Teacher Assessment Resources for Monitoring and Improving Instruction

TARMII 3.0 Quarterly Progress Report FY 2018 (January-March 2018)

Project Title: TARMII 3.0

Cooperative Agreement Number: AID-674-A-12-00023 Agreement Start and End Dates: 01/04/2016 – 31/03/2019 Project Locations: South Africa (Gauteng and North-West Provinces) Donor : USAID

Contents 1. INTRODUCTION ...... 4 2. PROJECT ACTIVITIES ...... 4 2.1 Project Management and Coordination ...... 4 2.2 TARMII 3 Software Development ...... 4 2.3 Item Improvement and Development ...... 5 2.4 Performance Evaluation ...... 7 2.5 Teacher Professional Development ...... 9 3. CONCLUSION ...... 14

Acronym List

CAPS Curriculum Assessment and Policy Statement DBE Department of Basic Education DG Director-General HOD Head of Department HSRC Human Sciences Research Council ICT Information Communication and Technology ME Measurement and Evaluation TARMII 3.0 Teacher Assessment Resources for Monitoring and Improving Instruction FDSD Functional Design Specification Document HL Home Language

1. INTRODUCTION

This report provides a narrative on the TARMII 3.0 project activities for the period, January to March 2018. The main highlight of this reporting period, was the training of Foundation Phase teachers, HoDs, District Officials and Provincial Coordinators on the TARMII 3.0 System in the North-West and Gauteng provinces as well as the commencement of school-based support for teachers in the North-West province.

2. PROJECT ACTIVITIES

The project activities during this period include project management and coordination; further modification of the TARMII 3 software components; finalisation of item review and quality assurance processes; development of Grade 3 formal tests; the advertisement of tender for grade 4 and 6 item development; training of Foundation Phase teachers, HoDs, District Officials and Provincial Coordinators on the TARMII system; and undertaking of the baseline in selected English medium primary schools in North-West and Gauteng provinces.

2.1 Project Management and Coordination

Two full time personnel were appointed on the TARMII project from January 2018. The TARMII Project Manager, Dr Joseph Kivilu was appointed on a one year contract from 10 January 2018 – 10 January 2019. He was appointed to manage the TARMII project by working closely with the DBE in order to bring synergy between personnel in the various directorates within the DBE and the HSRC. The second appointee was Michelle Clarke, who was appointed as the TARMII Training and Resource Development Coordinator on an 11 month contract, from 10 January to 10 December 2018. Her responsibility is to manage the assessment review process, facilitate training of teachers and district officials, and coordinate the school-based coaching of teachers in the use of the TARMII system. The two staff members have joined the TARMII 3.0 team with vigour and are deeply involved in the activities of the project.

2.2 TARMII 3 Software Development There was continuous testing and modification of all components of the software. These modifications were as a result of inputs received during the training of teachers as well as at the working session with the DBE team held on 14 March 2018. The modifications included providing an animated indicator to show the test build process, additional message pop-ups,

4 securing the reference material in a test, adding the Foundation Phase font file to the system, developing a print facility for learner tests and adjusting the Test Builder navigation process. The next important aspect of this process is also to ensure that the TARMII 3.0 system is ready for mass traffic.

Working session with the DBE The HSRC scheduled a TARMII working session with the relevant units at the DBE. This was held at the HSRC on 14 March 2018 and attended by five personnel from the Examinations and Assessment unit. The aim of the session was to provide the DBE with an opportunity to engage with the system in order to determine if it meets the requirements of the DBE. The DBE officials including Dr Rufus Poliah voiced their satisfaction with the system and suggested the following for consideration: The system should have a Test Builder which will be based on a Test Framework. The system should allow for bulk printing of an entire class test. The open ended questions should allow for marks and comments. The teacher should have more control over the learner administration process.

During this period the Software Developer completed the following tasks: Modifying and re-coding a number of functions of the software. Preparing the hosting server. Developing a load testing plan to simulate one million concurrent users of the TARMII site. Performing smoke tests - these tests ensure that the vital functions of the software work and that the system is stable. Conducting load tests and generating blockages of the system so as to optimize its functionality.

2.3 Item Improvement and Development

The activities related to item improvement and development of English Home Language (HL) entailed quality assurance and approval of items by the DBE, uploading and cleaning of approved items for publication as well as the development of formal assessments. By end of March 2018 a total of 2597 items were completed and loaded on the system. Table 1 below shows the number of items uploaded and then published onto the system.

Table 1: Uploading and publishing of items Grade 1 2 3 Term 1 219 items published on TARMII 111 items published on 227 items published on system TARMII system TARMII system Term 2 169 items uploaded and 211 items uploaded and 300 items uploaded and published on the TARMII published on the TARMII published on the TARMII system system system Term 3 173 Items published on the 245 Items published on the 226 Items published on the TARMII system TARMII system TARMII system Term 4 149 items uploaded onto the 223 uploaded and published 344 uploaded and published TARMII system onto the TARMII system onto the TARMII system Total 710 790 1097

Development of formal assessments Following the development of Grade 3 Term 1 assessments in December 2017, the HSRC submitted the draft tests consisting of 30 items for Form A and 29 items for Form B to the DBE for quality assurance. The DBE proposed that a working session be held with the three Gauteng-based test developers to show them how they should respond to the comments made by the DBE. This session was held on the 18th of March 2018 and was facilitated by Ms Devagie Maistry of the DBE. Modifications were made to the assessments and resubmitted to DBE for quality assurance and final approval.

Testing of Grade 3 items In preparation for testing of Grade 3 items at the sampled 21 schools in North-West and Gauteng, we have selected items from the item bank in terms of content, level of difficulty and cognitive levels to administer to Grade 3 learners in May 2018. The data collected will be used to compute psychometric characteristics to determine the quality of the test items.

Development of items for Grades 4 & 6 The advertisement for a service provider to develop English language items for Grades 4 and 6 was placed on the Government Tender Bulletin on the 16th of March 2018 and a briefing session held on the 22nd of March 2018. . Unfortunately, only two potential service providers attended the compulsory briefing. A close look at their profiles revealed that they did not have the requisite expertise to do the job satisfactorily. Thus, a decision was made by the Supply

Chain Management team of the HSRC to cancel the tender and initiate a fresh advertisement. A second advertisement will be placed in the Government Tender Bulletin in April 2018

2.4 Performance Evaluation

A sample of 21 Gauteng and North-West English medium schools were identified as TARMII ready schools in terms of availability of laptops/desktops for teachers and learners and strong internet connection for use in teaching and learning. Following the identification of project schools, the Director-General of the DBE, Mr HM Mweli wrote a letter to the HoDs of each province, namely, Ms SM Semaswe, Acting HoD North-West and Mr E. Mosuwe Superintendent-General Gauteng informing them about project implementation in the identified TARMII schools. The HSRC received copies of the letters to schools in February 2018. The HSRC made contacts with the identified schools to conduct a baseline on assessment and use of ICT in schools. The following questionnaires were administered in schools from 9 February to 23 March 2018:

a) School questionnaire on ICT and assessment practices- this was completed by principals. It required information about ICT integration in teaching and learning; the perception of the principal on the use of ICT for teaching and learning in the school; expenditure on ICT at the schools; teacher professional support and development in ICT and the challenges schools faced with the implementation of ICT

b) Teacher questionnaire on ICT and assessment practices – this was completed by Grade 3 teachers and required information on classroom assessment practices, ICT in teaching, learning and assessment; teacher professional development in ICT and challenges faced by teachers with the implementation of ICT.

c) School-based infrastructure survey to determine the status of ICT at the school – this was completed by teacher responsible for ICT at the school.

Table 2 and 3 show list of schools visited for baseline data collection.

Table 2: List of North-West schools Name of school District 1 Rustenburg-Noord Bojanala 2 Swartruggens Bojanala 3 David Brink Bojanala 4 Sonop, Bojanala 5 Pansdrift, Bojanala 6 Tlhabane West Bojanala

7 Fields College Bojanala 8 Lichtenburg Bojanala 9 Platinum Village Bojanala 10 Sol Platjee Ngaka-Modiri Molema 11 Diphetogo Ngaka-Modiri Molema

Table 3: List of Gauteng schools Name of school District 1 Bedfordview Ekurhuleni North 2 Blairgowrie Johannesburg North 3 Mogobeng Ekurhuleni South 4 Chief Luthuli Ekurhuleni North 5 Laerskool Theo Wassenaar Johannesburg central 6 Laerskool Voorbrand Johannesburg central 7 Rientondale Tshwane North 8 Robert Hicks Tshwane North 9 Braamfischerville Johannesburg West 10 Glenhazel Johannesburg East

Out of the 11 North-West schools visited, two schools, namely, Rustenburg-Noord and David Brink refused to participate in the project stating that the use of technology in teaching and learning is not a priority in their schools.

Out of the 10 Gauteng schools which were identified as TARMII ready in November 2017, it was found during the baseline data collection that the TARMII cannot be implemented in five of the schools (i.e. Blairgowrie, Braamfischerville, Chief Luthuli, Laerskool Voorbrand, Mogobeng primary schools) because of the following reasons: loss of internet connectivity loss of laptops in December 2017 due to theft and the district removed laptops for use by learners due to theft occurring in some of the schools. The TARMII team sought to replace the five schools. This was done through the Gauteng TARMII Provincial Coordinator, Ms Thelma Morake of Examinations and Assessments. The new replacement schools, namely, Edleen, Emmarentia, Meredale, Tamaho, Tswelelo and Robertsham primary schools were identified before the 17th of March training and were invited to the training session. Baseline data was collected at the six schools from the 16th - 20th of April 2018.

2.5 Teacher Professional Development

This component entails training of all Grade 3 computer literate teachers and Foundation Phase school HoDs, principals, provincial and district education officials on the TARMII 3 online system as well as school-based support provided to Grade 3 teachers and their HoDs.

Six hours once-off training sessions on the TARMII system were scheduled for both Gauteng and North-West schools. These sessions took place on the following Saturdays: 10 February, 24 February and 17 March 2018. Training in the Gauteng province was centralised while in the North-West it was district-based. All training sessions were done in hotels that had strong internet connectivity to ensure smooth running of the training. The HSRC hired laptops for use during Gauteng and North-West Bojanala district training sessions. The North-West Ngaka Modiri Molema district provided laptops and ensured that chrome was installed on all laptops ahead of the training. All training sessions were facilitated by full time HSRC staff and participants trained on the following aspects of the TARMII system:

Why do we assess learners and why do we use a rubric? TARMII access and registration - how to access the TARMII website and register as the teacher; Website navigation - navigating through the TARMII website and all its functions; Learner management - creating and managing the class list; Building a test – accessing the item bank and creating a test; Publish test- preparing the final test and making it available to learners to take the test; Assess learner test – retrieving completed and submitted tests from learners and finalising the marking; Interpreting reports – accessing, reading and interpreting learner performance reports; Use of resources – accessing and using the various video and PDF resources.

At the end of the training session, each training attendee received a TARMII user manual which would help to guide them on the use of the system. Furthermore, each teacher was given a practical activity to go through all the functions of the system when they return to their respective schools.

Training in the North-West Province: Ngaka-Modiri Molema district

The TARMII 3 software training took place in Mahikeng, on Saturday, 10th of February 2018. A total of 16 school, district and provincial officials participated in training. Teachers and HoDs were from the three schools identified in the district, namely Diphetogo, Sol Plaatje and Lichtenburg primary schools. In addition to teachers and their HoDs, a principal from Sol Plaatje primary school, the E-Learning District Coordinator, the Provincial E-Learning Coordinator and two Provincial Officials from Examinations and Assessment units participated in the training session.

Training in the North-West Province: Bojanala district

Training in Bojanala district was held on the 24th of February 2018 in Rustenburg. The following six schools were represented: Swartruggens Intermediate, Tlhabane West, Fields College, Sonop, Pansdrif, and Platinum Village. A total of 18 teachers, the district official and the provincial Mahikeng sub-district official participated in the training.

Training in the Gauteng province

This centralised training took place on Saturday, 17th of March 2018 in Midrand. The following eleven schools were represented: Bedfordview, Glenhazel, Edleen, Theo Wassenaar, Emmarentia, Robertsham, Rietondale, Meredale, Tamaho, Tswelelo and Robert Hicks primary schools. A total of 36 teachers and their HoDs, three school principals and five education officials involving two Gauteng Provincial officials and district officials from Gauteng West, Ekurhuleni North and Johannesburg East districts participated in training.

Evaluation of training on the TARMII system

Teachers, principals and provincial and district education officials who participated in the training sessions held in Gauteng and North-West provinces on 10 February, 24 February and 17 March completed a training evaluation tool. A total of 72 participants completed the evaluation tool. The Table below provides a summary of results from an evaluation tool used to assess the perception of teachers, principals and district/provincial officials about the TARMII system. Overall, a positive perception about the TARMII system is noted from teachers followed by district and provincial officials then school principals. In terms of creating a class list on the system, 77% of teachers agreed that this was an easy task. This is in line with what we observed in training sessions. Some of the teachers in Gauteng went ahead and

10 captured class lists of Grade 3 classes. For example, a principal of Tswelelo primary managed to complete capturing class lists during training.

Table 4: Responses from training respondents

Teacher Principal District/Provincial response (n=55) response Officials response Itemised responses (%) (n=4) (n=13) (%) (%) Ease to create and manage a class list on the TARMII system Disagree 2.2 0.0 2.2 Unsure 0.0 0.0 2.2 Agree 77.8 4.4 11.1 Ease to build an assessment test (both OEQ and MCQs) Disagree 2.2 0.0 0.0 Unsure 11.1 0.0 6.7 Agree 64.4 4.4 11.1 Managed to mark and score items easily on the system Disagree 2.2 0.0 0.0 Unsure 6.5 0.0 4.3 Agree 69.6 4.4 13.1 Ease to access remedial resources. Disagree 4.3 0.0 0.0 Unsure 15.2 4.3 10.9 Agree 58.7 0.0 6.5 TARMII 3.0 will save me time when preparing assessments Disagree 4.3 0.0 0.0 Unsure 10.9 0.0 2.2 Agree 65.2 4.3 13.0 TARMII 3.0 system has an appropriate balance of visual, audio and text options for teacher support. Disagree 4.3 0.0 0.0 Unsure 4.3 0.0 0.0 Agree 70.2 4.2 17.1

With regard to building an assessment test from the TARMII system, 64 percent of teachers, 11 percent of district and provincial officials agreed that it was easy to build a test. We observed a slight deviation with the ease to access remedial resources with only 59 percent of teachers agreeing to having accessed remedial resources. The TARMII system was also reported to have an appropriate balance of visual, audio and text options for teacher support. We noted that 70 percent of teachers and 17 percent of district and provincial officials agree that TARMII has appropriate visuals. Sixty five percent of teachers believe that the TARMII system will save them time when preparing assessments.

School-based support and monitoring in the North-West

During the current reporting period, school-based support and monitoring on the use of the TARMII software took place in Ngaka Modiri Molema district on 19th and 20th February and in Bojanala district on the 6th – 8th March 2018. These support sessions were provided by full time HSRC staff. During the support we got teachers to log in to their computers to check if they managed to complete the assignment given to them at the training session. We found that not all teachers had completed the task. They mentioned that they could not find time to work on the system and some had problems with internet connectivity. In instances where teachers had not accessed TARMII since training, we provided on-site refresher training and had focus group interviews with them to determine the challenges and possible successes that they may have in the use of the TARMII system at their schools. School-based support and monitoring in the North-West province revealed the following:

Possible successes with TARMII implementation TARMII has the potential to help learners who cannot write but are able to type using the keyboard. A teacher at Lichtenburg primary said that although internet connection is not always good, she wanted to use the TARMII system. She planned to administer paper-based tests accessed from the TARMII system and online tests to the learners to determine learner performance when using both paper-based and on-line methods. If the results show that some of the learners perform better when tested using the TARMII online system, the school will write to the district requesting concession to allow some of the learners to take tests using computers. She undertook to share the results with the HSRC.

The interest of teachers and the HoD at Diphetogo primary school prompted the principal to purchase a Wi-Fi router. The principal informed us that he, the Chairperson of the School Governing Body and the district E-Learning Coordinator went together to purchase the modem for use in the TARMII project. Since the school is a no-fee school, the principal called a parents meeting on 18th February to request parents to support implementation of TARMII by paying for monthly data to run TARMII. On the day of our visit, 20th February we also trained two teachers who were involved in ICT at the school. Before we left the school, we noticed that the ICT teachers together with the district E-Learning Coordinator were working on the laptops and labelling

them for use by the learners. The support that the district E-Learning Coordinator offers to participating schools has potential for success of the TARMII project

Techers from Platinum village in Bojanala were preparing to train other Foundation Phase teachers on the use of the TARMII system so that they programme can be used in the entire Foundation phase

The HoD and teachers of Tlhabane West primary in Bojanala indicated that they would like to implement TARMII at their school. The District Subject Advisor, Ms Vetshe was present during the support session.

Challenges with TARMII implementation In Bojanala district, teachers from four schools, namely, Fields college, Pansdrift, Sonop, and Swartruggens indicated that they had not used TARMII since returning from training and were unlikely to use it due to;

The school does not have strong internet connection to allow for learners to take the test on-line and would not consider strengthening internet connection due to funding. Teachers are keen to use the system but they would log on to TARMII at home to access tests and administer the test through pen and paper. One teacher mentioned that TARMII is not for now but for the future. The focus of the project should be targeted at high schools and not primary schools.

The school has prioritised ICT for the senior and FET phases and has set aside funding for building a school hall. Therefore, the use of technology in teaching and learning is not a priority at this stage.

TARMII would be useful if only used to draw assessment items and administer them through pen and paper. TARMII would benefit children from enriched home environments where technology is used in their homes. One teacher said: “TARMII would not work in the foundation phase”

TARMII was not discussed at the management meeting after training because it is not a priority at the school

Although none of the teachers in Bojanala had accessed the TARMII system after training, Tlhabane-West and Platinum Village primary schools are the only two schools that are keen to continue using the TARMII system.

Following the visits the HSRC decided to get replacement schools to test the use of TARMII as an online system. We contacted the Bojanala district official, Ms Vetshe to identify English medium schools that meet the TARMII requirement. We received the following names of schools: Sunriseview primary, Zinniaville combined, Central primary and Kloofview primary. We visited the four schools in April and plan to train teachers in May 2018. The principal of Kloofview offered to provide the training venue and computers for use during training.

3. CONCLUSION

Despite a few challenges experienced with the uptake of the project by schools, we have made tremendous progress in implementing the project. The TARMII 3.0 system is almost complete and we are engaging with the DBE to start the transition process for the handover of the TARMII system to the DBE. The information we received from teachers and education officials from the field has helped us to fine tune the system to improve on its functionality. Although item development has been very slow we have set targets to ensure that the Item development team is able to deliver quality items.

Training of teachers and education officials in both NW and Gauteng was very successful and we are currently providing school based support. It was evident during school based support that there is a need for teachers to be allocated time in the computer labs for TARMII. The evaluation and monitoring component is providing feedback to the other project components based on field observations.

APPENDICES Planned Activities for April – June 2018 Activities 2018 Project Components April May June TARMII 3.0 software Continue with modification of the TARMII system Preparing the system for Grades 4 and 6 item development Teacher Development School-based support and monitoring in North-West and Gauteng schools Item improvement and development Uploading and publishing of Grades 1 and 2 Term 2 items and remaining Term 4 items onto the TARMII system Finalisation of Term 1 and 3 formal tests Development of Term 2 and 4 formal tests Administration of the Grades 3 pre-test at selected schools Performance Evaluation Conduct baseline in a total of 9 replacement schools in Gauteng and North-West primary schools Analysis of baseline data and report writing

Performance Monitoring Plan: January –March 2018

FY End of Performance Indicators 2016 FY 2016 FY 2017 FY 2017 FY 2018 FY 2018 FY 2019 FY 2019 Baseline Project (quantitative)1 Targe Actual Target Actual Target Actual Target Actual Target t Objective 1: modification to increase the functionality of the TARMIIfp software Indicator 1.1 User Requirement 0 1 1 1 1 1 1 Specification (URS) document Indicator 1.2 Functional Design 0 1 1 1 1 1 1 Specification (FDS) document Indicator 1.3 Topic Tree Platform 0 1 1 1 1 1 1 Indicator 1.4:User Registration 0 1 1 1 1 1 1 module Indicator 1.5 Item Bank 0 1 1 1 1 1 1 functionality Indicator 1.6 Word Upload 0 1 1 1 1 1 1 module Indicator 1.7 Number of TARMII 0 3 0 3 3 3 3 software user manual developed Objective 2: Item Improvement and Development Indicator 2.1 Number of TARMII English assessment items 0 1300 0 2700 2754 reviewed Indicator 2.2 Number of English assessment items quality assured 0 1300 0 3000 1917 2597 by the DBE Indicator 2.3 Number of English 0 0 0 0 0 0 0 items versioned

1 This is the quantitative description of the indicators. The baseline for future measurements is established at the beginning of the project. Targets for all out years and for the end of the project are also established at the beginning of the project. However, the PMP will be updated at the time of the work plan with previous FY actuals and updated targets for out years, if relevant. 17

Objective 3: Teacher Professional Development Component Indicator 3.1Number of training 0 2 0 2 2 2 2 2 materials developed Indicator 3.2 Number of 0 60 0 60 55 60 teachers trained Indicator 3.3Number of teachers receiving on-going school-based 0 60 0 42 55 60 support Indicator 3.4 Number of HoDs 0 20 0 22 20 trained Indicator 3.5Number of 0 10 0 10 5 10 Curriculum Advisors trained Indicator 3.6 Number of e- Learning Specialists and Teacher 0 6 0 6 2 6 Centre Managers trained Indicator 3.7 Number of leaners 800 0 800 800 participating in the program Objective 4: Performance Evaluation Indicator 4.1 Baseline to determine teacher assessment and teaching practices using the 0 0 0 60 0 60 60 software (Gauteng and North- West) Indicator4.2 End-line survey to determine the impact of training 0 0 0 60 0 60 60 and ongoing support on teacher assessment and teaching practices