Technical Assistance Consultant’s Report
Project Number: 45052 Regional–Capacity Development Technical Assistance (R–CDTA) August 2015
Quality Primary Education in the North Pacific (Financed by the Japan Fund for Poverty Reduction)
Prepared by: Development Strategists International Consulting Incorporated
This consultant’s report does not necessarily reflect the views of ADB or the Government concerned, and ADB and the Government cannot be held liable for its contents. (For project preparatory technical assistance: All the views expressed herein may not be incorporated into the proposed project’s design.
FINAL REPORT
TA 8066 REG: QUALITY PRIMARY EDUCATION IN THE NORTH PACIFIC
Submitted by
DEVELOPMENT STRATEGISTS INTERNATIONAL CONSULTING, INC. (PHILIPPINES)
15 August 2015
ABBREVIATIONS
ADB Asian Development Bank CDTA Capacity Development Technical Assistance COFA Compact of Free Association DMC Developing Member Countries DMF Design and Monitoring Framework DOE Department of Education DRE Division of Research and Evaluation DSIC Development Strategists International Consulting, Inc. EGLA Early Grade Learning Assessment EGLA-L Early Grade Learning Assessment – Literacy EGLA-N Early Grade Learning Assessment – Numeracy EGRA Early Grade Reading Assessment EMIS Education Management Information System FGD Focus Group Discussion FSM Federated States of Micronesia JFPR Japan Fund for Poverty Reduction JICA Japan International Cooperation Agency MOE Ministry of Education MOU Memorandum of Understanding NDOE National Department of Education PAT Palau Achievement Test PERA Palau English Reading Assessment PLAN-A Primary Literacy and Numeracy Assessment PLD Professional Learning Development PREL Pacific Resources for Education and Learning QAT Quarterly Assessment Test QPENP Quality Primary Education in the North Pacific (“the Project”) PAL Palau (Republic of) RETA Regional Technical Assistance RMI Republic of the Marshall Islands SABER System Assessment and Benchmarking for Education Results SDOE State Department of Education SAL Short Assessment for Language SAM Short Assessment for Mathematics TA Technical Assistance TAR Technical Assistance Report TEGRA Tonga Early Grade Reading Assessment TOR Terms of Reference US United States (of America) WB World Bank
TA 8066: REG - Quality Primary Education in the North Pacific 3 Funded by Asian Development Bank
EXECUTIVE SUMMARY
TThe Quality Primary Education in the North Pacific (QPENP), a regional Capacity Development Technical Assistance (CDTA) was the first initiative by the Asian Development Bank (ADB), particularly in the Pacific Region to support the development of a new learning assessment tool for use in primary education grades. The three North Pacific countries included in the regional CDTA – Federated States of Micronesia (FSM), Republic of the Marshall Islands (RMI) and Republic of Palau (PAL) – requested the Asian Development Bank to support implementation of Early Grade Learning Assessment (EGLA) in their respective countries1. This regional CDTA is financed by the Japan Fund for Poverty Reduction (JFPR) administered by ADB.
The CDTA also intended to address relevant issues currently faced by the education sector in the Pacific countries particularly in improving educational achievements and measuring learning outcomes that will inform better approaches to learning basic skills in literacy and numeracy and eventually impact on improved access to quality education for primary students in FSM, RMI, and Palau.
To carry out this CDTA, the ADB has engaged the services of Development Strategists International Consulting, Inc. (DSIC), Philippines, hereafter referred to as the Managing Consultant or “Consultant”. The project was implemented from September 2013 to July 2015.
Summary of Major Achievements
The major achievements of the CDTA include the following:
i. Development and piloting of the EGLA Literacy and Numeracy to 8 pilot and 2 non-pilot schools in FSM and 5 pilot and 2 non-pilot schools in RMI. Two baseline surveys were conducted, one in February 2014 and another in February 2015.
ii. All the 13 pilot schools in both RMI and FSM included in the project were provided comprehensive professional development programs for teachers to implement individualized and group learning approaches to improve literacy and numeracy learning. A total of 823 students were assessed in February 2015 as part of the baseline pilot surveys of EGLA.
iii. Three major PD programs were implemented in November 2013, June 2014 and October 2014. A total of 521 teachers were trained – 164 in November 2013, 114 in June 2014 and 243 in October-November 2014.
1 Asian Development Bank, March 2012, Technical Assistance Report: Primary Quality Education in the North Pacific. (para. 1).
TA 8066: REG - Quality Primary Education in the North Pacific 4 Funded by Asian Development Bank
iv. Reprinted over a hundred of literacy materials in the first languages (Marshallese, Kosraean, and Pohnpeian) and in English. These learning materials are now set up in reading corners of the pilot schools. Additionally, math kits and manipulatives were also distributed to all pilot schools in the three project sites. These materials were distributed in June 2014 and November 2014 during the PD for teachers. With these materials, the pilot schools were able to set up reading and numeracy corners in their schools.
v. The EMIS system of RMI was reviewed and an EMIS proposal was submitted to RMI MOE that eventually sought support for US funding. A Workshop on EMIS was conducted to FSM EMIS Staff in Ponhpei in July 2014. The EMIS Staff of the four states of FSM attended the workshop.
vi. The EGLA Results were shared and presented in workshops and conference in:
° Pacific Early Age Readiness and Learning (PEARL) in Nadi, Fiji on May 4-6, 2015
° PACIFIC ISLANDS BILINGUAL BICULTURAL ASSOCIATION (PIBBA) in Kosrae, FSM on June 23-25, 2015
° Micronesia Teacher Education Conference (MTEC) in Ponhpei, FSM on July 21-24, 2015
° Pacific Educational Conference in Majuro, RMI on July 28-31, 2015.
vii. A review of the assessment system of Palau was completed and the Final Report was submitted in January 2015. A workshop to present the results of the review was held in October 2014, and a follow-up capacity building program for MOE to draft the Palau National Student Assessment Policy Framework was conducted on July 14-16, 2015.
viii. The TA also supported RMI in drafting their Bilingual Education Policy. The policy was enacted by the Government of the Republic of the Marshall Islands.
ix. Prepared two knowledge products – a 12-page Informational Booklet on the QPENP and a three-minute video of the EGLA. The 12-page booklet was distributed during the conference and workshops held in PIBBA, MTEC and PEC.
Conclusions
The pilot project has shown very promising results in a very short timeframe. The motivation to continue this effort is evident from the feedback received from the participants. It has also become clear that, with specific and focused efforts that
TA 8066: REG - Quality Primary Education in the North Pacific 5 Funded by Asian Development Bank
increase understanding of actual student learning while improving teaching practice, it is possible to effect significant positive impacts in learning outcomes.
The challenge is to systematize and institutionalize such efforts more broadly. It must be acknowledged that other contextual factors may have influenced the positive gains experienced within such a short period of time. Nevertheless, the momentum and commitment generated warrants the project’s further support, as it has shown new ways of approaching assessment and providing support to teachers and student learning.
Future efforts would ideally focus on more institutionalization of teacher professional development, including its linkages to pre-service education, while ensuring more bilingual and other targeted learning resources in this effort.
Sustainable change in teacher practice through deeper conceptual and pedagogical understanding takes time. Community and parent support will also be key to promoting more comprehensive, accountable, and transparent ways to improve the quality of education for the future young people in the North Pacific. By maintaining a strong focus on providing critical foundations in literacy and numeracy, it is hoped that many more young people will succeed throughout their education and in life.
Issues and Recommendations
As with any project, there were a number of issues that arose over the life of QPENP which required attention from all major stakeholders. These included scheduling issues; local capacity; and communication. None of these issues, however, have had major negative impacts on the final outputs of the project and have been able to be resolved through dialogue between ADB, MOE/DOE and the consulting team. At all times, the focus has been on achieving project outcomes and solutions were found to ensure that these were met efficiently.
Issue: The implicit intention of the project is to provide capacity building to the MOE/DOE staff, particularly in developing the EGLA and designing and implementing professional development of teachers. The implementation of the EGLA, that is, conducting the assessment tool, called for a large number of MOE/DOE staff and the skills and experience required were not always readily available. The Consultants organized series of formal and informal learning sessions with the staff to ensure the development of knowledge and skills to properly implement EGLA when the project ends. However, more pressing is to expand the EGLA to all schools in a phased process. For instance, for Pohnpei and Majuro, they intend to expand EGLA beginning with 2 schools, and then 2 or 3 schools may be added each year. There was a concern about the time and resources required, hence, the need for phasing.
Recommendation 1: That a follow up Technical Assistance be provided to scale up EGLA beyond pilot schools and project sites. All beneficiaries of this project strongly articulated the need for a follow-up Technical Assistance to expand and scale the use of EGLA. It was suggested that cluster approach to implementation would be introduced as an approach in the scaling up of EGLA using random samples of students. A Technical Assistance, which could be provided by ADB and/or other development partners may
TA 8066: REG - Quality Primary Education in the North Pacific 6 Funded by Asian Development Bank
be sought to ensure that the achievements and success gained from the project, particularly EGLA, be sustained. Through this TA, EGLA will be further implemented to other schools beyond the pilot schools and control schools included in this project, as well as expanding it to other states in FSM and other atolls in Majuro. Furthermore, the TA would intend to enhance the capacity of MOE and NDOE to implement EGLA by institutionalizing EGLA into their assessment system, thereby requiring capacitating the assessment and data management units of the MOE and NDOE. The proposed EGLA TA would also include training of MOE/NDOE staff in monitoring and training of more assessors. Specifically, the following are the activities that may be implemented under the suggested follow-up TA:
i. Develop a plan to implement and monitor implementation of a possible second QPENP.
ii. Prepare an EGLA Training Manual for literacy and numeracy that addresses the need for standardized administration and scoring system.
iii. Develop a larger assessment item bank for literacy and numeracy in both languages – (i) items for training assessors, (ii) items to be used as practice sets for students, (iii) items that can be made available to use as formative tools; and (iv) items that will be kept for Department/State-wide level use.
iv. Comprehensive training program for EGLA (or other assessment) data management and analysis, using available software packages in order to ensure that assessment running records of students are maintained, monitored to inform policy decisions related to curriculum, assessment and evaluation.
v. Develop learning resources, particularly to support the language education policy focused on writing to read and numeracy.
vi. Support and strengthen department/ministry leadership in literacy and numeracy development.
Issue : The design of the project was to provide professional development to support the literacy and numeracy learning. The project also triggered the formation of learning communities through engagement of school principals in all the pilot schools. The PD were designed and implemented with the principals being involved very heavily. However, the beneficiaries would like to see more principals and teachers collaborate in the activities, not only in EGLA implementation but also in designing and implementing professional development programs. They also want a TA to institutionalize professional development for curriculum officers, subject specialists, teachers, and principals and a support to develop a monitoring and evaluation of teachers’ performance and diagnostic tools for teachers with corresponding training program for implementation.
Recommendation 2: A support to enhance quality pedagogy through improved teacher training and education – both pre-service education and in-service teacher training. There is a strong need to institutionalize quality pedagogy by supporting not only in-
TA 8066: REG - Quality Primary Education in the North Pacific 7 Funded by Asian Development Bank
service teacher training programs (INSETT) in the Pacific countries, but also pre-service teacher education and training (PRESET). While the College of Micronesia (COM) in FSM and College of the Marshall Islands (CMI) have been providing pre-service education and training, there is a need to modernize the PRESETT programs focusing on the present issues surrounding the education system in the Pacific countries. Literacy and numeracy levels are still struggling compared to other regional neighbors. There is a need to develop a PRESETT and INSETT programs that blend and integrate literacy and numeracy in the core learning areas. Likewise, the emphasis on bilingual education is becoming a prominent issue in the school system, the need to train would-be teachers to teach in a bilingual learning environment and to provide learning resources that promote bilingual skills development – that is, becoming competent in their own language (such as Marshallese, Pohnpeian, or Kosraean) and likewise in the English language. Moreover, assessment is becoming an integral part of the curriculum development and implement, hence, the PRESETT and INSETT should also be developed in such as way that assessment of skills will be improved in order to support the EGLA implementation more effectively. Specifically, the institutionalization of quality pedagogy through a supported technical assistance should include the following activities:
i. Develop a policy on PRESETT and INSETT that supports the implementation of the six components of quality pedagogy addressing the needs from department staff to classroom teachers.
ii. Support both COM and CMI (and maybe Palau Community College) in reviewing their pre-service programs and provide support in terms of capacity building and organizational development of teacher training unit or department.
iii. Support the enhancement of the pre-service curriculum to include the teaching of bilingual students, how to teach L1 (Marshallese, Kosraean, Ponhpeian, or even Palauan for Palau) and teaching numeracy in a bilingual approach.
iv. Strengthen learning communities where ministry/department level staff work closely with school principals, and principals working closely with teachers as well as to engage community and family. The PD that includes the engagement of community and family will ensure that culture and language will be enhanced and preserved.
v. Strengthen a communication cycle/system between parents and teachers, and between principals and the department.
vi. Develop a long-term plan for professional learning for teachers, including career path, salary, and professionalization of teachers. This activity may lead to the development of a Teacher Development Policy Framework.
vii. Expand existing INSETT or PD programs being implemented by the ministry or departments of education in the project sites.
TA 8066: REG - Quality Primary Education in the North Pacific 8 Funded by Asian Development Bank
Issue. One of the components of the project was focused on EMIS and data management of students’ assessment records. However, while this was not given much emphasis just like the piloting of EGLA and PD programs. Most of the technical assistance given through this project was to help the MOE for RMI to strengthen their EMIS and for capacity building and training for EMIS staff for FSM. This was the result of re-focusing of the project during the Inception phase. Hence, it is strongly recommended that a further support on EMIS and management of student assessment data be provided.
Recommendation 3: Strengthen EMIS and management of student assessment data and other school records. There is a clamor for a more extensive support to strengthen the EMIS and management of student assessment data. To the stakeholders of the project, they feel these two strategic programs would allow them to make better decisions and could inform better policy programming. Hence, it is suggested that a follow through of the EMIS support be provided with the following proposed activities:
i. Further review of the EMIS of FSM, particularly Kosrae and Pohnpei States.
ii. Develop a capacity building of state level staff on EMIS and data management of student assessment records. A short-term overseas training and study tour may be a good program for key staff of state DOE and MOE for RMI. If possible, support the development of EMIS Specialists in each state to ensure that data management will be handled properly.
iii. Provide a comprehensive training of NDOE staff to manage student assessment data, including training on scoring, analyzing and reporting of EGLA data and results. A better way of presenting EGLA results may be identified and NDOE staff will be trained accordingly.
iv. A school level training should also be designed to train teachers to analyze classroom assessment data such as student profiling, class profiling and school profiling. This way, even at the school level, the core standards may be determined using student assessment data and other information.
v. As originally conceptualized, it may be good to enhance the system for providing student unique identification numbers or Student Education Numbers or ID that will be used from Kindergarten to Grade 12.
Issue. While the project has provided learning resources by re-printing out-of-print learning resources developed under previous projects and other organizations, it is evident for the need to develop a more appropriate resources for literacy and numeracy that support the bilingual nature of the curriculum being implemented in the Pacific countries covered in this project. Moreover, the states and countries included in this project have been strengthening their bilingual education policy, that is, to institutionalize the development of competencies in both the national language/ vernacular and English language. Under this project, the RMI has been supported in the finalization of bilingual education policy.
TA 8066: REG - Quality Primary Education in the North Pacific 9 Funded by Asian Development Bank
Recommendation 4: Develop bilingual literacy materials and activities. There is a dearth of learning resource in the local language. While the project had re-printed materials in local language, these are not enough resource to support the literacy development in the local language. The teachers of Sansrik Elementary School in Kosrae, for example, have develop readers in Kosraean language and this could be further enhanced. Related to this, there is the need to institutionalize bilingual education through a development of bilingual education policy. Hence, the following activities under this recommendation are given as follows:
i. Train and encourage teachers to develop readers and numeracy learning materials in the local languages. The outputs of the teachers, just like those from Sansrik Elementary School, can be enhanced and published through a support of a TA grant or through the federal support funds.
ii. Support each state to develop a bilingual education policy and provide technical and financial assistance in implementing the policy.
Issue: Finally, the review of the assessment system of Palau has resulted in initially supporting the MOE of Palau to conduct an organizational diagnosis of the assessment and research unit – Division of Research and Evaluation (DRE). Also, a workshop was conducted to support the MOE to develop its National School Assessment Policy Framework. However, there is still a strong to support Palau in their assessment system to further improve and streamline their examination and assessment systems. Specifically, the following activities may be implemented through another TA:
i. Support in the finalization of the National School Assessment Policy Framework of Palau as well as its implementation, particularly in the streamlining of examination and assessment activities.
ii. Provide capacity building to the DRE who will eventually spearhead the implementation of the assessment policy framework.
iii. Provide capacity building to MOE Subject Specialists, Assessment Staff, Principals, and teachers in assessment – from state-wide assessment to classroom level assessment.
iv. Further support the MOE in upgrading its examination materials aligned with their curriculum vis-à-vis the national school assessment policy framework.
TA 8066: REG - Quality Primary Education in the North Pacific 10 Funded by Asian Development Bank
ACKNOWLEDGMENTS
The Development Strategists International Consulting (DSIC) would like to acknowledge and thank the following for their partnership and collaboration, dedication and ownership of QPENP activities:
• Hon. Minister Dr. Hilda C. Heine, Ministry of Education, RMI
• Hon. Secretary Dr. Rufino Mauricio, National Department of Education, FSM
• Hon. Minister Sinton Soalablai, Ministry of Education, Republic of Palau
• Director Joseph Villazon, State Department of Education, Pohnpei State, FSM
• Director Lyndon Cornelius, State Department of Education, Kosrae State, FSM
We would also like to express our thanks to the Chiefs of Curriculum, Curriculum Specialists, Special Education Specialists, Teacher Training Specialists, and Testing and Assessments in FSM and RMI who served as members of the EGLA Teams and Professional Development Teams. Without their support and cooperation, the development and piloting of EGLA as well as the conduct of Professional Development Programs of teachers could not been smoothly and successfully implemented. These teams worked closely with the DSIC Consultants to achieve the outcomes that are highlighted in this report.
The guidance provided by Ms. Chimi Thonden, Education Specialist of ADB has always been a guiding light to the conduct of this TA from the beginning to end.
DSIC also greatly appreciates the Management of MOE Palau, whose coordination and support has been instrumental to the success of the completion of the review of the assessment system of Palau.
The consulting team has been professional, flexible and committed to the success of this project. We would especially like to acknowledge Dr. Richard Gonzales, who very ably led the team. Likewise, the members of the consulting team – Dr. Robert Early, Dr. Diana Guild, Mr. Steve French, Dr. Marylin Low, Dr. Alan Parkes, and Dr. Gillian Tasker, whose ideas, flexibility and commitment realized the great success of this project.
We would finally like to express our gratitude to all of the other stakeholders in the education sectors in FSM, RMI and Palau, donor partners particularly JICA, curriculum officers, principals, teachers and students, who gave up their time to contribute to QPENP activities.
TA 8066: REG - Quality Primary Education in the North Pacific 11 Funded by Asian Development Bank
I. INTRODUCTION
A. Context and Background
2. The Quality Primary Education in the North Pacific (QPENP) Project, a regional Capacity Development Technical Assistance (CDTA) was the first initiative by the Asian Development Bank (ADB), particularly in the Pacific Region to support the development of a new learning assessment tool for use in primary education grades. The three North Pacific countries included in the regional CDTA – Federated States of Micronesia (FSM), Republic of the Marshall Islands (RMI) and Republic of Palau (PAL) – requested the Asian Development Bank to support implementation of Early Grade Learning Assessment (EGLA) in their respective countries2. This regional CDTA is financed by the Japan Fund for Poverty Reduction (JFPR) administered by ADB.
3. The CDTA also intended to address relevant issues currently faced by the education sector in the Pacific countries particularly in improving educational achievements and measuring learning outcomes that will inform better approaches to learning basic skills in literacy and numeracy and eventually impact on improved access to quality education for primary students in FSM, RMI, and Palau. Also, the EGLA outcomes in the three participating countries were envisioned to contribute to further comparative analysis in the region, in particular, building on the success of EGRA in Tonga and Vanuatu in the South Pacific.
4. The CDTA recognize that over the past years, the countries in the North Pacific region have made some advances in terms of systematic assessment of student performance. However, there are still significant challenges and constraints that need to be addressed to ensure more scientific knowledge solutions and best practices be put in place. Although all the North Pacific countries included in this CDTA have been conducting at least one national assessment to measure learning outcomes, particularly at the primary education level, it is acknowledged that there is still room for improvement in their respective assessment system. Moreover, while early grade reading assessment (EGRA) is ongoing in some Pacific countries in partnership with the World Bank, some of the North Pacific countries still had to introduce and enhance their programs towards more systematic assessment of early literary and numeracy.
5. Furthermore, this CDTA intended to support the introduction of the EGLA in grades 3 and 5 of selected primary schools in the states of Kosrae and Pohnpei in FSM and Majuro Atoll in RMI. The primary objective of this project was to assist the participating countries set up the process of systematically measuring how well children in the early grades of primary schools or education are acquiring literacy (reading and writing) and numeracy (mathematical arithmetic) skills, and drive more effective efforts to improve performance in these core learning areas through the introduction of EGLA. The
2 Asian Development Bank, March 2012, Technical Assistance Report: Primary Quality Education in the North Pacific. (para. 1).
TA 8066: REG - Quality Primary Education in the North Pacific 12 Funded by Asian Development Bank
successful experience of Tongan EGRA (TEGRA) was used in piloting EGLA in FSM and RMI. This TA used the theoretical foundation that the purpose of assessment in early grades is to support learning and that assessment sets the definition about what children can do and what they are ready to learn next3.
6. The CDTA assignment intended to provide support in, i) developing and piloting of EGLA in selected schools; ii) strengthening learning approaches through quality professional development of teachers; iii) review of the EMIS4; and iv) disseminating EGLA results regionally and for replication nationwide. Furthermore, the TA conducted an external review of the Student Assessments in Palau with the goal of enhancing its present assessment system5.
B. Profiles of Participating Countries
7. The country profiles of the three North Pacific countries are similar, yet they have distinct characteristics. A brief description of each country is given below as well as the map showing the North Pacific countries.
3 Shepard, L., Kagan, S.L.& Wurtz, R. (1998). Principles and recommendations for Early Childhood Assessment. 4 This output is in the process of being modified based on actual need of RMI and FSM because of some on-going similar initiatives in these countries. 5 Asian Development Bank: Summary of Minor Changes in Scope to RETA 3088
TA 8066: REG - Quality Primary Education in the North Pacific 13 Funded by Asian Development Bank
Federated States of Micronesia
8. The Federated States of Micronesia (FSM) is a federation of four states – namely, Yap, Chuuk, Pohnpei and Kosrae. Each state has its own government and identify. The physical isolation of the country and the distances between the four states pose particular development challenges6. These characteristics also add challenges in the coordination between state and national governments in reform efforts, and the effective use of development partner funding. It achieved its independence from the United States (US) through a Compact of Free Association (COFA) that gave the islands liberty to govern themselves but FSM is still dependent economically and foreign aid from the US.7
9. Just like other islands in the North Pacific, FSM predominantly lives through aid from foreign countries such as the US and Japan. It is reported that the former US Trust Territories receive around $100 million annually from the US alone. Due to the dependence from foreign aid, poverty and unemployment are at high levels. A large portion of the population lives on poverty level and lacking basic services, particularly clean water.
10. The education system of FSM is based on the colonial systems of German and Japanese education. The first program implemented in FSM was organized by the German colonizers. During World War II (WWII), the Japanese established their own public school system that led to an expansion in the number of schools in FSM. Right after the WWII when the compact between the US and FSM was made, the US gave more emphasis on the educational system of FSM. The US government implemented a program that doubles the government’s budget for education and led to an increase in the number of enrollees for all levels. An eight-year public school is mandatory for all students of FSM.
11. At present, the FSM Constitution provides a policy for decentralized educational system that enables citizens of the FSM to participate fully in the development of the islands as well as to become familiar with the Pacific community and the world. To this end, the purpose of education in the FSM shall be to develop its citizens in order to prepare them for participation in self-government and economic social development; to function as a unifying agent; to bring to the people a knowledge of their islands, the economy, the government, and the people who inhabit the islands; to preserve Micronesian culture and traditions; to convey essential information concerning health, safety, and protection of the island environment; and to provide its citizens with the social, political, professional and vocational skills required to develop the Nation.8
6 Asian Development Bank (2012). Federated States of Micronesia Fact Sheet. 7 BBC.(2013). Micronesia profile. Retrieved September 12, 2013, from http://www.bbc.co.uk/news/world-asia-pacific- 15494620. 8 FSM Const., art. IX, § 4, as amended.
TA 8066: REG - Quality Primary Education in the North Pacific 14 Funded by Asian Development Bank
Republic of the Marshall Islands
12. The Republic of the Marshall Islands (RMI) is located north of the Equator. The Marshall Islands are divided into two chains of coral atolls9 and islets - the Western Chain and the Eastern Chain. RMI was under US rule in the years after WWII and gained its independence in 1986 through a Compact of Free Association. This treaty indicates that the islands will govern themselves but will still receive military and economic aid from the US.10
13. Economically, in spite of the annual aid of roughly $65 million the RMI receives from the US, the islands do not have enough income. The inhabitants of the islands primarily live through fishing and farming.
14. The state schools in the Marshall Islands function under the supervision of the Ministry of Education, but now recently re-organized under the Public School System. There are 75 public schools in the country that serve about 10,500 students. The student- teacher ratio in the country is 24:1. Only a small percentage of the teachers in primary and secondary levels have gained college and graduate degrees.11
15. The Education Act emphasizes the development of the “student” as the primary goal for the development of education in RMI. The goals for the delivery of education in RMI are stated in the Education Regulations of the Education Act of 199112 and are summarized as follows:
i) The ability to think for one’s self, assuming responsibility for one’s own welfare in an unselfish manner.
ii) The aspiration to be productive and recognize that each individual has a part to play in society and must be willing to do one’s share towards contributing to the common good.
iii) The realization that self-reliance requires communication and cooperation with others and is not a totally independent or isolated endeavor.
iv) The recognition of the need for long term planning and the value of making immediate sacrifices to maximize future benefits and for the intelligent use of resources; understanding that the future is not really predictable and that the individual must develop skills and attitudes which enables him or her to adapt successfully to the changes that will occur in tomorrow’s society.
v) The concern for an efficient return on one’s labor and the realization of the economic potential of traditional subsistence skills.
vi) The ability to critically evaluate various lifestyles, behaviors, customs and social
9 Atolls are coral deposits on the crater rims of underwater volcanoes 10 BBC, 2013; Infoplease, n.d.. 11 Republic of Marshall Islands, n.d. 12 Education Act of 1991 and Education Regulations, RMI
TA 8066: REG - Quality Primary Education in the North Pacific 15 Funded by Asian Development Bank
mores, retaining traditional values whenever possible and modifying other behaviors to meet changes in society and the environment.
Republic of Palau
16. Republic of Palau is located southeast of the Philippines. It consists of about 200 islands. However, only eight of which are permanently inhabited. It became an independent nation in 1994 when it signed a Compact of Free Association with the US after 47 years of being a United Nations Trust Territory under the administration of the US.
17. Through the COFA, Palau has been receiving annual financial assistance from the US for general government operations plus additional support for postal, aviation, and weather services, as well as for infrastructure, education, health care, and other programs. Tourism is the main industry in Palau, with major visitors from Japan, Korea and Taiwan, which makes about 85% of total visitors to Palau. Palau’s form of government is democratic and led by a president.13
18. The educational system of Palau is led by the Ministry of Education. The country follows the 12-year educational program, also known as K-12 program in the US. Through its constitutional mandate and public laws, Palau’s education is free and compulsory for all children aged 6 to 17, or until graduation from high school. Approximately 18 schools provide primary education (Grades 1 to 8) to the students of Palau.
19. The Palau Education Master Plan 2006-2016 clearly states its aim to increase student achievement through improvement in governance and policy setting, curriculum and instruction, school administration and management, personnel management, and facilities and support services. These goals are anticipated to lead to the achievement of the vision of the MOE that “Our students will be successful in the Palauan society and the world. To accomplish this vision, the MOE has declared the following mission: The Republic of Palau Ministry of Education, in partnership with students, parents, and the community, is to ensure student success through effective curriculum and instruction in a conducive learning environment”.
13 Encyclopedia of Nations, 2013
TA 8066: REG - Quality Primary Education in the North Pacific 16 Funded by Asian Development Bank
Table 1. Summary of Country’s Profile and Information
FSM RMI Palau
Location Oceania, island group Oceania, two Oceania, group of in the North Pacific archipelagic island islands in the North Ocean, about three- chains of 29 atolls, Pacific, southeast of quarters of the way each made up of the Philippines from Hawaii to many small islets and Indonesia five single islands in the North Pacific
Nationality Micronesian Marshallese Palauan Chuukese Kosrean Pohnpeian Yapese
Capital Palikir Majuro Melekeok
Population 106,104 (July 2013 est.) 69,747 (July 2013 est.) 21,108 (July 2013 est.)
Language English (official and Marshallese Both Palauan and common language), English are official Chuukese, Kosraean, languages, but Pohnpeian, Yapese, Palauan is the local Ulithian, Woleaian, language. Nukuoro, Kapingamarangi
Literacy 89% of the total 93.7% of the total 92% of the total (definition: age 15 population population population and over can read Male: 91% Male: 93.6% Male: 93% and write) Female: 88% Female: 93.7% Female: 90% (1980 est.) (1999 est.) (1980 est.)
Government Constitutional Constitutional Constitutional government in free government in free government in free association with the association with the association with the US; Compact of Free US; Compact of Free US; Compact of Free Association entered Association entered Association entered into force on 3 Nov into force on 21 Oct into force on 1 Oct 1986 and the 1986 and the 1994. amended compact amended compact entered into force in entered into force in May 2004 May 2004
Source: Summarized from The World Fact Book www.cia.gov/library/publications/the-world-factbook/geos/
TA 8066: REG - Quality Primary Education in the North Pacific 17 Funded by Asian Development Bank
C. Objectives of the Final Report
20. The main objective of this Final Report is to describe how the project was implemented according to approach and strategy agreed among the ADB, MOE/DOEs and the Consultants. Moreover, this Final Report also discusses that major achievements of the projects as well as the challenges encountered will inform lessons learned and recommendations.
21. The Final Report also tries to account some milestones achieved during the process of implementation and towards the completion vis-à-vis the design and monitoring framework (DMF) put forward for this project.
22. This report also details the framework and methodology that were followed during the implementation of the CDTA, particularly the development of EGLA tools, conduct of professional development for teachers to support the teaching of literacy and numeracy, review of the Educational Management Information System (EMIS), and review and evaluation of the assessment system of Palau.
D. Organization of the Inception Report
23. The Final Report is organized and structured as follows:
I. Introduction
II. Overview of the Technical Assistance
III. CDTA Implementation Strategies and Work Plan
IV. The EGLA Baseline Survey Results
V. Review of the Assessment System of Palau
VI. Summary, Conclusions, Recommendations
VII. Annexes
TA 8066: REG - Quality Primary Education in the North Pacific 18 Funded by Asian Development Bank
II. OVERVIEW OF THE TECHNICAL ASSISTANCE
A. Introduction
24. In September 2010, representatives from 11 ministries of education of Pacific countries met in Papua New Guinea and agreed to support a new learning assessment tool for use in the early primary education grades.14 In another meeting of secretaries of education in the Pacific held in 2011, they further discussed the strategy on Early Grade Reading Assessment (EGRA) application and agreed to roll out EGRA in the region, utilizing the Tonga experience as a benchmark of success.
25. From those meetings and leveraging from the EGRA experiences in the Pacific and in other parts of the world, this regional CDTA has been initiated as the first project in the Pacific developing member countries (DMCs) to have an Early Grade Learning Assessment (EGLA) that includes both literacy and numeracy. The initiative was anchored on the experiences of EGRA in other Pacific countries and around the world and attempt to offer it as an option for the Pacific countries, particularly those in the North Pacific region.
26. Thus, a concept note of this CDTA was developed and eventually approved by the ADB Vice President (Operation 2) on 8 August 2011. Consequently, the TA was approved in March 2012. The main goal of this CDTA is to achieve improvement in the quality of primary education and to ensure that the EGLA outcomes in the two15 participating countries will contribute to further comparative analysis in the region.
27. The CDTA – the Quality Primary Education in the North Pacific was designed to develop and trial new inputs in learning assessments, teacher professional development, and data management to improve quality in the primary education in the northern Pacific Micronesian nations of RMI, the FSM, and to evaluate student assessment system in Palau.
28. The Project operated in the RMI in five selected schools on Majuro Atoll, and in the FSM, the project worked with all six schools in Kosrae State and two selected schools in Pohnpei State.
14 ADB, 2012. Technical Assistance Report: Quality Primary Education in the North Pacific. 15 The TAR indicated three participating countries but changes were done in the scope to the RETA
TA 8066: REG - Quality Primary Education in the North Pacific 19 Funded by Asian Development Bank
B. Issues
29. The Regional Technical Assistance Report (TAR) identified three major issues, namely:
i) Access to education and outcomes in the Pacific. Pacific DMCs have made significant progress towards universal access to primary education, with an average enrollment rate of more than 80%. In spite of this, it was noted that students’ learning outcomes in some of these countries still lag significantly behind improvements in access to schooling. Moreover, Pacific countries like Cook Islands, RMI, FSM and Papua New Guinea who participated in the Programme for International Student Assessment (PISA) performed poorly and were not included in the high-achieving groups in literacy and mathematics assessments. Nearly one-third of the students in Pacific DMCs were at the lowest proficiency level in the PISA results in 2007, compared with only 9% of European students who were at that level. Similar results were also noted for mathematics, wherein nearly one-third of students of Pacific DMCs performed at the lowest level in the PISA assessment.16
ii) Challenges in measuring learning outcomes. Most studies show that failure of children to acquire basic skills in literacy and numeracy would lead to higher repetition and dropouts at early grades. The acquisition and development of these skills is very critical at early grades. Hence, it is important to ensure that early diagnosis of these skills is done. However, very few countries, including the Pacific DMCs are giving attention to measuring how children at early grades are acquiring literacy and numeracy skills17. National assessments are being implemented by many countries with the intention to measure learning outcomes against defined national standards reflected in their national curriculum, but almost all national as well as international assessment are using the traditional paper-and-pencil tests administered to students. The practice of national assessments using paper-and pencil tests is also implemented in both FSM and RMI. However, these assessments are not necessarily able to measure literacy and numeracy skills comprehensively that could inform policy and instructional support as well as teachers’ professional development programs. Some literacy and numeracy skills are proven to be assessed more effectively using performance assessment tools. Hence, the Department of Education (DOE) of FSM and MOE of RMI articulated their need to be supported in developing and carrying out comprehensive and systematic assessment of learning outcomes, particularly in literacy and numeracy domains.
iii) Progress in student performance assessment systems in the Pacific. Through the Pacific Resources for Education and Learning (PREL), one-on-one assessments called Short Assessment for Language (SAL) and Short Assessment
16 Education and Pacific Peoples in New Zealand. http:..www.stats.govt.nz/browse_for_stats/people_and_communities/ pacific_peoples/pacific-progress-education/schooling.aspx 17 Early Grade Literacy Assessment Meeting. April 2011. The World Bank. Sydney. Australia.
TA 8066: REG - Quality Primary Education in the North Pacific 20 Funded by Asian Development Bank
for Mathematics (SAM) have been introduced in FSM and RMI. However, these assessments are being used on a limited scale at the moment in both FSM and RMI and their intention is more on providing data for teachers at the classroom level. Hence, the need to have a standardized assessment tool for literacy and numeracy skills of Grades 3 and 5 to systematically measure these skills and provide information for policy decision and teachers’ capacity development programs. The introduction of Early Grade Reading Assessment (EGRA) in the Pacific, particularly in South Pacific countries, such as Tonga, Papua New Guinea and Vanuatu, encourages some North Pacific countries, particularly FSM and RMI to consider EGRA or similar measures to ensure that their students would be at par with their neighboring countries. Hence, the need to develop assessment tools to measure literacy and numeracy for these two countries is imperative.
30. Learning from the experiences of EGRA in the Pacific, the MOE of RMI and DOE of FSM have expressed their need to develop a new learning assessment tool for early primary education, particularly for Grades 3 and 5. The main objective of assessment is to systematically measure how well children in the early grades of primary school are acquiring and developing literacy in both English and local language and numeracy skills. Eventually, the Project will bring about more effective efforts to improve programs in the development of these core learning areas through programmatic instructional support using evidences from assessment data.
C. Expected Impact and Outcome
31. As indicated in the TAR, the expected impact of the QPENP will be improved access to quality education for primary school students in FSM, Palau, and RMI. The outcome will be that participating governments integrate EGLA in selected schools for Grades 3 and 5 and develop individualized learning and assessment approaches.
D. The Design and Monitoring Framework
32. The Design and Monitoring Framework (DMF) of the CDTA intended to achieve five outputs18 with key activities. Table 2 presents the DMF showing the achievements and accomplishments as July 31, 2015.
18 Outputs and activities presented in this Final Report are revised based on the results of Inception Workshops in Palau, FSM and RMI.
TA 8066: REG - Quality Primary Education in the North Pacific 21 Funded by Asian Development Bank
Table 2: The Project’s DMF and Achievement and Accomplishments as of 31 July 201519
Design Performance Targets Data Sources Achievements and Summary and Indicators with and Reporting Accomplishments Baselines Mechanisms Impact By 2017: Achievements:
Improved Quality of primary Midterm and Based on the 2015 EGLA Baseline access to school education Interim Reports Survey, the performance of students quality improved; test scores of the indicated an increase of education for in literacy and Consultants performance in both literacy and primary school numeracy in numeracy ranging from 24% to 32%. students in the intervention grades FSM and the improved by 20% RMI (baseline will be assessed in 2013)
Outcome Achievements:
Participating Nine (9) target primary Inception, 133% of the target was achieved. governments schools in FSM and Midterm and integrate RMI implement EGLA Interim Reports Twelve primary schools were learning and in early grades by of the included in the pilot and baseline assessment 2015 (baseline: 0 in Q1 Consultants surveys in 2014 and 2015 (RMI: 5, FSM: approaches 2013) 7) - promoted under EGLA in Moreover, 4 control schools (RMI: 2, nine schools FSM-Pohnpei: 2) were assessed in 2015.
Individualized and Achieved. All the 12 pilot schools in group learning both RMI and FSM included in the approach project were provided operationalized in comprehensive professional FSM, Palau and RMI by development programs for teachers 2015 (baseline: 0 in Q1 to implement individualized and 2013) group learning approaches to improve literacy and numeracy learning.
Three major PD programs were implemented in November 2013, June 2014 and October 2014.
Palau was not provided such program because the focus was on the review of their assessment system. The review of the assessment system was completed on January
19 The contract of DSIC with ADB for this TA ended on 31 July 2015.
TA 8066: REG - Quality Primary Education in the North Pacific 22 Funded by Asian Development Bank
Design Performance Targets Data Sources Achievements and Summary and Indicators with and Reporting Accomplishments Baselines Mechanisms 2015 (see separate report)
Outputs Achievements:
1. Pilot EGLA in EGLA piloted in nine Midterm and Achieved. nine (9) (9) schools in 2 Interim Reports schools countries by 2015 of the The EGLA was developed in (baseline: 0 in Q1 Consultants November 2013, a tool that provides 2013) a detailed picture of student performance levels in literacy and numeracy.
12 primary schools were included in the pilot (RMI: 5, FSM: 7) and 4 control schools (RMI: 2, FSM-Pohnpei: 2).
EGLA Baseline Survey was conducted in February 2014, and another baseline survey was done for another cohort of Grade 3 and 5 in February 2015.
2. Develop Teachers trained in Inception, Achievement: individualized learning and teaching Midterm and and group approaches in 9 Interim Reports Achieved 133% of the target schools learning schools in 2 countries of the and 100% of the target countries. approaches by 2015 (baseline: 0 in Consultants Three major PD programs were Q1 2013) implemented in November 2013, June 2014 and October 2014. A total of 521 teachers were trained – 164 in November 2013, 114 in June 2014 and 243 in October-November 2014.
Dates MAJ PNI KSA Nov 87 32 45 2013 L N L N L N June 44 31 - - 24 15 2014 Oct 44 29 21/ 22/ 28 60 2014 21 61* Total** 88 60 42 83 52 75 L: Literary PD N: Numeracy PD * first number is Week 1 and second number is Week 2 ** Some participants attended both Literacy and Numeracy, participants also include JICA and Peace Corps Volunteers in all the project locations.
TA 8066: REG - Quality Primary Education in the North Pacific 23 Funded by Asian Development Bank
Design Performance Targets Data Sources Achievements and Summary and Indicators with and Reporting Accomplishments Baselines Mechanisms
3. Integration of EMIS and EGLA Midterm and Achievements: EGLA with the integration feasibility Interim Reports EMIS conducted by 2015 of the The EMIS system of RMI was reviewed considered (baseline: 0 in Q1 Consultants and a proposal EMIS was submitted 2013) to RMI MOE that eventually sought support from US funding. * This EMIS component was re-focused during A Workshop on EMIS was conducted the inception stage at to FSM EMIS Staff in Ponhpei in July the request of RMI and 2014. The EMIS Staff of the four states FSM. of FSM attended the workshop.
4. EGLA data EGLA findings Final Report of Achievements: dissemination presented in at least the Consultants regionally and three regional events 133% achieved. replication by 2015 nationwide The EGLA Results were presented in:
1) Workshop on Pacific Early Age Readiness and Learning (PEARL) in Nadi, Fiji on May 4-6, 2015
2) PACIFIC ISLANDS BILINGUAL BICULTURAL ASSOCIATION (PIBBA) in Kosrae, FSM on June 23-25, 2015
3) Micronesia Teacher Education Conference (MTEC) in Pohnpei, FSM on July 21-24, 2015
4) Pacific Educational Conference in Majuro, RMI on July 28-31, 2015.
5. Conduct Completed Review Final Review Achievements: review of the Report of the Palau Report, Interim Assessment Assessment System Progress Report Achieved. System of Palau A review was conducted and the Final Report was submitted in January 2015.
A workshop to present the results of the review was held in October 2014, and follow-up capacity building
TA 8066: REG - Quality Primary Education in the North Pacific 24 Funded by Asian Development Bank
Design Performance Targets Data Sources Achievements and Summary and Indicators with and Reporting Accomplishments Baselines Mechanisms program for MOE to draft the Palau National Student Assessment Policy Framework was also conducted on July 14-16, 2015.
E. The Terms of Reference of Consultants
33. To carry out this CDTA, the ADB has engaged the services of Development Strategists International Consulting, Inc. (DSIC), Philippines, hereafter referred to as the Managing Consultant or “Consultant”.
34. During the Inception Workshops held in Koror for Palau, Kosrae and Pohnpei for FSM and Majuro for RMI, slight modification to the approach for the CDTA has been decided and agreed upon by the participating countries and the ADB. The modification of the approach and focus required some revisions in the specific TORs of the Consultants. Specifically, Output 3 was re-focused to the review of existing EMIS without looking into actual integration of EGLA data into the system.
35. The table below shows the original and revised TORs of each individual consultant that guided them in the implementation of the project.
Table 3. Comparison between Original and Revised TOR
Original TOR Revised TOR
Team Leader (TL): Team Leader: Dr. Richard Gonzales Dr. Richard Gonzales i. The TL will work closely with the host i. No revision government and ADB to provide in-depth ii. No revision experience and knowledge of the political iii. No revision economy in the countries and in the iv. No revision education sector, in particular. ii. During the analysis phase, the TL will: (i) New TORs: assist in the institutional analysis and v. Supports the Data Management arrangements, and in the analysis of the Specialist 1 conduct the review of most viable options for introducing the the assessment system of Palau. EGLA and individualized learning and vi. Coordinates with MOE/DOE and assessment approaches in reading and ADB in the organization of the numeracy; (ii) provide a view on the regional workshop to disseminate political economy of countries to assess EGLA outcomes. the feasibility of the EGLA; and (iii) help to vii. Spearheads the preparation of the ensure the involvement of all the key EGLA Monograph and other related stakeholders during the TA, especially knowledge products. during the development of and discussion viii. Prepares quarterly reports, draft final of the implementation strategy. report and final report.
TA 8066: REG - Quality Primary Education in the North Pacific 25 Funded by Asian Development Bank
Original TOR Revised TOR iii. During the development of the implementation strategy, the TL, together with his team members will (i) ensure the proper design of the consultation and decision making process within MOEs, with other government agencies, and with all other key stakeholders; (ii) ensure the quality of the focus group discussion, required capacity development, based on extensive knowledge of key stakeholders in the sector; (iii) guide the consultation process of the implementation plan; and (iv) facilitate a high-level workshop where agreement on the implementation strategy will be sought. iv. During the implementation of the project, the TL will ensure that the project is implemented according to the agreed implementation strategy. The TL will also ensure that the inputs of all consulting team members are of good quality.
EGLA Literacy Specialists: EGLA Literacy Specialists: Dr. Marylin Low & Dr. Robert Early Dr. Marylin Low & Dr. Robert Early i. Design, field test and complete tools and i. Design, field test and complete tools materials for reading assessment for and materials for literacy assessment Grades 2 and 5. for Grades 3 and 5. ii. Participates in various training sessions for ii. No revision MOEs, school teachers and JICA volunteers. New TORs: iii. Designs and conducts training for MOE/DOE staff to administer the newly developed EGLA-Literacy. iv. Prepares EGLA Manual that includes definition of constructs, administration, scoring and reporting procedures. v. Documents lessons learned in the development and pilot testing of EGLA-Literacy. vi. Provides support to RMI in providing background knowledge in developing Bilingual Policy through a one-day or two-day workshop.
TA 8066: REG - Quality Primary Education in the North Pacific 26 Funded by Asian Development Bank
Original TOR Revised TOR
EGLA Numeracy Specialist: EGLA Numeracy Specialist: Dr. Steve French Dr. Steve French i. Designs, field tests and completes tools i. Designs, field tests and completes and materials for reading assessment for tools and materials for numeracy Grades 2 and 5. assessment for Grades 3 and 5. ii. Participates in various training sessions for ii. No revision MOEs, school teachers and JICA volunteers. New TORs: iii. Designs and conducts training for MOE/DOE staff to administer the newly developed EGLA-Numeracy. iv. Prepares EGLA Manual that includes definition of constructs, administration, scoring and reporting procedures. v. Documents lessons learned in the development and pilot testing of EGLA-Numeracy. vi. Provides support to the Education Quality Specialists in conducting Professional Development for teachers. Education Quality Specialists: Education Quality Specialists: Dr. Gillian Tasker & Dr. Diana Guild Dr. Gillian Tasker & Dr. Diana Guild i. Provide inputs for quality improvement of i. Provide inputs for quality primary education with individualized improvement of primary education learning and assessment approaches in with individualized learning and reading and numeracy, based on EGLA assessment approaches in literacy assessment; and numeracy, based on EGLA ii. Analyze the implications of EGLA on the assessment assessment and examination system; ii. No revision iii. Prepare necessary teaching and learning iii. No revision materials as needed, in English. iv. Contribute to the national and iv. Contribute to the regional efforts, e.g. Asia regional studies on EGLA outcomes, South Pacific Board for Educational focused on contribution of teachers’ Literacy and Education Experience Survey, professional development to EGLA to complement the project for the purpose of regional comparison. Note:
Dr Gillian Tasker will be primarily in-charge of activities for Majuro, RMI, while Dr Diana Guild will oversee activities for Pohnpei. Both experts collaborate the activities in Kosrae.
TA 8066: REG - Quality Primary Education in the North Pacific 27 Funded by Asian Development Bank
Original TOR Revised TOR
Education Data Management Experts: Education Data Management Expert 1: Dr. Carlo Magno and Michael Dalton Dr. Carlo Magno & Dr. Richard Gonzales i. Responsible for analysis of the implications i. No revision of the EGLA implementation on the ii. Assigned to Education Data provision of timely, accurate and Management Expert 2. complete information on all schools, iii. No revision teachers and students. iv. Coordinates the national and ii. Determine how the current data regional studies on EGLA outcomes, collection, storage, and usage system focused on contribution of teachers’ needs to be reformed to implement EGLA professional development to EGLA. and how EGLA data can be used to link with individualized learning and New TORs: assessment approaches for reading and v. Supports the EGLA Team in test numeracy. planning and data iii. Work closely with the national statistics analysis/psychometric analysis and data management experts. vi. Designs and implements the review iv. Contribute to the regional efforts, e.g. Asia of the assessment system of Palau, South Pacific Board for Educational including preparation of the Literacy and Education Experience Survey, Assessment Report. to complement the project for the vii. Supports the EGLA Teams in purpose of regional comparison. conducting the psychometric analysis of the newly developed tools. viii. Prepares the EGLA Technical Manual that includes reports on the development of the EGLA and its psychometric properties. ix. Assists the TL in organizing the regional workshop to disseminate the EGLA outcomes. x. Assists the TL and EGLA Experts in preparing the EGLA Monograph
Education Data Management Expert 2: Dr. Allan Parkes & Mr. Scott Pontifex
New TORs: i. Reviews and assesses the present EMIS of FSM and RMI.
ii. Conducts scoping of business requirements and feasibility analysis to upgrade/revise the EMIS in collaboration with EMIS staff of FSM and RMI.
iii. Prepares a feasibility report toward upgrading of the EMIS of FSM and RMI including possible integration of
TA 8066: REG - Quality Primary Education in the North Pacific 28 Funded by Asian Development Bank
Original TOR Revised TOR
EGLA data into the system.
iv. Recommends hardware and software to upgrade the EMIS of FSM and RMI based on existing equipment and facilities available.
Note: The change of TOR for this position required reconsideration of present expert, as well as his availability because of the urgent request for this component to be completed earlier. Mr. Michael Dalton may be replaced.
Education Statistics and Data Management Education Statistics and Data Specialists: Management Specialists: Davidson Syne, Sally Ann de Brum & Winton Davidson Syne, Sally Anne de Brum & Clarence Winton Clarence i. Responsible for providing statistical support i. No revision for EGLA data processing and ii. No revision management in the targeted countries. iii. No revision ii. Assist the TL to prepare and arrange iv. No revision necessary capacity development for v. Not applicable for FSM and RMI stakeholders. iii. Work closely with the International New TORs: Education Data Management Specialist in vi. Assist the EGLA Specialists in EGLA analyzing the implications of the EGLA for administration and encoding of pilot providing timely, accurate, and complete and baseline data. information on all targeted schools, teachers, and students. iv. Take a lead in the consultations supporting the EGLA analysis. v. Provide inputs to the assessment and examination system analysis.
Pool of National Consultants: Pool of National Consultants: Destin Penland, Juanita Lawrence, Winton Destin Penland, Juanita Lawrence, Winton Clarence and Evelyn Joseph Clarence and Evelyn Joseph i. Provide local support to international i. No Revision. consultants during project implementation. New TORs: ii. Support the Education Quality Specialists in the organizing, conducting and evaluating professional development programs for teachers and MOE/DOE staff. iii. Support the Education Quality Specialists in gathering data on
TA 8066: REG - Quality Primary Education in the North Pacific 29 Funded by Asian Development Bank
Original TOR Revised TOR
teachers’ professional practices and teaching-learning approaches. iv. Support the TL in organizing seminars, workshops and project- related activities in their respective project site assignments. v. Perform monitoring and follow-up activities required by the International consultants while they are not in the field.
36. With the leadership and guidance of the Team Leader, the international consultants led and planned their respective areas of expertise in collaboration with the national consultants and government counterpart staff. The national consultants and government counterparts provided advise to the international consultants on the local situation that enabled them to adjust their plans, activities, data analyses and recommendations to local context and conditions.
37. The Minister of Education of RMI and the Directors of Department of Education of Kosrae State and Ponhpei States of FSM played a critical role in ensuring that project implementation was carried out very smoothly. They supported the team in ensuring that Ministry and Department of Education Staff, such as the Chiefs, Curriculum Specialists, Special Education Specialists, and Teacher Training Coordinators were involved in all aspects of the projects, such as development of the EGLA instruments and in the conduct of PD for teachers.
38. The Team Leader assumed the overall responsibility for the direction of the CDTA implementation and consolidation with the Team’s activities and findings, and ultimate responsibility for the final report and other knowledge products produced by the Project.
39. All consultants assisted the TL to prepare and conduct workshops, prepare reports, and did other tasks in their areas of expertise, that were identified during the CDTA implementation, in addition to specific TORs assigned to them.
TA 8066: REG - Quality Primary Education in the North Pacific 30 Funded by Asian Development Bank
III. CDTA IMPLEMENTATION STRATEGIES AND ACHIEVEMENTS
40. This section of the report describes and discusses the details of CDTA implementation strategies and work plans.
41. The Development Strategists International Consulting (DSIC) was awarded the contract for the management of the consulting services in May 2013. The consulting services began in October 2013 and completed in July 2015.
42. The implementation process required the consultants to perform field visits in October-November, 2013, February-March, 2014, June-July 2014, October-November, 2014, February-March 2015 and finally in May-July 2015 intermittently when they shared the results in various regional conferences and workshops and also provided additional capacity building activities beyond the teachers of the pilot schools for this project.
A. Key Objectives
43. To conduct this CDTA, the Managing Consultant’s approach was governed by a focus on timely delivery of a well-designed technical assistance that firmly reinforced the implementing agencies in each country and the ADB’s goal to improve quality education for primary students in these countries. It is fully compliant with both ADB and participating Pacific countries requirements to enable smooth TA implementation.
44. Emphasis was placed on designing the EGLA tools, and interventions and programs that were pragmatic and can be successfully implemented in the Pacific context and consideration of the specific needs of the project sites. The Consultants considered the TOR articulated in the TAR which revealed the key objectives of the assignments that included complementing sets of tasks as follows:
i) To develop and pilot EGLA in selected schools in two countries – FSM and RMI.
ii) To develop individual learning approaches through comprehensive professional development for teachers in FSM and RMI.
iii) To review and assess the existing EMIS in FSM and RMI.
iv) To disseminate the EGLA results in at least three regional conferences for replication nationwide or region-wide.
v) To conduct an external review of student assessment in Palau.
TA 8066: REG - Quality Primary Education in the North Pacific 31 Funded by Asian Development Bank
B. Guiding Principles: Participatory Approach and Learning from Experience
45. The Consultants believed that the success of QPENP hinged on achieving consensus on the proposed TA design among and between ADB, Ministries and Departments of Education in FSM, RMI and Palau, a wide range of government stakeholders, as well as the consultants. It was essential that views were obtained from a cross section of stakeholders in the subsector as well, including school principals and teachers and other government officials with operational knowledge of constraints and challenges of the education sector, especially in primary education sub-sector before the Consultants embarked on project implementation.
46. Therefore, the Consultant adopted a participatory and consultative approach to the TA project from the very beginning, engaging stakeholders in all aspects of project implementation. Participation were sought through a combination of focused consultative workshops, one-on-one discussions with key officials/stakeholders as well as both formal and informal discussion with a range of stakeholders from FSM, RMI and Palau. The team members’ combined experience, institutional knowledge, and networks were critical in making the consultative process a success. It resulted in developing very appropriate EGLA tools and designing a responsive professional development programs for teachers. The consultation process and working closely with the staff of the MOE and DOEs in Kosrae and Ponhpei also served as capacity building and PD for the staff.
47. Aside from examining the successful experience of the EGRA in various countries, particularly in the Pacific countries such as Tonga and Vanuatu, the team brought with them their experiences and knowledge in developing and implementing learning assessments and development of assessment management information systems including establishment of student unique identification number in Samoa, New Zealand, Australia, US, Vanuatu and the Philippines. The lessons learned during implementation of similar projects in these countries were used to ensure success and best practice.
48. Also, the Consultants consulted with and carefully examined other donors’ experiences, such as World Bank’s EGRA, JICA Mathematics Program in the Pacific, and understand how the approach they used could be further optimized into the CDTA. The team carefully examined and analyzed the achievements of the Tongan EGRA and similar projects in the region20 and determined how best to position the new ADB investment and support. The team’s experience in developing ADB projects, governance in education sector in the Pacific, as well as procurement and project implementation were also brought to bear in these areas.
C. Technical Approach and Methodology
49. Given the aggressive schedule and the wide scope of coverage and expected deliverables under the CDTA, the development and pilot of EGLA and the development of individualized and group learning approaches in two countries through
20 Example: Samoa’s Education Sector Project that developed its assessment policy framework, standardized its primary education exams and developing a student unique identifier system for data management on student performance.
TA 8066: REG - Quality Primary Education in the North Pacific 32 Funded by Asian Development Bank
comprehensive professional learning development for teachers, the consideration and scoping of the EMIS and dissemination of EGLA data in the region and replication nationwide, as well as the conduct of external review of the assessment system in Palau, the consultants were organized under a range of work streams which were carried out in parallel to each other with the Team Leader overseeing all work streams as presented in the figure below:
Figure 1. Work Stream and Consultants’ Organization
50. In addition, the completion of the CDTA was staged in three phases – analysis phase, development of implementation phase, and actual implementation phase that were closely led and monitored by the Team Leader. The overall process framework is presented below:
TA 8066: REG - Quality Primary Education in the North Pacific 33 Funded by Asian Development Bank
Figure 2. CDTA Implementation Process Framework
51. The overall process framework (Figure 2) illustrates the approach for the services and the methodology that was followed in carrying out all the activities. It shows how the project was implemented following a logical course with sequential and non- sequential inputs to project activities that produced desired outputs at various stages of the CDTA implementation. The details of the methodology for carrying out the activities for each identified output of the project are discussed in the following section.
D. Inception Workshops
52. Adhering to a participatory and collaborative approach to project implementation, the consultants conducted series of workshops in the four project sites in the three countries in order to have a participative planning of the implementation of the CDTA. The Consultants, in cooperation with ADB and the Ministry of Education/Department of Education of the participating countries – Palau, Federated States of Micronesia, and Marshall Islands, held one-day Inception Workshops in each project site with the following objectives:
i) To explain the overall scope and purpose of the project.
ii) To present the proposed work plan and implementation schedule of the project.
iii) To discuss the detailed activities and Action Plan with MOE/DOE partners and other stakeholders and development partners.
iv) To gather feedback and comments from ADB, government, development partners, and other stakeholders.
TA 8066: REG - Quality Primary Education in the North Pacific 34 Funded by Asian Development Bank
v) To gather information/data for the development and preparation of the Inception Report.
53. During the Inception Workshop in each project site, the agenda below was followed. The copy of the Inception Workshop Agenda is included in Annex A. The agenda included:
i) Greetings from Government (MOE or DOE)
ii) Remarks from ADB Officer
iii) Introduction of Consultants and Participants
iv) Overview of the QPENP presented by the Team Leader
v) Introduction of EGLA Literacy and Numeracy presented by EGLA Experts
vi) Implementation Plan of EGLA Development presented by Data Management Expert 1
vii) Professional Development and Instructional Support presented by Education Quality Specialist
54. During the Inception Workshop, the Steering Committee, EGLA Team and Professional Development Team for each project were organized. These EGLA Teams and Professional Development Teams worked very closely with the Team, a key to the successful implementation of the CDTA.
55. The second half of the Inception Workshop focused on planning exercise and development of action plans for the EGLA Group that informed the details of the project implementation. At the end of the one-day workshop, the Consultants together with ADB, Government and development partners and other stakeholders agreed on the work plan and schedule of the project as well as how to achieve the impact, outcomes, outputs and activities defined in the design and monitoring framework. Through these workshops, the Consultants were able to prepare and submit an Inception Report that is acceptable and agreeable to ADB and Government.
56. A total of 126 participants attended the Inception Workshops. The participants included but not limited to:
i) Representatives from Government Ministries
ii) Officers and Staff of the Ministry of Education/Department of Education
iii) Representatives of development partners from JICA, USAID, AusAID, UN, etc.
TA 8066: REG - Quality Primary Education in the North Pacific 35 Funded by Asian Development Bank
iv) School principals and selected teachers
v) Representative from Colleges and other institutions
vi) ADB Officers and ADB Country Staff
vii) Consultants from DSIC
57. The table below shows the dates, place and number of participants of the inception workshops:
Table 4. Date, Place and Number of Participants for the Inception Workshops
Date Place Number of Participants21 September 5, Ministry of Education 27 2013 Teacher Training Conference Room Koror, Palau
September 9, State Department of Education 30 2013 Conference Room, Kosrae, FSM
September 12, Special Education Center 30 2013 Conference Room, Pohnpei, FSM
September 17, RMI International Convention Center 39 2013 Majuro, RMI
21 List of Participants was included in the Inception Report.
TA 8066: REG - Quality Primary Education in the North Pacific 36 Funded by Asian Development Bank
IV. THE EARLY GRADE LEARNING ASSESSMENT (EGLA): CONCEPTUALIZATION AND DEVELOPMENT
58. This section describes the EGLA as an assessment tool and how it was conceptualized, developed, and piloted in RMI and in the FSM. The EGLA is one of the major deliverable of this project.
A. What is EGLA?
59. The EGLA is a formative assessment tool that provides a detailed picture of student performance levels in literacy and numeracy. EGLA is envisioned to be for multiple purposes – tailored teacher professional development, identifying appropriate learning resources, and building accountability.
60. The EGLA was developed for the FSM and the RMI in a collaborative manner alongside with education authorities of each project site, with intensive capacity building, piloting, analysis, and extensive trials.
61. The new assessment tool is administered individually to capture literacy and numeracy practices of students in grades 3 and 5. It is intended to be delivered to a statistically valid and representative sample of students (except in places with very small populations), and may not even need to be taken to every school. However, the tool could also be used as an individualized assessment instrument as part of an overall student assessment strategy.
62. The assessments require 10 minutes per child for each of the five components outlined in Table 5 below:
Table 5: Five Core Components of the EGLA
Grade 3 Grade 5
Interview in the local language Interview in the local language Literacy in the local language Literacy in the local language Numeracy in the local language Numeracy in the local language Literacy in English Literacy in English Numeracy in English Numeracy in English Source: Development Strategists International Consulting
63. The EGLA is administered once a year across an appropriate sample with the primary aim of providing detailed measures for state and national level policy makers
TA 8066: REG - Quality Primary Education in the North Pacific 37 Funded by Asian Development Bank
and educational authorities. It will inform them about the overall status of literacy and numeracy learning in the school system.
64. The first component of EGLA - the interview, is individually administered in the L1. It compiles basic demographic information about each subject, but also elicits rich contextual information that contributes to building a sociolinguistic profile of the population. Details about such matters as language use in the home, and access to literacy and numeracy resources and inputs outside of school, will provide information about home-school connections that will support the interpretation of the EGLA data gathered.
65. EGLA is targeted at Grades 3 and 5 levels because it is recognized that basic skills need to be acquired early, as they are more difficult to acquire later. Failure to gain a good start at the early levels will put students at a disadvantage in future learning contexts. If the gap between readers and emergent-readers, and numerate and emergent-numerate children is not addressed at an early grade, it will continue to widen as they progress through schooling22. The time and effort required to provide repair and remedial support at later stages makes it imperative that every effort is made to ensure that good conceptual foundations and skills are acquired early.
66. EGLA is introduced at selected elementary schools in RMI and FSM. The primary objective sets up the process of systematically measuring how well children in the early grades are acquiring effective literacy and numeracy practices, and in turn this will drive increased efforts to improve performance in these core learning areas. The overall success of the EGRA framework from elsewhere in the world and in the Pacific informed the development and piloting of the EGLA tool in Kosrae, Pohnpei and Majuro. The approach is based on the theoretical foundation that the purpose of assessment in early childhood education is to support learning, and that assessment sets the definition about what a child can do and what they are ready to learn next.23
67. The purpose of the EGLA then is:
• To collect baseline data of early learners’ basic literacy and numeracy practices (i.e. patterned ways of using technology, knowledge and skills to accomplish tasks).
• To collect information about the context within which early learners become literate and numerate in Majuro, RMI, and in the FSM states of Kosrae and Pohnpei.
• To build local capacity to administer the EGLA, analyze the data, and make evidence-based decisions to inform policy, pedagogy, resourcing, and
22 Good, R. H., & Kaminski, R. A. (Eds.). (2002). Dynamic indicators of basic early literacy skills (6th ed.). Eu- gene, OR: Institute for Development of Educational Achievement. 23 Shepard, L., S.L. Kagan, & E. Wurtz. 1998. Principles and recommendations for early childhood assessments. Washington, DC: National
TA 8066: REG - Quality Primary Education in the North Pacific 38 Funded by Asian Development Bank
professional learning for educators. The ultimate result should then be improved literacy and numeracy outcomes.
68. The QPENP project has developed and piloted EGLA in a total of 13 pilot schools and four control schools and provided technical assistance for individualized and group approaches to literacy and numeracy learning. It may also assist with plans by educational authorities in RMI and FSM to replicate the approach nationwide, and with any efforts to disseminate EGLA results regionally. The project also has the capacity to integrate EGLA with the local EMIS.
B. Fundamental Conceptual Considerations
69. The ability to read, understand and create texts (literacy) and to be able to draw on an underlying control and understanding of mathematical concepts and skills (numeracy) are fundamental and foundational skills used to establish and develop life- long learning.
70. EGLA allows for the assessment of the control of foundation skills, and for the ability to use them meaningfully along a developmental path leading to being fully literate and numerate.
• Grade 3 is considered to be an appropriate level to undertake the assessment of the control of foundation skills in literacy and numeracy.
• Grade 5 is considered to be an appropriate level to evaluate if the foundation skills have been extended and integrated into the child’s learning, and can be utilized meaningfully in relevant tasks that demonstrate ongoing development.
Box. 1: Why assess Grades 3 and 5?
- Based on international research and best practices, these foundational skills should be acquired early. They are difficult to attain later, especially as the gap between learners and non-learners widen. Without a good start, non-learners can fall further behind in all future learning through school, and so early diagnosis of problem areas enables targeted treatment.
- Assessing in Grade 3 provides information about the acquisition of foundational skills that should be addressed and improved before the middle grades.
- Assessing in Grade 5 provides information on students’ progress from the early stages to more developed levels of becoming genuinely literate and numerate. This information can be used to tailor learning for a school, a class, or an individual student, so that learners are equipped to handle as they are exposed to a wider range of texts and mathematical challenges.
TA 8066: REG - Quality Primary Education in the North Pacific 39 Funded by Asian Development Bank
71. EGLA is grounded in the service of learning, and has a strong formative motivation:
• What are the students’ reading and mathematical strengths at school and classroom levels in both languages?
• What areas of literacy and numeracy learning in both languages need additional support?
• What contextual factors promote literacy and numerical development?
72. EGLA is designed to provide answers to these questions, by providing rich data- sets on an annual basis, that over the years will become an extensive database of longitudinal information. Educational authorities can use this information to plan for improved outcomes, and to track progress towards them.
73. In order to assess student performance in literacy and numeracy, it is necessary to deconstruct the notions of literateness and numerateness. What does it mean to say that a person is “literate” or “numerate”? Are there different answers to this question for different stages of development? Both of these terms are clearly holistic and composite cover terms for overall competencies that are made up of complex subsets of skills and conceptual understanding.
74. While broad measures are often cited (e.g. statistics on literacy levels on a country-by-country basis), the question of how such percentages are arrived at, or how such information is to be interpreted, is fraught with difficulties. There are many variables that may interact in many ways to result in a particular child performing well, or not well at all, in reading a particular text on a particular occasion. Or, if help is to be given to a child who is struggling in mathematics, then such assistance needs to target particular areas of need while continuing to build on student strengths.
75. Therefore, an important step in the development of the EGLA instruments for literacy and numeracy was to go back to the basics of our understanding of what really constitutes the ability to read and write well, or the ability to count and compute with skill. What are the components of these abilities? ‘Deconstructing’ them means breaking them down into their ‘constructs’, or ‘building blocks’.
76. There are two advantages to this. The first is that by breaking down these skills into their component parts, it is then possible to develop questions and tasks that will directly assess the child’s level of understanding and skill for each component. This means that it is possible to know exactly what is being measured in each part of the assessment tool, and in turn, areas of strengths and areas of need can be clearly identified.
77. Secondly, because we are able to obtain measures for each of the discrete or individual components, it will be possible to ensure that any subsequent informed actions can be directly targeted to the areas that are identified as needing most attention.
TA 8066: REG - Quality Primary Education in the North Pacific 40 Funded by Asian Development Bank
78. Fortunately, modern educational theory provides a good understanding of what factors go together to produce literate and numerate individuals.
Essential Theoretical Underpinning of Literacy Development
Literacy practices are uses of the tools of literacy (e.g. texts, paper and pencil, digital media) in combination with the decoding and encoding processes of reading and writing (often now extended to include the processing of images and multimodal and interactive texts), informed by knowledge of genres, modalities, registers, styles, and grammars24. Hull & Moje, 2012
79. With EGLA, students can engage in multiple tasks of various literacy practices that elicit evidence of their literacy knowledge and skills. The tasks are designed to reflect successful literacy learning experiences that:
• Build upon prior knowledge and cultural practices
• Capitalize on the social nature of learning
• Situate literacy learning within a larger purposeful and relevant activity
• Engage literacy learners as makers of meaning.
Essential Theoretical Underpinning of Numeracy Development
80. The numeracy component for EGLA is premised on the definition that Pacific Numeracy is the:
“Knowledge and skills necessary to empower a person to be able to use numbers in mathematical processes, as well as the language of mathematics for a variety of purposes, with respect to everyday life.”25 Literacy, Numeracy and Life Skills, 2006, p4
81. The numeracy component of EGLA is an assessment tool that is designed to elicit quality information about the knowledge and mental strategies that students use to solve problems in mathematics. As with the literacy component, the numeracy component takes the form of an individual assessment with students. This is necessary because:
24 Hull, G., & Moje, E. (2012, January). What is the development of literacy the develop- ment of? Paper presented at the Understanding Language Conference. Stanford, CA. 25 Earle, D. (2015). Updating the Adult and Literacy Life Skills Survey: Estimating change in skills distribution since 2006. New Zealand: Ministry of Education.
TA 8066: REG - Quality Primary Education in the North Pacific 41 Funded by Asian Development Bank
i) Uncovering mental strategies involves finding out how students solve problems. While paper and pencil assessment tasks can determine whether or not a student is able to calculate correct answers, a one-to-one interview also provides opportunities for observation and probing questions to reveal the strategies that students use when solving problems;
ii) The interview process is invaluable for teachers’ own professional development as it provides insights into students’ common misconceptions and errors so that teaching and learning activities can be better designed to meet student’s needs.
82. The EGLA numeracy component is designed according to current international best practice, and incorporates the following concepts:
i) That the teaching and learning of mathematics is cyclical in nature. Mathematical understanding requires strategies to develop new knowledge, and the knowledge serves as a foundation for the development of new strategies.
ii) Students need to demonstrate their problem solving skills with word problems in familiar contexts.
iii) Students need the opportunity to use a variety of materials to solve problems.
iv) It is accepted that frequently there are several strategies that may lead to a correct answer and that students who are exposed to alternative approaches by sharing with their peers, benefit from a richer learning experience and develop a stronger number sense.
v) Teachers who are able to assess accurately an individual child’s progress on a framework are better able to design suitable learning activities for them to progress to the next step.
C. Purposes of the EGLA for Literacy and Numeracy
83. The primary purpose of the EGLA for Literacy and Numeracy is to assist the participating countries Box 2: Why assess literacy set up the process of systematically measuring how and numeracy? well children in the early grades of primary schools or education are acquiring literacy (reading and The ability to read, understand, writing) and mathematical (numeracy) skills, and to and create text (literacy) and to drive more effective efforts to improve performance draw on an underlying in these core learning areas. While there have been understanding and control of several initiatives to develop tools to assess literacy mathematical concepts in some Pacific states, the Quality Primary Education (numeracy) are fundamental in the North Pacific (QPENP) Project is the first skills used to begin, establish, and maintain lifelong learning.
TA 8066: REG - Quality Primary Education in the North Pacific 42 Funded by Asian Development Bank
initiative to support the development of a new learning assessment tool that includes both literacy and numeracy for use in primary education grades.
84. Through the Pacific Resources for Education and Learning (PREL), one-on-one assessments called Short Assessment for Literacy (SAL) and Short Assessment for Mathematics (SAM) have previously been introduced in FSM and RMI. However, these assessments are being used on a limited scale in both FSM and RMI and their intention is more on providing data for teachers at the classroom level. Hence, it was needed to have a broader standardized assessment tool for literacy and numeracy skills of Grades 3 and 5 to systematically measure these skills and provide information for policy decision and teachers’ capacity development programs.
85. A key feature of the education policies in the two countries is a commitment to the notion of bilingualism. The vernacular languages of this region function as national languages, and there is strong loyalty to them, reinforced by the important role the languages serve in all areas of national life, including the religious domain. While there is some residual reticence to the use of these languages in education, there is generally widespread acceptance of the underlying principles that:
• children’s first exposure to learning, including initial instruction in literacy and numeracy, should be through the medium of the child’s first language. [This is readily implemented in situations where there is just a single vernacular, as in RMI and Kosrae, but is a more complex consideration in Pohnpei where there are significant language communities besides the predominant Pohnpeian community, such as the Nukuoro and Kapingamarangi communities.]
• while the use of the local language may have been regarded as a transitional measure, leading to a fairly early switch to English as the sole language of instruction, there are significant cultural, social and pedagogical reasons for extending the use of the vernacular as an instructional language throughout the elementary cycle, and later. [The timing and ultimate extent of the inclusion of English as an instructional language has to be decided for each situation.]
• a key outcome for education in the RMI and FSM is to produce bilingual individuals who are able to function in both their local language and English. As the impact of English grows, Micronesian nations themselves are becoming increasingly bilingual, and the compelling reality is that a good knowledge of English is regarded as an absolute requirement for success in modern life. On the other hand, there is a recognition of the diluting and intrusive pressure that English places on local languages, and the school is seen as having an important role in local language maintenance and development.
86. As a result, curriculum outcomes specify achievement targets for students in both languages at the targeted grades. Therefore, in order for the EGLA assessment to provide a comprehensive picture of system performance, it is necessary to evaluate literacy and numeracy performance in both languages in relation to appropriate
TA 8066: REG - Quality Primary Education in the North Pacific 43 Funded by Asian Development Bank
curriculum standards. After undertaking an initial sociolinguistic profile interview, students who undergo EGLA assessment will therefore participate in four separate evaluations:
• Literacy for L1
• Literacy for English
• Numeracy for L1
• Numeracy for English
87. The EGLA tool can therefore be regarded as a formative assessment measure for literacy and numeracy at the level of the system as a whole, but can also provide data to inform literacy practices at school and classroom levels.
Box 3: Why assess both in first language and English?
- An important feature of the EGLA is that it is a bilingual assessment instrument Micronesian-designed for the context, and provide the only means for educators to directly compare the results for English with those of the first language, opening up many avenues for investigating aspects of bilingual learning and achievement.
- It is widely recognized that mastery of the English language is important to broaden young people’s future opportunities. However, there is an equally growing understanding of the need to ensure that Micronesian languages and cultures are also retained and developed. Promoting bilingual competency in both English and indigenous first languages is the approach that both the FSM and the RMI use in their public school systems.
- International research demonstrates the benefits that accrue when children’s first (and continuing) experiences in education make use of knowledge of their first language. In fact, the best predictor of language development in English as a second language is the level of development of language proficiency in the mother tongue.
- If the student is expected to have high levels of proficiency in both languages, then knowing students’ specific strengths and the areas where they face challenges will allow more appropriate instructional responses to be developed.
- Project Consultants supported a workshop in the RMI to develop a new language education policy that supports the importance of both Marshallese as the first language and English as the language of international interaction. Similar language policy developments are taking place elsewhere in Micronesia. Educators have recognized that the EGLA is a very useful assessment tool that can provide information to track progress toward the first and second language proficiency goals envisaged by these policy initiatives.
TA 8066: REG - Quality Primary Education in the North Pacific 44 Funded by Asian Development Bank
D. Description of the EGLA Literacy and Numeracy Assessments
88. The EGLA tool as developed in the QPENP project is an integrated package of materials, samples of which are reduplicated in the annex as a separate package. One goal in developing the materials was to keep them as straightforward and self- explanatory as possible, but it was also found that one of the implementational challenges was to ensure systems were in place for keeping track of the various components that are used.
89. For both literacy and numeracy, training materials were developed for assessors, and it was found that a minimum of one day was required for training assessors in the use of the literacy tool, and another day for the numeracy. The logistics of identifying and assigning assessors is a factor that needs careful thought. Some assessors are comfortable with being trained to assist with assessments for literacy and numeracy in both levels, and in both languages, while others prefer to stick to assessing just one level, in either literacy or numeracy, and in one language only. At a practical level, assessors need to be able to endure undertaking many hours of sitting through the student assessments, and they need to recognize the need to be efficient in keeping the process moving.
90. Before going to the schools, it is necessary to have access to school rolls, so that pre-determined random selections of respondents or “examinees” can be identified. Without this, there may be a temptation for principals or teachers to send along their better students for assessment. Assistance from the school principals in identifying rooms and in assigning some of their staff to assist with marshaling children to the test venue in small groups is imperative.
91. As each child comes to the venue, first, they will undergo the general interview, only in their first language. These are straightforward questions and any competent readers and speakers of the local language can conduct the interviews. As part of the interview, the subject is asked to indicate their willingness to undergo the assessment. They need to know that this is not a test in the usual sense. The interview form has all responses available in bubble format, so that there is no need for transcription or manual handling of data. The forms can be electronically scanned.
92. On completion of the interview, each child is given an assessment checklist, with their name and ID number on it, which they carry with them through the process. Each child is also provided with a new sharpened pencil that they use throughout and keep afterwards.
93. In order to help children overcome any discomfort about the assessment, it is expected that they would complete the literacy and numeracy assessments in their vernacular language before completing the assessments in English. This sequence enables them to become familiar with questions in their first language before tackling the parallel questions in English with a different assessor.
TA 8066: REG - Quality Primary Education in the North Pacific 45 Funded by Asian Development Bank
94. The students should be told that the questions in the English assessment might be similar to those in their first language but answers would be different in order to avoid any attempts to memorize responses from the first assessment.
95. For each assessment, there are assessor scripts, other test materials and response forms. The assessor scripts contain the full content of the assessment, and assessors are encouraged to work directly from the script at all times. It ensures consistency of delivery, and keeps the assessor closely on task. After numerous times of going through the assessment, the assessor will be less dependent on the script, and may be able to proceed without it, but it is important to train using the script more or less word-for-word.
96. The other test materials for literacy include a short printed text either in the student’s language or English, a student answer sheet, which is used to record samples of the student’s writing, and a set of word cards which, as will be explained further later, are only used with subjects who are clearly unable to complete the full assessment.
97. The other test materials for numeracy include a test booklet, which contains all the tasks that the student will be asked to do. For some tasks, colored blocks and paper shapes are provided for the student to work with. Students are also given a piece of paper to use as a worksheet as required.
98. For both literacy and numeracy, it is important to point out that provision is made for situations where a student with a very low level of skill is encountered. In numeracy, for each task, there are a series of four questions given, in ascending levels of difficulty. The instructions to the assessor point out several places where, for example, if a student cannot give any answer to question 1 or 2, then it will not be possible for them to correctly answer subsequent questions on that topic. Thus, the assessor will simply move to the next task without asking the additional questions.
99. In literacy, some students taking the assessment will be classified as emergent- readers. The first task is to read the word “Name” on the answer sheet, and then to do what it says, i.e. fill in the blank with their name. The next task is to read and respond to a very simple sentence on the answer sheet, such as “What is the name of your school?”, “Where do you live?”. Students who are unable to complete these very basic tasks would struggle with the rest of the assessment, and there would be no benefit in forcing them to endure it. So after these initial tasks, the literacy component takes a two-track approach. Students who appear to be emergent-readers are taken through some further short tasks designed to find out if they did have at least some initial or emerging level of skill in some basic elements of literacy, while those who complete the initial tasks complete the longer version of the assessment.
100. For each assessment, there is a response form in bubble format. For numeracy, these can be completely filled in as the assessment proceeds, while for literacy, they can be mostly marked in real time, but there are some aspects of assessing student writing which may require the assessor to take a moment to fill in after the student has completed the assessment.
TA 8066: REG - Quality Primary Education in the North Pacific 46 Funded by Asian Development Bank
101. Having completed all assessments as indicated by the check marks on the assessment checklist, which is collected, the student can return to class. Students kept the pencil given to them at the start, and they were also given a sweet to thank them for their participation.
102. An interesting difference in the seating arrangements has been developed. It was found out that for literacy, the idea of the assessment being conceptualized as the assessor and the student reading a story text together resulted in the most natural seating arrangement being side-by-side. Assessors had to learn to keep their response forms on the far side away from the eyes of the student, but sitting beside each other seemed appropriate for the reading tasks.
103. The numeracy assessment was conducted with the assessor facing the student across a table. This ensured the student would not be able to read the assessor’s script or the response form which were contained in a folder. The Question Booklet was placed between the assessor and the student oriented to read and answer. A blank sheet of paper was used to mask the questions and moved to reveal each item in turn. The student could respond to questions orally, by pointing to features in the question booklet or by writing a computation on a separate sheet of paper.
104. The assessor conducted the assessment by following the instructions and words written in the script. Instructions required the assessor to point out features in each item in the Text or the Question Booklet, the words of the question, an illustration and, for some items a choice of answers.
E. Content Coverage
105. The Literacy and Numeracy domains of the EGLA tool incorporate constructs based on existing frameworks for assessing literacy and numeracy skill sets for the early grades. The specific constructs that have been selected are the standards and commonly shared international competencies for literacy and numeracy that are common to the current curriculum statements in the states of Kosrae and Pohnpei in FSM and Majuro in RMI. Tables 2 and 3 present the main concepts included in EGLA literacy and numeracy for Grades 3 and 5.
Table 6. Content coverage for EGLA Literacy for Grades 3 and 5
Content Coverage for Grades 3 and 5
• Foundational skills for reading alphabetical principle phonological awareness phonics word recognition fluency
TA 8066: REG - Quality Primary Education in the North Pacific 47 Funded by Asian Development Bank
• Vocabulary word/phrase knowledge and use • Comprehension listening reading • Writing meaning grammar conventions spelling Source: Development Strategists International Consulting (DSIC)
106. EGLA literacy is designed to gather information about these important basic skills and, in doing so, highlights the integrated nature of literacy. However, it also recognizes that the ability of the student to undertake the literacy tasks in the assessment is dependent on underlying language competency, and in fact the nature of the interaction between the assessor and the subject within the assessment context provides an opportunity to observe this. Therefore, at various points, the assessor makes an evaluation of the student’s language practices, especially their ability to understand and respond to instructional language, and to engage in oral interaction with the assessor. The level of understanding of the language of instruction is critical, and the students’ understanding of what is being asked of them will impact the quality of the practices. It is important to track this information through the administration of EGLA.
107. The information gathered from EGLA literacy can therefore support and inform:
• Learning goals and success criteria articulated in the classroom that improve literacy practices
• Multiple and varied literacy tasks within a larger, purposeful project
• Strategic approaches to nested activities that are both explicit and implicit, planned and impromptu teaching and learning opportunities
• Feedback and feed forward
• Self-reflection by teachers and students.
TA 8066: REG - Quality Primary Education in the North Pacific 48 Funded by Asian Development Bank
Table 7. Content coverage for EGLA Numeracy for Grades 3 and 5
Content Coverage for Content Coverage for
Grade 3 Grade 5
• Counting • Place Value
• Place value • The Four Operations
• Addition and subtraction • Decimals
• Multiplication and division • Fractions
• Statistics and probability • Statistics and probability
• Geometry and • Geometry and measurement measurement
Source: Development Strategists International Consulting (DSIC)
108. Within and between these main concepts in both grades, there are opportunities for students to apply knowledge and strategies to word problems in familiar contexts, and to demonstrate higher thinking skills such as the application of a concept or drawing a conclusion.
F. Development of the EGLA Literacy and Numeracy Tools
109. The development of the EGLA tools began with a series of inception workshops in September 2013. The QPENP team met with representatives of each M/DOE to outline the project and to nominate teams of personnel to undertake responsibility for EGLA and for Professional Development.
110. The first inception workshop was held in Kosrae, followed by similar workshops in Pohnpei and Majuro. Consultations were held to identify the purpose of the EGLA tools and timelines were agreed for the development phase including the creation, trialing, and pre-testing of the tool, and for the administration of the first Baseline Survey in February 2014.
TA 8066: REG - Quality Primary Education in the North Pacific 49 Funded by Asian Development Bank
Definition of EGLA Literacy Constructs
111. The contents of the literacy tool were outlined above. These are now detailed more specifically, in terms of the actual tasks that were developed to match each of the constructs that were being assessed. The items in Table 5 refer to the tasks that can be identified from the scripts and response forms for the literacy assessments.
Table 8. Literacy Constructs and Items
CONSTRUCTS ITEMS GRADE 3 ITEMS GRADE 5 Alphabetic 3f Know names of letters 3d Know names of letters principle OR OR 5e 5e Phonological 3b, 3d Blends phonemes to N/A awareness OR 5c make word Phonics 3c, Know and apply letter- 3b, 3c Know and apply letter 3eOR sound OR sound 5d correspondences 5c, 5d correspondences Word 2a Read high frequency 2a Read high frequency recognition 3a OR words accurately by 3a OR 5b words accurately by 5b sight sight Fluency 7a Read with appropriate 7a Read with appropriate rate. rate. Foundationalskills for reading 7b Read with accuracy 7b Read with accuracy and automaticity. and automaticity. 7c Read with 7c Read with expression/prosody. expression/prosody. Vocabulary 6a, 6c Determine the general 6a, 6c Determine the general acquisition or domain-specific or domain-specific word. word.
6b, 6d Determine the meaning 3g, 6b, Determine the meaning of general and domain- 6d of general and domain- specific words in a text specific words in a text Vocabulary using multiple strategies (e.g., affixes) Listening 1, 9 Social interactional 1, 9 Social interactional comprehension language language 3gOR5 Instructional language 3eOR5f Instructional language f 4a Academic language 4a Academic language
Comprehension OR10a OR10a
TA 8066: REG - Quality Primary Education in the North Pacific 50 Funded by Asian Development Bank
CONSTRUCTS ITEMS GRADE 3 ITEMS GRADE 5 Reading 5a Identify the main topic 5a Identify the main topic comprehension of the text of the text 8a, 8c Answer questions about 8a, 8c Answer questions about details in a text details in a text 8b Locate answer in the 8b Locate answer in the text text Writing for 2b, 8d Respond to a text in 2b, 8d Respond to a text in meaning writing. writing. 4b OR Spell grade-level 4b OR Spell grade-level 10b appropriate words and 10b appropriate words and phrases (dictation) phrases (dictation)
Writing 3h, 4c Use appropriate 3f, 3h,4c Use appropriate OR punctuation, OR 10c punctuation, 10c capitalization, and capitalization, S/V syntax agreement and syntax Source: Development Strategists International Consulting (DSIC)
112. Engaging in an act of literacy requires the simultaneous activation of a number of skills, and so not all tasks can be seen as focusing on the assessment of just a single competency. It will be seen that in the EGLA assessment, different constructs are assessed at different points. The automated marking of the answer sheets provides for all marks relating to each individual construct to be collated and reported.
113. Also, as can be seen from the answer sheet, assessments in literacy performance are not always objective, and not always scorable as a simple Yes/No response. At a number of places, assessors are asked to make what are essentially impressionistic evaluations, and to assign a scalar evaluation from 1 to 4. In these cases, rubrics are provided to guide the assessors to the appropriate score. Further, an important part of the assessor training is to provide samples of responses and to coach assessors into being able to make the appropriate evaluations with a high level of consistency and reliability. The training materials for literacy cover this matter in considerable detail.
114. One of the key aims of the literacy component in EGLA was to make the assessment task interesting and authentic for the students. Rather than developing a series of disconnected questions about unrelated tasks, all of the activities in the assessment were built around the interaction of the student with an informational text. The use of an informational text rather than a narrative recognizes the differences between these two broad types of text, the familiarity of most students with narrative, and the increasing need to gain the reading and writing skills required for informational text. The text was written for the purposes of the assessment, but considerable care was taken to ensure that the ideas were culturally, socially, and environmentally appropriate, and was about a topic that students could readily relate to and understand within the context of their own experience.
TA 8066: REG - Quality Primary Education in the North Pacific 51 Funded by Asian Development Bank
115. It was found that students became engaged in the task of understanding the story, and the specific technical tasks they were asked to perform in relation to the text of the story, as in the diagram below, did not seem artificial or contrived. The focus for the student was around learning about the content of the text, interacting with its wording, and responding to its message, while in the background the assessor was able to make careful measures of the student’s level of skill in the specified sub-components of literacy practice.
Figure 2. The Flow of EGLA Literacy Tasks (Source: Development Strategists International Consulting (DSIC)
Definition of EGLA Numeracy Constructs
116. Initially, the Numeracy constructs were selected according to the common main strands of the Mathematics Curriculum for the Republic of the Marshall Islands, and two of the Federated States of Micronesia – Kosrae and Pohnpei. Six broad curriculum tasks were identified as appropriate for Grade 3 and six for Grade 5. For each task, 4 items are used to assess levels of understanding of mathematical content and skills to provide an individual numerical measure of performance. Each item provides a score of 2, 1 or 0 where 2 represents a correct response using an efficient strategy, 1 represents a response demonstrating a degree of understanding though perhaps not the use of most
TA 8066: REG - Quality Primary Education in the North Pacific 52 Funded by Asian Development Bank
efficient strategy, and 0 represents an incorrect answer or no response. The total of the response measures for the four items for each task provides the individual performance level according to the scale: a total of 7 or 8: Advanced; 5 or 6: Proficient; 2, 3 or 4: Developing; 0 or 1: Beginning.
Table 9. Structure of Numeracy Assessment
Grade 3 Grade 5 Construct Item Concept Construct Item Concept 1.1 Counting on from a 1.1 Identifying the place given two-digit number value of a digit for a number up to 100,000, given one example 1.2 Counting on through 1.2 Interpreting a one hundred numeral represented by a picture of place value blocks, given
Task1 Task1 one example Counting
1.3 Naming numbers PlaceValue 1.3 Identifying the before and after a smallest (or largest) of hundred ‘teens’ four 3-digit numerals number 1.4 Step counting in a 1.4 Identifying the largest simple pattern (or smallest) of three decimal numerals
2.1 Using ten as a counting 2.1 Demonstrating the unit addition of a 4-digit and a 3-digit numeral 2.2 Identifying one, two 2.2 Using subtraction to and three digit solve a word problem numerals in a familiar context 2.3 Identifying the smallest 2.3 Using multiplication (or largest) of four 3- to solve a word Task2 Task2 digit numerals problem in a familiar PlaceValue context FourOperations 2.4 Identifying the largest 2.4 Using division for (or smallest) of three sharing to solve a amounts of money word problem in a familiar context
3.1 Demonstrating addition 3.1 Identifying the of two 1-digit numbers position of a decimal als n & n& tion using counters on a number line, Task3 Task3 Decim Additio Subtrac given one example
TA 8066: REG - Quality Primary Education in the North Pacific 53 Funded by Asian Development Bank
3.2 Adding a 2-digit and 1- 3.2 Demonstrating the digit number without addition of a 2- using counters. decimal place number to a 1- decimal place number 3.3 Calculating the 3.3 Demonstrating the difference between a multiplication of two 2-digit and a 1-digit 1-decimal place number numbers
3.4 Applying subtraction of 3.4 Solving a word two 2-digit numbers to problem involving a word problem simple decimals using
logic or division
4.1 Applying multiplication 4.1 Matching a fraction facts to contextual to its representation word problem using shaded circles
4.2 Using number sense to 4.2 Adding two fractions calculate a product with the same denominator 4.3 Using knowledge of 4.3 Subtracting a simple
Task4 multiplication tables to Task4 fraction from one divides a 2-digit number Fractions whole by a 1-digit number 4.4 Applying multiplication 4.4 Using fractions to Multiplication Division & tables knowledge to a solve a word problem word problem involving in a familiar context sharing.
5.1 Identifying the 5.1 Making a statement categories represented to interpret data in a in a pictograph double bar graph 5.2 Calculating the 5.2 Calculating the total frequency for one number of items category in a represented in a pictograph single bar graph 5.3 Calculating the 5.3 Calculating the difference in average (mean)
Task5 frequencies for two Task5 number of items in categories in a the single bar graph pictograph
StatisticsProbability & 5.4 Identifying the most StatisticsProbability & 5.4 Identifying the most likely outcome for likely outcome for a drawing one cube from spinner given four a bag of six coloured diagrams cubes
TA 8066: REG - Quality Primary Education in the North Pacific 54 Funded by Asian Development Bank
6.1 Selecting a square (or 6.1 Identifying an obtuse rectangle) from a (or acute) angle from picture of six shapes four diagrams 6.2 Identifying the number 6.2 Identifying the name of sides (or corners) of of a triangle in a the square (or geometrical diagram rectangle) (equilateral or isosceles) 6.3 Folding a paper square 6.3 Identifying a picture (or rectangle) to of rotation (or demonstrate an reflection) Task6 Task6 alternative line of transformation from a symmetry choice of three familiar contexts. 6.4 Calculating the 6.4 Choosing a Geometry& Measurement Geometry& Measurement perimeter of a square statement that best (or rectangle) for a matches an word problem and illustration of two diagram, with correct pencils (shorter or units. longer) Source: Development Strategists International Consulting (DSIC)
117. An analysis across tasks makes it possible for identifying levels of understanding of other key mathematical concepts. This requires differentiation within the range of responses for individual items, particularly for those generating a score of 2 or 1. The following table identifies these response items with a third level notation, i.e. 6.1.2 indicates a correct response, (a square), while 6.1.1 would indicate a partially correct response identifying a shape with some common properties of a square, (a rectangle). So a response that is neither a square nor a rectangle would be coded as 6.1.0.
Table 10. Responses to Numeracy Tasks
Grade 3 Grade 5 Concept Responses Concept Responses Counting less than 100 1.1.2 Number representations 1.1.2 1.2.1 1.1.1 2.2.1 1.2.2 1.2.1 3.1.2 3.1.1 4.1.2 4.1.1 Step counting 1.4.2 Ordering numbers 1.3.2 1.4.1 1.3.1 2.1.2 1.4.2 1.2.1 1.4.1 3.1.2 3.1.1 Measurement units 2.4.2 Operation algorithms for 2.1.1 2.4.1 whole numbers 2.2.1
TA 8066: REG - Quality Primary Education in the North Pacific 55 Funded by Asian Development Bank
6.4.2 2.3.1 2.4.1 Adding 3.1.2 Addition & multiplication 3.2.2 3.1.1 algorithms with decimals 3.2.1 3.2.2 3.3.2 3.2.1 3.3.1 Subtracting 3.3.2 Addition & subtraction 4.2.2 3.3.1 algorithms with fractions 4.2.1 3.4.2 4.3.2 3.4.1 4.3.1 4.4.1 Multiplication tables 4.1.2 Geometrical terms 6.1.2 4.1.1 6.1.1 4.2.2 6.2.2 4.2.1 6.2.1 4.3.2 6.3.2 4.3.1 6.3.1 4.4.2 4.4.1
Dividing or sharing 4.3.2 Measurement representation 3.1.2 4.3.1 and units 3.1.1 4.4.2 3.4.2 4.4.1 3.4.1 6.4.2 6.4.1 Solving word problems 3.4.2 Solving word problems 2.2.2 3.4.1 2.2.1 4.1.2 2.3.2 4.1.1 2.3.1 4.4.2 2.4.2 4.4.1 2.4.1 5.2.2 3.4.2 5.2.1 3.4.1 5.3.2 4.4.2 5.3.1 4.4.1 6.4.2 5.2.2 6.4.1 5.2.1 Source: Development Strategists International Consulting (DSIC)
118. It is also possible to match EGLA Numeracy Items that relate to the stages described in the Number Framework that underlies the New Zealand Numeracy Development Project.
119. The Number Framework. At the core of the Numeracy Development Project is the Number Framework. This framework was established to help teachers, parents, and students to understand the requirements of the Number knowledge and Number strategies sections of the New Zealand Curriculum.
120. In the two main sections of the framework, the distinction is made between strategy and knowledge. The strategy section describes the mental processes students
TA 8066: REG - Quality Primary Education in the North Pacific 56 Funded by Asian Development Bank
use to estimate answers and solve operational problems with numbers. The knowledge section describes the key items of knowledge that students need to learn. It is important that students make progress in both sections of the framework.
121. The strategy section of the framework consists of a sequence of global stages. Progress through the stages indicates an expansion in knowledge and in the range of strategies that students have available. By matching student responses to the EGLA Numeracy tool to the Number Framework, department of education officials can identify common gaps in numeracy understanding, knowledge and skills that teachers need to address as the appropriate ‘next-steps’ for learning.
122. The application of number knowledge and mental strategies is often described as 'number sense'. A strongly developed number sense provides the essential foundation for algebraic thinking.
Figure 4. The Number Framework (Source: http://new.nzmaths.co.nz/numeracy-development-projects-number-framework)
123. EGLA Numeracy response items relating to the NZ Number Framework are:
Grade 3 Grade 5 Responses Responses Stage 0: The student is unable to consistently count 1.1.0 Emergent a given number of objects because they lack knowledge of counting sequences and/or one-to-one correspondence. Stage 1: One-to- The student is able to count a set of 1.1.1, 1.2.0 one counting objects or form sets of objects but cannot solve problems that involve joining and 3.1.0 separating sets.
TA 8066: REG - Quality Primary Education in the North Pacific 57 Funded by Asian Development Bank
Stage 2: The student is able to count a set of 1.1.2, 1.2.1, Counting from objects or form sets of objects to solve 1.3.0, 1.4.0 one on materials simple addition and subtraction problems. The student solves problems by counting all the objects. Stage 3: The student is able to visualise sets of 1.2.2, Counting from objects to solve simple addition and 1.3.1,1.4.1 one by imaging subtraction problems. The student solves problems by counting 2.1.0, all the objects. 3.1.1, 3.2.0, 3.3.0, 3.4.0 Stage 4: The student uses counting on or counting 1.3.2, 1.4.2 Advanced back to solve simple addition or counting subtraction tasks. 2.1.1
3.1.2, 3.2.1, 3.3.1, 3.4.1 Stage 5: Early The student uses a limited range of 2.1.2 2.1.2, 2.2.1, additive part- mental strategies to estimate answers and 2.3.1 whole solve addition or subtraction problems. 3.2.2, 3.3.2, These strategies involve deriving the 3.4.2 answer from known basic facts, (eg. doubles, fives, making tens). 4.1.0. 4.2.1, Stage 6: The student can estimate answers and 4.1.1, 4.2.2, 2.2.2, 2.3.2, Advanced solve addition and subtraction tasks 4.3.1, 4.4.1 2.4.1, additive/early involving whole numbers mentally by multiplicative choosing appropriately from a broad 3.3.2, 3.2.2 part-whole range of advanced mental strategies (eg. place value positioning, rounding and compensating or reversibility). The student uses a combination of known facts and a limited range of mental strategies to derive answers to multiplication and division problems, (eg. doubling, rounding or reversibility). Stage 7: The student is able to choose 3.1.2, 2.4.2 Advanced appropriately from a broad range of multiplicative mental strategies to estimate answers and 4.3.2, 4.4.2 3.4.1 part-whole solve multiplication and division problems. These strategies involve partitioning one 4.1.2 or more of the factors, (eg. place value partitioning, rounding and compensating, reversibility). Stage 8: The student can estimate answers and 3.1.2, 3.4.2, Advanced solve problems involving the multiplication proportional part- and division of fractions and decimals 4.2.2, 4.3.2, whole using mental strategies. These strategies 4.4.1, 4.4.2 involve recognizing the effect of number size on the answer and converting
TA 8066: REG - Quality Primary Education in the North Pacific 58 Funded by Asian Development Bank
decimals to fractions where appropriate. These students have strongly developed number sense and algebraic thinking. Source: Development Strategists International Consulting (DSIC)
The Student Preliminary Interview
124. The analysis of the EGLA literacy and numeracy data with the student preliminary Interview responses also provides a rich source of research information that the DoE/MoE can draw from to determine the future professional learning needs of principals, teachers and communities to enhance quality learning in classrooms.
125. The 17-question interview provides demographic data for each student. While some questions have a specific mathematical focus (e.g. questions 5, 6 & 7), every other question has some relevance for both literacy and numeracy learning, and are therefore useful indicators to distinguish factors that may promote or hinder learning in both fields.
126. A correlation of students’ responses to the preliminary interview questions with levels of performance in the assessment has the potential for providing answers to the following sample research questions:
For literacy:
i. Are there aspects of the students’ language experience outside of the classroom that are predictive of their language performance in the classroom?
ii. To what extent does family involvement, support and participation have an effect on students’ levels of literacy in their vernacular and in English?
iii. Does access to TV correlate with better language skill generally, or with better skill in either L1 or L2?
iv. Can any correlations be made with the non-school environment for students who show the best levels of literacy ability in both languages?
v. Can any factors be identified which seem to guarantee that students do not perform well in reading and writing their own language?
For numeracy:
i. Which contextual factors relate to high numeracy outcomes in Kosraean/Marshallese/Pohnpeian?
ii. Which contextual factors relate to high numeracy outcomes in English?
TA 8066: REG - Quality Primary Education in the North Pacific 59 Funded by Asian Development Bank
iii. Do students who perform well in literacy in the vernacular language also perform well in numeracy in the vernacular language (and vice versa)?
iv. Do students who perform well in literacy in English also perform well in numeracy in English (and vice versa)?
v. Is there any evidence of gender difference in numeracy outcomes at Grades 3 or 5?
Initial Item Writing and Review
127. The writing of the first draft of the EGLA instruments was undertaken in Majuro, RMI. As in all locations, small teams were formed made up primarily of the resident subject specialists, working closely with the QPENP Literacy and Numeracy Consultants. Agreement was reached to structure both the Grade 3 and Grade 5 tools according to the key constructs as identified and discussed above.
128. The assessment tool needed to be structured in such a way that it could be administered as a one-to-one interview for an assessor and student, and be conducted within a 10-12 minute time frame. Tasks were developed that addressed an appropriate sample of knowledge and skills for each construct, according to their importance, sequence in conceptual development. There needed to be a range of levels of difficulty, and a balance of knowledge and skills, requiring a range of simple to advanced thinking, and incorporating problems with meaningful (familiar) contexts. Separate sets of tasks were developed in English for the two grades.
129. In literacy, the tasks were largely oriented around the interaction with an authentic text, and in numeracy, twenty-four items (questions) were written for each grade reflecting the requirements above.
130. Once the English versions were prepared, the local language versions were worked on. Some of the M/NDOEs have recognized language experts and translators who became involved in this task. This proved to be quite a complex process because it needed to be understood that the vernacular language versions were not identical translations of the English versions, but were parallel versions of the same format and same level of difficulty. It would not make sense to pose exactly the same problem or question first in one language, and then in the other language.
131. For literacy, this meant that completely new stories had to be written, and new sets of questions and tasks based on the content and wording of the new text had to be prepared.
132. For numeracy, twenty-four further questions for each grade were written using different examples, numbers or information from the English version. These parallel items were questions of equal complexity that assessed the same concepts, although frequently incorporating an alternative context.
TA 8066: REG - Quality Primary Education in the North Pacific 60 Funded by Asian Development Bank
133. All of the vernacular language materials were either written first in English and then translated. If they were written in the vernacular, it was necessary for them to be back-translated to English so that both the Consultants and local language specialists who are members of the EGLA team could verify the accuracy of the content.
134. Translation of items was a time-consuming task resulting in a number of key issues that generated debate, and bringing about some useful collaboration between the mathematics and language specialists. One particular difficulty lay with technical words that do not have vernacular equivalents. To translate some technical words into Marshallese generates a phrase or description of a concept rather than a specific name, so an item requiring a student to identify a rectangle from a set of pictures of other quadrilaterals is not of the same complexity in different languages. Further, it emerged that in a Marshallese classroom, even at Grade 2 or 3, a rectangle would be identified by its English term. This dilemma was resolved by agreement that the EGLA numeracy tool should primarily assess mathematical understanding rather than complications related to literacy, and so, if a student is normally introduced to a particular concept using an English word, then any EGLA Numeracy tool in the vernacular version should also use the familiar English word.
135. Another issue arose with the language choice for numerals. It was agreed that where numerals were written as words or figures, assessors would use only English for the English EGLA numeracy version and only Marshallese for the vernacular EGLA numeracy version.
136. Many Micronesian languages complex orthographies with various special characters involving diacritics, and many subject experts lacked confidence in their own ability to write the standard form of the language correctly, resulting in the preparation, editing and correction of numerous rounds of drafts. Also, some of the languages have highly marked stylistic variants, and there was a lot of discussion about whether more formal or less formal styles should be used in the EGLA assessment materials. As always with language issues, local personnel did not always agree among themselves what might be the most appropriate translation or rendering for a particular expression.
Development of the Pre-Test Forms
137. The various assessment documents were designed and prepared as the EGLA literacy and numeracy assessment tools. These included the assessment administration forms, including the script for each assessor to follow when interviewing a student, the tools used in the interview (literacy text, and numeracy question booklet), and the response forms for assessors to record a student’s responses to each item. Eight sets of these documents were prepared, both literacy and numeracy for Grade 3 in the vernacular, Grade 3 in English, Grade 5 in the vernacular, and Grade 5 in English.
138. In these first versions of the assessment forms, the assessors recorded a student’s responses on a form by ticking a box labelled 1 for a correct answer or a box labelled 0 for an incorrect answer. Acceptable correct answers were specified but assessors were asked to record alternative and incorrect responses in a separate box to create
TA 8066: REG - Quality Primary Education in the North Pacific 61 Funded by Asian Development Bank
evidence of the range of responses that each item generated. These responses were useful for modifying the response forms in later revisions of the EGLA tools. Also, in the first Majuro versions of the numeracy tool, assessors were also required to total the number of correct responses among the four items of each task and to tick whether the student performed according to the scale: 4-advanced, 3-proficient, 2-developing, 1 or 0- beginning. However, this was dropped in later versions as the response forms were prepared with bubbles for recording answers, and the overall evaluations were automatically generated from the database.
Pilot Testing and Pre-Testing
139. For the initial trial or pre-testing, the RMI Ministry of Education nominated a team of education officers to be trained as assessors for EGLA, along with the specialists who had developed the tools. Morning was designated for trial or pre-testing at Delap Elementary School with a target of 10 students each from Grades 3 and 5 for each assessor. However, the assessing took much longer than anticipated due to unfamiliarity with the items and scripts so this target number of students was not achieved. Nevertheless, a valuable amount of data was collected from the pre-testing that resulted in considerable refinements of the assessment procedure and the instrument. Modifications were made in items to change the difficulty level, context, choice of words, technical terms and to compose alternative translations.
140. To speed up the assessment process, additional personnel were recruited and trained for the Pre-testing of the EGLA Numeracy tool. Procedures were refined to enable the assessment to be conducted more efficiently by having some of the initial preliminary interviews administered by teachers and EGLA assessors and to vary the order of the assessments according to availability of the assessors. Some assessors were more flexible and could assess in either language, and more of the QPENP consultants participated as additional assessors as necessary to avoid hold-ups.
141. The time for each assessment varied according to student ability and language competence. A particular issue in numeracy was the question of how long a student should be given to think about a problem before calculating an answer, or how much time should be given to them to demonstrate their own methods of solution that were not always the more efficient strategies. It has become evident that some students did struggle to answer some of the items due to lack of understanding of the instructions or language, or for items that they had yet to acquire the knowledge or skills necessary to answer. Hence, it was agreed that assessors could omit the more complex items should a student struggle with some of the easier items within a task to avoid unnecessary frustration for the student, and omitted items were identified appropriately on the response sheet. This is where the idea of having both a short version and a long version of the literacy component was developed.
142. The pretesting of the EGLA tools on Majuro was conducted with Grade 4 and Grade 6 students at Delap and Rita Elementary schools over four days. Grades 4 and 6 were chosen to be assessed for the pretest because those students were more likely have covered the full range of curriculum topics included in the assessment than the current Grade 3 and Grade 5 students and thus provide a more reliable evaluation of
TA 8066: REG - Quality Primary Education in the North Pacific 62 Funded by Asian Development Bank
the suitability of the tool. The teams of assessors were doubled at eight and though the first assessments were very slow, the pace increased as the assessors became more familiar with the scripts and requirements. The data was collated ready for entering into the Ministry computer for analysis of the psychometric properties. Feedback was sought from the assessors on the content and administration of the assessment tools in Majuro and this provided valuable information for future revisions.
Item Review and Revisions
143. Two further rounds of try-out and pre-testing of the EGLA tools were subsequently completed in Kosrae and Pohnpei. In each location, the same steps were repeated, checking curriculum documents for key content and skills, developing task constructs, writing parallel sets of items and translating one set of questions and instructions into the vernacular languages. However, it proved possible to adapt and build on the material developed in Majuro and further modified in Kosrae that saved considerable time. Previously successful items were often retained or sometimes modified with a fresh context suitable for the local culture. New items were written to replace the ones that had proved too complex or took students too long to answer. In literacy, the idea of using a set of word cards as a supplementary tool for the weakest students was developed, and in numeracy, some items involving terms such as symmetry and perimeter were redrafted to assess the concept rather than the knowledge of technical words by providing a practical task such as folding paper or calculating the distance an ant would walk around an object.
144. The accumulated evidence of common student errors, strategies and misunderstandings enabled the response forms to be further redesigned for Pohnpei. In numeracy, the previous two-way responses (correct or incorrect) were expanded to record an additional category. Bubble sheets were created to record three possible responses: fully correct, a partial answer or a correct intermediate step, and incorrect. In each location, it was possible to improve the procedures to administer the EGLA assessment tool and to develop a set of instructions for assessors to follow to ensure greater uniformity in its use.
Development of the Final Form
145. The first Baseline Survey was scheduled in February 2014. The development of the EGLA tools for the Baseline Survey followed a review of the pretesting in Majuro, Kosrae and Pohnpei and the reports of the reliability and validity of the EGLA revealed in psychometric testing.
146. Adhering to the constructs that had been agreed to in all three locations, two parallel Grade 3 and Grade 5 assessment tools were developed in English drawing on the items that had proved reliable and valid during the pretesting. One version was used for the common English assessment tool and the other was translated into Marshallese, Kosraean and Pohnpeian for the vernacular versions. Once again, the work of preparing, checking and validating the vernacular language versions of the assessment tools proved to be a more demanding and time-consuming task than had been
TA 8066: REG - Quality Primary Education in the North Pacific 63 Funded by Asian Development Bank
anticipated. The translations were communicated by email over the Christmas/New Year period and despite a tight timeline to allow for printing, the task was completed satisfactorily.
147. In literacy, the format of the response form was revised to include more information about the rubrics that assessors were to follow in evaluating performance in some tasks.
148. In numeracy, the format of the Question Booklet was retained and supplemented with some Grade 3 manipulatives such as counters, cubes and paper squares and rectangles for folding, At Grade 5, pictures of familiar manipulatives in the Question Booklet were used rather than actual solid objects,
149. Also, the Assessment Administration Forms for both literacy and numeracy were standardized to give a common look. The Response Forms continued with the bubble sheet formats first adopted in Pohnpei.
150. Assessors were also provided with the EGLA literacy and numeracy instructions documents that outlined the procedures and expectations of each person involved in administering the EGLA Baseline Survey. This document served as a guide during the assessors’ training workshop and as a reference during the initial implementation.
151. To facilitate the assessment for a larger group of subjects, more M/NDOE staff were trained to assist in administering the Baseline Survey. However, the logistics of the assessment needed to be well-planned, as it was found that having eight separate assessment stations in one standard size classroom is workable, but more than that the room is crowded and noisy.
152. Twelve Ministry staff were trained as assessors in RMI for Numeracy and 12 for Literacy, 8 Department staff were trained as assessors in Kosrae for Numeracy and 8 for Literacy, and 16 Department staff were trained as assessors in Pohnpei for Numeracy and 11 for Literacy (including 2 from FSM National Department of Education). The Baseline Survey was administered in 5 schools in RMI, 6 schools in Kosrae and 4 schools in Pohnpei. A total of 120 students were surveyed in Pohnpei both at Grade 4 and Grade 6. The student totals for RMI and Kosrae were 120 and 104 respectively, in each of the two grades.
153. The EGLA assessment goes hand-in-hand with professional development support in all locations, and it is hoped that even though the PD inputs are necessarily brief and limited, they will result in some improvements in what happens in the classrooms, and further, in student performance. The only way to measure this is to have an experimental group and a control group of subjects. In addition to the five schools in Majuro that are participating in the QPENP project, whose teachers are being exposed to more in- service training, and which are being provided with some additional resources, the EGLA instrument was taken to two further schools with a total of 40 subjects. These constitute the control group, and it remains to be seen after a further round of assessments in 2015 whether any positive impact of the very limited PD in the project’s experimental schools
TA 8066: REG - Quality Primary Education in the North Pacific 64 Funded by Asian Development Bank
has taken place, compared to the control schools where there has been no similar intervention.
G. Psychometric Properties of the EGLA based on the Pretesting
Item Characteristics
154. The items of the EGLA literacy and numeracy for grades 3 and 5 were calibrated using Item Response Theory (IRT), more specifically the Rasch model. The Rasch model is an item response theory approach to analyzing test items as opposed to the classical test theory. Each item on a test has its own item characteristic curve that describes the probability of getting each particular item right or wrong given the ability of the test takers.26
155. The Rasch model27 is a one-parameter model where item discrimination and guessability is constant. The calibration of test item difficulty in Rasch model is independent of the person used for the calibration. This means that the test items’ difficulty remains unchanged even if the test is administered to different samples28. The Rasch model is a probabilistic uni-dimensional model which asserts that:
i. the easier the question, the more likely the student will respond correctly to it, and
ii. the more able the student, the more likely he/she will pass the question compared to a less able student.
156. Responses of students that follow these two assumptions are said to indicate that an item is fit. When data fit the model, the relative difficulties of the questions are independent of the relative abilities of the students, and vice versa. Items that fit the model were determined using the Mean Square Values within 0.8 to 1.2 and/or z values above 2.00.
157. Majority of the items in the literacy and numeracy fits the Rasch Model. Hence, it could be said that items in the EGLA literacy and numeracy assure probability of being reliable and have strong capability to measure the best characteristics of the test items.
Test Characteristic Curves of the EGLA Literacy
158. The following are the test characteristic curves for the literacy indicating the acceptability of the items:
26 Kaplan, R. M., & Saccuzzo, D. P. (1997). Psychological testing: principles, applications, and issues (4th ed). Pacific Grove, CA: Brooks-Cole. 27 The Rasch model, named after Georg Rasch, is a psychometric model for analyzing categorical data, such as answers to questions on a reading assessment or questionnaire responses, as a function of the trade-off between (a) the respondent's abilities, attitudes or personality traits and (b) the item difficulty. 28 Rasch, G. (1977). On Specific Objectivity: An attempt at formalizing the request for generality and validity of scientific statements. The Danish Yearbook of Philosophy, 14, 58-93.
TA 8066: REG - Quality Primary Education in the North Pacific 65 Funded by Asian Development Bank
Figure 5. TCC of Literacy 3 L2 – RMI Figure 6. TCC of Literacy 3 L1 – RMI
Figure 7. TCC of Literacy 5 L2 – RMI Figure 8. TCC of Literacy 5 L1 – RMI
Figure 9. TCC of Numeracy 3 L2 – Kosrae Figure 10. TCC of Literacy 3 L1 – Kosrae
TA 8066: REG - Quality Primary Education in the North Pacific 66 Funded by Asian Development Bank
Figure 11. TCC of Numeracy 5 L2 – Kosrae Figure 12. TCC of Literacy 5 L1 – Kosrae
Figure 13. TCC of Numeracy 3 L2 – Pohnpei Figure 14. TCC of Literacy 3 L1 – Pohnpei
Figure 15. TCC of Numeracy 5 L2 – Pohnpei Figure 16. TCC of Literacy 5 L1 – Pohnpei
TA 8066: REG - Quality Primary Education in the North Pacific 67 Funded by Asian Development Bank
Test Characteristic Curves of the EGLA Numeracy
159. The following are the test characteristic curves for the numeracy:
Figure 17. TCC of Numeracy 3 L2 – RMI Figure 18. TCC of numeracy 3 L1 – RMI
Figure 19. TCC of Numeracy 5 L2 – RMI Figure 20. TCC of numeracy 5 L1 – RMI
Figure 21. TCC of Numeracy 5 L2 – Kosrae Figure 22. TCC of numeracy 5 L1 – Kosrae
TA 8066: REG - Quality Primary Education in the North Pacific 68 Funded by Asian Development Bank
Figure 23. TCC of Numeracy 5 L2 – Kosrae Figure 24. TCC of numeracy 5 L1 – Kosrae
Figure 25. TCC of Numeracy 3 L2 – Pohnpei Figure 26. TCC of numeracy 3 L1 – Pohnpei
Figure 27. TCC of Numeracy 3 L2 – Pohnpei Figure 28. TCC of numeracy 3 L1 – Pohnpei
TA 8066: REG - Quality Primary Education in the North Pacific 69 Funded by Asian Development Bank
Reliability of EGLA
160. The reliability of the EGLA was established in two ways. One is the internal consistency of items. Another is the parallel form of reliability in two language versions of the assessment: English and the native language. Based on the internal consistency values obtained through Cronbach’s alpha, item and person reliability are all acceptable.
161. For the correlation among the L1 and L2 version of the EGLA, majority of the correlation coefficient values are significant (r>.20).
Table 11. Internal Consistencies of Literacy
RMI Kosrae Pohnpei Gr 3 Gr 5 Gr 3 Gr 5 Gr 3 Gr 5 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 Cronbach’s .87 .91 .92 .93 .84 .78 .84 .79 .91 .92 .90 .93 Alpha
Item .98 .95 .98 .93 .99 .96 .99 .86 .98 .99 .98 .85
Person .98 .96 .98 .91 .99 .72 .98 .72 .91 .93 .97 .91
Source: Development Strategists International Consulting
Table 12. Internal Consistencies of Numeracy
RMI Kosrae Pohnpei Gr 3 Gr 5 Gr 3 Gr 5 Gr 3 Gr 5 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 Cronbach’s .74 .77 .72 .72 .85 .77 .76 .79 .85 .90 .71 .70 Alpha
Item .94 .95 .91 .94 .97 .79 .96 .77 .83 .85 .72 .75
Person .92 .95 .91 .95 .99 .72 .96 .79 .87 .88 .73 .69
Source: Development Strategists International Consulting
Validity of the EGLA
162. The validity of the EGLA was obtained through content validity, convergent validity, and factorial validity.
163. Content validity was established by constricting the items of the EGLA based on the educational standards in Micronesia – in the RMI and FSM. The items were reviewed based on the standards both by curriculum specialists and international consultants.
TA 8066: REG - Quality Primary Education in the North Pacific 70 Funded by Asian Development Bank
164. Convergent validity was established by correlating the constructs of the literacy and correlating the constructs of numeracy. The correlations were conducted by levels and by country. Significant and positive correlations were obtained (r>.20) in all the constructs.
165. Factorial validity was established by conducting a Confirmatory Factor Analysis (CFA). A measurement model that was constructed composed of the factor of the numeracy and literacy constructs. The indicators were the items under each construct. The latent constructs were inter-correlated because the domains that were related to the procedure would establish convergence of the factors. Significant parameter estimates should be produced to establish the relationship among the latent constructs. The components should have significant estimates as well in order to provide proofs of inclusion of the component for their respective latent constructs. The results of the CFA showed that the model was supported by the data with adequate fit indices obtained.
Table 13. Fit of the Measurement Model for Literacy
RMI Kosrae Pohnpei Gr 3 Gr 5 Gr 3 Gr 5 Gr 3 Gr 5 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 χ2 744.0 790.1 773.8 15.1 66.1 690.1 64.8 65.2 1690.9 1710.0 1029.5 1117.6 3 1 9 4 3 1 9 3 9 4 7 3 RMS .08 .06 .08 .04 .01 .05 .01 .01 .09 .09 .09 .08 RMSE .09 .05 .09 .06 .02 .04 .02 .01 .09 .08 .09 .10 A CFI .92 .92 .95 .93 .95 .94 .97 .94 .90 .91 .90 .90 GFI .98 .91 .93 .94 .94 .93 .97 .94 .90 .91 .90 .91 NFI .94 .97 .99 .96 .95 .95 .91 .93 .91 .92 .91 .91 CFI = Comparative Fit Index, GFI=Goodness-of-Fit IndexNFI= Bentler-Bonett Normed Fit Index; PGI = RMS = Root Mean Square; RMSEA = Root Mean Square of Error Approximation Source: Development Strategists International Consulting
Table 14. Fit of the Measurement Model for Numeracy
RMI Kosrae Pohnpei Gr 4 Gr 6 Gr 4 Gr 6 Gr 4 Gr 6 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 L2 L1 χ2 13.1 15.1 33.2 35.1 180.8 225.0 265.5 194.6 373.1 466.8 282.9 289.6 1 4 4 6 3 5 3 9 7 4 6 4 RMS .03 .04 .06 .07 .01 .01 .01 .01 .09 .09 .08 .08 RMSE .04 .06 .08 .09 .01 .02 .02 .01 .06 .09 .03 .08 A CFI .92 .93 .90 .90 .92 .90 .90 .90 .92 .90 .97 .98 GFI .95 .94 .91 .91 .92 .90 .90 .90 .90 .92 .91 .91 NFI .94 .96 .95 .91 .95 .91 .91 .92 .95 .91 .93 .90 CFI = Comparative Fit Index, GFI=Goodness-of-Fit IndexNFI= Bentler-Bonett Normed Fit Index; PGI = RMS = Root Mean Square; RMSEA = Root Mean Square of Error Approximation Source: Development Strategists International Consulting
TA 8066: REG - Quality Primary Education in the North Pacific 71 Funded by Asian Development Bank
V. THE EGLA BASELINE SURVEY RESULTS: SUMMARY OF MAJOR FINDINGS
166. This section presents and discusses the results and major findings of the EGLA baseline surveys conducted in 2014 and in 2015. Separate country reports are provided to each of the project sites – Kosrae, Pohnpei and Majuro.
A. Profile of Respondents of the 2014 - 2015 Baseline Survey
167. The EGLA was administered twice during the project implementation. It was first administered in February 2014 and again in February 2015.
168. In February 2014, the EGLA Literacy and Numeracy tools were administered in Majuro, RMI to 159 Grade 3 students and 160 Grade 5 students in four schools. Five schools were classified as experimental or pilot schools – Ajeltake Elementary School, Delap Elementary School, Laura Elementary School, Rita Elementary School, and Woja Elementary School while the two other schools were tagged as control group – Rairok ES and Uliga ES. In Kosrae, FSM, the EGLA Literacy was administered to 134 Grade 3 and 128 Grade 5 students, while EGLA Numeracy was administered to 132 Grade 3 and 152 Grade 5 students in all seven schools in Kosrae, namely – Tafunsak Elementary School, Sansrik Elementary School, Lelu Elementary School, Walung Elementary School, Malem Elementary School and Utwe Elementary School. Finally, in Ponhpei, the EGLA Literacy and Numeracy were administered to 119 Grade 3 and 120 Grade 5 students in four schools in Ponhpei – Kolonia Elementary School and Ohmine Elementary School as experimental or pilot schools, while Netts Elementary School and Awak Elementary School were the control or non-pilot schools.
169. In the second baseline survey administered in February 2015, the same schools in all projects were covered and the consultants ensured that approximately the same number of students were assessed in both EGLA literacy and numeracy. A total of 413 Grade 3 students were assessed in three projects sites, while 410 Grade 5 were assessed from the same sample schools. Overall, there were 823 students who were assessed during the second baseline survey.
Table 15. Distribution of Respondents for 2015 Baseline Survey according to Grade and Gender by Project Site
Project Site Grade 3 Grade 5 Male Female Total Male Female Total Majuro, RMI 83 77 160 80 79 159 Kosrae, FSM 52 52 104 46 58 102 Pohnpei, FSM 75 74 149 76 73 149 Total 210 203 413 202 210 410 Source: Development Strategists International Consulting
TA 8066: REG - Quality Primary Education in the North Pacific 72 Funded by Asian Development Bank
B. Major Findings, Results and Impact of EGLA Literacy and Numeracy
170. The first EGLA baseline was carried out in February 2014. Following this, input a professional development program for teachers took place in three short phases, and learning resources were provided to target schools. The project activities stimulated a new level of engagement across the education sectors in each project site, with the curriculum specialists spending more time in schools, and working more collaboratively with principals and teachers to implement planned strategies in the target schools.
171. The second EGLA baseline or endline, was conducted in February 2015 with a new cohort of Grade 3 and Grade 5 students, using basically the same instrument with minor refinements. Aggregated results29 in both baseline and the endline showed the following:
i. Student performance in English and their first language are highly correlated. However, students performed better in their first language.
ii. In literacy, while students performed well in fluency, vocabulary, reading comprehension, and writing for meaning, students are still significantly challenged in foundational skills for reading, such as alphabetic principles, phonological awareness, and phonics.
iii. In numeracy, students showed some mastery in geometry, measurement, and statistics. However, they also showed that they are still at a more basic developmental stage when given the tasks of operations, particularly multiplication and division.
iv. Information about some social and home-related factors was gathered through the student interview as part of the EGLA administration process, with the anticipation that such factors might explain or at least correlate with language and numeracy proficiency patterns. However, no significant factors were identified which, when analyzed could be seen as having a significant impact and influence on literacy and numeracy performance tasks.
172. The main reason why a second baseline survey was administered was to determine whether there was a difference after one year with some interventions such as provision of learning resources and evidence-based professional development programs. After one of interventions, the following changes were observed between the EGLA baseline and endline:
i. Across all project sites, the percentage of proficient to advanced readers increased by an average of 21%, leading to a decreased percentage of readers at the lower beginning and developing stages.
29 More detailed presentation and discussion of results in each project site are in Country Reports which are annexed in main report.
TA 8066: REG - Quality Primary Education in the North Pacific 73 Funded by Asian Development Bank
ii. The percentage of proficient to advanced numerates increased by an average of 19%, which also led to a decreased percentage of beginning and developing numerates. iii. Looking at the results across different learning components, the number of students responding correctly at endline increased by 25% on average. iv. Students who performed well in the vernacular, tended to perform well in English. Performing well in one language in numeracy and literacy seems to be predictive of performing well in the other language. This runs contrary to the former prevailing view that in order to ensure good development in English, the use of the vernacular should be downplayed, and instead reinforces the idea that building a strong foundation in the vernacular will be supportive of eventual improved outcomes in English. v. The factors or areas of EGLA literacy and numeracy that were identified as areas of challenges in 2014 were still identified as such in 2015. However, the percentage of students getting correct responses increased.
TA 8066: REG - Quality Primary Education in the North Pacific 74 Funded by Asian Development Bank
C. Results of EGLA Literacy
173. The EGLA results presented in this report summarize the comparison between 2014 and 2015 percentage of correct response in each construct in every project site.
174. Grade 3 Literacy (First Language). When the children were assessed using their first language, that is, Pohnpeian, Kosraean, and Marshallese, similar results have emerged. There are more children who answered correctly in 2015, than in 2014. Except for Fluency for Pohnpeian sample, all percentages increased. However, in the same sample, it is evident that children significantly improved in response in Word Recognition and Listening comprehension. The following figures show the comparison of 2014 and 2015 percentage of correct response in EGLA Literacy Grades 3 and 5 in first language and English.
Figure 29. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 3 Literacy (First Languages – Pohnpeian, Kosraean and Marshallese)
175. Grade 3 Literacy (English). For each of the nine literacy constructs assessed in Grade 3 (Alphabetic principle, Phonological awareness, Phonics, High frequency words, Fluency, Vocabulary, Listening comprehension, Reading comprehension, and Writing) there were generally overall increases in performance in all locations. The sample graphs shown here are for Pohnpei, Kosrae, and Majuro. Majuro and Kosrae samples, however, show a slight decline in one factor - Writing. Further analysis will determine if such exceptions are still within margins of sampling error, or have some other explanation.
TA 8066: REG - Quality Primary Education in the North Pacific 75 Funded by Asian Development Bank
Figure 30. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 3 Literacy (English)
176. Grade 5, Literary Vernacular Language. For the vernacular language, the results are consistent with the results of English wherein percentage of correct response increased in all sub-factors, for Kosrae and Majuro, there was a slight decrease of percentage of correct response for writing for meeting. The decrease in this subject area was only noted in Grade 3 Literacy English. However, in general, the results still imply that in there is an increase in the percentage of readers compared to last year’s results.
Figure 31. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 5 Literacy (First Language)
TA 8066: REG - Quality Primary Education in the North Pacific 76 Funded by Asian Development Bank
177. Grade 5 Literacy, English. For each of the eight literacy constructs assessed at Grade 5 (Alphabetic principle, Phonics, High frequency words, Fluency, Vocabulary, Listening comprehension, Reading comprehension, and Writing) there were increases in performance in all locations.
Figure 32. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 5 Literacy (English)
TA 8066: REG - Quality Primary Education in the North Pacific 77 Funded by Asian Development Bank
D. Results of EGLA Numeracy
178. The following figures show the comparison of 2014 and 2015 percentage of correct response in EGLA Numeracy Grades 3 and 5 in the first language and English.
179. Grade 3 Numeracy (First Language). When Grade 3 students were assessed using their first language for numeracy, results also showed significant increase in the percentage of students answering the questions and performing tasks correctly. Significant increased in performance were noticed in counting, place values and statistics. However, basic operations remain an area of challenge.
Figure 33. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 3 Numeracy (First Language)
180. Grade 3 Numeracy (English). Student performance in the six constructs for numeracy in Grade 3 (counting, place value, addition and subtraction, multiplication and division, statistics, and geometry) showed significant increased from 2014 to 2015 results. However, addition and subtraction as well as multiplication and division remain the areas of challenge in all project sites.
TA 8066: REG - Quality Primary Education in the North Pacific 78 Funded by Asian Development Bank
Figure 34. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 3 Numeracy (English)
181. Grade 5 Numeracy (First Language). When Grade 5 students were assessed using their first language in numeracy, it was revealed that performance of students in 2015 is better than those students who took the same assessment in 2014.
Figure 35. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 5 Numeracy (First Language)
TA 8066: REG - Quality Primary Education in the North Pacific 79 Funded by Asian Development Bank
182. Grade 5 Numeracy (English). Among Grade 5 students, percentage of students answering the questions correctly and performing the tasks also increased. However, in all project sites, there are still about one-third of the students not being able to answer the items correctly.
Figure 36. Comparison of 2014 and 2015 Percentage of Correct Response on Grade 5 Numeracy (English)
TA 8066: REG - Quality Primary Education in the North Pacific 80 Funded by Asian Development Bank
E. Utilization of EGLA Results and Professional Development of Teachers
How can EGLA Results be Used by Education Authorities?
183. The results of the EGLA are robust and very informative. Using the results of the EGLA, the curriculum and assessment specialists can identify which areas of the curriculum content deserve the greatest attention for supporting teachers in classrooms, make plans for school wide professional development initiatives, provide manipulatives or other resources, and determine how best to assist with lesson planning and assessment. Pedagogical and conceptual development issues can be identified and the next steps for learning determined.
184. Education administrators can compare the levels of achievement for different schools. Schools, on the other hand, can compare the achievement of different classes at the same level for particular learning concepts.
185. The experience of developing and implementing the EGLA assessments has provided the education administrators with a deeper level of understanding and commitment to students’ achievement in the classroom. Education administrators and teachers involved have a stronger appreciation for where the students are, and of the need for more support to teachers.
How was the Professional Development Structured?
186. Education administrators in all the three project What is one key feature of locations followed through after the professional quality pedagogy? development workshops with more concentrated time in the schools, implementing observations, debriefs, and support for teachers. The professional Research shows that lessons development workshops required an unusually high that involve a variety of level of interaction between ministry and activities are more likely to departmental specialists, principals, and teachers. This result in increased student resulted in a new recognition of the need for engagement and improved commitment and cooperation to address the learning. challenges of improving teacher quality. Therefore, teachers should 187. Over the year and half of the project, three ensure that each teaching phases of capacity building took place for teachers, point is reinforced through a principals, and subject and curriculum specialists. range of different activities.
188. Phase 1: Improvement in student achievement is strongly linked to the quality of teaching. An initial workshop explored quality pedagogy, composed of various factors and strategies that are known to lead to improved teaching and learning in classrooms. Processes introduced were designed to ensure that both teachers and students know specifically what they are going to learn, how they are going to learn it, and how they will know when they have learnt it. Essential
TA 8066: REG - Quality Primary Education in the North Pacific 81 Funded by Asian Development Bank
aspects for improvement focused on teachers providing students with high quality, constructive feedback and feed-forward, and building positive classroom relationships. Participants were introduced to a protocol for conducting supportive peer observations and feedback.
189. Phase 2: For the second workshop, participants used the findings of the first EGLA assessment to formulate professional development initiatives designed to improve the quality of teaching and learning for numeracy and literacy. Early-grade teaching strategies relating to the understanding of quality pedagogy to numeracy, and literacy were trialed and practiced with students and teachers. Examples in literacy trainings included a focus on:
i. Decoding phonetically,
ii. Using context to determine word meaning,
iii. Reading fluently, and
iv. Responding in writing to what has been read.
190. Phase 3: The third input, with sections devoted to numeracy and literacy, saw participants consolidating their conceptual knowledge and skills, modelling, trialling and practising further quality teaching strategies. Example strategies enabling students to make sense with numeracy included:
i. Using number stories with familiar concepts
ii. Ensuring students are first introduced to a new concept by handling a number of appropriate manipulatives
iii. Encouraging students to develop mental images as strategies to solve problems
iv. Using appropriate algorithms to calculate answers and justifying strategy choices.
191. Participants learned to pose problems in familiar contexts, plan teaching episodes, demonstrate the use of manipulatives, introduce strategies that develop mental images, and to model interactions with learners to ensure concepts are understood. In most workshop components, trainers trialed activities with students and teachers in order to develop their modeling skills to work in school classrooms. Teachers worked collaboratively to plan lessons, and to observe and give feedback and feed- forward to each other.
F. Resources Provided to School to Support Literacy and Numeracy
192. Math Resources. Teachers of targeted Grade 3 and 5 classes were provided with math kits that contained tools for teaching the standard curriculum topics on number
TA 8066: REG - Quality Primary Education in the North Pacific 82 Funded by Asian Development Bank
properties, operations, statistics and probability, geometry and measurement for that particular level. The teachers also participated in professional development workshops focused on particular manipulatives to identify needs from the first EGLA survey, particularly counting strips, number frames, number arrows, based 10 blocks, and number lines.
193. In addition, a handbook was prepared for early grade math teachers, “Making Sense with Numeracy”, which includes a short introduction to numeracy, an outline of the necessary stages to develop math strategies and knowledge, and an illustrated guide to using manipulatives and other practical teaching tools to develope understanding of place value, the four operations with whole numbers, fractions and decimals, and important applications in statistics and geometry. Worksheets from various math internet sites have been reproduced for photocopying, since there are many teachers in Micronesia who are yet to have access to computers for these important resources.
194. Literacy Resources. Local teams in Majuro, Kosrae, and Pohnpei developed an inventory of readers, books, word charts, school dictionaries and other resources that they wanted printed in the local language and English. The books include a variety of topics, are written at a range of reading levels and offer multiple copies per student in grades 3 and 5 classrooms. Some resources were new, and had not been printed before, but the project was also able to reprint a number of previously developed items that were no longer available.
195. Professional development sessions included setting up reading corners, highlighting the importance of creating an environment in which reading is clearly valued. When the books arrived on the island, teachers established Reading Corners in their classrooms with students helping to determine how the books would be organized, displayed and used. The reading corner provides a quiet and comfortable place for students to read independently or with a partner or small group. In these classrooms, students know that reading matters.
G. Support to the Development of Bilingual Policy of RMI
196. One strategic additional involvement was with the RMI MOE’s initiative in language policy development for education. Project consultants ran a one-day seminar that brought together key stakeholders from educational institutions and traditional leadership to consider key issues involved in developing a bilingual education policy that jointly serves the educational and national cultural aspirations of the country. The basic design framework for the policy that was developed in the seminar forms the basis of a more elaborate language policy statement that is currently being approved for RMI. Similar language policy developments are also currently taking place elsewhere in Micronesia.
197. These policy initiatives throughout the north Pacific are strongly in favour of a bilingual education approach, and this has strengthened the value that educators are seeing in the information that EGLA can provide to them, as a bilingual assessment tool.
TA 8066: REG - Quality Primary Education in the North Pacific 83 Funded by Asian Development Bank
VI. REVIEW OF THE ASSESSMENT SYSTEM OF PALAU
198. This section discusses the review done to investigate the assessment system of Palau. This is a special output of the project that was specifically requested by the MOE of Palau in lieu of the introduction and piloting of the EGLA.
A. Introduction
199. One of the major changes of the scope of the CDTA was for Palau not to participate in the piloting of EGLA in the North Pacific. Instead, the MOE of Palau requested that an external review of its many existing assessments (Palau Achievement tests in all subjects, Quarterly assessment in all subjects, English reading pre- and post, PILNA, mathematics placement test, etc.) be conducted. Hence, the focus of this work stream or sub-output was for the Consultants to conduct an independent review that would inform the MOE Palau of the best way forward in regards to its assessment system. The Team Leader was assigned to be the primary responsible consultant for designing and conducting the review of the assessment programs of Palau. Using international benchmarks and best practices, the Team Leader performed review in terms of relevance, assessment objectives/purposes, content and coverage, utilization of assessment data, impact on teaching and learning, acceptability to the community and other stakeholders.
200. This output recognized that educational outcomes, more specifically, assessment results are common indicators of the level of performance and learning attained by students of a particular country. These assessment results are generated by using national and public examinations. In an outcome-based perspective, the results of national exams indicate how well the students have attained national/international standards and reflect the level of competency attained. The results of national and/or state-wide examinations would also have direct implications on the quality of educational programs and instruction provided by schools in general.
201. In the case of the Ministry of Education in the Republic of Palau, assessment was conducted in the classroom level and the system-wide level. The classroom level assessment would refer to quizzes, tests, and short projects that are given with a formative and/or summative purpose. The system-wide assessment refers to the national exams provided for the students that the Ministry of Education and schools administer. These system-wide assessment include the Palau English Reading Assessment (PERA), Quarterly Assessment Test (QAT), Palau Achievement Test (PAT), and the Math Placement test30.
202. The Palau English Reading Assessment (PERA) measures achievement levels in reading in the English language. The PERA measures reading covering the areas of phonemic awareness, phonemics, vocabulary, comprehension, and fluency. It is
30 Ministry of Education, Palau, 2013.
TA 8066: REG - Quality Primary Education in the North Pacific 84 Funded by Asian Development Bank
administered from grades 1 to 8 in pre and post test within the school year. The results place students in four performance levels: Advanced, proficient, basic, and developing.
203. The Quarterly Assessment Test (QAT) in Palau measures students’ mastery of the learning targets for the subjects English, mathematics, Palauan studies, science, and social studies. The QAT is administered four times in a school year from grades 1 to 8. The QAT has a high stake because it makes up 25% of the students’ grade for the quarter.
204. The Palau Achievement Test (PAT) measures students’ achievement for English, mathematics, Palauan studies, science, and social studies. The test is based on the curricular framework that was developed in 2010. The PAT is administered annually for grades 4, 6, 8, 10, and 12.
205. The Math Placement Test is administered to students planning to enter Grade 9 at Palau High School for the purpose of placing the students in the appropriate mathematics course. Students who pass the test (score of 75% and above) are placed on an algebra course while students who fail (below 75%) are placed on an applied mathematics course.
206. Aside from the four national assessments, the students in Palau also take the SAT, SAT10, PILNA, TOEFL, and ASVAP that are all mandated by the Government based on laws enacted by the Congress.
207. The primary concern of the Ministry of Education in Palau was to determine the status of their assessment system. In this study, the assessment system would refer to the four national tests administered to students (PERA, QAT, PAT, and Math placement Test). More specifically, they need to make some decisions whether there is a need to maintain the national tests administered to students and improve aspects on the administration and function of the present assessment system. Given this context, the report provided a review of the present assessment system in the Ministry of Education in the Republic of Palau. The review also provided a detailed analysis of the assessment system in order to help the Ministry of Education make sound decisions about the national assessments conducted. The review included report and analysis of the assessment system in terms of its relevance, objectives/purposes, content and coverage, utilization of assessment data, impact on teaching and learning, acceptability to the community and other stakeholders. The review also includes evaluating the status of the Palau assessment system using the criteria of the System Assessment and Benchmarking for Education Results (SABER) developed by the WB. The criteria of the SABER include the enabling context, system alignment, and assessment quality.
B. Framework for the Review
208. The first part of the study reviewed the Palau assessment system of the Ministry of Education in terms of its relevance, objectives/purposes, content and coverage, utilization of assessment data, impact on teaching and learning, acceptability to the community and other stakeholders. Table 1 contains the scope and coverage of each criteria. The definition and scope of the six criteria were adapted from the Riding and
TA 8066: REG - Quality Primary Education in the North Pacific 85 Funded by Asian Development Bank
Stufflebeam31. The Joint Committee on Evaluation developed a set of standards to evaluate the conduct of evaluation. The definitions of the standards were used in the study to determine the scope of the six criteria. The six criteria specifically matches the standards created by the Joint Committee on Evaluation.
Table 16. Scope of the Criteria in Reviewing the Assessment System of Palau Ministry of Education
Criteria Scope Indicators
Relevance Information collected is • Survey and Focus Group responsive to the needs of the Discussion (FGD) among schools and students teachers, education specialists, selected students and employers and management team of the MOE
Objective/Purpose Described purposes and • Survey and FGD among procedure teachers, education specialists, selected students and employers and management team of the MOE • Content analysis of the test items and manuals
Content and coverage Information sources, • Survey and FGD among framework, curriculum, teachers, education standards, validity, and specialists, and reliability management team of the MOE • Content analysis of the test items and manuals
Utilization of assessment data The assessment system serving • Survey and FGD among the needs of its intended users teachers, education (Ridings & Stufflebeam, 1991). specialists, selected employers and management team of the MOE
Impact on teaching and Follow through of the results • Survey and FGD among learning among the stakeholders. selected employers, teachers, education specialists, and management team of the
31 Riding, R., & Stufflebeam, D. (1991). Standards for evaluation of educational programs projects and materials. NJ: McGraw Hill.
TA 8066: REG - Quality Primary Education in the North Pacific 86 Funded by Asian Development Bank
Criteria Scope Indicators
MOE
Acceptability to the The assessment is conducted • Survey and FGD among community and stakeholders legally, ethically, and with due selected employers and regard for the welfare of those parents, teachers, involved in the assessment as education specialists, and well as those affected by its management team of the results MOE. Other government and private stakeholders may be asked to participated in FGD and surveys.
209. The System Assessment and Benchmarking for Education Results (SABER) is an initiative that helps countries examine and strengthen the performance of their education system (in this case assessment system) to achieve learning for all. The concept of SABER was used to examine the assessment system of the MOE in Palau. Additionally, the SABER instruments were examined for adaptation in the study. As an outcome, the assessment system of the MOE of Palau was assigned a level of development gauging from the extent of the assessment system they have successfully regulated. The scales range from latent to emerging to establish to advanced. The study reviewed the four national exams of the MOE in Palau using the criteria enabling context, system alignment, and assessment quality.
Table 12. Scope of the Criteria in Evaluating the Assessment System of Palau Ministry of Education
Criteria Scope Indicator
Enabling context The wider context in which the • A policy framework for assessment activity takes place and assessment activities the extent to which it is supportive of • A regular budget for the assessment. This refers to the assessment Activities broad policy framework within which • Training of teachers in the assessment is carried out, the assessment techniques institutional structures for carrying out the assessment or using the assessment results, the availability of sufficient and stable sources of funding, and the existence of effective human resources.
System Alignment The extent to which the assessment • Evidence of an alignment activity is aligned with the rest of the between assessment education system. This involves the activities and official learning connection between assessment standards or curriculum
TA 8066: REG - Quality Primary Education in the North Pacific 87 Funded by Asian Development Bank
Criteria Scope Indicator
activities and the education system’s learning goals, standards, and curriculum.
Assessment quality The technical quality of the • Regular collection of system instruments and procedures used in level data on student the assessment activity. Validity and learning levels reliability of the assessment tools • Timely dissemination of data on student learning achievement to stakeholders • Quality assurance of classroom assessment activities
C. Purpose and Methodology of the Review Study
210. The overall purpose of the review was to determine the effectiveness of the assessment system for the Palau MOE. The first part of the review described the features of the national assessment in the areas of (1) Relevance of assessment, (2) assessment objectives/purposes, (3) content and coverage, (4) utilization of assessment data, (5) impact on teaching and learning, and (6) acceptability to the community and other stakeholders.
211. The description of the six (6) areas were obtained through survey and FGDs. The second part of the report evaluated the assessment system (focused on National Tests) in Palau Ministry of Education using the SABER criteria that included enabling context, system alignment, and assessment quality. A checklist was also developed to evaluate the assessment system on the three criteria. The results provided recommendations on the assessment system of Palau. More specifically, the recommendations were classified on the following: What’s working? What’s not working? What should be done differently? And, how to keep what is working and improve what is not.
D. Summary of Results32
212. The overall purpose of the review was to determine the status and effectiveness of the assessment system in Palau that is administered by the MOE. Using international benchmarks and best practices, the review of the assessment system of Palau was focused on: (i) Relevance, (ii) Clarify of assessment objectives and purposes, (iii) Content and scope, (iv) Level of utilization of assessment data, (v) Impact on teaching and learning , and (vi) Acceptability to the community and other stakeholders.
213. The review process also evaluated the assessment system using SABER criteria such as: (i) enabling context, (ii) system alignment, and (iii) quality of assessment.
32 A more detailed report was submitted to the MOE of Palau through ADB in January 2015.
TA 8066: REG - Quality Primary Education in the North Pacific 88 Funded by Asian Development Bank
214. Overall, the assessment system of Palau that includes the administration of the QAT, PAT, PERA and MPT is very much in place. Hence, it can be inferred that the assessment of Palau is emerging, that is starting to be established and progressively developing.
215. The relevance of the assessment system is low to medium. The information and data collected through the assessment tools are responsive to the needs of the teachers, students, MOE, and parents/employers to a varying degree. The extent to which the tests address the needs of the students and identify the strengths and weakness of students’ learning also vary from one test to another.
216. The clarity of purpose and content of the assessment system is medium. There is a consensus among the respondents and stakeholders that the purposes of the tests are clearly articulated. However, they do not seem to have a clear understanding of the content of the tests. Teachers claimed that they are strictly following a learning plan on a quarterly basis but they often realized that the items included in the tests (particularly for QAT) are not aligned with the learning targets for that quarter. Students also expressed their limited appreciation of the purpose of PAT because this test has no value added in any way into their final grades.
217. Utilization of assessment results is low. The review data reveal that assessment results and data are not maximally used and utilized for policy decision, improvement and teaching and learning and overall improvement of the school system. Results are not systematically disseminated.
218. Impact of assessment to teaching and learning is low. Because results are not systematically disseminated to teachers, students and even parents, the impact of the assessment system is low. Students find in particular that PAT as a mere exercise, because they are very much informed about the use and value of the PAT results. While MOE organizes an annual meeting among the principals to share the results of the tests, these are not cascaded down to the classroom level. Teachers are seldom provided with the results, even for class level or school level results.
219. Acceptability by the users and stakeholders is medium. The level of acceptance and appreciation of the various stakeholders – students, teachers, principals, parents/employers and MOE officers varies from one test to another. Although there is a generally positive acceptance among the stakeholders, they also have some reservations, particularly on the valuation of QAT into the final grades of the students. The students do not completely appreciate the value of PAT, while teachers do not see the relevance of the PERA because of the inability of the MOE to share the results and information about the tests on a timely manner.
220. In terms of the enabling context of the assessment system, the enabling context is emerging. While the assessment system is in place, the policies that are supporting the implementation are not clearly articulated and disseminated, making the impact of the assessment into the overall educational system weak.
TA 8066: REG - Quality Primary Education in the North Pacific 89 Funded by Asian Development Bank
221. The alignment of the assessment system with the curriculum and school system is established. The review discovered that in general, tests are very much aligned with the curriculum and agreed learning standards.
222. The quality of assessment is emerging. By and large, the quality of the assessment tools are satisfactory but there are still some areas that need to be further improved, such as ensuring the cultural locatedness or validity of the test items, improving the density of the items on each page, and providing clearer pictures and figures in the test booklets. Similarly, the quality of data collection is well defined but it also necessitates some particularly on how the system disseminates results.
223. On the MOE capacity, the institutional capacity of the MOE is emerging. The infrastructure and human resources needed to implement the assessment is in place, but still require some significant improvement.
224. Other factors that are considered emerging are – (i) system mechanism to ensure quality of examinations; (ii) maintaining credibility of the exam results; and (iii) teachers are tasked to participate in the implementation and administration of assessments.
225. The major issues identified are in the areas of institutional management, assessment tools, and capacity building.
E. Recommendations and MOE’s Response
226. From the results of the review, the following recommendations are:
i. Recommendation 1: That the MOE consider developing a National School Assessment Policy Framework that will define the purpose, objectives, uses and implementing guidelines of the assessment system anchored on the national education policy and curriculum policy framework.
ii. Recommendation 2: That a functional review of the MOE unit that is in-charge of assessment activities be conducted.
iii. Recommendation 3: That the assessment tools be reviewed, revised, and updated with a support from technical advisers.
iv. Recommendation 4: That a feasibility study be conducted to determine the best timing for test administration in order to ease students from taking the tests at the same time during the fourth quarter and to prevent examination fatigue among students and even teachers.
v. Recommendation 5: That the assessment will be streamlined, particularly PAT and QAT.
vi. Recommendation 6: That a mechanism to ensure quality assurance of the assessment tools be installed including organizing an assessment technical review group.
TA 8066: REG - Quality Primary Education in the North Pacific 90 Funded by Asian Development Bank
vii. Recommendation 7: That the MOE will develop a system for reporting and dissemination of test results.
viii. Recommendation: 8: That a comprehensive capacity development for MOE staff, teachers and principals be developed and implemented to ensure that the assessment system in place become more effective and have greater impact. The areas of capacity building may include, among others –
° Classroom assessment techniques for teachers ° Development of standardized assessment and standard-based assessment for MOE staff ° Development of easy-to-understand test reports and dissemination system ° Using test results for evidence-based learning ° Conduct of institutional research using test results ° Conduct of validity and reliability research studies for MOE staff and principals ° Preparing students for examinations – tips on test taking behaviors
227. Given the recommendations above, it was strongly suggested that the MOE will convene a National Assessment Committee or Council (NAC) to:
i. Review the relevance and usefulness of all the tests, particularly QAT and PAT; ii. Discuss streamlining of the assessment tools – less but better quality and more effective and efficiently implementation process; iii. Consider a proposal to conduct a functional analysis of MOE structure on effective governance of the assessment system; iv. Propose the conduct of studies using assessment results to support teaching and learning, including validity, reliability, utility and efficacy of the tests; v. Design a capacity building program to support teachers in using assessment results; and vi. Develop a program for advocacy and information campaign about the tests – using primers and other media.
228. As a response of the MOE to the recommendations, they requested the Team Leader of this TA to provide support in developing a National School Assessment Policy Framework. The framework will serve as the main document to guide the assessment system of the Republic. This request was approved by ADB and a three-day workshop was held on July 14-16, 2015 at the MOE attended by the Management Committee. This workshop resulted in drafting the Palau National Student Assessment Policy Framework and an institutional review of the Division of Research and Evaluation, the unit in-charge of assessment at the MOE of Palau. The draft of the Palau National Assessment Policy Framework is in Annex J.
TA 8066: REG - Quality Primary Education in the North Pacific 91 Funded by Asian Development Bank
IV. SUMMARY, CONCLUSIONS AND RECOMMENDATIONS
229. This section summarizes the major achievements of the project, conclusions and recommendations.
A. Summary of Achievements
230. The major achievements of the CDTA include the following:
i. Development and piloting of the EGLA Literacy and Numeracy to 8 pilot and 2 non-pilot schools in FSM and 5 pilot and 2 non-pilot schools in RMI. Two baseline surveys were conducted, one in February 2014 and another in February 2015.
ii. All the 13 pilot schools in both RMI and FSM included in the project were provided comprehensive professional development programs for teachers to implement individualized and group learning approaches to improve literacy and numeracy learning. A total of 823 students were assessed in February 2015 as part of the baseline pilot surveys of EGLA.
iii. Three major PD programs were implemented in November 2013, June 2014 and October 2014. A total of 521 teachers were trained – 164 in November 2013, 114 in June 2014 and 243 in October-November 2014.
iv. Reprinted over a hundred of literacy materials in the first languages (Marshallese, Kosraean, and Pohnpeian) and in English. These learning materials are now set up in reading corners of the pilot schools. Additionally, math kits and manipulatives were also distributed to all pilot school in the three project sites. These materials were distributed in June 2014 and November 2014 during the PD for teachers. With these materials, the pilot schools were able to set up reading and numeracy corners in their schools.
v. The EMIS system of RMI was reviewed and a proposal EMIS was submitted to RMI MOE that eventually sought support from US funding. A Workshop on EMIS was conducted to FSM EMIS Staff in Ponhpei in July 2014. The EMIS Staff of the four states of FSM attended the workshop.
vi. The EGLA Results were shared and presented in workshops and conference in:
° Pacific Early Age Readiness and Learning (PEARL) in Nadi, Fiji on May 4-6, 2015
° PACIFIC ISLANDS BILINGUAL BICULTURAL ASSOCIATION (PIBBA) in Kosrae, FSM on June 23-25, 2015
TA 8066: REG - Quality Primary Education in the North Pacific 92 Funded by Asian Development Bank
° Micronesia Teacher Education Conference (MTEC) in Ponhpei, FSM on July 21-24, 2015
° Pacific Educational Conference in Majuro, RMI on July 28-31, 2015.
vii. A review of the assessment system of Palau was completed and the Final Report was submitted in January 2015. A workshop to present the results of the review was held in October 2014, and follow-up capacity building program for MOE to draft the Palau National Student Assessment Policy Framework was conducted on July 14-16, 2015.
viii. The TA also supported RMI in drafting their Bilingual Education Policy. The policy was enacted by the Government of the Republic of the Marshall Islands.
ix. Prepared two knowledge products – a 12-page Informational Booklet on the QPENP and a three-minute video of the EGLA. The 12-page booklet was distributed during the conference and workshops held in PIBBA, MTEC and PEC.
B. Conclusions
231. The pilot project has shown very promising results in a very short timeframe. The motivation to continue this effort is evident from the feedback received from the participants. It has also become clear that, with specific and focused efforts that increase understanding of actual student learning while improving teaching practice, it is possible to effect significant positive impacts in learning outcomes.
232. The challenge is to systematize and institutionalize such efforts more broadly. It must be acknowledged that other contextual factors may have influenced the positive gains experienced within such a short period of time. Nevertheless, the momentum and commitment generated warrants the project’s further support, as it has shown new ways of approaching assessment and providing support to teachers and student learning.
233. Future efforts would ideally focus on more institutionalization of teacher professional development, including its linkages to pre-service education, while ensuring more bilingual and other targeted learning resources in this effort.
234. Sustainable change in teacher practice through deeper conceptual and pedagogical understanding takes time. Community and parent support will also be key to promoting more comprehensive, accountable, and transparent ways to improve the quality of education for the future young people in the North Pacific. By maintaining a strong focus on providing critical foundations in literacy and numeracy, it is hoped that many more young people will succeed throughout their education and in life.
TA 8066: REG - Quality Primary Education in the North Pacific 93 Funded by Asian Development Bank
C. Discussions of Issues and Recommendations
235. As with any project, there were a number of issues that arose over the life of QPENP which required attention from all major stakeholders. These included scheduling issues; local capacity; and communication. None of these issues, however, have had major negative impacts on the final outputs of the project and have been able to be resolved through dialogue between ADB, MOE/DOE and the consulting team. At all times, the focus has been on achieving project outcomes and solutions were found to ensure that these were met efficiently.
236. Issue: The implicit intention of the project is to provide capacity building to the MOE/DOE staff, particularly in developing the EGLA and designing and implementing professional development of teachers. The implementation of the EGLA, that is, conducting the assessment tool, called for a large number of MOE/DOE staff and the skills and experience required were not always readily available. The Consultants organized series of formal and informal learning sessions with the staff to ensure that will be able develop knowledge and skills to properly implement EGLA when the project ends. However, the more pressing is to expand the EGLA to all schools in a phased process. For instance, for Pohnpei and Majuro, they intend to expand EGLA beginning with 2 schools, and then 2 or 3 schools may added each year. There was a concern about the time and resources required therefore needs phasing.
237. Recommendation 1: That a follow up Technical Assistance be provided to scale up EGLA beyond pilot schools and project sites. All beneficiaries of this project strongly articulated the need for a following Technical Assistance to expand and scale the use of EGLA. It was suggested that cluster approach to implementation as random samples of students would be introduced as an approach in the scaling up of EGLA. A Technical Assistance, which may be provided by ADB and/or other development partners may be sought to ensure that the achievements and success gained form the project, particularly EGLA be sustained. Through this TA, EGLA will be further implemented to other schools beyond the pilot schools and control schools included in this project, as well as expanding it to other states in FSM and other atolls in Majuro. Furthermore, the TA would intend to enhance the capacity of MOE and NDOE to implement EGLA by institutionalizing EGLA into their assessment system, thereby requiring capacitating the assessment and data management units of the MOE and NDOE. The proposed EGLA TA would also include training of MOE/NDOE staff in monitoring and training of more assessors. Specifically, the following are the activities that may be implemented under the suggested follow-up TA:
i. Develop a plan to implement and monitor implementation of a possible second QPENP.
ii. Prepare an EGLA Training Manual for literacy and numeracy that addresses the need for standardized administration and scoring system.
iii. Develop a larger assessment item bank for literacy and numeracy in both languages – (i) items for training assessors, (ii) items to be used as practice
TA 8066: REG - Quality Primary Education in the North Pacific 94 Funded by Asian Development Bank
sets for students, (iii) items that can be made available to use as formative tools; and (iv) items that will be kept for Department/State-wide level use.
iv. Comprehensive training program for EGLA (or other assessment) data management and analysis, using available software packages in order to ensure that assessment running records of students are maintained, monitored to inform policy decisions related to curriculum, assessment and evaluation.
v. Develop learning resources, particularly to support the language education policy focused on writing to read and numeracy.
vi. Support and strengthen department/ministry leadership in literacy and numeracy development.
238. Issue: The design of the project was to provide professional development to support the literacy and numeracy learning. The project also triggered the formation of learning communities through engagement of school principals in all the pilot schools. The PD were designed and implemented with the principals being involved very heavily. However, the beneficiaries would like to see more principals and teachers collaborative activities, not only in EGLA implementation but also in designing and implementing professional development programs. They also want a TA to institutionalize professional development for curriculum officers, subject specialists, teachers, and principals and a support to develop a monitoring and evaluation of teachers’ performance and diagnostic tools for teachers with corresponding training program for implementation.
239. Recommendation 2: A support to enhance quality pedagogy through improved teacher training and education – both pre-service education and in-service teacher training. There is strong need to institutionalize quality pedagogy by supporting not only in-service teacher training programs (INSETT) in the Pacific countries, but also pre-service teacher education and training (PRESET). While the College of Micronesia (COM) in FSM and College of the Marshall Islands (CMI) have been providing pre-service education and training, there is a need to modernize the PRESETT programs focusing on the present issues surrounding the education system in the Pacific countries. Literacy and numeracy levels are still struggling compared to other regional neighbors, there is a need to develop a PRESETT and INSETT programs that blend and integrate literacy and numeracy in the core learning areas. Likewise, the emphasis on bilingual education is becoming a prominent issue in the school system, the need to train would teachers to teach in a bilingual learning environment and to provide learning resources that promote bilingual skills development – that is, becoming competent in their own language (such as Marshallese, Pohnpeian, or Kosraean) and likewise in the English language. Moreover, assessment is becoming an integral part of the curriculum development and implement, hence, the PRESETT and INSETT should also be developed in such as way that assessment skills of skills will be improved in order to support the EGLA implementation more effectively. Specifically, the institutionalization of quality pedagogy through a supported technical assistance should include the following activities:
TA 8066: REG - Quality Primary Education in the North Pacific 95 Funded by Asian Development Bank
i. Develop a policy on PRESETT and INSETT that supports the implementation of the six components of quality pedagogy addressing the needs from department staff to classroom teachers.
ii. Support both COM and CMI (and maybe Palau Community College) in reviewing their pre-service programs and provide support in terms of capacity building and organizational development of teacher training unit or department.
iii. Support the enhancement of the pre-service curriculum to include the teaching of bilingual students, how to teach L1 (Marshallese, Kosraean, Ponhpeian, or even Palauan for Palau) and teaching numeracy in a bilingual approach.
iv. Strengthen learning communities where ministry/department level staff work closely with school principals, and principals working closely with teachers as well as to engage community and family. The PD that includes the engagement of community and family will ensure that culture and language will be enhanced and preserved.
v. Strengthen a communication cycle/system between parents and teachers, and between principals and the department.
vi. Develop a long-term plan for professional learning for teachers, including career pathing, salary, and professionalization of teachers. This activity may lead to the development of a Teacher Development Policy Framework.
vii. Expand existing INSETT or PD programs being implemented by the ministry or departments of education in the project sites.
240. Issue. One of the components of the project was focused on EMIS and data management of students’ assessment records. However, while this was not given much emphasis just like the piloting of EGLA and PD programs. Most of the technical assistance given through this project was to help the MOE for RMI to strengthen their EMIS and for capacity building and training for EMIS staff for FSM. This was the result of the re-focusing of the project during the Inception phase. Hence, it strongly recommended that a further support on EMIS and management of student assessment data be provided.
241. Recommendation 3: Strengthen EMIS and management of student assessment data and other school records. There is an obvious clamor for a more extensive support to strengthen the EMIS and management of student assessment data. To the stakeholders of the project, they feel these two strategic programs would allow them to make better decisions and could inform better policy programming. Hence, is suggested that a follow through of the EMIS support be provided with the following proposed activities:
i. Further review of the EMIS of FSM, particularly Kosrae and Pohnpei States.
TA 8066: REG - Quality Primary Education in the North Pacific 96 Funded by Asian Development Bank
ii. Develop a capacity building of state level staff on EMIS and data management of student assessment records. A short-term overseas training and study tour may be a good program for key staff of state DOE and MOE for RMI. If possible, support the development of EMIS Specialists in each state to ensure that data management will be handled properly.
iii. Provide a comprehensive training of NDOE staff to manage student assessment data, including training on scoring, analyzing and reporting of EGLA data and results. A better way of presenting EGLA results may identified and NDOE will staff will be trained accordingly.
iv. A school level training should also be designed to train teachers to analyze classroom assessment data such as student profiling, class profiling and school profiling. This way, even at school level, the core standards may be determined using students assessment data and other information.
v. As originally conceptualized, it may be good to enhance the system for providing student unique identification numbers or Student Education Numbers or ID that they use from Kindergarten to Grade 12.
242. Issue. While the project has provided learning resources by re-printing out-of-print learning resources developed under previous projects and other organizations, there is an evident need to develop a more appropriate resources for literacy and numeracy that support the bilingual nature of the curriculum being implemented in the Pacific countries covered in this project. Moreover, the states and countries included in this project have been strengthening their bilingual education policy, that is, to institutionalize the development of competencies in both the national language/ vernacular and English language. Under this project, the RMI has been supported in the finalization of bilingual education policy.
243. Recommendation 4: Develop bilingual literacy materials and activities. There is a dearth of learning resource in the local language. While the project had re-printed materials in local language, these are not enough resource to support the literacy development in the local language. The teachers of Sansrik Elementary School in Kosrae, for example, have develop readers in Kosraean language and this could be further enhanced. Related to this, is the need to institutionalize bilingual education through a development of bilingual education policy. Hence, the following activities under this recommendations are given as follows:
i. Train and encourage teachers to develop readers and numeracy learning materials in the local languages. The outputs of the teachers, just like those from Sansrik Elementary School, can be enhanced and published through a support of a TA grant or through the federal support funds.
ii. Support each state to develop a bilingual education policy and provide technical and financial assistance in implementing the policy.
TA 8066: REG - Quality Primary Education in the North Pacific 97 Funded by Asian Development Bank
244. Issue: Finally, the review of the assessment system of Palau has resulted in initially supporting the MOE of Palau to conduct an organizational diagnosis of the assessment and research unit – Division of Research and Evaluation (DRE). Also, a workshop was conducted to support the MOE to develop its National School Assessment Policy Framework. However, there is still a strong to support Palau in their assessment system to further improve and streamline their examination and assessment systems. Specifically, the following activities may be implemented through another TA:
i. Support in the finalization of the National School Assessment Policy Framework of Palau as well as its implementation, particularly in the streamlining of examination and assessment activities.
ii. Provide capacity building to the DRE who will eventually spearhead the implementation of the assessment policy framework.
iii. Provide capacity building to MOE Subject Specialists, Assessment Staff, Principals, and teachers in assessment – from state-wide assessment to classroom level assessment.
iv. Further support the MOE in upgrading its examination materials aligned with their curriculum vis-à-vis the national school assessment policy framework.
TA 8066: REG - Quality Primary Education in the North Pacific 98 Funded by Asian Development Bank