How Do We Use Educational Evaluation to Improve?

Home , Tzotzil

NATIONAL EDUCATIONAL EVALUATION POLICY GAZETTE IN MEXICO Traveling, learning along the way, and arriving Year 2. No. 4 / March - june 2016 www.inee.com.mx Published each four months with the aim of fostering the dialogue of the National Educational Evaluation System How do we use educational evaluation to improve?

Voices from the inee The Mexican Teresa Bracho González experience Jorge Antonio Hernández Uralde Sonora Francisco Miranda López Zacatecas Agustín Caso Raphael Querétaro

Analyses and proposals Guillermo M. Cejudo International panorama Gabriela Pérez Yarahuán Arlo Kempf-Canada Francisco Abarca Guzmán Peter McLaren-United States

Special supplement to the Mexican Educational With texts in Amuzgo of Guerrero Evaluation Policy and Tzotzil of Chiapas.

The logbook National Educational Evaluation Policy Gazette in Mexico Year 2. No. 4 / March-June 2016

IN OUR OWN HAND OUR VOICE IN THE CLASSROOM

Evaluating to improve: Effective use for improvement; Can the project for certifying is it possible? what can we do to make it educational quality via iso 2 Members of the Board 20 happen? 40 9001-2000 improve education in of Governors of the inee Teresa Bracho González Zacatecas? Adolfo Castruita Monreal

FROM THE DESK NAUTICAL LETTER DOSSIER: USE OF INFORMATION, TWO PROPOSALS Second year: A National Educational To build the narrative Evaluation Policy for Promoting OTHER PERSPECTIVES 4 of evaluation 23 Improvement Gazette No. 4. How to foster the VOICES FROM THE CONFERENCE ROADMAP use of evaluations? 42 Guillermo M. Cejudo The Dangers of Educational The meaning and use, for the Francisco Abarca Guzmán Extremism: High-Stakes improvement of Education, 5 Standardized Evaluation in US 26 of the results of the Evaluation The Use Radar K-12 Education of Basic Teaching-Learning Gabriela Pérez Yarahuán Peter McLaren Conditions (ecea) and the 46 Arlo Kempf Evaluation of Learning Outcomes in Elementary Education (Planea) Education-policy decisions: the Jorge Antonio Hernández Uralde link between evidence and values Andrés Eduardo Sánchez Moguel Visit the Gazette digital version in the 11 Francisco Miranda López Raquel Ahuja Sánchez inee web at: www.inee.edu.mx.

FACING OUR CHALLENGES Theinee guidelines: building bridges between evaluation, Challenges, aims and work: 31 use and educational improvement decisions and educational Arcelia Martínez Bordón 14 evaluation in Querétaro José Alfredo Botello Montes 2016 Guideline-issuance Program An overview of education and 34 evaluation in Sonora: how to Information systems on a par 17 overcome the challenges with the Educational Reform: Ernesto De Lucas Hopkins 37 innovation for strategic data management Agustín Caso Raphael Arturo Cervantes Trejo Cecilia Galas Taboada

inee National Educational Evaluation Policy Gazette in Mexico, Year 2, No. 4, http://www.inee.edu.mx march-june 2016, is a publication of the National Institute for Education- Twitter al Evaluation by the Educational Policy and Regulations Unit, under the @ineemx care of the General Direction of Guidelines for the Educational Improve- inee Facebook ment. Barranca del Muerto No. 341, Col. San José Insurgentes, Del. Benito http://goo.gl/axitPa Juárez, C.P. 03900, México, D. F. Tel. (55) 62.70.66.00. inee Youtube http://goo.gl/fHRDvC Coments: Laura Athié, Responsible editor ([email protected] / [email protected]). 2

ENGLISH

IN OUR OWN HAND

Evaluating to improve: to drive improvement; if its results just sit that decision-makers and others involved is it possible? on the shelf, their likelihood of informing in education access the said findings and decision-making will decrease, and hence use them to make improvements; (iv) how they must be disseminated as necessary, via the results of the tests pertaining to the Na- the right means and to the right and people, tional Plan for the Evaluation of Learning and suitably used for the purposes that that (Spanish acronym: Planea) and the Evalua- Members of the Board their nature and technical features equip tion of Basic Teaching and Learning Condi- of Governors of the National them to be used, based on awareness of tions (Spanish acronym: ecea) can be used Institute of Educational Evaluation their consequences and certainty that they to plan improvement paths in schools and (spanish acronym: inee) will serve to foster the improvements in school districts; (v) what needs to be done to education that the National Institute for the foster greater use of the evaluations carried Evaluation (Spanish acronym: inee) wants out by the inee for planning purposes; (vi) to achieve in the context of the Educational whether the design of the evaluations needs In order for the link between evaluation Reform in Mexico. to be changed; (vii) whether the evaluation results and educational improvement to We at the inee are convinced that evalu- findings need to be disseminated in a differ- occur, it is necessary that both education- ation findings constitute essential input for ent way; (viii) who we need to liaise with to authority decision-makers at all levels and the creation of the National Educational foster greater use of the said findings; (ix) also teachers develop the competencies Policy, and that for the latter to be optimally what features an information system needs formulated, the voices of all those involved to possess to ensure that it is optimally used that will enable them to understand the in education, both nationwide and at the by decision-makers; and (x) what propos- contents and scope of the said results local level, must be heard if we are to get a als the model for the creation of guidelines […], that their value and importance be clear idea both of the realities and problems needs to include in order to improve educa- acknowledged, that they be considered kind of the education system, and also of the so- tion. of accountability, and that the technical lutions needed by a diverse, multicultural Based on the aforesaid questions, in this expertise of the evaluating bodies and country such as ours. Since evaluation is a edition of the Gazette we include some lo- the professional role that they play be mere means, rather than an end in itself, we cal viewpoints from the states of Querétaro, recognized. must not only create good evaluations, but Sonora and Zacatecas, along with two pro- Governing Document also use them, and therefore the inee’s main posals by Mexican specialists, as to ways in the National Policy for aim is to ensure that evaluation leads to im- which evaluation findings can be used. We the Evaluation of Education provement – i.e. that it informs decision- also include an article by Peter McLaren and making aimed at making education more Arlo Kempf, two of the most influential the- equitable and raising its quality. orists in the field of critical pedagogy, who ducation is a human right and plays a With the above aim in mind, in the talk about the limits of standardized evalua- decisive role in developing people’s abil- fourth edition of the Gazette of the Nation- tion in the light of the experience with such Eities, skills and capacity to enjoy life,1 and, al Evaluation Policy in Mexico, we seek to evaluation in Canada and the usa. for its part, educational evaluation is a tool ascertain: (i) what the uses of the evalua- The Gazette also includes contributions designed to improve education and thus to tion that is being produced by the National by the heads of various inee units, by Te- make good on the right to universal high- Evaluation System (Spanish acronym: snee) resa Bracho, a member of the inee’s Board quality education. -primarily by the inee, but also by the edu- of Governors, and by other inee officials But despite all the good intentions and cation authorities- are; (ii) what the biggest who, with regard to the necessary connec- high expectations, the truth is that there is achievements of -and challenges facing- the tion between evidence and fundamental no automatic link between evaluation and said evaluation are; (iii) whether the said values, describe the progress achieved in improvement. Such a connection may fail evaluation is indeed used, and, if it is, by our own evaluation system, make proposals to occur when evaluation is not pertinent or whom and how; (iv) what we can do to fos- for the use and dissemination of evaluation relevant, or not properly focused, yielding ter more use; and (v) if the said evaluation findings, and present an analysis of the Pla- results that are not germane, or when it is serves to improve education. nea and ecea results. These reflections are not timely or it takes so long to give feedback More precisely, this edition of the Ga- accompanied by a text that talks about the on its results that the crucial opportunity for zette sets out to find out (i) what types of creation of guidelines that can foster the use them to have an impact is lost. Also, if there evaluation findings tend to be used for of evaluation to improve education via the is not adequate communication about an purposes of policy and program design; (ii) influencing of education policy. evaluation, resistance to it might arise. what the biggest challenges are regarding This year’s first edition of theGazette In all cases, the main thing is that people the dissemination and use of the said find- also includes a section devoted to the Na- must know about, and use, evaluation if it is ings; (iii) what needs to be done to ensure tional Educational Evaluation Policy (Span- 3

National Educational Evaluation Policy Gazette in Mexico

ish acronym: pnee) that will continue to appear in all the subsequent editions, since the latter is the benchmark for the development of evaluation aimed at educational improvement that the inee wants to promote. Given the importance of this topic, we are also including a special supplement devoted to it, developed by all the members of the inee’s Board of Governors, two experts from unesco’s International Institute for Educa- tional Planning in Buenos Aires, and the Da- vid Rockefeller Center for Latin American Studies, who talk about the achievements of the pnee and the challenges it faces, based on an analysis of its recently published Gov- erning Document. It should not be forgotten that 2016 is a crucial year for education in Mexico. We are exactly halfway through a 6-year presi- dential term of office marked by an educational reform that gave rise to ambitious evaluation projects aimed at professionalizing teachers and also set up the National Educational Evaluation System (Spanish acronym: snee), which, possibly the only system of its kind in the world, is outstanding for vigorously promoting the development of measurement and evaluations, not only of learning outcomes, but also of various components, processes and results, as part of a systematic effort to link evaluation to different regulatory and technical endeavors in order both to ensure that it is transparent, equitable, fair and objective and also to foster cooperation at the institutional level among the different education authorities and promote strategies for influencing education policies and programs, improving education services, raising the latter’s quality, co’s primary schools via the findings ofecea . be overcome to ensure its use, and its po- and achieving more equitable results. At the end of the same year, based on the re- tential for informing educational policy and In 2015, important changes were achieved sults of the competitive examinations for en- achieving improvement. in the evaluation of student achievement and try to the teaching profession, the inee issued We would like the debate initiated in learning outcomes via Planea, which was its first guidelines aimed at improving initial this 4th edition of the Gazette to be ongoing implemented for the first time in the final elementary-level teacher training. and thought-provoking while also opening years of primary, lower-secondary and up- Thus it bears asking whether all the work the door to input resulting in new ideas and per-secondary education. We carried out a described above will have an impact on the practices that consolidate the new wave of Consulta Previa, Libre e Informada a Pueblos formulation of educational policy, when this improvement-focused evaluation that is un- Indígenas sobre Evaluación Educativa (“Pre- will happen, and what needs to be done to derway in our country. liminary, Free, Informed Consultation of the ensure it. Indigenous Peoples”) about the education Our guest contributors to this edition 1 See: Amartya Sen. (1999). Development as Free- received by the indigenous peoples and the of the Gazette reflect not only about edu- dom. www.uio.no/studier/emner/matnat/ifi/ evaluations carried out in their schools, as cational evaluation per se, but also about well as ascertaining the conditions in Mexi- its achievements, the challenges that must 4

ENGLISH

FROM THE DESK

Second year: opens people’s eyes and gives us a chance es for readers’ comments and forums for To build the narrative to tell out stories after listening to those of exchanging ideas and opinions: http:// other people. www.inee.edu.mx. of evaluation Beyond numbers there are words, and, 4. We’re looking for information as a ba- beyond words, people – the teacher who sis for making decisions and would like tells us that he does not feel he has been rep- to know what responses are received to resented in the Gazette, the education min- the questions posed in each edition. ister who talks about his plan, the specialist Since people need input in order to make he art of recounting events, experienc- who, in these pages, proposes an evaluation decisions, twice a year we will be includes, tragedies, the enormous problems model aimed at solving the big problems that ing a special supplement like the one to Twe face and how we deal with reality is as old the National Educational Evaluation System this 5th edition, which is entitled The Na- as the world and we practice it via the print- has identified in Mexico’s education system tional Educational Evaluation System ed page, emails, blogs, video clips, books, and seeks to solve. Each author-protagonist and the National Educational Evaluation reports with graphs and statistics and docu- reads and recounts, thus contributing to a Policy: Progress and Viewpoints. mentaries, telling stories that give rise to shared account of the endeavor to imple- 5. We decision-makers need to have a say; comments, bring together shared attitudes, ment a countrywide policy….and, in their there are a lot of us, in different spheres answer questions and clear up doubts, doing emails and responses to our surveys, the of action. In response to this comment, so by means of words that combine to create readers of the said account have expressed the Gazette is publishing the following in- a narrative, captivate our audience, make it the following needs: vitation and will be welcoming your input remember and, at the end of the day, leave so as to ensure that all those people who it with a new experience, some queries, or 1. We’d like to have our concerns ad- work in schools, research centers, insti- something to reflect on. dressed. In response to this request, as tutes and supervision and evaluation of- Today this way of communicating things of this year, the National-Education-Pol- fices are also represented. has been -and is still- known as storytelling, icy Gazette will be including, in its “Our which, as explained by Mo Yan, the win- Readers” section, a dialogue with those Specifications for contributions: In digi- ner of the 2012 Nobel Prize for Literature, who seek clarification. tal form (Microsoft Word file), double spac- whose name in Chinese means “don’t talk”, 2. We need to get a better understanding ing between lines. Arial-bold 12-point font. simply means letting the rest of the world of the National Educational Evalua- Maximum size: 12 single-sided a5 sheets, to know about things: tion Policy. In answer to this comment, be sent before the closing date to: lathie@ in this 4th edition of the Gazette and in inee.edu.mx. All contributions will be re- I know that nebulous terrain exists in the subsequent ones, we will be including a viewed by the Editorial Board. hearts and minds of every person, terrain policy report, with graphs, about the de- By working together to write this ac- that cannot be adequately characterized in cisions made -and the projects carried count of evaluation, which reaches 22 coun- simple terms of right and wrong or good out- in relation to the seven main compo- tries and thousands of readers, we are, in the and bad, and this vast territory is where a nents of the National Educational Evalu- words of Mo Yan, “reading about society and writer gives free rein to his talent. So long ation Policy. life, which, together, constitute a big, invis- as the work correctly and vividly describes 3. We’d like to share our opinions and ible book” where, in just a few pages, people this nebulous, massively contradictory ter- suggestions. In response to this concern, can understand the whys and wherefores. rain, it will inevitably transcend politics the Gazette is becoming interactive and “What I did,” says Yan, “was to tell my stories and be endowed with literary excellence.1 will be including two capsules featuring in my own way”. video interviews with specialists on its The same is true of theGazette, which microsite in each edition, as well as spac- 1 Mo Yan, Sorytellers. Nobel Speech. December enters its second year of publication today, 7th, 2012. © Nobel foundation 2012. https:// resolved not to be silenced, but rather to go goo.gl/d0GiZC on telling the story, to find that grey area in If you want to contribute to the Gazette, please take note of the following information: order, on each of its pages, to tell the story of evaluation, to talk about the educational Edition Topics Closing date reform that began in 2013, and to publish Context, diversity and educational the articles of its contributors who, also part Gazette No. 5 evaluation: towards fair, inclusive May 15th, 2016 of the reform, have expressed their opinions (July-October, 2016) education? about it in the course of five editions. Gazette No. 6 This way of explaining complex ideas in Evaluation and local capacities: (November 2016- August 15th, 2016 simple words is, precisely, a form of story- can federalism be rebuilt? telling – the transmission of a message that February 2017) 5

National Educational Evaluation Policy Gazette in Mexico

VOICES FROM THE CONFERENCE and other education stakeholders in the US. THE SPECIAL GUEST While recent education reforms, including the Every Student Succeeds Act (essa), The Dangers of may be a move away from one-size-fits-all Educational Extremism: approaches to understanding student success, there remains a complex web of local, High-Stakes state, and federal tests and testing mandates. Further, the data driven culture of so-called Standardized Evaluation accountability persists throughout US edu- in US K-12 Education cation, with students, teachers, adminis- trators, schools, and districts increasingly understood through the narrow prism of “We write this with humility, as testing results. a modest missive to education Recognizing the diversity of educational stakeholders in diverse Latin realities across Latin America at the national, American contexts […]. These are subnational, and intranational levels, this challenging times for policy makers, article offers a humble and general caution and the question of how we use for Latin American policy makers and stakeholders, with reference to the hazards of the evaluation and assessment in our overuse of standardized assessment. The first schools is fundamental to our vision section below introduces the current context and philosophy of education”, say of high-stakes testing in the US, with a look at Peter McLaren, the father of the the quantity, frequency, and duration of test- Critical Pedagogy, and Arlo Kempt, ing. The next section situatesUS K-12 stan- teaching and learning expert, in dardized testing practices in international this analysis about what they call perspective. The third section looks at the the Global Educational Reform politics of testing as well as the surrounding Movement. myths and contested rationales underpin- ning high-stakes testing such as data driven accountability. The article then turns to Peter McLaren consider implications for education in Latin College of Educational Studies. Chapman America. A short conclusion follows. University University Of California at Los Angeles Current Context of US High-stakes (ucla) Standardized Testing in US K-12 [email protected] Classrooms In the US, innumerable and wide-ranging Arlo Kempf standardized tests (ST) are used to measure Coordinator of Secondary Education, the learning, understanding, and aptitude Master of Teaching Program of children. Some are diagnostic, indicating Department of Curriculum, Teaching and they are geared to assess students’ knowl- Learning, oise/University of Toronto edge and ability in order to inform subse- [email protected] quent teaching activities and foci. These may not accurately be called high-stakes, as they generally do not impact whether students Introduction advance or not, and student performance The past fifteen years have seen a radical on these assessments is unlikely to impact overhaul of education in the United States. school funding, school reputation, or teach- Among the most significant elements of this er salary. We can thus set these aside in this transformation has been the expansion and discussion, and focus exclusively on ST that acceleration of the use of high-stakes stan- is not diagnostic, and on those tests which dardized testing. Despite major concerns are used across US districts and/or states. about accuracy, student well-being, and so- In the US, the quantity of testing is dif- cial and economic inclusion from various ferent from state to state and from district stakeholders the use of testing is currently to district. Many states administer standard- at an all-time high, and correlates to a host ized tests in the first and second grades, how- of negative consequences for K-12 students ever, many begin ST as early as kindergarten 6

ENGLISH

(roughly ages 3-5) and continue all the way US K-12 Evaluation in International from lower to upper secondary school), se- through to high school graduation (roughly Comparative Perspective lection into different programmes (e.g. into ages 16-18). In addition to state- and district- The United States is an international outlier vocational or academic programmes), or se- mandated testing, the Common Core State not only as far as the quantity of standard- lection into university programmes.”8 While Standards Initiative (ccssi) sets out guide- ized testing used, but also as far as the de- all of the 65 countries have at least some lines for curriculum, delivery, and assessment gree to which standardized tests are used to assessment and/or evaluation at the lower (including st) in states that have adopted the determine educational success (on numer- and/or upper secondary levels, there is sig- Common Core (a strong majority as of Janu- ous levels). Students in the US write more nificant variation, and the United States is ary 2016). In December of 2015, US President standardized tests more often than most among a very small number of countries that Barack Obama signed the Every Student Suc- students around the world. These standard- uses national/state and additional examina- ceeds Act (essa) which is an overhaul of the ized evaluations are used not just to under- tions at both the lower and upper secondary previous No Child Left Behind Act (nclb). stand student success and ability but also levels. Although the US uses relatively few While it is possible that ccssi and essa may to evaluate teachers, schools, districts, and high-stakes entrance exams for qualification ultimately reduce the quantity of standard- states under the banner of data-driven ac- for secondary and postsecondary opportu- ized testing, it is clear that current use of these countability. With the United States at one nities, the intensity of testing beginning as high stales assessment is unprecedented in end of the spectrum as far as testing, Finland early pre-K is extraordinary and interna- US history. A study by the American Center is at the other. The Scandinavian nation uses tionally unparalleled. for Progress (2014) found that students were only one standardized test, at the end of sec- writing up to 20 standardized tests per year, ondary school. Sixteen-year-old Britons, by The Myths of Accountability, with an average of ten per year in grades three contrast, write a series of high-stakes tests Impartiality, and Equity: The through 12.1 These figures support research (approximately 15–20), which determines Challenges and Rationale for conducted by the American Federation of their path forward in secondary school and Standardized Testing in the US Teachers.2 Additionally, Kempf’s (2016) study thereafter in potential opportunities to con- Among the most common justifications for of teachers in Chicago found teachers admin- tinue to college or university. the use of standardized testing in the US are ister an average of 11 standardized tests per The Program for International Student that st offers an impartial measure of stu- year, while Education Week suggests that in Assessment (pisa) administered by the Or- dents’ accomplishments, skills, and under- many urban schooling districts, students are ganization for Economic Co-operation and standings in a given subject area or areas; writing an average of 113 standardized tests Development (oecd) offers educational that these impartial methods provide an between pre-k and twelfth grade.3 Although measures of 65 countries around the world, equitable educational opportunity for low there are not many US jurisdictions in which and provides a broad international perspec- income and racial minority students (mod- students write fewer than five standardized tive on testing usage. One measure relevant erating the educational impacts of race and tests per year, the research mentioned above for understanding testing in comparative income); and that such measures provide indicates there are many in which students perspective is pisa’s look at the use of exam- accountability to taxpayers as far as a return write up to 20 or more on a yearly basis. inations at the lower and upper secondary on taxpayer dollars. It is worth discussing Generally speaking, quantitative in- levels. According to the oecd, lower sec- each of these pieces separately here before creases in the use of standardized tests cor- ondary generally refers to schooling for stu- moving on. respond to quantitative increases in class- dents aged 10 to 16. It begins between “the room time allocated for test preparation. A ages of 10 and 13 and ends between the ages Impartiality 2014 New York Times article described the of 13 and 16. It is compulsory across oecd The most noticeable barrier to impartial- situation in Florida where “many schools and partner countries. It marks the end of ity, or objectivity, is inaccuracy. In the US, this year will dedicate on average 60 to 80 compulsory education in a number of coun- countless incidents of cheating have raised days out of the 180-day school year to stan- tries.”6 Upper secondary is typically two to serious questions about the validity of the dardized testing (st). In a few districts, tests three years, with students generally entering use of standardized testing results. Cheating, were scheduled to be given every day to at at 15 or 16 years of age. however, is not the biggest issue as far as ob- least some students.”4 Former US Assistant While pisa does not offer data on st at jectivity and impartiality and st. In order for Secretary of Education Diane Ravitch sug- the elementary level (what the oecd calls standardized tests to be objective they have gested that in late 2013, many districts were the pre-primary and primary levels) findings to be clear (what they claim to measure must allocating “20 percent of the school year” to on the use of st for older students are im- be what they are designed to measure, and as test preparation.” The American Federation portant. pisa differentiates between assess- well correspond exactly to what they do mea- of Teachers study found that each year, ap- ments and examinations. Assessments “take sure); they must be accurate (based exclu- proximately 20 days of instructional time are stock of students’ performance in order to sively on classroom and school learning, with used in test taking, administrative activities make decisions on future instruction or to no interference); and they must be reason- and test preparation.5 It is safe to say that on summarise performance for information ably value-free, safeguarding that culture and the low end in the United States, ten percent purposes” and “do not have direct tangible income do not privilege or handicap any stu- of class time is used for activities related to consequences for students.”7 Alternately, ex- dents in preparing for or succeeding on the st, with a number of jurisdictions devoting aminations “determine students’ pathway to test (nor any teacher in preparing students 20 percent or more. upper levels of education (e.g. the transition for it). To the question of clarity (whether 7

National Educational Evaluation Policy Gazette in Mexico

standardized tests measure what they claim can also transform social difference into ac- to measure, well-known US critic of ST Alfie ademic advantage and disadvantage. When Kohn argues that “[n]orm-referenced tests test results are used to judge schools, dis- were never intended to measure the quality tricts, and increasingly teachers for things of learning or teaching” and suggests “the over which they may have limited or no con- main objective of these tests is to rank, not to trol, the dangerous role of testing becomes rate; to spread out the scores, not to gauge the clear. Students, teachers, schools, and dis- quality of a given student or school.”9 Norm- tricts are understood in either a positive or referenced testing in schools compares one negative light based, sometimes exclusively, student’s performance to that of all others on ST scores. Successful results often corre- taking the same test. This is different from late to higher incomes among parents, while evaluating what a student has learned or by related contrast, low test scores correlate what a student knows, and additionally tells to higher rates of poverty.15 us nothing about changes in an individual The Trends in International Mathemat- student’s academic progress.10 ics and Science Study (timss), released in Numerous academic researchers have 2011, reported wide differences between argued that st frequently fails to meet suffi- the reading and math performance of eighth cient standards of both reliability and valid- grade white, African American, and Latino ity.11 Put simply, we cannot ensure that ST students.16 Additionally, there had been no data are reliable as far as understanding the change in the gap between the performance academic ability of individual children. CRe- of 17-year-old African American and white ATE, a group of researchers from Chicago, students on the National Assessment of explains the difference between reliability Educational Progress in both reading and and validity using the metaphor of two bro- math since 2004.17 Using graduation rates ken scales. One gives you a different weight as a measure of school success, there was a each time you step on it (unreliable) and the 20 percent decline among students in high- other is always ten pounds over (it is reliable poverty schools, while rates for students in but invalid).12 American academic William low-poverty schools remained the same.18 Popham suggests that validity is the most important component of any evaluation.13 Accountability This is a criterion, however, that standard- As Kempf has argued elsewhere19, the driv- ized testing has not met in the opinion of ing rhetorical force in the push for ST is many scholars (of different political persua- the idea of accountability. It is a compelling sions) for the last 25 years. discourse. Who would disagree that teachers should be held to high standards, that The Myth of a Level Playing Field students should be well educated and cared Kempf (2016) notes that a great deal of stan- for, that money from taxpayers should be dardized evaluation unavoidably assesses carefully spent, that public servants with in- what students have learned out of school. adequate performance should be forced to For example, the third grade student who improve or lose their job, or that we should spends her Sundays in a private chess camp have clear tools for understanding how well will come to school with an advantage over our education systems are working? These a classmate who spent his Sunday in Bible common sense arguments make up simple school. Each will have been gaining knowl- and seductive justifications for standardiza- edge but one is learning ideas, language, and tion, however, there is no proof st (or stan- analytical processes that are validated and dardization of education more generally) recognized as legitimate in-school knowl- accomplishes any of these objectives. Exces- edge. As far as testing, the issue here is not sive use of standardized testing can profes- which weekend learning is better but in- sionally disadvantage teachers and adminis- stead that one student is coming to school trators who work with student populations with a tool for success, a tool to which most with low test results. Gauging student and students have no access. Our use of stan- teacher ability through st can lead some dardized tests may then equate this advan- teachers to prefer teaching only those stu- tage with intelligence.14 dent populations who are likely to be high Standardized testing thus not only fails performing according to test results, and in- to account for economic, cultural, and other deed push them away from serving students differences within student populations, it who are not. 8

ENGLISH

After over fifteen years of increased use from the constructive, experiential, and so- dents from poor families already experience of high-stakes standardized testing, we are cial elements of learning. Internationally, unhealthy levels of stress hormones, which no closer ensuring accountability to the the United States is a radical standardized can impair their neural development at a public. Further, as Kempf has argued20 ST testing outlier, testing more and more often very early age.32 What are the implications has failed to produce gains such as better than anywhere else. Although the US uses of high-stakes, standardized tests in a digi- support of students with diverse learning fewer make-or-break examinations (such as talized world where school administrations needs, more inclusive curriculum, and in- middle school or high school entrance ex- attempt to increase the ram and gigabytes in creased teacher access and application of ams) than some other places, the intensity of classroom computers rather than find ways research into teaching and learning. Indeed, standardized testing (beginning as early as to increase the shrinking attention spans of standardization has largely hindered rather Pre-K) sets it apart from many other educa- students that result from intense computer than promoted these improvements. There tion cultures. Most teachers assess students use as well as other factors? Economic in- is a painful gap, then, between blanket calls very frequently, and the vast majority of equality can affect early childhood develop- for accountability and educational improve- that assessment would make no sense ex- ment in many areas including brain development. Whereas there is no doubt that edu- pressed as simply as a number. Often lost in ment, educational achievement, and mental cation in the United States faces many prob- the standardized classroom is the context, health. The holistic burden should thus not lems, there is no proof that standardization the fecundity, and the unique challenges, be placed solely on the shoulders of teachers, will fix any of them. strengths, and opportunities which that are since it is impossible for even the best teach- From a wider international perspective, taken into consideration in the course of ers to overcome these obstacles facing their Finnish educationalist Pasi Sahlberg sug- rich and useful assessment and evaluations. students despite the excellence of their in- gests we are now witnessing a “Global Edu- That education is a political activity coeval structional programs and the high quality of cational Reform Movement” (germ) and with economic life is by now an accepted their pedagogical skills. The struggle for edu- argues systems that have experienced in- commonplace in the nomenclature of educational reform is the struggle for economic creased standardization (specifically a move cational theorists. Given the grim reality of justice. There is no other way to put it. toward test-based accountability, standard- persistent poverty in the United States—and ization, school choice, and human capital the hard truth that a large segment of the Considerations for Latin America model thinking) have seen a corollary de- new jobs being delivered by the transna- The greatest impediment to educational cline in academic performance.21 Citing 12 tional capitalist economy pays little more success and prosperity is not a lack of good years of oecd/pisa data from 2000 to 2012, than poverty-level wages—even if poor teaching or inadequate testing and account- Sahlberg traces the decline in mathemat- parents can raise their children like middle- ability but rather economic and social in- ics learning outcomes in the United States, class parents, their chances of escaping pov- equality. This is global truth with particular England, Canada, Australia, New Zealand, erty through education is far from assured. resonance in Latin America. If this is the case, the Netherlands, and Sweden—all jurisdic- Seventy-four percent of the entering class at then following in the footsteps of the United tions, which have embraced the germ ap- top-tier colleges and universities come from States as far as high-stakes testing will likely proach, to albeit different degrees.22 In the the ranks of the economically well-off; that worsen national and subnational issues rather United States in particular, over 25 percent is, from the top quartile of families by in- than help to solve problems facing education of students fail to meet oecd/pisa’s base- come group.29 Most of the economic growth stakeholders in Latin America. The solution line for math proficiency, and nationally the in the U.S. is benefiting the rich, and low- to educational reform and the quality of social United States ranks 27th among oecd coun- wage jobs are becoming a permanent part life in general in the US relates powerfully to tries.23 While standardization does not cor- of the U.S. economy. Already, one in four economic rights—indeed to rights that are not relate with academic improvement, oecd/ Americans has a job that pays poverty-level always attached to educational reform. This pisa data on social and economic equal- wages. Indeed, fighting for the right to have is equally true of countries throughout Latin ity tells us that improvements in social and a decent and remunerative job would do America. Education reform in democratic economic equality correlate with improved more for educational reform than current contexts must ultimately enhance educational performance on oecd/pisa evaluations,24 a policies involving testing.30 opportunities for the popular majorities—it conclusion supported as well by the work of What are the moral and educational must recast society through the promise of Wilkinson and Pickett.25 The United States, implications surrounding the use of multi- political democracy by extending educational which has the same levels of inequality to- ple-choice tests that rely on memorization democracy into the realm of the social and day as it did in the 1790s,26 also lags in its and test-taking skills in our contemporary economic through a political process of pop- educational performance. world; a world where electronic and digital ular self-emancipation. While, short of this Internationally renowned educational- media shape our neural networks, mediate lofty goal, there may be many efforts that can ist Andy Hargreaves muses that: “account- our perceptual emotional networks, and re- be made to improve education, administering ability is what remains when responsibility organize our senses and neuropsychological high-stakes testing is not among them. has been subtracted.”27 As Kempf has sug- structures as well as speed up our “orienting Testing is clearly linked to the global gested28, the push for accountability de- response” regardless of content—all which marketplace, educating for specific forms of professionalizes teachers’ work; establishes combine to create attentional and memory labor power that will serve the interests of competition at the center of our education difficulties later on in school?31 Compound the transnational capitalist economy. How- system; and alienates students and parents this dilemma with the fact that many stu- ever, critics such as John Marsh (2011) make 9

National Educational Evaluation Policy Gazette in Mexico

the case that education should be treated education stakeholders in diverse Latin Amer- as a political—not a market—phenomenon ican contexts. We recognize the US indeed has and we believe that there is merit to this ar- a great deal to learn from the tremendous suc- gument. Marsh notes that the US does not cesses of various Latin American educational generate many more poor people than other jurisdictions, as well as the concomitant chal- countries. European countries achieve low- lenges faced by policy makers in these regions. er poverty rates because they provide more These are challenging times for policy makers, social programs aimed at the poor and un- and the question of how we use evaluation employed. Without government programs, and assessment in our schools is fundamental Sweden would have 26.7 percent of its popu- to our vision and philosophy of education. The lation living in poverty. With a strong social questions surrounding testing are never su- safety net, however, including policy con- perficial since they are linked to more pressing centrations on education, the poverty rate philosophical questions that probe deeply into is 5.3 percent.33 To be sure, education helps the very nature of knowledge. For instance, some people enter the labor market, and in- how can we construct viable explanatory sys- directly might create a few more jobs, but tems of classification, interpretation, and eval- what we need is direct job creation, higher uation from the underlying phenomenological wages, and better redistribution programs. descriptions of lived experiences of students, Marsh notes, citing various research- whether through logical inferences or critical/ ers, that among children whose parents have dialectical theories? This is a major task that identical levels of education, those children confronts not only test-makers but education who lived in unequal countries performed stakeholders at all levels. We must not simply worse on tests of adult literacy. Children of ally ourselves with compatible ideological in- parents with college degrees in general per- terpretations about testing but must be willing form the same, whether they live in Finland to challenge all of our fraudulent assumptions. (one of the most economically equitable We must risk new understandings that run countries) or the US (one of the most eco- contrary to our entrenched opinions. The pol- nomically inequitable countries within the itics of high-stakes testing must remain open so-called developed world). But children to disinterested reflection. This involves a de- in the US whose parents only attained high dogmatization about the contemporary merits school will perform worse on literacy tests of high-stakes testing and their fundamental than children in Finland whose parents have necessity. We need to be able to decondition the same education as their US counterparts. the beliefs and assumptions of our working This is because economic inequality affects epistemology; to de-reify and de-automatize the quality of family life, in areas of health, everyday reality; and reconsider conditioned security, rates of substance abuse, etc.34 attributes that clutter our commonplace ob- Education has been made the only avail- servations about measuring and assessing able means of addressing injustices that what we deem most pertinent to the process arise from economic disparity, and to us of learning. Without this epistemic decoloni- this constitutes one of the worst crimes of zation as our guide, we are doomed to repeat capitalism. In our view, the answer to im- and re-entrench the worst elements of capital- proving education lies in creating more eco- ism in schooling; a back breaking load to be nomic equality and equity. This is a lesson borne most painfully by those already at the that all capitalist contexts (in Latin America margins of educational power and success. and elsewhere) need to learn, as the United States seems oblivious to it. Focusing on 1 Melissa LazarÍn, Testing Overload in Amer- testing and accountability will only further ica’s Schools (Washington, DC: Center for exacerbate already dire conditions for the American Progress, 2014), 2. Arlo Kempf, marginalized and working-class communi- The Pedagogy of Standardized Testing: The ties throughout Latin America. Radical Effects of Standardization in US and Canadian Education, (New York: Palgrave, Conclusion 2016), 15-17. In this article we have attempted, albeit briefly, 2 Howard Nelson, Testing More, Teaching Less, to highlight and survey the perils of the cur- What America’s Obsession with Student Test- rent approach to education and student evalu- ing Costs in Money and Lost Instructional Time ation undertaken in US K-12 schooling. We (Washington, DC: American Federation of write this with humility, as a modest missive to Teachers, 2013), 6–7. 10

ENGLISH

5 Howard Nelson, Testing More, Teaching Less, Testing Separating Wheat Children from Chaff What America’s Obsession with Student Test- Children,” AlfieKohn.Org, 2002, accessed May, ing Costs in Money and Lost Instructional Time 3, 2013, http://goo.gl/AwhS6w; Peter Sacks, (Washington, DC: American Federation of Standardized Minds: The High Price of America’s Teachers, 2013), 6–7. Testing Culture and What we can do to Change it 6 Organization for Economic Cooperation and De- (Cambridge, MA: Perseus, 1999); and others. velopment, Reviews of National Policies for Edu- 12 Chicagoland Researchers and Advocates for cation: Improving Lower Secondary Schools in Transformative Education. Research Brief 1: Norway (Paris: oecd, 2011), 9. Arlo Kempf, The Testing Today in Context: History, Impact, and Pedagogy of Standardized Testing: The Radical Alternatives (Chicago: CReATE, 2012), 4, ac- Effects of Standardization in US and Canadian cessed January 10, 2014, http://www.createchi- Education, (New York: Palgrave, 2016), 15-17 cago.org/. 7 Organization for Economic Cooperation and De- 13 William Popham, Classroom Assessment: What velopment. pisa 2012 Results: What Makes Schools Teachers Need to Know (Boston, MA: Pearson, Successful? Resources, Policies and Practices Vol- 2011), 83. ume 4 (Paris: oecd Publishing, 2013), 147. Arlo 14 Arlo Kempf, The Pedagogy of Standardized Kempf, The Pedagogy of Standardized Testing: The Testing: The Radical Effects of Standardization Radical Effects of Standardization in US and Cana- in US and Canadian Education, (New York: Pal- dian Education, (New York: Palgrave, 2016), 15-17 grave, 2016), 18. 8 Organization for Economic Cooperation and De- 15 See: Arlo Kempf, The Pedagogy of Standard- velopment. pisa 2012 Results: What Makes Schools ized Testing: The Radical Effects of Standard- Successful? Resources, Policies and Practices Vol- ization in US and Canadian Education, (New ume 4 (Paris: oecd Publishing, 2013), 147. York: Palgrave, 2016), 15-17.; James Popham, 9 Alfie Kohn. “Standardized Testing and its Vic- Classroom Assessment: What Teachers Need to tims,” Education Week, September 27, 2000, ac- Know (Boston, MA: Pearson, 2011); James Po- cessed October 1, 2014, http://goo.gl/7ejJKN. pham, The Truth about Testing: An Educator’s 10 Arlo Kempf, The Pedagogy of Standardized Call to Action (Alexandria, VA: ascd, 2001); Testing: The Radical Effects of Standardization Richard Murnane and John P. Papay, “Teach- in US and Canadian Education, (New York: Pal- ers’ Views on No Child Left Behind: Support grave, 2016), 15-17. for the Principles, Concerns about the Prac- 11 See: Chicagoland Researchers and Advocates tices,” Journal of Economic Perspectives 24, for Transformative Education. Research Brief 1: no. 3 (2010): 151–66; Wayne Au, Unequal by Testing Today in Context: History, Impact, and Design: High-Stakes Testing and the Standard- Alternatives (Chicago: CReATE, 2012), 4, ac- ization of Inequality (New York: Routledge, cessed January 10, 2014, http://www.createchi- 2007); Linda Darling-Hammond, “Race, In- cago.org/; Pasi Sahlberg, Finnish Lessons: What equality and Educational Accountability: The Can the World Learn from Educational Change Irony of ‘No Child Left Behind,’” Race Ethnic- in Finland? (New York, NY: Teachers College ity and Education 10 (2007): 245–260; David Press, 2011); Richard Thorndike, Measurement Hursh, “Exacerbating Inequality: The Failed and Evaluation in Psychology and Education (8th Promise of the No Child Left Behind Act. Race ed.) (Upper Saddle River, NJ: Prentice Hall, 2011); Ethnicity and Education 10, no. 3 (2007): Howard Wainer, Uneducated Guesses: Using Evi- 295–308; Kate Menken, “Teaching to the Test: dence to Uncover Misguided education Policies ( How No Child Left Behind Impacts Language Princeton, NJ: Princeton University Press, 2011); Policy, Curriculum, and Instruction for Eng- James Popham, Classroom Assessment: What lish Language Learners,” Bilingual Research Teachers Need to Know (Boston, MA: Pearson. Journal 30, no. 2 (2006): 521–546; Rod Paige, 3 Arlo Kempf, The Pedagogy of Standardized Test- 2011); James Popham, “Instructional Insensitiv- “No Child Left Behind: The Ongoing Move- ing: The Radical Effects of Standardization inUS ity of Tests: Accountability’s Dire Drawback,” ment for Public Education Reform. Harvard and Canadian Education, (New York: Palgrave, Phi Delta Kappan 89 (2007):146–155; James Educational Review 76, no. 4 (2006): 461–473; 2016), 15-17. Matthew Lynch, “Education Offi- Popham, The Truth About Testing: An Educator’s John Rogers, “Forces of Accountability? The cials to Re-examine Standardized Testing,” Edu- Call to Action (Alexandria, VA: ASCD, 2001); Power of Poor Parents in No Child Left Be- cation Week, October 22, 2014, accessed Octo- Linda Darling-Hammond, The Flat World and hind,” Harvard Educational Review 76 (2006): ber 23, 2014, http://goo.gl/s6czCW Education: How America’s Commitment to Eq- 611–641; Jennifer Booher-Jennings, “Below 4 Lizette Alvarez, “States Listen as Parents Give uity will Determine our Future (New York, NY: the Bubble: ‘Educational Triage’ and the Texas Rampant Testing an F,” New York Times , No- Teachers College Press, 2010); Alfie Kohn, What Accountability System,” American Educational vember 9, 2014, accessed November 10, 2014, Does it Mean to be Well Educated? (Boston, MA: Research Journal 42 (2005): 231–268; Frances http://goo.gl/QeLhYd. Beacon Press, 2004); Alfie Kohn, “Standardized Contreras, “Access, Achievement, and Social 11

National Educational Evaluation Policy Gazette in Mexico

Capital: Standardized Exams and the Latino http://nces.ed.gov/pubsearch/pubsinfo.asp. tional Change (Thousand Oaks, CA: Corwin College-Bound Population” Journal of His- 19 Arlo Kempf, The Pedagogy of Standardized Press, 2009), 62. panic Higher Education 4 (2005): 197–214; Testing: The Radical Effects of Standardization 28 Arlo Kempf, The Pedagogy of Standardized Jo Boaler, “When Learning no Longer Mat- in US and Canadian Education, (New York: Testing: The Radical Effects of Standardization ters—Standardized Testing and the Creation Palgrave, 2016), 14-20. in US and Canadian Education, (New York: of Inequality,” Phi Delta Kappan 84 (2003): 20 Arlo Kempf, The Pedagogy of Standardized Palgrave, 2016), 14-20. 502–506; Mary Barksdale-Ladd, and Karen Testing: The Radical Effects of Standardization 29 John Marsh, Class Dismissed? Why We Cannot Thomas, “What’s at Stake in High-Stakes Test- in US and Canadian Education, (New York: Pal- Teach or Learn Our Way Out of Equality (New ing: Teachers and Parents Speak Out,” Journal grave, 2016), 14-20. York: Monthly Review Press, 2011), 89. of Teacher Education 51 (2000): 384–397; and 21 Pasi Sahlberg, “Global Educational Reform Move- 30 John Marsh, Class Dismissed? Why We Cannot others. ment is Here!” (PasiSahlberg.com, N.D.), accessed Teach or Learn Our Way Out of Equality (New 16 Provasnik, Stephen, et al., “Highlights From September 6, 2014, http://goo.gl/52GsWm. Rich- York: Monthly Review Press, 2011), 86. timss 2011: Mathematics and Science ard Wilkinson and Kate Pickett. The Spirit Level: 31 Norman Doidge, The Brain That Changes Itself: Achievement of U.S. Fourth- and Eighth-Grade Why More Equal Societies Almost Always Do Bet- Stories of Personal Triumph from the Frontiers of Students in an International Context,” National ter (London: Allen Lane, 2009). Brain Science (New York: Viking, 2007), 309-10. Center for Education Statistics, 2012, accessed 25 Richard Wilkinson and Kate Pickett, The Spirit 32 John Marsh, Class Dismissed? Why We Cannot May 2, 2014, www.nces.ed.gov/timss. Level: Why Equality is Better for Everyone (Lon- Teach or Learn Our Way Out of Equality (New 17 National Center for Education Statistics (2012), don: Penguin, 2010). York: Monthly Review Press, 2011), 57. The Nation’s Report Card: Trends in Academic 26 Pasi Sahlberg, “What Canada can Learn from Fin- 33 John Marsh, Class Dismissed? Why We Cannot Progress 2012, accessed October 21, 2014, land” (2014 R.W. Jackson Lecture, Toronto, On- Teach or Learn Our Way Out of Equality (New http://nces.ed.gov/pubsearch/pubsinfo.asp. tario, April 17, 2014). Webcast available at https:// York: Monthly Review Press, 2011), 182. 18 National Center for Education Statistics (2012), webcasts.welcome2theshow.com/oise/2230. 34 John Marsh, Class Dismissed? Why We Cannot The Nation’s Report Card: Trends in Academic 27 Andy Hargreaves and Dennis Shirley, The Teach or Learn Our Way Out of Equality (New Progress 2012, accessed October 21, 2014, Fourth Way: The Inspiring Future for Educa- York: Monthly Review Press, 2011), 210.

VOICES FROM THE CONFERENCE

Education-policy ecisions are taken in different spheres decisions: the link and at different levels, and involve dif- Dferent hierarchies. How one defines what between evidence and the decision-making process is, or should be, depends on the frame of reference used values and the role assigned to knowledge, values or analysis within the said frame. If one believes that knowledge or rea- “Values open up new horizons and soning are involved in the said process, then public and human aspirations, while data will play a fundamental role in ascer- evidence can point to both obstacles taining what view of reality informs deci- and possibilities”, says the author. sion-making. However, since this very belief “Evidence can support precise, in the rationality and “value” of evidence correct diagnoses of situations or would also appear to assume an encounter with values, then, in order to analyze deci- problems, but values provide the sions, we must understand how they take social leverage needed to guide shape and how they put facts and values into interventions aimed at addressing practice, as well as how convictions, ideas the said problems or situations and and interests interact with facts and reali- justify changes that are both possible ties, and vice versa (Parsons, 2013). and necessary”, he assures. Assuredly recent discussions about public-policy analysis have been pervaded by the growing conviction that policy-implementa- Francisco Miranda López tion decisions must be based on evidence. An Head of the Educational Policy important influence in this regard is the so- and Regulation Unit of the inee called evidence-based-policy (ebp) approach, [email protected] whereby public-policy decisions and choices 12

ENGLISH

among public-policy options must be based that lead to subjective, arbitrary, biased from the use of numerical scales to measure on the overt, purposeful use of the best avail- decision-making; the next is the concept of and observe aspects and facets of realityit able evidence stemming from facts, data and the rigorously constructed “evidence-based . Hence, by establishing objective magni- relationships rooted in empirical conclusions approach” as a means of avoiding simplis- tudes, we can more accurately measure and that are not founded on the subjective views tic, mechanical judgments; and the third is define the problem, breaking it down into its of a small number of protagonists, but rather the concept of “evidence as an objective af- different components and rendering it open on objective measurement using relevant, re- firmation” that stresses the measurement of to analysis whereby it can be mathematically liable methods and techniques (Davis, 2000; problems as a prerequisite for sizing them or logically duplicated so as to delve deeper Aguilar, 2005; Bracho, 2010). up, analyzing them and thereby coming up into it, create scenarios for experimentation Although the above idea is very con- with possible solutions. and the establishment of predictability, or vincing and has much to recommend it, it Evidence deemed to be “hard” is as- associate with other variables to ascertain does not completely take stock of the role sumed to consist of relevant information how “flexibly” it behaves. Thus, how sensible that other subjective, judgmental factors that does not arise from people’s subjectiv- and efficient decisions are depends on the play in the decision-making process. While ity – i.e. information that is not distorted by extent to which they are based on the mea- EBP helps to ensure that decisions are more perceptions or opinions based on sympathy, surement of the problems analyzed, the pre- rational and objective, guarding against hearsay or empathy vis-à-vis a given fact or cision with which their magnitudes are es- decisions that are whimsical, doctrinal, ir- situation. What distinguishes hard data is tablished, and their potential variability, all rational or headstrong, it does not take into that they exist independently of people and, of which, together, enable us to plan precise account the fact that all policy-making deci- although they may give rise to the latter’s interventions aimed at solving, diminishing, sions involve value judgments - i.e. ultimate- actions or behaviors, constitute an external, limiting or redefining them (Aguilar,2005 ; ly it does not envisage other logics, based on objective reality that entails its own thought Muñoz and González, 2010), and also to es- principles and values, that also inform and processes and behavior (Hitch and McK- tablish criteria and mechanisms to measure justify the decisions that are made and can ean, 1962; Alain and Smith, 1971; Quade, the success of the said interventions. play an important part in mediating and le- 1982; Corzo, 2013). Hence, it is considered veraging the data, and the models stemming that, when a decision is based on the best Proposals based on values therefrom, for the purpose of making deci- available evidence, it is more likely to solve and principles sions about public policy (Simon, 1984; Ma- the problem that it is seeking to solve or ad- Just as our everyday concept of what is good jone, 1989; Bracho, 2011). dress, so long as it is not the butt of personal and bad, correct or incorrect, fair or unfair What we wish to stress, in this brief ar- judgements, disagreements between people influences our decisions and actions, in the ticle, is that EBP logic needs to serve as a or groups, or the opinions of those who play same way the ideas and conceptions of so- basis for making rational, value-related deci- a part in formulating it, implementing it, or ciety and politicians about certain values sions, rather than being used to support deci- analyzing its results. influence the orientations of the policy de- sions that only contemplate the desired ends, On the other hand, evidence is com- cisions that are made, functioning as a kind since we are convinced that this will enable monly associated with the procedures or of router in our assessment of the reality us to make the data more meaningful and methods that are used to produce it, which we perceive. With varying intensity, these orient the way they are used to solve prob- is not synonymous with “hard data”, since routers serve, at least initially, to channel lems, provide input to debates, and inform the latter are found, while evidence is always or generate the meaning of everything we policy-making, which, in modern “public- built, rather than floating around waiting to normally experience in our private lives, in policy-governed countries”, needs, among be caught, being the intentional result of a the public domain, and in political or insti- other things, to involve citizens, respect and method that involves various conceptual tutional life. further human rights, solve social problems, resources, disciplinary frameworks, techni- In the vast literature about research and foster democratic government, transpar- cal resources and operational contexts that into values, one can identify at least three ency and accountability, all of which are basic are different from -and in many ways more core concepts that bear keeping in mind principles adherence to which is all the more complex than- those required for data analy- in the context of decision-making and essential in government policy-making. sis (Aguilar, 2005; Muñoz y González, 2010). public-policy processes. First, values are With regard to the aforesaid imperatives Hence, the method renders the evidence interpreted as principles, becoming ethical and requisites, we go on below to provide a valid by enabling it to be replicated and mainstays, ontological bases and compass- brief breakdown of the different meanings compared, thus depending on who proposes es that guide our behavior, second, they are assigned to evidence and values, and of their or propounds it, rather than on the different like joints or interfaces between collective, felicities and shortfalls. mental and technical operations that under- political and private or small-group life, Evidence-based reflection prop it and can render it “falsifiable” via dif- and third they serve as touchstones for It can be argued that the notion of evidence ferent testing or refutation mechanisms. judging the actions and behavior of our fel- rests on at least three core concepts that Finally, strongly associated with the low men. support its claims to be valid for purposes concepts of “hard data” and “hard method”, As principles, values delimit meaning, of decision-making. The first of these is the we use the term “magnitude” to refer to evi- and hence govern people’s behavior, also concept of “hard evidence” that supports dence. One of the most characteristic fea- speaking to individuals and binding them decision-making, as against “soft concepts” tures of all hard evidence is that it results via their belief in, or acceptance of, the con- 13

National Educational Evaluation Policy Gazette in Mexico

tents or attitudes that they prescribe or de- strategies aimed at serving the public good. of those involved or benefited, and also meet termine (Aguilar, 2007; Weimer and Vining, In other words, policies and the decisions the legitimacy criteria that underpin demo- 2010; Bracho, 2011; Arellano and Blanco, that stem from them are the result of a tense, cratic governments, as well as satisfying 2013, Merino, 2013). In this way values have composite process that potentially balances transparency and accountability criteria. to do with the foundations, origins, main te- the gathering and use of evidence and the This is especially pertinent for evalua- nets or raison d’être of things, thus forging creation of meaning and structural content tion policy, an area in which one seeks not strong links among individuals and groups oriented by fundamental principles or values. only to make effective, legitimate decisions that have -or seek to create- common fac- While evidence provides empirical sup- that render evaluation relevant, fair, ad- tors that identify and unify them and are port and underpins the linkage between equate and transparent in order to ascertain recognizable in all physical, cultural, social ends and means that is part and parcel of the state or condition of education, but also or historical contexts. all policy decisions, values can increase the to support education-policy decisions. However, values serve not only as prin- amount of knowledge that the said evidence The aim of evaluation policy is both to ciples in the abovementioned way, but also can contribute by demanding more data and remedy problems and fill gaps in evaluation, as social levers or instruments that enable measurements regarding the basic fulfill- and also to foster improvement. In other communication between -and containment ment of moral, legal or human obligations. words, one seeks, on the one hand, to create and development of- individuals and so- However, the opposite may also be the case, a connection between evidence and values ciety, a role that they play when they push since, as result of data regarding viability that makes it possible to ascertain the state and guide individual action towards more and its technical and scientific underpin- of the components, processes and results meaningful social spheres such as groups, ning, or based on the making of technical or of the Mexican Education System, and, on communities, countries or states (Schwartz, historical comparisons during political de- the other hand, to build logical, empirical, 1999; Lindblom, 1992; Asthama, Richardson bate, evidence can dampen the ideological value-based and policy links between the and Halliday, 2002). Values serve as impor- or political zeal of some decision-makers to status quo and the proposed policy action, tant mechanisms for reduplicating the social address social problems and make the nec- which entails going beyond the strict terrain order, either for the purpose of strengthen- essary means available. of evaluation and entering the domain of ing identification between individuals and Values open up new horizons and public planning, government administration and larger social groups and affirming the for- and human aspirations, while evidence can policy-making. mer’s membership of the latter, or in order point to both obstacles and possibilities. The processes whereby evidence and to achieve congruence between sociopoliti- Evidence can help us to determine what is values are linked or separated in education cal criteria and the criteria governing indi- politically feasible, but values and princi- policy would seem to imply similar pro- vidual or group behavior. ples can lead to concerted political efforts cesses in government policy-making. Even Finally, values are closely linked to the to overcome limitations and open up new when evidence and the weight thereof are criteria whereby we interpret and evaluate worlds of possibility. Evidence can support the same, or government action adheres to reality and human behavior, and on which precise, correct diagnoses of situations or the same value benchmarks in both evalu- we base value judgements and translate them problems, but values provide the social le- ation policy and education policy, in fact into certain judgments about the state, nature verage needed to guide interventions aimed there will always be margins of linkage or of -or results stemming from- a given thing, at addressing the said problems or situations separation that are sure to determine the ef- event or process pertaining to material, so- and justify changes that are both possible fectiveness, credibility and, ultimately, the cial or human reality. Moreover, by func- and necessary. direction of government action aimed at tioning in this way, values serve not only to In short, the relationship between evi- improving education, with regard both to determine the place we can assign to things dence and government policy can by no quality and to equity. within given categories, but may also be open means be linear or automatic. As Eugene In the worst case, the aforesaid dynamic to some kind of inner grading or stratification Bardach asserts (1980), in politics no meth- could lead to tensions and differences in the that enables us to measure the distance be- odology is universally valid, and so, rather building of consensus and the positing of ba- tween certain features or traits of the afore- than seeking methodological anarchism, sic values, not to mention the problems as- said things and certain desired values. one should, to the extent that the context sociated with issues of legitimacy (i.e. that of allows, move towards plurality, which im- data, methodology and measurability) vis- Linkage in evaluation policy plies various interventions that are not just à-vis an educational problem and the public As is clear from the above comments and technical, but also involve different ethical, debate to which the latter can give rise. concepts, initial conclusions regarding political and social criteria. The quest for grounds for decisions -above all political de- fundamental balanced decisions that are References cisions- must be based on a coordinated ef- technically sound on the one hand, and so- Aguilar, L. F. (2005). Introductory study, in Majone, fort to reach decisions that are both effective cially acceptable, fiscally responsible and G. Evidencia, argumentación y persuasión de la and legitimate. In the final analysis we are institutionally transparent on the other formulación de políticas. Mexico: cncpyap/fce. talking about decision-making that is uni- hand, is probably one of the most sensitive Aguilar, L. F. (2007). El estudio de las políticas fied and structured in line with needs and aspects of public-policy formulation, added públicas. Mexico: Porrúa. contexts where reasoning based on ends and to which such decisions need to be partici- Alain C. E. and Smith, W. (1971). How much is reasoning based on values come together in pative, take into account the points of view enough. New York: Harper and Row. 14

ENGLISH

Arellano, D. and Blanco, F. (2013). Políticas públicas y Polaris, S. A. de C. V. Muñoz, M. and González, J. (2010). Valoración de democracia. Mexico: Instituto Federal Electoral. Corzo, F. (2013). Diseño de políticas públicas. la calidad de la evidencia y fuerza de las reco- Asthama, S., Richardson, S., and Halliday, J. Mexico: Grupo Editorial y de Investigación mendaciones. Barcelona: Sociedad Española (2002). Partnership working in public policy Polaris S. A. de C. V. de Medicina Familiar y Comunitaria. provision. Social Policy and Administration, Davies, P. (2000). The relevance of systematic Parsons, D. W. Public Policy: An Introduction to Vol. 36, No.7, pp. 780. reviews to educational policy and practice. the Theory and Practice of Policy Analysis, Bardach, E. (1980). The Implementation Game. Oxford Review of Education, 26 (3 y 4), pp. Elgar, Edward Publishing, Inc. (1995). Pub- Cambridge: Themit Press. 365-378. lished in Spanish by Flacso, Mexico. (2007) Bracho, T. (2010). Políticas públicas basadas en la Hitch, C. J. and McKean, R. (1962). The Econom- Quade, E. S. (1982). Analysis (or Public Decisions). evidencia. La política pública como acción in- ics of Defense in the Nuclear Age. Cambridge: Amsterdam, North-Holland. formada y objeto de investigación, en Merino, Harvard University Press. Schwartz, S. (1999). A theory of cultural values M. and Cejudo, G. (Compilers). Problemas, Lindblom, Ch. (1992). El proceso de elaboración and some implications for work. Applied Psy- decisiones, soluciones. Enfoques de política de políticas públicas. Madrid: Ministerio para chology, 48 (1), pp. 23-47. pública. Mexico: fce/cide. las Administraciones Públicas. Simon, H. (1984). La nueva ciencia de la decisión Bracho, T. (2011). El papel de los valores en la Majone, G. (1989). Evidence, Argument and Per- gerencial. Buenos Aires: El Ateneo. evaluación de políticas para sistemas educati- suasion in the Policy Process. Yale University Weimer, D. and Vining, A. (2010). Policy Analysis: vos complejos. Revista Mexicana de Investig- Press. Concepts and Practice. USA: Routledge. ación Educativa, 16 (50). Merino, M. (2013). Políticas públicas. Ensayo so- Corzo, F. (2013). Diseño de políticas públicas. bre la intervención del Estado en la solución Mexico: Grupo Editorial y de Investigación de problemas públicos. Mexico: cide.

VOICES FROM THE CONFERENCE FACING OUR CHALLENGES

Challenges, aims and I: raising educational quality by improving work: decisions and school conditions so that students -whether children, youths or adults- can acquire the educational evaluation knowledge, and develop the skills, values and attitudes, that they need in order to in Querétaro have a rich personal, family and social, be responsible, committee citizens, and go on learning throughout their lives. In the interview transcribed below, which doesn’t just focus on II: ensuring equality in terms of access to, education, but also covers policy continuation in, and completion of, all the and syllabus design and the use of types, modalities and levels of education, information in the light of the recent with priority being afforded to the most dis- advantaged students, groups and locations. results achieved via evaluation, José III: expanding coverage by increasing, Alfredo Botello Montes, Minister of and making more flexible, the opportunity Education for the State of Querétaro, for people to enter, remain in, all the types, talks with the Gazette about his modalities and levels of education, ensuring states experiences with educational that they have access to cultural services and evaluation and the challenges that it resources, sports and recreational activities, faces. and scientific and technical amenities and events.

IV: Improving institutional management by consolidating the running and linkage of r. Botello Montes, who is Minister of the state-level education system in order to Education for the State of Querétaro, ensure that education policies are effective Mpoints out that the strategic aims of the min- continually evaluated, that resources are istry he heads include: used efficiently, transparently and account- 15

National Educational Evaluation Policy Gazette in Mexico

ably, that the needs of both students and society are satisfied, and that management Table 1. Planea 2015 results for the state of Querétaro centers on schools and classrooms. % of students Countrywide Asserting that the position he holds af- Placing countrwide placing in levels II average fords him a big opportunity to serve others to IV in his state, which is an important region of Mexico, Botello talks about the main input Language and 9th 54.6% 50.4% that the state of Querétaro has managed to Communication obtain via educational evaluation and how year of

the said input has been used in order to de- th sign the current policies and programs: 6 Mathematics 8th 43.2% 39.5% Primary School “Of course,” he says, “the evaluation currently being carried out by the National Educational Evaluation Institute (Spanish Language and 4th 76.3% 70.5% acronym: inee) is very important, since it Communication provides us with results that have an im-

pact at both the national and international School. levels, mainly in the areas of Language and Mathematics 3rd 42.5% 34.6% year of Secondary rd

Communication and Mathematical Rea- 3 soning (See Table 1 and Figure 1) ”. “For its part”, remarks Botello, “Planea Note: Pertains to the National Pan for the Evaluation of Learning Outcomes Examinatiion (Spanish acronym: has become a very important means of Planea) elementary-level evaluation in Mathematics and Language and Communication that was administered measuring the results of the local education in June, 2105, along with a questionnaire about coexistence skills, to 53,220 students in the final primary and system, due to its emphasis on the acquisi- secondary grades in 1,833 schools in Querétaro. Source: Ministry of Education of the State of Querétaro. tion of basic learning and curriculum-based competencies. After the Education Reform that began in our country a couple of years ago, it is essential that we obtain precise Figure 1: 2015 Planea upper-secondary-level results for Querétaro information about the suitability for the teaching profession of our teacher trainees The results of the 2015 Planea upper-secondary-level examination were as follows: and in-service teachers based on their per- 42.2% of the students in Querétaro placed in the highest performance levels (II and formance in the said evaluations”. IV) in Language and Communication (i.e. reading comprehenension), compared with ”Furthermore, Planea helps us to ascer- 36% countrywide, the best results, in descending order, being obtained in the states st nd rd th tain our students’ progress and find out our of Puebla (1 place), Baja California (2 place), Durango (3 place) , Jalisco (4 place) th state-level education systems areas of op- and Querétaro (5 place). portunity for ongoing improvement, as well 22.2% of the students in Querétaro placed in the highest performance levels in to measure our performance at the national Mathematics, compared with an 18.8% countrywide average), the best results, in de- st nd and international levels”. scending order, being obtained in the states of Durango (1 place), Puebla (2 place), rd th th th Regarding the aforesaid areas of oppor- Sonora (3 place) , Sinaloa (4 place), Baja California (5 place), Querétaro (6 place) th tunity and the design of policies and pro- and Chihuahua (7 place). grams aimed at exploiting them, Botello, a former federal congressman, comments: Source: Ministry of Education of the State of Querétaro “We need to work with the teachers, the managers and the students themselves to hone their skills in the important areas of language, communication and mathematical reasoning”. ”Communication, for its part, is an area cies that are needed in order to make such ”Though we’ve already addressed the of opportunity steeming from the Educa- improvements (See Figure 2) ”. latter area -i.e. including it in the curricu- tional Reform. We have the School Tech- “We should take advantage not only of lum- we haven’t given it the importance it nical Committees (STCs), in which parents, the instruments that are currently available, merits, notwithstanding the fact that it’s the teachers and the education authorities work such as the School and Zonal Technical basis for all logical thought, which is why together in order to proactively promote Committees, but also of the direct links that we’re offering the largest number of work- ongoing improvement in schools, and these should exist among schhol principals, school shops, and why one of the policies we’re go- also serve to help school principals and mangement, teachers and parents. Having ing to implement this year focuses heavily teachers to monitor the educational com- noticed that the latter are not involved, on on language and mathematical reasoning”. munity’s endeavors to implement the poli- an ongoing or regular basis, in what goes 16

ENGLISH

on inside our schools, I believe that another Figure 2: Results of the 2015 competitive process for appointing area of opportunity exists to find ways to in- new elementary-level teachers. volve them more in thir children’s education and prevent them from always being on the th From the 11th to 19 of July, 2,341 of the 2,729 candidates who had applied to par- sidelines, and I’ve noted that out teachers ticpate in the selection process for appointment to elmentary-level teaching posts in are keen to help in this endeavor”. the State of Querétaro (i.e. 85.78% of the total registrees), with the following results: ”Additionally, given the size of our state’s 69.76% of the examinees (i.e. 1,633 out of 2,341 candidates) got a “Suitable” education system, we face the problem of grade, putting our state in first place countrywide, with the biggest percentage of turnover in our schoolsdue to retirement (See “Suitable” grades, as against a national average of 51.91%. Table 2). It’s hard to find enough class teachers 30.24% of the candidates (707 people) were awarded an “Unsuitable” grade, as and one of our challenges is to issue calls for compared with a national average of 48.09%. candidates to take the examinations for admis- 16.15% of the candidates who were graded “Suitable” placed in the Very High Per- sion to the National Teaching in order to cover formance group, 24.86% in the High Performance group, 27.72% in the Good Perfor- vacant positions with suitable staff”. mance group, and just 1.03% in the Satisfcatory Performance group. Assuredly, the dissemination of educa- 72.88% of the teacher-training-college graduates who took the examination got a tional-evaluation results is of the essence, “Suitable” grade, making Querétaro the state with the fourth highest number of such and on this topic Botello comments: graduates with “Suitable” grades countrywide. ”Though I only recently took up my post Just 26.01% of all the teacher-training-college graduates who took the examination in in October of 2015, one of my aims is to en- our state -i.e. one of the lowest percentages in all of Mexico’s 32 states- received an sure that our School Technical Commits are “Unsuitable” grade. duly set up and that both my own Minis- try and the Federal Ministry of Public Edu- Source: Ministry of Education of the State of Querétaro cation can provide them with feedback on evaluation results, though we also want to look at the issue of ensuring participation by the members of the said committees, which we believe to be a big challenge not only lo- Table 2. Coverage in the Querétaro Education System cally, but also countrwide”. A former teacher at the Monterrey Coverage: new age ranges1 Coverage: traditional age ranges2 Technical and Higher-studies Institute (Spanish acronym: itesm), Botello Montes Level Registered Population* % Registered Population* % stresses his state’s strengths: “The greatest strength I’ve found are the Pre-school 85 065 112 116 75.9 85 065 112 116 75.9 very professional teaching and managerial staff, who’ve worked their way up through Primary 248 608 227 087 109.5 248 608 264 884 93.9 the professional-teaching system.I’ve even Lower-sec- met met people with Ph.D’s and master’s ondary 120 780 113 458 106.5 120 780 113 548 106.4 degrees in education who are class teachers or school principals. Our staff are very well trained and enthusiastic”. Upper-sec- 75 932 113 568 66.9 75 932 113 320 67.0 ”In our Department of Elementary Edu- ondary cation we encountered three different plan- Lower-profes- 219 113 568 0.2 219 113 320 0.2 ning areas which would assume responsibil- sional ity for evaluation-related tasks. There was a primary section, a pre-school one and a Senior-high 75 713 113 568 66.7 75 713 113 320 66.8 secondary one, each operating separately without any links to the others. We’re cur- Higher** 65 790 185 466 35.5 65 790 183 815 35.8 rently reengineering their management so as to fuse them into a single area and ensure Source: Data: start of the 2014-2015 school-year. Statistical Summary. Data: start of the 2014-2015 very good communication among them, so school-year. Education Portal. Government of the State of Querétaro. as achieve a new, uniform evaluation system * Population projections: National Population Council (Spanish acronym: Conapo, April 2013 / ** that serves all the different types of educa- Bachelor’s Degree and Classroom-taught Advanced Technical Diploma. tion, rather than a jigsaw puzzle. We want to work as a team, a body whose arms, legs and head work in unison”. On the subject of the National Educa- tional Evaluation Policy and the Dialogues 17

National Educational Evaluation Policy Gazette in Mexico

with the Educational Autorities aimed at who are directly involved in evaluation, and 1 New age ranges: preschool: ages 3, 4 and 5: ages formulating it, he says: I also think that we should give feedback to 6 to 11; lower-secondary: ages 12 to 14; upper- “The National Educational Evaluation the rest of the education authorities on our secondary: ages 15 to 17; higher: ages 18 to 22. System Conference is an excellent mecha- views about the application of the evalu- 2 Traditional age ranges: preschool: ages 3, 4 and nism implemented by the inee. I was able to ations. In general, I find these meetings, 5; primary: ages 6 to 12; lower-secondary: ages attend the meeting that was held in October, where we can meet and understand each 13 to 15; upper-secondary: ages 16 to 18; high- and found that it was a very good forum for other, and exchange opinions, to be a very er: ages 19 to 23. giving input and exchanging opinions. They useful useful way of ensuring that sate-level showed us data indicating which actions policy doesn’t become divorced from federal need to be taken and provided us with feed- policy”. Visit the website of the Ministry of back as a basis for decision-making”. ”Our ultimate aim is to produce good, Education of the State of Querétaro ”I believe that particpation in the said well trained, well educated citizens. This is at: http://www.queretaro.gob.mx/ Dialogues should not be limited to the Min- the big task facing our country, its states and educacion/ istry of Education, but also involve the staff also, of course, its municipalities”.

VOICES FROM THE CONFERENCE FACING OUR CHALLENGES

An overview of “Furthermore, with the decentraliza- education and tion that confers autonomous status on the National Institute for Educational Evalua- evaluation in Sonora: tion (Spanish acronym: inee) the evaluation systems charged with fostering high-quality how to overcome the in education have been rendered legal and challenges institutionalized. In this regard, while not underestimating the challenges faced by Sonora with regard to coverage and satisfac- “To raise quality, we need an tion of the existing demand -above all at the evaluation system that produces upper-secondary and higher-studies levels- information that’s relevant for the government of our state, headed by our decision-making”, says Ernesto governor, Claudia Pavlovich Arellano, has De Lucas Hopkins, Minister of made high-quality an across-the-board cri- Education and Culture for the terion for the design of educational policies and programs”. State of Sonora, in the interview transcribed below, where he talks Evaluation for improvement about the strategies that he is Pavlovich Arellano, who held various gov- pursuing in order to once more ernment posts in the State of Sonora during make his state a leader in the field of the ten years before he took up his present education. post, asserts: “An evaluation system that produces timely, relevant information for purposes of decision-making, and is ac- For us to create high-quality educational countable both for its results and for the options, the State needs to take actions educational processes stemming from them, “aimed at achieving renewed levels of effi- is a prerequisite for raising the quality of ciency, effectiveness and equity in its policy- Sonora’s education system. The input from making;” states Ernesto De Lucas Hopkins, evaluation should be used to formulate gov- Minister of Education for the State of So- ernment policies aimed at raising the qual- nora. “In our country, the administration ity of education and fostering equal oppor- of President Enrique Peña Nieto is carrying tunity in education. The main input includes out an Educational Reform the main aim the generation, analysis and dissemination of which is to create a high-quality educa- of information and knowledge about edu- tion system for all, as made manifest in the cation, the design and implementation of a amendment to Article Three of the Mexican system of benchmarks for ascertaining the Constitution, which was voted into law by quality of education in our state, the carry- all the members of both houses of Congress”. ing out and promotion of research that un- 18

ENGLISH

derpins the design of specific programs in and the guidelines stemming therefrom that ceived the Sonoran education system at the line with the needs of Sonora to strengthen are established by the inee, to disseminate beginning of this administration, we face big areas such as infrastructure and equipment, results and recommendations for the design challenges that need to be tackled via gov- professional training and development, and of the of the pertinent ongoing-training and ernment policies based on strategies and remuneration systems. In the context of the professional-development programs, and to lines of action whose priority is to win back current reform, other very important aims sign such agreements with authorized gov- the trust of our state’s citizens by involving have to do with student learning outcomes ernment institutions as are needed to raise them in government decision-making and and the suitability of teaching and manage- the quality of education, etc.”. fostering cooperation in all the spheres of rial staff”. “Outstanding among the members of public life. These aims are enshrined in the “Finally, one of the current state govern- the previous administration (2009-2015) 2016-2021 State Development, which sees ment’s priorities is to include efficiency and is De Lucas, who was the national director the recovery of trust as a challenge that re- effectiveness in the use of public resources of ProMéxico (a trust fund of the Mexican volves around efficiency – i.e. an administra- as specific input requisites for government- Government, pertaining to the Ministry of tion that responds to the population’s needs policy design, along with the need for the Economy, that promotes international trade via responsible action, the proper handling results to be in harmony with the 2016-2021 and investment) and also of the Institute for of public resources, and the implementation State Development Plan and to manifest Mexicans Abroad (Spanish acronym: ime). of innovative government policies with the themselves in the Sectoral Education Plan We received an education system with seri- three essential features of mainstreaming, for the State of Sonora”. ous failings not only in its teacher-training equality and competitiveness”. and teacher-development programs, but “The aforesaid should manifest itself The challenges of information use also in its infrastructure, just to mention the in the 2016-2021 Sectoral Education Plan, “Now, with regard to the challenges posed most important ones, and hence the high which will orient our education system, and by evaluation,” says Pavlovich Arellano, a quality -attested to by various external eval- in which it will be essential for us to have one-time federal congressman, “it should be uations- that our state’s education was one a state-level evaluation system that enables stressed that we carried out a deep-reach- renowned for was eroded”. us to make timely decisions and provides ing, wide-ranging diagnosis of the educa- “On top of the above, the lack of infor- feedback about educational programs – i.e. tion sector both by system (input, process- mation about the Educational Reform and a dynamic, reliable system of benchmarks”. es and products) and by type and level of the dearth of clarity and transparency in the “It’s clear to us that, in order to once education, likewise evaluating the different way the lgspd and its processes were imple- more achieve the high-quality education policies and programs implemented by the mented led to a split among our teachers that we Sonorans’ were accustomed to, we’ll administration that preceded us. However, that had a strong impact on our society and need to need to ensure a minimal level of this was no easy task, since we found either impeded the achievement of the culture and normalcy in our state’s education system, no information and/or information that had employer-employee relationship that were give more administrative autonomy to our been tampered with, which made it hard for necessary for the signing of agreements and schools, albeit with full state-government us to find a clear, reliable starting point”. the implementation of strategies, including support, restart and analyze the training “For the above reason, it’s hard for us in the dissemination of information and the and professional-development programs for the current administration, and our succes- training of all those involved, to enable the teachers and other school staff in the context sors, to come up with timely, accurate infor- Reform to succeed. As a result, the general of the Educational Reform, work with the mation that enables us to take appropriate unease that prevailed among our teachers federal authorities to implement programs educational-policy decisions that provide delayed the implementation of these things for updating, maintaining and building edu- a basis for evaluating our performance, and led to chaos”. cational infrastructure and supplying equip- which is why we propose the relaunching “The lack of transparency in the use of ment and materials, and restart the support of a state-level evaluation system that can the public resources devoted to education and reward programs that made Sonora a work hand-in-hand with the inee to pro- led to the paralysis of educational programs leader in educational achievement and en- duce and disseminate input for the design and projects, especially at the upper second- abled it to reduce student lag and dropout and implantation of actions -above all those ary level. The decentralized subsystems that rates”. needed for our state to exercise the powers the state government received were suffer- assigned to for the purpose of carrying out ing from serious economic problems and, Strategies the Educational Reform – e.g. to suggest finally, a very worrying matter was the de- “Focusing on the aforementioned challeng- complementary profiles, norms and bench- cline in the educational-achievement results es,” says De Lucas Hopkins, “we’ve based our marks, to choose and train evaluators in of Sonora’s students”. actions on the following strategies: accordance with the pertinent inee guidelines, to punctually issue and broadcast pub- The design of the sectoral a) The transparent use of the public re- lic invitations to take part in teacher-selec- education program sources earmarked for education, a focus tion procedures, to implement the different “As can be seen”, says De Lucas Hopkins, on information systems that will enable evaluation processes in accordance with the who holds a master’s degree in Law and us to win back Sonorans’ trust in their General Law Governing the Professional Government from the Washington College authorities, and the cutting of red tape so Teaching Service (Spanish acronym: lgspd) of Law, “given the condition in which we re- as to simplify and reduce paperwork. 19

National Educational Evaluation Policy Gazette in Mexico

b) The dissemination of information about the Reform, it’s up to the state government results of the evaluation that showed Sonora both the Educational Reform and the to ensure the full exercise of the powers and to be the leader in educational achievement”. secondary laws and processes that form obligations that the lgspd confers on the “In this regard, I believe that one suc- part of it, based on the setting up of the local education authority for the purpose cessful endeavor that should be publicized State Office for the Coordination of the of ensuring the efficient, effective running and replicated is the New School Program Professional Teaching Service (Span- of the education system, and to this end that was designed based on the setting up ish acronym: (cnspd) and the Office for liaison and ongoing communication with of the State Evaluation System in 2004 in Liaison Between Sate Institutions and the aef and the inee are essential if we compliance with our state’s law governing the Federal Education Authorities and are to fully comply with the lgspd and the evaluation ant as result of the establish- the inee, the training of teachers so that guidelines that stem from it, or, failing this, ment of the inee. We need to preserve and they can duly and opportunely comply propose measures for perfecting and im- strengthen the essential features of the said with the different performance-evalu- proving education in our state based on the support program for school managers in ation processes, the encouragement of latter’s specific experiences and particulari- order for Sonora to once more become a people to take the evaluation for entry ties, without losing sight of the shared aim leader in educational quality”. to, and promotion within, the Profes- of strengthening both the local and national sional Teaching Service, ensuring that education systems and raising their quality”. its results are transparent, and the set- “We believe that, in order to raise edu- You can find out more about the Minis- ting up of support programs, including cational quality in our state, we need, among try of Education and Culture of the State scholarships and free school uniforms, other things, to: of Sonora at: http://www.sec-sonora. gob.mx/portal/index.php based on the transparent use of resources and the cutting of red tape. a) Save and strengthen the State-of-Sono- c) The strengthening of infrastructure, ra Institute for Educational Evaluation via the updating and maintenance of (Spanish acronym: ieees). schools and the provision of materials b) Set up a State Office for the Coordina- and furniture. tion of the Professional Teaching Ser- d) The setting up of a program to support vice. the analysis of learning-outcome evalu- c) Formulate government policy based on ation results, aimed at designing and the results of the Census of Elementary- implementing specific programs for level and Special-education Schools, each school and every group, based on Teachers and Students. mentoring and tutoring for students at d) As a matter of priority, set up a program risk of lagging behind or dropping out. for the training and professional devel- e) The raising of educational quality via a opment of elementary- and lower-sec- work program to help the schools that ondary-level teachers and school princi- got the lowest results in the Planea ex- pals. amination that the inee has been ad- e) Design educational programs based on ministering since the 2014-2015 school the results of the external evaluations”. year”. “Currently, we’re in the process of de- “Now high quality and equity,” De Lucas, signing strategies to reverse the Planea re- who was chairman of the Institutional Revo- sults, which constitute a big challenge, since lutionary Party (Spanish acronym pri) State Sonora placed 18th in Language and Com- Steering Committee for Sonora in 2008, munication, and next to last in Mathemat- goes on to say “are the foundations on which ics at the primary level, and 27th in both the our country’s -and hence our state’s- educa- aforementioned sections of the examination tion system should rest”. at the secondary level”. “The Sonoran Ministry of Education “In order to tackle this challenge, we believes that the monitoring of educational went about identifying the schools that got improvement should, as a matter of prior- the worst results, in order to come up with ity, be accompanied by the other things that strategies involving their principals, teach- enable students to achieve maximal learning ers, students and parents, while also not outcomes – i.e. teaching-learning materials forgetting to strengthen the schools that got and methods, organization, infrastructure, good results. Though the aforesaid strate- equipment, and hard work on the part of gies are still being implemented, we’re sure teachers and school principals”. that they’ll enable us to get better Planea “Since the national policy for raising results, since we have prior experience of educational quality is undeniably rooted in implementing support programs, based on 20

ENGLISH

OUR VOICE

Effective use for broad array of effects by multiple classes of improvement; what users” (Weiss, 1998). In practice, we have gradually pro- can we do to make it gressed from creating a small range of evaluation reports that were of limited relevance happen? or use to potential users, and merely con- centrated on complying with standards or satisfying requirements, to producing evaluations that increasingly take the said users into account, notwithstanding which we are “For the promise inherent in still very far from achieving optimal evalua- evaluation as an instrument for tions that take stock of all the relevant fac- improvement to be effectively tors and take stock of all the possible users. realized, it not only needs to be Hence, we still have a long way to go created in a thorough, fair and timely when it comes to fostering meaningful uses manner, but also must actually be of evaluation aimed at improving the education system. Often, as specialists in evalua- used by all those involved for the tion, we have mainly worried about the sci- clear, explicit purpose as serving entific quality of our work, and this has led as evidence to orient and inform us to focus on things such as the rigor of our decisions about improvement in each measuring tools and their analytical meth- area of practice”, says the author of ods, and the problems inherent in ensuring the following article, who proposes data integrity and security. Of course, these a tool for analyzing users within a things are our responsibility and constitute policy aimed at foster effective use. prerequisites for achieving improvement, but they do not suffice to bring it about. We must also view of policy as something au- Teresa Bracho González thentically public and seek to involve all the Member of the Board of the inee protagonists, at every level, in order to solve [email protected] problems. There are implicit beliefs or suppositions about the use of information stemming from educational evaluation, which, if we examine them from viewpoints such as y effective uses of evaluation, one means that of public policymaking, could tell us deliberate actions implemented by the some interesting, relevant things about the Bdifferent recipients or users of information successes and shortfalls of our work as eval- (i.e. protagonists), who -to different degrees uators of the education system so far. and for different purposes- use the results, For our evaluation to truly become a fac- methods or ideas derived from educational tor in the improvement of the said system, it evaluation as relevant input on which to invariably needs to be used at all levels and base decisions and actions aimed at bring- by all the protagonists in their particular ing about improvement in their particular spheres of action. But how do we ensure that spheres of action. this happens? Do we need to do something In its early stages, in the late 1960’s, it different in order to ensure that evaluation was considered that the sole function of is really useful? In my opinion, we evaluators evaluation, and of the information stem- need to change our viewpoints and practices a) Considering the topic of the uses of ming from it, was to impact the particular in at least the following three ways: we need evaluation starting from the latter’s program evaluated. However, as Carol Weiss to (a) take stock of the uses of evaluation conception – i.e. at Stage Zero explains, we are now aware that evaluation starting at the planning stage (what I call This topic must be put on the agenda – i.e. it has other dimensions, and that many other stage zero), (b) transcend the traditional out- must be one of the main concerns of evalu- aspects of it besides its results -e.g. its ideas look that underlies our behavior as evalua- ators from the moment when they start to and visions, theoretical frameworks, meth- tors, and (c) address the subject of the uses design an evaluation, which means chang- odological procedures and even its design- of evaluation aimed at “intervening in public ing the two typical practices. First we must can also be used, since “use encompasses a policy”, by: cease thinking about the uses of evaluations 21

National Educational Evaluation Policy Gazette in Mexico

after they have been completed, only then Some of the aforesaid assumptions may Tool for analyzing users as part of a -and not before- deciding which types of have worked when evaluation was seen as policy aimed at fostering effective use. reports, publications and methods of dis- being limited to the function of account- As already affirmed, it is essential that po- semination to use, since this way of thinking ability, providing information to top officials tential users be considered, and all the more leads us to lose an opportunity that cannot about the overall progress of the education so in the case of educational evaluation. If always be recovered later, given that it may systems they administered. They may have we wish to inform policymaking in its dif- not be possible to adapt the data gathered been reasonable when respect for -and trust ferent areas, and at its different levels, we during the evaluation process to suit uses in- external evaluation, and the incursion of must produce evidence expressed in the that were not envisaged at the design stage. professional evaluation specialists into pub- languages and forms that are appropriate for Second, we must stop believing that our job lic affairs, had to be engendered. However, the users at different levels, taking into ac- has ended when the results are published now that we have changed, and enriched, count the said people’s actual ability to take and disseminated, assuming that the uses our way of looking at education systems, on board, interpret and use the evidence in needed to change the system at its different and in a context where access to informa- question within their respective spheres of levels will arise spontaneously as a result of tion has become a right per se, any evalua- action. interest, understanding and assimilation on tion practice that mainly rests on the afore- Above all, it is essential that those who the part of the various protagonists who are said, or similar, assumptions prevents us produce the evidence -i.e. the evaluators- be interested in the workings of the education from achieving the longed-for link between aware that the aforesaid consideration of the system. In short, I propose that communi- evaluation and educational improvement, characteristics and needs of the target us- cation about evaluation extend beyond the and should be eschewed. ers must be the starting point for designing publication and dissemination that have tra- strategies capable of producing results that ditionally been carried out in Mexico. c) Seeing the use of evaluation as an can foster uses that lead to improvements in intervention in government policy our education system. b) Changing the “traditional” outlook Once it has been accepted that use should As a first step, I propose using the “ob- that underlies our behavior as be taken into account starting at the plan- servation areas” (Fig. 1) customarily used by evaluators of the National Education ning stage of evaluation, and we have ad- researchers to measure education-service System (Spanish acronym: sen) justed our outlook to match our practice, quality, which are clearly summarized by Linked to point a) above, one can assert the other important point is that we need to the Latin-American Laboratory for the As- that, precisely due to the specialized nature conceive of the use of evaluation as a form sessment of Educational Quality (Spanish of our task, we have fallen prey to a set of fal- of intervention in government policy, rather acronym: llece), an initiative coordinated lacies that we have not explicitly addressed than a matter of haphazard, spontaneous re- by unesco, and used by the latter for its in the course of our work, and hence not sponse to the interests of the various users achievement tests (i.e. the First Regional talked about enough. If we want to foster the of the evaluation data. Though those of us Comparative and Explanatory Study (Span- actual use of evaluation for purposes of im- who form the National Educational Evalua- ish acronym: perce), the Second Regional provement, we must, at the very least, question System (Spanish acronym: SNEE) need Comparative and Explanatory Study (Span- tion the implicit assumptions that: to have a broader discussion about the mat- ish acronym: serce), and the Third Regional ter, I propose at least two topics to be cov- Comparative and Explanatory Study (Span- 1. the important thing is to produce high- ered during the said debate. ish acronym: perce). quality evaluations, with our main chal- First, if we see the matter of use as a Then, as a second step, I propose a “mir- lenge as evaluators being to create in- policy in itself, rather than a spontaneous, ror” analysis (Fig. 2) taking the same agents formation of the required quality, which fortuitous consequence of evaluation, as and protagonists as in Fig.1 -i.e. students, can be achieved by developing highly already mentioned, then the non-use or re- teachers, school principals and authorities- rigorous and scientific instruments. jection of evaluation findings, or the failure as the potential initial users of any evalua- 2. publication and dissemination ensure to understand them, constitute the specific tion of educational quality (not forgetting access and divulgation, since the mere problem that the said policy should solve, that this second analysis is based on an eval- publication and dissemination of sci- and second, as is the case with all suggested uation of achievement, since, if the aforesaid entifically controlled evaluation results policies, in order to be deemed to constitute people are the ones who play a part in de- makes them accessible and understand- public ones, those we are talking about here termining quality, being able to take steps to able to those involved in the sen. require rigorous, participatory analysis of improve classroom education, they are also 3. users are fully able to understand the their potential usefulness and of the needs the ones to whom we should primarily ad- information that is produced, since edu- and expectations of all the members of the dress our results and keep foremost in our cational evaluations are mainly designed sen who are able to play a part in the lat- minds when carrying out any evaluation. to be used by specialists. ter’s improvement, to which end I go on to Though simple, this concept can enrich 4. due to his/her training and experience, propose a simple tool -one of many pos- the first approach at defining the products the evaluator is the one who knows best sible ones- that can be used to structure the that spring from evaluation results. How- what should be evaluated, what can be aforesaid processes of needs identification ever, since they should be defined in the evaluated, and how to do so. and categorization, dialogue and building of context of the National Educational Evalua- meaning related to evaluations. tion System (Spanish acronym: snee), I have 22

ENGLISH

inserted question marks in the four sections Fig. 1: Observation areas used in educational research of Fig. 2. As a matter of priority, we need to define the type of evaluations to be used Student Teacher and determine how the information yielded Family context Teaching environment by them can be utilized to foster high educational quality via consultations with the main potential users, among other things. Commitments, habits Curriculum and It should be stressed that this matter is and skills teaching-learning of the utmost importance strategically, es- management pecially in the context of the reforms currently underway in the sen, since, in order to exploit the potential of evaluation as a tool for improvement, the said evaluation Quality of student must not only be produced in a rigorous, fair learning outcomes and timely manner, but also actually used by all the players involved -each at his/her/its respective level- for the clear, explicit purpose of providing guidance so that the said Institutional Political and players can make decisions aimed at achiev- management institutional management ing improvement in their respective spheres of action. The use of educational evaluation -a topic that differs from, but is closely relat- School principal Government authorities ed to, that of the creation or production of School microcosm Educational macrocosm evaluation- is a subject that merits research in its own right, since the effective uses that Source: Laboratorio Latinoamericano de Evaluación de la Calidad de la Educa- are desired -i.e. ones that lead to improve- ción (llece), 2000:12. ments in the education system- will not occur spontaneously or automatically to the different existing or potential recipients of Fig. 2: Tool for identifying the potential users of evaluations in an evaluation findings. We on the supply side effective-use policy; protagonists in their contexts have some very important leeway in the endeavor to foster greater, more thoughtful demand and thus transform desired uses into Student Teacher improvement strategies. It is time for us to Family context Teaching environment embark on this task.

References Latin American Laboratory for the Assessment of ¿? ¿? the Quality of Education, llece. (2000). First international comparative study of language, mathematics, and associated factors for students in the third and fourth grade of primary Fostering of school. Second report. Santiago de Chile: high-quality student unesco. Available at: http://unesdoc.unesco. learning outcomes org/images/0012/001231/123143s.pdf Weiss, C. H. (1998). Have We Learned Anything New About the Use of Evaluation? American Journal of Evaluation, No. 1, pp. 21-33. ¿? ¿?

School principal Government authorities School microcosm Educational macrocosm

Source: Author-producer based on llece 2000:12. 23

National Educational Evaluation Policy Gazette in Mexico

NAUTICAL LETTER

A National Educational Evaluation Policy for Promoting Improvement

Topic: A National Educational Evaluation lenges inherent in the evaluation of the Na- members and make it possible to coordi- System (Spanish acronym: snee) that helps tional Education System (Spanish acronym: nate efforts, at both the national and local to improve educational quality in Mexico sen), and hence to explicitly help to build a levels, to establish the aims of evaluation shared, interlinked pnee agenda. and monitor its short- and medium-term The efforts made by the inee’s in 2015 progress1. Main aim: To give a brief referential over- to carry forward the building of the pnee In this regard, the pnee constitutes an view of the creation and current status of focused on (i) analysis and open discus- effort to give educational evaluation a big- the snee, and the outlook for the latter, that sion with those involved in the snee about ger role as an interface between the said will be of use to policy-makers and serve to the appropriateness of making a right-to- evaluation and the use of its results to im- plot out the path to be followed with re- high-quality-education emphasis the pnee’s prove education. gard to educational evaluation, based on guiding principle; (ii) the definition of the It should be stressed that, when refer- the Governing Document of the National mainstays of the pnee; (iii) the stipulation ring to the snee, we mean the three main Educational-Evaluation System (snee-gd). of the purviews and responsibilities of each components that it will embody – i.e. (i) of the authorities that make up the snee the snee’s Governing Document2; (ii) the based on a legal analysis of the powers and State-level Evaluation and Educational- 1. The snee-gd as a starting point competencies of the different entities in- Improvement Programs (Spanish acronym: Arising from reflection, the exchange of volved in educational evaluation; and (iv) peeme)3, and y (iii) the snee Medium-Term ideas and the reaching of consensus, the discussion, in the Conference, about the Program4. snee-gd provides a long-term panorama snee’s Governing Document with the fed- that both sets out to make good on the eral and state-level education authorities. 3. How can the pnee be used right to education and constitutes a new to improve education? agreement between the inee and the fed- The pnee will both ensure that the evalu- eral and local education authorities. 2. The advantages of having a National ations carried out throughout Mexico, at Since 2014, we have been holding the Educational Evaluation Policy both the federal and local levels, are objec- Dialogues with the Educational Authori- The pnee is envisaged as a tool for bring- tive, transparent, accurate and unbiased, ties, intended to be a forum for direct con- ing about educational improvement that and also make it possible to create institu- tact and high-level discussion between the will apportion the different evaluation tasks tional channels for coordination and coop- inee’s Board of Governors and the educa- among the members of the snee and guide eration that make the government actions tion authorities about the implementation their efforts, clarify which legal powers undertaken in the context of the snee more of the Educational Reform and the chal- have been conferred on each of the said effective, leading to better liaison between 24

ENGLISH

Diagram 1. Components of the pnee the Areas Responsible for Evaluation (Span- ish acronym: are’s) and the federal and state-level authorities, and also between the said are’s and other are’s, Higher Edu- cation Institutions, international organizations and ngo’s, thus furthering the carry- pneegd peeme snee mtp ing out of their projects and activities. It will also produce institutional and intervention mechanisms for improving state- level education services via the use of evalu- National Educational-Evaluation Policy ation results, making it possible to come up with improvement strategies mainly based on the said results. Finally, through the guidelines, the pnee will provide guidance which, while not breaking the law, will make it easier to comply with Medium-term snee Program quality of compulsory education and make it the legal mandates, informing educational Stemming from a review, analysis and in- more equitable via cooperation among the policy-making in order to raise the quality tegration process, the Medium-term snee protagonists and institutions involved in the of education. It should be stressed that the Program (snee-mtp) will put together the snee. said guidance can be given at the request of basic components of the snee developed by 3 This planning tool for analyzing and orienting the education authorities. each state, which will indicate which actions educational evaluation and improvement in are to be taken at the national and state lev- the states consists of an institutional program 4. What will the pnee’s next els respectively, as well as stipulating the that will facilitate the setting up of projects, step be in 2016? activities, deadlines, people in charge, and aims and actions pertaining to evaluation and goals pertaining to the federal and local educational improvement and goals, organiz- State-level Evaluation and Educational- education authorities and the inee. ing one or more evaluation projects that the improvement Programs You can find the full version of the Docu- state deems pertinent based on its diagnostic The peeme’s mentioned above will be de- ment Governing the National Educational- study. signed by the states with the inee’s help, Evaluation at: http://goo.gl/YMjkKP 4 Will set out (i) to define the legal, conceptual and this task requires collective work on the and methodological foundations and also the part of the state-level education authori- 1 Projected to 2020. implementation paths for carrying out the ties responsible for both elementary and 2 Includes the central evaluation-policy tenets evaluation processes defined in the snee, and lower-secondary education in order to put that orient and organize the work path, lines (ii) to establish the objectives, lines of action together a single state-level program that of action and perspective for 2020 and ad- and calendar for such evaluation processes as covers both levels. dress the constitutional obligation to raise the are eventually defined.

2) Roadmap: stages between January and July, 2016

1. Diagnostic study 2. Identification of problems Note: The inee will provide feedback at each stage 3. Projects and aims and the stage education authorities will make the per- 4. Actions tinent adjustments. 5. Schedules and benchmarks 6. Goals Would you like to learn more about the pnee? Look for the special supplement of this Gazet- te: "The snee and the National Educational Eva- luation Policy: Challenges and Perspectives". 25

National Educational Evaluation Policy Gazette in Mexico Roadmap for developing the 2016 PEEME’s 1) The regions

Northeast Baja California Coahuila, Durango, Nuevo León, Sonora San Luis Potosí, Tamaulipas Chihuahua

Coahuila South-Southheast Baja California Sur Nuevo Campeche, Chiapas, Guerrero, Oaxaca, León Durango Quintana Roo, Tabasco, Veracruz, Yucatán Sinaloa Northwest Zacatecas Baja California, Tamaulipas

Baja California Sur, Querétaro Nayarit San Luis Potosí Chihuahua, Sonora, Sinaloa Hidalgo Aguascalientes Yucatán Tlaxcala Jalisco Quintana Roo Puebla Guanajuato Veracruz Colima Michoacán Campeche Tabasco Estado de México Guerrero West Ciudad de México Chiapas Aguascalientes, Colima, Oaxaca Guanajuato, Jalisco, Michoacán, Cuernavaca Nayarit, Querétaro, Zacatecas Center Ciudad de México, Estado de México, Hidalgo, Morelos, Puebla, Tlaxcala

3) Calendar

Activities and deliverables Date Activities and deliverables Date First version of the Diagnostic Study January First version of planned Actions Feedback on planned Actions from the inee May Feedback on the Diagnostic Study from the inee February Adjustments to planned Actions Adjustments to the Diagnostic Study First version of Aims Regional meeting: first stage of peeme February Feedback on Aims from the inee June First version of the List of Problems Adjustments to Aims Feedback on the List of Problems from the inee March Regional meeting: third stage of the peeme June Adjustments to the List of Problems First version of Schedules and Benchmarks First version of Projects and Aims Feedback on Projects and Aims from the inee April Feedback on Schedules and Benchmarks from the inee July Adjustments to the Projects and Aims Adjustments to Schedules and Benchmarks

Regional meeting: second stage of peeme April Third Meeting of the inee-sep peeme Development July Second meeting of the inee-sep peeme Development Committee May Committee Fourth Meeting of the inee-sep peeme Development October Committee 26

ENGLISH

ROADMAP

The meaning and use, endeavors to measure students’ mastery of a set of essential learnings at different stages for the improvement of the compulsory-education process, the of Education, of the main results of both of them, which are outlined in this article, show aspects of our results of the Evaluation country’s educational status quo that are of Basic Teaching- closely interlinked with each other. After first presenting the main Planea results re- Learning Conditions* garding learning outcomes in the sixth year and the Evaluation of of primary school, we will go on to outline the basic conditions in Mexican primary Learning Outcomes in schools shown by the ecea results, and, Elementary Education** finally, we will venture some observations about the overlaps between the said two studies and the uses to which the information yielded by them can be put. Jorge Antonio Hernández Uralde Head of the National Education System’s Learning outcomes at the primary level Evaluation Unit (Spanish acronym: uesen) Planea was administered by the inee, at of the inee the end of the 2014-2015 school year, to a [email protected] representative sample of sixth-year primary-school pupils that was used as a basis Andrés Eduardo Sánchez Moguel for extrapolating the countrywide results General Director for the Evaluation that we will comment on below. The Min- of Educational Results of the National istry of Public Education (Spanish acronym: Education System’s Evaluation Unit sep) administered it in all the other prima- of the inee ry schools in Mexico so as to furnish each [email protected] school with a report on the results achieved by its pupils, duly contextualized. The tests Raquel Ahuja Sánchez that make up the said exam cover develop- General Prospectus Director for the mental areas on which many other types of National Education System’s Evaluation learning are based, such as Language and Unit of the inee Communication and Mathematical Reason- [email protected] ing, and which hence constitute two areas of the curriculum that serve as yardsticks or proxies for other areas, it being difficult, Introduction for example, to imagine students perform- In late 2015, the National Institute for the ing well in the sciences if their linguistic and Evaluation of Education (Spanish acronym: mathematical skills are weak. inee) published the preliminary results of Generally speaking, the Planea results at the Evaluation of Basic Teaching-Learning the primary level correlate with the low per- Conditions (Spanish acronym: ecea) and formance levels in the National Education the Evaluation of Learning Outcomes in El- System that previous evaluations such as ementary Education that forms part of the the Examination of Educational Quality and National Plan for the Evaluation of Learning Achievement (Spanish acronym: excale) Outcomes (Spanish acronym: Planea). Al- and the National Examination of Academic though the said evaluations have different, Achievement in Schools (Spanish acronym: independent, purposes, since ecea sets out enlace) had already been indicating. to ascertain the extent to which Mexico’s Almost half (49.5%) of the students compulsory-education institutions have the placed in Achievement Level 1 in Language basic wherewithal to function, while Planea and Communication, meaning that they face serious limitations when endeavoring * Spanish acronym: ecea to carry out the basic language tasks such as * * Spanish acronym: Planea understanding expository1 and literary texts 27

National Educational Evaluation Policy Gazette in Mexico

that they are required to master as part of Graph 1. Percentage of students per achievement level in Language and Communication, the primary-school curriculum, and com- nationwide and for each type of school: sixth grade of primary school, 2015 paring the communicative aims of different types of text, with only a third (33.2%) of the students who placed in Level 2 -pertaining to an indispensable minimal mastery of key Nationwide 49.5 33.2 14.6 2.6 curricular learnings- being able to carry out the said tasks. 3.7 Around 15% of sixth-year primary- Indigenous 80.0 16.0 0.3** school pupils placed in Achievement Level 4.6 5.4** 3, meaning that they can carry out inferential Community 67.9 26.7 tasks such as understanding a metaphor in General public 51.6 34.1 12.7 1.6 a story, writing topic sentences that express the main idea of a text and summarize the Private 13.3 33.9 38.9 13.9 author’s intentions, and identifying data, arguments and opinions, while only 3% of the Source: uesen inee said pupils are in achievement level 4, which N1 N2 N3 N4 means that they can deploy top-level cognitive skills to understand argumentative texts, deduce how an interview is structured, etc.. Just over 60% of students placed in Level 1 for Mathematical Reasoning, meaning Graph 2. Percentage of students per achievement level in Mathematical Reasoning, that they still have not acquired the funda- nationwide and for each type of school. Sixth grade of primary school, 2015 mental abilities pertaining to the said, such as the capacity to carry out basic operations (i.e. addition, subtraction, multiplication Nationwide 60.5 18.9 13.8 6.8 and division) using natural numbers, iden- 5.8 tify geometrical features such as height, par- allelism and right angles in simple shapes, or Indigenous 83.3 9.7 1.2 solve problems involving perimeters, while Community 69.2 17.2 10.3* 3.3* 19% of all sixth-year pupils placed in Level 2, meaning that can indeed perform the afore- General public 62.8 19.0 12.9 5.4 said tasks. Around 14% placed in Level 3 and Private 25.9 23.0 27.4 23.8 are also able to solve mathematical problems involving decimals, compare fractions and multiply them by natural numbers, and identify geometric progressions using rules, N1 N2 N3 N4 Source: uesen inee etc., while only 7% of the students who complete the sixth year of primary school managed to place in Level 4, meaning that, as well as having the aforesaid competencies, they can solve addition problems using frac- Graph 3. Percentage of students per achievement level in Language and Communication according to the level of marginalization. Sixth grade of primary school, 2015 tions, divide and multiply fractional numbers by natural numbers, calculate areas and work out averages and means. In short, a very significant portion of the students who complete primary school Nationwide 49.5 33.2 14.6 2.6 in our country is not achieving satisfactory learning outcomes. By splitting the student population’s learning outcomes into four Very high – high 61.8 28.5 8.8 0.9 levels, we can perceive the most significant differences among students who attend dif- Medium 49.2 34.7 14.0 2.1 Marginaliztion ferent types of school, as shown in Graphs Very low – low 34.2 37.9 12.6 5.3 1 and 2 below, which depict the said differences in the respective fields of Language and Communication and Mathematical Reasoning. As can be seen in these graphs, N1 N2 N3 N4 Source: uesen inee 28

ENGLISH

the types of school located in the most vul- and students from the 4th, 5th and 6th grades schools possessing at least one of the afore- nerable areas -i.e. indigenous and commu- and a representative from the board of the said facilities, but only 5% having all of them nity schools- have much lower achievement Parents’ Association, acted as informants. (See Graph 4). Almost none of our country’s levels than other public and private schools, Below, we present some of the results that public schools has all three of the afore- with 8 out of every 10 pupils in indigenous indicate differences in the educational con- said educational-support areas, all three schools and 7 out of every 10 pupils in com- ditions of Mexican students, and show that of which are mostly found in the general munity schools placing in achievement level the biggest shortfalls were in schools located single-level schools, of which 2.6% possess 1 in both Language and Communication in vulnerable contexts.5 them, as compared with 35% of all the pri- and Mathematical Reasoning. To support learning and the coverage vate schools. Two out of every three indige- This tendency for the most vulnerable of the curriculum, the schools have librar- nous schools have none of these three areas, populations to achieve the worst learning ies, media rooms and multiple-use rooms as while more than half (57.2%) of the general outcomes also becomes evident when one well as classrooms, with 60% of all Mexican multi-level schools, and over a third (37.1%) analyzes the distribution of achievement levels taking stock of the levels of margin- Graph 4. Percentages of schools with 1, 2 and 3 educational-support areas respectively:6 alization of the schools’ locations. Graph nationwide and by type of school7 3 below, which shows the distribution of achievement levels in Language and Com- munication according to local marginaliza- Informante: Director tion levels, shows that those locations with high or very high levels of marginalization 0.3 0.1 0.8 2.2 0.0 2.6 have the lowest achievement levels, while 100 the highest achievement levels are found in 5.3 7.6 those locations with low or very low levels of marginalization, with a similar trend being 17.1 22.8 80 32.8 29.0 37.5 found for Mathematical Reasoning.2 All the aforesaid data indicate that the in- 35.1 equality in learning outcomes is largely due to the way the education system Works, of- 60 35.0 fering different types of service to different 37.5 populations and, as far as can be see, helping 29.2 to aggravate social inequality rather than di- 40 minishing it. Some of the results showing this 66.0 68.6 other side of the coin are presented below. 57.2 20 42.6 Basic conditions in primary schools 37.1 25.8 ecea, an evaluation of operating conditions in schools that is carried out every four 7.5 years, measures the extent to which the said 0 institutions have the basic (not necessarily Nationwide Multi-level Single-level General General Private optimal) resources and processes for their indigenous indigenous multi-level single-level pupils to learn, the said basic wherewithal Type of school being those that every school, and all Mexi- can students, should have – i.e. physical With three areas With one areas resources such as infrastructure, furniture With two areas No area Source: uesen inee and teaching materials, teachers and management staff who work collaboratively, parent participation, effective use of time, and conviviality among pupils.3 of the general multi-level schools lack them. and seat where they can sit and write in good ecea was first administered in2014 to a While 81% of all Mexican state schools or fair condition, but again this is more the representative sample of our country’s pri- have the student-only bathrooms that are case in the private schools (99.5%) and the mary schools that comprised six types of essential for security and privacy (Graph 5), community schools (82%), where the Na- institutions – i.e. general single-level public this figure decreases in line with the degree tional Council for the Fostering of Education primary schools, general multi-level public of marginalization, dropping to just 44.4% in (Spanish acronym: conafe) is directly re- primary schools, single-level indigenous the community schools. sponsible for school equipment. Next come schools, multi-level indigenous schools, Classroom furniture is a basic part of the general schools (68.3% to 78.3%) and fi- community schools and private schools.4 school equipment. Nationwide, all the chil- nally the indigenous schools (53.5% to 59.2%) The principals of each school, plus teachers dren in 78.2% of all the groups have a desk [Graph 6]. 29

National Educational Evaluation Policy Gazette in Mexico

It can be seen that basic educational Graph 5. Percentages of schools with working toilets exclusively for students: materials such as free textbooks (FT’s) nationwide and by type of school are not opportunely distributed in suffi- cient quantities in all the different types of Informante: Director / lec school. Though80 % of the single-level general schools and the private schools received 100 FT’s before the start of the school year, the figure for general multilevel and indigenous schools is around 70%, and just over half (51.7%) for community schools, it being im- 80 44.4% portant to stress that, almost three months into the 2014-2015 school year, a significant 55.9% number of community schools (9.4%) re- 69.5% 60 ported that had still not received any FT’s. 81.3% 81.6% Moreover, three months after starting class- 92.8% 97.1% es, not all the children had a complete set of FT’s, as can be seen in the following table. 40 The results presented here are just an example of the inequality in children’s study conditions. The education system needs, at 55.6% the very least, to ensure that there is a gener- 20 44.1% al shared base level for all schools and hence 30.5% all students, though, from an equality view- 18.7% 18.4% 7.2% point, the schools and children in the poorer 0 2.9% locations should receive not just the same, Nationwide Community Multi-level Single-level General General Private but more, in order to be able to achieve their indigenous indigenous multi-level single-level learning targets. Type of school Some final thoughts Learning-outcome levels are also lower With student-only toilets Without student-only toilets Source: uesen inee in the types of school that function in the most precarious conditions and hence offer less wellbeing and learning opportunities to their students, who come from the poorest environments, as reflected in their results Graph 6. Percentages of schools whose students have a desk and seat and in the inequalities among the courses where they can sit and write: nationwide and by type of school offered. The schools in these conditions face a bigger challenge than the, both because Informante: Director / lec they cater to impoverished populations, and 100 also because they offer less favorable study conditions. Both ecea and Planea clearly show that 80 we urgently need to improve the educational services provided to the most vulnerable populations, as a matter of priority, aiming 60 to ensure not only that all our children attend school, but also that they all learn. In 99.5% 40 78.2% 82.0% 78.3% this regard, it is germane to ask the authori- 68.3% ties responsible for managing primary edu- 59.2% cation, to respond, with the participation of 53.5% 20 state-level ministries of education, groups of researchers, teachers, parents and other people involved in education, to the follow- 0 ing questions: (i) How can we can ensure that Nationwide Community Multi-level Single-level General General Private all students achieve the learning outcomes indigenous indigenous multi-level single-level that are indispensable at the primary level? Type of school (ii) Should the curriculum be structured Source: uesen inee 30

ENGLISH

3 More information about evaluation can be Table 1: Percentages of schools that received Free Textbooks before the start of the 2014-2015 found at: http://www.inee.edu.mx/index.php/ school year or at the start thereof and percentages of groups in which the students had a full proyectos/ecea set of FT’s in November, 2014. 4 It should be stressed that most of the multi- % of schools that received Free % of groups in which the level general schools -i.e. schools where a single Textbooks before the start of the students had a full set of FT’s teacher handles groups whose students are at Type of school 2014-2015 school year or at the (November, 2014). different levels- and the indigenous and commu- start thereof nity schools are situated in rural locations with high or very high levels of marginalization, while Community 51.7% 30.3% the general single-level schools and the private schools are located in urban areas with medium, Multi-level 70.9% 48.3% low or very low marginalization levels. indigenous 5 The results were obtained via self-administered Sincle-level questionnaires plus a personal interview with 67.0% 43.1% indigenous the Parents’ Association representative. 6 Educational-support areas, including libraries, General multi-level 72.6% 54.5% multiple-use rooms and media rooms. 7 Community schools are not included in the General single-level 80.6% 64.9% graph. 8 The basic operating framework for primary Private 80.0% 77.7% schools that was used in ecea 2014 can be consulted at: http://www.inee.edu.mx/index.php/ proyectos/ecea Fuente: uesen inee

ecea: http://www.inee.edu.mx/ index.php/proyectos/ecea around essential common primary-level mary level in order to promote their use for learning aims and others that adjust to dif- purposes of improving our schools. ferent contexts? (iii) How can we strengthen For its part, ecea uses a basic school- Planea: http://www.inee.edu.mx/ index.php/planea initial and in-service training for those who operation framework that includes 68 condi- teach more needy populations? (iv) How can tions split into 7 fields and 21 dimensions, we ensure that all the schools belonging to as well as 160 benchmarks that can be ana- the National Education System comply with lyzed by school communities and education basic operating norms and offer the same authorities at the school-zone, sector-head, learning opportunities to all? (v) How do we and state-education-authority levels, etc. for establish special incentives for teachers fac- the purpose of evaluating school conditions ing higher challenge levels? and (vi) How do and planning actions to improve them.8 we reconcile the need to use resources effi- In this article, I have tried to how the ciently with the fact that the universal right evaluation results can provide information to high-quality education implies affirmative that is pertinent to different protagonists action that requires us to spend more on the and inform decision-making in different most vulnerable populations? spheres, making different requirements of We still need to ascertain the extent each of them. Evaluation is useful to the ex- to which the ecea and Planea results are tent that the responsible authorities ponder known, and used, by school communities. its results and use them to make decisions. The implementation of Planea2015 includ- Those wishing to find out more about ed the gathering of information at both the the evaluations mentioned in this article national and state levels that was published should visit the following microsites via the by the inee, and also of information about inee portal. each school, which was published by the Ministry of Education so that the School 1 Texts whose main aim is to transmit informa- Technical Committees and the School So- tion and disseminate knowledge. cial-participation Committees might ana- 2 It is not possible to include all the information lyze it and reflect on local strengths and op- that is available about the sixth-year-primary portunities. Undoubtedly, we need to make Planea results for 2015 in this article. The com- several adaptations and improvements in plete results can be consulted at: http://www. the dissemination of the results at the pri- inee.edu.mx/index.php/planea 31

National Educational Evaluation Policy Gazette in Mexico

ROADMAP

The inee guidelines: The Educational Reform proclaimed in thereof, take too long to serve political pur- building bridges December of 2012 conferred autonomy on poses, so that decision makers, faced with the said Institute and empowered it to issue the need to act and respond rapidly, use between evaluation, policy recommendations -i.e. “guidelines”- whatever available information helps them aimed at informing decision-making about to underprop their decisions. uses and educational educational policy and thus serving as a Other factors that determine whether improvement bridge between evaluation and educational research and/or evaluation will be used are improvement. the profile of the researchers/evaluators, the In this article, which contains some re- prestige of the institution to which the lat- flections about how to create the aforesaid ter belong, and the type of research and/or inee guidelines and about their potential evaluation, added to which is the fact that Theinee issued its first guidelines for influencing decisions aimed at improve- quantitative studies still enjoy more prestige in 2015, after being empowered ment, we first summarize part of the debate than qualitative or meta-analytical ones, to do so by the 2012 Educational regarding the things that prevent evaluation and that overview studies still enjoy more Reform, which links evaluation from being used and transformed into gov- prestige than studies on a single topic - a to improvement. The author of ernment policy, and then go on to mention phenomenon that is also linked to the cur- the following article gives a brief two key aspects of the drawing up of guide- rently prevailing paradigms. lines that seek, precisely, to achieve such summary of the road that the transformation – namely, the participatory aforesaid Institute mapped out for defining of the problem being addressed and the creation of the said guidelines, the possible solutions to it, and the weighing stressing both the key role played of the scope and shortfalls of government by policy and program evaluations action at the state level. aimed at remedying the problem, shortfall or weakness that is being Difficulties inherent in the research- addressed, and also emphasizing evaluation pairing and its use the need to hear the viewpoints of Carol Weiss holds that, though the linear different protagonists. model whereby pure and applied research are turned into policy presupposes the existence of consensus between researchers and policymakers about the aims being pursued, nothing could be further from the truth, Arcelia Martínez Bordón since research and policymaking run par- General Director of Guidelines for allel to each other, occasionally coinciding, Educational Improvement but only tangentially, and, while academic Regulation and Educational-policy debate helps to establish the paradigms for Unit of the inee the discourse frameworks that govern polit- [email protected] ical action, it rarely defines the public-policy agenda. While it is assumed that, if research and evaluation comply with certain standards, ll government policies are responses by the rest is history, in fact, though playing a the State to social problems or to situa- key role in informing decision-making, the Ations that are deemed unsatisfactory or un- high quality of the said research and evalu- acceptable (Subirats, Knoepfel, Larrue and ation is no guarantee that they will be taken Varone, 2008). Drawn up in many ways, but stock of by some user groups, since other resulting, above all, from pressure applied things, such as political negotiation and by different protagonists for the existence of consensus seeking and forming, play a big- a social problem to be acknowledged, policy ger role in the use of evaluations (Moreles, agendas may also receive input from evalu- 2010). ations and research, and, indeed, the inee’s Another important factor preventing remain aim is that educational evaluation help search and evaluation from being used is the inform decision-making about how to raise fact that, in most cases, the said research or the quality of education. evaluation, and publication of the findings 32

ENGLISH

In reality, an evaluation-and-follow-up inee por las autoridades educativas [The ing, evaluation must also be comprehensive system will only be successful if the evalu- use of inee evaluation results by the educa- enough to enable people to see the wood ations it produces are used,1 though the in- tion authorities] (Silva, 2012), is very reveal- for the trees. However, this is no easy task, formation that the latter yield will not neces- ing, since the people involved in education since educational evaluation in Mexico has sarily be utilized. As already mentioned, the who were interviewed for the said study not progressed at the same pace vis-à-vis the fact that an evaluation is well designed and expressed very divergent opinions about the different components, processes and results of a high quality does not guarantee that it three topics covered – namely, familiarity of the National Education System (Spanish will be used as a basis for decision-making, with the information produced by the inee, acronym: sen), or in terms of the different and hence many evaluations and studies end the importance assigned to the said infor- evaluation types and methods that are used up sitting on the shelf. mation, and the latter’s use in, and impact and their aims, being unable to see the wood Another difficulty is the lack of dis- on, decision-making. Thus, for example, because it has focused more on student and semination strategies. Some evaluations are while most of the education officials inter- teacher components than on educational very technical and not easily understood by viewed at the state level knew that the inee policies and programs, syllabi, methods, those who might use them to make deci- existed, only one fifth of them reported that materials and schools. sions, while others reach conclusions -and they were familiar with the information that Now that it is autonomous, the inee is make recommendations- that are unfeasible it produces, in marked contrast to those in- addressing the abovementioned concern by or minimally relevant, given which we argue terviewed at the federal level, most of whom developing new evaluations, notwithstand- that the former must satisfy certain minimal said that they were familiar with, and had ac- ing which the paving stones of the road lead- requirements, such as the need to provide cess to, the said information. With regard to ing to a comprehensive evaluation that take timely, relevant, high-quality information, if the use of, and value assigned to, the afore- stock of the sen’s different components are they are to be used, added to which various mentioned information, the study reveals still being laid. key role players -who “own” them, agree that that the information produced by the inee they are useful and relevant, and promote is predominantly used for discussion or con- The guidelines: a new tool their utilization- must take part in their de- sultation – what Silva (2012:101) refers to as As already remarked, the inee guidelines, sign, implementation and dissemination. “using the data to build a discourse”: In- which are recommendations aimed at indeed, the said study (p. 102) concludes that forming decision-making about educational Education evaluation and the raising the “inee’s influence on the political process policy, establish specific goals and refer to of quality is limited” and also depends on how much key aspects of educational improvement Given the above, it is clear that there is no people know about the evaluations, how based on the identification of the main direct relationship between educational well they understand them, and how they problems, weaknesses or risks that must be evaluation and decision-making, and, hence, interpret them. addressed to fill a specific gap in the educa- perhaps an even less direct relationship be- This limited use of the results of the dif- tion system or remedy a particular shortfall tween evaluation and the raising of quality. ferent types of evaluation -not just of those in it. Evaluation may be formative, aim- carried out by the inee, but also those ad- Their validity is underpinned by evalu- ing to achieve improvement along the way ministered by other authorities such as the ation findings, available tried and tested based on its discoveries of what needs to Ministry of Public Education (Spanish acknowledge about the topic in question, past be changed, or summative, setting out to ronym: sep) and the state-level education decisions that have proven to be relevant make judgments about what has not worked authorities- is also referred to in one of the and effective, and designs legitimized via and thus fostering accountability, but re- findings of the Critical Inventory Studies consultations with various key players to gardless of whether it is formative or sum- (Spanish: Estudios de inventarios críticos) whom their contents have been divulged mative, evaluation should not be seen as a of institutions that carry out educational and their scope and viability confirmed. panacea for all the evils and shortfalls of the evaluation at both the elementary and the Hence, two key aspects of the guideline- education system, since, even though it can lower-secondary levels, which were com- creation process that seek to build bridges help to inform decision-making, suggest missioned by the inee in 2013. Indeed, in a between evaluation, the use of the latter, and changes, raise educational quality and foster series of interviews carried out as part of the improvement, are: (i) evaluation of the suc- accountability, it does not suffice to bring aforesaid study of evaluation inventories in cesses and shortfalls of government actions, about transformation. elementary education, various interviewees and (ii) participatory creation. asserted that both federal- and state-level The use of educational evaluation in evaluations had little or no impact on the a) Evaluation of the successes and Mexico institutional planning and/or redesigning of shortfalls of government actions In our country, the use of educational policies and programs, and it was also found To create guidelines, it is very important to evaluation as part of endeavors to improve that, at both the elementary- and the lower- evaluate government actions – i.e. the poli- teaching-learning practices and formulate secondary levels, most local-evaluation re- cies and programs that the Mexican govern- educational policy would appear to have sults were only disseminated in the schools ment has implemented in order address the been limited thus far. In this regard, the in- themselves, while the local authorities had problems in question. While the guidelines formation yielded by the study entitled El no system pertaining to evaluation results. are policy recommendations that seek to uso de los resultados de las evaluaciones del In order to better inform decision-mak- influence decision-making, the scope of the 33

National Educational Evaluation Policy Gazette in Mexico

improvement is defined based on the analy- build consensuses, and thus one of the main and, in due time, about possible changes to, sis or weighing of the government actions prerequisites for the production of the said or extensions of, the aforesaid tool. Another, taken, their shortfalls, and the challenges tools is that they be created in a participato- equally big challenge has to do with the fol- faced by them. ry manner, which .is why the model or path low-up and updating that the inee, jointly In this regard, the inee has also endeav- for the issuing of guidelines includes ongo- with NGO’s, represented by the Social Ad- ored to define a path or model for the evalu- ing consultation with different members of visory Council for Educational Evaluation ation of educational policies and programs society and educators as a crucial aspect of and other participants who wish to join in, that places more stress on the evaluation of the said guidelines’ creation. In view of this, decides to carry out, heavily based on the policies than on that of isolated programs, both when drawing up its proposal for the actions and commitments of the educa- given that tackling the various problems creation of the guidelines aimed at improv- tion authorities at which the guidelines are facing education usually involves more than ing the initial training of elementary-level aimed. Furthermore, the said path should a single program. For example, evaluating teachers that were issued in September of be plotted out with the aim of supporting the success of government actions aimed at 2015, and also in its proposal for the cre- the inee’s main strategy of linking evalua- reducing dropout rates in lower-secondary ation of the guidelines for improving policy tion to use and educational improvement. schools means analyzing the design, imple- relating to the education of children and mentation, results and impact of scholarship youths from migrant day-laborers’ families 1 MacKay. (2007). asserts that other important programs, early warning systems for detect- (which is still in the draft stage), among prerequisites in order for an evaluation system ing dropout risks, or of actions aimed at other people, the inee consulted members to be deemed successful are that there should increasing parent and community participa- of organized civil society, academics and be incentives for those who use the findings and tion and collaboration. government officials, who not only helped sanctions for those who do not, and that users Hence, the first policy evaluation carried to define the educational problem that the have the tools, training and funding that they out by the inee for the purpose of issuing guidelines were intended to address, but need in order to make the suggested changes. guidelines set out to evaluate actions and also collaborated in the search for alterna- programs for the education of children and tive solutions to the said problem. References youths from families of migrant day labor- Via consultation, an effort has been Mackay, K. (2007). “How to Build M&E Systems ers, reviewing the successes and shortfalls made to anticipate the different legal, bud- to Support Better Government”. World Bank of the educational-policy actions taken by getary, technical, political and administra- Independent Evaluation Group. Consulted the National Council for the Promotion of tive limitations that could prevent the said on the 26th of January, 2016 at: http://goo.gl/ Education (Spanish acronym: Conafe), and guidelines from being accepted and hamper W0Tird. the Program for Educational Inclusion and their implementation. Miranda López, F., Martínez Bordón, A., Mend- Equality (Spanish acronym: piee) of the De- ieta Melgar, G., (2013) “Inventario crítico de partment of Indigenous Education (Spanish Final comments instituciones y organismos públicos y priva- acronym: dgei) and the National Institute Given how difficult it is to turn evaluation dos de evaluación educativa para Educación for Adult Education (Spanish acronym: into government action, the inee has traced Básica y Media Superior”. Study commis- inea), as well as studying the implementa- out a path for creating and issuing guide- sioned by the inee. tion of the aforesaid actions at the local level lines, in accordance with which, as well as Moreles Vázquez, J. (2009). Uso de la investig- by means of selected case studies. gathering the relevant information yielded ación social y educativa: Recomendacio- by the evaluations and studies that are pro- nes para la agenda de investigación. Perfiles b) Participatory creation duced by the itself and other entities -which educativos, 31(124), 93-106. Consulted on The starting point for the creation of the help to diagnose the problem and propose the 26th of January, 2016, at: http://goo.gl/ guidelines was also an awareness that even possible solutions- the inee carries out an DpL3dg. the best evidence available does not suffice evaluation of the public policies implement- Silva Laya, M. (2012). “Seguimiento de uso de re- to inform policy. As asserted by Weiss and ed, designed, precisely, to ascertain the suc- sultados de evaluación del inee por parte de other authors, in order to ensure that re- cesses and shortfalls of the said policies. It is autoridades educativas”. Study commissioned search findings form part of the government at this stage that it is possible to encourage by the inee. From July, 2011 to July, 2012. agenda and play a part in decision-making, policy reassessment and the carrying out of Subirats H., J, Knoepfel, P., Larrue, C., Varone, F., we need to build the requisite consensuses. other actions that had not previously been (2008) Análisis y gestión de políticas públicas. In other words, the guidelines should not envisaged, and the inee also brings together Ariel, Ciencias Sociales. Spain. only be based on high-quality, methodologi- different protagonists to jointly analyze both Weiss, C. H. (1998). Have We Learned Anything cally sound evaluations and research, but the public problems for which a solution is New About the Use of Evaluation? American also have the biggest possible consensus and being sought and also the key improvements Journal of Evaluation, N° 1, 21-33. be adequately disseminated, which is why it that need to be made in order to solve the is necessary to engage in dialogue with the said problems. different key protagonists. Since the guidelines are a new tool, and In order for the guidelines that it is- an enormous opportunity, for linking evalu- sues to be legitimate and viable, and have a ation to change, it is important to continu- better chance of success, the inee needs to ously reflect about how they are created 34

ENGLISH

ROADMAP

2016 Guideline-issuance What are the topics for 2016? Project 2. Guidelines for improving Program* the education available to children and youths who live in indigenous Project 1. Guidelines for improving communities. the education available to the sons and daughters of migrant agricultural day In 2016, the National Institute laborers. Why are they necessary? for the Evaluation of Education The Mexican State is heavily indebted to this (Spanish acronym: inee) will work population when it comes to the provision of on five projects aimed at drawing up Why are they necessary? education. The “Preliminary, Free, Informed guidelines for improving compulsory The sons and daughters of migrant agricul- Consultation of the Indigenous Peoples” education. tural day laborers find it very hard to gain (Consulta Libre e Informada a Pueblos y access to education because: (a) their fami- Comunidades Indígenas: inee, 2014) reveals lies are continuously moving from farm to that these communities neither identify farm; (b) there are no educational services with the education that has been provided adapted to their conditions; and (c) they to them nor find it relevant. What considerations guide the work as child laborers. According to the “Educational Over- drafting of the guidelines? According to the 2009 National Survey view of Mexico’s Indigenous Population” The drafting of the guidelines should be of Day Laborers (Spanish acronym: enjo (Panorama Educativo de la Población In- based on the participative identification of 2009), the sons and daughters of migrant dígena de México: inee, 2015, preliminary the educational problems to be addressed agricultural day laborers spend an average version), there are big shortfalls in infra- and the collaborative proposal of solutions of 4.5 years in school, compared with the structure, material and teachers possessing to the said problems, with the aim being to national average of 8.1 years, in addition to the specific profile needed to work with the ensure not only that the guidelines are legiti- which around 78% of the said population is group in question, and 74.7% of all children mate, but also that they are feasible and in- living in a state of multidimensional poverty and youths who speak indigenous languages form decision-making that will help to raise Although the Mexican State provides are taught by people who do not speak the the quality of education and make it more education to the aforesaid population said languages, while speakers of indigenous equitable. through the National Council for the Pro- languages aged 15 or over only spend a total motion of Education (Spanish acronym: 6.7 years in school. The criteria for choosing the2016 Conafe), the Elementary-Education Pro- Though there is a plethora of govern- guidelines are that: gram for the Sons and Daughters of Mi- ment programs aimed at the aforesaid grant Agricultural Day Laborers (Spanish population,3 a preliminary review of the 1. evidence is available that makes it pos- acronym: Pronim),1 and the programs of the said programs, and of such evaluations as sible to identify educational problems, National Adult Education Institute (Spanish are available, reveals that they do not focus weaknesses or risks. acronym: inea),2 nevertheless the aforesaid on the indigenous population, do not afford 2. the problems identified must be tackled group has faded from the educational agen- preferential attention to the latter, do not in the context of current educational da, since, when the Pronim became part of adapt to its conditions or characteristics, policy the Program for Educational Inclusion and and are not interlinked. 3. there exist educational evaluations or Equity (Spanish acronym: piee) in 2014, its The preliminary results of the Evalua- research whereby it is possible to ascer- budget was reduced to a mere 44% of what it tion of Basic Teaching and Learning Condi- tain the extent of the backwardness and had been in 2013. tions (Spanish acronym: ecea; inee, 2015) its impact on educational quality and Moreover, the evaluations of Conafe and at the primary level confirm that there are equity. Pronim that were carried out reveal recur- big gaps between the indigenous schools rent problems – i.e. (i) high teacher turn- and other schools in terms of the quality of over and a shortage of teachers possessing their furniture, toilets, text books and other the specific profile needed to work with the teaching materials, and the number special- group in question; (ii) scant integration and ized teaching staff. inter-sectoral linkage among the different programs; and (iii) minimal information about the characteristics and educational paths of the population in question. 35

National Educational Evaluation Policy Gazette in Mexico

According to the National Plan for the Evaluation of Learning (Spanish acronym: Planea: inee 2015), 28.4% less indigenous students in the sixth grade of primary school reach achievement-level 1 in Language and Communication than do children in general public schools.

Project 3. Guidelines for reducing dropout rates at the lower-secondary level.

Why are they necessary? Mexico has one of the lowest percentages of youths between the ages of 15 and19 enrolled in upper-secondary education, added to its high temporary and permanent dropout rates levels, which exceed those recorded for all the other school levels. In the 2013-2014 school year, the dropout rates for primary, lower-secondary and upper- secondary school were 0.6%, 4.7% and 13.1% respectively (sep, 2014). Although policies and programs aimed at reducing dropout rates were put in place when the Department of Upper-secondary Education was set up in 2005,4 the problem persists, and the only two such programs to have been evaluated are:

1. the Scholarship Program, the evaluations of which make it clear that scholarships are not the best way of reducing dropouts, since the problem is more complex and other factors, such as the relevance of the education offered and school organization, need to be taken into account. 2. the Construye T (i.e. “Build Yourself”) program, the evaluations of which identify many reasons why young people drop out of school, including boredom and lack of motivation. 36

ENGLISH

Project 4. Guidelines for improving In 2015, the National Office for the Co- putting in place scholarship programs for im- the Tutoring Program (tp) for teachers ordination of the Professional Teaching Ser- poverished students, risk-prevention strategies, who have just joined the Professional vice (Spanish acronym: cnspd) issued the tutoring and support systems, and remedial Teaching Service (Spanish acronym: National Strategy for Ongoing Training and courses. In 2010, the Síguele. Caminemos jun- spd). Professional Development in Elementary tos, (“Keep Going; Let’s Walk Together”) pro- Education, which plots out lines of action gram was established with the aim of linking for the implementation of the training pro- previously unlinked efforts, followed, in2013 , Why are they necessary? grams “to be taken by the staff evaluated in by the Yo no abandono (“I Don´t Drop Out”) In this program’s first year, the provision accordance with the law”. In January 2016, it strategy, which included interventions at the of tutoring has proven to be a highly com- was announced that the sep’s departments academic, psycho-social and socioeconomic plex endeavor. Up to October, 2015, 9,795 of Elementary Education and Upper Sec- levels. elementary-school teachers had enrolled ondary Education would be responsible for in this program (i.e. only 45% of all those ongoing training. who had taken the diagnostic test at the Just like that pertaining to tutoring, end of their first year of service), and one this is a new policy that is still in the design of the main reasons for this low coverage stage, and hence it is imperative that, start- rate is the fact that very few tutors have ing at the said stage, the inee provide con- signed up to take the pertinent entrance sultancy and help in the formulation of the examination. ongoing-training proposal to be made in the context of the spd, since the achievement of The pt for new members of the spd, the aims set forth in the said proposal will, which started in the 2014-2015 school year in large measure, depend on the latter’s suc- and was to be offered for two years by the cess. local education authorities, was the first obligatory spd support program set up for What will the input be and what are the purpose of professionalizing teachers. the expectations for the 2016 guideline Since the spd is a recently designed sys- program? tem that has not been operating for very The input to the set of guidelines will be the long, the pt has not been running for very results of the evaluations of the spd in terms long either and there are no evaluations to of learning outcomes, basic school condi- ascertain of progress. However, since both tions, policy analysis and assessments of federal and state-level education officials all government actions, the first two of which over Mexico report big delays in implement- will make it possible to carry out a diagnosis ing these tutoring services, it is essential that of the situation to be addressed, while the we identify the bottlenecks in order to sug- last two will help to ascertain the successes gest timely ways to improve both the pro- and shortfalls of the actions taken by the cesses and the results. government in order to solve the problem.

* Presented at the Ordinary Meeting of Project 5. Guidelines for improving the inee’s Advisory Social Council on the the design and implementation of Evaluation of Education (Spanish acronym: the ongoing-training program in the conscee) on the 9th of February, 2016. context of the spd.

1 Currently a part of the Ministry of Public Edu- Why are they necessary? cation’s Program for Educational Inclusion and In Mexico the ongoing-training policy has Equity (Spanish acronym: piee). been marked by: (i) a marked disconnect 2 Via the 10-14 Education-for-Life-and-Work among the educational services and insti- Model (Spanish acronym: MEVyT 10-14). tutions responsible for initial and ongoing 3 Around 15 Federal Government programs have training and teacher updating; (ii) heav- been set up to improve the education provided ily fragmented teacher training programs; to the indigenous population. and (iii) scant detection and satisfaction of 4 The2008 Comprehensive Reform of Upper- regional and local training needs (National Secondary Education (Spanish acronym: riems Council for the Evaluation of Social Devel- 2008) set out to make upper-secondary educa- opment Policy, 2010 and 2012). tion relevant and facilitate the smooth passage of students between the different subsystems, 37

National Educational Evaluation Policy Gazette in Mexico

ROADMAP

Information systems order to help evaluate the performance of on a par with the the National Education System (Spanish acronym: sen) and inform the taking of educational reform: decisions aimed at improving it, and, in this regard, the plethora of data produced innovation for strategic will be of great use in providing an over- data management* view of compulsory education in Mexico (Spanish acronym: sep). Broadly conceived, as required by the Educational Reform, Even though it is still in the such information should facilitate decision- development phase, one can making, foster accountability, and promote already foresee the great strategic transparency based on open-government contribution that the Comprehensive practices. Evaluation-results System (Spanish An information system on a par with acronym: sire) -to be launched in the the Reform, and in line with the inee’s brief, should take stock of the needs of the stra- second half of 2016- will make to the tegic players in order to fulfill its purpose. working of the National Institute for Based on a user focus, as set forth in Figure the Evaluation of Education (Spanish 1 below, the said system should offer a selec- acronym: inee) and the National tion of relevant, high-quality in a space that Educational-evaluation System enables it to be nimbly administered and (Spanish acronym: snee), and the used while maximally exploiting the poten- impact that the said system will have tial of the available data. on evidence-based decision-making Developed by the Institute with the and policy and syllabus design. The support of the Institute of Geography of the authors of the following article discuss National Autonomous University of Mexico the said sire and its probable effects. (Spanish acronym: unam), the sire brings together evaluation results and key information from the snee and the geographical, Agustín Caso Raphael demographic and socioeconomic contexts, Head of the Unit for the Dissemination and, as a strategic project aimed at creating and Promotion of the Evaluation Culture institutional value for the snee, provides an of the inee overview of education based on the ecolog- [email protected] ical-systems model, bringing information from multiple sources together in a single Arturo Cervantes Trejo platform for simultaneous use and, at all Director General of the Comprehensive times, affording a geographical overview via Evaluation- Results System of the inee geospatial analytical tools, as well allowing [email protected] people to consult, analyze, mine and down- load data. Cecilia Galas Taboada Given the sen’s large size and complex Deputy Director of the Comprehensive structure, the sire facilitates geographi- Evaluation-Results System of the inee cal analysis of the latter by indicating the [email protected] location of schools, along with contextual information, and thus providing an overview that helps us to understand how the many factors that determine the real state of education in our country are intercon- s a result of the Educational Reform and nected. It depends, for its functioning, on the setting up of the snee, we need to a business-intelligence operating model in Aensure that the different participants in the which the relevance of each piece of data said system have access to reliable, relevant, is weighed in terms of the extent to which high-quality information about the state of it enables institutions to achieve the aims compulsory education in our country in of the Reform. 38

ENGLISH

Figure 1. Characteristics of strategic information systems The said system, designed to satisfy the operational and data needs of the inee and other Educational-evaluation System of the entities, takes stock of all the working and tactical information that stem from the In- Design aimed Contains a selection stitute and the snee in order to choose, and at giving the user a of information that is of bring together, all those strategic data that pleasant experience enable us to evaluate the Mexican education use to the user system’s components and results. Based on this focus on the value of information, and in line with the mission and Accessible Relevant aims of the inee and the National Educa- tional-evaluation System, the sire compiles municati m on and manages a data bank that operates via o C the components shown in the following dia-

t gram. O n n e g m oi ve • Data bank: data base for inputting the ng impro results of educational evaluations and High-quality Integrating strategically choosing key information about both the sen and the geographical and socio-demographic context of education that are standardized to facilitate Significantly brings Ensures that the data interlinked us. together information included in its bank are of • Geoportal: a technical tool for the car- from multiple sources to high quality rying out of consultations and spatial create strategic value analyses of a selection of evaluation- result data and data stored in other data bases pertaining to the system. • Reports: documents statistics or sys- Source: Author-produced. tems available via Internet that compile key information, stemming from the data bank’s contents, about topics having to do with the results of the evaluations of the sen, and include access to contents, benchmarks, statistics and other educational-evaluation resources. Figure 2. Components of the sire • Downloading of information sources: process whereby access is afforded to links, databases and files containing pri- Environment conducive to analysis, consultation mary or processed information that can Data bank and visualization be downloaded from the system subject to legislation for the protection of personal data. Data mining Geoportal • Knowledge community revolving around the use of the system: forum for collaboration that enables progress and inno- Downloading vation in the education system to be divulged Reports of information provides advice to users about the use of infor- sources mation and tools, and serves as a means of leveraging the sire’s new capabilities. Knowledge community revolving around the use of the system Among the sire’s most outstanding features are its technological structure based on free-code software and its data model, which make it possible to operate jointly Source: Author-produced based on the conceptual design of version 5.0 of the sire. with other systems in order to exchange in- 39

National Educational Evaluation Policy Gazette in Mexico

formation more safely and efficiently. The management strategy, and the creation of References system’s structure also makes it possible to a knowledge community, in order to foster Diario Oficial de la Federación. (11th of Septem- access information from multiple sources the adoption of a system that helps users to ber, 2013). Ley General de Educación. Mex- and carry out consultation and analysis at navigate the system in a simple, effective ico. different breakdown levels, whether geo- way, doing so at different levels according to inee. (2015). Diseño conceptual del sire v5.0. graphical, or according to population fea- their needs. Dirección General del Sistema tures, type of educational service or user This sire was not only designed and de- Integral de Resultados de las Evaluaciones. Mex- interests or needs. veloped based on the best international prac- ico: inee. Its versatility is a direct response to the tices, but is also becoming a touchstone for inee. (2013). Estatuto Orgánico del Instituto need to make the information in question the work being done in other countries and Nacional para la Evaluación de la Edu- available to different audiences that play the top-level panels of entities such as the cación. Mexico. inee. different roles in education and have dif- Organization for Economic Cooperation and ferent degrees of specialization. The ap- Development (oecd) and unesco’s State-of- plications and tools will be available -and the-World’s-Education-Systems initiative. useful- to high-level decision makers, state- At the national level, it is becoming a level technical teams, researchers, members strategic tool enabling the inee and the of ngo’s, teachers, parents and students in members of the National Educational-eval- accordance with the latter’s characteristics uation System to function, as well as a valu- and technical abilities. able strategic tool that enables the Institute Since the implementation of such an and each of the snee’s members to per- all-embracing system presupposes an inform the tasks entrusted to them and sup- stitutional and cultural evolution towards ports decision-making, transparency and Those wishing to find out more about the sire should visit the following link: knowledge-management models that sup- accountability in the area of educational https://www.sire.igg.unam.mx/ port institutional strategies, and assumes evaluation. the development of technical and human capacities for using highly advanced tools, * We thank Marisol Sánchez Barrera for her help the sire envisages the design of a change- in describing the sire’s conceptual design. 40

ENGLISH

IN THE CLASSROOM

Can the project for exceed the latter’s expectations, and even those of his/her parents. certifying educational 2. Leadership on the part of the school quality via ISO 9001- principal, which implies wielding authority rationally and convincing people 2000 improve education rather than giving orders. in Zacatecas? The project aimed at certifying education in the state of Zacatecas using ISO international standards is a response to social and academic needs in a context where in- Can high-quality education be stitutions are seeking to base their actions achieved by means of inter-level on parameters that enable them to measure linkage? Conceiving of supervision the efficiency of public education services as a basis for educational decision- and detect problems in the different levels making, Adolfo Castruita Monreal of educational administration. analyzes educational policy in Implementation of the iso 9000 the state of Zacatecas, which is standard in Zacatecas applying iso 9001-2000 guidelines to By reviewing and analyzing the implications elementary-education. of adopting iso 9001-2000 in the state of Zacatecas for educational policy, it is hoped to significantly change the way schools are Adolfo Castruita Monreal perceived by society and redefine what kind Primary-school-supervision unit number of schools the latter wants, since it may well 100 Ministry of Education of the State of be the case that changes in teaching meth- Zacatecas ods, materials, classroom dynamics, as well [email protected] as school equipment and maintenance, constitute part of the new educational model aimed at improving learning outcomes and Theiso 9001-2000 standard thus enabling pupils to make sense of today’s in education realities. iso standard iwa2-2003/nmx-cc-023- It is pertinent, for purposes of this proj- imnc:2004, quality-management systems: ect, that Zacatecas is implementing the guidelines for the implementation of iso High Quality Schools Program, with the 9001-2000 in education, adopted as part of State Council for Social Participation in Ed- the International Workshop Agreements ucation cooperating in all the latter’s stages, (IWA’s) is deemed to be the general standard from training to the adjudication, selection for the implementation of ISO certification and monitoring of each of the Strategic in schools, since it enables us to analyze dif- Plans pertaining to all the schools that have ferent aspects of the quality and manage- signed up for this general research project ment of state education. aimed at raising the quality of education in Such certification sets out to promote our state. ongoing improvement in the most common Given the above, the said Council de- aspects of school operation and optimize cided that two elementary-level institutions learning outcomes by analyzing institution- participating in the High Quality Schools al needs, shortcomings and strengths. Oria program -i.e. the Salvador Varela Resén- Razo (2003) lists the guiding principles for diz morning-shift primary school in the implementing the ISO 9001-2000 standard municipality of Zacatecas and the General in Mexican schools as: Lázaro Cárdenas del Río general secondary school in the city of Fresnillo- should seek 1. A focus on getting to know the student iso 9001:2000 certification. and his/her educational needs, in order After just over one year and three to provide fast, effective solutions that months of joint efforts by teachers, manage- 41

National Educational Evaluation Policy Gazette in Mexico

ment and support staff and parents, in May, Via this certification, the said institution, 2004, the aforesaid two schools obtained which is one of the best schools in the mu- the said certification, which was endorsed nicipality, taking in 400 students a year, and by the International Certification Network graduating a similar number who achieve (IQNet) company, a subsidiary of SAI Global very good results in subsequent educational Limited, so that the abovementioned Salva- levels, is endeavoring to raise the quality of dor Varela Reséndiz primary school, located the services it renders. in Laboristas street in the Zacatencas bor- In order for a school to implement an ough now has iso 9001-2000 certification ongoing-improvement program such as the in management processes for high-quality one proposed here, it needs to make some education. changes to its internal organization in or- Following the guidelines set forth in the der to achieve more efficient teaching, and iwa2 governing document, the said school’s this implies the widespread use of different pre-enrollment, enrollment, student-entry, teaching methodologies, support materials teaching-learning, evaluation, certification and equipment in order to facilitate learn- and graduate-monitoring processes -all run ing. in accordance with the standardized guide- One of the priorities is that technologi- lines contained in the quality manual per- cal resources be used not only by pupils to taining to the model proposed by the self- increase their knowledge and acquire skills, same school- were certified. but also by teachers and administrative staff, The aforesaid model seeks to reduce since the most important task is that of de- the number of shortfalls in the rendering signing and producing teaching materials, of educational services, increase academic especially when one considers that teachers productivity, make optimal use of teach- are specialists in the subjects that they teach ing time, increase commitment in pupils and should be capable of developing teach- and their parents, and set up an ongoing- ing methods that satisfy their pupils’ needs. improvement process by making daily pro- As the research draws to an end, the cedures more systematic and monitoring General Lázaro Cárdenas del Río school is quality. putting the finishing touches to its efforts to Despite the above efforts, the school in be recertified for a further three years, and question is constantly criticized by -and it bears mentioning that the said institu- often receives complaints from- some par- tion’s achievements are due to its principal’s ents for various reasons such as rejecting leadership and the clear commitment of its requests for children to join certain groups, teachers. forming a first grade with so-called “rec- ommended pupils”, expelling children with Conclusions low performance levels, having teachers The lessons learned from this educational who constantly make heavy demands on policy adopted in the state of Zacatecas in their pupils, and, sometimes, ill-humoredly order to achieve significant changes in the scolding parents and pupils – so much so current conception of what a school consists that, having reached the end of the proof and what type of school that society needs gram, it is hard put to gain recertification, spring from the endeavor to ascertain the and even runs the risk losing the certifica- strengths and weaknesses of each school’s tion it currently has. teaching methods. practices and materials, The pre-enrollment, enrollment, stu- equipment and upkeep from a private-sec- dent-entry, teaching-learning, evaluation, tor viewpoint so as to formulate the perti- certification and graduate-monitoring pro- nent ongoing-improvement plan. cesses of the General Lázaro Cárdenas del Ideally, each elementary-level school Río federal secondary school, located in should draw up its own proposal for im- Fresnillo, Zacatecas, are likewise certified, proving learning outcomes, analyzing also being run in accordance with the stan- achievement during each school year, and dardized guidelines contained in the manual proposing actions likely to result in ongo- pertaining to the private-education quality ing improvement, since the aim is to raise model. pupils’ achievement levels and equip them 42

ENGLISH

DOSSIER: USE OF INFORMATION, TWO PROPOSALS with skills, as well as to augment their cul- OTHER PERSPECTIVES tural capital, and endow them with the in- tellectual tools that they need in order to go on learning. Gazette No. 4. How and evaluations have been carried out -and, to foster the use of in the case of Mexico, specific mechanisms References constructed- for the purpose of ascertaining Normas 9000. “Guía de inicio”, consulted on No- evaluations? the extent to which government heed evalu- vember 30th, 2007. http://www.normas9000. ators’ recommendations, notwithstanding com/RegistrosRequeridosPorISO.htm which utilization remains a major challenge iso 9000, iso/iwa2/wd1n4 and Mexican Min- for this whole system of rules and processes istry of Education (Spanish acronym: SEP) “The generally accepted tenet that that is founded on the premise that its evalu- standards (2002). International Workshop evaluations do not lead to change ations will be used. Agreement 2. Quality Management Sys- due to the lack of interest of those In this article, after giving a brief intro- tems-Guidelines for the application of iso in charge of programs or the scant duction to the problem, we present an anal- 9001:2000 to education. evaluation culture does not suffice”, ysis of how the federal government tends ______The Ten Most Frequently asked ques- say the authors of the following to use evaluations, and, based on the latter, tions about iso, in iso 9001-2000, What is make some proposals as to how their use ISO and how does it work? Global Engineer- article, who give three reasons why can be fostered. ing documents, consulted on November 15th, evaluation results are not used to 2007 at http://global.ihs.com/ make decisions, four reasons why The problem of use ORIA Razo, Vicente. (2003). “La calidad de la recommendations are not heeded, With regard to social development, in our educación”, in iso 9001:2000 en la educación going on, finally, to describe two uses country an evaluation system that is com- mexicana. Ediciones virtuales, sep, Mexico. to which evaluations are put. pulsory, generalized, systematic and stan- ______(2003). “Los ocho principios básicos dardized has taken shape, and these features para la aplicación de iso/iwa2:2003, directri- have become an international touchstone – ces para aplicar la norma iso 9001-2000 en la Guillermo M. Cejudo above all for foreign governments that seek educación”, in iso 900-2000 en la educación Academic Secretary of the Centro de to ascertain how effective such interven- mexicana. Ediciones virtuales, sep, Mexico. Investigación y Docencia Económicas tions are, with a view to improving them. SCHMELKES, Sylvia. (1992). Hacia una mejor (cide) According to the National Council for the calidad de nuestra escuela. Publicaciones sep. [email protected] Evaluation of Social-development Policy ______(2000). “La Evaluación de los Apren- (Spanish acronym: Coneval), based on the dizajes en la Educación Básica”, en Educación Francisco Abarca Guzmán compulsory evaluations of federal budget- 2001 No. 70 (March), Mexico. Master’s degree in Public Policy from the ing programs, over 3,300 recommendations cide have been made to an average of 118 pro- [email protected] grams that were evaluated each year from 2008 to 2013. Although evaluation of our country’s development policy has proved hen a government official makes to be a very useful means of producing in- public-policy decisions, s/he has to formation for purposes of decision-making, Wtake many factors into account, including whether about budgeting or regarding the budgetary constraints, political expediency, redesign of programs, nevertheless one of managerial ability, citizens’ expectations the suppositions that is most prejudicial for and the available evidence. Such decisions the evaluation of government policy is the are not usually taken by a single person, via assumption that there is a direct link be- an orderly, rational process where the aims tween the latter and the significant improve- are not in dispute, but, rather, there is sel- ment of programs. dom enough evidence available. What can Such concern about using results exists we do, under such conditions, to ensure that in all evaluation systems and is explained by the decisions taken result in improved pub- the difficulty of setting up mechanisms that lic policies that are more likely to actually coherently make explicit the cause-and- solve the problems that they address? effect relationships among the information For several years now, one of the most fre- produced, the protagonists and processes quent answers to the above question stresses involved, and the results expected of the the need to generate evaluations that provide evaluation. The commonly accepted as- information that that guides government sumption that evaluations do not lead to decision-making, and hence institutions have change due to lack of interest on the part of been set up charged with evaluating policy, those responsible for the programs and en- 43

National Educational Evaluation Policy Gazette in Mexico

tities evaluated, or to the scantiness of the “evaluation culture”, does not suffice to -ex plain their use at a broader level. Regardless of the purpose for which an evaluation is carried out, its perceived success or failure tends to be based on the extent to which the information it yields is used. All evaluation endeavors should be conceived of in terms of how useful their findings are for improving the results of a given social intervention. The specialized literature on the subject has identified two main types of evaluation depending on the use that is to be made of the information produced. On the one hand, formative evaluation stresses feedback aimed at improving a policy’s end product (Weiss, 1998), while, on the other hand, summative evaluation enables us to determine the merit or value of a policy and make a “last judgment” about it (Russ-Eft and Preskill, 2001). Thus, the information produced by evaluations can help to redirect efforts and the commitment of resources, or to put an end to a set of actions that is not yielding results, and, in either case, use implies the taking of policy actions as result of a growing understanding of the causes of a public problem (Pawson, 2013; Cejudo and Maldonado, 2011). There are various reasons why the results of an evaluation are not used to inform decision-making. The literature suggests reasons ranging from institutional and policy factors to the individual attributes of decision-makers (González, 2016). Gener- ally speaking, the features and quality of the evaluation system, and the characteristics of the evaluators, are the main factors that can impede use.

1. Features of the evaluation system. The or there is nobody responsible for imple- to involve the right agents to make real, management, design and commission- menting them, they it will be unlikely to and not merely operational, changes and ing of the evaluation (done at the mo- serve as input for decision-making. the match between evaluation-system ment when the program is being de- 3. Features of the evaluees. Certain attri- guidelines and the rest of the of the ac- signed) can lead to temporary flaws or butes of evaluations and the recipients of countability systems. In short, it is a flaws in the process (e.g. if a bad or ad their results -i.e. the degree to which the matter of whether the recipient of the hoc evaluator is hired). evaluation is institutionalized and the evaluation results wants -and is able- to 2. Evaluation quality. If an evaluator car- coexistence of decision-making ability, act on the evaluator’s recommendations. ries out a shoddy analysis, his/her find- managerial ability and spending power ings will probably be of little or no use, in the person/entity that receives the How are evaluations used in Mexico? and if an evaluator’s conclusions are too evaluation results- that can explain how Based on the results published by Coneval in general or abstract, useable recommen- and why the evaluations are used. It is its Follow-up Reports on Aspects of Federal dations will not be able to de made. If the matter of the ability of organizations to Programs that are Subject to improvement, said recommendations are not feasible react to evaluations, including the ability we can assert (after Cejudo and Abarca, 2016) 44

ENGLISH

that those evaluation results that identify so called Aspects Open to Improvement (AOI’s) are indeed used by the entities responsible for budgeting. Since 2008, most of the said AOI’s are categorized by the aforesaid entities as “specific”, meaning that, besides acknowledging the need to improve these aspects, the entities in question acknowledge that they are directly responsible for making the recom- mended improvements. Moreover, year after year, except for the period 2012-2013, those responsible for using the information have, in most cases, recognized the high priority of the best proposals stemming from evaluations. In this regard, there is evidence showing that the information produced by the evaluation system is well received by those in charge of programs, who see it as an important information source. Nevertheless, the aforesaid use varies. For example, the programs with the biggest budgets make more use of information that enables them to improve aspects having to do with their running or implementation, while, on the contrary, the programs with the smallest budgets use the information stemming from the compulsory evaluations to improve their own design (Cejudo and Abarca, 2016). This is because the former programs have been able to carry out additional research projects and evaluations that provide them with more accurate diagnoses of the problems that they face, while, for the latter, the compulsory evaluations are the only source of information, and hence, for the said low-budget programs, the quality of the said information, and therefore the quality of the evaluations themselves, plays a basic role in improving their interventions in the public sphere (Abarca, 2014). Besides understanding how evaluations are used, we need to ascertain why the recommendations stemming from them are passed over by those in charge of programs. It has been found, in the case of Mexico, that recommendations are not heeded mainly for the following four reasons (Cejudo and Abarca, 2016):

1. Lack of synchronization. The recommendations that are made had already been considered by those in charge of the program as a result of previous evaluations, meaning that there is a lag between the times when evaluations are carried out and their implementation. 2. Limited empowerment. The recommen- 45

National Educational Evaluation Policy Gazette in Mexico

dations cannot be heeded because they clear identification of those responsible the conditions needed for the said evalua- exceed the program’s brief, which attests for implementation, and this means that tions to engender decisions that improve to ignorance on the part of the evaluator. “quality control” is not limited to the policy results already exist when the recom- 3. Lack of clarity. The recommendations due and timely delivery of information mendations are made. are ambiguously expressed by the evalu- within the terms of reference, but im- ators, or are vague to the evaluation for- plies interaction aimed at assuring that References mat. recommendations are heeded in accor- Abarca, F. (2014). Evaluar con múltiples propósi- 4. Lack of resources. The recommendations dance with the same terms. tos: efectos del sistema de evaluación mexi- cannot be acted on due to lack of money, cano en los programas de desarrollo social. staff or time. b) Organizational ability to act on Master’s Degree Thesis. Mexico:cide . recommendations Cejudo, G. and Abarca, F. (2016). ¿Cómo se usan The evaluation system itself can take i) It is crucial to identify the specific peo- las evaluaciones? Un análisis de cómo los stock of all these aspects in order, in the first ple responsible for acting on the recom- programas reaccionan a lo que recomiendan place, to improve the quality of the informa- mendations so as ensure that decision- las evaluaciones. Mexico, unpublished. tion offered to those in charge of the pro- making ability, managerial ability, and, Cejudo, G. and Maldonado, C. (2011). De las grams, and, in the second place, to increase ideally, the ability to make budget deci- recomendaciones a las acciones. Mexico: the probability that the said information will sions, all coincide. clear-cide-sfp. be used. The following section of this ar- ii) All those involved in decision-making, González, D. (2016). Usos de la información de Se- ticle contains recommendations on how to both inside and outside the organiza- guimiento y Evaluación (sye) en las Políticas achieve this. tion, should be made aware of the rec- Públicas. Mexico, doctoral thesis, cide. ommendations and the implications of Consejo Nacional de Evaluación. (2009). Informe What can be done to increase their participation. de Seguimiento a los Aspectos Susceptibles de evaluation use? iii) Forums need to set up for interaction Mejora de Programas Federales 2008. Mexico. Perhaps the most important point is that it is between the evaluator and the decision- Consejo Nacional de Evaluación. (2010). Informe hard to impose use – i.e. if a government of- makers, so that the government officials de Seguimiento a los Aspectos Susceptibles de ficial doesn’t know what s/he can use evalua- understand the reasons for the recom- Mejora de Programas Federales 2009. Mexico. tion results for, how they were produced, or mendations and link them to their basic Consejo Nacional de Evaluación. (2011). Informe what their relevance is, s/he is very unlikely activities. In other words, the officials de Seguimiento a los Aspectos Susceptibles de to act on the recommendations in question. should not see the heeding of the said Mejora de Programas Federales 2010. Mexico. In this regard, deliberate interventions are recommendations as a mere bureaucrat- Consejo Nacional de Evaluación. (2012). Informe needed in the following two areas: ic formality, but rather as a decision, that de Seguimiento a los Aspectos Susceptibles de helps them to get results. Mejora de Programas Federales 2011. Mexico. a) Evaluation quality Consejo Nacional de Evaluación. (2013). Informe i) The processes for commissioning the de Seguimiento a los Aspectos Susceptibles de evaluations need to be improved, keep- Conclusions Mejora de Programas Federales 2012. Mexico. ing a balance between involving those To turn findings into recommendations, Consejo Nacional de Evaluación. (2014). Informe responsible for making key decisions there must be a dialogue between the evalua- de Seguimiento a los Aspectos Susceptibles de (so that they do not see the evaluation tor and the person in charge of the program, Mejora de Programas Federales 2013. Mexico as something alien) and maintaining the and this can only happen if the former has Pawson, R. (2013). The Science of Evaluation. A independence of the evaluator (so that reliable information, has carried out a sound Realist Manifesto. London: Sage. closeness to the decision makers does analysis, and makes timely, relevant recom- Russ-Eft, A. and Preskill, H. (2001). Evaluation in not jeopardize the impartiality of the mendations on the one hand, and the latter, Organizations. New York: Basic Books. findings). on the other hand, considers the evaluation Weiss, C. (1998). Evaluation. New York: Prentice ii) The evaluation needs to be designed to be necessary and useful, understands the Hall. and developed in such a way that the recommendations and sees in them an op- information gathered, and the decisions portunity to improve his/her performance documented, produce a cumulus of evi- and achieve the aims set for his/her orga- dence. nization. If the aforesaid conditions are not iii) It is necessary to promote the hiring of fulfilled, even if the program head chooses better evaluators, who have ample theo- to comply with the recommendations as retical and working knowledge of the a matter of obligation, is forced to act on programs and combine methodologi- them, and produces follow-up reports, the cal soundness with expertise about the evaluations will not be as useful as expected. workings of the public sector. In short, rather than being a putative iv) It is necessary to demand that the evalu- aim or an afterthought, the use of evalua- ators make precise, relevant, feasible, tions must be at the core of decision-making timely recommendations involving the about policies and their assessment, so that, 46

ENGLISH

DOSSIER: USE OF INFORMATION, TWO PROPOSALS OTHER PERSPECTIVES

The Use Radar However, the fact that such evidence is available is no guarantee that it will serve to bring about desirable social changes, since the possession of robust evidence does not suffice to ensure that knowledge is convert- Though evaluation is certainly ed into the decisions needed to improve the essential in the context of the lives of millions of people who suffer from National Educational-evaluation fundamental privations. Policy, it is no less crucial that Without any doubt, in order to produce its results be used to drive evidence that has an impact on the making improvement. How do we decide of decisions about government policies or programs, we need rigorous scientific re- how to use evaluation evidence search that proves that given actions lead and recommendations? What are to the changes that are expected in the tar- the basic dimensions of evaluation get population. However, it is not enough use? Dr. Pérez Yarahuán posits four to determine, via research, whether a giv- components -acknowledgement, en circumstance is produced by a specific quality, credibility and participation- government action or program. Evidence that form the tips of a radar’s is also a result of the analysis of processes crosshairs. of cause and effect and the identification of those factors that play a part in the logic of change. Since the causative chain (program theory) that stems from a desirable change must be analyzed in terms of the transfor- Gabriela Pérez Yarahuán mations that occur in people’s behavior in Research professor attached to the Centro response to an external factor or stimulus, de Investigación y Docencia Económicas and also of the proper implementation of (cide); Research coordinator, clear the said stimulus, intervention or program Center for Latin America. by those responsible for taking action, then [email protected] evidence consists in proving that government actions are effective, and providing logical programmatic explanations for this he ways in which knowledge is acquired effectiveness and for the processes used to have changed radically over the last 20 achieve it (Rossi, Lipsey & Freeman, 2004). Tyears in most parts of the world. Today, sci- In other words, evidence is basically gath- ence possesses tools for obtaining data and ered via a comprehensive process of evalu- specific information about nearly all the as- ation that implies scientific ways of acquir- pects of human life -biological, economic, ing knowledge about the different aspects cognitive and social- at a speed that was of the logic of government action and poli- heretofore undreamt of, added to which cy implementation. processing capacity has increased substan- However, the said evaluation will not tially due to the improvement and large- be turned into evidence that is used for scale availability of technology over recent purposes of improvement unless certain decades. Sophisticated techniques for the features able to convert knowledge into quantitative and qualitative analysis of in- action are explicitly considered. In this re- formation have proliferated as a result of the gard, the present article posits at least four ability of multiple members of an increas- dimensions, none of which suffices, in itself, ingly interconnected social and academic to ensure the instrumental use of evalua- community to collaborate, and nowadays tion, though, together, they should inform the scientific community has the tools need- decision-making about evaluation policy. ed to generate more evidence about -and ex- The said dimensions areacknowledgement, plain- the factors that affect so many aspects quality, credibility and the participation in of our lives. the evaluation process of those responsible 47

National Educational Evaluation Policy Gazette in Mexico

Figura 1. The Use Radar

Quality

Aknowledgement Credibility

Participation

for taking action, and the use of evaluation study curricula, teacher training, scholar- is the result of correct decisions in all four ships, provision of materials, building of of them. infrastructure, expansion of school hours, To clarify this idea, I posit a metaphori- etc.) result in changes in one aspect or an- cal “Use Radar”1 (See Fig. 1) that should other of people’s lives (e.g. terminal efficien- guide decision-making about evaluation cy, educational inclusion or student pupil policy. performance). Below, I explain the aforesaid four di- When applied to evaluation, the term, mensions, how they relate to each other, use, has different meanings (Johnson,et al., why they affect the achievement of use, giv- 2009; Leviton & Hughes, 1981). The evi- ing some examples to illustrate them, not- dence acquired via evaluation can be used withstanding which the following two brief to legitimize an action that has been carried notes are required to clarify the concepts of out or learn about it, but it can also be used evaluation and use. to make specific changes –e.g. in the type of goods or services provided, the inclusion cri- Type of evaluation and type of use teria for the population served- or even as a The evaluation that is referred to in this ar- justification for cancelling the intervention. Credibility ticle mainly consists of research into specific When we talk about the use of evaluation to programs or interventions for the purpose make improvements, we have in mind this of ascertaining if the actions taken (new type of instrumental use, which differs from 48

ENGLISH

other kinds of use because it is hoped that ongoing requisite for the establishment of gram’s continuation, rather than affecting specific actions, aimed at improving key as- patterns of action and behavior. its design and modifying its components in pects of interventions or programs, will en- Fort the last 10 years, there has been order to improve its results. sue from the evaluation process. Some other wide recognition in Latin America of the types of use are symbolic use (or use in order need to evaluate government programs and Credibility as a prerequisite of use to exert influence) and conceptual use (Her- policies. A recent study of evaluation sys- Evaluation is more likely to be used for im- bert, 2014), which may set out to legitimize tems in the said region remarks that most of provement if its findings are perceived by decisions or trigger a learning process, but its countries have passed fundamental laws the relevant protagonists to stem from a do not result in specific actions regarding governing government actions that include totally unbiased process that includes ade- -or changes to- government programs. obligations to evaluate the latter (Pérez quate research techniques and practices and All evaluations do not necessarily lead to Yarahuán & Maldonado Trujillo, at press). considers and synthesizes all the relevant instrumental changes, since not all evalua- Nevertheless, although evaluation is widely information in the recommendations that tions are intended to have this sort of direct acknowledged, its low level of utilization is arise from the data analyzed. impact. For example, in the academic world, repeatedly stressed. For evaluation to be highly credible, it the purpose of evaluation-based research Although it is essential that the State’s must be perceived as part of a legitimate, is to gather scientific evidence that helps to duty to evaluate be acknowledged if evalu- transparent process based on adequate profes- bear out hypotheses about factors that can ation results are to form part of the formu- sional criteria, and it must also be clear, since affect human behavior (Levitt & List,2009 ), lation and implementation of better, more its credibility depends on the transparency of with the hope that such an accumulation effective government policy, as the afore- its design, on the information that it uses, and if evidence in a given sector will influence mentioned study shows, acknowledging on the independence of the evaluators. political debate and eventually lead to the evaluation is not tantamount to using it. The external evaluations of social pro- formulation of more effective policies and grams that have been coordinated by the greater accountability (Chelimsky, 2006). The importance of quality in National Council for the Evaluation of So- However, the management-by-results evaluation cial-development Policy (Spanish acronym: model, which has flourished in govern- For the its results to be used for purposes Coneval) since 2007 are a very good example ment organizations, places special stress of improvement, evaluation must be based of credibility. Over the last decade, Coneval, on evidence-based decision-making – i.e. on reliable data, appropriate methodol- which has played a key role in the creation of it emphasizes a commitment to make de- ogy, explicit hypotheses and solid theories. a legitimate, transparent evaluation system cisions regarding budgeting and set up or These attributes are indispensable requi- that uses high-quality information, has cre- close down programs based on the results of sites for endowing evaluation with quality, ated or coordinated more than 1,000 evalu- studies and evaluations, having a very strong and when evaluation lacks the latter, it will ations, including, of course, evaluations of focus on instrumental use. be only sparsely used or run the serious education programs and various program- For evaluation to be used instrumentally, risk of not improving action and hence not matic processes thereof, such as design, the protagonists who make decisions about yielding good intervention results. Though impact and performance, and thanks to the design and implementation of govern- high technical quality is a prerequisite, it the said organization, we currently possess ment programs and policies need to be con- does not guarantee that evaluation will be a significant body of information and data vinced that it is “useable”, and this will only used to improve programs. One example regarding key social-program benchmarks. be possible based on a set of evaluation-proof high-quality evaluation whose use for However, to what extent has this evidence cess attributes that includes acknowledg- instrumental improvement is limited is the been used for improvement? The findings of ment, quality, credibility and participation, Education, Health and Nutrition Program a study of budgeting programs for elemen- which are sine qua non dimensions for the (Spanish acronym: Progresa), which is cur- tary education from 2000 to 2014 indicate design of evaluation processes or systems rently called Prospera, having previously that there has generally been scant use of that permit a high degree of use. been named Oportunidades. Though there external evaluations of programs aimed at can be no doubt that the evaluation of this modifying the latter’s design via operating The acknowledgment of evaluation program, the findings of which started to rules (Pérez Yarahuán, 2015). The factors To form part of routine government ad- circulate in 1999 (International Food Policy most likely to promote changes in the said ministration and be used to improve poli- Research, 2001), has been very highly ac- operating rules are evaluation design, the cies and programs, evaluation needs to be knowledged for its high technical quality, programs’ contextual variables, and the po- acknowledged and accepted by a group in- it is equally true that the said evaluation, litical environment. volved protagonists. By “acknowledged”, we in its early stages, was used to legitimate So the credibility of evaluation is impor- not only mean included in the in the formal and continue the said program, rather than tant, but its design plays a fundamental role rules (laws, decrees, regulations, etc.) that to improve its design or implementation. in determining its use, due not only to the are in place, but also perceived as a respon- While the intention of the evaluation was nature of what is being evaluated, but also sibility both by those in government and by to enable Progresa to survive a change of because it is in the design stage that funda- citizens. This is essential for the building of government in a period of political uncer- mental issues such as the role of the protag- an “evaluation culture” whereby the evalu- tainty, it should not be denied that its use onists (evaluees, evaluators and decision- ation of government policies becomes an was circumscribed to achieving the pro- makers) are determined. 49

National Educational Evaluation Policy Gazette in Mexico

Participation The four dimensions of our use radar Johnson, K., Lija O. Greenseid, Stacie A. Toal, Though empirical studies of use have often -acknowledgement, quality, credibility and Jean A. King, Frances Lawrenz & Boris found that both the quality of evaluation, and the participation- can provide orientation Volkov. (2009). Research on Evaluation Use: also its relevance, credibility and acknowledge- for an effective evaluation policy. In order A Review of the Empirical Literature from ment, affect its use (Johnsonet al., 2009), we for educational evaluation to be used to 1986 to 2005. American Journal of Evalua- nevertheless need to establish a more general bring about improvement, we must rec- tion, 30(3), 377–410. framework for understanding change mecha- ognize that high-quality evidence is essen- Leviton, L. & Hughes, E. (1981). Research on the nisms. In this regard, it has been proposed that tial….but then so are the evaluation’s ac- Utilization of Evaluation. Evaluation Review, theories be formulated linking different types knowledgement and credibility, and also the 5(4), 525–548. of use and different levels of analysis (individ- mechanisms of participation, appropriation Levitt, S. D. & List, J. A. (2009). Field Experiments ual, group and collective). In accordance with and socialization. in Economics: The Past, the Present, and the this viewpoint, the change brought about by Future. European Review, 53(1), 1–18. an evaluation should be explained in terms of 1 This concept is based on Diermier’s “Trust Ra- Pérez Yarahuán, G. (2015). La influencia de la a mechanism that affects attitudes and actions dar”, 2011. evaluación externa en los programas guber- and has the ultimate aim of creating social namentales para la educación básica. Revista wellbeing (Mark & Henry, 2004). References Mexicana de Investigación Educativa, XX The instrumental use of evaluation does Chelimsky, E. (2006). The Purposes of Evaluation (66), pp. 685–710. not happen automatically after a recommen- in a Democratic Society. In The Sage hand- Pérez Yarahuán, G. & Maldonado Trujillo, C. (En dation has been made. Evaluation is done in book of Evaluation. London: sage. prensa). Panorama de los sistemas nacionales specific contexts that affect the nature of its Herbert, J. L. (2014). Researching Evaluation In- de monitoreo y evaluación en América Latina. input, and, for it to have an impact on the fluence: A Review of the Literature. Evalua- Mexico City: Centro de Investigación y Do- design and implementation of programs, we tion Review, 38(5), 388–419. cencia Económicas/clear. need to put in place a mechanism for learn- International Food Policy Research. (2001). Is Rossi, P. H., Lipsey, M. W. & Freeman, H. E. ing about, understanding and socializing its progresa Working? Summary of the Results (2004). Evaluation. A systematic Approach findings whose design includes the partici- of an Evaluation by ifpri. undated: ifpri. (Seventh). Thousand Oaks:sage . pation of those evaluated. National Educational Evaluation Policy Gazette in Mexico