Australian Council for Educational Research (ACER) ACEReSearch

2005 - Using data to support learning 1997-2008 ACER Research Conference Archive

2005

Using Data to Support Learning (Conference Proceedings)

Australian Council for Educational Research (ACER)

Follow this and additional works at: https://research.acer.edu.au/research_conference_2005

Part of the Educational Assessment, Evaluation, and Research Commons

Recommended Citation Australian Council for Educational Research (ACER), "Using Data to Support Learning (Conference Proceedings)" (2005). https://research.acer.edu.au/research_conference_2005/2

This Book is brought to you by the 1997-2008 ACER Research Conference Archive at ACEReSearch. It has been accepted for inclusion in 2005 - Using data to support learning by an authorized administrator of ACEReSearch. For more information, please contact [email protected]. Australian Council for Educational Research ACEReSearch

2005 - Using data to support learning Research Conferences

8-28-2008 Using Data to Support Learning (Conference Proceedings) Australian Council for Educational Research (ACER)

Recommended Citation Australian Council for Educational Research (ACER), "Using Data to Support Learning (Conference Proceedings)" (2008). 2005 - Using data to support learning. http://research.acer.edu.au/research_conference_2005/2

This Book is brought to you for free and open access by the Research Conferences at ACEReSearch. It has been accepted for inclusion in 2005 - Using data to support learning by an authorized administrator of ACEReSearch. For more information, please contact [email protected]. RESEARCH CONFERENCE 2005 Research Conference 2005

Using Data to Using Data to Support Learning Using Data to Support Support Learning

7–9 AUGUST, 2005 GRAND HYATT HOTEL, MELBOURNE,VICTORIA

Conference Proceedings ConferenceConference ProceedingsProceedings Contents

Foreword v

Keynote papers 1 Gage Kingsbury 2 Benchmarks and growth and success…Oh, my! Lorna Earl 6 From accounting to accountability: Harnessing data for school improvement John Hattie 11 What is the nature of evidence that makes a difference to learning?

Concurrent papers 23 Gabrielle Matters 24 Good data, bad news, good policy making… Lynn Tozer and Marilyn Holmes 32 Moving on from Count Me In Too: Evidence-based teaching and learning in numeracy in the early and middle years of schooling Max Smith 38 Getting SMART with data in schools: Lessons from NSW Glenn Rowley and Peter Congdon 46 Data-driven school improvement through the VCE Data Service Rosemary Cahill 58 ‘Getting it Right’ Symposium – Paper 1 Getting it Right … using the right data effectively Lawrence Ingvarson 63 ‘‘Getting it Right’ Symposium – Paper 2 Getting professional development right Marion Meiers 72 ‘Getting it Right’ Symposium – Paper 3 Evaluation of the Getting it Right Literacy and Numeracy Strategy in Western Australian schools Concurrent papers continued Wayne Craig 84 Data and school improvement – A school perspective Reg Allen 87 Using the evidence of student achievement for improvements at individual, class and school level John DeCourcy 93 Using HSC data to give principals leverage Michele Bruniges 102 An evidence-based approach to teaching and learning Philip Holmes-Smith 106 Assessment for learning: Using Statewide Literacy & Numeracy tests as diagnostic tools Carmel Richardson 111 Data-informed research and practice: Evaluating student achievement in secondary schools Jim Tognolini 126 Using online assessment to inform teaching and learning in primary and secondary classrooms David Axworthy 127 Turning data into information that improves learning:The WA experience Ken Rowe 131 Evidence for the kinds of feedback data that support both student and teacher learning Hilary Hollingsworth 147 Learning about teaching and teaching about learning: Using video data for research and professional development Teresa Angelico 152 An evidence-based approach to improvement:A case study of the Victorian Catholic Sector

Poster presentations 157

Conference program 161

Conference delegates 165 Research Conference 2005 Planning Committee Professor Geoff Masters CEO, Conference Convenor, ACER Dr John Ainley Deputy CEO and Research Director National and International Surveys, ACER Dr Ken Rowe Research Director, Learning Processes and Contexts, ACER Ms Margaret Forster Research Director, Assessment and Reporting, ACER Ms Marion Meiers Senior Research Fellow, ACER Ms Carmel Richardson Senior Research Fellow, ACER Ms Kerry-Anne Hoad Manager, Centre for Professional Learning, ACER

Copyright © 2005 Australian Council for Educational Research

19 Prospect Hill Road Camberwell VIC 3124 AUSTRALIA www.acer.edu.au

ISBN 0-86431-684-4

Design and layout by Integral Graphics and Wendy McGregor Editing by Caroline Glascodine Printed by Print Impressions ForewordForeword

v Geoff Masters Australian Council for Educational Research Research Conference 2005 is the tenth national Research Conference.Through our research conferences, ACER provides significant opportunities at the national level Geoff Masters is Chief Executive Officer of the for reviewing current research-based knowledge in key areas of educational policy Australian Council for Educational Research and practice. A primary goal of these conferences is to inform educational policy (ACER). Professor Masters is an international and practice. authority in educational measurement and student assessment and has published extensively Research Conference 2005 brings together key researchers, policy makers and in these fields. Early in his career he developed teachers from a broad range of educational contexts from around Australia and the widely-used partial credit model for the overseas.The conference addresses the theme ‘Using data to support learning’. statistical analysis of rating scales and professional judgements. Although much of his research has We are sure that the papers and discussions from this research conference will been focused on questions of validity and make a major contribution to the national and international literature and debate reliability in large-scale tests and surveys, on the effective use of data. Professor Masters has a special interest in using developments in modern measurement We welcome you to Research Conference 2005, and encourage you to engage in theory to construct improved tools for conversation with other participants, and to reflect on the research and its professional practitioners. connections to policy and practice.

Professor Geoff N Masters Chief Executive Officer, ACER

Research Conference 2005 vi KeynoteKeynote paperspapers

1 Benchmarks and growth and success … Oh, my!

Abstract easy one, it is clear that it involves the amount that a school helps students In order to inform decisions in our grow in their knowledge, and in their schools, information about student love of learning. It seems clear that a achievement has to be accurate and model for school success that doesn’t timely.The information also has to be include the growth of an individual child presented in a fashion which encourages is not a very useful model. teachers and schools’ personnel to make the best possible decisions. One This paper will discuss some recent G. Gage Kingsbury of the most basic pieces of information research concerning US attempts to University of Minnesota concerns whether the school is doing a use student proficiency standards to good job educating its students. identify schools that are struggling. It will also discuss a model that combines G. Gage Kingsbury (Ph.D., Psychology, University This paper will discuss some recent of Minnesota, 1984) is the Director of Research growth and standards to improve our research concerning attempts in the for the Northwest Evaluation Association ability to identify successful schools. (NWEA). He served as a member of the United States to use student proficiency Finally, it will discuss the use of an NWEA board of directors for seven years. His levels and content standards to identify assessment system that fosters primary area of focus is in the application of Item schools that are struggling. It will also improvement in education. Response Theory to practical assessment discuss a model that combines growth applications. Since developing his first and standards to improve our ability to computerized adaptive test in 1976, Gage has Research on US identify successful schools. Finally, it will designed adaptive achievement tests that are attempts to identify currently in use by over 1000 agencies discuss the use of an assessment system throughout the United States. This includes the that fosters improvement in education. struggling schools development of the first adaptive test used As long as there have been schools, operationally in K-12 education. In addition, he The US federal government has used has developed procedures for adaptive testing there has been the question of which several approaches to identifying that are currently in use in many operational school is the best. From sports teams ‘schools at risk’ in the past.To use less adaptive tests used in selection, certification, and to beautiful grounds to academic loaded language, let’s call this the ‘search licensure, from military testing to the health competitions, this question is discussed for schools that aren’t very successful’. professions. daily in coffee shops around the world. The current approach that the ‘feds’ are Gage has published or presented over sixty While it is clear that there is no using to identify less successful schools studies dealing with item banking, item response ‘correct’ answer to this question, it is is seen in the AYP (Adequate Yearly theory, and computerized adaptive testing. He not for lack of trying. has served on the editorial boards for several Progress) provisions of the No Child peer-review journals dealing with measurement In the United States, many folks think that Left Behind Act. Under this legislation, and assessment. Gage has also served as a public education is not doing as well as it schools are judged to be successful or developer of the American Council on Education might. However, these same folks will not depending on the percentage of standards for computerized adaptive testing and defend with all their might the quality of students in each grade and subgroup the Association of Test Publishers guidelines for who can successfully reach a defined computerized test development and use. education and the quality of teachers at their child’s school.The reason for this level of proficiency in reading and strong defence is simple. Parents can see mathematics.The details of the level of how their son or daughter grows in proficiency and the content being school from day to day and from year to assessed are left to the states to decide. year.While they might not be able to The approach taken in No Child Left quantify ‘school success’, they can see Behind (NCLB) does not include the their daughter learning to read and growth of individual students. Instead, it growing into a person with profound looks at the percentage of students who capabilities and potential. happen to be able to clear a single While the answer to the question of proficiency hurdle on a single test on a what makes a successful school is not an single day of the school year.While this

Research Conference 2005 2 can be an important piece of information, 3 The current system does not my state department of education, it isn’t the most important element to necessarily lead to better placement are both schools equally effective? look at when measuring school success. for students in low performing schools. A prudent person would probably Researchers investigating this issue have The examples shown above indicate answer this question ‘I don’t know’.We raised the following four concerns: that students who move to schools can’t judge student growth by looking with higher percentages of students 1 Single point-in-time analyses may at a student’s current level, and without meeting the standard may not get a reflect demographics rather than knowing anything about student growth better education. As Kim and effectiveness. They cannot distinguish in a school, we can hardly judge Sunderman (2004a) note, students between schools that accelerate whether that school is successfully who take advantage of transfer skills and those that allow students educating its students. It is possible that opportunities afforded under NCLB to languish. Cross-sectional some of the students in one school often move from schools with measures do not tell us whether exceeded the state performance support for low performing students students entered with high or low standards before they came to this to more affluent schools that do not skills or whether they have gained school. Status relative to the have remedial reading programs, or lost ground as a result of performance standards is not sufficient tutors or supplemental Title I money. instruction. Flicek and Wong (2003) to identify individual or school success. characterise the cross-sectional 4 Expectations of AYP need to be Both student status and student growth percent-proficient model as one of tempered by looking at observed results are needed to paint a complete picture the least valid evaluation methods. in exemplary schools. In his 2003 of a school’s effectiveness. Schools that serve primarily English- address, as president of the American The graph below shows how students’ speaking students who are not in Educational Research Association, fifth grade mathematics status (Average poverty tend to have higher results. Robert Linn illustrated the gulf Score) and growth (Growth Index) The data do not show which between NCLB expectations and compare in a group of several hundred schools have been effective with the observed performance. Using state elementary schools from throughout population that they serve (Kim & and NAEP data from across the the United States (McCall, Kingsbury, & Sunderman, 2004b; Baker & Linn, country, Linn projected that reaching Olson, 2004). Several findings are clear 2002; Buchanan, 2004). 100% proficiency in twelve years from the graph, but the most important would be highly unlikely. He called for 2 The NCLB model does not take the are the following: the use of research to establish goals performance of students above or far that are stringent, but feasible. • Schools with very similar status levels below the standard into account. may differ greatly in the amount of When the goal is to get the greatest One of the primary outcomes of NCLB growth they cause in their students number of students to meet the has been renewed discussion about what (schools A and G, for example) constitutes school success and what standard in a year, schools quite • Schools with cause vary similar growth school accountability models should look sensibly direct efforts at those for students with very different status like. Although the law and its performing just below the cut-off levels (schools A and D, for example) point. Schools earn no credit for implementation have not been • A high-performing school may not improving skills of the lowest straightforward nor without controversy, be one where you would want your performing students or for getting this extended dialogue and the children enrolled (consider school F, gifted student to work to their associated research will definitely improve for instance). capacity. Critics have pointed to this our knowledge of how schools work for feature of NCLB as a disincentive to students. Consider a person comparing These findings mean that some schools excellence, encouraging states to set the Adequate Yearly Progress of two are consistently more effective in low standards in order to schools and asking the following question: causing growth for their students, concentrate on fewer students and If two schools finish the regardless of the students they work look better in public reports instructional year with the same with.This is important information (Marion et al., 2002). percentage of students above the about the success that a school is proficiency levels established by having with its students.

Using Data to Support Learning 3 will result in proficiency within a pre-defined period of time; 8 Quadrant I Quadrant II • Each student is assessed at least twice yearly, and the student’s 6 growth is calculated and compared 4 to the growth target; • The school gets credit toward 2 A D Expected dex success for each student reaching or n Growth 0 exceeding their growth target; and owth I owth

r • The school is judged a success if its G -2 total credit exceeds a pre-defined F -4 performance level. G That is the entire process. It can be -6 Quadrant IV Quadrant III implemented in any setting that has

-8 Overall Average defined curriculum standards and 200 205 210 215 220 225 230 235 240 proficiency levels, and uses a Average Score measurement instrument that is vertically scaled. It allows every student Figure 1 Comparison of average mathematics scores and growth index to ‘count’ in the measurement of school for grade 5 students by school success, by requiring that very high and very low achieving students continue to It is clear that implementation of NCLB The Hybrid Success grow, and it leads every student to provides US schools with a variety of Model proficiency and beyond.While current challenges, and many opportunities to legislation tries to help those students make education better. Students and An example of the category of models who are struggling, the HSM process educators deserve to know what is that include both growth and judges school success by looking at the expected of them, and states’ efforts to proficiency is Kingsbury and Houser’s success of every student in the school. set content standards and standards of (1997) Hybrid Success Model.To The use of HSM should create a performance have clearly helped measure success of a school with this climate with rigorous but attainable schools bring greater focus to model, we measure academic growth of standards, to the benefit of all students. improving achievement. Pursuit of each student in the school.To the improvement requires that public policy, extent that students are growing as An assessment system resources, and sanctions to be applied much or more than expected and that serves students in a purposeful and prudent fashion. growing towards or beyond proficiency, A high-quality assessment system must This study makes clear that a key the school can be judged a success.To meet accountability requirements, but it element that is not represented in determine this: also must serve the needs of each NCLB metrics is individual growth.A • Each student is given a growth student enrolled in the schools. In order more complete accountability system target each year, in each content to achieve this goal, the system might would reward schools for the growth area of interest; include the following components: they nurture in students. Proficiency • The growth target, if achieved, will standards are useful in measuring status, require every student to grow as • Content standards that are fairly but they can create inequity by focusing much as a pre-defined comparison complete, and flexible to change; schools on the relatively small number group; • Performance standards that can be of students who are nearly proficient, • If the student is below the measured along a stable scale that and diverting their attention from those proficiency level, the growth target measures growth across grades; who are far from proficient. will be higher, requiring growth that • Performance standards that have

Research Conference 2005 4 consistent meaning across grades • Assessments that are targeted at Kingsbury, G. G. & Houser, R. (1997). and across subject areas; each student’s instructional level, not Using data from a level testing system • Accurate measurement of student the middle of a grade range. Targeted to change a school district. In The achievement and growth; tests or adaptive tests provide the Rasch tiger ten years later: Using IRT most accurate measurement techniques to measure achievement in • Reporting of results to teachers and available today. schools. Chicago, IL: National administrators in a timely fashion; • A model for examining school success Association of Test Directors. • Measurement of student that incorporates both status and achievement that allows the Linn, R. L. (2003). Accountability: growth. One such model that is identification of areas of strength Responsibility and reasonable currently in use is the Hybrid and areas of concern; expectations. Educational Researcher, Success Model. It incorporates Vol. 32, No. 7, pp. 3–13. • A procedure for changing reasonable growth for each student instruction based on areas of as one aspect of success, and McCall, M. S., Kingsbury, G. G., & Olson, concern and areas of strength; incorporates additional growth that A. (2004). Individual Growth and • Measurement of school success that will bring every student to the School Success. Portland, OR: allows the identification of areas of proficiency level as another aspect. Northwest Evaluation Association. concern and areas of strength; • A reporting system that fosters the Marion, S.,White, C., Carlson, D., • A procedure for using information use of data to improve education.A Erpenbach,W. J., Rabinowitz, S., & about school success to change variety of models for systemic, data- Sheinker, J. (2002). Making valid and policy based on areas of concern based change exist, but each one reliable decisions in determining and areas of strength; depends on providing meaningful adequate yearly progress ASR-CAS • A model for systemic effectiveness reports to the people who need Joint Study Group on Adequate Yearly that allows a school district to them before they get stale. Progress, Council of Chief State measure its improvement across School Officers:Washington, D.C. schools; and References • A procedure to improve a school Baker, E. L. & Linn, R. L. (2002). Validity system based on information about issues for accountability systems. CSE systemic effectiveness. Technical Report 585, 2002. National A simple set of tools can be used to Center for Research on Evaluation, make the assessment system described Standards, and Student Testing. Los above a reality.These tools enable an Angeles: UCLA. organisation to craft a strong assessment Buchanan, B. (2004). Defining ‘adequate system.The system will be able to meet yearly progress’. American School accountability needs and provide Board Journal, February, 2004. accurate information to students and teachers.The set of tools includes: Kim, J. & Sunderman, G. L. (2004a). Does NCLB provide good choices for • A measurement system that includes students in low-performing schools? a stable, cross-grade measurement Cambridge, MA:The Civil Rights scale. An example is found in the Project at Harvard University. NWEA RIT scale, which has demonstrated stability over more Kim, J. & Sunderman, G. L. (2004b). Large than 20 years, and which allows mandates and limited resources: State detailed characterisation of a response to the No Child Left Behind student’s achievement against a map Act and implications for accountability. of skills that common to a wide Cambridge, MA:The Civil Rights variety of curricula. Project at Harvard University.

Using Data to Support Learning 5 From accounting to accountability: Harnessing data for school improvement

There was a time in education when implemented large-scale assessment decisions were based on the best systems, established indicators of judgements of the people in authority. It effectiveness, set targets, created was assumed that school leaders, as inspection or review programs, tied professionals in the field, had both the rewards and sanctions to results and responsibility and the right to make many combinations of the above decisions about students, schools and (Whitty et al., 1998; Leithwood, Edge, & even about education more broadly. Jantzi, 1999). Large-scale assessment Dr. Lorna Earl They did so using a combination of and testing has moved from being an intimate and privileged knowledge of instrument for decision-making about Aporia Consulting Ltd the context, political savvy, professional students to being the lever for holding training and logical analysis. Data played schools accountable for results Dr. Lorna Earl is Director, Aporia Consulting Ltd. almost no part in decisions. In fact, (Firestone et al., 1998). Leaders in and a recently retired Associate Professor in the there was not much data available states, districts and schools are required Theory and Policy Studies Department and Head of the International Centre for Educational about schools. Instead, leaders relied on to demonstrate their progress to Change at OISE/UT. Her career has spanned their tacit knowledge to formulate and the public. research, policy and practice in school districts, execute plans. provincial government and academe. After 25 Not only are schools being judged using years as a Research Officer and Research In the past several decades, a great deal data, many of the reforms also assume Director in school districts, she was the first has changed.The 21st century has been or require a capacity on the part of Director of Assessment for the Ontario dubbed the ‘information age’.There has schools and school leaders to use data Education Quality and Accountability Office. been an exponential increase in data internally to identify their priorities for From there she moved to OISE/UT. and information, and technology has change, to evaluate the impact of the Lorna is a teacher and a researcher with a made it available in raw and unedited decisions that they make, to understand background in psychology and education and a forms in a range of media. Like many their students’ academic standing, to doctorate in Epidemiology and Biostatistics. She others in the society, educators are establish improvement plans and to has worked for over 20 years in schools and school boards and, as a leader in the field of trying to come to grips with this vast monitor and assure progress (Herman assessment and evaluation, has been involved in deluge of new and unfiltered & Gribbons, 2001). School leaders are consultation, research and staff development with information, and to find ways to finding themselves faced with challenges teachers' organizations, ministries of education, transform this information into that are ill-structured with more than a school boards and charitable foundations. knowledge and ultimately into single, right answer.They are faced with Throughout her career, she has concentrated her constructive action. the daunting task of anticipating the efforts on policy and program evaluations, as a future and making conscious vehicle to enhance learning for pupils and for adaptations to their practices, in order organizations. She has done extensive work in Data as a policy lever to keep up and to be responsive to the the areas of literacy and the middle years but has Accountability and data are at the heart concentrated her efforts on issues related to environment.There is not enough time of contemporary reform efforts evaluation of large-scale reform and assessment for adaptation by trial and error or for (large-scale and classroom) in many venues worldwide. Accountability has become experimentation with fads that around the world. the watchword of education, with data inevitably lose their appeal. In this holding a central place in the current context, research studies, evaluations wave of large-scale reform. Policy and routine data analyses offer makers are demanding that schools mechanisms for streamlining and focus on achieving high standards for all focusing planning and actions in schools. students, and they are requiring evidence of progress from schools that Viewed from this vantage point, data are is conceived of explicitly in a language not ‘out there’.They are, and should be, of data (Fullan, 1999). Nations, states, an important part of an ongoing process provinces, and school districts have of analysis, insights, new learning and changes in practice in all schools and

Research Conference 2005 6 districts. Data provide tools for the investigation necessary to plan KNOWLEDGE POOR appropriate and focused improvement strategies. Synthesising and organising 1980s 1980s data in different ways stimulates Uniformed Uniformed reflection and conjecture about the professional prescription nature of the problem under judgement NATIONAL PROFESSIONAL consideration. Over time, this process PRESCRIPTION JUDGEMENT gives rise to defensible plans for changes. 2000s 1990s Uniformed Uniformed professional Accountability redefined: prescription judgement from surveillance to informed professional KNOWLEDGE RICH judgement When all is said and done, school Figure 1 leaders are the ones who are accountable for the work of the school. High-stakes accountability systems can Choosing accountability educational agenda, but they began to create a sense of urgency and provide through informed draw on research and other evidence ‘pressure’ for change. However, real to inform their policies. accountability is much more than professional judgement Barber sees the 2000s as an era of accounting (providing information or Michael Barber (2002), a national policy ‘informed professional judgement’, in justifications in an annual report or a advisor on education in England, uses the which control of education ought to be press release or even student report following graphic to describe trends in returned to educators, but now with cards). It is a moral and professional educational reform over the past 50 years explicit requirements to be informed responsibility to be knowledgeable and as a function of the knowledge base on professionals. And that means using fair in teaching and in interactions with which it has been founded and the locus evidence and research to justify and students and their parents. It engenders of responsibility and decision-making. support educational decisions. respect, trust, shared understanding, and mutual support. He portrays the 1970s as a time of Many school leaders are ready for ‘uninformed professional judgement’, in ‘informed professionalism’ but that which educators operated largely as requires a concerted emphasis on Accounting is gathering, individuals within broad policy becoming and staying ‘informed’. organising and reporting guidelines, relying on their personal information that describes professional perspectives to make Using data to ‘take performance. decisions.The 1980s were a time of charge of change’ ‘uninformed prescription’ where Accountability is the governments took direct control of Using data does not have to be a conversation about what the education and dictated prescriptive mechanical or technical process that information means and how it directions, often without appealing to denigrates educators’ intuition, teaching fits with everything else that any knowledge base other than their philosophy and personal experience. In we know, and about how to own ideological views. National or fact, using data wisely is a human use it to make positive changes. federal programs proliferated, with thinking activity that draws on personal Earl & LeMahieu, 1997 centrally directed curriculum and views but also on capturing and assessment systems. In the 1990s organising ideas in some systematic way, governments still controlled the turning the information into meaningful

Using Data to Support Learning 7 actions and making the interpretation Educators need to use data in many Inquiry habit of mind public and transparent (Senge, 1990). different contexts – to establish their Having data is a beginning, but it is not current state, to determine The first stage of the process is both enough. Schools need to move from improvement plans, to chart simple and profound. Professional being data-rich to being information- effectiveness of their initiatives and to decisions in schools have historically rich and knowledge-rich as well. monitor their progress towards their been based on tacit knowledge, goals.This process can serve a model at knowledge that is embedded in Information becomes knowledge when any stage in their planning and as a individual experiences and involves it is shaped, organised and embedded in guide as they become comfortable with intangible factors like personal belief a context that gives it meaning and using data in their work. In another and values. But, schools today are very connectedness. Using data is not publication we have identified what we complex places and the kinds of separate from planning and from believe are the key capacities for challenges that demand reflection, routine decisions in schools. Instead, leaders in a data-rich world (Earl & consideration of many points of view data are a necessary part of an ongoing Katz, 2002). Leaders for informed and attention to context and evidence. process of analysis, insight, new learning professionalism will need to: As Fullan (2001) argues: and changes in practice. Synthesising and organising data in different ways • develop an inquiry habit of mind, Schools are beginning to discover stimulates reflection and conjecture that new ideas, knowledge creation, • become data literate and inquiry and sharing are essential to about the nature of the problem under • create a culture of inquiry in their solving learning problems in a consideration and provides the vehicle school community. rapidly changing society. for investigating and planning focused improvement strategies. The panels in the graphic are organised An inquiry habit of mind for around the three key capacities and use organisational improvement means The implications for leaders are vast. If the painting metaphor to detail the developing a habit of using inquiry and data are to become part of the fabric of process of using data. reflection to think about where you are, school improvement, however, leaders in schools must become active players in the data-rich environment that surrounds them (Earl & LeMahieu, 1997). Inquiry habit Data Culture of mind literacy of inquiry School leaders as data artists Setting the canvas Blocking the canvas The image grows Using data for improvement puts school leaders into new roles in which What is our purpose? What do we want What is included in they must operate like artists, painting a What roles do to know? this picture? we play? What will we do as a gallery full of pictures to characterise What data do result of our new the complexities and subtleties of the Who are the we need? subject. Artists are always gathering and audiences? knowledge? using data.They are constantly observing, investigating, and responding Planning this The first strokes Displaying the to colours, textures, and images. And, picture picture How do we make they use their considerable interpretive What do we think sense of this? How will we engage talent and experience to draw the we know? the audiences? What does it salient features to the foreground, Where do we How can we show all mean? emphasise important dimensions and want to go? what we have learned? communicate a mood and a message to the audience. Figure 2 Painting as a metaphor for making data-informed decisions

Research Conference 2005 8 where you are going, how you will get Educators can draw on many different mood it creates, and so on.This is also there, and then turn around and rethink forms of evidence – research studies, where technical assistance becomes an the whole process to see how well it is test results, surveys, observations, important part of the process. working and make adjustments. testimonies and witnesses all qualify as Educators are not likely to have the data.The challenges come in deciding technical expertise to do all of the Setting the canvas what data are appropriate and useful necessary analyses and they don’t need for their purposes, ensuring the quality to become data analysts.What is much Artists begin their work by preparing of the data and doing the kinds of more important is that leaders are their canvas and deciding about the analyses and interpretations that will aware of the value and the constraints dimensions and scope of the work. For help them make sense of the data. that are associated with various kinds of educational leaders, setting the canvas data, as they use it to think about their means establishing the background for Blocking the canvas work.Then they can call on others to an issue, deciding why they are serve as ‘critical friends’ to help them dedicating resources (especially time) to Once the team is beginning to get a with analysis and even with the this issue and identifying all of the feel of the contours of the issue, they interpretation. people who need to be involved in one can begin to think about what data will way or another. Before making any help them make the image visible to serious educational decisions, the themselves and others.They are ready A culture of inquiry leadership team needs to be explicit to decide what data they need – to Educational change depends on about their purpose, about who should choose their palette of colours, define collaborative professional learning.We be involved in the decision; about the the scope of the work and make have known for a long time that audience for the judgement and about decisions about composition and mandating change doesn’t work. their own responsibility in the decision- design.This is not as simple a process as Mandates may create an awareness that making process. it may appear. Getting the right data changes are necessary but real change depends on asking the right questions. depends on people working in schools, Planning this picture engaging in new learning, individually and The first strokes In the second panel, the team situates collectively, to refresh their knowledge, the issue by establishing the current The value associated with data come understandings and skills and to deal state of affairs and explicitly deciding from skill in discerning the quality of the with and take charge of change. about the ideal outcome of their work. data, organising it, thinking about what it Becoming inquiry-minded and data It is important to have a clear picture of might mean and using it wisely to make literate are major changes in practice the present before jumping into making decisions. Making sense of data, like that are consistent with the notion of plans and some image of what you are painting pictures, is an iterative process. professional learning communities and hoping to accomplish. One idea leads to another. Some ideas that warrant concerted attention to lose credibility in the process. Others new shared learning.When educators Data literacy get clearer. New information leads the come to the planning process as work in a different direction. investigators, wanting to understand and Most school districts have lots of data interested in working together and with available in their district information At this point, the team considers data in others to find the best solutions, they systems, although they may not be a range of different configurations, find themselves engaged in a very easily accessible or organised in a way spends time trying to make sense of it different kind of organisation; one that that they can be easily used by through analysis, discussion and values dissenting voices and is individual schools. Schools are also likely interpretation and transforms data into determined to generate and share to have various kinds of other formal knowledge that they can use.This is the knowledge, even when the new and informal data that tend not to be process that determines what the knowledge may mean having to make electronically stored – data like picture looks like – what story it tells, dramatic changes and even reinvent classroom records, classroom what images come into foreground and themselves. assessments and program descriptions. which recede into background; what

Using Data to Support Learning 9 The image grows never-ending process and there is never Stuart Foundation. Los Angeles, CA: a single final image. Instead, each image Center for the Study of Evaluation As the team considers the data and is one in a series that will emerge as (CSE) University of California, Los talks about what they are learning, their the team revisits the issue and Angeles. painting begins to materialise and they considers what has changed and what become more aware that there are Leithwood, K., Edge, K. & Jantzi, D. needs adjustment. When schools many possible interpretations and many (1999) Educational Accountability:The engage in ongoing school improvement, possible strategies for improving what State of the Art. Gutersloh: they find themselves in a continuous they do in schools. But even more Bertelsmann Foundation Publishers. cycle of change. It gets easier as they important, the data suggest that there is internalise and embed the technical Senge, P.M. (1990) The Fifth Discipline: work to be done. It is time to use their skills, organisational processes and The Art and Practice of the Learning new learning to change what they values into routines in the culture of Organization. London: Century are doing. the school. Business. Displaying the picture Whitty, G. Power, S. & Halpin, D (1998) References Devolution and choice in education:The The team also finds that they are not Barber, M. (2002). From good to great: school, the state and the market. alone.There are many people in the large-scale reform in England. Paper Buckingham: Open University Press. community who care deeply about presented at Futures of Education what happens in schools.They can start conference, April 23, 2002. Zurich, to think about what they need to Switzerland. communicate to whom and about how others can contribute to their ongoing Earl, L. & Katz, S. (2002). Leading schools quest for deeper understanding and in a data rich world. In K. Leithwood,, better solutions. P.Hallinger, G. Furman, P.Gronn, J. MacBeath, B. Mulford & K. Riley. The The painting metaphor gives the second international handbook of leadership team a process for using data educational leadership and to produce a static image of an issue a administration. Dordrecht, NL, Kluwer. point in time. Once there is an initial image, it becomes the basis for public Earl, L. & LeMahieu, P.(1997). Rethinking engagement and for changing practices. assessment and accountability. In A. Hargreaves (Ed.), Rethinking In this metaphor, the picture is the Educational Change with Heart and stimulus for action, not the end result. Mind. 1997 ASCD Yearbook. The process now shifts to sharing what Alexandria,VA. has been learned, listening carefully to the responses from the various people Firestone,W., Mayrowetz, D., & Fairman, who care and deciding what has to J. (1998). Performance-based happen next.This is not a showcase assessment and instructional change: event; it is an ongoing, active exchange The effects of testing in Maine and of ideas and decisions about action. Maryland. Educational Evaluation and Policy Analysis, 20(2), 95-113. A gallery full of Fullan, M. (1999). Change forces:The paintings sequel. London. Falmer. Using data to make decisions is hard Herman, J. & Gribbons, B. (2001). work. Although it may be tempting to Lessons learned in using data to mount the picture and accept the support school inquiry and continuous accolades, educational change is a improvement: Final report to the

Research Conference 2005 10 What is the nature of evidence that makes a difference to learning?

Schools are awash with data, and the problem that should be in front of us. accountability movement is requesting Asking whether and how to send data that they collect even more.This back to schools is the wrong question. presentation locates the teachers as A major theme of this presentation is critical in the ‘evidence’ cycle. It that we must be more mindful of the demonstrates a model for assisting ‘interpretations’ we wish to make from teachers to ascertain the nature and any data collected as it is the use evidence to make a difference to ‘interpretations’ that are critical, rather John Hattie learning.This model permits other key than data itself. Of course, the quality of stakeholders (principals, Ministries, the data reflects on the validity of the University of Auckland parents, students) to then share this interpretations, but it is the latter which evidence. It outlines studies in schools should be uppermost in our minds John Hattie is Professor in the Faculty of that have been using the model and when we (a) collect data, and (b) Education at Auckland University, New Zealand, and Director of Project asTTle (Assessment Tools then develops a system-wide return interpretations to those we wish for Teaching and Learning). His areas of research accountability model based on this to influence. include measurement models and their evidence that makes the difference to In the meantime, while volumes of data application to educational problems, meta-analysis, teaching and learning. and models of teaching and learning. Over the are extruded about and from schools, past four years Professor Hattie has headed a Schools are awash with data, and I have teaching continues without the benefits team introducing a model of assessment for yet to find a Department or Ministry of of such data.There is still a philosophy teachers in all schools in NZ, and thus providing Education which does not have so that assumes teachers know how and schools with evidence based information about the teaching and learning. much data that debate is more what data to collect to best enhance concerned with issues such as data learning, and many of these assumptions Professor Hattie’s current research projects cover: warehouses, executive information are based on folk philosophies, poor • Meta-analysis of ADD, item-order effects, systems, web pages, data portals, and measurement, and shaky data.We still physical training on body image, and synthesis of meta-analysis on teaching and learning the use of Access, Oracle, or other teach in a manner we did 150 years • Weighting models of self-concept, social mega-data systems. Soon after this ago (see Cuban & Tyack, 1995), with a desirability estimates, goodness-of-fit indices of bounty is collected, someone begins to preponderance of talking (about unidimensionality in structural equation ask “How can we return it to the 70–80% of the time, see Yair, 2000), modelling and item response modelling schools?” At last year’s Round Table on deciding on activities that aim to engage • Validation of models of teacher expertise, and Assessment in Sydney, for example, rather than choosing activities that development of professional teaching standards there were many discussions about the reflect on curricula intentions that aim volumes of data that can be readily to challenge.We are loosing the minds returned, and how it could be and hearts of the students (particularly ‘massaged’ and presented to schools in during early adolescence, when dis- the most digestible form. It was also engagement is already a ‘cool’ attribute) noted, in passing it seemed, that the and we are also losing the voters as schools were not that enamoured with their belief about the quality of receiving so much data – they were not schooling declines. sure what to do with it, and were Because of such criticism (and also concerned by the time and workload because it seems good practice), it is involved in reading and digesting it. not uncommon for systems then to Hence, there is the desire to find more invent ‘accountability’ systems to drive acceptable ways to return ‘their’ data the teachers to get more and more back to the schools. It seems, once learning out of their charges. One form again, there is an effort to solve the of accountability assumes that if only problem in front of us rather than the we could name, shame, and blame with

Using Data to Support Learning 11 evidence, we could get those teachers downward (e.g.,‘accommodating’ special (Hattie & Peddie, 2003) published a operating at higher levels of efficiency. education students out of the test study based on school reports to Another form of accountability assumes room, retain back students from moving parents from 156 schools in New that if only we could collect sufficient up a grade, using suspensions and not Zealand. Only 12 included information system-wide evidence, we could enrolling students who may detract relating to the official curriculum levels; convince the parents/voters not to be from the scores of high schools, and so half included no information on critics. Both miss the mark. on). It is not the model worth achievement relative to any standard; moving towards. half talked about students in agricultural Most depend on the thermometer terms (developing, needing more, theory of traction – although the abject As Robert Linn (2003) has emerging, growing); and half included a failures of this model are already demonstrated, it would take an specific section relating to effort. On causing untoward damage to our innovation of atomic bomb proportions the basis of these reports 98% of profession of teaching, to the role of to get the average yearly gains needed students had positive comments about principals, and leading to students'lack to reach the stated goals by 2012 – the their achievement, were putting in of engagement. Perhaps the most visible target year (he estimated that it will effort, and were ‘a pleasure to teach/joy form of accountability that illustrates take 150 years at the rate of annual to have in my class’.With few these issues is the US ‘No Child Left yearly progress of the past 10 years to exceptions, the majority of students in Behind’.While I see some merit in reach the targets set for 2012). As these schools were achieving above some of its claims (e.g., ensuring all Australia moves towards national average! No wonder parents demand students, and not just the ‘average’ testing, it will become more awash with more ‘tests’, accountability, and ‘teacher- student, succeeds within a school) the data, it will de-contextualise schools, proof’ information from our schools. implications of this USA-wide lead to more claims for ‘school choice’, accountability system have become increase the flight out of the public Second, there is not a lot of evidence most clear in its negative effects. It has schools, will lead to more schools in that the massive increases in made the teachers teach what they lower socioeconomic areas stumbling state/federal monies have made a expect is coming in the test; it ensures and more schools in higher difference to the quality of teaching and students are focused on this teaching; it socioeconomic areas cruising, and, most learning. Hanushek (2005) has judges the success of the school in of all, it will feed the belief that the presented information (in current terms of whether teachers are doing quality of schooling in the dollars allowing for inflation) of changes this job of teaching to the test; it rids State/Territory/Australia is declining. I in public schools’ resources in the the school day of ‘peripherals’ that are see none of this enhancing the quality United States over the past 40 years not tested (such as physical education, of teaching and learning. (Figure 1).The achievement curve music, art, and self-respect); it cuts (from NAEP) has remained constant But we need to remind ourselves who vocational and career education over this period. If we, as educationalists is asking for more tests – it is incorrect programs in high schools which are in classrooms and schools do not to blame the politicians.They are clearly desperately needed by many students provide the evidence that increased listening to the voters – who want whose alternative is to drop out, and it resources make a difference to student more accountability (which they punishes those who do not do their job learning and outcomes, then we will interpret as tests and data) in the same and teach to the test (see Hattie, soon be on the back foot, arguing way politicians wish to return evidence Brown, & Keegan, in press; Linn, 2003; why there should not be decreases that their investment in schooling is Shepard, 2000). It ensures that (a) there in resources. paying off. Let me make two is a quick gain as all learn how to ‘game’ claims here. My major theme is that we need the test, (b) that the curriculum is models of school/teacher/student altered downwards to ensure that First, schools have failed in their efforts accountability located at the system there is reasonable success for more to provide appropriate and defensible and school level that maximises the students, (c) it introduces procedures to data to parents about their children – probability of enhancing learning and remove those who may bring scores hence the clamour for more tests.We outcomes. Indeed, we must develop an

Research Conference 2005 12 Public school resources, US NAEP performance 1960–2000 (17 year olds)

320 1960 1980 2000 310 300 290 Pupil-teacher ratio 25.8 18.7 17.3 280 % master’s degree 24 50 56 270 260 Median experience 11 12 15 250 math reading science writing Spending/pupil $2,235 $5,124 $7,591 1970 1980 1990 1999

Figure 1 Indicators of changing public school resources and NAEP achievement over the past 40 years in the USA accountability system that is located from the student level upwards, directly involving and influencing the teacher Identifying that which matters and principal level, as such a system is more likely to have major effects on the quality of teaching and learning. Such a Teachers system, which I intend to outline, can Students also serve the systems’ needs of providing evidence of curricula, Home resources, and equity issues. Peers Schools Principal What makes a difference to teaching Figure 2 Percentage of achievement variance and learning? The reason for locating the power of data to enhance student outcomes at This is a summary of what is, not what discussions of ‘choice’ too often means the teacher level comes from the many should be – as I certainly can note the that schools get to choose the students recent studies on the epicentre of power of peers as co-learners, the role they want (and many students in certain casual effects on learning: the teachers. of principals to make a difference to neighbourhoods are denied the choice At this same ACER conference, two instructional leadership, and so on. It is they want). Maybe there is merit in clear, that the major factor in this ‘choice’ but most of us get what comes years ago I presented on the factors equation is the student – but most of through the school gates from the local that make a difference to teaching and you have to take what the areas. Similarly absurd notions of brain learning and divided them into six parts neighbourhood produces and waves, learning styles, multiple of the cake (Hattie, 2003):

Using Data to Support Learning 13 intelligences and other pop-educ claims chokepoint to the enhancement of teaching, and rarely does random are more befitting brain surgeons than learning outcomes for students.While assignment occur. It is this form of the cut and thrust of the teaching and there can be sharing of activities and teacher-available data that is of interest learning conundrums.The major stories about students and incidents, it to my forms of accountability. influence on student learning is the is rare to hear discussions among Such classroom-based data is also teacher, and here is where I wish to teachers about the levels of contested – and while it can consist of locate the issue of ‘What data would understanding, the degree of challenge scores on tests, it can also consist of support a teacher to enhance teaching and expectations required and attained teacher judgements, student ratings, and and learning?’ and thus how can we – such that each year teachers revisit so on – provided such evidence can be devise systems to ensure that such data the students in terms of their internal defensibly accumulated and is open to is obtained, and when obtained that it beliefs about what levels of scrutiny. It is the judgements or makes a difference? While there are performance are required – allowing interpretations based on these data that other sources of data useful to a system, students to gain or drop according to is of most interest.The asTTle model the key to any accountability model these (often untested) beliefs about the outlined in the presentation allows such should orient around this question. desired levels of progression (Robinson evidence to be defensible accumulated & Lai, 2005;Timperley, 2005). One of and contested – and this is how it the major purposes of an accountability What are ‘learning should be.We must contest the system is to assist in articulating a outcomes’? evidence – as that is the basis of a common language of progression. common understanding of progression. This question begs the question:What is it is that we wish to enhance? This The nature of ‘data’ question has occupied the minds of The location of curricula reformers for decades, and we Before venturing into the ‘evidence’ starts in seem to experience a once-a-decade- recommended model, it is important to the classroom bump where the old curricula is comment on the nature of ‘data’, as this repackaged, new names invented, much is a most contested term. A current fad The argument in this presentation is is added and little is subtracted, and the radiating out from the United States is that the location of evidence that classrooms continue on much as the notion of evidence-based decision- makes a difference to teaching and before.The latest craze, begun by the making – and this term has been learning must be located at the ‘teacher’ OECD is to include key competencies hijacked to mean a very narrow form level. Of course, the students are or ‘essence’ statements and this seems, of evidence. Liberty and Miller (2003), implicitly involved – but they are not the at long last, to get closer to the core of for example, consider ‘evidence-based’ core.This is because it is most common what students need. Key competencies relates to meeting peer-review to locate students in groups (i.e., include thinking, making meaning, standards, and including evidence classrooms) critically influenced by the managing self, relation to others, and directly impacting on children’s learning teacher. Indeed, my theme is that if we participating and contributing. Indeed, (not correlates, see Scriven, 1988).This form the accountability model around such powerful discussions must ensue cuts out so much of today’s literature providing teachers with excellent around the nature of what are ‘student and I note an excellent summary of the diagnostic and formative evidence, we outcomes’ as this should inform what surviving literature by Alton-Lee (2003). have not only an excellent model but kinds of data need to be collected to But an extra condition has been added, one that influences teaching and thence enhance teaching and learning. that of the type of research designed to learning. Basing a model on students collect data: preferably random can help those students who learn in a Outcomes from curricula must have a assignment to various groups (Mosteller diagnostic and formative manner about sense of achievement progression. From & Boruch, 2002).While this may be such accountability evidence but this our New Zealand research, it is most exemplary, it is not the only design of would exclude most students. Similarly, defensible to claim that a common merit. Moreover, in classrooms, teachers basing it on parent’s privileges (those understanding among teachers of still need to base their evidence on who have the home-resources to add progression is probably the greatest data from their students and from their value to this evidence) would again

Research Conference 2005 14 exclude so many parents, particularly least it moves the discussion beyond strategies, they are hierarchical, and that those who do not have command of the status of the students, which is what it is desirable that students, as early as the language of schooling and learning must accrue from state-/nation-wide possible, learn to strategise using the (Clinton & Hattie, 2005). models and to include the critical highest step in the hierarchy. My point is questions relating to growth. not to question the merit of this claim The first part of the model is to address (although see Ell, 2002; van Gardaren, teachers’ expectations and target setting, The asTTle model is based around 2002) but to highlight that number as these are key drivers in the three major questions:Where are we operations are considered in most need enhancement of learning – or can be the going? How are we going? and Where of curriculum innovation. greatest barrier to such enhancement to next? Thus, target setting is critical, as (Rubie, Hattie, & Hamilton, in press). is evidence of the gap between current We have accumulated evidence based These expectations also are and targeted performance, and the on about 25,000 students undertaking underpinned by the teacher’s conception manner in which teachers are going to over 1500 items from across the of progression. During the conference reduce this gap for all students. Other mathematics curricula (from the asTTle presentation I will demonstrate a target graphs from the asTTle application will norming sample).Then we can present setting process for individual students be shown that will demonstrate how a the growth of number (in its three that allows immediate aggregation to the national system can provide evidence forms) and can see a steep learning class and school level to ask whether the on these issues, in an immediate way, to curve right throughout the Years 5 to 12. target setting is reasonable, enhancing, teachers and students.There is evidence But in Geometric Knowledge we can and defensible.The critical features of individual student achievement, class see a shaky start in primary school; there include the following: it is in the language achievement, the distribution of is a decline and then no growth during of teaching and learning and not achievement across cohorts, school- Years 5 to 7; and then over the latter assessment; it leads to discussion among wide analyses, and linkages to years of schooling, a less steep growth fellow teachers about the nature of appropriately challenging curricula than for Number.There should be a teaching and learning; and it provides materials.These analyses can be major set of questions here about the school leaders with information to form conducted at the individual as well as at teaching of geometry in primary schools a school-wide discussion about targets. the cohort, class, and school levels. – perhaps dropping it completely! Similarly, I will demonstrate a school We can drill down deep below this profile also provided by the asTTle Evidence-based level of aggregation and also ask about package that shows current performance curricula development specific objectives within Number and and how it can be used to evaluate the within Geometry, and this is the nature degree of attaining these targets. Curriculum is also a contested domain, of evidence-based curriculum Similarly, for both sets of evidence the and too often, it is resolved by asking a development. Such discussion, based on national norms (for the country, or for group of experts to devise a new evidence about learning can contest ‘schools like mine’ can be interpolated). version – often tinkering at the edges, deeply held beliefs about what should choosing new names to dominate the The emphasis is on growth, and avoids be undertaken in the name of centre, and the teachers do much the many of the current problems with curriculum form, and can lead to asking same as they did before. Instead, it is value-added models.The latter have direct questions about where the argued, curriculum development should been too dependent on measuring only curriculum needs to be reformed, and start with evidence based on what at two time points, with all the where to be left alone. students know and can do. incumbent problems (Cronbach & Furby, 1970).The current model, Take mathematics as an example. It is Evidence based within- however, incorporates many time points easy to imagine a group of ‘experts’ school development and is thus conducive to an interrupted arguing for some new twist or time series analysis – which has much development in mathematics.The There are many within-school debates more power to provide information on current vogue seems to be number about the nature of evidence that the value added by teachers and strategies, and in New Zealand a group makes a difference to learning? Let me schools (see Hattie & Rowe, 2004). At has decided there are six of these illustrate six.

Using Data to Support Learning 15 1 The importance of asking (.40 effect-sizes). Anything less is holding Rigdon, 1997;Timperley, 2004).Turning relative questions of back a student, as at least half the effects such evidence into tools for teachers is effectiveness can attain growth greater than .40. the key to evidence-based teaching. Timperley (2005), for example, worked If you could sum up all the studies on 2 The use of effect-sizes in with teachers in one low what makes a difference to students’ classrooms to underpin the socioeconomic area, and began by achievement, there are very few that do discussion on effectiveness collecting a range of literacy not report some success. Nearly achievement information: The power of effect-sizes (the everything enhances achievement, thus This information, on individual difference between two groups or any teacher claiming that they can show student profiles and held in each between two time points divided by evidence of enhanced learning is not teacher’s filing cabinet, was vast saying much. For example, based on my their pooled standard deviation) is and encompassing.The more syntheses of evidence on this question relatively easy to implement in schools. formal assessments (using (Hattie, 1999; in prep), I have Phillips, McNaughton, and MacDonald standardised measures like the determined the effect-sizes of over 100 (2001) have used effect-sizes in their Reading Observation Survey, Clay, major innovations from over 300,000 implementation of school-wide literacy 1993) was considered by the studies. For example: zero is when there programs in schools from lower teachers as something collected is no effect on achievement, a negative socioeconomic areas, with much for the assistant principal’s use, not theirs.Teachers considered that effect is when the innovation reduces success.Their success is not only to the most relevant planning achievement, and a positive is when the provide policy makers with evidence of the success, but more importantly to evidence was anecdotal innovation enhances achievement.These observational data collected on a innovations include structural changes assist teachers in the delivery of the literacy program. daily basis in their classrooms.They (reducing class size, ability grouping), considered such data was relevant curricula innovations, teacher effects Another advantage of using effect-sizes and trustworthy in contrast to the (questioning, direct instruction, reciprocal is that they force schools to have clear more formally collected teaching), and so on.Virtually everything goals and standards of student information.The assistant principal, we do enhances achievement (note how performance, as only then can teachers however, was concerned about the few are below the zero effect-size).The collect and review information to low quality of these anecdotal critical question is whether we can inform themselves about their levels of observation data particularly implement those effects that enhance success with their students in reaching because they did not give the teachers an understanding of the achievement by more than the average those standards (Newmann, King, & adequacy of their students’ progress in comparison with other 1.00 students in the country.When Average Effect-size = .40 explaining the national data for 0.80 their students, teachers had many reasons to exclude the 0.60 information (the national kids are not like mine, I teach to the best 0.40 of my ability given whom I am given, I should not ‘teach to the 0.20 Effect-size test’, the tasks are not ‘authentic’, others fail to understand what my 0.00 kids can do, I have too many -0.20 students in my class, I need more time if this is going to impact on -0.40 me, and so on. Timperley (2005) Figure 3

Research Conference 2005 16 Timperley highlighted the power of diseases that affected Maori after is that teachers need to move away ‘surprise’ to ensure more ownership by the arrival of the British colonists. from considering achievement data as the teachers:‘One of the ways in which Success criteria: By week 3 of this saying something about the student, and data can be powerful in creating change unit, students will be able describe the start considering achievement data as is the possibility that they may be trends in Maori population between saying something about their teaching. If discrepant with previous beliefs and 1820 and 1920. By the end of the students do not know something, or create surprise, thus challenging those unit, the students will be able to cannot process the information, this beliefs’ (Schutzwohl, 1998;Timperley & explain the effect of British colonisation should be cues for teacher action, Robinson, 2001).When teachers on Maori health at the beginning of particularly teaching in a different way compared their students’ growth with the twentieth century and how it (the first time did not work!). Merely that of students of other teachers, they influenced Maori population trends ascribing to the student the information and make predictions about the health were surprised.The most important that they can or can not do something effects on indigenous peoples by aspect of this study was moving the is not as powerful as ascribing to the colonising countries. teachers from expressing outcomes in teacher what they have or have not terms of the students they received, the How success criteria will be taught well. working conditions of teaching and assessed: Students will be able to write a paragraph that relates three A similar powerful idea is that teachers learning, to a set of contingencies based pieces of information: the arrival of have differing conceptions of on learning outcomes. British diseases, the population assessment (Brown, 2004), and trends of Maori, the contribution of understanding these differing 3 The importance of learning previously unknown disease to the conceptions may be critical before intentions and success criteria decline in population. encouraging teachers to collect more Using effect-sizes, or any evidence of Evidence is now easy – it relates the evidence. Brown (2004) has discovered enhancement comes back to the issues teacher’s intention (from the four major conceptions: assessment of merit and worth of the outcomes. curriculum) to the task and activities, improves teaching and learning, Within the classroom we have clearly specifies the criteria the teacher assessment makes schools and teachers articulated these as learning intentions would use to judge student learning, accountable, assessment makes students and success criteria. Our work in and indicates how data could be accountable, and assessment is schools too often shows that students collected specific to these criteria. And irrelevant. If teachers consider rarely know the learning criteria for a even more powerful if the learning assessment is irrelevant, then this needs particular lesson, are confused as to intention, success criteria, and to be attended to before inviting such what success would look like for this assessment are shared with the teachers to consider evidence-based intention (often claiming that something students (as they commence the task). models of teaching and learning.They long, spelled correctly, and neat is At a minimum, it stipulates the notion will depend overly on anecdotal indicative of the success criteria), and of what the learning outcomes are, can evidence, believing that completion of do not see how the assessment relates lead to debates about sufficiency, assigned tasks (regardless of difficulty to the success criteria nor the learning challenge, appropriateness, time, and challenge) and similar such intentions.We have spent much time resources, and can indicate to other engagement-related activities are more writing about making learning intentions teachers and students (and parents) the critical that any ‘surprises’ and evidence and success criteria explicit, and have level and depth of the learning. based on dependable testing seen many classes and schools procedures. transform with these simple but 4 Assessment data is optimised It may be necessary for teachers to powerful ideas (Clarke,Timperley, & when teachers conceive such listen to students more closely and thus Hattie, 2003).To illustrate: data as about them (and not use other sources of classroom Learning intention: ‘To understand about the students) evidence. Bishop, Berryman,Tiakiwai, the causes and effects of events that and Richardson (2003) interviewed have shaped the lives of a group of One of the powerful ideas in evidence- Mäori students about how to best people.’ The context might be the based models of teaching and learning improve their educational achievement.

Using Data to Support Learning 17 The students claimed that the major that the student has dependable applied (King & Newmann, 2000). changes needed to be how teachers evidence on which to base their Every school contains a diverse related and interacted with Mäori decision-making. Instead, we so often mix of teachers and students with students in their classrooms.Too often promote the power of self-regulation varying competencies and attitudes and a unique set of these interactions were based on deficit but fail to realise that it is premised on social, cultural and political theorising by teachers about these evidence of learning performance. conditions, all of which have a students, and too often these powerful influence on teaching and relationships were based on denying 6 Enhancing teacher learning (Bryk, Sebring, Kerbow, that the students had a rich cultural performance to improve Rollow, & Easton, 1998; Lytle & heritage that they brought to the student learning is Cochran-Smith, 1994).These classroom.This led to low expectations conditional upon evidence complex conditions often present of Mäori students and collecting obstacles for teachers attempting evidence to confirm these beliefs, thus Timperley (2005) recently noted that to apply new ‘generic’ learning creating a downward spiralling, self- ‘the notorious lack of success of from conventional professional development programs to their fulfilling prophecy of Mäori student teacher PD is too well known to keep own classroom practice (Clement achievement and failure. Based on these hiding or assuming that we should & Vandenberghe, 2000; DuFour, student experiences, the Team continue as if this evidence is not 1999; Hord, 1997; Lashway, 1998; developed a professional development aplenty (DuFour & Eaker, 1999; Lewis, Leo & D'Ette, 2000; Leonard & intervention, that when implemented 1997; Louis & Leithwood, 1998; Leonard, 1999; Louis & Leithwood, with a group of 11 teachers in four Timperley, 2005;Wald & Castleberry, 1998; McLaughlin, 1993; schools, was associated with improved 2000).’ A major reason for this lack of Rosenholtz, 1989; Smylie, 1995). success is that too much professional learning, behaviour and attendance From such an analysis,Timperley development for teachers does not outcomes for Mäori students. recommends developing a culture of have enhancement of student learning Similarly, Irving (2005) has found that using data to support learning and how as the contingency of success.Too students are very adept at identifying this ‘needs a mind shift that will rock the often, PD is more related to working excellence in teaching and the major foundations of what we do and how we conditions (of teachers and students), question may be ‘Why primary and do it’. She proposed five elements of and correlates of student learning. secondary teachers do not use more professional learning communities: Indeed, in her recent synthesis of student evaluation of teaching?’ Irving literature Timperley was able to locate 1 The development of shared values used the standards of the National only 17 articles that related the and expectations about children, Board for Professional Teaching effects of PD on student learning! learning, teaching and teachers’ roles Standards to create a student She continued: and the relationship of these to the evaluation instrument (for high school environment (Bryk et al. 1999; Louis Generic delivery models of much mathematics). Using a sample of NBC et al. (1996). and non-NBC teachers, he found that external professional development 2 The collective focus on student students could dependably discriminate have often proved ineffective in creating the depth of shared learning that then becomes part between these two groups of teachers. professional knowledge needed if of the normative control of the The data are there but is the courage staff are to address complex professional community (Bryk to use it there? teaching and learning issues in et al., 1999). their schools, particularly in those 3 Collaboration, whereby professional 5 Movement towards student schools facing challenging empowerment of teaching circumstances (DuFour & Eaker, communities foster the sharing of expertise and faculty members call and learning 1999; Lewis, 1997; Louis & Leithwood, 1998;Wald & on each other to discuss the If you believe in student self-assessment, Castleberry, 2000). Part of the development of skills and create self-monitoring, self-teaching, self-learning, depth required is an understanding shared understandings of and self-responsibility – then it is critical of the contextual conditions in effective practice which the new learning must be

Research Conference 2005 18 4 Deprivatised practice, and much curricula, and by comparisons to about what is worth learning in our time and opportunity to talk to appropriate national and local standards schools, and to develop a common each other about teaching. of performance.The major sources of language about the progression of this 5 Reflective dialogue, implies self- evidence relate to diagnosis and learning as students advance through awareness about one’s work as a formative assessment models and are their schooling. teacher through engaging in in-depth centred on three major questions: conversations about teaching and Where are we going? How are we References learning (Louis, Marks et al., 1996). going? and Where to next? All analyses Best evidence can be conducted at the individual as Alton-Lee, A. (2003). synthesis. These all require a serious commitment well as at the cohort, class, and school http://www.minedu.govt.nz/ to evidence, debates about the levels.The evidence can also be used to index.cfm?layout=document&docume contested nature and value of evidence, contest deeply held beliefs about what ntid=8646&data=l and actions based on evidence.This is a should be undertaken in the name of Ministry of Education,Wellington, major culture shift for many schools, curriculum reform, and can lead to New Zealand. where privatised teaching occurs, asking direct questions about where the Bishop, R., Berryman, M.,Tiakiwai, S., & discussion is more about curriculum curriculum needs to be reformed, and Richardson, C. (2003). Te Kötahitanga: and students and less about teaching, where it should be left alone. The Experiences of Year 9 and 10 and evidence of growth in learning is Mäori Students in Mainstream rarely shared across the school. Within schools, this evidence-based Classrooms accountability model can be used to ask . Poutama Pounamu Research and Development Centre/ Concluding comments relative questions about the effectiveness of teaching, can be recast Mäori Education Research Institute The major argument of this in terms of learning intentions and (MERI), School of Education, presentation is to move the discussion success criteria, and evidence provided University of Waikato. away from data towards interpretations, about the quality of teaching rather Brown, G. L.T. (2004). Teachers’ from student outcomes to teaching than the quality of the students that a conceptions of assessment. Unpublished successes and improvements, and from school receives. It is important to doctoral dissertation, University of accountability models located about consider teachers’ conceptions of Auckland, New Zealand. schools to located first in the classroom assessment, and to use evidence as the Bryk, A., S. (1999). Policy lessons from to support such evidence-based basis for professional development Chicago's experience with teaching and learning.The asTTle programs. Perhaps students’ evaluations decentralization. In D. Ravitch (Ed.), model, which has been developed in of teaching could be also used as part Brookings Papers on Education Policy New Zealand, will be used in the of this evidence base. Keynote presentation to demonstrate (Washington, DC: Brookings Institution The move to collecting more data such a model. By locating evidence in Press), 67-128. needs to be stopped and the move to the classroom we can improve the Bryk, A. S., Sebring, P.B., Kerbow, D., making more defensible interpretations quality of information and Rollow, S., & Easton, J. Q. (1998). about teaching and learning upgraded interpretations sent to students, Charting Chicago school reform: to priority levels. Evidence that informs parents, Ministries, Ministers, and thence Democratic localism as a lever for teachers about their teaching is the the community.We can influence the change. Boulder, CO:Westview Press. most critical evidence that can be major agent that influences student and (ED421594). provided and too many current models learning – the teacher, can highlight the ignore such evidence. It is possible to Clarke, S.,Timperley, H., & Hattie, J. A. C. debate about what is worth teaching, devise a national accountability model (2003). Assessing formative assessment. and, most importantly, can begin to based on evidence critical to teachers, Hodder Moa Beckett, New Zealand. establish a teacher-shared language and such a model can also serve to about the achievement progression. Clay, M. M. (1993). An observation survey evaluate the state of learning in the of early literacy achievement. Auckland, The model is based on target setting, nation, to provide evidence for New Zealand: Heinemann Education. on ensuring the implementation of the curriculum reform, to create debate

Using Data to Support Learning 19 Clement, M., & Vandenberghe, R. (2000). Hattie, J.A.C. (in prep.). Influences on Leonard, L. J., & Leonard, P.E. (1999). Teachers' professional development: A student learning. Book in preparation. Reculturing for collaboration and solitary or collegial (ad)venture? Teaching leadership. Journal of Educational Hattie, J.A.C. (2003, October). Teachers and Teacher Education, 16, 81-101. Research, 92, 237-242. make a difference:What is the research Clinton, J. M., & Hattie, J. A. C. (2005). evidence? Keynote presentation at the Lewis, A. (1997). A new consensus When families learn the language of Building Teacher Quality:The ACER emerges on the characteristics of schools: Evaluation of the Flaxmere Annual Conference, Melbourne, good professional development. The Project. Ministry of Education, Australia. Harvard Education Letter, 13(3), Wellington. May/June. Hattie, J. A. C., Brown, G.T., & Keegan, P. Cronbach, L.J., & Furby, L. (1970). How (2003, January).The Advantages and Liberty, K., & Miller, J. (2003). Research should we measure ‘change’ – Or Disadvantages of Statewide and/or as a resource for evidence-based should we? Psychological Bulletin, 74, National Testing. Hawaii International practice. Dunmore Press, Palmerston 68-80. Conference on Education, Honolulu. North, NZ. Cuban L. and Tyack, D.B. (1995) Hattie, J.A.C., & Rowe, K. (2004). Review Linn, R.L. (2003). Accountability: Tinkering toward utopia: a century of of current indicators constructed using responsibilities and reasonable public school reform. Cambridge Mass: NEMP data: Consultancy report to the expectations. Educational Researcher. Harvard University Press. New Zealand Ministry of Education. 32 (7), 3-13. Ministry of Education,Wellington, New DuFour, R. (1999). Challenging role: Louis, K. S., & Leithwood, K. (1998). Zealand. Playing the part of principal stretches From organizational learning to one's talent. Journal of Staff Hord, S. M. (1997). Professional learning professional learning communities. In Development, 20(4), 1-4. communities: Communities of continuous K. Leithwood & K. S. Louis (Eds.), inquiry and improvement. Austin,TX: Organizational learning in schools (pp. DuFour, R., & Eaker, R. (1999). Professional Southwest Educational Development 275-285). Lisse: Swets & Zeitlinger. learning communities at work: Best Laboratory [On-line]. Available: practices for enhancing student Louis, K. S., Kruss, S. D., & Marks, H. M. http://www.sedl.org/pubs/catalog/items achievement. Bloomington, IN: National (1996). School wide professional /cha34.html. Educational Service (ED426 472). community. In F.M. Newmann & Irving, E. (2005).The development and Associates (Eds.), Authentic Ell, F. (2002). Strategies and thinking validation of a student evaluation achievement: Restructuring schools for about number in children aged 9-11 instrument to identify highly intellectual quality (pp. 179-207). San years (Tech. Rep. No. 17). Auckland, accomplished mathematics teachers. Francisco: Jossey-Bass Publishers. NZ: University of Auckland, Project Unpublished Doctoral dissertation. asTTle. Lytle, S. J., & Cochran-Smith, M. (1994). University of Auckland. Inquiry, knowledge, and practice. In Hanushek, E. (2005, May). Alternative King, M. B., & Newmann, F. M. (2000). Sandra Hollingsworth & Hugh Sockett Policy Options: Class Size Reduction Will teacher learning advance school (Eds.), Teacher research and educational and Other Possibilities. Keynote goals? Phi Delta Kappan, 81, 576-580. reform: 93rd NSSE yearbook, Part l (pp. presentation at the International 22-51). Chicago: National Society for Conference on Learning Effectiveness Lashway, L. (1998). Creating a learning the Study of Education, University of and Class size. Hong Kong. organization. ERIC Digest, Number Chicago Press. 121: ED420897. Hattie, J.A.C., & Peddie, R. (2003). McLaughlin, M. (1993).What matters School reports:‘Praising with faint Leo,T., & D'Ette, C. (2000). Launching most in teachers' workplace context? damns’. Set: Research information for professional learning communities: In J.W. Little & M. McLaughlin (Eds.), teachers, 3, 4-9. Beginning actions. Issues About Teachers work (pp. 79-103). New York: Change, 8(1), [On-line], Available: Hattie, J.A.C. (1999, June). Influences on Teachers College Press. http://www.sedl.org/change/issues/ student learning. Inaugural Professorial issues81/. Address, University of Auckland.

Research Conference 2005 20 Phillips, G., McNaughton, S., & Smylie, M. A. (1995).Teacher learning in van Gardaren, D. (2002). Paper and MacDonald, S. (2001). Picking up the the workplace: Implications for school pencil strategy assessment items for the pace: Effective literacy interventions for reform. In T. R. Guskey & M. Huberman New Zealand Number Framework: Pilot accelerated progress over the transition (Eds.), Professional development in study. (Tech. Rep. No. 18). Auckland, into decile 1 schools.Wellington: education: New paradigms and practices NZ: University of Auckland, Project Ministry of Education. (pp. 92-113). New York:Teachers asTTle. College, Columbia University. Mosteller, F., & Boruch, R. (2002). Wald, P.J., & Castleberry, M. S. (Eds.). Evidence matters: randomized trails in Timperley, H. S. (2004). School-based (2000). Educators as learners: Creating education research. Brookings Institute, influences on teachers' learning and a professional learning community in Washington DC. expectations. In F. Columbus (Ed.), your school. Alexandria,VA.: Association Teaching and Teacher Issues: Nova for Supervision and Curriculum Newmann, F. M., King, M. B., & Rigdon, Science (forthcoming). Development. (ED439099). M. (1997). Accountability and school performance: Implications from Timperley, H. S. (2005). Instructional Yair, G. (2000). Educational battlefields in restructuring schools. Harvard Leadership Challenges:The Case of America:The tug-of-war over Educational Review, 67, 41-74. Using Student Achievement Information students’ engagement with instruction. for Instructional Improvement, In Review. Sociology of Education, 73, 247-269. Robinson,V. M. J., & Lai, M. K. (2005). Practitioner research for educators: A Timperley, H. S., & Robinson,V. M. J. guide to improving classrooms and (1998). Collegiality in schools: Its nature schools.Thousand Oaks, CA: Corwin and implications for problem solving. Press. Educational Administration Quarterly, 34(Supplemental December), 608-629. Rosenholtz, S. (1989).Workplace conditions that affect teacher quality Timperley, H. S., & Robinson,V. M. J. and commitment: Implications for (2001). Achieving school improvement teacher induction programs. The through challenging and changing Elementary School Journal, 89, 421-439. teachers' schema. Journal of Education Change, 2, 281-300. Rubie, C. M., Hattie, J. A. C., & Hamilton, R. J. (in press). Expecting the best for Timperley, H. S., & Robinson,V.M.J. New Zealand students: teacher (2003). Partnership as intervention expectations and academic outcomes. strategy in self-managing schools. British Journal of Educational Psychology. School Effectiveness and School Improvement, 14(3), 249-274. Schutzwohl, A. (1998). Surprise and schema strength. Journal of Timperley, H., Robinson,V. M. J., & Experimental Psychology: Learning, Bullard,T. (1999). Strengthening Memory and Cognition, 24, 1182-1199. education in Mangere and Otara: First evaluation report. Auckland: University Scriven, M. (1988).The new crisis in of Auckland. teacher evaluation: the improper use of ‘research-based’ indicators’, Timperley, H. S. & Wiseman, J. (2003). Professional Personnel Evaluation News, The sustainability of professional 1(1), 4–8. development in literacy: Part 2. School- based factors associated with high Shepard, L. A. (2000).The role of student achievement (Wellington, New assessment in a learning culture. Zealand: Ministry of Education): Educational Researcher, 29(7), 4-14. http://www.minedu.govt.nz/index.cfm? layout=document&documentid =8638&data=l.

Using Data to Support Learning 21 Research Conference 2005 22 ConcurrentConcurrent paperspapers

23 Good data, bad news, good policy making …

Introduction system; this paper covers two – our experience with policy makers before, The New Basics Trial in Queensland during and after the Trial; and the highs (2000–04) was about improving and lows of what teachers will (and will educational outcomes. At its heart was not) do. the idea that, to do this, there must be an orchestration of the message We learnt that policy makers come and systems of curriculum, teaching and go – the ones you finish with are often assessment – and that these changes not the ones you started with. (The Gabrielle Matters must be in practices, not merely in same is true of teachers and principals!) Australian Council for Educational Research, statements of intention or expectation. Commitment to school reform can wax Queensland What were the changes? We changed and wane, and be influenced by factors the curriculum by introducing three outside anyone’s control.We learnt that To June 2005 – Director, Assessment & New suites of Rich Tasks covering three 3- there are some teachers who are Basics, Queensland Department of Education and excited by opportunities and grasp the the Arts year spans from Year 1 to Year 9.To determine their curriculum plans, nettle for the betterment of their From October 2005 – Principal Research Fellow teachers had to map backwards from students.There are also some who are and Manager Brisbane Office of the Australian not excited, and who avoid the nettle. Council for Educational Research these tasks – each of whose specification was given on a single A3 Professor Matters has been keenly interested in There were different challenges in the educational measurement since she was a pup – page [Education Queensland. (2004)]. primary and secondary years – the Trial as a teacher, school administrator, bureaucrat, We changed assessment by introducing deliberately spanned the two. In the researcher, advisor, test developer, writer, and a system of social moderation aimed at primary years, the challenge was to the mother. Her major achievements include: achieving state-wide comparability.This view of the teacher as the fount of all • teaching students to use their minds well system required teachers to talk among knowledge that mattered; in the (taught Chemistry and Physics for 20 years in themselves and compare their opinions secondary years the challenge was to Independent, state and Catholic schools, the existence of ‘silos’ that including 5 years as a deputy principal) about student work, not just within their school but also across schools. compartmentalise knowledge and the • leading teams of talented educators on various disciplines.Teachers’ threshold knowledge projects (e.g. QCS Test,Assessment & And we changed teaching by ‘upping Reporting, New Basics) was often found wanting (especially in the ante’ intellectually, challenging Mathematics and the physical sciences), • writing journal articles, books and conference teachers professionally, and connecting papers on test design and marking, test-taking but we also often found teachers willing behaviour, assessment/testing formats, the what was done in the classroom to the to learn new approaches, new concepts underachievement of boys (before the topic real world. and new skills. became trendy), the curriculum wars, the The Rich Tasks at the centre of the relationship between pedagogy and assessment, The New Basics research findings were ‘death by assessment’, school reform, and New Basics embodied the changes that considered by the Minister for standards-based assessment we sought.They were rich in the sense Education in presenting the • compiling reports including Reinventing Years of having variety, scope and depth; in Government’s position on how to 10–12 in State Schools (1999, with Richard requiring academic rigour; and in being improve student learning and to Smith et al.), and The New Basics Research multidisciplinary. Student performances increase comparability of assessment Report (2004) on Rich Tasks were assessed in rich and reporting across schools [Education • having fun working across academe and the ways – the final grade was not the Queensland (2005)]. bureaucracy. result of some scoring algorithm but of on-balance judgements made by Evidence-based policy teachers considering each performance from multiple perspectives. making We learnt many things from the Trial The New Basics research and about many areas of the education evaluation reports (Department of

Research Conference 2005 24 Education and the Arts, 2004; Australian have been identified as applying across features of high-quality Council for Educational Research, 2004) the State beyond New Basics and performance and the features of were considered by the Queensland beyond state schools. acceptable performance; Minister for Education in presenting the This paper spans the trial period • Rich Task assessment model – a Government’s position on how to (2000–04) and the immediate post-trial variant of the traditional improve student learning and to period (2005), showing how research criteria/standards matrix; increase comparability of assessment evidence informed policy-making. • Moderation strategy – four stages and reporting across schools beginning with clarification of task (Department of Education and the New Basics Framework intent and concluding with Arts, 2005). ratification of teacher judgements The New Basics approach to A comprehensive history of the of the standard of student work; development of the New Basics curriculum, teaching, assessment, • Common-format reports – of the Framework and its implementation in reporting, and school organisation was results of formative assessment at 58 state schools can be found in ‘The developed and trialled because of a the end of a 3-year span, as an New Basics: Narrative and widespread recognition and acceptance overall grade for each Rich Task in Commentary’, within the research in 1999–2000 that major changes in the suite, with associated legend report. A summary of the key education were absolutely essential, for ease of interpretation, and with components follows. particularly in the compulsory years of state-wide comparability assured. schooling. • What is taught: Four categories of This view is confirmed by our most essential practices for new times: Differences between recent research, which, furthermore, Life pathways and social futures, KLAs and New Basics strongly suggests that change is still Multiliteracies and communications needed.The New Basics research media, Active citizenship, and At the same time as the New Basics program demonstrated ways in which it Environments and technologies. Program was being developed and might be possible to bring about such • How it is taught: Four categories of implemented in Queensland, another change. effective teaching strategies: program of educational reform was in Intellectual quality, Connectedness progress across Australia – the Key There were four aspects to the New to the wide world, Recognition of Learning Areas (KLAs).The differences Basics trial: development of an difference, and Social support. between the two need to be integrated framework for curriculum, understood. pedagogy and assessment for new • How learning is displayed:Three times; implementation of the suites of transdisciplinary tasks in • The KLA curriculum is organised framework in volunteer state schools three 3-year spans within Years 1–9: into eight areas, which are based on that were selected, quarantined, First suite – 5 tasks; second suite – composite fields of knowledge, each resourced and supported; a research 7; third suite – 8. Rich Tasks are with its own content and context. program comprising 25 individual published in an A3 ‘artbook’ as a The New Basics idea organises a research activities; and an independent collection of double-page spreads, futures-oriented curriculum into external evaluation. one for each task, giving the task four categories, each of which has description; New Basics referents; an explicit orientation towards Ultimately, there were three objects of targeted repertoires of practice; task researching, understanding, and learning from the New Basics trial: the specs; ideas, hints and comments; coming to grips with newly New Basics per se (which was the aim task parameters; and assessment emerging economic, social and of the exercise), the management of criteria. cultural conditions. intervention, and the education system • How the evidence of learning is • Within the New Basics Framework, itself. In this paper, I focus on two of assessed and reported: productive pedagogies are a them: one, the strengths and • mandatory rather than desirable weaknesses of the New Basics idea in Pre-set standards for each task – component. practice; and two, the critical issues that as indicated by the desirable

Using Data to Support Learning 25 • In the KLA idea, outcomes are Basics Research Papers: In Essence’ in and public administration. It can be expressed in terms of what students the research report).The research hard to find the resources (people, are expected to know and to be program used a mixed methods money and priorities) needed to able to do within a composite of approach – from case study to support the long-term specific fields of knowledge at multilevel modelling, critical discourse developments that bring real and certain stages. In the New Basics analysis to psychometrics. substantial change. idea, outcomes are expressed as The method, results and conclusions for 5 The Department of Education, in Rich Tasks – the specific activities each of the research activities were opening itself up to information with real-world value and use, scrutinised by the Framework Research about the state/health of the system through which students are able to Advisory Group, four internationally through the trialling of the New display their grasp of important recognised researchers working Basics, reveals aspects of a mature ideas and skills. independently of Education Queensland system that is ready to face the • The KLA idea incorporated a (EQ).The conduct of the research and demands, obstacles, uncertainties staggered implementation of the validity of the research findings and risks of successful operation in syllabuses as they became available were the subject of a commissioned the 21st century. over a span of years.The New external evaluation. 6 The changes needed to align the Basics idea made all Rich Tasks schools and classrooms of public available at once. Eleven key messages education with the needs of the • The KLA syllabuses are silent on the future can be achieved without high body of evidence required for The following messages from the additional cost and without assessment.The New Basics research report relate specifically to detracting from the ‘old’ basics. New Basics in the EQ context in which documents are prescriptive. 7 As a curriculum project alone, the Trial took place. • The KLA syllabuses do not contain unaccompanied by a powerful assessment criteria. In Queensland, 1 The Trial of the New Basics assessment system and the the Core Learning Outcomes are provided value for money, development of schools as learning indicators of standards. Rich Tasks demonstrating the capacity of the organisations, the New Basics is not have task-specific grading masters. New Basics package as a complete likely to have continuing impact. The desirable features are indicators system. 8 Since the New Basics is about of standards. 2 The New Basics package fundamental change in schooling, it • The KLA syllabuses follow a (curriculum, teaching, moderated will be necessary to ensure schools constructivist approach to learning. assessment and reporting) can be (and EQ structures) are learning The New Basics Rich Tasks realise used to revitalise the education organisations. the reconceptualist paradigm. system, to reform schools, and to 9 Real, sustained and substantive achieve the student learnings changes in professional practices, Research program necessary for the new world. which are not at the heart of 3 Schools and teachers experienced teaching and schooling, are not A research program was developed real challenges but also significant effected overnight or on the basis of around three key questions: Is the New rewards in doing New Basics, in the an edict. Basics likely to lead to the changes that development of the professional are wanted? Is the New Basics likely to 10 Any approach to extension should community, the public accountability, be accepted? Is the New Basics feasible be sensitive to the preconditions and the links with the world outside on an extended basis? These three identified during the Trial for the classroom. questions spring from the central optimising the chances of success research question: Is the New 4 Associated with educational for schools implementing the New Basics viable? innovation are real tensions in Basics package. accommodating the oft-competing 11 Any implementation of changes These questions gave rise to 25 demands for academic excellence based on learnings from the New separate research activities (see ‘New

Research Conference 2005 26 Basics should be gradual, consistent • Year 9 – individual discourse in Strengths and with the need to develop the formal registers, project weaknesses according capacity of the system and management of group its schools. endeavours, non- to the external traditional learning evaluation Selected research frames. ⇑ Quality of student work • Test scores of students in trial findings ⇑ schools on the International Schools’ Development of an assessment • The Rich Tasks were found to be Assessment (ISA) (a standardised system richer than the best student work test of Reading Literacy, ⇑ Changes in approaches to teaching from non-trial schools (themselves Mathematical Literacy and Writing), ⇔ Performance on standardised tests selected to be the best of their improved significantly over time, but ⇓ type) in Year 6, and as rich in Years 3 Congruence with other aspects of did so to an extent not significantly the school system and its context and 9. different from the extent of ⇓ Differential impact between year • In general, Queensland teachers improvement of students in non-trial levels take the view that assessment is schools (including non-state schools). relevant to, and has a positive effect • Year 6 students in trial schools who Research involving real student work on, teaching and learning. identified as Aboriginal or Torres indicated that the New Basics could Simultaneously, the general view is Strait Islander students improved lead to the types of changes that are that assessment lacks validity and is more than other students in the wanted, with student performances inaccurate.Teachers do not appear domain of Reading Literacy on ISA. changing not just in depth but also in to be convinced that assessment is nature. Research also indicated that • Very few students (from trial and a tool for school accountability (the moderated assessment could deliver non-trial schools alike) performed opposite of their NZ counterparts). shifts in teachers’ classroom practices. very highly according to the criteria Reactions of principals and teachers to • Students in New Basics schools held for assessing problem-solving on the external testing and the subsequent their own on conventional World Class Tests. standardised tests of literacy and performance of students on those tests • Queensland students are not test- numeracy. indicated a lack of test-wiseness that wise, and state-school teachers (of could be detrimental to Queensland • Teachers’ participation in various students in Years 3 to 9) do not students in other stressful testing stages of moderation was one of have a positive attitude to external environments. the most important contributors to tests. professional skills enhancement and System blockages included the transfer • The QSRLS-observed decline in to developing confidence in applying of principals out of New Basics schools intellectual quality and connectedness the model for grading students’ Rich in juncture years, IT processes, and the from primary to Year 8 was Task performances. changing role of district director during checked. • The New Basics assessment system the Trial. • Students in trial schools rated is able to withstand pressure and teacher classroom practice in three There were different challenges in the respond to challenges that arise in of the four dimensions of a measure primary and secondary years – the Trial the quest for comparability. of ‘enacted pedagogy’ higher than deliberately spanned the two. In the • Factors that might explain did students in non-trial schools. primary years the challenge was to the students’ performances across view of the teacher as the fount of all • Teachers were surprised that some Rich Tasks are: knowledge that mattered; in the of their students performed so well. • Year 3 – technology, performing, secondary years the challenge was to • A not-insignificant proportion of verbal language; the existence of silos that students met the ambitious compartmentalise knowledge and the • Year 6 – non-traditional learning aspirational standards set for award disciplines.Teachers’ threshold frames; of A-grade.

Using Data to Support Learning 27 knowledge was often wanting trial schools different from what is Possible responses to this less-than- (especially in mathematics and the happening in non-trial schools? palatable news were many and varied, physical sciences), but teachers were as were the options for action. willing to learn new approaches, new Findings not just about concepts and new skills. state schools Options for action Change was – and still Because comparative studies were One could tinker with the existing situation, but the results would not is – needed incorporated into the New Basics Research Program, it was necessary to meet future needs. One could ignore The New Basics Research Report collect data from many more schools what the research is saying, but the suggests deeper issues about the state than just the trial schools. Data were tension between what is needed and of education in Queensland than those collected from other state schools and what has been achieved is already identified in 1999–2000, and also from some non-state schools. For widely known. One could put resources suggests how change can be achieved. certain studies, matched or ‘like’ schools into more documents, or more bolt-on These perspectives are supported by were selected whereas, for other professional development programs – the findings from another study – studies, schools were selected because the usual response in such situations – Assessment & Reporting Framework they were known to be outstanding but this is an expensive solution that (ARF) Pilot Study (Education (‘the best’).Therefore, the critical issues has been tried in other places at other Queensland, 2003). that were identified apply across times. One could interpret teachers’ Queensland schooling in Years 1–9, need for support in their basics An Assessment and Reporting Taskforce beyond the New Basics. (assessment, pedagogy, curriculum) as was established in 2001 because it was showing that they need very detailed clear that there was no coherent Critical issues specifications, but this approach would approach to assessment and reporting de-skill the profession. in P–10. In 2003, the ARF Pilot Study The expression of the critical issues is Since the cost of the Trial ($10.7m over explored assessment in the context of deceptively simple and falls under five four years) was, amongst other things, KLA syllabus implementation in headings – curriculum, schools, schools’ the cost of finding out that change can Queensland state schools. communities, teachers, and the occur and can be accepted, one could education system. As a result of these two studies, draw on the research evidence for together with the earlier Queensland • There are large gaps between the methods of bringing about change.This School Reform Longitudinal Study intended curriculum and the is not to say, however, that simply (QSRLS) (The University of enacted curriculum. extending the New Basics is the answer. Queensland, 2001), the Department of • Some schools can handle change And so, in June 2004, the Director- Education came to possess a large and meet future needs; some act to General stated: volume of hard data and rigorous contain or neutralise change. analyses about what is happening in December 2003 marked the end of the Trial, but that was by no classrooms, which made it possible to • Diversity in the nature and intensity means the end of the New Basics. describe crucial aspects of education of, and attitudes to, the relationships I have authorised 58 schools that across all state schools Years 1–9 (a between schools and their communities is huge. have been involved to date to description that can reasonably be continue with the New Basics extrapolated to P–10). • In general, teachers do not possess while we take the time to reflect high levels of content knowledge, Some of the research studies were on the learnings from the Trial and are not confident about assessment, determine how they can be absolute:What is happening in trial and are not sure what students transferred to all schools. schools? Some were longitudinal: How are learning. have trial schools changed over time? Later this year, the Minister for Many of the studies, however, were • Queensland education’s message Education will present the comparative: Is what is happening in system lacks coherence. Government’s position on how best to report student

Research Conference 2005 28 achievement and school construct and explore new Teaching values performance information.The on- problems, new learning strategies and going program of work, to which new solutions. Upping the intellectual ante the New Basic research and Connecting students to the wider Focused and uncluttered: Curriculum evaluation will contribute, will world extend this into the development planning requires a principled selection of recommendations to of learnings from various disciplines and Generating a supportive classroom Government on how to achieve skills (social, cultural, cognitive and environment greater integration of curriculum, linguistic) rather than universal coverage Recognising difference. teaching, assessment and reporting of prescribed ‘atomistic’ learning in our schools. outcomes. Students study fewer things Assessment system It seemed that the way forward was to in greater depth in order to achieve values identify the core values of New Basics greater levels of understanding. Rigour and incorporate those into all schools, Fluid and responsive: Curriculum setting aside territorial aggrandisement development does not focus on sets of Comparability and simply using what is useful.This is documents and lists of outcomes that Validity not the same as arguing for or against have been composed over several years Accountability. implementing the New Basics in more in committee but, rather, is thought of schools, all schools or no schools. in terms of a renewable and criticisable Action values resource that is dynamic, changing in Core values of the New relation to new contexts, renewed and Prescribing the required outputs (goals) Basics in action sustained by teachers and curriculum but not the way to get there (process) developers. Developing school–community links Did the New Basics per se trigger the Closing the loop with monitoring, desired changes in trial schools? Transdisciplinary: The transdisciplinary feedback and support Probably not. It was more likely to be (or multi-disciplinary) approach to the different way of doing business – of teaching and learning draws on Strengthening teachers’ content ‘doing school’ according to the core practices and skills across disciplines knowledge and assessment skills values of the program; namely, while attempting to retain the integrity through built-in, not bolt-on, approaches curriculum values, teaching values, of each discipline; as opposed to the to professional development assessment system values, and action thematic or interdisciplinary approach Enhancing learning organisations at values. Curriculum values are expanded that seek links between disciplines often school and system levels on below. For brevity, other values are with a dilution of discipline-specific Using program values to drive planning merely listed under each of the other expertise. Caution: Before ‘going trans’, and organisation. three headings. teachers need to be able to work confidently with the disciplines. Queensland Curriculum, Curriculum values The ‘old’ basics: The old basics remain Assessment and at the heart of the New Basics but are Futures orientation: Curriculum is not considered to be sufficient as the Reporting (QCAR) designed around tasks that prepare substance of modern education.The students for new workplaces, Framework New Basics, in emphasising the skills technologies and cultures. Some of the The policy-makers decided that any that students need to complete tasks involve traditional ways of doing policy statement and action plan for intellectually challenging, integrated, real- things; others are responses to new progressing the integration of life tasks, should not sacrifice basic skills times. Some of them require existing curriculum, teaching, assessment and development. practices and skills, some require the reporting should be based on research, blending of old and new, while others including the New Basics research.They require students and teachers to sought and received Cabinet

Using Data to Support Learning 29 endorsement of 12 evidence-based Authentic Highlights of the new characteristics of an effective schooling Knowledges and skills from real-world QCAR Framework system. sources such as industry flow freely • Define what is essential curriculum back and forth between the wider Clear governance for all students in Years P–10; community and the learning There are clearly articulated roles and environment. • Set standards of student responsibilities for all parties involved in achievement in the essential policy making and practice in Inclusive curriculum. curriculum, teaching, assessment and • Create a bank of assessment tools Individual needs and learning styles are reporting. for teachers that link to the essential accommodated, diversity is recognised curriculum and standards. Research-based and celebrated, and student participation in decision-making is • Establish, at key points in the P–10 Decisions on educational policy and encouraged. years, rigorous comparable practice in curriculum, teaching, assessment against the defined assessment and reporting are informed Supportive standards. by rigorous research. • Specify a common framework for Students receive clear guidelines on reporting student achievement what they are learning, how they will be Equality of opportunity against standards. assessed, and how they can influence Every young Queenslander, regardless practices in the classroom. Students are The policy direction for the framework of economic or social circumstances, is encouraged to take risks in a safe was developed by the Department of given the opportunity to acquire environment and be responsible for Education and the Arts in collaboration essential knowledges, skills, their own behaviour and learning. with the Queensland Catholic Education understandings and capacities. Commission, the Association of Accountable Independent Schools Queensland, Transparent Education Queensland, and the Schools convey high expectations and Queensland Studies Authority (QSA). Queensland schools are able to students are able to demonstrate their The Queensland Government will set benchmark their performance on the learning through valid assessment tasks, parameters to guide the creation of the basis of data about schools and assessment results that are materials and tools that make up the performance as well as data on student reported on are comparable across the QCAR Framework.The QSA will achievement in areas of learning at key State. develop the materials and tools in junctures. consultation with key stakeholders, ready Teacher professional community Flexible for implementation state-wide in 2008. Teachers participate in sustained Curriculum is readily renewable and intellectual work, and use a range of References responsive to new contexts. teaching strategies to provide flexible All documents cited in this paper can and innovative learning experiences for be found at: Intellectually challenging individual students and groups of Learners study fewer things in greater students. http://www.education.qld.gov.au/corporate/ depth, achieving deeper levels of newbasics Adaptable understanding. Learning experiences http://www.education.qld.gov.au/corporate/ draw on specific fields of knowledge as There is a willingness to try new ways newbasics/pdfs/summaryfindings.pdf) well as integrate ideas, concepts and of working and be responsive to http://www.education.qld.gov.au/curriculum/ information across fields of knowledge. emerging technologies, and societal and assessment organisational change. http://www.education.qld,gov.au/qcar

Research Conference 2005 30 Education Queensland (2005). Smarter learning:The Queensland Curriculum, assessment and reporting. Brisbane: Queensland Department of Education and the Arts. Education Queensland (2004). New Basics: Rich tasks. (2nd--gold cover ed.). Brisbane: Queensland Department of Education (Assessment & New Basics Branch). Education Queensland. (2004). The New Basics research report. Retrieved 19 August, 2004, from http://education.qld.gov.au/corporate/n ewbasics/html/research/research.html. Smith, R. A., Matters, G. N., Cosier, I. L., & Watson, D. J. (1999). Reinventing Years 10–12 in state schools: A model of multiple learning pathways for new times (Report to the Strategic Policy Branch, Education Queensland).

Using Data to Support Learning 31 Moving on from Count Me In Too: Evidence- based teaching and learning in numeracy in the early and middle years of schooling

New Zealand developed the Early Numeracy Project for Years 1–3 in 2000–2001, based on the New South Wales’ Count Me In Too, and much has happened in mathematics education since. Change is inevitable and numeracy has moved on.Today the New Zealand Number Framework, the Diagnostic Lynn Tozer Marilyn Holmes Interview and Teaching Model now underpin numeracy teaching practice in Dunedin College of Education, Dunedin College of Education, over 14,000 classrooms from Year 1–9. New Zealand New Zealand Important developments to date have Lynn Tozer is a Mathematics Adviser at the Marilyn Holmes is an Adviser in Mathematics at included a flexible national database and Dunedin College of Education where she has the Dunedin College of Education. Mrs Holmes web site, well-developed supporting worked in Educational Support to primary holds a Masters in Science from Curtin University materials and data-rich annual evaluation schools since 1999. Lynn has been a deputy of Technology and is currently working on her reports which inform future direction principal and has taught at all year levels of the doctoral studies. and expectation of achievement. primary school. She also has a passion for literacy, She is a highly experienced primary teacher, and is a trained Reading Recovery teacher. During Because the Numeracy Project is having taught children aged from 5 to 13 years in 1999–2000 Lynn spent time at the University of evolving, further development and many different types of schools in every decade Delaware as a Fulbright Scholar. Lynn is an consolidation will continue. since the 1960s. original member of the numeracy development team having been involved since 2000 in all pilot Mrs Holmes has been a Senior Lecturer in This paper gives a brief background to projects, Count Me In Too, the Early Numeracy Mathematics Education at Auckland College of the Numeracy Project and outlines, Project, the Advanced Numeracy Project and the Education, 1997–2001 and has also lectured in through a story, how evidence-based Intermediate Project. She is also currently the several curriculum areas at the Dunedin College teaching is an integral part of Numeracy Project Coordinator for schools in the of Education, 1994–1996. classroom practice. Otago-Southland region and a National She is one of the original group of lecturers, Coordinator, contributing to Ministry policy mathematics advisers and Ministry personnel who developments in numeracy. Lynn has also been Introduction developed the idea that grew into the Numeracy co-author of several mathematics publications and Project as it is today. Mrs Holmes has contributed writes for the New Zealand maths web site. Change is inevitable in education. As further as a facilitator for the project pilots since New Zealand pushes towards its goal of Her day-to-day facilitation work is focused on the 2000: Count Me In Too Project, Early Numeracy a knowledge based economy the impact numeracy professional development of practicing Project, Advanced Numeracy Project, and teachers, is student-centred and predominantly Intermediate Numeracy Project. is on a teaching community where classroom-based. change is not carefully drip fed but is She continues to inform the development of the geared towards an accelerated pace. project through her involvement in teacher professional development as well as contributing Managing change becomes an issue that to teacher resources: Ministry of Education is imposed on teachers who generally numeracy booklets,‘Figure It Out’ booklets and regard it with trepidation or uncertainty. the New Zealand Mathematics web site. Mesnik (2004) writes that “practically all forms of human behaviour involve decision making under the supervision of a mysterious guide – uncertainty”. How can teachers then be certain that they are making the best decisions for

Research Conference 2005 32 children’s learning in mathematics? It is Numeracy Project, was implemented into existing knowledge and strategies ‘what especially hard when their New Zealand schools in 2001. Milestone do they know’, the framework gives the professionalism appears to be Reports and Evaluations have annually direction ‘where am I taking them’ and undermined by statements such as documented evidence from teachers, the teaching model gives the vehicle to “…transition to evidence based facilitators, researchers and policy analysts make connections between the process practice has yet to occur in and have continued to inform further of teaching and the child’s process of education…this is despite there being development of the Numeracy Project It learning ‘how will I get there’. millions of studies that move education must be stressed that this is not a static The New Zealand Number Framework beyond craft and opinion” and “many of project; it is an evolving discipline.Today has two main sections: knowledge and those outside education find it hard to the project includes work in schools for strategy.The dichotomy is for teaching believe, after a century of scientific and Years 1-8 children; a growing secondary purposes and to focus teachers on technological progress… that the component for Years 9-10;Te Poutama teaching strategic thinking through the primary aim of teacher education is to Tau an initiative in Maori; and supporting teaching model. It is the strategy reproduce rather than to improve material (refer www.nzmaths.co.nz for framework that requires teachers to existing teaching practice” (Hattie, 2003, further information). reflect on their beliefs about how p12). Shulman (1989) gives credence to Facilitators, principals, and teachers are children learn and what children know teachers by acknowledging “their interdependent in effectuating the about number concepts.The Strategy accumulated wisdom of practice which successful implementation of the section describes the mental processes in many cases is as important a source Numeracy Project. Schools are children use to solve operational of guidance for practice as the theory encouraged to either work in the project problems with numbers. It consists of a or empirical principles”. by syndicates or as whole school sequence of nine global strategy stages Experienced teachers are not to be professional development. Having several (see Figure 1) with three operational underestimated; they have a wealth of teachers involved at once means the domains: addition and subtraction, knowledge.The sample they draw their commitment is easier to maintain, multiplication and division and evidence from is the hours, the months through the support they give each proportions and ratios. and the years they spend with several other. Built into the contract between the Ministry and the school is an obligation hundreds of children.The New Zealand from the principal to participate in initial Numeracy Project helps teachers to Counting strategies workshops, with a moral responsibility to manage the inevitable changes, allay their 0. Emergent lessen the workload of teachers by uncertainties, challenge their beliefs and engaging in one major professional 1. One to one counting at the same time create an environment development; the Numeracy Project. 2. Counting from one on where teachers feel a certainty about Materials their professional judgments for children’s learning (Higgins, 2001, 2002, 2003; Irwin, Key aspects 3. Counting from one by 2002, 2003;Thomas & Tagg, 2004; Three aspects have been named as Imaging Thomas,Tagg & Ward, 2003;Thomas & pivotal to the Numeracy Project; the 4. Advanced Counting Ward, 2001, 2002). teacher development programme, framework and diagnostic interview Part-whole strategies Background (Young-Loveridge, 2004). However, it 5. Early Additive could be argued that the teacher The New South Wales Department of 6. Advanced Additive development programme is the cocoon Education and Training initiative Count 7. Advanced Multiplicative around the framework, diagnostic Me In Too (Years 1-3 children) provided 8. Advanced Proportional interview and the teaching model a well researched base from which to (Holmes & Tozer, 2004).The triangulation develop the New Zealand Numeracy Source: Ministry of Education, 2004 of these three, form a strong core for the Project. Following a successful pilot of teaching learning process.The interview Count Me In Too in 2000, the Early provides the teacher with the child’s Figure 1 The number framework

Using Data to Support Learning 33 The Knowledge section describes the A cameo involving one child is used to Knowledge data show that Joanne can key items of knowledge that students show how data could be understood read and sequence numbers both need to learn to provide the foundation and acted upon. Joanne is forwards and backwards up to 100. for strategy development and consists representative of many Stage 4 Numbers bigger than this prove of five aspects; numeral identification, (Advanced Counting) children and it is challenging. She has already shown that number sequence and order, grouping/ through her data that we trace her her backward counting over the decade place value, basic facts and written journey and what it means for her. is not secure. She is unable to recognize recording. common unit fractions. Her place value Joanne’s story knowledge is limited to knowing how Data collection many tens there are in numbers to 100 Joanne has transferred to Kapai School only. She appears to know basic facts to The gathering of data and subsequent at the beginning of her fifth year at 10, her doubles to 20 and teen numbers. analysis are ‘part and parcel’ of the school. In her interview, her teacher The teacher looks to the Number Numeracy Project.The assessment tool notes that although Joanne is normally Framework to provide specific is an individual task-based oral interview outgoing and confident, she is now in which carefully sequenced questions identifiable learning steps when planning tentative in her responses and has are couched to elicit a variety of for Joanne’s needs.The analysis of one difficulty explaining her thinking. Her responses.Teachers are supported with short line of data show that, for Joanne, interview data show: their first diagnostic interviews by their as a Year 5 child, these results are cause facilitators who coach teachers in the for concern because at Kapai School use of the tool and in clarifying their Strategies knowledge 80% of the children, at the beginning of understandings of children’s thinking Add/Sub Mult/Div. Prop/Ratios FNWS BNWS Year 3, are Stage 4 in addition and about number. Ongoing formative Frac. Grouping/PV Basic Facts subtraction. Behavioural observations assessments can be complex but for an Stage 4 4 2-4 4 4 2-3 4 4 show Joanne’s hesitancy, the use of fingers, her need for reassurance, slow empowered teacher who knows, Data interpretation gives a simple clear responses and the absence of understands and applies the detail of profile statement.With reference to the mathematical language to explain her the number framework to daily Number Framework, what strategies in thinking. It would suggest that the interaction and observation these are the three domains did Joanne show power of the face to face interview made simpler. that placed her at Stage 4? To solve the cannot be overstated when compared single digit addition problem 8+5, A characteristic of good data is its with a silent pen and paper test. potential to help teachers make good Joanne counts on from 8, saying decisions about children’s learning. Data 9,10,11,12,13. She keeps track on her Joanne’s class tell a story. Each Diagnostic Interview fingers.The problem 37-9 is solved by provides the classroom teacher with counting back on her fingers, however At a class level the results of individual immediate and detailed information in crossing the decade, 30 is omitted interviews provide substantial diagnostic about the child’s number knowledge and her answer is 27. Counting on or assessment information.There are 28 and mental strategies and allows back from the biggest number is her children in Joanne’s class. Each set of behavioural observations to be made in most advanced addition and subtraction individual’s data contributes to a detailed response to the oral questions. strategy. Her response to a class profile, part of which is in Figure 2. multiplication problem presented as an The diverse range of children’s Understanding data and array, brings out fingers again as she skip numeracy needs as evidenced by the actions for improvement counts the rows in 5s.To solve the data cannot be ignored.The challenge is simple problem _ of 12, she shares out for the classroom teacher to plan a The two questions uppermost in the beans one by one into three piles, classroom programme in response to teachers’ minds should be:What does it and answers 4, voiced as a question. the highlighted strengths and all mean and; how can we use it to Joanne looks to the teacher for weaknesses in student knowledge and improve children’s achievements? confirmation for many of her responses. strategies.To effectively manage a

Research Conference 2005 34 Biographies Strategies KnowledgeEdit

Name DOB Year Ethnicity Gender Add Mult Proport FNW BNW Num Fraction Place Basic Status Action (dd/mm/yyyy) Sub Div S S ID Value Facts

Edit

To m 10/05/1996 5 Maori m 4 4 2–4 5 4 na 2–3 4 4 Active 186825

Room 5

Edit NZ Tara 1/10/1995 5 European f 6 6 5 6 6 na 5 5 5 Active 186829

Room 5

Edit

Tim 21/03/1996 5 Maori m 5 5 5 6 6 na 5 5 5 Active 186830

Room 5

Edit NZ Tania 18/12/1995 5 European f 5 6 5 5 5 na 5 5 6 Active 186833

Room 5

Edit NZ Tina 3/02/1996 5 European f 5 5 4 5 5 na 4 5 4 Active 186837

Room 5

Edit NZ Travis 18/05/1996 5 European m 4 4 2–4 5 2 na 2–3 4 2–3 Active 186840

Room 5

Figure 2 response to this data, children are • providing immediate scaffolding and assessment.The teacher shares her grouped according to their strategy feedback observations, and a clear direction for stage. Joanne joins five others from her • making connections to existing the next lesson is openly discussed and class for accelerated learning of part- concepts understood by all parties. whole thinking strategies taught through • stimulating metacognition and self the numeracy teaching model; using assessment Joanne’s school materials, using imaging and using number properties. Research indicates • shared identification of next Management at Kapai School has made the importance of including children in learning steps. it clear that teachers will gather valid and consistent diagnostic and formative their learning (Alton-Lee, 2003; Askew, These teaching techniques are all part data throughout the year so that a Brown, Rhodes, Johnson & Wiliam, of the synchronicity of quality detailed school-wide numeracy picture 2002). Joanne’s teacher does this by: responsiveness to Joanne and what her emerges. Problem-solving discussions data tell us. Intuitive and informally • discussing the learning intention of can now focus on the story told by collected evidence informs the next the lesson data. From a close analysis of trends, teaching step for the experienced • using open and high order questions achievement targets will be set and teacher. However,“the intuitive and these are critical to informing the • allowing wait time for considered implicit” must be “made explicit” to the direction for future staff professional responses children (Timperley & Parr, 2004, p69). development. Alton-Lee (2003), in her • actively listening Therefore, the teacher concludes the best evidence synthesis of quality strategy lesson with Joanne’s group in • encouraging student to student teaching states that “the gathering and conversations an informal and immediate student self

Using Data to Support Learning 35 analysis of high quality student from school to school.This ensures the References achievement data and the use of continuity of student information for externally referenced benchmarks have successive classroom teachers. Askew, M., Brown, M., Rhodes,V., been found to be powerful tools in Johnson, D., & Wiliam, D.(1997). A national numeracy story is the last bringing about changes in teacher Effective teachers of numeracy. London: which the data tell.The national trends practice that facilitates higher King’s College. and patterns revealed in the data achievement for students” (p19). (Christensen, 2003; Higgins, 2003; Irwin; Alton-Lee, A. (2003) Best Evidence Timperley’s research in 2004 also found 2003;Thomas,Tagg and Ward, 2003) Synthesis: Quality Teaching for Diverse that significantly higher achievement was continue to confirm that the Numeracy Students in Schooling.Wellington: accomplished when syndicate and Project is having a significant positive Ministry of Education school staff meetings were focused on impact on children’s achievement and Christensen, I. (2004). An evaluation of data analysis and discussing to inform expectations and practice for Te Poutama Tau, 2003: Exploring Issues implementation issues arising from the the school, in the classroom and in Mathematics Education.Wellington: evidence. In their study,Thomas and Tagg ultimately, for the child. Ministry of Education (in press) found that thirteen longitudinal schools, which set school- Conclusion Hattie, J. (2003). New Zealand Education wide targets for numeracy and collected Snapshot:With specific reference to the achievement information to rate It is clear that data tell stories for all; the Yrs 1-13 Years. Paper presented to performance, outperformed the six child, the teacher, the school, the parents Knowledge Wave 2003–the Leadership longitudinal study schools which did not. and the nation.Without informed Forum. Auckland, New Zealand. analysis or in fact “interrogation” of data Irwin, K. (2002). An evaluation nof the and a precise understanding of the story Joanne’s country Numeracy Exploratory Study Years 7-10, it tells, little or moderate impact will be 2001: Exploring Issues in Mathematics Kapai School results are entered into made on children’s achievement.The Education.Wellington: Ministry of the national database located on-line. data provides the base for constructive Education. This database has been contributed to management of the implicit complexities by all schools involved in the Numeracy of teaching in a dynamic classroom. Irwin, K. (2003). An evaluation of the Project since 2000. By the end of 2005 “There is impartiality about achievement Numeracy Project for Years 7-10, 2002: data on over 400 000 students will data if it is handled legitimately and that, Exploring Issues in Mathematics have been entered.The annual analysis unlike a person expressing an opinion, Education.Wellington: Ministry of of this data has provided a rich data is an objective messenger that is Education. statistical source upon which Kapai hard to shoot” (www.nzmaths.co.nz/ Mesnik, I. (2004). Common-Sense Logic School can base its numeracy numeracy/Lead_Teacher). for Managing Uncertainty. In A, achievement comparisons with a Rogerson, (Ed.), The Mathematics national picture.The database provides, Never before have we had such a Education into the 21st Century Project. by norm referencing, the capacity for complete picture of a child’s mathematical Proceedings of the International Kapai School to compare, by year level understandings of number (A. Robertson, Conference;The Future of and domain, the performance of its personal communication, May 26, 2005). Mathematics Education, pp1-6. children with other schools of similar Nor have teachers had the opportunity decile and with children of similar to be certain and confident that they can Shulman, L. (1986, February).Those and are, making a significant impact on ethnicity or gender.The facility also who understand: Knowledge growth in children’s achievement. allows Kapai to make in-school teaching. Educational Researcher, 15(2), comparisons of its own progress from 4-14. Acknowledgement the beginning of the year, between Timperley, H.S. (2004). Enhancing years and over time. A strength of the The opinions expressed in this paper Professional Learning Through Evidence- database is the facility to transfer both do not necessarily represent those of Based Inquiry. Paper presented at group and individual student data from Dunedin College of Education or the Symposium on Teaching Quality. class to class and, in the near future Ministry of Education, New Zealand. Waipuna Lodge; Auckland.

Research Conference 2005 36 Timperley, H., and Parr, J. (2004). Using Holmes, M.,Tozer, L. (2004). An Effective Evidence in Teaching Practice: Identification and Response to Implications for Professional Learning. Children’s Learning Needs in Number. Auckland: Hodder Moa Beckett In A, Rogerson, (Ed.), The Mathematics Publishers Limited. Education into the 21st Century Project. Proceedings of the International Thomas, G., and Tagg,A. (In press). Conference;The Future of Evidence for Expectations: Findings from Mathematics Education, pp60-66. the Numeracy Project Longitudinal Study. Young-Loveridge, J.M. (2004). Patterns of Performance and Progress on the Thomas, G.,Tagg,A. (2004). An Numeracy Projects 2001-2002: evaluation of the Early Numeracy Exploring Issues in Mathematics Project, 2003: Exploring Issues in Education.Wellington: Ministry of Mathematics Education.Wellington: Education. Ministry of Education Thomas, G.,Tagg,A., & Ward, J. (2003). An evaluation of the Early Numeracy Project, 2002: Exploring Issues in Mathematics Education.Wellington: Ministry of Education Thomas G., & Ward, J.(2001). An Evaluation of the Count Me In Too Pilot: Exploring Issues in Mathematics Education.Wellington: Ministry of Education. Thomas G., & Ward, J. (2002). An Evaluation of the Early Numeracy Project, 2001: Exploring Issues in Mathematics Education.Wellington: Ministry of Education. Higgins, J. (2001) An Evaluation of the Year 4-6 Numeracy Exploratory Study: Exploring Issues in Mathematics Education.Wellington: Ministry of Education. Higgins, J. (2002). An evaluation of the Advanced Numeracy Project, 2001: Exploring Issues in Mathematics Education.Wellington: Ministry of Education. Higgins, J. (2003). An evaluation of the Advanced Numeracy Project, 2002: Exploring Issues in Mathematics Education.Wellington: Ministry of Education.

Using Data to Support Learning 37 Getting SMART with data in schools: Lessons from NSW

Abstract development.The paper will describe how SMART can play a crucial part in Lifting the performance of New South striking a balance between internal and Wales (NSW) students in literacy, external assessment, and between numeracy and other key outcome areas assessment for instruction and to world-class standards is a central assessment for accountability. priority of this Government.The crucial responsibilities, shared between schools Introduction * and the system, for effective educational Max Smith provision are articulated in the most Governments everywhere are seeking Department of Education and Training, recent Framework for School reassurance that their school systems New South Wales† Development and Accountability for are delivering the results students need NSW government schools.The role of to succeed in an increasingly complex Max Smith [B.Ec Dip.Ed M.Ed.Stud PhD] is 108 very senior officers, School society. At the same time, schools have Manager, Systems, Data Analysis and Regional Education Directors, recently appointed been given greater autonomy and Support with the Educational Measurement and to regions across the state, is to ensure freedom to manage their own affairs School Accountability Directorate, Department of and develop school-specific instructional Education and Training, New South Wales. Dr the effective implementation of this Smith’s research over the past decade has framework.The aim is to consolidate and improvement strategies.These two focused on the development and use of school and focus existing accountability, sets of forces have given rise to more performance indicators including measures of improvement and reporting policies to sophisticated monitoring and analysis value added, relative effectiveness and attitude improve and enrich student outcomes. systems to ensure that standards are towards schooling.This work is supported by an improving and that schools are being extensive background in teacher and classroom- Essential to this framework is the vast adequately supported in their work. based research, and 17 years of teaching store of information available within the experience in secondary schools. Current projects system and its schools on student A consolidated school development include the development of reporting and analysis outcomes: academic, social and and accountability framework, the tools for use with external test and assessment data, and tools for gathering and analysing affective. Accessing, managing, analysing Framework for Development and information on social and affective outcomes in and interpreting this store of Accountability, has been devised by the schools and their communities. His regional information are tasks fundamental to NSW Department of Education and support roles include responsibility for the the success of the Department, its Training to bring together elements analysis of outcomes information, school schools, and for high quality provision contained in existing departmental reporting, school review policy, and software policies and agreements. development and computer systems. for the students in their care. Significant challenges have been overcome The Department’s accountability and through the development of state-of- improvement functions have been the-art information and communication supplemented in successive restructures, systems (ICT) that bring complex data most recently and explicitly through the to the finger tips of staff in schools and appointment of 78 School Education regions in highly usable forms. An Directors with line management outstanding example is the School responsibility for principals, both formally Measurement, Assessment and through annual review and professionally Reporting Toolkit (SMART) that in terms of leadership support and facilitates the gathering, monitoring, professional growth.Thirty School analysis and reporting of data in NSW Development Officers attached to public schools. regions have also been recently This paper looks at the strengths of the appointed to provide additional SMART package, its role in engendering leadership and guidance to schools in educational progress in NSW and the self-evaluation, planning, development, plans the Department has for its future data analysis and reporting.

Research Conference 2005 38 The design for equitable testing, assessment and monitoring tools involves clarifying the purpose and underlying constructs for the assessment, and identifying the sorts of inferences that can be drawn from the assessment (Willingham & Cole, 1997). The use of instruments developed using modern measurement theory, based on concepts of cognitive processes in learning and inferential methods, provides a foundation for testing that permits a more meaningful interpretation of achievement in relation to a defined latent trait (Mislevy, 1993). As Cunningham (2005) points out, Figure 1 NSW framework for school development and accountability these tests have the potential to incorporate both multiple-choice and store of information held by schools The framework for constructed response items so long as and the system on the achievements school development both components are contributing to and demographic features of students. the measurement of the same and accountability The paper will now consider more construct.The strength of this approach closely the role of testing and The Framework for School is seen across the test development assessment in NSW. Development and Accountability is process, in item analysis and reporting based upon the following principles: Testing and assessment and in establishing a set of linked tests • the need for accurate, reliable data to report in relation to set standards about each school The tension between the twin goals of across time (Thissen & Wainer, 2001). development and accountability is There is greater scope for such • supplementation of student reflected in the imperative to strike a instruments to serve the needs of outcomes data with the results of balance between instructional macro-reporting while providing more in-school evaluation assessment and performance useable information at more local levels. • evaluation of school performance as measurement at both the school and The experience in NSW is that the the first step in a process of school system levels. Cizek (2005) regards high development and improvement right sort of tests and support materials stakes [accountability] tests as incapable can strike a balance between the micro • effective school planning to of providing high-quality information for and macro reporting levels and win stimulate the development cycle instructional purposes and queries if support from the profession for state- • school planning will suggest goals relative group performances have wide testing. In a wide-reaching that form the basis for reporting to anything meaningful to say at the school evaluation of assessment practices in the community level.The NSW experience is that NSW, Eltis (2003) found surprisingly • balance between school accountability testing and assessment programs can little hostility remaining amongst and development activities effectively serve these two purposes at teachers towards the expanded level of once but only if the design of the tests • evaluation of school performance to state-wide testing in NSW. State-wide is appropriate and there are provide the foundation for tests have come to be valued by school mechanisms in place to convey the reporting on accountability. leaders, teachers and parents for their critical diagnostic and performance- diagnostic scope, as well as for their A framework based on these principles related messages to the right people in ability to locate the performance of the will necessarily lean heavily on the vast a flexible and timely manner. school’s students relative to other

Using Data to Support Learning 39 students across the State. Eltis found improvement and accountability systems. School Measurement, Assessment and the availability of quality diagnostic The ICT systems will now be discussed. Reporting Toolkit (SMART for short), and information from the testing programs made it available to all schools and professional learning opportunities School Measurement, participating in all the NSW state-wide for teachers and school leaders were Assessment and testing programs including the Basic major contributors to the growing Skills Test and Primary Writing success of these tests. Reporting Toolkit Assessments in Years 3 and 5, the (SMART) Computer Skills Assessment (CSA) in Internal and external Year 6, and the English Language and The NSW Department’s Data on Disk Literacy Assessment (ELLA) and assessments software was developed in 1997 Secondary Numeracy Program (SNAP) following recommendations put forward Parker and Rennie (1998) raise the issue in Years 7 and 8.The most recent by schools in the 1995 Review of the of the relative value placed on internal additions to the package provide Basic Skills Test for software to be and external assessments. Clearly, both analysis of the School Certificate and developed that would enable schools to forms of assessment can provide Higher School Certificate examinations. analyse their results electronically, freeing measures of achievement; however, NSW’s innovative Essential Skills in school staff from the many hours needed external (most often test-based) Science Assessment will come on-line in to analyse and copy data from the paper measures remain the focus of many, but trial form in 2006. Other modules versions of the reports. Prototypes were no longer all, tertiary institutions and under current development will see the developed and trialled in 150 schools employers. Assessments based on a package expanded to include teacher across NSW in 1997 and 1998. wider sampling of the curriculum over a assessments of students, questionnaire Feedback from the trialling was used to greater period of time are arguably a and survey tools, and an assessment strengthen the software’s functionality more valid but under-reported construct item data bank. Figure 2 presents the and in 1999 the software was rolled out compared with one-off external tests. structure of the SMART package in to schools.The software has undergone Nevertheless, lower value is often placed graphical form. significant enhancements since 1999 and on the internal school-based assessment, provides schools with what can be Plate 1 even within education systems and described as an outstanding set of schools themselves.The origins of this analytical tools to support schools in perception probably lie in the perceived SMART is an outstanding making informed decisions on pedagogy, limitations of school-based assessments schooling outcomes analysis quality teaching and learning and as expressed in the concern for package. It offers Principals and improving student learning outcomes.The consistency of teacher judgements, senior teachers in NSW software has been widely accepted and between teachers and over time, and schools an innovative tool for used in all NSW government primary between students, and hence concern analysing and comparing a schools, and also by South Australian for the fairness of the assessment (Linn school's learning achievement government and Catholic schools, NSW & Gronlund, 2000). Eltis (2003) suggests results by drawing on the most Catholic Education Commission schools, a closer alignment is needed between extensive student outcomes NSW Independent schools and many internal and external assessments, using database in Australia.The overseas International Schools, including external assessments as a part of a sophistication of the package one in PNG. broader framework for reporting and means that in-school and cross-validating internal school-based Until 2004 the software was only across-schools comparisons assessments. available to schools participating in the can be made easily, Basic Skills Testing Program. Since then incorporating data drawn from In NSW, the provision of high quality tests the Educational Measurement and a number of years across a and ICT systems to align internal and School Accountability Directorate has variety of assessment external assessments are seen as crucial consolidated and streamlined the situations. I believe the to effective assessment practice and a key reporting software, now known as the applicability of the package to development in the State’s school

Research Conference 2005 40 school summary information and assist schools in a climate of the ability to regenerate detailed and administrators to “drill- evidenced based decision reports on individual student for the down” into the test results in a making is world class and as far information of parents and teacher very systematic and logical way. as this writer is aware is not • access to analysis for groups of The feedback is linked directly matched by equivalent software students across the performance to the curriculum so; in that in any other state in Australia bands (skill bands) including the sense it reinforces the notion or country. students in custom groups – this is that the curriculum is the particularly powerful for providing unifying construct underpinning Professor John Pegg evidence of improved student teaching, learning and Director, National Centre of Science, learning outcomes and effective assessment. Information and Communication pedagogy Technology, and Mathematics Education Professor Jim Tognolini • schools can access information for Rural and Regional Australia concerning the performance of Research Director, System and University of New England, NSW School Testing students on individual items at the General Manager, Sydney Office group or school levels – this Australian Council for Educational includes patterns of student NSW legislation protects privacy and Research responses and additional distractor personal information, and prevents the information detailing the reason why publication of test results for individuals students chose particular options. or schools that could be used for the • Information investigating Individual creation of league tables. One of the Plate 2 Student Responses to various core functions of SMART is to ensure questions. Details provided include links to the relevant NSW Syllabus, that all data is appropriately locked I firmly believe that for large- to ESL scales and advice as to preventing unauthorised access to scale assessment programs where to access teaching strategies individual and school results.This has there is a need to make an to support teaching and learning. been achieved through a sophisticated overt link that the data that • The most recent release of SMART process of data encryption and unique emerge from such programs includes linkages to teaching and passwords that still enable backward are part of a continuous stream learning strategies and resources compatibility to previous data sets.The of information that tracks the with wide-reaching implications for passwords also manage permissions to progress of students and professional learning and program access various levels of the data. programs over time.Another development. Regional passwords allow access to all part of the stream is the data school data for the Region and its that emerges from the teaching • Local area, customised comparative associated education areas, and school and learning process that takes school groups, like school group and passwords only allow access to their place on a day-to-day basis. state trend data is available to all data at administrative and class Together they give a better schools.Trends for various test aspects teacher levels. picture of student progress are also available disaggregated, for example, by gender. The data and analysis functionality than either of them individually. • Item analysis enables schools to available to schools for the testing SMART provides a conceptual filter information on the programs is impressive and includes: link between the state-wide performance of the school or for • the ability to create Custom Groups assessment programs and various groups of students for of students such as class groups and school-level use of the same particular skills.This is an extremely students involved in special data in promoting student powerful tool in filtering programs such as Reading Recovery learning.The Program is user- performance, for example, by friendly and enables teachers • tables in PDF format, including literacy/numeracy, by subject and by sub-strands.

Using Data to Support Learning 41 • A common reporting scale is used The school-based assessment module NSW DET has a wealth of material and to map student progress between, will form the nexus between external expertise to construct a test item for example,Year 3 and Year 5 or and internal assessments and has the databank. From our extensive testing Year 7 and Year 8 potential to see the consistency of activities we have a huge store of test • Student Progress Maps allow the teacher judgements and hence the items from which to construct an user to identify individual students significance placed on school-based Assessment Item Databank.We and drill down to the specific assessments, greatly enhanced. estimate that we hold some 20,000 performance of the individual items if we include both published student directly from the graph. Survey module items and items that have been trialled but not published. • Important information on the overall SMART will soon have a fully functional progress of students within the school module for accessing the Department’s In the construction of the Assessment can be discovered when they are surveys and questionnaires, designing Item Databank we recognised three compared to the rest of the state in custom surveys and analysing and essential requirements. Firstly, the value added terms. Individual student presenting these results.This important number and type of items authentically performance can be compared to all initiative makes explicit the connection reflect the nature and emphases of the other students of similar prior ability. between academic, social and affective syllabus outcome to be measured. Value- added measures are also outcomes.This part of the toolkit will Secondly, the items meet accepted prominent in the new packages facilitate the collection of data from standards of content validity and developed for Years 10 and 12. students, teachers, parents and the psychometric quality.Thirdly, the item • Schools can export data from the community. It is planned for release by bank is easy to use and maintain. It is software into various spreadsheet the end of the 2005 school year. vital that classroom teachers can easily applications.This enables schools to manage the test item database and add additional information to Assessment Item Data Bank build tests to their specifications. individual student data and then From test items created by the perform further analysis. Educational Measurement and School Conclusion Other SMART modules Accountability Directorate we are NSW schools are taking more constructing a databank of test items responsibility for their own School-based assessment from those not required for current performance, are subject to closer module testing programs. At this stage, it is public scrutiny and are finding new simply going under the title of ways of improving student outcomes in The most recent developments in ‘Assessment Item Databank’. a world of ever-more demanding SMART include a module for the As a classroom teacher, using released standards. In the context of the NSW capture, analysis and reporting of teacher and aligned test items in our State's Framework for School Development school-based assessments.There is no Assessment Item Databank, you will be and Accountability, an effective school universally available software in NSW to able to select test items to assess how will be one that is constantly striving to help with capturing data in schools on well your students are meeting syllabus enhance its educational provision the achievements of students against outcomes. Once you select the through a process of self-evaluation, standards frameworks.The module will reporting, review and development facilitate data entry and data import, assessment items, you will then be able to print out a test and answer key. within the resources available to it. A make links to external tests and fundamental premise underlying the examination results, allow schools to Now that you know how well your framework is that neither accountability design biannual reports to parents on the students have mastered the syllabus nor school improvement efforts, on achievement of students, allow analysis of outcomes, you can then visit the linked their own, will be sufficient to produce the data and facilitate the accumulation teaching strategies site to identify best improved student outcomes. and presentation of data for school and practices in education to teach and or system planning, and for annual reporting re-teach specific outcomes. NSW public schools are being to parents and the community. presented with quantitative data

Research Conference 2005 42 Figure 2 Structure and functions of the SMART package

Using Data to Support Learning 43 comparing school performance with The SMART toolkit is cutting edge consequences. In R. P.Phelps (Ed.) state-wide achievement levels, value- technology for schools. Nowhere else Defending Standardized Testing. added measures generated from nationally and perhaps internationally Mahwah, New Jersey: Lawrence external tests and examinations, tools do schools have access to such a Erlbaum Associates. sophisticated analysis package that for the collection of both quantitative Cunningham, G. K. (2005). Must high- enables the manipulation and and qualitative data from within the stakes mean low quality? Some testing investigation of student performance. school and its community, and highly program implementation issues. In R. P. NSW schools are uniquely positioned flexible, diagnostic tools for the Phelps (Ed.) Defending Standardized through their access to the SMART management, analysis and presentation Testing. Mahwah, New Jersey: package to consider detailed of outcomes information.The availability Lawrence Erlbaum Associates. of very senior staff in regions assisting information to support specific and each school to undertake self tailored intervention strategies for Eltis, K. (2003). Time to Teach,Time to evaluation and planning in which the improving student learning outcomes. Learn: Report on the Evaluation of community plays a part, and which Outcomes Assessment and Reporting in incorporates analysis of the statistics References NSW Government Schools, State of NSW, Department of Education and and the setting of targets for school Cizek, G. J. (2005). High-stakes testing: Training. development completes the framework. contexts, characteristics, critiques, and

Research Conference 2005 44 Linn, R.L., & Gronlund N.E. (2000). Measurement and assessment in teaching (8th ed.). Upper Saddle River, New Jersey: Merill Prentice Hall. Meisels, S.J., Dorfman, A., & Steele, D. (1995). Equity and Excellence in group-administered and performance- based assessments. In M.T. Nettles and A.L. Nettles (Eds), Equity and Excellence in Educational Testing and Assessment. Norwell, Massachusetts: Kluwer Academic Publishers. Mislevy, R. (1993). Foundations of a new test theory. In N. Frederiksen, R.J. Mislevy and I.I. Bejar (Eds), Test Theory for a New Generation of Tests. Hillsdale, New Jersey: Lawrence Erlbaum Associates. Parker, & Rennie (1998). Equitable assessment strategies. In B.J. Fraser and K.G.Tobin (Eds), International handbook of science education. Great Britain: Kluwer Academic Publishers. Thissen, & Wainer (2001). Test scoring. Mahwah, New Jersey: Lawrence Erlbaum Associates. Willingham,W.W., & Cole, N.S. (1997). Gender and fair assessment. Hillsdale, New Jersey: Lawrence Erlbaum Associates.

* The opinions expressed in this paper are those of the author and not necessarily those of the NSW Department of Education and Training.

† I wish to acknowledge the very significant contributions to this paper made by the Director and my colleagues in the Department’s Educational Measurement and School Accountability Directorate.

Using Data to Support Learning 45 Data-driven school improvement through the VCE Data Service

Abstract As the holder of student achievement data spanning three sectors and four levels, the Victorian Curriculum and Assessment Authority (VCAA) has a responsibility to provide these data to schools in ways that enable school staff to use them effectively and easily. Glenn Rowley Peter Congdon With the discontinuation of the Victorian Curriculum and Victorian Curriculum and publication of school achievement Assessment Authority Assessment Authority indices, the VCAA was forced to confront a range of issues surrounding Glenn Rowley has been General Manager for Peter joined the Victorian Curriculum and the question of which data belonged to Policy, Measurement and Research at the Victorian Assessment Authority in 2002 as the Manger of Curriculum and Assessment Authority since Educational Measurement. Prior to this the student, which was the property of February 2002. He began his career as a appointment he was the Head of Assessment the school, and which belonged to the secondary teacher in Victorian schools before Services Psychometric Support Team at the general public. In 2002, a new balance undertaking graduate studies in educational Australian Council of Educational Research, where was struck. A key component in this measurement and evaluation at the University of he worked for 11 years. Peter’s main focus of balance was the introduction of the Toronto, where he completed a Masters degree work has been in large-scale testing, involving data in 1972 and a PhD in 1975. analysis, equating and reporting of educational VCE Data Service. outcomes at individual and group levels. Following lecturing appointments at the University The VCE Data Service is an online of Toronto and La Trobe University, he joined the Peter is a member of the National Measurement service that connects schools to the staff of the Faculty of Education at Monash Advisory Group of the Australian Government. entire VCE data set going back to 1998, University in 1983, lecturing in research He has participated in educational standard and provides them with the capacity to methodology and school assessment. He was setting exercises for the Malaysian Government. generate a range of analyses related to appointed Associate Professor in 1990, Associate He has studied psychometrics at the University of Dean (Research) in 1991 and Associate Dean Chicago along with various statistical analysis and their own school, and how its results (Staff) in 2001. He has been a member of the educational measurement courses at The compare to those of other schools in Scaling Committee of the Victorian Tertiary University of Melbourne. Peter specialises in data the State, schools in the same sector Admissions Centre since 1994, and a member of analysis techniques including Rasch measurement, (government, Catholic and the ACER Council and Board of Directors. multi-faceted analyses, test equating and Independent), and to schools in its Like differential item functioning. Since joining the VCAA, Glenn has had major School Group. involvement with the delivery of the AIM testing Peter has produced a number of book chapters program, the changed arrangements for the and journal articles in educational measurement Through VCEDS, senior management publication of VCE data and the development and along with numerous presentations at national teams in each school can easily implementation of the VCE Data Service. He is and international conferences. generate charts and tables addressing a currently working on the development of a new range of questions including the ‘Like Schools’ measure. following: • How were our school’s overall results this year? Were they better or worse than in previous years? How do they compare with schools like ours? • How did our students perform, study by study in terms of completions, Study Scores, examinations and school assessment grades?

Research Conference 2005 46 • How did our school’s results privacy and the public’s right to know. achievers than when placed in a class of compare to reasonable The issue became particularly acute in lower achievers. In schools in which expectations? Did our students 2002, in relation to the publication of students are generally of high ‘ability,’ perform as well as students of data on school performance in the end- students do perform better than would comparable ability in other schools? of-schooling Victorian Certificate of have been predicted from their ability • Is our student cohort changing Education (VCE). alone. In schools in which students are over time, in ability and/or generally of lesser ‘ability,’ students do achievement? Public reporting of the perform less well than would have been predicted from their ability alone. • How can we develop better VCE, 1996–2001 Hence we should have expected to find understandings of the patterns of From 1996 until 2001, the Victorian higher achievement indices in schools group performance by identifying Board of Studies and its successor, the with higher ability students. And, year how individuals contribute to those VCAA, had provided information to after year, that is what was found. patterns? newspapers to facilitate the publication The six-year adventure with This paper provides an account of the of tables documenting VCE performance achievement indices came to an end in first two and a half years of the VCE by school. Conscious of differences in 2002.The VCAA took a view similar to Data Service, outlining the user school intake, it was decided in 1996 that of Goldstein, who had observed, in feedback and the VCAA response to it, not to publish raw achievement data, relation to the value-added measures and the growth in usage over the but instead to publish an index, in which that he had pioneered, that: period. It concludes with lessons raw achievement is adjusted according learned and plans for the future. to students’ performance on another set their use as public accountability of measures, the General Achievement measures, e.g. in the form of When educators study their Test (GAT). performance tables or ‘value schools and classes, they seek an added league tables’ is answer to the ageless question: Is The GAT is administered in June of the inappropriate and would destroy it good because we’ve been doing same year, and its key purpose is to their credibility and usefulness. If it for a long time, or is it good enhance the quality control measures they were ever to become ‘high because we have tangible evidence on VCE assessment. It is presented to stakes’ pieces of information like of its worth? In many instances students as a measure of their ‘general the current DfEE league tables of one must conclude the former knowledge and skills in the areas of examination results, then they because no evidence exists to would inevitably become distorted written communication; mathematics, support the latter (Johnson, 1997). and no longer reflect any science and technology; and humanities, underlying reality of school Because of its unique role in Victorian arts and the social sciences.’ performance (Goldstein, 1999). education, the Victorian Curriculum and The achievement indices were created Assessment Authority (VCAA) has an In arriving at this position, the VCAA using a multiplicative model in which important role to play in assisting was forced to confront a range of the three components of the GAT schools in their efforts to become issues surrounding the question of were included at the level of the more effective.The VCAA is the holder which data belonged to the student, student, but school GAT means were of data on student achievement that which was the property of the school, not. It had very different properties to a span three sectors (government, and which belonged to the general measure of growth that could have public. In 2002, a new balance was Catholic, Independent) and four levels been developed had the full multilevel (Years 3, 5, 7 and 11–12), and at each struck. A key component in this balance modelling described by Goldstein was the VCE Data Service. level it has a responsibility to provide (1995, 1999) and Goldstein, Huiqi, Rath, these data to schools in ways that & Hill (2000) been used.There are Public reporting of the enable school staff to use them many reasons for this. effectively and easily. VCE since 2002 It is well known that a student of a Like other jurisdictions,Victoria has had given ability is likely to perform better From 2002, the achievement indices to deal with issues of accountability, when placed in a class of higher were discontinued. Instead, a wide

Using Data to Support Learning 47 range of information is published, Since 2004, the professional • How were our school’s overall results including information about: development program has been this year? Were they better or worse followed by a series of consultations in than in previous years? How do they • school programs (enrolments in which VCAA staff have met with school compare with schools like ours? VCE and VCAL, range of VCE and leadership teams for confidential advice VET studies offered) • How did our students perform, on the analyses they can generate from study by study? Are there important • student achievements (satisfactory their own school data.To provide this differences among groups within the completions in VCE,VCAL and VET assistance,VCAA staff have travelled to school (e.g., gender, class groupings) units, median study scores, all corners of the State, meeting with and between exams and percentage of high achievers) staff from more than 100 schools in coursework assessments? • student pathways (percentage applying each of the past two years. • How did our school’s results for and achieving tertiary selection and compare to reasonable various employment options). What does the VCE expectations? Did our students The information that is made public is Data Service provide to perform as well as students of the information that the public can use schools? comparable ability in other schools? to make decisions about schools. The VCE Data Service does not provide • Is our student cohort changing over Further, more detailed information is, of schools with a predefined set of reports. time? Can we detect trends in course, available from school web sites. Instead, it puts schools in touch with the student ability from year to year, and entire VCE data set going back to 1998, if so, are these changes matched by The introduction of the and invites users to generate reports in changes in student performance VCE Data Service response to the issues and research over the same period? questions that address their concerns. • Can we develop better The detailed information needed by They can, of course, only draw out understandings of the patterns of schools in their planning for school analyses related to their own school, group performance by identifying improvement is now made available and how its results compare to those of how individuals contribute to those through a new online service known as other schools in the state, schools in the patterns? Have the results of the VCE Data Service (VCEDS).This same sector (government, Catholic and particular students or groups of service was launched in November independent), and to schools in its Like students distorted overall patterns 2002, following several years in which a School Group. of achievement? limited form of ‘value-added’VCE reporting had been successfully trialled As in the trial that preceded it, VCEDS School staff are also able to design their in a sample of schools.The reports provides ability-adjusted estimates in own analyses by entering data fields available to schools through VCEDS which adjustment is made in each unique to the school and so introducing include, but are not limited to, the study for: comparisons tailored to their specific ability adjusted measures pioneered in • each of the three component GAT needs. For example, one school might the VCE Data Project. scores (Written Communication, like to see results broken down by campus, another by language The information provided by VCEDS is Mathematics/Science/Technology and Arts/Humanities) background, and another by residential confidential to the school, and access to location. An inventive use by one school • the mean GAT score of all students it is strictly controlled to protect its has been to monitor the long-term in the school taking the study, and privacy. Professional development effectiveness of an innovative middle gender. programs for teachers are provided in school program by comparing VCE February/March each year, focusing on But this is only a small part of the performance several years later. the range of reports available to schools information provided.Through VCEDS, Comparisons between the performance and how they can be obtained and senior management teams in every interpreted. Each year, these sessions school can easily generate charts and of males and females are, of course, have been attended by teachers from tables addressing questions like routinely provided, along with those between 200 and 300 schools. the following: between class groupings and student

Research Conference 2005 48 year levels. Schools can use VCEDS to • more intuitive reporting of adjusted In summary, users have drawn most monitor the performance of Year 11 estimates, and heavily on the following report types, students taking accelerated VCE • capacity to study the performance listed in the logical sequence that users programs, and use their findings to of individuals and their contribution might have approached them, rather assess the wisdom of the advice that to reports on overall performance. than their frequency of usage, which is students are receiving as they approach documented in Table 1 (over the page): their VCE studies. These and a number of other enhancements were included in a major • All VCE Studies (overall Study Score For each report, VCEDS provides users enhancement of the system in time for distribution) with a choice of a graph, a table, or the 2004 data upload. • Study score distributions, showing both.These reports may be sent directly multiple studies to the printer, or pasted into another User response to these improvements • Study score and assessment grade application, such as a Word document. has been monitored in the course of distributions (breakdowns within a In this way, schools can use VCEDS to 27 group professional development report publicly to parents, or sessions attended by 617 senior school single study) confidentially to department heads or staff and 102 individual school • Study score distributions, plotted individual teachers.With the information consultations in 2005, and has been over time overwhelmingly positive. provided through VCEDS, schools are • Adjusted score distributions, able to examine all aspects of their Total usage has grown steadily since the showing multiple studies students’ achievements in a more release of the 2004 data in the first • Adjusted score distributions detailed way than ever before, and to week of February 2005. Figure 1 shows (breakdowns within a single study) use the knowledge they gain to identify the cumulative hits recorded on the • Adjusted score distributions, plotted problems and build upon their strengths. first days of February and each month over time until June.The most heavily-accessed Schools’ experience report to this date has been Report 17 • Scatterplots of achieved versus with the VCE Data (Student Results by Study), which expected study scores shows a scatterplot of achieved versus • Results for individual students, Service predicted Study Scores for each study, across studies. The VCE Data Service in now in its third with 1692 hits in the four-month year of operation. After the first full period. Another seven reports have year, evaluation largely focused on the exceeded 1000 hits in this period. question ‘How can we make this better?’ During 2004, feedback was 18000 systematically sought, and the key improvements sought included: 16000 14000 • the capacity to generate detailed reports on assessment grades and 12000 scaled (VTAC) ENTER Subject 10000 Scores as well as Study Scores in 8000 each study; 6000 Cumulative hits Cumulative • access to student results by Home 4000 School and/or by Assessing School 2000 (for schools that share teaching 0 programs); 1 February 1 March 1 April 1 May 1 June • capacity to trace GAT scores over Date time and their relationship with achievement scores Figure 1 Total VCEDS usage Feb–June 2005

Using Data to Support Learning 49 Table 1 Cumulative VCEDS usage by report type: Feb-June 2005

Report type Mar 1 April 1 May 1 June 1

1. All VCE Studies (Study Score distributions) 296 740 876 1080 2. Assessment Grades (frequencies of each) 188 482 584 722 3. Study Score frequencies 104 265 330 418 4. Completions (Units 3 & 4 & qualifying for a Study Score) 65 160 200 268 5. Study Scores of 40 and above, over time if required 149 374 522 660 6. Location of school within Like Schools Grouping 70 186 233 325 7. Multiple VCE Studies (Study Scores) 284 804 949 1167 8. Single VCE Study (Study Scores and Graded Assessments) 242 769 922 1198 9. Single VCE Study (Study Scores over time) 246 728 895 1152 10. Multiple VCE Studies (adjusted scores) 250 678 802 1016 11. Single VCE Study (adjusted scores) 170 514 601 752 12. Single VCE Study (adjusted scores by time) 233 717 866 1087 13. Single Study adjusted scores (subgroup comparisons)) 179 521 611 796 14. Single Study (achieved versus predicted scores) 218 602 723 949 15. GAT and Study Score distributions (single studies) 178 511 613 795 16. GAT and Study Score distributions over time 155 395 494 699 17. Student results by Study (scatterplots of achieved versus predicted) 338 1101 1307 1692 18. Individual student results across Studies 298 781 895 1116

The ‘hit count’ records each time a user constantly finding new ways of using the part of their limited professional chooses that item from the next higher- information that they now have the development funds on buying in the level menu. For each hit, many reports power to access. expertise that will enable them to can be produced.The usage reports do become self-sufficient, and in our view But empowerment can be resource- not enable us to record the number of this would be money well spent, intensive. Over 2004 and 2005,VCAA actual reports accessed, printed or particularly if it leads to self-reliance staff have conducted 50 Professional pasted into other documents, nor can and not to continuing dependence on Development seminars in all areas of we determine the number of schools assistance from outside the school. Victoria, with attendance totalling 1600 accessing the service. Changes to the senior school staff. In addition, they have There are a number of issues that the system currently in train will provide held 236 individual consultations with VCAA needs to address. Some of these the latter information. staff from 161 different schools. Over are relatively minor, such as how many time, the need for this level of support years of data the system should retain Lessons learned from may decline, but with an influx of new for everyday access. User feedback is the VCE Data Service users each year, it is unlikely to go away. telling us that the current seven years is more than sufficient, and that when the The VCE Data Service is built on a Beyond the existing support program, 2005 results are uploaded, one or more philosophy of empowerment. It the capacity of VCAA staff to support years could be dropped. challenges its users, and those who rise schools to provide individual support is to the challenge are excited by the limited. Many schools are comfortably A vital issue for the VCAA to resolve is capacity that they gain by using it. self-sufficient already, and others can the uneasy compromise between the Feedback from users has been quickly become so with minimal desire for ease of access and the need consistently positive, and schools are support. Schools may wish to spend to maintain the confidentiality of the

Research Conference 2005 50 data.To protect confidentiality, access APPENDIX has been restricted to the Principal of each school, the VASS coordinator, and Sample reports from the VCE Data Service other users as nominated by the Principal.This may be over-restrictive, particularly in schools where the Principal is not aware of the service or nervous about security issues.The VCAA is planning consultations with user-Principals about the possibility of setting up different categories of users with access to different reports within the service. For example, we could have the category ‘Super User,’ with access to everything; a ‘Department Head’ category, with access to reports for a defined set of Studies (such as a Key Learning Area), and a ‘Subject Teacher’ category, with access to one Study only. These and a number of other issues will be resolved by consultation with users and implemented in time for the upload of the 2005 results in January 2006. References Goldstein, H. (1995). Multilevel statistical models (2nd ed.) London: Edward Arnold. Goldstein, H. (1999). Using value-added data for school improvement purposes. Oxford Review of Education, 25, 469-483. Goldstein, H., Huiqi, P., Rath,T., & Hill, N. (2000). The use of value-added information in judging school performance. London: Institute of Education. Available at: http://www.ioe.ac.uk/hgpersonal/ Using-value-added-information.pdf Johnson, J. 1997. Data-driven school improvement, ERIC Digest, No. 109. Available at: http://www.ericdigests.org/ 1997-3/data.html

Using Data to Support Learning 51 Research Conference 2005 52 Using Data to Support Learning 53 Research Conference 2005 54 Using Data to Support Learning 55 Research Conference 2005 56 Using Data to Support Learning 57 ‘Getting it Right’ Symposium – Paper 1 Getting it Right … using the right data effectively

Abstract path are those involved in the Getting it Right Literacy and Numeracy Strategy. The Getting it Right - Literacy and Numeracy Strategy is a targeted and Getting it Right was a $27 million pre- coordinated program of additional election commitment of Western support for government primary Australia’s Gallop Labour government schools in Western Australia.The which came to power in March 2001. program provides additional specialist The commitment included the training teaching personnel, professional and deployment of 200 specialist Rosemary Cahill development and support to schools teachers to work in government schools Department of Education and Training, across the government school system. to improve levels of literacy and Western Australia The key purposes of the strategy are to numeracy among high needs students improve literacy and numeracy with a particular focus on Aboriginal Rosemary Cahill is Manager, Literacy and outcomes across government schools, students and other groups who lagged Numeracy in the K–10 Academic Standards and and to achieve greater parity of behind the general population.The initial Support Directorate of the Department of outcomes across all groups of students. 4-year commitment expired in July this Education and Training in Western Australia. She The model for teachers’ professional year, but the strategy has proven so has been central to the planning and management successful that maintenance of current of the Getting it Right Literacy and Numeracy learning incorporates many features of Strategy from inception in 2001. Her teaching effective professional development staffing and support has been assured experience includes Special Education, ESL and identified in current research findings. into the foreseeable future. mainstream primary settings and her curriculum Central to that professional learning has development experience includes writing teacher been how to select, collect and analyse Planning resource materials in support of bidialectal education (Solid English and coordination of the credible diagnostic and summative The pre-election commitment set Deadly Ways to Learn package). student performance data to inform the down certain parameters for planning teaching and learning cycle, whole- (including the number of specialist school planning and resource allocation. teachers to be involved by 2005, broad A trend that is evident in all levels of outcomes, focus cohorts and budget) planning undertaken in recent years by but it left open details about the model the Department of Education and for implementation. In a nutshell, key Training in Western Australia has been implementation details that were careful and deliberate analysis of subsequently determined are as follows: performance data to inform resource • Specialist Teachers work in high allocation, actions and adjustments to needs schools. Not every school will actions.Through a series of get a turn. Relative needs are complementary teaching and learning determined through a combination professional learning initiatives being of systemic quantitative data and implemented in all WA government more localised qualitative data: schools at present, every principal and Western Australian Literacy and most teachers within the system Numeracy Assessment (WALNA) regularly hear about the virtues of data is factored against teacher formative assessment and ‘evidence- numbers to allocate full-time based planning’ as key features of equivalent (FTE) teacher time to outcomes focused education. An districts, then District Directors use increasing number of teachers and school performance data (WALNA principals have embraced these virtues; plus other school data) to prominent among the teachers and allocate the district FTE to high principals that have gone down this needs schools.

Research Conference 2005 58 • There is now a total of 300 Specialist and values of the Curriculum school system’s Outcomes and Teachers of literacy or numeracy Framework. Specialist Teachers do not Standards Framework says about working in 300 schools across WA, routinely withdraw groups of outcomes within the English and occupying (in either a full-time or a students from a class, and the Mathematics learning areas because part-time capacity) the overall classroom teacher maintains those documents capture the commitment of 200 FTE by 2005. responsibility for the progress of all version of literacy and numeracy • Specialist Teacher allocations are students in the class. In collaboration that WA government school made for 2-year periods. After an with the principal and other staff, the teachers are required to pursue. initial 2-year allocation, schools Specialist Teacher also facilitates This point is significant in the remain eligible, on the basis of implementation of a whole school context of evidence-based planning demonstrated need, for subsequent approach to literacy or numeracy and teaching because what counts 2-year allocations. and the systematic collection and for credible and useful data is analysis of student performance data. significantly coloured by what • Schools get a literacy or a numeracy version of literacy and numeracy Specialist Teacher.The rationale here • Principals of participating schools you are seeking to teach. is that participating schools are are required to set a 2-year school often challenging settings, and there improvement target for literacy or • A deep and thorough sense of would be a risk of overstretching numeracy, then to track and report ‘where their students are at’ in resources, energy and goodwill if (through normal quality assurance relation to progress towards those teachers in participating schools processes) their school’s progress outcomes – what they already were expected to make significant towards that improvement target. know and can do, and what they gains in literacy and numeracy at the • Principals of participating schools have yet to learn. same time. Rather, they are attend a 2-day induction workshop • A broad repertoire of teaching encouraged to focus whole- before the strategy commences in practices from which to select so they heartedly on improving one thing at their schools and Specialist Teachers can properly match student learning a time. participate in seven 3-day training experiences to student needs. workshops spaced across the two • Resources and attention are divided Clearly, the above three domains of years, the first of which occurs equally between literacy and professional knowledge are necessarily before they begin in the role. numeracy. Historically, literacy and linked in a cyclic process of assessment, Between workshops, a member of numeracy initiatives in WA have teaching and learning. Accordingly, the the Central Team provides ongoing unfolded as initiatives that focus on professional learning and support support and site visits to Specialist literacy … and numeracy if you’ve provided through the Getting it Right Teachers and principals. still got time.The ‘fifty-fifty’ split strategy gave balanced attention to within Getting it Right was a each. Given the theme of this strategic decision to redress this Professional learning conference, however, the focus to be imbalance, further influenced by the and support taken from here on with this paper will fact that First Steps in Mathematics be how Getting it Right was designed The basic premise of the Getting it research and resource development to help teachers and principals to make Right Literacy and Numeracy Strategy is was completed in 2002. more effective use of data about their that teachers make the difference. • Specialist Teachers (STs) work students’ achievement using a range of Further, that teachers are best shoulder-to-shoulder in classrooms sources, including the Literacy Net, equipped to make the biggest with colleagues, assisting with the Numeracy Net, diagnostic First Steps difference when they have: collection and analysis of student tools and WALNA data. performance data, using that data to • A deep and thorough understanding inform planning, modelling lessons of the outcomes students are The focus on data and team-teaching.The methodology required to learn. In particular, that modelled by STs is outcomes they know what the WA Curriculum From the first workshop that principals focused and reflects the principles Framework and the government and Specialist Teachers attend, they are

Using Data to Support Learning 59 told very clearly that the Getting it by their teachers does not tell them external body has told them to do Right strategy is a data-driven initiative much about progress in relation to our something about ‘issue A’ while they and that they will be required to make Outcomes and Standards Framework. believe a more pressing problem is systematic use of data in their own ‘issue B’.Thirdly, targets have become a Target-setting is often where the schools and classes to inform decisions. rallying-point for schools, giving them a ‘rubber hits the road’ for schools There are two significant data-driven tangible and collective focus. All staff involved in Getting it Right. Most tasks for principals which are outlined know that their school’s target is schools spend the first term looking during their 2-day induction workshop informed by good decisions about good back into their WALNA data in greater and are constantly revisited through data, so they can see the logic in detail to detect trends and gaps, and subsequent school site visits by directing resources, time and energy then turn to supplementary assessment members of the Central Team: target towards the priority areas captured in tools to gain a more fine-grained and setting and the allocation of Specialist their target. diagnostic picture of current Teacher time within the school.The performance and priority needs.The focus on data in classrooms is manifest Allocation of Specialist Teacher professional learning workshops as systematic monitoring of student conducted for principals and Specialist time within the school learning to inform planning.This is a Teachers anticipate this need by the large part of the support and advice To achieve a balance between impact inclusion of sessions about analysing provided by Specialist Teachers when and coverage and to avoid spreading WALNA data, using the Literacy Net, working with classroom colleagues. Specialist Teachers so thinly that they Numeracy Net and ESL Bandscales, and become ineffective, it is recommended getting to know developmental phases, Target-setting that Specialist Teachers work about 1 teaching emphases and diagnostic day per fortnight or half a day per week The only extra reporting requirement uses of First Steps English and with each classroom colleague he/she is placed on schools involved in Getting it Mathematics resources. asked to support. It follows that if a Right is that they have to set a Several important benefits have school received 0.8 FTE, the Specialist challenging yet realistic school emerged from the way target-setting Teacher could be expected to work improvement target for literacy or has applied in the Getting it Right with about 8 classroom colleagues – numeracy, and to track and report strategy. Firstly, the need to set targets more if they were highly experienced, progress towards that target.To ensure in relation to the outcomes of the less if they were inexperienced or had that school improvement targets are English and mathematics learning areas very challenging classes. It also follows genuinely challenging yet realistic, has forced principals and teachers to that some classroom colleagues will not principals are required to gain make sure they know what those receive direct support from a Specialist endorsement for the target from his/her outcomes are about. It is felt that a Teacher so principals need to decide District Director. Progress towards school significant factor giving rise to this who will get the support and who will improvement targets is then reported benefit is that schools were not told not.To inform these decisions, principals through normal quality assurance what their target would be, nor what are directed back to the performance processes with District Directors. measure they had to use. Rather, they data of students in various classes, plus The spectre of target-setting is generally looked at the strengths and limitations consideration of qualitative data greeted by principals with some horror. of various assessment instruments, regarding teacher strengths and other To start with, if you are going to set a determined the extent to which those forms of support available in the school. target for a point in the future, you first instruments yielded information in need to know a fair bit about the relation to our outcomes, and made an Systematic monitoring of platform from which you are working, informed decision. Secondly, schools individual student and class needs and many principals know that the data have a strong sense of ownership As the focus shifts from whole-school they have about their school’s current relating to the targets they have set, so to class and on to individual students, performance is, at best, rather thin. have a much clearer sense of purpose the imperative to use data to inform When they start digging, many also find with their planning in relation to those planning continues. In classrooms, it is that a lot of the data that is collected targets. It is not the case that an

Research Conference 2005 60 necessary to supplement the them.While they are under no illusions exercised by District Directors in quantitative evidence of class trends about the accountability agenda and reviewing school targets is unknown, and individual needs with more fine- how systemic data contributes to that so it would not be prudent to grained assessments of progress agenda, they also understand how claim success on the basis of this towards the outcomes.To this end, WALNA data and other qualitative finding alone. Specialist Teachers are trained to help sources of data can be blended to their colleagues to use the First Steps provide a rich picture of progress and Systemic WALNA data diagnostic maps and the Literacy Net, needs and are essential tools to To guard against freedom of information the ESL Bandscales and the Numeracy support and inform school, class and requests for school-by-school WALNA Net to monitor progress. Collectively, individual improvements. data which could reveal individual student these fine-grained qualitative assessment performance in small schools,WALNA tools help teachers determine the What the data tell us data is not centrally collated school-by- nature of any difficulties that students so far school so rates of improvement in are experiencing so they can be more ‘Getting it Right schools’ are not easily focused and systematic in their planning Three layers of data collection have compared with those of ‘non-Getting it and teaching. been established to monitor the extent to which Getting it Right is proving Right schools’. Individual schools who have An early assumption made by some of effective in meeting its intended been participating in the strategy for the teachers with whom Specialist outcomes and to inform adjustments more than two years, however, provided Teachers worked was that the Specialist that may be warranted: school target WALNA data to Central Office for Teacher would ‘do’ the assessment or setting, systemic WALNA data and an analysis at the end of 2004.When ‘do’ the planning – or better still, take external evaluation being conducted by compared with 2001 data, the 2004 the more difficult students away – so the ACER. literacy or numeracy data from these the classroom teacher could get on schools indicated an overall improvement. with the real business of teaching. Not School target-setting Given that students in many participating so. Rather, Specialist Teachers worked schools are highly transient, these gains alongside their classroom colleagues to The requirement for schools to are significant: students who have jointly determine student learning and negotiate with District Directors benefited from the strategy may have future needs, then to work out realistic yet challenging improvement moved prior to the test, only to be targets and to report progress in appropriate instructional foci and to replaced by other students who are relation to those targets was outlined plan accordingly.The mismatch between struggling and have not been at the earlier.There is clear qualitative early assumptions and the actual school long enough to benefit. support Specialist Teachers were evidence that target-setting has been primed to provide needed to be pivotal to getting schools to make It is noteworthy that gains were less carefully managed by principals. In time, constructive use of assessment evident in difficult-to-staff locations in however, classroom colleagues have information. Further, that positioning which teacher turnover is high.This is to found that the time spent analysing target-setting as being primarily about be expected in a strategy which seeks work samples and probing students’ improvement rather than accountability to effect student improvements by thinking through insightful tasks and has given principals the confidence to building the capacity of teachers – if the questions is a good investment because be open about what worked, what teachers with increased capacity are it enables them to be far more efficient didn’t work and what adjustments they constantly moving, their improved and effective teachers. intend to make in the future. teaching practice will benefit students in the school they move to, not those in A breakthrough achieved in Getting it The vast majority of participating the school they leave.This finding signals Right schools is that principals and schools have reported that they either the need to consider supplementary teachers are inclined to view data as a met or exceeded their school ways to support such schools, many of ‘friend’ they can use to support their improvement targets.This could be which are in the country and receive a work rather than as something that is taken to be a very positive result. On large proportion of graduate teachers. used primarily by others to ‘check’ on the other hand, the degree of rigour

Using Data to Support Learning 61 Over time, it is expected that the very Concluding comments gradual improvements evident in WALNA data over the past decade will Among the principals and staff at accelerate, and that these gains will be schools participating in Getting it Right, most prominent at the tail. It is too early, there is almost universal agreement that however, to detect such trends. Further, that this strategy is making a significant when they do become evident, it will not difference in their schools.While the be possible to attribute them entirely to calibre of Specialist Teachers, the Getting it Right because this is one of leadership provided by the principals of several improvement initiatives supporting participating schools and the quality and WA government schools at present. depth of expertise provided by the central Getting it Right team are pivotal ACER evaluation ‘people’ factors contributing to the strategy’s success, a number of An independent, external evaluation has structural factors that are particular to been commissioned by the Australian the model adopted have also Council for Educational Research contributed to its success. Central (ACER) to monitor teacher among these is the systematic and development goals of Getting it Right, in deliberate use of high quality data that particular, the extent to which Specialist tells us something important about the Teachers, their classroom colleagues things that matter at every stage and and their principals: every level of Getting it Right as it a Develop improved understandings, unfolds. confidence and teaching skills in relation to literacy and numeracy; b Understand Curriculum Framework outcomes relating to literacy and numeracy, especially those set out in the English and Mathematics learning areas; c Collect and analyse credible diagnostic and summative student performance data to inform the planning and teaching cycle; d Participate in cohesive, data-driven, whole-school planning for literacy and numeracy; and e Participate in and engender two- way home–-school collaboration and communication in support of literacy and numeracy development. The ACER has recently submitted a final report from this evaluation and its findings will be made public in due course.

Research Conference 2005 62 ‘Getting it Right’ Symposium – Paper 2 Getting professional development right

Rosemary Cahill, in her paper in these beliefs and practice, suggests it would conference proceedings, has described be worthwhile examining its main the Getting it Right (GiR) strategy for components in relation to research on improving levels of literacy and professional learning for teachers. numeracy among high needs students. This paper examines the Western Linking policy to Australian Getting it Right reform as a practice strategy for professional learning and Lawrence Ingvarson compares it with research on the The challenge of building strong links characteristics of effective PD. between reform policy and Australian Council for Educational Research implementation is a perennial one in The Getting it Right Strategy is clearly a education. A common refrain in Dr Lawrence Ingvarson, (B.Sc. Dip.Ed., M.A., comprehensive and well-resourced evaluation reports of educational Academic Dip. Ed, PhD) is the Research Director of reform strategy with its main emphasis ACER's Teaching and Learning research program. He reform efforts is the lack of fit between on building professional capacity among is internationally recognised as expert in the areas of ambitious goals for school improvement teacher education, professional development, school teachers and principals.The data we and the resources necessary to bring improvement, the measurement of change and the gathered as part of the evaluation, evaluation of educational programs. As a former about significant change in practice. through school and classroom Associate Professor of Education at Monash Policy makers can also have quite naive observations, interviews and surveys, University, he taught courses on educational expectations about how easy it is to evaluation, school improvement and professional left us in no doubt that the strategy bring about educational change, not development at master and doctoral levels. He is a was highly regarded by teachers and member of the National Staff Development Council, understanding that the kinds of change principals and was having a significant the American Evaluation Association and was a that really matter in education are not member of the Ministerial Advisory Committees for impact on practice.This paper will focus structural changes but those that build the Victorian Institute of Teaching and the mainly on GiR work focused on Development of a TAFE Centre. teacher capacity and professional improving numeracy teaching. He has written several commissioned reports on culture.There are no short cuts to school improvement and professional development Most teachers we observed and for the OECD and the Commonwealth educational improvement. interviewed were readily able to give Government, including the National review of Peterson, McArthy and Elmore’s (1996) Teacher Education (1984) and a National Strategy specific examples of how the GiR research, for example, cast doubt on for the Improvement of Science Teaching (1992). strategy had transformed their Recent publications include Valuing Teachers’Work: the capacity of ‘restructuring’ reforms in mathematics teaching.This comment New Directions on Teacher appraisal, published by the United States to benefit classroom ACER (1994) and Professional Credentials: Standards from a teacher is typical: for Primary and Secondary Science Teaching in practice.This was because: Australia, commissioned by the Australian Science I don’t set limits to my Changing practice is primarily a Teachers’ Association (1995). expectations, or their expectations, problem of teacher learning, not a for what they can learn any more He has worked extensively in Australia, the UK and problem of organisation ... School the USA on reforms related to professional ... because I know they can get structures can provide development, the quality of teaching and teacher there. Because of the diagnostic career structures. He was commissioned by the opportunities for the learning of tools, I’m listening much more to Australian Council of Deans of Education in 1993 to new teaching practices and new their thought processes, to how prepare a report on teacher registration and the strategies for student learning, but accreditation of teacher education in Australia. He they work it out. I’m getting them structures, by themselves do not also conducted extensive research on the to reflect more, orally, to find out cause learning to occur ... School introduction of the Advanced Skills Teacher in what thought processes they are Australia in the early 1990s and was recently invited structure follows from good using. So I can tell much better by Elsevier press to edit a book on “Assessing practice, not vice versa. (Peterson, whether they really understand or Teachers for Professional Certification” in the series McArthy & Elmore, 1996, p. 149) Advances in Program Evaluation.The book brings not – pen and paper tests don’t together the considerable research and development tell you that. This is a lesson we understand well in work conducted on teaching standards and performance assessments by the US National Board The success of the Getting it Right Australia, since the disappointments of for Professional Teaching Standards since it was Strategy in linking State Government school management reforms in the established in 1987. policy to significant change in teachers’ 1990s.There was no logic to these

Using Data to Support Learning 63 reforms linking changes in school into some of the data we have for about half a day each week for each management to teacher learning and gathered in evaluating the strategy. teacher, for two years. new practices. Over the past decade, While it is to be expected that there Central training: Specialist Teachers increasing numbers of researchers have will be significant differences in receive extensive and intensive training identified the existence of an active, implementation across schools, there is and support from a central GiR team in accountable professional community no doubt that the strategy is having using the First Steps in Mathematics within and across schools as important significant effects in schools where it has materials and in research related to for effective teacher development and been implemented as planned.What I learning the mathematics. The training high quality teaching (Little & McLaughlin, would like to do here is bring out the and support takes place over two years 1993; Louis, Kruse & Marks, 1996). key features of the strategy by – 21 days spread over seven three-day comparing it with research on Richard Elmore from Harvard has spent training sessions run by Central characteristics of effective professional many years studying the problem of GiR staff. development. ‘scaling up’ good educational practices. Working ‘shoulder –to –shoulder’: In a recent comment on the US ‘No Before making that comparison it is Initially, Specialist Teachers work Child Left Behind Act’, and the necessary to give a brief outline of the unrelenting pressure to improve schools key components of the GiR strategy. alongside colleagues, helping them in without corresponding improvement in the collection and analysis of student teachers’ skills, he states,‘In its least Main components of the performance data, using that data to inform planning, modelling lessons, and desirable face, educational reform can Getting it Right Strategy become a kind of conspiracy of team teaching.The classroom teacher ignorance: policymakers mandating for professional learning retains responsibility for the mathematics learning of the children in results they do not themselves know The components listed below give only his or her class. how to achieve, and educators an overview of the main ‘pieces’ in the pretending they do know what to do GiR strategy.This list should be read in In a typical week Specialist Teachers but revealing through their actions that conjunction with Rosemary’s ACER spend half an hour or so planning the they don’t.’ conference paper, which places the GiR next week’s session/s and an hour or so A feature of the WA Getting it Right Strategy in a broader context of state teaching with that teacher using the Literacy and Numeracy Strategy is the educational reform. activities they had planned together. From time to time, the Specialist depth of understanding it reveals of Curriculum: A high quality, research- Teacher might assist the teacher to run what it takes for reform policies to based curriculum development penetrate to the level of everyday resource – First Steps in Mathematics. a diagnostic test to monitor progress in practice.The Strategy is primarily about Teachers use this resource understanding and identify difficulties enhancing the capacity of existing collaboratively to plan the school’s students might be experiencing.The teachers to meet the needs of children mathematics curriculum, to plan Specialist Teacher is not to act as a at risk. Rosemary Cahill has revealed learning activities tailored to students in support teacher or routinely teach that this is a targeted and coordinated their classroom and to map groups of students withdrawn from a program that directs serious money at development in their mathematical class.The Strategy places heavy reliance a serious problem.The strategy reveals thinking. on the professional judgement of the a sophisticated understanding of the teacher and on informing that complexities of change and the Specialist Teachers: High needs schools judgement. conditions that need to be in place if are given a Specialist Teacher allocation Preparation of school principals: professional development is to make a for a two-year period. Schools select a Special sessions are held for principals difference to student learning highly regarded teacher with interest in target setting and in identifying outcomes. and expertise in mathematics numeracy. The Specialist Teacher is released from specific actions they can take to Rosemary has described the overall classroom duties to work ‘shoulder to support the work of the Specialist strategy. Marion has provided a glimpse shoulder’ with a number of colleagues, Teacher.

Research Conference 2005 64 Whole school approach: While the considerable.There is, however, an The content of professional Specialist Teachers work mainly in the emerging synthesis of findings from development is critically important early grades, they also works towards these studies about the conditions that to its effectiveness.While the lifting awareness about the GiR Strategy foster professional learning that relates content varies with the goals of among all staff and implementing a to improved student learning outcomes, the school, the content of PD should deal directly with what whole-school approach to improving particularly in the core areas of literacy students are expected to learn numeracy outcomes. and numeracy. and the instructional strategies that Hawley and Valli (1999) summarise this research and experience have Comparing the GiR research in a list of nine principles for shown are effective. Strategy with research the design of effective professional This characteristic of effective on effective professional learning (Table 1).The GiR Strategy will professional learning emphasises the learning be discussed in relation to each of overriding importance of what teachers these principles. learn, as opposed to how they learn it. There are many lists of characteristics 1. Hawley and Valli’s first principle for As Kennedy (1999) puts it, the form of of effective professional development the design of effective professional professional learning turns out to be activities. Few are grounded in rigorous learning states that: less important than the what, the research based on examining the effects substance or content.This finding The content of professional of professional learning programs on challenges the strong emphasis that has development (PD) focuses on what student learning outcomes.This should been placed for many years on the students are to learn and how to not be surprising as the methodological address the different problems processes or structures used in problems in tracing the links between students may have in learning professional development activities, such PD and improved student learning are the material. as whether they are planned Table 1

Principles for the Design of Effective Professional Development (Hawley & Valli, 1999)

1. The content of professional development (PD) focuses on what students are to learn and how to address the different problems students may have in learning the material. 2. Professional development should be based on analyses of the differences between (a) actual student performance and (b) goals and standards for student learning. 3. Professional development should involve teachers in the identification of what they need to learn and in the development of the learning experiences in which they will be involved. 4. Professional development should be primarily school-based and built into the day-to-day work of teaching. 5. Professional development should be organized around collaborative problem solving. 6. Professional development should be continuous and on-going, involving follow-up and support for further learning-including support from sources external to the school that can provide necessary resources and new perspectives. 7. Professional development should incorporate evaluation of multiple sources of information on (a) outcomes for students and (b) the instruction and other processes that are involved in implementing the lessons learned through professional development. 8. Professional development should provide opportunities to gain an understanding of the theory underlying the knowledge and skills being learned. 9. Professional development should be connected to a comprehensive change process focused on improving student learning.

Using Data to Support Learning 65 collaboratively or whether they are gap between actual student the following week to help the children one-off or long term. It turns out that performance and goals for student to overcome those difficulties.Though knowledge is the key when it comes to learning. Goals for student learning there is not enough space to document generative professional learning, also provide a basis for defining it here, extensive research underpins particularly when it leads to deeper what teachers need to learn and a the diagnoses of student learning and yardstick for improving PD. understanding of the content that the learning activities to promote better students are to learn, the research on This principle emphasises the understanding. how students learn that content and importance of focusing professional As an aside, it was common to hear the nature of the problems different learning on data and feedback from teachers in GiR schools express students have in learning that content. one’s own students, especially data considerable surprise about the about where those students are in The Getting it Right Strategy is firmly expertise and confidence they had relation to where they could be, or based on this kind of content focus.The accumulated in analysing student should be, in their development. ‘what’ that occupies most of the GiR performance when they met with Contrary perhaps to initial concerns professional learning is knowledge teachers from non-GiR schools at about standards for student learning about mathematics, research about how ‘making consistent judgements’ expressed some years ago, research- students learn that content and the meetings. based standards have proved to be an stages in their developing important lever for fostering productive 3. Hawley and Valli’s third principle understanding.Training sessions for dialogue about the purposes of links to the previous two principles. Specialist Teachers are rich with education and have given teachers opportunities to deepen understanding Professional development should something to be collegial about. Some about mathematics concepts, and to involve teachers in the identification of the most effective professional become more perceptive about the of what they need to learn and in learning now comes through activities the development of the learning nature of learning difficulties. Recent that help teachers to ‘moderate’ or experiences in which they will research articles are available. Back in compare their own students’ work and be involved. schools, the Specialist Teacher works development with that of other with teachers to find out what the Adherence to this principle teachers’ students.These activities children know and what they need to ensures that PD is relevant.When provide a valuable means of learn next, then they plan how they will teachers help design their own ‘deprivatising’ teachers’ practices and learning, they are likely to feel a work together to bring about that opening up more avenues for feedback greater sense of involvement in learning.These meetings focus on and professional accountability. the PD experience.Teachers are selecting appropriate learning activities most likely to use what they learn for children that will progress specific This principle is at the very heart of the when PD development is focused understandings in mathematics.The GiR strategy. At almost every meeting on solving problems in their focus is on strengthening, not between a Specialist Teacher and a particular contexts. supplanting, the professional judgement classroom teacher, they will be Together these first three principles of the teacher. examining the work that students did stress the importance of making the previous week in response to the 2. Hawley and Valli’s second principle of practice, and evidence about practice, learning activities they chose.They will effective professional learning states that: the site for professional learning. interpret this work, making use of Practice-based professional learning Professional development should be Diagnostic Maps, student outcomes represents a major shift from traditional based on analyses of the differences levels and Key Understandings.They use views of professional learning based on between actual student performance this work to sort students into groups participation in ‘courses’. This is not to and goals and standards for student according to the difficulties they are learning. imply that courses and other activities having and their phase of development such as workshops, conferences and Professional development that is with respect to the mathematical seminars do not have an important role based on analysis of student concepts in question.They will then in supporting professional learning. But learning helps teachers close the plan appropriate learning activities for

Research Conference 2005 66 these kinds of activities are only the are expected to teach and the means The Getting it Right Strategy achieves ‘front end’ of the change process.We to be more discerning about their this penetration to the level of practice have known for a long time that the students’ learning of that content. As almost painlessly. However, the availability ‘back end’, the implementation stage of one would expect, teachers varied in and the training of the Specialist Teachers the change process, is where the hard their openness to First Steps, but the are crucial – and the fact that the work has to be – supporting teachers benefits reported by other teachers Specialist Teacher is usually another as they test new approaches in their and the availability of the Specialist teacher from the same school.The own classrooms (Fullan, 1982). Very Teacher as an extra resource in ‘shoulder to shoulder’ concept is few PD strategies put the level of planning and teaching usually proved irresistible to most teachers who do not resources into the implementation and too difficult to resist. want to be told what to do, but do want continuation stages that GiR does. to know anything that helps them help We asked teachers how they saw the their students learn better.The Specialist The third principle runs counter to Specialist Teacher, and this response Teachers have the kind of in-depth conventional wisdom about professional is typical: training from the GiR team that makes development in some respects. Getting As someone who is a bit more them a valuable resource in negotiating it Right identifies what teachers need to knowledgeable, but one of us. It’s easy the complex First Steps Curriculum know and be able to do to teach to go to her.We know she is there to Development Resources.The ‘shoulder mathematics more effectively rather change the way we teach mathematics. to shoulder’ notion captures the notion than what they might want to know. of partnership well – that ‘we are going But what they need to know in the GiR 4. Hawley and Valli’s fourth principle to work together’. Despite our initial Strategy has a strong foundation in states that: scepticism about the possibility of such a research and proven practice. Spending Professional development should be relationship, we did not come across any more time on mathematics may not be primarily school-based and built into teachers who did not value highly the the highest priority for some teachers. the day-to-day work of teaching. opportunity to work with the Specialist In fact they may avoid PD courses in Teachers learn from their work. Teacher in their school. mathematics and, as some teachers we Learning how to teach more interviewed admitted, they may cover effectively on the basis of 5. Hawley and Valli’s fifth principle the mathematics part of the curriculum experience requires that such relates closely to the fourth: in a less than enthusiastic manner.With learning be planned for and Professional development should be GiR, the Specialist Teachers take the evaluated. Learning needs arise organized around collaborative knowledge and the professional learning and should be met in real problem solving. to the teacher where they work and contexts. Curriculum development, Without collaborative problem where they can test it out. assessment, and decision-making processes are all occasions for solving, individual change is Many teachers we interviewed in the learning.When built into these possible, but school change is not. course of the evaluation made routine practices, PD powerfully Collaborative problem-solving comments along the lines that the GiR addresses real needs. activities allow educators to work numeracy strategy made them feel together to identify both problems This principle has been promoted for more like a ‘professional’.When pressed and solutions. Activities may many years. Over thirty years ago, to explain what they meant, they would include interdisciplinary teaming, people were promoting ‘school-based in- curriculum development and say they felt more like ‘experts’. They service education’, or ‘school-focused critique, collaborative action now had knowledge that gave them a professional development’. It can mean research, and study groups. stronger basis for interpreting student little, as in simply transferring passive learning outcomes and deciding what The GiR Strategy builds on long course modes of PD into the school on students needed next.The GiR strategy experience that effective professional curriculum days. The difficulty is in deliberately avoids telling teachers how learning opportunities arise from building opportunities for teachers to be to teach, but it does aim to provide collaborative work on authentic actively engaged as professional learners teachers with deeper knowledge about teaching tasks and problems. Motivation in the context of their day-to-day work. (and interest in) the mathematics they to engage in this kind of learning

Using Data to Support Learning 67 increases with evidence of improved follow-up and support ensures that happen without the other key student understanding and enjoyment. professional development ingredients in acquiring new skills; The fact that there is a brief time span contributes to real change and modelling of the theory and in GiR between a planning meeting, continuous improvement. opportunities to practice the ideas teaching together and meeting again to This component of professional learning yourself and receive feedback.The examine student work and review the design is probably one of the major Specialist Teacher brings these learning activities greatly helps.There is strengths of the GiR Strategy for opportunities into the classroom. a direct connection between learning, improving learning opportunities for Research has indicated it may take two application and feedback. disadvantaged students. Perhaps the to three years for the kind of significant What becomes possible with the greatest weakness of professional changes in pedagogy that GiR calls for resources that the GiR Strategy makes learning for teachers is the lack of to take hold (Hodges, 1996). Under the available is a movement toward the funding for follow-up and support GiR Strategy, schools were funded for notion of the school as a professional when teachers come to implement the at least two years, and the support for organisation. Professional organisations, as innovation in their own classrooms.This a Specialist Teacher often continued described by Weick and McDaniel is when the need for support is at its into a third year. Schools often put (1989), recognise that professional work highest if professional learning is to additional funding of their own to is not just ‘up front’ work. Professional translate into practice. extend the number of teachers that work requires the ‘back room’ work of First Steps in Mathematics is a complex Specialist Teachers could work with. interpretation to inform decision-making. package of resources for diagnosing 7. Hawley and Valli’s seventh research- Work structures in professional students’ developing understanding of based principle states that: organisations recognise that effective mathematics and planning and teaching requires time during the implementing teaching programs to Professional development should working day to bring values and improve student learning. Left at the incorporate evaluation of multiple sources of information on (a) expertise to bear on the non-routine school door, or even explained at some outcomes for students and (b) the problems involved in meeting the central professional development event, instruction and other processes learning needs of all students.This it is very unlikely that teachers would that are involved in implementing principle, like the others, requires strong use these resources. At first reading, the the lessons learned through leadership at the school level to ensure GiR material is vast and rather professional development. collaborative work is actively supported impenetrable.Working ‘shoulder –to - When done right, evaluation of and that the Specialist Teacher are able to shoulder’ with the Specialist Teacher professional development yields say ‘no’ to other demands on their time. turns the learning process into many important lessons for refining 6. Hawley and Valli’s sixth research- small achievable steps. professional development.Without based principle states that: The GiR strategy has an ambitious such evaluation, future opportunities for teachers to learn Professional development should be vision for mathematics classes. Students may not be productive. Multiple continuous and on-going, involving will be actively engaged in constructing sources of information should be follow-up and support for further their own mathematical knowledge. used, including teacher portfolios, learning-including support from Teachers will know how to tap into this observations of teachers, peer sources external to the school that thinking.Teachers will be adept at evaluations, and student can provide necessary resources and promoting mathematical thinking and performance. Lessons become new perspectives. maintaining high quality discussion of most clear when evaluators collect Adoption and implementation of mathematical ideas.The need for props data during different stages of the effective practices requires like worksheets and textbooks will fade change process. continued learning.Therefore the away.This kind of pedagogy will not A valuable aspect of the GiR strategy design of professional development happen without a deep understanding was the realisation that evaluation must provide time to apply new of the mathematics and how children should be built into the strategy early ideas and, sometimes, must draw on learn the mathematics. Neither will it additional outside expertise. Such on. ACER was contracted to conduct

Research Conference 2005 68 the evaluation in mid 2003 over the 8. Hawley and Valli’s eighth research- al., 1993; Fennema, et al., 1996) showed next two years when new cohorts of based principle states that: the futility of PD that focused on Specialist Teachers were still being teaching techniques, as opposed to Professional development should trained. Funding for Specialist Teachers provide opportunities to gain an deepening teachers’ understanding of often continued into the third year.This understanding of the theory research about the development of made it possible to track changes over underlying the knowledge and skills children’s mathematical thinking within time and for the evaluation team to being learned. particular content domains. Expansion feed information back to the GiR team. and elaboration of the professional Because beliefs filter knowledge knowledge base leads to what they The key questions for the evaluation and guide behaviour, professional called ‘generative’ or sustained change concerned the impact of the GiR development must address (Franke et al., 1998).This understanding Strategy on teachers’ knowledge and teachers’ beliefs, experiences, and was a necessary condition for significant practice, though not student outcomes. habits. Furthermore, specific shifts in teachers’ beliefs and practices. The GiR team made a policy decision knowledge and skills that work in one setting, sometimes do not Effective pedagogy depends on early in the evaluation not to use work in others.When teachers knowledge of subject matter and how Western Australia Literacy and have a good understanding of the students learn it. Numeracy Assessment (WALNA) data theory behind particular practices for assessing student outcomes. Funding and programs, they can adapt the In the GiR Strategy, Specialist Teachers for the evaluation enabled several strategy they learned about to the have 21 days of PD over two years sources of data about the impact of the circumstances in which the teacher focused on this kind of knowledge.The GiR strategy to be gathered. is trying to use it. experience of gaining this knowledge led several Specialist Teachers to say These sources included visits to schools This principle relates closely to Principle spontaneously that,‘I’m feeling like a to conduct structured classroom 1 and the central importance of the professional for the first time’. Specialist observations and interviews with content that is learned in professional Teachers draw on this knowledge back teachers, Specialist Teachers and development. Change in practice is in their schools in working with principals.The ACER team visited more likely to be pervasive when it is classroom teachers.Their weekly twenty schools on three occasions in an informed by theory in which the meetings, where they examine student attempt to trace changes that could be educator involved has confidence. work from the previous week, identify attributed to the GiR Strategy. Surveys Reforms such as First Steps set types of misunderstanding and select of teachers, Specialist Teachers and ambitious goals for teachers and learning activities appropriate to those principals were also conducted on two students, especially that mathematics students, provide an authentic context occasions – late in 2003 and late in lessons will be characterised by lively in which to link the research to 2004.The surveys included innovative discussion of significant mathematical practice.This real work context brings methods for gathering information ideas. More teachers will help students teachers’ current beliefs, experiences, about the impact of the Strategy on test their own mathematical and habits to the fore – a necessary teachers’ knowledge and practice. constructions, and think critically about condition for change to happen. Teachers were presented with mathematical procedures. For some scenarios that called for them to apply Working ‘shoulder to shoulder’ means teachers, this involves a transformation what they had learned from the GiR the Specialist Teacher can bring useful in their knowledge, beliefs and practices Strategy; for example, about diagnosing knowledge to the core work of that goes to the heart of their identity student understanding and selecting planning and teaching. Practice is as a teacher. It was common for learning activities to promote key deprivatised. In the best situations, teachers to state in interviews that,‘I’ll understandings. Later in the evaluation, Specialist Teachers model new practices never teach maths the same way again’, it was common for Specialist Teachers frequently and teachers receive plenty as a result of their work with the and principals to show the evaluation of informal feedback as they try the Specialist Teacher. team evidence of improved outcomes practices out for themselves.This in numeracy that they attributed to the Earlier research, on which First Steps in protected environment enables GiR Strategy. Mathematics draws (e.g. Carpenter et teachers to take risks and experience

Using Data to Support Learning 69 different types of learning themselves. period of time. Principals have being free to work alongside colleagues, Teachers see the benefits of what they customised training in the kind of individually and in groups, the Specialist are learning in their students’ enjoyment support they can provide to enable Teacher makes it more possible for the of the activities. Specialist Teachers to work effectively. school to review in depth how well Clear guidelines are provided about students are being served.The Specialist 9. Hawley and Valli’s ninth research- what the Specialist Teacher’s role entails Teachers act as a bridge between based principle states that: – and what it does not.Time for research and the ‘dailiness’ of teaching. Professional development should be Specialist Teachers and classroom They help to break down isolation and integrated with a comprehensive teachers to plan and teach together is the persistence of privacy in teaching. change process focused on built into the timetable. While we found variation from school improving student learning. to school in the way the role was Improving teacher capabilities Concluding comment implemented, the role itself was greatly without changing the conditions valued in every case.We were that influence the opportunities to The GiR Strategy is consistent with surprised how most specialist teachers, use these capabilities is often research about the characteristics of who came from within the ranks of the counter-productive.These effective designs for professional staff, were accepted and valued in their conditions include time and learning. It illustrates how far we have new role.When asked how she saw the opportunities to try new practices, come over the last thirty years or so Specialist Teacher in her school, one adequate funding, technical since professional development was assistance, and sustained central teacher expressed the views of many equated mainly with one-off teachers we spoke with:‘She’s a bit office follow through.Thus, unless workshops. In these final remarks, I professional development is more knowledgeable, but she is still one would like to draw attention to one designed as part of a larger change of us. It is easy to go to her.We know interesting aspect of the GiR Strategy process, it is not likely to she is there to change the way we that take us beyond Hawley and Valli’s be effective. teach maths, but that’s OK’. list of principles. The fact that GiR PD is part of a One way to think about the Specialist broader reform strategy is clearly a Freeing up expertise:The role Teacher role is as a means of ‘freeing up strength of the GiR Strategy.The main of the Specialist Teacher in the expertise’ in the school and making it components of this strategy were listed GiR design more available.When you see a earlier. Data about student learning Specialist Teacher at work with outcomes has been used to identify an The role of a well-trained Specialist individual teachers and with year level undeniable need.The strategy has been Teacher is pivotal to the success of GiR. teams of teachers, assisting with the planned on several levels, from the Without the Specialist Teacher, it is hard diagnostic maps, with the Numeracy centre to the Region, the school and to see how any of the Hawley and Valli Net, the rotation of classroom activities the classroom – and over an extended principles could be implemented, yet, and so on, you wonder why this role time period. It has strong political and they make no mention of such a role in and this type of leadership has not financial backing from the Minister.The their list of conditions that appear to been a normal part of school staffing focus on building professional capacity nurture effective professional before.Teachers think that the most as the means of improving learning development. important source of useful ideas for outcomes in disadvantaged areas is their teaching is other teachers, yet The Specialist Teacher concept points clear. First Steps in Mathematics is a school organisation often makes that to a new teacher leadership role that is well-researched and comprehensive expertise inaccessible as teachers are worth considering as a more curriculum development resource. locked away in the isolation of their permanent component of school Funding for each school is substantial own classrooms. One thing that young staffing. Specialist teachers do what and typically equivalent to an extra staff teachers value highly is the chance to formally appointed school leaders member’s salary.There is a strong see expert teachers at work and to get ought to do, but rarely actually do.They central team to provide training for the helpful feedback from them about their make the concept of an accountable Specialist Teachers over an extended own teaching. Greater opportunities for professional community a reality. In

Research Conference 2005 70 modelling and feedback are key features Fennema, E., Carpenter, A., Franke, M., achievement: Restructuring schools for of the GiR strategy. Levi, L., Jacobs,V. & Empson, S. (1996). intellectual quality. San Francisco: A longitudinal study of learning to use Jossey-Bass. The GiR Strategy puts resources where children’s thinking in mathematics they are most likely to have an impact Weick, K. E. & McDaniel, R. R. (1989). instruction. Journal of Research in on student opportunities to learn.The How professional organisations work: Mathematics Education, 27, 403-434. English have been looking at Implications for school organisation and ‘remodelling’ teaching (Collarbone, Franke, M., Carpenter, A., Fennema, E., management. In T. Sergiovanni and J. H. 2004). Part of the motivation for this Ansell, E. & Behrend, J. (1998). Moore (Eds.), Schooling for tomorrow: arose from studies of teacher workload Understanding teachers’ self-sustaining Directing reform to issues that count. and stress. Remodelling includes generative change in the context of Allyn & Bacon, Boston, pp. 330-355. stripping non-teaching clerical and professional development. Teaching administrative tasks that limit the time and Teacher Education, 14(1), 67-80. and energy that teachers have for Fullan, M. (1982).The meaning of teaching. It has also included a very large educational change. New York: investment in new teaching assistant Teachers College Press. roles in schools.The WA GiR strategy raises the question about whether a Hawley,W. & Valli, L. (1999).‘The more effective approach might be to essentials of effective professional place extra resources, if they are development: A new consensus’ in available, into freeing up expert teachers Darling-Hammond, Linda and Sykes, from time to time to work shoulder to Gary, Editors (1999) Teaching as the shoulder in the way that the GiR Learning Profession. Handbook of Policy developers have insisted. GiR legitimates and Practice, Jossey-Bass Publishers, the deprivatisation of teaching. Some San Francisco. teachers found this uncomfortable at Hodges, H. L. B. (1996). Using research first, but by the second year, when it had to inform practice in urban school: 10 become obvious that colleagues were key strategies for success. Educational gaining a great deal from the Policy, 10(2), 223-252. partnership, they usually came on board. Most teachers and principals in WA GiR Kennedy, M. (1999). Form and schools were in no doubt that the GiR substance in mathematics and science Strategy was giving them a greater professional development. NISE Brief opportunity to improve student learning 3 (2) Madison,WI: National Institute outcomes than any other strategy they for Science Education, University of had experienced. Wisconsin. Peterson, P.L., McArthy, S. J. & Elmore, R. References (1996). Learning form school restructuring. American Educational Carpenter, A., Ansell, E., Franke, M., Research Journal, 3(1), 119- 154. Fennema, E. & Weisbeck, L. (1993). A study of children’s problem solving Little, J. & McLaughlin. M. (Eds.) (1993). processes. Journal for Research in Teachers’ work: Individuals, colleagues Mathematics Education, 24, 428-441. and context. New York:Teachers College Press. Collarbone, P.(2004). Touching tomorrow: Remodelling in English Schools. Paper Louis, K. S., Kruse, S. & Marks, H. (1996). presented at Making Schools Better Schoolwide professional community. In Conference, August 2004, Melbourne. F. M. Newman (Ed.) Authentic

Using Data to Support Learning 71 ‘Getting it Right’ Symposium – Paper 3 Evaluation of the Getting it Right Literacy and Numeracy Strategy in Western Australian schools

Abstract Teachers. A review of the data collected from principals presents a positive This paper and the paper by Dr. account of an initiative that provides Lawrence Ingvarson are companion ongoing expert help to teachers in the pieces to Rosemary Cahill’s account of school, as they work ‘shoulder to the intentions of the Getting it Right shoulder’ in planning, and in classrooms. Literacy and Numeracy Strategy, and the model of professional development Principals’ perspectives on which the strategy is based. In these Marion Meiers papers, we present some findings from School principals were well positioned Australian Council for Educational Research the ACER evaluation of Getting it Right. to provide information about the We provide detailed results from the impact of the work of the Getting it Marion Meiers is a Senior Research Fellow at surveys of principals undertaken in Right Specialist Teachers in their school. ACER. Her research interests lie in the fields of 2003 and 2004, including findings of the We interviewed school principals on literacy and teacher professional development. use of data to improve planning.These She has directed the ACER Longitudinal Literacy three occasions in twenty schools, and and Numeracy Study (LLANS) since 1998. She results provide evidence of the impact gained a very positive picture of the has considerable experience in undertaking of the initiative, and evidence of an responses to Getting it Right. In one evaluations of teacher professional development, increase of the impact of the strategy school, during the evaluation team’s first and was the project director for the ACER over time.We identify key features of visit, the principal noted that teachers’ evaluation of the Western Australian Department the model of professional learning that of Education initiative, Getting it Right. confidence was ‘going through the roof’, underpin the strategy, and link this to and he reported that the value of Marion has lectured in teacher education at other research findings on effective having a Specialist Teacher had been Monash, Deakin and RMIT universities. She has professional development. contributed actively to the teaching of English and mentioned during performance literacy as a secondary teacher, consultant, The main purpose of the Australian management reviews.The Specialist curriculum writer and policy officer in the Council for Educational Research’s Teachers’ skills and knowledge, the Victorian Department of Education, and in evaluation was to provide the Western practicality of her advice and her ‘street leadership roles in teacher professional credibility’ had impacted on the school. development. She has played an active role in Australian Department of Education English and literacy teacher professional and Training with information about the The Specialist Teacher’s role of associations, and from 1993–1996 was the effectiveness of the Getting it Right providing in-class support was non- Executive Liaison Officer of the Australian Literacy and Numeracy Strategy (GiR- negotiable in the school. He noted that Literacy Federation. She is currently a member of LNS) in developing expertise relating to finding time for collaborative planning the National Council of the Australian Literacy the teaching of literacy and numeracy. had been difficult, particularly because Educators’ Association. She has an extensive of the number of teachers working in record of publications, including journal articles, The evaluation was focused on the chapters in books, and textbooks, in relation to impact of the GiR-LNS professional tandem pairs. the teaching and learning of English and literacy. development, on changes in school Several months later, in a second practices and on changes in classroom interview the same principal described teaching practices. the consolidation of the strategies Principals, Specialist Teachers and their initiated in connection with Getting it classroom teacher colleagues were Right in the previous year: surveyed in Term 4 2003 and again in The English policy is giving Term 4 2004. Other important direction to the whole school … evaluation information was collected GiR is focusing on writing as a from visits to a selected number of starting point … the Literacy Net schools, and from observations of the is being taken up … We’re not training sessions for the Specialist trying to cover too much …

Research Conference 2005 72 without GiR we wouldn’t have that the Getting it Right strategy was The most frequently cited reason for been able to implement the policy closely connected with the programs selecting classroom teachers to work …. Our Specialist teacher works in listed.There were strong links, for with the Specialist Teachers was the year class, providing ongoing, accessible example, between Getting it Right and level at which the teachers taught.The support. the Curriculum Improvement Program, needs of students was the next most The two surveys of principals, increasing over the course of a year. frequently listed criterion.The willingness of teachers to work with the Specialist conducted with a twelve 12-month In the 2003 survey, principals were Teacher was identified by a small interval, provided a range of detailed asked:‘What were the most important number of respondents.This question range of evidence about the impact of criteria used in selecting which was not asked in the 2004 survey. Getting it Right over time.The classroom teachers would work with descriptive results of the evaluation the Getting it Right Specialist Teacher?’ We were interested in the extent of questionnaires completed by principals The collaborative working relationships practical support schools provided to in 2003 and 2004 show that the between the Specialist Teacher and the Specialist Teachers, and so asked initiative was rated highly, and on some classroom teachers appears to be a principals about resources provided by dimensions, rated more highly in the critical factor in the effectiveness of the school to support the work of the second survey. Getting it Right, and it was interesting to Specialist Teacher.The frequencies investigate the reasons that principals shown in Table 3 show the levels of School context gave for selecting teachers to work with provision of resources. A suitable Principals were asked to identify the the Specialist Teacher.We found that a workspace was provided in almost all extent to which the Getting it Right range of reasons was cited, and cases, but phone, computer and email strategy was connected to other funded constructed a set of categories from an access were provided less frequently. In school programs. Getting it Right is examination of the responses. Space both surveys, the majority of principals intended to bring about improved was provided on the survey to list three reported that they had made learning opportunities for students, and criteria, although many principals chose timetabling arrangements to allow for coherence with other school only to list one or two.Table 2 shows collaborative planning. In view of the improvement programs is desirable. the categories and frequencies for each importance of collaborative planning in Table 1 shows the responses to this category, sorted according to the aspect the GiR strategy, it is interesting to note question. Overall, the responses indicate listed first, second and third. that 77% in 2003 and 83% of schools in

Table 1 Connections between GiR and other programs in 2003 and 2004

To a To a m i nor To a major To what extent is Getting it Right Not at all moderate extent extent connected to the following programs % extent % % %

a) the Curriculum Improvement Program? 1 5 26 68 n = 142/141 0 4 16 81 b) the Students at Educational Risk strategy? 0 7 8 85 n = 143/140 0 2 21 77 c) the Commonwealth Literacy and Numeracy Program? 11 7 24 59 n = 123/116 10 5 22 64 d) the Aboriginal Educational Operational Plan? 9 26 33 31 n = 132/131 9 1 31 42 e) other programs? (please specify) 6 6 17 71 n = 109/108 6 3 12 79

Using Data to Support Learning 73 Table 2 Criteria for selecting classroom colleagues in 2003

First Second Third Selection criteria for teachers to work criteria criteria criteria with Specialist Teachers listed % listed % listed % n = 139 n = 103 n = 61

Year level/s 45 15 8 Teachers’ willingness to work with ST (choice) 7 9 5 Teachers’ capacity for collaboration 4 5 5 Needs of students in classes 11 18 8 Teachers requesting to be involved 0 1 3 Attitude to the concept of GiR 0 0 0 Teachers’ interest in change in pedagogy 2 2 8 Other 3 11 10 Availability of common meeting time 1 Nil 3 Level of teachers’ needs 2 8 10 Small school, all involved 2 1 0 Misread question, described selection criteria for STs 24 31 39

Table 3 Resources provided by the school in 2003 and 2004

What resources has the school provided to No Yes support the work of the Specialist Teacher? % % n = 144/141

6 94 a) A suitable workspace for the Specialist Teacher 3 97 b) Phone, computer, and email access for the Specialist 21 79 Teacher 14 87 c) Timetabling and staffing arrangements to allow for the 8 92 collaborative planning time needed by the Specialist 4 96 Teacher and teacher colleagues d) Time, in addition to duties other than teaching time 23 77 (DOTT), for collaborative planning 17 83 e) A budget for the Specialist Teacher to purchase 20 80 resources for literacy/numeracy teaching 7 92

2004 provided time in addition to Setting targets had a key role in this process, duties other than teaching (DOTT) for supporting and working with the collaborative planning.The results The second section of the Specialist Teacher in the development of indicate that schools were providing questionnaire was designed to collect realistic and challenging targets, and slightly more practical support for information about setting targets for negotiating these targets with the the Specialist Teachers in 2004 than improving literacy and numeracy District Director. in 2003. outcomes within the school. Principals

Research Conference 2005 74 Table 4 Involvement in target setting reported that in 2003 the targets had not been modified, and 34% reported Who was involved in setting targets? that the targets had not been modified (tick as many boxes as apply) No Yes in 2004.Table 6 shows the frequencies % % n = 144/141 of responses to suggested reasons for modification. a) The Specialist Teacher 4 96 11 89 The most common reasons for modifying the targets in both 2003 and b) You, the principal, alone 83 17 2004 were the availability of further 87 14 information about student performance c) All members of the school leadership team 49 51 and further review of the data. Advice 55 45 from GiR team members prompted d) A literacy or numeracy working party 69 31 modification in 42% of schools and 43% 57 43 in 2004. Advice from District Office staff was almost never involved in either year. e) The whole staff 54 46 47 53 The impact of Getting f) District Office staff 95 5 it Right 92 8 g) Parents 89 11 The third section of the questionnaire 90 10 for principals included a series of questions designed to gather h) District Director 95 5 information about the principals’ 94 6 impressions of the impact of the Getting it Right strategy in the school. Table 4 shows that in both 2003 and Framework Outcomes to a moderate These responses provided insights into 2004 almost all of the schools involved or major extent in both years.The most the initial impact of the strategy, as they the Specialist Teacher in target setting. In frequently used sources of information refer to the end of the first or second almost half of the respondents’ schools were ‘other quality student achievement year of operation of the strategy in the all members of the school leadership data’ (95% to a moderate or major schools, and to the impact after another team or the whole staff were involved. extent in 2003, and 96% in 2004) and year had passed.The 2004 responses District Office staff and parents were the needs of students (94% (2003) and provided information about the longer- rarely involved. 94% (2004) to a moderate or major term impact of Getting it Right. extent).The 2004 data confirmed the The use of data to set targets to The first question in this section of the 2003 data, indicating that schools were improve learning is a key aspect of survey focused on a variety of drawing on a variety of information in Getting it Right, and so principals were outcomes in the school that had setting targets. asked about the data that had been resulted from the Getting it Right used in setting targets.The descriptive Principals were also asked about the strategy.These results are shown in results indicate that all the data sources extent to which schools modified the Table 7.The greatest impact reported suggested in the survey question had targets once they had been set, and the was in relation to teachers and teaching been used to a considerable extent. sources of information and advice practices. Over 90% of respondents in Western Australia Literacy and leading to modification. both the 2003 and 2004 surveys Numeracy Assessment (WALNA) data Sixty-three per cent of respondents reported that the Getting it Right was used to a moderate or major reported that the targets had been strategy was, to a moderate or major extent in 74% of schools in 2003 and in modified during 2003, and 66% extent, leading to more effective 86% of schools in 2004. Eighty-one per reported that they had been modified literacy/numeracy teaching practices, cent of schools used Curriculum during 2004.Thirty-seven per cent benefits to teachers, teachers being

Using Data to Support Learning 75 Table 5 Data used in setting targets in 2003 and 2004

To a To a m i nor To a major To what extent was each of the following Not at all moderate extent extent important in setting targets? % extent % % %

a) WALNA data 9 17 25 49 n = 139/137 4 10 20 66 b) other quality student achievement data 2 3 25 70 n = 131/129 2 2 25 71 c) Curriculum Framework learning outcomes for English 2 18 40 41 or mathematics 5 14 33 47 n = 131/133 d) the needs, experiences and interests of those students 1 6 25 69 most in need of help with literacy or numeracy 2 5 22 72 n = 137/135

Table 6 Modifying targets 2003 and 2004

No Yes What led to targets being modified? % %

a) More information about student performance 48 52 became available 52 48 n = 144/141 b) Advice was provided by Getting it Right team members 58 42 n = 144/141 57 43 c) Advice was provided by District Office staff 95 5 n = 144/141 94 6 d) Further review of student achievement data, 58 41 such as the WALNA data, or information gained from 45 55 the Literacy /Numeracy Net n = 144/141 more confident about teaching literacy reported that a coherent whole school Mathematics student outcomes of the or numeracy, and teachers being better literacy or numeracy plan had been Curriculum Framework also increased: at diagnosing students’ learning needs. implemented, compared with 73% in 73%, 2003, to 92%, 2004 (to a the previous year. moderate or major extent). It is interesting to note the increase in the extent to which principals reported Principals also reported an increase in The effective use of student performance that Getting it Right had impacted on the consistent use of the Literacy Net, data to improve planning had also several outcomes between 2003 and from 68% (to a moderate or major increased from 84% in 2003 to 91% in 2004.These results are indicative of the extent) in 2003 to 82% (to a moderate 2004 (to a moderate or major extent). longer-term impact of the initiative. or major extent) in 2004.The principals There was also an increase between reported teachers’ increased 2003 and 2004 in the extent to which it In 2004, 87% (to a moderate extent understanding of the English or was reported that more reflective use and to a major extent) of respondents

Research Conference 2005 76 Table 7 Getting it Right has led to these outcomes in 2003 and 2004

To a To a m i nor To a major To what extent has the Getting it Right strategy Not at all moderate extent extent led to the outcomes listed below? % extent % % %

a) more effective literacy/numeracy teaching practices 0 6 39 55 n = 144/139 0 7 33 60 b) The implementation of a coherent literacy/numeracy 5 22 38 35 plan for the whole school n = 142/138 2 11 45 42 c) Consistent use of the Literacy/Numeracy Net across 14 19 33 35 the school n = 141/139 8 11 35 47 d) improved learning outcomes for students at risk 0 12 40 48 n = 143/139 0 8 37 55 e) improved learning outcomes for all students 2 15 49 35 n = 142/139 1 14 40 45 f) more effective use of student performance data to plan 0 16 43 41 teaching and learning activities n = 143/138 1 7 38 53 g) Improved school results in WALNA testing 16 31 36 17 n = 124/128 7 21 40 32 h) more effective reporting to parents on students’ 11 38 39 12 improvement in literacy/numeracy skills n = 141/139 5 31 45 19 i) Teachers have a clearer understanding of the English 4 23 52 21 or Mathematics student outcomes of the Curriculum 1 8 47 45 Framework n = 143/139 j) the teachers have benefited from working with the 0 3 22 75 Getting it Right Specialist Teacher n = 143/139 0 4 15 81 k) teachers are more confident about teaching literacy 1 8 41 51 or numeracy n = 142/139 0 6 30 64 l) teachers are better at diagnosing students’ learning 1 11 48 40 needs n = 142/139 1 10 38 51 m) more reflective use of performance data to improve 3 19 46 32 planning at the whole school level n = 143/137 0 9 42 50 was being made of performance data to to 53% (to a moderate or major understanding of literacy and numeracy improve planning at the whole school extent) in 2003. curriculum and pedagogy, and how to level: 78%, 2003, to 92%, 2004, (to a link performance data to students’ Overall, these results indicate the moderate or major extent). needs.The results are shown in Table 8. principals’ impressions that Getting it Almost none of the principals In 2004 72% (to a moderate or major Right has led to a range of outcomes in responded using the ‘not at all’ option. extent) of principals reported that their schools. Responses to the other three options schools results in WALNA testing had Principals were asked about the impact (to a minor, moderate or major extent) improved across the school, compared of Getting it Right on their own were spread across the options.These

Using Data to Support Learning 77 Table 8 Impact on principals’ knowledge and understanding in 2003 and 2004

To a To a m i nor To a major To what extent has the work of Getting it Right Not at all moderate extent extent Specialist Teacher … % extent % % %

a) broadened your understanding of literacy or numeracy 4 24 48 24 curriculum and pedagogy? 1 16 50 33 n = 143/139 b) increased your knowledge of how to link your 5 32 39 25 school’s performance data to student needs in literacy 1 22 45 32 and numeracy n = 142/139

Table 9 Impact on teachers’ professional learning in 2003 and 2004

How would you rate their (professional development activities in which teachers at your school have GiR ST GiR ST GiR ST GiR ST participated over the past three years) relative much less less more much more impact, in terms of improving student learning impact impact impact impact outcomes, compared with the impact of teachers’ % % % % work with the Getting it Right Specialist Teacher?

n = 139/137 1 3 42 54 0 5 34 61

Table 10 GiR strategy meeting important school needs in 2003 and 2004

Is the Getting it Right strategy meeting any Yes No important needs in your school? % % n = 123/136

98 2 98 2 results indicate that principals were which teachers at their school had professional development activities, and reporting some level of impact on their participated over the past three years this had increased to 61% in 2004. knowledge and understanding, and that with the impact of their teachers’ work The surveys included a number of this had increased by the time of the with the Getting it Right Specialist open-ended questions, so that principals second survey. Teacher.The results shown in Table 9 could provide their own reasons and indicate a very strong trend to rating A question was designed to produce a explanations to further questions about involvement in Getting it Right as having general estimate of principals’ the impact of the Getting it Right more impact (42%, 2003, 34%, 2004) viewpoints on the impact of the strategy.These responses were and much more impact (54%, 2003, Getting it Right strategy on teachers’ examined and categorised into 61%, 2004).That is, more than half of professional learning. Principals were common responses. All responses were the respondents indicated in 2003 that asked to compare the impact of all the read by trained assessors, and scored involvement in Getting it Right has professional development activities in according the described categories. much more impact than other

Research Conference 2005 78 Table 11 GiR meeting important school needs

First Second Third need need listed need listed listed GiR meeting school needs n = 139/133 n = 123/120 n = 88/86 % % %

Identifying, diagnosing, monitoring and assisting students 18 15 13 at risk 12 8 4 Improving pedagogy in literacy/numeracy 26 15 15 17 8 11 Increasing teachers’ awareness of strategies to 4 15 6 improve learning 11 13 7 Improving teachers’ content knowledge 4 5 5 2 2 1 Enhancing literacy/numeracy learning 6 6 3 10 5 4 Improving assessment practices 2 5 5 1 5 4 Catering better for a range of student needs 4 6 3 2 6 4 Other 3 6 7 6 7 16 Whole-school planning for lit/num development 3 6 11 6 14 12 Improving data gathering and analysis 3 2 5 6 4 7 Focused use of school budget 1 1 1 0 1 2 Helping focus teacher learning (professional development) 8 4 6 5 3 9 Availability of ‘on-hand’ expert support; modelling 13 8 5 of lit/num strategies 11 7 8 Teachers’ engagement in collaborative planning and 6 6 16 sharing expertise 11 17 12 Improving home-school links 0 3 1 0 2 0

Principals were asked whether or not affirmative, they were then asked to list The responses shown in Table 11 the Getting it Right strategy was how Getting it Right had helped to indicate that, in 2003 and 2004, two meeting important needs in their meet these needs.Table 11 captures the school needs were most commonly school. As the results in Table 10 reasons they listed. Space was provided reported as having been met by the indicate, in both surveys almost all for three reasons to be listed. Getting it Right strategy.The first of (98%) agreed that this was the case. Respondents listed a varying number of these was the need to identify, needs, accounting for the different diagnose, monitor and assist students at If the principals responded in the numbers of responses. risk.The second need was related to

Using Data to Support Learning 79 Table 12 Better ways of meeting school needs 2003 and 2004

First way Second way Yes. Better ways of meeting school needs listed listed than GiR? n = 27/23 n = 12/5 % %

More opportunity for additional professional learning 19 8 for all staff 13 0 Need both GiR Literacy and Numeracy STs 4 8 4 0 More FTE 22 17 4 20 More differentiated resourcing 26 8 0 20 Even more support for GiR additional assistance 15 0 to schools 26 0 Linking to other agency support 0 17 Other 15 42 44 60 the improvement of pedagogy in professional learning and more time for staff. Support from the school literacy or numeracy teaching. Other the Specialist Teacher were mentioned administration, and school organisational needs that were identified as being met as better ways of meeting school needs. support were mentioned more than included increasing teachers’ awareness other factors. Principals were given the opportunity of strategies to improve learning and to note the factors that had facilitated Although reference to the Getting it the need for teachers to engage in or hindered the Getting it Right Right training program for teachers was collaborative planning and sharing strategy in their school. limited, the emphasis on the effectiveness of expertise. of the Specialist Teachers implies the The range of facilitating factors shown Principals were asked if they thought effectiveness of the training received by in Table 13 is of interest.They relate to that there were better ways of meeting the Specialist Teachers, as well as the the school context, the effectiveness of their school’s needs than the Getting it strength of their interpersonal skills and the Specialist Teacher, and to aspects of Right strategy. Most replied ‘no’ to this knowledge of literacy and numeracy educational change, such as teachers’ question (88%, n = 135) suggesting that content and pedagogy. receptiveness to change.The pattern of their impressions of the value of the responses is similar for 2003 and A number of factors were identified by strategy were positive. A small number 2004.While the frequencies for many the principals as having hindered the responded ‘yes’ – there were better categories are small, the range of implementation of the Getting it Right ways.The results in 2004 were very factors identified by principals provides strategy in their schools.The descriptive similar, with 84% (n = 135) replying useful insight into the operation of results are seen in Table 14, and are ‘No’.The responses of the small Getting it Right.The most frequently similar for 2003 and 2004.Two factors number who answered this question listed facilitating factor was the general were mentioned more often than the negatively were categorised, and the effectiveness of the particular Specialist others identified. Of all factors listed for results are shown in Table 12. Teacher in that school.The next most the first time, 28% related to lack of From the small number of respondents, frequently listed factor was the support time for collaboration. Staff resistance more opportunities for staff and cooperation of the whole school to working with the Specialist Teacher,

Research Conference 2005 80 Table 13 Factors that facilitated Getting it Right in the school in 2003 and 2004 or to the Getting it Right approach to providing additional assistance First factor accounted for 11% of factors listed for listed the first time in 2003, and this increased What has facilitated GiR in school? n = 131/134 to 26% in 2004. % Sustaining changes to Te a m approach (involving, for example, ST, SAER 2 coord., Admin.) 5 teaching practice Support and cooperation from whole staff 1 brought about by the 8 Getting it Right strategy Support by school Admin. 8 Finally, principals were asked about 10 plans that schools had made to sustain School organisational support, including time for 5 changes that may have brought about collaboration 13 by the Getting it Right strategy.The System-level support for GiR 3 range of plans reported was 5 categorised.The descriptive results for Additional time provided by school for work of GiR ST 2 the first and second plans listed are 4 shown in Table 15.The most interesting General effectiveness of the GiR ST 39 result is the increase in reports 22 between 2003 and 2004 that Other 3 collaborative planning and in-class 2 support will be continue: from 9% in 2003 to 24% in 2004.This suggests Use of GiR to focus whole school on improving 2 increasing recognition of the value of student outcomes 2 this key aspect of the Getting it Right Collaborative planning and review 6 strategy, affirming one of the strengths 7 of the model of professional learning GiR supports local needs & meets system requirements 1 that underpins Getting it Right. 4 GiR provides resource at point of teaching in 1 A positive view the classroom 1 Overall, the descriptive results of the The GiR training for STs 4 responses to the questionnaires 2 completed by principals in 2003 and Teachers receptiveness to change 2 2004 present a positive view of the 1 Getting it Right strategy.The results Teachers’ willingness to ask for help 0 provide insights into many features of 1 the strategy that principals connect to Data-based incentive from need for school to improve 8 improved outcomes in their schools. student outcomes 10 Coherence with other school initiatives 1 0 Observed effectiveness of strategies promoted by GiR 2 0 ST from within school 1 1

Using Data to Support Learning 81 Table 14 Factors that hindered Getting it Right in the school

First factor listed What has hindered GiR? n = 131/122 %

Lack of direction, poor administration of GiR 2 (at system level) 0 Insufficient funds 7 4 Lack of time (eg, for collaboration) 28 25 Short timeline (only 2 years) 2 1 Timetabling constraints 2 0 Staff turnover 6 11 Change of ST 3 4 Other 14 19 Sharing GiR ST with another school 2 1 Inappropriate ST 2 5 Staff resistance 11 26 Difficult to change some teachers’ practice 5 3 Staff not focused on students’ learning needs 0 0 Principal needed more briefing at commencement 3 0 Negative effects of GiR program title 10 0 Staff misunderstanding of GiR ST role 8 1 Student transience 5 1 Unwillingness to use DOTT for GiR 0 1 Not whole school (K-7) in focus 1 0

Research Conference 2005 82 Table 15 Plans for sustaining changes in 2003 and 2004

First plan Second plan listed listed Plans for sustaining GiR changes? n = 137/136 n = 83/87 % %

Maintain ST role through other funding 7 1 (eg, CLNP, or further GiR funding) 6 2 Providing school resources/funding 7 11 4 6 School will continue to fund ST position 5 4 4 7 Developing whole school literacy/numeracy plan 15 10 13 9 Continue to treat GiR as integral part of 4 2 teachers’ learning 10 1 Ongoing direct monitoring of student outcomes in 1 6 all classes 2 3 Continue collaborative planning and in-class support 9 10 24 14 Other 12 15 4 24 Extend collaborative planning to whole school 4 5 7 8 Embed GiR changes in school teaching and/or 20 17 assessment practices 16 12 Increase the number of teachers involved 1 1 2 7 Implement GiR as designed at system level 4 1 0 0 Introduce timetable changes 2 0 0 0 Provide more PD for teachers 9 18 2 0 Share good practice within the school 2 0 (eg staff meetings, visiting other teachers’ classrooms) 6 1

Using Data to Support Learning 83 Data and school improvement – A school perspective

Over the past decade,Victorian achievement regardless of background government schools have become or location.The Blueprint emphasises accustomed to the use of data as a the need for an accountability and means of evaluating school improvement focus that is responsive to performance and identifying potential school needs and focuses on improving areas for improvement. student outcomes and now the charter In many instances however, the data has process is evolving into a four-year been relatively unsophisticated and the strategic planning process. Wayne Craig data has not been readily available to or Another of the Blueprint Flagship Department of Education and Training, useable by teachers, and students rarely Strategies is the implementation of a Victoria had access to this information. performance and development culture The Victorian Education Department’s across all schools.The core of this Wayne Craig was appointed as Regional Director Accountability and Improvement performance and development culture of Education for Melbourne’s Northern framework, underpinned by a three- is the provision of multiple sources of Metropolitan Region early in 2005. year school charter, required schools to feedback to teachers so that teachers Prior to this appointment Wayne was principal at monitor, evaluate and report on can constantly monitor and improve Box Hill Senior Secondary College. Under performance data across a range of their performance. Wayne’s leadership Box Hill Senior developed a areas. Data sets include: significant reputation as an innovative school In an attempt to drive improved providing diverse programs and pathways. • Like School Group – a measure of outcomes, schools such as Box Hill Box Hill Senior played a leading role in the the socioeconomic status of the Senior Secondary College have integration of vocational education in senior school population developed their own data gathering schooling and established unique sporting •AIM* results processes to inform both teachers and programs in basketball, football and tennis. • VCE** Results against ‘Like Schools’ students about performance. In 2005, Box Hill Senior established the Middle Years Tennis School to cater for 150 students • Vocational provision and Box Hill Senior Secondary College – a seeking to combine middle years education with achievement Years 10 to 12 school of 600 students a strong sporting program. • Real Retention – is unusual in the Victorian context. As one of two ‘stand-alone’ senior colleges, Box Hill also developed innovative approaches to • Student Absence the use of ICT as both a teaching and it has no feeder schools and relies on management tool. • Parent Opinion ‘word of mouth’ recommendations to Wayne commenced his working life as a research • Staff Opinion secure its enrolment. A third of Box Hill officer for a major retail company and later taught The Framework was more a students travel three to four hours per mathematics and science at a number of technical day to and from school and the school schools in Melbourne’s east and south east. compliance mechanism than an improvement tool.The data has developed an enviable reputation underpinning the Framework assisted in for innovative programs across a broad identifying issues, but was relatively range of programs. In 2005, as part of unsophisticated and is not readily usable the Blueprint’s Leading Schools Fund, the in schools as the basis for improving school established the Middle Years student achievement. Tennis School.The Middle School has 55 students from Years 5 to 9 In 2003 Victorian Education Minister combining tennis with schoolwork and Lynne Kosky launched a Blueprint for this enrolment will increase to 150 Government Schools with a moral students in 2006. purpose to improve student

*AIM – Achievement Improvement Monitor: a statewide literacy and numeracy test administered at Years 3, 5 and 7.

** VCE – The Victorian Certificate of Education

Research Conference 2005 84 Box Hill Senior uses a broad range of Students also complete a number of using scanning technology. Students electronic tools to gather information tasks on Websurvey (web-based evaluate their approaches to learning, on student attitudes, aspirations, software developed by the school) indicate whether they have achieved at expectations and achievement. Some of which include: the level they expected, factors contributing to their success and steps this information is fed back to students • A learning styles test that is they can take to lift their performance. as part of a mentoring process and intended to raise student awareness The reports are a key component of some goes to teachers to assist in and reinforces the need for teachers the Student Progress Conferences that improving their performance. to consider different learning styles are held with parents and students. There are several crucial preconditions as they plan teaching programs; Success factors and improvement for the implementation of the Box Hill • A self-assessment of approaches to options are also forwarded to each of approach: learning.This includes questions like the student’s teachers. ‘How do you rate yourself on • Clarity and unity of purpose commitment to study? Very Teachers at Box Hill have electronic • Unless the school has a strong committed? Committed? Not access to all the data regarding students sense of direction, data is more committed? How much time do you and are encouraged to reflect on this likely to confuse than clarify or expect to spend on homework information as part of the school’s assist. each day?’; performance management process. • A strong emphasis on use of data • Questions related to future Two other significant data sets are also • Using data that is potentially employment such as,‘Being highly used to assist staff to improve the confronting for teachers is a paid is … Very important? quality of their teaching. gradual process. Important? Not important?; The first is an analysis of individual • The use of ICT for collection and • Questions on achievement student performance at Year 12 in the analysis of data expectations in each subject; VCE.The analysis takes into account • ICT is essential to collect, analyse • Questions related to involvement in three key factors: and distribute data. part-time work, sport and the arts • Ability as measured by the General (outside school); and • High levels of trust Achievement Test that is • Teachers and students must be • Questions related to the student’s administered by the Assessment confident that data will be used access to ICT at home. Authority during Year 12; appropriately. Box Hill Senior has also developed a • Gender, as girls generally In its work with students the school web-based attendance and progress outperform boys; and uses a well-developed mentoring reporting system that monitors student • Year level, as students taking one program that assists students to take attendance by the half-hour and Year 12 subject at Year 11 generally responsibility for their performance.This provides a monthly progress report in outperform Year 12 students taking program supports students in reflecting each subject.This progress report also five subjects. the match between achievement levels includes a grade point average on student performance. Other variables such as attendance, age and aspirations and expectations. Much and student family occupation can also of the data at the heart of the Individual student mentoring sessions be included in the analysis. mentoring process is also used to are conducted with a teacher several support improved teaching. times per year. The student is Teachers have information on the encouraged to reflect on levels of performance of each student in their Data collection and use achievement compared to expectations classes and are able to look at how and aspirations and what strategies those students performed in other To get a picture of student ability levels might be used to lift achievement. classes.The data indicates whether all students sit ACER Literacy and students performed at, above or below Numeracy tests on entry to the school. At the end of each semester, students what would have been expected on complete their own report on progress

Using Data to Support Learning 85 the basis of ability, gender and year spoke about how they always knew level. when they had a really bad lesson or really good lesson but the surveys The data lays a rich foundation for provided detailed information across a discussion in performance management range of performance areas. sessions, in learning area meetings and between groups of teachers.These Thirdly, student opinion gave teachers discussions have encouraged teachers an insight on aspects of their to modify their teaching practice and performance that could be improved. have led to changes in class organisation Most teachers looked at the survey in some instances. results and identified two or three indicators that were lower than others. Another key data set for Box Hill These aspects became the teachers is student opinion on the improvement focus for the next twelve quality of teaching. Using scanning months. technology, all staff members are expected to survey classes during the Finally, the survey data added another year. Students are asked to respond to layer of rich discussion to the schools questions such as: performance management process and assisted the school administration to • My teacher is an expert in this identify professional development needs subject. for the staff as a whole. • My teacher is well prepared. The strategic use of data at Box Hill • My teacher checks understanding. Senior Secondary College supports • In this class, students are treated as improved student achievement and a adults. similar, but broader, approach is now • In this class, high standards of work being developed to serve government are expected, schools in Melbourne’s Northern Implementing this approach required a Metropolitan Region. A regional data significant leap of faith on the part of service is being developed to provide teachers.There was some concern, teachers and students with a range of particularly from experienced teachers, information on student achievement, as to whether students would actually ability, aspirations and expectations. assess teacher performance fairly. At the The data service will support a range of same time, some students expressed other school improvement initiatives to doubt as to whether teachers would be rolled out over the next two years. respond to teacher opinion. There were four clear outcomes from the student opinion surveys. Firstly, students treated the surveys seriously. Secondly, teachers were highly regarded by students and this had a significant impact on morale. Most teachers had never received any formal feedback from students in their classes. During performance review many teachers

Research Conference 2005 86 Using the evidence of student achievement for improvements at individual, class and school level

Abstract analyses of large data sets often provide little guidance to the individual teacher, Techniques using student work as direct who must be concerned with the and visible evidence of achievement, of complex particularity of individual the repertoires of practice of students students and groups of students. and teachers, provide a powerful opportunity for teachers and schools Of course, these statements about seeking to improve the learning of the teachers and students rest on assumed students they have.This is a purpose archetypes of: Reg Allen different from that of the analyst • learning as including (but not Tasmania Qualifications Authority modelling patterns in large data sets of restricted to) broad and deep test scores or the concerns with understanding Dr Reg Allen is the Chief Executive Officer of the complex causality found in small-n • the teacher as professional, inquiring Tasmanian Qualifications Authority. He has 30 studies and the methods consequently and reflecting on practice to achieve years of experience as a teacher and subject differ. Critical elements of techniques master, board member and teacher more learning by more students for using student work include the representative, consultant, deputy director and • the student as a whole person, living value of seeking a student, rather than director, in the diverse education contexts of in and across a time and place and Australia, England, and the USA. He has acted as subject or teacher, perspective, open to embedded in cultures1. expert education advisor in Australia and both the official – what is recognised as America, including being a leading member of the part of school – and the unofficial – Such archetypes push into the Framework Research Advisory Group for unrecognised factors that underpin background those data techniques that Queensland's New Basics in 2001-2004. students’ practices. are more suited to the notions of His principal previous role has been as Deputy teacher as technician, following codified Director of the Queensland Board of Senior This paper describes the nature, use instructions in the use of some test Secondary School Studies where he played a key and importance of some powerful scores to focus coaching effort for gains role in the design and implementation of techniques through which teachers can in terms of a uni-dimensional latent Queensland's Student Education Profile for senior use data to improve student learning. school students, including the integration of trait.The techniques explored in this vocational education in post-compulsory school For a teacher, the central purpose of paper can help teachers to identify education. He has produced over 40 publications analysing data is to improve the learning teacher and student repertoires of across diverse education specialities, including of one or more particular students.That 2 national studies and the development of policies practice . Luke et al. (2005) describe the for the Australasian Curriculum Assessment and is, the individual teacher and the school hypothesis that effective teaching Certification Authorities (ACACA). take the students who come to them involves ‘weaving’ – shifting kinds and and seek to improve the learning of levels of knowledge as needed. In these those students.This purpose is different terms, teachers draw on repertoires of from that of the sociologist seeking to practice as they work with students, understand patterns of participation, or weaving these together. Students draw that of the policy analyst seeking to on repertoires of practice, some of understand the impact, if any, of policy which they bring with them from settings.The possibly powerful outside the school and others which generalisations about a handful of key are learned, developed or modified variables produced by nomothetic through their experience of school.

1See Gutierrez and Rogoff (2003).

2‘Repertoires of practice’ is a helpful term from cultural sociology now being increasingly used in discussions of pedagogy. Its broader meaning refers to the idea that regularities in our performances or actions (language, gestures, rituals, routines, rhetorics) can be understood in terms of ‘toolkits’, set of models, from which we select and combine (more-or-less unreflectively)(Sheffy, 1997).

Using Data to Support Learning 87 Indeed, learning itself can usefully be really is identical and that each unit is a understanding what students bring with seen as the development by the separate, isolated possible world that is them and their knowledge, student of particular repertoires of unaffected by what happens to the understanding and acceptance of ‘doing practice3. other units’ (Brady & Seawright, 2004). school’ – Bourdieu’s habitus and Bernstein’s recontextualisation – their The teacher’s concern with improving Secondly, while the methods of what de ‘feel for the game’ (Cooper & Dunne, the learning of particular students Meur and Rihoux (2002) call quali- 1998), for making sense of students’ means some distinctive characteristics quantitative comparative analysis4 offer responses to various types of for data gathering and analysis. the researcher an opportunity to mathematics assessments. understand patterns of complex First, the methods and results of what causality in small-n populations5, they Data about what actually happens in Ragin (1997) calls the variable-oriented present significant technical and school6 can be relatively direct or researcher are not useful – there are operational challenges to the teacher 7 too few students, too many facets to indirect . Direct data includes student seeking to understand and improve 8 consider and the students interact with work – potentially the most valuable what is happening in a particular each other, with the teacher and with outer sign of internal activity – and classroom. 9 their wider socio-cultural contexts. structured classroom observations . Teachers often seem intuitively aware Thirdly, students bring their complete More indirect data includes the that some fundamental assumptions selves with them when they interact evidence from student and teacher required by statistical studies seeking to with school – a ‘dunno’ can be laconic, reflections (through conversations and find effective educational treatments resentful, defensive, uninterested, surveys) and test results. Student and (‘taking this action causes that effect’) diversionary or intentionally misleading teacher comments and reflections are are not sound. Brady and Seawright, rather than a simple attestation of a more indirect in that they are (2004) discuss this in terms of the deficit of not knowing (not statements about what people think is potential failure of a key assumption remembering, never encountered, didn’t happening as mediated through their required by controlled or randomised realise) easily remedied by an instruction ways of seeing the world10.Teacher experiments, one which implies that session. Cooper and Dunne (1999) have statements about enacted practices, in ‘each supposedly identical treatment shown from UK data the importance of particular, often seem strongly coloured

3Especially so when we seek learning that is powerful, transferable and oriented towards meeting a future of unpredictable demands and opportunities.

4See http://smalln.spri.ucl.ac.be/. Ragin (1987) uses the term Qualitative Comparative Analysis (QCA).The methods in QCA have, however, a quantitative aspect in the inclusion of Boolean minimisation algorithms originally developed in digital electronics.

5Katz et al. (2005) present an interesting comparison of the results of fuzzy set QCA (Cronqvist, 2003) and regression analysis in an analysis of causality in economic development in Spanish America.

6The term ‘enacted curriculum’ is helpful, but can be misleading. Studies of the ‘enacted curriculum’ seem to focus more on what teachers say about what they do, more than on what happens or how students experience it. See for example http://www.secsupport.org/overview.htm for materials and Porter (2004) for a discussion of differences between the intended (standards), enacted (teacher priorities) and assessed (tests) curriculum.

7In this context,‘direct’ and ‘indirect’ are similar but not identical to the distinction historians draw between ‘primary’ and ‘secondary’ sources.

8‘Student work’ is used here in the most general sense, not restricted to culminating performances, formal assessment or testing.

9See, for example, the coding scheme used in the Queensland School Reform Longitudinal Study (Education Queensland, 2001).The need for trained observers and multiple observations over time (a single lesson does not sample the complexity of practices that could be part of a teacher’s repertoire) make this type of evidence less accessible on a regular basis.

10Student comments can be very helpful, nonetheless. See Cooper and Dunne (1999) for examples of the insights that are only readily apparent with the use of student comments – students who gave the ‘wrong’ answer to ‘realistic’ problems sometimes did so because they took the ‘realistic’ setting of a task at face value. Student comments can also remind us of the gap between our intentions and their interpretations.The author once sought feedback about the Queensland Year 12 Writing Task from a small group of students selected from the highest achievers – a group that included students who write for pleasure and profit.The Writing Task was designed to provide students with opportunities to do their very best writing, to showcase their writing skills in their preferred genre whatever that might be.They told us, however, that this was ‘school’ and that school doesn’t want your best writing, only the writing that fits its expectations.

Research Conference 2005 88 by their intentions and their feelings In summary, for teachers seeking to In the late 1990s, the author led the about what ought to be happening. improve student learning, as a data development and piloting of a resource source, student work is more easily for Queensland senior secondary Tests provide teachers with indirect accessible than structured classroom schools seeking to review their evidence about what is happening – an observations, it provides more direct, practices (Allen & Bell, 1999).This estimate of the ability (and propensity) involved a skilled facilitator using a considered to underpin particular visible and complete evidence of both structured process centred on student knowledge and skills – an indirect student and teacher repertoires of 11 work. As well as the critical elements indication about aspects of what has practice than do test scores and it listed above, a particular characteristic happened.Tests are, of course, coloured supports the types of analysis needed of the process was that it took a by their sample nature and by the by the classroom and school situation student-centred focus rather than varying ways students choose (or don’t of small numbers and complex causality. subject focus.That is, it sought to use a choose) to respond to them. It may There is growing interest in the use of set of collections of the work of seem so obvious and simple that tests student work to improve learning – individual students12 as the direct can provide diagnostic evidence – so that see, for example, www.lasw.org, evidence for asking questions about the a teacher knows what needs to be done Cushman (1996), Little et al. (2003). enacted values, priorities and practices for the student to learn more. Practice is Critical elements include the need: in the set of subjects experienced by a more complex. For example, as part of student.To make the task as the reform of its lycées, France had • for expert facilitation and carefully straightforward as possible, the sets of developed in the 1990s a national designed protocols – teachers can student work were, for the first stage of program of testing specifically designed to find the task of looking at the work review, chosen to be those of students be diagnostic, for teachers to adapt their itself both difficult – they want to who were generally successful – the pedagogy to meet the needs of their (re-)mark it – and troublingly – students who were not ‘resistant’ to the students (see, for example, http://artic.ac- there are notions of territory, of enacted culture of the school, who besancon.fr/espagnol/pages/evalsec.htm). privacy and perhaps worries about knew how to ‘play the game’.The In mathematics, for example, the teacher being judged and found wanting techniques encouraged by the facilitator codes student responses, and tables in • to avoid ‘deficit’ models – the and the protocols could be seen, the teacher’s guide suggest that various students and/or the teacher ‘didn’t roughly, in terms of the ‘hermeneutic combinations of successes and failures get it’ (Little et al., 2003) – but to circle’13 or, in simpler terms, as the sorts are associated with different needs and look for the attitudes, values, of interpretations that historians and proposed remediations. Not surprisingly, priorities and ways of doing things anthropologists practise when teachers (for example, http://www.ac- that are evidenced in the work (and documents are the only evidence they versailles.fr/pedagogi/anglais/joinin/miseen the presence or absence of teacher have for understanding some social placeremediation2nde.htm ) find it not so comments and other signs) practices14. An initial focus on the simple – there’s a lot of work scoring and • to identify and see through and surface, obvious features of the then coding responses, there are students beyond what is being taken for evidence, including any evidence of with widely varying backgrounds, widely granted by the teacher and teacher comments, codes and signs, was varying responses to the test situation followed with closer examination of and other familiar problems. the student.

11Gutierrez and Rogoff (2003) emphasise the importance of a focus on activities rather than individual traits.

12Queensland had a system of externally moderated school-based assessment for high-stakes subjects. Students typically took six subjects each studied for two years.The moderation process required compilation of a folio (collection) of a student’s work providing the evidence that supported the final decision about a level of achievement (criteria and standards based) in a subject.Thus, a collection of a single student’s work for this project was a set of five or six folios each containing a variety of tasks.

13A reading of the texts in the light of pre-judgements is subjected to critical examination in the light of the texts.

14There are cuneiform texts from Sumerian schoolrooms that give us some (limited) picture of their enacted values, priorities and practices. See Sylvan (2004) for an account of methodological issues involved in using these sorts of perspectives.

Using Data to Support Learning 89 what activities seemed to be • a school’s view that there was (1999) exploration of the varied emphasised and what de-emphasised in effective use of technology across reasons students had for giving practice – using the assumption that the the curriculum was not supported ‘incorrect’ responses to mathematics students whose work was being looked by sets of student work that test items. at would seek to maximise their return showed, for example, that This technique looks at all the evidence for effort and thus enacted attitudes, computers were being used mostly in the artefacts (the student work), values and priorities could be inferred as electronic typewriters including teacher and student from the evidence. • across one school’s curriculum, the marginalia, the ‘unofficial’ as well as the Taking a student focus rather than a enacted variety and complexity of official.There’s much potential value in subject focus was often a particular ‘problem-solving’ was less than taking as complete a view as possible. A challenge for participating teachers, as individual subject areas believed it look at the marginalia of students’ was seeing the implicit, enacted priorities to be – the sort of result that only responses to some items on a rather than the intended or designed. comes readily through teachers Queensland Core Skills Test showed, for taking a whole student rather than example, that there were many At the end of the process, teachers’ an individual subject perspective students who have in effect learned findings included that: • across the curriculum of individual that anything that looks at all • some generally desired behaviours students there was a narrower ‘mathematical’ is not for them; (for example, clear and accurate range of extended writing than they regardless of how carefully an item has written expression, clear expected been constructed to provide a friendly mathematical argument) were in • with the ideas from this sort of and easy entry to the task. A study of practice review they could draw useful Queensland Year 10 Mathematics folios rewarded/encouraged/required in interpretations of the patterns in included a wonderful example centred only one subject – with the QCS16 test score data they had. on the problematic nature of so-called consequences that the behaviour ‘problems’ but illustrating the was exhibited within but not These findings are probably not recontextualising, the demands on outside that classification; the surprising, of course.They illustrate, knowledge, skills, attitudes and values of knowledge and skills did not transfer however, the potential of this sort of ‘doing school’. Here was a ‘problem’ from one situation to another technique for developing teachers’ (actually not a problem – the answer understanding of the impact of • in some schools, there appeared to was obvious on inspection).The full text practices rather than intentions and of be greater reward for effort for (student response and teacher the importance of seeking to careful presentation than for serious marginalia) strongly suggested the understand school from a student intellectual rigour – these schools following scenario: the student wrote perspective. Once teachers were often started the process because down the obvious answer, remembered familiar with the practice and of concerns that their students that ‘working’ had to be shown and techniques of this sort of study, there performed relatively less well in constructed some semblance of it (it was real additional value in a successful higher order thinking tasks than they didn’t work); the teacher attempted to follow-up review using the work of expected follow the working, couldn’t, gave up students who did not experience • what was declared to be the official and then marked the response as success – a more challenging task (less correct, giving full credit. intention of an assessment task was evidence, more possible interpretations) not necessarily what was but potentially very fruitful, as These approaches are, especially in the 15 rewarded/favoured in practice demonstrated by Cooper & Dunne’s shorter term, essentially ameliorative

15In audiences in different countries, the author has found general recognition that a university assignment that is declared as ‘wanting your own opinion’ should not be taken simply at face value.

16Schools were provided with comprehensive and detailed score analyses of their year 12 students performances in the Queensland Core Skills Test – a test of generic skills, using multiple choice, short response and extended writing formats to assess students’ achievements in terms of common curriculum elements, such as reading, writing, evaluating, synthesising, judging, inferring, deducing.

Research Conference 2005 90 rather than fundamentally reconstitutive. louder than the text.To improve De Meur, G., & Rihoux, B. (2002). Teachers need data gathering and student learning, however, the direct L’Analyse Quali-Quantitative Comparée analysis techniques that work in the and comprehensive evidence of (AQQC-QCA). Louvain-La-Neuve: here-and-now, providing some ways to achievement in the point-at-able form Academia Bruylant. improve matters for their students. in which it appears in student work Education Queensland. (2001). Avoiding not only deficit models but also provides a data source that can be used Queensland School Reform Longitudinal both an emphasis on intentions and too to generate rich analyses. Study: Final Report,Volume 1. Brisbane, restrictive a focus17, the use of student Queensland: Education Queensland. work provides a practicable basis for References identifying key aspects of what is and Gutiérrez, K., & Rogoff, B. (2003). Reviewing what might be, at an individual, class and Allen, R., & Bell, E. (1999). Cultural Ways of Learning: Individual Senior school level. Starting at the school level . Brisbane: Queensland Board of Traits or Repertoires of Practice. builds the skills to look at what is rather Senior Secondary School Studies. Educational Researcher, 32(5), 19-25. than what is intended and the consensus Brady, H., & Seawright, J. (2004, 28 Katz, A.,Vom Hau, M., & Mahoney, J. building involved in working at this level August). Framing social enquiry: from (2005). Explaining the great reversal in supports the individual teacher in models of causation to statistically based Spanish America. Sociological Methods looking at student repertoires of practice causal inference. Paper presented at & Research, 33(4), 539-573. at the class and the individual level. At the 2004 Annual Meeting of the the same time, teachers develop their American Political Science Association. Little,W., Gearhart, M., Curry, M., & understanding of the range of teacher Kafka, J. (2003). Looking at student Cooper, B., & Dunne, M. (1998, 6-11 repertoires of practice18. work for teacher learning, teacher September). Social class, gender, equity community, and school reform. Phi One of the practical challenges of using and national curriculum tests in Delta Kappan, 85(3), 185-192. student work noted by Little et al. mathematics. Paper presented at the (2003) is the tendency for teachers to Mathematics Education and Society: Luke, A., Freebody, P.,Shun, L., & select culminating,‘show’ pieces. It can be An International Conference, Gopinathan, S. (2005). Towards salutary to collect the full set of student University of Nottingham. research-based innovation and reform: work completed by a student across all Singapore schooling in transition. Cooper, B., & Dunne, M. (1999). classes for, say, a four-week period – the (forthcoming in Asia Pacific Journal of Assessing children's mathematical author has observed cases where there Education, retrieved 8 June, 2005, from knowledge: Social class, sex and was little if any artefactual evidence of http://crpp.nie.edu.sg/course/view.php?i problem-solving. Open University Press. any worthwhile activity by students and d=262) involvement by teachers. Some will say, Cronqvist, L. (2003). Using multi-value Porter, A. (2004). Curriculum assessment. of course, that the really important logic synthesis in social science. Paper Retrieved 8 June, 2005, from learning is necessarily not19 evidenced by presented at the 2nd General http://www.secsupport.org/pdf/curricas anything anyone can produce – the Conference of the European sess.pdf classroom is a private space. Consortium for Political Research, Marburg, Germany. Ragin, C. (1987). The comparative Such claims are not refuted by method. University of California Press. dismissing them as defensiveness, an Cushman, K. (1996). Looking unwillingness to be accountable – collaboratively at student work: an Ragin, C. (1997).Turning the tables: there’s a scholarly tradition that the essential toolkit. Horace, 13(2), 1-20. How case-oriented research silences, the gaps, the interstices speak challenges variable-oriented research. Comparative Social Research, 16, 27-42.

17Strong classifications serve as a constraint on teachers, of course, as well as students.

18Including the filters through which they see and understand student responses – there’s a lot of taken-for-granted understanding of what it is to ‘do school’, there can be wide differences between what we say and what students hear.

19This is the strong version of this claim – a weaker version is ‘not necessarily’.

Using Data to Support Learning 91 Sheffy, R. (1997). Models and Habituses: Problems in the idea of cultural repertoires. Canadian Review of Comparative Literature / Revue Canadienne de Littérature Comparée, 24(1), 35-47. Sylvan, D. (2004). In the field: An alternative pedagogy for qualitative methods courses. Paper presented at the 2004 Annual Meeting of the American Political Science Association.

Research Conference 2005 92 Using HSC data to give principals leverage

‘Without data, I’m just another Visscher and Coe (2002) develop a person with an opinion’ heuristic for the interpretation of (Barry McGaw, ACER Research School Performance Feedback Systems Conference 2002) (SPFS) which looks at the system in What makes the difference in student terms of its: achievement? What elements among • design process ‘what makes the difference’ can a school • features (the validity of the input principal influence? How does the information, the accessibility of the principal influence these for the better? John DeCourcy data, whether the output is standard How do teachers best take account of St Andrews College, New South Wales or tailored to the school, the extent the pedagogical information available in of support for use of the system, etc.) data? How can we use the data available Dr John DeCourcy is College Principal of St to address these questions? • implementation process (the use of Andrews College, a Catholic dual-campus tailored user training, promotion of secondary school of 1200 students in Western The topic of this conference is ‘Using user participation, the monitoring of Sydney. His academic background is in theoretical data to improve student learning’. Data structural chemistry, educational measurement implementation, etc.) will act to improve student learning and developmental psychology. Five years ago, • within-school organisational features broadly across a school only if the data seeking a better way of understanding and using (the school’s and teachers’ capacity achievement data from his own school, he began become the principal’s agenda, and the to deal with innovation, the extent the project with the Catholic Education data will become the principal’s agenda to which the system requires Commission (NSW) providing a multilevel only if s/he sees the data as a useful resources, the extent to which new analysis of HSC data described in this paper, and lever to achieve worthwhile outcomes. has gone on to provide this analysis to all skills must be developed, etc.). Catholic secondary schools in the State. John is So the question becomes:‘How do you also currently president of the Australian College get data to a form where it will provide Each of these four aspects of the of Educators in his part of Sydney. the principal with leverage s/he can use system bear upon the fifth and critical and trust?’This paper draws on a five- aspect: the usage of the SPFS (whether year project involving over 120 it will be for instrumental, conceptual, secondary schools in New South Wales symbolic, or strategic use). The choice to outline what has been learned about of dominant usage pattern then affects the most effective ways to engage the sixth characteristic of the system, its principals and teachers with a particular intended and unintended effects. set of achievement data.The learnings There are many examples of SPFS from this project may well have where failure to take adequate notice applicability in other settings. of the features, implementation, or Principals and teachers can be reluctant organisational characteristics of the to engage with data because their system leads to utterly inappropriate professional intuition leads them to be usage of the system, and undesirable defensive about data analysis which unintended effects (Amrein & Berliner, purports to attribute large differences 2003; Braun and Mislevy, 2005).The in achievement to schools or teachers, intention of the project described in where the difference actually lies in this paper is to produce a usage pattern factors beyond their control (O’Day, that is instrumental: the data becomes 2002).What is needed is a form of an instrument in the principal’s and analysis that separates out the factors teachers’ hands to monitor and improve that do lie within the control of pedagogy and students’ performance. teachers, and gives a valid and easily As an instrument, the data is presented interpreted analysis of these factors. in a way that gives the principal

Using Data to Support Learning 93 leverage to support and effect the confidence intervals of the year; the analysis of these results from innovation that has a positive effect on measurement are +/– 9%! the project is available for downloading student achievement. before the start of the following school The product of the project as supplied year, and the report on the overall HSC to schools is an electronic file, consisting is available from June each year.The Context of five parts: project is supported by a web site For each of the last five years, the • The Primary Analysis of each subject, (http://stage.cecnsw.catholic.edu.au/hsca/) project has been a cooperative showing a comparative learning gain which has both a secure section where agreement, conducted under the (with confidence intervals) of the schools and systems can obtain their auspices of the Catholic Education mean result achieved in the subject own data, and an open section Commission (CECNSW) between the with that achieved by similar containing the Manual (DeCourcy, (now) 125 Catholic secondary schools students in other schools 2005a) for the project, and a series of of New South Wales to pool the • The Secondary Analysis of each annotated PowerPoint files which can results of their 14,000+ students in the be used by principals and others in Higher School Certificate (HSC) subject, showing a comparison of the mean result achieved in this professional development activities with examinations to enable a multilevel staff. Each year, a number of seminars analysis (Goldstein, 1995; Goldstein, subject with firstly state average and secondly the average obtained in all on the use of the analysis are Rasbash, Plewis, Draper, Browne,Yang, conducted under the auspices of Woodhouse, & Healy, 1998) to be of their other subjects by the students in this subject CECNSW for those whose role it is to conducted across both the aggregate introduce the analysis to staff. results and each of the 80,000+ • The Trends Analysis for each subject, results in individual subjects.The showing the three measures from Initially, most principals met this project statistical methodology of the analysis the primary and secondary analyses with a healthy degree of scepticism and is described in the Appendix to over the last six years, and suspicion; over the five years of the this paper. showing any second-order effects project, this has changed for most to for each year insight and enthusiasm as they have The central concept of the project is seen the connection between the ‘comparative learning gain’: what is the • The School Database containing presentation of the data and their comparison in the performance of the both the input data and the results knowledge of their schools. students in this subject in this school of the analysis for each student in with that of similar students in other each subject, along with aggregations What we’ve learned schools, where ‘similar’ is taken as at the student, subject and school students of equivalent prior level.The database in particular We’ve learned (Rowe, 2000, 2001, achievement two years earlier in the allows for further investigation of 2004a) that it’s teachers who make the School Certificate, of the same gender the student- and class-level difference; whole-school effects are and of the same socioeconomic status information small compared to the effect of (SES). For the teacher in the HSC individual teachers. Multilevel analysis • The Report (DeCourcy, 2005b) to course, each of prior achievement, with all variables converted to normal- CECNSW on the performance of gender and SES is a given, each is liable equivalent deviates as described in the Catholic schools generally in the to have a bearing on achievement, and appendix partitions the variance HSC, any issues arising from the each must be discounted if pedagogical sources for student aggregate Tertiary analysis and a series of statistical effects are to be inferred. Entrance Score (TES); a similar process appendices. can be undertaken for a subject such as A second important aspect of the The process for delivery of the analysis Drama. analysis has been the inclusion of to schools is centred on supporting the confidence intervals (uncertainties) in The contrast between the school effect principal in his/her work with staff. the graphical presentation of results. An in these two analyses is not surprising. Students and schools receive the results apparent improvement of 2% in For a TES, students will have experienced of the HSC in mid-December each average achievement is not significant if at least five different teachers, and usually

Research Conference 2005 94 better than expectation.‘They weren’t a very good group’ implies that expectations should have been low, and that achievement is in line with Y10 Achievement expectation. Both responses beg the Gender question of an appropriate level of School SES expectation, which can be addressed StudentSES using multilevel modelling. School Residual We’ve learned that most practitioners are engaged with the data not through a consideration of the analytic techniques as summarised in the Appendix, but through the use of a valid graphical presentation of the Figure 1 Sources of variance in 2004 TES results of that analysis. For each subject in a school, the Primary Analysis is simply presented as a comparison of ‘Achieved’ with ‘Expected’, building on the unpacking of assumptions describing Y10 Achievement above, showing confidence intervals. Gender Learning to interpret a graph such as School SES this is the focus of the seminar program StudentSES and the manual.The diagonal line where School achieved equals typical is the line of Residual average comparative learning gain in this subject.The centre of the ellipse is the value that this subject in this school achieves as an average achieved standard score, against the average Figure 2 Sources of variance in 2004 HSC Drama typical standard score as outlined in equation (13) in the Appendix.The axes six or seven.The effect seen is an these.When previously the only of the ellipse are determined by the average across all of these standard for comparison for schools confidence intervals of the means, subjects/teachers. Put differently, the data was with state average, or with the derived as outlined in the Appendix. point to a consistent mix of teaching and school’s previous results, there are When the ellipse is completely above pedagogy experienced by students.The predictable responses to results above the diagonal as in this case, the achieved point of leverage for principals is to see average or those below average.Those result is above what students of the those subjects and teachers where the above were greeted with,‘Haven’t we same prior achievement, gender and comparative learning gain is high and to done well!’Those below were dismissed SES have achieved elsewhere.When the build on these strengths; similarly, to see with,‘They weren’t a very good group ellipse intersects the diagonal, it is ‘in the those where it is low and target this year’. Both of these responses rely range of expectation’.When it is appropriate interventions. on assumptions of the comparison of completely below, it is ‘below achievement with expectation.The We’ve learned that in order to engage expectation’. In the case illustrated ‘Haven’t we done well’ response is a principals and teachers with data, you above, the principal and the teacher can claim that compared to what might need to begin with the assumptions indeed be confident that ‘we have done reasonably have been expected of this they make about data, and unpack well’ even though the results may have group of students, they have done been below state average.

Using Data to Support Learning 95 the project give principals a basis for seeing whether the curriculum and pedagogical interventions they apply are having an effect. We’ve learned that SES is related to only a tiny proportion of the variance in aggregate results (as shown in Fig.1), but it may be a bit larger in some individual

Achieved subjects.There has been criticism (Marks, Rowe & Beavis, 2003; Rowe, 2004) of some analyses of achievement data which purport to show large SES effects but are in fact statistically invalid. The 2004 analysis in this project shows Typical 1.1% of the TES variance related to variance in the school-level Farish index (Farish, 2004) and 1.9% related to Figure 3 Typical primary analysis graph for a subject variance in the postcode-average for the individual student (Australian Bureau of Statistics, 2004). 20 We’ve learned that the real test of the 18 validity and utility of a data analysis for a 16 principal lies in his/her ability to 14 recognise in the graphical 12 representation of the subject what s/he 10 knows of what has happened within the 8 school. For the first three years of the 6 analysis, there was simply a single-year 4 snapshot of data.When the data was 2 summarised over time in a trends 0 graph, principals began in a large way to 2001 2002 2003 2004 engage with the data. Fig. 5 shows on HSC of Year particular school’s trend on the primary (comparative learning gain) measure.

Figure 4 Percentage of variance in learning gain related to gender When the principal saw this, he immediately identified the reasons for We’ve learned that gender and SES do relating to appropriately differentiated the drop in 2001 from what he knew make a difference in results, but they are pedagogy, enabling both boys and girls of what had happened in that subject in not variables which schools can change. to engage with the curriculum at their the school, and was convinced of the The approach therefore has been to point of need and learning style are validity of the data analysis process. account for the variable, discount it (by considerable. In 2004, 31 subjects We’ve learned that engagement with factoring it into the typical or expected showed significant gender effects, with data is like peeling the layers of an score as shown in equation (13)), and 30 of these favouring females.The size onion: different audiences begin and end look at the pedagogy. of the significant effects ranged from their engagement at different levels of 2.3% (Mathematics) to 16.7% (Food Gender is related to 10–14% of the the data. For district, diocesan or system Technology).The longitudinal data from variance in TES, favouring girls.The issues officers, the beginning level of interest is

Research Conference 2005 96 2000 2001 2002 2003 2004

Figure 5 Example of a trend graph for primary analysis in one subject in a school whole-state, leading down to system, subject, which can then be further Amrein and Berliner’s (2003) terms, we then to individual schools and often investigated from the trends graph in aim for a low-stakes analysis so that stopping at that level. For a teacher, the that subject. If the second-order effect in teachers engage; if the analysis becomes initial point of interest is the department the subject is significant, this is noted on a high-stakes accountability exercise, within the school, leading down to the trends graph and can have (see then the focus shifts to dealing with the subject, then to class then to individual below) significant utility in developing analysis, rather than using the analysis to student.Teachers do not become pedagogy. Roadmaps have been deal with the pedagogy.There are many engaged if they do not have student- developed for the use of a number of methods of engagement between level data, with each student identified other audiences for the analysis. principals and teachers: some are by name. Hence the database that is outlined below as the levers a principal We’ve learned that a school provided as part of the package has the might use.The manual (DeCourcy, performance feedback system like this facility for schools to convert student ID 2005a) gives more detail on most. has to be responsive to the needs of numbers to names, and or any user to the users, as strongly stated by Visscher Lever 1 for the principal is to ask for begin and end their consideration of the and Coe (2002). Many of the elements the production of a brief report on data at their points of interest. of the analysis, including the web site, each subject, addressing just four We’ve learned that principals and the available PowerPoint files, the questions: teachers can be overwhelmed by a large manual and the database have been • What have you been doing, dataset, but that you do need to provide provided following the expressed needs and why? the large dataset to enable each to of those using the analysis. • How is it going? follow his or her particular point of We’ve learned that once the principal is enquiry or interest. Hence, we have • How do you know? engaged with the analysis, s/he will developed ‘roadmaps’ through the • What do you plan to do next? begin to use it as a lever to move the analysis package to give at least an initial pedagogy and curriculum of the school. The third question demands that the way of logically engaging with the data. A teacher engage with the analysis in typical roadmap for a principal takes The idea of leverage order to substantiate their answer to him/her from the manual (DeCourcy, the second question.The fourth 2005a), to the trend graph ‘Overall The analysis gives principals and question becomes the answer, the School Result’, to Numeric Report 4, the teachers an external point of reference following year, to the first question. ‘school summary’ from the database.This for discussion about pedagogy and for There is not room in this sort of summary unpacks the overall school attempts to improve both pedagogy analysis for blame-the-students result to see the effect of each different and thereby student achievement. In responses, unless the teacher can

Using Data to Support Learning 97 hypothesise a distinctive characteristic Lever 5 is to engage teachers with Lever 6 is in ‘further factor’ analysis. of the particular group of students. If second-order effects in the data.The Explanations of why a particular result s/he can, then dealing with it becomes comparative learning gain shown in has been achieved in the primary the answer to the fourth question. Fig. 3 is a representation of the mean analysis of comparative learning gain learning gain for the whole class, a first- often come back to hypotheses at the Lever 2 is the Overall School Result order effect. Equation (3) in the individual-student level. For example, report, which uses the layout of Fig. 5 appendix has a second-order statistic prior study of the subject in earlier above to plot over time the aggregate u which represents the school-level years, class size, frequency with which comparative learning gain for all 1j residual of the slope of the line of best some students arrived on the late bus students in the school.The single aim of fit for each of the individual students and the differing effectiveness of an increasing comparative learning gain within that subject within that school. different teachers in multi-class subjects on this is a valid and stringent target for Obviously, the line of best fit for one might be hypothesised. A crude test of all in the school. school may be parallel to, steeper than, any of these can be simply performed, Lever 3 is the School Summary or shallower than the typical line of using the data supplied to the school. provided as part of the database, which best fit through the students in all The database supplied to the school for ranks each subject from the highest to schools.Where it is significantly steeper, each student in each subject includes a the lowest comparative learning gain. the comparative learning gain of the calculation (using equation (14) from There is potential for misuse here if the students in the higher end of the the Appendix) of the Achieved and idea behind confidence limits in distribution has been relatively better, a Typical results for that student in the measurement is not understood.The statistic summarised by a simple ‘+H’. subject.The mean of each of these fact that the differences between the Where it is significantly shallower, the gives the coordinates of the centre of comparative learning gain in different comparative learning gain of the the ellipse in the primary analysis. For a subjects are small, particularly when students in the lower end of the categorical hypothesis, such as the compared to the uncertainties is distribution has been relatively better, students who had previously studied illustrated by the relatively large size of ‘+L’. For the principal and teachers, the subject, it is straightforward to gain the ellipses in the primary analysis these second-order effects are the means of the sub-groups, and then (Fig. 3). Hence the School Summary recorded in the School Summary compare how they plot. lists subjects simply with a statement of report, the Trends Analysis, and Lever 7 is in monitoring participation in whether the subject is above, within the individual subject reports. A subject different subjects, particularly in those range of, or below expectation. Its where the focus is on supporting which are most challenging.There is a advantage is that one can easily see struggling students and allowing the temptation for able students those subjects which are close to the capable to fend for themselves will be experiencing their first taste of really boundary between these categories. identified by a string of +L results. A having to struggle with a subject to subject where the focus is on the Lever 4 is to engage with teachers in drop to lower levels of the subject. achievement of the best students and terms of comparative learning gain.The Marsh (1991), Marsh, Chessor, Craven the remainder are allowed to find their focus of future planning is always and Roche (1995), Marsh and Rowe own level will gain +H results. For the around ways in which the comparative (1996), Marsh, Hau and Craven (2003) discussion between principal and learning gain might be improved, as this and the data from this project show teacher, it is a valid aim if you have is the most reliable way of improving that the key to outstanding results in achieved a +H one year to strive to outcomes. For this reason, subject trend higher-level subjects lies in the keep those gains and attempt a +L the graphs such as that shown in Fig. 5 are combination of high participation by following year, all the while keeping the the focus of attention.The aim is to students, positive challenge from first-order effect positive and increasing. keep the graph going up. In dealing with teachers, and appropriate pedagogy.We Similarly, a +L one year can lead to an these, the most powerful leverage should be ensuring that more students aim for a +H the following year.The comes from the simple questions.‘Can take on challenging subjects, rather than pedagogical direction is towards you just explain to me why this graph seeking to advise students out of differentiated instruction. looks like this?’ the subject.

Research Conference 2005 98 Which lever or combination of levers a two marks for each student is his/her and sij is the student-level measure of principal or teacher chooses to use is ‘HSC mark’ in the subject. HSC marks socioeconomic status, taken as the dependent on the school and the for each subject were re-scaled to the postcode-average of the fourth SEIFA students.Together, they form a powerful mean and standard deviation of the x index (Australian Bureau of Statistics, set of tools to address pedagogical values, to give yk values for each student. 2004). In this equation, following change. Goldstein et al. (1998): Within each subject k the values of the mean for each school j of x and y were β β Conclusion k 0ij = 0 + u0j + e0ij (2) obtained. Since both x and yk lie on the β β The international research and the data same scale, the comparison of the and 1j = 1 + u1j (3) from this project show that using data means yjk and xjk is then a crude Allowing for variation of at the as an accountability mechanism, comparison of achieved result with 1 second level detects school effects producing league tables which amplify what might be expected from students where the extent of the impact of prior tiny and statistically non-significant of a similar level of performance two performance varies from school to variations between schools into large years earlier. school, and reveals second-order effects differences in rank, is not effective in The standard error of the independent as described. Allowing for gender and improving student performance.What is variable can be estimated in the usual SES effects detects that part of the effective is valid analysis of data, variance in the HSC mark which can be presenting the results of the analysis in way as , where n is the group attributed directly to gender or SES, an engaging way, targeting professional size for school j in subject k. However, and is not part of the school effect. development to support use of the Goldstein (1995, p. 3) notes that such a analysis and then engaging teachers in method is likely to underestimate the This allows an estimation of the professional development to support standard error of the dependent expected mean result in school j in changes in pedagogy. variable, since it assumes a random subject k to be given as : sampling from the population and in It is the teachers who make the β β β β β this study we are specifically assessing yˆjk = 0 + 1xij + 2gij + 3Sj + 4sij difference. non-random (school) effects on the (4) groupings of the dependent variable. and attributes school effects as being: Appendix: statistical To estimate the standard error in the methodology for the SchoolEffect = u0j + u1j xij (5) dependent variable and to investigate multilevel analysis any gender or SES effects, a model is There is often more than one class of a given subject in a school. It would be For the total sample, the School fitted to the data using MLwiN ideal to construct a three-level model Certificate results in English-literacy, multilevel modelling software (Goldstein for each subject, with students nested Mathematics, Science, Australian et al., 1998) for each subject.This gives within classes within schools. However, Geography and History were converted the value of for student i within school j the data as supplied do not include to standard scores x , x , x , x and x , studying subject k as: e m s h g class designation, so this is not possible. based on the whole-of-state means and A multilevel model was then fitted to Lever 6, described above and in the standard deviations in each test. the data for each subject allowing manual (DeCourcy, 2005a), gives a way For each subject k in the Higher School second-level variation in the 1 value in which this separation may be Certificate, the x values from two years and including gender, achieved by individual schools. earlier were obtained for all students y = β x + β x + β g + β S + β s A second analysis is then performed taking the subject k.Within the Higher ijk 0ij 0 1j ij 2 ij 3 j 4 ij with each set of data, using the School Certificate, each student is (1) separate SC results, giving models awarded a scaled exam mark and a where g is the gender of student i in structured thus: (school-based) assessment mark that is ij school j, S is the school-level measure moderated for each school against the j of socioeconomic status (Farish, 2005), examination mark.The mean of these

Using Data to Support Learning 99 β β β β β β β β β The process compares subject with yijk = 0ijx0 + 1jxEij + 2jxMij + 3jxSij + 4jxHij + 5jxGij + 6gij + 7Sj + 8sij (6) subject within the HSC using students β β where 0ij = 0 + u0j + e0ij (7) common to pairs of subjects to derive β β a mapping of the Board of Studies 1j = 1 + u1j (8) (BOS) marks to a new ‘UAImark’ for β β 2j = 2 + u2j (9) each student in each subject. From these a ‘UAI mean’ for each HSC β = β + u (10) 3j 3 3j subject is derived and published β β (Cooney, 2005).These UAI mean then 4j = 4 + u4j (11) vary over a wide range, representing β β 5j = 5 + u5j (12) the relative performance of the cohort taking the particular subjects. (For giving example, Mathematics Extension 2 has β β β β β β β β β yˆjk = 0 + 1xEij + 2xMij + 3xSij + 4xHij + 5xGij + 6gij + 7Sj + 8sij (13) a UAI mean of approx 44/50; at the other end of the scale, Construction and has a mean of approx 16/50.) ˆ β β β β β β β β β yijk = 0 + 1xEij + 2xMij + 3xSij + 4xHij + 5xGij + 7gij + 8Sj + 8sij (14) Schools in NSW are given no information about individual student’s Finally, the process of equations (6) – (13) In both cases, the effect inferred is UAI or TES. However, it is possible to is repeated with all variables converted relative. A ‘+H’ is necessarily a ‘–L’, and take the individual student marks as to normal equivalent deviates in order vice versa. Neither a ‘+H’ nor a ‘+L’ an provided by the BOS and to map them to obtain overall relationships between inference of a deficiency in the teaching to gain reasonably accurate TES values the variances in the dependent and and learning: it is simply an observation using the published data of the independent variables. of an effect. Universities Admission Centre (Cooney, Derivation of the 2005).This is done by a simple linear Conversion to the mapping, such that a value t is gained as second-order effect k Tertiary Entrance the TES equivalent of the BOS mark yk where y lies between the mapping In equation (3) above, the residual u1j Score scale k is significant in educational terms for points b and bd, where these points map schools. If the value is positive and The process of producing the to the UAI/TES scale uc and ud thus: University Admission Index (‘UAI’, significantly above 1 (MLwiN provides Cooney, 2000) derives a measure of yk – bc both the value and standard error of tk = .(ud – uc ) + uc (15) the residual) then the school has the student’s performance compared bd – bc provided significantly greater learning with a whole-of-age group cohort. (A gain for the higher-achieving end of the index similar to the UAI is produced in Comparison with State student distribution than is found in each state in Australia, the Equivalent average other schools in this subject. Such a National Tertiary Entrance Rank As a part of the feedback to schools, result is depicted with the designation (ENTER).) In NSW the UAI is one of the six presentations of the data ‘+H’ in the school report of the subject. produced from a Tertiary Entrance that is provided is the comparison of Score (TES).The TES is a mark out of If the value is negative and significantly school and state mean in the subject.To 500, consisting of the aggregate of the below then the school has provided place the differences between school 1 best 10 units of the student’s re-scaled significantly greater learning gain for the and state means in all subjects on the scores, including a minimum of 2 units lower-achieving end of the student same scale, the difference that is of English. (Most subjects are 2-unit in distribution than is found in other schools reported (∆tkm) is the difference on the in this subject.This is depicted with the value, giving a mark out of 100.) TES one-unit scale. designation ‘+L’ in the school report.

Research Conference 2005 100 Comparison between Cooney, G., (2005). Report on the Marsh, H.W., Chessor, D., Craven, R. & this subject and all scaling of the 2004 NSW Higher Roche, L. (1995).The effects of gifted School Certificate. Sydney: Universities and talented programs on academic others in the school Admission Centre; NSW Vice- self-concept: the big fish strikes again, Comparisons of the different means of Chancellors Conference Technical American Educational Research Journal, subjects within the school carry little Committee on Scaling 32, (2), 285-319. information when the subjects are on DeCourcy, J. S. (2005a). HSC data Marsh, H.W., Hau, K.T., & Craven, R. the BOS scale. Even when the marks analysis:The manual, Sydney: Catholic (2003).The big-fish-little-pond effect are re-scaled as described above to the Education Commission. stands up to scrutiny, American TES scale, the fact that different subjects Psychologist, 59, 4, 269-271. DeCourcy, J. S. (2005b). Report on HSC attract candidatures of varying ability 2004. Sydney: Catholic Education O’Day, J. A. (2002). Complexity, means that little can be gained by direct Commission. accountability and school comparison of means. improvement, Harvard Educational Farish, S. (2004). Funding Arrangements However, if one uses the NSW DET Review, 72, 3, 293-329. for Non-government Schools 2005- method (Smith, 1999) a clear-cut 2008: Recalculation of the Modified A Rowe, K. J. (2000). Assessment, league comparison between subjects within the Socioeconomic Status (SES) Indicator tables and school effectiveness: school can be obtained. Because the TES using 2001 Australian Bureau of Consider the issues and ‘Let’s Get Real’, process places all marks on a common Statistics Census Data Canberra: Journal of Educational Enquiry, 1, 1, 73-98. scale, the comparison of each student’s Department of Education, Science and mark in a subject t with the mean value Rowe, K. J., (2001). Equal and different? k Training. of that student’s results in each of his/her Yes, but what really matters?, other subjects tim gives a measure of the Goldstein, H., (1995). Multilevel Riverview Boys Education Conference, extent to which the individual student’s Statistical Models London: Arnold, 2nd October 2001. performance in the subject is ahead of or edition. Rowe, K. J. (2004a).‘The Importance of behind other subjects.The mean of these Goldstein, H., Rasbash, J., Plewis, I., Teaching: Ensuring better schooling by individual values for all students in the Draper, D., Browne,W.,Yang, M., building teacher capacities that subject t then gives a reliable kjm Woodhouse, G. & Healy, M., (1998). A maximize the quality of teaching and comparison of subjects within the school. User’s Guide to MlwiN, University of learning provision – implications of London: Multilevel Models Project, findings from the international and tkjm = tk – tim (16) Version 1.0 Australian evidence-based research’; Paper presented at the ‘Making Schools Marks, G. N., Rowe, K.J., & Beavis, A. References Better’ Conference, Melbourne (2003). Australian Schools not so University, 26-27 August 2004. Amrein, A. L., & Berliner, D. C. (2003). ‘Undemocratic’, Campus Review, ‘The effects of high-stakes testing on June/July 34. Rowe, K. J. (2004b).‘Analysing and student motivation and learning’, Reporting Performance Indicator Data: Marsh, H.W., (1991). Failure of high- Educational Leadership, 60, 5, 32-38 ‘Caress’ the data and user beware!’ ability high schools to deliver academic Paper presented at the 2004 Public Australian Bureau of Statistics (2004). benefits commensurate with their Sector Performance and Reporting SEIFA 2001: The socioeconomic index students’ ability levels, American Conference, Sydney 19-22 April 2004. for Areas. Canberra: Self Educational Research Journal, 28, (2), Braun, H. I., & Mislevy, R. (2005).‘Intuitive 445-464. Smith, M. (1999). Methods for within- school comparison of subject results Test Theory’, Phi Delta Kappan, 86,7, Marsh, H.W. & Rowe, K. J., (1996).The Pers. Comm. 489-497 negative effects of school average Cooney, G. (2000).The Universities ability on academic self-concept: An Visscher, A. J. & Coe, R. (Eds), (2002). Admission Index (UAI). Sydney: application of multilevel modelling, School improvement through Universities Admission Centre Australian Journal of Education, 40, performance feedback, Lisse,The 65-87. Netherlands: Swets and Zeitlinger.

Using Data to Support Learning 101 An evidence-based approach to teaching and learning

A Greek philosopher might suggest that include teacher observation, tests, peer evidence is what is observed, rational assessment and practical performance – and logical; a Fundamentalist – what you and constitutes the information and know is true; a Post Modernist – what data that is used to gauge the you experience; a Lawyer – material educational attainment and progress of which tends to prove or disprove the individuals; groups; and cohorts; and existence of a fact and that is admissible increasingly, the effectiveness of in court; a Clinical Scientist – programs and performance of Michele Bruniges information obtained from observations educational systems. and/or experiments; and a teacher – Information and assessment data are Department of Education and Training, what they see and hear. Australian Capital Territory increasingly used for multiple purposes, The past decade has seen a high level including national and international Michele Bruniges (Dip T., Grad Dip Ed. Studies., of engagement and commitment by comparisons of standards of learning M.Ed. Ph.D.) has experience teaching in both Australian schools to the collection, and educational attainment (Timmins, primary and secondary schools. She has also held analysis and interpretation of 2004). Increased pressures at a local the positions of Senior Curriculum Adviser, information about students to inform level to meet accountability Assessment and Reporting, Chief Education teaching and learning. Rapid changes in requirements, and to deliver improved Officer, Mathematics and Assistant Director of results across the cohort have ‘put data School Assessment and Reporting for the NSW society, economics and technology, the Department of Education and Training. increased demand for accountability, to an increasing array of use’ (Timmins, and the need to prepare all students to 2004, p. 2) in schools. During 1999, Michele received an award for excellent service to public education and training be citizens in an increasingly globalised in NSW. The following year, Michele was world, has cultivated the increased Why is an evidenced- appointed Director of Strategic Information and requirement to inform and improve based approach to Planning with responsibility for leading and education through various evidence- teaching and learning directing systems performance, information based approaches. systems and corporate and strategic planning. In important? the same year, she was awarded a Churchill However, while evidence is one way to Fellowship to study the analysis, monitoring and support the core business of schools As realised by many educationalists, an reporting of student achievement in education –maximising student learning and evidence-based approach to teaching systems and research studies in the United States and learning is crucial to maximising and the Netherlands. outcomes – evidence in and of itself is not sufficient to maximise student student outcomes.We need to ‘know’ – Michele was appointed Assistant Director- outcomes. If we are serious about to have evidence about the General, School Education Services NSW in 2003 performance of our students in order with a strong interest in educational measurement developing and maintaining an evidence- to support them to achieve high quality issues, school culture and the process of managing based culture of improvement in change. In early 2004, Michele was appointed teaching and learning, the unique and educational outcomes. Regional Director,Western Sydney with priorities specialised knowledge, skills, experience There are four major ways in which we including a renewed focus on supporting frontline and professional capacity of teachers can use the information we gain from teachers and school staff and the provision of must be valued as fundamental quality responses to local issues. In January 2005, assessment (our evidence) to maximise Michele took up the position of Chief Executive components of any evidence process. student learning and outcomes.These of the ACT Department of Education and That is, the way in which evidence is include using evidence to: Training. obtained, collated, interpreted and results • improve the focus of our teaching strategically utilised, must be interlinked (a diagnostic capacity) with, and influenced by, the profession. • focus students’ attention on their What is evidence? strengths and weaknesses (a motivation capacity) Evidence is obtained through various • improve programming and planning forms of assessment – which may (a means of program assessment)

Research Conference 2005 102 • report on an assessment (a means reporting (in favour of, or in achievement scales that articulate a of communicating student collaboration with, the more traditional continuum of typical development in a achievement) norm-referenced assessment) by specified domain. Once defined, these Australian education systems as the maps can be used to describe quality In order to most effectively support primary means to describe students’ student achievement at both a point in students to achieve quality educational achievements and progress has enabled time and over time.This development outcomes, the process of evidence to the use of data to identify particular has also provided the means to inform teaching and learning must be strengths or weaknesses in curriculum establish where individual students are an explicit and accountable one, which terms at the classroom, school and in a continuum of learning the essential is equitable, representative, valid, and system levels. One example of this has starting point from which to develop a reliable. been the development of assessment relevant and appropriate learning Sharing the secret rubrics. Rubrics have been powerful in pathway. supporting student learning in their The increased use of information and simplistic form by providing a list of Quality teachers make assessment data to inform teaching and criteria, or ‘what counts’ in a project or the difference learning brings a largely recognised assignment; and in providing a scale increased need for assessment that is describing the characteristics of a range We know that quality teachers make a an open and accountable process about of student work.This tool creates the significant difference to the learning what really matters, what students structure for important conversations outcomes of students. John Hattie’s should know, and a process that about assessment by providing students (2003) recent rigorous and exhaustive provides the best information to them with informative feedback about their research has provided profound and on how they can improve. work and detailed evaluations of final powerful evidence to support this products (Department of Education conviction – ‘excellence in teaching is Assessment should not be a covert Tasmania). the single most powerful influence on mission, but rather a process defined by achievement’.The design, collection and the importance of transparency and Criterion-referenced assessment sheds response to findings are intimately information sharing which is directed by light on many of the previously linked to the art of effective teaching positioning the needs of students as protected secrets of assessment. In the and will impact significantly on student paramount. Providing students with past, the details of assessment have educational achievement. minimal and nondescript information usually remained teacher-only about assessment is an antiquated information. However, increasingly so, In many disciplines, field professionals approach, which has the potential to teachers and students are engaging in are predominantly identified as having disengage students from an important conversations about assessment that the most astute and profound aspect of their learning experience and involves a common language.These knowledge, skills, experience and limit their capacity for achievement. Being conversations are crucial to provide the professional capacity to make open with students about the once held learner with an opportunity and judgements about the most effective secrets of assessment, and engaging impetus to discuss how goals are set, way to obtain, collate, interpret and students in associated questioning and how performance is measured, and apply evidence. Professional educators conversation, provides a greater how performance can be improved. have a unique and specialised capacity opportunity for all students to achieve Significantly, they enable the learner to to lead and contribute to evidence- high quality educational outcomes. experience an active role in the based approaches to teaching and assessment process.They also provide learning – because, it is they who know The development of assessment that important feedback for teachers that best, the ‘subject’ matter and the makes explicit the standards, criteria can be used to respond to students’ individual.Teachers are distinguished and feedback for students has been particular needs. from other professions by their deep recognised as a significant development knowledge of how the learning process in describing and quantifying student Advances in educational measurement occurs.This places teachers in an achievement and progress.The have paved the way for the inimitable position to utilise a range of adoption of criterion-referenced introduction of progress maps or

Using Data to Support Learning 103 profession-specific, as well as locally interrogate evidence.The value of collaboration, are in a powerful position specific, skills, knowledge and evidence does not necessarily lie solely to direct and sustain this alignment, in experiences, to improve the educational in the description that it provides of order to provide effective education. outcomes of their students. student achievement – but rather, the In identifying the variables that impact way in which this description is While it is necessary to value, or at on student learning, Hattie (2003) interrogated and understood in order least consider, all sources of evidence, confirms that within schools, teachers to develop and apply appropriate we must not hesitate to recognise that account for about 30% of the variance strategies to improve student learning. It teachers are often in a leading position in student achievements – the major is fair to say that traditionally the role of to identify and act on the best way in source of ‘within-school’ variance.There the teacher in this process has been which to obtain and assess the is also a ‘growing body of evidence that undervalued. However, if evidence is to worthiness and weight of the diverse the use of high-quality, targeted be used most effectively, the capacity of range of evidence collected about assessment data, in the hands of school the teacher to ask the right questions students. Just as the judgement and staff trained to use it effectively, can of evidence, to examine the how and authority of a doctor is respected in the improve instruction’ (Protheroe, 2003) why of evidentiary results, and to assessment he/she makes of a patient, and consequently, student outcomes. respond with the most effective and the medication he/she prescribes Furthermore, Nancy Protheroe suggests strategies, must be realised as to achieve an outcome of health and that educators who have learned to paramount. well-being, so too should the effectively use assessment data have professional expertise of teachers be While it is critical to realise and support often ignited change and achieved valued and trusted, in the quest for the role of teachers in leading and positive results.This evidence provides a high-quality educational results. contributing to evidence-based compelling argument of the importance approaches to teaching and learning, it of continuing development of the Teachers are in a unique position to is also important to consider that teaching profession, and that in have an extensive and well-developed teachers have a responsibility to the particular, teachers are supported to range of strategies and techniques that profession, as well as a broader social play a leading role in evidence-based can be used to identify and meet the responsibility, to account for decisions approaches to teaching and learning. current needs of a diverse range of that are made. In times of increased This includes supporting teachers to students – and, moreover, to match the change, it is necessary that the teaching see and learn from each other’s work future desired achievements of the profession builds strong links with and experiences, in order to expand students to a plan for action. No, research communities in order to the circle of professional collaboration teachers cannot necessarily predict the understand the most current directed towards student achievement, future! However, they do have a rich developments about learning and and developing ways to ensure that the capacity to accumulate a broad-ranging development to enhance and sharpen best teachers are retained in the area repertoire of strategies that enable them their knowledge. For, if we are to of greatest impact – the classroom. to match a strategy to a student’s needs. support the notion that the creativity, With this knowledge base, teachers are ingenuity and expertise of teachers be Conclusion able to make informed judgements valued and prioritised, the thinking and about how best to work towards It is the ‘evidence’ that we are instruction of teachers must be further developing students, selecting presented with that often informs relevant, perceptive, dynamic and assessment strategies that accurately decisions that are made about student forward looking. reflect what it is that our students know; learning, and about the health of use evidence to support students for Alan Luke (1999) argues that effective education. However, evidence alone is further achievement; and prepare education requires alignment of the not sufficient to maximise student students to be active and contributing three key message systems that exist in outcomes. Quality teachers are a citizens, now and into the future. education: curriculum, pedagogy and fundamental part of the recipe for assessment. Luke’s argument is a successful evidence-based approaches Furthermore, teachers are in a powerful one, and teachers, enabled by to teaching and learning.The distinctive position to be able to professional autonomy and knowledge, skills, experience and

Research Conference 2005 104 professional capacity of teachers must be valued as essential ingredients in meeting the goals of the core business of education systems and ensuring that educational attainment across the nation continues to rise. References Hattie, John (2003). Teachers make a difference:What is the research evidence? Paper given at the Australian Council for Education Research Annual Conference on: Building Teacher Quality. Luke, Allan (1999): Education 2010 and new times:Why equity and social justice still matter, but differently, Paper presented to the Education Queensland Online Conference, 20 October 1999. Available online at: http://education.qld.gov.au/corporate/n ewbasics/docs/onlineal.doc Timmins, Robyn. (2004). Putting the nation to the test, is there room for improvement? Paper presented at the 9th Annual Assessment Roundtable: Assessing Assessment Conference, Sydney, New South Wales, 7-9 November 2004 P.1 Protheroe, Nancy (2001). Improving teaching and learning with data-based decisions: Asking the right questions and acting on the answers, Educational Research Service: Making a Difference in Our Children’s Future, Available online at: http://www.ers.org/spectrum/ sum01a.htm, p.1. Department of Education Tasmania, (2005), Available online at: http://www.education.tas.gov.au/ocll/cu rrcons/profreadings/andrade.htm

Using Data to Support Learning 105 Assessment for learning: Using Statewide Literacy & Numeracy tests as diagnostic tools

Assessment of, ‘for’ and ‘as’ learning grouping both assessment as, and under the one title of ‘formative assessment’.They do, however, make a assessment for learning distinction between this formative The Victorian Government’s Blueprint for assessment and summative assessment. Government Schools (Department of Black and Wiliam (1998c) argue that Education and Training, 2004a) promised raising the standards of learning has the development of guidelines ‘to assist become an international priority but Philip Holmes-Smith government schools with school-based governments throughout the world decision-making in areas of curriculum have put most of their efforts into School Research, Evaluation and development, pedagogy, assessment and summative assessment type initiatives Measurement Services reporting’. Subsequent draft assessment such as ‘National, State, and district advice (Department of Education and standards; target setting; enhanced Philip Holmes-Smith B.Sc., Grad Dip Curriculum Training, 2004b) suggests that Admin., M.Ed. is the principal consultant at School programs for the external testing of Research, Evaluation and Measurement Services assessment for improved student students’ performance; surveys such as (SREAMS), an independent educational research learning and deep understanding NAEP (National Assessment of consultancy business. Philip has taught Science, requires the range of assessment Educational Progress) and TIMSS (Third Biology and Mathematics in Victorian government practices to be used including: International Mathematics and Science secondary schools. He is also an experienced Study); initiatives to improve school educational program evaluator, consultant, • Assessment for learning [which] is planning and management; and more researcher and psychometrician/statistician whose undertaken to ascertain students’ current research, evaluation and measurement prior knowledge, perceptions and frequent and thorough inspection’. projects and interests lie in the areas of teacher misconceptions and to monitor However, Black and Wiliam (1998b, effectiveness and school improvement; school student learning progress … [and] 1998c) argue that the impact of all accountability models; improving literacy, to inform teaching practice and these reforms have amounted to little. numeracy and science teaching; the use of They, and others, argue that there is learning technologies to improve student curriculum planning in order to now compelling evidence to show that outcomes; students at educational risk; and large- support students’ future learning and scale, state achievement testing programs understanding. (formative assessment) ‘the important message now including benchmarking. confronting the educational community • Assessment as learning [which] is that assessment, which is explicitly Philip established SREAMS at the beginning of focuses on constructive feedback designed to promote learning [formative 1997 and is currently contracted to a number of from the teacher and on developing State education departments and curriculum assessment] is the single most powerful the student’s capacity to self assess authorities around Australia conducting tool we have for both raising standards and reflect on their learning – to curriculum program evaluations, educational and empowering lifelong learners’ improve their future learning and research and psychometric analysis of state-wide (Assessment Reform Group, 1999). testing programs. understanding. (ongoing assessment) • Assessment of learning [which] makes The pattern of assessment efforts in judgements about what the student Australia certainly mimics that of many has learned in relation to the other countries such as the United teaching and learning goals. … [and] Kingdom and the United States in that should be comprehensive and reflect each State Government conducts its the learning growth over the period own version of Literacy and Numeracy assessed. … (summative assessment) summative testing at Years 3, 5 and 7 (and will be extended to Year 9 by Assessment for, assessment as, and 2007) and the expenditure on these assessment of learning are certainly not policy initiatives greatly overshadows new concepts. For example, Black and any policy initiatives that promote Wiliam (1998a, 1998b, 1998c) make formative assessment. Reporting of little distinction between assessment these data is dominated by summative

Research Conference 2005 106 type reports and accountability units one or more outcomes from the State’s the bottom of the report shows that around the country look at these data curriculum standards, responses to each thirteen of the twenty-one students (or in terms of what they say about overall item reflect students’ knowledge, 62% of the group) answered this school performance. perceptions and misconceptions with question correctly. Furthermore, the respect to the standard being tested. summary data shows that across the Using Statewide Let us look at two items from Victoria’s whole State, 59% of students answered Literacy & Numeracy English and Mathematics tests to this item correctly. In other words, demonstrate the use of Statewide although less than two-thirds of the tests as diagnostic tools Literacy & Numeracy tests as students in this group answered the item Despite this apparent incongruence diagnostic tools. correctly, this percentage was very similar to the percentage of students in the between what the research suggests Look at Table 1. It shows for each student whole State. will have the greatest impact on raising at the school the response to each item. the standards of learning (formative If a student answered the item correctly, a More interesting, however, is to note that assessment) and the actual practice that ‘tick’ is entered in the body of the table; if of those students in this group who most jurisdictions around the world a student answered the item incorrectly, answered the item incorrectly, all of have implemented (summative the incorrect response is entered into the them gave the same incorrect answer ‘c’. assessment), it is too often overlooked table. Look at the highlighted item No. 6. (This can be seen by either running your that summative type tests actually The answer key shows that the correct eye down the column for item No. 6 or contain valuable formative (or answer is option ‘d’ and that the first by computing a tally for each response diagnostic) information. three students (David Billsdon, Stephane as has been done in the last four lines of For example, in Australia, because each Byrne and Anne Conlan) answered this the table.) By reviewing the item, State bases its tests on its version of item correctly.The fourth student teachers at this school can gain some the national curriculum profiles and (Rhonda Connor), however, gave the very interesting diagnostic information because each item on the tests reflects incorrect response ‘c’. Summary data near about this group of students.

Table 1 Student response report for Year 3 Reading

Using Data to Support Learning 107 The letter from Paul clearly says ‘Please don’t give me any more homework’. Any student who gave the answer ‘a’ can’t even read for literal interpretations from this text. Most students would probably first approach answering this question by looking at each letter and trying to find a one-to- one match between the question and the text. For most students, matching up ‘Which student wants to some extra work’ with ‘Please don’t give me any more homework’ is a simple task and option ‘a’ would be eliminated easily. Although no one in this school gave ‘a’ as the answer, there were many in the State that did indicating a low level of literal reading skills amongst such students. The letter from Alice says ‘Please don’t give us any more projects to do’. Matching up ‘Which student wants to some extra work’ with ‘Please don’t give us any more projects to do’ is a slightly harder task because it requires students to have a higher level of vocabulary understanding to be able to equate extra work and more projects. Again, although no one in this school gave ‘b’ as the answer, there were many in the State that did, indicating a low level of literal reading skills amongst such students. It should be noted, however, that these students may be reading at a higher level that those students who answered ‘a’ – their problem may be more to do with a limited vocabulary. The letter from Vlad says,‘Thank you for all the homework you give us’ but makes no mention of whether or not he would like to do some extra work. The letter from Rosa says,‘I am glad that you don’t give us homework on the weekends’ but does ask,‘Please can I do a project on the solar system to share with our class’. Students who answered either ‘c’ or ‘d’ can probably

Research Conference 2005 108 comprehend the literal interpretation of some powerful information. If that seven of the eighteen students (or option ‘a’ or ‘b’ and have eliminated supported by information from similar 39% of the group) answered this them as possible correct answers.This items, it would be possible for the question correctly. Furthermore, the places these students at a higher teachers to conclude that reading the summary data shows that across the reading level than those who gave literal meaning of text is probably a skill whole State, only 47% of students answer ‘a’ or ‘b’.What separates already mastered by all students in Year answered this item correctly. Although students who gave the correct answer 3 at this school.Therefore, it would less than half the students in the State ‘d’ from those who answered ‘c’ is the probably be a waste of time devoting answered the item correctly, even fewer level of inference these students are too many learning and teaching students in the school answered this able to make from the text.Those who opportunities to this skill. On the other item correctly. answered ‘c’ are incorrectly making the hand, although about two-thirds of More interesting, however, is to note inference that liking homework (‘Thank students can make correct inferences that of those students in this group you for all the homework you give us’) from text, making inferences from text who answered the item incorrectly, with wanting to do even more of it. On has not been mastered by all Year 3 most of them (another 39%) gave the the other hand, those who answered ‘d’ students at this school and additional same incorrect answer ‘d’. (Again, this have been able to infer correctly that learning and teaching opportunities in can be seen by either running your eye wanting to do a ‘project on the solar this area are warranted. down the column for item No. 17 or by system’ means that Rosa wants to ‘do Table 2 gives a Numeracy example. computing a tally for each response as some extra work’ even with the Again, it shows for each student at the has been done in the last four lines of distracter statement from Rosa that she school the response to each item. Look the table.) Again, by reviewing the item, was ‘glad that you don’t give us at the highlighted item No. 17.The teachers at this school can gain some homework on the weekends’. answer key shows that the correct very interesting diagnostic information The diagnostic information from this answer is option ‘b’.The summary data about this group of students. item gives the teachers at this school near the bottom of the report shows

Table 2 Student response report for Year 3 Mathematics

Using Data to Support Learning 109 To answer this question, students need to Furthermore, some students, but not results from the tests actually contain employ at least two different skills. First, many, also have a problem matching valuable formative information that could they must be able to match numbers numbers written in word format with the inform teaching practice and curriculum written in word format with the equivalent number written in numerical planning in order to support students’ equivalent number written in numerical format.These students need to be future learning and understanding. format.That is, fifteen hundred means 15 identified and supported. Research suggests that much could be lots of 100. Second, they need to have gained (in terms of raising the standards Interestingly, this item also appeared on some understanding of place value.That of learning) by supporting classroom the Year 5 Numeracy test as a link is, 1500 signifies one lot of a thousand teachers to make better use of item1. At Year 5, 82% of all students in (which indicates 10 lots of 100) plus 5 formative assessment. One place to start the State answered this item correctly lots of 100 (i.e. 15 lots of 100). should be to support teachers in using and almost the same number of Statewide Literacy & Numeracy tests as In this school, about 6% of the Year 3 students in the school answered diagnostic tools. students gave the incorrect answer ‘a’ correctly. However, of the remaining (150) as their answer.These students students answering incorrectly, by far show no understanding of either the greatest number gave ‘d’ as their References concept. No one gave answer ‘c’ answer, indicating that there are still a Assessment Reform Group (1999). indicating that the ‘15’ in 10 015 fooled few students in Year 5 who do not fully Beyond the black box. Cambridge: no one. However, 39% of the Year 3 understand the concept of place value. University of Cambridge. students in this school chose the incorrect answer ‘d’ (15 100), suggesting Summary Black, P., & Wiliam, D. (1998a). that while these students may be able Assessment and classroom learning. to match numbers written in word Assessment of learning dominates Assessment in Education 5(1) pp. 7-71. format (‘fifteen hundred’) with an assessment efforts around the world, Black, P., & Wiliam, D. (1998b). Inside the equivalent number that somewhat and systems, whether intentionally or black box: Raising standards through resembles its numerical format (15 unintentionally, typically portray such classroom assessment. Kings College 100), they do not fully understand the programs as the best means of raising London. concept of place value.That is, they the standards of learning. At the Black, P.& Wiliam, D. (1998c). Inside the have incorrectly read ’15 100’ as classroom level, such programs are rarely black box: Raising standards through meaning 15 lots of 100. appreciated and most teachers have little faith in either the reliability or validity of classroom assessment. Phi Delta Again, the diagnostic information from this such State-mandated tests.While it is Kappan Vol. 80(2) pp.139-148 item gives the teachers at this school clear that school administrators take the Department of Education and Training some powerful information. If supported results of such programs very seriously, (2004a). Blueprint for Government by information from similar items, it would few classroom teachers give any more Schools. Melbourne: Department of be possible for these teachers to conclude than cursory attention to the results for Education and Training that place value is a concept not their own class.This is a shame for a understood well by many students. number of reasons. First and foremost Department of Education and Training Therefore, additional learning and teaching amongst these reasons is that item-level (2004b).Website: www.sofweb.vic.edu.au/ opportunities in this area are warranted. blueprint/fs1/assess.asp

1A link item is an item placed on two different Year level tests to aide in the establishment of a common performance scale across the different Year levels.

Research Conference 2005 110 Data-informed research and practice: Evaluating student achievement in secondary schools

Abstract bureaucrats and researchers far removed from the heart of education – The advantage of ‘ability-adjusted’ students and teachers in schools – analyses of educational data is their whereas data-informed evaluation of capacity to provide fairer assessments student achievement and school of school and student achievement than performance is the term preferred by reliance on raw scores alone. School practitioners.This is not a pedantic performance evaluations based on wordplay – it highlights a key difference students’ unadjusted (raw) marks favour Carmel Richardson in the attitudes and practices of system schools with higher intakes of bright personnel compared with those of Australian Council for Educational Research and advantaged students.The learning school staff. At system-level, the focus is gains of middle and lower ability on a top-down, reform driven Carmel Richardson is a Senior Research Fellow at students are overlooked, and the judgement of schools in terms of their the Australian Council for Educational Research. achievements of students and schools in students’ achievements, with teachers She taught in secondary schools for more than disadvantaged areas are not valued, twenty years, mainly Maths, English and Special typically assumed responsible when Education, before commencing doctoral studies at while the focus is concentrated on underperformance is identified. For The University of Melbourne. Carmel’s main those achieving the highest marks.With school staff, the emphasis is on research area is in analysing ability-adjusted ‘ability-adjusted’ analyses of school data, integrating their external and internal student achievement data to provide schools and any student who achieves higher marks quantitative results with their qualitative teachers with information to track the learning than similar ability peers is progress of each student. She analysed the data data, to more comprehensively inform acknowledged as having performed for the Hong Kong Quality Education Fund their monitoring of student (1997–2000) as part of the project to develop well.This paper describes findings from achievement and their school’s Performance Indicators for Measuring Value- a series of ‘ability-adjusted’ analyses performance. added Improvement of Primary and Secondary conducted within individual schools, Students’ Academic Performance. Carmel has where students’Victorian Certificate of But what processes are required to conducted large-scale multilevel modelling Education (VCE) results were analysed ensure that both educational systems analyses of Victorian final secondary assessments at student and class levels. Staff and schools accurately and fairly assess (1994–2000) for all secondary schools in Victoria, student and school achievement? Australia. Carmel moved to ACER in 2003, and is members were assisted with verification now providing detailed ‘within-school’ multi-level and interpretation of their data to Simplistic rankings place schools and analyses of Victorian Certificate of Education data ensure positive use within their school. classes with large numbers of bright at student and class levels for more than one This research led to a number of students as the top performers, while hundred Victorian secondary schools.This work is practitioners seeking ‘ability-adjusted’ real achievements in schools and classes being extended to primary schools as well as to with more disadvantaged and lower the junior and middle years in secondary schools. analyses of their junior and/or middle- ability students are ignored. Clearly, as Carmel is Project Director of ACER’s Data secondary students’ achievements, as Interpretation Service. they recognised the benefits of this far as possible, all variables that affect data-informed approach.The impact in student learning ought be taken into terms of improving teaching and account, if genuine ‘value-added’ learning, and the on-going challenges educational performance is to be inherent in designing each school’s recognised. Research in the School database, aligned with curriculum and Effectiveness and School Improvement assessment policy, are discussed. (SE&SI) tradition has consistently identified individual student ability and Background and context prior attainment as key factors associated with student achievement; Data-driven quality assurance is a and socioeconomic status is the most popular term used by system

Using Data to Support Learning 111 commonly debated contextual variable • The value of an ability measure, been developed in their earlier years of (Hattie, 2003; Hill & Richardson, 2001; appropriately verified and schooling, although they are advised to Hill & Rowe, 1996, 1998; Mortimore et comprehensively interpreted, for look at sample questions and past al., 1988; Schereens & Creemers, 1989; more accurate evaluation of school papers.The GAT is a component of the Teese & Polesel, 2003). and student performance. statistical moderation process used by the Victorian Curriculum and Teachers need ability and prior • The evidence regarding the real Assessment Authority (VCAA) in their attainment data on each student at the gender effect, illustrating the error in monitoring of the school assessed work start of the school year to monitor the general statement that ‘boys are and examinations used to calculate the student progress effectively, and to underperforming in relation to girls’, student’s VCAA Study Score; which was provide parents with valid reports on based on overall patterns in the achievement measure for this their child’s learning gains each aggregated data. research. semester. Schools are hampered from • The dangers associated with achieving these goals for many reasons, referring to class-level variance, that The methodology and modelling one being the negative attitude towards is, the class residual, as the followed the process initiated by Hill & ability measures held by some class/teacher effect, or even more Turner in 1995 for 10 schools in the educators, due in part, to the misuse of misleading, as the teacher effect. pilot version, and further developed by ‘intelligence’ tests throughout the last • The need for each school to Rowe in 1998 for 50 schools in the trial century.The major reason, however, is develop a ‘within-school’ database to VCE Data Project (Rowe, 1999). Effect the lack of an ability measure, and enable ‘ability-adjusted’ monitoring of sizes were calculated for the five system level failure to supply schools student and school performance. explanatory variables – three student with developmentally appropriate ability measures, school mean ability in • The need for better resourcing to attainment measures, scored on a each Study, and student gender, used to ensure instructional effectiveness common metric, longitudinal scale. generate each student’s expected VCE within schools, and greater focus on Currently, school reports do not results. School residuals, representing monitoring system-level effectiveness. provide the next year’s teacher with the difference between the predicted indicators of the standard that students The research background 1. and achieved VCE scores, were plotted have achieved in terms of clearly for the 20 subjects each year. ‘Across-schools analyses’ delineated skills and knowledge within Consistency in the school’s each subject; nor do they give parents For the seven years, 1994 to 2000, performance across the 20 Studies each indicators of their child’s achievement in ability and attainment data for all VCE year, and stability in each subjects over relation to his or her potential.There students in every government, Catholic time, were noted (Richardson, 2000a). are divergent views in the educational and Independent school, were analysed The school residuals at this time were community about the merits and at student and school levels, in a series interpreted as indicators of the school’s demerits of schools’ reports that of variance components models for ‘ability-adjusted’ position amongst VCE indicate student ability and achievement each of the 20 largest VCE Studies. providers across the State (Rowe, in relation to school-aged peers. 1999). The measure of ability for this ‘across- This paper discusses one approach schools’ research was the General Several concerns surfaced when these where ability and achievement data Achievement Test (GAT), based on ‘ability-adjusted’ results were shared with were analysed, electronically displayed general knowledge and skills in three schools. Practitioner-informed and comprehensively interpreted to domains:Written Communication (GAT explanations for the patterns in their assist school staff in monitoring student c); Mathematics, Science and Technology school’s subject residuals were not and school performance.This work was (GAT m); and Humanities, the Arts and congruent with interpretations typically built on ‘value-added’ analyses from two Social Sciences (GAT h). Each year, made by system-level bureaucrats and large-scale research projects, which students are informed that they do not academic researchers, who rely too involved multi-level modelling of VCE have to do any special preparation for often on statistical analyses alone, and results over the past ten years. Key the GAT, as the basic writing and frequently ‘got it wrong’ (Richardson, findings included: reasoning skills being assessed have 2001). In Victoria’s League Tables,

Research Conference 2005 112 published from 1996 to 2000, there school’ analyses had to precede ‘across- For the two-level (students in classes) were blatant examples of schools being schools’ analyses, to ensure fair variance components modelling for the incorrectly highlighted by the media as evaluation of school performance. In ‘within-schools’ analyses, the six ‘top performers’ or unfairly labelled as 2001, the ‘within-school’ analyses were explanatory variables used to predict ‘failing schools’ based on system level developed and trialled. Over the next the student’s VCE marks were: data analyses with ability ‘supposedly’ three years, 16 schools (2002 VCE • Three student ability (GAT) taken into account. Detailed examination data), 90 schools (2003 VCE data), and measures:Written Communication of these data revealed the problems 105 schools (2004 VCE data) voluntarily (GAT c), with such gross school rankings, when, participated in this research, that is, they Mathematics/Science/Technology regardless of the standard achieved, half effectively funded it. (GAT m) and the schools in the State had to be below One of the problems highlighted in the Arts/Humanities/Social Sciences the median, by definition; and one school ‘across-schools’ research involved the (GAT h) has to be ‘bottom of the ladder’ each use of the VCAA Study Score as the year.This ‘so-called’ accountability rating • Class mean ability (the mean ability achievement mark, since it is the rarely affected elite schools in wealthy for all students, with student ability student’s rank, relative to all other VCE suburbs, and impacted most negatively calculated as the average (AvGAT), students within each Study. However, on schools in poorer areas (Richardson, of the three GATs). this rank is unsuitable for comparisons 2002). • Gender (males = 0, females = 1) across an individual student’s VCE The key difference between the system- Studies, and when comparing class and • Year level (Year 12 students = 0, level VCE data provision to schools subject achievement within schools. In Year 11 students = 1). since 1994 and this in-depth research recognition of this,The Victorian Tertiary Effect sizes, and proportion of variance project was the latter’s inclusion of Admissions Centre (VTAC) transforms at student and class levels, were qualitative research undertaken with VCE Study Score ranks to marks for calculated for each school, which schools (Richardson, 2000b). Staff calculation of students’ ENTER included government, Catholic, and feedback and suggestions were (Equivalent National Tertiary Entrance Independent schools, small and large integrated into subsequent data analyses Rank). As students and schools in schools, single-sex and co-educational each year, and macros and a software Victoria are not given students’ final schools, and urban, regional and rural package were developed to display the VCE marks, only their Study Score schools. Both student and class residuals data so as to facilitate data display and ranks, the Scaling Guide that VTAC were examined for both typical and interpretation.The more informative, publishes to schools each year was atypical patterns, where the residual visual presentation of both raw and used to calculate the student’s VCE was the difference between the ability-adjusted data in this project was marks for this research.The difference predicted mark, based on student ability preferred by school staff to the residual between VCAA Study Scores (marks) (3 GATs), gender, year level, and class plots provided at system level, because and VTAC Scaled Scores (ranks) is mean ability (AvGAT), and the achieved teachers could verify the raw data, and illustrated and discussed further in the VCE mark. Raw and ability-adjusted explain some of the patterns in the section on class level analyses below. VCE results were summarised in graphs analysed results. In response to the and tables, and provided to each school positive feedback from schools, this The second problem identified in the on a CD.To ensure that the analyses doctoral research developed into an initial research was that year level was a and the interpretations were statistically independent Data Interpretation key variable predicting student sound and educationally meaningful, the Service, now operated by ACER. performance in some Studies, and researcher and practitioners discussed differentially so in some schools.This and debated the results in a The research background 2 factor needed to be included in the professional learning seminar.This ‘Within-school analyses’ modelling. Students typically complete allowed for the rich contextual VCE Units 1 and 2 in Year 11, and VCE knowledge available within the school The comprehensive verification of the Units 3 and 4 in Year 12, although some to be taken into account in evaluating data and review of the multi-level students study one or more VCE Units student, class and school performance. analyses indicated that in-depth,‘within- 3 and 4 in Year 11.

Using Data to Support Learning 113 Broad consistency in effect sizes and patterns, and the depth of these ‘within- associated with higher VTAC marks, school patterns were found for the school’ analyses and data interpretations, r = 0.6435); and the variation between 2002-2004 ‘within-schools’ research and to be examined. In all graphs, whether at students at each ability point. For the 1994-2000 ‘across-schools’ student, class or school levels, individual example, around the State mean ability population-level research, especially in student ability (AvGAT) was plotted on level (AvGAT ~ 20), the range for terms of the average magnitude of the X-axis, and achievement (VTAC individual students in this school in 2004 variance explained – between 5-15% mark) on the Y-axis. Drop-down menus varied from 13 to 43 VTAC marks. at the second level (classes for the beside each graph enabled results for This was the typical range across VCE ‘within-schools’ research; and schools particular students, classes or subjects to ability and achievement data both within for the ‘across-schools’ research). It is be highlighted as white circles, against schools and across schools, except for not possible in this paper to discuss all the background of dark diamonds schools with fee-paying overseas results of these two large-scale projects, displaying the school’s VCE results for students, where a distinctly different however different aspects of the the year. For example, in Figure 1, the pattern was evident. Further examination research have been presented in diamonds pinpoint the ability and of these schools led to the understanding greater detail at seminars and national achievement scores for every student that two separate analyses were needed and international Conferences (Hong completing their VCE in this school in for such schools, to avoid the distortion Kong, 2000a; Melbourne, 2000; 2004.The white intersecting lines on the that results when ability-adjusted data for Denmark, 2001; Sydney, 2003; graph indicate State means (AvGAT = overseas students are included in the Melbourne, 2003; Sydney, 2004). A 20,VTAC Study mark = 30), and the VCE data analyses. series of seminars will be held after the diagonal line crossing the diamonds is Conference, where more in-depth the school’s regression line. No student Figure 1 demonstrates that while ability displays and explanations of the or class has been selected here (the plays a large part in students’ final research findings will be presented. blank space in the legend beside the academic achievement, the range in white circle in the graph’s title). marks at every level of ability is considerable, thus ability alone does not Student ability patterns Noteworthy features are the general determine final achievement.The multi- pattern of achievement increasing with Some patterns from one school are level modelling revealed that in this student ability (higher GAT scores outlined below to enable indicative school in 2004, individual student ability as measured by performance on the three GATs accounted for 41% of the variation in student marks. Class mean ability, gender and year level explained around 3% of student differences in VCE achievement.Thus, a total of 44% of the variation in student’s scores was explained by the six factors modelled, with differences between classes accounting for 9% of the variance.The remaining 47% of unexplained variance in VCE results in this school was associated with factors not measured in these analyses.This unaccounted-for variation was what the discussions with staff in each school were intended to uncover, and were usually attributed to student effort, motivation and aspirations, teacher skill, school and Figure 1 Scatterplot of Students’ Mean GAT Score (Ability), home resources. Plotted against Students’VTAC Marks (Achievement).

Research Conference 2005 114 Figure 2 (left). Scatterplot of Students’ Ability and Achievement, with Boys Highlighted (white circles). Figure 3 (right). Scatterplot of Students’ Ability and Achievement with Year 11 Students Highlighted.

From this graph of overall school VCE (Figure 2), and Year 11 VCE students ‘Which boys are performing better than achievement (Figure 1), individual averaged almost two marks more than similar ability boys within the school?’ students could be identified using a Year 12 VCE students (Figure 3). In and ‘What factors are influencing some ‘Who is …?’ button.This allowed these two graphs, the dashed line low ability boys to perform well, while obvious outliers in the data to be represents the school mean other low ability boys do poorly?’These immediately identified, and their ability performance for boys (Figure 2), and two questions can be re-worded for and achievement data further examined the school mean achievement for Year girls, as the same situation applies – at by reference to the tables and graph on 11 students (Figure 3). every ability level, there are groups of girls achieving both above and below the Student page (see Figure 8 below) At each ability level (Figure 2), there are their predicted score.The relevant focus of the CD. In addition, displays of school boys achieving VCE marks higher (white for teachers and schools is ‘which boys’ gender and year level patterns selected circles above the regression line) and and ‘which girls’ were under-performing, from the following eight options:Year lower (white circles below the when evaluating under-performance. 11 students,Year 12 students, males, regression line) than the school’s This research has more potential than females,Year 11 males,Year 11 females, average across all Studies.This graph the Federal government’s response to Year 12 males,Year 12 females, could be indicates how misleading the gross the ‘boys’ under-performance’ problem highlighted on the Figure 1 graph. statement that ‘girls are outperforming than male-only scholarships, as it boys in VCE’ is, given the range in marks enables positive examples of low ability Gender and year level for boys at each level of ability.The boys (and girls) who are performing typical pattern was that bright boys patterns well to be identified, and the factors achieved as well, if not better than that contributed to their success can be Similar patterns were evident in terms bright girls, but more lower-ability boys evaluated and shared with all schools. of the range for both ability and performed worse than lower-ability achievement data when gender and girls.The mean gender effect at class The same questions can be asked year level were examined in the ‘within- and school levels, in both the ‘within- regarding year level and other variables school’ analyses.The effect size for schools’ and ‘across-schools’ research, known to affect student achievement: gender in the variance components was due to the poor results of some of for example,‘Which Year 11 students, modelling for this school’s VCE 2004 the lower ability boys, not because all highlighted as white circles in Figure 3, scores was 1.2 and the effect size for boys are performing worse than all girls. performed better (or worse) than year level was 1.7.That is, girls averaged similar ability peers?’ Careful The educationally more informative just over one mark higher than boys examination of this graph, and questions in terms of gender ought be:

Using Data to Support Learning 115 Figure 4 (left). Class Mean GAT and Class Mean VCAA Score. Figure 5 (right). Class Mean GAT and Class Mean VTAC Score discussions with students and teachers class mean GAT scores are plotted those below the line are not to accurately discern the reasons against class mean VCAA Study Scores; performing as well as predicted, based associated with each student’s and in Figure 5, class mean GAT scores on the variables adjusted for in the performance, enabled positive action to are plotted against class mean VTAC modelling, and on the overall pattern in be taken where deemed necessary, for Scaled Scores, for the selected school. the school’s data. A range of these students are the school’s current performance is evident at each ability The pattern in the data in Figure 4 is of Year 12’s. level. Clearly, there are factors other concern, with some low ability classes than ability which influence class achieving higher class mean VCAA achievement, and hence the results for Class-level analyses scores than some high ability classes in all students in each class were this school in 2004.The correlation When class-level data were first examined, along with the patterns between these two class level variables examined in 2001, the need for the within and across teachers for all VCE was zero, indicating no discernible ‘ability-adjusted’ analyses to be classes in the school. conducted on the student’s mark relationship between class average GAT scores and class average VCAA Study (VTAC Scaled Score), not the relative Class and teacher State rank (VCAA Study Score) within Scores (ranks). patterns each Study, became evident.The However, in Figure 5, when VCE marks following two graphs illustrate this at (class mean VTAC Scaled Scores) were The following two class graphs (Figures class level, where its effect is strongest; plotted against ability (class average 6 & 7) provide examples of VCE results and were prepared to assist teachers to GAT scores), the expected pattern for for two English classes (A and B), taught understand the difference between the educational data was found (r = 0.660), by the same teacher (Teacher 5) in the VCAA Study Score and the VTAC with higher ability classes generally same school in the same year. Scaled Score.This pattern occurred also achieving higher marks than lower In teacher 5’s first English class (A), at individual student level, although to a ability classes. In Figure 5, classes more students were above than below variable degree, as students’ marks are (diamonds) above the regression line the school’s regression line, while in differentially affected by the impact of are interpreted as performing better Class (B), the reverse pattern was VTAC’s scaling of Studies. In Figure 4, than expected within this school, while

Research Conference 2005 116 Figure 6 (left). English Teacher 5, Class A Figure 7 (right). English Teacher 5, Class B found.Yet in both classes, most students • Who decides if this difference is 6 & 7). Data for a high performing were on or around the school’s mean educationally meaningful? student in Class A (Student 258) in line, that is, they performed as expected • What process determines where Figure 8, and a low performing student given their ability. In class B, however, the line of acceptable ability- in English Class B (Figure 9) are now two students were ten or more marks adjusted performance is drawn, and examined. As can be seen, both below the regression line, noticeably how is this authenticated? students performed in similar fashion in dropping below the majority of the English as they did in their other classes. school’s 2004 VCE cohort (densely Detailed examination and informed Student 258 (AvGAT = 18.3) in English clustered diamonds). In particular, discussion of the data were necessary Class A, with a mark of 34, contributed Student 429, identified as the third when evaluating the factors believed to to Class A’s positive residual (1.6), and highest in terms of ability (AvGAT = influence overall class achievement. generally achieved at or above expected 23.7), with a mark of only 16, strongly More often, the effect of individuals or level in all Studies. Student 429 (Figure affected the mean achievement in small groups of students within a class 9) generally achieved less than Teacher 5’s second English class. appeared to have a greater effect than an individual teacher on the class mean expected, relative to other students of Class A (Figure 6) had a positive class achievement. Frequently a student who similar ability in this school in 2004, and residual (1.6) and Class B (Figure 7) a achieved high marks in one class also contributed to the negative residual for negative class residual (–1.4).This is scored positively in their other Studies, his English class.This leads to the interpreted as the mean English mark and vice versa. question:“To what extent can the for Class A was 1.6 marks higher than English teacher be held responsible for predicted, and Class B’s mean was 1.4 Student level analyses the English marks of these two marks lower than expected, given the students?”These data, of themselves, do six factors adjusted for in the multi-level Note that in Figures 8 & 9, the white not and can not tell us whether Teacher modelling. Obvious questions are: circles illustrate an individual student’s 5 was a good, average or poor teacher performance, set against the results for of English in either class. • ‘Why the difference?’ all students (diamonds).The English School staff generally attribute student • In what ways is the teacher mark for each student is shown as a effort and interest, or lack thereof, as responsible for the three mark white square to assist with comparison the main explanatory factors in student difference in the two English and location of these students in their performance. Students with positive class means? respective English class graphs (Figures

Using Data to Support Learning 117 Figure 8 (left). VCE results for student 258, English Class A Teacher 5 (highlighted) Figure 9 (right). VCE results for student 429, English Class B Teacher 5 (highlighted) residuals were described as those who not automatically associate the class ongoing learning, and capacity to had high aspirations, gave appropriate residual with the measure of the acknowledge both ‘good’ lessons and time to the subject regularly throughout teacher’s effectiveness, as academics and ‘difficult’ lessons.These teachers were the year, and took notice of teacher system level staff tend to do. not paraded as ‘perfect’ teachers or persons, but as genuine educators, who feedback and instructions for improving The fact is that few researchers have loved learning, had strong discipline their work. Students with negative actually analysed data that could be knowledge and love of their subject residuals were usually said to have low authentically considered to have material, and were able to communicate motivation, lack of home support for included valid measures on which well and sustain positive relationships learning; and in some cases, illness and teacher effects could be calculated. with students. Note that no evidence trauma were relevant factors. Until such measures are defined and was provided in the sense of these Results of this research (Richardson, gathered, claims of teacher effect sizes, qualities being measured as they were 2004b) indicate that the class residual calculated from multi-level models of merely observations of, and attributions ought not be referred to as the teacher students in schools, or even students in made by, their peers within the school. residual. Even reference to variance at classes, must be more closely examined. As a consequence of these data- this level as the class/teacher residual, However, it is important to note that, in informed discussions, many teachers needs caution, given the unit of analysis schools where such analyses were independently selected areas of focus is merely aggregated student level data, conducted over several years, teachers for themselves for their current VCE not specifically measured teacher or whose class residuals were strongly class variables. In multi-level modelling positive year after year were often the teaching – more examination practice, research, the class residual is simply the ones that colleagues named as ‘high- greater monitoring of student written difference between the adjusted mean performing’ teachers.This was work in class throughout the year. for all students in a class, compared substantiated with detailed reference to Examples of instances where a negative with the adjusted mean for all other the individual teacher’s behaviours in class residual was attributed to a ‘poor’ classes in the school.Yet in the vast terms of curriculum contribution, teacher were rare, but some class majority of cases in this research, assessment practice, student feedback, patterns did generate concern. Further negative class-level residuals were and collaboration within the school. investigation into their students’ clearly influenced by factors associated Other characteristics of ‘top teachers’ performance in their other Studies was with a small group of students. acknowledged by VCE staff in this undertaken in discussions with the Principals and senior staff in schools did research were openness to their own, teacher, as was consideration of

Research Conference 2005 118 contextual factors that may have accounted for the less-than-expected achievement. In some schools, additional support was given in terms of formal and informal mentoring. Some teachers were encouraged to develop contact with subject networks groups for improved access to curriculum and assessment resources and information. School level analyses In Figure 10, all class residuals (dark diamonds) for this school in 2005 are plotted in rank order from lowest to highest, with all English KLA classes (white diamonds) selected.The bounded line around each diamond indicates the 95% Confidence Interval for each class residual. As is the case in all schools, the majority of class residuals in the school are within ±2 marks of their expected achievement on the vertical scale. Residuals for Teacher 5’s two English Figure 10 Ranked Class Residuals in this School, 2004 (dark diamonds), classes (Class A’s residual 1.6, and Class English Class Residuals (white diamonds), and Teacher 5’s two English Classes (squares). B’s residual – 1.4) are highlighted as grey ranked residual plots, without any could identify instances where the squares (Figure 10). Because their reference to the multi-variate, multi- student and class residuals were respective confidence intervals do not level factors influencing these results, let inaccurate, and make appropriate overlap on this class residual plot, alone acknowledgement of the adjustments in their evaluation of their statisticians consider that there is a unmeasured (and possibly school’s performance. statistically significant difference between unmeasurable) factors.Too often, these two classes, and some then refer to negative subject or class residuals are this as the ‘teacher effect’. However, Within schools misrepresented as the teacher effect, detailed examination of individual research, years 7-11 simply because the patterns across students’ results in Teacher 5’s two English students are hidden. Only when lower Staff in schools who had access to this classes revealed that the difference was level (student) data are examined is this level of detailed student and class data largely associated with performance of problem avoided. A more detailed quickly recognised what they described several students in each class. discussion and interpretation of as ‘the value of a good ability measure’ The data and discussion associated with patterns in residual plots will be to provide them with value-added Figure 10 provides one example of the presented in a series of seminars to be information on their students’ academic misinterpretation that can occur when held at ACER later this year. performance (Richardson, 2002, 2003a). statistical analyses alone are used to In some schools, senior staff set about Principals and teachers preferred estimate school, subject and teacher obtaining an independent measure of scatterplots (Figures 1–9) to residual performance. student ability at the key learning stages plots (Figure 10), when examining their – entry to secondary school, and in At system-level, and in ‘League Table’ school’s data, as the former better Year 10 when there was a focus on summaries, so-called ‘failing schools’ and illustrated the meaning of student and work experience, careers advice and ‘top schools’ are identified from such class residuals.Within the school, staff VCE subject choices.

Using Data to Support Learning 119 Research into practice Urrbrae Agricultural High School (UAHS) is a specialist agricultural school located in suburban Adelaide. The school has 1000 students, all of whom are selected to enter the school. As with many schools, one of our goals is to achieve excellent learning outcomes, in particular, as expressed in our Strategic Plan:‘To achieve excellent learning outcomes which allow our graduates to be skilled contributors to our community’.This generated debate within our school about an operational definition of excellent learning outcomes. Subsequently, consideration was also given to the second strategic goal:‘To achieve growth of social capital Figure 11 Typical Scatterplot of Student Ability and Achievement for a community that is socially and (based on teacher’s semester grades/marks). environmentally sustainable’. Defining excellence in terms of tertiary Concerns were often raised about the highlighting the reality that, when writing education entry scores was not relationship between the results reports for students, teachers’ grades appropriate for or relevant to many of students receive on school reports, and are based on both observed behaviours our students, and also left us to work their academic performance as and examined subject material over the with data available only after students measured on external assessments. semester. Some teachers give positive had left the school.We wanted the Figure 11 illustrates the typical pattern grades to ‘reward’ students for effort, capacity to monitor progress of all found when internal school assessments and to encourage lower ability students. students towards the goals as they (in this case, semester report grades) Figure 11 reflects the high variability moved through the school. Debate over are plotted against external measures amongst teachers when assessing the meaning of excellence led to a (in this case, an intake ability test). student achievement, sometimes found belief that, for us, it would be for the White circles represent one school’s even when moderation procedures are school to make a positive impact on Year 8 Mathematics results for in place in the school. student achievement. On an individual Semester 1, 2003, set against all Year 8 basis, excellence was defined in relation Many schools are developing processes students’ subject results (diamonds), to the student’s starting point.To to support their teachers in monitoring with teachers’ grades converted to monitor achievement in each learning and improving their assessment and marks (A+ = 20, A = 19, A– = 18, area, we needed a measure of student reporting practices, and some schools B+ = 17, etc.). ability.The proposal to collect baseline have already begun this venture The diamonds and circles on the Y-axis data was controversial in the school, towards becoming a data-informed (vertical line at zero ability score) with some fears raised from past school (see poster displays at this represent students not assessed on the memories or myths of IQ tests. Conference for examples). Sally ability measure on entry to the school, However, staff members recognised that Paterson now outlines the way her and the missing data on the X-axis our school not only had goals relating to school embarked on the task of ‘using indicate students no longer at the the quality of academic learning, but also data to support learning’. school. Note the lack of correlation to the development of social capital, as between ability and teacher grades, stated above, and gathering a multi-

Research Conference 2005 120 dimensional student profile would give the factors that predict student most cases, however, it was not that us the opportunity to monitor all achievement as being individual ability assessment tasks had to be created and aspects of each student’s development. (50%), the influence of school, home marked which was new, but the and peers (20%), and the quality of requirement for the content and tasks A commitment was made to establish a teaching (30%).To allow us to focus on to be common between classes database with a comprehensive array of the impact we can have, as school staff, studying the same course material. information gathered for each student, we need to be clear about the factors There was some additional work for including a measure of ability in four we cannot influence, such as the the teachers to participate in domains – verbal reasoning, numerical student’s ability on entry to the school. moderation of the results. reasoning, abstract reasoning and visual spatial reasoning, a measure of students’ To monitor our progress in achieving The Design and Technology faculty thinking style/learning preference, and excellent outcomes, it was necessary to drew up a task that Year 8 students their self-reported attitudes to learning ensure that teachers were using at least completed at the end of their semester and to the school’s focus areas.These some common assessment tasks, of study in this area. It was designed to attitudes were expressed on a school- completed individually and under assess all of the desired outcomes of developed survey.Teachers also supervision.Teachers were involved in the course and was completed across a collected an example of student writing moderating these assessments. In some number of lessons.This model conducted in class throughout the first learning areas, new assessment tasks promoted discussion among the semester, to provide a baseline against were devised and in other areas, curriculum coordinator group and which development in students’ written existing ones fitted the appropriate inspired interest within other learning expression could be mapped. criteria. In all cases, the assessments areas in devising similar tasks. were referenced to Level 4 outcomes Mathematics and Science already used Research has shown that what the of the State curriculum framework. common tests and these results were student brings to the learning situation Discussion of the need for, and the correlated against the ability data. predicts 50% of their achievement.The structure of, these assessment tasks led collection of baseline ability data gave This project is still at a very early to an interesting professional debate. In the school the opportunity to identify stage; however, an examination of some situations, teachers raised the starting point for each student. Year 8 Mathematics data in Figure 12 questions about the increased workload Some research (Hattie, 2003) described (r = 0.656) in this school in 2005 this change in assessment required. In

Figure 12 Scatterplot of Student Ability and Year Level Common Subject Assessment, 2005

Using Data to Support Learning 121 indicates a pattern closer to the the profile, we did not want to foster is only our Year 8 students who are expected relationship between ability views such as:‘My child is top (or involved. As they move through the and achievement data, than for example bottom) of the class’.The discussion we school, achievement data will continue that shown in Figure 11 (Mathematics, want to have with parents and students to be correlated against the intake Year 8, 2003). is about how well the student is ability data.There are several questions achieving in relation to their own as yet unanswered: How will we best Once data from each learning area is ability-predicted achievement. For the represent a student’s Year 9 correlated against the baseline data, attitudinal data, we have offered to achievement levels against their Year 8 teachers are asked to examine and report this to parents in a face-to-face levels? Is there a meaningful way to do reflect upon individual and group discussion with a staff member to that? Will the results of each learning variations from the ability-predicted fully flesh out the implications of area be correlated against each ability results. In some cases, there may be this information. strand separately, or only against the clear, non-school explanations for general reasoning or some combination under- or over-achievement. Most The intention was to ensure of these? research indicates that the more likely consistency in achievement reporting explanation for those variations is a across the year level, as well as to build The point in conducting this research is teacher effect.Teachers are being capacity for monitoring each student’s to lead to our teaching processes being supported to develop the expertise to progress throughout their time in the data-informed and as a consequence, analyse the data and work towards school in ability-adjusted terms, more effective.We believe the focus on finding and addressing the possible commonly referred to as the value- the student’s own real learning progress reasons for the variation in added contribution of the school to will contribute to improved performance, from that expected given student academic achievement. relationships between the teacher, their ability. student and parents. We did not meet our goal of reporting As an example, some of our teachers to parents at the end of the semester. see one cause of under-achievement in This was because it became clear that Conclusion our senior school as student considerable professional discussion still Schools have extensive data – as participation in vocational education needed to occur for the teaching staff student records and reports, in staff programs, which take the students out to feel comfortable with their capacity offices, administration areas and of their normally scheduled classes.This to answer questions from parents and archives, and of course, the vital issue will not be able to be investigated students regarding these data.There are information carried in teachers’ heads. with our new database for some years other sensitivities as well. For example, Some secondary schools have intake (until this year’s Year 8 students reach if all results are available to staff, how data or scholarship results, but few Year 11 and 12); however it is an will performance of individual learning schools have gathered the example of the enquiry that will be areas be seen by staff of other learning comprehensive data required for possible, as a result of our commitment areas? If we continue to send home effective monitoring of student to developing a longitudinal database. student grades as well as the correlated achievement, as identified in the ability and achievement data, will Another issue of debate has been the research discussed above. parents question results that may reporting of the correlated data appear anomalous? Some teachers are A measure of student ability, against (achievement to ability). From the finding the stated expectation that they which to evaluate student attainment outset, parents were informed that the can influence the quality of students’ via common tasks and moderated correlation of their child’s achievement learning outcomes to be, at least to subject assessment for each year level, is with their child’s ability would be some degree, quite confronting. essential to provide schools with the reported to them.The intention was to capacity for ability-adjusted monitoring do this at the end of each semester. As we proceed with this project, of each student’s learning progress, at There was never any intention of opportunities and questions continue regular intervals.The explicit purpose reporting the raw data from the to arise. Our commitment is to run this for developing a school database is to student profile. For the ability section of project for five years at least. In 2005, it support learning at all levels within the

Research Conference 2005 122 school, so that trends over time can be factors that affect student learning, both type of data analyses and displays of identified, with early detection of issues positively and negatively then adjust VCE results similar to that displayed in leading to remediation and extension their practice accordingly.This research Figure 10, and currently used at system indicated that school leaders and level in Victoria. However, managing the school’s teachers need considerable time to academic database so that appropriate Improved collaboration between examine their value-added data.The information can be readily accessed researchers and practitioners can lead graphs and tables of data provided on when needed requires time, skill and to more truly data-informed analyses, if their CD encouraged staff to reflect on financial commitment. Planning is the voices of all stakeholders are the student and class patterns, and to essential to ensure that all data are represented, and not dominated by discuss and debate their attributions for formatted and integrated, as students, system-level statistical analyses that are the factors impacting student and class teachers, year level co-ordinators, not independently verified. It is possible scores.The capacity to highlight each curriculum and welfare staff, to conduct more equitable evaluation student within the school, examine administration and management all of student and school performance, performance in depth at individual require different analyses, report both across and within schools student, class and subject levels, as formats, and levels of access. In addition, (Richardson, 2004a).This VCE research developed in this research, was new to, security, regular updating and archiving found support for Rowe’s (2003) and positively received by, school of information also require attention. statement that ‘All too frequently leadership teams. It is hoped that this systems, schools and teachers (my To begin the task of value-added research can be extended in the future emphasis) have lacked credible monitoring of student and school to include student feedback, as it would information regarding the magnitude of performance, the following steps are be of interest to record the factors their relative contributions to recommended: students considered were major performance and effectiveness’. Greater Step 1. Enter all current educational influences on their results, and whether effort needs to be focused on research data available in the school in they believed they had achieved to within schools where it is possible to relevant spreadsheets in the their potential. validly identify the factors influencing school’s academic database. However, the extensive, multi-level students’ achievements for both boys Step 2. Arrange for appropriate factors that affect student learning have and girls and for low, average and high analyses to be conducted, with yet to be definitively identified and ability students. output formatted to ensure modelled. For this to occur, a re- School effectiveness research (SER) and user-friendly access to, and allocation of resource provision at system-level analyses still over- interpretation of, all tables, system level is needed, so that valid and emphasise teacher effectiveness, and fail graphs and summary reliable curriculum measures and to take into account the multi-level information. assessment protocols are available in all structure within which teaching and Step 3. Use this school database to schools. Clearly defined subject learning operate. Student responsibility initiate informed discussion and knowledge and skills, in appropriate for learning (at senior secondary) and debate around the following developmental stages with common system-level accountability need questions: metric assessment scales, would enable appropriate attention so that resources teachers to report valid learning gains are diverted to research that has the • Do these data provide us with for each student. potential to identify and verify sources answers to the questions we have of variation at student, class and teacher been asking? When contextual information and data- informed interpretation are lacking at levels within and across schools. • What questions remain the level at which the data were unanswered, and what further data We do not yet have appropriate gathered, class and subject residuals are are needed to respond to these? measures of the verifiable teacher often misrepresented as evidence of behaviours explicitly linked to student • What new questions have emerged? teacher performance. Principals and achievement that can be validly The challenge for school staff when teachers are rightfully concerned about reported in terms of effect size. Claims reviewing their data is to identify the their performance being judged by the regarding the proportion of variance

Using Data to Support Learning 123 explained at the so-called ‘teacher’ or The value added by the school is usually Presentation at the 14th International ‘class/teacher’ level were not supported estimated in terms of student and group Congress for School Effectiveness and in this detailed, evidence-based research performance above that of their peers. Improvement (ICSEI) Conference, (Richardson, 2003b, 2004b), where Yet it is rare for all academic Copenhagen, January 3-6, 2001. interpretations were validated in characteristics such as ability, past Hill, P.W. & Rowe, K. J. (1996). Multilevel discussions with senior staff within performance in the subject area, modelling in school effectiveness schools.This does not mean that teaching and learning strategies, and research. School Effectiveness and teachers and the quality of teaching are contextual variables such as gender and School Improvement, 7 (1), 1-34. not vitally important influences on SES at student and school levels to be student achievement. Just as we rarely comprehensively measured. This level of Hill, P.W. & Rowe, K. J. (1998). Modelling have measures of student motivation data is just not available yet in Australia. student progress in studies of and aspiration, time on task and degree educational effectiveness. School Of all the States in Australia, because of of private tutoring, illness and personal Effectiveness and School Improvement, 9 the ability and achievement measures trauma, all of which affect student (3), 310-333. performance, so too, we do not yet collected for the VCE,Victoria has the Mortimore, P., Sammons, P., Stoll, L. have the comprehensive data needed greatest potential to take the lead in Lewis, D. & Ecob, R. (1988). School to identify the teacher behaviours and developing research to identify positive matters: the junior years. In P. attitudes that positively impact student teaching and student learning effects Mortimore, Effective schools: current performance across all ability levels. (Richardson, 2004a). One way that this could be achieved is, for example, if the impact and future potential. Director's For more authentic evaluation of Hay-McBer (2000) Teacher Effectiveness Inaugural Lecture, Institute of teacher performance, and calculation of variables were measured and integrated Education, University of London. genuine teacher effects, valid measures of with the type of ‘within-school’ analyses February 7, 1995. teaching knowledge, skills and described above. More than 100 Richardson, C.A. (2000a). Stability and behaviours demonstrated to make a Victorian schools have already consistency in students’VCE positive difference to student demonstrated their commitment to achievements, 1994-1998. Presentation achievement, are needed.While some ‘using data to improve learning’. It is at the 13th International Congress for research quotes ‘characteristics of now time for both the Federal and School Effectiveness and Improvement effective teachers’ (Sammons, 1999) no State Governments to collegially (ICSEI):“Global Networking for definitive studies have measured these support and extend this research. Quality Education”, Hong Kong, variables over time.The Hay-McBer January 4-8, 2000. (2000) research on teacher References effectiveness provides one way of Richardson, C.A. (2000b). Positive teacher conceptualising a matrix of factors that Hattie, J. A. (2003). Teachers make a effects at VCE level: Evidence, observation could be modelled to further our difference:What is the research and reflections. Presentation at the Annual understanding in this area. evidence? The Australian Council for Postgraduate Students’ Conference, Educational Research, (ACER) Faculty of Education,The University of Many research reports aggregate one Conference, 2003, Building Teacher Melbourne, December 2, 2000. or more student measures to create Quality:What does the research tell second-level variables, then discuss this Richardson, C.A. (2001). The Role of the us? Melbourne, October, 19-21, 2003. aggregated group variance, be it General Achievement Test (GAT) in VCE class/subject or school-level, in terms of Hay-McBer. (2000). Research into results, 1994-2000. Presentation at the the ‘class/teacher effect’, sometimes the Teacher Effectiveness. A Model of Annual Postgraduate Students’ ‘teacher effect’. Results from ‘within- Teacher Effectiveness. Research Conference, Faculty of Education,The school’ analyses in this research Report, No 216. DfEE, U.K. University of Melbourne, December 2, indicated that even when student data Hill, P.W. & Richardson, C. A. (2001). The 2001. are aggregated to class level (or subject role of contextual effects in predicting Richardson, C.A. (2002). The value of level) it is misleading to name this as the academic progress of students in value-added analyses of VCE results. the ‘teacher effect’. Hong Kong secondary schools.

Research Conference 2005 124 Presentation at IARTV Invitational effectiveness research [Special Issue]. Seminar, Melbourne. International Journal of Educational Research, 13, 691-706. Richardson, C.A. (2003a). The data- informed school: Positive use of student Teese, R., & Polesel, J. (2003). information for teachers. Presentation at Undemocratic schooling; Equity and the Australian Council for Educational equality in mass secondary education Research (ACER), November, 2003. in Australia. Melbourne,VIC: Melbourne University Press. Richardson, C.A. (2003b). Fairer methods of assessment:Value-added analyses of final secondary results. Presentation at the 15th International Congress for School Effectiveness and Improvement( ICSEI), Sydney, Australia, January, 2003. Richardson, C.A. (2004a). Effectiveness and Quality Assurance in Education. Presentation to the Board,Victorian Schools Innovation Commission, 22nd July, 2004 Melbourne. Richardson, C.A. (2004b). Do teachers make a difference at VCE? Exploring empirical evidence and reflecting on findings. Presentation to the 9th Annual Roundtable: Assessing Assessment, 7-9 November, 2004, Crowne Plaza Sydney. Rowe, K.J. (1999). VCE Data Project: 1994-1998. Concepts, Issues, Directions and Specifications. Melbourne.Victorian Board of Studies. Rowe, K. J. (2003).The importance of Teacher Quality as a key determinant of students’ experiences and outcomes of schooling. ACER Conference, Building Teacher Quality: What does the research tell us? Melbourne, 19-21 October, 2003. Sammons, P.(1999). School Effectiveness. Coming of age in the twenty-first century. Swets and Zeitlinger, B.V., Lisse,The Netherlands. Scheerens, J. & Creemers, B.P.M.(1989). Conceptualizing school effectiveness. In Developments in school

Using Data to Support Learning 125 Using online assessment to inform teaching and learning in primary and secondary classrooms

Abstract • users with the ability to monitor student progress over time. In the 1980’s there was a conscious effort around Australia and in many This presentation will describe other countries around the world to instrument and its underlying rationale, shift the focus in assessment from show how the feedback can be used to notions of passing and failing to those inform teaching and learning, and of monitoring growth; from comparing discuss ways that the instrument and students against each other to building the feedback might be developed Jim Tognolini up an image of what it is that students further in the future to ensure that the Australian Council for Educational Research know and can do at particular stages in advantages that accrue from their development; and, from collecting information technology are being fully Professor Tognolini is the Research Director marks to summarise performance to harnessed in an attempt to continually (System and School Testing) and General Manager providing students and teachers with improve learning. of ACER’s Sydney office. He has extensive information from assessment activities experience in the areas of educational measurement, psychometrics, planning, survey that can be used to help diagnose research and quantitative research methodology. potential weaknesses and strengths and For over 20 years he has provided psychometric lead to improved learning. advice, research and project management, Government testing program management and The Australian Council for Educational co-ordination. He has been a consultant to Research (ACER) has recently Victorian Board of Studies – Computer Adaptive developed an online assessment Testing Project, the Ministry of Education - WA instrument that takes advantage of the MSE, the Secondary Education Authority of WA, latest advances in psychometric theory Board of Studies, NSW, Department of Education to provide schools and students with a and Training, NSW and the Victorian Curriculum and Assessment Authority powerful tool to support learning at school and at home. Professor Tognolini has extensive experience in liaising with representatives at all levels of government and education both within Australia The instrument and overseas on assessment and research provides related issues.

Professor Tognolini has just produced the Report • an online testing program in of the Consultancy to Research International Best mathematics and English for the Practice in Outcomes-based Assessment Related equivalent of Years 3 to 10; to Post-Compulsory Education in conjunction • tests that assess the generic skills with Professor David Andrich and Professor Sam Ball. that underpin learning in the subject areas of English and mathematics; • tests that are tailored to the individual needs of children; • instantaneous feedback using student reports and progress maps to show where a child is located on a continuum of performance that is linked directly to learning in the classroom; and,

Research Conference 2005 126 Turning data into information that improves learning:The WA experience

Introduction within a powerful and interrelated socio-political system.There are a This paper will look at some examples number of potential enablers and of the way in which the Western blockers operating in the system that Australian Department of Education need to also be considered. For this and Training is presenting student reason, the resources that we have performance data and transforming it developed to bring information to into information to assist teachers to teachers has been integrated into a modify their teaching practices and David Axworthy whole of system approach which has improve the learning of their students. implications for the teacher, principal, Department of Education and Training, John Hattie’s work has shown that of district director and central executive of Western Australia those variables over which an education the organisation. Locally we refer to the system can have some control, it is the whole paradigm as Assessment literacy David Axworthy is Director of Standards and behaviour of teachers that has the or Assessment for learning.This approach Accountability at the Western Australian has included: Department of Education and Training and is greatest impact on student learning responsible for the policy on school accountability (Hattie, 2003). It is because of this • Policy; and for the system of educational measurement, research finding and its congruence • Resources; reporting and school review. He has extensive with the intuitive knowledge of those experience in all areas of the education system who work in schools that we need to • Professional learning; and from classroom teacher to senior executive and is • Consultancy support. known for his systems approach to organisational ensure that student performance data change and reform. His background is in reaches teachers in a way that informs In keeping with the theme of the educational psychology and measurement and he their approach to teaching practices. conference, this paper focuses on the has undertaken work as a consultant to education authorities in England and to the Medical While Blanchard’s proposition resources component; more specifically Research Council (MRC) in Scotland. (Blanchard et al., 1999) that people the approach that we have taken to without information cannot make good data and its use. However, in displaying decisions appears self-evident, we all the resources, the linkage to the other know from our own experience that components will become evident. As the mere presence of performance part of the Assessment literacy data does not necessarily lead people approach resources have been and are to make good decisions.Teachers are being developed around student no exception to this, despite – or performance data across all learning perhaps because – of the fact that they areas and phases of schooling.We have are surrounded by a plethora of data also been working with ACER on the on student behaviour and performance, development of assessments on the there is no automatic and universal social outcomes of schooling both inter- adjustment to their teaching practice in and intra-personal.This paper will limit response. Herein, lies the nub of the itself to a discussion of the resources issue that we have been working on in related to the performance data Western Australia what needs to produced by the Western Australian happen to performance data to turn it Literacy and Numeracy Assessments into the kind of information that will (WALNA) in Years 3, 5 and 7 and some cause teachers to modify their teaching comments on Monitoring Standards in practice? Education Year 9 (MSE 9). While the classroom teacher is the The WALNA is a curriculum-based critical target for this work, we know assessment that tests students’ that teachers engage in their work knowledge and skills in numeracy,

Using Data to Support Learning 127 reading, spelling and writing. Annual to ensure that tests can be placed information, as it is a response to a testing commenced in 1998 with the against a common scale from year to senior management directive. In other assessment of Year 3 students in year across each of the year groups, for words, the paradigm developed is an reading, writing, and spelling. Over the each of the areas assessed. example of top-down directive meeting next two years numeracy was added bottom-up demand. It should be recognised that while the and the assessment extended to Years psychometric properties of the test are 5, and 7. MSE 9 was used to assess Year of critical importance to those of us Presentation of the 9 students in reading, viewing and interested in measurement, it cuts little data mathematics in 2004 and is being ice with teachers.The things that extended to include writing and science (This part of the presentation will involve increase the validity of the tests in the for 2005. Both sets of tests are whole a ‘show and tell’ of the different electronic eyes of teachers are: cohort tests for students in public data displays including hot links that schools in WA and are also used by • The direct and explicit linkage cannot be reproduced in a paper medium.) between each test item and a Catholic and independent schools in Data is presented in different ways to corresponding element of the WA and some other states and the many different audiences that curriculum (Outcomes and territories. receive it. Parents, teachers, principals, Standards Framework); In looking at the data and its use to district directors and the community support learning this paper will focus • The involvement of classroom through the media all receive reports of on the following issues: teachers in the panelling of items the results following the annual for consideration; assessment in August.The general • The validity of the data; • The trialling of sample items in principle that has been applied to these • The way in which the data is actual classes; and reports is that data is provided in a presented; • The use of teachers as markers and form that provides the most useful • The knowledge and skills required the associated training that goes evidence for the role of the audience. In by teachers to interpret the data; with it. other words, the data is reconstructed • The way that data is managed at into information to support the Thus, we have a standardised test with the school site and the support decision-making and judgements that a beautiful set of psychometric provided for teachers to work with are the responsibility of the specific numbers that specifically measures the data; and group to whom it is supplied.This will important facets of the curriculum be illustrated through examples of the • Teachers’ capacity to transform data outcomes which teachers want their data presented to class teachers, school into information. students to achieve, and a growing principals and district directors. recognition by teachers that their Underlying properties professional expertise has been Class teachers of the test incorporated into the test’s design and construction. To assist the class teacher make Fundamental to any discussion of the appropriate decisions on their teaching When we were starting out it was use of data is the quality of the data.To programs, the data is constructed so as clearly apparent that those teachers what extent does the test result tell us to give detailed information on each who had been directly involved in the about the nature of learning that has student’s performance on each test process of test construction were far occurred? The WALNA is developed in item.The electronic worksheet on more open to the more ‘diagnostic’ use accordance with standards of best which the data is presented enables of the assessment results than those psychometric practice (Wright & Stone, teachers to cross-reference items to who saw the test as another off-the- 1979) in terms of item response and the component of the curriculum being shelf standardised assessment. In fact, Rasch analysis.The internal reliability for assessed and to undertake a miscue the development of the approach that every test instrument is greater than analysis to gain further insight into a we have adopted is as much the result r 0.8 and a rigorous regime of student’s learning. horizontal and vertical equating is used of a response from teachers for more

Research Conference 2005 128 Item by item analysis enables the a group or a whole class.The super- Year 3 to Year 5 and on to Year 7). production of learning profiles of profile also provides teachers with Principals are assisted to look at aspects individuals or groups of students.These some ideas about the components of of ‘value-adding’ by an analysis of the profiles are represented on a set of the curriculum that represent the actual performance of a year group ‘super-profiles’ in English (reading) and logical next steps to teach in line with compared with that predicted from the mathematics (number, measurement, the notion of proximal development socioeconomic status of the students, space and chance and data). Each of the (Vygotsky, 1978). their earlier performance or both of five super-profiles have been generated these factors combined. Often teachers observe a pattern in the from a Rasch analysis of the elements results of a particular student or a group As with the data presented to teachers, of the curriculum that have been of students, which indicates that the data is presented to principals in a assessed, not just in this test but components of the curriculum have or way that causes them to ask questions throughout the history of testing in WA have not been learnt but the teachers are and to triangulate the test data with with MSE and WALNA. Each profile has unable to confirm whether or not this is that from other sources.The data also involved the analysis of over 1,000 test true. In other words, the teachers are leads the principals to ask questions items and 50,000 students. Any one test unsure how to specifically assess a about the performance of specific measures a relatively small sample of particular part of the curriculum.To help students or groups of students, which the total profile, usually approximately teachers to assess curriculum outcomes, a requires an analysis of class data by a 30 items. However, because Rasch set of resource books has been class teacher.Thus, principals are modelling has enabled these elements generated that use past test items to encouraged to support their teachers to be placed in a hierarchy along with explain in detail that part of the in accessing the class level data and in many other elements, it is possible to curriculum and how distracters have been learning how to better understand it. project a student’s score from this test chosen to exemplify faulty learning. By onto the profile and predict how that working with these examples, teachers District Directors student would have performed on get a deeper understanding of the other elements. Similarly, groups of curriculum outcomes in question and District Directors have the students or a whole class may be how to assess whether students have responsibility of determining whether a profiled in the same way. achieved them.Teachers often report that school is providing a quality education A critical feature of the way that the by looking at the assessment resources to each of its students.They also data is provided to teachers is that it they gain some insight into how to teach operate in collaboration with the encourages them to ask questions of the curriculum outcomes better.These school principal to improve the overall the data and triangulate this information resources may be viewed on the following performance of each school.To assist with evidence that they have from website: http://www.eddept.wa.edu.au/ the district director, the data is classroom observations, assessments mse/Assessment. presented in ways that provide the and judgements of the same students’ highest level of summarised and learning. In this way, the data that has Principals analysed data alongside other been gathered from a one-off test is information on the Department’s not privileged over that collected by While principals are able to access the intranet web site. Data is again teachers in the course of their teaching. same data as the class teacher, they also presented in ways that cause them to The data from the test suggests to receive the data reconstructed to ask questions and triangulate this data teachers the possibility that some provide comparative evidence of their on student performance with that aspects of the curriculum may have school level performance over time and coming from other sources.The data been learnt and others not yet learnt. If with that of other schools.To assist presented on the web site focuses on teachers are able to confirm this from principals to make judgements about the trend and ‘value-added’ aspects rather their own observations and classroom effectiveness of particular programs or than straight descriptive statistics. It assessments, it provides them with clear policies on student learning, information leads District Directors to ask questions information about what they need to is constructed on cohort growth over of principals that require the principal teach, whether to an individual student, time (in WA primary schools this is from to have engaged in an analysis of their

Using Data to Support Learning 129 own data at both the school and class References level. In this way, Directors are encouraged to support principals to Blanchard, K., Carlos, J., & Randolph, A. understand their school’s data and to (1999). The three keys to develop the analytic skills within the empowerment. Berett-Koehler. school to use the data to plan for Hattie, J. A. (2003).Teachers make a improvement. difference.What is the research evidence? Paper presented at the Conclusion 2003 Research Conference of the Australian Council for Education In Western Australia, we do not believe Research. that we have solved the problem of transforming data into information Vygotsky, L. S. (1978). Mind and society: which leads teachers to improve the The development of higher mental learning of their students, but we do processes. Cambridge, MA: Harvard believe that we are well into the University Press. journey and that we have learnt a lot. Wright, B. D., & Stone, M. H. (1979). The following are just some of the Best test design (Rasch Measurement things that we have learnt: Series) M E S a press. • Teachers are willing to look at test data if it gives them some insight into their students’ learning of relevant curriculum outcomes. • Creating a dichotomy between test results and teacher judgements that privileges one over the other is counterproductive. Assisting teachers to see how test results can refine and sharpen their judgements is very powerful. • Presenting data in ways that encourage teachers to take on a questioning, problem-solving role (scientist practitioner) with respect to their students’ learning causes changes in their teaching practice and results in improvements in student learning. • Working with teachers’ line mangers and their line managers (principals and district directors) enables a system of support for working with class teachers and is more likely to lead to sustainable development in the longer term.

Research Conference 2005 130 Learning about teaching and teaching about learning: Using video data for research and professional development

Although video technology has been developments in teaching and learning available for several decades, the over time (for example, Lampert and collection and use of classroom video Ball’s mathematics teaching study, see data for supporting and improving Lampert & Ball, 1998). teaching and learning can still be Professional development projects considered to be in its infancy. A variety incorporating video data also vary in of research and professional scope and scale. For example, from development projects have made use projects designed by national and State of video data, revealing promising initial Hilary Hollingsworth education organisations, university outcomes and identifying many faculties, and independent organisations, Education Consultant,Victoria possibilities for its use. However, to projects initiated at the local relatively little systematic research has school level. Hilary Hollingsworth is an education consultant been conducted on the feasibility and and researcher who works with schools, effectiveness of various types and uses In both research and professional universities, and education organisations in development contexts, a variety of Australia and the United States. Her current of video in education (Brophy, 2004). methodologies have been developed for interests which focus on the use of video cases This paper outlines the nature and for teacher professional learning were generated collecting, storing, retrieving, coding, virtues of video data, and describes through her work over the last several years at navigating, and analysing classroom video several Australian examples of research LessonLab in Los Angeles, California.While there, data. Over recent years technologies for she was the representative for ACER working on and professional development projects storing and showing video have the Third International Mathematics and Science that utilise classroom video data. It proliferated (for example, tape, laserdisc, Study (TIMSS) 1999 Video Study. As a Senior reports some of the formative Researcher for that study, she shared CD Rom, DVD, and web streaming), and evaluations attesting to the positive responsibility for the development, some sophisticated software/technology outcomes of these projects, as well as implementation and analyses of the video data platforms have been designed to some of the challenges associated with coding scheme, and the authoring of the function purposefully as research tools international and Australian reports. In addition to them. Finally, it anticipates possible (for example, vPrism and Studiocode), or her research role in the TIMSS 1999 Video Study, future directions for the use of as tools for practitioners to explore and she worked with school systems, school districts, classroom video data for supporting universities, professional development work with video data (for example, and improving teaching and learning. organisations, and textbook publishers across the LessonLab’s Visibility platform). United States, as a Director for the Teacher Learning Division of LessonLab.This work The nature and virtues Several authors have reported detailed involved the design and implementation of video of classroom video data descriptions of the virtues of video data cases in a unique and powerful web-based (see for example, Brophy, 2004; Clarke technology platform. She has previously worked Research projects incorporating video & Hollingsworth, 2000; Stigler & as a lecturer at the University of Melbourne, a Hiebert, 1999). Among these are the mathematics consultant in Victorian schools, and a data vary in scope and scale, from large primary teacher in Victoria. She has published international video surveys capturing capacity of video to: preserve papers related to her research in mathematics single lesson snapshots of classroom classroom activity so that it can be education and teacher professional learning, as activity (for example, the Third ‘slowed down’ to enable detailed well as resource books for teachers and parents. International Mathematics and Science examinations of teaching and learning Study (TIMSS) 1999 Video Study, from multiple perspectives; reveal involving the videotaping, coding, and alternatives through comparative analysis of over 1000 lessons in seven analysis; and, stimulate discussions about countries, see Hiebert et al., 2003), to choices related to teaching and learning. studies that involve the videotaping of Brophy (2004) notes that video offers one classroom across extended periods unique affordances especially powerful (months or years), capturing for supporting teaching and learning:

Using Data to Support Learning 147 Videos show both teacher and Science and Training under the Grants anticipated that these video examples student actions and thus can be for National Literacy and Numeracy will be invaluable in communicating the viewed from both the teacher’s and Strategies and Projects Programme. results of the study. the student’s perspective, allowing With the aim of ‘identifying teaching A further research project building attention to both teaching and practices that lead to improved literacy directly on the methodology developed learning issues. (Brophy, 2004, p. 299) outcomes for children in the early years in the In Teachers’ Hands study has of schooling’ (Louden et al., in press, p. commenced in 2005.The Student Projects utilising video iv), the study design included the Growth Study, Effective Teaching:An data collection, observation, and analyses of Evidence Based Approach, being video of literacy teaching sessions in In what ways are classroom video data conducted by Edith Cowan University, is classrooms where students’ literacy being used in efforts to support and an evidence-based enquiry into the performance had been assessed the improve teaching and learning in characteristics of effective teaching in previous year as ‘more than expected’, Australia? To illustrate some of the Western Australian Government ‘as expected’, or ‘less than expected’. possibilities, descriptions of several Schools.This study will make use of Using a Classroom Literacy projects that the author is associated video data in a similar way to In Observation Schedule (CLOS) that was with are presented below.The Teachers’ Hands, and will focus on two based on the project literature review, examples include research and areas: literacy in pre-school and Year 1, the classroom video data were analysed professional development projects, and mathematics in Year 8. for the presence or absence of 33 conducted by national education literacy teaching practices considered organisations, universities, and schools. Example 2 - Use of research important to effective literacy teaching. Each project uses different These teaching practices were grouped video data for teacher methodologies for data collection, into six dimensions: participation, professional learning: Engaging analysis, and use, and while it is beyond knowledge, orchestration, support, in excellence in mathematics the scope of this paper to include differentiation, and respect. details about these, related references teaching,ACER and AAMT are provided where appropriate.The Quantitative and qualitative methods Designed by ACER and AAMT, Engaging intent of this section is to provide a were used to analyse the video data in in Excellence in Mathematics Teaching is sense of the ways classroom video data the study. Quantitative analysis, included a pilot professional learning program in are being used, and provoke thought a simple descriptive analysis by frequency which teachers (1) conduct a self- about possibilities for their further use. to provide a picture of the teaching evaluation against the AAMT Standards Each of the projects included in this practices demonstrated by each teacher. for Excellence in Teaching Mathematics, section exploit the technology’s unique For this purpose researchers used (2) design and undertake a customised, affordances, and tailor the use of the vPrism, a software package that enables needs-based, workplace, professional video – and in several cases the video detailed, time-linked coding of video. learning program, and (3) present the itself – to their specific project goals. Further qualitative analysis of the video outcomes of their program to fellow data provided ‘a textured and nuanced participants1. A key opportunity for Example 1 - Use of video data in account of the application of each of the participants of the program is to research: Edith Cowan University 33 literacy teaching practices’ by the observe selected examples of public teachers videotaped (Louden et al., in Classroom video data was an integral release classroom video data from the press, p.v). component of the research study, In TIMSS 1999 Video Study, and interpret Teachers’ Hands: Effective Literacy In addition to the publication of a them in terms of the AAMT Standards. Teaching Practices in the Early Years of written report of the study findings a The mathematics component of the Schooling conducted by Edith Cowan web site is being developed that TIMSS 1999 Video Study examined University and funded by the Australian includes video examples of the 33 teaching practices in seven countries Government Department of Education, literacy teaching practices. It is through in-depth analysis of 638 eighth-

1For details regarding the Engaging in Excellence program, see Peck, Hollingsworth & Morony, 2004.

Research Conference 2005 148 grade mathematics lesson videotapes2. teacher’s assumptions about reflections.These activities were Public release video data from the Study accepted and expected practice purposefully linked to the theoretical is available on a set of CD ROMs using no longer apply. In this situation, content of the lectures and tutorials LessonLab’s Visibility technology teachers are more inclined to student teachers attended, and to their platform3. Each CD ROM contains video interrogate the videotape and, by school practicum experiences. Rather implication, their own practice.The and related materials including time- than having only one example of unfamiliarity of what they are linked transcripts in both English and the practice from which to draw viewing challenges their professional insights at this stage of their native language, time-linked indexes assumptions about what is enabling efficient navigation around acceptable, competent teaching course, student teachers had multiple different segments in the lessons, lesson practice. (Clarke & Hollingsworth, examples to examine and compare graphs displaying plans of the lessons, 2000) through the use of the video data. time-linked images of textbook and In a unit evaluation survey, student worksheet pages, and time-linked Example 3 - Use of video data teachers nominated ‘Becoming a commentaries on the lessons.Teachers in preservice education: Edith Teacher’ as the most satisfying unit in participating in the Engaging in Excellence Cowan University their first year education program. And, pilot program had the opportunity to tutors of the unit reported increased In 2003 and 2004, faculty in the School use LessonLab’s cutting edge technology quality and depth of understanding by of Education at Edith Cowan University to observe and analyse teaching student teachers generated through developed and trialed a preservice practices and learning environments their observation and analysis of the education unit,‘Becoming a Teacher’, that from seven countries.This experience classroom video data. stimulated discussion about alternative made use of authentic classroom video practices and provoked teachers to data to create links between traditional The success of this first unit led to the reflect on their own practice. face-to-face lecturing and tutorials, development of additional units using practicum experience, and online the Visibility platform across the second Participants in the Engaging in Excellence learning opportunities.The unit was half of 2004, for implementation in the program reported that the TIMSS video designed for first year students to take in 2005 academic year. Professional materials were an extremely useful part first semester of their teacher education learning workshops were conducted for of the initial workshop. Participants course. A major goal of the initiative was academic staff in the School of ‘valued the rare opportunity to look into for teachers entering the profession to Education to assist them to the classrooms of teachers in other understand the importance of reconceptualise their pedagogy to countries and observe their strategies developing sound professional practices effectively incorporate classroom video and learning environments.They also – such as observation, analyses, and data into their education units. In 2005, valued the analysis of these videos reflection – and to gain experience with four units are making use of classroom against the [AAMT] Standards’ (Peck, these practices. video data in the Visibility platform, and Hollingsworth & Morony, 2004, pp. 375- several more are currently under 376).This evaluation is in accordance The ‘Becoming a Teacher’ unit was development for 2006. with the views of Clarke and developed in LessonLab’s Visibility Hollingsworth (2000): technology platform.That platform Example 4 - Use of video data enabled the design of a tailored, When teachers view videotapes of for teacher professional learning: interactive ‘Course’ that could be directly classrooms the familiarity of the authored by faculty members. As part of Lesson study at Ballarat and classroom setting can reduce the Clarendon College,Victoria power of the video clip to catalyse the course, student teachers were teacher reflection. However, if the engaged in viewing classroom videos and Over the past four years, staff at videotaped lessons are taken from completing ‘Tasks’ and ‘Forums’ associated Ballarat and Clarendon College in a very different culture, the with their observations, analyses, and Victoria have engaged in whole-school

2Details of the TIMSS 1999 Video Study are reported in Hiebert et al., 2003, and in Hollingsworth, Lokan & McCrae, 2003.

3For details regarding LessonLab’s Visibility software, see: www.lessonlab.com

Using Data to Support Learning 149 strategic professional development. one focused on numeracy.These groups utilising video data of their own Through this process a focus on literacy are continuing to work on the classroom teaching. Each member of and numeracy has emerged, and one refinement of their lessons through the academic staff is videotaped initiative that is currently being 2005.Video data of their work are teaching a lesson at different stages in implemented to provide opportunities being collected and used in two ways. the year.The video is viewed by the for teachers to work on improving their First, each of the collaborative group teacher, and by a colleague who is a teaching in these areas is a local meetings is videotaped so that teachers designated performance manager.The adaptation of the Japanese ‘Lesson can reflect on the group discussions. video is then used as a point of specific Study’ model of teacher learning. These videotapes also provide a record reference to discuss teacher of the group’s progress for evaluative performance, and to provide feedback Lesson study is an ongoing professional and training purposes. Second, each of focused on improvement.The use of learning experience involving small the group’s research lessons is each teacher’s classroom video data in groups of teachers meeting regularly to videotaped so that teachers can this way, provides real evidence of engage in a collaborative process of observe and analyse them, and refer to teaching performance and evidence of lesson planning, implementation, specific examples in them for discussion improvement in teaching performance evaluation and refinement. Key to their in the lesson study meetings. over time. It also contextualises work is the hypothesising of anticipated performance management as part of student responses, the testing of those Teachers who have been involved in the ongoing learning process of hypotheses, and the refinement of the the first two lesson study groups at the teachers within the ongoing learning lesson design.The groups typically meet College have reported that the process community of the school. weekly or bi-weekly for several hours they have engaged in has had a and focus on only a few lessons over significant positive impact on their the year with the aim of perfecting teaching of literacy and numeracy. In Promising initial them. Once the lessons have been particular, they consider that the outcomes, challenges, refined to a point of ‘readiness’ where opportunities to reflect on their own and future directions the group feels they can not perfect and others’ practice, become aware of them any further – usually after several new alternatives, engage in serious In each of the projects described above, months or even years – they are shared questions and discussions about promising initial outcomes regarding the with other teachers and other schools, content and pedagogy, and develop use of video data have been reported complete with development and test observational and analytical skills, have through formative evaluations. In the information, and expected student led to improvements in their teaching. A context of research, classroom video responses to questions and problems. further lesson study group was formed data are highly valued because they Skills gained through the detailed recently, making a total of three groups enable rich and detailed studies of the process of observation and analysis in active in the school during 2005. complex activities of teaching and lesson study transfer to teachers’ work learning, and their reusability allow for on other lessons. As Hiebert and Stigler Example 5 - Use of video data examination from different perspectives (2000) suggest,‘lesson study reverses for teacher professional learning: for different purposes.The report of the the relationship prevalent in the United Performance management at Negotiation of Meaning Project conducted through the University of States [and Australia] between Ballarat and Clarendon College, improving teaching and improving Melbourne and edited by Clarke (2001), teachers.Working on improving Victoria evidences the value of multi-perspective teaching yields teacher development, As part of the whole school strategic analysis of research video data. 4 rather than vice versa.’ approach to professional development Classroom video data also offer a At Ballarat and Clarendon College, two described in Example 4, Ballarat and plethora of opportunities for teacher’s lesson study groups formed in Term 4 Clarendon College staff also participate professional learning.Video data provide of 2004; one focused on literacy and in a performance management program new avenues for teachers, schools,

4For further reading about lesson study see Stigler & Hiebert, 1999.

Research Conference 2005 150 universities, professional developers, and important issue to address will be the mathematics in seven countries: Results professional organisations, to engage in need for summative evaluation of from the TIMSS 1999 Video Study. rigorous and serious observation and projects. As Brophy (2004) contends: Washington, D.C.: National Center for analyses of classroom activity to support Education Statistics. Along with continual and improve teaching and learning. developments in the state of the Hollingsworth, H., Lokan, J., & McCrae, While positive outcomes of projects art of both the technology itself B. (2003). Teaching mathematics in using classroom video data are being and its applications for teacher Australia: Results from the TIMSS 1999 reported and celebrated, a number of education purposes [and Video Study. Melbourne: ACER. research], the field now needs challenges have also emerged. systematic research designed to Jacobs, J., Garnier, H., Gallimore, R., Elsewhere, authors have recorded provide summative evaluation of Hollingsworth, H., Givvin, K.B., Rust, K., details of practical and logistical the effectiveness of video-based Kawanaka,T., Smith, M.,Wearne, D., challenges associated with the programs and assessment of the Manaster, A., Etterbeek,W., Hiebert, J. collection, analysis, and use of video trade-offs involved in alternative & Stigler, J.W. (2003). TIMSS 1999 data, including cost, time, quality of approaches. (Brophy, 2004, p. 304) Video Study Technical Report:Volume 1: video production, ethics, copyright, and Video data offer tremendous Mathematics Study. NCES 2003-012. technology constraints (Brophy, 2004; opportunity for authentic learning Washington, DC: U.S. Department of Clarke & Hollingsworth, 2000; Jacobs et about teaching and teaching about Education, National Center for al., 2004). In the Australian context, each learning, and provide a rich resource for Education Statistics. of these challenges exists; however, as helping set future directions for evidenced by the number of projects Lampert, M., & Ball, D. (1998).Teaching, improvement. already under way, none are multimedia, and mathematics: insurmountable. Perhaps the most Investigations of real practice. New pertinent challenge associated with the References York:Teachers College Press. use of video data in Australia relates to Brophy, J. (Ed.) (2004). Using video in Louden,W., Rohl, M., Barratt Pugh, C., developing a culture among teachers teacher education.The Netherlands: Brown, C., Cairney,T., Elderfield, J., and teacher educators that values and Elsevier. House, H., Meiers, M., Rivalland, J., & embraces the collection and use of Clarke, D. J. & Hollingsworth, H. (2000). Rowe, K. (in press). In teachers’ hands: video data. Seeing is understanding. Journal of Staff Effective literacy teaching practices in The ideas and examples presented in Development, 21, 40-43. the early years of schooling. Mt Lawley, this paper represent only some of the WA: Edith Cowan University. Clarke, D. J. (Ed.) (2001). Perspectives possibilities for using video data to on practice and meaning in Peck, R., Hollingsworth, H., & Morony, support and improve teaching and mathematics and science classrooms. W. (2004). Engaging with excellence in learning. Other groups are working with Kluwer Academic Press: Dordrecht, mathematics teaching: Creating video data in different ways, and Netherlands. excellence in the learning knowledge and expertise in the area is environment. In B.Tadich, S.Tobias, C. growing. It is possible, and in some Hiebert, J., & Stigler, J. (2000). A Brew, B. Beatty, & P.Sullivan (Eds.), cases planned, that projects like the proposal for improving classroom Towards excellence in mathematics (pp. examples described earlier (as well as teaching: Lessons from the TIMSS 367-376). Brunswick: Mathematical others not reported here) be scaled up video study. Elementary School Journal, Association of Victoria. to extend to schools, universities, and 101(1), 3-20. professional organisations across the Stigler, J.W., & Hiebert, J. (1999). The Hiebert, J., Gallimore, R., Garnier, H., country. As skill and efficiencies in video teaching gap: best ideas from the Givvin, K.B., Hollingsworth, H., Jacobs, data capture, store, and use develop, world’s teachers for improving education J., Miu-ying Chui, A.,Wearne, D., Smith, and as the teaching profession in the classroom. New York:The Free M., Kersting, N., Manaster, A.,Tseng, E., embraces the collection and use of Press. Etterbeek,W., Manaster, C., Gonzales, video data, even more options will P., & Stigler, J. (2003). Teaching open up. As the field expands, an

Using Data to Support Learning 151 An evidence-based approach to improvement: A case study of the Victorian Catholic Sector

The evidence based approach is integral empirical evidence in making decisions to efforts by the Victorian Catholic about how to deliver instruction. sector to continually improve its There has been a concerted effort in effectiveness.There has been a the Victorian Catholic sector to concerted effort in the Victorian broaden the professional experience Catholic sector to broaden the and judgement of system personnel, professional experience and judgement school leaders and teachers by locating of system personnel, school leaders and it within the available evidence and teachers by locating it within the Teresa Angelico generating research studies to available evidence and generating continually explore and test it.This Catholic Education Office research studies to continually explore paper illustrates the instrumental role of and test it. Evidence generation and evidence generation and transfer in Dr Teresa Angelico is Assistant Director in Policy transfer have been instrumental in and Governance at the Catholic Education Office shaping the design of literacy and shaping the design of literacy and Melbourne with a responsibility for the processes numeracy programs, informing the numeracy programs.There is an of policymaking, governance, knowledge content of professional learning increasing emphasis on using evidence management, legal services and communications. initiatives, and informing decision-making Teresa has a background in primary teaching and as a tool for professional learning and related to improving the overall in higher education through her involvement in a to inform decision making related to performance of schools and the system range of areas of study in education and improving the overall performance of sociology. She has had extensive experience in as a whole. the provision consultancy services to schools schools. In addition to using evidence to assisting them to improve educational outcomes facilitate reform within the sector, the Literacy and numeracy for students living in poverty. She has a strong Catholic sector system authority has interest in school improvement and effectiveness, used evidence to make a case for program development in particular ways in which research can inform improving the level of financial support policy development. Her doctoral thesis explored The systemic approach to the collection provided by the State Government for this relationship. of student and teacher data through the Victorian Catholic schools. Literacy Advance research project There are 483 Catholic primary, provided schools with evidence about secondary and special schools the impact of various literacy programs representing over 180,000 student on student achievement.The data enrolments in Victoria. A key priority for collected included students’ literacy the Victorian Catholic sector system assessment and information about authority over the last decade has been schools, classes, teachers and students. the building of capacity for the system, The strong performance of students schools and classrooms to continually attending schools where the Children’s improve their effectiveness. A critical Literacy Success Strategy (CLaSS) feature of the capacity building agenda program was being implemented led to has been the centrality of evidence as an increase in the number of schools the basis for decision-making and action. implementing the program. Ainley and The impetus for this has been the Fleming (2003, pxi) indicate an increase evidence-based movement in education from 11 per cent of schools in 1999 to defined by Whitehurst (2001) as the 79 per cent in 2002.The key elements integration of professional wisdom of the CLaSS approach include the (which is acquired through experience following: the importance of designating and consensus) with the best available a literacy coordination position in the school; the provision of a designated

Research Conference 2005 152 time for literacy in the daily timetable; from 2001 to 2003, developed twelve effective and efficient ways of improving and the opportunity for students to have effective scaffolding approaches (Lewis leader and teacher quality. Over the access to intervention program support. & Lindsay, 2005). past years, there has been a concerted effort to increase the level of The two-year longitudinal study showed The performance levels of students confidence and skills of personnel that the CLaSS program delivered enrolled in schools participating in these working at both the system and school positive benefits for students in the projects were measured through the level through a range of professional Catholic sector (Ainley & Fleming, use of clinical interviews. Student data learning activities, including sponsorship 2000). Five years on, the longitudinal was collected at the beginning and the to undertake accredited courses.There study highlighted the importance of a end of the project and compared to is a heightened awareness of the need good foundation to literacy beginning in students enrolled in non-participating to incorporate data literacy in the range the preparatory year. Enjoyment of schools to test the effectiveness of of professional learning initiatives. activities is a further element vital to curriculum and assessment approaches continued growth in literacy, pointing to and instruments. For instance, the collection of the importance of providing literacy documented evidence, as part of the At the end of 2004, a research study was activities and processes that engage and development of a Professional Portfolio commissioned to identify factors affecting motivate students at different year is central to the Leadership Standards high student achievement at the VCE levels. A challenge remains to find initiative which is designed for teachers level.The research undertaken by Kaye effective strategies with which to assist aspiring to work towards formal Stacey (University of Melbourne) students who are experiencing leadership positions and undertaken in identified a number of variables that are difficulties in literacy beyond Year 1, and partnership with the Australian Council positively correlated with high to lessen the widening gap between for Educational Research (ACER). performance such as a larger student students with differing abilities (Ainley & Participants are required to document cohort; a broader range of mathematics Fleming, 2003). an initiative in which they managed and subject offerings; high quality program lead a change effort with colleagues in Decisions informing the design of provision for all students; a culture of high their school, providing both quantitative numeracy programs have been based achievement; and the capacity to attract and qualitative evidence to support the on evidence arising from three major and retain quality teachers. A seminar attainment of standards for practice research initiatives undertaken in was held on 3 June 2005 for the purpose related to five areas of school life and partnership with a team of researchers of reflecting on the research evidence operations.The five areas include the associated with the Australian Catholic emerging from these research studies. following: the Faith Community; A Vision University (ACU).The Early Numeracy Participants identified a number of for the Whole School;Teaching and Research Project (ENRP), implemented strategies for supporting high Learning; People and Resources; and from 1999 to 2001, developed a achievement in mathematics education in Pastoral and Community. hierarchical framework to monitor the Victorian Catholic sector. Suggestions children’s understanding and arising from this seminar will be published Project participants record action taken, development of mathematics called in a Seminar Series Paper and school and evidence of professional growth growth point.This framework allows personnel will be invited to respond. includes elements such as content and teachers to track children’s Suggestions include the importance of practice of teaching, communication and mathematical advancement through the making student performance data interpersonal skills, assessment and use of a clinical interview.The Middle accessible and the provision of reporting, evidence of professional Years Research Project (MYNRP), professional development support. reading and contribution to the implemented from 2001 to 2002, was profession. Reports from participants instrumental in informing the Professional learning indicate that the documentation is development of a tool called Rich facilitating learning as it provides a basis Assessment Tasks to track students’ Given the importance of combining for analysis of practice.Through this understanding. Finally, the Researching professional wisdom with empirical analysis, participants acquire a Numeracy Teaching Approaches in the evidence, the Catholic sector is faced heightened awareness of their strengths Primary School Project, implemented with an ongoing challenge to find and weaknesses. As one teacher notes,

Using Data to Support Learning 153 the documentation ‘made me think student enrolments; student learning of this data depends on professional about the possible impact of my outcomes; Religious Education; school judgement. As McGurn and Farrar teaching and leading, and how I must finances; school facilities; and personnel. (2005 forthcoming) indicate there is a plan for this in advance’ (Audley & Further reports will be developed over need for careful reflection about McDonald, 2005). time including in a seventh area of contextual factors impacting on results Student Wellbeing. and a need to consider results over a period of time. School improvement The School Improvement Reports Data collection is central to the overall provide schools with summative In addition to systemic information process of school improvement. reports of their own data over time as services, Catholic schools also collect Schools use a range of data sources to well as comparisons with all Victorian data at the local level. Evidence about inform the identification of priorities for Catholic Schools and Like Schools (by school climate and community member the school as a whole, for the enrolment bands, funding category, perceptions are collected through professional learning of not only the language background or socioeconomic attitudinal surveys. O’Donnell (2005 staff as a whole but also of the status).The development of additional forthcoming) principal of St Gregory individual members. Data is collected reports and enhancements to existing the Great Primary Doncaster, outlines a through external system-based sources reports will be informed by the number of survey instruments that will and through school-level initiatives. recommendations from a recent trial be implemented every two years.The and review of the information system. staff surveys include The School as a According to Pascoe and Jane (2005), Another initiative to be released later Workplace and the Psychological Health all Victorian Catholic principals and staff this year is an extensive online of Staff.The student data survey focuses Catholic members have access to the reporting system for literacy on social outcomes and the Parent Education Victoria Network (CEVN) assessments in primary schools and a Opinion survey collects data about Information and Support portal which detailed financial report. Professional perceived teaching quality, academic has been created to support school development and consultancy services rigour, customer responsiveness and improvement.This site provides online are provided to strengthen school general satisfaction. dynamic resources including directories, leaders’ capacity to analyse their own Schools recognise the importance of document repositories, profiles and data and use it in their planning for using data to build individual student reports of personnel, finance and school improvement. student outcomes data, professional profiles in order to support them in development programs and links to Catholic schools also access the their learning. Presentation College support programs. Victorian Certificate of Education Windsor (in McGunn & Farrar, 2005, (VCE) Data Service, providing them forthcoming), is currently drawing CECV An important feature of the with information about their students’ together information from AIM testing, Information and Support portal is the performance in any studies or in all primary school reports, transition School Profile Report online (SIR) which studies offered by their school.The interviews and student absenteeism in provides a one-page summary for each service allows schools to extract the middle and senior years.The school school and a link to a suite of School information about performance over is also devising a survey designed to Improvement Reports that school time and in specific assessment tasks. track students’ movements from school leaders can use to inform their own For instance, data has been used at with a view to supporting students in planning for ongoing school Presentation College Windsor to making decisions about course options improvement. Data are dynamically enhance the effectiveness of VCE study and pathways from schooling. Data on drawn from the system database to periods and had led to the students’ part-time work habits will also provide up-to-date reports to users development of a formal trial exam be collected with a view to integrating providing schools’ individual and program, with the English exam being the teaching of time management skills. comparative data and indicators of their assessed externally to provide feedback The school is interested in finding out attainment of system-wide targets. to students and their teachers on areas whether part-time work and heavy Currently, reports are provided for six for improvement (McGurn & Farrar, involvement in extra-curricular activities key dimensions of school programs: 2005 forthcoming).The interpretation has the potential to adversely affect

Research Conference 2005 154 student performance. schools, the recurrent expenditure was improvement and effectiveness.The 6% lower than the average priorities underpinning the Systemic improvement Government school and 33% lower development and review of a number than the average Independent school. of the strategy and operational plans1 The need for well-grounded empirical The study also shows that Catholic are reflecting issues arising from the evidence to support the case for schools achieve better than average research studies2. For instance, the increasing the level of funding from the educational outcomes on a range of needs-based formula3 used to allocate State Government to Victorian Catholic measures, with the increment over funding to schools was reviewed and a schools led to the commissioning of average outcomes being particularly new policy is being developed on three research studies in 2004 by the pronounced for students from lower schools fees which is designed to system authority (these can be accessed socioeconomic backgrounds. Social encourage schools to keep fees at at www.cecv.melb.catholic.edu.au). A capital is identified as a factor in affordable levels. case for change could not be made on achieving higher than average Most importantly, the research evidence the basis of perceptions and educational outcomes at lower costs. judgements of personnel working has been instrumental in supporting the within the Catholic sector. The The research findings were campaign to increase the level of State Affordability of Catholic Schools in Victoria disseminated to politicians, senior public Government funding for Victorian undertaken in partnership with Monash servants and academics involved in the Catholic schools.The campaign has University, showed that Catholic schools high level discussions. A seminar was been successful in bringing about an are becoming less affordable and with held to provide researchers with an improvement in funding for not only fewer lower income families able to opportunity to present their findings, needy Catholic schools but also a attend. The Welfare Needs of Victorian and responses were invited. Seminar number of needy Independent schools. Catholic Schools, undertaken in Reports were developed and published Increased support for needy non- partnership with the University of in hard copy and online. In addition, the government schools evident in the Melbourne, indicated that students and print media was invited to a briefing 2005–06 State Budget was provided as parents are seeking help from schools and provided with a media release part of the overall budget for social in a range of welfare areas as schools statement and the full report and, as a disadvantage. On 18 May 2005, the are becoming the point of care for the result, all projects received coverage. Catholic sector signed a landmark local community. Many principals report Catholic school communities were agreement with the State Government that they and their staff are under- invited to regional briefings to respond for a four-year period. resourced to deal with the range of to and discuss the research evidence. In needs encountered. communicating research findings in the Conclusion public arena, the Catholic sector has The Contribution of Catholic Finally, been able to more clearly articulate a The evidence-based approach is Schools to the Victorian Economy and the discourse about Catholic schools that integral to efforts by the Victorian Community , undertaken in partnership emphasises their contribution to the Catholic sector to continually improve with Victoria University, shows that in achievement of public goals of its effectiveness. Evidence generation 2002 the per pupil Victorian Catholic excellence and equity. and transfer has been instrumental in school recurrent expenditure for shaping the design of literacy and primary schools was 21% lower than The research findings are currently numeracy programs.There is an the average Government school and informing strategic and policy increasing emphasis on using evidence 31% lower than the average development processes with a view to as a tool for professional learning and Independent school. For secondary supporting the overall process of to inform decision-making related to

1For example, the strategy plan for the Archdiocese of Melbourne the 2006 -10 plan, the communication strategy and the policy reform strategy.

2For example, the centrality of pastoral care in Catholic schools, the sponsorship of accredited training in welfare studies and the support for specific innovative practices.

3The existing formula allocates recurrent government grants on a ‘needs-based’ formula that favours low socioeconomic school communities. It is intended that the review process will result in a strengthened weighting of grants to lower socioeconomic regions and schools.

Using Data to Support Learning 155 improving the overall performance of Learning Matters Vol 10. No.2. schools. In addition to using evidence to Forthcoming. Melbourne: Catholic facilitate reform within the sector, the Education Office. Catholic sector system authority has O’Donnell, D. (2005). Using Attitudinal used evidence to make a case for Surveys to Inform Policy, Planning and improving the level of financial support Teaching Learning Matters Vol 10. No.2. provided by the State Government for Forthcoming. Melbourne: Catholic Victorian Catholic schools. Education Office. The reliance on evidence underpins the Pascoe, S. & Jane, G. (2005). Radical long-standing practice of forging Innovation in Education. Paper delivered collaborative partnerships with a wide at the International Congress for range of personnel associated with School Effectiveness and Improvement research organisations such as the (ICSEI) Conference, Barcelona. Australian Council for Educational Research (ACER), the University of Whitehurst, G. (2001). Educational Melbourne, Australian Catholic research and improvement.Washington: University (ACU),Victoria University, United States Department of and Monash University.These Education. partnerships have been significant in ensuring that data collection, analysis and reporting methods have been of high quality. In addition, system efficiency is enhanced through the promotion of system-based data services. References Audley, P.& McDonald, A (2005). The Professional Standards of Practice for Leaders in Catholic Schools Learning Matters Vol 10. No. 2. Forthcoming Melbourne: Catholic Education Office. Ainley, J. & Fleming, M. (2000). Learning to read in the early years. Catholic Education Commission of Victoria, Melbourne. Ainley, J., & Fleming, M. (2003). Five years on: Literacy advance in the primary years. Melbourne: Catholic Education Commission of Victoria Lewis, G. & Lindsay, S. (2005). Playing the Numbers Game Learning Matters Vol 10. No.2. Forthcoming. Melbourne: Catholic Education Office. McGurn, K. & Farrar, P.(2005). Turning Data Analysis into Classroom Outcomes

Research Conference 2005 156 PosterPoster presentationspresentations

157 1 Daniel Balacco 2 Lis Turner Framework to plan, monitor, evaluate and communicate progress Department of Education and Children’s Department of Education and Training, of student learning; Services South Australia Western Australia • To assist teachers in interpreting A model of data analysis for Assessment for Learning assessment data and using these to improve their teaching strategies; continuous school improvement Lighthouse Project – Geraldton and in South Australia Schools WA • Using the MSE Assessing Students’ During 2004, the ‘High Performance The Midwest District Education Office exemplars and other system Outcomes’ trial was established with in conjunction with the Standards and resources to support consistency four site teams and districts. The trial Accountability Directorate last year in teacher judgements. aimed to examine the principle of developed an Assessment for Learning ‘Making Data Count’ and trial the Lighthouse Project that was run for 3 Carmel Richardson application of Victoria Bernhardt’s eight schools in the Midwest over approaches to data analysis for semester two. Australian Council for Educational Research continuous school improvement. The project aimed to build skills, ACER’s Data Interpretation This poster will present a Multiple knowledge and understandings of Measures Venn Diagram which will school-based teaching leaders to sustain Service display the model presented in Victoria implementation of policy initiatives. ACER’s Data Interpretation Service Bernhardt’s (2004) ‘Data Analysis for WA Plan for Government Schools has (DIS) seeks to make schools data Continuous School Improvement’ text. an objective of:“Building assessment informed rather than data driven. It The model identifies the four categories literacy: Assisting schools and teachers measures the relationship between of data that are important for to enhance their skills in collecting, student ability and current achievement. continuous school improvement and analyzing and using student also describes what the data tell us by performance information for intersecting these categories of data. improvement purposes”. The DIS analyses a range of Several SA schools have engaged with performance data provided by the The purpose of the project was to this model, and it is now being further school. It does so in the context of build the leadership capacity within and explored by all the districts in SA as expected performance as measured by across a cluster of schools through part of the data4learning.conf expo in an objective instrument the students sit. focusing on recurrent, collaborative, June 2005. The school’s application of This instrument tests innate skills rather data driven processes in which teachers the model will also be presented in the than learned knowledge. This analysis regularly examine student work and poster display. enables the school to identify individual assessment results for the purpose of student’s actual performance against making teaching and learning expected performance within a adjustments. particular subject, across subjects and This was achieved by: over time. As such, it recognises learning • Developing understandings of needs as well as discerning patterns current research from Black and across classes, subjects and year levels. Wiliam (1998),Wiggins and In this way student data informs McTighe (1998) and Stiggins (2002); teaching and learning practices and helps monitor subject outcomes. • Developing evidence-based approaches to plan for student improvement; • Using the Curriculum Framework and Outcomes and Standards

Research Conference 2005 158 4 Peter Weddell 6 Frank Keighley 7 Louise Ellis1, National Awards for Quality Schools Department of Education and Training, Herbert Marsh and Australian Capital Territory Rhonda Craven2 National Awards for Quality 1Australian Council for Educational Research Using multi data sources to Schooling 2University of Western Sydney support improvement and The poster will display the achievements of winners in the 2004 achievement in ACT schools Using an evidence-based National Awards for Quality Schooling. This poster describes the process and research approach to examine Using an evidence-based approach to outcomes of developing a system the impact of peer support research, planning, implementation and report based on the over-arching reporting, applicants were required to concept of the ACT School Excellence The early adolescent years are marked tell the story of an innovative school Initiative, combining qualitative and by a confluence of change, including improvement project or initiative that quantitative sources of information in biological, psychological and social demonstrated improved and sustainable moving to a new paradigm of system developments, as well as the move outcomes for students. The winning reporting, using the four domains of from primary to secondary school. applications reflected innovative work schooling from the ACT's School Awareness of the problems facing across all learning areas, school Improvement Framework as the adolescents has led to the promotion management, social outcomes and the organising themes. of school-based intervention strategies curriculum. to help students maintain positive self-concepts and overcome their adjustment difficulties. However, 5 Deborah Hartman evidence-based research on the and Victoria Clay effectiveness of peer support programs Boys in Schools Program is currently lacking.We have sought to Family Action Centre address this void in previous research University of Newcastle by empirically examining the effectiveness of a widely-used peer support program on both Year 7 Boys Education: – identity, students and their peer support leaders learning and relationships (Year 10/11 students).The findings of The Boys in Schools Program poster this research have important display (with research by Deborah implications and suggest that the Hartman and Victoria Clay) will feature provision of peer support has the evidence from research into boys potential to make a significant education that links male identity, contribution to schools’ efforts to learning and relationships. It will orchestrate positive outcomes not only highlight research methods and practical for early adolescents, but also for older tools that support teachers to develop students who implement the program. appropriate pedagogy and assessment methods to enhance academic self-concept, general self-concept and the specific learning of boys.

Using Data to Support Learning 159 Research Conference 2005 160 ConferenceConference programprogram

161 Sunday 7 August 6.00 – 7.30 Welcome reception Lumina, Grand Hyatt Hotel

Monday 8 August

9.00 Conference opening Savoy Ballroom Professor Geoff Masters, Chief Executive Officer, ACER 9.30 Keynote address Savoy Ballroom ‘Benchmarks and growth and success…Oh, my!’ Dr Gage Kingsbury, Research Director, Northwest Evaluation Association (NWEA), USA Chair Dr John Ainley, Deputy Chief Executive Officer, ACER; Research Director, National and International Surveys 10.30 Morning tea 11.00 Concurrent sessions 1 Session A: Savoy 1 Session B: Kensington Session C: Savoy 2 Session D: Savoy 3 ‘Good data, bad news, good ‘Moving on from Count Me ‘Getting SMART with data in ‘Data-driven school policy making…’ In Too: Evidence-based schools: Lessons from NSW’ improvement through the Dr Gabrielle Matters, Principal teaching and learning in Dr Max Smith, Manager, Data VCE Data Service’ Research Fellow and Manager, numeracy in the early and Analysis and Regional Support, Dr Glenn Rowley, General Brisbane Office, ACER middle years of schooling’ NSW Department of Education Manager Policy Measurement and Chair Margaret Forster, ACER and Training Lynn Tozer and Marilyn Holmes, Research VCAA & Peter Congdon, Chair Marion Meiers, ACER Dunedin College of Education Manager Educational Measurement New Zealand VCAA Chair Kerry-Anne Hoad, ACER Chair Barry McCrae, ACER

12.15 Lunch and poster displays 1.15 Concurrent sessions 2 Session E: Savoy 1 Session F: Savoy 2 Session G: Savoy 3 Session H: Kensington ‘Getting it Right’ Symposium ‘Data and school ‘Using the evidence of ‘Using HSC data to give Dr Lawrence Ingvarson, Research improvement – A school student achievement for principals leverage’ Director,Teaching and Learning, perspective’ improvements at individual, Dr John DeCourcy, Principal, ACER, Marion Meiers, Senior Wayne Craig, Director, Northern class and school level’ St Andrew’s College, Marayong Research Fellow, ACER and Metropolitan Region, Department Dr Reg Allen, CEO,Tasmania NSW Rosemary Cahill, Manager, Chair of Education and Training VIC Qualifications Authority Andrew Jackson, Curriculum Directorate, Chair Nick Thornton, Chair Kerry-Anne Hoad, ACER ACER, APC Department of Education and ACER, APC Training WA Chair Peter McGuckian, ACER

2.30 Afternoon tea 3.00 Concurrent sessions 3 Session I: Kensington Session J: Savoy 2 Session K: Savoy 3 Session L: Savoy 1 ‘An evidence-based approach ‘Assessment for learning: ‘Data-informed research ‘Using online assessment to to teaching and learning’ Using Statewide Literacy and practice: Evaluating inform teaching and learning Dr Michele Bruniges, CEO, & Numeracy tests as student achievement in in primary and secondary ACT Department of Education diagnostic tools’ secondary schools’ classrooms’ and Training Philip Holmes-Smith, Director Carmel Richardson, Senior Professor Jim Tognolini, Research Chair Sheldon Rothman, ACER School Research Evaluation and Research Fellow, ACER Director, System and School Measurement VIC Chair Kerry-Anne Hoad, ACER Testing, ACER Chair Marion Meiers, ACER Chair Alison Elliott, ACER

4.15 Close of discussion 7.00 Conference dinner Savoy Ballroom Tuesday 9 August

9.15 Keynote address Savoy Ballroom ‘From accounting to accountability: Harnessing data for school improvement’ Associate Professor Lorna Earl, Ontario Institute for Studies in Education, University of Toronto, Canada Chair Dr John Ainley, ACER 10.30 Morning tea 11.00 Concurrent sessions 4 Session M: Savoy 2 Session N: Savoy 1 Session O: Savoy 3 Session P: Kensington ‘Turning data into ‘Evidence for the kinds of ‘Learning about teaching and ‘An evidence-based approach information that improves feedback data that support teaching about learning: Using to improvement:A case study learning:The WA experience’ both student and teacher video data for research and of the Victorian Catholic Dr David Axworthy, Research learning’ professional development’ Sector’ Director, Performance and Dr Ken Rowe, Research Director, Dr Hilary Hollingsworth, Dr Teresa Angelico, Assistant Accountability, Department of Learning Processes and Contexts, Consultant,Victoria Director, Catholic Education Office, Education and Training WA ACER Chair Kerry-Anne Hoad, ACER Melbourne Chair Marion Meiers, ACER Chair Pamela Macklin, ACER Chair Deirdre Jackson, ACER

12.15 Lunch and poster displays 1.15 Keynote address Savoy Ballroom ‘What is the nature of evidence that makes a difference to learning?’ Professor John Hattie, Auckland University NZ Chair Dr John Ainley, ACER 2.30 Closing address Savoy Ballroom Professor Geoff Masters, CEO, ACER 3.00 Close of Conference Research Conference 2005 164 ConferenceConference delegatesdelegates

165 Dinner Table No. Delegate Name Delegate Organisation

33 Ms Ibtisam Abu-Duhou Department of Employment, Education & Training, NT 32 Mrs Vanessa Aguirre St. Michael's Primary, NSW 2 Dr John Ainley ACER, Deputy CEO Mr Nerissa Albon Grey Street Primary School,VIC Miss Gail Allen Orrvale Primary School,VIC 15 Ms Kelly Allen Woodridge Senior High School, QLD 14 Mr Peter Allen Christchurch College of Education, NZ Dr Reg Allen Tasmanian Qualifications Authority 32 Ms Paula Allt Cerdon College, NSW 38 Ms Jennifer Amaranathan University of Auckland, NZ 19 Ms Prue Anderson ACER, Assessment & Reporting Ms Robyn Anderson North Geelong Secordary College,VIC Mr John Anderton Tuart College,WA Mr Stephen Andrews Catholic Education Office, NSW 5 Dr Teresa Angelico Catholic Education Office,VIC Ms Dale Anthony PLC Sydney, NSW Ms Cathy Archbold Department of Education & The Arts, QLD Mrs Marian Armstrong Lakeside Secondary College,VIC Mrs Mary Asikas Seaford 6-12 School, SA 22 Dr Mark Askew Broken Bay Diocese, NSW Professor Tania Aspland USC, QLD 9 Mrs Kerrie Atkins DEST, Canberra, ACT 37 Mrs Julie Atkinson Pascoe Vale Girls College,VIC Ms Julia Audova Nagle College, NSW Mr Michael Avery Catholic Education Sandhurst,VIC 5 Dr David Axworthy Dept. of Education,WA 37 Mrs Sandra Bailey Brisbane Girls Grammar School, QLD Mrs Christina Baker Fountain Gate Primary School,VIC 25 Mr Robert Baker St Philip's Christian College, NSW 5 Mr Rodney Baker Department of Education,WA 18 Mr Tim Baker Busselton Primary School,WA 2 Mr Daniel Balacco DECS, SA Ms Helen Bandrowski Catholic Education Sandhurst,VIC Mrs Sheena Barber Catholic Education Office,WA Ms Karyn Bartholomai Noosa District State High, QLD Mr Nigel Bartlett Association of Independent Schools of Victoria Mr Gerard Bate Glen Waverley Secondary College,VIC Ms Marg Batt St Ann's School,VIC Mr Craig Battams St John the Apostle School, SA Ms Cheryl Baulch Box Forest College,VIC 24 Mrs Joy Bedford Our Lady of the Sacred Heart, SA

Research Conference 2005 166 Mrs Vivienne Belcher Christchurch College of Education, NZ 14 Mrs Lynne Bell Nerang State High School, QLD Mr Luciano Benedet Catholic Education Office, NSW Mrs Annette Bennet Kingswood College,VIC Mrs Susan Benson Catholic Education Office,WA Ms Helen Bettes St Francis Xavier College,VIC Mrs Leonie Beuhler Toormina Public School, NSW 7 Ms Sue Beveridge Centre for Learning Innovation, NSW Mrs Pitsa Binnion McKinnon Secondary College,VIC Ms Jenny Blackall Erindale College, ACT Mr Robert Blackley St Joseph's College,VIC Mrs Diana Blain All Saints Catholic Senior College, NSW Dr Alan Bliss Melbourne Grammar School,VIC Mr Edgar Bliss Catholic Education, SA 21 Mr Ian Boon Frenchville State School, QLD 34 Mr David Boots DECS, SA Mr Chris Boulton Nannup District High School,WA Ms Christina Bounds All Saints Catholic Senior College, NSW 32 Mr Christopher Bounds Parramatta Marist High School, NSW Mr Robert Bourke-Finn Sebastopol College,VIC Mr Steve Bousfield Westminster School, SA Ms Karen Bowker Gormandale Primary School,VIC Mrs Denise Bowley Barker School,VIC Mr Michael Box Aquinas College,VIC 26 Mrs Jane Boyle Western Heights College,VIC Mrs Judith Boyle Eltham High School,VIC Mrs Judy Brawn Swan Hill Primary School,VIC Mr Scott Briant Department Education & Training,VIC Ms Helen Briggs Kingswood Primary School,VIC Ms Sharon Broadbent Northfield Primary School, SA 16 Mr Michael Brohier Somerset College, QLD Mr Paul Brooks Catholic Education Office Sydney, NSW Ms Bernadette Brouwers St Patrick's College,VIC 36 Dr Brian Brown DET,WA 21 Mr Bruce Brown Scotch College,VIC 21 Mr Christopher Brown St Paul's College, NSW Mr Luke Brown Our Lady of Lourdes, QLD Mr Richard Brown Toowoomba Christian College, QLD Mr Vin Brown St Joseph's College,VIC Ms Claire Browne Edith Cowan University,WA Mr Nicholas Browne The Peninsula School,VIC Mr Jeff Bruce Cherrybrook Technology High School, NSW 1 Dr Michele Bruniges ACT Department of Education & Training

Using Data to Support Learning 167 16 Mr Peter Brunt Holy Spirit School, Geelong,VIC Ms Kaye Brunton Victoria University, NZ Mrs Dionne Bryant Amaroo School, ACT Mrs Gloria Bryant St Patrick's Primary School, NSW Ms Esperanza Buen Center for Educ. Measurement, Philippines 34 Mr Patrick Burford Department of Employment, Education & Training, NT Mr Jeffrey Burn Sacred Heart College,VIC Mrs Eleanor Burt Christchurch College of Education, NZ Ms Rhonda Caddy NSW DET 7 Ms Rosemary Cahill Department of Education & Training,WA Mrs Trish Caldow St Thomas More Primary,VIC Mr Clyde Campbell Eatons Hill State School, QLD Dr Mary Cannon Warrandyte High School,VIC Mr Gerald Carey Gleeson College, SA Mr Bruce Carothers Victoria Point State School, QLD Mrs Meg Carpenter Victoria Point State School, QLD Ms Beverley Casey Ruyton Girls' School,VIC 16 Mr Greg Cater Somerset College, QLD Mr Christopher Challen Orminston College, QLD 33 Mr Noel Chamberlain Albany Senior High School,WA Ms Anne Charles Catholic Education Office,TAS Mr Andrew Chisholm McKinnon Secondary College,VIC Mr Paul Chivers Stockdale Road Primary School,VIC 38 Ms Wei Chun Wang Swinburne University,VIC Mr Michael Ciccarelli Catholic Education Office WA 19 Ms Sue Clarke ACER, Cunningham Library,VIC Ms Victoria Clay University of Newcastle, NSW Ms Carol Cockburn Education Assessment Australia, NSW 13 Mr Ian Coffey St Peter's College,VIC Ms Linda Collings ISCA, ACT 40 Ms Rosalyn Comport St Stephen’s College, QLD 5 Mr Peter Congdon Victorian Curriculum & Assessment Auth. Mrs Judy Connell Catholic Education Office,VIC 21 Mr Vince Connor CEO Bathurst Mrs Doreen Conroy Bathurst High School, NSW 3 Ms Jocelyn Cook DET,WA 41 Dr Maxine Cooper Griffith University, QLD 28 Mr Andrew Coote Ravenswood School for Girls, NSW 11 Mr Noel Corpe Dingley Primary School,VIC 16 Mr Steve Corrigan Corpus Christi Primary School, NSW 4 Dr Ian Cosier Queensland Studies Authority 25 Mrs Christine Cottell-Mayhew Albany Junior High School, NZ 37 Mr Neil Couch University of Waikato, NZ

Research Conference 2005 168 Mr Patrick Coughlan St Anthony's School, QLD 21 Ms Meg Cowey Department of Education & Training,WA 6 Mr Wayne Craig Department of Education & Training,VIC 31 Mr Adam Crawford Balwyn North Primary School,VIC 10 Ms Anne-Maree Creenaune Catholic Education Office, NSW 30 Mr Alan Crierie St Michael's College, SA 1 Dr. Brian Croke ACER Board Member 26 Mr Pedro Cruz Emmanuel Christian Comm. School,WA 38 Dr Everarda Cunningham Swinburne University,VIC 35 Mr Craig Curry Department of Education & Training, ACT 6 Mr Walter Czernezkyj Urrbrae Agricultural High School, SA 36 Mrs Bronwyn Davies DET,WA 13 Mr Hugh Davies Lyneham High School, ACT Mrs Susan Davies Kuyper Christian School, NSW Ms Rhonda Davis Braemar College,VIC Mr Barry Dean Brisbane Boys' College, QLD Dr Lenore Decenteceo Center for Education Measurement, Philippines 4 Dr John DeCourcy St Andrew's College, NSW Ms Bronwyn Dell Liddiard Road Primary School,VIC Mrs Donna Denham Brisbane Catholic Education Centre, QLD 29 Ms Jenny Denham Melville Senior High School,WA Ms Susan Dennett DE&T,VIC 13 Mrs Lee Denton St Oliver's Primary School, NSW Dr Joanne Deppeler Monash University,VIC 10 Mr Mark Dickenson Office for Educ. Review,TAS Mrs Deborah Dickson Salford School, NZ Mr Paul Dillon Brisbane School of Distance Education,VIC 24 Mrs Rebecca Dillon Trinity College, SA Ms Leanne Distefano Killester College,VIC 32 Mr Peter Dodd Radford College, ACT 3 Mr Alan Dodson Department of Education & Training,WA 32 Mr Paul Dolan Cerdon College, NSW 21 Mr Steve Donatti Department of Education & Training,WA 19 Ms Trish Donohue NSW DET Dr Jenny Donovan Educational Assessment Australia, NSW Mr Tim Dorning St Michael's Grammar,VIC 30 Mr Jamie Dorrington St Stephen's College, QLD Mr Harold Dover Curriculum Coporation,VIC Mr Robert Dowdell Sydney Boys High School, NSW 36 Mr Geoffrey Dowling Athelstone Primary School, SA Ms Kylie Doye DET,VIC Mr Greg Duggan Catholic Education Office, Sydney, NSW Ms Jennifer Duggan Fountain Gate Primary School,VIC

Using Data to Support Learning 169 Mrs Debbie Dunwoody Methodist Ladies' College,VIC Mrs Gaye Durnsford St Laurence's College, QLD Mrs Anne Dwyer Toorak College,VIC 2 Assoc. Prof. Lorna Earl OISE-UT, Canada Ms Kim Eckersley Monash University,VIC Ms Kate Edge DECS, SA Ms Gayle Edmends St Thomas More Primary,VIC Mr John Edwards Catholic Education Office, NSW Miss Lorraine Edwards Glenroy West Primary,VIC 7 Ms Debbie Efthymaides Department of Employment, Education & Training, NT Ms Jessica Elderfield Edith Cowan University,WA 4 Dr Alison Elliot ACER, Early Childhood Education 23 Mr Geoffrey Elliott Mt Tarcoola Primary School,WA 12 Mrs Jane Elliott ACER, Professional Learning Mr Ron Elliott Traralgon Secondary College,VIC Mrs Brigitte Ellis Catholic Education Commission, NSW Mr Geoff Ellis Carwatha College,VIC 10 Dr Louise Ellis ACER, Learning Processes & Contexts Ms Jennifer Emery DECS, SA 34 Mrs Lorraine Evan Department of Employment, Education & Training, NT Ms Gail Evans Seaford 6-12 School, SA Ms Angela Falkenberg Hampstead Primary School, SA 34 Mr Brian Featherston DECS, SA Mrs Wendy Featherston DECS, SA Miss Robyn Field Glenfield Public School, NSW Mr Patrick Finnerty Nagle College 14 Mr Jonathan Fisher NZCER Dr Tom Fisher St Leonard's College,Vic. Mr Grant Fitzgerald Catholic Education Sandhurst,VIC Mrs Mary Fitz-Gerald Mater Christi College,VIC Mr Ernie Fleming Flora Hill Secondary College,VIC Mr Andrew Forbes Catholic Education Commission, NSW Ms Catherine Ford Wantirna College,VIC 2 Ms Margaret Forster ACER, Assessment & Reporting Mr David Fox Griffith High School, NSW Mr Gary Francis Department of Education & The Arts, QLD Ms Mel Francis Melbourne Girls' College,VIC Ms Kerry Franklin Mentone Girls' Grammar,VIC 32 Dr Jennifer Fraser St. Michael's Primary NSW Mrs Helen Freeman Albany Rise Primary,VIC 4 Ms Janet Frost St Andrew's College, NSW 29 Mr Peter Furey Brighton Grammar School,VIC 29 Mrs Marion Gaertner-Jones Annesley College, SA

Research Conference 2005 170 Mr Graham Gallas Australian National University, ACT Mrs Christina Gamble All Saints' College,WA 28 Mr David Garner Ravenswood School for Girls, NSW Mrs Mary Ann Gatt-Petrini Holy Spirit Primary, NSW Mrs Kaye Gauci Glenroy West Primary,VIC Mrs Judith Gaunt St Catherine's School, SA Miss Lyndal Gaunt Overnewton College,VIC 31 Mrs Janette Gee Belmont City College,WA 20 Mr Tony George Aberfoyle Park High School, SA 34 Mrs Tassi Georgiadis DECS, SA Ms Deborah Gibbs Massey University of Educ., NZ Mr Kevin Gillan Warren Blackwood Educ. Office,WA Mr Tim Gilmour Nagle College, NZ Mrs Fiona Godfrey Canberra Girls Grammar, ACT Mrs Kelly Godsall-Folster Salford School, NZ 41 Mr Christopher Gold St Michael’s College, QLD Mr Martin Gooding Knox Grammar School, NSW 31 Ms Fiona Gordon Beaconhills College,VIC 14 Mr Ron Gorman AISWA 26 Mrs RoseMarie Gosper St Mary's Primary School, NSW 14 Ms Valerie Gould AISWA Mr Michael Gowland Bethany Catholic Primary School,VIC Mr Chris Graham Catholic Education Office, Parramatta Mrs Lorraine Grant Salford School, NZ Dr Jan Gray Edith Cowan University,WA Mrs Dianne Grech Bentley Park Collelge, QLD 2 Mr Alan Green DECS, SA Miss Jackie Green North Melbourne Primary School,VIC 21 Mr Phil Green Department of Education & Training,WA Mrs Anna Hagan C.E.D., Massey University, NZ 12 Mrs Sue Hale Dromana Secondary College,VIC 13 Mr Michael Hall Lanyon High School, ACT Mr Nick Hamer-Smith Lyndhurst Secondary College,VIC Miss Jodie Hancock St. Anthony's School, SA Mr Rob Handreck Balwyn High School,VIC Ms Rosie Harden Yankalilla Area School, SA Ms Jacinta Harding Emerson School,VIC 5 Mrs Helen Hardman Catholic Education Office,VIC Mr Tony Harkness Brisbane Catholic Education Mr Tim Harms Immanuel College, SA Mr Peter Harrison Tyers Primary School,VIC Ms Jan Hart The Scots College, NSW Ms Deborah Hartman University of Newcastle

Using Data to Support Learning 171 3 Professor John Hattie University of Auckland, NZ Mrs Jane Haustropher Traralgon Special Development School,VIC 13 Mrs Janet Hawes St Bernadette's Primary, NSW Mr Michael Hayes MLC School, NSW Dr Anne Hazell DECS, SA Ms Maureen Hemmings Catholic Education Office, NSW Ms Jenny Henderson MacFarlane Primary School, NT Mrs Lyn Henshall Tintern Schools,VIC 33 Ms Ellen Herden Department of Employment, Education & Training, NT 14 Mrs Loreen Herlihy Nerang State High School, QLD 3 Ms Bronwyn Hession Catholic Schools Office, NSW Mr Peter Hibbins Department of Education,VIC Mr Ken Hickling St George School, NSW Mrs Denise Hills Catholic Education Sandhurst,VIC 25 Mrs Marie Therese Hirschhorn St Agnes Catholic High School, NSW Mr Ian Hislop Yakamia Primary School,WA 5 Ms Kerry-Anne Hoad ACER, Manager, Professional Learning Mr Russell Hodgson Toongabbie Primary School,VIC 41 Mrs Kathryn Hodson Multi Serve Education Trust, NZ Mr Steve Hoffman East Doncaster Secondary College,VIC Mrs Pamela Holko Lauriston Girls' School,VIC Dr Hilary Hollingsworth Education Consultant Mrs Maree Holmes St Francis Xavier School,VIC 8 Ms Marilyn Holmes Dunedin College of Education 8 Mr Philip Holmes-Smith School Research Eval. & Meas. Services,VIC Mr Kevin Holohan McCarthy Catholic College, NSW Mrs Ann Hooper Shelford Girls Grammar,VIC Ms Terri Hopkins Catholic Education Office,VIC Mr Michael Hopkinson Catholic Schools Office, NSW Mr Robert Horwood St Peter's School,VIC Ms Helen House Edith Cowan University,WA Mr Tony Hughes Holy Family Primary, NSW Mr Don Hyatt Wellington Sec. College,VIC 3 Dr Lawrence Ingvarson ACER,Teaching & Learning 35 Ms Gail Inniss Department of Education,VIC 40 Mrs Christine Ireland Cranbourne South Primary School,VIC Mr Alan Irvine The Entrance Public School, NSW Mrs Sue-Anne Irving Woodport public School, NSW 6 Mr Andrew Jackson ACER, Australian Principals Centre 6 Mrs Deirdre Jackson ACER, Assessment Services 37 Miss Helen Jackson Pascoe Vale Girls College,VIC 25 Mr Michael Jackson Albany Junior High School, NZ 17 Mrs Patricia Jackson William Carey Christian School, NSW

Research Conference 2005 172 35 Ms Cathy Jackson-Grant Brisbane Catholic Education 41 Ms Patricia Jacobsen Catholic Education Office, NSW Ms Mary Jacquier Catholic Education Office, SA Mr Michael James Barker College, NSW 34 Ms Irene Janiszewska DECS, SA 40 Mrs Heather Jatan Department of Education,TAS Ms Cecille Jeffery Catholic Education Office,VIC Ms Pauline Jelleff Glengarry Primary School,VIC Mr Gary Johnson Cherrybrook Technology High School, NSW Mrs Maree Johnson St Joseph's College,VIC 23 Ms Jayne Johnston Professional Learning Institute,WA Ms Jenny Johnston Assoc. Indep. Schools of Tas. 24 Mr Stuart Johnston Trinity College, SA 10 Ms Lois Joll PCAPP,WA Mrs Annmaree Jones Wembley Primary School,VIC Ms Bernadette Jones Ballajura Comm. College,WA Mr Garry Jones Dunsborough Primary School,WA 3 Ms Glynis Jones Panatley Pty Ltd, NSW 22 Mr Kevin Jones Delany College, NSW Ms Sue Jones Seaford 6-12 School, SA 14 Mrs Christine Joyce NZCER Mrs Jo Karaolis St Lucy's School, NSW 9 Ms Irene Kaspar DEST, Canberra 28 Mr Dianne Katalinic Sacred Heart Primary, NSW Mr Chris Kay Donvale Christian College,VIC 9 Mr Frank Keighley Department of Education & Training, ACT Mr Peter Kellett Grace Luthern College, QLD Ms Debra Kelliher Catholic Education Office, NSW Mrs Helen Kelly Bethany Catholic Primary School,VIC 40 Mr Joseph Kelly Cranbourne South Primary School,VIC 37 Mr Peter Kendall Essendon Keilor Secondary College,VIC Mr David Kersivien Our Lady of the Rosary School, NSW Mr Stephen Kimonides Hawthorn Secondary College,VIC 11 Mrs Sheereen Kindler Parkdale Secondary College,VIC 33 Ms Anna King Department of Employment, Education & Training, NT 26 Mrs Janis King James Cook High School, NZ 38 Ms Pam King University of Auckland, NZ 26 Mr Stuart King James Cook High School, NZ 1 Dr Gage Kingsbury Northwest Evaluation Assoc. (NWEA), USA Ms Elizabeth Klienhenz ACER,Teaching & Learning 12 Mr Philip Knight Parkdale Secondary College,VIC 33 Mrs Suzanne Knight Department of Education & Training,WA 40 Mr Andrew Koerbin Department of Education,TAS

Using Data to Support Learning 173 Mr Trevor Kowitz Department of Education & The Arts, QLD 36 Mrs Michaelle Kriening DET,WA Ms Carmel Kriz Catholic Schools Office, NSW Mr Graeme Krutli Traralgon South Primary School,VIC Mrs Adrianne Kupsch DECS, SA 36 Mrs Kay Kuzich Department of Education and Training,WA 18 Mr Michael Laidler Marymount College, QLD 35 Mrs Helen Lambert Association of Independent Schools, SA Dr Julie Landvogt The Ithaka Project 37 Mrs Ruth Lang Brisbane Girls Grammar School, QLD 17 Mr Andrew Latham St Paul's Primary School, NSW Ms Marian Lawless Orrvale Primary School,VIC Mr Terry Lawless North Melbourne Primary School,VIC Ms Jane Leaker DECS, SA 33 Ms Gail Ledger University of Auckland, NZ Dr Denis Lee Our Lady of the Sacred Heart College, NSW Mr Lynden Leppard Department of Education,TAS Mr Craig Lesley Catholic Education Office, Parramatta 17 Ms Jenny Lew Campbelltown P.A. High School, NSW 36 Ms Estelle Lewis The Assoc. of Indep. Schools, NSW 1 Ms Jenny Lewis Aust. Council for Educ. Leaders Dr Peter Lewis Christchurch Grammar School 24 Dr Michael Liddle Trinity College, SA Mr Cameron Lievore St Aidan's Primary School, NSW 42 Associate Professor Kaye Lipson Swinburne University Ms Caron Little Edith Cowan University,WA Ms Sarah Livingston Monbulk College,VIC 35 Ms Karen Livingstone Brisbane Catholic Education Mrs Sharon Livingstone Dunedin College of Education, NZ Ms Debra Lockett Alawa Primary School, NT 20 Mr John Lockhart Chancellor State College, QLD Ms Susan Lohmeyer Education Services & Programs, SA 8 Dr Jan Lokan Former ACER 9 Mr Steve Lusby South Pacific Board for Educ. Assess, 22 Mr Ian Luxton The Southport School, QLD Miss Sam MacCulloch Catholic Schools Office, NSW Ms Lynne Macdonald Plenty Parklands Primary,VIC Mr Anthony MacDougal St Patrick's College, NSW 2 Ms Pamela Macklin ACER, Deputy CEO Ms Chris MacLean Traralgon Network,VIC 20 Mr Jake Madden St Augustine's School, NSW Ms Teresita Maghanoy Center for Educ. Measurement, Philippines Ms Sandra Mahar DE&T Victoria

Research Conference 2005 174 Mrs Patricia Maidens Xavier College, NSW 18 Mrs Danuta Maka Our Lady Queen of Peace Primary, NSW 16 Mrs Jennifer Malone Holy Spirit School, Geelong Ms Marlene Manahan Bethany Catholic Primary School,VIC 22 Mr Michael Maniska Kincoppal-Rose Bay, NSW 30 Mrs Anne Marrins Our Lady of Mt Carmel, NSW 2 Mr Steve Marshall DECS, SA 18 Ms Janice Martin St Anthony's School, NSW Ms Mary Martin Griffith High School, NSW 23 Mr Ian Martineau Emmanuel Anglican College, NSW 28 Mrs Mary Mason The Geelong College,VIC Ms Debra Masters Ministry of Education, NZ 1 Professor Geoff Masters ACER, CEO 4 Dr Gabrielle Matters Qld Dept. of Educ. And the Arts 4 Mr Graham Maxwell Queensland Studies Authority Ms Pamela McAlister Brisbane Waters Sec. College, QLD Mr John McCarthy DE&T,VIC 35 Ms Cheryl McCashney DE&T,VIC Mrs Nancy McCracker Overnewton College,VIC 9 Prof. Barry McCrae ACER, Assessment & Reporting Mrs Sandi McCue MacFarlane Primary School, NT Ms Mary McDonnell Killester College,VIC 18 Ms Cecely McGeachie Our Lady of Mercy College, NSW 9 Mr Peter McGuckian ACER, International Development 15 Ms Heidi McGuinness Tara Anglican School for Girls, NSW 11 Mr Chris McGuire Parkdale Secondary College,VIC Mr Stephen McIllhatton CEO - Lismore, NSW 33 Mrs Gayle McIlraith University of Auckland, NZ 36 Mrs Marilyn McKee DET,WA 15 Mrs Carmel McKelvie Education Queensland 7 Ms Rosa McKenna Dept. of Employ, Educ. & Training, NT Mrs Carol McKenzie Fernworth Primary School, NZ 23 Mr Ian McKenzie Kambrya College,VIC 26 Mrs Jennifer McKeown St Mary's Primary School, NSW Mr Graeme McKinnon Brunswick Secondary College,VIC Ms Jennie McKinnon Lauriston Girls' School,VIC 40 Mrs Roma McKinnon Cranbourne South Primary School,VIC Ms Helen McLean Crestwood High School, NSW Mr Glenn McMahon Catholic Education Sandhurst,VIC 11 Mr Tim McMahon Parkdale Secondary College,VIC Mr Tony McMahon St Joseph's School,VIC 17 Dr Greg McPhan New England Girls' School, NSW Ms Rhonda McPhee Department of Education,VIC

Using Data to Support Learning 175 Ms Bernadette McPherson Catholic Education Sandhurst,VIC Ms Robyn Meddows Nagle College, NSW 7 Ms Marion Meiers ACER,Teaching & Learning Mrs Ewa Mekina Department of Education,TAS 19 Ms Juliette Mendelovits ACER, Assessment & Reporting Mr Bradley Merrick Barker College, NSW 16 Mr Kevin Mills St Patrick's Primary School, NSW 15 Mr Jason Milner Education Queensland 15 Mr David Mitchell Education Queensland Mr Leigh Mitchell St Thomas More Primary,VIC Mrs Shirley Molloy Notre Dame College,VIC Ms Ingrid Molnar Northfield Primary School, SA Mrs Marlene Monahan Bethany Catholic Primary School,VIC Mr Geoff Moore Christchurch College of Educ., NZ 25 Ms Christine Moore Georgiou Camberwell Primary School,VIC 19 Mr John Morath ACER, Assessment Services 4 Ms Deirdre Morris ACER Press Mr Peter Moulds Christian Outreach College, QLD 19 Assoc. Professor Kathryn Moyle Australian National University 8 Mr Wayne Muir University of Melbourne,VIC Mr Peter Mulcahy Education Queensland Mr Andrew Mullally St Agnes School,VIC 24 Ms Linda Munns Trinity College, SA Mrs Maureen Munro Guthrie St Primary School,VIC 26 Mr Gordon Murdoch Department of Education & Training,WA Ms Pauline Murphy Our Lady of the Manger School, SA Mr Atholl Murray The Grange P-!2 Primary,VIC Ms Virginia Murray Hawthorn Secondary College,VIC A/Prof Rosalind Murray-Harvey Flinders University, SA 33 Mr Karl Mutch University of Auckland, NZ Dr Michael Nagel Forest Lake College, QLD Mrs Jill Nash Beechworth Primary School,VIC 24 Mrs Tina Neate Our Lady of the Sacred Heart, SA 39 Mr Greg Nethery Catholic Education Office, Parramatta 41 Mrs Hannah Newman Education Consultant, NSW 6 Ms Margaret Newman Consultant,VIC Mr Yui-kin Ng Hong Kong Institute of Education Mr Stephen Nicholas Great Lakes College Senior Campus Mr Terry Niebling Lourdes Hill College, QLD Ms Sue Nilsen Assoc. of Indep. Schools,VIC Mr Gordon Nolte Glenroy West Primary,VIC 20 Mrs Jill Noonan St Augustine's School, NSW Mr William Norris Renaissance Learning Australia,VIC

Research Conference 2005 176 41 Mrs Ann Northover Multi Serve Education Trust, NZ 4 Mr Brian Nott Queensland Studies Authority Ms Rosalie Nott Catholic Educ. Commission, NSW Mrs Jenny Nykoluk St Thomas More Primary,VIC 40 Mr Andrew Oakley Department of Education,TAS Mr Lynton Oakten Hawthorn Secondary College,VIC Ms Anne Oberholzer IEB. South Africa 40 Ms Debbie O'Brien Office of Board of Senior Sec. Studies, ACT Mrs Dianne O'Brien Catholic Education Sandhurst,VIC Mr Anthony O'Byrne Aquinas College,VIC Ms Liz O'Callaghan Mary MacKillop School, NSW 10 Mr Seamus O'Grady Catholic Education Office, NSW 26 Mrs Tami O'Hare Western Heights College,VIC Ms Harriet Olney Edith Cowan University,WA 17 Mr Ludo Onstein Ascham School, NSW 23 Mr Glenn O'Shaughnessy Kambrya College,VIC 31 Ms Amanda O'Shea Snowtown Area School, SA Mrs Mary Oski CEOM,VIC Mr Ivan Pagett Woodport Public School 39 Ms Jill Palfitt University of Auckland, NZ Mrs Carolyn Pannett Fernworth Primary School, NZ 1 Ms Helen Paphitis Salisbury High School, SA 31 Mr Dave Parker Golden Grove Primary School, SA Dr Stephen Parkin Warrandyte High School,VIC Mr Warren Parkinson Brisbane School of Distance Education, QLD Mr Santo Passarello Combined Catholic Colleges, NSW 6 Ms Sally Paterson Urrbrae Agricultural High School, SA 38 Ms Joanne Patrick University of Auckland, NZ Mr Don Patterson Korumburra Sec. College,VIC 18 Mr Bob Peacock Marymount College, QLD Ms Dianne Peck DET,VIC 38 Mrs Joy Pedersen Department of Employment, Education & Training, NT Mr Geoff Pell Taylors Lakes Sec. College,VIC Mrs Ili Pelletier Shelford Girls Grammar,VIC Ms Jude Pentecost Ministry of Education, NZ Mr George Perini DE&T, Eastern Region,VIC Ms Lee-Anne Perry All Hallows' School, QLD Mr Terence Pestana Challis Primary School,WA 35 Ms Judy Pettiford Wanniassa School, ACT Mrs Terese Phillips Catholic Education Office,TAS Ms Irene Poh Chisholm Catholic College, QLD Mr Garry Pollard Lyndale Secondary College,VIC Mr George Porter Sebastopol College,VIC

Using Data to Support Learning 177 14 Mr Mark Porter Woodcroft College, SA Ms Rhonda Potter Carwatha College,VIC Ms Kondilo Prades Lyndhurst Secondary College,VIC 20 Mrs Coralee Pratt Roberts McCubbin Primary,VIC Mr Norman Pratt University of Auckland, NZ Mrs Marie Previte St Laurence's College, QLD Mr David Price Department of Education & Training,WA Mrs Johanna Prins Kuyper Christian School, NSW 17 Ms Robyn Pulbrook Campbelltown P.A. High School, NSW 8 Dr Nola Purdie ACER, Learning Processes & Contexts Mr Peter Quigley Catholic Education Office, SA 20 Mr Gerard Quinn Chancellor State College, QLD Ms Judi Quinn DECS, SA 9 Mrs Anna Raivoce South Pacific Board for Educ. Assess, Mr John Reed Kosciusko Street Primary School,VIC Mrs Sharon Reeves Alawa Primary School, NT Mr Marc Reicher St Leo's Catholic College, NSW Ms Barbara Reinfeld DECS, SA Ms Anna Rerakis Curriculum Coporation,VIC Mr Christopher Reynolds Trinity College, SA 6 Ms Carmel Richardson ACER, Assessment Services 29 Ms Rhonda Rickards MLC School, NSW 15 Dr Elizabeth Ricketts Holy Trinity Primary, NSW Dr Julie Rimes Kilburn Institute,TAS Dr Deborah Robertson Holy Name School,WA 5 Mrs Claire Robinson-Pope Victorian Curriculum & Assessment Authority Mr Jaime Rodriguez Mount St Joseph, NSW 37 Ms Barbara Rogers Essendon Keilor Sec. College,VIC Assoc. Prof. Mary Rohl Edith Cowan University,WA Mr Jim Rolfe Catholic Education Sandhurst,VIC Ms Lynne Rolley Indept. Educ. Union of Australia 23 Ms Jennifer Roosmalen International Grammar School, NSW 30 Mr Tony Rose St Anthony's Primary School, NSW Mr Jim Ross Glen Waverley Sec. College,VIC 10 Dr Sheldon Rothman ACER,Transitions & Economics of Educ. Mrs Jan Rowe Balwyn North Primary School,VIC 1 Dr Ken Rowe ACER, Learning Processes & Contexts 1 Dr Glenn Rowley Victorian Curriculum & Assessment Auth. Mr Tullio Rubinich Department of Education & Training,WA Ms Fran Rufus Western Heights College,VIC 17 Mr Robert Ruggeri St Margaret Mary's School, NSW Mr Cameron Ryall Upwey High School,VIC Mrs Erica Ryan Catholic Schools Office, NSW

Research Conference 2005 178 Ms Julie Ryan Our Lady of Mercy College,VIC 39 Ms Sophie Ryan Catholic Education Office, Parramatta Ms Janice Sampson Easy Mark Pty Ltd,WA Mrs Susan Sanders Our Lady of Mercy College,VIC Ms Cristina Sandri Highgate Primary School,WA Mr Jonathan Savrine MLC School, NSW Sr Margaret Scroope Catholic Education Commission, NSW 35 Mrs Lynda Secombe Assoc. of Independent Schools, SA Mr Jesus Sevilla, Jr Center for Educ. Measurement, Philippines Mr Brennan Sharpley Catholic Education Office, NSW 16 Mr James Sheedy Holy Spirit School, Geelong Ms Debra Sheehan Overnewton College,VIC 18 Ms Leigh Sheldon Mid West Education District,WA Mrs Anne Sheldrick Glenfield Public School, NSW Ms Julile Shelton Glen Waverley Sec. College,VIC 39 Ms Bev Shepherd University of Auckland, NZ 29 Mr John Shortill Geographe Primary School,WA Mrs Judith Sibbald Our Lady of the Rosary School, NSW Ms Kerry Sidaway Western Heights College,VIC Ms Julie Simon Ballajura Comm. College,WA 10 Mr Philip Slattery Catholic Education Office, NSW 15 Ms Ann Small St Andrews College, NSW Mrs Sandra Smart Niddrie Secondary College,VIC Ms Alison Smith Brisbane South District Educ. QLD 24 Ms Barbara Smith ACER Press Ms Dorothy Smith Lauriston Girls' School,VIC 8 Dr Max Smith Department of Education & Training, NSW 29 Mr Vaughan Smith Caulfield Grammar School,VIC Mr Christopher Smyth Ms Susanne Somerville View Bank College,VIC Mr Arthur Soumalias Lakeside Sec. College,VIC Mr Wayne Spradbury St Patrick's College,TAS Mrs Barbara Spurr Christchurch College of Educ, NZ 7 Mrs Carolyn Stanistreet DEST, Canberra Mr Harry Stassinopoulos Seaford 6-12 School, SA Dr Andrew Stephanou ACER, Assessment & Reporting Mrs Jennifer Stirlilng All Saints College, NSW Mr Kerry Stirling Catholic Education Office, Parramatta Mr Christopher Stock St Paul's College,VIC 30 Mr Iain Stoddart Deans Marsh Primary School,VIC Br Robert Sutton St Patrick's Marist College, NSW Mrs Margaret Swain Alinjarra Primary School,WA 18 Mr Gary Swayn Education QLD

Using Data to Support Learning 179 Mr William Sweeney Prince Alfred College, SA Ms Susan Swift Department of Education,WA Ms Debra Talbot Association of Independent Schools NSW 28 Mrs Jenny Tap Ravenswood School for Girls, NSW Ms Carmel Tapley CSO, NSW 17 Mr David Taplin Newington College, NSW 13 Mrs Merryn Taplin Sydney Grammar School, NSW 12 Ms Margaret Taylor ACER, Professional Learning 7 Ms Elizabeth Tchacos DEST, Canberra 20 Ms Wendy Teasdale-Smith Aberfoyle Park High School, SA Ms Vivienne Tellefson Brunswick Secondary College,VIC 22 Ms Erica Thomas Kincoppal-Rose Bay, NSW 25 Mrs Jan Thomas Albany Junior High School, NZ Mr Ian Thompson Overnewton Anglican Comm. College,VIC Mrs Julie Thompson Assoc. of Independent Schools, NSW 20 Mr David Thomson Chancellor State College, QLD 6 Mr Nick Thornton ACER, Australian Principals Centre Mr Phillip Tierney St Joachim's School,VIC Ms Maureenna Tilley St James' Primary School,VIC 3 Dr. Helen Timperley University of Auckland Ms Lidia Tizian Box Forest College,VIC Ms Rosa Tocchet St Thomas More Primary,Vic Dr Jim Tognolini ACER, System & School Testing 24 Mr Christopher Tome Orange High School, NSW Mr Michael Tonnet Holy Spirit Primary, NSW 31 Mrs Clare Touhy Beaconhills College,VIC 8 Ms Lyn Tozer Dunedin College of Education, NZ Dr Deborah Trengove St Leonard's College,VIC 16 Mrs Jose Trimble Holy Spirit School, Geelong Ms Maureen Trotter Monbulk College,VIC Mr Michael Tucker King's College,VIC 15 Mrs Cheryl Turner Tara Anglican School for Girls, NSW Mr David Turner Kenilworth State School, QLD 25 Ms Dianne Turner Sevenoaks Senior College,WA 19 Ms Lis Turner Department of Education & Training,WA 38 Mr Martin Turner University of Auckland, NZ 28 Dr Pauline Turner The Geelong College,VIC 9 Mr Ross Turner ACER, National & International Surveys Mr David Tyson Wantirna College,VIC Ms Liz Vanden Brink Seaford 6-12 School, SA 29 Mrs Margaret Varady Sydney Girl's High School, NSW 21 Dr Peter Veenker Canberra Institute of Tech. 30 Mrs Sue Veling Our Lady of Mt Carmel, NSW

Research Conference 2005 180 22 Ms Maria Versace OLMC Parramatta, NSW 39 Mr Nic Vidot Catholic Education Office, Parramatta Ms Ann-Marie Vine Wadalba Community School, NSW 39 Mrs Rosalie Vine Catholic Education Office, Parramatta 7 Ms Anna Vlass Box Hill Senior Sec. College,VIC Mrs Anne Walker Fernworth Primary School, NZ 32 Mr Barry Walker Nayland College, NZ Mrs Anne Wallington Lauriston Girls' School,VIC 39 Mr John Walsh Catholic Education Office, Parramatta Mr Stephen Walsh St Leo's Catholic College, NSW 10 Mrs Sue Walsh Catholic Education Office, NSW Mrs Patrea Walton Corinda State High School, QLD Mrs Sylvia Walton Tintern Schools,VIC 12 Mrs Glenice Warner Parkdale Secondary College,VIC Mr Steve Warner Wembley Primary School,VIC 8 Mr David Wasson Department of Education & Training, NSW Mr Jim Watterston DE&T, Eastern Region,VIC Mr Phillip Waugh Aust. International School, Hong Kong 22 Mrs Evelyn Waygood St Vincent's College, NSW 12 Mrs Helen Weadon Damascus College,VIC Mrs Judith Weavers Massey University CED, NZ Mr Stephen Webber St Peter's College, SA 1 Ms Di Weddell DEST, Canberra 1 Mr Peter Weddell National Awards for Quality Schooling, ACT 25 Mr Mark Weegberg Our Lady of Sion College,VIC 36 Mr Dirk Wellham Anglican Church Grammar School, QLD Mr Andrew Wells DECS, SA Mr Ward Westphal Department of Education, ACT Ms Jo-Ann Wheeler Merri Creek Primary School,VIC Ms Helen Whelan Assoc. of Indep. Schools of SA 5 Br Tony Whelen Catholic Schools Office, NSW 2 Ms Wendy Whitham DEST, Canberra 34 Ms Miriam Wilde DECS, SA 3 Dr Helen Wildy Murdoch University,WA Mr Adrian Wiles Luther College,VIC 31 Mrs Jenny Williams Beaconhills College,VIC Mrs Janelle Wills Matthew Flinders Angl. College, QLD 39 Mrs Barbara Wilson University of Auckland, NZ Mr Paul Woodham Department of Education,TAS 13 Mrs Carolyn Woodhouse Footscray City College,VIC Mr Keith Woodward DE&T Nothern Region,VIC 38 Ms Jodie Woolcock Dept. of Employ, Educ. & Training, NT 41 Mr Andy Worrall Multi Serve Education Trust, NZ

Using Data to Support Learning 181 Ms Dionne Wright Merri Creek Primary School,VIC Mrs Marg Wright Christchurch College of Educ., NZ 19 Ms Susan Wright DET, NSW 29 Mr Warrick Wynne MLC,VIC 30 Mrs Michelle Yager Chisholm Primary School, NSW Mr Aaron Young Department of Education & Training,WA 28 Mr Alec Young Flickwjick Pty Ltd,TAS Ms Cynthia Young Hawthorn Secondary College,VIC

702 delegates listed as of Monday 18 July 2005

Research Conference 2005 182