DEELSA/PISA/BPC/M(99)1 OECD PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT (PISA) Seventh meeting of the Board of Participating Countries 4-5 October 1999, Paris SUMMARY OF MAIN OUTCOMES Introduction 1. The seventh meeting of the Board of Participating Countries (BPC) supervising the OECD Programme for International Student Assessment (PISA), was held 4-5 October 1999 at the OECD Headquarters in Paris. 2. The primary objectives of this meeting were to: • Review the outcomes from the PISA field trial. • Establish the review and decision making process through which the PISA main study assessment instruments will be developed and evaluated and to provide guidance for the development of the main study questionnaires. • Decide on procedures for the public release of PISA assessment materials with the aim to find an appropriate balance between: i) satisfying public interest in the PISA assessment materials; ii) ensuring the continued security of all assessment materials that are used in the main study or that may be used in future survey cycles; and iii) ensuring that the different stakeholders in participating countries, including schools, have equal access to these materials. • Review the draft terms of reference for the second PISA survey cycle, to establish priorities for the further development of existing and new assessment domains, to review the proposed work task deliverables and management deliverables, and to establish the timeline for the tendering process. • Establish the PISA budget for 2000. 3. The meeting was attended by delegates from Australia, Austria, Belgium, Canada, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The meeting was chaired by Mr. Owen from the United States. 1 DEELSA/PISA/BPC(98)32 4. Following the opening of the meeting by Mr. Owen, the adoption of the agenda [doc. ref. DEELSA/PISA/BPC(99)A2] and the adoption of the summary record from the fifth meeting of the BPC [doc. ref. DEELSA/PISA/BPC(99)M1], Mr. Alexander reported on: − Progress in the external evaluation of PISA and the composition of the PISA Scientific Review Panel. − Plans for the next INES General Assembly, to be held 13-15 September 2000 in Tokyo and the role and importance of PISA for this General Assembly. − Progress in the development of the International Life Skills Survey (ILSS) as well as on work undertaken under the programme on the Definition and Selection of Competencies (DESECO). − Options for the future development of PISA, highlighting the perspectives of longitudinal survey components which would allow PISA to examine key transitions within education systems and between education and employment, the incidence and effects of early school leaving, the contribution of work experience to skill development and the attitudes, aspirations and behaviour of youth entering the labor market. 5. Afterwards, Mr. Schleicher reported on progress in the implementation of PISA [doc. Ref. DEELSA/PISA/BPC(99)12], including: • the finalisation and verification of the field trial instruments; • the implementation of the PISA field trial in participating countries; • the verification and analysis of the field trial data; • the continued development of the PISA dissemination strategy, • the development of the main study student and school context questionnaires; • the development of a first draft of the terms of reference for the second PISA survey cycle. • the outcomes of a meeting of national and international experts and administrators involved in the implementation of PISA that had been held 4-6 June 1999 with the objective to identify strengths and weaknesses in the item and test development process, the PISA survey operations, quality standards and procedures for quality assurance, and the PISA communication mechanisms. 6. Mr. Schleicher also explained how the issues that had been raised by the BPC at its last meeting had been addressed and led participants through the areas requiring further resolution by the BPC. 7. The BPC: − WELCOMED the progress made in the implementation of PISA to date and the ways in which the decisions and recommendations made by the BPC at its last meeting had been implemented. 2 DEELSA/PISA/BPC/M(99)1 • NOTED the report on the linguistic equivalence of the national assessment instruments [doc. Ref. DEELSA/PISA/BPC(99)16]. Development of the main study assessment instruments 8. Mr. Schleicher introduced a proposal designed to: i) determine whether the field trial assessment satisfies the requirements for PISA and to ii) guide the selection of items for the main study and the evaluation of the main study assessment instruments [doc. Ref. DEELSA/PISA/BPC(99)14]. 9. The BPC: • WELCOMED the proposal as an adequate basis for the development and assessment of the main study instruments. • NOTED the role of the PISA cultural review panel in verifying that the PISA main study assessment instruments do not violate accepted cultural positions in participating countries, the role of the consortium and the PISA functional expert groups in establishing the technical quality of the assessment instruments and the role of National Project Managers in finalising the selection of items at their meeting 22-26 November 1999. • ASKED the BPC Executive Group to review at the beginning of December whether the proposed steps were implemented appropriately and to report to the BPC on the outcomes of this review before 10 December 1999. 10. Mr. Adams reported on the initial outcomes from the analysis of the field trial data. 11. The BPC: • NOTED with satisfaction that, with exception of Competency Class 3 in the Mathematics domain, the field trial item pool provides an appropriate coverage of the PISA assessment frameworks, that detailed item audits are available that document the quality of the items and that appropriate mechanisms are in place to ensure that the match between the frameworks and the item pool will still be adequate after criteria relating to the cultural appropriateness and the psychometric properties of the items have been applied. Dissemination strategy for PISA outcomes 12. Mr. Schleicher introduced the PISA dissemination strategy that had been developed in the light of the BPC’s discussions at its meeting 1-3 March 1999 [doc. Ref. DEELSA/PISA/BPC(99)10]. 13. The BPC: − WELCOMED the proposals as an appropriate basis for the development of PISA reports and publications. • SUGGESTED that the thematic reports should take a policy-oriented perspective and be presented in an accessible format and that they should place PISA outcomes in the context of other qualitative and quantitative work at the OECD. 3 DEELSA/PISA/BPC(98)32 • SUGGESTED that the initial international report be accompanied by a background paper that would explain the objectives and nature of the PISA frameworks and instruments, describing and illustrating with sample items what is measured and what is not measured in the first survey cycle. • SUGGESTED that the thematic reports should place greater emphasis on the information obtained through the international option on “self-regulated learning”. • NOTED the importance of a simultaneous release of the international and national initial reports which would allow for a contextualised discussion of the findings from the international analyses and ASKED the Secretariat to clarify the timing for the release of the data that would be required for the preparation of national reports. • NOTED that it was planned to publish one thematic report in 2001, three in 2002 and two in 2003, with the sequencing of the publications determined by the priorities identified by the BPC, the methodological complexity of the required analyses and the extent to which external data would need to be utilised. • NOTED that the international data set was scheduled to be released at the beginning of 2002 and that this dataset would allow participating countries to pursue analyses and reports beyond the international publications. • NOTED that the primary resource for the development of the thematic reports would be the international experts from the PISA Functional Expert Groups and the PISA Scientific Review Panel but suggested that also outside expertise and key educational stakeholders be involved in the preparation of the reports. − SUGGESTED that the proposed optional report on the profile of reading literacy be incorporated as an integral part of the PISA reporting plans and that it should include an in- depth review of the definition and scope of the PISA literacy concept as defined in PISA framework with considerations of its internal and external validity. − NOTED that it was important for the reports to provide a perspective that goes beyond a labour market focus and exploits the full potential of the PISA instruments. − NOTED that the different nature of the PISA and IALS instruments would limit the possibility of comparing the knowledge and skills of 15 year-olds with those of the adult population. • NOTED the importance of presenting outcomes in ways that are interpretable for policy makers and, in particular, to express student achievement through descriptive proficiency levels. • SUGGESTED that the reports on the relationship between social background and student achievement should have a stronger systemic perspective. • UNDERLINED the importance of the report on school factors related to quality and equity while noting the analytic difficulties involved in estimating school effects, given the age- based design of PISA and the varying institutional structures in participating countries. 4 DEELSA/PISA/BPC/M(99)1 • SUGGESTED that for the report on school factors, aspects of the interaction between gender and school factors as well as the impact of student motivation should be considered. • AGREED that the thematic report on the organisation of learning should be deferred to the second survey cycle when a domain with a closer relationship to the curriculum would be at the centre and when information on learning and instruction would be given greater weight in the context questionnaires.