DEELSA/PISA/BPC/M(99)1 1 OECD PROGRAMME for INTERNATIONAL STUDENT ASSESSMENT (PISA) Seventh Meeting of the Board of Participatin

DEELSA/PISA/BPC/M(99)1

OECD PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT (PISA)

Seventh meeting of the Board of Participating Countries 4-5 October 1999, Paris

SUMMARY OF MAIN OUTCOMES

Introduction

1. The seventh meeting of the Board of Participating Countries (BPC) supervising the OECD Programme for International Student Assessment (PISA), was held 4-5 October 1999 at the OECD Headquarters in Paris.

2. The primary objectives of this meeting were to:

• Review the outcomes from the PISA field trial.

• Establish the review and decision making process through which the PISA main study assessment instruments will be developed and evaluated and to provide guidance for the development of the main study questionnaires.

• Decide on procedures for the public release of PISA assessment materials with the aim to find an appropriate balance between: i) satisfying public interest in the PISA assessment materials; ii) ensuring the continued security of all assessment materials that are used in the main study or that may be used in future survey cycles; and iii) ensuring that the different stakeholders in participating countries, including schools, have equal access to these materials.

• Review the draft terms of reference for the second PISA survey cycle, to establish priorities for the further development of existing and new assessment domains, to review the proposed work task deliverables and management deliverables, and to establish the timeline for the tendering process.

• Establish the PISA budget for 2000.

3. The meeting was attended by delegates from Australia, Austria, Belgium, Canada, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The meeting was chaired by Mr. Owen from the United States.

1 DEELSA/PISA/BPC(98)32

4. Following the opening of the meeting by Mr. Owen, the adoption of the agenda [doc. ref. DEELSA/PISA/BPC(99)A2] and the adoption of the summary record from the fifth meeting of the BPC [doc. ref. DEELSA/PISA/BPC(99)M1], Mr. Alexander reported on:

− Progress in the external evaluation of PISA and the composition of the PISA Scientific Review Panel.

− Plans for the next INES General Assembly, to be held 13-15 September 2000 in Tokyo and the role and importance of PISA for this General Assembly.

− Progress in the development of the International Life Skills Survey (ILSS) as well as on work undertaken under the programme on the Definition and Selection of Competencies (DESECO).

− Options for the future development of PISA, highlighting the perspectives of longitudinal survey components which would allow PISA to examine key transitions within education systems and between education and employment, the incidence and effects of early school leaving, the contribution of work experience to skill development and the attitudes, aspirations and behaviour of youth entering the labor market.

5. Afterwards, Mr. Schleicher reported on progress in the implementation of PISA [doc. Ref. DEELSA/PISA/BPC(99)12], including:

• the finalisation and verification of the field trial instruments;

• the implementation of the PISA field trial in participating countries;

• the verification and analysis of the field trial data;

• the continued development of the PISA dissemination strategy,

• the development of the main study student and school context questionnaires;

• the development of a first draft of the terms of reference for the second PISA survey cycle.

• the outcomes of a meeting of national and international experts and administrators involved in the implementation of PISA that had been held 4-6 June 1999 with the objective to identify strengths and weaknesses in the item and test development process, the PISA survey operations, quality standards and procedures for quality assurance, and the PISA communication mechanisms.

6. Mr. Schleicher also explained how the issues that had been raised by the BPC at its last meeting had been addressed and led participants through the areas requiring further resolution by the BPC.

7. The BPC:

− WELCOMED the progress made in the implementation of PISA to date and the ways in which the decisions and recommendations made by the BPC at its last meeting had been implemented.

2 DEELSA/PISA/BPC/M(99)1

• NOTED the report on the linguistic equivalence of the national assessment instruments [doc. Ref. DEELSA/PISA/BPC(99)16].

Development of the main study assessment instruments

8. Mr. Schleicher introduced a proposal designed to: i) determine whether the field trial assessment satisfies the requirements for PISA and to ii) guide the selection of items for the main study and the evaluation of the main study assessment instruments [doc. Ref. DEELSA/PISA/BPC(99)14].

9. The BPC:

• WELCOMED the proposal as an adequate basis for the development and assessment of the main study instruments.

• NOTED the role of the PISA cultural review panel in verifying that the PISA main study assessment instruments do not violate accepted cultural positions in participating countries, the role of the consortium and the PISA functional expert groups in establishing the technical quality of the assessment instruments and the role of National Project Managers in finalising the selection of items at their meeting 22-26 November 1999.

• ASKED the BPC Executive Group to review at the beginning of December whether the proposed steps were implemented appropriately and to report to the BPC on the outcomes of this review before 10 December 1999.

10. Mr. Adams reported on the initial outcomes from the analysis of the field trial data.

11. The BPC:

• NOTED with satisfaction that, with exception of Competency Class 3 in the Mathematics domain, the field trial item pool provides an appropriate coverage of the PISA assessment frameworks, that detailed item audits are available that document the quality of the items and that appropriate mechanisms are in place to ensure that the match between the frameworks and the item pool will still be adequate after criteria relating to the cultural appropriateness and the psychometric properties of the items have been applied.

Dissemination strategy for PISA outcomes

12. Mr. Schleicher introduced the PISA dissemination strategy that had been developed in the light of the BPC’s discussions at its meeting 1-3 March 1999 [doc. Ref. DEELSA/PISA/BPC(99)10].

13. The BPC:

− WELCOMED the proposals as an appropriate basis for the development of PISA reports and publications.

• SUGGESTED that the thematic reports should take a policy-oriented perspective and be presented in an accessible format and that they should place PISA outcomes in the context of other qualitative and quantitative work at the OECD.

3 DEELSA/PISA/BPC(98)32

• SUGGESTED that the initial international report be accompanied by a background paper that would explain the objectives and nature of the PISA frameworks and instruments, describing and illustrating with sample items what is measured and what is not measured in the first survey cycle.

• SUGGESTED that the thematic reports should place greater emphasis on the information obtained through the international option on “self-regulated learning”.

• NOTED the importance of a simultaneous release of the international and national initial reports which would allow for a contextualised discussion of the findings from the international analyses and ASKED the Secretariat to clarify the timing for the release of the data that would be required for the preparation of national reports.

• NOTED that it was planned to publish one thematic report in 2001, three in 2002 and two in 2003, with the sequencing of the publications determined by the priorities identified by the BPC, the methodological complexity of the required analyses and the extent to which external data would need to be utilised.

• NOTED that the international data set was scheduled to be released at the beginning of 2002 and that this dataset would allow participating countries to pursue analyses and reports beyond the international publications.

• NOTED that the primary resource for the development of the thematic reports would be the international experts from the PISA Functional Expert Groups and the PISA Scientific Review Panel but suggested that also outside expertise and key educational stakeholders be involved in the preparation of the reports.

− SUGGESTED that the proposed optional report on the profile of reading literacy be incorporated as an integral part of the PISA reporting plans and that it should include an in- depth review of the definition and scope of the PISA literacy concept as defined in PISA framework with considerations of its internal and external validity.

− NOTED that it was important for the reports to provide a perspective that goes beyond a labour market focus and exploits the full potential of the PISA instruments.

− NOTED that the different nature of the PISA and IALS instruments would limit the possibility of comparing the knowledge and skills of 15 year-olds with those of the adult population.

• NOTED the importance of presenting outcomes in ways that are interpretable for policy makers and, in particular, to express student achievement through descriptive proficiency levels.

• SUGGESTED that the reports on the relationship between social background and student achievement should have a stronger systemic perspective.

• UNDERLINED the importance of the report on school factors related to quality and equity while noting the analytic difficulties involved in estimating school effects, given the age- based design of PISA and the varying institutional structures in participating countries.

4 DEELSA/PISA/BPC/M(99)1

• SUGGESTED that for the report on school factors, aspects of the interaction between gender and school factors as well as the impact of student motivation should be considered.

• AGREED that the thematic report on the organisation of learning should be deferred to the second survey cycle when a domain with a closer relationship to the curriculum would be at the centre and when information on learning and instruction would be given greater weight in the context questionnaires.

− SUGGESTED that the report meeting the needs of low and high achieving students should focus on successful performance under adversary circumstances.

− WELCOMED the PISA newsletter and brochure as useful instruments for the ongoing dissemination of PISA related information.

Public release of PISA assessment material

14. Mr. Schleicher introduced proposals for the public release of a small set of field trial test items which would aim to provide a balanced and appropriate representation of important aspects of the PISA frameworks and instruments and which could be used by participating countries to describe and illustrate the PISA instruments. Technically, this initial set of illustrative items could be made available at the beginning of February 2000. It would include items from the field trial that will not be used in the PISA main study but which still meet PISA requirements of subject-matter and cultural appropriateness and technical quality.

15. The BPC:

• NOTED the importance of publicly accessible sample items that countries could use to illustrate and communicate the nature of the PISA instruments to the national scientific and policy communities.

• WELCOMED the proposal presented by the Secretariat [doc. Ref. DEELSAS/PISA/BPC(99)15] as providing an appropriate balance between: i) satisfying public interest in the PISA assessment materials; ii) ensuring the continued security of all assessment materials that are used in the main study or that may be used in future survey cycles; and iii) ensuring that the different stakeholders in participating countries, including schools, have equal access to these materials.

• NOTED that the sample items should represent the diversity of PISA item types, cover a range of aspects from the frameworks and cover a range of item difficulties.

• SUGGESTED the release of the sample assessment materials in a contextualised form in which it would become clear how the items relate to the different dimensions and aspects in the PISA assessment frameworks.

• NOTED that it will be desirable to ensure that the different stakeholders in PISA, including participating schools, have equal access to these materials and ASKED the Secretariat to facilitate this by making the set of illustrative items available in printed form as well as in the form of an interactive web site.

5 DEELSA/PISA/BPC(98)32

• ASKED the Secretariat to explore whether the set of illustrative items could be released earlier than in February 2000.

• COMMITED to ensure that no assessment material beyond the items that have been internationally designated for public release will be made public in participating countries.

• ASKED the Secretariat to provide PISA National Project Managers with guidelines and instructions that would describe which types of national and international PISA related documentation could be released publicly and which not.

Context questionnaires and analysis plans

16. Ms. Tamassia reported from the PISA questionnaire review meeting that had been held 13-14 September 1999 in Luxembourg and which had brought together national and international experts and the consortium with the aim to review the outcomes from the field trial and to establish options for the further development of the PISA context questionnaires.

17. Mr. Adams reported on progress in the development of the PISA context questionnaires and on efforts underway to better focus the context questionnaires and to achieve greater depth and clarity in areas of particular relevance to the BPC’s policy priorities. Mr. Adams also outlined the constructs and instruments that are being proposed to address the BPC’s reporting plans [doc. Ref. DEELSA/PISA/BPC(99)17].

18. The BPC:

− WELCOMED progress made in the development of questionnaires that respond to the analytic needs of PISA.

− WELCOMED the criteria proposed by the consortium for the construction of the main study questionnaires (the questionnaires should contribute to indicators of enduring cross-national relevance, focus on areas that are amenable to policy development, be applicable in most participating countries and be based on standardised cross-national coding schemes).

− REQUESTED that a framework for the context questionnaires be developed as quickly as possible that would include information on how the information collected through the context questionnaires would be used for the analyses required for the PISA thematic reports.

− WELCOMED the conceptual work on the definition and measurement of the economic, social and cultural capital of students and their families as well as the measurement of motivation, family structure and home engagement but ASKED the Secretariat to document the measures that are being developed for this purpose and to assess their validity and reliability (considering also the outcomes of validity studies in participating countries).

− NOTED the desire of some countries to shorten testing time for individual students but DECIDED to maintain the current scope and length of the instruments.

• NOTED the difficulties in measuring school career pathways in an international comparative fashion, given the differences in national institutional structures, but ASKED the Secretariat to work with interested countries on the alignment of corresponding national options.

6 DEELSA/PISA/BPC/M(99)1

• NOTED that the added value of rotating the PISA student context questionnaires for analytical purposes would be limited, given remaining questions on the quality of the components designated for rotation, CONSIDERED that the increased costs of rotation in terms of complexity and resource requirements were not warranted, and therefore DECIDED to use a single form for the student context questionnaires in the main study.

Development of the terms of reference for the second PISA survey cycle

19. Mr. Owen reported from a planning meeting of the INES Network A that had been held 20-21 May 1999 with the aim to establish options for the design and development of the second survey cycle of PISA.

20. Afterwards, Mr. Owen introduced the draft terms of reference for the second survey cycle that had been developed by Network A [doc. Ref. DEELSA/PISA/BPC(99)18] and which comprised the technical aspects of the project, arrangements regarding the contractor’s role and required activities and the proposed schedule, work task deliverables and management deliverables.

21. The BPC:

− WELCOMED the draft as an appropriate basis for the development of the call for tender for the second PISA survey cycle and agreed to submit further comments in writing to the Secretariat before 25 October 1999.

− THANKED Mr. Alexander for the successful translation of the political commitment of Member countries to evaluate the outcomes of education within an international comparative framework into a unique institutional framework designed to provide valid, reliable and comparable data on student achievement on a regular basis.

− WELCOMED the proposal to develop a coherent and integrated framework for reading, mathematics, and science during the second survey cycle that would operate within the concept of literacy that was adopted by the Board of Participating Countries but review the definitions and revise and expand them as necessary in order that they are aligned across the domains.

− WELCOMED the proposal to place the assessment of mathematical literacy in the second survey cycle within the curricular context of participating education systems but ASKED Network A to clarify the objectives and nature of the proposed curriculum analysis.

− WELCOMED the proposal to launch the tendering process already in April 2000 (with the evaluation of bids during June, the selection of the contractor during July/early August and the negotiation and award of the contract by September 2000), thus allowing for more time for conceptual and instrument development during the second survey cycle.

− NOTED that the Network A data strategy provided for the assessment of a broader range of knowledge and skills than are currently implemented in PISA but EMPHASISED that it was the role of the BPC to decide which of the assessment domains would be taken up in which of the PISA survey cycles.

7 DEELSA/PISA/BPC(98)32

− AGREED that PISA should continue to progressively integrate the assessment of cross- curriculum competencies into the survey instrument and CONSIDERED that the assessment of problem-solving skills would be an appropriate goal for the second survey cycle and should therefore form part of the international call for tender.

− NOTED that the feasibility to assess problem-solving skills within the existing survey design and budgetary framework for PISA would only be known once the bids to the international call for tender become available and therefore DECIDED to defer a decision on the implementation of an assessment of problem-solving skills, as well as a decision on the methods through which such an assessment would best be undertaken (e.g. as an integral part of PISA or as an international option) to when the bids become available.

− NOTED the need to find an appropriate balance between, on the one hand, continuity in the framework and design of PISA in order to establish reliable trend lines for educational outcomes and on the other hand leaving sufficient room for innovation,.

− NOTED the need to co-ordinate developmental work for the second survey cycle with related efforts in this area, in particular the International Life Skills Survey.

− NOTED that the assessment of reading as a foreign language based on the existing PISA reading instruments would only partially address the desire of countries to assess the language skills of students and CONSIDERED that the utility and feasibility of this proposal could only be assessed once the bids to the international call for tender become available and that, if exercised, such an assessment should become an international option.

− WELCOMED the proposal for an international option of over-sampling the PISA student population in the second survey cycle in order to administer a longitudinal follow-up student questionnaire (through a household survey) in subsequent cycles of PISA but suggested that the development of instruments for the longitudinal follow-up should be the subject of a separate call for tender.

− NOTED the growing interest among non-Member countries to participate in PISA and ASKED the Secretariat to develop a strategy for the participation of non-Member countries for review by the BPC at its next meeting.

− DECIDED to maintain the existing PISA policy for providing the assessment material in two source languages but SUGGESTED that bidders review the PISA translation procedures to best accomplish this.

Election of Vice Chair

22. The BPC elected Ms. Chiara Croce (Italy) as a Vice Chair for the BPC for a period of three years as per the operating rules of the BPC, replacing Mr. Guillermo Gil (Spain).

Adoption of the Budget

23. Mr. Schleicher introduced the proposed budget for the calendar year 1999.

8 DEELSA/PISA/BPC/M(99)1

24. The BPC:

− Unanimously ADOPTED the PISA budget for the calendar year 2000 [doc. ref. DEELSA/PISA/BPC(99)19] and committed to the country contributions listed in column 10 of Table 2 in document DEELSA/PISA/BPC(99)19;

− DECIDED that the 2000 PISA budget should continue to be based on the existing PISA scale of country contributions in order to ensure that country contributions for 2000 would not exceed the estimates that had been provided in 1997.

− NOTED that country contributions were due in January 1999 but ASKED the Secretariat to continue working with countries on identifying flexible arrangements for the mode and timing of the payment of country contributions;

− AGREED to submit the proposed budget to the Budget Committee of the OECD.

Next meeting

25. The next meeting of the BPC will be held 13-15 March 2000.