PISA Sample Tasks, Texts, Item Statistics, and Item Maps for Mathematics, Reading Literacy, Science, and Problem Solving
Total Page:16
File Type:pdf, Size:1020Kb
Sample tasks from PISA from http://www.erc.ie/pisa PISA Sample Tasks, Texts, Item Statistics, and Item Maps for Mathematics, Reading Literacy, Science, and Problem Solving Table of Contents Interpreting the Tables of the PISA Item Statistics 2 PISA Item Difficulty 2 Example Interpretations of PISA Sample Items 4 Mathematics 6 Unit: Exchange Rate 6 Unit: Staircase 8 Unit: Exports 9 Unit: Growing Up 11 Unit: Skateboard 13 Unit: Number Cubes 16 Unit: Walking 18 Unit: Test Scores 20 Unit: Carpenter 21 Unit: Robberies 22 Unit: Internet Relay Chat 23 Unit: Earthquake 25 Reading Literacy 26 Unit: The Gift 26 Unit: Labour Force 32 Science 36 Unit: Semmelweis 36 Unit: Ozone 40 Problem Solving 43 Unit: Cinema Outing 43 Unit: Library System 45 Unit: Irrigation 48 1 Interpreting the Tables of the PISA Item Statistics Sample tasks and items for mathematics and problem solving have been adapted from the OECD's initial reports for PISA 2003 (OECD, 2004a, 2004b). The layout has been compacted somewhat and is not identical to that presented to students in the test booklets. As no new items for reading literacy or science were released following the PISA 2003 assessment, sample items for these domains are drawn from a pool of items released following the 2000 assessment (see OECD, 2001; Shiel et al., 2001). Sample tasks for mathematics originate from both PISA 2000 and PISA 2003. The PISA cycle from which the task originates is given in brackets after the title. For items shown, their classification in the framework is given (e.g., content area, process, context and item type). The science items are categorised according to science processes as described in the PISA 2000 framework. These have been changed since 2000. Since there is no one-to-one match between the two sets of processes, the science items shown here are classified by the 2000 framework. The first and second PISA 2000 science processes, 'recognising questions that can be answered via scientific research' and 'identifying scientific evidence', are similar to the PISA 2003 science process of 'understanding scientific investigation'. The third PISA 2000 science process, 'applying scientific knowledge in the situation presented', is similar to the PISA 2003 science process of 'describing, explaining and predicting scientific phenomena', although some of these items might also be classified as 'interpreting scientific evidence and conclusions'. The fourth PISA 2000 science process, 'critically evaluating scientific evidence/data', is similar to 'interpreting scientific evidence and conclusions'. The item statistics that are provided here include the percentage of students providing a correct response, the percentages giving an incorrect response and the percentages not responding, for Ireland and the OECD. These statistics are from the 2000 and 2003 PISA databases (http://www.pisa.oecd.org). Item difficulties are reported in terms of the proficiency levels at which the items are located in case of mathematics, reading, and problem solving. In cases where an item offers both partial credit (PC) and full credit (FC), item difficulties and proficiency levels for both levels of credit are given. Marking guides for the questions are also given. PISA Item Difficulty For each item, whether from 2000 or 2003, the information under ‘PISA Item Difficulty’ gives the scale score that represents the location of the item on the relevant achievement scale (in terms of average performance of a fixed number of pupils drawn from each OECD country). Each scale has a mean of 500.0 and a standard deviation of 100.0. The reader is referred to the four item maps that follow, for examples of how items can be located along achievement/proficiency scales. 2 Irish 25th percentile Irish 75th percentile (445.0) (561.9) Irish Mean y (502.8) lit i b Irish 10th percentile Irish 90th percentile (393.1) OECD Mean nt a (613.9) (500.0) Stude 358 359 420 421 482 483 544 545 606 607 668 669 Below Lev 1 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 358 359 420 421 482 483 544 545 606 607 668 669 lty Exports Q1 (426.3) Exports Q2 (565.0) Staircase Q1 Number Cubes Q2 Walking Q1 (611.0) (420.9) Item Difficu (503.5) The PISA 2003 Combined Mathematics Scale: Cut-points for Proficiency Levels, Scores of Students in Ireland at Key Markers, and Difficulties of Selected Items Irish 25th Irish 75th percentile Irish Mean percentile (460.2) (515.5) (576.9) y OECD Irish 90th lit i Irish 10th Mean percentile b percentile (494.2) (622.1) nt a (401.3) Stude 335 336 407 408 480 481 552 553 625 626 Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 335 336 407 408 480 481 552 553 625 626 lty Labour Q1 (476.6) Gift Q3 partial credit (538.4) Item Difficu Gift Q4 Gift Q2 Labour Q2 full credit (631.1) (366.6) (447.5) The PISA 2003 Reading Literacy Scale: Cut-points for Proficiency Levels, Scores of Students in Ireland at Key Markers, and Difficulties of Selected Items 3 Irish 25th percentile Irish Mean Irish 75th percentile (441.8) (505.4) (571.9) Irish 10th percentile OECD Mean Irish 90th percentile y (383.9) (499.6) (624.5) lit i b nt a Stude 353 354 419 420 574 575 636 637 Below 10pc 10-25pc 26-75pc 76-90pc Above 90pc 353 354 419 420 574 575 636 637 lty Semmelweis Q1 Partial Credit Semmelweis Q2 (494.9) (625.7) Semmelweis Q3 (471.4) Semmelweis Q1 Full Credit (650.1) Item Difficu The PISA 2003 Science Scale: Cut-points for Key Percentile Intervals, Scores of Students in Ireland at Key Markers, and Difficulties of Selected Items Irish 10th Irish 25th Irish Mean y percentile percentile (498.5) Irish 90th lit i (394.6) (444.9) Irish 75th percentile b percentile (600.7) OECD Mean nt a (555.3) (500.0) Stude 405 406 499 500 592 593 Below Level 1 Level 1 Level 2 Level 3 405 406 499 500 592 593 Library System Q2 partial lty Irrigation Q3 (531.3) Library System Q1 credit (658.1) (436.8) Irrigation Q1 (497.2) Library System Q2 full credit (692.9) Item Difficu The PISA 2003 Problem Solving Scale: Cut-points for Proficiency Levels, Scores of Students in Ireland at Key Markers, and Difficulties of Selected Items 4 The raw item difficulties (Deltas) were transformed to the 500–100 scales which assume a response probability of 0.62 using the following linear transformations: • (((Delta – (-0.1344))/1.2838)*100) + 500 for mathematics • (((Delta – (0.5076))/1.1002)*100) + 500 for reading • (((Delta – (-0.0933))/1.1086)*100) + 500 for science • (((Delta – (-0.0973))/1.1751)*100) + 500 for problem solving. Example Interpretations of PISA Sample Items In the sample mathematics item Exchange Rate Question 1, the item difficulty across OECD countries is 406.1 points. This means that the item is located almost a full standard deviation below the OECD mean of 500.0. Further, since the item is between 358.3 and 420.4 score points on the mathematics scale, the item is at proficiency Level 1. The OECD average percentage correct for the item is 79.7%, while, in Ireland, it is slightly higher at 83.2%. The OECD average ‘missingness’ for the item is 6.6%, while in Ireland, it is 3.5%. The sample mathematics item Growing Up Question 2 is an item on which both partial and full credit are available, depending on the completeness of a student’s answer. The fully- correct version of the item has a difficulty estimate of 525.3, while the partially-correct version has an estimate of 419.3. The full-credit estimate is at proficiency Level 3, while the partial credit version is on the border between Levels 1 and 2. On average across OECD countries, 54.7% of students received full credit for their responses, while in Ireland, 51.5% did so. The corresponding percentages for partial credit were 28.1% and 35.8% respectively. Again, missingness in Ireland (3.4%) is lower than the OECD average level (7.5%). 5 MATHEMATICS UNIT: EXCHANGE RATE (2003) Context: Public. Mei-Ling from Singapore was preparing to go to South Africa for 3 months as an exchange student. She needed to change some Singapore dollars (SGD) into South African rand (ZAR). EXCHANGE RATE QUESTION 1 (Item code: M413Q01) Domain: Quantity. Item type: Short constructed response. Mei-Ling found out that the exchange rate between Singapore dollars and South African rand was: 1 SGD = 4.2 ZAR. Mei-Ling changed 3000 Singapore dollars into South African rand at this exchange rate. How much money in South African rand did Mei-Ling get? Key: Full credit: 12 600 ZAR (unit not required); no credit: Other responses, missing. Process: Reproduction. Understand a simple problem, and link the given information to the required calculation. PISA Item Difficulty Item statistics % OECD % Ireland Scale score Correct 79.7 83.2 406.1 Incorrect 13.8 13.4 Level Missing 6.6 3.5 1 Total 100 100 EXCHANGE RATE QUESTION 2 (Item code: M413Q02) Domain: Quantity. Item type: Short constructed response. On returning to Singapore after 3 months, Mei-Ling had 3 900 ZAR left. She changed this back to Singapore dollars, noting that the exchange rate had changed to: 1 SGD = 4.0 ZAR. How much money in Singapore dollars did Mei-Ling get? Key: Full credit: 975 SGD (unit not required); no credit: Other responses, missing. Process: Reproduction. Understand a simple problem, and decide that division is the right procedure to go with.