Table of Contents Welcome from Conference Host – Cambridge ESOL 3 Table of Contents LTRC Cambridge 2010 Conference Organising Committee 4 Message from Chair of LTRC Cambridge 2010 Organising Committee 4 Acknowledgements 6 Address of ILTA Business Office 6 Proposal Reviewers 7 Award Winners 8 Award Committees 9 TIRF Awards for the Language Assessment Research Priority 10 New BAAL SIG for Testing, Evaluation and Assessment (TEA) 10 SAGE Online Access 10 About ILTA 12 Message from the ILTA President 13 Sponsors and Exhibitors 16 LTRC 2011 – University of Michigan 23 Programme Overview 25 Full Programme 30 The Messick Award 44 UCLES/ILTA Lifetime Achievement Award 47 Abstracts 51 Pre-Conference Workshops 51 Symposia 54 Research Papers 61 Posters 90 Works in Progress 100 Presenter Index 111 Additional Information for Delegates 114 In Memoriam – Caroline Clapham 118 Notes 121 Maps 126 32nd Language Testing Research Colloquium, Cambridge – 12–16 April 2010 1 66585585 PP_LTRC_LTRC 22010.indd010.indd 1 229/03/20109/03/2010 115:085:08 2 32nd Language Testing Research Colloquium, Cambridge – 12–16 April 2010 66585585 PP_LTRC_LTRC 22010.indd010.indd 2 229/03/20109/03/2010 115:085:08 Welcome from the Conference Host – Cambridge ESOL Dr Michael Milanovic, Cambridge ESOL On behalf of the University of Cambridge and the Local Organising Committee in my own department, Cambridge ESOL Examinations, I am very pleased to welcome the 32nd Language Testing Research Colloquium to Cambridge. Rather I should say “welcome back” to the many old friends who are here again for the second LTRC in Cambridge. The last time we had the honour of hosting this event was in the Summer of 1993 when the 15th LTRC was hosted in Cambridge and in Arnhem (Netherlands). This was a double first; it was the first time it was held outside North America, and the first (and only time) it was held in two countries! The subsequent internationalisation of LTRC has been a major success, and since 1993 the event has been held in nine countries as far apart as Finland and Australia. I think this is testimony to the worldwide interest in language testing and the importance now placed on high quality tests and professional standards. This is a very important aim for our University and we are very pleased to support this trend. Over the last 17 years we have seen a shift of emphasis in our field, so that the technical aspects of testing which were a main focus in the 1980s and early 1990s are now balanced alongside a growing concern for the social and educational aspects of language assessment. Welcome from the Conference Host – Cambridge ESOL In the Local Organising Committee we wanted to come up with a theme which could encompass this broad perspective and allow for a multi-faceted conference with something to appeal to everyone, no matter what angle you come from. The title we chose, Crossing the threshold: investigating levels, domains and frameworks in language assessment most obviously conjures up the literal notions of language levels and proficiency frameworks – such as the CEFR. This is clearly represented in the programme and in the pre- conference workshops. I am pleased to note, however, that papers have also been accepted which treat this theme in a more figurative sense, where “crossing the threshold” refers to the crossing of a physical boundary (e.g. entry into a country or into a profession), and where the domain or framework in question forms part of a social or political system in which language and language assessment play an important part. In this respect too, I think the programme promises to be very varied and no doubt stimulating. I wish you a very pleasant and successful stay in Cambridge and I look forward to meeting many of you during the conference. Michael Milanovic CEO, University of Cambridge ESOL Examinations 32nd Language Testing Research Colloquium, Cambridge – 12–16 April 2010 3 66585585 PP_LTRC_LTRC 22010.indd010.indd 3 229/03/20109/03/2010 115:085:08 LTRC Cambridge 2010 Conference Organising Committee Conference Organisers Dr Lynda Taylor (Chair), University of Cambridge ESOL Examinations (Consultant) Dr Michael Milanovic, University of Cambridge ESOL Examinations (CEO) Dr Nick Saville, University of Cambridge ESOL Examinations Sarah Brydie, University of Cambridge ESOL Examinations Keri Ridout, University of Cambridge ESOL Examinations Xi Xiaoming – ILTA Liaison, Educational Testing Service Sara Cushing Weigle – ILTA Liaison, Georgia State University Robert Ranieri, Prime Management Anna Trott, Prime Management Message from Chair of LTRC Cambridge 2010 Organising Committee Dear friends and colleagues, LTRC Cambridge 2010 Conference Organising Committee As anyone who has ever been involved knows, organising our annual Language Testing Research Colloquium (LTRC) can be a complex and time-consuming endeavour. This year was no exception, and the large and unexpected increase in proposals submitted for the 2010 conference made it even more of a challenge. People sometimes wonder how decisions are made when scheduling the event and selecting the programme content. This year, in the interests of transparency and accountability, I thought it might be helpful to summarise some of the behind-the-scenes planning and decision-making that went on this year which we trust will result in an enjoyable and stimulating conference for all of us. The LTRC 2010 Call for Proposals was launched at the March 2009 event in Denver and appeared on the conference website in April 2009. In that Call for Proposals the Local Organising Committee provided some guidance on the chosen theme, identifying potential questions of interest, highlighting thematic strands and suggesting related sub-themes. Our intention was to offer a broad background canvas with plenty of scope for interpretation but with the aim of achieving a reasonably coherent and cohesive programme. We also offered guidance on what differentiates a symposium from a session of research papers, in the hope that this would avoid the frustration that can arise when a symposium becomes simply a sequence of loosely connected, individual research papers with little time for plenary discussion. We included in the Call for Proposals explicit evaluation criteria, based partly on those used by AAAL, to assist colleagues in conceptualising and crafting their proposal. With an eye on publishing a potential conference proceedings after the event, we also invited people to theme- code their abstract proposals on submission to help us get ahead with the advance planning of such a volume. By the proposal submission deadline on 30 September 2009, we had received a total of 298 abstracts. Though we imagined Cambridge might be an attractive venue, we had certainly not anticipated a 50% increase on submissions from the previous year! The overall submission figure breaks down as follows: 11 symposia, 159 research papers, 76 works-in-progress (WiPs) and 52 posters. This necessitated assembling a team of 40 willing members from right across the international language testing community to peer review and grade up to 16 mixed abstracts each over a period of 3–4 weeks. The LTRC 2010 reviewer team is listed on page 7 and we should be grateful to them for their service to our community. It was particularly good to be able to welcome some first-time reviewers to the team and each of these was paired with a more experienced LTRC reviewer. When assigning abstracts for review at the outset, we took care to avoid any potential conflict of interest; reviewers themselves were also asked to alert us to any case of this on receipt of their 4 32nd Language Testing Research Colloquium, Cambridge – 12–16 April 2010 66585585 PP_LTRC_LTRC 22010.indd010.indd 4 229/03/20109/03/2010 115:085:08 allocation but in only one instance did a reviewer ask us to reassign one of the abstracts allocated to them. Every research paper, WiP and poster submission was blind reviewed and graded independently by 2 reviewers. Each of the 11 symposia, however, was reviewed and graded by 4 independent and experienced reviewers. The whole process was done online; all reviewers were sent a set of guidelines, which reiterated the evaluation criteria together with the usual 4-point rating scale used for grading LTRC abstracts: 1 = Reject; 2 = Possibly accept if space permits; 3 = Good, accept; 4 = Outstanding, definitely accept. Given the large number of submissions for LTRC 2010, reviewers were encouraged to use the full length of the rating scale to help discriminate between stronger and weaker proposals. Following completion of this stage, reviewer grades were averaged to generate a final overall grade for each proposal. All proposals were then rank ordered according to this final grade, i.e. 4, 3.5, 3, 2.5, etc, and also by presentation category, i.e. symposia, research papers, WiPs and posters. The highest scoring proposals in each of the 4 categories were considered eligible for inclusion in the programme. At this point it became clear that adopting the plenary-only format for LTRC 2010 meant there would not be space to include some high-quality proposals in the programme. For this reason the Local Organising Committee (in consultation with the ILTA Board) decided to schedule several parallel research paper and symposium sessions on the Thursday and Friday. This allowed us to increase the number of research paper slots from about 22 to 40, and to increase the number of symposium slots from 2 to 4. Our venue facilities meant we could accommodate around 20 posters and 18 WiPs. We were sensitive to the fact that there is a longstanding affection for the LTRC plenary format and there have been mixed reactions to parallel sessions at past LTRCs. However, as well as allowing as many people as possible to present and take part in the event, we believed this decision would enable us to offer a broad and stimulating conference program for all participants, who now come to LTRC from many parts of the world and represent a growing range of professional LTRC Cambridge 2010 Conference Organising Committee interests.