Acknowledgments The LTRC 2014 Organizing Committee Chair John de Jong VU University Amsterdam Gea Dreschler Bert Weltens Rebecca Present-Thomas Helena van Veen Mike Hannay Pearson, London Veronica Benigno Alex Perry-Dean We are grateful for generous support from our Sponsors: Table of Contents Getting Around Amsterdam 4 President's Message to LTRC 2014 6 Message from LTRC Chair 7 Awards 2014 8 Conference Program Schedule 9 Pre-Conference Workshops 10 Abstracts Wednesday June 4 15 Abstracts Thursday June 5 40 Abstracts Friday June 6 85 Conference Program Details Day by Day 118 LTRC 2014 Reviewers and Volunteers 121 Index 122 3 Getting Around Amsterdam A comprehensive system of trains, buses, trams, and the metro ensure convenient public transportation in and around Amsterdam. Travelling from Schiphol to Amsterdam From the arrivals hall of Schiphol Airport, you can take an NS (the Dutch railway operator) train to the city center (station: Amsterdam Centraal) or VU University (station: Amsterdam Zuid). If you have a bank card with an appropriate chip and pin code, you can purchase your train tickets from the yellow NS machines near the airport exit. It is also possible to purchase tickets and ask questions at the NS desk, which is identified on the map below by the blue NS logo in front of Arrivals 2. Trains from Schiphol leave several times per hour. Single, 2nd-class tickets cost approximately €2.60 to Amsterdam Zuid (~8 minutes travel time) or €4.00 to Amsterdam Centraal (~15 minutes). Around Amsterdam with buses, trams, and metro Once you're in Amsterdam, you'll want to switch to local transportation. Single-use tickets for trams and buses can be purchased on board for about €2.80 and are valid for one hour. However, we highly recommend purchasing a multi-day ticket for unlimited access to GVB buses, trams, and metro during your stay. You can purchase multi-day tickets in advance at the GVB ticket offices at Amsterdam Centraal and Amsterdam Zuid train stations. (Note that Connexxion buses and NS trains are not included in the multi-day tickets.) Getting to/from VU University The university is a short walk (~10 minutes) from train station Amsterdam Zuid. Additionally, the following local (GVB) lines, as shown on the campus map, will bring you directly to the VU Hoofdgebouw (Main Building): Tram lines 5, 16, and 24 Metro line 51 Buses 176, 241, 242, and 310 4 Getting Around Amsterdam At VU University All workshops, presentations and exhibitions will take place in the Hoofdgebouw (Main Building), which is located at de Boelelaan 1105. The opening reception will take place at the Basket campus pub (next to the parking lots in the middle of the main campus block). Main building VU The Basket Conference dinner Restaurant Café In de Waag is located on the Nieuwmarkt in the city center. From VU University, metro line 51 (direction: Amsterdam Centraal) stops directly at Nieuwmarkt, just across the square from the restaurant. Alternatively, tram lines 5, 16 and 24 will all take you to Dam Square, an 8-minute walk from the restaurant. 5 President's Message to LTRC 2014 Welcome to LTRC Amsterdam! - the 36th Language Testing Research Colloquium since our humble beginnings in a Boston hotel room in 1979. The conference organizers, chaired by John de Jong, and including Gea Dreschler, Bert Weltens, Rebecca Present-Thomas, Helena van Veen, Mike Hannay, Veronica Benigno, and Alex Perry-Dean, have put together a rich and varied program on the important theme of Towards a Universal Framework. The last time LTRC met in this lovely country was in August, 1993, when we experienced a two- nation colloquium, Part One in Cambridge and Part Two in Arnhem - the link between then and now being John de Jong. We're very fortunate this time around to enjoy the hospitality of VU University, and the facilities are not only first-rate but inviting and congenial, encouraging that most important aspect of any LTRC - socializing with old friends and discovering new ones. Here in Amsterdam, we are being treated to a wealth of information and inspiration regarding language assessment, including four pre-conference workshops, 75 paper presentations, 4 symposia, 30 work in progress presentations, 19 posters, 7 plenary sessions, and 3 keynotes, featuring, the Samuel J. Messick Memorial Lecture, the Distinguished Achievement Award Lecture, and the John Trim Award Lecture. I'm sure we will all come away with intellectual riches that will guide and inspire language testing research for a long time. We are also being indulged by wonderful social events, including the opening reception at The Basket and the conference banquet at In de Waag, surely the oldest banquet venue in LTRC history. Will there be any surprises during the banquet? It's an LTRC tradition that remarkable, even magical things happen at the banquet. We'll see... ILTA, our professional organization, of which LTRC is the annual conference, continues to work toward its purpose of promoting the improvement of language testing throughout the world. We work with other professional organizations such as AAAL, TESOL, and the many regional and national language testing organizations worldwide to disseminate information about language assessment. We have an informal working group of experts in the assessment of aviation English working with the International Civil Aviation Organization. Each year we publish an updated ILTA Language ILTA Bibliography and a Doctoral Dissertation Bibliography – we thank Tineke Brunfaut, of Lancaster University, for taking this project over this year from Annie Brown who has been conscientiously compiling the bibliographies for many years. Thank you Annie! We recognize excellence in language assessment research through the Cambridge/ILTA Distinguished Achievement Award and the ILTA Best Article Award. We promote education in language assessment with our annual Workshops and Meetings Awards. We encourage the professional development of MA and PhD students in language assessment by providing travel grants and a student presentation award. Jon Trace, University of Hawai’i, and Maryam Wagner, University of Toronto, were the recipients of the Student Travel Award this year. We look forward to hearing the many student presentations at this LTRC and to seeing which will win the Lado Best Student Paper Award. So, thank you very much to the organizing committee of this year’s LTRC and to all of you who are participating in the Colloquium this year. There is no doubt that it will be a memorable and merry meeting! All best wishes, Dan Douglas ILTA President 6 Message from LTRC Chair John de Jong On behalf of the Organizing Committee I am very happy to welcome you all to Amsterdam. It has been an interesting year preparing this meeting of the language testing minds. From the moment of putting out the call for papers correspondence has been proliferous. The call resulted in some negative and some positive reactions, but it did attract a large number of proposals: 224 in total at an interesting rate of submission as shown in the first graph. The 224 proposals were each reviewed by Proposal submission by date three raters generating 672 ratings divided over 73 volunteer raters (see page121) . 100% 90% The scoring rubrics were the same for the 80% four types of proposals (paper, poster, WIP 70% and symposium). Ratings resulted in 69% 60% 50% of the proposals being accepted (score of 4 40% or higher on scale of 1 to 6 ). Overall rater 30% agreement did not meet high-stakes 20% standards with 30% of exact agreement 10% 0% and a further 38% with a maximum 17 19 21 23 25 27 29 31 2 4 6 8 10 12 14 16 18 20 22 24 difference of 1 score point between raters. The highest acceptance rate (87%) was for October November posters, followed by symposia (80%). Acceptance rate for WIPs was 65% whereas 56% of the papers proposals were accepted. A total 18 proposals were withdrawn for various reasons. The program opens each day Theme popularity with an award lecture, followed by one or two plenary Supporting learners sessions after which we Test transparency continue in four different Supporting teachers streams and come together Diagnosing progress again at the end of the day in a Defining proficiency plenary session. For the Describing progress streams we endeavoured to Regional variations group papers by topic, but this Academic discipline was only partly successful as Job-related descript some proposals indicated up to Age-related descript five different themes of the 10 0% 5% 10% 15% 20% themes we suggested in the call for proposals and up to 9 keywords. Judging from the popularity of themes (shown in the second graph) it would seem that we are in general interested in serving the community of language learners and language teachers and hope to produce transparent information on learners ‘progress towards proficiency. There are many contributions that offer interesting perspectives in relation to the conference theme of seeking to define a universal framework and I am sure we will have interesting and heated debates in and around the campus. As of the time of this programme going to print we have 262 registered participants from 35 countries and from close to 200 different institutions. My wish for all of us is that we will be able to make some headway towards understanding what it means for learners to achieve progress in mastering a language. 7 Awards 2014 Award Committees: ILTA Student Travel Awards Chair: Tony Green (University of Bedfordshire, UK) Elvis Wagner (Temple University, USA) Jessica Wu (The Language Training & Testing Center (LTTC), Taiwan) ILTA Best Article Award Chair: Ute Knoch (University of Melbourne, Australia) ILTA Workshops and Meetings Awards Chair: Anthony Green (University of Bedfordshire, UK) Margaret (Meg) Malone (Center for Applied Linguistics, USA) Elvis Wagner (Temple University, USA) Award Winners: ILTA Student Travel Awards Jon Trace, University of Hawai'i, Paper: Building a better rubric: Towards a more robust description of academic writing proficiency (with Gerriet Janssen and Valerie Meier).