Online Diagnostic Assessment: Potential and Limitations (The Case of DIALANG in the Greek Junior High School Context)
Total Page:16
File Type:pdf, Size:1020Kb
Research Papers in Language Teaching and Learning Vol. 3, No. 1, February 2012, 293‐310 ISSN: 1792‐1244 Available online at http://rpltl.eap.gr This article is issued under the Creative Commons License Deed. Attribution 3.0 Unported (CC BY 3.0) Online Diagnostic Assessment: Potential and Limitations (The case of DIALANG in the Greek Junior High School Context) [Διαγνωστική Αξιολόγηση online: Δυνατότητες και Περιορισμοί (Η εφαρμογή του DIALANG σε Ελληνικό Γυμνάσιο)] Vassiliki Baglantzi This study explored the potential of DIALANG to serve placement purposes in the 1st grade of Greek state junior high school. The DIALANG system of language assessment is an application of the Common European Framework of Reference (Council of Europe, 2001), which assesses the user’s level in five skills (listening, writing, reading, vocabulary and structures) and places them on one of the six (A1‐C2) CEFR levels. It is a valid and reliable, online diagnostic test, which makes extensive use of self‐assessment and provides several types of feedback. So far, DIALANG has been mainly used by adult learners in 14 languages in self‐access, low‐stakes contexts around Europe. However, the research focus in this study is whether DIALANG can replace teacher made placement tests for younger learners of English. Teacher‐made tests are often criticized on the grounds of validity, reliability and content, therefore the use of a standardized alternative would facilitate placement procedures in a valid and reliable way. 20 students took DIALANG and then reported on their experience through a questionnaire. The students were also involved in self‐assessing their reading, writing and listening skills using the DIALANG self‐assessment statements. Their teacher also used the DIALANG scales to assess the students’ proficiency level in these three skills. The research findings showed that DIALANG can be a highly practical and useful placement tool provided that it is limited to the assessment of three skills (listening, writing and reading) out of the five it can assess. Results also showed that neither the students nor the teacher are adequately familiar with the DIALANG statements and consequently the CEFR skills level descriptors, although these levels are extensively used in the description of textbooks, language courses and exams in the ELT field in Greece. Η παρούσα μελέτη ερευνά τις δυνατότητες της αξιολόγησης DIALANG να εξυπηρετήσει κατατακτήριους σκοπούς στην Α’ τάξη του δημόσιου Γυμνασίου στην Ελλάδα. Το σύστημα γλωσσικής αξιολόγησης DIALANG είναι μια εφαρμογή του Κοινού Ευρωπαïκού Πλαισίου Αναφοράς των Γλωσσών (Council of Europe, 2001), που αξιολογεί το επίπεδο του χρήστη σε πέντε δεξιότητες (κατανόηση προφορικού λόγου, παραγωγή γραπτού λόγου, κατανόηση γραπτού λόγου, λεξιλόγιο και δομές) και τον τοποθετεί σε ένα από τα έξι επίπεδα (Α1‐Γ2) 293 Baglantzi / Research Papers in Language Teaching and Learning 3 (2012) 293‐310 του Πλαισίου. Είναι ένα έγκυρο και αξιόπιστο διαγνωστικό τεστ (Council of Europe, 2001), διαθέσιμο από το Διαδύκτιο, το οποίο κάνει εκτενή χρήση της αυτοαξιολόγησης και παρέχει διαφορετικά είδη ανατροφοδότησης. Ωστόσο, μέχρι τώρα έχει χρησιμοποιηθεί κυρίως από ενήλικους μαθητές 14 γλωσσών σε περιβάλλοντα αυτοπρόσβασης και χαμηλού διακυβεύματος σε όλη την Ευρώπη. Ο ερευνητικός στόχος αυτής της μελέτης είναι αν η αξιολόγηση DIALANG μπορεί να αντικαταστήσει τα κατατακτήρια τεστ που φτιάχνονται από τους καθηγητές και απευθύνονται σε μαθητές μικρότερης ηλικίας. Τα τεστ που φτιάχνονται από τους καθηγητές έχουν συχνά δεχτεί κριτική ως προς την εγκυρότητα, την αξιοπιστία και το περιεχόμενό τους, επομένως, εναλλακτικά η χρήση ενός σταθμισμένου τεστ, θα διευκόλυνε την κατατακτήρια διαδικασία με έγκυρο και αξιόπιστο τρόπο. 20 μαθητές έκαναν το τεστ DIALANG και συμπλήρωσαν ένα ερωτηματολόγιο σχετικά με την εμπειρία τους. Οι μαθητές επίσης αξιολόγησαν τις δεξιότητές τους στην κατανόηση και παραγωγή γραπτού λόγου και την κατανόηση προφορικού λόγου χρησιμοποιώντας τις προτάσεις αυτοαξιολόγησης του DIALANG. Επιπλέον, η καθηγήτριά τους χρησιμοποίησε τις κλίμακες DIALANG για να αξιολογήσει το επίπεδο των μαθητών της σε αυτές τις τρεις δεξιότητες. Τα ευρήματα της έρευνας έδειξαν ότι η αξιολόγηση DIALANG μπορεί να αποτελέσει ένα άκρως πρακτικό και χρήσιμο κατατακτήριο εργαλείο με την προϋπόθεση ότι θα περιοριστεί στην αξιολόγηση τριών δεξιοτήτων (κατανόηση και παραγωγή γραπτού λόγου και κατανόηση προφορικού λόγου) από τις πέντε που μπορεί να αξιολογήσει. Τα αποτελέσματα επίσης έδειξαν ότι ούτε οι μαθητές ούτε η καθηγήτρια είναι αρκετά εξοικειωμένοι με τις προτάσεις του DIALANG και κατ’ επέκταση με την περιγραφή των επιπέδων δεξιοτήτων του Κοινού Ευρωπαïκού Πλαισίου Αναφοράς των Γλωσσών, παρά το ότι τα επίπεδα αυτά χρησιμοποιούνται ευρέως στην περιγραφή βιβλίων, προγραμμάτων γλωσσών, και εξετάσεων στον τομέα της Διδασκαλίας της Αγγλικής Γλώσσας στην Ελλάδα. Key words: DIALANG, diagnostic assessment, placement tests, self‐assessment. Introduction Every year, in September, teachers of English in Greek state junior high schools give students of the 1st grade a placement test in order to stream them into two levels: less and more advanced learners (Ministry of Education, 1996). There are no ready‐made, commercially available or specially‐designed placement tests which can work for every institution and every language programme, since placement tests are constructed for particular situations and are based on the key features of different levels in each situation. As Hughes (2003, p. 17) states: “[Successful placement tests] are tailor‐made rather than bought off the peg.” Thus, the design of a valid and reliable placement test is undertaken by the teachers. Since placement is based on the exclusive responsibility of diagnosing what a pupil can or cannot do, using a diagnostic test could be an alternative. The problem in this case lies in the fact that diagnostic tests are scarce and very often other types of tests, such as proficiency or progress tests are used for diagnostic purposes (Alderson, 2005; Hughes, 2003). The DIALANG language assessment system is an application of the Common European Framework of Reference (CEFR; Council of Europe, 2001). It is a free, online, diagnostic test which places test takers at the six CEFR levels (A1‐C2) in five skills: listening, writing, reading, vocabulary and structures. It involves self‐ assessment and provides a variety of feedback types. Its validity and reliability have been established by groups of testing experts in Universities around Europe and it has been widely used as it can assess fourteen European 294 Baglantzi / Research Papers in Language Teaching and Learning 3 (2012) 293‐310 languages. Studies that report on the DIALANG system use (Floropoulou, 2002; Haahr et al, 2004; Huhta et al, 2002; Yang, 2003) refer to adult test takers who self‐access DIALANG. The present study researches the possibility of offering teachers of English in Greek state junior high schools an alternative placement procedure, which can replace teacher‐made placement tests, whose validity and reliability are often criticized. Another aim is to assess students’ familiarity with self‐assessment procedures and shed light on the students’ perceptions of the CEFR skill level descriptors. A third aim is to assess the extent to which the teacher can accurately match students’ performance to the CEFR skill level descriptors and how familiarity with the CEFR can enhance the accuracy of their assessment. The DIALANG test: components and feedback When entering the DIALANG site (http://www.lancs.ac.uk/researchenterprise/ dialang/about), users first choose the language and the skill to be tested. They can also choose the language (mother tongue or other) of the test rubrics, self‐assessment statements and feedback. Having chosen the above, DIALANG users are presented with the Vocabulary Size Placement Test (VSPT). Their task is to study a list of verbs and decide which are real and which are not. Because of the assumption that “the size of someone’s vocabulary can be used as a rough guide to other language skills”(Meara, 1992, p. 5) this test is used as a quick way of profiling the user’s vocabulary size, and determines the level of difficulty (easy, medium, difficult) of the test that follows. Another optional placement procedure is a set of self‐assessment statements (18 for each skill) that are available in 3 skills (Listening, Writing and Reading). Vocabulary and Grammar are not included since they are more language specific and, therefore, difficult to develop in relation to all languages (Alderson, 2005). Test takers are presented with “Can do” statements, which are arranged in order of increasing difficulty, and are asked to click on the ‘Yes’ or ‘No’ answer. The aim of this part of the test is two‐fold: On the one hand, combined with the VSPT it provides an estimate of the user’s level, according to which DIALANG will be adjusted. More importantly, though, it is an opportunity for the users to reflect on their language ability and be involved in their own assessment. The system calculates the user’s level and, when the test is over, it compares this level with the one decided according to the test results. It can provide possible explanations of any discrepancies between them in the feedback provided. Self‐assessment is an optional component and the users can skip it, but, as with the VSPT, they are advised to take it and are warned about the consequences if they do not. If a user decides to skip both the VSPT and the self‐assessment