Automating Second Language Acquisition Research: Integrating Information Visualisation and Machine Learning Helen Yannakoudakis Ted Briscoe Theodora Alexopoulou Computer Laboratory Computer Laboratory DTAL University of Cambridge University of Cambridge University of Cambridge United Kingdom United Kingdom United Kingdom [email protected] [email protected] [email protected] Abstract levels. In particular, we build a visual user in- terface (hereafter UI) which aids the develop- We demonstrate how data-driven ap- ment of hypotheses about learner grammars us- proaches to learner corpora can support ing graphs of linguistic features discriminating Second Language Acquisition research when integrated with visualisation tools. pass/fail exam scripts for intermediate English. We present a visual user interface support- Briscoe et al. (2010) use supervised discrimi- ing the investigation of a set of linguistic native machine learning methods to automate the features discriminating between pass and assessment of ‘English as a Second or Other Lan- fail ‘English as a Second or Other Lan- guage’ (ESOL) exam scripts, and in particular, the guage’ exam scripts. The system displays First Certificate in English (FCE) exam, which directed graphs to model interactions assesses English at an upper-intermediate level between features and supports exploratory search over a set of learner scripts. We (CEFR level B2). They use a binary discrimina- illustrate how the interface can support tive classifier to learn a linear threshold function the investigation of the co-occurrence that best discriminates passing from failing FCE of many individual features, and discuss scripts, and predict whether a script can be clas- how such investigations can shed light on sified as such. To facilitate learning of the clas- understanding the linguistic abilities that sification function, the data should be represented characterise different levels of attainment appropriately with the most relevant set of (lin- and, more generally, developmental aspects guistic) features. They found a discriminative fea- of learner grammars. ture set includes, among other feature types, lexi- cal and part-of-speech (POS) ngrams. We extract 1 Introduction the discriminative instances of these two feature 3 The Common European Framework of Reference types and focus on their linguistic analysis . Ta- for Languages (CEFR)1 is an international bench- ble 1 presents a small subset ordered by discrimi- mark of language attainment at different stages of native weight. learning. The English Profile (EP)2 research pro- The investigation of discriminative features can gramme aims to enhance the learning, teaching offer insights into assessment and into the linguis- and assessment of English as an additional lan- tic properties characterising the relevant CEFR guage by creating detailed reference level descrip- level. However, the amount and variety of data tions of the language abilities expected at each potentially made available by the classifier is con- level. As part of our research within that frame- siderable, as it typically finds hundreds of thou- work, we modify and combine techniques devel- sands of discriminative feature instances. Even oped for information visualisation with method- if investigation is restricted to the most discrim- ologies from computational linguistics to support inative ones, calculations of relationships be- a novel and more empirical perspective on CEFR 3Briscoe et al. (2010) POS tagged and parsed the data 1http://www.coe.int/t/dg4/linguistic/cadre en.asp using the RASP toolkit (Briscoe et al., 2006). POS tags are 2http://www.englishprofile.org/ based on the CLAWS tagset. 35 Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH, pages 35–43, Avignon, France, April 23 - 24 2012. c 2012 Association for Computational Linguistics tween features can rapidly grow and become over- Feature Example whelming. Discriminative features typically cap- VM RR (POS bigram: +) could clearly ture relatively low-level, specific and local prop- , because (word bigram: −) , because of erties of texts, so features need to be linked to the necessary (word unigram: +) it is necessary that scripts they appear in to allow investigation of the the people (word bigram: −) *the people are clever contexts in which they occur. The scripts, in turn, VV∅ VV∅ (POS bigram: −) *we go see film need to be searched for further linguistic prop- NN2 VVG (POS bigram: +) children smiling erties in order to formulate and evaluate higher- level, more general and comprehensible hypothe- Table 1: Subset of features ordered by discriminative weight; + and − show their association with either ses which can inform reference level descriptions passing or failing scripts. and understanding of learner grammars. The appeal of information visualisation is to gain a deeper understanding of important phe- tions we describe in detail the visualiser, illustrate nomena that are represented in a database (Card et how it can support the investigation of individual al., 1999) by making it possible to navigate large features, and discuss how such investigations can amounts of data for formulating and testing hy- shed light on the relationships between features potheses faster, intuitively, and with relative ease. and developmental aspects of learner grammars. An important challenge is to identify and assess To the best of our knowledge, this is the first the usefulness of the enormous number of pro- attempt to visually analyse as well as perform jections that can potentially be visualised. Explo- a linguistic interpretation of discriminative fea- ration of (large) databases can lead quickly to nu- tures that characterise learner English. We also merous possible research directions; lack of good apply our visualiser to a set of 1,244 publically- tools often slows down the process of identifying available FCE ESOL texts (Yannakoudakis et al., the most productive paths to pursue. 2011) and make it available as a web service to In our context, we require a tool that visu- other researchers5. alises features flexibly, supports interactive inves- tigation of scripts instantiating them, and allows 2 Dataset statistics about scripts, such as the co-occurrence We use texts produced by candidates taking the of features or presence of other linguistic proper- FCE exam, which assesses English at an upper- ties, to be derived quickly. One of the advantages intermediate level. The FCE texts, which are of using visualisation techniques over command- part of the Cambridge Learner Corpus6, are pro- line database search tools is that Second Lan- duced by English language learners from around guage Acquisition (SLA) researchers and related the world sitting Cambridge Assessment’s ESOL users, such as assessors and teachers, can access examinations7. The texts are manually tagged scripts, associated features and annotation intu- with information about linguistic errors (Nicholls, itively without the need to learn query language 2003) and linked to meta-data about the learners syntax. (e.g., age and native language) and the exam (e.g., We modify previously-developed visualisation grade). techniques (Di Battista et al., 1999) and build a visual UI supporting hypothesis formation about 3 The English Profile visualiser learner grammars. Features are grouped in terms of their co-occurrence in the corpus and directed 3.1 Basic structure and front-end graphs are used in order to illustrate their rela- The English Profile (EP) visualiser is developed tionships. Selection of different feature combi- in Java and uses the Prefuse library (Heer et nations automatically generates queries over the al., 2005) for the visual components. Figure 1 data and returns the relevant scripts as well as as- shows its front-end. Features are represented sociations with meta-data and different types of errors committed by the learners4. In the next sec- 5Available by request: http://ilexir.co.uk/applications/ep- visualiser/ 4Our interface integrates a command-line Lucene search 6http://www.cup.cam.ac.uk/gb/elt/catalogue/subject/ tool (Gospodnetic and Hatcher, 2004) developed by Gram custom/item3646603/ and Buttery (2009). 7http://www.cambridgeesol.org/ 36 Figure 1: Front-end of the EP visualiser. by a labelled node and displayed in the central where sk ∈ S, 1 ≤ k ≤ N, exists() is a panel; positive features (i.e., those associated with binary function that returns 1 if the input fea- passing the exam) are shaded in a light green tures occur in sk, and 0 ≤ score(fj, fi) ≤ 1. colour while negative ones are light red8. A field We group features in terms of their relative co- at the bottom right supports searching for fea- occurrence within sentences in the corpus and dis- tures/nodes that start with specified characters and play these co-occurrence relationships as directed highlighting them in blue. An important aspect is graphs. Two nodes (features) are connected by the display of feature patterns, discussed in more an edge if their score, based on Equation (1), is detail in the next section (3.2). within a user-defined range (see example below). Given fi and fj, the outgoing edges of fi are mod- 3.2 Feature relations elled using score(fj, fi) and the incoming edges Crucial to understanding discriminative features using score(fi, fj). Feature relations are shown is finding the relationships that hold between via highlighting of features when the user hovers them. We calculate co-occurrences of features at the cursor over them, while the strength of the re- the sentence-level in order to extract ‘meaningful’ lations is visually encoded in the edge width. relations and possible patterns of use. Combi- nations of features that may be ‘useful’ are kept For example, one of the highest-weighted pos- while the rest are discarded. ‘Usefulness’ is mea- itive discriminative features is VM RR (see Ta- sured as follows: ble 1), which captures sequences of a modal Consider the set of all the sentences in the cor- auxiliary followed by an adverb as in will al- pus S = {s1, s2, ..., sN } and the set of all the fea- ways (avoid) or could clearly (see).