Oral Proficiency Standards and Foreign Language Teacher

264 SUMMER 2013

Oral Proﬁciency Standards and Foreign Language Teacher Candidates: Current Findings and Future Research Directions

Eileen W. Glisan Indiana University of Pennsylvania Elvira Swender American Council on the Teaching of Foreign Languages Eric A. Surface SWA Consulting Inc.

Abstract: The renewed national focus on teacher quality and effectiveness has resulted in more rigorous standards that describe the knowledge and skills required of teacher candidates across all disciplines. In the area of foreign languages, three sets of professional standards address the oral proficiency of teachers in the target languages they teach across the career continuum. For teacher candidates, the ACTFL/NCATE Program Standards for the Preparation of Foreign Language Teachers (2002) establish minimum oral proficiency levels based on the ACTFL Proficiency Guidelines—Speaking (2012). Utilizing ACTFL Oral Proficiency Interview (OPI) data, this study examines to what extent candidates are attaining the ACTFL/NCATE Oral Proficiency Standard of Advanced Low in most languages or Intermediate High in Arabic, Chinese, Japanese, and Korean. Findings indicate that 54.8% of candidates attained the required standard between 2006 and 2012 and that significant differences emerged for language, year tested, and university program results. Further research that takes into account additional contextual information about candidates and programs will inform continuing professional dialogue about the oral proficiency of teacher candidates entering the profession.

Key words: Advanced Low, NCATE, proﬁciency, Standards, teacher preparation Eileen W. Glisan (PhD, University of Pittsburgh) is Professor of Spanish and Foreign Language Education at Indiana University of Pennsylvania, Indiana, Pennsylvania. Elvira Swender (Doctor of Arts, Syracuse University) is Director of Professional Programs at the American Council on the Teaching of Foreign Languages, White Plains, New York. Eric A. Surface (PhD, North Carolina State University) is the President and Principal Scientist at SWA Consulting Inc., an applied research and consulting firm based in Raleigh, North Carolina. Foreign Language Annals, Vol. 46, Iss. 2, pp. 264–289. © 2013 by American Council on the Teaching of Foreign Languages. DOI: 10.1111/flan.12030 Foreign Language Annals VOL. 46, NO. 2 265

Introduction for teacher preparation. Foreign language In the field of education, the past decade teacher preparation programs that undergo has seen an increasing emphasis on teacher NCATE review and seek national recogni- accountability for student learning, at least tion by ACTFL/NCATE must provide partially in response to the national de- evidence that their programs are preparing mand for “highly qualified” teachers pro- candidates who can demonstrate the per- posed through the No Child Left Behind formance described in the standards. In Act (NCLB) (2001). The need to identify conjunction with the development of the teachers of high quality has resulted in ACTFL/NCATE Program Standards, in more rigorous standards related to the 2002, the Interstate New Teacher Assess- preparation of teachers across disciplines, ment and Support Consortium (INTASC1), particularly with respect to the develop- with support from ACTFL, designed its ment of their content knowledge and skills Model Licensing Standards for Beginning (see, for example, NCATE, 2008). This Foreign Language Teachers, which may national endeavor to develop teacher stand- also be used by participating states to ards has prompted widespread discussion license teachers and induct them into the and research exploring the type of content‐ profession. To complete the career contin- specific knowledge teachers should pos- uum, in 2001, the National Board for sess. This discussion has been particularly Professional Teaching Standards (NBPTS) prevalent in the field of mathematics (Ball, created the World Languages Other Than Thames, & Phelps, 2008; Kleickmann et al., English Standards, which describe the 2013), but it has also occurred in most knowledge and skills necessary for accom- other fields, including science (Lederman, plished language teachers who elect to earn 1999) and physical education (Siedentop, National Board certification; these stand- 2002). ards were updated in 2010 and renamed the Although dialogue concerning the World Languages Standards (NBPTS, 2001, content knowledge and skills of foreign 2010). language teachers dates back to the late One area that all three sets of standards 1980s (ACTFL, 1988), more recently three address is the need for teachers to demon- sets of teacher standards have sparked a strate a high level of oral proficiency in the renewed discussion regarding the content foreign language to be taught in the knowledge and skills that foreign language classroom, the target language (TL).2 How- teachers should possess as they move ever, the standards that have garnered the across the career continuum from teacher most attention in this regard are the ACTFL/ candidates to beginning teachers to accom- NCATE Program Standards, because they plished teachers. In 2002, NCATE ap- stipulate that foreign language teacher proved the ACTFL/NCATE Program preparation programs desiring national Standards for the Preparation of Foreign recognition must assume responsibility for Language Teachers (ACTFL, 2002) (here- determining whether their teacher candi- after referred to as the ACTFL/NCATE dates reach specific levels of oral proficiency Program Standards), which define the as outlined in the standards. expectations for teacher candidates who The requirement that teacher candi- complete a teacher preparation program dates demonstrate the requisite level of and earn teacher certification. NCATE, proficiency on the Oral Proficiency Inter- which is recognized by the U.S. Department view (OPI) was the result of two years of of Education and the Council for Higher professional dialogue, during which time Education as an accrediting agency for foreign language professionals participated schools, colleges, and departments of in extensive discussions at state, regional, education, determines which colleges of and national levels before reaching consen- education meet rigorous national standards sus. According to Glisan (2013), the 266 SUMMER 2013 standard regarding the oral proficiency of Oral Proficiency Expectations of teacher candidates reflects the voice of the ACTFL/NCATE Program foreign language education profession in- Standards asmuch as it materialized through bottom‐ The ACTFL/NCATE Program Standards for up consensus building in the field. However, the Preparation of Foreign Language Teach- since the publication of the ACTFL/NCATE ers establish the expectations that teacher Program Standards in 2002 and their use by candidates must demonstrate prior to com- programs of foreign language teacher prep- pleting college or university programs of aration seeking national recognition by teacher preparation in foreign languages ACTFL/NCATE, professional discussion (ACTFL, 2002). To this end, foreign has ensued regarding (1) the extent to language teacher preparation programs which teacher candidates are achieving the seeking national recognition must: requisite levels of oral proficiency prior to completing their preparation programs (see, 1. Require that their teacher candidates for example, Burke, 2013), and (2) the role complete an ACTFL OPI, which is of the preparation program in assisting its administered and double‐rated through candidates in reaching these levels (Glisan, Language Testing International (LTI), 2013; Huhn, 2012). This discussion has the ACTFL testing office. LTI schedules, taken on increased importance as expect- administers, and reports ACTFL‐certified ations defining teacher candidates’ oral ratings for all academic, commercial, and proficiency have evolved into a high‐stakes governments testing clientele. testing requirement, not only for teacher 2. Establish exit levels of proficiency accord- preparation programs seeking ACTFL/ ing to the ACTFL/NCATE Program Stand- NCATE recognition but also for teacher ards—Advanced Low for most languages candidates who reside in the 23 states that or Intermediate High in Arabic, Chinese, currently require a specific level of oral Japanese, and Korean. The expected levels proficiency for teacher licensure3 (Chamb- of oral proficiency are defined according to less, 2012). the ACTFL Proficiency Guidelines 2012— A cogent professional dialogue regard- Speaking (ACTFL, 2012). ing the success with which teacher candidates and programs of teacher preparation The setting of a proficiency level by are meeting national oral proficiency ex- language was based on research conducted pectations—be they for purposes of pro- by the Foreign Service Institute (FSI) in the gram completion or state licensure—must 1980s regarding the length of time necessary be based on empirical data rather than to reach specific levels of oral proficiency in anecdotal evidence and speculation. Up to a target language and considered how this point, however, there has been a dearth similar the TL is to the learner’s native of published research examining the extent language (Liskin‐Gasparro, 1982). For ex- to which candidates in foreign language ample, “…learners will typically need con- teacher preparation programs are meeting siderably more instructional time to reach the ACTFL/NCATE proficiency expect- the Advanced level of proficiency in Arabic ations. This study analyzes OPI data over than in Spanish when the learners’ native six years, from 2006 to 2012, in an effort to language is English” (Liskin‐Gasparro, describe the level at which foreign language 1982, as cited in Chambless, 2012, p. teacher candidates are meeting the ACTFL/ S144). The U.S. government developed a NCATE Oral Proficiency Standard and system for grouping languages into four to determine how these results may differ categories: Group I includes languages such across languages, years in the sample, as French, Italian, Portuguese, and Spanish; and university programs of teacher Group II includes German, Greek, and preparation. Foreign Language Annals VOL. 46, NO. 2 267

Hindi; Group III includes Hebrew, Polish, 1999, 2012) have become widely accepted Russian, Serbo‐Croatian, Thai, and Viet- in the foreign language profession as the namese; and Group IV includes Arabic, framework most extensively used to mea- Chinese, Japanese, and Korean (Liskin‐ sure oral language ability. Born as a result of Gasparro, 1982). This grouping of languages collaboration among U.S. government test- into categories is based on how different ing agencies, ACTFL, and the Educational each language is from English and how long Testing Service, the guidelines have been it would take to reach comparable levels of “institutionalized in foreign language pro- proficiency in that language. These catego- fessional circles in the United States” not ries have been used in OPI‐related research only for purposes of oral assessment but also (e.g., Surface & Dierdorff, 2003) and as the basis for teacher standards, curricu- language policy–related work, such as lum frameworks, and instructional planning foreign language skill–based pay (Dierdorff (Liskin‐Gasparro, 2003, p. 484).4 The & Surface, 2008). The ACTFL/NCATE current 2012 guidelines describe the tasks Program Standards specify that teacher that speakers can successfully complete at candidates should be able to demonstrate each of five major levels—Distinguished,5 oral proficiency at a minimum level of Superior, Advanced, Intermediate, and Nov- Advanced Low for Group I, II, and III ice—together with the content, context, languages, e.g., French, German, Hebrew, accuracy, and discourse types associated Italian, Russian, and Spanish, while candi- with these tasks (Swender & Vicars, 2012). dates who teach languages in Group IV They are an instrument for assessing (Arabic, Chinese, Japanese, Korean) must functional language ability and are used to demonstrate proficiency at a minimum level rate ACTFL OPIs. of Intermediate High (ACTFL, 2002). The OPI is the tool used to assess a While the national standards reflect an speaker’s oral proficiency—that is, the level articulated set of expectations for the career of functional ability “to use the language continuum of the foreign language teacher, effectively and appropriately in real‐life individual states also require teachers to situations” (Swender & Vicars, 2012, meet standards for earning teacher licensure. p. 1). Adapted from the earlier interview As seen above, the three sets of national procedure used by the FSI, the OPI was standards are aligned in terms of their setting developed as a structured 10‐ to 30‐minute of Advanced Low as the minimum level of recorded conversation between a certified oral proficiency for language teachers, and in OPI interviewer and the individual whose recent years, an increasing number of states proficiency is being assessed. While the have been adopting an OPI requirement for interview follows the OPI standard protocol licensure. Of the 23 states that currently consisting of a warm‐up, multiple level require the OPI (i.e., as opposed to those that checks and probes, and a wind‐down, it use written exams such as the PRAXIS II mirrors a typical spontaneous conversation Content Knowledge Tests), all but seven inasmuch as the interviewer’s line of ques- follow the recommendations of the ACTFL/ tioning and posing of tasks are determined NCATE Program Standards or require a by the way in which the interviewee higher proficiency level. responds6 (Swender & Vicars, 2012). Over the years, the original face‐to‐face interview Assessing Teachers’ Oral format was expanded to include the telephonic interview, which is a commonly used Proficiency: The ACTFL form of the OPI, as well as the OPI by Proficiency Guidelines and computer, called the OPIc (Swender & the OPI Vicars, 2012). The purpose of both the Since their inception in 1982, the ACTFL OPI and the OPIc is to obtain a “ratable Proficiency Guidelines—Speaking (1986, sample” that provides evidence of the 268 SUMMER 2013

interviewee’s level of oral proficiency ac- level of Advanced Low. In short, Advanced cording to the criteria of the ACTFL Low speakers can: Proficiency Guidelines—Speaking (ACTFL, 2012). The OPI is a global assessment Participate in most informal and some inasmuch as it “establishes a speaker’s level formal conversation on topics related to of consistent functional ability (patterns of school, home, and leisure activities and strength) as well as the upper limits of that about some topics related to employment, ability (patterns of weakness)” and does not current events, and matters of public and measure discrete pieces of knowledge about community interest; the language (Swender & Vicars, 2012, p. 2). Narrate and describe in the major time There are four major assessment criteria frames of past, present, and future in used to rate an OPI: paragraph‐length discourse with some control of aspect;7 Global tasks or functions performed in Handle appropriately the linguistic chal- the language. lenges presented by a complication or Asking and answering simple questions, unexpected turn of events in a social narrating and describing, supporting opin- situation; ions, etc. Speak with sufficient control of basic Contexts/Content Areas—the sets of structures and generic vocabulary to be circumstances, linguistic or situational, understood by native speakers of the in which these tasks are performed, and language, including those unaccustomed topics that relate to these contexts. to nonnative speech (Swender & Vicars, Context (in a restaurant in Mexico) 2012, p. 42; see the Appendix for a Content (food and drink) detailed description of the Advanced Low The accuracy with which the tasks are level). performed, i.e., the comprehensibility of the message. As a rationale for requiring Advanced How grammar, vocabulary, and other Low, the ACTFL/NCATE standards present language features affect the precision, the need for teachers to “speak in spontane- clarity, and appropriateness of the message. ous, connected discourse” and be able to Who can understand the message and what provide the essential TL input and “type of degree of empathy is required on the part of classroom environment that is necessary for the interlocutor. language acquisition to occur” (ACTFL, The oral text type that is produced in the 2013, n.p.). Because the decision regarding performance of the tasks. the acceptable minimum level of proficiency Discrete words or phrases, sentences, para- came about through professional consensus graphs, or extended discourse (Swender & in our field, it is widely believed that teachers Vicars, 2012, p. 2). who cannot speak in connected discourse and in major time frames will not be effective As each speech sample is rated accord- in planning, teaching and assessing commu- ing to the criteria described above and not in nication tasks in the three modes as defined relation to the performances of other speak- in the Standards for Foreign Language ers, the OPI is a criterion‐referenced, rather Learning in the 21st Century (National than a norm‐referenced, assessment. Standards, 2006), nor will they be able to guide students in interacting with others in The Advanced Level on the OPI interpersonal contexts. As explained earlier, the various sets of As some states have established Inter- national standards for foreign language mediate High as the minimum level for state teachers include an expectation that teach- licensure—a requirement that has sparked ers speak the TL at a minimum proficiency some professional debate in the field—it is Foreign Language Annals VOL. 46, NO. 2 269

important to highlight the differences be- hensibility. In sum, teachers with oral tween Intermediate High and Advanced proficiency at the Advanced level are able Low. Intermediate High speakers: to communicate both in larger quantity and at a higher level of quality. Their linguistic Converse with ease and confidence when skills allow them to have the spontaneity dealing with the routine tasks and social and automaticity that teachers need to situations of the Intermediate level; model and engage students in sustained Handle successfully uncomplicated tasks classroom communication. Because their and social situations requiring an ex- speech breaks down when attempting change of basic information related to more complicated and sustained speech, their work, school, recreation, particular teachers with proficiency at the Intermediate interests, and areas of competence; level, even at Intermediate High, may lack Handle a substantial number of tasks spontaneity and automaticity, and, in fact, associated with the Advanced level, but may not consistently communicate the they are unable to sustain performance of intended message. all of these tasks all of the time; i.e., typically when they attempt Advanced‐ level tasks, their speech exhibits one or Review of the Literature on Oral more features of breakdown, such as the Proficiency Levels failure to carry out fully the narration or Research examining the oral proficiency description in the appropriate major time levels of foreign language teachers is scant frame, an inability to maintain paragraph‐ at best, as acknowledged by Chambless length discourse, or a reduction in breadth (2012) in her recent literature review of and appropriateness of vocabulary; this topic. In the 1980s and 1990s, several Can generally be understood by native studies were published that examined the speakers unaccustomed to dealing with oral proficiency levels of students after nonnatives, although interference from completing a particular number of years of another language may be evident and a language study in high school (Glisan & pattern of gaps in communication may Foltz, 1998; Huebner & Jensen, 1992) or in occur (Swender & Vicars, 2012, p. 61). college (Magnan, 1986; Tschirner & Heilen- man, 1998). However, the vast majority of Foreign language teachers with Inter- this research did not include the use of mediate High proficiency function most official OPIs, conducted by certified OPI comfortably at the Intermediate level, using testers and double‐rated through LTI, nor sentence‐level speech in the present time did it seek to examine the proficiency levels frame and speaking within concrete, every- of teachers or teacher candidates. In what is day contexts and uncomplicated situations. perhaps the first publication of official OPI When attempting the tasks across the ratings, Swender (2003) reported the levels contexts and content areas of the Advanced of undergraduate foreign language majors level, while they may be able to perform at reflecting data from 501 official OPIs con- the Advanced level much of the time, they ducted between 1998 and 2002 in Mandarin, are not able to consistently sustain this level French, German, Italian, Japanese, Russian, at all times. A main distinguishing feature of and Spanish. These interviews were con- Intermediate High speakers is that their ducted either in the face‐to‐face format or via language demonstrates patterns of break- telephone, and according to current ACTFL down, i.e., their linguistic accuracy declines, OPI testing protocol, they were double‐rated their fluency suffers, they are unable to and certified by ACTFL through LTI. consistently communicate their intended Although the students who completed these message in the appropriate time frame, and OPIs were all language majors in their junior there are typically gaps in their compre- or senior years, the data were not 270 SUMMER 2013

disaggregated for teacher education candi- 92.2% achieved Advanced Low or higher, dates. Swender reported that the greatest possibly for the same reasons (pp. 29–34). concentration of ratings (55.8%) occurred in Sullivan (2011) conducted the only other the Intermediate High/Advanced Low range, study that has reported OPI data specifically with 47% of language majors demonstrating for teacher preparation candidates in an oral proficiency at the Advanced Low level or attempt to examine the strategies used by higher (2003, p. 523). teacher candidates in preparation for taking The first large‐scale study examining the OPI. Data from her survey of 734 teacher the OPI scores of teacher candidates was candidates in Spanish, Mandarin, French, conducted by Swender, Surface, and Ham- English, Italian, German, Arabic, and Russian lyn in 2007, using OPI data collected on revealed that 72% reported OPI ratings of 3,198 teacher candidates from 10 states over Advanced Low or higher (p. 243). This a two‐year period, from 2005 to 2007.8 The percentage is much higher than the results candidates in this study, which included of the Swender et al. (2007) study, perhaps at both heritage and native speakers of the TL, least in part because 200 candidates reported had completed traditional programs of being native speakers of the TL and would teacher preparation, and some were licensed presumably obtain a high proficiency rating. by their states through alternative certifica- Further, it is important to note that OPI tion routes. The study analyzed the number results in Sullivan’s study were self‐reported of candidates who reached the ACTFL‐/ by teacher candidates, rather than data that NCATE‐required level as well as those who were obtained through LTI, i.e., not official met the state licensure level. Languages OPI ratings. An interesting finding in Sulli- included French, German, Italian, Manda- van’s investigation was that the candidates rin, and Spanish, as well as a category called who reported having been successful on “other world languages” that grouped the OPI reported spending an average of together languages such as Arabic, Canton- 19 hours per week using the TL outside of ese, Haitian, Hebrew, Japanese, Korean, class, more than triple the number of hours Polish, Portuguese, and Russian. OPI results spent by candidates who were unsuccessful in confirmed that 59.5% of teacher candidates reaching the requisite OPI level. achieved the ACTFL/NCATE requirement In sum, the principal findings of the few of Advanced Low (or Intermediate High in studies that have examined OPI data from the case of Group IV languages) on their first foreign language teacher candidates are as attempt and that of those who did not meet follows: the level on the first try, 38.5% met the requirement on their second OPI attempt. In The majority of teacher candidates in addition, the study illustrated that 85% of these studies have demonstrated profi- teacher candidates reached the minimum ciency at a minimum level of Advanced OPI level set by their respective states on Low, thereby meeting the ACTFL/ their first attempt. In analyzing data for NCATE standard. individual languages, the following repre- There are differences across TLs with sent the percentages of candidates who respect to the number of candidates reached the Advanced Low level as required achieving the requisite OPI levels. by ACTFL/NCATE: French: 73.3%, Of the teacher candidates who do not German: 72.8%, Italian: 84.8%, and Spanish: reach the ACTFL/NCATE level on their 81%. In Mandarin, 100% of teacher candi- first attempt at the OPI, it appears that a dates achieved the ACTFL/NCATE mini- percentage of them are able to reach the mum level of Intermediate High, possibly level in a second OPI. because they were primarily native or There is some evidence that particular heritage speakers of the language, and in strategies for preparing teacher candidates the category of “other world languages” to succeed on the OPI may have a role to Foreign Language Annals VOL. 46, NO. 2 271

play in whether or not they demonstrate priate ACTFL/NCATE oral proficiency fi oral pro ciency at the requisite level. standards changed over time from 2006 to 2012? The current study seeks to build on the Swender et al. (2007) study by analyzing Finally, in addition to exploring how many OPI data trends from a longer six‐year candidates met the ACTFL/NCATE Pro- period, investigating factors not examined gram Standards requirement, the study also ’ earlier (e.g., attainment of the Oral Profi- examined possible differences in candidates ciency Standard across university teacher attainment of the standard across university preparation programs) and deriving future teacher preparation programs: research needs. RQ4. Are there significant differences in the number of candidates attaining the fi The Study ACTFL/NCATE Oral Pro ciency Stan- dard across university programs of As the primary focus of this study was to teacher preparation? ascertain how well foreign language teacher candidates met the ACTFL/NCATE Oral fi fi Pro ciency Standard, the rst research Methodology question was: Sample and Data Collection RQ1. How many teacher candidates met fi ACTFL provided OPI data for 2,881 foreign the ACTFL/NCATE Oral Pro ciency language teacher candidates who tested Standard for their language for years – from 2006 to 2012 as part of meeting the 2006 2012? ACTFL/NCATE Oral Proficiency Standard In view of the fact that languages differ in from 48 identified university teacher prepa- terms of time required to reach certain levels ration programs. Because 734 candidates did of proficiency as noted above, the second not report their university affiliation, these research question sought to determine test results were not used for analyses related whether teacher candidates in certain lan- to program results, although these data guages were more likely to meet the were used in all other analyses. Teacher standard for their language: candidates tested in 11 different languages: fi Arabic, Mandarin, French, German, He- RQ2. Are there signi cant differences brew, Italian, Japanese, Korean, Portuguese, across languages in the rates at which Russian, and Spanish (see Table 1). Nine teacher candidates met the appropriate fi candidates tested in more than one lan- ACTFL/NCATE Oral Pro ciency guage. For these cases, each language was Standard? treated as a separate test taker. Thus, there Given that it has been nearly a decade since were 2,890 tests included in the analysis for the release of the ACTFL/NCATE Program RQ1–3 and 2,156 included in the analysis Standards, the study also sought to deter- for RQ4. mine whether there have been changes over time in the number of teacher candidates Measures who met the oral proficiency requirement. ACTFL OPI and Advanced Level Check Therefore, the third research question The ACTFL OPI is the test of record for the addressed the extent to which teacher ACTFL/NCATE Oral Proficiency standard. candidates met the appropriate ACTFL/ All OPIs in the sample were conducted and NCATE Oral Proficiency Standard over the recorded by ACTFL‐certified OPI testers in course of the 2006–2012 period: the respective languages and were rated RQ3. How have the rates at which according to the ACTFL Proficiency Guide- teacher candidates have met the appro- lines (1999, 2012). All official ACTFL OPIs 272 SUMMER 2013

TABLE 1 academic institution conducted the inter- ‐ ‐ Languages Tested view in a face to face format, assigned an advisory rating, and submitted the recorded Language Sample interview to ACTFL for a blind second rating. Both telephonic interviews and Arabic 9 academic upgrades are considered Chinese (Mandarin) 142 official OPIs and were treated as such in French 400 this study. German 113 Of the 2,890 interviews, 2,396 were full Hebrew 1 OPIs, and 494 were Advanced Level Checks, Italian 105 a truncated version of the OPI designed to Japanese 8 determine whether or not a candidate has Korean 1 met the Advanced Low level (see Table 2). Portuguese 4 The outcome of the Advanced Level Check Russian 13 is stated in terms of qualified (“Q”) if the Spanish 2,094 candidate meets or exceeds an Advanced Low level of speaking proficiency or not n ¼ 2,890 qualified (“NQ”) if the candidate’s speech does not meet the criteria for Advanced Low (Language Testing International [LTI], 2013). It is important to note that, because administered through LTI receive two the goal of the Advanced Level Check is to independent ratings to confirm the level determine whether or not the candidate has and maintain interrater reliability; in cases of met the minimal level of Advanced Low, the disagreement on the two ratings, a third interview does not yield results to determine rating is obtained. This procedure has been the exact OPI rating—that is, how far below found to yield highly reliable ratings of oral or above the Advanced level the candidate’s proficiency (Surface & Dierdorff, 2003; speaking ability actually is. However, for SWA Consulting Inc. 2012). descriptive purposes, ACTFL often classifies Of the 2,890 OPIs in the sample, 2,828 a rating of Q as Advanced Low and NQ as were conducted via telephone, scheduled by Intermediate High. Figure 1 depicts the LTI, conducted by an ACTFL‐certified OPI overall distribution of OPI and Advanced tester, rated by the tester, and then indepen- Level Check ratings in the study using this dently second‐rated by a different ACTFL‐ convention. Figure 2 illustrates the distribu- certified tester for a blind second rating. The tion of OPI ratings only. Because the remaining 62 tests were “academic up- Advanced Level Check does not result in a grades,” in which a certified tester at an specific OPI rating, these data were not

TABLE 2 OPI and Advanced Level Checks 2006–2012

2006 2007 2008 2009 2010 2011 2012 Total OPIs 141 261 326 365 521 403 379 2,396 Advanced Level Checks 39 89 99 52 73 68 74 494 Total 180 350 425 417 594 471 453 2,890 Foreign Language Annals VOL. 46, NO. 2 273

FIGURE 1 ACTFL OPI and Advanced Level Check Score Distribution for Teacher Candidates in Sample

n ¼ 2,890. Sample contains testing data from 2006–2012 and across test takers regardless of whether they indicated a specific program when testing. This includes 494 Advanced Level Checks, using the convention of coding Q as AL and NQ as IH.

included in analyses that examined actual vanced Low or Intermediate High) for their OPI ratings of candidates. respective languages on the ACTFL OPI or Advanced Level Check. Therefore, the Meeting Standard ratings on these assessments were dichoto- The research questions focus on whether or mized into a two‐level dependent variable not candidates attained the standard (Ad- (met standard, did not meet standard) for

FIGURE 2 ACTFL OPI Score Distribution for Teacher Candidates in Sample

n ¼ 2396. Sample contains testing data from 2006–2012 and across test takers regardless of whether they indicated a specific program when testing. This excludes the 494 Advanced Level Checks included in the overall sample. 274 SUMMER 2013

analysis. If a candidate received Advanced At the university program level, the Low or higher in French, German, Hebrew, categorical variable of whether a university Italian, Portuguese, Russian, and Spanish, program was a public or private institution the candidate was coded as meeting stan- was included in a post hoc analysis for RQ4. dard, whereas a candidate who received a The university program level variables were rating of Intermediate High or lower in these continuous and calculated from the data in languages was coded as not meeting stan- the study—the number of years their dard. In Arabic, Mandarin, Japanese, and candidates had been testing, the overall Korean, candidates who were rated Inter- number of candidates tested, and the mediate High or above were coded as number of languages tested. meeting the standard while candidates whose scores were Intermediate Mid or below were coded as not meeting the Analytic Procedure The choice of a dichotomous dependent standard. The use of the dichotomous variable (two levels)—met standard or did dependent variable permitted inclusion of not meet standard—had an impact on the the data from OPIs and Advanced Level analytic procedure choices. Checks in the analyses. RQ1 Categorical Independent Variables To address the first research question, Several categorical independent variables frequencies are presented for candidates were included in this study to determine who met and did not meet the ACTFL/ whether or not structural factors influenced NCATE oral proficiency standard. In addi- standard attainment in the sample. Specifi- tion, because the university program was cally, language, year tested, and university not indicated for some candidates, results program affiliation were investigated. These are presented in two categories: institution factors may be proxies for unmeasured known and institution unknown. To deter- factors that influence proficiency or stan- mine whether the form in which the dard attainment. This influence is acknowl- assessment was taken (OPI or Advanced edged in the ACTFL/NCATE Oral Level Check) had a statistically significant Proficiency Standard to some degree in effect on candidates’ success in meeting the that differences in the time it takes a native ACTFL/NCATE Oral Proficiency Standard speaker of English to learn the language in this sample, a chi‐square analysis was (language learning difficulty) is incorporat- conducted. The same approach was used to ed into the proficiency level set for lan- determine whether there was a statistically guages. Although year tested could be significant difference in standard attainment treated as a continuous independent vari- for candidates with identified institutions able—and actually was in one analysis and those candidates without. (correlation for RQ3)—all three variables were treated as categorical independent variables for the main analyses. Language RQ2–4 and university program were nominal To address RQ2–4, a series of logistic variables—that is, there was no intrinsic regressions was conducted to investigate if ordering of the categories. The numbers pre‐existing categorically independent var- indicate category membership; higher num- iables (language, year tested, and program) bers do not indicate “more” of something, significantly influenced the odds of whether but merely a new category. Treating year candidates met or did not meet the ACTFL/ tested as a categorical variable in our main NCATE Oral Proficiency Standard. Logistic analyses was used to determine whether regression can handle any type of indepen- differences between years were “blips” or dent variable and a dichotomous dependent trends. variable (i.e., two categories; met standard or Foreign Language Annals VOL. 46, NO. 2 275

did not meet standard). Logistic regression dichotomous independent variables created provides an omnibus test of model fit on the from the other categories (all the other chi‐square distribution (x2) to determine if languages in this example). This recoding there is a statistically significant overall approach is standard practice in logistic effect of the independent variable or varia- regression with categorical independent bles in the model (i.e., p value for the variables with multiple levels (Cohen omnibus x2 is < or ¼ 0.05). Logistic re- et al., 2003). gression has certain advantages over other ForRQ2,Spanishwasusedasthe techniques used with dichotomous depen- referent group because it was the largest dent variables (e.g., chi‐square analysis of an language in terms of tests in the sample. For individual independent variable), such as RQ3, 2006 was used as the referent group allowing the researcher to control for other because it was the first year of testing in the factors and to approximate the percentage of sample. For RQ4, the largest single university variance in the dependent variable ac- program by candidates who tested in the counted for by each independent variable sample was used as the referent group. For the (i.e., McFadden’s Rho‐squared). Given the logistic regression results, the interpretation RQ2–4 goals and the data, logistic regression of p values for the significance of the was an appropriate method of analysis dichotomous independent variable’s effect (Cohen, Cohen, West, & Aiken, 2003). on the odds of the dependent variable is in For RQ3, because year can be treated as reference to the referent group’s relationship a continuous variable as well a categorical with the dependent variable. Continuing our variable, a correlation analysis (r) was language example above, the p values pre- conducted to determine whether year was sented for all other languages were interpreted related to standard attainment. A statisti- in comparison to the referent group (Span- cally significant r coefficient would indicate ish). Given our focus on description (not a significant increasing (positive sign) or prediction), the p values along with the decreasing (negative sign) trend in attain- percentages of candidate attainment should ment. The logistic regression allowed for the be interpreted to indicate whether candidates investigation of the effect of each year. in each language met the standard at a For RQ2–4, the language, year, and significantly higher or lower percentage university program were used as categorical than candidates in Spanish. variables in the logistic regressions. These For RQ4, a post hoc multilevel logistic variables were recoded and entered into the regression was conducted to explore the logistic regression as a series of dichotomous impact of several university program level variables for each category or level in variables—public/private status, the number the variable (e.g., for language: whether of years their candidates have been testing, the a candidate tested in French, whether a overall number of candidates tested, and the candidate tested in German, etc.), except for number of languages tested. Multilevel logis- the category or level chosen as the referent tic regression is similar to logistic regression group. For language, for example, each except it acknowledges that data can be candidate would receive a 1 for the language influenced by hierarchical structure, such as tested and 0 for all other languages, unless it candidates being nested within university was the referent group. In analysis, the programs. A two‐level model splitting the referent group (Spanish in the case of variance in candidate attainment (dependent language) did not receive a dichotomous variable) between the university program and variable as did the other categories or levels candidate levels was conducted. This model (other languages in this example) because it allowed for approximating variance associat- served as a category or level against which ed with university program membership and the others were compared. Members of the individual candidates. It also focused the the referent group received 0 for all the independent variables on the appropriate 276 SUMMER 2013 level of variance—such as program‐level RQ2: Are there significant differences independent variables predicting program‐ across languages in the rates at which level variance in attainment—providing a fl teacher candidates met the more appropriate test of the in uence of appropriate ACTFL/NCATE Oral program‐level variables. Proficiency Standard? Significant differences in the rates at which Results candidates attained the standard were found across different languages (x2 (10) ¼ RQ1. How many teacher candidates 155.013, p < 0.001). As seen from Table 4, met the ACTFL/NCATE Oral the rate of attaining the standard for Proficiency Standard for their Spanish—the referent group in this language for 2006–2012? analysis—was 51%. Mandarin, French, As seen in Table 3, of the 2,890 tests and Italian had significantly higher attain- administered from 2006 to 2012, 54.8% ment rates than Spanish (all p values (n ¼ 1,584) met the appropriate ACTFL/ < 0.001), while German and Portuguese NCATE Oral Proficiency Standard for their had a similar attainment rate as Spanish language. For those who indicated the (p ¼ 0.248, 0.329, respectively; i.e., not a university they attended, 52.2% met the statistically significant difference). Due to standard, while 62.3% of those who did not low sample sizes, results were inconclusive indicate an institution met the standard. for the other languages. Of note, the findings This finding represented a statistically for languages with the same underlying significant difference in attainment between standard (Advanced Low) as Spanish were the two groups (x2 (1)¼ 22.261, mixed, with some having similar levels of p < 0.001). Across languages for which a standard attainment (e.g., German) and rating of Advanced Low ACTFL/NCATE others having significantly higher levels of Oral Proficiency is required, a significant standard attainment (e.g., French). statistical difference was found between the percentage of candidates meeting the stan- RQ3: To what extent have the rates at dard on the ACTFL Advanced Level Check and those candidates meeting standard which teacher candidates met the on the ACTFL OPI (x2 (1) ¼ 18.857, appropriate ACTFL/NCATE Oral p < 0.001). This finding indicates that a Proficiency Standard changed over greater proportion of candidates achieved time from 2006 to 2012? the standard on the ACTFL Advanced Level At first glance, the data in Table 5, Table 6, Check (62.2%; n ¼ 298 of 479) than on the and Figure 3 seem to suggest that the rates of ACTFL OPI (51.3%; n ¼ 1,155 of 2,251). meeting the standard appeared to be

TABLE 3 Results for Meeting the ACTFL/NCATE Oral Proficiency Standard 2006–2012

Sample Met Standard Did Not Meet Standard n % n %

Combined 1,584 54.8% 1,306 45.2% Institution Known 1,126 52.2% 1,030 47.8% Institution Unknown 457 62.3% 277 37.7%

n ¼ 2,890. Includes ACTFL OPIs and Advanced Level Checks. Foreign Language Annals VOL. 46, NO. 2 277

TABLE 4 Percent of Sample Meeting the ACTFL/NCATE Oral Proficiency Standard by Language Category: 2006–2012

Sample Met Standard Did Not Comparison to Meet Standard Spanish n % n % p value

Arabic 1 11.1% 8 88.9% 0.047 Chinese (Mandarin) 126 88.7% 16 11.3% < 0.001 French 256 64% 144 36% < 0.001 German 51 45.1% 62 54.9% 0.248 Hebrew 0 0% 1 100% Italian 81 77.1% 24 22.9% < 0.001 Japanese 2 25% 6 75% 0.168 Korean 1 100% 0 0% Portuguese 1 25% 3 75% 0.329 Russian 2 15.4% 11 84.6% 0.024 Spanish 1,062 50.7% 1,032 49.3% – Combined 1,584 54.8% 1,306 45.2%

n ¼ 2,890. Distribution of pass rates differs signiﬁcantly across language (x2 (10) ¼ 155.013, p < 0.001). Spanish is the language of reference for p value calculations. p values < or ¼ 0.05 are statistically signiﬁcant. Hebrew and Korean had only one test, so p values could not be estimated.

TABLE 5 Percent of Sample Meeting the ACTFL/NCATE Oral Proficiency Standard by Year

Year n Met Comparison to 2006 n % p value

2006 180 112 62.2% — 2007 350 227 64.9% 0.713 2008 425 245 57.6% 0.241 2009 417 262 62.8% 0.717 2010 594 278 46.8% < 0.001 2011 471 242 51.4% < 0.001 2012 453 217 47.9% < 0.001

n ¼ 2,890. Sample contains testing data across test takers regardless of whether they indicated a specific program when testing. Results differ significantly by year (x2 (6) ¼ 68.348, p < 0.001). The p value indicates whether the pass rate for that year differed significantly from the pass rate in 2006. 278

TABLE 6 Percent of Sample Meeting the ACTFL/NCATE Oral Proficiency Standard by Language by Year

Language 2006 2007 2008 2009 2010 2011 2012 n % Met n % Met n % Met n % Met n % Met n % Met n % Met

Arabic –– –– –– –– 8 0% 1 100% –– Mandarin 2 100% 4 50% 6 100% 24 100% 32 56% 38 97% 36 100% French 24 33% 50 56% 63 71% 56 55% 92 65% 64 59% 51 71% German 4 75% 13 46% 15 20% 20 50% 29 34% 21 48% 11 46% Hebrew –– –– –– –– –– 10% –– Italian 1 100% 21 95% 11 82% 16 81% 21 71% 16 63% 19 63% Japanese –– 3 33% 3 0% 1 100% 1 0% –– –– Korean –– –– –– –– 1 100% –– –– Portuguese –– –– –– –– –– 1 100% 3 0% Russian –– –– –– –– 12 8% 1 100% –– Spanish 149 64% 259 64% 327 54% 300 57% 398 40% 328 44% 333 38% n ¼ Total number of candidates testing in a given year in a given language. The % Met is the percentage of that number who met the standard. UMR2013 SUMMER Foreign Language Annals VOL. 46, NO. 2 279

FIGURE 3 Percent of Sample Meeting the ACTFL/NCATE Oral Proficiency Standard by Year

Total candidates per year: 2006 (n ¼ 180), 2007 (n ¼ 350), 2008 (n ¼ 425), 2009 (n ¼ 417), 2010 (n ¼ 594), 2011 (n ¼ 471), 2012 (n ¼ 453). declining over time; the correlation between Figure 3 provides a graphic presentation of year of testing and whether or not the the decreasing trend and clearly shows a standard was met was r ¼0.110, distribution shift in 2010 with an increase in p < 0.001, statistically indicating a declin- the percentage of candidates who did not ing rate of standard attainment over time. A achieve the standard. Table 6 presents the logistic regression was conducted to further number of candidates who met the standard investigate this finding and explore the by language and year. effect of each year. In addition, because In an attempt to explain the drop in the RQ2 found a significant effect for language number of candidates who achieved the and thus to control for the effects of standard in 2010, university programs were language, language variables were included compared from 2006–2009 to 2010–2012 in the regression, as in RQ2. using a pos thoc analysis. Significant differ- Substantial differences across years ences in the number of programs participat- were found, even after controlling for ing in the data collection for these two time language (x2(6) ¼ 68.348, p < 0.001). Af- periods were found (x2 (47) ¼ 444.956, ter examining the rate at which candidates p < 0.001). Of the 48 programs that met the standard across years, the research- provided data for the study, only 28 did so ers found that the rates were relatively during both time periods; 10 programs only similar from 2006 to 2009 (i.e., not participated in 2006–2009; and 10 only significantly different). After 2009, rates of participated in 2010–2012. Therefore, it is meeting the standard dropped significantly, possible that differences in program affilia- with each subsequent year after 2009 having tion accounted for this drop in the rates of significantly lower (p < 0.001) attainment attaining the standard. To examine the rates than in 2006. Years 2010–2012 had possible effect of program affiliation and similar rates to each other, and no significant the decline in meeting the ACTFL/NCATE differences were found between these years. Oral Proficiency Standard in 2010, the data Table 5 shows the percentages of candidates were reanalyzed using only programs that overall who met the standard for each year provided data for both time periods. This with the p values in relationship to 2006. analysis revealed that the significant effect of 280 SUMMER 2013

TABLE 7 Number of University Foreign Language Teacher Preparation Programs in Sample

Sample 2006 2007 2008 2009 2010 2011 2012

Total n 11 17 22 33 35 24 21

“year tested” was unchanged. That is, in were similar (x2 (47) ¼ 384.422, 2010, 2011, and 2012, a significant decline p < 0.001; 12.9% of the approximated in achieving the standards was found when variability). compared to 2006–2009. The decline was Table 7 illustrates the number of consistent for the programs in the sample in university programs represented by candi- both time periods; thus, it is unlikely that it dates in the sample for each year of the was the effect of the entry new programs to study. Eight percent of 48 programs in the the OPI testing program. No additional data sample had 20% or fewer of their candidates are available to aid in understanding the who tested meet the ACTFL/NCATE Oral significant decline over time in this sample. Proficiency Standard requirement for their language. In comparison, 27% of the 48 RQ4: Are there significant differences programs had 80% or more of their in the number of candidates attaining candidates meet the requirement for their language. However, many of these programs the ACTFL/NCATE Oral Proficiency identified in the sample had very few Standard across university programs candidates. Looking only at the 38 programs of teacher preparation? with at least five candidates, 26% of these 38 Only data from the 2,156 candidates at the programs had 80% or more of their 48 university teacher preparation programs9 candidates meet the ACTFL/NCATE Oral who identified their program were included Proficiency requirement for their language, in this analysis. Of the 2,156 tests included while only one program had 20% or fewer of in this analysis, 1,933 were OPIs and 223 its candidates meet the requirement. were Advanced Level Checks. To analyze When examining the five university RQ4, data were restricted to only those programs that tested the largest number of university programs that were identified in candidates in the sample (see Table 8), the the sample. Results indicated significant researchers found substantial variability in differences in the number of candidates the number of candidates meeting the attaining the ACTFL/NCATE standard ACTFL/NCATE Oral Proficiency Standard, across university programs (x2 (47) ¼ ranging from only 22.1% of candidates 434.573, p < 0.001). These differences meeting the standard to 87.6% of candidates among programs accounted for 14.6% of meeting the standard. the total approximated as in “total approxi- Table 9 presents data for the five mated variability” in whether the standard university programs that had the highest was met. In other words, 14.6% of candidate rate of their candidates reaching the ACTFL/ differences in attaining the standard were NCATE Oral Proficiency Standard after related to program affiliation for the reduced having tested a minimum of five candidates. sample. Given previous results, the analysis Eighty‐eight percent or more of the candi- was also conducted controlling for language dates in these five programs successfully met and year tested by the candidate (i.e., RQ2 the standard. Table 10 presents data for the and RQ3 were supported), and the results five programs from which at least five Foreign Language Annals VOL. 46, NO. 2 281

TABLE 8 Test Results for University Programs With Largest Number of Candidates in Sample

Sample n tested Met Standard n %

Program 1 346 168 48.6% Program 2 306 90 29.4% Program 3 193 169 87.6% Program 4 113 25 22.1% Program 5 143 53 37.1%

TABLE 9 University Programs With Highest Rates of Candidates Attaining Oral Proficiency Standard

Institution Pass Public vs. Years #of #of Rate Private Testing Students Languages Tested

A 100% Private 2 5 2 B 90% Public 6 39 1 C 89% Public 4 28 2 D 89% Public 2 53 4 E 88% Public 5 193 5

Only programs with ﬁve or more students over a span of at least two years between 2006 and 2012 were included in this table.

TABLE 10 University Programs With Lowest Rates of Candidates Attaining Oral Proficiency Standard

Institution Pass Public vs. Years #of #of Rate Private Testing Students Languages Tested

F 37% Public 4 143 6 G 36% Public 5 72 2 H 29% Public 6 306 7 I 22% Private 7 113 4 J 13% Private 4 15 1

Only programs with ﬁve or more students over a span of at least two years between 2006 and 2012 were included in this table. 282 SUMMER 2013

candidates were tested and for which the since the publication of the ACTFL/NCATE least number of candidates attained the Program Standards in 2002. Perhaps the standard. In these programs, only 37% or chief question is quite simply: “Are foreign fewer of the teacher candidates met the language teacher candidates reaching the standard. Unfortunately, the only informa- ACTFL/NCATE Oral Proficiency Standard tion available about these programs per- according to the languages they will teach?” tained to their status as public/private, the Based on the data from this study, it is clear number of years their candidates had been that slightly more than half (54.8%) of the testing, the overall number of candidates teacher candidates are reaching the requisite tested, and the number of languages tested. levels of oral proficiency. This percentage is A multilevel logistic regression was somewhat lower than what was reported in performed as a post hoc analysis to confirm the Swender et al. (2007) study, which the previous results and investigate whether or revealed that 59.5% of teacher candidates not any of the four program‐level data points met the required level. This difference could that were available on all 48 programs in the be explained by virtue of the fact that the study had an impact on the attainment of the 2007 study included teacher candidates who standard by candidates in those programs. The completed alternative routes to certification, approximated variance in the met or did not many of whom are typically native speakers meet standard outcome was partitioned into and/or heritage learners and presumably university program–level variance and candi- would obtain proficiency ratings in the date‐level variance. Results indicated that 85% Advanced to Superior ranges. of the approximated variance was attributed to While the overall majority of candidates candidate‐level factors, while 15% was attrib- are achieving the standard, data reveal large uted to university program–level factors. This and important differences across languages. was consistent but slightly higher than the Unfortunately, it is difficult to explain these logistic regression results. differences in the absence of demographic Multilevel logistic regression permitted information about the candidates. Differ- analysis of university program character- ences may possibly be attributed to learners’ istics to determine if these explained any of differing language backgrounds, including the above 15% of variance. The character- such factors as length of time studying the istics examined were whether the university language and presence or intensity of study was public or private, the number of abroad experiences. As an example, one languages tested, the number of years tested, might hypothesize that test results were and the number of candidates tested. These significantly higher for Mandarin and Italian four variables did not explain a significant as compared to those for Spanish because a amount of variance as a set (x2 (4) ¼ 1.686, greater number of candidates were native p ¼ 0.793), nor were any of the program speakers or heritage learners in those two characteristics significant independently. In languages. However, in the absence of addition, influence of test type was investi- descriptive candidate data, this statement gated and was not a significant independent remains purely speculative. As a greater variable for this subsample. No additional number of candidates sought certification in data on specific program factors were French and Spanish, perhaps a more available to help explain what might be interesting question is why the results for driving these differences. French were significantly higher than those for Spanish. Figures 4 and 5 illustrate the OPI ratings for Spanish and French respec- Discussion tively.10 Over the course of the six‐year This analysis of OPI data for 2006–2012 has period, an average of 61.5% of French shed light on several important questions candidates obtained OPI ratings at the that the field has been attempting to answer Advanced Low level or higher, while an Foreign Language Annals VOL. 46, NO. 2 283

FIGURE 4 ACTFL OPI Score Distribution for Teacher Candidates in Spanish

n ¼ 1,716. Sample contains testing data from 2006–2012 and across test takers regardless of whether they indicated a specific program when testing. This excludes the 378 Spanish Advanced Level Checks included in the overall sample. average of only 49% of Spanish candidates A perplexing finding is that the rate at obtained ratings at the Advanced Low level which teacher candidates met the ACTFL/ or higher. It is important to bear in mind that NCATE Oral Proficiency Standard signifi- this difference may in part be due to the fact cantly fell in 2010, with 2010–2012 remain- that there were 1,716 candidates tested in ing fairly consistent. At first glance, this may Spanish and only 324 in French. Regretta- seem counterintuitive to what appears to be bly, additional contextual information that happening in the field inasmuch as an might provide a clearer picture to help increasing number of foreign language explain these findings is lacking. teacher preparation programs are obtaining

FIGURE 5 ACTFL OPI Score Distribution for Teacher Candidates in French

n ¼ 324. Sample contains testing data from 2006–2012 and across test takers regardless of whether they indicated a specific program when testing. This excludes the 76 French Advanced Level Checks included in the overall sample. 284 SUMMER 2013

national recognition by ACTFL/NCATE, their proficiency, and providing opportuni- leading one to surmise that program quality ties for candidates to build proficiency is improving and thus the quality of the outside of the classroom. Given that the teacher candidates must be improving also. results of the Sullivan (2011) study pointed While a possible explanation could attribute to a possible relationship between the this decline to a natural outcome of more amount of time that candidates spend in programs beginning to require the OPI for the target language outside of class and their the first time, data analysis results do not success in meeting the Oral Proficiency confirm this. In the absence of specific Standard, program factors could be influen- program information, we can only surmise tial in this regard, particularly if specificTL that perhaps programs made curricular activities beyond the classroom were re- changes or instituted new requirements, quired. For example, Sullivan found that which may have had an impact on the candidates who were successful in meeting attainment of the Oral Proficiency Standard, the Oral Proficiency Standard reported or that their teacher candidate population reading newspapers and literature, watching changed in some way (e.g., candidates television and movies, and practicing with entered with lower oral proficiency than native speakers for weeks or even months before). However, an analysis of future OPI prior to the OPI; in contrast, unsuccessful data trends over time as well as descriptive candidates reported listening to music as information about preparation programs their most common preparation strategy. and their candidates are needed to confirm Unfortunately, we do not know to what these hypotheses. extent the programs in our study encour- Perhaps the most interesting finding for aged and promoted TL use outside of the the field is the significant difference in the classroom. Clearly, more information is number of candidates attaining the ACTFL/ needed about these programs to explain NCATE Oral Proficiency Standard across the success of their candidates in reaching university teacher preparation programs. higher levels of oral proficiency and to While some programs graduated 88–100% provide a lens into how programs might of candidates who reached the Oral Profi- create the “culture of proficiency” presented ciency Standard, there were also programs in in Huhn’s (2012) study. which as few as 13% of candidates attained It merits mentioning two unanticipated the standard. The findings of this study seem findings related to candidate differences in to indicate that teacher candidates may have attaining the standard. First, the percentage a greater likelihood of attaining the Oral of candidates who met the standard was Proficiency Standard if they attended one of significantly higher for the group of candi- the university programs listed in Table 9 dates who did not report the name of a than if they attended a program listed in university program (see Table 3) than for the Table 10. This claim, if confirmed through group reporting their university (62.3% further research, would corroborate the compared to 52.2%). The second unexpect- finding of Huhn (2012), who examined ed finding, reported above, is that the the programs that were considered to be success rate of candidates on the Advanced most innovative and also most successful in Level Check was significantly higher than obtaining ACTFL/NCATE national recogni- the rate of those testing on the OPI. In both tion. These programs not only required the cases, data are lacking to account for these OPI but “worked to create a culture of oral significant differences. As reporting pro- proficiency” (p. S173) in their programs gram affiliation and selecting the assessment by instituting ongoing oral assessment are individual choices unless they are throughout the curriculum, establishing dictated by the program, candidate data benchmark proficiency testing goals, pro- could potentially answer questions about viding regular feedback to candidates on the differences in attaining the standard Foreign Language Annals VOL. 46, NO. 2 285

between these groups. In short, the absence percentage of candidates reaching the stan- of descriptive data regarding both candi- dard than do others. Third, OPI trends over dates and programs prevent further explo- time must continue to be examined in light ration of these findings. of the findings of the six‐year period reported in this study. Fourth, further Study Limitations and research is needed to explore program‐level Directions for Future Research factors that may impact attainment of the While findings from this study shed light on Oral Proficiency Standard within each how well foreign language teacher candidates program, as suggested in the Huhn literature are meeting the ACTFL/NCATE Oral Profi- review (2012). For example, to what extent ciency Standard, the lack of additional do factors such as program size, number of demographic information (e.g., first language, faculty, or opportunities for study abroad education, previous foreign language experi- impact how well candidates meet proficien- ence, study abroad) prevents both further cy standards prior to graduation? Perhaps if explanations of candidate performance and more were known about the specific features conclusions about the influence of language, of some programs that are found to impact year tested, and program affiliation. More- the success of candidates in reaching over, a large percentage of the sample did not the Oral Proficiency Standard, it would indicate a specificuniversityaffiliation. Given be possible to recommend programmatic that the rates of attaining the ACTFL/NCATE changes for other programs that aspire to Oral Proficiency Standard differed significant- assist their candidates in this regard. Phase II ly between candidates who did and did not of ACTFL’s Research Priorities Project for provide a university program affiliation, 2013–2014 is, in fact, sponsoring research program‐specific conclusions are limited. that investigates the characteristics of suc- Further, the features of the university (e.g., cessful foreign language teacher preparation public or private) and certification program programs (see http://www.actfl.org). (e.g., number of languages or candidates, years of testing) for which data were collected did not help explain differences in the Conclusion: A Call for percentage of candidates who successfully Professional Dialogue met the standard. This study has presented data that report This study has raised several questions the degree to which teacher candidates are that merit future research. First, as OPIs reaching the ACTFL/NCATE Oral Proficien- continue to be administered, it is imperative cy Standard although differences among that demographic information be collected languages, differences over time, differences so that the interplay among factors such as across programs, and differences between native language, educational background, formats of the OPI remain unexplained. One length of foreign language study, experience might argue that the most positive outcome using the TL outside of the classroom, and of this study is the impetus for professional study abroad experience may be under- dialogue that it provides. Questions that stood. In addition, more detailed data on merit urgent attention include: teacher candidates themselves may shed light on why those taking the Advanced 1. Are we satisfied that only a little over half Level Check were more successful in of teacher candidates are attaining the attaining the standard. Second, additional ACTFL/NCATE Oral Proficiency Stan- work is needed to examine differences dard? If we are not satisfied, what can we among languages in attaining the Oral do as a field to produce teachers with Proficiency Standard: It is curious that higher levels of oral proficiency? certain languages, including languages with- 2. What are the features of university in the same language category, had a higher teacher preparation programs whose 286 SUMMER 2013

candidates have a high success rate in ples (InTASC, 2011). The new standards fi meeting the ACTFL/NCATE Oral Pro - are intended to be professional practice ciency Standard? standards, not only for “beginning” 3. Given that candidates in many teacher teachers. To this end, InTASC removed preparation programs are reaching the the “N” from its name and is now called standard, how can we use their successes the Interstate Teacher and Assessment to inform and assist other programs? Support Consortium (InTASC). 4. How can we urge teacher preparation 2. For more information on the oral programs that have as low as a 13% proficiency expectations for beginning ‐ success rate on the OPI to engage in self FL teachers, see http://programs.ccsso. study and implement systemic change to org/content/pdfs/ForeignLanguageStan- ensure the success of their candidates in dards.pdf. For more information on the this area? Glisan (2013) has called for expectations for accomplished FL teach- language departments to refocus from ers, see http://www.nbpts.org/userfiles/ simple completion of required courses to file/worldlanguages_standards.pdf. fi ‐ demonstration of pro ciency based out- 3. In the literature review published by comes, but how can this type of wide- Chambless in 2012, 21 states were spread change be realized? identified as requiring the OPI and/or 5. How can we develop a systematic Writing Proficiency Test (WPT). Since approach for gathering necessary demo- that time, two additional states began graphic data on teacher candidates that requiring the OPI and/or WPT: Hawaii would help explain the results that have requires a minimal level of Intermediate emanated from this study? High on the OPI for languages that use a Roman‐based alphabet (such as Spanish) Hopefully this study will serve as a catalyst and a minimal level of Intermediate Mid in prompting continued research and dia- for languages that use a non‐Roman logue across the profession as we work based alphabet such as Mandarin; Okla- toward the goal of preparing teachers who homa requires a minimum level of ’ meet the profession s expectations for Intermediate High for all available world speaking the TL at the expected level of languages. fi fi pro ciency in which they are certi ed to 4. For a brief history of the ACTFL teach in our classrooms. Proficiency Guidelines and the OPI, see Liskin‐Gasparro (2003). Acknowledgments 5. The “Distinguished” level was added to fi — The authors would like to express our the ACTFL Pro ciency Guideline sincere thanks to Helen Hamlyn at LTI for Speaking in 2012. However, it is not her contribution in providing the OPI data yet being assessed through the OPI. for this study and to Milton Cahoon, Gwen 6. For more information about the format Good, and Matt Borneman at SWA Consult- of the OPI, see Swender (2003), and for ing Inc., for their work on the data analysis. information regarding interrater reliabil- We are also grateful to Dr. Richard Donato, ity of the assessment, see Surface and University of Pittsburgh, for his helpful Dierdorff (2003). comments and suggestions on earlier drafts 7. Aspect is a verbal category that indicates of the manuscript. whether an action or state is viewed as completed or in progress (I went/I was going), instantaneous or enduring (The Notes sun came out/The sun was shining), 1. In 2011, InTASC released its revised momentary or habitual (They vacationed Model Core Teaching Standards, which at the shore/They used to vacation at the replace the earlier INTASC Core Princi- shore). Aspect may be indicated by Foreign Language Annals VOL. 46, NO. 2 287

prefixes, suffixes, infixes, phonetic tion analysis for behavioral sciences (3rd ed.) changes in the root verb, or the use of Mahwah, NJ: L. Erlbaum Associates. auxiliaries (Swender & Vicars, 2012, p. Dierdorff, E. C., & Surface, E. A. (2008). If you pay for skills, will they learn? Skill change 43). ‐ 8. Although this study also analyzed WPT and maintenance under a skill based pay system. Journal of Management, 34, 721– data, that information will not be 743. reviewed here as it is beyond the scope Glisan, E. W. (2013). On keeping the target of the current study. language in language teaching: A bottom‐up 9. Specific university programs are not effort to protect the public and students. identified in our study. Modern Language Journal, 92, 539–542. 10. As per earlier discussion, Advanced Level Glisan, E. W., & Foltz, D. A. (1998). Assessing Checks were not included in these students’ oral proficiency in an outcome‐based analyses. curriculum: Student performance and teacher intuitions. Modern Language Journal, 82,1–18. References Huebner, T., & Jensen, A. (1992). A study of foreign language proficiency‐based testing in ACTFL. (1982). ACTFL provisional proficiency secondary schools. , guidelines. Yonkers, NY: Author. Foreign Language Annals 25, 105–115. ACTFL. (1986). ACTFL proficiency guidelines. Hastings‐on‐Hudson, NY: Author. Huhn, C. (2012). In search of innovation: Research on effective models of foreign ACTFL. (1988). ACTFL provisional program language teacher preparation. Foreign Lan- guidelines for foreign language teacher educa- guage Annals, 45, S163–S183. tion. Foreign Language Annals, 21,71–82. Interstate Teacher Assessment and Support ACTFL. (1999). ACTFL proficiency guide- Consortium (InTASC). (2011). Model core lines—Speaking. Retrieved from http://www. teaching standards: A resource for state dialogue. actfl.org/files/public/Guidelinesspeak.pdf Washington, DC: Council of Chief State fi ACTFL. (2002). ACTFL/NCATE standards for School Of cers. Retrieved March 26, 2013, from http://www.ccsso.org/Documents/2011/ the preparation of foreign language teachers. Yonkers, NY: Author. InTASC_Model_Core_Teaching_Standards_ 2011.pdf ACTFL. (2012). ACTFL proficiency guide- Kleickmann, T., Richter, D., Kunter, M., lines—Speaking (3rd ed.) Alexandria, VA: Author. Elsner, J., Besser, M., Krauss, S., & Baumert, J. (2013). Teachers’ content knowledge and ACTFL. (2013). NCATE FAQs. Retrieved pedagogical content knowledge: The role of from http://www.actfl.org/http%3A/actfl.org/ structural differences in teacher education. professional‐development/program‐review‐ Journal of Teacher Education, 64,90–106. services/actflncate/actflncate/ncate‐faqs#4 Language Testing International (LTI). (2013). Ball, D. L., Thames, M. H., & Phelps, G. Level check. Retrieved from http://www.lan- (2008). Content knowledge for teaching: guagetesting.com/level‐check What makes it special? Journal of Teacher Lederman, N. G. (1999). Teachers’ under- Education, 59, 389–407. standing of the nature of science and class- Burke, B. M. (2013). Looking into a crystal room practice: Factors that facilitate or impede ball: Is requiring high‐stakes language profi- the relationship. Journal of Research in Science ciency tests really going to improve world Teaching, 36, 916–929. language education? Modern Language Journal, ‐ fi 92, 529–532. Liskin Gasparro, J. (1982). ETS oral pro ciency testing manual. Princeton, NJ: Educational Chambless, K. S. (2012). Teachers’ oral Testing Service. fi pro ciency in the target language: Research ‐ fi on its role in language teaching. Foreign Liskin Gasparro, J. (2003). The ACTFL pro - ciency guidelines and the oral proficiency Language Annals, 45, S141–S162. interview: A brief history and analysis of their Cohen, J., Cohen, P., West, S. G., & Aiken, L. survival. Foreign Language Annals, 36, 483– S. (2003). Applied multiple regression/correla- 490. 288 SUMMER 2013

Magnan, S. S. (1986). Assessing speaking Surface, E. A., & Dierdorff, E. C. (2003). proficiency in the undergraduate curriculum: Reliability and the ACTFL oral proficiency Data from French. Foreign Language Annals, interview: Reporting indices of interrater 19, 429–438. consistency and agreement for 19 languages. Foreign Language Annals, 36, 507–519. National Board for Professional Teaching Standards (NBPTS). (2001). World languages SWA Consulting Inc. (2012). Reliability study other than English standards. Arlington, VA: of the ACTFL OPI in Chinese, Portuguese, Author. Russian, Spanish, German, and English for the American Council on Education report (Techni- National Board for Professional Teaching cal report). Raleigh, NC: Author. Standards (NBPTS). (2010). World languages standards. Arlington, VA: Author. Retrieved Swender, E. (2003). Oral proficiency testing in March 26, 2013, from http://www.nbpts.org/ the real world: Answers to frequently asked userfiles/file/TEACH_WorldLanguages_Web questions. Foreign Language Annals, 36, 520– %20v2.pdf 526. NCATE. (2008). Professional standards for the Swender, E., Surface, E. A., & Hamlyn, H. accreditation of teacher preparation institutions. (2007, November). What proficiency testing is Washington, DC: Author. telling us about teacher certification candidates. Presentation at the ACTFL Annual Confer- National Standards. (2006). Standards for ence, San Antonio, TX. foreign language learning in the 21st century (SFLL). Lawrence, KS: Allen Press. Swender, E., & Vicars, R. (2012). ACTFL oral proficiency interview tester training manual. No Child Left Behind Act of 2001 (NCLB). Alexandra, VA: ACTFL. (2001) Public Law (PL) 107‐110, Title II 20 U. S.C. 6601 sec. 2101. Retrieved March 26, 2013, Tschirner, E., & Heilenman, L. K. (1998). from http://www2.ed.gov/policy/elsec/leg/esea02/ Reasonable expectations: Oral proficiency 107‐110.pdf goals for intermediate‐level students of Ger- man. Modern Language Journal, 82, 147– Siedentop, D. (2002). Content knowledge for 158. physical education. Journal of Teaching in Physical Education, 21, 368–377. Submitted January 15, 2013 Sullivan, J. H. (2011). Taking charge: Teacher candidates’ preparation for the oral proficiency Accepted March 19, 2013 interview. Foreign Language Annals, 44, 241– 257.

APPENDIX Advanced Low Oral Proficiency

Speakers at the Advanced Low sublevel are able to handle a variety of communicative tasks. They are able to participate in most informal and some formal conversations on topics related to school, home, and leisure activities. They can also speak about some topics related to employment, current events, and matters of public and community interest. Advanced Low speakers demonstrate the ability to narrate and describe in the major time frames of past, present, and future in paragraph‐length discourse with some control of aspect. In these narrations and descriptions, Advanced Low speakers combine and link sentences into connected discourse of paragraph length, although these narrations and descriptions tend to be handled separately rather than interwoven. They can handle appropriately the essential linguistic challenges presented by a complication or an unexpected turn of events. Responses produced by Advanced Low speakers are typically not longer than a single paragraph. The speaker’s dominant language may be evident in the use of false cognates, literal translations, or the oral paragraph structure of that language. At times their discourse may be Foreign Language Annals VOL. 46, NO. 2 289

minimal for the level, marked by an irregular flow, and containing noticeable self‐correction. More generally, the performance of Advanced Low speakers tends to be uneven. Advanced Low speech is typically marked by a certain grammatical roughness (e.g., inconsistent control of verb endings), but the overall performance of the Advanced‐level tasks is sustained, albeit minimally. The vocabulary of Advanced Low speakers often lacks specificity. Nevertheless, Advanced Low speakers are able to use communicative strategies such as rephrasing and circumlocution. Advanced Low speakers contribute to the conversation with sufficient accuracy, clarity, and precision to convey their intended message without misrepresentation or confusion. Their speech can be understood by native speakers unaccustomed to dealing with non‐natives, even though this may require some repetition or restatement. When attempting to perform functions or handle topics associated with the Superior level, the linguistic quality and quantity of their speech will deteriorate significantly.

Source: http://actﬂproﬁciencyguidelines2012.org/speaking#Advanced