Rosetta Stone Version 3 Falls Short of Manufacturer's Claims

UNIVERSITY OF MARYLAND CENTER FOR ADVANCED STUDY OF LANGUAGE

Rosetta Stone Version 3 Falls Short of Manufacturer’s Claims

Katharine Nielson, MA Suzanne Freynik, MA Purpose—To determine whether CASL’s review of Rosetta Stone® Version 2 is still accurate and whether Version 3 may have affected the outcomes of our previous empirical study. Conclusions—Some features of Rosetta Stone V3 have changed; however, Executive Summary these changes would not have affected the outcomes of our empirical study. Relevance—As a stand-alone package, Rosetta Stone is unlikely to be the Purpose solution to the U.S. Government’s language learning needs. CASL review and research in 2007 TTO 2101 E.4.1 | CDRL A017 | DID DI-MISC 80508B | Contract No. H98230-07-D-0175 revealed shortcomings with Rosetta Stone® Version 2 (V2) software. Ro- setta Stone recently released Version attempts to mimic conversational Learners in our study might have 3 (V3), claiming significant improve- practice through multiple-choice had more familiarity with conver- ments over V2 of its product. To de- drills. However, these drills fall short sational vocabulary, but they would termine whether our V2 review is still of the “real-life simulations” that the not have had more conversational accurate and whether V3 may have website claims have been included practice. Further, those who disliked affected the outcomes of our previous in V3. the lack of explicit writing instruc- empirical study, we compared the two 4. Although the V3 lessons are or- tion or the tedium of the drills versions. ganized in more or less the same would not have had a substantively way as the V2 lessons, V3 includes different experience with V3. Conclusions grammatical concepts that V2 did not include. The new concepts Relevance 1. As in V2, the images in V3 are not include features of Arabic or Spanish culturally relevant. In fact, 90 per- that would be especially difficult for Rosetta Stone’s claims about the cent of the photos in the Spanish and second-language learners. innovativeness of the product, as well Arabic programs are identical. as the language learning outcomes 5. Although some features of the V3 possible after its use, are generally 2. As in V2, information about how software provide more dynamic overstated. The software does not a language’s writing system works elements, the program is not truly provide the dynamic environment is not provided in V3. Learners are interactive. For example,V3 tells required to practice using the shown examples of the language’s learners when they should try an language in context. Rosetta Stone script and are expected to figure out exercise again but does not give might be a useful tool to supplement how the writing system works on them a chance to participate in vocabulary acquisition in a more their own. conversations. well-rounded language course, but 3. Unlike V2, V3 includes exercises 6. While some features of Rosetta as a stand-alone package Rosetta that might help learners with basic Stone V3 have changed, these Stone is unlikely to be the solution greetings and other real-life conver- changes would not have affected the to the U.S. Government’s language sations. The new Milestone feature outcomes of our empirical study. learning needs.

Claim by Rosetta Stone about V3: Conclusions Executive Report Version 3 uses Contextual Forma- Our review of Rosetta Stone V3 tion™ to ask the learner to write Purpose reveals that while some features have new language in response to been updated and improved, many conversational prompts.”8 In ad- To help meet its language learning have remained the same. In this sec- dition, “Version 3 immerses you goals, the U.S. Government has in- tion, we revisit a number of problems immediately in the new language, vested millions of dollars in the online encountered with Rosetta Stone V2, providing just the right context to foreign-language training program Ro- present the claim made by Rosetta prompt you to speak, pronounce, setta Stone®. In 2007 the University of Stone addressing each problem (if read and write in the very first les- Maryland Center for Advanced Study available), and review the content of son! Intuitive, sequential learning of Language (CASL) empirically exam- the new course with respect to the builds progressively and makes ined the effectiveness of Rosetta Stone problem identified. every lesson count. Version 2 (V2) in Arabic, Chinese, and Spanish. Results of that study and a As in V2, the images in V3 are not Review: The prompts for the writing previous CASL evaluation describing 1 culturally relevant. drills in the newest version of Rosetta 1 Stone are more conversational than the the program’s potential effectiveness Problem with V25: The images used prompts used in V2. That is, learners revealed shortcomings with the soft- throughout the course are the same re- are sometimes asked to respond in ware, both with respect to the delivery gardless of the language being taught, writing to conversational exchanges. of claims made by the company and in so they are not culturally relevant. the usefulness of Rosetta Stone for self- However, V3 continues to take an study by beginning language learners. Claim by Rosetta Stone about V36: inductive approach to teaching the written language. It does not explicitly For example, the software did not pro- Our new global tapestry of explain how the writing system for a vide resources for learners to practice people images reflects the diver- language works, and it assumes that using the language in a communicative sity of people and cultures from learners will be able to map the writ- context, and the courses lacked any which our language products ten language onto the sounds they are consideration of culture. derive. We hope you’ll find these hearing (while they are simultaneously After CASL published its review fascinating. of V2 and while the empirical study learning to map those sounds to the was underway, Rosetta Stone released Review: While the images have been meaning pictured). Version 3 (V3) in several different updated for V3, 90 percent of the pho- For example, the very first writing languages,2 including Spanish and tos are the same for both Spanish and lesson in Lesson 1 shows four different Arabic. The company has claimed that Arabic.7 The photos are not language- images with the noun describing each “version 3 significantly improves our specific and are not culturally relevant. image written in the target language. product,” and should compensate for In other words, they do not immerse Students are asked to match the writing many of the problems identified by the language learners in the target culture to the image. The next screen shows a CASL review and empirical study. because they depict images of people letter or syllable from one of the previ- in all cultures. ously displayed words written out, and What this study investigated students are expected to match it to As in V2, information about how a the appropriate sound. The software Because V3 was released after our language’s writing system works 2 provides no explanation with reading review and during our empirical study is not provided in V3. of V2, our current evaluation3 aimed or writing activities, and learners are to determine (1) whether our expert Problem with V2: Rosetta Stone V2 expected to learn these skills through review of the software is still accurate does not explain how the writing sys- trial and error. and (2) whether V3 has any potential tem works. Learners are expected to Unlike V2, V3 includes exercises implications for the results of our em- figure out the writing system of each that might help learners with pirical study.4 language on their own. The learner- 3 basic greetings and other real-life In this report we first revisit the issues log comments from the CASL 2007 conversation. with V2 raised by CASL researchers, empirical study indicated that the par- identify the new V3 claims made by the ticipants were particularly frustrated Problem with V2: At the end of the software manufacturers, and review V3 with the non-Roman scripts, which entire course, learners have not ac- in both Spanish and Arabic. We then caused users to either seek outside quired fundamental material, such as consider whether using V3 during our resources in order to learn the writ- basic greetings or how to introduce empirical study might have changed the ing system or abandon the program themselves. In addition, there is no results. altogether. dialogue or interaction, but instead

short, unrelated sentences that have no ion does expose learners to the words, milestones is estimated to take 10 real-life counterpart. Finally, the single it does not show them how they are minutes. Therefore, of the 200 or so participant in our empirical study who used in an actual conversation. Rather, hours necessary for this course, a total completed the 200-hour V2 course had the words simply mark the beginning of 40 minutes is designed for practice the following comment: “While Roset- and the end of the lesson. in communication. These drills are ta Stone does teach a lot of words, they Some of the picture-concept map- not the “real-life simulations” that the are not always the words you need to ping exercises present images that website says have been included in have an actual conversation.” are related to one another in the form V3. of a conversational exchange (e.g., Claim by Rosetta Stone about V3: Although the V3 lessons are “My house is big.” “My house is organized in more or less the New Contextual Formation™ bigger than your house.”). Again, this 4 same way as the V2 lessons, V3 feature uses real-world simula- exchange is more communicative in includes grammatical concepts tions to give you the benefits you nature than the examples used in the that V2 did not include. need to succeed. New Milestone exercises of V2, but it hardly provides feature lets you try out your new the “immersion” environment claimed Problem with V2: The grammar les- language knowledge in real-life by the manufacturers. sons appear to be sequenced by the simulations. The most significant gesture V3 notion of moving from simple to com- makes toward communicative lan- plex grammar. This simple to com- Review: Rosetta Stone V3 appears to guage use is the Milestone feature that plex sequencing is known as a covert be more relevant in terms of commu- ends each of the four units, where the grammatical syllabus. It is commonly nicative language use. Nearly all of learner participates in a dialogue with found in mass-market language mate- the grammar drills in V2 encouraged pictured interlocutors by choosing the rials, despite publishers’ claims to be learners to use the third person. V3, on written utterance that fits each scenar- adopting a more innovative approach. the other hand, includes a variety of io. The interlocutors are presented in In addition, V2 provides no informa- grammar drills that encourage learners the style of a cartoon strip, with photos tion on cultural-grammatical issues to use the first person, which is more scrolling across the scene and speech such as the difference between formal likely to be needed in conversation. bubbles indicating who is speaking to and familiar verb forms in Spanish or In addition, while V2 overlooked whom. The dialogue presented in this the difference between dual and plural very basic items like greetings, V3 feature is identical in every language, noun marking in Arabic. includes these from the start. How- and while conversation-like, the ex- ever, the greetings are not necessarily Claim by Rosetta Stone about V3: changes are stilted and inauthentic, as presented in a communicative con- in the following example: Improved intuitive, sequential text. A number of drills begin with a learning makes every lesson greeting, go on to address unrelated Hello. count and build progressively. grammar and vocabulary, and then end Hello. with an equally unrelated farewell. Are all the flowers the same color? Review: Lessons in V3 do not appear For example, the Core lesson in Unit No, the flowers are not all the same to be organized differently than in V2. 1 begins with two images: a man and color. What color flowers do you The organization of the materials is a woman. The word hello is written want? still based on the shift from simple to above each picture and is heard as I would like red flowers. more complex grammar, though the each image is highlighted. Learners do How many would you like? focus in V3 is on different gram- nothing on this screen, but they do see I would like 35. Thank you. matical concepts. V3 does incorporate the words and hear them. Then the ac- You’re welcome. some of the important grammatical tual lesson begins and learners match Good morning. concepts that the second version words (which have nothing to do with Good morning. overlooked. greetings) to images and complete Do you sell eggs? Specifically, the drills in V3 provide simple grammar drills. The last screen Yes, I sell eggs. How many eggs do more practice on some points of Ara- of the Core lesson is of two different you need? bic or Spanish grammar that might people leaving. The word goodbye is I need 48 eggs. be difficult for speakers of other written above each picture and learners Thank you. languages. For example, V3 Arabic hear it as the images are highlighted. You’re welcome. includes additional drills that illustrate The hello and goodbye screens are The above dialogue is unrealistic the difference between dual and plural unrelated to the material presented in and, more importantly, does not give marking on nouns, case-marking the rest of the lesson. While presenting learners enough time for this type of on nouns, and the conjugations for greetings and salutations in this fash- communicative drill. Each of the four verbs according to gender as well as

© 2008 University of Maryland. All rights reserved. 3 ROSETTA STONE VERSION 3 FALLS SHORT number, all of which were introduced Although some features of the V3 Similarly, although the technology only sporadically in V2. V3 Span- 5 software provide more dynamic behind the speech tools appears to be ish illustrates the difference between elements, the program is not truly improved, learners still do not receive formal and informal second person interactive. any feedback other than the chance to and no longer focuses extensively on compare the graph of their utterances Problem with V2: The Rosetta Stone the infrequently used present progres- to that of a native speaker. Unlike Dynamic Immersion Method9 en- sive tense. In both Spanish and Arabic, in V2, learners have more time to vironment is quite impoverished in V3 introduces first, second, and third practice speech in V3. The lessons that comparison with the rich immersion person agreement all within the first rely on the speech recognition software environments of naturalistic lan- lesson, which is an improvement over allow learners to link to a separate guage acquisition. Rosetta Stone V2 V2, where first person agreement is screen where they can listen and record is restricted to a limited vocabulary not introduced until Lesson 5 and sec- each statement multiple times before surrounding the set of images. The ond person agreement is ignored until continuing to the next question. They only dynamic and interactive elements the last lesson. are able to spend more time working are the distinctive tones and symbols In order to accommodate this more on pronunciation, but they have no intended to indicate that a selection diverse array of grammatical forms, chance to actually interact in the target among four choices is correct or incor- V3 dispenses with some of the unusual language. rect and the comparison of a visual vocabulary that permeated V2; V3 representation of the learner’s pronun- While some features of Rosetta does not have as many drills about ciation to a native speaker’s graph. Stone V3 have changed, these ducks and horses and people crouch- 6 changes would not have affected ing under tables. Rather, V3 intro- Claim by Rosetta Stone about V3: the outcomes of our empirical duces a sparser (and arguably more Rosetta Stone does not claim to have study. useful) core vocabulary and then drills changed their Dynamic Immersion the same items repeatedly with gram- Method environment; however, the The updated “global tapestry of im- matical manipulations. For instance, a company describes a new feature that ages,” while more visually appealing, set of drills in V2 might depict people attempts to provide another form of does not change the format of the jumping off of many different things, customization. program and would have had no effect while a comparable set of drills in New Adaptive Recall™ Lan- on our empirical study. And while the V3 depicts different combinations guage feature tracks progress addition of the Contextual Formation of people writing. The V2 manipula- to reinforce your strengths and and Milestone features is a gesture tion illustrates the names of all the revisit needs. New proprietary toward more communicative language, things people can jump off of, whereas speech recognition technology the features do not provide practice for the V3 manipulation illustrates how gets you speaking from the start authentic conversations with native the verb write agrees with different and new speech analysis tools speakers. Therefore, while learners in subjects. perfect your pronunciation. our study might have had more famil- Unfortunately, in addition to paring iarity with conversational vocabulary, down vocabulary items, V3 also does Review: In terms of the dynamic and they would not have had any more away with some necessary grammati- interactive components, the course practice with completing conversa- cal concepts. While V2 had lamenta- continues to provide feedback via a tions. It doubtful that the single bly few exercises involving the past negative tone. In other words, learn- participant of the study who completed and future tenses, V3 has even fewer, ers are not given any hints on how to the Oral Proficiency Interview would adhering almost exclusively to the select the correct answer, even after have been better equipped to converse present tense. In addition, the attempts choosing the incorrect answer repeat- in Arabic. to include language-specific gram- edly. Rather than receive the correct The inductive method of instruc- matical information are inconsistent. answer, learners receive feedback tion has remained the same, so those While it is standard conversational about when they should attempt the students who disliked learning through practice in both Spanish and Arabic lesson again. The V3 Adaptive Recall guessing would not have had a differ- to drop pronouns before conjugated feature records user performance, ent experience with the new product. verbs, only the Spanish version shows determines how many items the learner In addition, the technology required to examples of this throughout the drills. answered correctly, and then makes run the program is the same, so using a The Arabic version of the software suggestions for future study. When ac- new version would probably not have remains a translation of the English cessing the program on the suggested changed any of the technical issues, version and retains pronouns in every date, the learner is prompted to com- such as learners being unable to use example, which does not reflect how plete the old lesson before continuing the software from secure facilities or the language is actually used. with new work. while using dial-up Internet access.

While the overall look and feel of manufacturers concerning the innova- 4 For a review of the content of V3, see the course is more modern, it remains tiveness of their product as well as the Appendix I. very much the same program. Learners language learning outcomes possible 5 All problems were taken from the CASL technical reports mentioned previously still do little more than match pictures after use are generally overstated. (M.7 Review of Technology Mediated to words and sounds. The language is While it is possible that learners using Language Training Programs and E.3.2. somewhat more communicative, but this product might learn some con- Rosetta Stone Findings). the program does not provide the op- versational phrases, the software does 6 Unless otherwise indicated, all claims by portunity for communicative practice. not provide the dynamic environment Rosetta Stone were taken directly from the company’s website on February 9, The “simulations” involve selecting required to practice using the language 2008. words from multiple-choice drop- in context. Rosetta Stone might be a 7 See Appendix II for a complete compari- down menus, so they do not prepare useful tool to supplement vocabulary son of V3 Spanish and Arabic. learners to actually speak in the target acquisition in a more well-rounded 8 This claim was excerpted from written language. Finally, the one-size-fits-all- language course, but as a stand-alone communication with Rosetta Stone Tech- languages approach has not changed, package it is unlikely to be the solution nical Support on May 7, 2008. and while there have been some to the U.S. Government’s language 9 The Dynamic Immersion Method is Ro- setta Stone’s language learning philoso- attempts to adapt the courses to fit lan- learning needs. phy. They claim that it “will teach you a guage-specific needs, these have been new language the same way you learned your first language: by directly associating haphazard. Since the structure of the Endnotes course remains essentially the same, words (written and spoken) with objects, actions and ideas that convey meaning.” those who disliked the lack of explicit 1 In July 2007 CASL prepared a technical The website goes on to say that Rosetta writing instruction or the tedium of the report reviewing several technology-me- Stone “uses the Dynamic Immersion drills would not have had a substan- diated language training (TMLT) products, Method to simulate a real-life immersion which included evaluations of Rosetta tively different experience with V3. experience” (Retrieved May 7, 2008 from Stone Version 2 (V2) in Spanish and Ara- http://tinyurl.com/6hpvyw). bic. In May 2007 CASL began conducting an empirical study testing the effective- Relevance ness of Rosetta Stone V2 in Arabic, Corresponding Author and Reprints: Katharine Nielson, MA, University of Maryland Center for Advanced Study of Chinese, and Spanish, and a report with Our review of Rosetta Stone V3 the findings from this study was prepared Language, [email protected], www.casl.umd.edu. reveals that while some problems with in March 2008. Funding/Support: This material is based upon work supported, in whole or in part, with funding from the United V2 have been addressed, there is still 2 According to the Rosetta Stone website, States Government. Any opinions, findings and conclusions more room for improvement. Further, the following languages were initially or recommendations expressed in this material are those of launched in Version 3: Arabic, English the author(s) and do not necessarily reflect the views of the it is unlikely that using V3 would have University of Maryland, College Park and/or any agency or (U.S.), English (U.K.), French, German, entity of the United States Government. The Contracting had much of an effect (if any) on the Italian, Portuguese (Brazil), Russian, Officer’s Representative for this project is Susan K. Luce, outcomes of our previous empirical Spanish (Latin American), and Spanish Deputy Director, National Center for Language and Culture Research, (301) 226-8845, [email protected]. study, which examined the effects of (Spain). V2 on language learning. 3 In order to complete the research, CASL Therefore, our conclusions and obtained licenses to access Rosetta Stone and researchers systematically recommendations remain the same. investigated all of the lessons in Spanish The claims made by the Rosetta Stone and Arabic.

Appendix I

Overview of the Changes to the Structure of Rosetta Stone V3 Our review of V3’s new features found some variety in the format of the exercises with the addition of a Milestone feature that provides a written simulation of a dialogue. The program continues to rely on the concept-image mapping approach, and students complete exercises where they either match pictures to text or sound, or respond to picture-based prompts in speech or text. While V2 was largely focused on vocabulary exercises, V3 includes more grammar drills. For example, in V3 Spanish, in Lesson 1 of the first unit, learners are prompted to choose the article that agrees with a noun. In addition,V3 now includes exercises on subject-verb agreement and spelling. See Table 1 for a comparison of exercises included in versions 2 and 3.

Table 1. A comparison of the exercise topics in Rosetta Stone Version 2 and Version 3.

Version 2 Exercises Version 3 Exercises • Adding, subtracting, multiplying and dividing • Ages • Body parts • Body parts • Clothing • Calendar terms • Comparatives and superlatives • Cities and countries • Comparing quantities and amounts • Clothing • Countries • Colors • Descriptive adjectives, including colors • Comparing and contrasting • Family relationships • Descriptions • Food, eating, and drinking • Family relationships • Furniture and instruments • Greetings • Giving directions • Household items • Names • Introductions • Numbers • Landmarks • Pets • Numbers • Professions and human conditions • Personal hygiene • Seasons of the year • Questions • Streets, maps, and landmarks • Shopping • The alphabet • Speaking • Times of day • Sports • Vehicles • Times of day

The V3 exercises are divided between those that require rote memorization of grammatical or spelling rules and those that require learners to recall specific vocabulary words.

Appendix II

Lesson-by-lesson comparison of V3 Arabic with V3 Spanish We conducted an exhaustive comparison of the software in each language to determine whether Rosetta Stone V3 continued to rely on a one-course-for-all-languages approach. In addition, we wanted to determine whether the same photos were used throughout all the courses. See Figure 1 for a graphical representation of the similarities and differences between the Arabic and Spanish versions of the software.

Additional practice: Country-specific: 0.8% of 2.2% of the exercises the Arabic drills are altered provide additional to be Arabic-specific in a practice with the same Grammatical adjustment: superficial sense. For exercise that appears in 1.4% of the Arabic drills example, where the the Spanish version. For emphasize some other Spanish drill says, ‘I write example, in Lesson 1.2, grammatical distinction Spanish,’ the Arabic says, the Arabic version has Dual marking: 1.8% (beyond the dual/plural ‘I write Arabic.’ Likewise, two exercises related to of all the drills focus on distinction) that Spanish the money drills in Spanish expressing the posses- the Arabic distinction lacks (e.g., one seems sometimes use pesos for sive that correspond to a between the plural and designed to highlight units where the Arabic single Spanish exercise. the dual. case marking on nouns). drills use Egyptian pounds.

Random difference: 0.8% of the Arabic drills differ from their Spanish counterparts in random ways for which we cannot discern a motivation. For example, Arabic drill 20 in Lesson 1.3 is identical to Spanish drill 17, except that Direct translation: The the Arabic drill substitutes majority (92.4%) of the ‘pen’ where the Spanish drill exercises in the Arabic says ‘egg.’ version are translations of Spanish exercises. English interference: 0.3% That is not to say that of the Arabic drills differ from they are calqued; rather the Spanish because they they are the appropriate are translated from the Arabic equivalents of the English version. Thus, where same phrases in Spanish the Spanish version has a (e.g., whereas Spanish Pedro or a Mr. Rodriguez, the expresses age with Tiene Direct translation 92.4% Arabic version often has a X años, literally ‘She has John or a Mr. Jones. X years,’ Arabic uses 3omeruhaa X senawaat, Lexical adjustment: 0.3% ‘Her age is X years.’). of the Arabic drills introduce a lexical distinction that does not appear in the Spanish. For instance, while the Spanish version (in Lesson 3.4) uses the same verb for brushing teeth and brushing hair, Arabic has two distinct verbs and gives each its own drill.

Figure 1. Graphical representation of differences between Rosetta Stone V3 Arabic and Spanish.

Appendix II, continued

The general reliance on translation from English (albeit less rigid than in V2) still gives rise to a number of difficulties. Among these are the tendency to provide subject pronouns even though Arabic is a language that generally omits these. The same tendency has been recognized and corrected in a number of the Spanish drills, but the Arabic version apparently maintains superfluous pronouns because its drills have been translated from English. Also, Arabic speech acts, such as greeting and thanking, involve different procedures than corresponding speech acts in Spanish and English. For instance, Arabic speakers in responding to greetings tend to amplify them (e.g., a common response to marhaba ‘hello’ would be marhabtain ‘two hellos’). Many Arabic speech acts also entail formulaic phrases of religious origin (e.g., the response to ‘How are you?’ is often ‘I’m well, thank God.’) Because most of the drills in the Arabic version of Rosetta Stone are translated from an English template, they miss these differences.

Rosetta Stone greetings (from Lesson 1.1) A. Marhaba. Hello. B. Marhaba. Hello. (from Lesson 2.4) A. Kaif haalik? How are you? B. Ana mariida. I’m sick. Kaif haalik? How are you? A. Ana biHair. I’m well.

Authentic Iraqi greeting sequence (from the Linguistic Data Consortium’s transcripts of recorded Iraqi Arabic conversational telephone speech) A. Alu. Hello. B. Alu. Hello. A. Alu Alu. Hello hello. B. Alu shlawnik, aHuuii 3aziiz? Hello, how are you my dear brother? A. Alu marhaba Allah yesalmik Hello, greetings, God rest you. B. Kaif haaluk? Shlawnik ziin inta How are you? You’re well, Inshallah. I hope (lit. God willing). A. Alhamdullilah biHair, Allah Thank God I’m well. God yesalmik . . . rest your soul . . .

Authentic Modern Standard Arabic greeting sequence (between a native and a nonnative speaker) A. ahlan. Welcome. B. ahlan biik. Welcome to you, too. A. Kiifik? How are you? B. Alhamdulilah. Thanks be to God. A. Ma huii aHbarik? What’s your news? B. aHbarii djayid. All of it good. A. hamdullilah . . . Thank God . . .