FINAL REPORT Roumen Vesselinov, Ph.D. Visiting Assistant Professor Queens College | City University of New York Measuring the Effectiveness [email protected] of Rosetta Stone® (718) 997-5444 January 2009 Executive Summary Introduction This is the first scientific study that de- PROJECTIONS This research project focused on study- cisively determines the effectiveness of 1. It is projected, that after 70 hours of study ing Spanish as a foreign language. The Rosetta Stone Spanish software. with Rosetta Stone Spanish the average participants were randomly selected from WebCAPE score would be sufficient to respondents to an advertisement. The MAIN FINDINGS fulfill the requirements for the first semes- respondents were reviewed on demograph- 1. After 55 hours of study with Rosetta ter for any college Spanish course (e.g. 3, 4, ics and language skills. People below age of Stone, students will significantly improve 5, or 6 semester Spanish course). This 19 and above 70 were excluded from the their Spanish language skills. projection needs to be statistically tested pool of potential participants. Also peo- further with a follow-up study of 70-100 ple who declared advanced knowledge of 2. After 55 hours of study with Rosetta hours of study with Rosetta Stone Spanish. Spanish or, were of Hispanic origin were Stone we can expect with 95% confidence excluded from the pool. The participants that the average WebCAPE score will be on 2. Rosetta Stone Spanish can be considered were randomly assigned to two samples: the level that will be sufficient to cover the equivalent to one semester of six semester Facility and Remote (At home). After the requirements for one semester of study in a course study of Spanish. It is possible to sampling was done, the respondents were college that offers six semesters of Spanish. investigate further which of the universi- approached with the information of their ties and colleges who use WebCAPE have assignment to a particular sample. 3. After 55 hours of study with Rosetta six semester Spanish course and find the The participants were given equal Stone Spanish significant proportion of average tuition for one Spanish semester. opportunity to study Spanish in a home students (56%-72%) will increase their oral environment and at Rosetta Stone Facil- proficiency with at least one level. The first four findings are backed by statis- ity. The length of the study was limited tical sample survey analysis. They are gen- to 55 hours which was strictly followed 4. Rosetta Stone Spanish Software is evalu- eralizable to the general population. The and controlled by the investigators. The ated extremely favorably by its users. After projections are simple extrapolation of the participants were instructed to use only 55 hours of use almost unanimously (80%- survey sample results and are not backed Rosetta Stone software during the study. 90%) the users agreed that Rosetta Stone by statistical evidence. Additional research No other tools or materials were used by Spanish software was easy to use, very is needed to scientifically confirm these two participants as far as we know. helpful, enjoyable, and very satisfac- conjectures. tory, and that they will recommend this software to others. Measuring the Effectiveness of Rosetta Stone | Queens College, City University of New York 1 Part 1. Sample Description The participants in the study were ran- refused and the people who accepted our The final sample of 135 people had domly selected from a pool of 6409 re- offer. We tested for differences on gender, 57% female participants, with mean age of spondents from the Washington DC Metro age, race, education, income, foreign lan- 39 years. Of the whole sample, 78.5% had Area (Washington DC, Northern Virginia guage knowledge, and employment status. college degree or above, 80% had full time and Maryland). 22% of the pool had some 210 people were approached for the or part time job and 33% knew foreign lan- knowledge of Spanish (69% Novice and 31% study. Of them 34 never responded, and 176 guage (not Spanish). The median income Beginner). The pool was predominantly fe- started the study and 135 finished success- was between $50,000 and $75,000. Below male (63.3%). Race decomposition was Cau- fully. The dropout rate was 23% (41 dropped are some detailed distributions by main de- casian (52.6%), Black (34.3%), Asian (8.1%), out of 176). There were no statistically sig- mographic variables. and Other (5%). The average age was 41 nificant differences between the dropouts years with the oldest person being 91 years (n=41) and the final sample (n=135) on old. gender, age, race, education, income, for- The pool was highly educated with eign language knowledge, and employment Table 1. Race Decomposition 74.7% having college degree, M.A. or Ph.D. status. In other words we have no reason to Race Percent The majority (75.9%) of the pool were em- believe that people who dropped out were African American/Black 21.4 ployed either FT or PT. The median income any different than people who stayed in. Asian 8.4 was $50,000-$75,000, with range from below Our final sample consisted of 135 peo- Caucasian 61.1 $25,000 to more than $150,000. ple, divided into two groups: Facility (n=70) Native American/Alaskan 0.8 The two samples were randomly se- and Remote (n=65). There were no statisti- lected from this pool. Only one person had cally significant differences between the two Other 8.3 to switch the sample because of internet groups on: gender, age, race, education, in- connection issues while studying remotely come, and employment status. The only dif- shortly after the beginning of the study. ference was on foreign language knowledge with Remote sample having 21.5% with DEMOGRAPHICS foreign language versus 42.9% for the Facil- There were no statistically significant dif- ity sample. This parameter was not part of Table 2. Education Decomposition ferences between the people who were ap- the initial sample design and does not affect Education Percent proached to participate in the study but our analysis. High School Diploma/GED 3.0 Some College 18.5 Mean = 40.81 Std. Dev. = 13.452 College Degree 42.2 Figure 1. Pool Distribution by Age N = 6,404 MA/PhD 36.3 500 400 Table 3. Income 300 Income ($1,000) Percent Frequency <25 10.0 25-50 25.0 200 50-75 20.0 75-100 14.2 100 100-125 11.7 125-150 9.2 >150 10.0 0 20 40 60 80 100 Age Measuring the Effectiveness of Rosetta Stone | Queens College, City University of New York 2 Part 2. Software Evaluation Table 4. Software Evaluation All participants after using Rosetta Stone Spanish software for 55 hours were Easy to Use Easy Helpful Enjoyed Satisfied Recommend asked to evaluate 5 statements about their experience with Rosetta Stone Spanish with Strongly Disagree 0.8 0.8 0.8 3.1 3.9 possible answers ranging from Strongly Disagree 1.6 3.1 3.9 5.4 2.3 Disagree to Strongly Agree on a 5-point scale where the middle is neutral. Neutral 3.1 7.8 7.8 14.7 9.3 Q1 (Easy). Rosetta Stone Spanish is easy Agree 46.5 44.2 40.3 49.6 37.2 to use. Q2 (Helpful). Rosetta Stone Spanish is Strongly Agree 48.1 44.2 47.3 27.1 47.3 helpful in teaching me the language. Q3 (Enjoyed). I enjoyed learning Spanish with Rosetta Stone. Table 5. Software Evaluation. Confidence Intervals Q4 (Satisfied). I am satisfied with Rosetta Stone Spanish. 95% Confidence Interval Dimension Percent “Agree” or Q5 (Recommend). I would recommend “Strongly Agree” Lower Limit Upper Limit Rosetta Stones software to others who are interested in learning Spanish. Easy 94.6 90.7 98.5 If we consolidate the “Agree” Helpful 88.4 82.8 94.0 and “Strongly Agree” we get the results in Table 5. Enjoyed 87.6 81.9 93.3 These results are extraordinarily con- Satisfied 76.7 69.5 84.0 vincing that Rosetta Stone Spanish is ex- tremely easy to use, very helpful, and enjoy- Recommend 84.5 78.3 90.8 able to work with. Finally, between 85% and 91% said they would recommend this soft- ware to others. Part 3. Outcome Measures (WebCAPE) In this study we used as one of our primary questions. Negative or zero score can be in- A student at a college with 6 Spanish courses outcome measures the WebCAPE test terpreted in the sense that the participant will need at least 204 points on WebCAPE (Web-based Computer Adaptive Placement did not take the test seriously or that there to move or be placed in Semester 2. Respec- Exam) developed by the Perpetual Technol- were other obstacles because the test is adap- tively a student at a college with 5 Spanish ogy Group (https://www.aetip.com/Prod- tive and every question depends on the an- courses will need at least 234 points; with ucts/CAPE/CAPE2.cfm). swer of the previous question. In that respect 4 Spanish courses – at least 270 points, and This is a well established1 test for negative scores should be interpreted very with 3 courses – at least 281 points. Spanish with impeccable validity (correla- cautiously or excluded from the analysis. tion coefficient=0.91) and reliability (test- Table 6. Suggested Calibration Scores retest = 0.86). According to their website, more than 500 colleges and universities use WebCAPE Suggested Calibration Scores WebCAPE for placement. Among them are Spanish: (3) Courses Spanish: (4) Courses Spanish: (5) Courses Spanish: (6) Courses Harvard University, Boston University, Van- Sem 1 Below 280 Sem 1 Below 270 Sem 1 Below 324 Sem 1 Below 204 derbilt University, Brown University, Queens Sem 2 281 - 351 Sem 2 270 - 345 Sem 2 234 - 311 Sem 2 204 - 288 College, CUNY, University of South Caro- lina, etc.