1 TODAY, IT’S ALL ABOUT MEASURING RESULTS

If you’re looking for the best English language solution for your organization, you want learners to see results. But how can you be sure that your language solution is working?

Voxy has built excellent relationships with major educational institutions in the Amer- icas and Asian markets, including , Colombia, , the , and China—more than half of the largest and most prestigious universities in Brazil are currently partnering with us. They all came to Voxy because our personalized English solution works, and because we measure learner outcomes to prove it.

We’ll define a set of research-based principles that can be used to calculate outcomes, and illustrate how each one is measured at Voxy.

LET’S GET STARTED

We give learners what they need in order to be able to do the tasks that are important to them. Dr. Katharine Nielson “ Voxy Chief Education Officer ” 2 INTRODUCING POWERFUL ENGLISH

Learners need English skills that will make a difference in the real world. To nail a sales presentation, to score a better job, to earn a raise. Through self-study, tutoring, per- sonalized coaching, and assessment, learners can improve their proficiency to tackle the tasks that matter to them most. This is Powerful English.

FOR AN ACCURATE AND HOLISTIC ASSESSMENT OF LEARNER SUC- CESS, VOXY USES A COMBINATION OF METRICS TO DEMONSTRATE POWERFUL ENGLISH OUTCOMES.

Second language acquisition (SLA) is a complex process, so it’s important to consider the learner’s full experience when choosing a valid set of metrics. From motivation and cognitive strengths to personal interests and work habits, every learner is unique.

By identifying what factors will affect the results of each metric, you can recognize specific areas where a solution is performing optimally, the level of overall engagement, and how to improve to increase learner success. Once enough baseline data is obtained, you can also compare learners against their peers to benchmark progress and identify ways to refine your methodology to deliver the most effective learning solution possible.

3 TIME ON TASK FREQUENCY

METRICS ACHIEVEMENT INSTRUCTOR FEEDBACK FOR MEASURING 7LEARNER OUTCOMES LEARNER SATISFACTION REAL-WORLD SUCCESS

PROFICIENCY IMPROVEMENT

4 HOW WE DO IT Time on task is measured automatically by the number of minutes learners spend engaging with learning activities, reviewing their customized Word Bank, attending tutoring sessions, and reading the Grammar Guide.

Total Time on Task TIME ON TASK 3830 in the First 30 Days of Using Voxy LEARNERS One of the single biggest predictors of a learner’s future success 3190 in acquiring a new skill is the amount of time spent practicing. LEARNERS

2160 LEARNERS 1816 LEARNERS 1495 LEARNERS

1043 LEARNERS 753 LEARNERS 363 LEARNERS 359 228 LEARNERS 182 LEARNERS LEARNERS

0-1 1-2 2-4 4-8 8-16 16-32 32-64 64-128 128-256 256-512 512 + minutes minutes minutes minutes minutes minutes minutes minutes minutes minutes minutes

Over 40% of Voxy leaners spend between two and nine hours in their first month of using Voxy. This sample includes learners using the product in blended as well as standalone implementations, with differing amounts of recommended usage, based on implementation and intended outcomes.

PRO TIP

Learners should spend time practicing with authentic content, including texts, video, and audio recordings with native speakers. This will prepare them to engage with the language outside of an instructional setting, and give them the practice they need to accomplish real-life tasks.

5 HOW WE DO IT Like time on task, frequency is automatically measured by the number of times learners engage with the Voxy platform. 76% OF LEARNERS USE VOXY AT LEAST TWICE 76% PER WEEK ON AVERAGE OF LEARNERS USE VOXY AT LEAST TWICE PER WEEK ON AVERAGE 76% OF LEARNERS USE VOXY AT LEAST TWICE FREQUENCY PER WEEK ON AVERAGE 76% Spaced practice, or practice at regular intervals—even for shorter OF LEARNERS USE 76% VOXY AT LEAST TWICE periods of time—isOF LEARNERS more effective USE than massed practice, or practice PER WEEK ON AVERAGE sessionsVOXY that are AT longer LEAST but TWICEless frequent. In other words, measuring not justPER the WEEK number ON of AVERAGEhours learners have studied but also the PRO TIP number of times they’ve spent in practice is an important metric Learners should try to practice using the target language as much as they can. Even if it’s only for five minutes, taking the time to listen to for demonstrating outcomes. a conversation or review some vocabulary words on a regular basis will speed up the rate of acquisition.

6 76% OF LEARNERS USE 76% VOXY AT LEAST TWICE OF LEARNERS USE PER WEEK ON AVERAGE VOXY AT LEAST TWICE PER WEEK ON AVERAGE 76% OF LEARNERS USE VOXY AT LEAST TWICE PER WEEK ON AVERAGE 61% % 61% OF VOXY LEARNERS 61 OF VOXY LEARNERS 61% 61% OF VOXY LEARNERS GET AT OF VOXY LEARNERS 61% LEAST AN 85% OF VOXY LEARNERS OF VOXY LEARNERS ON THEIR UNIT ACHIEVEMENT TESTS GET AT LEAST AN SCORE GET AT LEAST AN AT LEAST SCORE AT LEAST 85% get at least an ON THEIR UNIT ACHIEVEMENT TESTS 85% % ON THEIR UNIT 85% 85 ON THEIR UNIT 85% ACHIEVEMENT TESTS ON THEIR UNIT ACHIEVEMENT TESTS ACHIEVEMENT TESTS ON THEIR UNIT ACHIEVEMENT TESTS

61% OF VOXY LEARNERS % HOW WE DO IT 61 Voxy creates a custom achievement test at the end of every unit using all OF VOXY LEARNERS the words and topics that have been covered, but giving new examples on 61% ACHIEVEMENT the test to determine whether learners can apply what they’ve learned. SCORE OF VOXY LEARNERS AT LEAST % Assessing how well learners understand what they’ve been taught 85 GET AT LEAST AN is achievement testing. This lets you measure whether learners ON THEIR UNIT GET AT LEAST AN understand the grammar, vocabulary, and concepts in the materials ACHIEVEMENT they’ve been85% reading and listening to. 61% TESTS ON THEIR UNIT OF VOXY LEARNERS 85% ACHIEVEMENT TESTS

ON THEIR UNIT ACHIEVEMENT TESTS SCORE AT LEAST 61% OF VOXY LEARNERS 85% SCORE AT LEAST ON THEIR UNIT ACHIEVEMENT TESTS 85% ON THEIR UNIT ACHIEVEMENT TESTS

PRO TIP

Learners can improve their achievement test scores by frequently reviewing what they’ve learned. Using idle down time to try to recall relevant vocabulary words is a good way to keep concepts fresh.

7 85% ON THEIR UNIT ACHIEVEMENT TESTS 61% OF VOXY LEARNERS SCORE AT LEAST Vocabulary VOCABULARY 88% VOCABULARY 88% VOCABULARY Vocabulary 88% 88% 88% VOCABULARY VOCABULARY 88%

VOCABULARY VOCABULARY Vocabulary VOCABULARY 88% 88% 88% VOCABULARY GRAMMAR VOCABULARY 88% VOCABULARY Vocabulary 88% 88% 88% VOCABULARY VOCABULARY88% 67%

VOCABULARY LISTENING88% 88% VOCABULARY 88% 67% VOCABULARY VOCABULARY

88% 88% VOCABULARY GRAMMAR VOCABULARY GRAMMAR SPEAKING

88% Keep it up! 88% Keep it up! 88% Keep it up! VOCABULARY SPEAKING GRAMMAR VOCABULARY GRAMMAR SPEAKING 8888%% Keep it up! 80% Keep it up! 67%74% Keep it up! 88% Keep it up! 88% Keep it up! 88% Keep it up! VOCABULARY WRITING LISTENING READING LISTENING 88% 88% Keep it up! 88% Keep it up! 88% Keep it up! WRITING LISTENING READING % WRITING LISTENINGVOCABULARY READING 88 75% Keep it up! 60% Try this skill! 66% Try this skill! 88% Keep it up! 88% Keep it up! 88% Keep it up! 67%

VOCABULARY GRAMMAR SPEAKING

88% Keep it up! 88% Keep it up! 88% Keep it up! VOCABULARYVOCABULARY GRAMMARSPEAKING SPEAKING GRAMMAR VOCABULARY GRAMMAR SPEAKING % % 88 88 88% % Keep it up! HOW% Keep it up! WE DO% Keep it up!IT 88 88% Keep it up! 74 80% Keep it up! 80 74% Keep it up! % % % VOCABULARY Keep it up! Keep it up! 88 Keep it up! 88 Keep it up! 88 Keep it up! GRAMMAR WRITING Keep it up! Voxy collects feedback directly from its English tutors, who provide per- sonalized, real-time feedback to learners through live one-on-one tutoring WRITING LISTENING READING VOCABULARY GRAMMAR SPEAKING 88% 88% 88% sessions that are fully integrated into the product. Read an example below. WRITING LISTENING READING 88% Keep it up! 88% Keep it up! 88% Keep it up! SPEAKING Keep it up! LISTENING Keep it up! READING Keep it up! WRITING LISTENING READING 88% 88% 88% WRITING LISTENING READING 75% Keep it up! 60% Try this skill! 66% Try this skill! % Keep it up! 360% TryFEEDBACK this skill! % Try this skill! Keep it up! Keep it up! Keep it up! 75 60 66 Hello, it was lovely speaking with you today! You were able to answer a lot of sponta- 88% Keep it up! 88% Keep it up! 88% Keep it up! neous questions, and you did a great job of responding and expressing your thoughts. There were only a few minor grammatical mistakes … Great job overall today—you did a wonderful job of speaking!

WRITING LISTENING READING VOCABULARY 88% GRAMMAR % SPEAKING 80% Keep it up! Keep it up! 74 Keep it up! 88% 88% 88% VOCABULARYVOCABULARY GRAMMARGRAMMAR SPEAKINGSPEAKING Keep it up! Keep it up! Keep it up! % % 88 88 % % WRITING 88 % % % % LISTENING 60% READING 66 75 88 %KeepKeep it up!it up! 74 % Keep it it up! up! 80% KeepKeep it up!it up! Try this skill! Try this skill! Keep it up! 88 74 80 VOCABULARY Keep it up! Keep it up! GRAMMAR WRITING Keep it up!

VOCABULARY GRAMMAR SPEAKING 88% 88% 88% WRITING LISTENING READING WRITING LISTENING READING SPEAKING Keep it up! LISTENING Keep it up! READING Keep it up! 88% 88% 88% INSTRUCTOR FEEDBACK 75% Keep it up! 60% Try this skill! 66% Try this skill! 75% Keep it up! 60% Try this skill! 66% Try this skill! Keep it up! Keep it up! Keep it up! All test-based assessment is subject to error and can be affected by any number of factors, from the learner’s state of mind to a bad Internet connection. So the qualitative feedback

from certified English tutors is invaluable in assessing the effective- PRO TIP WRITING LISTENING READING ness of instruction. Instructors can offer a cumulative assessment VOCABULARY 88% GRAMMAR 74% SPEAKING 80% Instructors need to be trained in how to reliably offer feedback—using Keep it up! based onKeep multiple it up! exchanges with a learner,Keep it up! triangulating the data 88% 88% 88% rubrics and checklists, as well as free responses—so that learners can be offered by more traditional forms of testing. evaluated andVOCABULARY compared in a standardizedGRAMMAR way. SPEAKING Keep it up! Keep it up! Keep it up! LISTENING % WRITING % 60% READING 66 75 88% Keep it up! 74% Keep it up! 80% Keep it up! Try this skill! Try this skill! Keep it up! 8

WRITING LISTENING READING

75% Keep it up! 60% Try this skill! 66% Try this skill! HOW WE DO IT Learner satisfaction is captured through surveys, conversations with the LEARNER SATISFACTION Learner Success Team, and unsolicited feedback. Learners are often the best judges of how much they’ve improved their language skills. Because they’re continuously evaluating their own ability to use the language, the ease with which they’ve learned it, and their happiness with the course materials, using learner feedback to measure outcomes is a logical choice. 4.13/5 4.13/5 LEARNER LEARNER SATISFACTION SATISFACTION

In a survey of students at Universidade Metodista de São Paulo, learners rated their satisfaction with Voxy an average of 4.13 out of 5 stars.

PRO TIP

Learners who can see that they’re making progress and feel that their needs are being met tend to be more satisfied. A language-learning program that responds to their needs, concerns, and questions will likely receive higher satisfaction scores.

9 HOW WE DO IT Measuring real-world success can be as simple as collecting self-reported data from learners, or as complex as measuring performance in simulated, real-world conditions.

Voxy is helping me with one of the greatest achievements a person could ever have in life: having a good qualification for the job market. I was approved in my job interviews in “ English and I owe that to Voxy. Thanks Voxy! Beatriz , Voxy Learner, São Paulo, Brazil ”

REAL-WORLD SUCCESS Voxy’s pedagogical approach is based on the principles of task- based language learning, or using English to complete meaningful tasks. One of the best measures of how well learners have improved is whether they’re able to accomplish their personal goals, such as PRO TIP achieving a specific score on a standardized test or successfully Learners can improve their English in the real world by seizing ev- ery opportunity to practice under genuine conditions—from seeking traveling to a new country. This is one of the most difficult outcomes out opportunities to speak English with customer service agents to to measure at scale, but one of the most important. selecting English as the default language for online applications.

10 HOW WE DO IT Voxy measures proficiency improvement with the Voxy Proficiency Assess- ment™ (VPA™), an in-product exam that evaluates reading and listening SCORE PROFICIENCY IMPROVEMENT SCORE proficiency as well as vocabulary and grammatical competence. The VPA™ IMPROVEMENT is offered at the start of each course and every three months thereafter. 67.1 IMPROVEMENT 80 55.8 Global language proficiency is the ability to read, write, speak, and 16% 70 PERCENTAGE OF Proficiency Correlates with Voxy Usage listen in the target language. When assessing listening proficiency, SCORE IMPROVEMENT 48.4 60 for example, one would consider a learner’s ability to do things 16 11.1% like comprehend basic instructions, make inferences, and draw 14 10.4% 50 conclusions. 12 40 10

8 30

6 276+ 20 3.4 0.5% ACTIVITIES 4 +THIRD VPA 0-137 138-275 276+ 10 ACTIVITIES ACTIVITIES ACTIVITIES COMPLETED 2 OPTIMAL USAGE LOW USAGE MEDIUM USAGE OPTIMAL USAGE & THIRD VPA COMPLETED 0 0 VOXY USAGE LOW 0-137 MEDIUM OPTIMAL OPTIMAL USAGE Learners who complete at least 276 activities and take three Voxy Proficiency USAGE ACTIVITIES USAGE USAGE & THIRD VPA COMPLETED Assessment™ exams at regular intervals see an average score improvement of 16%. 138-275 276+ ACTIVITIES ACTIVITIES

3.4 --> 0.5% 48.4 --> 10.4% 55.8 -- > 11.1% 67.1 --> 16.0% PRO TIP

Ideally, proficiency assessment should be domain agnostic. This means that outcomes shouldn’t depend on what topics the learner knows more or less about. If learners are tested on a domain they’re familiar with, such as hospitality, they’re likely to outperform others who are less familiar with hospitality subjects. Many factors can affect proficiency improvement, so it should never be used exclusively to measure learner outcomes.

11 LET’S REVIEW... VOXY’S 7 METRICS FOR MEASURING LEARNER OUTCOMES

TIME ON TASK is one of the biggest predictors of success when acquiring a new skill

FREQUENCY of practice affects long-term results

ACHIEVEMENT shows how well learners understand the material they have been taught

INSTRUCTOR FEEDBACK is key in assessing the effectiveness of instruction

LEARNER SATISFACTION is measured through feedback on the user experience

REAL-WORLD SUCCESS means accomplishing personal goals

PROFICIENCY IMPROVEMENT includes the ability to read, write, speak, and listen in another language

12 POWERFUL ENGLISH IN ACTION

Powerful English is about more than simply measuring outcomes—it’s­ also about how to capitalize on that information once it’s been collected. With the data gathered from the seven metrics, we can:

TRACK AND ASSESS LEARNER PROGRESS HOLISTICALLY

APPLY INSIGHTS FROM THE DATA TO ACCELERATE AND ENHANCE THE LEARNING EXPERIENCE

Demonstrating learner success is only part of the equation. To continue delivering the best language solution possible, we can identify patterns in the data and address specific areas for product improvement to benefit the learner even more.

When you take the next step and apply insights from the data to the product, you can do more than provide learners with what they want—you can give them exactly what they need.

www.voxy.com

13 CONCLUSION

DON’T JUST ASSUME YOUR ENGLISH LANGUAGE SOLUTION IS WORKING. CHOOSE A PROVIDER THAT DELIVERS PROVEN RESULTS.

Measurable learner success

Data-driven product improvement

Personalized learning for maximum results

OUTCOMES MATTER. LET’S TALK. 14 + ABOUT VOXY LEARN MORE Our mission is to change the way the world learns and teaches Online Language Learning in The Workplace: Maximizing Efficiency, Effectiveness, and Time-on-Task English. We’re calling for an end to the one-size-fits-all approach: Voxy’s Chief Education Officer Dr. Katharine Nielson reviews We want learners to see results from day one and for organizations empirical research on effective online language learning. to receive the best-in-class service they deserve. We’re the only provider to offer a fully personalized English course Evaluating the Impact of Blended Learning on Performance and Engagement of Second Language Learners based exclusively on real-world content. Voxy’s integrated web Voxy’s team of academic experts investigates learner engagement. and mobile platform provides an exceptionally versatile solution, whether you’re looking to supplement in-person instruction, Using Voxy for the TOEFL introduce cutting-edge technology into your blended learning An analysis of learner performance and engagement. curriculum, or deliver a comprehensive standalone English course. With unparalleled support services and a team of academic experts, The Voxy Approach Voxy is designed to meet your organization’s unique needs. Dr. Katharine Nielson presents an overview of Voxy’s methodology.

Contact us VOXY IS HEADQUARTERED IN [email protected]

www.voxy.com 15