Erider™ Field Test Final Report July 3, 2014 Revised October 10, 2014 Revised March 25, 2015

eRider™ Field Test Final Report

July 3, 2014 Revised October 10, 2014 Revised March 25, 2015

Prepared By Pat Hahn, TEAM OREGON Communications Manager

For TEAM OREGON Motorcycle Safety Program TEAM OREGON eRider™ Field Test Final Report

Table of Contents 1. Introduction ...... 4 2. Objective ...... 4 3. Executive Report ...... 4 4. Background ...... 6 A. TEAM OREGON ...... 6 Mandatory Training ...... 6 B. TEAM OREGON Basic Rider Training Curriculum 1984-Present ...... 8 5. eRider™ vs. BRT Classroom ...... 9 A. eRider™ Development and Curriculum Overview ...... 9 B. Curriculum Template ...... 9 C. Curriculum Development Process ...... 10 D. Proof-of-Concept (Chapters 1-2b) Field Test Results ...... 13 E. Curriculum Overview ...... 14 Learning Objectives ...... 14 6. The eRider™ Field Test ...... 15 A. Design and Modifications ...... 15 B. Marketing ...... 19 C. Registration and Troubleshooting ...... 19 D. Data Collection and Analysis ...... 21 Statistical Analysis ...... 21 Curriculum Comparison ...... 22 E. Survey Results ...... 22 Student End-of-Course Evaluations ...... 22 Student Post-then-Pre-Course Surveys...... 26 Survey Results – Conclusions ...... 28 F. Student and Instructor Feedback...... 28 Online Student Comments ...... 28 Instructor Surveys ...... 30

TEAM OREGON eRider™ Field Test Final Report

G. Pass Rates, Knowledge and Skill Test Results ...... 31 Field Tests 1 and 2 ...... 31 Demographics ...... 32 Pass Rates...... 33 BRT Knowledge Test (BRT and ONB) ...... 35 Online Quizzes (ONB and ONI) ...... 36 Skill Test (BRT, ONB, IRT, ONI) ...... 38 Pass Rates, Knowledge and Skill Test Results – Conclusions ...... 38 7. Conclusions and Recommendations ...... 39 Conclusions ...... 39 Recommendations ...... 41 8. List of Appendices ...... 41

TEAM OREGON eRider™ Field Test Final Report

1. Introduction

The purpose of this report is to provide detail on the background and development of the eRider™ online classroom, its field test design and protocol, the field test results and recommendations.

2. Objective

The objective of the field test was to compare the eRider™ online classroom to the Basic Rider Training (BRT) classroom to determine its suitability as an approved classroom option for TEAM OREGON beginning and intermediate rider training students. The field test also measured how well each curriculum addresses the National Highway Traffic Safety Administration (NHTSA) Model National Standards for Entry-Level Motorcycle Rider Training and TEAM OREGON priority areas.

3. Executive Report

TEAM OREGON has a strong history of developing effective motorcycle rider training curricula, designing the BRT classroom in 2003 and BRT range in 2004 – both are approved by Oregon Department of Transportation (ODOT) for a motorcycle endorsement testing waiver. TEAM OREGON developed Intermediate Rider Training (IRT) shortly thereafter, which is also a course approved by ODOT for a skill test waiver. TEAM OREGON also developed Advanced Rider Training (ART), Police ART, Police High-Speed and Rider Skills Practice (RSP) to meet the various needs of Oregon motorcycle riders.

In 2010-2014, to better meet the needs of Oregon riders and to address the newly published Model National Standards for Entry Level Rider Training, TEAM OREGON developed and field tested the eRider™ online classroom. eRider™ is not a redesign of any existing curricula – it was created brand new from the ground up. The curriculum topics are informed by the National Standards, with extra emphasis placed on TEAM OREGON priority areas.

As part of the eRider™ field test, TEAM OREGON conducted an analysis and comparison of eRider™ and BRT curricula to measure how well each addresses the National Standards. In the final tally, eRider™ exceeds BRT – the National Standards are addressed 633 times by eRider™ and 538 times by BRT. When BRT range exercises are taken into account, eRider™ addresses every standard in the National Standards.

The eRider™ curriculum focuses heavily on eight curriculum priority areas based on needs of Oregon riders:

TEAM OREGON eRider™ Field Test Final Report

• Risk awareness • Riding strategy • Scanning • Cornering and vehicle control skills • Judgment • Impairment • Riding gear • Group riding

The comparison of eRider™ and BRT to the National Standards also demonstrated that eRider™ exceeds BRT in addressing TEAM OREGON curriculum priority areas by a margin of 34-50 percent.

TEAM OREGON conducted two eRider™ field tests to evaluate effectiveness. The first showed slightly lower pass rates and knowledge test scores, so the curriculum was modified for a second field test. The second field test showed no difference in pass rates and test scores. This report combines the results of both field tests.

The field test design focused on student outcomes, comparing the online classroom and range to those of the traditional classroom and range. In all, 1,086 basic and intermediate students participated in the field test – 440 in the beta group and 646 in the control group. The outcomes measured were:

• End-of course evaluation scores • Post-then-pre-course survey scores • Course completion status (pass rate) • Knowledge test scores • Skill test scores

The results of these measures were mixed, but overall demonstrated that eRider™ meets the approved instructional standard set by BRT classroom. Among the findings:

1. In the end-of-course evaluations, basic students gave higher ratings to the traditional classroom; intermediate students gave higher ratings to the online classroom. 2. In the post-then-pre-course surveys, students gave consistently higher ratings to the online classroom. 3. Students in all groups reported significant increases in knowledge of motorcycle riding topics after completing the course for both online and traditional classroom.

TEAM OREGON eRider™ Field Test Final Report

4. Feedback from students and instructors about the online classroom was very positive. 5. BRT and eRider™ students are equally likely to pass the course. 6. BRT and eRider™ students scored the same on the BRT knowledge test; eRider™ students’ scores on the online quizzes were equivalent to that of BRT students. Therefore, the eRider™ knowledge test function meets the same assessment standard as the BRT knowledge test. 7. There was no difference in skill test scores between traditional and online students. Therefore, the eRider™ classroom provides instruction that meets the instructional standard set by BRT for skill testing.

Taken all together, the results demonstrate eRider™ meets and/or exceeds the BRT classroom, and is an effective and viable option for TEAM OREGON basic and intermediate students. The results also demonstrate that the current traditional classroom should not be discontinued at this time. TEAM OREGON recommends that the eRider™ online classroom curriculum be approved by ODOT for use starting in 2015.

4. Background

A. TEAM OREGON

Since 1984, TEAM OREGON, headquartered at Oregon State University, has been responsible for the design, implementation and management of the rider training arm of the Oregon motorcycle safety program. Active partnerships include ODOT Transportation Safety Division, Driver and Motor Vehicle Services (DMV) and the Governor’s Advisory Committee on Motorcycle Safety (GAC) among many others including site sponsors and community colleges.

Mandatory Training

In 1989, completion of an approved basic rider training course became mandatory in Oregon for new riders younger than 19 who wanted a motorcycle endorsement. In 1998, this law was extended to include all new riders younger than 21. In 1991, DMV began waiving the state skill test for a motorcycle endorsement for any student who completed the TEAM OREGON basic course. In 1995, DMV also began waiving the state knowledge test for basic course graduates. In 2004, DMV began waiving the state skill test for intermediate course graduates – the intermediate courses being a shortened version of the approved basic course.

TEAM OREGON eRider™ Field Test Final Report

Figure 1.

Mandatory Training and Riders Trained

14,000 2012 Under─ 41 12,000 2013 Under─ 51 2011 Under─ 31 10,000

8,000

6,000 Students Trained Students

4,000 1998 Under─ 21

2,000 1989 Under─ 19

- 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

This figure shows the growth of rider training in Oregon since 1984 and the years training became mandatory for new riders. TEAM OREGON currently trains more than 12,000 students a year.

In 2009, Oregon law was modified further to require all new riders to complete an approved training course, phased in over five years based on age:

January 1, 2011 All New Riders 30 and under

January 1, 2012 All New Riders 40 and under

January 1, 2013 All New Riders 50 and under

January 1, 2014 All New Riders 60 and under

January 1, 2015 All New Riders of Any Age

TEAM OREGON eRider™ Field Test Final Report

B. TEAM OREGON Basic Rider Training Curriculum 1984-Present

TEAM OREGON began training riders in 1984 using curricula provided by the Motorcycle Safety Foundation (MSF). In 1986, the MSF Motorcycle Rider Course: Riding and Street Skills (MRC: RSS) became the approved curriculum for novice riders in Oregon.

In 1996, MSF announced it would create a new curriculum to replace the MRC: RSS, and in 2001 released the Basic Rider Course (BRC). TEAM OREGON elected to evaluate the BRC before using it in the field, and created the BRC Task Force in February 2001. The Task Force concluded that the classroom component of the BRC was insufficient, as acknowledged in the BRC Field Test Final Report:

“(The) Task Force was not willing to subject TEAM OREGON students to a classroom curriculum they felt was incomplete. Therefore, the classroom portion of the BRC was not tested.”

A field test of the range portion was conducted during the 2002 training season. This field test involved 354 students who were given BRC range, and a comparable number of students who were given MRC: RSS range. Both groups were administered an end-of- course skills test that combined the skills tests from BRC and MRC: RSS.

It was the unanimous recommendation of the task force that TEAM OREGON not adopt the BRC (classroom or range) as presented. This recommendation and a full report were provided to ODOT in November of 2002. Based on the findings of the field test, ODOT concluded that the BRC was not an appropriate curriculum for the novice riders of Oregon (and would not make a suitable replacement for the MRC: RSS) and directed TEAM OREGON to not adopt the BRC.

The MSF announced that it would no longer support the MRC: RSS curriculum (student textbooks, instructor guides, range cards, videos etc.) starting in 2004. Given the directive by ODOT to not adopt the BRC and the discontinued support of MRC: RSS, TEAM OREGON developed a new curriculum to replace the MRC: RSS.

The Basic Rider Training (BRT) classroom was field tested and deployed in 2003, followed in 2004 by BRT range. For more detail, see Appendix A. Shortly thereafter, IRT and RSP were deployed to meet the needs of riders with some experience. In 2004, DMV began waiving the state skill test for a motorcycle endorsement for students who completed IRT.

TEAM OREGON eRider™ Field Test Final Report

5. eRider™ vs. BRT Classroom

A. eRider™ Development and Curriculum Overview

In 2010, after gaining much experience designing and deploying the Oregon Parks and Recreation Department’s ATV safety training online classroom (Appendix B), TEAM OREGON began work on its own online classroom. This new curriculum, called eRider™, was meant to serve as an alternative to the seven-hour traditional basic classroom. The curriculum development team consisted of these subject matter experts:

• Steve Garets – TEAM OREGON director • Ray Pierce – TEAM OREGON training manager • Bob Reichenberg – motorcycle safety expert and technical writer • Stacey Axmaker – Idaho STAR director • Tom Mehren – SoundRIDER! journalist • Pat Hahn – TEAM OREGON communications manager • David Kertzner – instructional designer • A-VIBE Web Development – website design and deployment

B. Curriculum Template

The eRider™ online classroom was designed specifically to address the NHTSA Model National Standards for Entry-Level Rider Training (Appendix C). The standards were developed by an expert working group convened by NHTSA in 2008, among them was TEAM OREGON Director Steve Garets. The National Standards details 168 standards in six basic categories:

1. Motorcycle Pre-ride Tasks 2. Vehicle Control Skills 3. Street Strategies 4. Roadway Management Skills 5. Tasks Related to Carrying Passengers, Cargo, Group Riding, and Touring 6. Factors Adversely Affecting Rider Performance

The purpose of the National Standards was to provide specific, strong, and measurable education standards as tools to ensure students receive the level of information and experience necessary to properly prepare them for real-world riding situations. In addition to providing that foundation, the National Standards permit great flexibility in course development and delivery and facilitate growth and improvement in state

TEAM OREGON eRider™ Field Test Final Report

education systems. TEAM OREGON used the National Standards to inform the design of the new eRider™ curriculum.

The National Standards establish baseline content that all entry-level riders should be taught in motorcycle rider training classes held in United States. States are encouraged to work with curriculum developers to not only include lessons that address the National Standards but to also go beyond the standards where needed to address state- specific priorities, crash causes and trends. Curricula tailored to local and regional needs, in addition to delivering baseline content, is expected to produce better- informed students and safer riders. TEAM OREGON was careful to assess and clarify Oregon motorcyclist safety priorities first, before any curriculum content was developed, reflecting the National Standards template.

It should be noted that the National Standards are meant to be used as educational standards, not a curriculum. The National Standards document encourages developers to address some tasks, such as rider responsibility, protective gear, distractions and impairments in multiple places throughout the developer’s curriculum design. TEAM OREGON curriculum designers did just that, and threaded important priority-area themes throughout the content. Curriculum developers were also advised in the National Standards to determine what formats, activities, resources and tests should be employed to support the standards, including how much time and attention should be spent on a particular issue and in what medium the instruction should take place (i.e. classroom, range, etc.) With the National Standards as a guideline, TEAM OREGON began the development of a brand-new classroom curriculum.

C. Curriculum Development Process

TEAM OREGON used a two-stage process to develop the six chapters that comprise the eRider™ curriculum. The first stage, completed during 2010-11, was the development of the first two chapters (riding gear and motorcycle controls) for an initial “proof-of- concept” field test. The second stage, completed during 2012-14, was to refine the first two chapters and develop the remaining four chapters. This section describes generally the procedures applied to both stages of curriculum development.

During the initial curriculum framing process, Steve Garets met with his core team of subject matter experts to discuss TEAM OREGON’s priority areas. This group, which included Ray Pierce, Bob Reichenberg, Stacey Axmaker, Tom Mehren and Pat Hahn, met regularly over the course of several months and arrived at a consensus for eight

TEAM OREGON eRider™ Field Test Final Report

curriculum priority areas, based on state and national data, ODOT priorities, and many collective years of experience in the needs of new riders:

• Risk awareness • Judgment • Riding strategy • Impairment • Scanning • Riding gear • Cornering and vehicle control • Group riding skills

It should be noted that, with the exception of cornering, the primary areas of focus for the classroom were mental riding skills. The physical skills of straight-line riding, turning, shifting, stopping, emergency braking and swerving are covered thoroughly during the range portion of the class and therefore not as heavily emphasized in the online classroom. The eRider™ online classroom was developed to be supported by range exercises.

eRider™ online classroom is not a redesign of any existing curricula – it was created intentionally “from scratch.” Tom Mehren of SoundRIDER!, a contractor for TEAM OREGON, wrote the draft curriculum text informed by the National Standards, with emphasis on TEAM OREGON priority areas. Mr. Mehren was chosen to write the curriculum because he was a longtime proponent of safe and responsible motorcycle operation, had a solid history of motorcycle journalism in the Pacific Northwest, and no affiliation with any state or national motorcycle rider training program or curriculum provider. Mr. Mehren’s inclusion in the process was critical to keeping the new curriculum free from any relationship, real or imagined, with current basic rider training classroom curricula offerings in the state, region or nation. No existing curricula materials were provided, and Mr. Mehren was instructed to avoid any such reference.

When the draft curriculum was complete, Bob Reichenberg was contracted as editor to ensure the content addressed every standard in the National Standards, every component of the TEAM OREGON priority areas, was technically accurate and instructionally sound. Mr. Reichenberg worked closely with Pat Hahn to ensure the final curriculum document met all the objectives needed for a basic motorcycle safety classroom curriculum. Great care was taken by Mr. Reichenberg and Mr. Hahn to make certain there was no content taken from any existing basic rider training classroom (e.g. BRT, BRC, MRC: RSS, etc.) anywhere in the curriculum.

TEAM OREGON eRider™ Field Test Final Report

Once the curriculum text was reviewed and approved by Mr. Garets, David Kertzner, a contractor for A-VIBE Web development, applied a comprehensive instructional design process to prepare it for the online classroom environment. This involved a full analysis of the curriculum content, breaking it up into six chapters and 100 individual modules, and rearranging the content as necessary for a smooth and logical instructional flow. In addition to introductory and chapter-quiz modules, Mr. Kertzner suggested more than 20 different module types to ensure variety, interactivity and an engaging online format. Mr. Kertzner worked closely with Pat Hahn to emphasize TEAM OREGON priority areas when conceiving and designing the interactive features to be used in the classroom. Mr. Kertzner, Mr. Garets and Mr. Hahn worked closely with A-VIBE to ensure that the online learning modules were designed as simply as possible for lowest costs and maximum educational effectiveness.

Figure 2.

eRider™ Production by the Numbers

500 video clips

350 images

88 final video presentations

52 interactive features

17 extras (models/actors)

12 professional motorcycle riders (TEAM OREGON instructors)

4 dedicated motorcycles

2 professional actors

1 voiceover narrator

1 enclosed trailer w riding gear/props

When the instructional design was complete, A-VIBE worked with TEAM OREGON to develop 100 “storyboards” to coordinate asset procurement and production (Figure 2). The storyboards detailed, scene for scene, every photo, image, graphic or computer animation to be used in eRider™. Asset production planning, recording, filming and

TEAM OREGON eRider™ Field Test Final Report

photography took place over six months and traveled to more than 20 filming locations in Portland, McMinnville, Carlton and Corvallis.

When filming was complete, Mr. Hahn worked closely with a video editor, graphic designer and A-VIBE to create the videos and interactive features that comprise the eRider™ online classroom. Video editing and production took place over three months. A-VIBE staff embedded the final assets into the eRider™ learning management system, a self-contained, Web-based, content-managed online administrative structure that gives TEAM OREGON full control of its content, student records and reporting tools.

D. Proof-of-Concept (Chapters 1-2b) Field Test Results

In fall of 2011, when the first two chapters of the eRider™ online classroom were complete, TEAM OREGON conducted a proof-of-concept test to determine if online classroom was feasible for motorcycle safety training (Appendix D). The field test compared results from the online classroom and range to those of the traditional classroom and range using student test scores. Instructors also administered a pre- course motorcycle controls quiz to students, completed a day-one student performance evaluation and post-course instructor debrief.

Overall, there were no significant differences between eRider™ students and traditional students by any measure. It was decided that online training, as carried out in Chapters 1-2b of the eRider™ online classroom, was suitable for educating riders about riding gear and control location and operation. eRider™ students were equally prepared for range training, knowledge testing and skill testing as those who completed the traditional classroom. Therefore, TEAM OREGON continued with the development of eRider™ Chapters 3-6 and a field test of the entire curriculum.

TEAM OREGON completed multiple internal reviews of eRider™ and one external review (including ODOT and GAC) to test the online classroom, fix any bugs, make any needed modifications and assess its appropriateness for basic motorcycle rider training students. In all, more than 200 modifications were made to content, design, navigation, function and administrative tools based on these reviews to prepare eRider™ to test on basic and intermediate students. TEAM OREGON submitted the online classroom to ODOT, the Oregon Transportation Safety Committee (OTSC) and GAC for approval to conduct a field test of the curriculum in spring of 2014. Permission for the field test was approved by OTSC and GAC, and granted by ODOT Transportation Safety Division Administrator Troy Costales February 14, 2014.

TEAM OREGON eRider™ Field Test Final Report

E. Curriculum Overview

The objective of eRider™ is to prepare new and returning motorcyclists to ride safely on the street. Upon completion, riders will know how to:

• Select appropriate riding gear • Identify common hazards and interpret clues in the riding environment • Use lane positioning to see and be seen, avoid hazards and maximize space • Identify conditions and impairments that affect rider safety and performance • Operate motorcycle or scooter controls to safely manage position, speed and path

The eRider™ online classroom uses multiple learning modules and activities to introduce and reinforce key motorcycle safety concepts. Review questions and quizzes are embedded in each chapter to help students evaluate their learning. Each chapter’s final activity asks students to apply key concepts and strategies to real-world situations.

Learning Objectives

Chapter 1 – Riding Gear Riders will learn the risks associated with operating a motorcycle and the importance, function and characteristics of proper protective gear.

Chapter 2 – Controls Riders will learn the location and operation of motorcycle or scooter controls, how to get underway, shift and stop. These skills are also practiced during the riding phase of the course.

Chapter 3 – Taking Control Riders will learn strategic lane positioning to see and be seen and the process of scanning to identify hazards, clues and escape routes. Riders will also learn strategies to avoid hazards and correct responses for dealing with hazards.

Chapter 4 – Riding Skills Riders will learn the proper techniques for cornering, braking and swerving. These skills are also practiced during the riding phase of the course.

Chapter 5 – Riding in the Real World

TEAM OREGON eRider™ Field Test Final Report

Riders will learn techniques for passing and adjusting to surface hazards and conditions of reduced traction and visibility. Riders will also learn proper techniques and cautions for riding in groups, carrying passengers and cargo and riding long distances.

Chapter 6 – Ready to Ride Riders will learn the requirements to ride legally in Oregon, what it means to be ready to ride and the factors that adversely affect rider performance. Riders will also learn the effects of impairments such as distraction, emotion, fatigue, alcohol and other drugs on riding.

The eRider™ full curriculum text with National Standards citations can be found in Appendix E. The National Standards can be found in Appendix C.

6. The eRider™ Field Test

A. Design and Modifications

In spring of 2014, the final eRider™ field test compared student results from the online classroom and range to those of the traditional classroom and range using student test scores, student evaluations of the course content and student evaluations of their own learning. The student knowledge test (K-test) consists of 50 questions with a requirement of 80 percent or greater answered correctly for a passing score. The student range test (riding skill evaluation) requires 20 “penalty” points or fewer for a passing score. The post-course student evaluations generally use a scale of 1-10, with 1 being “least” and 10 being “greatest” in 24 categories. The field test was also designed to collect demographic and behavior data to aid in future planning and implementation. In all, more than 40 student variables were observed and collected during the field test.

Beta group basic courses were scheduled March – May 2014 at Portland, Salem and Eugene training sites. A control group was selected from the same or nearby sites between during the same time period.

(continued on next page)

TEAM OREGON eRider™ Field Test Final Report

Figure 3.

Field Test Basic Courses (Planned) 50

40 34 36 30

eRider™ BRT BRT

For basic rider training students, 34 beta and 36 control courses were scheduled from March 14 to May 11.

Beta group intermediate courses were scheduled March – May 2014 at Portland training sites. A control group was selected from the same or nearby sites during the same time period.

Figure 4.

Field Test Intermediate Courses (Planned) 50

30 22 23 20

eRider™ IRT IRT

For intermediate rider training students, 22 beta and 23 control courses were scheduled from March 15 to May 11.

Estimating an average of 10.5 students per course, this schedule could generate 600- 700 students each in the beta and control groups.

TEAM OREGON eRider™ Field Test Final Report

All eRider™ students were required to complete the online classroom by noon Thursday before the first range training session. To integrate the eRider™ classroom into the existing range schedule, the range start days, start times, pre-range student orientation and some instructional terms were modified slightly. BRT and IRT classroom sessions were eliminated from Friday and Saturday course schedules; however, the one-hour BRT classroom session on Sunday remained so instructors could administer the written K-test. The K-test was identical to the test taken by all BRT students in 2014 and required for eRider™ students to receive a course completion card and qualify for the motorcycle endorsement qualification and testing waiver.

Based on feedback collected during the internal and external reviews, a bypass quiz was inserted into the Chapter 2 (Controls) introduction. To develop the bypass, TEAM OREGON tested an IRT controls quiz and analyzed student responses in the fall of 2013 (see Appendix H). The quiz allows eRider™ BRT and IRT students who are familiar with motorcycle or scooter controls a one-time opportunity to “test out” of Chapter 2. A score on the bypass quiz of 80 percent or better allows students to bypass the chapter. A score of 60-79 percent allows students to bypass the chapter, with the recommendation that they complete the chapter. Students who score 0-59 percent on the quiz are required to complete the chapter.

The bypass quiz had its intended effect. Students who didn’t need to learn motorcycle controls were able to bypass Chapter 2 for a more efficient classroom experience.

Figure 5.

eRider™ Chapter 2 Bypass Quiz 100 81 80 57 60

BRT IRT

In all, 57 percent of BRT students and 81 percent of IRT students used the bypass.

TEAM OREGON eRider™ Field Test Final Report

Bypassing Chapter 2 had no effect on student outcomes: The course passing rate for students who used the bypass was slightly better than those who completed the chapter, but the difference was not significant.

Figure 6.

Course Pass Rate Chapter 2 Bypass vs. Chapter 2 Completed

100 92 90 83 75 80 60 eRider™ BRT 40 eRider™ IRT 20 0 Bypass Completed

The pass rate for students who bypassed Chapter 2 was 82.9% for eRider™ BRT and 92.3% for eRider™ IRT. The overall pass rate for those groups was 75.8% and 90.3%, respectively.

The first day of range instruction was scheduled to begin a half hour earlier than the traditional schedule to allow instructors time to process student paperwork, provide the student orientation and introduce instructor hand signals. Instructional terminology in some riding exercises was changed to eliminate terms and phrases from BRT not included in the eRider™ curriculum, including ONE-C, SMOG-C, SIPDE, “ready-aim-fire” and “outside-inside-outside.” Due to the earlier start times and the range-only nature of day-one course meetings, late-arriving student policy for day one was modified for the eRider™ courses, as well.

To eliminate potential bias, the instructors were not given access to the online classroom prior to the field test. However, at general instructor update meetings during winter and spring of 2014, instructors were given overviews of the online classroom and field test plan, including review of eight eRider™ learning modules and review of the modifications to start times, student orientation, etc. Instructor response to the new curriculum at the updates was very positive.

TEAM OREGON contacted instructors assigned to eRider™ courses personally by phone so staff could ensure they were aware of these course modifications. Full detail of

TEAM OREGON eRider™ Field Test Final Report

course modifications and instruction was also provided in writing via e-mail to each lead instructor and by hard copy in the course file.

B. Marketing

Overall, courses using the eRider™ online classroom were marketed in the same manner as traditional courses. In the Rider Education Management System (REMS) student registration program, two new courses were added for the field test: ONB (online basic) and ONI (online intermediate). These courses were marketed online as eRider™ BRT and eRider™ IRT, respectively. New Web pages were created, specific to each course, which included important BRT and IRT information as well as additional detail about the online classroom, computer system requirements and completion requirements. Throughout the TEAM OREGON website, notifications were posted to raise awareness of the online- classroom option for select courses and sites.

Registration for ONB and ONI courses was voluntary: The choice of whether to take a course with an online or traditional classroom was left to students. At no time during the courses were the students informed that the course they were taking was part of a field test. There were no efforts on the part of TEAM OREGON or its sponsors to assign any particular student or group of students to either the beta or control groups. However, it is recognized that selection bias may be a concern, since students volunteering for the beta group may be more experienced with computer technology than those who choose the traditional course.

C. Registration and Troubleshooting

At the start of the field test, eRider™ registration was completed only by phone. Based on the successful design of the 2011 field test of Chapters 1-2, this allowed phone staff to verbally clarify system requirements, course completion requirements, deadlines and other details. However, due to low demand for rider training in March 2014 and very low demand for online-classroom courses during that time, several early ONB and ONI courses were cancelled.

(continued on next page)

TEAM OREGON eRider™ Field Test Final Report

Figure 7.

Field Test Basic Courses - Beta Group (Actual) 50

30 20 20 10 10

eRider™ BRT eRider™ IRT

In all, TEAM OREGON conducted 30 eRider™ field test courses – 20 eRider™ BRTs and 10 eRider™ IRTs

To boost enrollment, TEAM OREGON made online registration available for eRider™ courses. Courses immediately started filling up. Instead of a pre-registration phone conversation, staff called students after they registered to remind them of completion requirements, deadlines and the need to read their confirmation letters carefully.

Figure 8.

Field Test Students 800 646 600

400 306

200

Beta Group Control Group

In all, the field test used data from 306 beta-group students and 646 control-group students.

Three TEAM OREGON staff members provided technical support for eRider™ students during business hours. One staff member provided support after hours and on weekends. Staff was able to respond immediately to solve system requirements issues, confirmation and account validation problems and other technical difficulties. In all, 26

TEAM OREGON eRider™ Field Test Final Report

students contacted TEAM OREGON by phone or e-mail for help resolving eRider™ technical issues. The majority of issues reported were system “bugs” due to the Web browsers (i.e. Internet Explorer) students were using, and resolved by recommending students switch or upgrade browsers. There was no negative feedback to the recommendations to switch browsers.

Staff also monitored online student activity closely to ensure classroom completion before the start of the first range training session. Only two students who registered for an eRider™ course were rescheduled due to insurmountable technical difficulties or an inability to complete the classroom on time. On average, most students completed the online classroom six days before the start of the first range session.

D. Data Collection and Analysis

Statistical Analysis

All data were analyzed by Oregon State Master of Public Health (epidemiology) graduate student Jing Li, supported by Dr. Andrew Houseman, Ph.D., associate professor in the OSU College of Public Health and Human Sciences. Data were collected in four general categories: • REMS student records • eRider™ student accounts • End-of-course student evaluation forms • “Post-then-pre” student surveys

Samples of field test documents and instruments can be found in Appendix H. Staff created unique Excel worksheets for each 12-student course in both beta and control groups to record REMS and eRider™ data. These worksheets were added to a master Excel file upon completion.

The OSU Center for Teaching and Learning advised that additional surveys be given to students for the field test: A pre-course survey, then a post-then-pre-course survey. The pre-course survey measured student riding experience, knowledge and expectations of the course in 11 topic areas. The post-then-pre-course (followup) survey asked students to re-evaluate pre-course knowledge and expectations, then measured post-course knowledge and outcomes in 10 topic areas. (“Level of experience with computer technology,” the last question on the pre-course survey, was omitted from the post- course survey.) Results could be compared to the US Department of Education’s Evaluation of Evidence-Based Practices in Online Learning meta-analysis (see Appendix

TEAM OREGON eRider™ Field Test Final Report

L). At the end of the post-course survey, respondents were also prompted to volunteer to participate in a long-term evaluation of rider training effectiveness.

In all, 952 students were categorized into four different groups, based on course type: BRT, ONB, IRT or ONI. The objective was to evaluate the effectiveness of eRider™ courses compared to traditional courses. Therefore, the analysis process was designed to compare eRider™ (ONB and ONI) and traditional (BRT and IRT) students’ performance. Because the four courses represent two different “experience” levels (basic and intermediate), all the comparisons conducted were BRT vs. ONB and IRT vs. ONI. The student outcomes to be analyzed were:

• End-of course evaluation scores • Post-then-pre-course evaluation scores • Course completion status (pass rate) • Knowledge test scores • Skill test scores

All statistical analysis except the post-then-pre-course survey was performed using R project for statistical computing (http://www.r-project.org/). The post-then-pre-course survey comparison was performed using independent sample t-tests and Wilcoxon rank tests (http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test).

Curriculum Comparison

TEAM OREGON conducted an analysis and comparison of both eRider™ and BRT curricula to measure how well each curriculum addresses the National Standards. When compared to the National Standards, eRider™ exceeds BRT by addressing the standards 633 times, compared to 538 times by BRT. When BRT range exercises are taken into account, eRider™ addresses every standard in the National Standards. The comparison of eRider™ and BRT to the National Standards also demonstrated that eRider™ exceeds BRT in addressing TEAM OREGON curriculum priority areas by a margin of 34-50 percent. For full detail, see Appendices F and G.

E. Survey Results

Student End-of-Course Evaluations

TEAM OREGON students complete an evaluation form at the end of every course. The evaluation forms ask students to rate, on a scale of one to 10 (10 being highest), various

TEAM OREGON eRider™ Field Test Final Report

aspects of the course including the registration process, instructor coaching and overall satisfaction.

Student responses to three questions on the evaluation form were collected and analyzed for the eRider™ field test:

A. The range exercises improved my riding skills B. The classroom presentation improved my knowledge C. The course prepared me to ride on the street D. I would recommend the course to friends

In total, 922 field test students completed an end-of-course evaluation form. The scores were consistently high, with means ranging from 8.7 to 9.6 (Table 1).

Table 1. Evaluation Evaluation BRT ONB IRT ONI Questions Score 95% 95% 95% 95% CI CI CI CI A. The range exercises improved Mean 9.6 9.4 8.8 9.0 my riding skills. (N) (376) (185) (251) (109) B. The classroom presentation Mean 9.4 8.9 8.7 9.0 improved my knowledge. (N) (376) (184) (251) (109) C. The course prepared me to Mean 8.8 8.9 8.8 9.1 ride on the street.* (N) (369) (183) (250) (110) D. I would recommend the Mean 9.7 9.5 9.3 9.3 course to friends. (N) (966) (196) (477) (110) * Due to a programming error, control group responses to statement D were not collected. The controls for Statement D were sampled from 2013 student end-of-course evaluation data. The mean responses scores for Statement D in 2013 were not significantly different from those in 2014.

Statements A and D received the highest scores from basic students, particularly those who took a course with traditional classroom (BRT). Each question received the lowest scores from traditional-intermediate (IRT) students. Statement C received the highest scores from online-intermediate (ONI) students.

All responses to Statement A were high, and there was no significant difference in “The range exercises improved my riding skills” among basic and intermediate groups (Figure 9).

TEAM OREGON eRider™ Field Test Final Report

Figure 9.

Student Evaluation A – Range Improved Riding Skills

10 9.6 9.4 8.8 9.0

0 BRT ONB IRT ONI

Average student responses to Question A ranged from 8.8 to 9.6. The differences between traditional and online course student responses were not significant.

Mean student responses to Statement B ranged from 8.7 to 9.4. All scores for Statement B were high, but the differences in “The classroom presentation improved my knowledge” were significant between traditional and online courses for each group. BRT students had mean evaluation scores 0.5 points greater (p-value <0.05, 95% CI 0.28- 0.74) than ONB students. ONI students had mean evaluation scores 0.3 points greater (p-value <0.05, 95% CI 0.01-0.7) than IRT students (Figure 10).

(continued on next page)

TEAM OREGON eRider™ Field Test Final Report

Figure 10.

Student Evaluation B – Classroom Improved Knowledge

10 9.4 8.9 8.7 9.0

0 BRT ONB IRT ONI

Average student responses to Question B ranged from 8.7 to 9.4. The differences between traditional and online student responses were small, but significant.

For Statement C, the difference between BRT and ONB students was not significant. However, ONI students had mean evaluation scores 0.3 points greater (p-value <0.05, 95% CI 0.04-0.6) than IRT students (Figure 11). This difference was significant.

Figure 11.

Student Evaluation C – Course Prepared Me for Street

10 8.8 8.9 8.8 9.1

0 BRT ONB IRT ONI

Average student responses to Question C ranged from 8.8 to 9.1. The difference between traditional and online basic course student responses is not significant. The difference between traditional and online intermediate course student responses was small, but significant.

TEAM OREGON eRider™ Field Test Final Report

All responses to Statement D were very high, but there was no significant difference in “I would recommend this course to friends” among basic and intermediate groups (Figure 12). It is worth noting here, in respect to student responses to Statement D, that basic and intermediate students in Oregon are required by law to take the course. That these scores are so high suggests a positive and very worthwhile rider education experience.

Figure 12.

Student Evaluation D – Would Recommend to Friends 9.7 10 9.5 9.3 9.3

0 BRT ONB IRT ONI

Average student responses to Question D ranged from 9.3 to 9.7. The differences between traditional and online course student responses were not significant.

Student Post-then-Pre-Course Surveys

TEAM OREGON students also completed an online “post-then-pre” course survey after their course. The survey asked students to rate, on a scale of one to 10 (10 being highest), their knowledge of various motorcycle safety topics before and after taking the course. This survey format allows students to rate course curricula and effectiveness through an evaluation of their own learning.

Students rated their knowledge and engagement in 10 topic areas: 1. Features and function of protective riding gear 2. Control location and operation 3. Basic riding skills 4. Riding strategies 5. Advanced riding skills 6. Passing, surface hazards, weather 7. Passengers, group riding and touring 8. Effects of impairments

TEAM OREGON eRider™ Field Test Final Report

9. Classroom engagement 10. Practice-riding (range) engagement Students in all groups reported significant increases in knowledge of motorcycle riding topics after completing the course. Not surprisingly, basic students had significantly greater increases in learning than intermediate students (Figure 13).

Figure 13.

Post-Then-Pre-Course Survey Results

10 9.1 8.5 8.7 8.8 8 7.3 7.4

6 5.4 5.3

0 BRT ONB IRT ONI

Pre-course Post-course

Students rated eRider™ courses consistently higher overall than courses using traditional classroom, but the differences were not significant.

The most significant increases among BRT students were riding strategies, advanced riding skills, and passing, surface hazards and weather. ONB students had similar results. Significant increases for ONB students also included basic riding skills and carrying passengers, group riding and touring. Intermediate student surveys did not reflect as great a change in knowledge as basic students, but all score differences in every question in the surveys were statistically significant (p-value<0.05).

The average score of the post-course survey was 8.5 for BRT and 8.7 for ONB. The difference is consistent but not significant – BRT students reported the same post- course level of knowledge as ONB students. A consistent result was found between IRT and ONI, with an average post survey of 8.8 and 9.1, respectively. This difference is also consistent but not significant.

TEAM OREGON eRider™ Field Test Final Report

Survey Results – Conclusions

Student scores for all TEAM OREGON courses were consistently high.

• In the end-of-course evaluations, basic students gave slightly but significantly higher ratings to the traditional classroom; intermediate students gave slightly but significantly higher ratings to the online classroom; these responses balance out. • In the post-then-pre-course surveys, all students gave slightly but consistently higher ratings to courses using online classroom. • Students in all groups reported significant increases in knowledge of motorcycle riding topics after completing the course.

Taken all together, these results demonstrate eRider™ meets the current BRT standard and is an effective and viable option for TEAM OREGON basic and intermediate students, but the current traditional classroom should not be discontinued at this time.

Full details of the survey results can be found in Appendix J.

F. Student and Instructor Feedback

Online Student Comments

Student comments submitted online through the eRider™ website or e-mail were tracked and recorded. In total, 34 students (11 percent of the beta group) submitted 41 comments. Almost all comments were positive. Even when a student was having trouble navigating the site, encountering “bugs” or lacking necessary technical requirements, their satisfaction with eRider™ was good.

When students submitted a comment online, they were also asked to rate the course in multiple areas on a scale of 1-5, with 5 being highest. Overall satisfaction with the course among these responses averaged 4.3 out of 5.

(continued on next page)

TEAM OREGON eRider™ Field Test Final Report

Figure 14. Online Student Comment Form Ratings 25

19 20

10 6 5 2 2 1 0 Very Satisfied Very Unsatisfied 5 4 3 2 1

In all, 19 of 30 student responses rated overall satisfaction 5 out of 5.

Almost every student (24 of 26) who submitted a comment form and responded to the question, “Did you find the course worthwhile?” answered in the affirmative (Figure 15).

Figure 15.

Online Student Comment Form – Did You Find the Course Worthwhile? 25 24

5 2

0 Yes No

Only 2 of 26 student responses “did not find the course worthwhile.”

It is worth noting that the two students who gave the course a 1 rating were the same two who responded they did not find the course worthwhile.

TEAM OREGON eRider™ Field Test Final Report

Instructor Surveys

Instructors for the beta groups completed post-course surveys to compare their experience with eRider™ students to that of typical students, to see if there were any noticeable differences. The questions asked of instructors were:

1. How prepared were eRider™ students for the range (riding gear, driver’s license, waivers)? 2. How did eRider™ students perform on Day 1 of range? 3. How did eRider™ students perform on Day 2 of range? (ONB only) 4. How did eRider™ students perform in the classroom (K-test)? (ONB only)

Instructors were asked to rate student performance on a scale from 1-10 in which 1 was “worse,” 5 was “same” and 10 was “better.”

Figure 16.

Instructor Survey Results 10

6 5.5 5.3 5.6 4.5 4

0 Question 1 Question 2 Question 3 Question 4

Instructors rated students’ performance on the range as slightly better, and on the K-test slightly worse, than traditional BRT students.

There were minimal differences reported by instructors: eRider™ students were rated as slightly better prepared for the range and performed slightly better on the range day one and day two, but performed slightly worse on the K-test.

Instructors from nine of the 30 eRider™ courses submitted comments that students were generally less familiar with maximum braking concepts and took longer to “catch on” (i.e. meet objectives) for maximum braking on the range. Students also

TEAM OREGON eRider™ Field Test Final Report

demonstrated poor performance on K-test question 21, which measured understanding of maximum braking technique. (It should be noted that 18 of 20 eRider™ BRT courses used afternoon range schedules, which meant K-tests were administered before the second range session – when maximum braking exercises are conducted.) Based on this feedback, the eRider™ modules addressing braking and maximum braking will be modified to emphasize key concepts and techniques (see Appendix I). If student performance problems persist, additional modifications will be made to ensure student understanding and performance in maximum braking situations.

G. Pass Rates, Knowledge and Skill Test Results

Field Tests 1 and 2

Early results showed that eRider™ BRT (ONB) students had lower knowledge test scores than traditional (BRT) students. While it is not surprising that BRT students would perform better than ONB students on a BRT knowledge test, this represented a shortcoming of the eRider™ curriculum and needed to be corrected. An analysis of tests taken by ONB students provided insight into the differences between ONB and BRT student scores. Six test questions were missed by ONB students more than 18 percent of the time (the rate at which further analysis is done by the TEAM OREGON training manager). The test questions are in bold font, the correct answers in italics:

#15: Scanning is aggressive and purposeful

#16: A good head turn in a corner means the rider’s head and eyes are looking as far as possible through the turn

#19: The most common cause of single-vehicle motorcycle crashes is failure to negotiate turns/curves

#21: When applying maximum straight-line braking squeeze the front brake with increasing pressure

#28: Of the following, which is a leading cause of multi-vehicle collisions? Motorcyclists failing to be visible in traffic

#37: Countersteering means press left to go left, press right to go right

TEAM OREGON eRider™ Field Test Final Report

Figure 17.

Knowledge Test Questions Most Often Missed (ONB)

100

60 46% 40 28% 27% 29% 18% 19% 20

0 Q#15 Q#16 Q#19 Q#21 Q#28 Q#37

Six questions were missed by more than 18 percent of ONB students. Question #21 refers to maximum braking technique, a weakness also noted by instructors. The eRider™ curriculum was modified to better emphasize the related concepts.

As a result of the knowledge test analysis of questions missed by ONB students, TEAM OREGON modified the eRider™ curriculum to better emphasize those concepts. Most items were corrected with minor text modifications. However, because instructors also noted weaker understanding of maximum braking concepts (see Section F) among eRider™ students, more substantive modifications were made in the topic area of maximum braking. A summary of the eRider™ curriculum modifications can be found in Appendix I.

After these modifications were completed, TEAM OREGON conducted another, smaller field test measuring only pass rate and knowledge test scores. This second field test consisted of 12 ONB courses in September and October 2014 totaling 134 students. ONB students in the second field test performed as well as BRT students in pass rate, BRT knowledge test and skill test scores. Instructor feedback no longer identified maximum braking concepts as a concern.

Demographics

In all, 1,086 students enrolled in courses scheduled for the eRider™ field test. There were 646 students in the control group and 440 in the beta group. Approximately 80 percent of the students were male. The majority of students rode a motorcycle on the range; only 77 of 1,086 students rode a scooter, and six students used their own

TEAM OREGON eRider™ Field Test Final Report

vehicles. Table 2 shows the demographic distribution of the 952 students who participated in the first field test.

Table 2. Traditional eRiderTM

Course Type BRT IRT ONB ONI Total, n 391 255 193 113 Female 113 16 53 15 Male 278 239 140 98 Age 16-20 60 0 18 0 21-30 140 74 72 28 31-40 93 77 56 32 41-50 52 62 26 29 51-60 39 33 14 20 61+ 7 9 7 4 Vehicle Type Motorcycle 371 243 176 105 Scooter 19 11 17 8 Student Owned 1 1 0 0

Pass Rates

To successfully complete the course and earn a completion card, basic (BRT and ONB) students were required to pass both the BRT knowledge and skill tests; intermediate students were required to pass only the IRT skill test. (Additionally, ONB and ONI students were also required to complete 10 online quizzes, discussed later.) During the first field test, IRT students had the highest pass rate; ONB students had the lowest. During the second field test, ONB students had a slightly different pass rate than BRT students; the difference was not significant. ONI and IRT students also had slightly different pass rates; that difference was also not significant (Table 3).

(continued on next page)

TEAM OREGON eRider™ Field Test Final Report

Table 3. Course Passed Course Type N (%)

BRT 316 (80.2) ONB 1* 147 (75.8) ONB 2** 107 (79.8) IRT 237 (92.9) ONI 102 (90.3) *Field test #1 **Field test #2

The overall pass rates for field test students are in approximate alignment with historical pass rates. The pass rates for field test BRT and ONB students are below the BRT pass rate for the past four years (Figure 18).

Figure 18.

Course Pass Rates – Basic

100 84.4 80.2 79.9 80 75.8

0 Average 2010-2013* BRT ONB 1** ONB 2***

The BRT pass rate 2010-2013 was 84.4 percent, higher than basic field test groups. *The historical rate includes retest students. ** Field Test 1. *** Field Test 2.

The second field test showed that BRT and ONB students were equally likely to pass the course.

The pass rates for both IRT and ONI students are similar to the IRT percent pass rate for the last four years (Figure 19).

TEAM OREGON eRider™ Field Test Final Report

Figure 19.

Course Pass Rates – Intermediate

100 92.8 92.9 90.3

0 Average 2010-2013* IRT ONI

The IRT pass rate for 2010-2013 was 92.8 percent, slightly different from intermediate field test groups. *The historical rate includes retest students.

The difference in pass rates between field test students and students from previous years is most likely because the field test pass rates do not include retest students, which the historical rate does.

BRT Knowledge Test (BRT and ONB)

Basic students (BRT and ONB) were required to complete the BRT knowledge test, an in- person test consisting of 50 questions and proctored by a TEAM OREGON instructor. This test served as an objective measure to compare BRT and eRider™ classroom curricula. Intermediate students (IRT and ONI) were not required to complete this test – their knowledge was assessed with a knowledge test administered by DMV. The results of this comparison are found in Table 4.

Table 4. Course Type Failed Test Knowledge Test Score P-value N (%) Mean (SD)

BRT 12 (3.1) 92 (5.9) ONB 1* 11 (5.7) 88 (9.0) ONB 2* 2 (1.5) 91 (6.5) Comparison <0.0001

*Field test #1 **Field test #2

TEAM OREGON eRider™ Field Test Final Report

During the first field test, 12 BRT students (3.1 percent) failed the in-person knowledge test, compared to 11 ONB students, who had a higher failure rate of 5.7 percent, a significant difference. During the second field test, only 2 ONB students (1.5 percent) failed the in-person knowledge test, compared to 12 BRT students, who had a rate of 3.1 percent. The difference was significant.

During the first field test, the average test score for BRT students was 92 out of a possible 100; the average ONB score was 88. The difference was significant, and interpreted as ONB students missing two more test questions than BRT students. This finding also helped explain why the original ONB pass rate was 4.4 points lower than BRT in the first field test. The second field test showed ONB knowledge test scores were only slightly lower than BRT scores. The difference was not significant (Figure 20).

Figure 20.

Knowledge Test Scores – BRT vs. ONB

100 92 88 91

0 BRT ONB 1* ONB 2**

ONB students scored slightly lower than BRT students on the BRT knowledge test. The difference in Field Test 2 is not significant. * Field Test 1. ** Field Test 2.

Based on this evidence, TEAM OREGON concludes that eRider™ meets the instructional standard set by BRT classroom and can be approved as a valid classroom option.

Online Quizzes (ONB and ONI)

To complete the online classroom, all eRider™ students (ONB and ONI) were required to take 10 online quizzes, consisting of a minimum 83 questions. At the end of each eRider™ chapter, students completed a graded quiz. Chapters 1, 3, 4 and 5 also contain mid-chapter quizzes, however, these are not graded. On each of the six graded quizzes,

TEAM OREGON eRider™ Field Test Final Report

eRider™ students must answer a minimum of 8-10 questions correctly to proceed. (Every time a student answers a question incorrectly, he or she is asked an additional question.) In total, there are 366 quiz questions in the eRider™ online question bank. Students are required to answer at least 58 of them correctly to complete the course. The quiz questions come from the same TEAM OREGON question bank from which BRT test questions are derived.

The eRider™ chapter quizzes use a “semi-open-book” test format. Students can leave the quizzes at any time using a refresh-my-memory function and review course material. When students return to the quiz, however, they are asked a different question from the same topic area.

The online quizzes are graded on a scale of 0-100, same as BRT knowledge test. The pass rate of ONB and ONI students (both 93 percent) for the online quizzes was consistent with the pass rate of BRT students taking the BRT knowledge test (92 percent).

Figure 21. Knowledge Test Scores – BRT K-Test vs. Online Quizzes

100 92 93 93

0 BRT ONB ONI

The scores of ONB and ONI students on eRider™ chapter quizzes were not significantly different from the scores of BRT students on the BRT knowledge test.

Because the average scores for the eRider™ chapter quizzes were not significantly different from that of the BRT knowledge test, TEAM OREGON concludes that the eRider™ classroom prepares students for the eRider™ online quizzes as well as BRT classroom prepares students for the BRT knowledge test. This evidence also

TEAM OREGON eRider™ Field Test Final Report

demonstrates that the eRider™ knowledge test function meets the same assessment standard as the BRT knowledge test.

Skill Test (BRT, ONB, IRT, ONI)

A skill test was administered to students in all four course types, and required students to score 20 or fewer penalty points to pass. The results for the skill test are found in Table 41. There were no significant differences between traditional and online students.

Table 5. Course Failed Test N (%) Skill Test Score P-value Type Mean (SD)

BRT 56 (14.3) 14 (8.3) ONB 32 (16.5) 14 (8.5) Comparison > 0.05 IRT 18 (7.1) 11 (6.4) ONI 11 (9.7) 11 (6.3) Comparison >0.05

Among basic courses, 56 BRT students (14.3 percent) failed the skill test and 32 ONB students (16.5 percent) failed. Among intermediate courses, 18 IRT students (7.1 percent) failed the skill test and 11 ONI students (9.7 percent) failed. Neither difference was significant. The average skill test score of BRT and ONB students was the same at 14 points. The average skill test score of IRT and ONI students was the same at 11 points. These differences were also not significant.

Because none of the skill test score differences were significant, TEAM OREGON concludes that courses using the eRider™ classroom (ONB and ONI) provide instruction that meets the instructional standard set by BRT for skill testing.

Pass Rates, Knowledge and Skill Test Results – Conclusions

The results are mixed, but overall, eRider™ (after being modified according to Appendix I) meets the standard set by BRT classroom.

• The course pass rate was higher for intermediate students than for basic students • There was no difference between course pass rates of traditional classroom students and online students

TEAM OREGON eRider™ Field Test Final Report

• There was no difference between knowledge test scores of ONB and BRT students • ONB students were only half as likely as BRT students to fail the BRT knowledge test • There was no difference between BRT knowledge test scores and online quiz scores • There was no difference between basic and intermediate online quiz scores • The online quizzes are equivalent to the proctored knowledge test • There was no difference between traditional and online students on the skill test

7. Conclusions and Recommendations

Conclusions

TEAM OREGON has a strong history of developing effective motorcycle rider training curricula, and eRider™ meets and/or exceeds the current standard. eRider™ is not a redesign of any existing curricula – it is brand new curriculum informed by the National Standards and focused on TEAM OREGON priority areas.

When compared to the National Standards, eRider™ exceeds BRT by addressing the standards 633 times, compared to 538 times by BRT. When BRT range exercises are taken into account, eRider™ addresses every standard in the National Standards.

The comparison of eRider™ and BRT to the National Standards also demonstrated that eRider™ exceeds BRT in addressing TEAM OREGON curriculum priority areas by a margin of 34-50 percent.

The field test focused on student outcomes comparing the online classroom and range to those of the traditional classroom and range. The results of the measures were mixed, but overall demonstrated that eRider™ meets the approved instructional standard set by BRT classroom. Among the findings:

1. In the end-of-course evaluations, basic students gave higher ratings to the traditional classroom; intermediate students gave higher ratings to the online classroom; these responses balance out. 2. In the post-then-pre-course surveys, students gave consistently higher ratings to the online classroom.

TEAM OREGON eRider™ Field Test Final Report

3. Students in all groups reported significant increases in knowledge of motorcycle riding topics after completing the course for both online and traditional classroom. 4. Feedback from students and instructors about the online classroom was very positive. 5. BRT and eRider™ students are equally likely to pass the course. 6. BRT and eRider™ students scored the same on the BRT knowledge test; eRider™ students’ scores on the online quizzes were equivalent to that of BRT students. Therefore, the eRider™ knowledge test function meets the same assessment standard as the BRT knowledge test. 7. There was no difference in skill test scores between traditional and online students. Therefore, the eRider™ classroom provides instruction that meets the instructional standard set by BRT for skill testing.

Table 6.

eRider™ Compared to BRT Classroom – Conclusions Does Measure Not Meets Exceeds Meet Addresses National Standards* √ Addresses TEAM OREGON Priority Areas √ End-of-Course Student Evaluations (Basic) √ End-of-Course Student Evaluations (Intermediate) √ Post-then-Pre-Course Surveys √ Increase in Student Knowledge √ Student and Instructor Feedback √ Course Pass Rate √

BRT Knowledge Test Scores √

eRider™ Knowledge Test Function √ Skill Test Scores √ * BRT does not address all National Standards in its current form

TEAM OREGON eRider™ Field Test Final Report

classroom should not be discontinued. TEAM OREGON recommends that the eRider™ online classroom curriculum be approved by ODOT for use starting in 2015.

Recommendations

1. Adopt eRider™ as an approved classroom curriculum (with modifications according to Appendix I). 2. Offer eRider™ as student classroom option per the integration plan (Appendix K). 3. Do not discontinue the traditional BRT and IRT classroom option at this time. 4. Adopt the eRider™ online quizzes as equivalent to the BRT knowledge test for the DMV knowledge test waiver. 5. Adopt BRT and IRT courses conducted using eRider™ for DMV skill test waivers. 6. Monitor eRider™ student performance and modify the curriculum as needed.

8. List of Appendices

Appendix A – BRT Classroom Field Test Report (2004) Appendix B – eLearning Case Study - OPRD ATV Safety Appendix C – Model National Standards for Entry-Level Motorcycle Training Curricula Appendix D – Proof-of-Concept (Chapters 1-2b) Field Test Appendix E – eRider™ Curriculum Text with National Standards Citations Appendix F – eRider™ vs. BRT Curriculum Comparison Appendix G - eRider™ vs. BRT Appendix H – eRider™ Field Test Instruments End-of-Course Student Evaluation Form Instructor Surveys IRT Controls Quiz Results and Instruments Online Student Comment Form Sample Pre-Course Survey Pre-then-Post-Course Survey Subject Matter Expert Review Form BRT Knowledge Test – May Be Provided Upon Written Request eRider™ Website Access – May Be Provided Upon Written Request Appendix I – Curriculum Modification Plan Appendix J – Student Survey Results Appendix K – Integration Plan Appendix L – Online Learning Meta-Analysis