safety

Article Blueprint for a Simulation Framework to Increase Driver Training Safety in North America: Case Study

Kevin F. Hulme 1,* , Rachel Su Ann Lim 1, Meghan Bauer 2, Nailah Hatten 3, Helena Destro 4, Brenden Switzer 5, Jodie-Ann Dequesnay 6, Rebecca Cashmore 7, Ian Duncan, Jr. 8, Anand Abraham 9, Jacob Deutsch 10, Nichaela Bald 11, Gregory A. Fabiano 12 and Kemper E. Lewis 13

1 The Stephen Still Institute for Sustainable Transportation and Logistics, University at Buffalo, Buffalo, NY 14260, USA; [email protected] 2 Department of Psychology, College of Arts and Sciences, University at Buffalo, Buffalo, NY 14260, USA; [email protected] 3 Gilbane Building Company, Baltimore, MD 21230, USA; [email protected] 4 Warsaw Central School District, Warsaw, NY 14569, USA; [email protected] 5 Buffalo Public Schools, Buffalo, NY 14202, USA; [email protected] 6 The Boeing Company, Los Angeles, CA 90245, USA; [email protected] 7 Williamsville Central School District, Williamsville, NY 14051, USA; [email protected] 8 Honda R&D America’s Inc., Raymond, OH 43067, USA; [email protected] 9 Adient, Plymouth, MI 48170, USA; [email protected] 10 GE Aviation, Cincinnati, OH 45215, USA; [email protected] 11 Edwards Vacuum, Sanborn, NY 14132, USA; [email protected]


12 Department of Psychology, College of Arts, Sciences & Education, Florida International University,
 Miami, FL 33199, USA; gfabiano@fiu.edu 13 Citation: Hulme, K.F.; Lim, R.S.A.; Department of Mechanical and Aerospace Engineering, School of Engineering and Applied Sciences, Bauer, M.; Hatten, N.; Destro, H.; University at Buffalo, Buffalo, NY 14260, USA; [email protected] * Correspondence: [email protected]; Tel.: +1-716-645-5573 Switzer, B.; Dequesnay, J.-A.; Cashmore, R.; Duncan, I., Jr.; Abraham, A.; et al. Blueprint for a Abstract: Despite numerous recent advances in the classroom and in-vehicle driver training and Simulation Framework to Increase education over the last quarter-century, traffic accidents remain a leading cause of mortality for Driver Training Safety in North young adults—particularly, those between the ages of 16 and 19. Obviously, despite recent advances America: Case Study. Safety 2021, 7, in conventional driver training (e.g., classroom, in-vehicle, Graduated Driver Licensing programs), 24. https://doi.org/10.3390/ this remains a critical public safety and public health concern. As advanced vehicle technologies safety7020024 continue to evolve, so too does the unintended potential for mechanical, visual, and/or cognitive driver distraction and adverse safety events on national highways. For these reasons, a physics-based Academic Editor: Garrett Mattos modeling and high-fidelity simulation have great potential to serve as a critical supplementary com- ponent of a near-future teen-driver training framework. Here, a case study is presented that examines Received: 29 December 2020 the specification, development, and deployment of a “blueprint” for a simulation framework in- Accepted: 15 March 2021 tended to increase driver training safety in North America. A multi-measure assessment of simulated Published: 1 April 2021 driver performance was developed and instituted, including quantitative (e.g., simulator-measured),

Publisher’s Note: MDPI stays neutral qualitative (e.g., evaluator-observed), and self-report metrics. Preliminary findings are presented, with regard to jurisdictional claims in along with a summary of novel contributions through the deployment of the training framework, as published maps and institutional affil- well as planned improvements and suggestions for future directions. iations. Keywords: driving simulation; physics-based modeling; human factors; driver training; driver education; safety; public health; modeling and simulation (M&S)

Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article 1. Introduction distributed under the terms and Traffic accidents continue to be a primary safety and public health concern and are conditions of the Creative Commons a leading cause of death for young, inexperienced, and newly-licensed drivers (aged Attribution (CC BY) license (https:// 16–19)[1,2]. Per mile driven, young drivers are almost 300% more likely (than drivers in creativecommons.org/licenses/by/ older age groups) to be involved in a fatal driving accident [2]. Statistics from the same 4.0/).

Safety 2021, 7, 24. https://doi.org/10.3390/safety7020024 https://www.mdpi.com/journal/safety Safety 2021, 7, 24 2 of 24

source indicate an unfortunate gender effect: the crash fatality rate for young males is more than 200% higher than that for young females. Furthermore, crash risk has long been measured to be particularly high just after licensure [3,4]; per mile driven, crash rates are 150% higher for (just-licensed) 16-year-olds than for slightly older drivers who even have a mere 2–3 years of experience. Recent scientific evaluations [5] have demonstrated that current driver education programs (e.g., classroom, in-vehicle, Graduated Driver Licensing programs [6]) do not necessarily produce safer drivers. While some training programs have been found to be effective for procedural skill acquisition and others improve driver perception of dangers and hazards, conventional driver training does not provide teenagers with sufficient hands- on, seat-time exposure to the varied challenges of driving, which can be visual, manual, and cognitive [7] in nature. Novice drivers in training need to safely learn essential skills (e.g., basic vehicle mechanical operation, steering, turning, and traffic management), without the complexity of the normal driving environment. Historically, driver licensing in the United States has served mostly as a formality—if the student does not flagrantly violate the rules of the road during their brief on-road evaluation, he or she passes the driving exam [8]. In various countries in Europe (e.g., the Czech Republic, Finland, France, the Netherlands, and Slovakia), driver training requirements are much more rigorous and often include a significant component in addition to classroom and in-vehicle training assessments—through simulator-based training (SBT) [9]. A relevant example from the literature is the TRAINER project [10], which proposes a framework for SBT (in Europe, including implementations based in Spain, Belgium, Sweden, and Greece) and offers result comparisons between training implementation on various classes of simulators and across simulator fidelities. In recent times, SBT has been implemented for supplementary teen driver education and performance assessment [11–15] with numerous key advantages. For example, train- ing simulators have been implemented to improve driver understanding and detection of hazards and are useful for enhancing the situational awareness of novice drivers [16,17]. Furthermore, training in the controlled environment of a simulator provides critical ex- posure to many traffic situations (and some that are rarely encountered while driving a physical vehicle), trial-and-error (i.e., freedom-to-fail), safety/repeatability, and digital, real-time metrics of driving performance [18]. Training in a simulator can be extremely engaging—especially so for young trainees—and provides young learners with critical additional seat-time to common driving situations in an educational setting that is active and experiential [19,20]. Likewise, other recent studies [21] have investigated correlations between simulation performance (e.g., violations, lapses, errors, driving confidence) and natural tendencies observed when driving a physical vehicle (on-road). Despite these advantages and advances, SBT also has some noteworthy drawbacks. A simulator, by definition, is an imitation of reality, and for some trainees, it is difficult to overcome this perception to provide training that is meaningful. This inherent absence of both “presence” (i.e., a sense of “being there”, that is natural, immediate, and direct) [22] and “immersion” (i.e., the objective sensory notion of being in a real-world situation or setting) [23] in the simulator can result in negative training [24] and acquired skills might not be appropriately applied when transferred to the “real world”. Accordingly, prior to implementation, a simulator must be properly verified and validated (i.e., designed and implemented such that the simulator model serves as an accurate representation of the “real world”) both (a) to promote positive training for participants and (b) to ensure that the human behavior data collected are accurate and realistic [25–27]. Another primary concern with many simulator implementations is environment maladaptation, more commonly referred to as “simulator sickness” [28]. This side effect is often caused by “postural instability” [29]—a sensory conflict between what is observed and what is felt when interacting within the artificial environment that commonly manifests itself through a number of symptoms, including sweating, dizziness, headaches, and nausea. Fortunately, past studies have demonstrated that simulator sickness is less frequent Safety 2021, 7, 24 3 of 24

and less severe when experimental duration is modulated, especially so among young individuals [30]. Finally, simulator environment affordability is a major consideration. The fidelity of the simulator (e.g., field-of-view, user controls, cabin, motion capabilities, and various others) should be sufficient to promote positive training while remaining within a price domain (i.e., acquisition, maintenance, operation, upgrades) that is achievable for a typical training classroom. Ultimately, researchers are still trying to determine (and objectively measure) how simulator exposure compares to actual, real-world driving exposure. Establishing the efficacy of supplementary simulator exposure in young driver training requires evidence- based empirical studies; this very notion stands as the primary emphasis of this paper. Implementing a targeted sequence of SBT exercises and employing a companion holistic multi-measure assessment, the impact of the training framework was investigated on various aspects of observed young driver performance. Metrics included: (a) quantitative measures (e.g., speed and compliance behavior at intersections), as monitored and calcu- lated by the physics-based vehicle dynamics mathematical model that underlies the driving simulator engine; (b) evaluator qualitative measures, as a critical “human-in-the-loop” component to augment numerical assessment; (c) module questionnaires and self-report, as a means to monitor transfer knowledge (e.g., pre- and post-module and pre- and post- program); and (d) exit surveys/anecdotal evidence, as a means for teen trainees (and their parents) to comment on the advantages and disadvantages of the SBT framework. In Figure1, a “theory of change” (i.e., a logic model) illustrates both the overall workflow and the primary novelties of SBT mechanisms that have been developed and deployed for this work. In Research Methodology, the design and development of the training environment (including two implemented simulator types) is detailed; the suite of SBT modules (environments) are described and justified; the experimental cohort is described, a teenaged cohort who were recruited over a 2+ year timeframe. In Results and Discussion, preliminary findings are presented by successfully instituting a holistic performance assessment, the intended outcomes of which could have profound training benefits. In Broader Impacts of our Methodology and Implementation are discussed the numerous advantages of using games and modeling and simulation (M&S) in training. The paper culminates with Conclusions and Future Work. The primary goal is that the overarching return on investment for this work will be a template for reducing untoward Safety 2021, 7, x FOR PEER REVIEW 4 of 24 driving outcomes, which could lead to improved national driver safety in the long term.

Figure 1. Teen driving framework—a blueprint theory of change (with simulation). Figure 1. Teen driving framework—a blueprint theory of change (with simulation). 2. Research Methodology As a primary component of the training framework, a simulator training environ- ment was specified and composed that would allow for presentation of training content at suitable fidelity (i.e., both physical and psychological) while simultaneously enabling measurement and observation of human performance alongside a holistic assessment of participant outcomes. The training environment was designed to allow for much-needed training “seat time” while enabling experiential reinforcement of basic driving skills and also enabling a trainee’s freedom to fail. In other words, the environment promoted repe- tition and was such that the learner could take risks (i.e., where real-world consequences were greatly reduced). This allowed failure to be reframed as a necessary component of learning and made students more resilient in the face of obstacles and challenges (e.g., [31,32]. This study was approved on 21 October 2015 by the ethics committee of 030 Uni- versity at Buffalo Internal Review Board with the code 030-431144.

2.1. Training Environment Design The novel implementation offered two simulator types of varying features and phys- ical fidelity. The fixed-base simulator is comprised of a racing frame/chair, a high-fidelity spring-resisted steering wheel (with a 240° range of motion), game-grade foot pedals (gas, brake, and clutch), and a PC-grade sound system. The visual system consists of a four- wall arrangement (each 6′ high × 8′ wide) that provides drivers with forward, left, right, and rearward Wide Super eXtended Graphics Array Plus (WSXGA+) -resolution (1680 × 1050) views arranged at 90° (see Figure 2). The motion-based simulator is comprised of a 6-Degrees of Freedom (DOF) full-motion electric hexapod platform, a two-seat sedan pas- senger cabin, a moderate-fidelity steering wheel (with force-feedback capability and a 900° range of motion), high-fidelity foot control pedals (with spring-resisted gas and clutch pedals and pressure modulation on the brake pedal), and a 2.1 sound system. The visual system consists of a 16′ diameter theater arrangement, with a 6′ vertical viewing

Safety 2021, 7, 24 4 of 24

2. Research Methodology As a primary component of the training framework, a simulator training environ- ment was specified and composed that would allow for presentation of training content at suitable fidelity (i.e., both physical and psychological) while simultaneously enabling measurement and observation of human performance alongside a holistic assessment of participant outcomes. The training environment was designed to allow for much-needed training “seat time” while enabling experiential reinforcement of basic driving skills and also enabling a trainee’s freedom to fail. In other words, the environment promoted repeti- tion and was such that the learner could take risks (i.e., where real-world consequences were greatly reduced). This allowed failure to be reframed as a necessary component of learning and made students more resilient in the face of obstacles and challenges (e.g., [31,32]. This study was approved on 21 October 2015 by the ethics committee of 030 University at Buffalo Internal Review Board with the code 030-431144.

2.1. Training Environment Design The novel implementation offered two simulator types of varying features and physi- cal fidelity. The fixed-base simulator is comprised of a racing frame/chair, a high-fidelity spring-resisted steering wheel (with a 240◦ range of motion), game-grade foot pedals (gas, brake, and clutch), and a PC-grade sound system. The visual system consists of a four-wall arrangement (each 60 high × 80 wide) that provides drivers with forward, left, right, and rearward Wide Super eXtended Graphics Array Plus (WSXGA+) -resolution (1680 × 1050) views arranged at 90◦ (see Figure2). The motion-based simulator is com- prised of a 6-Degrees of Freedom (DOF) full-motion electric hexapod platform, a two-seat Safety 2021, 7, x FOR PEER REVIEWsedan passenger cabin, a moderate-fidelity steering wheel (with force-feedback capability 5 of 24

and a 900◦ range of motion), high-fidelity foot control pedals (with spring-resisted gas and clutch pedals and pressure modulation on the brake pedal), and a 2.1 sound system. The visual system consists of a 160 diameter theater arrangement, with a 60 vertical viewing capabilitycapability and and which which implements implements 6 Full High 6 Full definition High (FHD)-resolutiondefinition (FHD)-resolution (1920 × 1080) (1920 × 1080) projectorsprojectors to fullyto fully envelop envelop the simulator the simulator participant participant (see Figure3 (see). Figure 3).

FigureFigure 2. Fixed-base2. Fixed-base simulator. simulator.

Figure 3. Motion-based simulator.

The accompanying simulation software environment was developed in-house (using C++) so that its design could be tailored to exact training needs. While participating on the simulator, the driver adjusts the on-board Human Interface Devices (HID) controls (Stage 1), and these inputs are then delivered to a Windows-based workstation where a simula- tion analysis is performed (Stage 2). Physics-based model changes manifest themselves in required updates to three forms of output rendering: (i) the scene graphics state (Stage 3a), (ii) the motion platform state (Stage 3b), and (iii) the audio state (Stage 3c) of the sim- ulation. Notionally, this signal path is illustrated in Figure 4, whose design and imple- mentation has previously been described elsewhere [33]. Each of these primary I/O states are briefly described.

STAGE 3 STAGE 1 STAGE 2 O utput In p u t (D river C o n tro ls) Simulation Analysis Rendering

a 2+b 2=c 2

Steering Pedals Paddle Shifte rs Com putation and Visual Cues Motion Cues Audio Cues Physics-based m odeling

Figure 4. I/O signal pathway for training framework.

Safety 2021, 7, x FOR PEER REVIEW 5 of 24

Safety 2021, 7, x FOR PEER REVIEW 5 of 24 capability and which implements 6 Full High definition (FHD)-resolution (1920 × 1080) projectors to fully envelop the simulator participant (see Figure 3).

capability and which implements 6 Full High definition (FHD)-resolution (1920 × 1080) projectors to fully envelop the simulator participant (see Figure 3).

Safety 2021, 7, 24 5 of 24

Figure 2. Fixed-base simulator. Figure 2. Fixed-base simulator.

FigureFigure 3. 3.Motion-based Motion-based simulator. simulator.

TheThe accompanying accompanying simulation simulation software software en environmentvironment environment was developed was developed in-house in-house (using (using C++)C++) so soso that that that its its its design design design could could could be be tailored be tailored tailored to toex exactactto ex trainingact training training needs. needs. needs.While While participating While participating participating on the on on the the simulator, the driver adjusts the on-board Human Interface Devices (HID) controls simulator,simulator, the the driver driver adjusts adjusts the on-boardthe on-board Human Hu Interfaceman Interface Devices Devices (HID) controls (HID) controls(Stage (Stage (Stage 1), and these inputs are then delivered to a Windows-based workstation where 1),1), and and these these inputs inputs are are then then delivered delivered to a Windows-basedto a Windows-based workstation workstation where a wheresimula- a simula- tiona simulation analysis is analysis performed is performed (Stage 2). Physic (Stages-based 2). Physics-based model changes model manifest changes themselves manifest in tion analysis is performed (Stage 2). Physics-based model changes manifest themselves in requiredthemselves updates in required to three updates forms to of three output forms rendering: of output (i) rendering: the scene (i)graphics the scene state graphics (Stage 3a),staterequired (ii) (Stage the updatesmotion 3a), (ii) platform the to motionthree state forms platform (Stage of state3b),output and (Stage (iii)rendering: 3b), the andaudio (iii)(i) state the audio(Stagescene state 3c)graphics of (Stage the sim- state 3c) (Stage ulation.of3a), the (ii) simulation. Notionally, the motion Notionally, this platform signal this pathstate signal is (Stage illustrated path 3b), is illustrated andin Figure (iii) inthe 4, Figure whoseaudio4, state whosedesign (Stage designand imple- 3c) and of the sim- mentationimplementationulation. Notionally, has previously has previously this been signal described been path described elsewhis illustrated elsewhereere [33]. in Each [33 Figure]. of Each these 4, of whose theseprimary primary design I/O states I/O and imple- arestatesmentation briefly are briefly described. has described.previously been described elsewhere [33]. Each of these primary I/O states are briefly described.

STAGE 3 STAGE 1 STAGE 2 O utput In p u t (D river C o n tro ls) Simulation Analysis STAGE 3 STAGE 1 STAGE 2 Rendering O utput In p u t (D river C o n tro ls) Simulation Analysis Rendering a 2+b 2=c 2

Steering Pedals Paddle Shifte rs Com putation anda 2+b 2=c 2Visual Cues Motion Cues Audio Cues Physics-based m odeling Steering Pedals Paddle Shifte rs Com putation and Visual Cues Motion Cues Audi o Cues Physics-based m odeling FigureFigure 4.4. I/OI/O signal signal pathway pathway for for training training framework. framework.

2.1.1. StageFigure 1: Input—Driver4. I/O signal pathway Controls for training framework. User input commands are obtained with DirectInput (Microsoft), a subcomponent of DirectX, which allows the programmer to specify a human-interface device (e.g., steering wheel, gas pedal, brake pedal). The state of the input device (e.g., rotational position of the steering wheel, translational position of the pedals) can be polled with minimal latency in near real-time.

2.1.2. Stage 2: Simulation Analysis The simulation computer uses the control inputs to perform a numerical analysis whose outputs are used to render the primary output components of the simulation (i.e., visual, haptic, aural). Vehicle dynamics are calculated using the classical Bicycle Model of the automobile [34,35]. This simplified physics-based model has two primary degrees of freedom (DOF), which are yaw rate (i.e., rotational velocity) and vehicle sideslip angle (i.e., the angle between the direction in which the wheels are pointing versus the direction in which they are actually traveling). Forward velocity is added to the baseline model as a third primary DOF for the present implementation. Primary model inputs are: steering Safety 2021, 7, 24 6 of 24

wheel (rotation) angle and the longitudinal forces of the tires, and primary model outputs include vehicle velocities, accelerations, and tire forces. We then employ basic numerical integration (i.e., Euler’s method) to solve the equation outputs that define the current state of the vehicle (e.g., longitudinal/lateral velocity, yaw and rate, and longitudinal/lateral position). Updates to the state equations have to occur rapidly, so the step size between the updated and previous state takes place at a rate of 60 Hz. The artificial vehicles that populate the virtual driving environment alongside the human-driven vehicle are independently represented by a physics-based traffic model [36] that attempts to emulate actual human behavior (i.e., including driving errors and lapses).

2.1.3. Stage 3: Outputs—Visual, Haptic and Audio Once calculated, vehicle output states are subsequently converted into the six motion DOFs for haptic cueing: (heave, surge, sway, roll, pitch, yaw). Due to the finite stroke length of the electrical actuators on the motion platform, this conversion requires limiting, scaling, and tilt coordination (i.e., a procedure commonly employed that helps to sustain longitudinal and lateral acceleration cues with the gravity vector through pitch and roll, respectively [37]) by way of the washout filtering [38,39] technique. The updated DOFs are sent to the motion platform computer via datagram packets (UDP) for the entire simulation. Packet delivery is accomplished using Win64 Posix multithreads; in this manner, the update rate of the motion process is independent from that of the graphics process to avoid Central Processing Unit (CPU) / Graphics Processing Unit (GPU) bottlenecks. Simulation graphics have been developed using OpenGL, an industry-standard 3D graphics Application Programing Interface (API). The simulation framework employs OpenAL for sound events that are commonplace in a ground vehicle scenario: ignition/shutdown, engine tone, and cues for tire-surface behaviors and hazards.

2.2. Training Modules The training modules that were designed for this framework allow driving participants to engage with training content in a progressive manner. In this way, training exposure begins simple and gradually increases in complexity while additional tasks and driving challenges are introduced. These simulation-based driving modules were specified and designed to complement existing driver training methodologies (e.g., in-class and in- vehicle components, primarily) while each targets a specific skillset (i.e., mechanical, visual, cognitive, or multiple) widely acknowledged for safe driving. The training modules can be decomposed into three primary classifications: (a) closed course (CC) “proving ground” tracks (without traffic); (b) standard residential driving (both with/without accompanying traffic); and (c) challenge scenarios (always with traffic). These classifications are described and illustrated next.

2.2.1. Closed Course Driving The closed-course training environments allow us to evaluate human performance during specialized maneuvers without traffic. The skid pad [40] training course (Figure5) is a conventional automotive proving ground environment that is invaluable to help students better understand the complex relationship between speed and heading (i.e., steering angle). The tri-radial speedway (Figure6) integrates straightaways with curves and necessitates speed adjustments at the segment transitions (i.e., careful interplay between acceleration, braking, and turning). The figure-8 test track (Figure7) allows learners to experience both clockwise and counterclockwise turns in immediate succession. Note that performance data were not collected during these closed course scenarios. Regardless, they served as an invaluable mechanism for providing novice drivers with additional and repeatable hands-on practice within the simulator among varied geometric conditions. This critical experiential exposure and “seat time” served as an advantageous supplement to the drives where driver performance was quantified. Safety 2021, 7, x FOR PEER REVIEW 7 of 24 Safety 2021, 7, x FOR PEER REVIEW 7 of 24

ground” tracks (without traffic); (b) standard residential driving (both with/without ac- ground” tracks (without traffic); (b) standard residential driving (both with/without ac- companying traffic); and (c) challenge scenarios (always with traffic). These classifications companying traffic); and (c) challenge scenarios (always with traffic). These classifications are described and illustrated next. are described and illustrated next. 2.2.1. Closed Course Driving 2.2.1. Closed Course Driving The closed-course training environments allow us to evaluate human performance The closed-course training environments allow us to evaluate human performance during specialized maneuvers without traffic. The skid pad [40] training course (Figure 5) during specialized maneuvers without traffic. The skid pad [40] training course (Figure 5) is a conventional automotive proving ground environment that is invaluable to help stu- is a conventional automotive proving ground environment that is invaluable to help stu- dents better understand the complex relationship between speed and heading (i.e., steer- dents better understand the complex relationship between speed and heading (i.e., steer- ing angle). The tri-radial speedway (Figure 6) integrates straightaways with curves and ing angle). The tri-radial speedway (Figure 6) integrates straightaways with curves and necessitates speed adjustments at the segment transitions (i.e., careful interplay between necessitates speed adjustments at the segment transitions (i.e., careful interplay between acceleration, braking, and turning). The figure-8 test track (Figure 7) allows learners to acceleration, braking, and turning). The figure-8 test track (Figure 7) allows learners to experience both clockwise and counterclockwise turns in immediate succession. Note that experience both clockwise and counterclockwise turns in immediate succession. Note that performance data were not collected during these closed course scenarios. Regardless, performance data were not collected during these closed course scenarios. Regardless, they served as an invaluable mechanism for providing novice drivers with additional and they served as an invaluable mechanism for providing novice drivers with additional and repeatable hands-on practice within the simulator among varied geometric conditions. Safety 2021, 7, 24 repeatable hands-on practice within the simulator among varied geometric7 of conditions. 24 This critical experiential exposure and “seat time” served as an advantageous supplement This critical experiential exposure and “seat time” served as an advantageous supplement to the drives where driver performance was quantified. to the drives where driver performance was quantified.

Figure 5. The skid pad scenario (CC). FigureFigure 5. 5.The The skid skid pad pad scenario scenario (CC). (CC).

Safety 2021, 7, x FOR PEER REVIEW 8 of 24

Figurea notional 6. The depiction tri-radial speedwayof the residential scenario (CC). training environment, which can be endeavored Figure 6. The tri-radial speedway scenario (CC). Figureeither 6. withoutThe tri-radial or with speedway accompanying scenario (CC). traffic (as shown). 2.2.2. Residential Driving 2.2.2. Residential Driving The primary SBT environment is based upon a five square mile training area inspired The primary SBT environment is based upon a five square mile training area inspired by an actual residential region located in suburban Western New York State (i.e., Buffalo- by an actual residential region located in suburban Western New York State (i.e., Buffalo- Niagara region) and adjacent to the North campus of the University at Buffalo (where this Niagara region) and adjacent to the North campus of the University at Buffalo (where this research was performed). The residential environment includes actual suburban streets, research was performed). The residential environment includes actual suburban streets, including 2- and 4-lane roads and a 1-mile linear segment of the New York State (290) including 2- and 4-lane roads and a 1-mile linear segment of the New York State (290) Thruway. The residential simulation construct also includes road signs, lane markings, Thruway. The residential simulation construct also includes road signs, lane markings, and traffic control devices and allows the trainee to practice basic safe driving (e.g., speed and traffic control devices and allows the trainee to practice basic safe driving (e.g., speed maintenance, traffic sign and signal management) within a controlled, measurable, and maintenance, traffic sign and signal management) within a controlled, measurable, and familiar environment inspired by “real world” locations. Refer to Figure 8, which presents familiar environment inspired by “real world” locations. Refer to Figure 8, which presents FigureFigure 7. 7.The The figure-8 figure-8 scenario scenario (CC). (CC).

2.2.2. Residential Driving The primary SBT environment is based upon a five square mile training area inspired

by an actual residential region located in suburban Western New York State (i.e., Buffalo- Niagara region) and adjacent to the North campus of the University at Buffalo (where this research was performed). The residential environment includes actual suburban streets, including 2- and 4-lane roads and a 1-mile linear segment of the New York State (290) Thruway. The residential simulation construct also includes road signs, lane markings, and traffic control devices and allows the trainee to practice basic safe driving (e.g., speed maintenance, traffic sign and signal management) within a controlled, measurable, and familiar environment inspired by “real world” locations. Refer to Figure8, which presents a notional depiction of the residential training environment, which can be endeavored eitherFigure without 8. Residential or with driving accompanying scenario. traffic (as shown).

2.2.3. Challenging Scenarios Finally, within the residential driving environment, driving challenges were pre- sented that entailed additional vigilance and attentiveness on the part of the trainee through multitasking. To practice these skillsets, which are less common on the roadway (and therefore, less frequently encountered during in-vehicle training), a series of special hazard (challenge) scenarios were devised, during which to observe and measure trainee response and driving performance. These challenge scenarios included the following: 1. A construction zone with roadway cones, narrow lanes, and large vehicles 2. An aggressive driver (tailgater) segment in a narrowed section of roadway 3. Speed modulation segments, including a section with speed bumps (see Figure 9) 4. A complex highway merge with large vehicles and inclement weather 5. An animal crossing within a quiet cul-de-sac (see Figure 10) According to previously published statistics [41], young and inexperienced adult drivers are at elevated risk to be involved in a vehicle accident, underestimate hazardous situations, and drive more frequently in risky situations than more experienced drivers. Directly related to these driving deficiencies, the leading causes of fatal crashes among teens include: (i) speeding, (ii) driving too fast for road conditions, (iii) inexperience, (iv) distractions, and v) driving while drowsy or impaired. In specification and design efforts for the training modules (and accompanying evaluation protocols), it remained critical for us to explicitly address many of these prevailing weaknesses within the supplemental training framework. Accordingly, thoughtful consideration was given to the layout of each of the 120 min instructional sessions that comprise the 10 h simulation course. Of the documented five leading causes of fatal crashes, the training modules explicitly address the first four: (i) influencing the driver to be mindful of posted speed limits, (ii) making sure the trainee is monitoring speed in varied driving conditions, (iii) exposing new driv- ers to driving situations they have never seen before—and may only rarely encounter, and (iv) challenge scenarios that cause external distraction and demand multitasking through

Safety 2021, 7, x FOR PEER REVIEW 8 of 24

a notional depiction of the residential training environment, which can be endeavored either without or with accompanying traffic (as shown).

Safety 2021, 7, 24 8 of 24

Figure 7. The figure-8 scenario (CC).

FigureFigure 8. 8.Residential Residential driving driving scenario. scenario.

2.2.3.2.2.3. Challenging Challenging Scenarios Scenarios Finally, within the residential driving environment, driving challenges were presented Finally, within the residential driving environment, driving challenges were pre- that entailed additional vigilance and attentiveness on the part of the trainee through multitasking.sented that Toentailed practice additional these skillsets, vigilance which an ared lessattentiveness common on on the the roadway part of (and the trainee Safety 2021, 7, x FOR PEER REVIEWtherefore, through lessmultitasking. frequently encountered To practice during these in-vehicleskillsets, training),which are a seriesless common of special on hazard the roadway 9 of 24

(challenge)(and therefore, scenarios less were frequently devised, encountered during which toduri observeng in-vehicle and measure training), trainee a response series of special Safety 2021, 7, x FOR PEER REVIEWand hazard driving (challenge) performance. scenarios These challengewere devised, scenarios during included which the to following: observe and measure trainee9 of 24

1.response A construction and driving zone performance. with roadway These cones, narrowchallenge lanes, scenarios and large included vehicles the following: cognitive/mechanical/visual workload. A specific driving module to address the fifth (i.e., 2.1. AnA construction aggressive driver zone (tailgater) with roadway segment cone in a narroweds, narrow sectionlanes, ofand roadway large vehicles 3.2.impairment) SpeedAn aggressive modulation has been driver segments, designed (tailgater) including and segment aincorporated section in witha narrowed speed for bumpsanother section (see researchof Figure roadway9) project [42,43] but 4.cognitive/mechanical/visual A complex highway merge workload. with large vehiclesA specific and driving inclement module weather to address the fifth (i.e., 3.was Speednot explicitly modulation included segments, within including the current a section work with scope. speed bumpsThis training (see Figure prospect 9) is dis- 5.impairment) An animal has crossing been withindesigned a quiet and cul-de-sac incorporated (see Figure for another 10) research project [42,43] but 4.wascussed notA complex furtherexplicitly highwayin included Future merge Work. within with the large current vehicles work and scope. inclement This training weather prospect is dis- 5.cussed An further animal in crossing Future Work.within a quiet cul-de-sac (see Figure 10) According to previously published statistics [41], young and inexperienced adult drivers are at elevated risk to be involved in a vehicle accident, underestimate hazardous situations, and drive more frequently in risky situations than more experienced drivers. Directly related to these driving deficiencies, the leading causes of fatal crashes among teens include: (i) speeding, (ii) driving too fast for road conditions, (iii) inexperience, (iv) distractions, and v) driving while drowsy or impaired. In specification and design efforts for the training modules (and accompanying evaluation protocols), it remained critical for us to explicitly address many of these prevailing weaknesses within the supplemental training framework. Accordingly, thoughtful consideration was given to the layout of each of the 120 min instructional sessions that comprise the 10 h simulation course. Of the

documented five leading causes of fatal crashes, the training modules explicitly address FigureFigure 9. Speed9. Speed modulation modulation (challenge). (challenge). Figurethe first 9. four:Speed (i) modulation influencing (challenge). the driver to be mindful of posted speed limits, (ii) making sure the trainee is monitoring speed in varied driving conditions, (iii) exposing new driv- ers to driving situations they have never seen before—and may only rarely encounter, and (iv) challenge scenarios that cause external distraction and demand multitasking through

FigureFigure 10. 10.Animal Animal crossing crossing (challenge). (challenge). Figure 10. Animal crossing (challenge). 2.3. Instructional Sessions and Performance Measures 2.3. InstructionalIn advance of Sessions each instructional and Performance session, Measures an acclimation drive (of duration five minutes)In advanceis offered ofprimarily each instructional to serve as a warm-upsession, procedurean acclimation for drivers drive to become(of duration fa- five miliarminutes) with is the offered controls primarily and environment to serve asof ath warm-upe simulator procedure [44]. Each forof the drivers five training to become fa- sessionsmiliar with incorporates the controls new elementsand environment (e.g., traffic of behavior) the simulator and training [44]. Eachcontent of (e.g., the fivechal- training lengesessions scenarios) incorporates that build new upon elements the previo (e.g.,us traffic and scaffold behavior) driving and trainingknowledge content in a pro- (e.g., chal- gressivelenge scenarios) manner. Finally,that build at the upon conclusion the previo of theus fifth and training scaffold module, driving each knowledge participant in a pro- isgressive given a manner.culminating Finally, “final atexam” the conclusionin the form ofof a the virtual fifth road training test. This module, road test each serves participant as a cumulative mechanism to examine specific training outcomes that were encountered is given a culminating “final exam” in the form of a virtual road test. This road test serves explicitly within each of the previous five instructional sessions (and the training modules usedas a cumulativefor each). Refer mechanism to Table 1 to for examine a summary specific of the training instructional outcomes sessions, that an were overview encountered ofexplicitly the training within module(s) each of that the wereprevious deployed five instructionalfor that session, sessions and a brief (and a thedescription training of modules theused primary for each). driving Refer features to Table for each1 for instructional a summary session. of the instructional sessions, an overview of the training module(s) that were deployed for that session, and a brief a description of Tablethe 1.primary Overview driving of Training features Modules for for each each instructional Instructional Session. session. Session Training Module(s) Residential Drive Description/Features Table 1. Overview of Training Modules for each Instructional Session. Practice drives for participant acclimation (e.g., the steering wheel, the Acclimation paddle shifter signal indicators, and the gas and brake pedals) and the Session TrainingResidential Module(s) Residential Drive Description/Features drive simulatorPractice visual/motion/sounddrives for participant cues. acclimation Note that this(e.g., drive the issteering encountered wheel, at the Acclimation thepaddle beginning shifter of eachsignal of indicators,the five training and sessions.the gas and brake pedals) and the Residential Course has no traffic and contains three left turns, three right turns (in- drive Residential, simulator visual/motion/sound cues. Note that this drive is encountered at Session 1 cluding one designated as “no right on red”), five traffic lights, and four closed-course the beginning of each of the five training sessions. stop signs. Course length: 3.2 miles. Course has no traffic and contains three left turns, three right turns (in- Residential, Session 1 cluding one designated as “no right on red”), five traffic lights, and four closed-course stop signs. Course length: 3.2 miles.

Safety 2021, 7, 24 9 of 24

According to previously published statistics [41], young and inexperienced adult drivers are at elevated risk to be involved in a vehicle accident, underestimate hazardous situations, and drive more frequently in risky situations than more experienced drivers. Di- rectly related to these driving deficiencies, the leading causes of fatal crashes among teens include: (i) speeding, (ii) driving too fast for road conditions, (iii) inexperience, (iv) dis- tractions, and v) driving while drowsy or impaired. In specification and design efforts for the training modules (and accompanying evaluation protocols), it remained critical for us to explicitly address many of these prevailing weaknesses within the supplemental training framework. Accordingly, thoughtful consideration was given to the layout of each of the 120 min instructional sessions that comprise the 10 h simulation course. Of the documented five leading causes of fatal crashes, the training modules explicitly address the first four: (i) influencing the driver to be mindful of posted speed limits, (ii) making sure the trainee is monitoring speed in varied driving conditions, (iii) exposing new drivers to driving situations they have never seen before—and may only rarely encounter, and (iv) challenge scenarios that cause external distraction and demand multitasking through cognitive/mechanical/visual workload. A specific driving module to address the fifth (i.e., impairment) has been designed and incorporated for another research project [42,43] but was not explicitly included within the current work scope. This training prospect is discussed further in Future Work.

2.3. Instructional Sessions and Performance Measures In advance of each instructional session, an acclimation drive (of duration five minutes) is offered primarily to serve as a warm-up procedure for drivers to become familiar with the controls and environment of the simulator [44]. Each of the five training sessions incorporates new elements (e.g., traffic behavior) and training content (e.g., challenge scenarios) that build upon the previous and scaffold driving knowledge in a progressive manner. Finally, at the conclusion of the fifth training module, each participant is given a culminating “final exam” in the form of a virtual road test. This road test serves as a cumulative mechanism to examine specific training outcomes that were encountered explicitly within each of the previous five instructional sessions (and the training modules used for each). Refer to Table1 for a summary of the instructional sessions, an overview of the training module(s) that were deployed for that session, and a brief a description of the primary driving features for each instructional session.

Table 1. Overview of Training Modules for each Instructional Session.

Session Training Module(s) Residential Drive Description/Features Practice drives for participant acclimation (e.g., the steering wheel, the paddle shifter signal indicators, and the gas and Acclimation Residential brake pedals) and the simulator visual/motion/sound cues. drive Note that this drive is encountered at the beginning of each of the five training sessions. Course has no traffic and contains three left turns, three right Residential, Session 1 turns (including one designated as “no right on red”), five closed-course traffic lights, and four stop signs. Course length: 3.2 miles. Course has no traffic and contains four left turns, two right turns (including one designated as “no right on red”), six traffic Residential, closed-course, Session 2 lights, a roundabout intersection, and three stop signs. challenge Challenges (2): construction zone and speed modulation. Course length: 3.4 miles. Course introduces traffic and contains four left turns, two right Residential, closed-course, turns (including one designated as “no right on red”), eight Session 3 challenge traffic lights, and one stop sign. Challenges (2): construction zone and speed modulation. Course length: 3.2 miles. Safety 2021, 7, 24 10 of 24

Table 1. Cont.

Session Training Module(s) Residential Drive Description/Features Course introduces traffic and contains one left turn, five right turns (including one designated as “no right on red”), six traffic Residential, closed-course, Session 4 lights, and five stop signs. Challenges (3): construction zone, challenge speed modulation, and animal crossings. Course length: 4.0 miles. Course introduces traffic and contains three left turns, four right Residential, closed-course, turns (including one designated as “no right on red”), six traffic Session 5 challenge lights, and three stop signs. Challenges (2): tailgater scenario and speed modulation. Course length: 3.8 miles. Course introduces traffic and contains two left turns, six right turns (including one designated as “no right on red”), seven Virtual Residential, challenge traffic lights, and five stop signs. Challenges (4): construction Road Test zone, speed modulation, animal crossings, and highway merge. Course length: 4.7 miles.

Table2 summarizes the experimental protocol for each of the instructional simula- tion modules. Eligible participants came for their first visit accompanied by their legal guardian. At the beginning of this first session, parents and teens provided their informed consent and assent, respectively, to participate in the study. Each session began with a number of pre-surveys and self-report questionnaires issued in advance of the simulator exercises. This was followed by a video briefing detailing driving rules and regulations pertinent to that session’s specific content and a brief acclimation drive. Teens experienced the actual simulator exercises in pairs (i.e., one driver and one passenger). There were brief breaks/transitions placed in between driver/passenger load/unload and to assist with mitigating simulator sickness. The total duration of each of the five sessions was approximately two hours.

Table 2. Experimental Protocol for Each Instructional Session.

Activity Duration Informed consent/assent (first visit only) 10–15 min Pre-surveys and questionnaires 10–15 min Training module video briefing 10 min Acclimation drive 5–10 min Simulator training modules (as driver) 1 30 min Breaks/transitions 5–10 min Simulator training modules (as passenger) 1 30 min Post-surveys and questionnaires 10 min Discussion/summary/session de-briefing 5–10 min Total Module Time: 120 min (approx.) 1 Includes seat-time (and repetition) for the closed-course, residential, and challenge scenarios. To provide a more holistic understanding of simulation-based training on driving performance, the holistic performance measure assessment consisted of three primary subcomponents:

2.3.1. Quantitative Measures A printed simulator score report collected driver performance data for the entire duration of each training module for each instructional session. The score report rated per- formance on basic driving skills including: (a) travel speed on every street, (b) travel speed at every stop sign, and (c) excursion speed and traffic light state (i.e., red/yellow/green) while traversing both into and out of every stoplight zone. Penalties were assigned when drivers were traveling above the speed limit or when they failed to fully stop at controlled intersections. Cumulatively, these penalties resulted in a final percentage score for each Safety 2021, 7, 24 11 of 24

category (i.e., travel speed, stop signs, stop lights), which were then combined to create an aggregated total (0–100) score. Note that the simulator also compiled an (X/Y) trace of lane position in a separate data file. These data were plotted and any obvious/noteworthy trends were observed by the trainee post-experiment, but trainee compliance was not (currently) monitored as a component of the score report cumulative score.

2.3.2. Qualitative Measures The simulator training evaluator completed an evaluation checklist to subjectively rate the trainee’s performance (in real-time) on various driving skills (e.g., lane maintenance, use of turn indicators, and speed fluctuations while turning) on a 4-point Likert scale. Training evaluators also completed handwritten instructor (HI) comments, later coded to create scores for the number of positive and negative comments (positive HI; negative HI) issued per instructional session, per driver. These tabulations likewise resulted in a difference metric (difference HI), which quantified the difference between the positive and negative comments issued per instructional session, per driver.

2.3.3. Self-Report Measures A session-based questionnaire (SBQ) measured pre- and post-session knowledge of basic driving facts, but specifically for driving and roadway details that were pertinent to each of the five instructional sessions and the corresponding simulator training module. The simulator exercises were intended to reinforce driving skills introduced during the pre-SBQ (and an accompanying video briefing), and this information might serve as first- time exposure for many novice teen drivers. Subsequently, during the simulator exercise, drivers directly practiced these skills hands-on such that the session-targeted subject matter should be better understood once the post-SBQ was endeavored at the conclusion of each training module. Likewise, a Motion Sickness Assessment Questionnaire (MSAQ) [45] was employed to query how each driver felt before and after they drove the simulator to assess if, and to what degree, simulation had a negative impact on physiological response (e.g., nausea, dizziness, headache). In the full version of the questionnaire, there were 16 questions total, each rated on a 10-point (0–9) Likert scale, with each question addressing one of four possible symptom types (i.e., gastrointestinal; central; peripheral; sopite-related). The MSAQ resulted in sub-scores for each of these four categories, as well as an “overall” sickness score (i.e., ranging from 0 to 144). The MSAQ was administered post-experiment, for modules 1, 3, and 5. Finally, a participant satisfaction form was issued upon program exit to solicit overall program rating, comments, and suggestions. Separate forms were given both to the drivers (students) and to their parents. The student form asked a series of questions (10-point Likert scale) and also contained three open-ended questions asking students for different types of feedback on the program. The parent form assessed what parents thought of the program as a whole and was emailed to parents after their child had completed the entire 10 h program. Open-ended feedback comments (from students as well as parents) were later tabulated coded to provide trend metrics.

2.4. Experimental Cohort The study cohort size was N = 132 (66 male, 66 female). For deployment, only teenagers (ages 14–17, permitted in New York State, but unlicensed) were recruited who had little or no actual driving experience as a requirement of participation—per our advanced screening procedures. The intention was to recruit only participants who would be unbiased by pre-existing knowledge of actual on-road driving. This, would enable us to promote positive training on the simulator with a “clean slate” of truly novice/green drivers. Other eligibility criteria were strictly health-related: each candidate participant could not be susceptible to seizures, be claustrophobic, be afraid of dark environments, or prone to extreme motion sickness (e.g., car, plane, sea). Primary subject recruitment Safety 2021, 7, 24 12 of 24

sources included digital e-mailings to all regional high schools within a 60 min drive of campus (where the study was conducted), printed fliers posted about the campus, word- of-mouth referrals through families who had already completed the program (as well as related programs on the driving simulator), and printed advertisements within a popular community newspaper. The Institutional Review Board (IRB) at the University at Buffalo reviewed and approved our study protocol in advance of its deployment. Subjects were minimally compensated for their time (a $50 gift card) after completing all five of their two-hour SBT modules.

3. Results and Discussion Previously published guidelines [46] have summarized general requirements for the successful implementation of a motion simulator intended for driver evaluation. These requirements include: (a) concerns about measurable driver performance criteria, (b) per- formance with respect to simulator fidelity, and (c) any demerits associated with a simulator implementation related to maladaptation and simulator sickness. During deployment and subsequent collection and analyses of data, these guidelines were attempted compre- hensively through a variety of data forms. Accordingly, our results are decomposed into three primary categories of performance measurement: (i) quantitative (i.e., as measured by the simulator), (ii) qualitative (i.e., as observed by the training expert who monitored each instructional session), and (iii) surveys and questionnaires (i.e., as self-reported by driving participants and their parents, as appropriate). As a result, the overall analysis is representative of a truly multi-measure assessment that enabled a thorough and holistic evaluation of simulator-based driving performance. Results were tabulated, analyzed, and reported using a combination of both Microsoft Excel and the R statistical software package.

3.1. Quantitative Measures Frequency histograms were composed to illustrate score report total scores as collected and calculated by the computational engine of the driving simulator. These data are representative of core safe driving performance skills as programmed and monitored in real-time by the simulator (e.g., travel speed and behavior at intersections). Refer to Figure 11, which plots the average cohort total score on the x axis for each of the five training sessions (S1–S5), as well as the virtual road test (VRT) on the y axis. The figure demonstrates a general overall compliance to safe driving habits within the simulator; however, it indicates that there may be a ceiling effect beyond the first session. For the remainder of the sessions, ratings from the score report were biased toward the upper threshold of the rating scale, which is a direct indicator of expected improvement of cohort core driving skills (with continued practice/exposure). This trend culminated with the virtual road test, where observed (mean) cohort scores were the highest when compared against any of the training modules offered within each of the simulator training sessions. Embedded within each series are the corresponding standard deviations for these scores report total scores. As expected, there was a wider standard deviation for the first session; however, these deviations slowly and steadily reduced as training progressed, culminating with the smallest standard deviation for the virtual road test. This is a likely indicator that positive training was maintained and actually improved as the program progressed, as scores tended to increase more uniformly (i.e., with less variation) for the entire training cohort. Safety 2021, 7, x FOR PEER REVIEW 13 of 24

Safety 2021, 7, x FOR PEER REVIEW 13 of 24 however, these deviations slowly and steadily reduced as training progressed, culminat- ing with the smallest standard deviation for the virtual road test. This is a likely indicator that positive training was maintained and actually improved as the program progressed, however,as scores tendedthese deviations to increase slowly more uniformlyand steadily (i.e., reduced with less as variation) training progressed, for the entire culminat- training ingcohort. with the smallest standard deviation for the virtual road test. This is a likely indicator that positive training was maintained and actually improved as the program progressed, Safety 2021, 7, 24 13 of 24 as scores tended to increase more uniformly (i.e., with less variation) for the entire training cohort.

Figure 11. Score report totals.

In Figure 12, observed results are compared across simulator fidelities among mem-

FigureFigurebers 11.of 11. theScore Score cohort. report report totals. Note totals. that all recruited participants were assigned to a cohort (i.e., fixed- base or motion-based) for all training sessions. Although it may have been ideal to have all participantsInIn Figure Figure 12, observed12, operate observed results both areresults simulator compared are compar across types, simulator edenabling across fidelities asimulator comparison among membersfidelities of fidelities among withinmem- ofeach the cohort.participant, Note that the all overall recruited cohort participants size were(N = assigned132) provided to a cohort a (i.e.,sufficient fixed-base basis for compari- orbers motion-based) of the cohort. for allNote training that sessions.all recruited Although partic itipants may have were been assigned ideal to have to a allcohort (i.e., fixed- participantsbaseson of or human motion-based) operate performance both simulator for all across types,training enablingsimulator-fidelities. sessions. a comparison Although of For fidelities it the may first within have session, each been cohortideal to results have participant,allwere participants higher the overallon operatethe cohort fixed-base both size (simulatorN simulator= 132) provided types, (but a enablingwith sufficient much basisa comparison wider for comparison result of variance); fidelities withinfor the ofeachsecond human participant, session, performance observed the across overall simulator-fidelities.results cohort were size nearly(N = For 132) equal the provided first across session, a simulator sufficient cohort results types,basis for and compari- for the werethird, higher fourth, on the and fixed-base fifth sessions simulator (as (but well with as much the widervirtual result road variance); test), observed for the results were secondson of session, human observed performance results were across nearly simulator-fidelities. equal across simulator types,For the and first forthe session, third, cohort results fourth,wereslightly higher and higher fifth on sessions for the the fixed-base (as motion-based well as the simulator virtual simulator. road (but test), with The observed muchgeneral results wider hypothesis were result slightly forvariance); these observa- for the highersecondtions foris session,that the motion-basedfor theobserved first simulator.session, results the were The fixed-b general nearlyase hypothesis equal simulator across for thesewas simulator easier observations to types, learn andand operatefor the isthird,for that a fornovice fourth, the first trainee, and session, fifth and the sessions the fixed-base second (as simulator wellsession as wastheserved easiervirtual as to an learnroad inflection andtest), operate observed point. for However, results wereonce aslightlythe novice cohort trainee, higher became and for the thesuitably second motion-based acclimated session served simulator. to as an and inflectioncomfortable The general point. with hypothesis However, the companion once for these simulator observa- thetraining cohort becameenvironments, suitably acclimated the very to presence and comfortable of motion with the cues companion to accompany simulator the simulation trainingtions is environments, that for the thefirst very session, presence the of fixed-b motionase cues simulator to accompany was the easier simulation to learn and operate visuals,forvisuals, a novice as as well well trainee, as the as greater the and greater inducedthe second induced sense session of bothsens presenceeserved of both andas presence an immersion inflection and within point.immersion the However, within once the motion-basedthemotion-based cohort became simulator simulator suitably display display system,acclimated system, provided to providan a higher-qualityd comfortableed a higher-quality (i.e., with more the authentic) companion (i.e., more authentic)simulator learningtraininglearning environment environments,environment and therefore and the therefore very served presence to served better promoteof to motion better positive promote cues training to positiveaccompany overall. training the simulation overall. visuals, as well as the greater induced sense of both presence and immersion within the motion-based simulator display system, provided a higher-quality (i.e., more authentic) learning environment and therefore served to better promote positive training overall.

FigureFigure 12. 12.Simulator Simulator fidelity fidelity comparison. comparison.

Finally,Finally, individual individual components components of the score of the report score were report investigated were forinvestigated noteworthy for noteworthy trends. Refer to Table3, which presents a summary decomposed by simulator fidelity. In general,Figuretrends. 12. trainees Refer Simulator to showed Table fidelity stronger 3, which comparison. compliance presents to a postedsummary speed decomposed limits on the fixed-baseby simulator fidelity. In simulator.general, Astrainees shown inshowed the second stronger and third complian columns ofce the to Table,posted trainees speed drove limits almost on the fixed-base twicesimulator. asFinally, fast over As individual shown the speed in limit componentsthe onsecond the higher-fidelity and of the third score motioncolumns report simulator of were the (2.6Table,investigated vs. 1.5 trainees mph for drove noteworthy almost overtrends.twice the as speed Refer fast limit, overto Table on the average), speed3, which resulting limit presents on in slightlythe ahigh summary lowerer-fidelity speed decomposed sub-scores. motion Thissimulator by result simulator is (2.6 vs. fidelity. 1.5 mph In general, trainees showed stronger compliance to posted speed limits on the fixed-base simulator. As shown in the second and third columns of the Table, trainees drove almost twice as fast over the speed limit on the higher-fidelity motion simulator (2.6 vs. 1.5 mph

Safety 2021, 7, x FOR PEER REVIEW 14 of 24

over the speed limit, on average), resulting in slightly lower speed sub-scores. This result is somewhat surprising, as one would think that a higher-fidelity environment (both in terms of display system and motion capabilities) would result in more authentic driving behaviors and stronger vigilance to posted speed limits. These trends were actually ob- Safety 2021, 7, 24 served for the latter two score report sub-scores, where stronger compliance14 of 24 was observed toward intersection regulation (stop signs and streetlights) on the motion-based simulator relative to similar observations with the fixed-base simulator (i.e., reflected by the final somewhattwo columns surprising, of the as oneTable). would think that a higher-fidelity environment (both in terms of display system and motion capabilities) would result in more authentic driving behaviors and stronger vigilance to posted speed limits. These trends were actually observed for theTable latter 3. two Score score Report report Metrics sub-scores, by Simulator where stronger Type compliance(Entire Cohort). was observed toward intersection regulation (stop signs and streetlights) on the motion-based simulator relative Speed Stop Sign Street Light to similar observations with the fixed-base simulator (i.e., reflected by the final two columns Simulator Type Average Speeof thed table).over Speed Limit (mph) Sub-Score Sub-Score Sub-Score (0–100) (0–100) (0–100) Table 3. Fixed-base Score1.5 Report Metrics by Simulator Type (Entire 89.5 Cohort). 92.6 92.8 Motion-based 2.6 Average Speed Speed 87.3 Stop Sign96.1 Street Light 95.1 Simulator Type over Speed Sub-Score Sub-Score Sub-Score Limit (mph) (0–100) (0–100) (0–100) 3.2.Fixed-base Qualitative Measures 1.5 89.5 92.6 92.8 Motion-basedAlthough quantitative 2.6 measures 87.3 served as the 96.1 primary mechanism 95.1 for trainee per- formance assessment, training evaluators also performed a more subjective qualitative as- 3.2.sessment Qualitative (upon Measures pairs of trainees) during each instructional session. In this way, certain categoriesAlthough of quantitative trainee performance measures served could as thebe primarymonitored mechanism and measured for trainee that per- would be more formance assessment, training evaluators also performed a more subjective qualitative assessmentchallenging (upon to program pairs of trainees) a simulator during eachto quantify instructional with session. any meaningful In this way, accuracy. cer- In so do- taining, categories it helped of us trainee to maintain performance an couldever-critical be monitored “human-in-the-loop” and measured that would as a critical be constituent moreof the challenging overall human to program performance a simulator assessment. to quantify with To any this meaningful end, Figure accuracy. 13 presents In an example sodepiction doing, it helped of the usqualitative to maintain asse an ever-criticalssment for “human-in-the-loop” a single participant, as ain critical the form con- of an instructor stituent of the overall human performance assessment. To this end, Figure 13 presents anevaluation example depiction checklist of the (IEC). qualitative The IEC assessment was perf for aormed single participant,across three in the qualitative form of performance ancategories, instructor evaluation each rated checklist on a (0–4) (IEC). Likert The IEC scal wase performedand observed across over three qualitativethe five primary instruc- performancetional sessions. categories, The categories each rated onrepresent a (0–4) Likert an assessment scale and observed of (a) overall over the lateral five lane stability, primary(b) vigilance instructional for using sessions. turn The signals categories to representindicate anturns assessment and lane of (a) changes, overall lateral and (c) maintaining laneappropriate stability, (b) speed/distance vigilance for using speci turnfically signals toamidst indicate the turns “challenge” and lane changes, segments and of the training (c) maintaining appropriate speed/distance specifically amidst the “challenge” segments ofmodules. the training For modules. this particular For this particular trainee trainee (sample), (sample), overall, overall, qualitative qualitative ratingsratings generally im- generallyproved improvedacross the across five thetraining five training sessions—particularly, sessions—particularly, compliance compliance toto “turn “turn signal indica- signaltors”, indicators”, which demonstrated which demonstrated steady steady improvement improvement across across all all fivefive sessions. sessions.

FigureFigure 13. 13.Sample Sample instructor instructor evaluation evaluation (qualitative). (qualitative).

AsAs a mechanisma mechanism by which by which to culminate to culminate these qualitative these qualitative observations observations across the across the en- entire cohort, project evaluators simultaneously completed handwritten instructor (HI) comments,tire cohort, later project coded evaluators to create scores simultaneously for the number completed of positive and handwritten negative qualita- instructor (HI) com- tivements, performance later coded feedback to commentscreate scores issued for per the trainee, number per session. of positive These tabulationsand negative qualitative performance feedback comments issued per trainee, per session. These tabulations re- sulted in a difference metric, per training module. To this end, Figure 14 illustrates the

Safety 2021, 7, x FOR PEER REVIEW 15 of 24

Safety 2021, 7, 24 positive (POS) and negative (NEG) scores, as well as the difference15 of 24score (DIFF), averaged, across the entire training cohort and issued per training module for each of the primary five instructional sessions. The general observed tendency is that the difference metric is resultedpositively in a difference correlated metric, with per trainingadditional module. simulator To this end, exposure, Figure 14 illustratesas shown the in the plot. As the pro- positivegram (POS)progressed, and negative students (NEG) scores, continued as well as theto differenceprogressively score (DIFF), demonstrate averaged, additional positive across the entire training cohort and issued per training module for each of the primary fivedriving instructional habits sessions. and, through The general experiential observed tendency expo issure that with the difference the simulation, metric practiced gradual isperformance positively correlated improvement with additional amidst simulator varied exposure, and asmo shownre complex in the plot. driving As the situations. Note that programalthough progressed, the positive/negative students continued to evaluation progressively demonstratedifferential additional increased positive as intended, the overall driving habits and, through experiential exposure with the simulation, practiced gradual performancequantity improvementof negative amidst comments varied and issued more complexwere ob drivingserved situations. to steadily Note that increase across the five althoughtraining the sessions, positive/negative as illustrated. evaluation One differential likely increased reason asfor intended, this observation the overall is the gradual addi- quantitytion of of more negative challenging comments issued driving were observedscenarios to steadilyas sessions increase proceeded. across the five Facing a new situation training sessions, as illustrated. One likely reason for this observation is the gradual addi- tionfor of the more first challenging time creates driving a scenarios new potential as sessions for proceeded. young driver Facing a error, new situation and this may have resulted forin the additional first time creates negative a new potentialdemerits for from young driverthe huma error,n and evaluator, this may have even resulted as the student continued into additional master negative core driving demerits from skills the humanfrom evaluator,previous even exposure. as the student Likewise, continued toas training progressed master core driving skills from previous exposure. Likewise, as training progressed across trainingacross modules training and modules instructional and scenarios, instructional the evaluator sc likelyenarios, exhibited the increased evaluator ex- likely exhibited in- pectationscreased as expectations students began toas master students baseline began driving to skills master and perhaps baseline felt an driving increased skills and perhaps felt responsibilityan increased to steadily responsibility reinforce more to advancedsteadily drivingreinforce behaviors. more advanced driving behaviors.

FigureFigure 14. Evaluation14. Evaluation summary. summary. DIFF, difference DIFF, score; difference POS, positive; score; NEG, POS, negative. positive; NEG, negative. 3.3. Self-Report Measures 3.3.As Self-Report a tertiary mechanism Measures for the holistic driver performance evaluation, quantitative and qualitativeAs a tertiary measures mechanism were supplemented for the with holistic a series driver of self-report performance metrics. Anevaluation, quantitative overall data comparison found that post-SBQ scores were improved (with significance) as comparedand qualitative to the pre-SBQ measures scores, (t(132) were = +3.64,supplementedp = 5.19 × 10− with9). Figure a series 15 plots of the self-report results metrics. An over- (preall vs. data post) comparison of the session-based found questionnaire that post-SBQ (SBQ) cohort sc averageores were across eachimproved of the five (with significance) as instructionalcompared sessions. to the When pre-SBQ individual scores, sessions (t(132) were = compared, +3.64, p the = 5.19e post-SBQ−9). scoresFigure were 15 plots the results (pre higher (with significance) than pre-SBQ scores for three of the five instructional sessions: Sessionvs. post) 1, t(132) of the = +8.89, session-basedp =3.18 × 10− 11questionnaire; Session 2, t(132) (S = +5.00,BQ) cohortp = 6.02 ×average10−5; and across each of the five Sessioninstructional 5, t(132) = +7.92,sessions.p = 2.24 When× 10−7 .individual The lack of observed sessions improvement were compared, for the other the post-SBQ scores twowere individual higher sessions (with (i.e., significance) Sessions 3 and than 4) could pre-SBQ indicate scores that there for was three not sufficient of the five instructional ses- correlation between the content of the questionnaires and the embedded training content -11 −5 withinsions: the Session simulator 1, exercises t(132) designed= +8.89, explicitly p =3.18e as the; Session focal point 2, for t(132) those sessions.= +5.00, p = 6.02e ; and Session 5, t(132) = +7.92, p = 2.24e−7. The lack of observed improvement for the other two individual sessions (i.e., Sessions 3 and 4) could indicate that there was not sufficient correlation be- tween the content of the questionnaires and the embedded training content within the simulator exercises designed explicitly as the focal point for those sessions.

Safety 2021, 7, x FOR PEER REVIEW 16 of 24

SafetySafety 20212021, 7,, x7, FOR 24 PEER REVIEW 16 of 24 16 of 24

Figure 15. SBQ (pre/post).

The Motion Sickness Assessment Questionnaire (MSAQ) was issued post-simulator in Sessions 1, 3, and 5. An overall evaluation found that, as expected, simulator sickness FigurewasFigure 15.not 15.SBQ identified SBQ (pre/post). (pre/post). to be a primary concern for these simulator experiments that were de- signed and deployed for this young driving cohort. Figure 16 displays the cohort mean TheThe Motion Motion Sickness Sickness Assessment Assessment Questionnaire Questio (MSAQ)nnaire was (MSAQ) issued post-simulator was issued post-simulator inscore Sessions (reported 1, 3, and as 5. Ana percentage overall evaluation of the foundoverall that, MSAQ as expected, score: simulator 144) and sickness the number of partic- wasipantsin Sessions not identifiedwho 1,reported 3, to and be a 5.cumulative primary An overall concern scores evaluation for theseof greater simulator found than that, experiments 14 as (15) expected, of that the were simulatortotal MSAQ sickness scale designed(alsowas notreported and identified deployed as a topercentage for be this a primary young of driving the concern total cohort. forN = these Figure132 cohort).simulator 16 displays Standard experiments the cohort deviations that were for thede- meanthreesigned score sessions and (reported deployed (1, as3, aand percentagefor 5) this were young of significantly the overalldriving MSAQ co largehort. score: only Figure 144) relative and 16 thedisplays to number the reportedthe cohort means; mean ofscore participants (reported who reportedas a percentage cumulative of scores the overall of greater MSAQ than 14 score: (15) of 144) the total and MSAQ the number of partic- scalethey (also were reported 2.56%, as 2.55%, a percentage and of4.44%, the total respectively, N = 132 cohort). which Standard is an deviations indicator for that there was a thecomparativelyipants three sessionswho reported (1, wide 3, and dispersion 5)cumulative were significantly of scoresresults. large of To onlygreater this relative point, than to note the14 reported(15) that of Figure means;the total 16 alsoMSAQ displays scale theya(also maximum were reported 2.56%, (MAX) 2.55%, as a andMSAQpercentage 4.44%, score respectively, of across the total the which cohortN = is 132 an for indicator cohort). each thatof Standard the there three was deviationssessions a in forwhich the comparativelythisthree metric sessions widewas (1, dispersioncollected 3, and of5)and results.were analyzed. significantly To this point,While note largethe that extremes only Figure relative 16 arealso noteworthy displays to the areported outliers, means; the maximum (MAX) MSAQ score across the cohort for each of the three sessions in which this overallthey were trends 2.56%, of the 2.55%, data and are 4.44%,favorable. respectively, The low MSAQwhich isscores an indicator across the that cohort there are was at- a metriccomparatively was collected wide and analyzed. dispersion While of the results. extremes To are this noteworthy point, note outliers, that the Figure overall 16 also displays trendstributed of the to data the areyouthful favorable. age The demographic low MSAQ scores (14–17 across), who the cohortare rarely are attributed susceptible to anything tomorea themaximum youthful than “mild” age(MAX) demographic simulator MSAQ (14–17),score maladaptation. across who are the rarely cohort The susceptible experimental for each to of anything the design three more (seesessions Table in 2) which suc- thancessfullythis “mild” metric imparted simulator was collected maladaptation. simulation and analyzed.best-practices The experimental While and design the intentionally extremes (see Table 2are) successfullybalanced noteworthy the 2outliers, h training the impartedsessionsoverall simulationtrends with breaks, of best-practicesthe datatimely are transitions and favorable. intentionally betw The balanced eenlow driver/passenger MSAQ the 2 hscores training across sessions activities, the cohort video aretrain- at- withtributed breaks, to timely the transitionsyouthful betweenage demographic driver/passenger (14–17 activities,), who video are rarely training, susceptible survey to anything activities,ing, survey andperhaps activities, most and critically—simulator perhaps most experimentcritically—simulator durations that experiment fell below a durations that 15fellmore min below maximum. than a “mild”15 min simulatormaximum. maladaptation. The experimental design (see Table 2) suc- cessfully imparted simulation best-practices and intentionally balanced the 2 h training sessions with breaks, timely transitions between driver/passenger activities, video train- ing, survey activities, and perhaps most critically—simulator experiment durations that fell below a 15 min maximum.

FigureFigure 16. 16.Motion Motion Sickness Sickness Assessment Assessment Questionnaire Questionnaire (MSAQ) summary. (MSAQ) summary.

Finally,Finally, exit exit surveys surveys were issued were toissued both the to teenboth trainees the teen and trainees their parents, and and their these parents, and these data further yielded valuable information about the training program. For the teen trainees, twodata questions further were yielded administered valuable on information a 1 to 10 Likert abou scale.t Thethe firsttraining asked, program. “Overall, rate For the teen train- yourees,Figure experience two 16. questions Motion in the Sickness training were program”,administeredAssessment and Questionnaire the on second a 1 to asked, 10 (MSAQ) Likert “Overall, scale.summary. rate theThe amount first asked, “Overall, ofrate time your that was experience spent on the in driving the training simulator”. progra Refer tom”, Figure and 17 the, which second co-analyzes asked, both “Overall, rate the series.amount OverallFinally, of time “experience” exit that surveys was ratings werespent on issued theon simulatorthe to driving both was th observedesimulator”. teen trainees to be highlyRefer and favorable,to their Figure parents, 17, which and these co- analyzesdata further both yielded series. valuableOverall “experience”information abou ratingst the on training the simulator program. was For observed the teen train-to be highlyees, two favorable, questions with were approximately administered 110 on aof 1 the to 10N =Likert 132 cohort scale. (83%)The first rating asked, their “Overall, overall experiencerate your experience as being an in 8/10 the trainingor better. progra In contrast,m”, and “time the onsecond simula asked,tor” ratings“Overall, were rate con- the siderablyamount of lower, time withthat wasa weighted spent on cohort the driving average simulator”. of just 5.7/10. Refer This to was Figure likely 17, because which notco- enoughanalyzes time both spent series. was Overall spent allowing “experience” teens toratings drive onthe the simulator simulator (performing was observed simulator- to be highly favorable, with approximately 110 of the N = 132 cohort (83%) rating their overall experience as being an 8/10 or better. In contrast, “time on simulator” ratings were con- siderably lower, with a weighted cohort average of just 5.7/10. This was likely because not enough time spent was spent allowing teens to drive the simulator (performing simulator-

Safety 2021, 7, 24 17 of 24 Safety 2021, 7, x FOR PEER REVIEW 17 of 24 Safety 2021, 7, x FOR PEER REVIEW 17 of 24

with approximately 110 of the N = 132 cohort (83%) rating their overall experience as beingexercises), an 8/10 as or opposed better. In contrast, to session “time time on simulator” spent on ratingsother wereactivities considerably (e.g., watching lower, videos, tak- exercises), as opposed to session time spent on other activities (e.g., watching videos, tak- withing asurveys, weighted watching cohort average partner of just drive, 5.7/10. experime This was likelynt debrief). because Likewise, not enough teens time were afforded ing surveys, watching partner drive, experiment debrief). Likewise, teens were afforded spentthe opportunity was spent allowing to answer teens to driveopen-ended the simulator questi (performingons related simulator-exercises), to their training as experiences; opposedthe opportunity to session time to spentanswer on otheropen-ended activities (e.g.,questi watchingons related videos, to taking their surveys, training experiences; three questions were assessed: “What did you learn about safe driving?” (375 total com- watchingthree questions partner drive, were experiment assessed: debrief). “What Likewise, did you teens learn were affordedabout safe the opportunity driving?” (375 total com- ments received), “What did you like most about your simulator training program?” (330 toments answer received open-ended), “What questions did related you like to their most training about experiences; your simulator three questions training were program?” (330 assessed:total comments “What did received you learn), and about “What safe did driving?” you dislike (375 total about comments the receiveddriving), “Whatsimulation training?” total comments received), and “What did you dislike about the driving simulation training?” did(112 you total like comments most about received your simulator). The three training most program?” common (330 responses total comments (reported received), as percentages) and(112 “What total didcomments you dislike received about). theThe driving three simulationmost common training?” responses (112 total (reported comments as percentages) for each question are summarized in Figure 18. Most prominently, trainees mentioned receivedfor each). The question three most are common summarized responses in (reported Figure as18. percentages) Most prominently, for each question trainees mentioned learning about hazard management and the importance of maintaining an appropriate arelearning summarized about in Figurehazard 18. Mostmanagement prominently, and trainees the mentionedimportance learning of maintaining about hazard an appropriate managementfollowing distance and the importance and speed of maintaining (relative to an cond appropriateitions). following Similarly, distance trainees and reported most following distance and speed (relative to conditions). Similarly, trainees reported most speedliking (relative the realism to conditions). of the Similarly,simulator, trainees the informative reported most nature liking the of realism the training of the modules, and simulator,liking the the realism informative of naturethe simulator, of the training the modules,informative and the nature hands-on of the engagement training modules, and the hands-on engagement of driving a simulator for training. By way of criticism, trainees ofthe driving hands-on a simulator engagement for training. of driving By way of a criticism,simulator trainees for training. suggested By that way the of lane criticism, trainees suggested that the lane markers and signs were difficult to see; despite acclimation, the markerssuggested and signs that werethe difficultlane markers to see; despite and signs acclimation, were thedifficult steering to wheel see; and despite pedals acclimation, the weresteering difficult wheel to adjust and to pedals (i.e., a commonwere difficult complaint to ofadjust all driving to (i.e., simulators); a common and complaint some of all driv- steering wheel and pedals were difficult to adjust to (i.e., a common complaint of all driv- suggesteding simulators); that the graphics and some were suggested not realistic. that the graphics were not realistic. ing simulators); and some suggested that the graphics were not realistic.

Figure 17. Teen Assessment Experience/Time on Driving Simulator. FigureFigure 17. 17.Teen Teen Assessment Assessment Experience/Time Experience/T on Drivingime on Simulator. Driving Simulator.

Figure 18. Teen Assessment Open-ended Summary. FigureFigure 18. 18.Teen Teen Assessment Assessment Open-ended Open-ended Summary. Summary. In a similar manner, parents were issued a series of questions when their teen com- InIn a similara similar manner, manner, parents parents were issued were a seriesissued of a questions series of when questions their teen when com- their teen com- pletedpleted the the training training program. program. Overall Overall ratings wereratings extremely were positive,extremely as parents positive, nearly- as parents nearly- pleted the training program. Overall ratings were extremely positive, as parents nearly- unanimouslyunanimously suggested suggested that (a) that simulator (a) simulator training shouldtraining constitute should a mandatoryconstitute first a mandatory first stepunanimously in driver training, suggested (b) they wouldthat (a) recommend simulator the trai coursening to othershould parents, constitu and (c)te thea mandatory first step in driver training, (b) they would recommend the course to other parents, and (c) the programstep in successfullydriver training, instructed (b) theirthey child would basic recomme driving skills.nd the Parents course were to also other afforded parents, and (c) the program successfully instructed their child basic driving skills. Parents were also afforded program successfully instructed their child basic driving skills. Parents were also afforded the opportunity to answer open-ended questions related to their own perceptions of their the opportunity to answer open-ended questions related to their own perceptions of their teen’s training experiences; three questions were assessed (each, on a 3-point Likert scale): teen’s training experiences; three questions were assessed (each, on a 3-point Likert scale): “As a result of their training, did you see your child paying more attention to road signs, “As a result of their training, did you see your child paying more attention to road signs, road hazards, and general road compliance?” (i.e., translation of training experiences from road hazards, and general road compliance?” (i.e., translation of training experiences from simulator to real-world), “How much do you think your child enjoyed being part of the simulator to real-world), “How much do you think your child enjoyed being part of the SBT program?”, and “Overall, how would you rate this program?” The Likert-style results SBT program?”, and “Overall, how would you rate this program?” The Likert-style results (reported as response percentages) for each question are summarized (favorably) in Fig- (reported as response percentages) for each question are summarized (favorably) in Fig- ure 19. As with their teens, parents were afforded the opportunity to answer open-ended ure 19. As with their teens, parents were afforded the opportunity to answer open-ended questions (positive and negative) related to their child’s training experiences (126 total questions (positive and negative) related to their child’s training experiences (126 total

Safety 2021, 7, 24 18 of 24

the opportunity to answer open-ended questions related to their own perceptions of their teen’s training experiences; three questions were assessed (each, on a 3-point Likert scale): “As a result of their training, did you see your child paying more attention to road signs, Safety 2021, 7, x FOR PEER REVIEWroad hazards, and general road compliance?” (i.e., translation of training experiences from 18 of 24 Safety 2021, 7, x FOR PEER REVIEW 18 of 24 simulator to real-world), “How much do you think your child enjoyed being part of the SBT program?”, and “Overall, how would you rate this program?” The Likert-style results (re- ported as response percentages) for each question are summarized (favorably) in Figure 19. Ascomments with their received teens, parents). For were each afforded category, the opportunity the three tomost answer common open-ended responses questions (reported as per- (positivecomments and received negative)). For related each to category, their child’s the training three experiencesmost common (126 totalresponses comments (reported as per- centages) are summarized in Figure 20. Parents opined that the primary benefits were that receivedcentages)). For are each summarized category, the threein Figure most common20. Parents responses opined (reported that the as primary percentages) benefits were that teens enjoyed driving the simulator and that the instructors were of high quality; the third areteens summarized enjoyed indriving Figure 20the. Parentssimulator opined and that that the the primary instructors benefits were were of that high teens quality; the third enjoyedhighest driving frequency the simulator response and (much that the less instructors significant were) was of highthat quality;teens seemed the third to have reduced highest frequency response (much less significant) was that teens seemed to have reduced highestdriving frequency anxiety response (while driving) (much less significantas a benefit) was of that their teens simulator seemed to exposure. have reduced However, parents driving anxiety (while driving) as a benefit of their simulator exposure. However, parents drivingalso noted anxiety that (while they driving) would as ahave benefit liked of their to simulatorhave seen exposure. more However,simulator parents time and increased alsoalso noted noted that that they they would would have likedhave to liked have seento ha moreve seen simulator more time simulator and increased time and increased graphics realism on the simulator, and many parents also noted the desire to attempt the graphicsgraphics realism realism on the on simulator, the simulator, and many and parents many also parents noted thealso desire noted to attempt the desire the to attempt the simulatorsimulator themselves. themselves. The third The highestthird highest frequency frequency response (much response less significant (much) less was significant that ) was that simulator themselves. The third highest frequency response (much less significant) was that parentsparents wished wished for higherfor higher visual visual realism realism for the simulator. for the simulator. parents wished for higher visual realism for the simulator.

Figure 19. Parent (direct) Assessment Likert-style feedback. FigureFigure 19. 19.Parent Parent (direct) (direct) Assessment Assessment Likert-style Likert-style feedback. feedback.

Figure 20. Parent Assessment Open-ended Summary. FigureFigure 20. 20.Parent Parent Assessment Assessment Open-ended Open-ended Summary. Summary. 4.4. Broader Broader Impacts Impacts of Our of Methodologyour Methodology and Implementation and Implementation 4. Broader Impacts of our Methodology and Implementation TheThe outcomes outcomes of the of current the current research research aimed to demonstrate aimed to demonstrate the effectiveness the of physics- effectiveness of phys- The outcomes of the current research aimed to demonstrate the effectiveness of phys- basedics-based M&S andM&S its potentialand its topotential improve roadwayto improve safety roadway and public safety health withinand public North health within Americanics-based young M&S driver and training. its potential Such a demonstration to improve could roadway result insafety a more and widespread public health within North American young driver training. Such a demonstration could result in a more wide- deploymentNorth American of these advanced young driver technologies training. and augmentation Such a demonstration of similar training could programs result in a more wide- acrossspread the deployment nation. Agencies of that these might advanced be interested technologi in such technologyes and augmentation would include high of similar training spread deployment of these advanced technologies and augmentation of similar training programs across the nation. Agencies that might be interested in such technology would programs across the nation. Agencies that might be interested in such technology would include high schools, driver training agencies, and law enforcement. The successful im- include high schools, driver training agencies, and law enforcement. The successful im- plementation of SBT frameworks could, over time, result in improved driving practices plementation of SBT frameworks could, over time, result in improved driving practices for the teenage demographic, where a disproportionate number of negative driving out- for the teenage demographic, where a disproportionate number of negative driving out- comes (e.g., crashes, injuries) continue to occur [1,2]. Ultimately, this could lead to various comes (e.g., crashes, injuries) continue to occur [1,2]. Ultimately, this could lead to various other benefits to society, including an increased level of safety on the roadways for all other benefits to society, including an increased level of safety on the roadways for all drivers, resulting in a reduction in accidents, injuries, and loss of life. In turn, this could drivers, resulting in a reduction in accidents, injuries, and loss of life. In turn, this could result in reduced insurance rates for all drivers, particularly those in the highest-risk pool result in reduced insurance rates for all drivers, particularly those in the highest-risk pool (i.e., young males). (i.e., young males). Likewise, by leveraging the groundwork established in this paper, similar training Likewise, by leveraging the groundwork established in this paper, similar training programs could be developed for other vehicle types for large equipment civilian vehicle programs could be developed for other vehicle types for large equipment civilian vehicle simulator training, including commercial truck driving. Recent research [47–49] indicates simulator training, including commercial truck driving. Recent research [47–49] indicates that skills received from exposure within a large-vehicle simulator (e.g., trucks, aviation) that skills received from exposure within a large-vehicle simulator (e.g., trucks, aviation) can directly transfer to skills required to operate an actual truck/aircraft—and are learned can directly transfer to skills required to operate an actual truck/aircraft—and are learned as quickly (and sometimes faster) than is the case with live/physical training. Within this as quickly (and sometimes faster) than is the case with live/physical training. Within this

Safety 2021, 7, 24 19 of 24

schools, driver training agencies, and law enforcement. The successful implementation of SBT frameworks could, over time, result in improved driving practices for the teenage demographic, where a disproportionate number of negative driving outcomes (e.g., crashes, injuries) continue to occur [1,2]. Ultimately, this could lead to various other benefits to society, including an increased level of safety on the roadways for all drivers, resulting in a reduction in accidents, injuries, and loss of life. In turn, this could result in reduced insurance rates for all drivers, particularly those in the highest-risk pool (i.e., young males). Likewise, by leveraging the groundwork established in this paper, similar training programs could be developed for other vehicle types for large equipment civilian vehicle simulator training, including commercial truck driving. Recent research [47–49] indicates that skills received from exposure within a large-vehicle simulator (e.g., trucks, aviation) can directly transfer to skills required to operate an actual truck/aircraft—and are learned as quickly (and sometimes faster) than is the case with live/physical training. Within this context, simulators—when employed in addition to (but not as a replacement for) hands-on training—can be particularly useful for instruction of gear-shifting and linear and angular maneuvers of increasing complexity. Likewise, in the military, it remains an ongoing challenge to determine efficient and effective mechanisms for simulator-based training (SBT). It has been emphasized that what matters most for maximizing SBT ef- fectiveness is an environment that fosters psychological fidelity (e.g., cognitive skills) by applying established learning principles (e.g., feedback, measurement, guided practice, and scenario design) [50,51]. Too often, the evaluation of training outcomes focuses on how the simulator itself functions, instead of on how humans perform and behave within the simulator [49,51]. The design and development of a training framework that demonstrates sufficient physical fidelity, while focusing on the analysis of human factors and objective metrics for performance within the training environment, remains a point of emphasis for the current work.

5. Conclusions and Future Work Traffic accidents remain a primary safety concern and are a leading cause of death for young, inexperienced, and newly licensed drivers [1–4]. For young driver education, it therefore remains an utmost public health priority to determine how technology can be better employed to increase the overall quality and effectiveness of current driver training methods. Supplemental training afforded by realistic physics-based modeling and hands- on simulation provides much-needed “seat time”, that is, practice and exposure within an engaging and authentic setting that promotes experiential learning. Concrete behavioral and performance feedback can be implemented to enhance and promote positive training, which could help to reduce negative driving outcomes (e.g., crashes, tickets, property damage, and fatalities) and thereby improve young driver safety over the long term. In this paper, a detailed case study was described that examined the specification, development, and deployment of a “blueprint” for a simulation-centric training framework intended to increase driver training safety in North America (see Figure1). Points of focus in this paper have included an overview of the numerous advantages of expanding the use of advanced simulation and gaming elements in current and future driver training; consideration toward simulator fidelity (e.g., including motion capabilities and field- of-view); and simulator-based driving environments to enhance young driver training (e.g., including closed-course, residential, and challenge scenarios to accompany baseline environment acclimation). A multi-measure (holistic) assessment of simulated driver performance was developed and demonstrated, including quantitative, qualitative, and self-report rubrics, from which the following trends were observed: • Cohort score report total scores exhibited a general upward trend across the entire program, and these trends were moderately enhanced with the presence of motion cues (quantitative). Safety 2021, 7, 24 20 of 24

• Both negative and positive feedback evaluator comments increased as the training modules proceeded. The overarching hypothesis is that the increase in positive feedback was counteracted by the driving session evaluator’s tendency for increased “bias” in the form of growing expectations as the program proceeded (qualitative). • Post-SBQ scores were higher than pre-SBQ scores (due to module-specific knowledge gained due to simulator exposure), as expected. Motion sickness (by way of MSAQ) was found to be insignificant, which can be credited to the experimental design (see Table2) and to the young demographics of this specific cohort (self-report). • Teens rated their experience on the simulator favorably but suggested that more of the overall session time should have been spent on the simulator itself rather than on related activities associated with the training program. They recognized the simulator as an effective way to learn but disliked some of the visual elements and control aspects of the simulator (self-report). • Almost all parents agreed that the simulator program should be a first step toward future driver training. Furthermore, per their own observations of subsequent teen driving performance, favorable feedback was issued for specific driving skillsets gained, including improved recognition to road signs and road hazards (self-report). This paper concludes with numerous suggestions and recommendations that would enhance future implementations of simulator training to further improve teen driver safety.

5.1. Improved Metrics for Quantifying Simulator Performance The simulator score report served as a primary measure for quantifying driver perfor- mance and behavior during the driving simulator exercises. This robustness of this score report toward comprehensively quantifying simulator performance stands to be improved from various perspectives. Future implementations would attempt to quantify driver compliance to lateral lane position, primarily during turn and lane-change maneuvers, as well as measuring ability to maintain lane-centric behavior during hazard events and multitasking; would devise enhanced scoring metrics that are specific to each of the three course environments to monitor overall progress and compliance across the five instruc- tional sessions; and would incorporate more explicit metrics for each of the specialized hazard challenges.

5.2. Improvements to Experimental Training Environment As a primary segment of teen (and parent) surveys, attempts were made to better understand elements of the current simulator framework that demand improvement in future implementations. Many teens suggested that signs and lane markings were difficult to see; although every attempt was made to ensure clarity and visibility of simulation environment objects, improvements are still required. Many teens suggested that the simulator (steering/pedals) were difficult to operate; this remains a common complaint across all driving simulators—particularly so with the feel of the brake pedal [52]. It is never feasible to make a simulator “feel” like everyone’s own vehicle, both physically (e.g., the resistive force on the brake pedal) and visually (e.g., the reduced sensation of longitudinal motion—or vection [53]—while braking within a virtual environment). Nonetheless, ongoing improvements including additional acclimation and device “tuning” to each driver should be considered. Finally, various teens (and parents) suggested the need for improved simulator graphics/realism toward a modern-day “gaming” standard.

5.3. Impairment Training Modules The current implementation of SBT content was designed explicitly to promote posi- tive training (i.e., enhancement of knowledge, skills, behaviors [24]) to improve roadway safety. This was attempted by explicitly addressing the top five (documented) causes for teen driving accidents and fatalities in New York State [41], with “impairment” being one of those five. It is well known that teen drivers frequently speak and text-message on their cell phones while driving [54]. Core simulator exercises are in development that Safety 2021, 7, 24 21 of 24

compare graphical representations of the teen’s driving to provide concrete performance and behavioral feedback. Related studies have investigated internal distractions (mind- wandering) while driving [33,55], which is well-suited to a simulator-based deployment. Finally, according to data published over the last decade, nearly one-fifth of teenagers killed in automobile accidents were under the influence (i.e., illicit or prescription drugs, alcohol), and young drivers are more 17 times more likely to be involved in a fatal accident when they have a blood alcohol concentration of 0.08% (i.e., the legal limit in New York State) than when they have not been drinking [56]. Simulator-based exercises have been designed and installed [42] that enable teen drivers to perform “routine” driving tasks with/without a simulated state of driver impairment that similarly includes concrete driver performance feedback.

5.4. Driving Exercises that Promote Reverse Egress One major deficiency in the present deployment is its current inability to perform a reverse maneuver within the simulator. Particularly within a driver education context, this feature could be extremely useful and effective toward improving young driver safety. For example, reverse maneuvers within a crowded parking area, 2-point turns (e.g., on a one-way street), 3-point turns (e.g., into a driveway), and the classic parallel-parking maneuver all require an accurate simulation model for reverse vehicle motion. All of these maneuvers are encountered during every day driving, and many are requirements on a standard New York State road test examination [57].

5.5. Driving Exercises for Driver Training on Rural Roads Previous research [58] has verified that fatality risk is elevated more so on rural roads than urban roads for all drivers and particularly teen drivers [59]. Per miles traveled, there have been estimated to be 30–40% more crashes and deaths in rural areas than in urban areas [59]. Likely causes of these untoward events are attributed to: remote areas with which young drivers are less familiar; low-lighting conditions during nighttime driving; increased propensity for elevated speeds on roads that are less traveled [60]; and as a result of these factors—an elevated likelihood for induced cognitive distraction (e.g., mindlessness, task-unrelated thought [33,55]) during the driving task. As such, these are all critical factors for which to provide exposure—and subsequently examine human performance behaviors—in future teen driving SBT research.

5.6. Longitudinal Data to Measure Efficacy of SBT To determine the long-term efficacy of employing simulation for teen drivers on a broader scale, it would be beneficial to observe the impact of such training exposure upon lifetime driving statistics. In New York State, the MV-15 report [61] provides vital information regarding traffic accidents, moving violations, and other negative driving outcomes of program participants. In future implementations of similar programs, it would be useful to obtain consent/assent to collect/analyze/compare reports culled from participants enrolled in simulation-based training programs to similar data culled from state and national averages. If favorable, such empirical data—presently unavailable in the literature—could influence future policy for future safety standards related to SBT courseware.

Author Contributions: Conceptualization, K.E.L. and K.F.H.; methodology, K.F.H.; software, G.A.F. and K.F.H.; validation, K.F.H.; formal analysis, K.F.H., M.B., H.D. and N.H.; investigation, R.S.A.L., M.B., N.H., H.D., B.S., J.-A.D., R.C., I.D.J., A.A., J.D. and N.B.; data curation, M.B. and K.F.H.; writing— original draft preparation, K.F.H. and R.S.A.L.; writing—review and editing, R.S.A.L. and K.F.H.; visualization, K.F.H. and R.S.A.L.; supervision, K.F.H.; project administration, N.H., H.D., M.B. and K.F.H.; funding acquisition, K.E.L. and K.F.H. All authors have read and agreed to the published version of the manuscript. Safety 2021, 7, 24 22 of 24

Funding: This research was funded by the National Science Foundation (NSF), grant number 1229861, the Industry/University Cooperative Research Center (I/UCRC) for e-Design. This research effort served as a timely contribution to its “Visualization and Virtual Prototyping” track. Institutional Review Board Statement: The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of the University at Buffalo Internal Review Board (protocol code 030-431144 on 21 October 2015). Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: The data presented in this study are available on request from the corre- sponding author. The data are not publically available due to participant privacy and confidentiality concerns related to the terms of the study IRB. Acknowledgments: Toward this extensive, multi-year endeavor, the authors wish to gratefully acknowledge the financial and technical support provided by Moog, Inc. (East, Aurora, NY) that further enabled the novel application of M&S to improve STEM, training engagement, and young driver safety. Lastly, authors Hulme and Lim are grateful for the ongoing support of the University at Buffalo Stephen Still Institute for Sustainable Transportation and Logistics (SSISTL). Conflicts of Interest: The authors declare no conflict of interest.

References 1. Centers for Disease Control and Prevention (CDC). WISQARS (Web-Based Injury Statistics Query and Reporting System). US Department of Health and Human Services. September 2019. Available online: https://www.cdc.gov/injury/wisqars/index.html (accessed on 12 November 2020). 2. Insurance Institute for Highway Safety (IIHS). Fatality Facts 2017: Teenagers. Highway Loss Data Institute. December 2018. Available online: https://www.iihs.org/topics/fatality-statistics/detail/teenagers (accessed on 12 November 2020). 3. Mayhew, D.R.; Simpson, H.M.; Pak, A. Changes in collision rates among novice drivers during the first months of driving. Accid. Anal. Prev. 2003, 35, 683–691. [CrossRef] 4. McCartt, A.T.; Shabanova, V.I.; Leaf, W.A. Driving experiences, crashes, and teenage beginning drivers. Accid. Anal. Prev. 2003, 35, 311–320. [CrossRef] 5. Thomas, F.D., III; Blomberg, R.D.; Fisher, D.L. A Fresh Look at Driver Education in America; Report No. DOT HS 811 543; National Highway Traffic Safety Administration: Washington, DC, USA, 2012. 6. Vaa, T.; Høye, A.; Almqvist, R. Graduated driver licensing: Searching for the best composition of components. Lat. Am. J. Manag. Sustain. Dev. 2015, 2, 160. [CrossRef] 7. Strayer, D.L.; Cooper, J.M.; Goethe, R.M.; McCarty, M.M.; Getty, D.J.; Biondi, F. Visual and Cognitive Demands of Using In-Vehicle Information Systems; AAA Foundation for Traffic Safety: Washington, DC, USA, 2017. 8. Rechtin, M. Dumb and Dumber: America’s Driver Education is Failing Us All—Reference Mark. motortrend.com (Online Article), Published 20 June 2017. Available online: https://www.motortrend.com/news/dumb-dumber-americas-driver-education- failing-us-reference-mark/ (accessed on 26 November 2020). 9. Helman, S.V.; Fildes, W.; Oxley, B.; Fernández-Medina, J.; Weekley, K. Study on Driver Training, Testing and Medical Fitness— Final Report; European Commission, Directorate-General Mobility and Transport (DG MOVE): Brussels, Belgium, 2017; ISBN 978-92-79-66623-0. 10. Nalmpantis, D.; Naniopoulos, A.; Bekiaris, E.; Panou, M.; Gregersen, N.P.; Falkmer, T.; Baten, G.; Dols, J.F. “TRAINER” project: Pilot applications for the evaluation of new driver training technologies. In Traffic & Transport Psychology: Theory and Application; Underwood, G., Ed.; Elsevier: Oxford, UK, 2005; pp. 141–156, ISBN 9780080443799. 11. Imtiaz, A.; Mueller, J.; Stanley, L. Driving Behavior Differences among Early Licensed Teen, Novice Teen, and Experienced Drivers in Simulator and Real World Potential Hazards. In Proceedings of the IIE Annual Conference and Expo 2014, Montreal, QC, Canada, 31 May–3 June 2014. 12. Mirman, J.H.; Curry, A.E.; Winston, F.K.; Wang, W.; Elliott, M.R.; Schultheis, M.T.; Thiel, M.C.F.; Durbin, D.R. Effect of the teen driving plan on the driving performance of teenagers before licensure: A randomized clinical trial. JAMA Pediatrics 2014, 168, 764–771. [CrossRef][PubMed] 13. Milleville-Pennel, I.; Marquez, S. Comparison between elderly and young drivers’ performances on a driving simulator and self-assessment of their driving attitudes and mastery. Accid. Anal. Prev. 2020, 135, 105317. [CrossRef][PubMed] 14. Martín-delosReyes, L.M.; Jiménez-Mejías, E.; Martínez-Ruiz, V.; Moreno-Roldán, E.; Molina-Soberanes, D.; Lardelli-Claret, P. Efficacy of training with driving simulators in improving safety in young novice or learner drivers: A systematic review. Transp. Res. Part F Traffic Psychol. Behav. 2019, 62, 58–65. [CrossRef] 15. Campbell, B.T.; Borrup, K.; Derbyshire, M.; Rogers, S.; Lapidus, G. Efficacy of Driving Simulator Training for Novice Teen Drivers. Conn. Med. 2016, 80, 291–296. [PubMed] Safety 2021, 7, 24 23 of 24

16. De Groot, S.; De Winter, J.C.F.; Mulder, M.; Wieringa, P.A. Didactics in simulator based driver training: Current state of affairs and future potential. In Proceedings of the Driving Simulation Conference North America, Iowa City, IA, USA, 12–14 September 2007. 17. McDonald, C.C.; Goodwin, A.H.; Pradhan, A.K.; Romoser MR, E.; Williams, A.F. A Review of Hazard Anticipation Training Programs for Young Drivers. J. Adolesc. Health 2015, 57 (Suppl. 1), S15–S23. [CrossRef] 18. Fuller, R. Driver training and assessment: Implications of the task-difficulty homeostasis model. In Driver Behaviour and Training; Dorn, L., Ed.; Ashgate: Dublin, Ireland, 2007; Volume 3, pp. 337–348. 19. Hulme, K.F.; Kasprzak, E.; English, K.; Moore-Russo, D.; Lewis, K. Experiential Learning in Vehicle Dynamics Education via Motion Simulation and Interactive Gaming. Int. J. Comput. Games Technol. 2009, 2009, 952524. [CrossRef] 20. Hulme, K.F.; Lewis, K.E.; Kasprzak, E.M.; Russo, D.-M.; Singla, P.; Fuglewicz, D.P. Game-based Experiential Learning in Dynamics Education Using Motion Simulation. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Orlando, FL, USA, 29 November–2 December 2010. 21. Wickens, C.; Toplak, M.; Wiesenthal, D. Cognitive failures as predictors of driving errors, lapses, and violations. Accid. Anal. Prev. 2008, 40, 1223–1233. [CrossRef][PubMed] 22. Chiu-Shui, C.; Chien-Hui, W. How Real is the Sense of Presence in a Virtual Environment?: Applying Protocol Analysis for Data Collection. In Proceedings of the CAADRIA 2005: 10th International Conference on Computer Aided Design Research in Asia, New Delhi, India, 28–30 April 2005. 23. Berkman, M.I.; Akan, E. Presence and Immersion in Virtual Reality. In Encyclopedia of Computer Graphics and Games; Lee, N., Ed.; Springer: Berlin/Heidelberg, Germany, 2019. [CrossRef] 24. Groeger, J.A.; Banks, A.P. Anticipating the content and circumstances of skill transfer: Unrealistic expectations of driver training and graduated licensing? Ergonomics 2007, 50, 1250–1263. [CrossRef][PubMed] 25. Kaptein, N.; Theeuwes, J.; van der Horst, R. Driving Simulator Validity: Some Considerations. Transp. Res. Rec. 1996, 1550, 30–36. [CrossRef] 26. Godley, S.T.; Triggs, T.J.; Fildes, B.N. Driving simulator validation for speed research. Accid. Anal. Prev. 2002, 34, 589–600. [CrossRef] 27. Bella, F. Driving simulator for speed research on two-lane rural roads. Accid. Anal. Prev. 2008, 40, 1078–1087. [CrossRef] 28. Stoner, H.A.; Fisher, D.; Mollenhauer, M., Jr. Simulator and scenario factors influencing simulator sickness. In Handbook of Driving Simulation for Engineering, Medicine, and Psychology; CRC Press: Boca Raton, FL, USA, 2011. 29. Pettijohn, K.; Pistone, D.; Warner, A.; Roush, G.; Biggs, A. Postural Instability and Seasickness in a Motion-Based Shooting Simulation. Aerosp. Med. Hum. Perform. 2020, 91, 703–709. [CrossRef] 30. Keshavarz, B.; Ramkhalawansingh, R.; Haycock, B.; Shahab, S.; Campos, J. Comparing simulator sickness in younger and older adults during simulated driving under different multisensory conditions. Transp. Res. Part F Traffic Psychol. Behav. 2018, 54, 47–62. [CrossRef] 31. Gee, J.P. What Games Have to Teach Us about Learning and Literacy; Palgrave Macmillan: New York, NY, USA, 2007. 32. Lee, J.; Hammer, J. Gamification in Education: What, How, Why Bother? Acad. Exch. Q. 2011, 15, 146. 33. Hulme, K.F.; Androutselis, T.; Eker, U.; Anastasopoulos, P. A Game-based Modeling and Simulation Environment to Examine the Dangers of Task-Unrelated Thought While Driving. In Proceedings of the MODSIM World Conference, Virginia Beach, VA, USA, 25–27 April 2016. 34. Milliken, W.F.; Milliken, D.L. Race Car Vehicle Dynamics; Society of Automotive Engineers: Warrendale, PA, USA, 1995. 35. Rajamani, R. Vehicle Dynamics and Control; Springer: New York, NY, USA, 2006; ISBN 0-387-26396-9. 36. Raghuwanshi, V.; Salunke, S.; Hou, Y.; Hulme, K.F. Development of a Microscopic Artificially Intelligent Traffic Model for Simulation. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Orlando, FL, USA, 1–4 December 2014. 37. Romano, R. Non-linear Optimal Tilt Coordination for Washout Algorithms. In Proceedings of the AIAA Modeling and Simulation Technologies Conference and Exhibit, AIAA 2003, Austin, TX, USA, 11–14 August 2003; p. 5681. [CrossRef] 38. Bowles, R.L.; Parrish, R.V.; Dieudonne, J.E. Coordinated Adaptive Washout for Motion Simulators. J. Aircr. 1975, 12, 44–50. [CrossRef] 39. Sung-Hua, C.; Li-Chen, F. An optimal washout filter design for a motion platform with senseless and angular scaling maneuvers. In Proceedings of the 2010 American Control Conference, ACC 2010, Baltimore, MD, USA, 30 June–2 July 2010; pp. 4295–4300. [CrossRef] 40. Akutagawa, K.; Wakao, Y. Stabilization of Vehicle Dynamics by Tire Digital Control—Tire Disturbance Control Algorithm for an Electric Motor Drive System. World Electr. Veh. J. 2019, 10, 25. [CrossRef] 41. New York State Department of Health (NYSDOH). Car Safety—Teen Driving Safety, (Online Statistics Portal). 2020. Available online: https://www.health.ny.gov/prevention/injury_prevention/teens.htm (accessed on 16 November 2020). 42. Fabiano, G.A.; Hulme, K.; Linke, S.M.; Nelson-Tuttle, C.; Pariseau, M.E.; Gangloff, B.; Lewis, K.; Pelham, W.E.; Waschbusch, D.A.; Waxmonsky, J.; et al. The Supporting a Teen’s Effective Entry to the Roadway (STEER) Program: Feasibility and Preliminary Support for a Psychosocial Intervention for Teenage Drivers with ADHD. Cogn. Behav. Pract. 2011, 18, 267–280. [CrossRef] 43. Fabiano, G.; Schatz, N.K.; Hulme, K.F.; Morris, K.L.; Vujnovic, R.K.; Willoughby, M.T.; Hennessy, D.; Lewis, K.E.; Owens, J.; Pelham, W.E. Positive Bias in Teenage Drivers with ADHD within a Simulated Driving Task. J. Atten. Disord. 2015, 22, 1150–1157. [CrossRef] Safety 2021, 7, 24 24 of 24

44. Overton, R.A. Driving Simulator Transportation Safety: Proper Warm-up Testing Procedures, Distracted Rural Teens, and Gap Acceptance Intersection Sight Distance Design. Ph.D. Thesis, University of Tennessee, Knoxville, TN, USA, 2012. Available online: https://trace.tennessee.edu/utk_graddiss/1462 (accessed on 2 December 2020). 45. Gianaros, P.; Muth, E.; Mordkoff, J.; Levine, M.; Stern, R. A Questionnaire for the Assessment of the Multiple Dimensions of Motion Sickness. Aviat. Space Environ. Med. 2001, 72, 115–119. 46. De Winter, J.C.; de Groot, S.; Mulder, M.; Wieringa, P.A.; Dankelman, J.; Mulder, J.A. Relationships between driving simulator performance and driving test results. Ergonomics 2009, 52, 137–153. [CrossRef][PubMed] 47. Hirsch, P.; Bellavance, F.; Choukou, M.-A. Transfer of training in basic control skills from a truck simulator to a real truck. J. Transp. Res. Board 2017, 2637, 67–73. [CrossRef] 48. Clinewood. Are Driving Simulators Beneficial for Truck Driving Training? 2018. Available online: https://clinewood.com/2018 /02/19/driving-simulators-beneficial-truck-driving-training/ (accessed on 22 November 2020). 49. Salas, E.; Bowers, C.A.; Rhodenizer, L. It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. Int. J. Aviat. Psychol. 1998, 8, 197–208. [CrossRef][PubMed] 50. Straus, S.G.; Matthew, W.L.; Kathryn, C.; Rick, E.; Matthew, E.B.; Timothy, M.; Christopher, M.C.; Geoffrey, E.G.; Heather, S. Collective Simulation-Based Training in the U.S. Army: User Interface Fidelity, Costs, and Training Effectiveness. Santa Monica, CA: RAND Corporation. 2019. Available online: https://www.rand.org/pubs/research_reports/RR2250.html (accessed on 18 November 2020). 51. Hays, R.T.; Michael, J.S. Simulation Fidelity in Training System Design: Bridging the Gap between Reality and Training; Springer: New York, NY, USA, 1989. 52. Wu, Y.; Boyle, L.; McGehee, D.; Roe, C.; Ebe, K.; Foley, J. Modeling Types of Pedal Applications Using a Driving Simulator. Hum. Factors 2015, 57, 1276–1288. [CrossRef][PubMed] 53. Kemeny, A.; Panerai, F. Evaluating perception in driving simulation experiments. Trends Cogn. Sci. 2003, 7, 31–37. [CrossRef] 54. Delgado, M.K.; Wanner, K.; McDonald, C. Adolescent Cellphone Use While Driving: An Overview of the Literature and Promising Future Directions for Prevention. Media Commun. 2016, 4, 79. [CrossRef][PubMed] 55. Hulme, K.F.; Morris, K.L.; Anastasopoulos, P.; Fabiano, G.A.; Frank, M.; Houston, R. Multi-measure Assessment of Internal Distractions on Driver/Pilot Performance. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Orlando, FL, USA, 30 November–4 December 2015. 56. Centers for Disease Control and Prevention (CDC). Vital Signs, Teen Drinking and Driving. 2020. Available online: https: //www.cdc.gov/vitalsigns/teendrinkinganddriving/index.html (accessed on 13 October 2020). 57. New York State Department of Motor Vehicles (NYSDMV). Pre-Licensing Course Student’s Manual (Form MV-277.1 PDF Download). 2020. Available online: https://dmv.ny.gov/forms/mv277.1.pdf (accessed on 13 October 2020). 58. Senserrick, T.; Brown, T.; Quistberg, D.A.; Marshall, D.; Winston, F. Validation of Simulated Assessment of Teen Driver Speed Management on Rural Roads. Annual proceedings/Association for the Advancement of Automotive Medicine. Assoc. Adv. Automot. Med. 2007, 51, 525–536. 59. Insurance Institute for Highway Safety (IIHS). Fatality Facts 2018—Urban/Rural Comparison. 2018. Available online: https: //www.iihs.org/topics/fatality-statistics/detail/urban-rural-comparison (accessed on 2 November 2020). 60. Henning-Smith, C.; Kozhimannil, K.B. Rural-Urban Differences in Risk Factors for Motor Vehicle Fatalities. Health Equity 2018, 2, 260–263. [CrossRef][PubMed] 61. New York State Department of Motor Vehicles (NYSDMV). REQUEST FOR CERTIFIED DMV RECORDS (Form MV-15 PDF Download). 2019. Available online: https://dmv.ny.gov/forms/mv15.pdf (accessed on 2 November 2020).