applied sciences

Article The Effect of Training in Virtual Reality on the Precision of Hand Movements

Sergo Martirosov, Petr Hoˇrejší * , Pavel Kopeˇcek,Marek Bureš and Michal Šimon

Department of Industrial Engineering and Management, University of West Bohemia, Univerzitní 2732/8, 301 00 Plzeˇn,Czech Republic; [email protected] (S.M.); [email protected] (P.K.); [email protected] (M.B.); [email protected] (M.Š.) * Correspondence: [email protected]

Abstract: The main point of the work was to use virtual reality to discover its benefits on training, specifically on the precision of hand movements in specific settings, and then evaluate its effects both for virtual reality and the transfer of the results to the real world. A virtual reality simulation was created using the Unity3D game engine and real-world experimental material was also prepared. A total of 16 participants took part in the training, which lasted for approximately one month. Once the data was gathered from both the virtual reality and real-world tests, we carried out in-depth statistical analysis. The results suggest positive outcomes in most aspects in virtual reality training productivity, but only partial transfer of the training benefits to the real world scenario. The possible reasons for this are described in the work and suggestions are given to duplicate the study with different variables to try to achieve different results.



Keywords: virtual environments; training; manuals and instructions; controls and input devices;
 human error Citation: Martirosov, S.; Hoˇrejší,P.; Kopeˇcek,P.; Bureš, M.; Šimon, M. The Effect of Training in Virtual Reality on the Precision of Hand Movements. 1. Introduction Appl. Sci. 2021, 11, 8064. https:// VR technology is used in different areas of life for various purposes. VR is the most doi.org/10.3390/app11178064 immersive type of reality technology that creates an artificial environment to inhabit. This immersive, computer-generated environment blocks out sensory input from the outside Academic Editor: world and uses visual and auditory cues to make the virtual world seem quite real. It is Giuseppe Marannano immersive because of its simulated environment that manages to trick your subconscious mind to start treating this illusion as real. Received: 13 August 2021 The main point of this work is to perform an experiment in VR and RW and find Accepted: 27 August 2021 out if precision, accuracy and time factors change and, if so, by how much and in which Published: 31 August 2021 direction. Accuracy and precision are alike only in the fact that they both refer to the quality of measurement, but they are very different indicators of measurement. Accuracy Publisher’s Note: MDPI stays neutral is the degree of closeness to the true value, whereas precision is the degree to which an with regard to jurisdictional claims in instrument or process will repeat the same value. In other words, accuracy is the degree of published maps and institutional affil- veracity while precision is the degree of reproducibility [1]. iations. This work focuses on just one way of measuring the precision of hand movements using one specific type of simulation, but the types of training simulations for each job should definitely have their own variants and styles of simulation. Some examples of professions that require high hand-to-eye coordination and precision are given below. VR Copyright: © 2021 by the authors. training in these cases could drastically improve the skills required to perform the tasks, as Licensee MDPI, Basel, Switzerland. it would allow unlimited training in any circumstances. This article is an open access article VR technology has become very popular and is used in many different areas such distributed under the terms and as entertainment, military, healthcare, education, engineering, etc. VR is a simulated conditions of the Creative Commons Attribution (CC BY) license (https:// environment that is created with computer technology and presented to the user in such a creativecommons.org/licenses/by/ way that the user starts to feel as if they are in a real environment. A simulation is a model 4.0/). of the RW wherein the user has the ability to interact with the environment [2]. Simulations

Appl. Sci. 2021, 11, 8064. https://doi.org/10.3390/app11178064 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 8064 2 of 19

are helpful and useful as they provide a realistic context in which individuals can explore, experiment and see immediate results as they create models of their own or try theories on the modelled concept [3]. Depending on the number of senses simulated in VR, such as vision, hearing, touch, balance, even smell, the immersion level in the artificial world can vary. The technological advances of computer hardware and software have made it possible to incorporate 3D Virtual Reality in innovative applications for teaching, training, and learning [4–6]. It has been noted that there is a strong potential value of ergonomics for virtual environments and vice versa [7]. A virtual environment or virtual world is a computer-generated 3D representation of real or fictional environments. A user can interact in such an environment independently at the same pace as one would experience events in the real world [8]. VR can support realistic and immersive simulation and enables the transfer of skills taught in VR into real contexts, as well as the provision of multi-user, embodied, and interactive active learning [9]. VR is believed to be a promising tool for training and complex problem solving, where weighing multiple variables and situational decision making are required [9–12]. Not only VR but also augmented reality (AR), which is often used in combination with or supplemented by VR, has been studied intensively and proven to be a useful training tool. One study [13] suggests using virtual training with augmented reality in assembly tasks as it can greatly reduce the time taken to perform assembly tasks compared to classical methods. Having skilled and trained employees is very important, and a study [14] proposed brand new smart solutions for designing and presenting work instructions and showed a comparison of different types of virtual, augmented, and conventional assembly instructions. The results of the research show promising future potential. One of the uses of VR can be helping those with autism to lessen social anxiety and frustration and improve academic and social skills. Individuals with autism suffer from the impairment of social functions, which leads to difficulties in social interaction, communication, and emotion recognition [15,16]. The fact that VR systems can allow users to explore immersive 3D environments from any location at any time could have a profound impact on science education [17]. VR allows the exploration of hidden phenomena and distant locations and the manipulation of otherwise immutable objects [18]. For instance, VR gives medical students the possibility to explore delicate internal organs that would otherwise require cadaver dissection [19]. When working with VR applications, it is important to understand that eventually the problem of cyber sickness arises, and it is very helpful to know where it is coming from and how to combat it if possible. Visually induced motion sickness is a syndrome that occasionally occurs when physically stationary individuals view compelling visual representations of self-motion. It may also occur when detectable lags are present between head movements and recomputation and the presentation of the visual display in helmet- mounted displays [20]. When creating a simulation in VR, thinking ahead about all the potential factors that could negatively influence participants in any way is crucial, as cyber sickness can have negative effects on experimental results. Research suggests there are various relationships between different factors, such as VR training experience, engagement, immersion, etc. and the level of stress from training in VR [21]. Thus, having that and other factors influencing cyber sickness in mind, we tried to create a VR training simulation that was as simple and easy as possible. There is also evidence that the weight of the head-mounted display can cause discomfort in the users’ necks and shoulders, and if participants have to rotate their heads due to an activity in VR, their discomfort level grows [22]. In our training the participants had to simply stand in one place and look down at the table where the shape models were located. Rotation of the head was not required. We tried to minimize this and other factors that could cause VR sickness. As cyber sickness is very similar to motion sickness, there are many aspects that they share, thus making it possible to borrow some solutions from motion sickness and apply them to cyber sickness. Appl. Sci. 2021, 11, x FOR PEER REVIEW 3 of 19

down at the table where the shape models were located. Rotation of the head was not required. We tried to minimize this and other factors that could cause VR sickness. As cyber sickness is very similar to motion sickness, there are many aspects that they share, thus making it possible to borrow some solutions from motion sickness and apply them to cyber sickness. There is a wide variety of research being conducted in the field of VR on cyber sick- ness: cyber sickness in different types of moving images [23], comparison of one-screen and three-screen displays [24,25], relationship between age and sickness level [26], and a

Appl. Sci. 2021, 11, 8064 gender effect on cyber sickness [27,28]. Research on virtual environments3 of 19 (VEs) has pro- vided converging evidence that being placed in a VE with a head-mounted display (HMD) can lead to motion sickness (MS) [29] Previous studies suggest that a twenty-mi- nuteThere exposure is a wide to varietyVREs (virtual of research reality being environments) conducted in the fieldcan increase of VR on cyber sickness (CS) sickness:symptoms cyber in sickness over 60% in different of participant types of moving [30]. It images has been [23], comparisonnoted that of some one-screen users might exhibit andsymptoms three-screen of CS, displays both [ 24during,25], relationship and after between experien agecing and VR sickness [31,32]. level CS [26 is], anddifferent from MS ain gender that the effect users on cyber are stationary, sickness [27 ,28but]. Researchthey have on a virtual compelling environments sense (VEs)of motion has as the visual provided converging evidence that being placed in a VE with a head-mounted display (HMD)imagery can changes lead to motion [33,34]. sickness (MS) [29] Previous studies suggest that a twenty- minuteCurrently exposure to there VREs is (virtual a debate reality concerning environments) VR can technology increase cyber and sickness VR cyber (CS) sickness. Some symptomsargue that in the over problem 60% of participant of VR sickness [30]. It has is beeninherent noted thatin this some technology users might itself exhibit and that as long symptomsas there is of mismatch CS, both during between and after what experiencing is visually VR [perceived31,32]. CS is and different what from is physically MS sensed, infixing that the the users VR aredisplay stationary, properties but they will have not a compelling address the sense root of motion cause as[35]. the visual imagery changes [33,34]. Currently there is a debate concerning VR technology and VR cyber sickness. Some argue2. Materials that the problem and Methods of VR sickness is inherent in this technology itself and that as long as2.1. there Software is mismatch and Hardware between what Equipment is visually perceived and what is physically sensed, fixing the VR display properties will not address the root cause [35]. The VR simulation was created using the Unity3D game engine. The tests were run 2.on Materials a VR-ready and Methods Acer Predator notebook, with Intel Core i7-6700HQ @ 2.60 GHz, 16 gb Ram 2.1.and Software Nvidia and GeForce Hardware GTX Equipment 1060-6gb. The VR headset was a HTC Vive Pro, which has a resolutionThe VR simulationof 1440 × 1600 was created pixels using per eye the Unity3D(2880 × 1600 game pixels engine. combined), The tests were a run90 Hz refresh rate, on a VR-ready Acer Predator notebook, with Intel Core i7-6700HQ @ 2.60 GHz, 16 gb Ram andand Nvidia a 110 GeForcedegrees GTX field 1060-6gb. of view. The All VR of headset these wasparameters a HTC Vive allowed Pro, which participants has a to have as resolutioncomfortable of 1440 a VR× 1600 experience pixels per as eye possible. (2880 × 1600 pixels combined), a 90 Hz refresh rate, and a 110 degrees field of view. All of these parameters allowed participants to have as2.2. comfortable Design a VR experience as possible. 2.2. DesignThe environment in VR was a simple room with no extra objects in it, in order not to distractThe environmentparticipants in with VR was unnecessary a simple room visual withno noise extra (see objects Figure in it, in1). order Apart not from the room toitself, distract the participantsonly 3D models with unnecessary available visualin the noisescene (see were Figure a 1.051). Apartm high from table, the an object (line, roomcircle, itself, sinus the wave) only 3D of models reference available to indraw the scene on, a were model a 1.05 of m a high glue table, gun an or object caulking gun, de- (line, circle, sinus wave) of reference to draw on, a model of a glue gun or caulking gun, dependingpending on on the scene,scene, which which was was either either a one-handed a one-handed or two-handed or two-handed version. The version. The line, line,circle, circle, and and sinus sinus wave wave were were greengreen with with medium medium transparency transparency on them, on whereas them, whereas the the draw- drawinging was was done done with with opaque opaque yellowyellow with with Unity3D’s Unity3D’s line renderer. line renderer.

FigureFigure 1. 1.Empty Empty VR environment.VR environment.

The laboratory with assigned VR tracking space can be seen in Figure2. The laboratory with assigned VR tracking space can be seen in Figure 2. Appl. Sci. 2021, 11, 8064 4 of 19 Appl. Sci. 2021,, 11,, xx FORFOR PEERPEER REVIEWREVIEW 4 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 19

Figure 2. Space for VR testing with tracking stations. Figure 2. Space for VR testing with tracking stations. 2.3.2.3. 3D3D ModelsModels 2.3. 3D Models TheThe modelsmodels ofof thethe glueglue gungun (see(see FigureFigure3 3)) and and caulking caulking gun gun (see (see Figure Figure4 )4) were were first first The models of the glue gun (see Figure 3) and caulking gun (see Figure 4) were first downloadeddownloaded fromfrom thethe webweb andand slightlyslightly adjustedadjusted inin BlenderBlender softwaresoftware forfor ourour experiment.experiment. downloaded from the web and slightly adjusted in Blender software for our experiment.

Figure 3. Glue gun model in 3D Viewer. Figure 3. Glue gun model in 3D Viewer.

.. . Figure 4. Caulking gun model in 3D Viewer. Figure 4. Caulking gun model in 3D Viewer. 2.4.2.4. ShapeShape Models2.4. Shape Models 2.4. Shape 2.4. Shape Models Line,Line, circle,circle, andand sinesine wavewave modelsmodels werewere modelledmodelled inin BlenderBlender softwaresoftware usingusing curvecurve Line, circle, and sine wave models were modelled in Blender software using curve objectsobjects andand werewere veryvery thin,thin, inin orderorder toto createcreate a aa strict strictstrict movement movementmovement pattern patternpattern that thatthat participants participantsparticipants objects and were very thin, in order to create a strict movement pattern that participants hadhad toto followfollow (see(see FiguresFigures5 5–7).–7). The The thinner thinner the the objects objects were, were, the the less less would would be be the the scatter scatter had to follow (see Figures 5–7). The thinner the objects were, the less would be the scatter ininin precision.precision.precision. in precision.

Figure 5. Line model in 3D Viewer (default grey/unchanged colour). Figure 5. Line model in 3D Viewer (default(default grey/unchangedgrey/unchanged colour). Appl. Sci. 2021, 11, 8064 5 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 19

Figure 6. Sinus wave model in 3D Viewer (default grey/unchanged colour). FigureFigure 6.6. SinusSinus wavewave modelmodel inin 3D3D ViewerViewer (default(default grey/unchangedgrey/unchanged colour).

Figure 7. Circle model in 3D Viewer (default grey/unchanged colour). FigureFigure 7.7. CircleCircle modelmodel inin 3D3D ViewerViewer (default(default grey/unchangedgrey/unchanged colour). 2.5. Procedure 2.5.2.5. ProcedureProcedure One session lasted around 20–25 min, depending on the participants’ abilities. Each OneOne sessionsession lastedlasted aroundaround 20–2520–25 min,min, dependingdepending onon thethe participants’participants’ abilities.abilities. EachEach participant was instructed to draw with the glue gun (one hand) and caulking gun (both participantparticipant waswas instructedinstructed toto drawdraw withwith thethe glueglue gungun (one(one hand)hand) andand caulkingcaulking gungun (both(both hands) following the line on the table, as precisely and as quickly as possible, but it was hands)hands) followingfollowing thethe lineline onon thethe table,table, asas preciselyprecisely andand asas quicklyquickly asas possible,possible, butbut itit waswas completely up to them which of the two aspects (precision or speed) they would choose completelycompletely upup toto themthem whichwhich ofof thethe twotwo aspectsaspects (precision(precision oror speed)speed) theythey wouldwould choosechoose over the other. They just needed to do it the best way they could. overover thethe other.other. TheyThey justjust neededneeded toto dodo itit thethe bestbest wayway theythey could.could. Participants were handed HTC Vive Pro controllers and placed in a specific location ParticipantsParticipants werewere handedhanded HTCHTC ViveVive ProPro controllerscontrollers andand placedplaced inin aa specificspecific locationlocation in the real environment, so that the tracking stations could follow their hand movements inin thethe realreal environment,environment, soso thatthat thethe trackingtracking stationsstations couldcould followfollow theirtheir handhand movementsmovements throughout the whole experiment to avoid any jitter or freeze. Despite having the best throughoutthroughout thethe wholewhole experimentexperiment toto avoidavoid anyany jitterjitter oror freeze.freeze. DespiteDespite havinghaving thethe bestbest conditions, this still happened sometimes, so the drawing on which the tracking was jit- conditions,conditions, thisthis still still happened happened sometimes, sometimes, so so the the drawing drawing on on which which the the tracking tracking was was jittery jit- tery had to be erased and redone. Using specialized gloves/trackers for such experi- hadtery tohad be erasedto be anderased redone. and Usingredone. specialized Using sp gloves/trackersecialized gloves/trackers for such experiments/tasks for such experi- ments/tasks is the ideal situation as it provides maximum precision. But in this case, two isments/tasks the ideal situation is the ideal as it situation provides as maximum it provides precision. maximum But precision. in this case, But two in Vivethis case, tracking two stationsVive tracking were usedstations to track were the used hand to track movements the hand in VR.movements The lighting in VR. and The positioning lighting and of Vive tracking stations were used to track the hand movements in VR. The lighting and thepositioning users during of the the users experiment during the were experiment taken into were account; taken both into trackingaccount; stationsboth tracking had clear sta- positioning of the users during the experiment were taken into account; both tracking sta- viewstions had of the clear participant, views of andthe participant, the lighting and in the the room lighting was in also the enabled room was to sufficient also enabled levels. to tions had clear views of the participant, and the lighting in the room was also enabled to Thesufficient drawing levels. started The ondrawing button started press and on button finished press on button and finished release. on button release. sufficient levels. The drawing started on button press and finished on button release. OnceOnce thethe VRVR scenescene started,started, participantsparticipants wouldwould bebe ‘spawned’‘spawned’ rightright inin frontfront ofof thethe table, Once the VR scene started, participants would be ‘spawned’ right in front of the table, asas everythingeverything waswas positionedpositioned (both(both inin VRVR andand thethe RW)RW) inin aa wayway toto makemake thethe experienceexperience as everything was positioned (both in VR and the RW) in a way to make the experience forfor participantsparticipants asas comfortablecomfortable asas possible.possible. TheThe experimentexperiment waswas conductedconducted inin suchsuch aa wayway for participants as comfortable as possible. The experiment was conducted in such a way thatthat allall thethe objectsobjects requiredrequired forfor thethe experimentexperiment (and(and therethere werewere nono extraextra objectsobjects inin thethe that all the objects required for the experiment (and there were no extra objects in the environment)environment) werewere locatedlocated rightright inin frontfront ofof thethe user,user, atat thetheclosest closestpossible possibledistance. distance. SoSo environment) were located right in front of the user, at the closest possible distance. So whenwhen itit comes comes to to the the potential potential problem problem of stereopsisof stereopsis (difficulty (difficulty focusing focusing on farther on farther objects), ob- when it comes to the potential problem of stereopsis (difficulty focusing on farther ob- injects), this in particular this particular case, everything case, everything was done was in done such in a such way soa way that so the that users the could users easilycould jects), in this particular case, everything was done in such a way so that the users could focuseasily onfocus the on task the at task hand at and hand the and objects the objects in front in of front them. of Thethem. drawing The drawing was done was using done easily focus on the task at hand and the objects in front of them. The drawing was done Unity3Dusing Unity3D game engines game engines line renderer line renderer component, component, which draws which points draws on points the position on the of posi- the using Unity3D game engines line renderer component, which draws points on the posi- gluetion of gun the or glue caulking gun or gun caulking edge when gun edge the VR when controller the VR movement controller movement changes and changes the button and tion of the glue gun or caulking gun edge when the VR controller movement changes and isthe pressed. button Theis pressed. distance The at whichdistance the at points which could the points be drawn could was be adjusteddrawn was in suchadjusted a way in the button is pressed. The distance at which the points could be drawn was adjusted in that,such whena way drawn, that, when the points drawn, were the sopoints close were to each so close other to that each it createdother that the it illusion created of the a such a way that, when drawn, the points were so close to each other that it created the continuousillusion of a yellow continuous line. Ifyellow the distance line. If wasthe distance adjusted was for higheradjusted values, for higher such asvalues, 1 cm, such the illusion of a continuous yellow line. If the distance was adjusted for higher values, such pointsas 1 cm, would the points be drawn would 1 cm be apartdrawn from 1 cm each apart other from while each the other participant while the held participant the controller held as 1 cm, the points would be drawn 1 cm apart from each other while the participant held andthe controller drew in VR, and and drew with in this VR, big and distance with this between big distance points, between there would points, be there no continuous would be the controller and drew in VR, and with this big distance between points, there would be line,no continuous but simply line, points but in simply 3D space. points Additionally, in 3D space. the Additionally, more points the there more were, points the morethere no continuous line, but simply points in 3D space. Additionally, the more points there Appl.Appl. Sci.Sci.2021 2021,,11 11,, 8064 x FOR PEER REVIEW 66 of 1919 Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 19

were, the more accurate the data that could be obtained. In this experiment, the distance accuratewere, the the more data accurate that could the bedata obtained. that could In thisbe obtained. experiment, In this the experiment, distance was the less distance than a were,was less the than more a accuratemillimetre. the The data smaller that could the distancebe obtained. between In this the experiment, point centres, the the distance more millimetre.was less than The a smallermillimetre. the distanceThe smaller between the distance the point between centres, the the point more centres, points wouldthe more be pointswas less would than bea millimetre. drawn. The smaller the distance between the point centres, the more drawn.points would be drawn. pointsTime would and be distance drawn. were automatically calculated using our own Unity3D VR simu- TimeTime andand distancedistance werewere automatically automatically calculated calculated using using our our own own Unity3D Unity3D VR VR simula- simu- lation.Time There and is distanceno need wereto consider automatically the subjec calculatedtive time usingcompression our own factor Unity3D [36], VRas we simu- are tion.lation. There There is is no no need need to to consider consider the the subjective subjective time time compression compression factor factor [[36],36], asas wewe areare obtainingobtaininglation. There onlyonly is objective objectiveno need to realreal consider data.data. the subjective time compression factor [36], as we are obtaining only objective real data. 2.6.2.6. ResultsResults 2.6. Results 2.6. ResultsFiguresFigures 8 8–13,–13, which which are are the the results results from from the the VR VR experiment, experiment, show show the the trends trends in in the the Figures 8–13, which are the results from the VR experiment, show the trends in the resultsresultsFigures afterafter 778–13, sessionssessions which ofof thearethe experimenttheexperiment results from fromfrom th allalle theVRthe users’experiment,users’ combinedcombined show data,data, the trendswhichwhich isinis thethe results after 7 sessions of the experiment from all the users’ combined data, which is the averageresultsaverage after distancedistance 7 sessions (millimetres)(millimetres) of the experiment betweenbetween thethe from drawndraw alln the pointspoints users’ andand combined thethe actualactual data, staticstatic which 3D3D objectsobjects is the average distance (millimetres) between the drawn points and the actual static 3D objects (line,average(line, circle,circle, distance sine)sine) and(millimetres)and thethe averageaverage between timetime inin the secondsseconds drawn itit points tooktook to toand performperform the actual eacheach static task.task. The3DThe objects trendtrend (line, circle, sine) and the average time in seconds it took to perform each task. The trend towardstowards(line, circle, lessless sine) distancedistance and betweenbetweenthe average thethe pointstimepoints in and andseconds objectsobjects it took isis better.better. to perform TheThe samesame each goesgoes task. forfor The thethe trend timetime towards less distance between the points and objects is better. The same goes for the time requirementsrequirementstowards less distance toto performperform between thethe tasks.tasks. the points and objects is better. The same goes for the time requirements to perform the tasks.

Figure 8. One hand line avg. time to distance trend. FigureFigure 8.8. One hand line avg.avg. time to distance trend. Figure 8. One hand line avg. time to distance trend.

Figure 9. Both hands line avg. time to distance trend. Figure 9. Both hands line avg. time to distance trend. FigureFigure 9.9. BothBoth handshands lineline avg.avg. timetime toto distancedistance trend.trend.

Figure 10. One hand circle avg. time to distance trend. Figure 10. One hand circle avg. time to distance trend. FigureFigure 10.10. OneOne handhand circlecircle avg.avg. timetime toto distancedistance trend.trend. From the graphs above, it can clearly be seen that all the results from one hand and both hands VR experimental results of average time and distance improved from the first day all the way to the last. It can even be concluded that, in VR, no more than 4 sessions are Appl. Sci. 2021, 11, 8064 7 of 19

Appl. Sci. 2021, 11, x FOR PEER REVIEWrequired, as, after the 4th session, the trend of improvement slowed down, thus indicating7 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 19 that the benefits were sufficient right up until that point.

FigureFigure 11.11. BothBoth handshands circlecircle avg.avg. timetime toto distancedistance trend.trend. Figure 11. Both hands circle avg. time to distance trend.

Figure 12. One hand sine avg. time to distance trend. FigureFigure 12.12. OneOne handhand sinesine avg.avg. time toto distancedistance trend.trend.

Figure 13. Both hands sine avg. time to distance trend. FigureFigure 13.13. Both hands sinesine avg.avg. time to distance trend. From the graphs above, it can clearly be seen that all the results from one hand and 3. MethodsFrom the (RW) graphs above, it can clearly be seen that all the results from one hand and both hands VR experimental results of average time and distance improved from the first 3.1.both Equipment hands VR experimental results of average time and distance improved from the first day all the way to the last. It can even be concluded that, in VR, no more than 4 sessions day all the way to the last. It can even be concluded that, in VR, no more than 4 sessions are required,The same as, models after the as in4th VR session, were used the tr forend the of experiment improvement on paper,slowed but down, this timethus theyindi- are required, as, after the 4th session, the trend of improvement slowed down, thus indi- werecating a that real the glue benefits gun and were a caulkingsufficient gun right (see up Figuresuntil that 14 point. and 15 ). The sizes of the real modelscating that were the very benefits similar were to the sufficient ones in right VR, and up theuntil drawings that point. on the paper were similar to the3. Methods 3D models (RW) of the lines in VR. Below are shown the real models of the glue gun and caulking3. Methods gun. (RW) 3.1. Equipment 3.1. EquipmentAlthough the glue gun uses plastic as a glue that melts when attached to an electric socket,The in same this experimentmodels as in a VR pencil were lead used on for the the tip experiment of the glue on gun paper, was used.but this This time was they a The same models as in VR were used for the experiment on paper, but this time they muchwere a easier real solutionglue gun then and using a caulking melted gun plastic (see and Figures allowed 14 and us to 15). get The a continuous sizes of the line real on were a real glue gun and a caulking gun (see Figures 14 and 15). The sizes of the real themodels paper. were very similar to the ones in VR, and the drawings on the paper were similar models were very similar to the ones in VR, and the drawings on the paper were similar to theThe 3D caulkingmodels of gun the hadlines a in new VR. unopenedBelow are shown silicone the tube. real Similarly,models of insteadthe glue ofgun using and to the 3D models of the lines in VR. Below are shown the real models of the glue gun and siliconecaulking for gun. drawing, a small pencil was inserted in the nozzle. This was an easier solution caulking gun. and producedAlthough better the glue drawing gun uses results. plastic as a glue that melts when attached to an electric Although the glue gun uses plastic as a glue that melts when attached to an electric socket, in this experiment a pencil lead on the tip of the glue gun was used. This was a socket, in this experiment a pencil lead on the tip of the glue gun was used. This was a much easier solution then using melted plastic and allowed us to get a continuous line on much easier solution then using melted plastic and allowed us to get a continuous line on the paper. the paper. Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 19

The caulking gun had a new unopened silicone tube. Similarly, instead of using sili- Appl. Sci. 2021, 11, 8064 The caulking gun had a new unopened silicone tube. Similarly, instead of using8 of sili- 19 cone for drawing, a small pencil was inserted in the nozzle. This was an easier solution cone for drawing, a small pencil was inserted in the nozzle. This was an easier solution and produced better drawing results. and produced better drawing results.

Figure 14. Glue gun with pencil lead inside. FigureFigure 14.14. GlueGlue gungun withwith pencilpencil leadlead inside.inside.

Figure 15. Caulking gun with small pencil attached. FigureFigure 15.15. CaulkingCaulking gungun withwith smallsmall pencilpencil attached.attached. 3.2. Paper/RW Material 3.2.3.2. Paper/RWPaper/RW MaterialMaterial For both sessions for the paper experiment, 5 A4 format sheets of paper were used ForFor bothboth sessionssessions for for the the paper paper experiment, experiment, 5 A45 A4 format format sheets sheets of paperof paper were were used used for for each participant. eachfor each participant. participant. 1. One sheet of paper contained 2 lines on one side for one-hand drawing (glue gun), 1.1. One sheet of paper contained 2 lineslines onon oneone sideside forfor one-handone-hand drawingdrawing (glue(glue gun),gun), and 2 lines on the back of the sheet for both-hands drawing (caulking gun). and 2 lines on the back of the sheet for both-hands drawingdrawing (caulking(caulking gun).gun). 2. One sheet of paper with 2 circles on both sides for one-hand drawing (glue gun). 2.2. One sheet of paper with 2 circles on bothboth sidessides forfor one-handone-hand drawingdrawing (glue(glue gun).gun). 3. One sheet of paper with 2 circles on both sides for both-hands drawing (caulking gun). 3.3. One sheet of paper with 2 circ circlesles on on both both sides sides for for both-h both-handsands drawing drawing (caulking (caulking gun). gun). 4. One sheet of paper with 2 sine waves on both sides for one-hand drawing (glue gun). 4.4. One sheet of paper with 2 sine waves on both sides for one-hand drawing (glue gun). 5. One sheet of paper with 2 sine waves on both sides for both-hands drawing (caulking 5.5. One sheet of paper with with 2 2 sine sine waves waves on on bo bothth sides sides for for both-hands both-hands drawing drawing (caulking (caulk- gun). inggun). gun). In Figures 16–18 are photographs of the paper sheets with red lines drawn on them. InIn FiguresFigures 1616–18–18 areare photographsphotographs ofof thethe paperpaper sheetssheets withwith redred lineslines drawndrawn onon them.them. In addition, you can see the drawings made by participants with the glue gun and the InIn addition,addition, youyou cancan seesee thethe drawingsdrawings mademade byby participantsparticipants withwith thethe glueglue gungun andand thethe caulkingcaulking gungun (using(using pencil).pencil). TheThe firstfirst numbernumber (e.g.,(e.g., 5.75.7 oror 7.3)7.3) indicatesindicates thethe timetime inin secondsseconds caulking gun (using pencil). The first number (e.g., 5.7 or 7.3) indicates the time in seconds takentaken toto drawdraw thethe line.line. TheThe secondsecond numbernumber (1(1 oror 2)2) indicatesindicates thethe one-handone-hand (glue(glue gun)gun) oror Appl. Sci. 2021, 11, x FOR PEER REVIEWtaken to draw the line. The second number (1 or 2) indicates the one-hand9 (glueof 19 gun) or two-handtwo-hand (caulking(caulking gun)gun) test.test. two-hand (caulking gun) test.

FigureFigure 16. One-handOne-hand sine sine drawing. drawing.

Figure 17. One-hand line drawing.

Figure 18. Both-hands circle drawing.

3.3. Procedure Participants were positioned near the table where all the paper materials and glue gun or caulking gun were prepared. Once a participant was given a tool, they proceeded to draw the line on the paper. All participants were instructed to draw the lines onto the prepared red drawings as fast and precisely as possible. Choosing either higher precision or speed was completely up to them. It just needed to be done to the best of their abilities. Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 19

Appl. Sci. 2021, 11, 8064 9 of 19

Figure 16. One-hand sine drawing. Figure 16. One-hand sine drawing.

FigureFigure 17.17. One-handOne-hand lineline drawing.drawing. Figure 17. One-hand line drawing.

Figure 18. Both-hands circle drawing. FigureFigure 18.18. Both-handsBoth-hands circlecircle drawing.drawing. 3.3. Procedure 3.3.3.3. ProcedureProcedure Participants were positioned near the table where all the paper materials and glue ParticipantsParticipants werewere positionedpositioned nearnear thethe tabletable wherewhere allall thethe paperpaper materialsmaterials andand glueglue gun or caulking gun were prepared. Once a participant was given a tool, they proceeded gungun oror caulkingcaulking gungun werewere prepared.prepared. OnceOnce aa participantparticipant waswas givengiven aa tool,tool, theythey proceededproceeded to draw the line on the paper. All participants were instructed to draw the lines onto the toto drawdraw thethe lineline onon thethe paper.paper. AllAll participantsparticipants werewere instructedinstructed toto drawdraw thethe lineslines ontoonto thethe prepared red drawings as fast and precisely as possible. Choosing either higher precision preparedprepared redred drawingsdrawings as as fastfast andand preciselyprecisely as as possible.possible. ChoosingChoosing eithereither higherhigher precisionprecision or speed was completely up to them. It just needed to be done to the best of their abilities. oror speedspeed waswas completelycompletely upup toto them.them. ItIt justjust neededneeded toto bebe donedone toto thethe bestbest ofof theirtheir abilities.abilities. During the test, the experimenter would hold the paper so that it did not move and also measured the time with a stopwatch. After each drawing, the time to finish the drawing and the test type (1 or 2) were written down beside each drawing. Then the paper was either changed or turned to continue the process until all the drawings were finished. Overall, the whole process could take around 5 min, depending on the participant’s decision to go either for speed or accuracy.

3.4. Calculations Although the experiment on the paper measured the same factors as in VR, doing the calculations here in the same way as with the VR data would be a huge challenge or maybe even impossible. Therefore, a different route was chosen. In order to measure the accuracy of the paper drawings, a transparent plastic sheet was prepared on which was printed each type of shape with multiple points on them, which used as a reference to check if the drawings went through them or not. These covers then were placed onto the actual paper drawing results, and the points were counted. Only the points that touched the drawing Appl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 19

During the test, the experimenter would hold the paper so that it did not move and also measuredDuring the the test, time the with experimenter a stopwatch. would After hold each the drawing, paper so the that time it did to finishnot move the drawingand also measured the time with a stopwatch. After each drawing, the time to finish the drawing and the test type (1 or 2) were written down beside each drawing. Then the paper was and the test type (1 or 2) were written down beside each drawing. Then the paper was either changed or turned to continue the process until all the drawings were finished. Overall,either changed the whole or turnedprocess tocould continue take around the process 5 min, until depending all the ondrawings the participant’s were finished. deci- Overall, the whole process could take around 5 min, depending on the participant’s deci- sion to go either for speed or accuracy. sion to go either for speed or accuracy. 3.4. Calculations 3.4. Calculations Although the experiment on the paper measured the same factors as in VR, doing the Although the experiment on the paper measured the same factors as in VR, doing the calculations here in the same way as with the VR data would be a huge challenge or maybe evencalculations impossible. here Therefore,in the same a way different as with route the wasVR data chosen. would In order be a huge to measure challenge the oraccuracy maybe even impossible. Therefore, a different route was chosen. In order to measure the accuracy of the paper drawings, a transparent plastic sheet was prepared on which was printed Appl. Sci. 2021, 11, 8064 of the paper drawings, a transparent plastic sheet was prepared on which was printed10 of 19 each type of shape with multiple points on them, which used as a reference to check if the drawingseach type wentof shape through with multiplethem or not. points These on thcoveem,rs which then were used placed as a reference onto the to actual check paper if the drawings went through them or not. These covers then were placed onto the actual paper drawing results, and the points were counted. Only the points that touched the drawing mademadedrawing byby theresults,the participantparticipant and the were werepoints counted.counted. were counted. IfIf thethe drawndrawn Only shapetheshape points (grey)(grey) that andand touched thethe redred the figurefigure drawing hadhad made by the participant were counted. If the drawn shape (grey) and the red figure had somesome whitewhite spacespace inin between,between, thethe pointpoint waswas countedcounted asas aa miss.miss. BelowBelow youyou cancan seesee allall 33 some white space in between, the point was counted as a miss. Below you can see all 3 coverscovers forfor eacheach drawing.drawing. TheThe lineline covercover hadhad 5252 points,points, thethe circlecircle 112112 points,points, andand thethe sinussinus 8181covers points.points. for each drawing. The line cover had 52 points, the circle 112 points, and the sinus 81 points. UsingUsing thisthis transparenttransparent sheet sheet (see (see Figures Figures 19 19––2121),), it wasit was possible possible to accuratelyto accurately calculate calcu- thelate numbertheUsing number this of pointstransparent of points that that weresheet were present (see present Figures on on the 19– the drawn21), drawn it lineswas lines possible of of the the participants. toparticipants. accurately TheseThesecalcu- late the number of points that were present on the drawn lines of the participants. These calculationscalculations requiredrequired thethe manualmanual trackingtracking ofof allall thethe pointspoints andand addingadding themthem toto thethe datadata calculations required the manual tracking of all the points and adding them to the data table.table. TheThe VRVR experiment, whereinwherein datadata waswas automaticallyautomatically gatheredgathered inin aa JsonJson file,file, requiredrequired lesslesstable. manualmanual The VR work.work. experiment, Due to wherei thethe calculationscalculationsn data was beingbeingautomatically donedone manually,manually, gathered the thein a number numberJson file, ofof required pointspoints less manual work. Due to the calculations being done manually, the number of points (density)(density) calculatedcalculated for for the the RW RW paper paper tests tests was was limited. limited. All All the the calculations calculations were were done done by by a singlea(density) single person person calculated in in an an objective for objective the RW manner. manner. paper tests was limited. All the calculations were done by a single person in an objective manner.

FigureFigure 19.19. TransparentTransparent sheetsheet forfor lineline pointspoints calculationscalculations(paper (paperexp.). exp.). Figure 19. Transparent sheet for line points calculations (paper exp.).

Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19 Figure 20. Transparent sheet for sinus points calculations (paper exp.). FigureFigure 20.20. TransparentTransparent sheetsheet forfor sinussinus pointspointscalculations calculations (paper (paper exp.). exp.).

FigureFigure 21.21. TransparentTransparent sheetsheet forfor circlecircle pointspointscalculations calculations(paper (paperexp.). exp.).

4.4. ResultsResults PrecisionPrecision in the the RW RW tests tests was was calculated calculated diff differentlyerently from from in VR. in VR. The The more more points points pre- presentsent on the on thedrawn drawn shape, shape, the higher the higher the score. the score.FiguresFigures 22–26 show 22–26 that show the time that measure- the time measurementsments in seconds in secondsare the same are the as for same the as VR for measurements, the VR measurements, but when but it comes when to it comesprecision to precisionmeasurements, measurements, we count wethe countnumber the of number points that of points are positioned that are positioned correctly on correctly the prede- on thefined predefined visual shapes visual on shapes the paper on the (line, paper circle, (line, sine). circle, sine).

Figure 22. One hand line avg. time to accuracy trend.

Figure 23. Both hands line avg. time to accuracy trend.

Figure 24. Both hands circle avg. time to accuracy trend. Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19 Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19

Figure 21. Transparent sheet for circle points calculations (paper exp.). Figure 21. Transparent sheet for circle points calculations (paper exp.). Figure 21. Transparent sheet for circle points calculations (paper exp.). 4. Results 4. Results 4. ResultsPrecision in the RW tests was calculated differently from in VR. The more points pre- Precision in the RW tests was calculated differently from in VR. The more points pre- sent onPrecision the drawn in the shape, RW tests the higher was calculated the score. diff Figureserently 22–26 from show in VR. that The the more time points measure- pre- sent on the drawn shape, the higher the score. Figures 22–26 show that the time measure- mentssent on in the seconds drawn are shape, the same the higher as for thethe VRscore. measurements, Figures 22–26 but show when that it comesthe time to measure-precision Appl. Sci. 2021, 11, 8064 ments in seconds are the same as for the VR measurements, but when it comes to precision11 of 19 measurements,ments in seconds we are count the same the number as for the of VRpoints measurements, that are positioned but when correctly it comes on to the precision prede- measurements, we count the number of points that are positioned correctly on the prede- finedmeasurements, visual shapes we counton the the paper number (line, of circle, points sine). that are positioned correctly on the prede- fined visual shapes on the paper (line, circle, sine). fined visual shapes on the paper (line, circle, sine).

FigureFigure 22.22. OneOne handhand lineline avg.avg. time toto accuracyaccuracy trend.trend. Figure 22. One hand line avg. time to accuracy trend. Figure 22. One hand line avg. time to accuracy trend.

Figure 23. Both hands line avg. time to accuracy trend. FigureFigure 23.23. BothBoth handshands lineline avg.avg. timetime toto accuracyaccuracy trend.trend. Figure 23. Both hands line avg. time to accuracy trend.

Appl. Sci. 2021, 11, x FOR PEER REVIEWFigure 24. Both hands circle avg. time to accuracy trend. 12 of 19 Figure 24. Both hands circle avg. time to accuracy trend. FigureFigure 24.24. BothBoth handshands circlecircle avg.avg. timetime toto accuracyaccuracy trend.trend.

FigureFigure 25.25. OneOne handhand circlecircle avg.avg. timetime toto accuracyaccuracy trend.trend.

Figure 26. One hand sine avg. time to accuracy trend.

Results from the RW experiments, which were only done twice, on the first and on the last day of the experiment, indicate improvements only in the time variable, whereas the accuracy of the drawing worsened compared to the first attempt. One of the interest- ing changes that could be incorporated in such an experiment would be adding one more RW test in between the 7 VR tests to see if the results would be the same. Although there are multiple factors, some of the reasons why these results were as they are could be for- getting the RW task, the type of the test itself, lack of motivation, etc.

5. In-Depth Statistical Evaluation MATLAB R2017a software was used for the statistical evaluation. The main goal was to decide if there was a difference between the first and last attempt (VR and paper), in order to see if the participants improved after the process of training in VR.

5.1. Methodology of statistical evaluation Statistical hypothesis testing was used for the statistical processing of the experi- mental data. A statistical hypothesis is a statement that relates to an unknown property or parameters of the probability distribution of a random variable. In statistical testing, we always have two types of hypotheses:

Hypothesis 1 (H1). Null hypothesis 𝐻 means the hypothesis, the validity of which we verify using a test.

Hypothesis 2 (H2). Alternative hypothesis 𝐻 means the hypothesis that denies the validity of the null hypothesis.

To test the null hypothesis, we must always choose a significance level 𝛼, which is the probability of rejection of a true null hypothesis (type I error). We always require this probability to be small, so we choose, according to our habit, to be 𝛼 = 0.05. The decision to reject/not-reject the null hypothesis is made using the p-value of the test. We reject the null hypothesis if the p-value of the test is less than the level of significance. More about hypothesis testing can be found in [37]. Appl. Sci. 2021, 11, x FOR PEER REVIEW 12 of 19

Appl. Sci. 2021, 11, 8064 12 of 19

Figure 25. One hand circle avg. time to accuracy trend.

FigureFigure 26.26. OneOne handhand sinesine avg.avg. time toto accuracyaccuracy trend.trend.

ResultsResults fromfrom the the RW RW experiments, experiments, which which were were only only done done twice, twice, on theon the first first and and on the on lastthe daylast day of the of experiment,the experiment, indicate indicate improvements improvements only only in the in timethe time variable, variable, whereas whereas the accuracythe accuracy of the of drawingthe drawing worsened worsened compared compared to the to the first first attempt. attempt. One One of theof the interesting interest- changesing changes that that could could be incorporated be incorporated in such in such an experiment an experiment would would be adding be adding one moreone more RW testRW intest between in between the 7 the VR 7 tests VR tests to see to if see the if results the results would would be the be same. the same. Although Although there there are multipleare multiple factors, factors, some some of the of reasons the reasons why thesewhy these results results were aswere they as are they could are becould forgetting be for- thegetting RW task,the RW the task, type the of thetype test of itself,the test lack itself, of motivation, lack of motivation, etc. etc.

5.5. In-DepthIn-Depth StatisticalStatistical EvaluationEvaluation MATLABMATLAB R2017aR2017a softwaresoftware waswas usedused forfor thethe statisticalstatistical evaluation.evaluation. TheThe mainmain goalgoal waswas toto decidedecide ifif therethere waswas aa differencedifference betweenbetween thethe firstfirst andand lastlast attemptattempt (VR(VR andand paper),paper), inin orderorder toto seesee ifif thethe participantsparticipants improvedimproved afterafter thethe processprocess ofof trainingtraining inin VR.VR. 5.1. Methodology of Statistical Eevaluation 5.1. Methodology of statistical evaluation Statistical hypothesis testing was used for the statistical processing of the experimen- Statistical hypothesis testing was used for the statistical processing of the experi- tal data. A statistical hypothesis is a statement that relates to an unknown property or mental data. A statistical hypothesis is a statement that relates to an unknown property parameters of the probability distribution of a random variable. In statistical testing, we alwaysor parameters have two of typesthe probabilit of hypotheses:y distribution of a random variable. In statistical testing, we always have two types of hypotheses:

Hypothesis 1 (H1). Null hypothesis H0 means the hypothesis, the validity of which we verify usingHypothesis a test. 1 (H1). Null hypothesis 𝐻 means the hypothesis, the validity of which we verify using a test.

Hypothesis 2 (H2). Alternative hypothesis HA means the hypothesis that denies the validity of theHypothesis null hypothesis. 2 (H2). Alternative hypothesis 𝐻 means the hypothesis that denies the validity of the null hypothesis. To test the null hypothesis, we must always choose a significance level α, which is the probabilityTo test the of null rejection hypothesis, of a true we null must hypothesis always choose (type Iaerror). significance We always level require𝛼, which this is probabilitythe probability to be of small, rejection so we of choose,a true null according hypothesis to our (type habit, I error). to be Weα = always0.05. The require decision this toprobability reject/not-reject to be small, the null so we hypothesis choose, according is made using to our the habit,p-value to be of 𝛼 the test.= 0.05. WeThe reject decision the nullto reject/not-reject hypothesis ifthe thep null-value hypothesis of the test is ismade less thanusing the the level p-value of significance. of the test. We More reject about the hypothesisnull hypothesis testing if the can p be-value found of in the [37 test]. is less than the level of significance. More about hypothesisStatistical testing tests canare be usedfound to in verify [37]. the validity of the null hypothesis. There are parametric and nonparametric tests. Parametric tests are those for which we must know the distribution of data, and they usually assume a normal distribution. Nonparametric tests do not require this assumption. Three tests were used in this work: 1. Jarque–Bera test 2. Paired-sample t-test 3. Wilcoxon signed ranked test Jarque–Bera test First, the data obtained were tested for normality using the Jarque–Bera [38] test in order to decide whether we should use parametric or non-parametric statistical tests. Appl. Sci. 2021, 11, 8064 13 of 19

Parametric tests can only be used for data with normal distribution; non-parametric tests are used for data with different distributions [37]. The Jarque–Bera test tested the null hypothesis that the measured data follow a normal distribution against the alternative hypothesis that the measured data do not follow a normal distribution. If the p-value is greater than the significance level α = 0.05, we do not reject the null hypothesis of data normality and we can use a parametric test (paired-sample t- test). Otherwise, we reject the null hypothesis of data normality, and we have to use a non-parametric test (Wilcoxon signed rank test). Paired-sample t-test This paired-sample t-test [39] is parametric and can only be used for data that comes from a normal distribution. We test the null hypothesis that the distribution of the observed variable (time/distance/ accuracy) is identical in both groups (first and last attempt) and has the same mean (expected value) against the alternative hypothesis that the distribution of the observed variables is not identical in both groups and has a different mean. We reject the null hypothesis if the p-value of the test is less than the significance level α = 0.05, which means that there is a statistically significant difference between the first and last attempt. If we do not reject the null hypothesis, there is no statistically significant difference. Wilcoxon signed rank test If data does not come from a normal distribution, in the next step, we use the para- metric Wilcoxon test for two paired samples [40]. It is the non-parametric equivalent of the paired-sample t-test. We test that the null hypothesis data in both groups (first and last attempt) are samples from continuous distributions with equal medians against the alternative that they are not. The method of evaluation and conclusions are the same as for the paired-sample t-test.

5.2. VR Experiment Here these two hypotheses were verified:

Hypothesis 3 (H3). Distance-reducing trend (increasing drawing accuracy) for VR testing is to be expected.

Hypothesis 4 (H4). Time-reducing trend (increasing drawing speed) for VR testing is to be expected.

We tested the statistically significant difference between the first and seventh (last) attempt. First, the normality test was done. In Tables1 and2, we can see the p-values of the test for both observed variables (distance and time) and for all six versions of the experiment. Table 1. VR distance—Jarque–Bera test.

VR Distance One Hand Line One Hand Circle One Hand Sine both Hands Line both Hands Circle both Hands Sine Attempt 171717171717 p-value 0.001 0.500 0.035 0.500 0.246 0.416 0.390 0.500 0.056 0.024 0.234 0.391

Table 2. VR time—Jarque–Bera test.

VR Time One Hand Line One Hand Circle One Hand Sine both Hands Line both Hands Circle both Hands Sine Attempt 171717171717 p-value 0.002 0.041 0.500 0.018 0.500 0.323 0.500 0.369 0.173 0.255 0.153 0.051

As can be seen, the p-value was in some cases lower than the significance level α = 0.05. This means that not all the data sets come from a normal distribution. Therefore, Appl. Sci. 2021, 11, 8064 14 of 19

the non-parametric Wilcoxon signed rank test was used. It was tested whether there was a difference in significance level between the first and last attempt of the VR experiment for all the variants (one hand/both hands, line/circle/sinus wave). In Table3, we can see the results for the distance variable. The p-value is lower than the significance level α = 0.05 for all variants. This means that we reject the null hypothesis and that there is a statistically significant difference between the first and last attempt. We know from Figures8–13 that the average values of the distance variable decreased. This test confirmed this change with a significance level α = 0.05, and this means that the drawing accuracy increased, and Hypothesis H1 is confirmed.

Table 3. VR Distance—Wilcoxon signed rank test.

VR Distance One Hand Line One Hand Circle One Hand Sine both Hands Line both Hands Circle both Hands Sine p-value 0.002 0.001 0.004 0.006 0.002 0.001

Table4 shows the results for the time variable. Again, in all cases the p-value was lower than the significance level α = 0.05. This means that we reject the null hypothesis and that there is a statistically significant difference between the first and last attempt. As we know from Figures8–13, this difference is the shorter time in the last attempt than in the first. This confirms Hypothesis H2 about the expected time-reducing trend (increasing drawing speed) for VR testing.

Table 4. VR time—Wilcoxon signed rank test.

VR Time One Hand Line One Hand Circle One Hand Sine both Hands Line both Hands Circle both Hands Sine p-value 0.030 0.023 0.024 0.026 0.006 0.026

Real world experiment In the real world experiment, these two hypotheses were verified:

Hypothesis 5 (H5). Accuracy-increasing trend for RW tests (on paper) is to be expected.

Hypothesis 6 (H6). Time-reducing trend (increasing drawing speed) for RW tests (on paper) is to be expected.

As in the VR experiment, the difference was tested between the first and last (second) attempts. First, the normality test was done. The results of the normality test can be seen in Tables5 and6 for all versions of the experiment.

Table 5. Paper accuracy—Jarque–Bera test.

Paper One Hand both Hands both Hands One Hand Line One Hand Sine both Hands Sine Accuracy Circle Line Circle Attempt 1 2 1 2 1 2 1 2 1 2 1 2 p-value 0.043 0.500 0.016 0.435 0.488 0.162 0.035 0.500 0.135 0.500 0.371 0.185

Table 6. Paper time—Jarque–Bera test.

Paper One Hand both Hands both Hands One Hand Line One Hand Sine both Hands Sine Time Circle Line Circle Attempt 121212121212 p-value 0.089 0.086 0.471 0.144 0.500 0.209 0.052 0.224 0.358 0.118 0.206 0.143 Appl. Sci. 2021, 11, x FOR PEER REVIEW 15 of 19

Appl. Sci. 2021, 11, 8064 15 of 19 Table 5. Paper accuracy—Jarque–Bera test.

Paper Accuracy One Hand Line One Hand Circle One Hand Sine Both Hands Line Both Hands Circle Both Hands Sine Attempt 1 2 In1 the case2 of the accuracy1 variable,2 see1 Table52. The p-value1 was2 in some1 cases lower2 p-value 0.043 0.500than 0.016 the significance 0.435 level0.488 α =0.1620.05. This0.035 means 0.500 that not0.135 all of the0.500 data sets0.371 come from0.185 a normal distribution. Therefore, the non-parametric Wilcoxon signed rank test was used. InIn the case case of of the the time time variable, variable, see see Table Table 6. All6. Allp-valuesp-values were weregreater greater than the than signif- the significanceicance level level𝛼 =α 0.05=;0.05 we ;do we not do reject not reject the null the nullhypothes hypothesisis of data of data normality, normality, and and we wecan canuse usethe thepaired-sample paired-sample t-test.t-test. In Table7, we can see the results for the accuracy variable. The p-value was lower than the significanceTable level 6. Paperα = 0.05 time—Jarque–Berain the case of the test. one-hand line and circle and the both-hands line. This means that, in these cases, there was a statistically significant difference between Paper Time One Hand Linethe first One and Hand second Circle attempt. One Hand As Sine we know Both from Hands Figures Line Both 22–24 Hand, thiss Circle difference Both Hands was in Sine the Attempt 1 2reduction 1 in accuracy.2 This1 is the opposite2 of1 what we2 assumed.1 In the2 other versions1 of2 the p-value 0.089 0.086experiment, 0.471 we0.144 did not0.500 reject the0.209 null hypothesis,0.052 0.224 so there 0.358 was no0.118 statistically 0.206 significant 0.143 difference between the first and second attempt. This means that accuracy did not decrease significantlyIn Table in7, thesewe can cases, see the but result neithers for did the it accuracy improve. variable. Hypothesis The p H1-value was was therefore lower notthan confirmed. the significance level 𝛼 = 0.05 in the case of the one-hand line and circle and the both-hands line. This means that, in these cases, there was a statistically significant differ- ence betweenTable 7. Paper the first accuracy—Wilcoxon and second attempt. signed As rank we know test. from Figures 22–24, this difference was in the reduction in accuracy. This is the opposite of what we assumed. In the other Paper One Hand One Hand One Hand both Hands both Hands both Hands versions of the experiment, we did not reject the null hypothesis, so there was no statisti- Accuracy Line Circle Sine Line Circle Sine cally significant difference between the first and second attempt. This means that accuracy p-value 0.011did not decrease 0.034 significantly 0.083 in these cases, 0.020 but neither did 0.063 it improve. Hypothesis 0.095 H1 was therefore not confirmed. Table8 shows the results ( p-values) of the paired-sample t-test for the time variable. BasedTable on the 7. results,Paper accuracy—Wilcoxon we can say that there signed is rank a significant test. difference between attempts 1 and 2 in the case of the one-hand circle, sinus wave, and both-handsBoth Hands line, circle, and sinus Paper Accuracy One Hand Line One Hand Circle One Hand= Sine Both Hands Line Both Hands Sine wave, with a significance level α 0.05 %. There is not a significantCircle difference between the p-value 0.011 first and second0.034 attempt in the0.083 experiment with0.020 the one-hand line0.063 at the significance0.095 level α = 0.05 % (but there is at the level 0.1%). As we know from Figures 23–27, the trend of the time variableTable 8 shows declined the inresults all cases. (p-values) This means of the that paired-sample we have confirmed t-test for Hypothesisthe time variable. H2 in all versions of the experiment with the exception of the one-hand line at the significance Based on the results, we can say that there is a significant difference between attempts 1 level α = 0.05 %. However, even in this case, it can be seen from Figure 22 that some and 2 in the case of the one-hand circle, sinus wave, and both-hands line, circle, and sinus improvement has taken place, so we can confirm the Hypothesis H2 for this version as wave, with a significance level 𝛼 = 0.05 %. There is not a significant difference between well but at the significance level of 0.1 (a higher level of uncertainty). the first and second attempt in the experiment with the one-hand line at the significance level 𝛼=0.05 % (but there is at the level 0.1%). As we know from Figures 23–27, the Table 8. Paper time—paired-sample t-test. trend of the time variable declined in all cases. This means that we have confirmed Hy- One HandpothesisOne H2 Handin all versionsOne of Hand the experimentbothHands with the exceptionboth Hands of the one-handboth Hands line at Paper Time Line the significanceCircle level 𝛼=0.05Sine . % However,Line even in this case,Circle it can be seen fromSine Figure p-value 0.09122 that some 0.021 improvement 0.041has taken place, so 0.016 we can confirm 0.018 the Hypothesis 0.014H2 for this version as well but at the significance level of 0.1 (a higher level of uncertainty).

FigureFigure 27.27. BothBoth handshands sinesine avg.avg. time to accuracy trend.trend. 6. Discussion

The main point of this work was to discover the effects/benefits of using virtual reality for training. The training was focused on improvements to the precision of hand Appl. Sci. 2021, 11, 8064 16 of 19

movements using a specific type of VR simulation that would allow users/participants to perform similar movement tasks over a prolonged period of time (approximately one month). As VR is said to be a very effective technology in this sphere, we created a VR simulation using the Unity3D game engine for this experiment. This engine proved itself to be very powerful and easy to use, as it had already been used for other types of simulations prior to this. The simulation for this work took several months to prepare and conduct. In addition to VR tests, RW experiments also took place. VR tests took place in seven sessions, whereas there were only two RW tests, on the start and end dates of the experiment. This was done to analyse the effects of training not only in VR itself but also the transfer of skills to the RW. Once the experiment was done, the data were collected. VR data was gathered inside the Unity3D game engine during the test process itself. Then the data about precision (in Json format) were exported to Excel, where they were processed in pre-prepared macro templates. RW data on performing the line movements (line, circle, sine wave) was processed at the end of the experiment. Lastly, RW data was exported to Excel templates, and the precision results were calculated and analysed using specific macros. The results suggest very positive outcomes for VR training in particular. As the partic- ipants had to perform on each of seven sessions four attempts of each type of simulation (six overall: one hand/two hands, line/circle/sine wave); in the end, the results improved drastically, and they actually improved from the start to the end. It was pointed out that possibly only four training sessions for this type of training could be sufficient as, after the fourth attempt, although there were improvements, they were not that impressive. When it comes to the RW tests, the results suggest that the transfer of skill from VR to RW was only partial. A study suggested similar results, wherein the training in VR produced positive results in that virtual scenario, but when it came to transferring the training to the RW scenario, no significant or no benefits were found [41]. Only the time value improved from the first attempt compared to the second one. Multiple reasons were suggested for this, such as participant forgetfulness, noninterest, tiredness, etc. It is recommended to perform such an experiment with a much bigger pool of participants, in-depth questionnaires, and possibly perform RW tests more frequently between the VR tests to remind the participants what they are training for. Additionally, it could also be the type of simulation that resulted in these outcomes. In order to be sure about some factors that could cause these non-beneficial effects, more research and experimentation should be performed on top of this work. As this experiment and training look very similar to welding training simulations, it is interesting to point out a couple of studies that delved into training participants in this field and evaluated the transfer of that training to practical cases. A study [42] suggests that VR welding simulations are good not only for novice welders but also for experienced ones as well. The training in VR welding is then used in practice in RW cases. As the results show some difference in quality between novice and experienced welders, data also suggests that even experienced welders can struggle with higher difficulty welds in VR, which can in turn allow them to train and become better at new levels. Thus, suggesting that there is a positive correlation and transfer of skills from VR training to the real world depending on the simulation type/difficulty. On the other hand, research [43] on different types of welding difficulty tasks shows that, the higher the difficulty of the welding task in VR, the less effective VR training becomes, thus requiring supplementary RW training. But for easy and medium types of training difficulty, VR is considered to be a great tool, which suggests a positive outcome in the transfer of skills. It is important to note that the weight factor can play some role in the fatigue and performance of the participants during the tests. In our experiment, all the participants were young and healthy, thus using controllers in VR and simple common tools which did not exceed a weight of 1 kg (caulking gun—both hands), with the duration of the Appl. Sci. 2021, 11, 8064 17 of 19

experiment for a maximum of 15 min with breaks if needed, would not cause a significant hurdle for the participants to perform the task they were given. Although the weight of the two controllers and the caulking gun were almost the same (the caulking gun was slightly heavier), and one controller had almost the same weight as the glue gun, there was no significant point in conducting the experiment in VR and RW with identical weights. The experiment was very similar in some ways both in VR and RW, but some things—weight, controller vs. RW tool, etc.—could not be matched identically, thus measuring the effect of the weight difference on the result was not one of our goals. One possibility would be to use a tracker mounted on the tool instead of the controller, which could give better results, as the grip feeling and weight in VR and RW tests would be almost identical. The results of such tests would and should definitely give different results. This experiment was just a preliminary test of such a type of training in VR. Initially, the analysis and conclusions may look rough, but to make them more fine-tuned, more such tests should be performed and in different variations. Those results will then be compared, and more precise and expanded versions of the outcomes could be discussed. In this case, seven tests in VR were performed, whereas in the RW, there were only two. Initially, the idea to perform the RW test on the start date and end date of the VR tests seemed sufficient, and the results were believed to be more positive. As it came out at the end, the results in the RW tests were not so satisfactory, which suggests many different ideas, such as the type of the experiment, lack of repetitions, etc. The only way to answer those ideas precisely is to repeat the same and different experiments multiple times, with different variables. This work suggested using this type of training only in one way. Using information from this work and performing similar research could eventually result in different out- comes. Those outcomes then would be used to prove some other important points about using VR for training and its benefits. As mentioned, the difficulty of the simulation and supplementary RW training could play a big role in the transfer of skills. Future work should be targeted at delving into the factors described above more deeply, in order to prove or disprove the hypothesis.

7. Conclusions Four hypotheses were tested using statistical hypothesis testing and set out in Section6. VR experiment

Hypothesis 3 (H3). Distance-reducing trend (increasing drawing accuracy) for VR testing is to be expected.

Hypothesis 4 (H4). Time-reducing trend (increasing drawing speed) for VR testing is to be expected.

Results: Both hypotheses were confirmed. There were statistically significant changes (in terms of improvement) for both the monitored variables (distance, time). Real world experiment

Hypothesis 5 (H5). Accuracy-increasing trend for RW tests (on paper) is to be expected.

Hypothesis 6 (H6). Time-reducing trend (increasing drawing speed) for RW tests (on paper) is to be expected.

Results: Only Hypothesis H2 was confirmed for the time variable. There was no change or a change for the worse for the accuracy variable.

Author Contributions: Conceptualization, S.M. and P.H.; methodology, P.H., P.K. and M.B.; software, S.M.; validation, P.K., S.M. and M.B.; formal analysis, S.M. and P.H.; investigation, S.M.; resources, S.M., P.H. and M.Š.; data curation, S.M.; writing—original draft preparation, S.M.; writing—review Appl. Sci. 2021, 11, 8064 18 of 19

and editing, S.M., P.H. and M.B.; visualization, S.M. and P.H.; supervision, P.H. and P.K.; project administration, M.Š.; funding acquisition, M.Š. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by the Internal Science Foundation of the University of West Bohemia under Grant SGS-2021-028 Developmental and Training Tools for the Interaction of Man and the Cyber–Physical Production System’ and with the financial support of the European Union, as part of the project entitled development of capacities and environment for boosting the international, intersectoral and interdisciplinary cooperation, project reg. No. CZ.02.2.69/0.0/0.0/18_054/0014627. Institutional Review Board Statement: The study was approved by the internal university Ethics Committee. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: https://drive.google.com/file/d/1pgmPETqm7QhgJMjGUAEhG3 ChctiVkML0/view?usp=sharing (accessed on 26 August 2021). Conflicts of Interest: The authors declare no conflict of interest.

References 1. Accuracy, vs. Precision. Available online: https://www.forecast.app/faqs/what-is-the-difference-between-accuracy-and- precision (accessed on 29 March 2021). 2. Thurman, R.A. Instructional simulation from a cognitive psychology viewpoint. Educ. Technol. Res. Dev. 1993, 41, 75–89. [CrossRef] 3. Rieber, L.P. Computer-based micro worlds: A bridge between constructivism and direct instruction. Educ. Technol. Res. Dev. 1992, 40, 93–106. [CrossRef] 4. Abulrub, A.G.; Attridge, A.N.; Williams, M.A. Virtual reality in engineering education: The future of creative learning. In Proceedings of the Global Engineering Education Conference (EDUCON), Amman, Jordan, 4–6 April 2011; pp. 751–757. [CrossRef] 5. Gregory, S.; Gregory, B.; Reiners, T.; Fardinpour, A.; Hillier, M.; Lee, M.; Basu, A. Virtual worlds in Australian and New Zealand higher education: Remembering the past, understanding the present and imagining the future. In Proceedings of the ASCILITE 2013 “Electric Dreams”, Sydney, Australia, 1–4 December 2013; Macquarie University: Sydney, NSW, Australia, 2013. ISBN 9781741384031. 6. Jou, M.; Wang, J. Investigation of effects of virtual reality environments on learning performance of technical skills. Comput. Hum. Behav. 2013, 29, 433–438. [CrossRef] 7. Wilson, J.R. Virtual environments and ergonomics: Needs and opportunities. Ergonomics 1997, 40, 1057–1077. [CrossRef] 8. Mitchell, P.; Parsons, S.; Leonard, A. Using virtual environments for teaching social understanding to 6 adolescents with autistic spectrum disorders. J. Autism Dev. Disord. 2007, 37, 589–600. [CrossRef] 9. Cheng, Y.; Wang, S.H. Applying a 3D virtual learning environment to facilitate student’s application ability-the case of marketing. Comput. Hum. Behav. 2011, 27, 576–584. [CrossRef] 10. Bertram, J.; Moskaliuk, J.; Cress, U. Virtual training: Making reality work? Comput. Hum. Behav. 2015, 43, 284–292. [CrossRef] 11. Dede, C. Millennial learning styles. Educ. Q. 2005, 28, 7–12. 12. Dede, C. Immersive interfaces for engagement and learning. Science 2009, 323, 66–69. [CrossRef] 13. Hoˇrejší, P. Augmented Reality System for Virtual Training of Parts Assembly. Procedia Eng. 2015, 100, 699–706. [CrossRef] 14. Hoˇrejší, P.; Novikov, K.; Šimon, M. A Smart Factory in a Smart City: Virtual and Augmented Reality in a Smart Assembly Line. IEEE Access 2020, 8, 94330–94340. [CrossRef] 15. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 4th ed.; American Psychiatric Association: Washington, DC, USA, 2000; ISBN 0-89042-062-9. 16. Hooper, S.R.; Poon, K.K.; Marcus, L.; Fine, C. Neuropsychological characteristics of school-age children with high-functioning autism: Performance on the NEPSY. Child Neuropsychol. 2006, 12, 299–305. [CrossRef] 17. Merchant, Z.; Goetz, E.T.; Cifuentes, L.; Keeney-Kennicutt, W.; Davis, T.J. Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Comput. Educ. 2014, 70, 29–40. [CrossRef] 18. Lee EA, L.; Wong, K.W. Learning with desktop virtual reality: Low spatial ability learners are more positively affected. Comput. Educ. 2014, 79, 49–58. [CrossRef] 19. Nicholson, D.T.; Chalk, C.; Funnell WR, J.; Daniel, S.J. Can virtual reality improve anatomy education? A randomised controlled study of a computer-generated three-dimensional anatomical ear model. Med. Educ. 2006, 40, 1081–1087. [CrossRef] 20. Hettinger, L.J.; Riccio, G.E. Visually Induced Motion Sickness in Virtual Environments. Presence Teleoperators Virtual Environ. 1992, 1, 306–310. [CrossRef] 21. Lackey, S.J.; Salcedo, J.N.; Szalma, J.L.; Hancock, P.A. The stress and workload of virtual reality training: The effects of presence, immersion and flow. Ergonomics 2016, 59, 1060–1072. [CrossRef][PubMed] Appl. Sci. 2021, 11, 8064 19 of 19

22. Kim, E.; Shin, G. User discomfort while using a virtual reality headset as a personal viewing system for text-intensive office tasks. Ergonomics 2021. [CrossRef] 23. Kuze, J.; Ukai, K. Evaluation of Visual Fatigue Caused by Motion Images; Elsevier: Amsterdam, The Netherlands, 2008; Volume 29, pp. 159–166. [CrossRef] 24. Seay, A.F.; Krum, D.M.; Hodges, L.; Ribarsky, W. Simulator Sickness and Presence in a High FOV Virtual Environment. In Proceedings of the IEEE Virtual Reality Conference 2001 (VR2001), Yokohama, Japan, 13–17 March 2001; pp. 299–300. [CrossRef] 25. Chompoonuch, J.; Kazuhiko, H. Study on Parallax Affect on Simulator Sickness in One-Screen and Three-screen Immersive Virtual Environment. Sch. Inf. Telecommun. Eng. 2011, 4, 34–39. 26. Arns, L.L.; Cerney, M.M. The Relationship Between Age and Incidence of Cyber sickness Among Immersive Environment Users. In Proceedings of the IEEE Virtual Reality, Bonn, Germany, 12–16 March 2005; pp. 267–268. [CrossRef] 27. Parsons, T.D.; Larson, P.; Kratz, K.; Thiebaux, M.; Bluestein, B.; Buckwalter, J.G.; Rizzo, A.A. Sex Differences in Mental Rotation and Spatial Rotation in a Virtual Environment. Neuropsychologia 2004, 42, 555–562. [CrossRef] 28. Suma, E.A.; Finkelstein, S.L.; Clark, S.; Goolkasian, P.; Hodges, L.F. Effects of Travel Technique and Gender on a Divided Attention Task in a Virtual Environment. In Proceedings of the IEEE Symposium on 3D User Interface, Waltham, MA, USA, 20–21 March 2010; pp. 27–34. [CrossRef] 29. Dizio, P.; Lackner, J.R. Alleviation of Motion Sickness and Postural Instability during and after Virtual Environment Exposure; Techni- cal Report; Prepared for Naval Air Warfare Center Training Systems Division: Orlando, FL, USA, 1998. 30. Regan, E.; Price, K. The Frequency and Occurrence and Severity of Side-Effects of Immersion in Virtual Reality. Aviat. Space Environ. Med. 1994, 65, 527–530. 31. Bowman, D.; Kruijff, E.; LaViola, J.J.; Poupyrev, I. An introduction to 3-D user interface design. Teleoperators Virtual Environ. 2001, 10, 96–108. [CrossRef] 32. Rich, C.J.; Braun, C.C. Assessing the impact of control and sensory compatibility on sickness in virtual environments. In Pro- ceedings of the Human Factors and Ergonomics Society 40th Annual Meeting, Philadelphia, PA, USA, 2–6 September 1996. [CrossRef] 33. Hale, K.S.; Stanney, K.M. Handbook of Virtual Environments; Earlbaum: New York, NY, USA, 2002; ISBN 9780805832709. 34. Lathan, R. Tutorial: A brief introduction to simulation sickness and motion programming. Real Time Graph. 2001, 9, 3–5. 35. Michael, P. Motion Sick in Cyberspace. IEEE Comput. Graph. Appl. 1998, 18, 16–21. [CrossRef] 36. Mullen, G.; Davidenko, N. Time Compression in Virtual Reality. Timing Time Percept. 2021, 9, 1–16. [CrossRef] 37. Reif, J. Methods of Mathematical Statistics; University of West Bohemia, Faculty of Applied Sciences: Plzeˇn,Czech Republic, 2004; pp. 52–76, 157–169. ISBN 80-7043-302-7. 38. Jarque-Bera Test–MATLAB jbtest. MathWorks–Makers of MATLAB and Simulink–MATLAB & Simulink. Available online: https://www.mathworks.com/help/stats/jbtest.html (accessed on 9 February 2021). 39. One-Sample and Paired-Sample t-Test–MATLAB t-Test. MathWorks–Makers of MATLAB and Simulink–MATLAB & Simulink. Available online: https://www.mathworks.com/help/stats/ttest.html (accessed on 9 February 2021). 40. Wilcoxon Signed Rank Test–MATLAB Signrank. MathWorks–Makers of MATLAB and Simulink–MATLAB & Simulink. Avail- able online: https://www.mathworks.com/help/stats/signrank.html (accessed on 9 February 2021). 41. Kozak, J.J.; Hancock, P.A.; Arthur, E.J.; Chrysler, S.T. Transfer of training from virtual reality. Ergonomics 1993, 36, 777–784. [CrossRef] 42. Byrd, A.P.; Anderson, R.G.; Stone, R. The Use of Virtual Welding Simulators to Evaluate Experienced Welders; Industrial and Manufacturing Systems Engineering Publications. 2015, p. 114. Available online: https://lib.dr.iastate.edu/imse_pubs/114 (accessed on 26 August 2021). 43. McLaurin, E.J.; Stone, R.T. Comparison of Virtual Reality Training vs. Integrated Training in the Development of Physical Skills. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2012, 56, 2532–2536. [CrossRef]