sensors

Article Feedback from HTC Vive Sensors Results in Transient Performance Enhancements on a Juggling Task in Virtual Reality

Filip Borglund 1,†, Michael Young 1,† , Joakim Eriksson 1 and Anders Rasmussen 2,3,*

1 Virtual Reality Laboratory, Design Sciences, Faculty of Engineering, Lund University, 221 00 Lund, Sweden; fi[email protected] (F.B.); [email protected] (M.Y.); [email protected] (J.E.) 2 Department of Neuroscience, Erasmus University Medical Center, 3000 DR Rotterdam, The Netherlands 3 Associative Learning, Department of Experimental Medical Science, Medical Faculty, Lund University, 221 00 Lund, Sweden * Correspondence: [email protected]; Tel.: +46-(0)-709746609 † Filip Borglund and Michael Young contributed equally.

Abstract: Virtual reality headsets, such as the HTC Vive, can be used to model objects, forces, and interactions between objects with high perceived realism and accuracy. Moreover, they can accurately track movements of the head and the hands. This combination makes it possible to provide subjects with precise quantitative feedback on their performance while they are learning a motor task. Juggling is a challenging motor task that requires precise coordination of both hands. Professional jugglers throw objects so that the arc peaks just above head height, and they time their throws so that the second ball is thrown when the first ball reaches its peak. Here, we examined whether it is possible to learn to juggle in virtual reality and whether the height and the timing of the throws


can be improved by providing immediate feedback derived from the motion sensors. Almost all
 participants became better at juggling in the ~30 min session: the height and timing of their throws Citation: Borglund, F.; Young, M.; improved and they dropped fewer balls. Feedback on height, but not timing, improved performance, Eriksson, J.; Rasmussen, A. Feedback albeit only temporarily. from HTC Vive Sensors Results in Transient Performance Enhancements Keywords: juggling; feedback; HTC Vive; learning; timing; virtual reality on a Juggling Task in Virtual Reality. Sensors 2021, 21, 2966. https:// doi.org/10.3390/s21092966 1. Introduction Received: 18 March 2021 Rapid technological advancements have enabled researchers to obtain rich data about Accepted: 22 April 2021 Published: 23 April 2021 our own physiological state and movements. We can get feedback on activity levels and heart rate from smart phones and watches. Other sensors can give us information

Publisher’s Note: MDPI stays neutral about muscle activity or limb movements. Accumulating evidence shows that providing with regard to jurisdictional claims in feedback derived from movement sensors can enhance motor learning. For example, published maps and institutional affil- feedback from Electromyography (EMG) electrodes can help singers adjust the tone of iations. muscles involved in singing [1]. Feedback can also enhance postural control in elders [2]. Compared to qualitative feedback, providing quantitative feedback enhances learning of the crawl stroke in swimming [3]. Studies also show that feedback can improve the timing of movements. Thus, providing participants with data from periocular EMG data allows them to control the closure of their eyes with impressive timing accuracy [4]. There is even Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. speculation that one could use feedback in an immersive virtual reality (VR) environment This article is an open access article to train empathy [5]. distributed under the terms and Virtual reality (VR) refers to technology that enables human–computer interaction conditions of the Creative Commons that simulates as many senses as possible. More precisely, it is commonly used to describe Attribution (CC BY) license (https:// interactive computer-generated 3D environments that the user can interact with and that creativecommons.org/licenses/by/ incorporate sensory feedback. Other related technologies include augmented reality (AR), 4.0/). in which digital content is superimposed on a user view of the real world, and mixed

Sensors 2021, 21, 2966. https://doi.org/10.3390/s21092966 https://www.mdpi.com/journal/sensors Sensors 2021, 21, 2966 2 of 11

reality (MR), in which the real world and digital context co-exist and interact [6]. Recently, VR products utilizing some form of head mounted display (HMD) and positional tracking, such as the Oculus Rift and HTC Vive headsets, have grown in popularity. In this study, we use the HTC Vive, which has an accuracy that is superior to that of other HMDs, making it a promising tool for scientific and clinical purposes [7,8]. Juggling can refer to any type of activity that involves object manipulation. However, the word is most commonly used to describe toss juggling, which means throwing and catching objects (usually balls or clubs) in different patterns. Juggling is a mentally chal- lenging task that requires advanced sensorimotor coordination [9]. Practicing juggling leads to changes in cortical white matter architecture lasting at least four weeks, although the changes do not correlate with performance [10]. To succeed, it is important to throw the projectiles in a consistent pattern and to pay close attention to the timing and the height of the throws. Throwing the projectiles with shallow trajectories result in smaller errors, but it also increases the risk of collisions and reduces the amount of time that the juggler has to deal with the objects. A higher trajectory results in more time to correct mistakes but also a greater amplification of errors. The optimal height to throw the objects varies depending the number of objects—more objects necessitates higher throws [11]. While some professional jugglers can juggle using only proprioception and somatosensory input, novice and intermediate jugglers are dependent on their vision and cannot juggle for more than a few seconds if they close their eyes [12]. For this overwhelming majority, it is helpful if the zenith of the objects’ trajectories is within their visual field. Indeed, aspiring jugglers are often advised to look at the highest point of the trajectory and throw the next ball when the previous ball reaches the top [11,13]. The timing and height of throws can be taught by trial-and-error or by having a professional juggler tell the aspiring juggler when a throw was too high/low or too fast/slow. Evidently, these strategies work; however, it is conceivable that the training would be more efficient with more precise feedback. Despite being a complex sensorimotor task, skillful juggling relies heavily on a few measurable parameters, making it a useful model for studying motor learning in humans. Using VR to study and train motor skills is not new. Some studies suggests that subjects can learn simple motor tasks in Virtual Reality, at a rate comparable to that in a conventional training paradigm [14,15]. Moreover, skills acquired in VR leads to improved performance when switching to a conventional paradigm, although the performance following training in VR is slightly poorer compared to the performance when someone was trained in a conventional paradigm from the onset [15]. A number of studies shows that training within VR can improve performance and also help restore or improve motor ability in patients [16]. Nevertheless, some studies suggest that improvements in a VR context does not always generalize to real-life settings. For example, one recent study showed that playing a dart game available on HTC Vive resulted in worsening of real- life performance compared to real-life training; indeed, the dart throws actually got less accurate than dart throws prior to training in VR [17]. In other words, when it comes to the efficacy of improving motor skills through training in VR, results have been mixed. A recent review concluded that more research is necessary to study motor learning in different VR systems as well as to establish how learning using these systems translates to the real world [18]. Prior studies have examined different juggling variants in VR. In one study, the subjects learned to bounce a virtual ball on a monitor by moving a physical handle [19]. Another study, in which subjects were trained on a timing coincidence task, again showed that learning did occur and that learning in VR led to improvements outside VR [20]. Studies using other VR devices show that feedback can have a positive effect on motor learning [21], and that the VR experience feels realistic [22]. The purpose here was to see if new commercially available HMDs can be used to improve the performance on specific parameters that are key to juggling performance. Since the HTC Vive can simulate several sensory modalities, it can induce a sense of immersion—a feeling of being present in the digital world. This immersion, combined with precise motion tracking [7,8] and control Sensors 2021, 21, 2966 3 of 11

over the visual and auditory feedback, makes the HTC Vive and similar HMDs promising candidates for examining what type of feedback is optimal for learning. In this study, we provided participants with timing and height performance feedback, presented visually to the participants within the VR environment, to test if such feedback can help participants improve their throwing technique when juggling. The control group performed the exact same task but did not receive feedback following the throws. The study was designed to test whether it is possible to learn to juggle in an HMD VR setting and whether providing feedback on parameters crucial to juggling—the timing and height of throws—can improve performance on these parameters.

2. Materials and Methods 2.1. Participants Participants were recruited through a website dedicated to finding study participants (studentkaninen.se). People with prior juggling skills—defined as being able to juggle for more than a few catches—were not admitted to this study. This was done to ensure that existing skills did not confound the results. A total of 33 subjects participated. Of these 33 participants, 8 were excluded because they did not appear to understand the task or never grew accustomed to the VR environment. After the experiment, the participants were asked to fill out the NASA Task Load Index (TLX) [23], with one additional question added: asking how realistic the experience felt.

2.2. Materials In this study, we used the HTC Vive (HTC Corporation, Taipei, Taiwan) VR HMD sys- tem and associated controllers. The HTC Vive (Figure1A) is a consumer VR headset with accompanying controllers and a tracking system. Though not perfect, the tracking system associated with this headset is better compared to other commercially available consumer tracking systems [7]. The head mounted display on the HTC Vive features a 100 de- grees horizontal field of view, a 110 degrees vertical view, and two 1080 pixel × 1200 pixel screens. The headset was driven by a PC with an Intel i5-8400 CPU, 16 GB of RAM, and one Nvidia gtx 1070i graphics card. The simulation was able to run in a framerate of at least 90 frames per second (which is the nominal framerate using SteamVR). No framerate drops below 90 Hz were recorded). Hence the latency induced by the VR-system was expected to be around 11 ms. The HTC Vive has two controllers (Figure1B) with multiple methods of input, such as a touchpad, a “trigger button” (shaped like the trigger of a pistol), and a “grip button” on the side of the controller (see Figure1B). The controllers and headset are tracked by the Lighthouse tracking system, which uses two base stations that send out infrared signals that the controllers receive and convert into position data. This lets the computer track the position and rotation of the headset and controllers within the n where the base stations are located. Recent tests show that the HTC Vive is associated with accurate motion tracking [8,24,25].

2.3. Creating the Virtual Reality Environment The juggling simulator was created in the game engine and development environ- ment Unity. For certain features we used the SteamVR Unity Plugin version 2018.3.7f1 (Valve Corporation, Bellevue, WA, USA), IDE: Visual Studio 2017 (Microsoft Corporation, Redmond, WA, USA). This plugin contains several convenient GameObjects, components, and scripts (Figure1C). For example, it contains a Player GameObject that adds a camera GameObject, whose position and rotation is automatically linked to the head mounted display. The Player GameObject also contains two hands GameObjects (Figure1F) that are connected to the VR controllers. For the purposes of our application, we changed the model to just have the hands in red gloves without the virtual controllers, so that we could have a higher perceived realism when juggling. The SteamVR Unity Plugin also includes a system that allows developers to make any object in the environment interactable. The interaction system also has a Throwable component; when added to an object it lets the Sensors 2021, 21, 2966 4 of 11

Sensors 2021, 21, x FOR PEER REVIEW 4 of 11 user grab and throw the object when hovering over it by pressing and releasing the trigger or grip buttons on the controllers.

FigureFigure 1.1.Hardware Hardware and and user user interface. interface. The The HTC HTC Vive Vive headset headset and and the the base base station station is shown is shown in (A in); the(A); HTC the handHTC controllerhand con- istroller shown is inshown (B). In in ( (CB),). theIn ( interfaceC), the interface of Unity, of which Unity, was which used was to used create to the create juggling the juggling simulation, simulation, is shown. is shown. (D) shows (D) the shows TV withinthe TV the within VRenvironment the VR environment that subjects that receivedsubjects received instructions instructions on. (E) shows on. (E the) shows hand the and hand the ball and used the ball in the used juggling in the simulator.juggling simulator. (F) shows ( aF picture) shows of a thepicture two-ball of the juggling two-ball tutorial juggling that tutorial subjects that were subjects shown were in VR shown and asked in VR to and imitate. asked The to heightimitate. and The timing height feedback and timing given feedback to subjects given in to the subjects feedback in groupthe feedback is shown group in (G is, Hshown), respectively. in (G,H), respectively.

2.4.2.3. ProcedureCreating the Virtual Reality Environment ForThe ajuggling graphical simulator illustration was of created the training in the protocol,game engine see Figure and development2. Participants environ- were splitment into Unity. the twoFor group—acertain features feedback we group used andthe aSteamVR control group—inUnity Plugin an alternating version 2018.3.7f1 pattern. However,(Valve Corporation, since we had Bellevue, to exclude WA, USA), more participantsIDE: Visual Studio in the 2017 control (Microsoft group, theCorporation, last four participantsRedmond, WA, were USA). all assigned This toplugin the feedback contains group. several The convenient feedback groupGameObjects, (n = 12) received compo- feedbacknents, and on scripts their throws; (Figure the 1C). ‘control For example, group’ ( nit= contains 13) did nota Player receive GameObject feedback. Both that groupsadds a werecamera introduced GameObject, to the whose juggling position simulation and rotati in theon same is automatically way. First, they linked were to given the onlyhead onemounted ball to display. get used The to thePlayer physics GameObject and interaction also contains dynamics two of hands the simulation GameObjects (Figure (Figure2E). The1F) participantsthat are connected were asked to the to completeVR controllers five consecutive. For the purposes catches with of oneour ball,application, alternating we handschanged for the each model throw. to Becausejust have many the hands participants in red hadgloves very without limited the VR virtual experience, controllers, the first so fewthat practicewe could throws have a were higher done perceived at 60% realism gravity, when meaning juggling. that theThe ball SteamVR fell slower Unity than Plugin in realalso life.includes Performance a system on that these allows throws developers in 60% gravity to make were any not object analyzed in the further. environment After five in- successfulteractable. catches, The interaction the gravity system was increased also has toa Throwable 100%, thus givingcomponent; the subject when an added experience to an moreobject akin it lets to thatthe user of throwing grab and and throw catching the ob a ballject inwhen real hovering life. over it by pressing and releasingIn the the two-ball trigger stage, or grip the buttons subjects on first the watched controllers. two floating hands throw two balls in an optimal fashion. Then subjects picked up one ball in each hand and proceeded to throw the2.4. ballsProcedure in arcs from one hand to the other. The subjects were not given instructions about which hand to start throwing with, but most subjects started with their dominant hand. For a graphical illustration of the training protocol, see Figure 2. Participants were Both groups completed a total of 200 throws with two balls. The first 50 baseline trials did split into the two group—a feedback group and a control group—in an alternating pat- not involve any feedback. After the first 50 trials, participants assigned to the feedback tern. However, since we had to exclude more participants in the control group, the last group received feedback on the following 100 trials; on trials 51–100, they received feedback four participants were all assigned to the feedback group. The feedback group (n = 12) on the height of their throws; on trials 101–150, they received feedback on the timing of theirreceived throws. feedback The feedbackon their throws; was presented the ‘control on the group’ HMD (n screens = 13) did immediately not receive following feedback. eachBoth trial.groups Height were feedbackintroduced was to presentedthe juggling on si twomulation bars with in the a greensame andway. a First, red zone. they Twowere crossesgiven only represented one ball theto get height used of to the the throws physic froms and the interaction right and leftdynamics hand respectively. of the simulation If the cross(Figure appeared 2E). The below participants the midline were in asked the green to complete zone, it meant five consecutive that the altitude catches was with too low.one Ifball, the alternating cross was above hands the for midline, each throw. it meant Beca thatuse the many throw participants was too high. had Timingvery limited feedback VR wasexperience, presented the infirst a similarfew practice way. throws A cross were on a done horizontal at 60% bar gravity, with greenmeaning and that red the zones ball denotedfell slower the than relative in real timing life. Performance of the throws on withthese thethrows left andin 60% right gravity hand. were Thus, notfeedback analyzed wasfurther. presented After five immediately successfulafter catches, each the trial gravity in a waywas thatincreased was easy to 100%, for the thus participants giving the tosubject interpret an experience (see Figure more1G,H). akin On to the that last of 50throwing post-test and trials catching (151–200), a ball performance in real life. was

Sensors 2021, 21, 2966 5 of 11

Sensors 2021, 21, x FOR PEER REVIEW 5 of 11

again assessed without any feedback. The post-test trials were used to assess whether the feedback had an effect lasting beyond the trials with the feedback.

Figure 2. Training protocol. All participants first completed an initial practice phase in which Figure 2. Training protocol. All participants first completed an initial practice phase in which par- ticipantsparticipants got accustomed got accustomed to the to VR the VRenvironment environment and and learned learned to throw to throw and and catch catch one one ball. ball. Next, Next, all all participantsparticipants completed completed 50 50 throws throws with with two two balls balls.. After After these these 50 50 throws, throws, the feedback group receivedre- ceivedheight height feedback feedback on the on next the next 50 trials 50 trials and and then then timing timing feedback feedback on theon the next next 50 trials.50 trials. The The control controlgroup group simply simply continued continued to practice to practice without withou any quantitativet any quantitative input. The input. session Thefinished session withfinished another with50 throws another in 50 which throws no in feedback which no was feedback given. See was Figure given.1 forSee largerFigure versions 1 for larger of the versions screenshots. of the screenshots. 2.5. Measures InA the principal two-ball aim stage, of this the studysubjects was first to watched examine two if the floating height hands and timing throw of two juggling balls inthrows an optimal can be fashion. optimized Then by subjects providing picked immediate up one performance ball in each feedbackhand and in proceeded VR. But how to throwdoes onethe balls define in optimalarcs from height? one hand The to optimal the other. height The subjects and relative were timing not given of throws instructions varies aboutslightly which depending hand to onstart the throwing height and with, the but personal most subjects preference started of the with juggler. their dominant The more hand.objects Both juggled, groups the completed higher the a throwstotal of also 200 needthrows to bewith [9, 11two]. However,balls. The jugglersfirst 50 baseline handling trials2–3 objectsdid not throws involve the any ball feedback. so that they After see thethe zenith,first 50 and trials, they participants throw the next assigned ball when to the the feedbackprevious group one has received just reached feedback the zenithon the [11follo]. Basedwing 100 on this,trials; combined on trials with51–100, observations they re- ceivedof a professional feedback on juggler the height (Filip of Borglund), their throws we; definedon trials the 101–150, optimal th heightey received as 25 feedback cm above oneye-height the timing and of wetheir defined throws. the The optimal feedback time was to throw presented the second on the ball HMD as 25 screens ms after immedi- the first atelyball reachedfollowing its each peak. trial. It is Height important feedback to note was that presented even professional on two bars jugglers with a will green not and throw a redat thezone. exact Two same crosses height represented each time. the However, height forof the this throws experiment, from thethe mostright important and left hand thing respectively.was to have If a the reference cross appeared point, to seebelow if our the participants midline in the would green adapt zone, to it feedback. meant that Before the altitudethe participants was too low. started If the throwing cross was the above balls themselves,the midline, optimalit meant throws that the as throw defined was above, too high.were Timing shown feedback to subjects was in thepresented juggling in tutorial a similar in way. the VR A environmentcross on a horizontal (see Figure bar1 F).with To greenquantitatively and red zones assess denoted the subjects’ the relative throws, timing we of compared the throws the with height the left and and relative right timinghand. Thus,of each feedback throw was to the presented optimal immediately throws. This after gave each us threetrial in measures a way that that was we easy used for in the the participantsstatistical analysis: to interpret absolute (see Figure height 1G,H). error forOnthe the rightlast 50 and post-test the left trials hand, (151–200), and the absoluteperfor- manceerror inwas the again relative assessed timing without of the secondany feedback. throw. InThe addition, post-test we trials recorded were used the number to assess of whetherdrops before the feedback each successful had an effect attempt. lastin Ag dropbeyond was the defined trials with as a the collision feedback. between a ball and the floor or a table. If the first ball was dropped before the second ball was thrown, an 2.5.attempt Measures would not be registered and the height and timing would not be saved. This made it possible for a person to drop more than twice per registered attempt. A principal aim of this study was to examine if the height and timing of juggling throws3. Results can be optimized by providing immediate performance feedback in VR. But how does3.1. Analysisone define optimal height? The optimal height and relative timing of throws varies slightly depending on the height and the personal preference of the juggler. The more To test whether the juggling performance improved as a result of the VR training, objects juggled, the higher the throws also need to be [9,11]. However, jugglers handling we used a combination of standard t-tests and linear mixed effects models to predict the 2–3 objects throws the ball so that they see the zenith, and they throw the next ball when number of drops, the absolute height error of throws with the right and the left hand, the previous one has just reached the zenith [11]. Based on this, combined with observa- and the timing error of the throws. In the linear mixed effects model, we used throw tions of a professional juggler (Filip Borglund), we defined the optimal height as 25 cm number (1–200) and group (feedback/control) as fixed effects and subject as a random effect above(MATLAB eye-height code isand provided we defined as Supplementary the optimal time Material). to throw Linear the second mixed ball effects as 25 models ms after are thestatistically first ball reached robust and its unlikepeak. It ANOVAs is important they to take note into that account even professional individual differences jugglers will [26 ]. not throw at the exact same height each time. However, for this experiment, the most im- portant thing was to have a reference point, to see if our participants would adapt to feed-

Sensors 2021, 21, x FOR PEER REVIEW 6 of 11

back. Before the participants started throwing the balls themselves, optimal throws as de- fined above, were shown to subjects in the juggling tutorial in the VR environment (see Figure 1F). To quantitatively assess the subjects’ throws, we compared the height and rel- ative timing of each throw to the optimal throws. This gave us three measures that we used in the statistical analysis: absolute height error for the right and the left hand, and the absolute error in the relative timing of the second throw. In addition, we recorded the number of drops before each successful attempt. A drop was defined as a collision be- tween a ball and the floor or a table. If the first ball was dropped before the second ball was thrown, an attempt would not be registered and the height and timing would not be saved. This made it possible for a person to drop more than twice per registered attempt.

3. Results 3.1. Analysis To test whether the juggling performance improved as a result of the VR training, we used a combination of standard t-tests and linear mixed effects models to predict the num- ber of drops, the absolute height error of throws with the right and the left hand, and the timing error of the throws. In the linear mixed effects model, we used throw number (1–200) Sensors 2021, 21, 2966 and group (feedback/control) as fixed effects and subject as a random effect (MATLAB6 of 11 code is provided as Supplementary Material). Linear mixed effects models are statistically robust and unlike ANOVAs they take into account individual differences [26]. To assess if feedback had an effect lasting beyond the feedback session, we compared the perfor- manceTo assess of the if feedback feedback had group an effect and the lasting control beyond group the on feedback the last session,50 trials we(post-test) compared when the neitherperformance group ofreceived the feedback any feedback. group and the control group on the last 50 trials (post-test) when neither group received any feedback. 3.2. Drops 3.2. Drops Overall, there was a clear improvement in the juggling performance over the entire Overall, there was a clear improvement in the juggling performance over the entire session (Figure 3). On average, the number of drops decreased by 0.0045 ± 0.0021 per session (Figure3). On average, the number of drops decreased by 0.0045 ± 0.0021 per throw throw (p < 0.0001, CI = (0.0041–0.0049)). Thus, following 200 throws subjects will on aver- (p < 0.0001, CI = (0.0041–0.0049)). Thus, following 200 throws subjects will on average age drop 200 × 0.0045 = 0.9 balls less before a successful attempt. This was also verified in drop 200 × 0.0045 = 0.9 balls less before a successful attempt. This was also verified in a paired t-test comparing throws 41–50 with the last ten throws (191–200). On average, the a paired t-test comparing throws 41–50 with the last ten throws (191–200). On average, participants dropped 0.448 ± 0.507 balls less on the last 10 trials. This difference was sta- the participants dropped 0.448 ± 0.507 balls less on the last 10 trials. This difference tistically significant: t(24) = 4.42, p = 0.00018, CI = (0.24–0.65), Cohen’s d = 0.89. Neverthe- was statistically significant: t(24) = 4.42, p = 0.00018, CI = (0.24–0.65), Cohen’s d = 0.89. less, there was no statistically significant difference between the feedback and the control Nevertheless, there was no statistically significant difference between the feedback and the groupcontrol − group0.027 ±− 0.1290.027 (p± =0.129 0.833, (p CI= 0.833,= (−0.281–0.226)), CI = (−0.281–0.226)), indicating indicating that the feedback that the feedback did not affectdid not the affect number the numberof dropped of dropped balls. balls.

FigureFigure 3. 3. Drops.Drops. The The figure figure shows shows the thenumber number of drop of dropss before before each eachsuccessful successful two-ball two-ball throw throw for allfor participants. all participants. Each Eachdot represents dot represents the number the number of drops of before drops one before successful one successful attempt attemptfor one for participant.one participant. The line The in line the inmiddle the middle shows showsthe median the median value. There value. is Therea clear isdecline a clear in decline the number in the dropsnumber before drops each before successful each successful attempt as attempt the part asicipants the participants spend more spend time more in the time simulator, in the simulator, sug- gestingsuggesting that thatparticipants participants improved improved their their perfor performancemance over over time. time. The Theboxplot boxplot to the to right the right shows shows the distribution of change in drops. For each participant, the average number of drops on the last 10 trials (191–200) was subtracted from the number of drops on throws 41–50. 3.3. Height Using two similar mixed effects models, we looked at the absolute height error for the right and the left hand. For the right hand (see Figure4A,B), the height error decreased by 0.48 ± 0.034 mm on average per throw (p < 0.0001, CI = (−0.55–−0.41 mm)). Over the 200 trials, this adds up to a 200 × 0.48 = 96 mm improvement. Left hand height errors followed a similar pattern (Figure4C,D). The error got smaller by 0.45 ± 0.037 mm per throw (p < 0.0001, CI = (−0.52–−0.37 mm)), or 200 × 0.45 = 90 mm over the 200 trials. Over the 200 trials, those receiving feedback did not perform significantly better than the control group (Figure5A,B). For the right hand the difference between the feedback group and the control group was −8.7 ± 24.9 mm (p = 0.73, CI = (−57–40 mm)). For the left hand the difference was 3.2 ± 25.7 mm (p = 0.90, CI = (−40–53 mm)). In line with this, performance on the post-test trials (151–200) did not differ between the feedback and the control group (Figure5A–C). However, within the feedback session (trials 51–100), there was a significant interaction. Participants who received feedback improved more than participants in the control group, for both hands (p < 0.01). To follow up on this, we ran two t-tests in which we compared the height error in the 10 trials before the feedback with the errors in the last 10 trials of the height feedback phase. For the right hand, those receiving feedback improved by 99.6 ± 139.1 mm on average while participants in the control group actually became worse by 10.8 ± 75.2 mm on average. For the left hand, the feedback improved by Sensors 2021, 21, x FOR PEER REVIEW 7 of 11

the distribution of change in drops. For each participant, the average number of drops on the last 10 trials (191–200) was subtracted from the number of drops on throws 41–50.

3.3. Height Using two similar mixed effects models, we looked at the absolute height error for the right and the left hand. For the right hand (see Figure 4A,B), the height error decreased by 0.48 ± 0.034 mm on average per throw (p < 0.0001, CI = (−0.55–−0.41 mm)). Over the 200 trials, this adds up to a 200 × 0.48 = 96 mm improvement. Left hand height errors followed a similar pattern (Figure 4C,D). The error got smaller by 0.45 ± 0.037 mm per throw (p < 0.0001, CI = (−0.52–−0.37 mm)), or 200 × 0.45 = 90 mm over the 200 trials. Over the 200 trials, Sensors 2021, 21, 2966those receiving feedback did not perform significantly better than the control group (Fig- 7 of 11 ure 5A,B). For the right hand the difference between the feedback group and the control group was −8.7 ± 24.9 mm (p = 0.73, CI = (−57–40 mm)). For the left hand the difference was 3.2 ± 25.7 mm109.4 (p ±= 0.90,96.6 mmCI = on (−40–53 average mm)). while In the line control with this, group performance improved by on 5.3 the± post-105.6 mm in the test trials (151–200)same did period. not differ In agreement between with the feedback the linear and mixed the effects control model group these (Figure were 5A– significant, for C). However, withinthe right the hand, feedback t(23) =session 2.50, p =(trials 0.02, 51–100), CI = (19–202 there mm), was Cohen’s a significant d = 0.99; interac- as well as for the tion. Participantsleft who hand received t(23) = 2.57, feedbackp = 0.017, impr CIoved = (20–188 more mm),than participants Cohen’s d = 1.03in the (Figure control5D,E). group, for both hands (p < 0.01). To follow up on this, we ran two t-tests in which we compared the height3.4. Timing error in the 10 trials before the feedback with the errors in the last 10 trials of the height feedbackWith another phase. mixed For the effects right model, hand, we those modelled receiving the feedback timing error, improved again using throw by 99.6 ± 139.1 andmm groupon average as fixed while effects participants and subject in asthe a control random group effect. actually As with became the height, there worse by 10.8 ±was 75.2 a mm small on but average. significant For the decrease left hand, in the the absolute feedback timing improved error over by 109.4 the 200 ± trials (see 96.6 mm on averageFigure 4whileE,F ). the On control average, group the timing improved error gotby 5.3 smaller ± 105.6 by 0.068mm in± the0.015 same ms per trial or × period. In agreement200 0.068 with =the 13.6 linear ms over mixed 200 ef trials.fects Thismodel effect these was were statistically significant, significant for the (p < 0.0001, CI = (−0.099–−0.0386 ms)). However, there was no differences between the feedback right hand, t(23) = 2.50, p = 0.02, CI = (19–202 mm), Cohen’s d = 0.99; as well as for the left group and the control group; not in the post-test and not within the timing feedback hand t(23) = 2.57, p = 0.017, CI = (20–188 mm), Cohen’s d = 1.03 (Figure 5D,E). session (Figure5C,F).

Figure 4. Height and timing errors (mean ± SEM) for participants receiving feedback (top row, colored lines/dots) and Figure 4. Height and timing errors (mean ± SEM) for participants receiving feedback (top row, controls (bottomcolored row, lines/dots) black lines). and controls Panels A –(bottomE shows row, the differenceblack lines). between Panels the A–E optimal shows height the difference of throws be- and the height of the actualtween throws the optimal with the height right hand of thro (Aws,B), and and the the height left hand of the (C, Dactual). Panels throws (E,F with) shows the theright timing hand error. (A,B), The color representsand the the stage left of hand the experiment: (C,D). Panels blue (E, representF) shows baselinethe timing trials, error. before The thecolor participant represents received the stage any of feedback; the red represent throws on trials where the participants received feedback on the height of their throws; yellow represent trials where participants received timing feedback; and purple represent post-feedback trials where participants did not receive any feedback.

Sensors 2021, 21, x FOR PEER REVIEW 8 of 11

experiment: blue represent baseline trials, before the participant received any feedback; red repre- Sensors 2021, 21, 2966 sent throws on trials where the participants received feedback on the height of their throws; yel-8 of 11 low represent trials where participants received timing feedback; and purple represent post-feed- back trials where participants did not receive any feedback.

FigureFigure 5.5. EffectEffect ofof heightheight and and timing timing feedback. feedback. (A (A–C–C) displays) displays the the height/timing height/timing of of the the throws throws before feedback,before feedback, at the endat the of end the feedbackof the feed sessions,back sessions, and on and the on post-test. the post-test. (D–F )(D displays–F) displays how how much the height/timingmuch the height/timing of the throws of the improved throws improved during the du feedbackring the feedback session.This session. was This calculated was calculated by taking by taking the average error on the last 10 trials before the feedback session (throws 41–50) and the average error on the last 10 trials before the feedback session (throws 41–50) and subtracting the subtracting the average error on the last 10 trials of the feedback session (41–50 for height; 91–100 average error on the last 10 trials of the feedback session (41–50 for height; 91–100 for timing), and for timing), and then comparing the change to the change in the control group. Participants receiv- thening feedback comparing on theheight change improved to the the change height in of the their control throws group. more Participants than those receivingwho did not feedback receive on heightfeedback. improved However, the heightthis was of not their the throws case for more thethan timing those feedback. who did not receive feedback. However, this was not the case for the timing feedback. 3.4. Timing 4. Discussion With another mixed effects model, we modelled the timing error, again using throw The aim of this study was to examine if participants can learn to juggle in VR and and group as fixed effects and subject as a random effect. As with the height, there was a whether feedback on the height and timing of the throws can enhance juggling performance. small but significant decrease in the absolute timing error over the 200 trials (see Figure Almost all participants improved their performance. They dropped fewer balls and the height4E,F). On as wellaverage, as the the timing timing oferror their got throws smaller improved by 0.068 ± significantly 0.015 ms per over trial theor 200 200 × trials, 0.068 suggesting= 13.6 ms over that 200 it is trials. indeed This possible effect towas learn statistically to juggle significant in a VR environment. (p < 0.0001, FeedbackCI = (−0.099– on the−0.0386 height ms)). of the However, throws resultedthere was in no temporary differences enhancements. between the However,feedback group the relative and the timing con- oftrol the group; throws not did in the not post-test improve and as a not result within of timing the timing feedback. feedback At thesession end (Figure of the session, 5C,F). all participants dropped fewer balls, but the height and the timing of the throws did not differ4. Discussion between the feedback and the control group. We found some evidence that feedback in VRThe transiently aim of this improves study jugglingwas to examine performance. if participants Participants can receivinglearn to juggle height in feedback VR and didwhether improve feedback the height on the of height their throwsand timing more of than the thethrows control can group.enhance However, juggling timingperfor- feedbackmance. Almost was not all associated participants with improved similar benefits, their performance. and on the post-test They dropped throws fewer there wereballs noand differences the height between as well theas the performance timing of oftheir the throws feedback improved group and significantly control group. over The the fact200 thattrials, the suggesting height feedback that it hadis indeed a temporary possible effect to learn suggests to juggle that itin can a VR be useful,environment. and perhaps Feed- repeatedback on the feedback height sessionsof the throws could resulted consolidate in temporary the improvements. enhancements. However, the rel- ativeSome timing participants of the throws appeared did not to improve have difficulties as a result focusing of timing on feedback. the height At and the timingend of ofthe the session, throws all while participants also catching dropped the balls.fewer Itba islls, unclear but the whether height theseand the difficulties timing of were the due to the limited time given to adapt to the VR environment, or if they were due to their limited juggling experience. Again, a potential solution might be to have multiple sessions or longer sessions. VR gives researchers the opportunity to control much of the sensory input received by the subject. However, in some respects it differs from real-life practice. One major limitation of VR, especially for tasks such as juggling, is the limited peripheral Sensors 2021, 21, 2966 9 of 11

vision. In our juggling exercise, participants could sometimes not see if they had actually caught the balls in their hands. The limited haptic stimulation ability of the HTC Vive controllers did not improve this matter. Recently released glove controllers from Valve allows one to track every finger separately. These improvements will add to the perceived realism of the task. Still, most participants felt that our juggling simulation—despite its limitations—was realistic. During the VR juggling session, participants improved on all measured parameters: the number of drops; the height of the throws; and the relative timing of the throws. In short, the participants learned the task. While some participants were unaccustomed to the VR environment and therefore needed more time to get used to it, no one experienced any motion sickness. The entire session lasted 30–40 min, and none of the participants reported experiencing any postural or visual fatigue. The fact that participants can learn a complex motor task in a short amount of time (~30 min) in a VR environment opens up for many possible applications. Basic motor programs for all kinds of tasks, such as playing sports, steering vehicles, or rehabilitation following injuries or strokes, might benefit from training in an immersive VR Environ- ment [27]. Not only does training in VR allow one to provide subjects with real-time quantitative feedback; it also allows trainers or care givers to get accurate and quantitative measurements on the progress that the subject makes. Indeed, HMDs could potentially be used to assess motor performance in clinical settings; for example, when assessing people for neuropsychiatric disorders. Currently, such motor assessments are carried out by clinicians, which means that personal biases and preconceptions can affect the results. Moreover, the ability to change fundamental constants, such as gravity, or to change the sensory inputs in a controlled fashion, enables researchers to assess subjects’ ability to adapt to such changes, which can be useful for research purposes as well as training purposes. Within VR, one can directly manipulate the visual input received by the subject. This means that one could remove motor noise that has been shown to slow down motor learning [28]. This could prove particularly valuable for cerebellar patients who appear to fail to learn tasks because of increased motor noise [29]. Earlier studies have shown that training in a VR environment can be even better than training in real life, though there is also a risk of becoming over-dependent on the VR feedback [21]. Overall, the literature does not provide a clear answer as to whether performance improvements in VR successfully translates to better performance in real life [17,18].

5. Conclusions Following a ~30 min training session in an immersive VR environment, participants exhibited clear signs of learning a simple ball juggling task. Moreover, feedback on specific performance parameters—specifically the height and timing of throws—resulted in transient improvements. These findings highlight the potential of VR technologies to improve sports performance and for rehabilitation of motor deficits. Future studies should examine if further training sessions consolidates improvements and to what extent improvements in VR translates to improvements in the real world.

Supplementary Materials: The following are available online at https://www.mdpi.com/article/10 .3390/s21092966/s1, Juggling_Anal.mlx, jugglingdata.mat. Author Contributions: Conceptualization, F.B., M.Y., J.E. and A.R.; methodology, F.B., M.Y., J.E. and A.R.; software, F.B., M.Y. and J.E.; validation, F.B., M.Y., J.E. and A.R.; formal analysis, F.B., M.Y. and A.R.; investigation, F.B. and M.Y.; resources, J.E., and A.R.; data curation, F.B., M.Y. and A.R.; writing— original draft preparation, F.B., M.Y. and A.R; writing—review and editing, A.R.; visualization, A.R.; supervision, A.R. and J.E.; project administration, A.R. and J.E.; funding acquisition, A.R. and J.E. All authors have read and agreed to the published version of the manuscript. Funding: This study was supported by grants to Anders Rasmussen from the Swedish Research Council (2020-01468), the Crafoord Foundation (20180704 & 20200729), the Segerfalk foundation (2019-2246), Åke-Wibergs foundation (M18-0070 & M19-0375), Fredrik & Ingrid Thurings foundation Sensors 2021, 21, 2966 10 of 11

(2018-00366 & 2019-00516), Pia Ståhls Foundation (20012), Magnus Bergvalls Foundation (2020-03788), and Anna-Lisa Rosenbergs foundation. Institutional Review Board Statement: The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the local ethical committee at Lund University (dnr: 2017/785). Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: Data and code is provided as supplementary material. Conflicts of Interest: The authors declare no conflict of interest.

References 1. Pettersen, V.; Westgaard, R.H. Muscle activity in professional classical singing: A study on muscles in the shoulder, neck and trunk. Logoped. Phoniatr. Vocol. 2004, 29, 56–65. [CrossRef][PubMed] 2. Yasuda, K.; Saichi, K.; Iwata, H. Haptic-based perception-empathy biofeedback enhances postural motor learning during high-cognitive load task in healthy older adults. Front. Med. 2018, 5, 149. [CrossRef][PubMed] 3. Rouhana, J.; Ferry, F.; Toussaint, L.; Boulinguez, P. Knowledge of results and explicit instruction: Efficiency of learning the crawl stroke in swimming. Percept. Mot. Skills 2002, 95, 895–896. [CrossRef] 4. Rasmussen, A.; Jirenhed, D.-A. Learning and timing of voluntary blink responses match eyeblink conditioning. Sci. Rep. 2017, 7, 3404. [CrossRef][PubMed] 5. Schoeller, F.; Bertrand, P.; Gerry, L.J.; Jain, A.; Horowitz, A.H.; Zenasni, F. Combining virtual reality and biofeedback to foster empathic abilities in humans. Front. Psychol. 2018, 9, 2741. [CrossRef][PubMed] 6. McMillan, K.; Flood, K.; Glaeser, R. Virtual reality, augmented reality, mixed reality, and the marine conservation movement. Aquat. Conserv. 2017, 27, 162–168. [CrossRef] 7. Niehorster, D.C.; Li, L.; Lappe, M. The Accuracy and precision of position and orientation tracking in the HTC vive virtual reality system for scientific research. Iperception 2017, 8, 2041669517708205. [CrossRef] 8. Borrego, A.; Latorre, J.; Alcañiz, M.; Llorens, R. Comparison of oculus rift and htc vive: Feasibility for virtual reality-based exploration, navigation, exergaming, and rehabilitation. Games Health J. 2018, 7, 151–156. [CrossRef] 9. Sloane, N.J.A.; Wyner, A.D. Scientific aspects of juggling. In Claude E. Shannon: Collected Papers; Wiley-IEEE Press: Hoboken, NJ, USA, 1993; pp. 850–864. [CrossRef] 10. Scholz, J.; Klein, M.C.; Behrens, T.E.J.; Johansen-Berg, H. Training induces changes in white-matter architecture. Nat. Neurosci. 2009, 12, 1370–1371. [CrossRef] 11. Beek, P.J.; Lewbel, A. The science of juggling. Sci. Am. 1995, 273, 92–97. [CrossRef] 12. Sánchez García, R.; Hayes, S.J.; Williams, A.M.; Bennett, S.J. Multisensory perception and action in 3-ball cascade juggling. J. Mot. Behav. 2013, 45, 29–36. [CrossRef][PubMed] 13. van Santvoord, A.A.M.; Beek, P.J. Phasing and the pickup of optical information in cascade juggling. Ecol. Psychol. 1994, 6, 239–263. [CrossRef] 14. Anglin, J.M.; Sugiyama, T.; Liew, S.-L. Visuomotor adaptation in head-mounted virtual reality versus conventional training. Sci. Rep. 2017, 7, 45469. [CrossRef][PubMed] 15. Anglin, J.; Saldana, D.; Schmiesing, A.; Liew, S. Transfer of a skilled motor learning task between virtual and conventional environments. In Proceedings of the 2017 IEEE Virtual Reality (VR), Los Angeles, CA, USA, 18–22 March 2017; pp. 401–402. 16. Adamovich, S.V.; Fluet, G.G.; Tunik, E.; Merians, A.S. Sensorimotor training in virtual reality: A review. NeuroRehabilitation 2009, 25, 29–44. [CrossRef] 17. Drew, S.A.; Awad, M.F.; Armendariz, J.A.; Gabay, B.; Lachica, I.J.; Hinkel-Lipsker, J.W. The trade-off of virtual reality training for dart throwing: A facilitation of perceptual-motor learning with a detriment to performance. Front. Sports Act. Living 2020, 2, 59. [CrossRef][PubMed] 18. Levac, D.E.; Huber, M.E.; Sternad, D. Learning and transfer of complex motor skills in virtual reality: A perspective review. J. Neuroeng. Rehabil. 2019, 16, 121. [CrossRef] 19. Nickl, R.W.; Ankarali, M.M.; Cowan, N.J. Complementary spatial and timing control in rhythmic arm movements. J. Neurophysiol. 2019, 121, 1543–1560. [CrossRef] 20. Quadrado, V.H.; da Silva, T.D.; Favero, F.M.; Tonks, J.; Massetti, T.; de Monteiro, C.B. Motor learning from virtual reality to natural environments in individuals with Duchenne muscular dystrophy. Disabil. Rehabil. Assist. Technol. 2019, 14, 12–20. [CrossRef] 21. Levac, D.E.; Sveistrup, H. Motor learning and virtual reality. In Virtual Reality for Physical and Motor Rehabilitation; Weiss, P.L., Keshner, E.A., Levin, M.F., Eds.; Springer: New York, NY, USA, 2014; pp. 25–46. ISBN 9781493909681. 22. Meyer, B.; Gruppe, P.; Cornelsen, B.; Stratmann, T.C.; Gruenefeld, U.; Boll, S. Juggling 4.0: Learning Complex motor skills with augmented reality through the example of juggling. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings; ACM: New York, NY, USA, 2018; pp. 54–56. 23. Hart, S.G. Nasa-Task Load Index (NASA-TLX); 20 years later. Proc. Hum. Fact. Ergon. Soc. Annu. Meet. 2006, 50, 904–908. [CrossRef] Sensors 2021, 21, 2966 11 of 11

24. Jost, T.A.; Drewelow, G.; Koziol, S.; Rylander, J. A quantitative method for evaluation of 6 degree of freedom virtual reality systems. J. Biomech. 2019, 97, 109379. [CrossRef][PubMed] 25. Spitzley, K.A.; Karduna, A.R. Feasibility of using a fully immersive virtual reality system for kinematic data collection. J. Biomech. 2019, 87, 172–176. [CrossRef][PubMed] 26. Krueger, C.; Tian, L. A comparison of the general linear mixed model and repeated measures ANOVA using a dataset with multiple missing data points. Biol. Res. Nurs. 2004, 6, 151–157. [CrossRef][PubMed] 27. Holden, M.K. Virtual environments for motor rehabilitation: Review. Cyberpsychol. Behav. 2005, 8, 187–211. [CrossRef][PubMed] 28. Therrien, A.S.; Wolpert, D.M.; Bastian, A.J. Increasing motor noise impairs reinforcement learning in healthy individuals. eNeuro 2018, 5.[CrossRef] 29. Therrien, A.S.; Wolpert, D.M.; Bastian, A.J. Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noise. Brain 2016, 139, 101–114. [CrossRef]