Touch-Based Interaction for Mobile Virtual Reality with Clip-On Lenses

applied sciences

Article “Blurry Touch Finger”: Touch-Based Interaction for Mobile Virtual Reality with Clip-on Lenses

Youngwon Ryan Kim 1, Suhan Park 2 and Gerard J. Kim 2,* 1 Korea Electronics Technology Institute, Gwangju 61011, Korea; [email protected] 2 Digital Experience Laboratory, Korea University, Seoul 02841, Korea; [email protected] * Correspondence: [email protected]

Received: 15 October 2020; Accepted: 5 November 2020; Published: 8 November 2020

Abstract: In this paper, we propose and explore a touch screen based interaction technique, called the “Blurry Touch Finger” for EasyVR, a mobile VR platform with non-isolating flip-on glasses that allows the fingers accessible to the screen. We demonstrate that, with the proposed technique, the user is able to accurately select virtual objects, seen under the lenses, directly with the fingers even though they are blurred and physically block the target object. This is possible owing to the binocular rivalry that renders the fingertips semi-transparent. We carried out a first stage basic evaluation assessing the object selection performance and general usability of Blurry Touch Finger. The study has revealed that, for objects with the screen space sizes greater than about 0.5 cm, the selection performance and usability of the Blurry Touch Finger, as applied in the EasyVR configuration, was comparable to or higher than those with both the conventional head-directed and hand/controller based ray-casting selection methods. However, for smaller sized objects, much below the size of the fingertip, the touch based selection was both less performing and usable due to the usual fat finger problem and difficulty in stereoscopic focus.

Keywords: virtual reality; object selection; pointing; ray casting; task performance; usability

1. Introduction Mobile virtual reality (M-VR) loosely refers to the immersive content platforms that are based on the smartphone [1]. M-VR in its typical form uses a head-set, into which the smartphone is inserted to lend its display (see Figure1, left). The head-set itself only provides the display optics, simple interaction device (e.g., button or touch pad) and world isolation [2,3]. M-VR is often contrasted with the higher-end (and priced) PC based VR (PC-VR) in which the display integrated head-set is separated from and tethered to a PC [4,5]. M-VR is generally regarded, aside from being mobile and self-contained, more convenient, inexpensive, and targeted for casual use. Nevertheless, M-VR in such a typical form is still not sufficient for a wide adoption by the public mass; the head-set is still somewhat bulky, interaction methods awkward and the ever-needed and must-be-handy smartphone becomes inaccessible during when used for VR (after the insertion). Recently, a new breed of M-VR, as called “EasyVR,” has appeared in the market in the form of flipping or clipping a cheap non-isolating magnifying lenses on the smartphone (see Figure1, right). Surprisingly, despite being open and the user’s peripheral view not shut from the outside world, such a display configuration still gives a good level of immersion with the magnified view covering much of the human’s field of view [6]. Given its much-improved convenience and the advantage of quick setting, it serves as a viable alternative configuration for M-VR (there is even a fold-out version built-into the smartphone case [7]). The open configuration also makes the immediate switch to the regular smartphone mode quite possible and vice versa (e.g., switch between VR and texting, see Figure2).

Appl. Sci. 2020, 10, 7920; doi:10.3390/app10217920 www.mdpi.com/journal/applsci

Article “Blurry Touch Finger”: Touch-Based Interaction for Mobile Virtual Reality with Clip-on Lenses

Youngwon Ryan Kim 1, Suhan Park 2 and Gerard J. Kim 2,*

1 Korea Electronics Technology Institute, Gwangju 61011, Korea; [email protected] 2 Digital Experience Laboratory, Korea University, Seoul 02841, Korea; [email protected] * Correspondence: [email protected]

Received: 15 October 2020; Accepted: 5 November 2020; Published: date

Abstract: In this paper, we propose and explore a touch screen based interaction technique, called the “Blurry Touch Finger” for EasyVR, a mobile VR platform with non-isolating flip-on glasses that allows the fingers accessible to the screen. We demonstrate that, with the proposed technique, the user is able to accurately select virtual objects, seen under the lenses, directly with the fingers even though they are blurred and physically block the target object. This is possible owing to the binocular rivalry that renders the fingertips semi-transparent. We carried out a first stage basic evaluation assessing the object selection performance and general usability of Blurry Touch Finger. The study has revealed that, for objects with the screen space sizes greater than about 0.5 cm, the selection performance and usability of the Blurry Touch Finger, as applied in the EasyVR configuration, was comparable to or higher than those with both the conventional head-directed and hand/controller based ray-casting selection methods. However, for smaller sized objects, much below the size of the fingertip, the touch based selection was both less performing and usable due to the usual fat finger problem and difficulty in stereoscopic focus.

Keywords: virtual reality; object selection; pointing; ray casting; task performance; usability

1. Introduction Mobile virtual reality (M-VR) loosely refers to the immersive content platforms that are based on the smartphone [1]. M-VR in its typical form uses a head-set, into which the smartphone is inserted to lend its display (see Figure 1, left). The head-set itself only provides the display optics, simple interaction device (e.g., button or touch pad) and world isolation [2,3]. M-VR is often contrasted with the higher-end (and priced) PC based VR (PC-VR) in which the display integrated head-set is separated from and tethered to a PC [4,5]. M-VR is generally regarded, aside from being mobile and self-contained, more convenient, inexpensive, and targeted for casual use. Nevertheless, M-VR in such a typical form is still not sufficient for a wide adoption by the public mass; the Appl. Sci. 2020, 10, x FOR PEER REVIEW 2 of 18 Appl.head-set Sci. 2020 is, 10still, 7920 somewhat bulky, interaction methods awkward and the ever-needed 2 ofand 18 must-be-handyFigure 1. Typical smartphone mobile virtualbecomes reality inaccessible with the smartpho during whenne inserted used andfor VRclosed (after into the an inexpensiveinsertion). headset (left), and “EasyVR” in which light magnifying lenses are clipped on the smartphone in a non-isolating way.

Recently, a new breed of M-VR, as called “EasyVR,” has appeared in the market in the form of flipping or clipping a cheap non-isolating magnifying lenses on the smartphone (see Figure 1, right). Surprisingly, despite being open and the user’s peripheral view not shut from the outside world, such a display configuration still gives a good level of immersion with the magnified view covering much of the human’s field of view [6]. Given its much-improved convenience and the advantage of quick setting, it serves as a viable alternative configuration for M-VR (there is even a fold-out version built-intoFigure the 1. Typical smartphone mobile virtualcase [7]). reality The with open the conf smartphoneiguration inserted also makes and closed the immediate into an inexpensive switch to the Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci regularheadset smartphone (left), and mode “EasyVR” quite in poss whichible light and magnifying vice versa lenses (e.g., are switch clipped between on the smartphoneVR and texting, in a see Figurenon-isolating 2). way.

Figure 2. Main features of “EasyVR”: (1) quick put-on, (2) no inconvenient mounting on the head, Figure 2. Main features of “EasyVR”: (1) quick put-on, (2) no inconvenient mounting on the head, (3) seamless switch between regular smartphone and VR modes, (4) touch screen based interaction (3) seamless switch between regular smartphone and VR modes, (4) touch screen based interaction possible (and no additional interaction devices—self-contained). possible (and no additional interaction devices—self-contained). One of the most important elements that makes “EasyVR” easy is perhaps the possibility of using the touchOne screenof the for most interaction. important Conventional elements that M-VR make set-upss “EasyVR” have o ffeasyered is the pe head-directionrhaps the possibility (or gaze) of andusing touch the touch pad (or screen button) for oninteraction. the side ofConvention the head-setal M-VR for the set-ups lack ofhave any offered better method.the head-direction The use of(or the gaze) touch and screen touch in pad EasyVR (or button) brings on about: the side (1) of the the usual head-set and familiarfor the lack touch of basedany better user method. experience The (includinguse of the the touch swipe screen and flick),in EasyVR (2) natural brings bimanual about: (1) interaction, the usual and and thereby, familiar (3) touch more directbased anduser eexperiencefficient interaction, (including and the (4) swipe no need and for flick), an additional (2) natural interaction bimanual device. interaction, and thereby, (3) more directEven and though efficient the interaction, open set-up and allows (4) no theneed fingers for an to additional touch and interaction manipulate device. objects on the touch screen,Even they though are seen the through open set-up the magnifying allows the lens. fingers While to touch the virtual and manipulate objects will objects be seen on in the focus touch by design,screen, thethey fingers are seen are through not and the seen magnifying blurry by lens. the binocular While the rivalry. virtual Interestingly,objects will be this seen very in focus feature by makesdesign, the the target fingers object are not visible and andseen thereby blurry by the the touch binocular screen rivalry. interaction Interestingly, feasible. Thisthis very also partlyfeature solvesmakes the the so target called object “fat finger” visible problem and thereby in touch the based touc smartphoneh screen interaction interaction feasible. (see Section This 3also for morepartly details)solves [the8]. so called “fat finger” problem in touch based smartphone interaction (see Section 3 for moreIn details) this paper, [8]. we explore this idea of the touch screen based interaction for EasyVR, called the “BlurryIn Touchthis paper, Finger we (BTF)” explore (see this Figure idea 3of). the We to carrieduch screen out the based first interaction stage basic for usability EasyVR, evaluation called the to“Blurry validate Touch the proposed Finger (BTF)” idea: (see whether Figure the 3). EasyVR We carried user out can the in factfirst select stage virtualbasic usability objects accuratelyevaluation usingto validate BTF, compared the proposed to e.g., idea: the whet conventionalher the EasyVR interaction user can techniques. in fact select The virtual evaluation objects through accurately the performanceusing BTF, compared of object selection, to e.g., the which conventional is one of the interaction most important techniques. and basicThe evaluation interactive VRthrough task [9the], wouldperformance provide of a goodobject yardstick selection, for which the practicality is one of the and most feasibility important of EasyVR and basic and interactive the associated VR “Blurrytask [9], Touchwould Finger” provide method. a good yardstick for the practicality and feasibility of EasyVR and the associated “Blurry Touch Finger” method. Appl. Sci. 2020, 10, 7920 3 of 18 Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 of 18

FigureFigure 3. 3. DirectlyDirectly selecting selecting a avirtual virtual object object with with the the fing fingerer on on the the touch touch screen. screen. The The touching touching finger finger is blurred,is blurred, but but because because of ofit, it,the the target target object object behind behind it itis is visible, visible, and and reasonably reasonably accurate accurate selection selection is is possible.possible. Both Both two/one two/one hand/finger hand/finger interactions interactions are possible.

InIn the the following following sections, sections, we we first ﬁrst review review other other related related research research work work and and describe describe the the details details of of thethe BTF BTF method. method. Then Then we we present present the the usability usability expe experimentriment and and its its results. results. The The paper paper is is concluded concluded withwith a a brief brief summary, summary, discussion and directions for future work.work.

2.2. Related Related Work Work ObjectObject selection selection in in the the 3D 3D virtual virtual space space has has been studied extensively [10,11]. [10,11]. While While a a variety variety of of methodsmethods exist, exist, virtual virtual ray ray casting casting [9] [9] is is one one of of the the most most common common and and popular popular approaches, approaches, in in which which a a 3D3D (virtual) rayray emanatesemanates from from the the hand hand (whose (whose position position is tracked is tracked by a sensor),by a sensor), and the and user the “shoots” user “shoots”for the object for the of interest.object of This interest. method This is quitemethod intuitive is quite in intuitive the sense in that the it sense is a natural that it abstraction is a natural of abstractionthe real-world of the finger real-world pointing. finger A slight pointing. variant A is slight the virtual variant flashlight is the virtual in which flashlight a volumetric in which cone isa volumetricformed with cone the is apex formed at the with hand the toward apex at the the target hand object toward [12 ].the Several target diobjectfferent [12]. forms Several of ray different casting formsmethods of existray dependingcasting methods on how exist the origin depending and direction on how of thethe rayorigin is determined and direction (and thusof the where ray theis determined“cursor” is situated,(and thus e.g., where using the the “cursor” hand positionis situated,/orientation e.g., using (hand the hand directed), position/orientation hand orientation (hand and directed),head position hand (head-hand orientation directed), and head headposition position (head-hand/orientation directed), (head head directed) position/orientation [13]. (head directed)To apply [13]. the virtual ray casting or flashlight techniques, the user would need to either employ a handTo apply tracking the sensing virtual ray device casting that or would flashlight require techniques, a cumbersome the user process would ofneed calibrating to either itsemploy spatial a handcoordinates tracking with sensing that ofdevice the view that point,would or require recognize a cumbersome and track the process hand of from calibrating the head its mounted spatial coordinatescamera or sensor with that which of wouldthe view restrict point, the or hand recognize position and to track stay withinthe hand the from sensing the volumehead mounted thereby cameramaking or the sensor ray/cone which casting would di restfficult.rict the In hand the context position of to EasyVR, stay within the usethe ofsensing hand-worn volume sensor thereby or makingcontroller the (a ray/cone separate module),casting difficult. would be In rather the co prohibitiventext of EasyVR, and go againstthe use its of very hand-worn spirit and sensor purpose or controller(i.e., self-containment). (a separate module), would be rather prohibitive and go against its very spirit and purposeThus, (i.e., another self-containment). possibility is to directly sense (from the self-contained EasyVR unit) the hand movementThus, another (and its gestures)possibility using is to computer directly sense vision (from techniques the self-contained or employing EasyVR environment unit) capturethe hand or movementdepth sensors (and from its gestures) the self- contained using computer VR unit. vision Small techniques depth cameras or employing (mounted environment on the head mounted capture ordisplay depth or sensors on body) from have the been self- used contained to employ VR 3Dunit. hand Small gesture depth based cameras interaction (mounted [14–21 on]. the head mountedWhile display hand basedor on body) pointing have of been remote used objects to em byploy the 3D ray hand casting gesture is natural based and interaction familiar, [14–21]. it can be difficultWhile due hand to the based lack pointing of sufficient of remote proprioceptive objects by feedback the ray casting [22–26] is and natural incur and significant familiar, fatigue it can be on difficultthe wrist due by theto the off setlack between of sufficient the target proprioceptive object/cursor feedback and the [22–26] hand [and27]. incur significant fatigue on the wristOn theby the other offset hand, between current the M-VR target invariably object/cursor employs and the the hand head-directed [27]. (gaze-based) cursor to selectOn an the object other with hand, the time-basedcurrent M-VR or side-button invariably drivenemploys final the confirmation. head-directed This (gaze-based) is primarily cursor because to selectthe use an of object a separate with interactionthe time-based controller or side-bu is discouragedtton driven at final first toconfirmation. make M-VR This as self-contained is primarily becauseas possible. the Head-directeduse of a separate target interaction designation controller is problematic is discouraged because theat first tasks to of make previewing M-VR andas self-containedexploring the worldas possible. and particular Head-directed target object target selection designation are coupled. is problematic For instance, because view-fixed the tasks menu of previewingsystem cannot and be exploring used if the the head-directed world and particular cursor is used.target Head-directed object selection (or are gaze coupled. based) For interaction instance, is view-fixedeven more problematicmenu system with cannot navigation be used control if the head-directed (although this papercursor doesis used. not dealHead-directed with it yet) (or because gaze based) interaction is even more problematic with navigation control (although this paper does not deal with it yet) because the user cannot freely view the world as the moving direction is determined Appl. Sci. 2020, 10, 7920 4 of 18 the user cannot freely view the world as the moving direction is determined by the gaze direction [28]. Using the head (vs. hand) is more tiring as well [28]. Time-based confirmation is slow, and the use of the side button is less direct. Companies have realized the limitation and have recently introduced separate controllers [29]. In this work, we explore the touch based method, because (1) hand tracking with a single camera is still challenging and prone to errors and latency, (2) depth sensors are still not a common feature on the smartphones yet and (3) the touch based method allows the use of two grabbing hands. Pierce et al. suggested the “Sticky Finger” as one of their virtual image plane interaction techniques similar to the Blurry Touch Finger, for selecting and manipulating an object in virtual environments [27]. With the Sticky Finger, the user interacts with the 2D projections that 3D objects in the scene make on one’s image plane. Thus, in a monoscopic environment, the method essentially a 2D interaction technique, as the selection is made on the image plane, as also categorized as such by [30]. When using a virtual ray, the “cursor” position (drawn by projecting the user’s fingertip position orthographically) is the point on the image plane collided with the ray [31]. Blurry Touch Finger works similarly except that the selection is made over the “real” image screen space (i.e., touch screen), under the magnifying glass with stereoscopy. Even though the target object is far, it seems as if attached to the finger, and leverages well on the already familiar touch interaction with the smartphone. There have been few proposals to use the back of the smartphone as the interaction space for M-VR [32,33]. Setting aside the need to install another touch sensor on the back of the M-VR unit, the method of selecting from the “back” is not intuitive especially when the selection is attempted among crowded objects with an occluded view (as seen from the front).

3. Blurry Touch Finger Figure3 illustrates the concept of Blurry Touch Finger (BTF). First, we assume an “open” EasyVR set-up with a magnifying glass clipped on the smartphone so that the hands/fingers are used to both hold the unit and touch the screen (see Figure3). Even though the open set-up allows the fingers to touch and manipulate objects on the touch screen, they are seen through the magnifying lens. By design, while the virtual objects will be seen in focus, fingers will be blurred and seen in double vision. Nevertheless, the user is still able to point to and touch the target object near-correctly, similarly to the case of point shooting [34]. Moreover, the target object is visible as if the blurred fingers are transparent due to the binocular rivalry. Binocular rivalry refers to phenomenon of visual perception in which perception alternates between different images presented to each eye [35]. That is, the finger is seen by only one eye (due to the vignette) and the alternative views (one containing the finger and the other not) make the finger effectively semi-transparent, making the imagery behind it completely visible. A similar concept has been applied for real environment object selection in an augmented reality setting [11]. BTF partly solves the so called “fat finger” problem which touch based interaction suffers from for selecting small objects due to the occlusion by the touching finger [8]. Both one-handed and two-handed interactions are possible. For the former, one hand will be used solely for holding the device, while for the latter, both hands are holding the device with fingers extended out to touch the left or right screen. Note that when both hands/fingers are used, the left or right finger is used one at a time (not simultaneously) to select different objects Figure4 illustrates the workings of the Blurry Touch Finger. The touch screen is magnified into a larger virtual imagery by the lenses, and the 3D target object (red) is seen in focus, fused for 3D depth perception, in the direction of the purple dotted line (emanating from the mid- point between the two eyes (Cyclopean eye) by a phenomenon called the Allelotropia [36]). Even though this seems to be the line direction with which the user consciously will try to point to the target object and touch the corresponding position on the touch screen, our observation has found that the user naturally makes the touch input on where the target object is rendered on the touchscreen (where the red dotted lines cross the touchscreen either on the left or right, the purple circles in Figure4). Since the left and right images Appl. Sci. 2020, 10, 7920 5 of 18

Appl. Sci. 2020, 10, x FOR PEER REVIEW 5 of 18 are assumed to be fused and not visible separately, this ability seems to be very surprising. Currently, wesurprising. do not have Currently, a clear physiologicalwe do not have explanation a clearto physiological this observation. explanation Nevertheless to this the observation. experiment inNevertheless this paper demonstratesthe experiment that in this such paper is the demonstrat case and highlyes that accuratesuch is the (in case par and with highly controller-based accurate (in ray-castingpar with controller-based method) selection ray-casting is possible. method) selection is possible.

FigureFigure 4.4. Working ofof thethe BlurryBlurry TouchTouch Finger.Finger. TheThe actualactual touchtouch isis mademade notnot alongalong thethe perceivedperceived viewview directiondirection ofof 3D3D targettarget objectobject fromfrom thethe CyclopeanCyclopean eye,eye, butbut inin thethe directiondirection fromfrom thethe respectiverespective eyeeye dependingdepending onon wherewhere thethe useruser touchestouches (on(on thethe leftleft oror rightright screenscreenspace spaceimage). image).

NoteNote thatthat thethe redred dotteddotted lineslines areare notnot exactlyexactly aa straightstraight lineline duedue toto thethe refractionrefraction byby thethe lenseslenses (but(but treatedtreated straightstraight forfor nownow forfor simplicitysimplicity andand thethe amountamount ofof refractionrefraction cancan bebe computed).computed). That is, thethe useruser cancan touchtouch eithereither thethe rightright oror leftleft screenscreen toto selectselect thethe fusedfused targettarget objectobject (when(when anan objectobject onlyonly appearsappears inin eithereither thethe rightright oror leftleft imagery,imagery, thethe useruser willwill makemake thethe touchtouch inputinput inin thethe respectiverespective sideside only).only). Therefore,Therefore, a a virtual virtual ray ray emanating emanating from from the the respective respective eye (righteye (right eye if eye right if halfright of half touch of screentouch isscreen used is and used vice and versa) vice isversa) used is to used intersect to intersect the target the object target for object selection. for selection.

4.4. Pilot Experiment: BTF-R vs. Head-Directed Selection Experiment Design Experiment Design Thus, we first ran a small pilot study, comparing one form of BTF, namely BTF-R to the head Thus, we first ran a small pilot study, comparing one form of BTF, namely BTF-R to the head directed method. The BTF is in its bare form, making the object selection upon the first touch (BTF-T). directed method. The BTF is in its bare form, making the object selection upon the first touch The other variant BTF makes the selection at the release of the finger, after the user makes the initial (BTF-T). The other variant BTF makes the selection at the release of the finger, after the user makes touch and adjusts the touch finger position for a more accurate final selection (BTF-R). the initial touch and adjusts the touch finger position for a more accurate final selection (BTF-R). We only shortly describe the pilot experiment and its results (almost the same experimental We only shortly describe the pilot experiment and its results (almost the same experimental procedure was used in the main experiment as described in the following sections). The user was procedure was used in the main experiment as described in the following sections). The user was presented with a sequence of 12 objects in one nominal size (about 0.4 0.4 cm—48 pixels in the screen presented with a sequence of 12 objects in one nominal size (about× 0.4 × 0.4 cm—48 pixels in the space as recommended by the Android user interface guideline [37]) that appeared in a balanced order screen space as recommended by the Android user interface guideline [37]) that appeared in a on the 4 3 grid positions. At each appearance of the object, ten subjects were asked to select the object balanced× order on the 4 × 3 grid positions. At each appearance of the object, ten subjects were asked as fast and accurately as possible, using two methods, (1) BTF-R and (2) head-directed. To eliminate any to select the object as fast and accurately as possible, using two methods, (1) BTF-R and (2) possible biases, both methods were tested using an open configuration, that is, with the flip-on lenses head-directed. To eliminate any possible biases, both methods were tested using an open and the final confirmation (for Head-directed) was done using a simple touch on the openly accessible configuration, that is, with the flip-on lenses and the final confirmation (for Head-directed) was done touchscreen (vs. time-based or using a side button). The object selection time, error (the distance using a simple touch on the openly accessible touchscreen (vs. time-based or using a side button). between the finger/cursor and target object), and general usability (through a survey) were measured The object selection time, error (the distance between the finger/cursor and target object), and as the dependent variables. general usability (through a survey) were measured as the dependent variables. The results in Figure 5 show that the Head-directed method did incur higher error (with statistical significance) and task completion time. The Cohen’s d-value for the distance error (BTF-R Appl. Sci. 2020, 10, 7920 6 of 18

Appl. Sci.The 2020 results, 10, x FOR in Figure PEER 5REVIEW show that the Head-directed method did incur higher error (with statistical6 of 18 significance) and task completion time. The Cohen’s d-value for the distance error (BTF-R vs. vs.Head-directed) Head-directed) was was approximately approximately 1.26 1.26 (i.e., (i.e., large large e ffeffect)ect) and and for for thethe selectionselection time,time, the value was was aroundaround 0.27 0.27 (small (small to to moderate moderate effect). effect).

FigureFigure 5. 5. SelectionSelection performance performance (above) (above) and and usability usability (below) (below) between between BTF-R BTF-R and and head-directed head-directed methods.methods. Note thatthat higher higher score score means means more more positive positive toward toward usability usability for the categoriesfor the categories of preference of preferenceand ease ofand use, ease and of vice use, versaand vice for mentalversa for and me physicalntal and demand.physical demand. The asterisk The markasterisk (*) mark indicates (*) indicatesstatistically statistically signiﬁcant significant diﬀerence. difference.

TheThe usability usability survey survey gave gave mixed mixed results: results: e.g., e.g., the the Head-directed Head-directed felt felt to to be be easier easier to to use, use, yet yet BTF BTF waswas preferred preferred and and generally generally had had less mental and ph physicalysical demand. Note Note that that the the pilot pilot test test was was conductedconducted not not using using the the much much more more inconvenient inconvenient cl closedosed head-set. head-set. Thus, Thus, at at the the least, least, considering considering the the aforementionedaforementioned general general disadvantages disadvantages of of the the Head-directed Head-directed method method (such (such as as the the coupling coupling between between generalgeneral viewing andand selection), selection), we we claim claim that that BTF-R BTF-R is a is viable a viable selection selection method method for EasyVR, for EasyVR, and worthy and worthyof comparison of comparison to the controller-based to the controller-based ray casting ray method.casting method.

5.5. Main Main Experiment: Experiment: BTF-T, BTF-T, BTF-R BTF-R and and CONT CONT TheThe purpose purpose of of the the main main experiment experiment is is to to validat validatee the the idea idea of of the the Blurry Blurry Touch Touch Finger Finger at at the the rudimentaryrudimentary level level first, ﬁrst, particularly particularly to to the the case case of of virtual virtual ray ray casting casting with with a a hand-gripped hand-gripped controller, controller, asas this this is is arguably arguably the the most most popular popular and and stable stable selection selection method method today today (but (but on on PC-VR), PC-VR), and and thus thus servesserves as aa goodgood reference reference for for the the level level of performance of performance [38]. [38]. On the On other the hand,other head-directedhand, head-directed method methodis the most is the prevalent most prevalent way of object way selectionof object onsele M-VR,ction evenon M-VR, though even we highlightedthough we highlighted its well-known its well-knowndisadvantages disadvantages (see Section 2(see). Section 2).

5.1. Experimental Design In this main experiment, we compared the user performances of two styles of the Blurry Touch Finger selection on the EasyVR set-up with regards to the state of the art, controller based ray casting method on the PC-VR as the conventional case (CONT). This would highlight, if any, the potential of BTF (and EasyVR) across different state of the art VR platforms. The main experiment was designed with two factors (3 selection methods—BTF-T, BTF-R and CONT and 3 object sizes—Small, Medium and Large) within subject repeated measure. We hypothesized that BTF-R would show equal or better performance/usability as compared to CONT. BTF-T is expected to be insufficient for general usage due to its nature to rely solely on the first touch, thus producing significant selection errors due to personal differences (e.g., in one’s dominant eye, stereoscopic fusion) and modelling error (e.g., assumed eye position). BTF-R should compensate for them by the additional drag-to-release. Note that EasyVR with a separate controller was not tested because (1) EasyVR pursues self-containment as a convenient mobile VR platform without any extra devices required other than the smart phone itself (and attached foldable lenses), and (2) holding the smartphone in one hand and controller in the other would be very difficult and unrealistic.

5.2. Experimental Task In the main experiment, the user was similarly presented with a sequence of 12 objects (all in one of three sizes) that appeared randomly on the 4 3 grid positions. At each appearance of the object, × the subject was asked to select the object as fast and accurately as possible (but with more emphasis on speed to ensure minimum level accuracy). Without such an instruction (e.g., just as fast as possible), we were concerned subjects would just disregard the accuracy aspect. A single selection task was finished upon the initial touch for BTF-T, upon release for BTF-R and upon the trigger button press by the index finger for the CONT. A sufficiently large fixed time-out value was given for the user to select all given objects (although no subjects failed). Selection of distant 3D objects with three different sizes (screen space size: Small—0.4 cm, Medium—0.5 cm, and Large—0.6 cm) was tested, all situated at a fixed distance (7.3 virtual space unit away from the view point) at 12 different locations laid out in a 4 3 grid. × Object selection time (between target object appearance and confirmation events), accuracy (assessed in two ways: the exact planar distance on the screen space to the intended target and number of successful selections) and general usability were measured as the dependent variables. An unsuccessful selection means that the center position of the touch input was not contained within the target object.

5.3. Experimental Set-Up In the experiment, for the two BTFs, we used the Samsung Galaxy S7 smartphone (152 g, screen size: 5.1 inches, combined screen resolution: 1440 2560 pixels [39]). with the Homido-mini VR magnifying × glasses (22 g, lens-to-screen distance: 70 mm [40]), giving it a virtual image of approximately 85 degree FOV. As for the CONT, the HTC Vive HMD (555 g, screen resolution: 2160 1200 or 1080 1200 per × × eye, FOV: 110 degree) and the two pose-tracking controllers were used (whose experimental content ran on the connected PC) [4]. A virtual ray (emanating from the respective eye, e.g., the right eye when right half of the touch screen is touched and vice versa) and a circular cursor were drawn at the end of the ray in stereo. The virtual ray was provided so that the interface looked the same as in the popular controller based ray casting methods (e.g., Tilt Brush/HTC Vive [4]) and give a better understanding of the 3D environment. For this experiment, the eye dominance was asked of but not put into consideration in terms of adjusting the direction of the virtual ray. The cursor and the ray guides the user make the selection, but for the BTF-T, it is of not much a use other than for just providing a confirmation of the selection, Appl. Sci. 2020, 10, 7920 8 of 18 since the selection is made right upon the direct initial touch. Note that the ray/cursor only appears only upon touch in the cases of BTF’s (e.g., only one ray/cursor when one finger is touched, and two when both fingers touch the screen), and two rays/cursors always appear in the case of CONT that uses two active controllers (see Figure6). Appl.Appl. Sci.Sci. 20202020,, 1010,, xx FORFOR PEERPEER REVIEWREVIEW 88 ofof 1818

FigureFigure 6.6. TheThe ray/cursors(s) rayray/cursors(s)/cursors(s) appearance when using the BTF’s BTF’s and and CONT CONT as as a a guideguide forfor thethe selection.selection. AsAs forfor BTF-T,BTF-T, itit appearsappears onlyonly afterafter thethe touchtouch isis made,mamade,de, andand forfor BTF-R,BTF-R, itit remains remains whilewhile the the finger ﬁngerfinger staysstays touchedtouched untiluntil released.released. The The finger ﬁngerfinger is is not not shown shown forfor clarity.clarity.

ForFor thethethe two twotwo BTFs, BTFs,BTFs, the thethe smartphone smartphonesmartphone based basedbased M-VR M-VRM-VR units wereunitsunits held werewere by heldheld both byby hands bothboth and handshands touch andand interacted touchtouch withinteractedinteracted the two withwith thumbs, thethe twotwo while thumbs,thumbs, for the whilewhile baseline forfor thethe case, baselinebaseline two controllerscase,case, twotwo controllerscontrollers in each held inin each ineach the heldheld left in andin thethe right leftleft andhandsand rightright were handshands used werewere (see Figureusedused (see(see7). FigureFigure 7).7).

FigureFigure 7.7.7. TheTheThe experimental experimentalexperimental set-up: set-up:set-up: (1) (1)(1) for forfor the thethe BTF-T BTF-TBTF-T and andand BTF-R BTF-RBTF-R using usingusing a smartphone aa smartphonesmartphone based based EasybasedM-VR EasyEasy (M-VRM-VRleft) and ((leftleft (2))) andand CONT (2)(2) CONTCONT using the usingusing PC thethe based PCPC VR basedbased with VRVR the withwith HTC thethe Vive HTCHTC HMD ViveVive/controllers HMD/controllersHMD/controllers (right). ((rightright).).

ForFor now, now, thethe calibrationcalibration betweenbetween thethe realreal andand virtualvirtual space,space, e.g.,e.g., betweenbetween the the touchtouch screenscreen andand virtualvirtual imageimage planeplane (in(in thethe casecase ofof BTF’s)BTF’s) waswas donedone byby aa simplesimple linearlinear scalingscaling transformation.transformation. As As for for thethe 3D 3D controllers controllers (CONT), (CONT), the the calibration calibration was was not not necessary necessary since since the the system system already already operated operated with with respectrespect toto thethe head-sethead-set (default(default viewview position).position).

5.4.5.4. Experimental Experimental Procedure Procedure TenTen paid paidpaid subjects subjectssubjects (eight (eight(eight men menmen and andand two twotwo women womenwomen between betweenbetween the thethe ages agesages of 23 ofof and 2323 and 29,and mean 29,29, meanmean= 27.9 ==/ SD27.9/SD27.9/SD= 5.36) == participated5.36)5.36) participatedparticipated in the inin experiment. thethe experiment.experiment. After collectingAfterAfter collectingcollecting their basic theirtheir background basicbasic backgroundbackground information, information,information, the subjects thethe weresubjectssubjects briefed werewere about briefedbriefed the purposeaboutabout thethe of the purposepurpose experiment ofof thth andee experimentexperiment given instructions andand givengiven for the instructionsinstructions experimental forfor tasks. thethe Asexperimentalexperimental the subjects tasks.tasks. were AsAs the thethe same subjectssubjects from the werewere pilot thethe experiment, sasameme fromfrom they thethe werepilotpilot alreadyexperiment,experiment, familiar theythey with werewere the alreadyalready BTF-R, familiarthusfamiliar an additional withwith thethe BTF-R,BTF-R, short training thusthus anan was additionaladditional given to shortshort allow trainingtraining the subjects waswas togivengiven become toto allowallow familiar thethe withsubjectssubjects the to VRto becomebecome set-up, familiartwofamiliar other withwith compared thethe VRVR methods set-up,set-up, (BTFtwotwo other andother controller comparedcompared based) methodsmethods and the(BTF(BTF experimental andand controllercontroller process. based)based) All and subjectsand thethe wereexperimentalexperimental right handed, process.process. had AllAll long subjectssubjects experiences werewere rightright in using hanhan theded,ded, smartphone hadhad longlong experiencesexperiences some experiences inin usingusing with thethe thesmartphonesmartphone M-VR or somePC-VR.some experiencesexperiences Throughout withwith the thethe practice M-VRM-VR sessions, oror PC-VR.PC-VR. the subjectsThrougThroughouthout mostly thethe tried practicepractice and adjustedsessions,sessions, thethethe way subjectssubjects they mostlymostly would triedtouchtried andand and adjustedadjusted select the thethe object wayway for theythey higher wouldwould accuracy. touchtouch andand selectselect thethe objectobject forfor higherhigher accuracy.accuracy. TheThe subjectssubjects satsat andand heldheld thethe M-VRM-VR unitunit (smartph(smartphoneone ++ clip-onclip-on glass)glass) withwith twotwo hands,hands, oror worewore (strapped(strapped toto thethe head)head) thethe HMDHMD wiwithth controllerscontrollers inin eacheach hand,hand, andand lookedlooked intointo thethe experimentalexperimental contentcontent comfortably.comfortably. TheThe exactexact viviewew positionposition waswas adjustedadjusted individuallyindividually soso thatthat thethe experimentalexperimental contentcontent couldcould bebe fusedfused inin stereostereo forfor thethe properproper depdepthth perception.perception. TheThe studystudy includedincluded ninenine differentdifferent treatmentstreatments (or(or conditions),conditions), namely,namely, threethree interfacesinterfaces ×× threethree objectobject sizes;sizes; inin eacheach treatmenttreatment condition,condition, 1212 differentdifferent targettarget objectsobjects laidlaid inin thethe 44 ×× 33 gridgrid popositionssitions werewere toto bebe selected,selected, comprisingcomprising ofof oneone trialtrial block,block, andand blockblock trialstrials repeatedrepeated 1212 timestimes (a(a totaltotal ofof 33 ×× 33 ×× 1212 ×× 1212 == 12961296 objectobject selections).selections). Appl. Sci. 2020, 10, 7920 9 of 18

The subjects sat and held the M-VR unit (smartphone + clip-on glass) with two hands, or wore (strapped to the head) the HMD with controllers in each hand, and looked into the experimental content comfortably. The exact view position was adjusted individually so that the experimental content could be fused in stereo for the proper depth perception. The study included nine different treatments (or conditions), namely, three interfaces three object sizes; in each treatment condition, × 12 different target objects laid in the 4 3 grid positions were to be selected, comprising of one trial × block, and block trials repeated 12 times (a total of 3 3 12 12 = 1296 object selections). × × × Each treatment was presented in a balanced fashion using the Latin-square methodology per subject. After the whole sessions, the subjects were asked to fill out a usability questionnaire (shown in Table1). We paraphrased the de facto standard usability survey questions from works like the NASA TLX [41] and SUS [42] to fit our purpose. It included standard categories such as ease of use, learnability, fatigue, and general satisfaction. The experiment took about one hour per subject.

Table 1. The general usability survey question.

NO./CATEGORY QUESTIONS How easy you felt the interface to be: Q1: Ease of use (1: Very diﬃcult—7: Very easy) How natural and intuitive you felt the interface to be: Q2: Naturalness (1: Very contrived—7: Very natural) How easy it was for you to learn the interface: Q3: Ease of learning (1: Very diﬃcult—7: Very easy) Rate the overall level of satisfaction of using the interface: Q4: General Satisfaction (1: Very unsatisfying—7: Very satisfying) How fatigued you were after using the interface: Q5: Fatigue (1: Very fatigued—7: Very fatigued at all) Rate how fast you felt you could accomplish the task: Q6: Speed (1: Very slowly—7: Very quickly) Rate how accurate you felt you could accomplish the task: Q7: Accuracy (1: Very inaccurately—7: Very accurately)

The standard two-way ANOVA was conducted that examined the eﬀect of the two factors (1) interface type (BTF-F, BTF-R, CONT) and (2) object size (large, medium, small) on user performances (selection time, distance error, number of unsuccessful selection—numerical data). The Tukey’s Student Test was used for the pairwise comparisons. For the usability analysis, the Mann-Whitney U Test was used since the questionnaires were rather subjective in nature although scored in the 7 level Likert scale.

5.5. Results and Discussion

5.5.1. Interface Type The analysis showed a signiﬁcant main eﬀect for the interface type (Figure8) to all three user performance categories (selection time: [F(2,81) = 8.417, p < 0.001 *], distance error: [F(2,81) = 147, p < 0.001, number of unsuccessful selections: [F(2,81) = 139.99, p < 0.001 *]). Appl. Sci. 2020, 10, 7920 10 of 18 Appl. Sci. 2020, 10, x FOR PEER REVIEW 10 of 18

FigureFigure 8.8. Quantitative performance measures (average selection time, average distancedistance errorerror andand averageaverage numbernumber of of unsuccessful unsuccessful selections selections in percentage)in percentage) among among the three the interfacethree interface types. Thetypes. asterisk The markasterisk (*) mark indicates (*) indicates statistically statistically signiﬁcant significant diﬀerence. difference.

UsingUsing thethe postpost hochoc Tukey’sTukey’s PairwisePairwise Comparisons,Comparisons, both BTF-TBTF-T andand BTF-RBTF-R showedshowed fasterfaster performanceperformance than the CONT due due to to its its directness directness (BTF-T (BTF-T vs. vs. BTF-R: BTF-R: p =p 0.991;= 0.991; BTF-T BTF-T vs. vs.CONT: CONT: p = p0.002= 0.002 *; BTF-R *; BTF-R vs. CONT: vs. CONT: p = 0.001p = 0.001 *). *). As for the distance error, BTF-R and CONT showed a similar level of accuracy with significant difference to the much lower that of BTF-T (BTF-T vs. BTF-R: p < 0.001 *; BTF-T vs. CONT: p < 0.001 *; BTF-R vs. CONT: p = 0.092). Appl. Sci. 2020, 10, 7920 11 of 18

As for the distance error, BTF-R and CONT showed a similar level of accuracy with significant difference to the much lower that of BTF-T (BTF-T vs. BTF-R: p < 0.001 *; BTF-T vs. CONT: p < 0.001 *; BTF-RAppl. vs. Sci. CONT: 2020, 10,p x =FOR0.092). PEER REVIEW 11 of 18 For the number of unsuccessful object selection, BTF-R showed the best performance among For the number of unsuccessful object selection, BTF-R showed the best performance among the the three (but statistically no difference from CONT). Similarly with the distance error, BTF-T clearly three (but statistically no difference from CONT). Similarly with the distance error, BTF-T clearly showed an unusable level of object selection performance (BTF-T vs. BTF-R: p < 0.001 *; BTF-T vs. showed an unusable level of object selection performance (BTF-T vs. BTF-R: p < 0.001 *; BTF-T vs. CONT:CONT:p < 0.001 p < 0.001 *; BTF-R *; BTF-R vs. vs. CONT: CONT:p p= = 0.946).0.946).

5.5.2.5.5.2. Object Object Size Size ANOVAANOVA showed showed a signiﬁcant a significant main main eeffectﬀect for target object object size size (Figure (Figure 9) 9for) for all all(selection (selection time: time: [F(2,81)[F(2,81)= 4.894, = 4.894,p = p0.010 = 0.010 *], *], distance distance error: error: [F(2,81)[F(2,81) == 5.939,5.939, p p= =0.0040.004 *, number *, number of unsuccessful of unsuccessful selections:selections: [F(2,81) [F(2,81)= 103.10, = 103.10,p

FigureFigure 9. Quantitative 9. Quantitative performance performance measures measures (average selection selection time, time, average average distance distance error error and and averageaverage number number of unsuccessful of unsuccessful selections selections in percentage) in percentage) among among the threethe three target target object object sizes. sizes. The The asterisk markasterisk (*) indicates mark (*) statistically indicates statistically signiﬁcant significant diﬀerence. difference. Appl.Appl. Sci. Sci. 20202020, ,1010, ,x 7920 FOR PEER REVIEW 1212 of of 18 18

Using the post hoc Tukey’s Pairwise Comparisons, the selection were significantly faster with large Usingtargets the (Large post vs. hoc Small: Tukey’s p =Pairwise 0.008 *). Comparisons,Yet, the error distance the selection when were selecting significantly large targets faster were with significantlylarge targets higher (Large than vs. Small:when selectingp = 0.008 small *). Yet, targets the error (Large distance vs. Small: when p = selecting 0.003 *). large targets were significantlyFor the rate higher of unsuccessful than when selecting selection, small all size targetss had (Large significant vs. Small: differencesp = 0.003 (Large *). vs. Medium: p = 0.016For *, Large the rate vs. of Small: unsuccessful p < 0.001 selection, *, Medium all vs. sizes Small: had p significant < 0.001 *). diHowever,fferences for (Large just BTF-R, vs. Medium: there wasp = 0.016no significant *, Large vs.difference Small: pbetween< 0.001 *,the Medium varied vs.sizes Small: (they pexisted< 0.001 only *). However, for BTF-T for and just CONT) BTF-R, (Figurethere was 10). no This significant is meaningful difference because between there the were varied nosizes significant (they existed differences only forin the BTF-T selection and CONT) time between(Figure 10the). BTF-R This is and meaningful the CONT because condition. there This were in nodicates significant that if the diff erencestarget is in small the selectionand the user time needsbetween to act the precisely, BTF-R and BTF-R the could CONT be condition. the way to This go. indicates that if the target is small and the user needs to act precisely, BTF-R could be the way to go.

Figure 10. BTF-T is fast and accurate for large target objects. The asterisk mark (*) indicates statistically Figuresignificant 10. diBTF-Tfference. is fast and accurate for large target objects. The asterisk mark (*) indicates statistically significant difference. Another interesting fact is that when using BTF-T, if the target objects were large, BTF-T was good enoughAnother to compete interesting with fact the otheris that two when interfaces—equally using BTF-T, if the accurate target thanobjects BTF-R were or large, CONT BTF-T and fasterwas goodthan enough CONT butto compete slightly slowerwith the than other BTF-R two (Figureinterfaces—equally 10). The Cohen’s accurate d values than BTF-R were around or CONT 0.34 and for fasterBTF-T than vs. CONT BTF-R but and slightly 0.76 for slower BTF-T than vs. CONT BTF-R for(Figure theselection 10). The Cohen’s time. Considering d values were the around fact that 0.34 the forsize BTF-T of the vs. large BTF-R targets and in 0.76 the for experiment BTF-T vs. matches CONT for the the smallest selection size time. required Considering from the the Android fact that Design the sizeGuidelines, of the large any size targets that isinbigger the experiment than the “large” matches size the in this smallest experiment size required can benefit from from the using Android BTF-T. DesignThe Guidelines, results follow any thesize usual that is speed-accuracy bigger than the trade-o “large”ff with size respectin this toexperiment object size. can The benefit smaller from the usingobject BTF-T. is, more time was spent to select it, but with nearly equal accuracy for BTF-T (Figure 10) and betterThe accuracy results follow with BTF-R, the usual by the speed-accuracy visible target trade-off (and guiding with cursor)respect andto object less fat size. finger The problem.smaller the object is, more time was spent to select it, but with nearly equal accuracy for BTF-T (Figure 10) and better5.5.3. accuracy Usability with BTF-R, by the visible target (and guiding cursor) and less fat finger problem. The responses to the survey generally showed a similar level of usability among the three interface 5.5.3. Usability types (Figure 11). Statistically significant differences were found only on the categories of ease of learningThe (Mann-Whitney:responses to thep -valuesurvey= generally0.002 *) and showed perceived a similar accuracy level (Mann-Whitney: of usability amongp-value the= 0.001 three *). interfaceBTF-T was types regarded (Figure the 11). least Statistically easy tolearn significant while differences also clearly were faster found (although only noton the with categories a statistical of easesignificance) of learning and (Mann-Whitney: less accurate. Thep-value overall = 0.002 trend *) and was perceived that BTF-T accuracy was rated (Mann-Whitney: marginally less p-value usable =(with 0.001 partial *). BTF-T statistical was regarded support) the than least the othereasy to two, lear mostn while probably also clearly due to thefaster lack (although of method not to with correct a statisticalthe initial significance) touch position. and less accurate. The overall trend was that BTF-T was rated marginally less usable (with partial statistical support) than the other two, most probably due to the lack of method to correct the initial touch position. Appl. Sci. 2020, 10, 7920 13 of 18 Appl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 18

Figure 11. Responses to to the survey which generally show showeded a similar level of usability among the three interface interface types. types. BTF-T BTF-T was was the the least least easy easy to lear to learn,n, and and perceived perceived to be to fast be fastbut less but accurate. less accurate. The asteriskThe asterisk mark mark (*) indicates (*) indicates statistically statistically significant significant difference. difference. 6. Supplement Experiment: Smaller Objects at Different Depths 6. Supplement Experiment: Smaller Objects at Different Depths The pilot and main experiment were conducted with the user selecting (relatively large) objects The pilot and main experiment were conducted with the user selecting (relatively large) objects of different sizes at a fixed depth. For further validation, a supplement experiment was performed of different sizes at a fixed depth. For further validation, a supplement experiment was performed for selecting even smaller objects presented at different depths. The smallest object size tested in the for selecting even smaller objects presented at different depths. The smallest object size tested in the main experiment was about 0.4 cm 0.4 cm which was about the smallest recommended object size main experiment was about 0.4 cm × 0.4 cm which was about the smallest recommended object size for mobile application [37]. However, in VR applications (that often operate at different depths) such a for mobile application [37]. However, in VR applications (that often operate at different depths) guideline may not be meaningful. such a guideline may not be meaningful. 6.1. Experimental Procedure 6.1. Experimental Procedure The experimental procedure was mostly same as the main experiment except that that equal sized targetThe objects experimental were placed procedure at three depthswas mostly such thatsame the as closestthe main appeared experiment with theexcept screen that space that sizeequal of sizedabout target 0.4 cm, objects second were closest, placed 0.3 cmat three and farthestdepths such 0.2. These that the sizes closest were appeared not only significantly with the screen below space the size ofof anabout average 0.4 cm, fingertip second (1.5 closest, cm 1.5 0.3 cm cm for and the farthest index and 0.2. 2 cmThese2 cmsizes for were the thumb), not only but significantly also smaller × × belowthan the the object size of sizes an average tested in fingertip the main (1.5 experiment. cm × 1.5 cm Only for BTF-R the index (who and exhibited 2 cm × the2 cm best for performancethe thumb), butin the also main smaller experiment) than the and object CONT sizes were tested compared in the main with experiment. regards to theOnly selection BTF-R performance.(who exhibited Thus, the bestthe supplementperformance experiment in the main was experiment) designed withand CO twoNT factors were (2 compared selection methods—BTF-Rwith regards to the and selection CONT performance.and 3 object sizesThus, (screen the supplemen space)—0.2t experiment cm, 0.3 cm, was and 0.4designed cm) within with subjecttwo factors repeated (2 selection measure. methods—BTF-RWe hypothesized thatand CONTCONT wouldand 3 showobject bettersizes performance(screen space)—0.2 as compared cm, 0.3 to cm, BTF-R, and as0.4 the cm) fat within finger subjectproblem repeated is starting measure. to become We a major hypothesized performance that deterrent. CONT Onlywould BTF-R show and better CONT performance were compared as comparedwith regards to toBTF-R, the selection as the fat performance. finger problem Figure is starting 12 shows to thebecome test virtual a major environment. performance deterrent. OnlyAs BTF-R in the and main CONT experiment, were compared the user with was similarlyregards to presented the selection with performance. a sequence of Figure 12 objects 12 shows (all in theone test of three virtual sizes) environment. that appeared randomly at three different depths. At each appearance of the object, the subject was asked to select the object as fast and accurately as possible. Object selection time and general usability were measured as the dependent variables. Thirteen paid subjects (eight men and two women between the ages of 23 and 29, mean = 27.9/SD = 5.36) participated in the experiment. The subjects were replaced with a different group from the main/pilot, because the same original could not be summoned again, and in addition, the new subject group could reaffirm the previous results with less bias. The study involved six different treatments (or conditions), namely, two interfaces × Appl. Sci. 2020, 10, 7920 14 of 18 three object sizes; in each treatment condition, 12 different target objects were to be selected, comprising of one trial block, and block trials repeated 12 times (a total of 2 3 12 12 = 864 object selections). × × × Each treatment was presented in a balanced fashion using the Latin-square methodology per subject. After the whole sessions, the subjects were asked to fill out the same usability questionnaire. The same analysis methods were applied as in the main experiment. Appl. Sci. 2020, 10, x FOR PEER REVIEW 14 of 18

Figure 12. Selecting target objects at different diﬀerent depths whosewhose screen spaces were smaller than in the main experiment and the average ﬁngertipfingertip size.size.

6.2. ResultsAs in the and main Discusstion experiment, the user was similarly presented with a sequence of 12 objects (all in one ofFigure three 13 sizes) shows that the appeared average selectionrandomly time at th betweenree different BTF-R depths. and CONT At each for appearance the smaller of object the selectionobject, the collectively. subject was asked ANOVA to select revealed the object a statistically as fast and signiﬁcant accurately di ﬀaserence possible. (p

6.2. Results and Discusstion Figure 13 shows the average selection time between BTF-R and CONT for the smaller object selection collectively. ANOVA revealed a statistically significant difference (p < 0, F = 3.333). The performance for each individual size followed the similar trend (specific results not given here). As the object size went well below the smallest recommended size for mobile interaction, the fat finger problem have affected the touch based performance negatively, and likewise the usability as illustratedFigure 13.in FigureThe average 14. selectionIn all categories, time between the BTF-F CONT and CONTexhibited withwith higher objectsobjects withwithusab sizessizesility, belowbelow with 0.4 0.4statistical cm.cm. significanceTheThe asteriskasterisk for Q2markmark and (*)(*) Q3.indicatesindicates statisticallystatistically signiﬁcantsignificant didifference.ﬀerence.

Figure 14. Responses to the usability survey for selecting objects with sizes below 0.4 cm between BTF-R and CONT. The asterisk mark (*) indicates statistically significant difference.

More seriously, our observation and post-briefing revealed that the subjects had quite a difficulty in fusing the stereoscopic imagery with the EasyVR set-up, contrast to the main experiment in which no such phenomenon occurred. For some reason, subjects had trouble being able to focus on the small objects as laid out at different depth. This goes to show that additional interface would be needed to resolve this problem.

7. Conclusions In this paper, we proposed, Blurry Touch Finger, a touch screen based interaction technique an open mobile VR platform with fingers accessible to the screen. We emphasize that the main objective was to show effectiveness of BTF on EasyVR (not EasyVR itself as a framework), as a viable alternative to existing selection method like the gaze/button and even controller based ray-casting methods. Our work partly demonstrated the utility of the proposed method; the object selection performance was as good as when using head-directed and the hand-driven ray casting methods. Binocular rivalry is usually a phenomenon that affects the visual perception in a negative way. The blurriness caused by the rivalry incidentally made it possible for the user to look through the finger, and added by the sense of proprioception, and accurately select the wanted object. In addition, such touch based selection is already a quite familiar task through the usual interaction with the smartphones. However, even with the after-release technique and the see-through nature of the “blurry” touch selection, for objects fairly below the size of the fingertip, the touch based performance Appl. Sci. 2020, 10, x FOR PEER REVIEW 15 of 18

Appl. Sci.Figure2020 ,13.10, The 7920 average selection time between BTF-F and CONT with objects with sizes below 0.4 cm.15 of 18 The asterisk mark (*) indicates statistically significant difference.

FigureFigure 14.14. Responses to the usability survey for selecting objects with sizes below 0.4 cm betweenbetween BTF-RBTF-R andand CONT.CONT. TheThe asteriskasterisk markmark (*)(*) indicatesindicates statisticallystatistically signiﬁcantsignificant didifference.ﬀerence.

MoreMore seriously,seriously, our our observation observation and and post-briefing post-brief revealeding revealed that the that subjects the subjects had quite had a di quitefficulty a indifficulty fusing thein stereoscopicfusing the imagerystereoscopic with theimagery EasyVR with set-up, the contrastEasyVR to set-up, the main contrast experiment to the in whichmain noexperiment such phenomenon in which occurred.no such phenomenon For some reason, occurre subjectsd. For had some trouble reason, being subjects able to had focus trouble on the being small objectsable to asfocus laid on out the at dismallfferent objects depth. as Thislaid goesout at to different show that depth. additional This interfacegoes to show would that be additional needed to resolveinterface this would problem. be needed to resolve this problem. 7. Conclusions 7. Conclusions In this paper, we proposed, Blurry Touch Finger, a touch screen based interaction technique an In this paper, we proposed, Blurry Touch Finger, a touch screen based interaction technique an open mobile VR platform with fingers accessible to the screen. We emphasize that the main objective open mobile VR platform with fingers accessible to the screen. We emphasize that the main objective was to show effectiveness of BTF on EasyVR (not EasyVR itself as a framework), as a viable alternative was to show effectiveness of BTF on EasyVR (not EasyVR itself as a framework), as a viable to existing selection method like the gaze/button and even controller based ray-casting methods. alternative to existing selection method like the gaze/button and even controller based ray-casting Our work partly demonstrated the utility of the proposed method; the object selection performance methods. Our work partly demonstrated the utility of the proposed method; the object selection was as good as when using head-directed and the hand-driven ray casting methods. performance was as good as when using head-directed and the hand-driven ray casting methods. Binocular rivalry is usually a phenomenon that affects the visual perception in a negative way. Binocular rivalry is usually a phenomenon that affects the visual perception in a negative way. The blurriness caused by the rivalry incidentally made it possible for the user to look through The blurriness caused by the rivalry incidentally made it possible for the user to look through the the finger, and added by the sense of proprioception, and accurately select the wanted object. finger, and added by the sense of proprioception, and accurately select the wanted object. In In addition, such touch based selection is already a quite familiar task through the usual interaction addition, such touch based selection is already a quite familiar task through the usual interaction with the smartphones. with the smartphones. However, even with the after-release technique and the see-through nature of the “blurry” touch However, even with the after-release technique and the see-through nature of the “blurry” selection, for objects fairly below the size of the fingertip, the touch based performance suffered from touch selection, for objects fairly below the size of the fingertip, the touch based performance the usual fat finger problem and difficulty in focus. Additional interactional provision can help alleviate this problem, e.g., by relying on techniques such as the “scaled” grab [43], in which the potential target can be effectively be enlarged with respect to the user for easier selection. Although the EasyVR platform (or the use of open clip-on lenses) is not as popular as Google Cardboard or regular head-set based VR at this time, we believe that users and developers will now understand that EasyVR offers reasonable immersion despite being open (refer to [6] by the same authors) and can possibly allow similar user experience to regular non-VR (e.g., touch based smartphone) mobile gaming. Thus, we believe there is a big potential in the future for further popularizing VR and smearing its usage into our everyday lives. Furthermore, there are many other basic VR tasks (e.g., object manipulation and rotation, navigation, menu selection) with which BTF can be extended and applied to. Note that any interaction methods for EasyVR is bound to involve touch based selection, e.g., of interface buttons. In fact, we have already begun the study of navigation methods in EasyVR [44] in which it was observed that no users have any difficulty in interacting with the touch screen buttons and interfaces using the Blurry Appl. Sci. 2020, 10, 7920 16 of 18

Touch Finger. Even the BTF selection method itself can be realized in a variety of forms (e.g., look of cursor, generalizing into a cone-shaped ﬂashlight metaphor). Continued studies e.g., with respect to the Fitts’ law with a larger number of participants are needed to fully validate and characterize the interaction performance and explore the design space of the Blurry Touch Finger as the main interface for EasyVR. In addition, we assumed that EasyVR was in general more convenient to use than closed M-VR (such as Cardboard/GearVR). However, there may be negative eﬀects e.g., to the level of immersion/presence due to peripheral distraction and touch based interaction. This is subject of another on-going work [6]. Touch based interaction is generally more suited for direct interaction with the target objects, however, there may be instances where indirect interaction is more appropriate in VR. Accidental touches can also be problematic. Nevertheless, our study has given a promising glimpse into the potential of the touch based interaction for M-VR, especially in the context of bringing EasyVR to a larger user base. We envision EasyVR/BTF can be used for causal touch-driven games and quick reviewing of 360 videos. It can also work to provide dual modes in serious games and social networking situation, allowing the user to seamlessly switch between regular smart phone and immersive viewing with the same touch interaction.

Author Contributions: Conceptualization, Y.R.K. and G.J.K.; methodology, Y.R.K. and G.J.K.; software, Y.R.K., S.P.; validation, Y.R.K., S.P.; formal analysis, Y.R.K., S.P. and G.J.K.; investigation, Y.R.K. and G.J.K.; resources, Y.R.K. and G.J.K.; data curation, Y.R.K., S.P.; writing—original draft preparation, Y.R.K. and G.J.K.; writing—review and editing, Y.R.K. and G.J.K.; visualization, Y.R.K., S.P.; supervision, G.J.K.; project administration, G.J.K.; funding acquisition, G.J.K. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2020-2016-0-00312) supervised by the IITP (Institute for Information & communications Technology Planning & Evaluation). Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Lewis, P.Daydream VR vs. HTC Vive & Oculus Rift|Mobile VR vs. PC. Availableonline: http://www.pcadvisor. co.uk/review/wearable-tech/daydream-vr-vs-htc-vive-oculus-rift-mobile-vr-vs-pc-vr-3647738/ (accessed on 8 October 2020). 2. Google Cardboard. Available online: http://vr.google.com/cardboard/ (accessed on 8 October 2020). 3. Samsung. Samsung Gear VR. Available online: http://www.samsung.com/global/galaxy/gear-vr/ (accessed on 8 October 2020). 4. HTC. HTC Vive. Available online: http://www.vive.com/kr/ (accessed on 8 October 2020). 5. Oculus. Oculus VR. Available online: http://www.oculus.com/ (accessed on 8 October 2020). 6. Kim, Y.R.; Kim, G.J. Presence and immersion of “easy” mobile VR with open flip-on lenses. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, Gothenburg, Sweden, 8–10 November 2017; pp. 1–7. 7. Figment VR, Figment VR. Available online: http://www.figmentvr.com/ (accessed on 8 October 2020). 8. Siek, K.A.; Rogers, Y.; Connelly, K.H. Fat finger worries: How older and younger users physically interact with PDAs. In Proceedings of the 10th IFIP TC13 International Conference on Human-Computer Interaction, Rome, Italy, 12–16 September 2005; pp. 267–280. 9. Bowman, D.A.; Hodges, L.F. An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In Proceedings of the 1997 Symposium on Interactive 3D Graphics, Providence, RI, USA, 27–30 April 1997; pp. 35–ff. 10. Bowman, D.A.; Kruijff, E.; Laviola, J.J.; Poupyrev, I. 3D User Interfaces: Theory and Practice; Addison-Wesley Professional: Redwood City, CA, USA, 2004. 11. Yu, J.E.; Kim, G.J. Blurry (sticky) finger: Proprioceptive pointing and selection of distant objects for optical see-through based augmented reality. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Merida, Mexico, 19–23 September 2016; pp. 336–337. Appl. Sci. 2020, 10, 7920 17 of 18

12. Bolt, R.A. “Put-that-there” Voice and gesture at the graphics interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, Seattle, WA, USA, 14–18 July 1980; pp. 262–270. 13. Liang, J.; Green, M. JDCAD: A highly interactive 3D modeling system. Comput. Graph. 1994, 18, 499–506. [CrossRef] 14. Colaço, A.; Kirmani, A.; Yang, H.S.; Gong, N.W.; Schmandt, C.; Goyal, V.K. Mime: Compact, low power 3D gesture sensing for interaction with head mounted displays. In Proceedings of the 26th Annual ACM Symposium on User interface Software and Technology, St. Andrews, UK, 8–11 October 2013; pp. 227–236. 15. Bowman, D.A.; Hodges, L.F. Formalizing the design, evaluation, and application of interaction techniques for immersive virtual environments. J. Vis. Lang. Comput. 1999, 10, 37–53. [CrossRef] 16. Lee, G.A.; Bai, H.; Billinghurst, M. Automatic zooming interface for tangible augmented reality applications. In Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry, Singapore, 2–4 December 2012; pp. 9–12. 17. Wachs, J.P.; Kölsch, M.; Stern, H.; Edan, Y. Vision-based hand-gesture applications. Commun. ACM 2011, 54, 60–71. [CrossRef] 18. Mistry, P.; Maes, P. SixthSense: A wearable gestural interface. In Proceedings of the SIGGRAPH Asia 2009 Art Gallery and Emerging Technologies: Adaptation, Yokohama, Japan, 16–19 December 2009; pp. 85–85. 19. Ha, T.; Woo, W. Bare hand interface for interaction in the video see-through HMD based wearable AR environment. In Proceedings of the International Conference on Entertainment Computing, Cambridge, UK, 20–22 September 2006; pp. 354–357. 20. Piumsomboon, T.; Clark, A.; Umakatsu, A.; Billinghurst, M. Poster: Physically-based natural hand and tangible AR interaction for face-to-face collaboration on a tabletop. In Proceedings of the 2012 IEEE Symposium on 3D User Interfaces (3DUI), Costa Mesa, CA, USA, 4–5 March 2012; pp. 155–156. 21. Prisacariu, V.A.; Reid, I. 3D hand tracking for human computer interaction. Image Vis. Comput. 2012, 30, 236–250. [CrossRef] 22. Gielen, C.; van den Heuvel, P.; van Gisbergen, J. Coordination of fast eye and arm movements in a tracking task. Exp. Brain Res. 1984, 56, 154–161. [CrossRef][PubMed] 23. Scarchilli, K.; Vercher, J.L. The oculomanual coordination control center takes into account the mechanical properties of the arm. Exp. Brain Res. 1999, 124, 42–52. [CrossRef][PubMed] 24. Bernier, P.M.; Chua, R.; Franks, I.M. Is proprioception calibrated during visually guided movements? Exp. Brain Res. 2005, 167, 292–296. [CrossRef][PubMed] 25. Helsen, W.F.; Elliott, D.; Starkes, J.L.; Ricker, K.L. Coupling of eye, ﬁnger, elbow, and shoulder movements during manual aiming. J. Mot. Behav. 2000, 32, 241–248. [CrossRef][PubMed] 26. Li, Z.; Jarvis, R. Real time hand gesture recognition using a range camera. In Proceedings of the Australasian Conference on Robotics and Automation (ACRA), Sydney, Australia, 2–4 December 2009; pp. 21–27. 27. Pierce, J.S.; Forsberg, A.S.; Conway, M.J.; Hong, S.; Zeleznik, R.C.; Mine, M.R. Image plane interaction techniques in 3D immersive environments. In Proceedings of the 1997 Symposium on Interactive 3D Graphics, Providence, RI, USA, 27–30 April 1997; p. 39. 28. Bowman, D.A.; Koller, D.; Hodges, L.F. A methodology for the evaluation of travel techniques for immersive virtual environments. Virtual Real. 1998, 3, 120–131. [CrossRef] 29. Samsung. Samsung GearVR With Controller. Available online: http://www.samsung.com/us/mobile/virtual- reality/gear-vr/gear-vr-with-controller-sm-r324nzaaxar/ (accessed on 8 October 2020). 30. Poupyrev, I.; Ichikawa, T.; Weghorst, S.; Billinghurst, M. Egocentric object manipulation in virtual environments: Empirical evaluation of interaction techniques. Comput. Graph. Forum 1998, 17, 41–52. [CrossRef] 31. Lee, S.; Seo, J.; Kim, G.J.; Park, C.M. Evaluation of pointing techniques for ray casting selection in virtual environments. In Proceedings of the International Conference on Virtual Reality and its Application in Industry, Hangzhou, China, 9–12 April 2002; Volume 4756, pp. 38–44. 32. Gugenheimer, J.; Dobbelstein, D.; Winkler, C.; Haas, G.; Rukzio, E. Facetouch: Enabling touch interaction in display ﬁxed uis for mobile virtual reality. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 16–19 October 2016; pp. 49–60. 33. Lee, J.; Kim, B.; Suh, B.; Koh, E. Exploring the front touch interface for virtual reality headsets. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 7–12 May 2016; pp. 2585–2591. Appl. Sci. 2020, 10, 7920 18 of 18

34. Point Shooting. Available online: https://en.wikipedia.org/wiki/Point_shooting (accessed on 8 October 2020). 35. Alais, D.; Blake, R. Binocular Rivalry; MIT Press: Cambridge, MA, USA, 2005. 36. Allelotropia. Available online: https://en.wikipedia.org/wiki/Binocular_vision (accessed on 8 October 2020). 37. Android User Interface Guideline. Available online: https://developer.android.com/guide/practices/ui_ guidelines/index.html (accessed on 8 October 2020). 38. Cournia, N.; Smith, J.D.; Duchowski, A.T. Gaze-vs. hand-based pointing in virtual environments. In Proceedings of the CHI’03 Extended Abstracts on Human Factors in Computing Systems, Ft. Lauderdale, FL, USA, 5–10 April 2003; pp. 772–773. 39. Samsung. Samsung Galaxy S7. Available online: http://www.samsung.com/global/galaxy/galaxy-s7/ (accessed on 8 October 2020). 40. Homido. Homido Mini VR Glasses. Available online: http://homido.com/en/mini (accessed on 8 October 2020). 41. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Adv. Psychol. 1988, 52, 139–183. 42. Brooke, J. SUS: A “quick and dirty” usability. Usability Eval. Ind. 1996, 189, 4–7. 43. Mine, M.R.; Brooks, F.P., Jr.; Sequin, C.H. Moving objects in space: Exploiting proprioception in virtual-environment interaction. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA, 3–8 August 1997; pp. 19–26. 44. Kim, Y.R.; Choi, H.; Chang, M.; Kim, G.J. Applying touchscreen based navigation techniques to mobile virtual reality with open clip-on lenses. Electronics 2020, 9, 1448. [CrossRef]

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional aﬃliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).