![PSEUDO-HAPTIC DISPLAY of MASS and MASS DISTRIBUTION DURING OBJECT ROTATION in VIRTUAL REALITY 2079 Mass Illusion Was a Demonstration of Such Effect [24]](http://data.docslib.org/img/650b6cfbe4a58b251e958e3ed87d742a-1.webp)
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020 2077
Pseudo-Haptic Display of Mass and Mass Distribution
During Object Rotation in Virtual Reality Run Yu and Doug A. Bowman Abstract—We propose and evaluate novel pseudo-haptic techniques to display mass and mass distribution for proxy-based object manipulation in virtual reality. These techniques are specifically designed to generate haptic effects during the object’s rotation. They rely on manipulating the mapping between visual cues of motion and kinesthetic cues of force to generate a sense of heaviness, which alters the perception of the object’s mass-related properties without changing the physical proxy. First we present a technique to display an object’s mass by scaling its rotational motion relative to its mass. A psycho-physical experiment demonstrates that this technique effectively generates correct perceptions of relative mass between two virtual objects. We then present two pseudo-haptic techniques designed to display an object’s mass distribution. One of them relies on manipulating the pivot point of rotation, while the other adjusts rotational motion based on the real-time dynamics of the moving object. An empirical study shows that both techniques can influence perception of mass distribution, with the second technique being significantly more effective. Index Terms—Virtual reality, object manipulation, object rotation, human perception, pseudo-haptics
1 INTRODUCTION Haptic sensation is an important part of a virtual reality (VR) motion have also proven effective, but only in highly controlled experience [5, 20]. It helps us understand the (simulated) physical settings [41, 42]. It remains to be seen whether this approach can be aspect of a virtual environment (VE) and is critical for enhancing the applied to rotation in a VR setting with unconstrained input. This sense of presence [12, 14, 34, 47]. However, providing effective haptic paper intends to fill this gap by taking a close look at how rotational feedback during interactive object manipulation in VR is difficult. In motion could be used to generate mass-related illusions in VR object manipulation, modern commercial VR systems often let the interaction. user hold a proxy (called a hand-held controller), and pick up different In this paper, we first investigate whether scaling an object’s virtual objects using this same tool [44]. A particular proxy provides rotational movement will influence the perception of its total mass the same haptic stimuli for all virtual objects, which contradicts the relative to other objects. We present statistical evidence from a scenario of holding different objects with distinct mass properties. psycho-physical experiment to support this hypothesis. Beyond the Thus, an item that is intended to be heavy may suddenly feel as light perception of an item’s total mass, we recognize that object rotation is as the proxy once it is picked up [45]. Deviations from the user’s also affected strongly by the distribution of mass within an object. For expectation (like this mismatch of heaviness) could reduce the VE’s example, swinging a golf club where most of the mass is concentrated coherence [36-38, 40], potentially leading to decreased plausibility in the club head feels very different than swinging a metal rod made [39, 40]. This problem can be addressed through active haptic of a single homogeneous material, even if both have the same total rendering, which directly displays haptic forces in real-time (e.g., mass. With this observation, we discuss the design and through a robotic arm attached to the user’s hand), but such implementation of two pseudo-haptic techniques to display mass approaches are often cumbersome and expensive [13, 45]. distribution within an object. One of them relies on manipulating the Apart from haptic stimulation, visual stimuli can also influence the pivot point of rotation, while the other adjusts the C/D ratio on the fly perception of heaviness (e.g., [33]). This phenomenon has been based on the real-time dynamics of the object’s movement. Our leveraged in VR through pseudo-haptic techniques that change the experiment shows that both techniques can influence perceptions of user’s perception of virtual objects without altering the physical proxy mass distribution, with the second technique being much more [18]. For example, varying the control/display (C/D) ratio of an effective. object’s translation directly influences the perception of its total mass As far as we are aware, this is the first effort focusing on pseudo- [7]. A larger C/D ratio makes objects feel heavier, since the same hand haptic display of mass and mass distribution during unconstrained movement results in less visual movement of the object. This rotational control in VR. The techniques presented here are designed technique has been applied in VR settings to successfully alter mass to be easily applicable to common proxy-based manipulation in VR perception with the user holding the same physical proxy [31, 35]. systems. The contributions of this paper include: Similar approaches of scaling visual movement during rotational • Empirical evidence that changing the C/D ratio of rotational motion in VR effectively influences the user’s perception of an object’s total mass. • Two novel pseudo-haptic techniques for displaying mass • distribution during rotation in VR, along with empirical evidence • that at least one of these techniques is very effective in creating such effects.
2 RELATED WORK Before introducing pseudo-haptic techniques for generating mass- related perceptions in immersive VEs, we discuss research describing how humans fundamentally perceive heaviness. Existing research has Manuscript received 10 Sept. 2019; accepted 5 Feb. 2020. suggested that the perception of heaviness is a multimodal one, Date of publication 18 Feb. 2020; date of current version 27 Mar. 2020. influenced by both haptic and visual information. Charpentier’s size- Digital Object Identifier no. 10.1109/TVCG.2020.2973056
1077-2626 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information. 2078 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020 mass illusion was a demonstration of such effect [24]. This illusion which directly addresses the challenge of displaying mass distribution this metaphor, which maps the hand’s relative movement to the the reference cube with a 1:1 C/D ratio has a mass of 1kg, another occurs when the weight of a larger item is underestimated compared by introducing novel hardware that combines active and passive haptic movement of a remotely placed object. This is similar to how we use cube with a mass of 5kg would have a C/D ratio of 1:0.2 (calculated to a smaller one with the same mass. Amazeen et al. later suggested display. Experiments found that this device leads to a more realistic a mouse to drag icons in a desktop environment, which does not by 5kg/1kg). that the true source of this illusion is not the change in size, but instead and enjoyable experience compared to an equivalent proxy with fixed require the hand and the object to be co-located. The reason we use the change in the object’s rotational inertia, which describes how mass mass distribution. Zenner’s thesis also provides a thorough overview remote manipulation is to enhance the applicability of the proposed Table 1: Scaling values used in experiment I (values represent the is distributed over the volume [1]. Masin et al. conducted an of materials related to this issue [44]. Similar to this, Cheng et al. techniques—not requiring the physical proxy and the virtual object to reciprocal of C/D ratio) experiment that asked participants to look at an object before, during proposed a liquid-based haptic proxy that can dynamically change its be co-located makes them compatible to other pseudo-haptic or after lifting it, and the influence from vision was only found when mass properties, but no formal evaluation was conducted [6]. Our techniques that are based on displacing the virtual object from the '$)) $#(! $,# '$)) $#(! *% they could see it while lifting [23]. This result suggests that the work attempts to avoid the need for special-purpose hardware by proxy (e.g., [7, 31, 35]), which inherently results in remote control. object’s motion might be a more crucial factor compared to its static displaying mass distribution using pseudo-haptics alone. Our technique for producing a pseudo-haptic display of total mass appearance. during rotation is in direct analogy with the technique of changing C/D One thing worth noting is that this experiment only evaluates if Other studies (also outside the context of VR) have shown that 3 APPROACH ratio in translation [7]. It’s also very similar to how rotational motion different rotational C/D ratios could lead to the perception of one item heaviness can be effectively perceived by visual information alone. is scaled to influence perceived heaviness in the experiments by Streit In object manipulation, the object’s motion is causally linked to the being heavier than the other. It does not evaluate the perception of the Runeson et al. demonstrated that the weight of an item can be et al. [41, 42]. We change the C/D ratio for rotation based on the absolute value of the object’s mass or the actual mass ratio between accurately judged by watching a point light display of a person lifting force exerted onto it [32, 33]. For rotation, this correlation can be object’s mass—a heavier object will have a larger C/D ratio (i.e., its expressed in the rotational version of Newton’s second law: two objects. Using the previous example, even though we set the C/D it [33]. Similar effectiveness of visual information on mass perception rotational motion is scaled down) and vice versa. The C/D ratio is ratios based on the objects’ virtual mass, we don’t claim that using a was also shown in an experiment by Bingham [4]. Runeson et al. (1) applied to rotational motion using several steps. First, for every frame, C/D ratio of 1:0.2 could lead to the perception that the item weighs proposed the “kinematic specification of dynamics (KSD) principles”, the orientation (originally expressed as a quaternion) of the hand-held 5kg or that it feels 5 times heavier than the 1kg reference cube. It only In this equation, is the torque,$ 8 # which is the rotational analog of which explains how dynamics (force and mass) could be interpreted force. The overall torque of object manipulation can come from both proxy is extracted. Then the difference in the orientation between the tests if the one with a C/D ratio of 1:0.2 feels heavier than the reference through kinematics (motion) using the physics-based causality hand input and the $object’s gravitational force. is the moment of last frame and the current frame is used to calculate an axis and an with 1:1. We consider the perception of the absolute values of mass between the two [32, 33]; the novel algorithms presented in this paper inertia, which describes mass-related properties of the object; and angle, which represent the proxy’s relative rotation between these two and mass difference as future work. are based on a similar physics-based model. is the angular acceleration, which concerns the object’s motion. is frames. The appropriate C/D ratio is applied to scale the angle while To eliminate any influence from translational motion or the Closely related to the experiments in this paper, Streit et al. tested the dynamics of the system, which is perceived hapticially through# keeping the same axis. Finally, these values are used to rotate the translational C/D ratio (which has already been shown to effectively how haptically perceived rotational inertia interacts with visual muscular tension, while is the kinematics, which is seen visually$ by virtual object. produce a pseudo-haptic illusion of mass perception [7]), both virtual displacement of motion in generating heaviness perception [41, 42]. its movement. According to the KSD principle [32, 33], scaling up the This technique loosely conforms to the KSD principle under cubes were fixed in place and only had three degrees-of-freedom In these experiments, participants were asked to rotate a handle visual motion ( ) while #maintaining the same haptic cues ( ) would equation (1). Assuming the pivot point is at the CoM, the only active (DOF) of rotation. In the real world, the user held a physical proxy in attached to a fixed frame while watching the manipulated motion on a imply a decrease in I and vice versa. We leverage this interplay torque in the system is the torque from hand input, since torque due to her dominant hand, with its rotational motion mapped to the currently 2D screen. The result suggests that perception of heaviness can be between the two# sensory channels to manipulate the perception$ of the gravity becomes zero if the rotational axis goes through the CoM. As selected cube using the assigned C/D ratio. Another hand-held consistently affected by both sensory channels, and this influence virtual object’s mass. long as the mass distribution is constant, increasing the C/D ratio has controller in the non-dominant hand was used to select one of the conforms to the physics-based correlation between kinematics and Compared to translational motion, what is unique about rotation is the effect that the same exerted torque from the hand will result in less cubes using ray-casting, as shown in Figure 3A. The user indicated dynamics. that both the haptic information ( ) and mass-related characteristics visual rotation of the object, which in turn increases the total mass of which cube felt heavier by clicking on the button next to the The aforementioned studies were not performed in immersive ( in (1) are only meaningful when tied to a specific rotational axis, the specified in (1). The reason why we say this is a loose corresponding cube (Figure 1) using the same ray. VEs, but the multimodal nature of haptic perception has been $ conformation is that changing C/D ratios relate to changes in velocity, and different axes will result in different values for them. is defined recognized and leveraged for desirable effects in VR. Since building as: a vector generated by the cross product of the radius vector (from not acceleration. But this approach appears to be “close enough” to 4.2.2 Procedure and Measures a matching physical prop for every virtual object in a VE is not the axis of rotation to the point of application of force) and$ the force effectively influence the perception of relative mass between objects. With three repetitions for each of the eight C/D ratio levels shown in practical in most situations, one line of research explores the vector, and is the sum of the products of each particle’s mass in the Table 1, each participant made 24 total mass judgments. The order of 4.2 Experiment I practicality of mapping multiple virtual items to one physical prop rigid body with the square of its distance from the rotational axis (for the trials was randomized. For each trial, if the cube with a larger C/D without the user noticing it, which is achieved by exploiting the a more thorough explanation of the dynamics of rigid body rotation, We designed a psycho-physical experiment to evaluate the ratio was selected as the heavier one, we considered it as a “correct” dominance of visual information to alter haptic sensations. Kohli et al. please refer to [9]). In addition to the object’s total mass, how mass is effectiveness of this technique. The experiment tests whether a user response. For each participant, we measured the percentage of correct proposed the concept of “redirected touching” [15] and found that this distributed relative to the axis matters to both of these concepts by can perceive a mass difference between two virtual objects rotated responses for each level of C/D ratio as the primary quantitative manipulation could have minimal effect on task performance [16] and definition. Thus, when discussing rotation, we cannot treat the entire with different C/D ratios. measure. users could adapt to it very well in a training task [17]. Azmandian et object as a point mass, but rather we need to understand its mass Using correct responses alone potentially puts us at risk that we 4.2.1 Experimental Design al. proposed “haptic retargeting” [2] and Lohse proposed “haptic distribution relative to the rotational axis. Consider again the golf club were only measuring participants’ ability to guess the correct answer change blindness” [21] to accomplish similar effects. Zenner et al. example: the forces required to rotate the golf club change drastically Figure 1 shows the virtual environment used in this experiment. In this instead of actually measuring their perception—users may have reported the detection thresholds when the user’s hand is displaced depending on where it is held and which direction it is rotated. This scene, two cubes with the same appearance (size, shape, and texture) guessed that the object that rotates slower “should” be the heavier one, visually, which could be applied for haptic retargeting purposes [46]. special feature of rotational motion gives us an opportunity to not only were placed side by side. The user was asked to rotate them in turn but they may not have been experiencing an actual sensation of mass Fujinawa et al. described a novel method for computationally manipulate the perception of an object’s total mass, but also the and make a two-alternative forced choice about which of them felt difference. To alleviate this issue, we also adopted a post-experiment designing a hand-held controller so its haptic properties could perception of how mass is distributed at a finer level of granularity. heavier than the other. The rotational control of the two cubes was questionnaire on the quality of induced haptic perception, proposed by convincingly represent a virtual counterpart even though they look It is important to point out that we treat total mass and mass scaled by difference C/D ratios. If the technique is effective, we Rietzler et al. [30]. Customized to fit the context of this experiment, different [8]. distribution as two separate physical properties in designing the expected that users would perceive that objects rotating with larger the questionnaire asked participants how much they agreed with the The interaction between visual and haptic perception has been pseudo-haptic techniques in this paper. On the one hand, two objects C/D ratios would feel heavier than the other, and vice versa. In this following six statements on a seven-point Likert scale (expressed to leveraged to influence many types of haptic sensations in VR, creating can have the same shape and total mass but distinct characteristics in experiment, the pivot point of rotation was always at the geometric the user using the labels “strongly disagree,” “disagree,” “somewhat a research area called “pseudo-haptics” [28]. It has found successful mass distribution; on the other hand, two objects can have the same center of the cube. disagree,” “neither agree nor disagree,” “somewhat agree,” “agree,” applications in changing the perception of friction [19], mass [7, 31, shape and the same relative mass distribution between different parts or “strongly agree”): 35], shape [3], stiffness [10], softness [27], torque [25] and force [29]. of the body, but different total masses (e.g., because one object is made Q1: I could feel that the two cubes have different weight Pseudo-haptic display of mass is usually achieved by visually of a different material). In sections 4 and 5, we describe techniques for Q2: The representation of weight difference felt realistic displacing the hand and/or the object, so a sense of heaviness is generating these two illusions and evaluate their effectiveness. In the Q3: I had the feeling of manipulating real objects generated though kinesthetic cues [7, 31, 35]. Research also finds that paper’s conclusion, we will discuss how these two aspects of pseudo- Q4: The representation of weight difference is sufficient for VR such a technique could be used to control fatigue during object lifting haptics can be combined for practical use. Q5: I liked this representation of weight difference [43]. Q6: I felt I have control Regarding the perception of an object’s mass distribution, Lukos In the questionnaires used in this paper, we exchanged the term 4 PSEUDO-HAPTIC DISPLAY OF TOTAL MASS DURING et al. [22] found that people change how they grab an object (in the Figure 1: The virtual environment for experiment I. “mass” with “weight”. Even though they have different meanings in ROTATION real world) if they can visually predict its center of mass (CoM). We physics and technically “mass” was the more accurate term here, we For each trial, the system randomly picked one of the cubes to assume that this will also hold true in VR, which is strong motivation felt “weight” was a more understandable expression for a general 4.1 The Technique assign it a non-1:1 rotational C/D ratio, while the other cube had a to develop techniques that adjust rotational motion based on an audience and could bring up what we were looking for in the ratio of 1:1 (used as the reference). The levels of the first C/D ratio are object’s mass distribution. Hashiguchi et al. used diminished reality to All the interaction techniques in this paper are based on the common questionnaires all the same, namely the subjective sensation of listed in Table 1 (note that the table lists the reciprocal of C/D ratios), hide part of a real rod to influence the perception of its mass and mass “simple virtual hand” metaphor, in which a zero-order mapping maps heaviness when manipulating these virtual objects. with values centered around 1.0. We can think of these values as being distribution [11]. Zenner et al. developed a “weight-shifting dynamic the hand-held controller’s movement to the object’s movement Each participant first went through an introduction session, which linearly scaled by the virtual mass of the items. For example, assuming haptic proxy” that could change its mass distribution on the fly [45], (translation and rotation) [26]. We use the “remote control” version of explained the task and the interface. Next, we provided a training YU AND BOWMAN: PSEUDO-HAPTIC DISPLAY OF MASS AND MASS DISTRIBUTION DURING OBJECT ROTATION IN VIRTUAL REALITY 2079 mass illusion was a demonstration of such effect [24]. This illusion which directly addresses the challenge of displaying mass distribution this metaphor, which maps the hand’s relative movement to the the reference cube with a 1:1 C/D ratio has a mass of 1kg, another occurs when the weight of a larger item is underestimated compared by introducing novel hardware that combines active and passive haptic movement of a remotely placed object. This is similar to how we use cube with a mass of 5kg would have a C/D ratio of 1:0.2 (calculated to a smaller one with the same mass. Amazeen et al. later suggested display. Experiments found that this device leads to a more realistic a mouse to drag icons in a desktop environment, which does not by 5kg/1kg). that the true source of this illusion is not the change in size, but instead and enjoyable experience compared to an equivalent proxy with fixed require the hand and the object to be co-located. The reason we use the change in the object’s rotational inertia, which describes how mass mass distribution. Zenner’s thesis also provides a thorough overview remote manipulation is to enhance the applicability of the proposed Table 1: Scaling values used in experiment I (values represent the is distributed over the volume [1]. Masin et al. conducted an of materials related to this issue [44]. Similar to this, Cheng et al. techniques—not requiring the physical proxy and the virtual object to reciprocal of C/D ratio) experiment that asked participants to look at an object before, during proposed a liquid-based haptic proxy that can dynamically change its be co-located makes them compatible to other pseudo-haptic or after lifting it, and the influence from vision was only found when mass properties, but no formal evaluation was conducted [6]. Our techniques that are based on displacing the virtual object from the '$)) $#(! $,# '$)) $#(! *% they could see it while lifting [23]. This result suggests that the work attempts to avoid the need for special-purpose hardware by proxy (e.g., [7, 31, 35]), which inherently results in remote control. object’s motion might be a more crucial factor compared to its static displaying mass distribution using pseudo-haptics alone. Our technique for producing a pseudo-haptic display of total mass appearance. during rotation is in direct analogy with the technique of changing C/D One thing worth noting is that this experiment only evaluates if Other studies (also outside the context of VR) have shown that 3 APPROACH ratio in translation [7]. It’s also very similar to how rotational motion different rotational C/D ratios could lead to the perception of one item heaviness can be effectively perceived by visual information alone. is scaled to influence perceived heaviness in the experiments by Streit In object manipulation, the object’s motion is causally linked to the being heavier than the other. It does not evaluate the perception of the Runeson et al. demonstrated that the weight of an item can be et al. [41, 42]. We change the C/D ratio for rotation based on the absolute value of the object’s mass or the actual mass ratio between accurately judged by watching a point light display of a person lifting force exerted onto it [32, 33]. For rotation, this correlation can be object’s mass—a heavier object will have a larger C/D ratio (i.e., its expressed in the rotational version of Newton’s second law: two objects. Using the previous example, even though we set the C/D it [33]. Similar effectiveness of visual information on mass perception rotational motion is scaled down) and vice versa. The C/D ratio is ratios based on the objects’ virtual mass, we don’t claim that using a was also shown in an experiment by Bingham [4]. Runeson et al. (1) applied to rotational motion using several steps. First, for every frame, C/D ratio of 1:0.2 could lead to the perception that the item weighs proposed the “kinematic specification of dynamics (KSD) principles”, the orientation (originally expressed as a quaternion) of the hand-held 5kg or that it feels 5 times heavier than the 1kg reference cube. It only In this equation, is the torque,$ 8 # which is the rotational analog of which explains how dynamics (force and mass) could be interpreted force. The overall torque of object manipulation can come from both proxy is extracted. Then the difference in the orientation between the tests if the one with a C/D ratio of 1:0.2 feels heavier than the reference through kinematics (motion) using the physics-based causality hand input and the $object’s gravitational force. is the moment of last frame and the current frame is used to calculate an axis and an with 1:1. We consider the perception of the absolute values of mass between the two [32, 33]; the novel algorithms presented in this paper inertia, which describes mass-related properties of the object; and angle, which represent the proxy’s relative rotation between these two and mass difference as future work. are based on a similar physics-based model. is the angular acceleration, which concerns the object’s motion. is frames. The appropriate C/D ratio is applied to scale the angle while To eliminate any influence from translational motion or the Closely related to the experiments in this paper, Streit et al. tested the dynamics of the system, which is perceived hapticially through# keeping the same axis. Finally, these values are used to rotate the translational C/D ratio (which has already been shown to effectively how haptically perceived rotational inertia interacts with visual muscular tension, while is the kinematics, which is seen visually$ by virtual object. produce a pseudo-haptic illusion of mass perception [7]), both virtual displacement of motion in generating heaviness perception [41, 42]. its movement. According to the KSD principle [32, 33], scaling up the This technique loosely conforms to the KSD principle under cubes were fixed in place and only had three degrees-of-freedom In these experiments, participants were asked to rotate a handle visual motion ( ) while #maintaining the same haptic cues ( ) would equation (1). Assuming the pivot point is at the CoM, the only active (DOF) of rotation. In the real world, the user held a physical proxy in attached to a fixed frame while watching the manipulated motion on a imply a decrease in I and vice versa. We leverage this interplay torque in the system is the torque from hand input, since torque due to her dominant hand, with its rotational motion mapped to the currently 2D screen. The result suggests that perception of heaviness can be between the two# sensory channels to manipulate the perception$ of the gravity becomes zero if the rotational axis goes through the CoM. As selected cube using the assigned C/D ratio. Another hand-held consistently affected by both sensory channels, and this influence virtual object’s mass. long as the mass distribution is constant, increasing the C/D ratio has controller in the non-dominant hand was used to select one of the conforms to the physics-based correlation between kinematics and Compared to translational motion, what is unique about rotation is the effect that the same exerted torque from the hand will result in less cubes using ray-casting, as shown in Figure 3A. The user indicated dynamics. that both the haptic information ( ) and mass-related characteristics visual rotation of the object, which in turn increases the total mass of which cube felt heavier by clicking on the button next to the The aforementioned studies were not performed in immersive ( in (1) are only meaningful when tied to a specific rotational axis, the specified in (1). The reason why we say this is a loose corresponding cube (Figure 1) using the same ray. VEs, but the multimodal nature of haptic perception has been $ conformation is that changing C/D ratios relate to changes in velocity, and different axes will result in different values for them. is defined recognized and leveraged for desirable effects in VR. Since building as: a vector generated by the cross product of the radius vector (from not acceleration. But this approach appears to be “close enough” to 4.2.2 Procedure and Measures a matching physical prop for every virtual object in a VE is not the axis of rotation to the point of application of force) and$ the force effectively influence the perception of relative mass between objects. With three repetitions for each of the eight C/D ratio levels shown in practical in most situations, one line of research explores the vector, and is the sum of the products of each particle’s mass in the Table 1, each participant made 24 total mass judgments. The order of 4.2 Experiment I practicality of mapping multiple virtual items to one physical prop rigid body with the square of its distance from the rotational axis (for the trials was randomized. For each trial, if the cube with a larger C/D without the user noticing it, which is achieved by exploiting the a more thorough explanation of the dynamics of rigid body rotation, We designed a psycho-physical experiment to evaluate the ratio was selected as the heavier one, we considered it as a “correct” dominance of visual information to alter haptic sensations. Kohli et al. please refer to [9]). In addition to the object’s total mass, how mass is effectiveness of this technique. The experiment tests whether a user response. For each participant, we measured the percentage of correct proposed the concept of “redirected touching” [15] and found that this distributed relative to the axis matters to both of these concepts by can perceive a mass difference between two virtual objects rotated responses for each level of C/D ratio as the primary quantitative manipulation could have minimal effect on task performance [16] and definition. Thus, when discussing rotation, we cannot treat the entire with different C/D ratios. measure. users could adapt to it very well in a training task [17]. Azmandian et object as a point mass, but rather we need to understand its mass Using correct responses alone potentially puts us at risk that we 4.2.1 Experimental Design al. proposed “haptic retargeting” [2] and Lohse proposed “haptic distribution relative to the rotational axis. Consider again the golf club were only measuring participants’ ability to guess the correct answer change blindness” [21] to accomplish similar effects. Zenner et al. example: the forces required to rotate the golf club change drastically Figure 1 shows the virtual environment used in this experiment. In this instead of actually measuring their perception—users may have reported the detection thresholds when the user’s hand is displaced depending on where it is held and which direction it is rotated. This scene, two cubes with the same appearance (size, shape, and texture) guessed that the object that rotates slower “should” be the heavier one, visually, which could be applied for haptic retargeting purposes [46]. special feature of rotational motion gives us an opportunity to not only were placed side by side. The user was asked to rotate them in turn but they may not have been experiencing an actual sensation of mass Fujinawa et al. described a novel method for computationally manipulate the perception of an object’s total mass, but also the and make a two-alternative forced choice about which of them felt difference. To alleviate this issue, we also adopted a post-experiment designing a hand-held controller so its haptic properties could perception of how mass is distributed at a finer level of granularity. heavier than the other. The rotational control of the two cubes was questionnaire on the quality of induced haptic perception, proposed by convincingly represent a virtual counterpart even though they look It is important to point out that we treat total mass and mass scaled by difference C/D ratios. If the technique is effective, we Rietzler et al. [30]. Customized to fit the context of this experiment, different [8]. distribution as two separate physical properties in designing the expected that users would perceive that objects rotating with larger the questionnaire asked participants how much they agreed with the The interaction between visual and haptic perception has been pseudo-haptic techniques in this paper. On the one hand, two objects C/D ratios would feel heavier than the other, and vice versa. In this following six statements on a seven-point Likert scale (expressed to leveraged to influence many types of haptic sensations in VR, creating can have the same shape and total mass but distinct characteristics in experiment, the pivot point of rotation was always at the geometric the user using the labels “strongly disagree,” “disagree,” “somewhat a research area called “pseudo-haptics” [28]. It has found successful mass distribution; on the other hand, two objects can have the same center of the cube. disagree,” “neither agree nor disagree,” “somewhat agree,” “agree,” applications in changing the perception of friction [19], mass [7, 31, shape and the same relative mass distribution between different parts or “strongly agree”): 35], shape [3], stiffness [10], softness [27], torque [25] and force [29]. of the body, but different total masses (e.g., because one object is made Q1: I could feel that the two cubes have different weight Pseudo-haptic display of mass is usually achieved by visually of a different material). In sections 4 and 5, we describe techniques for Q2: The representation of weight difference felt realistic displacing the hand and/or the object, so a sense of heaviness is generating these two illusions and evaluate their effectiveness. In the Q3: I had the feeling of manipulating real objects generated though kinesthetic cues [7, 31, 35]. Research also finds that paper’s conclusion, we will discuss how these two aspects of pseudo- Q4: The representation of weight difference is sufficient for VR such a technique could be used to control fatigue during object lifting haptics can be combined for practical use. Q5: I liked this representation of weight difference [43]. Q6: I felt I have control Regarding the perception of an object’s mass distribution, Lukos In the questionnaires used in this paper, we exchanged the term 4 PSEUDO-HAPTIC DISPLAY OF TOTAL MASS DURING et al. [22] found that people change how they grab an object (in the Figure 1: The virtual environment for experiment I. “mass” with “weight”. Even though they have different meanings in ROTATION real world) if they can visually predict its center of mass (CoM). We physics and technically “mass” was the more accurate term here, we For each trial, the system randomly picked one of the cubes to assume that this will also hold true in VR, which is strong motivation felt “weight” was a more understandable expression for a general 4.1 The Technique assign it a non-1:1 rotational C/D ratio, while the other cube had a to develop techniques that adjust rotational motion based on an audience and could bring up what we were looking for in the ratio of 1:1 (used as the reference). The levels of the first C/D ratio are object’s mass distribution. Hashiguchi et al. used diminished reality to All the interaction techniques in this paper are based on the common questionnaires all the same, namely the subjective sensation of listed in Table 1 (note that the table lists the reciprocal of C/D ratios), hide part of a real rod to influence the perception of its mass and mass “simple virtual hand” metaphor, in which a zero-order mapping maps heaviness when manipulating these virtual objects. with values centered around 1.0. We can think of these values as being distribution [11]. Zenner et al. developed a “weight-shifting dynamic the hand-held controller’s movement to the object’s movement Each participant first went through an introduction session, which linearly scaled by the virtual mass of the items. For example, assuming haptic proxy” that could change its mass distribution on the fly [45], (translation and rotation) [26]. We use the “remote control” version of explained the task and the interface. Next, we provided a training 2080 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020 session to ensure that the participants could understand the ratio except those close to 1.0. Even at these levels, the average Figure 4). As mentioned in Section 4.2.2, although the identification CoM (where torque due to gravity does not exist). Thus, keeping a experimental procedure, interact with the virtual scene, and complete correctness was greater than 80%. of the heavier cube only demonstrates that the user could correctly constant C/D ratio specifies that there is no torque due to gravity, the two-alternative forced choice tasks. This training session included We treat the correctness percentage as ordinal data. For C/D ratios guess the result, the positive result of the questionnaire here indicates which means that the CoM is at the pivot. Note that the C/D ratio two trials with one of the cubes exhibiting downscaled rotation using of 1/0.2, 1/0.7, 1/1.3, 1/1.5, 1/1.8, the answers were 100% correct so that the illusion was indeed perceived as subjective sensations of mass should be technically be adjusted according to the direction of rotation the C/D ratio of 1/0.2, and another two trials with one cube showing no statistical test was needed. For the rest of the cases, one-sample differences. The results from Q6 of the questionnaire also show that since could change according to this direction. However, for simple upscaled rotation using the C/D ratio of 1/1.8. After the training, each Wilcoxon signed rank tests show that the results are all significantly adjusting the rotational C/D ratio did not compromise the perceived objects like cubes (used in the experiment described in Section 5.2) participant completed the 24 experimental trials. During both the higher than random chance of 50%, with p-values of 4.103e-14 (C/D controllability. Overall, this technique and experiment complement that rotate around the CoM, the slight change of would have little training and the experiment, no feedback was given after each trial; = 1/0.5), 0.0001 (C/D = 1/0.9) and 0.0003 (C/D = 1/1.1). the previous works by Dominjon et, al. that used translational C/D influence, so we decided to ignore this factor to simplify the technique instead, the system simply played a sound indicating the conclusion ratio to generate the perception of mass [7] and the experiments by and keep a constant C/D ratio. of the current trial and immediately loaded the next one. Finally, we Streit et, al. [41, 42] that used constrained rotation. Based on these gave the participant the post-experiment questionnaire. 1.00 results, adjusting the C/D ratio for both translation and rotation in VR 5.1.2 Dynamic Torque Technique (DT) object manipulation should generate a convincing illusion of mass One limitation of PV is that it requires modifying the pivot of rotation. 4.2.3 Apparatus and Participants variation. But in practical VR applications, it is preferred that users can grasp Participants sat in a chair to complete the experiment. A wireless HTC Figure 3 shows that users’ ability to perceive mass differences does objects at any point, motivating the need for a more general-purpose VIVE Pro head-mounted display was used to display the virtual 0.75 decrease as the C/D ratio gets closer to 1.0. A limitation of this current technique. The second technique, called “dynamic torque,” achieves environment (Figure 2A). This headset has a resolution of 1080 x 1200 experiment is that, due to our choice of C/D ratio levels, we were not by adjusting the rotational C/D ratio using the real-time mechanics of pixels per eye (2160 x 1200 pixels combined), a refresh rate of 90 Hz able to find the threshold of mass difference that will lead to the rotating object given an arbitrary pivot. and a field of view of 110 degrees. The user used a VIVE Pro detectability close to random chance. We consider finding that The general idea is shown in Figure 6. Assume we are using the controller (Figure 2C) in her non-dominant hand to select between the 0.50 threshold as part of the future work. same square-shaped virtual object in which the CoM is at the bottom two cubes and give answers using ray-casting, while a VIVE tracker right, but the rotational pivot is now at the geometric center rather than (Figure 2B) was used in her dominant hand to rotate the virtual cubes. 5 PSEUDO-HAPTIC DISPLAY OF MASS DISTRIBUTION at the CoM. This means that gravity will generate a torque relative to
The VE displayed a virtual representation of the VIVE Pro controller Correct percentage DURING ROTATION this pivot (shown in Figure 6 as a vector pointing into the 2D plane). (and the ray emitted from its tip) so the user could easily select items This torque affects how difficult it is to rotate the object based on its with ray-casting. The VIVE tracker itself was not displayed since 0.25 5.1 Techniques relationship with the vector of the rotational axis. If they lie in the visually seeing its orientation could potentially make it apparent that Recognizing that mass distribution also plays an important role in same direction (the angle between them is smaller than 90 degrees, as the cube’s rotation was scaled. The headset, controller, and tracker rotational motion, we leverage the same interplay between kinesthetic in the case of the clockwise rotation in Figure 6), the torque due to were all tracked by an HTC Lighthouse 2.0 inertial-optical tracking torque cues and visual movement cues to display pseudo-haptic mass gravity will aid the intended rotation. Hence, we scale up the rotational system. The interface and environment were developed in Unity 0.00 illusions at a finer level of granularity. We propose two different motion using a smaller C/D ratio in this case. Conversely, if the torque 2018.3.2f1. Rendering was performed by a PC with an Intel Core i7- 0.2 0.5 0.7 0.9 1.1 1.3 1.5 1.8 techniques, both leveraging the intuitive relationship between torque, due to gravity is opposed to the direction of the rotational axis, we 1 / Control display ratio 8700k CPU, 16GB of RAM, and an Nvidia GTX 1070 GPU. rotational motion and mass expressed by (1). scale down the rotation with a larger C/D ratio, as in the counter- Figure 3: Percentage of correct responses at each C/D ratio in clockwise rotation in Figure 6. experiment I. Error bars represent standard deviation. 5.1.1 Pivot Technique (PV) The first technique, called “the pivot technique,” simply sets the pivot Figure 4 shows the results from the questionnaire. One-sample point of rotation at the object’s CoM while maintaining a constant C/D Wilcoxon signed rank tests show that the mean responses are ratio for rotation around any axis. We expect that users will be able to significantly higher than 0 for all questions, with p-values of 0.0004 naturally feel where the CoM is when it coincides with the rotational (Q1), 0.0006 (Q2), 0.0008 (Q3), 0.0002 (Q4), 0.002 (Q5), 0.0001 pivot point. Figure 5 illustrates this idea in 2D: imagine that the virtual $)) $# #() (Q6). object is said to be made of the same material but does not have even ' ) $#$ ) ' ) $#$ '+ )-(! ( '+ )-(! ( density across its body. Starting from the upper-left corner, the density *%'$)) $# $,#'$)) $# 3 gradually becomes greater as we move towards the lower-right corner. (% (% x With this mass distribution, the CoM will lie close to the lower-right
Figure 2: A. the user used the VIVE tracker to manipulate the object 2 corner instead of at the geometric center. Using PV, the object rotates Figure 6: Illustration of the DT technique. The torque due to gravity is while holding the VIVE Pro controller to give answers and select x x around this CoM, as illustrated by Figure 5. a vector pointing into the 2D plane (marked ‘.’). It affects the C/D ratio between the two cubes in the first experiment. B: The VIVE tracker. C: xx x based on the axis and direction of rotation. The VIVE Pro controller. 1 DT requires that rotational motion should be scaled according to A key aspect of the experiment was the choice of the physical the direction of rotation at each instant. Thus, we need to adjust the proxy, as its physical shape and mass could potentially bias mass- 0 rotational C/D ratio on a frame-to-frame basis. In the following related perception. Since experiment I was about distinguishing the paragraphs, we derive how this ratio can be computed for each frame
overall mass of two virtual objects that look identical, as long as the Score (7 point Likert) in a way that is both feasible and intuitive. −1 same physical proxy is used in the same way when manipulating the For each frame, the so-called rotational C/D ratio is the ratio two cubes, the shape and mass of the proxy should not bias the user’s between the amount of rotation of the physical proxy ( ) and the choice. On the other hand, it is desirable to choose a proxy that is small −2 amount of rotation of the virtual object ( ). Given a desired C/D ratio 40 and light enough to be easy for the user to hold and rotate. We ended and measured in the current frame, we can compute . Figure 5: Illustration of the PV technique. The object rotates around 5 up using the VIVE tracker (Figure 2B) as it is lightweight and easy to First we recognize that there are two sources of torque in this the CoM instead of the geometric center. 40 5 manipulate with one or two hands. −3 system: the torque from hand input ( ) and the torque from Throughout the experiment, the participant sat at a table, so he Q1 Q2 Q3 Q4 Q5 Q6 In remote object manipulation, we expect the user to feel that he is gravity ( ). We re-write (1) into (2): could put the controller down if he wanted to use both hands to Question $0-2. gripping the object right at the pivot point of rotation. Hence, PV /4 (2) manipulate the VIVE tracker (Figure 2A). The table could also Figure 4: Box plot of subjective questionnaire results in experiment I. creates the illusion that the user is always gripping the object at its $ Mean values are indicated by an ‘x.’ provide elbow support if needed to mitigate potential fatigue. CoM. Intuitively, this is how people tend to grab an object in the real We turn this equation$0-2. into6$ a general/ 8 # relationship between torque, A total of 17 participants (4 females, 13 males) were recruited, all world , because gravity does not create any torque relative to the CoM, mass distribution and rotation that loosely follows the KSD principle: 4.3 Discussion between 20 and 30 years old. Among them, only two of them making the object more controllable [22]. PV is designed so that users (3) identified their left hand as the dominant hand. Experiment II (Section The results of experiment I strongly support the effectiveness of the see a constant rotational speed (C/D ratio) around the pivot point, 5.3) also used this same group of participants. C/D ratio technique for producing a pseudo-haptic effect making one indicating that this point is the CoM, and thus giving the user In (3), is the torque$0-2. from6$/4 hand5 input,5 representing how much kinesthetic effort the user is exerting to rotate the object in that frame. virtual object seem heavier than another. Not only could participants information about the object’s mass distribution. 0-2. $ 4.2.4 Results robustly identify the heavier object, even with small mass differences This technique also loosely conforms to the KSD principles The value of this component should be determined by how the hand- Figure 3 shows the correctness results. As the figure shows, (Figure 3), but also they affirmed the technique’s power through the following equation (1). According to (1), the overall torque of the held proxy is being rotated. The other torque component, , is the participants achieved almost 100% correctness at all levels of C/D questionnaire, in which they indicated that the mass difference felt specified virtual torque generated by gravity acting on the object’s system should be influenced by gravity, which in turn should affect $/4 strong (Q1), realistic (Q2, Q3), sufficient (Q4) and desirable (Q5) (see the rotational speed (C/D ratio), unless the pivot coincides with the mass, projected onto the rotational axis. Note that since we are using YU AND BOWMAN: PSEUDO-HAPTIC DISPLAY OF MASS AND MASS DISTRIBUTION DURING OBJECT ROTATION IN VIRTUAL REALITY 2081 session to ensure that the participants could understand the ratio except those close to 1.0. Even at these levels, the average Figure 4). As mentioned in Section 4.2.2, although the identification CoM (where torque due to gravity does not exist). Thus, keeping a experimental procedure, interact with the virtual scene, and complete correctness was greater than 80%. of the heavier cube only demonstrates that the user could correctly constant C/D ratio specifies that there is no torque due to gravity, the two-alternative forced choice tasks. This training session included We treat the correctness percentage as ordinal data. For C/D ratios guess the result, the positive result of the questionnaire here indicates which means that the CoM is at the pivot. Note that the C/D ratio two trials with one of the cubes exhibiting downscaled rotation using of 1/0.2, 1/0.7, 1/1.3, 1/1.5, 1/1.8, the answers were 100% correct so that the illusion was indeed perceived as subjective sensations of mass should be technically be adjusted according to the direction of rotation the C/D ratio of 1/0.2, and another two trials with one cube showing no statistical test was needed. For the rest of the cases, one-sample differences. The results from Q6 of the questionnaire also show that since could change according to this direction. However, for simple upscaled rotation using the C/D ratio of 1/1.8. After the training, each Wilcoxon signed rank tests show that the results are all significantly adjusting the rotational C/D ratio did not compromise the perceived objects like cubes (used in the experiment described in Section 5.2) participant completed the 24 experimental trials. During both the higher than random chance of 50%, with p-values of 4.103e-14 (C/D controllability. Overall, this technique and experiment complement that rotate around the CoM, the slight change of would have little training and the experiment, no feedback was given after each trial; = 1/0.5), 0.0001 (C/D = 1/0.9) and 0.0003 (C/D = 1/1.1). the previous works by Dominjon et, al. that used translational C/D influence, so we decided to ignore this factor to simplify the technique instead, the system simply played a sound indicating the conclusion ratio to generate the perception of mass [7] and the experiments by and keep a constant C/D ratio. of the current trial and immediately loaded the next one. Finally, we Streit et, al. [41, 42] that used constrained rotation. Based on these gave the participant the post-experiment questionnaire. 1.00 results, adjusting the C/D ratio for both translation and rotation in VR 5.1.2 Dynamic Torque Technique (DT) object manipulation should generate a convincing illusion of mass One limitation of PV is that it requires modifying the pivot of rotation. 4.2.3 Apparatus and Participants variation. But in practical VR applications, it is preferred that users can grasp Participants sat in a chair to complete the experiment. A wireless HTC Figure 3 shows that users’ ability to perceive mass differences does objects at any point, motivating the need for a more general-purpose VIVE Pro head-mounted display was used to display the virtual 0.75 decrease as the C/D ratio gets closer to 1.0. A limitation of this current technique. The second technique, called “dynamic torque,” achieves environment (Figure 2A). This headset has a resolution of 1080 x 1200 experiment is that, due to our choice of C/D ratio levels, we were not by adjusting the rotational C/D ratio using the real-time mechanics of pixels per eye (2160 x 1200 pixels combined), a refresh rate of 90 Hz able to find the threshold of mass difference that will lead to the rotating object given an arbitrary pivot. and a field of view of 110 degrees. The user used a VIVE Pro detectability close to random chance. We consider finding that The general idea is shown in Figure 6. Assume we are using the controller (Figure 2C) in her non-dominant hand to select between the 0.50 threshold as part of the future work. same square-shaped virtual object in which the CoM is at the bottom two cubes and give answers using ray-casting, while a VIVE tracker right, but the rotational pivot is now at the geometric center rather than (Figure 2B) was used in her dominant hand to rotate the virtual cubes. 5 PSEUDO-HAPTIC DISPLAY OF MASS DISTRIBUTION at the CoM. This means that gravity will generate a torque relative to
The VE displayed a virtual representation of the VIVE Pro controller Correct percentage DURING ROTATION this pivot (shown in Figure 6 as a vector pointing into the 2D plane). (and the ray emitted from its tip) so the user could easily select items This torque affects how difficult it is to rotate the object based on its with ray-casting. The VIVE tracker itself was not displayed since 0.25 5.1 Techniques relationship with the vector of the rotational axis. If they lie in the visually seeing its orientation could potentially make it apparent that Recognizing that mass distribution also plays an important role in same direction (the angle between them is smaller than 90 degrees, as the cube’s rotation was scaled. The headset, controller, and tracker rotational motion, we leverage the same interplay between kinesthetic in the case of the clockwise rotation in Figure 6), the torque due to were all tracked by an HTC Lighthouse 2.0 inertial-optical tracking torque cues and visual movement cues to display pseudo-haptic mass gravity will aid the intended rotation. Hence, we scale up the rotational system. The interface and environment were developed in Unity 0.00 illusions at a finer level of granularity. We propose two different motion using a smaller C/D ratio in this case. Conversely, if the torque 2018.3.2f1. Rendering was performed by a PC with an Intel Core i7- 0.2 0.5 0.7 0.9 1.1 1.3 1.5 1.8 techniques, both leveraging the intuitive relationship between torque, due to gravity is opposed to the direction of the rotational axis, we 1 / Control display ratio 8700k CPU, 16GB of RAM, and an Nvidia GTX 1070 GPU. rotational motion and mass expressed by (1). scale down the rotation with a larger C/D ratio, as in the counter- Figure 3: Percentage of correct responses at each C/D ratio in clockwise rotation in Figure 6. experiment I. Error bars represent standard deviation. 5.1.1 Pivot Technique (PV) The first technique, called “the pivot technique,” simply sets the pivot CoM Figure 4 shows the results from the questionnaire. One-sample point of rotation at the object’s CoM while maintaining a constant C/D pivot Wilcoxon signed rank tests show that the mean responses are ratio for rotation around any axis. We expect that users will be able to significantly higher than 0 for all questions, with p-values of 0.0004 naturally feel where the CoM is when it coincides with the rotational gravity through CoM (Q1), 0.0006 (Q2), 0.0008 (Q3), 0.0002 (Q4), 0.002 (Q5), 0.0001 pivot point. Figure 5 illustrates this idea in 2D: imagine that the virtual Rotation in the Rotation against (Q6). direction of the direction of object is said to be made of the same material but does not have even gravity scales gravity scales torque due to gravity density across its body. Starting from the upper-left corner, the density up rotation down rotation speed 3 gradually becomes greater as we move towards the lower-right corner. speed x With this mass distribution, the CoM will lie close to the lower-right Figure 2: A. the user used the VIVE tracker to manipulate the object corner instead of at the geometric center. Using PV, the object rotates Figure 6: Illustration of the DT technique. The torque due to gravity is 2 a vector pointing into the 2D plane (marked ‘×’). It affects the C/D ratio while holding the VIVE Pro controller to give answers and select x x around this CoM, as illustrated by Figure 5. between the two cubes in the first experiment. B: The VIVE tracker. C: xx x based on the axis and direction of rotation. The VIVE Pro controller. 1 DT requires that rotational motion should be scaled according to A key aspect of the experiment was the choice of the physical the direction of rotation at each instant. Thus, we need to adjust the proxy, as its physical shape and mass could potentially bias mass- 0 CoM rotational C/D ratio on a frame-to-frame basis. In the following related perception. Since experiment I was about distinguishing the paragraphs, we derive how this ratio can be computed for each frame
overall mass of two virtual objects that look identical, as long as the Score (7 point Likert) in a way that is both feasible and intuitive. −1 same physical proxy is used in the same way when manipulating the For each frame, the so-called rotational C/D ratio is the ratio two cubes, the shape and mass of the proxy should not bias the user’s between the amount of rotation of the physical proxy ( ) and the choice. On the other hand, it is desirable to choose a proxy that is small −2 amount of rotation of the virtual object ( ). Given a desired C/D ratio 40 and light enough to be easy for the user to hold and rotate. We ended and measured in the current frame, we can compute . Figure 5: Illustration of the PV technique. The object rotates around 5 up using the VIVE tracker (Figure 2B) as it is lightweight and easy to First we recognize that there are two sources of torque in this the CoM instead of the geometric center. 40 5 manipulate with one or two hands. −3 system: the torque from hand input ( ) and the torque from Throughout the experiment, the participant sat at a table, so he Q1 Q2 Q3 Q4 Q5 Q6 In remote object manipulation, we expect the user to feel that he is gravity ( ). We re-write (1) into (2): could put the controller down if he wanted to use both hands to Question $0-2. gripping the object right at the pivot point of rotation. Hence, PV /4 (2) manipulate the VIVE tracker (Figure 2A). The table could also Figure 4: Box plot of subjective questionnaire results in experiment I. creates the illusion that the user is always gripping the object at its $ Mean values are indicated by an ‘x.’ provide elbow support if needed to mitigate potential fatigue. CoM. Intuitively, this is how people tend to grab an object in the real We turn this equation$0-2. into6$ a general/ 8 # relationship between torque, A total of 17 participants (4 females, 13 males) were recruited, all world , because gravity does not create any torque relative to the CoM, mass distribution and rotation that loosely follows the KSD principle: 4.3 Discussion between 20 and 30 years old. Among them, only two of them making the object more controllable [22]. PV is designed so that users (3) identified their left hand as the dominant hand. Experiment II (Section The results of experiment I strongly support the effectiveness of the see a constant rotational speed (C/D ratio) around the pivot point, 5.3) also used this same group of participants. C/D ratio technique for producing a pseudo-haptic effect making one indicating that this point is the CoM, and thus giving the user In (3), is the torque$0-2. from6$/4 hand5 input,5 representing how much kinesthetic effort the user is exerting to rotate the object in that frame. virtual object seem heavier than another. Not only could participants information about the object’s mass distribution. 0-2. $ 4.2.4 Results robustly identify the heavier object, even with small mass differences This technique also loosely conforms to the KSD principles The value of this component should be determined by how the hand- Figure 3 shows the correctness results. As the figure shows, (Figure 3), but also they affirmed the technique’s power through the following equation (1). According to (1), the overall torque of the held proxy is being rotated. The other torque component, , is the participants achieved almost 100% correctness at all levels of C/D questionnaire, in which they indicated that the mass difference felt specified virtual torque generated by gravity acting on the object’s system should be influenced by gravity, which in turn should affect $/4 strong (Q1), realistic (Q2, Q3), sufficient (Q4) and desirable (Q5) (see the rotational speed (C/D ratio), unless the pivot coincides with the mass, projected onto the rotational axis. Note that since we are using 2082 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020 a zero-order mapping between the physical proxy’s movement and the these other seven having the same mass). The heavier sub-cube is increase the influence of the physical proxy’s rotation, so the same virtual object’s movement, the axis of rotation for the object is solely randomly picked by the system and unknown to the user, since the kinesthetic effort from the hand will result in more rotation of the # #) ' # ), #) ),$) # &* ( determined by the rotation direction of the proxy in that frame. The sub-cubes all have the same appearance. The system uses either PV or virtual object. Enlarging will increase the influence of torque due # #)!- ') #'#$" # $ torque from gravity should not influence this direction; it should only DT to display the mass distribution, and the user must judge which to gravity, so the resulting C/D ratio will fluctuate more dramatically & influence the amount of rotation. This reduces the problem from 3D sub-cube is heavier after manipulating the object (an eight-alternative when the rotational axis changes. Enlarging will increase the rotation to 2D rotation. Thus, torque due to gravity is projected onto forced choice task). influence of the moment of inertia of the virtual object, so that it is 80% ' the rotational axis. By adjusting the ratio between the mass of the heavier sub-cube more reluctant to move. After empirical testing, we set the following On the right side of (3), is the specified moment of inertia of the and the lighter ones, the CoM of the entire box will move along the values: ; ; . virtual object relative to the rotational axis. Again, is the amount diagonal of the heavy sub-cube (shown in two dimensions in Figure Note that since the display of mass distribution is independent from 5 % & ' of rotation for the virtual object, which is the value we are looking for. 7, right). The larger this ratio is, the closer the CoM is placed to the the display 8 of total mass8 (explained 8 at the end of Section 3), the overall 5 Next, we reform the general relationship of (3) into an equation by center of the heavy sub-cube. We set the total mass of the cube at 100g scale of the C/D ratio for both translation and rotation should not affect 60% giving each component an empirically determined constant and used four levels of mass ratio to create four conditions, as shown the perception of mass distribution. One can uniformly scale the C/D ( : in Table 2. These values were chosen so that the positions of the CoM ratio of translation (Dominjon et, al. [7]) and/or rotation (Section 4) to would be evenly spread on the diagonal, as shown on the right side of display the object’s total mass while still using either PV or DT to % & ' (4) : Figure 7. For each condition, PV used the corresponding CoM position display the mass distribution. Thus, we do not need to separate Technique 40% DT We approximate the% 0-2. kinesthetic& /4 effort' from5 5 hand input ( by $ 6 $ 8 as the rotational pivot, while DT used the corresponding values in translation from rotation in this experiment (unlike experiment I, in PV the product between the physical proxy’s moment of inertia ( and 0-2. Table 2 to drive its computation of using formula (7). which we allowed only rotation, since the C/D ratio of translation how much it was rotated in the current frame ( : $ : 40 To carry out the actual calculation of (7), several values on its right would affect the perception of total object mass). To maximize the Correct percentage : 5 40 (5) side need to be computed or empirically set. First, we use a constant potential applicability of these techniques, we gave the user full six : value to approximate based on the size and mass of the VIVE DOF control in this experiment. To solve for , we re-organize (5) into (6): 20% %4040 6&$/4 8'55 tracker: . 40 5 (6) ' 5.2.2 Procedure and Measures 40 Table8 2: Mass ratio conditions in experiment II. This experiment used a within-subject design—each participant used In this formula,5 all the% 40 components40 & /4 on 'the5 right side can be 8 9 6 $ : both techniques in turn for the same set of tasks. The order of computed ( , empirically set , or measured in Condition Condition Condition Condition presentation was counter-balanced. For each technique, there were 0% real-time from the user’s input . 1 2 3 4 40$/45: 9%&': four repetitions for each of the four conditions (Table 2), resulting in Directly using (6) may cause a problem: the torque due to gravity Heavy sub- 3:1 19:319:3 13:1 33:1 940: 30g 47.5g 65g 82.5g a total of 32 trials Mass ratio: heavy cube / light cube can counter the hand’s intended rotation while having a very large cube mass value, which could result in a rotation angle of zero or in the opposite per participant. For each technique, the order of the Light sub- 9 !! 7 ! 7 direction of the hand’s input. This would break the zero-order 10g 7.5g 5g 2.5g 16 trials was randomized. For each trial, the position of the heavy sub- Figure 8: Percentage of correct responses by each technique in cube mass ! " : mapping of the control. To overcome this issue, we set a minimum cube was also randomized. each mass ratio condition in experiment II. Error bars represent Heavy/light scale for the object’s rotation to be 10% of the physical proxy’s 3:1 6.33:1 13:1 33:1 The participant indicated which sub-cube was perceived to be standard deviation. ratio rotation (a maximum C/D ratio of 1/0.1), so the zero-order mapping heavier by selecting the corresponding numbered button (Figure 7, questionnaire on that technique before repeating the entire process for from hand input to the object’s movement is always maintained. Thus, left) using ray-casting. Each participant’s percentage of correct the second technique. Finally, the preference questionnaire was The virtual cube has a size of , with each we use the following formula to calculate the angle the virtual object answers was used as the primary quantitative measure. Note that since presented. sub-cube having a size of . Given the mass should rotate in each frame: this is an eight-alternative forced choice, random chance would lead properties (Table 2) and size of the object, 7 and 7 can be calculated to a 12.5% correctness rate. 5.2.3 Apparatus and Participants (7) using their definitions in physics given 7 a rotational 7 axis [9]. Here we /4 5 We also included the questionnaire [30] used in experiment I to Similar to Experiment I, we needed to carefully choose a proxy to briefly introduce how these values are computed.$ evaluate the quality of the haptic sensation. Modified to match the In the next5 8 ;9section, %we40 show40 6 the &values$/4: used'5 in the computation40< of avoid biasing the user’s mass-related perception. Specific to this (7) based on the specific virtual object used in the experiment. is computed by first calculating the torque due to gravity context of this experiment, the questionnaire asks the participant to experiment, the proxy’s shape and mass distribution need to be taken using the cross product between the radius vector (the vector from /4 / rate agreement with these statements on the same 7-point Likert scale: into consideration, as asymmetric shape or unevenly distributed mass pivot$ to CoM) and the object’s total gravitational force (marked $as Q1: I could feel that one cube is heavier than the other seven could potentially influence the user’s perception of which (virtual) for the big cube) : Q2: The representation of weight distribution felt realistic sub-cube was the heavier one. Ideally, the proxy would have evenly 5 Q3: I had the feeling of manipulating a real object distributed mass in all directions, which would make it generic enough Q4: The representation of weight distribution is sufficient for VR to represent the cube at an arbitrary orientation without biasing the Then this vector of is projected onto the unit vector of the $/ 8 75 Q5: I liked this representation of weight distribution distribution of mass (note the orientation of the proxy does not rotational axis to compute . / Q6: I felt I have control necessarily match the orientation of the virtual cube at any time The computation of $ is a little more complicated. For a composite /4 We also asked participants to indicate which of the two techniques because of the scaled C/D ratio). Thus, the ideal proxy is a sphere built object such as the big cube $we used, can be generated by adding the 5 they preferred according to the same criteria. Specifically, we asked with homogeneous material and even density. However, it was non- moment of inertia of each individual component. Hence, is the sum 5 the following questions: trivial to track a sphere’s rotation accurately in our lab. We ended up of the moment of inertia from each sub-cube (marked as ): Q1: Which technique gave you a stronger sense of weight 5 using the same VIVE tracker (Figure 2B) as a reasonable compromise, +-(** distribution (one of the sub-cubes is heavier than the others)? 51 as its form factor is symmetric along two dimensions. Its overall light ) Q2: Which representation of weight distribution felt more To compute each , the parallel axis theorem is applied. This weight also helps to reduce the unwanted influence of the physical 5 81*& 51 realistic ? ) '(** ( theorem states that, if we know an object’s moment of inertia when it proxy’s shape and mass distribution. 51 Q3: Which technique gave you a sensation that’s closer to the is rotating about an axis passing through its CoM (marked as Hence, the same apparatus from experiment I (Section 4.2.3) was $(( ! $ feeling of manipulating real objects? for each sub-cube), we can further deduce its moment of inertia used here. The VIVE tracker was used to manipulate the object and Q4: Which representation of weight distribution felt more when it’s rotating around any axis that’s parallel to the original one. the VIVE controller was used to select answers. The participants were 51+3, sufficient for VR applications? Figure 7: Left: VE for experiment II. Right: the four possible CoM Such calculation uses the distance between the two axes ( ) and the the same group of 17 people who completed experiment I. They Q5: Which technique do you prefer? positions in experiment II. mass of the object (marked as for each sub-cube): participated in experiment II immediately following experiment I, in Q6: Which technique felt easier to control? a single session of approximately 40 minutes. 5.2 Experiment II 5 Participants first went through a tutorial session, in which the ' concept of mass distribution and the experimental task were 5.2.4 Results To evaluate the effectiveness of the two proposed techniques, we For our cube object,51 8 is51+3, listed6 in Table5 2, can be computed designed an experiment to test whether the mass distribution displayed using the distance from each sub-cube’s geometric center (also the explained. Similar to experiment I, participants first went through a Figure 8 presents the correctness results from experiment II. Again, 5 through these techniques could be correctly perceived. CoM for each sub-cube) to the current rotation axis. is training session to practice the task four times with the first technique. we treated the correctness percentage as ordinal data and performed generated using the following formula no matter the direction of All of these four trials used the 33:1 ratio as the mass ratio between an ordinal logistic regression, using the technique and mass ratio 51+3, 5.2.1 Experimental Design rotation, with being the length of the sub-cube’s edge: the heavy sub-cube and each of the light sub-cubes (condition 4 in condition as the two predictors. Both technique (p < 0.0001) and mass The experimental task was to manipulate the yellow cube shown in Table 2). After the training session, the participant was asked to ratio condition (p < 0.001) were significant factors, and there was no complete the 16 experimental trials. The system did not give any Figure 7, left, which is composed of eight sub-cubes at the corners. & significant interaction between these two factors. We conducted a Finally, we need to set the constants , and . These values feedback on correctness during the practice or experimental trials. Among these sub-cubes, one is heavier than the other seven (with 51+3, 8( 5 pair-wise comparison between PV and DT in each condition using determine the influence of each component of (7). Enlarging will After using the first technique, participants completed the Wilcoxon signed rank test with Bonferroni correction. The result % & ' % YU AND BOWMAN: PSEUDO-HAPTIC DISPLAY OF MASS AND MASS DISTRIBUTION DURING OBJECT ROTATION IN VIRTUAL REALITY 2083 a zero-order mapping between the physical proxy’s movement and the these other seven having the same mass). The heavier sub-cube is increase the influence of the physical proxy’s rotation, so the same virtual object’s movement, the axis of rotation for the object is solely randomly picked by the system and unknown to the user, since the kinesthetic effort from the hand will result in more rotation of the # #) ' # ), #) ),$) # &* ( determined by the rotation direction of the proxy in that frame. The sub-cubes all have the same appearance. The system uses either PV or virtual object. Enlarging will increase the influence of torque due # #)!- ') #'#$" # $ torque from gravity should not influence this direction; it should only DT to display the mass distribution, and the user must judge which to gravity, so the resulting C/D ratio will fluctuate more dramatically & influence the amount of rotation. This reduces the problem from 3D sub-cube is heavier after manipulating the object (an eight-alternative when the rotational axis changes. Enlarging will increase the rotation to 2D rotation. Thus, torque due to gravity is projected onto forced choice task). influence of the moment of inertia of the virtual object, so that it is 80% ' the rotational axis. By adjusting the ratio between the mass of the heavier sub-cube more reluctant to move. After empirical testing, we set the following On the right side of (3), is the specified moment of inertia of the and the lighter ones, the CoM of the entire box will move along the values: ; ; . virtual object relative to the rotational axis. Again, is the amount diagonal of the heavy sub-cube (shown in two dimensions in Figure Note that since the display of mass distribution is independent from 5 % & ' of rotation for the virtual object, which is the value we are looking for. 7, right). The larger this ratio is, the closer the CoM is placed to the the display 8 of total mass8 (explained 8 at the end of Section 3), the overall 5 Next, we reform the general relationship of (3) into an equation by center of the heavy sub-cube. We set the total mass of the cube at 100g scale of the C/D ratio for both translation and rotation should not affect 60% giving each component an empirically determined constant and used four levels of mass ratio to create four conditions, as shown the perception of mass distribution. One can uniformly scale the C/D ( : in Table 2. These values were chosen so that the positions of the CoM ratio of translation (Dominjon et, al. [7]) and/or rotation (Section 4) to would be evenly spread on the diagonal, as shown on the right side of display the object’s total mass while still using either PV or DT to % & ' (4) : Figure 7. For each condition, PV used the corresponding CoM position display the mass distribution. Thus, we do not need to separate Technique 40% DT We approximate the% 0-2. kinesthetic& /4 effort' from5 5 hand input ( by $ 6 $ 8 as the rotational pivot, while DT used the corresponding values in translation from rotation in this experiment (unlike experiment I, in PV the product between the physical proxy’s moment of inertia ( and 0-2. Table 2 to drive its computation of using formula (7). which we allowed only rotation, since the C/D ratio of translation how much it was rotated in the current frame ( : $ : 40 To carry out the actual calculation of (7), several values on its right would affect the perception of total object mass). To maximize the Correct percentage : 5 40 (5) side need to be computed or empirically set. First, we use a constant potential applicability of these techniques, we gave the user full six : value to approximate based on the size and mass of the VIVE DOF control in this experiment. To solve for , we re-organize (5) into (6): 20% %4040 6&$/4 8'55 tracker: . 40 5 (6) ' 5.2.2 Procedure and Measures 40 Table8 2: Mass ratio conditions in experiment II. This experiment used a within-subject design—each participant used In this formula,5 all the% 40 components40 & /4 on 'the5 right side can be 8 9 6 $ : both techniques in turn for the same set of tasks. The order of computed ( , empirically set , or measured in Condition Condition Condition Condition presentation was counter-balanced. For each technique, there were 0% real-time from the user’s input . 1 2 3 4 40$/45: 9%&': four repetitions for each of the four conditions (Table 2), resulting in Directly using (6) may cause a problem: the torque due to gravity Heavy sub- 3:1 19:319:3 13:1 33:1 940: 30g 47.5g 65g 82.5g a total of 32 trials Mass ratio: heavy cube / light cube can counter the hand’s intended rotation while having a very large cube mass value, which could result in a rotation angle of zero or in the opposite per participant. For each technique, the order of the Light sub- 9 !! 7 ! 7 direction of the hand’s input. This would break the zero-order 10g 7.5g 5g 2.5g 16 trials was randomized. For each trial, the position of the heavy sub- Figure 8: Percentage of correct responses by each technique in cube mass ! " : mapping of the control. To overcome this issue, we set a minimum cube was also randomized. each mass ratio condition in experiment II. Error bars represent Heavy/light scale for the object’s rotation to be 10% of the physical proxy’s 3:1 6.33:1 13:1 33:1 The participant indicated which sub-cube was perceived to be standard deviation. ratio rotation (a maximum C/D ratio of 1/0.1), so the zero-order mapping heavier by selecting the corresponding numbered button (Figure 7, questionnaire on that technique before repeating the entire process for from hand input to the object’s movement is always maintained. Thus, left) using ray-casting. Each participant’s percentage of correct the second technique. Finally, the preference questionnaire was The virtual cube has a size of , with each we use the following formula to calculate the angle the virtual object answers was used as the primary quantitative measure. Note that since presented. sub-cube having a size of . Given the mass should rotate in each frame: this is an eight-alternative forced choice, random chance would lead properties (Table 2) and size of the object, 7 and 7 can be calculated to a 12.5% correctness rate. 5.2.3 Apparatus and Participants (7) using their definitions in physics given 7 a rotational 7 axis [9]. Here we /4 5 We also included the questionnaire [30] used in experiment I to Similar to Experiment I, we needed to carefully choose a proxy to briefly introduce how these values are computed.$ evaluate the quality of the haptic sensation. Modified to match the In the next5 8 ;9section, %we40 show40 6 the &values$/4: used'5 in the computation40< of avoid biasing the user’s mass-related perception. Specific to this (7) based on the specific virtual object used in the experiment. is computed by first calculating the torque due to gravity context of this experiment, the questionnaire asks the participant to experiment, the proxy’s shape and mass distribution need to be taken using the cross product between the radius vector (the vector from /4 / rate agreement with these statements on the same 7-point Likert scale: into consideration, as asymmetric shape or unevenly distributed mass pivot$ to CoM) and the object’s total gravitational force (marked $as Q1: I could feel that one cube is heavier than the other seven could potentially influence the user’s perception of which (virtual) for the big cube) : Q2: The representation of weight distribution felt realistic sub-cube was the heavier one. Ideally, the proxy would have evenly 5 Q3: I had the feeling of manipulating a real object distributed mass in all directions, which would make it generic enough Q4: The representation of weight distribution is sufficient for VR to represent the cube at an arbitrary orientation without biasing the Then this vector of is projected onto the unit vector of the $/ 8 75 Q5: I liked this representation of weight distribution distribution of mass (note the orientation of the proxy does not rotational axis to compute . / Q6: I felt I have control necessarily match the orientation of the virtual cube at any time The computation of $ is a little more complicated. For a composite /4 We also asked participants to indicate which of the two techniques because of the scaled C/D ratio). Thus, the ideal proxy is a sphere built object such as the big cube $we used, can be generated by adding the 5 they preferred according to the same criteria. Specifically, we asked with homogeneous material and even density. However, it was non- moment of inertia of each individual component. Hence, is the sum 5 the following questions: trivial to track a sphere’s rotation accurately in our lab. We ended up of the moment of inertia from each sub-cube (marked as ): Q1: Which technique gave you a stronger sense of weight 5 using the same VIVE tracker (Figure 2B) as a reasonable compromise, +-(** distribution (one of the sub-cubes is heavier than the others)? 51 as its form factor is symmetric along two dimensions. Its overall light ) Q2: Which representation of weight distribution felt more To compute each , the parallel axis theorem is applied. This weight also helps to reduce the unwanted influence of the physical 5 81*& 51 realistic ? ) '(** ( theorem states that, if we know an object’s moment of inertia when it proxy’s shape and mass distribution. 51 Q3: Which technique gave you a sensation that’s closer to the is rotating about an axis passing through its CoM (marked as Hence, the same apparatus from experiment I (Section 4.2.3) was $(( ! $ feeling of manipulating real objects? for each sub-cube), we can further deduce its moment of inertia used here. The VIVE tracker was used to manipulate the object and Q4: Which representation of weight distribution felt more when it’s rotating around any axis that’s parallel to the original one. the VIVE controller was used to select answers. The participants were 51+3, sufficient for VR applications? Figure 7: Left: VE for experiment II. Right: the four possible CoM Such calculation uses the distance between the two axes ( ) and the the same group of 17 people who completed experiment I. They Q5: Which technique do you prefer? positions in experiment II. mass of the object (marked as for each sub-cube): participated in experiment II immediately following experiment I, in Q6: Which technique felt easier to control? a single session of approximately 40 minutes. 5.2 Experiment II 5 Participants first went through a tutorial session, in which the ' concept of mass distribution and the experimental task were 5.2.4 Results To evaluate the effectiveness of the two proposed techniques, we For our cube object,51 8 is51+3, listed6 in Table5 2, can be computed designed an experiment to test whether the mass distribution displayed using the distance from each sub-cube’s geometric center (also the explained. Similar to experiment I, participants first went through a Figure 8 presents the correctness results from experiment II. Again, 5 through these techniques could be correctly perceived. CoM for each sub-cube) to the current rotation axis. is training session to practice the task four times with the first technique. we treated the correctness percentage as ordinal data and performed generated using the following formula no matter the direction of All of these four trials used the 33:1 ratio as the mass ratio between an ordinal logistic regression, using the technique and mass ratio 51+3, 5.2.1 Experimental Design rotation, with being the length of the sub-cube’s edge: the heavy sub-cube and each of the light sub-cubes (condition 4 in condition as the two predictors. Both technique (p < 0.0001) and mass The experimental task was to manipulate the yellow cube shown in Table 2). After the training session, the participant was asked to ratio condition (p < 0.001) were significant factors, and there was no complete the 16 experimental trials. The system did not give any Figure 7, left, which is composed of eight sub-cubes at the corners. & significant interaction between these two factors. We conducted a Finally, we need to set the constants , and . These values feedback on correctness during the practice or experimental trials. Among these sub-cubes, one is heavier than the other seven (with 51+3, 8( 5 pair-wise comparison between PV and DT in each condition using determine the influence of each component of (7). Enlarging will After using the first technique, participants completed the Wilcoxon signed rank test with Bonferroni correction. The result % & ' % 2084 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020 confirmed that DT significantly outperformed PV in the first three the high effectiveness of DT in producing a pseudo-haptic display of conditions when the mass ratio was 3:1, 19:3 or 13:1, but not the last mass distribution. condition when this ratio was 33:1 (p = 0.008, 0.011, 0.044, 0.068 PV, on the other hand, seems to be much weaker in generating this respectively for conditions 1-4). We also compared the result of each illusion. Even though participants gave correct answers at a technique in each condition against random chance (12.5%) using significantly higher rate than random in two of the four conditions, one-sample Wilcoxon signed rank tests with Bonferroni correction. these conditions had very high mass ratios (13:1 and 33:1). Correctness levels for DT were significantly higher than random Participants using PV did not find the illusion to be strong (Q1), chance in all conditions (p = 0.002, 0.001, 0.001, 0.001 for conditions realistic (Q2, Q3), sufficient (Q4), or desirable (Q5), and it was 1-4). However, PV outperformed random chance significantly only generally considered as the sub-optimal choice when compared to DT. when the mass ratio was 13:1 and 33:1 (p = 2.044, 0.264, 0.010, 0.001 This could be seen in both the individual evaluation of PV (Figure 9) for conditions 1-4). These results are annotated in Figure 8. and the comparison between the two techniques (Figure 10). Figure 9 shows the results from the individual technique Both techniques were considered to be very easy to control (Q6), questionnaire. Wilcoxon signed rank tests showed that DT had and slightly more participants thought PV was more controllable. This significantly higher scores than PV on Q1, Q2, Q4 and Q5. The p- is not a surprise, as PV maintains the 1:1 rotational C/D ratio values are 0.008 for Q1, 0.027 for Q2, 0.386 for Q3, 0.043 for Q4, throughout, while DT changes it on the fly. When controllability is at 0.042 for Q5 and 0.503 for Q6. Responses for DT were significantly high priority, PV may still have some value even though its pseudo- higher than 0 for all the questions, with p-values of 0.0002, 0.008, haptic illusion seems to be much weaker. 0.001, 0.003, 0.008 and 0.005 for Q1-Q6. In the case of PV, the results A limitation of the DT technique is that there are several are only significantly higher than 0 on Q3 and Q6. The p-values are empirically determined constants in our algorithm. These were 0.128, 0.862, 0.0002, 0.606, 0.833 and 0.0005 for Q1-Q6. These specifically tweaked to fit the current experiment, and they may need results from statistical analysis are annotated in the figure. to be further modified when using DT for a new VE. Another limitation is that the influence from the mass distribution of the # #) ' # ), #) ),$) # &* ( physical proxy is not explored in this experiment, and we consider it as part of the future work. A limitation of the current experimental # #)!- ') # procedure is that we cannot be sure that the fixed order of the two experiments had no impact on the result; future studies could validate our findings without this ordering limitation. 3
x 2 x x 15 x x x x 1 x
x Technique 10 DT 0 x Choice PV x x DT
count PV
Score (7 point Likert) −1
5
−2
−3 0
Q1 Q2 Q3 Q4 Q5 Q6 Q1 Q2 Q3 Q4 Q5 Q6 Question Question Figure 10: Results of the preference questionnaire from experiment II. Figure 9: Results of the individual technique questionnaire in experiment II. Mean values are indicated by an ‘x.’ 6 CONCLUSIONS AND FUTURE WORK Figure 10 shows the results from the preference questionnaire. This research contributes several pseudo-haptic techniques to Except that slightly more participants believed that PV is easier to influence mass-related perceptions during rotational motion in VR control than DT, more people preferred DT on all other criteria. object manipulation. Experiment I showed that one can display a difference in total masses by scaling objects’ rotational C/D ratios. In 5.3 Discussion addition, we also found empirical evidence that by altering the C/D The results of experiment II show that DT is a powerful technique in ratio dynamically based on the physical principles that explain the generating the perception of mass distribution. When using DT, motion of rotation, we can effectively display certain aspects of mass participants could usually perceive which part of the object was distribution. These two pseudo-haptic techniques can be applied heavier even when the CoM was only slightly off the geometric center. simultaneously—scaling the base C/D ratio to alter the perception of Evaluation of subjective user experience also shows that the effect was total mass while adjusting this base C/D ratio dynamically relative to perceived to be strong (Q1), realistic (Q2, Q3), and sufficient (Q4), itself to display mass distribution. and that the technique was appreciated (Q5). The fact that a non-ideal The techniques can also be seamlessly combined with translation- proxy was used (Section 5.2.3) further demonstrates the power and based pseudo-haptic techniques to further enhance the display of applicability of this technique. Overall, this empirical evidence shows mass-related properties. In any VR application requiring realistic object manipulation, these techniques can be directly applied, which YU AND BOWMAN: PSEUDO-HAPTIC DISPLAY OF MASS AND MASS DISTRIBUTION DURING OBJECT ROTATION IN VIRTUAL REALITY 2085 confirmed that DT significantly outperformed PV in the first three the high effectiveness of DT in producing a pseudo-haptic display of could potentially enhance plausibility and the overall sense of [13] J. Hummel, J. Dodiya, R. Wolff, A. Gerndt, and T. Kuhlen, "An conditions when the mass ratio was 3:1, 19:3 or 13:1, but not the last mass distribution. presence. Such effects could also have a positive impact on the evaluation of two simple methods for representing heaviness in condition when this ratio was 33:1 (p = 0.008, 0.011, 0.044, 0.068 PV, on the other hand, seems to be much weaker in generating this enjoyment and engagement of the VR experience, which are crucial immersive virtual environments," in 2013 IEEE Symposium on 3D User respectively for conditions 1-4). We also compared the result of each illusion. Even though participants gave correct answers at a for certain applications such as entertainment, training and Interfaces (3DUI). pp. 87-94, IEEE, 2013. technique in each condition against random chance (12.5%) using significantly higher rate than random in two of the four conditions, psychotherapy. [14] B. E. Insko, M. Meehan, M. Whitton, and F. Brooks, “Passive haptics one-sample Wilcoxon signed rank tests with Bonferroni correction. these conditions had very high mass ratios (13:1 and 33:1). In the future, we plan to study the physical proxy’s shape, mass significantly enhances virtual environments,” Doctoral dissertation, Correctness levels for DT were significantly higher than random Participants using PV did not find the illusion to be strong (Q1), and mass distribution, to see how these factors could affect the University of North Carolina at Chapel Hill, 2001. chance in all conditions (p = 0.002, 0.001, 0.001, 0.001 for conditions realistic (Q2, Q3), sufficient (Q4), or desirable (Q5), and it was strength of the pseudo-haptic illusion. A follow-up study should also [15] L. Kohli, "Redirected touching: Warping space to remap passive haptics," 1-4). However, PV outperformed random chance significantly only generally considered as the sub-optimal choice when compared to DT. determine the threshold when the mass difference between the two in 2010 IEEE Symposium on 3D User Interfaces (3DUI). pp. 129-130, when the mass ratio was 13:1 and 33:1 (p = 2.044, 0.264, 0.010, 0.001 This could be seen in both the individual evaluation of PV (Figure 9) objects become indistinguishable. Moreover, we’d like to investigate IEEE, 2010. for conditions 1-4). These results are annotated in Figure 8. and the comparison between the two techniques (Figure 10). how the parameters used in these techniques connect to the perceived [16] L. Kohli, M. C. Whitton, and F. P. Brooks, "Redirected touching: The Figure 9 shows the results from the individual technique Both techniques were considered to be very easy to control (Q6), absolute values that quantify mass, mass difference and mass effect of warping space on task performance," in 2012 IEEE Symposium questionnaire. Wilcoxon signed rank tests showed that DT had and slightly more participants thought PV was more controllable. This distribution. To continue this line of research, we also plan to combine on 3D User Interfaces (3DUI). pp. 105-112, IEEE, 2012. significantly higher scores than PV on Q1, Q2, Q4 and Q5. The p- is not a surprise, as PV maintains the 1:1 rotational C/D ratio these two pseudo-haptic effects with translation-based techniques and [17] L. Kohli, M. C. Whitton, and F. P. Brooks, "Redirected Touching: values are 0.008 for Q1, 0.027 for Q2, 0.386 for Q3, 0.043 for Q4, throughout, while DT changes it on the fly. When controllability is at study their use in more realistic virtual environments, which could Training and adaptation in warped virtual spaces," in 2013 IEEE 0.042 for Q5 and 0.503 for Q6. Responses for DT were significantly high priority, PV may still have some value even though its pseudo- potentially identify measurable benefits on broad user experience Symposium on 3D User Interfaces (3DUI). pp. 79-86, IEEE, 2013. higher than 0 for all the questions, with p-values of 0.0002, 0.008, haptic illusion seems to be much weaker. from adding these pseudo-haptic illusions (e.g., on perceived [18] A. Lécuyer, “Simulating haptic feedback using vision: A survey of 0.001, 0.003, 0.008 and 0.005 for Q1-Q6. In the case of PV, the results A limitation of the DT technique is that there are several presence, fun and engagement). research and applications of pseudo-haptic feedback,” Presence: are only significantly higher than 0 on Q3 and Q6. The p-values are empirically determined constants in our algorithm. These were Teleoperators and Virtual Environments, vol. 18, no. 1, pp. 39-53, 2009. 0.128, 0.862, 0.0002, 0.606, 0.833 and 0.0005 for Q1-Q6. These specifically tweaked to fit the current experiment, and they may need REFERENCES [19] A. Lecuyer, S. Coquillart, A. Kheddar, P. Richard, and P. Coiffet, "Pseudo-haptic feedback: Can isometric input devices simulate force results from statistical analysis are annotated in the figure. to be further modified when using DT for a new VE. Another [1] E. L. Amazeen, and M. T. Turvey, “Weight perception and the haptic feedback?," in Proceedings of IEEE Virtual Reality 2000. pp. 83-90, limitation is that the influence from the mass distribution of the Journal of size–weight illusion are functions of the inertia tensor,” IEEE, 2000. # #) ' # ), #) ),$) # &* ( physical proxy is not explored in this experiment, and we consider it Experimental Psychology: Human perception and performance, vol. 22, [20] M. C. Lin, and M. Otaduy, Haptic rendering: foundations, algorithms, as part of the future work. A limitation of the current experimental no. 1, pp. 213, 1996. # #)!- ') # procedure is that we cannot be sure that the fixed order of the two and applications: AK Peters/CRC Press, 2008. [2] M. Azmandian, M. Hancock, H. Benko, E. Ofek, and A. D. Wilson, [21] A. L. Lohse, C. K. Kjær, E. Hamulic, I. G. Lima, T. H. Jensen, L. E. experiments had no impact on the result; future studies could validate "Haptic retargeting: Dynamic repurposing of passive haptics for our findings without this ordering limitation. Bruni, and N. C. Nilsson, "Leveraging Change Blindness for Haptic enhanced virtual reality experiences," in Proceedings of the 2016 chi 3 Remapping in Virtual Environments," in 2019 IEEE 5th Workshop on conference on human factors in computing systems. pp. 1968-1979, Everyday Virtual Reality (WEVR). pp. 1-5, IEEE, 2019. ACM, 2016. [22] J. Lukos, C. Ansuini, and M. Santello, “Choice of contact points during [3] Y. Ban, T. Kajinami, T. Narumi, T. Tanikawa, and M. Hirose, x multidigit grasping: effect of predictability of object center of mass 2 "Modifying an identified curved surface shape using pseudo-haptic x 15 location,” Journal of Neuroscience, vol. 27, no. 14, pp. 3894-3903, 2007. x effect," in 2012 IEEE Haptics Symposium (HAPTICS). pp. 211-216, x [23] S. C. Masin, and L. Crestoni, “Experimental demonstration of the sensory x x IEEE, 2012. x basis of the size-weight illusion,” Perception & Psychophysics, vol. 44, 1 x [4] G. P. Bingham, “Kinematic form and scaling: Further investigations on no. 4, pp. 309-312, 1988. the visual perception of lifted weight,” Journal of Experimental x [24] D. J. Murray, R. R. Ellis, C. A. Bandomir, and H. E. Ross, “Charpentier Psychology: Human Perception and Performance, vol. 13, no. 2, pp. 155, Technique (1891) on the size—weight illusion,” Perception & Psychophysics, vol. 10 DT 1987. 0 x Choice 61, no. 8, pp. 1681-1685, 1999. PV [5] D. Bowman, E. Kruijff, J. J. LaViola Jr, and I. Poupyrev, 3D User x x DT [25] A. Paljic, J.-M. Burkhardtt, and S. Coquillart, "Evaluation of pseudo- count PV Interfaces: Theory and Practice: Addison-Wesley, 2004. haptic feedback for simulating torque: a comparison between isometric [6] C.-H. Cheng, C.-C. Chang, Y.-H. Chen, Y.-L. Lin, J.-Y. Huang, P.-H. Score (7 point Likert) and elastic input devices," in 12th International Symposium on Haptic −1 Han, J.-C. Ko, and L.-C. Lee, "GravityCup: a liquid-based haptics for Interfaces for Virtual Environment and Teleoperator Systems, 2004. simulating dynamic weight in virtual reality," in Proceedings of the 24th 5 HAPTICS'04. Proceedings. pp. 216-223, IEEE, 2004. ACM Symposium on Virtual Reality Software and Technology. p. 51, [26] I. Poupyrev, M. Billinghurst, S. Weghorst, and T. Ichikawa, "The go-go −2 ACM, 2018. interaction technique: non-linear mapping for direct manipulation in [7] L. Dominjon, A. Lécuyer, J.-M. Burkhardt, P. Richard, and S. Richir, VR," in Proceedings of the 9th annual ACM symposium on User "Influence of control/display ratio on the perception of mass of interface software and technology. pp. 79-80, ACM, 1996. manipulated objects in virtual environments," in Proceeding of IEEE −3 [27] P. Punpongsanon, D. Iwai, and K. Sato, “Softar: Visually manipulating 0 Virtual Reality 2005 (VR 2005). pp. 19-25, IEEE, 2005. haptic softness perception in spatial augmented reality,” IEEE Q1 Q2 Q3 Q4 Q5 Q6 Q1 Q2 Q3 Q4 Q5 Q6 [8] E. Fujinawa, S. Yoshida, Y. Koyama, T. Narumi, T. Tanikawa, and M. Question transactions on visualization and computer graphics, vol. 21, no. 11, pp. Question Hirose, "Computational design of hand-held VR controllers using haptic 1279-1288, 2015. Figure 10: Results of the preference questionnaire from experiment II. shape illusion," in Proceedings of the 23rd ACM Symposium on Virtual Figure 9: Results of the individual technique questionnaire in [28] A. Pusch, and A. Lécuyer, "Pseudo-haptics: from the theoretical Reality Software and Technology. p. 28, ACM, 2017. foundations to practical system design guidelines," in Proceedings of the experiment II. Mean values are indicated by an ‘x.’ ONCLUSIONS AND UTURE ORK [9] H. Goldstein, C. Poole, and J. Safko, "Classical mechanics," AAPT, 6 C F W 13th international conference on multimodal interfaces. pp. 57-64, ACM, 2002. Figure 10 shows the results from the preference questionnaire. This research contributes several pseudo-haptic techniques to 2011. influence mass-related perceptions during rotational motion in VR [10] N. Gurari, K. J. Kuchenbecker, and A. M. Okamura, "Stiffness [29] A. Pusch, O. Martin, and S. Coquillart, "Hemp--hand-displacement- Except that slightly more participants believed that PV is easier to discrimination with visual and proprioceptive cues," in World Haptics object manipulation. Experiment I showed that one can display a based pseudo-haptics: a study of a force field application," in 2008 IEEE control than DT, more people preferred DT on all other criteria. 2009-Third Joint EuroHaptics conference and Symposium on Haptic difference in total masses by scaling objects’ rotational C/D ratios. In Symposium on 3D User Interfaces (3D UI). pp. 59-66, IEEE, 2008. addition, we also found empirical evidence that by altering the C/D Interfaces for Virtual Environment and Teleoperator Systems. pp. 121- [30] M. Rietzler, F. Geiselhart, J. Frommel, and E. Rukzio, "Conveying the 5.3 Discussion 126, IEEE, 2009. ratio dynamically based on the physical principles that explain the perception of kinesthetic feedback in virtual reality using state-of-the-art The results of experiment II show that DT is a powerful technique in [11] S. Hashiguchi, S. Mori, M. Tanaka, F. Shibata, and A. Kimura, generating the perception of mass distribution. When using DT, motion of rotation, we can effectively display certain aspects of mass hardware," in Proceedings of the 2018 CHI Conference on Human distribution. These two pseudo-haptic techniques can be applied "Perceived weight of a rod under augmented and diminished reality Factors in Computing Systems. p. 460, ACM, 2018. participants could usually perceive which part of the object was visual effects," in Proceedings of the 24th ACM Symposium on Virtual simultaneously—scaling the base C/D ratio to alter the perception of [31] M. Rietzler, F. Geiselhart, J. Gugenheimer, and E. Rukzio, "Breaking the heavier even when the CoM was only slightly off the geometric center. Reality Software and Technology. p. 12, ACM, 2018. Evaluation of subjective user experience also shows that the effect was total mass while adjusting this base C/D ratio dynamically relative to tracking: Enabling weight perception using perceivable tracking offsets," itself to display mass distribution. [12] H. G. Hoffman, "Physically touching virtual objects using tactile in Proceedings of the 2018 CHI Conference on Human Factors in perceived to be strong (Q1), realistic (Q2, Q3), and sufficient (Q4), augmentation enhances the realism of virtual environments," in The techniques can also be seamlessly combined with translation- Computing Systems. p. 128, ACM, 2018. and that the technique was appreciated (Q5). The fact that a non-ideal Proceedings of IEEE 1998 Virtual Reality Annual International based pseudo-haptic techniques to further enhance the display of [32] S. Runeson, “On visual perception of dynamic events,” Doctoral proxy was used (Section 5.2.3) further demonstrates the power and Symposium (Cat. No. 98CB36180). pp. 59-63, IEEE, 1998. applicability of this technique. Overall, this empirical evidence shows mass-related properties. In any VR application requiring realistic dissertation, Doctoral Dissertation at the Faculty of Social Sciences, object manipulation, these techniques can be directly applied, which University of Uppsala, University of Uppsala, 1976. 2086 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 5, MAY 2020
[33] S. Runeson, and G. Frykholm, “Visual perception of lifted weight,” [40] R. T. Skarbez, “Plausibility illusion in virtual environments,” Doctoral Journal of Experimental Psychology: Human Perception and dissertation, The University of North Carolina at Chapel Hill, 2016. Performance, vol. 7, no. 4, pp. 733, 1981. [41] M. Streit, K. Shockley, and M. A. Riley, “Rotational inertia and [34] E.-L. Sallnäs, K. Rassmus-Gröhn, and C. Sjöström, “Supporting presence multimodal heaviness perception,” Psychonomic bulletin & review, vol. in collaborative environments by haptic force feedback,” ACM 14, no. 5, pp. 1001-1006, 2007. Transactions on Computer-Human Interaction (TOCHI), vol. 7, no. 4, [42] M. Streit, K. Shockley, M. A. Riley, and A. W. Morris, “Rotational pp. 461-476, 2000. kinematics influence multimodal perception of heaviness,” Psychonomic [35] M. Samad, E. Gatti, A. Hermes, H. Benko, and C. Parise, "Pseudo-Haptic Bulletin & Review, vol. 14, no. 2, pp. 363-367, 2007. Weight: Changing the Perceived Weight of Virtual Objects By [43] Y. Taima, Y. Ban, T. Narumi, T. Tanikawa, and M. Hirose, "Controlling Manipulating Control-Display Ratio," in Proceedings of the 2019 CHI fatigue while lifting objects using pseudo-haptics in a mixed reality Conference on Human Factors in Computing Systems. p. 320, ACM, space," in 2014 IEEE Haptics Symposium (HAPTICS). pp. 175-180, 2019. IEEE, 2014. [36] R. Skarbez, F. P. Brooks, and M. C. Whitton, "Immersion and coherence [44] A. Zenner, “Investigating weight distribution in virtual reality proxy in a visual cliff environment," in Proceeding of IEEE Virtual Reality interaction,” Master thesis, Universität des Saarlandes, Masterarbeit, 2017 (VR 2017). pp. 397-398, IEEE, 2017. 2016. [37] R. Skarbez, F. P. Brooks Jr, and M. C. Whitton, "Immersion and [45] A. Zenner, and A. Krüger, “Shifty: A weight-shifting dynamic passive coherence in a stressful virtual environment," in Proceedings of the 24th haptic proxy to enhance object perception in virtual reality,” IEEE ACM Symposium on Virtual Reality Software and Technology. p. 24, transactions on visualization and computer graphics, vol. 23, no. 4, pp. ACM, 2018. 1285-1294, 2017. [38] R. Skarbez, F. P. Brooks Jr, and M. C. Whitton, “A Survey of Presence [46] A. Zenner, and A. Krüqer, "Estimating Detection Thresholds for and Related Concepts,” ACM Computing Surveys (CSUR), vol. 50, no. 6, Desktop-Scale Hand Redirection in Virtual Reality," in 2019 IEEE pp. 96, 2017. Conference on Virtual Reality and 3D User Interfaces (VR2019). pp. 47- [39] R. Skarbez, S. Neyret, F. P. Brooks, M. Slater, and M. C. Whitton, “A 55, IEEE, 2019. psychophysical experiment regarding components of the plausibility [47] M. Ziat, T. Rolison, A. Shirtz, D. Wilbern, and C. A. Balcer, "Enhancing illusion,” IEEE transactions on visualization and computer graphics, vol. virtual immersion through tactile feedback," in Proceedings of the 23, no. 4, pp. 1369-1378, 2017. adjunct publication of the 27th annual ACM symposium on user interface software and technology. pp. 65-66, ACM, 2014.