Mechanisms of Integrating Vibrotactile and Force Cues for 3D User Interaction Within Virtual Environments

University of Calgary PRISM: University of Calgary's Digital Repository

Graduate Studies The Vault: Electronic Theses and Dissertations

2018-08-29 Mechanisms of Integrating Vibrotactile and Force Cues for 3D User Interaction within Virtual Environments

Tarng, Stanley

Tarng, S. (2018). Mechanisms of Integrating Vibrotactile and Force Cues for 3D User Interaction within Virtual Environments (Unpublished master's thesis). University of Calgary. Calgary. AB. doi:10.11575/PRISM/32876 http://hdl.handle.net/1880/107698 master thesis

University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission. Downloaded from PRISM: https://prism.ucalgary.ca UNIVERSITY OF CALGARY

Mechanisms of Integrating Vibrotactile and Force Cues for 3D User Interaction within Virtual

Environments

Stanley Tarng

A THESIS

SUBMITTED TO THE FACULTY OF GRADUATE STUDIES

IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE

DEGREE OF MASTER OF SCIENCE

GRADUATE PROGRAM IN ELECTRICAL AND COMPUTER ENGINEERING

CALGARY, ALBERTA

AUGUST, 2018

A model for the mechanism of integration for vibrotactile and force cues in the haptic modality is important to facilitate task performance of human users in a three-dimensional (3D) virtual environment (VE). To investigate this mechanism, the research in this thesis used maximum likelihood estimation (MLE) and proposed proportional likelihood estimation (PLE) as the models of the integration. Two significant findings are as follows: (1) I found that based on the task accuracy, MLE was unable to integrate vibrotactile and force cues despite its ability to integrate cues of different modalities in literature; (2) Integration with PLE revealed that the mechanism of integration of vibrotactile and force cues may not be entirely additive as assumed in MLE. This work sheds an insight for proper model of integration between vibrotactile and force cues for interactive tasks in VEs.

ii Table of Contents

ABSTRACT ...... II

TABLE OF CONTENTS ...... III

LIST OF TABLES ...... V

LIST OF FIGURES AND ILLUSTRATIONS...... VI

1 INTRODUCTION ...... 1 1.1 Research Motivation ...... 5 1.2 Objectives and Contributions ...... 6 1.3 Thesis Organization ...... 9

2 BACKGROUND ...... 10 2.1 Virtual Environments ...... 10 2.2 Haptic Modality ...... 12 2.2.1 Haptics Devices in VEs ...... 13 2.3 Multi-sensory Integration...... 15

3 APPLICABILITY OF MAXIMUM LIKELIHOOD ESTIMATION FOR VIBROTACTILE AND FORCE CUE INTEGRATION ...... 18 3.1 Maximum Likelihood Estimation in Cue Integration ...... 19 3.2 Experiment 1: Reliance Unbiased ...... 22 3.2.1 Methods...... 22 3.2.2 Results ...... 29 3.3 Experiment 2: Reliance Biased ...... 32 3.3.1 Methods...... 32 3.3.2 Results ...... 33 3.4 Experiment 3: Intensity Reduced ...... 36 3.4.1 Methods...... 36 3.4.2 Results ...... 37 3.5 General Discussion ...... 40 3.6 Conclusion ...... 44

4 PROPORTIONAL LIKELIHOOD ESTIMATION FOR VIBROTACTILE AND FORCE CUE INTEGRATION...... 45 4.1 Proportional Likelihood Estimation ...... 46 4.2 Analyses with PLE and Discussion ...... 49 4.3 Potential Applications ...... 52

5 SUMMARY AND FUTURE WORK ...... 55 5.1 Summary ...... 55 5.2 Future Work ...... 57

iii BIBLIOGRAPHY ...... 59

A COPYRIGHT TRANSFER ...... 67

B UNIVERSITY OF CALGARY ETHICS APPROVAL ...... 75

C DATA COLLECTION ...... 78 C.1 Objective Data ...... 78 C.2 Subjective Data ...... 79 C.2.1 Cybersickness Questionnaire ...... 80 C.2.2 Perceptual Questionnaire ...... 81 C.2.3 NASA Task Load Index (NASA-TLX) ...... 82

D LIST OF SOFTWARE TOOLS...... 83

iv List of Tables

2.1: Different nerve receptors for the sense of touch...... 12

3.1: Formulation of MLE weights for vibrotactile and force cues...... 21

3.2: The means and standard deviations of the subjective data and their ANOVA results among all testing blocks in Experiment 1 ...... 31

3.3: The means and standard deviations of the subjective data and their ANOVA results among all testing blocks in Experiment 2 ...... 35

3.4: The means and standard deviations of the subjective data and their ANOVA results among all testing blocks in Experiment 3...... 39

4.1: Summary of formulation of MLE and PLE for vibrotactile and force cues ...... 47

v List of Figures and Illustrations

1.1: Examples of devices providing visual feedback in VEs; (a) Oculus Rift; (b) CAVE ...... 1

1.2: Examples of haptic devices; (a) PHANToM Omni with force feedback on the stylus; (b) VibroTac bracelet with six vibrators in each segment; and (c) HTC Vive controllers with circular touchpad control and vibration feedback...... 3

2.1: Haptic devices for force and vibrotactile feedback respectively: (a) PHANToM Omni with a force feedback stylus to be held by a user; (b) VibroTac bracelet, to be wrapped around the users’ limbs for vibrotactile feedback; ...... 14

3.1: MLE for an integration of vibrotactile and force cues. Accuracy is the normalized estimates from a series of detection tasks by human users...... 19

3.2: The VE and devices utilized for 3D user interaction: (a) the visual scene in the VE; (b) 3D shutter glasses and an IR emitter; (c) a VibroTac bracelet; (d) ear plugs and an E4 wristband; and (e) an Omni device...... 23

3.3: (a) The visual scene in the VE and (b) the transmission line curved due to gravity between two towers with forces delivered to the participant...... 24

3.4: Locations of both vibrotactile and force cues: (a) a co-located setting; and (b) a dis- located setting...... 26

3.5: Examples of cue profiles for all testing blocks: (a) the V_co and V_dis blocks; (b) the F_only block; and (c) the FV_co and FV_dis blocks...... 27

3.6: Average task completion time of all testing blocks for experiment 1. [Error bars represent standard errors.] ...... 30

3.7: Average task accuracy of all testing blocks for experiment 1. [Error bars represent standard errors. The symbol of ** and *** denotes Bonferroni significant differences with p < 0.01 and p < 0.001 respectively.] ...... 30

3.8: MLE analyses of task accuracy data for Experiment 1 in (a) co-located and (b) dis- located settings...... 32

3.9: Average task completion time of all testing blocks for experiment 3. [Error bars represent standard errors.] ...... 34

3.10: Average task accuracy of all testing blocks for Experiment 2. [Error bars represent standard errors. The symbol of *** denotes Bonferroni significant differences with p < 0.001.] ...... 34

3.11: MLE analyses of task accuracy data for Experiment 2 in (a) co-located and (b) dis- located settings...... 36

vi 3.12: Average task completion time of all testing blocks for Experiment 3. [Error bars represent standard errors.] ...... 38

3.13: Average task accuracy of all testing blocks for Experiment 3. [Error bars represent standard errors. The symbol of *** denotes Bonferroni significant differences with p < 0.001.] ...... 38

3.14: MLE analyses of task accuracy data for Experiment 3 in (a) co-located and (b) dis- located settings...... 40

3.15: Gaussian-distributed estimations of MLE on the experimental data in: (a) reliance- unbiased co-located; (b) reliance-unbiased dis-located; (c) reliance-biased co-located; (d) reliance-biased dis-located; (e) intensity-reduced co-located; (f) intensity-reduced dis-located settings...... 43

4.1: MLE for a cue integration with mean and amplitude (therefore variance) mismatch from the cue collaboration...... 46

4.2: Gaussian-distributed estimations of MLE and PLE and their weights on the experimental data in: (a) reliance-unbiased co-located; (b) reliance-unbiased dis-located; (c) reliance- biased co-located; (d) reliance-biased dis-located; (e) intensity-reduced co-located; (f) intensity-reduced dis-located settings...... 50

vii

Chapter 1

Introduction

Computer-based virtual environments (VE) are systems which provides simulated virtual objects for user interaction in real time. The primary appeal of a VE is its ability to provide an experience to the user that is both immersive and realistic. This is accomplished by constructing the virtual objects to appear and behave similarly to real-world objects. The appearance of the virtual objects, while important, does not offer the complete experience of a VE. The ability to interact with these virtual objects vastly improves the experience and the usefulness of a VE [1].

VEs have been used in applications in oil and gas exploration [2,3], manufacturing [4], construction [5], medical surgery [6,7], arts [8], education [9], military [2], and entertainment

[2,10]. Many of these applications require a human user to interact with the VE in some way.

(a) (b)

Figure 1.1: Examples of devices providing visual feedback in VEs; (a) Oculus Rift; (b) CAVE

These user interactions frequently provide feedback to the user via the visual modality exemplified by both personal devices such as head-mounted displays (HMDs) and wall-scale systems such as Computer Automated Virtual Environments (CAVEs) as seen in Figure 1.1.

HMDs such as Oculus Rift as illustrated in Figure 1.1 (a) are personal devices that uses optics, visual displays, and head tracking to create a VE for the user, while CAVEs use large displays and 3D glasses to provide a VE simultaneously to multiple users.

VEs offers a unique combination of visuals and interactivity that can be developed and deployed relatively quickly and cheaply compared to using real-world counterparts. As affordable virtual reality devices become more readily available, the application of VEs have become an attractive solution to enhance industrial or research processes by human users and experts alike. Increasingly, users demand more information to be delivered by the VE with increased precision and accuracy. Since the visual modality is the primary source of information for humans, providing and receiving feedback in this modality is natural for both the creators and users of VEs. Adding additional information visually is often the most straightforward method.

However, humans can only visually focus and process a limited amount of information at a given time [11]. Therefore, too much information delivered via the visual modality can overwhelm a user. If too much information is presented visually, the sheer amount of information can become unmanageable to the user. This can result in incomplete or incorrect interpretation of the information by the user. To alleviate this information overload in the visual modality, information from other sensory modalities is necessary. For this, creators of VEs incorporates feedback of other modalities (e.g. haptics – the sense of touch, and auditory) to simplify the

delivery of information as well as add information to provide a sense of presence and interactivity [1,6,11–16]. Many HMD equipment manufacturers have already incorporated haptic and auditory devices in their development kit, which greatly increases their use in VEs overall.

Haptic and auditory information are delivered through various cues, which stimulate different sensory receptors of the user. A cue is a signal that can be extracted from a corresponding sensory input to the human sensory receptors. For example, cues of the haptic modality include the sub-categories of kinesthetic force and pressure, cutaneous textures of surfaces (i.e., vibration, shape, smoothness), moisture, and temperature. In the real world, multiple sub-categorical cues are typically encountered in a combination together at the same time. When manipulating an object in the real world, the user expects haptic feedback in contact with the object. This contact integrates multiple sub-categorical cues of force, position, temperature, surface texture etc. together to provide information about the object to the user. VE creators often desire to recreate this type of interaction with virtual objects in the VE once visual representation of the object are delivered. Hence in a VE, we must also deliver multiple sub- categorical cues to the user in a similar way. Indeed, it has been shown that multiple cues, in

(a) (b) (c)

Figure 1.2: Examples of haptic devices; (a) PHANToM Omni with force feedback on the stylus; (b) VibroTac bracelet with six vibrators in each segment; and (c) HTC Vive controllers with circular touchpad control and vibration feedback. 3

some combinations thereof, are generally effective at giving a sense of presence and enhancing user performance [11]. Two commonly used types of feedbacks in the haptic modality are through vibrotactile and force cues. This arises from both availability and relative ease of acquiring devices that can deliver these haptic cues, such as vibration motors and joysticks commonly used in video game applications illustrated in Figure 1.2.

Vibrotactile and force cues can be delivered to the user in either co-located or dis-located settings. When interacting with physical objects for manipulation in the real world, the user expects feedback directly in contact with the object. That is, the cues are co-located to stimulate different sensory receptors for the user’s cognitive response. While real world interactions have the cues co-located with the object when interactions take place, it may not be possible to deliver cues in co-located settings in a VE [17,18]. This may be due to restrictions in the design and/or implementation of haptic devices [17]. Instead, dis-located cues can be delivered to the user.

Therefore, both settings are important for understanding how cues are integrated together for creating intuitive VEs. Studies have commonly reported an observation that the co-located cues offer better performance in the tasks of perception, discrimination, and tool steering [17,19–23].

Nevertheless, dis-located cues remain a mean to improving user performance in the VE [1]. To best facilitate user performance in either setting, a general mechanism of integrating vibrotactile and force cues would be necessary. However, few efforts have been devoted to investigate this mechanism.

1.1 Research Motivation

As the application of vibrotactile and force cues in VEs become more widespread due to the availability of haptic feedback devices, there is a need for a generic mechanism for integration the two cues. However, the specific integration of vibrotactile and force cues in the haptic modality is not well understood. Current implementations of these cues in VEs generally rely on either real-world scenario (haptic recording) [7,24], or analytical models verified with user testing [24,25]. The simulated cues implemented based on real-world scenarios provides convincing and realistic results to the users because they are delivering pre-recorded feedback to the user. This type of simulated feedback closely matches those from the real-world interaction.

However, this implementation is not easily extrapolated to scenarios outside of their domain. The cues implemented based on analytical models and user testing can also be convincing to the user.

However, these are usually simplified and may not have a high degree of accuracy and precision.

A model of integration at the fundamental level can elucidate accurate and precise vibrotactile and force cues in VEs while being applicable to multiple scenarios. Therefore, it is desirable to a conduct a deeper exploration into the nature of human integration of vibrotactile and force cues.

Existing reports on various generic mechanisms of integration applied to multi-sensory cues integration offers possible candidates for vibrotactile and force cue integration. The mechanisms of cues integration examined in existing reports includes maximum likelihood estimation (MLE)

[13,26–28], information integration theory [29] and signal detection theory [30–32]. MLE was followed in many of these reports, such as when explaining the integrations of visual and haptic

cues [13], force and position cues [27], as well as auditory and visual cues [33]. Many of these integrations have used co-located cues and followed MLE. Exceptions are co-located force cues and cutaneous cues (cues delivered to the nerves in the skin) under task-specific models

[20,22,23]. Due to the task-specific nature of these models, they are not generally applicable to other interactive tasks in VEs. Also, these reports examined integration of sensory cues of different modalities instead of sub-categorical cues of the same modality. A mechanism of integrating vibrotactile and force cues, two sub-categorical cues of the haptic modality, may not necessarily follow the same mechanism as integration of two cues of different modalities. It is hypothesized in this thesis that MLE may not be followed when integrating vibrotactile and force cues, but instead follow other integration mechanism.

1.2 Objectives and Contributions

The main objective of this thesis is to investigate the mechanisms of human integration of vibrotactile and force cues. To reach this objective, the research conducted for this thesis has the following two aims:

Aim 1. To investigate the applicability of MLE for integrating vibrotactile and force cues.

As MLE is commonly used in integration of other multi-sensory cues, this is

necessary to establish a baseline for comparison with the model of integration

proposed in Aim 2. This also serves to verify the hypothesis of the thesis.

Aim 2. To provide a model of integration for integrating vibrotactile and force cues. This

provides insight to the underlying mechanisms of the integration. In this thesis, I 6

created and proposed Proportional Likelihood Estimation (PLE) for this purpose.

PLE is a model of integration based on MLE but with proportions of contribution of

individual cues taken into consideration.

For Aim 1, MLE’s suitability for integrating vibrotactile and force cues were examined because it was applied most frequently in existing reports. Under both co-located and dis-located conditions, as well as under different user bias and cue intensity, MLE was found to be inadequate as the integration model. The initial investigation was published in a flagship IEEE conference as follows:

 A. Erfanian, S. Tarng, Y. Hu, J. Plouzeau, and F. Merienne, (8 pages in IEEE double

column format; full-paper submission for two-staged and double-blinded review) “Force

and vibrotactile integration for 3D user interaction within virtual environments,” Proc.

IEEE Symposium on 3D User Interfaces (IEEE 3DUI), Los Angeles, CA, Mar. 2017, pp.

87-94. [Acceptance rate ~30%]

 A. Erfanian, S. Tarng, Y. Hu, J. Plouzeau and F. Merienne, (2-page in IEEE double-

column format; full-paper submission for double-blinded review), "Mechanism of

integrating force and vibrotactile cues for 3D user interaction within virtual

environments", Proceedings of the IEEE Virtual Reality Conference (IEEE VR), Los

Angles, CA, USA, Mar. 2017, pp. 257-258. [Acceptance rate ~ 30%.]

Further findings have been published in a reputable IEEE conference as follows:

 S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, (6-page in IEEE double-column format;

full-paper submission for double-blinded review) “An Exploration on the Integration of

Vibrotactile and Force Cues for 3D Interactive Tasks,” Proceedings of the IEEE

Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Reutlingen, Germany,

Mar. 2018. [Acceptance rate ~ 30%.]

 S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, (2 pages in IEEE double column format;

full-paper submission for two-staged and double-blinded review) “Vibrotactile and Force

Collaboration within 3D Virtual Environments,” Proc. IEEE International Conference on

Computer Supported Cooperative Work in Design (IEEE CSCWD), Nanjing, China, May

2018, pp. 330-335.

For Aim 2, PLE was proposed for integrating vibrotactile and force cues as MLE was found to be inadequate in modeling the integration. This has resulted in a journal manuscript to be submitted to a leading IEEE journal as follows (copyright transfers see Appendix A).

 S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, (10 pages in IEEE double column

format) “Additive or Not: A Question for Integrating Vibrotactile and Force Cues in 3D

Interactive Tasks,” to be submitted to IEEE Transactions on Human-Machine Systems

(IEEE THMS), Aug. 2018.

The research of this thesis was done in collaboration with the Laboratory of Electronics and

Digital Imaging (LE2I) of Arts et Métiers ParisTech in France. The experiments conducted for this thesis used human participants with approval by the University of Calgary Conjoint

Faculties Research Ethics Board under the ethics clearance according to the Canadian Tri- council Ethics Guidelines (Appendix B).

1.3 Thesis Organization

This thesis is organized into five chapters, including this Introduction chapter. Chapter 2 is a review to provide background material for the haptic senses of vibrotactile and force, and the state of the art in multi-sensory integration.

Chapter 3 describes the research and experiments conducted for Aim 1. This chapter begins with an overview of the mathematical formulation of MLE and how it applies to multi-sensory integration. Three experiments were conducted to investigate the suitability of MLE applied to the integration of vibrotactile and force cues. In these experiments, the participants of the experiment were asked to complete a detection task with the aid of vibrotactile and/or force cues.

The physical locations of application of these cues were either co-located or dis-located. The three experiments were similar and varied only in the participants’ bias and the intensity of the vibrotactile cues they received. The results of these experiments confirmed the hypothesis that

MLE is unsuitable as a model of integration for vibrotactile and force cues.

Chapter 4 addresses Aim 2, in which PLE was proposed as the model of integration for vibrotactile and force cues. This chapter describes the mathematical formulation of PLE in contrast to MLE. The experimental results from Chapter 3 were then analyzed with PLE, and the results presented in this chapter. PLE was found to be able to integrate vibrotactile and force cues to match the empirical observations in the experiments. The implications of PLE’s mathematical formulation confirms the hypothesis that the integration of vibrotactile and force cues follow a different integration mechanism than MLE.

Chapter 5 summarizes this thesis work and proposes future work. 9

Chapter 2

Background

As described in Chapter 1, VEs are increasing employing haptic devices to aid in the user interaction as well as deliver more information to the user. Understanding of haptic senses and cues allows for effective integration of haptic devices into VEs. This understanding forms the foundation of effective use of multiple sensory cues in concert with each other in a VE. While the visual modality remains the most important in a VE [34,35], the haptic modality provides additional information to facilitate user performance [36,37]. In literature, the integration of multi-sensory cues, such as visual and haptics, have been investigated [13], but the integration of multiple sub-categorical cues of the same modality is not well understood. In this chapter, VEs are first described. Afterwards, background information on haptic senses – primarily vibrotactile and force – are introduced. Then the current state of the art in multi-sensory integration is discussed.

2.1 Virtual Environments

A VE is a computer-based system that uses high-end human-computer devices for human- computer interaction. A VE can be as simple as a simulation running on a common desktop computer (with low immersion) to wall/room sized 3D projections, or head mounted displays with exoskeleton devices (high immersion). Typically, a VE simulate an artificial environment

that simulates the real world for user interaction [38]. However, due to the artificial nature of the

VE, they allow for the introduction of environments and interactions completely unrealizable in the real world [38]. To make this simulated environment immersive to the user, sensory cues and interactions are provided. The user can interact and manipulate the VE with devices such as computer mice, spatial tracking systems, game controllers, etc. [39]. Coupled with the user interactions are the feedback from the VE. By using various devices, VEs can provide feedback to the user with sensory cues such as visual, auditory, olfactory, and haptic cues [12]. These cues, in some combination, are generally effective at creating an immersive environment and enhance user performance in the VE [6,15,16].

Immersion is the sensation of being in an environment, whether physically and/or mentally

[39]. VEs offers physical immersion through stimulating body’s senses through technology and devices. A VE is able to replace or augment the stimulus appropriate to the environment being simulated, therefore offering physical immersion. The VE system’s level of immersion depends on the display technologies (including sensory displays) and the rendering software [2]. In visual immersion alone, there are many factors such as field of view, display size, display resolution, stereoscopy, head tracking, lighting realism, frame rate, etc. influencing the level of immersion of the system [2]. Depending on the application, some of the factors would be more important than others. For example, lighting realism for abstract information like information modeling would not be measured in the same way as a VE simulating the real world.

Mental immersion, commonly referred to as “presence”, is the state of mind where the user is deeply engaged in the VE so that there is a willing suspension of disbelief [2,39]. Sense of

presence is a psychological response and is much more difficult to evaluate compared to physical immersion. Presence is an individual and is also context-dependent psychological response [2].

This has to do with the user experience of believing to “be there”. The same user may even experience different levels of presence with the same system at different times based on recent history and differing state of mind [2]. While presence is subjective to each individual user and difficult to quantify, it is generally possible to induce a high degree of presence in the user with good physical immersion. VE creators often desire their VEs to have a high degree of presence to facilitate user performance. Therefore, understanding how human responds to physically immersive cues, such as haptics, is important the creation of VEs.

2.2 Haptic Modality

Haptics is the modality of touch that allows the users to “feel” an object and sense their environment [40]. It is a perceptual system composed of two subsystems, cutaneous and kinesthetic [18], Compared to the visual or auditory modalities, haptics is especially effective at discerning the properties of surfaces and material objects [18]. Haptics allows for users to sense features of an object such as weight, shape, texture, and temperature [40]. It is the sense that

Perception Nerve Receptor Cold Krause end-bulbs Skin stretch Ruffini’s end organ Texture and slow vibrations Meissner’s corpuscle Deep pressure and fast vibrations Pacinian corpuscle Table 2.1: Different nerve receptors for the sense of touch. 12

most directly corresponds to object manipulation in a VE, and therefore is important to understand when using haptics in a VE.

Typically, humans use their hands to perceive the sense of touch. Table 2.1 shows the different nerve receptors for this purpose [41]. Humans receive tactile stimuli from the nerve receptors underneath their skin when perceiving an object through the sense of touch. This allows for the perception of temperature, pressure, and texture of objects [41]. Additionally, when humans manipulate objects with their hands, not only do they receive information about the object from the sense of touch (i.e. from cutaneous receptors), they also receive information through the muscle and motor systems (i.e. kinesthetics). Kinesthetics perception allows humans to perceive force being applied on body parts as well as their positions in space [42]. This form of perception receives stimuli from receptors on the muscle tendons and joints, which allows for sensing object weights and stiffness. Overall, the combination of cutaneous and kinesthetic perception provides the complete haptic information.

2.2.1 Haptics Devices in VEs

Chapter 1 briefly mentioned a few of the haptic devices currently available in Figure 2.1. There are many other haptic devices available, ranging from simple piezoelectric actuators used in electronics to wearable exoskeletons providing force feedback to entire human limbs [40]. These devices recreate the sense of touch by stimulating the appropriate receptors. Haptic devices are valuable to researchers for the ability to create carefully controlled haptic virtual objects, allowing for research into the human sense of touch. When used in a VE, haptic devices generally share the purpose of assisting the user in the manipulation of virtual objects. Haptic 13

(a) (b)

Figure 2.1: Haptic devices for force and vibrotactile feedback respectively: (a) PHANToM Omni with a force feedback stylus to be held by a user; (b) VibroTac bracelet, to be wrapped around the users’ limbs for vibrotactile feedback; devices may also be used for the remote control of machines, such as those used in medical surgery [40].

As described in Chapter 1, vibrotactile and force cues in the haptic modality, are frequently used to aid the user in manipulating objects in VEs. This arises from the availability of devices capable of delivering these cues. An example of a device capable of providing a sense of touch with force feedback is the PHANToM Omni device (Geomagic Inc., USA) [43], shown in Figure

2.1 (a). The Omni device can deliver force feedback of up to 0.88 N continuously to its stylus.

The user can manipulate the style with 6 degrees of freedom in a workspace the size of

160x120x70 mm. The Omni device has a resolution of 450 dpi, allowing precise manipulation of virtual objects through the manipulation of its stylus.

A different example of a device capable of providing vibrotactile feedback is the VibroTac bracelet (SENSODRIVE Gmbh, Germany) [36], shown in Figure 2.1 (b). Powered by an 800 mAh Lithium-ion battery, the VibroTac bracelet consists of 6 independent vibration segments.

The vibrations are generated by rotating an unbalanced mass at the end of the motor shaft of a cylindrical DC motor. The vibrations generated by the bracelet are approximately 180 Hz, which is within the maximum stimulation of skin mechanoreceptors of 250 Hz [44]. The ViborTac allows the user to receive vibrotactile feedback and was developed for use with collision 14

feedback or to provide guidance information [36]. However, it can be adapted to other applications as well.

For adequate immersion of a user in a VE while interacting with a haptic device, the haptic device must update its operating state. For example, it must detect object collision and update its display in reaction to the collision. This must be done quickly and updated frequently at a rate of

1 kHz or higher, otherwise the quality of the immersion for the user will be inconsistent [45].

Haptic devices are not without their limitations. For example, the Omni device has a limited maximum force output due to mechanical limitations. It also has a limited workspace in which it can operate. There may also be strict requirements in their integration with other displays to ensure meaningful and reliable feedback. For example, a surgical VE offers both haptic and visual feedback to the user. It is undesirable to have noticeable mismatches between the haptic feedback and the visual feedback [45]. However, limited haptic feedback can still provide an enhancement to user performance in VEs [45]. Therefore, both the Omni and VibroTac can be useful haptic devices despite their limitations, and is useful for the experiments conducted for this thesis.

2.3 Multi-sensory Integration

Multi-sensory integration is the study of how information from different sensory modalities are interpreted and integrated by the nervous system [46]. Examples of sensory modalities are visual, auditory, olfactory, and haptic. In everyday life, humans constantly receive information in the form of sensory cues in multiple modalities. Every object being interacted with have cues 15

corresponding to each of the sensory modalities. It is with the integration of all the cues from these modalities that humans receive a complete sense of an object. Literature suggests that the visual modality often influences information from other senses [34]. This is the case because visual information tends to be more reliable than other sources of information. However, this can also be a source of illusion. This is known as the ventriloquist effect [47]. In this particular case of multi-sensory integration, the visual cues dominate over the auditory cues for spatial localization. This has the effect of making the auditory cue appear to come from a different spatial location than in actuality. This effect has been exploited by performers since antiquity

[47] and continues to be a fascinate multi-sensory integration phenomenon. It illustrates that the result of multi-sensory integration is not obvious and can be surprising.

In the real world, cues are usually expected to be co-located. When we grasp an object in the real world we receive haptic cues co-located with the visual cues; likewise, when we drop an object we may receive auditory cues as it hit the floor co-located with the visual cues, and perhaps kinesthetic cues to our hands indicating the drop. In a VE, ideally all sensory cues should be co-located to simulate the real world as closely as possible [1,17,18]. However, precise co-location of haptic cues can be technically difficult to achieve in a VE compared to co-location of visual and auditory cues. This generally arise from the physical or hardware limitation of haptic devices [17]. Nevertheless, dis-located cues can still be provided to the user in a VE.

Therefore, it is important to understand both co-located and dis-located cues settings. As described in Chapter 1, while co-located cues offer better user performance in a VE compared to dis-located cues, dis-located cues still offer an avenue to improve user performance in a VE [1].

Many studies have focused on multi-sensory integration involving the visual modality because of the visual dominance, there are still research into the integration of non-visual modalities. However, little effort exists in investigating two sub-categorial cues of the same modality, such as vibrotactile and force cues. As described in Chapter 1, existing reports on multi-sensory integration focused on the integration of visual and haptic cues [13], force and position cues [27], as well as auditory and visual cues [33]. The majority of the investigations into multi-sensory integration have used co-located cues and followed MLE [13]. Therefore,

MLE offers a foundation for investigation into the integration of vibrotactile and force cues in this work.

Chapter 3

Applicability of Maximum Likelihood Estimation for

Vibrotactile and Force Cue Integration1

As discussed as Aim 1 in Chapter 1, MLE was investigated for its applicability to integrate vibrotactile and force cues. Since MLE is frequently used to integration cues of different modalities, it follows that this model of integration is a suitable baseline for comparison with other integration models [13,26–28]. Using haptic devices, it is feasible to have both co-located and dis-located settings of vibrotactile and force cues in the haptic modality. Therefore, both settings are important for understanding how cues are integrated together for creating intuitive

VEs. In this thesis, it is hypothesized that the rules of MLE may not be followed when integration vibrotactile and force cues, two sub-categorical cues of the same modality. A series of three experiments were conducted to verify this hypothesis.

This chapter begins with an overview of MLE in the context of multi-sensory integration.

Then the three experiments conducted are presented, followed by a discussion of the findings.

1 Parts of this chapter are published/to be submitted to: S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, “Vibrotactile and Force Collaboration within 3D Virtual Environments,” Proc. IEEE International Conference on Computer Supported Cooperative Work in Design, Nanjing, China, May 2018, pp. 330-335. S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, “Additive or Not: A Question for Integrating Vibrotactile and Force Cues in 3D Interactive Tasks,” to be submitted to IEEE Transactions on Human-Machine Systems, 10 pages (IEEE double-column format), Aug. 2018.

Figure 3.1: MLE for an integration of vibrotactile and force cues. Accuracy is the normalized estimates from a series of detection tasks by human users.

The three experiments used human participants with approval by the University of Calgary

Conjoint Faculties Research Ethics Board according to the Canadian Tri-council Ethics

Guidelines (Appendix B).

3.1 Maximum Likelihood Estimation in Cue Integration

Human sensory system responds to sensory cues in the environment. These cues are then used by the human perceiver to estimate corresponding physical properties of the source of the cues.

Each estimate has uncertainties caused by sources both intrinsic and extrinsic to the perceiver

[30]. Intrinsic sources may include sensory noise or poorly learned discrimination; and extrinsic sources may include environmental noise or signal attenuation [30]. Over many estimates, the results can be modeled as a Gaussian distribution. Human behavior and decision making process have also been shown to be Gaussian distributed in nature [13,48].

Each individual cue in MLE are assumed to be Gaussian distributed. Figure 3.1 shows each cue has the characteristic parameters of mean (μ), corresponding to the mean of the estimates;

and variance (σ2), corresponding to the certainty of the estimates. These parameters are independent from each other [13]. A third parameter, amplitude (A) can be used in place of variance, as the two are inversely related to each other. This Gaussian distribution can be described as:

(푎−휇)2 1 − 퐺̂ = 푒 2휎2 (3.1) √2휋휎2

The rules of MLE, when applied to integrate N individual cues, the resulting cue integration to be estimated (퐺̂) is also Gaussian distributed. This integration method yields a predicted

Gaussian distribution of the cue integration based on empirical observations of the individual cues [13,28]. The prediction (퐺̂) of an integration among N individual cues is estimated by summing the weighted empirical observations (퐺̂푖, 푖 ∈ {1, . . , 푁}) of the cues as follows [13]:

푁 퐺̂ = ∑ 푊푖퐺̂푖, (3.2) 푖=1 where the weight,

2 1/휎푖 푊푖 = 푁 2, (3.3) ∑푗=1 1/휎푗 of the i-th cue’s observation is related to its standard deviation (휎푖), given that each observation is a Gaussian distribution. Thus, the prediction (퐺̂) yielded by Eq. (3.2) is also a Gaussian distribution. It is also noted that Eq. (3.3) implies the sum of the individual weights must be 1.

When MLE is applied to two individual cues, such as vibrotactile and force cues, the formulation can be simplified, and the characteristic parameters estimated as the following

Integration Method MLE

2 1/휎푉 Vibrotactile 푊 = 푊 = 푊 2 = (3.6) 푉 휇푉 휎푉 2 2 1/휎푉 + 1/휎퐹

1/휎2 Weights 퐹 Force 푊 = 푊 = 푊 2 = (3.7) 퐹 휇퐹 휎퐹 2 2 1/휎푉 + 1/휎퐹 Table 3.1: Formulation of MLE weights for vibrotactile and force cues.

휇 = 퐸(푊휇푉퐺푉 + 푊휇퐹퐺퐹) (3.4) for estimating the mean, where E() is the expected value function; and

2 휎 = 푉푎푟(푊 2 퐺 + 푊 2 퐺 ) 휎푉 푉 휎퐹 퐹 (3.5)

for estimating the variance, where Var() is the variance function. 퐺푉 and 퐺퐹 are the Gaussian distribute vibrotactile and force cues respectively and can be treated as random variables (RVs).

The weights (푊 , 푊 , 푊 2 and 푊 2) of each cues’ observation are described in Table 3.1. In 휇푉 휇퐹 휎푉 휎퐹

MLE, there is only one set of weights applied to both mean and variance integration, as described by Eq. (3.6) and (3.7). The formulation of these weights in MLE not only entirely depends on the variance of the individual cues, they also can only be positive numbers with an upper limit of 1. This implies that cue integration in MLE can only be positive additions of the cues. This has the consequence that the resulting mean must fall between the two individual cues’ means and the variance necessarily decrease. This can be shown with the rule when a RV is multiplied by a constant:

푉푎푟(푐푋) = 푐2푉푎푟(푋) (3.8)

Since the weights must be less than 1, the resulting variance decrease by the square of the weights.

3.2 Experiment 1: Reliance Unbiased2

To verify the applicability of MLE, we conducted an empirical study consists of three experiments. The three experiments share the same VE, procedures, and data analyses methods.

The experiments differ from each other in participants, who are all unique; as well as the participants’ bias towards either vibrotactile or force cues. In this study, we developed a 3D stereoscopic VE for human participants to undertake the interactive task.

3.2.1 Methods

VIRTUAL ENVIRONMENT: The VE in the study was created using Unity 3D game engine

(version 5.3.4f). All visual and haptic components of the VE were managed using the C# language. The VE and devices utilized for user interaction are depicted in Figure 3.2. As shown in Figure 3.2 (a), the stereoscopic visual scene of the VE was projected onto a screen. A pair of

3D glasses and allowed each participant to view the scene. A vibrotactile bracelet delivered a vibrotactile cue to the right hand or the forearm of the participant. As depicted in Figure 3.2 (d), a pair of ear plugs were used to block out the noise generated by the vibrotactile bracelet when

2 Experiment 1 and its ANOVA and MLE analyses were done in collaboration with A. Erfanian at the University of Calgary.

(c)

(a)

(d)

(b) (e)

Figure 3.2: The VE and devices utilized for 3D user interaction: (a) the visual scene in the VE; (b) 3D shutter glasses and an IR emitter; (c) a VibroTac bracelet; (d) ear plugs and an E4 wristband; and (e) an Omni device. delivering the vibrotactile cue. A PHANToM Omni device (Geomagic Inc., USA), as shown in

Figure 3.2 (e), inputted interactive commands and provided the force cue to the participant’s right hand. Two Unity 5 plugins were applied to activate the VibroTac bracelet and the Omni device. A vibrotactile plugin was made by the Art et Métiers, France; and a force plugin was provided by the Digital Design Studio at the Glasgow School of Art, United Kingdom.

Each participant sits on a chair and employed the Omni device on a small table, as illustrated in Figure 3.2 (a), with their right hand for 3D interaction, as illustrated in Figure 3.2 (a). The chair was positioned far away at a distance proportional to the screen size to maintain the same field of view between experiments. The participant views the visual scene of the VE through a stereo camera attached to the front of the drone. The visual scene consisted of a high-powered transmission line in a mountainous region, illustrated in Figure 3.3 (a). The transmission line was curved towards the ground due to its own weight while being supported by two towers. These supporting towers were separated at 60.00 m. The participant was required to fly and guide a

(a)

(b)

Figure 3.3: (a) The visual scene in the VE and (b) the transmission line curved due to gravity between two towers with forces delivered to the participant. drone using the Omni device along the transmission line. This is illustrated in Figure 3.3 (b). A robotic arm with a loop-shaped clamp at the end was attached to the bottom of the drone. The transmission line was covered by this clamp for detecting defects. The defects were delivered as vibrotactile and/or force cues to the participant, through the VibroTac bracelet and/or the Omni device respectively. These defects were not visually discernable from the transmission line.

PARTICIPANTS: Ten participants (with a mean age of 26.78 ± 5.77 years old), naïve to the purpose of the study with no prior experiences with the VE, took part in the experiment. The 24

number of participants is greater than the necessary 8 groups according to Lehr’s formula [49]. A pre-screening assessment consisted of an Edinburgh handedness test, an Ishihara color-blindness test, and a Randot stereo test (Stereo Optical Inc.) was conducted to verify the eligibility of each participant. From these assessments, every participant was right-handed with regular color vision, and had normal or corrected-to-normal vision with the stereo acuity of at least 40” of arc.

The participants in this experiment were unbiased towards either the vibrotactile or the force cue.

HARDWARE: A high-end graphics workstation with 2 Intel Xeon X5690 CPU processors, 2 nVidia QuadroPlex 7000 graphics cards plus 4 Quadro 6000 graphic adapters, 32 GB of RAM and a 64-bit Windows® 7 operating system was used to run the VE. The visual scene was projected on the center screen (3x3 m2, with the participant 2.5 m away) of a CAVE (automatic virtual environment) using two projectiondesign F30SX projectors (Barco Fredrikstad, Belgium).

A VibroTac bracelet (SENSODRIVE Gmbh, Germany) presented in Figure 3.2(c) delivered the vibrotactile cues as described in the procedure.

PROCEDURE: The 3D interactive task the participant undertook was to detect defects using the robotic arm on the flying drone over a curved transmission line. The participant pointed the stylus tip of the Omni device along the transmission line while pressing the dark-gray button on the stylus to move the drone. The clamp of the robotic arm covered the transmission line for sensing its surface, highlighted in Figure 3.3 (a) by the red circle. The Omni device delivered a continuous guiding force (퐹⃗푔) tangentially to the line to guide the participant to fly the drone along the line. There is also a frictional force opposing this continuous guiding force (퐹⃗푣). The total of the guiding force and frictional force is 0.50 N. The magnitude was slightly above the 25

(a) (b)

Figure 3.4: Locations of both vibrotactile and force cues: (a) a co-located setting; and (b) a dis-located setting. threshold of the human force perception [50] and under the device’s maximum force for continuous exertion (0.88 N). In addition to the continuous force applied by the Omni device, vibrotactile and/or force cues were delivered to each participants right hand (and/or forearm) when a defect is encountered on the transmission line (Figure 3.4). The force cue was delivered to the participant through the stylus of the Omni device on the right hand for a duration of 1.00 s excluding the pre- and after-ramping of 0.25 s. The force cue was 0.10 N (퐹⃗푐) in addition to the continuous guiding force. Therefore, when a force cue was applied, the resulting total force delivered to the participant was 퐹⃗푔 + 퐹⃗푐 + 퐹⃗푣 . These forces are depicted in Figure 3.3 (b).

The vibrotactile cue was applied either co-located with the Omni device on the hand or dis- located (as depicted in Figure 3.4) from the Omni device on the forearm. The vibration was 1.00 s long at a frequency of ~200 Hz, which was a frequency easily perceptible by the participant

[51]. When placed in the co-located setting, the vibration motor was placed on the skin covering the first dorsal interosseous muscle between the thumb and the index finger. In dis-located setting, the motor is placed at the carpi radialis longus muscle of the forearm.

Figure 3.5: Examples of cue profiles for all testing blocks: (a) the V_co and V_dis blocks; (b) the F_only block; and (c) the FV_co and FV_dis blocks.

Each participant declared his/her detection of a defect while navigating over the transmission line by pressing down both buttons on the stylus of the Omni device. There were 5 cue profiles, as presented in Figure 3.5, with each profile being one testing block:

• V_co: The defect cue signal consisted only the vibrotactile cue co-located with the Omni

stylus.

• V_dis: The defect cue signal consisted only a vibrotactile cue dis-located from the Omni

stylus at the forearm.

• F_only: The defect cue signal consisted only a force cue delivered by the Omni stylus.

• FV_co: The defect cue signal consisted of the combined vibrotactile cue of the V_co

profile in the co-located setting with the force cue of the F_only profile.

• FV_dis: The defect cue signal consisted of the combined vibrotactile cue of the V_dis

profile in the dis-located setting with the force cue of the F_only profile.

A practice block was present prior to each testing block for the participant to learn to fly the drone as well as detecting defects. A total of 15 defects was located randomly on the transmission line and their locations differed in all practice and testing blocks. The participant answered one cybersickness [52] and one perceptual questionnaire [53] after each testing block.

The total length of the procedure was about 1.75 hours, including the pre-assessment and a 2- minute break between any two blocks for each participant. The order of the testing blocks was counter-balanced among all participants.

DATA COLLECTION AND ANALYSES: First, both objective and subjective data were collected in for each testing block. Objective data such as task completion time and task accuracy of detecting defects were logged by the VE to record the participant’s performance. Subjective data was gathered through the cybersickness and perceptual questionnaires. The perceptual questionnaire covered the perceived usefulness (Usf), effectiveness (Eff), pleasure (Pls) and workload (Wkl) components of the VE. Using the first three components of a continuous variation of the Likert scale bounded between 0% and 100%, the first three components were constructed [54]. Each question related directly to the participant’s actions for consistency [55].

The workload component was constructed using the NASA task load index (TLX) [56]. All subjective data were numeric for comparison, except cybersickness data. For detailed descriptions see Appendix C.

Second, cybersickness was assessed using the responses gathered from the questionnaire

[52]. If a participant had suffered from cybersickness, his/her data was excluded from analyses.

Third, other objective and subjective data were compared using one-way Analysis of

Variance (ANOVA) [57]. ANOVA is suitable for testing for significant statistical differences between several groups while assuming the truth of null hypothesis. While other methods such as the t-test exists for testing for significant differences between two groups, ANOVA is able to test multiple groups simultaneously. This is advantageous because a single test with ANOVA reduces the likelihood of falsely rejecting the null hypothesis compared multiple pair-wise tests.

A requirement for using ANOVA is the data must normally distributed. Using Anderson-Darling normality tests [58], data eligibility for using ANOVA analyses was determined. This normality test also determined if the cues to be modeled is indeed Gaussian. Significant differences from

ANOVA analyses were then evaluated with Bonferroni post hoc tests [57]. These analyses served to establish the baseline for using both MLE.

Lastly, MLE was applied to the task accuracy data. The MLE weights were calculated with formulas in Table 3.1. These weights were then used to compute the prediction of the integration between the vibrotactile and force cues. Refer to Appendix D for a list of software tools used to create the VE and for data analyses.

3.2.2 Results

None of the participants suffered from cybersickness. This was confirmed by the physiological data and the responses to the cybersickness questionnaire. Thus, the data of all participants were used in the analyses. Normality tests on other objective and subjective data showed that all the data were normally distributed and therefore eligible for ANOVA analyses.

Figure 3.6: Average task completion time of all testing blocks for experiment 1. [Error bars represent standard errors.]

Figure 3.7: Average task accuracy of all testing blocks for experiment 1. [Error bars represent standard errors. The symbol of ** and *** denotes Bonferroni significant differences with p < 0.01 and p < 0.001 respectively.]

ANOVA analyses of the completion time (Figure 3.6) revealed no significant difference among all testing blocks [F(4, 49) = 1.39; p > 0.05]. The similar completion time resulted from each block having the same number of the defects at different locations. ANOVA analyses on the task accuracy (Figure 3.7) resulted in a significant difference among all testing blocks however

[F(4, 49) = 4.53; p < 0.05]. Three pairs of the testing blocks (F_only vs. V_co; F_only vs.

V_dis; and F_only vs. FV_dis) were differentiable in task accuracy from the Bonferroni post hoc test. F_only block was significantly more error prone then the V_co, V_dis and FV_dis blocks.

Subjective Data Usf (%) Eff (%) Pls (%) Wkl V_co 73 ± 8 73 ± 11 71 ± 12 109 ±31 V_dis 75 ± 9 75 ± 13 72 ± 12 104 ±34 Testing Blocks F_only 76 ± 15 62 ± 22 56 ± 21 116 ±30 (mean ± SD*) FV_co 74 ± 11 73 ± 11 72 ± 13 110 ±46 FV_dis 77 ± 9 74 ± 11 75 ± 12 107 ±32 F (4, 49) 0.58 3.77 3.95 0.52 ANOVA p < 0.05 —   —

*SD: Standard deviation. Table 3.2: The means and standard deviations of the subjective data and their ANOVA results among all testing blocks in Experiment 1

As described in Table 3.2, ANOVA analyses yielded no significant difference on the perceived usefulness and workload among all testing blocks. However, a significant difference was present for perceived pleasure and effectiveness. Post hoc tests revealed a significant difference between the F_only block and every other testing block for the perceived pleasure and effectiveness. This indicated that the vibrotactile cue promoted a better accuracy, effectiveness, and pleasure than the force cue alone.

MLE analyses of the task accuracy data revealed that MLE was unable to predict the cue integration, as seen in Figure 3.8. In the co-located case of Figure 3.8 (a), MLE prediction

(FV_co (MLE)) was able to reasonable match the mean of the observed integration (FV_co), however variance was unmatched. In the dis-located case of Figure 3.8 (b), MLE failed to match in either mean or variance.

(a) (b)

Figure 3.8: MLE analyses of task accuracy data for Experiment 1 in (a) co-located and (b) dis-located settings.

3.3 Experiment 2: Reliance Biased

Experiment 2 replicated most of Experiment 1 while altering the conditions slightly. The only alteration was that the participants were verbally instructed to favor the vibrotactile cue before each testing block of the experiment, thereby inducing a vibrotactile reliance bias. This bias was introduced to obtain results differing to those of Experiment 1 to evaluate MLE under different conditions.

3.3.1 Methods

VIRTUAL ENVIRONMENT: We used the same VE from the Experiment 1.

PARTICIPANTS: Ten participants unique from those of Experiment 1 (mean age of 27.13 ±

5.13 years old) took part in the study and underwent the same pre-screening tests with the same eligibility requirements as Experiment 1. Prior to conducting the experiment, the participants were verbally instructed to be biased towards the vibrotactile cue.

HARDWARE: The stereoscopic visual scene of the VE was projected on a screen (2.4×2.4 m2, participant sat 2 m away) using an Acer U5200 projector (Acer Inc., Taiwan) operated at 120

Hz. A pair of 3D shutter glasses and an IR emitter (nVidia Inc., Santa Clara, USA) allowed each participant to view the scene. A VibroTac bracelet delivered the vibrotactile cues. The VE was run on a Dell Precision WorkStation T3500 (Dell Inc., USA) under the Microsoft Windows® 7 operating system with an Intel Xeon E5507 CPU, an nVidia Quadro FX4800 graphics card, and

8 GB of RAM.

PROCEDURE: The procedures are the same as those are Experiment 1.

DATA COLLECTION AND ANALYSES: We used the same data collection and analyses methods as those of Experiment 1.

3.3.2 Results

Confirmed by the physiological data and the responses to the cybersickness questionnaire, none of the participants suffered from cybersickness. Therefore, data from all participants were used in the analyses. Objective and subjective data were all normally distributed and therefore eligible for ANOVA analyses.

Figure 3.9: Average task completion time of all testing blocks for experiment 3. [Error bars represent standard errors.]

Figure 3.10: Average task accuracy of all testing blocks for Experiment 2. [Error bars represent standard errors. The symbol of *** denotes Bonferroni significant differences with p < 0.001.]

From the objective data, the task completion time of each testing block was depicted in

Figure 3.9. Similar to Experiment 1, there were no significant different among all testing blocks from ANOVA analyses of the completion time [F(4, 49) = 1.65, p > 0.05]. Again similar to

Experiment 1, a significant difference in task accuracy was present among all blocks [F(4, 49) =

24.08, p < 0.01] as shown in Figure 3.10. Post hoc tests indicated that the difference came from the F_only block as well, compared to other blocks. In other words, F_only block was significantly less accuracy then other blocks.

Subjective Data Usf (%) Eff (%) Pls (%) Wkl V_co 67 ± 18 63 ± 17 65 ± 18 126±38 V_dis 60 ± 22 57 ± 19 61 ± 22 138±26 Testing Blocks F_only 65 ± 20 53 ± 23 60 ± 22 145±28 (mean ± SD*) FV_co 65 ± 15 61 ± 20 69 ± 22 144±24 FV_dis 60 ± 24 56 ± 23 64 ± 23 139±33 F (4, 49) 0.65 0.92 0.83 1.79 ANOVA p < 0.05 — — — 

* SD: Standard deviation. Table 3.3: The means and standard deviations of the subjective data and their ANOVA results among all testing blocks in Experiment 2

Summarized in Table 3.3, ANOVA analyses of the subjective data revealed that workload gave a significant difference among testing blocks. Bonferroni post hoc tests yielded that the difference arose from the V_co vs. F_only blocks [F(1, 9) = 1.84, p < 0.05] and the V_co vs.

FV_co blocks [F(1, 9) = 2.46, p < 0.05]. ANOVA analyses of perceived usefulness, effectiveness and pleasure revealed no significant difference among the testing blocks. This indicated the presence of vibrotactile cue promoted better accuracy and reduced the workload of the task.

Analyses of task accuracy with MLE data revealed that MLE was unable to predict the cue integration, as seen in Figure 3.11. The MLE prediction did not match the observed integration in the co-located case of Figure 3.11 (a) nor dis-located case of Figure 3.11 (b). In the co-located case, the observed integration had higher accuracy than either individual cues while this effect did not occur in the dis-located case. This agrees with existing work that co-location of cues facilitates better task performance.

(a) (b)

Figure 3.11: MLE analyses of task accuracy data for Experiment 2 in (a) co-located and (b) dis-located settings.

3.4 Experiment 3: Intensity Reduced

Both Experiment 1 and 2 indicated that strong vibrotactile cues enhanced the detection accuracy.

However, if weaker vibrotactile cues are delivered, would that reduce the detection accuracy or increase the variance? Also, would the relatively stronger force cues compared to vibrotactile cues change the integration so that it might match MLE more closely? In Experiment 3, again

Experiment 1 was replicated with only one difference. The only difference compared to

Experiment 1 was that the perceived intensity of the vibrotactile cue was halved. This was done to reduce the task enhancing effect from delivering vibrotactile cues for the detection task seen in

Experiment 1 and 2. It should be noted that the participants did not have a reliance bias that was present in Experiment 2.

3.4.1 Methods

VIRTUAL ENVIRONMENTS: Experiment 3 used the same VE as Experiment 1.

PARTICIPANTS: Ten participants unique from those of Experiment 1 and 2 (mean age of

24.80 ± 4.35 years old) took part in the study. The participants also underwent the same pre- screening tests with identical eligibility requirements as the previous experiments. Ethics approval was attained at our institute.

HARDWARE: A graphics workstation with an Intel Xeon E5-2650 v4 CPU processor with nVidia Quadro M5000 graphics card and 32 GB of RAM running with Microsoft Windows®10 operating system was used to execute the VE. The visual scene was projected on the center screen (3x3 m2, with the participant 2.5 m away) of a CAVE. The vibrotactile cue was delivered with a vibrotactile device built in-house. The device consisted of a motor driver board build with an Atmel SM4S Xplained Pro evaluation kit (Atmel Corp., USA), and a vibrator with a diameter of 10 mm (Precision Microdrives, UK). A Windows® driver in C++ was developed to allow the

VE to operate the vibrotactile device. In contrast to the previous two experiments, this vibrotactile device was configured to deliver the vibrotactile cue at half intensity as perceived by the participants [59].

PROCEDURE: The procedures are the same as those are Experiment 1.

DATA COLLECTION AND ANALYSES: We used the same data collection and analyses methods as those of Experiment 1.

3.4.2 Results

All data from all participants were eligible for analyses as none of the participants suffered from cybersickness, indicated by the physiological data and the responses to the cybersickness

Figure 3.12: Average task completion time of all testing blocks for Experiment 3. [Error bars represent standard errors.]

Figure 3.13: Average task accuracy of all testing blocks for Experiment 3. [Error bars represent standard errors. The symbol of *** denotes Bonferroni significant differences with p < 0.001.] questionnaire. All other objective and subjective data were normally distributed as indicated by normality tests and therefore eligible for ANOVA analyses.

Similar to Experiment 1 and 2, there were no significant different among all testing blocks from ANOVA analyses of the completion time shown in Figure 3.12 [F(4, 49) = 1.01, p > 0.05].

A significant difference in task accuracy shown in Figure 3.13 was present among all blocks

[F(4, 49) = 36.64, p < 0.01] as well. Post hoc tests indicated that, similar to Experiment 1 and 2, the difference came from the F_only block compared to other blocks. 38

Subjective Data Usf (%) Eff (%) Pls (%) Wkl V_co 73 ± 12 68 ± 14 64 ± 19 131±24 V_dis 70 ± 13 71 ± 12 64 ± 20 134±31 Testing Blocks F_only 65 ± 8 62 ± 15 63 ± 20 132±27 (mean ± SD*) FV_co 66 ± 17 67 ± 14 68 ± 15 123±32 FV_dis 64 ± 12 65 ± 11 62 ± 14 133±28 F (4, 49) 0.31 0.36 0.62 0.33 ANOVA p < 0.05 — — — —

*SD: Standard deviation. Table 3.4: The means and standard deviations of the subjective data and their ANOVA results among all testing blocks in Experiment 3.

ANOVA analyses of the subjective data, presented in Table 3.4, indicated that there were no significant different among testing blocks for the metrics used. This result was not unexpected due to the reduced perceived intensity of the vibrotactile cue in this experiment. By reducing the perceived intensity of the vibrotactile cue, its relative effect compared to the force cue was reduced. This eliminated the subjective improvement from introducing the vibrotactile cue as seen in Experiment 1 and 2. However, task accuracy was still significantly improved by the presence of the vibrotactile cue regardless of the participants subjective response. It is possible that the subjective responses are less sensitive than the objective metric, however this investigation is out of the scope of this thesis.

Seen in Figure 3.14, MLE was unable to predict the cue integration in either mean or variance. The observed integration in the co-located case of Figure 3.14 (a) and dis-located case of Figure 3.14 (b) did not match their respective MLE prediction. Due to the reduced vibrotactile cue intensity in the experiment, the participants were less certain about the vibrotactile cue

(a) (b)

Figure 3.14: MLE analyses of task accuracy data for Experiment 3 in (a) co-located and (b) dis-located settings. compared to both Experiment 1 and Experiment 2. This is seen as an increase in variance of the individual vibrotactile cues in both Figure 3.14 (a) and Figure 3.14 (b).

3.5 General Discussion

The outcomes of both task completion time and task accuracy were similar across the three experiments. The reliance bias towards the vibrotactile cue in Experiment 2 and the reduced vibrotactile cue intensity in Experiment 3 had no effect on accuracy. This agreed with existing work on sensing surfaces with tactile cues [18].

The ANOVA analyses of both subjective and objective data resulted in similar outcomes across the three experiments, which validates the replication of the experiments. Task accuracy were significantly increased when vibrotactile cues were used. Vibrotactile cue is often used as a notification tool such as vibration alarms [36]. It is likely that the vibrotactile cues served as a notification to the participants to the existence of a defect, hence the increased task accuracy. It is also possible that the presence of the continuous force reduces the efficient perception of the

force cue. Furthermore, the visual scene of the flying drone along the transmission line involves changes in position of the clamp with respect to the line. These changes could influence the integration of both force and vibrotactile cues, as reported in literature [27].

For all three experiments, we estimated Gaussian-distributed observations of the individual cues, and yielded Gaussian-distributed observations and predictions of their cue integration with

MLE. For both co-located and dis-located settings, the observations and predictions were derived and presented in Figure 3.15. It is noted that in almost all cases, the combined observations

(FV_co and FV_dis) resulted in higher accuracy in the detection task, which agreed with the

ANOVA analyses.

MLE failed for five out of six cases presented in Figure 3.15 (b), (c), (d), (e), and (f) when integrating observations of the individual cues. When compared to the observation of the combined cues, the MLE cue integration did not match in either mean nor variance in these five cases. Figure 3.15 (a) presented an exception, which is a case of a reasonable mean-match.

However, MLE remained unable to explain the variance mismatch in this exceptional case. The

MLE integration of the reliance-biased data and vibrotactile intensity reduced data and their corresponding combined observations were unmatched, as seen in Figure 3.15 (c), (d), (e), and

(f). In the reliance-biased case, the combined observations were close to those of the V_co and

V_dis blocks. This closeness was especially pronounced in the dis-located setting, as depicted in

Figure 3.15 (d). Because of the vibrotactile bias, it is plausible the participants relied heavily on the vibrotactile cue, with the effect being magnified when applied in the dis-located setting.

In the intensity reduced setting (Experiment 3), the variance of the force cue only observations were lower than the vibrotactile only observations, as depicted in Figure 3.15 (e) and (f). This is in contrast with the individual cue observations in unbiased and reliance-biased data, both exhibited a lower variance in vibrotactile observations than force observations. This can be attributed directly to the intensity of the vibrotactile cue being less than the other two experiments. This resulted in the participants having less certainty in the vibrotactile cue compared to the force cue, and therefore a higher variance in the vibrotactile observations.

The participants still utilized the force cue to a certain degree in all cases, even in the case where the participant is vibrotactile biased and the cues are delivered in the dis-located settings.

This behavior was more pronounced when the force cue was delivered at a co-location with the vibrotactile cue. This manifested as a decrease in variance of the combined observation (i.e. combined cues resulted in more certainty in the detection task) as depicted in Figure 3.15 (b), (c),

(d), (e) and (f). The exception, depicted in Figure 3.15 (a), still indicated the participants took account of the information from the force cues. In this case, the participants relied on the force cue much more than in all other settings.

Co-located Dis-located

(a) (b)

Reliance Unbiased Reliance

Reliance Biased Reliance

(e) (f)

Intensity Reduced Intensity

Figure 3.15: Gaussian-distributed estimations of MLE on the experimental data in: (a) reliance-unbiased co-located; (b) reliance-unbiased dis-located; (c) reliance-biased co- located; (d) reliance-biased dis-located; (e) intensity-reduced co-located; (f) intensity-reduced dis-located settings.

In the reliance unbiased case, the distribution of the co-located vibrotactile and force cues setting (FV_co) had a smaller mean and larger variance than the dis-located cue setting (FV_dis).

This implies that the co-located setting is less accurate and less certain than the dis-located setting. This disagrees with observations that co-located cues improves task performance [1].

However, it is possible that vibrotactile cue dis-located from the force cue caused the participant to be more certain about the vibrotactile cue as a notification. Co-located with the vibrotactile cue, the force cue may have acted as extraneous noise to the vibrotactile cue for the participant.

3.6 Conclusion

The result of the attempt of MLE to integrate vibrotactile and force cues did not yield satisfactory integrations compared to the respective empirical observations. Mismatches of mean and variance were both present between MLE predictions and the observations. This implies that the integration of vibrotactile and force cues do not follow the rules of MLE. In Chapter 4, these mismatches will be investigated, and Proportional Likelihood Estimation is proposed as the model of integration.

Chapter 4

Proportional Likelihood Estimation for Vibrotactile and

Force Cue Integration3

In Chapter 3, MLE was found to be inadequate when employed to integrate vibrotactile and force cues, both in the haptic modality. When MLE was applied to integrate individual vibrotactile and force cues, the result was not a satisfactory match to the observed cue combination in either co- located or dis-located settings. However, the mathematical formulation of MLE provided a hint towards the next steps of investigation. In MLE, it is assumed that the cue integration must be additive. This is shown by the integration weights of each individual cue in MLE must necessarily be positive. Therefore, what if this assumption is not necessarily true for vibrotactile and force cue integration? What modifications of MLE are needed to remedy the mean and variance mismatches found in Chapter 3? To resolve the mismatches, Proportional Likelihood

Estimation (PLE) was created and proposed for this thesis, without the additive assumptions of

MLE. PLE instead formulated the integration weight such that they are not limited to positive

3 Parts of this chapter are published/to be submitted to: S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, “Vibrotactile and Force Collaboration within 3D Virtual Environments,” Proc. IEEE International Conference on Computer Supported Cooperative Work in Design, Nanjing, China, May 2018, pp. 330-335. S. Tarng, A. Erfanian, Y. Hu, and F. Merienne, “Additive or Not: A Question for Integrating Vibrotactile and Force Cues in 3D Interactive Tasks,” to be submitted to IEEE Transactions on Human-Machine Systems, 10 pages (IEEE double-column format), Aug. 2018.

numbers. This allows PLE prediction to correctly match the observed integration through the use of negative weights. The presence of negative weights implies the mechanism of integration behind vibrotactile and force cues is not the same as the mechanism behind integration of cues of different modalities.

This chapter describes the formulation of PLE and applies it to the results of the three experiments in Chapter 3. The results indicated that the integration of vibrotactile and force cues is not necessarily positive reinforcements of the two individual cues.

4.1 Proportional Likelihood Estimation

As MLE was inadequate when integrating vibrotactile and force cues, modifications to it was necessary. MLE yielded mismatches between the MLE prediction and the observed cue integration (Figure 4.1) in both mean and variance/amplitude when applied. The goal of PLE is to resolve these mismatches. As mean and variance are independent from each other in a

Figure 4.1: MLE for a cue integration with mean and amplitude (therefore variance) mismatch from the cue collaboration. 46

) ) ) )

)

2 4

. .

3.1 3.4 3.5 4 4

( ( ( ( (

푉

2 2

푉 푉

퐹푉

휎

휎 휎

휎

−

− −

−

퐹

2 2

퐹 퐹푉

휎

휎 휎

2 2

푉 퐹

휎 휎

푊 푊

) )

1 3

PLE

. .

4 4

( (

푉

)

퐹푉

휇

퐹

휇

)

퐺

−

퐹

−

퐹

−

퐺

휎

퐹

) 퐹

퐹

퐹푉

휇

휇 휎

휇

휇퐹

푊

−

푎

푊

(

−

푉 퐹

푉

푒

휇 휇

퐺

푉

푊 푊

푉

휎

퐺

휎

휋

휇푉

푊

(

푊

√

(

퐸

푉푎푟

퐺

(3.7) (3.8)

휇

퐹

휎

푉

퐹

휎

푉

휎

MLE

푉

퐹

휎

푊

퐹

푉

휇

푊

ormulation of MLE and PLE for vibrotactile and force cues vibrotactile and for force and PLE MLE of ormulation

퐹

푉

푊

Force

Vibrotactile

Mean

Summary of f Summary

Variance

Method

Integration

Prediction Weights Integration Integration

47 Table

Gaussian distribution, the two mismatches can be handled separately. Therefore two pairs of weights, one for mean and one for variance, can be used instead of a single pair of weights employed by MLE to integrate both mean and variance.

Table 4.1 provides a summary of MLE and PLE formulation. Equations from Chapter 3 are included here for comparison and completeness. Similar to MLE, each individual cue in PLE is estimated as a Gaussian distribution. Also similar to MLE, the resulting cue integration to be estimated (퐺̂) is also Gaussian distributed (Eq. (3.1)). The estimation of the characteristic parameters (mean and variance) are also done in a similar way (Eqs. (3.4) and (3.5)).

Where PLE differs from MLE is the formulation of individual weights (푊 , 푊 , 푊 2 and 휇푉 휇퐹 휎푉

푊 2) for each cue used for the cue integration. MLE only has one set of weights applied to both 휎퐹 mean and variance integration, as described by Eqs. (3.6) and (3.7). The formulation of these weights in MLE depends only on the variance of the individual cues, with the resulting weights necessarily bounded between 0 and 1. In contrast to MLE, mean weights and variance weights in

PLE are independently calculated as mean and variance are independent from each other. The mean weights (푊휇푉 and 푊휇퐹) of individual cues in PLE are derived by using mean differences between the observations of the cues and their empirical integration. This is described in Eqs.

(4.1) and (4.3), where μV, μF, and μFV are the means of the observations of the vibrotactile cue, the force cue and their cue integration, respectively. Variance weights (푊 2 and 푊 2) are 휎푉 휎퐹

2 2 2 calculated in a similar fashion in Eqs. (4.2) and (4.4), where 휎푉 , 휎퐹 and 휎퐹푉 represents the variance of the observations of the vibrotactile, force cue, and their cue integration respectively.

Essentially, the weights are proportional to the difference between the mean/variance of the integration and the mean/variance individual cues, hence the name proportional likelihood.

In essence, the weights of each cue are what their MLE weights would be to ensure a match in mean and variance between the prediction and the observation of the cue integration. When the mean and variance PLE weights are used together, the integration would match the observation. Due to the formulations of the weights in PLE, the weights are not limited to positive numbers, nor do they have an upper limit of 1. Therefore, integration with PLE is not necessarily positive reinforcements of individual cues, as implied by MLE. This can offer insights into the mechanism of integration between vibrotactile and force cues.

4.2 Analyses with PLE and Discussion

Similar to the method MLE was applied to the task accuracy data in the three experiments in

Chapter 3, PLE was also applied to the task accuracy data. In Figure 4.2, the resulting cue integrations was overlaid on Figure 3.15 and the MLE and PLE weights were summarized and included. MLE integration from Chapter 3 is included for comparison with PLE.

In contrast to MLE, PLE matched the observations of the cue integration in all cases of

Figure 3.15 as expected from the formulation. All weights, including MLE weights, were also listed with their respective integration in the figure. PLE required both amplitude and variance weight to have a sufficient match to the empirical observation. Using amplitude weights alone did not remedy the variance/amplitude mismatch; likewise, using variance weights alone did not remedy the mean mismatch either. A Gaussian model of a cue is described by two independent 49

Co-located Dis-located

(a) (b)

푾 푾 ퟐ 푾 푾 ퟐ 푾푽 흁푽 흈푽 푾푽 흁푽 흈푽 0.638 0.565 -0.472 0.554 1.43 2.3

푾 푾 ퟐ 푾 푾 ퟐ 푾푭 흁푭 흈푭 푾푭 흁푭 흈푭 0.362 0.435 1.47 0.446 -0.43 -1.3

Reliance Unbiased Reliance

푾 푾 ퟐ 푾 푾 ퟐ (c) 푾푽 흁푽 흈푽 (d) 푾푽 흁푽 흈푽 0.691 1.05 1.26 0.672 0.973 1.16

푾 푾 ퟐ 푾 푾 ퟐ 푾푭 흁푭 흈푭 푾푭 흁푭 흈푭 0.309 -0.05 -0.26 0.328 0.027 -0.16

Reliance Biased Reliance

푾 푾 푾 ퟐ 푾 푾 푾 ퟐ (e) 푽 흁푽 흈푽 (f) 푽 흁푽 흈푽

0.473 1.08 -7.71 0.384 1.03 -0.06

푾 푾 ퟐ 푾 푾 ퟐ 푾푭 흁푭 흈푭 푾푭 흁푭 흈푭 0.527 -0.08 8.71 0.616 -0.03 1.06

Intensity Reduced Intensity

Figure 4.2: Gaussian-distributed estimations of MLE and PLE and their weights on the experimental data in: (a) reliance-unbiased co-located; (b) reliance-unbiased dis-located; (c) reliance-biased co-located; (d) reliance-biased dis-located; (e) intensity-reduced co-located; (f) intensity-reduced dis-located settings.

variables: mean and variance. It follows that to describe an integration fully, it is necessary to process the two variables independently.

The weights derived using PLE were also not limited to positive weights unlike MLE.

Indeed, despite the additive assumption in both MLE and PLE, PLE resulted in some cues with negative weights in the integration. MLE assumes entirely additive roles of individual cues in their integration. That is, cue integrations are wholly positive reinforcement of the entire set of cues.

Wholly positive reinforcement of cues might be true in the integration between visual and haptic cues, between visual and auditory cues, as well between force and position cues. These integrations are all between different modalities. However, if the mechanism of integration behind two sub-categorial cues of the same modality is not wholly positive reinforcements of each other, this may not be true. When integrating the vibrotactile and force cues, two sub- categorial cues of the same modality, PLE could remedy both the mean and variance mismatches resulted from MLE. Shared by both MLE and PLE, as seen in Eq. (3.4) and (3.5), both predicts the cue integration by additively combine the weighted observations of all individual cues.

However, to sufficiently match the prediction with the empirical observations, negative weights were necessary as shown by PLE. This may hint that because vibrotactile and force cues both belongs to the haptic modality, together they are not simply additively combined as in the case of integration cues of different modalities.

An example where multi-sensory cue integration followed the rules of MLE is the integration between visual and haptic cues. In literature, it is found that visual cues decoding correlates well

to the occipital lobe along with the ventral pathway in the central nervous system [60]. Haptic cues decoding was mainly found in the parietal lobe, including the frontal cortex [60]. These two pathways do not overlap with each other and is well suited for MLE as a model of integration. In contrast, if the two cues being integrated are both in the same modality, such a haptics, they may undergo the same processing in similar areas of the nervous system. Such a case may result in unknown interaction between the two sub-categorical cues of the same modality. However, while the two cues may not be positively reinforcing each other, it does not mean that one of the cues is being ignored. A negative cue integration is still nevertheless a cue integration. Thus, the additive assumption may not hold true for integration cues in the same modality. This agrees with the hypothesis that the integration of vibrotactile and force cues follow other integration mechanisms than MLE. In the future, a different model of integration of the vibrotactile and force cues might need a model without additive assumptions.

4.3 Potential Applications

The VE and the detection task for the study was chosen to be easily transferable to other applications. The curved wire navigation in the VE is similar to the exploration of virtual 3D fluid models used for investigating airflow around a vehicle or airplane, as well as fluid flow in pipes and reservoirs. The task of power line defect detection in remote areas using a drone is applicable to the difficulties in the real world to accomplish this task. Drone piloting using haptic devices is also of potential military and commercial applications.

Based on the observations of the study, there are two aspects that can affect the implementation of vibrotactile and force cues in VEs. First, the vibrotactile cue works exceptionally well as a source of notification compared force cues when used for the same task.

However, using force cues in collaboration with vibrotactile cues still facilitated a better detection result. Exploratory tasks where users examine abstract models in VEs through both visual and haptic displays should also combine multiple modalities for notifications. This can facilitate more intuitive exploration and result in more efficient retrieval of information from these models. VE equipment should therefore offer both vibrotactile and force feedback capabilities wherever possible and provide notifications with both types of cues for better results.

Many VE systems are currently only able to deliver one type of haptic cue, such as the Omni device with force cues and HTC Vive controllers with vibrotactile cues. Currently this is due to hardware limitations, but recent work in wearable haptics might offer possible solutions in the future [61].

Second, PLE was shown to be able to measure contributions between two separate cues to a degree through the weights. While a more comprehensive study is required to evaluate the precise measure from these weights, they serve as an indicator to the nature of the integration. It is possible to apply this to other types of integrations. When two sensory cues are perceived to have a common cause, they are integrated. However, humans are constantly bombarded with many sights, sounds, odors, and haptic cues. Therefore the human nervous system continuously needs to determine which cues should be integrated, and which should be processed separately

[62]. Understanding this process is important for effective use of cues in VEs. Ineffective cues

results in the user missing crucial information. The cues can be subconsciously ignored because their implementation leads the user to conclude the cues are unimportant. This is inattention blindness, where a user fails to perceive a cue due to perceptual overload [63]. It is possible that

PLE can be used to indicate if the cues are integrated or processed separately instead, however further work in required.

Chapter 5

Summary and Future Work

This chapter summarizes the achievements and results of the research of this thesis and provides possible extensions of this work in the future.

5.1 Summary

As described in Chapter 1, VEs are quickly becoming an attractive solution to enhance industrial and research processes. Existing hardware have primarily driven the use of the visual modality in these VEs. Indeed, because visual modality is the most important sense to human users, it is desirable to have VEs that are immersive and convincing in that modality. However, with the increase of popularity of VEs comes the increase of users’ demand in both technical and feature capabilities of these VEs. Adding features and information through the visual modality is the most straightforward method, but this can easily overwhelm the user if it is not carefully executed. With limited amount of visual focus and information processing capability of the users, information from other modalities becomes desirable to avoid overwhelming the visual modality.

Currently, many hardware systems for VEs have purpose-built haptic devices, such as game controllers. These devices offer interactivity for a user to interact with aspects of the VE. Many of these haptic devices also provides active haptic feedback, frequently in the form of 55

vibrotactile or force cues. These cues can be delivered in either co-located or dis-located settings when used in a VE. While co-located cues offer better performance in the task of perception, discrimination, and tool steering, dis-located cues remain a mean to improve user performance in the VE. While vibrotactile and force cues are frequently delivered together in VEs, few efforts have been devoted to the investigation of the mechanism of their cue integration. A general mechanism of integrating vibrotactile and force cues is necessary to best facilitate user performance in VEs.

This research first aimed to evaluate MLE as a mechanism of integration for vibrotactile and force cues. MLE was chosen because it was shown to be followed when integrating multi- sensory cues, such as visual and haptic cues. Three experiments were conducted with a combination of co-located and dis-located vibrotactile and/or force cues under varying cue bias and cue intensity. MLE was unable to integration the individual vibrotactile and force cues to satisfactorily match the observed combination of the two cues. This provided a confirmation that the mechanism of integration of vibrotactile and force cues, two sub-categorical cues of the same modality, does not necessarily follow the same mechanism as integration of two cues of different modalities.

The second aim of this research was to provide a model of integration for integrating vibrotactile and force cues. For this, PLE was proposed as a modification to MLE. PLE revealed the additive assumption of cues in MLE may not be correct when integrating vibrotactile and force cues. It was seen that it is possible for the two cues to not be additive with each other in a

cue integration. In the future, a model of integration without the additive assumption of MLE is required when modeling the integration of vibrotactile and force cues.

5.2 Future Work

The three experiments conducted for this research only covered a few specific combinations of the many possible variations to apply vibrotactile and force cues to the participants. This research has explored possible effects of participant bias towards either cues to the integration, however it remains to be just one specific case. It is possible to bias the participants differently to obtain different results from the cue integration. This research also explored the effects of a varied vibrotactile cue intensity and obtained a reasonable corresponding change in the cue integration. However, it remains possible to conduct experiments to have different cue intensity levels. The cue intensities can also be absolute or relative to the other cue. Due to the limitation of time and resources devoted for this master’s thesis, many possible variations were removed from the scope of the study. Nevertheless, these types of changes may reveal interesting changes to the vibrotactile and force cue integration and reveal addition insights.

The power transmission line in the VE used in the experiments are curved surfaces. This is useful for engineering purposes because curved surfaces appear in many statistical models explored in VEs. However, other types of surfaces may be desirable to explore since this research is fundamental in nature. Different types of surfaces are necessary to further confirm the findings in this research.

While the current research proposed PLE as a model of integration of vibrotactile and force cues, it does not serve this purpose fully. Instead, PLE primarily offered insights behind the mechanism of integration with the weights of the integration. PLE currently remains a model of insufficient predictive power as it relies on the observation of the combined cues. Additional cases can be obtained with different experiment conditions to find more integration weights, and an emerging pattern may be gleaned. There are also other models of integration used in literature when integrating multi-sensory cues, such as signal detection theory and information integration theory. These models can be investigated for their suitability for integrating vibrotactile and force cues as they have different fundamental assumptions unlike those of MLE.

The possibility that due to both being sub-categorical cues of the haptic modality, vibrotactile and force cue integration is treated differently than two cues of different modalities offer interesting investigation avenues. Two haptic cues using the same processing pathways in the central nervous system may integrate differently than two cues of different modalities using two different pathways. Investigation with functional magnetic resonance imaging (fMRI) or electroencephalography (EEG) may reveal interesting insights into how multiple cues are integrated in general.

Bibliography

[1] D. Swapp, V. Pawar, and C. Loscos, “Interaction with co-located haptic feedback in

virtual reality,” Virtual Real., vol. 10, no. 1, pp. 24–30, 2006.

[2] D. A. Bowman, R. P. Mcmahan, and V. Tech, “Virtual Reality: How Much Immersion Is

Enough? (Cover story),” Computer, vol. 40, no. 7, pp. 36–43, 2007

[3] E. M. Lidal, T. Langeland, C. Giertsen, J. Grimsgaard, and R. Helland, “A decade of

increased oil recovery in virtual reality,” IEEE Comput. Graph. Appl., vol. 27, no. 6, pp.

94–97, 2007.

[4] A. K. Bejczy, “Virtual reality in manufacturing,” in Re-engineering for Sustainable

Industrial Production: Proceedings of the OE/IFIP/IEEE International Conference on

Integrated and Sustainable Industrial Production, Lisbon, Portugal, May 1997, pp. 48–60

[5] D. Reiner, D. Stricker, G. Klinker, and S. Muller, “Augmented Reality for Construction

Tasks: Doorlock Assembly,” Int. Work. Augment. Realit, no. October, pp. 31–46, 1998.

[6] S. Gallo et al., “Towards multimodal haptics for teleoperation: Design of a tactile thermal

display,” in Proceedings of the 2012 12th IEEE International Workshop on Advanced

Motion Control (AMC), 2012, pp. 1–5..

[7] M. Agus, A. Giachetti, E. Gobbetti, G. Zanetti, A. Zorcolo, B. Picasso, and S. S.

Franceschini, “A haptic model of a bone-cutting burr,” Stud. Health Technol. Inform., vol.

94, no. February, pp. 4–10, 2003.

[8] D. Monaghan, J. O’Sullivan, N. E. O’Connor, B. Kelly, O. Kazmierczak, and L. Comer,

“Low-cost creation of a 3D interactive museum exhibition,” Proc. 19th ACM Int. Conf. 59

Multimed. - MM ’11, vol. 1, no. 212, p. 823, 2011.

[9] L. Freina and M. Ott, “A literature review on immersive virtual reality in education: State

of the art and perspectives,” Proc. eLearning Softw. Educ. (eLSE), Bucharest, Rom. April,

2015.

[10] M. Zyda, “From visual simulation to virtual reality to games,” Computer, vol. 38, no. 9,

pp. 25–32, 2005.

[11] N. Ouarti, A. Lécuyer, and A. Berthoz, “Haptic motion: Improving sensation of self-

motion in virtual worlds with force feedback,” in IEEE Haptics Symposium, 2014, pp.

167–174.

[12] H. Q. Dinh, N. Walker, L. F. Hodges, C. Chang Song, and A. Kobayashi, “Evaluating the

importance of multi-sensory input on memory and the sense of presence in virtual

environments,” in Proceedings IEEE Virtual Reality, 1999, pp. 222–228.

[13] M. O. Ernst and M. S. Banks, “Humans integrate visual and haptic information in a

statistically optimal fashion.,” Nature, vol. 415, no. 6870, pp. 429–433, 2002.

[14] O. Gerovichev, P. Marayong, and A. M. Okamura, “The Effect of Visual and Haptic

Feedback on Manual and Teleoperated Needle Insertion,” MICCAI ’02 Proc. 5th Int.

Conf. Med. Image Comput. Comput. Interv. I, pp. 147–154, 2002.

[15] S. J. Lederman, A. Martin, C. Tong, and R. L. Klatzky, “Relative performance using

haptic and/or touch-produced auditory cues in a remote absolute texture identification

task,” in Proceedings - 11th Symposium on Haptic Interfaces for Virtual Environment and

Teleoperator Systems, 2003, pp. 151–158.

[16] R. Sigrist, G. Rauter, R. Riener, and P. Wolf, “Augmented visual, auditory, haptic, and

multimodal feedback in motor learning: A review,” Psychon. Bull. Rev., vol. 20, no. 1, pp.

21–53, 2013.

[17] J. D. Brown, S. Member, and R. B. Gillespie, “The Effect of Force / Motion Coupling on

Motor and Cognitive Performance,” in IEEE World Haptics, 2011, pp. 197–202.

[18] S. J. Lederman and R. L. Klatzky, “Haptic perception: A tutorial,” Attention, Perception,

Psychophys., vol. 71, no. 7, pp. 1439–1459, Oct. 2009.

[19] C. Pacchierotti, M. Abayazid, S. Misra, and D. Prattichizzo, “Steering of flexible needles

combining kinesthetic and vibratory force feedback,” in 2014 IEEE/RSJ International

Conference on Intelligent Robots and Systems, 2014, pp. 1202–1207.

[20] C. Pacchierotti, A. Tirmizi, G. Bianchini, and D. Prattichizzo, “Enhancing the

Performance of Passive Teleoperation Systems via Cutaneous Feedback,” IEEE Trans.

Haptics, vol. 8, no. 4, pp. 397–409, Oct. 2015.

[21] D. Prattichizzo, C. Pacchierotti, and G. Rosati, “Cutaneous Force Feedback as a Sensory

Subtraction Technique in Haptics,” IEEE Trans. Haptics, vol. 5, no. 4, pp. 289–300, 2012.

[22] Z. F. Quek, S. B. Schorr, I. Nisky, A. M. Okamura, and W. R. Provancher, “Augmentation

Of Stiffness Perception With a 1-Degree-of-Freedom Skin Stretch Device,” IEEE Trans.

Human-Machine Syst., vol. 44, no. 6, pp. 731–742, Dec. 2014.

[23] Z. F. Quek, S. B. Schorr, I. Nisky, W. R. Provancher, and A. M. Okamura, “Sensory

Substitution and Augmentation Using 3-Degree-of-Freedom Skin Deformation

Feedback,” IEEE Trans. Haptics, vol. 8, no. 2, pp. 209–221, Apr. 2015.

[24] A. M. Okamura, R. J. Webster, J. T. Nolin, K. W. Johnson, and H. Jafry, “The haptic

scissors: cutting in virtual environments,” in Proceedings of 2003 IEEE Int. Conf. Robot.

Autom,, vol. 1, no. 1, pp. 828–833, 2003.

[25] F. Zheng, W. F. Lu, Y. S. Wong, and K. W. C. Foong, “An analytical drilling force model

and GPU-accelerated haptics-based simulation framework of the pilot drilling procedure

for micro-implants surgery training,” Comput. Methods Programs Biomed., vol. 108, no.

3, pp. 1170–1184, 2012.

[26] J. M. Hillis, M. O. Ernst, M. S. Banks, and M. S. Landy, “Combining sensory

information: mandatory fusion within, but not between, senses.,” Science, vol. 298, no.

5598, pp. 1627–1630, 2002.

[27] K. Drewing and M. O. Ernst, “Integration of force and position cues for shape perception

through active touch,” Brain Res., vol. 1078, no. 1, pp. 92–100, 2006.

[28] M. Rohde, L. C. J. van Dam, and M. O. Ernst, “Statistically Optimal Multisensory Cue

Integration: A Practical Tutorial,” Multisens. Res., vol. 29, no. 4–5, pp. 279–317, 2015.

[29] R. Ratcliff and G. Mckoon, “The Diffusion Decision Model: Theory and Data for Two-

Choice Decision Tasks,” Neural Comput., vol. 20, no. 4, pp. 873–922, 2008.

[30] S. K. Lynn and L. F. Barrett, “‘Utilizing’ Signal Detection Theory,” Psychol. Sci., vol. 25,

no. 9, pp. 1663–1673, 2014.

[31] L. A. Jones and H. Z. Tan, “Application of psychophysical techniques to haptic research,”

IEEE Trans. Haptics, vol. 6, no. 3, pp. 268–284, 2013.

[32] L. Van Hulle, S. Van Damme, and G. Crombez, “Valid cues for auditory or

somatosensory targets affect their perception: A signal detection approach,” Perception,

vol. 42, no. 2, pp. 223–232, 2013.

[33] A. Perez-Bellido, M. O. Ernst, S. Soto-Faraco, and J. Lopez-Moliner, “Visual limitations

shape audio-visual integration.,” J. Vis., vol. 15, no. 14, p. 5, 2015.

[34] I. B. Witten and E. I. Knudsen, “Why seeing is believing: Merging auditory and visual

worlds,” Neuron, vol. 48, no. 3, pp. 489–496, 2005.

[35] C. Spence, C. Parise, and Y. Chen, “The Colavita Visual Dominance Effect - The Neural

Bases of Multisensory Processes,” in The Neural Bases of Multisensory Processes, Boca

Raton, FL: CRC Press, 2011, pp. 523–550.

[36] S. Schätzle, T. Ende, T. Wüsthoff, and C. Preusche, “VibroTac: An ergonomic and

versatile usable vibrotactile feedback device,” Proc. - IEEE Int. Work. Robot Hum.

Interact. Commun., pp. 670–675, 2010.

[37] C. Pacchierotti, M. Abayazid, S. Misra, and D. Prattichizzo, “Teleoperation of steerable

flexible needles by combining kinesthetic and vibratory feedback,” IEEE Trans. Haptics,

vol. 7, no. 4, pp. 551–556, 2014.

[38] S. R. Ellis, “What are virtual environments?,” Comput. Graph. Appl. IEEE, vol. 14, no. 1,

pp. 17–22, 1994.

[39] W. R. Sherman and A. B. Craig, Understanding virtual reality: Interface, application, and

design. Elsevier, 2002.

[40] M. L. McLaughlin, J. P. Hespanha, and G. S. Sukhatme, “Introduction to haptics,” Touch

in Virtual Environments: Haptics and the Design of Interactive Systems, vol. 64. Prentice-

Hall, 2002.

[41] R. S. Johansson and J. R. Flanagan, “Coding and use of tactile signals from the fingertips

in object manipulation tasks,” Nat. Rev. Neurosci., vol. 10, no. 5, pp. 345–359, 2009.

[42] N. H. Bakker, P. J. Werkhoven, and P. O. Passenier, “The effects of proprioceptive and

visual feedback on geographical orientation in virtual environments,” Presence

Teleoperators Virtual Environ., vol. 8, no. 1, pp. 36–53, 1999.

[43] “Touch Specifications,” 2018. [Online]. Available: https://www.3dsystems.com/haptics-

devices/geomagic-touch/specifications.

[44] A. M. Murray, R. L. Klatzky, and P. K. Khosla, “Psychophysical characterization and

testbed validation of a wearable vibrotactile glove for telemanipulation,” Presence

Teleoperators Virtual Environ., vol. 12, no. 2, pp. 156–182, 2003.

[45] G. Robles De La Torre, “The Importance of the Sense of Touch in Virtual and Real

Environments,” IEEE Multimed., vol. 13, no. 3, pp. 24–30, 2006.

[46] B. E. Stein, T. R. Stanford, and B. A. Rowland, “The Neural basis of Multisensory

Integration in the Midbrain: Its organisation and Maturation,” Hear. Res., vol. 258, pp. 4–

15, 2009.

[47] D. Alais and D. Burr, “Ventriloquist Effect Results from Near-Optimal Bimodal

Integration,” Curr. Biol., vol. 14, no. 3, pp. 257–262, 2004.

[48] J. I. Gold and M. N. Shadlen, “The neural basis of decision making,” Annu Rev Neurosci,

vol. 30, pp. 535–574, 2007.

[49] G. Van Belle, Statistical Rules of Thumb. Hoboken, NJ, USA: Wiley, 2011.

[50] K. Kim, D. W. Rattner, and M. A. Srinivasan, “Virtual reality-based laparoscopic surgical

training: The role of simulation fidelity in haptic feedback,” Comput. Aided Surg, no. 9(5),

pp. 227–234, 2004.

[51] J. M. Walker, A. A. Blank, P. A. Shewokis, and M. K. Omalley, “Tactile Feedback of

Object Slip Facilitates Virtual Object Manipulation,” IEEE Trans. Haptics, vol. 8, no. 4,

pp. 454–466, 2015.

[52] R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, “Simulator sickness

questionnaire: an enhanced method for quantifying simulator sickness,” Int. J. Aviat.

Psychol., vol. 3, no. 3, pp. 203–220, 1993.

[53] A. Erfanian, S. Tarng, Y. Hu, J. Plouzeau, and F. Merienne, “Force and vibrotactile

integration for 3D user interaction within virtual environments,” in 2017 IEEE Symposium

on 3D User Interfaces (3DUI), 2017, pp. 87–94.

[54] I. E. Allen and C. A. Seaman, “Likert scales and data analyses,” Qual. Prog., vol. 40, no.

7, pp. 64–65, 2007.

[55] P. Cairns, “Experimental methods in human-computer interaction,” Encyclopedia of

Human-Computer Interaction, 2nd Ed. Interaction Design Foundation, 2016.

[56] S. G. Hart, “Nasa-Task Load Index (NASA-TLX); 20 Years Later,” in Proceedings of the

Human Factors and Ergonomics Society Annual Meeting, 2006, vol. 50, no. 9, pp. 904–

908.

[57] D. C. Montgomery, Design and Analysis of Experiments, 2nd ed. John Wiley & Sons,

2008.

[58] C. Forbes, M. Evans, N. Hastings, and B. Peacock, Statistical Distributions. John Wiley &

Sons, 2011.

[59] I. Hwang, J. Seo, M. Kim, and S. Choi, “Vibrotactile perceived intensity for mobile

devices as a function of direction, amplitude, and frequency,” IEEE Trans. Haptics, vol. 6,

no. 3, pp. 352–362, 2013.

[60] S. Peltier, R. Stilla, E. Mariola, S. LaConte, X. Hu, and K. Sathian, “Activity and effective

connectivity of parietal and occipital cortical regions during haptic shape perception,”

Neuropsychologia, vol. 45, no. 3, pp. 476–483, 2007.

[61] M. Maisto, C. Pacchierotti, F. Chinello, G. Salvietti, A. De Luca, and D. Prattichizzo,

“Evaluation of wearable haptic systems for the fingers in Augmented Reality

applications,” IEEE Trans. Haptics, vol. 10, no. 4, pp. 511–522, 2017.

[62] A. D. B. Craig, T. Neurosci, and R. J. Dolan, “Decision Theory : What ‘ Should ’ the

Nervous System Do?,” Science (80-. )., vol. 318, no. October, pp. 606–610, 2007.

[63] N. Lavie and Y. Tsal, “Perceptual load as a major determinant of the locus of selection in

visual attention,” Percept. Psychophys., vol. 56, no. 2, pp. 183–197, 1994.

Appendix A

74 Appendix B

University of Calgary Ethics Approval

(Redacted)

75 (Redacted)

76 (Redacted)

77 Appendix C

Data Collection

C.1 Objective Data

The objective data logged by the VE consists of the defects encountered and the corresponding cues delivered. Any user response through the buttons on the Omni device is also logged. All events are timestamped. A sample of the raw log is included below:

78 C.2 Subjective Data

The following questionnaires have been used for evaluating cybersickness, usefulness, effectiveness, pleasure, and workload. If the participant responds with “Severe” in any category, or a total of three “Moderate” responses, the participant is evaluated to have cybersickness. The perceptual questionnaire is evaluated with continuous variation of the Likert scale bounded between 0% and 100% [54]. Pleasure is measured through responses of Q7; usefulness is the average of Q1 and Q4; effectiveness if the average of Q2 and Q5. The workload component was constructed using the NASA task load index (TLX) [56]. The workload rating is calculated by dividing each Subscale Rating is by the corresponding Relative Importance rank. The results are then summed together for the final workload rating.

79 C.2.1 Cybersickness Questionnaire

C.2.2 Perceptual Questionnaire

C.2.3 NASA Task Load Index (NASA-TLX)

Appendix D

List of Software Tools

This appendix briefly lists the software tools and source code for the implementation of the VE used in the experiments conducted for this thesis as well as statistical analyses tools used. To obtain a copy of the original source code, please refer to my MSc research supervisor, Dr. Hu.

The VE developed for the experiments conducted for this thesis was developed using the following:

 Unity 5.3.2f1: This software was used as a development environment for modeling and

scripting the VE software.

 Microsoft Visual Studio 2015: This software was used as an integrated development

environment (IDE) for developing the VE software. The C# language was used.

 MiddleVR 1.6.2.f1: This software was used to integrate the VE software with the VE

equipment at the University of Calgary Collaboration Centre.

The following tools were used to conduct statistical analyses describe in Chapter 3.

 Microsoft Excel: This tool was used to organize subjective and objective data, calculate

means, standard deviations, and standard error, as well as draw statistical diagrams.

 MATLAB: This tool was used to perform MLE and PLE on objective data and draw the

integration diagrams in Chapter 3 and Chapter 4, as well as perform the following

statistical analyses:

o Anderson-Darling normality test

o One-way repeated measures analysis of variance (ANOVA) o Post-hoc Bonferroni test for multiple comparisons