2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) Evaluation of Simulator Sickness for 360◦ Videos on an HMD Subject to Participants’ Experience with Virtual Reality Majed Elwardy* Hans-Jurgen¨ Zepernick† Yan Hu‡ Thi My Chinh Chu§ Veronica Sundstedt¶ Blekinge Institute of Technology SE-37179 Karlskrona, Sweden ABSTRACT In contrast to the large technological advancements supporting novel immersive visual and interactive applications, the understand- Virtual reality (VR) has seen tremendous advances in head-mounted ing about the level of simulator sickness and cybersickness is less displays (HMDs), optics, media quality, and other improvements developed but similar important to assure customer satisfaction. that facilitate immersive experiences. With the occurrence of new Given that viewing VEs on HMDs is a rather new option for many technologies like Cloud VR and networked VR video services, ap- consumer verticals, studies on simulator sickness that engage partic- plications such as 360◦ video streaming are becoming more popular ipants with different levels of experience with VR are also needed. within the broader consumer markets. As a result, VR content is accessible to customers with rather different levels of experiences with immersive media, i.e., never, sometimes, or often use of VR. 1.1 Related Work The question, therefore, arises to which degree simulator sickness Related recent work is typically based on the simulator sickness is induced to viewers depending on their experiences with VR on ◦ questionnaire (SSQ) proposed in [17]. In [4], a VR simulator of a HMDs. In this paper, simulator sickness is evaluated for 360 videos forestry crane for loading logs onto a truck was evaluated in terms that were shown on an HTC Vive Pro HMD to participants having of quality of experience (QoE) aspects and SSQ to reveal the effects different levels of experience with VR on HMDs. The modified ab- of latency for task completion. It was shown that the display delay solute category rating with hidden reference (M-ACR-HR) method strongly influenced sickness symptoms causing some test subjects was used in a subjective experiment for video quality assessment to discontinue the experiment. The work reported in [5] evaluated within two subsequent sessions along with a simulator sickness ques- the physical discomfort by recording the SSQ for the Oculus Rift tionnaire (SSQ). A statistical analysis of the SSQ scores is performed HMD in three experiments against the Google Cardboard and 3D to reveal the relationship between simulator sickness and partici- TV. Compared to the two low-cost devices, the Oculus Rift HMD pants’ experiences with VR regarding: (1) Individual symptoms, (2) turned out to have great potential for conducting basic research in Pairwise comparison of symptoms, and (3) Symptom clusters of this field. However, additional in-depth studies were suggested to nausea, oculomotor, disorientation, and total score. It is shown that better understand the relationship between exposure duration, ex- the simulator sickness symptoms, in general, are slightly or rarely ◦ perimental design, and simulator sickness as well as other HMDs. perceived across the different experience levels for the selected 360 In [11], the impact of video content type on users’ VR sickness and videos. The results indicate that the reported simulator sickness in- physiological response was studied engaging 26 participants with creases in the second session for participants that never used VR on visual stimuli shown on three generations of Oculus Rift HMDs, HMDs. Sufficiently long breaks between sessions should therefore Samsung Gear VR HMD, and a 2D TV screen. Findings of this be accounted for in the M-ACR-HR method to avoid that simulator research include that participants’ personal content preferences af- sickness influences quality rating. fect their sickness perception, skin conductance strongly correlates Keywords: Immersive media, 360° videos, subjective experiments, with sickness effects, and that both SSQ and also simpler question- M-ACR-HR method, simulator sickness questionnaire. naires give good results. In [15], QoE and simulator sickness of a smart-exercise-bike VR system was evaluated using the SSQ and physiological signals. It was shown that texture quality and frame 1 INTRODUCTION rate have a statistically significant impact on perceived quality but In recent years, immersive multimedia such as virtual reality (VR) not on the SSQ scores. Further, the synthetic VEs presented to the and augmented reality (AR) have seen increased applications ranging participants appear to trigger higher levels of sickness symptoms from digital games to the general consumer markets. The trend compared to studies focusing on omnidirectional videos. The levels of viewing virtual environments (VEs) on head-mounted displays of prior experience with VR appeared to also have a significant im- (HMDs) is due to large technological improvements in hardware, pact on simulator sickness with more experienced participants being processors, and software. 5G mobile networks are being developed less affected. In [21], subjective quality of omnidirectional videos to provide the high data rates needed for this type of applications with at different bit rates and resolutions was evaluated along with an ultra-low latency and stringent reliability constraints [12]. Similarly, analysis of SSQ scores and head motion behavior. It was found that the visions for future 6G mobile networks foresee to take VEs even simulator sickness increases with increased test time for the videos further to serve holographic verticals and societies [13]. shown on an HTC Vive Pro HMD and Oculus Rift HMD but can be reduced by taking sufficiently long breaks between sessions. The *e-mail: [email protected] finding that simulator sickness increases with increased time was †e-mail: [email protected] confirmed in [25] and the role of extended breaks to reduce SSQ ‡e-mail: [email protected] scores was pointed out. A similar experimental setting was studied §e-mail: [email protected] in [22] with the obtained SSQ scores suggesting different levels of ¶e-mail: [email protected] simulator sickness depending on the content. Further, it was shown that the female participants developed significantly higher simulator sickness (see also [10]). The work in [26] investigated visual and cognitive aftereffects of using HMDs and their relationship to the re- porting of sickness on the SSQ in an application-based study. In [1], 978-1-7281-6532-5/20/$31.00 ©2020 IEEE 477 DOI 10.1109/VRW50115.2020.00097 the relationship between SSQ scores and participants’ dropout rates 2 EXPERIMENTAL DESIGN due to sickness symptoms in driving simulators was studied. The SSQ data was obtained as part of a larger subjective experiment Similar as the SSQ [17] was developed from the Pensacola mo- on 360◦ video quality assessment. The stimuli were viewed on tion sickness questionnaire (MSQ) [16] to better capture symptoms an HMD by participants having different levels of experience with of simulator-induced sickness compared to motion-induced sickness, VR. The subjective experiment was broken down into a pilot study several questionnaires have been proposed in recent years to bet- with a small group of experts and a main study with participants ter evaluate immersive media-induced sickness. A virtual reality not directly involved in quality assessment of visual stimuli. As sickness questionnaire (VRSQ) was proposed in [18] based on the noted in Recommendation ITU-T P.910 [14], the aim of a pilot study responses of 24 participants that were given target selection tasks on (4-8 experts) is to obtained indicative results before conducting a VR device. The results show that the SSQ proportional score of the main study with a larger number of participants. Results of nausea-related symptoms is the lowest which motivated to consider this subjective experiments addressing the impact of participants’ only nine symptoms for the VRSQ that contribute to oculomotor and experience on 360◦ video quality assessment regarding opinion disorientation. In [23], the suitability of short, single-scale versions scores have been reported in [7]. In the following sections, the of the SSQ and presence questionnaire (PQ) for omnidirectional design of this experiment is described to the extent needed for the videos in comparison to the respective long versions was investi- understanding of the SSQ analysis. Additional details about the gated. The results show that the short questionnaires can be used to design of this subjective experiment can be found in [7]. assess strong effects such as impact of content but cannot replace the long versions when it comes to assess the impact of coding parame- 2.1 Stimuli, Software, and Equipment ters such as resolution, bit rate, and frame rate on simulator sickness ◦ and presence. In [20], the SSQ is compared with the cybersickness Four natural 360 reference video scenes of 8K resolution (2D width) with different content, complexity, and motion were selected from questionnaire (CSQ) [6], VRSQ [18], and French SSQ (FSSQ) [3] ◦ in VEs developed for VR. The results obtained in this study with 32 the VQA-ODV database [19], [2]. For each of the 8K 360 video participants and 7 different VEs presented in 9 sessions show that scenes, given in equirectangular projection format, additional refer- ence videos with 6K, 4K, 2K, and optimal resolution (OR) [29] were CSQ and VRSQ give better indicators of validity compared to SSQ ◦ and FSSQ when it comes to evaluating cybersickness. produced. A set of 360 test videos was generated by compressing the 360◦ reference videos using five different quantization parame- 1.2 Contribution and Paper Structure ters (QPs), i.e., QP=22, 27, 32, 37, and 42. In this way, 120 360◦ videos with different quality levels were obtained: (1) 20 reference In view of the above discussion, in this paper, we evaluate the impact videos with five resolutions and four scenes, (2) 100 test videos rep- of participants’ experience with VR on simulator sickness using the resenting five resolutions, five QPs, and four scenes.