RESEARCH ARTICLE HTC Vive MeVisLab integration via OpenVR for medical applications Jan Egger1,2*, Markus Gall1, JuÈrgen Wallner3, Pedro Boechat3, Alexander Hann4, Xing Li5, Xiaojun Chen5, Dieter Schmalstieg1 1 Institute of Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16c/II, Graz, Austria, 2 BioTechMed-Graz, Krenngasse 37/1, Graz, Austria, 3 Medical University of Graz, Department of Oral and Maxillofacial Surgery, Auenbruggerplatz 5/1, Graz, Austria, 4 Department of Internal Medicine I, Ulm University, Albert-Einstein-Allee 23, Ulm, Germany, 5 Shanghai Jiao Tong University, School of Mechanical Engineering, Shanghai, China a1111111111 a1111111111 * [email protected] a1111111111 a1111111111 a1111111111 Abstract Virtual Reality, an immersive technology that replicates an environment via computer-simu- lated reality, gets a lot of attention in the entertainment industry. However, VR has also great OPEN ACCESS potential in other areas, like the medical domain, Examples are intervention planning, train- ing and simulation. This is especially of use in medical operations, where an aesthetic out- Citation: Egger J, Gall M, Wallner J, Boechat P, Hann A, Li X, et al. (2017) HTC Vive MeVisLab come is important, like for facial surgeries. Alas, importing medical data into Virtual Reality integration via OpenVR for medical applications. devices is not necessarily trivial, in particular, when a direct connection to a proprietary PLoS ONE 12(3): e0173972. https://doi.org/ application is desired. Moreover, most researcher do not build their medical applications 10.1371/journal.pone.0173972 from scratch, but rather leverage platforms like MeVisLab, MITK, OsiriX or 3D Slicer. These Editor: Hans A. Kestler, University of Ulm, platforms have in common that they use libraries like ITK and VTK, and provide a conve- GERMANY nient graphical interface. However, ITK and VTK do not support Virtual Reality directly. In Received: August 26, 2016 this study, the usage of a Virtual Reality device for medical data under the MeVisLab plat- Accepted: March 1, 2017 form is presented. The OpenVR library is integrated into the MeVisLab platform, allowing a Published: March 21, 2017 direct and uncomplicated usage of the head mounted display HTC Vive inside the MeVisLab platform. Medical data coming from other MeVisLab modules can directly be connected per Copyright: © 2017 Egger et al. This is an open access article distributed under the terms of the drag-and-drop to the Virtual Reality module, rendering the data inside the HTC Vive for Creative Commons Attribution License, which immersive virtual reality inspection. permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Data is available on Introduction figshare at https://figshare.com/articles/Cranial_ Defect_Datasets/4659565. Virtual Reality (VR) places a user inside a computer-generated environment. This paradigm is Funding: The work received funding from becoming increasingly popular, due to the fact that computer graphics have progressed to a BioTechMed-Graz in Austria (https:// point where the images are often indistinguishable from the real world. The computer-gener- biotechmedgraz.at/en/, ªHardware accelerated ated images previously presented in movies, games and other media are now detached from intelligent medical imagingº), the 6th Call of the the physical surroundings and presented in new immersive head-mounted displays (HMD), Initial Funding Program from the Research & like the Oculus Rift, the PlayStation VR or the HTC Vive (Fig 1). Virtual reality not only places Technology House (F&T-Haus) at the Graz University of Technology (https://www.tugraz.at/ a user inside a computer-generated environment ± it is able to completely immerse a user in a en/, ªInteractive Planning and Reconstruction of virtual world, removing any restrictions on what a user can do or experience [1±3]. Beside Facial Defectsº, PI: Jan Egger) and was supported movies, games, and other media, the medical area has great potential for the newly VR devices, PLOS ONE | https://doi.org/10.1371/journal.pone.0173972 March 21, 2017 1 / 14 HTC Vive for medical applications by TU Graz Open Access Publishing Fund. Dr. Xiaojun Chen receives support by the Natural Science Foundation of China (www.nsfc.gov.cn, Grant No.: 81511130089) and the Foundation of Science and Technology Commission of Shanghai Municipality (Grants No.: 14441901002, 15510722200 and 16441908400). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors declare no competing interests. Fig 1. Illustration of a person using the HTC Vive and controllers (photo filtered with PRISMA 2.2.1 under an iPhone 6s running iOS 9.3.3). https://doi.org/10.1371/journal.pone.0173972.g001 because they are also now able to process and display high-resolution medical image data acquired with modern CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) scanners [4]. Examples are surgery trainers and simulators, and preoperative surgical planning [5], [6]. Others already working in the area of medical Virtual Reality are Nunnerley et al. [7], who tested the feasibility of an immersive 3D virtual reality wheelchair training tool for people with spinal cord injury. They designed a wheelchair training system that used the Oculus Rift headset and a joystick. Newbutt et al. [8] studied the usage of the Oculus Rift in autism patients. Their study explores the acceptance, willingness, sense of presence and immersion of participants with autism spectrum disorder (ASD). The examination of digital pathology slides with the virtual reality technology and the Oculus Rift has been explored by Farahani et al. [9]. They applied the Oculus Rift and a Virtual Desktop software to review lymph node cases for digital pathology. A usability comparison between HMD (Oculus Rift) and stereoscopic desk- top displays (DeepStream3D) in a VR environment with pain patients has been performed by Tong et al. [10]. Twenty chronic pain patients assessed the severity of physical discomforts by trying both displays, while watching a virtual reality pain management demo in a clinical set- ting. An interactive 3D virtual anatomy puzzle for learning and simulation has been designed and tested by Messier and collages [11]. The virtual anatomy puzzle is supposed to help users to learn the anatomy of various organs and systems by manipulating virtual 3D data. It was implemented with an Oculus Rift. A computer-based system, which can record a surgical pro- cedure with multiple depth cameras and reconstruct in three dimensions the dynamic geome- try of the actions and events that occur during the procedure, has been introduced by Cha et al. [12]. Equipped with a virtual reality headset, such as the Oculus Rift, the user was able to PLOS ONE | https://doi.org/10.1371/journal.pone.0173972 March 21, 2017 2 / 14 HTC Vive for medical applications walk around the reconstruction of the procedure room, while controlling the playback of the recorded surgical procedure with simple VCR controls (e.g., play, pause, rewind, and fast for- ward). Xu et al. [13] studied the accuracy of the Oculus Rift during cervical spine mobility measurement by designing a virtual reality environment to guide participants to perform cer- tain neck movements. Subsequently, the cervical spine kinematics was measured by both the Oculus Rift tracking system and a reference motion tracker. The Oculus Rift has been applied by Kim et al. [14] as a cost-effective tool for studying visual-vestibular interactions in self- motion perception. The vection strength has been measured in three conditions of visual compensation for head movement (1. compensated, 2. uncompensated and 3. inversely com- pensated). King [15] developed an immersive VR environment for diagnostic imaging. The environment consisted of a web server acquiring data from volumes loaded within the 3D Slicer platform [16] and forwarding them to a Unity application (https://unity3d.com/) to ren- der the scene for the Oculus Rift. However, to the best of the authors' knowledge, no work described the integration the HTC Vive into MeVisLab (http://www.mevislab.de/) platform [17] yet. We developed and implemented a new module for the medical prototyping platform MeVisLab that provides an interface via the OpenVR library to head mounted displays, enabl- ing the direct and uncomplicated usage of the HTC Vive in medical applications. Unlike the Oculus Rift, the room-scale tracking offered by the HTC Vive allows walking around virtual objects, which enables a more advanced immersion and inspection. This contribution is organized as follows: Section 2 introduces details of the methods, Sec- tion 3 presents experimental results and Section 4 concludes the paper and gives an outlook on future work. Methods In this section, the materials and methods that have been used for the integration of the HTC Vive into the MeVisLab environment via the OpenVR library are introduced. Datasets For testing and evaluating our software integration, we used multiple high-resolution CT (Computed Tomography) scans from the clinical routine. The resolution of the scans consisted of 512x512 voxels in the x- and the y-direction with an additional few hundred slices in the z- direction. The scans varied in anatomy and location of pathology, including datasets from patient skulls with cranial defects. In Fig 2, 3D visualizations of patient skulls with cranial defects on the left (left) and right side (right) are shown. The medical scans are freely available for download and usage in own research projects, however, we kindly ask users cite our work [18], [19]. All relevant data are hosted at the public repository Figshare. Please see data hosted at Figshare at the following URL: https://figshare.com/articles/Cranial_Defect_Datasets/4659565. Note that the datasets from our clinical partners have not been altered or downsampled in any way. Hence, we assess the visual quality and evaluate the frames per second (fps) when dis- playing original sized scans inside the HTC Vive.