UPTEC F 18058 Examensarbete 30 hp November 2018 3D visualisation of breast reconstruction using Microsoft HoloLens Amanda Norberg Elliot Rask Abstract 3D visualisation of breast reconstruction using Microsoft HoloLens Amanda Norberg Elliot Rask Teknisk- naturvetenskaplig fakultet UTH-enheten The purpose of the project is to create a Mixed Reality (MR) application for the 3D visual-isation of the result of a breast reconstruction surgery. The application is to Besöksadress: be used beforesurgery to facilitate communication between patient an surgeon Ångströmlaboratoriet Lägerhyddsvägen 1 about the expected result.To this purpose Microsoft HoloLens is used, which is a Hus 4, Plan 0 pair of Mixed Reality (MR) glassesdeveloped and manufactured by Microsoft that has a self-contained, holographic renderingcomputer. For the development of the Postadress: MR application on the Hololens, MixedRealityToolkit-Unity is used which is a Unity- Box 536 751 21 Uppsala based toolkit available. The goal of the application is thatthe user can scan a torso of a patient, render a hologram of the torso and attach to it aprefabricated breast Telefon: which possibly follows the patient’s specification. 018 – 471 30 03 Telefax: To prepare a prefabricated breast, a 3D model of the breast is first created in the 018 – 471 30 00 3D modellingsoftware Blender. It then gets its texture from a picture taken with the HoloLens camera.The picture is cropped to better fit the model and uploaded as a Hemsida: 2D texture which is thenattached to the prefabricated breast, which is imported http://www.teknat.uu.se/student into Unity. To scan objects, the HoloLens’s operating system feature Surface Observer is used. Theresulting mesh from the observer is cropped using a virtual cube, scaled, moved and rotatedby the user. The cropped mesh is then smoothed using the Humphrey’s Classes smoothingalgorithm. To fuse the smoothed mesh with the prefabricated breast model, the Unitycomponents: Colliders and Transforms are used. On a collision the breast’s transform parentis set to the mesh’s transform, making the objects transforms behave depending on eachother. The MR application has been developed and evaluated. The evaluation results show that thegoal has been achieved successfully. The project demonstrates that the Microsoft HoloLensis well suited for developing such medical applications as breast reconstructive surgery vi-sualisations. It can possibly be extended to other surgeries such as showing on a patient’sbody how the scar will look after a heart surgery, or a cesarean section. Handledare: Christian Alex Ämnesgranskare: Ping Wu Examinator: Tomas Nyberg ISSN: 1401-5757, UPTEC F18 058 Contents 1Introduction 6 1.1 Background . 7 1.2 Purposeandgoals................................. 8 1.3 Overview of the project . 8 1.4 Tasks........................................ 8 1.5 Outline....................................... 9 2MixedRealityandMicrosoftHoloLens 10 2.1 MixedReality................................... 10 2.2 MicrosoftHoloLens ................................ 11 2.2.1 Hardware . 11 2.2.2 HoloLensRS4Preview . 12 2.2.3 Development tools . 13 3Theory 13 3.1 Mesh........................................ 13 3.2 Meshsmoothingalgorithms . 14 3.2.1 Laplacian Smoothing Algorithm . 14 3.2.2 Humphrey’s Classes Smoothing Algorithm . 15 4Methodandimplementation 15 4.1 Unity........................................ 15 4.1.1 GameObject . 16 4.1.2 Component ................................ 16 4.1.3 Gestures . 17 4.1.4 User Interface . 17 4.1.5 MeshFilter MeshRenderer . 17 4.1.6 Colliders.................................. 18 4.1.7 Spatial mapping . 18 4.1.8 Image processing . 19 4.2 Creatingbreastprefabricate . 20 4.3 Navigating the application . 21 4.4 Scanning a targeted object . 22 4.5 Smoothing of the obtained object mesh . 22 4.6 Photographareaandrenderitastextureofobject. 22 4.7 Mergingscannedobjectandbreastprefabricate . 24 5Resultsanddiscussion 24 5.1 Result . 24 5.1.1 Interacting with the application . 24 3 5.1.2 Scenes . 25 5.2 Discussion . 30 5.2.1 Limitations of the concept . 30 5.2.2 Imitating the human skin with textures . 30 5.2.3 Breast prefabrication . 30 5.2.4 Thesurfaceobserver ........................... 31 5.2.5 Thecurrentspatialobserver . 31 6 Conclusion and future work 32 6.1 Future....................................... 32 4 Abbreviations 3D Three dimensional API Application Programming Interface AR Augmented Reality CPU Central Processing Unit HC algorithm Humphrey’s Classes algorithm MR Mixed Reality MRT Mixed Reality Toolkit UI User Interface UX User experience VR Virtual Reality 5 1 Introduction “Our way of visualizing the body in medicine has historically been as a two- dimensional abstraction. Now through mixed reality, we can view it in three dimensions and that is pretty transformative [1],” - Dr. Simon Kos, Chief Medical Officer, Microsoft As the advancement of technology moves forward, so does the vast applications of it within medicine. New technology and software can facilitate diagnostics, education and even guide surgeons through surgeries by the aid of holograms projected on the skin of patients [2]. Mixed Reality aims to be a seamless integration between the virtual and real world. It is a computer augmented environment where "real data are sensed and used to modify users’ interactions with computer mediated worlds beyond conventional dedicated visual displays" [3]. Holograms are 3D objects that are made of light and sound. A Mixed Reality device such as Microsoft’s HoloLens display holograms while the background is still the real world. The holograms can be created, altered and implemented in an application for a MR device. Central in the devices that offer MR is an understanding of the environment in which the user operates. The device uses sensors to perceive environmental input such as surfaces, boundaries, sound and location [4]. With the use of MR, holograms of different parts of the human body such as organs can be viewed and interacted with by physicians in a 3D scope. The object can be scaled, rotated and enhanced at the fingertips of the user. It can also be placed, or "anchored", in the physical space so that the user can walk around, closer or further away from the object. These kinds of interactions with a program is a great advancement in the field of Human Computer Interactions. It allows for a feeling of simplicity on behalf of the end user when intuitive hand motions can control and interact with the application. Figure 1.1: Venn diagram visualisation of the interplay between humans, environment and computers where Mixed Reality is in the intersection of all three [4]. 6 This thesis aims to investigate the possibility of and implement a proof of concept MR application for viewing a holographic aesthetic result of a surgery directly on the patient’s body when viewed through a MR device. More specifically to show the possible result of a breast reconstruction before the actual reconstruction surgery for a patient who has undergone a mastectomy. This by creating a hologram of a new breast as a 3D object. The 3D object will have the same texture as the skin on the chest that will be expanded into a new breast. 1.1 Background A mastectomy is the surgical procedure where the breast is removed. In a total mastectomy the full breast with tissue and nipple are removed as shown in Figure 1.2 [5]. Figure 1.2: Illustration of the removal of a breast in a total mastectomy [6]. Some patients have their breast reconstruction at the same time as their mastectomy proce- dure. However, the most recommended practise for breast reconstruction post mastectomy is to wait some months until the patient is finished with possible additional treatments like chemotherapy [7]. When reconstructing the breast, the expansion can be done in several ways: with an implant, tissue from the own body, or a combination of the two [7]. It can be difficult for the patient to make a sufficiently informed decision on whether to undergo breast reconstruction. A recent, although very small, study shows that more than half of women have made decisions about breast reconstructive surgery that did not align with their goals or preferences [8]. "As breast cancer providers, we need to talk about the pros and cons of surgery to help women make treatment choices. Shared decision-making between the surgeon and patient would be particularly useful for this decision. We need to connect pa- tients with decision aids to help them really think through what is most important to them [9]." - Clara Lee, M.D., Ohio State University Comprehensive Cancer Center 7 The HoloLens is a good tool for 3D visualisation of breast reconstruction. This leads to the purpose and goals of the project that are addressed in the following section. 1.2 Purpose and goals The purpose of this project is to develop an application in Unity for a Microsoft HoloLens. The application aims to scan a patient’s torso using the HoloLens and render a hologram using that data, to then be merged together with a prefabricated breast. The prefabricated breast should also be able to get its texture from a picture of the chest area taken using the HoloLens. 1.3 Overview of the project The idea is that the many advanced sensors in a MR device can be used to obtain sufficiently accurate scan of the human torso to fit the purpose of the application. The Microsoft HoloLens offers an API called Spatial Mapping that uses the device’s sensors to scan the surroundings and render them into a mesh. The mesh can then be processed in a program and used by the application as the basis for attaching a hologram of a breast. Development of the application based on the Hololens will be done in three steps: 1) scan a specified area of the human body and store it in 3D format using the Hololens 2) render an object (e.g., breast) to be placed on the body at a specified position; the object is scalable and sizable 3) place the rendered object on the body at a specified position in the AR environment 1.4 Tasks The main part of the project will consist of writing and altering scripts in the programming language C#.