DOCSLIB.ORG

3D Visualisation of Breast Reconstruction Using Microsoft Hololens

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
3D Visualisation of Breast Reconstruction Using Microsoft Hololens UPTEC F 18058 Examensarbete 30 hp November 2018 3D visualisation of breast reconstruction using Microsoft HoloLens Amanda Norberg Elliot Rask Abstract 3D visualisation of breast reconstruction using Microsoft HoloLens Amanda Norberg Elliot Rask Teknisk- naturvetenskaplig fakultet UTH-enheten The purpose of the project is to create a Mixed Reality (MR) application for the 3D visual-isation of the result of a breast reconstruction surgery. The application is to Besöksadress: be used beforesurgery to facilitate communication between patient an surgeon Ångströmlaboratoriet Lägerhyddsvägen 1 about the expected result.To this purpose Microsoft HoloLens is used, which is a Hus 4, Plan 0 pair of Mixed Reality (MR) glassesdeveloped and manufactured by Microsoft that has a self-contained, holographic renderingcomputer. For the development of the Postadress: MR application on the Hololens, MixedRealityToolkit-Unity is used which is a Unity- Box 536 751 21 Uppsala based toolkit available. The goal of the application is thatthe user can scan a torso of a patient, render a hologram of the torso and attach to it aprefabricated breast Telefon: which possibly follows the patient’s specification. 018 – 471 30 03 Telefax: To prepare a prefabricated breast, a 3D model of the breast is first created in the 018 – 471 30 00 3D modellingsoftware Blender. It then gets its texture from a picture taken with the HoloLens camera.The picture is cropped to better fit the model and uploaded as a Hemsida: 2D texture which is thenattached to the prefabricated breast, which is imported http://www.teknat.uu.se/student into Unity. To scan objects, the HoloLens’s operating system feature Surface Observer is used. Theresulting mesh from the observer is cropped using a virtual cube, scaled, moved and rotatedby the user. The cropped mesh is then smoothed using the Humphrey’s Classes smoothingalgorithm. To fuse the smoothed mesh with the prefabricated breast model, the Unitycomponents: Colliders and Transforms are used. On a collision the breast’s transform parentis set to the mesh’s transform, making the objects transforms behave depending on eachother. The MR application has been developed and evaluated. The evaluation results show that thegoal has been achieved successfully. The project demonstrates that the Microsoft HoloLensis well suited for developing such medical applications as breast reconstructive surgery vi-sualisations. It can possibly be extended to other surgeries such as showing on a patient’sbody how the scar will look after a heart surgery, or a cesarean section. Handledare: Christian Alex Ämnesgranskare: Ping Wu Examinator: Tomas Nyberg ISSN: 1401-5757, UPTEC F18 058 Contents 1Introduction 6 1.1 Background . 7 1.2 Purposeandgoals................................. 8 1.3 Overview of the project . 8 1.4 Tasks........................................ 8 1.5 Outline....................................... 9 2MixedRealityandMicrosoftHoloLens 10 2.1 MixedReality................................... 10 2.2 MicrosoftHoloLens ................................ 11 2.2.1 Hardware . 11 2.2.2 HoloLensRS4Preview . 12 2.2.3 Development tools . 13 3Theory 13 3.1 Mesh........................................ 13 3.2 Meshsmoothingalgorithms . 14 3.2.1 Laplacian Smoothing Algorithm . 14 3.2.2 Humphrey’s Classes Smoothing Algorithm . 15 4Methodandimplementation 15 4.1 Unity........................................ 15 4.1.1 GameObject . 16 4.1.2 Component ................................ 16 4.1.3 Gestures . 17 4.1.4 User Interface . 17 4.1.5 MeshFilter MeshRenderer . 17 4.1.6 Colliders.................................. 18 4.1.7 Spatial mapping . 18 4.1.8 Image processing . 19 4.2 Creatingbreastprefabricate . 20 4.3 Navigating the application . 21 4.4 Scanning a targeted object . 22 4.5 Smoothing of the obtained object mesh . 22 4.6 Photographareaandrenderitastextureofobject. 22 4.7 Mergingscannedobjectandbreastprefabricate . 24 5Resultsanddiscussion 24 5.1 Result . 24 5.1.1 Interacting with the application . 24 3 5.1.2 Scenes . 25 5.2 Discussion . 30 5.2.1 Limitations of the concept . 30 5.2.2 Imitating the human skin with textures . 30 5.2.3 Breast prefabrication . 30 5.2.4 Thesurfaceobserver ........................... 31 5.2.5 Thecurrentspatialobserver . 31 6 Conclusion and future work 32 6.1 Future....................................... 32 4 Abbreviations 3D Three dimensional API Application Programming Interface AR Augmented Reality CPU Central Processing Unit HC algorithm Humphrey’s Classes algorithm MR Mixed Reality MRT Mixed Reality Toolkit UI User Interface UX User experience VR Virtual Reality 5 1 Introduction “Our way of visualizing the body in medicine has historically been as a two- dimensional abstraction. Now through mixed reality, we can view it in three dimensions and that is pretty transformative [1],” - Dr. Simon Kos, Chief Medical Officer, Microsoft As the advancement of technology moves forward, so does the vast applications of it within medicine. New technology and software can facilitate diagnostics, education and even guide surgeons through surgeries by the aid of holograms projected on the skin of patients [2]. Mixed Reality aims to be a seamless integration between the virtual and real world. It is a computer augmented environment where "real data are sensed and used to modify users’ interactions with computer mediated worlds beyond conventional dedicated visual displays" [3]. Holograms are 3D objects that are made of light and sound. A Mixed Reality device such as Microsoft’s HoloLens display holograms while the background is still the real world. The holograms can be created, altered and implemented in an application for a MR device. Central in the devices that offer MR is an understanding of the environment in which the user operates. The device uses sensors to perceive environmental input such as surfaces, boundaries, sound and location [4]. With the use of MR, holograms of different parts of the human body such as organs can be viewed and interacted with by physicians in a 3D scope. The object can be scaled, rotated and enhanced at the fingertips of the user. It can also be placed, or "anchored", in the physical space so that the user can walk around, closer or further away from the object. These kinds of interactions with a program is a great advancement in the field of Human Computer Interactions. It allows for a feeling of simplicity on behalf of the end user when intuitive hand motions can control and interact with the application. Figure 1.1: Venn diagram visualisation of the interplay between humans, environment and computers where Mixed Reality is in the intersection of all three [4]. 6 This thesis aims to investigate the possibility of and implement a proof of concept MR application for viewing a holographic aesthetic result of a surgery directly on the patient’s body when viewed through a MR device. More specifically to show the possible result of a breast reconstruction before the actual reconstruction surgery for a patient who has undergone a mastectomy. This by creating a hologram of a new breast as a 3D object. The 3D object will have the same texture as the skin on the chest that will be expanded into a new breast. 1.1 Background A mastectomy is the surgical procedure where the breast is removed. In a total mastectomy the full breast with tissue and nipple are removed as shown in Figure 1.2 [5]. Figure 1.2: Illustration of the removal of a breast in a total mastectomy [6]. Some patients have their breast reconstruction at the same time as their mastectomy proce- dure. However, the most recommended practise for breast reconstruction post mastectomy is to wait some months until the patient is finished with possible additional treatments like chemotherapy [7]. When reconstructing the breast, the expansion can be done in several ways: with an implant, tissue from the own body, or a combination of the two [7]. It can be difficult for the patient to make a sufficiently informed decision on whether to undergo breast reconstruction. A recent, although very small, study shows that more than half of women have made decisions about breast reconstructive surgery that did not align with their goals or preferences [8]. "As breast cancer providers, we need to talk about the pros and cons of surgery to help women make treatment choices. Shared decision-making between the surgeon and patient would be particularly useful for this decision. We need to connect pa- tients with decision aids to help them really think through what is most important to them [9]." - Clara Lee, M.D., Ohio State University Comprehensive Cancer Center 7 The HoloLens is a good tool for 3D visualisation of breast reconstruction. This leads to the purpose and goals of the project that are addressed in the following section. 1.2 Purpose and goals The purpose of this project is to develop an application in Unity for a Microsoft HoloLens. The application aims to scan a patient’s torso using the HoloLens and render a hologram using that data, to then be merged together with a prefabricated breast. The prefabricated breast should also be able to get its texture from a picture of the chest area taken using the HoloLens. 1.3 Overview of the project The idea is that the many advanced sensors in a MR device can be used to obtain sufficiently accurate scan of the human torso to fit the purpose of the application. The Microsoft HoloLens offers an API called Spatial Mapping that uses the device’s sensors to scan the surroundings and render them into a mesh. The mesh can then be processed in a program and used by the application as the basis for attaching a hologram of a breast. Development of the application based on the Hololens will be done in three steps: 1) scan a specified area of the human body and store it in 3D format using the Hololens 2) render an object (e.g., breast) to be placed on the body at a specified position; the object is scalable and sizable 3) place the rendered object on the body at a specified position in the AR environment 1.4 Tasks The main part of the project will consist of writing and altering scripts in the programming language C#.
Load more

Recommended publications
  • Samsung Announces New Windows-Based Virtual-Reality Headset at Microsoft Event 4 October 2017, by Matt Day, the Seattle Times
    Samsung announces new Windows-based virtual-reality headset at Microsoft event 4 October 2017, by Matt Day, The Seattle Times Samsung is joining Microsoft's virtual reality push, Microsoft also said that it had acquired AltspaceVR, announcing an immersive headset that pairs with a California virtual reality software startup that was Windows computers. building social and communications tools until it ran into funding problems earlier this year. The Korean electronics giant unveiled its Samsung HMD Odyssey at a Microsoft event in San ©2017 The Seattle Times Francisco recently. It will sell for $499. Distributed by Tribune Content Agency, LLC. The device joins Windows-based immersive headsets built by Lenovo, HP, Acer and Dell, and aimed for release later this year. Microsoft is among the companies seeking a slice of the emerging market for modern head-mounted devices. High-end headsets, like Facebook-owned Oculus's Rift and the HTC Vive, require powerful Windows PCs to run. Others, including the Samsung Gear VR and Google's Daydream, are aimed at the wider audience of people who use smartphones. Microsoft's vision, for now, is tied to the PC, and specifically new features in the Windows operating system designed to make it easier to build and display immersive environments. The company also has its own hardware, but that hasn't been on display recently. Microsoft's HoloLens was a trailblazer when it was unveiled in 2015. The headset, whose visor shows computer-generated images projected onto objects in the wearer's environment without obscuring the view of the real world completely, was subsequently offered for sale to developers and businesses.
    [Show full text]
  • Holographic Computer
    MECHANICAL ENGINEERING | DECEMBER 2018 | P.33 bioengineering WEARABLE Holographic Computer Microsoft’s HoloLens mixed reality headset is a breakthrough in human-computer interaction. STORY BY AGAM SHAH • ILLUSTRATION BY ZINA SAUNDERS he Kinect camera, which headset, Thyssenkrupp is equipping Microsoft released in 2010, elevator service technicians with brought unprecedented HoloLens to visualize and identify T levels of gesture and voice problems ahead of a job, and have interaction to video games. “When remote, hands-free access to technical we created Kinect for Xbox 360, and expert information when onsite. we aimed to build a device capable The company says that has helped of recognizing and understanding improve service times. Ford is also people so that computers could using HoloLens, enabling their design operate in ways that are more and engineering teams to visualize human,” said Alex Kipman, technical that projects the graphics onto a full-scale models in 3-D. They’ve fellow at Microsoft, who helped lens. As a user walks around a room converted processes that used to invent the camera. or turns his head, the position or take weeks down to days, and more Microsoft wanted to build on orientation of the graphical image is easily and securely share ideas across the success of the Kinect’s sensor altered so that it appears to the user the company and consider more technology by parlaying it into a as occupying a consistent location concepts than before. And NASA has radically new device—a mixed reality in space. provided HoloLens to astronauts on headset that would project computer- HoloLens has emerged as the go- the International Space Station as a generated images seemingly into real to device for engineers to integrate holographic instruction manual and a space.
    [Show full text]
  • Innovation in Shared Virtual Spaces for Mental Health Therapy
    Innovation in Shared Virtual Spaces for Mental Health Therapy Jake Backer Advisor: Professor Soussan Djamasbi Worcester Polytechnic Institute May 23, 2021 This report represents the work of one or more WPI undergraduate students submitted to the faculty as evidence of completion of a degree requirement. WPI routinely publishes these reports on the web without editorial or peer review. 1 Contents 1 Abstract 3 2 Introduction 4 3 Background 5 3.1 Mental Health . .5 3.2 Telehealth . .5 3.3 Augmented Reality (AR) . .6 3.4 AR Therapy . .7 3.5 Avatars . .7 4 Designing the Application 8 4.1 Infrastructure . .8 4.1.1 Networking . .9 4.1.2 Movement . 11 4.1.3 Audio Communication . 12 4.1.4 Immersive Experience . 14 4.1.5 Telemetry . 15 5 User Studies 17 5.1 Results . 17 5.2 Discussion and Future User Studies . 19 6 Contribution and Future Work 21 References 24 2 1 Abstract Rates of mental health disorders among adolescents and younger adults are on the rise with the lack of widespread access remaining a critical issue. It has been shown that teletherapy, defined as therapy delivered remotely with the use of a phone or computer system, may be a viable option to replace in-person therapy in situations where in-person therapy is not possible. Sponsored by the User Experience and Decision Making (UXDM) lab at WPI, this IQP is part of a larger project to address the need for mental health therapy in situations where patients do not have access to traditional in-person care.
    [Show full text]
  • New Realities Risks in the Virtual World 2
    Emerging Risk Report 2018 Technology New realities Risks in the virtual world 2 Lloyd’s disclaimer About the author This report has been co-produced by Lloyd's and Amelia Kallman is a leading London futurist, speaker, Amelia Kallman for general information purposes only. and author. As an innovation and technology While care has been taken in gathering the data and communicator, Amelia regularly writes, consults, and preparing the report Lloyd's does not make any speaks on the impact of new technologies on the future representations or warranties as to its accuracy or of business and our lives. She is an expert on the completeness and expressly excludes to the maximum emerging risks of The New Realities (VR-AR-MR), and extent permitted by law all those that might otherwise also specialises in the future of retail. be implied. Coming from a theatrical background, Amelia started Lloyd's accepts no responsibility or liability for any loss her tech career by chance in 2013 at a creative or damage of any nature occasioned to any person as a technology agency where she worked her way up to result of acting or refraining from acting as a result of, or become their Global Head of Innovation. She opened, in reliance on, any statement, fact, figure or expression operated and curated innovation lounges in both of opinion or belief contained in this report. This report London and Dubai, working with start-ups and corporate does not constitute advice of any kind. clients to develop connections and future-proof strategies. Today she continues to discover and bring © Lloyd’s 2018 attention to cutting-edge start-ups, regularly curating All rights reserved events for WIRED UK.
    [Show full text]
  • Microsoft Aims at Apple with High-End Pcs, 3D Software 26 October 2016
    Microsoft aims at Apple with high-end PCs, 3D software 26 October 2016 unlocking a more natural and immersive way to create on the thinnest display ever built," said Microsoft vice president Terry Myerson, The new PC has an ultra-HD 4.5K display, offering more pixels than most new high-definition televisions. It will be available in "limited quantities" for the upcoming holiday season, with more units available in 2017. The new device drew a mixed response from analysts. "Yes, this is very cool. But what's the market for a $3,000 computer you can draw on? How many people have that job?" asked Benedict Evans of the Microsoft chief executive officer Satya Nadella talks at a venture firm Andreessen Horowitz in a tweet. Microsoft news conference October 26, 2016 in New York But Avi Greengart at the research firm Current Analysis said the new Microsoft products "are Microsoft launched a new consumer offensive aimed at setting the high mark for the Windows Wednesday, unveiling a high-end computer that ecosystem." challenges the Apple iMac along with an updated Windows operating system that showcases three- dimensional content and "mixed reality." The US tech giant announced its first desktop computer, called Surface Studio, a $3,000 high- end "all-in-one" device that aims at creative professionals, a segment dominated by Apple. "We're creating a new category that transforms your desk into a creative studio," Microsoft chief executive Satya Nadella said at the unveiling in New York. With a large, 28-inch (71-centimeter) hinged touchscreen display touted as "the thinnest desktop monitor ever created," Surface Studio Microsoft Corporate VP of Devices, Panos Panay adds to the Microsoft lineup of tablet and laptop introduces Microsoft Surface Studio on October 26, 2016 devices for the premium segment.
    [Show full text]
  • Musashi Seimitsu Industry Co., Ltd
    Musashi Seimitsu Industry Co., Ltd. Auto parts manufacturer installs production line remotely during travel ban with Dynamics 365 Remote Assist Musashi Seimitsu Industry is an automotive parts manufacturer with production Customer and sales facilities in 14 countries and 35 hubs worldwide. The global brand slogan, Musashi Seimitsu Industry Co., Ltd. Power to Value, has represented its innovative manufacturing style since 1938. This (http://www.musashi.co.jp/) can-do spirit has protected corporate infrastructure amidst COVID-19. By merging the real and digital worlds using HoloLens 2 with Dynamics 365 Remote Assist, Product or Service the company launched a new production line in Mexico without a single engineer - Microsoft Dynamics 365 leaving Japan. Musashi Seimitsu is embracing the post COVID-19 new normal with Mixed Reality its swift digital transformation. - Microsoft Dynamics 365 Remote Assist - Microsoft HoloLens 2 - Microsoft 365 New product launch almost delayed due to - Microsoft Teams COVID-19 Industry Musashi Seimitsu, manufacturer of powertrains and other parts for 2- and 4-wheeled autos for more Manufacturing than 82 years, engages in every stage of the process, from development to sales. Originating in Japan, the company’s plants now reach 35 countries and global regions, including the European Union, Asia, Organization Size and the Americas. Musashi Seimitsu is a firmly established global supplier, with overseas markets Over 10,000 accounting for approximately 85 percent of sales. Country Japan In early 2020, COVID-19 began its global advance and by late March, overseas business trips had been cancelled. Things worsened as plants in each country scrambled to prevent the spread of infection and by April, most production lines were shut down.
    [Show full text]
  • Arxiv:2008.11239V1 [Cs.CV] 25 Aug 2020 Overlaying Digital Twin on the Real World
    HoloLens 2 Research Mode as a Tool for Computer Vision Research Dorin Ungureanu Federica Bogo Silvano Galliani Pooja Sama Xin Duan Casey Meekhof Jan Stuhmer¨ ∗ Thomas J. Cashman Bugra Tekin Johannes L. Schonberger¨ Pawel Olszta Marc Pollefeys Microsoft Abstract Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with inte- grated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand Figure 1. Microsoft HoloLens 2. tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer viding access to all raw image sensor streams – including vision as well as robotics and encourage contributions from depth and IR. Released together with a public repository the research community. collecting auxiliary tools and sample applications [4], Re- search Mode promoted the use of HoloLens as a powerful tool for doing research in computer vision and robotics [1]. 1. Introduction HoloLens 2 (Fig. 1), announced in 2019, brings a num- Mixed reality technologies have tremendous potential to ber of improvements with respect to the first-generation de- fundamentally transform the way we interact with our envi- vice – like a dedicated DNN core, articulated hand tracking ronment, with other people, and the physical world.
    [Show full text]
  • Evaluating the Microsoft Hololens Through an Augmented Reality Assembly Application" (2017)
    Mechanical Engineering Conference Presentations, Mechanical Engineering Papers, and Proceedings 5-5-2017 Evaluating the Microsoft oloLeH ns through an augmented reality assembly application Gabriel Evans Iowa State University, [email protected] Jack Miller Iowa State University, [email protected] Mariangely Iglesias Pena Iowa State University, [email protected] Anastacia MacAllister Iowa State University, [email protected] Eliot H. Winer Iowa State University, [email protected] Follow this and additional works at: https://lib.dr.iastate.edu/me_conf Part of the Applied Mechanics Commons, and the Manufacturing Commons Recommended Citation Evans, Gabriel; Miller, Jack; Pena, Mariangely Iglesias; MacAllister, Anastacia; and Winer, Eliot H., "Evaluating the Microsoft HoloLens through an augmented reality assembly application" (2017). Mechanical Engineering Conference Presentations, Papers, and Proceedings. 179. https://lib.dr.iastate.edu/me_conf/179 This Article is brought to you for free and open access by the Mechanical Engineering at Iowa State University Digital Repository. It has been accepted for inclusion in Mechanical Engineering Conference Presentations, Papers, and Proceedings by an authorized administrator of Iowa State University Digital Repository. For more information, please contact [email protected]. Evaluating the Microsoft oloLeH ns through an augmented reality assembly application Abstract Industry and academia have repeatedly demonstrated the transformative potential of Augmented Reality (AR) guided assembly instructions. In the past, however, computational and hardware limitations often dictated that these systems were deployed on tablets or other cumbersome devices. Often, tablets impede worker progress by diverting a user's hands and attention, forcing them to alternate between the instructions and the assembly process. Head Mounted Displays (HMDs) overcome those diversions by allowing users to view the instructions in a hands-free manner while simultaneously performing an assembly operation.
    [Show full text]
  • Innovative Learning Environments in STEM Higher Education
    SPRINGER BRIEFS IN STATISTICS Jungwoo Ryoo Kurt Winkelmann Editors Innovative Learning Environments in STEM Higher Education Opportunities, Challenges, and Looking Forward SpringerBriefs in Statistics SpringerBriefs present concise summaries of cutting-edge research and practical applications across a wide spectrum of felds. Featuring compact volumes of 50 to 125 pages, the series covers a range of content from professional to academic. Typical topics might include: • A timely report of state-of-the art analytical techniques • A bridge between new research results, as published in journal articles, and a contextual literature review • A snapshot of a hot or emerging topic • An in-depth case study or clinical example • A presentation of core concepts that students must understand in order to make independent contributions SpringerBriefs in Statistics showcase emerging theory, empirical research, and practical application in Statistics from a global author community. SpringerBriefs are characterized by fast, global electronic dissemination, standard publishing contracts, standardized manuscript preparation and formatting guidelines, and expedited production schedules. More information about this series at http://www.springer.com/series/8921 Jungwoo Ryoo • Kurt Winkelmann Editors Innovative Learning Environments in STEM Higher Education Opportunities, Challenges, and Looking Forward Editors Jungwoo Ryoo Kurt Winkelmann Pennsylvania State University Valdosta State University Altoona, PA, USA Valdosta, GA, USA This book is an open access publication.
    [Show full text]
  • Virtual & Augmented Reality
    EXCERPTED FROM THE ORIGINAL: See inside cover for details. FINANCIAL BACKING GLOBAL INTEREST SHIPPING OUT SELLING OUT INTEREST IN THE PAST READY TO BUILD A WAVE OF CONTENT ON THE WAY RETAIL VALUE HOME REDESIGN, REIMAGINED EASIER TO IMAGINE YOURSELF AT HOME Video • Jaunt The Ecosystem • NextVR Virtual Reality / Augmented Reality • VRSE • Oculus Story Studio • GoPro • IG Port Processors Games • TI Applications • Sony • Qualcomm • Ubisoft 3D Audio • STMicro Graphics • CCP Games • Nvidia • TI • Realtek • Oculus Story Studio • AMD • Wolfson • Himax • Tammeka Games • Qualcomm • Realtek • MediaTek • Pixel Titans • Intel • Capcom Augmented Reality Virtual Reality Engineering • Microsoft HoloLens • Facebook Oculus Head–mounted devices • Autodesk • Google Glass • Samsung Gear VR • Dassault Systèmes • Magic Leap • Google Cardboard • IrisVR • Atheer • HTC Vive • Visidraft • Osterhout Design • Sony PSVR • MakeVR Memory Group • Vuzix iWear (DRAM/SSD) • VR Union Claire • Micron Healthcare • Samsung • Psious • SK Hynix • zSpace • Toshiba • Conquer Mobile • 3D Systems Social • Altspace VR • High Fidelity • Podrift Commerce • Sixense {shopping} • Matterport {real estate} Display • Samsung • JDI Cameras • Himax • 360Heros • Crystal • GoPro Odyssey 3D Lenses • Nokia OZO • Wearality • Jaunt NEO • Zeiss • Matterport Pro 3D Components • Canon • Nikon Haptics • Largan • Alps Position/ Room Tracker • AAC • Hon Hai • Nidec • Pegatron • Flex • Jabil • HTC Motion Sensors • Leap Motion • InvenSense • TI • STMicro • Honeywell The Ecosystem Virtual Reality / Augmented
    [Show full text]
  • Mixed Reality
    Mixed Reality Matt Fedorovich Authorized Reseller Lead, Immersive Technology Agenda Part One 01 Remote Assist Overview Part Two 02 Device and Infrastructure Readiness Part Three 03 End User Readiness Part Four 04 Recent Stories Remote Work for the Future Remote Assistance Remote Collaboration • “See what I see” • Everyone in virtual space – AR, VR or MR • Leverage physical world for interaction • Leverage digital world for interaction • Cross-Platform • Cross-Platform • Integration into existing systems is key • Integration into existing systems is key • Solution : D365 Remote Assist • Solutions : Spatial.io & AltSpace VR Remote Assist Overview Work together from anywhere. Microsoft Dynamics 365 Dynamics 365 Remote Assist Empower technicians to collaborate more efficiently by working together from different locations with Dynamics 365 Remote Assist on HoloLens, Android, or iOS devices. Enabled by Microsoft Teams, OneDrive and Dynamics 365. Dynamics 365 Guides Operators learn during the flow of work by providing holographic instructions when and where they are needed. These instruction cards are visually tethered to the place where the work must be done, and can include images, videos, and 3D holographic models How It Works iOS & Android Device Microsoft HoloLens 2 Front Line Technician Remote Expert Technician needs help. Login to Remote Assist Receives call on their Teams Mobile or PC Application and calls expert Device Requires Device Requires Remote Assist App Remote Assist Microsoft Teams Teams License On iOS / Android License On PC or Mobile (w/ Video) Per user Remote Assist App OR On Microsoft HoloLens 2 Per Device Key Journey Points Envisioning & Planning Readiness & Support Innovation & Integration Accelerate Infrastructure Readiness Identify key components that are needed before you think about devices and end users.
    [Show full text]
  • Mixed Reality & Microsoft Hololens
    Mixed Reality & Microsoft HoloLens Laboratorio di Realtà Virtuale Manuel Pezzera – [email protected] Mixed Reality • “Mixed reality (MR) is a spectrum of immersive experiences, connecting and blending physical and digital worlds together in augmented-reality and virtual-reality applications” [cit. Microsoft] Mixed Reality Augmented and Virtual Reality • Virtual Reality: it tricks your eyes into thinking that you are somewhere else. • Augmented reality: it is an extra dimension over VR since it can show the real world around wearer with graphic overlayed. • AR can also totally immerse user just like VR by simply showing pixels everywhere with no transparency to the real world. • In virtual world, users are best advised to stay seated to avoid collision with physical objects which they cannot see in the real world. What is a hologram? • A hologram is an object like any other object in the real world, with only one difference: instead of being made of physical matter, a hologram is made entirely of light. • Holographic objects can be viewed from different angles and distances, just like physical objects. They can also interact with the environment (e.g., bouncing ball on a table) • Hologram do not offer any physical resistance when touched or pushed because they don’t have a mass. Mixed reality vs Augmented Reality • MR can be mentioned as an enhanced form of Augmented Reality (AR). In AR the visible natural environment is overlaid with a layer of digital content. • But the case is slightly different with MR, as virtual objects are integrated into and responsive to the real world. For instance, a virtual ball under your desk would be blocked from a user’s view unless he bends.
    [Show full text]