DOCSLIB.ORG

Developing a Kinect Based Holoportation System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Developing a Kinect Based Holoportation System Wright State University CORE Scholar Browse all Theses and Dissertations Theses and Dissertations 2018 Developing a Kinect based Holoportation System Soumya Chiday Wright State University Follow this and additional works at: https://corescholar.libraries.wright.edu/etd_all Part of the Computer Engineering Commons, and the Computer Sciences Commons Repository Citation Chiday, Soumya, "Developing a Kinect based Holoportation System" (2018). Browse all Theses and Dissertations. 2231. https://corescholar.libraries.wright.edu/etd_all/2231 This Thesis is brought to you for free and open access by the Theses and Dissertations at CORE Scholar. It has been accepted for inclusion in Browse all Theses and Dissertations by an authorized administrator of CORE Scholar. For more information, please contact [email protected]. DEVELOPING A KINECT BASED HOLOPORTATION SYSTEM A Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science by SOUMYA CHIDAY B.Tech., Jawaharlal Nehru Technological University, India, 2016 2018 Wright State University WRIGHT STATE UNIVERSITY GRADUATE SCHOOL December 13, 2018 I HEREBY RECOMMEND THAT THE THESIS PREPARED UNDER MY SUPERVISION BY Soumya Chiday ENTITLED Developing A Kinect Based Holoportation System BE ACCEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science. ________________________________ Yong Pei, Ph.D. Thesis Director ________________________________ Mateen M. Rizki, Ph.D. Chair, Department of Computer Science Committee on Final Examination ________________________________ Yong Pei, Ph.D. ________________________________ Mateen M. Rizki, Ph.D. (Co-Advisor) ________________________________ Paul Bender, Ph.D. ________________________________ Barry Milligan, Ph.D. Interim Dean of the Graduate School ABSTRACT Chiday, Soumya. M.S. Department of Computer Science and Engineering, Wright State University, 2018. Developing a Kinect based Holoportation System. Holographic communication and distributed collaboration offer great potential for empathic computing to help remove the cost, distance, language and expertise barriers in many social and economic activities. Recent advances in AR-enhanced communication as evident by Microsoft Holoportation technology demonstrate the progress toward fully immersive collaborations distributed and remotely. Current holoportation system requires the use of extensive camera-arrays and powerful server system due to the computation demand and sensory needs to capture and reconstruct the subject of interests. Thus, they suffer in mobility and applicability in real-world scenarios. In this thesis, we present an ultra-portable holoportation system design that requires only modest hardware supports that include a Kinect v2 sensor and PC/Laptop in addition to the HoloLens devices. Using Unity 3D engine on Universal Windows Platform and HoloToolKit sharing service, we have implemented the software for holoportating a 3D holograms of a person from a remote location to a local HoloLens device over the Internet. iii This low-cost and ultra-portable holoportation system could be potentially used in various industries for skill training, real-time assistance from subject expert. iv TABLE OF CONTENTS 1. INTRODUCTION ....................................................................................................................... 1 1.1. Telepresence ......................................................................................................................... 1 1.2. Augmented Reality ............................................................................................................... 1 1.3. Potential Applications in Different Fields ............................................................................ 2 2. MICROSOFT HOLOLENS ......................................................................................................... 4 2.1. What is HoloLens? ................................................................................................................ 4 2.2. Specifications ........................................................................................................................ 5 2.2.1. Cameras and Sensors: .................................................................................................... 6 2.2.2. Voice Commands: .......................................................................................................... 7 2.2.3. Hand Gestures: ............................................................................................................... 7 2.2.4. Spatial Mapping: ............................................................................................................ 8 2.3. Limitations ............................................................................................................................ 9 2.4. HoloLens and Kinect Collaboration ..................................................................................... 9 3. MICROSOFT KINECT ............................................................................................................. 11 3.1. What is Kinect? ................................................................................................................... 11 3.2. Kinect SDK ......................................................................................................................... 12 3.3. Cameras and Data Streams ................................................................................................. 12 3.3.1. IR Projector and Depth Camera: .................................................................................. 12 3.3.2. RGB Camera: ............................................................................................................... 13 3.3.3 Skeletal Data: ................................................................................................................ 14 3.3.4. Construct 3D holographic video: ................................................................................. 14 4. HOLOPORTATION .................................................................................................................. 15 4.1. What is Holoportation? ....................................................................................................... 16 4.1.1. Cave Camera Set-Up:................................................................................................... 17 4.2. Mobile Holoportation .......................................................................................................... 18 4.3. Limitations .......................................................................................................................... 19 5. IMPLEMENTATION OF KINECT-BASED HOLOPORTATION PROTOTYPE SYSTEM 20 5.1. Unity 3D Engine ................................................................................................................. 20 v 5.2. HoloToolKit ........................................................................................................................ 22 5.3. HoloToolKit Sharing Service ............................................................................................. 23 5.4. Components of Hologram ................................................................................................... 24 5.4.1. Mesh:............................................................................................................................ 24 5.4.2. Texture: ........................................................................................................................ 24 5.5. Server Design ...................................................................................................................... 25 5.6. Client Design ...................................................................................................................... 27 5.7. Holographic Emulation ....................................................................................................... 28 5.8. Building an UWP Application ............................................................................................ 30 5.8.1. Player Settings in Unity: .............................................................................................. 30 5.8.2. Build Settings in VS2017: ........................................................................................... 30 5.8.3. Deploy: ......................................................................................................................... 31 5.9. Implementation Results ...................................................................................................... 31 5.10. Rear End of Hologram ...................................................................................................... 32 5.10.1. Scanned 3D Model for Rear End: .............................................................................. 34 6. CONCLUSION AND FUTURE WORK .................................................................................. 35 REFERENCES .............................................................................................................................. 36 vi LIST OF FIGURES Figure Page Figure 1. An expert helping customer remotely ...…………………………………. 3 Figure 2.1. Components of HoloLens …. ……………………………...………...... 5 Figure 2.2. Bloom Gesture give this Home Page on HoloLens … …...……...……. 8 Figure 2.3 Spatial Mapping of Living Room ….…………………………………... 9 Figure 3.1. Components of Kinect ….……………………………… …………….
Load more

Recommended publications
  • Samsung Announces New Windows-Based Virtual-Reality Headset at Microsoft Event 4 October 2017, by Matt Day, the Seattle Times
    Samsung announces new Windows-based virtual-reality headset at Microsoft event 4 October 2017, by Matt Day, The Seattle Times Samsung is joining Microsoft's virtual reality push, Microsoft also said that it had acquired AltspaceVR, announcing an immersive headset that pairs with a California virtual reality software startup that was Windows computers. building social and communications tools until it ran into funding problems earlier this year. The Korean electronics giant unveiled its Samsung HMD Odyssey at a Microsoft event in San ©2017 The Seattle Times Francisco recently. It will sell for $499. Distributed by Tribune Content Agency, LLC. The device joins Windows-based immersive headsets built by Lenovo, HP, Acer and Dell, and aimed for release later this year. Microsoft is among the companies seeking a slice of the emerging market for modern head-mounted devices. High-end headsets, like Facebook-owned Oculus's Rift and the HTC Vive, require powerful Windows PCs to run. Others, including the Samsung Gear VR and Google's Daydream, are aimed at the wider audience of people who use smartphones. Microsoft's vision, for now, is tied to the PC, and specifically new features in the Windows operating system designed to make it easier to build and display immersive environments. The company also has its own hardware, but that hasn't been on display recently. Microsoft's HoloLens was a trailblazer when it was unveiled in 2015. The headset, whose visor shows computer-generated images projected onto objects in the wearer's environment without obscuring the view of the real world completely, was subsequently offered for sale to developers and businesses.
    [Show full text]
  • Holographic Computer
    MECHANICAL ENGINEERING | DECEMBER 2018 | P.33 bioengineering WEARABLE Holographic Computer Microsoft’s HoloLens mixed reality headset is a breakthrough in human-computer interaction. STORY BY AGAM SHAH • ILLUSTRATION BY ZINA SAUNDERS he Kinect camera, which headset, Thyssenkrupp is equipping Microsoft released in 2010, elevator service technicians with brought unprecedented HoloLens to visualize and identify T levels of gesture and voice problems ahead of a job, and have interaction to video games. “When remote, hands-free access to technical we created Kinect for Xbox 360, and expert information when onsite. we aimed to build a device capable The company says that has helped of recognizing and understanding improve service times. Ford is also people so that computers could using HoloLens, enabling their design operate in ways that are more and engineering teams to visualize human,” said Alex Kipman, technical that projects the graphics onto a full-scale models in 3-D. They’ve fellow at Microsoft, who helped lens. As a user walks around a room converted processes that used to invent the camera. or turns his head, the position or take weeks down to days, and more Microsoft wanted to build on orientation of the graphical image is easily and securely share ideas across the success of the Kinect’s sensor altered so that it appears to the user the company and consider more technology by parlaying it into a as occupying a consistent location concepts than before. And NASA has radically new device—a mixed reality in space. provided HoloLens to astronauts on headset that would project computer- HoloLens has emerged as the go- the International Space Station as a generated images seemingly into real to device for engineers to integrate holographic instruction manual and a space.
    [Show full text]
  • Innovation in Shared Virtual Spaces for Mental Health Therapy
    Innovation in Shared Virtual Spaces for Mental Health Therapy Jake Backer Advisor: Professor Soussan Djamasbi Worcester Polytechnic Institute May 23, 2021 This report represents the work of one or more WPI undergraduate students submitted to the faculty as evidence of completion of a degree requirement. WPI routinely publishes these reports on the web without editorial or peer review. 1 Contents 1 Abstract 3 2 Introduction 4 3 Background 5 3.1 Mental Health . .5 3.2 Telehealth . .5 3.3 Augmented Reality (AR) . .6 3.4 AR Therapy . .7 3.5 Avatars . .7 4 Designing the Application 8 4.1 Infrastructure . .8 4.1.1 Networking . .9 4.1.2 Movement . 11 4.1.3 Audio Communication . 12 4.1.4 Immersive Experience . 14 4.1.5 Telemetry . 15 5 User Studies 17 5.1 Results . 17 5.2 Discussion and Future User Studies . 19 6 Contribution and Future Work 21 References 24 2 1 Abstract Rates of mental health disorders among adolescents and younger adults are on the rise with the lack of widespread access remaining a critical issue. It has been shown that teletherapy, defined as therapy delivered remotely with the use of a phone or computer system, may be a viable option to replace in-person therapy in situations where in-person therapy is not possible. Sponsored by the User Experience and Decision Making (UXDM) lab at WPI, this IQP is part of a larger project to address the need for mental health therapy in situations where patients do not have access to traditional in-person care.
    [Show full text]
  • Windows 10 Enterprise E3 in CSP - Windows Deployment | Microsoft Docs
    07/01/2020 Windows 10 Enterprise E3 in CSP - Windows Deployment | Microsoft Docs Windows 10 Enterprise E3 in CSP 08/23/2017 • 16 minutes to read • +5 In this article Compare Windows 10 Pro and Enterprise editions Deployment of Windows 10 Enterprise E3 licenses Deploy Windows 10 Enterprise features Related topics Windows 10 Enterprise E3 launched in the Cloud Solution Provider (CSP) channel on September 1, 2016. Windows 10 Enterprise E3 in CSP is a new offering that delivers, by subscription, exclusive features reserved for Windows 10 Enterprise edition. This offering is available through the Cloud Solution Provider (CSP) channel via the Partner Center as an online service. Windows 10 Enterprise E3 in CSP provides a flexible, per- user subscription for small- and medium-sized organizations (from one to hundreds of users). To take advantage of this offering, you must have the following: Windows 10 Pro, version 1607 (Windows 10 Anniversary Update) or later, installed and activated, on the devices to be upgraded Azure Active Directory (Azure AD) available for identity management Starting with Windows 10, version 1607 (Windows 10 Anniversary Update), you can move from Windows 10 Pro to Windows 10 Enterprise more easily than ever before— no keys and no reboots. After one of your users enters the Azure AD credentials associated with a Windows 10 Enterprise E3 license, the operating system turns from Windows 10 Pro to Windows 10 Enterprise and all the appropriate Windows 10 Enterprise features are unlocked. When a subscription license expires or is transferred to another user, the Windows 10 Enterprise device seamlessly steps back down to Windows 10 Pro.
    [Show full text]
  • New Realities Risks in the Virtual World 2
    Emerging Risk Report 2018 Technology New realities Risks in the virtual world 2 Lloyd’s disclaimer About the author This report has been co-produced by Lloyd's and Amelia Kallman is a leading London futurist, speaker, Amelia Kallman for general information purposes only. and author. As an innovation and technology While care has been taken in gathering the data and communicator, Amelia regularly writes, consults, and preparing the report Lloyd's does not make any speaks on the impact of new technologies on the future representations or warranties as to its accuracy or of business and our lives. She is an expert on the completeness and expressly excludes to the maximum emerging risks of The New Realities (VR-AR-MR), and extent permitted by law all those that might otherwise also specialises in the future of retail. be implied. Coming from a theatrical background, Amelia started Lloyd's accepts no responsibility or liability for any loss her tech career by chance in 2013 at a creative or damage of any nature occasioned to any person as a technology agency where she worked her way up to result of acting or refraining from acting as a result of, or become their Global Head of Innovation. She opened, in reliance on, any statement, fact, figure or expression operated and curated innovation lounges in both of opinion or belief contained in this report. This report London and Dubai, working with start-ups and corporate does not constitute advice of any kind. clients to develop connections and future-proof strategies. Today she continues to discover and bring © Lloyd’s 2018 attention to cutting-edge start-ups, regularly curating All rights reserved events for WIRED UK.
    [Show full text]
  • Surface Hub 2S Admin Guide
    Surface Hub 2S Admin Guide Surface Hub 2S coming soon; Pre-release products shown; products and features subject to regulatory certification/approval, may change, and may vary by country/region. Surface Hub 2S has not yet been authorized under U.S. Federal Communications Commission (FCC) rules; actual sale and delivery is contingent on compliance with applicable FCC requirements. This documentation is an early release of the final documentation, which may be changed prior to final commercial release and is confidential and proprietary information of Microsoft Corporation. This document is provided for informational purposes only and Microsoft makes no warranties, either express or implied, in this document. © 2019. Microsoft Corporation. All rights reserved Introduction .................................................................................................................................................. 1 Welcome to Surface Hub 2S ......................................................................................................................... 1 New User Experience and Features ........................................................................................................................ 1 Microsoft Teams ..................................................................................................................................................... 1 New form factor and hardware changes ................................................................................................................ 2 Surface
    [Show full text]
  • Study on the Effects of New Information Technologies on the Abuse and Exploitation of Children
    Study on the Effects of New Information Technologies on the Abuse and Exploitation of Children on the Technologies of New Information Study on the Effects Study on the Effects of New Information Technologies on the Abuse and Exploitation of Children UNITED NATIONS OFFICE ON DRUGS AND CRIME Vienna Study on the Effects of New Information Technologies on the Abuse and Exploitation of Children UNITED NATIONS New York, 2015 © United Nations, May 2015. All rights reserved, worldwide. This report has not been formally edited and remains subject to editorial changes. The contents of this report do not necessarily reflect the views or policies of UNODC or contributory organizations and neither do they imply any endorsement. The designations employed and the presentation of material in this publication do not imply the expression of any opinion whatsoever on the part of the Secretariat of the United Nations concerning the legal status of any country, territory, city or area, or of its authorities, or concerning the delimitation of its frontiers or boundaries. Information on uniform resource locators and links to Internet sites contained in the present publication are provided for the convenience of the reader and are correct at the time of issue. The United Nations takes no responsibility for the continued accuracy of that information or for the content of any external website. Publishing production: English, Publishing and Library Section, United Nations Office at Vienna. Acknowledgements This report was prepared pursuant to ECOSOC resolution 2011/33 on Prevention, protection and international cooperation against the use of new information technologies to abuse and/or exploit children by Conference Support Section, Organized Crime Branch, Division for Treaty Affairs, UNODC, under the supervision of John Sandage (former Director, Division for Treaty Affairs), Sara Greenblatt and Loide Lungameni (former and current Chief, Organized Crime Branch, respectively), and Gillian Murray (former Chief, Conference Support Section).
    [Show full text]
  • Security and Compliance in Microsoft Teams - Microsoft Teams | Microsoft Docs
    3/26/2020 Overview of security and compliance in Microsoft Teams - Microsoft Teams | Microsoft Docs Security and compliance in Microsoft Teams 02/29/2020 • 9 minutes to read • +23 • Applies to: Microsoft Teams In this article Security Compliance Information Protection Architecture Licensing Location of data in Teams Compliance standards Related topics ） Important As a customer of Office 365, you own and control your data. Microsoft does not use your data for anything other than providing you with the service that you have subscribed to. As a service provider, we do not scan your email, documents, or teams for advertising or for purposes that are not service-related. Microsoft doesn’t have access to uploaded content. Like OneDrive for Business and SharePoint Online, customer data stays within the tenant. You can check out more about our trust and security related information at the Microsoft Trust Center. Teams follows the same guidance and principles as the Microsoft Trust Center. Microsoft Teams is built on the Office 365 hyper-scale, enterprise-grade cloud, delivering the advanced security and compliance capabilities our customers expect. For more information on planning for security in O365, please review our O365 content. The O365 security roadmap is a good place to start. For more information on planning for compliance in O365, you can start with the plan for security and compliance article. This article will provide further information about Teams-specific security and compliance. You should review these Microsoft Mechanics videos about security and compliance: Microsoft Teams Essentials for IT: Security and Compliance (12:42 min) Microsoft Teams Controls for Security and Compliance (10:54 min) https://docs.microsoft.com/en-us/microsoftteams/security-compliance-overview 1/8 3/26/2020 Overview of security and compliance in Microsoft Teams - Microsoft Teams | Microsoft Docs Security Teams enforces team-wide and organization-wide two-factor authentication, single sign-on through Active Directory, and encryption of data in transit and at rest.
    [Show full text]
  • EPIC-Amicus-US V. Miller
    No. 18-5578 UNITED STATES COURT OF APPEALS FOR THE SIXTH CIRCUIT UNITED STATES OF AMERICA Plaintiff-Appellee, vs. WILLIAM J. MILLER, Defendant-Appellant. On Appeal from tHe United States District Court for the Eastern District of Kentucky Case No. 2:16-cr-00047-1 The Hon. David L. Bunning BRIEF OF AMICUS CURIAE ELECTRONIC PRIVACY INFORMATION CENTER (EPIC) IN SUPPORT OF APPELLANT Marc Rotenberg Counsel of Record Alan Butler Electronic Privacy Information Center 1718 Connecticut Avenue, N.W. Suite 200 WasHington, DC 20009 (202) 483-1140 October 17, 2018 UNITED STATES COURT OF APPEALS FOR THE SIXTH CIRCUIT Disclosure of Corporate Affiliations and Financial Interest Sixth Circuit Case Number: 18-5578 Case Name: United States v. Miller Name of counsel: Alan Butler Pursuant to 6th Cir. R. 26.1, Electronic Privacy Information Center Name of Party makes the following disclosure: 1. Is said party a subsidiary or affiliate of a publicly owned corporation? If Yes, list below the identity of the parent corporation or affiliate and the relationship between it and the named party: No. 2. Is there a publicly owned corporation, not a party to the appeal, that has a financial interest in the outcome? If yes, list the identity of such corporation and the nature of the financial interest: No. CERTIFICATE OF SERVICE I certify that on ____________October_______ 17,_______ 2018___________ the foregoing document was served on all parties or their counsel of record through the CM/ECF system if they are registered users or, if they are not, by placing a true and correct copy in the United States mail, postage prepaid, to their address of record.
    [Show full text]
  • Practical Use of Approximate Hash Based Matching in Digital Investigations
    Digital Investigation 11 (2014) S18–S26 Contents lists available at ScienceDirect Digital Investigation journal homepage: www.elsevier.com/locate/diin Practical use of Approximate Hash Based Matching in digital investigations Petter Christian Bjelland*, Katrin Franke, André Årnes 1 Norwegian Information Security Laboratory (NISlab), Gjøvik University College, Norway abstract Keywords: Approximate Hash Based Matching (AHBM), also known as Fuzzy Hashing, is used to Digital forensics identify complex and unstructured data that has a certain amount of byte-level similarity. Approximate Matching Common use cases include the identification of updated versions of documents and Evidence analysis fragments recovered from memory or deleted files. Though several algorithms exist, there Data discovery has not yet been an extensive focus on its practical use in digital investigations. The paper Malware forensics addresses the research question: How can AHBM be applied in digital investigations? It fo- cuses on common scenarios in which AHBM can be applied, as well as the potential sig- nificance of its results. First, an assessment of AHBM for digital investigations with respect to existing algorithms and requirements for efficiency and precision is given. Then follows a description of scenarios in which it can be applied. The paper presents three modes of operation for Approximate Matching, namely searching, streaming and clustering. Each of the modes are tested in practical experiments. The results show that AHBM has great potential for helping investigators discover information based on data similarity. Three open source tools were implemented during the research leading up to this paper: Autopsy AHBM enables AHBM in an existing digital investigation framework, sddiff helps under- standing AHBM results through visualization, and makecluster improves analysis of graphs generated from large datasets by storing each disjunct cluster separately.
    [Show full text]
  • Microsoft Aims at Apple with High-End Pcs, 3D Software 26 October 2016
    Microsoft aims at Apple with high-end PCs, 3D software 26 October 2016 unlocking a more natural and immersive way to create on the thinnest display ever built," said Microsoft vice president Terry Myerson, The new PC has an ultra-HD 4.5K display, offering more pixels than most new high-definition televisions. It will be available in "limited quantities" for the upcoming holiday season, with more units available in 2017. The new device drew a mixed response from analysts. "Yes, this is very cool. But what's the market for a $3,000 computer you can draw on? How many people have that job?" asked Benedict Evans of the Microsoft chief executive officer Satya Nadella talks at a venture firm Andreessen Horowitz in a tweet. Microsoft news conference October 26, 2016 in New York But Avi Greengart at the research firm Current Analysis said the new Microsoft products "are Microsoft launched a new consumer offensive aimed at setting the high mark for the Windows Wednesday, unveiling a high-end computer that ecosystem." challenges the Apple iMac along with an updated Windows operating system that showcases three- dimensional content and "mixed reality." The US tech giant announced its first desktop computer, called Surface Studio, a $3,000 high- end "all-in-one" device that aims at creative professionals, a segment dominated by Apple. "We're creating a new category that transforms your desk into a creative studio," Microsoft chief executive Satya Nadella said at the unveiling in New York. With a large, 28-inch (71-centimeter) hinged touchscreen display touted as "the thinnest desktop monitor ever created," Surface Studio Microsoft Corporate VP of Devices, Panos Panay adds to the Microsoft lineup of tablet and laptop introduces Microsoft Surface Studio on October 26, 2016 devices for the premium segment.
    [Show full text]
  • Azure Kinect Body Tracking Sdk Reference
    Azure Kinect Body Tracking Sdk Reference Curatorial Mic unburden vivaciously and ineloquently, she imbody her planometer feminizes digitally. Participating Adam intermediates some Ealing and squegs his mooters so darkly! Right-hand and flutiest Zebulon slabber her sink menaced roomily or sightsee thriftily, is Sheff turdine? It is a meter away team commit to the kinect body tracker is not putting off Azure Kinect body tracking currently uses a DNN model to impair the skeleton the. Azure Kinect Masterclass Angles LightBuzz. Slightly off before we actually built its own priorities, or ideas are multiple example applications and you just slightly off before it also viewed these references. Net framework development, but i wanted with. My money when depth. Press with a reference documentation to refer to develop their imaginations come to determine whether we try again later one without my proyect there are categorized as sensor? Customers are tracked joints and review few extra adapters in another tab for tracking kinect dk has assigned different container format. MicrosoftAzureKinectBodyTracking 101 NuGet Gallery. The reference pages and kinect sdk is based on how developers stopped making sure that were used on your computer. Eddie using azure kinect or. According to sow body index frames coming from free body tracking SDK More. See details on hardware recommendations in Microsoft's Azure Kinect documentation. For more information visit the official documentation of Azure Kinect DK. Add C wrapper for Azure Kinect Body Tracking SDK Azure. Kinect camera to play them with so. If the Microsoft Azure Kinect Sensor SDK provides a reference in Python it wound be.
    [Show full text]