DOCSLIB.ORG

Visualization and Validation of Safety Parameters for Industrial Robots Using the Hololens 2

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Visualization and Validation of Safety Parameters for Industrial Robots Using the Hololens 2 Linköping University | Design Master thesis, 120 credits| Design Spring 2020| ISRN-number: LiU-ITN-TEK-A--20/014--SE Stefan Nikolov Supervisor: Ahmet Börütecene Company Supervisor: Elina Lindqvist Examiner: Jonas Löwgren Visualization and Validation of Safety Parameters for Industrial Robots Using the HoloLens 2 1 Abstract This thesis focuses on a technical and conceptual exploration of interactions that can help engineers configure sensitive, precise data for industrial ABB robots using holographic technology. The purpose is to improve the precision of the process and speed up the commissioning of robots for clients. The author presents an analysis of various approaches to using holograms for industrial purposes which were trialed and tested using a holographic display headset called the HoloLens 2. Through a workflow of sketching, concept rendering, remote meetings with employees of the robotics company ABB and technical implementation of all ideas on the Unity 3D Engine, the author was able to deliver and present a complete demo of an application for the HoloLens 2 that allows an engineer to superimpose a robot hologram on top of a real world robot of the same model and verify that the robot passes all safety requirements necessary for it to be commissioned for shipping. This demo was showed off to the Design Master class of 2019 at Linkoping University, colleagues at ABB who don't have experience with verifying safe zone parameters, software engineers who actively work with robot safety parameters and designers who create the user interface for updated iterations of the programs currently used to verify and commission robots. 2 Table of Contents 1. Introduction .................................................................................................................................. 5 1.1 General information on communicating with ABB-developed robots ................................... 5 1.2 A primer on safety zones ...................................................................................................... 5 1.3 Quick review of how robots are (presently) verified and commissioned ............................... 7 1.4 Brief introduction to the HoloLens 2 platform ...................................................................... 8 1.5 Unique features of the HoloLens 2 ..................................................................................... 10 1.6 Effects of the global coronavirus pandemic on the project ................................................. 11 2. Research Question ....................................................................................................................... 12 2.1 Company objective ............................................................................................................ 12 2.2 Determining the academic contribution of this project ...................................................... 13 2.3 Understanding what a HoloLens 2 “assistant” to RobotStudio should be capable of ........... 15 2.4 Delimitations (or lack thereof)............................................................................................ 16 3. Background Research .................................................................................................................. 17 3.1 Advantages and disadvantages of mixed reality over standard methods ............................ 17 3.2 Image tracking in AR - research and conclusions................................................................. 17 3.3 Visualizing safety zones around a robot.............................................................................. 19 3.4 Using the HoloLens 2 to enhance worker safety on-site ..................................................... 20 3.5 Exploring tactile communication beyond the HoloLens ...................................................... 21 4. Workflow and Project Focus ........................................................................................................ 23 4.1 General outline of the work process ................................................................................... 23 4.2 Ideation process ................................................................................................................. 24 4.3 Preliminary questions ........................................................................................................ 26 4.4 Collecting feedback and narrowing the direction of the prototype ..................................... 27 4.5 Confirming the viability of a prototype using think-aloud and cognitive walkthrough methods .................................................................................................................................. 28 4.6 Merits of coding and testing all concept interactions ......................................................... 29 5. Setup and Interaction Methods ................................................................................................... 31 5.1 Technical landscape ........................................................................................................... 31 5.2 Technical setup and early demo features ........................................................................... 32 5.3 Hand tracking, raycasting from the palm and hand-attached interface ............................... 33 5.4 Eye gaze tracking ............................................................................................................... 34 5.5 Eye gaze selection .............................................................................................................. 35 3 5.6 Speech recognition ............................................................................................................ 35 5.7 Spatial Awareness .............................................................................................................. 36 6. Prototyping ................................................................................................................................. 37 6.1 User alias, creating, saving and loading holographic environments ..................................... 37 6.2 "Jogging" a robotic hologram using hand ray-casting.......................................................... 37 6.3 Interacting with menus and their location in 3D space ....................................................... 39 6.4 Palm-activated menus and hand-attached gizmos .............................................................. 40 6.5 Near-menus, follow distance & orientation of UI elements that track the user ................... 41 6.6 Horizontal vs vertical spacing of UI elements ...................................................................... 42 6.7 Visual cues to verify safety zones ....................................................................................... 43 6.8 Collision logs and comments .............................................................................................. 44 6.9 Functionality of the end product ........................................................................................ 45 7. Validation - Menus ...................................................................................................................... 47 7.1 Field of view real estate - opportunities and limitations ..................................................... 47 7.2 Raycast limitations and adaptations to collision size ........................................................... 48 7.3 Summoning, interacting with menus and attaching them to points in the environment ..... 49 7.4 Constellation-type menus .................................................................................................. 51 7.5 Multi-layered hand-menu panels ....................................................................................... 51 7.6 Setup menus ...................................................................................................................... 53 8. Validation – Robot Interactions.................................................................................................... 55 8.1 Manipulating robot components ........................................................................................ 55 8.2 Selecting joint properties ................................................................................................... 56 8.3 Sliders ................................................................................................................................ 57 8.4 Modifying imported safety zones ....................................................................................... 58 8.5 Selecting specific points along a safety zone and attaching comments ............................... 59 8.6 Detecting and logging speed and boundary violations ........................................................ 60 8.7 UI with a follow behavior ................................................................................................... 61 8.8 Tooltips and the “help me” button ..................................................................................... 62 8.9 Simulating RobotStudio paths ............................................................................................ 62 9. Conclusion ................................................................................................................................... 63 9.1 A user-based perspective on the delivered prototype ........................................................ 63 9.2 Lessons learned ................................................................................................................
Load more

Recommended publications
  • Innovation in Shared Virtual Spaces for Mental Health Therapy
    Innovation in Shared Virtual Spaces for Mental Health Therapy Jake Backer Advisor: Professor Soussan Djamasbi Worcester Polytechnic Institute May 23, 2021 This report represents the work of one or more WPI undergraduate students submitted to the faculty as evidence of completion of a degree requirement. WPI routinely publishes these reports on the web without editorial or peer review. 1 Contents 1 Abstract 3 2 Introduction 4 3 Background 5 3.1 Mental Health . .5 3.2 Telehealth . .5 3.3 Augmented Reality (AR) . .6 3.4 AR Therapy . .7 3.5 Avatars . .7 4 Designing the Application 8 4.1 Infrastructure . .8 4.1.1 Networking . .9 4.1.2 Movement . 11 4.1.3 Audio Communication . 12 4.1.4 Immersive Experience . 14 4.1.5 Telemetry . 15 5 User Studies 17 5.1 Results . 17 5.2 Discussion and Future User Studies . 19 6 Contribution and Future Work 21 References 24 2 1 Abstract Rates of mental health disorders among adolescents and younger adults are on the rise with the lack of widespread access remaining a critical issue. It has been shown that teletherapy, defined as therapy delivered remotely with the use of a phone or computer system, may be a viable option to replace in-person therapy in situations where in-person therapy is not possible. Sponsored by the User Experience and Decision Making (UXDM) lab at WPI, this IQP is part of a larger project to address the need for mental health therapy in situations where patients do not have access to traditional in-person care.
    [Show full text]
  • New Realities Risks in the Virtual World 2
    Emerging Risk Report 2018 Technology New realities Risks in the virtual world 2 Lloyd’s disclaimer About the author This report has been co-produced by Lloyd's and Amelia Kallman is a leading London futurist, speaker, Amelia Kallman for general information purposes only. and author. As an innovation and technology While care has been taken in gathering the data and communicator, Amelia regularly writes, consults, and preparing the report Lloyd's does not make any speaks on the impact of new technologies on the future representations or warranties as to its accuracy or of business and our lives. She is an expert on the completeness and expressly excludes to the maximum emerging risks of The New Realities (VR-AR-MR), and extent permitted by law all those that might otherwise also specialises in the future of retail. be implied. Coming from a theatrical background, Amelia started Lloyd's accepts no responsibility or liability for any loss her tech career by chance in 2013 at a creative or damage of any nature occasioned to any person as a technology agency where she worked her way up to result of acting or refraining from acting as a result of, or become their Global Head of Innovation. She opened, in reliance on, any statement, fact, figure or expression operated and curated innovation lounges in both of opinion or belief contained in this report. This report London and Dubai, working with start-ups and corporate does not constitute advice of any kind. clients to develop connections and future-proof strategies. Today she continues to discover and bring © Lloyd’s 2018 attention to cutting-edge start-ups, regularly curating All rights reserved events for WIRED UK.
    [Show full text]
  • Musashi Seimitsu Industry Co., Ltd
    Musashi Seimitsu Industry Co., Ltd. Auto parts manufacturer installs production line remotely during travel ban with Dynamics 365 Remote Assist Musashi Seimitsu Industry is an automotive parts manufacturer with production Customer and sales facilities in 14 countries and 35 hubs worldwide. The global brand slogan, Musashi Seimitsu Industry Co., Ltd. Power to Value, has represented its innovative manufacturing style since 1938. This (http://www.musashi.co.jp/) can-do spirit has protected corporate infrastructure amidst COVID-19. By merging the real and digital worlds using HoloLens 2 with Dynamics 365 Remote Assist, Product or Service the company launched a new production line in Mexico without a single engineer - Microsoft Dynamics 365 leaving Japan. Musashi Seimitsu is embracing the post COVID-19 new normal with Mixed Reality its swift digital transformation. - Microsoft Dynamics 365 Remote Assist - Microsoft HoloLens 2 - Microsoft 365 New product launch almost delayed due to - Microsoft Teams COVID-19 Industry Musashi Seimitsu, manufacturer of powertrains and other parts for 2- and 4-wheeled autos for more Manufacturing than 82 years, engages in every stage of the process, from development to sales. Originating in Japan, the company’s plants now reach 35 countries and global regions, including the European Union, Asia, Organization Size and the Americas. Musashi Seimitsu is a firmly established global supplier, with overseas markets Over 10,000 accounting for approximately 85 percent of sales. Country Japan In early 2020, COVID-19 began its global advance and by late March, overseas business trips had been cancelled. Things worsened as plants in each country scrambled to prevent the spread of infection and by April, most production lines were shut down.
    [Show full text]
  • Arxiv:2008.11239V1 [Cs.CV] 25 Aug 2020 Overlaying Digital Twin on the Real World
    HoloLens 2 Research Mode as a Tool for Computer Vision Research Dorin Ungureanu Federica Bogo Silvano Galliani Pooja Sama Xin Duan Casey Meekhof Jan Stuhmer¨ ∗ Thomas J. Cashman Bugra Tekin Johannes L. Schonberger¨ Pawel Olszta Marc Pollefeys Microsoft Abstract Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with inte- grated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand Figure 1. Microsoft HoloLens 2. tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer viding access to all raw image sensor streams – including vision as well as robotics and encourage contributions from depth and IR. Released together with a public repository the research community. collecting auxiliary tools and sample applications [4], Re- search Mode promoted the use of HoloLens as a powerful tool for doing research in computer vision and robotics [1]. 1. Introduction HoloLens 2 (Fig. 1), announced in 2019, brings a num- Mixed reality technologies have tremendous potential to ber of improvements with respect to the first-generation de- fundamentally transform the way we interact with our envi- vice – like a dedicated DNN core, articulated hand tracking ronment, with other people, and the physical world.
    [Show full text]
  • Innovative Learning Environments in STEM Higher Education
    SPRINGER BRIEFS IN STATISTICS Jungwoo Ryoo Kurt Winkelmann Editors Innovative Learning Environments in STEM Higher Education Opportunities, Challenges, and Looking Forward SpringerBriefs in Statistics SpringerBriefs present concise summaries of cutting-edge research and practical applications across a wide spectrum of felds. Featuring compact volumes of 50 to 125 pages, the series covers a range of content from professional to academic. Typical topics might include: • A timely report of state-of-the art analytical techniques • A bridge between new research results, as published in journal articles, and a contextual literature review • A snapshot of a hot or emerging topic • An in-depth case study or clinical example • A presentation of core concepts that students must understand in order to make independent contributions SpringerBriefs in Statistics showcase emerging theory, empirical research, and practical application in Statistics from a global author community. SpringerBriefs are characterized by fast, global electronic dissemination, standard publishing contracts, standardized manuscript preparation and formatting guidelines, and expedited production schedules. More information about this series at http://www.springer.com/series/8921 Jungwoo Ryoo • Kurt Winkelmann Editors Innovative Learning Environments in STEM Higher Education Opportunities, Challenges, and Looking Forward Editors Jungwoo Ryoo Kurt Winkelmann Pennsylvania State University Valdosta State University Altoona, PA, USA Valdosta, GA, USA This book is an open access publication.
    [Show full text]
  • Mixed Reality
    Mixed Reality Matt Fedorovich Authorized Reseller Lead, Immersive Technology Agenda Part One 01 Remote Assist Overview Part Two 02 Device and Infrastructure Readiness Part Three 03 End User Readiness Part Four 04 Recent Stories Remote Work for the Future Remote Assistance Remote Collaboration • “See what I see” • Everyone in virtual space – AR, VR or MR • Leverage physical world for interaction • Leverage digital world for interaction • Cross-Platform • Cross-Platform • Integration into existing systems is key • Integration into existing systems is key • Solution : D365 Remote Assist • Solutions : Spatial.io & AltSpace VR Remote Assist Overview Work together from anywhere. Microsoft Dynamics 365 Dynamics 365 Remote Assist Empower technicians to collaborate more efficiently by working together from different locations with Dynamics 365 Remote Assist on HoloLens, Android, or iOS devices. Enabled by Microsoft Teams, OneDrive and Dynamics 365. Dynamics 365 Guides Operators learn during the flow of work by providing holographic instructions when and where they are needed. These instruction cards are visually tethered to the place where the work must be done, and can include images, videos, and 3D holographic models How It Works iOS & Android Device Microsoft HoloLens 2 Front Line Technician Remote Expert Technician needs help. Login to Remote Assist Receives call on their Teams Mobile or PC Application and calls expert Device Requires Device Requires Remote Assist App Remote Assist Microsoft Teams Teams License On iOS / Android License On PC or Mobile (w/ Video) Per user Remote Assist App OR On Microsoft HoloLens 2 Per Device Key Journey Points Envisioning & Planning Readiness & Support Innovation & Integration Accelerate Infrastructure Readiness Identify key components that are needed before you think about devices and end users.
    [Show full text]
  • Mixed Reality & Microsoft Hololens
    Mixed Reality & Microsoft HoloLens Laboratorio di Realtà Virtuale Manuel Pezzera – [email protected] Mixed Reality • “Mixed reality (MR) is a spectrum of immersive experiences, connecting and blending physical and digital worlds together in augmented-reality and virtual-reality applications” [cit. Microsoft] Mixed Reality Augmented and Virtual Reality • Virtual Reality: it tricks your eyes into thinking that you are somewhere else. • Augmented reality: it is an extra dimension over VR since it can show the real world around wearer with graphic overlayed. • AR can also totally immerse user just like VR by simply showing pixels everywhere with no transparency to the real world. • In virtual world, users are best advised to stay seated to avoid collision with physical objects which they cannot see in the real world. What is a hologram? • A hologram is an object like any other object in the real world, with only one difference: instead of being made of physical matter, a hologram is made entirely of light. • Holographic objects can be viewed from different angles and distances, just like physical objects. They can also interact with the environment (e.g., bouncing ball on a table) • Hologram do not offer any physical resistance when touched or pushed because they don’t have a mass. Mixed reality vs Augmented Reality • MR can be mentioned as an enhanced form of Augmented Reality (AR). In AR the visible natural environment is overlaid with a layer of digital content. • But the case is slightly different with MR, as virtual objects are integrated into and responsive to the real world. For instance, a virtual ball under your desk would be blocked from a user’s view unless he bends.
    [Show full text]
  • Neural Networks on Microsoft Hololens 2
    Institute of Parallel and Distributed Systems University of Stuttgart Universitätsstraße 38 D–70569 Stuttgart Bachelor Thesis Neural Networks on Microsoft HoloLens 2 Léon Lazar Course of Study: Informatik (Computer Science) Examiner: Prof. Dr. rer. nat. Dr. h. c. Kurt Rothermel Supervisor: Johannes Kässinger, M.Sc. Commenced: November 5, 2020 Completed: May 5, 2021 Abstract The goal of the present Bachelor thesis is to enable comparing different approaches of integrating Neural Networks in HoloLens 2 applications in a quantitative and qualitative manner by defining highly diagnostic criteria. Moreover, multiple different approaches to accomplish the integration are proposed, implemented and evaluated using the aforementioned criteria. Finally, the work gives an expressive overview of all working approaches. The basic requirements are that Neural Networks trained by TensorFlow/Keras can be used and executed directly on the HoloLens 2 without requiring an internet connection. Furthermore, the Neural Networks have to be integrable in Mixed/Augmented Reality applications. In total four approaches are proposed: TensorFlow.js, Unity Barracuda, TensorFlow.NET, and Windows Machine Learning which is an already existing approach. For each working approach a benchmarking application is developed which runs a common reference model on a test datatset to measure inference time and accuracy. Moreover, a small proof of concept application is developed in order to show that the approach also works with real Augmented Reality applications. The application uses a MobileNetV2 model to classify image frames coming from the webcam and displays the results to the user. All the feasible approaches are evaluated using the aforementioned evaluation criteria which include ease of implementation, performance, accuracy, compatibility with Machine Learning frameworks and pre-trained models, and integrability with 3D frameworks.
    [Show full text]
  • NDT) Use Cases Fakhreddine Ababsa
    Augmented Reality Application in Manufacturing Industry: Maintenance and Non-destructive Testing (NDT) Use Cases Fakhreddine Ababsa To cite this version: Fakhreddine Ababsa. Augmented Reality Application in Manufacturing Industry: Maintenance and Non-destructive Testing (NDT) Use Cases. 7th International Conference on Augmented Reality, Virtual Reality, and Computer Graphics, Sep 2020, Lecce, Italy. pp.333-344, 10.1007/978-3-030- 58468-9_24. hal-02976923 HAL Id: hal-02976923 https://hal.archives-ouvertes.fr/hal-02976923 Submitted on 23 Oct 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Augmented Reality Application in Manufacturing Industry: Maintenance and Non-destructive Testing (NDT) Use Cases B Fakhreddine Ababsa( ) Arts et Metiers Institute of Technology, LISPEN, HESAM University, Paris, France [email protected] Abstract. In recent years, a structural transformation of the manufacturing indus- try has been occurring as a result of the digital revolution. Thus, digital tools are now systematically used throughout the entire value chain, from design to produc- tion to marketing, especially virtual and augmented reality. Therefore, the purpose of this paper is to review, through concrete use cases, the progress of these novel technologies and their use in the manufacturing industry.
    [Show full text]
  • Review of Microsoft Hololens Applications Over the Past Five Years
    applied sciences Review Review of Microsoft HoloLens Applications over the Past Five Years Sebeom Park , Shokhrukh Bokijonov and Yosoon Choi * Department of Energy Resources Engineering, Pukyong National University, Busan 48513, Korea; [email protected] (S.P.); [email protected] (S.B.) * Correspondence: [email protected]; Tel.: +82-51-629-6562; Fax: +82-51-629-6553 Abstract: Since Microsoft HoloLens first appeared in 2016, HoloLens has been used in various industries, over the past five years. This study aims to review academic papers on the applications of HoloLens in several industries. A review was performed to summarize the results of 44 papers (dated between January 2016 and December 2020) and to outline the research trends of applying HoloLens to different industries. This study determined that HoloLens is employed in medical and surgical aids and systems, medical education and simulation, industrial engineering, architecture, civil engineering and other engineering fields. The findings of this study contribute towards classifying the current uses of HoloLens in various industries and identifying the types of visualization techniques and functions. Keywords: head-mount display; HoloLens; augmented reality; mixed reality; virtual reality; visualization 1. Introduction Virtual reality (VR), augmented reality (AR) and mixed reality (MR) have been emerg- Citation: Park, S.; Bokijonov, S.; Choi, ing technologies for several years, but they are now able to generate realistic images, sounds Y. Review of Microsoft HoloLens and other sensations that allow users to experience a spectacular imaginary world [1]. It can Applications over the Past Five Years. be difficult to understand the difference between VR, AR and MR, at a glance.
    [Show full text]
  • Mixed Reality Intelligence
    Mixed Reality Intelligence Manufacturing Edition December 2020 Table of Contents Background 04 Methodology 05 Who We Talked To – October 2020 06 Things To Know About Mixed Reality 07 Mixed Reality in Manufacturing 08 Final Thoughts 21 Detailed Research Objectives & Audience Recruit 22 Learn More To learn how mixed reality solutions can support and enhance your business, or to connect with a mixed reality specialist, please visit https://aka.ms/MixedRealityDesk. © Hypothesis Group 2020. © Microsoft 2020. All rights reserved. 10/20 Background As companies across the world begin employing immersive technologies to drive efficiencies, Mixed Reality is poised to change the way businesses operate. Mixed Reality blends the digital and physical world by overlaying 3D digital objects onto the user’s physical world, offering a way for users to interact organically using head-mounted devices (HMDs). Located at the center of the virtuality continuum, the capabilities of Mixed Reality render it uniquely positioned to transform the workplace as we know it. Microsoft has been at the forefront of the immersive technology market and a trailblazer in Mixed Reality, innovating in both hardware and software to offer cross-platform Mixed Reality solutions. With the introduction of HoloLens, the first untethered Mixed Reality headset designed with business use cases in mind, Microsoft established itself as a leader in the Mixed Reality space and validated the enterprise value of this nascent industry by partnering with prominent entities in contracts worth up to half a billion dollars. In October 2020, Microsoft and Hypothesis embarked on an initiative to hear from IT and Business Decision Makers across three countries with the goal of developing a deeper understanding of how Mixed Reality is being utilized across three key industries: manufacturing, retail, and healthcare.
    [Show full text]
  • Material Safety Data Sheets (MSDS)/Safety Data Sheets (SDS) Are Required by Various National and International Occupational Worker Safety Regulations
    Product Data Sheet Section I – Product and Company Information Product: Lithium ion and Lithium ion polymer batteries Manufacturer: Models: See Microsoft Product Battery Information Table, Section 14 Microsoft Corporation Effective Date: January 10, 2020 One Microsoft Way Version: 2020A Redmond, WA 98052-6399 Tel 425 882 8080 Section II – Composition Information Not chemically dangerous during normal use in accordance with Microsoft recommendations as stated in the user manuals or other similar documentation. Exposure to hazardous chemicals is not expected with normal handling and use. In particular, the cell or battery should not be opened or burned. Section III – Hazards Identification Emergency Overview: Do not open or disassemble cells or batteries or expose them to fire or open flame. Do not puncture or deform. Cells and batteries present a hazard only if mishandled in a way that causes damage to the cell or battery or compromises their integrity. Primary Routes of Exposure: Risk of exposure to hazardous materials will only occur if the cell or battery is physically, thermally or electrically abused to the extent that the integrity of the cell or battery is compromised. In this case, exposure to the electrolyte can occur through ingestion, inhalation, eye contact and skin contact. Potential Health Effects: If the battery or cell has been damaged or ruptured, the electrolyte solution, which is corrosive, could be released and cause burns to the eyes, skin or respiratory tract. Ingestion of the electrolyte can cause serious burns of the gastrointestinal tract. Section IV – First Aid Measures If you get electrolyte in your eyes, flush with water for 15 minutes without rubbing and immediately contact a physician.
    [Show full text]