Overcoming Limitations of the Hololens for Use in Product Assembly
Total Page：16
File Type：pdf, Size：1020Kb
https://doi.org/10.2352/ISSN.2470-1173.2019.2.ERVR-179 © 2019, Society for Imaging Science and Technology Overcoming Limitations of the HoloLens for Use in Product Assembly Jack Miller, Melynda Hoover, Eliot Winer; Iowa State University - Ames, Iowa Abstract use in an AR work instruction assembly application. The research presented here will focus entirely on the development of the Augmented Reality (AR) is an emerging technology that could application and the novel techniques used to mitigate the limitations greatly increase training efficiency and effectiveness for assembly of the HoloLens. This application was also used to conduct a user processes. In fact, studies show that AR may reduce time and study comparing the effectiveness of work instructions delivered via errors by as much as 50%. While many devices are available to an AR HMD, AR tablet, tablet model-based instructions and display AR content, the use of Head Mounted Displays (HMDs) is desktop model-based instructions [2]. The results of this study are the most interesting for manufacturing and assembly as they free presented in a separate paper. For the study, due to all the custom up the user’s hands. Due to the emerging nature of this technology, enhancements made to the HMD system, it was necessary to there are many limitations including input, field of view, tracking validate that work instructions and other parts of the system were and occlusion. The work presented in this paper explores the use of functioning together properly. To accomplish this, a separate data the Microsoft HoloLens to deliver AR work instructions for visualization application was developed. This application fused data product assembly. An AR assembly application was developed to collected from the HoloLens device, an Empatica E4 physiological guide a trainee through the assembly of a mock aircraft wing. To wrist sensor, and errors via post-process grading of the trainee’s ensure accurate instructions were displayed, novel techniques completed assembly. This fusion through the data visualization were developed to mitigate the HoloLens’ tracking and display application allowed the analyst to better understand the training limitations. Additionally, a data visualization application was session and identify any issues in the AR application. In addition to developed to validate the training session and explore trends validating the work instructions, the analyst may also discover new through data collected from the HoloLens along with wearable high-level trends about the trainee and the training environment physiological sensor data. through the fusion of these data sources. Through the delivery and validation of these work instructions, this work seeks to enhance the effectiveness of training for maintenance and assembly operations Introduction using an emerging commodity HMD. Augmented Reality (AR) is an emerging technology that could greatly increase training efficiency and effectiveness. In fact, Henderson & Feiner found that AR may reduce time and errors by Background as much as 50% for maintenance and assembly operations [1]. AR Augmented reality is a technology that has been heavily consists of computer-generated visual information overlaid on top researched since the 1990’s and shows promise for a variety of of a view of the real world. These computer-generated visuals are applications. AR involves merging computer-generated visuals with often delivered via a tablet or Head Mounted Display (HMD). a view of the real world. These visuals provide additional Traditionally, tablets have been the preferred method of delivering information to what is seen in reality. In a seminal paper, Azuma AR content due to the technological constraints of HMDs and the surveyed the various applications of AR and found that it will have commercial availability of tablets. HMDs of the past tended to be a major impact in the fields of medical, robotics, entertainment, expensive custom-built solutions for research purposes and not visualization and notably manufacturing and repair [3]. viable for commercial use. However, in recent years several Additionally, Krevelen & Poelman found similar use cases and commercially available AR HMDs have come to market and show many more including education, military training and others [4]. promise for use in maintenance operations. The main advantage of Manufacturing is a large area for AR research including many use AR HMDs over tablets is that they overlay the computer-generated cases for industry and the military. Specifically, AR guided visuals directly over the users view of the real world while allowing assembly and disassembly tasks make up 30% of published research them to remain hands free. This is ideal for maintenance training as for AR manufacturing [5]. the trainee is likely to be working with their hands while receiving Augmented Reality Assembly Applications work instructions. Numerous papers have explored the use of AR for the delivery AR HMDs show great potential for use in maintenance of work instructions. The first AR work instruction application was operations and training. However, due to the emerging nature of this proposed by Caudell & Mizell to display work instructions for an technology, many of the components that make up these AR HMDs aircraft manufacturing process through a transparent HMD [6]. are still under development. Therefore, there are many limitations However, the technology at that time was still in its infancy. Since within the technology that must be mitigated including input this seminal paper, researchers have developed many systems to methods, field of view, tracking and occlusion. These limitations are explore possibilities of this technology. especially important to maintenance and assembly tasks as it is Prior work has shown many benefits of using AR to deliver important to ensure the trainee receives accurate work instructions work instructions. Studies have shown that AR delivered work and therefore has an effective training session. instructions drastically reduce errors when compared to traditional Specifically, this paper discusses the novel techniques work instruction delivery methods [7]–[9]. Furthermore, Baird & employed to mitigate the limitations of the Microsoft HoloLens for IS&T International Symposium on Electronic Imaging 2019 The Engineering Reality of Virtual Reality 2019 179-1 Barfield found that in addition to reducing errors, AR also decreased displayed. Many past devices had issues with dim displays and low the time to complete assembly tasks. In a similar study, Henderson resolution leading to virtual objects appearing farther away than and Feiner found that AR delivered instructions significantly they should [4]. Even though recent advances in technology have reduced the amount of time to locate tasks [1]. This study also found improved resolution and opacity in many devices, there are still that AR reduced head and neck movements leading to reduced many improvements that can be made. Occlusion has been found to physical workload. Additionally, since the information is displayed effectively give depth perception cues that may overcome when it needs to be for a task, the user does not have to recall the limitations in resolution and opacity [19]. Through accurate information from 2D instructions, decreasing mental workload [10]. occlusion, the user may better understand the proper representation These benefits lead to enhanced assembly performance and of the instructions given [20], [21]. therefore reduced costs. Additional research is needed to understand and mitigate these limitations for use in AR work instruction applications. While Augmented Reality Hardware hardware improvements of this emerging technology will be very AR assembly instructions are often delivered through beneficial in the future, many of the current limitations can be transparent HMDs, tablets and smartphones. Due to commercial mitigated through novel development techniques. The work in this availability and relatively low cost, tablets and smartphones have paper explores the limitations of the Microsoft HoloLens and been the preferred AR platform [11]. However, the main drawback provides novel solutions to overcome them. of these devices for delivering assembly instructions is that typically users must hold the device when using it. This is not ideal as the user will likely be working with their hands when receiving work Methodology instructions. The methodology section describes the challenges faced when AR HMDs may overcome this issue through a hand-free developing an AR assembly application for a commercially experience. In the past, AR HMDs have been expensive custom- available AR HMD. An AR application was developed to guide a built solutions created for specific use cases. This has been a large trainee through the assembly of a mock aircraft wing, shown in pitfall for the use in many applications as they are not readily Figure 1. This assembly includes a 46-step process of picking, available. However, in recent years numerous commercially placing, and assembling a variety of parts and fasteners. These steps available AR HMDs have come to market. Some of the most popular require the trainee to navigate through three separate stations: a parts currently available are the Microsoft HoloLens, Daqri Smart table, fastener bins, and assembly station. The parts table includes Glasses and Meta 2 [12]–[14]. These devices can track the large wooden parts to be placed and assembled. The fastener bins environment while displaying