THEVIRTUALMANUFACTURINGSTATION A Framework for Collaborative Assessment of Manual Assembly Tasks MICHAELOTTO aus Memmingen DISSERTATION zur Erlangung des Doktorgrades - Dr. rer. nat. - der Fakultät für Ingenieurwissenschaften, Informatik und Psychologie der Universität Ulm Institut für Medieninformatik Fakultät für Ingenieurwissenschaften, Informatik und Psychologie Universität Ulm 2020 acting dean: Prof. Dr.-Ing. Maurits Ortmanns referees: Prof. Dr. Enrico Rukzio, Ulm University Prof. Dr.-Ing. Gabriel Zachmann, University of Bremen day of defense: 10/23/2020 Michael Otto: The Virtual Manufacturing Station: A Framework for Collaborative Assessment of Man- ual Assembly Tasks, Doctoral dissertation. ©2020 This document was typeset in LATEX using the typographical look-and-feel classicthesis developed by André Miede. classicthesis is available for both LATEX and LYX: https://bitbucket.org/amiede/classicthesis/ This document uses graphics from flaticon.com by the author "Freepik", "Good Ware", "pixelmeetup", "monkik", "geotatah", "Vitaly Gorbachev" and "Gregor Cres- nar". Thank you. ABSTRACT In the automotive industry, markets are demanding more product models, derivatives and extra equipment with shorter life-cycles. Due to these effects, planning of manual assembly is becoming more com- plex and diverse. With the current mostly physical mock-up produc- tion validation methods, these changes cause considerable increases in production planning costs, product preparation time and and put required quality levels at risk. The use of virtual assessment methods during the production validation phase is a promising countermea- sure for these effects. As of yet, there is no holistic view on virtual production validation in the literature since related publications either offer self-contained, practical approaches or theoretical constructs without direct appli- cability. In order to bridge this gap, this doctoral thesis focuses on the analysis, development, integration and evaluation of collabora- tive, virtual methods for assessments of manual assembly processes in the manufacturing industry. This research focuses on the question whether collaborative virtual environments can support production validation workshops, so that verification criteria can be assessed in the same quality, less time and with lower costs compared to hardware-based workshops. A new system is being developed and proposed, called the "Virtual Manufacturing Station" (VMS). It is a framework for holistic virtual production validation. The VMS consists of a multi-display environ- ment, sensors and software components so that it can be used in in- teractive, collaborative, virtual production validation workshops. In order to provide production validation engineers with such a virtual framework, six theoretical key properties are derived for the VMS: "collaborative virtual environments", "multi-user support", "original size visualization", "natural user interfaces", "integration of physical and digital mock-ups" and "asymmetric/symmetric output." This the- oretical framework is based on four research areas with each con- tributing to at least one of the theoretical key properties. These areas are "VR simulation software", "markerless, full-body motion capture", "large high-resolution displays" and "spatial augmented reality." This doctoral thesis presents advances in basic human computer interaction research, technology, production validation methodology substantiated by the following studies: Two contextual inquiry stud- ies on virtual production validation, two technological evaluations using a markerless full-body motion capture system presented, a sys- tematic design space analysis for spatial augmented reality, a stan- dardized benchmark for VR assessments of manual assembly tasks, a iii size perception study, and five studies on basic research related to vir- tual production validation. The latter research studies cover a broad investigation scope, such as measurement of task completion times, error rates and qualitative feedback. Overall, these studies have demonstrated that the VMS framework is reliable and applicable for collaborative virtual production valida- tion workshops. Although this research has been conducted for the automotive sector, the presented VMS framework is also applicable to the manufacturing industry in general. The VMS methods and tools discussed contribute to higher workshop collaboration perfor- mance, lower task completion times, reduced preparation work and a reduced dependency on physical mock-ups. The VMS reduces the overall costs in production validation while simultaneously maintain- ing the validation quality. iv PUBLICATIONS The following publications are sorted chronologically. Some ideas, texts and figures have appeared previously in the following list of directly related core [C] publications: [C1] Michael Otto, Michael Prieur, and Enrico Rukzio. “Using Scal- able, Interactive Floor Projection for Production Planning Sce- nario.” In: Proceedings of the Ninth ACM International Confer- ence on Interactive Tabletops and Surfaces, Poster. ACM ITS 2014. Reuses according to the author’s rights of ACM Digital Li- brary. Dresden, Germany: Association for Computing Machin- ery, 2014, pp. 363–368. doi: 10.1145/2669485.2669547. [C2] Michael Otto, Philipp Agethen, Florian Geiselhart, and Enrico Rukzio. “Towards Ubiquitous Tracking: Presenting a Scalable, Markerless Tracking Approach Using Multiple Depth Cam- eras.” In: Proceedings of the EuroVR 2015. EuroVR 2015. Best Industrial Paper Award. Lecco, Italy: European Association for Virtual Reality and Augmented Reality, 2015. url: https: //www.eurovr-association.org/wp-content/uploads/2019/ 11/06112017_Proceedings-1.pdf. [C3] Florian Geiselhart, Michael Otto, and Enrico Rukzio. “On the Use of Multi-Depth-Camera Based Motion Tracking Systems in Production Planning Environments.” In: Proceedings of the 48th CIRP Conference on Manufacturing Systems. CIRP CMS 2015. Vol. 41. CIRP CMS. Licensed under CC BY-NC-ND 4.0, https://creativecommons.org/licenses/by-nc-nd/4.0/. Ischia, Italy: Elsevier, 2015, pp. 759–764. doi: 10 . 1016 / j . procir . 2015.12.088. [C4] Michael Otto, Michael Prieur, Philipp Agethen, and Enrico Rukzio. “Dual Reality for Production Verification Workshops: A Comprehensive Set of Virtual Methods.” In: Proceedings of the 6th CIRP Conference on Assembly Technologies and Systems. CIRP CATS 2016. Vol. 44. CIRP CATS 2016. Licensed under CC BY-NC-ND 4.0, https://creativecommons.org/licenses/by- nc-nd/4.0/. Gothenburg, Sweden, 2016, pp. 38–43. doi: 10 . 1016/j.procir.2016.02.140. [C5] Michael M. Otto, Philipp Agethen, Florian Geiselhart, Michael Rietzler, Felix Gaisbauer, and Enrico Rukzio. “Presenting a Holistic Framework for Scalable, Marker-Less Motion Cap- turing: Skeletal Tracking Performance Analysis, Sensor Fu- sion Algorithms and Usage in Automotive Industry.” In: Jour- nal of Virtual Reality and Broadcasting, JVRB 3.13 (2016). Li- v censed under Digital Peer Publishing Lizenz (v2, de), http:// www.dipp.nrw.de/lizenzen/dppl/dppl/DPPL_v2_en_06-2004.html, pp. 1–10. doi: 10.20385/1860-2037/13.2016.3. [C6] Philipp Agethen, Felix Gaisbauer, Michael Otto, and Enrico Rukzio. “Interactive Simulation for Walk Path Planning within the Automotive Industry.” In: Proceedings of the 51st CIRP Con- ference on Manufacturing Systems. CIRP CMS 2018. Vol. 72. Li- censed under CC BY-NC-ND 4.0, https://creativecommons.org/ licenses/by-nc-nd/4.0/. Stockholm, Sweden, 2018, pp. 285– 290. doi: 10.1016/j.procir.2018.03.223. [C7] Michael Otto, Eva Lampen, Philipp Agethen, Mareike Lan- gohr, Gerald Masan, and Enrico Rukzio. “Evaluation on Per- ceived Sizes Using Large-Scale Augmented Floor Visualiza- tion Devices.” In: Proceedings of the 8th ACM International Sym- posium on Pervasive Displays. ACM PerDis 2019. Reuses accord- ing to the author’s rights of ACM Digital Library. Palermo, Italy: Association for Computing Machinery, 2019, pp. 1–7. doi: 10.1145/3321335.3324951. [C8] Michael Otto, Eva Lampen, Felix Auris, Felix Gaisbauer, and Enrico Rukzio. “Applicability Evaluation of Kinect for EAWS Ergonomic Assessments.” In: Proceedings of the 52nd CIRP Con- ference on Manufacturing Systems. CIRP CMS 2019. Vol. 81. Li- censed under CC BY-NC-ND 4.0, https://creativecommons.org/ licenses/by-nc-nd/4.0/. Ljubljana, Slovenia, 2019, pp. 781–784. doi: 10.1016/j.procir.2019.03.194. [C9] Michael Otto, Eva Lampen, Philipp Agethen, Mareike Lan- gohr, Gabriel Zachmann, and Enrico Rukzio. “A Virtual Real- ity Assembly Assessment Benchmark for Measuring VR Per- formance & Limitations.” In: Proceedings of the 52nd CIRP Con- ference on Manufacturing Systems. CIRP CMS 2019. Vol. 81. Li- censed under CC BY-NC-ND 4.0, https://creativecommons.org/ licenses/by-nc-nd/4.0/. Ljubljana, Slovenia, 2019, pp. 785–790. doi: 10.1016/j.procir.2019.03.195. [C10] Michael Otto, Eva Lampen, Philipp Agethen, Gabriel Zach- mann, and Enrico Rukzio. “Using large-scale augmented floor surfaces for industrial applications and evaluation on perceived sizes.” In: Personal and Ubiquitous Computing (2020). Licensed under CC BY 4.0, https://creativecommons.org/licenses/by/4.0/, pp. 1–28. doi: 10.1007/s00779-020-01433-z. Further [F] co-authored publications that are not directly related to the thesis’ topic are: vi [F1] Philipp Agethen, Michael Otto, Felix Gaisbauer, and Enrico Rukzio. “Presenting a Novel Motion Capture-Based Approach for Walk Path Segmentation and Drift Analysis in Manual As- sembly.” In: Proceedings of the Sixth International Conference