Computation Offloading Of Augmented Reality In Warehouse Order Picking Creative Technology Bachelor of Science Thesis Harald Eversmann July, 2019 UNIVERSITY OF TWENTE Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS) Supervisor Dr. Job Zwiers Critical observer Dr. Randy Klaassen Client Gerben Hillebrand CaptureTech Corporation B.V. [Page intentionally left blank] 1 ABSTRACT A novel method is proposed to implement computation offloading in augmented reality (AR) technology, such that said technology can be of more use for industrial purposes, and in this case specifically, warehouse order picking. The proposed method utilises a wireless connection between the AR device in question and a server and lets them communicate with each other via video and MJPEG live streams. Experiments show promising results for the prototype, but not yet in terms of fully offloading the AR devices workload. It is expected that rising technologies like faster Wi-Fi connection can help in the successful conclusion of fully offloading AR devices. 2 3 ACKNOWLEDGEMENTS The author would like to express his deep gratitude towards CaptureTech Corporation B.V. and in particular Gerben Hillebrand for the opportunity, knowledge, and resources to make this research project possible. The author thanks Dr. J. Zwiers for his continuous supervision and assistance throughout the entire graduation project in question. Additionally, the author would also like to ex- press his thanks to Dr. R. Klaassen for his role as critical observer during the process of this research. 4 TABLE OF CONTENTS ABSTRACT ........................................................................................................................................................ 2 ACKNOWLEDGEMENTS .................................................................................................................................... 4 LIST OF TABLES ................................................................................................................................................ 7 LIST OF FIGURES ............................................................................................................................................... 8 LIST OF ABBREVIATIONS .................................................................................................................................. 9 I. INTRODUCTION .......................................................................................................................................... 10 II. BACKGROUND ........................................................................................................................................... 12 AUGMENTED REALITY ............................................................................................................................................ 12 USER TEST ........................................................................................................................................................... 12 OFFLOADING ........................................................................................................................................................ 14 POTENTIAL SUPPORTING DEVICES.............................................................................................................................. 16 PROBLEM STATEMENT ............................................................................................................................................ 17 III. IDEATION .................................................................................................................................................. 18 CREATIVE TECHNOLOGY DESIGN PROCESS ................................................................................................................... 18 STAKEHOLDERS ..................................................................................................................................................... 18 OBSERVATIONS ..................................................................................................................................................... 18 DEVELOPMENT ..................................................................................................................................................... 19 COMMUNICATION ................................................................................................................................................. 20 SENDING AND RECEIVING IMAGES TO PROCESS .......................................................................................................... 20 OFFLOADING AS TOTAL BACKEND ............................................................................................................................ 21 IMAGE RECOGNITION ............................................................................................................................................. 21 USER EXPERIENCE ................................................................................................................................................. 22 INTERACTION DIAGRAM .......................................................................................................................................... 23 REQUIREMENTS .................................................................................................................................................... 24 IV. IMPLEMENTATION .................................................................................................................................... 26 FIRST PROTOTYPE .................................................................................................................................................. 26 SECOND PROTOTYPE .............................................................................................................................................. 27 THIRD PROTOTYPE ................................................................................................................................................. 29 SETTING UP THE SERVER ........................................................................................................................................ 32 ANDROID APPLICATION .......................................................................................................................................... 33 V. EVALUATION ............................................................................................................................................. 36 POSSIBLE EVALUATIONS .......................................................................................................................................... 36 TEST SETUP .......................................................................................................................................................... 37 COLLECTED DATA .................................................................................................................................................. 38 DISCUSSION ......................................................................................................................................................... 39 EVALUATION CONCLUSION ...................................................................................................................................... 40 VI. CONCLUSION ............................................................................................................................................ 41 VII. DISCUSSION AND FUTURE RESEARCH ...................................................................................................... 42 IV. REFERENCES ............................................................................................................................................. 43 APPENDIX A: QUESTIONNAIRE ....................................................................................................................... 45 APPENDIX B: CONSENT FORM........................................................................................................................ 46 5 APPENDIX C: SEQUENCE DIAGRAM COMMANDS ........................................................................................... 48 APPENDIX D: FIRST PROTOTYPE ..................................................................................................................... 49 APPENDIX E: CODE FOR MEASURING TIME BETWEEN FRAMES ...................................................................... 50 APPENDIX F: SECOND PROTOTYPE WITH ZBAR .............................................................................................. 51 APPENDIX G: THIRD PROTOTYPE WITH VIDEOSTREAM AND WHITE BACKGROUND ...................................... 52 APPENDIX H: IMAGE PROCESSING SERVER CODE ........................................................................................... 53 APPENDIX I: ANDROID APPLICATION ............................................................................................................. 55 6 LIST OF TABLES Page number Table number Description 25 3.1 A list of requirements for the prototype, classified in general and prototype specific necessities. 27 4.1 The measurements of time between two frames for the first prototype. 29 4.2 The time between two frames measured for the program with ZBar implemented. 31 4.3 The time between two frames measured for the program with Imutils and a transparent back- ground. 31 4.4 The time between two frames measured for the program