Experimental Test report Document information Project Title 6th Sense Project Number E.02.25 Project Manager Fraunhofer Austria Deliverable Name Verification Report “Experimental Test Report” Deliverable ID Del 4.2 Edition 00.01.01 Template Version 03.00.00 Task contributors Fraunhofer Austria, FREQUENTIS AG, Fraunhofer FKIE, subcontracted by Fraunhofer Austria Abstract The project Sixth Sense postulates that the users “body language" is different at “good” and “bad” decisions. The project follows the idea to use the whole body language of a user for communicating with a machine. In our case it is an Air Traffic Controller (ATCO) with an Air Traffic Tower CWP. Specifically we intend to analyse the correlation of the change of the behaviour of an ATCO - expressed through his body language - with the quality of the decisions he is making. For that, an experiment was set up, data about the user behaviour was collected, explored and analysed. This document is the test report of the proof of concept for the Sixth Sense prototype and its core components. Results of our work may be used for an early warning for “bad” situations about to occur or decision aids for the ATCO. Authoring & Approval Prepared By - Authors of the document. Name & Company Position & Title Date Volker Settgast/Fraunhofer Austria Project Contributor 22.06.2015 Nelson Silva / Fraunhofer Austria Project Contributor 01.07.2015 Carsten Winkelholz, Jesscia Schwarz/ Subcontracted Project 07.07.2015 Fraunhofer FKIE Contributor Michael Poiger / Frequentis AG Project Contributor 08.07.2015 Florian Grill / Frequentis AG Project Contributor 08.07.2015 Reviewed By - Reviewers internal to the project. Name & Company Position & Title Date Theodor Zeh / Frequentis Technical Coordinator 15.07.2015 Eva Eggeling / Fraunhofer Austria Project Manager 15.07.2015 Approved for submission to the SJU By - Representatives of the company involved in the project. Name & Company Position & Title Date Theodor Zeh / Frequentis Technical Coordinator 30.07.2015 Eva Eggeling / Fraunhofer Austria Project Manager 30.07.20125 Rational for rejection None. Document History Edition Date Status Author Justification 00.00.01 20/06/2015 Draft Eva Eggeling New Document 00.00.03 07/07/2015 Update Volker Settgast merged version 00.00.04 09/07/2015 Update all merged version 00.00.08 15/07/2015 Update all merged version Submission 00.01.00 30/07/2015 Eva Eggeling Merged Version Version 00.01.01 15/09/2015 Review Version Eva Eggeling/all Resubmission Intellectual Property Rights (foreground) This deliverable consists of SJU foreground. 2 of 98 ©SESAR JOINT UNDERTAKING, 2011. Created by Fraunhofer Austria, FREQUENTIS AG for the SESAR Joint Undertaking within the frame of the SESAR Programme co-financed by the EU and EUROCONTROL. Reprint with approval of publisher and the source properly acknowledged. Table of Contents TABLE OF CONTENTS ..................................................................................................................................... 3 LIST OF TABLES ................................................................................................................................................ 4 LIST OF FIGURES .............................................................................................................................................. 4 EXECUTIVE SUMMARY .................................................................................................................................... 6 1.1 PURPOSE OF THE DOCUMENT ............................................................................................................... 7 1.2 INTENDED READERSHIP......................................................................................................................... 7 1.3 ACRONYMS AND TERMINOLOGY ........................................................................................................... 7 2 THE EXPERIMENT ..................................................................................................................................... 9 2.1 EXPERIMENTAL SETUP ......................................................................................................................... 9 2.2 OPERATIONAL SCENARIO ................................................................................................................... 10 2.3 ROLES AND RESPONSIBILITIES ........................................................................................................... 11 2.4 TECHNICAL SETUP OF THE EXPERIMENT ............................................................................................ 12 2.5 AMQ BROKER ..................................................................................................................................... 13 2.6 THE HUMAN MACHINE INTERFACE (HMI) .......................................................................................... 13 3 PERFORMING THE EXERCISES .......................................................................................................... 16 3.1 PROFILE OF PARTICIPANTS ................................................................................................................. 16 3.2 DATA ANALYSIS, EXPLORATION AND VISUALIZATION ......................................................................... 17 3.2.1 Heart Rate vs Observations List ................................................................................................ 18 3.2.2 Eye-Tracker and mouse Analysis .............................................................................................. 19 3.2.3 Simple Metrics and Data Exploration ........................................................................................ 20 4 RESULTS ................................................................................................................................................... 23 4.1 WORKLOAD ESTIMATES BASED ON QUESTIONNAIRES ....................................................................... 23 4.2 HINTS FOR H1 - EXPLORING THE SENSOR DATA ............................................................................... 28 4.2.1 Sixth Sense Prototype Framework for Data Exploration ........................................................ 31 4.2.2 Categorization of Metrics regarding mental Aspects .............................................................. 34 4.2.3 Research Questions .................................................................................................................... 38 4.2.4 Data Exploration and Analysis ................................................................................................... 40 4.3 HINTS FOR H2 - ANALYSIS OF THE ARRIVAL AND DEPARTURE WORKFLOWS ................................... 54 4.3.1 Implementation ............................................................................................................................. 54 4.3.2 Results of the Analysis of ATC Workflow Steps ...................................................................... 55 4.3.3 Machine Learning Experiments ................................................................................................. 57 4.4 EVENT TRACE ANALYSIS .................................................................................................................... 61 4.4.1 Variable Length Markov Models (VLMM) ................................................................................. 61 4.4.2 Scatterplot Matrix for Measures ................................................................................................. 64 4.4.3 Visualization of Sequential Patterns .......................................................................................... 64 4.4.4 Insights regarding interaction sequences ................................................................................. 66 4.4.5 States corresponding to outliers and around ........................................................................... 75 4.5 CONCLUSION ....................................................................................................................................... 77 4.6 FUTURE WORK .................................................................................................................................... 79 REFERENCES ................................................................................................................................................... 80 APPENDIX A ...................................................................................................................................................... 82 A.1 TECHNICAL VERIFICATION DETAILS OF EXERCISE 1 AND 2 ............................................................... 82 A.1.1 Kinect ............................................................................................................................................. 87 A.1.2 Speech Recognition ..................................................................................................................... 92 APPENDIX B QUESTIONNAIRES ............................................................................................................ 94 3 of 98 ©SESAR JOINT UNDERTAKING, 2011. Created by Fraunhofer Austria, FREQUENTIS AG for the SESAR Joint Undertaking within the frame of the SESAR Programme co-financed by the EU and EUROCONTROL. Reprint with approval of publisher and the source properly acknowledged. List of tables Table 1 - Description of the workflow steps ...........................................................................................