Fakult¨at fur¨ Informatik CSR-21-02 Adaptive User Interface for Automotive Demonstrator Hasan Aljzaere · Owes Khan · Wolfram Hardt Juli 2021 Chemnitzer Informatik-Berichte Adaptive User Interface for Automotive Demonstrator Master Thesis Submitted in Fulfilment of the Requirements for the Academic Degree M.Sc. Dept. of Computer Science Chair of Computer Engineering Submitted by: Hasan Aljzaere Student ID: 322205 Date: 22.12.1988 Supervising tutor: Prof. Dr. Dr. h. c. W. Hardt M. Sc. Owes Khan 1 Abstract The BlackPearl in the Computer Engineering Department is an Automotive Demonstrator, which has a variety of sensors, and users can control these via the server. The server is responsible for the remote interaction, the Smart Queue, and the Raspberry Pi display for human interaction. The Automotive Demonstrator consists of four components, which are installed on the CE-Box: Main QML Application, Main Server, Live Stream, and Smart Queue. All of these servers are running on three single-board computers (Raspberry Pi 3B): Main, BlackPearl, and Camera servers. The Automotive Demonstrator is built with the latest version from both Qt and NodeJS, and the components can access, store and exchange the data in JSON format. The BlackPearl will be controlled via four types of interaction methods: Web server, Voice commands (Sparrow), Pi Display, and Gamepad. The outcome of this thesis is a configurable and adaptive User Interface for Automotive Demonstrator, and this can be easily updated, customized, and accessible for new applications without the need to update or rebuild the program. Keywords: User Interaction, Adaptive UI, Configurable UI, Smart Queue, Raspberry Pi, Qt/QML, NodeJS 2 Zusammenfassung Der BlackPearl ist im Fachbereich Technische Informatik ein Automotive Demonstrator, der über eine Vielzahl von Sensoren verfügt, die der Benutzer über einen Server steuern kann. Der Server ist für die Ferninteraktion zuständig und die Smart Queue sowie der Raspberry Pi Display für die menschliche Interaktion. Der Automotive Demonstrator besteht aus vier Komponenten, die auf der CE-Box installiert sind: Haupt-QML-Anwendung, Hauptserver, Live-Stream und Smart Queue. Alle diese Server laufen auf drei Einplatinencomputern (Raspberry Pi 3B): Haupt-, BlackPearl- und Kamera-Server. Der Automotive Demonstrator ist mit der neuesten Version von sowohl Qt als auch NodeJS ausgestattet, und die Komponenten können auf die Daten im JSON-Format zugreifen, speichern und austauschen. Der BlackPearl wird über vier Arten von Interaktionsmethoden gesteuert: Webserver, Sprachbefehle (Sparrow), Pi-Display und Gamepad. Das Ergebnis dieser Arbeit ist eine konfigurierbare und anpassungsfähige Benutzeroberfläche für den Automotive Demonstrator, die leicht aktualisiert, angepasst und für neue Anwendungen zugänglich gemacht werden kann, ohne dass das Programm aktualisiert oder neu erstellt werden muss. Keywords: User Interaction, Adaptive UI, Configurable UI, Smart Queue, Raspberry Pi, Qt/QML, NodeJS 3 Content Abstract ....................................................................................................................... 2 Zusammenfassung ...................................................................................................... 3 Content ........................................................................................................................ 4 List of Figures .............................................................................................................. 7 List of Tables ............................................................................................................... 9 List of Code Snippets ................................................................................................ 10 List of Abbreviations .................................................................................................. 11 1 Introduction ........................................................................................................ 12 1.1 Motivation ..................................................................................................... 12 1.2 Technical Background .................................................................................. 13 1.2.1 YellowCar .............................................................................................. 13 1.2.2 BlackPearl ............................................................................................. 15 1.2.3 Qt Framework ........................................................................................ 18 1.2.4 NodeJs .................................................................................................. 20 1.2.5 CMUSphinx ........................................................................................... 23 1.2.6 GTK ....................................................................................................... 24 1.2.7 Felgo ...................................................................................................... 24 1.2.8 Kivy ........................................................................................................ 25 1.2.9 ElectronJS ............................................................................................. 26 1.2.10 Django .................................................................................................... 26 1.3 Problem Statement....................................................................................... 27 1.4 Requirements ............................................................................................... 27 1.4.1 Main requirements ................................................................................. 28 1.4.2 Current vs proposed system .................................................................. 28 1.4.3 Why Qt and NodeJS .............................................................................. 30 2 State of Art ......................................................................................................... 32 2.1 BlackPearl: Extended Automotive Multi-ECU Demonstrator Platform .......... 32 2.2 Adaptive User Interaction ............................................................................. 34 4 2.2.1 A Framework for Adaptive User Interface Generation based on User Behavioural Patterns .......................................................................................... 34 2.2.2 A RESTful Architecture for Adaptive and Multi-device Application Sharing.. ............................................................................................................. 35 2.2.3 Model-based adaptive user interface based on context and user experience evaluation ........................................................................................ 37 2.3 Speech Assistance ....................................................................................... 41 3 Concept .............................................................................................................. 43 3.1 Main Concept ............................................................................................... 43 3.2 Adaptivity and Configurability ....................................................................... 46 3.3 Smart Queue ................................................................................................ 47 3.4 User Interaction ............................................................................................ 48 3.4.1 Remote UI ............................................................................................. 48 3.4.2 BlackPearl Display ................................................................................. 49 3.4.3 Voice UI ................................................................................................. 50 3.4.4 Gamepad ............................................................................................... 51 4 Implementation ................................................................................................... 53 4.1 Main implementation .................................................................................... 53 4.2 Application Programming Interface (API) ..................................................... 54 4.3 Rules Management ...................................................................................... 56 4.4 BlackPearl Server......................................................................................... 57 4.4.1 Main Server ........................................................................................... 58 4.4.2 Web Interface ........................................................................................ 59 4.4.3 Custom App ........................................................................................... 59 4.4.4 Web Interface Settings .......................................................................... 60 4.4.5 Camera Server ...................................................................................... 62 4.5 Smart Queue ................................................................................................ 63 4.6 Application Framework ................................................................................. 65 4.7 User Interface ............................................................................................... 71 4.7.1 Remote User Interaction ........................................................................ 72 4.7.2 Voice User Interaction ..........................................................................