SC1-DTH-03-2018 Adaptive smart working and living environments supporting active and healthy ageing
Smart environments for person-centered sustainable work and well- being
D5.3: sustAGE ICT ecosystem and Integration Report †
Abstract: This deliverable presents the integration report which refers to the release of the 1st Integrated Prototype of the sustAGE system. The work gives important perception towards the release of the final version of the sustAGE integrated prototype, ensuring a smooth and effective integration of all components
Contractual Date of Delivery 31/12/2020 Actual Date of Delivery 20/01/2021 Deliverable Security Class Public Editor Spyridon Vantolas (AEGIS), Vassilis Tountopoulos (AEGIS), Manos Karabinakis (AEGIS) Contributors Maria Pateraki (FORTH), Evangelos Loutsetis (FORTH), Manolis Lourakis (FORTH), Michalis Maniadakis (FORTH), Kostas Papoutsakis (FORTH), Iraklis Varlamis (FORTH), Adria Mallol- Ragolta (UAU), Christos Pikridas (AUTH), Ion-Anastasios Karolos (AUTH), Vito Nitti (IMA), Frank Werner (SAG), Antonio Ascolese (IMA) Quality Assurance Adria Mallol-Ragolta (UAU), Maria Pateraki (FORTH)
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
The sustAGE Consortium Foundation for Research and Technology – Hellas (FORTH) Greece Centro Ricerche FIAT (CRF) Italy Software AG (SAG) Germany IMAGINARY Srl (IMA) Italy Forschungsgesellschaft für Arbeitsphysiologie und Arbeitsschutz e.V. (IFADO) Germany Heraklion Port Authority AE (HPA) Greece AEGIS IT Research UG (AEGIS) Germany Universität Augsburg (UAU) Germany Aristotle University of Thessaloniki (AUTH) Greece Universidad Nacional de Educacion a Distancia (UNED) Spain
sustAGE 2 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Document Revisions & Quality Assurance
Internal Reviewers 1. Maria Pateraki (FORTH) 2. Adria Mallol-Ragolta (UAU)
Revisions Version Date By Overview 1.0 20/01/2021 FORTH Final Version 0.7 19/01/2021 FORTH, UAU 2nd review completed 0.6 18/01/2021 AEGIS 2nd Draft for Internal Review 0.5 15/01/2021 FORTH, UAU 1st review completed 0.4 13/01/2021 AEGIS 1st Draft for Internal Review 0.3 10/01/2021 All contributors Second draft (common editing) 0.2 12/12/2020 AEGIS First draft 0.1 21/11/2020 AEGIS ToC
sustAGE 3 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Table of Contents LIST OF ACRONYMS AND ABBREVIATIONS ...... 6 EXECUTIVE SUMMARY ...... 7 1 INTRODUCTION ...... 8 1.1 PURPOSE OF THE DOCUMENT ...... 8 1.2 INTENDED READERSHIP ...... 8 1.3 RELATIONSHIP WITH OTHER SUSTAGE DELIVERABLES ...... 8 2 1ST INTEGRATED PROTOTYPE OVERVIEW ...... 9 2.1 IMPROVEMENTS BETWEEN THE MVP AND THE 1IP ...... 9 2.2 ADDRESSING FINDINGS FROM THE MVP EVALUATION ...... 15 2.2.1 Technical issues at the sensor level ...... 15 2.2.2 Technical Issues at the sensor level ...... 16 2.2.3 Usability analysis of the smartphone and smartwatch apps ...... 17 2.2.4 Usability of Cognitive Games ...... 18 2.3 HIGH LEVEL ARCHITECTURE ...... 19 2.4 SECURITY ...... 19 2.5 DEPLOYMENT ...... 22 2.5.1 1IP as an Android native application ...... 22 2.5.2 Infrastructure and components deployment ...... 22 2.5.3 Data synchronization ...... 24 3 USE CASE SCENARIOS ...... 25 3.1 SCENARIOS RATIONALE ...... 25 3.2 MICRO-MOMENTS AND RECOMMENDATIONS ...... 25 3.3 MIMO TO RECOMMENDATION WORKFLOW EXAMPLE ...... 29 3.4 1IP USE CASE SCENARIOS ...... 31 3.4.1 Outside workplace scenario ...... 31 3.4.2 CRF use case scenario during work time ...... 32 3.4.3 HPA use case scenario ...... 34 3.5 COMMUNICATION MESSAGES ...... 35 3.5.1 Example message flow ...... 36 4 SUSTAGE SYSTEM COMPONENTS ...... 38 4.1 IOT INFRASTRUCTURE ...... 38 4.2 MONITORING LAYER ...... 39 4.2.1 Multi-source real-time localization ...... 39 4.2.2 Monitoring of user actions ...... 40 4.2.3 Speech and sentiment analysis ...... 42 4.3 DATA STREAMING LAYER ...... 42 4.3.1 sustAGE Bridge ...... 42 4.3.2 Universal Messaging (UM) ...... 43 4.4 PERSONALISATION LAYER ...... 44 4.4.1 Streaming Analytics (APAMA CEP) ...... 44 4.4.2 Knowledge abstraction ...... 45 4.4.3 Multi-modal State and Trait Estimation ...... 46 4.4.4 Knowledge Base ...... 46 4.5 RECOMMENDATION LAYER ...... 47 4.5.1 Temporal Causality and Reasoning ...... 47 4.5.2 Recommendation Engine ...... 48 4.6 INTERACTION LAYER ...... 49 4.6.1 Dialog Manager ...... 49 4.6.2 Dashboard ...... 50 4.6.3 Game based training ...... 50 5 MOBILE APP USER INTERFACE ...... 52 6 INTEGRATION ...... 55 6.1 SUSTAGE DATABASES ...... 55 6.2 SUSTAGE REST API ...... 55
sustAGE 4 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
7 CONCLUSIONS AND NEXT STEPS ...... 58 8 REFERENCES ...... 59
List of Tables Table 1: Sensor/data advancements from the MVP to the 1IP ...... 10 Table 2: Components advancements from the MVP to the 1IP ...... 11 Table 3:Technical issues at sensor level ...... 16 Table 4:Technical issues at application logic ...... 16 Table 5: Privacy and security requirements and solution ...... 20 Table 6: 1IP Micro Moments ...... 25 Table 7: Episode Types introduced in the 1IP ...... 46 Table 8: Rest API functions ...... 56
List of Figures Figure 1: sustAGE high-level architecture ...... 19 Figure 2: sustAGE deployment diagram ...... 23 Figure 3: Message flow diagram ...... 36 Figure 4: Mixed Event Processing Capabilities within the CEP [12]...... 44 Figure 5: Eclipse-based EPL Designer ...... 45 Figure 6: CG Arcade and Challenge modes ...... 50 Figure 7: Cognitive games ...... 51 Figure 8: Screenshots from the mobile app (I) ...... 52 Figure 9: Screenshots from the mobile app (II) ...... 53
sustAGE 5 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
List of Acronyms and Abbreviations
1IP: First Integrated Prototype App: Application CG: Cognitive Game DM: Dialog Manager EAV: Entity–Attribute–Value FLT: Feels Like Temperature HR: Heart Rate HRRest: Heart Rate Rest HRV: Heart Rate Variability KA: Knowledge Abstraction KB: Knowledge Base MiMo: Micro-moment MVP: Minimum Viable Prototype OSH: Occupational Safety and Health RE: Recommendation Engine SSL: Secure Sockets Layer UAM: User Actions Monitoring UM: Universal Messaging
sustAGE 6 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Executive Summary This document presents the work carried out for the development and delivery of the first Integrated Prototype (1IP) of the sustAGE project. The implementation of the 1IP is supported by an operational IoT infrastructure in real-life working environments and the deployment of software modules which are part of the sustAGE solution. Based on the MVP implementation and the results of the first cycle of evaluation, the 1IP is a significantly improved version of the sustAGE prototype, addressing the majority of issues revealed by the first evaluation cycle and introducing new features. Several improvements were integrated by including additional functionalities, either by implementing new features, integrating modules not previously integrated in the MVP or by upgrading and refining already integrated components. More emphasis is given to personal data protection aspects by adopting different privacy and security mechanisms. More specifically, the deliverable is organized into the seven sections outlined below.
Section 1 introduces the deliverable, highlighting its intended readership and relationship to other sustAGE deliverables.
Section 2 presents an overview of the 1IP with emphasis on the achievements since the MVP prototype. It also presents the technical and usability issues revealed during the first cycle of the evaluation and the actions taken to resolve these issues. This section also presents the updated high-level architecture of the sustAGE solution and details the deployment of IoT devices and software components. Finally, it describes the actions taken to ensure privacy and security.
Section 3 describes the details of the use case scenarios selected for the implementation of the 1IP and presents a full sequence of system data samples. The selected use case scenarios guided the definition of new Micro-Moments and advanced personalised recommendations.
Section 4 presents the status and the work carried out for each specific component of sustAGE, highlighting the advancements of each component compared to the MVP implementation.
Section 5 covers the usability improvements made in the user interface of the mobile app based on the MVP evaluation results and it describes the new design of the mobile app.
Section 6 depicts aspects regarding the integration of new features.
Section 7 highlights the overall conclusions and briefly presents the next steps and milestones.
sustAGE 7 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
1 Introduction 1.1 Purpose of the document
This document aims to present the development of the sustAGE 1IP, which is the second delivered integrated version of sustAGE following the MVP (Minimum Viable Prototype) described in D5.2. The 1IP implementation includes all proposed modules and gives important perception towards the release of the final version of the sustAGE integrated prototype, ensuring a smooth and effective integration of all components, considering their availability, interoperability, and performance requirements.
The deliverable addresses the overall work performed to implement functionalities required for the 1IP, using the MVP and the results from the first evaluation cycle as basis for further developments and improvements. It also introduces new components and features, describes the updated deployment plan and outlines the implemented use cases with the associated Micro Moments (MiMos), along with specifications of the messages exchanged among different components.
Results from the evaluation of the 1IP (D6.3) will guide the developments towards the implementation and deployment of the final sustAGE prototype. 1.2 Intended readership D5.3 is a public document (PU) and therefore is intended for the European Commission, the sustAGE Project Officer, the members of the sustAGE consortium, members of other H2020- funded projects as well as the general public. 1.3 Relationship with other sustAGE deliverables
In general, this work will provide insights regarding the implementation and deployment of the sustAGE integrated system. Thus, this deliverable is strongly connected to the work reported in D5.1 “sustAGE Architecture Definition and System Orchestration Specifications” and D5.2 “sustAGE MVP Integration Report”.
Also, this work is guided by the results of the first evaluation cycle reported in D6.2 “MVP evaluation”, and is related to the technologies/tools reported in D7.5 “Initial Draft of Exploitation Activities”.
This deliverable also relates to the deliverables of Work Package 3 (Monitoring and Communication Technologies) and Work Package 4 (Personalized Multi-level Recommendations on Work Ability and Well-Being).
Finally, this deliverable is closely related with the other deliverables of Work Package 5, specifically:
● D5.4 “HCI Knowledge Base”, due in Month 30 of the project ● D5.5 “Game-based Training Activities Base”, due in Month 36 of the project ● D5.6 “sustAGE Final Integration Report”, due in Month 39 of the project
sustAGE 8 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
2 1st Integrated Prototype overview
The 1IP of sustAGE is the first feature-complete integrated version of the sustAGE system. The sustAGE system evolved from the MVP by including additional functionalities, either by implementing new features, integrating modules not previously integrated in the MVP or by upgrading and refining already integrated components. Built on top of the MVP prototype, the 1IP solution addresses the majority of issues revealed by the first evaluation cycle and introduces new features.
The sustAGE MVP represented the initial integrated version of the sustAGE system, which demonstrated the system functionality based on predefined basic scenarios for both pilot sites. The MVP was a simple prototype with minimum functionalities that showcased the potential of the proposed solution by integrating the core system components and validated that the goal is approached in a reasonable manner.
The 1IP implementation follows the architecture described in D5.1 and D5.2, which is further updated to accommodate the requirements of the integrated solution (Section 2.2).
In short, data from the IoT infrastructure, comprising of smart sensors, cameras, and mobile devices, are collected and processed through the lower-level components of the system. These data facilitate the monitoring of different aspects of user actions and environmental variables. Processing at the lower-level system components stays close to the original sources, and privacy-sensitive information (visual and speech data) is not sent to the upper layers of the platform. Data are further consumed by the higher-level components of the system (e.g., Knowledge Abstraction (KA), Recommendation Engine (RE)) and all data and information in the system is streamed in real-time through the Universal Messaging bus, which provides the brokering service for modules communication and interaction.
Based on a number of predefined micro-moments, the modules of the Personalisation Layer process the incoming information, detect key episodes and populate/update the Knowledge Base with user-specific knowledge. The information from the Knowledge Base is then used by the next processing layer to determine proper recommendations according to the dynamic situation (location, time, activity) of the user in real-time. After a recommendation is issued, the user is notified via different channels, i.e., via mobile and smartwatch notifications. The user’s response to the received recommendation is sent back to the system and recorded in the Knowledge Base.
2.1 Improvements between the MVP and the 1IP
The transition period from the MVP to the 1IP included updates on system specifications, end-user requirements and scenarios, which in turn brought updates to the components of the sustAGE system.
Table 1 and Table 2 summarise the advancements concerning the sensors and components from the MVP to the 1IP, in a compact but informative form. The work carried out for each component is described in detail in Section 4.
sustAGE 9 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Table 1: Sensor/data advancements from the MVP to the 1IP
Sensors/Data MVP Status 1st Integrated Prototype Status
Smartwatch sensor Heart rate information was In addition to heart rate, beat-to- data broadcasted to the sustAGE beat intervals, accelerometer Bridge. measurements in the x-, y- and z- axes, and steps information are broadcasted to the sustAGE Bridge. Smartwatch app A network icon was the only In addition to the network icon, element used in the app new feedback elements were providing feedback to the introduced to the app: short users regarding the data vibration when the data transmission status. transmission is interrupted, and symbolic representation of the received recommendations displayed on the screen of the watch. Smartphone Xiaomi Mi9 devices were Additional devices Xiaomi Mi10 used in the experiments. were acquired to support a larger testing group for HPA.
Microphone/speech The speech sample recorded The speech sample recorded is data was directly transferred to the encrypted on the smartphone sustAGE Bridge. Implemented before being transferred to the on a dedicated smartphone sustAGE Bridge. Functionality app. integrated in the sustAGE smartphone app. Environmental Data from temperature, In addition to the data used in the sensors humidity, noise and luminance MVP, data from the wind sensor sensors were transmitted to the are also transmitted for the HPA sustAGE Bridge. pilot. Data from the luminance sensor are only used in the CRF pilot. Camera CRF pilot: Two Stereolabs CRF pilot: Two additional ZEDTM stereo cameras were Stereolabs ZED2TM stereo cameras mounted on portable tripods, of higher resolution and wider installed at fixed locations on field of view will be used. The the side of the production line. four sensors form two pairs of Each camera acquires colour sensors, installed alongside the left and depth-aware visual data and the right area of the
sustAGE 10 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
streams, transferred and production line, while each pair is processed locally by a desktop connected to a single desktop computer via wired, USB computer via wired USB connection. connection. HPA pilot: The Blackfly GigE HPA pilot: No modifications to camera was mounted outside the camera sensor. the crane cabin. Out of the 4.5 mm, 3.5 mm and 6mm lenses, the latter was selected. Localization sensors Localization data from the Transmission frequency of smartphone are transmitted localization data from the every second. smartphone was set to one per minute at CRF and outside workplaces, and one per second at the HPA workplace. Additional u- blox sensors are deployed for camera georeferencing in the HPA pilot.
Table 2: Components advancements from the MVP to the 1IP
Component name MVP Status 1st Integrated Prototype status
Multi-source real-time A separate app for The localization component in the location tracking system localization on the smartphone has been integrated in smartphone. the main sustAGE app. Low-cost GNSS sensors have been assessed for camera georeferencing. User Actions CRF pilot: Assessed and CRF pilot: Implemented a new Monitoring employed state-of-art acquisition module to acquire and methods to perform 3D/2D combine time-synchronised visual pose estimation and tracking data by the cameras from both of the user's body in the sides of the assembly line presence of severe (connected to two different occlusions. A benchmark computers). Implemented a new dataset was compiled that methodology for the visual comprises a set of multi- recognition of four types of modal sensory data abnormal user body postures and sequences (15 sequences). ergonomic risk classification based on the MURI risk analysis HPA pilot: Developed a method. Additional multimodal model-based 6DoF object sensory data (comprising 20 tracker to track the port sequences) was acquired and crane’s spreader. A semantic annotations (ground
sustAGE 11 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
benchmark dataset was truth) were compiled to extend the captured comprising video CRF benchmark dataset. sequences during HPA pilot: trained a CNN with a loading/unloading operation set of manually annotated images of containers. in order to support re-detection in cases the tracking is lost. This aims to support the model-based Functionalities implemented 6DoF tracker by detecting the but not integrated as part of spreader and determining its the MVP. approximate pose in single images.
Speech and sentiment Extraction of paralinguistic Automatic transcription and analysis features and basic translation, when necessary, of the communication with the speech samples received, and UM. Functionalities encryption of the actual message implemented but not before publishing it in the UM. integrated as part of the MVP. sustAGE Bridge sustAGE Bridge was set up CRF pilot: Due to particular for both pilot sites in order security policies and restrictions to receive data from the in place at CRF, a new sustAGE sensors of the IoT Bridge was set up in FORTH in ecosystem and the order to receive data from the components of the users’ mobile devices. Security Monitoring Layer, and mechanisms have been forward them to the UM. implemented in order to secure connectivity with all mobile Raspberry Pi 3 was used for devices. the sustAGE Bridge. Raspberry Pi 4 was used for the sustAGE Bridge.
Universal Messaging All events and message type Configuration of security related (UM) definitions implemented on features (i.e., authentication, TLS separated topics/channels. encryption) are provided.
Streaming Analytics Implementation of two Extension of correlators to (APAMA) correlator modules which support the detection of abnormal relate to user monitoring body postures (including average (e.g., quality of sleep, HR on risk, episodeType and Avg, High HR, localization, occurrences) and component geofencing, geo-state
sustAGE 12 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
localization, user break health/status monitoring. monitoring). Environmental conditions (i.e., FLT, Wind, Noise, Luminance) are monitored and specific averages over 60 and 300 seconds are continuously computed from moving windows.
Knowledge Abstraction Process device activity to A more flexible approach to estimate sleep hours and detect when a user wakes up has time to work for the been implemented based on the individual users. Provide streamed HR data from the watch. input to the RE for the synthesis of relevant Mapping of real-time HR and recommendations. HRV data to appropriate semantic scales.
Inference of user action intensity based on HR data.
Multi-modal State and Implementation of initial Implementation of the initial Trait Estimation Speech Emotion version of the sub-component Recognition (SER) models responsible for the task of Human using the paralinguistic Activity Recognition (HAR) from features extracted by the smartwatch measurements. speech and sentiment analysis component. Functionalities implemented but not integrated as part of the MVP. Evaluation of the SER models was performed off-line.
Knowledge Base Creation of Knowledge Implemented functions to record database, Episode Type more Episode Types. definition and implementation, Entity- attribute-value (EAV) model.
sustAGE 13 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Temporal Causality and Check every morning Implement the Physical Activity- Reasoning whether it is a CG-Day for Day, similar to the CG-Day. the user, notify the RE and Examine the temporal properties set appointments for an of HR and HRV states to infer the afternoon reminder to play fatigue state of the user. CGs. Examine the evolution of posture deviation events to infer how it will proceed in the near future and inform the RE.
Recommendation Handling of UM messages Implementation of the scenarios Engine mainly from Knowledge of the 1IP, which mainly target Abstraction and APAMA, user heart rate condition (to detect and communication with the fatigue), detection of muscular Knowledge Base for stress based on user abnormal retrieving user information, postures (CRF), monitoring safety when it is needed. distances (HPA), and user daily Implementation of the commitment to a schedule of scenarios of the MVP physical and mental training. version, which mainly targeted extreme environmental conditions and user sleep quantity and user arrival at work.
Dialog Manager Decoding and formatting of Improvements to receive the new the recommendations recommendations targeted for the produced by the 1IP and display the symbolic Recommendation Engine. representation of the recommendations to be displayed on the watch. Moreover, updates were made to the information encapsulated in the output message produced by this component.
Game Based Training Basic connection to the UM, Messages and statistics for UM Activities only two test games. and KB. Four new games. New graphical design, multi-language support. Challenge and Arcade modes. Android 10 compatibility. Setting menu, achievements page,
sustAGE 14 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
tutorials.
Dashboard Not available. Connectivity to KB and analytics DB to provide role-based access to recorded data (multivariate data, information extraction, query acceleration).
2.2 Addressing findings from the MVP evaluation
The first cycle of evaluation revealed some technical and usability issues on the sustAGE MVP. Along with comments and observations from the technical partners during the implementation phase, the consortium created a roadmap to resolve these issues towards the release of the 1IP.
2.2.1 Technical issues at the sensor level The accuracy and reliability of the sensors was assessed in real-world conditions over longer periods against reference sensors to detect systematic errors as well as potential noise in the sensor measurements. For the environmental sensors, the differences were within normal limits, although for the HPA case the differences of 1 degree in temperature and 4Rh in humidity are attributed to the outdoor location of the sensors and the local microclimate. For the positioning sensors, the accuracy levels for both the indoor and outdoor cases were consistent with the findings previously observed by other researchers. In the indoor case, the average horizontal and vertical accuracy was estimated at 0.37 m and 0.68 m respectively, whereas, in the case of smartphones for outdoor positioning, 60% of the measured locations were within 5m of their true positions. These aspects need to be further considered when combining localization information with information from other sensors and modules to set limits in accuracy, regarding detection of user states and safety thresholds.
Furthermore, the communication at the lower system level, namely between the sensors and the streaming layer modules, was evaluated considering input/output consistency, latency, potential packet loss, the wireless communication overhead, the scalability potential, etc. From the evaluation performed focusing on the wellness sensor (i.e., the smartwatch) during the MVP, we concluded that there were no critical issues regarding the data transmission when sending heart rate data from the smartwatch to the sustAGE Bridge. However, we detected duplicate timestamps in a small percentage of the packets received, which we plan to investigate and resolve towards the deployment of the first integrated prototype. Also, we detected a Bluetooth timeout firing on the smartphone, which stopped the data transmission. Unfortunately, fixing this issue requires the manual intervention of the users, as the sustAGE smartphone app cannot resolve it automatically from the back end.
sustAGE 15 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Table 3:Technical issues at sensor level
Issue Resolution
Accuracy of positioning sensors Further assessment of the extent of integration with other measurements of higher accuracy.
Duplicated timestamps Upgrade of the back-end of the smartwatch app with temporal buffers storing sensor readings in order to overcome the asynchrony between methods. Furthermore, assigning the timestamp to the current measurements available in a synchronous method.
Bluetooth timeout Implementation of a controlling mechanism in the sustAGE Bridge to detect devices that are not successfully transferring data, so their users can be notified to take a corrective action.
2.2.2 Technical Issues at the sensor level The integration of system components and the application logic was tested against basic application scenarios. Participants at both pilot sites followed a predefined routine for two consecutive days in order to use and assess all the components of the sustAGE MVP system. The assessment revealed different technical issues relating to the storage of user/device data in the database, shortcomings in the localization app and the transmission of the user location to estimate different location states of the user, downtime of specific sensors and modules, and user-triggered updates in the phone and apps that prevented the system from working properly, hindering the detection of MiMos and as a result the reception of the recommendations on the user device. Despite the technical problems, the participants highlighted the following issues as the most important ones to be addressed in upcoming releases of the system: ● Provision of an indication to the user informing when the connection on the smartwatch is working and when not or an alert when there is a disconnection. ● Difficulties in interacting with the smartwatch itself and the smartwatch app. ● Reduction of the number and size of the devices as in some cases these limited the user’s activities (CRF pilot). ● Ease the speech recording procedure, as a dedicated app was required for the MVP. ● Increase of the font size for people with sight limitations. ● Inclusion of information on how to use the system via a “help box”.
Table 4:Technical issues at application logic
Issue Resolution
Shortcoming in transmission of Source code for localization integrated with the main user location, localization as a sustAGE app. separate app
sustAGE 16 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Downtime on specific sensors and Implementation of controlling mechanisms to check modules the integrity of the sensor data received by the system and to check the status of the system components in the architecture.
User-triggered updates Adjustment of the smartphone settings disabling the notifications referred to application updates, and improvement of the instructions provided to the users during the preparatory activities for the evaluation.
Data transmission feedback on the In addition to the connectivity icon, the smartwatch smartwatch app triggers a short vibration to notify users about data transmission issues.
App activation on the smartwatch Improvement of the instructions provided to the users and reserving some time during the preparatory activities for users to familiarise themselves with the smartwatch.
Ease of the speech recording Integration of the developed code in the back-end of procedure the Android native sustAGE app, so it can be offered as an app functionality.
Reduction of the number of Consideration of using armbands for specific devices devices for the CRF pilot in order to reduce the number of devices users should carry in their pockets.
Improve the font size Modified front-end design.
Include information on how to use An information page in the app with instructions and the system via a “help box” FAQs will be included in the 2IP.
2.2.3 Usability analysis of the smartphone and smartwatch apps
The usability evaluation of the MVP prototype aimed at identifying usability-related problems when users use the sustAGE smartphone and smartwatch apps. Three different instruments were adopted to assess individual factors that contribute to the concept of usability, namely understandability, learnability, operability, usability, effectiveness, and satisfaction.
The heuristic evaluation supported the identification of usability problems in the design of the user interface by analysing the outcomes of the participants after freely interacting with the interface. In total, 34 usability problems were identified and those with severity score 4 (the highest score, which indicates usability catastrophes) were prioritized. The task scenarios evaluation assessed the usability of the interface by guiding participants from CRF and HPA to perform certain tasks, while the 4 Likert Scale questionnaire allowed the understanding of users’ perception after using the interface. The last instrument aimed at evaluating the user’s satisfaction.
One of the main concerns raised by the users was the multiple number of apps involved in the MVP system. The main sustAGE app, for the MVP version, was developed using the EXPO
sustAGE 17 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 framework, whereas the speech recording, the user localization and the serious games were implemented in dedicated apps based on different development frameworks (Android Native, Unity and C#). In addition, the Garmin Connect app was required to enable connectivity and data transmission from the smartwatch to the sustAGE Bridge. Therefore, one of the main resolutions during the 1IP implementation phase was to reduce the number of apps and adopt a common development framework.
From the analysis of the usability feedback received during the MVP evaluation phase, we clustered the usability problems into the following categories and highlight the most relevant issues below.
● Front-end design of the smartphone app ○ Small font size ○ Different font sizes and families used throughout the app ○ Inconsistent button design ○ Misleading buttons on the menu bar ○ Unsuitable design of the screen with the history of recommendations ○ Lack of interaction elements providing feedback to the users, such as a soft vibration or an acoustic element when pressing a button ● System status feedback ○ Some users reported that the smartphone and the smartwatch seemed isolated one from the other ○ Users did not receive feedback when there were transmission issues between the smartwatch and the sustAGE Bridge unless they specifically looked at the watch display and realized that the connectivity icon was missing ○ Some users reported that providing the wake-up and go-to-sleep times in the system was not very intuitive ● Recommendations ○ Some users did not find the procedure of rejecting recommendations intuitive ○ Some users did not understand whether the recommendations should be answered by the yes/no buttons ○ Some users were confused with the meaning and the content displayed in the recommendations screen of the smartphone app ● Smartwatch app ○ Problems interacting with the smartwatch ○ Problems to activate the smartwatch app ○ Lack of navigation elements in the smartwatch app ● Speech recorder ○ Feedback on the status of the recording was missing, according to some users
Users’ feedback is the most valuable source of information to solve usability-related issues. Therefore, the different aspects summarised here have driven the implementation and improvement of the sustAGE smartphone and smartwatch apps. The usability-related aspects included and improved in the smartphone app for the 1IP are highlighted in Section 5 of the current deliverable.
2.2.4 Usability of Cognitive Games Both games used in the MVP evaluation (Find-It, Match-It) were rated as being simple and lacking in different levels. It was considered important to increase cognitive stimulation to lead players to improve personal performances instead of repeating the same games, which
sustAGE 18 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 becomes monotonous and dull after a while. Moreover, changes in sound, music and buttons were suggested as well as improving the feedback to the user and supporting a faster response time. The enhancements and extensions of the cognitive games that were implemented for the 1IP are discussed in Section 4.6.3.
2.3 High level Architecture The architecture of the 1IP is based on the version that was reported in D5.2 (Figure 4.1). Guided by the integration process, we have created a new layer, namely the Interaction Layer, which includes the components responsible for the interaction between the end-users and the system. The introduction of the new layer provides better structure and understanding of the architecture. During the MVP implementation, we introduced a listener module that was necessary for some types of sensors to send their data before they were forwarded to the sustAGE Bridge. Conceptually, the listener and the Bridge component have similar roles in the system architecture, so we have decided to depict them as a single module with all the functionality included.
Figure 1: sustAGE high-level architecture 2.4 Security The sustAGE platform handles heterogeneous data sources in terms of integration, sensing and analysis to provide basic learning and reasoning mechanisms to support the system’s personalised recommendation services in the workplace and outside the workplace. Therefore, security and privacy aspects need to be considered for handling personal information.
sustAGE 19 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Considering the need to comply with the General Data Protection Regulation (GDPR) and to ensure that personal data are processed fairly and transparently, the “privacy by design” approach is embedded in the sustAGE architecture and different privacy and security mechanisms are adopted based on the requirements set in D5.1 to guarantee the privacy of personal data and the platform security. Table 5 denotes the implementation of these privacy and security mechanisms in a compact form while more detailed description follows.
Table 5: Privacy and security requirements and solution
ID Requirement 1IP Implementation
PSR1 Authentication and Users are registered to the platform with a device authorisation and remain completely anonymous. Client certificate mechanisms are used to authenticate all the mobile devices connected to the system modules for enhanced security transmitting data to authorized/authenticated recipients only.
PSR2 Access control management Users are assigned specific roles within the platform and permissions on the various functionalities and information are restricted to the defined roles.
PSR3 Personal data dissociation Personal information and end users pseudo names are completely dissociated from any other data collected.
PSR4 Aggregation and Collected or imported data are systematically anonymization of anonymized and no one can access individual experimental data profiles (apart from privileged predefined user roles, e.g., authorized researchers of the coordinating organization - FORTH). Only aggregated, processed and anonymized data are accessible.
All connections between modules and components PSR5 Secure data acquisition and over HTTP or MQTT are encrypted using SSL. exchange
PSR6 Prior informed consent No data collected for users who have not previously provided their written consent.
sustAGE 20 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
PSR7 User-controlled personal Users are always in control of their data and information personal information. Personal settings can change at any time without affecting the overall system operation.
PSR8 Transparent data Data updates, processing and analytics must be management available to users at any time.
PSR9 Database security and Database security practices (max security options encryption of DB Server, firewall protection, regular updates and patches, secure backups in different physical locations) are applied and encryption of data applied to sensitive information (Secure User Data storage).
PSR10 Right to be forgotten End users have the option to request their profile deletion and all their relevant information stored in the sustAGE system.
Authentication, authorization, and encryption Authentication and authorization mechanisms have been implemented to grant permission to system modules and components for connecting to the UM. All connections between modules and components over HTTP or MQTT are encrypted using Secure Sockets Layer (SSL). Interactions between components, as well as authentication at the Dashboard, are secured by SSL, while user account information is held at the Secure User Data storage component, a separate data repository in which all data (i.e., name, surname, username and password of all registered participants) and key associations (pseudoID associations of all study participants) reside in encrypted form. Access to encrypted data is only provided following user authentication based on granted role for access. In this manner, the appropriate level of security is guaranteed. The sustAGE smartphone app provides the users with the functionality to record their own voice. This information is transferred through the Internet, so that it can be received by the sustAGE Bridge. However, speech provides sensible information in terms of users’ privacy and therefore needs to be processed carefully. For this purpose, the smartphone app encrypts the audio file recorded before it is transferred through the Internet, and the sustAGE Bridge decrypts the file upon reception. Client certificates mechanisms are exploited to authenticate all the mobile devices connected to the system modules for enhanced security. Due to particular security policies and restrictions at the CRF premises, all outgoing and incoming connections to the CRF network are forbidden. To overcome this issue, we set up a VPN SSL tunnel between the CRF Bridge and the FORTH Bridge for the data to be transmitted to the UM.
sustAGE 21 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Data collection and anonymisation/obfuscation practices
No data are being collected for users that have not previously given their written consent. Prior to transmitting personal data, data anonymization/obfuscation transformations procedures are applied to ensure that all data (direct and strong indirect identifiers) and metadata references could not make the data subject identifiable. Participants usage data (i.e., IoTs sensors data, mobile app data, aggregated data stored in GDPR compliant cloud) are associated by design with a not identifiable ID, by the use of which no one may identify, directly or indirectly a subject. The pseudo-anonymized data remain as such, while transmitted, analysed, aggregated and stored in the system. Records of IDs associations (i.e., ID - realID) are stored in an encrypted form within the Secure User Data Storage (FORTH). Pseudo-anonymised data are stored within the main cloud repository (SAG). Whenever sustAGE processes personal data, this action takes place in the FORTH computer room (Greece). In no case resulting sensible data leave the EU (FORTH acts as a Data controller). Google Cloud text-to-speech services might reside outside the UE. However, Google Cloud services are GDPR compliant, and we transfer for processing pitch-shifted versions of the original samples with pseudo-anonymised file names to preserve the anonymity of the users.
System Roles and Data Accessibility A Role-Based Access Control (RBAC) mechanism has been implemented to specific user roles (admin, manager, user), limiting the access to the Dashboard services. The system roles and data accessibility are described below:
● Admins: The sustAGE admins will be able to manage end-users from all sites and perform actions such as system monitoring, services check, etc. ● Developers: The developers will have access to their services and the data that these services may need. ● Managers: The managers will have access on all users’ (workers) ID data, processed data and limited access to the raw data coming from the sensors. ● Users (workers) will have access to all the data their devices are displaying.
2.5 Deployment
2.5.1 1IP as an Android native application
Following the MVP, the Consortium agreed on moving the sustAGE smartphone app into an Android-based native application (Section 2.1). By doing so, the sustAGE app can integrate the localisation and the speech recording apps into a single app, reducing the number of applications users need to install in their devices. Additionally, this implementation allows better communication between the different functionalities that were previously implemented in separate applications. This approach also allows the communication between the smartphone and the smartwatch, so the system can provide information to users directly on their smartwatches.
2.5.2 Infrastructure and components deployment
The deployment of the sensing devices and software components of the sustAGE 1IP extends the deployment followed during the MVP implementation to enhance privacy and security,
sustAGE 22 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 incorporates new components and improves the performance on data collection, analysis and storage.
Figure 2 depicts the deployment of the sustAGE devices and components in the work environment, the FORTH hub, and the cloud infrastructure. The arrows between different components show the connections in terms of privacy and security, as described in Section 2.4.
Figure 2: sustAGE deployment diagram
The IoT sensing infrastructure has been installed and configured in the premises of the two pilots along with software modules that process raw data collected by the sensors and need to be close to the source to avoid network latencies before being processed. Each local workstation hosts the Monitoring User Actions module and a local sustAGE Bridge component, which receives data from the vision, environmental and localization sensors and forwards them to the sustAGE Bridge at FORTH. The watch measurements and the speech samples are directly transferred to the sustAGE Bridge at FORTH
The intermediate layer hosted at FORTH includes the FORTH Bridge, which acts as a hub collecting data from both sustAGE Bridges deployed at the two pilots and forwards data. It also hosts the Speech and Sentiment Analysis module and the Secure User Data Storage to ensure privacy of personal data.
The cloud layer consisting of the cloud infrastructure provided by SAG, AEGIS and IMA hosts the UM and the components of the Personalisation, Recommendation and Interaction layers, and the corresponding databases. These components perform analysis on non-personal
sustAGE 23 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 identifiable data from both pilot sites. The data volume flowing in the UM may increase in high levels while the aggregation and statistical analysis of those data require strong computational power and enough storage resources ensured by this cloud-based installation.
Finally, users interact with the system with wearable sensors and apps installed in their smartphone and smartwatch.
2.5.3 Data synchronization In order for the system modules to process the data collected from the sensors and compute higher level information, it is necessary that all these data are synchronised. Data is being sent via different devices (Raspberry Pi, Arduino, Smartphone, Smartwatch, PCs), which generally have unsynchronized internal clocks. To overcome this issue, all devices and modules are synced with a specific Network Time Protocol (NTP) server (namely gr.pool.ntp.org), thus bounding the time difference/offset to a tolerable amount. All devices that have access to the Internet are being synced directly with the assigned NTP server. The devices located within the CRF premises, where there is no access to the Internet, are being synced with the FORTH Bridge. More specifically, the FORTH Bridge, which is synced with the assigned NTP server, is being used as a local NTP server from which the CRF Bridge is synced to. The CRF Bridge is also used as a local NTP server with which the rest of the devices within the CRF segregated network are synced to. Synchronized devices allow the generation of temporally ordered messages which are time-stamped with a number representing the milliseconds elapsed since the UNIX Epoch (i.e., 1st January 1970). The Android OS restricts time synchronisation by not allowing users to select the NTP server to synchronise their device to. Instead, synchronization is entirely dependent on information from the network. To synchronise the smartphone with the rest of the system, the back-end of the sustAGE smartphone app queries the assigned NTP server with a predefined frequency. Statistical processing of the roundtrip time from this query determines the offset between the current NTP and device time, which allows for the compensation of the discrepancy of the latter in the timestamps of smartphone-related measurements. As the smartwatch time is automatically synchronised with the time on the smartphone, the time offset is also transferred to the smartwatch app via Bluetooth and used to synchronise smartwatch measurements with the assigned NTP server.
sustAGE 24 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
3 Use case scenarios 3.1 Scenarios rationale The scenarios selected for the sustAGE 1IP cover both pilot sites (Heraklion Port Authority - HPA and Centro Ricerche FIAT - CRF) and refer to basic scenarios of the workers’ typical workday routine. These use-case scenarios will be further refined and extended throughout the ongoing sustAGE development process. The scenarios are mapped to the workflows initially presented in “D5.1 sustAGE architecture definition and system orchestration specifications”, which describes the main interactions among system components and users that result in recommendations both within and away from the workplace. The general use- case scenarios, as analytically described in “D6.1 sustAGE evaluation strategy and ethics manual”, consist of two domain-specific scenarios for the working context of CRF and HPA and one cross-domain scenario for user activities performed outside the work environment. In brief, the targeted scenario for CRF refers to activities of assembly line workers performing highly-standardized repetitive short tasks. The HPA use case and derived scenario involves port workers supporting the cargo operations at the commercial terminal of HPA. The third cross-domain scenario focuses on activities away from the workplace and targets both CRF and HPA workers, considering their everyday routine and wellbeing. 3.2 Micro-moments and recommendations Based on the selected scenarios and the individual conditions for the port and the factory pilots, a set of micro-moments have been defined, which cover the workers’ day within and outside of the working environment. Each MiMo is triggered by certain measurements and results in issuing the corresponding recommendations. In particular, the continuous monitoring of user activities (e.g., interaction with the mobile phone, going to work, job-task implementation, etc.), user physical condition (e.g., heart rate, heart rate variability) and schedule (e.g., cognitive games’ or physical training schedule), as well as the monitoring of the work environment conditions (e.g., temperature, humidity, etc.) allow the identification of events (e.g., sensor values above critical thresholds) or state changes (e.g., user entering the work area). The propagation of the detected information through the system components collectively triggers the processing of specific MiMos, which result in notifications, recommendations and alerts that are sent to the respective user. More specifically, the MiMos are associated with moments of special interest for the user, usually related to potential user needs, and the respective recommendations are communicated to the user through the most convenient and acceptable channels. Recent user-specific information and statistics (e.g., recent HR or HRV values, body stress values, cognitive game performance, etc.) are critical in this respect, and, to get this information, the Recommendation Engine relies on information stored in the Knowledge Base.
The MiMos are listed in the following table, including the pilot which they refer to, their identifying code, a short description, the information stored in the KB and the associated recommendations/notifications that are generated.
Table 6: 1IP Micro Moments
Pilot MiMo Description Knowledge stored in Recommendations (R)/ Notifications (N) / Alerts
sustAGE 25 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Code KB* (A)
Outside the Work Environment
- Wake up time - Sleep earlier (R) CRF/HPA ON User wakes - Sleep hours - Time until work starts (N) up - Sleep quality - Play Cognitive Game (R) - Time to start work - Perform physical training - Recommendation (R) - Response to recommendation
- Go to Sleep time CRF/HPA OFF User goes to sleep
- Heart Rate - Suggestion to calm/slow CRF/HPA W High heart rate - Avg. Heart Rate down (R) detection - High Heart Rate - Recommendation - Response to recommendation
CRF/HPA CG User is idle at - HR and HRV - Suggest to play a CG (R) home - Date & time of the - Suggest to do a physical training activity schedule activity (R) - Training activity within the day - Weather - Date & time of playing game - Game ID, score, game results, duration
HPA/CRF TEST Perform the - HRRest test schedule - Remind worker to perform periodic the test (for determining his HRRest test heart rate when at rest - HRRest)
In the Work Environment
sustAGE 26 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
- Arrival time - Time until work starts (N) CRF/HPA L1 User arrives at - Minutes delayed - Delayed X times in the last (Constant/f the workplace - Response to Y days (N) ixed recommendation - Adjust sleep time (R) arrival time in CRF / Dynamic arrival time in HPA)
- Leave time CRF/HPA L2 User leaves (Constant/f the workplace ixed leave time in CRF / Dynamic leave time in HPA)
- High/Low Feels Like - Suggest to the shift CRF EnvTemp High/Low Temperature (FLT) manager to check and Temperature - Recommendation adjust environmental detected in the - Response to conditions (R) work recommendation - Recommend workers to environment hydrate and take some rest (R)
- High/Low Feels Like - Recommend workers to HPA EnvTemp High/Low Temperature (FLT) protect themselves from Temperature - Recommendation high/low temp (R) detected in the - Response to - Recommend leading work recommendation worker to pause operations environment due to extremely high temperature conditions (R)
- High wind - Suggest to the shift leader HPA EnvWind Extreme wind - Recommendation to consider pausing speed detected - Response to operations due to extreme in the work recommendation wind conditions (R) environment - Recommend crane operator to stop operations (A)
sustAGE 27 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
- High noise - Suggest to the shift CRF/HPA EnvNoise High noise - Recommendation manager (CRF) or the level detected - Response to leading worker (HPA) to in the work recommendation take measures against noise environment exposure (R) - Alert the worker to move away from the source of noise (A)
- Low illumination - Suggest to the shift leader CRF/HPA EnvIllumin Low - Recommendation or the leading worker to ation illumination - Response to take measures against low level detected recommendation illumination (R) in the work environment
- Heart Rate - Suggest to the shift CRF/HPA W High heart rate - Avg. Heart Rate manager or the leading detection - High Heart Rate worker to recommend a - Recommendation small break to the worker - Response to (R) recommendation
CRF/HPA F Fatigue - Heart Rate - Notify and advise the shift detection - Heart Rate Variability manager to assign easier - Sleep hours tasks (N) - Recommend to the shift manager to stop ops and grant a break - Suggest to the worker to go to bed earlier, to rest in the morning and to relax before going to bed (to promote a better sleep) (R)
CRF Break Break during - Recommendation - Suggest to the worker to work shift - Response to carry out a short relaxation detected recommendation activity (R) - Weather condition - Suggest to play a CG (R) - Break schedule and - Suggest to perform a short duration physical activity (R)
CRF IDLE Micro-break - HR and HRV - Recommend flex activities between cycles - Camera (R)
CRF Wdev Worker is - Camera (vision) - Suggest to the shift having a bad manager or the leading body posture worker to recommend a for a long small break to the worker period (R)
sustAGE 28 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
- Recommendation to the leading worker to modify posture for oncoming cycles (preventive action) (N) - Notify the leading worker regarding the observation of body posture (N)
HPA VL Close - Accident Alert - Alert the dock workers and proximity to a - Recommendation the leading worker to move high-risk zone - Response to away from the dangerous detected recommendation area (A) - Worker localization - Alert the crane operator to - Spreader 3D Oriented check the crane (A) Bounding Box (OBB)
*Measurements from location and environmental sensors are constantly stored in the KB so that components may retrieve information such as the current temperature or latest known position of a certain worker at any time.
An important outcome of this process is the knowledge gained during the execution of the MiMo’s steps. This includes not only sensor-related information, such as localization information, user states and environmental conditions, but also abstracted or aggregated information derived by the KA component, such as user-specific averages (e.g., average sleep time) or aggregates (e.g., number of times that high heart rate was detected). This information is stored in the KB for future reference, mainly by the components of the Personalisation Layer, which need to obtain contextual information in order to improve the proposed recommendations.
3.3 MiMo to recommendation workflow example
To better showcase the series of internal steps triggered by a MiMo, we examine in detail two examples involving MiMo:WDev and MiMO:F.
Steps of MiMO:WDev - Detection of abnormal body postures
1. Body posture-related data from cameras are sent to the User Actions Monitoring (UAM) module. 2. UMA detects abnormal body postures per line-worker, per assembly cycle/station. 3. After the observed cycle is completed, the resulting information is sent to the UM. 4. APAMA aggregates information for a period of 6 cycles/rounds and then for 36 cycles/rounds per worker. 5. Aggregations are stored in the KB. 6. KA receives the 36 cycles value, retrieves the last 6 cycles values from the KB and checks metrics for each body part over stress thresholds and the relevant message is sent to the UM.
sustAGE 29 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
7. The RE receives the “abnormal posture” message and issues a notification regarding the observation of body posture to the worker or a recommendation to the team leader and the line-worker for a micro-break for the worker (=< 5 min) and a recommendation to the line-worker to modify posture for oncoming cycles. 8. The DM receives the recommendation and asks the KB for the latest localization state. Then, the DM selects the Italian/Greek text for the recommendation, checks whether the worker is at work or at home to decide whether to additionally issue the recommendation as a voice-based message, and pushes the recommendation to the mobile app. 9. The user receives a push notification, and a vibration and a symbolic representation of the current recommendation in the smartwatch. 10. The user responds to the recommendation received and the response is stored in the KB.
Steps of MiMO:F - Detection of fatigue level
1. Heart rate values are sent from the smartwatch to the Bridge and are forwarded to the UM every second. 2. APAMA monitors the “heart rate” messages, derives moving averages for the past minute and 5 minute periods from the Heart Rate Variability metric. Heart Rate Variability averages are posted in the UM and registered in KB. 3. The KA compares the real-time data with personalised HR-related information stored in the KB to detect fatigue events. 4. If a fatigue level change is detected, a fatigue event is stored in the KB 5. In case the fatigue level for a user changes from normal to high, the RE is triggered to notify the user and suggest to take a micro-break and do some stretching if he/she feels tired. 6. The DM receives the recommendation and asks the KB for the latest localization state. Then, the DM selects the Italian/Greek text for the recommendation, checks whether the worker is at work or at home to decide whether to additionally issue the recommendation as a voice-based message, and pushes the recommendation to the mobile app. 7. The user receives a push notification, and a vibration and a symbolic representation of the current recommendation in the smartwatch. 8. The user responds to the recommendation received and the response is stored in the KB.
sustAGE 30 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
3.4 1IP use case scenarios
3.4.1 Outside workplace scenario
Cross-domain use case scenario (Outside the work environment)
In the morning before going to work
● The user (worker) switches on the mobile phone and wears the wristwatch. Heart rate information is detected, and the activation time is captured ● The system starts measuring location and time, computes approximate sleep duration and shows average sleep compared to the previous night (statistics) ● The system collects outside environmental metrics from sensors, shows current environmental conditions (statistics) ● The system checks the user’s schedule for cognitive training activity, physical activity, heart rate rest test, and time to start work Notifications/Recommendations ● Recommend going to sleep earlier on this day (provide a reminder in the evening 30 min earlier than the recommended time) ● Notify the user on time remaining to arrive at work ● Recommend to the user to perform a cognitive game activity if the current day is the day for a cognitive training activity (provide a reminder in the evening) ● Recommend to the user to perform a physical activity if the current day is the one for a physical exercise (provide a reminder in the evening) ● Recommend to the user to perform a heart rate rest test ● Recommend to the user to hydrate often if the temperature is high
● Monitor heart rate levels and check for incidences of abnormal heart rate Notifications/Recommendations
● If the user’s heart rate is above normal, notify him/her and ask whether he/she feels ok
Afternoon
● Monitor location, time, schedule, cognitive game performance and physical activity
Notifications/Recommendations
● Reminders to play cognitive games or perform a physical activity
Evening
● Invoke a short dialog if the user was reminded to perform a physical activity to assess
sustAGE 31 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
whether the user performed the physical activity and derive the user’s state
End of day
● The user switches off the mobile phone, turns off the smartwatch app and takes the smartwatch off
3.4.2 CRF use case scenario during work time
CRF use case scenario
Arrive at work
● Measure location and time, access the user’s schedule and determine whether the user is on time, delayed or early Notifications/Recommendations ● Notify the user that he/she should start work in X minutes ● Notify the user - in case of being late - that he/she has been late for X times in the last Y days ● Recommend to the user - in case of being late - to go to sleep earlier this evening
Task-cycle execution in the assembly line
● Monitor temperature, noise, illuminance, humidity and detect outlying conditions Notifications/Recommendations ● Recommend to the shift manager to check and appropriately adjust work environment conditions ● Recommend to the line workers to hydrate, take some rest during their break
● Monitor heart rate levels and check for the incidence of high heart rate events ● Compare heart rate levels with weekly or monthly averages for the same worker Notifications/Recommendations ● Notify the line worker about high heart rate and ask whether he/she is feeling ok (button press to confirm) ● Recommend to the shift manager to provide a short break to the worker with high heart rate or heart rate deviation (i.e., bring a human in the loop to decide on how to best handle the given situation) ● Notify the line worker through the smartwatch to check the mobile device and receive the recommendation for a short break and a proper hydration
sustAGE 32 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
● Monitor the user's heart rate variability as an indication of fatigue Notifications/Recommendations ● Recommend to the worker to rest and sleep earlier (in the afternoon - when at home) ● Remind the user to do physical activities (during the weekend)
● Detect abnormal (suboptimal in terms of ergonomics) body postures (flexion angle of waist, height of working arm, rotation angle of the waist, flexion and stretching angle of the knee) during task cycle execution and deviations from “typical” postures based on the MURI risk analysis method. Derive averages over 6 and 36 task cycle executions per line worker. To be extended to weekly/monthly averages and analysed by the workforce analytics. Notifications/Recommendations ● Notification to the line worker regarding the observation of bad body posture and recommendation to modify posture for oncoming cycles (preventive action) ● Recommend to the shift manager to provide a micro-break of =<5 min to the worker
Idle periods1
● The system estimates workers’ location and also checks outdoor environmental conditions during scheduled short or lunch breaks
Notifications/Recommendations
● Recommend to the worker to relax or perform stretching training ● If the weather permits, recommend to do an outdoor walk for 5 minutes ● If the weather is bad, recommend to stay inside and play a CG
End of work schedule/ Leave work
● Measure location and time, access the worker’s schedule and extract information regarding the time the worker has left the workplace
1 Idle periods are considered to be time intervals between short cycle tasks, at the end of the daily shift, 10 minute breaks or 45 minute lunch break.
sustAGE 33 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
3.4.3 HPA use case scenario
HPA use case scenario
Arrive at work
● Measure location and time, access the user’s schedule and determine whether the worker is on time, delayed or early Notifications/Recommendations ● Notify the user that he/she should start work in X minutes ● Notify the user - in case of being late - that he/she has been late for X times in the last Y days ● Recommend to the user - in case of being late - to go to sleep earlier this evening
During loading/unloading of containers
● Monitor temperature, illuminance, noise, humidity and detect outlying conditions Notifications/Recommendations ● Recommend to the dock workers and the crane operator to hydrate, and rest due to high temperature ● Recommend to the leading worker to suspend operations due to extreme wind conditions
● Monitor heart rate levels and check for the incidence of high heart rate ● Compare heart rate levels with weekly or monthly averages for the same worker Notifications/Recommendations Crane operator ● Notification to the crane operator of high heart rate, ask if the crane operator is feeling ok (button press to confirm) ● Recommend to the leading worker to grant a short break to the crane operator and pause operations ● Notify the crane operator through the smartwatch to check his mobile device and receive the recommendation for a short break and hydration Dock worker ● Notification to the dock worker of high heart rate, ask if he/she is feeling ok (button press to confirm) ● Recommend to the leading worker to grant a short break to a dock worker ● Notify the dock worker through the smartwatch to check the mobile device and receive the recommendation for a short break and hydration
sustAGE 34 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
● Monitor the heart rate variability of the user as an indication of fatigue Notifications/Recommendations ● Recommend to the worker to rest and go to sleep earlier (in the afternoon - when at home) ● Remind user to do physical activities (during the weekend)
● Monitor safe clearance distances (distance of workers from moving containers during their vessel loading/unloading) Notifications/Recommendations ● Alarm to the dock workers, the leading worker and the crane operator when there is a person in the vicinity of the moving crane and within the designated “fall zone” (hazard of cargo/twist locks falling)
During idle time
● The system estimates workers’ location, fatigue level and also checks outdoor environmental conditions during short breaks or the lunch break determined by the team leader
Notifications/Recommendations
● Recommend to the user to relax or perform stretching training ● If the weather permits, recommend to do an outdoor walk for 5 minutes ● If the weather is bad, recommend to stay inside and play a CG
End of work schedule/ Leave work
● Measure location, time, access the user’s schedule and extract information regarding the time the worker has left the workplace
3.5 Communication messages
All sustAGE modules communicate with the system following the message-based communication protocol offered by the UM bus to ensure high throughput of real-time streaming data and ensure scalability for additional needs.
This means that information is not directly sent to specific recipients, but it is instead published as a message to the common UM. Receivers of the message have to subscribe to the UM and wait for such messages to be sent. All components follow this publish/subscribe pattern.
sustAGE 35 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
This communication is facilitated by the use of topics, which allow publishers to publish messages of one category to a specific topic and subscribers to only subscribe to the topics that interest them.
3.5.1 Example message flow
The flow of information among system components for MiMo:F and MiMo:WDev, which were described in Section 3.3, is depicted in Figure 3.
Figure 3: Message flow diagram
A detailed scenario of a CRF worker who strains his muscles during work due to assuming abnormal postures is presented in D4.1 (Section 7.2). Next, we describe the data flow and the exchanged messages for the identification of the fatigue level of a worker, which applies in both the CRF and HPA use cases.
Fatigue Level Identification Example Data from the smartwatch, including heart rate measurements, flow through the bridge to the UM bus. The messages contain information about the current heart rate and heart rate variability of the worker:
{"msgtype": "userheartRateW", "isTest": true, "timestamp": 1608714869005, "userId": 215, "heartRate": 94, "hrv": [693,724]}
sustAGE 36 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
These messages are then consumed in real time by the Streaming Analytics module (APAMA), which aggregates information by extracting averages for the last 60 seconds and 5 minutes. The extracted information is posted to the corresponding UM topic and is consumed by the KB:
{"msgtype": "userheartRateAverageAP", "time": 1608714882125, "userId": 215, "areaId": 0, "episodeType": "AverageHRV60", "value": 712.985}
{"msgtype": "userheartRateAverageAP", "time": 1608714882125, "userId": 215, "areaId": 0, "episodeType": "AverageHRV300", "value": 718.343}
Simultaneously, the KA implements regular, periodic comparisons of real-time data with personalised stored HR-related information to detect fatigue events. As soon as a change in the fatigue status is detected, the information is published in the UM and registered in the KB: {"msgtype": "FatigueLevelKA", "time": 1608714884160, "userId": 215, "areaId": 0, "episodeType": "FatigueHigh", "value": 0}
If the level change indicates a high fatigue level, the Recommendation Engine is triggered to notify the user and suggest an action:
{"msgtype": "userFatigueRec", "timestamp": 1608714885960, "userId": 215, "recomID": 869579858, "recomParameters": "", "recomType": "R_fatigue", recomShort: "R_fatigue", timeframeToaccept:0}
The Dialog Manager receives the recommendation message and formats it to the appropriate notification for the user:
{"msgtype":"userFatigueDM", "timestamp": 1608714885960, "userId": 215, "recomID": 869579858, "recomType": "R_fatigue", "recommendationMessage": "You seem tired. Maybe it is good time to take a micro-break and do some stretching.”, "recommendationTTS": false, recomCluster: "MP", “timeframeToaccept”: 2, “messageExplicitAcceptance": “Are you better now?”}
sustAGE 37 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
4 sustAGE system components 4.1 IoT infrastructure
Smartwatch devices: A major upgrade since the MVP regarding the IoT infrastructure is the expansion of the information transferred from the smartwatch to the sustAGE Bridge. Specifically, while the MVP version only transferred heart rate information to the sustAGE Bridge, the current version additionally transfers beat-to-beat intervals, accelerometer measurements in the x-, y-, and z-axes, and steps information. Due to the asynchrony of the methods provided by the Garmin SDK to read sensor measurements, the back-end of the smartwatch app implements several buffers to avoid the loss of sensor readings. We envision that this multimodal data will help monitor the sustAGE users more accurately and provide richer information to the high-level components of the sustAGE architecture for use in their internal processing and decision-making processes.
During the testing phase of this upgraded version of the smartwatch app, we identified a significant loss of data after running the app for a few minutes. From the MVP evaluation, we knew that there was no significant data loss when transferring only heart rate information to the sustAGE Bridge. Therefore, it seemed reasonable to assume that the current issue was related to the increase of the information to be broadcast. After further investigations, we discovered that this loss was caused by an overflow of a queue in the memory of the device storing the data before this was transferred to the sustAGE Bridge. Additionally, we observed that the time between consecutive POST requests from the smartwatch to the bridge kept increasing as the app was running.
These observations allowed us to conclude that the data transmission issue was caused by the large size of the message to be transferred to the bridge at every request. The Vivoactive 3 performs a character encoding when sending the data to the Internet. Accelerometer measurements were converted into m/s2 units and sent as float numbers, which caused a critical overhead on the number of characters to first encode and then transfer to the sustAGE Bridge. To overcome this issue, we read and encapsulate the accelerometer measurements in milli G units, as directly provided by the sensors of the smartwatch, which allows the encoding and transmission of integers. Fixing this issue, consecutive POST requests from the smartwatch to the bridge are performed steadily, approximately every 3 seconds.
Smartwatch microphone: The microphone of the smartphone is also one of the sensors involved in the IoT infrastructure used in sustAGE. The first integrated prototype of the system integrates the functionality for users to record their own voice in the main sustAGE smartphone app. Furthermore, in order to secure the audio file recorded by the app, it is encrypted before being transferred (sent through the Internet) and finally decrypted when successfully received by the sustAGE Bridge.
Environmental sensors: Sensors for measuring temperature, humidity, pressure, air quality, dust concentration, luminance, and noise that are Arduino and Raspberry Pi compatible were integrated for the MVP. In addition to these sensors, a wind sensor is being used for the HPA use case, measuring the wind speed in meters per second (m/s), which is used as a parameter for calculating the feels-like temperature (FLT), while on the MVP we were only calculating the average temperature.
sustAGE 38 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Based on the measured illumination values from the luminance sensor and in an effort to assess glare aspects via individual person-centred maximum values for illumination in both pilot sites, it was evident that there are intrinsic difficulties to estimate these values based on the overall functionalities of the sustAGE modules and sensors’ setup. Although we monitor general lighting (overall conditions) and not task-related lighting (at the workstation), still the difficulties mainly relate to estimating both the intensity and the direction of the light sources with respect to each individual person and more specifically to where each person is looking at under the assumption of free spaces, thus without obstacles between the light source and the subject. For the measurements to be reliably extracted, one would need additional information: a) set fixed light positions, thus map the light sources in the indoor environment and derive the sun azimuth based on time, date, location in the outdoor environment; b) estimate the humans’ head pose c) create and update maps of the environment in real-time considering moving and static objects along with their material properties. Due to the above complexities, we reside on monitoring minimum luminance conditions for outdoor and indoor workplace environments. Cameras: For the CRF use case, two wide-angle stereo cameras (Stererolabs ZED2TM) were installed in addition to the two existing stereo cameras (Stereolabs ZEDTM). The cameras are installed on both sides of the assembly line to support an improved visual coverage of the workstations with less occlusions. This modification was necessary due to changes in the assembly actions and the replacement of the door mounting trolley with a new bulkier one.
4.2 Monitoring Layer
4.2.1 Multi-source real-time localization
Indoor positioning: For indoor positioning of users in CRF’s pilot area, a real-time location tracking system has been developed for the MVP. This system uses special Ultra Wide Band Beacon devices which transmit signals that allow the computation of 3D coordinates and associate accuracies via trilateration algorithms based on Time Difference of Arrival (TDoA).
Outdoor positioning: For outdoor positioning, the smartphones carried by the users are used. The smartphones selected for sustAGE can receive signals from multiple navigation satellite constellations currently in operation, such as GPS, GLONASS, Galileo and Beidou. This achieves increased position accuracy and robustness against interferences. Smartphone location when outdoors is obtained from multi-GNSS measurements that are processed with algorithms developed for the MVP. This functionality has been integrated into the app used for the first integrated prototype.
An additional issue concerned the localization frequency which was automatically changed on certain occasions. Specifically, in the CRF use-case, when a worker enters the CRF work environment, the system automatically adjusts the frequency of the messages sent to the sustAGE Bridge in order to reduce battery drainage.
Highly accurate outdoor positioning for georeferencing: Camera georeferencing for the HPA use-case is essential in order to align the camera’s local coordinate system with the ground coordinate system employed by GNSS receivers. Towards this end, we combined exterior orientation from a known target with incremental pose changes inferred from accurate multi-GNSS positioning [1]. For the latter, we chose a low-cost device, specifically the u-blox ZED-F9P high-precision receiver. ZED-F9P is a multi-band GNSS module with
sustAGE 39 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 integrated u-blox multi-band Real-time Kinematic (RTK) technology for centimetre-level three-dimensional accuracy. It comes with built-in support for standard RTCM corrections, supporting centimetre-level navigation from local base stations or from virtual reference stations (VRS) in a network positioning setup. For that reason, the Networked Transport of RTCM via Internet Protocol (NTRIP) application level protocol was used in the present study. Hence, GNSS data are easily distributed from a PC server to many GNSS users who have Internet access through a mobile IP network (e.g., GSM, GPRS) or via local WiFi networks. An NtripCaster server streams RTCM data in various versions via the Internet, using a network of permanent GNSS stations covering a large part of Greece including the study area of the Heraklion port. The GNSS module was connected with a Raspberry Pi kit in order to be properly configurable for the application demands. The RTK technique was implemented for position estimation. This methodology relies on a number of permanent reference base stations. Therefore, the RTCM data (version 3.1) was retrieved (via NTRIP) from the nearest GNSS base station, which is equipped with a high-quality geodetic receiver and antenna. Thus, each timely position coordinates, derived by the used multi-band GNSS receiver, delivers centimetre-level accuracy in seconds (fix solution) by combining GNSS signals from multiple frequency bands (L1/L2/L5). The estimated coordinates in easting, northing (E, N) axes are expressed in the TM87 Transverse Mercator cartographic projection which is the official projection of the Greek national geodetic system HGRS87 according to proper transformation parameters.
4.2.2 Monitoring of user actions This component is responsible for detecting typical behaviour patterns and action/events of users in the work environments, in a non-obtrusive manner using visual data acquired by four stationary stereo cameras (CRF pilot) and a single camera mounted on a moving harbour crane (HPA pilot). Prior information from the real-time user location tracking component (Sec. 4.2.1) could possibly be fused with vision-based extracted information to enhance the performance of this system module. However, further investigations need to be carried out concerning the accuracy of localization both in the indoor and outdoor cases.
Based on the analysis of the two end-user work scenarios initially described in D6.1, we formalized the monitoring of user actions for the CRF pilot around the concept of detecting suboptimal performance during assembly task execution towards reducing the ergonomic risk and improving occupational safety. Regarding the HPA pilot, the focus is on improving occupational safety through monitoring of critical areas, moving containers and workers during loading/unloading operations. Of main interest is the real-time tracking of the port crane’s spreader2 in order to define a threat volume in the form of an oriented bounding box around it. Estimating the distance to the safety volume from the location of each worker allows to determine whether safe clearance distances are maintained from the spreader and the container. Worker locations are provided by the GNSS receivers integrated in their smartphones. A spreader or moving container that moves too close to a worker triggers alerts directed to both the particular worker and the crane operator.
For the CRF pilot, we exploit data computed by vision-based 3D body pose estimation and tracking methods to perform recognition of abnormal (suboptimal in terms of ergonomics) user body postures during the production phase of the manufacturing process. The recognized
2 A spreader is a device used for lifting containers. It is placed between the container and the lifting crane and has four locking mechanisms that attach to the corners of the container.
sustAGE 40 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 body postures are classified against medium or high ergonomic risk aiming at the elimination or to lower the MURI factor (body postures or activities that indicate increased ergonomic risk that further imposes stress on the musculoskeletal health of workers) according to the Lean manufacturing functions and production system [6,7]. The method developed relies on spatio-temporal Graph Neural Networks models combined with the soft-DTW technique for differentiable temporal alignment of data sequences or time-series. The model is trained for human body posture detection and classification using the well-known datasets of TUM [8], UW-IOM [9] and a new dataset of abnormal body postural assessment compiled by FORTH and CRF based on a series of image sequences captured at the CRF workplace (currently still under development). Moreover, detected deviations of the duration among task executions during the same or consecutive working days will be considered and analysed. This method relies on a semi-supervised recognition scheme based on temporal co-segmentation and alignment of long sequences of actions and attention mechanisms. We focus on the analysis of the three users for assembly task cycles. As part of the 1IP, the following aspects were considered: augmentation of the camera system with two additional wide-angle stereo cameras (Stereolabs ZED2TM) to the two existing stereo cameras (Stereolabs ZEDTM), testing and resolution of camera placement task within the CRF working environment in order to maximize the visual observation/coverage of areas with respect to all three workstations of the production line, resolution of connectivity, data transmission, time synchronization and storage performance issues for the visual data streams acquired by the four cameras connected to two different computers, acquisition of a series of multimodal data sequences from the actual operational environment to facilitate the development and testing of novel vision algorithms, and testing of messages to be sent to the UM using actual results from captured data sequences. The first version of the underlying algorithms as part of the component functionality is available in the first integrated prototype.
A series of video sequences were acquired during the assembly of car doors in all three of the working stations using different users, including both reference (normal) sequences and sequences with different types of anomalies in the task execution.
For the HPA pilot, a camera with a narrower field of view was installed on a mobile harbour crane and test videos were acquired during vessel loading/unloading of containers from the actual operational environment. As part of the 1IP, a model-based 6 DoF (orientation plus translation) object pose tracker was developed. This tracker is applied to tracking the port crane’s spreader. As the spreader lacks discriminant features, tracking is based on straight line segments. Provided with a 3D mesh model of the spreader and an approximate pose, the tracker renders the model, removes lines corresponding to hidden edges and matches them against the line segments detected in an image. Then, the differential pose that best aligns the two sets of lines is estimated using robust regression techniques and finally used to provide a correction in the spreader’s pose for use in the subsequent frame.
Additional efforts were directed towards detecting the spreader and determining its approximate pose in single images to be used for bootstrapping the tracker in cases where tracking is lost. To this end, a CNN was trained with a number of manually annotated images and experiments indicated that a 2D bounding box containing the spreader can be determined when at least 40% of the spreader is visible. Further investigations included training the CNN for regressing 6D pose parameters based on image samples from the T-LESS dataset [2]3
3 http://cmp.felk.cvut.cz/t-less/
sustAGE 41 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 captured from a systematically sampled view sphere with 10 deg step in elevation and 5 deg step in azimuth.
Further efforts focused on the derivation, implementation and validation of a methodology for continuously georeferencing the moving camera that is rigidly mounted on the crane. This step is essential in order to align the camera’s local coordinate system with a ground coordinate system such as the one employed by GNSS receivers. The methodology relies on a set of visually distinct ground control points to georeference the camera at a reference location, combined with a stream of position measurements acquired from a triplet of compact, low-cost RTK GNSS receivers (cf. Sec. 4.2.1). By combining exterior orientation from a known target with incremental pose changes inferred from accurate multi-GNSS positioning, the full 6 DoF pose of the camera is updated with low processing overhead and without requiring the continuous visual tracking of ground control points.
In both HPA and CRF cases, the image data transmitted by wire is processed by the respective module in the local PC and the extracted information is forwarded to a listener sub- module as part of the bridge functionality. Raw pixel data remains always at the local PCs.
4.2.3 Speech and sentiment analysis
This component first automatically transcribes the speech samples recorded by the users into text. To achieve this, we use the off-the-shelf Automatic Speech Recognition (ASR) service provided by Google Cloud. Before sending the speech sample to the Cloud, we apply pitch shifting to the original audio file in order to preserve the anonymity of the users. This process is language-dependent. Hence, the speech and sentiment analysis component first checks the ID of the corresponding user, which is embedded in the file name of the media file sent from the smartphone, and then decides the configuration parameters required to invoke the ASR. Greek and Italian transcriptions are automatically translated into English, so these can be fed into English sentiment analysis models. In this regard, initial sentiment analysis models have been trained using the CMU-MOSEI dataset [3], comparing the performance of hierarchical networks with and without attention mechanisms.
Additionally, this component also extracts paralinguistic features from the raw speech samples, so these can be fed into the speech emotion recognition models, which reside in the multimodal state and trait estimation component in the overall sustAGE architecture. The mechanism to transfer these features from one component to the other is by publishing them to the UM. Furthermore, the transcribed text is also encrypted and published to the UM, so it can be retrieved by the DM. 4.3 Data Streaming Layer
4.3.1 sustAGE Bridge The sustAGE Bridge is a component that receives data gathered from all sensors of the IoT ecosystem (sensors and Monitoring Layer) and forwards them for further processing via the UM. In the MVP version, the sustAGE Bridge functionality relied on the open-source Home Assistant software. Since Home Assistant did not provide support for implementing custom methods that we needed to use, we implemented our custom modular gateway based on the Raspbian OS of the Raspberry Pi. The hardware was upgraded to Raspberry Pi 4, enhancing
sustAGE 42 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506 the overall performance of the bridge. Enhanced security mechanisms have been implemented for all devices that connect to the bridge to send data encrypted.
Due to particular policies and restrictions for the CRF case and also taking into account that users’ sensitive personal data should be transmitted to devices that are located within FORTH premises (FORTH acts as a Data Controller), another bridge was set up at FORTH. The FORTH Bridge is used to a) receive data from the CRF Bridge and forward them to the UM and b) to synchronise the CRF Bridge with the assigned NTP server, since the CRF Bridge is not allowed to have access to the Internet. The two bridges are connected via an SSL VPN tunnel provided by CRF. Mobile devices (smartphones and smartwatches), from both use cases, are now sending data to the FORTH Bridge.
4.3.2 Universal Messaging (UM)
Event handling in sustAGE relies on a messaging paradigm which is different from conventional, service-based architectures as it permits high-speed, low latency and high throughput, avoiding time-consuming communication handshakes. Messaging operates on a TCP/IP enabled network and is mainly supported by the messaging bus, an intermediary that supports multiple communication interfaces, each one defined by a protocol and a port. As messaging bus, the Universal Messaging4 from Software AG is used, which acts as a central component within the sustAGE architecture. Message transport relies on transfer of data through a common channel in an asynchronous manner. Messages are sent from an application without knowing if the receiving application is up and ready at the same time. In the same way, an application that was not connected to the messaging system for whatever reason, might receive a message, which was sent some time ago right after it successfully reconnects to the messaging system. Asynchronous messaging strengthens the decoupling of the applications and allows integrators to choose between broadcasting messages to multiple receivers, routing a message to one of many receivers, or other topologies as they see fit.
A messaging event is the object that is published to the messaging bus (channel, queue, peer- to-peer, or data group) and is passed on to subscribed consumers as required. An event may be a simple byte array or contain more complex structures, such as complex data structures, dictionaries, Google Protocol Buffers, JSON, or JMS events. The underlying messaging model is publish/subscribe [10], an asynchronous messaging model where the sender (publisher) of a message and the consumer (subscriber) of a message are decoupled. The two simply share a common channel/topic. The publisher publishes data to the channel, which exists within the messaging bus. As messages arrive on a channel, the server automatically sends them onto all consumers subscribed to the channel. The messaging system involves aspects such as a set of Message Channels/Topic (the way that two applications can message each other), Message Endpoints (connectivity to a message channel), and Message Body (actual pieces of information exchanged among applications). As the message body, a well- defined and known JSON structure has been individually chosen. Message events are handled by the message bus via topics to separate data from different sources into logical channels rather than using filtering. In addition, separated data can utilize DataStreams for a single target, which facilitates the communication between different platform modules but also reduces the communication overhead as modules only receive data relevant for their proper operation.
4 https://www.softwareag.com/corporate/products/az/universal_messaging/default
sustAGE 43 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
4.4 Personalisation Layer
4.4.1 Streaming Analytics (APAMA CEP)
The Streaming analytics component is provided through a CEP (Complex Event Processing) engine which is directly attached to the sustAGE messaging bus to retrieve real-time events. In sustAGE, APAMA5 from Software AG is used as the streaming analytics engine as it is able to filter, aggregate, enrich and analyse a high throughput of data streaming event sources to identify simplex and complex patterns to visualize business and factories in real-time, detect urgent situations, and automate immediate actions [11]. The streaming analytics components consist of two modules, which is an editor – to define the rules in EPL (Event Process Language) – and the Complex Event Processing Engine (CEP) for interpreting EPL at run time and correlating events. The correlator parts hereby act as a container for EPL applications that can be injected and removed dynamically at run-time without disrupting other running applications. EPL rules injected into the correlator define events and patterns of events that an application is interested in. These are passed to internal units within the correlator for high-performance execution. Unique event patterns can be modified programmatically and there is no need for fixed-stream networks. The application logic is executed when event patterns are matched, like a traditional “call-back” or “listener”, in which the application code can perform any operation, including generating output events.
At the highest level of abstraction, there is incoming data, a query to execute over that data and an index to be used to speed up the query. Rather than applying the index to the high volumes of changing data, like in traditional systems, the CEP instead applies the index to the query. That is, the query is deconstructed, and its elements are used to update an index. Now, when subsequent data is received, it is immediately compared to the index to execute the query; the data does not need to be stored, no indexes are updated, and there is no need for a secondary trigger. This creates a simpler and more efficient architecture that is suited to event processing of big data in motion. Following this notion, the CEP supports a wide range of mixed events that range from discrete event matching, streaming aggregation, and detection of long-running event patterns. Acting on this event patterns, derived aggregates and computations results (that is, the sustAGE events) are in parallel published to the messaging channel for further processing by other sustAGE components.
Figure 4: Mixed Event Processing Capabilities within the CEP [12].
5 http://www.apamacommunity.com/
sustAGE 44 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
The CEP tool incorporates the use of EPL designed specifically for event driven applications to offer a native capability to use event attributes, their time of occurrence, and any relationship among the events as fundamental programming constructs. In scenarios with fast moving data, EPL offers clear advantages of the structured relational algebra of SQL, which focuses upon transactional data requirements. In support of designing and writing CEP applications and services, an Eclipse based designer is provided for development, debugging, testing, profiling, back testing and deployment, as shown in the figure below.
Figure 5: Eclipse-based EPL Designer
4.4.2 Knowledge abstraction
The Knowledge Abstraction (KA) is responsible for transforming continuous flows of sensory information into compactly described knowledge items. The KA is directly linked to the UM bus, thus receiving in real-time information from all the environmental and physiological sensory sources. The information is accumulated and passed through knowledge extraction filters that determine the occurrence of MiMo events. The output of the KA is sent back to the UM, which enables the storage of extracted MiMos to the Knowledge Base (KB) and additionally forwards the information to all modules “listening” to the Knowledge Abstraction topic for further processing.
The KA considers two levels of knowledge extraction filters. At the lower level, the univariate or single modality filters are implemented based on statistical aggregation metrics, rules applied on sensory values to capture changes in the state of users, and univariate time- series features that are algorithmically extracted from sensory flows. At the higher level, the KA considers the combinatorial abstraction of multivariate sensory flows. This is supported by Machine Learning algorithms that cluster low level states to develop higher level representations encoding multi-modal information that may correspond to different contexts for the user-environment pair. The transitions between high-level representations are expected to reveal typical paths of how the wellness of users evolves over the day and how it is correlated with locations, environment states and certain times of the day.
sustAGE 45 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
4.4.3 Multi-modal State and Trait Estimation
The publication of the paralinguistic features extracted from the speech and sentiment analysis component in the UM triggers this component. These features are then fed into the speech emotion recognition model in order to infer the emotional state of the user. This information is finally sent to the UM, so it can be retrieved by the DM or other components interested in the current emotional state of the user. Current efforts in this component are focused on investigating the optimal speech emotion recognition models, with their features and internal configurations.
In this component also resides the human activity recognition model, which aims to automatically recognise the activities users are doing by exploiting the information sensed with the smartwatch. For tackling this problem, we will mainly use accelerometer information, but we will also investigate the suitability of fusing this data with heart rate or steps information. The outcomes of this model will be published in the UM with the aim to help the Temporal Causality and Reasoning (TCR) and the Recommendation Engine (RE) components to enrich the decision-making process regarding when to issue new recommendations to the users.
4.4.4 Knowledge Base
The Knowledge Base (KB) is the mechanism that collects and stores in the Knowledge database data related to the workers and their environment. The KB reads information flows in the UM and populates the database with episode types. Episode types are predefined types of events that indicate state-changes of the users or the environment.
Episode types contain high-level information derived from the processing of data gathered from the sensing infrastructure. This information represents the captured user action, behaviour or other relevant information, while at the same time hides all low-level technical details regarding how this information was exactly acquired. Episode types provide the basis for the development of personalised recommendations.
Following the development process towards the final release of the sustAGE solution, new episode types will be added. The representation of these events in the Knowledge Base follows the Entity–Attribute–Value (EAV) model, allowing the incorporation of new types of events without the need to modify the database structure.
In addition to the initial list of episode types defined for the MVP implementation, new types are added to represent different or more complex user states for the 1IP release. Table 7 shows the episode types added in the KB after the MVP.
Table 7: Episode Types introduced in the 1IP
EpisodeType Short Description
HR_vLow Very low heart rate detected
sustAGE 46 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
HRV_vLow Heart rate variability
HRV_Low
HRV_Normal
HRV_High
HRV_vHigh
Fatigue_High Fatigue level detection
Fatigue_Low
NoFatigue
Posture_riskHigh Abnormal posture detection
Posture_riskMedium
Posture_riskLow
4.5 Recommendation Layer
4.5.1 Temporal Causality and Reasoning
The sustAGE system interacts with users for long periods of time to observe their behaviour and physiological state at different times of the day, in different conditions and different contexts, which are further abstracted to memorable items and stored in the KB with the help of the KA module. The Temporal Causality and Reasoning (TCR) module exploits the information stored in the KB to adjust the ordinary performance of the composite system to the personalised characteristics and needs of the users.
The TCR uses the previous data in three different ways in order to adjust the current performance. These are related to:
• the implementation of personalised reasoning mechanisms that take into account the historical data of the individual users and additionally the evolution of their actions and their physical/mental condition along the day, • the prediction of situations in which a user may be in the near or mid-term future, in order to foresee uncomfortable situations and potential health risks for the given user, and • the timely optimised implementation of the interventions proposed by the system, adequately coordinated with the daily program of users in order to not interfere with their activities and enhance the user acceptance of the sustAGE recommendations.
sustAGE 47 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
4.5.2 Recommendation Engine The role of the Recommendation Engine (RE) module is to provide actionable recommendations that improve safety and well-being at work to individual workers and their supervisors. The module takes advantage of the input messages that arrive to the UM from the KA and the TCR modules, which provide information about the current and predicted user states, extreme environmental conditions that may affect user performance or health status, and send the recommendations at the right moment. The RE delivers three types of recommendations: ● notifications, which mainly refer to low priority actions that do not require immediate response, but relate to scheduled training activities (physical or cognitive training), to sleep quality, etc. They increase workers’ awareness about their progress in the scheduled training activities, remind them to hydrate on a hot day or avoid prolonged exposure to low temperatures on a cold day. ● recommendations, which refer to medium priority actions that require a user response in a short-term period. They are usually linked to reminders for performing an activity in their idle time, to avoid exposure to hard environmental conditions (noise, temperature, humidity, etc), to avoid musculoskeletal stress from bad body postures, fatigue, etc. ● alerts, which refer to high priority actions that require immediate user response, mainly for safety purposes. A heart related incident, such as high heart rate or unsafe working conditions (e.g., extremely high temperature or strong wind, presence in the vicinity of a moving container) are some of the alert recommendations addressed to workers. Most of the alerts are also sent to the shift managers or specific workers related to the risk (e.g., the crane operator on a windy day), in order to initiate a break. In order to avoid repeating low or medium priority recommendations in a short time, the recommendation engine consults the Knowledge Base, before sending a recommendation. It checks for the most recent recommendation of the same type sent to the same user and sends again only when a time threshold has passed. The communication with the KB API allows the RE to retrieve the daily activity and shift schedules of the worker, as well as the daily breaks, the recent progress in training, etc. Recommendations are sent to the UM and handled by the Dialog Manager, which also invokes explicit user feedback, in order to validate that the recommendation has been accepted or was helpful for the user. The MVP implementation has been refined in this 1IP by adding more recommendations related to daily shift and break schedule, as well as to the weakly physical and cognitive training of the workers. Body posture and heart rate monitoring allows to detect body stress and fatigue in this version and issue recommendations and alerts to the workers, as well as to notify their supervisors. From an implementation point of view, a set of rules is used to trigger the respective recommendations. The rules employ thresholds which are similar for all workers in the 1st integrated version and the partners are working on the dynamic personalisation of these thresholds based on workers’ response to the recommendations, using a continuous learning approach.
sustAGE 48 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
4.6 Interaction Layer
4.6.1 Dialog Manager
This component is invoked when the system needs to communicate with the users, either to issue new recommendations or to notify about problems detected with the sensors, for instance a connectivity loss between the smartwatch and the sustAGE Bridge. The DM is therefore mainly triggered by the output of the recommendation engine, which includes an encoded version of the recommendation to issue to the user. The reception of messages from the recommendation engine triggers several processes in the DM. First, the DM extracts the encoded recommendation from the JSON-based message received, and queries the KB through a GET method to retrieve the corresponding sentence, or short text, that needs to be communicated to the user. From this GET request, the DM receives the decoded version of the current recommendation in English, Greek and Italian, the priority level associated with the current recommendation, and the follow-up question, if any, to know whether the recommendation was helpful and the time to wait until issuing it. Subsequently, the DM also extracts the ID of the user to whom the current recommendation is targeted to. With this information, the DM queries the KB one more time through a GET method to determine the preferred language of the user. This information is used to select the language with which the recommendation will be finally displayed to the user. Lastly, the DM includes specific parameters determined by previous components in the architecture, if any, in the recommendation itself before being issued to the user.
The DM is also responsible for determining the communication means by which recommendations are issued to the users. As the main modality, users will always receive the recommendations in a textual form in their smartphones. Furthermore, we see advantages on issuing the recommendations on the smartwatch too. For instance, during working hours users have several tasks to accomplish, and being regularly interrupted for reading the recommendations on their smartphones can be disturbing. In order to overcome this issue, we map each recommendation to a specific priority level. When the DM issues a specific recommendation to the user’s smartphone, the back-end of the sustAGE smartphone app forwards the associated priority level to the smartwatch app. If the recommendation received is classified as low priority, the watch vibrates and the display turns green for a few seconds. If the recommendation received is classified as medium priority, the watch vibrates and the display turns yellow for a few seconds. Finally, if the recommendation received is classified as high priority, the watch vibrates and the display turns red for a few seconds. With this mechanism, we aim to help users filter the recommendations they receive. If the current recommendation has a low priority, users do not need to check the recommendation immediately and therefore can continue with their working duties. However, if the current recommendation has a high priority, users should pause their current activity and check immediately the recommendation the system provided. Additionally, when users are at home, which is checked by querying the KB and retrieving the current location state of the user, the recommendation will also be provided as a voice-based message using the Text-To-Speech (TTS) functionality available in Android.
Towards the second integrated prototype, we plan that the DM will exploit the outputs of the speech and sentiment analysis, and the multimodal state and trait estimation components by engaging users into short empathetic interactions. To support the natural language understanding task, we will use the RASA open-source tool [4].
sustAGE 49 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
4.6.2 Dashboard The sustAGE Dashboard is a user interface that provides different views and visualisations on workforce status and related key performance indicators (KPI) at one glance. It is designed to provide the link between stakeholders of sustAGE and their need for different insights of the platform. It enables a dynamical view on metrics, performance, safety and human factors relevant to optimise workforce performance along with data projections. The Dashboard enables visualisations within sustAGE and consists of three components for end-user visualisation (Grafana), data caching (Workforce DB) and intermediary data computations (workforce analytics runner). The visualisation component is based on Grafana, a free and open dashboard solution that provides different visualisation diagrams and options and a role-based access. Hereby dynamic and reusable dashboards are feasible that support flexible visualisations with a multitude of options. The workforce analytics runner component collects information from all available information sources and databases. Based on this data, it is capable of executing scripts and generating enhanced insights which are otherwise not feasible through the streaming analytics or mere database views. These are foremost visualisation tasks that involve complex computations or the use of multiple side-constraints. The runner script is also capable of querying the machine learning components from the predictive analytics module which complements the knowledge from the data streams and database views by considering certain patterns which may have an impact on future trends. As these computations may take up to several minutes, the workforce analytics runner is periodically but asynchronously executed, like a batch script.
The Workforce database is used as a visualisation cache and keeps temporary data generated by the workforce analytics runner script, predictive analytics module or simply short-term, intermediary results. The Workforce DB component is only accessible through the Dashboard and filled with data by a workforce analytics runner job, which frequently queries the databases and computes queries which cannot directly be read from the database.
4.6.3 Game based training This component comprises a set of mini games designed and developed to train the cognitive skills that usually decrease with age. The idea behind this tool is to use the motivation and engagement typical of games to increase compliance of users when they have to follow a personalised training plan. Clinical requirements and main aspects of the design were produced in activities included in WP2. The app is installed on the Android devices as a standalone app, completely dedicated to the games. Nevertheless, we are in the process of functionally integrating it with the whole system, which will allow the game to adapt to each single player their real skills and needs, possible improvements, etc. The app presents two different game modes, namely “challenge” and “arcade”.
Figure 6: CG Arcade and Challenge modes sustAGE 50 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Currently, there are six available games (Find it, Match it, Get your bearings, Crazy matrix, Creative sequences, Grab and dash) in both these sections (Figure 7). The two first games, already available in the version tested during the MVP, were modified in gameplay and graphical aspects according to the feedback collected during the MVP evaluation. Furthermore, starting from requirements presented in D2.3 “Mental training specifications”, four new games were designed, developed and integrated in the same app. In the “challenge mode” each user can find, three times a week, only the recommended games (maybe just some of the six), in a recommended difficulty and with the recommended parameters (e.g., speed of items on screen, number of distractors, etc). In the “arcade mode”, available just for some minutes a day, users can find all the 6 games and they can decide which one(s) to play. In this game mode, the difficulty can change during the match, as in an endless game, becoming gradually more complex. Tutorials and achievements were also implemented with the aim to make the app easier to use and more engaging.
Figure 7: Cognitive games
sustAGE 51 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
5 Mobile app user interface
Several design improvements have been introduced in the front-end of the version used for the first integrated prototype with the aim to address the usability problems identified by the users during the MVP evaluation. For the 1IP, a completely new smartphone app has been designed and implemented. The reasons for this update are twofold: i) the conversion of the app into a fully native Android app, and ii) the inclusion of new usability elements aimed to resolve the usability problems encountered by the users during the MVP evaluation.
We first removed the menu bar and re-designed the home screen of the app. The home screen (Figure 8.a) now contains new, visually appealing and self-explanatory buttons as the main elements of the home screen. Each button offers a specific functionality to the users. Specifically, displaying the history of recently received recommendations, a summary of the results obtained after playing the CGs, the microphone button, checking the measurements from the environmental sensors, their profile information, and their heart rate.
Figure 8: Screenshots from the mobile app (I)
Compared to the smartphone app used for the MVP, in the new app we have removed the activity page, which was reported as problematic and confusing, as users’ wake-up and go-to- sleep times are now automatically detected based on the stream of data from the wristwatch. Therefore, users do not need to report their wake-up and go-to-sleep times manually.
In the heart rate dedicated screen, in addition to displaying the current heart rate of the user, we designed and implemented a mechanism by pressing a button for users to follow a resting procedure. Pressing this button triggers specific mechanisms in the architecture to update the baseline heart rate measurements for the current user. This screen also provides instructions to help users undertake the baseline measurement successfully (Figure 8.b).
sustAGE 52 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
The profile page has been updated giving the option to the user to select preferred language between English, Italian or Greek (Figure 8.c).
When users press the microphone button, a toast appears at the bottom of the screen to let the users know that the recording has just started. When they press the button again to conclude the recording, a new toast appears to let the users know that the recording has just finished.
Figure 9: Screenshots from the mobile app (II)
Following users’ feedback from the MVP evaluation, we have also re-designed the recommendations page. Each recommendation is now displayed and formatted as a small card. The heading of the card contains the temporal information, with the content of the actual recommendation in its central part. Furthermore, the heading of each card is coloured according to the priority level of each recommendation in order to help users quickly filter the recommendations they have received (Figure 9.a).
In the environment page the user gets information about environmental conditions at the workplace. The data come from sensors installed at the pilot sites (Figure 9.b).
The game results page provides information about the performance of the users in each game. The content of the page is updated each time the user completes his/her CGs activity. Currently, the displayed information contains the high score achieved in each game and the highest level reached (Figure 9.c).
sustAGE 53 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Finally, we introduce a typing sound and a soft vibration on the smartphone when the user presses any button in the app. These will be perceived by the users as feedback elements and increase their confidence while using the app.
sustAGE 54 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
6 Integration 6.1 sustAGE Databases The relational database that has been selected for the MVP to provide the data management capabilities required by the sustAGE Knowledge database was designed to be expandable to support the next versions of the sustAGE platform. The user-related events in the sustAGE Database follow the Entity–Attribute–Value (EAV) model. EAV is selected as an appropriate data model for the KB because it allows a general-purpose representation of episode types and their associated values. The initial design achieved this target so there was no need for significant changes while several new features were added in the system. The only addition in the database structure was to support the monitoring of user actions and the detection of abnormal body postures. This process produces and consumes more complex data thus the database structure is enhanced with new tables and relations to support these demands. The sustAGE uses other relational databases for specific backup or demonstration functionalities. The Replay database stores sensing data to replay for demonstration and development purposes while the Analytics database stores all messages flowing in the UM mainly for backup and possibly for further analysis during the next steps of implementation.
6.2 sustAGE REST API
Information stored in the sustAGE database needs to be accessible by various components to retrieve the user context while receiving messages that refer to a specific user. The stored knowledge might also be used for more generic processing, like, for example, to derive personal statistics or even global metrics based on historical data that will be found in the KB. Moreover, components might need to store or update information in a direct manner, without having to use the message-driven communication scheme. For example, a user’s response to a received recommendation is a valuable piece of feedback which can be stored directly in the relevant table of the database.
In order to fulfil the need for this type of direct communication, a REST API has been developed to permit controlled access to the data stored in the sustAGE database. Representational state transfer (REST) is an architectural style for software that defines a set of constraints to be used for creating web services [5]. Web services conforming to the REST architectural style, called RESTful web services, provide interoperability by allowing the requesting systems to access and manipulate textual representations of web resources by using a uniform and predefined set of stateless operations.
An initial set of RESTful services has been developed to satisfy the needs of the MVP use cases, and was updated accordingly to support the 1IP functionalities.
Table 8 presents the functions added to the REST API to support the 1IP implementation.
sustAGE 55 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
Table 8: Rest API functions
Method Endpoint
GET /sustage/knowledge/getUserBreakSchedule/{user_id}/{month}/{day}
Get the scheduled break times for a specific user at a given date
GET /sustage/knowledge/getUserLastHR/{user_id}
Get the last recorded heart rate average (60 sec) for the given user
GET /sustage/knowledge/getUserLastXHRMax/{user_id}/{x}
Get the last x captured heart rate averages and their average value for a given user
GET /sustage/knowledge/userToPlayCG/{user_id}
Get information for CG schedule for the given user
GET /sustage/knowledge/getUserLastTimeGame/{user_id}
Get the last time of CG activity performed for the given user
GET /sustage/knowledge/userToPerformHRRest/{user_id}/{dayOfTheWeek}
Get information regarding whether the user has to perform a Heart Rate rest test in a given day
GET /sustage/knowledge/userToDoPhysicalActivity/{user_id}
Get information for the physical activity schedule for the given user
sustAGE 56 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
The implementation of the REST API is based on Java EE and more specifically, Java Jersey library6, a JAX-RS Reference Implementation, and Gson7, a Java library provided by Google that can be used to convert Java Objects into their JSON representations.
6 Java Jersey library: https://eclipse-ee4j.github.io/jersey/ 7 Gson library: https://github.com/google/gson
sustAGE 57 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
7 Conclusions and next steps
The work carried out within this period has resulted into an operational infrastructure environment which supports the realization of the sustAGE 1IP as well as the deployment of the software modules and the IoT infrastructure considering additional privacy and security requirements.
The 1IP is a working integrated solution for both pilots, which cover a “normal” day of a worker inside and outside the working environment. The release of 1IP gives important perception towards the release of the envisioned sustAGE solution, ensuring a smooth and effective integration of the individual components.
The second cycle of evaluation that will follow will guide the developments towards the final release, along with the possible addition of new features and the refinement of the so far implemented solution.
sustAGE 58 January 20, 2021
SUSTAGE D5.3 SC1-DTH-03-2018/№ 826506
8 References
[1] M. Lourakis, M. Pateraki, I.-A. Karolos, C. Pikridas and P. Patias. Pose Estimation of a Moving Camera with Low-cost, multi-GNSS Devices. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXIV ISPRS Congress, XLIII-B2-2020, pp. 55-62, Nice, France, July 4-10, 2021.
[2] T. Hodaň, P. Haluza, Š. Obdržálek, J. Matas, M. Lourakis, X. Zabulis. T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 880-888, Santa Rosa, USA, 2017.
[3] A. Bagher Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2236–2246, Melbourne, Australia, July 2018.
[4] T. Bocklisch, J. Faulkner, N. Pawlowski, A. Nichol. Rasa: Open Source Language Understanding and Dialogue Management. In Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017.
[5] R. T. Fielding. Architectural Styles and the Design of Network-based Software Architectures, PhD. Dissertation, University of California, Irvine, 2000.
[6] M. F. Brito, A. L. Ramos, P. Carneiro, M. A. Gonçalves,. Ergonomic Analysis in Lean Manufacturing and Industry 4.0—A Systematic Review. In Lean Engineering for Global Development, pp. 95-127, Springer, Cham, 2019.
[7] T. Koukoulaki. The impact of lean production on musculoskeletal and psychosocial risks: An examination of sociotechnical trends over 20 years. Applied ergonomics, 45(2), pp. 198-212, 2014.
[8] M. Tenorth, J. Bandouch, and M. Beetz. The TUM kitchen data set of everyday manipulation activities for motion tracking and action recognition. In IEEE Int. Conf. Comput. Vis. Workshops, pp. 1089–1096, 2009.
[9] B. Parsa, E. U. Samani, R. Hendrix, C. Devine, S. M. Singh, S. Devasia, and A. G. Banerjee. Toward ergonomic risk prediction via segmentation of indoor object manipulation actions using spatiotemporal convolutional networks. IEEE Robotics and Automation Letters, 4(4):3153–3160, 2019.
[10]: Baldoni, R.; M. Contenti, and A. Virgillito. "The Evolution of Publish/Subscribe Communication Systems." Future Directions of Distributed Computing”, Springer Verlag LNCS Vol. 2584, 2003]
[11] Big Data Streaming Analytics Platforms, The Forrester Wave, July 17, 2014
[12] Apama 5.3 - Highlights & Future Directions, Software AG, GCS, April 2019
sustAGE 59 January 20, 2021