Affective Computing Overview of Theory, Techniques and Applications http://www.cs.unibo.it./difelice/ Context Aware Systems Prof. Marco Di Felice Department of Computer Science and Engineering University of Bologna Affective Computing q Affective computing is an emerging field of research that aims to enable intelligent systems to recognize, feel, infer and interpret human emotions (emotion-aware computing). ² Interdisciplinary research area: computer science (computer vision, IA, NLP, ..), psychology, cognitive science, social science. ² Include sentiment analysis (not covered in this presentation) and emotion recognition (focus on mobile applications here). AFFECTIVE COMPUTING: THEORY, TECHNIQUES AND APPLICATIONS MARCO DI FELICE 2 https://www.affectiva.com/product/affectiva-automotive-ai/ Affective Computing q USE CASES: SOCIAL MONITORING (I) Driver state monitoring ² Monitor levels of driver fatigue and distraction to enable appropriate alerts and interventions that correct dangerous driving. ² Monitor driver anger to enable interventions or route alternatives that avoid road rage. ² Address handoff challenge between driver and car in semi-autonomous vehicles. Occupant Experience Monitoring ² Personalize content recommendations ² Adapt environmental conditions AFFECTIVE COMPUTING: THEORY, TECHNIQUES AND APPLICATIONS MARCO DI FELICE 3 [1] M. Chen, Y. Zhang, Y. Li, S. Mao, V. C. M. Leung: EMC: Emotion-aware mobile cloud computing in 5G. IEEE Network 29(2): 32-38 (2015) [2] Yong Feng, Qiana Yin, Zhang Xiao, Han, Yu, Limei Peng, "EARS: Emotion-aware recommender system based on hybrid information fusion", Information Fusion, 46, March 2019, 141-146 Affective Computing CHEN2_LAYOUT.qxp_Layout 1 3/13/15 2:07 PM Page 33 q USE CASES: RECOMMENDER SYSTEMS (II) Y. Qian et al. Data collection (Historical data and data from social networks) ments when they are not available or when it is not conve- nient for the user to send his/her requests. Emotion-aware applications have great potential to Emotion Emotion-aware Mobile user recognition bring QoE provisioning to a new level and change the lifestyle of many people, which, however, also demands 5G network Emotion-aware more advanced technologies to yield effective solutions. service push recommender Fortunately, with MCC technology, computationally inten- sive tasks can be offloaded to the cloud instead of only by Remote cloud mobile devices [7]. Through the extensive use of cloud systems computing and cloud-assisted resources, mobile devices Data collection can break the shackles of their limited resources, the (Data from wearable models of mobile applications can be improved to enhance devices and user experience, and a user-centric paradigm can be smartphone) developed, which is also the core objective of 5G [8]. An important aspect of QoE provisioning is to offer personalized mobile service based on the user’s emotional Data collection Data preprocessing variations. This article proposes a novel style of applica- (multimedia emotional data) tions enabled by emotion-aware mobile cloud computing Sensing Computing resource (EMC) in 5G. EMC is design to provide emotion-aware resource mobile services by recognizing users’ emotionalArchitecture changes through cloud computing and big data analysis. EMC Architecture infers emotion based on affective computing, which is a complex process requiring a paradigmproposed shift in user model- in [1] Local cloudlet ing, for example, mode of operation, expression character- proposed in [2] istics, attitudes, preferences, cognitive styles, background Figure 1. The proposed EMC architecture. knowledge, and so on. The performance of affective com- Fig. 1. System architecture of EARS. puting largely depends on the quality of collected data, AFFECTIVE COMPUTING: THEORY, TECHNIQUES AND APPLICATIONS explicit information that can be processed through various approved the users, we can take some privacy preserving measures, such as which could be very large in volume and variety. The approaches for recommender systems, such as CF, SVD, and matrix anonymity, which protects the user’s privacy by hiding the user’s sen- 4 more types and larger scale of emotional data, the higher the quickly and efficiently, according to their physical andfactorization emo-[21]. However,MARCO DI FELICE social network data and user review sitive information, such as ID, name and so on. In fact, such a proces- data are implicit information whose features should be extracted by sing does not affect the performance of the recommendation system accuracy of emotional analysis results. Hence, in order to pro- tional status. available approaches. In particular, social information can be ex- proposed in this paper, because the two processes are vertical. We will vide accurate and timely emotion-aware services, affective To solve the challenging issues that arise in the EMCtracted archi- by social computing from social network data [22], and not discuss the data privacy preserving here. This work will be carried emotional information can be extracted from user review data by out in the future. computing is resource-intensive based on the analysis of emo- tecture, we investigate a comprehensive emotion-awaresentiment system analysis [23]. tional big data in the forms of text, video, social data, facial with the support of the latest technology advances,3. Algorithms such as and Models: After information fusion, the raw data are 3.2. Explicit feedback analysis features, and physiological signals, among others. mobile cloud computing and 5G. The main contributionspreprocessed, of and valuable information is extracted for re- commendations. Available algorithms and models are used to pro- On the basis of the algorithm mentioned above, we can provide the Recently, most existing healthcare systems only provide this work include: vide personalized services based on the hybrid information. following model: Assume that the user’s choice behavior is determined ff care for the user’s physiological status while EMC also takes •Enabling computation-intensive affective computing4. Recommendation in Services: In di erent scenarios, various re- by the user’s “selectivity” of the product. commendation services are appropriate: (i) statistics-based appli- Definition 1 assumes that user i and product j in potential product K- into account the user’s mental status. The existing healthcare mobile applications by MCC and big data analysis. cations that list the most popular items [24]; (ii) knowledge-based dimensional feature space are represented as a set of vectors applications that discover users’ interests according to historical systems exhibit the following features: •Providing users with resource cognition-based emotion- UUUUUUUUiiiikjjjjk(,12 ,, ), (, 12 ,, ). User characteristics and data [25]; and (iii) prediction-based applications that recognize product= characteristics⋯ = together determine⋯ the behavior of the user’s •Wearable and mobile devices are used to collect the user’s aware feedback by utilizing the abundant cloud resources fi users’ demands through advanced machine learning and arti cial choice, and thus Aij represents the absolute extent of the tendency of the physiological information. and the broadband bandwidth supported by 5G. intelligence [26]. product user ij: K •They only collect the user’s physical status, but are not •Identifying a novel EMC partitioning design trade-offIt is worth to noting that in the process of data collection and in- AUVUVij i· j ik jk formation fusion, the data we used may involve user s privacy in- aware of the user’s stress levels, unhealthy emotion, and guarantee users’ QoE while achieving optimized resource ’ ==k 1 (1) formation, such as emotional information. In order to protect privacy of ∑= even mental illness conditions. allocation under dynamic mobile network environments. However, the absolute tendency itself does not fully reflect the user’s •The existing mode based on wearable devices for collecting The remainder of this article is organized as follows. We physiological information will give a negative psychological first present the EMC architecture and then present a collab- implication that the user is in poor health. orative local cloudlet architecture for data collection and pre- In particular, for users in a mood of loneliness and depres- processing, as well as a proposed remote cloud for emotion sion, such a “conscious” way to collect and present their phys- recognition and service push. Next we introduce elastic emo- iological information may lead to even more serious mental tion-aware computing by joint cloud, communications, and illness. device resource optimization. Then we conclude the article. In contrast, with emotion care, the above problems can be alleviated. As examples, EMC can serve various scenarios as follows. Architecture • For elderly people, EMC can collect their physical condi- The proposed EMC architecture is shown in Fig. 1, which tions, as well as perceive their mental status, which allevi- includes three main components: a mobile terminal, a local ates their loneliness and other negative emotions. cloudlet, and a remote cloud. • For those working in a closed environment over a long peri- od, such as deep-sea or space exploration, EMC can be uti- EMC Infrastructure Functional Components lized to perceive their emotions and help adjust their In EMC, infrastructure functional components, which consist physical and mental state to ensure successful completion of mobile terminals, a local cloudlet, and a remote cloud,