Supporting Cognitive and Emotional Empathic Writing of Students Thiemo Wambsganss1,2, Christina Niklaus1, 3, Matthias Sollner¨ 4, Siegfried Handschuh1, 3 and Jan Marco Leimeister1, 4 1 University of St.Gallen fthiemo.wambsganss, christina.niklaus, siegfried.handschuh, [email protected] 2 Carnegie Mellon University [email protected] 3 University of Passau fchristina.niklaus, [email protected] 4 University of Kassel fsoellner, [email protected] Abstract We present an annotation approach to cap- turing emotional and cognitive empathy in student-written peer reviews on business mod- els in German. We propose an annotation scheme that allows us to model emotional and cognitive empathy scores based on three types of review components. Also, we conducted an annotation study with three annotators based on 92 student essays to evaluate our annota- tion scheme. The obtained inter-rater agree- Figure 1: Empathy annotation scheme. First, a text ment of α=0.79 for the components and the paragraph is classified into a peer review component multi-π=0.41 for the empathy scores indicate (strengths, weakness, improvement suggestions). Sec- that the proposed annotation scheme success- ond, the same annotator is then scoring the cognitive fully guides annotators to a substantial to mod- and emotional empathy level of the components based erate agreement. Moreover, we trained predic- on our annotation guideline on a 1-to-5 scale. tive models to detect the annotated empathy structures and embedded them in an adaptive writing support system for students to receive served experiences of another,” (Davis, 1983, p.1)1. individual empathy feedback independent of Empathy skills not only pave the foundation for an instructor, time, and location. We evalu- successful interactions in digital companies, e.g., ated our tool in a peer learning exercise with 58 students and found promising results for in agile work environments (Luca and Tarricone, perceived empathy skill learning, perceived 2001), but they are also one of the key abilities in feedback accuracy, and intention to use. Fi- the future that will distinguish the human work- nally, we present our freely available corpus of force and artificial intelligence agents from one 500 empathy-annotated, student-written peer another (Poser and Bittner, 2020). However, be- arXiv:2105.14815v1 [cs.CL] 31 May 2021 reviews on business models and our annotation sides the growing importance of empathy, research guidelines to encourage future research on the has shown that empathy skills of US college stu- design and development of empathy support dents decreased from 1979 to 2009 by more than systems. thirty percent and even more rapidly between 2000 to 2009 (Konrath et al., 2011). On these grounds, 1 Introduction the Organization for Economic Cooperation and Empathy is an elementary skill in society for daily Development (OECD) claims that the training for interaction and professional communication and empathy skills should receive a more prominent is therefore elementary for educational curricula role in today’s higher education (OECD, 2018). (e.g., Learning Framework 2030 (OECD, 2018)). 1Being aware that empathy is a multidimensional construct, It is the “ability to simply understand the other in this study, we focus on emotional and cognitive empathy person’s perspective [:::] and to react to the ob- (Spreng et al., 2009; Davis, 1983). To train students with regard to empathy, educa- et al., 2009), the perceived feedback accuracy (Pod- tional institutions traditionally rely on experiential sakoff and Farh, 1989), and the intention to use learning scenarios, such as shadowing, commu- (Venkatesh and Bala, 2008) in a controlled evalu- nication skills training, or role playing (Lok and ation with 58 students provided promising results Foster, 2019; van Berkhout and Malouff, 2016). for using our approach in different peer learning Individual empathy training is only available for a scenarios to offer quality education independent of limited number of students since individual feed- an instructor, time, and location. back through a student’s learning journey is often Our contribution is fourfold: 1) we derive a hindered due to large-scale lectures or the growing novel annotation scheme for empathy modeling field of distance learning scenarios such as Mas- based on psychological theory and previous work sive Open Online Classes (MOOCs) (Seaman et al., on empathy annotation (Buechel et al., 2018); 2) 2018; Hattie and Timperley, 2007). we present an annotation study based on 92 student One possible path for providing individual learn- peer reviews and three annotators to show that the ing conditions is to leverage recent developments in annotation of empathy in student peer reviews is re- computational linguistics. Language-based models liably possible; 3) to the best of our knowledge, we enable the development of writing support systems present the second freely available corpus for em- that provide tailored feedback and recommenda- pathy detection in general and the first corpus for tions (Santos et al., 2018), e.g., like those already empathy detection in the educational domain based used for argumentation skill learning (Wambsganss on 500 student peer reviews collected in our lecture et al., 2020a, 2021b). Recently, studies have started about business innovation in German; 4) we embed- investigating elaborated models of human emotions ded our annotation approach as predictive models (e.g., Wang et al.(2016), Abdul-Mageed and Un- in a writing support system and evaluated it with 58 gar(2017), Buechel and Hahn(2018), or Sharma students in a controlled peer learning scenario. We et al.(2020)), but available corpora for empathy hope to encourage research on student-written em- detection are still rare. Only a few studies address pathetic texts and writing support systems to train the detection and prediction of empathy in natural students’ empathy skills based on NLP towards a texts (Khanpour et al., 2017; Xiao et al., 2012), and, quality education independent of a student’s loca- to the best of our knowledge, only one corpus is tion or instructors. publicly available for empathy modelling based on news story reactions (Buechel et al., 2018). Past 2 Background literature therefore lacks 1) publicly available em- The Construct of Empathy The ability to per- pathy annotated data sets, 2) empathy annotation ceive the feelings of another person and react to models based on rigorous annotation guidelines their emotions in the right way requires empathy combined with annotation studies to assess the – the ability “of one individual to react to the ob- quality of the data, 3) the alignment of empathy served experiences of another” (Davis(1983), p.1). in literature on psychological constructs and theo- Empathy plays an essential role in daily life in ries, and 4) an embedding and real-world evalua- many practical situations, such as client communi- tion of novel modelling approaches in collaborative cation, leadership, or agile teamwork. Despite the learning scenarios (Rose´ et al., 2008). interdisciplinary research interest, the term empa- We introduce an empathy annotation scheme thy is defined from multiple perspectives in terms and a corpus of 500 student-written reviews that of its dimensions or components (Decety and Jack- are annotated for the three types of review compo- son, 2004). Aware of the multiple perspectives on nents, strengths, weaknesses, and suggestions for empathy, in this annotation study, we focused on improvements, and their embedded emotional and the cognitive and emotional components of em- cognitive empathy level based on psychological the- pathy as defined by Davis(1983) and Lawrence ory (Davis, 1983; Spreng et al., 2009). We trained et al.(2004). Therefore, we follow the ‘Toronto different models and embedded them as feedback Empathy Scale’ (Spreng et al., 2009) as a synthesis algorithms in a novel writing support tool, which of instruments for measuring and validating empa- provided students with individual empathy feed- thy. Hence, empathy consists of both emotional back and recommendations in peer learning scenar- and cognitive components (Spreng et al., 2009). ios. The measured empathy skill learning (Spreng While emotional empathy lets us perceive what other people feel, cognitive empathy is the human 2018). ability to recognize and understand other individu- Empathy Annotated Corpora and Annotation als (Lawrence et al., 2004). Schemes Only a few studies address the detec- Emotion and Empathy Detection In NLP, the tion and prediction of empathy in natural language detection of empathy in texts is usually regarded texts (e.g., Khanpour et al.(2017) and Xiao et al. as a subset of emotion detection, which in turn (2012)). Presenting the first and only available gold is often referred to as part of sentiment analysis. standard data set for empathy detection, Buechel The detection of emotions in texts has made ma- et al.(2018) constructed a corpus in which crowd- jor progress, with sentiment analysis being one workers were asked to write emphatic reactions to of the most prominent areas in recent years (Liu, news stories. Before the writing tasks, the crowd- 2015). However, most scientific studies have been workers were asked to conduct a short survey with focusing on the prediction of the polarity of words self-reported items to measure their empathy level for assessing negative and positive notions (e.g., and their personal distress based on Batson et al.