Conversation Model Fine-Tuning for Classifying Client Utterances in Counseling Dialogues Sungjoon Park 1, Donghyun Kim 2, Alice Oh 1 1 School of Computing, KAIST, Republic of Korea 2 Trost, Humart Company, Inc., Republic of Korea [email protected], [email protected] [email protected] Abstract office and with reduced financial burden com- pared to traditional face-to-face counseling ses- The recent surge of text-based online coun- sions (Hull, 2015). seling applications enables us to collect and In text-based counseling, the communication analyze interactions between counselors and environment changes from face-to-face counsel- clients. A dataset of those interactions can ing sessions. The counselor cannot read non- be used to learn to automatically classify the client utterances into categories that help verbal cues from their clients, and the client uses counselors in diagnosing client status and pre- text messages rather than spoken utterances to dicting counseling outcome. With proper deliver their thoughts and feelings, resulting in anonymization, we collect counselor-client di- changes of dynamics in the counseling relation- alogues, define meaningful categories of client ship. Previous studies explored computational ap- utterances with professional counselors, and proaches to analyzing the dynamic patterns of re- develop a novel neural network model for clas- lationship between the counselor and the client sifying the client utterances. The central idea of our model, ConvMFiT, is a pre-trained con- by focusing on the language of counselors (Imel versation model which consists of a general et al., 2015; Althoff et al., 2016), clustering topics language model built from an out-of-domain of client issues (Dinakar et al., 2014), and look- corpus and two role-specific language models ing at therapy outcomes (Howes et al., 2014; Hull, built from unlabeled in-domain dialogues. The 2015). classification result shows that ConvMFiT out- Unlike previous studies, we take a computa- performs state-of-the-art comparison models. tional approach to analyze client responses from Further, the attention weights in the learned model confirm that the model finds expected the counselor’s perspective. Client responses linguistic patterns for each category. in counseling are crucial factors for judging the counseling outcome and for understanding the sta- 1 Introduction tus of the client. So we build a novel catego- rization scheme of client utterances, and we base Some mental disorders are known to be treated ef- our categorization scheme on the cognitive behav- fectively through psychotherapy. However, people ioral theory (CBT), a widely used theory in psy- in need of psychotherapy may find it challenging chotherapy. Also, in developing the categories, we to visit traditional counseling services because of consider whether they are adequate for the unique time, money, emotional barriers, and social stigma text-only communication environment, and appro- (Bearse et al., 2013). Recently, technology- priate for the annotation of the dialogues as train- mediated psychotherapy services emerged to alle- ing data. Then using the corpus of text-based viate these barriers. Mobile-based psychotherapy counseling sessions annotated according to the programs (Mantani et al., 2017), fully automated categorization scheme, we build a novel conver- chatbots (Ly et al., 2017; Fitzpatrick et al., 2017), sation model to classify the client utterances. and intervention through smart devices (Torrado This paper presents the following contributions: et al., 2017) are examples. Among them, text- based online counseling services with professional • First, we build a novel categorization method counselors are becoming popular because clients as a labeling scheme for client utterances in can receive these services without traveling to an text-based counseling dialogues. 1448 Proceedings of NAACL-HLT 2019, pages 1448–1459 Minneapolis, Minnesota, June 2 - June 7, 2019. c 2019 Association for Computational Linguistics • Second, we propose a new model, Conversa- counseling dialogues, these categories are not di- tion Model Fine-Tuning (ConvMFiT) to clas- rectly applicable. Using text without non-verbal sify the utterances. We explicitly integrate cues, a client’s responses are inherently differ- pre-trained language-specific word embed- ent from the transcriptions of verbally spoken re- dings, language models and a conversation sponses which include categories such as ‘silence’ model to take advantage of the pre-trained (no response for more than 5 seconds) and ‘non- knowledge in our model. verbal referent’ (physically pointing at a person). Another relevant study, derived from text-based • Third, we empirically evaluate our model in counseling sessions with suicidal adolescents pro- comparison with other models including a poses 19 categories which we judged to be too state-of-the-art neural network text classifica- many to be practical for manual annotation (Kim, tion model. Also, we show typical phrases of 2010). counselors and clients for each category by investigating the attention layers. The last criterion of “meaningful to counselors” is perhaps the most important. To meet that cri- terion, we base the categorization process on the 2 Categorization of Client Utterances cognitive behavioral theory (CBT) which is the Client responses provide essential clues to un- underlying theory behind psychotherapy counsel- derstanding the client’s internal status which can ing. The details of using CBT for the categoriza- vary throughout counseling sessions. For exam- tion process is explained next. ple, client’s responses describing their problems Categorization Process and Results. In develop- prevail at the early stage of counseling (Hill, 1978; ing the categories, we follow the Consensual Qual- E. Hill et al., 1983), but as counseling progresses, itative Research method (Hill et al., 1997). Two problem descriptions decrease while insights and professional counselors with clinical experience discussions of plans continue to increase (Seeman, participated in this qualitative research method to 1949). Client responses can also help in predicting define the categorization. counseling outcomes. For example, a higher pro- To begin, we randomly sample ten client cases portion of insights and plans in client utterances considering demographic information including indicates a high positive effect of counseling (Hill, age, gender, education, job, and previous counsel- 1978; E. Hill et al., 1983). ing experiences. We then start the categorization Categorization Objective. Our final aim is to process with the fundamental components of the build a machine learning model to classify the CBT which are events, thoughts, emotions, and be- client utterances. Thus, the categorization of the havior (Hill et al., 1981). The professional coun- utterances should satisfy the following criteria: selors annotate every client utterance to with those • Suitable for the text-only environment: Cate- initial component categories with tags that add de- gories should be detected only using the text tail. For example, if an utterance is annotated as response of clients. ‘emotion’, we add ‘positive/negative’ or concrete label such as ‘hope’. If these tags are categorized • Available as a labeling scheme: The num- to be a new category, we add that category to the ber of categories should be small enough for list until it is saturated. When the number of cat- manual annotation by counseling experts. egories becomes more than 40, we define higher level categories that cover the existing categories. • Meaningful to counselors: Categories should In the second stage, annotators discuss and be meaningful for outcome prediction or merge these categories into five high-level cate- counseling progress tracking. gories. Category 1 is informative responses to Previous studies in psychology proposed nine and counselors, and category 2 is providing factual in- fourteen categories for client and counselor verbal formation and experiences. Categories 3 and 4 are responses, respectively, by analyzing transcrip- related to the client factors, expressing appealing tions from traditional face-to-face counseling ses- problems and psychological changes. The last cat- sions (Hill, 1978; Hill et al., 1981). But these cate- egory is about the logistics of the counseling ses- gories were developed for face-to-face spoken in- sions including scheduling the next session. The teractions, and we found that for online text-only categories in detail are as follows: 1449 Characteristic Informative Client Factors Process Factual Anecdotal Appealing Psychological Counseling Category Information Experience Problem Change Process Name (Fact.) (Anec.) (Prob.) (Chan.) (Proc.) Statement at the Statement of Brief mention of Clients experience Clients factors resolution stage counseling Explanation categorical contributing to the related to the of the appealing structure and information appealing problem appealing problem problem relationship • Objective • Experience • Negative • Positive • A message to Fact with others Emotion Prediction counselor • Living • Comments • Cognitive • Expectation, • Gratitude, Examples conditions from others distortion Determination Greetings • Demographic • Interpersonal • Coping • Time • Trauma information problems behaviors appointment • Limited • Interpersonal • Family • Self- • Questions about conditions situations problems awareness the consultation Table 1: Final Categorization of client utterances. Five