Domain Expert Platform for Goal-Oriented Dialogue Collection Didzis Gosko Arturs Znotins Inguna Skadina Normunds Gruzitis Gunta Nespore-Berzkalne Institute of Mathematics and Computer Science, University of Latvia Raina bulv. 29, Riga, Latvia [email protected] Abstract represent various dialogue scenarios, as well as knowledge of the specific domain. This informa- Today, most dialogue systems are fully or partly built using neural network architectures. tion has to be provided in specific data formats A crucial prerequisite for the creation of a goal- that in many cases are too complicated for domain oriented neural network dialogue system is a experts. Moreover, the required training datasets dataset that represents typical dialogue scenar- usually include various annotation layers, such as ios and includes various semantic annotations, named entities, dialogue acts, intents, etc. The e.g. intents, slots and dialogue actions, that are creation of such datasets is a complex task, and necessary for training a particular neural net- the datasets are not completely isolated and ab- work architecture. In this demonstration paper, stracted from the particular dialogue system. Thus, we present an easy to use interface and its back- end which is oriented to domain experts for domain experts that are involved in the creation the collection of goal-oriented dialogue sam- of the datasets must have a high-level understand- ples. The platform not only allows to collect ing of the overall structure of the dialogue system or write sample dialogues in a structured way, and its components, and how it is reflected in the but also provides a means for simple annota- annotated dialogue samples. tion and interpretation of the dialogues. The This demonstration paper address this issue by platform itself is language-independent; it de- presenting a web-based platform for the creation pends only on the availability of particular lan- 1 guage processing components for a specific and maintenance of dialogue datasets . The inter- language. It is currently being used to collect face of the platform is very simple and high-level: dialogue samples in Latvian (a highly inflected it allows a domain expert without detailed technical language) which represent typical communica- knowledge of the underlying dialogue system to tion between students and the student service. create and update a representative training dataset, as well as to maintain the underlying database of 1 Introduction domain- and organisation-specific information that Modeling of human-computer interaction through will be used for question answering. The platform dialogue systems and chatbots has raised a great in- provides tools for the creation of goal-oriented dia- terest among researchers and industry already since logue systems, in particular: the time when the first chatbot Eliza (Weizenbaum, 1966) was created. This interest has become viral • creation of datasets for dialogue systems that after the successful introduction of Siri (Bellegarda, provide (or generate) responses depending on 2013). Today, virtual assistants, virtual agents and user input, intents and on the previous actions chatbots are present everywhere: on mobile de- of the dialogue system; vices, on different social networking platforms, on • creation of datasets for dialogue systems that many websites and through smart home devices cover one or several topics; and robots. The virtual conversational agents are usually im- • slot filling, including slot filler (e.g. named plemented as end-to-end neural network models, or entity) normalization and annotation; their components are implemented through neural network architectures (Louvan and Magnini, 2020). • creation and maintenance of slot filler aliases; Such architectures require creation of datasets that 1http://bots.ailab.lv/ 295 Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 295–301 April 19 - 23, 2021. ©2021 Association for Computational Linguistics • creation and maintenance of knowledge base templates, and the typical dialogue scenarios. For and interactive response selection; the demonstration purposes, we have chosen three common topics: working hours, academic leave, • response generation, including the generation and enrollment requirements. Elaborated and an- of inflectional forms for syntactic agreement. notated sample dialogues constituting the training dataset have been specified by a domain expert us- Our platform not only supports collection of di- ing the dialogue management platform presented alogue scenarios, but also simulates prototypical in this paper. interaction between human and computer. The tool Since we focus on goal-oriented virtul assistants, has been successfully used for the creation of a the Hybrid Code Networks (HCN) architecture has dialogue dataset for the virtual assistant that sup- been selected for the implementation (Williams ports the work of the student service in relation to et al., 2017) allowing us to combine recursive three frequently asked topics: working hours and neural networks (RNN) with the domain-specific contacts of the personnel and structural units (e.g. knowledge and action templates of the dialogue sys- libraries), issues regarding academic leave, as well tem. The concrete dialogue system is implemented as enrollment requirements and documents to be within the DeepPavlov framework2. submitted (Skadina and Gosko, 2020). In the next chapters of this paper, we describe 3 Overall Architecture and Components our motivation to develop the platform, its overall architecture and main components, and the domain The platform presented in this paper is designed to expert interface and its main components. support three use cases: 2 Background and Motivation 1. To create and gradually improve a collection of dialogue samples necessary for developing For English and several other widely used lan- and testing a goal-oriented dialogue system. guages, many publicly available dialogue datasets have been created and are reused for different re- 2. To support (re-)training of a goal-oriented di- search and development needs (e.g., Budzianowski alogue system. et al.(2018), Zeng et al.(2020)). However, in the case of less-resourced languages, only few or 3. To support dialogue testing in the inference no training datasets are available (Serban et al., mode. Training and running a dialogue system 2018). To our knowledge, there is no publicly in the inference mode is performed through available dataset for Latvian, that could be used for the DeepPavlov framework by passing the goal-oriented dialogue system modelling. To over- goal-oriented bot model configuration along come this obstacle, some research groups machine- with relations to other objects that are specific translate existing English datasets into the low- to the platform. resourced languages, while others try to build train- Figure1 illustrates the architecture of the plat- ing datasets from scratch. When possible, crowd- form. Apart from the domain expert user interface sourcing, including gamification (Ogawa et al., described in detail in Section4, key components 2020), is used as well. However, there in no best of the platform are four databases for storing the recipe for obtaining or collecting dialogue sam- dialogue scenarios, the relevant entities and their ples for a specific NLP task (in our case, dialogue aliases for slot filling, the required external knowl- modeling) for a less-resourced language with a rel- edge for question answering, and reusable tem- atively small number of speakers. plates for response generation. The motivation of our work is the necessity to build virtual assistants in less-resourced settings. 3.1 Dialogue Database The practical use case to test the platform has been Dialogues created by the users of the platform (i.e., the everyday communication between students and by domain experts not end-users) are stored in the the student service of the University of Latvia. SQLite database Dialogues to support concurrent Since this communication has never been inten- modification. The dialogue database stores poten- tionally recorded, we started with the analysis of tial end-user utterances together with the respective data retrieved from an online student forum to iden- tify the most common topics, question and answer 2https://deeppavlov.ai/ 296 User Interface User Interface Web Browser Backend Server Knowledge base Dialogues Entities Templates SQLite SQLite SQLite SQLite Intent Named Entity Response Goal-Oriented Bot Slot Filling Identification Recognition Generation Figure 1: Architecture of the platform. The selected (grey) part of the diagram is language-specific and can be replaced or removed entirely. intents, slot values and the corresponding bot ac- 2020). To support entity classes of a particular do- tions. main, the NER model is trained on a larger general- Intents for the particular dialogue dataset are de- domain dataset (Gruzitis et al., 2018; Paikens et al., fined in a separate view of the platform’s interface. 2020) and a smaller domain-specific dataset. The The predefined intents are linked to utterances dur- combined model allows to recognize not only com- ing the dialogue writing process (for details, see monly used entity classes like persons, locations Section4) and later used for training. In our demon- and organizations, but also domain specific entities stration dialogue system, we use a Keras classifica- like job positions and working hours.