Programming in Natural Language with Fuse: Synthesizing Methods
Total Page：16
File Type：pdf, Size：1020Kb
Programming in Natural Language with fuSE: Synthesizing Methods from Spoken Utterances Using Deep Natural Language Understanding Sebastian Weigelt, Vanessa Steurer, Tobias Hey, and Walter F. Tichy Karlsruhe Institute of Technology Institute for Program Structures and Data Organization Karlsruhe, Germany [email protected], [email protected], [email protected], [email protected] Abstract few addressed learning new functionality from nat- ural language instructions (e.g. Le et al.(2013); The key to effortless end-user programming is Markievicz et al.(2017)). However, even recent natural language. We examine how to teach in- approaches still either restrict the language or are telligent systems new functions, expressed in (over-)fitted to a certain domain or application. natural language. As a first step, we collected 3168 samples of teaching efforts in plain En- We propose to apply deep natural language un- glish. Then we built fuSE, a novel system that derstanding to the task of synthesizing methods translates English function descriptions into from spoken utterances. Our approach combines code. Our approach is three-tiered and each modern machine learning techniques with infor- task is evaluated separately. We first classify mation retrieval and knowledge-based methods to whether an intent to teach new functionality grasp the user’s intent. As a first step, we have per- is present in the utterance (accuracy: 97.7% formed a user study to investigate how laypersons using BERT). Then we analyze the linguis- tic structure and construct a semantic model teach new functionality with nothing but natural (accuracy: 97.6% using a BiLSTM). Finally, language. In a second step, we develop fuSE (Func- we synthesize the signature of the method, tion Synthesis Executor). fuSE translates teaching map the intermediate steps (instructions in the efforts into code. On the basis of the gathered data method body) to API calls and inject con- we constructed a three-tiered approach. We first trol structures (F1: 67.0% with information determine, whether an utterance comprises an ex- retrieval and knowledge-based methods). In plicitly stated intent to teach a new skill. Then, an end-to-end evaluation on an unseen dataset we decompose these teaching efforts into distinct fuSE synthesized 84.6% of the method signa- tures and 79.2% of the API calls correctly. semantic parts. We synthesize methods by transfer- ring these semantic parts into a model that repre- 1 Introduction sents the structure of method definitions. Finally, we construct signatures, map instructions of the Intelligent systems became rather smart lately. One body to API calls, and inject control structures. easily arranges appointments by talking to a vir- tual assistant or controls a smart home through a 2 Related Work conversational interface. Instructing a humanoid robot in this way no longer seems to be futuristic. The objective of programming in natural language For the time being, users can only access built-in was approached from different perspectives over functionality. However, they will soon expect to the years. Quite a few approaches are natural lan- add new functionality themselves. For humans, the guage interfaces to code editors (Price et al., 2000; most natural way to communicate is by natural lan- Begel, 2004; Begel and Graham, 2005;D esilets´ guage. Thus, future intelligent systems must be et al., 2006). However, they assume that users lit- programmable in everyday language. erally dictate source code. Thus, these approaches Today’s systems that claim to offer program- are intended for developers rather than laypersons. ming in natural language enable laypersons to is- Other approaches such as Voxelurn by Wang et al. sue single commands or construct short scripts (e.g. (2017) aim to naturalize programming languages Mihalcea et al.(2006); Rabinovich et al.(2017)); to lower the hurdle for programming novices. usually no new functionality is learned. Only a Approaches for end-user programming in natu- 4280 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4280–4295 July 5 - 10, 2020. c 2020 Association for Computational Linguistics coffee is a beverage coffee is a beverage coffee is a beverage people like in order people like in order people like in order MISC procedure makeCoffee() to make coffee you to make coffee you to make coffee you st nd rd locate(Cup) 1 stage 2 stage DECL 3 stage put(Cup, have to locate the have to locate the have to locate the CoffeeMachine .Dispenser) teaching intent semantic struct. method synth. cup place it under cup place it under cup place it under press(CoffeeMachine .RedButton) the dispenser and the dispenser and the dispenser and press the red button press the red button press the red button Teaching SPEC Figure 1: Schematic overview of fuSE’s three-tiered approach. ral language take up the challenge of bridging the Markievicz et al.(2017), use task descriptions that semantic gap between informal spoken or written were intended to instruct humans to benchmark descriptions in everyday language and formal pro- their approach. However, often the assumed vocab- gramming languages. Early systems were syntax- ulary is rather technical (Lincoln and Veres, 2012). based (Winograd, 1972; Ballard and Biermann, Thus, the usability for laypersons is limited. 1979; Biermann and Ballard, 1980; Biermann et al., 1983; Liu and Lieberman, 2005). Some were al- 3 Approach ready capable to synthesize short scripts including The goal of our work is to provide a system for pro- control structures and comments, e.g. NLP for NLP gramming in (spoken) natural language. Layper- by Mihalcea et al.(2006). Others take the user in sons shall be enabled to create new functionality in the loop and create scripts with a dialog-driven ap- terms of method definitions by using natural lan- proach (Le et al., 2013). In further developments in- guage only. We offer a general approach, i.e. we do telligent assistants offer their service to assist with not restrict the natural language regarding wording programming (Azaria et al., 2016). Often these and length. Since spontaneous language often com- assistants support multi-modal input, e.g. voice prises grammatical flaws, disfluencies, and alike, and gestures (Campagna et al., 2017, 2019). Oth- our work must be resilient to these issues. ers combine programming in natural language with We decompose the task in three consecutive other forms of end-user programming, such as pro- steps. The rationale behind this decision is as fol- gramming by example (Manshadi et al., 2013) or lows. On the one hand, we can implement more programming by demonstration (Li et al., 2018). focused (and precise) approaches for each task, e.g. Some authors such as Landhaußer¨ et al.(2017) using machine learning for one and information and Atzeni and Atzori(2018a,b) take a knowledge- retrieval for another. On the other hand, we are based approach by integrating domain and environ- able to evaluate and optimize each approach indi- mental information in the form of ontologies. vidually. The stages of our three-tiered approach Suhr and Artzi(2018) employ a neural network are the following (see Figure 1 for an example): to learn a situational context model that integrates the system environment and the human-system- 1. Classification of teaching efforts: Deter- interaction, i.e. the dialog. Many recent approaches mine whether an utterance comprises an ex- integrate semantic parsing in the transformation plicitly stated teaching intent or not. process (Guu et al., 2017; Rabinovich et al., 2017; 2. Classification of the semantic structure: Chen et al., 2018; Dong and Lapata, 2018). Even Analyze (and label) the semantic parts of a though the natural language understanding capabil- teaching sequence. Teaching sequences are ities are often impressive, the synthesized scripts composed of a declarative and a specifying are still (semantically) erroneous in most cases. part as well as superfluous information. Additionally, learning of new functionality is not covered by approaches of that category so far. 3. Method synthesis: Build a model that repre- Programming in natural language is of particular sents the structure of methods from syntactic interest in the domain of humanoid robotics (Lau- information and classification results. Then, ria et al., 2001, 2002; She et al., 2014; Mei et al., map the actions of the specifying part to API 2016). People expect to teach them as they teach calls and inject control structures to form the human co-workers. Therefore, some authors, e.g. body; synthesize the method signature. 4281 The first two stages are classification problems. desc. w. (total) w. (unique) Thus, we apply various machine learning tech- sc. 1 (greet) 795 18,205 566 niques. The first stage is a sequence-to-single-label sc. 2 (coffee) 794 26,005 625 task, while the second is a typical sequence-to- sc. 3 (drinks) 794 33,001 693 sequence task. For the first we compare classical sc. 4 (table) 785 31,797 685 machine learning techniques, such as logistic re- total 3,168 109,008 1,469 gression and support vector machines, with neural network approaches including the pre-trained lan- Table 1: The number of descriptions, words, and unique words per scenario and in the entire dataset. guage model BERT (Devlin et al., 2019). For the second task we narrow down to neural networks and BERT. A more detailed description of the first two stages may be found in (Weigelt et al., 2020). completed the study. The share of male and female The implementation of the third stage is a com- participants is almost equal (50.5% vs. 49.5%); bination of syntactic analysis, knowledge-based more than 60% are native English speakers. Most techniques and information retrieval. We use se- of them (70%) had no programming experience at mantic role labeling, coreference analysis, and a all.