Rosetta Code: Improv in Any Language Piotr Mirowski1, Kory Mathewson1, Boyd Branch1,2 Thomas Winters1,3, Ben Verhoeven1,4, Jenny Elfving1 1Improbotics (https://improbotics.org) 2University of Kent, United Kingdom 3KU Leuven, Dept. of Computer Science; Leuven.AI, Belgium 4ERLNMYR, Belgium Abstract Rosetta Code provides improv theatre performers with artificial intelligence (AI)-based technology to perform shows understandable across many different languages. We combine speech recognition, improv chatbots and language translation tools to enable improvisers to com- municate with each other while being understood—or comically misunderstood—by multilingual audiences. We describe the technology underlying Rosetta Code, detailing the speech recognition, machine translation, text generation and text-to-speech subsystems. We then describe scene structures that feature the system in performances in multilingual shows (9 languages). We provide evaluative feedback from performers, au- Figure 1: Example of a performed Telephone Game. Per- diences, and critics. From this feedback, we draw formers are aligned and one whispers to their partner on analogies between surrealism, absurdism, and multilin- the right a phrase in a foreign language (here, in Swedish), gual AI improv. Rosetta Code creates a new form of language-based absurdist improv. The performance re- which is then repeated to the following performer, until the mains ephemeral and performers of different languages last utterance is voiced into automated speech recognition can express themselves and their culture while accom- and translation to show how information is lost. modating the linguistic diversity of audiences. in which it is performed. Given that improv is based on the Introduction connection between the audience and the performers, watch- Theatre is one of the most important tools we have for shar- ing improv in a foreign language severely limits this link. ing experiences and building cross-cultural understanding. This contrasts with scripted theatre, which has been sal- Moreover, theatre performers and audiences who speak dif- vaged from monolingual oblivion: Sophocles, Shakespeare, ferent languages are more connected than ever, thanks to in- and Sartre continue to be translated into many different lan- creasing ease of communication, dissemination of culture, guages, reinterpreted, and enjoyed by audiences around the translation, travel, and improvements in remote performance world. Improv has not had such an opportunity, and perfor- capabilities. In particular, improvised theatre (improv) is mance groups are bound to remain local or switch to English well positioned to connect culture given its universality, ac- as a lingua franca when performing internationally. cessibility, and low barriers to entry: improvisation tech- The art of improvisation is derived from the connections niques can be readily understood and internalized, and in a between performative layers, both between the performers, short manner of time, individuals from diverse cultures em- and between the performers and the audience. Improv em- pathize with each other while performing scenes together, braces the audience to create collaboratively together. In with deep characters, relationships, settings, motivations, this way improvisation is a democratic narrative, and the and subtext. Improv serves as a microcosm of cultural com- potential impacts of improvised theatre between performers munication; it is “the theatre of the people” in moment (Boal and audiences of different cultures and languages are sig- 2006). Improv is therefore an ideal test-bed to explore broad nificant. Most international improvisational collaboration is cultural and communication questions (Mathewson 2019). English based, but many regional festivals take place in the Improv is also a paradoxical cultural artifact. On one languages of the host region. These performances exclude hand, improv is ubiquitous and conveys universal messages audiences without knowledge of the performance language, about the human condition and the vagaries of life. On the and limits the contributions of improvisors who do not speak other hand, as a highly linguistic art form, improv is nearly the language. Without translation, improvisation misses out impossible to understand if you do not know the language on important voices due to language limitations. Proceedings of the 11th International Conference on Computational Creativity (ICCC’20) 115 ISBN: 978-989-54160-2-8 How can we create conditions so that improvisors from different cultures can improvise together in their own lan- guage? How can audiences understand performers using different languages? How might we grow our cultural com- munication and empathy while only being able to speak one language? Rosetta Code answers these questions, and gives theatrical improv a suite of software, scenes, and show struc- tures from which to advance and expand. The Methods section describes the technical details of the system, the challenges associated with improv in any lan- guage, and how we used our system in the context of the- atrical improvisation. In section Rosetta Code on Stage we provide results of using the system in three shows us- ing nine languages. We also present evaluative feedback Figure 2: Illustration of the visual interface used in Rosetta from performers, audiences, and critics. In section Re- Code as seen by the audience. The top part displays the lated Work, Historical Context and Discussion we situate choice of the language used for speech recognition and the Rosetta Code at the intersection of improvisational theatre latest recognised sentence (here French). The bottom part and language, present an exploration of the cultural impor- shows the target language for machine translation (here Pol- tance of multi-lingual artistic performance, and provide sev- ish), as well as the last few translation results. The buttons eral directions for future work. and input box (top) enable overriding speech recognition and activating / deactivating text-to-speech. Methods Artificial intelligence-based improvisation is an art form, where a robot and/or AI is used on stage as an improv stage dynamic vocal microphones (with an on-off button that partner (Bruce et al. 2000; Mathewson and Mirowski 2017a; can be triggered by the user), connected to the computer 2017b; Mirowski and Mathewson 2019; Jacob et al. 2019; via an analog-to-digital audio interface. Winters and Mathewson 2019; Liu et al. 2019; Mathewson An instantaneous translation system, e.g. Google Trans- 2019). That robot relies on a generative language model to • late API4, is used as communication channel to convert produce lines or actions in response to context, and can in recognised speech from one language into another. itself be seen as a computationally creative system. A vari- A surtitle visualization interface (Figure 2) that enables ation of that format, Improbotics, designed in 2016, con- • sists in letting human actors enunciate the lines: the chatbot the audience to follow the conversation using instanta- effectively whispers lines into the ears of human improvis- neous translation, and allows improvisors to modify trans- ers, who are only allowed to repeat exactly those lines, but lation language settings. are otherwise free to express themselves with a full vocabu- Text-to-speech synthesis API to automatically voice lary of physicality and emotions (Mathewson and Mirowski • translations. 2018). We have adopted this configuration for the Rosetta In-ear headphone interfaces (or earpieces) enable individ- Code show. • ual performers to listen to audio translation while still be- The core idea of Rosetta Code is to build on existing, ing able to follow other conversations. Our setup to trans- state-of-the-art language technology (Section: Technology mit sound from the computer to the improviser relies on Overview) to enable a palette of improvisational games (Sec- FM radio transmitters that can multicast information to tion: Improv Games). Rosetta Code thereby allows impro- multiple FM radio receivers worn by several improvisers. visors speaking different languages to perform multilingual improv theatre together on stage. Improvisational Chatbot System To respond meaning- fully to human improvisor input utterances, the AI improv Technology Overview system works by using a statistical language model to gen- The technical setup used in this project consists of several erate sentences in continuation of some context presented elements that can be seen as independent building blocks, as text. Previous versions of AI improvisation were built each corresponding to a piece of equipment or to an Appli- upon the neural network sequence-to-sequence architec- cation Programming Interface (API). ture (Sutskever, Vinyals, and Le 2014) trained on a pseudo- Speech recognition (e.g. Google Speech-to-text API1 or translation task from the context into the generated out- • Web Speech API2), which works in multiple languages3, put (Vinyals and Le 2015). For Rosetta Code, we rely on running in a browser application. In order to successfully the GPT-2 neural network transformer architecture (Radford capture the improviser’s voice while occluding ambient et al. 2019), trained on a large corpus of web pages, which 5 noise and other performers’ voices, we rely on handheld we fine-tuned on the OpenSubtitles corpus of film subti- tles (Tiedemann 2009). 1 https://cloud.google.com/speech-to-text 2 4 https://google.com/intl/en/chrome/demos/speech.html https://cloud.google.com/translate/docs