A JavaScript Framework for Crowdsourced Human-Robot Interaction Experiments: RemoteHRI Finley Lau,1 * Deepak Gopinath, 2,3 * Brenna D. Argall1,2,3 * Equal contribution 1Department of Computer Science, Northwestern University, Evanston, Illinois 2Department of Mechanical Engineering, Northwestern University, Evanston, Illinois 3Shirley Ryan AbilityLab, Chicago, Illinois fi[email protected], [email protected], [email protected] Abstract maintain stricter social distancing protocols, rendering an in- person study practically impossible. Particularly, in the sub- In this paper, we present RemoteHRI, a JavaScript (JS) soft- domain of assistive robotics, the end-user population (peo- ware framework for conducting Human-Robot Interaction (HRI) experiments in a web browser. Built with HRI re- ple with motor-impairments as a result of trauma or neuro- searchers in mind, RemoteHRI includes a flexible set of soft- degenerative disease) typically belongs to a high-risk group. ware tools that allows for rapid prototyping and quick de- Therefore, participation in an in-person study can pose sig- ployment of a wide range of laboratory-like experiments that nificant health risks. can be run online. RemoteHRI uses the state-of-the-art Re- 1 By contrast, simulation-based experiments are less expen- actJS framework to build standard HRI stimulus environ- sive and can enhance the usefulness of in-person studies. By ments such as grid worlds, differential drive cars, and robotic arms. As a result, the researcher can solely focus on the conducting experiments online, researchers can parallelize experimental design thereby saving valuable time and ef- data collection and achieve greater diversity in the recruit- fort. Code for RemoteHRI is available at https://github.com/ ment pool. Furthermore, online settings remove the need for argallab/RemoteHRI. in-person supervision, alleviating the health risks of being in close proximity to others. Exploratory HRI studies con- ducted in a simulated environment can also provide strong 1 Introduction priors, in terms of baseline models for human behavior and Evaluating progress in science and engineering heavily de- initial evaluation of robotics algorithms. These studies ad- pends on proper experimentation protocols. In the domain ditionally inform resourceful design of a real-world experi- of Human-Robot Interaction (HRI), proper experiments are ment. Performing human subject studies in a simulated en- needed both to understand the human decision-making pro- vironment is not novel. For example, jsPsych (De Leeuw cess while interacting with robots and to evaluate the success 2015) is a popular JavaScript library used by researchers of robotics autonomy algorithms in interacting with humans in the field of psychology to perform simple online behav- and other entities. ioral experiments. However, the stimulus plugins available In an ideal situation, researchers conduct HRI experi- in jsPsych do not cover the full space of rich and com- ments with real robotic systems. This provides researchers plex stimuli needed for HRI. The field of HRI research cur- with rich data that encodes the sensing and actuating com- rently lacks a comprehensive software framework for con- plexities of robot operation in the real world. However, de- ducting simulation-based experiments and relies on one-off signing and conducting HRI experiments on real robotic sys- solutions developed for specific projects, leading to wasted tems come with a great deal of challenges, especially in the time and repeated effort. Although primarily used for rein- academic setting. forcement learning research, OpenAI’s Gym (Brockman et First, real robotic systems are expensive, and academic al. 2016) provides a suite of simulated environments that are labs rarely own multiple robots of the same type. This dras- widely used for HRI experiments (Schaff and Walter 2020; tically limits the number of studies that could be run in par- Broad et al. 2020); however, the JS interface needed for allel, thereby making data collection extremely slow. Sec- crowd-sourced online studies is still under development. ond, subject recruitment for academic studies can suffer RemoteHRI provides HRI researchers a set of tools to from biases due to lack of diversity in the recruitment pool. design and conduct common HRI studies in a seamless Third, in-person subject studies typically require researchers manner. The framework is designed with the researcher in to be in close contact with the subjects. However, in light mind, specifically focusing on ease of use and rapid proto- of COVID-19, academic researchers are now required to typing. RemoteHRI derives its inspiration from frameworks Copyright c 2020, Association for the Advancement of Artificial such as jsPsych and supports flexible experiment design, an Intelligence (www.aaai.org). All rights reserved. easy-to-use researcher interface, and prepackaged standard 1https://reactjs.org/ HRI stimulus environments. RemoteHRI uses ReactJS for client-side rendering, which allows for increased flexibility, • Training Phase: In the training phase, the participant gets speed, and ease of development through its management of familiarized with the experimental setup. This phase also state and structural organization of components (groups of helps researchers establish a baseline performance, which elements displayed on the screen) as compared to vanilla could be a useful measure in data analysis. JavaScript. • Testing Phase: The testing phase typically consists of The key features of RemoteHRI are as follows: multiple blocks of experimental trials under different ex- • Researcher-Centric Design: RemoteHRI was built with perimental conditions. This phase may serve to collect researchers from different backgrounds and programming data (for example, human teleoperation data of robotic experience in mind and provides a low entry point for manipulators to build data-driven computational models easily prototyping and building HRI experiments, without of human decision making) or to evaluate an algorithm’s having to develop the client-side application (which typ- performance (for example, evaluation of a shared control ically requires researchers to learn to use graphics APIs, robot policy for assistive robotic manipulators). and physics libraries) or the server-side application. • Survey Phase: The participant is presented with ques- • Modular/Extensible: RemoteHRI is highly modular in tions related to their experience of interacting with the that the different applications of the framework operate robotic system. independently of each other and may be replaced with an Note that in a given experiment, possibly with the ex- equivalent module. For example, the RemoteHRI server ception of the consenting phase, phases may occur multi- could be replaced with a custom-built server without af- ple times in no particular order. RemoteHRI recognizes the fecting the client-side application. Furthermore, imple- need for such combinatorial flexibility in experimental de- mentation of different stimulus types is similar and there- sign and offers researchers an easy approach to specify any fore the framework can be easily extended to work with experimental flow. new types of stimuli. • Unified Researcher Interface: Regardless of the spe- 3 Framework Modules cific experimental domain or design choice, the researcher RemoteHRI consists of two main software modules: 1) the specifies experimental flow through a single JSON file, client-side application and 2) the server-side application. which we refer to as Experiment.json (discussed more in Participants interact with the client-side application in their detail in Section 4). RemoteHRI provides a GUI inter- browsers during an experiment, while the server-side appli- face and a set of utility functions allowing researchers to cation ensures the proper delivery of experiment content. In quickly generate Experiment.json. the subsequent subsections, we will describe the details of • Plug and Play: RemoteHRI experiments are completely the client-side and the server-side applications. controlled by the Experiment.json file. By specifying the stimuli and their respective properties used in the experi- 3.1 Client ment in Experiment.json, researchers can create entire ex- The client-side application is responsible for presenting the periments without changing the client-side application. stimulus for each trial during the course of the experiment. In Section 2 we describe a typical HRI experiment de- In RemoteHRI we consider two main classes of stimuli: sign and how it informs some of the design choices for Re- • Passive: A passive stimulus is an environment in which a moteHRI. The main software modules that constitute Re- participant ‘passively’ observes the presented stimuli (au- moteHRI are described in Section 3. The researcher inter- dio/video/text). For example, participants could be shown face for experiment specification is presented in Section 4, images of two different humanoid robots and asked which followed by conclusions and future work in Section 5. one has more human-like features. Participant responses to a passive stimulus depend on the stimulus type and can 2 HRI Experiment Design take many forms such as single answer/multiple choice, A quick analysis of different types of experiments conducted multiple answer/multiple choice, 5- or 7-point Likert in the field of HRI reveals various commonalities in ex- scale, or free text response. RemoteHRI provides a suite perimental