SWARMED: Captive Portals, Mobile Devices, and Audience Participation in Multi-User Music Performance Abram Hindle Department of Computing Science University of Alberta Edmonton, Alberta, Canada [email protected] ABSTRACT In this paper we present and propose a method of allow- Audience participation in computer music has long been ing anyone with a web-enabled mobile device to participate limited by resources such as sensor technology or the mate- in a performance by providing an interface to a shared com- rial goods necessary to share such an instrument. A recent puter music instrument over a wireless network (WiFi), or paradigm is to take advantage of the incredible popularity an Internet connection, via an interactive webpage. We di- of the smart-phone, a pocket sized computer, and other mo- rect users directly to the relevant pages by taking advantage bile devices, to provide the audience an interface into a com- of captive-wifi portal/hotspot technology. Thus the issue of puter music instrument. In this paper we discuss a method distributing and downloading an app is avoided, as the web- of sharing a computer music instrument’s interface with an interface of the instrument is distributed locally via WiFi audience to allow them to interact via their smartphones. or globally via the internet. We propose a method that is relatively cross-platform and Providing internet access to 20 or more devices can often device-agnostic, yet still allows for a rich user-interactive be prohibitive due to bandwidth constraints and latency is- experience. By emulating a captive-portal or hotspot we sues, thus in this paper we focus mostly on providing instru- reduce the adoptability issues and configuration problems ment interfaces via a local wifi-network configured much like facing performers and their audience. We share our experi- a captive-wifi portal in a hotel. Hotel wifi networks are often ences with this system, as well as an implementation of the captive-wifi portals that require authentication or payment system itself. in order to enable internet access, these captive portals in- tercept and replace your hypertexttransportprotocol (HTTP) requests with an authentication and payment interface. In- Keywords stead of asking you to authenticate, one can use a captive- Wifi, Smartphone, Audience Interaction, Adoption, Captive portal to drive the smart-phones to the computer music Portal, Multi-User, Hotspot instrument’s interface. For the rest of the paper we will describe and discuss SWARMED: a Shared Wifi Audience Reactive Musical Ex- 1. INTRODUCTION temporization Design. SWARMED is a captive-wifi-portal Audience interaction has classically been limited by phys- that serves up web-based user interfaces which allow users to ical and monetary resource constraints. If one wanted to interact with and control a running computer music instru- share an instrument with an audience and allow them to ment with their web-enabled devices such as smart-phones, interact often an entire installation fitted with sensors was tablets and laptops. SWARMED solves many tool adop- needed, or the audience needed to provide their own materi- tion and portability issues by relying on standardized and als. Smartphones have gained popularity in computer music portable web technologies to deliver interactive instruments. performance, with smart phone orchestras and groups be- Thus our contributions in this paper are: ing formed [10]. Often these groups share the same apps • We propose using captive-portal to drive WiFi users and play together either through their speakers, or over an instrument’s user interface (UI); MIDI, or via a mixer. One barrier to audience interaction in the case of smartphones was coding something that was • We describe the SWARMED system abstractly and cross-platform enough to run on the major vendors smart- concretely; phones [17]. In the case of the iPhone, one would even • need a developer license and their application would have We provide a free open-source implementation of the to appear in the app store. Yet aiming for the smartphone SWARMED system [5]; makes sense, each audience member essentially is bringing • We discuss our experience performing with SWARMED their own computer and running your software. Yet the and suggest future uses for such a system; least restrictive and most cross-platform method of utiliz- ing smartphones is not apps, it is their ability to view and 2. PREVIOUS WORK interact with webpages [1]. In this section we discuss some of the previous work rel- evant to audience collaboration, adoptability, and the use Permission to make digital or hard copies of all or part of this work for of portable digital devices in audience-interactive computer personal or classroom use is granted without fee provided that copies are music performances. not made or distributed for profit or commercial advantage and that copies In terms of experiments and discussions regarding audi- bear this notice and the full citation on the first page. To copy otherwise, to ence participation using smartphones and mobile devices republish, to post on servers or to redistribute to lists, requires prior specific Oh et al.[11] in “Audience-Participation Techniques Based permission and/or a fee. NIME’13, May 27 – 30, 2013, KAIST, Daejeon, Korea. on Social Mobile Computing” argue that smartphone ori- Copyright remains with the author(s). ented performances are convenient for both the audience 174 Instrument Server Handler Interactive HTML5/Javascript Application PA System Instrument Webserver Mixer or User Interface ... Audio Stream Name Software System/Component Unidirectional Communication Instrument Server Bidirectional Communication Handler ... 0 or more components Figure 1: Abstract Software Architecture and Communication of Software Components of the SWARMED Network and performer. Furthermore they discuss different kinds of ware [14]. Adoptability is different than portability. Porta- social interaction ranging from social network interaction bility is about whether or not software can run on a plat- (Tweetdreams [3]), through to interactive web applications. form, adoptability is whether or not the software is com- They highlight that audience interaction does not have to patible enough with a user such that the user adopts and be musical in nature, it can be about communicating back uses a tool. In the case of a computer music interfaces on to the performers or researchers. mobile devices, both portability and adoptability are im- Unsilent Night is an example of audience based participa- portant as the audience will have a wide range of mobile tion and adoption. Unsilent Night, by Phil Kline [9], invited devices (portability) but they will receive minimal training audience members to join in a performance of playing a lim- and yet must navigate to and learn the user interface (UI) ited set of audio samples in an open space with their own quickly: they will have to adopt the use of the UI. Soft- boom-boxes and portable speakers. Kline also provided an ware engineers try to make tools adoptable by integrating iPhone app one can install in order to participate. their tools into user interfaces that users already use (like TweetDreams, by Dahl et al. [3], is an excellent example Microsoft Office) [14]. of audiences interacting with a music instrument via a web Thus we propose a system different from previous systems interface (Twitter). The audience participates by tweeting that addresses user adoption via captive-portal technology, at the performer’s Twitter account. Their architecture is addresses portability via web-standard browsers, and at- very similar to SWARMED except that they rely on Twitter tempts to enable audience collaboration by leveraging the to mediate the messages. TweetDreams requires an internet mobile devices of the audience. connection and the use of Twitter. From an infrastructural point of view, Jesse Allison [1] describes an implementation of a distributed performance 3. ABSTRACT FRAMEWORK system that is web-based (HTML5) and uses an application In this section we describe the abstract structure of a SWA- webserver (Ruby on Rails). Allison argues that adhering to RMED instrument. A SWARMED instrument would in- HTML5 standards allows for client-side instrument UIs to clude the user, the performer, the network, the infrastruc- be distributed without having to address excessive porta- ture, and the computer music instrument under control. bility issues. This is depicted in Figure 1. massMobile by Weitzner et al. [17] describes a system The user interacts with the instrument by logging onto a meant to allow smartphones to interact with and Max/MSP captive-wifi-portal, a wireless network. The captive-portal via web standards (HTML5), over a variety of wireless net- whisks the user’s web browser away to an initial webpage works. Their configuration includes a persistent internet that determines the capability of the the user’s mobile de- accessible sound server running Max/MSP. Our work dif- vice and webbrowser and then navigates the user to appro- fers because we leveraged a captive-portal technique and do priate and compatible user interfaces. not necessarily rely on Max/MSP (although we tended to When the user interacts with the instrument their browser use CSound [15] on the backend). sends HTTP commands over the wireless network back to Tanaka [13] carefully evaluates the capabilities of mobile an application webserver. This application webserver is devices and their affordances for mobile music performance dedicated to passing on the user’s input to the computer and interaction. Tanaka’s work seeks to address the lim- music instrument. Application webservers, such as Ruby itations and affordances of the mobile devices and argues on Rails [1] or Mongrel2 [12], are used because they allow that developers of mobile instruments should pay attention handlers to run, which are functions or processes (running to how users use their devices. For instance many users programs) that respond to a request. These handlers often hold their phone with one hand and manipulate it with the interact with longer running processes, such as an instru- other, thus two-handed interaction is not an affordance of ment.