The Mobility is the Message: the Development and Uses of MobMuPlat.

Daniel Iglesia Iglesia Intermedia; Google, Inc. California, USA [email protected]

Abstract ulty. Due to the fertile and wide-ranging inter- ests of both groups’ composers, both groups aim This paper documents the development of to support a spectrum of hardware and . MobMuPlat, a software suite for running Pure Data on mobile devices. It chronicles initial The range of audio applications, drivers, and con- ensemble-oriented goals, pre- and post-launch de- troller hardware would often lead to technical is- velopment, and a few common user issues. It sues, consuming time in long rehearsals. Perform- also lists a range of projects which use MobMu- ers each had a multi-channel hemispherical speaker Plat, highlighting how the reduction of technolog- and subwoofer, combined with all the controllers ical barriers yields new and unanticipated results. and cabling required for various works. The sheer amount of physical equipment added logistical is- Keywords sues and long setup times. My goal was an ensemble performance model Pure Data, libpd, mobile, ensemble. to counteract these vexations. It would be self- contained, hand-held, mobile, battery-powered, 1 Definition with minimal software or hardware conflicts, min- MobMuPlat[1] is a platform, built on imal technical learning curve, and no cables to libpd[2,3], for running Pure Data (Pd) patches lay down ahead of time. An electronics ensemble on mobile devices. No text coding is needed. In should have the mobility of an acoustic ensemble: addition to running a patch, MobMuPlat handles the ability to walk onstage with minimal setup and the user interface, networking, interaction with perform as self-contained individual sound sources. mobile hardware and services (sensors, GPS), and Rather than sitting, tethered to laptop screens, the communication with external devices such as MIDI hardware should get out of the way, allowing mo- and HID devices. The GUI can be the "native" tion and "heads-up" ensemble interaction. Pd GUI, or a custom interface with additional fea- As someone personally invested in electronic tures and widgets suited to mobile (e.g. swipable ensemble repertoire, I also hoped for an ensemble pages, multi-touch). With a goal of development model requiring a significantly smaller budget, so and distribution on any OS, MobMuPlat is four that the model could exist beyond well-heeled mu- pieces of software: an iOS app, and Android app, sic departments. In both musical and educational an OSX editor application, and a cross-platform contexts, providing each student with a laptop is (Java Swing) editor application. In addition, mul- often infeasible, and can be an unnecessary tech- tiple networking protocols are implemented for nical distraction from an musical or educational robust group communication, reflecting MobMu- goal. This lower technical and financial barrier Plat’s original focus on ensemble performance. also meant acoustic ensembles could more easily perform pieces with electronics. 2 Origins Composers for PLOrk and Sideband predom- MobMuPlat sprang from a specific mix of in- inantly use audio software which are unavailable terests and frustrations. From 2010 to 2013, I on mobile operating systems (e.g. , Supercol- was a co-leader of the Princeton Laptop Orches- lider, ChucK); no one wants learn to write native tra (PLOrk)[4,5], a group comprised of rotating mobile applications for a single work. The for- classes of students. I was also a member of its tunate appearance of libpd, however, meant that sibling ensemble Sideband[6], comprised of a more composers could still develop in a familiar graph- stable group of graduate student, staff, and fac- ical audio environment, and deploy their work to mobile devices. Graphical environments also al- of my personal circle of composers and perform- low fast prototyping and tinkering in a way that ers. After launch, many other voices appeared, lower-level text coding does not, making Pd & including early users versed in Pd, users of sim- libpd (which run on nearly anything) a good in- ilar software, users in new performance config- tersection between composer-friendly and mobile- urations, and eventually longer-term users with OS-friendly. deeper needs and questions. This section tallies Unlike solo electronics performance, which MobMuPlat’s continued development after initial usually demands complex control of multiple launch. streams of audio, works for LOrks are often fairly Some missing features were apparent, and technically and instrumentally simple, with a sin- added shortly after launch. Composers work- gle synth or a sampler and a few parameters ex- ing with dynamic graphical scores had little re- posed to each player. Richness or complexity was course until an "LCD" object was added, with not required from any single player, but arose from simplified syntax inspired by Max’s LCD object. the combined behavior of the group. Mobile de- Complex audio control signals were not express- vices fit this model, in which complex GUIs, inter- ible/drawable until the "table" object was added, app routing, DAWs, or other laptop-only qualities with the ability to directly interact with Pd ta- were not necessary. In fact, the homogeneity and bles. For users looking to connect mobile to lap- limitations of mobile operating systems (in which top, direct unicast to a user-specified IP and port applications have limited power to influence the was added. By popular request, I added Audiobus overall state of the OS) is a benefit, cutting down support, allowing MobMuPlat to be a node in a on incompatibility and driver issues. It also helps larger app-to-app audio graph. that everyone already has one in their pocket. Up to this point, MobMuPlat was only on iOS; The missing pieces were: the Android version was released in late 2014. 1. A to control the Though it had the benefit of opening MobMuPlat patch, particularly one that was oriented to- to many, many more users, my primary motiva- wards mobile interaction (multi-touch con- tion was far more specific. The GameTrak tether trol, swiping through pages of content), controller has a unique ability to move electronics specifiable and editable as text. This would performance into physical/gestural space; it had be a set of widgets (sliders, knobs, toggles, become a staple of electronic ensemble repertoire XY sliders, etc) defined in JSON and ren- by this point[7]. HID devices remain publicly un- dered by native app code. supported on iOS, while Android, a more open platform, supports them natively. As an initial 2. Leveraging the mobile device’s sensors and goal of using mobile devices was to get perform- hardware connectivity (tilt, compass, GPS, ers away from a laptop screen, towards physical MIDI and USB controllers) as control input. heads-up performance, then supporting tether in- These would be routed between the native put was a high priority. Prime pieces of PLOrk app layer and the user’s patch. and Sideband repertoire (e.g. Jascha Narveson’s In Line[8]) were now portable to mobile. 3. Network communication (OSC, and one- Narveson also designed and wrote LANdini[9], to-many protocols that improved reliabil- a networking protocol which added guaranteed ity over pure multicast). Connectivity (ad- message ordering and delivery on top of UDP net- dresses, ports) would be managed natively, working. LANdini was an early feature of MobMu- and with messages routed to/from the the Plat, for the same reasons as tether support: to user’s patch. port existing LOrk repertoire to mobile. Its use on mobile devices, however, presented a limitation: Once the above were complete, and after initial power. LANdini was written for plugged-in lap- testing by colleagues, MobMuPlat was publicly re- tops, and so heavy CPU and messaging load was leased on iOS in January 2013. not a problem. The same load on a mobile device would yield fast battery drain (along with a no- 3 Post-launch development table heat). Though LANdini remains supported The development of MobMuPlat was influ- on MobMuPlat, a leaner counterpart was needed. enced first my own interests, then by the interests I wrote the Ping & Connect protocol in response, with a simple and unoriginal pattern. Like LAN- And here’s what is generated internally for the dini, each client pings out their existence (with an patch. "0-gui-rec" and "0-gui-send" are the optional player index, the better to refer to one handles with which the app GUI widget commu- another programmatically) and everyone creates nicates: direct connections to the pings they receive. This model is a common one, and was baked into the earliest ChucK pieces for PLOrk (e.g. Clix [10]). Though message receipt is not guaranteed, a good router will drop few packets over direct socket con- nections[11], and the performance and battery us- age is improved. After MobMuPlat had matured this far, I re- visited its overall learning curve. Creating a cus- tom interface shouldn’t always be necessary, and so I explored adding "native" Pd patch rendering. A user’s patch ought to be openable directly, dis- playing its toggles, sliders, comments, etc. Fortu- nately, work by Chris McCormick (PdDroidParty Note that there is no pass-through at the app for Android[12]) and Dan Wilcox (PdParty for GUI layer, e.g. something going to [s 0-gui-rec] iOS[13]), provided existing examples and reusable will not trigger an output in [r 0-gui-send]. code. Their pattern was to have users notate de- There are still, however, some complications. sired Pd GUI widgets with send/receives, commu- Atom widgets (e.g. float number boxes) remove nicating their state to the rest of the patch; this their inlet/outlet if the widget uses a send/receive works well with libpd’s structure, in which the na- name. This complicates the ability to connect tive app layer communicates via send/receive with to/from it. So in this case, the processing the audio patch. I, however, endeavored to use logic modifies the float box to not have internal patches without this modification, so that users send/receive names, restoring the inlet/outlet, and could drop in their patches as-is. This turned out then manually generate send/receive objects to to be non-trivial. A brief overview of the strategy mimic the intended behavior. follows. Here’s an initial state: 3.1 Native GUI processing

On selecting a patch to open, the patch (as And the post-processing state: text) is processed into two separate patches. The first is rendered as the app GUI, with auto- generated send/receive names for each widget. The second is the patch actually running within libpd, in which each widget is now surrounded with send/receive objects. All messages flowing to the patch widget are sent to the app GUI to render its visible counterpart; all messages flowing from the app GUI (in response to user interaction), are sent back into the patch widget. I attempt to illustrate this below. 4 Issues and future work Here’s an initial patch state defined by the This section covers a few common questions user: asked by users, combined with some possibilities for future development. The largest set of questions involves 1) the use of Pd-extended and external objects, and 2) the ability to generate a standalone app for an app store. To some, MobMuPlat’s ease of use im- plies that related tasks are also easy; they are not. Both these tasks are doable, but involve work- of MobMuPlat’s adoption has been the surprising ing with native mobile source code and IDEs, at ways in which it has been used. Some projects which point most users understandably lose inter- continue common uses of Pd (e.g. solo perfor- est. (The whole point of MobMuPlat, after all, mance, acoustic instrument augmentation, instal- was to avoid needing to do that.) Regarding stan- lation), while some highlight new use as ensemble dalone apps, I believe that unless one wishes to instrument or embedded device, and many high- monetize an app (which is difficult without a slick light uses completely specific to the nature of mo- custom interface), or one is paranoid about people bile devices (e.g. toys, running). This section con- looking at one’s patch, that installing MobMuPlat tains a sample of the ways MobMuPlat has been (or PdParty, PdDroidParty, etc) and sharing the used. patch is usually much simpler (and more open). • Mobility-based ensembles: Though The inability to create and monetize a standalone mobile-device-based ensembles have existed app also allows the community to avoid the ethi- for some time[14,15], some new ensembles cal complaints, occasionally made on Pd forums, specifically use mobile devices in order to about open source and personal/commercial gain. roam around public space. These include However, the questions regarding sharing an outgrowth of PLOrk[16] (with hand-made patches coincides with a related set of questions re- controllers), and El Smartphone Ensemble in garding mass distribution. A composer/performer Colombia[17,18]. I discuss my own mobile may wish to distribute a large-group work (e.g. a ensemble in the following section. sound walk or audience-participation piece) and Demonstration of scientific research hope to do so in as few steps as possible. Right • : now, there is no faster way than installing MobMu- MobMuPlat was used to demonstrate how Plat, and then downloading the patch and inter- the interaction of fireflies can be a model face files onto the device, and then returning to for synchronization within a group of de- MobMuPlat to select them. Future work may at- vices[19,20]. tempt a method of "pushing" patches to connected • Instrument design for solo perfor- devices (though this has the additional hurdle of mance systems: Toby Hendrick (aka otem requiring the group to connect to a local router); rellik) has built a suite of MobMuPlat-based this may help for small collaborative groups, but I instruments for his live rig[21]. doubt it will solve the large-group problem. • Acoustic instrument extension: For While I consider development of MobMuPlat real-time acoustic processing, mobile phones to be approaching feature completion, there’s still are smaller and less distracting than laptops, a few more open considerations for the future. and easier to deploy than embedded com- Watch and other wearable support is of interest, puters. Some examples include porting a since it allows even more heads-up musical inter- processing chain for trombone performance action. MobMuPlat has Android Wear support (including AudioBus)[22] and an embedded (for specifying watch widgets that communicate processor for violin[23]. with the mobile device) for some time; I person- ally use this to control a mobile device while us- • Concert works with audience interac- ing tether controllers. However I’ve not yet pub- tion: Celeste Oram recreated Vera Wyse licized this feature, nor investigated parity with Munro’s 1940 experimental work Skywave Apple watches, nor implemented editor support. Symphony, using audience’s mobile devices Additionally, my interest in networking protocols in place of radios[24]. will likely continue, and there’s constant discus- • Sound mapping, sound walks, and per- sion of new protocols; if any gain traction with a sonal surveillance: Some projects focus on large community I may consider integration. (At group experience in a specific location, in- the time of this article, there’s some chatter about cluding works for Seoul, Korea by Max Ne- Ableton’s new standard. No promises.) upert[25] and Hailuoto, Finland by Antye 5 Uses Greie-Ripatti[26]. Others create conceptual works for personal, private experience, such MobMuPlat was originally conceived with spe- as Oram’s Soft Sonic Surveillance[27] and cific uses in mind, but the primary personal benefit Beatrice Monastero’s Wandertroper[28]. • Toy development: One of the most en- dearing and unexpected uses of MobMuPlat is Notori, a project to recycle mobile phones into toy components[29,30].

• Research on running and music: Multi- ple projects have used MobMuPlat to mea- Two pages of the accompanying watch GUI, for sure how music motivates running perfor- selecting presets and controlling a subset of synth mance[31,32]. parameters. 5.1 The Google Mobile Orchestra The basis of our networking allows any per- The Google Mobile Orchestra (GMOrk), per- former to change any parameter of any other per- forms on tablets, with tether controllers, portable former. This can be useful to impart timbral or speakers, networking, and live video. It began in rhythmic unity when desired, and to inject chal- the LOrk model, with a repertoire of works, both lenging elements (e.g. "everyone send your com- adapted from the LOrk repertoire, and newly com- mands to your left") which yield unanticipated re- missioned for mobile. The LOrk model, based on sults. This pattern is indebted to early network changing student membership with varying lev- improvisers, particularly The Hub[33]. As every els of musical experience, results in works often musical parameter is broadcast over the network, using fairly simple high-level control of a musi- a central computer can then act as a recipient of cal process, within a directed score. The mem- this traffic, and re-renders the audio of all the play- bers of GMOrk, however, felt that lower-level con- ers, using the audio to render real-time video for trol, in a more self-directed improvisational con- 3D glasses[34]. text, would suit us better. While we retain the ex- References isting externally-composed repertoire, we have de- veloped our own performance patches with many [1] www.mobmuplat.com low-level parameters, custom real-time remapping [2] www.libpd.cc to physical controllers, and heavy use of network- [3] Brinkmann, Peter, et al. "Embedding Pure ing. Data with libpd." Proceedings of the Pure Data Convention. 2011. link [4] plork.princeton.edu [5] Trueman, Daniel, et al. "PLOrk: the Princeton , year 1." Proceedings of the In- ternational Conference. 2006. link [6] www.sidebandband.com [7] Freed, Adrian, et al. "Musical Applications and Design Techniques for the Gametrak Teth- ered Spatial Position Controller." Proceedings of the 6th Sound and Music Computing Confer- ence. 2009. link [8] www.sidebandchronicles.com/jascha- narveson-in-line/ [9] Narveson, Jascha, and Dan Trueman. "LAN- dini: a Networking Utility for Wireless LAN- based Laptop Ensembles." Proceedings of the Sound and Music Computing Conference. 2013. link [10] Smallwood, Scott, et al. "Composing for Lap- A page of a GMOrk interface, enabling control of, top Orchestra." Computer Music Journal 32.1, and tether mapping to, synth parameters. 2008. link [11] Cerqueira, Mark. "Synchronization over Net- [22] no-insects.blogspot.com/2014/10/meta- works for Live Laptop Music Performance." trombone-on-.html Master’s Thesis, Department of Computer Sci- [23] Overholt, Daniel, and Steven Gelineck. "De- ence, Princeton University. 2010. link sign & Evaluation of an Accessible Hybrid Vi- [12] http://droidparty.net/ olin Platform." Proceedings of the International Conference on New Interfaces for Musical Ex- [13] github.com/danomatika/PdParty pression. 2014. link [14] Wang, Ge, Georg Essl, and Henri Penttinen. [24] verawysemunro.nz/re-enacting-the- "Do Mobile Phones Dream of Electric Orches- Skywave-Symphony tras?" Proceedings of the International Com- puter Music Conference. 2008. link [25] choejeongeun.com/xD25-1-GPS-Audio-Walks [15] Oh, Jieun, et al. "Evolving The Mobile Phone [26] medium.com/@poemproducer/when-you- Orchestra." New Interfaces for Musical Expres- listen-it-listens-back-4c4d99bea8f8 sion. 2010. link [27] workandthe.work/II-Soft-Sonic-Surveillance [16] Snyder, Jeff, and Avneesh Sarwate. "The Mo- www.dawn.ul.ie/?/interactive-media/ Proceedings of the [28] bile Device Marching Band." 2014/BeatriceMonastero/ International Conference on New Interfaces for Musical Expression. 2014. link [29] www.yuichirock.com/notori/ [17] sonologiacolombia.wordpress.com/lab/ [30] Katsumoto, Yuichiro, and Masa Inakage. ensamble-de-smartphones/ "Notori: Reviving a Worn-Out Smartphone by Combining Traditional Wooden Toys with Mo- [18] J. J. Arango and D. M. Giraldo, "The Smart- bile Apps." SIGGRAPH Asia 2013 Emerging phone Ensemble. Exploring Mobile Computer Technologies. 2013. Mediation in Collaborative Musical Perfor- mance." Proceedings of the International Con- [31] www.marcobasaglia.com/improving- ference on New Interfaces for Musical Expres- running-motivation sion. 2016. link [32] Forsberg, Joel. "A Mobile Application for Im- [19] Nymoen, Kristian, Arjun Chandra, and Jim proving Running Performance Using Interactive Tørresen. "Firefly with Me: Distributed Syn- Sonification." 2014. chronization of Musical Agents." Awareness [33] Gresham-Lancaster, Scot. "The Aesthetics Magazine. 19 November 2013. link and History of The Hub: The Effects of Chang- [20] www.vimeo.com/67205605 ing Technology on Network Computer Music." Leonardo Music Journal. 1998. p39-44. link [21] www.youtube.com/playlist?list= PLpP3G7pBTV-IiLO1oSdR7hI79sktGp3kb [34] www.vimeo.com/178834011