The Mobility is the Message: the Development and Uses of MobMuPlat. Daniel Iglesia Iglesia Intermedia; Google, Inc. California, USA [email protected] Abstract ulty. Due to the fertile and wide-ranging inter- ests of both groups’ composers, both groups aim This paper documents the development of to support a spectrum of hardware and software. MobMuPlat, a software suite for running Pure Data on mobile devices. It chronicles initial The range of audio applications, drivers, and con- ensemble-oriented goals, pre- and post-launch de- troller hardware would often lead to technical is- velopment, and a few common user issues. It sues, consuming time in long rehearsals. Perform- also lists a range of projects which use MobMu- ers each had a multi-channel hemispherical speaker Plat, highlighting how the reduction of technolog- and subwoofer, combined with all the controllers ical barriers yields new and unanticipated results. and cabling required for various works. The sheer amount of physical equipment added logistical is- Keywords sues and long setup times. My goal was an ensemble performance model Pure Data, libpd, mobile, ensemble. to counteract these vexations. It would be self- contained, hand-held, mobile, battery-powered, 1 Definition with minimal software or hardware conflicts, min- MobMuPlat[1] is a platform, built on imal technical learning curve, and no cables to libpd[2,3], for running Pure Data (Pd) patches lay down ahead of time. An electronics ensemble on mobile devices. No text coding is needed. In should have the mobility of an acoustic ensemble: addition to running a patch, MobMuPlat handles the ability to walk onstage with minimal setup and the user interface, networking, interaction with perform as self-contained individual sound sources. mobile hardware and services (sensors, GPS), and Rather than sitting, tethered to laptop screens, the communication with external devices such as MIDI hardware should get out of the way, allowing mo- and HID devices. The GUI can be the "native" tion and "heads-up" ensemble interaction. Pd GUI, or a custom interface with additional fea- As someone personally invested in electronic tures and widgets suited to mobile (e.g. swipable ensemble repertoire, I also hoped for an ensemble pages, multi-touch). With a goal of development model requiring a significantly smaller budget, so and distribution on any OS, MobMuPlat is four that the model could exist beyond well-heeled mu- pieces of software: an iOS app, and Android app, sic departments. In both musical and educational an OSX editor application, and a cross-platform contexts, providing each student with a laptop is (Java Swing) editor application. In addition, mul- often infeasible, and can be an unnecessary tech- tiple networking protocols are implemented for nical distraction from an musical or educational robust group communication, reflecting MobMu- goal. This lower technical and financial barrier Plat’s original focus on ensemble performance. also meant acoustic ensembles could more easily perform pieces with electronics. 2 Origins Composers for PLOrk and Sideband predom- MobMuPlat sprang from a specific mix of in- inantly use audio software which are unavailable terests and frustrations. From 2010 to 2013, I on mobile operating systems (e.g. Max, Supercol- was a co-leader of the Princeton Laptop Orches- lider, ChucK); no one wants learn to write native tra (PLOrk)[4,5], a group comprised of rotating mobile applications for a single work. The for- classes of students. I was also a member of its tunate appearance of libpd, however, meant that sibling ensemble Sideband[6], comprised of a more composers could still develop in a familiar graph- stable group of graduate student, staff, and fac- ical audio environment, and deploy their work to mobile devices. Graphical environments also al- of my personal circle of composers and perform- low fast prototyping and tinkering in a way that ers. After launch, many other voices appeared, lower-level text coding does not, making Pd & including early users versed in Pd, users of sim- libpd (which run on nearly anything) a good in- ilar software, users in new performance config- tersection between composer-friendly and mobile- urations, and eventually longer-term users with OS-friendly. deeper needs and questions. This section tallies Unlike solo electronics performance, which MobMuPlat’s continued development after initial usually demands complex control of multiple launch. streams of audio, works for LOrks are often fairly Some missing features were apparent, and technically and instrumentally simple, with a sin- added shortly after launch. Composers work- gle synth or a sampler and a few parameters ex- ing with dynamic graphical scores had little re- posed to each player. Richness or complexity was course until an "LCD" object was added, with not required from any single player, but arose from simplified syntax inspired by Max’s LCD object. the combined behavior of the group. Mobile de- Complex audio control signals were not express- vices fit this model, in which complex GUIs, inter- ible/drawable until the "table" object was added, app routing, DAWs, or other laptop-only qualities with the ability to directly interact with Pd ta- were not necessary. In fact, the homogeneity and bles. For users looking to connect mobile to lap- limitations of mobile operating systems (in which top, direct unicast to a user-specified IP and port applications have limited power to influence the was added. By popular request, I added Audiobus overall state of the OS) is a benefit, cutting down support, allowing MobMuPlat to be a node in a on incompatibility and driver issues. It also helps larger app-to-app audio graph. that everyone already has one in their pocket. Up to this point, MobMuPlat was only on iOS; The missing pieces were: the Android version was released in late 2014. 1. A graphical user interface to control the Though it had the benefit of opening MobMuPlat patch, particularly one that was oriented to- to many, many more users, my primary motiva- wards mobile interaction (multi-touch con- tion was far more specific. The GameTrak tether trol, swiping through pages of content), controller has a unique ability to move electronics specifiable and editable as text. This would performance into physical/gestural space; it had be a set of widgets (sliders, knobs, toggles, become a staple of electronic ensemble repertoire XY sliders, etc) defined in JSON and ren- by this point[7]. HID devices remain publicly un- dered by native app code. supported on iOS, while Android, a more open platform, supports them natively. As an initial 2. Leveraging the mobile device’s sensors and goal of using mobile devices was to get perform- hardware connectivity (tilt, compass, GPS, ers away from a laptop screen, towards physical MIDI and USB controllers) as control input. heads-up performance, then supporting tether in- These would be routed between the native put was a high priority. Prime pieces of PLOrk app layer and the user’s patch. and Sideband repertoire (e.g. Jascha Narveson’s In Line[8]) were now portable to mobile. 3. Network communication (OSC, and one- Narveson also designed and wrote LANdini[9], to-many protocols that improved reliabil- a networking protocol which added guaranteed ity over pure multicast). Connectivity (ad- message ordering and delivery on top of UDP net- dresses, ports) would be managed natively, working. LANdini was an early feature of MobMu- and with messages routed to/from the the Plat, for the same reasons as tether support: to user’s patch. port existing LOrk repertoire to mobile. Its use on mobile devices, however, presented a limitation: Once the above were complete, and after initial power. LANdini was written for plugged-in lap- testing by colleagues, MobMuPlat was publicly re- tops, and so heavy CPU and messaging load was leased on iOS in January 2013. not a problem. The same load on a mobile device would yield fast battery drain (along with a no- 3 Post-launch development table heat). Though LANdini remains supported The development of MobMuPlat was influ- on MobMuPlat, a leaner counterpart was needed. enced first my own interests, then by the interests I wrote the Ping & Connect protocol in response, with a simple and unoriginal pattern. Like LAN- And here’s what is generated internally for the dini, each client pings out their existence (with an patch. "0-gui-rec" and "0-gui-send" are the optional player index, the better to refer to one handles with which the app GUI widget commu- another programmatically) and everyone creates nicates: direct connections to the pings they receive. This model is a common one, and was baked into the earliest ChucK pieces for PLOrk (e.g. Clix [10]). Though message receipt is not guaranteed, a good router will drop few packets over direct socket con- nections[11], and the performance and battery us- age is improved. After MobMuPlat had matured this far, I re- visited its overall learning curve. Creating a cus- tom interface shouldn’t always be necessary, and so I explored adding "native" Pd patch rendering. A user’s patch ought to be openable directly, dis- playing its toggles, sliders, comments, etc. Fortu- nately, work by Chris McCormick (PdDroidParty Note that there is no pass-through at the app for Android[12]) and Dan Wilcox (PdParty for GUI layer, e.g. something going to [s 0-gui-rec] iOS[13]), provided existing examples and reusable will not trigger an output in [r 0-gui-send]. code. Their pattern was to have users notate de- There are still, however, some complications. sired Pd GUI widgets with send/receives, commu- Atom widgets (e.g. float number boxes) remove nicating their state to the rest of the patch; this their inlet/outlet if the widget uses a send/receive works well with libpd’s structure, in which the na- name.