A Sensor-Driven Generative Music Platform
Total Page:16
File Type:pdf, Size:1020Kb
PourOver: A Sensor-Driven Generative Music Platform Kevin Schlei Christopher Burns Aidan Menuge University of University of Michigan University of Wisconsin-Milwaukee 1100 Baits Drive Wisconsin-Milwaukee 3223 N. Downer Ave. Ann Arbor, MI 48109-2085 3223 N. Downer Ave. Milwaukee, WI 53211 [email protected] Milwaukee, WI 53211 [email protected] [email protected] ABSTRACT [6] push the concept of `open works' that invite user input The PourOver Sensor Framework is an open iOS framework to guide playback [5]. designed to connect iOS control sources (hardware sensors, The role of the mobile device user as both audience mem- user input, custom algorithms) to an audio graph's param- ber and participant is explored in echobo [10] and Nexus eters. The design of the framework, motivation, and use [1]. In echobo, audience members join a `master musician' cases are discussed. The framework is demonstrated in an via a network, who controls high level musical structure. end-user friendly iOS app PourOver, in which users can run The NEXUS system provides a web-based distributed per- Pd patches with easy access to hardware sensors and iOS formance interface. Each blurs the performer / audience APIs. barrier through individual contributions to a larger collab- orative performance. PourOver looks to apply this model to an individual listening experience by following many am- Author Keywords bient, environmental, motion, and location changes. generative music, mobile music, sensor mapping The MoMu framework demonstrated how quickly acces- sible device parameters, paired with an included audio en- gine, could speed development of sensor-rich audio applica- ACM Classification tions [3]. The PSFramework aims to solve similar design [Applied computing] Sound and music computing, [Human- problems, but via a more generic protocol in which custom centered computing] User interface design, [Software and its objects can easily register as controller sources, and the tar- engineering] Software design engineering geted destination is not a particular audio engine. There are a number of projects that provide on-device GUI interactions to an underlying audio engine. The NexusUI 1. INTRODUCTION JavaScript framework enables rapid GUI prototyping for Generative music can be difficult to distribute in a way that instrument interfaces [11]. These pieces and instruments preserves its generative qualities. Fixed recordings can cap- expose their interactivity through the portal of the device ture a particular performance, but may be interpreted as a screen. canonic representation by the listener, especially upon mul- The urMus system provides a number of solutions for mo- tiple listenings. bile music creation: sensor streams, value scaling, controller Generative pieces can use sensors or external input sources mapping, and networked live-coding all integrated with the to add an interactive dimension to performance. Adding built-in audio engine [7][8]. Through the urMus interface, hardware requirements to the performance of a piece fur- musicians can design and perform pieces that leverage the ther complicates its distribution. device's multitude of interfaces. The system also allows for PourOver is an iOS app that provides a platform for cre- networked live coding where a desktop user can push code ation, playback, and distribution of sensor-driven algorith- to another user's mobile device [9]. This design solution, mic / generative pieces. It aims to be a music player app where some coding was split from the device, was a signifi- built for generative music. cant inspiration for this research. The underlying controller management is handled by PS- RjDj1 was an iOS app that filtered external audio to Framework, a new open-source framework for iOS. PSFrame- create a reactive listening experience. Later, a desktop work is built as a generic and extendable controller and API app, RJC1000, allowed users to customize their own RjDj management system. `scenes.' The PourOver app and framework also aims to give listeners a reactive listening experience. However, us- ing Pd to create pieces affords a more general, open, and 2. RELATED WORK customizable content creation system. The iOS platform has provided tools for distributing gener- Sensors2PD is a framework designed to quickly link An- ative, interactive, or multimedia-rich music apps. `Album droid device sensor data to Pd patches, and is the most apps' such as Bj¨ork's Biophilia [2] and Brian Eno's Bloom similar in scope to PourOver [4]. The system sends con- troller values to numbered receivers (sensor1v0, sensor1v1, etc.). The system also supports touch input (sensorTIDvx, sensorTIDvy) and WiFi router information (sensorW-ID). PourOver looks to extend this kind of connectivity with a Licensed under a Creative Commons Attribution generic protocol design and expandability. 4.0 International License (CC BY 4.0). Copyright remains with the author(s). NIME’16, July 11-15, 2016, Griffith University, Brisbane, Australia. 1 . http://rjdj.me/ 355 3. PSFRAMEWORK instance of that class, or use the existing instance. This The PourOver Sensor Framework (PSFramework) was de- setup avoids unnecessary instantiation of objects, excess veloped by the authors to provide simple connectivity from battery drain, and memory pressure. iOS data streams to an audio engine's parameters, while Once a generator is up and running, it handles the pulling automatically handling instantiating and clean up of source of data streams from its model. It then calls the PSCon- APIs. A variety of included control generating sources pro- trollerUpdating method updateValue:forControllerNamed: vide quick access to common sensor data. to send an updated data value through its corresponding The framework is open and extendable, allowing for new PSController and on to its final destination. hardware sensors or custom controller streams to be added 3.3 PSControllerDelegate as they appear. Programmers can engage with the iOS side of an app by creating and registering their own controller To install PSFramework, instantiate a PSControllerCoor- generating objects with the framework. dinator object and assign its controllerDelegate prop- PSFramework will be available on Github and installable erty. The PSControllerDelegate handles the reception of via CocoaPods. all PSController value changes. From there it can pass the updated value to the desired target. 3.1 PSController 3.4 Activity Modes and Controller Ranges The framework is built around managing PSController ob- jects. A PSController defines a data stream by name and Sensor-driven applications normally need to scale sensor transmits values in a normalized range. values to an appropriate range, since sensors can produce For example, a PSController can be created to handle wildly different values depending on the use case. Present- pitch attitude data from the gyroscope sensor. The con- ing the complete range of a sensor in all cases only passes troller would be initialized with the following properties: the issue to the user, who then must test for a practical range and scale their data accordingly. PSController( PSFramework attempts to solve this issue by supporting name:"CMMotionManager.deviceMotion.attitude.pitch", four different activity modes during playback: walking, run- min:-M_PI / 2.0, ning, cycling, and automotive. These modes were chosen to max:M_PI / 2.0 match the output of CMMotionActivityManager, an object ) that analyzes the current activity of the device. CMMotionActivityManager supplies real-time updates of Controller names should follow a DNS-like, `member of' the device activity along with an estimated confidence naming convention. In this instance, UIKit's CMMotion- property. In tests, the ability to predict whether the de- Manager class has a deviceMotion property, which has an vice was stationary or in motion worked fairly well, but was attitude property, which is a struct that contains pitch. skewed towards moving. Using the confidence property to While this naming convention is verbose, it creates a strong filter out less confident reports fixed the issue. bridge between the PSController name and the API it is A PSController can store separate value ranges for each referencing. of the four activity modes. When an activity mode is re- The gyroscope pitch sensor outputs values from -π/2 to ported, the PSController object checks for a range for that π/2. In this case, the sensor range is a fixed limit. In others, mode and adjusts its minimum and maximum values ap- such as CMPedometer.speed, a best estimate can be used for propriately. PSControllers that do not use different ranges expected controller output ranges. PSController requires a will ignore the change. value range because it normalizes its output. More infor- This system allows pieces to produce a similar outcome mation on controller ranges is discussed in 3.4. regardless of whether the user is walking or driving. Con- PSControllers are designed to minimize excessive updat- versely, a piece could be designed to follow mode changes ing. They store a sentValue property, which is checked and alter the music as they occur. for equality on subsequent updates. They also have an active property to indicate whether a receiver exists for 4. POUROVER APPLICATION that controller. PSControllers will avoid sending superflu- The iOS application PourOver has two major goals: provide ous messages based on these checks. This functionality can a platform to play sensor-driven algorithmic / generative be overridden in cases where the behavior is undesired, for