AuSynthAR: A simple low-cost modular based on Augmented Reality

Pedro Lucas Independent Researcher Guayaquil, Ecuador [email protected]

ABSTRACT in the musical and educational fields. AuSynthAR is a digital instrument based on Augmented Re- ality (AR), which allows sound synthesis modules to create 2. SYSTEM DESIGN simple sound networks. It only requires a mobile device, 2.1 Physical Space a set of tokens, a sound output device and, optionally, a The setup depicted in (Figure 1) shows the following rele- MIDI controller, which makes it an affordable instrument. vant elements: A mobile device, where the audio synthesis An application running on the device generates the sounds engine and the augmented reality application are executed; and the graphical augmentations over the tokens. a set of tokens, which are the fiducial markers that denotes sound modules; a plastic holder for the device, which allows Author Keywords the performer to interact with both hands; and, a MIDI Augmented Reality, Sound Synthesis, Mobile Music controller, which enables the user to play musical notes and increase the possibilities for controlling the virtual modules. CCS Concepts Speakers or any other sound output device can extend the physical setup. •Applied computing → Sound and music computing; •Human-centered computing → Mixed / augmented reality;

1. INTRODUCTION In the field of music interfaces, Augmented Reality (AR) has been used for controlling synthesizer parameters on high expressive setups, making music through unusual objects and exploring new kinds of instruments [3] [5] [1]. The related work remarks the use of AR as an important and novel element to enrich experiences for digital instru- ments, like the Reactable [2], which is a successful example that uses AR to achieve high expressiveness and collabora- tion for creating interesting sound materials. Some AR applications require expensive equipment [2] [1]; however, the computational power of mobile devices al- Figure 1: AuSynthAR physical setup. low us to implement affordable AR tools. In previous work, such as in [4], an AR electrical circuit simulator was im- The relationships among the hardware components are plemented and tested in a learning environment. In this shown in (Figure 2). The performer interacts with the to- proposal, the circuit simulator was replaced by a Networked kens and the input devices, which generate data that is pro- Modular Synthesizer; hence, the Audio Synthesis with Aug- cessed by the mobile device. The screen acts as a visualizer mented Reality - AuSynthAR was conceived. since the actions are performed with the fiducial markers. This work describes AuSynthAR as a low-cost synthesizer The built-in camera allows the device to gather the input based on AR for mobile devices, complemented with a tan- from the markers, and render the virtual augmentations gible user interface (TUI) which consists of tokens and a that produce sounds through an output device, according plastic holder. Optionally, MIDI controllers can be added to the internal synthesis engine. to the configuration. The synthesis approach is designed under the concept of a modular synthesizer. 2.2 Virtual Space Future directions for developing high expressive instru- The virtual objects are the graphics and audio modules that ments are exposed, as well as opportunities to conduct tests support the generation of sounds due to the physical ob- jects. The virtual space consists of the software components described in (Figure 3). AuSynthAR is a real-time system that is feeding from two Licensed under a Creative Commons Attribution input sources: A camera and external devices. The data 4.0 International License (CC BY 4.0). Copyright collected by the camera is managed through the camera remains with the author(s). controller and sent to an image recognition process, which NIME’19, June 3-6, 2019, Federal University of Rio Grande do Sul, identifies the fiducial markers and maps to the correspond- Porto Alegre, Brazil. ing virtual object on each of them according to a strategy

405 Performer Mobile Device

Screen

Fiducial Marker Central Processing Unit Audio Input Device

Built-in Camera Source Control or Effect

Sound Processor

Audio Output Device

Figure 4: Sound synthesis network.

Figure 2: Hardware components for AuSynthAR. For hearing the resulting audio configuration, the per- former needs to connect the last part of the chain to the denoted as the coordinates mapping in virtual world, which output module. The network presented in (Figure 4) shows translates information from the real world to the virtual en- the described elements in a real-time operation. vironment. Then, the graphics engine renders the 3D aug- mentations over the physical objects through the 3D objects 3. CONCLUSIONS and GUI rendering module. Finally, the synthesis engine This work presents AuSynthAR, a novel musical interface generates audio by using the synthesis modules according to that uses augmented reality (AR) technology for modular the connections and manipulations given by the performer sound synthesis with high accessibility. The proposed de- when using the markers; also, the system provides graphical sign establishes a physical setup as a combination of hard- feedback as well as waves visualizations. ware and software components supported on mobile devices. There is a high potential of this musical interface for Camera Controller novice music performers and learners regarding sound syn- User’s Input Controller thesis. Previous applications of AR denote the interest

Image Recognition Process of users in manipulating physical and virtual elements for

Graphical Feedback and achieving their creative goals, while overcoming the chal- Sound Synthesis Engine Wave Visualizations Coordinates Mapping lenges of connecting networks of components. The main in Virtual World contribution of this work is the construction of an affordable Synthesis Modules tool that can be installed and used easily by audio artists 3D Objects and GUI Rendering and people in general, to experience the block construction of sound and music. For future work, human-interaction testing focused on ex- Figure 3: Software components for AuSynthAR. pressiveness will be conducted to improve the design of this interface. Questionnaires and automatic logs are planning The software application for this prototype was imple- to be used. Also, an AR headset systems will be considered. mented in Unity 3D Game Engine and powered by the Other assessments will also be taken into account, such as augmented reality framework Vuforia. The application was learning capabilities on academic courses related with sound built for Android devices with MIDI support. synthesis. 2.3 Sound Synthesis Sound generation through the mobile device is achieved by 4. ACKNOWLEDGMENTS making the calculations of the samples that are going to The author would like to thank Dr. Enrique Pel´aez, profes- be played. Unity gives the possibility to procedurally build sor at ESPOL University, for his constructive criticism of audio via the OnAudioFilterRead callback, which can ma- the manuscript. nipulate the incoming audio buffer or generate new sounds, and fill a new buffer with the resulting samples. AuSyn- 5. REFERENCES thAR calculates basic sound waves and operations, fills the [1] BEHRINGER. DeepMind12 Augmented Reality. Retrived new buffer with the samples and played them back. from https://www.youtube.com/watch?v=w76lNhqiaCw, An example of an interconnected sound synthesis network 2017. is displayed in (Figure 4), with 3D characters as a feedback [2] G. Geiger, N. Alber, S. Jord`a,and M. Alonso. The for sound interactions. Each block represents a simple mod- Reactable : A Collaborative Musical Instrument for Playing and Understanding Music. Rolling Stone, pages 36–43, 2010. ule that contributes to the final output. [3] F. Liarokapis. Augmented Reality Scenarios for Guitar Learning. Proceedings of Eurographics UK Chapter, Theory 2.4 Operational Process and Practice of Computer Graphics 2005, pages 163–170, Once the setup is assembled, the performer will be able to 2005. generate sounds and music by arranging the tokens (mark- [4] P. Lucas, D. Vaca, F. Dominguez, and X. Ochoa. Virtual ers) physically and connecting them virtually. The connec- circuits: An augmented reality circuit simulator for tions can be established by approaching the tokens for an engineering students. Proceedings - IEEE 18th International Conference on Advanced Learning Technologies, ICALT automatic connection, and then, separating them while a 2018, pages 380–384, 2018. virtual wire appears between both. The circle-shaped to- [5] R. Sethi. OAK Turns Your Tabletop Into An Augmented kens that act as “knobs”, can be connected similarly; that Reality Musical Instrument. Retreived from is, when one token is connected to another, it can control a https://ask.audio/articles/oak-turns-your-tabletop-into-an- parameter, such as the amplitude or frequency, by rotating augmented-reality-musical-instrument, the virtual knob. 2016.

406