DOCSLIB.ORG

Copyright by Yago Parra Moutinho Stucky De Quay 2016

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Copyright by Yago Parra Moutinho Stucky De Quay 2016 Copyright by Yago Parra Moutinho Stucky de Quay 2016 The Dissertation Committee for Yago Parra Moutinho Stucky de Quay Certifies that this is the approved version of the following dissertation: The Development and Experience of Gesture and Brainwave Interaction in Audiovisual Performances Committee: Sharon Strover, Supervisor Bruce Pennycook, Co-Supervisor Shanti Kumar Russell Pinkston Yacov Sharir Rui Penha António Coelho The Development and Experience of Gesture and Brainwave Interaction in Audiovisual Performances by Yago Parra Moutinho Stucky de Quay, B.Music.; M.Multimedia Dissertation Presented to the Faculty of the Graduate School of The University of Texas at Austin in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy The University of Texas at Austin December 2016 Dedication To my parents, Isabel and José Acknowledgements I want to thank Bruce Pennycook, Yacov Sharir, and Sharon Strover for their significant intellectual and artistic contributions to this work. I also want to thank A. R. Rahman for inviting me to join his fantastic project with Intel, and Lakshman Krishnamurthy for his inspiring leadership in that project. Finally, I am grateful to have had Reema Bounajem supporting me emotionally throughout the writing process. v The Development and Experience of Gesture and Brainwave Interaction in Audiovisual Performances Yago Parra Moutinho Stucky de Quay, PhD The University of Texas at Austin, 2016 Supervisor: Sharon Strover Co-Supervisor: Bruce Pennycook Software and hardware developments in the 21st century have greatly expanded the realm of possibilities for interactive media. These developments have fueled an increased interest in using unconventional methods to translate a performer’s intentions to music and visuals. And at the heart of every computer mediated performance is the question of what human action causes what audiovisual effect. This research explores this question and others by examining two performances that I created; performances featuring audiovisual media that respond to a performer’s gestures and brainwaves. The aim is to improve our understanding of what factors influence interaction design within and across the performing arts. I argue that technology, collaborators, and transparency have a decisive impact on a performer’s actions and interactive media. Furthermore, my findings suggest that synchronicity, elegance and visual feedback are important characteristics that can enhance the bond between the performer and audiovisual content. Lastly, I explain how universities and companies can improve research and development by together creating a live show. vi Table of Contents CHAPTER 1: INTRODUCTION .............................................................................................. 1 Background .................................................................................................................. 1 Methods ........................................................................................................................ 2 Data collection .............................................................................................................. 6 Dissertation outline ...................................................................................................... 7 CHAPTER 2: LITERATURE REVIEW ................................................................................... 10 Embodied music cognition ......................................................................................... 11 Audiovisual binding ................................................................................................... 14 Mediatization on stage ............................................................................................... 18 Interaction transparency ............................................................................................. 20 Interim summary ........................................................................................................ 22 Brain-computer interfaces .......................................................................................... 23 Extracting relevant neural signals .............................................................................. 28 Interim summary ........................................................................................................ 31 Examples of interactive performances ....................................................................... 32 Brainwaves .............................................................................................................. 33 Motion tracking ....................................................................................................... 37 Biosignals ................................................................................................................ 39 CHAPTER 3: CURIE .......................................................................................................... 41 Context ....................................................................................................................... 44 Origin ...................................................................................................................... 44 Media technology .................................................................................................... 47 Sensors .................................................................................................................... 52 Development .............................................................................................................. 56 Musician wristbands ............................................................................................... 57 Gesture algorithm .................................................................................................... 60 Optical motion tracking .......................................................................................... 65 Radio motion tracking ............................................................................................. 67 Finger tracking glove .............................................................................................. 71 Audience wristbands ............................................................................................... 72 Software .................................................................................................................. 76 Mappings................................................................................................................. 84 Mapping development ............................................................................................ 89 Visuals..................................................................................................................... 98 Choreography ........................................................................................................ 101 Music..................................................................................................................... 104 Analysis .................................................................................................................... 106 Technology ........................................................................................................... 108 Transparency ......................................................................................................... 114 Collaboration ......................................................................................................... 122 CHAPTER 4: AD MORTUOS .............................................................................................. 133 Context ..................................................................................................................... 136 vii Origin .................................................................................................................... 136 Media technology .................................................................................................. 138 Development ............................................................................................................ 139 Eeg headset ........................................................................................................... 140 Tablet computer .................................................................................................... 143 Led lights .............................................................................................................. 145 Motion tracking ..................................................................................................... 146 Position tracking ................................................................................................... 147 Practice software ................................................................................................... 148 Mappings............................................................................................................... 150 Mapping development .......................................................................................... 150 Visuals................................................................................................................... 153 Choreography ........................................................................................................ 155 Music..................................................................................................................... 157
Load more

Recommended publications
  • The Microphone Feedback Analogy for Chatter in Machining
    Hindawi Publishing Corporation Shock and Vibration Volume 2015, Article ID 976819, 5 pages http://dx.doi.org/10.1155/2015/976819 Research Article The Microphone Feedback Analogy for Chatter in Machining Tony Schmitz UniversityofNorthCarolinaatCharlotte,Charlotte,NC28223,USA Correspondence should be addressed to Tony Schmitz; [email protected] Received 16 March 2015; Accepted 13 May 2015 Academic Editor: Georges Kouroussis Copyright © 2015 Tony Schmitz. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. This paper provides experimental evidence for the analogy between the time-delay feedback in public address systems and chatter in machining. Machining stability theory derived using the Nyquist criterion is applied to predict the squeal frequency in a microphone/speaker setup. Comparisons between predictions and measurements are presented. 1. Introduction with the workpiece surface causes the tool to begin vibrating and the oscillations in the normal direction to be copied Self-excited vibration, or regenerative chatter, in machining onto the workpiece [4]. When the workpiece begins its has been studied for many years. The research topic has second revolution, the vibrating tool encounters the wavy encouraged the application of mathematics, dynamics (linear surfaceproducedduringthefirstrevolution.Subsequently, and nonlinear), controls, heat transfer, tribology, and other
    [Show full text]
  • Recording and Amplifying of the Accordion in Practice of Other Accordion Players, and Two Recordings: D
    CA1004 Degree Project, Master, Classical Music, 30 credits 2019 Degree of Master in Music Department of Classical music Supervisor: Erik Lanninger Examiner: Jan-Olof Gullö Milan Řehák Recording and amplifying of the accordion What is the best way to capture the sound of the acoustic accordion? SOUNDING PART.zip - Sounding part of the thesis: D. Scarlatti - Sonata D minor K 141, V. Trojan - The Collapsed Cathedral SOUND SAMPLES.zip – Sound samples Declaration I declare that this thesis has been solely the result of my own work. Milan Řehák 2 Abstract In this thesis I discuss, analyse and intend to answer the question: What is the best way to capture the sound of the acoustic accordion? It was my desire to explore this theme that led me to this research, and I believe that this question is important to many other accordionists as well. From the very beginning, I wanted the thesis to be not only an academic material but also that it can be used as an instruction manual, which could serve accordionists and others who are interested in this subject, to delve deeper into it, understand it and hopefully get answers to their questions about this subject. The thesis contains five main chapters: Amplifying of the accordion at live events, Processing of the accordion sound, Recording of the accordion in a studio - the specifics of recording of the accordion, Specific recording solutions and Examples of recording and amplifying of the accordion in practice of other accordion players, and two recordings: D. Scarlatti - Sonata D minor K 141, V. Trojan - The Collasped Cathedral.
    [Show full text]
  • Distortion Improvement in the Current Coil of Loudspeakers Gaël Pillonnet, Eric Sturtzer, Timothé Rossignol, Pascal Tournier, Guy Lemarquand
    Distortion improvement in the current coil of loudspeakers Gaël Pillonnet, Eric Sturtzer, Timothé Rossignol, Pascal Tournier, Guy Lemarquand To cite this version: Gaël Pillonnet, Eric Sturtzer, Timothé Rossignol, Pascal Tournier, Guy Lemarquand. Distortion improvement in the current coil of loudspeakers. Audio Engineering Society Convention, May 2013, Roma, Italy. hal-01103598 HAL Id: hal-01103598 https://hal.archives-ouvertes.fr/hal-01103598 Submitted on 15 Jan 2015 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. AES Audio Engineering Society Convention Paper Presented at the 134th Convention 2013 May 4–7 Rome, Italy This Convention paper was selected based on a submitted abstract and 750-word precis that have been peer reviewed by at least two qualified anonymous reviewers. The complete manuscript was not peer reviewed. This convention paper has been reproduced from the author's advance manuscript without editing, corrections, or consideration by the Review Board. The AES takes no responsibility for the contents. Additional papers may be obtained by sending request and remittance to Audio Engineering Society, 60 East 42nd Street, New York, New York 10165-2520, USA; also see www.aes.org. All rights reserved.
    [Show full text]
  • Electronic Pipe Organ Using Audio Feedback
    Electronic Pipe Organ using Audio Feedback Seunghun Kim and Woon Seung Yeo Audio & Interactive Media Lab Graduate School of Culture Technology, KAIST Daejeon, 305-701, Korea [email protected], [email protected] ABSTRACT 2. RELATED WORK The organ is a musical instrument with a long history. Based This paper presents a new electronic pipe organ based on on its sound-producing mechanism, organs can be catego- positive audio feedback. Unlike typical resonance of a tube rized into several groups such as pipe, reed, electronic, and of air, we use audio feedback introduced by an amplifier, a digital organs. Virtually none of these, however, utilizes lowpass filter, as well as a loudspeaker and a microphone the idea of acoustic feedback we propose. in a closed pipe to generate resonant sounds without any Examples of acoustic feedback used for musical instru- physical air blows. Timbre of this sound can be manip- ments include the hybrid Virtual/Physical Feedback Instru- ulated by controlling the parameters of the filter and the ments (VPFI) [2], in which a physical instrument (e.g., a amplifier. We introduce the design concept of this audio pipe) and virtual components (e.g., audio DSP processors feedback-based wind instrument, and present a prototype such as a lowpass filter) constitute a feedback cycle. The that can be played by a MIDI keyboard. Audible EcoSystemics n.2 is a live electronic music based only on Larsen tones (i.e., sound generated by audio feed- back) [3]. In [4], Overholt et al. documents the role of feedback in 1. INTRODUCTION actuated musical instruments.
    [Show full text]
  • LM4651 & LM4652 Overture™ Audio Power Amplifier 170W Class D
    LM4651 & LM4652 Overture August 2000 LM4651 & LM4652 Overture™ Audio Power Amplifier 170W Class D Audio Power Amplifier Solution General Description Key Specifications The IC combination of the LM4651 driver and the LM4652 n Output power into 4Ω with < 10% THD. 170W (Typ) power MOSFET provides a high efficiency, Class D sub- n THD at 10W, 4Ω, 10 − 500Hz. < 0.3% THD (Typ) woofer amplifier solution. n Maximum efficiency at 125W 85% (Typ) The LM4651 is a fully integrated conventional pulse width n Standby attenuation. >100dB (Min) modulator driver IC. The IC contains short circuit, under voltage, over modulation, and thermal shut down protection Features circuitry. It contains a standby function, which shuts down n Conventional pulse width modulation. ™ the pulse width modulation and minimizes supply current. The LM4652 is a fully integrated H-bridge power MOSFET n Externally controllable switching frequency. IC in a TO-220 power package. Together, these two IC’s n 50kHZ to 200kHz switching frequency range. 170W Class D Audio Power Amplifier Solution form a simple, compact high power audio amplifier solution n Integrated error amp and feedback amp. complete with protection normally seen only in Class AB n Turn−on soft start and under voltage lockout. amplifiers. Few external components and minimal traces n Over modulation protection (soft clipping). between the IC’s keep the PCB area small and aids in EMI n Short circuit current limiting and thermal shutdown control. protection. The near rail-to-rail switching amplifier substantially in- n 15 Lead TO−220 isolated package. creases the efficiency compared to Class AB amplifiers.
    [Show full text]
  • Sonic Screwdriver’, Meant for Implementation in Iterations of Daniël Maalman’S Lost & Found Orchestra Sound Art Installations
    Abstract This thesis showcases the development process of a tool for achieving a sustained, non-percussive sound that captures the sonic essence of ceramic objects, a so-called ‘sonic screwdriver’, meant for implementation in iterations of Daniël Maalman’s Lost & Found Orchestra sound art installations. The tool is meant to use a different way of sound excitation than Maalman’s conventional method of simply tapping objects with cores of solenoids. This is done by means of creating an audio feedback loop on the surface of the objects, which allows the objects speak in their own voice by using its resonant frequencies in an audio feedback loop. The audio feedback loop is composed of a contact microphone, a surface transducer and an audio amplifier. The system achieves accurate pitch control of the audio feedback at the resonant frequencies of an object by means of a control signal being input into the audio feedback loop via a second surface transducer. The developed solution can be used as a powerful tool in the creation of many types of sound art. 1 Quinten Tenger | Creative Technology | University of Twente | 2019 Acknowledgements First of all, I would like to thank Daniël Maalman for giving me the opportunity to work on this project, inspiring me with his way of working and being available to answer questions day and night. It has been a truly interesting and rewarding challenge. Furthermore, I would like to thank Edwin Dertien for helping me stay on track in times of confusion and frustration and his continuous support and guidance, as well as Erik Faber’s.
    [Show full text]
  • Aalborg Universitet Towards an Open Digital Audio Workstation for Live
    Aalborg Universitet Towards an open digital audio workstation for live performance Dimitrov, Smilen DOI (link to publication from Publisher): 10.5278/vbn.phd.engsci.00028 Publication date: 2015 Document Version Publisher's PDF, also known as Version of record Link to publication from Aalborg University Citation for published version (APA): Dimitrov, S. (2015). Towards an open digital audio workstation for live performance: the development of an open soundcard. Aalborg Universitetsforlag. (Ph.d.-serien for Det Teknisk-Naturvidenskabelige Fakultet, Aalborg Universitet). DOI: 10.5278/vbn.phd.engsci.00028 General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. ? Users may download and print one copy of any publication from the public portal for the purpose of private study or research. ? You may not further distribute the material or use it for any profit-making activity or commercial gain ? You may freely distribute the URL identifying the publication in the public portal ? Take down policy If you believe that this document breaches copyright please contact us at [email protected] providing details, and we will remove access to the work immediately and investigate your claim. Downloaded from vbn.aau.dk on: April 30, 2017 THE DEVELOPMENT OF AN OPEN SOUNDCARD THE DEVELOPMENT OF FOR LIVE PERFORMANCE: AUDIO WORKSTATION AN OPEN DIGITAL TOWARDS TOWARDS AN OPEN DIGITAL AUDIO WORKSTATION FOR LIVE PERFORMANCE: THE DEVELOPMENT OF AN OPEN SOUNDCARD BY SMILEN DIMITROV DISSERTATION SUBMITTED 2015 SMILEN DIMITROV Towards an open digital audio workstation for live performance: the development of an open soundcard Ph.D.
    [Show full text]
  • The Sound of Feedback, the Idea of Feedback in Sound Seminar 25 January 2020
    The Sound of Feedback, The Idea of Feedback in Sound Seminar 25 January 2020 Canterbury Christ Church University Daphne Oram Building North Holmes Road Canterbury CT1 1NP (UK) Co-organisers: WinterSound Festival and the Composition, Improvisation and Sonic Art (CISA) Research Unit, Canterbury Christ Church University (Canterbury, Kent, UK) Music, Thought and Technology, Orpheus Institute (Ghent, Belgium) WinterSound: Celebrating the sonic underground We celebrate the sound of tomorrow with the fourth annual WinterSound festival. Fresh for 2020, we move to a new home, the Daphne Oram Building, named after one of the UK’s most experimental, risk-taking musicians. Daphne Oram was a pioneering composer who taught in Canterbury in the 1980s, and listened beyond the boundaries of style and convention to dream of sounds no one had heard before. We continue her legacy in seeking out sounds beyond the mainstream, with dynamic collisions of electronic music, jazz, sound installations, improvisation, circuit bending and audiovisual surprises. Thursday 23 January, 5PM-9PM, Daphne Oram Building “Experimental Songs, Broken Rhythms and Glitches” 5.00 - 6.30 Collaborations with C3U records, jazz and rap experiments Exhibition Space and CONTACT, Canterbury’s hottest electronic band. 6.30 - 7.00 Extra Normal Records: DJ Set Exhibition Space 7.00 - 8.00 Matthew Herbert presents Accidental Records DO 0.14 8.00 - 9.00 Mariam Rezaei - DJ Set Exhibition Space Friday 24 January, 5PM-9PM, Daphne Oram Building “Communities of Sound” Presenting the Splinter Cell string quartet, Free Range Orchestra and the debut of a brand new Canterbury band, TOTEM//TREES, featuring 80s legend Jack Hues, flautist Heledd Francis- Wright and US saxophonist Robert Stillman.
    [Show full text]
  • Nonlinear Echo Suppression Technology Enabling Quality Handsfree Talk for Compact Equipment
    Image/Voice Processing Component Solutions Nonlinear Echo Suppression Technology Enabling Quality Handsfree Talk for Compact Equipment HOSHUYAMA Osamu Abstract This paper introduces a nonlinear echo suppression technology that enables compact equipment such as a notebook PC or cellular phone to be employed for handsfree use without a headset. In the suppression of echo, (audio feedback from the loudspeaker to the microphone) the distorted echo produced by vibrations of the compact loudspeaker or cabinet has been particularly diffi cult in the past. However, the newly developed frequency domain processing technology makes it possible to cancel the echo at a level that permits high-quality talk. Keywords cellular phone, handsfree call, echo canceller, distortion, nonlinear 1. Introduction 2. Principles of Nonlinear Echo Generation Recently, the dissemination of remote conferences in enter- Let us assume that you are in Tokyo as shown in Fig. 1(a) prises, VoIP calls on PC, video phone on cellular phones and and that you are holding a handsfree talk with Osaka, and see the prohibition of the use of cellular phone during driving have how the voice is heard. First, voice “Hello” spoken at Osaka is increased the demand for a handsfree capability without using sent through the communication line and reproduced as “Hel- a headset or handset. The handsfree devices for this purpose lo” to the loudspeaker in Tokyo. Since the loudspeaker repro- currently use a linear echo canceller based on a linear adaptive duced volume is high in handsfree talk, “Hello” output from fi lter in order to improve the communication quality. However, the loudspeaker is fed back to the microphone and returned to this technology still poses a problem, which is the incapability Osaka with a delay.
    [Show full text]
  • (Diy) Punk Rock
    LISTENING IN THE LIVING ROOM: THE PURSUIT OF AUTHENTIC SPACES AND SOUNDS IN DALLAS/FORT WORTH (DFW) DO -IT-YOURSELF (DIY) PUNK ROCK Sean P eters Thesis Prepared for the Degree of MASTER OF ARTS UNIVERSITY OF NORTH TEXAS December 2017 APPROVED: Catherine Ragland, Major Professor Steven Friedson, Committee Member Rebecca Geoffroy-Schwinden, Committee Member Benjamin Brand, Interim Chair of the Division of Music History, Theory, and Ethnomusicology and Director of Graduate Studies in the College of Music John W. Richmond, Dean of the College of Music Victor Prybutok, Dean of the Toulouse Graduate School Peters, Sean. Listening in the Living Room: The Pursuit of Authentic Spaces and Sounds in Dallas/Fort Worth (DFW) Do-It-Yourself (DIY) Punk Rock. Master of Arts (Music), December 2017, 97 pp., 4 figures, bibliography, 98 titles. In the Dallas/Fort Worth (DFW) do-it-yourself (DIY) punk scene, participants attempt to adhere to notions of authenticity that dictate whether a band, record label, performance venue, or individual are in compliance with punk philosophy. These guiding principles champion individual expression, contributions to one's community (scene), independence from the mainstream music industry and consumerism, and the celebration of amateurism and the idea that everyone should "do it yourself." While each city or scene has its own punk culture, participants draw on their perceptions of the historic legacy of punk and on experiences with contemporaries from around the world. For this thesis, I emphasize the significance of performance spaces and the sonic aesthetic of the music in enacting and reinforcing notions of punk authenticity. The live performance of music is perceived as the most authentic setting for punk music, and bands go to great lengths to recreate this soundscape in the recording studio.
    [Show full text]
  • The Laser Microphone Authors: Mark Chounlakone, Julian Alverio Filtering Section: Justin Tunis
    The Laser Microphone Authors: Mark Chounlakone, Julian Alverio Filtering Section: Justin Tunis Abstract The human voice can generate sound waves in the range of 300 Hz to 3400 Hz [8]. These sound waves vibrate nearby objects, making it possible for an analog electronic device to convert these vibrations into an audio signal[1]. One way to accomplish this conversion from movement to audio is to use a "laser microphone", which reflects a laser off the vibrating object and uses a receiver to capture the laser's reflection. The reflection of the laser gets deflected as vibrations shift the surface of the vibrating object. Therefore, if a receiver takes in the oscillating laser signal from a fixed location, the receiver will detect the laser deflections caused by the vibrations that were originally produced from an audio signal. The receiver can then filter and amplify this signal, and output it as audio. Through this process the laser microphone effectively reproduces the audio that induced the object's vibrations. The laser microphone is able to reproduce audio detected from a vibrating surface with relatively high accuracy: less than 8% distortion. As an additional feature, the laser microphone is also able to transmit audio via amplitude-modulated laser signal, capture the laser signal, and output the audio. Thus by using a laser based system that captures oscillations in the position of the laser, the laser microphone is able to accurately reproduce both the audio that induced an object's vibrations and audio transmitted via laser communication. Introduction Human speech is composed of sound waves that vibrate the objects nearby.
    [Show full text]
  • Mood - a Personal Interpretation of the Legendary Analog Synthesizer by Eugenio Giordani, Ios App by Alessandro Petrolati All Rights Reserved 2017 Apesoft
    mood - A personal interpretation of the legendary Analog Synthesizer by Eugenio Giordani, iOS app by Alessandro Petrolati All rights reserved 2017 apeSoft Legal Notice All product names used in this product are trademarks of their respective owners and are in no way associated or affiliated with apeSofte Software. These trademarks of other manufacturers are used solely to identify the products of those manufacturers whose tones and sounds were studied and or recorded during development of this product. Thank you for purchasing the mood iOS synthesizer. This work is a tribute and a personal reinterpretation of the classic analog synthesizer we have known in the seventies. This is not the full reproduction of a particular subtractive synthesis instrument but the re-creation of a new one that incorporates some of the most important features that are present in the originals, including popular sound components of the eighties as Frequency Modulation synthesis module. After the great effort in recreating the iVCS3, we wanted to design a new instrument more freely and creatively, having in mind primarily the joy of playing with the sound and sharing with you this our passion. Again thanks to the incredible power and flexibility of Csound, it was possible in a relatively short time to turn from an idea into a finished project in a relatively short time. During this process we have added many new features to the original design as a fully programmable sixteen steps arpeggiator/sequencer, polyphonic functionality, a “vintager” parameter and the full compatibility with Audiobus and Inter-App Audio protocols. I hope you enjoy this synthesizer for you music and sound satisfaction.
    [Show full text]