Embodied Sonic Meditation and Its Proof-Of-Concept: “Resonance of the Heart”
Total Page：16
File Type：pdf, Size：1020Kb
Embodied Sonic Meditation and its Proof-of-Concept: “Resonance of the Heart” Jiayue Cecilia Wu Julius O. Smith Yijun Zhou Matthew James Wright UC Santa Barbara Stanford University Stanford University Stanford University [email protected] [email protected] [email protected] [email protected] ABSTRACT 1.2 George Lakoff’s “Embodied Cognition” This paper presents the concept of Embodied Sonic Medi- tation (ESM). ESM sonically explores the theory of “Em- On the other hand, George Lakoff et al.’s “Embodied bodied Cognition,” which argues that we reflect on daily Cognition” theory argues that high-level concepts such as events and understand abstract concepts, such as the time, space, arts, and mathematics are grounded in sen- aesthetics of music and art, through our physical body. sorimotor experience – a view that our sensorimotor ca- This concept is being introduced to undergraduate stu- pacities, bodies and environment are all central to shap- dents in a U.S. university as part of an experimental mu- ing our mental processes [8]. In other words, we are not sic pedagogical methodology. The goal is to improve the disembodied mind floating around; we are also made of students’ comprehension of the relationship of gestures flesh and bones – the fact that we have bodies strongly and sounds. To practice this approach, we designed and shapes our mind and cognition. We experience and learn realized a proof of concept audio-visual system named about the world through motor-based exploration accord- Resonance of the Heart (Chinese:印心). This system uses ing to limitations of our sensorimotor and perceptual sys- an infrared sensing device and touchless hand gestures to tem. Therefore, if we wanted to give someone something control a real-time tracking system producing various to think about (cognitive process), we should give sonic results. To track and estimate the subtle gestures of her/him something related to her/his bodily activities in- ten fingers that are not typically captured by any existing stead of something abstract. This is what we mean by sensing device, we implemented supervised learning al- “embodiment.” gorithms and an artificial neural network. Two novel electroacoustic vocal processing techniques, which in- 1.3 Gesture-Controlled Real-time Vocal Processing clude a Tibetan Throat Singing filter and a spectral-tilt Human-computer interactive design, input, mapping, and filter, were first implemented to simultaneously process control strategies have been developed to enhance human vocals based on the performer’s hand gestures with a vocal expression in real-time, such as [1], [6], and [11]. one-to-one mapping strategy. Although the uses of body movement in composition and instrumental design have been studied since the 1980’s 1. INTRODUCTION [2], few studies have been done in the context of electro- acoustic vocal performance. 1.1 Pauline Oliveros and “Deep Listening” Vocal performance is unique from other instrumental performances in several ways. First, to a vocalist, the In 1974, Pauline Oliveros disrupted traditional western body is the instrument: sound comes directly from a vo- music education by practicing an ancient eastern philo- calist’s body; there is no other sound generator. Moreo- sophical concept – meditating though sound. In Bud- ver, humans can naturally read body language and voices; dhism this is called “experiencing sonic Vedanā” [4]. She this begins at infancy and is refined to an art by adult- adopted this approach and further developed it into an hood [3]. Thus, a vocalist’s body gestures produce more improvising, composing, and teaching practice. Through complex perceived cognitive meanings in terms of com- sonic meditations, Oliveros advocated a “Deep Listen- municating emotions and expressing musicality to the ing” practice [12] that trains our ears and mind to con- audience compared to other instrument players [14]. sciously appreciate all sounds to increase sonic aware- In a previous study [21], we proposed the first empir- ness, thus creating profound effects on music making and ical evaluation methodology of a Digital Music Instru- listening. Actual sound making in Oliveros’ sonic medita- ment (DMI) for augmenting electroacoustic vocal per- tions was “primarily vocal, with sometimes hand clap- formance from the audience’s perspective. We found the ping or other body sounds. Occasionally, sound- relationship between the performer’s body movement and producing objects and instruments are used” [13]. Oli- vocal expression are crucial to the degree of perceived veros’ work in sonic meditation focused on the cognition engagement from the audience perspective. However, of sound. more investigation is needed to further understand this Copyright: © 2017 Jiayue Cecilia Wu et al. This is an open-access article profound vocal-gesture relationship in computer music. distributed under the terms of the Creative Commons Attribution License 3.0 Unported, which permits unrestricted use, distribution, and reproduc- tion in any medium, provided the original author and source are credited. 110 2017 ICMC/EMW 2. GOALS AND MOTIVATIONS an engagement with sonic awareness, self-exploration, and non-hierarchical social relationships of music crea- Inspired by Pauline Oliveros’ four decades of “Deep Lis- tion and appreciation. Students will be practicing ESM tening” practice and George Lakoff’s “embodied cogni- through listening, singing, music performance and im- tion,” we propose Embodied Sonic Meditation: a sonic provisation, field recording, and interactive music con- art practice based on the combination of sensing technol- trolled by motion capture. With the aid of music technol- ogy and human sensibility. Through this practice, we ogy, we aim to open up a safe, free, and non-judgmental encourage people to fully understand and appreciate ab- space to touch, move, and inspire students to express stract electric and electroacoustic sounds and how these their creative nature, embrace their inner selves, and gen- sounds are formed and transformed (cognitive process), uinely connect with others by enhancing their sonic by providing them interactive audio systems that can awareness and their ability to listen, understand, and tightly engage their bodily activities to simultaneously communicate through novel music expressions with em- create, sculpt, and morph the sonic outcomes themselves, bodied experience. using their body motions (embodiment). This ongoing project aims to further explore gesture-controlled, vocal- 3. SYSTEM ARCHITECTURE processing DMI design strategies and experimental sound education. In this preliminary research phase, we focus on elec- 3.1 Tracking Device and Its Optimization Solutions troacoustic vocal processing manipulations, particularly Because of its low price, lightweight, and portability, as the mapping strategies relating motion input/control and well as a lower latency and higher frame rate compared to the parameters of vocal processing, because of their sig- other sensors [16] such as Microsoft’s Kinect™, we nificance and lack of systematic studies in the field. chose a Leap Motion™ infrared sensor as our non- For teaching and proof-of-concept purposes, we de- attached tracking sensor to realize the gestural instrument veloped an interactive audio-visual system named “Reso- for this project. nance of the Heart.” The name of the system is borrowed The sensing algorithms and visual processing are from “印心”, a “Kōan” story in Chinese Zen Buddhism, implemented in Python. We implemented supervised which describes a Zen master and his disciple’s thoughts learning algorithms and an artificial neural network to resonant without verbal communication. This system estimate and track the subtle motions of ten fingers, enables students to learn and explore the sound-gesture which are not typically captured by existing sensing de- relationship using hand movement through sensing tech- vices. Two historically intractable problems of the Leap nology. We applied ancient Buddhist hand gestures Motion™ sensor are addressed: 1) the sensor cannot de- named Mudras [7] that have the hands and fingers tect overlapping hands, and 2) the sensor’s detection crossed or overlapped, as shown in Figure 1, to trigger range is spatially limited. Training examples included corresponding sonic effects. Meanwhile, dynamic hand seven Tibetan Buddhist Mudras that are overlapping hand motions are also mapped to the audio/visual system to gestures as well as fifty hand trajectories where a hand continuously control electroacoustic voice manipulations and a visualization of a 4-dimensional Buddhabrot fractal goes out of the sensor's range and later returns. Hand ges- [5]. The visual processing system is independent from the tures were treated as a classification problem with 62% audio processing system. The visual component is only accuracy. used in live performance. We use only the audio system With our system optimization solutions, the hand tra- in our teaching practice in order to let the students to fo- jectory is predicted when the tracking device loses its cus on sonic awareness. track. Instead of losing track and interrupting the smoothness of the gestural input and control, at least there are some data to be processed, thus improving the coherence and wholeness of the user experience. Our preliminary research shows the general potential of ap- plying machine learning to creating robust DMIs using unreliable sensors. 3.2 Audio-Visual Processing The user gives the system two inputs: vocals via a micro-