MEDIA SCORES a Framework for Composing the Modern-Day Gesamtkunstwerk
Total Page:16
File Type:pdf, Size:1020Kb
MEDIA SCORES A Framework for Composing the Modern-Day Gesamtkunstwerk PPPP AAA A TTT T Bachelor of the Arts in Media Arts University of Arizona, $%%& Master of Sciences in Media Arts and Sciences Massachusetts Institute of Technology, $%%+ Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial ful/llment of the requirements for the degree of Doctor of Philosophy in Media Arts aandnd Sciences at the Massachusetts Institute of Technology September $%2& © $%2& Massachusetts Institute of Technology. All rights reserved. Author: P A T Program in Media Arts and Sciences + August $%2& Certi/ed by: T M 78 9 Muriel R. Cooper Professor of Music and Media Program in Media Arts and Sciences ;esis Supervisor Accepted by: P <7< M = Alexander W. Dreyfoos Professor of Media Technology Associate Academic Head, Program in Media Arts and Sciences MEDIA SCORES A Framework for Composing the Modern-Day Gesamtkunstwerk PPPP AAA A TTT T Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial ful/llment of the requirements for the degree of Doctor of Philosophy in Media Arts and Sciences at the Massachusetts Institute of Technology A B S T R A C T Media Scores provide a theoretical and technical means to orchestrate multiple modalities in the creation of expressive works of art and performance. New technologies aDord numerous opportunities to tell stories and create expressive artworks through a variety of media. However, the tools of planning, composition, design, and creation of these works remain disjoint with respect to the artwork’s constituent disciplines and from the /nal experience. Media Scores extend the metaphor of a musical score to other modalities in order to facilitate the process of authoring and performing multimedia compositions, providing a medium through which to realize a modern-day Gesamtkunstwerk. Research into the representation and encoding of expressive intent provides the conceptual underpinnings for the development of novel interfaces for composing and performing with Media Scores. Using such a tool, the composer is able to shape an artistic work that may be performed through human and technological means in a variety of media and utilizing various modalities of expression. Media Scores oDer the possibility of authoring content that incorporates live performance data and the potential for audience participation and interaction. ;is paradigm bridges the extremes of the continuum from composition to performance, allowing for improvisatory compositional acts at performance-time. ;e Media Score also provides a common point of reference in collaborative productions, as well as the infrastructure for the real-time show control of any technologies used during a live performance. ;esis Supervisor: T M 78 9 Muriel R. Cooper Professor of Music and Media Program in Media Arts and Sciences Massachusetts Institute of Technology MEDIA SCORES A Framework for Composing the Modern-Day Gesamtkunstwerk PPPP AAA A TTT T ;e following person served as a reader for this thesis: ;esis Reader K P< Professor of Computer Science Media Research Lab New York University MEDIA SCORES A Framework for Composing the Modern-Day Gesamtkunstwerk PPPP AAA A TTT T ;e following person served as a reader for this thesis: ;esis Reader A M7D L, < Associate Professor of Cinematic Arts Interactive Media Division University of Southern California ACKNOWLEDGMENTS T M 78 9 ;ank you for welcoming me along on this remarkable six-year portion of the journey. I am ever-grateful for your guidance and the opportunities you have aDorded to me to participate in groundbreaking projects, to leverage my skills, expand my horizons, to grow, and to create. A M7D L I am most grateful to my dissertation committee for your time, expertise, K P< and wisdom. Our numerous discussions over the years have been exceedingly helpful in re/ning and broadening my ideas about the future of storytelling and the arts. Much of that has found its way into Media Scores and this thesis and will surely continue to inspire my future work. N< ON ;ank you for having been a part of this process. Your unique and varied insights and suggestions have proven to be an important part of the ideas explored in this thesis. M O P N N< T Nothing can parallel the love, devotion, and unwavering support that you have always given me. By your example and because of your encouragement, I am empowered to reach for my dreams. E J== ;rough all of the projects, the furies of creation, the obstacles, and the celebrations of accomplishments that we have shared over the past several years, I am very much indebted to you, your friendship, and support. B B NST I have had the great privilege to work alongside such a ta lented and creative S<N O9= group. You have encouraged and inspired me with your and ideas and AO< 9 T creations. More importantly, I’ve been most fortunate to have you as my E S8 8 friends. J <7 W T RUS77 K< ST + E O9 Your constant friendship throughout the years has been a comfort and reassurance. C 8< R V< My dear friends beyond the Lab have provided essential companionship S8 C8 W and diversions over the years. N 8 F8 CWS< K< T, Y. A DWS< D < K 8 N M 78 9 Figments would not have been possible without the additional contributions E K L<7Z of several people. My talented rehearsal cast helped develop the piece and H Z the performances that would ultimately be staged. Others helped me to J = H = shape the story, the technologies, and construct the performance EN< SW infrastructure. It was rewarding to have such a team to bring my ideas D < M before an audience. PW 78 W O Some of the work presented in this thesis was conducted in collaboration with organizations worthy of special acknowledgment. From material T support to unique opportunities to explore innovative ideas, I am grateful SN8 O78= to have worked with each. MIT MW=<7 T8 A= MIT MW=<7 T8 GW< AEZ I =<W 2% Media Scores C O N T E N T S Acknowledgments ..................................................................................... + Contents ................................................................................................... 22 List of Figures ........................................................................................... 2& 2 Introduction ................................................................................... 2\ 2.2 ;e Art-Work of the Future ......................................................... 2] 2.$ Objectives for Media Scores ......................................................... 2^ 2.& Re_ections on a Journey .............................................................. 2+ 2.` Culmination of Idea and Practice ................................................. 2+ 2.\ Terminology ............................................................................... $% 2.] ;esis Structure ........................................................................... $& $ Foundations and Precedents ........................................................... $\ $.2 Perception, Representation, and Mapping .................................... $\ $.$ Expectation and Information ....................................................... &% $.& Emotion ...................................................................................... &] $.` Synaesthesia and Multisensory Integration ................................... &a $.\ Semiotics and Abstract Representation ........................................ `% $.] Abstract Art ................................................................................ `% $.^ Visual Music ................................................................................ `$ $.a Projections of Time ..................................................................... `] $.+ Scripts and Scores ........................................................................ `a $.2% Production Practice ...................................................................... \& $.22 Performativity, Liveness, and Immersion ...................................... \^ & Systems in Performance .................................................................. ]& &.2 Death and the Powers ................................................................. ]] &.2.2 Disembodied Performance ....................................................... ]+ &.2.$ Robot Choreography ................................................................. ^2 &.2.& Mapping System ....................................................................... ^& &.2.` Render Designer ...................................................................... ^] &.$ Spheres and Splinters .................................................................. ^+ &.& Extending Sleep No More ........................................................... a% 22 Contents &.` A Toronto Symphony .................................................................. a^ &.\ Practice ........................................................................................ +2 &.\.2 Cues, Modes, and Triggers ........................................................ +$ &.\.$ Communications Protocols .......................................................+` &.] Design Principles for Performance Systems ................................. +^ ` Creating Media Scores ................................................................... 2%2 `.2 ;e Beginnings of an Idea.......................................................... 2%$ `.2.2 Personal Opera .......................................................................