Behavior, Animation, Music

behavior, animation, music: the music and movement of synthetic characters Marc Downie BA, MSci, Magdalene College, University of Cambridge (1998) submitted to the program in media arts and sciences, school of architecture and planning, in partial fulfilment of the requirements for the degree of master of science in media arts and sciences at the massachusetts institute of technology, february 2001 author program in media arts and sciences Bruce M. Blumberg assistant professor of media arts and sciences, Asahi broadcasting corporation career development professor of media arts and sciences, thesis supervisor Stephen A. Benton chair, department committee on graduate students, program in media arts and sciences massachusetts institute of technology 2000. all rights reserved. behavior, animation, music: the music and movement of synthetic characters Marc Downie submitted to the program in media arts and sciences, school of architecture and planning, in partial fulfilment of the requirements for the degree of master of science in media arts and sciences at the massachusetts institute of technology, february 2001 abstract This thesis begins with the idea that reactive, behavior-based ‘synthetic characters’ can become an appropriate platform for musical experimentation. This idea motivates the creation of a new behavior system for these characters. This system, in addition to providing the basis of the work described therein, appears to solve some outstanding problems in character creation. Next, work on the creation of characters’ motor systems is described, culminating in a new framework for characters to learn and understand their motor actions while remaining within an example-based animation domain. Finally, several musical applications of these systems and this character-based approach are discussed. These applications take the form of visual and audio interactive installations that consist of, in part, musical creatures built from this framework. thesis supervisor: Bruce M. Blumberg. assistant professor of media arts and sciences, Asahi broadcasting corporation career development professor of media arts and sciences behavior, animation, music: the music and movement of synthetic characters Marc Downie submitted to the program in media arts and sciences, school of architecture and planning, in partial fulfilment of the requirements for the degree of master of science in media arts and sciences at the massachusetts institute of technology, february 2001 Tod Machover Professor of Music and Media, Program in Media Arts and Sciences thesis reader Emilio Bizzi Eugene McDermott Professor in the Brain Sciences and Human Behavior thesis reader acknowledgements I would like to thank the following cast: Bruce Blumberg – direction; Bill Tomlinson – production; Michael Patrick Johnson – technical; Ari Benbasat – pastoral; Tod Machover – for my time in MediaLabEurope; the remainder of the synthetic characters group at The Media Lab with whom I’ve worked closely (they are: Rob Burke, Scott Eaton, Jesse Gray, Michal Hlavac, Damian Isla, Yuri Ivanov, Matt Berlin, Chris Kline, Ben Resner, Ken Russell, Song-Yee Yoon) – environmental; Linda Peterson – patience; my parents – understanding; Alison James – love. contents 17 list of figures 19 introduction 20 why a synthetic character? the synthetic characters group why characters? 21 two plateaus expressive characters intelligent interactive music 25 behavior 26 a new behavior system I context goals – learning goals – the year of the dog project shaping - learning by successive approximation 'shaping' and music 31 phase one — building blocks I the action-tuple computing expected values 33 percepts percept trees 39 phase two — dynamics state objects and decaying memory traces 41 the action-group a baseline action selection mechanism 44 credit assignment value updating 46 percept updating – model building extending the percept tree 50 phase three – models I introduction I Tr percepts simple generic one-dimensional models combinational models activation models other models 57 temporal models simple durations forward transition models exotic duration models temporal logics 65 aural models cepstral models 66 phase four – extensions I other values what is value? complex value implementations value chaining - the precedes relationship value signal processing 74 other action-groups hierarchical action-groups perception as action 78 other state – emotion and motivation soft state expectations about value expectations about the consequences of actions emotional tags 84 concluding remarks 87 animation 88 motor systems for synthetic characters I challenges for a character motor system aesthetic choices walking, shaking and moving your head ...in the correct style... supporting parameterized behavior support new motor actions motor level problem solving maintaining the illusion of (cartoon) physics finding your food 95 example based character animation – an overview sources of animation data a diversion - why example-based? 102 blend based motor systems I building up a motor system the animation player adding animation blending 110 using adverb parameters to solve problems a baseline motor system - so far so good... in joint representation blending & processing simple shaping replacing inverse-kinematics learning to replace inverse-kinematics where does this leave us? 123 graph-based motor systems I overview I technical development building up the node representation developing the distance metrics automatic topology generation re-editing animation data 134 towards a complete motor system moving around the graph complex labels motor program graphs learning to chase your tail – part 1 gesture graph generation learning to chase your tail – part 2 communication with the behavior system enforcing physical constraints 148 future extensions self-intersection removal persistent ‘mergeable’ graphs node models 154 concluding remarks 157 music 158 why do music this way? I (void *): a cast of characters interactive music score output methods 164 after (void *) — lessons learned the case against direct control the case against low risk music the case against radically different music creatures 168 sketches I common themes 170 sand:stone (installation) introduction music creature design status 173 three creatures (performance) introduction 175 soundcreatures (installation) introduction creature design aural Ac percept trees musical structures from behavior status 181 ozymandias (video) introduction off-line image processing – image creatures 185 exchange (installation) creature design - behavior musical worlds 190 listen (installation) introduction creature design extensions status 196 curve (technology) introduction scanned synthesis models (technology) artificial gestural control 203 beginnings and endings 204 technical influences behavior-based AI traditional machine-learning deliberative AI interactive music origins of music previous work inside the group 209 where this leaves us 215 (quaternions) preliminaries interpolation distance metric 219 (video contents) 221 references |16 list of figures 31 figure 1. introducing the action-tuple 34 figure 2. initial percept configuration might include “sound” and “movement” 36 figure 3. action trees can group similar classes of actions together to guide experimentation 37 figure 4. percept trees guide new action-tuple hypothesis generation 38 figure 5. building up a dog – part 1 40 figure 6. graph of saliency over time for a simple click 49 figure 7. building up a dog – part 2 51 figure 8. a Gaussian model forming a model of the world 55 figure 9. clique formation 60 figure 10. forward transition modelling 63 figure 11. low integer ratio blind duration models 64 figure 12. temporal relationships 68 figure 13. the lion problem 75 figure 14. schemes for hierarchical action-selection 82 figure 15. timescales in the behavior system 85 figure 16. duncan, and the shepherd 94 figure 17. finding your food (if you are a dog) 96 figure 18. key frame animation applied to a dog skeleton 103 figure 19. a simple ‘verb’ graph of animations 105 figure 20. radial basis functions for interpolation 108 figure 21. timewarping necessary for blending 111 figure 22. flow of command in a blend based motor system 115 figure 23. increasingly high “beg” models (skeleton only) for dog 116 figure 24. arrangement of basis functions in 3 space 117 figure 25. a five-spot basis function pattern for head interpolation 118 figure 26. SOM learning to look at objects 124 figure 27. the wrong and right segmentation of a biphasic ‘wave’ gesture 129 figure 28. building up the distance metric 130 figure 29. calculating closest transit time 131 figure 30. a highly connected motor graph generated from 3 animations – sniff, play and beg 133 figure 31. graph operation for reincorporating animation data 143 figure 32. learning and segmenting a gesture 145 figure 33. blended joint angles are not equal to blended end-effector positions 150 figure 34. adding blending back into graphs |18 159 figure 35. images from (void*) 160 figure 36. the (void*) music system 171 figure 37. synthetic chemistry for motivational variables 172 figure 38. images from sand:stone 176 figure 39. new sounds are incorporated as triggers and actions 180 figure 40. images of a sound creature (inverted) 186 figure 41. note-space metaphors are musically useful 189 figure 42. a music creature chasing down two notes (inverted) 191 figure 43. keeping a model population in a percept 195 figure 44. listen dancing frenetically 195 figure 44. two stills

Load more