Vol. 1 No. SIG 1(CVIM 1) IPSJ Transactions on Computer Vision and Image Media June 2008 Regular Paper Synthesis of Dance Performance Based on Analyses of Human Motion and Music y y TAKAAKI SHIRATORI 1 and KATSUSHI IKEUCHI 2 Recent progress in robotics has a great potential, and we are considering to develop a dancing humanoid robot for entertainment robots. In this paper, we propose three fundamental methods of a dancing robot aimed at the sound feedback system in which a robot listens to music and automatically synthesizes dance motion based on the musical features. The first study analyzes the relationship between motion and musical rhythm and extracts important features for imitating human dance motion. The second study models modifi- cation of upper body motion based on the speed of played music so that a humanoid robot dances to a faster musical speed. The third study automatically synthesizes dance performance that reflects both rhythm and mood of input music. with the music. 1. Introduction 2. Overview of Proposed Methods Since technology regarding humanoid robots is advancing rapidly, many research projects related Considering characteristics of actual dance per- to these robots have been conducted. To add to formances, there are various musical features af- this research, we are considering to enhance human fecting dance motion. We mainly focused on the dance moiton for entertainment robots, and aiming following correspondences between music and mo- at mimicing dance performance with a biped hu- tion: manoid robot. Rhythm Rhythm is one of the most important We developed an algorithm to enable a humanoid features for dance performance. Even novices robot to represent dance performance17). This al- can recognize musical rhythm, and easily clap gorithm is based on a Learning-from-Observation or wave their hands and legs in response to it. (LFO) paradigm that has a robot directly acquire Speed Dance motion should be synchronized the knowledge of what to do and how to do by ob- with the speed of musical rhythm. Usually, serving a human demonstration. This paradigm is when music gets faster, dance motion is modi- necessary because the difference in body structure fied to follow up the musical speed. between a robot and a human performer makes it Mood Some dance performances are much af- impossible to directly map human motion to a robot fected by musical moods such as happiness and that needs to maintain its balance. We designed task sadness. Even if we don’t dance to music, we models for leg motion of dance performance based feel quiet and relaxed when listening to relax- on contact states, and we have used these to auto- ing music such as a ballad, and we feel excited matically modify human motion for use by a robot. when listening to intense music such as hard However, there still remains the problem that this rock music. algorithm is done offline and a robot cannot lis- Based on these factors, we developed three meth- ten to music and respond to it when performing a ods to analyze and synthesize dance motion with dance. In this paper, we propose three fundamen- musical features based on human perceptions. tal techniques aimed at the achievement of dancing- The first study described in Section 3 is to ana- to-music ability. This ability, which we call sound lyze the relationship between motion and musical feedback system, indicates that people can synchro- rhythm and to extract important stop features in or- nize their dance motion with various musical fea- der to mimic human dance motion. The goal of tures such as rhythm, speed, and mood, even if they this study is to distinguish which features should are novices. The ultimate goal is that a robot listens be preserved for dance motion imitation. Accord- to currently played music and automatically syn- ing to observation of human dance motion, motion chronizes or composes dance motion synchronized rhythm is represented with a stop motion called a keypose, at which dancers clearly stop their move- y1 Department of Information and Communication Engineer- ments, and the motion rhythm is synchronized with ing, Graduate School of Information Science and Technol- ogy, The University of Tokyo musical rhythm when performing a dance. The pro- y2 Interfaculty Initiative in Information Studies, The Univer- posed method aims to reveal this relationship. sity of Tokyo The second study described in Section 4 is to 34 Vol. 1 No. SIG 1(CVIM 1) Synthesis of Dance Performance Based on Analyses of Human Motion and Music 35 model how to modify upper body motion based (HMM)7), or by applying a spatio-temporal isomap on the speed of played music. When we observed for dimensionality reduction8). Kahol et al.9),10) structured dance motion performed at a normal mu- proposed a motion segmentation method using ap- sic playback speed and motion performed at faster proximated physical parameters such as force, mo- music playback speed, we found that the detail of mentum and kinetic energy. We decided to follow each motion is slightly different while the whole of this biomechanical concept basically, and thus we the dance motion is similar in both cases. To prove defined that keyposes in dance motion as stopping this, we analyzed the motion differences in the fre- postures. quency domain, and obtained two insights on the In addition, we focused on the rhythm of dance omission of motion details. performance. The motion rhythm of most dance The third study described in Section 5 is to syn- performance corresponds to music rhtyhm, and thesize dance performance that is well matched to some prior work on character animation uses this the mood of the input music. We mainly focus property for animated motion synthesis1),12),13). So on intensity in dance performance as a mood fea- our method consists of a motion analysis step that ture. We designed an algorithm to synthesize new extracts stopping postures from motion and a mu- dance performance by assuming these relationship sic analysis step that extracts rhythm from music. between motion and music rhythm mentioned in the Combining motion and musical information allows first study, and the relationship between motion and the motion’s keyposes to be established. music intensity. However, dance motion with high 3.1 Rhythm Tracking from Music Sequence intensity is difficult to reproduce with a biped robot To estimate musical rhythm, we use the following due to balance maintenance, and our target in this known principles: study is CG character animation. But we believe Principle 1: A sound is likely to be produced that this method will be applicable to a robot in the consistent with the timing of the rhythm. future. Principle 2: The interval of the onset component is likely to be equal to that of the rhythm. 3. Keypose Extraction for Imitating Dance The onset component represents the spectral Motion26) power increase from the previous temporal frame, Mapping human motion to a humanoid robot is a and we use Goto et al.’s method5) for the extrac- difficult problem, and understanding what features tion of onset components. By applying an auto- in human motion are important and using the fea- correlation function to time series of onset compo- tures for reproduction with robots has been well nents, we can estimate the rhythm of music. studied. Some previous methods have actually ex- 3.2 Keypose Candidate Extraction from Mo- tracted abstract models by recognizing what to do tion Sequence and how to do it, and generating motion for robots Our motion analysis method is based on the from the models7),8),16),20),28). However, we found a speed of a performer’s hands, feet and center of problem that the traditional techniques tended to ex- mass (CM). In many forms of dance, including tract too many models from dance motion19), and it Japanese traditional dance, the movements of hands is nearly impossible to distinguish what is truly im- and feet have a strong relationship with the in- portant for dance performance imitation. This sec- tended expression of the whole body. Therefore, tion describes a novel method to analyze the rela- the speed of the hands and feet is useful for extract- tionship between important postures in dance mo- ing stop motions. However, this is not sufficient for tion and musical rhythms in order to understand keypose extraction because sometimes the dancer the essential features of dance motion. We refer to makes rhythm errors, or dances are varied by the these important postures as keyposes. preferences or the genders of performers, etc. So According to Flash et al.3), every human motion in addition to the motion of the hands and feet, our consists of several motion primitives, which de- algorithm uses the motion of the body’s CM calcu- note fundamental elements of human motion, and lated from standard mass distribution. The motion these primitives are segmented by detecting in- of the CM represents the motion of the whole body; stances when hands and feet stop their movements. thus, the effects of missteps and individual differ- In whole body motion, there are many methods to ences are less. Through this step, we extract motion segment human motion by detecting the local min- keypose candidates that satisfy the following crite- ima of end-effector speed and to classify the mo- ria: tion segment into several clusters by calculating ( 1 ) Dancers clearly stop their movements. co-occurrence21), by using Hidden Markov Models ( 2 ) Dancers clearly move their body parts dur- 36 IPSJ Transactions on Computer Vision and Image Media June 2008 Fig. 1 Keypose candidate extraction for hand and CM Fig. 3 Refinement of the keypose candidates using musical motions.