Speech Animation
A Practical Model for Live Speech-Driven Lip-Sync
Li Wei and Zhigang Deng ■ University of Houston
he signal-processing and speech-understand- whereas of"ine speech animation synthesis algo- ing communities have proposed several ap- rithms don’t need to meet such tight time con- proaches to generate speech animation based straints. Compared with forced phoneme align- onT live acoustic speech input. For example, based ment for prerecorded speech, this last challenge on a real-time recognized phoneme sequence, re- comes from the low accuracy of state-of-the-art searchers use simple linear smoothing functions to live speech phoneme recognition systems (such as produce corresponding speech animation.1,2 Other the Julius system [http://julius.sourceforge.jp] and approaches train statistical models (such as neural the HTK toolkit [http://htk.eng.cam.ac.uk]). networks) to encode the mapping between acous- To quantify the phoneme recognition accuracy tic speech features and facial movements.3,4 These between the prerecorded and live speech cases, we approaches have demonstrated randomly selected 10 prerecorded sentences and A simple, ef!cient, yet their real-time runtime ef!ciency extracted their phoneme sequences using the Ju- practical phoneme-based on an off-the-shelf computer, lius system, !rst to do forced phoneme-alignment approach to generating but their performance is highly on the clips (called of!ine phoneme alignment) and realistic speech animation in speaker-dependent because of the then as a real-time phoneme recognition engine. individual-speci!c nature of the By simulating the same prerecorded speech clip as real time based on live speech chosen acoustic speech features. live speech, the system generated phoneme output input starts with decomposing Furthermore, the visual realism of sequentially while the speech was being fed into it. lower-face movements and these approaches is insuf!cient, Then, by taking the of"ine phoneme alignment re- ends with applying motion so they’re less suitable for graphics sults as the ground truth, we were able to compute blending. Experiments and and animation applications. the accuracies of the live speech phoneme recogni- comparisons demonstrate the Live speech-driven lip-sync in- tion in our experiment. As Figure 1 illustrates, the realism of this synthesized volves several challenges. First, live speech phoneme recognition accuracy of the speech animation. additional technical challenges same Julius system varies from 45 to 80 percent. are involved compared with pre- Further empirical analysis didn’t show any pat- recorded speech, where expensive global optimiza- terns of incorrectly recognized phonemes (that is, tion techniques can help !nd the most plausible the phonemes often recognized incorrectly in live speech motion corresponding to novel spoken or speech), implying that to produce satisfactory live typed input. In contrast, it’s extremely dif!cult, if speech-driven animation results, any phoneme- not impossible, to directly apply such global opti- based algorithm must take the relatively low pho- mization techniques to live speech-driven lip-sync neme recognition accuracy (for live speech) into applications because the forthcoming (unavailable design consideration. Moreover, that algorithm yet) speech content can’t be exploited during the should be able to perform certain self-corrections synthesis process. Second, live speech-driven lip- at runtime because some phonemes could be in- sync algorithms must be highly ef!cient to en- correctly recognized and input into the algorithm sure real-time speed on an off-the-shelf computer, in a less predictable manner.
50 January/February 2014 Published by the IEEE Computer Society 0272-1716/15/$31.00 © 2015 IEEE 1.0 cy
ra 0.9
cu 0.8
Inspired by these research challenges, we pro- ac pose a practical phoneme-based approach for live n 0.7 speech-driven lip-sync. Besides generating realistic 0.6 gnitio 0.5 speech animation in real time, our phoneme-based co approach can straightforwardly handle speech in- 0.4 e re put from different speakers, which is one of the 0.3 major advantages of phoneme-based approaches 0.2 over acoustic speech feature-driven approaches phonem 0.1 ve
(see the “Related Work in Speech Animation Syn- Li 0 thesis” sidebar).3,4 Speci!cally, we introduce an ef- 1 2 3 4 5 6 7 8 9 10 !cient, simple algorithm to compute each motion Sentence no. segment’s priority and select the plausible segment based on the phoneme information that’s sequen- Figure 1. The Julius system. Its live speech phoneme recognition tially recognized at runtime. Compared with exist- accuracy varied from 45 to 80 percent. ing lip-sync approaches, the main advantages of our method are its ef!ciency, simplicity, and ca- pability of handling live speech input in real time.
Data Acquisition and Preprocessing We acquired a training facial motion dataset for this work by using an optical motion capture system and attaching more than 100 markers to the face of a female native English speaker. We eliminated 3D rigid head motion by using a sin- gular value decomposition (SVD) based statistical shape analysis method.5 The subject was guided to speak a phoneme-balanced corpus consist- ing of 166 sentences with neutral expression; the obtained dataset contains 72,871 motion frames (about 10 minutes of recording, with 120 frames per second). Phoneme labels and durations were (a) (b) automatically extracted from the simultaneously recorded speech data. Figure 2. Facial motion dataset. (a) Among the 102 markers, we used As Figure 2a illustrates, we use 39 markers in the 39 green markers in this work. (b) Illustration of the average face the lower face region in this work, which results markers. in a 117-dimensional feature vector for each mo- tion frame. We apply principal component analy- si –1 si+1 sis (PCA) to reduce the dimension of the mo- tion feature vectors, which allows us to obtain a …… compact representation by only retaining a small si number of principal components. We keep the !ve most signi!cant principal components to cover Figure 3. Motion segmentation. The grids represent motion frames,
96.6 percent of the motion dataset’s variance. where si denotes the motion segment corresponding to phoneme pi. All the other processing steps described here are When we segment a motion sequence based on its phoneme timing performed in parallel in each of the retained !ve information, we keep one overlapping frame (grids with slashes !lled) principal component spaces. between two neighboring segments. We evenly sample !ve frames (grids with red color) to represent a motion segment. Motion Segmentation For each recorded sentence, we segment its motion Motion Segment Normalization sequence based on its phoneme alignment and ex- Because a phoneme’s duration is typically short tract a motion segment for each phoneme occur- (in our dataset, the average phoneme duration is rence. For the motion blending that occurs later, 109 milliseconds), we could use a small number we keep an overlapping region between two neigh- of evenly sampled frames to represent the original boring motion segments. We set the length of the motion. We downsample a motion segment by overlapping region to one frame (see Figure 3). evenly selecting !ve representative frames (see
IEEE Computer Graphics and Applications 51 Speech Animation
Related Work in Speech Animation Synthesis
he essential part of visual speech animation synthesis is References Tdetermining how to model the speech co-articulation 1. M.M. Cohen and D.W. Massaro, “Modeling Coarticulation in effect. Conventional viseme-driven approaches need users Synthetic Visual Speech,” Models and Techniques in Computer to !rst carefully design a set of visemes (or a set of static Animation, Springer, 1993, pp. 139–156. mouth shapes that represent different phonemes) and 2. A. Wang, M. Emmi, and P. Faloutsos, “Assembling an Expressive then employ interpolation functions1 or co-articulation Facial Animation System,” Proc. SIGGRAPH Symp. Video Games, rules to compute in-between frames for speech animation 2007, pp. 21–26. synthesis.2 However, a !xed phoneme-viseme mapping 3. C. Bregler, M. Covell, and M. Slaney, “Video Rewrite: Driving scheme is often insuf!cient to model the co-articulation Visual Speech with Audio,” Proc. SIGGRAPH 97, 1997, pp. phenomenon in human speech production. As such, the 353–360. resulting speech animations often lack variance and realis- 4. Y. Cao et al., “Expressive Speech-Driven Facial Animation,” tic articulation. ACM Trans. Graphics, vol. 24, no. 4, 2005, pp. 1283–1302. Instead of interpolating a set of predesigned visemes, 5. Z. Deng and U. Neumann, “eFASE: Expressive Facial Animation one category of data-driven approaches generates speech Synthesis and Editing with Phoneme-Level Controls,” Proc. animations by optimally selecting and concatenating motion ACM SIGGGRAPH/Eurographics Symp. Computer Animation, units from a precollected database based on various cost 2006, pp. 251–259. functions.3–7 The second category of data-driven speech 6. S.L. Taylor, “Dynamic Units of Visual Speech,” Proc. 11th ACM animation approaches learns statistical models from data.8,9 SIGGRAPH/Eurographics Conf. Computer Animation, 2012, pp. All these data-driven approaches have demonstrated notice- 275–284. able successes for of"ine prerecorded speech animation 7. X. Ma and Z. Deng, “A Statistical Quality Model for Data- synthesis. Unfortunately, they can’t be straightforwardly Driven Speech Animation,” IEEE Trans. Visualization and extended for live speech-driven lip-sync. The main reason Computer Graphics, vol. 18, no. 11, 2012, pp. 1915–1927. is that they typically utilize some form of global optimiza- 8. M. Brand, “Voice Puppetry,” Proc. 26th Annual Conf. Computer tion technique to synthesize the most optimal speech-syn- Graphics and Interactive Techniques, 1999, pp. 21–28. chronized facial motion. In contrast, in live speech-driven 9. T. Ezzat, G. Geiger, and T. Poggio, “Trainable Video-Realistic lip-sync, forthcoming speech (or phoneme) information is Speech Animation,” ACM Trans. Graphics, vol. 21, no. 3, 2002, unavailable. Thus, directly applying such global optimiza- pp. 388–398. tion-based algorithms would be technically infeasible. 10. B. Li et al., “Real-Time Speech Driven Talking Avatar,” J. Several algorithms have been proposed to generate live Tinghua Univ. Science and Technology, vol. 51, no. 9, 2011, pp. speech-driven speech animations. Besides predesigned 1180 –1186. phoneme-viseme mapping,10,11 various statistical models 11. Y. Xu et al., “A Practical and Con!gurable Lip Sync Method (such as neural networks,12 nearest-neighbor search,13 and for Games,” Proc. Motion in Games (MIG), 2013. pp. 109–118. others) have been trained to learn audio-visual mapping 12. P. Hong, Z. Wen, and T.S. Huang, “Real-Time Speech-Driven for this purpose. The common disadvantages of these Face Animation with Expressions Using Neural Networks,” approaches is that the acoustics features used for training IEEE Trans. Neural Networks, vol. 13, no. 4, 2002, pp. 916–927. the statistical audio-to-visual mapping models are highly 13. R. Gutierrez-Osuna et al., “Speech-Driven Facial Animation speaker-speci!c, as pointed out by Sarah Taylor and her with Realistic Dynamics,” IEEE Trans. Multimedia, vol. 7, no. 1, colleagues.6 2005, pp. 33–42.
Figure 3); we represent each motion segment si as After this clustering step, we essentially convert
a vector vi that concatenates the PCA coef!cients the motion dataset to a set of AnimPho sequences. of the !ve sampled frames. Then, we further precompute the following two data structures: Clustering We apply a K-means clustering algorithm to the ■ AnimPho transition table, T, is an m × m table,
motion vector dataset {vi: i = 1…n}, where n is the where m is the number of AnimPhos. T(ai, aj) =
total number of motion vectors, and then use Eu- 1 if the AnimPho transition
clidean distance to compute the distance between motion dataset; otherwise, T(ai, aj) = 0. In this
two motion vectors—we call the center of each article, we say two AnimPhos ai and aj are con-
obtained motion cluster an AnimPho. Next, we nected if and only if T(ai, aj) = 1. We also de!ne
obtain a set of AnimPhos {ai: i = 1…m}, where m T(ai) = {aj|T(ai, aj) = 1} as the set of AnimPhos
is the total number of AnimPhos, and ai is the ith connected to ai. AnimPho. ■ Phoneme-AnimPho mapping table, L, is an h × m
52 January/February 2015 p1 p2 p3 table, where h is the number of phonemes used (we used 43 English phonemes in this work). L(p , a) = 1 if the phoneme label of a is pho- i j j 1 4 7 neme pi in the dataset; otherwise, L(pi, aj) = 0.
Similarly, we de!ne L(pi) ={aj|L(pi, aj) = 1} as the 2 5 8 set of AnimPhos that have the phoneme label pi.
3 6 9 Live Speech-Driven Lip-Sync Miss Miss In the work described here, two steps generate lip motion based on live speech input: AnimPho selection and motion blending. Figure 4. Naive AnimPho selection strategy. The circles on the top are the phonemes that come to the AnimPho Selection system sequentially. The square below each phoneme
Assuming phoneme pt arrives at the lip-sync sys- pi is L(pi), and the number in the squares denotes the tem at time t, our algorithm needs to !nd its cor- index of AnimPho. Dotted directional lines denote responding AnimPho, at, to animate pt. A naive existing AnimPho transitions in the dataset. The solution would be to take the AnimPho set, φ(pt) green AnimPhos are selected based on the naive
= L(pt) ^ T(at-1), which is the set of AnimPhos that AnimPho selection strategy. have the phoneme label pt and have a natural con- nection with at-1, and then choose one of them 100 percent, and the natural AnimPho connection as at by minimizing a cost function (for example, information (dotted directional lines in Figure 4) the smoothness energy de!ned in Equation 1). If wouldn’t be exploited.
φ(pt) is empty (called a miss in this article), then To solve both the oversmoothness and the inaccu- we reassign all the AnimPhos (regardless of their rate articulation issues, we need to properly utilize phoneme labels) as φ(pt): the AnimPho connection information, especially when the missing rate is high. In our approach,
we compute the priority value, vi, for each Anim- aEt = argmin dist ()()aBt−1 , ()η , (1) ηφ∈ ()pt Pho ai ∈ L(pt) based on the precomputed transition information. Speci!cally, the priority value of an where dist(E(at–1), B(η)) returns the Euclidean AnimPho ai is de!ned as the length of the longest distance between the ending frame of at–1 and the connected AnimPho sequence corresponding to the starting frame of η. If L(pt) ^ T(at–1) isn’t empty, this observed phoneme sequence (up to pt). The priority method can ensure the co-articulation by !nding value vi of ai is calculated as follows: the AnimPho at that’s naturally connected to at–1 in the captured dataset. However, it doesn’t work max,()vajj+ 1 ∃∈Lp()ti−1 aT∈ ()aj well in practical applications, primarily because the vi . (2) missing rate of this method is typically very high 1 Otherwise because of the limited size of the captured dataset.
As a result, the synthesized animations are often Then, we select AnimPho at by taking both the incorrectly articulated and oversmoothed (that is, priority value and the smoothness term into con- only minimizing the smoothness energy de!ned sideration. In our approach, we prefer to select the in Equation 1). AnimPho with the highest priority value; if multi- Figure 4 illustrates a simple example of this na- ple AnimPhos have the same high-priority values, ive AnimPho selection strategy. Let’s assume we we select the smoothest one when it’s connected initially select the third AnimPho for phoneme p1. with at-1.
When p2 comes, we !nd that none of the Anim- We use the same illustrative example in Figure
Phos in L(p2) can connect to the third AnimPho in 5 to explain our algorithm. We start by selecting the dataset, so we select the one with a minimum the third AnimPho for p1. When p2 comes, we !rst smoothness energy (assuming it’s the sixth Anim- compute the priority value for each AnimPho in
Pho). When p3 comes, again we !nd that none of L(p2) by using the AnimPho transition informa- the AnimPhos in L(p3) can connect to the sixth tion. The priority value of the fourth AnimPho,
AnimPho, so we select an AnimPho (assuming it’s v4, is 2, because the AnimPho sequence <1, 4> the seventh one) based on the smoothness energy. corresponds to the observed phoneme sequence
The missing rate of this simple example would be
IEEE Computer Graphics and Applications 53 Speech Animation
p1 p2 p3
■ The limited size of the motion dataset. In most practical applications, the acquired facial mo- tion dataset typically has a limited size due to 1 4 7 1 2 1 data acquisition costs and other reasons. For example, the average number of outgoing tran- 2 58 1 2 3 sitions from an AnimPho in our dataset is 3.45. Therefore, once we pick an AnimPho for pt–1, the 3 6 9 forward AnimPho transitions for pt are limited 1 1 1 (that is, on average 3.45). Miss In theory, to dramatically reduce the missing rate Figure 5. Priority-incorporated AnimPho selection (get close to zero), these two issues must be well strategy. Numbers in red boxes are the AnimPho taken care of. In this work, rather than capturing priority values. a much larger facial motion dataset to reduce the missing rate, we focus on how to utilize a limited value of the !fth AnimPho as 2. After we com- motion dataset to produce acceptable live speech-
pute priority values, we select one AnimPho for p2. driven lip-sync results. The motion-blending step Although we !nd that none of the AnimPhos in can handle the discontinuity incurred by the miss-
L(p2) is connected to the third AnimPho, instead ing event. of selecting the sixth AnimPho that minimizes the smoothness term, we choose the fourth or the Motion Blending !fth AnimPho (both have the same higher priority In this section, we describe how to generate the value). Assuming the !fth AnimPho has a smaller !nal facial animation by using a motion-blending
smoothness energy than the fourth AnimPho, we technique based on the selected AnimPho at and
select the !fth for p2. When phoneme p3 comes, the duration of phoneme pt (which is output from the eighth AnimPho has the highest priority value, a live speech phoneme-recognition system such
so we simply select it for p3. as Julius). Speci!cally, this technique consists of
As Figure 6 shows, compared with the naive three steps: selecting an AnimPho bt that both
AnimPho selection strategy, our approach can sig- connects to at–1 and has the most similar motion
ni!cantly reduce the missing rate. Still, the miss- trajectory with at; generating an intermediate mo-
ing rate in our approach is substantial (around 50 tion segment mt based on at–1, at, and bt; and doing
percent), for two reasons: motion interpolation on mt to produce the desired
number of animation frames for pt, based on the
■ The unpredictability of the next phoneme. At the inputted duration of pt. AnimPho selection step in live speech-driven
lip-sync, the next (forthcoming) phoneme in- Selection of bt. As mentioned earlier, at runtime, our formation is unavailable, so we can’t predict live speech-driven lip-sync approach still has a non- which AnimPho would provide the optimal trivial missing rate (on average, about 50 percent) transition to the next phoneme. at the AnimPho selection step. To solve the possible
0.9 0.8 Our method 0.7 Naive method
te 0.6 ra 0.5
Miss 0.4 0.3 0.2 0.1 0 12345678 Sentence no. Figure 6. The naive AnimPho selection strategy versus our approach. The naive approach has a higher missing rate when tested on the same eight phrases.
54 January/February 2015 a t , bt motion discontinuity issue between at–1 and at, we need to identify and utilize an existing AnimPho transition
bat = argmin dist η, t , (3) () at η∈Ta()t−1 , bt where dist(η, at) returns the Euclidean distance between at and η. Note that if at ∈ T(at–1), which Motion means a natural AnimPho transition exists from at–1 to at in the dataset, Equation 3 is guaranteed to return bt = at. p p t–1 t Time (b) Generation of an intermediate motion segment, mt.
After bt is identi!ed, we need to modify it to en- sure the motion continuity between the previous animation segment at–1 and the animation for pt a m t , t that we’re going to synthesize. Here, we use bt′ to bt dt denote the modi!ed version of bt. We generate an intermediate motion segment mt by linearly blend- Motion ing bt′ and at.
All the AnimPhos (including bt, at–1, and at) store the center of the corresponding motion seg- ment clusters, as described earlier; they don’t re- p p t–1 t Time tain the original motion trajectory in the dataset. (c) As a result, the starting frame of bt and the ending frame of at–1 may be slightly mismatched, although Figure 7. The motion-blending and interpolation process. (a) After the the AnimPho transition
bt′ and at, to obtain an intermediate motion seg- for phoneme pt based on its duration. We calculate ment mt, as Figure 7c illustrates. As described ear- the required number of frames N in !nal anima- lier, we use give evenly sampled frames to represent tion as N = duration(pt) × FPS, where duration(pt) a motion segment. Therefore, this linear blending denotes the duration of phoneme pt and FPS is the can be described as follows. resulting animation’s frames per second. In this work, we use the Hermit interpolation to i −1 i −1 compute in-between frames in two sample frames. mitt[]=1− ×bi′ []+ ×ait [],(()15≤≤i , (4) 4 4 The derivatives of the !ve sample frames are com- puted as follows: where mt[i] is the ith frame of motion segment mt. di[]51if = t−1 Motion interpolation. By taking the downsampled dit []= ()mitt[]+ 11−−mi[]/ 21if <
IEEE Computer Graphics and Applications 55 Speech Animation
Input: Sample frames, mt; the derivatives of sample frames, dt; the target number of frames, N. To better understand the accuracy of our ap- Output: The interpolated sequence, St. 1: for i = 1 → N do proach in different situations, we synthesized mo- 2: Find the upper and lower sample frame [lower, upper]= Bound(i) tion in three cases, as shown in Table 1. Here, live 3: α = (i - lower)/(upper - lower) speech phoneme recognition means that the pho- neme sequence is sequentially outputted from Julius 4: St[i] = HermitInterpolation(upper, lower, α) 5: end for by simulating the prerecorded speech as live speech input. Real-time facial motion synthesis means the Figure 8. Algorithm 1. For motion interpolation, we take the upper and facial motion is synthesized phoneme by phoneme, lower sample frames as input and then compute the in-between frame as described earlier. Our approach was used as the corresponding to α ∈ [0, 1]. synthesis algorithm in cases 1 and 2. We also compared our approach with a baseline Table 1. Three cases/con!gurations for our lip-sync experiments. method1 that adopts the classical Cohen-Massaro 7 Aproach Live speech phoneme recognition Real-time facial motion synthesis co-articulation model to generate speech anima- tion frames via interpolation. This method also Case 1 Not used Yes needs users to manually specify (or design) visemes, Case 2 Yes Yes which are a set of static mouth shapes that repre- Baseline Not used Yes sent different phonemes. To ensure a fair compari- son between our approach and the chosen baseline, 1 where dt–1 is the derivatives of the previous downs- the visemes used in the baseline’s implementation ampled motion segment. As illustrated in Figure 7c, were automatically extracted as the center (mean) the Hermit interpolation can guarantee the result- of the cluster that encloses all the representative 2 ing animation’s C smoothness. Finally, we use mt values of the corresponding phonemes.
and dt to interpolate in-between animation frames. Table 2 shows the comparison results. Case 2 has The motion interpolation algorithm can be described the largest RMSE errors in the three test cases due using Algorithm 1 (see Figure 8). Speci!cally, func- to the inaccuracy of live speech phoneme recogni- tion p=HermitInterpolation(upper, lower, tion. The RMSE error of case 1 is smaller than that α) takes the upper and lower sample frames as of the baseline approach in all three test cases, but input and computes the in-between frame corre- the animation results in case 1 are clearly better sponding to α ∈ [0, 1]. than those in the baseline approach in terms of motion smoothness and visual articulation. Results and Evaluations We implemented our approach using C++ without Runtime Performance Analysis GPU acceleration (see Figure 9). The off-the-shelf To analyze our approach’s runtime ef!ciency, we test computer we used in all our experiments is used three prerecorded speech clips, each of which an Intel core I7 2.80-GHz CPU with 3 Gbytes is about two minutes in duration. We sequentially of memory. The real-time phoneme recognition fed the speech clips into our system by simulating system is Julius, which we trained with the open them as live speech input. During the live speech- source VoxForge acoustic model. In our system, driven lip-sync process, our approach’s runtime the 3D face model consists of 30,000 triangles, performance was recorded simultaneously. Table and a thin-shell deformation algorithm is used to 3 shows the breakdown of its per-phoneme deform the face model based on synthesized facial computing time. From the table, we can make two marker displacements. To improve the realism of observations. synthesized facial animation, we also synthesized First, the real-time phoneme recognition step corresponding head and eye movements simul- consumes about 60 percent of our approach’s to- taneously by implementing a live speech-driven tal computing time on average. Its average motion head-and-eye motion generator.6 synthesis time is about 40 percent (around 40.85 ms) of the total computing time, which indicates Quantitative Evaluation that our lip-sync algorithm itself is highly ef!cient. To quantitatively evaluate the quality of our ap- Second, when we calculated the average phoneme proach, we randomly selected three speech clips number per second in the test data and the average with associated motion from the captured data- FPS, we computed the average FPS as follows: set and computed the root mean squared error
(RMSE) between the synthesized motion and the N Number of Frames()pi captured motion (only taking markers in the lower ∑ i=1 TotalCompute Tiime(p ) AverageFPS = i , (6) face into account, as illustrated in Figure 2). N
56 January/February 2015 Figure 9. Experimental scenario. Several selected frames synthesized by our approach. where N is the total number of phonemes in the test Table 2. RMSE of the three test speech clips. data; Number of Frames(pi) denotes the number Aproach Audio 1 Audio 2 Audio 3 of animation frames needed to visually represent Case 1 18.81 19.80 25.77 pi, which can be simply calculated as its duration (in seconds) multiplied by 30 frames (per second) Case 2 24.16 25.79 27.81 in our experiment; and Total Computing Time(pi) Baseline 22.34 22.40 26.17 denotes the total computing time used by our approach for phoneme pi including the real-time n the one hand, our approach is simple and phoneme recognition time and motion synthesis Oef!cient and can be straightforwardly imple- time. As Table 3 shows, the calculated average FPS mented. On the other, it works surprisingly well of our approach is around 27, which indicates that for most practical applications, achieving a good our approach is able to approximately achieve real- balance between animation realism and runtime time speed on an off-the-shelf computer, without ef!ciency, as demonstrated by our results. Because GPU acceleration. our approach is phoneme-based, it can naturally
Table 3. Breakdown of our approach’s per-phoneme computing time. Test audio Phoneme recognition Motion synthesis Motion blending Average total time Average phonemes Average frames per clip time (ms) (ms) (ms) (ms) per second second (FPS) 1 76.73 38.56 0.16 128.28 8.84 25.62 2 90.13 39.10 0.14 144.40 7. 29 30.03 3 93.80 44.89 0.15 142.23 7.43 26.77
IEEE Computer Graphics and Applications 57 Speech Animation
handle speech input from different speakers, Acknowledgments assuming the employed state-of-the-art speech This work was supported in part by NSF IIS-0914965, recognition engine can reasonably handle the NIH 1R21HD075048-01A1, and NSFC Overseas speaker-independence issue. and Hong Kong/Macau Young Scholars Collabora- Despite its effectiveness and ef!ciency, however, tive Research Award (project 61328204). We thank our current approach has two limitations. One, Bingfeng Li and Lei Xie for helping with the Julius the quality of visual speech animation synthesized system and for numerous insightful technical discus- by our approach substantially depends on the sions. Any opinions, "ndings, and conclusions or rec- ef!ciency and accuracy of the used real-time speech ommendations expressed in this material are those of (or phoneme) recognition engine. If the speech the authors and do not necessarily re!ect the views of recognition engine can’t recognize phonemes in the agencies. real time or does it with a low accuracy, then such a delay or inaccuracy will be unavoidably propagated to or even be exaggerated in the References employed AnimPho selection strategy. Fortunately, 1. B. Li et al., “Real-Time Speech Driven Talking with the continuous improvement of state-of- Avatar,” J. Tinghua Univ. Science and Technology, vol. the-art real-time speech recognition techniques, 51, no. 9, 2011, pp. 1180–1186. we anticipate that our approach will eventually 2. Y. Xu et al., “A Practical and Con!gurable Lip Sync generate more realistic live speech-driven lip-sync Method for Games,” Proc. Motion in Games (MIG), without modi!cations. Two, our current approach 2013. pp. 109–118. doesn’t consider affective state enclosed in live 3. P. Hong, Z. Wen, and T.S. Huang, “Real-Time Speech- speech and thus can’t automatically synthesize Driven Face Animation with Expressions Using corresponding facial expressions. One plausible Neural Networks,” IEEE Trans. Neural Networks, vol. solution to this problem would be to utilize state- 13, no. 4, 2002, pp. 916–927. of-the-art techniques to recognize the change of a 4. R. Gutierrez-Osuna et al., “Speech-Driven Facial subject’s affective state in real time based on his Animation with Realistic Dynamics,” IEEE Trans. or her live speech alone; such information could Multimedia, vol. 7, no. 1, 2005, pp. 33–42, 2005. then be continuously fed into an advanced version 5. Z. Deng and U. Neumann, “eFASE: Expressive Facial of our current approach that can dynamically Animation Synthesis and Editing with Phoneme-Level incorporate the emotion factor into the expressive Controls,” Proc. ACM SIGGGRAPH/Eurographics facial motion synthesis process. Symp. Computer Animation, 2006, pp. 251–259. As future work, in addition to extending the 6. B. Le, X. Ma, and Z. Deng, “Live Speech Driven current approach to various mobile platforms, we Head-and-Eye Motion Generators,” IEEE Trans. also plan to conduct comprehensive user studies Visualization and Computer Graphics, vol. 18, no. 11, to evaluate its effectiveness and usability. We also 2012, pp. 1902–1914. want to investigate better methods to increase 7. M.M. Cohen and D.W. Massaro, “Modeling phoneme recognition accuracy from live speech Coarticulation in Synthetic Visual Speech,” Models and to !ducially retarget 3D facial motion capture and Techniques in Computer Animation, Springer, 1993, data to any static 3D face models. pp. 139–156.
Li Wei is a PhD student in the Department of Computer Science at the University of Houston. His research interests include computer graphics and animation. Wei received a BS in automation and an MS in automation from Xiamen University, China. Contact him at [email protected].
Zhigang Deng is an associate professor of computer science at the University of Houston. His research interests include computer graphics, computer animation, and human-computer interaction. Deng received a PhD in computer science from the University of Southern California. Contact him at zdeng4@ uh.edu.
Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
58 January/February 2015