Premium
Accurate automatic visible speech synthesis of arbitrary 3D models based on concatenation of diviseme motion capture data
Author(s) -
Ma Jiyong,
Cole Ronald,
Pellom Bryan,
Ward Wayne,
Wise Barbara
Publication year - 2004
Publication title -
computer animation and virtual worlds
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.225
H-Index - 49
eISSN - 1546-427X
pISSN - 1546-4261
DOI - 10.1002/cav.11
Subject(s) - computer science , motion capture , speech synthesis , concatenation (mathematics) , viseme , dynamic time warping , motion (physics) , face (sociological concept) , artificial intelligence , sequence (biology) , set (abstract data type) , speech recognition , computer vision , waveform , image warping , sentence , mathematics , social science , radar , telecommunications , speech technology , combinatorics , sociology , biology , genetics , programming language
We present a technique for accurate automatic visible speech synthesis from textual input. When provided with a speech waveform and the text of a spoken sentence, the system produces accurate visible speech synchronized with the audio signal. To develop the system, we collected motion capture data from a speaker's face during production of a set of words containing all diviseme sequences in English. The motion capture points from the speaker's face are retargeted to the vertices of the polygons of a 3D face model. When synthesizing a new utterance, the system locates the required sequence of divisemes, shrinks or expands each diviseme based on the desired phoneme segment durations in the target utterance, then moves the polygons in the regions of the lips and lower face to correspond to the spatial coordinates of the motion capture data. The motion mapping is realized by a key‐shape mapping function learned by a set of viseme examples in the source and target faces. A well‐posed numerical algorithm estimates the shape blending coefficients. Time warping and motion vector blending at the juncture of two divisemes and the algorithm to search the optimal concatenated visible speech are also developed to provide the final concatenative motion sequence. Copyright © 2004 John Wiley & Sons, Ltd.