English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Generating vivid and diverse 3D co-speech gestures is crucial for variousapplications in animating virtual avatars. While most existing methods cangenerate gestures from audio directly, they usually overlook that emotion isone of the key factors of authentic co-speech gesture generation. In this work,we propose EmotionGesture, a novel framework for synthesizing vivid and diverseemotional co-speech 3D gestures from audio. Considering emotion is oftenentangled with the rhythmic beat in speech audio, we first develop anEmotion-Beat Mining module (EBM) to extract the emotion and audio beat featuresas well as model their correlation via a transcript-based visual-rhythmalignment. Then, we propose an initial pose based Spatial-Temporal Prompter(STP) to generate future gestures from the given initial poses. STP effectivelymodels the spatial-temporal correlations between the initial poses and thefuture gestures, thus producing the spatial-temporal coherent pose prompt. Oncewe obtain pose prompts, emotion, and audio beat features, we will generate 3Dco-speech gestures through a transformer architecture. However, considering theposes of existing datasets often contain jittering effects, this would lead togenerating unstable gestures. To address this issue, we propose an effectiveobjective function, dubbed Motion-Smooth Loss. Specifically, we model motionoffset to compensate for jittering ground-truth by forcing gestures to besmooth. Last, we present an emotion-conditioned VAE to sample emotion features,enabling us to generate diverse emotional results. Extensive experimentsdemonstrate that our framework outperforms the state-of-the-art, achievingvivid and diverse emotional co-speech 3D gestures. Our code and dataset will bereleased at the project page:https://xingqunqi-lab.github.io/Emotion-Gesture-Web/

EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation