English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Conversation is an essential component of virtual avatar activities in themetaverse. With the development of natural language processing, textual andvocal conversation generation has achieved a significant breakthrough. However,face-to-face conversations account for the vast majority of dailyconversations, while most existing methods focused on single-person talkinghead generation. In this work, we take a step further and consider generatingrealistic face-to-face conversation videos. Conversation generation is morechallenging than single-person talking head generation, since it not onlyrequires generating photo-realistic individual talking heads but also demandsthe listener to respond to the speaker. In this paper, we propose a novelunified framework based on neural radiance field (NeRF) to address this task.Specifically, we model both the speaker and listener with a NeRF framework,with different conditions to control individual expressions. The speaker isdriven by the audio signal, while the response of the listener depends on bothvisual and acoustic information. In this way, face-to-face conversation videosare generated between human avatars, with all the interlocutors modeled withinthe same network. Moreover, to facilitate future research on this task, wecollect a new human conversation dataset containing 34 clips of videos.Quantitative and qualitative experiments evaluate our method in differentaspects, e.g., image quality, pose sequence trend, and naturalness of therendering videos. Experimental results demonstrate that the avatars in theresulting videos are able to perform a realistic conversation, and maintainindividual styles. All the code, data, and models will be made publiclyavailable.

DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generation