Open AccessSeamsTalk: Seamless Talking Face Generation via Flow-Guided Inpainting
Author(s) -
Yeongho Jeong,
Gyeongman Kim,
Doohyuk Jang,
Jaeryong Hwang,
Eunho Yang
Publication year - 2024
Publication title -
ieee access
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.587
H-Index - 127
ISSN - 2169-3536
DOI - 10.1109/access.2024.3381992
Subject(s) - aerospace , bioengineering , communication, networking and broadcast technologies , components, circuits, devices and systems , computing and processing , engineered materials, dielectrics and plasmas , engineering profession , fields, waves and electromagnetics , general topics for engineers , geoscience , nuclear engineering , photonics and electrooptics , power, energy and industry applications , robotics and control systems , signal processing and analysis , transportation
Talking face generation aims to generate a face video to speak according to a given audio or driving video. Despite the importance of natural lower face movement, previous approaches have focused only on animating the lip, neglecting the connection between the modified lower face and the original background. As a result, the generated face is not smoothly integrated into the original video. To address this, we propose a new method to create a seamless talking face video by reformulating talking face generation as a conditional video in-painting. Moreover, since previous methods solely rely on referencing the original frame, the original frame’s lip shape influences a generated lip shape. Therefore, we devise a two-stage pipeline that leverages the original frame to reduce scene-specific information loss in the lower face and then utilizes multiple other frames to generate a desired lip shape. Experimental results demonstrate that our method generates a seamless talking face while maintaining similarly accurate lip shapes compared to existing methods.