English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The goal of this work is to reconstruct high quality speech from lip motionsalone, a task also known as lip-to-speech. A key challenge of lip-to-speechsystems is the one-to-many mapping caused by (1) the existence of homophenesand (2) multiple speech variations, resulting in a mispronounced andover-smoothed speech. In this paper, we propose a novel lip-to-speech systemthat significantly improves the generation quality by alleviating theone-to-many mapping problem from multiple perspectives. Specifically, weincorporate (1) self-supervised speech representations to disambiguatehomophenes, and (2) acoustic variance information to model diverse speechstyles. Additionally, to better solve the aforementioned problem, we employ aflow based post-net which captures and refines the details of the generatedspeech. We perform extensive experiments on two datasets, and demonstrate thatour method achieves the generation quality close to that of real humanutterance, outperforming existing methods in terms of speech naturalness andintelligibility by a large margin. Synthesised samples are available at ourdemo page: https://mm.kaist.ac.kr/projects/LTBS.

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos