English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Audio-visual speech recognition (AVSR) attracts a surge of research interestrecently by leveraging multimodal signals to understand human speech.Mainstream approaches addressing this task have developed sophisticatedarchitectures and techniques for multi-modality fusion and representationlearning. However, the natural heterogeneity of different modalities causesdistribution gap between their representations, making it challenging to fusethem. In this paper, we aim to learn the shared representations acrossmodalities to bridge their gap. Different from existing similar methods onother multimodal tasks like sentiment analysis, we focus on the temporalcontextual dependencies considering the sequence-to-sequence task setting ofAVSR. In particular, we propose an adversarial network to refine frame-levelmodality-invariant representations (MIR-GAN), which captures the commonalityacross modalities to ease the subsequent multimodal fusion process. Extensiveexperiments on public benchmarks LRS3 and LRS2 show that our approachoutperforms the state-of-the-arts.

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition