English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we introduce a novel audio-visual multi-modal bridgingframework that can utilize both audio and visual information, even withuni-modal inputs. We exploit a memory network that stores source (i.e., visual)and target (i.e., audio) modal representations, where source modalrepresentation is what we are given, and target modal representations are whatwe want to obtain from the memory network. We then construct an associativebridge between source and target memories that considers the interrelationshipbetween the two memories. By learning the interrelationship through theassociative bridge, the proposed bridging framework is able to obtain thetarget modal representations inside the memory network, even with the sourcemodal input only, and it provides rich information for its downstream tasks. Weapply the proposed framework to two tasks: lip reading and speechreconstruction from silent video. Through the proposed associative bridge andmodality-specific memories, each task knowledge is enriched with the recalledaudio context, achieving state-of-the-art performance. We also verify that theassociative bridge properly relates the source and target memories.

Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video