English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Speaker diarization in real-world videos presents significant challenges dueto varying acoustic conditions, diverse scenes, and the presence of off-screenspeakers, among other factors. This paper builds upon a previous study(AVR-Net) and introduces a novel multi-modal speaker diarization system,AFL-Net. Unlike AVR-Net, which independently extracts high-levelrepresentations from each modality, AFL-Net employs a multi-modalcross-attention mechanism. This approach generates high-level representationsfrom each modality while conditioning on each other, ensuring a morecomprehensive information fusion across modalities to enhance identitydiscrimination. Furthermore, the proposed AFL-Net incorporates dynamic lipmovement as an additional modality to aid in distinguishing each segment'sidentity. We also introduce a masking strategy during training that randomlyobscures the face and lip movement modalities, which increases the influence ofthe audio modality on system outputs.Experimental results demonstrate that ourproposed model outperforms state-of-the-art baselines, such as the AVR-Net andDyViSE. Moreover, an ablation study confirms the effectiveness of eachmodification. Some demos areprovided:https://yyk77.github.io/afl_net.github.io.

AFL-Net: Integrating Audio, Facial, and Lip Modalities with Cross-Attention for Robust Speaker Diarization in the Wild