English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Visual and audio events simultaneously occur and both attract attention.However, most existing saliency prediction works ignore the influence of audioand only consider vision modality. In this paper, we propose a multitasklearning method for visual-audio saliency prediction and sound sourcelocalization on multi-face video by leveraging visual, audio and faceinformation. Specifically, we first introduce a large-scale database ofmulti-face video in visual-audio condition (MVVA), containing eye-tracking dataand sound source annotations. Using this database, we find that soundinfluences human attention, and conversly attention offers a cue to determinesound source on multi-face video. Guided by these findings, a visual-audiomulti-task network (VAM-Net) is introduced to predict saliency and locate soundsource. VAM-Net consists of three branches corresponding to visual, audio andface modalities. Visual branch has a two-stream architecture to capture spatialand temporal information. Face and audio branches encode audio signals andfaces, respectively. Finally, a spatio-temporal multi-modal graph (STMG) isconstructed to model the interaction among multiple faces. With jointoptimization of these branches, the intrinsic correlation of the tasks ofsaliency prediction and sound source localization is utilized and theirperformance is boosted by each other. Experiments show that the proposed methodoutperforms 12 state-of-the-art saliency prediction methods, and achievescompetitive results in sound source localization.

Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos