English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we propose a deep learning based multi-speaker direction ofarrival (DOA) estimation with audio and visual signals by usingpermutation-free loss function. We first collect a data set for multi-modalsound source localization (SSL) where both audio and visual signals arerecorded in real-life home TV scenarios. Then we propose a novel spatialannotation method to produce the ground truth of DOA for each speaker with thevideo data by transformation between camera coordinate and pixel coordinateaccording to the pin-hole camera model. With spatial location informationserved as another input along with acoustic feature, multi-speaker DOAestimation could be solved as a classification task of active speakerdetection. Label permutation problem in multi-speaker related tasks will beaddressed since the locations of each speaker are used as input. Experimentsconducted on both simulated data and real data show that the proposedaudio-visual DOA estimation model outperforms audio-only DOA estimation modelby a large margin.

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function