English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite the remarkable success of deep multi-modal learning in practice, ithas not been well-explained in theory. Recently, it has been observed that thebest uni-modal network outperforms the jointly trained multi-modal network,which is counter-intuitive since multiple signals generally bring moreinformation. This work provides a theoretical explanation for the emergence ofsuch performance gap in neural networks for the prevalent joint trainingframework. Based on a simplified data distribution that captures the realisticproperty of multi-modal data, we prove that for the multi-modal late-fusionnetwork with (smoothed) ReLU activation trained jointly by gradient descent,different modalities will compete with each other. The encoder networks willlearn only a subset of modalities. We refer to this phenomenon as modalitycompetition. The losing modalities, which fail to be discovered, are theorigins where the sub-optimality of joint training comes from. Experimentally,we illustrate that modality competition matches the intrinsic behavior oflate-fusion joint training.

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)