English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Training multimodal foundation models is challenging due to the limitedavailability of multimodal datasets. While many public datasets pair imageswith text, few combine images with audio or text with audio. Even rarer aredatasets that align all three modalities at once. Critical domains such ashealthcare, infrastructure, or transportation are particularly affected bymissing modalities. This makes it difficult to integrate all modalities into alarge pre-trained neural network that can be used out-of-the-box or fine-tunedfor different downstream tasks. We introduce LoReTTa (Linking mOdalities with atRansitive and commutativE pre-Training sTrAtegy) to address this understudiedproblem. Our self-supervised framework unifies causal modeling and maskedmodeling with the rules of commutativity and transitivity. This allows us totransition within and between modalities. As a result, our pre-trained modelsare better at exploring the true underlying joint probability distribution.Given a dataset containing only the disjoint combinations (A, B) and (B, C),LoReTTa can model the relation A <-> C with A <-> B <-> C. In particular, weshow that a transformer pre-trained with LoReTTa can handle any mixture ofmodalities at inference time, including the never-seen pair (A, C) and thetriplet (A, B, C). We extensively evaluate our approach on a synthetic,medical, and reinforcement learning dataset. Across different domains, ouruniversal multimodal transformer consistently outperforms strong baselines suchas GPT, BERT, and CLIP on tasks involving the missing modality tuple.

Training Transitive and Commutative Multimodal Transformers with LoReTTa