English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

People perceive the world with multiple senses (e.g., through hearing sounds,reading words and seeing objects). However, most existing AI systems onlyprocess an individual modality. This paper presents an approach that excels athandling multiple modalities of information with a single model. In our"{SkillNet}" model, different parts of the parameters are specialized forprocessing different modalities. Unlike traditional dense models that alwaysactivate all the model parameters, our model sparsely activates parts of theparameters whose skills are relevant to the task. Such model design enablesSkillNet to learn skills in a more interpretable way. We develop our model forfive modalities including text, image, sound, video and code. Results showthat, SkillNet performs comparably to five modality-specific fine-tuned models.Moreover, our model supports self-supervised pretraining with the same sparselyactivated way, resulting in better initialized parameters for differentmodalities. We find that pretraining significantly improves the performance ofSkillNet on five modalities, on par with or even better than baselines withmodality-specific pretraining. On the task of Chinese text-to-image retrieval,our final system achieves higher accuracy than existing leading systemsincluding Wukong{ViT-B} and Wenlan 2.0 while using less number of activatedparameters.

One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code