English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-trained multimodal foundation models have demonstrated remarkablegeneralizability but pose challenges for deployment due to their large sizes.One effective approach to reducing their sizes is layerwise distillation,wherein small student models are trained to match the hidden representations oflarge teacher models at each layer. Motivated by our observation that certainarchitecture components, referred to as modules, contribute more significantlyto the student's performance than others, we propose to track the contributionsof individual modules by recording the loss decrement after distillation eachmodule and choose the module with a greater contribution to distill morefrequently. Such an approach can be naturally formulated as a multi-armedbandit (MAB) problem, where modules and loss decrements are considered as armsand rewards, respectively. We then develop a modified-Thompson samplingalgorithm named OPTIMA to address the nonstationarity of module contributionsresulting from model updating. Specifically, we leverage the observedcontributions in recent history to estimate the changing contribution of eachmodule and select modules based on these estimations to maximize the cumulativecontribution. We evaluate the effectiveness of OPTIMA through distillationexperiments on various multimodal understanding and image captioning tasks,using the CoCa-Large model (Yu et al., 2022) as the teacher model.

Module-wise Adaptive Distillation for Multimodality Foundation Models