English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

It has been well established that increasing scale in deep transformernetworks leads to improved quality and performance. However, this increase inscale often comes with prohibitive increases in compute cost and inferencelatency. We introduce Alternating Updates (AltUp), a simple-to-implement methodto increase a model's capacity without the computational burden. AltUp enablesthe widening of the learned representation, i.e., the token embedding, whileonly incurring a negligible increase in latency. AltUp achieves this by workingon a subblock of the widened representation at each layer and using apredict-and-correct mechanism to update the inactivated blocks. We presentextensions of AltUp, such as its applicability to the sequence dimension, anddemonstrate how AltUp can be synergistically combined with existing approaches,such as Sparse Mixture-of-Experts models, to obtain efficient models with evenhigher capacity. Our experiments on benchmark transformer models and languagetasks demonstrate the consistent effectiveness of AltUp on a diverse set ofscenarios. Notably, on SuperGLUE and SQuAD benchmarks, AltUp enables up to$87\%$ speedup relative to the dense baselines at the same accuracy.

Alternating Updates for Efficient Transformers