English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With the rapid growth in model size, fine-tuning the large pre-trainedlanguage model has become increasingly difficult due to its extensive memoryusage. Previous works usually focus on reducing the number of trainableparameters in the network. While the model parameters do contribute to memoryusage, the primary memory bottleneck during training arises from storingfeature maps, also known as activations, as they are crucial for gradientcalculation. Notably, neural networks are usually trained using stochasticgradient descent. We argue that in stochastic optimization, models can handlenoisy gradients as long as the gradient estimator is unbiased with reasonablevariance. Following this motivation, we propose a new family of unbiasedestimators called WTA-CRS, for matrix production with reduced variance, whichonly requires storing the sub-sampled activations for calculating the gradient.Our work provides both theoretical and experimental evidence that, in thecontext of tuning transformers, our proposed estimators exhibit lower variancecompared to existing ones. By replacing the linear operation with ourapproximated one in transformers, we can achieve up to 2.7$\times$ peak memoryreduction with almost no accuracy drop and enables up to $6.4\times$ largerbatch size. Under the same hardware, WTA-CRS enables better down-streaming taskperformance by applying larger models and/or faster training speed with largerbatch sizes.

Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model