English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The extensive memory footprint of pre-trained language models (PLMs) canhinder deployment in memory-constrained settings, such as cloud environments oron-device. PLMs use embedding matrices to represent extensive vocabularies,forming a large proportion of the model parameters. While previous work towardsparameter-efficient PLM development has considered pruning parameters withinthe transformer layers, pruning the embedding matrix as part of fine-tuning orinference has yet to be explored. We first demonstrate that a significantproportion of the vocabulary remains unused in these scenarios. We then proposea simple yet effective approach that leverages this finding to minimize thememory footprint of the embedding matrix. We show that this approach providessubstantial reductions in memory usage across a wide range of models and tasks.Notably, our approach maintains equivalent downstream task performance whileallowing a more efficient use of compute resources.

Frustratingly Simple Memory Efficiency for Pre-trained Language Models via Dynamic Embedding Pruning