English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Transformer-based models have exhibited superior performance in variousnatural language processing and computer vision tasks. However, these modelscontain enormous amounts of parameters, which restrict their deployment toreal-world applications. To reduce the model size, researchers prune thesemodels based on the weights' importance scores. However, such scores areusually estimated on mini-batches during training, which incurs largevariability/uncertainty due to mini-batch sampling and complicated trainingdynamics. As a result, some crucial weights could be pruned by commonly usedpruning methods because of such uncertainty, which makes training unstable andhurts generalization. To resolve this issue, we propose PLATON, which capturesthe uncertainty of importance scores by upper confidence bound (UCB) ofimportance estimation. In particular, for the weights with low importancescores but high uncertainty, PLATON tends to retain them and explores theircapacity. We conduct extensive experiments with several Transformer-basedmodels on natural language understanding, question answering and imageclassification to validate the effectiveness of PLATON. Results demonstratethat PLATON manifests notable improvement under different sparsity levels. Ourcode is publicly available at https://github.com/QingruZhang/PLATON.

PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance