English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Given a pre-trained language model, how can we efficiently compress itwithout retraining? Retraining-free structured pruning algorithms are crucialin pre-trained language model compression due to their significantly reducedpruning cost and capability to prune large language models. However, existingretraining-free algorithms encounter severe accuracy degradation, as they failto preserve the useful knowledge of pre-trained models. In this paper, wepropose K-pruning (Knowledge-preserving pruning), an accurate retraining-freestructured pruning algorithm for pre-trained language models. K-pruningidentifies and prunes attention heads and neurons deemed to be superfluous,based on the amount of their inherent knowledge. K-pruning applies an iterativeprocess of pruning followed by knowledge reconstruction for each sub-layer topreserve the knowledge of the pre-trained models. Consequently, K-pruning showsup to 58.02%p higher F1 score than existing retraining-free pruning algorithmsunder a high compression rate of 80% on the SQuAD benchmark.

Knowledge-preserving Pruning for Pre-trained Language Models without Retraining