English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Today, transformer language models serve as a core component for majority ofnatural language processing tasks. Industrial application of such modelsrequires minimization of computation time and memory footprint. Knowledgedistillation is one of approaches to address this goal. Existing methods inthis field are mainly focused on reducing the number of layers or dimension ofembeddings/hidden representations. Alternative option is to reduce the numberof tokens in vocabulary and therefore the embeddings matrix of the studentmodel. The main problem with vocabulary minimization is mismatch between inputsequences and output class distributions of a teacher and a student models. Asa result, it is impossible to directly apply KL-based knowledge distillation.We propose two simple yet effective alignment techniques to make knowledgedistillation to the students with reduced vocabulary. Evaluation of distilledmodels on a number of common benchmarks for Russian such as Russian SuperGLUE,SberQuAD, RuSentiment, ParaPhaser, Collection-3 demonstrated that ourtechniques allow to achieve compression from $17\times$ to $49\times$, whilemaintaining quality of $1.7\times$ compressed student with the full-sizedvocabulary, but reduced number of Transformer layers only. We make our code anddistilled models available.

Knowledge Distillation of Russian Language Models with Reduction of Vocabulary