English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretraining multilingual language models from scratch requires considerablecomputational resources and substantial training data. Therefore, a moreefficient method is to adapt existing pretrained language models (PLMs) to newlanguages via vocabulary extension and continued pretraining. However, thismethod usually randomly initializes the embeddings of new subwords andintroduces substantially more embedding parameters to the language model, thusweakening the efficiency. To address these issues, we propose a novelframework: \textbf{O}ne \textbf{F}or \textbf{A}ll (\textbf{\textsc{Ofa}}),which wisely initializes the embeddings of unseen subwords from targetlanguages and thus can adapt a PLM to multiple languages efficiently andeffectively. \textsc{Ofa} takes advantage of external well-aligned multilingualword embeddings and injects the alignment knowledge into the new embeddings. Inaddition, \textsc{Ofa} applies matrix factorization and replaces the cumbersomeembeddings with two lower-dimensional matrices, which significantly reduces thenumber of parameters while not sacrificing the performance. Through extensiveexperiments, we show models initialized by \textsc{Ofa} are efficient andoutperform several baselines. \textsc{Ofa} not only accelerates the convergenceof continued pretraining, which is friendly to a limited computation budget,but also improves the zero-shot crosslingual transfer on a wide range ofdownstream tasks. We make our code and models publicly available.

OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining