English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Cross-lingual transfer learning is an important property of multilinguallarge language models (LLMs). But how do LLMs represent relationships betweenlanguages? Every language model has an input layer that maps tokens to vectors.This ubiquitous layer of language models is often overlooked. We find thatsimilarities between these input embeddings are highly interpretable and thatthe geometry of these embeddings differs between model families. In one case(XLM-RoBERTa), embeddings encode language: tokens in different writing systemscan be linearly separated with an average of 99.2% accuracy. Another family(mT5) represents cross-lingual semantic similarity: the 50 nearest neighborsfor any token represent an average of 7.61 writing systems, and are frequentlytranslations. This result is surprising given that there is no explicitparallel cross-lingual training corpora and no explicit incentive fortranslations in pre-training objectives. Our research opens the door forinvestigations in 1) The effect of pre-training and model architectures onrepresentations of languages and 2) The applications of cross-lingualrepresentations embedded in language models.

Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings