English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Previous work has shown that the representations output by contextuallanguage models are more anisotropic than static type embeddings, and typicallydisplay outlier dimensions. This seems to be true for both monolingual andmultilingual models, although much less work has been done on the multilingualcontext. Why these outliers occur and how they affect the representations isstill an active area of research. We investigate outlier dimensions and theirrelationship to anisotropy in multiple pre-trained multilingual languagemodels. We focus on cross-lingual semantic similarity tasks, as these arenatural tasks for evaluating multilingual representations. Specifically, weexamine sentence representations. Sentence transformers which are fine-tuned onparallel resources (that are not always available) perform better on this task,and we show that their representations are more isotropic. However, we aim toimprove multilingual representations in general. We investigate how much of theperformance difference can be made up by only transforming the embedding spacewithout fine-tuning, and visualise the resulting spaces. We test differentoperations: Removing individual outlier dimensions, cluster-based isotropyenhancement, and ZCA whitening. We publish our code for reproducibility.

Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity