English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It is widely accepted that fine-tuning pre-trained language models usuallybrings about performance improvements in downstream tasks. However, there arelimited studies on the reasons behind this effectiveness, particularly from theviewpoint of structural changes in the embedding space. Trying to fill thisgap, in this paper, we analyze the extent to which the isotropy of theembedding space changes after fine-tuning. We demonstrate that, even thoughisotropy is a desirable geometrical property, fine-tuning does not necessarilyresult in isotropy enhancements. Moreover, local structures in pre-trainedcontextual word representations (CWRs), such as those encoding token types orfrequency, undergo a massive change during fine-tuning. Our experiments showdramatic growth in the number of elongated directions in the embedding space,which, in contrast to pre-trained CWRs, carry the essential linguisticknowledge in the fine-tuned embedding space, making existing isotropyenhancement methods ineffective.

How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy