English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Representational spaces learned via language modeling are fundamental toNatural Language Processing (NLP), however there has been limited understandingregarding how and when during training various types of linguistic informationemerge and interact. Leveraging a novel information theoretic probing suite,which enables direct comparisons of not just task performance, but theirrepresentational subspaces, we analyze nine tasks covering syntax, semanticsand reasoning, across 2M pre-training steps and five seeds. We identifycritical learning phases across tasks and time, during which subspaces emerge,share information, and later disentangle to specialize. Across these phases,syntactic knowledge is acquired rapidly after 0.5% of full training. Continuedperformance improvements primarily stem from the acquisition of open-domainknowledge, while semantics and reasoning tasks benefit from later boosts tolong-range contextualization and higher specialization. Measuring cross-tasksimilarity further reveals that linguistically related tasks share informationthroughout training, and do so more during the critical phase of learning thanbefore or after. Our findings have implications for model interpretability,multi-task learning, and learning from limited data.

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training