English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While the successes of transformers across many domains are indisputable,accurate understanding of the learning mechanics is still largely lacking.Their capabilities have been probed on benchmarks which include a variety ofstructured and reasoning tasks -- but mathematical understanding is laggingsubstantially behind. Recent lines of work have begun studying representationalaspects of this question: that is, the size/depth/complexity of attention-basednetworks to perform certain tasks. However, there is no guarantee the learningdynamics will converge to the constructions proposed. In our paper, we providefine-grained mechanistic understanding of how transformers learn "semanticstructure", understood as capturing co-occurrence structure of words.Precisely, we show, through a combination of mathematical analysis andexperiments on Wikipedia data and synthetic data modeled by Latent DirichletAllocation (LDA), that the embedding layer and the self-attention layer encodethe topical structure. In the former case, this manifests as higher averageinner product of embeddings between same-topic words. In the latter, itmanifests as higher average pairwise attention between same-topic words. Themathematical results involve several assumptions to make the analysistractable, which we verify on data, and might be of independent interest aswell.

How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding