English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite their omnipresence in modern NLP, characterizing the computationalpower of transformer neural nets remains an interesting open question. We provethat transformers whose arithmetic precision is logarithmic in the number ofinput tokens (and whose feedforward nets are computable using space linear intheir input) can be simulated by constant-depth logspace-uniform thresholdcircuits. This provides insight on the power of transformers using knownresults in complexity theory. For example, if $\mathsf L \neq \mathsf P$ (i.e.,not all poly-time problems can be solved using logarithmic space), thentransformers cannot even accurately solve linear equalities or check membershipin an arbitrary context-free grammar with empty productions. Our resultintuitively emerges from the transformer architecture's high parallelizability.We thus speculatively introduce the idea of a fundamental parallelism tradeoff:any model architecture as parallelizable as the transformer will obeylimitations similar to it. Since parallelism is key to training models atmassive scale, this suggests a potential inherent weakness of the scalingparadigm.

The Parallelism Tradeoff: Limitations of Log-Precision Transformers