English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Accurate syntactic representations are essential for robust generalization innatural language. Recent work has found that pre-training can teach languagemodels to rely on hierarchical syntactic features - as opposed to incorrectlinear features - when performing tasks after fine-tuning. We test what aspectsof pre-training are important for endowing encoder-decoder Transformers with aninductive bias that favors hierarchical syntactic generalizations. We focus onarchitectural features (depth, width, and number of parameters), as well as thegenre and size of the pre-training corpus, diagnosing inductive biases usingtwo syntactic transformation tasks: question formation and passivization, bothin English. We find that the number of parameters alone does not explainhierarchical generalization: model depth plays greater role than model width.We also find that pre-training on simpler language, such as child-directedspeech, induces a hierarchical bias using an order-of-magnitude less data thanpre-training on more typical datasets based on web text or Wikipedia; thissuggests that in cognitively plausible language acquisition settings, neurallanguage models may be more data-efficient than previously thought.

How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases