English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Targeted syntactic evaluations of language models ask whether models showstable preferences for syntactically acceptable content over minimal-pairunacceptable inputs. Most targeted syntactic evaluation datasets ask models tomake these judgements with just a single context-free sentence as input. Thisdoes not match language models' training regime, in which input sentences arealways highly contextualized by the surrounding corpus. This mismatch raises animportant question: how robust are models' syntactic judgements in differentcontexts? In this paper, we investigate the stability of language models'performance on targeted syntactic evaluations as we vary properties of theinput context: the length of the context, the types of syntactic phenomena itcontains, and whether or not there are violations of grammaticality. We findthat model judgements are generally robust when placed in randomly sampledlinguistic contexts. However, they are substantially unstable for contextscontaining syntactic structures matching those in the critical test content.Among all tested models (GPT-2 and five variants of OPT), we significantlyimprove models' judgements by providing contexts with matching syntacticstructures, and conversely significantly worsen them using unacceptablecontexts with matching but violated syntactic structures. This effect isamplified by the length of the context, except for unrelated inputs. We showthat these changes in model performance are not explainable by simple featuresmatching the context and the test inputs, such as lexical overlap anddependency overlap. This sensitivity to highly specific syntactic features ofthe context can only be explained by the models' implicit in-context learningabilities.

Language model acceptability judgements are not always robust to context