English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Analysing whether neural language models encode linguistic information hasbecome popular in NLP. One method of doing so, which is frequently cited tosupport the claim that models like BERT encode syntax, is called probing;probes are small supervised models trained to extract linguistic informationfrom another model's output. If a probe is able to predict a particularstructure, it is argued that the model whose output it is trained on must haveimplicitly learnt to encode it. However, drawing a generalisation about amodel's linguistic knowledge about a specific phenomena based on what a probeis able to learn may be problematic: in this work, we show that semantic cuesin training data means that syntactic probes do not properly isolate syntax. Wegenerate a new corpus of semantically nonsensical but syntactically well-formedJabberwocky sentences, which we use to evaluate two probes trained on normaldata. We train the probes on several popular language models (BERT, GPT, andRoBERTa), and find that in all settings they perform worse when evaluated onthese data, for one probe by an average of 15.4 UUAS points absolute. Althoughin most cases they still outperform the baselines, their lead is reducedsubstantially, e.g. by 53% in the case of BERT for one probe. This begs thequestion: what empirical scores constitute knowing syntax?

Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing