English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How should we compare the capabilities of language models (LMs) and humans? Idraw inspiration from comparative psychology to highlight some challenges. Inparticular, I consider a case study: processing of recursively nestedgrammatical structures. Prior work suggests that LMs cannot handle thesestructures as reliably as humans can. However, the humans were provided withinstructions and training, while the LMs were evaluated zero-shot. I thereforematch the evaluation more closely. Providing large LMs with a simple prompt --substantially less content than the human training -- allows the LMs toconsistently outperform the human results, and even to extrapolate to moredeeply nested conditions than were tested with humans. Further, reanalyzing theprior human data suggests that the humans may not perform above chance at thedifficult structures initially. Thus, large LMs may indeed process recursivelynested grammatical structures as reliably as humans. This case study highlightshow discrepancies in the evaluation can confound comparisons of language modelsand humans. I therefore reflect on the broader challenge of comparing human andmodel capabilities, and highlight an important difference between evaluatingcognitive models and foundation models.

Can language models handle recursively nested grammatical structures? A case study on comparing models and humans