English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

LLMs have demonstrated remarkable capability for understanding semantics, butthey often struggle with understanding pragmatics. To demonstrate this fact, werelease a Pragmatics Understanding Benchmark (PUB) dataset consisting offourteen tasks in four pragmatics phenomena, namely, Implicature,Presupposition, Reference, and Deixis. We curated high-quality test sets foreach task, consisting of Multiple Choice Question Answers (MCQA). PUB includesa total of 28k data points, 6.1k of which have been created by us, and the restare adapted from existing datasets. We evaluated nine models varying in thenumber of parameters and type of training. Our study indicates that fine-tuningfor instruction-following and chat significantly enhances the pragmaticscapabilities of smaller language models. However, for larger models, the baseversions perform comparably with their chat-adapted counterparts. Additionally,there is a noticeable performance gap between human capabilities and modelcapabilities. Furthermore, unlike the consistent performance of humans acrossvarious tasks, the models demonstrate variability in their proficiency, withperformance levels fluctuating due to different hints and the complexities oftasks within the same dataset. Overall, the benchmark aims to provide acomprehensive evaluation of LLM's ability to handle real-world language tasksthat require pragmatic reasoning.

PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities