English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

How reliably can we trust the scores obtained from social bias benchmarks asfaithful indicators of problematic social biases in a given language model? Inthis work, we study this question by contrasting social biases with non-socialbiases stemming from choices made during dataset construction that might noteven be discernible to the human eye. To do so, we empirically simulate variousalternative constructions for a given benchmark based on innocuousmodifications (such as paraphrasing or random-sampling) that maintain theessence of their social bias. On two well-known social bias benchmarks(Winogender and BiasNLI) we observe that these shallow modifications have asurprising effect on the resulting degree of bias across various models. Wehope these troubling observations motivate more robust measures of socialbiases.

The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks