English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Bias auditing of language models (LMs) has received considerable attention asLMs are becoming widespread. As such, several benchmarks for bias auditing havebeen proposed. At the same time, the rapid evolution of LMs can make thesebenchmarks irrelevant in no time. Bias auditing is further complicated by LMbrittleness: when a presumably biased outcome is observed, is it due to modelbias or model brittleness? We propose enlisting the models themselves to helpconstruct bias auditing datasets that remain challenging, and introduce biasmeasures that distinguish between different types of model errors. First, weextend an existing bias benchmark for NLI (BBNLI) using a combination ofLM-generated lexical variations, adversarial filtering, and human validation.We demonstrate that the newly created dataset BBNLI-next is more challengingthan BBNLI: on average, BBNLI-next reduces the accuracy of state-of-the-art NLImodels from 95.3%, as observed by BBNLI, to a strikingly low 57.5%. Second, weemploy BBNLI-next to showcase the interplay between robustness and bias: wepoint out shortcomings in current bias scores and propose bias measures thattake into account both bias and model brittleness. Third, despite the fact thatBBNLI-next was designed with non-generative models in mind, we show that thenew dataset is also able to uncover bias in state-of-the-art open-sourcegenerative LMs.  Note: All datasets included in this work are in English and they addressUS-centered social biases. In the spirit of efficient NLP research, no modeltraining or fine-tuning was performed to conduct this research.  Warning: This paper contains offensive text examples.

Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing