English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Innovations in annotation methodology have been a catalyst for ReadingComprehension (RC) datasets and models. One recent trend to challenge currentRC models is to involve a model in the annotation process: humans createquestions adversarially, such that the model fails to answer them correctly. Inthis work we investigate this annotation methodology and apply it in threedifferent settings, collecting a total of 36,000 samples with progressivelystronger models in the annotation loop. This allows us to explore questionssuch as the reproducibility of the adversarial effect, transfer from datacollected with varying model-in-the-loop strengths, and generalisation to datacollected without a model. We find that training on adversarially collectedsamples leads to strong generalisation to non-adversarially collected datasets,yet with progressive performance deterioration with increasingly strongermodels-in-the-loop. Furthermore, we find that stronger models can still learnfrom datasets collected with substantially weaker models-in-the-loop. Whentrained on data collected with a BiDAF model in the loop, RoBERTa achieves39.9F1 on questions that it cannot answer when trained on SQuAD - onlymarginally lower than when trained on data collected using RoBERTa itself(41.0F1).

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension