English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The predictions of question answering (QA)systems are typically evaluatedagainst manually annotated finite sets of one or more answers. This leads to acoverage limitation that results in underestimating the true performance ofsystems, and is typically addressed by extending over exact match (EM) withpre-defined rules or with the token-level F1 measure. In this paper, we presentthe first systematic conceptual and data-driven analysis to examine theshortcomings of token-level equivalence measures.  To this end, we define the asymmetric notion of answer equivalence (AE),accepting answers that are equivalent to or improve over the reference, andpublish over 23k human judgments for candidates produced by multiple QA systemson SQuAD. Through a careful analysis of this data, we reveal and quantifyseveral concrete limitations of the F1 measure, such as a false impression ofgraduality, or missing dependence on the question.  Since collecting AE annotations for each evaluated model is expensive, welearn a BERT matching (BEM) measure to approximate this task. Being a simplertask than QA, we find BEM to provide significantly better AE approximationsthan F1, and to more accurately reflect the performance of systems.  Finally, we demonstrate the practical utility of AE and BEM on the concreteapplication of minimal accurate prediction sets, reducing the number ofrequired answers by up to x2.6.

Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation