English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Annotating datasets for question answering (QA) tasks is very costly, as itrequires intensive manual labor and often domain-specific knowledge. Yetstrategies for annotating QA datasets in a cost-effective manner are scarce. Toprovide a remedy for practitioners, our objective is to develop heuristic rulesfor annotating a subset of questions, so that the annotation cost is reducedwhile maintaining both in- and out-of-domain performance. For this, we conducta large-scale analysis in order to derive practical recommendations. First, wedemonstrate experimentally that more training samples contribute often only toa higher in-domain test-set performance, but do not help the model ingeneralizing to unseen datasets. Second, we develop a model-guided annotationstrategy: it makes a recommendation with regard to which subset of samplesshould be annotated. Its effectiveness is demonstrated in a case study based ondomain customization of QA to a clinical setting. Here, remarkably, annotatinga stratified subset with only 1.2% of the original training set achieves 97.7%of the performance as if the complete dataset was annotated. Hence, thelabeling effort can be reduced immensely. Altogether, our work fulfills ademand in practice when labeling budgets are limited and where thusrecommendations are needed for annotating QA datasets more cost-effectively.

Practical Annotation Strategies for Question Answering Datasets