English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A growing body of recent evidence has highlighted the limitations of naturallanguage processing (NLP) datasets and classifiers. These include the presenceof annotation artifacts in datasets, classifiers relying on shallow featureslike a single word (e.g., if a movie review has the word "romantic", the reviewtends to be positive), or unnecessary words (e.g., learning a proper noun toclassify a movie as positive or negative). The presence of such artifacts hassubsequently led to the development of challenging datasets to force the modelto generalize better. While a variety of heuristic strategies, such ascounterfactual examples and contrast sets, have been proposed, the theoreticaljustification about what makes these examples difficult for the classifier isoften lacking or unclear. In this paper, using tools from information geometry,we propose a theoretical way to quantify the difficulty of an example in NLP.Using our approach, we explore difficult examples for several deep learningarchitectures. We discover that both BERT, CNN and fasttext are susceptible toword substitutions in high difficulty examples. These classifiers tend toperform poorly on the FIM test set. (generated by sampling and perturbingdifficult examples, with accuracy dropping below 50%). We replicate ourexperiments on 5 NLP datasets (YelpReviewPolarity, AGNEWS, SogouNews,YelpReviewFull and Yahoo Answers). On YelpReviewPolarity we observe acorrelation coefficient of -0.4 between resilience to perturbations and thedifficulty score. Similarly we observe a correlation of 0.35 between thedifficulty score and the empirical success probability of random substitutions.Our approach is simple, architecture agnostic and can be used to study thefragilities of text classification models. All the code used will be madepublicly available, including a tool to explore the difficult examples forother datasets.

Geometry matters: Exploring language examples at the decision boundary