English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Excessive reuse of holdout data can lead to overfitting. However, there islittle concrete evidence of significant overfitting due to holdout reuse inpopular multiclass benchmarks today. Known results show that, in theworst-case, revealing the accuracy of $k$ adaptively chosen classifiers on adata set of size $n$ allows to create a classifier with bias of$\Theta(\sqrt{k/n})$ for any binary prediction problem. We show a new upperbound of $\tilde O(\max\{\sqrt{k\log(n)/(mn)},k/n\})$ on the worst-case biasthat any attack can achieve in a prediction problem with $m$ classes. Moreover,we present an efficient attack that achieve a bias of $\Omega(\sqrt{k/(m^2n)})$ and improves on previous work for the binary setting ($m=2$). We alsopresent an inefficient attack that achieves a bias of $\tilde\Omega(k/n)$.Complementing our theoretical work, we give new practical attacks tostress-test multiclass benchmarks by aiming to create as large a bias aspossible with a given number of queries. Our experiments show that theadditional uncertainty of prediction with a large number of classes indeedmitigates the effect of our best attacks.  Our work extends developments in understanding overfitting due to adaptivedata analysis to multiclass prediction problems. It also bears out thesurprising fact that multiclass prediction problems are significantly morerobust to overfitting when reusing a test (or holdout) dataset. This offers anexplanation as to why popular multiclass prediction benchmarks, such asImageNet, may enjoy a longer lifespan than what intuition from literature onbinary classification suggests.

The advantages of multiple classes for reducing overfitting from test set reuse