English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The success of over-parameterized neural networks trained to near-zerotraining error has caused great interest in the phenomenon of benignoverfitting, where estimators are statistically consistent even though theyinterpolate noisy training data. While benign overfitting in fixed dimensionhas been established for some learning methods, current literature suggeststhat for regression with typical kernel methods and wide neural networks,benign overfitting requires a high-dimensional setting where the dimensiongrows with the sample size. In this paper, we show that the smoothness of theestimators, and not the dimension, is the key: benign overfitting is possibleif and only if the estimator's derivatives are large enough. We generalizeexisting inconsistency results to non-interpolating models and more kernels toshow that benign overfitting with moderate derivatives is impossible in fixeddimension. Conversely, we show that rate-optimal benign overfitting is possiblefor regression with a sequence of spiky-smooth kernels with large derivatives.Using neural tangent kernels, we translate our results to wide neural networks.We prove that while infinite-width networks do not overfit benignly with theReLU activation, this can be fixed by adding small high-frequency fluctuationsto the activation function. Our experiments verify that such neural networks,while overfitting, can indeed generalize well even on low-dimensional datasets.

Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension