English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Distances between data points are widely used in machine learningapplications. Yet, when corrupted by noise, these distances -- and thus themodels based upon them -- may lose their usefulness in high dimensions. Indeed,the small marginal effects of the noise may then accumulate quickly, shiftingempirical closest and furthest neighbors away from the ground truth. In thispaper, we exactly characterize such effects in noisy high-dimensional datausing an asymptotic probabilistic expression. Previously, it has been arguedthat neighborhood queries become meaningless and unstable when distanceconcentration occurs, which means that there is a poor relative discriminationbetween the furthest and closest neighbors in the data. However, we concludethat this is not necessarily the case when we decompose the data in a groundtruth -- which we aim to recover -- and noise component. More specifically, wederive that under particular conditions, empirical neighborhood relationsaffected by noise are still likely to be truthful even when distanceconcentration occurs. We also include thorough empirical verification of ourresults, as well as interesting experiments in which our derived 'phase shift'where neighbors become random or not turns out to be identical to the phaseshift where common dimensionality reduction methods perform poorly or well forrecovering low-dimensional reconstructions of high-dimensional data with densenoise.

The Curse Revisited: When are Distances Informative for the Ground Truth in Noisy High-Dimensional Data?