English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Over the last two decades, many exciting variable selection methods have beendeveloped for finding a small group of covariates that are associated with theresponse from a large pool. Can the discoveries from these data miningapproaches be spurious due to high dimensionality and limited sample size? Canour fundamental assumptions about the exogeneity of the covariates needed forsuch variable selection be validated with the data? To answer these questions,we need to derive the distributions of the maximum spurious correlations givena certain number of predictors, namely, the distribution of the correlation ofa response variable $Y$ with the best $s$ linear combinations of $p$ covariates$\mathbf{X}$, even when $\mathbf{X}$ and $Y$ are independent. When thecovariance matrix of $\mathbf{X}$ possesses the restricted eigenvalue property,we derive such distributions for both a finite $s$ and a diverging $s$, usingGaussian approximation and empirical process techniques. However, such adistribution depends on the unknown covariance matrix of $\mathbf{X}$. Hence,we use the multiplier bootstrap procedure to approximate the unknowndistributions and establish the consistency of such a simple bootstrapapproach. The results are further extended to the situation where the residualsare from regularized fits. Our approach is then used to construct the upperconfidence limit for the maximum spurious correlation and to test theexogeneity of the covariates. The former provides a baseline for guardingagainst false discoveries and the latter tests whether our fundamentalassumptions for high-dimensional model selection are statistically valid. Ourtechniques and results are illustrated with both numerical examples and realdata analysis.

Are Discoveries Spurious? Distributions of Maximum Spurious Correlations and Their Applications