Premium
Semisupervised inference for explained variance in high dimensional linear regression and its applications
Author(s) -
Tony Cai T.,
Guo Zijian
Publication year - 2020
Publication title -
journal of the royal statistical society: series b (statistical methodology)
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 6.523
H-Index - 137
eISSN - 1467-9868
pISSN - 1369-7412
DOI - 10.1111/rssb.12357
Subject(s) - estimator , mathematics , statistical inference , statistics , inference , linear model , linear regression , asymptotic distribution , algorithm , computer science , artificial intelligence
Summary The paper considers statistical inference for the explained varianceβ T Σ β under the high dimensional linear model Y = Xβ + ε in the semisupervised setting, where β is the regression vector and Σ is the design covariance matrix. A calibrated estimator, which efficiently integrates both labelled and unlabelled data, is proposed. It is shown that the estimator achieves the minimax optimal rate of convergence in the general semisupervised framework. The optimality result characterizes how the unlabelled data contribute to the estimation accuracy. Moreover, the limiting distribution for the proposed estimator is established and the unlabelled data have also proved useful in reducing the length of the confidence interval for the explained variance. The method proposed is extended to semisupervised inference for the unweighted quadratic functional‖ β ‖ 2 2 . The inference results obtained are then applied to a range of high dimensional statistical problems, including signal detection and global testing, prediction accuracy evaluation and confidence ball construction. The numerical improvement of incorporating the unlabelled data is demonstrated through simulation studies and an analysis of estimating heritability for a yeast segregant data set with multiple traits.