Maximum regularized likelihood estimators: A general prediction theory and applications

Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high‐dimensional statistics. In this paper, we derive guarantees for MRLEs in the Kullback–Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization and that the regularization is definite and positive homogenous. The results thus apply to a very large variety of models and estimators, such as tensor regression and graphical models with convex and non‐convex regularized methods. A main conclusion is that MRLEs are broadly consistent in prediction—regardless of whether restricted eigenvalues or similar conditions hold. Copyright © 2018 John Wiley & Sons, Ltd.