A statistical test for mixture detection with application to component identification in multidimensional biomolecular NMR studies

We introduce a statistical hypothesis test for detecting mixtures in a nonlinear regression model with mean regression function defined by a weighted sum of two multidimensional unimodal functions, where each unimodal function in the summation representing a component . Two regression components are mixed when the distance between their centres is small or the proportion of their contribution to the mean regression function is close to zero or one. Two challenges in model estimation under the null hypothesis of one regression component are that the proportion parameter describing the weighed contribution of each component lies on the boundary of the parameter space and that the model parameters are nonidentifiable. Therefore, the parameter estimators derived from standard nonlinear estimation approaches are inconsistent and unstable. To overcome these challenges, we study a penalized regression test statistic with a relatively simple quadratic approximation which can be used to simulate the quantiles of the test statistic under the null hypothesis. One leading application of the mixture testing procedure is the detection of mixed or overlapped components in multidimensional data generated from nuclear magnetic resonance (NMR) experiments for protein structure determination. It is important to de‐mix the components since each regression component provides specific information about the structure of the protein. In certain cases, the lack of a small number of essential components can lead to a significant deviation in the predicted structure. The Canadian Journal of Statistics 42: 36–60; 2014 © 2013 Statistical Society of Canada