Source enumeration via RMT estimator with adaptive decision criterion based on linear shrinkage estimation of noise eigenvalues using relatively few samples

Estimating the number of signals embedded in noise is a fundamental problem in array signal processing. The classic random matrix theory (RMT) estimator based on RMT does not consider the bias term of the expectation of the eigenvalue being tested, and thus, its detection performance will be affected by this bias term, and it also suffers from noise uncertainty in its decision threshold. In order to overcome these problems, more accurate expressions for the distribution and the bias term of the expectation of the eigenvalue being tested are firstly derived by utilising the linear shrinkage (LS) technique. Then, a novel LS‐RMT estimator is proposed by incorporating the bias term of the expectation of the eigenvalue being tested into the decision criterion of the RMT estimator. Moreover, both the increased under‐estimation probability and the increased over‐estimation probability of the RMT estimator incurred by this bias term are derived according to the sign of this bias term. Based on these results, a novel LS‐RMT estimator with adaptive decision criterion (termed as ‘LS‐RMT‐ADC estimator’) is proposed by utilising the LS‐RMT estimator, and the proposed LS‐RMT‐ADC estimator can adaptively select its decision threshold and determine the noise variance in the selected decision threshold. Therefore, it can overcome both the higher under‐estimation probability and the higher over‐estimation probability of the RMT estimator, and can also avoid the noise uncertainty in the decision threshold of the RMT estimator. Finally, simulation results are presented to show that the LS‐RMT‐ADC estimator outperforms the existing estimators.