Design of Incremental Echo State Network Using Leave-One-Out Cross-Validation

In most echo state networks (ESNs), the training error typically decreases as the network size increases, and thus the overfitting issue is widely existed. To solve this problem, an incremental ESN (IESN) is proposed by incorporating the leave-one-out cross-validation (LOO-CV) and the regularization method. First, the LOO-CV error is used to automatically identify the network architecture such that the overfitting problem is avoided to some extent. Second, the regularization technique is used to solve the ill-posed problem, and thus the IESN owns good robustness property. Third, the output weights are incrementally calculated by the fast SVD updating algorithm to reduce the ESN training time. Moreover, the stability and convergence of IESN are discussed to ensure its successful application. Simulation results demonstrate that the proposed IESN requires fewer reservoir nodes yet obtains much better performance than other existing ESNs.