English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

It is hard to train Recurrent Neural Network (RNN) with stable convergenceand avoid gradient vanishing and exploding problems, as the weights in therecurrent unit are repeated from iteration to iteration. Moreover, RNN issensitive to the initialization of weights and bias, which brings difficultiesin training. The Alternating Direction Method of Multipliers (ADMM) has becomea promising algorithm to train neural networks beyond traditional stochasticgradient algorithms with the gradient-free features and immunity tounsatisfactory conditions. However, ADMM could not be applied to train RNNdirectly since the state in the recurrent unit is repetitively updated overtimesteps. Therefore, this work builds a new framework named ADMMiRNN upon theunfolded form of RNN to address the above challenges simultaneously. We alsoprovide novel update rules and theoretical convergence analysis. We explicitlyspecify essential update rules in the iterations of ADMMiRNN with constructedapproximation techniques and solutions to each sub-problem instead of vanillaADMM. Numerical experiments are conducted on MNIST, IMDb, and textclassification tasks. ADMMiRNN achieves convergent results and outperforms thecompared baselines. Furthermore, ADMMiRNN trains RNN more stably withoutgradient vanishing or exploding than stochastic gradient algorithms. We alsoprovide a distributed paralleled algorithm regarding ADMMiRNN, namedP-ADMMiRNN, including Synchronous Parallel ADMMiRNN (SP-ADMMiRNN) andAsynchronous Parallel ADMMiRNN (AP-ADMMiRNN), which is the first to train RNNwith ADMM in an asynchronous parallel manner. The source code is publiclyavailable.

P-ADMMiRNN: Training RNN with Stable Convergence via An Efficient and Paralleled ADMM Approach