Open AccessRAPIDO: a rejuvenating adaptive PID‐type optimiser for deep neural networks
Author(s) -
Kim S.,
Park D.J.,
Chang D.E.
Publication year - 2019
Publication title -
electronics letters
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.375
H-Index - 146
eISSN - 1350-911X
pISSN - 0013-5194
DOI - 10.1049/el.2019.1593
Subject(s) - sigmoid function , mnist database , hyperbolic function , pid controller , artificial neural network , activation function , gradient descent , computer science , artificial intelligence , control theory (sociology) , quadratic equation , deep learning , softmax function , deep neural networks , algorithm , mathematics , engineering , control engineering , control (management) , temperature control , mathematical analysis , geometry
The authors present a novel gradient descent algorithm called RAPIDO for deep learning. It adapts over time and performs optimisation using current, past and future information similar to the PID controller. The proposed method is suited for optimising deep neural networks that consist of activation functions such as sigmoid, hyperbolic tangent and ReLU functions because it can adapt appropriately to sudden changes in gradients. They experimentally study the authors' method and show the performance results by comparing with other methods on the quadratic objective function and the MNIST classification task. The proposed method shows better performance than the other methods.