Open AccessTanhExp: A smooth activation function with high convergence speed for lightweight neural networks
Author(s) -
Liu Xinyu,
Di Xiaoguang
Publication year - 2021
Publication title -
iet computer vision
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.38
H-Index - 37
eISSN - 1751-9640
pISSN - 1751-9632
DOI - 10.1049/cvi2.12020
Subject(s) - activation function , computer science , artificial neural network , robustness (evolution) , hyperbolic function , convergence (economics) , artificial intelligence , function (biology) , task (project management) , image (mathematics) , algorithm , function approximation , pattern recognition (psychology) , mathematics , mathematical analysis , biochemistry , chemistry , evolutionary biology , biology , economics , gene , economic growth , management
Lightweight or mobile neural networks used for real‐time computer vision tasks contain fewer parameters than normal networks, which lead to a constrained performance. Herein, a novel activation function named as Tanh Exponential Activation Function (TanhExp) is proposed which can improve the performance for these networks on image classification task significantly. The definition of TanhExp is f ( x ) = x tanh( e x ). The simplicity, efficiency, and robustness of TanhExp on various datasets and network models is demonstrated and TanhExp outperforms its counterparts in both convergence speed and accuracy. Its behaviour also remains stable even with noise added and dataset altered. It is shown that without increasing the size of the network, the capacity of lightweight neural networks can be enhanced by TanhExp with only a few training epochs and no extra parameters added.