English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Activation functions play a pivotal role in determining the training dynamicsand neural network performance. The widely adopted activation function ReLUdespite being simple and effective has few disadvantages including the DyingReLU problem. In order to tackle such problems, we propose a novel activationfunction called Serf which is self-regularized and nonmonotonic in nature. LikeMish, Serf also belongs to the Swish family of functions. Based on severalexperiments on computer vision (image classification and object detection) andnatural language processing (machine translation, sentiment classification andmultimodal entailment) tasks with different state-of-the-art architectures, itis observed that Serf vastly outperforms ReLU (baseline) and other activationfunctions including both Swish and Mish, with a markedly bigger margin ondeeper architectures. Ablation studies further demonstrate that Serf basedarchitectures perform better than those of Swish and Mish in varying scenarios,validating the effectiveness and compatibility of Serf with varying depth,complexity, optimizers, learning rates, batch sizes, initializers and dropoutrates. Finally, we investigate the mathematical relation between Swish andSerf, thereby showing the impact of preconditioner function ingrained in thefirst derivative of Serf which provides a regularization effect makinggradients smoother and optimization faster.

SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function