English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Convolutional neural networks have been successful in solving many sociallyimportant and economically significant problems. This ability to learn complexhigh-dimensional functions hierarchically can be attributed to the use ofnonlinear activation functions. A key discovery that made training deepnetworks feasible was the adoption of the Rectified Linear Unit (ReLU)activation function to alleviate the vanishing gradient problem caused by usingsaturating activation functions. Since then, many improved variants of the ReLUactivation have been proposed. However, a majority of activation functions usedtoday are non-oscillatory and monotonically increasing due to their biologicalplausibility. This paper demonstrates that oscillatory activation functions canimprove gradient flow and reduce network size. Two theorems on limits ofnon-oscillatory activation functions are presented. A new oscillatoryactivation function called Growing Cosine Unit(GCU) defined as $C(z) = z\cos z$that outperforms Sigmoids, Swish, Mish and ReLU on a variety of architecturesand benchmarks is presented. The GCU activation has multiple zeros enablingsingle GCU neurons to have multiple hyperplanes in the decision boundary. Thisallows single GCU neurons to learn the XOR function without featureengineering. Experimental results indicate that replacing the activationfunction in the convolution layers with the GCU activation functionsignificantly improves performance on CIFAR-10, CIFAR-100 and Imagenette.

Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks