English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Activation functions play a crucial role in neural networks because they arethe nonlinearities which have been attributed to the success story of deeplearning. One of the currently most popular activation functions is ReLU, butseveral competitors have recently been proposed or 'discovered', includingLReLU functions and swish. While most works compare newly proposed activationfunctions on few tasks (usually from image classification) and against fewcompetitors (usually ReLU), we perform the first large-scale comparison of 21activation functions across eight different NLP tasks. We find that a largelyunknown activation function performs most stably across all tasks, theso-called penalized tanh function. We also show that it can successfullyreplace the sigmoid and tanh gates in LSTM cells, leading to a 2 percentagepoint (pp) improvement over the standard choices on a challenging NLP task.

Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks