English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we describe the use of recurrent neural networks to capturesequential information from the self-attention representations to improve theTransformers. Although self-attention mechanism provides a means to exploitlong context, the sequential information, i.e. the arrangement of tokens, isnot explicitly captured. We propose to cascade the recurrent neural networks tothe Transformers, which referred to as the TransfoRNN model, to capture thesequential information. We found that the TransfoRNN models which consists ofonly shallow Transformers stack is suffice to give comparable, if not better,performance than a deeper Transformer model. Evaluated on the Penn Treebank andWikiText-2 corpora, the proposed TransfoRNN model has shown lower modelperplexities with fewer number of model parameters. On the Penn Treebankcorpus, the model perplexities were reduced up to 5.5% with the model sizereduced up to 10.5%. On the WikiText-2 corpus, the model perplexity was reducedup to 2.2% with a 27.7% smaller model. Also, the TransfoRNN model was appliedon the LibriSpeech speech recognition task and has shown comparable resultswith the Transformer models.

TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling