English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, self-attention models such as Transformers have given competitiveresults compared to recurrent neural network systems in speech recognition. Thekey factor for the outstanding performance of self-attention models is theirability to capture temporal relationships without being limited by the distancebetween two related events. However, we note that the range of the learnedcontext progressively increases from the lower to upper self-attention layers,whilst acoustic events often happen within short time spans in a left-to-rightorder. This leads to a question: for speech recognition, is a global view ofthe entire sequence still important for the upper self-attention layers in theencoder of Transformers? To investigate this, we replace these self-attentionlayers with feed forward layers. In our speech recognition experiments (WallStreet Journal and Switchboard), we indeed observe an interesting result:replacing the upper self-attention layers in the encoder with feed forwardlayers leads to no performance drop, and even minor gains. Our experimentsoffer insights to how self-attention layers process the speech signal, leadingto the conclusion that the lower self-attention layers of the encoder encode asufficiently wide range of inputs, hence learning further contextualinformation in the upper layers is unnecessary.

When Can Self-Attention Be Replaced by Feed Forward Layers?