English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

The transformer is a fundamental building block in deep learning, and theattention mechanism is the transformer's core component. Self-supervised speechrepresentation learning (SSRL) represents a popular use-case for thetransformer architecture. Due to transformers' acausal behavior, the use oftransformers for SSRL has been predominantly focused on acausal applications.However, several media processing problems, such as speech processing, requirereal-time solutions. In this paper, we present an implementation of theattention module that enables training of SSRL architectures with low computeand memory requirements, while allowing real-time inference with low and fixedlatency. The attention module proposed in this paper includes two components,streaming attention (SA) and low-latency streaming attention (LLSA). The SArepresents our proposal for an efficient streaming SSRL implementation, whilethe LLSA solves the latency build-up problem of other streaming attentionarchitectures, such as the masked acausal attention (MAA), guaranteeing alatency equal to one layer even when multiple layers are stacked. We present acomparative analysis between the vanilla attention, which we will refer here asacausal attention (AA), the SA, and the LLSA, by training a streaming SSRL withautomatic speech recognition as downstream task. When training onlibrispeech-clean-100 and testing on librispeech-test-clean, our low-latencyattention module has a word error rate (WER) of 5.84%, which represents asignificant improvement over the MAA (WER = 13.82%). Our implementation alsoreduces the inference latency from 1.92 to 0.16 seconds. The proposedlow-latency module preserves many of the benefits of conventional acausaltransformers, but also enables latency characteristics that make it applicableto real-time streaming applications.

A low latency attention module for streaming self-supervised speech representation learning