English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Pre-trained speech Transformers have facilitated great success across variousspeech processing tasks. However, fine-tuning these encoders for downstreamtasks require sufficiently large training data to converge or to achievestate-of-the-art. In text domain this has been partly attributed tosub-optimality of the representation space in pre-trained Transformers. In thiswork, we take a sober look into pre-trained speech encoders and rewire theirrepresentation space without requiring any task-specific labels. Our methodutilises neutrally synthesised version of audio inputs along with frame maskingto construct positive pairs for contrastive self-supervised learning. When usedfor augmenting the wav2vec 2 encoder, we observe consistent improvement ofisotropy in the representation space. Our experiments on 6 speech processingtasks, exhibit a significant convergence speedup during task fine-tuning aswell as consistent task improvement, specially in low-resource settings.

Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing