English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Self-supervised speech representation learning (speech SSL) has demonstratedthe benefit of scale in learning rich representations for Automatic SpeechRecognition (ASR) with limited paired data, such as wav2vec 2.0. We investigatethe existence of sparse subnetworks in pre-trained speech SSL models thatachieve even better low-resource ASR results. However, directly applying widelyadopted pruning methods such as the Lottery Ticket Hypothesis (LTH) issuboptimal in the computational cost needed. Moreover, we show that thediscovered subnetworks yield minimal performance gain compared to the originaldense network. We present Prune-Adjust-Re-Prune (PARP), which discovers andfinetunes subnetworks for much better performance, while only requiring asingle downstream ASR finetuning run. PARP is inspired by our surprisingobservation that subnetworks pruned for pre-training tasks need merely a slightadjustment to achieve a sizeable performance boost in downstream ASR tasks.Extensive experiments on low-resource ASR verify (1) sparse subnetworks existin mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computationaladvantage and performance gain of PARP over baseline pruning methods. Inparticular, on the 10min Librispeech split without LM decoding, PARP discoverssubnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease comparedto the full model. We further demonstrate the effectiveness of PARP via:cross-lingual pruning without any phone recognition degradation, the discoveryof a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, andits applicability to pre-trained BERT/XLNet for natural language tasks.

PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition