English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Self-supervised speech representation learning has shown promising results invarious speech processing tasks. However, the pre-trained models, e.g., HuBERT,are storage-intensive Transformers, limiting their scope of applications underlow-resource settings. To this end, we propose LightHuBERT, a once-for-allTransformer compression framework, to find the desired architecturesautomatically by pruning structured parameters. More precisely, we create aTransformer-based supernet that is nested with thousands of weight-sharingsubnets and design a two-stage distillation strategy to leverage thecontextualized latent representations from HuBERT. Experiments on automaticspeech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERTenables over $10^9$ architectures concerning the embedding dimension, attentiondimension, head number, feed-forward network ratio, and network depth.LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks withthe HuBERT size, achieves comparable performance to the teacher model in mosttasks with a reduction of 29% parameters, and obtains a $3.5\times$ compressionratio in three SUPERB tasks, e.g., automatic speaker verification, keywordspotting, and intent classification, with a slight accuracy loss. The code andpre-trained models are available athttps://github.com/mechanicalsea/lighthubert.

LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT