English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Transformer-based speech self-supervised learning (SSL) models, such asHuBERT, show surprising performance in various speech processing tasks.However, huge number of parameters in speech SSL models necessitate thecompression to a more compact model for wider usage in academia or smallcompanies. In this study, we suggest to reuse attention maps across theTransformer layers, so as to remove key and query parameters while retainingthe number of layers. Furthermore, we propose a novel masking distillationstrategy to improve the student model's speech representation quality. Weextend the distillation loss to utilize both masked and unmasked speech framesto fully leverage the teacher model's high-quality representation. Ouruniversal compression strategy yields the student model that achieves phonemeerror rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERBbenchmark.

Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation