English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Self-supervised learning (SSL) has achieved notable success in many speechprocessing tasks, but the large model size and heavy computational cost hinderthe deployment. Knowledge distillation trains a small student model to mimicthe behavior of a large teacher model. However, the student architectureusually needs to be manually designed and will remain fixed during training,which requires prior knowledge and can lead to suboptimal performance. Inspiredby recent success of task-specific structured pruning, we propose DPHuBERT, anovel task-agnostic compression method for speech SSL based on jointdistillation and pruning. Experiments on SUPERB show that DPHuBERT outperformspure distillation methods in almost all tasks. Moreover, DPHuBERT requireslittle training time and performs well with limited training data, making itsuitable for resource-constrained applications. Our method can also be appliedto various speech SSL models. Our code and models will be publicly available.

DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models