English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We introduce a new approach for speech pre-training named SPIRAL which worksby learning denoising representation of perturbed data in a teacher-studentframework. Specifically, given a speech utterance, we first feed the utteranceto a teacher network to obtain corresponding representation. Then the sameutterance is perturbed and fed to a student network. The student network istrained to output representation resembling that of the teacher. At the sametime, the teacher network is updated as moving average of student's weightsover training steps. In order to prevent representation collapse, we apply anin-utterance contrastive loss as pre-training objective and impose positionrandomization on the input to the teacher. SPIRAL achieves competitive orbetter results compared to state-of-the-art speech pre-training method wav2vec2.0, with significant reduction of training cost (80% for BASE model, 65% forLARGE model). Furthermore, we address the problem of noise-robustness that iscritical to real-world speech applications. We propose multi-conditionpre-training by perturbing the student's input with various types of additivenoise. We demonstrate that multi-condition pre-trained SPIRAL models are morerobust to noisy speech (9.0% - 13.3% relative word error rate reduction on realnoisy test data), compared to applying multi-condition training solely in thefine-tuning stage. Source code is available athttps://github.com/huawei-noah/Speech-Backbones/tree/main/SPIRAL.

SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training