English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Continued pre-training (CP) offers multiple advantages, like target domainadaptation and the potential to exploit the continuous stream of unlabeled dataavailable online. However, continued pre-training on out-of-domaindistributions often leads to catastrophic forgetting of previously acquiredknowledge, leading to sub-optimal ASR performance. This paper presents FusDom,a simple and novel methodology for SSL-based continued pre-training. FusDomlearns speech representations that are robust and adaptive yet not forgetful ofconcepts seen in the past. Instead of solving the SSL pre-text task on theoutput representations of a single model, FusDom leverages two identicalpre-trained SSL models, a teacher and a student, with a modified pre-traininghead to solve the CP SSL pre-text task. This head employs a cross-attentionmechanism between the representations of both models while only the studentreceives gradient updates and the teacher does not. Finally, the student isfine-tuned for ASR. In practice, FusDom outperforms all our baselines acrosssettings significantly, with WER improvements in the range of 0.2 WER - 7.3 WERin the target domain while retaining the performance in the earlier domain.

FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learning