English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

This paper proposes a textless training method for many-to-many multilingualspeech-to-speech translation that can also benefit the transfer of pre-trainedknowledge to text-based systems, text-to-speech synthesis and text-to-speechtranslation. To this end, we represent multilingual speech with speech unitsthat are the discretized representations of speech features derived from aself-supervised speech model. By treating the speech units as pseudo-text, wecan focus on the linguistic content of the speech, which can be easilyassociated with both speech and text modalities at the phonetic levelinformation. By setting both the inputs and outputs of our learning problem asspeech units, we propose to train an encoder-decoder model in a many-to-manyspoken language translation setting, namely Unit-to-Unit Translation (UTUT).Specifically, the encoder is conditioned on the source language token tocorrectly understand the input spoken language, while the decoder isconditioned on the target language token to generate the translated speech inthe target language. Therefore, during the training, the model can build theknowledge of how languages are comprehended and how to relate them to differentlanguages. Since speech units can be easily associated from both audio and textby quantization and phonemization respectively, the trained model can easilytransferred to text-related tasks, even if it is trained in a textless manner.We demonstrate that the proposed UTUT model can be effectively utilized notonly for Speech-to-Speech Translation (S2ST) but also for multilingualText-to-Speech Synthesis (T2S) and Text-to-Speech Translation (T2ST), requiringonly minimal fine-tuning steps on text inputs. By conducting comprehensiveexperiments encompassing various languages, we validate the efficacy of theproposed method across diverse multilingual tasks.

Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation