English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The emergence of large language models (LLMs) has sparked significantinterest in extending their remarkable language capabilities to speech.However, modality alignment between speech and text still remains an openproblem. Current solutions can be categorized into two strategies. One is acascaded approach where outputs (tokens or states) of a separately trainedspeech recognition system are used as inputs for LLMs, which limits theirpotential in modeling alignment between speech and text. The other is anend-to-end approach that relies on speech instruction data, which is verydifficult to collect in large quantities. In this paper, we address theseissues and propose the BLSP approach that Bootstraps Language-SpeechPre-training via behavior alignment of continuation writing. We achieve this bylearning a lightweight modality adapter between a frozen speech encoder and anLLM, ensuring that the LLM exhibits the same generation behavior regardless ofthe modality of input: a speech segment or its transcript. The training processcan be divided into two steps. The first step prompts an LLM to generate textswith speech transcripts as prefixes, obtaining text continuations. In thesecond step, these continuations are used as supervised signals to train themodality adapter in an end-to-end manner. We demonstrate that thisstraightforward process can extend the capabilities of LLMs to speech, enablingspeech recognition, speech translation, spoken language understanding, andspeech conversation, even in zero-shot cross-lingual scenarios.

BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing