English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Language models trained on large-scale corpus often generate content that isharmful, toxic, or contrary to human preferences, making their alignment withhuman values a critical concern. Reinforcement learning from human feedback(RLHF) with algorithms like PPO is a prevalent approach for alignment but isoften complex, unstable, and resource-intensive. Recently, ranking-basedalignment methods have emerged, offering stability and effectiveness byreplacing the RL framework with supervised fine-tuning, but they are costly dueto the need for annotated data. Considering that existing large language models(LLMs) like ChatGPT are already relatively well-aligned and cost-friendly,researchers have begun to align the language model with human preference fromAI feedback. The common practices, which unidirectionally distill theinstruction-following responses from LLMs, are constrained by their bottleneck.Thus we introduce CycleAlign to distill alignment capabilities fromparameter-invisible LLMs (black-box) to a parameter-visible model (white-box)in an iterative manner. With in-context learning (ICL) as the core of thecycle, the black-box models are able to rank the model-generated responsesguided by human-craft instruction and demonstrations about their preferences.During iterative interaction, the white-box models also have a judgment aboutresponses generated by them. Consequently, the agreement ranking could beviewed as a pseudo label to dynamically update the in-context demonstrationsand improve the preference ranking ability of black-box models. Throughmultiple interactions, the CycleAlign framework could align the white-box modelwith the black-box model effectively in a low-resource way. Empirical resultsillustrate that the model fine-tuned by CycleAlign remarkably exceeds existingmethods, and achieves the state-of-the-art performance in alignment with humanvalue.

CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment