English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent studies have highlighted a phenomenon in large language models (LLMs)known as "the reversal curse," in which the order of knowledge entities in thetraining data biases the models' comprehension. For example, if a model istrained on sentences where entity A consistently appears before entity B, itcan respond to queries about A by providing B as the answer. However, it mayencounter confusion when presented with questions concerning B. We contend thatthe reversal curse is partially a result of specific model training objectives,particularly evident in the prevalent use of the next-token prediction withinmost causal language models. For the next-token prediction, models solely focuson a token's preceding context, resulting in a restricted comprehension of theinput. In contrast, we illustrate that the GLM, trained using theautoregressive blank infilling objective where tokens to be predicted haveaccess to the entire context, exhibits better resilience against the reversalcurse. We propose a novel training method, BIdirectional Casual languagemodeling Optimization (BICO), designed to mitigate the reversal curse whenfine-tuning pretrained causal language models on new data. BICO modifies thecausal attention mechanism to function bidirectionally and employs a maskdenoising optimization. In the task designed to assess the reversal curse, ourapproach improves Llama's accuracy from the original 0% to around 70%. We hopethat more attention can be focused on exploring and addressing these inherentweaknesses of the current LLMs, in order to achieve a higher level ofintelligence.

Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse