English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Language models (LMs) can generate hallucinations and incoherent outputs,which highlights their weak context dependency. Cache-LMs, which augment LMswith a memory of recent history, can increase context dependency and have shownremarkable performance in diverse language generation tasks. However, we findthat even with training, the performance gain stemming from the cache componentof current cache-LMs is suboptimal due to the misalignment between the currenthidden states and those stored in the memory. In this work, we presentHistAlign, a new training approach to ensure good cache alignment such that themodel receives useful signals from the history. We first prove our concept on asimple and synthetic task where the memory is essential for correctpredictions, and we show that the cache component of HistAlign is betteraligned and improves overall performance. Next, we evaluate HistAlign ondiverse downstream language generation tasks, including prompt continuation,abstractive summarization, and data-to-text. We demonstrate that HistAlignimproves text coherence and faithfulness in open-ended and conditionalgeneration settings respectively. HistAlign is also generalizable acrossdifferent model families, showcasing its strength in improving contextdependency of LMs in diverse scenarios. Our code is publicly available athttps://github.com/meetdavidwan/histalign

HistAlign: Improving Context Dependency in Language Generation by Aligning with History