English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Summarizing book-length documents (>100K tokens) that exceed the contextwindow size of large language models (LLMs) requires first breaking the inputdocument into smaller chunks and then prompting an LLM to merge, update, andcompress chunk-level summaries. Despite the complexity and importance of thistask, it has yet to be meaningfully studied due to the challenges ofevaluation: existing book-length summarization datasets (e.g., BookSum) are inthe pretraining data of most public LLMs, and existing evaluation methodsstruggle to capture errors made by modern LLM summarizers. In this paper, wepresent the first study of the coherence of LLM-based book-length summarizersimplemented via two prompting workflows: (1) hierarchically merging chunk-levelsummaries, and (2) incrementally updating a running summary. We obtain 1193fine-grained human annotations on GPT-4 generated summaries of 100recently-published books and identify eight common types of coherence errorsmade by LLMs. Because human evaluation is expensive and time-consuming, wedevelop an automatic metric, BooookScore, that measures the proportion ofsentences in a summary that do not contain any of the identified error types.BooookScore has high agreement with human annotations and allows us tosystematically evaluate the impact of many other critical parameters (e.g.,chunk size, base LLM) while saving $15K and 500 hours in human evaluationcosts. We find that closed-source LLMs such as GPT-4 and Claude 2 producesummaries with higher BooookScore than the oft-repetitive ones generated byLLaMA 2. Incremental updating yields lower BooookScore but higher level ofdetail than hierarchical merging, a trade-off sometimes preferred by humanannotators. We release code and annotations after blind review to spur moreprincipled research on book-length summarization.

BooookScore: A systematic exploration of book-length summarization in the era of LLMs