English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

While large language models (LLMs) have proven to be effective on a largevariety of tasks, they are also known to hallucinate information. To measurewhether an LLM prefers factually consistent continuations of its input, wepropose a new benchmark called FIB(Factual Inconsistency Benchmark) thatfocuses on the task of summarization. Specifically, our benchmark involvescomparing the scores an LLM assigns to a factually consistent versus afactually inconsistent summary for an input news article. For factuallyconsistent summaries, we use human-written reference summaries that we manuallyverify as factually consistent. To generate summaries that are factuallyinconsistent, we generate summaries from a suite of summarization models thatwe have manually annotated as factually inconsistent. A model's factualconsistency is then measured according to its accuracy, i.e.\ the proportion ofdocuments where it assigns a higher score to the factually consistent summary.To validate the usefulness of FIB, we evaluate 23 large language models rangingfrom 1B to 176B parameters from six different model families including BLOOMand OPT. We find that existing LLMs generally assign a higher score tofactually consistent summaries than to factually inconsistent summaries.However, if the factually inconsistent summaries occur verbatim in thedocument, then LLMs assign a higher score to these factually inconsistentsummaries than factually consistent summaries. We validate design choices inour benchmark including the scoring method and source of distractor summaries.Our code and benchmark data can be found at https://github.com/r-three/fib.

Evaluating the Factual Consistency of Large Language Models Through News Summarization