English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Although remarkable progress has been achieved in preventing large languagemodel (LLM) hallucinations using instruction tuning and retrieval augmentation,it remains challenging to measure the reliability of LLMs using human-craftedevaluation data which is not available for many tasks and domains and couldsuffer from data leakage. Inspired by adversarial machine learning, this paperaims to develop a method of automatically generating evaluation data byappropriately modifying existing data on which LLMs behave faithfully.Specifically, this paper presents AutoDebug, an LLM-based framework to useprompting chaining to generate transferable adversarial attacks in the form ofquestion-answering examples. We seek to understand the extent to which theseexamples trigger the hallucination behaviors of LLMs.  We implement AutoDebug using ChatGPT and evaluate the resulting two variantsof a popular open-domain question-answering dataset, Natural Questions (NQ), ona collection of open-source and proprietary LLMs under various promptingsettings. Our generated evaluation data is human-readable and, as we show,humans can answer these modified questions well. Nevertheless, we observepronounced accuracy drops across multiple LLMs including GPT-4. Ourexperimental results show that LLMs are likely to hallucinate in two categoriesof question-answering scenarios where (1) there are conflicts between knowledgegiven in the prompt and their parametric knowledge, or (2) the knowledgeexpressed in the prompt is complex. Finally, we find that the adversarialexamples generated by our method are transferable across all considered LLMs.The examples generated by a small model can be used to debug a much largermodel, making our approach cost-effective.

Automatic Hallucination Assessment for Aligned Large Language Models via Transferable Adversarial Attacks