English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Hallucination, posed as a pervasive challenge of multi-modal large languagemodels (MLLMs), has significantly impeded their real-world usage that demandsprecise judgment. Existing methods mitigate this issue with either trainingwith specific designed data or inferencing with external knowledge from othersources, incurring inevitable additional costs. In this paper, we presentOPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and aRetrospection-Allocation strategy, serving as a nearly free lunch to alleviatethe hallucination issue without additional data, knowledge, or training. Ourapproach begins with an interesting observation that, most hallucinations areclosely tied to the knowledge aggregation patterns manifested in theself-attention matrix, i.e., MLLMs tend to generate new tokens by focusing on afew summary tokens, but not all the previous tokens. Such partial over-trustinclination results in the neglecting of image tokens and describes the imagecontent with hallucination. Statistically, we observe an 80%$\sim$95%co-currency rate between hallucination contents and such knowledge aggregationpatterns. Based on the observation, OPERA introduces a penalty term on themodel logits during the beam-search decoding to mitigate the over-trust issue,along with a rollback strategy that retrospects the presence of summary tokensin the previously generated tokens, and re-allocate the token selection ifnecessary. With extensive experiments, OPERA shows significanthallucination-mitigating performance on different MLLMs and metrics, provingits effectiveness and generality. Our code is available at:https://github.com/shikiw/OPERA.

OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation