English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs)inference: explicit controllable text generation. Multi-modal LLMs empowermulti-modality understanding with the capability of semantic generation yetbring less explainability and heavier reliance on prompt contents due to theirautoregressive generative nature. While manipulating prompt formats couldimprove outputs, designing specific and precise prompts per task can bechallenging and ineffective. To tackle this issue, we introduce a novelinference method, Prompt Highlighter, which enables users to highlight specificprompt spans to interactively control the focus during generation. Motivated bythe classifier-free diffusion guidance, we form regular and unconditionalcontext pairs based on highlighted tokens, demonstrating that theautoregressive generation in models can be guided in a classifier-free way.Notably, we find that, during inference, guiding the models with highlightedtokens through the attention weights leads to more desired outputs. Ourapproach is compatible with current LLMs and VLMs, achieving impressivecustomized generation results without training. Experiments confirm itseffectiveness in focusing on input contexts and generating reliable content.Without tuning on LLaVA-v1.5, our method secured 69.5 in the MMBench test and1552.5 in MME-perception. The code is available at:https://github.com/dvlab-research/Prompt-Highlighter/

Prompt Highlighter: Interactive Control for Multi-Modal LLMs