English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

While current LLM chatbots like GPT-4V bridge the gap between humaninstructions and visual representations to enable text-image generations, theystill lack efficient alignment methods for high-fidelity performance onmultiple downstream tasks. In this paper, we propose \textbf{$M^{2}Chat$}, anovel unified multimodal LLM framework for generating interleaved text-imageconversation across various scenarios. Specifically, we propose an$M^{3}Adapter$ that efficiently integrates granular low-level visualinformation and high-level semantic features from multi-modality prompts. Uponthe well-aligned fused feature, $M^{3}Adapter$ tailors a learnable gatingstrategy to balance the model creativity and consistency across various tasksadaptively. Moreover, to further enhance the effectiveness of $M^{3}Adapter$while preserving the coherence of semantic context comprehension, we introducea two-stage $M^{3}FT$ fine-tuning strategy. This strategy optimizes disjointgroups of parameters for image-text alignment and visual-instructionrespectively. Extensive experiments demonstrate our $M^{2}Chat$ surpassesstate-of-the-art counterparts across diverse benchmarks, showcasing its prowessin interleaving generation, storytelling, and multimodal dialogue systems. Thedemo and code are available at\red{https://mattie-e.github.io/M2Chat.github.io}.

M$^{2}$Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation