English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent generative models have demonstrated impressive capabilities ingenerating realistic and visually pleasing images grounded on textual prompts.Nevertheless, a significant challenge remains in applying these models for themore intricate task of story visualization. Since it requires resolvingpronouns (he, she, they) in the frame descriptions, i.e., anaphora resolution,and ensuring consistent characters and background synthesis across frames. Yet,the emerging Large Language Model (LLM) showcases robust reasoning abilities tonavigate through ambiguous references and process extensive sequences.Therefore, we introduce \textbf{StoryGPT-V}, which leverages the merits of thelatent diffusion (LDM) and LLM to produce images with consistent andhigh-quality characters grounded on given story descriptions. First, we train acharacter-aware LDM, which takes character-augmented semantic embedding asinput and includes the supervision of the cross-attention map using charactersegmentation masks, aiming to enhance character generation accuracy andfaithfulness. In the second stage, we enable an alignment between the output ofLLM and the character-augmented embedding residing in the input space of thefirst-stage model. This harnesses the reasoning ability of LLM to addressambiguous references and the comprehension capability to memorize the context.We conduct comprehensive experiments on two visual story visualizationbenchmarks. Our model reports superior quantitative results and consistentlygenerates accurate characters of remarkable quality with low memoryconsumption. Our code will be made publicly available.

StoryGPT-V: Large Language Models as Consistent Story Visualizers