English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Diffusion models have emerged as the de facto paradigm for video generation.However, their reliance on web-scale data of varied quality often yieldsresults that are visually unappealing and misaligned with the textual prompts.To tackle this problem, we propose InstructVideo to instruct text-to-videodiffusion models with human feedback by reward fine-tuning. InstructVideo hastwo key ingredients: 1) To ameliorate the cost of reward fine-tuning induced bygenerating through the full DDIM sampling chain, we recast reward fine-tuningas editing. By leveraging the diffusion process to corrupt a sampled video,InstructVideo requires only partial inference of the DDIM sampling chain,reducing fine-tuning cost while improving fine-tuning efficiency. 2) Tomitigate the absence of a dedicated video reward model for human preferences,we repurpose established image reward models, e.g., HPSv2. To this end, wepropose Segmental Video Reward, a mechanism to provide reward signals based onsegmental sparse sampling, and Temporally Attenuated Reward, a method thatmitigates temporal modeling degradation during fine-tuning. Extensiveexperiments, both qualitative and quantitative, validate the practicality andefficacy of using image reward models in InstructVideo, significantly enhancingthe visual quality of generated videos without compromising generalizationcapabilities. Code and models will be made publicly available.

InstructVideo: Instructing Video Diffusion Models with Human Feedback