English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Diffusion models (DMs) have become the new trend of generative models andhave demonstrated a powerful ability of conditional synthesis. Among those,text-to-image diffusion models pre-trained on large-scale image-text pairs arehighly controllable by customizable prompts. Unlike the unconditionalgenerative models that focus on low-level attributes and details, text-to-imagediffusion models contain more high-level knowledge thanks to thevision-language pre-training. In this paper, we propose VPD (Visual Perceptionwith a pre-trained Diffusion model), a new framework that exploits the semanticinformation of a pre-trained text-to-image diffusion model in visual perceptiontasks. Instead of using the pre-trained denoising autoencoder in adiffusion-based pipeline, we simply use it as a backbone and aim to study howto take full advantage of the learned knowledge. Specifically, we prompt thedenoising decoder with proper textual inputs and refine the text features withan adapter, leading to a better alignment to the pre-trained stage and makingthe visual contents interact with the text prompts. We also propose to utilizethe cross-attention maps between the visual features and the text features toprovide explicit guidance. Compared with other pre-training methods, we showthat vision-language pre-trained diffusion models can be faster adapted todownstream visual perception tasks using the proposed VPD. Extensiveexperiments on semantic segmentation, referring image segmentation and depthestimation demonstrates the effectiveness of our method. Notably, VPD attains0.254 RMSE on NYUv2 depth estimation and 73.3% oIoU on RefCOCO-val referringimage segmentation, establishing new records on these two benchmarks. Code isavailable at https://github.com/wl-zhao/VPD

Unleashing Text-to-Image Diffusion Models for Visual Perception