English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

The field of image synthesis has made tremendous strides forward in the lastyears. Besides defining the desired output image with text-prompts, anintuitive approach is to additionally use spatial guidance in form of an image,such as a depth map. In state-of-the-art approaches, this guidance is realizedby a separate controlling model that controls a pre-trained image generationnetwork, such as a latent diffusion model. Understanding this process from acontrol system perspective shows that it forms a feedback-control system, wherethe control module receives a feedback signal from the generation process andsends a corrective signal back. When analysing existing systems, we observethat the feedback signals are timely sparse and have a small number of bits. Asa consequence, there can be long delays between newly generated features andthe respective corrective signals for these features. It is known that thisdelay is the most unwanted aspect of any control system. In this work, we takean existing controlling network (ControlNet) and change the communicationbetween the controlling network and the generation process to be ofhigh-frequency and with large-bandwidth. By doing so, we are able toconsiderably improve the quality of the generated images, as well as thefidelity of the control. Also, the controlling network needs noticeably fewerparameters and hence is about twice as fast during inference and training time.Another benefit of small-sized models is that they help to democratise ourfield and are likely easier to understand. We call our proposed networkControlNet-XS. When comparing with the state-of-the-art approaches, weoutperform them for pixel-level guidance, such as depth, canny-edges, andsemantic segmentation, and are on a par for loose keypoint-guidance of humanposes. All code and pre-trained models will be made publicly available.

ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems