English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Text-to-image diffusion models generate impressive and realistic images, butdo they learn to represent the 3D world from only 2D supervision? Wedemonstrate that yes, certain 3D scene representations are encoded in the textembedding space of models like Stable Diffusion. Our approach, Viewpoint NeuralTextual Inversion (ViewNeTI), is to discover 3D view tokens; these tokenscontrol the 3D viewpoint - the rendering pose in a scene - of generated images.Specifically, we train a small neural mapper to take continuous cameraviewpoint parameters and predict a view token (a word embedding). This tokenconditions diffusion generation via cross-attention to produce images with thedesired camera viewpoint. Using ViewNeTI as an evaluation tool, we report twofindings: first, the text latent space has a continuous view-control manifoldfor particular 3D scenes; second, we find evidence for a generalizedview-control manifold for all scenes. We conclude that since the view tokencontrols the 3D `rendering' viewpoint, there is likely a scene representationembedded in frozen 2D diffusion models. Finally, we exploit the 3D scenerepresentations for 3D vision tasks, namely, view-controlled text-to-imagegeneration, and novel view synthesis from a single image, where our approachsets state-of-the-art for LPIPS. Code available athttps://github.com/jmhb0/view_neti

Viewpoint Textual Inversion: Discovering Scene Representations and 3D View Control in 2D Diffusion Models