English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision-Language models like CLIP have been widely adopted for various tasksdue to their impressive zero-shot capabilities. However, CLIP is not suitablefor extracting 3D geometric features as it was trained on only images and textby natural language supervision. We work on addressing this limitation andpropose a new framework termed CG3D (CLIP Goes 3D) where a 3D encoder islearned to exhibit zero-shot capabilities. CG3D is trained using triplets ofpointclouds, corresponding rendered 2D images, and texts using natural languagesupervision. To align the features in a multimodal embedding space, we utilizecontrastive loss on 3D features obtained from the 3D encoder, as well as visualand text features extracted from CLIP. We note that the natural images used totrain CLIP and the rendered 2D images in CG3D have a distribution shift.Attempting to train the visual and text encoder to account for this shiftresults in catastrophic forgetting and a notable decrease in performance. Tosolve this, we employ prompt tuning and introduce trainable parameters in theinput space to shift CLIP towards the 3D pre-training dataset utilized in CG3D.We extensively test our pre-trained CG3D framework and demonstrate itsimpressive capabilities in zero-shot, open scene understanding, and retrievaltasks. Further, it also serves as strong starting weights for fine-tuning indownstream 3D recognition tasks.

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition