English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

3D occupancy prediction is an emerging task that aims to estimate theoccupancy states and semantics of 3D scenes using multi-view images. However,image-based scene perception encounters significant challenges in achievingaccurate prediction due to the absence of geometric priors. In this paper, weaddress this issue by exploring cross-modal knowledge distillation in thistask, i.e., we leverage a stronger multi-modal model to guide the visual modelduring training. In practice, we observe that directly applying features orlogits alignment, proposed and widely used in bird's-eyeview (BEV) perception,does not yield satisfactory results. To overcome this problem, we introduceRadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.By employing differentiable volume rendering, we generate depth and semanticmaps in perspective views and propose two novel consistency criteria betweenthe rendered outputs of teacher and student models. Specifically, the depthconsistency loss aligns the termination distributions of the rendered rays,while the semantic consistency loss mimics the intra-segment similarity guidedby vision foundation models (VLMs). Experimental results on the nuScenesdataset demonstrate the effectiveness of our proposed method in improvingvarious 3D occupancy prediction approaches, e.g., our proposed methodologyenhances our baseline by 2.2% in the metric of mIoU and achieves 50% in Occ3Dbenchmark.

RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation