English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, Large Language Models (LLMs) and Multimodal Large Language Models(MLLMs) have shown promise in instruction following and 2D image understanding.While these models are powerful, they have not yet been developed to comprehendthe more challenging 3D physical scenes, especially when it comes to the sparseoutdoor LiDAR data. In this paper, we introduce LiDAR-LLM, which takes rawLiDAR data as input and harnesses the remarkable reasoning capabilities of LLMsto gain a comprehensive understanding of outdoor 3D scenes. The central insightof our LiDAR-LLM is the reformulation of 3D outdoor scene cognition as alanguage modeling problem, encompassing tasks such as 3D captioning, 3Dgrounding, 3D question answering, etc. Specifically, due to the scarcity of 3DLiDAR-text pairing data, we introduce a three-stage training strategy andgenerate relevant datasets, progressively aligning the 3D modality with thelanguage embedding space of LLM. Furthermore, we design a View-AwareTransformer (VAT) to connect the 3D encoder with the LLM, which effectivelybridges the modality gap and enhances the LLM's spatial orientationcomprehension of visual features. Our experiments show that LiDAR-LLM possessesfavorable capabilities to comprehend various instructions regarding 3D scenesand engage in complex spatial reasoning. LiDAR-LLM attains a 40.9 BLEU-1 on the3D captioning task and achieves a 63.1\% classification accuracy and a 14.3\%BEV mIoU on the 3D grounding task. Web page:https://sites.google.com/view/lidar-llm

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding