English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Recent efforts to use natural language for interpretable driving focus mainlyon planning, neglecting perception tasks. In this paper, we address this gap byintroducing ROLISP (Risk Object Localization and Intention and SuggestionPrediction), which towards interpretable risk object detection and suggestionfor ego car motions. Accurate ROLISP implementation requires extensivereasoning to identify critical traffic objects and infer their intentions,prompting us to explore the capabilities of multimodal large language models(MLLMs). However, the limited perception performance of CLIP-ViT visionencoders in existing MLLMs struggles with capturing essential visual perceptioninformation, e.g., high-resolution, multi-scale and visual-related inductivebiases, which are important for autonomous driving. Addressing thesechallenges, we introduce HiLM-D, a resource-efficient framework that enhancesvisual information processing in MLLMs for ROLISP. Our method is motivated bythe fact that the primary variations in autonomous driving scenarios are themotion trajectories rather than the semantic or appearance information (e.g.,the shapes and colors) of objects. Hence, the visual process of HiLM-D is atwo-stream framework: (i) a temporal reasoning stream, receiving low-resolutiondynamic video content, to capture temporal semantics, and (ii) a spatialperception stream, receiving a single high-resolution frame, to captureholistic visual perception-related information. The spatial perception streamcan be made very lightweight by a well-designed P-Adapter, which islightweight, training-efficient, and easily integrated into existing MLLMs.Experiments on the DRAMA-ROLISP dataset show HiLM-D's significant improvementsover current MLLMs, with a 3.7% in BLEU-4 for captioning and 8.7% in mIoU fordetection.

HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving