English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this work, we investigate extending the comprehension of Multi-modal LargeLanguage Models (MLLMs) to regional objects. To this end, we propose to extractfeatures corresponding to regional objects as soft prompts for LLM, whichprovides a straightforward and scalable approach and eliminates the need forLLM fine-tuning. To effectively extract regional features from regular imagefeatures and irregular point cloud features, we present a novel and unifiedposition-assisted feature extraction module. Furthermore, training an MLLM fromscratch is highly time-consuming. Thus, we propose incrementally extendingexisting pre-trained MLLMs to comprehend more modalities and the regionalobjects of those modalities. Specifically, we freeze the Q-Former from BLIP-2,an impressive MLLM, and optimize the modality-specific Lora parameters inQ-Former and LLM for each newly introduced modality. The freezing of theQ-Former eliminates the need for extensive pre-training on massive image-textdata. The freezed Q-Former pre-trained from massive image-text data is alsobeneficial for the pre-training on image-region-text data. We name ourframework RegionBLIP. We pre-train RegionBLIP on image-region-text,point-cloud-text, and point-cloud-region-text data. Experimental results verifythat \Ours{} can preserve the image comprehension capability of BILP-2 andfurther gain a comprehension of the newly introduced point cloud modality andregional objects. The Data, Code, and Pre-trained models will be available athttps://github.com/mightyzau/RegionBLIP.

RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension