English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision-language (VL) pretrained models have achieved impressive performanceon multimodal reasoning and zero-shot recognition tasks. Many of these VLmodels are pretrained on unlabeled image and caption pairs from the internet.In this paper, we study whether representations of primitive concepts--such ascolors, shapes, or the attributes of object parts--emerge automatically withinthese pretrained VL models. We propose a two-step framework, CompositionalConcept Mapping (CompMap), to investigate this. CompMap first asks a VL modelto generate primitive concept activations with text prompts, and then learns toconstruct a composition model that maps the primitive concept activations (e.g.the likelihood of black tail or red wing) to composite concepts (e.g. ared-winged blackbird). We show that a composition model can be reliably learnfrom ground truth primitive concepts. We thus hypothesize that if primitiveconcepts indeed emerge in a VL pretrained model, its primitive conceptactivations can be used to learn a composition model similar to the onedesigned by experts. We propose a quantitative metric to measure the degree ofsimilarity, and refer to the metric as the interpretability metric. We alsomeasure the classification accuracy when using the primitive conceptactivations and the learned composition model to predict the compositeconcepts, and refer to it as the usefulness metric. Our study reveals thatstate-of-the-art VL pretrained models learn primitive concepts that are highlyuseful for fine-grained visual recognition on the CUB dataset, andcompositional generalization tasks on the MIT-States dataset. However, weobserve that the learned composition models have low interpretability in ourqualitative analyses. Our results reveal the limitations of existing VL models,and the necessity of pretraining objectives that encourage the acquisition ofprimitive concepts.

Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?