Semantic Description of Objects in Images Based on Prototype Theory

This research aims to build a model for the semantic description of objects based on visual features extracted from images. We introduce a novel semantic description approach inspired by the Prototype Theory. Inspired by the human approach used to represent categories, we propose a novel Computational Prototype Model (CPM) that encodes and stores the object’s image category’s central semantic meaning: the semantic prototype. Our CPM model represents and constructs the semantic prototypes of object categories using Convolutional Neural Networks (CNN). The proposed Prototype-based Description Model uses the CPM model to describe an object highlighting its most distinctive features within the category. Our Global Semantic Descriptor (GSDP) builds discriminative, low-dimensional, and semantically interpretable signatures that encode the objects’ semantic information using the constructed semantic prototypes. It uses the proposed Prototypical Similarity Layer (PS-Layer) to retrieve the category prototype using the principle of categorization based on prototypes. Using different datasets, we show in our experiments that: i) the proposed CPM model successfully simulates the internal semantic structure of the categories; ii) the proposed semantic distance metric can be understood as the object typicality score within a category; iii) our semantic classification method based on prototypes can improve the performance and interpretation of CNN classification models; iv) our semantic descriptor encoding significantly outperforms others state-of-the-art image global encoding in clustering and classification tasks.