Embedding Visual Words into Concept Space for Action and Scene Recognition

In this paper we propose a novel approach to introducing semantic relations into the bag-of-words framework. We use the latent semantic models, such as LSA and pLSA, in order to define semantically-rich features and embed the visual features into a semantic space. The semantic features used in LSA technique are derived from the low-rank approximation of word-document occurrence matrix by SVD. Similarly, by using the pLSA approach, the topic-specific distributions of words can be considered dimensions of a concept space. In the proposed space, the distances between words represent the semantic distances which are used for constructing a discriminative and semantically meaningful vocabulary. We have tested our approach on the KTH action database and on the Fifteen Scene database and have achieved very promising results on both.