Exploring the synergy between knowledge graph and computer vision for personalisation systems

In this paper, we explore the synergy between knowledge graph technologies and computer vision tools for personalisation systems. We propose two image user profiling approaches which map an image to knowledge graph entities representing the interests of a user who appreciates the image. The first one maps an image to entities which correspond to the objects appearing in the image. The second maps to entities which are depicted by visually similar images and which exist in the conceptual scope of the dataset within which further personalisation tasks are conducted. We show the superiority of our second approach against the baseline Google Cloud Vision API (label detection and web entity detection) in terms of accuracy metrics (precision, recall, MRR, nDCG). We also argue the importance of the capacity to create semantically useful profiles as the essence of many knowledge- or semantic-based personalisation systems is the semantic similarity calculation. We then apply our profiling approach in a novel personalisation use case where we seek to select the most appropriate images to display in recommendation banners. Our proposed knowledge-based approach tries to select the images which are the most in line with the semantic user profiles. We hypothesise that this image selection strategy allows to improve the user’s perception of the recommended items. We conduct a two-stage user study with a real commercial travel dataset (1,357 package tours in 136 countries and regions depicted by 11,614 images). The results of 32 participants allow us to observe the promising performance of our approach in terms of persuasion, attention, efficiency and affinity.