Open AccessCross-modal retrieval by an end to end way
Author(s) -
Bang Hu Yin,
Li Xiu
Publication year - 2020
Publication title -
iop conference series. materials science and engineering
Language(s) - English
Resource type - Journals
eISSN - 1757-899X
pISSN - 1757-8981
DOI - 10.1088/1757-899x/715/1/012082
Subject(s) - computer science , similarity (geometry) , modal , semantic similarity , information retrieval , image retrieval , modality (human–computer interaction) , explicit semantic analysis , image (mathematics) , artificial intelligence , process (computing) , task (project management) , natural language processing , semantic computing , semantic technology , semantic web , chemistry , management , polymer chemistry , economics , operating system
Cross-modal retrieval has attracted most attention in the recent years. For the image and text, how to measure the semantic similarity among them is still a challenging problem in cross-modal retrieval task. In our paper, we propose a cross-modal retrieval method that uses a CNN to obtain the semantic similarity between image and text. Most methods to solve this problem employ two separate parts for each modality to obtain the semantic similarity between them. However, in our work we just use a CNN to get the semantic similarity without having to use separate networks. In addition, we are aiming to solve the problem between long text and image, so we use the topic model to process the text. We evaluate our approach on Wikepedia dataset.