LSTM for Image Annotation with Relative Visual Importance

We consider the problem of image annotations that takes into account of the relative visual importance of tags. Previous works usually consider the tags associated with an image as an unordered set of object names. In contrast, we exploit the implicit cues about the relative importance of objects mentioned by the tags. For example, important objects tend to be mentioned first in a list of tags. We propose a recurrent neural network with long-short term memory to model this. Given an image, our model can produce a ranked list of tags, where tags for objects of higher visual importance appear earlier in the list. Experimental results demonstrate that our model achieves better performance on several benchmark datasets.