Confidence and Diversity for Active Selection of Feedback in Image Retrieval

Image search is a challenging problem because of the need to model any concept the user might want to retrieve. One recent solution to the problem allows the user to give feedback on the current set of results, by answering questions about how the relative attributes of individual returned images relate to his/her target image. We show how to ask more informative questions. In our active selection formulation that determines about which attribute the system should next ask a question, we account for the confidence of relative attribute models. In addition to asking about reliably modeled attributes, the system is also encouraged to ask diverse questions, by computing question diversity on both the attribute and image levels. We show that both of our novel active selection criteria, confidence and diversity, help improve search results on three datasets. Further, when used in combination, they boost performance more than either cue alone.