Relevance & Assessment: Cognitively Motivated Approach toward Assessor-Centric Query-Topic Relevance Model

This paper intends to introduce a novel model for query-topic relevance assessment from assessor and cognitive point of view in the sense that relevance is a multidimensional cognitive and dynamic conception. The focus of this presentation is concentrated on modeling the concept "Query Associative Vocabulary of Relevance" to emphasize the value of integrating intuitive, descriptive, multi-valued assessment, and agreement in the process of creating a query-topic relevance data. As this model differentiates between different types of query-topics and levels of relevance, it provides a facility to enhance the quality of relevance data by re-evaluating the resulted associative vocabulary at each cycle of refinement. This aspect is of importance, as it is directed toward extracting as much advantage from human assessment as possible. A prototype of this model has generated in an initial run a relevance dataset of 20.710 relevance assessor’s feedback and a co-occurrence matrix of 39607 terms distributed in intuitive, descriptive and document associative vocabularies. Most of the assessor feedback is descriptive produced by humans in context of establishing a relevance relationship between a query-topic and related documents. Furthermore, classifying query relevance datasets according to grades of agreements among judgments is useful as it gives a better overview of the performance of the considered system and the comparison of different datasets in context of consistency and performance becoming easier. Despite the importance of relevance in designing and evaluating Information Retrieval Systems as possible intercognitive systems, a consensus on definition is still debatable. However, considering relevance as a multidimensional cognitive and dynamic conception provides researcher with a research track to evaluate the performance of interactive and inter-cognitive processes in terms of the multidimensionality and cognitive aspects of relevance.