Premium
Imperfect referees: Reducing the impact of multiple biases in peer review
Author(s) -
Zhao Yun Wei,
Chi ChiHung,
Heuvel WillemJan
Publication year - 2015
Publication title -
journal of the association for information science and technology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.903
H-Index - 145
eISSN - 2330-1643
pISSN - 2330-1635
DOI - 10.1002/asi.23317
Subject(s) - context (archaeology) , prejudice (legal term) , computer science , reliability (semiconductor) , imperfect , publication bias , quality (philosophy) , meta analysis , psychology , confidence interval , social psychology , statistics , medicine , paleontology , power (physics) , linguistics , physics , philosophy , mathematics , epistemology , quantum mechanics , biology
Bias in peer review entails systematic prejudice that prevents accurate and objective assessment of scientific studies. The disparity between referees' opinions on the same paper typically makes it difficult to judge the paper's quality. This article presents a comprehensive study of peer review biases with regard to 2 aspects of referees: the static profiles (factual authority and self‐reported confidence) and the dynamic behavioral context (the temporal ordering of reviews by a single reviewer), exploiting anonymized, real‐world review reports of 2 different international conferences in information systems / computer science. Our work extends conventional bias research by considering multiple biases occurring simultaneously. Our findings show that the referees' static profiles are more dominant in peer review bias when compared to their dynamic behavioral context. Of the static profiles, self‐reported confidence improved both conference fitness and impact‐based bias reductions, while factual authority could only contribute to conference fitness‐based bias reduction. Our results also clearly show that the reliability of referees' judgments varies along their static profiles and is contingent on the temporal interval between 2 consecutive reviews.