Premium
Interrater reliability and convergent validity of F 1000 P rime peer review
Author(s) -
Bornmann Lutz
Publication year - 2015
Publication title -
journal of the association for information science and technology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.903
H-Index - 145
eISSN - 2330-1643
pISSN - 2330-1635
DOI - 10.1002/asi.23334
Subject(s) - inter rater reliability , hard rime , reliability (semiconductor) , psychology , convergent validity , peer review , sass , validity , computer science , clinical psychology , psychometrics , rating scale , developmental psychology , physics , world wide web , political science , power (physics) , quantum mechanics , law , internal consistency , meteorology
Peer review is the backbone of modern science. F 1000 P rime is a postpublication peer review system of the biomedical literature (papers from medical and biological journals). This study is concerned with the interrater reliability and convergent validity of the peer recommendations formulated in the F 1000 P rime peer review system. The study is based on about 100,000 papers with recommendations from faculty members. Even if intersubjectivity plays a fundamental role in science, the analyses of the reliability of the F 1000 P rime peer review system show a rather low level of agreement between faculty members. This result is in agreement with most other studies that have been published on the journal peer review system. Logistic regression models are used to investigate the convergent validity of the F 1000 P rime peer review system. As the results show, the proportion of highly cited papers among those selected by the faculty members is significantly higher than expected. In addition, better recommendation scores are also associated with higher performing papers.