Wanted: a ‘Reviewer Effectiveness Index‘

Another yearly individual performance assessment exercise has gone by in universities and research institutes around the world, and, once again, in the overwhelming majority of cases, a crucial aspect of researchers’ activity has been virtually ignored. As a rule, universities and research centers require data on grant monies awarded to their employees (or simply the amount of overhead they generated), on number of articles, books, or book chapters published, on awards, and on the number of invited talks. Information about committee memberships, internally or in national/ international organizations, is commonly requested. To the extent that they can bring prestige (and, therefore, ultimately money) to their institution, administrators are also interested in any editorial responsibilities researchers have. However, few and far between are the institutions that solicit information of any kind about the anonymous, behindthe-scene work of peer reviewing manuscripts for journals, or proposals for funding agencies. And yet, peer reviewing, a process that has often been criticized1 but for which no viable alternative has ever been proposed, occupies a sizeable portion of researchers’ time. A quick, back-of-the-envelope calculation demonstrates this clearly. Most journals and funding agencies require between two and four peer reviews for each manuscript or proposal they have to evaluate. Therefore, if one takes multi-authorship into account, as well as the fact that more manuscripts or proposals are submitted than are eventually published or funded, each researcher, on average, should probably review roughly three times as many manuscripts and research proposals as he/she is submitting. Someone publishing, say, five articles and submitting two proposals per year should be willing to review at least 15 manuscripts and 6 proposals, i.e. carry out 21 reviews in total per year. The time spent on each of them depends on the type of manuscript or proposal involved, on the amount (if any) of background reading needed to produce a good review, and on the level of detail with which the peer-review report is put together. An average of about 6 hours per review, from start to finish, seems reasonable, with some people spending considerably longer than that. Given that estimate, the individual mentioned above should theoretically devote roughly 126 hours per year to peer reviewing, equivalent to slightly over three full, 40-hours weeks of work, which is a sizeable fraction of his/her overall effort.2 In the last few years, not without a sense of alarm at this development, journal editors in a wide range of disciplines have reported increasing difficulties in finding peer reviewers.3 Statistics are lacking to determine how serious the problem is. Likewise, it is not entirely clear what other causes of the problem there may be, besides the lack of recognition of the peer-review process. Certainly, the fact that the pressure on researchers to attract grant monies has increased dramatically over the last decade, and that the number of manuscripts submitted for publication has skyrocketed (in particular from countries such as China or India that until recently did not have a strong publishing tradition)4 must be contributing factors. Nevertheless, regardless of the lack of information on the peer-review crisis, the perception of editors, ‘from the trenches’, seems difficult to dismiss. Their day-to-day observation is that, unlike in years passed, it is becoming less and less exceptional for them to have to issue 10–15 invitations before they can secure the peer reviews needed to assess a given manuscript. Most often, researchers declining invitations to review invoke the fact that they are too busy to add yet another item to their already overcommitted schedule. In 232 Philippe C. Baveye