ARTICLE REVIEW CHECKLIST: A CRITERION CHECKLIST FOR REVIEWING RESEARCH ARTICLES IN APPLIED PSYCHOLOGY 1

Over the past couple of years, a large group of reviewers and I have been developing a comprehensive checklist of criteria for reviewing research articles. The purpose of this effort was to provide a heuristic device of issues to think about when reviewing an article. As such, we hoped that the checklist might be a useful tool for reviewers, authors, and students. A two-part Delphi-like procedure was followed. In the first part, a preliminary checklist of criteria was developed and circulated to a large number of reviewers. It contained 93 criteria and was divided into 14 categories (e.g., literature review, sample, measures, procedures, analyses , conclusions, etc.). Reviewers were asked to examine the list and think about what they look for when reviewing an article, and then to modify the criteria on the list or add criteria to it. They suggested 860 additional items and 429 modifications to the original criteria. They were also asked to send in any existing unpublished checklists they might have, and several were obtained containing 135 more items. As the criteria were edited and condensed, special effort was made to incorporate all the content and much of the specific wording of the reviewers' suggestions. The resulting checklist had 246 criteria divided into 16 categories. In the second part of the study, the revised checklist was again circulated to the group of reviewers. This time they were asked to rate each criterion in terms of the weighting it should receive when reviewing an article , and they were asked about their background and experience.^ The information was used to develop the final version of the checklist. First, approximately 9% of the criteria were eliminated because they were relatively unimportant (e.g., received low ratings) or were ambiguous (e.g., many ratings left blank). Second, the checklist was simplified by grouping the criteria within each category into clusters of similar criteria. Finally , within each cluster, the criteria were listed in a very gross rank ordering of importance based on the ratings. The final checklist contained 223 criteria, 44 clusters, and 15 categories. Special thanks to the reviewers who not only provided the ideas and data for this checklist, but who also labor tirelessly with little recognition to make the review process work. This information was used to explore differences between reviewers on the weightings they assigned to the criteria. These data will be reported elsewhere.