Premium
Making the AGREE tool more user‐friendly: the feasibility of a user guide based on Boolean operators
Author(s) -
Scott N. Ann,
Moga Carmen,
Harstall Christa
Publication year - 2009
Publication title -
journal of evaluation in clinical practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.737
H-Index - 73
eISSN - 1365-2753
pISSN - 1356-1294
DOI - 10.1111/j.1365-2753.2009.01265.x
Subject(s) - computer science , reliability (semiconductor) , quality (philosophy) , guideline , set (abstract data type) , ambiguity , categorization , test (biology) , task (project management) , medical physics , data mining , artificial intelligence , medicine , paleontology , philosophy , power (physics) , physics , management , epistemology , pathology , quantum mechanics , economics , biology , programming language
Rationale, aims and objectives The Appraisal of Guidelines Research and Evaluation (AGREE) instrument is a generic tool for assessing guideline quality. This feasibility study aimed to reduce the ambiguity and subjectivity associated with AGREE item scoring, and to augment the tool's capacity to differentiate between good‐ and poor‐quality guidelines. Methods A literature review was conducted to ascertain what AGREE instrument adjustments had been reported to date. The AGREE User Guide was then modified by: 1 constructing a detailed set of instructions, or dictionary, using Boolean operators, and 2 overlaying seven criteria to categorize guideline quality. The feasibility of the Boolean‐based dictionary was tested by three appraisers using three randomly selected guidelines on low back pain management. The dictionary was then revised and re‐tested. Results Of the 52 published studies identified, 14% had modified the instrument by adding or deleting items and 35% had adopted strategies, such as using a consensus approach, to overcome inconsistencies and ensure identical item scoring among appraisers. For the feasibility test, Pearson correlation coefficients ranged from 0.27 to 0.81. Revision and re‐testing of the dictionary increased the level of agreement (range 0.41 to 0.94). Application of the revised dictionary not only decreased the variability of the domain scores, but also reduced the tool's reliability among inexperienced appraisers. Conclusion Appraisers found the Boolean‐based AGREE User Guide easier to use than the original, which improved their confidence in the tool. Good reliability was achieved in the feasibility test, but the reliability and validity of some of the changes will require further evaluation.