Premium
The validity and reliability of a signal impact assessment tool
Author(s) -
Rolfes Leàn,
Kolfschoten Judith,
Hunsel Florence,
Puijenbroek Eugène
Publication year - 2016
Publication title -
pharmacoepidemiology and drug safety
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.023
H-Index - 96
eISSN - 1099-1557
pISSN - 1053-8569
DOI - 10.1002/pds.4038
Subject(s) - medicine , reliability (semiconductor) , kappa , validity , gold standard (test) , clinical psychology , psychometrics , mathematics , power (physics) , physics , geometry , quantum mechanics
Background The Netherlands Pharmacovigilance Centre Lareb uses an operating Signal Impact Assessment Tool (SIAT) as aid in signal selection. SIAT prioritized signals into one of four categories: strong/moderate signal strength and similarly health impact. Although the SIAT has been used for many years, validity and reliability was never explored. Purpose The aim of this study is to test the validity and reliability of the operating and weight‐adjusted SIAT. Method For validity testing, judgments of a Delphi panel of three pharmacovigilance experts were used as a ‘gold standard’. First, the panel judged the weighting of the items included in the SIAT. Then, during two phases, the panel rated 40 signals in one of the four categories. Two researchers scored the signals using the SIAT. Panel judgments were compared with scores for the operating and weight‐adjusted SIAT. Inter‐ and intra‐observer variability was also tested. The Cohen's Kappa coefficient (k) was calculated. At least substantial agreement ( k > 0.6) was considered to be necessary for an acceptable reliability. Results Validity did not meet predefined criteria: operating and weight‐adjusted tool, respectively, k‐phase1 = 0.83 and 0.83, k‐phase2 = 0.18 and 0.36. Differences were found for signal strength and health impact. Inter‐ and intra‐observer variabilities were good, k of 0.78 and 0.72, respectively. Conclusions The SIAT was found to have an insufficient validity and proper reliability. Although SIAT scores should not be decisive in the decision making process, items included can be used as an aid to decide which signals deserve further investigation. Copyright © 2016 John Wiley & Sons, Ltd.