Premium
A rule‐based approach for automatically extracting data from systematic reviews and their updates to model the risk of conclusion change
Author(s) -
Bashir Rabia,
Dunn Adam G.,
Surian Didi
Publication year - 2021
Publication title -
research synthesis methods
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.376
H-Index - 35
eISSN - 1759-2887
pISSN - 1759-2879
DOI - 10.1002/jrsm.1473
Subject(s) - random forest , computer science , systematic review , decision tree , logistic regression , data mining , machine learning , artificial intelligence , receiver operating characteristic , change detection , classifier (uml) , precision and recall , statistics , medline , mathematics , political science , law
Few data‐driven approaches are available to estimate the risk of conclusion change in systematic review updates. We developed a rule‐based approach to automatically extract information from reviews and updates to be used as features for modelling conclusion change risk. Rules were developed to extract relevant information from published Cochrane reviews and used to construct four features: the number of included trials and participants in the reviews, a measure based on the number of participants, and the time elapsed between the search dates. We compared the performance of random forest, decision tree, and logistic regression to predict the conclusion change risk. The performance was measured by accuracy, precision, recall, F 1 ‐score, and area under ROC (AU‐ROC). One rule was developed to extract the conclusion change information (96% accuracy, 100 reviews), one for the search date (100% accuracy, 100 reviews), one for the number of included clinical trials (100% accuracy, 100 reviews), and 22 for the number of participants (97.3% accuracy, 200 reviews). For unseen reviews, the random forest classifier showed the highest accuracy (80.8%) and AU‐ROC (0.80). All classifiers showed relatively similar performance with overlapping 95% confidence interval (CI). The coverage score was shown to be the most useful feature for predicting the conclusion change risk. Features mined from Cochrane reviews and updates can estimate conclusion change risk. If data from more published reviews and updates were made accessible, data‐driven methods to predict the conclusion change risk may be a feasible way to support decisions about updating reviews.