We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Accuracy and Efficiency of Machine Learning-Assisted Risk-of-Bias Assessments in "Real-World" Systematic Reviews : A Noninferiority Randomized Controlled Trial.
- Authors
Arno, Anneliese; Thomas, James; Wallace, Byron; Marshall, Iain J.; McKenzie, Joanne E.; Elliott, Julian H.
- Abstract
<bold>Background: </bold>Automation is a proposed solution for the increasing difficulty of maintaining up-to-date, high-quality health evidence. Evidence assessing the effectiveness of semiautomated data synthesis, such as risk-of-bias (RoB) assessments, is lacking.<bold>Objective: </bold>To determine whether RobotReviewer-assisted RoB assessments are noninferior in accuracy and efficiency to assessments conducted with human effort only.<bold>Design: </bold>Two-group, parallel, noninferiority, randomized trial. (Monash Research Office Project 11256).<bold>Setting: </bold>Health-focused systematic reviews using Covidence.<bold>Participants: </bold>Systematic reviewers, who had not previously used RobotReviewer, completing Cochrane RoB assessments between February 2018 and May 2020.<bold>Intervention: </bold>In the intervention group, reviewers received an RoB form prepopulated by RobotReviewer; in the comparison group, reviewers received a blank form. Studies were assigned in a 1:1 ratio via simple randomization to receive RobotReviewer assistance for either Reviewer 1 or Reviewer 2. Participants were blinded to study allocation before starting work on each RoB form.<bold>Measurements: </bold>Co-primary outcomes were the accuracy of individual reviewer RoB assessments and the person-time required to complete individual assessments. Domain-level RoB accuracy was a secondary outcome.<bold>Results: </bold>Of the 15 recruited review teams, 7 completed the trial (145 included studies). Integration of RobotReviewer resulted in noninferior overall RoB assessment accuracy (risk difference, -0.014 [95% CI, -0.093 to 0.065]; intervention group: 88.8% accurate assessments; control group: 90.2% accurate assessments). Data were inconclusive for the person-time outcome (RobotReviewer saved 1.40 minutes [CI, -5.20 to 2.41 minutes]).<bold>Limitation: </bold>Variability in user behavior and a limited number of assessable reviews led to an imprecise estimate of the time outcome.<bold>Conclusion: </bold>In health-related systematic reviews, RoB assessments conducted with RobotReviewer assistance are noninferior in accuracy to those conducted without RobotReviewer assistance.<bold>Primary Funding Source: </bold>University College London and Monash University.
- Subjects
EXPERIMENTAL design; RESEARCH; RESEARCH methodology; EVALUATION research; RISK assessment; COMPARATIVE studies; RANDOMIZED controlled trials; RESEARCH funding
- Publication
Annals of Internal Medicine, 2022, Vol 175, Issue 7, p1001
- ISSN
0003-4819
- Publication type
journal article
- DOI
10.7326/M22-0092