We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Calibration drift in regression and machine learning models for acute kidney injury.
- Authors
Davis, Sharon E.; Lasko, Thomas A.; Guanhua Chen; Siew, Edward D.; Matheny, Michael E.; Chen, Guanhua
- Abstract
<bold>Objective: </bold>Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.<bold>Materials and Methods: </bold>Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.<bold>Results: </bold>Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.<bold>Conclusions: </bold>Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.
- Subjects
UNITED States; KIDNEY injuries; MACHINE learning; DECISION support systems -- Medical applications; ARTIFICIAL neural networks; MEDICAL decision making; ACUTE kidney failure; DECISION making; PROBABILITY theory; RESEARCH funding; VETERANS' hospitals; LOGISTIC regression analysis
- Publication
Journal of the American Medical Informatics Association, 2017, Vol 24, Issue 6, p1052
- ISSN
1067-5027
- Publication type
journal article
- DOI
10.1093/jamia/ocx030