We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
Prospective Validation of an Electronic Health Record–Based, Real-Time Suicide Risk Model.
- Authors
Walsh, Colin G.; Johnson, Kevin B.; Ripperger, Michael; Sperry, Sarah; Harris, Joyce; Clark, Nathaniel; Fielstein, Elliot; Novak, Laurie; Robinson, Katelyn; Stead, William W.
- Abstract
This cohort study evaluates the performance of a suicide attempt risk prediction model implemented in a vendor-supplied electronic health record to predict subsequent suicidal ideation and suicide attempt. Key Points: Question: How well do electronic health record–based suicide risk models perform in the clinical setting, and is performance generalizable? Findings: This cohort study of 30-day suicide attempt risk among 77 973 patients showed good performance in nonpsychiatric clinical settings at scale and in real time. Numbers needed to screen were reasonable for an algorithmic screening test that required no additional data collection or face-to-face screening to calculate. Meaning: Suicide attempt risk models can be implemented with accurate performance at scale, but performance is not equal in all clinical settings, which requires model recalibration and updating prior to deployment in new settings. Importance: Numerous prognostic models of suicide risk have been published, but few have been implemented outside of integrated managed care systems. Objective: To evaluate performance of a suicide attempt risk prediction model implemented in a vendor-supplied electronic health record to predict subsequent (1) suicidal ideation and (2) suicide attempt. Design, Setting, and Participants: This observational cohort study evaluated implementation of a suicide attempt prediction model in live clinical systems without alerting. The cohort comprised patients seen for any reason in adult inpatient, emergency department, and ambulatory surgery settings at an academic medical center in the mid-South from June 2019 to April 2020. Main Outcomes and Measures: Primary measures assessed external, prospective, and concurrent validity. Manual medical record validation of coded suicide attempts confirmed incident behaviors with intent to die. Subgroup analyses were performed based on demographic characteristics, relevant clinical context/setting, and presence or absence of universal screening. Performance was evaluated using discrimination (number needed to screen, C statistics, positive/negative predictive values) and calibration (Spiegelhalter z statistic). Recalibration was performed with logistic calibration. Results: The system generated 115 905 predictions for 77 973 patients (42 490 [54%] men, 35 404 [45%] women, 60 586 [78%] White, 12 620 [16%] Black). Numbers needed to screen in highest risk quantiles were 23 and 271 for suicidal ideation and attempt, respectively. Performance was maintained across demographic subgroups. Numbers needed to screen for suicide attempt by sex were 256 for men and 323 for women; and by race: 373, 176, and 407 for White, Black, and non-White/non-Black patients, respectively. Model C statistics were, across the health system: 0.836 (95% CI, 0.836-0.837); adult hospital: 0.77 (95% CI, 0.77-0.772); emergency department: 0.778 (95% CI, 0.777-0.778); psychiatry inpatient settings: 0.634 (95% CI, 0.633-0.636). Predictions were initially miscalibrated (Spiegelhalter z = −3.1; P =.001) with improvement after recalibration (Spiegelhalter z = 1.1; P =.26). Conclusions and Relevance: In this study, this real-time predictive model of suicide attempt risk showed reasonable numbers needed to screen in nonpsychiatric specialty settings in a large clinical system. Assuming that research-valid models will translate without performing this type of analysis risks inaccuracy in clinical practice, misclassification of risk, wasted effort, and missed opportunity to correct and prevent such problems. The next step is careful pairing with low-cost, low-harm preventive strategies in a pragmatic trial of effectiveness in preventing future suicidality.
- Subjects
SUICIDE risk factors; SCIENTIFIC observation; RESEARCH evaluation; CONFIDENCE intervals; HOSPITAL emergency services; ACADEMIC medical centers; ACQUISITION of data methodology; RESEARCH methodology evaluation; RESEARCH methodology; RACE; SURGICAL clinics; SUICIDAL ideation; RISK assessment; SEX distribution; RESEARCH funding; DESCRIPTIVE statistics; MEDICAL records; ELECTRONIC health records; PREDICTION models; RECEIVER operating characteristic curves; PREDICTIVE validity; LONGITUDINAL method; PSYCHIATRIC hospitals; EVALUATION
- Publication
JAMA Network Open, 2021, Vol 4, Issue 3, pe211428
- ISSN
2574-3805
- Publication type
Article
- DOI
10.1001/jamanetworkopen.2021.1428