We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
External Validation and Updating of a Statistical Civilian-Based Suicide Risk Model in US Naval Primary Care.
- Authors
Ripperger, Michael A.; Kolli, Jhansi; Wilimitis, Drew; Robinson, Katelyn; Reale, Carrie; Novak, Laurie L.; Cunningham, Craig A.; Kasuske, Lalon M.; Grover, Shawna G.; Ribeiro, Jessica D.; Walsh, Colin G.
- Abstract
Key Points: Question: How well does a civilian-based suicide risk model generalize in US Navy primary care? Findings: In this cohort study with 260 583 service members, domain and temporal validation showed internal retraining and external validation had similar performance. Updating with US Navy–specific factors added minimal improvement. Meaning: These findings suggest that civilian-based risk models might generalize to military health settings; prior to transferring risk models, external validation might demonstrate adequate performance in new settings and avoid costly internal development. This cohort study externally validates and updates a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care for use among US Navy service members. Importance: Suicide remains an ongoing concern in the US military. Statistical models have not been broadly disseminated for US Navy service members. Objective: To externally validate and update a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care. Design, Setting, and Participants: This retrospective cohort study used data collected from 2007 through 2017 among active-duty US Navy service members. The external civilian model was applied to every visit at Naval Medical Center Portsmouth (NMCP), its NMCP Naval Branch Health Clinics (NBHCs), and TRICARE Prime Clinics (TPCs) that fall within the NMCP area. The model was retrained and recalibrated using visits to NBHCs and TPCs and updated using Department of Defense (DoD)–specific billing codes and demographic characteristics, including expanded race and ethnicity categories. Domain and temporal analyses were performed with bootstrap validation. Data analysis was performed from September 2020 to December 2022. Exposure: Visit to US NMCP. Main Outcomes and Measures: Recorded suicidal behavior on the day of or within 30 days of a visit. Performance was assessed using area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), Brier score, and Spiegelhalter z-test statistic. Results: Of the 260 583 service members, 6529 (2.5%) had a recorded suicidal behavior, 206 412 (79.2%) were male; 104 835 (40.2%) were aged 20 to 24 years; and 9458 (3.6%) were Asian, 56 715 (21.8%) were Black or African American, and 158 277 (60.7%) were White. Applying the civilian-trained model resulted in an AUROC of 0.77 (95% CI, 0.74-0.79) and an AUPRC of 0.004 (95% CI, 0.003-0.005) at NBHCs with poor calibration (Spiegelhalter P <.001). Retraining the algorithm improved AUROC to 0.92 (95% CI, 0.91-0.93) and AUPRC to 0.66 (95% CI, 0.63-0.68). Number needed to screen in the top risk tiers was 366 for the external model and 200 for the retrained model; the lower number indicates better performance. Domain validation showed AUROC of 0.90 (95% CI, 0.90-0.91) and AUPRC of 0.01 (95% CI, 0.01-0.01), and temporal validation showed AUROC of 0.75 (95% CI, 0.72-0.78) and AUPRC of 0.003 (95% CI, 0.003-0.005). Conclusions and Relevance: In this cohort study of active-duty Navy service members, a civilian suicide attempt risk model was externally validated. Retraining and updating with DoD-specific variables improved performance. Domain and temporal validation results were similar to external validation, suggesting that implementing an external model in US Navy primary care clinics may bypass the need for costly internal development and expedite the automation of suicide prevention in these clinics.
- Subjects
SUICIDE risk factors; MILITARY hospitals; CONFIDENCE intervals; RESEARCH methodology; RETROSPECTIVE studies; RISK assessment; PRIMARY health care; UNITED States. Navy; DESCRIPTIVE statistics; RESEARCH funding; STATISTICAL models; LONGITUDINAL method; ALGORITHMS
- Publication
JAMA Network Open, 2023, Vol 6, Issue 11, pe2342750
- ISSN
2574-3805
- Publication type
Article
- DOI
10.1001/jamanetworkopen.2023.42750