1016 — Building and validating a machine learning model to predict SUD using integrated electronic health record and survey data
Lead/Presenter: Katherine Hoggatt,
San Francisco VA Health Care Center
All Authors: Hoggatt KJ (San Francisco VA Health Care System), Harris AHS (Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA; Department of Surgery, Stanford University School of Medicine, Palo Alto, CA) Washington DL (Center for the Study of Healthcare Innovation, Implementation, & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA; Division of General Internal Medicine and Health Services Research, UCLA David Geffen School of Medicine, Los Angeles, CA) Williams EC (Center of Innovation for Veteran Centered and Value-Driven Care, VA Puget Sound, Seattle, WA; Department of Health Services, University of Washington, Seattle, WA)
Substance use disorder (SUD) is a common diagnosis among VA patients. VA uses measures of SUD identification (i.e., the proportion of patients diagnosed with SUD at a facility) to monitor population health, measure healthcare delivery within and between systems, and track healthcare disparities for patients with SUD. These efforts are undermined when diagnoses are a poor proxy for true prevalence, which could occur due to variability in diagnostic practices. With better estimates of SUD prevalence we could distinguish variation in performance from variation in diagnoses. VA electronic health records (EHRs) have ample data that could be used to predict (estimate) SUD prevalence. However, building and validating a predictive model requires a criterion outcome in addition to the data inputs, and for SUD the criterion outcome measure is not in the EHR and requires primary data collection. Our objective was to develop and validate a machine learning model to produce better estimates of SUD prevalence.
We surveyed 6000 Veteran outpatients from 30 geographically-representative VA healthcare systems. A criterion measure for past-year SUD was collected via telephone interview conducted from 01/2018-04/2019 (51% response rate). The survey included validated items assessing DSM-5 criteria for substance-specific use disorders. We linked survey data to EHR data, including demographics, health conditions, health care utilization, medication history, SUD-related laboratory tests, SUD-related pharmacotherapy, receipt of alcohol screening, and receipt of a brief intervention. We trained machine learning models (LASSO, Ridge Regression, ElasticNet, random forest (RF), and gradient boosting machines (GBM)) with repeated 10-fold cross-validation and quantified model performance on a hold-out dataset. We assessed performance using measures of discrimination (sensitivity (Se), specificity (Sp), area under the ROC curve (AUROC)) and accuracy (logloss, Brier score).
Respondents were predominantly male (91.2%), White (68.2%), and ages 65 and older (54.8%). Survey-based SUD prevalence was greater than the diagnosis rate (12.8% vs. 8.5%). As expected, SUD diagnoses were a poor proxy for prevalent SUD (Se = 34%). All methods yielded models with similar discrimination (i.e., ability to correctly classify patients by SUD status): AUROC = 0.79-0.82. Two methods, RF and GBT, yielded predictions with lower log-loss (0.32-0.34) and lower Brier score (0.12 for both), indicating less distance between the observed and model-based predicted probabilities of SUD (i.e., higher accuracy). With an optimized threshold, the model-based sensitivity for detecting SUD was 77% or greater across models. Overall, GBM had slightly better performance than other methods.
A lack of SUD prevalence data can undermine efforts to monitor population health and ensure equitable healthcare access. Model-based prevalence estimates can complement health record diagnoses and help distinguish patterns in population health from variation in diagnosing.
Ensuring access to evidence-based SUD treatment is a VA priority, but these efforts are undermined when measures of disease burden and related healthcare quality/equity measures depend on documented diagnoses that do not accurately reflect true disease prevalence. Model-based estimates of SUD prevalence can be used to distinguish differences in performance from artifacts due to non-standardized diagnostic practices, facilitating comparisons across facilities and between VA-delivered and VA-purchased care.