Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

VA Health Systems Research

Go to the VA ORD website
Go to the QUERI website

IIR 11-088 – HSR Study

IIR 11-088
Developing and Validating a Veterans Affairs Cardiac Risk Score
Rodney A. Hayward, MD
VA Ann Arbor Healthcare System, Ann Arbor, MI
Ann Arbor, MI
Funding Period: September 2012 - August 2016
Evidence-based medicine (EBM) has hugely advanced medical care by answering the question: "Which treatments work on average in a population?" Personalized medicine seeks to utilize EBM and other tools to answer the patient's question: "Am I likely to benefit and how much?" Thus, a central goal of patient-centered medical care is to tailor treatment agreements toward individual patient risks, benefits, and preferences. This benefit is now recognized in major guidelines for cholesterol and aspirin. Effective tailoring, however, cannot be possible without accurate predictions of individual patients' risks of developing major medical illness.
While multivariate risk/benefit prediction to individually tailor treatments could greatly increase treatment precision, tools to facilitate such improvements in VHA are lacking. For cardio-and cerebrovascular disease (CCV), which is the leading cause of morbidity and mortality in the U.S., current risk prediction tools have substantial shortcomings, including requiring manual entry of risk factor information, being developed and calibrated on patient populations quite different from VHA's (i.e., poor generalization to other populations), and failing to use new data-mining techniques or the robust continuum of clinical data available in VA's electronic medical record (VA EMR). To address these CCV treatment concerns, this project developed a VA-based risk prediction score to address the major concern of external validity, in addition to developing novel clinical algorithms to tailor clinical CCV decision-making and risk/benefit communication to individual Veterans.

The main goal of this project was to create a maximally effective risk prediction tool that is conceptually and clinically distinct from other commonly used risk-prediction tools.
To reach this goal, this study had two overarching specific aims: 1) To develop and assess the merits of two competing approaches to developing VA EMR-derived CCV risk prediction tools, one using standard regression models (REG models) and one using machine learning models (ML Models); and, 2) To compare the accuracy and clinical impact of these VA EMR-derived CCV risk prediction tools to each other and to commonly used risk prediction models developed outside of VA, such as the Framingham and Euro SCORE risk tools using traditional risk assessment techniques and newly-developed, clinically-relevant algorithms.

Our study population consisted of established national VHA patients age 45 and older. All primary analyses were conducted on those without known CCV, while those with a history of CCV disease or those receiving a procedure for CCV conditions during the past two years were assumed to have known CCV disease and therefore excluded. We created a national, longitudinal cohort using laboratory and pharmacy data from the DSS National Data Extracts and CDW outpatient pharmacy; outpatient and inpatient ICD-9 and CPT codes from the SAS Medical Datasets and VA-CMS data; clinical measures from the Corporate Data Warehouse; cause of death from the National Death Index. Our analytic approaches included regression-based predictive models (REG models) utilizing logistic regression methods, in addition to using machine learning models (ML models) using Random Forests and Boosting analytic techniques. We conducted an internal validation of our major CCV and fatal CCV models using REG and ML by examining the performance of these models using two separate cohorts of randomly selected subjects consisting of two-thirds of the population for the development dataset and one-third for the validation dataset. We compared our developed risk score (VARS-CVD) to other commonly used risk prediction models developed outside of VA, including the American College of Cardiology/American Heart Association's (ASCVD) Score, Framingham Risk Score, UKPDS Risk Score, and EuroSCORE.

Using an outcome of CVD events during a five year follow-up period, our study population consisted of 1,512,092 patients (1,435,937 men and 76,155 women).

For Aim 1, the developed VARS-CVD score had good calibration and passable discrimination. Results include a C-statistic of 0.66 in men and 0.73 in women. Machine learning models did not offer any statistical advantages over traditional logistic regression.

For Aim 2, we compared the VARS-CVD score to the ASCVD score, which has recently become the field's standard based on ACC/AHA cholesterol guidelines. The VARS score had similar discrimination to the ASCVD score, but much better calibration. The ASCVD score predicted 63% more events than what we observed.

These findings suggest that large managed care organizations can use their own electronic health records and administrative outcome data to create a cardiovascular disease (CVD) risk prediction score with major advantages over those developed in traditional cohort studies. The ASCVD score had an AUROC (which is a measure of discrimination) that was similar to our internal, health-system specific VARS-CVD score; however, ASCVD systematically over-estimated the observed risk in our cohort by almost 60%.

Moreover, these findings demonstrate that our internally developed score is much better calibrated than the externally-developed score. Furthermore, the ASCVD score classified 46,508 patients as moderate or high-risk, a level at which statin therapy is recommended, while these same patients were classified as low-risk using the VARS-CVD risk tool.

We have developed a model that can be used as an automated VA EMR-based CCV risk prediction tool that can be used to help tailor clinical decision-making to individual Veteran circumstances, thereby increasing efficiency of care management and aiding patient-centered decision-making for Veterans. This project's findings should have dramatic policy and clinical implications. Our work demonstrates that clinical risk scores can be made efficiently within an individual healthcare system's EHR. This work could be followed at other managed care organizations, in addition to being adjusted to many other clinical conditions. As EHRs improve and "Big Data" becomes more central to patient self-management, our findings offer a blueprint for how these tools can be used in clinical practice, preventing CVD events, and guiding statin use for patients who are more likely to benefit. Furthermore, our VA-specific Cardiac Risk Score could have two major impacts on healthcare in VHA: 1) The VARS-CVD score could be easily integrated into practice via the VA EMR or a web-based interface; and, 2) By improving how we use the basic medications for CCV prevention, saving lives and expense.

External Links for this Project

NIH Reporter

Grant Number: I01HX000681-01A2

Dimensions for VA

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

Learn more about Dimensions for VA.

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.
    Search Dimensions for this project


Journal Articles

  1. Basu S, Sussman JB, Rigdon J, Steimle L, Denton BT, Hayward RA. Benefit and harm of intensive blood pressure treatment: Derivation and validation of risk models using data from the SPRINT and ACCORD trials. PLoS Medicine. 2017 Oct 1; 14(10):e1002410. [view]
  2. Zawistowski M, Sussman JB, Hofer TP, Bentley D, Hayward RA, Wiitala WL. Corrected ROC analysis for misclassified binary outcomes. Statistics in medicine. 2017 Jun 15; 36(13):2148-2160. [view]
  3. Basu S, Sussman JB, Berkowitz SA, Hayward RA, Yudkin JS. Development and validation of Risk Equations for Complications Of type 2 Diabetes (RECODe) using individual participant data from randomised trials. The lancet. Diabetes & endocrinology. 2017 Oct 1; 5(10):788-798. [view]
  4. Markovitz AA, Holleman RG, Hofer TP, Kerr EA, Klamerus ML, Sussman JB. Effects of Guideline and Formulary Changes on Statin Prescribing in the Veterans Affairs. Health services research. 2017 Dec 1; 52(6):1996-2017. [view]
  5. Vance MC, Wiitala WL, Sussman JB, Pfeiffer P, Hayward RA. Increased Cardiovascular Disease Risk in Veterans With Mental Illness. Circulation. Cardiovascular quality and outcomes. 2019 Oct 1; 12(10):e005563. [view]
  6. Hofer TP, Hayward RA. New Studies Do Not Challenge the American College of Cardiology/American Heart Association Lipid Guidelines. Annals of internal medicine. 2016 Dec 6; 165(11):827-828. [view]
  7. Sussman JB, Wiitala WL, Zawistowski M, Hofer TP, Bentley D, Hayward RA. The Veterans Affairs Cardiac Risk Score: Recalibrating the Atherosclerotic Cardiovascular Disease Score for Applied Use. Medical care. 2017 Sep 1; 55(9):864-870. [view]
Journal Other

  1. Hofer TP, Sussman JB, Hayward RA. New Studies Do Not Challenge the American College of Cardiology/American Heart Association Lipid Guidelines. Annals of internal medicine. 2016 May 17; 164(10):683-4. [view]
Conference Presentations

  1. Sussman JB, Wiitala WL, Zawistowski M, Hofer TP, Bentley DR, Vijan S, Hayward RA. The Veterans Affairs Cardiac Risk Score: Recalibrating the ASCVD Score for Applied Use. Paper presented at: Society of General Internal Medicine Annual Meeting; 2016 May 14; Hollywood, FL. [view]

DRA: Aging, Older Veterans' Health and Care, Health Systems, Cardiovascular Disease
DRE: Epidemiology, Treatment - Observational
Keywords: none
MeSH Terms: none

Questions about the HSR website? Email the Web Team

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.