Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

VA Health Systems Research

Go to the VA ORD website
Go to the QUERI website

PPO 10-088 – HSR Study

PPO 10-088
Evaluation of Methods for Summarizing and Reporting Hospital Outcome Performance
Amresh D Hanchate, PhD
VA Boston Healthcare System Jamaica Plain Campus, Jamaica Plain, MA
Boston, MA
Funding Period: January 2011 - December 2013
Crucial to the ongoing drive toward improving quality of inpatient care and outcomes is the ability to assess hospital performance. While VA has made great strides in automating and systematizing patient data, there has been little research in evaluating statistical methods that accurately identify excellent, average or poor performers.

Despite increased use of hospital profiling, there is concern regarding the appropriateness of the statistical profiling method used. We examine two commonly used methods: traditional logistic regression leading to observed to expected ratios and the random effects (RE) hierarchical logistic regression method used in the VA/CMS HospitalCompare program. The two methods result in different profiles of VA hospitals when examining 30-day mortality for acute myocardial infarction (AMI) patients. There is no gold standard for identifying which profiling method "gets it right". To evaluate the two methods, we used simulated data for which the true hospital performance was predetermined. Implications of findings for VA hospital profiling are assessed.

The simulated data consist of randomly generated measures of patient risk, hospital performance and patient dichotomous outcomes. We developed a series of simulated scenarios by incrementally varying hospital volumes and inter-hospital variation in performance (intra class correlation [ICC]). For each simulation we obtained standardized hospital risk-adjusted mortality rates (SMRs) using traditional and RE methods and classified hospitals into above-average, average and below-average SMR categories. These were contrasted with the true SMR category to calculate sensitivity, specificity and positive predictive values (PPV +/-).

Overall performance indicated higher discrimination from the traditional method, compared to the RE method, in identifying above-average SMR hospitals (c-statistic: 0.89 vs. 0.81, p-value<0.001) and below-average SMR hospitals (0.90 vs. 0.88, p-value=0.02). Both hospital volume and ICC were important factors in this performance differential. Traditional method indicates higher discrimination for the lowest quartile of hospitals by discharge volume; no such difference was noted for other volume quartiles. Similarly, for ICC of 1% and 2%, discrimination was higher for traditional method; this difference was negligible for ICC of 5% or greater.

To improve quality of hospital care we need valid methods for identifying the high and low performers. Our work provides a basis for evaluating current methods for identifying high and low performers after consideration of tradeoffs between sensitivity and specificity. It contributes to the growing literature on pros and cons of the current profiling methods.

External Links for this Project

NIH Reporter

Grant Number: I01HX000404-01

Dimensions for VA

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

Learn more about Dimensions for VA.

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.
    Search Dimensions for this project


Journal Articles

  1. Hanchate AD, Ash AS, Borzecki A, Abdulkerim H, Stolzmann KL, Rosen AK, Fink AS, Pugh MJ, Shokeen P, Shwartz M. How pooling fragmented healthcare encounter data affects hospital profiling. The American journal of managed care. 2015 Feb 1; 21(2):129-38. [view]
Conference Presentations

  1. Hanchate AD. Evaluation of Statistical Methods for Profiling Hospital Outcome Performance: The Case of Acute Myocardial Infarction 30-day Mortality. Poster session presented at: AcademyHealth Annual Research Meeting; 2012 Jun 24; Orlando, FL. [view]
  2. Hanchate AD. Evaluation of Statistical Methods for Profiling Hospital Outcome Performance: The Case of Acute Myocardial Infarction 30-day Mortality. Paper presented at: VA HSR&D / QUERI National Meeting; 2012 Jun 16; National Harbor, MD. [view]

DRA: Health Systems
DRE: Research Infrastructure, Treatment - Observational
Keywords: none
MeSH Terms: none

Questions about the HSR website? Email the Web Team

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.