HSR&D Home » Research » PPO 10-088 – HSR&D Study
Evaluation of Methods for Summarizing and Reporting Hospital Outcome Performance
Amresh D Hanchate, PhD
VA Boston Healthcare System Jamaica Plain Campus, Jamaica Plain, MA
Funding Period: January 2011 - December 2013
Crucial to the ongoing drive toward improving quality of inpatient care and outcomes is the ability to assess hospital performance. While VA has made great strides in automating and systematizing patient data, there has been little research in evaluating statistical methods that accurately identify excellent, average or poor performers.
Despite increased use of hospital profiling, there is concern regarding the appropriateness of the statistical profiling method used. We examine two commonly used methods: traditional logistic regression leading to observed to expected ratios and the random effects (RE) hierarchical logistic regression method used in the VA/CMS HospitalCompare program. The two methods result in different profiles of VA hospitals when examining 30-day mortality for acute myocardial infarction (AMI) patients. There is no gold standard for identifying which profiling method "gets it right". To evaluate the two methods, we used simulated data for which the true hospital performance was predetermined. Implications of findings for VA hospital profiling are assessed.
The simulated data consist of randomly generated measures of patient risk, hospital performance and patient dichotomous outcomes. We developed a series of simulated scenarios by incrementally varying hospital volumes and inter-hospital variation in performance (intra class correlation [ICC]). For each simulation we obtained standardized hospital risk-adjusted mortality rates (SMRs) using traditional and RE methods and classified hospitals into above-average, average and below-average SMR categories. These were contrasted with the true SMR category to calculate sensitivity, specificity and positive predictive values (PPV +/-).
Overall performance indicated higher discrimination from the traditional method, compared to the RE method, in identifying above-average SMR hospitals (c-statistic: 0.89 vs. 0.81, p-value<0.001) and below-average SMR hospitals (0.90 vs. 0.88, p-value=0.02). Both hospital volume and ICC were important factors in this performance differential. Traditional method indicates higher discrimination for the lowest quartile of hospitals by discharge volume; no such difference was noted for other volume quartiles. Similarly, for ICC of 1% and 2%, discrimination was higher for traditional method; this difference was negligible for ICC of 5% or greater.
To improve quality of hospital care we need valid methods for identifying the high and low performers. Our work provides a basis for evaluating current methods for identifying high and low performers after consideration of tradeoffs between sensitivity and specificity. It contributes to the growing literature on pros and cons of the current profiling methods.
External Links for this Project
NIH ReporterGrant Number: I01HX000404-01
Dimensions for VADimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.
If you have VA-Intranet access, click here for more information vaww.hsrd.research.va.gov/dimensions/
VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address. Search Dimensions for this project
DRA: Health Systems
DRE: Treatment - Observational, Research Infrastructure
MeSH Terms: none