Health Services Research & Development

Veterans Crisis Line Badge
Go to the ORD website
Go to the QUERI website

IIR 17-219 – HSR&D Study

New | Current | Completed | DRA | DRE | Portfolios/Projects | Centers | Career Development Projects

IIR 17-219
Benchmarking Hospital Quality: Template Matching versus Conventional Regression Approaches
Hallie C Prescott MD MSc
VA Ann Arbor Healthcare System, Ann Arbor, MI
Ann Arbor, MI
Funding Period: March 2018 - August 2020

BACKGROUND/RATIONALE:
Identifying and remediating low-quality care is at the heart of systematic quality improvement, particularly in VA. However, cross-hospital comparisons to identify under-performing hospitals are limited by differences in patient case-mix and illness severity. Clinicians consider the current approach to benchmarking with conventional regression to be unfair, unclear, and unhelpful. While substantial resources are devoted to hospital benchmarking, the current return on this investment is limited because clinicians do not understand or trust the methods. The National Academy of Medicine has recently called for investing into the science of performance measurement and for increasing its transparency and validity. Template matching has been proposed as an alternative methodological approach to benchmarking that is fair, clear, and helpful. However, this new approach has never been tested outside of limited research settings.

OBJECTIVE(S):
To test the utility of template matching for comparing quality of care across VA's diverse acute care hospitals, this project will assess the feasibility, accuracy, and interpretability of this approach. Specifically, the project will: (A1) Feasibility: Develop and optimize two template matching approaches for comparing 30-day mortality across VA hospitals. (A2) Accuracy: Compare the ability of template matching versus conventional regression to correctly identify under-performing hospitals. (A3) Interpretability: Compare the interpretability and credibility of hospital performance data generated from template matching versus conventional regression models with clinical leaders.

METHODS:
Aim 1 will develop and refine the statistical methodology for benchmarking 30-day mortality in VA acute care hospitals using two template matching approaches: (1) a single template and (2) personalized templates for each hospital. Multiple statistical approaches to matching will be considered to achieve the fairest comparisons across hospitals. Aim 2 will use simulation and real patient data to measure when and why template matching and conventional regression approaches may yield discordant assessments of hospitals. Aim 3 will develop the presentation of template matching data, then survey VA Chiefs of Medicine to compare the interpretability and credibility of performance reports using template matching versus conventional regression for hospital benchmarking, with follow-up semi-structured interviews with a subset of Chiefs to further understand template matching performance reports from the perspective of Chiefs of Medicine.

FINDINGS/RESULTS:
We have developed initial SAS code necessary to run template matching using a single template, as well as the SAS code to automate the simulations in Aim 2. We have developed the data-generation process for 4 initial scenarios, in which we will test template matching versus conventional regression:
-Scenario #1: No differences in case-mix of quality of care across hospitals
-Scenario #2: Differences in case-mix, but not quality of care
-Scenario #3: Differences in quality of care, but not case-mix
-Scenario #4: Differences in both case-mix and quality of care.
Our preliminary findings indicate that template matching is equivalent to conventional regression for Scenario #1 (both methods have a low rate of false-positive outlier detection).

IMPACT:
The overarching goal of this proposal is to improve the care of hospitalized Veterans. This proposal will apply promising, new template matching approaches for benchmarking to the diverse VA healthcare system and complete a multi-faceted evaluation of these approaches. We will consider not only the feasibility and methodological rigor of template matching, but also the interpretability, credibility and accountability of the data. We expect that template matching will be feasible for benchmarking VA acute care hospitals, that it will identify under-performing hospitals at least as well as current benchmarking with conventional regression, and that it will be more interpretable and credible to end-users.

PUBLICATIONS:

Journal Articles

  1. Prescott HC, Carmichael AG, Langa KM, Gonzalez R, Iwashyna TJ. Paths into Sepsis: Trajectories of Presepsis Healthcare Use. Annals of the American Thoracic Society. 2019 Jan 1; 16(1):116-123.
  2. Haak BW, Prescott HC, Wiersinga WJ. Therapeutic Potential of the Gut Microbiota in the Prevention and Treatment of Sepsis. Frontiers in immunology. 2018 Sep 10; 9:2042.
  3. Prescott HC. Preventing Chronic Critical Illness and Rehospitalization: A Focus on Sepsis. Critical care clinics. 2018 Oct 1; 34(4):501-513.
  4. Royer S, Prescott HC. Reply to Azithromycin: Short Course with Long Duration. Journal of hospital medicine. 2018 Aug 1; 13(8):583.
  5. Hensley MK, Prescott HC. Bad Brains, Bad Outcomes: Acute Neurologic Dysfunction and Late Death After Sepsis. Critical care medicine. 2018 Jun 1; 46(6):1001-1002.
  6. Prescott HC, Cope TM, Gesten FC, Ledneva TA, Friedrich ME, Iwashyna TJ, Osborn TM, Seymour CW, Levy MM. Reporting of Sepsis Cases for Performance Measurement Versus for Reimbursement in New York State. Critical care medicine. 2018 May 1; 46(5):666-673.
  7. Liu VX, Escobar GJ, Chaudhary R, Prescott HC. Healthcare Utilization and Infection in the Week Prior to Sepsis Hospitalization. Critical care medicine. 2018 Apr 1; 46(4):513-516.
  8. Meyer N, Harhay MO, Small DS, Prescott HC, Bowles KH, Gaieski DF, Mikkelsen ME. Temporal Trends in Incidence, Sepsis-Related Mortality, and Hospital-Based Acute Care After Sepsis. Critical care medicine. 2018 Mar 1; 46(3):354-360.
  9. Prescott HC, Chang VW. Overweight or obese BMI is associated with earlier, but not later survival after common acute illnesses. BMC geriatrics. 2018 Feb 6; 18(1):42.
  10. Royer S, DeMerle KM, Dickson RP, Prescott HC. Shorter Versus Longer Courses of Antibiotics for Infection in Hospitalized Patients: A Systematic Review and Meta-Analysis. Journal of hospital medicine. 2018 May 1; 13(5):336-342.
  11. Prescott HC, Angus DC. Enhancing Recovery From Sepsis: A Review. JAMA. 2018 Jan 2; 319(1):62-75.
  12. Govindan S, Prescott HC. Quick Sequential Organ Failure Assessment: Illness Severity Indicator, Clinical Decision Support Tool, or Both?. Critical care medicine. 2017 Nov 1; 45(11):1947-1949.
  13. Prescott HC, Costa DK. Improving Long-Term Outcomes After Sepsis. Critical care clinics. 2018 Jan 1; 34(1):175-188.
Journal Other

  1. Prescott HC, Angus DC. Postsepsis Morbidity. JAMA. 2018 Jan 2; 319(1):91.
  2. Royer S, Prescott HC. Next Steps for Confirming Bronchoalveolar Lavage Amlyase as an Useful Biomarker for Ventilator-Associated Pneumonia. [Editorial]. Critical care medicine. 2018 Jan 1; 46(1):165-166.


DRA: none
DRE: Treatment - Observational, Research Infrastructure, TRL - Applied/Translational
Keywords: Information Management, Quality Indicators
MeSH Terms: none