Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

VA Health Systems Research

Go to the VA ORD website
Go to the QUERI website

IIR 17-219 – HSR Study

IIR 17-219
Benchmarking Hospital Quality: Template Matching versus Conventional Regression Approaches
Hallie C Prescott, MD MSc
VA Ann Arbor Healthcare System, Ann Arbor, MI
Ann Arbor, MI
Funding Period: March 2018 - February 2022
Identifying and remediating low-quality care is at the heart of systematic quality improvement, particularly in VA. However, cross-hospital comparisons to identify under-performing hospitals are limited by differences in patient case-mix and illness severity. Clinicians consider the current approach to benchmarking with conventional regression to be unfair, unclear, and unhelpful. While substantial resources are devoted to hospital benchmarking, the current return on this investment is limited because clinicians do not understand or trust the methods. The National Academy of Medicine has recently called for investing into the science of performance measurement and for increasing its transparency and validity. Template matching has been proposed as an alternative methodological approach to benchmarking that is fair, clear, and helpful. However, this new approach has never been tested outside of limited research settings.

To test the utility of template matching for comparing quality of care across VA's diverse acute care hospitals, this project will assess the feasibility, accuracy, and interpretability of this approach. Specifically, the project will: (A1) Feasibility: Develop and optimize two template matching approaches for comparing 30-day mortality across VA hospitals. (A2) Accuracy: Compare the ability of template matching versus conventional regression to correctly identify under-performing hospitals. (A3) Interpretability: Compare the interpretability and credibility of hospital performance data generated from template matching versus conventional regression models with clinical leaders.

Aim 1 will develop and refine the statistical methodology for benchmarking 30-day mortality in VA acute care hospitals using two template matching approaches: (1) a single template and (2) personalized templates for each hospital. Multiple statistical approaches to matching will be considered to achieve the fairest comparisons across hospitals. Aim 2 will use simulation and real patient data to measure when and why template matching and conventional regression approaches may yield discordant assessments of hospitals. Aim 3 will develop the presentation of template matching data, then survey VA Chiefs of Medicine to compare the interpretability and credibility of performance reports using template matching versus conventional regression for hospital benchmarking, with follow-up semi-structured interviews with a subset of Chiefs to further understand template matching performance reports from the perspective of Chiefs of Medicine.

We have developed initial SAS code necessary to run template matching using a single template, as well as the SAS code to automate the simulations in Aim 2. We have developed the data-generation process for 4 initial scenarios, in which we will test template matching versus conventional regression:
-Scenario #1: No differences in case-mix of quality of care across hospitals
-Scenario #2: Differences in case-mix, but not quality of care
-Scenario #3: Differences in quality of care, but not case-mix
-Scenario #4: Differences in both case-mix and quality of care.
Our preliminary findings indicate that template matching is equivalent to conventional regression for Scenario #1 (both methods have a low rate of false-positive outlier detection).

The overarching goal of this proposal is to improve the care of hospitalized Veterans. This proposal will apply promising, new template matching approaches for benchmarking to the diverse VA healthcare system and complete a multi-faceted evaluation of these approaches. We will consider not only the feasibility and methodological rigor of template matching, but also the interpretability, credibility and accountability of the data. We expect that template matching will be feasible for benchmarking VA acute care hospitals, that it will identify under-performing hospitals at least as well as current benchmarking with conventional regression, and that it will be more interpretable and credible to end-users.

External Links for this Project

NIH Reporter

Grant Number: I01HX002546-01

Dimensions for VA

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

Learn more about Dimensions for VA.

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.
    Search Dimensions for this project


None at this time.

DRA: Health Systems
DRE: Treatment - Observational, Research Infrastructure, TRL - Applied/Translational
Keywords: Information Management, Quality Indicators
MeSH Terms: none

Questions about the HSR website? Email the Web Team

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.