Interpretability, credibility, and usability of hospital-specific template matching versus regression-based hospital performance assessments; a multiple methods study.

McGrath BM, Takamine L, Hogan CK, Hofer TP, Rosen AK, Sussman JB, Wiitala WL, Ryan AM, Prescott HC. Interpretability, credibility, and usability of hospital-specific template matching versus regression-based hospital performance assessments; a multiple methods study. BMC health services research. 2022 Jun 3; 22(1):739.

Related HSR&D Project(s)

• IIR 17-219 – Benchmarking Hospital Quality: Template Matching versus Conventional Regression Approaches

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

If you have VA-Intranet access, click here for more information vaww.hsrd.research.va.gov/dimensions/

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.    Search Dimensions for VA for this citation
* Don't have VA-internal network access or a VA email address? Try searching the free-to-the-public version of Dimensions

Search for Abstract from PubMed

---

Abstract:

BACKGROUND: Hospital-specific template matching (HS-TM) is a newer method of hospital performance assessment. OBJECTIVE: To assess the interpretability, credibility, and usability of HS-TM-based vs. regression-based performance assessments. RESEARCH DESIGN: We surveyed hospital leaders (January-May 2021) and completed follow-up semi-structured interviews. Surveys included four hypothetical performance assessment vignettes, with method (HS-TM, regression) and hospital mortality randomized. SUBJECTS: Nationwide Veterans Affairs Chiefs of Staff, Medicine, and Hospital Medicine. MEASURES: Correct interpretation; self-rated confidence in interpretation; and self-rated trust in assessment (via survey). Concerns about credibility and main uses (via thematic analysis of interview transcripts). RESULTS: In total, 84 participants completed 295 survey vignettes. Respondents correctly interpreted 81.8% HS-TM vs. 56.5% regression assessments, p? < 0.001. Respondents "trusted the results" for 70.9% HS-TM vs. 58.2% regression assessments, p? = 0.03. Nine concerns about credibility were identified: inadequate capture of case-mix and/or illness severity; inability to account for specialized programs (e.g., transplant center); comparison to geographically disparate hospitals; equating mortality with quality; lack of criterion standards; low power; comparison to dissimilar hospitals; generation of rankings; and lack of transparency. Five concerns were equally relevant to both methods, one more pertinent to HS-TM, and three more pertinent to regression. Assessments were mainly used to trigger further quality evaluation (a "check oil light") and motivate behavior change. CONCLUSIONS: HS-TM-based performance assessments were more interpretable and more credible to VA hospital leaders than regression-based assessments. However, leaders had a similar set of concerns related to credibility for both methods and felt both were best used as a screen for further evaluation.