Benefit-Of-The-Doubt Approaches for Calculating a Composite Measure of Quality

Shwartz M, Burgess JF, Berlowitz D. Benefit-Of-The-Doubt Approaches for Calculating a Composite Measure of Quality. Poster session presented at: AcademyHealth Annual Research Meeting; 2010 Jun 28; Boston, MA.

Related HSR&D Project(s)

• IIR 06-260 – Identifying and Characterizing High Performing VHA Nursing Homes

Research Objectives: Standard approaches for determining weights when calculating a composite measure of health care quality from individual quality indicators (QIs) include equal weighting, denominator-based weights, and judgment-based weights. Benefit-of-the-doubt (BOD) approaches have not been used in the health services area, though they have been used to calculate composite measures for profiling countries. Underlying these approaches is the assumption that relative performance on a set of indicators is, at least to some extent, a revealed preference by the organization about the relative importance of the indicators. A BOD approach recognizes these revealed preferences by assigning higher weights to indicators on which performance is better and lower weights to indicators on which performance is poorer. Specifically, weights are chosen to optimize the composite measure subject to a set of constraints. BOD weights have been shown to be Nash equilibria in an evaluation game between a regulator and an organization. Our objective is to describe two BOD approaches for calculating a composite measure of nursing home quality from retrospective data and to compare facility rankings from these approaches to rankings when standard approaches are used. Study Design: Data on the following 5 risk-adjusted QIs were used: pressure ulcer development, functional decline, behavioral decline, mortality and preventable hospitalizations. For each QI for each nursing home, we know the expected number of cases from a risk-adjustment model and the observed number of cases and can thus calculate the ratio of observed to expected cases (O/E ratio). We use several standard approaches for calculating a composite measure for each nursing home from the 5 individual O/E ratios: facility-specific denominator-based weights, overall denominator-based weights, and equal weighting. We then use two BOD approaches: simple linear programming (LP) models and data envelopment analysis (DEA). In both BOD approaches, constraints are added to limit adjustments of the weights to some percentage of overall denominator-based weights. Population Studied: The data were originally collected in 1998 from 32 Department of Veterans Affairs (VA) nursing homes selected to represent a balanced sample of different sizes, locations and quality of care. Principal Findings: If weights for the BOD approaches are constrained to be within + 75% of overall denominator-based weights, BOD approaches identified approximately the same high performing facilities as the standard approaches. Five of the 6 top-ranked facilities using overall denominator-based weights were ranked in the top 6 by the BOD approaches. BOD approaches have more of an impact on which facilities were identified as poor performers. Only 3 of 6 bottom-ranked facilities using overall denominator-based weights were ranked in the bottom 6 using BOD approaches. Conclusions: There are a variety of legitimate local context reasons related to a facility's patients, workforce, environment and strategic priorities that might partially explain its relative performance on a particular set of QIs. BOD approaches take these types of factors into account by allowing adjustments to policy-determined baseline weights. Implications for Policy, Delivery or Practice: BOD approaches should be attractive to providers and, in a collaborative environment, also to policy makers. Their use may increase acceptance of composite measures of quality.