1049 — Are Improvements in Measured Performance Driven by Better Treatment or "Denominator Management"?
Alex AH, VA Palo Alto; Chen C, VA Palo Alto; Rubinsky A, VA Palo Alto; Hoggatt K, VA Greater Los Angeles; Vanneman MV, VA Palo Alto;
In order to improve the quality of specialty substance use disorder (SUD) treatment, the US Department of Veteran Affairs (VA) in fiscal year (FY) 2003 included a Continuity of Care measure in executive performance plans. To qualify for the measure denominator, patients needed to attend three visits at an addiction treatment program. In order to meet the performance criterion (numerator), patients needed to attend two visits per month for three months after qualifying. Because performance measures, especially those tied to incentives and consequences, are intended to stimulate particular actions, the first goal of this study was to evaluate if the inclusion of this measure in executive performance plans resulted in increases in measured performance. Then, because performance measures can stimulate actions other than those intended, the second goal of this study was to examine if changes in performance were driven by unintended behaviors such as restricting who qualifies for the denominator by managing which patients received the qualifying dose of treatment.
We included all patients in all VA facilities between FY2000-FY2009. Four outcomes were calculated for each of 40 quarters straddling the inclusion of the measure in performance plans: 1) the actual performance measure - patients retained/patients qualified; 2) patients qualified/ patients diagnosed with SUD; 3) patients qualified/ total patients; 4) patients diagnosed with SUD/total patients. The interrupted time series design was analyzed with segmented regression.
Rate of performance on the performance measure increased significantly after the measure was tied to consequences (p < 0.01). However, the proportion of patients with SUD who qualified for the measure dropped significantly immediately after the measure was implemented (p = 0.03).
Changes in measured performance can be driven by intended or unintended actions. Measures may be particularly vulnerable to stimulating unintended actions when the criteria for denominator qualification are under the control of those being evaluated.
These results should motivate the development of measures that are less vulnerable to the problems described, and also the exploration of shadow measures to monitor and reduce unintended consequences.