3078 — Implementing Decision Theory in Evaluating Provider Performance
Pietz K, and Petersen LA, Houston HSR&D CoE; Draper D, University of California, Santa Cruz; Wang D, Reis B, Landrum CR, and Virani SS, Houston HSR&D CoE; Murawski J, and Bonner M, VISN 12; Woodard LD, Houston HSR&D CoE
Network administrators need accurate, easily understandable information on the performance of their medical care providers in order to make management decisions and provide meaningful feedback. Decision theory has been used in other fields to make choices based on possible consequences. We evaluated the application of decision theory together with a profiling methodology to assess providers’ performance in one VA Network.
We evaluated provider performance for blood pressure (BP) control among hypertensive patients over multiple quarters using a method developed in the UK for evaluating university performance. Risk adjustment variables were race, age, illness burden, number of medications, marital status, and VA priority group, coded as categorical variables. We compared a provider’s observed performance (proportion of patients with BP <140/90 mm Hg) to the expected performance based on the performance of all providers in the Network in the risk adjustment categories. The difference between observed and expected divided by the estimated standard error was compared to high and low thresholds to determine outlier status. Network administrators specified utilities for correct or incorrect outlier identification. Using Monte Carlo simulation, we found the optimal threshold to maximize the total utility by specifying providers whose unadjusted performance lies outside 80% of the normal distribution curve as true outliers. We also assessed specificity using null simulations with no quality differences.
We have assessed performance for 12 consecutive quarters. There were 53,584 hypertensive patients and 300 providers in the Network in the latest quarter. Using the optimized threshold, simulations showed that the average sensitivity was 92% in identifying high outliers and 79% for low outliers. The specificity was 98% for both high and low outliers.
This method of rating providers using risk adjustment reflects the relative weights that administrators place on different outcomes. Simulations showed that the methodology provides an accurate way to determine whether a provider’s performance is within acceptable limits.
We developed a methodology for quality assessment that incorporates decision theory utilities, determined in partnership with Network administrators. Such methods can be used for provider feedback and to guide quality improvement efforts that reflect Network priorities.