Health Services Research & Development

Veterans Crisis Line Badge
Go to the ORD website
Go to the QUERI website

2006 HSR&D National Meeting Abstract


1051 — Where Best to Invest? Model Information vs. Model Selection when Evaluating Facility Performance

Author List:
Sloan KL (VA Puget Sound HCS)
Burgess JF (VA Management Science Group)
Zhou C (Vanderbilt University)
Fishman P (Group Health Cooperative)
Zhou XH (VA Puget Sound HCS)
Wang L (VA Puget Sound HCS)

Objectives:
Profiling techniques promise to provide valid provider performance evaluation results by taking into account differential patient populations. However, little attention in the literature has been paid to the trade-offs involved in investing in more sophisticated statistical models vs. gathering of salient provider-specific information and their impact on the results of provider performance evaluation. Furthermore, although Bayesian and hierarchical approaches to risk adjustment expand analytic options for a proper leveling of the playing field; such approaches also require careful consideration of the policy consequences of the decisions made when choosing models.

Methods:
In order to illustrate how model and information selection affect risk adjusted outcomes, we examine variations in outpatient cost for 144 hospitals in the U.S. Veterans Health Administration system. We take as our starting point results from the standard, frequentist profiling of outpatient cost. We then consider the hierarchical arrangement of patients into facilities, apply alternative analytic methods, and compare which facilities are flagged as cost outliers. In a stepwise fashion we examine how increasingly elaborate models impact the assessment of facility performance. Contrasts include non-hierarchical vs. hierarchical models, frequentist vs. Bayesian techniques, and ignoring vs. including facility-level information, all interpreted from a healthcare management policy perspective.

Results:
We demonstrate that the choice of information to include is significantly more important than choice of statistical model specification. Specifically, failure to include relevant facility-level information substantively affects the profiling results (both point estimators and confidence intervals).

Implications:
Although there exist computational and explanatory costs associated with more complex model specifications, hierarchical models including relevant facility-specific information appear necessary in order to adequately understand inter-facility performance variations regarding outpatient cost. Furthermore, inclusion of this information better allows one to respond to criticism related to failure to account for special characteristics of poorly performing programs.

Impacts:
As performance evaluation becomes increasingly important and “Performance Pay” is now explicitly part of the physician reimbursement formula, a clear understanding of how performance drivers at various system levels influence outcomes is necessary in devising rational and fair reimbursement formulae. Failure to appropriately include relevant facility-specific information in performance evaluation risks drawing highly inaccurate conclusions.