BACKGROUND/RATIONALE:
Assessing provider and facility performance is an increasingly important activity for health care system administrators as they attempt to provide quality care in an era of budget constraints. Profiling, an analytic tool that uses epidemiological methods to evaluate and compare patterns of care, is the approach most commonly used, but few findings exist to guide the choice of either case-mix instrument or outlier identification method.

OBJECTIVE(S):
Our short-term objective is to provide a rigorous and exhaustive comparison of promising profiling methods across a set of policy-relevant measures, employing the most widely used and sophisticated case-mix instruments and outlier identification methods currently available. Our long-term objective is to help VHA policy makers determine the optimal procedure for identifying statistical outliers, i.e., facilities and VISNs that stand out from the majority, for purposes of both remediation and identifying exemplars.

METHODS:
The three main components of the study are profiling measures, case-mix instruments, and outlier identification methods. We will include five profiling measures (total cost, outpatient pharmacy cost, proportion of ischemic heart disease patients with LDL<100mg/dl, proportion of diabetic patients with HbgA1c<9%, and days to psychiatric readmission for psychiatric inpatients), four case-mix instruments (Adjusted Clinical Groups, Diagnostic Cost Groups, Chronic Illness and Disability Payment System, and the RxRisk-V), and two outlier identification methods (direct calculation and Bayesian hierarchical modeling). The study will examine possible outliers at the facility as well as at the VISN level using patient-level data. The study sample includes all VA users in FY 2001 for the total and pharmacy cost profiling measures. Subsamples of IHD patients, diabetes patients, and psychiatric inpatients will be identified from the full study sample for the LDL, HbgA1c and days to psychiatric readmissions profiling measures. The analysis will proceed in four steps: (1) Derive VA-specific weights for a set of commonly used case-mix instruments; (2) Determine which combination of case-mix instrument and profiling model best accounts for variations in the observed data using split-half validation; (3) Determine how alternative selections of case-mix instrument and profiling model impact outlier identification; and (4) Determine the stability of outlier status over time.

FINDINGS/RESULTS:
1) When profiling of LDL and HbgA1c, neither choice of case mix model nor statistical model specification substantively impact facility profiles. However, choice of "cut point" (e.g., LDL of 70 vs 100 vs 130 vs 160) does result in significant numbers of facilities having their outlier status change.

2) When profiling outpatient cost using frequentist and Bayesian hierarchical and non-hierarchical models, model choice had minimal impact on the assessment facility performance. However, addition of facility-level information to the hierarchical models resulted in very substantial differences in facility performance. We surmise that inclusion of appropriate and complete information is more important than model specification, for profiling applications.

IMPACT:
Given the substantial gap between profiling theory and current administrative practice, it seems reasonable to examine (1) how wide is this gap and (2) whether currently available technology can narrow (if not bridge) the gap. Our approach is to rigorously and systematically study five profiling measures: total cost, outpatient pharmacy cost, low density lipoprotein (LDL) control in the secondary prevention of ischemic heart disease (IHD), hemoglobin A1c (HgbA1c) control in the secondary prevention of diabetes mellitus, and days to psychiatric readmission. All measures are of high relevance to the national VA mission through their cost and/or clinical consequences, so valid and reliable profiling techniques would improve the evaluation of health care services and resource needs both within the VA as well as between VA and non-VA providers and facilities.

Even findings indicating irresolvable inconsistencies in profiling results would be useful to both policy-makers and researchers. Understanding the current limits of profiling methods would help policy-makers avoid redirecting resources based on unreliable profiling results. Researchers would also benefit by being able to direct their efforts toward developing improved case-mix assessment tools and/or statistical methods rather than studying spuriously identified “high quality” programs.

---

External Links for this Project

---

PUBLICATIONS:

Journal Articles

1. Sloan KL, Montez-Rath ME, Spiro A, Christiansen CL, Loveland S, Shokeen P, Herz L, Eisen S, Breckenridge JN, Rosen AK. Development and validation of a psychiatric case-mix system. Medical care. 2006 Jun 1; 44(6):568-80. [view]

Conference Presentations

1. Zhou XH. The impact of Case-mix Adjustment Methodology on Outlier Identification. Paper presented at: Health Policy Research Annual International Conference; 2003 Oct 17; Chicago, IL. [view]
2. Sloan K, Burgess J, Zhou C, Fishman P, Zhou X, Wang L. Where Best to Invest? Model Information vs. Model Selection when Evaluating Facility Performance. Paper presented at: VA HSR&D National Meeting; 2006 Feb 15; Arlington, VA. [view]

DRA: Health Systems
DRE: none
Keywords: Implementation, Quality assessment, Risk adjustment
MeSH Terms: none