3101 — AHRQ’s Patient Safety Indicators in Nine Years of VA Inpatient Data: Low Reliability Limits their Use as Performance Measures
West AN (White River Junction VAMC) , Weeks WB
(White River Junction VAMC), Bagian JP
(VA National Center for Patient Safety)
Recently there has been greater attention to establishing national quality standards in healthcare. The Agency for Healthcare Research & Quality has developed several Patient Safety Indicators (PSIs) to estimate rates of a broad variety of medical/surgical adverse events in inpatient care. The PSIs are intended to yield reliable comparisons among hospitals to promote higher quality throughout healthcare systems. These adverse events, however, are rare, some much more so than others, and the reliability of PSI rates has been little tested.
Using nine years (1997-2005) of hospital discharge data for the entire VA system, we tested the reliabilities of PSIs in several ways, relating our findings to two variables: the average annual number of adverse events that each PSI represents, and whether the PSI rate is calculated for individual hospitals or hospital networks (VISNs). We calculated one-year lagged correlations to test the stability of rank orders among hospitals or VISNs. We separated hospitals or VISNs into highest, middle, or lowest thirds on each PSI for each year, and counted those that were consistently in the extreme thirds across years. Using 2001 as a reference year, we calculated the percentages that were in the same thirds in other years. We also assessed time trends in PSI rates for the entire VA as well as by VISN or hospital.
We found that PSI rates, and quality groups based on them, were more stable for VISNs than for individual hospitals, but primarily only for the most common PSIs, i.e., those that reflect the highest numbers of adverse events. Only the most frequent PSIs yielded significant time trends, for the whole VA or for VISNs, but at no better than chance for hospitals.
Most PSIs are so rare that they are not reliable as performance measures, particularly for comparisons among hospitals. Reliability is better for the most frequent PSIs when applied to data aggregated for large healthcare systems.
The need for large patient samples from multiple hospitals severely limits the PSIs’ clinical utility, as well as their potential value to managers seeking to improve quality or to patients seeking optimal care.