1073 — Assessing the Validity of Selected Patient Safety Indicators (PSIs): Are the PSIs Ready to be Used as Hospital Performance Measures?
Shin MH (VA Boston Healthcare System, COLMR-COE), Borzecki AB
(Bedford VAMC, CHQOER-COE), Chen Q
(VA Boston Healthcare System, COLMR-COE), Loveland S
(VA Boston Healthcare System, COLMR-COE), Cevasco M
(VA Boston Healthcare System), Itani K
(VA Boston Healthcare System), Mull HJ
(VA Boston Healthcare System, COLMR-COE), Rosen AK
(VA Boston Healthcare System, COLMR-COE)
The Agency for Healthcare Research and Quality’s (AHRQ) Patient Safety Indicators (PSIs) screen for potentially preventable complications of inpatient care. Although designed as “indicators” rather than definitive measures, they are increasingly being used as performance measures. Thus, understanding their accuracy in detecting “true” safety events is important. As part of a comprehensive PSI validation study, we evaluated the positive predictive value (PPV) of 12 PSIs.
We identified flagged cases for each PSI from 28 representative hospitals by applying the AHRQ PSI software (v.3.1a) to FY03-07 VA administrative data. Using standardized tools and VistAWeb, nurse abstractors reviewed charts of flagged PSI cases. We calculated PPVs (i.e., # of true events/flagged cases), examined inter-rater reliability (IRR) between nurses, and reviewed reasons for false positive (FP) cases.
PPVs ranged from 30% (95% CI, 22-40%) (PSI #3, Pressure Ulcer [PU]) to 87% (95% CI, 79-92%) (PSI #14, Wound Dehiscence [WD]); median was 64%; IRR was > 90%. Common reasons for FPs included: conditions that were present on admission (POA), coding errors (e.g., for Pulmonary Embolism/Deep Vein Thrombosis, arterial thromboses were coded as venous), and coding algorithm limitations (e.g., for Hemorrhage/Hematoma, the code for hematoma drainage may also be used for abscess drainage). Three PSIs also specify excluding non-elective admissions. However, because VA data lacks an admission status variable, this resulted in a significant number of FPs (25-76%).
PSI algorithm and coding enhancements (e.g., adding POA and non-elective admission codes) would significantly increase the PPV of several PSIs. However, other FP causes, like coding errors, may not be as easily resolved without additional coder training. Based on our results, using the PSIs with lower PPVs for performance measurement seems premature. Further research is needed to improve the validity of specific PSIs and examine their utility for monitoring/tracking safety and quality improvement (QI).
Given the current intent to use specific PSIs for performance measurement, determination of whether these PSIs are appropriate for hospital reporting is critical. Some PSIs with low PPVs, such as PU, may be useful for QI or case-finding, while other PSIs with relatively high PPVs, like WD, may be appropriate for performance measurement.