3084 — Validating the Patient Safety Indicators (PSIs): Do They Reflect True Safety Events?
Rosen AK (Bedford COE), Borzecki A
(Bedford COE), Kaafarani H
(Boston COE), Hanchate A
(Boston University), Loveland S
(Bedford COE), Hartmann C
(Bedford COE), Shwartz M
(Boston COE), Shin M
(Bedford COE), Shimada S
(Bedford COE), Rivard P
Accurate measurement of patient safety-related events is challenging given their relative infrequency. The AHRQ Patient Safety Indicators (PSIs) use hospital administrative data to screen for potentially preventable complications, such as postoperative pulmonary embolism or deep vein thrombosis (PE/DVT) and iatrogenic pneumothorax (IP). Since there is little evidence that PSIs measure “true” safety events, we examined their criterion validity through chart review of flagged PSI cases. Site visits also will be conducted to examine attributional validity (i.e., the association between PSI rates and processes/structure of care). This multi-faceted approach will elicit important information regarding which PSIs should be used as performance measures and which should be used only as case-finding/screening tools.
We applied the AHRQ PSI software to FY03-FY07 PTF files to obtain risk-adjusted PSI rates. We conducted a retrospective chart review of flagged PSI cases using standardized abstraction tools. We examined inter-rater reliability between nurses and calculated rates of true and false positives.
Two PSIs (PE/DVT and IP) have been validated to date. Of 112 cases abstracted for PE/DVT, 49 were true events, yielding a positive predictive value (PPV) of 43.8% (95% CI, 34.4-53.4%). PPV for IP was higher (50%, 95% CI, 35.5%-64.5%). Inter-rater reliability was > 90% for both PSIs. Various coding-related issues accounted for most of the false positives for both PSIs. However, for IP most of the false positives resulted from IPs following percutaneous lung biopsies that are meant to be excluded per the PSI algorithm.
Improving PSI algorithms and expanding the codes used should increase the predictive value of the PSIs. PPV results suggest that with some relatively minor modifications (e.g., better procedure coding for lung biopsy), IP may be ready for use in performance measurement, whereas PE/DVT may be better suited as a screening tool. Site visit information should provide additional insights on how healthcare systems can use the PSIs. We are continuing to validate other PSIs and proceed with site visits.
As the PSIs are increasingly being used for hospital profiling and pay-for-performance, evaluating their accuracy is critical for determining which ones are most appropriate for these purposes.