Validating the Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators: How Well Do They Identify True Safety Events?

Ensuring patient safety,"freedom from accidental injury caused by medical care," is a high priority for the VA. A formal patient safety program was launched in 1997, promulgating the VA as a national leader in patient safety. This program included the establishment of the National Center for Patient Safety, developed to advance patient safety measurement and quality improvement (www.patientsafety.gov).

Despite the numerous patient safety initiatives underway, the VA does not yet have a national system in place for tracking patient safety events. The Patient Safety Indicators (PSIs), developed by the Agency for Healthcare Research and Quality (AHRQ) and released in 2003, represent a substantial contribution to the scientific detection of patient safety events. The PSIs are an evidence-based method designed to screen for potentially preventable adverse events that occur in the inpatient setting, such as complications following surgeries, other procedures, and some medical care. The PSIs are based on ICD-9-CM diagnosis and procedure codes obtained from administrative discharge data, making them readily available, cost-efficient, and easy to use. However, because they are based on administrative data (known for its coding variability), concerns related to their validity and reliability as patient safety "outcomes" are prevalent. Moreover, these concerns have heightened with increasing emphasis on the use of the PSIs for public reporting and pay-for-performance, rather than for how they were originally intended, for quality improvement and case-finding activities. The Centers for Medicare and Medicaid Services (CMS) currently posts six individual PSIs and a PSI composite on their Hospital Compare website and will be tracking these through their annual payment program (www.cms.gov/HospitalQualityInits/11_HospitalCompare.asp). It is only a matter of time before the VA follows suit.

PSIs Examined by Study Recognizing the need to understand whether the PSIs identify true events, the VA HSR&D Service funded a study to examine the validity of the PSIs: "Validating the Patient Safety Indicators in the VA: A Multi-Faceted Approach" (SDR-07-002). One important component of this four-year study (2003-2007) was to assess the criterion validity of the PSIs (i.e., do cases flagged by the PSI algorithm represent true events based on medical record review, "the gold standard"?). We selected 12 of the 20 hospital-level PSIs for study based on their relevance to the VA population, observed VA rates, and their potential preventability (see Table).

Our 28 hospitals were drawn from a nationally representative sample of VA acute-care hospitals, selected based on individual PSI counts, PSI composite rates, and geographic distribution. From each hospital, we randomly selected four PSI-flagged medical records for medical record abstraction (112 cases per PSI). Two nurse-abstractors reviewed electronic medical records (EMRs) using standardized data abstraction instruments and guidelines based on AHRQ-developed tools or developed de novo. The EMR was reviewed for the occurrence of a safety-related event; patient clinical characteristics; clinical processes related to the event; and patient outcomes. We examined inter-rater reliability between the nurses, setting a target of 90 percent agreement.

To assess criterion validity, we calculated the positive predictive validity (PPV) of each PSI and associated 95 percent Confidence Intervals (CIs). PPV was calculated by dividing the number of true positives (TPs) by the number of flagged cases. We also examined false positives (FPs) to determine why they were flagged and how the PSI might be improved.

PPVs for selected PSIs varied considerably, ranging from a low of 28 percent for Postoperative Hip Fracture to a high of 87 percent for Postoperative Wound Dehiscence. PPVs for the other PSIs were relatively moderate, ranging from 43 percent for PE/DVT to 75 percent for Postoperative Hemorrhage/Hematoma. Of the nine surgical PSIs, seven PPVs were >60 percent, demonstrating better predictive validity than the medical PSIs. A common reason for false positives included conditions that were present-on-admission. For example, among false positive cases of Pressure Ulcer, 83 percent were present-on-admission. Another primary cause for false positives was misidentification of non-elective admissions as elective.1

Given the relatively moderate PPVs found, we recommend that the PSIs should continue to be used primarily for quality improvement and case-finding activities. The possible exceptions to this are some of the surgical PSIs (those with the highest PPVs); however, even these should be used only for public reporting,given that coding revisions are needed to improve most of the indicators. Nonetheless, the PSIs are a step in the right direction; it is important to understand what the indicators detect and where the greatest opportunities for quality improvement and case finding lie.

  1. Rosen, A.K. et al. "Validating the Patient Safety Indicators (PSIs) in the Veterans Health Administration: Do They Accurately Identify True Safety Events?" Medical Care 2012; 50(1):74-85.