» Back to Table of Contents
Ensuring patient safety,"freedom from accidental injury caused by medical care," is a high priority for the VA. A formal patient safety program was launched in 1997, promulgating the VA as a national
leader in patient safety. This program included the establishment of the National Center for Patient Safety, developed to advance patient safety
measurement and quality improvement (www.patientsafety.gov).
Despite the numerous patient safety initiatives underway, the VA does not yet have a national system in place for tracking patient safety events. The
Patient Safety Indicators (PSIs), developed by the Agency for Healthcare Research and Quality (AHRQ) and released in 2003, represent a substantial
contribution to the scientific detection
of patient safety events. The PSIs are an evidence-based method designed to screen for potentially preventable adverse events that occur in the inpatient
setting, such as complications following surgeries, other procedures, and some medical care. The PSIs are based on ICD-9-CM diagnosis and procedure codes
obtained from administrative discharge data, making them readily
available, cost-efficient, and easy to use. However,
because they are based on administrative data (known for its coding variability), concerns related to their validity and reliability as patient safety
"outcomes" are prevalent. Moreover, these concerns have heightened with increasing
emphasis on the use of the PSIs for public reporting and pay-for-performance, rather than for how they were originally intended, for quality improvement
and case-finding activities. The Centers for Medicare and Medicaid Services (CMS) currently posts six individual PSIs and a PSI composite on their Hospital
Compare website and will be tracking these through their annual payment program (www.cms.gov/HospitalQualityInits/11_HospitalCompare.asp). It is only a matter of time before the VA follows suit.
Recognizing the need to understand whether the PSIs identify true events, the VA HSR&D Service funded a study to examine the validity of the PSIs:
"Validating the Patient Safety Indicators
in the VA: A Multi-Faceted Approach" (SDR-07-002). One important component of this four-year study (2003-2007) was to assess the criterion validity of the
PSIs (i.e., do cases flagged by the PSI algorithm represent true events based on medical record review, "the gold standard"?). We selected 12 of the 20
hospital-level PSIs for study based on their relevance to the VA population, observed VA rates, and their potential preventability (see Table).
Our 28 hospitals were drawn from a nationally representative sample of VA acute-care hospitals,
selected based on individual PSI counts, PSI composite rates, and geographic distribution.
From each hospital, we randomly selected four PSI-flagged medical records for medical record abstraction (112 cases per PSI). Two nurse-abstractors
reviewed electronic medical records (EMRs) using standardized data abstraction
instruments and guidelines based on AHRQ-developed tools or developed de novo. The EMR was reviewed for the occurrence of a safety-related event; patient
clinical processes related to the event; and patient outcomes. We examined inter-rater reliability
between the nurses, setting a target of 90 percent agreement.
To assess criterion validity, we calculated the positive predictive validity (PPV) of each PSI and associated 95 percent Confidence Intervals (CIs). PPV
was calculated by dividing the number
of true positives (TPs) by the number of flagged cases. We also examined false positives (FPs) to determine why they were flagged and how the PSI might be
PPVs for selected PSIs varied considerably, ranging from a low of 28 percent for Postoperative
Hip Fracture to a high of 87 percent for Postoperative Wound Dehiscence. PPVs for the other PSIs were relatively moderate, ranging from 43 percent for
PE/DVT to 75 percent for Postoperative Hemorrhage/Hematoma. Of the nine surgical PSIs, seven PPVs were >60 percent, demonstrating better predictive
validity than the medical PSIs. A common reason for false positives included conditions that were present-on-admission. For example, among false positive cases of
Pressure Ulcer, 83 percent were present-on-admission. Another primary cause for false positives was misidentification of non-elective admissions as
Given the relatively moderate PPVs found, we recommend that the PSIs should continue to be used primarily for quality improvement and case-finding activities. The possible exceptions to this are some of the surgical PSIs (those with the highest PPVs); however, even these should be used only for public reporting,given that coding revisions are needed to improve most of the indicators. Nonetheless, the PSIs are a step in the right direction; it is important to understand what the indicators detect and where the greatest opportunities for quality improvement and case finding lie.
Rosen, A.K. et al. "Validating the Patient Safety Indicators (PSIs) in the Veterans Health Administration: Do They Accurately Identify True Safety Events?" Medical Care 2012; 50(1):74-85.