BACKGROUND/RATIONALE:
Because the Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) are regarded as "screening tools" of potential patient safety events rather than as definitive measures, increasing use of these indicators for public reporting and pay-for-performance makes it critical that the PSIs measure true safety events and accurately reflect hospital safety performance.


OBJECTIVE(S):
1) Investigate the criterion validity of the PSIs (specifically, the positive predictive value, or PPV) by review of the VA's electronic medical record (EMR); 2) investigate the attributional validity of the PSIs by identifying explicit processes and structures of care associated with PSIs; 3) revise and improve the PSIs using multiple data sources and settings of care; and 4) assess the utility validity of the PSIs for quality improvement (QI) and performance measurement.

METHODS:
We identified PSI-flagged cases from 28 representative hospitals by applying the AHRQ PSI software (v.3.1a) to FY2003-2007 hospital discharge data. To examine criterion validity, we selected 12/17 PSIs; nurse-abstractors used standardized tools to review a random sample of flagged medical records (112 records per PSI) for the presence of true safety events. To examine attributional validity, we matched flagged cases with non-flagged cases for 3 PSIs (Postoperative Pulmonary Embolism/Deep Vein Thrombosis (PE/DVT), Postoperative Wound Dehiscence (PWD), and Accidental Puncture or Laceration (APL)) to examine chart-based process of care information; we also conducted site visits to determine whether facility-level practices were associated with facility-level variation in PSI Composite Score. We identified additional PSI events using multiple data sources by merging inpatient databases with 30-day post-discharge inpatient/outpatient data and VA Surgical Quality Improvement Program (VASQIP) data. As a final step, we collaborated with the National Center for Patient Safety (NCPS) on a QI project in 16 hospitals using the Virtual Breakthrough Series (VBTS) process to examine the utility validity of the PSI, Postoperative Respiratory Failure (PRF). We also conducted a formative evaluation (FE) with potential end-users to assess utility.

FINDINGS/RESULTS:
1) PPVs ranged from 28% (95% CI, 15-43%) for Postoperative Hip Fracture to 87% (95% CI, 79-92%) for PWD. Common reasons for false positives included conditions that were present-on-admission (POA), coding errors, and lack of coding specificity.

2) Among the 116 PE/DVT case-control pairs, we found similarities in terms of demographics, surgery type, and appropriate pharmaco-prophylaxis rates overall (62% vs. 72%; p=0.13; effect size=0.15). However, among the highest-risk patients (hip/knee replacements and cancer patients), controls were slightly more likely to receive appropriate pharmaco-prohylaxis (results not significant). We found no differences in processes of care between each of the 95 case-control pairs for PWD and APL, minor differences that were found were due to patients' underlying severity rather than quality of care shortfalls. Documentation of processes was frequently missing in EMRs. Site-visit analysis revealed moderate association between PSI Composite Scores and specific domains of structure/process, such as leadership, communication, and coordination. For example, high-performers (i.e., sites with low PSI Composite) had leaders that were very engaged in patient safety.

3) Applying modified PSI algorithms to post-discharge data yielded an additional 11,077 safety events among PSI-eligible cases over the 5-year period. Most cases were flagged within 14-days post-discharge. Using VASQIP data as the "gold standard" for 5 selected surgical PSIs that matched with VASQIP outcomes, we found that PSI sensitivity ranged from 20-68%, specificity from 99.1-99.8%, and PPV from 31-72%. Differences in adverse event (AE) definitions explained disagreement in safety event detection for all PSI-VASQIP comparisons, except PWD. Each of the PSIs failed to flag some AEs detected by VASQIP because of coding errors, particularly for PE/DVT and PWD. There were also many VASQIP-only (between 5-75% of the chart review sample) and PSI-only (between 69-100% of the chart review sample) cases that did not have discernible differences in AE definition or evidence of a coding error.

4) Teams (consisting of at least one surgeon or ICU specialist, coder, and nurse) actively participated in the VBTS with the VBTS faculty (the PSI team and NCPS staff) over the course of the 8-month QI project. Commonly implemented interventions to prevent PRF included nursing-related interventions, such as improving processes related to incentive spirometer use, and standardized physician orders for the Ventilator Associated Pneumonia bundle. Some teams reported reduced ventilator days and ICU readmissions for PRF. FE interviewees indicated interest in learning about using the PSIs: 1) to better target and supplement current safety measures and QI methods, and 2) for case-finding and QI.

IMPACT:
Although it is well-known that administrative data are subject to coding inaccuracies and variation across facilities, their use for identifying potential patient safety events is a significant advance over traditional methods for identifying safety events, such as incident reporting, root cause analyses, and expensive, resource-intensive chart review. The PSIs have high specificity, moderate to good sensitivity compared to VASQIP chart-review data, and moderate positive predictive validity using nurse-abstractor chart-reviewed data. Once the VA implements POA data and integrates it into the hospital discharge data, the PPVs of several PSIs should increase. Our results suggest that improving the coding of data as well as specific coding practices related to AE reporting, through education and training of medical coders, would also improve the overall validity of the PSIs. Interestingly, we did not detect significant differences in process of care information between case-control pairs when reviewing medical records; however, site visit results suggest that there appears to be an association between PSI Composite and certain domains of quality (e.g., leadership). Additional methods, such as direct observation of clinical practices, might be useful in clarifying the relationship between PSIs and processes of care. Finally, given that selected PSI rates are currently reported on the CMS Hospital Compare Website for private sector hospitals, and that VA PSI rates will likely be posted in the near future, QI interventions, such as VBTS programs, should be implemented to reduce the rates of all PSIs, not just PRF. A multi-faceted approach to validating the PSIs in the VA is needed to continue our efforts to revise and improve the PSIs.

---

External Links for this Project

---

PUBLICATIONS:

Journal Articles

1. Sullivan JL, Rivard PE, Shin MH, Rosen AK. Applying the High Reliability Health Care Maturity Model to Assess Hospital Performance: A VA Case Study. Joint Commission Journal on Quality and Patient Safety. 2016 Sep 1; 42(9):389-411. [view]
2. Chen Q, Hanchate A, Shwartz M, Borzecki AM, Mull HJ, Shin MH, Rosen AK. Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicator Rates Among Veteran Dual Users. American journal of medical quality : the official journal of the American College of Medical Quality. 2014 Jul 1; 29(4):335-43. [view]
3. Utter GH, Borzecki AM, Rosen AK, Zrelak PA, Sadeghi B, Baron R, Cuny J, Kaafarani HM, Geppert JJ, Romano PS. Designing an abstraction instrument: lessons from efforts to validate the AHRQ patient safety indicators. Joint Commission Journal on Quality and Patient Safety. 2011 Jan 1; 37(1):20-8. [view]
4. Mull HJ, Borzecki AM, Loveland S, Hickson K, Chen Q, MacDonald S, Shin MH, Cevasco M, Itani KM, Rosen AK. Detecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators. American journal of surgery. 2014 Apr 1; 207(4):584-95. [view]
5. Chen Q, Rosen AK, Cevasco M, Shin M, Itani KM, Borzecki AM. Detecting patient safety indicators: How valid is "foreign body left during procedure" in the Veterans Health Administration? Journal of the American College of Surgeons. 2011 Jun 1; 212(6):977-83. [view]
6. Chen Q, Borzecki AM, Cevasco M, Shin MH, Shwartz M, Itani KM, Rosen AK. Examining the relationship between processes of care and selected AHRQ patient safety indicators postoperative wound dehiscence and accidental puncture or laceration using the VA electronic medical record. American journal of medical quality : the official journal of the American College of Medical Quality. 2013 May 1; 28(3):206-13. [view]
7. Shin MH, Sullivan JL, Rosen AK, Solomon JL, Dunn EJ, Shimada SL, Hayes J, Rivard PE. Examining the validity of AHRQ's patient safety indicators (PSIs): is variation in PSI composite score related to hospital organizational factors? Medical care research and review : MCRR. 2014 Dec 1; 71(6):599-618. [view]
8. Borzecki AM, Kaafarani H, Cevasco M, Hickson K, Macdonald S, Shin M, Itani KM, Rosen AK. How valid is the AHRQ Patient Safety Indicator "postoperative hemorrhage or hematoma"? Journal of the American College of Surgeons. 2011 Jun 1; 212(6):946-953.e1-2. [view]
9. Borzecki AM, Kaafarani HM, Utter GH, Romano PS, Shin MH, Chen Q, Itani KM, Rosen AK. How valid is the AHRQ Patient Safety Indicator "postoperative respiratory failure"? Journal of the American College of Surgeons. 2011 Jun 1; 212(6):935-45. [view]
10. Borzecki AM, Cevasco M, Chen Q, Shin M, Itani KM, Rosen AK. Improving identification of postoperative respiratory failure missed by the patient safety indicator algorithm. American journal of medical quality : the official journal of the American College of Medical Quality. 2013 Jul 1; 28(4):315-23. [view]
11. Borzecki AM, Cevasco M, Mull H, Shin M, Itani K, Rosen AK. Improving the identification of postoperative wound dehiscence missed by the Patient Safety Indicator algorithm. American journal of surgery. 2013 Jun 1; 205(6):674-80. [view]
12. Bastian LA, Mattocks KM, Rosen AK, Hamilton AB, Bean-Mayberry B, Sadler AG, Klap RS, Yano EM. Informing policy to deliver comprehensive care for women veterans. Medical care. 2015 Apr 1; 53(4 Suppl 1):S1-4. [view]
13. Borzecki AM, Cowan AJ, Cevasco M, Shin MH, Shwartz M, Itani K, Rosen AK. Is development of postoperative venous thromboembolism related to thromboprophylaxis use? A case-control study in the Veterans Health Administration. Joint Commission Journal on Quality and Patient Safety. 2012 Aug 1; 38(8):348-58. [view]
14. Chen Q, Shin MH, Chan JA, Sullivan JL, Borzecki AM, Shwartz M, Rivard PE, Hatoun J, Rosen AK. Partnering With VA Stakeholders to Develop a Comprehensive Patient Safety Data Display: Lessons Learned From the Field. American journal of medical quality : the official journal of the American College of Medical Quality. 2016 Mar 1; 31(2):178-86. [view]
15. Cevasco M, Borzecki AM, Chen Q, Zrelak PA, Shin M, Romano PS, Itani KM, Rosen AK. Positive predictive value of the AHRQ Patient Safety Indicator "Postoperative Sepsis": implications for practice and policy. Journal of the American College of Surgeons. 2011 Jun 1; 212(6):954-61. [view]
16. Cevasco M, Borzecki AM, McClusky DA, Chen Q, Shin MH, Itani KM, Rosen AK. Positive predictive value of the AHRQ Patient Safety Indicator "postoperative wound dehiscence". Journal of the American College of Surgeons. 2011 Jun 1; 212(6):962-7. [view]
17. Zubkoff L, Neily J, Mills PD, Borzecki A, Shin M, Lynn MM, Gunnar W, Rosen A. Using a virtual breakthrough series collaborative to reduce postoperative respiratory failure in 16 Veterans Health Administration hospitals. Joint Commission Journal on Quality and Patient Safety. 2014 Jan 1; 40(1):11-20. [view]
18. Mull HJ, Borzecki AM, Chen Q, Shin MH, Rosen AK. Using AHRQ patient safety indicators to detect postdischarge adverse events in the Veterans Health Administration. American journal of medical quality : the official journal of the American College of Medical Quality. 2014 May 1; 29(3):213-9. [view]
19. Rosen AK, Chen Q, Borzecki AM, Shin M, Itani KM, Shwartz M. Using estimated true safety event rates versus flagged safety event rates: does it change hospital profiling and payment? Health services research. 2014 Oct 1; 49(5):1426-45. [view]
20. Smith EG, Zhao S, Rosen AK. Using the patient safety indicators to detect potential safety events among US veterans with psychotic disorders: clinical and research implications. International journal for quality in health care : journal of the International Society for Quality in Health Care. 2012 Aug 1; 24(4):321-9. [view]
21. Rosen AK, Itani KM. Validating the patient safety indicators in the Veterans Health Administration: are they ready for prime time? Journal of the American College of Surgeons. 2011 Jun 1; 212(6):921-3. [view]
22. Rosen AK, Itani KM, Cevasco M, Kaafarani HM, Hanchate A, Shin M, Shwartz M, Loveland S, Chen Q, Borzecki A. Validating the patient safety indicators in the Veterans Health Administration: do they accurately identify true safety events? Medical care. 2012 Jan 1; 50(1):74-85. [view]
23. Kaafarani HM, Borzecki AM, Itani KM, Loveland S, Mull HJ, Hickson K, Macdonald S, Shin M, Rosen AK. Validity of selected Patient Safety Indicators: opportunities and concerns. Journal of the American College of Surgeons. 2011 Jun 1; 212(6):924-34. [view]
24. Cevasco M, Borzecki AM, O'Brien WJ, Chen Q, Shin MH, Itani KM, Rosen AK. Validity of the AHRQ Patient Safety Indicator "central venous catheter-related bloodstream infections". Journal of the American College of Surgeons. 2011 Jun 1; 212(6):984-90. [view]

Conference Presentations

1. Shin M, Borzecki A, Cevasco M, Shwartz M, Loveland S, Chen Q, Rosen AK. Admission Coding: Implications for Patient Safety Indicator Rates in the Veterans Health Administration. Presented at: AcademyHealth Annual Research Meeting; 2011 Jul 11; Seattle, WA. [view]
2. Burgess JF, Shwartz M, Shwartz M, Stolzman KL, Pekoz E, Christiansen C, Berlowitz D. Analyzing the Relationship Between Cost and Quality Using a Bayesian Shrinkage Composite Measure of Quality. Paper presented at: AcademyHealth Annual Research Meeting; 2011 Jun 14; Seattle, WA. [view]
3. Mull HJ, Borzecki AM, Loveland S, Shin M, Chen Q, Rosen AK. Detecting Adverse Events with AHRQ Patient Safety Indicators: Assessing the Usefulness of Post-Discharge Administrative Data in the Veterans Health Administration. Poster session presented at: AcademyHealth Annual Research Meeting; 2012 Jun 24; Orlando, FL. [view]
4. Borzecki AM, O'Brien W, Mull H, Chen Q, Loveland S, Cevasco M, Shin M, Rosen AK. Do the Patient Safety Indicators Postoperative Respiratory Failure and Postoperative Wound Dehiscence Miss True Events? Poster session presented at: AcademyHealth Annual Research Meeting; 2011 Jun 12; Seattle, WA. [view]
5. Borzecki A, Shin M, Cevasco M, Rosen AK. How Valid is the AHRQ Patient Safety Indicator Postoperative Metabolic Derangements? Presented at: VA HSR&D National Meeting; 2011 Feb 16; National Harbor, MD. [view]
6. Loveland S A, Shin M, Chen Q, Zhao S, Hanchate AD, Rosen AK, Borzecki A. Trends in Readmission Rates within the Veterans Health Administration (VHA). Poster session presented at: VA HSR&D National Meeting; 2011 Feb 16; Washington, DC. [view]
7. Rosen AK, Borzecki A. Using Clinical Groupers to Study Episodes of Care. Poster session presented at: VA HSR&D National Meeting; 2011 Feb 18; National Harbor, MD. [view]
8. Borzecki AM, Cevasco M, Shin M, Chen Q, Rosen AK. Valid is the AHRQ Patient Safety Indicator Postoperative Metabolic Derangements? Poster session presented at: VA HSR&D National Meeting; 2011 Feb 18; National Harbor, MD. [view]
9. Rosen AK, Borzecki AM, Kaafarani H, Hanchate AD, Loveland S, Hartmann CW, Shwartz M, Rivard P, Shin M. Validating the Patient Safety Indicators (PSIs): Do They Reflect True Safety Events? Poster session presented at: VA HSR&D National Meeting; 2009 Feb 14; Baltimore, MD. [view]
10. Borzecki AM, Borzecki AM. Validating the Patient Safety Indicators in the Veterans Health Administration. Presented at: AcademyHealth Annual Research Meeting; 2009 Jun 30; Chicago, IL. [view]

DRA: Health Systems, Other Conditions
DRE: none
Keywords: Implementation, Research method, Risk factors, Safety
MeSH Terms: none