2012 National Meeting

3103 — Examining the Validity of the AHRQ Patient Safety Indicators (PSIs) in the VA: Are They Associated with Structures and Processes of Care?

Shin MHRivard P, and Sullivan J, Center for Organization, Leadership, and Management Research (COLMR); Solomon J, Center for Health Quality, Outcomes, and Economic Research; Hayes J, and Rosen AK, COLMR;

Using the AHRQ PSIs, we evaluated facility-level safety performance in collaboration with key VA stakeholders interested in patient safety. Although recent studies have examined the criterion validity of the PSIs, this is the first VA study to assess “attributional” validity (i.e., are PSI rates associated with structures/processes of care?). We determined the extent to which certain evidence-based structures/processes of care are associated with facility-level PSI rates.

Research teams (one clinician/one researcher) conducted site visits at six VA facilities and interviewed staff in diverse roles about structures/processes of care in 2009-2010. Sites were selected based on high/low PSI Composite Scores, geographic distribution, and surgical volume. Interviews targeted fourteen domains of evidence-based structures/processes of care (e.g., leadership, culture/climate, coordination, communication, quality monitoring). Interviews were audio-recorded and transcribed verbatim. Using a qualitative thematic approach, analysts coded the transcripts for evidence from the fourteen domains; we then conducted cross-site comparisons to examine differences in evidence-based structures/processes. Next, these results were compared to facility-level PSI rates, which were unblinded for this analysis.

Preliminary results show moderate association between PSI rates and structures/processes of care. Variations in certain structures/processes, such as leadership, were as expected (i.e., stronger leadership in facilities with lower PSI rates). In contrast, there was less association between PSI rates and other structures/processes of care, such as culture/climate and quality monitoring (i.e., stronger culture/climate or quality monitoring did not always equate to facilities having lower PSI rates). Analysis is ongoing; complete results will be available for presentation.

PSI rates appear to be associated with certain structures/processes of care, but variability within and across facilities is also notable. Given the complexity of evaluating safety, PSI rates alone may not provide a complete picture of facilities’ patient safety. More research may be needed to examine whether PSIs can be used to compare safety performance across facilities, and/or whether other variables may be confounding the relationship between structures/processes of care and PSI rates.

Since the PSI rates will soon be publicly reported on VA Hospital Compare, it is critical to establish the validity of the PSIs as outcomes for evaluating facility-level safety performance.