BACKGROUND/RATIONALE:
The Institute of Medicine has advocated using performance and outcome measures as a critical tool to improve the quality of care. VA leads the industry in performance measurement through its External Peer Review Program (EPRP), a comprehensive performance measurement system with over 90 indicators across six domains of health care value, including quality. VA's investment has significantly improved quality of care, with VA facilities outperforming both the public and private sector. Nevertheless, previous research by this project's PI indicates that, though most measures show improvement over time, there is significant variability in improvement across measures. Potential alternative explanations, such as organizational characteristics or "performing to the test" (i.e., health care facilities tend to concentrate their efforts exclusively on the areas in which they are being measured), are unsupported by these analyses. Research from industrial/organizational psychology suggests that two performance measure characteristics, accomplishment difficulty and documentation difficulty, could considerably bias the validity of a performance measure if not accounted for. In this pilot project we adapted and developed measures of these characteristics, and tested their impact on clinical performance over time.

OBJECTIVE(S):
The objectives of this pilot research are: (1) adapt and develop measures of documentation difficulty and accomplishment difficulty of outpatient quality EPRP measures, and (2) to assess the impact of documentation and accomplishment difficulty on VAMC performance trends of these measures.

METHODS:
DESIGN: This research is a mixed methods study combining a retrospective database review of selected EPRP measures with expert ratings of EPRP measure characteristics. We used growth curve analyses to examine the trajectory of performance on each EPRP measure over time, and included measure difficulty ratings as a predictor to explain performance differences between measures over time.

DATA SOURCE: We obtained facility level scores on 17 EPRP outpatient measures from 2000 to 2008. Data were acquired via a data use agreement with the Office of Quality and Performance.

DOCUMENTATION DIFFICULTY SCALE DEVELOPMENT AND ASSESSMENT: The research team developed a Guttmann type scale assessing the effort required to enter sufficient detail in the electronic medical record to satisfy the criteria for the relevant performance measure. Six primary care physician (PCP) experts experienced in EPRP independently rated the 17 measures using the scale; we then averaged the ratings to arrive at a single numerical assessment of documentation difficulty for each measure.

ACCOMPLISHMENT DIFFICULTY ASSESSMENT: Accomplishment difficulty was assessed via Functional Job Analysis, a longstanding methodology for describing work. The aforementioned PCP experts described in detail the tasks required to satisfy each performance measure; in instances where PCPs could not provide sufficient detail, task statements were fleshed out by clinical personnel with the appropriate expertise (e.g., gastroenterologist for colorectal cancer screening tasks). Each task was rated on ten dimensions of complexity (complexity with respect to things, data, people, reasoning, math, language, autonomy, worker technology, worker interaction, and human error consequence) by two independent raters (discrepancies were resolved by consensus); we then averaged dimensional ratings across component tasks to arrive at a single rating of complexity for each dimension, for each EPRP measure. The degree of worker interaction, human error consequence, autonomy, and a composite representing the mean of all ten dimensions were specifically examined as alternate measures of accomplishment difficulty.

DATA ANALYSIS: We conducted between-groups analyses of variance to test whether accomplishment and documentation difficulty varied significantly across EPRP measures. To determine the impact of these difficulty measures on EPRP scores over time, these ratings were used as covariates in growth curve analyses of the 17 EPRP measures.

FINDINGS/RESULTS:
DIFFERENCES ACROSS EPRP MEASURES IN DIFFICULTY RATINGS: EPRP measures varied considerably in the number of tasks required to achieve the behavior or outcome evaluated by the measure (mean number of tasks = 10.87, SD = 7.7, min = 5, max=33). On average, measures of chronic care following AMI (e.g., ASA or beta blockers at most recent visit) exhibited significantly higher composite ratings of accomplishment difficulty than diabetes or screening measures, though they did not significantly differ from immunization measures (mean z-score =.45, -.03, -.06, and -.07 respectively; F(3, 156)= 3.66, p=.014). No significant differences in documentation difficulty were found amongst measures, nor were difficulty ratings significantly correlated with the number of tasks required (r = -.33, p = .18).

EPRP SCORES AS A FUNCTION OF DIFFICULTY: Accomplishment difficulty, as measured by the average degree of worker interaction in an EPRP measure's component tasks, significantly predicted change over time: EPRP measures with higher worker interaction values exhibited steeper slopes than EPRP measures with lower worker interaction values (B=.011, SE =.003, p=.0004), suggesting that on average, measures requiring greater worker interaction (coordination) showed greater improvement over time than measures requiring less worker interaction.

IMPACT:
VA spends millions annually on its External Peer Review Program (EPRP) to maintain a valid, reliable, and useful clinical performance management system; however, this goal depends on the EPRP's ability to accurately capture the phenomena it is designed to assess. Our findings suggest that performance measures requiring greater interaction, which can be used as a proxy for coordination, require more time to achieve and sustain high levels of performance. These findings have important implications for primary care facilities' transition into Patient Aligned Care Teams (PACTs): as more clinical work is structured around teams, special attention should be paid to issues of coordination, both from measurement and workflow perspectives by researchers and policy makers alike.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX000320-01
Link: https://reporter.nih.gov/project-details/7870104

---

PUBLICATIONS:

Journal Articles

1. Hysong SJ, Amspoker AB, Petersen LA. A Novel Method for Assessing Task Complexity in Outpatient Clinical-Performance Measures. Journal of general internal medicine. 2016 Apr 1; 31 Suppl 1:28-35. [view]

Conference Presentations

1. Hysong SJ, Khan MJ, Amspoker AB, Petersen LA. All clinical performance measures are not created equal: The role of interaction among clinical personnel on measured performance. Poster session presented at: AcademyHealth Annual Research Meeting; 2012 Jun 25; Boston, MA. [view]
2. Hysong SJ. Organizational correlates of outpatient performance trends in VAMCs. Paper presented at: VA HSR&D National Meeting; 2011 Feb 16; Washington, DC. [view]

DRA: Health Systems
DRE: none
Keywords: none
MeSH Terms: none