The report is a product of the VA/HSR Evidence Synthesis Program.

Caution Advised for Pay-for-Performance Programs

Over the last decade, pay-for-performance (P4P) programs have been implemented in a variety of health systems, including the VHA, as a means to improve the efficiency and quality of healthcare. There has been a parallel increase in the number of studies examining the effects of P4P. A number of recent reviews have summarized this literature, but have found insufficient evidence to broadly characterize the balance of harms and benefits. However, financial incentives programs are complex interventions whose effects may depend, in part, on the settings in which they are implemented, the methods used for implementation, the populations targeted, and the characteristics of the incentive programs themselves.

Investigators with the VA Evidence-Based Synthesis Program located in Portland, OR conducted an evidence review to examine the positive and negative effects of P4P on process and health outcomes – and how implementation characteristics modify the effects of P4P programs. The review incorporates information from the following sources:

• Summary of evidence from an existing good quality systematic review, which included U.S. studies published primarily through December 2012;
• Update of the existing systematic review, including studies published from December 2012 through April 2014;
• Full review of studies in countries with health systems similar to VA (e.g., UK, Taiwan) published through April 2014; and
• Key informant (KI) interviews with 14 experienced P4P researchers.

Summary of Review
Investigators identified 94 articles that met inclusion criteria, in addition to those found in the prior systematic review. In general, P4P programs appear to have the potential to improve process of care outcomes over the short term, especially in ambulatory settings; however, there is insufficient evidence that P4P programs have beneficial effects on care processes over the long term – or on patient outcomes over any time period. Incentive programs tend to have the greatest absolute effect on care processes over the short term in settings with lower baseline levels of performance. In the United States in particular, the effects of P4P on health disparities are unclear, largely due to the lack of patient cultural variables that were collected and recorded. There is limited evidence in the UK's Quality and Outcomes Framework (QOF) and within VA that initial improvements may be sustained even after removal of the incentive. The value of incentive programs to stimulate incremental performance gains once initial improvements have been achieved is unclear. Also unclear is the influence of P4P above and beyond other quality initiatives often accompanying financial incentives, such as public reporting and information technology.

Findings from experts in the field (i.e., key informants) are congruent with previous qualitative work – that the potential negative unintended consequences of P4P may, in some circumstances, outweigh benefits, though there is relatively little good-quality empiric evidence examining the rates of harms from P4P. Following is more information about key findings.

Effects of P4P on Patient Outcomes and Processes of Care

• P4P programs in ambulatory settings can improve the proportion of patients receiving the care process targeted by an intervention. However, this is based on low-strength evidence because of inconsistencies across studies, lack of impact over the long term, heterogeneity of interventions studied and outcomes measured, as well as the typically small effect size.
• There is insufficient evidence that P4P programs have beneficial effects on patient outcomes over any time period.
• In hospital settings, studies evaluating the Premier Hospital Quality Incentive Demonstration and the Hospital Value-Based Purchasing programs in the U.S. report a limited effect on both processes of care and patient outcomes. However, a study evaluating the effect of P4P in the VA on processes of care found significant and sustained improvement on six of the seven measures examined. Internationally, studies evaluating hospital P4P programs report generally positive effects, with a slowing of improvements or a plateau over time.

Implementation Factors that Modify P4P Effectiveness

• Programs that emphasize measures that target process of care or clinical outcomes that are transparently evidence-based and viewed as clinically important may inspire more positive change than programs that use measures targeted to efficiency or productivity, or do not explicitly engage providers from the outset.
  • Findings from both the literature examining physician perceptions and KI interviews support the use of evidence-based measures that are congruent with providers' expectations for clinical quality. There also was strong agreement among KIs that provider buy-in is crucial.
• The incentive structure needs to carefully consider several factors, including incentive size, frequency, and target. In general, the QOF, with its larger incentives, has been more successful than programs in the U.S.
  • Key informants attribute this to incentives that are large enough to motivate behavior, but also caution that larger incentives may not be cost-effective and may result in gaming. KIs also stressed the importance of the attribution of the incentive to provider behavior, that incentivized measures be congruent with institutional priorities, address the needs of the institution at the local level, and are designed to best serve the local patient population.
• P4P programs should have the capacity to change over time in response to ongoing measurement of data and provider input.
  • Key informants strongly agreed that P4P programs should be flexible and evaluated on an ongoing and regular basis. They pointed to the QOF, which is evaluated annually. Moreover, since its inception it has undergone numerous adjustments, i.e., the measures incentivized and the thresholds associated with payments.
• P4P programs should target areas of poor performance and consider de-emphasizing areas that have achieved high performance.
  • Findings from studies of both the QOF and VA, as well as KI interviews, support that improvements associated with measures achieving high performance can be sustained after the measure has been de-incentivized. Consistent evaluation of the performance of – and adjustments to incentivized measures will allow institutions to shift focus and attention to the areas of greatest need for improvement.

Positive and Negative Unintended Consequences associated with Pay-for-Performance

• Most of the studies examining differential effects of P4P by race/ethnicity, socioeconomic, or other demographic characteristics came from the UK's QOF program. In general, there was no strong consistent evidence that P4P had different effects on different patient subgroups.
• Key informants stressed that gaming is likely to occur and that P4P programs should be designed with this assumption. To reduce the likelihood of gaming, P4P programs should have stakeholder input and buy-in, and should be based on precise, simple, evidence-based, and realistic measures.
• In the U.S., KIs expressed concern that higher-risk patients can now be easily identified using algorithms. A common theme among KIs was that incentive payments should be risk-adjusted to account for higher-risk patients.

Future Research
Despite numerous P4P programs, there is a need for higher-quality evidence to better understand whether these programs are effective in improving the quality of healthcare and patient health, and whether they result in negative unintended consequences. Studies examining P4P have been largely observational and primarily retrospective, or lack good matched comparison groups. In addition, one of the fundamental challenges is disentangling the individual effect of each intervention, as well as discerning contextual and implementation factors. There also is an urgent need to examine the implementation factors that may mediate or moderate program effectiveness, such as the influence of public reporting, the number and focus of measures, incentive size, structure, and target. In addition, more research is needed to better understand whether P4P differentially affects sub-populations of patients, and, if so, how best to mitigate health disparities and avoid unintended consequences.

VA is in a unique position to conduct rigorous and methodologically strong P4P research, examining not only P4P's effectiveness on processes of care and patient outcomes, but also implementation characteristics and unintended consequences.

Reference
Kondo K, Damberg C, Mendelson A, Motu'apuaka M, Freeman M, O'Neil M, Relevo R, Kansagara D. Understanding the intervention and implementation factors associated with benefits and harms of pay for performance programs in healthcare. VA-ESP Project #05-225; 2015.

A Cyberseminar session on this ESP Report will be held on November 16, 2015 from 2:00 to 3:00pm (ET). To register, go to the HSR&D Cyberseminar web page.

Please feel free to forward this information to others!

Read past HSR&D Management e-Briefs on the HSR&D website.

ESP is currently soliciting review topics from the broader VA community. Nominations will be accepted electronically using the online Topic Submission Form. If your topic is selected for a synthesis, you will be contacted by an ESP Center to refine the questions and determine a timeline for the report.

This Management e-Brief is provided to inform you about recent HSR&D findings that may be of interest. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs. If you have any questions or comments about this Brief, please email CIDER. The Center for Information Dissemination and Education Resources (CIDER) is a VA HSR&D Resource Center charged with disseminating important HSR&D findings and information to policy makers, managers, clinicians, and researchers working to improve the health and care of Veterans.

This report is a product of VA/HSR&D's Quality Enhancement Research Initiative's (QUERI) Evidence-Based Synthesis Program (ESP), which was established to provide timely and accurate synthesis of targeted healthcare topics of particular importance to VA managers and policymakers – and to disseminate these reports throughout VA.

See all reports online.