The science of implementation is maturing and gaining prominence as we seek to identify successful strategies for integrating promising research evidence into clinical practice. One aspect of implementation science that has not received consistent attention is the measurement of implementation success - that is, measurement of the extent to which an innovation is actually put into regular practice as intended in the targeted organizations. Yet the measurement of implementation success is critical to implementation science because it is essential in identifying effective implementation strategies and contributes to understanding the success or failure of the innovation outcomes.
To advance this aspect of implementation science by conducting an inventory of the measurement of implementation success in implementation studies in two HSR&D portfolios: Implementation Service Directed Projects and Implementation and Management.
The inventory included implementation studies in two VA HSR&D portfolios: Implementation Service Directed Projects and Implementation and Management Research. Beginning with the 200 studies in these portfolios, we screened project abstracts to identify implementation studies. Fifty-nine implementation studies were identified and PIs were contacted by e-mail to request project documents describing the project protocol and implementation measures used. Forty-five study teams responded with the requested materials. After reviewing the documents, we requested telephone interviews with PIs or designees to pursue details. Thirty-three teams responded and were interviewed. Detailed notes from document reviews and interviews were entered into matrices and reviewed qualitatively. The matrices included: 1) description and categorization of the innovation, 2) the scope of the study; 3) levels of organization at which implementation was targeted; 4) standards of implementation success; 5) data sources; 6) validation of measures; 7) implementation model used.
A substantial majority of studies in the inventory measure implementation success separate from outcome performance. However, the measures are not consistently well developed and there are no established standards across studies. Of the 33 studies in the final inventory, 28 (85%) formally measured implementation success. Of those 28, 50% used fidelity as the standard of implementation success and 18% used scope; 18% used multiple standards. Data sources for the measures included: electronic records, surveys, interviews, focus groups, observations, research records. Most measures were newly created. Most studies did not measure the implementation strategy used to introduce the innovation to the organization. Measurement standards and data sources did not vary systematically by type or scope of intervention.
The inventory results have immediate benefits in drawing attention to the importance of implementation success measurement and of offering examples of measurement tools. In the longer term, the findings contribute to implementation science by laying the groundwork for further work in this area. Further work is needed to develop robust measures, especially those that can take into account the multiple perspectives of innovation participants at different levels of the organization and that produce valid information from less than perfect data sources.
- VanDeusen-Lukas C, Hall CG, Gerena-Melia M. Challenges of measuring implementation success. Paper presented at: National Institutes of Health Conference on the Science of Dissemination and Implementation: Methods and Measurement; 2010 Mar 15; Bethesda, MD.
- VanDeusen-Lukas C, Silver J, Gerena-Melia M. An Inventory of measurement of implementation success. Poster session presented at: AcademyHealth Annual Research Meeting; 2009 Jun 28; Chicago, IL.
- VanDeusen-Lukas C, Silver J, Gerena-Melia M. Measuring Implementation Success. Presented at: VA QUERI National Meeting; 2008 Dec 10; Phoenix, AZ.