Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

VA Health Systems Research

Go to the VA ORD website
Go to the QUERI website

SHP 08-156 – HSR Study

SHP 08-156
Formative Evaluation in Implementation Research: Guideline Development
Susan L. Zickmund, PhD
VA Pittsburgh Healthcare System University Drive Division, Pittsburgh, PA
Pittsburgh, PA
Funding Period: June 2008 - December 2008
The VA has embraced the use of implementation research as a critical way to improve care by facilitating the adoption of proven medical practices. Yet implementation science still remains rudimentary. Challenges exist in judging the success of an implementation project. Rather than relying solely on hard outcomes, an investigator must also assess how effectively an intervention is put into place at a site. This assessment comes via a formative evaluation (FE) plan. The FE plan can contain such diverse data elements as the number of fliers distributed about the intervention, how often an intervention advocate speaks to participants, the content derived from team meeting notes, and themes extracted from focus groups. The diversity of such data elements can appear foreign to investigators trained in traditional health services research. The FE plan can also lead to confusion about what to measure, when, and how to integrate the various measures into a unified whole. Complicating matters is the fact that FE is often qualitative--such as interviews, focus groups, and meeting notes-which can make data difficult for investigators to interpret. Thus, there is a pressing need to provide clarity on how FE is to be used in implementation studies.

The specific aims of this Short Term Project are to: (1) Conduct a systematic review of VA funded implementation projects to classify the specific types of FE procedures employed; (2) Conduct in-depth telephone interviews with investigators of VA-funded implementation projects to establish which FE methods were used and which were considered most/least successful; and (3) Using the national experts gathered together under the umbrella of the Center for Health Equity Research and Promotion's Implementation Research Collaborative, develop a set of guidelines outlining best FE practices in VA implementation research.

This Short Term Project proposal offers a three-part approach: 1) a systematic review of the literature designed to clarify what investigators should consider appropriate for designing FE during implementation projects; 2) qualitative interviews with VA implementation investigators designed to show which FE methods were used as well as thematic differences between investigators; and 3) an expert panel, funded through a VA Implementation Collaborative Award, charged with considering the information needs of investigators and the ways of moving the recommendations for FE forward.

Of 74 articles found using search terms in our systematic review, 22 were relevant to our topic. A close analysis of these articles (see Table 1, Appendix G) found that the guidelines and recommendations for FE were very similar in general concept and execution, including those published in the VA Quality Enhancement Research Initiative (QUERI) journal Implementation Science and in other journals as well.

During the telephone interviews with investigators, a consensus emerged that conducting interviews with key providers and staff was the most helpful FE method and that surveys were considered least helpful. Investigators expressed frustration about the lack of resources needed to complete FE activities. They were also concerned that while FE required time and resources, it resulted in few publications in high impact Medical journals. Thematic differences were seen between investigators affiliated with QUERI Centers and those who were not. QUERI investigators in the interviews appeared to recognize at the outset the need for a detailed plan for FE, while investigators not affiliated with QUERI struggled to understand the value and purpose of FE. Finally, differences were found between investigators who did not have expertise in using mixed qualitative methods training and those who did. Those without mixed methods expertise were uncomfortable with qualitative data. Those who were experienced in using mixed methods were concerned about the quality of the data collection methods and about the potential bias resulting from an imbalance in power relations between those who conducted interviews (research assistants or supervisors) and the implementation team.

In the expert panel, members shared their own experiences in learning to use or understand FE. Suggestions were made to use logic models in FE as a way for investigators to increase their precision in mapping out FE plans.

This Short Term Project is designed to bridge gaps that may exist between recommendations and the real world challenges that investigators experience as well as to provide suggestions for furthering FE research. The results of the study can have wide application within the VA implementation community, thereby supporting the HSR&D research mission.

External Links for this Project

Dimensions for VA

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

Learn more about Dimensions for VA.

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.
    Search Dimensions for this project


None at this time.

DRA: Health Systems
DRE: none
Keywords: Implementation
MeSH Terms: none

Questions about the HSR website? Email the Web Team

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.