1049 — Balancing Internal Versus External Validity in Implementation Research

Zubkoff L (Sepulveda Center of Excellence (COE)), Karwoski J (Sepulveda COE), Mittman BS (Sepulveda COE)

Objectives:
Implementation research seeks to understand and evaluate implementation processes under routine circumstances within the full range of contexts found in real-world settings. In contrast to efficacy studies of medical interventions, in which heterogeneity of settings is minimized to enhance the researcher’s ability to isolate the intervention’s main effects, implementation researchers seek out diversity to evaluate intervention programs under realistic conditions representing the full heterogeneity of real-world contexts. In many cases, implementation studies also aim to achieve policy and practice improvement goals in addition to knowledge generation goals. Each of these features of implementation research poses distinct challenges for traditional research methods designed to maximize internal validity. This presentation offers guidance in maximizing external validity and real-world relevance of implementation studies within VA while retaining necessary levels of internal validity.

Methods:
The presentation is based on in-depth, structured reviews of four categories of prior literature, including literature (1) addressing goals and challenges of implementation research, (2) discussing tradeoffs between internal and external validity, (3) presenting strategies for maximizing external validity, and (4) proposing standards for reporting external validity. This literature is used to create a set of dimensions and design decision points for implementation researchers, accompanied by guidance regarding specific tradeoffs and factors to consider in balancing competing considerations.

Results:
In general, VA implementation researchers are better able to achieve high levels of real-world relevance and external validity (while minimizing threats to internal validity) than implementation researchers working in other settings, due to specific features of the VA healthcare delivery system and the implementation research infrastructure developed by HSR&D and QUERI. The presentation offers specific guidance for external validity and design decisions, explaining how to take advantage of the VA implementation research infrastructure to maximize the value of implementation studies.

Implications:
VA researchers should invest additional effort in explicitly confronting research design tradeoffs and reaching design decisions based on a careful plan for maximizing external and internal validity as appropriate for the specific study in question.

Impacts:
Better attention to external validity and real-world relevance in VA implementation research will increase the likely successful sustainability and further adoption of research-developed implementation strategies.