» Back to Table of Contents

Research Highlight

Evaluation of Innovators Network Aims to Understand Why Innovations Succeed, and If They Can be Scaled and Spread

In 2015, the VA Center for Innovation (VACI) launched the VA Innovators Network program. The goals of this program are to foster a culture of innovation and to continuously improve the care and experience of care encountered by Veterans. Innovators Network addresses these goals by empowering VA employees to test new ideas and by encouraging collaboration with stakeholders across VA programs. Specifically, the Innovators Network builds the innovation capacity of VA by: 1) teaching and training VA employees on innovation-related competencies (e.g., human-centered design, entrepreneurship); 2) creating an innovation development and funding pathway to generate novel ways of serving Veterans; and 3) supporting the VACI Diffusion of Excellence initiative by encouraging the identification and implementation of promising practices at VA Innovators Network sites. This work aligns with the overarching mission of the VACI which is to identify, test, and evaluate new approaches to meet the needs of Veterans through innovations rooted in data, design-thinking, and agile development.

The Innovators Network investment has yielded a diverse set of local leaders and projects. The ground-level leadership of Innovators Network is a team of Innovation Specialists embedded in 33 VA Medical Centers nationwide, who receive training and financial support through VA Innovators Network. Trained in core competencies including quality improvement, human-centered design,¹ and implementation science, these specialists represent a broad range of backgrounds and job descriptions and work directly with employees and Veterans to support innovation. Complementary to this support for local leadership development, Innovators Network also supports projects initiated by frontline employees through the Spark-Seed-Spread Investment program. This is a three-tiered funding model which, to date, has awarded approximately $10 million in Innovation Spark-Seed-Spread grant funds for more than 300 projects across 33 VA sites. The interventions supported by the Innovators Network include clinical, administrative, educational, and health information technology initiatives. They range from the development of new technology (e.g., through 3D-printing) to development of new programs, and to the reorganization of clinical and administrative workflow. A broad array of professionals are implementing these projects across a range of VA medical center environments.

In partnership with the VA Innovators Network and VA Quality Enhancement Research Initiative (QUERI), investigators at Bedford, Boston, Durham, Palo Alto, and Portland have joined forces to plan and conduct a national evaluation of the Innovators Network. Our mixed-methods, multi-site evaluation will address three key questions: 1) Who participates in Innovators Network? 2) Why do they participate? and 3) What is the impact of Innovators Network participation? These three questions guide the selection of the evaluation strategy. The evaluation further seeks to understand why a particular innovation was successful, and how and in what settings it can be scaled up and spread, recognizing that not all innovations are a perfect fit for all settings.

While the diversity of projects and settings is a considerable strength of the Innovators Network, it also poses challenges to effective evaluation. With such a large number of highly varied projects, it is not practical to perform a separate, customized evaluation for each awardee. Further, the lack of common metrics inherent in such an approach would hinder comparisons of performance across projects, making it more difficult to identify common success factors, challenges, and unintended consequences. We have attempted to address these and other challenges in designing the evaluation strategy and share a brief overview of our planned approach here.

Our evaluation, which began in April 2018, is seeking to understand who participates in the Innovators Network, and why. We will pursue quantitative analyses to identify organizational factors associated with applying to be an Innovators Network site and factors associated with subsequent applications for Spark-Seed-Spread grants. Additionally, through qualitative analysis of interviews with site leadership, Innovation Specialists, and select project leads, we will explore expectations for participation and how and whether they were met. Results from these analyses will allow Innovators Network leadership to anticipate how characteristics of sites may affect participation in the Network and will provide the foundation for defining the long-term impact of participation.

After examining who participates and what they hope to achieve through participation, we will study the impact of Innovators Network participation on individual sites. Qualitative data from interviews with Network participants will help identify and understand site-level impacts and experiences. Interview questions will explore the perceived impacts, benefits and challenges of Network participation, and observations of systemic or structural supports either observed or desired in order to sustain these impacts. Qualitative data will be triangulated with site level data with a focus on employee satisfaction, psychological safety, autonomy, and participation in decision making measures to examine the impact that participation in the Innovators Network has on employee experience.

Finally, for selected Innovators Network projects, we will measure impact, including factors associated with success, sustainability, and return on investment. Interviews with selected Network participants will identify perceived project impacts for Veterans, for VA employees, and for the VA system as a whole. Interviews will explore sites' use of specific metrics to track project impacts as well as successes and challenges. Where available, we will work with Innovators Network and project leaders to capture and interpret these metrics to gain a deeper understanding of project success. A business case framework will guide the return on investment (ROI) analysis for select innovation awardees. A business case for quality in health care can be defined as a financial ROI to the entity that invests in the intervention, in a reasonable time frame, that can be realized as real dollars, as a reduction in losses for a given program or population, or as avoided costs. A business case analysis differs from other types of economic analysis (e.g., cost-benefit, cost-effectiveness, cost-utility), which often measure costs and benefits from a societal perspective. The perspective of business case analysis is purposely narrow since its primary goal is to determine the sustainability of quality-focused innovations from the investing organization's perspective.

We believe the energy and strength of the Innovators Network stem from the diversity of Network participants and sites nationwide, and from the broad range of projects led by frontline employees. To assess impact, therefore, we must begin by ascertaining the goals and motivations of participants, then use this knowledge to understand the variety of successful outcomes sought and achieved. This is a three-year evaluation funded through March 2021 and we anticipate preliminary results in 2019. The VA Innovators Network is a potentially transformative initiative whose goal is to improve the care delivered to our nations' Veterans and to improve the workplace experience for those who serve them. This comprehensive evaluation of the VA Innovators Network will assess its impact and inform further operationalization and sustainability of processes to better support a culture of innovation and enable implementation of innovation development pathways across VA nationally.

References

1. 1. "Designing for Veterans, A Toolkit for Human-Centered Design," Department of Veterans Affairs Center for Innovation. October 2015; Version 1.0.