Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

VA Health Systems Research

Go to the VA ORD website
Go to the QUERI website

SHP 08-194 – HSR Study

SHP 08-194
Developing a PBLI QI Systems Impact Assessment Questionnaire
David C. Aron, MD MS
Louis Stokes VA Medical Center, Cleveland, OH
Cleveland, OH
Funding Period: April 2008 - September 2008
The Accreditation Council for Graduate Medical Education (ACGME) acknowledged the changing needs of physicians in training when it endorsed practice-based learning and improvement (PBLI): a core competency that is typically omitted from formal medical curriculum. The overall goal is to have residents competent to investigate and evaluate their own patient care practices, evaluate and integrate scientific evidence and be able to improve their practices. Available curriculums and assessment tools do not adequately address all of the components of PBLI and few attempt to capture the residents' ability to develop and implement clinically-based CQI projects that involve the practice setting and assess impact on the practice setting and/or organization. Curriculums and assessment tools without such foci miss the importance of system perspectives, opportunities for interprofessional team development, and meaningful impacts on patient care.

Our objectives are: (1) to evaluate pre-post data on a PBLI curriculum grounded on QI system projects; (2) by developing a PBLI QI assessment tool and manual for scoring PBLI knowledge and application; and (3) to provide preliminary data regarding the sustainability of the residents' systems projects and the impact of those projects on patient care and the organization.

PBLI curriculum was offered on alternate rotations (residents on alternate month were involved with a different curriculum). Pre-post rotation data was available from 6 PBLI QI Systems Curriculum blocks (n=50) and 5 comparison blocks (n=42). Data included closed- and open-ended questions designed to assess resident PBLI application skills, and the notes and presentation slides for the residents' presentation of their projects at the internal medicine's morbidity and mortality conference (IM MMC). Qualitative methods were used to identify scoring themes. Open and thematic coding was conducted and inter-coder reliability was evaluated. Project impact and sustainability were assessed by comparing the project focus and current organizational practices.

Complete pre-post data was available on 46 residents in the intervention group, and 42 residents in the comparison group. Codebooks were iteratively developed and inter-coder reliability assessed, with acceptable levels of coder agreement achieved. The two groups did not differ on pre-test levels for any items. As hypothesized, residents participating in the PBLI QI systems curriculum had greater increases in beliefs about ability to implement a CQI project and comfort developing data collection plans. The groups did not differ regarding changes in beliefs related to ability to develop a project and teach CQI principles. Also, as hypothesized, residents in the intervention (PBLI curriculum) demonstrated greater increase in total knowledge score (e.g., describe change concept, how a cause-effect diagram is created, elements of the improvement model), with the PBLI intervention being a significant predictor of post knowledge scores (partial eta squared = .224) after adjusting for pre knowledge levels and prior experience. Twelve of the 15 teams' suggestions based on their PBLI projects with practical systems-relevant outcomes were implemented and sustained beyond residents' project periods.

By developing a PBLI assessment tool and manual for scoring PBLI knowledge and application we have established a mechanism to identify resident competence in PBLI and systems thinking. The importance of developing and evaluating this knowledge is critical to training physicians capable of improving the care of veterans and contributing to the organizations mission of delivering high quality health care.

External Links for this Project

Dimensions for VA

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

Learn more about Dimensions for VA.

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.
    Search Dimensions for this project


Journal Articles

  1. Stulberg JJ, Delaney CP, Neuhauser DV, Aron DC, Fu P, Koroukian SM. Adherence to surgical care improvement project measures and the association with postoperative infections. JAMA : the journal of the American Medical Association. 2010 Jun 23; 303(24):2479-85. [view]
  2. Lawrence RH, Tomolo AM. Development and preliminary evaluation of a practice-based learning and improvement tool for assessing resident competence and guiding curriculum development. Journal of graduate medical education. 2011 Mar 1; 3(1):41-8. [view]
  3. Tomolo AM, Lawrence RH, Watts B, Augustine S, Aron DC, Singh MK. Pilot study evaluating a practice-based learning and improvement curriculum focusing on the development of system-level quality improvement skills. Journal of graduate medical education. 2011 Mar 1; 3(1):49-58. [view]

DRA: Health Systems
DRE: none
Keywords: Education Research, Practice patterns, Trainees
MeSH Terms: none

Questions about the HSR website? Email the Web Team

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.