Though education is one of VA's four missions, few studies have assessed the cost of VA's education program, or its impact on patient outcomes, professional education, or contribution to clinical productivity. VA spends $1billion annually to create 8,900 positions through which 30% of all residents in ACGME approved programs do clinical rotations. Although resident supervision is a key component to any residency training program, only a few studies have measured supervision or related it to patient health or educational outcomes. While ordinal measures for procedural care (surgery) are available, there are no scientifically validated measures to assess resident supervision in outpatient care.
This study will refine and test the Resident Supervision Index (RSI), a survey instrument designed to measure quantitatively the intensity of supervision residents receive in outpatient clinics.
The RSI was evaluated for content validity through an Expert Panel of 10 nationally selected, Graduate Medical Education experts from VA. The RSI was tested in general internal medicine and two subspecialty surgical clinics the Loma Linda VAMC. Patient care encounters were included if residents delivered care and the patient had a diagnosis of diabetes or depression. Based on these criteria, a VistA report was used to develop a cohort patient list from which 10 to 15 patients were randomly selected weekly during an 8-week period. The RSI was administered to the consenting resident and attending physicians after the supervisory encounter. Total reported minutes of attending supervision was re-tested for reliability within 24 hours. Concurrent validity compared attending physician versus resident responses. Agreement was based on kappa, intraclass correlation, correlation, and bias. Variation in supervision intensity was assessed by computing the percent of total time that the resident spent on behalf of the given patient when the attending physician was physically present in same room.
The Expert Panel judged RSI ver. 3.11 as having content validity to measure supervision intensity through minutes in supervision, resident case understanding and changes or confirmation of clinical findings and care. Feasibility: for residents: 75 (93.8%) of 80 residents consented and completed RSIs; for attending physicians: 37 (97%) of 38 physicians consented and completed RSI. Reliability: for residents: 125 of 148 RSI with retests agreed with kappa=.74, ICC=.95, r=.90, bias=.03 minutes of mean 7.7 minutes/encounter; for attending physicians: 132 of 143 captured retests, with kappa=1.00, ICC=.92, r=.85, and bias=.09 minutes of mean 8.3 minutes/encounter. Among 140 encounters where both resident and attending physician reported an RSI on a given supervision encounter, kappa=.72, ICC=.78, r=.64, bias=.22 minutes of mean 7.6 minutes/encounter. Intensity variability: Supervision intensity (percent of resident patient time that was directly supervised by the attending physician) averaged 28% over 110 study patients, with standard deviation of 14%, and a range per case from 2% to 75%. However, 50% of cases had intensity between 20% and 33%.
The Resident Supervision Index has content validity, is feasible in clinical settings, is a reliable measure of supervision, and did distinguish residents facing different levels of supervision. The RSI provides an opportunity for researchers assessing graduate medical education to assess supervision intensity and its impact on patient (quality, health), trainee (satisfaction, learning), and system-level (costs, patient retention, attending physician burden) outcomes. RSI applications may also contribute to VA policy makers, planners, and administrators by offering insight into graduate medical education (GME) in VA and the role GME plays in caring for the present and future health needs of our veterans.
- Byrne JM, Kashner M, Gilman SC, Aron DC, Cannon GW, Chang BK, Godleski L, Golden RM, Henley SS, Holland GJ, Kaminetzky CP, Keitz SA, Kirsh S, Muchmore EA, Wicker AB. Measuring the intensity of resident supervision in the department of veterans affairs: the resident supervision index. Academic Medicine. 2010 Jul 1; 85(7):1171-81.
- Kashner TM, Henley SS, Golden RM, Byrne JM, Keitz SA, Cannon GW, Chang BK, Holland GJ, Aron DC, Muchmore EA, Wicker A, White H. Studying the effects of ACGME duty hours limits on resident satisfaction: results from VA learners' perceptions survey. Academic Medicine. 2010 Jul 1; 85(7):1130-9.
- Kashner TM, Byrne JM, Chang BK, Henley SS, Golden RM, Aron DC, Cannon GW, Gilman SC, Holland GJ, Kaminetzky CP, Keitz SA, Muchmore EA, Kashner TK, Wicker AB. Measuring progressive independence with the resident supervision index: empirical approach. Journal of graduate medical education. 2010 Mar 1; 2(1):17-30.
- Kashner TM, Byrne JM, Henley SS, Golden RM, Aron DC, Cannon GW, Chang BK, Gilman SC, Holland GJ, Kaminetzky CP, Keitz SA, Muchmore EA, Kashner TK, Wicker AB. Measuring Progressive Independence with the Resident Supervision Index: Theoretical Approach. The journal of graduate medical education. 2010 Mar 1; 2:8-16.
- Cannon GW, Keitz SA, Holland GJ, Chang BK, Byrne JM, Tomolo A, Aron DC, Wicker AB, Kashner TM. Factors determining medical students' and residents' satisfaction during VA-based training: findings from the VA Learners' Perceptions Survey. Academic Medicine. 2008 Jun 1; 83(6):611-20.
Mental, Cognitive and Behavioral Disorders, Health Systems
Education Research, Mental Health Care, Trainees