BACKGROUND/RATIONALE:
One-third of all U.S. physician residents rotate through a VA medical center. While rotating through a VA medical center, the over 43,000 physician residents, interns, and fellows representing all clinical specialties, plus an additional 25,000 medical students, are involved in 37% of VA's physician services that provide care for over 2.5 million veterans. VA professional staff supervises these physician trainees when they are engaged in providing critical care to patients in VA medical centers. Despite the critical role that VA medical staff plays in supervising these physician trainees that ensures the quality care for its veterans and education opportunities to future health professionals within the learning, academic medical center, there are no scientifically valid, widely used measures that quantify the intensity of that supervision or assess the appropriateness of that supervision. In fact, little is understood how supervision impacts patient outcomes, guide trainees' progress towards competencies to enter independent practice, or contribute to producing clinical workload.

This study assesses the performance of an electronic, fully automated, Clinical Supervision Index [CSI] that was developed by VA and its Office of Academic Affiliation, to permit VA leadership to understand how costly supervision impacts patient's health outcomes, assess trainees' professional progress towards being able to engage in licensed, independent clinical practice, and measure trainee contributions to clinical workload.


OBJECTIVE(S):
We assess (i) CSI concurrent validity to determine if the lengths of supervised time that trainees and supervisors report in CSI surveys agree with time computed from direct observations, (ii) construct validity to determine if supervision intensity quantified by computing the Clinical Supervision Index from responses to the CSI surveys predictable, and (iii) and applicability to determine if adjusted CSI intensity scores reflect trainee's clinical competencies and predict patient care outcomes, patient satisfaction with trainee involved VA care, trainee education outcomes, and trainee satisfaction with their learning experiences.

METHODS:
To assess performance, data were obtained on a panel of patients seeking care over a one-year follow-up period at the Continuity Care Clinic at the VA Loma Linda Healthcare System, and who had a diagnosis of type II diabetes. Study also included physician trainees (medical students, interns, residents, and fellows) who engaged in outpatient care for the panel of consenting study patients during the patient enrollee's one-year follow-up period, and the physician trainee's attending physician supervisor. All three groups, the enrolled study patient, their physician trainee, and their attending physician supervisor, were enrolled and signed an IRB-VAMC R&D approved informed consent. Data came from (i) the electronic, automated administration through VA's electronic health record of the CSI surveys to both the trainee and his or her VA attending physician supervisor for all patient care encounters defined as an entry to the patient's medical record (progress note) for study enrolled patients during their respective one-year follow-up period; (ii) VistA extracts of the patient health record including clinical outcome data on enrolled patients during the one-year follow-up period, (iii) Computer Data Warehouse extracts through SQL to compute daily workloads per supervising attending physician and physician trainee by date, clinic stop code, facility location, and patient and provider identification numbers; (iv) face-to-face and telephone interviews administering at baseline and 4 quarterly follow-up surveys during the one-year follow-up using the Utilization and Cost Inventory, (v) web-based trainee satisfaction surveys administered to enrolled study physician trainees during the two-year trainee follow-ups, (vi) vetted observations where study staff observed trainee supervision during sampled study patients' clinical encounters, and (vii) records of trainees regarding their USMLE and in-service training examinations results during the 2-year trainee enrolled study period. The unit of analyses is the patient progress note that accounts for supervised encounters with the study patient in both outpatient and inpatient care settings. Analyses include intraclass correlations, mean response biases, and effect sizes computed from Generalized Estimating Equations to account for hierarchical trainee, supervisor, and patient encounter level data.

FINDINGS/RESULTS:
The original protocol specified n=300 study enrolled patients to power hypothesis testing. Between August 7, 2014 through September 28, 2016, a total n=370 patients (enrolled rate: 370/300=123%) were recruited, signed an IRB/R&D Committee approved informed consent, and completed one year follow-up (or were deceased n=5 from causes determined to be non study-related). Patient recruitment was not difficult, achieving a 98% enrollment rate among referrals. In fact, patients who had learned about the study from other patients but otherwise were not referred by clinical staff, came to us and asked if they could enroll in the study, claiming they wanted to be involved in a study that had trained, supervised, study staff observing the care their were receiving from their physician. Demographically, these completed patients were 76% retired, 5% unemployed, and 19% employed either full-time, part-time, or self-employed; 52% were married, with 34% divorced or separated, 7% widowed, and 6% never married; 51% were white or Caucasian, with 22% African American, 22% Hispanic or Latino, 1% Native American or Pacific Islander, or 4% other. The patient panel was highly educated, with 93% completing a high school education or higher, with 75% having post-high school graduate college or trade school, with 9% having a graduate degree. Only 4% of patients were female. The mean age was 63.7, median age was 65.0, and standard deviation (s.d.) was 8.98, with ages ranging from 31 - 88 years of old. The mean monthly income from all sources, including earned and unearned income, was $3,026/month (median=$2,500/month, s.d.=$2,544), with monthly income ranging from 0% (family supported) to $21,350. Study patients were permanent residents of the area, with 17% owning their own home, 37% buying their home, and 38% rending, with 8% living with a friend or relative or were homeless. Most patients were optimistic about their care, with over half of patients agreeing or strongly agreeing that if they get the care they needed from VA, their health, well-being, and quality of life would significantly improve. During their one-year follow-up, 33% of patients had used a non-VA provider for health care. The 117 patients who were using non-VA care averaged 5.78 clinic visits/yr. (s.d.=8.36). Study patients who used non-VA care averaged 25.61 visits/year (s.d.=22.88), with a median of 20 visits/year.

There were 98 physician residents who qualified for the study, signed an IRB-approved informed consent, and engaged in caring for an enrolled, consented study patients during a clinical encounter with a service date that fell within the respective patient's one year follow-up period. Our original estimates were n=90, for a trainee sample rate of 98/90=108.9%. Of these, 88 (90%) had completed the Learners' Perceptions Survey on line to describe their satisfaction with the clinical, learning, and working environments of their VA rotation. All 98 trainees completed the Trainee Supervision Survey for total 752 supervision encounters with study patients. Among the 98 enrolled residents, 56% had one supervision encounter where they completed the trainee supervision survey, 30% had between 2-4 such encounters, and 14% had 5 or more such observed supervision encounters. Among these supervised encounters, 216 were observed and recorded by study staff, as administered between March 25, 2015 and November 6, 2017.

There were a total 2,965 eligible supervision encounters with service dates falling between March 25, 2015 and November 6, 2017, where the trainee CSI survey was properly administered to the trainee engaged in patient care (i.e., signed the progress note). This compares with 2,400 such encounters powered in the original study protocol, for an enrollment rate of 123.5%. These supervision encounters involved a total 562 trainees for supervision encounters. Among these encounters, 2,948 provided complete responses to the survey, for a 99.4% survey completion rate. For the two-year period, the average trainee completed 5.28 (s.d.=7.08) surveys, with 95% of all trainees having completed 20 or fewer surveys during a two-year period. Overall, 1,836 (62.3%) included care discussion between trainee and supervisor, for a mean CSI value of m = 0.20 (s.d.=0.24), while 2,321 (78.7%) encounters possessed at least an attending physician in the area for a CSI = 0.71 (0.41), with resident assisting in care CSI = 0.41 (0.42), supervisor directly observing CSI = 0.12 (0.25), supervisor in area CSI = 0.35 (0.44). For the 628 completed outpatient supervision encounters, 371 (59.1%) included a care discussion with a mean CSI = 0.18 (0.21), with 531 encounters possessed at least an attending physician in the area (84.6%) for a patient care CSI = 0.80 (0.38), with resident assisting in care CSI = 0.37 (0.40), supervisor directly observing CSI = 0.12 (0.25), supervisor in area = 0.51 (0.47).

The system also captured responses from the trainee's supervising practitioner. The supervisor CSI survey was limited to oversight activities that describe what preparations the supervisor took to oversee the trainee's interaction with patients at the VA medical center. Among the 2,948 encounters where 562 trainees had completed a trainee survey, the supervising physician had been identified in 2,938 encounters (99.9%). However, after identifying the supervising practitioner and his or her VA email address, the computerized search engine captured only 289 submitted surveys (9.8%), with 243 (84%) of such captured surveys were completed. While critical measures of the CSI depend only on the trainee survey, the poor response to the supervisor survey is considered the result of using emails to get medical staff to complete the survey days following the actual encounter. We have recommended that the CSI system be changed so that the supervisor survey is captured at the time the supervisor signs the progress note. In the original protocol, we elected using emails to capture supervisor responses to CSI survey to accommodate those hospitals that do not require supervisors to sign the progress note. Reviewing supervisor responses, overall attending physicians reported spending a median 15.00 minutes engaged in a chart review, discussion with clinical staff, and direct assessment of the patient, with 35% of encounters the supervisor changed or annotated the medical chart, and 13% re-directed staff to alter a trainee's choice of diagnostic tests, interpretation, or treatment plan. Not surprisingly, psychiatry and medicine subspecialties (cardiology) reported the most hours a supervisor spends reviewing a trainee's chart, talking with staff, and directly contacting the patient to oversee the trainee. The least amount of time was provided by internal medicine and 'other' medicine. On the other hand, older supervisors were less likely to request clinical staff to alter the patient's care plan as offered by the trainee.

Predictors of supervision intensity were as expected (p<.001). Adjusting for nesting by patient and supervisor, all physician trainees regardless of level (medical student to PGY4 or higher), had a supervisor at least in the area whenever trainees were engaged in patient care. However, supervision varied by level of experience for higher levels of supervision. For example, intensity of direct observation supervision was less as trainees moved from medical student status, to PGY1 through PGY3, and into fellowship (PGY4+). On the other hand, trainees were increasing likely to assistant their supervisor engaged in patient care with increasing trainee experience. Finally, while most trainees received some case discussion with their supervisor, intensity generally increased with increasing experience, reflecting trainees who engage in more demanding tasks that require more skill levels. Supervision also varied by gender, age, and ethnicity of the trainee.

IMPACT:
Findings from this study are being provided to the Office of Academic Affiliations to roll out nationwide as a means to monitor the extent to which health professions trainees are supervised while engaged in the caring for patients at VA medical centers, and to measure trainee education progress as a fundamental performance assessment tool to evaluate whether the VA is completing its education and training mission. The CSI data collection software will be made available to further transitions of VA's electronic health record for automated data collection and analyses.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX001582-01
Link: https://reporter.nih.gov/project-details/8784044

---

PUBLICATIONS:

None at this time.

DRA: Health Systems
DRE: Diagnosis
Keywords: none
MeSH Terms: none