1094 — Measuring Use of Evidence-Based Psychotherapy for Post Traumatic Stress Disorder
Shiner B, NCPTSD, VERC; D'Avolio LW, and Nguyen TM, MAVERIC; Zayed MH, White River Junction VAMC; Watts BV, NCPS, VERC;
Recent studies have used administrative data to characterize the quality of care for PTSD in VHA clinics. A key limitation to these studies is that they cannot determine which psychotherapy protocols Veterans receive. Research shows that some psychotherapy protocols are more effective than others in the treatment of PTSD. Chart review can generate detailed information about clinical processes, including psychotherapists' reports of the protocols they used, but is time-consuming and difficult to apply on a large scale. Using natural language processing (NLP), we developed a method of evaluating the quality of psychotherapy that combines the scale and efficiency of administrative data analysis with the granularity of chart review.
We obtained administrative data and all clinical notes for Veterans who newly presented to the six outpatient VHA specialized PTSD clinics in New England during the first half of fiscal year 2010 (n = 1,924). We used the Automated Retrieval Console (ARC), a VA-developed NLP application, to detect whether empirically-supported psychotherapies endorsed by VA, prolonged exposure (PE) and cognitive processing therapy (CPT), were delivered. We began by manually coding all notes for 10% of the patients, and then trained ARC to replicate the manual coders' performance. We then used ARC to perform automated coding of the rest of the document set. Finally, we determined treatment receipt using both NLP and administrative data analysis.
Our manual coding team achieved a high level of agreement (K = 0.88). ARC's performance in replicating the manual coding team's ratings on the initial set of 7,546 notes was excellent (recall, precision, and f-measure >0.90). Therefore, we used ARC to code the entire set of 84,561 notes. Of these, 12,762 were coded as individual psychotherapy. Out of the individual psychotherapy notes, 169 were coded as PE and 575 as CPT. Patients received an average of 6.6 individual psychotherapy sessions in their initial 6 months. This was smaller than the estimate of 9.1 sessions made using administrative data. There were 121 patients (6.3%) that received at least one session of PE or CPT. Among the six sites, the rate ranged from 3.7% to 13.5%. Patients enrolled in these protocols received an average of 6.1 sessions of PE or CPT, but 14.4 sessions of any individual psychotherapy. The number was inflated to 16.1 sessions when using administrative codes.
We have developed a reliable method to measure the use of two evidence-based psychotherapy protocols for PTSD in VHA.
This method could be used to provide measurement to guide efforts to improve the use of evidence-based psychotherapy.