2011 HSR&D National Meeting Abstract
3090 — Automated Classification of Psychotherapy Note Text
Shiner B (VISN 1 VERC, NCPTSD), D'Avolio LW
(MAVERIC), Nguyen TM
(MAVERIC), Watts BV
(VISN 1 VERC, NCPS), Fiore LD
Recent studies have employed a count of individual psychotherapy encounters as a measure of quality of care for Veterans with PTSD. Due to potential billing errors, it is possible that this methodology results in an overestimate of the amount of psychotherapy delivered. Manually reviewing individual records can yield a more accurate picture. However, this labor-intensive method could not be efficiently applied to national treatment studies, which require automated technologies. We sought to adapt one such technology for this purpose.
We identified 100 consecutive Veterans presenting to a VA walk-in mental health clinic who screened positive for PTSD using the PTSD Checklist. We identified all notes billed as individual psychotherapy in the six months subsequent to a positive screen. Two psychologists independently reviewed the notes and classified them as to whether or not the encounter was indeed psychotherapy using a coding scheme previously published elsewhere. When they disagreed, a psychiatrist acted as adjudicator.
After classification of the notes, we loaded them into the Automated Retrieval Console (ARC), a VA-developed natural language processing program. Using our manual coding as a reference, ARC performed a series of iterative evaluations of different combinations of features and classifiers to determine the best classification model. We then validated the model against the manual coding.
We identified 221 individual psychotherapy encounters. In the manual review, we classified 57% of the associated notes as psychotherapy. The remainder were intakes, psychological testing, and case management notes. ARC ran a total of 52 different top scoring feature set-classifier combinations and determined the most appropriate algorithm using a 10-fold cross validation. The top scoring combination was word tokens plus a support vector machine-based classifier for a recall of 0.97 (akin to sensitivity), a precision of 0.90 (akin to sensitivity), and a harmonic mean of 0.93 (akin to ROC).
ARC replicated the performance of a team of mental health professionals in classifying psychotherapy notes almost perfectly.
ARC has potential to improve the accuracy of studies of quality of care for PTSD in the VA. Multisite validation studies and classification of multiple psychotherapy types would further increase the relevance of these findings.