skip to page content
Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

VA Health Systems Research

Go to the VA ORD website
Go to the QUERI website
Publication Briefs

Using Administrative Data to Measure Treatment for Veterans with PTSD May Overestimate Delivery of Psychotherapy


BACKGROUND:
In recent years, studies have used various methods to characterize the quality of care for PTSD delivered in VA outpatient clinics. One common method is counting the number of individual encounters administratively coded as psychotherapy as a measure of treatment adequacy. However, it is possible that this method results in an over-estimate of psychotherapy treatment, and that manually reviewing patient records, while more labor intensive, is more accurate. This study sought to determine whether using administrative data to determine the number of psychotherapy sessions Veterans receive is equivalent to manual record review. Investigators identified 100 consecutive Veterans who presented to one VA mental health clinic and screened positive for PTSD. All notes administratively coded as individual psychotherapy in the six months subsequent to a positive screen were identified. Two psychologists then independently reviewed the notes and classified whether or not the encounter was psychotherapy, using criteria from another published VA study. The manually-classified notes were used to develop an automated coding protocol using the Automated Retrieval Console (ARC), a VA-developed natural language processing program. ARC was then used to independently code the notes, and the performance of the automated coding program was compared to manual coding.

FINDINGS:

  • Of the 221 notes that were administratively coded as individual psychotherapy for PTSD, 126 (57%) were coded as individual psychotherapy after manual review of records. Thus, nearly half of the encounters that would have been counted as the provision of psychotherapy in large administrative studies appeared to be records of services other than psychotherapy (e.g., intakes, psychological testing).
  • Findings suggest that using counts of administrative codes over-estimates the amount of psychotherapy delivered to Veterans with PTSD. This suggests a potential limitation in current studies of the quality of care for PTSD in VA.
  • The ARC program replicated the performance of the manual coders in classifying psychotherapy notes very well. This suggests that ARC may help bridge the gap between the accuracy of manual coding and the scope of administrative coding. However, additional research is required on larger datasets prior to drawing statistically significant conclusions.

LIMITATIONS:

  • These findings are based on observations made at a single VA site.
  • The number of cases reviewed was not large enough to draw statistically significant conclusions.
  • Multisite validation of these findings and further development of ARC for this application is required before these findings can be considered generalizeable.

AUTHOR/FUNDING INFORMATION: This study was partly funded by HSR&D’s Consortium for Healthcare Informatics Research (CHIR). Dr. Shiner is the recipient of a VA New England Early Career Development Award and is part of the VAMC in White River Junction, VT.


PubMed Logo Shiner B, D’Avolio L, Nguyen T, et al. Automated Classification of Psychotherapy Note Text: Implications for Quality Assessment in PTSD Care. Journal of Evaluation in Clinical Practice 2011 Feb 14;e-pub ahead of print.

Related Briefs

» next 21 Data Use Briefs...


» next 224 Mental Health Briefs...


» next 90 PTSD Briefs...


What are HSR Publication Briefs?

HSR requires notification by HSR-funded investigators about all articles accepted for publication. These journal articles are reviewed by HSR and publication briefs or summaries are written for a select number of articles that are then forwarded to VHA Central Office leadership to keep them informed about important findings or information. Articles to be summarized are selected by HSR based on timeliness of the findings, interest of leadership, or potential impact on the organization. Publication briefs are written for only a small number of HSR published articles. Visit the HSR citations database for a complete listing of HSR articles and presentations.


Questions about the HSR website? Email the Web Team

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.