3070 — Identification and Classification of Medically Unexplained Symptoms in VA Clinical Documents
Palmer M (SLC IDEAS Center, VA Salt Lake City Health Care System), South BR
(SLC IDEAS Center, VA Salt Lake City Health Care System), Shen S
(SLC IDEAS Center, VA Salt Lake City Health Care System), Tuteja A
(SLC IDEAS Center, VA Salt Lake City Health Care System), Divita G
(SLC IDEAS Center, VA Salt Lake City Health Care System), Samore MH
(SLC IDEAS Center, VA Salt Lake City Health Care System), Gundlapalli AV
(SLC IDEAS Center, VA Salt Lake City Health Care System)
Our goal was to develop guidelines and an annotation schema that can be consistently applied to build a reference standard used to train and evaluate an automated Natural Language Processing (NLP) system to extract medically unexplained syndromes (MUS) found in VA clinical documents. These tasks also help characterize the occurrence of MUS in Operation Enduring Freedom (OEF)/Operation Iraqi Freedom (OIF) veterans.
We randomly sampled 492 clinical encounter documents from a cohort of OEF/OIF veterans who received health care services between 01/01/2007–12/31/2010 obtained from the Veterans Informatics Computing Infrastructure (VINCI). Annotated data represent spans of text identifying symptoms and contextual features for assertional information, and symptom duration. For a random sample of 15 documents annotated by four reviewers we report the number of annotations and the number that could be included in non-mutually exclusive symptom constellations for the three most common MUS: irritable bowel syndrome (IBS), fibromyalgia, or chronic fatigue syndrome (CFS). Post hoc review by two clinicians was used to classify annotated information into symptom constellations. We report agreement of clinician reviewers at the level of symptom classification.
The number of annotations (unique mentions) for four annotators was 1,358 (583) overall (mean of 17 annotations per document). Of these 1,010 (477) were identified as possible symptoms, 262 (53) assertions, and 86 (53) represented symptom duration. Kappa statistic for classification of annotated spans was low (< = 0.15) for all symptom clusters except IBS (0.48). Clinician post hoc review of the 477 unique symptom annotations revealed non-mutually exclusive symptom clusters of which 274 (58%) could potentially describe CFS, IBS 31 (6%), and 154 (32%) fibromyalgia.
These results illustrate the difficulty of classifying symptoms into MUS categories. Further development of our annotation guidelines will help improve performance of annotators and clinicians at each of the review stages.
Using an automated NLP system to identify and examine symptom clusters will provide data that can be used to continually assess the health status of OEF/OIF veterans. The ability to measure the burden of MUS symptoms has the potential to provide more comprehensive health care services to all veterans.