2011 HSR&D National Meeting Abstract
1042 — Combining Natural Language Processing and Statistical Text Mining: Classifying Fall-Related Progress Notes
Jarman J (HSR&D/RRD Center of Excellence: Maximizing Rehabilitation Outcomes), Luther SL
(HSR&D/RRD Center of Excellence: Maximizing Rehabilitation Outcomes), McCart JA
(HSR&D/RRD Center of Excellence: Maximizing Rehabilitation Outcomes), Berndt DJ
(University of South Florida)
The purpose of this research is to compare the classification accuracy of statistical text-mining (STM) using two sets of terms from VA progress notes: (1) all terms and (2) only medically-relevant terms (MRTs).
Progress notes for Veterans and two matched controls from one VAMC with ICD-9-CM codes documenting an outpatient visit for a fall-related injury (FRI) were identified using administrative data. This resulted in 247 Veterans and 5,009 text notes (NOTES). Each note was reviewed and classified by clinicians with 1,151 notes having documentation of treatment for a FRI and 3,858 having no documentation for a FRI. A GATE (General Architecture for Text Engineering) Natural Language Processing (NLP) pipeline was constructed to separate the MRTs from the common language. This was accomplished by using the pipeline to extract UMLS (Unified Medical Language System) concepts from the notes to create a second dataset (MR). After splitting each dataset into identical Training and Testing samples (70%/30%), STM was performed using SAS Text Miner 4.2 on the datasets (NOTES and MR) and several models were tested for each. The model with the best accuracy was selected for each dataset based on the statistics from the Training sample.
Across the samples, the average note was made up of 30% MRTs (105 terms). The following statistics are from the best STM models using the Test sample (NOTES/MR) (%): Accuracy – 91.53/92.59, Sensitivity – 81.79/85.22, Specificity – 94.45/94.80, Positive Predictive Value – 81.56/83.05, and Negative Predictive Value – 94.53/95.54.
A STM model using MRTs was able to classify the fall notes with slightly better accuracy than a model using the full text from the notes. These results imply that important patterns are not lost and that efficiency can be increased by removing common language when using STM. This can serve as a foundation for other data mining research, such as targeted information extraction and even clinical discovery.
FRIs are an important health care issue, particularly in the aging Veteran population. Administrative data likely under record the extent of the problem. Results of this study suggest the combination of NLP and STM can reliably identify FRIs in the ambulatory care setting.