3096 — Supporting VA EPRP Measures for Vaccination Using Text-Processing and NLP Methods

Gundlapalli AV (University of Utah), Phansalkar S (University of Utah), Shen S (University of Utah), Delisle S (University of Maryland, Baltimore Veterans Affairs Medical Center), Perl T (Johns Hopkins University)

Objectives:
The VA Office of Quality and Performance’s External Peer Review Program (EPRP) currently relies on manual chart review to identify documentation of influenza and pneumococcal vaccination. This study demonstrates the utility of using automated methods to extract vaccination information from VISTA.

Methods:
This study was a retrospective review of a random sample of 15,377 unique outpatient encounters selected for chart review for 70,743 patients having at least one outpatient encounter at either the VA Maryland or Salt Lake City Health Care Systems between October 2003 and March 2004. These encounters included the emergency department, ambulatory care, and specialty clinics where specific services would be provided for diagnoses related to acute infectious disease, and opportunities for prophylactic vaccinations. Our reference standard for identification of vaccination for influenza or pneumonia was based on 1) structured data sources (ICD-9-CM, CPT-4 codes, or recorded in the VISTA Immunizations log) indicating vaccination; or 2) the combination of structured data and chart review results identifying vaccination. The full texts of 76,500 notes corresponding to these encounters were processed using a Natural Language Processing (NLP) system called MedLEE. Additionally, we applied a simple text-processing approach to identify influenza or pneumococcal vaccination using string matching coupled with a negation detection algorithm called NegEx. Statistical performance of each method of identification of vaccination documentation was determined.

Results:
Altogether,1,740 (11%) patients were flagged from administrative data sources as having received either vaccination. In comparison, the simple text-processing approach flagged 2,613 (17%) patients and the NLP system 2,303 (15%) patients with vaccination documentation. For patients receiving outpatient services over the six month study period, 55% of patients from the VA Maryland Health Care System had any one of the structured data sources documenting either type of vaccination. Aggreement between combined data obtained coded data sources and immunization log data compared with NLP was high. Statistical performance of document identification by each method based on arbitrated chart review will be presented.

Implications:
Further validation of these methods is needed against available EPRP data.

Impacts:
Though we focus on identifying vaccination against influenza or pneumonia from non-structured VA data sources, these methods could be expanded to support other EPRP measures that rely on data abstraction from VISTA using manual chart review.