HSR&D Home » Research » IIR 06-119 – HSR&D Study
From Syndromic to Disease-Course Surveillance
Sylvain Delisle, MD MBA
Baltimore VA Medical Center VA Maryland Health Care System, Baltimore, MD
Funding Period: October 2007 - December 2011
Early illness detection is essential to minimize morbidity and mortality in the event of infectious disease outbreaks of public health significance e.g. severe influenza, SARS or bioterrorism. The automated monitoring of electronic health information offers the potential to enhance disease surveillance compared to manual case reporting. The selection of electronically extractable data elements affects an automated system's ability to detect events at the earliest stage. Today's most sophisticated systems are limited to a narrow range of electronic data types, often obtained from unrelated sources. A system that could relate a broad array of clinical data around unique individuals could identify both the incidence and the time course of illnesses, and thus be helpful not only in the recognition but also in the management of epidemics.
The overall objective was to automate the use of CPRS data to enhance efficiency, timeliness, and validity of detection of outbreak of acute respiratory infections (ARI). We hypothesized that expanding the scope of an automated surveillance system to include illness progression and severity would shorten the time it takes to recognize disease outbreaks that pose a significant threat to veterans and public health.
Our specific aims were to:
1) Refine and validate automated methods to interpret ARI progression and severity;
2) Develop and validate statistical signals for surveillance that combine disease incidence and severity;
3) Use simulated outbreaks of ARI to evaluate the sensitivity and specificity of statistical models that account for disease severity compared to models that do not;
4) Vary simulated ARI outbreak characteristics so as to optimize the automated surveillance methods developed in Aims 1 and 2 for timely, sensitive and specific outbreak detection;
5) Initiate real-time electronic transmission of health record data to support public health disease surveillance.
In August 2009, a project modification added the following aims:
6) Apply previously validated ARI case-detection algorithms (CDAs) that utilize CPRS data to create time-series of daily counts of patients with possible ARI;
7) Develop seasonally-adjusted auto-regressive integrated moving average (SARIMA) forecasting models for each of the above time-series;
8) Compare the performance of ARI surveillance systems that use alternative aberrancy-detection methods: a) one which utilizes long-term historical data for forecasting (SARIMA); or b) one which utilizes data from the recent past only.
1 Development of ARI case-detection algorithms (CDAs).
CPRS data was extracted and transferred to relational databases. To build CDAs, relevant data were grouped and classified along different dimensions of respiratory disease severity: diagnostic and procedures codes, type of health care delivery, therapeutic and monitoring efforts, physiological, laboratory and imaging results, and the free text of clinical notes and XR reports. CDAs were created using iterative regression and tested against a reference manual structured chart review for their ability to identify ARI. The most successful ARI CDAs combined free-text analyses and structured, and have been published in PLoS ONE, 2010 5(10) e13377 and in PLoS ONE, 2012 7(12):e51147. Different aspects of the text analyses have also been published (AMIA Annu Symp Proc 2008:692; AMIA Summits Transl Sci Proc Mar 1; 2008:36; AMIA Summits Transl Sci Proc Mar 1; 2010:56).
2 Development of whole-system simulations (WSS).
We recreated historical background case count time series by applying the most successful CDAs to past CPRS data. We injected factitious influenza cases to CDA-specific backgrounds using an age-structured influenza epidemic model. From the time of this injection, aberrancy-detection statistics were applied each successive day (n = 50-80) on paired background+injection vs. background-only time series. Each injection-prospective-surveillance cycle was repeated 52 times, each time shifting the injection to a different week of the study year. We computed three whole-system benchmarks: 1) the "Detection Delay", the average time from injection to the first true-positive signal, defined as a statistical alarm originating in the background+injection dataset but not present in the background-only dataset; 2) the "False-Alarm Rate" (FAR), defined as the number of unique false-alarms originating in the background-only dataset during the study year, divided by 365 days; 3) the "Caseload", the yearly number of cases included in the false alarms. System performance was compared using activity monitoring operating characteristic (AMOC) curves relating Detection Rate or Delay to FAR or Caseload.
1 Aims 1, 3 and 4.
We identified three ARI subgroups who may have more severe disease: 1) ARI patients with fever; 2) ARI patients who seek emergent care; 3) outpatients with pneumonia.
A) Targeting ARI patients with fever. Seven (7) percent of our ARI patients had fever, compared with 70% of influenza patients. Thus, monitoring for febrile-ARI reduced sensitivity by 30%, but also eliminated 93% of afebrile ARI background cases thereby improving signal-to-noise ratio (S/N). Eight ARI CDAs that regrouped information about diagnostic codes, cough remedies and a computerized text analysis seeking ARI symptoms in free-text of the clinical note were compared against otherwise identical CDAs that also required patients to have an elevation in body temperature when seen. AMOC curves revealed that for any given FAR within a practical range (0-10%), CDA that targeted febrile patients yield lower detection delays than CDA that targeted all ARI patients. Surveillance caseload was also greatly reduced at any given detection delay.
Status: Completed. These results are being prepared for publication.
B) Targeting ARI patients seeking urgent care. We observed that two-thirds of the cases with ARI were seen in urgent care areas, but that these areas received only 15% of total outpatient visits. Because of a favorable S/N ratio, we hypothesized that an ARI surveillance system that focused on urgent visits would outperform one that monitored all outpatient visits, even though that system would ignore a significant number (1/3) of outpatients with ARI. To test this hypothesis, we applied one of eight different ARI CDA to CPRS entries related to "All" or to "Urgent-only" outpatient encounters. WSS was then used to generate paired AMOC curves of otherwise identical surveillance systems that included either "All" or Urgent-only visits. In each case, simulated surveillance systems aimed at Urgent-only visits resulted in Detection Delays that were lower at any given FAR or Caseload.
Status: Completed. These findings are being prepared for publication.
C) Targeting ARI patients with pneumonia. We applied a sensitive ARI CDA (99%) to documents related to all outpatient visits from 01/01/2004 to 12/31/2006 (n = 2.7 million). All associated chest imaging (CXR) reports (n = 2,747) associated with CDA-flagged encounters (n = 22,960) were manually reviewed and classified as "non-negative" if findings could possibly support the diagnosis of pneumonia. We then reviewed all related CPRS entries to determine if the patients had a "Possible Pneumonia" (non-negative imaging report with either cough, sputum, fever/chills/night sweats, dyspnea or pleuritic chest pain) or "Pneumonia-in-Plan" (non-negative imaging report with pneumonia stated as a likely diagnosis by the provider). The review identified 370 reference cases with Possible Pneumonia and 250 with Pneumonia-in-Plan. CDAs aimed at detecting these reference cases using ICD-9 codes, chest imaging requests, elevated temperature and other structured data elements from CPRS were sensitive (90-100%) but inaccurate (positive predictive values (PPV) of 13-22%).
To improve CDA accuracy, the manually classified chest imaging reports served to train machine-learning software that produced an imaging report text classifier. When combined with EMR-based structured data and with text analyses aimed at ARI symptoms in clinical notes, the x-ray report text classifier increased the positive predictive value pneumonia by 20-70%, while sensitivities of 58-75 % were retained.
Status: Completed. These results have been published (PLoS One, 2013 8(8) e70944).
2 Aims 2 and 3.
To begin developing statistical signals for surveillance that combine disease incidence and severity, we have used a simplified categorization framework, assigning ARI cases as either outpatients (cat1), hospitalized (cat2), or dead (cat3). Even when using the most sensitive ARI CDA, there were only three patients who died within 24 hours of their ARI encounter (cat3) in the last 8 years in the Baltimore dataset. Thus, all of cat3 instances were set to generate an alert. To develop a daily score that incorporates information about disease severity, we assigned a range of weights by which cat2 counts were multiplied before being added to cat1 counts. We used WSS to evaluate this daily score over a broad set of weights using hospitalization rates consistent with severe influenza. The resulting AMOC curves indicated that system performance worsens with increased cat2 weight.
Only a small proportion of patients with ARI exhibit markers that suggest a more severe disease. The proportion of ARI patient who: 1) are febrile is 7%; 2) have an abnormal CXR is 2.5%; 3) are admitted to the hospital is 2.5%. Thus, daily case counts of patients who exhibit any one of these markers are low, with many days showing no single instance. To incorporate disease severity into ARI surveillance, we therefore had to develop statistical aberrancy detection approaches that focus on sparse datasets. We chose to develop a general yet parsimonious method that could account for both large and small shifts in daily counts: a generalized likelihood ratio (GLR) control chart for zero-inflated, Poisson-distributed (ZIP) data. Using a range of purely simulated datasets, we have now shown that GLR-ZIP approach outperforms a recently published alternative, CUSUM-ZIP. We have implemented GLR- ZIP detection models in software, determined model parameters and control limits over a range of simulated data compatible with our authentic background datasets of patients with pneumonia.
Status: The results comparing GLR-ZIP to CUSUM-ZIP are being prepared for publication. In a recent grant application, we propose to evaluate the relative performance of these new aberrancy detection methods for pneumonia surveillance.
3 Aim 5.
The above work begins to highlights the priorities for CPRS data extractions and processing and the most fruitful surveillance strategies. We have now succeeded in developing a JAVA application that automates the sequence of database queries, text processing and statistical analyses necessary to perform surveillance using operational CPRS data.
Status: Proof-of-concept programming completed but not fully tested. We are currently updating software documentation before release to the NLP and surveillance community.
4 Aims 6, 7 and 8.
To test the hypothesis that aberrancy-detection methods that incorporate seasonality and other long-term data trends would reduce the time required to discover an influenza outbreak, we created 8-year authentic background time series of daily case counts by applying either of two ARI CDA to CPRS-derived data: 1) CDA1, ICD-9 codes set; 2) CDA2, ICD-9 codes combined with text analyses seeking ARI symptoms. Specific seasonal autoregressive integrated moving average (SARIMA) models were developed for each background. We used WSS to compare systems that used either: a) CDC's EARS W2c, which makes statistical predictions using 4 weeks of past data; b) SARIMA. At any given FAR within a practical range (0-10%), the SARIMA methods yielded lower detection delays than EARS W2c. These data suggested that forecasting approaches that incorporate long-term data trends can significantly improve the performance of surveillance systems.
Status: Completed. These results are now being prepared for publication.
Our work indicates that information from a comprehensive EMR can be leveraged to improve both the effectiveness and the efficiency of automated surveillance systems for ARI. Fruitful sources of information are diagnostic codes, prescriptions, and the text of clinical note and chest imaging reports. Our work show that ARI surveillance systems should preferentially seek accurate rather than sensitive case-detection algorithms, and target subsets of ARI patients who may have more severe disease, such as those who are febrile and those who are seen in the emergency room. This work highlights the usefulness of developing simulation-based empiric approaches to guide the development of CPRS-based disease or event surveillance systems, in general. New pneumonia case-detection methods were developed that could serve not only to improve surveillance performance, but could also help evaluate the quality of care of patients with community-acquired pneumonia.
External Links for this Project
Dimensions for VADimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.
If you have VA-Intranet access, click here for more information vaww.hsrd.research.va.gov/dimensions/
VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address. Search Dimensions for this project
DRA: Infectious Diseases, Health Systems
DRE: Epidemiology, Technology Development and Assessment, Research Infrastructure
Keywords: Bioterrorism, Research method
MeSH Terms: none