BACKGROUND/RATIONALE:
Chronic Heart failure (HF) is the primary reason for discharge for veterans treated within the Veterans Administration health care system (VA). An integral part of VA's quality transformation has been the use of quality measurement as a feedback mechanism. Automating performance measurement for HF holds promise to lower the substantial labor-costs of manual chart abstraction by health professionals, conserving health professional time for more complex cases. Because of routine use of the electronic health record within VA and the increasing availability of information extraction (IE) techniques for text, automating data capture for performance measurement is a realistic pursuit. We used the Promoting Action on Research on Implementation in Health Services (PARIHS) framework as our conceptual framework for stakeholder engagement because prior studies have shown that Health Information Technology can be used as a facilitator to implement evidence-based clinical guidelines.

OBJECTIVE(S):
The overall objective of the research was to study the use of information extraction (IE) techniques and associated automation of treatment performance metrics for chronic heart failure patients within the VA.

METHODS:
The study included a sample of inpatients diagnosed from Region 1 with chronic heart failure discharged in fiscal year 2008 whose charts were abstracted by the External Peer Review Program (EPRP). We targeted key concepts for automated extraction based on the External Peer Review Program (EPRP) required documentation elements consisting of; left ventricular systolic function (LVSF) assessment, angiotensin-converting enzyme (ACE) inhibitor, or angiotensin receptor blocker (ARB) therapy for appropriate patients, and reasons why these patients may not be on medications, for example contraindications. We assembled the data elements for the HF performance indicators using structured data from VISTA and IE techniques with patient documents. We constructed and tested automated algorithms to generate the data needed for the measure.
In order to compare the data capture for the two distinctly difference work processes of EPRP manual abstraction and the automated IE process, we calculated sensitivity and specificity between; 1. EPRP results and the reference standard we developed through human review, and 2. EPRP results and the IE output, both at the patient and facility levels.
We undertook a stakeholder engagement process using a snowball sampling technique to identify interviewees. We developed an interview guide, summarized the main points in the interview, sent summaries to the interviewees for validation, and then used an applied thematic analysis approach with validated summaries to generated themes (codes) to answer our stakeholder research questions.

FINDINGS/RESULTS:
The study cohort was comprised of 1390 inpatients. The text notes for the patients in the cohort were obtained and organized into batches of one batch per patient. Of those batches, 314 were used for training and 771 were used for testing. We trained and tested the Congestive Heart Failure Information Extraction Framework (CHIEF) NLP system. The CHIEF NLP system had 98.9% sensitivity with a positive predictive value of 98.7% in classifying documented care as guideline-concordant.

We also identified the sections within documents where a high prevalence of the concepts were found across all documents. Identification of sections assisted the CHIEF NLP system in targeting relevant concepts for extraction.

Within the CHIEF NLP system, we developed the following modules pertinent to the inpatient performance measure for chronic heart failure: EF detection and extraction module (which we used as a proxy for LVSF assessed) with a sensitivity of 1, Positive Predictive Value (PPV) 99.0% and F measure 99.5%; classification of whether the EF was less than 40% or not with a sensitivity of 96.8%, PPV of 95.1% and F measure 95.9%; a medication detection module with a sensitivity of 99.1%, PPV 88.5% and F measure 93.5%; and a module which detected and extracted reasons for instances of non-guideline-concordant medication with a sensitivity of 26.8%, Positive Predictive Value (PPV) 90.6% and F measure 41.4%.

When compared to EPRP data, the overall sensitivity and PPV for the reference standard we developed through human review was found to be 89.44% and 49.90%, respectively. There were statistically significant differences (p<0.001) in PPV between facilities. The overall sensitivity and PPV for IE when compared to the EPRP data was 88.12% and 49.70%, respectively. The IE showed statistically significant differences in facility-specific performance in both sensitivity (p<0.05) and PPV (p<0.01). We hypothesize the differences are due to variations in document formats for each facility and also potentially relate to the documents sets for EPRP review and the automated process respectively. Specifically, EPRP reviews the entire record while we used a limited document set in our automation.

We interviewed 13 stakeholders during the stakeholder engagement. The interviewees included a variety of participants in positions within VA. We determined preliminary themes that we predict will assist uptake and adoption of the CHIEF system within VA. For example, we found that there is a culture of continual quality improvement with IT and healthcare system infrastructure providing a feedback loop to support the quality improvement efforts.

IMPACT:
The results of our research will fill a data and process gaps within the existing quality improvement and IT infrastructure. The use of the system and stakeholder engagement results will improve the efficiency of obtaining data about guideline-concordant care in of chronic heart failure both within and outside VA.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX000191-01
Link: https://reporter.nih.gov/project-details/7749813

---

PUBLICATIONS:

Journal Articles

1. Gobbel GT, Garvin J, Reeves R, Cronin RM, Heavirland J, Williams J, Weaver A, Jayaramaraja S, Giuse D, Speroff T, Brown SH, Xu H, Matheny ME. Assisted annotation of medical free text using RapTAT. Journal of the American Medical Informatics Association : JAMIA. 2014 Sep 1; 21(5):833-41. [view]
2. Kim Y, Garvin J, Goldstein MK, Meystre SM. Classification of Contextual Use of Left Ventricular Ejection Fraction Assessments. Studies in health technology and informatics. 2015 Jan 1; 216:599-603. [view]
3. Meystre SM, Kim Y, Gobbel GT, Matheny ME, Redd A, Bray BE, Garvin JH. Congestive heart failure information extraction framework for automated treatment performance measures assessment. Journal of the American Medical Informatics Association : JAMIA. 2017 Apr 1; 24(e1):e40-e46. [view]
4. Kim Y, Garvin J, Heavirland J, Meystre SM. Improving heart failure information extraction by domain adaptation. Studies in health technology and informatics. 2013 Jan 1; 192:185-9. [view]

VA Cyberseminars

1. Garvin JH, Heavirland J, Kelly N, Kim Y. Automating the Inpatient Chronic Heart Failure Quality Measures . [Cyberseminar]. 2013 Mar 18. [view]

Conference Presentations

1. Gobbel GT, Garvin JH, Reeves R, Cronin RM, Heavirland J, Williams J, Weaver AL, Jayaramaraja S, Giuse D, Speroff T, Brown SH, Xu H, Matheny ME. Assisted annotation of medical free text using RapTAT for Interactive Machine Learning. Paper presented at: American Medical Informatics Association Annual Symposium; 2013 Nov 19; Washington, DC. [view]
2. Garvin JH. Automated Heart Failure Quality Measurement. Poster session presented at: Heart Failure Society of America Annual Scientific Meeting; 2016 Sep 18; Orlando, FL. [view]
3. Garvin JH. Automated Measurement of Care Delivery to Heart Failure Patients within the Department of Veterans Affairs. Poster session presented at: American Medical Informatics Association Annual Symposium; 2015 Nov 14; San Francisco, CA. [view]
4. Garvin JH. Automating Heart failure Phenotyping with Natural Language Processing for Personalized medicine. Paper presented at: Korean Society for Bioinformatics and Systems Biology Annual Conference; 2016 Aug 19; Incheon, Korea. [view]
5. Meystre S, Kim Y, Garvin JH. Comparing Methods for Left Ventricular Ejection Fraction Clinical Information Extraction. Paper presented at: American Medical Informatics Association Translational Bioinformatics / Clinical Research Informatics Annual Joint Summits on Translational Science; 2012 Mar 19; San Francisco, CA. [view]
6. Meystre S, Kim Y, Redd A, Garvin JH. Congestive Heart Failure Information Extraction Framework (CHIEF) Evaluation. Presented at: American Medical Informatics Association Annual Symposium; 2014 Nov 17; Washington, DC. [view]
7. Kim Y, Garvin JH, Meystre S. Detecting Mentions and Values of Left Ventricular Ejection Fraction in Echocardiogram Reports. Poster session presented at: VA HSR&D / QUERI National Meeting; 2012 Jul 16; National Harbor, MD. [view]
8. Garvin JH, Heavirland J, Weaver AL, Hope CJ, Meystre S. Determining Section Types to Capture Key Clinical Data for Automation of Quality Measurement for Inpatients with Chronic Heart Failure. Poster session presented at: American Medical Informatics Association Annual Symposium; 2012 Nov 3; Chicago, IL. [view]
9. Garvin JH. Developing Natural Language Processing Systems for Healthcare. Presented at: American Medical Informatics Association Annual Symposium; 2015 Nov 17; Chicago, IL. [view]
10. Redd A, Kim Y, Meystre S, Heavriland J, Williams, J, Garvin JH, Weaver A. Effect of Pre-annotation on Annotation Time. Poster session presented at: American Medical Informatics Association Annual Symposium; 2014 Nov 18; Washington, DC. [view]
11. Meystre S, Kim Y, Williams J, Bray B, Garvin JH. Heart Failure Medication Detection and Prescription Status Classification in Clinical Narrative Documents. Paper presented at: American Medical Informatics Association Annual Symposium; 2015 Aug 21; San Paolo, Brazil. [view]
12. Kim Y, Garvin JH, Heavirland J, Meystre S. improving Detection of Reasons not to Take a Medication by Leveraging Medication Prescription Status. Poster session presented at: American Medical Informatics Association Annual Symposium; 2015 Nov 14; San Francisco, CA. [view]
13. Kim Y, Garvin J, Heavirland J, Meystre SM. Improving Heart Failure Information Extraction Domain Adaptation. Paper presented at: International Medical Informatics Association World Congress on Medical and Health Informatics; 2013 Aug 22; Copenhagen, Denmark. [view]
14. Kim Y, Garvin JH, Heavirland J, Weaver A, Meystre S. Medication Prescription Status Classification in Clinical Narrative Documents. Poster session presented at: American Medical Informatics Association Annual Symposium; 2014 Nov 18; Washington, DC. [view]
15. Kalsy M, Kelly N, Garvin JH, Goldstein M. Methods for an Early Stakeholder Engagement for Implementation of Health Information Technology. Poster session presented at: American Medical Informatics Association Annual Symposium; 2014 Nov 17; Washington, DC. [view]
16. Kalsy M, Kelly N, Garvin JH. Preliminary Themes Related to the Stakeholder Engagement for Automated Data Acquisition for Heart Failure. Poster session presented at: American Medical Informatics Association Annual Symposium; 2013 Nov 19; Washington, DC. [view]
17. Kim Y, Garvin JH, Heavirland J, Meystre S. Relatedness Analysis of LVEF Qualitative Assessments and Quantitative Values. Poster session presented at: American Medical Informatics Association Spring Congress; 2013 Mar 20; San Francisco, CA. [view]
18. Kalsy M, Kelly N, Goldstein M, Goldstein M, Garvin JH. Using a Sociotechnical Approach with Stakeholder for a Formative Evaluation of an NLP System. Poster session presented at: American Medical Informatics Association Annual Symposium; 2015 Mar 26; San Francisco, CA. [view]

DRA: Health Systems Science, Cardiovascular Disease
DRE: Diagnosis, Technology Development and Assessment, Research Infrastructure
Keywords: Cardiovasc’r disease, Quality assessment, Research method
MeSH Terms: none