Health Services Research & Development

Veterans Crisis Line Badge
Go to the ORD website
Go to the QUERI website

CRE 12-321 – HSR&D Study

New | Current | Completed | DRA | DRE | Portfolios/Projects | Centers | Career Development Projects

CRE 12-321
Cognitive Support Informatics for Nurse Medication Stewardship
Frank A Drews MS PhD
VA Salt Lake City Health Care System, Salt Lake City, UT
Salt Lake City, UT
Funding Period: November 2013 - April 2018

BACKGROUND/RATIONALE:
Acute change in mental status (AMSC) is a common clinical problem. Coordination between physicians and nurses are critical in prevention, monitoring, and patient safety. Quality of care requires effective coordination between clinical roles. Improving cognitive support for clinical team coordination is the goal of this work.

OBJECTIVE(S):
AIM 1: Characterize nurses' medication management information needs, documentation and communication processes associated with identifying and monitoring AMSC in inpatient settings.
AIM 2: Create a predictive model that integrates AMSC predictors and text descriptors with staffing data to support therapeutic decision-making, resource planning and collaboration
AIM 3: Design and test three cognitive support interventions: 1) Info-button decision-support provided through BCMA; 2) alerts imbedded in white board display; 3) display of documented mental status changes from the narrative electronic notes.
AIM 4: Implement and assess the impact of a comprehensive program for AMSC.

METHODS:
AIM 1:
Study 1: A national phone survey of 58 Nurse Managers of inpatient medical wards were randomly selected within all VA VISN (regional areas) to assess ward-level documentation practices and AMSC policies.
Study 2: Ethnographic observations were conducted in 3 inpatient floors (total=28) on the flow of mental status information, clinical decisions for patient care, and clinician's handoffs for older adults.
Study 3:Identification of mental models, information needs and decision tasks of 11 physicians regarding care of patients with AMSC using Cognitive Task Analyses and qualitative analysis.
Study 4: Randomly selected 30 patients > 65 years of age from 1 year of inpatient admissions to a SLC VA medical ward and evaluated documentation practices across nurses and physicians for AMSC.
Study 5: Three expert usability experts evaluated the VA eMAR/BCMA system using Nielson's heuristic taxonomy.

AIM 2
Study 1: Developed and tested a quantitative predictive model for daily risk of delirium from structured data predicting VINCI inpatient patients with orders for sitters and restraints.
Study 2: Validated the risk model in Study 1 using CAM (Confusion Assessment Method) assessed on 125 patients in a medical ward in SLC VA.
Study 3: Developed and tested an NLP model using topic-modeling to mental status terms associated with AMSC using progress notes in VINCI patient data.
Study 4: Validated NLP model using EPRP reviewers for 2017 charts assessing the VA delirium-based performance measure as gold standard.

AIM 3:
Study 1: Designed a CPRS clinical reminder that included physician assessment of risk at admission, delirium risk score, and an order set for increased nurse monitoring.
Study 2: Designed a dashboard for physicians that provides decision support in patients at risk for delirium and tested for usability, usefulness and impact on physician decision-making.
Study 3: An initial pilot survey of 17 physician's expectations and satisfaction regarding communication from nurses on mental status changes was conducted with SLC resident teams.
Study 4: Design decision support for nurses in CPRS to support nurse-physician communication that integrates physician notes, predictive risk scores, and nurses' assessments.

AIM 4:
Study 1: The full implementation of the program is ongoing and under the direction of the physician and nursing staff in SLC. They will direct the evaluation of the work.
Study 2: The implementation of an NLP automated program to support national EPRP performance assessment of delirium has been turned over to the directors of EPRP and the VA Office of Quality Assessment.

FINDINGS/RESULTS:
AIM 1
Study 1: Four themes from the interviews emerged: 1) "Fuzzy Concepts"- mental status terminology is often colloquial, vague, and imprecise; 2) "Grey Data" - data about mental status is often hidden or hard to find; 3) "Context is Critical"- mental status assessment and documentation depends on the patient situation; and 4) "Competing Goals"- nurses will document the mental status related to behavioral issues over other types of symptoms.
Structured data found: 1) Assessment for mental status limited to orientation (93%), and few identify risk (5%); 2) Preventative interventions were rarely used (10%). Sitters was the predominant intervention (88%); 3) formal stewardship, such as tracking ACMS, was rare (5%); and 4) formal CAM monitoring is rarely ordered (5%).

Study 2: Key deficits in communication were noted, specifically issues with communicating risk (80% of patients at risk were not identified nor was there increased monitoring) poor communication (delays in AMSC in 75% of patients with changes), no causal attribution was noted for 73% of patients with AMSC.

Study 3: 4 thematic areas emerged - all related to uncertainty: 1) Unavailable baseline information, 2) causal attribution is ambiguous; 3) information sources lack credibility, and 4) high perceived effort.

Study 4: 11/30 of patients had some documentation referring to AMSC. Physicians and nurses agreed on 10/11. ICD9 codes only identified 3 of the 30. Nurses focused largely on orientation while physicians had significantly more terms related to AMSC.

Study 5: 99 usability problems were identified with 15 rated as catastrophic. Situational awareness was affected at all levels for nurses caring for patients in the inpatient setting.

AIM 2-
Study 1: Accuracy (C statistic) maximized at 33 (C=73.1%). PPV was low (7.35%).

Study 2: C statistic for the ROC curve = 0.801.

Study 3: Three different topic-modeling methods (LDA and 2 ICD-based method) were tested. Keyword search method was highly specific but insufficiently sensitive (F-measure = 0.442). All 3-topic models had better recall but worse precision. LCD-2 had an F-measure of 0.677. 5/1000 topics found related to AMSC.

Study 4: a: NLP more accurate than EPRP reviewers with "presence of delirium" had an AUC=89.3% and "assessment of delirium" had an AUC=93%. Results b: An NLP pipeline ontology, called "POETenceph" was developed to rank clinical notes on the evidence for delirium using our realist ontology of encephalopathy, POETenceph correctly classified 65% of the documents.

AIM 3:
Study 1: The assessment was inserted in physician's H&P as a structured variable and is in use at present.

Study 2: A randomized trial compared regular CPRS access with dashboard display of delirium information using vignettes found that medical residents using the dashboard were significantly more likely to: 1) identify more risk factors (4.54 vs. 3.25, p = 0.123); 2) identify baseline mental status (93 % vs. 75% p=0.007); 3) have a more appropriate plan overall (p < 0.05) and express more confidence in their decision (p =. 0.04). Medication and baseline information was rated as the most useful.

Study 3: An initial pilot survey of 17 physician's expectations regarding communication from nurses on mental status identified five areas identified as most important: a) Lab values (glucose, BMP), b) Vital Signs, c) relevant medications, d) Onset/timing, brief situation and baseline, e) symptoms, and f) oxygen saturation. On ratings scales of 1 (low) to 7 (high), they reported they had to often request more information (M=4.8), rated the appropriateness overall as high (M=5.6), as was satisfaction (M=5.8).

Study 4: Design decision support for nurses in CPRS to support nurse-physician communication that integrates physician notes, predictive risk scores, and nurses' assessments. Results: Initial evaluation showed initial moderate ratings of satisfaction.

Aim 4
Study 1: The full implementation of the program is ongoing and under the direction of the physician and nursing staff in SLC. They will direct the evaluation of the work.

Study 2: The implementation of an NLP automated program to support national EPRP performance assessment of delirium has been turned over to the directors of EPRP and the VA Office of Quality Assessment.



IMPACT:
This study will significantly improve the care of patients with delirium by improving communication about risk and early identification of possible causes. We have developed an alert, documentation reminder dialogues, created a dashboard, and created an NLP module for EPRP performance measures.

PUBLICATIONS:

Journal Articles

  1. Cheng Y, Nickman NA, Jamjian C, Stevens V, Zhang Y, Sauer B, LaFleur J. Predicting poor adherence to antiretroviral therapy among treatment-naïve veterans infected with human immunodeficiency virus. Medicine. 2018 Jan 1; 97(2):e9495.
  2. Cheng Y, Sauer B, Zhang Y, Nickman NA, Jamjian C, Stevens V, LaFleur J. Adherence and virologic outcomes among treatment-naïve veteran patients with human immunodeficiency virus type 1 infection. Medicine. 2018 Jan 1; 97(2):e9430.
  3. LaFleur J, Bress AP, Rosenblatt L, Crook J, Sax PE, Myers J, Ritchings C. Cardiovascular outcomes among HIV-infected veterans receiving atazanavir. AIDS. 2017 Sep 24; 31(15):2095-2106.
  4. Williams ST, Lawrence PT, Miller KL, Crook JL, LaFleur J, Cannon GW, Nelson RE. A comparison of electronic and manual fracture risk assessment tools in screening elderly male US veterans at risk for osteoporosis. Osteoporosis international : a journal established as result of cooperation between the European Foundation for Osteoporosis and the National Osteoporosis Foundation of the USA. 2017 Nov 1; 28(11):3107-3111.
  5. Leecaster MK, Weir CR, Drews FA, Hellewell JL, Bolton D, Jones MM, Nebeker JR. Translation of Contextual Control Model to chronic disease management: A paradigm to guide design of cognitive support systems. Journal of Biomedical Informatics. 2017 Jul 1; 71S:S60-S67.
  6. Weir C, Brunker C, Butler J, Supiano MA. Making cognitive decision support work: Facilitating adoption, knowledge and behavior change through QI. Journal of Biomedical Informatics. 2017 Jul 1; 71S:S32-S38.
  7. Montgomery AE, Cusack M, Szymkowiak D, Fargo J, O'Toole T. Factors contributing to eviction from permanent supportive housing: Lessons from HUD-VASH. Evaluation and program planning. 2017 Apr 1; 61:55-63.
  8. Weir C, Gibson B, Taft T, Slager S, Lewis L, Staggers N. Mental Status Documentation: Information Quality and Data Processes. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 2017 Feb 10; 2016:1219-1228.
  9. Nelson SD, Poikonen J, Reese T, El Halta D, Weir C. The pharmacist and the EHR. Journal of the American Medical Informatics Association : JAMIA. 2017 Jan 1; 24(1):193-197.
  10. Brody AA, Gibson B, Tresner-Kirsch D, Kramer H, Thraen I, Coarr ME, Rupper R. High Prevalence of Medication Discrepancies Between Home Health Referrals and Centers for Medicare and Medicaid Services Home Health Certification and Plan of Care and Their Potential to Affect Safety of Vulnerable Elderly Adults. Journal of the American Geriatrics Society. 2016 Nov 1; 64(11):e166-e170.
  11. Kramer HS, Gibson B, Livnat Y, Thraen I, Brody AA, Rupper R. Evaluation of an Electronic Module for Reconciling Medications in Home Health Plans of Care. Applied clinical informatics. 2016 May 25; 7(2):412-24.
  12. Leecaster M, Toth DJ, Pettey WB, Rainey JJ, Gao H, Uzicanin A, Samore M. Estimates of Social Contact in a Middle School Based on Self-Report and Wireless Sensor Data. PLoS ONE. 2016 Apr 21; 11(4):e0153690.
  13. Nelson RE, Jones M, Leecaster M, Samore MH, Ray W, Huttner A, Huttner B, Khader K, Stevens VW, Gerding D, Schweizer ML, Rubin MA. An Economic Analysis of Strategies to Control Clostridium Difficile Transmission and Infection Using an Agent-Based Simulation Model. PLoS ONE. 2016 Mar 31; 11(3):e0152248.
  14. Del Fiol G, Mostafa J, Pu D, Medlin R, Slager S, Jonnalagadda SR, Weir CR. Formative evaluation of a patient-specific clinical knowledge summarization tool. International journal of medical informatics. 2016 Feb 1; 86:126-34.
  15. Stuart AR, Higgins TF, Hung M, Weir CR, Kubiak EN, Rothberg DL, Saltzman CL. Reliability in Measuring Preinjury Physical Function in Orthopaedic Trauma. Journal of Orthopaedic Trauma. 2015 Dec 1; 29(12):527-32.
  16. Graber CJ, Jones MM, Glassman PA, Weir C, Butler J, Nechodom K, Kay CL, Furman AE, Tran TT, Foltz C, Pollack LA, Samore MH, Goetz MB. Taking an Antibiotic Time-out: Utilization and Usability of a Self-Stewardship Time-out Program for Renewal of Vancomycin and Piperacillin-Tazobactam. Hospital pharmacy. 2015 Nov 24; 50(11):1011-24.
  17. Doing-Harris KM, Weir CR, Igo S, Shi J, Shao Y, Hurdle JF. POETenceph - Automatic identification of clinical notes indicating encephalopathy using a realist ontology. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 2015 Nov 5; 2015:512-21.
  18. Staggers N, Iribarren S, Guo JW, Weir C. Evaluation of a BCMA's Electronic Medication Administration Record. Western Journal of Nursing Research. 2015 Jul 1; 37(7):899-921.
  19. Weir CR, Staggers N, Gibson B, Doing-Harris K, Barrus R, Dunlea R. A qualitative evaluation of the crucial attributes of contextual information necessary in EHR design to support patient-centered medical home care. BMC medical informatics and decision making. 2015 Apr 16; 15(1):30.
  20. Mishra R, Bian J, Fiszman M, Weir CR, Jonnalagadda S, Mostafa J, Del Fiol G. Text summarization in the biomedical domain: a systematic review of recent research. Journal of Biomedical Informatics. 2014 Dec 1; 52:457-67.
  21. Spuhl J, Doing-Harris K, Nelson S, Estrada N, Del Fiol G, Weir C. Concordance of Electronic Health Record (EHR) Data Describing Delirium at a VA Hospital. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 2014 Nov 14; 2014:1066-71.
  22. Hope C, Estrada N, Weir C, Teng CC, Damal K, Sauer BC. Documentation of delirium in the VA electronic health record. BMC research notes. 2014 Apr 3; 7:208.
Journal Other

  1. Weir CR. Ensuring the Quality of Evidence: Using the Best Design to Answer Health IT Questions. [Book Review]. Studies in health technology and informatics. 2016 Jan 1; 222:90-101.
Center Products

  1. Weir CR. Delirium prevention and management. 2017 Sep 30.
  2. Samore MH, Rubin MA, Weir CR. Supplement on Theory and Innovation in Cognitive Support for Health Care Decision Making Supplement in the Journal of Biomedical Informatics. 2017 Jul 22.
  3. Del Fiol G. OpenInfoButton. 2017 Jul 1. Available from: http://www.openinfobutton.org.
  4. Gawron A, Gundlapalli AV, Gawron LM, Samore MH, Rubin MA, Jones MM, Weir CR. Natural Language Processing (NLP) Clinical Extraction Method. 2017 Jul 1.
  5. Weir CR, Samore MH, Chapman WW, Jones MM. Population health Dashboards. 2017 Jul 1.
VA Cyberseminars

  1. Nebeker JR, Del Fiol G, Weir CR. Integrating Pattern Matching and Active Thinking Support in Information Displays for Clinicians. JBI [Cyberseminar]. HSR&D. 2017 Aug 30.
Conference Presentations

  1. Weir CR. Adapting Nielsen's Design Heuristics to Dual Processing for Clinical Decision Support. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 21; Chicago, IL.
  2. Weir CR, Samore MH, Jones MM. Foraging for Information in the EHR: The Search for Adherence Related Information by Mental Health Clinicians. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 21; Chicago, IL.
  3. Weir CR. Why aren't they happy? An analysis of end-user satisfaction with Electronic health records. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 12; Chicago, IL.
  4. Weir CR, Gibson BS. The Information Quality of Mental Status Documentation and Data Processes. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 12; Chicago, IL.
  5. Weir CR, Gibson BS, Lewis LJ. Mental Status Documentation Information quality and Data Processes. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 12; Chicago, IL.
  6. Weir CR, Lewis LJ. Sense disambiguation through semantic interaction data. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 12; Chicago, IL.
  7. Weir CR, Lewis LJ. Development and Validation of an Electronic Health (EHR)-Based Risk Stratification Rule for Inpatient Delirium. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 12; Chicago, IL.
  8. Weir CR, Zeng Q. Identification and Use of Frailty Indicators from Text to Examine Associations with Clinical Outcomes Among Patients with Heart Failure. Distinguished Paper recipient. Paper presented at: American Medical Informatics Association Annual Symposium; 2016 Nov 12; Chicago, IL.


DRA: Aging, Older Veterans' Health and Care, Health Systems
DRE: Diagnosis, Treatment - Observational, Technology Development and Assessment
Keywords: Adverse Event Monitoring, Care Management Tools, Decision Support, Medication Management, Natural Language Processing, Personal Health Record
MeSH Terms: none