BACKGROUND/RATIONALE:
The VA's medication alert system automatically warns prescribers about potential drug allergies, drug-drug interactions, etc during order entry. These alerts are particularly important given that an ever-increasing number of medications are available for treatment and preventable medication errors contribute to many adverse drug events (ADEs). However, VA data suggest that many alerts do not aid prescriber decision-making. In addition, several barriers identified during routine VA prescribing relate to the design of medication alerts. The feasibility of prototyping and evaluating alert redesigns has not been previously demonstrated.

OBJECTIVE(S):
The objective of this pilot proposal was to investigate whether interface redesigns can effectively address some common problems associated with medication alerts and evaluate the feasibility of these methods for a larger, more complex investigation of alert redesigns. The specific aims for this study were as follows: Aim 1: Design new medication alert prototypes and evaluate the usability of these redesigns. Aim 2: Assess whether alert redesigns reduce common barriers associated with current medication alerts.

METHODS:
For Aim 1, we combined literature findings with input from an expert panel to create new medication alert prototypes. We redesigned the presentation of allergy data; navigation to drug-drug interaction details; timing of drug-laboratory alerts; and mechanism to enter a reason for overriding alerts. In addition, we attempted to streamline the presentation of information on alerts and reduce textual information overload. We also developed three complex, clinically-relevant patient scenarios to evaluate the redesigns. Scenarios were crafted to prompt 5 drug-allergy alerts, 11 drug-drug interaction alerts, and 3 drug-laboratory alerts. Safety probes were included in the scenarios to test potential alert design weaknesses. For Aim 2, simulation experiments were conducted with 20 VA outpatient, primary care prescribers (physicians, nurse practitioners, and pharmacists) in a within-subject comparison of current alerts versus redesigned alerts. Prescribers completed two 30 minute sessions, one for each alert design; in the first session, half of the prescribers received the VA alert design in the first session and half the prescribers received the redesigned alerts. Prescribers were asked to 'think aloud' and were video recorded. After the scenarios, prescribers also completed a computer system usability questionnaire (CSUQ) (7-point Likert scale; 1 = strongly disagree and 7 = strongly agree) and NASA TLX workload assessment. Summed responses to questionnaire subscales were analyzed with paired t-tests. Individual items that were added to the questionnaire to assess specific design features were analyzed with the Wilcoxon signed-rank test. NASA TLX was evaluated with a two-way analysis of variance (ANOVA). For safety probes and alerts, we analyzed how the two alert designs affected decision-making error rates. These results were analyzed with chi-squared McNemar tests and the Wilcoxon signed-rank test. For all statistical analyses, a p-value < 0.05 was considered significant.

FINDINGS/RESULTS:
The alert design type did not have a significant main effect on the overall NASA TLX workload assessment. However, the redesigned alerts improved usability across several dimensions, based upon a qualitative usability analysis of the video recordings as well as quantitative ratings from the usability questionnaire. Qualitative findings indicated that the layout of information on the redesigned alerts enhanced usability. In addition, the following CSUQ scores were significantly greater for redesigned alerts compared to current alerts: 'overall satisfaction', (redesigned alerts: Mean = 5.2, SD = 1.4 vs. current alerts: Mean = 4.3, SD = 1.2, p = .033); 'information quality' (redesigned alerts: Mean = 5.2, SD = 1.4 vs. current alerts: Mean = 4.1, SD = 1.2; p = 0.013); and 'interface quality' (redesigned alerts: Mean = 5.2, SD = 1.5 vs. current alerts: Mean = 4.3, SD = 1.3; p = 0.005). In addition, for redesigned alerts, the efficiency of viewing allergy histories was rated significantly higher (redesigned alerts: Mean = 6.0, SD = 1.3 vs. current alerts: Mean = 3.6, SD = 1.9; p = 0.001), as was the efficiency of entering the reason for overriding alerts (redesigned alerts: Mean = 5.6, SD = 1.6 vs. current alerts: Mean = 4.4, SD = 1.7; p = 0.032). Ratings were not significantly different across the two designs for the efficiency of viewing laboratory results or drug interaction monographs. Prescribers rated information quality greater for redesigned alerts, although much of the clinical information was consistent across the two alert designs. Modifying alerts to reduce textual information overload may enhance prescribers' perceptions about the quality of information presented by alerts and improve workflow.

For safety probes and decision-making, appropriate decisions were significantly greater for alert redesigns compared to current alerts for all three alert types: allergy alerts (redesigned alerts: Median = 5.0, SE = 0.1 vs. current alerts: Median = 3.5, SE = 0.2, p = 0.001); drug-laboratory alerts for low creatinine clearance (redesigned alerts: Median = 2.0, SE = 0.2 vs. current alerts: Median = 0.5, SE = 0.2, p = 0.001); and drug-drug interaction alerts (redesigned alerts: Median = 4.5, SE = 0.2 vs. current alerts: Median = 3.0, SE = 0.3, p = 0.028). Finally, the percentage of override reasons that were appropriate were significantly greater for redesigned alerts that offer a drop-down list of options (Median = 85%, SE = 3.0) compared to current alerts that rely on free-text entry (Median = 50%, SE = 6.0, p = 0.001). Thus, several aspects of decision-making were improved with the redesigned alerts, indicating that these new designs may enhance workflow and patient safety.

IMPACT:
This study addressed common design problems that weaken the effectiveness of VA medication alerts. Results can help the VA understand how to design alerts to better support clinical workflow during medication order entry. Findings may be used to improve medication alert design for physicians, nurse practitioners, and pharmacists, and may ultimately be used to reduce ADEs and enhance medication safety for veterans. Study results may also inform alert designs in other institutions and computerized provider order entry (CPOE) systems.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX000343-01
Link: https://reporter.nih.gov/project-details/7873593

---

PUBLICATIONS:

Journal Articles

1. Russ AL, Chen S, Melton BL, Johnson EG, Spina JR, Weiner M, Zillich AJ. A Novel Design for Drug-Drug Interaction Alerts Improves Prescribing Efficiency. Joint Commission Journal on Quality and Patient Safety. 2015 Sep 1; 41(9):396-405. [view]
2. Russ AL, Zillich AJ, Melton BL, Russell SA, Chen S, Spina JR, Weiner M, Johnson EG, Daggy JK, McManus MS, Hawsey JM, Puleo AG, Doebbeling BN, Saleem JJ. Applying human factors principles to alert design increases efficiency and reduces prescribing errors in a scenario-based simulation. Journal of the American Medical Informatics Association : JAMIA. 2014 Oct 1; 21(e2):e287-96. [view]
3. Russ AL, Chen S, Melton BL, Saleem JJ, Weiner M, Spina JR, Daggy JK, Zillich AJ. Design and Evaluation of an Electronic Override Mechanism for Medication Alerts to Facilitate Communication Between Prescribers and Pharmacists. The Annals of pharmacotherapy. 2015 Jul 1; 49(7):761-9. [view]
4. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. Prescribers' interactions with medication alerts at the point of prescribing: A multi-method, in situ investigation of the human-computer interaction. International journal of medical informatics. 2012 Apr 1; 81(4):232-43. [view]
5. Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL, Chaffee BW, Hartman C, Tamis V, Galbreth B, Glassman PA, Phansalkar S, van der Sijs H, Gephart SM, Mann G, Strasberg HR, Grizzle AJ, Brown M, Kuperman GJ, Steiner C, Sullins A, Ryan H, Wittie MA, Malone DC. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. Journal of the American Medical Informatics Association : JAMIA. 2015 Nov 1; 22(6):1243-50. [view]
6. Melton BL, Zillich AJ, Russell SA, Weiner M, McManus MS, Spina JR, Russ AL. Reducing prescribing errors through creatinine clearance alert redesign. The American journal of medicine. 2015 Oct 1; 128(10):1117-25. [view]
7. Russ AL, Fairbanks RJ, Karsh BT, Militello LG, Saleem JJ, Wears RL. The science of human factors: separating fact from fiction. BMJ quality & safety. 2013 Oct 1; 22(10):802-8. [view]

Journal Other

1. Melton BL, Zillich AJ, Weiner M, McManus MS, Spina JR, Russ AL. Alerts for low creatinine clearance: design strategies to reduce prescribing errors. [Abstract]. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 2014 Nov 15; 1(1):85. [view]
2. Russ AL, Militello LG, Saleem JJ, Fairbanks RJ, Wears RL. Response to separating fact from opinion: a response to 'the science of human factors: separating fact from fiction'. [Letter to the Editor]. BMJ quality & safety. 2013 Nov 1; 22(11):964-6. [view]
3. Melton BL, Russ AL, Zillich AJ, McManus MS, Weiner M. The Reply. The American journal of medicine. 2016 Jul 1; 129(7):e97. [view]
4. Russ AL, Weiner M, Saleem JJ, Wears RL. When 'technically preventable' alerts occur, the design--not the prescriber--has failed. [Letter to the Editor]. Journal of the American Medical Informatics Association : JAMIA. 2012 Nov 1; 19(6):1119; author reply 1119-20. [view]

Newspaper Articles

1. Kobylinski AL, Zillich AJ, Russ AL. Redesigning Medication Alerts to Support Prescriber Decision-Making. Topics in Patient Safety (TIPS). 2012 Nov 1: 1: 2. [view]

VA Cyberseminars

1. Russ AL. 9 Factors that Influence Prescribers' Response to Alerts at the Point-of-Care: Implications for VA Order Check Design. [Cyberseminar]. 2013 Jan 15. [view]
2. Russ AL, Saleem JJ, Fahner J, Baker DA. A Rapid Usability Evaluation (RUE) Method Implemented for VA Health Information Technology. [Cyberseminar]. 2010 Sep 10. [view]
3. Russ AL. Clinical Reminder Order Checks: Design Tips for VA Order Checks”, VA-Informatics Development and Education Academy (VA-IDEA). Invited LiveMeeting presentation for VA clinical application coordinators (CACs). [Cyberseminar]. 2011 Jun 29. [view]
4. Russ AL. Clinical Reminder Order Checks: Design Tips from VA Order Checks” VA-Informatics Development and Education Academy (VA-IDEA). Invited LiveMeeting presentation for VA clinical application coordinators (CACs). [Cyberseminar]. 2011 Jul 6. [view]
5. Russ AL. Computerized Patient Record System (CPRS) Clinical Workgroup: Usability Testing. [Cyberseminar]. 2010 Aug 26. [view]
6. Russ AL. Human Factors, Usability, and Warning Design Literature: An Overview. [Cyberseminar]. 2013 Feb 25. [view]
7. Russ AL. Redesigning Order Checks to Improve Safety. [Cyberseminar]. 2014 Mar 12. [view]

Conference Presentations

1. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. 9 Factors that Influence Prescribers' Response to VA Medication Safety Checks at the Point-of-Care. Paper presented at: VA HSR&D / QUERI National Meeting; 2012 Jul 14; National Harbor, MD. [view]
2. Melton BL, Zillich AJ, Weiner M, McManus MS, Spina JR, Russ AL. Alerts for Low Creatinine Clearance: Design Strategies to Reduce Prescribing Errors. Paper presented at: American Medical Informatics Association Annual Symposium; 2014 Nov 15; Washington, DC. [view]
3. Russ AL. Alternative Designs for Medication Alerts that Improve Efficiency and Medication Safety. Paper presented at: VistA Community Meeting; 2016 Jan 13; Indianapolis, IN. [view]
4. Russ AL, Zillich AJ, Melton B, Spina J, Weiner M, Russell SA, McManus MS, Kobylinski A, Doebbeling BN, Hawsey J, Puleo A, Johnson E, Saleem JJ. Applying Human Factors Principles to Improve Medication Alerts. Paper presented at: American Medical Informatics Association Annual Symposium; 2012 Nov 3; Chicago, IL. [view]
5. Russ AL, Chen S, Melton BL, Saleem JJ, Weiner M, Spina J, Daggy J, Zillich AJ. Design and Evaluation of an Override Mechanism to Enhance Medication Safety for Veterans. Paper presented at: VA HSR&D / QUERI National Meeting; 2015 Jul 8; Philadelphia, PA. [view]
6. Russ AL. Insights from Human Factors Engineering & Warnings Research. Paper presented at: Drug-Drug Interaction Clinical Decision Support Conference Series; 2013 May 14; Rockville, MD. [view]
7. Probst CA, Russ AL, Carayon P, Wolf L, Parker SH, Xiao Y. Learning about Healthcare: Preparing Human Factors Professionals for a Career in Healthcare. Paper presented at: Human Factors and Ergonomics Society Annual Meeting; 2012 Oct 1; Boston, MA. [view]
8. Chen S, Zillich AJ, Melton BL, Saleem JJ, Johnson E, Weiner M, Russell SA, McManus MS, Doebbeling BN, Russ AL. Redesigned Drug-Drug Interaction Alerts Reduced Medication Errors and Improved Prescribing Efficiency. Paper presented at: Great Lakes Pharmacy Resident Annual Conference; 2013 Apr 24; West Lafayette, IN. [view]
9. Chen S, Zillich AJ, Melton BL, Saleem JJ, Johnson E, Weiner M, Russell SA, McManus MS, Doebbeling BN, Russ AL. The Effect of Redesigned Computerized Drug-Drug Interaction Alerts on Medication Errors and Prescribing Efficiency. Poster session presented at: International Society for Pharmacoeconomics and Outcomes Research Annual Meeting; 2013 May 18; New Orleans, LA. [view]

DRA: Health Systems Science
DRE: Treatment - Observational, Technology Development and Assessment, Research Infrastructure
Keywords: Decision support, Pharmaceuticals, Safety
MeSH Terms: none