Many missed and delayed cancer diagnoses result from breakdowns in communication and coordination of abnormal findings suspicious for cancer, which often first emerge in the primary care setting. Our previous work in the VA has shown that delays in the follow-up of abnormal test results persist despite reliable delivery of test results through the electronic health record. Methods to detect these delays and identify "high risk" patients are underdeveloped and need to be optimized for use within Patient Aligned Care Teams (PACTs). We conducted pilot work to determine whether the use of electronic queries, or "triggers," can proactively identify patients at risk of delayed cancer diagnosis. Triggers prompted review of selected medical records with evidence of possible care delays (e.g., a chart with no documented follow-up of an abnormal chest X-ray after 30 days). More than half the charts identified by the triggers were confirmed on chart review to have missed follow-up (positive predictive values [PPVs] >50%). However, the processes by which our team confirmed these delays and communicated them to providers were inefficient and resource intensive.
Building on our pilot work, we propose to develop and test an innovative automated surveillance intervention to improve timely diagnosis and follow-up of five common cancers in primary care practice (colorectal, bladder, lung, hepatocellular, and breast). Our methodology will use the VA Informatics and Computing Infrastructure (VINCI) to trigger medical records with evidence of potential delays in follow-up of abnormal test results. To guide our work, we will use Sittig and Singh's 8-Dimension, Socio-Technical Model built on principles from clinical informatics and human factors. Our specific aims are to: 1) Evaluate the accuracy of a VINCI-based "real-time" automated surveillance system to identify patients at risk of missed or delayed diagnosis of 5 common cancers. 2) Establish how to integrate "real-time" surveillance and communication of information about at-risk patients into the point of PACT care through adoption of informatics and human factors engineering principles. 3) Evaluate effects of the automated surveillance intervention on timeliness of the diagnostic process.
Study sites include 3 facilities in VISN 12 (Hines VA Hospital, Jesse Brown VA Medical Center, and Madison VA Hospital) and 1 facility in VISN 16 (Houston VA Medical Center). In Aim 1, we will use an iterative approach to develop and test algorithms to "trigger" records lacking documented follow-up action after pre-defined diagnostic clues for cancer. Data elements needed to operationalize our triggers already exist as part of the Corporate Data Warehouse. We will apply trigger algorithms to test cohorts, compare their output against manual chart reviews to confirm delays, and use these data to modify the algorithms to improve trigger PPVs. The finalized triggers will be applied to validation cohorts to determine the final PPVs through the same methods. In Aim 2 we will use interviews, task analysis, participatory design techniques, and usability testing to ensure that the automated intervention will fit within the workflow of real-world clinical practice. We will determine the technical requirements to transmit data to the PACTs and explore the best ways of communicating the information to the PACT team. In Aim 3 we will conduct a randomized controlled trial with PACT providers in VISN 12 and VISN 16 randomly assigned to intervention or usual care. Intervention development is ongoing, and will consist of: 1) weekly/biweekly identification of patients at risk of diagnostic delays and making this data accessible to a designated facility-level recipient; and 2) communication to PACT team providers in VISN 12 and 16 about their patients that are experiencing potential delays. Our outcomes are the median time in days from diagnostic clue to follow-up action (e.g., time to colonoscopy after a positive hemoccult test) and the proportion of patients receiving appropriate and timely follow-up care.
Aim 1 Findings:
We have completed the development and validation of all 5 trigger algorithms. Overall, the triggers were successful in identification of delays in diagnostic evaluation after an abnormal test results suggestive of cancer. Four publications detailing the processes for the 5 trigger algorithms and each trigger's specific findings are detailed later in the Impact section:
- We developed and validated a trigger for diagnostic evaluation delays in lung cancer diagnosis. This trigger found that among 1,847 results coded by radiologists as "suspicious for malignancy" between January to December 2012, 635 (35%), were flagged by the trigger to have a potential delay in subsequent follow-up action. Review of 400 of these records confirmed 242 patients experienced delays (PPV=60.5%). Thus, application of triggers to the VA's corporate data warehouse enabled efficient identification of patients experiencing delays in follow up of possible lung cancer.
-We developed and validated a trigger for diagnostic evaluation delays in bladder cancer diagnosis. Among 5,857 patients with high-grade hematuria suggestive of possible bladder cancer between January 2012 to December 2014, this trigger found 495 patients potentially experiencing a delay in follow-up evaluation. Review of 400 of these records confirmed 232 experienced delays (PPV=58.0%). The findings indicate that a hematuria-based trigger could aid in the reduction of delays in bladder cancer diagnosis.
-We developed and validated a trigger for diagnostic evaluation delays in gastrointestinal (colorectal and hepatocellular) cancer diagnosis. The trigger found 1,073 patients with suspected colorectal cancer seen between January to December 2013. Review of 400 of these records confirmed 224 experienced delays (PPV=56.0%). The trigger identified 130 patients with suspected hepatocellular cancer that were seen between January 2011 to December 2014. Review of all 130 records confirmed delays in diagnostic evaluation for 107 of these patients (PPV=82.3%). The application of these triggers provides a more efficient way to detect delays in GI cancer diagnosis.
-We developed and validated a trigger for diagnostic evaluation delays in breast cancer diagnosis. Among 2,129 patients seen between January 2010 to May of 2015 with abnormal mammograms, this trigger identified 552 potentially experiencing a follow-up. Review of 400 of these records confirmed 283 experienced delays (PPV=70.8%). The mammography-related trigger has the potential to aid in timely follow up to prevent delays in diagnosis.
Aim 2 Findings:
Our Team traveled to VISN 12 twice in 2014 and completed 37 in-person interviews with PACT members and 2 in-person interviews with facility safety personnel. Formal analysis of interview data (collected for the most part during the trips to VISN 12 in 2014) is now complete. Due to the turnover of one of the experts, there were some delays in analysis of the cognitive task interviews. Content analysis was performed by using a distributed cognition approach to identify patterns of information transmission across people and artifacts. The findings from this data shows several challenges to test-result management and identifies team-based strategies to overcome these challenges. The challenges include information overload, coordination across distributed care, and demands on prospective memory. In order to deal with information overload, intermediaries, such as nurses, can aid in information exchange by filtering information to prevent the primary care provider (PCP) from being overburdened by low priority or irrelevant messages. Distribution of healthcare teams requires the use of various communication methods such as asynchronous communication. Aim 2 data has shown that test result follow-up processes are quite fragmented and non-standardized at the PACT level, and we need much more leadership support and organization-level interventions to improve follow-up.
Thus, through this research we aim to improve the safety and quality of care that veterans receive. Our findings provide important information on the effectiveness and value of "trigger-based" interventions to identify and reduce cancer-related diagnostic delays.
The following excerpts are from the conclusions of the Aim 1-related papers:
Lung Cancer Trigger Paper (published in Chest): An algorithm designed to identify patients at risk for delays in follow-up of abnormal imaging from a large national dataset performed with reasonable accuracy for use in the clinical setting. Future research to develop and refine similar algorithms more widely can potentially reduce delays in diagnostic evaluation and improve quality and safety of patient care.
Gastrointestinal (GI) Cancer Trigger Paper [contains data from colorectal and hepatocellular cancer triggers] (published in Clinical Gastroenterology and Hepatology): We developed and tested an algorithm to identify patients at risk for delayed diagnostic evaluation for GI cancers. Prospective use of such triggers can improve care delivery related to diagnostic evaluation of cancer.
Bladder Cancer Trigger Paper (published in Applied Clinical Informatics): We developed and tested an algorithm to identify patients at risk for delayed diagnostic evaluation after lab findings of high-grade hematuria and found a performance level conducive for future application in clinical practice. Such triggers may serve as a resource for clinicians, informaticians and patient safety professionals to help reduce delays in cancer care.
Breast Cancer Trigger Paper (published in Journal of the American College of Radiology): In this study, we developed and tested an algorithm to detect delays in follow-up of externally-performed mammography. We discovered that despite legal requirements regarding reporting and current processes for communicating mammography results to patients, a small percentage of patients continue to lack timely follow-up. Thus, clinical application of such triggers could track and reduce missed opportunities to act early on abnormal mammogram results.
- Meyer AND, Singh H. The Path to Diagnostic Excellence Includes Feedback to Calibrate How Clinicians Think. JAMA. 2019 Feb 8.
- Bundy DG, Singh H, Stein RE, Brady TM, Lehmann CU, Heo M, O'Donnell HC, Rice-Conboy E, Rinke ML. The design and conduct of Project RedDE: A cluster-randomized trial to reduce diagnostic errors in pediatric primary care. Clinical trials (London, England). 2019 Apr 1; 16(2):154-164.
- Bates DW, Singh H. Priorities In Patient Safety: The Authors Reply. Health affairs (Project Hope). 2019 Feb 1; 38(2):330.
- Mendonca SC, Abel GA, Gildea C, McPhail S, Peake MD, Rubin G, Singh H, Hamilton W, Walter FM, Roland MO, Lyratzopoulos G. Associations between general practice characteristics with use of urgent referrals for suspected cancer and endoscopies: a cross-sectional ecological study. Family Practice. 2018 Dec 12.
- Sittig DF, Wright A, Coiera E, Magrabi F, Ratwani R, Bates DW, Singh H. Current challenges in health information technology-related patient safety. Health Informatics Journal. 2018 Dec 11; 1460458218814893.
- Medford-Davis LN, Singh H, Mahajan P. Diagnostic Decision-Making in the Emergency Department. Pediatric Clinics of North America. 2018 Dec 1; 65(6):1097-1105.
- Bates DW, Singh H. Two Decades Since To Err Is Human: An Assessment Of Progress And Emerging Priorities In Patient Safety. Health affairs (Project Hope). 2018 Nov 1; 37(11):1736-1743.
- Berenson R, Singh H. Payment Innovations To Improve Diagnostic Accuracy And Reduce Diagnostic Error. Health affairs (Project Hope). 2018 Nov 1; 37(11):1828-1835.
- Giardina TD, Haskell H, Menon S, Hallisy J, Southwick FS, Sarkar U, Royse KE, Singh H. Learning From Patients' Experiences Related To Diagnostic Errors Is Essential For Progress In Patient Safety. Health affairs (Project Hope). 2018 Nov 1; 37(11):1821-1827.
- Murphy DR, Meyer AN, Sittig DF, Meeks DW, Thomas EJ, Singh H. Application of electronic trigger tools to identify targets for improving diagnostic safety. BMJ quality & safety. 2019 Feb 1; 28(2):151-159.
- Bergl PA, Nanchal RS, Singh H. Diagnostic Error in the Critically III: Defining the Problem and Exploring Next Steps to Advance Intensive Care Unit Safety. Annals of the American Thoracic Society. 2018 Aug 1; 15(8):903-907.
- Meyer AND, Thompson PJ, Khanna A, Desai S, Mathews BK, Yousef E, Kusnoor AV, Singh H. Evaluating a mobile application for improving clinical laboratory test ordering and diagnosis. Journal of the American Medical Informatics Association : JAMIA. 2018 Jul 1; 25(7):841-847.
- Sittig DF, Salimi M, Aiyagari R, Banas C, Clay B, Gibson KA, Goel A, Hines R, Longhurst CA, Mishra V, Sirajuddin AM, Satterly T, Singh H. Adherence to recommended electronic health record safety practices across eight health care organizations. Journal of the American Medical Informatics Association : JAMIA. 2018 Jul 1; 25(7):913-918.
- Nystrom DT, Singh H, Baldwin J, Sittig DF, Giardina TD. Methods for Patient-Centered Interface Design of Test Result Display in Online Portals. EGEMS (Washington, DC). 2018 Jun 26; 6(1):15.
- Smith MW, Hughes AM, Brown C, Russo And E, Giardina TD, Mehta P, Singh H. Test results management and distributed cognition in electronic health record-enabled primary care. Health Informatics Journal. 2018 Jun 1; 1460458218779114.
- Lemoine N, Dajer A, Konwinski J, Cavanaugh D, Besthoff C, Singh H. Understanding diagnostic safety in emergency medicine: A case-by-case review of closed ED malpractice claims. Journal of healthcare risk management : the journal of the American Society for Healthcare Risk Management. 2018 Jul 1; 38(1):48-53.
- Bhise V, Rajan SS, Sittig DF, Vaghani V, Morgan RO, Khanna A, Singh H. Electronic health record reviews to measure diagnostic uncertainty in primary care. Journal of evaluation in clinical practice. 2018 Jun 1; 24(3):545-551.
- Giardina TD, Baldwin J, Nystrom DT, Sittig DF, Singh H. Patient perceptions of receiving test results via online portals: a mixed-methods study. Journal of the American Medical Informatics Association : JAMIA. 2018 Apr 1; 25(4):440-446.
- Kwan JL, Singh H. General Internists in Pursuit of Diagnostic Excellence in Primary Care: a #ProudtobeGIM Thread That Unites Us All. Journal of general internal medicine. 2018 Apr 1; 33(4):395-396.
- Shah T, Patel-Teague S, Kroupa L, Meyer AND, Singh H. Impact of a national QI programme on reducing electronic health record notifications to clinicians. BMJ quality & safety. 2019 Jan 1; 28(1):10-14.
- Whitehead NS, Williams L, Meleth S, Kennedy S, Epner P, Singh H, Wooldridge K, Dalal AK, Walz SE, Lorey T, Graber ML. Interventions to Improve Follow-Up of Laboratory Test Results Pending at Discharge: A Systematic Review. Journal of hospital medicine. 2018 Feb 28.
- Bhise V, Meyer AND, Menon S, Singhal G, Street RL, Giardina TD, Singh H. Patient perspectives on how physicians communicate diagnostic uncertainty: An experimental vignette study. International journal for quality in health care : journal of the International Society for Quality in Health Care. 2018 Feb 1; 30(1):2-8.
- Olson APJ, Graber ML, Singh H. Tracking Progress in Improving Diagnosis: A Framework for Defining Undesirable Diagnostic Events. Journal of general internal medicine. 2018 Jul 1; 33(7):1187-1191.
- Bhise V, Rajan SS, Sittig DF, Morgan RO, Chaudhary P, Singh H. Defining and Measuring Diagnostic Uncertainty in Medicine: A Systematic Review. Journal of general internal medicine. 2018 Jan 1; 33(1):103-115.
- Lyratzopoulos G, Mendonca SC, Gildea C, McPhail S, Peake MD, Rubin G, Singh H, Hamilton W, Walter FM, Roland M, Abel GA. Associations between diagnostic activity and measures of patient experience in primary care: a cross-sectional ecological study of English general practices. The British journal of general practice : the journal of the Royal College of General Practitioners. 2018 Jan 1; 68(666):e9-e17.
- Richards KA, Ruiz VL, Murphy DR, Downs TM, Abel EJ, Jarrard DF, Singh H. Diagnostic evaluation of patients presenting with hematuria: An electronic health record-based study. Urologic oncology. 2018 Mar 1; 36(3):88.e19-88.e25.
- Murphy DR, Meyer AND, Vaghani V, Russo E, Sittig DF, Wei L, Wu L, Singh H. Electronic Triggers to Identify Delays in Follow-Up of Mammography: Harnessing the Power of Big Data in Health Care. Journal of the American College of Radiology : JACR. 2018 Feb 1; 15(2):287-295.
- Rogith D, Iyengar MS, Singh H. Using Fault Trees to Advance Understanding of Diagnostic Errors. Joint Commission Journal on Quality and Patient Safety. 2017 Nov 1; 43(11):598-605.
- Bhise V, Sittig DF, Vaghani V, Wei L, Baldwin J, Singh H. An electronic trigger based on care escalation to identify preventable adverse events in hospitalised patients. BMJ quality & safety. 2018 Mar 1; 27(3):241-246.
- Sittig DF, Singh H. Toward More Proactive Approaches to Safety in the Electronic Health Record Era. Joint Commission Journal on Quality and Patient Safety. 2017 Oct 1; 43(10):540-547.
- Murphy DR, Meyer AND, Vaghani V, Russo E, Sittig DF, Wei L, Wu L, Singh H. Development and Validation of Trigger Algorithms to Identify Delays in Diagnostic Evaluation of Gastroenterological Cancer. Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association. 2018 Jan 1; 16(1):90-98.
- Bhise V, Meyer AND, Singh H, Wei L, Russo E, Al-Mutairi A, Murphy DR. Errors in Diagnosis of Spinal Epidural Abscesses in the Era of Electronic Health Records. The American journal of medicine. 2017 Aug 1; 130(8):975-981.
- Sittig DF, Belmont E, Singh H. Improving the safety of health information technology requires shared responsibility: It is time we all step up. Healthcare (Amsterdam, Netherlands). 2018 Mar 1; 6(1):7-12.
- Gregory ME, Russo E, Singh H. Electronic Health Record Alert-Related Workload as a Predictor of Burnout in Primary Care Providers. Applied clinical informatics. 2017 Jul 5; 8(3):686-697.
- Kwan JL, Singh H. Assigning Responsibility To Close the Loop on Radiology Test Results. Diagnosis. 2017 Jun 16; doi.org/10.1515/dx-2017-0019.
- Murphy DR, Meyer AN, Vaghani V, Russo E, Sittig DF, Richards KA, Wei L, Wu L, Singh H. Application of Electronic Algorithms to Improve Diagnostic Evaluation for Bladder Cancer. Applied clinical informatics. 2017 Mar 22; 8(1):279-290.
- Davalos MC, Samuels K, Meyer AN, Thammasitboon S, Sur M, Roy K, Al-Mutairi A, Singh H. Finding Diagnostic Errors in Children Admitted to the PICU. Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. 2017 Mar 1; 18(3):265-271.
- Menon S, Singh H, Giardina TD, Rayburn WL, Davis BP, Russo EM, Sittig DF. Safety huddles to proactively identify and address electronic health record safety. Journal of the American Medical Informatics Association : JAMIA. 2017 Mar 1; 24(2):261-267.
- Pfoh ER, Engineer L, Singh H, Hall LL, Fried ED, Berger Z, Wu AW. Informing the Design of a New Pragmatic Registry to Stimulate Near Miss Reporting in Ambulatory Care. Journal of Patient Safety. 2017 Feb 28.
- Meyer AND, Murphy DR, Al-Mutairi A, Sittig DF, Wei L, Russo E, Singh H. Electronic Detection of Delayed Test Result Follow-Up in Patients with Hypothyroidism. Journal of general internal medicine. 2017 Jul 1; 32(7):753-759.
- Sittig DF, Wright A, Ash J, Singh H. New Unintended Adverse Consequences of Electronic Health Records. Yearbook of medical informatics. 2016 Nov 10;(1):7-12.
- Singh H, Graber ML, Hofer TP. Measures to Improve Diagnostic Safety in Clinical Practice. Journal of Patient Safety. 2016 Oct 20.
- Singh H, Zwaan L. Annals for Hospitalists Inpatient Notes - Reducing Diagnostic Error-A New Horizon of Opportunities for Hospital Medicine. Annals of internal medicine. 2016 Oct 18; 165(8):HO2-HO4.
- Singh H. Improving Diagnostic Safety in Primary Care by Unlocking Digital Data. Joint Commission Journal on Quality and Patient Safety. 2017 Jan 1; 43(1):29-31.
- Baldwin JL, Singh H, Sittig DF, Giardina TD. Patient portals and health apps: Pitfalls, promises, and what one might learn from the other. Healthcare (Amsterdam, Netherlands). 2017 Sep 1; 5(3):81-85.
- Singh H, Schiff GD, Graber ML, Onakpoya I, Thompson MJ. The global burden of diagnostic errors in primary care. BMJ quality & safety. 2017 Jun 1; 26(6):484-494.
- Giardina TD, Sarkar U, Gourley G, Modi V, Meyer AN, Singh H. Online public reactions to frequency of diagnostic errors in US outpatient care. Diagnosis (Berlin, Germany). 2016 Mar 1; 3(1):17-22.
- Fisk RJ, Kumar D, Murphy DR, Arya M. Complementing EMR-based Interventions to Improve Hepatitis C Screening. [Letter to the Editor]. Journal of translational internal medicine. 2018 Dec 1; 6(4):198-199.
- Murphy DR, Meyer AN, Russo E, Sittig DF, Wei L, Singh H. The Burden of Inbox Notifications in Commercial Electronic Health Records. JAMA internal medicine. 2016 Apr 1; 176(4):559-60.
- Singh H. Closing the Loop. A Guide to Safer Ambulatory Referrals in the EHR Era. Cambridge, Massachusetts: Institute for Healthcare Improvement / National Patient Safety Foundation; 2017 Nov 28. 42 p.
HSR&D or QUERI Articles
- Meyer A, Singh H. Using Electronic Health Records to Improve Diagnoses. FORUM : Translating Research into Quality Healthcare for Veterans. 2019 Jan 2; 2018(Winter): 4.
- Singh H. Expert Commentaries: Measurement of Diagnostic Errors Is a Key First Step to Their Reduction. [Newsletter] AHRQ National Quality Measures Clearinghouse (NQMC); 2015 Nov 29. Available from: http://www.qualitymeasures.ahrq.gov/expert/expert-commentary.aspx?id=49659.
- Singh H. Building Reliable Systems to Reduce Delays in Diagnosis: Training Curriculum and Seminars. Institute for Healthcare Improvement; 2015 Oct 13. Available from: http://www.ihi.org/education/WebTraining/Webinars/Web_Action/2016JanuaryBuildingReliableSystems/Pages/default.aspx.
- Baldwin JL, Bhise V, Sadana A, Singh H. Public Attitudes and Opinions on Medical Errors in India. Poster session presented at: Diagnostic Error in Medicine Annual International Conference; 2016 Nov 7; Hollywood, CA.
- Bhise V, Meyer A, Menon S, Singhal G, Street R, Giardina T, Singh H. Patient Perspectives on Communication of Diagnostic Uncertainty. Paper presented at: Diagnostic Error in Medicine Annual International Conference; 2016 Nov 7; Hollywood, CA.
- Giardina T, Menon S, Haskell H, Hallisy J, Southwick F, Sarkar U, Singh H. An Exploration of Patient and Family Self-Reports of Diagnostic Errors. Paper presented at: Diagnostic Error in Medicine Annual International Conference; 2016 Nov 7; Hollywood, CA.
- Meyer A, Thompson PJ, Desai S, Taylor JR, Wilson KK, Zhang L, Redmond N, Singh H. Methods for evaluating mobile applications for test ordering and diagnosis. Poster session presented at: Diagnostic Error in Medicine Annual International Conference; 2016 Nov 7; Hollywood, CA.
- Nystrom DT, Meyer A, Paull DE, Singh H. Virtual Patient Simulation: A Method to Study Diagnostic Process as an Emergent Aspect of Information Sampling Behavior. Presented at: Human Factors and Ergonomics Society Annual International Meeting; 2016 Sep 20; Washington, DC.
- Giardina T, Baldwin J, Sittig DF, Singh H. Patient Perceptions of Test Result Notification Via the Patient Portal. Poster session presented at: Public Health Informatics Conference; 2016 Aug 22; Atlanta, GA.
- Singh H. The 2016 NQF Report: Measurement is the First Step in Improving Health IT Safety. Paper presented at: National Institute of Standards and Technology: The Role of Standards in Managing and Mitigating Health IT Patient Safety Risks Workshop; 2016 Aug 7; Gaithersburg, MD.
- Classen D, Singh H. The Role of Standards in Managing & Mitigating Health IT Patient Safety Risks. Paper presented at: Agency for Healthcare Research and Quality National Institute of Standards and Technology Annual Conference; 2016 Jul 25; Gaithersburg, MD.
- Singh H. Improving Diagnostic Safety: The Next Grand Challenge & Opportunity for Informatics. Paper presented at: Health Informatics Society of Australia Annual Conference; 2016 Jul 25; Melbourne, Australia.
- Singh H. An Emerging Frontier for Health Services Research and Quality Improvement - Panelist for Diagnostic Safety. Presented at: AcademyHealth Annual Research Meeting; 2016 Jun 27; Boston, MA.
- Singh H. Correct and Timely Clinical Diagnosis: An Emerging National Patient Safety Priority – Keynote Speaker. Paper presented at: VA Madison VA Hospital Research Day Annual Assembly; 2015 May 22; Madison, WI.
- Singh H. Building Health Services Research and Research Careers – Keynote Speaker. Paper presented at: VA Madison VA Hospital Research Day Annual Assembly; 2015 May 22; Madison, WI.
Diagnosis, Treatment - Comparative Effectiveness
Cancer, Care Management Tools, Healthcare Algorithms, Knowledge Integration, Surveillance