Study Examines Relationship between Diagnostic Accuracy and Confidence Using Case Study Vignettes

BACKGROUND:
Diagnostic errors can lead to patient harm, but the factors contributing to diagnostic errors have not been explored as thoroughly as other patient safety concerns. While physician overconfidence has been identified as a potential contributor to diagnostic error, little is known about the relationship between physicians' diagnostic accuracy and their confidence (defined as diagnostic calibration) or how that relationship might change with the evolution of diagnosis, such as through patient history, examination, and testing. Even less is known about how confidence and diagnostic accuracy relate to resource use (e.g., specialized tests, referrals, second opinions). This case vignette-based study evaluated diagnostic accuracy, confidence, and diagnostic calibration during the evolution of the diagnostic process. It also evaluated how resource use might be related to those factors. Investigators recruited 118 general internists from an online physician community (www.QuantiaMD.com) and asked them to diagnose four previously validated case vignettes of variable difficulty (two "easier"; two "more difficult"). Cases were presented in a web-based format. Investigators assessed physicians' diagnostic accuracy, confidence in diagnostic accuracy, as well as diagnostic calibration and whether they would use any additional resources to diagnose the case. Diagnostic accuracy was scored by comparing participating physicians' responses to the true diagnoses provided by case study authors.

FINDINGS:

• Physicians correctly diagnosed 55% of the easier and 6% of the more difficult cases.
• Physicians' confidence averaged 7 out of 10 (10 being highest confidence) for easier cases and 6 for more difficult cases. The small decrease in confidence between easier and more difficult cases was unlikely to be practically or clinically meaningful in the face of a much larger drop in accuracy.
• Physicians' level of overconfidence increased with difficult cases (diagnostic calibration was worse).
• Although confidence increased as cases evolved through the diagnostic process and more data became available, diagnostic accuracy did not.
• Fewer diagnostic tests were requested when confidence was high, even when accuracy was low.

LIMITATIONS:

• Findings from an artifical diagnostic environment may not translate to real-world settings.
• Some physicians may have been unwilling to disclose diagnostic uncertainty.

IMPLICATIONS:

• If confirmed in real clinical settings, findings provide evidence needed to develop interventions to improve physicians' diagnostic calibration and resource use, and related diagnostic outcomes.

AUTHOR/FUNDING INFORMATION:
Drs. Meyer, Payne, Meeks, and Singh are part of HSR&D's Houston Center for Quality of Care and Utilization Studies.

Meyer AND, Payne VL, Meeks DW, Rao R, and Singh H. Physicians’ Diagnostic Accuracy, Confidence, and Resource Requests: A Vignette Study. JAMA Internal Medicine August 26, 2013;e-pub ahead of print.