Real-time analysis of Delphi panel data to facilitate iterative rating within meetings.

Weinstein JB, Hogan CK, Whittington T, Cano J, Seelye SM, Sussman JB, Bernstein SJ, Prescott HC. Real-time analysis of Delphi panel data to facilitate iterative rating within meetings. BMC research notes. 2025 Jul 31; 18(1):337, DOI: 10.1186/s13104-025-07409-7.

Related HSR&D Project(s)

• IIR 20-313 – Optimizing Veteran Recovery from Sepsis (OVeR-Sepsis)

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

If you have VA-Intranet access, click here for more information vaww.hsrd.research.va.gov/dimensions/

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.    Search Dimensions for VA for this citation
* Don't have VA-internal network access or a VA email address? Try searching the free-to-the-public version of Dimensions

Search for Abstract from PubMed

---

Abstract:

OBJECTIVE: The Modified Delphi panel process involves iterative ratings, often with lengthy lags between rounds to allow time for data analysis. We sought to develop, implement, and evaluate an approach for real-time data analysis and presentation to complete multiple rounds of ratings in a single panel meeting. RESULTS: We developed statistical code in Python to analyze Delphi panel ratings completed in Slido using a 9-point Likert scale. The code generates histograms of panel ratings. The median score is color-coded pink when formal disagreement criteria are met (regardless of median score), and when there is no disagreement, red for median scores of 1-3, yellow for 4-6, and green for 7-9. We implemented this approach for a Delphi panel on sepsis recovery that rated three sets of care practices. Round 1 rating, panel discussion, and round 2 rating all occurred in one day for each set of practices. Round 1 ratings were presented during discussion. This approach avoided "homework" outside meetings. All panelists either strongly agreed (57%) or agreed (43%) it was easy to understand the presentation of round 1 ratings. Most panelists either strongly agreed (50%) or agreed (21%) that they liked doing both rounds of rating during the meetings.