BACKGROUND/RATIONALE:
Decision-making is an essential cognitive activity of clinicians who use electronic records. Decisions about diagnosis and treatment can be described from both logical and "naturalistic" perspectives. Naturalistic decision-making is based on pattern recognition and "situation awareness", which suggests that high-quality, information-rich clinical notes may facilitate clinicians "getting the picture" of a patient in a decision making task. Previous work conducted by our group established that a statistical property of text, computed from the frequencies of words in documents vs their frequency in the entire document collection correlated positively with practitioner estimates of document quality. This statistic, known as tf*idf (term frequency X inverse document frequency), is used by bibliographic systems and search engines to retrieve results that are relevant to a query. When computed for all terms in a document, the summed tf*idf measures a document's "informativeness". Electronic charts contain a large volume of narrative text, and pose a cognitive challenge of information overload. The research question asked whether clinical assessments could be enhanced by selecting the most "informative" documents in a patient record to display compared to displaying all documents. A data review task using de-identified patient records was administered under a workplace-approximating time constraint.

OBJECTIVE(S):
The project had 3 specific aims:

1. Implement a document display enhancement suggested in prior analyses of interviews and experiments with VA CPRS users.
2. Measure the performance impact of the enhancement on a decision task requiring document review, using a simulated CPRS testing platform.
3. In a proof of concept exercise, demonstrate the feasibility of linking a simulated CPRS evaluation platform to data in a VA research data warehouse.

The enhancement chosen was to use summed tf.idf to determine which documents to display in a simulated chart. It was hypothesized that a view showing the 30 most-informative clinical notes from electronic records containing about 100 such notes would improve physicians', nurses' and administrators' ability to process records and correctly respond to questions about cases compared to a standard view of all notes. Record review time was limited to five minutes per case.

METHODS:
Aim 1: Summed tf.idf was computed for all 3.7 million documents in a 2003-2006 VA Puget Sound collection. From these, notes of six cases concurrently diagnosed with PTSD, Coronary Artery Disease, Diabetes and Hypertension and possessing about 100 notes were selected, de-identified and loaded into a simulated CPRS presentation. Neahmatullah's de-identification method, adapted for VA use, was applied.

Aim 2: The experiment presented notes of six cases formatted as they appear in a VA electronic record. In a block design each subject alternately viewed cases in standard form and enhanced form. Across subjects, the case presentation sequence was rotated to balance viewing of each form of each case. In the standard view, subjects saw charts containing all patient notes and in the enhanced view, subjects saw only the 30 "most informative" notes. Subjects were allowed 5 minutes to review the notes and answer questions before proceeding to the next case. Subjects were permitted to terminate the exercise at any point.

The test task was to answer 10 questions about the patient's clinical status. The questions were based on practice guidelines applicable to four diseases frequently found in veterans: PTSD, Diabetes, Hypertension and Coronary Artery Disease. "Correct" answers to questions as they applied to each case were judged by the PI based on review of the full set of notes.

It was hypothesized that subjects viewing the shortened record sets would answer questions more quickly, and that more responses would be correct. This was assessed by recording the total number of responses and the number of correct responses for each case.

Research subjects were recruited by e-mail from lists of primary practitioners (MD, ARNP, PA), nurses and administrative staff at VA Puget Sound. Clicking a link in the invitation e-mail opened the web-based test. 20 subjects were recruited: 13 physicians, 2 nurses and 5 administrators.

Aim 3: Methods developed at Puget Sound for creating de-identified patient notes and a simulated version of the CPRS were implemented in the VINCI research data warehouse.


FINDINGS/RESULTS:
Aim 1: A web-based simulated CPRS system displaying documents and managing recruitment and data collection was successfully deployed. Nearly 600 documents were anonymized with 99.9% accuracy, resulting in a 74-fold efficiency increase compared to manual de-identification.

Aim 2: Results showed that overall, the number of questions answered, and the number of correct answers was smaller in the enhanced View compared to the standard View. This was true for nine questions addressing historical facts (Questions 2, 4, 5, and 8-13) and four questions addressing facts dependent on most recent data (Questions 1, 3, 6, and 7).

Cases View Re- F* Correct F*
Role Viewed sponses Re-
sponses
---------------------------------------------------------------
All 44 Std 10.0 .87 5.8 .90
(N=20) 44 Enh 9.3 5.1

Practitioner 29 Std 10.9 NS 6.6 NS
(N=13) 29 Enh 9.9 5.9

Nurse 4 Std 9.5 NS 5.0 NS
(N=4) 4 Enh 10.0 4.5

Admini-
strative 11 Std 8.0 NS 3.9 NS
(N=5) 11 Enh 7.2 3.2

(*One way ANOVA, testing for main effect of View. None of the
comparisons were statistically significant.)

Nurses and Administrators made fewer correct responses in the enhanced Form compared to physicians. The final result was restricted by a low subject response rate which also adversely affected the balance between cell sizes shown below. Despite the small sample, the results showed no significant effect from the enhancement. The hypothesis that filtering by informativeness would improve performance was not supported.

#Adminstrations Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
Standard 4 3 12 9 11 5
Enhanced 10 11 4 5 4 10

Aim 3: It was demonstrated that clinical notes data outputs comparable to those produced for the experiment could be rendered from data in the VINCI data warehouse.

IMPACT:
Within the constraints of a small sample size the results indicated that filtering by a broad tf*idf informativeness measure did not enhance clinicians' ability to correctly answer questions about the current status of patients with common chronic conditions when strict time limits were imposed. Eliminating 2/3 of less-informative documents did not significantly degrade subject performance. The tf*idf statistic can be adjusted to select documents relevant to specific user questions. The results suggest that a topic-focused approach for selecting documents could prove more useful.

Compared to a 2009 study where the e-mail response rate was 44%, this study experienced a response rate of 12%. VA staff seem less likely to respond to e-mail than they were three years ago. Future studies of this sort might better be conducted on a larger scale or performed in a laboratory setting. The most positive impact of the study was demonstration of an efficient system for de-identifying VA records which will facilitate future realistic studies of electronic medical records usage.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX000295-01
Link: https://reporter.nih.gov/project-details/7869482

---

PUBLICATIONS:

None at this time.

DRA: Health Systems
DRE: Diagnosis, Treatment - Observational, Technology Development and Assessment
Keywords: none
MeSH Terms: none