3187 — Development and Validation of an Automated Functional Assessment Algorithm

Giang P, Bay Pines VA Healthcare System; Williams AE, Bay Pines VA Healthcare System; Argyros L, Bay Pines VA Healthcare System; Higginbottom J, Bay Pines VA Healthcare System; Alemi F, Washington, DC VAMC; Levy C, Denver/Seattle COIN;

Objectives:
Functional status (FS) is a key indicator of overall health and quality of life, particularly for patients in geriatric and long-term care settings. FS data are routinely documented in the Minimum Data Sets (MDS) for patients in VA Community Living Centers (CLCs); however, these data are not available for patients residing in alternative, long-term care settings including VA Medical Foster Homes (MFHs). An alternative is to derive FS scores from notes in electronic health records (EHRs). In this manner, FS scores can be calculated without requiring additional data collection efforts. This study developed and validated an automated text algorithm (ATA) that computes FS scores from clinical notes.

Methods:
Data were retrieved from the Department of Veterans Affairs (VA) Corporate Data Warehouse (CDW) and uploaded to the VA Informatics and Computing Infrastructure (VINCI). Clinical notes were obtained for all VA MFH patients and MDS data for VA CLC patients from 2009 to 2012. A sample of 165 MFH charts was randomly selected from the data pool to develop, test, and refine the ATA. Subsequently, the ATA was validated on MDS data from the CLC charts. Algorithm accuracy and concurrent validity were evaluated using Receiver Operating Characteristic Curves (ROC) and standard linear regression.

Results:
The ATA took on average 2.5 minutes to score patient charts. Charts contained an average of 1042 notes. For the MFH corpus, manually abstracted FS scores regressed on the ATA scores produced a high, adjusted R-squared (0.789, p < 0.001). The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) was (0.9643). Validation using CLC MDS data regressed on ATA scores produced an Adjusted R-Squared = 0.538 (p < 0.001). The AUC for these data was 0.877.

Implications:
The ATA applied to textual data efficiently generated a FS assessment with a good to high level of accuracy. There was some decline in performance when the algorithm was tested on the CLC MDS data.

Impacts:
ATA performance numbers indicate that the ATA may fill an important administrative and quality assurance information gap for current and emerging models of non-institutional long-term care delivery.