2012 HSR&D/QUERI National Conference Abstract
3092 — Adjustment of DOCS Neurobehavioral Measures for Rater Severity and Leniency
Pape TL, Hines VA, CMC3; Mallinson T, University of Southern California; Guernon A, Marianjoy Rehabilitation Hospital;
Clinician’s routinely determine impairment levels and monitor recovery by observing and rating a patient’s behavioral performance. This approach to measurement assumes that all raters apply a common understanding of a behavioral rating scale. To test this assumption, we examined the impact of rater severity/leniency on measures of neurobehavioral functioning derived from the Disorders of Consciousness Scale (DOCS).
This prospective cohort study involved 57 rehabilitation clinicians from seven sites who administered the DOCS to 174 patients who remained unconscious after severe traumatic brain injury. Clinicians administered DOCS tests together, but scored elicited responses independently. DOCS Sensory items (n = 20) were examined using the Winsteps and Many-Faceted Rasch measurement models. Analyses were conducted to examine specifically for the effects of rater severity and leniency on patient measures of neurobehavioral functioning.
Raters were largely occupational, speech and physical therapists (88%). Person reliability is strong (.93) and all sensory items and rating scale steps fit the measurement models (Infit statistics: 0.8 – 1.14). Three raters misfit (>1.2) the model and were removed from analyses. There were no interactions between raters and items or time. Winsteps and Facets analyses indicate that seven raters were unacceptably severe, where patients scored by these raters appear to have less functioning than they actually do. Four raters were unacceptably lenient and these patients’ neurobehavioral abilities were over-estimated. Comparing DOCS patient measures with Winsteps (unadjusted for rater severity/leniency) and Facets (adjusted) indicates that 42% of patient measures were over-or under-estimated because of rater severity or leniency.
The DOCS is a reliable test, and the yielded measures of patient neurobehavioral ability do not show bias across time, patients or items. Evidence indicates, however, that DOCS measures should be adjusted to eliminate the influence of rater severity/leniency.
The field of rehabilitation seldom considers the impact of rater severity and leniency on patient measures. These findings suggest a need to consider that some clinical raters are more severe or lenient in how they assign scores, and that measurement of patient abilities and treatment effectiveness can be under-or over-estimated if rater severity/leniency are not accounted for.