3138 — Exploring Nonresponse Bias for the Department of Veterans Affairs' Bereaved Family Survey
Ersek M, PROMISE Center, PVAMC; Smith D, PROMISE Center, PVAMC; Kuzla N, PROMISE Center, PVAMC; Thorpe J, PROMISE Center, PVAMC; Scott L, PROMISE Center, PVAMC;
Patients' and families' evaluations of healthcare are widely used as performance measures. Scores on these measures may be affected by nonresponse bias, resulting in inaccurate determinations of performance achievement. Our objective was to examine nonresponse bias for the mail version of the VA Bereaved Family Survey Performance Measure (BFS-PM) and evaluate the effect of nonresponse bias on facilities' BFS-PM scores and performance.
Our study was a retrospective analysis of a national sample of 20,540 Veterans who died in one of 146 VAMCs between October 2012 - September 2013. Data were collected as part of an ongoing quality improvement program. The BFS-PM is derived from a 19 item survey which covers key components of quality palliative care. The PM is based on one Likert-scale (Excellent to Poor) item and defined as the proportion of respondents (i.e., next of kin) who rated overall care for the deceased Veteran during the last month of life as "Excellent." After creating a model to predict the likelihood of response based on patient and clinical characteristics, we applied inverse probability weights to examine their effect on facilities' scores. We also evaluated facility performance before and after weighting for nonresponse vis-a-vis varying benchmarks.
We received 8,912 surveys (45% response rate). Using an18-variable model with the best C statistic (0.65) and lowest AIC (24154.07), we were able to calculate the inverse probability weight for nonresponse. Using these weights, we were able to achieve balance on key demographic and clinical variables between responders and nonresponders. The mean change in facility BFS-PM scores after weighting for nonresponse was -2%, (range: -10 to +11). The scores of 31% of facilities changed more than +/- 2%. The number of facilities meeting hypothetical benchmarks of 60, 70 and 80% also changed as a result of weighting for nonresponse.
Although nonresponse had only modest effects on the national BFS-PM mean score, its impact on individual facility-level scores varied greatly and affected facilities' ability to meet performance benchmarks.
Our results underscore the importance of appropriately addressing nonresponse in the use of quality of care metrics based on survey data.