BACKGROUND/RATIONALE:
Missing data affects nearly all studies. In observational studies, when data are not missing completely at random, one must rely on computationally expensive, post-experimental balancing procedures to address it. Examples include matching and propensity score adjustment. If the missing mechanism directly relates to the study's outcomes of interest, then the computations needed to address missing data become substantially more complex, relying on modeling procedures of uncertain validity. All currently available methods to address missing data must be applied after data collection ends, when the underlying bias can no longer be rectified. Furthermore, these methods do not always yield the correct answer and sometimes introduce their own bias into the results. Non-response bias is a common cause of non-random missing data in survey research.

OBJECTIVE(S):
Our primary objective was to test 3 interventions that, if successful, would reduce non-response bias at the design stage of a mailed survey.

In anticipation of a future cohort study, our secondary objective was to identify the best combination of the 3 interventions to minimize non-response bias in Veterans from Operations Enduring Freedom, Iraqi Freedom, or New Dawn (OEF/OIF/OND) who were applying for posttraumatic stress disorder (PTSD) disability benefits.

METHODS:
The study was a randomized, 3X2X2 factorial trial that assessed the impact of two types of cover letter manipulations and two honoraria on non-response bias to a mailed survey. The survey asked about PTSD and depression symptoms, functioning, background history, and trauma exposures. Participants were 480 men and 480 women who were randomly selected from a sampling frame of 14,630 male and 2,945 female OEF/OIF/OND Veterans, respectively, who had pending PTSD disability claims. Forty Veterans of each gender were assigned to each one of the 12 possible intervention arms using simple randomization without replacement.

The first cover letter manipulation, factor 1, varied what Veterans were told about the survey's content -namely, that the survey asked about "combat," about "unwanted sexual attention while in the military," or about "military and life experiences that could affect well-being." The second cover letter manipulation, factor 2, varied what Veterans were told about how their name was selected for the study: from a Department of Veterans Affairs "list of Veterans who served during OEF/OIF/OND" or from a Department of Veterans Affairs "list of Veterans who filed a disability claim." Factor 3 varied the honorarium Veterans received after completing the survey: $20 or $40. Veterans were told which honorarium they would receive in the cover letter.

Analysis was intention-to-treat. Outcomes included 1) survey response dichotomized as a "returned survey/did not return survey" variable, 2) the number of items missing from each scale in the survey, and 3) bias in the respondent pool relative to the total pool of Veterans selected to participate in the study. We used the American Association for Public Opinion Research definition #1 to calculate response rate. Item missingness was calculated as the percentage of unanswered items in a scale divided by its total number of items. We calculated Squared PBIAS (SqPBIAS) to assess respondent bias according to 8 key characteristics obtained from administrative data: age, race, combat exposure flags, Military Sexual Trauma screening results, service connection for PTSD, service connection for any disorder, psychiatric illness severity, and Charlson Comorbidity Index score. The SqPBIAS has an approximate Gamma distribution and represents the "distance," or bias, of the respondents in an intervention arm from the total pool of Veterans randomized to that arm. We report median distances to account for skewed distributions in some of the 8 characteristics. SqPBIAS medians are presented in a spider graph (see Quad chart) with grid lines ranging from 0.0 to 0.10. Zero (0.0) represents no bias; SqPBIAS less than 0.01 is considered "very good" or "highly satisfactory." SqPBIAS 0.0625 is potentially worrisome.

FINDINGS/RESULTS:
410 Veterans (41.5% of men and 44.0% of women) returned surveys. Across the 12 possible intervention groups, response rates ranged from 27.5% to 56.3%. Response rates were significantly higher overall when Veterans were offered $40 instead of $20 (p = 0.001) and when they were told their name was selected from a list of Veterans applying for disability benefits instead of a list of OEF/OIF/OND Veterans (p = 0.05). For both men and women, the lowest response rate (22.5% for men and 32.5% for women) was obtained when the cover letter said the survey would ask about military and life experiences, said their name had come from a list of OEF/OIF/OND Veterans, and offered a $20 honorarium.

Missing items across scales ranged from 0% to 10.4%. 55% of women and 61.3% of men answered every question on the survey. Men were most likely to answer all items when they were told the survey contained questions about combat and least likely when they were told the survey asked about military and life experiences (p = 0.04). The 3 intervention factors were not associated with women's probability of answering every item.

SqPBIAS medians ranged from 0.003 to 0.097 for women and 0.003 to 0.072 for men. As the spider graph shows, the "distance" or bias (SqPBIAS median = 0.097) was greatest for women randomized to the cover letter saying the survey would ask about military and life experiences, told them they had been selected for survey from a list of Veterans applying for disability benefits, and offered a $20 honorarium. The smallest distance (SqPBIAS median = 0.003) was seen for women randomized to the cover letter saying the survey would ask about unwanted sexual attention while in the military, told them they had been selected from a list of Veterans who filed a disability claim, and offered a $40 honorarium. The largest distance (SqPBIAS median = 0.072) occurred in the men randomized to the cover letter saying the survey would ask about military and life experiences, told them they had been selected from a list of OEF/OIF/OND Veterans, and offered a $20 honorarium. Men's smallest distance (SqPBIAS median = 0.003) was observed for the group randomized to the cover letter saying the survey would ask about combat, told them they had been selected from a list of OEF/OIF/OND Veterans, and offered a $40 honorarium.

In this study, the cover letter's content and incentive level influenced not only Veterans' decisions to return a survey or to complete all the items in a survey, but also influenced *which* Veterans would return and fully complete a survey. Specific cover letter content and a larger honorarium resulted in less biased respondent pools compared to other content and lower honoraria. Men and women responded differently to the cover letter content, suggesting that content might need to be tailored to each gender to minimize bias. These data suggest that attention to cover letter content might reduce the need for computationally expensive adjustments for bias post-survey.

IMPACT:
Mailed surveys are widely used in most scientific fields. These data show that attention to cover letter content--a very inexpensive process--could prevent the need for computationally expensive, complicated post-survey adjustments for bias. Although the impact of higher survey incentives on return rates is well known, we extended these results to show that larger incentives also reduce bias. Unfortunately, our $20 honorarium, which is quite generous and out of the range of many studies' budgets, was not sufficient to avoid potentially worrisome non-response bias. Bias adjustments typically require expert, time-consuming, expensive statistical input--so much so that even a $40 incentive could prove cost-saving. Future work should examine the cost trade-offs of larger incentives versus the costs of rehabilitating biased datasets.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX001564-01A1
Link: https://reporter.nih.gov/project-details/8864080

---

PUBLICATIONS:

Journal Articles

1. Murdoch M, Clothier BA, Beebe TJ, Bangerter AK, Noorbaloochi S. Impact of different cover letter content and incentives on non-response bias in a sample of Veterans applying for Department of Veterans Affairs disability benefits: a randomized, 3X2X2 factorial trial. BMC medical research methodology. 2022 Mar 6; 22(1):61. [view]
2. Murdoch M, Clothier BA, Kehle-Forbes S, Vang D, Noorbaloochi S. Impact of different cover letter information and incentives on Veterans' emotional responses to an unsolicited mailed survey about military traumas: a randomized, 3x2x2 factorial trial. BMC medical research methodology. 2022 Dec 1; 22(1):308. [view]

DRA: Health Systems
DRE: Treatment - Observational
Keywords: none
MeSH Terms: none