2012 HSR&D/QUERI National Conference Abstract
3050 — Multiple Testing and False Discovery Rate Control: Statistical Significance Adjustments in VA Health Studies
Glickman ME, Rao SR, Schultz MR, and Eisen SV, CHQOER, Bedford VAMC, MA;
The main objectives of this talk are: (1) to provide arguments that researchers should be using false discovery rate control rather than family-wise error rate control when performing multiple hypothesis tests, and (2) to clarify a common misconception about multiple testing that leads to inappropriate use of family-wise error rate control.
A common procedure when performing multiple hypothesis tests in health studies is to reduce the significance level for each test by controlling the family-wise error rate (FWER) such as through a Bonferroni correction. We argue that FWER control is generally applied inappropriately, as the reason for performing adjustments should not be based on the number of tests performed, but instead on the expected frequency of true null hypotheses. Such reasoning motivates the need for applying false discovery rate (FDR) adjustments, an approach that has frequent application in large genomic studies involving multitudes of hypothesis tests, but is all but absent from health services research and VA health research in particular. The FDR approach adjusts significance levels to ensure the frequency of true null hypotheses among rejected tests is controlled. The conceptual and practical advantages of FDR control are presented. We compare results of FWER and FDR control on a recently completed VA study involving post-deployed Veterans in which tests were performed on the relationship between mental health and substance use scores with gender, deployment operation, service component, and service branch.
In the study consisting of 44 p-values, 18 were significant at the 0.05 level. An FWER adjustment to the significance level via the Bonferroni procedure resulted in only 12 of the tests declared significant. However, using the FDR approach by Benjamini and Hochberg (1995), 17 of the tests were significant.
Adjusting conclusions of multiple hypothesis tests through FDR control is a principled alternative to conventional multiplicity adjustments based on FWER control. As a practical consequence, adjustments based on FDR control also have the advantage of generally being more powerful in detecting effects.
Researchers in the VA who perform multiple hypothesis tests in their studies should be aware of FDR adjustment as a modern approach to provide sound and reliable statistical inferences.