Different meta-analysis methods can change judgements about imprecision of effect estimates: a meta-epidemiological study.

Wang Z, Alzuabi MA, Morgan RL, Mustafa RA, Falck-Ytter Y, Dahm P, Sultan S, Murad MH. Different meta-analysis methods can change judgements about imprecision of effect estimates: a meta-epidemiological study. BMJ evidence-based medicine. 2023 Apr 1; 28(2):126-132.

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

If you have VA-Intranet access, click here for more information vaww.hsrd.research.va.gov/dimensions/

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.    Search Dimensions for VA for this citation
* Don't have VA-internal network access or a VA email address? Try searching the free-to-the-public version of Dimensions

Search for Abstract from PubMed

---

Abstract:

OBJECTIVES: To empirically evaluate five commonly used meta-analysis methods and their impact on imprecision judgements about effect estimates. The two fixed-effect model methods were the inverse variance method based on normal distribution and the Mantel-Haenszel method. The three random-effects model methods were the DerSimonian and Laird, the Hartung-Knapp-Sidik-Jonkman and the profile likelihood approaches. DESIGN: Meta-epidemiological study. SETTING: Meta-analyses published between 2007 and 2019 in the 10 general medical journals with the highest impact factors that evaluated a medication or device for chronic medical conditions and included at least 5 randomised trials. MAIN OUTCOME MEASURES: Discordance in the judgements of imprecision of effect estimates based on two definitions: when either boundary of 95% CI of the OR changed by more than 15% or changed in relation to the null. RESULTS: We analysed 88 meta-analyses including 1114 trials with an average of 12.60 trials per meta-analysis and average I of 26% (range: 0%-96%). The profile likelihood failed to converge in three meta-analyses (3%). Discordance in imprecision judgements based on the two definitions, respectively, occurred between the fixed normal distribution and fixed Mantel-Haenszel method (8% and 2%), between the DerSimonian and Laird and Hartung-Knapp-Sidik-Jonkman methods (19% and 10%), between the DerSimonian and Laird and profile likelihood methods (9% and 5%), and between the Hartung-Knapp-Sidik-Jonkman and profile likelihood methods (5% and 13%). Discordance was greater when fewer studies and greater heterogeneity was present. CONCLUSION: Empirical evaluation of studies of chronic medical conditions showed that conclusions about the precision of the estimates of the efficacy of a drug or device frequently changed when different pooling methods were used, particularly when the number of studies within a meta-analysis was small and statistical heterogeneity was substantial. Sensitivity analyses using more than one method may need to be considered in these two scenarios.