Comparing traditional and Bayesian approaches to ecological meta-analysis

Paula Pappalardo, Kiona Ogle, Elizabeth A. Hamman, James R. Bence, Bruce A. Hungate, Craig W. Osenberg

Research output: Contribution to journalArticlepeer-review

Abstract

Despite the wide application of meta-analysis in ecology, some of the traditional methods used for meta-analysis may not perform well given the type of data characteristic of ecological meta-analyses. We reviewed published meta-analyses on the ecological impacts of global climate change, evaluating the number of replicates used in the primary studies (ni) and the number of studies or records (k) that were aggregated to calculate a mean effect size. We used the results of the review in a simulation experiment to assess the performance of conventional frequentist and Bayesian meta-analysis methods for estimating a mean effect size and its uncertainty interval. Our literature review showed that ni and k were highly variable, distributions were right-skewed and were generally small (median ni = 5, median k = 44). Our simulations show that the choice of method for calculating uncertainty intervals was critical for obtaining appropriate coverage (close to the nominal value of 0.95). When k was low (<40), 95% coverage was achieved by a confidence interval (CI) based on the t distribution that uses an adjusted standard error (the Hartung–Knapp–Sidik–Jonkman, HKSJ), or by a Bayesian credible interval, whereas bootstrap or z distribution CIs had lower coverage. Despite the importance of the method to calculate the uncertainty interval, 39% of the meta-analyses reviewed did not report the method used, and of the 61% that did, 94% used a potentially problematic method, which may be a consequence of software defaults. In general, for a simple random-effects meta-analysis, the performance of the best frequentist and Bayesian methods was similar for the same combinations of factors (k and mean replication), though the Bayesian approach had higher than nominal (>95%) coverage for the mean effect when k was very low (k < 15). Our literature review suggests that many meta-analyses that used z distribution or bootstrapping CIs may have overestimated the statistical significance of their results when the number of studies was low; more appropriate methods need to be adopted in ecological meta-analyses.

Original languageEnglish (US)
JournalMethods in Ecology and Evolution
DOIs
StateAccepted/In press - 2020

Keywords

  • bias
  • confidence interval
  • coverage
  • credible interval
  • effect size
  • global climate change
  • log response ratio
  • sample size

ASJC Scopus subject areas

  • Ecology, Evolution, Behavior and Systematics
  • Ecological Modeling

Fingerprint Dive into the research topics of 'Comparing traditional and Bayesian approaches to ecological meta-analysis'. Together they form a unique fingerprint.

Cite this