Meta-Analysis as a Research Synthesis Methodology: Cause for Concern

ABSTRACT

The use of meta-analysis is growing in popularity. It is based on the fundamental notion of the effect size, and a critical assumption is that effect sizes based on different measures are directly comparable. In this article it is argued that the direct comparability of effect sizes across measures implies the invariance of the effect sizes across these measures. A model of standardized mean difference (SMD) effect size invariance is developed, based on multifacet generalizability theory, which shows that SMD effect size invariance requires certain validity invariance conditions to hold. One implication of these findings is that the direct comparability of SMD effect sizes based on different measurement procedures is an empirical matter requiring testing prior to conducting a meta-analysis. Findings are also discussed suggesting that violations of one of these conditions—universe score validity invariance—can bring about substantial differences across SMD effect sizes as a function of measurement procedure. These findings suggest the need for a more refined use of meta-analysis since meta-analytic results may be adversely impacted by the lack of direct comparability of effect sizes based on different measures.