Summary the Power, Potential, and Perils of Meta-Analysis: a Workshop on Integrative Reviews

Citation:

Steven P. Brown (1994) ,"Summary the Power, Potential, and Perils of Meta-Analysis: a Workshop on Integrative Reviews", in NA - Advances in Consumer Research Volume 21, eds. Chris T. Allen and Deborah Roedder John, Provo, UT : Association for Consumer Research, Pages: 353.

Advances in Consumer Research Volume 21, 1994      Page 353

SUMMARY

THE POWER, POTENTIAL, AND PERILS OF META-ANALYSIS: A WORKSHOP ON INTEGRATIVE REVIEWS

Steven P. Brown, Southern Methodist University

As its title implied, this special session discussed the power of meta-analysis (e.g., to overcome the limitations and assumptions inherent in statistical significance testing), its potential for testing theory and establishing prior probabilities for particular relationships, and the perils associated with statistical problems such as heteroskedasticity and truncation. The session questioned assumptions often made by researchers regarding error rates in empirical research and the meaning of statistical significance. It also questioned assumptions made about meta-analysis (e.g., that it is useful only for summarizing research results but not for theory development or testing). The session chair Kent B. Monroe of the University of Illinois described the background to the session proposal and introduced the speakers.

The first paper, "What Do Data Really Mean?," was delivered by Frank L. Schmidt of the Department of Management and Organization, University of Iowa. Professor Schmidt, one of the world's leading experts on meta-analysis, described the limitations of statistical significance testing and pointed out ways that meta-analysis overcomes these limitations. He observed that the assumption often made by researchers that error rates in their research approximate the alpha levels specified for Type I error is erroneous. Using examples based on sample sizes typical of consumer research, he showed that Type II error is a much more serious hazard, with rates exceeding .50. He also criticized the assumption that statistically non-significant effects are equal to zero, noting the failure to find an effect may usually be attributable to insufficient power in the study design and that non-significant effects are usually not equal to zero. Professor Schmidt concluded that statistical significance testing is "an addiction" that researchers should fight to break.

The second paper, "Meta-Analysis for Model Estimation," was delivered by Professor Donald R. Lehmann of Columbia University. Professor Lehmann discussed the uses of meta-analysis for establishing prior probabilities regarding the strength of relationships of interest in situations where data are sparse or expensive (e.g., new product forecasting). In such situations, estimates of patterns (e.g., life-cycle shapes) or relations among variables (e.g., advertising to awareness or attitude) often rely on other "relevant" information, usually estimates based on prior studies. He summarized some past work suggesting that generalization across products and situations is the rule rather than the exception. He concluded by suggesting how to perform meta-analysis across studies even when the individual studies do not contain sufficient data to allow for estimation of a relationship on a study-by-study basis.

The session's final paper, "Validity Threats in Meta-Analysis," was delivered by Murali Chandrashekaran of the University of Cincinnati (co-authored with Beth A. Walker of Arizona State University). This paper focused on two statistical threats to validity in regression of effect sizes from meta-analysis on potential moderator variables, namely heteroskedasticity and truncation. The paper proposed a maximum-likelihood estimation technique that yields greater power and efficiency than OLS estimation. A Monte Carlo study compared estimation methods and found the maximum-likelihood technique superior.

Discussion of the three papers was provided by Steven P. Brown of Southern Methodist University. He reviewed a number of assumptions commonly made by consumer researchers (e.g., that error rates approximate alpha rates for Type I error, that Type I error is the type of error that researchers should be most concerned with, that larger effect sizes are in some sense "better," that the null hypothesis is the appropriate test in most studies, and that meta-analysis is useful only for summarizing prior research results but not for developing or testing theory) and reviewed how the papers had shown these assumptions to be shaky at best and more often erroneous. He concluded that the three papers in sequence addressed issues related to the "power, potential, and perils" of meta-analysis as per the session's billing.

----------------------------------------