investigación psicológica.
International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079
Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.
International Journal of Psychological Research 151
Meta-analysis in Psychological Research.
El meta-análisis en la investigación psicológica.
Julio Sánchez-Meca and Fulgencio Marín-Martínez University of Murcia, Spain
ABSTRACT
Meta-analysis is a research methodology that aims to quantitatively integrate the results of a set of empirical
studies about a given topic. With this purpose, effect-size indices are obtained from the individual studies and the
characteristics of the studies are coded in order to examine their relationships with the effect sizes. Statistical analysis in
meta-analysis requires the weighting of each effect estimate as a function of its precision, by assuming a fixed- or a random-
effects model. This paper outlines the steps required for carrying out the statistical analyses in a meta-analysis, the different
statistical models that can be assumed, and the consequences of the assumptions in interpreting their results. The statistical
analyses are illustrated with a real example.
Key words: Meta-analysis, effect size, fixed-effects models, random-effects models, mixed-effects models.
RESUME�
El meta-análisis es una metodología de investigación que pretende integrar cuantitativamente los resultados de un
conjunto de estudios empíricos sobre un determinado problema. Con este propósito, se calculan índices del tamaño del
efecto y se codifican las características de los estudios con objeto de examinar su relación con los tamaños del efecto. El
análisis estadístico en meta-análisis requiere ponderar cada estimación del efecto en función de su precisión asumiendo un
modelo de efectos fijos o de efectos aleatorios. En este trabajo se presentan las etapas necesarias para realizar un meta-
análisis, los diferentes modelos estadísticos que pueden asumirse y las consecuencias de asumir dichos modelos en la
interpretación de sus resultados. Finalmente, los análisis estadísticos se ilustran con datos de un ejemplo real.
Palabras clave: Meta-análisis, tamaño del efecto, modelos de efectos fijos, modelos de efectos aleatorios, modelos de
efectos mixtos.
Article received/Artículo recibido: December 15, 2009/Diciembre 15, 2009, Article accepted/Artículo aceptado: March 15, 2010/Marzo 15/2010 Dirección correspondencia/Mail Address: Julio Sánchez-Meca, Dpto. Psicología Básica y Metodología, Facultad de Psicología, Campus de Espinardo, Universidad de Murcia, 30100-Murcia, Spain, E-mail: [email protected] Fulgencio Marín-Martínez, University of Murcia, Spain
INTERNATIONAL JOURNAL OF PSYCHOLOGICAL RESEARCH esta incluida en PSERINFO, CENTRO DE INFORMACION PSICOLOGICA DE COLOMBIA, OPEN JOURNAL SYSTEM, BIBLIOTECA VIRTUAL DE PSICOLOGIA (ULAPSY-BIREME), DIALNET y GOOGLE SCHOLARS. Algunos de sus articulos aparecen en SOCIAL SCIENCE RESEARCH NETWORK y está en proceso de inclusion en diversas fuentes y bases de datos internacionales. INTERNATIONAL JOURNAL OF PSYCHOLOGICAL RESEARCH is included in PSERINFO, CENTRO DE INFORMACIÓN PSICOLÓGICA DE COLOMBIA, OPEN JOURNAL SYSTEM, BIBLIOTECA VIRTUAL DE PSICOLOGIA (ULAPSY-BIREME ), DIALNET and GOOGLE SCHOLARS. Some of its articles are in SOCIAL SCIENCE RESEARCH NETWORK, and it is in the process of inclusion in a variety of sources and international databases.
International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079
Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.
152 International Journal of Psychological Research
Meta-analysis in Psychological Research
1. Introduction
In the last 30 years meta-analysis has become a very useful methodological tool for accumulating research
on a given topic. The huge growth of research in
psychology has made it very difficult to synthesize the
results in any field without the help of statistical methods to
summarize the evidence. Unlike traditional reviews on a given topic, which are essentially subjective in nature,
meta-analysis aims to imbue the research review with the
same scientific rigor that is demanded of empirical studies:
objectivity, systematization and replicability. Thus, meta-
analysis is a method used to quantitatively integrate the
results of a set of empirical studies on a given research question. With this purpose, the results of each individual
study included in a meta-analysis have to be quantified in
the same metric, usually by calculating an effect-size index,
and then the effect estimates are statistically analyzed in
order to: (a) obtain an average estimate of the effect
magnitude, (b) assess heterogeneity among the effect
estimates, and (c) search for characteristics of the studies
that can explain the heterogeneity (Cooper, 2010; Cooper,
Hedges, & Valentine, 2009; Hunter & Schmidt, 2004;
Lipsey & Wilson, 2001; Petticrew & Roberts, 2006).
As meta-analysis aims to integrate single studies, the analysis unit is not the participant, but the single study.
Therefore, the sample size in a meta-analysis is the number
of studies that it has been possible to recover regarding the
research question.
Meta-analysis is being applied in many different
fields in psychology, but especially in evaluating the
effectiveness of treatments, interventions, and prevention
programs in such settings as mental health, education,
social services, or human resources. Other psychological
fields where meta-analysis is also being applied include areas such as gender differences in childhood, adolescence
or with adults of many aptitudes and attitudes;
psychometric validity of employment tests, and reliability
generalization of psychological tests in general (Cook,
Cooper, Cordray et al., 1992). Nowadays, it is very
common to find meta-analytic studies on very different topics in any scientific psychology journal. Therefore,
clinicians and researchers should have a sufficient
knowledge base for correctly interpreting and/or carrying
out meta-analyses.
This article is divided into four sections. Firstly,
the phases in which a meta-analysis is carried out are
presented. Then we outline the main statistical methods in
meta-analysis. In the next section statistical methods for
meta-analysis are illustrated using a real example. Finally,
we present some concluding remarks.
2. Phases in a Meta-analysis
A meta-analysis is a scientific investigation and,
consequently, it involves carrying out the same phases as in an empirical study. However, some of the phases have a
few specificities that it is necessary to mention. Basically,
we can conduct a meta-analysis in six phases: (1) Defining
the research question; (2) literature search; (3) coding of
studies; (4) calculating an effect-size index; (5) statistical
analysis and interpretation, and (6) publication (Cooper,
2010; Egger, Davey Smith, & Altman, 2001; Lipsey &
Wilson, 2001; Littell, Corcoran, & Pillai, 2008; Sánchez-
Meca & Marín-Martínez, 2010.
(1) Defining the research question. As in any empirical study, the first step in a meta-analysis is to define
the research question as clearly and objectively as possible.
This implies proposing conceptual and operational
definitions of the different concepts and constructs related
to the research question. For example, in a meta-analysis
about the efficacy of psychological treatments of obsessive-
compulsive disorder (OCD), constructs such as
psychological treatment, obsessive-compulsive disorder,
and the measurement tools to assess efficacy were defined
in this phase (Rosa-Alcázar, Sánchez-Meca, Gómez-
Conesa, & Marín-Martínez, 2008).
(2) Literature search. Once the research question
is formulated, the next step consists of defining the
eligibility criteria of the single studies, that is, the
characteristics a study must fulfill in order to be included in
the meta-analysis. The selection criteria will depend on the
purpose of the meta-analysis, but it is always necessary to
specify the types of study designs that will be accepted
(e.g., only experimental designs, or also quasi-experimental
ones, etc.). For example, in the meta-analysis on OCD
(Rosa-Alcázar et al., 2008) in order to be included in the
meta-analysis the studies had to fulfill several criteria: (a) to apply a psychological treatment to adult patients with OCD;
(b) to include a control group with OCD patients; (c) to
report statistical data for calculating the effect sizes; (d) to
have at least 5 participants in each group, and (e) to be
published between 1980 and 2006.
In this phase the different strategies used to locate
the single studies are also specified. No meta-analysis is
complete without a search of electronic databases
specifying the keywords used (e.g., PsycInfo, MedLine,
ERIC). This search strategy is usually complemented by carrying out searches by hand of relevant journals and
books for the topic of interest, and by checking the
references of the papers included in the meta-analysis.
Additionally, it is very advisable to try to locate
unpublished papers that might fulfill the selection criteria,
in order to counteract publication bias. This can be done by
International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079
Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.
International Journal of Psychological Research 153
sending letters to well-known researchers in the field
requesting unpublished papers about the topic.
(3) Coding of studies. Once we have the single studies included in the meta-analysis, the next step is to
record the main characteristics of the studies in order to
later explain the heterogeneity exhibited by the effect sizes.
The characteristics of the studies, or moderator variables,
are classified as substantive, methodological, and extrinsic
variables. Substantive characteristics are those related to the
research question of the meta-analysis, whereas
methodological variables are characteristics related to the
study design. Finally, extrinsic variables refer to those
characteristics that, despite are not related with the subjects
nor the study design, could also have an influence in the results. In the OCD example (Rosa-Alcázar et al., 2008),
substantive characteristics coded in the studies included the
type of psychological treatment (e.g., cognitive therapy,
exposure techniques), the mean age of the participants and
the illness history (in years). Some of the methodological
characteristics coded included the type of design
(experimental versus quasi-experimental), attrition in the
posttest, and the sample size. Moreover, extrinsic variables
such as the country where the study was carried out and the
education profile of the main author were also coded.
The coding norms of the moderator variables are
written in a codebook. Some study characteristics are
difficult to code due to incomplete or ambiguous reporting
in the single studies. Therefore, the reliability of the coding
process should be analyzed. To this end, two (or more)
researchers should independently apply the codebook to all
or a sample of the single studies. Then, using the coding
records made by the researchers, agreement indices are
applied (e.g., kappa coefficients, intraclass correlations) in
order to assess the reliability of the coding process.
(4) Calculating an effect-size index. In the coding process of single studies, an effect-size index also has to be
calculated in order to quantify the results of each study in a
common metric. Depending on the study design and the
type of dependent variables (continuous, dichotomous),
different effect-size indices can be applied. Thus, when the
studies have a two-group design and the outcome measure is continuous, the most appropriate effect-size index is the
standardized mean difference or d. This is defined as the
difference between the two means divided by a pooled
within-study standard deviation. Furthermore, when the
dependent variable is dichotomous, several risk indices can be applied: (a) the risk difference, rd, defined as the
difference between the failure (or success) proportions for
the two groups; (b) the risk ratio, rr, defined as the ratio
between the two proportions, and (c) the odds ratio, or,
defined as the ratio between the odds of the two groups.
Finally, when the study applied a correlational design, a correlation coefficient can be used as the effect-size index
(e.g., the Pearson correlation coefficient, its Fisher’s Z
transformation, the point-biserial correlation coefficient, the
phi coefficient, etc.). Table 1 presents some of the usual
effect-size indices applied in meta-analysis together with
their estimated sampling variances, 2
i σ̂ , as they are used in
the statistical analyses of a meta-analysis (cf. Borenstein,
Hedges, Higgins, & Rothstein, 2009; Cooper et al., 2009).
Once the effect-size index most appropriate to the characteristics of the studies has been selected, it is applied
to each single study and its sampling variance is also
calculated with the corresponding formulas (cf., e.g.,
Borenstein et al., 2009). When a meta-analysis includes
studies with different designs (e.g., correlational and two-
group designs), there are formulas to transform different
effect-size indices into each other. For example, it is
possible to transform correlation coefficients into d indices,
and vice versa; or odds ratios into d indices (Sánchez-Meca,
Marín-Martínez, & Chacón-Moscoso, 2003).
(5) Statistical analysis and interpretation. The dataset in a meta-analysis is composed of a matrix where
the rows are the studies and the columns are the moderator
variables, the effect-size index calculated in each study, and
its sampling variance. With these data it is possible to carry
out statistical analyses, which have the following three
main objectives: (1) to calculate an average effect size and
its confidence interval; (b) to assess the heterogeneity of the
effect sizes around the average, and (c) to search for
moderator variables that can explain the heterogeneity
(Sutton & Higgins, 2008). The main characteristic of meta-
analysis is that statistical methods are used for integrating the study results. More details about how to statistically
analyze a meta-analytic database are presented in the next
point of this article.
(6) Publication. Finally, the results of a meta-
analysis have to be published following the same structure as any other scientific paper: Introduction, method, results,
and discussion and conclusions (Botella & Gambara, 2006;
Rosenthal, 1995). A literature review on the topic is
outlined in the introduction, together with definitions of the
constructs and variables implied in the research question,
and the objectives and hypotheses of the meta-analysis. In
the method section the following should be included: the
selection criteria of the studies, the search strategy of the
studies, the coding process of the study characteristics, the
effect-size index calculated in the single studies, and the
statistical analyses that were carried out in the meta-
analytic integration. In the results section the characteristics
of the studies are presented, together with the effect-size
distribution, the mean effect size, the heterogeneity
assessment, and the results of the statistical analyses for
searching for moderator variables related to the effect sizes.
Finally, in the discussion and conclusion section the results
International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079
Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.
154 International Journal of Psychological Research
of the meta-analysis are compared with previous ones, the
implications for future research are mentioned, and the
limitations and the main conclusions of the meta-analysis
are also outlined.
Table 1. Effect-size indices and their respective estimated within-study sampling variances
Effect-size index Ti Estimated sampling variance, 2
i σ̂
Mean difference CE