Module 7 - Planning the Meta-Analysis

Planning Your Meta-Analysis: From Systematic Review to Quantitative Synthesis

Planning a meta-analysis is a critical step in a systematic review. After expending significant effort to identify studies, the next stage involves quantitatively analyzing their data. This article outlines a general framework for quantitative analysis, defines meta-analysis, discusses its advantages, and explores the types of data and effect measures used.

By the end of this discussion, the key principles for a successful meta-analysis will be clear:

• Effective analysis demands a thoughtful approach to both qualitative and quantitative elements.
• A robust analytical framework is essential, specifying comparisons, data types, and summary measures, all of which must directly address the study question.
• Meta-analysis is the statistical combination of effect estimates from two or more separate and independent studies.

I. The Foundation of Systematic Review and Meta-Analysis

The results of a meta-analysis can be misleading if insufficient attention is paid to the initial stages of a systematic review.

A. Steps in a Systematic Review

1. Formulate the review question: This determines the eligibility criteria.
2. Identify, select, and critically appraise studies: Based on eligibility criteria and a predefined protocol.
3. Collect appropriate data: From the included studies.
4. Conduct meta-analysis: An optional component of a systematic review, focusing on planning the analysis of primary studies (where the unit of analysis is the individual study).

B. Qualitative vs. Quantitative Synthesis

Any systematic review analysis comprises two main components:

• Qualitative analysis (qualitative synthesis)
• Quantitative synthesis (meta-analysis)

1. Qualitative Synthesis: The Essential First Step Qualitative synthesis is the most crucial part of any systematic review. It involves a structured summary, description, and discussion of study characteristics that may influence the cumulative evidence. This means carefully examining studies and asking:

• Are the studies similar or different?
• Are the study participants comparable?
• Are the interventions homogeneous across included studies?
• Are the studies designed, conducted, and reported properly?
• Is there sufficient information from the studies for meta-analysis?

Significant effort should be dedicated to thorough qualitative synthesis before proceeding to quantitative analysis.

2. General Framework for Synthesis Questions Once a comprehensive qualitative synthesis is complete, the focus shifts to quantitative analysis (meta-analysis). This involves answering questions about the synthesized evidence:

• What is the direction of effect or association (e.g., is the intervention preventive or harmful)?
• What is the size of effect (the point estimate)?
• Are the effects consistent across studies?
• If results are inconsistent, why are they different?
• What is the strength of evidence for the effect? This judgment relies on assessing study design, risk of bias, and statistical measures of uncertainty. For example, a small, poorly reported study showing a strong effect may not warrant high confidence due to risk of bias concerns.

II. Framing Your Meta-Analysis Plan

The analysis plan must align directly with the aims of the review. Different analytical techniques are chosen based on the review’s objectives.

A. Aligning Analysis with Review Aims

1. Pair-wise Meta-analysis

• Goal: Obtain the size of effect from similar studies estimating the same effect, comparing two interventions (e.g., mechanical vs. manual chest compressions for cardiac arrest).
• Focus: This is the standard type of analysis covered in this discussion.

2. Network Meta-analysis

• Goal: Draw inferences about the comparative effectiveness of multiple interventions for the same condition (e.g., four classes of drugs with 16 different eye drops for primary open-angle glaucoma). Pair-wise meta-analysis would be insufficient here.
• Method: Addresses questions like “Which of these 11 drugs is most effective?” and “What is the probability that a drug will be ranked as the best?”
• Discussion: Covered in a separate lecture.

3. Meta-regression

• Goal: Evaluate the relationship between the size of an effect and certain study characteristics (e.g., is the efficacy of the BCG vaccine associated with latitude?).
• Method: Used to examine factors that differ across studies and answer questions not posed by individual studies (since each individual study would have only one baseline level for a characteristic).
• Discussion: Covered in a separate lecture.

4. Other Aims Regardless of the specific aim, the analysis plan must flow directly from the review’s objective, dictating the choice of meta-analytical technique.

III. Understanding Meta-Analysis: A Closer Look

Meta-analysis is an optional component of a systematic review; not all reviews must include one.

A. Defining Meta-Analysis

Meta-analysis is formally defined as “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings.” Alternatively, it is “a statistical analysis which combines the results of several independent studies considered by the analyst to be combinable.” As the systematic reviewer or data analyst, you must decide if it makes sense to combine a set of studies.

Conventional meta-analysis typically focuses on pair-wise comparisons of interventions or exposure-outcome relationships.

B. Deciding Which Studies to Combine

The most challenging decision for a reviewer is to determine which studies are appropriate for combination in a meta-analysis. Justification for combining results typically arises when:

• Studies are estimating, in whole or in part, a common effect.
• Studies address the same fundamental biological, clinical, or mechanistic question.

Example: Interferon Therapy for Hepatitis C Rarely will two studies be identical. Studies often differ (e.g., higher/lower interferon therapy dose, participants’ age/education/health, geographic location like Asia vs. North America, different Hepatitis C subtypes). The reviewer must judge whether these differences are small enough to warrant combining studies or too significant to allow for a meaningful combined analysis.

C. Visualizing Meta-Analysis: The Forest Plot

A typical figure showing meta-analysis results is called a Forest Plot.

• Each included study (e.g., “Kunif 1997”) is represented by its point estimate (a square) and a 95% confidence interval (a horizontal line).
• The size of the square for each study differs, with larger squares indicating a larger study and more weight in the meta-analysis.
• A line of no effect (or null value) is typically centered (e.g., a risk ratio of one).
• The meta-analytical result is represented by a diamond at the bottom of the plot. The center of the diamond indicates the pooled estimate, and its width reflects the 95% confidence interval for the combined effect.

D. The Weighted Average: Core of Meta-Analysis

In a forest plot, each study’s result (e.g., risk ratio) is summarized. The overall measure of effect is a weighted average of the results of the individual studies. The weight assigned to a trial is greater if it provides more information, primarily due to a larger sample size, which leads to increased precision.

1. Formula for Pooled Estimate The inverse variance weighted average formula is: $`\frac{\sum (Y_i \times W_i)}{\sum W_i}`$

Where:

• Y_i refers to the intervention effect, measure of association, or measure of the effect estimated in the ith study.
• W_i is the weight given to the ith study.

2. Calculating Variance for Confidence Intervals The variance for the weighted average is: $\frac{1}{\sum W_i}$ This variance is used to construct the 95% confidence interval for the pooled estimate.

3. Example Calculation Walkthrough Let’s illustrate with a hypothetical example using 6 studies and Odds Ratio (OR) as the measure of association.

Study 1 Data (from 2x2 table):

• Treated (Event/No Event): 12 / 53
• Comparison (Event/No Event): 16 / 49
• Odds Ratio (OR): 0.69
• Y_1 (Log OR): log(0.69) = -0.36
• Variance for Y_1: 0.18
• W_1 (Weight): 1 / Variance = 1 / 0.18 = 5.4

The same steps are followed to obtain the log OR, variance, and weight for each of the 6 studies.

Study	Odds Ratio	Log(OR) (Y_i)	Variance	Weight (W_i) = 1/Variance
1	0.69	-0.36	0.19	5.40
2	…	…	0.29	3.45
3	…	…	0.05	20.00
4	…	…	0.06	17.16
5	…	…	0.12	8.33
6	…	…	0.23	4.35
Total		-30.59		42.25

• Pooled Log OR: -30.59 / 42.25 = -0.72
• Pooled OR: exp(-0.72) = 0.48
• Variance for Pooled OR: 1 / 42.25 = 0.024
• The square root of this variance gives the standard error, which is used to calculate the 95% confidence interval for the pooled odds ratio.

This example illustrates the calculation for a fixed-effect meta-analysis, where weight is the inverse of the variance of the effect estimate. Other weighting methods exist, such as for random-effects meta-analysis, where the weight calculation differs slightly.

4. Software and Reporting Variations While these calculations can be done manually, most meta-analytical software performs them automatically, producing forest plots. Studies with larger weights (e.g., Study 4 in the example, with 41% relative weight) dominate the meta-analysis result.

It is crucial to note that a systematic review does not have to include a meta-analysis. If studies are highly heterogeneous (e.g., three studies for amblyopia with disparate visual acuity outcomes), authors may decide not to combine them statistically. In such cases, a forest plot can still be used to display individual study results without a pooled diamond. This highlights that meta-analysis is only appropriate when studies are comparable and homogeneous enough to estimate a similar effect.

E. Interpreting Meta-Analysis Results

Like any study, the meta-analysis estimate and confidence interval must be interpreted in the context of clinically important effect size.

• A statistically significant result might not be clinically important.
• A result that is not statistically significant may still be compatible with a clinically important effect.

Remember the principle: “Absence of evidence is not evidence of absence.” Interpretation should prioritize the clinical question over sole reliance on p-values or statistical significance.

IV. Why Conduct a Meta-Analysis? Advantages and Benefits

There are several compelling reasons to perform a meta-analysis:

A. Increased Power and Precision

If the included studies are homogeneous, meta-analysis can:

• Increase statistical power: Making it possible to detect effects as statistically significant.
• Increase precision: Resulting in narrower confidence intervals.
• Quantify effect size and uncertainty: Providing a precise overall estimate.
• Reduce interpretation problems: Mitigating issues due to sampling variation by looking at all studies together.
• Assess homogeneity/heterogeneity: Quantifying between-study variation.

Example: Intravenous Streptokinase for Acute Myocardial Infarction Early, smaller studies (often with 95% confidence intervals crossing the null value of one for mortality at three months) were underpowered to assess mortality. By combining 38 studies, a meta-analysis (represented by the diamond at the bottom) conclusively showed that streptokinase is highly effective in lowering mortality, a conclusion not evident from individual small studies alone.

B. Addressing Heterogeneity and Consistency

Meta-analysis helps to:

• Examine factors that differ across studies.
• Settle controversies arising from conflicting studies.

C. Answering Broader Research Questions

Meta-analysis can answer questions not posed by individual studies, as individual studies provide only one specific context or comparison.

1. Examining Factors Across Studies (Meta-regression) Example: Effectiveness of Toothpaste and Baseline Cavities A meta-regression can explore if toothpaste efficacy (preventable fraction of new cavities on the y-axis) is associated with baseline population levels of cavities (x-axis). Each bubble represents a study, its size proportional to its weight. A regression line can reveal, for instance, that higher baseline cavity levels might correlate with a greater preventable fraction of new cavities by toothpaste.

2. Comparative Effectiveness of Multiple Interventions (Network Meta-analysis) Example: Medical Interventions for Primary Open-Angle Glaucoma For a condition like primary open-angle glaucoma, where multiple drug classes and numerous individual drugs exist (e.g., 11 active drugs plus placebo/no treatment), clinicians need to know which is most effective.

• A “network plot” shows drugs as circles (size proportional to patient count) and lines connecting drugs that have been directly compared in randomized controlled trials.
• Since not all drugs are directly compared (e.g., Latanoprost vs. Carteolol), conventional pair-wise meta-analysis cannot directly compare all 11 drugs.
• Network meta-analysis allows for indirect comparisons and ranking of interventions, providing probabilities of a drug being best, second best, etc.

3. Generating New Hypotheses and Future Research Directions Systematic reviews and meta-analyses are crucial for identifying deficiencies in existing literature or evidence. This allows for:

• Recommendations for new studies: Detailing optimal study designs.
• Specification of parameters: For example, a Cochrane review on treatment for dental lesions noted the need for well-designed, long-term randomized trials. It further specified that new studies should standardize parameters such as initial lesion size, patient characteristics, tooth type/location, operator skill, clinical procedures, magnification devices, instrumentation, and materials.

V. When to Exercise Caution: Limitations of Meta-Analysis

Not every question requires a meta-analysis, and its utility is only as good as the studies it includes.

A. The “Garbage In, Garbage Out” Principle

If all included studies are biased or have design/reporting deficiencies, a meta-analysis will produce a very precise but biased result, which can be worse than the individual biased studies on their own.

B. Beware of Reporting Biases

Not all studies are reported, and those with statistically significant or positive results are more likely to be published than those with negative or null findings. This publication bias can distort meta-analytic conclusions.

C. The “Apples and Oranges” Dilemma

While combining truly disparate studies (“mixing apples and oranges”) is problematic, the core issue lies in the research question’s scope.

• A meta-analysis is not useful if the goal is to understand only “apples” but combines “apples” and “oranges.”
• However, if the goal is to understand “fruit” in general, combining them might be appropriate.

Studies must address the same fundamental question, though the question can (and often must) be broader than that posed in any individual study.

VI. Essential Considerations for Meta-Analysis

To conduct a meta-analysis, several critical questions must be addressed:

A. Key Questions to Address

• Which comparisons should be made (e.g., two interventions or multiple)?
• Which study results should be used for each comparison (e.g., 3-month mortality, 1-year mortality, or 24-month quality of life)?
• For each outcome, what is the best summary measure of effect for each comparison (e.g., risk ratio, odds ratio, or mean difference)?
• Are the results of studies similar for each comparison? This involves examining characteristics of participants, interventions, outcomes, and risk of bias across studies, in addition to visual inspection via forest plots.
• How reliable are those summaries (referring to the risk of bias of individual studies)?

VII. Data Types and Effect Measures in Meta-Analysis

This section serves as a refresher on data types and effect measures relevant to systematic reviews and meta-analysis.

A. Understanding “Effect” in Research

In systematic reviews, “effect” can refer to:

• The effect of healthcare interventions.
• The association between an exposure and an outcome.
• Prevalence or incidence.

When comparing two interventions or groups (common in randomized controlled trials), the effect is usually quantified as a difference or a ratio. These are often called effect measures for randomized controlled trials or measures of association for observational studies. Choosing the appropriate measure depends entirely on the type of data available.

B. Common Data Types in Human Research

1. Dichotomous Data: Two categories (e.g., alive/dead, event/non-event, heart attack/no heart attack).
2. Continuous Data: No gaps between levels (e.g., blood pressure, height, weight, vision measured as number of letters read).
3. Ordinal Data: Ordered categories (e.g., Likert scale: none, mild, moderate, severe; or number of repeated treatments: 1, 2, 3, 4).
4. Counts: For infrequent events (e.g., number of strokes, number of adverse events).
5. Time-to-Event Data: Survival time (e.g., time to cancer remission, time to death).

Example: Visual Acuity in Age-Related Macular Degeneration Study

• Loss of less than 15 letters from baseline: Dichotomous (either lost <15 letters or >=15 letters).
• Change in number of letters: Continuous (e.g., -8.2 +/- 16.3 letters).
• Number of repeated treatments (1, 2, 3, 4): Ordinal.

C. Choosing Measures for Dichotomous Data

Dichotomous data are typically represented in a 2x2 table (Test Intervention vs. Comparison Intervention, with Event vs. No Event counts). Measures include:

1. Risk Ratio (RR):

   • Definition: Ratio of risk (proportion/probability of event) in the treatment group divided by risk in the control group.
   • Features:
     • Easy to interpret and explain (“probability of having the event in the treatment group compared to the control group is X”).
     • Not typically reported in multivariate analyses.
     • Cannot be calculated if there are no events in the control (denominator) group.

2. Odds Ratio (OR):

   • Definition: Ratio of the odds of an event in the treatment group divided by the odds in the control group.
   • Features:
     • Best-developed statistical methodology, especially for covariate adjustment.
     • Can be calculated from study designs (e.g., case-control studies) that do not allow calculation of absolute risks or rates.
     • Can be difficult to interpret, especially when the baseline rate is above 20%.

   OR vs. RR:

   • Not the same: Calculated using different formulas, yielding different numerical values for the same 2x2 table.
   • OR is more extreme: The odds ratio is always further from the null value of one than the relative risk. For example, if RR = 0.8, OR will be less than 0.8 (e.g., 0.7).
   • Misinterpretation: If OR is misinterpreted as RR, the intervention effect will be overestimated (e.g., misinterpreting an OR of 0.7 as a 30% risk reduction instead of a 30% reduction in odds). Interpretation is fine if OR is understood as OR.
   • Mixed reporting: If some studies report RR and others OR, they should be analyzed separately by their reported measure of effect. A single meta-analysis can display multiple diamonds, one for each measure (e.g., separate pooled estimates for RR, OR, Hazard Ratio, Incidence Rate Ratio).

3. Risk Difference (RD) and Number Needed to Treat (NNT):

   • Risk Difference: Risk on treatment minus risk on control.
   • Number Needed to Treat (NNT): Inverse of the risk difference (1/RD).
   • Features:
     • Directly relates to the number of patients who benefit from therapy.
     • Fairly easy to interpret.
     • Can be calculated from any study design, even with zero events in either group (unlike ratios).
     • Less often constant, particularly with substantive variation in baseline risk.
     • Not often reported in studies or calculable if multivariate adjustment was performed.

   Consistency of Measures: When baseline risk varies across studies, consistency in one measure (e.g., risk difference) implies variability in another (e.g., risk ratio or odds ratio), and vice versa. For example, if risk difference is consistently -10% across studies with varying baseline mortality (30%, 17%, 62%), the risk ratios and odds ratios for those studies will be heterogeneous.

4. Handling Zero Cell Counts:

   • Problem: If there are no events in a treatment or control group (zero cell count), ratios (RR, OR) cannot be calculated directly.
   • Software Correction: Most meta-analytical software automatically adds a fixed value (typically 0.5) to all zero cells, though this can bias estimates towards no difference and overestimate variance. Non-fixed zero-cell corrections are also available.
   • Recommendation: If no events occur in both arms of a study, exclude that study from ratio-based meta-analysis or use risk difference instead, as it does not face this calculation problem.

D. Choosing Measures for Continuous Data

1. Mean Difference (MD):

   • Use: When all studies use the same scale for the outcome (e.g., blood pressure, change in refractive error in diopters).
   • Interpretation: Direct difference in means.

2. Standardized Mean Difference (SMD):

   • Use: When studies measure the same underlying construct (e.g., patient satisfaction, IQ) but use different scales or instruments (e.g., satisfaction scales from 0-100 vs. 0-5).
   • Definition: Difference in means between groups divided by the standard deviation of the outcome among participants.
   • Interpretation: The intervention favors by “X” standard deviations (e.g., 0.28 standard deviations). This can be harder for patients to interpret clinically than a direct mean difference.
   • Preference: Mean difference is generally preferred when possible; SMD is used when instruments differ.

E. Measures for Other Data Types (Counts, Time-to-Event)

• Counts: Can be dichotomized (any events vs. none), treated as continuous data for common events, or analyzed as rates using models like Poisson.
• Time-to-Event Data:
  • Can be dichotomized if patient status (event/no event) is known at a fixed time point for all patients (e.g., proportion with event before 12 months).
  • If studies report hazard ratios or use other survival analysis methods, these can be meta-analyzed directly.

F. Guiding Principles for Measure Selection

The choice of measure should always:

• Convey necessary clinically meaningful information. Consider measures useful for making clinical decisions (e.g., “reducing blood pressure by 20%” rather than a continuous measure for a patient).
• Be suitable to the study design.
• Be statistically appropriate and convenient to work with.

Conclusion

Planning a meta-analysis requires careful consideration of the systematic review’s aims, the characteristics of the included studies, and the types of data available. Understanding the nuances of qualitative synthesis, the principles of weighted averaging, and the appropriate application of various effect measures are fundamental to conducting a robust and clinically meaningful meta-analysis. While meta-analysis offers significant advantages in increasing precision and answering complex questions, it is crucial to recognize its limitations and apply it judiciously.

Core Concepts

• Qualitative Synthesis: A crucial initial step in a systematic review involving a structured summary, description, and discussion of study characteristics to assess their comparability and potential impact on cumulative evidence.
• Meta-Analysis (Quantitative Synthesis): The statistical combination of effect estimates from two or more separate and independent studies to integrate findings and derive a pooled estimate.
• Forest Plot: A graphical representation used in meta-analysis to display the results of individual studies and their combined effect, often including confidence intervals and study weights.
• Weighted Average: The method used in meta-analysis to combine individual study results, where each study’s contribution to the overall pooled estimate is proportional to its precision or sample size.
• Types of Data and Effect Measures: Categorization of outcome data (e.g., dichotomous, continuous) and the corresponding statistical measures (e.g., Risk Ratio, Odds Ratio, Mean Difference, Standardized Mean Difference) used to quantify intervention effects or associations.

Concept Details and Examples

Qualitative Synthesis

Detailed Explanation: Qualitative synthesis is the foundational step before considering quantitative analysis. It involves a thorough examination of individual study characteristics, such as participant demographics, intervention details, outcome definitions, and study design, to understand their similarities and differences. This critical assessment helps determine if studies are sufficiently homogeneous to be combined statistically and identifies potential sources of heterogeneity.

Examples:

1. A systematic reviewer assesses studies on a new diabetes medication, noting that some trials recruited only adults with Type 2 diabetes, while others included adolescents with Type 1, leading to a decision that these populations are too different for a combined meta-analysis.
2. When reviewing studies on a surgical procedure, a researcher identifies that some studies used minimally invasive techniques, while others used open surgery, necessitating a careful discussion in the qualitative synthesis about the clinical comparability of the interventions.

Common Pitfalls/Misconceptions: A common pitfall is rushing directly to meta-analysis without sufficient qualitative synthesis, which can lead to inappropriate pooling of vastly different studies, a concept often referred to as “mixing apples and oranges.” A misconception is that if studies aren’t perfectly identical, meta-analysis is impossible; however, qualitative synthesis helps identify acceptable variations and informs decisions about sub-group analyses or reasons for not pooling.

Application Scenario

A research team is planning a systematic review on the effectiveness of different behavioral interventions for reducing smoking rates among adolescents. They have identified 15 randomized controlled trials, some reporting continuous outcomes like daily cigarette count and others dichotomous outcomes like ‘quit status at 6 months’. The team needs to determine if a meta-analysis is feasible and how to handle the varied data types.

In this scenario, the team would first engage in qualitative synthesis to critically appraise each of the 15 studies, examining participant demographics, intervention components, and follow-up durations to assess homogeneity. Based on their findings, they would frame their analysis plan, deciding whether a meta-analysis is appropriate for all studies, or if they need to conduct separate analyses for different outcome types (using types of data and effect measures like Mean Difference for continuous data and Risk Ratio or Odds Ratio for dichotomous data) or even subgroup analyses if significant heterogeneity is found, which would be visualized and interpreted using a forest plot and its weighted average.

Quiz

1. Which of the following is considered the MOST important part of any systematic review, serving as a prerequisite for quantitative synthesis? a) Formulating the review question b) Meta-analysis (quantitative synthesis) c) Qualitative synthesis d) Identifying study eligibility criteria

2. True or False: A systematic review must always include a meta-analysis.

3. You are conducting a meta-analysis where some studies report the relative risk (RR) and others report the odds ratio (OR) for a dichotomous outcome. How should you typically handle this in your meta-analysis? a) Convert all ORs to RRs before pooling. b) Convert all RRs to ORs before pooling. c) Conduct separate meta-analyses for studies reporting RR and studies reporting OR. d) Randomly choose either RR or OR as the primary effect measure and convert all others to match.

4. Briefly explain one key advantage of performing a meta-analysis as discussed in the lecture.

---ANSWERS---

1. c) Qualitative synthesis

   • Explanation: The lecture emphasizes that qualitative synthesis is the “most important part of any systematic review” and a crucial step before moving to quantitative analysis, ensuring studies are comparable.

2. False

   • Explanation: The lecture explicitly states that “Meta-analysis is really an optional component of a systematic review. Not every single system systematic review have to have a meta-analysis,” especially if studies are too heterogeneous.

3. c) Conduct separate meta-analyses for studies reporting RR and studies reporting OR.

   • Explanation: The lecture advises analyzing studies by the measure of association or effect they reported, noting that RR and OR are different measures and should not be directly combined in the same pooled estimate unless specifically justified by advanced methods or very low event rates. The example given in the transcript shows separate meta-analyses for RR, OR, HR, and IRR.

4. One key advantage of performing a meta-analysis is to increase the power and precision of the overall effect estimate.

   • Explanation: By combining data from multiple studies, meta-analysis can detect effects that individual, smaller studies might miss due to insufficient power. It also provides a narrower confidence interval around the pooled estimate, leading to a more precise understanding of the intervention’s true effect.