Module 6 - Qualitative Synthesis and Interpreting Results

Qualitative Synthesis in Systematic Reviews: Understanding and Interpretation

Qualitative synthesis stands as the most challenging yet critically important component of a systematic review. While quantitative synthesis (meta-analysis) statistically combines numerical results, qualitative synthesis involves integrating findings from various studies without relying on statistical aggregation. This process demands deep understanding and thoughtful interpretation, transforming raw data into meaningful insights for the reader.

This article aims to clarify the role and methodology of qualitative synthesis within systematic reviews, demonstrating its significance in deriving comprehensive conclusions.

Understanding Systematic Reviews and Qualitative Synthesis

Traditionally, the terms “systematic review” and “meta-analysis” were often used interchangeably. However, in contemporary research, a clear distinction exists:

• A systematic review is a broad approach to identify, appraise, and synthesize all relevant evidence on a specific research question. It is the comprehensive and systematic process of gathering information.
• Meta-analysis is a type of quantitative synthesis that may or may not be part of a systematic review. It involves statistically combining numerical results from multiple studies.

Crucially, every meta-analysis requires a systematic review, but a systematic review does not always necessitate a meta-analysis. The qualitative synthesis fills this gap, providing understanding and interpretation beyond mechanistic data aggregation. It transforms factual descriptions into a coherent narrative that adds to the understanding of the phenomenon of interest.

Placement within a Systematic Review: While some systematic reviews (e.g., certain Cochrane reviews) place qualitative synthesis within the discussion section, it is best presented as a dedicated part of the results section. This emphasizes its role in presenting findings, rather than merely contextualizing them.

Why is Qualitative Synthesis Essential?

The necessity for qualitative synthesis arises from the inherent complexities and variabilities within research evidence. It allows reviewers to:

• Address Discordance Among Studies: Individual studies included in a review may differ in their research questions, patient populations, interventions, outcome measurements, or settings. Qualitative synthesis explores these differences and their potential impact on findings.

• Explain Differences Among Systematic Reviews: Even systematic reviews on the same topic might arrive at differing conclusions. This can stem from variations in how authors:

  • Selected studies for inclusion.
  • Extracted data.
  • Assessed study quality or risk of bias.
  • Decided whether studies could be quantitatively combined.

  A qualitative synthesis provides the framework to explain why these discrepancies exist and why a particular interpretation of the evidence is warranted. It goes beyond simply noting factual differences in tables or figures, providing deeper understanding.

Components of a Comprehensive Qualitative Synthesis

A robust qualitative synthesis is an assessment of the body of evidence that moves beyond simple factual descriptions or tables detailing study counts, exclusion reasons, study sizes, treatments, or quality scores. Its purpose is to convey the reviewer’s understanding of how an intervention might work (or not work), whether an association truly exists, for which groups it might be effective, and under what circumstances.

This process involves addressing a number of critical judgments and observations:

1. Assessing the Credibility of Evidence:

   • Uncertainty and Belief: Articulate the degree of certainty in the evidence. Are there specific studies that raise concerns due to incomplete reporting, unclear areas, or other issues?
   • Implications of Missing Evidence: Consider the impact of missing data, such as patients who discontinued treatment or dropped out. How might this affect the results, and what analytical approaches (e.g., sensitivity analysis) were considered to address it?
   • Impact of Technical and Methodological Flaws: Discuss how methodological shortcomings (e.g., lack of allocation concealment, absence of masking) in included studies might have influenced the reported outcomes.
   • Reasonableness of Quantitative Synthesis (Meta-analysis): Even if a meta-analysis is performed, discuss the rationale behind the decision to combine studies. What are the potential downsides of relying on a single summary estimate, especially for outcomes with limited evidence or significant dropout rates?

2. Identifying Patterns and Nuances in the Evidence:

   • Observed Patterns: Describe any discernible patterns in the results. For example, do studies in younger populations show more dramatic effects than those in older populations? Did subgroup analyses reveal a baseline characteristic as particularly important to outcomes?
   • Underlying Science and Interpretation: Explain how observed patterns help interpret findings or understand the underlying scientific mechanisms.
   • Absence of Patterns: If no clear patterns emerge, discuss the implications and any lingering questions about the findings.

3. Orienting the Reader to Study Characteristics:

   • Detailed Study Description: Describe the participants, interventions, comparators, and outcomes across the included studies.
   • Influence of Review Design Choices: Explain how the specific choices made in designing the systematic review (e.g., eligibility criteria) may have influenced the overall findings.
   • Impact of Study Design and Execution Differences: Discuss how variations in the design and execution of individual studies might have affected their results. Examples include:
     • Differences between outpatient and inpatient settings.
     • Whether studies were single-center or part of a multi-center trial with a coordinating center. The reviewer should investigate if results differed based on these characteristics.

Key Aspects to Describe in Qualitative Synthesis

To integrate findings and call attention to specific situations, groups, interventions, outcomes, or study methodologies, a qualitative synthesis should describe:

• Clinical and Methodological Characteristics: This includes study size, and whether specific subgroups (e.g., only older people) were included or excluded, and how this might have influenced findings.
• Strengths and Limitations: Discuss the strengths and limitations of the individual studies, including potential risks of bias or design flaws, and how these might contribute to differing results.
• Relevance and Applicability: Address the applicability of findings to different populations (e.g., how outcomes might differ in poor versus less poor populations). Consider the importance of:
  • Co-interventions: Additional treatments or interventions given alongside the primary one.
  • Setting: The environment in which the research was conducted.
  • Outcome Measures: Whether the outcomes measured were of direct importance to patients (e.g., quality of life) or primarily laboratory measurements. Contemporary reviews often prioritize patient-important outcomes.

Synthesizing and Concluding

Ultimately, a qualitative synthesis involves describing the nature of the evidence, interpreting it, evaluating its strengths and weaknesses, and reaching a conclusion. This conclusion, particularly regarding the appropriateness of combining studies quantitatively, should ideally be formed before any statistical meta-analysis is undertaken. It articulates whether a quantitative synthesis is warranted, and if so, why; or if not, why not.

The Institute of Medicine (IOM) standards for systematic reviews, specifically Standard 4.2 on conducting a qualitative synthesis, provide a helpful guide, offering general principles that can be adapted based on the specifics of each systematic review.

Key Takeaways

• Systematic reviews are integrative syntheses: They require systematically assembling and critically appraising evidence, considering both its qualitative and quantitative aspects.
• Quantitative synthesis is not always warranted: Reasons may include poor study quality or highly disparate results across studies.
• Qualitative synthesis explains disparities: When results are inconsistent (e.g., some studies showing benefit, others none), qualitative synthesis helps explore potential explanations. Conversely, if results are highly similar, it reinforces the findings and may encourage quantitative combination.

Examples of Qualitative Synthesis

While a detailed walkthrough is beyond the scope of this article, reviewing examples from published systematic reviews is highly encouraged to grasp the practical application of qualitative synthesis. Past systematic reviews, such as those conducted by students on topics like “abstinence-plus programs for HIV infection” or “metformin and polycystic ovary syndrome,” illustrate various approaches.

For instance, some examples demonstrate:

• How different systematic reviews on the same topic can include entirely different sets of primary studies, with only minor overlap. The Creanga systematic review, for example, highlighted the diverse study inclusions across similar meta-analyses.
• The impact of methodological choices, such as allocation methods (e.g., in a review on school-based interventions), on study outcomes.
• How factors like publication year (e.g., in a review on latent TB infection) or variations in interventions and outcomes across studies can influence therapy efficacy and reported side effects.

These examples underscore that qualitative synthesis is a nuanced area with no single “right” or “wrong” approach. However, it is an indispensable part of any systematic review. It is where the reviewer’s expertise in the topic shines, providing a crucial interpretive layer that guides the reader’s understanding of the evidence and the robustness of any quantitative findings.

Core Concepts

• Qualitative Synthesis: The process of integrating and interpreting the non-numerical aspects of findings from included studies in a systematic review, focusing on understanding patterns, strengths, and limitations.
• Systematic Review vs. Meta-analysis: A systematic review is a comprehensive process to identify, appraise, and synthesize all relevant evidence on a specific question, while meta-analysis is a specific statistical technique for quantitatively combining results from multiple studies, which may or may not be part of a systematic review.
• Role of Judgment in Systematic Reviews: Systematic reviews heavily rely on expert judgment at various stages, from study selection and data extraction to quality assessment and the interpretation of findings, which must be transparently acknowledged as it can influence the interpretation of results.

Concept Details and Examples

Qualitative Synthesis

Detailed Explanation: Qualitative synthesis involves a deep understanding and interpretation of the evidence base in a systematic review, moving beyond simple factual descriptions or tables. Its purpose is to convey the reviewer’s understanding of how an intervention works, for whom, and under what circumstances, by examining patterns, discrepancies, and contextual factors across studies. It helps to integrate findings, evaluate strengths and weaknesses, and inform the decision of whether a quantitative synthesis (meta-analysis) is warranted.

Examples:

1. Pattern Recognition: A qualitative synthesis might note that studies conducted in younger populations consistently show a more dramatic effect of an intervention compared to studies in older populations, suggesting age as a modifying factor.
2. Addressing Discordance: If some studies show a beneficial effect and others show no effect for the same intervention, the qualitative synthesis would explore potential reasons for this discordance, such as differences in dosage, patient comorbidities, or study methodology (e.g., outpatient vs. inpatient settings).

Common Pitfalls/Misconceptions:

• Conflating with Discussion Section: A common pitfall is treating qualitative synthesis merely as a discussion of findings, rather than a distinct section of the results where observations about the nature of the evidence itself are made before drawing conclusions.
• Over-reliance on Tables: Presenting tables of study characteristics without an accompanying narrative interpretation of what those characteristics mean for the body of evidence is insufficient for a true qualitative synthesis.

Systematic Review vs. Meta-analysis

Detailed Explanation: A systematic review is the overarching scientific method used to comprehensively identify, evaluate, and summarize all relevant evidence on a specific research question. It involves rigorous steps like defining criteria, searching databases, screening studies, extracting data, and assessing bias. Meta-analysis, on the other hand, is a specific statistical procedure that can be part of a systematic review, used to pool numerical results from multiple studies to obtain a single, more precise estimate of an effect.

Examples:

1. Systematic Review without Meta-analysis: A systematic review might investigate the effectiveness of different psychosocial interventions for chronic pain. If the included studies are too heterogeneous in terms of interventions, outcomes, or populations to be meaningfully combined statistically, the review would present a qualitative synthesis of the findings without a meta-analysis.
2. Systematic Review with Meta-analysis: A systematic review examining the efficacy of a specific drug for reducing blood pressure might include a meta-analysis if the included randomized controlled trials are sufficiently similar in design, patient population, intervention, and outcome measures to allow for statistical pooling of results.

Common Pitfalls/Misconceptions:

• Interchangeable Terms: The most significant pitfall is using “systematic review” and “meta-analysis” interchangeably. While often conducted together, a meta-analysis is a type of synthesis that may or may not be part of a systematic review.
• Meta-analysis as a Requirement: Believing that every systematic review must include a meta-analysis. A meta-analysis is only warranted when the studies are sufficiently homogeneous and of adequate quality for quantitative pooling.

Role of Judgment in Systematic Reviews

Detailed Explanation: Expert judgment is intrinsic to every stage of a systematic review, from formulating the research question and defining eligibility criteria to abstracting data, assessing risk of bias, and interpreting the synthesized evidence. These judgments are not arbitrary but are informed by the reviewer’s expertise and understanding of the topic. The transparency and justification of these judgments are crucial, as they can significantly influence the review’s findings and their interpretation.

Examples:

1. Eligibility Criteria Interpretation: During the screening phase, a reviewer might need to make a judgment call on whether a study’s “adolescent” population (defined as 12-18 years) fits the pre-defined eligibility criterion of “young adults” (18-25 years), requiring careful consideration of the research question’s nuances.
2. Risk of Bias Assessment: When assessing the risk of bias, a reviewer might judge a study to have “high risk of bias” due to unclear allocation concealment, even if other aspects are well-reported. This judgment, based on methodological expertise, influences the confidence in that study’s results and, consequently, the overall synthesis.

Common Pitfalls/Misconceptions:

• Bias vs. Judgment: Misconception that “judgment” inherently means “bias.” While judgment can introduce bias if not made transparently and systematically (e.g., using a pre-defined protocol), it is also essential for navigating the complexities and ambiguities inherent in research evidence.
• Mechanical Process: Believing that a systematic review is a purely mechanical process devoid of human interpretation. While systematic and rigorous, it still requires informed expert judgment, especially in the qualitative synthesis phase.

Application Scenario

A research team is conducting a systematic review on the effectiveness of mindfulness-based interventions for reducing anxiety in college students. They have identified 15 relevant studies, but find that these studies vary widely in intervention duration (4-12 weeks), delivery format (in-person vs. online), and anxiety measurement tools used. In this scenario, the lesson’s concepts would be crucial for navigating the variability. The team would need to employ qualitative synthesis to describe patterns, such as whether longer interventions or in-person formats yielded different anxiety reduction outcomes, and to discuss the impact of diverse measurement tools. Their judgment would be vital in deciding whether a meta-analysis is warranted, or if the heterogeneity is too great, making a qualitative synthesis the primary method of reporting findings and interpreting the overall body of evidence.

Quiz

Questions:

1. Multiple Choice: Which of the following is considered the “hardest part” and “most important” part of a systematic review, according to the lecture? a) Formulating the research question b) Conducting the search strategy c) Qualitative synthesis and interpreting results d) Data extraction and management

2. True/False: A meta-analysis can be conducted without first performing a systematic review.

3. Short Answer: Name two distinct types of patterns or factors a reviewer might describe in a qualitative synthesis to help understand why study results differ.

4. Multiple Choice: The purpose of a qualitative synthesis is to: a) Statistically combine numerical data from multiple studies. b) Simply list the characteristics of included studies in tables. c) Convey the reviewer’s understanding of how an intervention works, for whom, and under what circumstances. d) Replace the need for any discussion section in a systematic review.

5. Short Answer: Why is “judgment” considered so important and challenging in systematic reviews?

---ANSWERS---

1. c) Qualitative synthesis and interpreting results

   • Explanation: The lecturer explicitly states, “this is absolutely the hardest part of your systematic review. It’s also the most important.”

2. False

   • Explanation: The lecture clearly states, “A meta-analysis always requires a systematic review but not vice versa.”
3. • Examples: Differences in study population (e.g., younger vs. older people), differences in intervention characteristics (e.g., duration, setting - outpatient vs. inpatient), differences in methodological quality/risk of bias (e.g., allocation concealment, masking), or differences in outcome measures (patient-reported vs. laboratory measurements).
   • Explanation: The qualitative synthesis is where reviewers explore reasons for variability or patterns in the evidence.

4. c) Convey the reviewer’s understanding of how an intervention works, for whom, and under what circumstances.

   • Explanation: The lecture defines qualitative synthesis as going “beyond factual descriptions or tables” to “try to convey your understanding having looked at the data now in it’s entirety about how this intervention might be working or not working.”
5. • Explanation: Judgment is crucial because systematic reviews are not purely mechanical; they involve numerous decisions at every stage (e.g., study inclusion, risk of bias assessment, interpretation of disparate findings). These judgments are informed by expertise and can significantly affect the review’s interpretation, making transparency about them vital.