Module 5 - Minimizing Metabias

This article, adapted from a lecture by Kay Dickersin, provides a comprehensive overview of biases that can occur during the conduct of systematic reviews and meta-analyses, collectively referred to as “metabias.” It emphasizes the importance of established standards to minimize such biases and ensure the reliability of review findings.

---

Understanding and Minimizing Bias in Systematic Reviews: An Introduction to Metabias

Systematic reviews and meta-analyses have become indispensable tools in evidence-based practice across various fields, including medicine and public health. However, the quality and reliability of these reviews hinge on rigorous methodology and a keen awareness of potential biases. This article explores the evolution of standards for systematic reviews, delves into different forms of “metabias” – biases inherent in the review process itself – and discusses strategies for transparent reporting.

The Evolution of Systematic Review Standards

Historically, review articles were often conducted without explicit methodological guidelines, leading to inconsistencies in how literature was synthesized. During the 1980s and 1990s, particularly in medicine and public health, a growing recognition emerged that if primary research studies were held to high methodological standards, the reviews synthesizing them should be held to equally rigorous standards. This realization spurred a movement towards conducting reviews systematically, incorporating distinct method, results, and discussion sections.

Standards have progressively evolved. A pivotal moment occurred in 2011 when the Institute of Medicine (IOM) published comprehensive standards for conducting systematic reviews. This development was crucial, as systematic reviews were proliferating in the literature, often without adhering to any standardized methodology. These IOM standards provide a vital framework for both conducting and assessing systematic reviews.

While biases within individual studies included in a review are a separate concern, this discussion focuses on metabias, defined as bias arising from the process of conducting the systematic review itself. Minimizing metabias is paramount to producing trustworthy evidence syntheses. For further reading on metabias, a relevant article by Goodman, Dickersin, and others was published in Annals of Internal Medicine in 2010.

Three primary forms of metabias are discussed:

• Selection bias
• Information bias
• Bias in the analysis

These forms of bias will be explored in detail in the following sections.

I. Selection Bias in Systematic Reviews

Selection bias in systematic reviews primarily manifests as reporting biases, which encompass various ways in which the availability of study results can be skewed. Another important form is inclusion bias.

A. Reporting Biases: An Overview

1. Publication Bias: The Problem of Unpublished Studies Publication bias occurs when studies with certain findings (e.g., positive or statistically significant results) are more likely to be published than others. This skews the available evidence, making a comprehensive search for all relevant studies, published or unpublished, critical.

   • Evidence of Non-Publication: The likelihood of a study being published varies significantly. For example:

     • Clinical trials approved by a Barcelona Hospital Ethics Committee showed only 21% publication rates by a reasonable follow-up date.
     • Studies from the Johns Hopkins School of Medicine Ethics Committee had an approximately 81% publication rate.
     • Clinical trials funded by the NIH in the late 1970s exhibited a very high publication rate of about 93%. While specific figures vary, a general estimate of 50% non-publication is often cited, highlighting a broad range influenced by factors such as the initiating institution or funding source.

   • Positive Results More Likely to Be Published: A systematic review by Fu Jen Song (2009-2010) extensively examined reporting biases, demonstrating that positive results are consistently more likely to be published:

     • Inception Cohorts: Studies identified through ethics committees or funding agencies generally show that publication favors positive results. Across 15 such studies (from the 1980s, 90s, and 2000s), the summary odds ratio for positive results being published was approximately 3. Only one study (Stern, Australia) initially favored non-positive results, but its confidence interval crossed unity, indicating potential consistency with other findings.
     • Regulatory Cohorts: Studies submitted to regulatory authorities like the FDA also revealed a publication bias favoring positive findings. The odds ratio was about 5, although with wider confidence intervals due to smaller study numbers.

   • Impact of Unpublished Data on Meta-Analyses: The concern is whether including unpublished studies would significantly alter the conclusions of a systematic review. A study examining FDA data, comparing summary statistics with and without unpublished data, found that unpublished data sometimes increased and sometimes decreased the summary statistic (approximately 40% in each direction, with 7% showing no change). This suggests that unpublished data do not consistently push results in one direction, but their impact is undeniable. The IOM standards and other guidelines recommend comprehensive searches for all studies, published or not, due to the potential influence of unpublished results.

   • The Role of ClinicalTrials.gov and FDA Databases: Databases like ClinicalTrials.gov (a registry for clinical trials and observational studies, mandated by law for FDA-regulated drugs, biologics, and devices to post results) are crucial for accessing comprehensive study information. A 2013 study revealed that results posted on ClinicalTrials.gov are often more comprehensive and easier to understand than those published in journals for the same studies. Critically, half of the trials with results posted on ClinicalTrials.gov had no corresponding journal publication. Furthermore, reporting on participant flow, intervention effectiveness, and adverse events was more complete on ClinicalTrials.gov. Therefore, systematic reviewers should consult FDA and ClinicalTrials.gov databases to ensure comprehensiveness and mitigate the bias towards positive published findings.

2. Selective Outcome Reporting: Unreported Endpoints Selective outcome reporting occurs when a study’s results are published, but only a subset of the assessed outcomes are reported, often favoring statistically significant or desired results. This phenomenon has only been extensively studied in the last decade, particularly following Anlin Chan’s 2004 JAMA study.

   • Evidence of Outcome Switching: Chan’s study, which followed protocols submitted to Danish ethics committees, revealed a startling finding: in approximately two-thirds of cases, investigators changed the designated primary outcome between the study protocol and its full publication. This is problematic because the primary outcome is crucial for sample size calculation and for assuring readers that no “data dredging” occurred.
   • Statistically Significant Findings Preferred: The study also found that statistically significant findings were more likely to be reported than non-significant findings, suggesting a link between the chosen primary outcome and its statistical significance. For example, if an intervention shows no benefit for “death from all causes” but a positive effect on “death from a specific cause,” reporting only the latter constitutes selective outcome reporting.
   • Strategies for Identifying Unreported Outcomes: To uncover unreported outcomes and ensure an accurate reflection of initial study intentions, systematic reviewers must delve deeper than journal publications. This involves consulting:
     • FDA databases
     • ClinicalTrials.gov
     • The grey literature

3. The Grey Literature: Hard-to-Find Studies The grey literature encompasses published materials that are not typically found through commercial publishing channels. It includes:

   • Conference abstracts

   • Unpublished data (e.g., from contractors to government institutes, pharmaceutical companies)

   • Book chapters

   • Letters, theses, and dissertations

   • Impact on Systematic Reviews: Only about half of studies initially reported as conference abstracts are ever fully published in journal articles. Critically, studies fully published tend to show positive results more often than those remaining in the grey literature. Sally Hopewell’s systematic review on the grey literature demonstrated that its exclusion could affect the results of a meta-analysis, even if the effect was small in her analysis.

   • Challenges and Debates in Searching Grey Literature: While searching PubMed and other major databases is relatively straightforward, accessing grey literature often requires significant effort (e.g., locating physical conference proceedings, extensive digging). Despite the laboriousness, many researchers are reluctant to omit hand-searching conference abstracts and other grey literature sources due to their potential impact on systematic review and meta-analytic results, though the debate about the cost-benefit remains active.

B. Missing Data and Contacting Authors

Even with comprehensive database and grey literature searches, systematic reviewers may encounter missing data or unreported outcomes within otherwise published studies. In such cases, contacting the original authors is a common strategy.

• Effectiveness of Contact Methods: A systematic review of five studies in the Cochrane database (Young and Hopewell) explored the effectiveness of contacting authors:

  • Email generally yielded a better response rate compared to other methods.
  • Sending repeated emails to the same person using the same methods did not increase the likelihood of a response.
  • Telephoning can be an alternative if email is unsuccessful.
  • Asking for multiple pieces of information (e.g., five questions) in a single communication did not appear to influence the response rate, suggesting authors are willing to provide all requested data if they are already retrieving any. Systematic reviewers are encouraged to correspond with authors via email for missing data or methodological clarifications, ensuring clear and easy-to-answer requests.

• Challenges with Drug Companies: Obtaining unpublished data from drug companies is particularly challenging. While methods exist for specific data requests (e.g., detailing the exact study needed), general inquiries about “unpublished studies on X” are largely ineffective. Personal connections within the industry may facilitate access.

C. Inclusion Bias: Pre-knowledge Influencing Study Selection

Inclusion bias occurs when the systematic review author’s prior knowledge of study results influences the eligibility criteria or data abstraction process. This can lead to the deliberate inclusion or exclusion of studies to favor a particular outcome.

• Definition and Mechanism: If an author is aware of the findings of many available studies, they might subtly (or overtly) set eligibility criteria to ensure that specific studies with desired results are included, while those with undesired results are excluded. This also extends to how data are abstracted once a study is included.
• Example: Mammographic Screening Controversy: The controversy surrounding mammographic screening for breast cancer in women under 50 provides a clear illustration. Given that many of these trials are well-known, systematic review authors may set inclusion criteria with prior knowledge of which randomized trials (or other designs) will be included or excluded, potentially influencing the final meta-analysis results. This constitutes a significant metabias.

II. Information Bias in Systematic Reviews

Information bias in systematic reviews relates to the accuracy and completeness of how data are collected and evaluated from the included studies.

A. Core Concerns: Accuracy and Completeness

Key concerns include:

• The accuracy of quality assessment or risk of bias judgments for individual studies.
• The accuracy and completeness of data abstraction from these studies.

B. The Influence of Pre-Knowledge on Data Abstraction (Revisiting Inclusion Bias)

Beyond study selection, pre-existing knowledge can affect data abstraction. If an abstractor knows a study had positive results, they might unconsciously:

• Assess its quality more favorably.
• Be more likely to look for and abstract specific outcomes. Conversely, if a study had negative results, or if the abstractor knows the authors or the journal where it was published, their assessment of quality or data abstraction might be influenced. For instance, studies have shown that assessors might rate a study as better done if it has positive results or if the author is male.

C. Masking (Blinding) Reviewers to Study Details

To counteract such influences, the practice of masking (or blinding) the systematic review authors during data abstraction was considered. This involved removing information about the individual study’s authors, institutions, journal, and findings.

• Historical Practice vs. Current Evidence: In the past, “differential photocopying” – literally cutting up articles to present only the title and methods sections – was used to mask reviewers. However, studies investigating the impact of such masking (about five studies on the topic) have largely concluded that it makes no significant difference to the data extracted, with only one study from 1996 showing an effect. This is positive news for efficiency, as differential photocopying was time-consuming.
• Implications for Efficiency and Accuracy: The current understanding is that masking reviewers to these elements is generally unnecessary. Instead, the focus should be on having two independent abstractors and then comparing their extracted data to identify and resolve discrepancies.

D. Challenges in Data Abstraction

1. Extracting Data from Graphs: Often, the only way to obtain data for an outcome of interest from a published paper is by visually estimating values from a graph (e.g., proportions from a bar chart), which requires making assumptions about numerators and denominators. This is a challenging and judgment-laden task. While specialized software can assist in this, contacting the original authors for precise data is often the most reliable solution.

2. The Role of Abstractor Experience: A common question is whether the experience level of the abstractor affects the accuracy of data extraction. A 2009 study compared error rates between less experienced and expert abstractors. The findings were reassuring: error rates were similar regardless of experience. However, inexperienced abstractors did take longer to complete the task. While replication of this finding is warranted, it suggests that new systematic reviewers can achieve comparable accuracy to experts.

E. The Necessity of Duplicate Data Abstraction

One of the most robust protections against information bias and human error in data extraction is involving multiple abstractors.

• Methods of Data Extraction: Three main approaches can be used:

  1. Single reader extraction: One person extracts all data onto a form.
  2. Single data extraction with verification: One reader extracts data, and a second reader reviews and verifies it.
  3. Double independent data extraction: Two separate individuals independently extract data without prior consultation. They then compare their extractions and resolve any differences.

• Evidence for Double Extraction: A 2006 study by Buscemi compared single extraction with verification against double independent extraction. It found that double independent data extraction resulted in less inaccuracy and a lower overall error rate. While some express concern about the increased time and cost of double extraction, the IOM Systematic Review Standards suggest that for key results that will be combined in the meta-analysis, double extraction is preferred. For less critical data (e.g., investigators, journal, dates), single extraction with verification might suffice for systematic reviews with limited resources. In practice, many organizations, like the Cochrane Collaboration, consistently employ double data extraction.

• Reliance on Publication: Unfortunately, systematic reviewers cannot solely rely on what is presented in published journal articles due to prevalent reporting biases. Data discrepancies or missing information often necessitate cross-referencing with grey literature sources, FDA databases, and ClinicalTrials.gov. Errors are human, and dual data extraction serves as a critical safeguard against them.

III. Bias in the Analysis of Systematic Reviews

Bias in the analysis refers to potential distortions introduced during the statistical synthesis of data from included studies. This is a complex area, and understanding its nuances is crucial.

A. The Impact of Statistical Model Choice (Fixed vs. Random Effects)

Meta-analyses typically employ two main statistical models: fixed effects and random effects. The choice of model can influence the overall effect estimate, especially in certain circumstances.

• Understanding the Models: While the nuanced differences are beyond the scope of this discussion, the general consensus has evolved. Many researchers initially gravitate towards random effects models, but over time, preferences may shift, recognizing that no single model is universally “right” or “wrong.” The decision often depends on the specific context and characteristics of the included studies.

• When Model Choice Matters: Evidence from Villar Study: A study by Jose Villar, published in Statistics in Medicine, examined 84 Cochrane meta-analyses from the Cochrane Pregnancy and Childbirth Group to compare results obtained using fixed and random effects models, particularly in the presence of statistical heterogeneity (variation in results beyond chance across studies).

  • With Statistical Heterogeneity (21 meta-analyses): When significant statistical heterogeneity was present, discordance between the two models was observed in 5 out of 21 meta-analyses (approximately 25%). This means that one model might yield a statistically significant result (excluding unity), while the other might not. In such cases, the random effects model typically provides a more conservative estimate, yielding wider confidence intervals and being less likely to exclude unity (i.e., less likely to report statistical significance).
  • Without Statistical Heterogeneity (63 meta-analyses): In the absence of significant statistical heterogeneity, discordance between the models was much rarer, occurring in only 4 out of 63 meta-analyses (approximately 6%).
  • Implications: The key takeaway is that when statistical heterogeneity is present in a meta-analysis, employing a random effects model may offer a more conservative and appropriate approach, as it accounts for variation between studies beyond what can be attributed to random chance. When heterogeneity is low, the choice of model generally has minimal impact on the overall estimate.

B. Transparency and Assessment of Metabias in Published Reviews

Despite the importance of minimizing metabias, systematic reviews in the published literature often fall short in transparently reporting their methods or assessing potential biases within their own process.

• Evidence of Poor Reporting Quality: Studies assessing the quality of systematic reviews, for example, in urology and pediatric oncology, reveal significant gaps in adherence to methodological standards:
  • In a review of 57 urology systematic reviews, only about 40% performed duplicate screening and data extraction, and roughly 50% conducted a comprehensive literature search.
  • Fewer than 30% explicitly published their eligibility criteria.
  • While about 60% reported their analysis methods, only approximately 18% assessed the likelihood of publication bias. This indicates that the overall quality of published systematic reviews is often suboptimal, leading to a high probability of metabias. This highlights the ongoing need for improved education for systematic review authors, as well as for journal editors and peer reviewers.

C. Discrepancies Among Systematic Reviews on the Same Topic

A common concern is when multiple systematic reviews addressing the same topic arrive at different conclusions. While this can sometimes reflect legitimate differences in the scientific question asked or transparent methodological choices, it can also stem from unacknowledged biases.

• Legitimate Differences vs. Methodological Biases: Often, discrepancies arise from slightly different research questions, search strategies, or inclusion/exclusion criteria. If reviews are transparently reported, readers can understand these differences and assess which review is most relevant to their specific question. However, sometimes differences point to underlying methodological biases.

• Case Study: H. pylori Eradication: An intriguing example is the comparison of two systematic reviews on the efficacy of H. pylori eradication in non-ulcer dyspepsia: one by the Cochrane group and another by the AHRQ Evidence-based Practice Center (EPC).

  • Cochrane Review: Included many more studies and concluded that H. pylori eradication was favored (statistically significant).
  • AHRQ EPC Review: Included fewer studies and found no statistically significant evidence favoring H. pylori eradication.
  • Key Factors Influencing Discrepancies: This significant difference was attributable to several factors:
    • Search Dates: The Cochrane review’s search terminated six months later (May 2000 vs. December 1999 for EPC), leading to the inclusion of three additional trials/abstracts.
    • Included/Excluded Studies: Differences in which studies were ultimately selected.
    • Endpoint Handling: In cases where an abstract reported an endpoint that did not exactly match the reviewers’ interest, the Cochrane authors contacted the original study authors to obtain unpublished data for the precise outcome needed. In contrast, the EPC reviewers used the reported outcome if it was “close enough.” This subtle difference in data acquisition significantly impacted the overall findings.

• Conclusion: Reviews can legitimately differ due to variations in scientific method. However, methodological choices, such as the recency of literature searches and diligence in obtaining precise outcome data, can profoundly influence findings. Transparency in reporting these choices is critical for readers to interpret and compare reviews effectively.

IV. Transparent Reporting of Systematic Reviews

Transparent reporting is essential for readers to understand how a systematic review was conducted, assess its quality, and interpret its findings. Specific reporting guidelines have been developed to promote this transparency.

A. Essential Reporting Guidelines

• PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses): This is the leading reporting guideline for systematic reviews of clinical trials. It provides a checklist of essential items that should be included in a systematic review report.
• MOOSE (Meta-analysis of Observational Studies in Epidemiology): While somewhat outdated (published in 2000) but still useful, MOOSE provides reporting guidance for systematic reviews and meta-analyses of observational studies. An updated version of MOOSE is currently under development.

It is worth noting that in older literature, the term “meta-analysis” was sometimes used broadly to refer to systematic reviews, even those without a quantitative synthesis.

B. The PRISMA Flowchart: Visualizing the Review Process

A crucial component of transparent reporting, particularly emphasized by PRISMA, is the flowchart. This diagram visually illustrates the flow of information regarding the identification, screening, eligibility, and inclusion of studies in a systematic review. It maps:

• The total number of records identified through database searching and other sources.
• Records removed due to ineligibility or duplication.
• The final number of studies included for qualitative synthesis (systematic review) and quantitative synthesis (meta-analysis). This flowchart provides a clear, concise overview of the study selection process, similar to participant flow diagrams in clinical trials.

C. GRADE: Grading the Body of Evidence

GRADE (Grading of Recommendations Assessment, Development and Evaluation) is a system for summarizing the overall quality of a body of evidence. It is important to distinguish GRADE from the systematic review or meta-analysis itself; GRADE is a step that comes after the review.

• Purpose: GRADE addresses the question: “How good is the evidence?” It helps users determine the confidence they can place in the findings and whether more research is needed. It moves beyond simply presenting the review’s results to evaluate the strength and reliability of the evidence.
• Factors Influencing Grading: The quality of evidence is graded down for factors such as:
  • Risk of bias in the included studies.
  • Inconsistency or indirectness of results.
  • Imprecision of effect estimates.
  • Publication bias. The quality of evidence can be graded up for factors like:
  • A large effect size.
  • Evidence of a dose-response effect.
  • Plausible confounding that would reduce a spurious effect. GRADE is typically conducted as an intermediate step between a systematic review and the development of a clinical practice guideline, and thus, it may not always be an integral part of the systematic review publication itself.

Conclusion

Conducting a robust systematic review and meta-analysis requires careful attention to potential biases at every stage of the process. Understanding metabias – bias in the systematic review’s methodology itself – is crucial. The three key areas of metabias include:

• Selection bias, largely manifesting as reporting biases and inclusion bias.
• Information bias, related to the accuracy and completeness of data abstraction.
• Bias in the analysis, influenced by methodological choices such as the statistical model used.

To minimize metabias, systematic reviews should adhere to established standards, including:

• Conducting thorough and comprehensive searches for all relevant evidence (published and unpublished).
• Implementing rigorous procedures for data abstraction, ideally with duplicate independent extraction.
• Carefully considering the impact of methodological choices during data synthesis.

While systematic review methodology continues to evolve, these principles offer practical guidance. Authors conducting systematic reviews should strive to apply these methods to the best of their ability within their resources and, importantly, transparently discuss any limitations or deviations from recommended standards in their reports. This commitment to transparency and methodological rigor is vital for enhancing the trustworthiness and utility of systematic reviews in informing decision-making.

Core Concepts

• Metabias: Bias inherent in the process of conducting a systematic review itself, distinct from biases within the individual studies being reviewed.
• Selection Bias (in systematic reviews): A type of metabias where studies are included or excluded, or their results prioritized, in a way that introduces systematic error into the systematic review’s findings.
• Reporting Bias: A specific type of selection bias where the decision to publish or report research findings, or specific outcomes within studies, is influenced by the nature and direction of the results.
• Publication Bias: A reporting bias where studies with positive or statistically significant results are more likely to be published than those with negative or non-significant results.
• Selective Outcome Reporting: A reporting bias where certain outcomes from a study are preferentially reported (or omitted) in publications based on their statistical significance or direction.
• Inclusion Bias: A type of selection bias where the systematic review author’s prior knowledge of study results or investigators influences the eligibility criteria or data abstraction process.
• Information Bias (in systematic reviews): A type of metabias related to the accuracy and completeness of data extraction and quality assessment from the included studies.
• Bias in Analysis (in systematic reviews): A type of metabias related to the inappropriate choice or application of statistical models or analytical methods during the meta-analysis.
• Reporting Guidelines (PRISMA, MOOSE): Standardized checklists and flowcharts designed to ensure transparent and complete reporting of systematic reviews and meta-analyses.
• GRADE (Grading of Recommendations Assessment, Development and Evaluation): A structured approach to assess the quality of a body of evidence and the strength of recommendations, distinct from the systematic review itself.

Concept Details and Examples

Metabias

Metabias refers to systematic errors that arise from the way a systematic review is conducted, rather than from flaws in the primary studies it reviews. It highlights the importance of methodological rigor in synthesizing evidence, as even a perfectly executed primary study can be misrepresented if the review process is biased. Minimizing metabias ensures the systematic review’s conclusions are as objective and reliable as possible.

• Example 1: A systematic reviewer decides to only include studies published in English, inadvertently excluding a significant body of non-English literature that might present different findings, leading to an English-language bias (a form of selection bias within metabias).
• Example 2: A reviewer consistently rates the methodological quality of studies sponsored by a particular pharmaceutical company lower, regardless of the actual study design, due to a pre-conceived notion, introducing information bias into the review’s quality assessment.
• Common Pitfall: Confusing metabias with the risk of bias within the individual primary studies. Metabias pertains to the systematic review’s own methodology.

Selection Bias (in systematic reviews)

Selection bias in systematic reviews occurs when the choice of studies to include or exclude, or how they are found, systematically favors certain results. This can lead to a skewed representation of the evidence, where the systematic review’s overall findings do not accurately reflect the totality of available research. It underscores the necessity of a comprehensive and unbiased search strategy and transparent eligibility criteria.

• Example 1: A systematic review on a medical intervention exclusively searches PubMed, missing trials published in clinical trial registries or in journals not indexed by PubMed, which might disproportionately contain negative or non-significant results.
• Example 2: A reviewer designs eligibility criteria for a systematic review on breast cancer screening that specifically excludes trials known to have null results, ensuring that only studies showing a positive effect are included in the meta-analysis.
• Common Pitfall: Believing that merely identifying all relevant studies guarantees freedom from selection bias; the criteria for inclusion and how those identified studies are selected also matters.

Reporting Bias

Reporting bias is a type of selection bias where the likelihood of research findings being published or reported is influenced by the nature or direction of those findings. This means that a systematic review might only have access to a biased subset of all conducted studies or all measured outcomes, leading to an incomplete or distorted picture of the evidence base.

• Example 1: Studies with statistically significant positive results on a new drug are much more likely to appear in peer-reviewed journals than studies showing no effect or negative effects, making it harder for a systematic review to capture the full range of evidence.
• Example 2: A clinical trial measures ten different patient outcomes, but only the two outcomes that showed statistically significant improvements are described in detail in the final publication, while the other eight (non-significant) are merely mentioned or omitted entirely.
• Common Pitfall: Assuming that all completed research studies, or all outcomes measured within a study, are equally likely to be fully reported and accessible through standard literature searches.

Publication Bias

Publication bias specifically refers to the phenomenon where studies with positive or statistically significant results are more likely to be published than those with negative or non-significant results. This creates a skewed literature base, making it challenging for systematic reviewers to find and synthesize all relevant evidence, potentially leading to an overestimation of intervention effects.

• Example 1: An pharmaceutical company conducts five trials for a new medication; the two trials showing a positive effect are published in prominent journals, while the three trials showing no significant benefit are never submitted for publication, making the overall evidence appear more favorable than it is.
• Example 2: A systematic review of acupuncture for pain relief primarily identifies published studies with positive outcomes, even though many pilot studies and smaller trials with null findings were conducted but never published, leading to an artificially strong summary effect.
• Common Pitfall: Relying solely on conventional bibliographic databases (like PubMed) for study identification, as these primarily index published literature and are thus susceptible to publication bias.

Selective Outcome Reporting

Selective outcome reporting occurs when, within a single study, certain measured outcomes are preferentially reported in the publication based on their statistical significance or direction, while other measured outcomes are omitted or downplayed. This misrepresents the study’s true findings and can lead to biased conclusions within a systematic review, especially if the primary outcome was switched post-hoc.

• Example 1: A drug trial’s protocol specifies ‘reduction in hospitalizations’ as the primary outcome, but when this outcome isn’t statistically significant, the researchers instead emphasize a statistically significant reduction in ‘symptom severity’ in their publication, promoting it as the main finding.
• Example 2: A study investigating the side effects of a vaccine collects data on 20 different adverse events; only the two mildest and least frequent adverse events are detailed in the published paper, while more severe but less common adverse events are not mentioned.
• Common Pitfall: Not consulting study protocols or clinical trial registries (like clinicaltrials.gov) to ascertain all pre-specified outcomes, thus relying solely on the published paper which might have selectively reported results.

Inclusion Bias

Inclusion bias arises when systematic review authors, possessing prior knowledge of individual study results or characteristics (like investigator identity), allow this knowledge to influence their decisions on which studies to include or exclude. This can subtly or overtly manipulate the systematic review’s composition to align with pre-existing beliefs or desired outcomes, compromising objectivity.

• Example 1: A systematic reviewer, knowing that a particular prominent research group consistently produces positive results for a specific intervention, might implicitly lower their strictness when applying eligibility criteria to studies from that group, ensuring their inclusion.
• Example 2: For a controversial topic like dietary supplements, a review author might deliberately set very narrow inclusion criteria (e.g., only specific types of randomized controlled trials with very high quality scores) that happen to exclude several well-known studies with negative findings, shaping the overall conclusion.
• Common Pitfall: Believing that explicit, pre-defined eligibility criteria automatically prevent inclusion bias; the application of these criteria, especially when reviewers are not blinded, can still be influenced.

Information Bias (in systematic reviews)

Information bias in systematic reviews concerns the accuracy and completeness of data abstracted from included studies, and the reliability of quality or risk of bias assessments. Errors or inconsistencies in these steps can distort the synthesis of evidence, regardless of the primary studies’ actual quality or results. It emphasizes the need for rigorous and standardized data collection processes within the review.

• Example 1: A single reviewer abstracts all data from primary studies. In one study, they misread a graph, leading to an incorrect numerical value for an outcome, which then gets incorporated into the meta-analysis.
• Example 2: When assessing the risk of bias for a primary study, a reviewer, knowing the study’s positive results, subconsciously rates its methodological rigor higher than an identical study with negative results, introducing bias into the quality assessment.
• Common Pitfall: Assuming that once a study is included, its data will be extracted perfectly. Human error and subjective interpretation during data abstraction are significant risks.

Bias in Analysis (in systematic reviews)

Bias in analysis within systematic reviews refers to systematic errors introduced during the statistical synthesis of data, often due to an inappropriate choice of statistical model or analytical methods. Such choices can significantly alter the meta-analysis’s summary estimate or its statistical significance, potentially leading to misleading conclusions about the overall effect of an intervention or exposure.

• Example 1: A meta-analysis includes studies with high statistical heterogeneity (widely differing results), but the reviewers choose a fixed-effects model, which assumes a single true effect size, rather than a more conservative random-effects model that accounts for variation between studies. This might lead to a statistically significant pooled estimate that isn’t robust.
• Example 2: Reviewers perform multiple subgroup analyses without pre-specifying them, and then only report the one subgroup that yields a statistically significant result, a form of ‘data dredging’ at the meta-analysis level.
• Common Pitfall: Over-reliance on default statistical software settings without understanding the assumptions behind different meta-analysis models (e.g., fixed vs. random effects) and their appropriateness for the data’s heterogeneity.

Reporting Guidelines (PRISMA, MOOSE)

Reporting guidelines like PRISMA and MOOSE provide structured frameworks (checklists and flowcharts) to ensure that systematic reviews and meta-analyses are reported transparently and comprehensively. They detail essential elements that should be included in a review’s manuscript, enabling readers to critically appraise the methodology, results, and conclusions, and aiding reproducibility.

• Example 1 (PRISMA): A systematic review uses the PRISMA flowchart to visually document the number of records identified, screened, assessed for eligibility, and ultimately included in the review, clearly showing reasons for exclusion at each stage.
• Example 2 (MOOSE): A systematic review of observational studies on a specific exposure and outcome explicitly reports the methods used to identify and handle confounding factors in primary studies, following MOOSE guidelines for transparency in reporting observational evidence synthesis.
• Common Pitfall: Confusing reporting guidelines (what to report) with methodological standards (how to conduct a systematic review). While related, they serve different purposes; simply following reporting guidelines doesn’t guarantee a well-conducted review.

GRADE (Grading of Recommendations Assessment, Development and Evaluation)

GRADE is a systematic and transparent methodology for assessing the quality of a body of evidence and for developing health recommendations. It provides a structured approach to move from evidence to recommendations, considering factors like risk of bias, inconsistency, indirectness, imprecision, and publication bias to rate the overall certainty of evidence as high, moderate, low, or very low.

• Example 1: After conducting a systematic review on the effectiveness of a new cancer treatment, a guideline panel uses GRADE to rate the quality of evidence for survival benefit as ‘moderate’ due to a high risk of bias in some included studies and imprecision in the pooled estimate.
• Example 2: A public health organization uses the GRADE framework to evaluate the evidence for a new vaccination program, leading them to issue a ‘strong recommendation’ based on high-quality evidence of effectiveness and safety, coupled with high values and preferences for the intervention.
• Common Pitfall: Misusing ‘body of evidence’ to refer simply to a meta-analysis or systematic review. GRADE specifically applies to the entire collected evidence base for a particular outcome, considering its strengths and weaknesses, to inform recommendations.

Application Scenario

A research team is planning a systematic review and meta-analysis on the effectiveness of digital health interventions for managing chronic back pain. They aim to provide a definitive answer for clinical practice. The key concepts of metabias would be highly relevant here to ensure the review’s integrity. They would need to address selection biases by performing a comprehensive search beyond just common databases and checking for selective outcome reporting by comparing published results with trial registries. Information bias would be managed by having two independent reviewers extract data and assess study quality, then resolving discrepancies. Finally, they would consider bias in analysis by carefully choosing the meta-analysis model based on observed heterogeneity, and transparently report all methods using PRISMA guidelines to allow for critical appraisal.

Quiz

Questions:

1. Multiple Choice: Which of the following is an example of metabias as defined in the lecture? a) A primary study failing to blind participants to treatment assignment. b) A systematic review author knowing the results of individual studies before setting inclusion criteria. c) A randomized controlled trial having a small sample size, leading to imprecise results. d) A laboratory experiment’s results being influenced by contamination of samples.

2. True/False: According to the lecture, the choice between a fixed-effects and random-effects model for meta-analysis rarely influences the overall estimate, regardless of statistical heterogeneity.

3. Short Answer: Why is it important for systematic reviewers to consult clinical trial registries (like clinicaltrials.gov) in addition to published journal articles?

4. Multiple Choice: Which reporting guideline is specifically recommended for systematic reviews of clinical trials? a) STROBE b) CONSORT c) PRISMA d) MOOSE

5. Short Answer: Briefly explain the difference between ‘publication bias’ and ‘selective outcome reporting’.

---

ANSWERS:

1. b) A systematic review author knowing the results of individual studies before setting inclusion criteria.

   • Explanation: Metabias refers to bias in the systematic review process itself. Options a, c, and d describe biases or limitations within primary research studies, not the review process.

2. False.

   • Explanation: The lecture explicitly states that when there is significant statistical heterogeneity among studies, the choice between fixed-effects and random-effects models can indeed influence the overall estimate and confidence intervals, with random-effects often being more conservative.

3. Explanation: Consulting clinical trial registries is important to identify unpublished studies, find studies with non-positive results that might not be published, and check for selective outcome reporting. Registries often contain more comprehensive data about study protocols and outcomes than journal publications, helping to minimize reporting biases and ensure a more complete and accurate systematic review.

4. c) PRISMA.

   • Explanation: PRISMA is the recommended reporting guideline for systematic reviews and meta-analyses of clinical trials. STROBE is for observational studies, CONSORT is for primary clinical trials, and MOOSE (while for observational systematic reviews) is considered somewhat outdated compared to PRISMA.

5. Explanation: Publication bias refers to the tendency for entire studies (especially those with positive or significant results) to be published, while other completed studies (often with negative or non-significant results) remain unpublished. Selective outcome reporting occurs within a single published study, where certain outcomes that were measured are preferentially reported (or omitted) based on their statistical significance or direction, even if the study itself is published.