Systematic Review Methodology

"A systematic review attempts to identify, appraise, and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question." — Cochrane Handbook

Systematic Review vs Meta-Analysis

PRISMA Guidelines

PRISMA 2020 Flow Diagram

Identification
├── Records from databases (n = ___)
├── Records from other sources (n = ___)
└── Duplicates removed (n = ___)

Screening
├── Records screened (n = ___)
├── Records excluded (n = ___)
└── Reports sought for retrieval (n = ___)

Eligibility
├── Reports assessed for eligibility (n = ___)
├── Reports excluded (n = ___)
  ├── Reason 1 (n = ___)
  ├── Reason 2 (n = ___)
  └── Reason 3 (n = ___)

Included
├── Studies included in review (n = ___)
└── Studies included in meta-analysis (n = ___)

Search Strategy

PICO Framework

Example Search Strategy

# Database: MEDLINE via PubMed
# Population: Adults with type 2 diabetes
# Intervention: SGLT2 inhibitors
# Comparison: Placebo or other antidiabetics

("diabetes mellitus, type 2"[MeSH] OR "type 2 diabetes"[tiab])
AND
("sodium-glucose transporter 2 inhibitors"[MeSH] OR "SGLT2 inhibitor*"[tiab] 
 OR "empagliflozin"[tiab] OR "dapagliflozin"[tiab] OR "canagliflozin"[tiab])
AND
("cardiovascular diseases"[MeSH] OR "heart failure"[MeSH] OR "MACE"[tiab] 
 OR "cardiovascular outcome*"[tiab])
NOT
("type 1 diabetes"[tiab] OR "gestational"[tiab])

Inclusion and Exclusion Criteria

Common Eligibility Domains

Risk of Bias Assessment

Cochrane Risk of Bias Tool (RoB 2)

The Cochrane RoB 2 tool assesses five domains:

Each domain rated as: Low risk, Some concerns, or High risk.

Data Extraction

Standard Data Fields

GRADE Quality Assessment

GRADE Domains

Upgrade factors:

• ↑ Large effect (RR > 2 or < 0.5)
• ↑ Dose-response gradient
• ↑ All confounders would reduce the effect

Synthesis Without Meta-Analysis (SWiM)

When to Use SWiM

• Studies use different outcome measures or scales
• Studies are too heterogeneous to pool meaningfully
• Few studies (K < 3) are available
• Methodological diversity prevents valid pooling

SWiM Methods

Python Implementation

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# --- PRISMA flow diagram data ---
prisma_data = {
    'Stage': ['Identification', 'Screening', 'Eligibility', 'Included'],
    'Records': [4523, 2891, 312, 45],
    'Excluded': [1632, 2579, 267, 0]
}

print("=== PRISMA Flow ===")
for i, stage in enumerate(prisma_data['Stage']):
    print(f"{stage}: {prisma_data['Records'][i]} records")
    if prisma_data['Excluded'][i] > 0:
        print(f"  Excluded: {prisma_data['Excluded'][i]}")

# --- Risk of Bias Assessment ---
studies = [
    {'Study': 'Smith 2020', 'D1': 'Low', 'D2': 'Low', 'D3': 'Some concerns', 
     'D4': 'Low', 'D5': 'Low', 'Overall': 'Some concerns'},
    {'Study': 'Jones 2021', 'D1': 'Low', 'D2': 'Low', 'D3': 'Low', 
     'D4': 'Low', 'D5': 'Low', 'Overall': 'Low'},
    {'Study': 'Lee 2022', 'D1': 'High', 'D2': 'Some concerns', 'D3': 'Low', 
     'D4': 'Some concerns', 'D5': 'Low', 'Overall': 'High risk'},
    {'Study': 'Chen 2023', 'D1': 'Low', 'D2': 'Low', 'D3': 'Low', 
     'D4': 'Some concerns', 'D5': 'Low', 'Overall': 'Some concerns'},
    {'Study': 'Wang 2024', 'D1': 'Low', 'D2': 'Low', 'D3': 'Low', 
     'D4': 'Low', 'D5': 'Low', 'Overall': 'Low'},
]

df_rob = pd.DataFrame(studies)
print("\n=== Risk of Bias Summary ===")
print(df_rob.to_string(index=False))

# Traffic light plot
rob_matrix = df_rob[['D1', 'D2', 'D3', 'D4', 'D5']].values
color_map = {'Low': '#2ecc71', 'Some concerns': '#f39c12', 'High': '#e74c3c',
             'High risk': '#e74c3c'}

fig, ax = plt.subplots(figsize=(10, 5))
for i in range(len(rob_matrix)):
    for j in range(5):
        color = color_map.get(rob_matrix[i, j], '#95a5a6')
        ax.add_patch(plt.Rectangle((j, len(rob_matrix) - i - 1), 1, 1, 
                     facecolor=color, edgecolor='white', linewidth=2))

ax.set_xlim(0, 5)
ax.set_ylim(0, len(rob_matrix))
ax.set_xticks([0.5, 1.5, 2.5, 3.5, 4.5])
ax.set_xticklabels(['Randomization', 'Deviations', 'Missing Data', 
                     'Measurement', 'Reporting'])
ax.set_yticks([0.5 + i for i in range(len(rob_matrix))])
ax.set_yticklabels(df_rob['Study'].tolist()[::-1])
ax.set_title('Risk of Bias Traffic Light Plot')
plt.tight_layout()
plt.savefig('rob_traffic_light.png', dpi=150)
plt.show()

# --- GRADE Evidence Profile ---
grade_data = {
    'Outcome': ['MACE (3-point)', 'All-cause mortality', 'Heart failure hospitalization'],
    'Studies': [5, 4, 6],
    'Participants': [45000, 38000, 52000],
    'Risk of bias': ['Serious (-1)', 'Not serious', 'Serious (-1)'],
    'Inconsistency': ['Not serious', 'Serious (-1)', 'Not serious'],
    'Indirectness': ['Not serious', 'Not serious', 'Not serious'],
    'Imprecision': ['Not serious', 'Serious (-1)', 'Not serious'],
    'Publication bias': ['Undetected', 'Undetected', 'Undetected'],
    'Starting level': ['High', 'High', 'High'],
    'Final certainty': ['Moderate', 'Low', 'Moderate'],
    'Effect estimate': ['HR 0.86 (0.80-0.93)', 'HR 0.92 (0.84-1.01)', 'HR 0.72 (0.64-0.82)']
}

df_grade = pd.DataFrame(grade_data)
print("\n=== GRADE Evidence Profile ===")
print(df_grade.to_string(index=False))

# --- Study selection funnel ---
selection_data = {
    'Phase': ['Database search', 'Duplicate removal', 'Title/abstract screen',
              'Full-text review', 'Data extraction', 'Quality assessment', 'Final synthesis'],
    'Records': [4523, 2891, 892, 312, 45, 45, 45]
}

fig, ax = plt.subplots(figsize=(10, 6))
phases = selection_data['Phase']
counts = selection_data['Records']
colors = plt.cm.Blues(np.linspace(0.3, 0.9, len(phases)))
ax.barh(phases[::-1], counts[::-1], color=colors[::-1], edgecolor='black')
for i, (count, phase) in enumerate(zip(counts[::-1], phases[::-1])):
    ax.text(count + 50, i, str(count), va='center', fontsize=10)
ax.set_xlabel('Number of Records')
ax.set_title('Study Selection Funnel')
ax.grid(True, alpha=0.3, axis='x')
plt.tight_layout()
plt.savefig('study_selection.png', dpi=150)
plt.show()

# --- Inclusion/exclusion summary ---
print("\n=== Exclusion Reasons (Full-Text) ===")
exclusion_reasons = {
    'Wrong population': 87,
    'Wrong intervention': 62,
    'Wrong outcome': 45,
    'Wrong study design': 38,
    'Duplicate data': 21,
    'Insufficient data': 14
}
for reason, count in sorted(exclusion_reasons.items(), key=lambda x: -x[1]):
    print(f"  {reason}: {count} studies")

Key Takeaways

Next Steps

• -> Meta-Analysis
• -> Replication Crisis
• -> Adaptive Trial Design