Two-Proportion Z-Test

Hypothesis Testing

Comparing Rates Across Groups

The two-proportion z-test evaluates whether two groups have different success rates, forming the statistical backbone of A/B testing. It provides a rigorous framework for comparing binary outcomes.

Digital Marketing — Testing whether landing page designs produce different conversion rates
Medicine — Comparing treatment success rates in randomized clinical trials
Social Policy — Evaluating whether interventions reduce rates in different populations

The two-proportion test turns business questions into statistical evidence.

Tests whether two population proportions are equal: H₀: π₁ = π₂.

Two-Proportion Z-Test Statistic

z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}

Here,

$z$ =The z-test statistic
$\hat{p}_1, \hat{p}_2$ =Sample proportions from each group
$\hat{p}$ =Pooled proportion under H₀
$n_1, n_2$ =Sample sizes of each group

where $\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}$ is the pooled proportion.

Application: A/B Testing

Two-Proportion Z-Test for A/B Testing

import numpy as np
from scipy import stats

def two_prop_z_test(x1, n1, x2, n2, alternative='two-sided', alpha=0.05):
    p1, p2 = x1/n1, x2/n2
    p_pool = (x1 + x2) / (n1 + n2)
    
    se = np.sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
    z = (p1 - p2) / se
    
    if alternative == 'two-sided':
        p_val = 2 * stats.norm.sf(abs(z))
    elif alternative == 'greater':
        p_val = stats.norm.sf(z)
    else:
        p_val = stats.norm.cdf(z)
    
    # 95% CI for difference (not pooled SE)
    z_crit = stats.norm.ppf(1 - alpha/2)
    se_ci = np.sqrt(p1*(1-p1)/n1 + p2*(1-p2)/n2)
    ci = (p1-p2 - z_crit*se_ci, p1-p2 + z_crit*se_ci)
    
    print(f"p̂₁ = {x1}/{n1} = {p1:.4f}")
    print(f"p̂₂ = {x2}/{n2} = {p2:.4f}")
    print(f"Difference: {p1-p2:+.4f}")
    print(f"Pooled p̂: {p_pool:.4f}")
    print(f"z = {z:.4f}, p = {p_val:.6f}")
    print(f"95% CI for p₁-p₂: ({ci[0]:.4f}, {ci[1]:.4f})")
    print(f"Decision: {'Reject H₀' if p_val < alpha else 'Fail to reject H₀'}")
    return z, p_val

# A/B Test: New landing page (B) vs original (A)
# Conversions: A: 520/8000, B: 612/8000
print("=== Website A/B Test ===")
print("H₀: Conversion rates are equal")
print("H₁: Conversion rates differ\n")
two_prop_z_test(x1=520, n1=8000, x2=612, n2=8000)

# Clinical trial example
print("\n=== Drug vs Placebo ===")
print("Adverse events: Drug: 45/500, Placebo: 28/500\n")
two_prop_z_test(x1=45, n1=500, x2=28, n2=500, alternative='greater')

Relative Risk and Odds Ratio

Effect Size Measures for Proportions

def relative_risk(x1, n1, x2, n2):
    p1, p2 = x1/n1, x2/n2
    rr = p1 / p2
    # Log-based CI
    log_rr = np.log(rr)
    se_log = np.sqrt(1/x1 - 1/n1 + 1/x2 - 1/n2)
    ci = np.exp(log_rr + np.array([-1.96, 1.96]) * se_log)
    print(f"Relative Risk = {rr:.4f}")
    print(f"95% CI for RR: ({ci[0]:.4f}, {ci[1]:.4f})")
    return rr, ci

def odds_ratio(x1, n1, x2, n2):
    p1, p2 = x1/n1, x2/n2
    odds1, odds2 = p1/(1-p1), p2/(1-p2)
    OR = odds1 / odds2
    log_or = np.log(OR)
    se_log = np.sqrt(1/x1 + 1/(n1-x1) + 1/x2 + 1/(n2-x2))
    ci = np.exp(log_or + np.array([-1.96, 1.96]) * se_log)
    print(f"Odds Ratio = {OR:.4f}")
    print(f"95% CI for OR: ({ci[0]:.4f}, {ci[1]:.4f})")
    return OR, ci

print("=== Effect Size Measures ===")
relative_risk(612, 8000, 520, 8000)
odds_ratio(612, 8000, 520, 8000)

Key Takeaways

Summary: Two-Proportion Z-Test

Pooled proportion under H₀ assumes equal true proportions — use it in the test statistic
Separate proportions are used for the confidence interval (not assuming H₀ is true)
Relative Risk (RR) = p₁/p₂ — multiplicative effect, common in epidemiology
Odds Ratio (OR) = [p₁/(1-p₁)] / [p₂/(1-p₂)] — used in case-control studies, logistic regression
A/B testing in industry uses this test millions of times daily
Sample size matters: with n=8000 per group we can detect small differences (≈0.6% change)

Two-Proportion Z-Test — Comparing Two Proportions