Ethics in Statistics

"Statistics is the grammar of science — and like any language, it can be used to illuminate or to deceive." — Adapted from Karl Pearson

The ASA Ethical Guidelines

The Six Principles

1. Professional Integrity and Accountability

   • Strive for honesty, objectivity, and transparency
   • Acknowledge limitations and potential biases in analyses
   • Accept responsibility for professional work

2. Integrity of Data and Methods

   • Use appropriate statistical methods
   • Document data processing and analytic decisions
   • Distinguish between exploratory and confirmatory analysis

3. Responsibilities to Clients, Employers, and Others

   • Protect confidential information
   • Disclose potential conflicts of interest
   • Report results accurately and completely

4. Responsibilities Regarding Allegations of Misconduct

   • Address allegations of misconduct promptly
   • Cooperate with investigations

5. Competence and Judgment

   • Practice only in areas of competence
   • Seek statistical expertise when needed

6. Responsibilities to Other Statisticians

   • Respect colleagues' work
   • Acknowledge contributions appropriately

Responsible Use of Statistics

P-Hacking and Data Dredging

Common forms of p-hacking include:

Replication and Transparency

Remedies for P-Hacking:

1. Pre-registration: Specify hypotheses, methods, and analysis plans before data collection
2. Registered Reports: Journals accept papers based on methodology, before results are known
3. Open Data and Code: Share analysis code and (where ethical) data
4. Bayesian Methods: Shift from binary significance to continuous evidence measures
5. Effect Size Reporting: Report practical significance alongside statistical significance

Algorithmic Fairness

Formal Fairness Criteria

Bias in Algorithms

Fairness-Aware Machine Learning

Informed Consent

Ethical Challenges in Modern Data Science

Data Privacy

Differential Privacy

Regulatory Frameworks

Professional Responsibility

Conflicts of Interest

Landmark Case Studies

Case 1: The Tuskegee Syphilis Study (1932–1972)

Ethical violations: No informed consent, deception, withholding treatment, selection of a vulnerable population.

Impact: Led directly to the National Research Act (1974) and the Belmont Report (1979), establishing the modern framework of informed consent, beneficence, and justice in human subjects research.

Case 2: The Challenger Disaster (1986)

Engineers at Morton Thiokol warned that O-rings could fail at low temperatures. Management overruled them. Statistical analysis of prior launches showed a clear relationship between temperature and O-ring damage — but this analysis was not presented to decision-makers.

Ethical lesson: Statistical evidence must be communicated clearly and forcefully when lives are at stake. The failure was not in the statistics but in the communication of statistical evidence.

Case 3: Algorithmic Bias in Criminal Justice (COMPAS)

Case 4: P-Hacking in Psychosocial Research

Simmons, Nelson, & Simonsohn (2011) demonstrated that common research practices (optional stopping, selective outcome reporting, including/excluding covariates) allow researchers to "find" statistically significant effects with probability up to 61% when the true effect is zero — far exceeding the nominal 5% Type I error rate.

Python Implementation

import numpy as np
from collections import defaultdict

# --- Simulating Algorithmic Bias ---
np.random.seed(42)

def simulate_fairness_audit(n=10000):
    """Audit a classifier for fairness violations across groups."""
    # Generate data with different base rates
    group_a = np.random.binomial(1, 0.3, n)  # Base rate 30%
    group_b = np.random.binomial(1, 0.5, n)  # Base rate 50%

    # Simulate predictions (deliberately biased model)
    # Model has equal TPR but different FPR across groups
    def predict(true_labels, fpr, tpr):
        n = len(true_labels)
        pred = np.zeros(n, dtype=int)
        for i in range(n):
            if true_labels[i] == 1:
                pred[i] = 1 if np.random.random() < tpr else 0
            else:
                pred[i] = 1 if np.random.random() < fpr else 0
        return pred

    pred_a = predict(group_a, fpr=0.15, tpr=0.85)
    pred_b = predict(group_b, fpr=0.30, tpr=0.85)

    # Compute fairness metrics
    def metrics(y_true, y_pred):
        tp = np.sum((y_true == 1) & (y_pred == 1))
        fp = np.sum((y_true == 0) & (y_pred == 1))
        fn = np.sum((y_true == 1) & (y_pred == 0))
        tn = np.sum((y_true == 0) & (y_pred == 0))
        return {'tpr': tp/(tp+fn), 'fpr': fp/(fp+tn),
                'selection_rate': np.mean(y_pred),
                'precision': tp/(tp+fp) if (tp+fp) > 0 else 0}

    m_a = metrics(group_a, pred_a)
    m_b = metrics(group_b, pred_b)

    print("=== Fairness Audit ===")
    print(f"{'Metric':<20s} {'Group A':>10s} {'Group B':>10s} {'Ratio':>10s}")
    print("-" * 55)
    for key in ['selection_rate', 'tpr', 'fpr', 'precision']:
        va, vb = m_a[key], m_b[key]
        ratio = min(va, vb) / max(va, vb) if max(va, vb) > 0 else float('inf')
        print(f"{key:<20s} {va:>10.3f} {vb:>10.3f} {ratio:>10.3f}")

    # Demographic parity violation
    dp_diff = abs(m_a['selection_rate'] - m_b['selection_rate'])
    print(f"\nDemographic parity difference: {dp_diff:.3f}")
    print(f"Equalized odds (TPR difference): {abs(m_a['tpr'] - m_b['tpr']):.3f}")
    print(f"Equalized odds (FPR difference): {abs(m_a['fpr'] - m_b['fpr']):.3f}")

    return m_a, m_b

simulate_fairness_audit()

# --- Differential Privacy Simulation ---
def laplace_mechanism(true_value, sensitivity, epsilon):
    """Add Laplace noise for differential privacy."""
    noise = np.random.laplace(0, sensitivity / epsilon)
    return true_value + noise

def gaussian_mechanism(true_value, sensitivity, epsilon, delta):
    """Add Gaussian noise for (epsilon, delta)-differential privacy."""
    sigma = sensitivity * np.sqrt(2 * np.log(1.25 / delta)) / epsilon
    return true_value + np.random.normal(0, sigma)

print("\n=== Differential Privacy Demonstration ===")
true_count = 5000
sensitivity = 1  # Adding/removing one person changes count by at most 1
true_proportion = 0.42

for eps in [0.1, 0.5, 1.0, 2.0, 5.0, 10.0]:
    estimates = [laplace_mechanism(true_proportion, sensitivity, eps)
                 for _ in range(1000)]
    bias = np.mean(estimates) - true_proportion
    rmse = np.sqrt(np.mean((np.array(estimates) - true_proportion) ** 2))
    print(f"  ε={eps:5.1f}: Mean={np.mean(estimates):.4f}, "
          f"Bias={bias:+.4f}, RMSE={rmse:.4f}")

# --- P-Hacking Simulation ---
print("\n=== P-Hacking Simulation ===")
from scipy import stats

def simulate_phacking(n_experiments=10000, n_samples=50, true_effect=0):
    """Simulate the effect of p-hacking on false positive rate."""
    # Standard analysis
    standard_fps = 0
    for _ in range(n_experiments):
        x = np.random.normal(0, 1, n_samples)
        y = true_effect + np.random.normal(0, 1, n_samples)
        _, p = stats.ttest_ind(x, y)
        if p < 0.05:
            standard_fps += 1

    # P-hacked analysis (try multiple tests, report best)
    hacked_fps = 0
    for _ in range(n_experiments):
        x = np.random.normal(0, 1, n_samples)
        y = true_effect + np.random.normal(0, 1, n_samples)
        # Try 4 analyses: original, log-transformed, with/outlier removed, two-tailed→one-tailed
        tests = [stats.ttest_ind(x, y),
                 stats.ttest_ind(np.log(np.abs(x)+1), np.log(np.abs(y)+1)),
                 stats.ttest_ind(x[1:], y[1:]),
                 stats.ttest_ind(x, y, alternative='less')]
        pvals = [p for _, p in tests]
        if min(pvals) < 0.05:
            hacked_fps += 1

    print(f"  True effect = {true_effect}")
    print(f"  Standard FPR: {standard_fps/n_experiments:.3f} (nominal: 0.050)")
    print(f"  P-hacked FPR: {hacked_fps/n_experiments:.3f}")

simulate_phacking(true_effect=0)
simulate_phacking(true_effect=0.3)

# --- Bayesian Fairness Assessment ---
print("\n=== Bayesian Perspective on Fairness ===")
def bayesian_fairness_prior(n_a, pos_a, n_b, pos_b, prior=1):
    """Compute posterior probability that groups have different true rates."""
    # Beta-Binomial model
    post_a = (prior + pos_a, prior + n_a - pos_a)
    post_b = (prior + pos_b, prior + n_b - pos_b)

    # Monte Carlo comparison
    samples_a = np.random.beta(post_a[0], post_a[1], 100000)
    samples_b = np.random.beta(post_b[0], post_b[1], 100000)

    p_a_greater = np.mean(samples_a > samples_b)
    diff = np.mean(samples_a - samples_b)
    ci = np.percentile(samples_a - samples_b, [2.5, 97.5])

    print(f"  Group A: {pos_a}/{n_a} = {pos_a/n_a:.3f}")
    print(f"  Group B: {pos_b}/{n_b} = {pos_b/n_b:.3f}")
    print(f"  P(A > B): {p_a_greater:.3f}")
    print(f"  Mean difference: {diff:+.4f}")
    print(f"  95% CI for difference: ({ci[0]:.4f}, {ci[1]:.4f})")

bayesian_fairness_prior(1000, 300, 1000, 350)  # 30% vs 35%
bayesian_fairness_prior(100, 30, 100, 50)      # 30% vs 50%

Key Takeaways

Next Steps

• -> Statistics Career Guide
• -> Reporting Standards
• -> Open Science Practices