Bootstrap Methods — Resampling for Inference

Bootstrapping is a resampling method that estimates the sampling distribution of a statistic by sampling with replacement from the observed data. It provides standard errors and confidence intervals without distributional assumptions.

Bootstrap Principle

Algorithm

| Step | Action |

|------|--------|

| 1 | Draw a bootstrap sample $X^*_1, \ldots, X^*_n$ by sampling with replacement from the original data |

| 2 | Compute the statistic $\hat{\theta}^*$ from the bootstrap sample |

| 3 | Repeat steps 1-2 B times (typically B = 1,000 - 10,000) |

| 4 | Use the distribution of $\hat{\theta}^*_1, \ldots, \hat{\theta}^*_B$ for inference |

Bootstrap Standard Error

Bootstrap Confidence Intervals

Percentile Method

BCa (Bias-Corrected and Accelerated)

Types of Bootstrap

| Type | Resampling Unit | When to Use |

|------|----------------|-------------|

| Nonparametric | Individual observations | Default; no distributional assumptions |

| Parametric | From fitted distribution | When distribution is known |

| Block | Blocks of observations | Time series data |

| Wild | Residuals with sign changes | Heteroscedastic data |

Bootstrap Hypothesis Testing

Subsampling

A related method that samples without replacement with a smaller sample size $m < n$.

Python Implementation

import numpy as np

import matplotlib.pyplot as plt



np.random.seed(42)



# Original data

n = 200

true_mean = 5.0

true_std = 2.0

data = np.random.normal(true_mean, true_std, n)



# Observed statistic

obs_mean = np.mean(data)

print(f"Observed mean: {obs_mean:.3f}")



# Bootstrap

B = 5000

boot_means = np.zeros(B)

for b in range(B):

    sample = np.random.choice(data, size=n, replace=True)

    boot_means[b] = np.mean(sample)



# Bootstrap SE

boot_se = np.std(boot_means, ddof=1)

print(f"Bootstrap SE: {boot_se:.3f}")

print(f"Analytical SE: {true_std/np.sqrt(n):.3f}")



# Percentile CI

alpha = 0.05

ci_perc = np.percentile(boot_means, [100*alpha/2, 100*(1-alpha/2)])

print(f"Percentile 95% CI: [{ci_perc[0]:.3f}, {ci_perc[1]:.3f}]")



# BCa CI (simplified)

z0 = np.mean(boot_means < obs_mean)

z_alpha = 1.96

ci_bca_lower = np.percentile(boot_means, 100 * np.mean(boot_means < obs_mean - z_alpha * boot_se))

ci_bca_upper = np.percentile(boot_means, 100 * np.mean(boot_means < obs_mean + z_alpha * boot_se))

print(f"BCa CI (approx): [{ci_bca_lower:.3f}, {ci_bca_upper:.3f}]")



# Bootstrap distribution

plt.figure(figsize=(8, 5))

plt.hist(boot_means, bins=50, edgecolor='black', alpha=0.7)

plt.axvline(x=obs_mean, color='red', linestyle='--', label='Observed')

plt.axvline(x=true_mean, color='green', linestyle='--', label='True')

plt.xlabel('Bootstrap Mean')

plt.ylabel('Frequency')

plt.title('Bootstrap Distribution of the Mean')

plt.legend()

plt.show()



# Bootstrap hypothesis test: H0: mean = 4.5

theta0 = 4.5

p_value = np.mean(np.abs(boot_means - obs_mean) >= np.abs(obs_mean - theta0))

print(f"\nBootstrap test (H0: mean=4.5): p={p_value:.4f}")

Worked Example

Key Takeaways

Related Topics

• See Cross-Validation for resampling in model evaluation

• See AIC and BIC for model selection criteria

• See Multiple Imputation for another resampling-based approach