Bootstrap Confidence Intervals — Resampling-Based Inference

Foundations of Statistics

When Theory Meets Reality

Bootstrap methods use resampling to construct confidence intervals without distributional assumptions, making them applicable to virtually any statistic. They provide reliable inference even when traditional formulas break down.

Finance — Constructing confidence intervals for risk measures like VaR
Ecology — Estimating uncertainty for biodiversity indices
Machine Learning — Quantifying variability in model performance metrics

The bootstrap lets the data speak for itself.

What Is Bootstrap Confidence Intervals?

DfBootstrap Confidence Intervals

Bootstrap confidence intervals use resampling — repeatedly drawing samples with replacement from the observed data — to estimate the sampling distribution of a statistic without relying on distributional assumptions.

Core Concept

Bootstrap Resampling

\hat{\theta}^{*b} = T(X^{*b}), \quad b = 1, 2, \ldots, B

Here,

$\hat{\theta}^{*b}$ =Bootstrap estimate from the b-th resample
$T$ =Statistic function applied to the resample
$X^{*b}$ =The b-th bootstrap sample (drawn with replacement)
$B$ =Total number of bootstrap resamples

Theoretical Foundation

ThBootstrap Principle (Efron, 1979)

The empirical distribution $\hat{F}_n$ places mass $1/n$ at each observed data point $x_i$ . The bootstrap approximates the true sampling distribution of $T(F_n)$ by the distribution of $T(\hat{F}_n^*)$ where $\hat{F}_n^*$ is the distribution of a resample from $\hat{F}_n$ . As $n \\to \infty$ , $\hat{F}_n \\to F$ almost surely (by the Glivenko-Cantelli theorem), so the bootstrap distribution converges to the true sampling distribution.

Key insight: We replace the unknown population $F$ with the known empirical $\hat{F}_n$ , then use the computer to simulate what we cannot compute analytically.

Three Bootstrap CI Methods

Percentile Bootstrap CI

[\hat{\theta}^{*(\alpha/2)}, \quad \hat{\theta}^{*(1-\alpha/2)}]

Here,

$\hat{\theta}^{*(\alpha/2)}$ =The $\alpha/2$ quantile of bootstrap estimates
$\hat{\theta}^{*(1-\alpha/2)}$ =The $1-\alpha/2$ quantile of bootstrap estimates

Limitations of the Percentile Method

The percentile interval is not transformation-resymmetric and can have poor coverage when the statistic is biased. It works well only when the sampling distribution is approximately symmetric and the statistic is close to unbiased.

BCA (Bias-Corrected and Accelerated) CI

[\hat{\theta}^{*(\alpha_1)}, \quad \hat{\theta}^{*(\alpha_2)}]

Here,

$\alpha_1$ =$\Phi(\hat{z}_0 + (\hat{z}_0 + z_{\alpha/2})/(1 - \hat{a}(\hat{z}_0 + z_{\alpha/2})))$
$\alpha_2$ =$\Phi(\hat{z}_0 + (\hat{z}_0 + z_{1-\alpha/2})/(1 - \hat{a}(\hat{z}_0 + z_{1-\alpha/2})))$
$\hat{z}_0$ =Bias correction: fraction of bootstrap estimates $< \hat{\theta}$
$\hat{a}$ =Acceleration: measures skewness of the statistic's distribution

When to Use BCA

BCA adjusts for both bias ( $\hat{z}_0 \neq 0$ ) and skewness ( $\hat{a} \neq 0$ ). It is generally preferred over the percentile method and has better finite-sample coverage properties.

Studentized (Pivotal) Bootstrap CI

[\hat{\theta} - t^{*(1-\alpha/2)} \cdot \hat{\text{se}}^*, \quad \hat{\theta} - t^{*(\alpha/2)} \cdot \hat{\text{se}}^*]

Here,

$t^{*(p)}$ =$p$-th quantile of bootstrap $t$-statistics $T^{*b} = (\hat{\theta}^{*b} - \hat{\theta})/\hat{\text{se}}^{*b}$
$\hat{\text{se}}^*$ =Bootstrap standard error estimate

Worked Example: Median Bootstrap CI

Given data $X = \\{2.1, 3.5, 4.2, 1.8, 5.6, 3.9, 2.7, 4.8, 3.2, 6.1, 2.9, 5.0\\}$ ( $n = 12$ ), compute a 95% BCA bootstrap CI for the population median.

Step 1: Compute the sample median: $\hat{\theta} = \text{median}(X) = 3.7$ .

Step 2: Draw $B = 10{,}000$ bootstrap samples, compute the median of each.

Step 3: Apply the BCA correction:

$\hat{z}_0 = \\Phi^{-1}(\text{fraction of } \hat{\theta}^{*b} < 3.7)$
$\hat{a}$ is estimated via the jackknife: $\hat{a} = \frac{\sum_{i=1}^n (\hat{\theta}_{(\cdot)} - \hat{\theta}_{(i)})^3}{6\left[\sum_{i=1}^n (\hat{\theta}_{(\cdot)} - \hat{\theta}_{(i)})^2\right]^{3/2}}$

where $\hat{\theta}_{(i)}$ is the median computed on the sample with the $i$ -th observation deleted.

Step 4: Adjusted quantiles $\alpha_1, \alpha_2$ give the BCA interval.

Why the Median is Hard

The median is a non-smooth statistic — its influence function is discontinuous. The CLT applies (asymptotically normal), but finite-sample distributions are skewed. The bootstrap handles this automatically without requiring the analyst to derive the asymptotic variance.

Python Implementation: Three Bootstrap Methods

import numpy as np
from scipy import stats

np.random.seed(42)
data = np.array([2.1, 3.5, 4.2, 1.8, 5.6, 3.9, 2.7, 4.8, 3.2, 6.1, 2.9, 5.0])
theta_hat = np.median(data)
B = 10000

# Generate bootstrap samples and compute medians
boot_medians = np.array([np.median(np.random.choice(data, size=len(data), replace=True))
                         for _ in range(B)])

# Percentile CI
ci_percentile = np.percentile(boot_medians, [2.5, 97.5])

# BCA CI (using jackknife for acceleration)
n = len(data)
jack_medians = np.array([np.median(np.delete(data, i)) for i in range(n)])
jack_mean = np.mean(jack_medians)
a_hat = np.sum((jack_mean - jack_medians)**3) / (6 * np.sum((jack_mean - jack_medians)**2)**1.5)
z0 = stats.norm.ppf(np.mean(boot_medians < theta_hat))
z_alpha = stats.norm.ppf([0.025, 0.975])
alpha1 = stats.norm.cdf(z0 + (z0 + z_alpha) / (1 - a_hat * (z0 + z_alpha)))
ci_bca = np.percentile(boot_medians, alpha1 * 100)

# Studentized CI
def se_median(x, B_boot=1000):
    boots = np.array([np.median(np.random.choice(x, len(x), replace=True)) for _ in range(B_boot)])
    return np.std(boots)

se_hat = se_median(data)
boot_t = (boot_medians - theta_hat) / se_hat
ci_studentized = theta_hat - np.percentile(boot_t, [97.5, 2.5]) * se_hat

print(f"Sample median: {theta_hat:.3f}")
print(f"Percentile CI: [{ci_percentile[0]:.3f}, {ci_percentile[1]:.3f}]")
print(f"BCA CI:        [{ci_bca[0]:.3f}, {ci_bca[1]:.3f}]")
print(f"Studentized CI:[{ci_studentized[0]:.3f}, {ci_studentized[1]:.3f}]")

Coverage Comparison

ThBootstrap Coverage Accuracy

Under regularity conditions, the studentized bootstrap CI achieves correct asymptotic coverage:

\lim_{n\to\infty} P\left(\hat{\theta} - t^{*(1-\alpha/2)}\hat{\text{se}}^* \leq \theta \leq \hat{\theta} - t^{*(\alpha/2)}\hat{\text{se}}^*\right) = 1 - \alpha

with convergence rate $O(n^{-1})$ — faster than the $O(n^{-1/2})$ rate of the percentile method.

Bootstrap Failures

The bootstrap can fail when:

The statistic is not pivotal and the sample size is small
The data are not exchangeable (e.g., time series with dependence)
The parameter lies on the boundary of the parameter space (e.g., variance components) In these cases, use specialized bootstrap methods (block bootstrap, subsampling).

Key Takeaways

Summary: Bootstrap Confidence Intervals

Bootstrap CIs use resampling to estimate the sampling distribution without distributional assumptions
Percentile: simplest, but only reliable for symmetric, approximately unbiased statistics
BCA: adjusts for bias and skewness; preferred for general use
Studentized: best theoretical properties ( $O(n^{-1})$ coverage error) but requires computing the standard error for each resample
Requires $B \geq 10{,}000$ resamples for stable percentile and BCA intervals
Not a panacea: fails with dependent data, boundary parameters, or very small samples

Bootstrap Confidence Intervals — Resampling-Based Inference

Bootstrap Confidence Intervals — Resampling-Based Inference

When Theory Meets Reality

What Is Bootstrap Confidence Intervals?

DfBootstrap Confidence Intervals

Core Concept

Bootstrap Resampling

Theoretical Foundation

ThBootstrap Principle (Efron, 1979)

Three Bootstrap CI Methods

Percentile Bootstrap CI

BCA (Bias-Corrected and Accelerated) CI

Studentized (Pivotal) Bootstrap CI

Worked Example: Median Bootstrap CI

Python Implementation: Three Bootstrap Methods

Coverage Comparison

ThBootstrap Coverage Accuracy

Key Takeaways

Summary: Bootstrap Confidence Intervals

Premium Content

Need Expert Statistics Help?