Variance of a Random Variable

What is Variance?

"The variance is the moment of inertia of the probability distribution about its center of mass." — Persi Diaconis

Mathematical Formulation

Properties of Variance

Standard Deviation

The standard deviation $\sigma = \sqrt{\text{Var}(X)}$ restores the original scale of measurement, making it interpretable in the same units as $X$.

Chebyshev's Inequality

This inequality is remarkably general — it requires no assumption about the shape of the distribution. For $k=2$, it says at most $25\%$ of the probability mass lies beyond 2 standard deviations from the mean.

Worked Example: Discrete Random Variable

Worked Example: Continuous Random Variable

Worked Example: Real Data — Exam Scores

Python Implementation

import numpy as np
from scipy import stats

np.random.seed(42)

# Demonstrate variance properties with a Bernoulli(p) random variable
p = 0.6
n = 10000
samples = np.random.binomial(1, p, size=n)

# Empirical variance vs theoretical
empirical_var = np.var(samples, ddof=0)
theoretical_var = p * (1 - p)
print(f"Bernoulli(p={p}): empirical Var = {empirical_var:.4f}, theoretical Var = {theoretical_var:.4f}")

# Show Var(aX + b) = a^2 Var(X)
a, b_const = 3, 5
transformed = a * samples + b_const
print(f"Var({a}X + {b_const}) = {np.var(transformed, ddof=0):.4f}")
print(f"{a}^2 * Var(X)      = {a**2 * empirical_var:.4f}")

# Sum of independent RVs: Var(X+Y) = Var(X) + Var(Y)
samples_y = np.random.binomial(1, 0.3, size=n)
sum_var = np.var(samples + samples_y, ddof=0)
print(f"Var(X+Y) = {sum_var:.4f}")
print(f"Var(X) + Var(Y) = {np.var(samples, ddof=0) + np.var(samples_y, ddof=0):.4f}")

Python Implementation: Chebyshev Verification

import numpy as np

np.random.seed(42)

# Use an exponential distribution (skewed, not normal) to test Chebyshev
lam = 1.0
n = 100000
samples = np.random.exponential(1/lam, size=n)

mu = np.mean(samples)
sigma = np.std(samples)

# Empirical P(|X - mu| >= k*sigma) vs Chebyshev bound 1/k^2
for k in [1.5, 2, 3, 4]:
    empirical = np.mean(np.abs(samples - mu) >= k * sigma)
    bound = 1 / k**2
    print(f"k={k}: empirical P = {empirical:.4f}, Chebyshev bound = {bound:.4f}")

Python Implementation: Real Data Example

import numpy as np

# Exam scores from worked example
scores = np.array([72, 85, 91, 68, 78, 94, 82, 76, 88, 80])

# Population variance (divide by n)
pop_var = np.var(scores)
# Sample variance (divide by n-1)
sample_var = np.var(scores, ddof=1)

print(f"Mean: {np.mean(scores):.1f}")
print(f"Population variance: {pop_var:.2f}")
print(f"Sample variance:     {sample_var:.2f}")
print(f"Standard deviation:  {np.std(scores, ddof=1):.2f}")

# Manual computation for verification
mean = np.mean(scores)
manual_var = np.sum((scores - mean)**2) / (len(scores) - 1)
print(f"\nManual computation: {manual_var:.2f}")

Variance of Common Distributions

Variance in Machine Learning

Key Takeaways

"Variance is the price we pay for uncertainty." — Harry Markowitz