Taylor Series and Approximations

Why It Matters

What is a Taylor Series

Maclaurin Series

A Maclaurin series is simply a Taylor series centered at $a = 0$. It is the most commonly used form because derivatives at zero are often easier to compute.

Common Taylor Expansions

Memorizing these standard expansions is essential — they appear constantly in calculus, differential equations, and numerical methods.

Convergence

Not every Taylor series converges to its original function. Understanding convergence is critical.

Taylor's Theorem with Remainder

Approximation Applications

Taylor series are used to compute function values efficiently in practice.

Polynomial Approximation

Python Implementation

Basic Taylor Approximation

import numpy as np

def taylor_exp(x, n_terms=10):
    """Approximate e^x using the Maclaurin series."""
    result = 0
    for k in range(n_terms):
        result += x**k / np.math.factorial(k)
    return result

# Compare with numpy
x = 1.0
print(f"Exact:  {np.exp(x):.12f}")
print(f"Taylor: {taylor_exp(x, n_terms=10):.12f}")

# Output:
# Exact:  2.718281828459
# Taylor: 2.718281801146

Vectorized Taylor Approximation

import numpy as np

def taylor_sin(x, n_terms=8):
    """Vectorized Maclaurin series for sin(x)."""
    result = np.zeros_like(x, dtype=float)
    for k in range(n_terms):
        sign = (-1)**k
        power = 2*k + 1
        result += sign * x**power / np.math.factorial(power)
    return result

x = np.array([0.1, 0.5, 1.0, 2.0])
print(f"Exact:    {np.sin(x)}")
print(f"Taylor:   {taylor_sin(x)}")
print(f"Error:    {np.abs(np.sin(x) - taylor_sin(x))}")

SymPy Symbolic Taylor Series

from sympy import symbols, series, sin, cos, exp, log, oo

x = symbols('x')

# Taylor series expansions
print("e^x:", series(exp(x), x, 0, n=6))
print("sin(x):", series(sin(x), x, 0, n=8))
print("cos(x):", series(cos(x), x, 0, n=8))
print("ln(1+x):", series(log(1+x), x, 0, n=6))

# Compute specific derivatives
from sympy import factorial
def taylor_coeff(f, x, a, n):
    """Get the nth Taylor coefficient at point a."""
    return f.diff(x, n).subs(x, a) / factorial(n)

print(f" coefficient of x^3 in e^x: {taylor_coeff(exp(x), x, 0, 3)}")

Error Analysis

import numpy as np
import matplotlib.pyplot as plt

def taylor_error_analysis():
    """Compare Taylor approximations of different orders."""
    x = np.linspace(-3, 3, 200)
    exact = np.exp(x)
    
    errors = {}
    for n in [3, 5, 7, 10]:
        approx = np.zeros_like(x)
        for k in range(n + 1):
            approx += x**k / np.math.factorial(k)
        errors[n] = np.abs(exact - approx)
    
    # Plot
    plt.figure(figsize=(10, 6))
    for n, err in errors.items():
        plt.semilogy(x, err, label=f'n={n}')
    plt.xlabel('x')
    plt.ylabel('Absolute Error')
    plt.title('Taylor Series Error for e^x')
    plt.legend()
    plt.grid(True)
    plt.show()

taylor_error_analysis()

Applications in AI/ML

Linearization of Neural Networks

Gradient Descent Analysis

Taylor series explain why gradient descent works:

• The gradient $\nabla f$ is the first-order Taylor approximation
• Newton's method uses the second-order Taylor expansion for faster convergence
• Momentum methods can be viewed as modifying the Taylor approximation landscape

Taylor Series in Loss Functions

Many loss functions are analyzed using Taylor expansions:

• Cross-entropy loss near zero predictions (gradient explosion)
• Huber loss as a smooth approximation of MAE
• Softmax stability analysis (subtracting the max is equivalent to scaling the Taylor series)

Automatic Differentiation Connection

Automatic differentiation (AD) computes exact derivatives — the same derivatives needed for Taylor series. Modern ML frameworks like PyTorch and JAX use AD to:

• Compute gradients for backpropagation
• Enable higher-order derivatives for Taylor-based optimization
• Implement the Taylor-mode AD for efficient computation of Taylor coefficients

Common Mistakes

Interview Questions

Question 1: Why does $e^x$ have the simplest Maclaurin series?

Answer: Because all derivatives of $e^x$ are $e^x$ itself, and $e^0 = 1$. So $f^{(n)}(0) = 1$ for all $n$, giving the clean series $\sum \frac{x^n}{n!}$. No alternating signs, no zeros — every term contributes equally.

Question 2: How would you approximate $\sqrt{1.1}$ without a calculator?

Answer: Use the binomial series $(1+x)^\alpha$ with $x = 0.1$ and $\alpha = 1/2$:

The exact value is $\approx 1.0488088...$, so 3 terms give 5 digits of accuracy.

Question 3: Why is the Taylor series for $\ln(1+x)$ restricted to $|x| < 1$?

Answer: The coefficients $c_n = \frac{(-1)^{n+1}}{n}$ satisfy $\lim_{n\to\infty} |c_n/c_{n+1}| = 1$, giving $R = 1$. At $x = 1$, the series $1 - 1/2 + 1/3 - \cdots$ converges (alternating harmonic series), but at $x = -1$ it becomes $-1 - 1/2 - 1/3 - \cdots$, which diverges (harmonic series).

Question 4: What's the relationship between Taylor series and Fourier series?

Answer: Both decompose functions into simpler basis functions:

• Taylor series use polynomial basis $\{1, x, x^2, \ldots\}$ — best for local approximation
• Fourier series use trigonometric basis $\{1, \cos(nx), \sin(nx)\}$ — best for periodic functions

Taylor series require differentiability; Fourier series work for discontinuous functions. In higher dimensions, Taylor series extend to multivariate polynomials, while Fourier series extend to the Fourier transform.

Question 5: How do Taylor series relate to the concept of analyticity?

Answer: A function is analytic at a point if its Taylor series converges to the function in some neighborhood. All polynomial, exponential, trigonometric, and rational functions (away from poles) are analytic. However, $f(x) = e^{-1/x^2}$ (with $f(0)=0$) is smooth but not analytic — its Taylor series at 0 is identically zero, yet the function is nonzero for $x \neq 0$.

Question 6: Can you use Taylor series to solve differential equations?

Answer: Yes — the power series method assumes $y(x) = \sum_{n=0}^{\infty} a_n x^n$, substitutes into the ODE, and solves for the coefficients recursively. For example, solving $y' = y$ with $y(0) = 1$ gives $a_n = 1/n!$, recovering $e^x$. This method works for linear ODEs with polynomial coefficients and is the basis for special functions like Bessel functions and Legendre polynomials.

Practice Problems

Problem 1: Derive the Series

Problem 2: Error Estimation

Problem 3: Non-Standard Expansion

Problem 4: Real-World Application

Problem 5: Series Manipulation

Quick Reference

Cross-References

• Sequences and Series — foundations for understanding convergence
• Power Series — the general framework for Taylor series
• Maclaurin Series — the special case at $a = 0$
• L'Hôpital's Rule — can be proved using Taylor expansions
• Newton's Method — uses first-order Taylor approximation for root finding
• Numerical Integration — Simpson's rule is based on quadratic Taylor approximation
• Fourier Series — alternative decomposition using trigonometric basis
• Differential Equations — power series method for solving ODEs
• Optimization — second-order methods use Hessian (second Taylor term)
• Automatic Differentiation — computes exact Taylor coefficients via the chain rule
• Neural Tangent Kernel — linearization of neural networks via first-order Taylor expansion