Functional Analysis

Problem: Verify the Cauchy-Schwarz inequality for $\mathbf{x} = (1, 2, 3)$ and $\mathbf{y} = (4, 5, 6)$ in $\mathbb{R}^3$.

Solution:

Step 1: Compute the inner product.

Step 2: Compute the norms.

Step 3: Verify the inequality.

Step 4: Check: $32 \leq 32.833$ ✓

Equality condition: Cauchy-Schwarz holds with equality iff $\mathbf{x}$ and $\mathbf{y}$ are linearly dependent. Here $32 < 32.833$ (strict inequality), so $\mathbf{x}$ and $\mathbf{y}$ are linearly independent.

Problem: Project $f(x) = x$ onto the subspace spanned by $\{1, x^2\}$ in $L^2[0,1]$ with inner product $\langle f, g \rangle = \int_0^1 f(x)g(x)dx$.

Solution:

Step 1: First orthogonalize the basis using Gram-Schmidt.

Let $e_0 = 1$ (already normalized: $\|1\|^2 = \int_0^1 1 \, dx = 1$).

Let $v_1 = x^2 - \langle x^2, e_0 \rangle e_0 = x^2 - \int_0^1 x^2 dx \cdot 1 = x^2 - 1/3$.

Normalize: $\|v_1\|^2 = \int_0^1 (x^2 - 1/3)^2 dx = \int_0^1 (x^4 - 2x^2/3 + 1/9)dx = 1/5 - 2/9 + 1/9 = 1/5 - 1/9 = 4/45$.

Step 2: Project $f(x) = x$ onto $\text{span}\{e_0, e_1\}$:

Result: $\text{proj}(x) = \frac{15}{16}x^2 + \frac{3}{16}$

Verification: $\langle x - \text{proj}(x), 1 \rangle = \int_0^1 (x - 15x^2/16 - 3/16)dx = 1/2 - 15/32 - 3/16 = 0$ ✓

Problem: Compute the kernel $K(x, y) = (x^T y + c)^d$ for $d = 2$, $c = 1$, $x = (x_1, x_2)$ and $y = (y_1, y_2)$, and show it equals an inner product in a feature space.

Solution:

Step 1: Expand the kernel.

Step 2: Define the feature map $\phi: \mathbb{R}^2 \to \mathbb{R}^6$:

Step 3: Compute the inner product in feature space:

Result: The polynomial kernel of degree 2 with $c=1$ computes an inner product in a 6-dimensional feature space without explicitly computing $\phi(x)$ or $\phi(y)$.

For general $d$ and input dimension $n$, the explicit feature space has dimension $\binom{n+d}{d}$, which grows exponentially. The kernel trick avoids this exponential cost.

Problem: Find the spectral decomposition of the integral operator $T: L^2[0, \pi] \to L^2[0, \pi]$ defined by $Tf(x) = \int_0^\pi \sin(x)\sin(y) f(y) dy$.

Solution:

Step 1: Identify the operator. $T$ is a rank-1 operator since the kernel $K(x,y) = \sin(x)\sin(y)$ is separable.

Step 2: Write $Tf(x) = \sin(x) \int_0^\pi \sin(y) f(y) dy = \sin(x) \langle f, \sin \rangle$.

Step 3: Find eigenvalues and eigenvectors.

If $f = \sin$, then $T(\sin)(x) = \sin(x) \langle \sin, \sin \rangle = \sin(x) \cdot \frac{\pi}{2}$.

So $T(\sin) = \frac{\pi}{2} \sin$, meaning $\lambda_1 = \frac{\pi}{2}$ is an eigenvalue with eigenvector $e_1 = \sqrt{\frac{2}{\pi}} \sin(x)$ (normalized).

Step 4: For any $f$ orthogonal to $\sin$ (i.e., $\langle f, \sin \rangle = 0$):

So $Tf = 0$ for all $f \perp \sin$. This means all other eigenvalues are $\lambda_n = 0$ for $n \geq 2$.

Step 5: Spectral decomposition:

The operator is compact (rank 1), self-adjoint, and has one non-zero eigenvalue $\lambda_1 = \pi/2$.

Problem: Show that $\ell^2 = \{(x_n)_{n=1}^\infty : \sum_{n=1}^\infty |x_n|^2 < \infty\}$ is a Hilbert space.

Solution:

Step 1: Define the inner product $\langle x, y \rangle = \sum_{n=1}^\infty x_n \overline{y_n}$. This converges absolutely by Cauchy-Schwarz on partial sums.

Step 2: Verify inner product axioms:

• $\langle x, x \rangle = \sum |x_n|^2 \geq 0$ with equality iff $x = 0$ ✓
• $\langle x, y \rangle = \overline{\langle y, x \rangle}$ ✓
• Linearity in first argument ✓

Step 3: Show completeness. Let $\{x^{(k)}\}$ be a Cauchy sequence in $\ell^2$.

For each fixed $n$, $|x_n^{(k)} - x_n^{(m)}| \leq \|x^{(k)} - x^{(m)}\|$, so $\{x_n^{(k)}\}_{k=1}^\infty$ is Cauchy in $\mathbb{R}$, hence converges to some $x_n$.

Step 4: Show $x = (x_n) \in \ell^2$ and $x^{(k)} \to x$.

Given $\epsilon > 0$, choose $K$ such that $\|x^{(k)} - x^{(m)}\| < \epsilon$ for $k, m \geq K$. For any finite $N$:

Taking $N \to \infty$: $\|x^{(k)} - x\| \leq \epsilon$, so $x^{(k)} \to x$ in $\ell^2$.

Result: $\ell^2$ is complete, hence a Hilbert space.

Problem: Show that the differentiation operator $T: C[0,1] \to C[0,1]$ defined by $Tf = f'$ is not bounded with the supremum norm.

Solution:

Step 1: Consider the family of functions $f_n(x) = \sin(2\pi n x)$.

Step 2: Compute norms.

Step 3: Compute the operator norm.

Step 4: Since $2\pi n \to \infty$ as $n \to \infty$, we have $\|T\| = \infty$.

Result: The differentiation operator is unbounded on $C[0,1]$ with the supremum norm. This demonstrates that differentiation is a fundamentally different operation from integration (which is bounded). In quantum mechanics, the momentum operator $\hat{p} = -i\hbar \frac{d}{dx}$ is an unbounded self-adjoint operator.

Problem: Given data $\{(x_1, y_1), (x_2, y_2), (x_3, y_3)\} = \{(0, 1), (1, 3), (2, 5)\}$, find the function $f^*$ that minimizes $\sum_{i=1}^3 (y_i - f(x_i))^2 + \lambda\|f\|_{\mathcal{H}}^2$ using the RBF kernel $K(x, y) = e^{-\gamma(x-y)^2}$ with $\gamma = 1$ and $\lambda = 0.1$.

Solution:

Step 1: By the Representer Theorem, $f^*(\cdot) = \sum_{i=1}^3 \alpha_i K(x_i, \cdot)$.

Step 2: Define the kernel matrix $K_{ij} = K(x_i, x_j) = e^{-(x_i - x_j)^2}$:

Step 3: The optimal $\boldsymbol{\alpha}$ satisfies $(K + \lambda I)\boldsymbol{\alpha} = \mathbf{y}$:

Step 4: Solve the $3 \times 3$ system (e.g., via Gaussian elimination):

Step 5: The solution is $f^*(x) = 0.24 \cdot e^{-x^2} + 1.93 \cdot e^{-(x-1)^2} + 3.63 \cdot e^{-(x-2)^2}$.

Result: The function interpolates the data while remaining smooth (controlled by $\lambda$). The representer theorem reduced an infinite-dimensional problem to solving a $3 \times 3$ linear system.

Problem: Verify that the Fourier basis $\{e^{inx}/\sqrt{2\pi}\}_{n \in \mathbb{Z}}$ is orthonormal in $L^2[-\pi, \pi]$.

Solution:

Step 1: The inner product in $L^2[-\pi, \pi]$ is $\langle f, g \rangle = \int_{-\pi}^{\pi} f(x)\overline{g(x)} dx$.

Step 2: Compute $\langle e^{inx}, e^{imx} \rangle$:

Step 3: Evaluate the integral.

If $n = m$: $\int_{-\pi}^{\pi} e^{0} dx = 2\pi$

If $n \neq m$: $\int_{-\pi}^{\pi} e^{i(n-m)x} dx = \left[\frac{e^{i(n-m)x}}{i(n-m)}\right]_{-\pi}^{\pi} = \frac{e^{i(n-m)\pi} - e^{-i(n-m)\pi}}{i(n-m)} = \frac{2i\sin((n-m)\pi)}{i(n-m)} = 0$

Step 4: Therefore:

Result: The Fourier basis is orthonormal. By completeness, any $f \in L^2[-\pi, \pi]$ can be written as $f(x) = \sum_{n \in \mathbb{Z}} \hat{f}(n) e^{inx}$ with $\hat{f}(n) = \frac{1}{2\pi}\int_{-\pi}^{\pi} f(x) e^{-inx} dx$.

Problem: For a function $f^*$ in an RKHS $\mathcal{H}$ with $\|f^*\|_{\mathcal{H}} \leq B$, derive the generalization bound using the representer theorem.

Solution:

Step 1: By the representer theorem, $f^*(\cdot) = \sum_{i=1}^n \alpha_i K(x_i, \cdot)$.

Step 2: The RKHS norm constraint gives:

Step 3: For the squared loss, the representer theorem solution gives:

Step 4: The generalization error (expected risk minus empirical risk) can be bounded using the covering number of the RKHS ball:

where $\kappa$ is the minimum eigenvalue of $K$.

Step 5: By standard generalization theory:

where $\kappa_n$ captures the effective dimension.

Result: The generalization bound depends on $\|f^*\|_{\mathcal{H}}$ (model complexity), $\lambda$ (regularization), and $n$ (sample size). Larger RKHS norm or smaller $\lambda$ increases overfitting risk.

Problem: Prove Bessel's inequality for an orthonormal sequence $\{e_n\}$ in a Hilbert space $H$: $\sum_{n=1}^{\infty} |\langle x, e_n \rangle|^2 \leq \|x\|^2$.

Solution:

Step 1: Let $S_N = \sum_{n=1}^{N} \langle x, e_n \rangle e_n$ be the partial projection onto $\text{span}\{e_1, \ldots, e_N\}$.

Step 2: Compute the norm of the residual $x - S_N$:

Step 3: Since $\{e_n\}$ is orthonormal:

Step 4: Therefore:

Step 5: Since this holds for all $N$:

Result: Bessel's inequality follows from the non-negativity of the norm. Equality holds iff $\{e_n\}$ is a complete orthonormal basis (Parseval's identity).

Problem: Find the Riesz representative of the linear functional $f(x) = \int_0^1 x(t) \cdot t \, dt$ on $L^2[0,1]$.

Solution:

Step 1: By the Riesz representation theorem, there exists $g \in L^2[0,1]$ such that $f(x) = \langle x, g \rangle = \int_0^1 x(t) g(t) \, dt$ for all $x \in L^2[0,1]$.

Step 2: Comparing $f(x) = \int_0^1 x(t) \cdot t \, dt$ with $\langle x, g \rangle = \int_0^1 x(t) g(t) \, dt$:

We identify $g(t) = t$.

Step 3: Verify: $\|g\|^2 = \int_0^1 t^2 \, dt = 1/3$.

Step 4: The operator norm of $f$ is $\|f\| = \|g\| = \sqrt{1/3} = 1/\sqrt{3}$.

Result: The Riesz representative is $g(t) = t$. The functional $f$ measures the "first moment" of $x$, and its Riesz representative is the function $g(t) = t$ against which $x$ is integrated.

Problem: Show that the integral operator $T: L^2[0,1] \to L^2[0,1]$ defined by $Tf(x) = \int_0^1 K(x,y)f(y)dy$ with continuous kernel $K$ is compact.

Solution:

Step 1: Let $\{f_n\}$ be a bounded sequence in $L^2[0,1]$ with $\|f_n\|_2 \leq M$.

Step 2: The functions $g_n(x) = Tf_n(x) = \int_0^1 K(x,y)f_n(y)dy$ satisfy:

So $\{g_n\}$ is uniformly bounded.

Step 3: Since $K$ is continuous on the compact set $[0,1]^2$, it is uniformly continuous. For $|x_1 - x_2| < \delta$:

So $\{g_n\}$ is equicontinuous.

Step 4: By the Arzelà-Ascoli theorem, every subsequence of $\{g_n\}$ has a uniformly convergent subsequence, hence a convergent subsequence in $L^2[0,1]$.

Result: $T$ maps bounded sets to precompact sets, so $T$ is compact. This is a fundamental result: integral operators with continuous kernels are compact, and their spectral theory is well-understood via the Mercer theorem.

Problem: Extend the linear functional $f(x) = x_1 + x_2$ defined on the subspace $M = \{(x_1, x_2, 0) : x_1, x_2 \in \mathbb{R}\} \subset \mathbb{R}^3$ (with $\ell^2$ norm) to all of $\mathbb{R}^3$ with the same norm.

Solution:

Step 1: The norm of $f$ on $M$: For $\|x\| = 1$ with $x \in M$:

Equality when $x_1 = x_2 = 1/\sqrt{2}$, so $\|f\| = \sqrt{2}$.

Step 2: By Hahn-Banach, extend $f$ to $\tilde{f}$ on $\mathbb{R}^3$ with $\|\tilde{f}\| = \sqrt{2}$.

Step 3: One extension: $\tilde{f}(x) = x_1 + x_2 + 0 \cdot x_3 = x_1 + x_2$ (just ignore $x_3$).

Check norm: $|\tilde{f}(x)| = |x_1 + x_2| \leq \sqrt{2}\|x\|$ ✓

Step 4: Another extension: $\tilde{f}(x) = x_1 + x_2 + \alpha x_3$ where $|\alpha| \leq 1$.

For $\alpha = 1$: $\tilde{f}(x) = x_1 + x_2 + x_3$. Norm: $\|\tilde{f}\| = \sqrt{3} > \sqrt{2}$ ✗

For $\alpha = 0$: $\|\tilde{f}\| = \sqrt{2}$ ✓

Result: The extension $\tilde{f}(x) = x_1 + x_2$ (with coefficient 0 for $x_3$) preserves the norm. Hahn-Banach guarantees such extensions exist, but they may not be unique.

Problem: Let $T_n: \ell^2 \to \ell^2$ be defined by $T_n(x_1, x_2, \ldots) = (x_n, 0, 0, \ldots)$. Is $\{T_n\}$ pointwise bounded? Is it uniformly bounded?

Solution:

Step 1: Compute $\|T_n x\| = |x_n|$ for $x = (x_1, x_2, \ldots) \in \ell^2$.

Step 2: For fixed $x \in \ell^2$, $\sum |x_n|^2 < \infty$, so $|x_n| \to 0$ as $n \to \infty$. Therefore $\sup_n \|T_n x\| = \sup_n |x_n| < \infty$.

The family $\{T_n\}$ is pointwise bounded.

Step 3: By the Uniform Boundedness Principle (Banach-Steinhaus), since $\ell^2$ is a Banach space and $\{T_n\}$ is pointwise bounded, we have $\sup_n \|T_n\| < \infty$.

Step 4: Compute $\|T_n\| = \sup_{\|x\|=1} \|T_n x\| = \sup_{\|x\|=1} |x_n| = 1$ (achieved by $x = e_n$).