Given an input image $\mathbf{I}$ (height $H$, width $W$, channels $C$) and a kernel $\mathbf{K}$ (size $k \times k$, $C$ input channels, $F$ output channels), the convolution at position $(i, j)$ is:

Each output channel is a different filter applied to all input channels, summed together.

• Valid: No padding, output is smaller than input
• Same: Pad so output has same spatial size as input ($p = \lfloor k/2 \rfloor$ for stride 1)
• Zero-padding: Pad with zeros (most common)

Stride is the step size of the convolution. Stride 2 reduces spatial dimensions by approximately half, acting as a learnable downsampling.

Pooling reduces spatial dimensions and provides translation invariance:

• Max Pooling: $\text{MaxPool}(\mathbf{O})_{i,j} = \max_{m,n} \mathbf{O}(i \cdot s + m, j \cdot s + n)$
• Average Pooling: $\text{AvgPool}(\mathbf{O})_{i,j} = \frac{1}{k^2}\sum_{m,n} \mathbf{O}(i \cdot s + m, j \cdot s + n)$
• Global Average Pooling: Average each feature map to a single value (replaces FC layers)

LeNet-5 was the first successful CNN, designed for digit recognition:

Total: ~60K parameters. Showed that local receptive fields + weight sharing + pooling could learn hierarchical features.

AlexNet won ImageNet 2012 by a large margin, starting the deep learning revolution:

• 8 layers, 60M parameters
• ReLU activations instead of tanh (faster training)
• Dropout (p=0.5) for regularization
• Data augmentation (random crop, flip, color jitter)
• GPU training (2 GTX 580 GPUs)
• Local Response Normalization (now replaced by BatchNorm)

VGG showed that depth matters — using only 3×3 convolutions:

• 16-19 layers deep
• All 3×3 convolutions with stride 1, padding 1
• 2×2 max pooling after each conv block
• Three FC layers at the end
• 138M parameters (VGG-16)

Key insight: Two 3×3 convolutions have the same receptive field as one 5×5, but with fewer parameters and more non-linearity.

ResNet introduced skip connections to solve the degradation problem:

where $\mathcal{F}$ is the residual function and $\mathbf{x}$ is the identity shortcut.

Skip connections allow gradients to flow directly through the network, enabling training of 100+ layer networks.

The Inception module applies multiple filter sizes in parallel:

• 1×1 convolution (reduces channels)
• 3×3 convolution
• 5×5 convolution
• 3×3 max pooling

Outputs are concatenated. This captures features at multiple scales without choosing a single kernel size.

Used in MobileNet and EfficientNet for efficiency:

1. Depthwise convolution: Apply one filter per input channel
2. Pointwise convolution: 1×1 convolution to combine channels

Parameters: $k^2 \cdot C_{\text{in}} + C_{\text{in}} \cdot C_{\text{out}}$ vs $k^2 \cdot C_{\text{in}} \cdot C_{\text{out}}$ for standard convolution.

For 3×3 kernels: ~8-9× fewer parameters and FLOPs.

EfficientNet uniformly scales three dimensions:

subject to $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$, where $\phi$ is a compound coefficient.

This finds the optimal balance between depth, width, and resolution for a given compute budget.

1. Start simple: Increase complexity only when needed
2. 3×3 kernels: Two 3×3 beats one 5×5 (more non-linearity, fewer params)
3. Increase depth: But use skip connections beyond 20 layers
4. Channel progression: Double channels when spatial dims halve
5. Global average pooling: Replace FC layers to reduce parameters
6. BatchNorm: After every conv, before activation
7. Skip connections: Enable deeper training, improve gradient flow