CNNs for Image Data: Convolution, Pooling and Architectures

Why CNNs for Images?

Fully connected networks treat images as flat vectors, destroying spatial structure. A 224ײ24׳ image flattened becomes 150,528 features — connecting each to a hidden layer of 1,000 neurons requires 150 million parameters just for the first layer. CNNs solve this through two key principles:

Parameter Sharing: A single filter (kernel) slides across the entire image, reusing the same weights at every spatial location. A 3׳ kernel has only 9 parameters per channel, regardless of image size.

Translation Equivalence: Because the same filter scans all positions, a feature detected at one location can be recognized anywhere. The network learns what to detect, not where.

How this diagram works: This comparison reveals why CNNs are essential for image data. The left side shows a fully connected (FC) network that flattens a 224ײ24׳ image into 150,528 features, requiring over 150 million parameters just for the first hidden layer — making it computationally expensive and prone to overfitting. The right side shows a CNN that uses convolutional layers with small filters, preserving spatial structure while reducing parameters to 1-25 million. The key insights box highlights why CNNs work: parameter sharing means the same filter detects features everywhere, translation invariance allows recognizing objects regardless of position, and spatial hierarchies let the network learn increasingly complex patterns layer by layer.

The Convolution Operation

Convolution is the core building block. A small kernel (filter) slides over the input, computing element-wise multiplications and summing the results at each position.

Mathematical Definition

For a 2D input $X$ and kernel $K$, the output at position $(i, j)$ is:

where $k_h$ and $k_w$ are the kernel height and width.

Stride and Padding

Stride ($s$): How many pixels the kernel shifts at each step.

Padding ($p$): Zero-padding added around the input border.

where $W$ = input size, $K$ = kernel size, $P$ = padding, $S$ = stride.

Multi-Channel Convolution

For an RGB input with $C_{in}$ channels, each filter has shape $K \times K \times C_{in}$. The filter slides over all channels simultaneously, producing a single output channel:

A convolution layer with $C_{out}$ filters produces $C_{out}$ channels. Total parameters: $C_{out} \times (C_{in} \times K \times K + 1)$.

Pooling Layers

Pooling reduces spatial dimensions, providing translational invariance and reducing computation.

Max Pooling

Selects the maximum value within each pooling window. Preserves the strongest feature activations.

Average Pooling

Computes the mean value within each window. Global Average Pooling (GAP) averages each entire feature map to a single value, commonly replacing fully connected layers.

CNN Architecture: The Pattern

The canonical CNN follows a repeating pattern:

[Conv → ReLU → Pool] נN  →  [FC] נM  →  Output

Convolutional blocks extract hierarchical features:

• Early layers: Low-level features (edges, textures, colors)
• Middle layers: Mid-level features (patterns, parts, shapes)
• Late layers: High-level features (objects, scenes, concepts)

Receptive Field

The receptive field is the region of the original input that influences a particular neuron. As we stack layers, the receptive field grows:

Famous Architectures

LeNet-5 (1998)

Pioneering CNN for handwritten digit recognition. Introduced the Conv→Pool→FC paradigm.

Total: ~60K parameters

AlexNet (2012)

Won ImageNet by a large margin. Key innovations: ReLU activation, dropout, data augmentation, GPU training.

• 5 conv layers + 3 FC layers
• ~60M parameters
• ReLU instead of tanh → faster training
• Overlapping pooling (3׳, S=2)

VGG (2014)

Demonstrated that depth matters. Used only 3׳ convolutions with stride 1 and padding 1.

Key insight: Two 3׳ conv layers have the same receptive field as one 5׵ layer but with fewer parameters:

VGG-16: 13 conv layers + 3 FC layers = ~138M parameters.

ResNet (2015)

Introduced skip connections to solve the degradation problem — deeper networks shouldn't have higher training error.

Architecture Comparison

EfficientNet: Compound Scaling

EfficientNet scales three dimensions jointly using a compound coefficient $\phi$:

subject to $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$. This achieves better accuracy-efficiency tradeoffs than scaling any single dimension.

Feature Visualization

CNNs learn interpretable feature hierarchies:

• Layer 1: Edge detectors (Gabor-like filters), color blobs
• Layer 2: Corners, textures, simple patterns
• Layer 3: Object parts (eyes, wheels, textures)
• Layer 4: Object-level features (faces, dogs, buildings)
• Layer 5: Full objects and scenes

Implementation in PyTorch

Basic CNN

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),

            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.AdaptiveAvgPool2d((1, 1))
        )
        self.classifier = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        return self.classifier(x)

Residual Block

class ResidualBlock(nn.Module):
    def __init__(self, channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)

        self.shortcut = nn.Sequential()
        if stride != 1:
            self.shortcut = nn.Sequential(
                nn.Conv2d(channels, channels, 1, stride, bias=False),
                nn.BatchNorm2d(channels)
            )

    def forward(self, x):
        out = torch.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        return torch.relu(out)

Transfer Learning

import torchvision.models as models

model = models.resnet50(pretrained=True)

for param in model.parameters():
    param.requires_grad = False

model.fc = nn.Linear(model.fc.in_features, num_classes)

optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)

Transfer learning strategy:

1. Load pretrained weights (ImageNet)
2. Freeze early layers (feature extraction)
3. Replace final layer for your task
4. Fine-tune with small learning rate

Key Takeaways