CONVOLUTIONAL NEURAL NETWORKS

1. INTRODUCTION TO CNNs

What is a Convolutional Neural Network?

A Convolutional Neural Network (CNN) is a specialized type of neural network designed for processing grid-like data, particularly images. CNNs are inspired by the visual cortex of animals and are the foundation of modern computer vision.

Key Characteristics

• Spatial hierarchy: CNNs learn hierarchical patterns from low-level features (edges) to high-level concepts (objects)
• Parameter sharing: Same filters are applied across the entire image
• Translation invariance: Can detect features regardless of their position in the image
• Sparse connectivity: Each neuron connects only to a local region of the input

INPUT IMAGE (32×32×3)
        ↓
    ┌───────┐
    │  CONV │ → Detect edges, colors
    └───────┘
        ↓
    ┌───────┐
    │  POOL │ → Reduce size
    └───────┘
        ↓
    ┌───────┐
    │  CONV │ → Detect shapes
    └───────┘
        ↓
    ┌───────┐
    │  POOL │ → Reduce size
    └───────┘
        ↓
    ┌───────┐
    │  CONV │ → Detect objects
    └───────┘
        ↓
    ┌───────┐
    │   FC  │ → Classification
    └───────┘
        ↓
    OUTPUT (10 classes)
            

2. WHY CNNs FOR IMAGES?

The Problem with Fully Connected Networks

Input: 32×32×3 CIFAR-10 image
Flattened: 3,072 dimensional vector
Hidden layer: 1,024 neurons
Parameters: 3,072 × 1,024 = 3,145,728 parameters!

Problems with FC Networks

How CNNs Solve These Problems

• Preserve spatial structure: Input shape [batch, channels, height, width] - never flattened!
• Parameter efficiency: 3×3 conv with 64 filters = only 1,792 parameters
• Translation invariance: Same filter detects edges everywhere
• Better generalization: Fewer parameters = less overfitting

3. CNN ARCHITECTURE COMPONENTS

┌────────────────────────────────────────┐
│          STANDARD CNN ARCHITECTURE      │
├────────────────────────────────────────┤
│  Input Image (32×32×3)                 │
│           ↓                             │
│  ┌──────────────────────┐               │
│  │ Conv → ReLU → Pool   │ ×N            │
│  └──────────────────────┘               │
│           ↓                             │
│  ┌──────────────────────┐               │
│  │ Conv → ReLU → Pool   │ ×M            │
│  └──────────────────────┘               │
│           ↓                             │
│  Global Average Pooling                │
│           ↓                             │
│  Fully Connected → Output (10 classes) │
└────────────────────────────────────────┘
            

Layer Types

1. Convolutional layers: Extract spatial features
2. Activation layers: Introduce non-linearity (ReLU)
3. Pooling layers: Downsample spatial dimensions
4. Normalization layers: Stabilize training (BatchNorm)
5. Dropout layers: Regularization
6. Fully connected layers: Final classification

4. CONVOLUTIONAL LAYERS

What is Convolution?

Convolution is a mathematical operation that slides a small filter (kernel) over an input to produce a feature map.

Key Parameters

Output Size Formula

How Filters Work

• Early layers (low-level): Edge detectors, color blobs, simple textures
• Middle layers (mid-level): Corners, curves, simple shapes
• Deep layers (high-level): Object parts, complex patterns, semantic concepts

Parameter Count

Example: Conv2d(3, 64, kernel_size=3)
= (3 × 3 × 3 + 1) × 64 = 1,792 parameters

5. POOLING LAYERS

Purpose of Pooling

1. Reduce spatial dimensions → decrease computational cost
2. Increase receptive field → each neuron "sees" more
3. Add translation invariance → small shifts don't change output
4. Reduce overfitting → fewer parameters in subsequent layers

Types of Pooling

Size Reduction with MaxPool(2):
32×32 → MaxPool → 16×16
16×16 → MaxPool → 8×8
 8×8  → MaxPool → 4×4
 4×4  → MaxPool → 2×2
            

6. ACTIVATION FUNCTIONS

Why Activation Functions?

ReLU (Rectified Linear Unit)

Advantages:

• Fast to compute
• Helps with vanishing gradients
• Sparse activations (many zeros)
• Works well in practice

Disadvantages:

• "Dying ReLU" problem: neurons can get stuck at 0
• Not zero-centered

Other Activations

7. NORMALIZATION TECHNIQUES

Batch Normalization

Purpose: Normalize layer inputs to have mean=0, std=1

Benefits:

• Faster convergence
• Allows higher learning rates
• Less sensitive to initialization
• Acts as regularization

Typical Placement

Conv → BatchNorm → ReLU

nn.Conv2d(64, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(inplace=True)

8. REGULARIZATION IN CNNs

Dropout

Purpose: Prevent overfitting by randomly dropping activations

• During training: randomly set activations to 0 with probability p
• During inference: scale activations by (1-p)

Typical Usage

# Higher dropout in FC layers
nn.Dropout(0.5)

# Lower dropout after conv layers
nn.Dropout2d(0.25)

Weight Decay (L2 Regularization)

optimizer = optim.AdamW(
    model.parameters(),
    lr=1e-3,
    weight_decay=5e-4  # λ = 0.0005
)

Label Smoothing

criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

Prevents overconfident predictions and improves calibration

9. DATA AUGMENTATION

Why Data Augmentation?

• Increase effective dataset size
• Improve generalization
• Reduce overfitting
• Better calibration

Common Augmentations for CIFAR-10

Complete Pipeline

train_transform = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(0.2, 0.2, 0.2, 0.1),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=(0.4914, 0.4822, 0.4465),
        std=(0.2470, 0.2435, 0.2616)
    ),
    transforms.RandomErasing(p=0.15),
])

10. TRAINING CNNs

Loss Functions

criterion = nn.CrossEntropyLoss(label_smoothing=0.05)

• Combines LogSoftmax + NLLLoss
• Expects raw logits (before softmax)
• Formula: -log(softmax(logits)[target_class])

Optimizers

Learning Rate Schedules

# Cosine Annealing (smooth decrease)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=epochs, eta_min=1e-6
)

# Step LR (decrease by factor every N epochs)
scheduler = optim.lr_scheduler.StepLR(
    optimizer, step_size=30, gamma=0.1
)

# ReduceLROnPlateau (adaptive)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, mode='min', factor=0.5, patience=5
)

Gradient Clipping

torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0
)

Prevents exploding gradients, especially useful with high learning rates

11. CIFAR-10 DATASET

Dataset Overview

Class Distribution

1. Airplane
2. Automobile
3. Bird
4. Cat
5. Deer
6. Dog
7. Frog
8. Horse
9. Ship
10. Truck

Data Normalization

# CIFAR-10 statistics
mean = (0.4914, 0.4822, 0.4465)  # RGB
std = (0.2470, 0.2435, 0.2616)

normalize = transforms.Normalize(mean=mean, std=std)

Train/Val Split

# Typical split: 45k train / 5k val / 10k test
val_size = 5000
train_indices = list(range(0, 50000 - val_size))
val_indices = list(range(50000 - val_size, 50000))

12. MODEL EVALUATION

Key Metrics

Evaluation Function

def evaluate(model, dataloader, criterion, device):
    model.eval()  # IMPORTANT!
    running_loss = 0.0
    correct = 0
    total = 0

    with torch.no_grad():
        for inputs, targets in dataloader:
            inputs = inputs.to(device)
            targets = targets.to(device)

            outputs = model(inputs)
            loss = criterion(outputs, targets)

            running_loss += loss.item() * inputs.size(0)
            predictions = outputs.argmax(dim=1)
            correct += (predictions == targets).sum().item()
            total += targets.size(0)

    return running_loss / total, correct / total

13. CNN vs FULLY CONNECTED NETWORKS

Parameter Comparison

Spatial Awareness

Typical CIFAR-10 Results

14. IMPLEMENTATION GUIDE

Model Architecture

class CIFAR10CNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()

        self.features = nn.Sequential(
            # Block 1: 32×32 → 16×16
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),
            nn.Dropout(0.25),

            # Block 2: 16×16 → 8×8
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),
            nn.Dropout(0.35),

            # Block 3: 8×8 → 1×1
            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.AdaptiveAvgPool2d(1),
        )

        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Dropout(0.4),
            nn.Linear(256, 128),
            nn.ReLU(inplace=True),
            nn.Dropout(0.2),
            nn.Linear(128, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x

Training Setup

# Model, loss, optimizer
model = CIFAR10CNN().to(device)
criterion = nn.CrossEntropyLoss(label_smoothing=0.05)
optimizer = optim.AdamW(model.parameters(), lr=2e-3, weight_decay=5e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)

# Training loop
for epoch in range(num_epochs):
    model.train()
    for inputs, targets in train_loader:
        inputs, targets = inputs.to(device), targets.to(device)

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

    scheduler.step()

15. COMMON PITFALLS AND SOLUTIONS

Pitfall 1: Flattening at Input

def forward(self, x):
    x = x.view(x.size(0), -1)  # DESTROYS SPATIAL STRUCTURE!
    x = self.conv1(x)  # This will crash

def forward(self, x):
    # x shape: [batch, 3, 32, 32] - keep spatial structure!
    x = self.features(x)
    x = self.classifier(x)

Pitfall 2: Augmenting Test Data

test_transform = transforms.Compose([
    transforms.RandomCrop(32, padding=4),  # NO!
    transforms.ToTensor(),
])

test_transform = transforms.Compose([
    transforms.ToTensor(),  # No augmentation!
    transforms.Normalize(mean, std),
])

Pitfall 3: Not Using .eval()

with torch.no_grad():
    for inputs, targets in test_loader:
        outputs = model(inputs)  # BatchNorm/Dropout still active!

model.eval()  # IMPORTANT!
with torch.no_grad():
    for inputs, targets in test_loader:
        outputs = model(inputs)

Common Training Issues

SUMMARY

Key Takeaways

1. CNNs preserve spatial structure - NEVER flatten at input
2. Convolutions are parameter-efficient - same filter reused across image
3. Use BatchNorm + ReLU after Conv layers for stable training
4. Data augmentation is crucial - easily +5-10% accuracy
5. Always normalize inputs using dataset statistics
6. Don't augment test set - evaluate on clean data
7. Use proper train/val/test splits - 45k/5k/10k for CIFAR-10
8. Monitor both train and val metrics - detect overfitting
9. Save best model, not last epoch
10. Set seeds for reproducibility

Expected Results