CHAPTER 1.7b · RESIDUAL CONNECTIONS

Gradient Flow Comparison

Visualization of the vanishing gradient problem and how residual connections enable gradient flow in deep networks

Gradient Flow is the central challenge when training deep networks. Without skip connections, gradients vanish exponentially – with them, models with 100+ layers can be trained successfully.

📖 Learning Context ▼
🎯

Learning Objectives

  • Visually understand the vanishing gradient problem
  • Understand how skip connections solve the problem
  • Recognize the "Gradient Highway" through residuals
🗺️

Context: Where are we?

Step 7/8 Transformer Fundamentals

This visualization complements Residual & LayerNorm (1.7) with a dynamic representation of gradient flow during training.

💡

Why it matters

Without residuals, the lower layers of a deep network would practically not learn. The skip connection y = x + f(x) guarantees that gradients always have a direct path – even with 128 layers.

📌

Key Takeaways

  • Without skip: Gradient × layer weights at each layer → exponential vanishing
  • With skip: Gradient can flow directly (= 1) + layer contribution
  • Enables training of 100+ layer models

The Vanishing Gradient Problem

In deep neural networks, gradients are multiplied at each layer during the backpropagation process. Without skip connections, this leads to exponentially decreasing gradients – the lower layers barely learn. Residual connections create a "Gradient Highway" that enables direct gradient flow.

Controls

8
5

Gradient Strength

Strong (≥ 0.7)
Medium (0.3 - 0.7)
Weak (0.1 - 0.3)
Vanishing (< 0.1)

Without Residual Connections

Gradients vanish exponentially

With Residual Connections

Gradient Highway enables direct flow

Fig. 1 | Side-by-side comparison of gradient flow during backpropagation. Left: Traditional deep network shows severe vanishing gradient problem. Right: Residual network with skip connections maintains strong gradient flow through all layers.

Why Gradients Vanish

In traditional backpropagation, gradients are multiplied at each layer:

  • Gradient passes through weights and activation functions
  • In deep networks: many multiplications with values < 1
  • Exponential decay: 0.8^10 ≈ 0.107
  • Lower layers receive barely any learning signal

The Solution: Residual Connections

Skip connections create a "Gradient Highway":

  • Direct path for gradients through all layers
  • Formula: H(x) = F(x) + x instead of just F(x)
  • Gradient can pass skip connection unchanged
  • Enables training of networks with 100+ layers