Week 2: Deep Learning Fundamentals

Deep Learning for Macroeconomics — Honours, The University of Edinburgh
Instructor: Juan Zurita · juan.zurita@ed.ac.uk

Learning objectives

By the end of this notebook you will be able to:

  1. Explain how backpropagation computes gradients through a network
  2. Implement a training loop with mini-batch stochastic gradient descent
  3. Diagnose and fix overfitting using regularisation techniques
  4. Choose appropriate learning rates, optimisers, and architectures
# Setup
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn

torch.manual_seed(42)
np.random.seed(42)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f"PyTorch {torch.__version__} on {device}")

1. Gradient descent from first principles

In PNM you minimised functions using Newton’s method and BFGS. Neural-network training uses a simpler family of methods — gradient descent — because computing the full Hessian is too expensive when you have thousands of parameters.

The update rule

\[\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}(\theta_t)\]

where \(\eta\) is the learning rate — the single most important hyperparameter in deep learning.

# Gradient descent on a simple 2D function (Rosenbrock)

def rosenbrock(x, y):
    return (1 - x)**2 + 100 * (y - x**2)**2

# Gradient descent path
x, y = torch.tensor(-1.5, requires_grad=True), torch.tensor(1.5, requires_grad=True)
lr = 0.001
path = [(x.item(), y.item())]

for step in range(5000):
    f = rosenbrock(x, y)
    f.backward()
    with torch.no_grad():
        x -= lr * x.grad
        y -= lr * y.grad
    x.grad.zero_()
    y.grad.zero_()
    if step % 500 == 0:
        path.append((x.item(), y.item()))

path.append((x.item(), y.item()))
path = np.array(path)

# Plot
X, Y = np.meshgrid(np.linspace(-2, 2, 100), np.linspace(-1, 3, 100))
Z = (1 - X)**2 + 100 * (Y - X**2)**2

fig, ax = plt.subplots(figsize=(8, 6))
ax.contour(X, Y, np.log10(Z + 1), levels=30, cmap='viridis', alpha=0.6)
ax.plot(path[:, 0], path[:, 1], 'o-', color='#7a2318', markersize=4, linewidth=1.5, label='GD path')
ax.plot(1, 1, '*', color='gold', markersize=15, label='Minimum (1, 1)')
ax.set_xlabel('x'); ax.set_ylabel('y')
ax.set_title('Gradient Descent on the Rosenbrock Function', fontsize=12)
ax.legend()
plt.tight_layout()
plt.show()

2. Backpropagation — the chain rule, automated

Backpropagation is just the chain rule applied systematically through the network’s computation graph. PyTorch does this for you with .backward(), but understanding the principle is essential.

For a network \(f = W_2 \sigma(W_1 x + b_1) + b_2\):

\[\frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial f} \cdot \frac{\partial f}{\partial \sigma} \cdot \frac{\partial \sigma}{\partial (W_1 x + b_1)} \cdot \frac{\partial (W_1 x + b_1)}{\partial W_1}\]

Each factor is a matrix — the “Jacobian” of that step. Backpropagation computes these products right-to-left (reverse mode), which is efficient when there are many parameters but few outputs.

3. Optimisers: SGD, Adam, and L-BFGS

Optimiser Key idea When to use
SGD Vanilla gradient descent Baseline; rarely used alone
SGD + Momentum Accumulate past gradients Smoother convergence
Adam Adaptive per-parameter learning rates Default for most tasks
L-BFGS Quasi-Newton (uses curvature) Fine-tuning after Adam; small problems

For economic models, a common recipe is: start with Adam for 80% of training, then switch to L-BFGS for final refinement.

# Comparing optimisers on a toy function-approximation task

target_fn = lambda x: torch.sin(3 * x) * torch.exp(-0.5 * x**2)
x_train = torch.linspace(-3, 3, 200).reshape(-1, 1)
y_train = target_fn(x_train)

optimiser_configs = {
    'SGD (lr=0.01)': lambda p: torch.optim.SGD(p, lr=0.01),
    'SGD+Momentum': lambda p: torch.optim.SGD(p, lr=0.01, momentum=0.9),
    'Adam (lr=0.01)': lambda p: torch.optim.Adam(p, lr=0.01),
}

fig, ax = plt.subplots(figsize=(9, 5))

for name, make_opt in optimiser_configs.items():
    model = nn.Sequential(nn.Linear(1, 64), nn.Tanh(), nn.Linear(64, 1))
    # Same initialisation
    torch.manual_seed(0)
    for m in model:
        if isinstance(m, nn.Linear):
            nn.init.xavier_normal_(m.weight)
            nn.init.zeros_(m.bias)
    
    opt = make_opt(model.parameters())
    losses = []
    for ep in range(2000):
        pred = model(x_train)
        loss = nn.MSELoss()(pred, y_train)
        opt.zero_grad(); loss.backward(); opt.step()
        losses.append(loss.item())
    ax.semilogy(losses, linewidth=1.5, label=name)

ax.set_xlabel('Epoch'); ax.set_ylabel('MSE Loss')
ax.set_title('Optimiser Comparison', fontsize=12)
ax.legend(fontsize=10); ax.grid(True, alpha=0.3)
plt.tight_layout(); plt.show()

4. Overfitting and regularisation

Overfitting means the network memorises training data instead of learning the underlying function. Defences:

  • Early stopping: monitor validation loss; stop when it starts rising
  • Weight decay (L2 regularisation): penalise large weights: \(\mathcal{L}_{\text{total}} = \mathcal{L} + \lambda \|\theta\|^2\)
  • Dropout: randomly zero out neurons during training (forces redundancy)

For DEQNs, overfitting is less of a concern because we sample fresh data each epoch. But it matters when we fit surrogates to simulation output (Week 9).

5. Practical tips for economic applications

  1. Normalise inputs. Capital \(k \in [0.1, 100]\) → rescale to \([0, 1]\) or standardise.
  2. Start with Adam, lr=1e-3. Reduce if training is unstable; increase if too slow.
  3. Use learning-rate schedulers. ReduceLROnPlateau is simple and effective.
  4. Monitor the loss carefully. Plot it. If it plateaus, change the learning rate or architecture.
  5. Tanh for smooth economics. ReLU for fast prototyping. Softplus for positivity constraints.

Exercises

Exercise 1: Implement weight decay. Train the same network with weight_decay=0 and weight_decay=1e-4 in Adam. Compare training and validation loss curves.

Exercise 2: Learning rate sensitivity. Train with lr ∈ {0.1, 0.01, 0.001, 0.0001}. Plot all loss curves on one graph. What happens when lr is too high? Too low?

Next week: Automatic Differentiation for Economics — forward vs reverse mode, and why it matters for equilibrium conditions.