Week 1: From Numerical Methods to Neural Networks

Deep Learning for Macroeconomics — Honours, The University of Edinburgh
Instructor: Juan Zurita · juan.zurita@ed.ac.uk

Learning objectives

By the end of this notebook you will be able to:

Explain why traditional numerical methods struggle with high-dimensional economic models
Describe how a neural network works as a universal function approximator
Build, train, and evaluate a simple neural network in PyTorch
Approximate an economic function (a policy function from the growth model) using a neural network

Prerequisites

Programming & Numerical Methods (PNM), especially Weeks 5–8 on function approximation and VFI
Python, NumPy, Matplotlib (all from PNM)
No prior deep learning experience required

1. Why we need something beyond grids

In PNM you solved the neoclassical growth model using value-function iteration on a grid. That works beautifully for 1- or 2-dimensional state spaces. But modern macro models often have many more state variables:

Model	State variables	Grid points needed
Brock–Mirman (1 asset)	1	100
RBC with capital + labour	2	10,000
OLG with 56 cohorts	56	\(100^{56} \approx 10^{112}\)
Krusell–Smith (distribution)	∞	—

This is the curse of dimensionality: the number of grid points grows exponentially with the number of state variables. A 100-point grid in 10 dimensions has \(100^{10} = 10^{20}\) points — more than the number of grains of sand on Earth.

Key insight: Neural networks don’t use grids. They parameterise functions directly, and the number of parameters grows polynomially, not exponentially, with the dimension of the input.

Let’s see this concretely.

# Setup — run this cell first
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm

# For reproducibility
np.random.seed(42)

# Check if PyTorch is available
try:
    import torch
    import torch.nn as nn
    print(f"PyTorch version: {torch.__version__}")
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    print(f"Device: {device}")
except ImportError:
    print("PyTorch not found. Install with: pip install torch")
    print("Or use Google Colab (recommended) — it comes pre-installed.")

The exponential wall

Let’s visualise how grid-based methods scale compared to neural-network parameters:

# Curse of dimensionality: grids vs neural networks

dims = np.arange(1, 21)
grid_points_per_dim = 50
grid_total = grid_points_per_dim ** dims

# A neural network with 2 hidden layers of width 64:
# parameters = (d * 64 + 64) + (64 * 64 + 64) + (64 * 1 + 1)
nn_params = dims * 64 + 64 + 64 * 64 + 64 + 64 + 1

fig, ax = plt.subplots(figsize=(9, 5))
ax.semilogy(dims, grid_total, 'o-', color='#7a2318', linewidth=2, label='Grid points ($50^d$)')
ax.semilogy(dims, nn_params, 's-', color='#2a6e3f', linewidth=2, label='Neural network parameters')
ax.set_xlabel('Number of state variables ($d$)', fontsize=12)
ax.set_ylabel('Number of unknowns', fontsize=12)
ax.set_title('The Curse of Dimensionality: Grids vs Neural Networks', fontsize=13)
ax.legend(fontsize=11)
ax.set_xlim(1, 20)
ax.axhline(y=1e12, color='gray', linestyle='--', alpha=0.5)
ax.text(15, 2e12, 'Laptop memory limit (~1 TB)', color='gray', fontsize=9)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print(f"At d=10: grid needs {grid_total[9]:.1e} points, NN needs {nn_params[9]:,} parameters")
print(f"At d=20: grid needs {grid_total[19]:.1e} points, NN needs {nn_params[19]:,} parameters")

2. Neural networks as function approximators

You already know function approximation from PNM. Given a function \(f: \mathbb{R}^d \to \mathbb{R}\), we want to find a parameterised approximation \(\hat{f}(x; \theta)\) that is close to \(f\) everywhere on its domain.

In PNM, you used: \[\hat{f}(x; \theta) = \sum_{i=0}^{n} \theta_i \, T_i(x) \quad \text{(Chebyshev polynomials)}\]

In this course, we use: \[\hat{f}(x; \theta) = W_L \, \sigma(W_{L-1} \, \sigma(\cdots \sigma(W_1 x + b_1) \cdots) + b_{L-1}) + b_L\]

where \(\sigma\) is a nonlinear activation function (like ReLU or tanh), and \(\theta = \{W_1, b_1, \ldots, W_L, b_L\}\) are learnable weights and biases.

The Universal Approximation Theorem

Theorem (Hornik, 1991): A feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function on a compact subset of \(\mathbb{R}^d\), to any desired accuracy.

This is the theoretical foundation for everything we do in this course. Neural networks are at least as expressive as polynomials or splines — and in high dimensions, they are much more efficient.

Anatomy of a neural network

Let’s build one step by step. A network with one hidden layer computes:

\[\hat{f}(x) = W_2 \, \sigma(W_1 x + b_1) + b_2\]

where: - \(x \in \mathbb{R}^d\) is the input (e.g., capital stock \(k\)) - \(W_1 \in \mathbb{R}^{h \times d}\) maps inputs to \(h\) hidden neurons - \(b_1 \in \mathbb{R}^h\) is the hidden bias - \(\sigma\) applies element-wise (ReLU, tanh, etc.) - \(W_2 \in \mathbb{R}^{1 \times h}\) maps hidden layer to output - \(b_2 \in \mathbb{R}\) is the output bias

Building a network from scratch (NumPy)

Before we use PyTorch, let’s see exactly what a neural network computes by coding one in NumPy — no magic:

# A neural network from scratch — just matrix multiplications and nonlinearities

def relu(z):
    """ReLU activation: max(0, z)"""
    return np.maximum(0, z)

def neural_net_forward(x, W1, b1, W2, b2):
    """
    Forward pass of a 1-hidden-layer neural network.
    
    x:  input array, shape (n_samples, d)
    W1: weight matrix, shape (d, h)
    b1: bias vector, shape (h,)
    W2: weight matrix, shape (h, 1)
    b2: bias scalar
    
    Returns: output array, shape (n_samples, 1)
    """
    # Step 1: linear transformation
    z1 = x @ W1 + b1          # shape: (n_samples, h)
    
    # Step 2: nonlinear activation
    a1 = relu(z1)              # shape: (n_samples, h)
    
    # Step 3: output layer (linear)
    y = a1 @ W2 + b2           # shape: (n_samples, 1)
    
    return y

# Create a tiny network: 1 input, 4 hidden neurons, 1 output
d, h = 1, 4
W1 = np.random.randn(d, h) * 0.5
b1 = np.random.randn(h) * 0.5
W2 = np.random.randn(h, 1) * 0.5
b2 = np.array([0.0])

# Evaluate on a grid
x = np.linspace(-2, 2, 200).reshape(-1, 1)
y = neural_net_forward(x, W1, b1, W2, b2)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Left: the raw function
axes[0].plot(x, y, color='#7a2318', linewidth=2)
axes[0].set_title('Neural network output (random weights)', fontsize=12)
axes[0].set_xlabel('x')
axes[0].set_ylabel('f(x)')
axes[0].grid(True, alpha=0.3)

# Right: what each hidden neuron contributes
z1 = x @ W1 + b1
a1 = relu(z1)
for i in range(h):
    axes[1].plot(x, a1[:, i] * W2[i, 0], alpha=0.7, linewidth=1.5, label=f'Neuron {i+1}')
axes[1].plot(x, y, 'k--', linewidth=2, label='Sum (output)')
axes[1].set_title('Each hidden neuron\'s contribution', fontsize=12)
axes[1].set_xlabel('x')
axes[1].legend(fontsize=9)
axes[1].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

print(f"Total parameters: {W1.size + b1.size + W2.size + b2.size}")

Activation functions

The activation function \(\sigma\) is what makes neural networks nonlinear. Without it, stacking layers would just give another linear function. Common choices:

Name	Formula	When to use
ReLU	\(\max(0, z)\)	Default for most applications
Tanh	\(\tanh(z)\)	When outputs should be in \((-1, 1)\)
Sigmoid	\(1/(1 + e^{-z})\)	When outputs should be in \((0, 1)\)
Softplus	\(\ln(1 + e^z)\)	Smooth approximation to ReLU

For solving economic models, tanh is often preferred because policy functions tend to be smooth — and tanh is infinitely differentiable (important when we compute Euler-equation residuals later).

# Comparing activation functions

z = np.linspace(-4, 4, 200)

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(z, np.maximum(0, z), linewidth=2, label='ReLU')
ax.plot(z, np.tanh(z), linewidth=2, label='tanh')
ax.plot(z, 1 / (1 + np.exp(-z)), linewidth=2, label='Sigmoid')
ax.plot(z, np.log(1 + np.exp(z)), linewidth=2, label='Softplus')
ax.axhline(y=0, color='gray', linewidth=0.5)
ax.axvline(x=0, color='gray', linewidth=0.5)
ax.set_xlabel('z', fontsize=12)
ax.set_ylabel('σ(z)', fontsize=12)
ax.set_title('Common Activation Functions', fontsize=13)
ax.legend(fontsize=11)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

3. PyTorch — your deep learning toolkit

PyTorch is the framework we use throughout this course. It provides:

Tensors — like NumPy arrays but with GPU support and automatic differentiation
nn.Module — a clean way to define neural network layers
Autograd — computes gradients automatically (crucial for training and for economic applications)
Optimizers — Adam, SGD, L-BFGS, etc.

PyTorch in 5 minutes

If you know NumPy, you already know 80% of PyTorch:

# PyTorch basics — if you know NumPy, you know this

import torch
import torch.nn as nn

# Tensors work like NumPy arrays
a = torch.tensor([1.0, 2.0, 3.0])
b = torch.ones(3)
print("a + b =", a + b)
print("a @ b =", torch.dot(a, b))
print("shape:", a.shape)

# Converting between NumPy and PyTorch
x_np = np.linspace(0, 1, 5)
x_torch = torch.from_numpy(x_np).float()  # NumPy → PyTorch
x_back = x_torch.numpy()                   # PyTorch → NumPy
print(f"\nNumPy:   {x_np}")
print(f"PyTorch: {x_torch}")

# The key difference: requires_grad enables automatic differentiation
x = torch.tensor(2.0, requires_grad=True)
y = x**2 + 3*x + 1  # y = x² + 3x + 1
y.backward()          # compute dy/dx
print(f"\ny = x² + 3x + 1 at x=2: y = {y.item():.1f}")
print(f"dy/dx at x=2: {x.grad.item():.1f}  (exact: 2*2 + 3 = 7)")

Defining a network with `nn.Module`

PyTorch’s nn.Module is the standard way to build neural networks. You define the layers in __init__ and the computation in forward:

class SimpleNet(nn.Module):
    """A feedforward neural network with one hidden layer."""
    
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.layer1 = nn.Linear(input_dim, hidden_dim)   # W1, b1
        self.layer2 = nn.Linear(hidden_dim, output_dim)  # W2, b2
        self.activation = nn.Tanh()
    
    def forward(self, x):
        x = self.layer1(x)       # linear: W1 @ x + b1
        x = self.activation(x)   # nonlinear: tanh(...)
        x = self.layer2(x)       # linear: W2 @ (...) + b2
        return x

# Create the network
net = SimpleNet(input_dim=1, hidden_dim=32, output_dim=1)

# Count parameters
n_params = sum(p.numel() for p in net.parameters())
print(f"Network architecture: 1 → 32 → 1")
print(f"Total parameters: {n_params}")
print(f"\nLayer details:")
for name, param in net.named_parameters():
    print(f"  {name}: shape {list(param.shape)}, {param.numel()} parameters")

4. Training a neural network — learning by minimising a loss

Training a neural network means finding weights \(\theta^*\) that minimise a loss function:

\[\theta^* = \arg\min_\theta \; \mathcal{L}(\theta) = \arg\min_\theta \; \frac{1}{N} \sum_{i=1}^{N} \big(\hat{f}(x_i; \theta) - y_i\big)^2\]

This is exactly what you did in PNM when fitting Chebyshev coefficients — but now we use gradient descent rather than linear algebra, because the network is nonlinear in \(\theta\).

The training loop

Forward pass: compute \(\hat{f}(x; \theta)\)
Loss: compute \(\mathcal{L}(\theta)\)
Backward pass: compute \(\nabla_\theta \mathcal{L}\) (automatic differentiation does this)
Update: \(\theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}\) (gradient step)
Repeat

Let’s train our network to approximate a known function.

Application: approximating the Brock–Mirman policy function

Recall from PNM (Week 8) the Brock–Mirman growth model with log utility and full depreciation:

\[\max \sum_{t=0}^{\infty} \beta^t \ln c_t \quad \text{s.t.} \quad k_{t+1} = k_t^\alpha - c_t\]

The analytical policy function is \(c(k) = (1 - \alpha\beta) k^\alpha\).

Let’s train a neural network to learn this function from data points — as a warm-up before we learn to solve models without knowing the answer.

# Generate training data from the known policy function

alpha = 0.33
beta = 0.95

def true_policy(k):
    """Brock-Mirman analytical policy: c = (1 - αβ) k^α"""
    return (1 - alpha * beta) * k ** alpha

# Training data: sample capital values and compute true consumption
N_train = 200
k_train = torch.FloatTensor(N_train, 1).uniform_(0.1, 5.0)
c_train = (1 - alpha * beta) * k_train ** alpha

# Test data (different points, to check generalisation)
k_test = torch.linspace(0.1, 5.0, 500).reshape(-1, 1)
c_test = (1 - alpha * beta) * k_test ** alpha

print(f"Training samples: {N_train}")
print(f"Test samples: {len(k_test)}")
print(f"k range: [{k_train.min():.2f}, {k_train.max():.2f}]")
print(f"c range: [{c_train.min():.2f}, {c_train.max():.2f}]")

# Train the network

# Create a slightly larger network for this task
model = SimpleNet(input_dim=1, hidden_dim=64, output_dim=1)

# Optimiser: Adam (adaptive learning rate — usually the best default)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)

# Loss function: mean squared error
loss_fn = nn.MSELoss()

# Training loop
n_epochs = 2000
losses = []

for epoch in range(n_epochs):
    # Forward pass
    c_pred = model(k_train)
    
    # Compute loss
    loss = loss_fn(c_pred, c_train)
    
    # Backward pass (compute gradients)
    optimizer.zero_grad()
    loss.backward()
    
    # Update weights
    optimizer.step()
    
    # Record loss
    losses.append(loss.item())
    
    if (epoch + 1) % 500 == 0:
        print(f"Epoch {epoch+1:4d}/{n_epochs}  |  Loss: {loss.item():.2e}")

print(f"\nFinal loss: {losses[-1]:.2e}")

# Evaluate and visualise results

model.eval()  # switch to evaluation mode
with torch.no_grad():  # no need to track gradients for evaluation
    c_nn = model(k_test)

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

# 1. Policy function comparison
k_np = k_test.numpy().flatten()
axes[0].plot(k_np, c_test.numpy().flatten(), 'k-', linewidth=2, label='True: $c = (1-\\alpha\\beta)k^{\\alpha}$')
axes[0].plot(k_np, c_nn.numpy().flatten(), '--', color='#7a2318', linewidth=2, label='Neural network')
axes[0].scatter(k_train.numpy()[:20], c_train.numpy()[:20], c='gray', s=15, alpha=0.5, label='Training data (sample)')
axes[0].set_xlabel('Capital $k$', fontsize=11)
axes[0].set_ylabel('Consumption $c$', fontsize=11)
axes[0].set_title('Policy Function Approximation', fontsize=12)
axes[0].legend(fontsize=9)
axes[0].grid(True, alpha=0.3)

# 2. Approximation error
error = (c_nn.numpy().flatten() - c_test.numpy().flatten())
axes[1].plot(k_np, error * 100, color='#7a2318', linewidth=1.5)
axes[1].axhline(y=0, color='gray', linewidth=0.5)
axes[1].set_xlabel('Capital $k$', fontsize=11)
axes[1].set_ylabel('Error (%)', fontsize=11)
axes[1].set_title('Approximation Error', fontsize=12)
axes[1].grid(True, alpha=0.3)

# 3. Training loss
axes[2].semilogy(losses, color='#2a6e3f', linewidth=1)
axes[2].set_xlabel('Epoch', fontsize=11)
axes[2].set_ylabel('MSE Loss', fontsize=11)
axes[2].set_title('Training Loss Over Time', fontsize=12)
axes[2].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

max_err = np.max(np.abs(error))
print(f"Maximum absolute error: {max_err:.6f}")
print(f"Maximum relative error: {max_err / np.mean(np.abs(c_test.numpy())):.4%}")

5. Going deeper — more layers, more power

What happens when we add more hidden layers? A deep network (2+ hidden layers) can represent compositional structure that a shallow network needs exponentially more neurons to match.

For economic models, 2–3 hidden layers with 32–128 neurons each is usually enough. The table below gives rules of thumb:

Task	Architecture	Why
1-D policy function	1 × 32	Simple, smooth function
Growth model (2-D)	2 × 64	Moderate complexity
IRBC (10-D)	3 × 128	High-dimensional, needs depth
HA model (100+ D)	3 × 256	Very high-dimensional

# A deeper network: 2 hidden layers

class DeeperNet(nn.Module):
    """Feedforward network with 2 hidden layers."""
    
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, output_dim)
        )
    
    def forward(self, x):
        return self.net(x)

# Compare architectures
architectures = [
    ("1 × 16",  SimpleNet(1, 16, 1)),
    ("1 × 64",  SimpleNet(1, 64, 1)),
    ("2 × 32",  DeeperNet(1, 32, 1)),
    ("2 × 64",  DeeperNet(1, 64, 1)),
]

results = {}
for name, model in architectures:
    n_params = sum(p.numel() for p in model.parameters())
    optimizer = torch.optim.Adam(model.parameters(), lr=0.005)
    
    for epoch in range(2000):
        pred = model(k_train)
        loss = loss_fn(pred, c_train)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
    model.eval()
    with torch.no_grad():
        pred_test = model(k_test)
    max_err = torch.max(torch.abs(pred_test - c_test)).item()
    results[name] = {'params': n_params, 'max_error': max_err}
    print(f"{name:8s}  |  params: {n_params:5d}  |  max error: {max_err:.6f}")

print("\n→ Deeper networks with the same parameter count often achieve lower error.")

Exercise 1: Approximating a 2-D function

The Cobb–Douglas production function \(Y = K^\alpha L^{1-\alpha}\) takes two inputs. Train a neural network to approximate it.

Tasks:

Generate training data: sample \(K \in [0.1, 5]\) and \(L \in [0.1, 3]\) uniformly, compute \(Y = K^{0.33} L^{0.67}\).
Build a network with input_dim=2, one hidden layer of 64 neurons, and output_dim=1.
Train for 3,000 epochs. Plot the loss curve.
Compare the neural-network surface to the true surface using a 3D plot.

Click for solution

# Exercise 1 solution

# 1. Generate training data
N = 500
K_data = torch.FloatTensor(N, 1).uniform_(0.1, 5.0)
L_data = torch.FloatTensor(N, 1).uniform_(0.1, 3.0)
X_train = torch.cat([K_data, L_data], dim=1)  # shape: (N, 2)
Y_train = K_data ** 0.33 * L_data ** 0.67      # shape: (N, 1)

# 2. Build the network
model_2d = SimpleNet(input_dim=2, hidden_dim=64, output_dim=1)

# 3. Train
optimizer_2d = torch.optim.Adam(model_2d.parameters(), lr=0.005)
losses_2d = []

for epoch in range(3000):
    pred = model_2d(X_train)
    loss = loss_fn(pred, Y_train)
    optimizer_2d.zero_grad()
    loss.backward()
    optimizer_2d.step()
    losses_2d.append(loss.item())

# 4. Plot
K_grid = np.linspace(0.1, 5.0, 50)
L_grid = np.linspace(0.1, 3.0, 50)
KK, LL = np.meshgrid(K_grid, L_grid)
X_grid = torch.FloatTensor(np.column_stack([KK.ravel(), LL.ravel()]))

model_2d.eval()
with torch.no_grad():
    Y_nn = model_2d(X_grid).numpy().reshape(KK.shape)

Y_true = KK ** 0.33 * LL ** 0.67

fig = plt.figure(figsize=(14, 5))

ax1 = fig.add_subplot(131, projection='3d')
ax1.plot_surface(KK, LL, Y_true, cmap=cm.coolwarm, alpha=0.7)
ax1.set_title('True: $K^{0.33} L^{0.67}$', fontsize=11)
ax1.set_xlabel('K'); ax1.set_ylabel('L'); ax1.set_zlabel('Y')

ax2 = fig.add_subplot(132, projection='3d')
ax2.plot_surface(KK, LL, Y_nn, cmap=cm.coolwarm, alpha=0.7)
ax2.set_title('Neural network', fontsize=11)
ax2.set_xlabel('K'); ax2.set_ylabel('L'); ax2.set_zlabel('Y')

ax3 = fig.add_subplot(133)
ax3.semilogy(losses_2d, color='#2a6e3f')
ax3.set_xlabel('Epoch'); ax3.set_ylabel('MSE Loss')
ax3.set_title('Training loss', fontsize=11)
ax3.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

print(f"Max error: {np.max(np.abs(Y_nn - Y_true)):.6f}")

Exercise 2: Effect of activation function

Repeat the 1-D policy-function approximation using three different activation functions — ReLU, Tanh, and Sigmoid — and compare the results.

Tasks:

Train three networks (same architecture: 1 × 64) with nn.ReLU(), nn.Tanh(), and nn.Sigmoid().
Plot all three approximations against the true policy function.
Which activation gives the smoothest approximation? Which gives the lowest error?

Click for solution

# Exercise 2 solution

class FlexNet(nn.Module):
    def __init__(self, activation):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(1, 64),
            activation,
            nn.Linear(64, 1)
        )
    def forward(self, x):
        return self.net(x)

activations = {
    'ReLU': nn.ReLU(),
    'Tanh': nn.Tanh(),
    'Sigmoid': nn.Sigmoid()
}

fig, ax = plt.subplots(figsize=(9, 5))
ax.plot(k_np, c_test.numpy().flatten(), 'k-', linewidth=2.5, label='True policy')

for name, act in activations.items():
    model_act = FlexNet(act)
    opt = torch.optim.Adam(model_act.parameters(), lr=0.005)
    for ep in range(2000):
        pred = model_act(k_train)
        loss = loss_fn(pred, c_train)
        opt.zero_grad(); loss.backward(); opt.step()
    
    model_act.eval()
    with torch.no_grad():
        pred_test = model_act(k_test)
    err = torch.max(torch.abs(pred_test - c_test)).item()
    ax.plot(k_np, pred_test.numpy().flatten(), '--', linewidth=1.8,
            label=f'{name} (max err: {err:.5f})')

ax.set_xlabel('Capital $k$', fontsize=11)
ax.set_ylabel('Consumption $c$', fontsize=11)
ax.set_title('Effect of Activation Function on Policy Approximation', fontsize=12)
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print("→ Tanh typically gives the smoothest approximation for economic functions.")
print("  ReLU can introduce kinks. Sigmoid saturates and may converge more slowly.")

6. What’s next — from supervised learning to solving models

In this notebook, we knew the answer and trained the network to match it. That’s supervised learning — useful for building intuition, but not very exciting for an economist.

The real power comes in Week 4, where we learn to solve models without knowing the answer. The key idea — Deep Equilibrium Networks — is:

Instead of training on \((k, c)\) pairs, define a loss function from the model’s equilibrium conditions (e.g., the Euler equation). The network learns the policy function by minimising the residual of these conditions.

This is why automatic differentiation (Week 3) matters so much: we need to differentiate the network output with respect to its inputs to evaluate Euler equations, HJB equations, and other equilibrium conditions.

Roadmap

Week 1 (you are here)     Neural networks as function approximators
    ↓
Week 2                     Deep learning fundamentals (training, regularisation)
    ↓
Week 3                     Automatic differentiation (the secret weapon)
    ↓
Week 4                     Deep Equilibrium Networks — solving models!
    ↓
Weeks 5–10                 Stochastic models, constraints, PINNs, HA, estimation

Key takeaways

The curse of dimensionality makes grid-based methods infeasible beyond ~3 state variables. Neural networks scale polynomially.
A neural network is just matrix multiplications + nonlinearities — nothing mysterious.
The Universal Approximation Theorem guarantees that neural networks can approximate any continuous function.
PyTorch provides tensors (like NumPy), automatic differentiation (free gradients), and optimisers (Adam).
Training = minimising a loss function via gradient descent. The same loop appears throughout this course.

Next week: Deep Learning Fundamentals — gradient descent, backpropagation, overfitting, and regularisation.

Learning objectives

Prerequisites

1. Why we need something beyond grids

The exponential wall

2. Neural networks as function approximators

The Universal Approximation Theorem

Anatomy of a neural network

Building a network from scratch (NumPy)

Activation functions

3. PyTorch — your deep learning toolkit

PyTorch in 5 minutes

Defining a network with nn.Module

4. Training a neural network — learning by minimising a loss

The training loop

Application: approximating the Brock–Mirman policy function

5. Going deeper — more layers, more power

Exercise 1: Approximating a 2-D function

Exercise 2: Effect of activation function

6. What’s next — from supervised learning to solving models

Roadmap

Key takeaways

Defining a network with `nn.Module`