Recurrent Neural Networks

Summary

Designed to process sequential data, by processing elements one at a time, while maintaining a memory (hidden state) of previous inputs. Used in language, time-series data, audio, etc

Key

• Input vector $x_t$: the data point at the current time step.
• Hidden state $h_t$: a vector that represents the RNN’s memory, updated at each time step
• Output vector $y_t$: the output at each time step, which may depend on $h_t$

Updating

At each time step $t$, the RNN takes in the current input $x_t$, combines it with the previous hidden state $h_{t-1}$, produces a new hidden state $h_t$, and optionally produces an output $y_t$.

$\boxed{h_t=\tanh (W_x{x}_t+W_h{h}_{t-1}+b_h)}$

• $W_x$ = weight matrix that connects the input $x_t$ to the hidden layer
• $W_h$ = weight matrix that connects the previous hidden state $h_{t+1}$ to the current hidden state
• $b_h$ = bias term

Generating

$y_t=W_y{h}_t+b_y$

• $W_y$ = weight matrix for output layer
• $b_y$= bias for output layer

Backpropagation through time

In training a RNN, you need to minimize the loss over all previous time steps

1. Unroll the RNN through time (turning it into a deep feedforward network where each layer corresponds to a different time step)

2. Apply standard backprop through unrolled structure

3. Accumulate gradients for the shared weights at each time step

4. Update parameters

Forward pass $T$, inputs $x_1,...,x_T$ generate hidden states $h_1,...,h_T$ and outputs $y_1,...,y_T$. At each time step: $h_t=\tanh W_{xh}{x}_t+W_{hh}{h}_{t-1}+b_h$, $y_t=W_{hy}{h}_t+b_y$, $l_t=LOSS(y_t,{\hat{y}}_t)$

For a sequence of length

Backward pass

During the backward pass, we compute gradients of the total loss $\mathcal{L}={\sum }_{t=1}^T{\ell }_t$ with respect to all trainable parameters: $W_{xh}$, $W_{hh}$, and $W_{hy}$.

First, compute the output gradient at each time step:

• ${\delta }_t^y=\frac{\partial {\ell }_t}{\partial y_t}$
• $\frac{\partial \mathcal{L}}{\partial W_{hy}}={\sum }_{t=1}^T{\delta }_t^y{h}_t^{\top }$

Then compute gradients flowing into the hidden states:

• ${\delta }_t^h={\delta }_t^y{W}_{hy}^{\top }+{\delta }_{t+1}^h\cdot \frac{\partial h_{t+1}}{\partial h_t}$
  where $\frac{\partial h_{t+1}}{\partial h_t}=\text{diag}(1-h_{t+1}^2)\cdot W_{hh}^{\top }$ (from $\tanh$ derivative)

For each time step $t$, accumulate parameter gradients:

• $\frac{\partial \mathcal{L}}{\partial W_{hh}}+={\delta }_t^h\odot (1-h_t^2)\cdot h_{t-1}^{\top }$
• $\frac{\partial \mathcal{L}}{\partial W_{xh}}+={\delta }_t^h\odot (1-h_t^2)\cdot x_t^{\top }$

The gradients are summed across all time steps due to weight sharing across $t=1,...,T$.

Optionally, apply gradient clipping to avoid exploding gradients:

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Challenges

• Exploding/Vanishing Gradients: the issue of gradients shrinking or growing too large as they backpropagate through time
• Limited memory: Hard to remember patterns longer than 10-20 time steps

Specialized RNNs

LSTM GRU

PyTorch implementation

import torch
import torch.nn as nn
import torch.optim as optim
 
# Set seed for reproducibility
torch.manual_seed(42)
 
# Hyperparameters
input_size = 5    # dimension of x_t
hidden_size = 4   # dimension of h_t
output_size = 2   # dimension of y_t
seq_len = 6       # sequence length
batch_size = 1    
lr = 0.01
 
# Sample input and target
x_seq = torch.randn(seq_len, batch_size, input_size)  # shape: (T, B, D)
target_seq = torch.randint(0, output_size, (seq_len, batch_size))  # classification targets
 
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.Wxh = nn.Linear(input_size, hidden_size)
        self.Whh = nn.Linear(hidden_size, hidden_size)
        self.Why = nn.Linear(hidden_size, output_size)
        self.tanh = torch.tanh
 
    def forward(self, x_seq):
        T, B, _ = x_seq.size()
        h_t = torch.zeros(B, hidden_size)  # initialize hidden state
        outputs = []
 
        for t in range(T):
            x_t = x_seq[t]
            h_t = self.tanh(self.Wxh(x_t) + self.Whh(h_t))
            y_t = self.Why(h_t)
            outputs.append(y_t)
 
        return torch.stack(outputs)  # shape: (T, B, output_size)
 
# Instantiate model and optimizer
model = RNN(input_size, hidden_size, output_size)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
 
# Training step
model.train()
optimizer.zero_grad()
 
# Forward pass
y_preds = model(x_seq)  # shape: (T, B, output_size)
 
# Compute loss
loss = 0
for t in range(seq_len):
    loss += criterion(y_preds[t], target_seq[t])
 
# Backward pass (BPTT)
loss.backward()
 
# Gradient clipping (optional)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
 
# Parameter update
optimizer.step()
 
print(f"Loss: {loss.item():.4f}")