From a single neuron to GPT, one honest step at a time
This volume picks up where the Machine Learning field guide left off. Every entry follows the same dissection: a standard definition, the idea in plain English, the maths derived rather than just stated, a diagram for the concepts that are genuinely spatial or sequential, how it works step by step, honest strengths and weaknesses, and a fully computed worked example.
The sequencing and scope are shaped by MIT's 6.S191, Introduction to Deep Learning (Alexander Amini & Ava Soleimany), the same path from perceptron → sequence models → attention → generative modelling → reinforcement learning that the course itself follows, with a few extra frontier topics folded in.
FND-01 · Foundations
The Perceptron
One neuron, one weighted vote, one line drawn through the data.
“
Standard Definition
A perceptron is the simplest unit of a neural network: a single artificial neuron that computes a weighted sum of its inputs plus a bias, then passes the result through an activation function to produce an output.
The Idea
Every neural network, no matter how large, is built from this one repeating unit. A perceptron takes several inputs, multiplies each by a learned "importance" weight, adds them up along with a bias term, and squashes the result through a non-linear function. Geometrically, a single perceptron draws one straight line (or hyperplane) through the data and asks "which side are you on?" That's the entire computation, everything else in deep learning is this same operation, repeated and stacked.
Diagram · anatomy of a single neuron
z = w₁x₁ + w₂x₂ + w₃x₃ + b, then y = g(z)
The Maths
The forward computation is just two steps:
$$z = \sum_{i=1}^{n} w_ix_i + b = w^Tx+b \qquad y = g(z)$$
where $g$ is a non-linear activation function (see the next entry). Rosenblatt's original 1958 learning rule updates the weights whenever the perceptron misclassifies a point:
$$w \leftarrow w + \eta\,(y_{\text{true}}-y_{\text{pred}})\,x$$
where $\eta$ is the learning rate. If the prediction is already correct, nothing changes; if it's wrong, the weights nudge in whichever direction would have made that specific example classify correctly.
How It Works
Multiply each input by its corresponding weight.
Sum the results and add the bias.
Pass the sum through an activation function (originally a hard step function, giving 0 or 1).
If trained with the perceptron rule, compare to the true label and nudge weights toward correctness.
Strengths
The simplest possible building block, trivial to compute and update.
Guaranteed to converge if the data is linearly separable.
Weaknesses
Can only represent linearly separable functions, a single straight decision boundary.
Famously cannot learn XOR, which directly motivated stacking neurons into Multi-Layer Perceptrons.
Why it matters: nearly every architecture in this guide, CNNs, LSTMs, Transformers, is built by composing this exact same weighted-sum-plus-activation unit in different arrangements.
Worked Example
A perceptron with weights $w=(1,1)$ and bias $b=-1.5$, using a step activation ($g(z)=1$ if $z\ge0$, else $0$), computing the logical AND gate:
x₁
x₂
z = x₁+x₂-1.5
output
0
0
-1.5
0
0
1
-0.5
0
1
0
-0.5
0
1
1
0.5
1
This single neuron perfectly reproduces AND, a straight line ($x_1+x_2=1.5$) cleanly separates the single "1" case from the three "0" cases.
Now try XOR (output 1 only when inputs differ): the four points are (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0. Plot these four points: the two "1"s sit on opposite corners, and so do the two "0"s. No single straight line can separate them, whichever line you draw, it will always have one point of each class on both sides.
Result: a single perceptron solves AND perfectly but provably cannot solve XOR at all, regardless of what weights you choose. This exact limitation, identified by Minsky and Papert in 1969, is what motivated stacking neurons into hidden layers, the very next entry in this guide.
FND-02 · Foundations
Multi-Layer Perceptrons
Stack enough neurons in enough layers, and you can bend any boundary you like.
“
Standard Definition
A multi-layer perceptron (MLP) is a feedforward neural network composed of an input layer, one or more hidden layers of neurons with non-linear activations, and an output layer, capable in principle of approximating any continuous function given enough hidden units.
The Idea
A single perceptron draws one straight line. Stack a layer of several perceptrons, each drawing its own line, and feed their combined outputs into another perceptron, and suddenly you can carve out curved, even disconnected regions of space. Each hidden neuron learns to detect a different simple pattern; deeper layers combine those simple patterns into increasingly abstract ones. This is precisely how a network solves XOR: one hidden neuron can learn "is at least one input on", another learns "are both inputs on", and the output layer combines them into "exactly one is on".
Diagram · a fully-connected feedforward network
every neuron in one layer connects to every neuron in the next, "fully connected" / "dense"
The Maths
Each layer applies an affine transform followed by a non-linearity:
The Universal Approximation Theorem guarantees that a network with just one sufficiently wide hidden layer and a non-linear activation can approximate any continuous function on a bounded domain to arbitrary precision. It's an existence proof, not a construction guide, it says nothing about how many neurons are actually needed or how to find the right weights, which is exactly what backpropagation and gradient descent are for.
How It Works
Feed the input vector into the first layer.
Compute each layer's pre-activation (weighted sum + bias), then apply the activation function.
Pass the result forward as the input to the next layer.
Repeat until the output layer produces the final prediction.
Strengths
Can approximate arbitrarily complex functions, solves XOR and far harder problems.
A flexible, general-purpose building block underlying nearly every architecture.
Weaknesses
Fully-connected layers ignore structure, treating a 2D image or a sequence as an unordered flat vector.
Parameter count grows quickly, and deep stacks are prone to vanishing/exploding gradients without care.
Worked Example
A tiny network: 2 inputs, 2 hidden units (ReLU), 1 output (sigmoid). Input $x=(1.0, 0.5)$.
Hidden layer: with weights $W^{(1)}=\begin{bmatrix}0.3&-0.2\\0.5&0.1\end{bmatrix}$, bias $b^{(1)}=(0.1,-0.1)$:
Result: the network outputs 0.512, essentially a coin flip here since the weights are untrained, but this exact forward-pass mechanism, layer by layer, is what every prediction in every neural network in this guide reduces to.
FND-03 · Foundations
Activation Functions
The one non-negotiable ingredient that stops a deep network from collapsing into a straight line.
“
Standard Definition
An activation function is a non-linear function applied to a neuron's weighted input, introducing the non-linearity that allows neural networks to model complex relationships; without it, any stack of layers would collapse into a single linear transformation.
The Idea
Here's a fact that surprises a lot of newcomers: stacking ten linear layers with no activation functions between them is mathematically identical to a single linear layer. Linear functions compose into linear functions, no matter how many you chain together. The entire reason depth helps at all is the non-linear "kink" inserted between layers, it's what lets a network bend, fold, and combine simple features into rich, curved decision boundaries.
Diagram · the three classic activation shapes
sigmoid and tanh flatten at the extremes (saturate); ReLU stays linear for all positive inputs
Notice the derivatives of sigmoid and tanh both shrink toward zero as $|z|$ grows large, at $z=3$, sigmoid's derivative is already down to 0.045. Stack enough layers using these and the gradient shrinks multiplicatively at every layer during backpropagation, the vanishing gradient problem. ReLU's derivative is a constant 1 for any positive input, no matter how large, which is precisely why it became the default choice for deep networks (at the cost of a new problem: any neuron whose input is always negative gets a derivative of exactly 0 forever, the "dying ReLU" problem, addressed by variants like Leaky ReLU and GELU).
How It Works
Compute the neuron's weighted sum, $z$.
Pass $z$ through the chosen non-linear function to get the neuron's output.
During backpropagation, the function's derivative at that specific $z$ determines how much gradient signal passes backward through this neuron.
Strengths
Sigmoid/tanh give smooth, bounded, interpretable outputs (useful for gates and probabilities).
ReLU is cheap to compute and keeps gradients from vanishing on the positive side.
Weaknesses
Sigmoid and tanh saturate for large $|z|$, killing gradients in deep networks.
ReLU units can "die" permanently if they land in the negative region during training.
Worked Example
Evaluating all three functions and their derivatives at five points:
z
-3
-1
0
1
3
sigmoid (deriv)
0.047 (0.045)
0.269 (0.197)
0.500 (0.250)
0.731 (0.197)
0.953 (0.045)
tanh (deriv)
-0.995 (0.010)
-0.762 (0.420)
0.000 (1.000)
0.762 (0.420)
0.995 (0.010)
ReLU (deriv)
0 (0)
0 (0)
0 (0)
1 (1)
3 (1)
Result: at z=3, sigmoid's gradient has collapsed to 0.045 and tanh's to just 0.010, over 95% of the gradient signal is already gone at a single neuron. ReLU's gradient stays at a clean 1 for any positive z, no matter how large, which is exactly why very deep networks lean on it (or its variants) rather than sigmoid/tanh internally.
FND-04 · Foundations
Loss Functions
The single number a network spends its entire life trying to make smaller.
“
Standard Definition
A loss function quantifies the discrepancy between a model's predictions and the true targets, providing the single scalar objective that gradient descent minimises during training.
The Idea
A network's weights are just numbers, they have no notion of "good" or "bad" until you define a loss function that measures how wrong the current predictions are. Everything downstream, backpropagation, gradient descent, the entire training loop, exists purely to make this one number smaller. Choosing the loss is choosing what the network actually optimises for, get it wrong and the network will confidently learn the wrong thing.
The Maths
For regression, Mean Squared Error penalises large errors disproportionately (because of the square):
For multi-class classification, Categorical Cross-Entropy generalises this across $K$ classes using one-hot true labels $y_{i,k}$ and predicted probabilities $\hat{y}_{i,k}$ (typically from a softmax output):
Cross-entropy is steep when the model is confidently wrong (a predicted probability near 0 for the true class sends $-\log(\hat{y})$ toward infinity) and flat when the model is already correct, exactly the gradient signal you want for classification.
How It Works
Run the forward pass to get predictions.
Compare predictions to true labels using the chosen loss formula.
Feed this scalar loss into backpropagation to compute gradients with respect to every weight.
Strengths
MSE is smooth and simple, ideal when errors should be penalised quadratically.
Cross-entropy provides strong, well-behaved gradients exactly where classification is wrong.
Weaknesses
MSE is sensitive to outliers, one huge error dominates the whole loss.
Cross-entropy requires well-calibrated probability outputs (usually via softmax/sigmoid) to behave correctly.
Worked Example
Cross-entropy: a 4-class prediction, true class is index 2 (one-hot $y=[0,0,1,0]$), predicted probabilities $\hat{y}=[0.1,0.2,0.6,0.1]$:
$$\text{CCE} = -\log(0.6) = 0.511$$
If the model had instead been very confident and correct ($\hat{y}_2=0.95$), the loss would drop to $-\log(0.95)=0.051$; if it had been confidently wrong ($\hat{y}_2=0.05$), the loss would spike to $-\log(0.05)=3.00$.
MSE: a single regression prediction, true value 5.0, predicted 4.2:
$$\text{MSE} = (5.0-4.2)^2 = 0.64$$
Result: notice how steeply cross-entropy punishes confident wrongness (0.051 → 0.511 → 3.00 as confidence in the correct class drops), that steep gradient is exactly what drives a classifier to correct its mistakes quickly.
FND-05 · Foundations
Gradient Descent
Walking downhill in the dark, one small step at a time, always in the steepest direction you can feel.
“
Standard Definition
Gradient descent is an iterative optimisation algorithm that updates a model's parameters in the direction opposite to the gradient of the loss function, taking steps proportional to a learning rate, in order to find a local minimum of the loss.
The Idea
Imagine standing somewhere on a hilly landscape in thick fog, unable to see anything beyond your feet. To get to the bottom, a reasonable strategy is: feel which direction slopes downward most steeply right where you're standing, take a small step that way, and repeat. That's gradient descent, exactly. The "landscape" is the loss function plotted against every parameter in the network, and the "slope you can feel" is the gradient, the vector of partial derivatives of the loss with respect to each parameter.
Diagram · descending a loss curve, step by step
large first steps where the slope is steep, shrinking automatically as the bottom approaches
The Maths
The update rule for a parameter $w$ is:
$$w \leftarrow w - \eta \frac{\partial L}{\partial w}$$
where $\eta$ is the learning rate. Three flavours differ only in how much data is used to estimate the gradient at each step: Batch gradient descent uses the entire training set per step (accurate but slow); Stochastic gradient descent (SGD) uses a single example per step (fast, noisy); Mini-batch gradient descent, the near-universal default, uses a small batch (say 32-256 examples), balancing speed against a stable gradient estimate.
How It Works
Compute the gradient of the loss with respect to every parameter (via backpropagation).
Move every parameter a small step in the opposite direction of its gradient.
Repeat over many batches and many passes (epochs) through the data until the loss stops improving.
Strengths
Simple, general-purpose, and works for virtually any differentiable model.
Scales to models with billions of parameters via mini-batching.
Weaknesses
A learning rate that's too large overshoots and diverges; too small crawls forever.
Can get stuck in poor local minima or saddle points on complex loss surfaces (addressed by the Optimisers entry).
Worked Example
Minimise the toy loss $f(w)=(w-4)^2+3$ (a parabola with its true minimum at $w=4$), starting at $w=0$ with learning rate $\eta=0.3$. The gradient is $f'(w)=2(w-4)$.
Step
1
2
3
4
5
6
w
2.400
3.360
3.744
3.898
3.959
3.984
f(w)
5.560
3.410
3.066
3.010
3.002
3.0003
Step 1: $w=0-0.3\times2(0-4)=0-0.3(-8)=2.4$. Notice the steps shrink automatically as $w$ approaches 4, since the gradient itself shrinks (it's proportional to the distance from the minimum), no manual learning-rate schedule was needed for this to happen.
Result: after just 6 steps, $w=3.984$ and $f(w)=3.0003$, within 0.03% of the true minimum of exactly 3, using nothing but the local slope at each point.
FND-06 · Foundations
Backpropagation
The chain rule, applied with obsessive bookkeeping, run backwards through the entire network.
“
Standard Definition
Backpropagation is the algorithm that computes the gradient of a neural network's loss function with respect to every parameter, by applying the chain rule of calculus backwards through the network's computational graph, layer by layer.
The Idea
Gradient descent needs the gradient of the loss with respect to every single weight in the network, potentially billions of them. Computing each one independently from scratch would be catastrophically slow. Backpropagation's insight is that these gradients share enormous amounts of computation: the chain rule lets you compute the gradient at the output once, then reuse it, multiplying by local derivatives, as you walk backward through each layer toward the input. It's really just careful bookkeeping of "how much is each weight to blame for the final error", propagated backward one layer at a time.
∂L/∂x = ∂L/∂h · ∂h/∂z₁ · ∂z₁/∂x, each factor is a simple local derivative
The Maths
For a chain of computations $x \to z \to h \to L$, the chain rule gives the gradient with respect to any earlier variable as a product of local derivatives:
Backpropagation computes $\partial L/\partial h$ once at the output, then walks backward, multiplying by one local derivative per layer, reusing every previous result rather than recomputing from scratch. This is why it's efficient: computing gradients for all $n$ parameters costs roughly the same as one extra forward pass, not $n$ separate calculations.
How It Works
Forward pass: compute and cache every intermediate value, layer by layer, ending at the loss.
Backward pass: starting from the loss, compute the local derivative at each step and multiply it into the running gradient, moving backward toward the input.
At each weight, the accumulated gradient tells gradient descent exactly which direction and how much to adjust it.
Strengths
Computes exact gradients for every parameter in roughly one extra forward pass' worth of work.
Works automatically for any differentiable computational graph, which is exactly what modern autodiff libraries exploit.
Weaknesses
Repeatedly multiplying small local derivatives across many layers can vanish (or, with large derivatives, explode).
Requires caching every intermediate activation from the forward pass, which costs memory proportional to depth.
Worked Example
A tiny network: $x=2$, layer 1 has $w_1=0.5, b_1=0.1$ with ReLU; layer 2 has $w_2=-0.3, b_2=0.2$ with sigmoid; target $y=1$; loss is squared error.
With learning rate 0.5: $w_2 \leftarrow -0.3-0.5(-0.1458)=-0.227$, and $w_1 \leftarrow 0.5-0.5(0.0795)=0.460$.
Result: every single one of those seven numbers came from one multiplication per step, chained backward from the loss. That's the entire algorithm, applied here by hand to two weights, and applied by a library like PyTorch to billions of weights using exactly the same chain rule.
FND-07 · Foundations
Weight Initialisation
Where you start matters just as much as which direction you walk.
“
Standard Definition
Weight initialisation is the choice of starting values for a network's parameters before training begins; poor initialisation can cause activations to saturate or vanish within the first few layers, while principled schemes like Xavier/Glorot and He initialisation keep signal variance roughly stable across layers.
The Idea
Two naive choices both fail badly. Set every weight to zero, and by symmetry every neuron in a layer computes exactly the same gradient and stays identical forever, the network never breaks symmetry and effectively behaves like a single neuron per layer. Set weights to large random values instead, and pre-activation sums grow huge as they pass through each layer (since summing many random terms compounds their variance), pushing sigmoid/tanh neurons into their saturated, near-zero-gradient regions immediately. Good initialisation schemes choose the random weight variance specifically so that the scale of activations neither explodes nor shrinks as data flows through the network.
The Maths
Xavier/Glorot initialisation (designed for sigmoid/tanh) draws each weight from a distribution with variance:
where $n_{\text{in}}$ and $n_{\text{out}}$ are the number of input and output units of that layer, chosen so the variance of activations is roughly preserved going forward and the variance of gradients is roughly preserved going backward. He initialisation (designed for ReLU, which zeroes out roughly half its inputs) compensates by doubling the variance:
$$\text{Var}(W) = \frac{2}{n_{\text{in}}}$$
How It Works
Choose an initialisation scheme matched to the activation function used in that layer (He for ReLU, Xavier for sigmoid/tanh).
Draw every weight independently from a (typically Gaussian or uniform) distribution with the corresponding variance.
Initialise biases to zero (they don't suffer the symmetry problem weights do).
Strengths
Keeps signal flowing through arbitrarily deep networks without manual per-layer tuning.
Costs nothing extra at training time, it's a one-time setup choice.
Weaknesses
Still just a heuristic, it assumes roughly linear behaviour near initialisation and can't fully prevent instability in very deep or unusual architectures (which is part of why Batch Normalisation and residual connections exist).
Worked Example
A 5-layer tanh network, 100 units per layer, comparing naive initialisation (weight std = 1.0) against Xavier-style initialisation (weight std = $1/\sqrt{100}=0.1$):
Layer
Naive: pre-activation std
Naive: % units saturated
Xavier: pre-activation std
Xavier: % units saturated
1
9.92
79%
1.03
1%
2
10.32
83%
0.63
0%
3
10.02
88%
0.46
0%
5
9.92
78%
0.35
0%
Result: with naive initialisation, roughly 80% of every layer's units are saturated (pinned near tanh's flat extremes) at every single layer, meaning their local gradient is essentially zero, backpropagation has almost nothing to work with. Xavier initialisation keeps virtually every unit in tanh's sensitive, gradient-carrying middle region throughout all 5 layers.
FND-08 · Foundations
Optimisers: Momentum, RMSProp & Adam
Plain gradient descent forgets everything between steps. These remember.
“
Standard Definition
An optimiser is the specific rule used to update a neural network's parameters from the gradients computed by backpropagation; methods like Momentum, RMSProp, and Adam go beyond plain gradient descent by adapting the update using accumulated statistics of past gradients.
The Idea
Momentum is like a ball rolling downhill: instead of only reacting to the current slope, it keeps a "velocity" that accumulates consistent gradient direction over time, smoothing out noisy zig-zagging and speeding through shallow, consistent slopes. RMSProp takes a different angle: it tracks a running average of each parameter's squared gradient and divides the step by its square root, so parameters with consistently large gradients get smaller effective steps and parameters with small gradients get relatively larger ones. Adam (the near-universal default today) simply combines both ideas: momentum for direction, RMSProp-style per-parameter scaling for step size.
The Maths
Momentum:
$$v \leftarrow \beta v + (1-\beta)\nabla L \qquad w \leftarrow w-\eta v$$
RMSProp:
$$s \leftarrow \beta s + (1-\beta)(\nabla L)^2 \qquad w \leftarrow w - \frac{\eta}{\sqrt{s}+\epsilon}\nabla L$$
Adam (with bias-corrected estimates $\hat{m}, \hat{s}$ to counteract $m,s$ starting at zero):
$$m \leftarrow \beta_1 m+(1-\beta_1)\nabla L \qquad s \leftarrow \beta_2 s+(1-\beta_2)(\nabla L)^2$$
$$\hat{m}=\frac{m}{1-\beta_1^t} \qquad \hat{s}=\frac{s}{1-\beta_2^t} \qquad w \leftarrow w-\frac{\eta}{\sqrt{\hat{s}}+\epsilon}\hat{m}$$
How It Works
Compute the gradient at the current step via backpropagation, as always.
Update the running statistics (velocity for Momentum, squared-gradient average for RMSProp, both for Adam).
Use those statistics, rather than the raw gradient alone, to compute the actual parameter update.
Strengths
Momentum accelerates through consistent, shallow slopes and dampens oscillation.
Adam adapts per-parameter, requires little learning-rate tuning, and is robust across a huge range of problems.
Weaknesses
More hyperparameters and memory (Adam stores two running statistics per parameter).
Adam can sometimes generalise slightly worse than well-tuned plain SGD with momentum, an active area of debate in practice.
Worked Example
The same noisy gradient sequence, $[4.0, 3.6, -0.5, 3.2, 3.0]$ (mostly positive but with one contrary spike), fed to three optimisers starting at $w=0$ with learning rate 0.1:
Step
1
2
3
4
5
Plain SGD
-0.40
-0.76
-0.71
-1.03
-1.33
Momentum (β=0.9)
-0.04
-0.11
-0.17
-0.26
-0.37
Adam
-0.10
-0.20
-0.27
-0.35
-0.43
Look at step 3, where the gradient briefly flips to $-0.5$. Plain SGD immediately jerks backward (from -0.76 to -0.71, a step in the wrong direction relative to the overall trend). Momentum barely notices the blip, its accumulated velocity from four steps of consistently positive gradients carries it smoothly through.
Result: after 5 steps, plain SGD has moved erratically and unevenly (-1.33), while Momentum (-0.37) and Adam (-0.43) both progress smoothly and consistently in the dominant direction, ignoring the single noisy outlier gradient at step 3.
FND-09 · Foundations
Regularisation & Dropout
Deliberately handicapping a network during training so it can't just memorise the answers.
“
Standard Definition
Regularisation refers to any technique that constrains a neural network's capacity or training process in order to reduce overfitting and improve generalisation to unseen data, including L1/L2 weight penalties, dropout, and early stopping.
The Idea
A large enough network can simply memorise its training set, including its noise and quirks, rather than learning the underlying pattern, and then fail badly on new data. Dropout fights this by randomly switching off a fraction of neurons on every training step, forcing the network to avoid relying too heavily on any single neuron or fragile co-adapted group of neurons, effectively training a huge ensemble of thinned sub-networks that share weights. L1/L2 penalties add a cost for having large weights at all, nudging the network toward simpler solutions. Early stopping simply watches performance on a held-out validation set and halts training the moment it stops improving, before the network has a chance to start memorising noise.
The Maths
Dropout: during training, each unit is kept with probability $p$ (zeroed otherwise), and the surviving activations are scaled by $1/p$ ("inverted dropout") so the expected total signal matches what the layer will see at test time, when no units are dropped at all:
L2 weight decay adds $\lambda\sum w^2$ to the loss, which simply adds an extra $\lambda w$ term to every gradient:
$$w \leftarrow w - \eta(\nabla L_{\text{data}} + \lambda w)$$
How It Works
Dropout: at each training step, randomly zero out a fraction of neurons, rescale the survivors, backpropagate as normal; at test time, use every neuron with no dropout at all.
Weight decay: add a penalty term to the loss so gradient descent constantly nudges every weight a little toward zero, in addition to whatever the data itself demands.
Early stopping: track validation loss every epoch, and stop (or roll back to) the checkpoint right before it started rising.
Strengths
Dropout is simple, cheap, and remarkably effective across almost any architecture.
Weight decay is a one-line addition with a solid theoretical grounding (it's the same L2 idea as Ridge regression).
Weaknesses
Dropout slows convergence and needs its rate tuned (too high, and the network can't learn anything).
None of these substitute for having enough real training data in the first place.
Worked Example
Inverted dropout on activations $h=[2, 4, 1, 3, 5]$ with keep-probability $p=0.6$. Suppose the random mask zeroes out the second unit: $m=[1,0,1,1,1]$.
Notice the surviving values are scaled up (2 becomes 3.33), that's the "inverted" part: it exactly compensates for the fact that, on average, 40% of units are zero during training, so the expected sum of $\tilde{h}$ across many random masks equals the original sum of $h$, exactly what the layer will see at test time with no dropout at all.
L2 weight decay on a weight $w=2.0$ with data-gradient $0.5$ and $\lambda=0.1$, learning rate 0.5:
$$\text{plain update: } w \leftarrow 2.0-0.5(0.5)=1.75 \qquad \text{with L2: } w \leftarrow 2.0-0.5(0.5+0.1\times2.0)=1.65$$
Result: the L2 penalty pulls the weight an extra 0.10 toward zero beyond what the data alone asked for, a small, constant tax on every weight, every step, that discourages the network from relying on any single large weight.
FND-10 · Foundations
Batch Normalisation
Re-centre and re-scale every layer's activations, every batch, so training never drifts off a cliff.
“
Standard Definition
Batch normalisation is a technique that normalises the activations of a layer across a mini-batch to have zero mean and unit variance, then applies a learned scale and shift, stabilising and accelerating the training of deep networks.
The Idea
As a deep network trains, every layer's weights keep changing, which means the distribution of inputs arriving at every later layer keeps shifting too, a moving target that makes training slower and more fragile ("internal covariate shift"). Batch normalisation forces each layer's activations back to a standard, well-behaved distribution (mean 0, variance 1) at every step, then lets the network learn its own preferred scale and shift on top of that standard baseline if it needs to. In practice this allows much higher learning rates, speeds up convergence dramatically, and adds a mild regularising effect as a side benefit (since the batch statistics themselves are a little noisy).
The Maths
For a mini-batch of activations, compute the batch mean and variance, normalise, then apply a learned scale $\gamma$ and shift $\beta$:
$\gamma$ and $\beta$ are ordinary learnable parameters, trained by backpropagation just like any weight, they let the network undo the normalisation entirely if that turns out to be optimal (e.g. $\gamma=\sqrt{\sigma_B^2+\epsilon}, \beta=\mu_B$ recovers the original activations exactly).
How It Works
For each mini-batch during training, compute the mean and variance of each activation across the batch.
Normalise every activation using those batch statistics.
Scale and shift the result using learned parameters $\gamma$ and $\beta$.
At test time (when there's no "batch" to compute statistics from), use a running average of the mean/variance accumulated during training instead.
Strengths
Substantially speeds up and stabilises training, allowing much higher learning rates.
Reduces sensitivity to weight initialisation choices.
Weaknesses
Behaves inconsistently with very small batch sizes (the batch statistics become noisy and unreliable).
Adds complexity: training and inference now compute normalisation differently, a common source of subtle bugs.
Worked Example
A mini-batch of 8 activations from one neuron: $[2, 4, 4, 4, 5, 5, 7, 9]$.
Result: whatever scale the raw activations happened to arrive at (here, spread between 2 and 9), batch norm first forces them into a clean, standard range centred at zero, then lets the network's own learned $\gamma$ and $\beta$ decide the final scale and offset it actually wants, decoupling "does this layer's output have a sensible shape" from "what shape does this specific layer need".
CNN-01 · Convolutional Neural Networks
The Convolution Operation
A small pattern-detector, slid across every position in an image, looking for the same thing everywhere.
“
Standard Definition
Convolution is an operation that slides a small learnable filter (kernel) across an input, computing the dot product between the filter and each local patch it covers, producing a feature map that indicates where a particular pattern occurs in the input.
The Idea
A fully-connected layer treats every pixel as unrelated to its neighbours and needs a completely separate weight for every input-output pair, hopeless for a realistic image. Convolution makes two crucial assumptions instead: nearby pixels matter far more than distant ones (locality), and a useful pattern, an edge, a curve, a texture, is worth detecting the same way no matter where it appears in the image (translation invariance, achieved via weight sharing: the exact same small filter is reused at every position). This is why CNNs need drastically fewer parameters than a fully-connected network would for the same image, and why they generalise so well to objects appearing anywhere in frame.
Diagram · one filter position, sliding across the image
the 3×3 highlighted patch is multiplied element-wise with the kernel and summed into a single output cell; the window then slides one step and repeats
The Maths
For an input $I$ and kernel $K$ of size $k\times k$, the output feature map at position $(i,j)$ is:
The output size shrinks with each convolution unless the input is padded: for an $n\times n$ input, $k\times k$ kernel, stride $s$, and padding $p$, the output size is $\lfloor(n-k+2p)/s\rfloor+1$. A layer typically learns many filters in parallel, each producing its own feature map (its own "channel"), each specialised to detect a different pattern.
How It Works
Position the kernel over the top-left patch of the input.
Multiply element-wise and sum, producing one output value.
Slide the kernel by the stride length and repeat, until it has covered the entire input.
Learn the kernel's values via backpropagation, exactly like any other weight, so the network discovers which patterns are actually useful.
Strengths
Dramatically fewer parameters than a fully-connected layer over the same input.
Detects a pattern anywhere in the image using the same learned filter (translation invariance).
Weaknesses
Purely local by design, a single layer can't relate far-apart pixels without stacking many layers to grow the receptive field.
Not naturally invariant to rotation or scale, only translation.
Worked Example
A 5×5 image convolved with a 3×3 vertical-edge-detecting kernel $\begin{bmatrix}1&0&-1\\1&0&-1\\1&0&-1\end{bmatrix}$ (positive on the left, negative on the right, zero in the middle, exactly the shape that fires strongly when pixel intensity drops from left to right).
For the top-left 3×3 patch $\begin{bmatrix}1&2&3\\0&1&2\\1&0&1\end{bmatrix}$:
Sliding the kernel across the whole image produces a full 3×3 feature map: $\begin{bmatrix}-4&-2&4\\0&-4&-1\\1&0&-2\end{bmatrix}$.
Result: the strongly positive value (+4, top-right) marks where intensity jumps sharply from low on the left to high on the right, exactly a vertical edge, while negative values mark the opposite transition. A trained CNN's first-layer filters discover edge-detectors much like this one entirely on their own, purely by minimising the loss.
CNN-02 · Convolutional Neural Networks
Pooling Layers
Shrink the feature map on purpose, keeping only what matters most in each neighbourhood.
“
Standard Definition
Pooling is a downsampling operation that reduces the spatial size of a feature map by summarising each local neighbourhood with a single value, most commonly the maximum (max pooling) or the average (average pooling).
The Idea
After convolution detects "an edge is present around here", the exact pixel position of that edge is often less important than the fact that it's present at all somewhere in the neighbourhood. Pooling deliberately throws away precise spatial location in favour of a coarser, more robust summary, which makes the network's predictions more tolerant of small shifts, distortions, and noise in exactly where a feature appears, while also shrinking the feature map and cutting computation for every subsequent layer.
Diagram · 2×2 max pooling with stride 2
each coloured 2×2 region collapses to a single value, the maximum within that region
The Maths
For a pooling window of size $k\times k$ over feature map $S$:
$$\text{MaxPool}(i,j) = \max_{0\le m,nUnlike convolution, pooling has no learnable parameters at all, it's a fixed, deterministic summarisation rule.
How It Works
Divide the feature map into non-overlapping (or sometimes overlapping) windows, typically 2×2.
Replace each window with a single number: its maximum (max pooling) or its mean (average pooling).
Pass the smaller, pooled feature map on to the next layer.
Strengths
Reduces computation and memory for every subsequent layer.
Adds a degree of robustness to small translations and distortions.
Weaknesses
Discards precise spatial information, which can matter for tasks like segmentation that need exact pixel locations.
Many modern architectures replace pooling with strided convolutions instead, letting the network learn its own downsampling.
Worked Example
A 4×4 feature map: $\begin{bmatrix}1&3&2&4\\5&6&1&2\\2&1&0&3\\4&2&1&1\end{bmatrix}$, using 2×2 max pooling with stride 2.
Top-left window $\begin{bmatrix}1&3\\5&6\end{bmatrix}$ → max is 6. Top-right window $\begin{bmatrix}2&4\\1&2\end{bmatrix}$ → max is 4. Bottom-left $\begin{bmatrix}2&1\\4&2\end{bmatrix}$ → max is 4. Bottom-right $\begin{bmatrix}0&3\\1&1\end{bmatrix}$ → max is 3.
The equivalent average-pooling result would be $\begin{bmatrix}3.75&2.25\\2.25&1.25\end{bmatrix}$, noticeably smoother and less sensitive to the single large spike (the 6).
Result: a 4×4 map (16 numbers) becomes a 2×2 map (4 numbers), a 4× reduction in size, while max pooling specifically preserves the strongest signal in every neighbourhood, exactly the kind of information a later layer deciding "is this pattern present anywhere around here" needs most.
CNN-03 · Convolutional Neural Networks
CNN Architectures & Skip Connections
Stack convolutions deeper and deeper, until they stop working, then add one simple trick that fixes it.
“
Standard Definition
A CNN architecture is a specific arrangement of convolutional, pooling, and fully-connected layers; residual (skip) connections, introduced in ResNet, add a layer's input directly to its output, allowing gradients to flow through an identity path unimpeded even in extremely deep networks.
The Idea
Early CNNs (LeNet in 1998, AlexNet in 2012, VGG in 2014) established the basic recipe: stack convolution, activation, and pooling layers, then finish with fully-connected layers for classification, and simply making the stack deeper tended to keep improving accuracy. But researchers discovered something counterintuitive: past a certain depth (around 20 layers), accuracy started getting worse, not from overfitting, but because gradients were vanishing on the way back through so many layers, the deeper layers genuinely couldn't be trained properly. ResNet's fix, in 2015, was almost embarrassingly simple: let each block learn only the residual (the difference from its input) and add the original input back in via a direct "skip connection". This one change made networks with over 100 layers trainable for the first time.
Diagram · a residual block
output = F(x) + x, so even if the learned block F contributes nothing useful, the identity path still carries the signal (and its gradient) straight through
The Maths
A plain deep layer must learn the full desired mapping $H(x)$ directly. A residual block instead learns the residual $F(x)=H(x)-x$, and simply adds $x$ back:
$$\text{output} = F(x) + x$$
The reason this fixes vanishing gradients: differentiating the output with respect to $x$ gives $\partial\text{output}/\partial x = \partial F/\partial x + 1$. That extra +1 guarantees a direct, unimpeded gradient path all the way back to any earlier layer, regardless of how small $\partial F/\partial x$ becomes. Compare this to the earlier entry on Backpropagation: without the skip connection, gradients through $n$ layers multiply $n$ local derivatives together; with it, there's always at least the identity path contributing a clean gradient of exactly 1.
How It Works
Group a few convolutional layers into a "block".
Compute the block's normal output, $F(x)$.
Add the block's original input $x$ directly to $F(x)$ (using a small projection if the dimensions don't match).
Stack many such blocks; even at depths of 50, 101, or 152 layers (ResNet-50/101/152), gradients still reach the earliest layers.
Strengths
Makes networks with 100+ layers trainable, previously impossible.
Simple to implement, adds almost no extra computation.
Weaknesses
Requires matching dimensions between the input and the block's output (handled with a small extra projection layer when they differ).
Depth still isn't free, very deep networks remain expensive to train and run.
Worked Example
Consider a 10-layer chain where each layer's local gradient (how much it shrinks the gradient signal passing through) is 0.6, typical of a saturating activation deep in a network.
Without skip connections: the gradient reaching layer 1 is the product of all 10 local derivatives:
$$0.6^{10} = 0.006$$
Only 0.6% of the original gradient signal survives the trip, layer 1 barely learns anything, no matter how many training steps you run.
With skip connections: every layer's output is $F(x)+x$, so the identity path alone guarantees a gradient contribution of at least 1 reaching layer 1, completely independent of depth or how small each block's own local derivative is.
Result: the difference between "0.6% of the signal survives" and "the signal always survives via the identity path" is precisely why ResNet could go from the 19-layer VGG to 152 layers and beyond, and why residual connections now appear in almost every deep architecture, including Transformers.
CNN-04 · Convolutional Neural Networks
Data Augmentation
Manufacture more training examples for free, just by looking at the same image differently.
“
Standard Definition
Data augmentation is the practice of applying label-preserving transformations, such as rotation, flipping, cropping, or colour jittering, to training examples in order to artificially expand the effective size and diversity of a training set.
The Idea
A photo of a cat is still a photo of a cat if you flip it horizontally, crop it slightly, rotate it a few degrees, or brighten it. But to a network that has only ever seen that exact pixel arrangement, a shifted or rotated version might look completely unfamiliar. Data augmentation manufactures these variations on the fly during training, teaching the network to recognise the underlying pattern regardless of superficial changes in position, orientation, or lighting, effectively multiplying the training set's diversity without collecting a single new labelled image.
The Maths
Augmentations are typically simple geometric or photometric transforms. A rotation by angle $\theta$ around the image centre, for instance, maps each pixel coordinate $(x,y)$ via the standard 2D rotation matrix:
Other common augmentations include random horizontal flips, random crops (taking a smaller random window of the image), colour jitter (randomly perturbing brightness/contrast/saturation), and Cutout/random erasing (blanking out a random patch to force reliance on multiple cues rather than one).
How It Works
At each training step, apply a randomly-chosen combination of transforms to each image before it enters the network.
Keep the label unchanged (a rotated cat is still labelled "cat").
Train as normal; because the exact transform differs every epoch, the network effectively sees a new variant of every image every time.
Strengths
Free, effective regularisation, directly reduces overfitting on limited data.
Makes the model robust to real-world variation it will actually encounter (different angles, lighting, framing).
Weaknesses
Must be chosen carefully per task: flipping digits horizontally would turn a "6" into something resembling a "9", destroying the label.
Adds preprocessing overhead, and excessive augmentation can slow convergence or blur genuinely useful signal.
Worked Example
A pixel sits at position $(x,y)=(4,2)$ relative to the image centre. Rotating the whole image by $\theta=30°$:
So this pixel's value moves from $(4,2)$ to approximately $(2.46, 3.73)$ in the rotated image (with interpolation used to handle the non-integer coordinates). Every pixel in the image undergoes the same transform, and the "cat" label travels with the whole image unchanged.
Result: from a single labelled photo, applying random rotations, flips, and crops across training can generate an effectively unlimited stream of distinct-looking training examples, all sharing the same true label, at zero additional labelling cost.
SEQ-01 · Sequence Models
Recurrent Neural Networks
The same tiny network, run again and again, carrying a memory of everything it's seen so far.
“
Standard Definition
A Recurrent Neural Network is a neural network designed for sequential data that maintains a hidden state, updated at every time step from the current input and the previous hidden state, using the same set of weights at every step.
The Idea
A feedforward network has no memory: feed it the same input twice and it gives the same output twice, with no notion of what came before. Language, audio, and time series all depend fundamentally on order and context, "dog bites man" and "man bites dog" use identical words. An RNN solves this by keeping a running "hidden state" summarising everything relevant seen so far, and updating that state with each new input using the exact same weights every single time step, the same small network is simply applied again and again, once per element of the sequence.
Diagram · the same cell, unrolled across three time steps
this is one cell, drawn three times to show it processing a sequence, not three different cells
The Maths
At every time step $t$, the hidden state updates using the current input and the previous hidden state:
Crucially, $W_{xh}$, $W_{hh}$, and $W_{hy}$ are the exact same matrices at every time step, this weight sharing is what lets an RNN handle sequences of any length with a fixed number of parameters. Training uses Backpropagation Through Time (BPTT): unroll the recurrence into an ordinary (if very deep) computational graph, one copy per time step, and apply standard backpropagation across the whole unrolled chain.
How It Works
Initialise the hidden state (usually to zero) before the sequence begins.
At each time step, combine the current input with the previous hidden state to compute the new hidden state.
Optionally produce an output at each step (for tasks like language modelling) or only at the final step (for tasks like sentiment classification).
Train via BPTT: unroll across all time steps and backpropagate through the entire chain.
Strengths
Naturally handles variable-length sequences with a fixed parameter count.
Maintains a running memory of everything seen so far in the sequence.
Weaknesses
Struggles badly with long-range dependencies due to vanishing gradients through many time steps (the next entry in this guide).
Inherently sequential, step $t$ can't be computed until step $t-1$ finishes, which limits parallelisation during training compared to Transformers.
Worked Example
A tiny scalar RNN: $W_{xh}=0.4$, $W_{hh}=0.6$, $b_h=0.1$, $W_{hy}=0.8$, $b_y=-0.1$, initial hidden state $h_0=0$. Input sequence $x=[1.0,\;0.5,\;-1.0]$.
Result: notice $h_3$ depends on $x_3$, but also (through $h_2$) on $x_2$, and (through $h_2$ and $h_1$) on $x_1$ too, every hidden state carries a compressed trace of the entire sequence so far, exactly the "memory" that makes RNNs suited to sequential data.
SEQ-02 · Sequence Models
The Vanishing/Exploding Gradient Problem
Why a plain RNN forgets anything that happened more than about ten steps ago.
“
Standard Definition
The vanishing/exploding gradient problem refers to the tendency of gradients in deep or recurrent networks to shrink toward zero, or grow toward infinity, as they are propagated backward through many layers or time steps, since backpropagation multiplies many local derivatives together.
The Idea
Recall from Backpropagation that the gradient reaching an early layer is a product of local derivatives across every layer in between. In an RNN, "layers" are time steps, and the same recurrent weight matrix multiplies in at every single one. If that repeated multiplication factor is consistently less than 1 (a very common situation, since tanh's derivative maxes out at 1 and is usually much smaller), the gradient shrinks exponentially with sequence length, so by the time it reaches a hidden state from 20 steps ago, there's essentially nothing left to learn from. If the factor is instead consistently greater than 1, the gradient explodes exponentially instead, overflowing to huge, useless values. Either way, a plain RNN effectively can't learn dependencies spanning more than roughly 10-20 steps.
The Maths
The gradient of the loss with respect to a hidden state $n$ steps in the past involves a product of $n$ terms, each roughly $W_{hh}\cdot\tanh'(z)$:
Since $\tanh'(z)\le1$ always, and is often much smaller, repeated multiplication shrinks this product toward zero exponentially fast in $n$. This is exactly why Long Short-Term Memory networks (next entry) were invented: to provide a path for gradients that doesn't rely on repeated multiplication through a squashing non-linearity at every single step.
How It Works (as a diagnosis)
During training, monitor gradient magnitudes at different points in the unrolled sequence.
If gradients shrink toward zero for distant time steps, the network effectively can't learn long-range dependencies (vanishing).
If gradients grow enormous, training destabilises with huge, erratic weight updates (exploding), commonly mitigated with gradient clipping (capping the gradient's norm at a maximum value before applying the update).
Why this matters
Understanding this failure mode is what directly motivated LSTMs, GRUs, residual connections, and (eventually) attention-based architectures that don't rely on sequential recurrence at all.
Practical mitigations
Gradient clipping handles the exploding case directly and cheaply.
The vanishing case needs an architectural fix, gating mechanisms (LSTM/GRU) or entirely different information pathways (attention).
Worked Example
With $W_{hh}=0.6$ and a typical tanh derivative around 0.5, the repeated multiplication factor per step is $0.6\times0.5=0.3$. Over 10 time steps:
$$0.3^{10} = 0.0000059$$
Less than one-thousandth of one percent of the original gradient survives 10 steps back, a hidden state's influence from 10 steps ago is, for all practical training purposes, invisible to gradient descent.
Now suppose instead $W_{hh}=1.5$ with a tanh derivative near 1 (early in training, near the origin, where tanh is closest to linear):
$$1.5^{10} = 57.7$$
Result: the exact same architecture, with only the recurrent weight changed from 0.6 to 1.5, swings from "gradient vanishes to essentially zero" to "gradient explodes nearly 60×" over just 10 steps. This razor's-edge sensitivity, entirely dependent on whether a repeated multiplier sits just below or just above 1, is the core reason plain RNNs are so difficult to train on long sequences.
SEQ-03 · Sequence Models
Long Short-Term Memory (LSTM)
A conveyor belt for information, with three valves deciding what to drop, what to add, and what to reveal.
“
Standard Definition
An LSTM is a recurrent architecture that maintains a separate cell state, regulated by three learned gates (forget, input, and output), which control what information is discarded, added, and exposed at each time step, allowing gradients to flow across long sequences without vanishing.
The Idea
A plain RNN squeezes everything through a single tanh non-linearity at every step, exactly the repeated-multiplication trap that causes vanishing gradients. The LSTM's key innovation is to add a second, separate pathway, the cell state, that information can travel along with only additive (not repeatedly multiplicative-through-a-squashing-function) updates. Three sigmoid "gates" (each outputting values between 0 and 1, acting like a dial from "fully closed" to "fully open") control this cell state: the forget gate decides what old information to throw away, the input gate decides what new information to add, and the output gate decides what part of the cell state to actually expose as the hidden state this step.
Diagram · one LSTM cell, one time step
the top line is the cell state "conveyor belt": forget gate multiplies it, input gate + candidate add to it, output gate reveals part of it as hₜ
The Maths
All four gates/candidates are computed from the same concatenated input (current input $x_t$ and previous hidden state $h_{t-1}$):
Because $c_t$'s update is additive ($f_t\odot c_{t-1}$ plus a new term, rather than repeatedly passed through tanh), gradients can flow backward through many time steps largely unimpeded whenever the forget gate stays close to 1, directly solving the vanishing gradient problem for information the network has learned is worth keeping.
How It Works
Forget gate: looking at the current input and previous hidden state, decide what fraction of the old cell state to keep.
Input gate + candidate: decide what new information is worth adding, and how much of it to actually add.
Update the cell state: combine the surviving old memory with the new addition.
Output gate: decide what part of the (updated) cell state to reveal as this step's hidden state.
Strengths
Learns long-range dependencies far better than a plain RNN, the original motivation for its invention.
Gating gives the network explicit, learnable control over what to remember and forget.
Weaknesses
Four times the parameters of a plain RNN cell (four gate computations instead of one), slower to train.
Still fundamentally sequential, step $t$ needs step $t-1$'s output, which limits parallelisation (part of what motivated Transformers).
Worked Example
One time step: $x_t=1.0$, $h_{t-1}=0.2$, $c_{t-1}=0.5$ (using simplified scalar gates combining $x_t+h_{t-1}$ for illustration).
Result: the forget gate (0.668) kept roughly two-thirds of the old cell state, the input gate (0.627) let in a bit more than half of the new candidate information, and the output gate (0.613) exposed roughly 61% of the resulting memory as the new hidden state. Three independent, learned decisions, all made from the same two inputs.
SEQ-04 · Sequence Models
Gated Recurrent Units (GRU)
Almost everything LSTM does, with one fewer gate and no separate cell state to keep track of.
“
Standard Definition
A Gated Recurrent Unit is a simplified recurrent architecture that merges the LSTM's forget and input gates into a single update gate, and merges the cell state and hidden state into one, reducing parameter count while retaining most of the ability to model long-range dependencies.
The Idea
The GRU, introduced in 2014, asks: do we really need two separate states (cell state and hidden state) and three separate gates? Its answer is no. A single update gate decides, in one combined decision, how much of the old hidden state to keep versus how much of a new candidate to blend in (playing the combined role of the LSTM's forget and input gates), and a reset gate decides how much of the past hidden state to consider when computing that new candidate in the first place. Fewer moving parts, fewer parameters, but empirically similar performance to LSTM on many tasks.
Notice the update gate $z_t$ performs a single linear interpolation between the old hidden state and the new candidate, if $z_t\approx0$, the old state passes through almost entirely unchanged (a direct, low-resistance path for gradients, just like the LSTM's forget-gate-near-1 case); if $z_t\approx1$, the state is almost entirely replaced by the new candidate.
How It Works
Reset gate: decide how much of the previous hidden state matters for computing the new candidate.
Candidate: compute a proposed new hidden state using the current input and the (possibly reset) past.
Update gate: decide the blend ratio between keeping the old hidden state and adopting the new candidate.
Combine via that single interpolation to get the new hidden state, no separate cell state required.
Strengths
Fewer parameters and faster to train than an LSTM, often with comparable accuracy.
Simpler to reason about, one state, two gates, instead of two states and three gates.
Weaknesses
Lacks the LSTM's separate output gate, giving slightly less fine-grained control over what's exposed versus what's retained internally.
Which of LSTM or GRU wins is genuinely task-dependent, neither is a strict upgrade over the other.
Worked Example
One time step: $x_t=1.0$, $h_{t-1}=0.2$ (using simplified scalar gates combining $x_t+h_{t-1}$).
Result: the update gate (0.668) decided to lean mostly toward the new candidate (contributing 0.391) rather than the old hidden state (contributing only 0.066), a single dial controlling the whole "keep old vs adopt new" decision, versus the LSTM's two separate gates doing conceptually similar work.
SEQ-05 · Sequence Models
Sequence-to-Sequence & Encoder-Decoder Models
Read the whole sentence first, compress it into one vector, then write out the translation from that vector alone.
“
Standard Definition
A sequence-to-sequence (seq2seq) model maps an input sequence to an output sequence of possibly different length, typically using an encoder RNN to compress the entire input into a fixed-size context vector, and a decoder RNN that generates the output sequence conditioned on that vector.
The Idea
Machine translation, summarisation, and speech-to-text all share a structure: an input sequence of one length must become an output sequence of a possibly completely different length (a 5-word English sentence might become a 7-word French one). The encoder-decoder split handles this elegantly: an encoder RNN reads the entire input sequence, one token at a time, and its final hidden state becomes a single fixed-length "context vector" meant to summarise the whole input's meaning. A decoder RNN then starts from that context vector and generates the output sequence one token at a time, feeding each generated token back in as the input to generate the next.
The Maths
Encoder: process input tokens $x_1,\ldots,x_T$, keep only the final hidden state as context:
$$h_t^{\text{enc}} = f(x_t, h_{t-1}^{\text{enc}}) \qquad c = h_T^{\text{enc}}$$
Decoder: initialise its hidden state from $c$, then generate outputs autoregressively, each step conditioned on the previous output token:
$$s_t = f(y_{t-1}, s_{t-1}) \qquad P(y_t\mid y_{The whole model is trained end-to-end to maximise the likelihood of the correct output sequence given the input, one combined loss across every decoder time step.
How It Works
Feed the entire input sequence through the encoder, discarding all hidden states except the last.
Use that final hidden state to initialise the decoder.
Generate the output one token at a time, feeding each generated token back in as the next input, until a special "end of sequence" token is produced.
Strengths
Naturally handles input and output sequences of different, variable lengths.
A clean, general framework: translation, summarisation, and captioning are all just different encoder/decoder pairings.
Weaknesses
Forcing the entire input sequence through one fixed-size context vector is a severe bottleneck, quality degrades sharply on long sequences as early information gets diluted by the time the encoder reaches the end.
This exact bottleneck is what the Attention Mechanism (next entry) was invented to remove.
Worked Example
Translating "I am fine" (3 tokens) to French "Je vais bien" (3 tokens), though the lengths needn't match in general.
Encoding: the encoder RNN processes "I", then "am", then "fine", updating its hidden state each time. Only the final hidden state, after processing "fine", survives as the context vector $c$, everything about "I" and "am" must already be compressed into that single vector by this point, or it's lost.
Decoding: the decoder starts from $c$, generates "Je", then feeds "Je" back in to generate "vais", then feeds "vais" back in to generate "bien", then produces an end-of-sequence token to stop.
Result: a 3-word input became a 3-word output here, but the same mechanism handles a 20-word input becoming a 5-word summary just as naturally, at the cost of squeezing arbitrarily long inputs through one fixed-size bottleneck vector, which is precisely the limitation attention was designed to solve.
ATT-01 · Attention & Transformers
The Attention Mechanism
Instead of compressing everything into one vector, let the decoder look back and choose what matters, every single step.
“
Standard Definition
Attention is a mechanism that computes a weighted combination of a set of value vectors, where the weights are derived from how well a query matches each corresponding key, allowing a model to dynamically focus on the most relevant parts of its input at each step.
The Idea
Seq2seq's fixed context vector forces the entire input through one bottleneck. Attention removes that bottleneck by letting the decoder, at every output step, look back at every encoder hidden state and decide, freshly each time, which ones are most relevant right now. Concretely: a query (what am I looking for right now?) is compared against a set of keys (a short descriptor for each candidate piece of information), producing a relevance score for each; those scores are turned into weights (via softmax, so they're positive and sum to 1); and the actual information retrieved is a weighted sum of values using those weights. It's genuinely just a soft, differentiable version of a database lookup.
Diagram · one query attending over three key/value pairs
query · key similarity sets the weight (thicker = higher); the context vector is the weighted sum of values, here weighted 0.43 / 0.21 / 0.35
The Maths
Given a query $q$, keys $k_1,\ldots,k_n$, and values $v_1,\ldots,v_n$:
In practice, scores are usually scaled by $1/\sqrt{d_k}$ (the square root of the key dimension) before the softmax, to keep the dot products from growing too large in magnitude as dimensionality increases, which would otherwise push softmax into a near one-hot, hard-to-train regime.
How It Works
Compute a similarity score between the query and every key (typically a dot product).
Scale and normalise those scores into weights via softmax, so they're all positive and sum to 1.
Compute the context as the weighted sum of the values, using those softmax weights.
Strengths
Removes the fixed-bottleneck problem of plain seq2seq entirely, the decoder can access any part of the input directly.
Fully differentiable, and the resulting attention weights are often directly interpretable (which inputs mattered most).
Weaknesses
Computing attention over every key costs $O(n)$ per query, so $O(n^2)$ total for a sequence attending to itself, expensive for very long sequences.
Result: because the query points in the same direction as $k_1$, it gets the highest weight (0.434) and pulls the context vector toward $v_1=(10,0)$, but $k_3$'s partial similarity still contributes meaningfully (0.351), giving a context vector that blends all three values rather than picking just one, exactly the "soft" in soft attention.
ATT-02 · Attention & Transformers
Self-Attention & Multi-Head Attention
Every word looks at every other word in the same sentence, including itself, and decides what's relevant.
“
Standard Definition
Self-attention is attention applied within a single sequence, where the queries, keys, and values are all derived from the same input, allowing every position to attend to every other position; multi-head attention runs several such self-attention operations in parallel with different learned projections, then combines the results.
The Idea
Plain attention (previous entry) connects a decoder to an encoder. Self-attention applies the exact same mechanism within one sequence: every token generates its own query, key, and value (via three separate learned weight matrices), then attends over every other token's keys and values, including its own. This is how a Transformer resolves something like "the animal didn't cross the street because it was too tired", self-attention lets "it" directly attend to "animal" (rather than "street"), regardless of the words in between, something a plain RNN would have to carry across every intervening time step. Multi-head attention simply repeats this whole process several times in parallel, each "head" with its own learned $W_Q, W_K, W_V$ projections, so different heads can specialise in tracking different kinds of relationships (one head might track grammatical subject-verb agreement, another might track coreference like the "it" example above) before all the heads' outputs are concatenated and combined.
The Maths
From input embeddings $X$ (one row per token), project into queries, keys, and values using learned matrices:
$$Q = XW_Q \qquad K=XW_K \qquad V=XW_V$$
Then apply scaled dot-product attention across the whole sequence at once:
Project every token's embedding into a query, key, and value vector.
For every token, compute attention scores against every other token's keys (including itself), scale, and softmax.
Compute each token's new representation as the weighted sum of every token's value vectors.
Repeat this in parallel across multiple heads with different learned projections, then concatenate and combine.
Strengths
Directly connects any two tokens in a single step, regardless of distance, no vanishing gradient across time steps.
Fully parallelisable across the sequence (unlike an RNN's step-by-step dependency), which is what makes training on massive datasets practical.
Weaknesses
$O(n^2)$ memory and compute in sequence length, since every token attends to every other token.
Has no inherent sense of token order at all, which is exactly why Positional Encoding (a later entry) has to be added back in explicitly.
Worked Example
Two tokens with 2D embeddings $X=\begin{bmatrix}1&0\\0&1\end{bmatrix}$, using identity projections for Q and K ($W_Q=W_K=I$) and $W_V=\begin{bmatrix}0.5&0\\0&2\end{bmatrix}$:
Softmax each row: token 1's attention weights over [token 1, token 2] are $[0.670, 0.330]$; token 2's are $[0.330, 0.670]$ (by symmetry).
$$\text{token 1's new representation} = 0.670(0.5,0)+0.330(0,2) = (0.335,\;0.660)$$
$$\text{token 2's new representation} = 0.330(0.5,0)+0.670(0,2) = (0.165,\;1.340)$$
Result: each token's new representation blends its own value with the other token's, weighted by how similar their query/key vectors are, exactly self-attention's job. Every one of these numbers matches what full self-attention actually computes, this is not a simplification of the mechanism, just a small enough example to trace by hand.
ATT-03 · Attention & Transformers
The Transformer Architecture
"Attention Is All You Need": throw away recurrence entirely, and build the whole model out of self-attention and feedforward layers.
“
Standard Definition
The Transformer is a neural network architecture that processes entire sequences in parallel using stacked layers of multi-head self-attention and position-wise feedforward networks, with residual connections and layer normalisation around each, replacing recurrence entirely as the mechanism for modelling sequential dependencies.
The Idea
RNNs and LSTMs process a sequence one token at a time, which is both slow (no parallelism across time) and prone to losing information over long distances. The 2017 paper that introduced the Transformer asked a provocative question: what if we dropped recurrence entirely? The answer was to build the whole model from self-attention (which connects any two tokens directly, in one step, entirely in parallel) and simple position-wise feedforward layers, wrapped in residual connections (recall the ResNet entry: a direct identity path keeps gradients flowing) and layer normalisation (Batch Normalisation's cousin, but normalising across features within a single example rather than across a batch, which suits variable-length sequences better). This combination scales to enormous datasets and model sizes precisely because every token can be processed simultaneously rather than waiting for the ones before it.
Diagram · one Transformer encoder block (stacked N times)
the green paths are residual (skip) connections, exactly the same mechanism as the ResNet entry, wrapped around each sub-layer
The Maths
Each block computes two sub-layers, both wrapped in a residual connection and followed by layer normalisation:
The full architecture stacks $N$ of these blocks (the original paper used $N=6$; modern large language models use far more). The encoder-decoder version adds a third sub-layer to each decoder block: cross-attention, where the decoder's queries attend over the encoder's keys and values, the direct architectural descendant of the Attention Mechanism entry, now embedded inside a fully self-attentional model.
How It Works
Embed input tokens and add positional encoding (since self-attention alone has no notion of order).
Pass through $N$ stacked blocks, each applying multi-head self-attention then a feedforward network, with a residual connection and layer norm around each.
For encoder-decoder tasks, the decoder additionally cross-attends to the encoder's final output at every block.
A final linear layer plus softmax converts the last block's output into a probability distribution over the vocabulary (for generation tasks).
Strengths
Fully parallelisable across the sequence, dramatically faster to train at scale than RNNs/LSTMs.
Directly models any-to-any token relationships without the distance-dependent decay of recurrence.
Weaknesses
$O(n^2)$ attention cost makes very long sequences expensive (an active area of research: sparse and linear attention variants).
Needs large amounts of data and compute to reach its potential, it has no built-in inductive bias for sequence order or locality the way RNNs and CNNs do.
Why it matters: essentially every modern large language model, GPT, BERT, LLaMA, Claude itself, is built from stacks of exactly this block.
Worked Example
Tracing one token's journey through a single block, using the self-attention output computed in the previous entry: token 1 entered as $(1,0)$ and self-attention produced $(0.335, 0.660)$.
Residual add: $x + \text{Attention}(x) = (1,0)+(0.335,0.660)=(1.335,0.660)$, the original signal is preserved even though the attention sub-layer only contributed a partial adjustment.
Layer norm then rescales this vector to have zero mean and unit variance across its own features (not across the batch, that's the key difference from Batch Normalisation), then the feedforward network and a second residual-add-and-norm follow the same pattern.
Result: at every stage, the original signal survives via the residual path while the sub-layer (attention or feedforward) only ever needs to learn an adjustment on top of it, exactly the same "learn the residual, not the whole mapping" trick from the ResNet entry, now protecting gradient flow through dozens of stacked Transformer blocks instead of convolutional ones.
ATT-04 · Attention & Transformers
Positional Encoding
Self-attention sees a bag of tokens, not a sentence. This is how order gets smuggled back in.
“
Standard Definition
Positional encoding is a fixed or learned vector added to each token's embedding before it enters a Transformer, injecting information about the token's position in the sequence, since self-attention itself is permutation-invariant and has no inherent notion of order.
The Idea
Look back at the self-attention formula: it computes a weighted sum over every other token's value vectors, based purely on content similarity between queries and keys. Shuffle the input tokens into any order, and self-attention alone would produce exactly the same set of outputs, just permuted along with the input, it genuinely cannot tell "the cat sat on the mat" from "the mat sat on the cat" using content alone. The original Transformer's fix was elegant: add a fixed vector to every token's embedding, one that's unique to each position, using sine and cosine waves of different frequencies, so the model can learn to use these smooth, predictable patterns to infer relative and absolute position.
The Maths
For position $pos$ and embedding dimension index $i$ (out of $d_{\text{model}}$ total dimensions), the original Transformer uses:
Different dimensions oscillate at different frequencies (controlled by the $10000^{2i/d_{\text{model}}}$ term), lower dimensions cycle quickly, higher dimensions cycle very slowly, giving every position a unique fingerprint across the full vector, similar to how the hour, minute, and second hands of a clock together uniquely identify any moment even though each hand alone repeats. A key mathematical property: $PE_{pos+k}$ can be expressed as a linear function of $PE_{pos}$, which is thought to help the model learn to attend based on relative position, not just absolute position.
How It Works
Compute a fixed sinusoidal vector for every position in the sequence, using different frequencies across the vector's dimensions.
Add this vector directly to the corresponding token's embedding (not concatenate, simply add).
Feed the combined embedding-plus-position vector into the Transformer as usual; self-attention now has positional information baked directly into the content it's comparing.
Strengths
Requires no learned parameters at all (in the original sinusoidal form), and generalises to sequence lengths never seen during training.
Provides a smooth, structured signal the model can exploit for both absolute and relative position.
Weaknesses
Many modern models instead use learned positional embeddings, or newer schemes like RoPE (rotary position embedding), which often perform better in practice.
Simply adding position to content mixes two different kinds of information into one vector, an approximation that later architectures have tried to improve on.
Worked Example
Computing the 4-dimensional positional encoding for four different positions:
Position
PE (4 dimensions)
0
[0.000, 1.000, 0.000, 1.000]
1
[0.841, 0.540, 0.010, 1.000]
2
[0.909, -0.416, 0.020, 1.000]
5
[-0.959, 0.284, 0.050, 0.999]
Look at the first two dimensions (the fast-oscillating pair): they swing through very different values between positions 0, 1, 2, and 5, giving nearby positions clearly distinguishable codes. Now look at the last two dimensions (the slow-oscillating pair): they barely move at all across these same five positions, they're still in the very early part of a much longer cycle.
Result: exactly like a clock's second hand distinguishing nearby moments while its hour hand barely moves, the fast dimensions here give fine-grained local position information while the slow dimensions preserve a sense of coarse, long-range position, together making every position's full vector unique.
ATT-05 · Attention & Transformers
Pretraining Paradigms: BERT vs GPT
Two ways to make a Transformer learn language before you ever tell it what task you actually want.
“
Standard Definition
Pretraining is the process of training a large Transformer on a generic, self-supervised objective over massive unlabelled text before fine-tuning it on a specific downstream task; BERT uses a bidirectional encoder trained via masked language modelling, while GPT uses a unidirectional decoder trained via next-token prediction.
The Idea
Labelled data is scarce and expensive; raw text is nearly unlimited. Pretraining exploits this by inventing a task that needs no human labels at all, and just happens to require deep language understanding to solve well. BERT (encoder-only) randomly masks out some words in a sentence and trains the model to predict them using context from both directions, before and after the masked word, which is only possible because it never needs to generate text left-to-right. GPT (decoder-only) instead trains on plain next-token prediction: given everything so far, predict the next word, using only leftward context (since at generation time, the future genuinely doesn't exist yet). This single architectural choice, bidirectional-but-can't-generate vs unidirectional-but-can-generate, is why BERT excels at understanding tasks (classification, extraction) while GPT excels at open-ended generation.
The Maths
BERT's masked language modelling objective: replace a random subset of tokens with a [MASK] placeholder, then maximise the likelihood of recovering the original tokens using bidirectional context:
GPT enforces this "only look backward" constraint using a causal mask inside self-attention, which sets the attention score to $-\infty$ (so its softmax weight becomes exactly 0) for any position attending to a future token.
How It Works
Pretrain a large Transformer on a massive text corpus using the self-supervised objective (masking for BERT, next-token prediction for GPT).
The model develops rich internal representations of grammar, facts, and reasoning patterns purely from this objective, no task-specific labels involved yet.
Fine-tune (or, for large enough GPT-style models, simply prompt) the pretrained model on the actual downstream task, often needing far less labelled data than training from scratch.
Strengths
Learns from effectively unlimited unlabelled text, sidestepping the labelled-data bottleneck.
Representations transfer remarkably well across a huge range of downstream tasks.
Weaknesses
BERT cannot generate text autoregressively (no causal structure); GPT only ever sees left-to-right context, even when right-side context would help.
Pretraining at this scale requires enormous compute, largely inaccessible outside major labs.
Worked Example
BERT-style: given "The cat sat on the [MASK]", BERT uses both "The cat sat on the" (before) and nothing (nothing follows here, but in general it would also use right-side context) to predict the masked word, drawing on bidirectional context.
GPT-style next-token prediction: given the prompt "the cat", a toy model computes logits over a small vocabulary for the next word:
Word
cat
dog
the
runs
jumps
Softmax probability
0.102
0.084
0.015
0.459
0.340
The model samples (or greedily picks) "runs" as the next token, then repeats the whole process treating "the cat runs" as the new prompt, generating one word at a time, autoregressively.
Result: "runs" wins with 45.9% probability, notably higher than "jumps" at 34.0%, even though both are grammatically plausible continuations, this is exactly the mechanism behind every word a GPT-style model ever generates: predict a probability distribution over the entire vocabulary, sample from it, and repeat.
GEN-01 · Generative Models
Autoencoders
Compress the input down to its essence, then see how well you can rebuild it from just that.
“
Standard Definition
An autoencoder is a neural network trained to reconstruct its own input, using an encoder that compresses the input into a lower-dimensional bottleneck representation and a decoder that reconstructs the original input from that compressed representation.
The Idea
An autoencoder's task sounds almost trivially easy, output whatever you were given as input, until you add one crucial constraint: the information has to pass through a much narrower bottleneck layer in the middle. Because that bottleneck is too small to just copy every input value directly, the network is forced to discover a compressed, information-dense representation that captures whatever's most essential about the input, discarding redundancy and noise. Once trained, the encoder alone is a powerful learned compressor, and the bottleneck representation often turns out useful for entirely different downstream tasks, denoising, anomaly detection, or as a foundation for generative models like the VAE (next entry).
The Maths
The encoder maps input $x$ to a lower-dimensional code $z$; the decoder maps $z$ back to a reconstruction $\hat{x}$:
Training minimises the reconstruction error, typically mean squared error for continuous data:
$$\mathcal{L} = \|x-\hat{x}\|^2$$
With a linear encoder and decoder and MSE loss, an autoencoder's optimal solution turns out to span the exact same subspace as PCA's top principal components, autoencoders are, in a real sense, a non-linear generalisation of PCA, capable of learning curved manifolds that PCA's straight lines cannot.
How It Works
Pass the input through the encoder to obtain the compressed bottleneck code.
Pass the code through the decoder to reconstruct the original input.
Compute the reconstruction loss between the input and its reconstruction, and backpropagate through both decoder and encoder together.
Strengths
Learns useful compressed representations with no labels required at all (self-supervised).
The bottleneck forces genuinely informative compression, useful for denoising, anomaly detection, and pretraining.
Weaknesses
A plain autoencoder's bottleneck space has no guaranteed structure, so sampling a random point in it rarely decodes into anything realistic, this is exactly why VAEs exist.
Prone to simply memorising training examples if the bottleneck isn't constrained tightly enough.
Worked Example
Compressing a 4D input $x=(2.0, 4.0, 1.0, 3.0)$ down to 2 dimensions and back, using the best possible linear decoder for this encoder (the pseudo-inverse):
Result: even with the mathematically best possible linear decoder for this particular encoding, some reconstruction error is unavoidable, compressing 4 numbers into 2 necessarily throws information away. That's not a training failure, it's the entire point: the bottleneck forces the network to keep only what matters most for reconstruction, discarding whatever's most redundant.
GEN-02 · Generative Models
Variational Autoencoders (VAE)
Don't just compress to a point, compress to a small cloud of uncertainty, so the space between real examples still makes sense.
“
Standard Definition
A Variational Autoencoder is a generative model that encodes each input as a probability distribution (typically Gaussian, parameterised by a mean and variance) over the latent space rather than a single point, trained to both reconstruct the input and keep that distribution close to a standard normal prior, enabling new data to be generated by sampling from the latent space.
The Idea
A plain autoencoder's bottleneck space is an unstructured mess, nothing stops similar inputs from encoding to wildly different points, or guarantees that the space between two encoded points decodes into anything sensible. A VAE fixes this by encoding each input not as a single point but as a small Gaussian "cloud" (a mean and a variance), and adding a second loss term that pulls every one of these clouds toward a standard normal distribution. The effect is that the entire latent space becomes smooth and continuous, similar inputs land near each other, empty regions between encoded points still decode into plausible outputs, and, crucially, you can now generate entirely new data by simply sampling a random point from that standard normal space and running it through the decoder.
The Maths
The encoder outputs a mean $\mu$ and log-variance $\log\sigma^2$ for each input, rather than a single deterministic code. To actually sample from this distribution while still allowing gradients to flow through the sampling step, the VAE uses the reparameterisation trick:
(sampling the randomness from a fixed, parameter-free $\epsilon$, then combining it with the learned $\mu,\sigma$ via simple arithmetic, so backpropagation can flow through $\mu$ and $\sigma$ normally). The total loss combines reconstruction quality with a KL-divergence penalty pulling the latent distribution toward a standard normal:
Encode the input into a mean $\mu$ and variance $\sigma^2$ (instead of a single point).
Sample a latent code using the reparameterisation trick, $z=\mu+\sigma\odot\epsilon$.
Decode $z$ back into a reconstruction, and compute both the reconstruction loss and the KL penalty.
To generate new data after training, simply sample $z\sim\mathcal{N}(0,1)$ directly and decode it, no encoder needed at generation time.
Strengths
Produces a smooth, structured latent space that supports meaningful interpolation and genuine sampling.
Has a principled probabilistic foundation (it's optimising a real lower bound on the data likelihood).
Weaknesses
Generated samples tend to look noticeably blurrier than GAN or diffusion outputs, a known side effect of the MSE reconstruction term.
Balancing the reconstruction and KL terms is delicate, too much KL pressure and the model ignores the input; too little and the latent space loses its useful structure.
Worked Example
The encoder outputs $\mu=1.5$, $\log\sigma^2=-0.5$ for one latent dimension. A sample $\epsilon=0.8$ is drawn from $\mathcal{N}(0,1)$.
Result: the sampled $z=2.123$ is what actually gets decoded, notice it's stochastic, running this same input through the encoder twice would generate two slightly different $z$ values (different random $\epsilon$), which is exactly the mechanism that lets a VAE produce varied outputs for a given input, and the sizeable KL value (1.178) shows this particular latent distribution still sits fairly far from the standard normal target, exactly the pressure that continues shaping it during training.
GEN-03 · Generative Models
Generative Adversarial Networks (GAN)
A forger and a detective, locked in an arms race, each one only getting better because the other does too.
“
Standard Definition
A Generative Adversarial Network consists of two neural networks, a generator that creates synthetic data from random noise, and a discriminator that learns to distinguish real data from the generator's fakes, trained simultaneously in a minimax game where each network improves by trying to outperform the other.
The Idea
Instead of directly telling a generator network what "realistic" means (an extremely hard thing to specify with a formula), a GAN outsources that judgement to a second network, the discriminator, whose entire job is to get better and better at telling real examples from the generator's fakes. The generator, in turn, is trained purely to fool the discriminator. As training proceeds, both networks improve in lockstep: a better discriminator forces the generator to produce more convincing fakes, and a better generator forces the discriminator to become more discerning. At the theoretical end point, the generator produces samples indistinguishable from real data, and the discriminator can do no better than random guessing (50/50).
Diagram · the adversarial training loop
both networks train simultaneously: the discriminator learns real vs fake; the generator learns from the discriminator's feedback
The Maths
The full minimax objective, where $D$ is the discriminator, $G$ is the generator, $x$ is real data, and $z$ is random noise:
The discriminator wants to maximise this (correctly assigning high scores to real data and low scores to fakes); the generator wants to minimise it (making $D(G(z))$ as close to 1 as possible, fooling the discriminator). In practice, the generator is usually trained instead to maximise $\log D(G(z))$ directly (the "non-saturating" loss), since the original formulation provides very weak gradients early in training when the generator is still bad and $D(G(z))$ is near 0.
How It Works
Sample random noise $z$ and generate a fake example, $G(z)$.
Feed both a real example and the fake example to the discriminator; update the discriminator to better separate the two.
Feed a fresh fake example through the (now slightly better) discriminator; update the generator to make the discriminator's output on this fake closer to "real".
Repeat, alternating between the two updates, ideally converging toward the generator producing indistinguishable fakes.
Strengths
Produces remarkably sharp, realistic samples, historically far crisper than VAE output.
No explicit likelihood or reconstruction loss needed, "realistic" is learned implicitly via the adversarial game.
Weaknesses
Notoriously unstable to train, the generator and discriminator can easily fall out of balance.
Prone to "mode collapse", where the generator finds a small handful of convincing outputs and just keeps producing those instead of the full diversity of real data.
Worked Example
Early in training: the discriminator outputs $D(\text{real})=0.9$ (confidently correct) and $D(\text{fake})=0.3$ (correctly suspicious of the still-weak generator).
Now suppose training progresses and the generator improves enough that the discriminator starts being fooled more often, $D(\text{fake})$ rises to 0.6:
$$\text{Generator loss} = -\log(0.6)=0.511$$
Result: as the generator improves and starts fooling the discriminator more often, its own loss drops (from 1.204 to 0.511), exactly the signal that tells it "keep going in this direction". Meanwhile the discriminator, seeing its accuracy slip, will itself update to sharpen its judgement again, the adversarial back-and-forth that gives GANs their name.
GEN-04 · Generative Models
Diffusion Models
Learn to destroy an image with noise, one tiny step at a time, then learn to run that process backwards.
“
Standard Definition
A diffusion model is a generative model trained to reverse a gradual noising process, learning to predict and remove a small amount of noise at each step, so that starting from pure random noise and repeatedly applying the learned denoising step produces a realistic sample.
The Idea
Diffusion models sidestep GANs' adversarial instability entirely with a different idea, one grounded in physics (the diffusion of ink dropped into water). Take a real image and gradually add a small amount of Gaussian noise, repeated over many steps, until essentially nothing recognisable remains, pure static. This forward process is fixed and requires no learning at all. The interesting part is the reverse process: train a network to predict, at each noise level, what noise was just added, so it can be subtracted back out. Chain that learned denoising step backward, starting from pure random noise, and repeatedly removing a little predicted noise at a time, and you arrive at a realistic sample, one that was never present anywhere in the training set.
The Maths
The forward process adds Gaussian noise at each of $T$ steps according to a schedule $\beta_1,\ldots,\beta_T$:
A network $\epsilon_\theta(x_t, t)$ is trained to predict the noise $\epsilon$ that was added to produce $x_t$ from $x_{t-1}$, using a remarkably simple loss:
Sampling then runs this backward: start from pure noise $x_T\sim\mathcal{N}(0,1)$, and repeatedly use $\epsilon_\theta$ to estimate and subtract out a small amount of noise at each step, gradually revealing a coherent image over the reverse of the same $T$ steps.
How It Works
Training: take a real image, add a random amount of noise according to the schedule, and train the network to predict exactly what noise was added.
Sampling: start from pure random noise.
Repeatedly ask the trained network "what noise do you think is in this?", subtract a scaled portion of its estimate, and repeat for every step, from $T$ down to 0.
The final result, after all $T$ denoising steps, is a generated sample.
Strengths
Much more stable to train than GANs, a straightforward regression-style loss rather than an adversarial game.
Currently produces the highest-fidelity, most diverse samples of any generative approach for images.
Weaknesses
Sampling requires many sequential denoising steps (often dozens to hundreds), making generation much slower than a GAN's single forward pass.
Training and sampling are both computationally expensive at scale.
Worked Example
A single toy data value $x_0=2.0$, forward-noised over 4 steps with $\beta=0.1$ at each step (so $\alpha=1-\beta=0.9$), using a fixed illustrative noise value of 0.5 at each step:
Step
1
2
3
4
xₜ
2.056
2.108
2.158
2.205
signal retained (√ᾱₜ)
0.949
0.900
0.854
0.810
Step 1: $x_1=\sqrt{0.9}(2.0)+\sqrt{0.1}(0.5)=1.897+0.158=2.056$. Notice the "signal retained" fraction shrinks steadily (0.949 → 0.900 → 0.854 → 0.810), by step 4, only 81% of the original signal strength remains identifiable against the accumulated noise; continue this for the full schedule (often 1000 steps) and essentially none of the original signal remains, pure noise.
Result: the reverse (generative) process is exactly this table read backwards, starting from something like $x_4$ (pure noise) and, at each step, using the trained network's noise estimate to move back toward $x_3$, then $x_2$, then $x_1$, then finally $x_0$, a realistic sample.
RL-01 · Deep Reinforcement Learning
Markov Decision Processes & Q-Learning
Learn the value of every action in every situation, purely by trial, error, and a simple bootstrap update.
“
Standard Definition
A Markov Decision Process formalises sequential decision-making as states, actions, transitions, and rewards; Q-learning is a model-free reinforcement learning algorithm that learns the expected long-term reward of taking a given action in a given state, updating its estimate using the observed reward and its own current estimate of the best future action.
The Idea
Reinforcement learning is a different kind of problem from everything else in this guide: there's no fixed dataset of correct answers, only an agent interacting with an environment, receiving a reward signal, and trying to maximise its cumulative reward over time. A Markov Decision Process formalises this: at each step the agent is in some state, takes an action, receives a reward, and transitions to a new state, with the "Markov" property meaning the future depends only on the current state, not the full history. Q-learning learns a function $Q(s,a)$ estimating "how good is it to take action $a$ in state $s$, and then act optimally forever after", updated using the classic reinforcement learning trick of bootstrapping: using your own current (imperfect) estimate of the future to improve your estimate of the present.
The Maths
The Q-learning update rule, applied after observing a transition $(s, a, r, s')$:
where $\alpha$ is the learning rate, $\gamma$ is a discount factor (how much future reward matters relative to immediate reward), and the bracketed term is the temporal difference (TD) error, the gap between what you currently believe $Q(s,a)$ should be and a slightly-more-informed estimate using the reward you actually just received. In Deep Q-Networks (DQN), $Q(s,a)$ is approximated by a neural network instead of a lookup table, letting Q-learning scale to enormous state spaces (like raw pixels from an Atari game) that a table could never cover.
How It Works
Observe the current state, and choose an action (balancing exploring new actions against exploiting known good ones).
Take the action, observe the reward received and the new state reached.
Compute the TD target using the observed reward plus the discounted best-case value of the new state.
Nudge the current $Q(s,a)$ estimate toward that TD target, and repeat, over many episodes of interaction.
Strengths
Model-free, it never needs to know the environment's rules, only rewards and observed transitions.
With a neural network approximator (DQN), scales to very large, high-dimensional state spaces.
Weaknesses
Can be sample-inefficient, often needing enormous amounts of interaction to learn good values.
Only directly handles discrete action spaces, choosing among continuous actions needs different methods (like the Policy Gradient entry next).
Worked Example
Current estimate $Q(s,a)=2.0$. The agent takes action $a$, receives reward $r=5.0$, and lands in state $s'$ where the best next action is worth $\max_{a'}Q(s',a')=8.0$. Discount factor $\gamma=0.9$, learning rate $\alpha=0.1$.
Result: the old estimate (2.0) was far too pessimistic, this action led somewhere much better than expected, so the update nudges $Q(s,a)$ up to 3.02, a modest step toward the fully-informed target of 12.2, exactly the gradual bootstrapping that, repeated over many episodes, eventually converges to accurate action values.
RL-02 · Deep Reinforcement Learning
Policy Gradient Methods
Instead of learning action values and picking the best one, learn the action probabilities directly, and nudge whatever led to a good outcome.
“
Standard Definition
Policy gradient methods directly parameterise and optimise a policy, a probability distribution over actions given a state, by increasing the probability of actions that led to higher-than-expected returns and decreasing the probability of actions that led to lower ones.
The Idea
Q-learning learns action values and picks the best one indirectly. Policy gradient methods take a more direct route: parameterise a policy $\pi_\theta(a\mid s)$ (typically a neural network outputting a probability distribution over actions) and adjust $\theta$ directly to make good actions more likely. The core intuition, from the classic REINFORCE algorithm: after playing out a full trajectory and observing its total return $G$, treat every action taken along the way as "responsible" for that return, if $G$ was high, nudge up the probability of every action taken; if $G$ was low (or negative), nudge them down. Actions are reinforced or discouraged in direct proportion to how good the outcome that followed them turned out to be.
The Maths
The policy gradient theorem gives an unbiased estimator of the gradient of expected return with respect to the policy parameters:
For a simple 2-action softmax policy with logits (preferences) $h_1, h_2$, the probability of action 1 is $\pi(a_1)=\sigma(h_1-h_2)$, and the gradient of $\log\pi(a_1)$ with respect to $h_1$ has the clean closed form $1-\pi(a_1)$. The parameter update after taking action 1 and observing return $G$:
Directly optimises the thing you actually care about, expected return, rather than an intermediate value function.
Weaknesses
High variance gradient estimates from a single sampled trajectory, often needing many episodes (or variance-reduction tricks like a learned baseline) to train stably.
Tends to be less sample-efficient than value-based methods for problems where Q-learning applies cleanly.
Worked Example
A 2-action policy with logits $h_1=0.847$, $h_2=0$, giving $\pi(a_1)=\sigma(0.847)=0.700$. The agent takes action 1 and receives return $G=+10$ (a good outcome). Learning rate $\alpha=0.01$.
Result: the identical action, taken from the identical state, gets reinforced (0.700 → 0.706) when it's followed by a good return and discouraged (0.700 → 0.694) when followed by a bad one, policy gradient methods learn purely from this signal, repeated across thousands of trajectories, without ever needing to know the value of the action in advance.
FRONT-01 · Modern Frontiers
Transfer Learning & Fine-Tuning
Why train from scratch when a model that already understands images (or language) can just be nudged toward your specific task?
“
Standard Definition
Transfer learning is the practice of taking a model already trained on a large, general dataset and adapting it to a new, typically smaller and more specific task, either by fine-tuning some or all of its weights or by using it as a fixed feature extractor.
The Idea
A CNN trained on millions of general images learns early layers that detect edges, textures, and simple shapes, features that are useful for essentially any visual task, not just the one it was originally trained on. Rather than reinventing these general-purpose features from scratch on a small, specific dataset (say, a few thousand X-ray images), transfer learning starts from the pretrained network and adapts just the parts that need to change for the new task, usually replacing and retraining only the final layers, and often fine-tuning some or all of the earlier layers at a much lower learning rate. This is the exact same underlying principle behind the Pretraining Paradigms entry for language, general-purpose knowledge learned once, at enormous scale, transfers cheaply to countless specific downstream uses.
The Maths
Given a pretrained network's weights $\theta_{\text{pre}}$, fine-tuning simply continues gradient descent on the new task's loss $\mathcal{L}_{\text{new}}$, starting from $\theta_{\text{pre}}$ rather than a random initialisation:
with $\eta_{\text{fine}}$ typically much smaller than the original pretraining learning rate (to avoid destroying the useful pretrained features in just a few noisy updates). A common variant, feature extraction, freezes $\theta_{\text{pre}}$ entirely (setting its gradient contribution to zero) and only trains a small new "head" on top.
How It Works
Start from a model already pretrained on a large, general dataset.
Replace its final task-specific layer(s) with new ones matching the new task (e.g. a different number of output classes).
Either freeze the pretrained layers and train only the new head (feature extraction), or continue training some/all layers at a low learning rate (fine-tuning).
Train on the new, typically much smaller task-specific dataset.
Strengths
Achieves strong performance with far less task-specific data and compute than training from scratch.
General-purpose pretrained models (vision or language) are now widely and often freely available.
Weaknesses
"Catastrophic forgetting" is a real risk, fine-tuning too aggressively can erase the very general knowledge that made the pretrained model useful.
Works best when the new task is at least somewhat related to what the model was originally trained on.
Worked Example
A ResNet with 50 layers, pretrained on 1.2 million general images across 1,000 categories, needs to be adapted to classify just 3 types of factory defects from only 400 labelled photos.
Feature extraction approach: freeze all 50 pretrained layers, replace the final 1000-category output layer with a new 3-category one, and train only that new layer's weights on the 400 photos. Since only a tiny new layer is being trained, 400 examples is often plenty.
Fine-tuning approach: unfreeze the last 10 or so layers (which encode more task-specific features than the very general early layers) and continue training them, alongside the new output layer, at a learning rate perhaps 100× smaller than the original pretraining rate.
Result: either approach typically reaches strong accuracy on the defect-classification task using 400 images, something that would be hopeless training a 50-layer network entirely from scratch, since randomly-initialised deep networks generally need many thousands of examples per class just to learn basic edge and texture detectors that the pretrained model already has for free.
FRONT-02 · Modern Frontiers
Large Language Models & In-Context Learning
Scale a GPT-style Transformer far enough, and it starts learning new tasks just from examples in the prompt, no weight updates required.
“
Standard Definition
A Large Language Model is a Transformer-based model trained on massive text corpora at very large parameter scale; in-context learning refers to such a model's ability to adapt its behaviour to a new task purely from examples or instructions given in its input prompt, without any weight updates.
The Idea
Everything in the Attention & Transformers section of this guide, self-attention, multi-head attention, positional encoding, next-token prediction, scales up almost unchanged into modern LLMs; what changes is sheer scale: billions of parameters, trained on trillions of tokens, using enormous compute budgets. What emerges at this scale is genuinely surprising: models start exhibiting in-context learning, the ability to perform a brand new task described or demonstrated purely within the prompt (a handful of example input-output pairs, or even just an instruction), without any gradient update at all. This is fundamentally different from the Transfer Learning entry's fine-tuning, no parameters change; the model's fixed weights, combined with self-attention's ability to relate the prompt's examples to the current query, are enough on their own.
The Maths
Generation remains exactly the causal language modelling objective from the Pretraining Paradigms entry, autoregressive next-token prediction:
What's different at scale is empirical: "scaling laws" research has found that a model's loss follows a remarkably smooth, predictable power-law relationship with model size, dataset size, and compute, and that certain capabilities (like in-context learning) appear to emerge only once these scale past particular thresholds, rather than improving smoothly and gradually from the start.
How It Works
Pretrain a very large Transformer decoder on a massive, diverse text corpus using next-token prediction.
At inference time, provide a prompt containing instructions and/or a handful of example input-output pairs (few-shot prompting).
The model's self-attention relates the current query to the patterns demonstrated earlier in the same prompt.
Generate a continuation that applies the demonstrated pattern to the new query, entirely within a single forward pass, no training step involved.
Strengths
Adapts to new tasks instantly, at inference time, with no retraining or fine-tuning required.
A single model can flexibly handle an enormous range of tasks depending purely on the prompt.
Weaknesses
In-context learning is less reliable and more prompt-sensitive than a model explicitly fine-tuned for a specific task.
Training models at this scale requires enormous compute, data, and energy, concentrating the ability to build frontier LLMs in a handful of organisations.
Worked Example
A prompt demonstrates a task with two examples, then asks the model to continue the pattern:
No gradient update has occurred, the model's weights are frozen, exactly as they were after pretraining. But self-attention lets every token in "English: 'good night' → French:" attend back over the two demonstrated examples, picking up on the pattern "this is an English-to-French translation task" purely from their content and position in the prompt.
Result: the model completes the prompt with "bonsoir", correctly inferring and applying the translation pattern from just two examples, in a single forward pass, with zero parameter updates, a capability that simply doesn't exist in smaller Transformers trained the same way, it's an emergent property of scale.
FRONT-03 · Modern Frontiers
Graph Neural Networks
Not every input is a grid or a sequence. Sometimes the data itself is a network, and the model needs to be built around that.
“
Standard Definition
A Graph Neural Network is a neural network designed to operate directly on graph-structured data, learning node representations by repeatedly aggregating and combining information from each node's neighbours, a process known as message passing.
The Idea
Images have a fixed grid structure (which convolution exploits); sentences have a fixed sequential structure (which RNNs and Transformers exploit). But social networks, molecules, road networks, and knowledge graphs are fundamentally irregular, each node can have a different number of neighbours, with no natural "up/down/left/right" or "before/after" ordering. A GNN handles this by having every node repeatedly gather information from its immediate neighbours (message passing), combine it with its own current representation, and update itself, layer by layer, exactly analogous to how a CNN's receptive field grows with depth, a GNN node's "awareness" extends further across the graph with each additional message-passing layer, eventually incorporating information from nodes several hops away.
The Maths
A generic message-passing layer updates every node $v$'s representation $h_v$ by aggregating its neighbours $\mathcal{N}(v)$:
where AGGREGATE is a permutation-invariant function (mean, sum, or max are common choices, chosen specifically because a node's neighbours have no natural order to respect). Stacking $L$ such layers lets information propagate up to $L$ hops across the graph, directly analogous to how stacking convolutional layers grows a CNN's receptive field.
How It Works
Initialise every node with its own input features.
At each layer, every node gathers (aggregates) its neighbours' current representations.
Combine the aggregated neighbour information with the node's own representation, and update it.
Stack multiple layers so information propagates further across the graph with each one; use the final representations for node classification, link prediction, or whole-graph tasks.
Strengths
Directly models naturally graph-structured data (molecules, social networks, road networks) that grids or sequences distort.
The same learned aggregation function generalises across graphs of entirely different sizes and shapes.
Weaknesses
Deep GNNs suffer from "over-smoothing", after enough message-passing layers, every node's representation starts converging toward looking the same.
Can be computationally expensive on very large, densely connected graphs.
Worked Example
A 4-node star graph: node 0 (centre) connects to nodes 1, 2, and 3. Initial 2D features: $h_0=(1,0)$, $h_1=(0,1)$, $h_2=(2,1)$, $h_3=(1,2)$. Using mean aggregation and a simple 50/50 combination of self and neighbour information:
Result: node 0's new representation blends information from all three of its neighbours (1, 2, and 3) in a single message-passing step, while nodes 1, 2, and 3 each pick up a little of node 0's information. After just one more layer, information from nodes 1, 2, and 3 would begin reaching each other too, indirectly, via node 0, exactly how a GNN's "receptive field" grows with depth.
FRONT-04 · Modern Frontiers
Adversarial Examples & Robustness
A change too small for a human to notice can flip a confident, correct prediction into a confident, wrong one.
“
Standard Definition
An adversarial example is an input deliberately perturbed by a small, often imperceptible amount, specifically chosen to cause a neural network to produce an incorrect output with high confidence; robustness research studies both how to construct such perturbations and how to defend against them.
The Idea
Recall that backpropagation computes $\partial L/\partial x$, the gradient of the loss not just with respect to the network's weights, but with respect to the input too. That same gradient, ordinarily discarded once training finishes, reveals exactly which direction in input space would most increase the loss for a specific example. Nudge the input a tiny amount in that direction, small enough to be visually imperceptible to a human, and you can often flip the network's prediction entirely, and often with even higher confidence than the original correct prediction had. This exposes something unsettling: a network's decision boundary can sit remarkably close to seemingly typical inputs, in directions humans simply don't perceive as meaningful.
The Maths
The Fast Gradient Sign Method (FGSM), one of the simplest and most illustrative attacks, perturbs every input dimension by a fixed small amount $\epsilon$, in the direction of the sign of the loss gradient with respect to the input:
$$x_{\text{adv}} = x + \epsilon\cdot\text{sign}(\nabla_x L(x,y))$$
Using the sign rather than the raw gradient magnitude keeps every pixel's perturbation bounded to exactly $\pm\epsilon$, maximising the perturbation's effect on the loss for a given total perceptibility budget. A common defence, adversarial training, simply generates adversarial examples during training itself and includes them in the training set, teaching the network to be robust to exactly this kind of attack (though typically at some cost to accuracy on unperturbed inputs).
How It Works
Compute the gradient of the loss with respect to the input pixels, via ordinary backpropagation (stopping one step earlier than usual, at the input rather than the weights).
Take the sign of that gradient at every pixel (+1 or -1).
Add a small multiple of that sign pattern to the original input.
Feed the perturbed input to the network; despite looking essentially unchanged to a human, it can produce a confidently wrong prediction.
Why this matters
Exposes a fundamental gap between how neural networks and humans actually perceive similarity, with serious implications for safety-critical deployments (autonomous vehicles, security systems, medical imaging).
Open challenges
Most defences (including adversarial training) only guard against the specific attack types used during defence-development, novel attacks routinely defeat prior defences.
Robustness and standard accuracy often trade off against each other, more robust models frequently perform slightly worse on ordinary, unperturbed data.
Worked Example
A 4-pixel input $x=[0.5, 0.3, 0.8, 0.1]$, with loss gradient $\nabla_x L = [0.02, -0.05, 0.01, -0.03]$, and perturbation budget $\epsilon=0.05$.
Result: every pixel moved by exactly 0.05, in whichever direction increases the loss the most, an imperceptibly small, uniform nudge. Yet because this perturbation was computed precisely (not randomly) using the network's own gradient, applying it in real settings routinely flips predictions with high confidence, even though $x_{\text{adv}}$ would look identical to $x$ to any human observer.