The Attention Mechanism

From fixed weights $(L12)$ to context-dependent weights (today)

Dr. Lena Hartmann: Before we dive in: the stakes are huge. Attention is the trick that lets a model decide what to focus on, which is basically why Transformers scale to long context and become useful in real products. By the end, you will be able to write scaled dot product attention in NumPy and explain every matrix in it. And we will build it from things you already know: matrix multiplies and dot products.

Dr. Lena Hartmann: Here is the bridge from L12: a neural network layer is a fixed linear map plus a nonlinearity. Once training is done, the weight matrix is fixed.

Kai: So attention changes that? Like the effective weights change depending on the input?

Dr. Lena Hartmann: Exactly. The goal today is to see how a few matrix multiplications produce data dependent mixing weights, so each token can pull information from other tokens differently, depending on the context.

Attention as learnable retrieval over a set of vectors

Dr. Lena Hartmann: Think of attention like a tiny retrieval system inside the network. You have a set of vectors available, and for each position you generate a query that decides what to pull in.

Dr. Lena Hartmann: First, imagine the things you can retrieve are value vectors. They are the information you might want to copy or mix into your current representation.

Kai: So the query is like a search vector, and the model learns how to make good searches?

Dr. Lena Hartmann: Yes. And the output is not a hard pick like nearest neighbor; it is a weighted sum. The weights alpha tell you how much each value contributes.

The $Q-K-V$ pattern: three matrices from one input

Dr. Lena Hartmann: Now let me show you the core design pattern. We start with the sequence packed into a matrix X: one row per token, one column per feature.

Dr. Lena Hartmann: Visually, this first bullet is just bookkeeping: X lives in n by d space, meaning n tokens and d features per token.

Dr. Lena Hartmann: Next, we apply three different learned linear transformations to X to get Q, K, and V. Same input, three different projections.

Kai: Why not just use X directly? Why split into three versions?

Dr. Lena Hartmann: Because it lets the model learn different geometry for matching versus content. Keys are optimized for being matched against queries, while values are optimized for the information you want to move around.

Matching is dot products: who is relevant to whom?

Dr. Lena Hartmann: Here is the linear algebra heart: relevance is a dot product. Each query vector compares against each key vector.

Dr. Lena Hartmann: This equation says the score between token i and token j is the dot product of query i with key j. If you stack all queries and keys, you get the full score matrix by Q times K transpose.

Kai: So this is basically cosine similarity, but without dividing by norms?

Dr. Lena Hartmann: It is the same alignment intuition, but it is a raw dot product, so it is sensitive to vector lengths. Cosine similarity would also normalize by the norms. In practice, the model can learn appropriate scaling through the projections, and we will also add an explicit scaling factor later.

Dr. Lena Hartmann: And the second bullet is the big picture: S is an n by n table of who attends to whom, one row per query token.

Softmax turns scores into weights you can average with

Dr. Lena Hartmann: Dot products give raw scores, but we want weights that behave like proportions. Softmax is the standard way to do that.

Dr. Lena Hartmann: This equation says: for each query row i, take softmax across the scores in that row to get attention weights across all j positions.

Dr. Lena Hartmann: Row wise matters: each token i gets its own distribution over where to look.

Kai: And the sum to one constraint is why people say attention is like probabilities, right?

Dr. Lena Hartmann: Right. Not probabilities of truth, but a clean way to express a convex mixture: nonnegative weights that add up to one. That makes the next step a weighted average of values.

Scaled dot-product attention (the full formula)

Dr. Lena Hartmann: Now we can name the destination: this is scaled dot product attention, the exact computation used inside Transformers.

Dr. Lena Hartmann: Read this as a pipeline: first compute Q times K transpose, scale it, apply softmax to get weights, then multiply those weights by V to mix value vectors into new outputs.

Kai: So the final result is still a matrix, one output vector per token?

Dr. Lena Hartmann: Exactly. Y has n rows: each row is a context aware vector for that token, built as a weighted sum of value rows.

Kai: If the scores get too peaky, does that mean the model almost always copies from just one token and stops blending information?

Dr. Lena Hartmann: Yes, that can happen. As the key dimension grows, dot products tend to grow in magnitude, so softmax can become too peaky. Dividing by the square root of d k keeps gradients healthier.

Dr. Lena Hartmann: And notice how linear algebraic this is: just two matrix multiplications plus one nonlinearity, softmax, in the middle.

Let me show you in code: attention in NumPy (and PyTorch)

Dr. Lena Hartmann: Let me show you how attention works end to end on a tiny toy example, so we can follow the scores, the weights, and the final output step by step.

Dr. Lena Hartmann: Before we print anything, make a quick prediction. For the first query vector, which key do you think gets the largest score: key one, key two, or key three? Then we will run the code and compare to the printed scores and weights.

Dr. Lena Hartmann: This first bullet is the mental map: Q times K transpose produces the relevance table.

Kai: And softmax is what makes each row comparable, so the model can form a clean mixture?

Dr. Lena Hartmann: Exactly. Now, let us resolve the prediction for row zero. The printed scores for the first query have their maximum at key two, because the dot product with key two is 1 while key one and key three are 0, and scaling just divides them all by the square root of d k. Softmax then turns that into the largest weight on key two, with smaller but nonzero weights on the others, and that is why the first output row is mostly value two with a little contribution mixed in from the other values.

Visualize it: attention weights are a heatmap

Dr. Lena Hartmann: Attention becomes intuitive when you look at the weight matrix as an image: rows are queries, columns are keys, and brightness is how much gets copied.

Dr. Lena Hartmann: This code makes a heatmap with numbers inside the cells. If a row has one bright cell, that token is focusing sharply on one other token; if a row is spread out, it is averaging information.

Kai: So when people show attention maps, they are literally plotting that weight matrix?

Dr. Lena Hartmann: Yes. This bullet is the key readout: row i tells you where token i looks. And the next bullet is how to read a single cell: bright means token j strongly influences token i through the value vectors.

Dr. Lena Hartmann: A common pitfall: the weights are not the final content, they are just mixing coefficients. The content that moves is in V.

Multi-head attention: several geometries in parallel

Dr. Lena Hartmann: Single head attention is already powerful, but Transformers typically run several attention computations in parallel, called heads.

Dr. Lena Hartmann: This equation says head h runs the same attention pipeline, but it starts by projecting the same input X with its own learned matrices: W Q for head h, W K for head h, and W V for head h. So each head lives in its own learned similarity space.

Dr. Lena Hartmann: That is why the first bullet matters: different heads literally mean different dot products, because the projections change the vectors before matching.

Kai: So each head can learn a different kind of relevance, and the model combines them?

Dr. Lena Hartmann: Exactly. In practice, one head might track nearby tokens, another might lock onto a named entity, another might focus on punctuation cues. Then concatenation merges those views into one representation per token.

AI connection: why attention unlocked Transformers

Dr. Lena Hartmann: Now let us connect back to modern AI systems. The reason attention matters is not just math elegance; it changes what architectures can do efficiently.

Dr. Lena Hartmann: Self attention means every token can pull context from other tokens to update its representation, instead of relying on a fixed local window or a sequential pass.

Dr. Lena Hartmann: This second bullet is a performance story: computing Q times K transpose is a big matrix multiply, which GPUs are extremely good at. That is why Transformers parallelize so well.

Kai: Is that why people look at attention maps when debugging a model? Like, you can see what it focuses on?

Dr. Lena Hartmann: Yes, as a diagnostic. It is not the whole story, but it is one of the few internal objects that is easy to visualize. And the last bullet is the architectural recipe: attention blocks plus feedforward blocks stack to form the Transformer backbone.

One more crucial detail: masking (who is allowed to look where?)

Dr. Lena Hartmann: If you are thinking about language models, there is one missing piece: the model must not peek at future tokens during next token prediction. That is handled by masking.

Dr. Lena Hartmann: This equation shows the mask matrix M added to the scores before softmax. The mask does not change V; it changes which attention weights can become nonzero.

Dr. Lena Hartmann: Causal masking is the rule for autoregressive text generation: token i can only look at tokens up to i, not beyond.

Kai: And padding mask is more like a data cleanliness thing, so padding does not steal attention weight?

Dr. Lena Hartmann: Exactly. In code, the usual trick is to add a huge negative number to masked score entries, so after softmax they become essentially zero weight.

Checkpoint: can you narrate the pipeline without formulas?

Dr. Lena Hartmann: Quick checkpoint. If you can say the pipeline in plain English, you understand the mechanism well enough to implement it and debug it.

Dr. Lena Hartmann: Step one: turn the input matrix into queries, keys, and values using learned linear projections.

Dr. Lena Hartmann: Step two: compute dot product scores, then softmax each row to get a distribution over where each token should look.

Kai: Step three: use those weights to average the value vectors, giving each token a context aware update.

Exit ticket: compute a tiny attention step

Dr. Lena Hartmann: Exit ticket time. We will do one tiny numeric softmax and interpret what it means geometrically, then reflect on the scaling factor.

Dr. Lena Hartmann: For the practice: softmax of two numbers means exponentiate them and normalize. Exponent of 2 is about 7.39, exponent of 0 is 1. So the weights are about 7.39 over 8.39 and 1 over 8.39, which is about 0.88 and 0.12. The output vector y is mostly v one with a small contribution from v two.

Kai: So a two point attention is just a smooth choice between two options, not an all or nothing switch.

Dr. Lena Hartmann: For the reflection: dividing by the square root of d k keeps dot product magnitudes from exploding as the dimension grows, so softmax does not saturate too early. That stabilizes training and keeps gradients usable.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!