Mechanistic Interpretability

From LoRA's low-rank change to "what circuit changed?"

Dr. Lena Hartmann: Every time a model surprises you in production, the real question is not just what it outputs, but what internal computation caused it. Today we are going to discover mechanistic interpretability: treating a network like a machine made of vectors and matrices that we can test, modify, and verify. The good news is we start from familiar linear algebra moves: dot products, matrix multiplies, and a bit of geometry. We'll bridge from last lecture's LoRA idea into three concrete tools: read, write, and intervene on features.

Kai: So instead of just measuring accuracy, we want to point to a specific direction or subspace in the model and say: that is the thing doing the job?

Dr. Lena Hartmann: Exactly. In LoRA, we freeze the original weight matrix W and add a low-rank update Delta W equals B A: A compresses a d-dimensional hidden vector down to r, and B expands it back to d, so only r directions can be changed. Mechanistic interpretability asks: which directions, where, and what behavior do they implement?

Dr. Lena Hartmann: Keep that phrase in mind: few directions. We'll reuse it for activations too, not just weights.

Dr. Lena Hartmann: Today we shift the question: instead of which directions changed in W, we ask which directions changed in the activations. Activations are vectors in R to the d. If a new behavior appears, it should show up as movement along particular activation directions or a small subspace we can identify, measure, and even intervene on.

Kai: So the goal is to connect a behavior change to a specific direction in activation space, not just to a weight update?

Mechanistic interpretability in one sentence

Dr. Lena Hartmann: Mechanistic interpretability is like debugging, but your variables are vectors and your functions are matrices. We want an explanation that lives inside the computation, not just a story on top of it.

Dr. Lena Hartmann: This first bullet is the mindset shift: the output is produced by internal activation vectors and weight matrices, so our explanation should reference those objects directly.

Kai: Why is everyone obsessed with causal tests here? Can't we just find a neuron that correlates with something?

Dr. Lena Hartmann: Correlation is a hint, not a guarantee. The second bullet is the standard: you intervene, and the behavior changes in the predicted way. That is what turns a pattern into an explanation.

Dr. Lena Hartmann: And the third bullet is what we actually look for in practice: directions, subspaces, and multi layer circuits that you can describe and test.

Why linear algebra shows up everywhere inside networks

Dr. Lena Hartmann: Before we look at any fancy tooling, let's name the objects. Inside a model, you keep seeing the same template: take a vector, multiply by a matrix, apply a nonlinearity, repeat.

Dr. Lena Hartmann: This equation is our working picture: an input vector becomes an activation vector, and then another linear map produces outputs. Mechanistic interpretability often studies the activation vector in the middle, because it is a bottleneck you can read and edit.

Kai: So even if the whole network is nonlinear, locally we're still constantly doing matrix times vector in some space?

Dr. Lena Hartmann: Yes. And this second bullet is the key geometric idea: a row vector acts like a detector. If the dot product with the input is large, that neuron turns on. That is already a candidate for a feature.

Worked example: a tiny network with interpretable hidden units

Dr. Lena Hartmann: Let me show you in code. We will build a tiny hidden layer with two neurons, and we will make the weights so each neuron cleanly corresponds to one input feature. This is the simplest possible situation where interpretability feels obvious.

Dr. Lena Hartmann: In this code, look at the weight matrix: each row picks out one coordinate. When we run inputs through and apply relu, neuron zero fires when the first coordinate is positive, and neuron one fires when the second coordinate is positive.

Kai: This feels too clean compared to real models. In practice the weights won't be one hot like that, right?

Dr. Lena Hartmann: Totally. In this toy linear layer, each neuron is a clean detector, so its row vector acts like a feature direction for that neuron. But in real models the rows mainly label neurons and their pre activations, and features are often best thought of as directions in activation space that we recover with tools like probes or principal component analysis, rather than reading rows directly. Mechanistic interpretability is what we do when the feature is a messy direction, not a coordinate axis.

Reading a feature with a linear probe (code first)

Dr. Lena Hartmann: Now let's do the first real interpretability move: reading. We'll make up activation vectors and a label that depends on a hidden direction, then try to recover that direction from data with least squares.

Dr. Lena Hartmann: Prediction checkpoint before we look at the output: since the true direction is one, zero, should the learned probe point mostly along the first axis? And should the cosine similarity with the true direction be closer to one or closer to zero?

Kai: So the probe gives me a direction, but does that direction mean the model is using it, or just that it contains information?

Dr. Lena Hartmann: Now we reveal the printouts: if the probe is doing its job, w probe should look roughly like one, zero up to scaling, and the cosine similarity should be near one. If it came out near zero, that would mean the probe direction is basically unrelated to the true feature. If it came out negative, it found the same feature but flipped sign. These bullets are about reading: a probe says the information is linearly present. To argue the model uses it, we need the next tool: interventions.

Dr. Lena Hartmann: One more practical detail: that dot product score is a single number you can interpret as a linear readout. Higher means the activation points more along w, lower means it points against w, and you can turn it into a prediction with a threshold or by comparing scores across classes.

The geometry: features are directions, not single neurons

Dr. Lena Hartmann: Here is the mental model that scales beyond toy networks. In big models, a feature is rarely one neuron. It is a direction in a high dimensional space, spread across many coordinates.

Dr. Lena Hartmann: This first bullet says what we mean by a feature: a vector v. If you project the activation onto v and it is large, that feature is present, even if no single coordinate looks special.

Kai: And the subspace idea is like: instead of one direction, we track a whole plane of related concepts?

Dr. Lena Hartmann: Exactly. And the last bullet is the AI reason: because d is huge, the model can pack many features into overlapping coordinates. That packing is often called superposition, and it is why linear algebra language is so natural here.

The causal move: intervene on an activation direction

Dr. Lena Hartmann: Now we earn the word mechanistic. We do not just read a direction; we write to it. That means we intentionally add a vector to the activation and see what the model does next.

Dr. Lena Hartmann: This equation is the whole intervention idea: take the current activation and add alpha times v. If v truly corresponds to a feature, increasing alpha should increase that feature's effect downstream.

Kai: So alpha is like a knob. If turning it changes the right behavior, that is evidence we found something real?

Dr. Lena Hartmann: Yes. The first bullet is the procedure, and the second bullet is the standard of proof: not just any change, but a change that matches the meaning you think v represents.

Worked example: intervention changes the output in a predictable way

Dr. Lena Hartmann: Let me show you a clean causal check. We pretend the model's next decision is controlled by a single logit computed as a dot product between an output weight vector and the current activation.

Dr. Lena Hartmann: We will sweep alpha and add alpha times v to the activation. Before we run it, make a concrete prediction: what logit do you expect at alpha equals 2, compared to the baseline?

Kai: If v changes only the first coordinate, and the output weight on that coordinate is 2, then the logit should go up by 4 when alpha is 2, so it should be baseline plus 4.

Dr. Lena Hartmann: Exactly. When we print the results, you should see a straight line: each step of alpha increases the logit by 2 times alpha relative to baseline, and any tiny mismatch would just be floating point rounding. These bullets name the math: the logit is a dot product, and the intervention shift is proportional to the dot product between the output weights and your direction.

Connecting back to LoRA: low-rank structure as "few feature directions"

Dr. Lena Hartmann: Now we close the loop to last lecture. LoRA changes a weight matrix in a low rank way. A useful picture is: a matrix can be described by a few important directions, each with a strength.

Dr. Lena Hartmann: In the S V D, U and V give you orthonormal direction sets for the output and input, and Sigma contains the strengths along those directions, called singular values.

Kai: So where does the low rank part come in?

Dr. Lena Hartmann: Low rank means you can approximate W by keeping only the top few strengths and their matching direction pairs, which looks like a small sum of rank one pieces. That is the bridge to LoRA and interpretability: LoRA updates a small set of directions, and interpretability asks what behaviors those dominant directions correspond to.

Code: inspect a matrix via its top singular directions

Dr. Lena Hartmann: Let me show you S V D in the most interpretability flavored way: we construct a matrix that secretly has one dominant direction, then we recover that direction using numpy linalg svd.

Dr. Lena Hartmann: In the printouts, the singular values tell you how many strong directions there are. When the top singular value is much larger than the rest, a rank one approximation captures most of the matrix, and the relative error becomes small.

Kai: How does this help in a real model? We are not interpreting a random matrix; we want a behavior explanation.

Dr. Lena Hartmann: These bullets give the workflow: if one direction dominates, the matrix mostly pushes inputs aligned with v1 toward outputs aligned with u1. Then you can do interventions: feed in vectors along v1, or edit activations along u1, and measure the downstream change.

Dr. Lena Hartmann: That is the same read and write loop as before, now applied to a weight matrix rather than an activation space.

A practical checklist: from a hypothesis to a verified circuit

Dr. Lena Hartmann: Let's compress the whole lecture into a workflow you can actually run. Mechanistic interpretability is not one trick; it is a loop from hypothesis to test.

Dr. Lena Hartmann: Step one is choosing a behavior you care about, something you can measure repeatedly. If you cannot measure it, you cannot test causality.

Dr. Lena Hartmann: Step two is reading: recover candidate directions using linear algebra tools. Probes and P C A live in activation space; S V D lives in weight space; gradients give another way to propose a direction.

Kai: And step three is the make or break: edit the activation and see if the behavior shifts the way you predicted.

Dr. Lena Hartmann: Right. Step four is localization: you want to pinpoint where that direction lives in the model, so you can say not just that the feature exists, but which component implements it.

Exit ticket: can you predict an intervention effect?

Dr. Lena Hartmann: Time to lock it in with a tiny symbolic exercise. You will compute how a logit changes when we add a direction to an activation.

Dr. Lena Hartmann: For the practice: replace a prime into the dot product. You get p prime equals w dot a plus alpha times w dot v. So the change p prime minus p is alpha times w dot v. That single dot product tells you whether the intervention matters downstream.

Kai: So if w dot v is near zero, I can crank alpha and nothing happens, which means my direction is probably not on the causal path to p.

Dr. Lena Hartmann: For the reflection: a good answer mentions an intervention that reliably changes the targeted behavior, plus controls that show it is specific and not a side effect. That is the difference between information being present and information being used.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!