LoRA: Low-Rank Adaptation

From attention weights to fine-tuning headaches

Dr. Lena Hartmann: Modern AI systems feel like magic partly because attention lets models dynamically mix information, but that power comes with massive weight matrices that are expensive to fine-tune.

Dr. Lena Hartmann: Today we will discover LoRA, low-rank adaptation: a linear algebra trick that lets you adapt a big model by training a small update instead of all the weights.

Dr. Lena Hartmann: We will start from what you already know from attention, then do a tiny NumPy experiment, then formalize what low-rank means, and end with a practical parameter-count check.

Kai: So this is basically: how do I customize a transformer without paying the full training bill?

Why full fine-tuning hurts: parameter explosion

Dr. Lena Hartmann: Before we talk about LoRA, let us quantify the pain of full fine-tuning: a single linear layer can already be millions of parameters once the model width is large.

Dr. Lena Hartmann: For a weight matrix W, the parameter count scales like d out times d in, so these projection matrices grow quadratically with the hidden size.

Kai: Just to sanity check: if the layer is 4096 by 4096, that is about sixteen million parameters for one matrix, right?

Dr. Lena Hartmann: And it is not just the weights: with Adam, you store two extra tensors per parameter, the first and second moment estimates, so that is about two times the parameter memory just for optimizer state.

Kai: So even if I can fit the model for inference, training can blow up because I am also storing gradients and optimizer state for every single weight entry.

Core idea: restrict updates to a low-rank subspace

Dr. Lena Hartmann: Today’s core idea is to adapt a big model by restricting the weight update to live in a low-rank subspace, so we only learn a small number of meaningful directions of change instead of freely changing every entry.

Kai: When you say low rank here, do you mean the update can be written as a product of two skinny matrices, so it is limited to a few degrees of freedom?

Dr. Lena Hartmann: In other words, LoRA does not change the base matrix directly; it learns an update that is deliberately constrained to be low-rank, which keeps the adaptation lightweight.

Dr. Lena Hartmann: You can think of this as saying: even if the matrix is huge, we assume the task only needs a small number r of adjustment directions, with r much smaller than the input and output dimensions.

Let me show you in code: low-rank update and parameter savings

Dr. Lena Hartmann: Before any more theory, let’s ground this with a tiny NumPy experiment that shows what a low-rank update is and why it saves parameters. We will keep the dimensions small so we can inspect shapes and ranks quickly.

Dr. Lena Hartmann: Before we run anything, make two predictions. If we set r to 4 and form delta W as B times A, what rank do you expect for delta W, and why should it be capped at that value?

Kai: I would expect the rank to be at most 4, because the product goes through a 4-dimensional bottleneck, even if the output matrix is 64 by 64.

Dr. Lena Hartmann: Second prediction: compare parameter counts. A full 64 by 64 weight matrix has 4096 parameters. With LoRA, we store A and B, so roughly r times (d in plus d out). With r equals 4, that is 4 times (64 plus 64) equals 512. So what reduction factor do you expect when we print full over LoRA?

Kai: 4096 divided by 512 is 8, so I would expect about an eight times reduction in trainable parameters for this layer.

Geometry: a low-rank update changes only a few output directions

Dr. Lena Hartmann: Before we look at any specific example, keep one picture in mind: low rank is about how many independent directions we can change, not how big the change can be.

Kai: So rank is like the number of new degrees of freedom I add to the layer, and magnitude is just how strongly I use them.

Dr. Lena Hartmann: A rank one update means the adjustment is built from one output direction times one input pattern, so the model gets new freedom along a single added direction in the output space.

Dr. Lena Hartmann: When you increase the rank to r, you are not changing everything everywhere; you are adding r new directions the layer can express, which is why this can be efficient while still being powerful.

Formal LoRA definition: freeze $W$, train $A$ and $B$

Dr. Lena Hartmann: Now we can state LoRA cleanly as a small change to a standard linear layer: we keep the original weight matrix fixed and add a trainable update on top.

Kai: And the update is the low-rank piece, so I only train the factors instead of the full matrix, right?

Dr. Lena Hartmann: The key idea is that the update is low rank, meaning we represent it as the product of two smaller matrices, scaled by a factor that controls the update’s strength.

Dr. Lena Hartmann: So during training, we freeze W and only learn these two new matrices, which greatly reduces the number of trainable parameters while still letting the model adapt.

How the forward pass changes (and what stays the same)

Dr. Lena Hartmann: In this part, we will connect the idea of a low rank weight update to the actual computation a layer performs on an input.

Kai: I care about this because it tells me whether LoRA adds latency, like extra matrix multiplies, during inference.

Dr. Lena Hartmann: Start with the usual forward pass: the layer multiplies the input vector by the weight matrix to produce the output vector.

Dr. Lena Hartmann: With LoRA, we keep the original matrix W, and we add a small learned correction that is built from two thin matrices. In the forward pass, that shows up as an extra term: alpha over r times B of A of x.

Where LoRA plugs into attention in practice

Dr. Lena Hartmann: In this part, we connect LoRA to what the model is actually doing inside a transformer block, and we focus on attention because that is where most practical LoRA adapters get added.

Kai: Is the idea that I would add LoRA to the query and value projections first, since that tends to change behavior most, and leave other parts frozen?

Dr. Lena Hartmann: The key idea is simple: instead of retraining the whole model, we modify a small set of weight matrices that shape how queries, keys, values, and outputs are produced in attention.

Dr. Lena Hartmann: So when you hear that LoRA plugs into attention, what that usually means in practice is adding low rank updates to the attention projection weights, because a few new feature directions can be enough to shift behavior.

Worked check: merging LoRA equals explicit low-rank path

Dr. Lena Hartmann: On this slide, we will do a quick numerical check of a key idea in LoRA: two different-looking computations can represent the same linear update.

Kai: So we are verifying that doing the two-step path during the forward pass matches what you get if you precompute the update once and add it into the weight matrix.

Dr. Lena Hartmann: Before we look at the result, make a prediction: if we compute y using the explicit low rank path and also compute y using a single merged weight matrix, will the maximum absolute difference be exactly zero, or just very small?

Kai: I would expect it to be very small, not necessarily exactly zero.

Dr. Lena Hartmann: Exactly. When you run the code, the printed max absolute diff should come out extremely close to zero, like around 1e-6 to 1e-12. That is the telltale sign that the two expressions are algebraically the same, but floating point associativity and rounding make the last few bits differ depending on how the multiplications are grouped.

Choosing $r$ and $\alpha$: constraints vs capability

Dr. Lena Hartmann: LoRA is simple, but the knobs matter because they control how expressive your update space is. In practice, you’re balancing two goals at once: keeping the adaptation lightweight, while still giving it enough freedom to learn what your task needs.

Kai: If I double the rank, I roughly double the adapter parameters, so I should treat rank like my main capacity lever.

Dr. Lena Hartmann: The first knob is rank r, which sets the capacity of the weight update, delta W. If r is too small, you can’t represent the needed change and you underfit; if it’s too large, you start giving up the efficiency and memory savings that made LoRA attractive.

Dr. Lena Hartmann: The second knob is alpha, a scale that controls the overall magnitude of the update. Higher values can make learning faster but can also make training less stable, while smaller values tend to be steadier but may adapt more slowly. And alongside those, regularization choices like LoRA dropout or only adapting selected layers can help you keep capability without overfitting.

LoRA in one sentence (and one mental image)

Dr. Lena Hartmann: Let us condense the whole lecture into a tight summary you can reuse when reading papers or debugging fine-tunes.

Kai: I want a one-liner that I can repeat to my team, plus a mental model for why it works.

Dr. Lena Hartmann: Hold onto one sentence and one picture in your head: you keep the original weights fixed, and you only learn a small change to them.

Dr. Lena Hartmann: That change is constrained to be low rank, meaning it is built from two skinny matrices whose product recreates the update, so you store and train far fewer parameters.

Exit ticket: can you count the savings?

Dr. Lena Hartmann: Time to test the core skill: can you translate LoRA into a concrete parameter-count win? You will do a quick back-of-the-envelope comparison between training a full weight matrix and training a low-rank update, using the sizes given.

Kai: So I should count parameters for the full matrix, then count the LoRA parameters, and compare them as a ratio or savings?

Dr. Lena Hartmann: Exactly. First, compute the full count as d out times d in. Then for LoRA, count the two low-rank factors: one with r times d in parameters and one with d out times r parameters. Finally, decide when that low-rank change might be too limiting for the kind of adaptation you need.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!