Singular Value Decomposition

From PCA to $A = U\Sigma V^T$

Dr. Lena Hartmann: Before we dive in: this matters because almost every time we compress a model, reduce embedding dimension, or do fast retrieval, we are secretly using the same geometric tool. Today we are going to discover Singular Value Decomposition, usually called SVD. The core idea is simpler than it sounds: any matrix can be seen as rotate, then stretch, then rotate. We will start from PCA intuition, then do it in NumPy, then connect it back to AI workflows.

Kai: So PCA was about variance directions. Are you saying SVD is the thing underneath PCA?

Dr. Lena Hartmann: Exactly. Last time, we used eigenvectors to find the most important directions in the data matrix.

Dr. Lena Hartmann: But in real AI, the dimension can be thousands, and explicitly forming the covariance matrix can be expensive and can amplify numerical issues. We want a more direct tool that works well at scale.

Dr. Lena Hartmann: That tool is SVD: it gives you the key directions and the stretch amounts in one shot, and it will explain PCA and low-rank compression in the same language.

Why SVD shows up everywhere in AI

Dr. Lena Hartmann: When you hear model compression, dimensionality reduction, or low-rank adapters, think: we are throwing away tiny directions and keeping big ones.

Dr. Lena Hartmann: SVD gives a ranked list of stretch strengths, the singular values. Keeping the top ones often preserves most behavior with far fewer numbers.

Kai: Is this like saying a weight matrix has a few important directions, and the rest is small noise?

Dr. Lena Hartmann: Yes. And SVD is the clean way to write that idea as a structured approximation, which is why it keeps reappearing in AI engineering.

Geometric picture: rotate, stretch, rotate

Dr. Lena Hartmann: Here is the whole promise on one line: any matrix can be factored into three simple pieces.

Dr. Lena Hartmann: This equation says: take your matrix A, and rewrite it as U times Sigma times V transpose. Think of that as rotate, stretch, rotate.

Kai: So V transpose chooses the input directions that matter most?

Dr. Lena Hartmann: Exactly. Along those special input axes, the matrix does not shear or mix directions: it only scales by the singular values, then U just re-orients the result in output space.

Dr. Lena Hartmann: And that last piece, U, is the final rotation: after Sigma stretches the coordinates, U rotates the result into the correct output orientation.

Let me show you in code: computing SVD in NumPy

Dr. Lena Hartmann: Before more formulas, let me show you what you actually run. We ask NumPy for the SVD of a simple two by two matrix.

Dr. Lena Hartmann: Before you look at the output, make a prediction: will the reconstruction error be exactly zero, around 1e minus 15, or noticeably larger? Why? Then we call numpy linalg svd of A, build Sigma, multiply U times Sigma times V transpose, and measure that reconstruction error.

Kai: So if the error is basically zero, that means the factorization is exact, just with floating point rounding?

Dr. Lena Hartmann: Right. You usually see something around 1e minus 15 because of floating point rounding, not because the SVD failed. And those printed singular values are the key summary: they tell you the strength of the transformation along the best-aligned directions.

Visualize it: a circle becomes an ellipse

Dr. Lena Hartmann: Now for the geometry: a linear map is easiest to understand by watching what it does to simple shapes.

Dr. Lena Hartmann: Before we plot anything, make a prediction: when we apply A to every point on the unit circle, will the result stay a circle, become an ellipse, or do something else? And which direction do you think will be the longest?

Kai: So the longest radius of that ellipse corresponds to the biggest singular value?

Dr. Lena Hartmann: Yes. The major axis length matches the largest singular value, and the minor axis matches the smaller one. The SVD is basically naming the ellipse: V gives you the input directions that become the ellipse axes, and the singular values give the axis lengths.

Key fact: singular values measure maximum stretch

Dr. Lena Hartmann: Now that the ellipse picture is in your head, we can say it precisely: the top singular value is the biggest stretch A can apply to any unit vector.

Dr. Lena Hartmann: This equation says: among all vectors of length one, pick the one that makes the output length as large as possible. That maximum length is sigma one.

Kai: Where do the directions come from? How do we find the vector that achieves that max?

Dr. Lena Hartmann: This second equation answers it: the right singular vectors are eigenvectors of A transpose A, and the eigenvalues are the singular values squared. That is why S V D is tied to eigen stuff, but works even when A is not square.

What SVD guarantees (dimensions $+$ orthogonality)

Dr. Lena Hartmann: Here is the clean, general statement: SVD works for any real matrix, even rectangular ones.

Dr. Lena Hartmann: We will use this one equation, A equals U Sigma V transpose, to keep both the geometry and the bookkeeping straight. First we will focus on the reduced form: it reconstructs A while keeping only r singular directions. Then we will contrast it with the full form, where U and V become square and the orthogonality identities match those square sizes.

Kai: Orthonormal bases means those columns are like perpendicular unit directions, right?

Dr. Lena Hartmann: Exactly. In the reduced form, U and V have orthonormal columns, so U transpose times U and V transpose times V equal the r by r identity. In the full form, U and V are square, so those products equal the m by m and n by n identities. Either way, U and V do pure rotations or reflections, and Sigma does pure scaling along independent axes. The nonnegative, sorted singular values are what make SVD so useful for ranking importance.

Worked example: best rank-$k$ approximation

Dr. Lena Hartmann: Now the payoff: once you have SVD, you can build a rank k approximation by keeping only the first k singular directions.

Dr. Lena Hartmann: This equation says: keep only the first k columns of U and V, and the top k singular values in Sigma. That gives a simpler matrix A k.

Dr. Lena Hartmann: In the code, we generate a noisy low rank matrix, compute its SVD, reconstruct using k equals three, and print the relative error.

Kai: So if the singular values shrink fast after the first few, we can keep just those and lose very little?

Dr. Lena Hartmann: Exactly. A big drop is telling you the matrix is effectively low rank, which is the mathematical reason compression works.

Dr. Lena Hartmann: And that is the key practical rule: truncate the small singular values. Dropping those tiny components removes mostly noise and gives you a compressed representation with little loss.

SVD is PCA without extra work

Dr. Lena Hartmann: Let us close the loop with last lecture: PCA was about finding the best directions in your data.

Dr. Lena Hartmann: If you center your data matrix X and take its SVD, you get U, Sigma, and V transpose for X directly.

Kai: And V gives the principal components, like the axes we project onto?

Dr. Lena Hartmann: Yes. The squared singular values control how much variance each component captures, because they match eigenvalues of X transpose X up to scaling. If X is centered and the rows are samples, the sample covariance is one over n minus one times X transpose X, so each explained variance value is sigma i squared over n minus one. So SVD is the practical route to PCA.

AI connection: low-rank weights and adapters

Dr. Lena Hartmann: Here is the engineering translation: if a weight matrix is close to low rank, you can represent it with far fewer parameters.

Dr. Lena Hartmann: This equation is the idea: approximate W using only the top k singular directions. That is a principled version of keep the important stuff.

Kai: LoRA uses two small matrices. Is that basically trying to learn something like U times Sigma times V transpose, but as an update?

Dr. Lena Hartmann: That is the right mental model: LoRA enforces low rank structure on the change to the weights, and SVD tells you why low rank updates can be expressive when the action lives in a few dominant directions.

Trust but verify: orthogonality $+$ reconstruction (NumPy and PyTorch)

Dr. Lena Hartmann: Whenever you learn a decomposition, you should verify the claims with code, especially in high dimensional AI where numerical issues are real.

Dr. Lena Hartmann: This script checks three things: U transpose times U is identity, Vt times Vt transpose is identity, and the product reconstructs A. Here Vt is V transpose, so Vt times Vt transpose corresponds to the reduced form orthogonality, V transpose V equals identity.

Kai: If these checks fail, would that mean I messed up shapes, like full matrices versus reduced?

Dr. Lena Hartmann: Exactly. The most common bug is mixing the reduced and full forms. And if you are in deep learning code, torch linalg svd gives the same components with tensor types.

What you should remember about SVD

Dr. Lena Hartmann: Let us compress the whole lecture into three ideas you can use tomorrow.

Dr. Lena Hartmann: First: think geometrically. Any matrix is a rotation, then independent axis scaling, then another rotation.

Kai: Second: the singular values are like a sorted importance list of directions?

Dr. Lena Hartmann: Yes. And third: truncating that list gives the best low rank approximation, which is why SVD underlies PCA and many compression tricks in AI.

Exit ticket: choose $k$ from singular values

Dr. Lena Hartmann: Time to make a practical decision like you would in compression: how many components k do we keep?

Dr. Lena Hartmann: This formula defines an energy ratio: you square singular values, sum the first k, and divide by the total. That is a common rule of thumb for choosing k.

Dr. Lena Hartmann: For sigma equal to 5, 2, 1: the total squared energy is 25 plus 4 plus 1 equals 30. With k equals 1, you get 25 over 30, about 0.833, not enough. With k equals 2, you get 29 over 30, about 0.967, so the smallest k is 2.

Kai: When would low rank fail even if we keep a lot of components?

Dr. Lena Hartmann: If the singular values decay slowly, there is no clear cutoff, so compression hurts. Also, even with decent energy, the directions you drop might be small but crucial for rare behaviors, like safety edge cases or specific prompts.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!