PCA: Dimensionality Reduction

From eigenvectors to PCA directions

Dr. Lena Hartmann: This equation is the L07 idea: vector v is a direction that A only stretches by lambda.

Why PCA matters in AI systems

Dr. Lena Hartmann: Embeddings are model-produced feature vectors. If each query embedding is seven hundred sixty-eight dimensional, shrinking to one twenty-eight can cut memory and network costs a lot.

Dr. Lena Hartmann: Approximate nearest neighbor search still relies on many inner products, and fewer dimensions means fewer multiplies.

Dr. Lena Hartmann: Even when you do not deploy PCA, plotting the first two principal components is a great sanity check on whether data separates.

Dr. Lena Hartmann: The directions with tiny variance are often noise; removing them can actually improve downstream learning.

Let me show you in code: PCA from covariance $+$ eigenvectors

Dr. Lena Hartmann: Centering is crucial: PCA cares about variation around the mean, not absolute position in space.

Dr. Lena Hartmann: The covariance matrix summarizes how features vary together; symmetry is why we use the stable eigh routine.

Kai: So the biggest eigenvalue literally means the data spreads out most along that eigenvector?

Dr. Lena Hartmann: Exactly. PCA ranks directions by how much variance they capture, and eigenvalues give that ranking.

Geometric picture: the cloud chooses its own axes

Dr. Lena Hartmann: Visually, PC one is the direction that makes the projection of points as spread out as possible.

Dr. Lena Hartmann: The second component must be orthogonal to the first, so it captures the leftover variation.

Kai: So if I keep just PC one, I'm squashing the whole cloud onto that indigo line?

Dr. Lena Hartmann: Yes, and the key is: among all possible lines, that particular line loses the least information in a variance sense.

Key insight: maximize variance of the projection

Dr. Lena Hartmann: This equation says: pick a unit direction u, project every centered point onto it, and the variance of those scalars becomes u transpose S u.

Dr. Lena Hartmann: Think of z as the coordinate of each point along the chosen axis u.

Dr. Lena Hartmann: Without the unit length constraint, you could make variance arbitrarily large by just scaling u, which is not a meaningful choice of direction.

Kai: Wait, why does maximizing that expression force u to be an eigenvector?

Dr. Lena Hartmann: Try a quick prediction first: if S is diagonal with entries 9 and 1, which unit vector makes u transpose S u as large as possible?

Kai: I would pick u equals (1, 0), so all the weight is on the 9 direction, not the 1 direction.

Dr. Lena Hartmann: Yes, u equals (1, 0) gives u transpose S u equal to 9, and u equals (0, 1) would only give 1. That is exactly what the Rayleigh quotient is doing here: it picks the direction that puts as much weight as possible on the largest variance.

Dr. Lena Hartmann: In the general case, you can rotate into the eigenvector basis where S becomes diagonal, and u transpose S u becomes a weighted average of the eigenvalues with weights that sum to 1. That Rayleigh quotient is maximized by putting all the weight on the largest eigenvalue, so the maximizing u is the top eigenvector, and the maximum value equals the top eigenvalue.

Formal PCA definition (and why eigenvectors show up)

Dr. Lena Hartmann: This is the standard covariance construction: transpose times the centered data, scaled by one over n.

Dr. Lena Hartmann: This is the PCA objective: among unit directions, pick the one with largest projected variance.

Kai: So once we find those eigenvectors and sort by eigenvalue, we are done, and the ordering tells us which directions matter most?

Dr. Lena Hartmann: The principal components are eigenvectors u i, ordered by eigenvalues: biggest lambda means most variance captured.

Dr. Lena Hartmann: To reduce to k dimensions, we keep the first k eigenvectors as our new coordinate axes.

In practice: PCA via SVD (stable and fast)

Dr. Lena Hartmann: This factorization is the rotate stretch rotate story: V gives directions in feature space, and Sigma contains the strengths along those directions.

Dr. Lena Hartmann: To get k dimensions, just take the first k columns of V; that matrix becomes your projection basis.

Kai: So the singular values are basically telling me how much energy each component has?

Dr. Lena Hartmann: Exactly. Squared singular values, scaled by one over n, match the covariance eigenvalues.

Worked example: compress to $k=1$ and reconstruct

Dr. Lena Hartmann: Encoding is just a dot product with the principal direction, producing a single coordinate per point.

Dr. Lena Hartmann: Decoding stretches that one coordinate back along the same direction and adds the mean to put points back in the right place.

Kai: So the whole point is that this chosen line makes the reconstruction error as small as possible, compared to any other line?

Dr. Lena Hartmann: Yes. Maximizing variance and minimizing squared reconstruction error are two sides of the same PCA coin.

PCA as a linear layer you can plug into pipelines

Dr. Lena Hartmann: This is the deployment view: W k is a fixed weight matrix with k principal directions, and Z is your compressed representation.

Dr. Lena Hartmann: For a single vector x, you subtract the mean and apply a linear map. This is exactly how it would look inside a model graph.

Dr. Lena Hartmann: If you store billions of embeddings, even a two times reduction in dimension can be a big win for throughput.

Kai: Because if the data distribution shifts, the mean and directions are no longer the right ones?

Dr. Lena Hartmann: Exactly. PCA is data dependent, so you need to fit it on the same kind of vectors you will later compress.

How to choose $k$: explained variance

Dr. Lena Hartmann: Choosing k is a tradeoff: smaller k is cheaper, larger k keeps more variance. Explained variance gives a simple quantitative knob.

Dr. Lena Hartmann: Each eigenvalue lambda i is variance captured by component i, so dividing by total variance gives a percentage.

Dr. Lena Hartmann: In real work, you often pick k by a threshold like ninety percent explained variance, then sanity check downstream performance.

Kai: So I should not blindly choose k from variance; I should test retrieval quality or classifier accuracy too?

Dr. Lena Hartmann: Exactly. Explained variance is a good heuristic, but your evaluation metric is the final judge.

PCA checklist: what to do, what to watch

Dr. Lena Hartmann: Let us compress the whole story into a practical checklist you can reuse in ML projects.

Dr. Lena Hartmann: Always center. If you skip this, the first component might just point toward the mean offset instead of a true variation direction.

Dr. Lena Hartmann: Use eigenvectors of the covariance for small d, and S V D for stability and speed when n and d are large.

Dr. Lena Hartmann: Projection is just a matrix multiply: you are changing coordinates into the principal component basis.

Kai: And if variance looks good but my model gets worse, I probably removed information that matters for the task.

Dr. Lena Hartmann: Exactly. PCA keeps what varies most, not necessarily what predicts best, so always validate on the actual objective.

Exit ticket: identify the first principal direction

Dr. Lena Hartmann: Here is a covariance matrix that already has no cross-correlation: variance is nine on the first axis and one on the second.

Kai: Since variance is bigger on the first axis, is the first principal direction just the x axis?

Dr. Lena Hartmann: Yes. The eigenvectors are the standard axes here: the first principal direction is one, zero, and the top eigenvalue is nine.

Dr. Lena Hartmann: Answer check: the first principal direction is one, zero, and the top eigenvalue is nine.

Dr. Lena Hartmann: Reflect: PCA is great when your data truly lies near a low dimensional subspace, but it can fail if the important information is in small-variance directions or if the distribution changes.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!