Vector Spaces and Subspaces

From $L03$ to today: when is a solution set a "legal" space?

Dr. Lena Hartmann: Systems of equations are not just about finding one answer. In AI, the whole set of answers matters, because those sets become the spaces where we optimize, compress, and interpret models. Today we are going to discover vector spaces and subspaces, starting from what you already saw in L03. We will go from pictures and numpy experiments, to a clean test you can apply in seconds.

Dr. Lena Hartmann: Here is the bridge: last lecture, you saw that solutions can form geometric objects like lines and planes, not just single points.

Kai: So if the solutions make a line or plane, is that automatically a vector space?

Dr. Lena Hartmann: Not always. Today we will pin down the exact rules, and you will learn a quick subspace test that tells you immediately when a set behaves like a space you can do linear algebra inside.

Why AI cares: embeddings and weights live in spaces (often smaller ones)

Dr. Lena Hartmann: The stakes: every time you average word embeddings, do PCA, or train a linear probe, you are assuming your vectors live in a structure where addition and scaling behave predictably. That structure is a vector space, and subspaces are the smaller, cheaper regions we often restrict ourselves to in production AI.

Dr. Lena Hartmann: First, embeddings: we store meaning as vectors in a high-dimensional real space, and we want operations like add and scale to stay inside that space.

Kai: And subspaces are like picking a smaller set of directions to move in?

Dr. Lena Hartmann: Exactly. That last bullet hints at why: we often search for a direction that encodes a concept, and that direction lives inside a subspace we can analyze and manipulate.

Intuition: a vector space is closed under the moves you want to make

Dr. Lena Hartmann: Before any formal list of axioms, think like an engineer: you have a set of objects, and you want to do two operations all the time in ML: add them and scale them. A vector space is simply a set where those moves never kick you out.

Dr. Lena Hartmann: Move one is addition: if you combine two feature vectors, the result should still be a valid feature vector in your set.

Kai: Scaling is like changing the strength of a feature direction, right?

Dr. Lena Hartmann: Yes. And closure is the punchline: if your set is not closed, basic operations like averaging embeddings can produce something that is not even representable in the same space.

Geometric examples in $\mathbb{R}^2$: some sets behave, some break

Dr. Lena Hartmann: Let us ground this with pictures in two dimensions. You already trust that all of the plane works nicely with addition and scaling. The interesting question is which smaller sets inside the plane also behave nicely.

Dr. Lena Hartmann: First bullet: the whole plane is our default vector space example.

Dr. Lena Hartmann: Second bullet: a line through the origin is stable under add and scale. If you take two points on that blue line and add them, you stay on the line.

Kai: But the shifted line looks almost the same. Why does it fail?

Dr. Lena Hartmann: Because it does not contain the zero vector, and scaling breaks it: multiplying a point by zero gives the zero vector, which is not on the shifted line. One missing point ruins the whole closure story.

Let me show you in code: testing closure (subspace or not?)

Dr. Lena Hartmann: Before the formal definition, make a prediction in words: if we take points on a line through the origin, will adding them stay on the line, and will scaling by any number stay on the line? Now compare that to a shifted line that does not pass through the origin: what specific scalar do you think will break it?

Dr. Lena Hartmann: Here is a small runnable check: we define S as points of the form (t, 2t) and T as points of the form (t, 2t plus 1), then we test addition for S and we explicitly test scaling by a equals zero for both. When you run it, you should see: “S add closed? True”, “S scale closed (a=0)? True”, and then “T scale closed (a=0)? False; 0*p = [0. 0.].”

Kai: So for the shifted line, multiplying a point by zero gives the zero vector, and that zero vector is not on y equals 2x plus 1. That means scalar closure fails right away.

Dr. Lena Hartmann: Exactly. The printed “False” for T matches the geometry: scaling any point on a shifted line by zero lands at (0, 0), but (0, 0) does not satisfy y equals 2x plus 1. Meanwhile the “True” results for S match the fact that a line through the origin is closed under addition and scaling. And remember: a few printouts can build intuition, but a real subspace check needs a for-all argument, typically phrased as nonempty plus closure under all linear combinations.

The fast subspace test (what you actually use)

Dr. Lena Hartmann: Here is the practical rule. If you only remember one slide, remember this one: we are working inside R n with real scalars, and a subspace is a nonempty set where any linear combination of vectors from the set stays in the set.

Dr. Lena Hartmann: This equation is the one line closure test: let S be a subset of R n, assume the set is not empty, pick any two vectors in the set, multiply by any real numbers a and b, add them, and you must land back in the set.

Kai: Why do we need both scalars a and b? Why not just test add and scale separately?

Dr. Lena Hartmann: Because this single condition implies both: set b equal to zero to get closure under scaling, and set a and b both equal to one to get closure under addition. It also matches how we actually build vectors in practice: we form weighted sums. And once the set is nonempty and this holds inside R n with real scalars, the zero vector is automatic, so you can safely use all your linear algebra tools inside the set.

Vector space: the full rulebook (but we will use it lightly)

Dr. Lena Hartmann: A vector space is the full, official structure. It has a set of vectors, a notion of scalar, and rules that guarantee the operations behave consistently. Most of the time in ML, you rely on the consequences, not the full list.

Dr. Lena Hartmann: First bullet: you need to know what your vectors are and what scalars you allow. In our course, scalars will almost always be real numbers.

Kai: So the zero vector and these identity rules are not just details, they are part of what makes algebra safe?

Dr. Lena Hartmann: Exactly. The last bullet is the everyday version: if any weighted sum stays inside, you can build new vectors from old ones without leaving the space. That is why vector spaces power optimization and representation in AI.

Worked example: a solution set is a subspace (the null space)

Dr. Lena Hartmann: Now a complete worked example that ties directly to L03. We define S as all vectors in three dimensions that solve a homogeneous system. This set is called the null space of A.

Dr. Lena Hartmann: This equation defines the set: we are looking at all x that A times x equals zero, where A is the row vector one, two, minus one.

Kai: So to prove it is a subspace, we just check zero and linear combinations?

Dr. Lena Hartmann: Yes. First bullet: the zero vector works automatically in any homogeneous system. Second bullet: if two vectors satisfy the equation, any weighted sum satisfies it too, because matrix multiplication distributes over addition and respects scaling.

Let me show you in code: finding a basis for the null space

Dr. Lena Hartmann: Next, we verify the worked example computationally. The main idea is to have the computer find directions x that make A times x come out to zero, which is exactly what it means to be in the null space.

Dr. Lena Hartmann: In this code, numpy linalg svd of A gives us the right singular vectors in V. We pick a small tolerance, estimate the numerical rank from the singular values, and then take the remaining columns of V as a numerical basis for the approximate null space. Finally, we print A times the basis to confirm it is near zero, keeping in mind that the result depends on the tolerance choice.

Kai: So those basis vectors are like the independent directions that do not change the constraint?

Dr. Lena Hartmann: Exactly. And the second bullet is the AI connection: many problems bake in invariances or constraints, and they often become homogeneous linear equations. Then your allowed changes live in a null space subspace.

Subspaces you will actually meet in ML

Dr. Lena Hartmann: Let us connect subspaces to recognizable ML tooling. Subspaces are how we say, out of all possible changes, only these directions are allowed or useful.

Dr. Lena Hartmann: First bullet: PCA literally produces a small set of directions, and their span is the low dimensional subspace you project onto.

Kai: And LoRA feels like we are only allowed to move weights in a specific pattern, not arbitrarily.

Dr. Lena Hartmann: Yes, that is the second bullet: low rank updates form a structured subspace inside the huge space of all weight matrices. Third bullet: a linear probe searches within a very specific function family, which is also a kind of subspace idea.

Checkpoint: how to decide "subspace or not" in seconds

Dr. Lena Hartmann: Time to compress the lecture into a mental checklist you can apply on sight. This is the exact habit that saves you from mistakes when you read papers or implement methods.

Dr. Lena Hartmann: First bullet: if the set misses the zero vector, it is dead on arrival as a subspace.

Kai: Second bullet is the linear combination test: pick any two vectors in the set and you are not allowed to escape.

Dr. Lena Hartmann: And third bullet closes the loop with L03: the solution set of a homogeneous system is always a subspace. That is why null spaces, column spaces, and all those spaces in linear algebra show up so naturally.

Exit ticket: classify a set and explain why it matters

Dr. Lena Hartmann: Let us finish with a quick test you can do without any computation. You will classify a set using the subspace checklist, then connect it back to your own ML habits.

Dr. Lena Hartmann: For the practice: the condition x plus y equals zero describes a line through the origin. The zero vector satisfies it, and any linear combination of two solutions still satisfies the same equation, so yes, it is a subspace of R two.

Kai: Reflection wise, I think I assume closure any time I average embeddings or interpolate between two representations.

Dr. Lena Hartmann: Perfect. That reflection is the point: once you notice the assumption, you can check whether your data actually lives in a true vector space, or in something more constrained where averaging might not make sense.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!