Linear Transformations

Bridge from $L05$: when does a map have an $A^{-1}$?

Dr. Lena Hartmann: Last time, determinants and inverses told us when a matrix-based map is reversible, which is huge in debugging and understanding model behavior. Today we name the bigger object: a linear transformation, the rule that turns an input vector into an output vector.

Dr. Lena Hartmann: Recap: for n by n matrices acting on vectors in R to the n, det of A not equal to 0 means A is invertible. That is equivalent to the map being one to one and onto, or equivalently that the only vector sent to the zero vector is the zero vector. If the matrix is not square, there is no determinant, and the map can still reduce dimension.

Kai: So today is like: before asking if an inverse exists, we ask what kind of maps we are even talking about?

Dr. Lena Hartmann: Exactly. Driving question: what properties make a transformation simple enough to represent by a matrix, and geometric enough to reason about in AI?

Why AI people should care: layers are transformations of embeddings

Dr. Lena Hartmann: Every forward pass in a neural network is a stack of transformations: you take an embedding vector and repeatedly map it into new vectors. If you can picture what each map does to space, you can reason about capacity, collapse, and why training sometimes gets stuck.

Dr. Lena Hartmann: Linear transformations are the special ones that behave predictably under addition and scaling.

Kai: Is that why so many model blocks look like matrix multiply plus some nonlinearity?

Dr. Lena Hartmann: Yes: the matrix multiply part is linear, and that makes it analyzable and fast on hardware. Then nonlinearities add expressiveness on top.

Intuition: a transformation moves *all* vectors, not just one

Dr. Lena Hartmann: A transformation is not a single vector operation; it is a rule applied to every vector in the space. So we judge it by what it does to shapes like grids, squares, or circles.

Dr. Lena Hartmann: Think: if we know where the basis vectors go, we can predict where any vector goes.

Kai: Basis vectors meaning the standard axes directions, like the unit x and unit y arrows?

Dr. Lena Hartmann: Exactly. In two dimensions, knowing where those two arrows land is enough to reconstruct the whole transformation.

Let me show you in code: matrix as a transformation

Dr. Lena Hartmann: Before any formal definition, think of a two by two matrix as a rule that takes an input vector and produces a new vector. Our goal is to make that idea feel tangible: the same input always maps to the same output, and the mapping can stretch, shrink, shear, and rotate.

Dr. Lena Hartmann: First bullet: we pick a matrix that stretches and shears, and a test vector.

Kai: And the output is just matrix times vector, right? I want to see the numbers.

Dr. Lena Hartmann: Exactly, but before we reveal the computed values, make a prediction. Use the fact that A times e1 is the first column of A, and A times e2 is the second column of A. What do you expect for A times e1 and A times e2, and then we will compare with the printed results for those and for A times x.

Dr. Lena Hartmann: Now run it and compare. The output shows A times e1 equals [2.0, 0.5] and A times e2 equals [1.0, 1.0], which match the first and second columns of A. If you predicted [2.0, 1.0] for A times e1, you probably grabbed the first row by accident. For the test vector x equals [1.0, 2.0], the code prints A times x equals [4.0, 2.5]. Notice this also matches the column view: one copy of the first column plus two copies of the second column gives [2.0, 0.5] plus [2.0, 2.0] equals [4.0, 2.5].

Visualization: what happens to a unit square?

Dr. Lena Hartmann: Numbers are good, but geometry makes it stick. A clean way to see a linear transformation is to watch what it does to the unit square: its four corners move, and the square turns into a parallelogram.

Dr. Lena Hartmann: Before we compute anything, make a prediction: which two points determine the new edges of the transformed square? Think about where the points (1,0) and (0,1) go under the matrix.

Kai: Are those the images of the basis directions? Like A times e one and A times e two, which are exactly where (1,0) and (0,1) land?

Dr. Lena Hartmann: Exactly. Those two transformed points set the new edge directions, and then the last corner follows by adding them. In the code, we compute the transformed corner coordinates so you can compare the original list with the transformed list.

Key insight: the two rules that make a map linear

Dr. Lena Hartmann: Now we can say what makes these maps special. Linearity is exactly two compatibility rules: with addition, and with scaling.

Dr. Lena Hartmann: This equation is the whole definition packed into one line: the map of a combination equals the same combination of mapped vectors.

Kai: So if I double the input vector, the output doubles too, and sums behave the same way?

Dr. Lena Hartmann: Yes. And that is why a linear map is completely determined by what it does to basis vectors: every vector is built from them by scaling and adding.

Formal payoff: every linear map is matrix multiplication

Dr. Lena Hartmann: Here is the big practical result: for linear maps between finite-dimensional spaces, once you choose a basis, the transformation can be represented by a matrix. In machine learning, we almost always use the standard basis for R to the n, so linear means matrix times vector.

Dr. Lena Hartmann: More precisely, if T maps R to the n into R to the m, there exists an m by n matrix A such that applying T to x is the same as multiplying A times x.

Kai: Where does the matrix A come from, concretely? Is it just the transformed basis vectors stacked together?

Dr. Lena Hartmann: Exactly. Column j of A is the output of the transformation on basis vector e j, so multiplying by A mixes those columns using the coordinates of the input.

Trust but verify: testing linearity with random vectors

Dr. Lena Hartmann: A great habit in AI engineering is to write a quick check. If you think some operation is linear, test the defining property numerically on random inputs.

Dr. Lena Hartmann: First bullet: we generate random vectors u and v and random scalars alpha and beta.

Kai: And then compare transform of the combo versus combo of the transforms, up to floating point error?

Dr. Lena Hartmann: Yes. The code prints a small norm of the difference, which should be near zero for a matrix-based transformation.

AI connection: a dense layer is linear plus a bias (affine)

Dr. Lena Hartmann: In deep learning, a dense layer looks linear, but there is a subtle twist: the bias term shifts everything. That makes the full layer affine, not linear, but the matrix part is still our linear transformation.

Dr. Lena Hartmann: This equation is the standard layer form: matrix multiply, then add a bias vector.

Kai: So the bias breaks the rule about scaling, right? If I scale the input, the constant shift does not scale.

Dr. Lena Hartmann: Exactly. But many geometric intuitions still apply: the matrix rotates and stretches space, then the bias translates the result.

AI connection: attention is linear in V once weights are fixed

Dr. Lena Hartmann: In this AI example, the main idea is simple: once the attention weights are treated as fixed, the step that mixes the value vectors becomes a plain linear transformation. That connects directly to our theme of functions of the form x maps to A x.

Dr. Lena Hartmann: We can bundle everything computed from Q and K into a single weight matrix P. Using the common convention that tokens are rows, V is an n by d v matrix and P is an n by n matrix, so the output is O equals P V, which is linear in V.

Kai: So it is not linear in Q or K because of the softmax, but it is linear in V after the weights are computed?

Dr. Lena Hartmann: Exactly. After Q and K determine P, attention applies a data-dependent but fixed-for-the-moment matrix to the values. Concretely, each token output o i equals the sum over j of P i j times v j, so the value mixing step is a standard linear map.

Recap: a linear transformation is a predictable geometric machine

Dr. Lena Hartmann: Lets compress the whole lecture into three takeaways you can use immediately when reading ML code and papers.

Dr. Lena Hartmann: First: linear means it preserves addition and scaling, so mixtures of inputs map to mixtures of outputs.

Kai: Second: once I know where the basis vectors go, I basically know the whole map, because every vector is built from the basis.

Dr. Lena Hartmann: Third: in AI, matrix multiply is the workhorse linear transformation, and determinants and inverses from last time tell you when that transformation is reversible.

Exit ticket: compute and interpret a transformation

Dr. Lena Hartmann: Let's do one quick practice to make sure the mechanics and the meaning are both there. Then one reflection question to connect it to your ML intuition.

Dr. Lena Hartmann: Practice: compute the output vector by multiplying the matrix by the input vector; you can do it as a linear combination of the matrix columns.

Kai: So I should think: one times the first column plus minus one times the second column, giving the transformed vector?

Dr. Lena Hartmann: Yes. Reflection: in an embedding model, ask which directions get stretched and which get squashed by the weight matrix, because that changes similarities downstream.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!