Neural Networks as Linear Algebra

From vector search to neural networks: same math, bigger pipeline

Dr. Lena Hartmann: In L11 we scored similarity with a dot product: one vector times another vector gives one number.

Dr. Lena Hartmann: We treated each item as a d dimensional vector, and the whole retrieval problem became geometry in high dimensions.

Dr. Lena Hartmann: At scale, you do lots of dot products in one shot by multiplying a query matrix by a key matrix transpose.

Kai: So matrix multiply is the real engine under the hood, not just a math detail.

Dr. Lena Hartmann: A neural network just repeats that idea: take vectors, apply structured matrix operations, output new vectors you can search, classify, or generate with.

A forward pass is mostly one line of linear algebra

Dr. Lena Hartmann: This is the whole story of a basic layer: multiply by a matrix, add a bias, then apply a simple nonlinearity.

Dr. Lena Hartmann: In code, this is literally a matrix times a vector plus a vector; that is why GPUs and linear algebra libraries dominate deep learning performance.

Dr. Lena Hartmann: Understanding the dimensions tells you immediately what can and cannot multiply, and it gives you fast checks when something breaks in a model pipeline.

Dr. Lena Hartmann: The same building block shows up in embedding layers, in linear classifier heads, and inside transformer blocks; the names change but the math repeats.

Kai: So if I understand this one equation deeply, a lot of architectures stop feeling magical.

One neuron is a dot product plus a threshold shift

Dr. Lena Hartmann: This shows a dot product between weight vector w and input vector x, then a bias that shifts the score.

Dr. Lena Hartmann: If x lines up with w, the dot product is big; if x points away, the score is small or negative. That is a geometric way to think of a learned feature.

Dr. Lena Hartmann: The bias is like moving the decision threshold left or right, so the neuron is not limited to cutting space only through the origin.

Dr. Lena Hartmann: That last expression is a hyperplane: it splits space into two half spaces where the neuron score is positive versus negative.

Kai: So a neuron is basically linear classification geometry, but used as a building block inside bigger functions.

Dr. Lena Hartmann: After computing z, we usually pass it through an activation function to produce the neuron's output, like a gate that keeps the model from being purely linear.

A layer is many neurons at once: a matrix times a vector

Dr. Lena Hartmann: This is an affine map: a linear transformation W times x, then a translation by b.

Dr. Lena Hartmann: These dimensions are the debugging superpower: m outputs, d inputs. If the shapes do not match, the layer literally cannot run.

Dr. Lena Hartmann: Each row is one neuron scoring x. So a layer is just many dot products packaged into a single matrix multiply.

Dr. Lena Hartmann: In practice you do B inputs at once, so x becomes a matrix X; the same idea scales from vector to matrix with no new math.

Worked example: compute one affine layer by hand

Dr. Lena Hartmann: Here is our layer: a 2 by 2 weight matrix, a 2D input vector, and a 2D bias vector.

Dr. Lena Hartmann: When we multiply and add, we get an output vector y with two components: four point five and negative one.

Dr. Lena Hartmann: Each row of W dots with x. First row gives four, second row gives one, then the bias shifts them to four point five and negative one.

Dr. Lena Hartmann: This is exactly the line you would run: W at x plus b. The whole deep learning stack is built to make that line fast.

Kai: That makes sense now: rows act like separate scorers, and the result stacks those scores.

Nonlinearity: the tiny piece that makes depth powerful

Dr. Lena Hartmann: ReLU is the most common example: it keeps positive values and clamps negatives to zero.

Dr. Lena Hartmann: For vectors, you apply it coordinate by coordinate. So it is simple, but it changes the geometry a lot.

Dr. Lena Hartmann: ReLU makes the network behave like different linear maps in different parts of the input space, depending on which coordinates are active.

Dr. Lena Hartmann: That is the key: deep networks build complicated shapes by gluing together lots of simple linear pieces.

Kai: So the linear algebra is still there, but the on off pattern from ReLU decides which linear algebra you are using.

A neural network is a composition of affine maps and nonlinearities

Dr. Lena Hartmann: This is a two layer multilayer perceptron: first an affine map, then a nonlinearity, then another affine map.

Dr. Lena Hartmann: You can track how the dimension changes layer by layer. That is why model configs list hidden sizes.

Dr. Lena Hartmann: Everything the network can learn is stored in these matrices and bias vectors. Training is just changing those numbers.

Dr. Lena Hartmann: This is a classic sanity check: without the nonlinearity, you could multiply all the matrices together and get a single equivalent layer.

Kai: So the whole point of depth is not more matrices, it is matrices plus nonlinear gates.

Worked example: add a nonlinearity and a second layer

Dr. Lena Hartmann: From the previous layer output, ReLU keeps four point five and turns negative one into zero.

Dr. Lena Hartmann: Given a equals four point five, zero, what do you predict W two times a will be?

Kai: It should be two times four point five plus negative one times zero, so nine.

Dr. Lena Hartmann: Exactly: the output is nine. Notice the negative weight on the second coordinate has no effect here because ReLU already made that coordinate zero.

Dr. Lena Hartmann: Even though each step looks simple, the overall map from x to the final output is now nonlinear because ReLU changed the path.

Dr. Lena Hartmann: Modern networks just repeat this thousands or millions of times with bigger matrices, but the operations are still multiply, add, and apply a nonlinearity.

Kai: Seeing it with numbers helps. It is not mysterious, it is just a lot of these blocks stacked.

Real models run batches: matrix times matrix

Dr. Lena Hartmann: This shows the batched affine map: X is a batch of inputs, and one adds the bias to every row.

Dr. Lena Hartmann: Batch size B just means you stack B vectors into a matrix. The layer then outputs B vectors, also stacked.

Dr. Lena Hartmann: This is why deep learning speed talk quickly becomes matrix multiply optimization, memory bandwidth, and GPU kernels.

Dr. Lena Hartmann: Frameworks hide the ones vector by broadcasting, but mathematically it is the same batched affine map.

Kai: So the batch dimension is not new math, it is just stacking and using matrix multiply.

Training: backprop turns gradients into more matrix multiplies

Dr. Lena Hartmann: This equation says the gradient for the weight matrix is an outer product: error signal delta times the input transpose.

Dr. Lena Hartmann: And this shows how error flows backward: multiply by a transpose, then apply an elementwise gate from the activation derivative.

Dr. Lena Hartmann: When people say backprop is efficient, they mean it reuses the same kinds of operations as the forward pass: multiplies and elementwise ops.

Dr. Lena Hartmann: Frameworks avoid building gigantic Jacobian matrices; they push vectors through the chain rule, which is exactly these products.

Kai: So in practice, we never materialize a huge Jacobian, we just move gradient vectors backward through the same building blocks.

AI connection: transformers are linear algebra with clever wiring

Dr. Lena Hartmann: Attention is built from matrix products: Q times K transpose gives scores, softmax normalizes them, and then you mix V by another matrix multiply.

Dr. Lena Hartmann: Another linear algebra idea you will see in practice is low rank fine tuning, where you keep W fixed and add a small update that looks like B times A.

Dr. Lena Hartmann: Even in transformers, you repeatedly project vectors with learned matrices; the novelty is how you route information across tokens.

Dr. Lena Hartmann: When you understand the linear pieces, you can reason about bottlenecks, compression, and also probe representations using simple linear models.

Kai: So the linear algebra view is not just theory, it is a way to read papers and debug systems.

Takeaway: neural networks are linear algebra plus simple nonlinear gates

Dr. Lena Hartmann: Whenever you see a layer, translate it into matrix times vector plus bias, then a simple nonlinearity like ReLU or softmax.

Dr. Lena Hartmann: Production speed comes from doing many examples or many tokens at once, which turns everything into matrix times matrix.

Dr. Lena Hartmann: Without nonlinearities, depth would be pointless. With them, the network becomes a piecewise linear or otherwise nonlinear function.

Dr. Lena Hartmann: Training is not separate magic; it is the same linear algebra patterns running in reverse, guided by the chain rule.

Exit ticket: can you run a tiny network step correctly?

Dr. Lena Hartmann: Your job is to compute the vector inside ReLU first, then apply ReLU coordinate by coordinate.

Kai: Let me try: first row dot x is 1 times 2 plus negative 2 times negative 1, so that is 4. Then plus bias 0 stays 4.

Dr. Lena Hartmann: Good. Second row dot x is 0 times 2 plus 3 times negative 1, so negative 3. Then plus bias 1 gives negative 2. Now apply ReLU: 4 stays 4, negative 2 becomes 0.

Dr. Lena Hartmann: The matrix multiply and bias add are the linear and affine part; ReLU is the point where the function stops being globally linear and starts selecting a region.

Kai: So if I changed the input a bit and that second coordinate crossed zero, the network would switch to a different linear piece.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!