Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors reveal the special directions a matrix stretches or shrinks without changing their direction. They power ideas like PCA and help build intuition for how neural networks transform data.

From moving space to finding its hidden anchors

Dr. Lena Hartmann: We've always seen matrices as operators that transform spaces. But what if some directions don't get turned by these transformations?

Kai: So we're looking for something stable within all the motion?

Dr. Lena Hartmann: Yes, precisely. We're looking for unchanging directions, unlike our previous idea of matrices simply transforming everything.

Dr. Lena Hartmann: Today, we're focusing on directions that, after a matrix transformation, still lie on the same line. They may stretch, shrink, or flip, but they stay on the same line through the origin.

Dr. Lena Hartmann: This concept is captured by the equation: A times v equals lambda times v. A is our matrix, v is a special direction vector, and lambda is its scalar factor. This is the heart of eigenvalues and eigenvectors.

Why AI engineers should care

Dr. Lena Hartmann: Before we dive into the mathematical details, let me show why this topic keeps appearing in AI systems.

Dr. Lena Hartmann: The first use case is PCA. When we compress high dimensional data, we often ask which directions explain the most spread. Those directions come from eigenvectors.

Kai: So this is not just a textbook thing. It is connected to dimensionality reduction?

Dr. Lena Hartmann: Exactly. The second use case is graph ranking. If pages, users, or documents pass importance around, a stable score pattern is an eigenvector idea.

Kai: Like how Google's PageRank algorithm determines the importance of web pages?

Dr. Lena Hartmann: Yes, that's a perfect example. PageRank relies heavily on eigenvectors to compute those stable importance scores.

Dr. Lena Hartmann: And the third use case shows up in training dynamics. Some directions in parameter space grow quickly, some decay, and eigenvalues help describe those modes.

Kai: This sounds really foundational. So eigenvectors can help us understand stability and change in complex AI models?

Dr. Lena Hartmann: Indeed, they provide a powerful analytical lens for understanding the intrinsic behavior and stability of such systems.

Let me show you in code

Dr. Lena Hartmann: Let's look at the code. We apply one matrix to a few vectors.

Dr. Lena Hartmann: Before running, examine the matrix and the x-axis vector, one zero, and the diagonal vector, one one. What do you predict happens to their directions after this transformation?

Kai: Based on the matrix, I think the x-axis vector will stay on its line, maybe stretched, but the diagonal one looks like it might rotate.

Dr. Lena Hartmann: Good prediction! The output shows the x-axis vector transforms to two zero. It's scaled but stays on its line. This is the geometric signal.

Kai: But for the diagonal vector, one one, it transforms to three one. That clearly changed direction, so it's not special for this matrix, right?

Dr. Lena Hartmann: Correct. It rotated off its line. Of the vectors we tested, only the x-axis vector remained on its original line, only scaled.

Dr. Lena Hartmann: The random vector also changed direction, so it's not a special direction for this matrix.

Some directions do not turn

Dr. Lena Hartmann: Now imagine how a matrix transformation works. A matrix usually pushes an arrow, or vector, into a new direction. Think of it like stirring a pot, most things get moved around.

Kai: So most vectors just get scrambled, their original path is lost?

Dr. Lena Hartmann: Typically, yes, most vectors change their direction. The first bullet is the normal case. Most vectors are sent to a different line, so their angle changes.

Kai: So an eigenvector is not just any output vector. It is an input vector with a special behavior.

Dr. Lena Hartmann: Yes. The second bullet says the special input keeps its line after the matrix acts. It might stretch or compress, but it stays on the same line, just further along it or closer to the origin.

Kai: And the 'stretch amount' is a scalar value, not another vector?

Dr. Lena Hartmann: Yes, that's right. The third bullet names the amount of stretching or flipping. That number is the eigenvalue. It tells us precisely how much the eigenvector is scaled.

The pattern we are hunting

Dr. Lena Hartmann: We're hunting for a special pattern in vectors under matrix transformations. Typically, transformations twist and stretch.

Kai: Is this pattern what defines an eigenvector?

Dr. Lena Hartmann: Yes, this pattern defines an eigenvector. It's when a matrix A transforms vector v, but its direction doesn't change, only its length. So, A times v is a scaled version of v.

Kai: So it's like a special line in space that the matrix just scales?

Dr. Lena Hartmann: Precisely. The vector v is our initial input direction, the vector we're testing for this special behavior.

Kai: So v is the direction that stays aligned?

Dr. Lena Hartmann: Yes. A times v represents vector v after transformation. We're checking if its direction stays aligned with the original v.

Dr. Lena Hartmann: Finally, the Greek letter lambda is the scalar factor. It tells us how much v was scaled, stretched, compressed, or flipped.

Formal definition

Dr. Lena Hartmann: Now that we have an intuitive understanding, let's establish the precise mathematical definition of eigenvalues and eigenvectors.

Kai: So this is the official mathematical way to write what we just discussed?

Dr. Lena Hartmann: Yes, this is the formal algebraic expression: A v equals lambda v. This means A acts on v by scaling it by lambda.

Kai: And v is the eigenvector, and lambda is the eigenvalue, right?

Dr. Lena Hartmann: That's right. The eigenvector v represents the direction unchanged by transformation A.

Dr. Lena Hartmann: And lambda is the eigenvalue: the single number saying how much that direction is stretched, shrunk, or flipped when the matrix acts on it.

Kai: Why can't v be the zero vector?

Dr. Lena Hartmann: V must not be the zero vector. If it were, the equation becomes trivial for any lambda, providing no meaningful insight. Eigenvectors must represent a meaningful direction.

A matrix with two clear anchors

Dr. Lena Hartmann: Eigenvectors are special directions a matrix only scales, without rotation. This slide introduces a matrix with two clear candidates for such directions.

Kai: So we're looking for these anchors or stable points?

Dr. Lena Hartmann: Yes, directions that remain invariant. This symmetric matrix A, with entries two one, one two, will be our example. It's a simple two-by-two matrix to keep calculations clear.

Dr. Lena Hartmann: Consider the two vectors shown here. Vector one one is a 'together direction'. How do you think matrix A will transform it?

Kai: Does 'together direction' mean its x and y components are equal?

Dr. Lena Hartmann: Exactly. Vector one negative one is an opposite direction. These two orthogonal vectors form a natural basis for this space.

Dr. Lena Hartmann: We'll check how matrix A transforms these two candidates. Do they maintain their direction, only scaling?

Check the candidates directly

Dr. Lena Hartmann: Now, let's verify our candidate eigenvectors. Looking at the first equation, where we apply our matrix to the vector one, one, what do you predict the output will be, and will its direction be preserved?

Kai: So we're doing matrix-vector multiplication here, and I predict the output will be the vector three, three. Yes, its direction should be preserved.

Dr. Lena Hartmann: Yes, exactly. You're right, the output is three, three. As you can see, when we multiply our matrix by the vector one, one, it results in the vector three, three. This is three times the original vector! So its direction was preserved, and the scaling factor is three.

Dr. Lena Hartmann: Now consider the second candidate vector, one, minus one. What do you predict its transformation will yield? Will its direction be preserved, and if so, what's its scaling factor?

Kai: So the scale factors are our eigenvalues, right?

Dr. Lena Hartmann: That's right. When we apply the matrix to the vector one, minus one, it results in the vector one, minus one. So the direction was preserved, and the scaling factor is one. From these calculations, we've identified the corresponding scale factors, or eigenvalues, as three and one.

How to find them without guessing

Dr. Lena Hartmann: How do we systematically find eigenvectors and their eigenvalues, without guessing?

Dr. Lena Hartmann: We rearrange the eigenvalue equation into this homogeneous system: A minus lambda I, all times vector v, equals the zero vector.

Dr. Lena Hartmann: First, recall that an eigenvector, vector v, by definition, cannot be the zero vector. If the matrix A minus lambda I were invertible, meaning its determinant was non-zero, then the only solution to the equation A minus lambda I times vector v equals zero would be the zero vector.

Kai: Wait, why does having a non-zero solution mean the matrix is singular?

Dr. Lena Hartmann: Because a matrix with a non-zero determinant is invertible, and an invertible matrix sends only the zero vector to zero. So if A minus lambda I maps some non-zero v to zero, it cannot be invertible. That is what singular means.

Dr. Lena Hartmann: But since we require a non-zero vector v, it must be the case that this matrix, A minus lambda I, cannot be invertible. When a matrix is not invertible, we call it a singular matrix.

Dr. Lena Hartmann: And a key property of singular matrices is that their determinant must be zero.

Kai: So setting the determinant to zero is the key to finding lambda?

Dr. Lena Hartmann: Exactly. This characteristic equation, determinant of A minus lambda I equals zero, gives us the eigenvalues.

Kai: And once we find lambda, how do we find v?

Dr. Lena Hartmann: And it runs the other way too, which is what makes this a method rather than just a test. If the determinant of A minus lambda I is zero, that matrix is singular, and a singular square matrix always sends some non-zero vector to zero. So every root of the characteristic equation really does come with an eigenvector, none of them are false leads.

Dr. Lena Hartmann: Substitute each eigenvalue back into the first equation, A minus lambda I times v equals zero, and solve for vector v to get the eigenvectors.

Worked algebra for the example

Dr. Lena Hartmann: Let's apply that systematic approach to our example matrix. We'll start by constructing the matrix A minus lambda I, where lambda is subtracted from the diagonal elements.

Dr. Lena Hartmann: This is the determinant condition for our specific matrix. We substitute our matrix A and the identity matrix I, then form a new matrix whose determinant we need to calculate.

Kai: So we just take A and subtract lambda along the main diagonal?

Dr. Lena Hartmann: Yes, that's how we form the matrix A minus lambda I. When we calculate the determinant of this matrix, it simplifies quite nicely into a straightforward quadratic equation that we can easily solve.

Kai: And solving this quadratic equation gives us the lambdas we found before: 3 and 1?

Dr. Lena Hartmann: Precisely. The roots of this characteristic equation are the eigenvalues. They are the same values we observed when we directly checked our candidate vectors. This confirms our systematic method works.

After $\lambda$, find $\mathbf{v}$

Dr. Lena Hartmann: We've found our eigenvalues, lambda equals 3 and lambda equals 1. Now, let's find their corresponding eigenvectors by plugging each eigenvalue back into our rearranged equation.

Dr. Lena Hartmann: For eigenvalue three, substitute lambda equals 3. As shown, solving the system A minus 3I times v equals zero gives us eigenvectors of the form c times the vector one, one.

Kai: So for lambda equals 3, the eigenvectors are all vectors along the direction one, one?

Dr. Lena Hartmann: Exactly. Now, for eigenvalue one, we set up the system A minus I times v equals zero.

Dr. Lena Hartmann: Solving that system gives every vector of the form c times one, minus one. Any nonzero multiple works, because eigenvectors describe a direction rather than one particular arrow, so the whole line through one, minus one qualifies.

Kai: Why is it important that c is not zero?

Dr. Lena Hartmann: Eigenvectors must be non-zero vectors. If c were zero, it would not be a valid eigenvector.

Verify with NumPy

Dr. Lena Hartmann: Now let the computer do the same job and then verify the definition. Modern linear algebra libraries make this process very efficient.

Dr. Lena Hartmann: This complete script uses numpy linalg eig on the matrix. The important habit is the check at the bottom: matrix times vector should match value times vector.

Kai: I have seen that function in notebooks, but I did not know the columns were the directions.

Dr. Lena Hartmann: Yes. The first bullet names the function, and the second warns about the layout. Each column is one eigenvector corresponding to an eigenvalue.

Kai: Does NumPy always sort the eigenvalues, or can they appear in any order?

Dr. Lena Hartmann: NumPy's eig function doesn't guarantee a specific order for the eigenvalues or eigenvectors. If order matters, like for PCA, you typically need to sort them yourself.

Repeated transformations reveal the dominant direction

Dr. Lena Hartmann: Repeatedly applying matrix A reveals how vector directions change, crucial for understanding dynamic systems.

Kai: So applying a matrix many times is like iterating a process?

Dr. Lena Hartmann: Yes, A to the power of k times vector x means applying A k times, simulating the vector's evolution over steps.

Dr. Lena Hartmann: Eigenvalues influence this. A large absolute value of lambda means vectors in its eigenvector's direction grow rapidly, dominating the transformation.

Kai: So the largest eigenvalue wins in the long run?

Dr. Lena Hartmann: Generally, yes. If the absolute value of lambda is less than one, the component along that eigenvector fades to zero under repeated application. If it's greater than one, that component grows in magnitude.

Dr. Lena Hartmann: If lambda is negative, each application flips the vector component's direction, causing oscillation.

Kai: Does this relate to stability in dynamical systems, like how a system might settle or diverge?

Dr. Lena Hartmann: Absolutely. Eigenvalues indicate system stability; if all have absolute values less than one, the system converges.

Diagonalization is a change of viewpoint

Dr. Lena Hartmann: Diagonalization changes our point of view on what a matrix does. Instead of watching the matrix mix coordinates together, we choose new axes along which it only stretches, and in those axes the action becomes simple.

Kai: Is this a change of basis?

Dr. Lena Hartmann: Yes, to the eigenbasis. The core relationship is A equals P D P inverse, and it holds whenever A has enough linearly independent eigenvectors to fill an invertible matrix P. Let us take the three pieces one at a time.

Dr. Lena Hartmann: The columns of P are the eigenvectors of A, and together they form the new coordinate system we are moving into. Reading a vector in that system tells us how much of each eigendirection it contains.

Kai: And D holds all the eigenvalues?

Dr. Lena Hartmann: Yes, D is a diagonal matrix of A's eigenvalues. These are the scaling factors, or stretches, along each eigenvector direction. Simple scaling.

Dr. Lena Hartmann: P inverse transforms vectors into this new basis. D then scales. P transforms back to the original.

Dr. Lena Hartmann: Here is why that identity holds. Multiply A by P, whose columns are the eigenvectors. Column by column, A times v i is lambda i times v i, so A P is the same list of columns with each one scaled by its own eigenvalue. That is exactly P times the diagonal matrix D. So A P equals P D, and multiplying on the right by P inverse gives A equals P D P inverse.

Dr. Lena Hartmann: Diagonalization simplifies matrix operations by transforming to a coordinate system where it just stretches along its axes, clarifying its action.

PCA turns spread into directions

Dr. Lena Hartmann: Principal Component Analysis, or PCA, is a powerful technique for finding the most important directions of variance, or spread, within your data.

Kai: So PCA uses eigenvalues and eigenvectors to find these 'most important directions'?

Dr. Lena Hartmann: Exactly. PCA solves an eigenvalue problem for the data's covariance matrix. This equation, C u equals lambda u, is central to understanding how.

Dr. Lena Hartmann: Let me show you why, rather than just asserting it. Take any unit direction u and project the centred data onto it: each point x becomes the single number u transpose x. Square those projections and average them, and since the average of x x transpose is the covariance matrix C, the variance along u is u transpose C u. So it is not a definition, it falls out.

Kai: So finding the direction of most spread means maximising that expression?

Dr. Lena Hartmann: Exactly, and subject to u being a unit vector, otherwise you could inflate it by just making u longer. Attaching a Lagrange multiplier to that constraint and setting the gradient to zero gives C u equals mu u. That is an eigenvalue equation, so the directions of greatest spread are the eigenvectors of C, not by convention, but because the maximisation lands there.

Kai: And then what does the eigenvalue itself mean?

Dr. Lena Hartmann: Substitute back. The variance along an eigenvector u is u transpose C u, and C u is lambda u, so it becomes lambda times u transpose u, which is lambda. The eigenvalue is the variance along its own direction.

Dr. Lena Hartmann: Here, C represents the covariance matrix of your dataset, which captures how different features vary together.

Kai: So u are the principal components, the directions?

Dr. Lena Hartmann: That's right, u represents the principal directions. These are the directions in which your data shows the most variance.

Dr. Lena Hartmann: And lambda, the corresponding eigenvalue, tells us the amount of variance, or spread, along that specific principal direction. A larger lambda indicates a more significant direction of data spread.

PCA in runnable Python

Dr. Lena Hartmann: Now, let's explore how Principal Component Analysis can be implemented using the power of NumPy.

Dr. Lena Hartmann: The script first generates a two-dimensional data cloud and then centers it. It computes the covariance matrix, then calls NumPy's linear algebra function to extract eigenvalues and eigenvectors. The plotted arrow represents the first principal component direction, scaled by the square root of its eigenvalue, so its length is one standard deviation along that direction. The eigenvalue itself is the variance.

Kai: Then the covariance eigenvectors are literally the axes PCA uses.

Dr. Lena Hartmann: That's right. The first bullet emphasizes centering data because PCA measures spread around the mean. This step ensures variance is measured from a central point, giving a true picture of the data's dispersion.

Kai: So the eigenvectors of the covariance matrix are the principal components?

Dr. Lena Hartmann: Yes, they directly correspond to the principal components. The largest eigenvalue indicates the direction with the most variance, which is the first principal component, capturing the most significant spread in your data.

Common traps

Dr. Lena Hartmann: Before we conclude, let's address three common misunderstandings about eigenvalues and eigenvectors.

Dr. Lena Hartmann: First, the zero vector is never an eigenvector. It satisfies equations trivially and carries no direction. An eigenvector must represent a meaningful, non-zero direction.

Kai: The second trap means if I double an eigenvector, it is still on the same eigenvector line.

Dr. Lena Hartmann: Perfect. That's exactly right. Eigenvectors are about directions, not specific length. If v is an eigenvector, then any non-zero scalar multiple, like two v or negative five v, is also an eigenvector for the same eigenvalue. They represent the same direction.

Kai: So, are there always real eigenvalues, or can they be complex?

Dr. Lena Hartmann: Yes, they can be complex. Third, some transformations rotate so strongly that no real direction stays on its line, leading to complex eigenvalues. While important in advanced applications like oscillatory systems, we won't go deep into them now.

Checkpoint: spot the eigenpair

Dr. Lena Hartmann: Quick checkpoint. Here is a different matrix, M, and a vector. Decide whether they form an eigenpair.

Dr. Lena Hartmann: Matrix M sends the vector two, zero to the vector six, six. Remember that an eigenvector keeps its direction, and only its length may change.

Kai: So is six, six just a scaled copy of two, zero?

Dr. Lena Hartmann: Let us check. If six, six were lambda times two, zero, then two lambda would have to be six, giving lambda equals three. But the second component needs zero times lambda to equal six, and zero times three is zero, not six. The two components demand different answers, so no single lambda works. Two, zero is not an eigenvector of M.

The one-sentence takeaway

Dr. Lena Hartmann: To recap, the core insight of eigenvalues and eigenvectors can be summarized as one key takeaway. This is what I want you to remember from our discussion.

Dr. Lena Hartmann: This equation, A times v equals lambda times v, shows matrix A scales vector v by lambda, without changing its direction. It defines a stable direction.

Kai: So v is the direction, and lambda is how much it's scaled?

Dr. Lena Hartmann: Precisely. Vector v is the eigenvector: a special direction transformation A preserves. It's like finding fundamental axes of change.

Dr. Lena Hartmann: And lambda is the eigenvalue, quantifying the change. It tells us if the eigenvector is stretched, shrunk, or flipped, providing critical information about the transformation's effect.

Kai: And these ideas link directly to AI applications like PCA, right?

Dr. Lena Hartmann: Indeed. These concepts are powerful for AI: Principal Component Analysis for dimensionality reduction, search engine ranking, and analyzing stability in dynamical systems.

Exit ticket

Dr. Lena Hartmann: Before we wrap up, here's an exit ticket to help solidify your understanding of eigenvalues and eigenvectors. It has a practice problem to apply what you've learned and a reflection question to encourage deeper thought.

Dr. Lena Hartmann: First, look at the matrix given here, with entries four, zero on the top row and zero, two on the bottom, together with the vector one, zero. The matrix is diagonal, which is the easiest case: a diagonal matrix stretches each coordinate axis on its own, without mixing them.

Kai: So we need to multiply A by v first?

Dr. Lena Hartmann: Yes, calculate the matrix-vector product. For your practice problem, your task is to find the product of matrix A and vector v, which is A times v, and to determine the corresponding eigenvalue, lambda.

Dr. Lena Hartmann: Then, for a moment of reflection, think about what makes a direction stable in the context of linear transformations. What properties of an eigenvector allow it to maintain its line? Take your time to work through these before moving on, as they are crucial for reinforcing your understanding.

Keep going: the next lesson builds on this one.

Thanks for spending time with Thinking in Math. Leave your questions in the YouTube comments and we'll tackle them in future videos.