Embeddings: From Words to Vectors

From A approximately equals U Sigma V transpose $(L09)$ to word meaning as vectors

Dr. Lena Hartmann: Last time in Lecture 09, we used the singular value decomposition to treat a big matrix as rotate, stretch, rotate, and we saw how low-rank structure lets us compress without losing the main patterns.

Kai: So today we are taking that matrix compression mindset and applying it to language, like turning a huge table of words into something geometric we can work with?

Dr. Lena Hartmann: In this lecture, we pivot from compressing a matrix to representing meaning: we will store each word as a vector, so that similar words end up near each other in that vector space.

Dr. Lena Hartmann: Our guiding idea is an embedding matrix, E, that you can think of as a learned meaning table, and we will build intuition for why simple dot products between word vectors can drive tasks like search and attention.

Why embeddings matter in AI: everything becomes "find nearby vectors"

Dr. Lena Hartmann: A big reason embeddings matter is that they give us one common language for many different AI tasks: vectors in a space.

Kai: When people say vector database or semantic search, is that basically just storing these vectors and doing nearest neighbor lookup at scale?

Dr. Lena Hartmann: Once text, images, users, or products live as points, a lot of problems turn into the same question: what is closest to what, or what is most similar?

Dr. Lena Hartmann: That simple idea powers search, recommendations, and even how large language models compare tokens during attention, so "find nearby vectors" becomes a reusable tool across the whole stack.

Code-first idea: "lookup" is just multiplying by a one-hot vector

Dr. Lena Hartmann: Before we interpret the formula on the slide, keep one simple picture in mind: an embedding table is just a big list of vectors, one vector per token in the vocabulary.

Kai: So the word itself is not special, it is just an index, and the model learns a row of numbers for that index?

Dr. Lena Hartmann: When you look up a token, you are really choosing the one vector associated with that token, like pulling the correct card from a card catalog.

Dr. Lena Hartmann: In a moment we will write this as multiplying by a one hot vector: the one hot has a single one at the token’s index, so the multiplication keeps exactly that embedding and zeroes out the rest.

NumPy demo: build a tiny embedding matrix and do the "lookup"

Dr. Lena Hartmann: Let’s do a tiny, concrete embedding example: we will build a small table of word vectors, then retrieve the vector for one word in two different ways and compare the results.

Kai: Two ways meaning: either pick the row for the word’s token ID, or use a one hot vector and multiply to select that row?

Dr. Lena Hartmann: Exactly. Before we reveal anything, make a prediction: if the word is queen, will the direct row lookup equal the one hot times the matrix, and what two numbers do you expect for the queen vector given the numbers we coded into the embedding matrix?

Dr. Lena Hartmann: Now we print both results and a same check. If they ever differ, it is usually a shape issue or a type issue, not a different idea. In this setup, both methods should return the exact same two number vector for queen.

Visualization: embeddings turn tokens into points you can cluster

Dr. Lena Hartmann: Once tokens are points in space, geometry starts doing useful work for you.

Kai: Is the main reason we like a two dimensional plot just that we can see clusters, even though real embeddings might have hundreds or thousands of dimensions?

Dr. Lena Hartmann: An embedding takes each token and assigns it a location in a vector space, so the token becomes a point you can compare to other points.

Dr. Lena Hartmann: When you visualize those points in two dimensions, clusters often appear: words used in similar contexts end up near each other, and unrelated words drift apart.

Kai: And in practice, if the embedding space is learned well, I should be able to take a query vector and just ask for the nearest points to get candidates for search or recommendations?

Key similarity tool: dot product and cosine similarity

Dr. Lena Hartmann: Now we need the one operation that makes embeddings useful: a similarity score between vectors. The goal is simple: given two embeddings, return a single number that tells us how closely they point in the same direction.

Kai: If dot product depends on vector length, does that mean a really frequent word could get a bigger score just because its embedding has a larger norm?

Dr. Lena Hartmann: A common starting point is the dot product: it tends to be larger when two vectors are aligned and smaller when they are opposed or unrelated. It gives us an easy alignment score between a query vector and a candidate vector.

Dr. Lena Hartmann: But dot product also depends on vector length, so we often normalize and use cosine similarity instead. Cosine similarity compares direction rather than magnitude, and then we can rank candidates by sorting them by similarity to the query.

Worked example: nearest neighbors with cosine similarity

Dr. Lena Hartmann: We are going to do a worked nearest neighbor search in an embedding space using cosine similarity, using a tiny toy vocabulary so the ranking is easy to reason about.

Dr. Lena Hartmann: Before we sort anything, make a concrete prediction: for the query word queen, excluding queen itself, what will be the top three neighbors by cosine similarity, and which word will be last?

Dr. Lena Hartmann: After we compute cosine similarity against every token, queen itself will rank first with a score of about one, but for neighbor search we ignore that self match; among the remaining words you should expect king and woman near the top, with apple or orange at the bottom because their vectors point in the opposite direction.

Dr. Lena Hartmann: If woman ends up closer or farther than you expected, look at the hand-crafted coordinates: cosine similarity cares about direction, not length, so vectors that share a similar angle with queen will score higher even if their raw coordinates differ.

Where do embeddings come from? learning makes geometry

Dr. Lena Hartmann: We do not hand-design the embedding table in practice; we learn it so that useful predictions become easy.

Kai: What is the learning signal in real systems: is it mostly next word prediction, or can it be things like clicks and purchases too?

Dr. Lena Hartmann: The key idea is that the training task creates pressure to organize vectors so that the right choice scores higher than the wrong ones.

Dr. Lena Hartmann: When this works well, items that behave similarly in data end up close together in direction and distance, so geometry becomes a useful summary of meaning or behavior.

AI connection: attention is "embeddings $+$ dot products $+$ weighted sum"

Dr. Lena Hartmann: If embeddings are the dictionary, attention is how the model queries the dictionary on the fly.

Kai: So attention is basically comparing the current token vector to other token vectors, and then mixing information based on those comparison scores?

Dr. Lena Hartmann: Instead of using one fixed meaning for a word, attention builds a context-specific meaning by deciding which other words matter right now.

Dr. Lena Hartmann: Conceptually, it is embeddings turned into comparisons, and those comparisons turned into a weighted mixture of information to produce the next representation.

SVD connection: embedding matrices are huge, so we compress them

Dr. Lena Hartmann: Embeddings are just numbers arranged in a big table: one row per vocabulary item and one vector of length d per row. When the vocabulary is large, that table becomes one of the biggest chunks of parameters in the model.

Dr. Lena Hartmann: A classic idea from singular value decomposition is that many large matrices can be approximated well by something low rank. Instead of keeping the full embedding matrix E, we represent it using two smaller matrices whose inner dimension is r.

Dr. Lena Hartmann: That gives two practical wins: you store far fewer numbers, and some similarity computations can be reorganized to do most of the work in rank r, for example by precomputing B transpose times B. If you have heard of LoRA, treat it as a teaser: it also uses a low-rank product, but as an update to a weight matrix, and we will not unpack the shapes here.

Checkpoint: what you should be able to say out loud

Dr. Lena Hartmann: Let’s pause for a quick checkpoint. By the end of this lesson, you should be able to say, in your own words, what an embedding is, how we compare embeddings, and how training makes the space useful.

Kai: Let me try: an embedding is just a learned vector for each token, and we use dot product or cosine to turn those vectors into a ranking of what is most related to a query.

Dr. Lena Hartmann: First: an embedding table is just a matrix, and each row is a vector that represents one item, like a word, in d dimensions.

Dr. Lena Hartmann: Second and third: we turn those vectors into rankings using dot product or cosine similarity, and we train with an objective that moves vectors so that meaningful neighbors end up close together.

Exit ticket: compute a similarity, then interpret it

Dr. Lena Hartmann: Before we wrap up, here is your exit ticket: we are going to compute one cosine similarity and then say what that number means in plain language.

Kai: If I compute it and get something close to one, that means the two vectors point in almost the same direction, and if it is close to minus one, they point opposite, right?

Dr. Lena Hartmann: Remember the idea: cosine similarity is about direction. It asks whether two vectors point in a similar way, not whether they are long or short.

Dr. Lena Hartmann: Do this once by hand and you will never forget what cosine similarity is measuring, and then connect it to one spot in an AI system where picking the nearest vectors would be useful.

Thank you for watching!

Thanks for watching. Subscribe and share if you found this useful—see you next time!