When I first started learing about transfer I found a couple of concepts were the key to understand the transformer model properly and those are wordembedding or vectorization of text, attechteion mechanism, multi head attention, self-supervised learning. In this post I have tried to explain how they are linked together and how they form the build blocks of the transfomer model.
These ideas are linked because they form the core of how a Transformer language model learns from text and uses context.
Here’s the big picture:
Text ? tokens ? vectors ? multi-head attention ? contextual representations ? predictions
Self-supervised learning is the training process that teaches those components to make useful predictions.
Let’s build that up using:
“The cat slept because it was tired.”
1. Word embeddings: representing text as numbers
Neural networks work with numbers, not words. First, the sentence is split into tokens—words, parts of words, or punctuation.
Each token is mapped to a list of numbers called an embedding vector:
These numbers are illustrative. In a real model, embeddings are learned during training.
An embedding is a kind of vector: one that represents something, such as a token, in a numerical space. Learning can arrange that space so useful similarities and relationships become accessible to the model.
But there’s a limitation: an initial token embedding alone doesn’t explain what the token means in this particular sentence. The word “it” could refer to a cat, a car, or something else.
That’s where attention comes in.
2. Attention: using context to update representations
Attention lets a token gather information from other tokens.
In our sentence, when processing “it,” a useful attention pattern might give substantial weight to “cat.” This helps the model build a representation of “it” that includes relevant context.
The basic process is:
- Compare the current token with the available tokens.
- Assign weights indicating how relevant each one is.
- Take a weighted combination of information from those tokens.
Attention uses three vectors derived from each token’s current representation:
| Vector | Intuition |
| Query (Q) | What information am I looking for? |
| Key (K) | What information can I be matched on? |
| Value (V) | What information will I contribute if selected? |
These aren’t literal questions or labels; they’re learned numerical transformations.
Queries are compared with keys to produce attention weights. Those weights are then used to combine values:
[ \mathrm{Attention}(Q,K,V)
\mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V ]
In plain English: compare ? weight ? mix information.
When queries, keys, and values come from the same sequence, this is called self-attention. In a next-token language model, a mask prevents tokens from attending to future tokens.
3. Multi-head attention: several attention operations in parallel
One attention operation produces one way of combining information. Multi-head attention runs several attention operations, each with its own learned query, key, and value transformations.
This lets the model capture different relationships at the same time. For example, different heads might capture:
- Connections between pronouns and nouns.
- Relationships between actions and participants.
- Nearby word patterns.
- Dependencies spanning longer distances.
These are possible patterns, not fixed jobs assigned to particular heads.
The head outputs are joined and transformed into one output:
Transformer layers combine this attention with other components, including feed-forward networks. Stacking layers allows representations to become progressively more context-sensitive. Position information also helps the model account for token order.
4. Self-supervised learning: how the model learns all of this
So far, we’ve described the machinery. But how does it learn good embeddings and attention patterns?
Through self-supervised learning: training where the text itself supplies the target, rather than requiring people to label every example.
For a next-token language model:
| Input | Target from the original text |
|---|---|
| “The cat” | “slept” |
| “The cat slept” | “because” |
| “The cat slept because” | “it” |
The model:
- Converts input tokens into embeddings.
- Processes them through Transformer layers, including multi-head attention.
- Predicts a probability distribution over the next token.
- Measures how poorly its prediction matches the actual next token.
- Uses backpropagation to update its parameters—including embeddings and attention transformations.
Across many examples, capturing grammar, meaning, and contextual relationships helps the model predict better.
Another self-supervised approach is masked-token prediction: hide a token and train the model to recover it from the surrounding text.
How they fit together
| Concept | Its role |
|---|---|
| Vectors / embeddings | Represent tokens numerically. |
| Attention | Mix information from relevant token representations. |
| Multi-head attention | Learn multiple ways to mix that information in parallel. |
| Self-supervised learning | Train the model using targets derived from the data itself. |
The key distinction: embeddings and attention are parts of the model; self-supervised learning is a way to train that model.
And despite the similar names, self-attention and self-supervised learning mean different things: the first describes where attention gets its information; the second describes where training targets come from.
