Part 1: Foundations

Computers don't understand words—they only understand numbers. So we need to convert text into numbers. We do this by creating a vocabulary: a list of all words the model knows, where each word gets a unique ID.

For our calculator, the vocabulary might look like:

{ "zero": 0, "one": 1, "two": 2, "three": 3, ... "plus": 12, "minus": 13, ... }

Now we can convert our input:

"two plus three"
 ↓
["two", "plus", "three"]  → split into words
 ↓
[2, 12, 3]                → look up each word's ID

Each word becomes its ID from the vocabulary. The model never sees "two"—it only sees the number 2.

A single number like "2" doesn't tell the model much. Is "two" similar to "three"? Is it related to "plus"? To capture these relationships, we convert each token ID into a vector—a list of numbers that represents the word's meaning.

Think of it like coordinates. In a city, "123 Main St" tells you exactly where something is. Similarly, a vector like [0.2, -0.5, 0.8] places a word in "meaning space":

"two"   → [0.9, 0.1, 0.2, ...]   (e.g. 64 numbers)
"three" → [0.85, 0.15, 0.25, ...]  (e.g. 64 numbers)
"plus"  → [0.1, 0.8, 0.3, ...]    (e.g. 64 numbers)

Why so many numbers? The number of values (e.g. 64) is called the "embedding dimension"—a choice we make when designing the model.

Imagine describing a person with just 2 numbers: height and weight. That's useful, but limited. Now add age, income, years of education—each number captures a new dimension of who they are. With more numbers, you can distinguish between people more precisely.

The same idea applies to words:

• 2 dimensions: Can barely distinguish words
• 64 dimensions: Enough for a simple task like our calculator
• 12,000+ dimensions: What models like GPT-4 use to capture nuance in all of human language

More dimensions = more detail, but slower training. We'll use 64 as an example throughout this tutorial—it's small enough to understand but powerful enough for our calculator.

What do the individual numbers mean? We don't know! The model learns these values during training. But for our calculator, we can imagine what some dimensions might capture:

Notice how "two" and "three" have similar vectors (both are numbers), while "plus" is quite different (it's an operation). Words with similar meanings end up close together in this space.

Our input becomes:

[2, 12, 3] → [[0.9, 0.1, ...], [0.1, 0.8, ...], [0.85, 0.15, ...]]
              "two"          "plus"          "three"

We have a problem. Look at these two inputs:

"five minus three" → answer: "two"
"three minus five" → answer: "negative two"

The words are the same, but the order matters. With just embeddings, the model sees the same three vectors in both cases—it doesn't know which word came first!

The solution: we add position information to each embedding. Think of it like seat numbers in a theater—each word gets a fixed position marker:

"two plus three"

Position 1: "two"   → [0.9, 0.1, ...] + [position 1 info] → [0.92, 0.15, ...]
Position 2: "plus"  → [0.1, 0.8, ...] + [position 2 info] → [0.13, 0.85, ...]
Position 3: "three" → [0.85, 0.15, ...] + [position 3 info] → [0.88, 0.21, ...]

Now each vector contains both what the word is AND where it appears. The same word at different positions will have slightly different vectors.

Imagine you're reading a sentence and someone asks you what it means. You don't look at each word separately—you naturally consider how words relate to each other.

Right now, our model has three separate vectors for "two", "plus", and "three". But these vectors are like three people in separate rooms—they can't talk to each other. To solve "two plus three", the model needs to understand the relationship between these words.

This is what "attention" does. Think of it like a group discussion:

Imagine the words sitting in a meeting room:

"plus": Hey everyone, I'm an operation. Who am I working with?
"two":  I'm a number! I'm sitting before you.
"three": I'm also a number! I'm sitting after you.
"plus": Got it—I need to ADD "two" and "three" together.

In technical terms, each word asks a question ("what's relevant to me?") and every other word offers an answer ("here's my information"). The model assigns an attention weight to each pair—a score from 0 to 1 indicating importance.

When "plus" looks at the other words, it assigns high weights (0.8) to "two" and "three" because they're relevant, and low weights (0.1) to irrelevant words. These weights determine how much each word influences the final understanding.

After this "discussion", each word's vector gets updated with information from the others:

The transformer repeats this "discussion" through multiple layers (we'll use 2-4). Each round of discussion refines the understanding:

• Layer 1: Basic relationships ("plus connects two numbers")
• Layer 2: Deeper understanding ("we're computing 2 + 3")
• Layer 3: Final insight ("the answer should be 5")

Here's a simplified view of how learning works:

Training example #1:
  Input: "one plus two"  →  Model guesses: "seven" (wrong!)
  Correct answer: "three"
  Model adjusts: "Hmm, I should pay more attention to 'one' and 'two' when I see 'plus'"

Training example #2:
  Input: "four plus three"  →  Model guesses: "six" (closer!)
  Correct answer: "seven"
  Model adjusts: "I'm getting better at addition, but need more practice"

...after 1000 examples...

Training example #1000:
  Input: "five plus one"  →  Model guesses: "six" (correct!)
  Model has learned: when "plus" appears, add the numbers around it

Each wrong answer nudges the model's internal numbers (weights) slightly. After thousands of nudges, the model has "learned" that 'plus' means addition—without us ever explicitly programming that rule.

It's like teaching a child math: you don't explain the neural pathways in their brain—you just show them "2 + 3 = 5" enough times, and their brain figures out the patterns.

After the transformer layers, the model has a rich understanding of "two plus three". But it's still just vectors—we need an actual answer.

The output layer asks: "Given everything I've learned about this input, which word in my vocabulary is most likely to come next?"

It scores every word in the vocabulary:

For input "two plus three", the model scores each possible answer:

"zero"   → 0.1%
"one"    → 0.2%
"two"    → 0.3%
"three"  → 0.5%
"four"   → 2.1%
"five"   → 94.2%  ← highest!
"six"    → 1.8%
"seven"  → 0.4%
...
"plus"   → 0.01%
"minus"  → 0.01%

These percentages are called probabilities. They add up to 100% across all words in the vocabulary. The model is saying: "I'm 94.2% confident the answer is 'five'."

The Math Behind It

How does the model convert vectors into probabilities? Two steps:

1. Linear layer: Multiply the final vector by a weight matrix to get a "score" for each word. Higher score = model thinks this word is more likely.

final_vector (64 numbers) × weight_matrix (64 × 30) = scores (30 numbers)

scores = [-2.1, -1.8, -1.5, -0.9, 0.3, 4.2, 0.1, -0.5, ...]
           zero   one   two  three four  five  six seven

Where does the weight matrix come from? We create it, and the model learns its values during training.

• 64 = our embedding dimension (the size of each word vector)
• 30 = our vocabulary size (how many words the model knows)

2. Softmax: Convert raw scores into probabilities (0-100%) that sum to 100%. The formula is:

probability(word) = e^(score for word) / sum of e^(all scores)

For "five" with score 4.2:
  e^4.2 = 66.7
  sum of all e^scores = 70.8
  probability = 66.7 / 70.8 = 94.2%

Softmax has a useful property: it makes high scores much higher and low scores much lower. A score of 4.2 vs 0.3 becomes 94.2% vs 2.1%. This makes the model "confident" in its best guess.

We now have probabilities for every word. How do we pick the final answer? There are a few strategies:

Strategy 1: Greedy (pick the highest)

The simplest approach: always pick the word with the highest probability.

"five"  → 94.2%  ← Pick this one!
"four"  → 2.1%
"six"   → 1.8%
...

Output: "five"

This is called greedy decoding. It's deterministic—the same input always gives the same output. Perfect for math where there's only one right answer.

Strategy 2: Sampling (add randomness)

Instead of always picking the top word, we randomly choose based on the probabilities. Higher probability = more likely to be chosen, but not guaranteed.

Run 1: "five"  (94.2% chance → picked!)
Run 2: "five"  (94.2% chance → picked!)
Run 3: "four"  (2.1% chance  → lucky pick!)
Run 4: "five"  (94.2% chance → picked!)

This adds variety. When writing a story, you don't want the same words every time. Sampling makes the model more creative.

Strategy 3: Temperature (control randomness)

We can adjust how "confident" the model is using a parameter called temperature:

• Low temperature (0.1): Makes high probabilities even higher. Model becomes very confident, less creative.
• Temperature = 1: Use probabilities as-is.
• High temperature (2.0): Flattens probabilities. Model becomes more random, more creative.

Original:     "five" 94.2%, "four" 2.1%, "six" 1.8%
Low temp:     "five" 99.9%, "four" 0.05%, "six" 0.03%  (almost certain)
High temp:    "five" 60%, "four" 15%, "six" 12%        (more random)

What Do Real Models Use?

And that's it! The model outputs "five", and we've successfully computed "two plus three" = "five".