Why Tokens are Converted to Embeddings

Here is exactly why this translation to floating-point vectors (Embedding) is absolutely critical, broken down into the three things it enables:

1. It enables MEANING (Geometry of Semantics)

Computers can’t understand definitions, but they are incredible at geometry.

When the integer 789 fetches the embedding vector for "pizza", it pulls a set of ~4,000 floating-point numbers (e.g., [0.23, -1.44, 0.87, ...]). These numbers are coordinates in a high-dimensional space.

  • Words that are similar get coordinates that are physically close together in this space.

  • During training, the model adjusts these coordinates so that "pizza" and "burgers" end up near each other (both are foods), while "on" and "top" end up in a different neighborhood (prepositions).

  • The benefit: By measuring the distance (dot product) between numbers, the model can mathematically conclude, “Ah, ‘pizza’ and ‘burgers’ are interchangeable here,” without ever needing a human to tell it.

2. It enables CALCULUS (Learning from mistakes)

This is the non-negotiable reason. To train an LLM, we need to calculate the loss (how wrong the prediction was) and push that error backward through the network to adjust the weights.

  • With numbers: The model predicts "pizza" as a probability of 0.4 and "burgers" as 0.3. The correct answer (target) is a number: 1 for pizza, 0 for everything else. The loss function subtracts them: 1.0 - 0.4 = 0.6. We take the derivative (slope) of that error and use it to tweak the weights.

  • Without numbers: You cannot take a mathematical derivative of a string like "pizza". Calculus only works on continuous floating-point numbers. Translation to numbers is what makes backpropagation possible.

3. It enables PARALLEL MATRIX MULTIPLICATION (Speed)

Modern GPUs are essentially giant calculators optimized to multiply matrices together incredibly fast.

  • The integer 789 indexes a row in the embedding matrix.

  • The entire sentence tensor [42, 847, 336, 562] is used to slice out a matrix of 4 embedding vectors simultaneously.

  • The Attention mechanism is just: Q * K^T (massive multiplications of floating-point matrices). GPUs can do trillions of these operations per second. If we tried to process letters or strings directly, the GPU would be useless.

The Exact Step-by-Step “Translation” in your heap:

Let’s trace "I love to eat" through the modern pipeline:

 
 
Step What it is Data Type Why?
1. Raw Text "I love to eat" String Human-readable, but useless to a computer.
2. Tokenization [42, 847, 336, 562] Integers (Indexes) To find the right row in the vocabulary table. (Like a book index).
3. Embedding Lookup [[0.1, -0.5...], [0.9, 0.2...], ...] Floating-point vectors To do math. These numbers represent meaning, distance, and context.
4. Attention Math [2.4, -1.1, 0.7...] (New vectors) Floating-point To mix contexts. These numbers combine the meanings of “I”, “love”, “to”, and “eat” into a unified mathematical representation of that specific phrase.
5. Output Logits [0.1, 0.8, 0.05 ...] (over 50k vocab) Floating-point scores To rank probabilities. The highest score points back to an integer (e.g., index 789 for “pizza”).
6. Detokenization "pizza" String The model outputs the integer 789, and the tokenizer translates it back to text for you to read.

The ultimate irony:
The AI never actually “knows” that 789 is “pizza”. To the neural network, "I love to eat" is just a pattern of 16,000 floating-point numbers (4 words × 4,000 dimensions) that, through billions of matrix multiplications, happens to point toward another set of numbers that map to index 789.

Translation to numbers helps because language is discrete (words are either present or absent), but reality is continuous. Numbers allow the AI to blur the hard edges between words, find similarities, and mathematically calculate the most probable next step.

Loading