Hyperparameter Comparison Table
Embedding Dimension (Hidden Size)=Number of Heads×Dimension of Each Head
|
Hyperparameter Category |
Feature / Setting |
DeepSeek-V3 / R1 [3] |
ChatGPT (GPT-4 Baseline) |
Claude 3.5 Series (Estimate) |
|
Attention Core |
Hidden Size (Embedding Dim) |
7,168 |
12,288 |
8,192 |
|
Number of Layers |
61 |
120 |
80 to 96 |
|
|
Attention Heads per layer |
128 |
96 |
64 |
|
|
Per-Head Sizing |
Per-Head Seeker Size (Q or K)
Dimension Size |
128 |
128 |
128 |
|
|
Per-Head Meaning Size (V) |
40 |
128 |
128 |
| The Stitched Output
|
Total Output Rows (Attention Heads × V)
|
5,120
|
12,288
|
8,192
|
|
Layer Matrices |
Seeker ($W_Q$) Matrix Size |
$7168 \times 1536$ |
$12288 \times 12288$ |
$8192 \times 8192$ |
|
Label ($W_K$) Matrix Size |
$7168 \times 128$ |
$12288 \times 12288$ |
$8192 \times 8192$ |
|
|
Meaning ($W_V$) Matrix Size |
$7168 \times 7168$ |
$12288 \times 12288$ |
$8192 \times 8192$ |
|
|
Output Projection Matrix Size |
$5120 \times 7168$ |
$12288 \times 12288$ |
$8192 \times 8192$ |
|
|
Remaining Structural |
Vocabulary Size (Tokens) |
129,280 |
100,256 |
65,000 |
|
Max Context Length (Tokens) |
4,096 (Base) |
8,192 (Base) |
4,096 (Base) |
|
|
FFN Intermediate Dimension |
18,432 |
49,152 |
32,768 |
|
|
Total MoE Experts |
256 |
16 |
0 (Dense Structure) |
|
|
Active Experts per Token |
8 |
2 |
N/A |
|
|
Optimization Rules |
Peak Learning Rate (LR) |
$2.2 \times 10^{-4}$ |
$2.0 \times 10^{-4}$ |
$1.5 \times 10^{-4}$ |
|
Global Batch Size |
15,360 sequences [3] |
16,384 sequences |
8,192 sequences |
|
|
Total Tokens per Batch |
~63 Million [3] |
~134 Million |
~33 Million |
|
|
AdamW Optimizer Betas |
$\beta_1=0.9, \beta_2=0.95$ [3] |
$\beta_1=0.9, \beta_2=0.95$ |
$\beta_1=0.9, \beta_2=0.95$ |
|
|
Weight Decay |
0.1 [3] |
0.1 |
0.1 |
Key Takeaways from the New Additions
1. The FFN Intermediate Dimension (The Calculation Engine)
Right after the attention mechanism merges word contexts, the tensor passes into the Feed-Forward Network (FFN). Notice how the dimension size explodes here. For Claude, it jumps from 8,192 to 32,768 columns. The model temporarily inflates the vector size to give the GPU maximum mathematical space to compute factual knowledge before squashing it back down.
2. The Token Volume per Batch (Optimization)
Look at the Total Tokens per Batch. When DeepSeek runs its backward pass to rewrite its weight matrices, it evaluates roughly 63 million tokens simultaneously across thousands of connected GPUs. GPT-4 pushes this even higher to over 134 million tokens. This massive scale ensures the mathematical gradients (the change vectors) are highly stable and representative of general language patterns. [1]
3. The Shared Industry Standards
Notice that despite completely different company cultures and network sizes, all three architectures use a Weight Decay of 0.1 [3] and AdamW Betas of 0.9 and 0.95. These specific optimization hyperparameters have become universal constants in AI research because they prevent network collapse across almost all large-scale deep learning training setups.
- The Content Half (64 numbers): Handled by the lossy, compressed bottleneck pool.
- The Position Half (64 numbers): Left completely uncompressed to carry the precise geometric RoPE timeline. [1]
ONE SINGLE DEEPSEEK ATTENTION HEAD
[----------------------- 128 Numbers Total -----------------------]
[ 64 Numbers for Content ] + [ 64 Numbers for Position ]
(From Compressed Pool) (Raw, Uncompressed RoPE)
- ChatGPT (GPT-4):
\(96\text{\ heads}\times 128\text{\ per-head\ size}=12,288\text{\ Embedding\ Dimension}\) - Claude 3.5 Series:
\(64\text{\ heads}\times 128\text{\ per-head\ size}=8,192\text{\ Embedding\ Dimension}\) [1]
2. Why Are They Related Physically?
They are related because of a step called Slicing. [1]
When your [Batch, Length, 8192] tensor enters a Claude layer, the GPU doesn’t actually create 64 physically separate matrices. Instead, it takes that giant 8,192-dimensional vector and slices it like a loaf of bread into 64 equal chunks of 128 numbers each.
text
TOTAL EMBEDDING DIMENSION (e.g., 8,192 numbers)
[————————————————————-]
Slice 1 Slice 2 Slice 3 Slice 64
[128 nos] [128 nos] [128 nos] … [128 nos]
↓ ↓ ↓ ↓
Head 1 Head 2 Head 3 Head 64
Use code with caution.
Each Attention Head is assigned exactly one slice. If the number of heads didn’t divide perfectly into the embedding dimension, you would have leftover numbers, and the matrix math would break. [1]
3. The DeepSeek Exception (Multi-Head Latent Attention)
You might notice that DeepSeek-V3 breaks this rule if you look at the table:
\(128\text{\ heads}\times 128\text{\ per-head\ size}=16,384\)
But DeepSeek’s embedding dimension is only 7,168! [1, 2]
How is this possible? DeepSeek uses a custom invention called Multi-Head Latent Attention (MLA).
Instead of slicing the main embedding dimension directly, DeepSeek takes the 7,168 numbers and compresses them down into a tiny bottleneck pool of just 512 numbers (the “latent space”). [1, 2, 3, 4]
The 128 heads then pull and expand their data out of that 512-dimensional pool instead of the main embedding vector. This mathematical trick is a major reason why DeepSeek runs so much cheaper and faster than its competitors. [1, 2]
- (Layers 1–10) only handle basic surface patterns, like punctuation and simple grammar.
- (Layers 11–50) start combining those patterns into abstract meanings, like the sentiment of a sentence.
- Deep layers (Layers 50–120) handle high-level logic, fact retrieval, and long-range context tracking.
- ChatGPT (GPT-4 Baseline): Relies heavily on a massive physical neural network with 120 layers to absorb and process vast amounts of data.
- DeepSeek-R1: Uses a much shallower physical body (61 layers), but it is trained using a highly advanced Reinforcement Learning (RL) algorithm called GRPO. [1, 2]
Standard 120-Layer Model: Input → Pass through 120 layers (1 time) → Output (Fast)
DeepSeek-R1 (61 Layers): Input → Pass through 61 layers 🔄 (Looping thousands of times to "think") → Output (Highly Reasoned)
