attention is a weighted probability distribution that determines how much focus to place on different parts of an input sequence. While your formula specifically computes the attention weights (or alignment scores), the complete attention mechanism multiplies these weights by a Value matrix V to produce a weighted sum of the inputs.

 

Causal Self-Attention=  \(\text{Causal\ Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}+M\right)V\)

 

 

 

 

To determine the matrix dimensions for the attention equation, let’s break down each component based on standard Transformer architecture conventions (such as Vaswani et al., “Attention Is All You Need”).

Variable Definitions & Dimensions

• $N$ (or $T$): Sequence length (number of tokens in the input/output sequence).

• $d_k$: Dimensionality of the Queries ($Q$) and Keys ($K$).

• $d_v$: Dimensionality of the Values ($V$). (Often $d_v = d_k$, but they can be separate).

• $M$: Causal mask matrix (used in autoregressive models like GPT to prevent attending to future tokens).

Component-by-Component Matrix Dimensions

1. Query ($Q$): $(N \times d_k)$

2. Key ($K$): $(N \times d_k)$

3. Key Transpose ($K^T$): $(d_k \times N)$

4. Matrix Multiplication ($QK^T$):

   $$(N \times d_k) \times (d_k \N \text{ or } N \times d_k \text{ transposed}) \rightarrow (N \times N)$$

   This creates a square matrix representing the raw alignment score between every token and every other token.

5. Scaling Factor ($\sqrt{d_k}$): A scalar value, so it does not change the dimensions of $(N \times N)$.

6. Causal Mask ($M$): $(N \times N)$

   Added element-wise to the $(N \times N)$ score matrix.

7. Softmax Function: Applied row-wise, maintaining the $(N \times N)$ shape.

   Attention Weights Matrix: $(N \times N)$

8. Value Matrix ($V$): $(N \times d_v)$

9. Final Causal Attention Output:

   $$\text{Attention Weights } (N \times N) \times V (N \times d_v) \rightarrow (N \times d_v)$$

Summary Table

Matrix / Component	Mathematical Notation	Dimensions
Query ($Q$)	$Q$	$N \times d_k$
Key ($K$)	$K$	$N \times d_k$
Key Transpose ($K^T$)	$K^T$	$d_k \times N$
Score / Mask Matrix	$\frac{QK^T}{\sqrt{d_k}} + M$	$N \times N$
Attention Weights	$\text{softmax}(\dots)$	$N \times N$
Value ($V$)	$V$	$N \times d_v$
Final Output	$\text{Attention}(Q,K,V)$	$N \times d_v$

 

 

for example  

if We have single head

Sequence length number of tokens =2048

attention head size = 1000

Dimensionality of the Values=1000

Since you specified a single head, the token embedding dimension (or projection space for that layer) is 1000.

 

 

 

Matrix / Component Symbol Shape / Dimensions Explanation Query Matrix $Q$ $2048 \times 1000$ (Sequence Length $\times$ Query Dimension $d_k$) Key Matrix $K$ $2048 \times 1000$ (Sequence Length $\times$ Key Dimension $d_k$) Key Transpose $K^T$ $1000 \times 2048$ Transposed from $K$ Value Matrix $V$ $2048 \times 1000$ (Sequence Length $\times$ Value Dimension $d_v$) Attention Mask $M$ $2048 \times 2048$ Square matrix matching the sequence length ($N \times N$) Product $QK^T$ $QK^T$ $2048 \times 2048$ $(2048 \times 1000) \times (1000 \times 2048)$ resulting in $N \times N$ Scaled Score + Mask $\frac{QK^T}{\sqrt{d_{k}}} + M$ $2048 \times 2048$ Element-wise addition preserves dimensions Attention Weights $\text{softmax}(\dots)$ $2048 \times 2048$ Applied row-wise, preserving the $N \times N$ shape Final Output Causal Attention $2048 \times 1000$ $(2048 \times 2048) \times (2048 \times 1000)$ resulting in Sequence Length $\times d_v$

 

While

 

 

Matrix	Symbol	Shape / Dimensions	Explanation
Query Weight Matrix	$W_q$	$1000 \times 1000$	($d_{model} \times d_k$) — Projects input embedding to Query space
Key Weight Matrix	$W_k$	$1000 \times 1000$	($d_{model} \times d_k$) — Projects input embedding to Key space
Value Weight Matrix	$W_v$	$1000 \times 1000$	($d_{model} \times d_v$) — Projects input embedding to Value space
Output Weight Matrix	$W_o$	$1000 \times 1000$	($d_v \times d_{model}$) — Projects the attention output back to the model dimension

 

 

 

 

 

 

 

 

 

 

 

Gemini:

To create the causal mask matrix for your specific text data, we must first map the dataset into a Vocabulary (unique words) and determine our sequence length.

When dealing with multiple sentences of different lengths, we use the length of the longest sentence as our max sequence length ($N$). Any shorter sentences are extended using a special pad token (e.g., <PAD>).

Here is how the causal mask matrix is created step-by-step for your dataset.

Let’s break down your sentences into tokens (words).

1. Sentence 1: [I, love, to, eat, pizza, with, extra, cheese, on, top] (10 tokens)
2. Sentence 2: [I, love, to, eat, burgers, with, extra, ketchup] (8 tokens)
3. Sentence 3: [On, top, of, the, pizza, is, extra, cheese] (8 tokens)
4. Sentence 4: [Cheese, and, burgers, go, well, together] (6 tokens)

The longest sequence is Sentence 1, which contains 10 tokens. Therefore, our core causal mask matrix must be an $10 \times 10$ matrix ($N = 10$). Shorter sentences will use a slice of this matrix or combine it with a padding mask.

For any sequence processed by the model, the causal mask matrix is generated using a lower-triangular structure. It places 0 where attention is allowed (past and present) and $-\infty$ where it must be blocked (future). [1]

For Sentence 1 (or any sequence padded to 10 tokens), the matrix $M$ is structurally created as follows:

To visualize exactly how this affects the text, we map the tokens of your sentences against the matrix rows (Queries) and columns (Keys).

Every word can only look at itself and words to its left.

When a sentence is shorter, modern frameworks typically generate a causal mask for the active tokens and merge it with a Padding Mask to block the filler space completely.

For Sentence 4, the combined mask looks like this: