FFN Feed-Forward Network

FFN stands for Feed-Forward Network (sometimes called an MLP, or Multi-Layer Perceptron). It is a core component found inside every layer of a Transformer architecture, sitting right after the self-attention mechanism. While attention figures out relationships between words, the FFN processes each word/token individually to transform and store knowledge. [1, 2, 3, 4, 5]
How an FFN Works
An FFN consists of linear transformations separated by a non-linear activation function (such as GELU or ReLU). It typically operates in three steps: [1, 2, 3]
  • Expansion: The input vector is projected into a much higher-dimensional space (usually 4 times larger than the model’s base hidden size) to analyze complex hidden patterns. [1, 2]
  • Activation: A non-linear function filters the data, allowing the network to learn complex, non-linear relationships rather than just straight lines. [1, 2]
  • Contraction: A second linear layer shrinks the expanded dimensions back down to the model’s original embedding size. [1, 2, 3]
Why FFNs Matter
  • The Knowledge Store: FFN parameters act as the model’s “long-term memory,” holding factual knowledge, grammar rules, and world information learned during training. [1, 2]
  • Parameter Weight: FFN layers make up roughly two-thirds (66%) of a standard LLM’s total parameter count. [1, 2]
  • The “Thinker”: If self-attention acts as a detective gathering clues from surrounding context, the FFN acts as a lookup library that decodes and processes what those clues mean. [1, 2]

 

 

=-=-=-=-=-

 

 

 

 

Loading