I Tested 4 Transformer Feed-Forward Architectures on a 2M-Parameter GPT - Here's What Wins
Introduction
When LLaMA, Mistral, and DeepSeek all use SwiGLU in their feed-forward networks, it's easy to assume you should too. But does it actually matter at the tiny scales most of us experiment with? Or is it just another billion-parameter trick?
I wanted to know. So I ran a controlled experiment: I trained a 2M-parameter GPT from scratch 12 times, changing nothing but the FFN architecture and activation function. Same data, same seed, same optimizer. The result is a clean, reproducible benchmark that answers a very practical question: What should I use in my own small transformer?
Here's the whole setup, the numbers, and a step-by-step guide so you can replicate the experiment yourself.
1. The Training Pipeline (Built from Scratch)
Before we get to the architectures, a quick look at the pipeline. Everything is implemented in raw PyTorch - no HuggingFace Trainer, no PyTorch Lightning. This gives full control over every detail, which is critical for a fair comparison.
Pipeline steps:
Data Ingestion: Extract text from 6 PDF books (finance, risk management, economics) using PyMuPDF. Clean, deduplicate, and combine into a single corpus.
Tokenizer: Train a BPE tokenizer from scratch with vocabulary size 4096.
Model: A decoder-only Transformer (MiniGPT) with configurable FFN - more on this below.
Training: Hand-written loop with gradient clipping, validation loss, and perplexity tracking.
Reproducibility: Fixed random seed (42), deterministic cuDNN, verified initial loss, and a successful overfit test on a single batch.
The entire pipeline is open-source and can be run with a single command (see Section 7).
2. Experimental Setup
| Hyperparameter | Value |
|---|---|
| Vocabulary size | 4096 |
| d_model | 128 |
| Attention heads | 4 |
| Layers | 4 |
| Max sequence length | 64 |
| Dropout | 0.0 (disabled) |
| FFN hidden dim | 512 (d_model × 4) |
| Total params (std.) | 1.85M |
| Total params (gated) | 2.11M |
| Data | 750K tokens (6 books) |
| Train/Val split | 90% / 10% |
| Optimizer | AdamW (lr=1e-3) |
| Batch size | 128 |
| Epochs | 5 |
| Seed | 42 |
3. The Four FFN Architectures
Standard FFN (original Transformer, BERT, GPT-2)
FFN(x) = Linear2( Activation( Linear1(x) ) )
Two linear layers, activation in between. Simple, fast, well-understood.
Gated FFNs (SwiGLU, ReGLU, GiGLU)
Gate(x) = Activation( Linear_gate(x) )
Up(x) = Linear_up(x)
FFN(x) = Linear_down( Gate(x) * Up(x) )
Three linear projections. The "gate" controls what information flows through; the "up" path provides the content. This is what LLaMA and Mistral use.
The three variants differ only in the activation applied to the gate:
- SwiGLU: SiLU (swish)
- ReGLU: ReLU
- GiGLU: GELU
For each architecture, I tested all three activations - yes, even "ReGLU with GELU" - because we want to see if the activation itself matters independently of the gate idea.
4. Results: Training Loss After 5 Epochs
| FFN Type | Activation | Train Loss (↓) | Params |
|---|---|---|---|
| Standard | ReLU | ~0.82* | 1.85M |
| Standard | GELU | 0.78 | 1.85M |
| Standard | SiLU | 0.84 | 1.85M |
| SwiGLU | ReLU | 0.76 | 2.11M |
| SwiGLU | SiLU | 0.74 | 2.11M |
| SwiGLU | GELU | 0.75 | 2.11M |
| ReGLU | SiLU | 0.74 | 2.11M |
| ReGLU | GELU | 0.75 | 2.11M |
| ReGLU | ReLU | 0.76 | 2.11M |
| GiGLU | ReLU | 0.92 | 2.11M |
| GiGLU | GELU | 0.92 | 2.11M |
| GiGLU | SiLU | 0.92 | 2.11M |
Key takeaways from the table:
- SwiGLU+SiLU wins with a loss of 0.74, matching the LLaMA recipe.
- Standard+GELU is a very close second at 0.78 - only 5% worse.
- SiLU is the best activation for gated FFNs, consistently beating ReLU and GELU.
- GELU is the best for standard FFNs (0.78 vs 0.84 for SiLU, 0.82 for ReLU).
- GiGLU fails at this scale - its performance lags significantly behind other architectures.
5. Why Gating Helps (and Why Not Always)
In a gated FFN, the model learns which information to keep. Think of it like this:
- Standard FFN: "Process everything the same way."
- Gated FFN: "Decide what's important, then process that."
The gate can suppress noise, amplify relevant features, or even shut off irrelevant dimensions. That's powerful, but it comes with a cost: a third weight matrix, slightly slower training, and the need for enough model capacity to actually learn useful gate behaviors.
At 2M parameters, the gain is real but modest (0.04 loss). At 7B parameters, LLaMA and others show this gap widens considerably. So for tiny models, you can happily stick with Standard+GELU and keep things simple.
6. So, Which FFN Should You Use?
| Your Situation | My Recommendation |
|---|---|
| Model < 1M parameters | Standard + GELU |
| Model 1M-10M parameters | SwiGLU + SiLU or Standard + GELU (tie) |
| Model > 10M parameters | SwiGLU + SiLU |
| Memory or speed critical | Standard + GELU |
| Reproducing LLaMA/Mistral | SwiGLU + SiLU |
7. Reproduce This Experiment Yourself
Everything is in my GitHub repo: raw-pytorch-minigpt.
Clone and set up
git clone https://github.com/hussainnazary2/raw-pytorch-minigpt.git
cd raw-pytorch-minigpt
bash setup.sh
Run a specific experiment (e.g., SwiGLU+SiLU)
git checkout exp-ffn-swiglu-silu
python src/trainer.py
Each experiment has its own git tag. The config.yaml contains the exact hyperparameters. All runs use seed 42 - you'll get the same numbers.
If you want to run all experiments and build the table yourself, check the experiments/ folder for scripts.
8. What's Next
I'll keep expanding this benchmark with:
- Validation loss metrics for all configurations
- Larger model sizes (2M, 5M parameters)
- Different data domains
The goal is to build a practical, open-source reference for anyone training small transformers from scratch.
Have you tried different FFN architectures in your own models? I'd love to hear what worked for you - drop a comment or reach out.
This post is part of the "Building GPT from Scratch" series. Follow along for more experiments on what actually matters in transformer training.
Need help with your transformer experiments? Contact us to discuss architecture optimization and training strategies.