CHAPTER 8.2 · TOOLS

Parameter Calculator

Calculate the total parameter count of an LLM based on architecture details like layers, heads, dimensions, and MoE configuration.

The Parameter Calculator shows how architecture decisions determine model size: Layers, heads, dimensions, and MoE configuration add up to billions of parameters. Understand the relationships through experimentation.

📖 Learning Context ▼
🎯

Learning Objectives

  • Understand the parameter formula for Transformers
  • Learn which architecture decisions matter most
  • Be able to verify known models
🗺️

Context: Where are we?

Step 3/5 Chapter 8: Tools & Glossary

Calculators (3/5) make theory practical – calculate yourself what architecture decisions mean.

💡

Why It Matters

Parameters ≠ Quality, but Parameters = Cost. Understanding the formula helps evaluate model announcements and estimate hardware requirements.

📌

Key Takeaways

  • Main Factor: Hidden Dimension x Layers dominates
  • FFN: Makes up ~2/3 of parameters
  • Embeddings: Vocab x dmodel is fixed per model

Architecture Parameters

Number of Transformer blocks in the stack
Hidden state dimension
Multi-Head Attention heads
MHA: same as Q. GQA: fewer. MQA: 1
Feedforward intermediate dimension
Number of different tokens
FFN is replicated to N experts

Results

Total Parameters
0 Billion
Memory Required (FP16)
0 GB
Inference FLOPs per Token
0 TFLOPs

Parameter Distribution

Embedding Matrix 0 M
Attention (Q, K, V, O) 0 M
Feedforward Network 0 M
Layer Normalization 0 M
Output Projection 0 M
Formulas
Attention: 4 x dmodel² (Q, K, V, O)
FFN (ReLU): 2 x dmodel x dff (W1, W2)
FFN (SwiGLU): 3 x dmodel x dff (W, V, W2)
LayerNorm: 2 x dmodel per layer (Gain, Bias)
Embedding: vocab_size x dmodel
Output: dmodel x vocab_size (if no Weight Tying)
GQA Adjustment
Grouped Query Attention reduces the K/V projection. Instead of h x (dk+dv) only nkv_heads x (dk+dv). For Llama 2 70B: 64 Q-heads, 8 KV-heads.
MoE Efficiency
With Mixture of Experts, FFN parameters are replicated. But only top-k experts are active per token. Mixtral 8x7B: 47B total, 13B active.
Memory Considerations
FP16: 2 bytes per parameter. FP32: 4 bytes. INT8: 1 byte. INT4: 0.5 bytes. Additionally: Gradients (training), KV-Cache (inference), activations.
FLOPs Estimate
Inference: ~2 x parameters per token generated. Training: ~6 x parameters x tokens (forward + backward). For MoE: Only active parameters count.