Next-Token-Prediction

The heart of LLM pretraining is a surprisingly simple task: Given all previous tokens, predict the next one. This "self-supervised" task is why models can be trained on trillions of tokens with minimal labeling effort.

Mathematically, this is formulated as an autoregressive language model:

The model learns through backpropagation to adjust the probability distribution P so that it predicts the ground-truth token with the highest possible probability.

Step 1: Process Context
The Transformer processes all previous tokens: "The", "cat", "sat", "on", "the". Through Self-Attention, all tokens can attend to all other tokens and capture dependencies.

Step 2: Final Hidden State
After all Transformer blocks, the final hidden state emerges – a vector that compresses all context information.

Step 3: Projection to Vocabulary
A final linear layer projects this vector to the vocabulary size (typically: 50K - 128K tokens). This produces "Logits" for each possible token.

Step 4: Softmax Normalization
The softmax function converts logits to probabilities (0-1, sum = 1):

The token with the highest probability is typically selected – in the example "mat".

During training, the model always uses the ground-truth tokens as context, not its own predictions. This is called Teacher Forcing and significantly simplifies training.

The Exposure Bias Problem: During inference, however, the model must work with its own predictions – potentially including erroneous tokens. This discrepancy between training and inference can lead to error accumulation.

Some methods to mitigate this:

• Scheduled Sampling: During training, gradually replace real tokens with model predictions
• Autoregressive Fine-Tuning: Additional fine-tuning rounds after pretraining
• Decoding Strategies: Beam Search, Top-k, Top-p to find better sequences