Next-Token Prediction

Next-Token Prediction is a fundamental training objective for language models that learns to predict the most likely subsequent token in a sequence given all preceding tokens as context. This autoregressive approach enables models to develop comprehensive understanding of language patterns, syntax, semantics, and world knowledge through self-supervised learning on large text corpora. The technique operates by masking future tokens during training, forcing models to predict each token based solely on leftward context, creating a natural learning signal without requiring labeled data. Next-token prediction serves as the foundation for most modern large language models, enabling capabilities like text generation, conversation, and reasoning through iterative token sampling. Standard training uses teacher forcing: the model is conditioned on the ground-truth prefix rather than on its own earlier predictions, which produces exposure bias—a mismatch between training conditions and free-running generation—reduced afterwards by post-training stages that optimize the model on its own sampled outputs, such as reinforcement learning from human feedback. This training paradigm allows models to learn complex linguistic structures, factual knowledge, and reasoning patterns that emerge from statistical regularities in training data.

Related terms

• NeMo Parakeet
• NLU meaning

Related services: LLM development, Agentic AI consulting.

← Back to the AI Glossary