Pre-Training Checkpoint

A pre-training checkpoint is a snapshot of a model's parameters saved at the end of — or periodically during — the pretraining phase of training, before any post-training steps such as instruction tuning, RLHF, or tool-use alignment. The checkpoint captures the raw language-modeling behavior learned from the pretraining corpus and serves as the base from which all downstream variants (chat, code, reasoning, distilled sizes) are produced. Frontier-model training pipelines typically produce a small number of major-version checkpoints (e.g., GPT-4 → GPT-5 base) plus many intermediate ones used for capability research; minor-version public releases are usually derived from a mixture of the most recent checkpoint and the strongest post-training techniques available at release time. Pre-training checkpoints are massive (often hundreds of gigabytes), expensive to produce (months of compute on thousands of accelerators), and treated as proprietary infrastructure — though some labs have released base checkpoints under research licenses. The term "base model" and "pre-training checkpoint" are sometimes used interchangeably, though "base model" more often refers to the public-facing release while "pre-training checkpoint" emphasizes the artifact itself.