Timestep

A timestep is a fundamental unit of temporal progression in iterative AI systems, representing a single discrete interval during which a model reads input, performs computation, and updates its internal state. In reinforcement learning, each timestep captures a complete interaction cycle: an agent observes the current environment state, selects an action, receives a reward signal, and transitions to a new state. This structure allows learning algorithms to accumulate experience over sequences of timesteps and use that experience to refine decision-making policies through methods such as temporal-difference learning or policy gradients.

In recurrent neural networks and sequence models, timesteps define the granularity at which sequential data is processed. A language model, for instance, may treat each token as a timestep, updating its hidden state at every step to carry contextual information forward. In time-series forecasting, the choice of timestep length — whether seconds, hours, or days — directly determines what temporal patterns the model can detect and how far into the future it can reliably predict. Finer timesteps capture rapid dynamics but increase computational cost; coarser timesteps reduce complexity but may obscure important short-term variation.

The timestep also plays a central role in controlling the stability and convergence of learning algorithms. In temporal-difference methods, value estimates are propagated backward through timesteps via bootstrapping, and the discount factor applied at each step shapes how much the agent values immediate versus future rewards. In simulation environments, the timestep interval must be chosen carefully to balance physical fidelity with computational efficiency — a concern shared by both AI researchers and the broader scientific computing community.

Beyond reinforcement learning and sequence modeling, timesteps appear in diffusion models, where the generative process is indexed by a discrete noise schedule across hundreds or thousands of steps, and in neural ordinary differential equations, where continuous dynamics are approximated through discrete temporal increments. The concept is deceptively simple but sits at the core of nearly every AI system that reasons about change, causality, or sequential structure.