Prefix Cache
Provider-side storage of shared request prefixes, billing matched tokens at a reduced rate.
Prefix cache is a provider-side optimization that allows consecutive model requests with identical starting sequences to skip re-processing those shared tokens. When a request's prefix — typically the system prompt and conversation history up to a certain point — matches a recent request, the provider reuses its prior computation and bills those tokens as cache tokens at a significantly reduced rate.
The mechanism operates at the token level: any change to the prefix, even reordering files in a prompt, rewriting a system instruction mid-session, or inserting a timestamp near the top, invalidates the cache from that point onward. Subsequent tokens then process at full rate. Understanding what constitutes the prefix is critical to maximizing cache hit rates.
The tradeoff is between optimization and flexibility. Structuring prompts to maximize prefix reuse — keeping system prompts stable, batching similar requests together — can reduce inference costs substantially. But doing so constrains how dynamically the agent's context can change within a session, and forces careful engineering of what gets injected where.
Open questions include how to expose cache hit rates and cost savings to users in a actionable way, and whether providers will eventually offer stronger guarantees about cache persistence across longer time windows. Current cache invalidation behavior is provider-specific and not always predictable.