Semantic caching for large language models stores and reuses previous responses by matching the semantic meaning of queries rather than exact text matches. The system stores hidden-layer representations (embeddings) from the neural network along with the generated outputs in a cache database. When a new query arrives, the system compares its semantic representation to cached queries, and if similar enough, returns the cached response instead of processing through the full model. This can speed up responses by 10× and reduce GPU compute costs by 90% for repeated or similar queries.
The technology addresses the high computational cost of running large language models, which require expensive GPU resources for every inference. By identifying semantically similar queries, the cache can serve appropriate responses even when the exact wording differs. This is particularly valuable for applications with many similar queries, such as customer support, documentation systems, or educational platforms. The semantic matching enables the cache to work effectively even when users phrase questions differently. Available as a web API, the technology can be integrated into existing LLM applications to dramatically improve response times and reduce infrastructure costs while maintaining response quality.
Maintainers of Milvus, an open-source vector database for scalable similarity search.
An LLM gateway and observability platform that includes a built-in semantic cache to optimize API costs and latency.
Open-source observability platform for LLMs that provides caching features to reduce latency and costs.
In-memory data store provider that has integrated vector search and semantic caching capabilities directly into its platform.
A web infrastructure and security company that has already enabled PQC support for a significant portion of the internet.
Develops the leading open-source framework for orchestrating LLMs and retrieval systems.
Provides a managed vector database optimized for high-performance AI similarity search.
Open-source vector search engine with out-of-the-box modules for vectorization and RAG.
Serverless cache provider that has expanded into vector index support for AI workflows.
Paper
From Exact Hits to Close Enough: Semantic Caching for LLM EmbeddingsarXiv · Feb 7, 2026
Explores offline policies for semantic caching to reuse semantically similar requests via embeddings. Proposes polynomial-time heuristics and online policies combining recency, frequency, and locality to improve semantic accuracy in LLM responses.
Paper
ContextCache: Context-Aware Semantic Cache for Multi-Turn Queries in Large Language ModelsPVLDB / arXiv · Jun 1, 2025
Introduces ContextCache for multi-turn dialogues, using a two-stage retrieval architecture to integrate current and historical dialogue representations. Shows 10x lower latency than direct LLM invocation.
Paper
LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer InferenceIEEE / arXiv · Dec 1, 2025
Presents LLMCache, a layer-wise caching framework that accelerates transformer inference by reusing intermediate activations based on semantic similarity. Achieves up to 3.1x speedup with minimal accuracy degradation.
Paper
vCache: Verified Semantic Prompt CachingarXiv · May 2, 2025
Proposes vCache, a verified semantic cache with user-defined error rate guarantees. Uses online learning to estimate optimal similarity thresholds for cached prompts, outperforming static threshold baselines.
Paper
SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV CachingarXiv · Mar 26, 2025
Proposes SentenceKV to group tokens based on sentence-level semantic similarity, compressing representations into semantic vectors stored on GPU while offloading KV pairs to CPU.
Paper
Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via Semantic-Aware Knowledge CachingarXiv · Sep 22, 2025
Describes Cortex, a cross-region knowledge caching architecture using Semantic Elements (SE) and Semantic Retrieval Index (Seri) for fast candidate selection and validation, reducing latency and cost.
Paper
Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online AdaptationarXiv · Aug 1, 2025
Develops a principled framework for semantic cache eviction under unknown query and cost distributions, addressing the mismatch costs inherent in semantic caching.
