The context
you already computed.
First surfaced in Tandemly Briefing — 2026-06-02
RAG pipelines re-attach the same retrieved documents to every query. Agents reinject full conversation histories on each turn. Multi-user deployments send the same system prompt millions of times a day. LLM inference engines keep computing the same prefill, over and over, because nothing tracks which context was already seen.
Prefill is expensive.
And you're paying it twice.
LLM inference splits into two phases: prefill processes the input prompt and decode generates each output token. Prefill is compute-intensive and scales poorly with context length. The KV-cache exists to mitigate this, but it is fragile in ways that matter enormously in production.
The KV-cache stores the attention key-value pairs computed during prefill. When a new request shares an identical prefix with a cached request, the engine skips recomputation for that prefix. In practice, this is extremely common: system prompts, retrieved documents, and conversation histories repeat constantly across requests from different users.
The problem is ordering. KV-cache reuse requires that shared content appears at the exact same position in the token stream. If a RAG pipeline retrieves documents A, B, and C for one user and B, A, and C for the next, neither benefits from the other's cache even though they retrieved identical content. A single token insertion anywhere in the shared prefix invalidates reuse for everything that follows it.
No existing serving layer solves this at scale. Individual applications manage their own caching strategies in isolation. There is no shared index of which context blocks appear across requests, no mechanism to reorder prompts for cache alignment, and no deduplication pass that removes repeated content without degrading quality. ContextPilot fills this gap.
Between the application that constructs prompts and the inference engine that executes them, there is no component that understands cross-request context overlap. ContextPilot introduces exactly this layer: a context-aware middleware that intercepts prompts, finds shared blocks, and restructures them to maximize what the KV-cache can reuse. The inference engine needs no changes. The application needs no awareness of what other users are sending.
Four components.
One middleware layer.
ContextPilot is not a new inference engine or a new cache implementation. It is a layer that sits in front of existing engines like vLLM, SGLang, and llama.cpp, intercepting and restructuring requests before they arrive.
No model retraining. No changes to the inference engine's core attention mechanism. No modifications to existing application code beyond routing requests through the middleware. The technique is orthogonal to model architecture and works with any transformer-based LLM served by a compatible engine.
Up to 3x faster prefill.
36% fewer tokens.
Benchmarked against prior SOTA long-context serving methods, ContextPilot delivers large prefill speedups with preserved or improved reasoning quality across multiple deployment scenarios.
RAG pipelines that retrieve the same 5 to 10 documents across hundreds of concurrent queries saw the largest gains. The context index identifies the repeated document chunks immediately, the reorder pass aligns their position in every request's prefix, and the KV-cache hit rate climbs by more than 10x over unstructured serving. Prefill latency dropped to under one-third of baseline at peak load.
Agents that inject full conversation history on each turn send a growing prefix where earlier turns are identical across many requests for the same user. ContextPilot's reorder and deduplicate passes align these shared turn blocks and compress repeated content, cutting the effective token count by roughly one-third. Prefill latency improvements compound as context length grows.
Token reduction is not just a throughput gain on data-center hardware. On Apple Silicon deployments where memory bandwidth is the binding constraint, fewer tokens per request translates directly to lower peak memory pressure and more headroom for concurrent requests. ContextPilot's gains hold in this setting without engine modification.
Gains are largest when prompt overlap is high. Workloads with low cross-request context sharing will see smaller improvements: a deployment where every user sends entirely unique long-form documents will gain little from the reorder pass. Baseline your own workload before extrapolating published reduction figures. The paper reports results across enterprise document QA, multi-turn chat, and constrained-hardware scenarios; gains vary by task type and request distribution.
What this means
for building with LLMs.
The core insight is that prompt prefix overlap is latent value most deployments never capture. Before scaling more hardware, it is worth measuring how much of that value already exists in your workload.
Where to go
from here.
Applying these ideas starts with measurement. You cannot optimize context reuse you have not measured.