Prompt Caching Strategies

# Prompt Caching Strategies

> Implement multi-level caching for LLM interactions -- from exact prompt-response pairs to semantic similarity matches to component-level reuse -- reducing cost and latency by avoiding redundant inference.

## Problem

KV-cache optimization (the provider-side prefix cache) is powerful but limited: it only helps when the *exact same prefix* is reused across calls. In practice, many applications have a different problem:

- **Repeated queries**: Users ask the same question in slightly different words. Each variation triggers a full LLM call, even though the answer is identical or near-identical.
- **Multi-step workflows**: An agent pipeline runs embedding generation, then re-ranking, then summarization. If the pipeline reruns on similar inputs, every step recomputes from scratch.
- **Expensive intermediate results**: Embeddings, classification labels, and extracted entities are computed per-request even when the same document has been processed before.
- **Cost explosion at scale**: A system handling 10,000 requests/day where 30% are near-duplicates wastes 3,000 calls worth of compute daily.

Without application-level caching, you pay full inference cost for every call regardless of whether you have seen that input (or something very close to it) before.

## Solution

Build a multi-level cache that sits between your application and the LLM provider. Each level trades off precision for hit rate:

1. **L1 -- Exact Match**: Hash the full prompt and return a cached response if the hash matches. Zero ambiguity, fastest lookup, but only catches identical prompts.
2. **L2 -- Semantic Similarity**: Embed the prompt and search a vector store for cached prompts above a similarity threshold. Catches paraphrases and minor variations.
3. **L3 -- Component-Level**: Cache individual components (embeddings, tool outputs, extracted entities) independently. Even when the full prompt is novel, its parts may have been computed before.

Each level has its own TTL (time-to-live) and invalidation strategy. L1 entries expire quickly (minutes) because exact matches are fragile. L2 entries last longer (hours) because semantic similarity is more durable. L3 entries persist longest (days) because embeddings and entities change only when the underlying data changes.

## How It Works