The AI feature everyone loved becomes the line item finance flags, because LLM cost scales with usage in a way pilots never reveal. The good news: most enterprise LLM spend is optimizable without touching quality, because most of it is waste — oversized models, redundant calls, and bloated prompts. Here are seven techniques that reliably cut cost, roughly in order of impact.
1. Route to the right-sized model
The most common and most expensive mistake is using your most capable, most expensive model for every request. Most enterprise traffic is easy — classification, extraction, simple Q&A — and a smaller, cheaper model handles it at a fraction of the cost. Reserve the frontier model for the genuinely hard queries.
Implement a routing layer that sends each request to the cheapest model that can handle it, escalating to a larger model only when a cheap-model attempt fails a quality check or a classifier predicts difficulty. In our experience intelligent routing is the single largest lever, often cutting cost by more than half with no perceptible quality drop on the traffic that matters.
2. Cache aggressively at multiple layers
Enterprise queries repeat far more than teams expect. Exact-match caching returns a stored response for identical requests instantly and for free. Semantic caching goes further, serving a cached answer for questions that are semantically equivalent even when the wording differs, using an embedding similarity threshold.
Separately, use provider prompt caching (available across major providers) to cache the large static portion of your prompts — system instructions, few-shot examples, retrieved context — so you are not repaying to process the same tokens on every call. For agentic and RAG workloads with big fixed prompt prefixes, this alone is a substantial saving.
- Exact-match cache for identical requests.
- Semantic cache for equivalent-meaning queries above a similarity threshold.
- Provider prompt caching for large static prompt prefixes.
3. Trim the prompt
You pay per token, and prompts bloat silently over time — accumulated instructions, redundant few-shot examples, verbose retrieved context. Audit your prompts. Often you can cut a third of the input tokens with no quality loss by removing redundancy and tightening instructions.
In RAG especially, do not stuff 20 chunks into the context when a reranker can identify the 5 that matter. Fewer, better chunks cut input cost and frequently improve answer quality by reducing distraction. Cost optimization and quality here point in the same direction.
4. Cap and control output tokens
Output tokens typically cost several times more than input tokens, so verbose generation is disproportionately expensive. Set explicit max-token limits and instruct models to be concise. For structured tasks, request structured output (JSON) rather than prose so you generate only the data you need.
This is low-effort, high-return. A model that answers a classification task with a single label instead of a paragraph of reasoning-you-discard costs a fraction as much, and constraining output often makes downstream parsing more reliable too.
5. Batch and pick the right processing mode
Not every workload needs a real-time answer. For offline and bulk jobs — enrichment, classification, backfills, report generation — use provider batch APIs, which commonly offer around a 50% discount in exchange for asynchronous, higher-latency processing. If a job can wait an hour, it should not pay the real-time premium.
Segregate your traffic by latency requirement and route accordingly. Much of what teams run synchronously has no genuine real-time need and is quietly overpaying for immediacy nobody uses.
6. Self-host at the right scale
At high, steady token volume, self-hosting an open-weight model can undercut per-token API pricing — but only past a real break-even point, and only if you can keep the GPUs busy. An idle GPU is more expensive than an API call. Serve with an efficient stack (vLLM's continuous batching) and drive utilization high before this pencils out.
For most organizations, managed APIs are cheaper all-in once you account for the engineering and operational overhead of self-hosting. Do the math on your actual volume rather than assuming self-hosting is automatically cheaper; it is a genuine inflection, but a later one than enthusiasm suggests.
7. Make cost observable and attributable
You cannot optimize what you cannot see. Instrument token usage and cost per request, per feature, and per user, and put it on a dashboard next to your quality and latency metrics. Set budgets and alerts so a runaway agent or a viral feature triggers a notification, not a surprise invoice.
Attribution changes behavior. When teams can see what their feature costs, they optimize it. Without visibility, cost is an abstract aggregate nobody owns; with it, cost becomes an engineering metric that improves like any other. This is the meta-technique that makes the other six sustainable.
- Track token cost per request, feature, and user, not just aggregate spend.
- Set per-feature budgets with alerts to catch runaway usage early.
- Display cost alongside quality and latency so optimization has a target.
Key takeaways
- 1.Route each request to the cheapest capable model — usually the single biggest saving.
- 2.Cache at multiple layers: exact-match, semantic, and provider prompt caching for static prefixes.
- 3.Trim input prompts and cap output tokens; fewer, better RAG chunks cut cost and improve quality.
- 4.Use batch APIs for non-real-time work and only self-host past a real, utilization-justified break-even.
- 5.Make cost observable and attributable per feature so optimization becomes a normal engineering metric.