Tokens are a design signal
Token consumption is visible and easy to count, so it often becomes a proxy for AI cost. But a low token count can hide expensive retries, poor answers, manual correction, or the engineering burden of an overcomplicated pipeline. Efficiency has to be measured at the workflow level.
A useful metric connects total inference and operating cost to an accepted outcome: a resolved request, a reviewed document, a completed decision, or a safe robotic action. That denominator keeps optimization aligned with the business.
Remove work before optimizing prompts
The most effective token is often the one a model never needed to process.
- Use deterministic rules for exact validation, formatting, calculations, and policy checks.
- Retrieve only evidence relevant to the current task.
- Carry structured state instead of replaying an entire conversation.
- Summarize stable history once and retain links to the source.
- Avoid sending interface chrome, duplicated instructions, and unused metadata.
- Stop generation when a structured answer is complete.
These are architecture decisions. Prompt wording matters, but it cannot compensate for an indiscriminate context pipeline.
Route by difficulty
One model does not need to handle every step. A routing layer can send bounded classification, extraction, or transformation to a smaller model and escalate uncertain cases to a more capable one. Some tasks should bypass generation entirely.
Routing only works with evaluation. Define representative cases, confidence behavior, and the cost of a wrong route. A cheap model that creates expensive human rework is not an optimization.
Treat retrieval as an information budget
Retrieval-augmented generation is often described as “giving the model company knowledge.” In practice, retrieval selects a small evidence set under a context budget. Quality depends on document structure, metadata, permissions, ranking, and the ability to abstain when evidence is weak.
Measure whether retrieved passages support the answer. More chunks can reduce quality by obscuring the relevant evidence and increasing both latency and token use.
Cache what is actually stable
Provider prompt caching, application-level response caching, retrieved-document caching, and precomputed embeddings solve different problems. Choose a cache based on what remains stable and how quickly permissions or source information can change.
Never let a cache bypass current authorization. The key must represent the user, policy, source version, model configuration, and other factors that can alter the valid result.
Evaluate quality and economics together
Maintain a small operating scorecard:
- acceptance rate on representative cases;
- human correction or escalation rate;
- input and output tokens per accepted outcome;
- latency at normal and peak demand;
- model and infrastructure cost per accepted outcome;
- failure modes introduced by each optimization.
Review the scorecard when prompts, retrieval, tools, models, or source data change. Token efficiency is not a one-time compression exercise. It is disciplined system operation.