Cost, latency and reliability for agents
Agents make many calls, so small costs and delays multiply. Learn budgets, caching, parallel tool calls, timeouts, fallbacks and when to hand over to a person.
In 60 seconds
Cost, latency and reliability for agents
Agents make many calls, so small costs and delays multiply. Learn budgets, caching, parallel tool calls, timeouts, fallbacks and when to hand over to a person.
Where the cost comes from
- Input tokens grow at every step, because the whole context is usually sent again.
- Big tool results, like full web pages or files, fill the context quickly.
- Retries and loops repeat work.
- Output tokens, including long reasoning, often cost more per token than input. Check your provider's current prices.
Controls that work
| Control | What it does | How |
|---|---|---|
| Token and money budget | Stops a task that uses too much. | Count tokens per run from API responses; stop and report when the budget is reached. |
| Trim tool results | Keeps context small. | Return only needed fields; cut long text; store the rest and give a reference. |
| Prompt caching | Can reduce cost and delay when many calls share the same start. | Many providers offer it. Keep stable parts (instructions, tool definitions) at the start. Check how your provider does it. |
| Result caching | Avoids repeating the same tool call. | Cache read-only tool results for a short time, keyed by tool name and arguments. |
| Parallel tool calls | Saves waiting time. | Some APIs let the model ask for several independent tool calls at once; run them together. |
| Right-sized models | Saves cost on easy steps. | Use a smaller model for simple steps like routing or extraction, if tests show it is good enough. |
Timeouts, retries and fallbacks
import random, time
class BudgetExceeded(Exception):
pass
class Budget:
def __init__(self, max_tokens: int):
self.max_tokens, self.used = max_tokens, 0
def spend(self, tokens: int):
self.used += tokens
if self.used > self.max_tokens:
raise BudgetExceeded("used " + str(self.used) + " tokens")
def call_model(prompt: str, timeout: float) -> tuple:
# Placeholder: call your provider with this timeout.
# Returns (text, tokens_used). Here it sometimes fails, for the demo.
if random.random() < 0.3:
raise TimeoutError("model did not answer in time")
return "ok: " + prompt[:30], 500
def reliable_call(prompt: str, budget: Budget, tries: int = 3) -> str:
for attempt in range(tries):
try:
text, tokens = call_model(prompt, timeout=30)
budget.spend(tokens)
return text
except TimeoutError:
wait = (2 ** attempt) + random.random() # backoff with jitter
time.sleep(wait * 0.01) # shortened for the demo
return "FALLBACK: could not reach the model; handing over to a person."
budget = Budget(max_tokens=5000)
try:
for step in range(20):
print(step, reliable_call("step " + str(step), budget))
except BudgetExceeded as err:
print("Stopped:", err)Fallbacks and human handoff
- 1
Retry
For short, temporary failures, with backoff and a small limit. - 2
Fall back
Try a simpler path: another model, a cached answer, or a fixed workflow for the common case. - 3
Degrade honestly
Give a partial answer and say clearly what is missing. - 4
Hand over to a person
Pass the case with a short summary and the trace id, so they don't start from zero.
Latency: making it feel fast
- Run independent tool calls in parallel.
- Stream the final answer so the user sees text as it is written.
- Show progress for long tasks ("Checking 3 suppliers… 2 done").
- Cut steps: better tool design often removes whole round trips.
- Set a total time limit per task, not only per call.
max_tokens to 2000 and watch where it stops.Watch and read more
Lab
A cost and reliability plan for your agent, backed by numbers from your eval runs.
The problem
Use your agent and eval set from the lesson 10 lab (fake tools are fine; count "tokens" as characters divided by 4 if you have no real API).
- Record tokens and time per run. Report the average and the worst case per task.
- Add a per-task token budget and a total time limit. Show one scenario where each stops the run.
- Add a retry with backoff for a flaky fake tool, and a fallback message that hands over to a person with the trace id.
- Pick two changes from the lesson (trimming results, caching, parallel calls, a smaller model for one step). Predict their effect, try them, and compare with your prediction.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Adding prompt caching cut your average cost, but the worst-case cost did not change. What might explain this?Reveal
Caching helps when many calls share the same start. The worst-case runs are probably long loops or runs with huge tool results, where most tokens are new content, not the shared prefix. To fix the worst case, you need budgets, loop detection and trimming of big results, not only caching.
Q2Why can a fallback to a smaller, faster model make the system less reliable overall if you are not careful?Reveal
The fallback path runs rarely, so it gets less testing. When the main model is down, all traffic suddenly goes through a path that may fail more often or behave differently, for example with tool calls or output formats. Include the fallback path in your eval set and test it regularly, or fall back to a simpler honest message and a human instead.
Questions people ask
How do I estimate what an agent will cost?
Run your eval scenarios, record tokens in and out for every call, and compute the average and the worst case per task. Multiply by your provider's current prices and expected volume. The worst case matters: a few runaway runs can cost more than many normal ones.
Is the cheapest model always the cheapest choice?
Not always. A weaker model may need more steps, more retries or more human handoffs, so the cost per successful task can be higher. Measure cost per successful task, not cost per call.
What timeout should I use?
Look at your traces: how long do normal calls take? Set the timeout a bit above the slow end of normal, and set a separate total limit for the whole task. Very long reasoning calls may need longer timeouts than short ones.
Can caching give wrong answers?
Yes, if the data changed. Cache only read-only results, keep the cache time short for data that changes (prices, stock, train status), and never share cached private data between users.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.