LEVEL 05 · LESSON 44
Cost and latency
Optimize the cost and waiting time of a successful task, not an isolated model call.
What it is
Cost is the resource or monetary expense of a task. Latency is the time a user waits for a useful result. Both depend on the whole workflow, not only one model call.
Why it matters
A technically impressive assistant can be impractical if it is too slow or if correction and review costs outweigh its benefit.
A business example
A proposal assistant makes several searches and model calls before showing anything. The team removes redundant searches and runs independent lookups together, measuring whether quality remains acceptable.
How it actually works
Measure complete-task latency, time to first useful output, token usage, tool costs, infrastructure, and human effort. Long outputs, repeated context, and sequential agent steps can all add delay. Caching, routing, and parallel work can help when correctness and freshness are preserved. Evaluate typical and slow-tail cases, because a good average can hide frustrating waits. Compare cost per successful task rather than per request.
A common misconception
A cheaper token price does not automatically mean a cheaper business outcome. Retries and manual cleanup can dominate the total.
ONE SENTENCE TO REMEMBER
Optimize the cost and waiting time of a successful task, not an isolated model call.
CHECK YOUR UNDERSTANDING
Two assistants are compared. A has cheaper tokens; B costs more per request but rarely needs retries or manual cleanup. How should they be compared?
Show the answer
Answer: B.
Optimize the cost and waiting time of a successful task, not an isolated model call. A cheaper token price does not guarantee a cheaper outcome when retries and manual cleanup dominate, and averages can hide frustrating slow cases.
Take it into your next meeting
Where does most of the user’s waiting time actually go?
One more concept, understood.
Mark this lesson complete to keep your place.