Chapter 15 · Running AI in the real world
The cost of serving AI
Costs come from model size, tokens processed and hardware time; caching and smaller models cut them.
Chapter 15 · Running AI in the real world
Costs come from model size, tokens processed and hardware time; caching and smaller models cut them.
Input tokens: a coin each
Serving a model means running powerful hardware for every token it reads and writes. The bill grows with the number of tokens.
Running a model uses hardware time. Services turn that into a price per token, with input and output priced separately. A larger model needs more hardware per token, so its tokens cost more.
The starting prices are made-up examples in no particular currency (per million tokens). Replace them with the prices from your own plan, then try the two levers.
Move a lever to see the savings.
What mainly drives the cost of serving an AI app?
How does caching cut the cost?
Why send easy requests to a smaller model?