Chapter 9 · Talking to models: prompting and context
Prompt caching
Reusing an unchanged prompt prefix lets providers skip recomputing it, saving time and cost.
Chapter 9 · Talking to models: prompting and context
Reusing an unchanged prompt prefix lets providers skip recomputing it, saving time and cost.
Request 1: the model works through every part from scratch.
The model does not have to re-read what has not changed. It picks up from the bookmark.
Many apps send the same big start every time: rules, a long document, earlier chat. Only the last question is new.
Tap a part to say you edited it before the next request. Green parts are reused, coral parts are worked out again. The new question always changes.
Teaching prices: reading saved work costs 10% of normal and the first save costs 25% extra. Real prices differ, and real caches expire after a while.
What does prompt caching reuse?
You edit a word in the first part of a long prompt. What happens?
Where should the part that changes every time go?