Chapter 6 · The Transformer
The KV cache
When generating, the model stores past keys and values so it does not recompute them for every new token.
Chapter 6 · The Transformer
When generating, the model stores past keys and values so it does not recompute them for every new token.
No cache: redo all 1
With cache: only new 1
worked on this step reused from the shelf
Each new word needs the keys and values of every word before it. A cache keeps them on a shelf, so they are made only once.
A model writes a reply token by token. For each new token, attention looks back at every earlier token using their keys and values.
Slide how many tokens are written. A simple counter shows the work each way. These are teaching numbers.
Cache on: the work grows in a straight line, but the shelf grows too. Cells = 2 (key and value) × tokens × layers.
What does the KV cache save?
With a cache, what is computed for each new token?
What is the cost of the cache?