Chapter 15 · Running AI in the real world
Time to first token and tokens per second
Users feel two speeds: how long until the reply starts, and how fast it streams.
Chapter 15 · Running AI in the real world
Users feel two speeds: how long until the reply starts, and how fast it streams.
Two different speeds. The amber part is the wait before the first word. The green part is the reply streaming in.
Before the first word appears, the model reads your whole prompt. A longer prompt means a longer wait. This is the time to first token.
Set the wait and streaming speed, then tap Send. The reply is stored text played back at your speeds; no model runs. Words per token and reading speed are rough English averages.
A short pause before it starts: fine. Far faster than anyone can read; the extra speed is hardly noticed in chat.
What does time to first token measure?
A very long prompt will most likely make which speed worse?
Roughly how fast does streaming need to be to keep ahead of a reader?