Chapter 15 · Running AI in the real world
Latency vs throughput
Latency is how fast one answer arrives; throughput is how many answers per second a system serves.
Preparing latency vs throughput: sports car or bus?
Chapter 15 · Running AI in the real world
Latency is how fast one answer arrives; throughput is how many answers per second a system serves.
The car gets one person there fastest. The bus is slower for each person but moves far more people in the same time.
Latency asks: how long do I wait for my answer? Throughput asks: how many answers does the system produce per second for everyone?
One server, 100-token replies. The timings come from a made-up formula with illustrative numbers.
The biggest batch that meets your budget is 13, giving the whole server about 220 tokens/s. A stricter budget means smaller batches and lower throughput.
What does latency measure?
What usually happens when you make the batch bigger?
Which job suits a large batch best?