Chapter 6 · The Transformer
Queries, keys and values
Each token asks a question (query), advertises what it has (key), and shares content (value).
Chapter 6 · The Transformer
Each token asks a question (query), advertises what it has (key), and shares content (value).
“bank” now carries mostly: water
QueryKeyMatch%Value
The query is compared with every key. The best matches hand back their values. Those values blend into the word's new meaning.
Every word is turned into three lists of numbers: a query, a key and a value. Think of a library.
Pick a word, then turn the dials of its query. Watch the matches and the blend change.
Simplified: six hand-made numbers per word stand in for the long learned vectors of a real model. Coloured dots are those numbers: brighter means bigger.
Which part is compared with the keys of the other words?
What gets blended together to make a word's new meaning?
You turn up the “water” dial of the query for “bank”. What happens?