Chapter 6 · The Transformer
The feed-forward block
After attention mixes information across tokens, a feed-forward layer processes each token on its own.
Chapter 6 · The Transformer
After attention mixes information across tokens, a feed-forward layer processes each token on its own.
Attention lets tokens share what they know. The feed-forward block then works on each token separately. Layers repeat the pair.
After attention, each word carries a blend of the words around it. “bank” has now heard about “river”. But all it did was mix numbers.
Pick a word, then press Next to move through the four stages. Switch parts off to see what they add.
Input: what “bank” carries now.
Strongest: thing · Sense of “bank”: unclear
Same network, every word alone: She → person · sat → describing · by → water · the → water · river → water · bank → water
Simplified: six hand-made detector neurons replace the thousands a real model learns, and the “mix” is half the word and half its attention blend. Dimensions: thing, animal, person, water, money, describing.
Does the feed-forward block look at other words?
What is the order inside one layer?
You switch attention off for “bank” in “river bank”. What happens?