Chapter 4 · The deep learning era (2006–2016)2014
Attention is born
Attention lets the decoder look back at the most relevant input words for each output word.
Chapter 4 · The deep learning era (2006–2016)2014
Attention lets the decoder look back at the most relevant input words for each output word.
To write “le”, the decoder looks mostly at “the” (85%).
Brighter, thicker lines mean more attention. Notice that “noir” looks at “black” even though the word order changed.
In the last lesson, the decoder had only one squeezed vector. By the end of a long sentence, early words were blurry.
Tap an output word to see where its attention goes. Then blur the focus, or turn attention off.
To write “noir”, the decoder looks mostly at “black” (90%).
The scores are written by hand for these two sentences, so this is a simplified teaching version. A real model learns them.
What does the decoder have access to with attention?
What do attention weights always add up to?
Why did “noir” look at “black” though it comes later in the sentence?