Chapter 4 · The deep learning era (2006–2016)2014
Sequence to sequence
An encoder reads a whole sentence into a vector; a decoder writes the output sentence from it.
Chapter 4 · The deep learning era (2006–2016)2014
An encoder reads a whole sentence into a vector; a decoder writes the output sentence from it.
The encoder reads one word at a time and adds it to a single vector.
This was the idea of sequence to sequence models around 2014: an encoder, a decoder, and one vector in between.
A sentence is a list of words in order. Translation turns one list into another list, and the two lists can have different lengths.
Lengthen the sentence, then make the vector bigger. Green words came back out right; coral ones got lost.
Only 50% survived. One small vector cannot hold this much.
Chart: words recovered (up) against sentence length in words (across), for the vector size you chose. A simplified teaching version: the vectors come from a fixed random recipe, not from learning.
What does the encoder hand to the decoder?
Why do long sentences go wrong in this design?
In the lab, what helps more words survive?