Chapter 4 · The deep learning era (2006–2016)1997
LSTM: long short-term memory
LSTMs add gates that decide what to remember and forget, so memory lasts longer.
Chapter 4 · The deep learning era (2006–2016)1997
LSTMs add gates that decide what to remember and forget, so memory lasts longer.
Memory belt starts empty
The belt carries the memory along the sentence. Gates beside it choose what gets written, what stays, and what is read out.
A plain RNN squeezes its memory at every step, so a name from the start of a long sentence fades by the end. Predicting “she” needs the name “Meera” to survive.
Tap a gate to change it: open, half, closed. Goal: make the last output equal Meera's value (0.9) while ignoring the distractor Ravi. Try the presets first.
Final read-out at “market”: 0.3. Aim for 0.9. Meera has faded or been mixed with Ravi.
A teaching version: each word writes one number (Meera +0.9, Ravi −0.6) and the gates are closed, half or open. Real LSTMs have many numbers and learn smooth gates. Rule: belt = forget × old belt + input × new item.
What does the forget gate do?
You want to ignore a distracting word. Which gate do you shut?
Why can an LSTM remember longer than a plain RNN?