Chapter 4 · The deep learning era (2006–2016)
Recurrent neural networks
RNNs read sequences one step at a time, carrying a hidden memory forward.
Chapter 4 · The deep learning era (2006–2016)
RNNs read sequences one step at a time, carrying a hidden memory forward.
The network reads one word, mixes it with what it remembers, and passes the new memory on to the next word. The same step is repeated for every word: that repeating loop is why it is called recurrent.
“Dog bites man” and “man bites dog” use the same words. A network that sees all words as a bag cannot tell them apart. A sequence needs something that remembers what came before.
Type a sentence or pick an example, then step through it. Try “not bad”, then lower “memory kept per step” and watch early words fade.
Read “good”: a positive word, added to the faded memory.
After the whole sentence: negative feeling (mood -1).
A teaching version: the update rule is written by hand with two memory numbers and a short word list. A real RNN learns its rule and holds hundreds of numbers.
What does an RNN pass from one word to the next?
Why can an RNN tell “dog bites man” from “man bites dog”?
What is a weakness of simple RNNs?