Chapter 17 · AI and society
Interpretability
Interpretability research looks inside models to find the features and circuits behind their behaviour.
Preparing interpretability: looking inside
Chapter 17 · AI and society
Interpretability research looks inside models to find the features and circuits behind their behaviour.
Showing a diagonal
Each neuron in this toy network reacts to one pattern. Real interpretability research tries to find such features inside much bigger models.
We can see what goes in and what comes out. The millions of numbers in between do not explain themselves.
Tap pixels to draw. Watch which neuron lights up, then pick a label for each one. A tiny hand-made teaching network, not a real model.
The network says: Not sure
What does interpretability try to do?
You switch a neuron off and the answer changes. What does that suggest?
Are real models as tidy as this toy?