Chapter 17 · AI and society
Data poisoning and model theft
Attackers can plant bad data to change what a model learns, or copy a model by querying it many times.
Chapter 17 · AI and society
Attackers can plant bad data to change what a model learns, or copy a model by querying it many times.
Seeds go into the field. A few dark ones were slipped in.
Training data is the soil a model grows from. If someone slips bad examples in, the model can grow crooked, and nobody sees it in the finished model.
A model has no way to know which training examples are honest. If a few are wrong on purpose, it learns from them too. This is data poisoning. Large models learn from huge piles of text and images collected from the open internet, which anyone can add to.
A tiny model sorts numbers into A (low) or B (high) by drawing a line halfway between the two groups. Slide in bad examples that look like B but are labelled A.
honest Ahonest Bbad example, wrongly labelled A
100% right on honest test data
With only honest data the line sits neatly between the groups. A simplified teaching toy. Real poisoning and defences are far more varied.
Why is poisoned training data hard to notice?
How can a model be copied without touching its files?
Which is a sensible defence against poisoned data?