Chapter 17 · AI and society
Bias and fairness
Models learn patterns from data, including unfair ones; fairness needs measuring and correcting.
Chapter 17 · AI and society
Models learn patterns from data, including unfair ones; fairness needs measuring and correcting.
Two groups, a fair-looking rule.
A model can look neutral and still favour one group because of the data it learned from. Measuring outcomes by group is how we notice and correct it.
A model learns patterns from past data. If that data reflects unequal access or past unfair decisions, the model can repeat them, even when it never sees a group label.
Sixty invented applicants. Circles are Group A, squares are Group B. Green outline means the person could complete the course. Filled means chosen by your cut-offs. In this toy data, Group B had less test practice, so its scores run lower for the same ability.
Overall, the right call was made for 87% of applicants (chosen if qualified, not chosen if not). With these cut-offs, qualified people in one group are chosen noticeably less often. Try both balancing buttons: they give different cut-offs, and accuracy can move too. Applicants with ability of 50 or more count as qualified.
How can a model become unfair without anyone intending it?
Why can one cut-off score treat two groups differently?
What is the first step towards fixing unfairness?