Chapter 2 · Machine learning basics
Collecting and cleaning data
A model is only as good as its data: gaps, duplicates, errors and skew all leak into what it learns.
Chapter 2 · Machine learning basics
A model is only as good as its data: gaps, duplicates, errors and skew all leak into what it learns.
A messy pile of records…
Cleaning is usually the biggest part of a real AI project. Most of the work happens before any model sees the data.
Data comes from forms, sensors, people and old files. Real data has duplicates, typos, gaps and odd values, like 80 hours of study typed instead of 8.
A small model learns who passes from hours studied and hours slept. It is tested on 40 clean students it has never seen. Turn fixes on and off.
Tap a fix to see what it changes. Some faults matter more than others.
| # | Hours studied | Hours slept | Result | Flag |
|---|---|---|---|---|
| 1 | 2.4 | 8.6 | "Fail" | |
| 2 | 6.1 | 8.6 | "pass" | typo |
| 3 | — | 8.3 | "Fail" | gap |
| 4 | 3.6 | 4.3 | "Fial" | typo |
| 5 | 0.7 | — | "Fail" | gap |
| 6 | 97 | 6.7 | "Pass" | odd |
| 7 | 4.5 | 6.7 | "FAIL " | typo |
| 8 | 6.1 | 7.9 | "Pass" | |
| 9 | — | 6.1 | "Pass" | gap |
| 10 | 4.7 | 8.3 | "Pass" | |
| 11 | 3.6 | 6.4 | "Fail" | |
| 12 | 0.9 | 7.7 | "fail" | typo |
| 13 | 83 | 5.8 | "Pass" | odd |
| 14 | 1.8 | — | "Fial" | typo |
| 15 | — | 6.3 | "Fail" | gap |
| 16 | 4.6 | 4.5 | "Fail" | |
| 17 | 1.1 | 7.1 | "FAIL " | typo |
| 18 | 7.7 | 4.4 | "Pass" | |
| 19 | 6.1 | 8.6 | "Pass" | |
| 20 | 27 | 7 | "Fail" | odd |
| 21 | — | 5.6 | "Fail" | gap |
| 22 | 5.1 | 7.6 | "pass" | typo |
| 23 | 3.1 | — | "Fail" | gap |
| 24 | 8.7 | 7.1 | "Pas" | typo |
| 25 | 3.8 | 9 | "Pass" | |
| 26 | 4.8 | 4.9 | "Fail" | |
| 27 | — | 8.6 | "FAIL " | typo |
| 28 | 5.9 | 6.5 | "Pass" | |
| 29 | 1.7 | 6.2 | "Fail" | |
| 30 | 1.8 | 4 | "Fail" | |
| 31 | 3.6 | 4.3 | "Fial" | copy |
| 32 | — | 6.1 | "Pass" | copy |
| 33 | 83 | 5.8 | "Pass" | copy |
| 34 | — | 5.6 | "Fail" | copy |
| 35 | 4.8 | 4.9 | "Fail" | copy |
| 36 | 5.9 | 6.5 | "Pass" | copy |
This is a simplified teaching version: one tiny model, made-up students, four kinds of fault.
A model learns from a table that has many copies of the same row. What is the risk?
A label column holds 'Pass', 'pass ' and 'Pas'. What should you do?
The data is perfectly tidy but 95% of it shows one kind of example. What is the problem?