Chapter 5 · Turning language into numbers
One-hot vs dense vectors
One-hot vectors are huge and say nothing about meaning; dense vectors are small and capture similarity.
Chapter 5 · Turning language into numbers
One-hot vectors are huge and say nothing about meaning; dense vectors are small and capture similarity.
“cat” as one-hot: 40 numbers, a single 1.
One-hot spends a whole row on one word. A dense vector packs the word into a few numbers, and nearby numbers mean nearby meanings.
Give every word in the vocabulary its own slot. To write cat, put a 1 in the cat slot and 0 everywhere else. With 50,000 words, that is 50,000 numbers for every single word.
Pick two words. One-hot can only say same or different. The dense vector says how close they are. Tap a pair to try it.
The dense vectors here are the 3-number hand-made ones from the Embeddings lesson. “Alike” is measured by distance on that map. Real models learn hundreds or thousands of numbers.
How many numbers does a one-hot vector need for a 50,000-word vocabulary?
How alike does one-hot say 'cat' and 'kitten' are?
Why do models prefer dense vectors?