Chapter 5 · Turning language into numbers
Embeddings
An embedding turns a token or text into a list of numbers so that similar meanings land close together.
Chapter 5 · Turning language into numbers
An embedding turns a token or text into a list of numbers so that similar meanings land close together.
Nobody tells the map which words belong together. Words used in similar ways end up close, so animals gather, foods gather and vehicles gather.
A model cannot do anything with the letters c-a-t. It needs numbers. The simplest idea is to give every word an ID, but an ID like 4,217 says nothing about meaning.
This map has 40 words placed by hand. Turn it to see the third direction, tap a word to see its five nearest neighbours, or drop a mystery point and see who lives nearby.
Nearest to cat:
cat = [-0.72, 0.52, 0.20]
This is a simplified teaching map: 3 numbers per word, placed by hand. Real embeddings are learned from huge amounts of text and use far more numbers.
What does an embedding give a word?
On the map, which word would sit closest to 'kitten'?
Real embeddings usually use…