Chapter 6 · The Transformer2018
BERT: reading both ways
BERT learned by filling in masked words using context from both sides.
Chapter 6 · The Transformer2018
BERT learned by filling in masked words using context from both sides.
A word is hidden. The model has to guess it.
BERT hides some words while it learns and has to rebuild them. The clues come from the words on the left and on the right.
Take a sentence and replace some words with [MASK]. The model has to work out the hidden words. The correct answer is known, so it can learn from every mistake.
Tap words to hide them (up to three). Compare a guesser that looks only left with one that looks both ways.
Hidden word: “bank”. Violet underline = left clues, coral underline = right clues.
Chance given to the true word: 17%
Chance given to the true word: 100%
A simplified teaching version: the guesser counts which words sit near which in about forty short sentences. Real BERT learns this with a deep transformer on huge amounts of text. Try “the bank gave her a loan”: the left side alone cannot tell, the right side can.
How is BERT trained?
Why is the right-hand context useful?
BERT is best suited to…