Chapter 6 · The Transformer2018
GPT: generative pretraining
GPT models are decoder-only transformers trained to predict the next token, scaled up generation after generation.
Chapter 6 · The Transformer2018
GPT models are decoder-only transformers trained to predict the next token, scaled up generation after generation.
Same recipe each time: a decoder that predicts the next token. Only the scale changes.
Parameter counts as reported in each model’s paper. The largest GPT-2 version is shown.
GPT stands for generative pretrained transformer. It is a decoder-only transformer trained to predict the next token on a huge amount of ordinary text. No labels are needed: the next word is the answer.
Use the arrows to move each card up or down so the earliest is at the top. Then press Check. Cards from the same year can go in either order.
Far larger again. It could pick up a task from a few examples written in the prompt.
Researchers showed that loss falls smoothly and predictably as models, data and compute grow.
Models were further trained on human ratings so they follow instructions and answer helpfully.
An encoder pretrained by filling in masked words, reading both ways.
An encoder-decoder that treats every task as text in, text out.
A bigger decoder that wrote long, fluent text and could try some tasks with no fine-tuning at all.
A decoder-only transformer pretrained to predict the next token, then fine-tuned for particular tasks.
A new design built from attention alone, first made for translation with an encoder and a decoder.
A short, neutral history, not a complete one. Many other models and papers matter too. Years and sizes are as reported in each model’s paper.
What is a GPT-style model trained to do?
What mainly changed from GPT to GPT-2 to GPT-3?
What did scaling laws show?