Chapter 8 · Shaping models: tuning and alignment2021
Instruction tuning
Training on instruction–answer pairs teaches a model to follow requests.
Chapter 8 · Shaping models: tuning and alignment2021
Training on instruction–answer pairs teaches a model to follow requests.
Translate “cat” into French.
…Translate “dog” into French. Translate “bird” into French. Translate
Translate “cat” into French.
“Cat” in French is “chat”.
A base model only continues text. After training on instruction and answer cards, the same model replies to the request.
A pretrained model learned to predict the next token. Ask it a question and it may carry on with more questions, because that is what text often looks like.
Tick the pairs you want in the deck. Then see how the tuned model handles four new requests. Some pairs are flawed. Can you spot them?
“Thank you” is “dhanyavaad”.
Answers tidily…Summarise in one sentence: The train was late, so we waited. Summarise
Rambles on like a base modelA confident answer that is wrong or off the point.
Learned a mistakeA confident answer that is wrong or off the point.
Learned a mistakeThe checker looks at length, repeats and empty answers. It cannot judge truth, so you still must.
A simplified toy: a few hand-written pairs and a rule instead of a real network.
What does a base model usually do with a question?
What goes into an instruction tuning dataset?
You add pairs with wrong answers to the deck. What happens?