Chapter 8 · Shaping models: tuning and alignment2021
LoRA and efficient fine-tuning
LoRA trains a few small add-on matrices instead of all the weights, making tuning cheap.
Chapter 8 · Shaping models: tuning and alignment2021
LoRA trains a few small add-on matrices instead of all the weights, making tuning cheap.
Big weights stay frozen
Instead of changing every weight, LoRA learns a small correction made from two thin matrices.
Full tuning updates every weight and needs memory for all of them. With LoRA the original weights are frozen. They never change.
Pick a model size, then slide the rank and choose which matrices get add-ons. Compare full tuning with LoRA.
32 layers, width 4096, vocabulary 50,000 (invented sizes).
Try rank 1, then 128. A higher rank can learn a richer correction but trains more numbers. Simplified: it uses the earlier lesson’s parameter count and ignores biases.
What happens to the original weights in LoRA?
A frozen 1,000 × 1,000 weight gets rank-8 add-ons. How many numbers are trained for it?
What does a higher rank give you?