Chapter 8 · Shaping models: tuning and alignment
Mixture of experts
A router sends each token to a few expert sub-networks, so a huge model only uses part of itself per token.
Chapter 8 · Shaping models: tuning and alignment
A router sends each token to a few expert sub-networks, so a huge model only uses part of itself per token.
A token arrives at the router
A router chooses a few experts for each token. The others stay idle for that token.
In a transformer block, the feed-forward part holds many of the weights. Mixture of experts replaces it with several smaller copies, called experts.
Type a sentence. Tap a token to see its router scores. Change the number of experts and top-k, then watch the active share of parameters.
More experts raise the total, while a small top-k keeps the active share low. Simplified: the router is a fixed made-up rule, not a trained layer. Spot any expert that does far more work than the rest? Real training adds a balancing term to avoid that.
What does the router decide?
A model has 8 experts and uses the top 2 per token. What is true?
Why is load balancing needed?