Re:Quadrillion-parameter transformers and how to train them
by in-silico on Sunday November 08, 2026 @01:05PM
(#50415551)
> The dataset is not held fixed in practice
The dataset is held fixed here because it's a control variable, and control variables are how you isolate relationships in science experiments.
The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.