News for hackers. Stuff that matters.
Re:Quadrillion-parameter transformers and how to train them by TGower on Sunday November 08, 2026 @10:13AM (#50413981)
> Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget.

The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.

Re:Quadrillion-parameter transformers and how to train them by in-silico on Sunday November 08, 2026 @01:05PM (#50415551)
> The dataset is not held fixed in practice

The dataset is held fixed here because it's a control variable, and control variables are how you isolate relationships in science experiments.