News for hackers. Stuff that matters.
Re:Optimizing ML training with metagradient descent by yorwba on Wednesday March 26, 2025 @06:58AM (#43480827)
Run a grid of hyperparameters for small models of different sizes to find out how the optimal values change as you scale up (the "scaling laws"), then extrapolate to predict performance at even larger scales, then do a single large run and hope that your predictions aren't too far off.

There are no comments on this comment yet.