No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon
A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases.
They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments.
(This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)
Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters.
There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)
I wonder if it would be worth starting with low precision and then over time raise precision to zero in on a local min / max.
1. newer optimizations algorithms converge faster (Adam, adagrad)
2. lower-precision numerics can be used (TF32 is 19bit, bf16 is popular)
3. matrix multiplications are implemented in hardware (tensor cores, AMX)
4. neural networks are compiled directly (TVM, halide)
5. multi-device network topologies are tuned for training (e.g. nvlink improvements)
6. companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)