Empirically, perhaps due to the lottery ticket hypothesis, one needs to wait a lot longer for small models to reach a useful loss, so if one is impatient it almost always pays off to train the largest model that a given compute configuration can support.