Re:Train TensorFlow models faster and at lower cost on Cloud...
by gdahl on Wednesday December 12, 2018 @09:28PM
(#18669441)
To clarify, we found the range shrink if one trains for a fixed number of epochs and expand if one trains for a fixed number of steps. So tuning can be easier or harder depending on the budget.
https://arxiv.org/abs/1811.03600
We looked at a bunch of different model architectures and datasets and found that you can get speedups from larger batch sizes up until a certain point, but that point is different for different datasets and architectures. The range of good hyperparameters is also narrower for larger batch sizes, which makes tuning harder.