Re:Facebook Trains ImageNet in 1 Hour
by breatheoften on Tuesday June 13, 2017 @04:43AM
(#14543579)
This explanation of why larger batches might make optimization harder makes a fair amount of sense to me. This comment on Quora helps a bit as well - with a slightly different approach to characterizing the trade off of batch size/accuracy/runtime performance.
I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain.
Are there any optimizers that dynamically scale the batch size up/down based on an online metric?
https://www.quora.com/Intuitively-how-does-mini-batch-size-a...
I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain.
Are there any optimizers that dynamically scale the batch size up/down based on an online metric?