News for hackers. Stuff that matters.
Re:Facebook Trains ImageNet in 1 Hour by breatheoften on Tuesday June 13, 2017 @12:50AM (#14542910)
I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?

It seems to my naive view like it should be "nice" from an accuracy perspective to look at more samples before making an adjustment to the network weights ...?

In general, does changing the batch_size hyperparameter make a lot of difference on different problems ...? Does the right value for batch size tend to be problem specific and/or network architecture specific?

Re:Facebook Trains ImageNet in 1 Hour by IanCal on Tuesday June 13, 2017 @05:35AM (#14543719)
> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?

Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together.

Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next step would keep taking you up that hill.

Now, if you batched together two direction tests, what would happen? You'd average together your left and right and end up with moving nowhere. Having both at the same time doesn't give you better information about how you move if you only see the result after averaging.

This interestingly maps to something we see in humans, though I'm struggling to find a decent paper on it (from the PRISM lab in Birmingham, UK if anyone else has any luck, think the person doing the research might have been called Chris). Simple adaptation tasks, in this case learning to control a joystick that has a clockwise/anticlockwise force applied to it, don't work well if you try and learn both one thing and the opposite straight away. However, sleeping in-between learning each left you able to do both well. Perhaps this was early results though.

Batch tradeoffs:

https://stats.stackexchange.com/questions/164876/tradeoff-ba...

https://arxiv.org/abs/1609.04836

Re:Facebook Trains ImageNet in 1 Hour by jononor on Tuesday June 13, 2017 @05:53AM (#14543770)
If this is the case, could one get improved learning by mixing large and small batches?
Re:Facebook Trains ImageNet in 1 Hour by nullc on Tuesday June 13, 2017 @01:02AM (#14542941)
[Warning: far from an expert here]

No, batches also help you escape local minima.

Re:Facebook Trains ImageNet in 1 Hour by sxyuan on Tuesday June 13, 2017 @02:56AM (#14543269)
[Another non-expert commenting]

Your comment didn't make sense to me at first, but I think I get it now. Even if you were able to fit the entire dataset into memory, batches are still a good idea, because optimizing on the entire dataset is non-convex and will likely lead you into a local minimum. However, what is a local minimum for one batch may not be a local minimum for the next batch, which helps you escape.

This explains why optimization gets harder with very large batch sizes - the gradients for different batches become more similar (as they resemble the "global" gradient more closely), so you become more susceptible to local minima. I think this also explains why the learning rate scaling helps - it increases the variance across gradients, and helps you escape local minima.

Re:Facebook Trains ImageNet in 1 Hour by breatheoften on Tuesday June 13, 2017 @04:43AM (#14543579)
This explanation of why larger batches might make optimization harder makes a fair amount of sense to me. This comment on Quora helps a bit as well - with a slightly different approach to characterizing the trade off of batch size/accuracy/runtime performance.

https://www.quora.com/Intuitively-how-does-mini-batch-size-a...

I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain.

Are there any optimizers that dynamically scale the batch size up/down based on an online metric?

Re:Facebook Trains ImageNet in 1 Hour by oerpli on Tuesday June 13, 2017 @03:30AM (#14543365)
One of the better articles I found about this topic when I've been learning for my exam: http://sebastianruder.com/optimizing-gradient-descent/index....