Too large batch size

Question

I experiment with CIFA10 datasets. With my model I found that the larger the batch size, the better the model can learn the dataset. From what I see on the internet the typical size is 32 to 128, and my optimal size is 512-1024. Is it ok? Or are there any things which I should take a look at to improve the model. Which indicators should I use to debug it?

P.S. It seems that the gradient is too noisy and if we have a larger sample size, it reduces the noise.

FYI Tradeoff batch size vs. number of iterations to train a neural network — Franck Dernoncourt, May 01 '17 at 00:27
@FranckDernoncourt Thanks for the link but it seems that there's a bug in my model somewhere — Konstantin Solomatov, May 01 '17 at 00:45
This link gives a good overview of the whole point of minibatching. It'll be slower and generally less efficient, but can help with huge datasets and help jolt you out of poor local minima: http://sebastianruder.com/optimizing-gradient-descent/index.html — generic_user, Jun 02 '17 at 15:09

PickleRick · Answer 1 · 2018-01-12T20:23:08.893

8

Read the following paper. It's a great read. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, Nitish Shirish Keska et al, ICLR 2017.

There are many great discussions and empirical results on benchmark datasets comparing the effect of different batchsizes. As they conclude, large batchsize causes over-fitting and they explain it as it converges to a sharp minima.

The code is also available here.

edited Jan 12 '18 at 20:23

answered Oct 24 '17 at 12:26

PickleRick

758
6
20

Link only answers are not welcome on CV. Furthermore you do not have to write in capital letters. – Ferdi Oct 24 '17 at 12:48

score 2 · Answer 2 · answered Oct 11 '20 at 09:09

2

The too-large batch size can introduce numerical instability and the Layer-wise Adaptive Learning Rates would help stabilize the training.

answered Oct 11 '20 at 09:09

Lerner Zhang

6,636
1
41
75

Too large batch size

2 Answers2