0

I have a very basic question on the optimization algotithm, when I'm adjusting weights and biases in a NN, should I:

  1. Forward propagate and backpropagate to calculate gradient descent (DC) for each batch once and then repeat for iterations_number times.

or

  1. Forward propagate and backpropagate to calculate gradient descent (DC) one batch for iterations_number times and then continue with the next batch.
Oliver Mohr Bonometti
  • 103
  • 2

1 Answers1

0

One iteration means that you do one forward pass and one backward pass for a single batch containing batch_size examples. Then you move on to the next batch. Also see this post on SE SO.

Note that this is not identical to an epoch. An epoch is only completed when all examples of your dataset have been passed through your network. And as long as your batch size is less than the number of datapoints you will need $\frac{n}{\text{batch size}}$ (with $n$ being the sample size) iterations to complete an epoch. Also see this answer on SE DS.

Jonathan
  • 5,410
  • 1
  • 9
  • 21
  • I see, so answer is option 1. and the correct terminology should be epochs_number instead of iterations_number as the second is just $\frac{n}{batch size}$. – Oliver Mohr Bonometti Jan 16 '21 at 18:13
  • @OliverMohrBonometti Yes, then it would be correct. The number of iterations is then calculated as following: iterations_number = (n / batchsize) * epochs_number. – Jonathan Jan 17 '21 at 08:49