6

I know that Dropout will remove neurons randomly to reduce over-fitting.

If Dropout is going to remove neurons, why are those neurons built? We could remove those neurons from the architecture.

Why don't we do that?

apaderno
  • 107
  • 6
Naren Babu R
  • 226
  • Its a complexity penalty. rather than building 100 different networks with different numbers of nodes and seeing which gives better validation results, we train one network with ( a couple of different values for drop out) – seanv507 Oct 02 '22 at 08:34
  • I recommend reading some explanations of dropout, such as in this question. https://stats.stackexchange.com/questions/241645 – Sycorax Oct 04 '22 at 03:08

3 Answers3

23

To add to @frank's answer, the reason using dropout is not the same as training a smaller network is that the neurons that are dropped out are randomly selected each time the weights are updated. So while on each iteration only some of the neurons are used and updated, over the entire training cycle all the neurons are trained. According to Jason Brownlee's A Gentle Introduction to Dropout for Regularizing Deep Neural Networks, dropout can be thought of as training an ensemble of models in parallel.

Lynn
  • 1,707
  • Never really thought of it that way, but I suppose it is analogous to bagging a bunch of weakly trained NNs. Of course in the actual implementation in code and in the back propagation of gradients it's treated as a whole. – Andrew Holmgren Oct 03 '22 at 13:52
18

The neurons are only dropped temporarily during training. They are not dropped from the network altogether. It is just that it turns out that we get better weights if we randomly set them to zero, temporarily, so the other neurons "think" they cannot "rely" on the other neurons and have to "perform well themselves". The neural network that you get out at the end contains all the neurons.

frank
  • 10,797
4

The goal of dropout isn't to disable neurons permanently. It is to prevent the network from splitting into separate sections functioning in parallel and not utilizing interconnections between them. An extreme, undesired case would be a network relying on only a single neuron processing certain feature.

Dropout disables neurons randomly during training which forces the consecutive layer to optimize weights coming from other inputs, and thus making a better use of all the possible connections available in the architecture.

user1079505
  • 161
  • This seems like an odd way to explain it, because the things you say are problems aren't obviously problematic, at least to me. From the perspective of human software design, being able to split software into separate sections that each do their own thing is highly desirable, and relying on other system components to work correctly is the norm. Even if we step away from human designs, in the world of explainable AI, finding out that a single neuron processes a specific feature would be a huge win. – Daniel Wagner Oct 03 '22 at 14:04
  • 1
    @DanielWagner Perhaps it's odd to you, but this line of reasoning was used by the authors of the method, whose abstract says "[Dropout] prevents units from co-adapting too much." https://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf?utm_content=buffer79b43&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer, – John Madden Oct 03 '22 at 16:47
  • @DanielWagner thanks, that's a good example. One of the reasons we like to split the code is to avoid running the same code, on the same data, twice. If you want to recognize handwritten letters, you don't want to run a function checking if it might be an i and start with searching for a dot, and then run another function checking if it might be a j and search for a dot again. You'd rather prefer to search for the dot once, and reuse the search outcome for both i and j. That would either save you CPU or allow to write a more advanced dot finder. This is what dropout helps to achieve. – user1079505 Oct 03 '22 at 17:27