How are deep-learning NNs different now (2016) from the ones I studied just 4 years ago (2012)?

Question

It is said in Wikipedia and deeplearning4j that Deep-learning NN (DLNN) are NN that have >1 hidden layer.

These kind of NN were standard at university for me, while DLNN are very hyped right now. Been there, done that - what's the big deal?

I heard also that stacked NN are considered deep-learning. How is deep-learning really defined?

My background of NN is mostly from university, not from jobs:

• studied applications of NN in industry
• had about 5 courses on artif. intel. & mach. learn. - though maybe 2 of them on NN
• used NN for small, simple project on image recognition - used 3 layer feed-forward NN
• did not do real research (as in doctor thesis) on them

Answer 1

You are right in that the basic concept of a deep NN hasn't changed since 2012. But there have been a variety of improvements to the ways in which deep NNs are trained that have made them qualitatively more powerful. There are also a wider variety of architectures available today. I've listed some developments since 2012, grouped by training improvements and architecture improvements:

Improvements to training deep NNs

• Hardware: The most obvious change is just the inexorable progression of Moore's law. There is more computing power available today. Cloud computing also makes it easy for people to train large NNs without needing to buy a huge rig.

• Software: The open source software for deep learning is really enormously improved from 2012. Back in 2012 there was Theano, maybe Caffe as well. I'm sure there are some others, too. But today we also have TensorFlow, Torch, Paddle, and CNTK, all of which are supported by large tech companies. This is closely related to the hardware bullet point since many of these platforms make it easy to train on GPUs, which drastically speeds up training time.

• Activation functions: The use of ReLU activation functions is probably more widespread these days, which makes training very deep networks easier. On the research side, there is a wider variety of activation functions being studied, including leaky ReLU, parametric ReLU, and maxout units.

• Optimization algorithms: There are more optimization algorithms around today. Adagrad and Adadelta just been introduced in 2011 and 2012, respectively. But we now also have the Adam optimizer and it's become a very popular choice.

• Dropout: In the past few years, dropout has become a standard tool for regularization when training neural networks. Dropout is a computationally inexpensive form of ensembling for NNs. In general, a set of models trained on random samples of the dataset will outperform a single model trained on the entire dataset. This is difficult to do explicitly for NNs because they are so expensive to train. But a similar effect can be approximated just by randomly "turning off" neurons on each step. Different subgraphs in the NN end up getting trained on different data sets, and thereby learn different things. Like ensembling, this tends to make the overall NN more robust to overfitting. Dropout is a simple technique that seems to improve performance in almost every case, so it's now used de rigueur.

• Batch normalization: It's been known for a while that NNs train best on data that is normalized --- i.e., there is zero mean and unit variance. In a very deep network, as the data passes through each layer, the inputs will be transformed and will generally drift to a distribution that lacks this nice, normalized property. This makes learning in these deeper layers more difficult because, from its perspective, its inputs do not have zero mean and unit variance. The mean could be very large and the variance could be very small. Batch normalization addresses this by transforming the inputs to a layer to have zero mean and unit variance. This seems to be enormously effective in training very deep NNs.

• Theory: Up until very recently, it was thought that the reason deep NNs are hard to train is that the optimization algorithms get stuck in local minima and have trouble getting out and finding global minima. In the last four years there have been a number of studies that seem to indicate that this intuition was wrong (e.g., Goodfellow et al. 2014). In the very high dimensional parameter space of a deep NN, local minima tend not to be that much worse than global minima. The problem is actually that when training, the NN can find itself on a long, wide plateau. Furthermore, these plateaus can end abruptly in a steep cliff. If the NN takes small steps, it takes a very long time to learn. But if the steps are too large, it meets a huge gradient when it runs into the cliff, which undoes all the earlier work. (This can be avoided with gradient clipping, another post-2012 innovation.)

New architectures

• Residual networks: Researchers have been able to train incredibly deep networks (more than 1000 layers!) using residual networks. The idea here is that each layer receives not only the output from the previous layer, but also the original input as well. If trained properly, this encourages each layer to learn something different from the previous layers, so that each additional layer adds information.

• Wide and deep networks: Wide, shallow networks have a tendency to simply memorize the mapping between their inputs and their outputs. Deep networks generalize much better. Usually you want good generalization, but there are some situations, like recommendation systems, in which simple memorization without generalization is important, too. In these cases you want to provide good, substantive solutions when a user makes a general query, but very precise solutions when the user makes a very specific query. Wide and deep networks are able to fulfill this task nicely.

• Neural turing machine: A shortcoming of traditional recurrent NNs (whether they be the standard RNN or something more sophisticated like an LSTM) is that their memory is somewhat "intuitive". They manage to remember past inputs by saving the hidden layer activations they produce into the future. However, sometimes it makes more sense to explicitly store some data. (This might be the difference between writing a phone number down on a piece of paper vs. remembering that the number had around 7 digits and there were a couple of 3s in there and maybe a dash somewhere in the middle.) The neural Turing machine is a way to try to address this issue. The idea is that the network can learn to explicitly commit certain facts to a memory bank. This is not straightforward to do because backprop algorithms require differentiable functions, but committing a datum to a memory address is an inherently discrete operation. Consequently, neural Turing machines get around this by committing a little bit of data to a distribution of different memory addresses. These architectures don't seem to work super well yet, but the idea is very important. Some variant of these will probably become widespread in the future.

• Generative adversarial networks: GANs are a very exciting idea that seems to be seeing a lot of practical use already. The idea here is to train two NNs simultaneously: one that tries to generate samples from the underlying probability distribution (a generator), and one that tries to distinguish between real data points and the fake data points generated by the generator (a discriminator). So, for example, if your dataset is a collection of pictures of bedrooms, the generator will try to make its own pictures of bedrooms, and the discriminator will try to figure out if it's looking at real pictures of bedrooms or fake pictures of bedrooms. In the end, you have two very useful NNs: one that is really good at classifying images as being bedrooms or not bedrooms, and one that is really good at generating realistic images of bedrooms.

Answer 2

Deep learning (and in particular deep convoluted neural networks) is very popular right now because of recent, substantial improvements in various machine vision / object detection challenges. Deep convents in particular have been used to reduce ImageNet classification errors from 28% in 2010 to 7% in 2014. Deep learning is certainly overhyped but that shouldn't detract from the fact that there have been meaningful improvements in the field of machine vision.

Answer 3

A similar question was asked on CV: difference between neural network and deep learning:

Deep learning = deep artificial neural networks + other kind of deep models.

Deep artificial neural networks = artificial neural networks with more than 1 layer. (see minimum number of layers in a deep neural network)

FYI: How true is this slide on deep learning claiming that all improvements from 1980s are only due to much more data and much faster computers?