Is there any proven disadvantage of transfer learning for CNNs?

Question

Suppose I know that I want to use a ResNet-101 architecture for my specific problem. There are ReseNet-101 models trained on ImageNet.

Is there any disadvantage of using those pre-trained models and just resetting the last (few) layers to match the specific number of classes or should that be the default option?

Please don't simply post your gut feeling... I have a gut feeling as well, but I want to know.

Answer 1

Based on my experience, not just for ImageNet, if you have enough data it's better to train your network from scratch. There are numerous reasons that I can explain why.

• First of all, I don't know whether you have had this experience or not but I've trained complicated CNNs neworks with over 25 million parameters. After reaching 95% accuracy, after convergence, I changed the learning rate to a bit bigger number to find another probable local minimum. I've not found any answer till today but whenever I did so, my accuracy decreased signifincantly all the time and it never imporoved even after more than thosands of epochs.

• The other problem is that whenver you use transfer learning, your training data should have two options. First of all, the distribution of the training data which your pre-trianed model has used should be like the data that you are going to face during test time or at least don't vary too much. Second, the number of training data for transfer learning should be in a way that you don't overfit the model. If you have a few number of labeled training data, and your typical pre-trained model, like ResNet has millions of parameters, you have to be aware of overfitting by choosing appropriate metrics for evaluation and good testing data which represent the real distribution of the real population.

• Next problem is that you can not remove layers with confidence to reduce the number of parameters. Basically the number of layers, as you yourself know, is a hyper-parameter that there is no consensus on how to be chosen. If you remove the convolutional layers from the first layers, again based on experience, you won't have good learning because of the nature of the architecture which finds low level features. Furthermore, if you remove first layers you will have problem for your dense layers, because the number of trainable parameters changes in this way. Densely connected layers and deep convolutional layers can be good points for reduction but it may take time to find how many layers and neurons to be diminished inorder not to overfit.

If you don't have enough data and there is already a pre-trained model, you can do something that can help you. I divide my answer to two parts for this situation:

The pre-trained model does not have common labels for most of its classes

If this is the case, you can forget all the fully connected layers and replace new ones. Basically what they do is just classifying the features that the network has found to reduce the error rate. About the convolutional layers you have to consider two major points:

1. Pooling layers try to sum up the information in a local neighbourhood and make a higher representation of the inputs. Suppose that the inputs of a pooling layer have nose, eyes, eybrows and so on. Your pooling layers somehow attempt to check whether they exist in a neighbourhood or not. Consequently, convolutional layers after pooling layers usually keep information which may be irrelevant to your task. There is a downside for this interpretation. The information may be distributed among different activation maps as Jason Yosinski et al. have investigated this nature at Understanding neural networks through deep visualization where you can read One of the most interesting conclusions so far has been that representations on some layers seem to be surprisingly local. Instead of finding distributed representations on all layers, we see, for example, detectors for text, flowers, fruit, and faces on conv4 and conv5. These conclusions can be drawn either from the live visualization or the optimized images (or, best, by using both in concert) and suggest several directions for future research and These visualizations suggest that further study into the exact nature of learned representations—whether they are local to a single channel or distributed across several .... A partial solution can be keeping the first layers which find low level features that are usually shared among different data distributions and removing deeper convolutional layers which find higher abstractions.

2. As stated, the main problem of convolutional layers is that the information they find may be distributed among different activation maps. Consequently, you can not be sure by removing a layer you can have better performance or not.

The pre-trained model has some common classes with your labels

If this is the case you can visualize the activations using the techniques described here. They have shown that although human face is not a label in ImageNet, some of internal activation maps get activated by face. This observation can be seen for other labels too. For instance, netwroks for deciding a scene contains a car or not are usually sensetive to roads and trees. The image below shows that which parts of ouputs get activated for which parts of images. This can help you when you don't have enough data and you have to use transfer learning.

Based on the answer here The standard classification setting is an input distribution $p(X)$ and a label distribution $p(Y|X)$. Domain adaptation: when $p(X)$ changes between training and test. Transfer learning: when $p(Y|X)$ changes between training and test. In other words, in DA the input distribution changes but the labels remain the same; in TL, the input distributions stay the same, but the labels change. Consequently, domain adaption problems also can be considered for the mentioned solutions.

Answer 2

I would not do that if your data is very different from the data in Imagenet. This is not typically the case, since Imagenet has lots of images representing many different things. However, let's say that your data comes from pictures taken by telescopes. In that case, even the most basic features (the very first layers) of a model trained on Imagenet will not be useful to your model, and in this case it might be more convenient to train the model from scratch using random initialization. One thing you can do, and that it might work better, is use the Imagenet model as your initial parameters and update all the parameters of the network.

However, in most cases your images look like a subset of Imagenet, being transfer learning a very powerful technique.

Answer 3

I found this Q. a year or so after it was asked, and I still don't think there are any studies that have comprehensively tested this.

When most people discuss transfer learning, what they think of is:

"Will imagenet weights help me get better accuracy (or at least speed up training) when using [insert popular google/facbook AI/microsoft architecture] on my data."

In reality, if I have medical images of arm bones, and I've previously trained a model on leg bones, that would probably be more applicable transfer learning than using imagenet weights... so stating that "de novo is better" or "transfer learning is better" is kind of meaningless, because you may just not have access to more relevant weights for transfer learning than imagenet.

The early layers always contain 'simple' features, and these are probably always somewhat transferable to any other visual problem; providing the scale of the objects of interest in the images are not drastically different from the original data. But to answer the question directly: no, there is no “proven” disadvantage to transferring weights from a previously trained network.