{"id":6802,"date":"2018-02-05T10:06:41","date_gmt":"2018-02-05T10:06:41","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=6802"},"modified":"2025-07-30T21:03:02","modified_gmt":"2025-07-30T15:33:02","slug":"transfer-learning","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/transfer-learning\/","title":{"rendered":"Transfer Learning for Deep Learning with CNN"},"content":{"rendered":"
\n

In this blog, we will study Transfer Learning. As this Transfer Learning concept relates with deep learning and CNN also. Although, will use graphs and images to understand Transfer Learning concept.<\/span><\/p>\n

\n

Introduction to Transfer Learning<\/h3>\n<\/div>\n
We can say transfer learning is a machine learning<\/a> method. In this, a model developed for a task that was reused<\/span> as the starting point for a model on a second task.<\/div>\n<\/div>\n
\n
\"Introduction<\/a>

Introduction to Transfer Learning<\/p><\/div>\n<\/div>\n

\n
<\/div>\n
\n
Transfer learning is the most popular approach in deep learning<\/a>. In this, we use pre-trained models as the starting point on computer vision. Also, natural language processing<\/a> tasks given the vast compute and time resource. Although, we need to develop neural network models.<\/div>\n<\/div>\n<\/div>\n
<\/div>\n
As transfer learning is related<\/span> to many problems. Such as multi-task learning and concept drift. Although it is not exclusively<\/span> an area of study for deep learning.<\/div>\n
<\/div>\n
\n
\"Introduction<\/a>

Introduction to Transfer Learning<\/p><\/div>\n<\/div>\n

<\/div>\n
\n
\"Introduction<\/a>

Introduction to Transfer Learning<\/p><\/div>\n<\/div>\n

\n
\n

What is a Pre-Trained Model?<\/h3>\n<\/div>\n
\n
To solve a problem, we need to have a pre-trained model of similar problem. Instead of building a model from scratch to solve a similar problem, we use the model trained on other problem as a starting point.<\/div>\n<\/div>\n<\/div>\n

How can I use Pre-trained Models?<\/h3>\n

As there is a predefined aim to use a pre-trained model. Also, a concept of transfer learning plays an important role in a pre-trained model.<\/p>\n

While choosing a pre-trained model, one should be careful in their case. If the problem statement we have at hand is very different from the one on which the pre-trained model was trained \u2013 the prediction we would get would be very inaccurate.<\/p>\n

As already many pre-trained architectures are directly available for use in the Keras library. Imagenet<\/strong>\u00a0data set has been widely used to build various architectures since it is large enough (1.2M images) to create a generalized model. Although, the problem statement comes in training a model.<\/p>\n

That can correctly classify the images into 1,000 separate object categories. Moreover, these 1,000 image categories represent object classes that we come across in our day-to-day lives. Such as species of dogs, cats, various household objects, vehicle types etc.<\/p>\n

We use transfer learning to generalize into images outside the ImageNet dataset. This happens only in case of a pre-trained model. Also, we use fine-tuning model for the modifications in a pre-trained model. Since we assume that the pre-trained network has been trained quite well.<\/p>\n

Thus, we don’t want to modify the weights too soon and too much. While modifying we generally use a learning rate smaller than the one used for initially training the model.<\/p>\n

Ways to Fine tune the model<\/h3>\n
\"Ways<\/a>

Ways to Fine tune the model<\/p><\/div>\n

a. Feature extraction<\/strong> \u2013 For a feature extraction mechanism, we use a pre-trained model as in this we can remove the output layer. Further, we have to use the entire network as a fixed feature extractor for the new data set.<\/p>\n

b. Use the Architecture of the pre-trained model \u2013 <\/strong>According to a dataset, at the time of initializing and training model, we use its architecture.\u00a0<\/strong>
\n<\/b><\/p>\n

c. Train some layers while freeze others<\/b> \u2013 There is one more way to use a pre-trained model i.e to train model partially. Further, we have to keep the weights of initial layers of the model frozen. While have to retrain only higher layers.\u00a0 We can try and test as to how many layers to be frozen and how many to be trained.<\/p>\n

The below diagram should help you decide on how to proceed with using the pre-trained model in your case \u2013<\/p>\n

Scenario 1 \u2013 Size of the Dataset is small while the Data similarity is very high \u2013<\/strong> As in this particular case, we do not require to retain the model, as data similarity is very high.<\/p>\n

Although, according to our problem statement, we need to customize and modify the output layers. As we have use pre-trained model here as a feature extractor.<\/p>\n

Further, to identify the new set of images have cat or dogs, we use trained models on Imagenet. Here we require similar images to Imagenet to categorize the two outputs – cats or dogs.<\/p>\n

Finally, at last in this case, we have to modify dense layers. Also, have to put the final softmax layers to output 2\u00a0 categories instead of 1000.<\/p>\n

Scenario 2 \u2013 Size of the data is small as well as data similarity is very low<\/strong>\u00a0\u2013 As in this case, we have to freeze the initial (let\u2019s say k) layers of the pre-trained model. Also, as freezing complete, then train the remaining(n-k) layers again.<\/p>\n

Although, keep in mind that the top layers would be customized to the new data set.\u00a0 Also, initial layers are kept pre-trained by their smaller size. But, keep frozen weights of those layers.<\/p>\n

Scenario 3 \u2013 Size of the data set is large however the Data similarity is very low <\/strong>\u2013 Particularly, in this case, neural network training<\/a> would be more effective. As it’s having a large data set. Also, the main thing is that the data we use is different.<\/p>\n

As we use data is different from data we use in training. Hence, its best to train the neural network from scratch according to your data.<\/p>\n

Scenario 4 \u2013 Size of the data is large as well as there is high data similarity \u2013 <\/strong>We can say this is the final and the ideal situation. As pre-trained models are more effective in this case. Also, we can use this model in very good manner.<\/p>\n

We have to just use the model is to retain the architecture of the model and the initial weights of the model. Moreover,\u00a0 we can retrain this model using the weights as initialized in the pre-trained model.<\/p>\n

\n
\n

Inductive learning and Inductive Transfer<\/h3>\n<\/div>\n
\n
We use this form of transfer learning in the deep learning<\/a>, called for an inductive transfer. Thus, it’s an area where the scope of possible models is narrowed<\/span> in a beneficial way. Although, this model fit into a different but related task.<\/div>\n<\/div>\n<\/div>\n
<\/div>\n
\n
\"Inductive<\/a>

Inductive learning and Inductive Transfer<\/p><\/div>\n<\/div>\n

\n
\n

How to Use Transfer Learning?<\/h3>\n<\/div>\n
\n
Two common approaches for transfer learning are as follows:<\/div>\n<\/div>\n
\n