{"id":111110,"date":"2023-01-13T11:00:06","date_gmt":"2023-01-13T05:30:06","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=111110"},"modified":"2023-01-13T11:00:04","modified_gmt":"2023-01-13T05:30:04","slug":"optimising-model-parameters-using-pytorch","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/","title":{"rendered":"Optimising Model Parameters Using PyTorch"},"content":{"rendered":"

After a model is built, we train it using our dataset. To do so, we must calculate the loss and optimise the model accordingly. Optimising the parameters is the essential step of the training process because getting the correct (not exact) weights is why we take the burden of dealing with such a huge dataset in the first place. In this article we will see Optimising Model Parameters Using PyTorch.<\/p>\n

How Do We Optimise a Model?<\/h3>\n

When a batch of the dataset passes through the network and outputs are computed, we compare it to the given label of each input and calculate the loss. Then we calculate the gradient of the loss for each parameter (in this case, weights) and try to adjust it to reduce the error. After we have done this with diverse and significant training examples, the weights get generalised, and we say that the training process is complete. There are many ways we can optimise our network, the most popular method being gradient descent.<\/p>\n

What is Gradient Descent?<\/h3>\n

Let J(\u04e8) be the loss function and \u04e8j the parameter we want to optimise. We update \u04e8j as follows:<\/p>\n

\u04e8j:= \u04e8j - \u03b1 \u2202J(\u04e8) \/\u2202\u04e8j<\/pre>\n
Where \u2202J(\u04e8) \/\u2202\u04e8j is the gradient of the loss with respect to the parameter under consideration, and \u03b1 is the learning rate.<\/p>\n
We are basically choosing a point at the loss curve and finding the gradient with respect to a parameter. Depending on the gradient, we are moving a small step of size \u03b1 in a direction where the loss minimises. The next time we do this step, we take the last step\u2019s final point as the starting point and again calculate the gradient and repeat the process. After a few steps depending on the learning rate, we finally reach an optimum value where the loss is minimised.<\/p>\n
Using PyTorch to Optimise the Model Parameters<\/h3>\n
PyTorch has a package called optim, which makes it easy for us to optimise the parameters during the training process. To use it, all we have to do is create an object of torch.optim specifying the model parameters and the learning rate.<\/p>\n
After we have made the model, we import torch.optim and then create an object of optim:<\/p>\n
import torch.optim as optim\r\noptimiser=optim.SGD(net.parameters(),lr=0.01)<\/pre>\n
In this example, we are using Stochastic Gradient Descent to optimise our model. There are other optimisers too which can be used like Adam, ASGD etc. The arguments we have passed are the learning rate and net.parameters(), where the net is an instance of the class of our model.<\/p>\n
The main role of our optimiser is in the training loop.<\/p>\n
for i in range(epoch):\r\n    optimiser.zero_grad()\r\n    output = model(input)\r\n    loss = loss_fn(output, target)\r\n    loss.backward()\r\n    optimiser.step()\r\n<\/pre>\n
In the second line, we call the optimiser.zero_grad to set all the gradients of the network parameters to zero after going through the network once because if we don\u2019t do it, then the gradients of the previous iteration will add up to the gradients of the present iteration and give wrong results.<\/p>\n
Then we find the output of one iteration and the loss. After that, we backpropagate through our network to store the gradients of the parameters and then call the optimiser.step() to optimise the model. It is as simple as that. Our model is now optimised for the current step, and we may move to the next iteration.<\/p>\n
Example (FASHION MNIST classification)<\/h3>\n
a. Importing the required libraries<\/h4>\n
import torch\r\nimport torchvision\r\nimport torch.nn as nn\r\nimport torch.optim as optim\r\nfrom torchvision import transforms,datasets\r\nimport torch.nn.functional as F\r\nimport matplotlib\r\nimport matplotlib.pyplot as plt\r\n<\/pre>\n
b. Preparing the dataset<\/h4>\n
train=datasets.FashionMNIST(\"\", train=True,download=True,transform=transforms.Compose([transforms.ToTensor()]))\r\ntest=datasets.FashionMNIST(\"\", train=False,download=True,transform=transforms.Compose([transforms.ToTensor()]))\r\n \r\ntrainset=torch.utils.data.DataLoader(train,batch_size=64,shuffle=True)\r\ntestset=torch.utils.data.DataLoader(test,batch_size=64,shuffle=True)\r\n<\/pre>\n
Output-<\/h4>\n
\"loading<\/a><\/p>\n
c. Building the Neural Network<\/h4>\n
class ConvNet(nn.Module):\r\n    def __init__(self):\r\n            super(ConvNet,self).__init__()\r\n            \r\n            self.model=nn.Sequential(nn.Conv2d(1,10,5,padding=2),\r\n                        nn.ReLU(),\r\n                        nn.AvgPool2d(2,stride=2),\r\n                        \r\n                        nn.Conv2d(10,20,5,padding=0),\r\n                        nn.ReLU(),\r\n                        nn.AvgPool2d(2,stride=2),\r\n                        \r\n                        nn.Flatten(),\r\n                        nn.Linear(500,250),\r\n                        nn.ReLU(),\r\n                        nn.Linear(250,100),\r\n                        nn.ReLU(),\r\n                        nn.Linear(100,10)\r\n                       )\r\n    \r\n    \r\n    \r\n    \r\n    \r\n    def forward(self, x):\r\n      \r\n        fitt = self.model(x)\r\n        return fitt\r\n<\/pre>\n
d. Initialising the Hyperparameters<\/h4>\n
Hyperparameters are adjustable parameters that alter, or rather control, the training process of the model.<\/p>\n
l_rate:( Learning rate) –<\/strong> It determines the speed at which the model moves down to the optimum solution in the learning rate.<\/p>\n
Batch_size –<\/strong> Determines the number of samples that are fed to the model at once for training.<\/p>\n
Optimiser<\/strong> – This is where we select the optimiser we need for the given task.<\/p>\n
l_rate=0.0001\r\nbatch_size=64\r\n\r\nepoch=10\r\ncnn_model=ConvNet()\r\n \r\n \r\ncel=nn.CrossEntropyLoss()\r\noptimizer=optim.Adam(cnn_model.parameters(),l_rate)\r\n<\/pre>\n
e. Training loop<\/h4>\n
def training(dataloader, model, loss_fn, optimizer):\r\n    size = len(dataloader.dataset)\r\n    for batch, (X, y) in enumerate(dataloader):\r\n        # Compute prediction and loss\r\n        pred = model(X)\r\n        loss = loss_fn(pred, y)\r\n \r\n        # Backpropagation\r\n        optimizer.zero_grad()\r\n        loss.backward()\r\n        optimizer.step()\r\n \r\n        if batch % 100 == 0:\r\n            loss, current = loss.item(), batch * len(X)\r\n            print(f\"loss: {loss:>7f}  [{current:>5d}\/{size:>5d}]\")\r\n<\/pre>\n
f. Training the Model<\/h4>\n
for i in range(epoch):\r\n  print(\"Epoch\",i)\r\n  training(trainset,cnn_model,cel,optimizer)\r\n<\/pre>\n
Output-<\/strong><\/p>\n
\"training<\/a><\/p>\n
\"Training<\/a><\/p>\n
\"output<\/a><\/p>\n
\"Training<\/a><\/p>\n
We can see the loss gradually decreasing and then roughly oscillating between a narrow range of values. It indicates that our model’s parameters have reached very close to the minimum of the loss curve, and our model is now optimal.<\/p>\n
Important Optimisers in PyTorch:<\/h3>\n
1. Adadelta<\/strong><\/p>\n
This optimiser implements the Adadelta algorithm, which is a gradient descent algorithm with an adaptive learning rate per dimension.<\/p>\n
2. Adagrad<\/strong><\/p>\n
The Adagrad optimiser implements the Adagrad algorithm, which has an adaptive learning rate per component.<\/p>\n
3. Adam<\/strong><\/p>\n
The Adam optimiser is a combination of two gradient descent algorithms- Momentum and Root Mean Square Propagation. The momentum algorithm speeds up the descent process based on the exponential weighted average, and the Root Mean Square Propagation method calculates the RMS values instead of the cumulative sum.<\/p>\n
4. AdamW<\/strong><\/p>\n
It is a derivative of the Adam optimiser with improved implementation of weight decay.<\/p>\n
5. SparseAdam<\/strong><\/p>\n
SparseAdam optimiser is an implementation of Adam optimiser. It is best suited for sparse tensors.<\/p>\n
6. Adamax<\/strong><\/p>\n
It is an Adam optimiser with a generalised infinite norm.<\/p>\n
7. LBFGS<\/strong><\/p>\n
This optimiser approximates the Broyden\u2013Fletcher\u2013Goldfarb\u2013Shanno algorithm using limited memory.<\/p>\n
8. NAdam<\/strong><\/p>\n
It is an Adam optimiser with Nesterov-accelerated adaptive movement estimation.<\/p>\n
Summary<\/h3>\n
PyTorch makes it very easy for us to optimise our neural networks owing to its optim package. All we have to do is import torch.optim, create an object, and in the training loop, call optimiser.zero_grad, loss.backward and optimiser.step in the correct order.<\/p>\n","protected":false},"excerpt":{"rendered":"
After a model is built, we train it using our dataset. To do so, we must calculate the loss and optimise the model accordingly. Optimising the parameters is the essential step of the training...<\/p>\n","protected":false},"author":5,"featured_media":111439,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[26498],"tags":[27188],"class_list":["post-111110","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pytorch-tutorials","tag-optimising-model-parameters-using-pytorch"],"yoast_head":"\nOptimising Model Parameters Using PyTorch - DataFlair<\/title>\n<meta name=\"description\" content=\"PyTorch makes it very easy for us to optimise our neural networks owing to its optim package. Learn Optimising Model Parameters Using PyTorch.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Optimising Model Parameters Using PyTorch - DataFlair\" \/>\n<meta property=\"og:description\" content=\"PyTorch makes it very easy for us to optimise our neural networks owing to its optim package. Learn Optimising Model Parameters Using PyTorch.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2023-01-13T05:30:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/optimising-model-parameters-using-pytorch.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Optimising Model Parameters Using PyTorch - DataFlair","description":"PyTorch makes it very easy for us to optimise our neural networks owing to its optim package. Learn Optimising Model Parameters Using PyTorch.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/","og_locale":"en_US","og_type":"article","og_title":"Optimising Model Parameters Using PyTorch - DataFlair","og_description":"PyTorch makes it very easy for us to optimise our neural networks owing to its optim package. Learn Optimising Model Parameters Using PyTorch.","og_url":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2023-01-13T05:30:06+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/optimising-model-parameters-using-pytorch.webp","type":"image\/webp"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823"},"headline":"Optimising Model Parameters Using PyTorch","datePublished":"2023-01-13T05:30:06+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/"},"wordCount":889,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/optimising-model-parameters-using-pytorch.webp","keywords":["Optimising Model Parameters Using PyTorch"],"articleSection":["PyTorch Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/","url":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/","name":"Optimising Model Parameters Using PyTorch - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/optimising-model-parameters-using-pytorch.webp","datePublished":"2023-01-13T05:30:06+00:00","description":"PyTorch makes it very easy for us to optimise our neural networks owing to its optim package. Learn Optimising Model Parameters Using PyTorch.","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/optimising-model-parameters-using-pytorch.webp","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/optimising-model-parameters-using-pytorch.webp","width":1200,"height":628,"caption":"optimising model parameters using pytorch"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/optimising-model-parameters-using-pytorch\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"PyTorch Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pytorch-tutorials\/"},{"@type":"ListItem","position":3,"name":"Optimising Model Parameters Using PyTorch"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/7f83c342f5d1632d6f7b4b0b0f447823","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4cf3a74600d131330b8c481d519afd1574093ed89f6d3396a95393ad223eb7cd?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team creates expert-level guides on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our goal is to empower learners with easy-to-understand content. Explore our resources for career growth and practical learning.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/111110","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=111110"}],"version-history":[{"count":3,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/111110\/revisions"}],"predecessor-version":[{"id":111445,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/111110\/revisions\/111445"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/111439"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=111110"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=111110"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=111110"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}