PyTorch Terminologies That You Must Know

In this article, we will learn about PyTorch Terminologies. So let’s see the PyTorch Glossary!!

PyTorch Terminologies

1. Batch Size

It is the number of samples that are fed into the neural network at once. When a model is trained, we don’t train it on all the samples at once but by it’s small subset. This helps the network to generalise better, improving the performance of the model on the test set.

2. Gradient

Gradient is the rate of change of output with respect to any input parameter. While training a neural network we often need to calculate the gradient of the loss with respect to the inputs to backpropagate and fit the model to hte training set. Since it is a crucial operation PyTorch provides an inbuilt function called autograd that can calculate the required gradients.

3. Distributed Data-Parallel

Distributed Data-Parallel implies training the same model on multiple GPUs to save time. Sometimes, the samples in the training set may be so numerous that it may take days to completely train the model using a single GPU. To avoid this delay we can load a model on multiple GPUs and give each of them only a subset of the training set, exhausting all samples available to us. This will train the neural network in a shorter duration and when it’s done we can use the aggregated model.

4. Dynamic Graphs

Earlier the neural network graphs were static, meaning that once trained and modelled they could not be updated with new data. Dynamic Graphs are the ones that can learn and improve on getting new information. This is widely used in social media, netflix’s, or any OTT platform for that matter, recommender system, financial transactions etc.

5. Convolution Operation

A convolution operation implies computing the dot product between the filter matrix of order n x m and all the possible n x m sub-matrices of the input. It is mainly used to identify the patterns like edges, circles, brightness etc in an image.

6. Stride

In the convolution operation, stride is the measure of column rows of the input matrix that the filter matrix moves between two convolution operations. Smaller teh stride larger will be the feature map, i.e output matrix, and vice-versa.

7. ATen

It is a core tensor library on which many operations are built. It forms the foundation on which the interfaces in PyTorch are built.

8. Epoch

Epoch is the number of times the training set has passed through the neural network while training.

9. TorchScript

It is a static subset of Python which can be used to represent neural networks.We can load a torchscript code even if we have no python dependency.

10. Kernel

A kernel is a single or multiple block of memory where all the operations of a model are carried out.

Summary

There are some terms which when used in the context of PyTorch have some specific meaning. However, these are more or less similar to their usual definitions.