Torch.nn in PyTorch

The basic construct of different parts of Neural Networks remains the same for different applications. Therefore it would make little sense to code the standard modules and classes over and over again. PyTorch torch.nn module contains all the required neural layers, activation functions, loss functions and all the other essential building blocks required for a neural network project. Our only job is to design the model by invoking all these features as and when required. It eliminates the need to code even the basic methods from scratch. Let’s learn more about it.

Containers in PyTorch

a. Module:

Module is the base class for all Neural Network classes. Every Neural Network class we build inherits this class as it has some features that are already built. These are common among all models, saving us time and avoiding repeating the same code repeatedly.

b. Sequential:

It is a sequential container to which modules get added in the order they are executed.

c. ModuleList:

It holds all the submodules in a list which can be called and indexed like python lists.

d. ModuleDict:

It holds modules as a key-value pair, just like a python dictionary.

e. ParameterList:

It holds all the parameters in a list and can be indexed like python lists.

f. ParameterDict:

It holds parameters in a key-value pair and can be indexed like a python dictionary.

g. register_module_forward_pre_hook:

It registers a forward pre-hook that is common to all modules.

h. register_module_forward_pre_hook:

It registers a global forward hook for all the modules.

i. register_module_backward_hook:

It registers a global backward hook which is common to all the modules.

j. register_module_full_backward_hook:

It registers a backward hook common to all the modules.

Convolution Layers in PyTorch

It is a collection of PyTorch’s inbuilt convolutional layers.

a. nn.Conv1d:

It performs one-dimensional convolution over the input data given to it.

b. nn.Conv2d:

It performs two-dimensional convolution over the input data.

c. nn.Conv3d:

It performs three-dimensional convolution over the input data.

d. nn.ConvTranspose1d:

It performs one-dimensional transposed convolution and can be considered as the gradient of the 1D convolution with respect to the inputs.

e. nn.ConvTranspose2d:

It performs transposed two-dimensional convolution over the input image and can be considered as the gradient of Conv2d with respect to the input.

f. nn.ConvTranspose3d:

It performs transposed 3-D convolution over the input and can be considered as the gradient of Conv3d with respect to the input.

g. nn.LazyConv1D:

It performs one-dimensional convolution with lazy initialisation of the input arguments.

h. nn.LazyConv2D:

It performs two-dimensional convolution with lazy initialisation of the input arguments.

i. nn.LazyConv3D:

It performs three-dimensional convolution with lazy initialisation of the input arguments.

j. nn.LazyConvTranspose1D:

It is an nn.ConvTranspose1D function with lazy initialisations of the input arguments.

k. nn.LazyConvTranspose2D:

It is an nn.ConvTranspose2D function with lazy initialisations of the input arguments.

l. nn.LazyConvTranspose3D:

It is an nn.ConvTranspose3D function with lazy initialisations of the input arguments.

m. nn.Unfold:

It extracts the sliding local blocks from a batched input tensor.

n. nn.Fold:

It combines an array of sliding local blocks to form a large containing tensor.

PyTorch Pooling layers:

Pooling refers to reducing the size of the activation map.

a. nn.MaxPool1d:

It applies one-dimensional max pooling over the input signal.

b. nn.MaxPool2d:

It applies two-dimensional max pooling over the input signal.

c. nn.MaxPool3d:

It applies three-dimensional max pooling over the input signal.

d. nn.MaxUnpool1d:

It calculates the partial inverse of MaxPool1D.

e. nn.MaxUnpool2d:

It calculates the partial inverse of MaxPool2D.

f. nn.MaxUnpool3d:

It calculates the partial inverse of MaxPool3D.

g. nn.AvgPool1d:

It is used to calculate the average pool of one-dimensional input.

h. nn.AvgPool2d:

It is used to calculate the average pool of two-dimensional input

i. nn.AvgPool3d:

It is used to calculate the average pool of three-dimensional input.

j. nn.FractionMaxPool2d:

It is used to calculate the fractional max pool of two-dimensional input.

k. nn.FractionMaxPool3d:

It is used to calculate the fractional max pool of three-dimensional input.

l. nn.LPool1d:

It is used to calculate the power average pooling of one-dimensional input.

m. nn.LPool2d:

It is used to calculate the power average pooling of two-dimensional input.

n. nn.AdaptiveMaxPool1d:

It is used to calculate the adaptive max-pooling of one-dimensional input.

o. nn.AdaptoveMaxPool2d:

It is used to calculate the adaptive max-pooling of two-dimensional input.

p. nn.adaptiveMaxPool3d:

It is used to calculate the adaptive max-pooling of three-dimensional input.

q. nn.AdaptiveAvgPool1d:

It is used to calculate the adaptive average pooling of one-dimensional input.

r. nn.AdaptiveAvgPool2d:

It is used to calculate the adaptive average pooling of two-dimensional input.

s. nn.AdaptiveAvgPool3d:

It is used to calculate the adaptive average max-pooling of three-dimensional input.

Padding Layers in PyTorch

It refers to adding extra pixels to increase the dimension of the input matrix.

a. nn.ReflectionPad1d:

It is used to pad the input tensor using the reflection of the one-dimensional input boundary.

b. nn.ReflectionPad2d:

It is used to pad the input tensor using the reflection of the two-dimensional input boundary.

c. nn.ReflectionPad3d:

It is used to pad the input tensor using the reflection of the three-dimensional input boundary.

d. nn.ReplicationPad1d:

It is used to pad the input tensor using the reflection of the one-dimensional input boundary.

e. nn.ReplicationPad2d:

It is used to pad the input tensor using the replication of the two-dimensional input boundary.

f. nn.ReplicationPad3d:

It is used to pad the input tensor using the replication of the three-dimensional input boundary.

g. nn.ZeroPad2d:

It is used to pad the input tensor with zeros for two-dimensional input.

h. nn.ConstantPad1d:

It is used to pad the input tensor with constants for one-dimensional input.

i. nn.ConstantPad2d:

It is used to pad the input tensor with constants for two-dimensional input.

j. nn.ConstantPad3d:

It is used to pad the input tensor with constants for three-dimensional input.

Non-linear Activations in PyTorch

a. nn.ELU:

#image#

b. nn.Hardshrink:

It applies hardshrink function to the input tensor inputwise.

#IMAGE#

c. nn.Hardsigmoid:

#IMAGE#

d. nn.Hardtanh:

#IMAGE#

e. nn.Hardswish:

#image#

f. nn.LeakyReLU:

#image#

g. nn.LogSigmoid:

#image#

h. nn.PReLU:

#image#

i. nn.ReLU:

#image#

j. nn.ReLU6:

#image#

k. nn.RReLU:

#image#

Here, ‘a’ is randomly assigned from the uniform distribution.

l. nn.SELU:

SELU(x)=scale∗(max(0,x)+min(0,α∗(exp(x)−1)))

α=1.6732632423543772848170429916717
And scale=1.0507009873554804934193349852946

m. nn.CELU:

#image#

n. nn.GELU:

#image#

Where, Φ(x) is the Cumulative Distribution Function for Gaussian Distribution.

o. nn.Sigmoid:

#image#

p. nn.SiLU:

#image#

q. nn.Mish:

#image#

r. nn.Softplus:

#image#

s. nn.Softshrink:

#imsge#

t. nn.Softsign:

#image#

u. nn.Tanh:

#image#

v. nn.Tanshrink:

#image#

w. n.Threshold:

#image#

a is the value we want to replace x with.

x. nn.GLU:

#image#

Where a and b are the first and the second half of the matrix respectively.

y. nn.Softmin:

#image#

z. nn.Softmax:

#image#

aa. nn.Softmax2d:

It applies Softmax at each spatial location.

bb. nn.LogSoftmax:

#image#

cc. nn.AdaptiveLogSoftmaxWithLoss:

It is used to partition the labels into clusters so as to minimise the cost of accessing the most frequent labels.

PyTorch Normalisation Layers

a. nn.BatchNorm1d:

It applies the batch normalisation of the input tensor of up to three-dimensions.

b. nn.BatchNorm2d:

It applies batch normalisation of the input tensor of four-dimensions.

c. nn.BatchNorm3d:

It applies batch normalisation of the input tensor of five-dimensions.

d. nn.LazyBatchNorm1d:

Applies batch normalisation over three-dimension input tensor with lazy initialisation of num_features argument of nn.BatchNorm1d.

e. nn.LazyBatchNorm2d:

Applies batch normalisation over four-dimension input tensor with lazy initialisation of num_features argument of nn.BatchNorm2d.

f. nn.LazyBatchNorm3d:

Applies batch normalisation over five-dimension input tensor with lazy initialisation of num_features argument of nn.BatchNorm3d.

g. nn.GroupNorm:

It applies Group Normalisation over a mini-batch of inputs.

h. nn.SyncBatchNorm:

It applies Batch Normalisation over an N-Dimensional input (a mini-batch of [N-2]D inputs).

i. nn.InstanceNorm1d:

It applies instance normalisation over the input tensor of three-dimension.

j. nn.InstanceNorm2d:

It applies instance normalisation over the input tensor of four-dimension.

k. nn.InstanceNorm3d:

It applies instance normalisation over the input tensor of five-dimension.

l. nn.LazyInstanceNorm1d:

It applies instance normalisation over the input tensor of three-dimension with lazy initialisation of the num_features argument.

m. nn.LazyInstanceNorm2d:

It applies instance normalisation over the input tensor of four-dimension with lazy initialisation of the num_features argument.

n. nn.LazyInstanceNorm3d:

It applies instance normalisation over the input tensor of five-dimension with lazy initialisation of the num_features argument.

o. nn.LayerNorm:

It applies Layer Normalisation over a mini-batch of input tensors.

p. nn.LocalResponseNorm:

It applies local response normalisation over an input signal which comprises several input planes.

PyTorch Recurrent Layers:

a. nn.RNN:

It applies a multi-layer Elman RNN with tanh or ReLU non-linear activation.

#image#

b. nn.LSTM:

It applies the multilayer LSTM (Long Short Term Memory) RNN to the input signal. All the elements go through the following operations:

#image#

c. nn.GRU:

It applies the multi-layer gated recurrent unit (GRU) to the input signal. All the elements go through the following operations:

#image#

d. nn.RNNCell:

It is an Elman cell with tanh or ReLU being the activation function.

#image#

e. nn.LSTMCell:

It is a LSTM cell.

#image#

f. nn.GRUCell:

It is a gated recurrent unit cell.

#image#

PyTorch Linear Layers:

a. nn.Identity:

It is a placeholder identity operator that is argument-insensitive.

b. nn.Linear:

It applies a linear transform, #mage#, to the input signal.

c. nn.Bilinear:

It applies a bilinear transform,#IMAGE# to the input data.

PyTorch Dropout Layers:

a. nn.Dropout:

It randomly zeroes some elements of the input tensor depending on the probability depending on the Bernoulli distribution.

b. nn.Dropout2d:

It randomly zeros the elements of an entire channel.

c. nn.Dropout3d:

It randomly zeros the elements of an entire channel of a 3-D input.

d. nn.FeatureAlphaDropout:

It randomly masks out an entire feature(channel).

PyTorch Sparse Layers:

a. nn.Embedding:

It stores the embeddings of a fixed dictionary and size.

b. nn.EmbeddingBag:

Computes sums or means of a collection of samples(‘bags’) of embeddings without instantiating the intermediate embeddings.

PyTorch Distance Functions:

a. nn.CosineSimilarity:

It computes and returns the cosine similarity between two inputs in the given dimension.

b. nn.PairwiseDistance:

It computes the pairwise distance between two vectors.

PyTorch Loss Function:

a. nn.L1Loss:

It measures the Mean Absolute Error between the input x and target y.

b. nn.MSELoss:

It measures the Mean Squared Error between the input x and target y.

c. nn.CrossEntropyLoss:

It measures the Cross Entropy Error between the input x and target y.

d. nn.CTCLoss:

It measures the Connectionist Temporal Classification loss.

e. nn.NLLLoss:

It measures the Negative Log Likelihood loss.

f. nn.PoissonNLLLoss:

It measures the Negative Log Likelihood loss with Poisson Distribution Target.

g. nn.GaussianNLLLoss:

It measures the Gaussian Negative Log Likelihood loss.

h. nn.KLDivLoss:

It measures the Kullback-Leibler loss.

i. nn.BECLoss:

It measures the Binary Cross Entropy between the probabilities of input and the target.

j. nn.BECWithLogitsLoss:

It combines a Sigmoid layer and BCELoss in one single class.

PyTorch Vision Layers

a. nn.PixelShuffle:

It rearranges elements in a tensor with shape (∗,C×r2,H,W) to a tensor with shape (∗,C,H×r,W×r) with r being the upscale factor.

b. nn.PixelUnshuffle:

It reverses the PixelShuffle operation.

c. nn.Upsample:

It upsamples the multi-channel input data.

d. nn.UpsamplingNearest2d:

It upsamples the multi-channel input data by applying the 2D nearest neighbour sampling.

e. nn.UpsamplingBilinear2d:

It upsamples the multi-channel input data by applying 2D bilinear sampling.

PyTorch Shuffle Layers

a. nn.ChannelShuffle:

It shuffles the input data by dividing the input channels into several groups keeping the original tensor shape.

PyTorch DataParallel Layers:

a. nn.DataParallel:

It implements data parallelism at the module level.

b. nn.parallel.DistributionParallel:

It implements distributed data parallelism at the module level.

PyTorch Utilities:

a. Clip_grad_norm:

It clips the norm of the gradient of an iterable parameter.

b. Clip_grad_value:

It clips the gradient of an iterable parameter when it crosses a certain threshold.

c. Parameters_to_vectors:

It merges all the parameters into one vector.

d. Vectors_to_parameters:

It converts the vectors to parameters.

e. prune.BasePruningMethod:

It is an abstract base class which can create new pruning techniques.

f. prune.PruningContainer:

It is a container that holds several pruning methods for the purpose of iterative pruning.

g. prune.Identity:

It does not prune any units. Instead, it generates the pruning parametrization with a mask of ones.

h. prune.RandomUnstructured:

It prunes an entire channel in a tensor at random.

i. prune.L1Unstructured:

It prune units in a tensor by zeroing out the ones with the lowest L1-norm.

PyTorch Quantized Functions:

It enables the storage of tensors and performs computations on them at low bandwidth.

a. Lazy Module Initialisations:

A mixin for modules that lazily initialise parameters.

Summary

torch.nn can be considered to be the soul of PyTorch as it contains all the essential modules required for Deep Learning tasks like designing the convolutional or recurrent layers, defining the loss function and preprocessing the given data to ease the fitting of data to the neural network.