![\"what](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/what-is-backpropagation.webp\")
<\/a><\/p>\nThis simple network works as follows. First, the input is multiplied by w1, which equals z1. Z1 is the input of the next neuron. The output of the second neuron is f_1(z). Here, f_1 is the activation function of the second neuron. All the layers go through this process. The last layer’s output is the network’s output. Now, this output is compared to the given label for the input. For argument’s sake, let us consider the squared loss.<\/p>\n
Loss=(\u00bd)* (y-fl)2<\/pre>\nNow we will adjust the weights to minimise this loss. We can accomplish this by taking the derivative of the loss for the weights we have to adjust.<\/p>\n
\u10dbLoss\/\u10dbw1=\u10dbf1\/\u10dbw1 * \u10dbLoss\/\u10dbf1\r\n\r\nAlso;\r\n \r\n\u10dbLoss\/\u10dbf1=\u10dbf2\/\u10dbf1*\u10dbLoss\/\u10dbf2\r\n\r\nUsing the chain rule;\r\n\r\n\u10dbLoss\/\u10dbf1=\u10dbf2\/\u10dbf1 *\u10dbf3\/\u10dbf2*\u10dbf4\/\u10dbf3*...............\u10dbfL\/\u10dbfL-1\r\n<\/pre>\nFirst, the penultimate layer adjusts its weight depending on the loss. Then the layer before that is adjusted, and the process goes on. This process of computing the output using the present weights and then propagating backwards to adjust these weights layer by layer is called backpropagation.<\/p>\n
The same process we described above can be used for layers with multiple neurons.<\/p>\n
PyTorch \u201cbackward()\u201d function<\/h3>\n
We can calculate the gradient of a tensor with respect to some other tensor in a neural network only after the “backward()” function is called. It initiates the backpropagation process, and the pointer moves from the tensor in consideration of its gradient function, followed by the calculation of the gradient after reaching the reference tensor.<\/p>\n
Jacobian<\/h3>\n
Jacobian is a matrix which contains all the possible derivatives of a function concerning one or more independent variables.<\/p>\n
Autograd, while computing the gradient using backpropagation, is actually calculating the Jacobian product of the loss vector. However, we need not pay much attention to it as it all happens in the background.<\/p>\n
How does autograd work?<\/h3>\n
The autograd stores the gradients in a Directed Acyclic Graph (DAG). A tensor has parameters – data, grad, grad_fn, is_leaf and requires_grad.<\/p>\n
![\"computation](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2023\/01\/computation-graph.webp\")
<\/a><\/p>\n