{"id":1281,"date":"2021-12-01T08:28:04","date_gmt":"2021-12-01T08:28:04","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/12\/01\/application-of-differentiations-in-neural-networks\/"},"modified":"2021-12-01T08:28:04","modified_gmt":"2021-12-01T08:28:04","slug":"application-of-differentiations-in-neural-networks","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/12\/01\/application-of-differentiations-in-neural-networks\/","title":{"rendered":"Application of differentiations in neural networks"},"content":{"rendered":"<div id=\"\">\n<p id=\"last-modified-info\">Last Updated on November 26, 2021<\/p>\n<p>Differential calculus is an important tool in machine learning algorithms. Neural networks in particular, the gradient descent algorithm depends on the gradient, which is a quantity computed by differentiation.<\/p>\n<p>In this tutorial, we will see how the back-propagation technique is used in finding the gradients in neural networks.<\/p>\n<p>After completing this tutorial, you will know<\/p>\n<ul>\n<li>What is a total differential and total derivative<\/li>\n<li>How to compute the total derivatives in neural networks<\/li>\n<li>How back-propagation helped in computing the total derivatives<\/li>\n<\/ul>\n<p>Let\u2019s get started<\/p>\n<div id=\"attachment_4963\" class=\"wp-caption aligncenter\"><img aria-describedby=\"caption-attachment-4963\" loading=\"lazy\" class=\"size-full wp-image-4963\" data-cfsrc=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/11\/freeman-zhou-plX7xeNb3Yo-unsplash.jpg\" alt=\"Application of differentiations in neural networks\" width=\"640\" height=\"480\"><img decoding=\"async\" aria-describedby=\"caption-attachment-4963\" loading=\"lazy\" class=\"size-full wp-image-4963\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/11\/freeman-zhou-plX7xeNb3Yo-unsplash.jpg\" alt=\"Application of differentiations in neural networks\" width=\"640\" height=\"480\"><\/p>\n<p id=\"caption-attachment-4963\" class=\"wp-caption-text\">Application of differentiations in neural networks<br \/>Photo by <a href=\"https:\/\/unsplash.com\/photos\/plX7xeNb3Yo\">Freeman Zhou<\/a>, some rights reserved.<\/p>\n<\/div>\n<h2>Tutorial overview<\/h2>\n<p>This tutorial is divided into 5 parts; they are:<\/p>\n<ol>\n<li>Total differential and total derivatives<\/li>\n<li>Algebraic representation of a multilayer perceptron model<\/li>\n<li>Finding the gradient by back-propagation<\/li>\n<li>Matrix form of gradient equations<\/li>\n<li>Implementing back-propagation<\/li>\n<\/ol>\n<h2>Total differential and total derivatives<\/h2>\n<p>For a function such as $f(x)$, we call denote its derivative as $f'(x)$ or $frac{df}{dx}$. But for a multivariate function, such as $f(u,v)$, we have a partial derivative of $f$ with respect to $u$ denoted as $frac{partial f}{partial u}$, or sometimes written as $f_u$. A partial derivative is obtained by differentiation of $f$ with respect to $u$ while assuming the other variable $v$ is a constant. Therefore, we use $partial$ instead of $d$ as the symbol for differentiation to signify the difference.<\/p>\n<p>However, what if the $u$ and $v$ in $f(u,v)$ are both function of $x$? In other words, we can write $u(x)$ and $v(x)$ and $f(u(x), v(x))$. So $x$ determines the value of $u$ and $v$ and in turn, determines $f(u,v)$. In this case, it is perfectly fine to ask what is $frac{df}{dx}$, as $f$ is eventually determined by $x$.<\/p>\n<p>This is the concept of total derivatives. In fact, for a multivariate function $f(t,u,v)=f(t(x),u(x),v(x))$, we always have<br \/>$$<br \/>frac{df}{dx} = frac{partial f}{partial t}frac{dt}{dx} + frac{partial f}{partial u}frac{du}{dx} + frac{partial f}{partial v}frac{dv}{dx}<br \/>$$<br \/>The above notation is called the total derivative because it is sum of the partial derivatives. In essence, it is applying chain rule to find the differentiation.<\/p>\n<p>If we take away the $dx$ part in the above equation, what we get is an approximate change in $f$ with respect to $x$, i.e.,<br \/>$$<br \/>df = frac{partial f}{partial t}dt + frac{partial f}{partial u}du + frac{partial f}{partial v}dv<br \/>$$<br \/>We call this notation the total differential.<\/p>\n<h2>Algebraic representation of a multilayer perceptron model<\/h2>\n<p>Consider the network:<\/p>\n<div id=\"attachment_4963\" class=\"wp-caption aligncenter\"><img aria-describedby=\"caption-attachment-4963\" loading=\"lazy\" class=\"size-full wp-image-4963\" data-cfsrc=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/3\/30\/Multilayer_Neural_Network.png\" width=\"640\" height=\"480\"><img decoding=\"async\" aria-describedby=\"caption-attachment-4963\" loading=\"lazy\" class=\"size-full wp-image-4963\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/3\/30\/Multilayer_Neural_Network.png\" width=\"640\" height=\"480\"><\/p>\n<p id=\"caption-attachment-4963\" class=\"wp-caption-text\">An example of neural network. Source: <a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Multilayer_Neural_Network.png%22\">https:\/\/commons.wikimedia.org\/wiki\/File:Multilayer_Neural_Network.png<\/a><\/p>\n<\/div>\n<p>This is a simple, fully-connected, 4-layer neural network. Let\u2019s call the input layer as layer 0, the two hidden layers the layer 1 and 2, and the output layer as layer 3. In this picture, we see that we have $n_0=3$ input units, and $n_1=4$ units in the first hidden layer and $n_2=2$ units in the second input layer. There are $n_3=2$ output units.<\/p>\n<p>If we denote the input to the network as $x_i$ where $i=1,cdots,n_0$ and the network\u2019s output as $hat{y}_i$ where $i=1,cdots,n_3$. Then we can write<\/p>\n<p>$$<br \/>begin{aligned}<br \/>h_{1i} &amp;= f_1(sum_{j=1}^{n_0} w^{(1)}_{ij} x_j + b^{(1)}_i) &amp; text{for } i &amp;= 1,cdots,n_1\\<br \/>h_{2i} &amp;= f_2(sum_{j=1}^{n_1} w^{(2)}_{ij} h_{1j} + b^{(2)}_i) &amp; i &amp;= 1,cdots,n_2\\<br \/>hat{y}_i &amp;= f_3(sum_{j=1}^{n_2} w^{(3)}_{ij} h_{2j} + b^{(3)}_i) &amp; i &amp;= 1,cdots,n_3<br \/>end{aligned}<br \/>$$<\/p>\n<p>Here the activation function at layer $i$ is denoted as $f_i$. The outputs of first hidden layer are denoted as $h_{1i}$ for the $i$-th unit. Similarly, the outputs of second hidden layer are denoted as $h_{2i}$. The weights and bias of unit $i$ in layer $k$ are denoted as $w^{(k)}_{ij}$ and $b^{(k)}_i$ respectively.<\/p>\n<p>In the above, we can see that the output of layer $k-1$ will feed into layer $k$. Therefore, while $hat{y}_i$ is expressed as a function of $h_{2j}$, but $h_{2i}$ is also a function of $h_{1j}$ and in turn, a function of $x_j$.<\/p>\n<p>The above describes the construction of a neural network in terms of algebraic equations. Training a neural network would need to specify a *loss function* as well so we can minimize it in the training loop. Depends on the application, we commonly use cross entropy for categorization problems or mean squared error for regression problems. With the target variables as $y_i$, the mean square error loss function is specified as<br \/>$$<br \/>L = sum_{i=1}^{n_3} (y_i-hat{y}_i)^2<br \/>$$<\/p>\n<h2>Finding the gradient by back-propagation<\/h2>\n<p>In the above construct, $x_i$ and $y_i$ are from the dataset. The parameters to the neural network are $w$ and $b$. While the activation functions $f_i$ are by design the outputs at each layer $h_{1i}$, $h_{2i}$, and $hat{y}_i$ are dependent variables. In training the neural network, our goal is to update $w$ and $b$ in each iteration, namely, by the gradient descent update rule:<br \/>$$<br \/>begin{aligned}<br \/>w^{(k)}_{ij} &amp;= w^{(k)}_{ij} \u2013 eta frac{partial L}{partial w^{(k)}_{ij}} \\<br \/>b^{(k)}_{i} &amp;= b^{(k)}_{i} \u2013 eta frac{partial L}{partial b^{(k)}_{i}}<br \/>end{aligned}<br \/>$$<br \/>where $eta$ is the learning rate parameter to gradient descent.<\/p>\n<p>From the equation of $L$ we know that $L$ is not dependent on $w^{(k)}_{ij}$ or $b^{(k)}_i$ but on $hat{y}_i$. However, $hat{y}_i$ can be written as function of $w^{(k)}_{ij}$ or $b^{(k)}_i$ eventually. Let\u2019s see one by one how the weights and bias at layer $k$ can be connected to $hat{y}_i$ at the output layer.<\/p>\n<p>We begin with the loss metric. If we consider the loss of a single data point, we have<br \/>$$<br \/>begin{aligned}<br \/>L &amp;= sum_{i=1}^{n_3} (y_i-hat{y}_i)^2\\<br \/>frac{partial L}{partial hat{y}_i} &amp;= 2(y_i \u2013 hat{y}_i) &amp; text{for } i &amp;= 1,cdots,n_3<br \/>end{aligned}<br \/>$$<br \/>Here we see that the loss function depends on all outputs $hat{y}_i$ and therefore we can find a partial derivative $frac{partial L}{partial hat{y}_i}$.<\/p>\n<p>Now let\u2019s look at the output layer:<br \/>$$<br \/>begin{aligned}<br \/>hat{y}_i &amp;= f_3(sum_{j=1}^{n_2} w^{(3)}_{ij} h_{2j} + b^{(3)}_i) &amp; text{for }i &amp;= 1,cdots,n_3 \\<br \/>frac{partial L}{partial w^{(3)}_{ij}} &amp;= frac{partial L}{partial hat{y}_i}frac{partial hat{y}_i}{partial w^{(3)}_{ij}} &amp; i &amp;= 1,cdots,n_3; j=1,cdots,n_2 \\<br \/>&amp;= frac{partial L}{partial hat{y}_i} f\u2019_3(sum_{j=1}^{n_2} w^{(3)}_{ij} h_{2j} + b^{(3)}_i)h_{2j} \\<br \/>frac{partial L}{partial b^{(3)}_i} &amp;= frac{partial L}{partial hat{y}_i}frac{partial hat{y}_i}{partial b^{(3)}_i} &amp; i &amp;= 1,cdots,n_3 \\<br \/>&amp;= frac{partial L}{partial hat{y}_i}f\u2019_3(sum_{j=1}^{n_2} w^{(3)}_{ij} h_{2j} + b^{(3)}_i)<br \/>end{aligned}<br \/>$$<br \/>Because the weight $w^{(3)}_{ij}$ at layer 3 applies to input $h_{2j}$ and affects output $hat{y}_i$ only. Hence we can write the derivative $frac{partial L}{partial w^{(3)}_{ij}}$ as the product of two derivatives $frac{partial L}{partial hat{y}_i}frac{partial hat{y}_i}{partial w^{(3)}_{ij}}$. Similar case for the bias $b^{(3)}_i$ as well. In the above, we make use of $frac{partial L}{partial hat{y}_i}$, which we already derived previously.<\/p>\n<p>But in fact, we can also write the partial derivative of $L$ with respect to output of second layer $h_{2j}$. It is not used for the update of weights and bias on layer 3 but we will see its importance later:<br \/>$$<br \/>begin{aligned}<br \/>frac{partial L}{partial h_{2j}} &amp;= sum_{i=1}^{n_3}frac{partial L}{partial hat{y}_i}frac{partial hat{y}_i}{partial h_{2j}} &amp; text{for }j &amp;= 1,cdots,n_2 \\<br \/>&amp;= sum_{i=1}^{n_3}frac{partial L}{partial hat{y}_i}f\u2019_3(sum_{j=1}^{n_2} w^{(3)}_{ij} h_{2j} + b^{(3)}_i)w^{(3)}_{ij}<br \/>end{aligned}<br \/>$$<br \/>This one is the interesting one and different from the previous partial derivatives. Note that $h_{2j}$ is an output of layer 2. Each and every output in layer 2 will affect the output $hat{y}_i$ in layer 3. Therefore, to find $frac{partial L}{partial h_{2j}}$ we need to add up every output at layer 3. Thus the summation sign in the equation above. And we can consider $frac{partial L}{partial h_{2j}}$ as the total derivative, in which we applied the chain rule $frac{partial L}{partial hat{y}_i}frac{partial hat{y}_i}{partial h_{2j}}$ for every output $i$ and then sum them up.<\/p>\n<p>If we move back to layer 2, we can derive the derivatives similarly:<br \/>$$<br \/>begin{aligned}<br \/>h_{2i} &amp;= f_2(sum_{j=1}^{n_1} w^{(2)}_{ij} h_{1j} + b^{(2)}_i) &amp; text{for }i &amp;= 1,cdots,n_2\\<br \/>frac{partial L}{partial w^{(2)}_{ij}} &amp;= frac{partial L}{partial h_{2i}}frac{partial h_{2i}}{partial w^{(2)}_{ij}} &amp; i&amp;=1,cdots,n_2; j=1,cdots,n_1 \\<br \/>&amp;= frac{partial L}{partial h_{2i}}f\u2019_2(sum_{j=1}^{n_1} w^{(2)}_{ij} h_{1j} + b^{(2)}_i)h_{1j} \\<br \/>frac{partial L}{partial b^{(2)}_i} &amp;= frac{partial L}{partial h_{2i}}frac{partial h_{2i}}{partial b^{(2)}_i} &amp; i &amp;= 1,cdots,n_2 \\<br \/>&amp;= frac{partial L}{partial h_{2i}}f\u2019_2(sum_{j=1}^{n_1} w^{(2)}_{ij} h_{1j} + b^{(2)}_i) \\<br \/>frac{partial L}{partial h_{1j}} &amp;= sum_{i=1}^{n_2}frac{partial L}{partial h_{2i}}frac{partial h_{2i}}{partial h_{1j}} &amp; j&amp;= 1,cdots,n_1 \\<br \/>&amp;= sum_{i=1}^{n_2}frac{partial L}{partial h_{2i}}f\u2019_2(sum_{j=1}^{n_1} w^{(2)}_{ij} h_{1j} + b^{(2)}_i) w^{(2)}_{ij}<br \/>end{aligned}<br \/>$$<\/p>\n<p>In the equations above, we are reusing $frac{partial L}{partial h_{2i}}$ that we derived earlier. Again, this derivative is computed as a sum of several products from the chain rule. Also similar to the previous, we derived $frac{partial L}{partial h_{1j}}$ as well. It is not used to train $w^{(2)}_{ij}$ nor $b^{(2)}_i$ but will be used for the layer prior. So for layer 1, we have<\/p>\n<p>$$<br \/>begin{aligned}<br \/>h_{1i} &amp;= f_1(sum_{j=1}^{n_0} w^{(1)}_{ij} x_j + b^{(1)}_i) &amp; text{for } i &amp;= 1,cdots,n_1\\<br \/>frac{partial L}{partial w^{(1)}_{ij}} &amp;= frac{partial L}{partial h_{1i}}frac{partial h_{1i}}{partial w^{(1)}_{ij}} &amp; i&amp;=1,cdots,n_1; j=1,cdots,n_0 \\<br \/>&amp;= frac{partial L}{partial h_{1i}}f\u2019_1(sum_{j=1}^{n_0} w^{(1)}_{ij} x_j + b^{(1)}_i)x_j \\<br \/>frac{partial L}{partial b^{(1)}_i} &amp;= frac{partial L}{partial h_{1i}}frac{partial h_{1i}}{partial b^{(1)}_i} &amp; i&amp;=1,cdots,n_1 \\<br \/>&amp;= frac{partial L}{partial h_{1i}}f\u2019_1(sum_{j=1}^{n_0} w^{(1)}_{ij} x_j + b^{(1)}_i)<br \/>end{aligned}<br \/>$$<\/p>\n<p>and this completes all the derivatives needed for training of the neural network using gradient descent algorithm.<\/p>\n<p>Recall how we derived the above: We first start from the loss function $L$ and find the derivatives one by one in the reverse order of the layers. We write down the derivatives on layer $k$ and reuse it for the derivatives on layer $k-1$. While computing the output $hat{y}_i$ from input $x_i$ starts from layer 0 forward, computing gradients are in the reversed order. Hence the name \u201cback-propagation\u201d.<\/p>\n<h2>Matrix form of gradient equations<\/h2>\n<p>While we did not use it above, it is cleaner to write the equations in vectors and matrices. We can rewrite the layers and the outputs as:<br \/>$$<br \/>mathbf{a}_k = f_k(mathbf{z}_k) = f_k(mathbf{W}_kmathbf{a}_{k-1}+mathbf{b}_k)<br \/>$$<br \/>where $mathbf{a}_k$ is a vector of outputs of layer $k$, and assume $mathbf{a}_0=mathbf{x}$ is the input vector and $mathbf{a}_3=hat{mathbf{y}}$ is the output vector. Also denote $mathbf{z}_k = mathbf{W}_kmathbf{a}_{k-1}+mathbf{b}_k$ for convenience of notation.<\/p>\n<p>Under such notation, we can represent $frac{partial L}{partialmathbf{a}_k}$ as a vector (so as that of $mathbf{z}_k$ and $mathbf{b}_k$) and $frac{partial L}{partialmathbf{W}_k}$ as a matrix. And then if $frac{partial L}{partialmathbf{a}_k}$ is known, we have<br \/>$$<br \/>begin{aligned}<br \/>frac{partial L}{partialmathbf{z}_k} &amp;= frac{partial L}{partialmathbf{a}_k}odot f_k'(mathbf{z}_k) \\<br \/>frac{partial L}{partialmathbf{W}_k} &amp;= left(frac{partial L}{partialmathbf{z}_k}right)^top cdot mathbf{a}_k \\<br \/>frac{partial L}{partialmathbf{b}_k} &amp;= frac{partial L}{partialmathbf{z}_k} \\<br \/>frac{partial L}{partialmathbf{a}_{k-1}} &amp;= left(frac{partialmathbf{z}_k}{partialmathbf{a}_{k-1}}right)^topcdotfrac{partial L}{partialmathbf{z}_k} = mathbf{W}_k^topcdotfrac{partial L}{partialmathbf{z}_k}<br \/>end{aligned}<br \/>$$<br \/>where $frac{partialmathbf{z}_k}{partialmathbf{a}_{k-1}}$ is a Jacobian matrix as both $mathbf{z}_k$ and $mathbf{a}_{k-1}$ are vectors, and this Jacobian matrix happens to be $mathbf{W}_k$.<\/p>\n<h2>Implementing back-propagation<\/h2>\n<p>We need the matrix form of equations because it will make our code simpler and avoided a lot of loops. Let\u2019s see how we can convert these equations into code and make a multilayer perceptron model for classification from scratch using numpy.<\/p>\n<p>The first thing we need to implement the activation function and the loss function. Both need to be differentiable functions or otherwise our gradient descent procedure would not work. Nowadays, it is common to use ReLU activation in the hidden layers and sigmoid activation in the output layer. We define them as a function (which assumes the input as numpy array) as well as their differentiation:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a5978344a947293034\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nimport numpy as np<\/p>\n<p># Find a small float to avoid division by zero<br \/>\nepsilon = np.finfo(float).eps<\/p>\n<p># Sigmoid function and its differentiation<br \/>\ndef sigmoid(z):<br \/>\n    return 1\/(1+np.exp(-z.clip(-500, 500)))<br \/>\ndef dsigmoid(z):<br \/>\n    s = sigmoid(z)<br \/>\n    return 2 * s * (1-s)<\/p>\n<p># ReLU function and its differentiation<br \/>\ndef relu(z):<br \/>\n    return np.maximum(0, z)<br \/>\ndef drelu(z):<br \/>\n    return (z &gt; 0).astype(float)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">np<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Find a small float to avoid division by zero<\/span><\/p>\n<p><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">finfo<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">eps<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Sigmoid function and its differentiation<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">sigmoid<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">exp<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">500<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">500<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">dsigmoid<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sigmoid<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e \">s *<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># ReLU function and its differentiation<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">relu<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">maximum<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">drelu<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>We deliberately clip the input of the sigmoid function to between -500 to +500 to avoid overflow. Otherwise, these functions are trivial. Then for classification, we care about accuracy but the accuracy function is not differentiable. Therefore, we use the cross entropy function as loss for training:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a59783452500007575\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# Loss function L(y, yhat) and its differentiation<br \/>\ndef cross_entropy(y, yhat):<br \/>\n    &#8220;&#8221;&#8221;Binary cross entropy function<br \/>\n        L = &#8211; y log yhat &#8211; (1-y) log (1-yhat)<\/p>\n<p>    Args:<br \/>\n        y, yhat (np.array): 1xn matrices which n are the number of data instances<br \/>\n    Returns:<br \/>\n        average cross entropy value of shape 1&#215;1, averaging over the n instances<br \/>\n    &#8220;&#8221;&#8221;<br \/>\n    return -(y.T @ np.log(yhat.clip(epsilon)) + (1-y.T) @ np.log((1-yhat).clip(epsilon))) \/ y.shape[1]<\/p>\n<p>def d_cross_entropy(y, yhat):<br \/>\n    &#8220;&#8221;&#8221; dL\/dyhat &#8220;&#8221;&#8221;<br \/>\n    return &#8211; np.divide(y, yhat.clip(epsilon)) + np.divide(1-y, (1-yhat).clip(epsilon))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># Loss function L(y, yhat) and its differentiation<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Binary cross entropy function<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0L = &#8211; y log yhat &#8211; (1-y) log (1-yhat)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Args:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0y, yhat (np.array): 1xn matrices which n are the number of data instances<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Returns:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0average cross entropy value of shape 1&#215;1, averaging over the n instances<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">T<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">log<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">T<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">log<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">d_cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8221; dL\/dyhat &#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">divide<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">divide<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>In the above, we assume the output and the target variables are row matrices in numpy. Hence we use the dot product operator <code>@<\/code> to compute the sum and divide by the number of elements in the output. Note that this design is to compute the <strong>average cross entropy<\/strong> over a <strong>batch<\/strong> of samples.<\/p>\n<p>Then we can implement our multilayer perceptron model. To make it easier to read, we want to create the model by providing the number of neurons at each layer as well as the activation function at the layers. But at the same time, we would also need the differentiation of the activation functions as well as the differentiation of the loss function for the training. The loss function itself, however, is not required but useful for us to track the progress. We create a class to ensapsulate the entire model, and define each layer $k$ according to the formula:<br \/>$$<br \/>mathbf{a}_k = f_k(mathbf{z}_k) = f_k(mathbf{a}_{k-1}mathbf{W}_k+mathbf{b}_k)<br \/>$<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a59783454077363955\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nclass mlp:<br \/>\n    &#8221;&#8217;Multilayer perceptron using numpy<br \/>\n    &#8221;&#8217;<br \/>\n    def __init__(self, layersizes, activations, derivatives, lossderiv):<br \/>\n        &#8220;&#8221;&#8221;remember config, then initialize array to hold NN parameters without init&#8221;&#8221;&#8221;<br \/>\n        # hold NN config<br \/>\n        self.layersizes = layersizes<br \/>\n        self.activations = activations<br \/>\n        self.derivatives = derivatives<br \/>\n        self.lossderiv = lossderiv<br \/>\n        # parameters, each is a 2D numpy array<br \/>\n        L = len(self.layersizes)<br \/>\n        self.z = [None] * L<br \/>\n        self.W = [None] * L<br \/>\n        self.b = [None] * L<br \/>\n        self.a = [None] * L<br \/>\n        self.dz = [None] * L<br \/>\n        self.dW = [None] * L<br \/>\n        self.db = [None] * L<br \/>\n        self.da = [None] * L<\/p>\n<p>    def initialize(self, seed=42):<br \/>\n        np.random.seed(seed)<br \/>\n        sigma = 0.1<br \/>\n        for l, (insize, outsize) in enumerate(zip(self.layersizes, self.layersizes[1:]), 1):<br \/>\n            self.W[l] = np.random.randn(insize, outsize) * sigma<br \/>\n            self.b[l] = np.random.randn(1, outsize) * sigma<\/p>\n<p>    def forward(self, x):<br \/>\n        self.a[0] = x<br \/>\n        for l, func in enumerate(self.activations, 1):<br \/>\n            # z = W a + b, with `a` as output from previous layer<br \/>\n            # `W` is of size rxs and `a` the size sxn with n the number of data instances, `z` the size rxn<br \/>\n            # `b` is rx1 and broadcast to each column of `z`<br \/>\n            self.z[l] = (self.a[l-1] @ self.W[l]) + self.b[l]<br \/>\n            # a = g(z), with `a` as output of this layer, of size rxn<br \/>\n            self.a[l] = func(self.z[l])<br \/>\n        return self.a[-1]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">mlp<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8221;<\/span><span class=\"crayon-s\">&#8216;Multilayer perceptron using numpy<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8216;<\/span><span class=\"crayon-s\">&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">__init__<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">lossderiv<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;remember config, then initialize array to hold NN parameters without init&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># hold NN config<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">layersizes<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">activations<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">derivatives<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">lossderiv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">lossderiv<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># parameters, each is a 2D numpy array<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">L<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">initialize<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">seed<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">42<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">seed<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">seed<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">sigma<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">insize<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">outsize<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">enumerate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">zip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">randn<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">insize<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">outsize<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sigma<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">randn<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">outsize<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sigma<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">forward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">x<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">enumerate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># z = W a + b, with `a` as output from previous layer<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># `W` is of size rxs and `a` the size sxn with n the number of data instances, `z` the size rxn<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># `b` is rx1 and broadcast to each column of `z`<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># a = g(z), with `a` as output of this layer, of size rxn<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The variables in this class <code>z<\/code>, <code>W<\/code>, <code>b<\/code>, and <code>a<\/code> are for the forward pass and the variables <code>dz<\/code>, <code>dW<\/code>, <code>db<\/code>, and <code>da<\/code> are their respective gradients that to be computed in the back-propagation. All these variables are presented as numpy arrays.<\/p>\n<p>As we will see later, we are going to test our model using data generated by scikit-learn. Hence we will see our data in numpy array of shape \u201c(number of samples, number of features)\u201d. Therefore, each sample is presented as a row on a matrix, and in function <code>forward()<\/code>, the weight matrix is right-multiplied to each input <code>a<\/code> to the layer. While the activation function and dimension of each layer can be different, the process is the same. Thus we transform the neural network\u2019s input <code>x<\/code> to its output by a loop in the <code>forward()<\/code> function. The network\u2019s output is simply the output of the last layer.<\/p>\n<p>To train the network, we need to run the back-propagation after each forward pass. The back-propagation is to compute the gradient of the weight and bias of each layer, starting from the output layer to the input layer. With the equations we derived above, the back-propagation function is implemented as:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a59783455087918602\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nclass mlp:<br \/>\n    &#8230;<\/p>\n<p>    def backward(self, y, yhat):<br \/>\n        # first `da`, at the output<br \/>\n        self.da[-1] = self.lossderiv(y, yhat)<br \/>\n        for l, func in reversed(list(enumerate(self.derivatives, 1))):<br \/>\n            # compute the differentials at this layer<br \/>\n            self.dz[l] = self.da[l] * func(self.z[l])<br \/>\n            self.dW[l] = self.a[l-1].T @ self.dz[l]<br \/>\n            self.db[l] = np.mean(self.dz[l], axis=0, keepdims=True)<br \/>\n            self.da[l-1] = self.dz[l] @ self.W[l].T<\/p>\n<p>    def update(self, eta):<br \/>\n        for l in range(1, len(self.W)):<br \/>\n            self.W[l] -= eta * self.dW[l]<br \/>\n            self.b[l] -= eta * self.db[l]<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">mlp<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-sy\">.<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">backward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># first `da`, at the output<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">lossderiv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">reversed<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">enumerate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># compute the differentials at this layer<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">T<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">axis<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">keepdims<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">T<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">update<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">eta<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">l<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">range<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e \">eta *<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e \">eta *<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>The only difference here is that we compute <code>db<\/code> not for one training sample, but for the entire batch. Since the loss function is the cross entropy averaged across the batch, we compute <code>db<\/code> also by averaging across the samples.<\/p>\n<p>Up to here, we completed our model. The <code>update()<\/code> function simply applies the gradients found by the back-propagation to the parameters <code>W<\/code> and <code>b<\/code> using the gradient descent update rule.<\/p>\n<p>To test out our model, we make use of scikit-learn to generate a classification dataset:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a59783457949204012\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nfrom sklearn.datasets import make_circles<br \/>\nfrom sklearn.metrics import accuracy_score<\/p>\n<p># Make data: Two circles on x-y plane as a classification problem<br \/>\nX, y = make_circles(n_samples=1000, factor=0.5, noise=0.1)<br \/>\ny = y.reshape(-1,1) # our model expects a 2D array of (n_sample, n_dim)<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">datasets <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">make_circles<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">metrics <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-v\">accuracy<\/span><span class=\"crayon-sy\">_<\/span>score<\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Make data: Two circles on x-y plane as a classification problem<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">make_circles<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_samples<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1000<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">factor<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">noise<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reshape<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-p\"># our model expects a 2D array of (n_sample, n_dim)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>and then we build our model: Input is two-dimensional and output is one dimensional (logistic regression). We make two hidden layers of 4 and 3 neurons respectively:<\/p>\n<p><img class=\"aligncenter size-large wp-image-13102\" data-cfsrc=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/11\/neuralnetwork.png\" alt=\"\" width=\"800\"><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-large wp-image-13102\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/11\/neuralnetwork.png\" alt=\"\" width=\"800\"><\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a59783458855788897\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# Build a model<br \/>\nmodel = mlp(layersizes=[2, 4, 3, 1],<br \/>\n            activations=[relu, relu, sigmoid],<br \/>\n            derivatives=[drelu, drelu, dsigmoid],<br \/>\n            lossderiv=d_cross_entropy)<br \/>\nmodel.initialize()<br \/>\nyhat = model.forward(X)<br \/>\nloss = cross_entropy(y, yhat)<br \/>\nprint(&#8220;Before training &#8211; loss value {} accuracy {}&#8221;.format(loss, accuracy_score(y, (yhat &gt; 0.5))))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># Build a model<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">mlp<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">relu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">relu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">sigmoid<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">drelu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">drelu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dsigmoid<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">lossderiv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">d_cross_entropy<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">initialize<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">forward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">loss<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Before training &#8211; loss value {} accuracy {}&#8221;<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">format<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">loss<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">accuracy_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>We see that, under random weight, the accuracy is 50%:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a59783459352658025\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nBefore training &#8211; loss value [[693.62972747]] accuracy 0.5<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>Before training &#8211; loss value [[693.62972747]] accuracy 0.5<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>Now we train our network. To make things simple, we perform full-batch gradient descent with fixed learning rate:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a5978345a049726136\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\n# train for each epoch<br \/>\nn_epochs = 150<br \/>\nlearning_rate = 0.005<br \/>\nfor n in range(n_epochs):<br \/>\n    model.forward(X)<br \/>\n    yhat = model.a[-1]<br \/>\n    model.backward(y, yhat)<br \/>\n    model.update(learning_rate)<br \/>\n    loss = cross_entropy(y, yhat)<br \/>\n    print(&#8220;Iteration {} &#8211; loss value {} accuracy {}&#8221;.format(n, loss, accuracy_score(y, (yhat &gt; 0.5))))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-p\"># train for each epoch<\/span><\/p>\n<p><span class=\"crayon-v\">n_epochs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">150<\/span><\/p>\n<p><span class=\"crayon-v\">learning_rate<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.005<\/span><\/p>\n<p><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">n<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">range<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_epochs<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">forward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">backward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">update<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">learning_rate<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">loss<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Iteration {} &#8211; loss value {} accuracy {}&#8221;<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">format<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">loss<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">accuracy_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>and the output is:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a5978345b308150169\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nIteration 0 &#8211; loss value [[693.62972747]] accuracy 0.5<br \/>\nIteration 1 &#8211; loss value [[693.62166655]] accuracy 0.5<br \/>\nIteration 2 &#8211; loss value [[693.61534159]] accuracy 0.5<br \/>\nIteration 3 &#8211; loss value [[693.60994018]] accuracy 0.5<br \/>\n&#8230;<br \/>\nIteration 145 &#8211; loss value [[664.60120828]] accuracy 0.818<br \/>\nIteration 146 &#8211; loss value [[697.97739669]] accuracy 0.58<br \/>\nIteration 147 &#8211; loss value [[681.08653776]] accuracy 0.642<br \/>\nIteration 148 &#8211; loss value [[665.06165774]] accuracy 0.71<br \/>\nIteration 149 &#8211; loss value [[683.6170298]] accuracy 0.614<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p>Iteration 0 &#8211; loss value [[693.62972747]] accuracy 0.5<\/p>\n<p>Iteration 1 &#8211; loss value [[693.62166655]] accuracy 0.5<\/p>\n<p>Iteration 2 &#8211; loss value [[693.61534159]] accuracy 0.5<\/p>\n<p>Iteration 3 &#8211; loss value [[693.60994018]] accuracy 0.5<\/p>\n<p>&#8230;<\/p>\n<p>Iteration 145 &#8211; loss value [[664.60120828]] accuracy 0.818<\/p>\n<p>Iteration 146 &#8211; loss value [[697.97739669]] accuracy 0.58<\/p>\n<p>Iteration 147 &#8211; loss value [[681.08653776]] accuracy 0.642<\/p>\n<p>Iteration 148 &#8211; loss value [[665.06165774]] accuracy 0.71<\/p>\n<p>Iteration 149 &#8211; loss value [[683.6170298]] accuracy 0.614<\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<p>Although not perfect, we see the improvement by training. At least in the example above, we can see the accuracy was up to more than 80% at iteration 145, but then we saw the model diverged. That can be improved by reducing the learning rate, which we didn\u2019t implement above. Nonetheless, this shows how we computed the gradients by back-propagations and chain rules.<\/p>\n<p>The complete code is as follows:<\/p>\n<div id=\"urvanov-syntax-highlighter-61a6a5978345c103447002\" class=\"urvanov-syntax-highlighter-syntax crayon-theme-classic urvanov-syntax-highlighter-font-monaco urvanov-syntax-highlighter-os-mac print-yes notranslate\" data-settings=\" minimize scroll-mouseover disable-anim\">\n<p><textarea class=\"urvanov-syntax-highlighter-plain print-no\" data-settings=\"dblclick\" readonly><br \/>\nfrom sklearn.datasets import make_circles<br \/>\nfrom sklearn.metrics import accuracy_score<br \/>\nimport numpy as np<br \/>\nnp.random.seed(0)<\/p>\n<p># Find a small float to avoid division by zero<br \/>\nepsilon = np.finfo(float).eps<\/p>\n<p># Sigmoid function and its differentiation<br \/>\ndef sigmoid(z):<br \/>\n    return 1\/(1+np.exp(-z.clip(-500, 500)))<br \/>\ndef dsigmoid(z):<br \/>\n    s = sigmoid(z)<br \/>\n    return 2 * s * (1-s)<\/p>\n<p># ReLU function and its differentiation<br \/>\ndef relu(z):<br \/>\n    return np.maximum(0, z)<br \/>\ndef drelu(z):<br \/>\n    return (z &gt; 0).astype(float)<\/p>\n<p># Loss function L(y, yhat) and its differentiation<br \/>\ndef cross_entropy(y, yhat):<br \/>\n    &#8220;&#8221;&#8221;Binary cross entropy function<br \/>\n        L = &#8211; y log yhat &#8211; (1-y) log (1-yhat)<\/p>\n<p>    Args:<br \/>\n        y, yhat (np.array): nx1 matrices which n are the number of data instances<br \/>\n    Returns:<br \/>\n        average cross entropy value of shape 1&#215;1, averaging over the n instances<br \/>\n    &#8220;&#8221;&#8221;<br \/>\n    return -(y.T @ np.log(yhat.clip(epsilon)) + (1-y.T) @ np.log((1-yhat).clip(epsilon))) \/ y.shape[1]<\/p>\n<p>def d_cross_entropy(y, yhat):<br \/>\n    &#8220;&#8221;&#8221; dL\/dyhat &#8220;&#8221;&#8221;<br \/>\n    return &#8211; np.divide(y, yhat.clip(epsilon)) + np.divide(1-y, (1-yhat).clip(epsilon))<\/p>\n<p>class mlp:<br \/>\n    &#8221;&#8217;Multilayer perceptron using numpy<br \/>\n    &#8221;&#8217;<br \/>\n    def __init__(self, layersizes, activations, derivatives, lossderiv):<br \/>\n        &#8220;&#8221;&#8221;remember config, then initialize array to hold NN parameters without init&#8221;&#8221;&#8221;<br \/>\n        # hold NN config<br \/>\n        self.layersizes = tuple(layersizes)<br \/>\n        self.activations = tuple(activations)<br \/>\n        self.derivatives = tuple(derivatives)<br \/>\n        self.lossderiv = lossderiv<br \/>\n        assert len(self.layersizes)-1 == len(self.activations),<br \/>\n            &#8220;number of layers and the number of activation functions does not match&#8221;<br \/>\n        assert len(self.activations) == len(self.derivatives),<br \/>\n            &#8220;number of activation functions and number of derivatives does not match&#8221;<br \/>\n        assert all(isinstance(n, int) and n &gt;= 1 for n in layersizes),<br \/>\n            &#8220;Only positive integral number of perceptons is allowed in each layer&#8221;<br \/>\n        # parameters, each is a 2D numpy array<br \/>\n        L = len(self.layersizes)<br \/>\n        self.z = [None] * L<br \/>\n        self.W = [None] * L<br \/>\n        self.b = [None] * L<br \/>\n        self.a = [None] * L<br \/>\n        self.dz = [None] * L<br \/>\n        self.dW = [None] * L<br \/>\n        self.db = [None] * L<br \/>\n        self.da = [None] * L<\/p>\n<p>    def initialize(self, seed=42):<br \/>\n        &#8220;&#8221;&#8221;initialize the value of weight matrices and bias vectors with small random numbers.&#8221;&#8221;&#8221;<br \/>\n        np.random.seed(seed)<br \/>\n        sigma = 0.1<br \/>\n        for l, (insize, outsize) in enumerate(zip(self.layersizes, self.layersizes[1:]), 1):<br \/>\n            self.W[l] = np.random.randn(insize, outsize) * sigma<br \/>\n            self.b[l] = np.random.randn(1, outsize) * sigma<\/p>\n<p>    def forward(self, x):<br \/>\n        &#8220;&#8221;&#8221;Feed forward using existing `W` and `b`, and overwrite the result variables `a` and `z`<\/p>\n<p>        Args:<br \/>\n            x (numpy.ndarray): Input data to feed forward<br \/>\n        &#8220;&#8221;&#8221;<br \/>\n        self.a[0] = x<br \/>\n        for l, func in enumerate(self.activations, 1):<br \/>\n            # z = W a + b, with `a` as output from previous layer<br \/>\n            # `W` is of size rxs and `a` the size sxn with n the number of data instances, `z` the size rxn<br \/>\n            # `b` is rx1 and broadcast to each column of `z`<br \/>\n            self.z[l] = (self.a[l-1] @ self.W[l]) + self.b[l]<br \/>\n            # a = g(z), with `a` as output of this layer, of size rxn<br \/>\n            self.a[l] = func(self.z[l])<br \/>\n        return self.a[-1]<\/p>\n<p>    def backward(self, y, yhat):<br \/>\n        &#8220;&#8221;&#8221;back propagation using NN output yhat and the reference output y, generates dW, dz, db,<br \/>\n        da<br \/>\n        &#8220;&#8221;&#8221;<br \/>\n        assert y.shape[1] == self.layersizes[-1], &#8220;Output size doesn&#8217;t match network output size&#8221;<br \/>\n        assert y.shape == yhat.shape, &#8220;Output size doesn&#8217;t match reference&#8221;<br \/>\n        # first `da`, at the output<br \/>\n        self.da[-1] = self.lossderiv(y, yhat)<br \/>\n        for l, func in reversed(list(enumerate(self.derivatives, 1))):<br \/>\n            # compute the differentials at this layer<br \/>\n            self.dz[l] = self.da[l] * func(self.z[l])<br \/>\n            self.dW[l] = self.a[l-1].T @ self.dz[l]<br \/>\n            self.db[l] = np.mean(self.dz[l], axis=0, keepdims=True)<br \/>\n            self.da[l-1] = self.dz[l] @ self.W[l].T<br \/>\n            assert self.z[l].shape == self.dz[l].shape<br \/>\n            assert self.W[l].shape == self.dW[l].shape<br \/>\n            assert self.b[l].shape == self.db[l].shape<br \/>\n            assert self.a[l].shape == self.da[l].shape<\/p>\n<p>    def update(self, eta):<br \/>\n        &#8220;&#8221;&#8221;Updates W and b<\/p>\n<p>        Args:<br \/>\n            eta (float): Learning rate<br \/>\n        &#8220;&#8221;&#8221;<br \/>\n        for l in range(1, len(self.W)):<br \/>\n            self.W[l] -= eta * self.dW[l]<br \/>\n            self.b[l] -= eta * self.db[l]<\/p>\n<p># Make data: Two circles on x-y plane as a classification problem<br \/>\nX, y = make_circles(n_samples=1000, factor=0.5, noise=0.1)<br \/>\ny = y.reshape(-1,1) # our model expects a 2D array of (n_sample, n_dim)<br \/>\nprint(X.shape)<br \/>\nprint(y.shape)<\/p>\n<p># Build a model<br \/>\nmodel = mlp(layersizes=[2, 4, 3, 1],<br \/>\n            activations=[relu, relu, sigmoid],<br \/>\n            derivatives=[drelu, drelu, dsigmoid],<br \/>\n            lossderiv=d_cross_entropy)<br \/>\nmodel.initialize()<br \/>\nyhat = model.forward(X)<br \/>\nloss = cross_entropy(y, yhat)<br \/>\nprint(&#8220;Before training &#8211; loss value {} accuracy {}&#8221;.format(loss, accuracy_score(y, (yhat &gt; 0.5))))<\/p>\n<p># train for each epoch<br \/>\nn_epochs = 150<br \/>\nlearning_rate = 0.005<br \/>\nfor n in range(n_epochs):<br \/>\n    model.forward(X)<br \/>\n    yhat = model.a[-1]<br \/>\n    model.backward(y, yhat)<br \/>\n    model.update(learning_rate)<br \/>\n    loss = cross_entropy(y, yhat)<br \/>\n    print(&#8220;Iteration {} &#8211; loss value {} accuracy {}&#8221;.format(n, loss, accuracy_score(y, (yhat &gt; 0.5))))<\/textarea><\/p>\n<div class=\"urvanov-syntax-highlighter-main\">\n<table class=\"crayon-table\">\n<tr class=\"urvanov-syntax-highlighter-row\">\n<td class=\"crayon-nums \" data-settings=\"show\">\n<div class=\"urvanov-syntax-highlighter-nums-content\">\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>40<\/p>\n<p>41<\/p>\n<p>42<\/p>\n<p>43<\/p>\n<p>44<\/p>\n<p>45<\/p>\n<p>46<\/p>\n<p>47<\/p>\n<p>48<\/p>\n<p>49<\/p>\n<p>50<\/p>\n<p>51<\/p>\n<p>52<\/p>\n<p>53<\/p>\n<p>54<\/p>\n<p>55<\/p>\n<p>56<\/p>\n<p>57<\/p>\n<p>58<\/p>\n<p>59<\/p>\n<p>60<\/p>\n<p>61<\/p>\n<p>62<\/p>\n<p>63<\/p>\n<p>64<\/p>\n<p>65<\/p>\n<p>66<\/p>\n<p>67<\/p>\n<p>68<\/p>\n<p>69<\/p>\n<p>70<\/p>\n<p>71<\/p>\n<p>72<\/p>\n<p>73<\/p>\n<p>74<\/p>\n<p>75<\/p>\n<p>76<\/p>\n<p>77<\/p>\n<p>78<\/p>\n<p>79<\/p>\n<p>80<\/p>\n<p>81<\/p>\n<p>82<\/p>\n<p>83<\/p>\n<p>84<\/p>\n<p>85<\/p>\n<p>86<\/p>\n<p>87<\/p>\n<p>88<\/p>\n<p>89<\/p>\n<p>90<\/p>\n<p>91<\/p>\n<p>92<\/p>\n<p>93<\/p>\n<p>94<\/p>\n<p>95<\/p>\n<p>96<\/p>\n<p>97<\/p>\n<p>98<\/p>\n<p>99<\/p>\n<p>100<\/p>\n<p>101<\/p>\n<p>102<\/p>\n<p>103<\/p>\n<p>104<\/p>\n<p>105<\/p>\n<p>106<\/p>\n<p>107<\/p>\n<p>108<\/p>\n<p>109<\/p>\n<p>110<\/p>\n<p>111<\/p>\n<p>112<\/p>\n<p>113<\/p>\n<p>114<\/p>\n<p>115<\/p>\n<p>116<\/p>\n<p>117<\/p>\n<p>118<\/p>\n<p>119<\/p>\n<p>120<\/p>\n<p>121<\/p>\n<p>122<\/p>\n<p>123<\/p>\n<p>124<\/p>\n<p>125<\/p>\n<p>126<\/p>\n<p>127<\/p>\n<p>128<\/p>\n<p>129<\/p>\n<p>130<\/p>\n<p>131<\/p>\n<p>132<\/p>\n<p>133<\/p>\n<p>134<\/p>\n<p>135<\/p>\n<p>136<\/p>\n<p>137<\/p>\n<p>138<\/p>\n<p>139<\/p>\n<p>140<\/p>\n<p>141<\/p>\n<p>142<\/p>\n<p>143<\/p>\n<\/div>\n<\/td>\n<td class=\"urvanov-syntax-highlighter-code\">\n<div class=\"crayon-pre\">\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">datasets <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">make_circles<\/span><\/p>\n<p><span class=\"crayon-e\">from <\/span><span class=\"crayon-v\">sklearn<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">metrics <\/span><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">accuracy_score<\/span><\/p>\n<p><span class=\"crayon-e\">import <\/span><span class=\"crayon-e\">numpy <\/span><span class=\"crayon-st\">as<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">np<\/span><\/p>\n<p><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">seed<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Find a small float to avoid division by zero<\/span><\/p>\n<p><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">finfo<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">eps<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Sigmoid function and its differentiation<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">sigmoid<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">exp<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">500<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">500<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">dsigmoid<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sigmoid<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e \">s *<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">s<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># ReLU function and its differentiation<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">relu<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">maximum<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">drelu<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">astype<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-t\">float<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Loss function L(y, yhat) and its differentiation<\/span><\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Binary cross entropy function<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0L = &#8211; y log yhat &#8211; (1-y) log (1-yhat)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Args:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0y, yhat (np.array): nx1 matrices which n are the number of data instances<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0Returns:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0average cross entropy value of shape 1&#215;1, averaging over the n instances<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">T<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">log<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">T<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">log<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">\/<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">d_cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8221; dL\/dyhat &#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">divide<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">divide<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">clip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">epsilon<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-t\">class<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">mlp<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8221;<\/span><span class=\"crayon-s\">&#8216;Multilayer perceptron using numpy<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0&#8216;<\/span><span class=\"crayon-s\">&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">__init__<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">lossderiv<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;remember config, then initialize array to hold NN parameters without init&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># hold NN config<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tuple<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tuple<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">tuple<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">lossderiv<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">lossderiv<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\"><\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;number of layers and the number of activation functions does not match&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\"><\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;number of activation functions and number of derivatives does not match&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">all<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">isinstance<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-t\">int<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">and<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">n<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\"><\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;Only positive integral number of perceptons is allowed in each layer&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># parameters, each is a 2D numpy array<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">L<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">None<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">L<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">initialize<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">seed<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">42<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;initialize the value of weight matrices and bias vectors with small random numbers.&#8221;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">seed<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">seed<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">sigma<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.1<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">insize<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">outsize<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">enumerate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">zip<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-o\">:<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">randn<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">insize<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">outsize<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sigma<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">random<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">randn<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">outsize<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">sigma<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">forward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">x<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Feed forward using existing `W` and `b`, and overwrite the result variables `a` and `z`<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Args:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0x (numpy.ndarray): Input data to feed forward<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">x<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">enumerate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># z = W a + b, with `a` as output from previous layer<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># `W` is of size rxs and `a` the size sxn with n the number of data instances, `z` the size rxn<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># `b` is rx1 and broadcast to each column of `z`<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">+<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># a = g(z), with `a` as output of this layer, of size rxn<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">return<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">backward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;back propagation using NN output yhat and the reference output y, generates dW, dz, db,<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0da<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Output size doesn&#8217;t match network output size&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-s\">&#8220;Output size doesn&#8217;t match reference&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># first `da`, at the output<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">lossderiv<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">reversed<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">list<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-e\">enumerate<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-p\"># compute the differentials at this layer<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">*<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">func<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">T<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">np<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">mean<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">axis<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">keepdims<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-t\">True<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">@<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-i\">T<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">z<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dz<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">shape<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">shape<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">shape<\/span><\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">assert<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">==<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">da<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">shape<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-e\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">def <\/span><span class=\"crayon-e\">update<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">eta<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><span class=\"crayon-s\">&#8220;Updates W and b<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Args:<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0eta (float): Learning rate<\/span><\/p>\n<p><span class=\"crayon-s\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0&#8220;<\/span><span class=\"crayon-s\">&#8220;&#8221;<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">l<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">range<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">len<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">W<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e \">eta *<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">dW<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">b<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">-=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e \">eta *<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-r\">self<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">db<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">l<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Make data: Two circles on x-y plane as a classification problem<\/span><\/p>\n<p><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">make_circles<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_samples<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">1000<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">factor<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">noise<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-cn\">0.1<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">y<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">reshape<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-p\"># our model expects a 2D array of (n_sample, n_dim)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">shape<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># Build a model<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">mlp<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">layersizes<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-cn\">2<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">4<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">3<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">activations<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">relu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">relu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">sigmoid<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">derivatives<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-v\">drelu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">drelu<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">dsigmoid<\/span><span class=\"crayon-sy\">]<\/span><span class=\"crayon-sy\">,<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">lossderiv<\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-v\">d_cross_entropy<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">initialize<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">forward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-v\">loss<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Before training &#8211; loss value {} accuracy {}&#8221;<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">format<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">loss<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">accuracy_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span class=\"crayon-p\"># train for each epoch<\/span><\/p>\n<p><span class=\"crayon-v\">n_epochs<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">150<\/span><\/p>\n<p><span class=\"crayon-v\">learning_rate<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.005<\/span><\/p>\n<p><span class=\"crayon-st\">for<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-i\">n<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-st\">in<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">range<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n_epochs<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-o\">:<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">forward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">X<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-v\">a<\/span><span class=\"crayon-sy\">[<\/span><span class=\"crayon-o\">&#8211;<\/span><span class=\"crayon-cn\">1<\/span><span class=\"crayon-sy\">]<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">backward<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">model<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">update<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">learning_rate<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-v\">loss<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">=<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">cross_entropy<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<p><span class=\"crayon-h\">\u00a0\u00a0\u00a0\u00a0<\/span><span class=\"crayon-e\">print<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-s\">&#8220;Iteration {} &#8211; loss value {} accuracy {}&#8221;<\/span><span class=\"crayon-sy\">.<\/span><span class=\"crayon-e\">format<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">n<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-v\">loss<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-e\">accuracy_score<\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">y<\/span><span class=\"crayon-sy\">,<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-sy\">(<\/span><span class=\"crayon-v\">yhat<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-o\">&gt;<\/span><span class=\"crayon-h\"> <\/span><span class=\"crayon-cn\">0.5<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><span class=\"crayon-sy\">)<\/span><\/p>\n<\/div>\n<\/td>\n<\/tr>\n<\/table><\/div>\n<\/p><\/div>\n<h2>Further readings<\/h2>\n<p>The back-propagation algorithm is the center of all neural network training, regardless of what variation of gradient descent algorithms you used. Textbook such as this one covered it:<\/p>\n<p>Previously also implemented the neural network from scratch without discussing the math, it explained the steps in greater detail:<\/p>\n<h2>Summary<\/h2>\n<p>In this tutorial, you learned how differentiation is applied to training a neural network.<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>What is a total differential and how it is expressed as a sum of partial differentials<\/li>\n<li>How to express a neural network as equations and derive the gradients by differentiation<\/li>\n<li>How back-propagation helped us to express the gradients of each layer in the neural network<\/li>\n<li>How to convert the gradients into code to make a neural network model<\/li>\n<\/ul><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/machinelearningmastery.com\/application-of-differentiations-in-neural-networks\/<\/p>\n","protected":false},"author":0,"featured_media":1282,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1281"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1281"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1281\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1282"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1281"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1281"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1281"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}