{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":1027.859914,"end_time":"2025-11-09T11:30:16.440207","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2025-11-09T11:13:08.580293","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"349573e6","cell_type":"markdown","source":"## 1. Introduction","metadata":{"papermill":{"duration":0.012224,"end_time":"2025-11-09T11:13:13.708505","exception":false,"start_time":"2025-11-09T11:13:13.696281","status":"completed"},"tags":[]}},{"id":"07550348","cell_type":"markdown","source":"We will construct a CNN using NumPy, building the the forward and backward propagation, as well as the training algorithm, all from scratch. Also we will provide math explanations in $\\LaTeX$ that will help understanding the model's construction. Then we will train our model to perform multiclass classification over the MNIST dataset.","metadata":{"papermill":{"duration":0.010578,"end_time":"2025-11-09T11:13:13.729791","exception":false,"start_time":"2025-11-09T11:13:13.719213","status":"completed"},"tags":[]}},{"id":"3bd92a03","cell_type":"markdown","source":"To refresh any Deep Learning concept used in this work, you can reference the Deep Learning section of the [Standford CS229 Notes](https://cs229.stanford.edu/main_notes.pdf). For convolutions in particular, the [Standford CS231n Notes](https://cs231n.github.io/convolutional-networks/) could help.","metadata":{}},{"id":"2f7ac453","cell_type":"markdown","source":"Our CNN will have the following structure:\n\\begin{array}{lcc}\n\\textbf{Layer} & \\textbf{Output Shape} & \\textbf{Parameters}\\\\ \\\\\n\\text{Input} & \\text{(1, 28, 28)} & \\text{0} \\\\\n\\text{Convolution 2D} & \\text{(4, 25, 25)} & \\text{68} \\\\\n\\text{Max Pooling} & \\text{(4, 12, 12)} & \\text{0} \\\\\n\\text{ReLu} & \\text{(4, 12, 12)} & \\text{0} \\\\\n\\text{Flattening} & \\text{(576,)} & \\text{0} \\\\\n\\text{Linear} & \\text{(10,)} & \\text{5770} \\\\\n\\text{Softmax} & \\text{(10,)} & \\text{0}\n\\end{array}","metadata":{"papermill":{"duration":0.01069,"end_time":"2025-11-09T11:13:13.751502","exception":false,"start_time":"2025-11-09T11:13:13.740812","status":"completed"},"tags":[]}},{"id":"414b0445","cell_type":"markdown","source":"For a consistent mathematical notation throughout the project, consider the following:\n1. An input MNIST image as $x \\in \\mathbb{R}^{1 \\times 28 \\times 28} \\:$. Grayscale, thus 1 channel. And $28 \\times 28$ pixels.\n2. Labels as one hot encoded vectors: $y \\in \\mathbb{R}^{10}$ such that if $x$ represents digit $d \\in \\set{0, \\dots,9}$, then $y_{d + 1} = 1$ and $y_j = 0, \\: \\forall j \\ne d + 1$\n3. Our network as a parametrized function $g_\\theta$ with $\\theta$ as the learnable parameters.\n4. The predictions for an example $x$ as $\\hat{y} := g_\\theta (x) = \\begin{bmatrix} p(y_1 = 1 | x; \\theta) \\\\ \\vdots \\\\ p(y_{10} = 1 | x; \\theta) \\end{bmatrix} \\in \\mathbb{R}^{10}$","metadata":{"papermill":{"duration":0.010556,"end_time":"2025-11-09T11:13:13.772471","exception":false,"start_time":"2025-11-09T11:13:13.761915","status":"completed"},"tags":[]}},{"id":"084bb6aa","cell_type":"markdown","source":"### 1.1 Setup and Configuration","metadata":{"papermill":{"duration":0.010399,"end_time":"2025-11-09T11:13:13.814352","exception":false,"start_time":"2025-11-09T11:13:13.803953","status":"completed"},"tags":[]}},{"id":"e5187b60","cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport math\nnp.random.seed(69)\n\nconfig = {\n 'conv_size': 4,\n 'conv_channels': 4,\n 'pool_size': 2,\n 'learning_rate': 0.1,\n 'batch_size': 32,\n 'num_batches': 260,\n 'dev_size': 500\n}","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:13.836854Z","iopub.status.busy":"2025-11-09T11:13:13.836462Z","iopub.status.idle":"2025-11-09T11:13:13.845447Z","shell.execute_reply":"2025-11-09T11:13:13.844470Z"},"papermill":{"duration":0.022119,"end_time":"2025-11-09T11:13:13.846832","exception":false,"start_time":"2025-11-09T11:13:13.824713","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"50d27b1b","cell_type":"markdown","source":"## 2. Forward Propagation","metadata":{"papermill":{"duration":0.01047,"end_time":"2025-11-09T11:13:13.868091","exception":false,"start_time":"2025-11-09T11:13:13.857621","status":"completed"},"tags":[]}},{"id":"580ec968","cell_type":"markdown","source":"Forward propagation is the process in which our neural network produces a prediction for a given example. Ie, computing $\\hat{y} = g_\\theta (x)$. This $g_\\theta$ can be seen as a composition of the $r$ layers of the network, also parametrized functions $f^{[1]}_{\\theta_1}, \\dots, f^{[r]}_{\\theta_r}$. Therefore, by defining each layers' function, we can compute $g_\\theta$ by composing them: $g_\\theta = f^{[1]}_{\\theta_1} \\circ \\dots \\circ f^{[r]}_{\\theta_r}$.","metadata":{"papermill":{"duration":0.01048,"end_time":"2025-11-09T11:13:13.888897","exception":false,"start_time":"2025-11-09T11:13:13.878417","status":"completed"},"tags":[]}},{"id":"fd20c773","cell_type":"markdown","source":"

1.1 Convolution Layer","metadata":{"papermill":{"duration":0.010364,"end_time":"2025-11-09T11:13:13.909461","exception":false,"start_time":"2025-11-09T11:13:13.899097","status":"completed"},"tags":[]}},{"id":"b50070fb","cell_type":"markdown","source":"The goal of this layer is to extract meaningful and representative features of an image (edges, shapes, textures, etc). The parameters of this layer consist of:\n\n1. Weights: arranged in kernels $K \\in \\mathbb{R}^{K_{w} \\times K_{h}} \\: (K_{w} \\le Width, \\: K_{h} \\le Height)$. We will consider $C_{out} \\times C_{in}$ kernels, one for each output and input channel ($C_{in} = 1$ for MNIST, but we will make it more general). This way, weights are stored in a 4D tensor: $W \\in \\mathbb{R}^{C_{out} \\times C_{in} \\times K_{w} \\times K_{h}}$.\n\n2. Biases: $b \\in \\mathbb{R}^{C_{out}}$.","metadata":{"papermill":{"duration":0.010323,"end_time":"2025-11-09T11:13:13.930117","exception":false,"start_time":"2025-11-09T11:13:13.919794","status":"completed"},"tags":[]}},{"id":"c12b48db","cell_type":"markdown","source":"This layer can be represented as $$Conv_{\\mathcal{W},b}: \\mathbb{R}^{C_{in} \\times Width \\times Height} \\to \\mathbb{R}^{C_{out} \\times (Width - K_{w} + 1) \\times (Height - K_{h} + 1)},$$ $$Conv_{k, i, j} (x)= \\sum_{c = 1}^{C_{in}} \\sum_{n = 1}^{K_{w}} \\sum_{m = 1}^{K_{h}} W_{k,c,n,m}\\cdot x_{c, \\ i+n-1, \\ j+m-1} + b_k \\in \\mathbb{R}$$","metadata":{"papermill":{"duration":0.010366,"end_time":"2025-11-09T11:13:13.950725","exception":false,"start_time":"2025-11-09T11:13:13.940359","status":"completed"},"tags":[]}},{"id":"3db65ad9","cell_type":"code","source":"def forward_convolution(conv_W, conv_b, data):\n \"\"\"\n Compute the output from a convolutional layer given the weights and data.\n\n conv_W is of the shape (# output channels, # input channels, convolution width, convolution height )\n conv_b is of the shape (# output channels)\n \n data is of the shape (# input channels, width, height)\n\n The output should be the result of a convolution and should be of the size:\n (# output channels, width - convolution width + 1, height - convolution height + 1)\n\n Returns:\n The output of the convolution as a numpy array\n \"\"\"\n\n conv_channels, _, conv_width, conv_height = conv_W.shape\n\n input_channels, input_width, input_height = data.shape\n\n output = np.zeros((conv_channels, input_width - conv_width + 1, input_height - conv_height + 1))\n\n for x in range(input_width - conv_width + 1):\n for y in range(input_height - conv_height + 1):\n for output_channel in range(conv_channels):\n output[output_channel, x, y] = np.sum(\n np.multiply(data[:, x:(x + conv_width), y:(y + conv_height)], conv_W[output_channel, :, :, :])) + conv_b[output_channel]\n\n return output","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:13.973176Z","iopub.status.busy":"2025-11-09T11:13:13.972604Z","iopub.status.idle":"2025-11-09T11:13:13.979575Z","shell.execute_reply":"2025-11-09T11:13:13.978688Z"},"papermill":{"duration":0.019904,"end_time":"2025-11-09T11:13:13.980925","exception":false,"start_time":"2025-11-09T11:13:13.961021","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"3ff7ec2d","cell_type":"markdown","source":"

1.2 Max Pooling Layer","metadata":{"papermill":{"duration":0.010532,"end_time":"2025-11-09T11:13:14.002020","exception":false,"start_time":"2025-11-09T11:13:13.991488","status":"completed"},"tags":[]}},{"id":"4a3b037a","cell_type":"markdown","source":"The goal of this layer is to reduce the dimension of the features extracted by the convolution layer, while emphasizing the most important ones. We will consider the stride $(S_w, S_h)$ size equal to the pooling size $(P_w, P_h)$.","metadata":{"papermill":{"duration":0.010738,"end_time":"2025-11-09T11:13:14.023940","exception":false,"start_time":"2025-11-09T11:13:14.013202","status":"completed"},"tags":[]}},{"id":"60b9fcc7","cell_type":"markdown","source":"This layer can be represented as: $$MaxPool: \\mathbb{R}^{C \\times W \\times H} \\to \\mathbb{R}^{C \\times \\lfloor \\frac{W}{P_w} \\rfloor \\times \\lfloor \\frac{H}{P_h} \\rfloor},$$ $$MaxPool_{c, i, j} (x) = \\max_{\\substack{0 \\le n < P_w \\\\ 0 \\le m < P_h }} x_{c, \\ i \\cdot S_w + n, \\ j \\cdot S_h+ m}$$","metadata":{"papermill":{"duration":0.01095,"end_time":"2025-11-09T11:13:14.045336","exception":false,"start_time":"2025-11-09T11:13:14.034386","status":"completed"},"tags":[]}},{"id":"e06daa81","cell_type":"code","source":"def forward_max_pool(data, pool_width, pool_height):\n \"\"\"\n Compute the output from a max pooling layer given the data and pool dimensions.\n\n The stride length should be equal to the pool size.\n\n data is of the shape (# channels, width, height)\n\n The output should be the result of the max pooling layer and should be of size:\n (# channels, width // pool_width, height // pool_height)\n\n Returns:\n The result of the max pooling layer\n \"\"\"\n input_channels, input_width, input_height = data.shape\n \n output = np.zeros((input_channels, input_width // pool_width, input_height // pool_height))\n\n for x in range(0, input_width - pool_width + 1, pool_width):\n for y in range(0, input_height - pool_height + 1, pool_height):\n\n output[:, x // pool_width, y // pool_height] = np.amax(data[:, x:(x + pool_width), y:(y + pool_height)], axis=(1, 2))\n\n return output","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.068215Z","iopub.status.busy":"2025-11-09T11:13:14.067850Z","iopub.status.idle":"2025-11-09T11:13:14.073817Z","shell.execute_reply":"2025-11-09T11:13:14.072932Z"},"papermill":{"duration":0.019499,"end_time":"2025-11-09T11:13:14.075315","exception":false,"start_time":"2025-11-09T11:13:14.055816","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"b8f7feb5","cell_type":"markdown","source":"

1.3 ReLu Layer","metadata":{"papermill":{"duration":0.010311,"end_time":"2025-11-09T11:13:14.096147","exception":false,"start_time":"2025-11-09T11:13:14.085836","status":"completed"},"tags":[]}},{"id":"32b73914","cell_type":"markdown","source":"The goal of this layer is to introduce non-linearities into the network, to ensure that more complex features can be represented. It may also help with training stability and gradient flow.","metadata":{"papermill":{"duration":0.01039,"end_time":"2025-11-09T11:13:14.116969","exception":false,"start_time":"2025-11-09T11:13:14.106579","status":"completed"},"tags":[]}},{"id":"c0fa6ad6","cell_type":"markdown","source":"This layer can be represented as: $$ReLu: \\mathbb{R}^{C \\times W \\times H} \\to \\mathbb{R}^{C \\times W \\times H}$$ $$ReLu(x)_{c, i, j} = \\max (0, x_{c, i, j})$$ ","metadata":{"papermill":{"duration":0.010446,"end_time":"2025-11-09T11:13:14.137717","exception":false,"start_time":"2025-11-09T11:13:14.127271","status":"completed"},"tags":[]}},{"id":"a7d54280","cell_type":"code","source":"def forward_relu(x):\n \"\"\"\n Compute the relu function for the input x.\n\n Args:\n x: A numpy float array\n\n Returns:\n A numpy float array containing the relu results\n \"\"\"\n x[x<=0] = 0\n\n return x","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.160419Z","iopub.status.busy":"2025-11-09T11:13:14.159745Z","iopub.status.idle":"2025-11-09T11:13:14.164276Z","shell.execute_reply":"2025-11-09T11:13:14.163484Z"},"papermill":{"duration":0.017586,"end_time":"2025-11-09T11:13:14.165643","exception":false,"start_time":"2025-11-09T11:13:14.148057","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"87c2dc89","cell_type":"markdown","source":"

1.4 Flattening Layer","metadata":{"papermill":{"duration":0.010339,"end_time":"2025-11-09T11:13:14.186680","exception":false,"start_time":"2025-11-09T11:13:14.176341","status":"completed"},"tags":[]}},{"id":"572fc85b","cell_type":"markdown","source":"This layer will reshape the data into a 1D vector. Implementation is so simple we won't do it now explicitly.","metadata":{"papermill":{"duration":0.010229,"end_time":"2025-11-09T11:13:14.207289","exception":false,"start_time":"2025-11-09T11:13:14.197060","status":"completed"},"tags":[]}},{"id":"3b96ea38","cell_type":"markdown","source":"

1.5 Linear Layer","metadata":{"papermill":{"duration":0.01031,"end_time":"2025-11-09T11:13:14.227896","exception":false,"start_time":"2025-11-09T11:13:14.217586","status":"completed"},"tags":[]}},{"id":"540a9f05","cell_type":"markdown","source":"The goal of this layer is to capture patterns and make sophisticated features to rely on, by lienally combining the features obatained by the convolution and pooling. The parameters of this layer consist of:\n1. Weights: $W \\in \\mathbb{R}^{M \\times N}$\n2. Biases: $b \\in \\mathbb{R}^N$","metadata":{"papermill":{"duration":0.010259,"end_time":"2025-11-09T11:13:14.249156","exception":false,"start_time":"2025-11-09T11:13:14.238897","status":"completed"},"tags":[]}},{"id":"c6a92249","cell_type":"markdown","source":"The layer can be represented as: $$L_{W, b}: \\mathbb{R}^M \\to \\mathbb{R}^N,$$ $$L_{W, b} (x) = x^T W + b$$","metadata":{"papermill":{"duration":0.010522,"end_time":"2025-11-09T11:13:14.269934","exception":false,"start_time":"2025-11-09T11:13:14.259412","status":"completed"},"tags":[]}},{"id":"dfa86655","cell_type":"code","source":"def forward_linear(weights, bias, data):\n \"\"\"\n Compute the output from a linear layer with the given weights, bias and data.\n weights is of the shape (input # features, output # features)\n bias is of the shape (output # features)\n data is of the shape (input # features)\n\n The output should be of the shape (output # features)\n\n Returns:\n The result of the linear layer\n \"\"\"\n return data.dot(weights) + bias","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.292540Z","iopub.status.busy":"2025-11-09T11:13:14.291786Z","iopub.status.idle":"2025-11-09T11:13:14.296341Z","shell.execute_reply":"2025-11-09T11:13:14.295458Z"},"papermill":{"duration":0.017564,"end_time":"2025-11-09T11:13:14.297763","exception":false,"start_time":"2025-11-09T11:13:14.280199","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"28f3a75c","cell_type":"markdown","source":"

1.6 Softmax Layer","metadata":{"papermill":{"duration":0.01046,"end_time":"2025-11-09T11:13:14.318752","exception":false,"start_time":"2025-11-09T11:13:14.308292","status":"completed"},"tags":[]}},{"id":"7b013471","cell_type":"markdown","source":"The goal of this layer is to output our final predictions for the network. For multiclass classification we choose the 'softmax' function as our final layer, since it outputs probaility shaped quantities* while mantaining good statistical properties and interpretability.","metadata":{"papermill":{"duration":0.010627,"end_time":"2025-11-09T11:13:14.339715","exception":false,"start_time":"2025-11-09T11:13:14.329088","status":"completed"},"tags":[]}},{"id":"74f30b06","cell_type":"markdown","source":"This function can be represented as:$$softmax : \\mathbb{R}^N \\to \\mathbb{R}^N$$ $$softmax(x_1, \\dots, x_N)_i = \\frac{e^{x_i}}{\\sum_{j = 1}^N e^{x_j}}, \\: i = 1, \\dots, N$$\n*Notice how: $softmax(x_1 , \\dots , x_{N}) > 0 , \\: \\sum_{i = 1}^{N} softmax(x_1, \\dots , x_{N})_i = 1$.","metadata":{"papermill":{"duration":0.011558,"end_time":"2025-11-09T11:13:14.361643","exception":false,"start_time":"2025-11-09T11:13:14.350085","status":"completed"},"tags":[]}},{"id":"b1a8a9f0","cell_type":"code","source":"def forward_softmax(x):\n \"\"\"\n Compute softmax function for a single example.\n The shape of the input is of size # num classes.\n x: A 1d numpy float array of shape number_of_classes\n\n Returns:\n A 1d numpy float array containing the softmax results of shape number_of_classes\n \"\"\"\n\n x = x - np.max(x,axis=0)\n exp = np.exp(x)\n s = exp / np.sum(exp,axis=0)\n return s","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.384598Z","iopub.status.busy":"2025-11-09T11:13:14.384330Z","iopub.status.idle":"2025-11-09T11:13:14.389036Z","shell.execute_reply":"2025-11-09T11:13:14.388253Z"},"papermill":{"duration":0.017843,"end_time":"2025-11-09T11:13:14.390309","exception":false,"start_time":"2025-11-09T11:13:14.372466","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"0b3845ae","cell_type":"markdown","source":"

1.7 Loss Function","metadata":{"papermill":{"duration":0.01046,"end_time":"2025-11-09T11:13:14.411468","exception":false,"start_time":"2025-11-09T11:13:14.401008","status":"completed"},"tags":[]}},{"id":"8c480c47","cell_type":"markdown","source":"The goal of a loss funtion is to evaluate our model's performance by comparing a predicted label $\\hat{y}$ against a true known one $y$. For multiclass classification we choose the 'cross-entropy' loss function, since its minimization is equivalent to the maximization of the training data likelihood.","metadata":{"papermill":{"duration":0.010592,"end_time":"2025-11-09T11:13:14.432831","exception":false,"start_time":"2025-11-09T11:13:14.422239","status":"completed"},"tags":[]}},{"id":"25eab567","cell_type":"markdown","source":"It can be represented as: $$l_{ce}: \\mathbb{R}^N \\times \\mathbb{R}^N \\to \\mathbb{R},$$ $$l_{ce} (\\hat{y}, y) = -\\sum_{i = 1}^{N} y_i \\log(\\hat{y_i})$$","metadata":{"papermill":{"duration":0.01066,"end_time":"2025-11-09T11:13:14.453990","exception":false,"start_time":"2025-11-09T11:13:14.443330","status":"completed"},"tags":[]}},{"id":"886f2081","cell_type":"code","source":"def forward_cross_entropy_loss(probabilities, labels):\n \"\"\"\n Compute the output from a cross entropy loss layer given the probabilities and labels.\n\n probabilities is of the shape (# classes)\n labels is of the shape (# classes)\n\n The output should be a scalar\n\n Returns:\n The result of the cross-entropy loss layer\n \"\"\"\n\n result = 0\n\n for i, label in enumerate(labels):\n if label == 1:\n result += -np.log(probabilities[i])\n\n return result","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.476274Z","iopub.status.busy":"2025-11-09T11:13:14.476007Z","iopub.status.idle":"2025-11-09T11:13:14.480630Z","shell.execute_reply":"2025-11-09T11:13:14.479983Z"},"papermill":{"duration":0.017793,"end_time":"2025-11-09T11:13:14.482082","exception":false,"start_time":"2025-11-09T11:13:14.464289","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"ce667831","cell_type":"markdown","source":"

1.8 Full Forward Propagation","metadata":{"papermill":{"duration":0.010451,"end_time":"2025-11-09T11:13:14.503140","exception":false,"start_time":"2025-11-09T11:13:14.492689","status":"completed"},"tags":[]}},{"id":"e9838c4c","cell_type":"markdown","source":"Finally, we can perform the complete forward propagation by composing each of our layers outputs.\nThe code implementation will also have the posibility of computing the loss $l_{ce} (\\hat{y} , y)$ if a label $y$ is given.","metadata":{"papermill":{"duration":0.010358,"end_time":"2025-11-09T11:13:14.523829","exception":false,"start_time":"2025-11-09T11:13:14.513471","status":"completed"},"tags":[]}},{"id":"b590f1ec","cell_type":"code","source":"def forward_prop(data, labels, params):\n \"\"\"\n Implement the forward layer given the data, labels, and params.\n \n Args:\n data: A numpy array containing the input (shape is 1 by 28 by 28)\n labels: A 1d numpy array containing the labels (shape is 10)\n params: A dictionary mapping parameter names to numpy arrays with the parameters.\n This numpy array will contain W1, b1, W2 and b2\n W1 and b1 represent the weights and bias for the hidden layer of the network\n W2 and b2 represent the weights and bias for the output layer of the network\n\n Returns:\n A 2 element tuple containing:\n 1. A numpy array The output (after the softmax) of the output layer\n 2. The average loss for these data elements\n \"\"\"\n\n W1 = params['W1']\n b1 = params['b1']\n W2 = params['W2']\n b2 = params['b2']\n pool_size = config['pool_size']\n \n first_convolution = forward_convolution(W1, b1, data)\n first_max_pool = forward_max_pool(first_convolution, pool_size, pool_size)\n first_after_relu = forward_relu(first_max_pool)\n flattened = np.reshape(first_after_relu, (-1))\n logits = forward_linear(W2, b2, flattened)\n y = forward_softmax(logits)\n loss = forward_cross_entropy_loss(y, labels)\n\n return y, loss","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.546804Z","iopub.status.busy":"2025-11-09T11:13:14.546039Z","iopub.status.idle":"2025-11-09T11:13:14.552471Z","shell.execute_reply":"2025-11-09T11:13:14.551683Z"},"papermill":{"duration":0.019676,"end_time":"2025-11-09T11:13:14.553833","exception":false,"start_time":"2025-11-09T11:13:14.534157","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"89cff86e","cell_type":"markdown","source":"

1.9 Making Predictions","metadata":{"papermill":{"duration":0.010744,"end_time":"2025-11-09T11:13:14.575286","exception":false,"start_time":"2025-11-09T11:13:14.564542","status":"completed"},"tags":[]}},{"id":"d445abab","cell_type":"markdown","source":"To make a prediction on a set of unlabeled examples, we just need to apply our forward propagation function on each of them, and select the maximum probability class candidate.","metadata":{"papermill":{"duration":0.010635,"end_time":"2025-11-09T11:13:14.596366","exception":false,"start_time":"2025-11-09T11:13:14.585731","status":"completed"},"tags":[]}},{"id":"54cef265","cell_type":"code","source":"def model_predict(data, params, forward_prop_function):\n \"\"\"\n Make predictions over a list of examples.\n \n Args:\n examples: A numpy array containing data examples.\n params: A dictionary mapping parameter names to numpy arrays with the parameters.\n forward_prop_functions: A function for forward propagation of the model\n\n Returns:\n A list containing the predicted class for each example\n \"\"\"\n predictions = []\n \n print(\"Making predictions...\")\n \n for i in range(data.shape[0]):\n y, _ = forward_prop_function(data[i], [], params)\n final_prediction = np.argmax(y)\n predictions.append(final_prediction)\n\n print(\"Predictions completed\")\n \n return predictions","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.619721Z","iopub.status.busy":"2025-11-09T11:13:14.618656Z","iopub.status.idle":"2025-11-09T11:13:14.624464Z","shell.execute_reply":"2025-11-09T11:13:14.623714Z"},"papermill":{"duration":0.018884,"end_time":"2025-11-09T11:13:14.625794","exception":false,"start_time":"2025-11-09T11:13:14.606910","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"62703c09","cell_type":"markdown","source":"## 3. Backward Propagation","metadata":{"papermill":{"duration":0.010614,"end_time":"2025-11-09T11:13:14.646842","exception":false,"start_time":"2025-11-09T11:13:14.636228","status":"completed"},"tags":[]}},{"id":"0c0c4e73","cell_type":"markdown","source":"

3.0 Backpropagation Strategy","metadata":{"papermill":{"duration":0.010391,"end_time":"2025-11-09T11:13:14.667756","exception":false,"start_time":"2025-11-09T11:13:14.657365","status":"completed"},"tags":[]}},{"id":"3865c8cc","cell_type":"markdown","source":"For training the neural network using gradient descent, we need to compute the gradient of the loss function: $J := l_{ce}(g_\\theta (x), y)$ with respect to $\\theta$. We will use the backpropagation strategy for this purpose.","metadata":{"papermill":{"duration":0.010534,"end_time":"2025-11-09T11:13:14.689260","exception":false,"start_time":"2025-11-09T11:13:14.678726","status":"completed"},"tags":[]}},{"id":"58717ebd","cell_type":"markdown","source":"Let $u^{[k]}$ be the output of the $k$-th layer, $k = 0, 1, \\dots, 7$. This means:$$u^{[0]} = x, \\: u^{[1]} = Conv_{\\mathcal{W}, b} (x), \\: \\dots\\:, \\: u^{[6]} = g_\\theta(x), \\: u^{[7]} = J$$\n\nSimillarly, let $\\theta ^{[k]}$ be the paramaters of the $k$-th layer (could be none), $k = 0, \\dots, 7$.

\n\nNotice that $\\forall k \\in \\set{1, \\dots, 7}$, on the forward propagation section we defined each layer's output $u^{[k]}$ as a function of its input $u^{[k - 1]}$ and parameters $\\theta^{[k]}$, so for every layer we will be able to calculate $\\frac{\\partial u^{[k]}}{\\partial \\theta ^{[k]}}$ and $\\frac{\\partial u^{[k]}}{\\partial u^{[k - 1]}}$. $(3)$

Derivatives from $(3)$ might depend on the $u^{[k-1]}$ value, so for the backpropagation strategy we will compute every $u^{[k]}$ in advance, in a \"forward pass\".","metadata":{"papermill":{"duration":0.010724,"end_time":"2025-11-09T11:13:14.710348","exception":false,"start_time":"2025-11-09T11:13:14.699624","status":"completed"},"tags":[]}},{"id":"12de2881","cell_type":"markdown","source":"With this notation, applying the chain rule $\\forall k \\in \\set{1, \\dots, 7}$, gives us: $$\\frac{\\partial J}{\\partial u^{[k - 1]}_i} = \\sum_j \\frac{\\partial J}{\\partial u^{[k]}_j} \\cdot \\frac{\\partial u^{[k]} _j}{\\partial u^{[k -1]}_i} \\text{ (1) }$$ $$\\frac{\\partial J}{\\partial \\theta ^{[k]}_i} = \\sum_j \\frac{\\partial J}{\\partial u^{[k]}_j} \\cdot \\frac{\\partial u^{[k]} _j}{\\partial \\theta ^{[k]}_i} \\text{ (2) }$$\n\nNow, consider we know $\\frac{\\partial J}{\\partial u^{[k]}}$ for a fixed $k$. Since we can always calculate $(3)$, equation $(2)$ enables us to compute $\\frac{\\partial J}{\\partial \\theta ^{[k]}_i}$. However, equation (1) enables us to compute $\\frac{\\partial J}{\\partial u^{[k - 1]}_i}$, so that we can inductively repeat this computation for $k-1$. Since at $k=7$ $\\frac{\\partial J}{\\partial u^{[k]}} = \\frac{\\partial J}{\\partial J}$ is known, by induction we are able to compute $\\frac{\\partial J}{\\partial \\theta ^{[k]}_i} \\: \\forall k \\in \\set{0, \\dots\\ 7}$. That's what we wanted. ","metadata":{"papermill":{"duration":0.010542,"end_time":"2025-11-09T11:13:14.731317","exception":false,"start_time":"2025-11-09T11:13:14.720775","status":"completed"},"tags":[]}},{"id":"994ef145","cell_type":"markdown","source":"For each layer $k$ we can define the step of the induction as a function that take as inputs $\\frac{\\partial J}{\\partial u^{[k]}}$ (induction hypothesis) and $u^{[k-1]}$ (for computing $(3)$ when it becomes necessary), and outputs $\\frac{\\partial J}{\\partial \\theta ^{[k]}}$ (the gradient we wanted) and $\\frac{\\partial J}{\\partial u^{[k-1]}}$ (induction step) relying on formulas $(1), (2)$ for the computation.\n\n$$\\mathcal{B_k}: (u^{[k-1]}, \\frac{\\partial J}{\\partial u^{[k]}}) \\mapsto (\\frac{\\partial J}{\\partial \\theta ^{[k]}}, \\frac{\\partial J}{\\partial u^{[k - 1]}})$$","metadata":{"papermill":{"duration":0.010924,"end_time":"2025-11-09T11:13:14.753470","exception":false,"start_time":"2025-11-09T11:13:14.742546","status":"completed"},"tags":[]}},{"id":"aa3d4192","cell_type":"markdown","source":"

3.1 Backward Convolution","metadata":{"papermill":{"duration":0.01063,"end_time":"2025-11-09T11:13:14.774796","exception":false,"start_time":"2025-11-09T11:13:14.764166","status":"completed"},"tags":[]}},{"id":"3786064a","cell_type":"markdown","source":"From the convolution layer, $$u^{[k]}_{p, i, j} = \\sum_{c = 1}^{C_{in}} \\sum_{n = 1}^{K_{w}} \\sum_{m = 1}^{K_{h}} W_{p,c,n,m}\\cdot u^{[k-1]}_{c, \\ i+n-1, \\ j+m-1} + b_p \\in \\mathbb{R}$$\nWe get the following outputs for the backward function:\n$$\\frac{\\partial J}{\\partial W_{p,c,n,m}} = \\sum_{i}\\sum_{j}\\frac{\\partial J}{\\partial u^{[k]}_{p,i,j}}\\; u^{[k-1]}_{c,\\,i+n-1,\\,j+m-1}$$ $$\\frac{\\partial J}{\\partial b_{p}} = \\sum_{i}\\sum_{j} \\frac{\\partial J}{\\partial u^{[k]}_{p,i,j}}$$ $$\\frac{\\partial J}{\\partial u^{[k-1]}_{c,i,j}} = \\sum_{p=1}^{C_{out}}\\sum_{n=1}^{K_w}\\sum_{m=1}^{K_h} W_{p,c,n,m}\\; \\frac{\\partial J}{\\partial u^{[k]}_{p,\\,i-n+1,\\,j-m+1}}$$ ","metadata":{"papermill":{"duration":0.010807,"end_time":"2025-11-09T11:13:14.796474","exception":false,"start_time":"2025-11-09T11:13:14.785667","status":"completed"},"tags":[]}},{"id":"588b8e33","cell_type":"code","source":"def backward_convolution(conv_W, conv_b, data, output_grad):\n \"\"\"\n Compute the gradient of the loss with respect to the parameters of the convolution.\n\n See forward_convolution for the sizes of the arguments.\n output_grad is the gradient of the loss with respect to the output of the convolution.\n\n Returns:\n A tuple containing 3 gradients.\n The first element is the gradient of the loss with respect to the convolution weights\n The second element is the gradient of the loss with respect to the convolution bias\n The third element is the gradient of the loss with respect to the input data\n \"\"\"\n\n out_channels, input_channels, conv_width, conv_height = conv_W.shape\n _, input_width, input_height = data.shape\n\n # Bias\n dJdb = np.sum(output_grad, axis=(1, 2))\n\n dJdu = np.zeros(data.shape)\n dJdW = np.zeros(conv_W.shape)\n for i in range(input_width - conv_width + 1):\n for j in range(input_height - conv_height + 1):\n for c in range(out_channels):\n\n # Weights\n dJdW[c] += output_grad[c, i, j] * data[:, i:(i+conv_width), j:(j+conv_height)]\n\n # Data\n dJdu[:, i:(i+conv_width), j:(j+conv_height)] += output_grad[c, i, j] * conv_W[c, :, :, :]\n\n return dJdW, dJdb, dJdu","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.819157Z","iopub.status.busy":"2025-11-09T11:13:14.818819Z","iopub.status.idle":"2025-11-09T11:13:14.825806Z","shell.execute_reply":"2025-11-09T11:13:14.824848Z"},"papermill":{"duration":0.020311,"end_time":"2025-11-09T11:13:14.827296","exception":false,"start_time":"2025-11-09T11:13:14.806985","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"7eff4c21","cell_type":"markdown","source":"

3.2 Backward Max Pool","metadata":{"papermill":{"duration":0.010848,"end_time":"2025-11-09T11:13:14.848968","exception":false,"start_time":"2025-11-09T11:13:14.838120","status":"completed"},"tags":[]}},{"id":"fcb078af","cell_type":"markdown","source":"From the max pooling layer, $$u^{[k]}_{c, i, j} = \\max_{\\substack{0 \\le n < P_w \\\\ 0 \\le m < P_h }} u^{[k-1]}_{c, \\ i \\cdot S_w + n, \\ j \\cdot S_h+ m}$$\nWe get the following outputs for the backward function: $$\\frac{\\partial J}{\\partial u^{[k-1]}_{c,\\,i \\cdot S_w + n,\\, j \\cdot S_h + m}} = \\begin{cases} \\frac{\\partial J}{\\partial u^{[k]}_{c,i,j}}, & \\text{if } (n,m) = (n^*,m^*), \\\\ 0, & \\text{otherwise.} \\end{cases}$$\n","metadata":{"papermill":{"duration":0.010967,"end_time":"2025-11-09T11:13:14.870483","exception":false,"start_time":"2025-11-09T11:13:14.859516","status":"completed"},"tags":[]}},{"id":"9afa9a6a","cell_type":"code","source":"def backward_max_pool(data, pool_width, pool_height, output_grad):\n \"\"\"\n Compute the gradient of the loss with respect to the data in the max pooling layer.\n\n data is of the shape (# channels, width, height)\n output_grad is of shape (# channels, width // pool_width, height // pool_height)\n\n output_grad is the gradient of the loss with respect to the output of the backward max\n pool layer.\n\n Returns:\n The gradient of the loss with respect to the data (of same shape as data)\n \"\"\"\n \n channels, out_width, out_height = output_grad.shape\n dJdu = np.zeros(data.shape)\n\n # For each pooling block\n for block_i in range(out_width):\n for block_j in range(out_height):\n for c in range(channels):\n\n # Get data indices of the corresponding pooling block position\n i, j = pool_width * block_i, pool_height * block_j \n\n # Find indexes of max element of data in current block (relative indexes to the block + block position)\n rel_k, rel_l = np.unravel_index(np.argmax(data[c, i:(i+pool_width), j:(j+pool_height)]), (pool_width, pool_height))\n k, l = rel_k + i, rel_l + j\n # Derivative is 1 for the max indeces, 0 else, so we simplify the multiplication as:\n dJdu[c, k, l] = output_grad[c, block_i, block_j]\n \n return dJdu","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.893775Z","iopub.status.busy":"2025-11-09T11:13:14.892978Z","iopub.status.idle":"2025-11-09T11:13:14.899594Z","shell.execute_reply":"2025-11-09T11:13:14.898661Z"},"papermill":{"duration":0.019957,"end_time":"2025-11-09T11:13:14.901148","exception":false,"start_time":"2025-11-09T11:13:14.881191","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"c386378c","cell_type":"markdown","source":"

3.3 Backward ReLu","metadata":{"papermill":{"duration":0.010568,"end_time":"2025-11-09T11:13:14.922525","exception":false,"start_time":"2025-11-09T11:13:14.911957","status":"completed"},"tags":[]}},{"id":"dc7e3605","cell_type":"markdown","source":"From the ReLu layer, $$u^{[k]}_{c, i, j} = max(0, u^{[k-1]}_{c, i, j})$$ \n\nWe get the following outputs for the backward function: $$\\frac{\\partial J}{\\partial u^{[k-1]}_{c,i,j}} = \\frac{\\partial J}{\\partial u^{[k]}_{c,i,j}} \\cdot \\mathbf{1}\\!\\left(u^{[k-1]}_{c,i,j} > 0\\right)$$","metadata":{"papermill":{"duration":0.010451,"end_time":"2025-11-09T11:13:14.943591","exception":false,"start_time":"2025-11-09T11:13:14.933140","status":"completed"},"tags":[]}},{"id":"c5ac777e","cell_type":"code","source":"def backward_relu(x, grad_outputs):\n \"\"\"\n Compute the gradient of the loss with respect to x\n\n Args:\n x: A numpy array of arbitrary shape containing the input.\n grad_outputs: A numpy array of the same shape of x containing the gradient of the loss with respect\n to the output of relu\n\n Returns:\n A numpy array of the same shape as x containing the gradients with respect to x.\n \"\"\"\n\n grad_outputs[x<=0] = 0\n return grad_outputs","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:14.966605Z","iopub.status.busy":"2025-11-09T11:13:14.965764Z","iopub.status.idle":"2025-11-09T11:13:14.970169Z","shell.execute_reply":"2025-11-09T11:13:14.969577Z"},"papermill":{"duration":0.017495,"end_time":"2025-11-09T11:13:14.971519","exception":false,"start_time":"2025-11-09T11:13:14.954024","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"bc9417a6","cell_type":"markdown","source":"

3.4 Backward Flattening","metadata":{"papermill":{"duration":0.010579,"end_time":"2025-11-09T11:13:14.992690","exception":false,"start_time":"2025-11-09T11:13:14.982111","status":"completed"},"tags":[]}},{"id":"76fdc289","cell_type":"markdown","source":"Again, this layer is simple enough to avoid explicit implementation at the moment.","metadata":{"papermill":{"duration":0.010802,"end_time":"2025-11-09T11:13:15.014028","exception":false,"start_time":"2025-11-09T11:13:15.003226","status":"completed"},"tags":[]}},{"id":"aeeafbe6","cell_type":"markdown","source":"

3.5 Backward Linear","metadata":{"papermill":{"duration":0.010528,"end_time":"2025-11-09T11:13:15.034968","exception":false,"start_time":"2025-11-09T11:13:15.024440","status":"completed"},"tags":[]}},{"id":"02a4a26f","cell_type":"markdown","source":"From the linear layer, $$u^{[k]} = u^{[k-1] \\, T} W + b$$ We get the following outputs for the backward function: $$\\frac{\\partial J}{\\partial W} = u^{[k-1]} \\otimes \\frac{\\partial J}{\\partial u^{[k]}}$$ $$\\frac{\\partial J}{\\partial b} = \\frac{\\partial J}{\\partial u^{[k]}}$$ $$\\frac{\\partial J}{\\partial u^{[k-1]}} = W \\frac{\\partial J}{\\partial u^{[k]}}$$","metadata":{"papermill":{"duration":0.010271,"end_time":"2025-11-09T11:13:15.055747","exception":false,"start_time":"2025-11-09T11:13:15.045476","status":"completed"},"tags":[]}},{"id":"b13196c7","cell_type":"code","source":"def backward_linear(weights, bias, data, output_grad):\n \"\"\"\n Compute the gradients of the loss with respect to the parameters of a linear layer.\n\n See forward_linear for information about the shapes of the variables.\n\n output_grad is the gradient of the loss with respect to the output of this layer.\n\n This should return a tuple with three elements:\n - The gradient of the loss with respect to the weights\n - The gradient of the loss with respect to the bias\n - The gradient of the loss with respect to the data\n \"\"\"\n\n # Weights\n dJdW = np.outer(data, output_grad)\n # Bias\n dJdb = output_grad\n\n # Data\n dLdu = weights\n dJdu = dLdu.dot(output_grad)\n\n return dJdW, dJdb, dJdu","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.078351Z","iopub.status.busy":"2025-11-09T11:13:15.077654Z","iopub.status.idle":"2025-11-09T11:13:15.082635Z","shell.execute_reply":"2025-11-09T11:13:15.081822Z"},"papermill":{"duration":0.018004,"end_time":"2025-11-09T11:13:15.084074","exception":false,"start_time":"2025-11-09T11:13:15.066070","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"4bd9b5fc","cell_type":"markdown","source":"

3.6 Backward Cross Entropy","metadata":{"papermill":{"duration":0.0106,"end_time":"2025-11-09T11:13:15.105307","exception":false,"start_time":"2025-11-09T11:13:15.094707","status":"completed"},"tags":[]}},{"id":"9d9f023d","cell_type":"markdown","source":"Understanding the cross-entropy backpropagation first, will be helful for computing the softmax one afterwards.","metadata":{"papermill":{"duration":0.010538,"end_time":"2025-11-09T11:13:15.126297","exception":false,"start_time":"2025-11-09T11:13:15.115759","status":"completed"},"tags":[]}},{"id":"4f8b317a","cell_type":"markdown","source":"From the cross-entropy loss, $$J = u^{[k]} = -\\sum_{i = 1}^{N} y_i \\log(u^{[k - 1]}_i) = -\\sum_{i = 1}^{N} y_i \\log(\\hat{y}_i)$$\n\nWe get the following outputs for the backward function: $$\\frac{\\partial J}{\\partial u^{[k-1]}_i} = \\frac{\\partial J}{\\partial \\hat{y}_i} = -\\frac{y_i}{\\hat{y}_i}$$\n","metadata":{"papermill":{"duration":0.010622,"end_time":"2025-11-09T11:13:15.147587","exception":false,"start_time":"2025-11-09T11:13:15.136965","status":"completed"},"tags":[]}},{"id":"fffc6305","cell_type":"code","source":"def backward_cross_entropy_loss(probabilities, labels):\n \"\"\"\n Compute the gradient of the cross entropy loss with respect to the probabilities.\n\n probabilities is of the shape (# classes)\n labels is of the shape (# classes)\n\n The output should be the gradient with respect to the probabilities.\n\n Returns:\n The gradient of the loss with respect to the probabilities.\n \"\"\"\n\n return -labels / probabilities","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.171658Z","iopub.status.busy":"2025-11-09T11:13:15.171341Z","iopub.status.idle":"2025-11-09T11:13:15.175557Z","shell.execute_reply":"2025-11-09T11:13:15.174935Z"},"papermill":{"duration":0.019042,"end_time":"2025-11-09T11:13:15.176959","exception":false,"start_time":"2025-11-09T11:13:15.157917","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"cffa485a","cell_type":"markdown","source":"

3.7 Backward Softmax","metadata":{"papermill":{"duration":0.010469,"end_time":"2025-11-09T11:13:15.198053","exception":false,"start_time":"2025-11-09T11:13:15.187584","status":"completed"},"tags":[]}},{"id":"4cd449f9","cell_type":"markdown","source":"From the softmax layer, $$u^{[k]}_i= \\frac{e^{u^{[k-1]}_i}}{\\sum_{j = 1}^N e^{u^{[k-1]}_j}}, \\: i = 1, \\dots, N$$\n\nWe get the following outputs for the backward function: $$\\frac{\\partial J}{\\partial u^{[k-1]}} = (\\text{diag}(u^{[k]}) - u^{[k]} (u^{[k]})^T) \\cdot \\frac{\\partial J}{\\partial u^{[k]}}$$","metadata":{"papermill":{"duration":0.01044,"end_time":"2025-11-09T11:13:15.218968","exception":false,"start_time":"2025-11-09T11:13:15.208528","status":"completed"},"tags":[]}},{"id":"e863dd9b","cell_type":"markdown","source":"This implementation can be numerically unstable because of the possible subtraction of close numbers. However, we know $\\frac{\\partial J}{\\partial u^{[k]}_i} = \\frac{y_i}{\\hat{y}_i}$ from the backward cross-entropy, and the labels are one-hot-encoded. This allows for a simplifiction of the formula that avoids overflow: $$\\frac{\\partial J}{\\partial u^{[k-1]}} = u^{[k]} - y$$\n","metadata":{"papermill":{"duration":0.010398,"end_time":"2025-11-09T11:13:15.239832","exception":false,"start_time":"2025-11-09T11:13:15.229434","status":"completed"},"tags":[]}},{"id":"e574a2ce","cell_type":"code","source":"def backward_softmax(x, grad_outputs):\n \"\"\"\n Compute the gradient of the loss with respect to x.\n grad_outputs is the gradient of the loss with respect to the outputs of the softmax.\n\n Args:\n x: A 1d numpy float array of shape number_of_classes\n grad_outputs: A 1d numpy float array of shape number_of_classes\n\n Returns:\n A 1d numpy float array of the same shape as x with the derivative of the loss with respect to x\n \"\"\"\n \n return forward_softmax(x) - (grad_outputs!=0).astype(int)","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.261927Z","iopub.status.busy":"2025-11-09T11:13:15.261596Z","iopub.status.idle":"2025-11-09T11:13:15.266229Z","shell.execute_reply":"2025-11-09T11:13:15.265305Z"},"papermill":{"duration":0.01745,"end_time":"2025-11-09T11:13:15.267649","exception":false,"start_time":"2025-11-09T11:13:15.250199","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"b9d52f94","cell_type":"markdown","source":"

3.8 Full Backpropagation","metadata":{"papermill":{"duration":0.010636,"end_time":"2025-11-09T11:13:15.289182","exception":false,"start_time":"2025-11-09T11:13:15.278546","status":"completed"},"tags":[]}},{"id":"0430e194","cell_type":"markdown","source":"Finally, we can compute the full backpropagation. First we will compute every $u^{[k]}$ in a forward pass, then we will apply every backward function in a descending order to compute every gradient $\\frac{\\partial J}{\\partial \\theta^{[k]}}$.","metadata":{"papermill":{"duration":0.010666,"end_time":"2025-11-09T11:13:15.310294","exception":false,"start_time":"2025-11-09T11:13:15.299628","status":"completed"},"tags":[]}},{"id":"31a76ff8","cell_type":"code","source":"def backward_prop(data, labels, params):\n \"\"\"\n Implement the backward propagation gradient computation step for a neural network\n \n Args:\n data: A numpy array containing the input for a single example\n labels: A 1d numpy array containing the labels for a single example\n params: A dictionary mapping parameter names to numpy arrays with the parameters.\n This numpy array will contain W1, b1, W2, and b2\n W1 and b1 represent the weights and bias for the convolutional layer\n W2 and b2 represent the weights and bias for the output layer of the network\n\n Returns:\n A dictionary of strings to numpy arrays where each key represents the name of a weight\n and the values represent the gradient of the loss with respect to that weight.\n \n In particular, it should have 4 elements:\n W1, W2, b1, and b2\n \"\"\"\n\n W1 = params['W1']\n b1 = params['b1']\n W2 = params['W2']\n b2 = params['b2']\n pool_size = config['pool_size']\n\n # Forward pass\n first_convolution = forward_convolution(W1, b1, data)\n first_max_pool = forward_max_pool(first_convolution, pool_size, pool_size)\n first_after_relu = forward_relu(first_max_pool)\n flattened = np.reshape(first_after_relu, (-1))\n logits = forward_linear(W2, b2, flattened)\n y = forward_softmax(logits)\n\n # Backward pass\n grad = backward_cross_entropy_loss(y, labels)\n grad = backward_softmax(logits, grad)\n gradW2, gradb2, grad = backward_linear(W2, b2, flattened, grad)\n grad = backward_relu(first_max_pool, grad.reshape(first_max_pool.shape))\n grad = backward_max_pool(first_convolution, pool_size, pool_size, grad)\n gradW1, gradb1, grad = backward_convolution(W1, b1, data, grad)\n\n grads = {}\n grads['W1'] = gradW1\n grads['b1'] = gradb1\n grads['W2'] = gradW2\n grads['b2'] = gradb2\n \n return grads","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.333549Z","iopub.status.busy":"2025-11-09T11:13:15.332740Z","iopub.status.idle":"2025-11-09T11:13:15.341731Z","shell.execute_reply":"2025-11-09T11:13:15.340815Z"},"papermill":{"duration":0.0224,"end_time":"2025-11-09T11:13:15.343251","exception":false,"start_time":"2025-11-09T11:13:15.320851","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"665b8617","cell_type":"markdown","source":"## 4 Model Training","metadata":{"papermill":{"duration":0.011819,"end_time":"2025-11-09T11:13:15.365798","exception":false,"start_time":"2025-11-09T11:13:15.353979","status":"completed"},"tags":[]}},{"id":"6a933976","cell_type":"markdown","source":"We will now construct our model's training algorithm. But first we will define some simple auxiliar functions, for computing the accuracy of predictions and for initializing the model's parameters.","metadata":{"papermill":{"duration":0.011149,"end_time":"2025-11-09T11:13:15.387588","exception":false,"start_time":"2025-11-09T11:13:15.376439","status":"completed"},"tags":[]}},{"id":"d49ddf2d","cell_type":"code","source":"def compute_accuracy(output, labels):\n \"\"\"\n Compute the accuracy for a set of predictions, with a given set of true labels.\n \n Returns:\n Accuracy of the predictions.\n \"\"\"\n correct_output = np.argmax(output,axis=1)\n correct_labels = np.argmax(labels,axis=1)\n\n is_correct = [a == b for a,b in zip(correct_output, correct_labels)]\n\n accuracy = sum(is_correct) * 1. / labels.shape[0]\n return accuracy\n\ndef get_initial_params():\n \"\"\"\n Compute the initial parameters for the neural network.\n\n There should be four parameters for this model:\n W1 is the weight matrix for the convolutional layer\n b1 is the bias vector for the convolutional layer\n W2 is the weight matrix for the output layers\n b2 is the bias vector for the output layer\n\n Weight matrices should be initialized with values drawn from a random normal distribution.\n The mean of that distribution should be 0.\n The variance of that distribution should be 1/sqrt(n) where n is the number of neurons that\n feed into an output for that layer.\n\n Bias vectors should be initialized with zero.\n \n Returns:\n A dict mapping parameter names to numpy arrays\n \"\"\"\n conv_size = config['conv_size']\n pool_size = config['pool_size']\n conv_channels = config['conv_channels']\n \n size_after_convolution = 28 - conv_size + 1\n size_after_max_pooling = size_after_convolution // pool_size\n\n num_hidden = size_after_max_pooling * size_after_max_pooling * conv_channels\n\n return {\n 'W1': np.random.normal(size = (conv_channels, 1, conv_size, conv_size), scale=1/ math.sqrt(conv_size * conv_size)),\n 'b1': np.zeros(conv_channels),\n 'W2': np.random.normal(size = (num_hidden, 10), scale = 1/ math.sqrt(num_hidden)),\n 'b2': np.zeros(10)\n }","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.411388Z","iopub.status.busy":"2025-11-09T11:13:15.410724Z","iopub.status.idle":"2025-11-09T11:13:15.418512Z","shell.execute_reply":"2025-11-09T11:13:15.417682Z"},"papermill":{"duration":0.021168,"end_time":"2025-11-09T11:13:15.419951","exception":false,"start_time":"2025-11-09T11:13:15.398783","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"31d06902","cell_type":"markdown","source":"We will train our model using Batch Gradient Descent. For simplicity we will not be training our model on the whole dataset but instead we will fix some small number of batches (no epochs), to ensure training is fast and efficient.

","metadata":{"papermill":{"duration":0.011079,"end_time":"2025-11-09T11:13:15.441705","exception":false,"start_time":"2025-11-09T11:13:15.430626","status":"completed"},"tags":[]}},{"id":"f249bb76","cell_type":"code","source":"def model_training(train_data, train_labels, dev_data, dev_labels, forward_prop_function, \n backward_prop_function, learning_rate=0.1, batch_size=16, num_batches=200):\n \"\"\"\n Train the neural network using batch gradient descent.\n\n Args:\n train_data: A numpy array containing the training examples\n train_labels: A numpy array containing the training labels\n dev_data: A numpy array containing the dev examples\n dev_labels: A numpy array containing the dev labels\n forward_prop_function: A function for the forward propagation of the model\n backward_prop_function: A function for the backward propagation of the model\n The learning rate for batch gradient descent\n The batch size\n The number of batches for the training\n \n Returns:\n The learned parameters for the model.\n The dev set cost history throughout training\n The dev set accuracy history throughout training\n \"\"\"\n \n dev_cost_history = []\n dev_acc_history = []\n \n params = get_initial_params()\n \n print('Starting Neural Network Training\\n')\n \n for batch in range(num_batches):\n \n if batch % 10 == 0:\n print(f'Processing {batch} / {num_batches}')\n \n batch_data = train_data[\n (batch_size * batch):(batch_size * (batch + 1))]\n batch_labels = train_labels[(batch_size * batch):(batch_size * (batch + 1))]\n \n total_grad = {}\n \n for item, label in zip(batch_data, batch_labels):\n \n grads = backward_prop_function(item, label, params)\n \n for key, value in grads.items():\n \n if key not in total_grad:\n total_grad[key] = np.zeros(value.shape)\n \n total_grad[key] += value / batch_data.shape[0]\n \n params['W1'] = params['W1'] - learning_rate * total_grad['W1']\n params['W2'] = params['W2'] - learning_rate * total_grad['W2']\n params['b1'] = params['b1'] - learning_rate * total_grad['b1']\n params['b2'] = params['b2'] - learning_rate * total_grad['b2']\n \n if (batch + 1) % 20 == 0:\n \n dev_predictions = []\n dev_cost = 0.\n \n for item, label in zip(dev_data, dev_labels):\n \n y, loss = forward_prop_function(item, label, params)\n \n dev_predictions.append(y)\n dev_cost += loss\n \n dev_cost /= dev_data.shape[0]\n dev_cost_history.append(dev_cost)\n \n dev_accuracy = compute_accuracy(dev_predictions, dev_labels)\n dev_acc_history.append(dev_accuracy)\n \n print(f'Cost = {dev_cost}, Accuracy = {dev_accuracy}')\n \n print('Neural Network Training Completed')\n\n return params, dev_cost_history, dev_acc_history","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.465374Z","iopub.status.busy":"2025-11-09T11:13:15.465073Z","iopub.status.idle":"2025-11-09T11:13:15.474266Z","shell.execute_reply":"2025-11-09T11:13:15.473506Z"},"papermill":{"duration":0.022767,"end_time":"2025-11-09T11:13:15.475667","exception":false,"start_time":"2025-11-09T11:13:15.452900","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"1912e4d1","cell_type":"markdown","source":"## 5. Model Application: MNIST Classification","metadata":{"papermill":{"duration":0.010921,"end_time":"2025-11-09T11:13:15.497302","exception":false,"start_time":"2025-11-09T11:13:15.486381","status":"completed"},"tags":[]}},{"id":"626e4dac","cell_type":"markdown","source":"After building our CNN, it's time to train and test it on the MNIST dataset.","metadata":{"papermill":{"duration":0.010758,"end_time":"2025-11-09T11:13:15.518649","exception":false,"start_time":"2025-11-09T11:13:15.507891","status":"completed"},"tags":[]}},{"id":"4490517c","cell_type":"markdown","source":"

5.1 Data Loading and Visualiztion","metadata":{"papermill":{"duration":0.010568,"end_time":"2025-11-09T11:13:15.539677","exception":false,"start_time":"2025-11-09T11:13:15.529109","status":"completed"},"tags":[]}},{"id":"07c5c95a","cell_type":"markdown","source":"We will load the training and testing data, see its shape and visualize some training examples.","metadata":{"papermill":{"duration":0.010462,"end_time":"2025-11-09T11:13:15.560591","exception":false,"start_time":"2025-11-09T11:13:15.550129","status":"completed"},"tags":[]}},{"id":"04d515b4","cell_type":"code","source":"train_data = np.loadtxt('/kaggle/input/digit-recognizer/train.csv', delimiter=',', skiprows=1)\ntest_data = np.loadtxt('/kaggle/input/digit-recognizer/test.csv', delimiter=',', skiprows=1)\n\np = np.random.permutation(train_data.shape[0])\ntrain_data = train_data[p]\n\nlabels = train_data[:, 0]\ntrain_data = train_data[:, 1:]\n\ntrain_data = np.reshape(train_data, (train_data.shape[0], 1, 28, 28))\ntest_data = np.reshape(test_data, (test_data.shape[0], 1, 28, 28))\nprint(f'Training data shape: {train_data.shape}')\n\nfig, axes = plt.subplots(1,10, figsize=(10, 10))\nfor i, ax in enumerate(axes.flat):\n ax.imshow(train_data[i][0])\n ax.set_title(labels[i].astype(int))\n ax.axis('off')\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:15.583536Z","iopub.status.busy":"2025-11-09T11:13:15.582774Z","iopub.status.idle":"2025-11-09T11:13:20.687250Z","shell.execute_reply":"2025-11-09T11:13:20.686330Z"},"papermill":{"duration":5.117863,"end_time":"2025-11-09T11:13:20.688826","exception":false,"start_time":"2025-11-09T11:13:15.570963","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"80faf439","cell_type":"markdown","source":"

5.2 Data Preprocessing","metadata":{"papermill":{"duration":0.011323,"end_time":"2025-11-09T11:13:20.711742","exception":false,"start_time":"2025-11-09T11:13:20.700419","status":"completed"},"tags":[]}},{"id":"e29c0697","cell_type":"markdown","source":"We will ensure training labels are one-hot-encoded. We will allocate some of the training examples to a dev set. And we will z-score normalize our training data for a more successful training.","metadata":{"papermill":{"duration":0.011488,"end_time":"2025-11-09T11:13:20.734457","exception":false,"start_time":"2025-11-09T11:13:20.722969","status":"completed"},"tags":[]}},{"id":"b2d77ace","cell_type":"code","source":"one_hot_labels = np.zeros((labels.shape[0], 10))\none_hot_labels[np.arange(labels.shape[0]), labels.astype(int)] = 1\n\ndev_size = config['dev_size']\ndev_data = train_data[:dev_size]\ndev_labels = one_hot_labels[:dev_size]\ntrain_data = train_data[dev_size:]\ntrain_labels = one_hot_labels[dev_size:]\n\nmean = np.mean(train_data)\nstd = np.std(train_data)\n\ntrain_data = (train_data - mean) / std\ndev_data = (dev_data - mean) / std\ntest_data = (test_data - mean) / std","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:20.758777Z","iopub.status.busy":"2025-11-09T11:13:20.758470Z","iopub.status.idle":"2025-11-09T11:13:21.363379Z","shell.execute_reply":"2025-11-09T11:13:21.362404Z"},"papermill":{"duration":0.619332,"end_time":"2025-11-09T11:13:21.365132","exception":false,"start_time":"2025-11-09T11:13:20.745800","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"4b7c9899","cell_type":"markdown","source":"

5.3 Training the Model","metadata":{"papermill":{"duration":0.013166,"end_time":"2025-11-09T11:13:21.389685","exception":false,"start_time":"2025-11-09T11:13:21.376519","status":"completed"},"tags":[]}},{"id":"f403a382","cell_type":"markdown","source":"Now that our data is ready, we can train the model we have built from scratch, and visualize the learning curve.","metadata":{"papermill":{"duration":0.011724,"end_time":"2025-11-09T11:13:21.413086","exception":false,"start_time":"2025-11-09T11:13:21.401362","status":"completed"},"tags":[]}},{"id":"de894924","cell_type":"code","source":"learning_rate = config['learning_rate']\nbatch_size = config['batch_size']\nnum_batches = config['num_batches']\n\nparams, dev_cost_history, dev_acc_history = model_training(train_data, train_labels, dev_data,\n dev_labels, forward_prop, backward_prop, learning_rate, batch_size, num_batches)\n\nfig, (ax_dc, ax_da) = plt.subplots(2)\n\nax_dc.plot(dev_cost_history, 'r')\nax_dc.set_xlabel('Time')\nax_dc.set_ylabel('Cost')\n\nax_da.plot(dev_acc_history,'b')\nax_da.set_xlabel('Time')\nax_da.set_ylabel('Accuracy')\n\nplt.tight_layout()\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:13:21.437567Z","iopub.status.busy":"2025-11-09T11:13:21.437240Z","iopub.status.idle":"2025-11-09T11:21:45.701830Z","shell.execute_reply":"2025-11-09T11:21:45.701043Z"},"papermill":{"duration":504.279008,"end_time":"2025-11-09T11:21:45.703372","exception":false,"start_time":"2025-11-09T11:13:21.424364","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"518a3302","cell_type":"markdown","source":"We achieved 95% Accuracy on the Dev Set!","metadata":{"papermill":{"duration":0.012889,"end_time":"2025-11-09T11:21:45.729915","exception":false,"start_time":"2025-11-09T11:21:45.717026","status":"completed"},"tags":[]}},{"id":"209d48ee","cell_type":"markdown","source":"

5.4 Predicting the Test Set","metadata":{"papermill":{"duration":0.012928,"end_time":"2025-11-09T11:21:45.755965","exception":false,"start_time":"2025-11-09T11:21:45.743037","status":"completed"},"tags":[]}},{"id":"e1cc4f6f","cell_type":"markdown","source":"Finally, we will compute our predictions on the test set.","metadata":{"papermill":{"duration":0.012695,"end_time":"2025-11-09T11:21:45.781769","exception":false,"start_time":"2025-11-09T11:21:45.769074","status":"completed"},"tags":[]}},{"id":"4adb3b05","cell_type":"code","source":"predictions = model_predict(test_data, params, forward_prop)\n\nids = np.arange(1, len(predictions) + 1)\nsubmission = np.column_stack([ids, predictions])\n\nnp.savetxt('submission.csv', submission, fmt='%d', header='ImageId,Label', comments='', delimiter=',')","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:21:45.811043Z","iopub.status.busy":"2025-11-09T11:21:45.810715Z","iopub.status.idle":"2025-11-09T11:30:15.331895Z","shell.execute_reply":"2025-11-09T11:30:15.330731Z"},"papermill":{"duration":509.551028,"end_time":"2025-11-09T11:30:15.346062","exception":false,"start_time":"2025-11-09T11:21:45.795034","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"4b5f465b","cell_type":"markdown","source":"Some of the predictions look like this:","metadata":{"papermill":{"duration":0.013093,"end_time":"2025-11-09T11:30:15.372894","exception":false,"start_time":"2025-11-09T11:30:15.359801","status":"completed"},"tags":[]}},{"id":"5b7bee14","cell_type":"code","source":"fig, axes = plt.subplots(1,10, figsize=(10, 10))\nfor i, ax in enumerate(axes.flat):\n ax.imshow(test_data[i][0])\n ax.set_title(predictions[i].astype(int))\n ax.axis('off')\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2025-11-09T11:30:15.400862Z","iopub.status.busy":"2025-11-09T11:30:15.400563Z","iopub.status.idle":"2025-11-09T11:30:15.876644Z","shell.execute_reply":"2025-11-09T11:30:15.875765Z"},"papermill":{"duration":0.49181,"end_time":"2025-11-09T11:30:15.878006","exception":false,"start_time":"2025-11-09T11:30:15.386196","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"275d0f1f","cell_type":"markdown","source":"

Thanks for Reading!

\nI hope you learned something new! Consider upvoting if you think this notebook could help others, and feel free to play with the CNN config by yourself to get a better accuracy!\n\n**Author**: Ángel Valencia","metadata":{"papermill":{"duration":0.013716,"end_time":"2025-11-09T11:30:15.905671","exception":false,"start_time":"2025-11-09T11:30:15.891955","status":"completed"},"tags":[]}}]}