{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"},{"sourceId":9111211,"sourceType":"datasetVersion","datasetId":5499203}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Building a Neural Network From Scratch! 🚀","metadata":{}},{"cell_type":"markdown","source":"## Table of Contents\n* [1. Introduction](#1)\n* [2. Vectors and Matrices](#2)\n* [3. Fundementals of Neural Networks](#2)\n    - [3.1. Introduction to Linear Regression and Perceptrons](#3.1)\n    - [3.2. Understanding Sigmoid Neurons](#3.2)\n    - [3.3. Neural Network Architecture](#3.3)\n    - [3.4. Learning with Gradient Descent](#3.4)\n    - [3.5. The Backpropagation Algorithm](#3.5)\n* [4. The MNIST Dataset](#4)\n    - [4.1. A Brief Exploration](#4.1)\n    - [4.2. Preparing The Data For Our Network](#4.2)\n* [5. Building a Neural Network from Scratch](#5)\n    - [5.1. Initialization](#5.1)\n    - [5.2. Forward Propagation](#5.2)\n    - [5.3. Training with Stochastic Gradient Descent (SGD)](#5.3)\n    - [5.4. Updating Weights and Biases](#5.4)\n    - [5.5. Backpropagation](#5.5)\n    - [5.6. Calculating Accuracy](#5.6)\n    - [5.7. Computing the Cost Derivative](#5.7)\n    - [5.8. Supporting Functions](#5.8)\n    - [5.9. Bringing Everything Together](#5.9)\n* [6. Classifying Handwritten Digits](#6)\n    - [6.1. Network Initialization](#6.1)\n    - [6.2. Training the Network](#6.2)\n    - [6.3. Evaluating Performance](#6.3)\n    - [6.4. Future Enhancements](#6.4)","metadata":{}},{"cell_type":"markdown","source":"<a id='1'></a>\n## [1. Introduction](#1)\n\nWelcome to this journey into neural networks and deep learning! If you've ever wondered how computers recognize handwritten digits, play games better than humans, or even drive cars, you're in the right place. This notebook will guide you step-by-step through building a neural network from scratch, applying it to the famous MNIST dataset for classifying handwritten digits.\n\nNeural networks, inspired by the human brain, are key to artificial intelligence. These networks, made up of layers of interconnected \"neurons,\" learn complex patterns and make intelligent decisions. While the concept may seem daunting, breaking it into smaller steps makes it more approachable.\n\nWe’ll follow the structure of Michael Nielsen’s textbook Neural Networks and Deep Learning, which provides excellent explanations and code examples. Nearly all the concepts and code we use come from this resource.\n\nHere’s a quick overview of what we’ll cover:\n\n* **Vectors & Matrices Review**: We’ll briefly revisit the linear algebra needed to grasp neural networks.\n* **Neural Network Basics** : Learn about linear regression, perceptrons, sigmoid neurons, network architecture, and training via gradient descent.\n* **MNIST Dataset**: Get hands-on with this classic dataset in machine learning.\n* **Building a Neural Network from Scratch**: Implement forward propagation, backpropagation, and stochastic gradient descent to classify digits.\n\nThis notebook is designed to be engaging and accessible, even if you’re new to neural networks. By the end, you’ll understand how they work and gain the skills to build and improve them. However, for a more thorough understanding, I highly recommend reading [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/). \n\nThere’s also a [GitHub repo](https://github.com/dannybozbay/Neural-Networks-From-Scratch) with the framework we build here , along with optimized versions that are faster and more effective.\n\nBefore we begin, remember: learning neural networks can be challenging, especially if you’re not familiar with linear algebra and calculus. Don’t let that discourage you! Anyone can grasp these concepts with persistence. If you get stuck, ask questions, seek help, and keep going. The reward is well worth it!\n\nNow, let’s dive into the exciting world of neural networks and deep learning!","metadata":{}},{"cell_type":"markdown","source":"<a id='2'></a>\n## [2. Vectors and Matrices](#2)\n\nIn this section, we cover the foundational linear algebra concepts necessary for implementing a neural network. This includes how to represent vectors and matrices, and perform key operations such as dot products, matrix multiplication, and element-wise (Hadamard) products.\n\n### Vectors\n\nA **vector** is a one-dimensional array of numbers and can be either a **row vector** or a **column vector**.\n\n* A **column vector** is an $n \\times 1$ matrix, denoted in bold lowercase, such as $\\textbf{v}$, and appears as:\n\n    $$\n    \\textbf{v} = \n    \\begin{bmatrix}\n    v_{1} \\\\\n    v_{2} \\\\\n    \\vdots \\\\\n    v_{n}\n    \\end{bmatrix}\n    $$\n    \n* A **row vector** is a $1 \\times n$ matrix, also denoted in bold lowercase, such as $\\textbf{u}$, and appears as:\n\n    $$\n    \\textbf{u} = \n    \\begin{bmatrix}\n    u_1 & u_2 & \\ldots & u_n\n    \\end{bmatrix}\n    $$\n\nTo convert between column and row vectors, use the transpose operation:\n\n$$\n\\begin{bmatrix}\nv_{1} \\\\\nv_{2} \\\\\n\\vdots \\\\\nv_{n}\n\\end{bmatrix}^{\\top}\n=   \n\\begin{bmatrix}\nv_1 & v_2 & \\ldots & v_n\n\\end{bmatrix}\n$$\n\nFor convenience in inline text, we write column vectors as row vectors with the transpose operation applied:\n\n\n$$\n\\textbf{v} =\n\\begin{bmatrix}\nv_1 & v_2 & \\ldots & v_n\n\\end{bmatrix}^\\top\n$$\n\n### Matrices\n\nMatrices are two-dimensional arrays of numbers and are denoted in bold uppercase. For example, a matrix $\\textbf{A}$ with dimensions $m \\times n$ is represented as:\n\n$$\n\\textbf{A} =   \\begin{bmatrix}\n    a_{11} & a_{12} & \\cdots & a_{1n}\\\\\n    a_{21} & a_{22} & \\cdots & a_{2n}\\\\\n    \\vdots & \\vdots & \\ddots & \\vdots\\\\\n    a_{m1} & a_{m2} & \\cdots & a_{mn}\\\\\n  \\end{bmatrix}\n$$\n\nEach element of the matrix is denoted by $a_{ij}$, where $i$ refers to the row and $j$ refers to the column.\n\n### Operations\n\nHere are some key matrix operations used in neural networks:\n\n1. **Dot Product**: The dot product of two vectors $\\textbf{v}$ and $\\textbf{u}$ is calculated as:\n\n    $$\n    \\textbf{v} \\cdot \\textbf{u} = \\sum_{i=1}^{n} v_i u_i\n    $$\n    \n    The result is a scalar (single number).\n    \n    \n2. **Matrix-Vector Multiplication**: When multiplying a matrix $\\textbf{A}$ by a column vector $\\textbf{v}$, the result is another column vector:\n\n    $$\n    \\textbf{u} = \\textbf{A}\\textbf{v}\n    $$\n\n    This operation is performed by taking the dot product of each row of $\\textbf{A}$ with the vector $\\textbf{v}$:\n    \n    $$\n    \\textbf{A}\\textbf{v} = \n    \\begin{bmatrix}\n    \\sum_{j=1}^n a_{1j}v_j \\\\\n    \\sum_{j=1}^n a_{2j}v_j \\\\\n    \\vdots \\\\\n    \\sum_{j=1}^n a_{mj}v_j\n    \\end{bmatrix}\n    $$\n    \n\n3. **Matrix-Matrix Multiplication**: To multiply two matrices $\\textbf{A}$ and $\\textbf{B}$, where $\\textbf{A}$ is $m \\times n$ and $\\textbf{B}$ is $n \\times p$, the result is an $m \\times p$ matrix $\\textbf{C}$:\n\n    $$\n    \\textbf{C} = \\textbf{AB}\n    $$\n    \n    where elements given by the dot product of the corresponding row of $\\textbf{A}$ and the column of $\\textbf{B}$:\n    \n    $$\n    c_{ij} = \\sum_{k=1}^n a_{ik}b_{kj}\n    $$\n\n\n4. **Hadamard Product**: The Hadamard product (element-wise multiplication) of two matrices $\\textbf{A}$ and $\\textbf{B}$ of the same dimensions is:\n    \n    $$\n    \\textbf{A} \\odot \\textbf{B} = \n    \\begin{bmatrix}\n    a_{11}b_{11} & a_{12}b_{12} & \\cdots & a_{1n}b_{1n} \\\\\n    a_{21}b_{21} & a_{22}b_{22} & \\cdots & a_{2n}b_{2n} \\\\\n    \\vdots & \\vdots & \\ddots & \\vdots \\\\\n    a_{m1}b_{m1} & a_{m2}b_{m2} & \\cdots & a_{mn}b_{mn} \\\\\n    \\end{bmatrix}\n    $$\n    \n    \n5. **Gradient of a Scalar-Valued Function with Respect to a Vector**:  For a scalar-valued function $C$ with respect to a column vector $\\textbf{x}$, the gradient is also a vector given by:\n\n    $$\n    \\nabla_{\\textbf{x}}C = \\frac{\\partial C}{\\partial \\textbf{x}} =\n    \\begin{bmatrix}\n    \\frac{\\partial C}{\\partial x_1} \\\\\n    \\frac{\\partial C}{\\partial x_2} \\\\\n    \\vdots \\\\\n    \\frac{\\partial C}{\\partial x_n}\n    \\end{bmatrix}\n    $$\n    \n    where each element represents the partial derivative of $C$ with respect to the corresponding element of $\\textbf{x}$.\n    \n    \n6. **Gradient of a Scalar-Valued Function with Respect to a Matrix**: For a scalar-valued function $C$ that depends on a matrix $\\textbf{A}$, the gradient with respect to $\\textbf{A}$ is another matrix:\n\n    $$\n    \\nabla_{\\textbf{A}} C = \\frac{\\partial C}{\\partial \\textbf{A}} =\n    \\begin{bmatrix}\n    \\frac{\\partial C}{\\partial a_{11}} & \\frac{\\partial C}{\\partial a_{12}} & \\cdots & \\frac{\\partial C}{\\partial a_{1n}} \\\\\n    \\frac{\\partial C}{\\partial a_{21}} & \\frac{\\partial C}{\\partial a_{22}} & \\cdots & \\frac{\\partial C}{\\partial a_{2n}} \\\\\n    \\vdots & \\vdots & \\ddots & \\vdots \\\\\n    \\frac{\\partial C}{\\partial a_{m1}} & \\frac{\\partial C}{\\partial a_{m2}} & \\cdots & \\frac{\\partial C}{\\partial a_{mn}} \\\\\n    \\end{bmatrix}\n    $$\n    \n    Here, each element of the gradient matrix $\\nabla_{\\textbf{A}} C$ represents the partial derivative of the scalar function $C$ with respect to the corresponding element $a_{ij}$ of the matrix $\\textbf{A}$.\n    ","metadata":{}},{"cell_type":"markdown","source":"<a id='3'></a>\n## [3. Fundementals of Neural Networks](#3)\n\nWith the basics of linear algebra covered, I welcome you to the fundamentals of neural networks! This section lays the groundwork for understanding how neural networks operate and why they are so powerful.\n\nWe'll start with the basics of linear regression and perceptrons, which are the building blocks of more complex networks. You'll then be introduced to sigmoid neurons and the architecture of neural networks, which define how these networks are structured and how they process information. We'll also cover essential training techniques, including gradient descent and the backpropagation algorithm, which are crucial for teaching neural networks to learn from data. \n\nBy the end of this section, you'll have a solid foundation in the key concepts that underpin neural networks, preparing you for the exciting task of building your own neural network from scratch.","metadata":{}},{"cell_type":"markdown","source":"<a id='3.1'></a>\n### [3.1. Introduction to Linear Regression and Perceptrons](#3.1)\n\n**Linear Regression**\n\nLinear regression is one of the most fundamental techniques in machine learning. It models the relationship between a dependent variable $y$ (the output we want to predict) and one or more independent variables $x$ (the input features).\n\nFor a single input variable, the relationship is described by the equation of a straight line:\n\n$$y = wx + b$$\n\nwhere: \n* $y$ is the dependent variable (the output or target value we want to predict).\n* $x$ is the independent variable (the input feature).\n* $w$ is the weight (or slope) of the line.\n* $b$ is the bias (or intercept).\n\nWhen we have multiple input variables, say $n$ inputs, the equation extends to:\n\n$$ y = w_1x_1 + w_2x_2 + \\ldots w_nx_n + b $$\n\nHere, each $x_i$ is an independent variable with a corresponding weight, $w_i$, and $b$ is a constant bias term.\n\n**Perceptrons**\n\nNow let's connect this to perceptrons. A perceptron is a basic unit of a neural network that performs a similar linear computation to linear regression but with a twist—it produces a binary output (0 or 1):\n\n<div style=\"width:100%;text-align: center;\"> <img align=middle src=\"https://s7280.pcdn.co/wp-content/uploads/2020/05/box-16.png\" alt=\"perceptron\" style=\"height:200px;\"> </div>\n\nIn the example shown above, the perceptron has three inputs: $x_1$, $x_2$, and $x_3$. It's output is determined by the following rule:\n   \n1. Each input $x_j$ is assigned a weight $w_j$.\n2. The weighted sum of the inputs is calculated: $\\sum_j w_jx_j$.\n3. The sum is compared to a threshold value.\n    \nThe output is 0 if the sum is less than or equal to the threshold, and 1 if it is greater. Mathematically, this is expressed as:\n    \n$$\n\\begin{equation}\n  \\text{output} =\\begin{cases}\n    0, & \\text{if    $\\sum_j w_jx_j \\leq \\text{threshold}$}\\\\\n    1, & \\text{if    $\\sum_j w_jx_j > \\text{threshold}$}\n  \\end{cases}\n\\end{equation}\n$$\n    \nExpressed this way, you can see that the weights $w_j$ represent the importance of each input. If $w_1$ is very small, the contribution of $x_1$ to the weighted sum will be small, suggesting that $x_1$ is not very important. Conversely, if $w_1$ is large, $x_1$ will have a significant contribution to the weighted sum, indicating that $x_1$ is important.    \n\nNow suppose we have a neuron with $n$ inputs. Then to simplify, we can rewrite the sum of weighted inputs as a dot product $\\textbf{w}\\cdot \\textbf{x}$. We also replace the threshold with what's known as the perceptron's bias, $b\\equiv -\\text{threshold}$. Thus, output rule becomes:\n\n$$\n\\begin{equation}\n   \\text{output}=\\begin{cases}\n    0, & \\text{if    $\\textbf{w}\\cdot \\textbf{x} + b\\leq 0 $}\\\\\n    1, & \\text{if    $\\textbf{w}\\cdot \\textbf{x} + b > 0$}\n  \\end{cases}\n\\end{equation}\n$$\n\nHere, $\\textbf{w}$ and $\\textbf{x}$ represent the vectors of the weights and inputs, respectively. The bias $b$, which is a scalar, determines how easy it is for the perceptron to output a 1. A large positive bias makes it easier for the perceptron to fire, while a large negative bias makes it harder.\n    \nIn essence, perceptrons are like small linear regression machines that classify inputs into binary categories based on a weighted sum of the inputs. That's all there is to them! ","metadata":{}},{"cell_type":"markdown","source":"<a id='3.2'></a>\n### [3.2. Understanding Sigmoid Neurons](#3.2)\n\n**The Need for Activation Functions**\n\nPreviously, we explored perceptrons, which output either 0 or 1 based on a linear combination of inputs. While useful for simple tasks, perceptrons have limitations: well...they produce binary outputs and only capture linear relationships. To address these issues, we use activation functions, which allow neurons to produce a range of outputs and introduce non-linearity into our networks.\n\nAmong various activation functions, the sigmoid function is a widely used choice due to its smooth, non-linear characteristics.\n\n**Sigmoid Neurons**\n\nLet's introduce the sigmoid neuron, a more advanced type of neuron that overcomes the limitations of perceptrons. Just like a perceptron, a sigmoid neuron has inputs $x_1, x_2, \\ldots$, where each input $x_j$ is associated with a weight $w_j$, and there's an overall bias $b$.\n\nThe key difference lies in the output. Instead of being just 0 or 1, the output of a sigmoid neuron is given by the sigmoid function $\\sigma(z)$, where $z = \\textbf{w} \\cdot \\textbf{x} + b$. The sigmoid function is defined as:\n\n$$\n\\boxed{\n\\sigma(z) \\equiv \\frac{1}{1 + e^{-z}}}\n$$\n\nThus, the output of a sigmoid neuron with inputs $x_1, x_2, \\ldots$, weights $w_1, w_2, \\ldots$, and bias $b$ is:\n\n$$\n\\sigma(\\textbf{w} \\cdot \\textbf{x} +b) = \\frac{1}{1 + \\text{exp}\\left(-\\sum_{j}w_jx_j - b\\right)}\n$$\n\nAt first glance, the sigmoid function might seem complex, but it's intuitive once you understand its properties. The function smoothly maps any input value to a value between 0 and 1, making it suitable for tasks requiring probability estimation.\n\n**Comparing Sigmoid Neurons to Perceptrons**\n\nDespite their differences, sigmoid neurons share similarities with perceptrons. To see this, let's examine the behavior of the sigmoid function under extreme values of $z$ (where $z = \\textbf{w} \\cdot \\textbf{x} + b$):\n\n* When $z$ is a large positive number, $e^{-z}$ approaches 0, so $\\sigma(z)$ approaches 1. This is similar to the perceptron outputting 1 when its input is above a certain threshold.\n\n* When $z$ is a large negative number, $e^{-z}$ approaches infinity, so $\\sigma(z)$ approaches 0. This resembles the perceptron outputting 0 when its input is below a threshold.\n\nIt’s only when $z = \\textbf{w} \\cdot \\textbf{x} + b$ is of moderate size that the sigmoid neuron deviates significantly from the perceptron model. This non-linearity is what gives sigmoid neurons their power, allowing neural networks to learn and model complex patterns in data.\n\n**Visualizing the Sigmoid Function**\n\nThe algebraic form of the sigmoid function isn't so important. What really matters is its shape when plotted. Here's the shape:\n\n<div style=\"width:100%;text-align: center;\"> <img align=middle src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/1200px-Logistic-curve.svg.png\" alt=\"perceptron\" style=\"height:300px;\"> </div>\n\nThe sigmoid curve is S-shaped, smoothly transitioning from 0 to 1. This smooth transition allows the neuron to output a range of values, providing more nuanced outputs than the hard threshold of a perceptron.\n\n**Summary**\n\nSigmoid neurons enhance the capabilities of neural networks by introducing non-linearity, allowing the network to model more complex relationships. By understanding sigmoid neurons, we take a significant step toward grasping the inner workings of more advanced neural network architectures. In the next section, we will explore how these neurons are organized into networks to solve complex problems.","metadata":{}},{"cell_type":"markdown","source":"<a id='3.3'></a>\n### [3.3. Neural Network Architecture](#3.3)\n\n#### Layers of a Neural Network\n\nA simple neural network, like a single perceptron, works similarly to linear regression but with an added non-linear activation function. When you hear about neural networks with many layers, these are known as deep neural networks. Despite their complexity, all neural networks, no matter how deep, consist of the same basic components: input layers, hidden layers, and output layers.\n\n**Input Layer**:\n\n* The input layer is the first layer of the network. Neurons in this layer, called input neurons, receive the raw input data. For example, if the input is a 28x28 grayscale image, the input layer would have 784 neurons, each representing a pixel's intensity scaled between 0 and 1.\n\n**Output Layer**:\n\n* The output layer is the last layer of the network. It contains neurons that produce the final result. For instance, in a network designed to determine whether an image depicts the digit \"9,\" the output layer might contain a single neuron. An output value less than 0.5 indicates \"not a 9,\" while a value greater than 0.5 indicates \"a 9.\"\n\n**Hidden Layers**:\n\n* Between the input and output layers are the hidden layers. These layers are called \"hidden\" because they are neither inputs nor outputs. Hidden layers are crucial for the network's ability to learn complex patterns and representations. A simple network may have only one hidden layer, while deep networks have two or more hidden layers. :<div style=\"width:100%;text-align: center;\"> <img align=middle src=\"https://lh6.googleusercontent.com/proxy/Ri-hfKMZvEVc9jgbEC9Naf_ndZvOagjA4Xo0us-5i4vkWa43we6Sl2ix3q-f7Sk48aEMnXCmTIYXnAKEj01xJcGvkmOA9gY8TF-ZF_0\" alt=\"perceptron\" style=\"height:100%;margin-top:3rem;\"> </div> \n\n**Designing the Input and Output Layers**\n\nDesigning the input and output layers is usually straightforward:\n\n* **Input Layer**: Its design is dictated by the form of the input data. For example, each pixel of an image can be an input neuron.\n\n* **Output Layer**: Its design depends on the task. In binary classification tasks, such as identifying if an image is a \"9\" or not, a single output neuron is sufficient.\n\n**Designing Hidden Layers**\n\nDesigning hidden layers is more of an art than a science. There are no simple rules for determining the optimal number of hidden layers or the number of neurons in each layer. Researchers rely on heuristics and experience. The key is to balance network complexity with computational efficiency:\n\n* **Complexity vs. Training Time**: Increasing the number of hidden layers can enhance the network's ability to learn complex patterns but also increases training time and the risk of overfitting. Overfitting happens when the network learns the training data too well, including noise and outliers, resulting in poor performance on new data. We'll dicuss this in more detail later on.\n\n#### Notation \n\nPreviously, we described the computation of a single neuron's output using vectors for inputs ($\\boldsymbol{x}$) and weights ($\\boldsymbol{w}$), yielding a scalar output, $a$, through the sigmoid activation function:\n\n$$\na = \\sigma (\\underbrace{\\textbf{w} \\cdot \\textbf{x} + b}_\\text{$z$}) = \\sigma(z)\n$$\n\nHere, $z = \\textbf{w} \\cdot \\textbf{x} + b$ is the weighted sum of inputs, a key term when we discuss backpropagation. However, in neural networks, there are often multiple layers and many neurons. To handle this complexity, we need to extend our notation to describe entire layers of neurons.\n\n**Extending to a Layer of Neurons**\n\nIn a layer of neurons, the input remains a vector $\\boldsymbol{x}$, but each neuron has its own set of weights and a bias. These weights are represented by a matrix, and the biases by a bias vector. The sigmoid function is applied element-wise to compute a vector of activations for the neurons in that layer.\n\nKey Notation:\n\n* $w_{jk}^l$: Weight from the $k$-th neuron in layer $(l-1)$ to the $j$-th neuron in layer $l$.\n* $b_j^l$: Bias for the $j$-th neuron in layer $l$.\n* $a_j^l$: Activation of the $j$-th neuron in layer $l$.\n\nFor example, $w_{24}^3$ refers to the weight connecting the 4th neuron in layer 2 to the 2nd neuron in layer 3.\n\n<div style=\"width:100%;text-align: center;\"> <img align=middle src=\"https://lh4.googleusercontent.com/proxy/MgW1IDikGx7WRNQ10vPON_cmUsGeziaONeIOTPFxOIVjQ0FNX_TykjCd-2LxEJb7Yp9kqs_Y3mOPoLqWxde1yHxOelbXDdwD8DWRGGo\" alt=\"perceptron\" style=\"height:100%;\"> </div> \n\nWith this notation, we can write the relationship between activations of consecutive layers:\n\n$$\na_j^l = \\sigma \\left( \\sum_k w^l_{jk} a^{l-1}_k + b^{l}_{j} \\right)\n$$\n\nIn matrix form, this becomes:\n\n$$\n\\boxed{\n\\textbf{a}^l = \\sigma \\left( \\textbf{W}^l \\textbf{a}^{l-1} + \\textbf{b}^l \\right) = \\sigma (\\textbf{z}^l )}\n$$\n\nWhere:\n\n* $\\boldsymbol{a}^l$: Activations in layer $l$.\n* $\\boldsymbol{W}^l$: Weight matrix from layer $(l-1)$ to layer $l$.\n* $\\boldsymbol{a}^{l-1}$: Activations from the previous layer.\n* $\\boldsymbol{b}^l$: Bias vector in layer $l$.\n* $\\boldsymbol{z}^l$: Weighted sum of inputs for layer $l$.\n","metadata":{}},{"cell_type":"markdown","source":"<a id='3.4'></a>\n### [3.4. Learning with Gradient Descent](#3.4)\n\n**Introducing the Cost Function**\n\nNow that we understand how neural networks are structured, the next question is: how do they learn? Learning in this context means adjusting the network's weights and biases so that its output closely matches the target value for all training inputs.\n\nTo understand this, let's suppose we measure the performance of a network using a cost function, defined as:\n\n$$\n\\boxed{\nC(\\textbf{W}, \\textbf{b}) \\equiv \\frac{1}{2n}\\sum_\\textbf{x} ||y(\\textbf{x}) - \\textbf{a}||^2\n}\n$$\n\nHere:\n\n* $\\textbf{W}$ represents all the weights in the network.\n* $\\textbf{b}$ represents all the biases.\n* $n$ is the total number of training inputs.\n* $\\textbf{a}$ is the network's output for input $\\textbf{x}$.\n* The sum is over all training inputs $\\textbf{x}$.\n\nThis cost function (known as mean squared error or MSE) is a scalar and has two important properties:\n\n1. It is non-negative: $C(\\textbf{W}, \\textbf{b}) \\geq 0$, since each term in the sum is non-negative.\n\n2. The cost decreases as the network output $\\textbf{a}$ gets closer to the target value $y(\\textbf{x})$.\n\nOur objective is to minimize the cost, which means finding the weights and biases that make $C(\\textbf{W}, \\textbf{b})$ as small as possible. This is achieved using the gradient descent algorithm.\n\n**Gradient Descent**\n\nGradient descent is used to minimize the cost function by adjusting the parameters $\\textbf{W}$ and $\\textbf{b}$.To illustrate, let's consider minimizing a general function $C(\\textbf{v})$, where $\\textbf{v}$ is a vector of two variables:\n\n$$\n\\textbf{v} = \\begin{bmatrix}\n       v_1 \\\\\n       v_2 \\\\\n     \\end{bmatrix}\n$$\n\n\n<div style=\"width:100%;text-align: center;\"> <img align=middle src=\"https://lh4.googleusercontent.com/proxy/ShxqdtX5DNBuVbSoTv-6swpfR8ear7Afm_TZClKhqr0S2zRNsjaSHyDpNVEtgq585914-D4VRmobGNbzObMFc4yQkSuniINlAhU1wHYZ-y_fCmWAGevO\" alt=\"perceptron\" style=\"height:100%;\"> </div> \n\nWe want to find the lowest point of $C$, similar to a ball rolling down to the bottom of a valley. This intuitive idea guides us toward minimizing the function.\n\nWhen we move by a small amount $\\Delta v_1$ in the $v_1$ direction, and a small amount $\\Delta v_2$ in the $v_2$ direction, calculas tells us that the function $C$ changes approximately by:\n\n$$\n\\Delta C \\approx \\frac{\\partial{C}}{\\partial{v_1}}\\Delta v_1 +  \\frac{\\partial{C}}{\\partial{v_2}}\\Delta v_2\n$$\n\nTo roll down the valley, we need to choose $\\Delta v_1$ and $\\Delta v_2$ in such a way that $\\Delta C$ is negative. To figure this out, we define $\\Delta \\textbf{v}$ as the vector of changes in $\\textbf{v}$:\n\n$$\n\\Delta \\textbf{v} = \\begin{bmatrix}\n       \\Delta v_1 \\\\\n       \\Delta v_2 \\\\\n     \\end{bmatrix}\n$$\n\nWe also define the *gradient* of $C$ as the vector of partial derivatives:\n\n$$\n\\nabla C = \\begin{bmatrix}\n       \\frac{\\partial{C}}{\\partial{v_1}} \\\\\n       \\frac{\\partial{C}}{\\partial{v_2}} \\\\\n     \\end{bmatrix}\n$$\n\nWith these defintions, the change in $C$ becomes:\n\n$$\n\\Delta C \\approx \\nabla C \\cdot \\Delta\\textbf{v}\n$$\n\nNow, suppose we choose the change in $\\textbf{v}$ as:\n\n$$\n\\boxed{\n\\Delta \\textbf{v} = -\\eta \\nabla C}\n$$\n\nwhere $\\eta$ is a small, positive paramater called the *learning rate*. The corresponding change in the cost function becomes:\n\n$$\n\\Delta C \\approx -\\eta \\nabla C \\cdot \\nabla C = -\\eta ||\\nabla C||^2 \\leq 0\n$$\n\nThis shows that $C$ will always decrease (or stay the same), which is exactly what we want!\n\nIn the gradient descent algorithm, we repeatedly compute the gradient $\\nabla C$ and adjust the parameters $\\textbf{v}$ by moving in the opposite direction, effectively \"rolling down\" the slope. The update rule for $\\textbf{v}$ is:\n\n$$\n\\textbf{v} \\rightarrow \\textbf{v}' = \\textbf{v} - \\eta \\nabla C,\n$$\n\nThis process is repeated over and over, continually decreasing $C$, with the goal of eventually reaching the global minimum.\n\n\n**Applying Gradient Descent to Minimize the Cost**\n\nNow that we understand gradient descent, let's see how it's applied in the context of neural networks. The network learns by adjusting its parameters—weights $\\textbf{W}$ and biases $\\textbf{b}$—to minimize the cost function:\n\n$$\nC(\\textbf{W}, \\textbf{b}) \\equiv \\frac{1}{2n}\\sum_\\textbf{x} ||y(\\textbf{x}) - \\textbf{a}||^2\n$$\n\nSpecifically, we update the weights and biases by taking steps in the opposite direction to the gradients of the cost function:\n\n$$\n\\begin{equation}\n\\begin{aligned}\n\\textbf{W} \\rightarrow \\textbf{W}' & = \\textbf{W} - \\eta \\frac{\\partial C}{\\partial \\textbf{W}} \\\\\n\\textbf{b} \\rightarrow \\textbf{b}' & = \\textbf{b} - \\eta \\frac{\\partial C}{\\partial \\textbf{b}} \\\\\n\\end{aligned}\n\\end{equation}\n$$\n\nIn component form, for each weight $w_{jk}$ and bias $b_j$, the update rules are: \n\n$$\n\\boxed{\n\\begin{align*} \nw_{jk} \\rightarrow w'_{jk} & = w_{jk} - \\eta \\frac{\\partial{C}}{\\partial{w_{jk}}} \\\\\nb_j \\rightarrow b_j' & = b_j - \\eta \\frac{\\partial{C}}{\\partial{b_j}}\n\\end{align*}}\n$$\n\nBy repeatedly applying these update rules, we gradually adjust the weights and biases, effectively \"rolling down the hill\" toward the minimum of the cost function, where the network performs best.\n\n**Stochastic Gradient Descent**\n\nWhen working with a large number of training inputs, calculating the full gradient $\\nabla C = (\\frac{\\partial C}{\\partial \\textbf{W}}, \\frac{\\partial C}{\\partial \\textbf{b}})$ for a large dataset can be slow, as the cost function is given by:\n\n$$\nC = \\frac{1}{n}\\sum_{\\textbf{x}}C_{\\textbf{x}}\n$$\n\nwhere\n\n$$\nC_{\\textbf{x}} \\equiv \\frac{1}{2}||y(\\textbf{x})-\\textbf{a}||^2\n$$\n\nHere, $C$ is the average cost over all training inputs $\\textbf{x}$, and $C_{\\textbf{x}}$ is the cost for each individual input. Calculating the gradient $\\nabla C$ involves computing the gradients $\\nabla C_{\\textbf{x}}$ for every input, which can be very time-consuming.\n\nTo address this, we use *stochastic gradient descent* (SGD). Instead of computing the gradient over the entire dataset, SGD approximates $\\nabla C$ using a small, randomly selected subset of data known as a mini-batch.\n\nFor a mini-batch $\\mathcal{B}$ of size $m$, the gradient approximation is:\n\n$$\n\\nabla C \\approx \\frac{1}{m} \\sum_{\\textbf{x} \\in \\mathcal{B}} \\nabla C_{\\textbf{x}}\n$$\n\nHere, the sum is over the inputs $\\textbf{x}$ in the mini-batch $\\mathcal{B}$. The update rules for the weights $w_{jk}$ and biases $b_j$ in this context are:\n\n$$\n\\boxed{\n\\begin{align*} \nw_{jk} \\rightarrow w'_{jk} & = w_{jk} - \\frac{\\eta}{m} \\sum_{\\textbf{x} \\in \\mathcal{B}}  \n\\frac{\\partial{C_{\\textbf{x}}}}{\\partial{w_{jk}}} \\\\\nb_{j} \\rightarrow b'_{j} & = b_{j} - \\frac{\\eta}{m} \\sum_{\\textbf{x} \\in \\mathcal{B}}  \n\\frac{\\partial{C_{\\textbf{x}}}}{\\partial{b_{j}}}\n\\end{align*}}\n$$\n\nThus, for each mini-batch, the weights and biases are updated based on the average gradient computed over that mini-batch. This process is repeated for all mini-batches in an epoch, providing a more efficient approximation to the full gradient descent.","metadata":{"execution":{"iopub.status.busy":"2024-08-06T13:15:56.712647Z","iopub.execute_input":"2024-08-06T13:15:56.713486Z","iopub.status.idle":"2024-08-06T13:15:56.718975Z","shell.execute_reply.started":"2024-08-06T13:15:56.713453Z","shell.execute_reply":"2024-08-06T13:15:56.717707Z"}}},{"cell_type":"markdown","source":"<a id='3.5'></a>\n### [3.5. The Backpropagation Algorithm](#3.5)\n\n#### Why We Need Backpropagation\n\nIn the previous section, we discussed using gradient descent to minimize the cost function, which measures a network's prediction error. However, to actually apply gradient descent, we need a way of calculating the gradients of the cost function with respect to each weight and bias.\n\nThis is where the backpropagation algorithm comes in. By leveraging the chain rule from calculus, backpropagation efficiently computes all necessary gradients in a single backward pass through the network. These gradients are then used in gradient descent to update weights and biases, helping minimize the cost function and improve the network's predictions. This efficiency makes backpropagation essential for training modern deep learning models.\n\n#### Understanding the Gradients\n\nBefore we dive into backpropagation, let's first focus on what we're trying to compute—the gradients.\n\nFor each weight $w_{jk}^l$ or bias $b_j^l$, the gradient of the cost function with respect to that parameter, $\\frac{\\partial C}{\\partial w_{jk}^l}$ or $\\frac{\\partial C}{\\partial b_j^l}$, measures how sensitive the cost function $C$ is to changes in that parameter.\n\n* **Large Gradient**: A small change in the weight or bias significantly affects the cost, meaning this parameter is crucial and requires careful adjustment.\n\n* **Small Gradient**: A small change has little effect on the cost, indicating that this parameter has less influence on the network's performance.\n\nBackpropagation efficiently calculates these gradients, and stochastic gradient descent (SGD) uses them to update the weights and biases, reducing error and improving the network's accuracy.\n\n#### The Core Idea of Backpropagation\n\nThe essence of backpropagation is to propagate the *error* from the output layer backward through the network. The error $\\delta^l_j$ of neuron $j$ in layer $l$ is defined as:\n\n$$\n\\boxed{\n\\delta^l_j = \\frac{\\partial C}{\\partial z^l_j}}\n$$\n\nWe call this the error because it measures how much the weighted input $z^l_j$ of neuron $j$ affects the overall cost $C$. To explain, imagine a demon at neuron $j$ in layer $l$ who can slightly alter the neuron's input by adding a small change $\\Delta z^l_j$. Instead of outputting $\\sigma(z^l_j)$, the neuron now outputs $\\sigma(z^l_j + \\Delta z^l_j)$. This small change propagates forward, ultimately changing the cost by $ \\Delta C = \\frac{\\partial C}{\\partial z^l_j} \\Delta z^l_j$\n\nIf $\\frac{\\partial C}{\\partial z^l_j}$ is large, the demon can significantly reduce the cost by choosing $\\Delta z^l_j$ with the opposite sign to $\\frac{\\partial C}{\\partial z^l_j}$. On the other hand, if $\\frac{\\partial C}{\\partial z^l_j}$ is near zero, the neuron is nearly optimal, and any perturbation has little effect on the cost. Thus, $\\frac{\\partial C}{\\partial z^l_j}$ serves as a measure of how far the neuron is from optimal performance.\n\nUsing backpropagation, we can compute the error vector $\\boldsymbol{\\delta}^l$ for each layer and relate these errors to the gradients of interest: $\\frac{\\partial C}{\\partial w_{jk}^l}$ and $\\frac{\\partial C}{\\partial b_j^l}$.\n\n#### The Four Fundemental Equations Behind Back Propagation\n\nBackpropagation is based on four fundamental equations that allow us to compute both the error $\\delta^l$ and the gradient of the cost function. While these equations may not be immediately clear, understanding them requires time and patience. This summary serves as an introduction to these concepts. For a deeper dive into the mathematics of backpropagation, I highly recommend Michael Nielsen's Neural Networks and Deep Learning, especially Chapter 2, which offers an excellent explanation and shares much of the notation used here.\n\n1. **Error in the Output Layer**:\n\n    The error $\\delta_j^L$ for the $j$-th neuron in the output layer $L$ is:\n    \n    $$\n    \\delta_j^L = \\frac{\\partial C}{\\partial a_j^L} \\sigma '(z_j^L)\n    $$\n    \n    In matrix form, this becomes:\n        \n    $$\n    \\boxed{\n    \\boldsymbol{\\delta}^L = \\nabla_{\\textbf{a}} C \\odot \\sigma'(\\textbf{z}^L)}\n    $$\n    \n    Where:\n    \n    * $\\boldsymbol{\\delta}^L$ is the vector of errors for the neurons in the output layer.\n    \n    * $\\nabla_{\\textbf{a}} C$ is the gradient of the cost function with respect to the output activations.\n    \n    * $\\sigma'(\\textbf{z}^L)$ is the vector of derivatives of the activation function with respect to the weighted inputs.\n    \n    * $\\odot$ represents the Hadamard product (element-wise multiplication).\n    \n    For quadratic cost (MSE), $\\nabla_{\\textbf{a}} C = (\\textbf{a}^L - \\textbf{y})$, so this simplifies to:\n    \n    $$\n    \\boxed{\n    \\boldsymbol{\\delta}^L = (\\textbf{a}^L - \\textbf{y}) \\odot \\sigma'(\\textbf{z}^L)}\n    $$\n    \n\n2. **Error in the Hidden Layers**\n\n    The error $\\delta_j^l$ for the $j$-th neuron in a hidden layer $l$ is given by:\n    \n    $$\n    \\delta_j^l = \\left( \\sum_k \\delta_k^{l+1} w_{kj}^{l+1} \\right)\\sigma '(z_j^l)\n    $$\n    \n    In matrix form, this is more efficiently expressed as:\n    \n    $$\n    \\boxed{\n    \\boldsymbol{\\delta}^l = \\left( \\left(\\textbf{W}^{l+1}\\right)^T \\boldsymbol{\\delta}^{l+1} \\right) \\odot \\sigma' (\\textbf{z}^l)}\n    $$\n    \n    This equation allows us to propagate the error backward through the network.\n        \n        \n3. **Gradient of the Cost Function with Respect to Biases**\n\n    The gradient of the cost function with respect to the bias $b_j^l$ for the $j$-th neuron in layer $l$ is: \n    \n    $$\n    \\frac{\\partial C}{\\partial b_j^l} = \\delta_j^l\n    $$\n    \n    In matrix form, this becomes:\n    \n    $$\n    \\boxed{\n    \\frac{\\partial C}{\\partial \\textbf{b}^l} = \\boldsymbol{\\delta}^l}\n    $$\n    \n    This indicates that the gradient of the cost with respect to the biases is simply the error in that layer.\n    \n    \n4. **Gradient of the Cost Function with Respect to Weights**\n\n    The gradient of the cost function with respect to the weight $w_{jk}^l$ in layer $l$ is:\n\n    $$\n    \\frac{\\partial C}{\\partial w_{jk}^l} = a_k^{l-1}\\delta_j^l\n    $$\n    \n    In matrix form, this becomes:\n    \n    $$\n    \\boxed{\n    \\frac{\\partial C}{\\partial \\textbf{W}^l} = \\boldsymbol{\\delta}^l \\left( \\textbf{a}^{l-1} \\right)^T}\n    $$\n    \n    This tells us that the gradient of the cost with respect to the weights is the product of the error at layer $l$ and the activations from the previous layer.\n    \n\n**Summary: The Four Equations of Backpropagation**\n\n$$\n\\boxed{\n\\begin{align*} \n& (1) \\quad \\boldsymbol{\\delta}^L = \\nabla_{\\textbf{a}} C \\odot \\sigma'(\\textbf{z}^L) \\\\\n& (2) \\quad \\boldsymbol{\\delta}^l = \\left( \\left(\\textbf{W}^{l+1}\\right)^T \\boldsymbol{\\delta}^{l+1} \\right) \\odot \\sigma' (\\textbf{z}^l) \\\\\n& (3) \\quad \\frac{\\partial C}{\\partial \\textbf{b}^l} = \\boldsymbol{\\delta}^l \\\\\n& (4) \\quad \\frac{\\partial C}{\\partial \\textbf{W}^l} = \\boldsymbol{\\delta}^l \\left( \\textbf{a}^{l-1} \\right)^T\n\\end{align*}}\n$$\n\nThese four equations form the foundation of backpropagation, enabling the efficient computation of gradients needed for updating weights and biases during training.\n\n#### The Backpropagation Algorithm Explained\n\nThe backpropagtion algorithm follows these steps:\n\n1. **Input**: Start by setting the input activation $\\textbf{a}^1 = \\textbf{x}$ for the input layer.\n\n2. **Feedforward**: For each layer $l = 2, 3, \\dots, L$ (from the first hidden layer to the output layer):\n\n    * Compute the weighted input $\\textbf{z}^l = \\textbf{W}^l \\textbf{a}^{l-1} + \\textbf{b}^l$.\n    \n    * Apply the activation function to get $\\textbf{a}^l = \\sigma(\\textbf{z}^l)$.\n\n\n3. **Compute Output Error**: At the output layer $L$, compute the error vector using:\n\n    $$\n    \\boldsymbol{\\delta}^L = \\nabla_{\\textbf{a}} C\n    \\odot \\sigma'(\\textbf{z}^L)\n    $$\n\n4. **Backpropagate the Error**: For each hidden layer $l = L-1, L-2, \\dots, 2$ (from the output layer back to the first hidden layer):\n\n    * Compute the error for each layer using:\n    \n        $$\n        \\boldsymbol{\\delta}^l = \\left( \\left(\\textbf{W}^{l+1}\\right)^T \\boldsymbol{\\delta}^{l+1} \\right) \\odot \\sigma (\\textbf{z}^l)\n        $$\n\n5. **Calculate Gradients**: Use the computed errors to get the gradients of the cost function:\n    \n    * For biases: $\\frac{\\partial C}{\\partial \\textbf{b}^l} = \\boldsymbol{\\delta}^l$\n    \n    * For weights: $\\frac{\\partial C}{\\partial \\textbf{W}^l} = \\boldsymbol{\\delta}^l \\left( \\textbf{a}^{l-1} \\right)^T$\n\nThis step-by-step process shows why it's called \"backpropagation\"—the errors are propagated backward from the output layer through the network.\n\n#### Backpropagation with Mini-Batch Gradient Descent\n\nRemember, we want to combine backpropagation with **stochastic gradient descent** (SGD), where the gradients are calculated over mini-batches of data rather than a single training example. Here's how to apply backpropagation in this context:\n\n1. **Input a Mini-Batch**: Select a batch of $m$ training examples. \n\n2. **For each training example  $\\textbf{x}$**:\n\n    * Set the input activation $\\textbf{a}^{\\textbf{x},1}$ and:\n        \n        * **Feedforward**: For each layer $l = 2, 3, \\dots, L$, compute $\\textbf{z}^{\\textbf{x},l}$ and $\\textbf{a}^{\\textbf{x},l}$.\n        \n        * **Output Error**: Compute $\\boldsymbol{\\delta}^{\\textbf{x},L}$ for the output layer.\n        \n        * **Backpropagate**: For each $l = L-1, L-2, \\dots, 2$, compute $\\boldsymbol{\\delta}^{\\textbf{x},l}$.\n        \n3. **Update Weights and Biases**:\n\n    * For each layer $l = L, L-1, \\dots, 2$, update the weights and biases:\n    \n        $$\n        \\begin{equation}\n        \\begin{aligned}\n        \\textbf{W}^l & \\rightarrow \\textbf{W}^l - \\frac{\\eta}{m}\\sum_{\\textbf{x}}\\boldsymbol{\\delta}^{\\textbf{x},l}(\\textbf{a}^{\\textbf{x},l-1})^T \\\\\n        \\textbf{b}^l & \\rightarrow \\textbf{b}^l - \\frac{\\eta}{m}\\sum_{\\textbf{x}}\\boldsymbol{\\delta}^{\\textbf{x},l}\n        \\end{aligned}\n        \\end{equation}\n        $$\n        \n    Here, $\\eta$ is the learning rate, and the summation is over all examples in the mini-batch.\n    \n\nIn practice, this algorithm is repeated for multiple mini-batches over several epochs of training. This ensures that the network progressively learns by adjusting weights and biases based on feedback from each mini-batch. Soon, we'll implement this algorithm in code.","metadata":{}},{"cell_type":"markdown","source":"<a id='4'></a>\n## [4. The MNIST Dataset](#4)\n\n\nBefore we dive into building our neural network framework, it's beneficial to first explore the dataset we'll be using—the MNIST dataset. This will help us familiarize ourselves with the data and allow us to plan how it will be structured for our network (e.g., data format, structure, and type). We'll outline a format that our models can work with, and then transform the MNIST data accordingly. If you decide to use a different dataset later, you'll just need to format it to match this structure.","metadata":{}},{"cell_type":"markdown","source":"<a id='4.1'></a>\n### [4.1. A Breif Exploration](#4.1)\n\nThe MNIST dataset contains 70,000 grayscale images of handwritten digits (0-9), each sized at 28x28 pixels. It's a popular dataset for training and testing machine learning models due to the relatively simple yet challenging task of digit recognition. The data is split into two sets: 42,000 images for training and 28,000 for testing, a 60-40 split.\n\nLet’s load the MNIST dataset and examine its structure.","metadata":{}},{"cell_type":"code","source":"# Essentials\nimport random\nimport numpy as np\nimport pandas as pd\nimport time\nfrom typing import List, Optional\n\n# Visualization\n!pip install scienceplots\nimport matplotlib.pyplot as plt\nimport scienceplots\n\n# Plot settings\nplt.style.use([\"science\", \"ieee\", \"notebook\", \"grid\", \"std-colors\"])\nplt.rcParams[\"legend.fontsize\"] = \"small\"\nplt.rcParams[\"legend.loc\"] = \"best\"\nplt.rcParams[\"figure.autolayout\"] = True\nplt.rcParams[\"figure.dpi\"] = 100","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:39.485940Z","iopub.execute_input":"2024-09-17T13:01:39.486455Z","iopub.status.idle":"2024-09-17T13:01:52.188466Z","shell.execute_reply.started":"2024-09-17T13:01:39.486412Z","shell.execute_reply":"2024-09-17T13:01:52.186505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_data():\n    train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\n    test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\n    return train, test\n\ntrain, test = load_data()\n\nprint(\"Train shape: \", train.shape)\nprint(\"Test shape: \", test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:52.191062Z","iopub.execute_input":"2024-09-17T13:01:52.191509Z","iopub.status.idle":"2024-09-17T13:01:56.500032Z","shell.execute_reply.started":"2024-09-17T13:01:52.191445Z","shell.execute_reply":"2024-09-17T13:01:56.498604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Preview the top 5 rows of the training data\ntrain.head(5)","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:56.501957Z","iopub.execute_input":"2024-09-17T13:01:56.502436Z","iopub.status.idle":"2024-09-17T13:01:56.523947Z","shell.execute_reply.started":"2024-09-17T13:01:56.502395Z","shell.execute_reply":"2024-09-17T13:01:56.522698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Preview the top 5 rows of the test data\ntest.head(5)","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:56.527057Z","iopub.execute_input":"2024-09-17T13:01:56.527583Z","iopub.status.idle":"2024-09-17T13:01:56.549739Z","shell.execute_reply.started":"2024-09-17T13:01:56.527522Z","shell.execute_reply":"2024-09-17T13:01:56.548583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Data Summary**:\n\n* Data is stored in Pandas DataFrames.\n* Each row represents a 28x28 grayscale image of a handwritten digit.\n* Each image has 784 features (pixels), with values from 0 (lightest) to 9 (darkest).\n* Training images are labeled with the digit they depict.\n* There are 42,000 images in the training set and 28,000 in the test set.\n\nBy converting each row in the dataframe to 28x28 array, we can visualise the handwritten digits:","metadata":{}},{"cell_type":"code","source":"def plot_digits(train_df=None, test_df=None, cols=4):\n    if isinstance(train_df, pd.DataFrame) and not train_df.empty:\n        print(\"Plotting training data...\")\n        # Pop labels out from the dataframe\n        train_labels = train_df.pop('label')\n        # Convert each row to a (28, 28) numpy array\n        train_arrays = train_df.apply(lambda row: row.values.reshape(28, 28), axis=1)\n        \n        # Define the number of rows for the subplot grid based on the number of columns\n        num_images = len(train_arrays)\n        rows = num_images // cols if num_images % cols == 0 else (num_images // cols) + 1\n        \n        # Create subplots with calculated rows and 4 columns\n        fig, axes = plt.subplots(rows, cols, figsize=(cols * 3, rows * 3))\n        axes = axes.flatten()  # Flatten to iterate easily\n\n        for ax, label, image in zip(axes, train_labels, train_arrays):\n            ax.imshow(image, cmap='gray')\n            ax.set_title(f\"Digit: {label}\")\n            ax.axis('off')\n\n        # Hide any remaining empty subplots\n        for i in range(num_images, len(axes)):\n            axes[i].axis('off')\n\n        # Adjust layout and show the plot\n        plt.tight_layout()\n        plt.show()\n\n    if isinstance(test_df, pd.DataFrame) and not test_df.empty:\n        print(\"Plotting test data...\")\n        # Convert each row to a (28, 28) numpy array for test_df\n        test_arrays = test_df.apply(lambda row: row.values.reshape(28, 28), axis=1)\n\n        # Define the number of rows for the subplot grid based on the number of columns\n        num_images = len(test_arrays)\n        rows = num_images // cols if num_images % cols == 0 else (num_images // cols) + 1\n\n        # Create subplots with calculated rows and 4 columns\n        fig, axes = plt.subplots(rows, cols, figsize=(cols * 3, rows * 3))\n        axes = axes.flatten()  # Flatten to iterate easily\n\n        for ax, image in zip(axes, test_arrays):\n            ax.imshow(image, cmap='gray')\n            ax.axis('off')\n\n        # Hide any remaining empty subplots\n        for i in range(num_images, len(axes)):\n            axes[i].axis('off')\n\n        # Adjust layout and show the plot\n        plt.tight_layout()\n        plt.show()\n\n    if (train_df is None or train_df.empty) and (test_df is None or test_df.empty):\n        print(\"No valid training or test data provided. Please provide a non-empty DataFrame.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-17T13:01:56.551192Z","iopub.execute_input":"2024-09-17T13:01:56.551595Z","iopub.status.idle":"2024-09-17T13:01:56.567423Z","shell.execute_reply.started":"2024-09-17T13:01:56.551560Z","shell.execute_reply":"2024-09-17T13:01:56.565906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the first 8 digits from the train and test sets\nplot_digits(train[:8], test[:8])","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:56.569126Z","iopub.execute_input":"2024-09-17T13:01:56.569648Z","iopub.status.idle":"2024-09-17T13:01:57.951727Z","shell.execute_reply.started":"2024-09-17T13:01:56.569614Z","shell.execute_reply":"2024-09-17T13:01:57.950495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='4.2'></a>\n### [4.2. Preparing The Data For Our Network](#4.2)\n\nAt the start of this chapter, we discussed the importance of defining a data format compatible with our models. Let's now establish that format and transform the MNIST dataset accordingly\n\n#### Data Format\n\nOur models will work with three distinct sets of data, each stored as a list. Each element in these lists corresponds to a unique sample of data. Specifically:\n\n* **Training data**: A list of tuples, where each tuple contains the (inputs, label) for an individual sample. This data is used to train the model, allowing it to learn and adjust its parameters.\n\n* **Validation data**: A list of tuples, where each tuple contains the (inputs, label) for an individual sample. This data is used to fine-tune the model and select the best parameters by evaluating its performance on unseen data during training. \n\n* **Test data**: A list of inputs with no associated labels. This data is used to assess the final performance of the trained model on completely new data, and make predictions.\n\nFor both inputs and labels:\n\n* **Inputs**: Represented by column vectors, $\\textbf{x}$. Since we’ll use NumPy for matrix operations, inputs should be stored as 2-dimensional NumPy arrays.\n* **Training labels**: Also represented by column vectors, $\\textbf{y}$, they should be stored as 2-dimensional NumPy arrays.\n* **Validation labels**: Should remain as integers since they’re not used in the training process.\n\n#### Transforming the MNIST Dataset\n\nWe will now transform the MNIST dataset to match the format outlined above. In addition, we also:\n\n* Split 5,000 samples from the training set to create a validation set.\n* Normalize the input values to the range [0, 1], which is preferred for neural networks.","metadata":{}},{"cell_type":"code","source":"def vectorize_label(j):\n    \"\"\"\n    Convert a digit (0-9) into a corresponding 10-dimensional unit vector.\n\n    Parameters:\n    j (int): The digit to be converted into a one-hot encoded vector (0-9).\n\n    Returns:\n    numpy.ndarray: A 10-dimensional unit vector with a 1.0 in the jth position and 0.0 elsewhere.\n    \"\"\"\n    e = np.zeros((10, 1))\n    e[j] = 1.0\n    return e","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:57.953317Z","iopub.execute_input":"2024-09-17T13:01:57.953703Z","iopub.status.idle":"2024-09-17T13:01:57.960240Z","shell.execute_reply.started":"2024-09-17T13:01:57.953672Z","shell.execute_reply":"2024-09-17T13:01:57.958782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def prep_data(n_valid=5000):\n    \"\"\"\n    Prepare the training, validation, and test datasets for the neural network.\n\n    This function performs the following steps:\n    1. Loads the data.\n    2. Shuffles the training data and separates a portion for validation.\n    3. Normalizes input data by scaling pixel values to the range [0, 1].\n    4. Converts input data into 784-dimensional column vectors (flattened images).\n    5. One-hot encodes the training labels.\n    6. Converts validation labels to NumPy integers.\n    7. Zips the training and validation inputs with their respective labels into lists of tuples.\n    \n    Returns:\n    - train (list of tuples): List of (input, label) tuples for training.\n    - valid (list of tuples): List of (input, label) tuples for validation.\n    - test (list): List of input column vectors for testing.\n    \"\"\"\n    # Load the training and test datasets\n    train, test = load_data()\n    \n    # Shuffle the training data and create a validation set\n    shuffled_train = train.sample(frac=1, random_state=0).reset_index(drop=True)\n    validation = shuffled_train.iloc[:n_valid]  # Select the first n_valid rows for validation\n    train = train.drop(validation.index)  # Remove validation rows from the training data\n    \n    # Separate features (inputs) and labels for training and validation data\n    X_train, y_train = train.drop(columns='label'), train['label']\n    X_valid, y_valid = validation.drop(columns='label'), validation['label']\n    X_test = test.copy()  # Keep a copy of the test data\n    \n    # Normalize the input data by scaling pixel values to the range [0, 1]\n    X_train = X_train / 255.0\n    X_valid = X_valid / 255.0\n    X_test = X_test / 255.0\n        \n    # Reshape inputs into 784-dimensional column vectors (28x28 images flattened)\n    X_train = [np.reshape(x, (784, 1)) for x in X_train.values]  \n    X_valid = [np.reshape(x, (784, 1)) for x in X_valid.values]  \n    X_test = [np.reshape(x, (784, 1)) for x in X_test.values]   \n    \n    # One-hot encode the training labels\n    y_train = [vectorize_label(y) for y in y_train.to_numpy()] \n    \n    # Convert validation labels to NumPy integers\n    y_valid = [np.int64(y) for y in y_valid] \n    \n    # Create lists of tuples for training and validation data\n    train = list(zip(X_train, y_train))\n    valid = list(zip(X_valid, y_valid))\n    \n    # Create a list of input vectors for the test data\n    test = list(X_test)\n\n    return train, valid, test","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:57.962245Z","iopub.execute_input":"2024-09-17T13:01:57.962628Z","iopub.status.idle":"2024-09-17T13:01:57.976636Z","shell.execute_reply.started":"2024-09-17T13:01:57.962598Z","shell.execute_reply":"2024-09-17T13:01:57.975231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare the data for our neural network\ntrain, validation, test = prep_data()\n\nprint(\"Train size:\", len(train))\nprint(\"Validation size:\", len(validation))\nprint(\"Test size:\", len(test))","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:01:57.978227Z","iopub.execute_input":"2024-09-17T13:01:57.978698Z","iopub.status.idle":"2024-09-17T13:02:02.995258Z","shell.execute_reply.started":"2024-09-17T13:01:57.978656Z","shell.execute_reply":"2024-09-17T13:02:02.994004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='5'></a>\n## [5. Building a Neural Network From Scratch](#5)\n\nCongratulations on making it this far! You've grasped the core concepts of neural networks and prepared the MNIST dataset for training. Now, we're about to embark on the most thrilling part of our journey: constructing a neural network from scratch!\n\nIn this chapter, we'll build a neural network step-by-step by developing a class called `Network`. We'll meticulously walk through each component, and by the end, you'll have a fully functional neural network ready to tackle real-world tasks.\n\nHere’s our roadmap for building the network:\n\n1. **Initialization**: Setting up the network’s structure, including weights and biases.\n\n2. **Forward Propagation**: Computing the network’s output by passing inputs through each layer.\n\n3. **Training with Stochastic Gradient Descent (SGD)**: Training the network using mini-batch gradient descent.\n\n4. **Updating Weights and Biases**: Adjusting the network’s parameters based on the gradients.\n\n5. **Backpropagation**: Calculating gradients for the cost function to update weights and biases.\n\n6. **Calculating Accuracy**: Evaluating the network’s performance on given data.\n\n7. **Computing the Cost Derivative**: Deriving the cost function with respect to output activations.\n\n8. **Supporting Functions**: Implementing essential functions like the sigmoid activation and its derivative.\n\n9. **Putting Everything Together**: With all the core methods defined and explained, assembling the complete and final class.\n\nOur implementation will be highly customizable, allowing you to experiment with different network architectures and observe how various configurations impact performance. This hands-on approach will not only enhance your understanding but also equip you for more advanced exploration in the field of neural networks.\n\nLet’s start by introducing the `Network` class, which will form the foundation of our neural network. This class will encompass the network’s structure and methods for training and evaluation.","metadata":{}},{"cell_type":"code","source":"class Network:\n    \"\"\"\n    A class representing a neural network for feedforward propagation and stochastic gradient descent training.\n\n    Attributes:\n    - sizes (list of int): List containing the number of neurons in each layer of the network.\n    - num_layers (int): Number of layers in the network, including input and output layers.\n    - biases (list of ndarray): List of bias vectors for each layer (excluding the input layer).\n    - weights (list of ndarray): List of weight matrices for each layer (connections between consecutive layers).\n\n    Methods:\n    - __init__(sizes): Initializes the network with given layer sizes.\n    - feedforward(a): Computes the output activations for a given input using feedforward propagation.\n    - SGD(training_data, epochs, mini_batch_size, eta, validation_data=None, monitor_training_accuracy=False, monitor_validation_accuracy=False): Trains the network using mini-batch stochastic gradient descent.\n    - update_mini_batch(mini_batch, eta): Updates weights and biases using backpropagation for a given mini-batch.\n    - backprop(x, y): Computes gradients for the cost function using backpropagation for a single training example.\n    - cost_derivative(output_activations, y): Computes the derivative of the cost function with respect to output activations.\n    - accuracy(data, convert=False): Calculates the accuracy of the network on the given data set.\n    \"\"\"","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:02:02.998924Z","iopub.execute_input":"2024-09-17T13:02:03.000204Z","iopub.status.idle":"2024-09-17T13:02:03.006720Z","shell.execute_reply.started":"2024-09-17T13:02:03.000155Z","shell.execute_reply":"2024-09-17T13:02:03.005239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='5.1'></a>\n### [5.1. Initialization](#5.1)\n\nWe start by initializing the network with the given layer sizes. We’ll create the `__init__` method to set up the network’s structure, including weights and biases:","metadata":{}},{"cell_type":"code","source":"def __init__(self, sizes):\n    \"\"\"\n    Initializes the neural network with given layer sizes and random weights and biases.\n\n    Parameters:\n    - sizes (list of int): List containing the number of neurons in each layer. The length of the list represents the number of layers in the network, including the input and output layers.\n    \"\"\"\n    self.num_layers = len(sizes)\n    self.sizes = sizes\n    # Initialize biases for all layers except the input layer with Gaussian distribution\n    self.biases = [np.random.randn(y, 1) for y in sizes[1:]]\n    # Initialize weights for all layers with Gaussian distribution\n    self.weights = [np.random.randn(y, x) / np.sqrt(x) for x, y in zip(sizes[:-1], sizes[1:])]","metadata":{"execution":{"iopub.status.busy":"2024-09-17T13:02:03.008461Z","iopub.execute_input":"2024-09-17T13:02:03.008984Z","iopub.status.idle":"2024-09-17T13:02:03.031142Z","shell.execute_reply.started":"2024-09-17T13:02:03.008941Z","shell.execute_reply":"2024-09-17T13:02:03.029708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Understanding the `__init__` Method\n\nThe `__init__` method in the `Network` class sets up the neural network’s architecture:\n\n1. **Network Architecture**: The `sizes` list defines the number of neurons in each layer (e.g., `sizes = [784, 30, 10]` creates a network 784 input neurons, 30 hidden neurons, and 10 output neurons).\n\n2. **Initializing Biases**: Biases are stored in a list of numpy arrays, one for each layer after the input layer, initialized with random values from a standard Guassian using `np.random.randn(y, 1)`.\n\n3. **Initializing Weights**: Weights are stored similarly in a list of numpy arrays, with each matrix containing random values for connections between layers, generated with `np.random.randn(y, x) / np.sqrt(x)`. Note, that the weights are initialized from a Gaussian with mean 0 and standard deviation $1/\\sqrt{n_{\\text{in}}}$. This small change helps neural networks learn faster.\n\n**Why Random Initialization?**: Randomly initializing weights and biases ensures that neurons start with different parameters, promoting diverse learning. Using a normal distribution helps maintain balanced data flow, reducing the risk of training issues like vanishing or exploding gradients.","metadata":{}},{"cell_type":"markdown","source":"<a id='5.2'></a>\n### [5.2. Forward Propagation](#5.2)\n\nNext, we’ll implement the `feedforward` method, which calculates the output of the network for a given input by passing it through each layer.","metadata":{}},{"cell_type":"code","source":"def feedforward(self, a):\n    \"\"\"\n    Perform feedforward propagation for a single input vector.\n\n    Parameters:\n    - a (ndarray): Input activation vector, shape (input_size, 1).\n\n    Returns:\n    - ndarray: Output activation vector after propagation through the network, shape (output_size, 1).\n    \"\"\"\n    # Iterate through each layer, updating the activations using the weights and biases\n    for b, w in zip(self.biases, self.weights):\n        # Compute the weighted input and apply the sigmoid activation function\n        a = sigmoid(np.dot(w, a) + b)  \n    return a  # Return the final output activation vector","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.348236Z","iopub.execute_input":"2024-09-17T11:59:40.349416Z","iopub.status.idle":"2024-09-17T11:59:40.362850Z","shell.execute_reply.started":"2024-09-17T11:59:40.349377Z","shell.execute_reply":"2024-09-17T11:59:40.361747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Understanding the `feedforward` Method\n\nThe `feedforward` method feeds an input activation vector `a` through the network to produce an output:\n\n1. **Purpose**: It calculates the activations for all layers, starting from the input layer, and returns the network's final output.\n\n2. **Layer-by-Layer Processing**: Using a `for` loop, it iterates through each layer's biases (`b`) and weights (`w`), paired together with Python’s `zip` function.\n\n3. **Activation Calculation**: For each layer, $l$, it computes the activations:\n\n    $$\n\\textbf{a}^l = \\sigma \\left( \\textbf{W}^l \\textbf{a}^{l-1} + \\textbf{b}^l \\right) = \\sigma (\\textbf{z}^l)\n    $$\n    \n    This process is repeated through all layers, ultimately returning the output of the network.","metadata":{}},{"cell_type":"markdown","source":"<a id='5.3'></a>\n### [5.3. Training with Stochastic Gradient Descent (SGD)](#5.3)\n\nWe’ll now add the SGD method to train the network using mini-batch stochastic gradient descent. This method will include monitoring options for training and validation accuracy.","metadata":{}},{"cell_type":"code","source":"def SGD(\n    self,\n    training_data,\n    epochs,\n    mini_batch_size,\n    eta,\n    validation_data=None,\n    monitor_training_accuracy=False,\n    monitor_validation_accuracy=False,\n):\n    \"\"\"\n    Trains the network using mini-batch stochastic gradient descent.\n\n    Parameters:\n    - training_data (list of tuples): List of tuples (x, y) representing training inputs and target outputs.\n    - epochs (int): Number of epochs for training.\n    - mini_batch_size (int): Size of each mini-batch.\n    - eta (float): Learning rate.\n    - validation_data (list of tuples, optional): If provided, the network will be evaluated on this data after each epoch.\n    - monitor_training_accuracy (bool, optional): If True, monitors accuracy on training data after each epoch.\n    - monitor_validation_accuracy (bool, optional): If True, monitors accuracy on validation data after each epoch.\n\n    Returns:\n    - tuple: Two lists containing training accuracy and validation accuracy per epoch, if monitoring is enabled.\n    \"\"\"\n    n = len(training_data)  # Total number of training samples\n    training_acc, validation_acc = [], []  # Lists to store accuracy per epoch\n    t0 = time.time()  # Start time for measuring training duration\n\n    # Iterate over the number of epochs\n    for j in range(1, epochs + 1):\n        # Shuffle the training data to ensure randomness\n        random.shuffle(training_data)\n\n        # Create mini-batches from the shuffled training data\n        mini_batches = [\n            training_data[k : k + mini_batch_size]\n            for k in range(0, n, mini_batch_size)\n        ]\n\n        # Iterate over each mini-batch and update the network's weights and biases\n        for mini_batch in mini_batches:\n            # Perform gradient descent on the mini-batch\n            self.update_mini_batch(mini_batch, eta)\n\n        # Print progress of the current epoch and elapsed time\n        print(f\"Epoch {j} complete (elapsed time: {time.time() - t0:.2f}s)\")\n        INDENT = \" \" * 4  # Indentation for printing accuracy results\n\n        # Monitor and print training accuracy if enabled\n        if monitor_training_accuracy:\n            accuracy = self.accuracy(training_data, convert=True)\n            training_acc.append(accuracy)\n            print(f\"{INDENT}Accuracy on training data: {accuracy:.2f}%\")\n\n        # Monitor and print validation accuracy if enabled\n        if monitor_validation_accuracy:\n            accuracy = self.accuracy(validation_data)\n            validation_acc.append(accuracy)\n            print(f\"{INDENT}Accuracy on validation data: {accuracy:.2f}%\")\n\n    return training_acc, validation_acc","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.364412Z","iopub.execute_input":"2024-09-17T11:59:40.364907Z","iopub.status.idle":"2024-09-17T11:59:40.377904Z","shell.execute_reply.started":"2024-09-17T11:59:40.364867Z","shell.execute_reply":"2024-09-17T11:59:40.376737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Understanding the `SGD` Method\n\nThe `SGD` method implements mini-batch stochastic gradient descent, which is the core technique for training our neural networks efficiently.\n\n1. **Purpose**: The `SGD` method implements mini-batch stochastic gradient descent, which is the core technique for training our neural networks efficiently. It trains the network by updating the weights and biases using small subsets of the training data, called mini-batches, across multiple passes through the dataset (epochs).\n\n\n2. **Parameters**:\n\n    * `training_data`: A list of tuples `(x, y)`, where `x` is the input and `y` is the target output.\n    \n    * `epochs`: The number of times the network will go through the entire training dataset.\n    \n    * `mini_batch_size`: The number of training samples in each mini-batch.\n    \n    * `eta` The learning rate, $\\eta$, which controls how much the weights and biases are adjusted.\n    \n    * `validation_data` (optional): A set of data used to evaluate the network's performance after each epoch. Like `training_data`, it's a list of tuples `(x, y)`, where `x` is the input and `y` is the target output.\n    \n    * `monitor_training_accuracy` and `monitor_validation_accuracy` (optional): If set to `True`, the method tracks and displays accuracy on training and validation data after each epoch. \n    \n    \n2. **The Training Process**:\n\n    * The method starts by shuffling the training data to ensure randomness. This helps the network generalize better by exposing it to different mini-batches in each epoch.\n    \n    * The shuffled data is then split into mini-batches. Each mini-batch is a subset of the training data, which helps speed up training and improves generalization.\n    \n    * For each mini-batch, the method calls `update_mini_batch`, which is a method we'll define next. This function performs the actual updates to the network's weights and biases based on the mini-batch data.\n    \n3. **Monitoring Progress**: After each epoch, the method can monitor and print accuracy on the training and validation data if `monitor_training_accuracy` or `monitor_validation_accuracy` are set to `True`. This provides insight into how well the network is learning and helps track its performance over time.\n\n4. **Ouput**: The method returns lists of training and validation accuracy for each epoch if monitoring is enabled, allowing you to analyze the network’s learning curve.\n\nBy using mini-batches, the `SGD` method efficiently trains the network, making it possible to handle large datasets and improve the model’s ability to generalize to new data. Next, we’ll look at how the `update_mini_batch` method fine-tunes the weights and biases for each mini-batch.","metadata":{}},{"cell_type":"markdown","source":"<a id='5.4'></a>\n### [5.4. Updating Weights and Biases](#5.4)\n\nIn the `SGD` method, the actual updates to the network's weights and biases are handled by the `update_mini_batch` method. Its role is to receive a mini-batch of training data and update the weights and biases based on the rules of stochastic gradient descent (SGD). Let's build that method below:","metadata":{}},{"cell_type":"code","source":"def update_mini_batch(self, mini_batch, eta):\n    \"\"\"\n    Update the network's weights and biases by applying stochastic gradient descent on a minibatch using backpropagation.\n\n    Parameters:\n    - mini_batch (list of tuples): List of tuples (x, y) representing mini-batch inputs and corresponding target outputs.\n    - eta (float): Learning rate.\n    \"\"\"\n    # Initialize gradient accumulators for biases and weights\n    nabla_b = [np.zeros(b.shape) for b in self.biases]\n    nabla_w = [np.zeros(w.shape) for w in self.weights]\n\n    # Compute gradients for each training example in the mini-batch using backpropagation\n    # And add to the accumulators\n    for x, y in mini_batch:\n        delta_nabla_b, delta_nabla_w = self.backprop(x, y)\n        nabla_b = [nb + dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]\n        nabla_w = [nw + dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]\n\n    m = len(mini_batch)  # Mini-batch size\n\n    # Update weights and biases using the stochastic gradient descent rules\n    self.weights = [w - (eta / m) * nw for w, nw in zip(self.weights, nabla_w)]\n    self.biases = [b - (eta / m) * nb for b, nb in zip(self.biases, nabla_b)]","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.379519Z","iopub.execute_input":"2024-09-17T11:59:40.379975Z","iopub.status.idle":"2024-09-17T11:59:40.395586Z","shell.execute_reply.started":"2024-09-17T11:59:40.379935Z","shell.execute_reply":"2024-09-17T11:59:40.394442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Understanding the `update_mini_batch` Method\n\n1. **Purpose**: This method updates the network's weights and biases for a given mini-batch of training data using the update rules of stochastic gradient descent (SGD).\n\n2. **Parameters**:\n\n    * `mini_batch` A list of tup;es `(x, y)`, where `x` is the input and `y` is the target output. This mini-batch is a small subset of the entire training dataset.\n    \n    * `eta`: The learning rate, controlling the step size for the updates.\n    \n    \n3. **Gradient Accumulation:** The method initializes `nabla_b` and `nabla_w` to accumulate the gradients for biases and weights, respectively, for each layer. These lists start as zero arrays, matching the dimensions of the network’s biases and weights.\n\n    $$\n    \\nabla C = \\left( \\underbrace{\\frac{\\partial C}{\\partial \\textbf{W}^1}, \\ldots, \\frac{\\partial C}{\\partial \\textbf{W}^L}}_\\text{nabla_w}, \\underbrace{\\frac{\\partial C}{\\partial \\textbf{b}^1}, \\ldots, \\frac{\\partial C}{\\partial \\textbf{b}^L}}_\\text{nabla_b}  \\right)\n    $$\n    \n\n4. **Gradient Computation**: For each training example in the mini-batch, the method computes the gradients of the cost function using backpropagation (`backprop`), returning `delta_nabla_b` and `delta_nabla_w`. These are added to the accumulators `nabla_b` and `nabla_w`.\n\n    \n5. **Updating Weights and Biases**: After processing the entire mini-batch, the method updates the weights and biases using the accumulated gradients and the rules of stochastic gradient descent.\n   \nThis method allows the network to iteratively minimize prediction errors by updating the weights and biases after processing each mini-batch. In the next section, we'll build the `backprop` method used to calculate the gradient of each individual training example in the mini-batch.","metadata":{}},{"cell_type":"markdown","source":"<a id='5.5'></a>\n### [5.5. Backpropagation](#5.5)\n\nWe've seen how the `update_mini_batch` method updates the network's weights and biases using gradients from each mini-batch. A key part of this process is the `backprop` method, which uses backpropagation to calculate the gradients for individual training examples. Let's explore how `backprop` works:","metadata":{}},{"cell_type":"code","source":"def backprop(self, x, y):\n    \"\"\"\n    Computes gradients for the cost function using backpropagation for a single training example.\n\n    Parameters:\n    - x (ndarray): Input data for a single training example, shape (input_size, 1).\n    - y (ndarray): Corresponding target output for the training example, shape (output_size, 1).\n\n    Returns:\n    - tuple: Two lists containing gradients for biases and weights for each layer.\n        - nabla_b (list of ndarray): Gradients for biases, each element has shape (neurons_in_layer, 1).\n        - nabla_w (list of ndarray): Gradients for weights, each element has shape (neurons_in_next_layer, neurons_in_layer).\n    \"\"\"\n    # Initialize gradients for biases and weights\n    nabla_b = [np.zeros(b.shape) for b in self.biases]\n    nabla_w = [np.zeros(w.shape) for w in self.weights]\n\n    # Feedforward pass: compute activations for each layer\n    activation = x\n    activations = [x]  # Store activations for each layer\n    zs = []  # Store weighted inputs (z vectors) for each layer\n    for b, w in zip(self.biases, self.weights):\n        z = np.dot(w, activation) + b\n        zs.append(z)\n        activation = sigmoid(z)\n        activations.append(activation)\n\n    # Backward pass: Output Layer\n    delta = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])\n    nabla_b[-1] = delta\n    nabla_w[-1] = np.dot(delta, activations[-2].T)\n    \n    # Backward pass: Hidden Layers\n    for l in range(2, self.num_layers):\n        z = zs[-l]\n        sp = sigmoid_prime(z)\n        delta = np.dot(self.weights[-l + 1].T, delta) * sp\n        nabla_b[-l] = delta\n        nabla_w[-l] = np.dot(delta, activations[-l - 1].T)\n\n    return nabla_b, nabla_w","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.397331Z","iopub.execute_input":"2024-09-17T11:59:40.398161Z","iopub.status.idle":"2024-09-17T11:59:40.412136Z","shell.execute_reply.started":"2024-09-17T11:59:40.398116Z","shell.execute_reply":"2024-09-17T11:59:40.410969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Restating the Equations of Backpropagation\n\nBefore exploring the `backprop` method, let's restate the key backpropagation equations:\n\n$$\n\\boxed{\n\\begin{align*} \n& (1) \\quad \\boldsymbol{\\delta}^L = \\nabla_{\\textbf{a}} C \\odot \\sigma'(\\textbf{z}^L) \\\\\n& (2) \\quad \\boldsymbol{\\delta}^l = \\left( \\left(\\textbf{W}^{l+1}\\right)^T \\boldsymbol{\\delta}^{l+1} \\right) \\odot \\sigma' (\\textbf{z}^l) \\\\\n& (3) \\quad \\frac{\\partial C}{\\partial \\textbf{b}^l} = \\boldsymbol{\\delta}^l \\\\\n& (4) \\quad \\frac{\\partial C}{\\partial \\textbf{W}^l} = \\boldsymbol{\\delta}^l \\left( \\textbf{a}^{l-1} \\right)^T\n\\end{align*}}\n$$\n\n\n#### Understanding the `backprop` Method\n\n1. **Purpose**: The `backprop` method computes the gradients of the cost function with respect to the weights and biases for a single training example. These gradients are used in the update_mini_batch method to update the network parameters during training.\n\n2. **Paramaters**: \n\n    * `x`: The input data for a single training example (column vector).\n    \n    * `y`: The corresponding target output for the training example (also a column vector).\n    \n    \n3. **Initialization of Gradients**:\n    \n    * Two lists, `nabla_b` and `nabla_w`, are initialized to store the gradients for biases and weights for the example `(x, y`). These are initially zero arrays that match the shapes of the biases and weights\n\n4. **Feedforward Pass**:\n\n    * The method performs a feedforward pass to compute the `activations` and weighted inputs (`zs`) for each layer, storing these for use in the backward pass.\n    \n    \n5. **Backward Pass**: \n\n    * **Output Layer**: The error term (`delta`) for the output layer is calculated using Equation (1):\n    \n    $$\n    \\boldsymbol{\\delta}^L = \\nabla_{\\textbf{a}} C \\odot \\sigma'(\\textbf{z}^L)\n    $$\n    \n    This is computed in the code as:  `delta = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])`. The gradients for the output layer's weights and biases are computed using Equations (3) and (4):\n    \n    $$\n    \\frac{\\partial C}{\\partial \\textbf{b}^L} = \\boldsymbol{\\delta}^{L}, \\quad \\frac{\\partial C}{\\partial \\textbf{W}^L} = \\boldsymbol{\\delta}^{L} \\left( \\textbf{a}^{L-1} \\right)^T\n    $$\n    \n    These gradients are stored at the end of the lists `nabla_b[-1]` and `nabla_w[-1]`.\n    \n    * **Hidden Layers**:  For each hidden layer, starting from the last and moving backward, the `delta` is computed using Equation (2):\n    \n    $$\n\\boldsymbol{\\delta}^l = \\left( \\left(\\textbf{W}^{l+1}\\right)^T \\boldsymbol{\\delta}^{l+1} \\right) \\odot \\sigma' (\\textbf{z}^l)\n    $$\n    \n    The weights and biases are updated similarly to the output layer.\n\n6. **Returning the Gradients**:  The method returns `nabla_b` and `nabla_w`, which contain the gradients for the biases and weights of the single training example. These are then used by `update_mini_batch` to adjust the network parameters.\n\nIn summary, the `backprop` method calculates the gradients of the cost function for a single training example by first performing a feedforward pass, then using a backward pass to compute the necessary gradients. Next, we’ll define the `cost_derivative` function used in this process.","metadata":{}},{"cell_type":"markdown","source":"<a id='5.6'></a>\n### [5.6. Computing the Cost Derivative](#5.6)\n\nIn backpropagation, we used the following equation to compute the error (`delta`) for the output layer:\n\n$$\n\\boldsymbol{\\delta}^L = \\nabla_{\\textbf{a}} C \\odot \\sigma'(\\textbf{z}^L)\n$$\n\nHere, $\\nabla_{\\textbf{a}} C$ represents the partial derivatives of the cost function with respect to the output activations. To calculate this, we use the `cost_derivative` method.\n\nSince we're using the quadratic cost function, defined for a single training input $\\textbf{x}$ as:\n\n$$\nC_{\\textbf{x}} = \\frac{1}{2}||\\textbf{y}(\\textbf{x}) - \\textbf{a}||^2\n$$\n\nThe derivative with respect to the output activations $\\textbf{a}$ is:\n\n$$\n\\nabla_{\\textbf{a}} C_{\\textbf{x}} = \\textbf{a} - \\textbf{y}\n$$\n\nNow, let's implement the `cost_derivative` method to calculate this.","metadata":{}},{"cell_type":"code","source":"def cost_derivative(self, output_activations, y):\n    \"\"\"\n    Compute the derivative of the cost function with respect to the output activations.\n\n    Parameters:\n    - output_activations (ndarray): Output activations from the network, shape (output_size, 1).\n    - y (ndarray): True output values, shape (output_size, 1).\n\n    Returns:\n    - ndarray: Vector of derivative of the cost with respect to output activations, shape (output_size, 1).\n    \"\"\"\n    return output_activations - y","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.413674Z","iopub.execute_input":"2024-09-17T11:59:40.414106Z","iopub.status.idle":"2024-09-17T11:59:40.427609Z","shell.execute_reply.started":"2024-09-17T11:59:40.414077Z","shell.execute_reply":"2024-09-17T11:59:40.426610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='5.7'></a>\n### [5.7. Calculating Accuracy](#5.7)\n\nWith the core methods of our `network` class in place, we now need a way to track the network’s performance during training. The `accuracy` method does this by calculating the percentage of correct predictions on a given dataset. This method was used in the `SGD` method when monitoring was enabled. Let’s go ahead and implement it:","metadata":{}},{"cell_type":"code","source":"def accuracy(self, data, convert=False):\n    \"\"\"\n    Calculate the percentage of correct predictions made by the network on the given data.\n\n    Parameters:\n    - data (list of tuples): List of tuples (x, y) representing inputs and desired outputs.\n    - convert (bool, optional): If True, indicates that the true output data (Y) is in vectorized form and needs to be converted to label format.\n\n    Returns:\n    - float: Percentage of correct predictions made by the network on the given data.\n    \"\"\"\n    if convert:\n        results = [(np.argmax(self.feedforward(x)), np.argmax(y)) for x, y in data]\n    else:\n        results = [(np.argmax(self.feedforward(x)), y) for x, y in data]\n\n    return sum(int(x == y) for x, y in results) / len(data) * 100","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.429085Z","iopub.execute_input":"2024-09-17T11:59:40.429432Z","iopub.status.idle":"2024-09-17T11:59:40.442834Z","shell.execute_reply.started":"2024-09-17T11:59:40.429403Z","shell.execute_reply":"2024-09-17T11:59:40.441725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Understanding the `accuracy` Method\n\n1. **Purpose**: The `accuracy` method evaluates how well the network is performing by calculating the accuracy on a given dataset. Accuracy is determined as the percentage of correct predictions the network makes compared to the true labels.\n\n2. **Paramaters**: \n\n    * `data`: A list of tuples `(x, y)`, where `x` is the input data and `y` is the expected output (or label).\n    \n    * `convert` (optional): A boolean flag indicating whether the true output `y` is in vectorized form (one-hot encoding) and needs to be converted back to label format. If `True`, the method will convert the one-hot encoded vectors back to their original label format before comparing predictions. This only needs to be done when the `data` provided is the training data, whose expected outputs `y` are vectorized. \n    \n3. **Prediction and Evaluation**:\n\n    * **Making Predictions**: For each input `x` in `data`, the network generates a prediction using the `feedforward` method. The predicted label is identified as the index of the neuron with the highest activation in the output layer (`np.argmax(self.feedforward(x))`).\n    \n    * **Comparison with True Labels**: The predicted label is then compared with the true label `y`. If `convert` is `True`, `y` is also converted to a label using `np.argmax(y)`.\n    \n    * **Calculating Accuracy**: The method counts the number of correct predictions (where the predicted label matches the true label) and divides this by the total number of examples in the dataset. The result is multiplied by 100 to express accuracy as a percentage. This is the value that's returned by the method.\n    \nThe accuracy method is crucial for tracking the network's learning progress during training. By monitoring accuracy on both the training and validation sets after each epoch, we can assess how well the network is generalizing to unseen data.","metadata":{}},{"cell_type":"markdown","source":"<a id='5.8'></a>\n### [5.8. Supporting Functions](#5.8)\n\nWe've built the core methods of our network class, including forward propagation, backpropagation, stochastic gradient descent, and accuracy evaluation. Two essential functions used throughout are the sigmoid activation function, $\\sigma(\\textbf{z})$, and its derivative, $\\sigma'(\\textbf{z})$. These functions are critical for both feedforward and backpropagation.\n\nThe sigmoid function is defined as:\n\n$$\n\\sigma(\\textbf{z}) = \\frac{1}{1 + e^{-\\textbf{z}}}\n$$\n\nIts derivative is:\n\n$$\n\\sigma'(\\textbf{z}) = \\sigma(\\textbf{z}) \\cdot \\left( 1 - \\sigma(\\textbf{z}) \\right)\n$$\n\nBoth can be implemented simply as follows:","metadata":{}},{"cell_type":"code","source":"def sigmoid(z):\n    \"\"\"\n    Compute the sigmoid function.\n\n    Parameters:\n    - z (ndarray): Input value or array.\n\n    Returns:\n    - ndarray: Sigmoid of the input value or array.\n    \"\"\"\n    return 1.0 / (1.0 + np.exp(-z))\n\n\ndef sigmoid_prime(z):\n    \"\"\"\n    Compute the derivative of the sigmoid function.\n\n    Parameters:\n    - z (ndarray): Input value or array.\n\n    Returns:\n    - ndarray: Derivative of the sigmoid function evaluated at the input value or array.\n    \"\"\"\n    return sigmoid(z) * (1 - sigmoid(z))","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.446610Z","iopub.execute_input":"2024-09-17T11:59:40.446992Z","iopub.status.idle":"2024-09-17T11:59:40.454502Z","shell.execute_reply.started":"2024-09-17T11:59:40.446955Z","shell.execute_reply":"2024-09-17T11:59:40.453251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='5.9'></a>\n### [5.9. Bringing Everything Together](#5.9)\n\nNow that we’ve built and understood each component of our neural network, it’s time to bring it all together. In this final part of the chapter, we’ll integrate all the methods we’ve developed into a complete `Network` class, which will serve as the foundation of our neural network framework. This is the moment where everything we’ve worked on so far comes together to create a powerful tool capable of learning from data.","metadata":{}},{"cell_type":"code","source":"class Network:\n    \"\"\"\n    A class representing a neural network for feedforward propagation and stochastic gradient descent training.\n\n    Attributes:\n    - sizes (list of int): List containing the number of neurons in each layer of the network.\n    - num_layers (int): Number of layers in the network, including input and output layers.\n    - biases (list of ndarray): List of bias vectors for each layer (excluding the input layer).\n    - weights (list of ndarray): List of weight matrices for each layer (connections between consecutive layers).\n\n    Methods:\n    - __init__(sizes): Initializes the network with given layer sizes.\n    - feedforward(a): Computes the output activations for a given input using feedforward propagation.\n    - SGD(training_data, epochs, mini_batch_size, eta, validation_data=None, monitor_training_accuracy=False, monitor_validation_accuracy=False): Trains the network using mini-batch stochastic gradient descent.\n    - update_mini_batch(mini_batch, eta): Updates weights and biases using backpropagation for a given mini-batch.\n    - backprop(x, y): Computes gradients for the cost function using backpropagation for a single training example.\n    - accuracy(data, convert=False): Calculates the accuracy of the network on the given data set.\n    - cost_derivative(output_activations, y): Computes the derivative of the cost function with respect to output activations.\n    \"\"\"\n\n    def __init__(self, sizes):\n        \"\"\"\n        Initializes the neural network with given layer sizes and random weights and biases.\n\n        Parameters:\n        - sizes (list of int): List containing the number of neurons in each layer. The length of the list represents the number of layers in the network, including the input and output layers.\n        \"\"\"\n        self.num_layers = len(sizes)\n        self.sizes = sizes\n        # Initialize biases for all layers except the input layer with Gaussian distribution\n        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]\n        # Initialize weights for all layers with Gaussian distribution\n        self.weights = [np.random.randn(y, x) / np.sqrt(x) for x, y in zip(sizes[:-1], sizes[1:])]\n\n    def feedforward(self, a):\n        \"\"\"\n        Perform feedforward propagation for a single input vector.\n\n        Parameters:\n        - a (ndarray): Input activation vector, shape (input_size, 1).\n\n        Returns:\n        - ndarray: Output activation vector after propagation through the network, shape (output_size, 1).\n        \"\"\"\n        # Iterate through each layer, updating the activations using the weights and biases\n        for b, w in zip(self.biases, self.weights):\n            a = sigmoid(\n                np.dot(w, a) + b\n            )  # Compute the weighted input and apply the sigmoid activation function\n        return a  # Return the final output activation vector\n\n    def SGD(\n        self,\n        training_data,\n        epochs,\n        mini_batch_size,\n        eta,\n        validation_data=None,\n        monitor_training_accuracy=False,\n        monitor_validation_accuracy=False,\n    ):\n        \"\"\"\n        Trains the network using mini-batch stochastic gradient descent.\n\n        Parameters:\n        - training_data (list of tuples): List of tuples (x, y) representing training inputs and target outputs.\n        - epochs (int): Number of epochs for training.\n        - mini_batch_size (int): Size of each mini-batch.\n        - eta (float): Learning rate.\n        - validation_data (list of tuples, optional): If provided, the network will be evaluated on this data after each epoch.\n        - monitor_training_accuracy (bool, optional): If True, monitors accuracy on training data after each epoch.\n        - monitor_validation_accuracy (bool, optional): If True, monitors accuracy on validation data after each epoch.\n\n        Returns:\n        - tuple: Two lists containing training accuracy and validation accuracy per epoch, if monitoring is enabled.\n        \"\"\"\n        n = len(training_data)  # Total number of training samples\n        training_acc, validation_acc = [], []  # Lists to store accuracy per epoch\n        t0 = time.time()  # Start time for measuring training duration\n\n        # Iterate over the number of epochs\n        for j in range(1, epochs + 1):\n            # Shuffle the training data to ensure randomness\n            random.shuffle(training_data)\n\n            # Create mini-batches from the shuffled training data\n            mini_batches = [\n                training_data[k : k + mini_batch_size]\n                for k in range(0, n, mini_batch_size)\n            ]\n\n            # Iterate over each mini-batch and update the network's weights and biases\n            for mini_batch in mini_batches:\n                # Perform gradient descent on the mini-batch\n                self.update_mini_batch(mini_batch, eta)\n\n            # Print progress of the current epoch and elapsed time\n            print(f\"Epoch {j} complete (elapsed time: {time.time() - t0:.2f}s)\")\n            INDENT = \" \" * 4  # Indentation for printing accuracy results\n\n            # Monitor and print training accuracy if enabled\n            if monitor_training_accuracy:\n                accuracy = self.accuracy(training_data, convert=True)\n                training_acc.append(accuracy)\n                print(f\"{INDENT}Accuracy on training data: {accuracy:.2f}%\")\n\n            # Monitor and print validation accuracy if enabled\n            if monitor_validation_accuracy:\n                accuracy = self.accuracy(validation_data)\n                validation_acc.append(accuracy)\n                print(f\"{INDENT}Accuracy on validation data: {accuracy:.2f}%\")\n\n        return training_acc, validation_acc\n\n    def update_mini_batch(self, mini_batch, eta):\n        \"\"\"\n        Update the network's weights and biases by applying stochastic gradient descent on a minibatch using backpropagation.\n\n        Parameters:\n        - mini_batch (list of tuples): List of tuples (x, y) representing mini-batch inputs and corresponding target outputs.\n        - eta (float): Learning rate.\n        \"\"\"\n        # Initialize gradient accumulators for biases and weights\n        nabla_b = [np.zeros(b.shape) for b in self.biases]\n        nabla_w = [np.zeros(w.shape) for w in self.weights]\n\n        # Compute gradients for each training example in the mini-batch using backpropagation\n        # And add to the accumulators\n        for x, y in mini_batch:\n            delta_nabla_b, delta_nabla_w = self.backprop(x, y)\n            nabla_b = [nb + dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]\n            nabla_w = [nw + dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]\n\n        m = len(mini_batch)  # Mini-batch size\n\n        # Update weights and biases using the stochastic gradient descent rules\n        self.weights = [w - (eta / m) * nw for w, nw in zip(self.weights, nabla_w)]\n        self.biases = [b - (eta / m) * nb for b, nb in zip(self.biases, nabla_b)]\n\n    def backprop(self, x, y):\n        \"\"\"\n        Computes gradients for the cost function using backpropagation for a single training example.\n\n        Parameters:\n        - x (ndarray): Input data for a single training example, shape (input_size, 1).\n        - y (ndarray): Corresponding target output for the training example, shape (output_size, 1).\n\n        Returns:\n        - tuple: Two lists containing gradients for biases and weights for each layer.\n            - nabla_b (list of ndarray): Gradients for biases, each element has shape (neurons_in_layer, 1).\n            - nabla_w (list of ndarray): Gradients for weights, each element has shape (neurons_in_next_layer, neurons_in_layer).\n        \"\"\"\n        # Initialize gradients for biases and weights\n        nabla_b = [np.zeros(b.shape) for b in self.biases]\n        nabla_w = [np.zeros(w.shape) for w in self.weights]\n\n        # Feedforward pass: compute activations for each layer\n        activation = x\n        activations = [x]  # Store activations for each layer\n        zs = []  # Store weighted inputs (z vectors) for each layer\n        for b, w in zip(self.biases, self.weights):\n            z = np.dot(w, activation) + b\n            zs.append(z)\n            activation = sigmoid(z)\n            activations.append(activation)\n\n        # Backward pass: compute gradient of the cost function\n        delta = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])\n        nabla_b[-1] = delta\n        nabla_w[-1] = np.dot(delta, activations[-2].T)\n\n        for l in range(2, self.num_layers):\n            z = zs[-l]\n            sp = sigmoid_prime(z)\n            delta = np.dot(self.weights[-l + 1].T, delta) * sp\n            nabla_b[-l] = delta\n            nabla_w[-l] = np.dot(delta, activations[-l - 1].T)\n\n        return nabla_b, nabla_w\n\n    def accuracy(self, data, convert=False):\n        \"\"\"\n        Calculate the percentage of correct predictions made by the network on the given data.\n\n        Parameters:\n        - data (list of tuples): List of tuples (x, y) representing inputs and desired outputs.\n        - convert (bool, optional): If True, indicates that the true output data (Y) is in vectorized form and needs to be converted to label format.\n\n        Returns:\n        - float: Percentage of correct predictions made by the network on the given data.\n        \"\"\"\n        if convert:\n            results = [(np.argmax(self.feedforward(x)), np.argmax(y)) for x, y in data]\n        else:\n            results = [(np.argmax(self.feedforward(x)), y) for x, y in data]\n\n        return sum(int(x == y) for x, y in results) / len(data) * 100\n\n    def cost_derivative(self, output_activations, y):\n        \"\"\"\n        Compute the derivative of the cost function with respect to the output activations.\n\n        Parameters:\n        - output_activations (ndarray): Output activations from the network, shape (output_size, 1).\n        - y (ndarray): True output values, shape (output_size, 1).\n\n        Returns:\n        - ndarray: Vector of derivative of the cost with respect to output activations, shape (output_size, 1).\n        \"\"\"\n        return output_activations - y\n\n\n# Supporting Functions\n\ndef sigmoid(z):\n    \"\"\"\n    Compute the sigmoid function.\n\n    Parameters:\n    - z (ndarray): Input value or array.\n\n    Returns:\n    - ndarray: Sigmoid of the input value or array.\n    \"\"\"\n    return 1.0 / (1.0 + np.exp(-z))\n\n\ndef sigmoid_prime(z):\n    \"\"\"\n    Compute the derivative of the sigmoid function.\n\n    Parameters:\n    - z (ndarray): Input value or array.\n\n    Returns:\n    - ndarray: Derivative of the sigmoid function evaluated at the input value or array.\n    \"\"\"\n    return sigmoid(z) * (1 - sigmoid(z))","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.456017Z","iopub.execute_input":"2024-09-17T11:59:40.456340Z","iopub.status.idle":"2024-09-17T11:59:40.485488Z","shell.execute_reply.started":"2024-09-17T11:59:40.456310Z","shell.execute_reply":"2024-09-17T11:59:40.484331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='6'></a>\n## [6. Classifying Handwritten Digits](#6)\n\nIn the previous chapter, we built our implementation of neural networks from scratch - the `Network` class. Now, we will put this implementation to the test by applying it to a practical task: classifying handwritten digits.\n\n\nIn this final chapter, we will:\n\n1. **Initialize the Network**: Set up the architecture of our neural network to handle digit classification.\n2. **Train the Network**: Use our training data to optimize the network's weights and biases.\n\n3. **Evaluate Performance**: Assess how well the network performs on unseen data and analyze the accuracy of its predictions.\n\n4. **Explore Enhancements**: Identify areas where the model could be improved and consider possible future directions.\n\nBy the end of this chapter, we will have a functioning neural network capable of recognizing handwritten digits, along with insights into its performance and potential improvements.","metadata":{}},{"cell_type":"markdown","source":"<a id='6.1'></a>\n### [6.1. Network Initialization](#6.1)\n\nTo begin, we define the architecture of our neural network, consisting of three layers: 784 input neurons, 100 hidden neurons, and 10 output neurons. Here’s the code to initialize the network:","metadata":{}},{"cell_type":"code","source":"# Initialize the neural network with 784 input neurons, one hidden layer of 100 neurons, and 10 output neurons\nnet = Network([784, 100, 10])","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:40.486943Z","iopub.execute_input":"2024-09-17T11:59:40.487357Z","iopub.status.idle":"2024-09-17T11:59:40.504798Z","shell.execute_reply.started":"2024-09-17T11:59:40.487318Z","shell.execute_reply":"2024-09-17T11:59:40.503586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **Input Layer (784 neurons)**: This layer corresponds to the 784 pixels in each 28x28 image, one neuron per pixel.\n* **Hidden Layer (100 neurons)**: We've chosen 100 neurons for this layer, balancing complexity and performance. This choice isn’t strictly optimal, but it provides enough flexibility for the network to learn and achieve good accuracy (typically above 95%).\n* **Output Layer (10 neurons)**: This layer consists of 10 neurons, each representing one of the digits from 0 to 9.\n\nWhen designing a neural network, there are no fixed rules for determining the number of hidden layers or neurons. This process involves experimentation, intuition, and experience. The architecture we’ve chosen is simple and effective for the task of digit classification, but there’s always room for optimization.\n\nIt’s important to note that adding more hidden neurons increases the model’s complexity, which can improve performance but also lengthen training times. For faster training with only a slight reduction in accuracy (to around 94%-95%), you could reduce the hidden layer size to 30 neurons.","metadata":{}},{"cell_type":"markdown","source":"<a id='6.2'></a>\n### [6.2. Training the Network](#6.2)\n\nNow that we’ve set up the architecture, we can move on to training our network using stochastic gradient descent (SGD). The training process involves evaluating the network’s ability to classify digits by tracking its performance on both the training and validation datasets at each step.\n\nHere’s how the training is structured:\n\n* **Epochs**: The network will train for 30 epochs, meaning the entire training dataset is passed through the network 30 times.\n* **Mini-batch Size**: After every 10 training examples, the network’s weights and biases are updated.\n* **Learning Rate ($\\eta$)**: Set to 3.0, this controls the size of the adjustments made during gradient descent.\n\nFor optimal results, we would normally train the network on the entire training set. However, for demonstration, we're using a subset (the validation data) to track performance.","metadata":{}},{"cell_type":"code","source":"# Train the network using stochastic gradient descent (SGD)\ntraining_acc, validation_acc = net.SGD(\n    training_data=train,\n    epochs=30,\n    mini_batch_size=10,\n    eta=3.0,\n    validation_data=validation,\n    monitor_training_accuracy=True,\n    monitor_validation_accuracy=True,\n)","metadata":{"execution":{"iopub.status.busy":"2024-09-17T11:59:41.666890Z","iopub.execute_input":"2024-09-17T11:59:41.667274Z","iopub.status.idle":"2024-09-17T12:11:15.322500Z","shell.execute_reply.started":"2024-09-17T11:59:41.667245Z","shell.execute_reply":"2024-09-17T12:11:15.321101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since each instance of the network class initializes its weights and biases randomly from a Gaussian distribution, the initial starting point in the parameter space will differ with each run. This means the '*path*' taken during stochastic gradient descent (SGD) to minimize the cost function will vary based on these starting values.\n\n* **Different Starting Points**: Some instances might start far from the global minimum, requiring more epochs to converge. Others may start closer, allowing them to reach the minimum faster.\n* **Local Minima**: The more problematic scenario occurs when the network initializes near a local minimum, where it may get stuck and fail to reach the global minimum.\n\nWith enough epochs, the network should ideally converge to a similar result, regardless of its starting point. However, this is not always guaranteed, and variations between training runs are common due to this randomness. If your monitoring results differ from mine, it's likely due to this initial variation.\n\nTo address this, you can explore more sophisticated initialization techniques or alternative cost functions to ensure more reliable convergence. If your results are significantly off—say, your validation accuracy is well below 90%—you can simply reinitialize the `Network` class and train again for better results.","metadata":{}},{"cell_type":"markdown","source":"<a id='6.3'></a>\n### [6.3. Evaluating Performance](#6.3)\n\nNow that we've trained our network, we can evaluate its performance by comparing the training accuracy and validation accuracy over the 30 epochs. We could simply read off the accuracies from the monitoring above, but it's always helpful to visualize the results with a plot!","metadata":{}},{"cell_type":"code","source":"def plot_metrics(\n    datasets: Optional[List[List[float]]] = None,\n    labels: Optional[List[str]] = None,\n    title: str = \"\",\n    xlabel: str = \"Epochs\",\n    ylabel: str = \"\",\n    is_accuracy: bool = False,\n) -> plt.Figure:\n    \"\"\"\n    Plots multiple datasets on a single graph with optional labeling and customization.\n\n    Parameters:\n    - datasets (Optional[List[List[float]]]): A list containing the datasets to be plotted.\n      Each dataset should be an iterable of numeric values. Must provide at least one dataset.\n    - labels (Optional[List[str]]): A list of labels corresponding to each dataset.\n      The number of labels must match the number of datasets. Defaults to None.\n    - title (str): The title of the plot. Defaults to an empty string.\n    - xlabel (str): The label for the x-axis. Defaults to \"Epochs\".\n    - ylabel (str): The label for the y-axis. If `is_accuracy` is True, this is overridden to \"%\".\n      Defaults to an empty string.\n    - is_accuracy (bool): A flag to indicate if the y-axis represents accuracy. If True,\n      the y-axis is set to a percentage scale (0-100%) and the plot title is set to \"Classification Accuracy\".\n      Defaults to False.\n\n    Returns:\n    - fig (matplotlib.figure.Figure): The matplotlib figure object containing the plot.\n\n    Raises:\n    - ValueError: If no datasets are provided or if the number of labels does not match the number of datasets.\n    \"\"\"\n\n    # Set default values for datasets and labels if they are None\n    datasets = datasets or []\n    labels = labels or []\n\n    # Validate that at least one dataset is provided\n    if not datasets:\n        raise ValueError(\"A dataset must be provided.\")\n\n    # Check if labels are provided and their count matches the number of datasets\n    if labels and len(labels) != len(datasets):\n        raise ValueError(\n            \"The number of labels provided must match the number of datasets provided.\"\n        )\n\n    # Create a new figure for the plot\n    fig = plt.figure()\n\n    # Determine the maximum length of the datasets\n    max_length = max(len(data) for data in datasets)\n    x_values = np.arange(\n        1, max_length + 1\n    )  # X-values representing epochs from 1 to max_length\n\n    # Plot each dataset with its corresponding label and x-values\n    for data, label in zip(datasets, labels):\n        plt.plot(\n            x_values[: len(data)], data, label=label\n        )  # Use x_values to align with epoch numbers\n\n    # Adjust plot settings for accuracy metric if applicable\n    if is_accuracy:\n        plt.title(r\"Classification Accuracy (%)\")\n        plt.ylim(top=100)\n        plt.ylabel(r\"%\")\n    else:\n        plt.title(title)\n        plt.ylabel(ylabel)\n\n    # Set x-axis properties based on xlabel\n    if xlabel == \"Epochs\":\n        plt.xlim(1, max_length)\n\n        # Determine tick interval based on max_length\n        if max_length <= 10:\n            interval = 1\n        elif max_length <= 30:\n            interval = 5\n        elif max_length <= 50:\n            interval = 10\n        elif max_length <= 100:\n            interval = 20\n        elif max_length <= 300:\n            interval = 50\n        elif max_length <= 500:\n            interval = 100\n        elif max_length <= 1000:\n            interval = 200\n        else:\n            interval = 100  # Fallback for larger sizes, adjust if needed\n\n        # Set the x-ticks using the determined interval, starting from 1\n        ticks = np.arange(0, max_length + 1, interval)\n        ticks[0] = 1\n        plt.xticks(ticks)\n\n    plt.xlabel(xlabel)\n    plt.legend()\n\n    # Return the figure object for further manipulation or saving\n    return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-17T12:12:16.001984Z","iopub.execute_input":"2024-09-17T12:12:16.002486Z","iopub.status.idle":"2024-09-17T12:12:16.018891Z","shell.execute_reply.started":"2024-09-17T12:12:16.002447Z","shell.execute_reply":"2024-09-17T12:12:16.017230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot accuracies\naccuracy_plot = plot_metrics(\n    [training_acc, validation_acc],\n    [\"Training Data\", \"Validation Data\"],\n    is_accuracy=True,\n)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-17T12:12:16.672401Z","iopub.execute_input":"2024-09-17T12:12:16.672825Z","iopub.status.idle":"2024-09-17T12:12:17.479046Z","shell.execute_reply.started":"2024-09-17T12:12:16.672792Z","shell.execute_reply":"2024-09-17T12:12:17.477773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Comparing Results\n\nWhile both the training and validation accuracies show steady improvement, there is a noticeable gap between the two. This means the network performs better on the training data than on unseen validation data - an effect called overfitting.\n\n* **Overfitting**: Overfitting occurs when the network becomes too specialized to the training data, learning noise and details that do not generalize well to unseen data.\n\n#### Conclusion\n\nThe network performs well, reaching an accuracy of 98.6% on the training set and 95.9% on the validation set. However, the gap between these two accuracies indicates the model has begun to overfit, meaning it has become too tailored to the training data and may not generalize as well to new, unseen data. To address overfitting, techniques such as early stopping, regularization, or dropout could be applied to improve generalization.\n\nWith the training complete, let’s use the trained model to make predictions on the test set and prepare a submission!","metadata":{}},{"cell_type":"code","source":"# Predict on test data\npredictions = [np.argmax(net.feedforward(x)) for x in test]\n\n# Create a submission dataframe\nsubmission = pd.DataFrame({\n        \"ImageId\": range(1, len(predictions) + 1),\n        \"Label\": predictions\n    })\n\n# Save dataframe as csv file\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head(10)","metadata":{"execution":{"iopub.status.busy":"2024-09-17T12:14:10.791549Z","iopub.execute_input":"2024-09-17T12:14:10.792038Z","iopub.status.idle":"2024-09-17T12:14:12.255780Z","shell.execute_reply.started":"2024-09-17T12:14:10.792002Z","shell.execute_reply":"2024-09-17T12:14:12.254575Z"},"trusted":true},"execution_count":null,"outputs":[]}]}