{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Introduction \nThis notebook is an implementation of [Neural Network and Deep learning ebooks by Micheal Nelson](http://neuralnetworksanddeeplearning.com/index.html). \n\nFew points on the notebook :-\n1. The dataset used in the book and this one is quite different in size. In the book the training data has 50000 samples. Here we have 42000.\n2. In the book there is a different validation dataset, here we will create a validation dataset from 42000 training dataset provided.\n3. The implementation in the book is in Python 2.7, that's why the code here won't exactly match with the book as I have changed several things to make it compatible for Python 3.\n\n\nFor a more detailed understanding of NN, do read this ebook. and try to understand each function that is written."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import scale, StandardScaler","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Reading the kaggle dataset in our environment.\ndata = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndata.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Writing this command to see which column is the target column.\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Scaling the data.\nOne thing, you will notice here is that the data that is used in the book is already standardized but kaggle data is not. So before doing anything let's scale our predictor space."},{"metadata":{"trusted":true},"cell_type":"code","source":"X = scale(data.drop('label', axis = 1).values)\ny = np.array(data['label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now will create our validation dataset to measure the performance in each epoch. You can select any number of test size here."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train and test split of the data\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 2000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data preprocessing\nNow we will do 2 things.\n1. The algorithm expects us to pass the 10-dimensional vector for the response and not just the label. For e.g if first sample has the lable 3, we will create a vector as [0,0,0,1,0,0,0,0,0,0]. For that we have a function called vectorized result.\n2. Another thing is that we are reshaping the predictor space and then binding the sample with their response vector using zip function. This is just because the algorithm is written that way. In more sophisticated implementation, this is usually not required."},{"metadata":{"trusted":true},"cell_type":"code","source":"def vectorized_result(j):\n    \"\"\"Return a 10-dimensional unit vector with a 1.0 in the jth\n    position and zeroes elsewhere.  This is used to convert a digit\n    (0...9) into a corresponding desired output from the neural\n    network.\"\"\"\n    e = np.zeros((10, 1))\n    e[j] = 1.0\n    return e\n\ntraining_inputs = [np.reshape(x, (784, 1)) for x in X_train]\ntraining_results = [vectorized_result(y) for y in y_train]\ntraining_data = list(zip(training_inputs, training_results))\nvalidation_inputs = [np.reshape(x, (784, 1)) for x in X_test]\nvalidation_data = list(zip(validation_inputs, y_test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below is the implementation of Neural Network from scratch. \n\nThis code is directly copied from the book. Please read the more detailed explaination of the function in book in Chapter 1. I have made few changes as some parts were giving error because of the changes in Python 2 and 3."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nnetwork.py\n~~~~~~~~~~\n\nA module to implement the stochastic gradient descent learning\nalgorithm for a feedforward neural network.  Gradients are calculated\nusing backpropagation.  Note that I have focused on making the code\nsimple, easily readable, and easily modifiable.  It is not optimized,\nand omits many desirable features.\n\"\"\"\n\n#### Libraries\n# Standard library\nimport random\n\n# Third-party libraries\nimport numpy as np\n\nclass Network(object):\n\n    def __init__(self, sizes):\n        \"\"\"The list ``sizes`` contains the number of neurons in the\n        respective layers of the network.  For example, if the list\n        was [2, 3, 1] then it would be a three-layer network, with the\n        first layer containing 2 neurons, the second layer 3 neurons,\n        and the third layer 1 neuron.  The biases and weights for the\n        network are initialized randomly, using a Gaussian\n        distribution with mean 0, and variance 1.  Note that the first\n        layer is assumed to be an input layer, and by convention we\n        won't set any biases for those neurons, since biases are only\n        ever used in computing the outputs from later layers.\"\"\"\n        self.num_layers = len(sizes)\n        self.sizes = sizes\n        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]\n        self.weights = [np.random.randn(y, x)\n                        for x, y in zip(sizes[:-1], sizes[1:])]\n\n    def feedforward(self, a):\n        \"\"\"Return the output of the network if ``a`` is input.\"\"\"\n        for b, w in zip(self.biases, self.weights):\n            a = sigmoid(np.dot(w, a)+b)\n        return a\n\n    def SGD(self, training_data, epochs, mini_batch_size, eta,\n            test_data=None):\n        \"\"\"Train the neural network using mini-batch stochastic\n        gradient descent.  The ``training_data`` is a list of tuples\n        ``(x, y)`` representing the training inputs and the desired\n        outputs.  The other non-optional parameters are\n        self-explanatory.  If ``test_data`` is provided then the\n        network will be evaluated against the test data after each\n        epoch, and partial progress printed out.  This is useful for\n        tracking progress, but slows things down substantially.\"\"\"\n        if test_data: n_test = len(test_data)\n        n = len(training_data)\n        for j in range(epochs):\n            random.shuffle(training_data)\n            mini_batches = [\n                training_data[k:k+mini_batch_size]\n                for k in range(0, n, mini_batch_size)]\n            for mini_batch in mini_batches:\n                self.update_mini_batch(mini_batch, eta)\n            if test_data:\n                print(\"Epoch {0}: {1} / {2}\".format(\n                    j, self.evaluate(test_data), n_test))\n            else:\n                print(\"Epoch {0} complete\".format(j))\n\n    def update_mini_batch(self, mini_batch, eta):\n        \"\"\"Update the network's weights and biases by applying\n        gradient descent using backpropagation to a single mini batch.\n        The ``mini_batch`` is a list of tuples ``(x, y)``, and ``eta``\n        is the learning rate.\"\"\"\n        nabla_b = [np.zeros(b.shape) for b in self.biases]\n        nabla_w = [np.zeros(w.shape) for w in self.weights]\n        for x, y in mini_batch:\n            delta_nabla_b, delta_nabla_w = self.backprop(x, y)\n            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]\n            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]\n        self.weights = [w-(eta/len(mini_batch))*nw\n                        for w, nw in zip(self.weights, nabla_w)]\n        self.biases = [b-(eta/len(mini_batch))*nb\n                       for b, nb in zip(self.biases, nabla_b)]\n\n    def backprop(self, x, y):\n        \"\"\"Return a tuple ``(nabla_b, nabla_w)`` representing the\n        gradient for the cost function C_x.  ``nabla_b`` and\n        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar\n        to ``self.biases`` and ``self.weights``.\"\"\"\n        nabla_b = [np.zeros(b.shape) for b in self.biases]\n        nabla_w = [np.zeros(w.shape) for w in self.weights]\n        # feedforward\n        activation = x\n        activations = [x] # list to store all the activations, layer by layer\n        zs = [] # list to store all the z vectors, layer by layer\n        for b, w in zip(self.biases, self.weights):\n            z = np.dot(w, activation)+b\n            zs.append(z)\n            activation = sigmoid(z)\n            activations.append(activation)\n        # backward pass\n        delta = self.cost_derivative(activations[-1], y) * \\\n            sigmoid_prime(zs[-1])\n        nabla_b[-1] = delta\n        nabla_w[-1] = np.dot(delta, activations[-2].transpose())\n        # Note that the variable l in the loop below is used a little\n        # differently to the notation in Chapter 2 of the book.  Here,\n        # l = 1 means the last layer of neurons, l = 2 is the\n        # second-last layer, and so on.  It's a renumbering of the\n        # scheme in the book, used here to take advantage of the fact\n        # that Python can use negative indices in lists.\n        for l in range(2, self.num_layers):\n            z = zs[-l]\n            sp = sigmoid_prime(z)\n            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp\n            nabla_b[-l] = delta\n            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())\n        return (nabla_b, nabla_w)\n\n    def evaluate(self, test_data):\n        \"\"\"Return the number of test inputs for which the neural\n        network outputs the correct result. Note that the neural\n        network's output is assumed to be the index of whichever\n        neuron in the final layer has the highest activation.\"\"\"\n        test_results = [(np.argmax(self.feedforward(x)), y)\n                        for (x, y) in test_data]\n        return sum(int(x == y) for (x, y) in test_results)\n\n    def cost_derivative(self, output_activations, y):\n        \"\"\"Return the vector of partial derivatives \\partial C_x /\n        \\partial a for the output activations.\"\"\"\n        return (output_activations-y)\n\n#### Miscellaneous functions\ndef sigmoid(z):\n    \"\"\"The sigmoid function.\"\"\"\n    return 1.0/(1.0+np.exp(-z))\n\ndef sigmoid_prime(z):\n    \"\"\"Derivative of the sigmoid function.\"\"\"\n    return sigmoid(z)*(1-sigmoid(z))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net = Network([784,30,10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above corresponds to :-\n1. Input layer with 784 neurons.\n2. 1 hidden layer with 30 neurons.\n3. Output layer with 10 neurons."},{"metadata":{"trusted":true},"cell_type":"code","source":"net.SGD(training_data, 30, 10, 3.0, test_data = validation_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is giving us an accuracy of 93.15% in epoch 21. Please note that scaling is very important, if not done the result will be far worse than this.\n\nWe will try more neurons in the hidden layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"net = Network([784,100,10])\nnet.SGD(training_data, 30, 10, 3.0, test_data = validation_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will now try with the new cost function **Cross Entropy** (Chapter 2) with the same activation function, **sigmoid function.** Cross entropy will address the issue of neuron saturation that causes learning slowdown."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"network2.py\n~~~~~~~~~~~~~~\n\nAn improved version of network.py, implementing the stochastic\ngradient descent learning algorithm for a feedforward neural network.\nImprovements include the addition of the cross-entropy cost function,\nregularization, and better initialization of network weights.  Note\nthat I have focused on making the code simple, easily readable, and\neasily modifiable.  It is not optimized, and omits many desirable\nfeatures.\n\n\"\"\"\n\n#### Libraries\n# Standard library\nimport json\nimport random\nimport sys\n\n# Third-party libraries\nimport numpy as np\n\n\n#### Define the quadratic and cross-entropy cost functions\n\nclass QuadraticCost(object):\n\n    @staticmethod\n    def fn(a, y):\n        \"\"\"Return the cost associated with an output ``a`` and desired output\n        ``y``.\n\n        \"\"\"\n        return 0.5*np.linalg.norm(a-y)**2\n\n    @staticmethod\n    def delta(z, a, y):\n        \"\"\"Return the error delta from the output layer.\"\"\"\n        return (a-y) * sigmoid_prime(z)\n\n\nclass CrossEntropyCost(object):\n\n    @staticmethod\n    def fn(a, y):\n        \"\"\"Return the cost associated with an output ``a`` and desired output\n        ``y``.  Note that np.nan_to_num is used to ensure numerical\n        stability.  In particular, if both ``a`` and ``y`` have a 1.0\n        in the same slot, then the expression (1-y)*np.log(1-a)\n        returns nan.  The np.nan_to_num ensures that that is converted\n        to the correct value (0.0).\n\n        \"\"\"\n        return np.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)))\n\n    @staticmethod\n    def delta(z, a, y):\n        \"\"\"Return the error delta from the output layer.  Note that the\n        parameter ``z`` is not used by the method.  It is included in\n        the method's parameters in order to make the interface\n        consistent with the delta method for other cost classes.\n\n        \"\"\"\n        return (a-y)\n\n\n#### Main Network class\nclass Network(object):\n\n    def __init__(self, sizes, cost=CrossEntropyCost):\n        \"\"\"The list ``sizes`` contains the number of neurons in the respective\n        layers of the network.  For example, if the list was [2, 3, 1]\n        then it would be a three-layer network, with the first layer\n        containing 2 neurons, the second layer 3 neurons, and the\n        third layer 1 neuron.  The biases and weights for the network\n        are initialized randomly, using\n        ``self.default_weight_initializer`` (see docstring for that\n        method).\n\n        \"\"\"\n        self.num_layers = len(sizes)\n        self.sizes = sizes\n        self.default_weight_initializer()\n        self.cost=cost\n\n    def default_weight_initializer(self):\n        \"\"\"Initialize each weight using a Gaussian distribution with mean 0\n        and standard deviation 1 over the square root of the number of\n        weights connecting to the same neuron.  Initialize the biases\n        using a Gaussian distribution with mean 0 and standard\n        deviation 1.\n\n        Note that the first layer is assumed to be an input layer, and\n        by convention we won't set any biases for those neurons, since\n        biases are only ever used in computing the outputs from later\n        layers.\n\n        \"\"\"\n        self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]\n        self.weights = [np.random.randn(y, x)/np.sqrt(x)\n                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]\n\n    def large_weight_initializer(self):\n        \"\"\"Initialize the weights using a Gaussian distribution with mean 0\n        and standard deviation 1.  Initialize the biases using a\n        Gaussian distribution with mean 0 and standard deviation 1.\n\n        Note that the first layer is assumed to be an input layer, and\n        by convention we won't set any biases for those neurons, since\n        biases are only ever used in computing the outputs from later\n        layers.\n\n        This weight and bias initializer uses the same approach as in\n        Chapter 1, and is included for purposes of comparison.  It\n        will usually be better to use the default weight initializer\n        instead.\n\n        \"\"\"\n        self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]\n        self.weights = [np.random.randn(y, x)\n                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]\n\n    def feedforward(self, a):\n        \"\"\"Return the output of the network if ``a`` is input.\"\"\"\n        for b, w in zip(self.biases, self.weights):\n            a = sigmoid(np.dot(w, a)+b)\n        return a\n\n    def SGD(self, training_data, epochs, mini_batch_size, eta,\n            lmbda = 0.0,\n            evaluation_data=None,\n            monitor_evaluation_cost=False,\n            monitor_evaluation_accuracy=False,\n            monitor_training_cost=False,\n            monitor_training_accuracy=False):\n        \"\"\"Train the neural network using mini-batch stochastic gradient\n        descent.  The ``training_data`` is a list of tuples ``(x, y)``\n        representing the training inputs and the desired outputs.  The\n        other non-optional parameters are self-explanatory, as is the\n        regularization parameter ``lmbda``.  The method also accepts\n        ``evaluation_data``, usually either the validation or test\n        data.  We can monitor the cost and accuracy on either the\n        evaluation data or the training data, by setting the\n        appropriate flags.  The method returns a tuple containing four\n        lists: the (per-epoch) costs on the evaluation data, the\n        accuracies on the evaluation data, the costs on the training\n        data, and the accuracies on the training data.  All values are\n        evaluated at the end of each training epoch.  So, for example,\n        if we train for 30 epochs, then the first element of the tuple\n        will be a 30-element list containing the cost on the\n        evaluation data at the end of each epoch. Note that the lists\n        are empty if the corresponding flag is not set.\n\n        \"\"\"\n        if evaluation_data: n_data = len(evaluation_data)\n        n = len(training_data)\n        evaluation_cost, evaluation_accuracy = [], []\n        training_cost, training_accuracy = [], []\n        for j in range(epochs):\n            random.shuffle(training_data)\n            mini_batches = [\n                training_data[k:k+mini_batch_size]\n                for k in range(0, n, mini_batch_size)]\n            for mini_batch in mini_batches:\n                self.update_mini_batch(\n                    mini_batch, eta, lmbda, len(training_data))\n            print(\"Epoch %s training complete\" % j)\n            if monitor_training_cost:\n                cost = self.total_cost(training_data, lmbda)\n                training_cost.append(cost)\n                print(\"Cost on training data: {}\".format(cost))\n            if monitor_training_accuracy:\n                accuracy = self.accuracy(training_data, convert=True)\n                training_accuracy.append(accuracy)\n                print(\"Accuracy on training data: {} / {}\".format(\n                    accuracy, n))\n            if monitor_evaluation_cost:\n                cost = self.total_cost(evaluation_data, lmbda, convert=True)\n                evaluation_cost.append(cost)\n                print(\"Cost on evaluation data: {}\".format(cost))\n            if monitor_evaluation_accuracy:\n                accuracy = self.accuracy(evaluation_data)\n                evaluation_accuracy.append(accuracy)\n                print(\"Accuracy on evaluation data: {} / {}\".format(\n                    self.accuracy(evaluation_data), n_data))\n            print\n        return evaluation_cost, evaluation_accuracy, \\\n            training_cost, training_accuracy\n\n    def update_mini_batch(self, mini_batch, eta, lmbda, n):\n        \"\"\"Update the network's weights and biases by applying gradient\n        descent using backpropagation to a single mini batch.  The\n        ``mini_batch`` is a list of tuples ``(x, y)``, ``eta`` is the\n        learning rate, ``lmbda`` is the regularization parameter, and\n        ``n`` is the total size of the training data set.\n\n        \"\"\"\n        nabla_b = [np.zeros(b.shape) for b in self.biases]\n        nabla_w = [np.zeros(w.shape) for w in self.weights]\n        for x, y in mini_batch:\n            delta_nabla_b, delta_nabla_w = self.backprop(x, y)\n            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]\n            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]\n        self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*nw\n                        for w, nw in zip(self.weights, nabla_w)]\n        self.biases = [b-(eta/len(mini_batch))*nb\n                       for b, nb in zip(self.biases, nabla_b)]\n\n    def backprop(self, x, y):\n        \"\"\"Return a tuple ``(nabla_b, nabla_w)`` representing the\n        gradient for the cost function C_x.  ``nabla_b`` and\n        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar\n        to ``self.biases`` and ``self.weights``.\"\"\"\n        nabla_b = [np.zeros(b.shape) for b in self.biases]\n        nabla_w = [np.zeros(w.shape) for w in self.weights]\n        # feedforward\n        activation = x\n        activations = [x] # list to store all the activations, layer by layer\n        zs = [] # list to store all the z vectors, layer by layer\n        for b, w in zip(self.biases, self.weights):\n            z = np.dot(w, activation)+b\n            zs.append(z)\n            activation = sigmoid(z)\n            activations.append(activation)\n        # backward pass\n        delta = (self.cost).delta(zs[-1], activations[-1], y)\n        nabla_b[-1] = delta\n        nabla_w[-1] = np.dot(delta, activations[-2].transpose())\n        # Note that the variable l in the loop below is used a little\n        # differently to the notation in Chapter 2 of the book.  Here,\n        # l = 1 means the last layer of neurons, l = 2 is the\n        # second-last layer, and so on.  It's a renumbering of the\n        # scheme in the book, used here to take advantage of the fact\n        # that Python can use negative indices in lists.\n        for l in range(2, self.num_layers):\n            z = zs[-l]\n            sp = sigmoid_prime(z)\n            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp\n            nabla_b[-l] = delta\n            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())\n        return (nabla_b, nabla_w)\n\n    def accuracy(self, data, convert=False):\n        \"\"\"Return the number of inputs in ``data`` for which the neural\n        network outputs the correct result. The neural network's\n        output is assumed to be the index of whichever neuron in the\n        final layer has the highest activation.\n\n        The flag ``convert`` should be set to False if the data set is\n        validation or test data (the usual case), and to True if the\n        data set is the training data. The need for this flag arises\n        due to differences in the way the results ``y`` are\n        represented in the different data sets.  In particular, it\n        flags whether we need to convert between the different\n        representations.  It may seem strange to use different\n        representations for the different data sets.  Why not use the\n        same representation for all three data sets?  It's done for\n        efficiency reasons -- the program usually evaluates the cost\n        on the training data and the accuracy on other data sets.\n        These are different types of computations, and using different\n        representations speeds things up.  More details on the\n        representations can be found in\n        mnist_loader.load_data_wrapper.\n\n        \"\"\"\n        if convert:\n            results = [(np.argmax(self.feedforward(x)), np.argmax(y))\n                       for (x, y) in data]\n        else:\n            results = [(np.argmax(self.feedforward(x)), y)\n                        for (x, y) in data]\n        return sum(int(x == y) for (x, y) in results)\n\n    def total_cost(self, data, lmbda, convert=False):\n        \"\"\"Return the total cost for the data set ``data``.  The flag\n        ``convert`` should be set to False if the data set is the\n        training data (the usual case), and to True if the data set is\n        the validation or test data.  See comments on the similar (but\n        reversed) convention for the ``accuracy`` method, above.\n        \"\"\"\n        cost = 0.0\n        for x, y in data:\n            a = self.feedforward(x)\n            if convert: y = vectorized_result(y)\n            cost += self.cost.fn(a, y)/len(data)\n        cost += 0.5*(lmbda/len(data))*sum(\n            np.linalg.norm(w)**2 for w in self.weights)\n        return cost\n\n    def save(self, filename):\n        \"\"\"Save the neural network to the file ``filename``.\"\"\"\n        data = {\"sizes\": self.sizes,\n                \"weights\": [w.tolist() for w in self.weights],\n                \"biases\": [b.tolist() for b in self.biases],\n                \"cost\": str(self.cost.__name__)}\n        f = open(filename, \"w\")\n        json.dump(data, f)\n        f.close()\n\n#### Loading a Network\ndef load(filename):\n    \"\"\"Load a neural network from the file ``filename``.  Returns an\n    instance of Network.\n\n    \"\"\"\n    f = open(filename, \"r\")\n    data = json.load(f)\n    f.close()\n    cost = getattr(sys.modules[__name__], data[\"cost\"])\n    net = Network(data[\"sizes\"], cost=cost)\n    net.weights = [np.array(w) for w in data[\"weights\"]]\n    net.biases = [np.array(b) for b in data[\"biases\"]]\n    return net\n\n#### Miscellaneous functions\ndef vectorized_result(j):\n    \"\"\"Return a 10-dimensional unit vector with a 1.0 in the j'th position\n    and zeroes elsewhere.  This is used to convert a digit (0...9)\n    into a corresponding desired output from the neural network.\n\n    \"\"\"\n    e = np.zeros((10, 1))\n    e[j] = 1.0\n    return e\n\ndef sigmoid(z):\n    \"\"\"The sigmoid function.\"\"\"\n    return 1.0/(1.0+np.exp(-z))\n\ndef sigmoid_prime(z):\n    \"\"\"Derivative of the sigmoid function.\"\"\"\n    return sigmoid(z)*(1-sigmoid(z))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net = Network([784,30,10], cost = CrossEntropyCost)\nnet.large_weight_initializer()\nnet.SGD(training_data, 30, 10, 0.5, evaluation_data = validation_data, monitor_evaluation_accuracy=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Very little improvement using cross entropy 93.2%. Let's try with 100 neurons."},{"metadata":{"trusted":true},"cell_type":"code","source":"net = Network([784,100,10], cost = CrossEntropyCost)\nnet.large_weight_initializer()\nnet.SGD(training_data, 30, 10, 0.5, evaluation_data = validation_data, monitor_evaluation_accuracy=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This time we have much better results, with 94.1% accuracy in epoch 13."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}