{"cells":[{"metadata":{},"cell_type":"markdown","source":"A neural network written in Numpy validated as a XOR gate, Pandas 8x8 MNIST Binary Digit classifier and 28x28 MNIST 10 Class Digit classifier.\n\n\nResources Used\n\nhttp://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf\n(backward step)\n\nhttps://gist.github.com/tsbertalan/3288114\n(learning rate, splitting delta & update steps, xor)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nfrom math import tanh","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Datasets for Supervised Learning\n\nA dataset for supervised learning consists of many rows of information, each paired with an expected output for the predictor.\n\nThe predictor, neural network in this case, will iterate over each row of information and generate its own predicted value for each one. This value will then be compared with the expected value(given by the dataset) and the network will update accordinly in attempt to minimize wrong predictions.\n\nIn order to see how well the predictor does in the end, a set of data will be needed to evaluate it. It is important that the test set data does not contain samples that the network has already seen as theese may bias the results. Normally a dataset is split into 2 parts, 80% of the data as a training set & 20% for the test set.\n\nWhen training a neural network on images, ie the MNIST Digits dataset, images are commonly flattened before being given to the predictor."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.datasets import load_iris\nimport pandas as pd\n\n# Load dataset to dataframe\ndata = load_iris()\nX = pd.DataFrame(data.data, columns=data.feature_names)\nX['expected_label'] = data.target\n\n# Shuffle rows\nX = X.reindex(np.random.permutation(X.index))\n\nX.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Activation Functions\n\nThe output of each neuron is passed through an activation function before being pushed to the next layer or output.\n\nThese make the output of each layers of neurons have nonlinear properties. \nWithout activation functions, there would be little(no?) value in adding more layers to the network. Each layer would be a linear transform, so a network without activation functions would just be a series of linear transforms, and a series of linear transforms could be represented by a single one."},{"metadata":{"trusted":true},"cell_type":"code","source":"@np.vectorize\ndef sigmoid(x):\n    return tanh(x)\n\n@np.vectorize\ndef sigmoid_prime(x):\n    return 1.0 - x**2\n\n\nLEAK = 0\n@np.vectorize\ndef relu(x):\n    return x if x > 0 else (LEAK * x)\n\n@np.vectorize\ndef relu_prime(x):\n    return 1 if x > 0 else LEAK\n\n\nactivation_map = {\n    'sigmoid': {\"f\": sigmoid, \"f '\": sigmoid_prime},\n    'relu': {\"f\": relu, \"f '\": relu_prime},\n    }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Visualize Activation Functions\nimport matplotlib.pyplot as plt\n\nfor activation_type in activation_map:\n    X = np.arange(-4, 4, .1)\n    \n    for key, value in activation_map[activation_type].items():\n        Y = [value(v) for v in X]\n        \n        plt.plot(X, Y, label=key)\n\n    plt.title(activation_type)\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cost Functions\nA cost function is the way to measure how far the output of the predictor is away from the desired value."},{"metadata":{"trusted":true},"cell_type":"code","source":"class MeanSquaredError:\n    \"\"\"\n    Mean squared error -- for regression problems.\n    \"\"\"\n    def __call__(seself, real, target):\n        return (1 / len(real)) * np.sum((target - real)**2)\n\n    def derivative(self, real, target):\n        return 2 / len(real) * (real - target)\n\n\nclass CrossEntropyLoss:\n    \"\"\"\n    Cross Entropy loss -- for categorical problems.\n    \"\"\"\n    def __call__(self, real, target):\n        return -real[np.where(target)] + np.log(np.sum(np.exp(real)))\n\n    def derivative(self, real, target):\n        return (1 / len(real)) * (real - target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Neural Networks\n\nThe goal of a neural network is to approximate an arbitrary function. For example, we can use a neural network to approximate the function that 28x28 grayscale images of handwritten numbers to a number 0-9 -- the MNIST digit recognition task."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class NN:\n    def __init__(self, layers, layer_activations, loss_prime, learning_rate=.5):\n        self.layers = layers\n        self.layer_activations = layer_activations\n        self.learning_rate = learning_rate\n        self.loss_prime = loss_prime\n\n        assert len(self.layer_activations) == len(self.layers) - 1, \"Number activations incorrect.\"\n\n        self.w = []\n        for i, layer in enumerate(self.layers[:-1]):\n            # w[in, out]\n            matrix = np.random.uniform(-.1, .1, size=(layer, self.layers[i+1]))\n\n            self.w.append(matrix)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Feedforward Network\n\nMost networks trained w/ gradient descent have a feedforward structure. This means the network is organized as a series of layers, with the output of every neuron in layer i being passed into every neuron in layer i+1.\n\nThe leftmost layer of neurons are inputs, with their activation set to whatever the input values are. The information from the initial layer is pushed through the corresponding synapses into the next layer, and so on to the next layer until the final layer has been updated. \n\nFor categorical tasks, the output layer is usually interpreted by measuring which output neuron has the highest activation, ie argmax(output_fires). If a neural network's task was to say whether an image had a 0 or a 1 (a cat or a dog, an iris or a rose, ...) in it, the output layer would have 2 neurons, whichever had the highest activation for a given input would correspond to the networks guess."},{"metadata":{"trusted":true},"cell_type":"code","source":"def forward(self, x):\n    \"\"\"\n    Network estimate y given x.\n    \"\"\"\n    fires = [np.copy(x)]\n\n    for i in range(len(self.layers) - 1):\n        x = np.matmul(fires[-1], self.w[i])\n\n        fires.append(activation_map[self.layer_activations[i]]['f'](x))\n\n    return fires[-1], fires\n\nNN.forward = forward","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning\n\nThe learning in a neural network is done by updating the weights of the synapses in an intelligent manner in attempt to increase the accuracy of the network.\n\nFor this, I have implemented the stochastic gradient descent algorithm to update weights.\n\nGradient descent is moving along a curve in a way to find a trough based on the curves derivatives in every direction(via directional derivative). In this scenario, the goal is to move along the curve of the cost function - f(weight_1, weight_2, .... target) -> error in order to minimize the wrong predictions of the network. This curve has as many dimensions as there are weights in the network.\n\nStochastic GD is an approximation of gradient descent when not all gradient info is known. In this network, every prediction and corresponding prediction error is used to evaluate the gradient at a certain point -- with the same network, different input states may generate different guesses of the gradient. Mini batch stochastic gradient descent is a good way to alleviate this issue but is not implemented here.\n\nBackpropogation pushes the error of the output layer backwards through the network in order to update the weights of each layer based on what synapses are thought to have contributed most to the incorrect prediction."},{"metadata":{"trusted":true},"cell_type":"code","source":"def backward(self, real, target, fires):\n    \"\"\"\n    Update weights according to directional derivative to minimize error.\n    \"\"\"\n    ## Error for output layer\n    error = self.loss_prime(fires[-1], target)\n    \n    delta = activation_map[self.layer_activations[-1]][\"f '\"](fires[-1]) * error\n\n    deltas = [delta]\n\n    ## Backpropogate error\n    for i in range(len(self.layers) - 3, -1, -1):\n        error = np.sum(deltas[0] * self.w[i+1], axis=1)\n\n        delta = activation_map[self.layer_activations[i]][\"f '\"](fires[i+1]) * error\n\n        deltas.insert(0, delta)\n\n    for i in range(len(self.layers) - 2, -1, -1):\n        self.w[i] -= self.learning_rate * deltas[i] * fires[i].reshape((-1, 1))\n\nNN.backward = backward","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Onehot Encoding\n\nAssign one expected value to each output of the neural network.\n\nie w/ 3 classes\n\n0 -> [1 0 0]\n\n1 -> [0 1 0]\n\n2 -> [0 0 1]"},{"metadata":{"trusted":true},"cell_type":"code","source":"def onehot(value, n_class):\n    output = np.zeros(n_class)\n\n    output[value] = 1.\n\n    return output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### XOR Gate"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Setup Dataset\n# ([in1, in2], expected)\ndata = [([0, 0], [0]), ([0, 1], [1]), ([1, 0], [1]), ([1, 1], [0])]\ndata = np.array(data)\n\n# shuffle and split info and labels\nidx = np.random.randint(0, 4, size=4000)\nX, y = data[idx, 0], data[idx, 1]\n\n## Setup NN\ncost = MeanSquaredError()\nnetwork = NN([2, 2, 1], ['sigmoid', 'sigmoid'], cost.derivative)\n\n## Train\nerror = 0\nfor i, expected in enumerate(y):\n    real, fires = network.forward(X[i])\n\n    network.backward(real, expected, fires)\n\n    error += (1 / 2) * (expected - real)**2\n\n    if i % 400 == 399:\n        print(error)\n        error = 0\n\n## Evaluate\nX, y = data[:, 0], data[:, 1]\nfor i, expected in enumerate(y):\n    real, fires = network.forward(X[i])\n    print(X[i], '->', real)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Binary Digit classification"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Read Dataset\nN_CLASS = 2\nEPOCH = 10\n\nimport sklearn.datasets\nX, y = sklearn.datasets.load_digits(n_class=N_CLASS, return_X_y=True)\n\n## Setup\ncost = CrossEntropyLoss()\nnetwork = NN([64, N_CLASS], ['sigmoid'], cost.derivative, 10**-3)\n\n## Train\nerror = 0\nfor e in range(EPOCH):\n    # shuffle dataset between epoch\n    idx = [i for i in range(len(y))]\n    np.random.shuffle(idx)\n    X = X[idx]\n    y = y[idx]\n    \n    for i, expected in enumerate(y):\n        real, fires = network.forward(X[i])\n\n        target = onehot(expected, N_CLASS)\n        network.backward(real, target, fires)\n\n        error += cost(real, target)\n\n    if not e % 1:\n        print(error)\n        error = 0\n\n## Evaluate\nconfusion = {}\naccuracy = 0\nfor i, expected in enumerate(y):\n    real, fires = network.forward(X[i])\n\n    guess = np.argmax(real)\n    if expected not in confusion:\n        confusion[expected] = {}\n    if guess not in confusion[expected]:\n        confusion[expected][guess] = 0\n    confusion[expected][guess] += 1\n\n    accuracy += int(guess == expected)\n\nprint(f\"Accuracy: {accuracy / len(y)}\")\nprint(confusion)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Full MNIST"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Read Dataset\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndataset = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ndataset.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X, y = dataset[dataset.columns[1:]].values, dataset['label'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Split dataset into train and test set\nsplit_idx = int(y.size * .8)\n\ntrain_X, train_y = X[:split_idx], y[:split_idx]\ntest_X, test_y = X[split_idx:], y[split_idx:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Setup NN\nN_CLASS = 10\nEPOCH = 10\n\ncost = CrossEntropyLoss()\n\nnetwork = NN([784, 32, N_CLASS], ['sigmoid', 'sigmoid'], cost.derivative, 10**-4)\n\n## Train\nerror = 0\nfor e in range(EPOCH):\n    # shuffle dataset between epoch\n    idx = [i for i in range(len(train_y))]\n    np.random.shuffle(idx)\n    train_X = train_X[idx]\n    train_y = train_y[idx]\n\n    for i, expected in enumerate(train_y):\n        real, fires = network.forward(train_X[i])\n\n        target = onehot(expected, N_CLASS)\n        network.backward(real, target, fires)\n\n        error += cost(real, target)\n\n    if not e % 1:\n        print(error)\n        error = 0\n\n        \n## Evaluate\nn_correct = 0\nfor i, expected in enumerate(test_y):\n    real, fires = network.forward(test_X[i])\n\n    n_correct += np.argmax(real) == expected\n\nprint(f\"Correct: {n_correct / test_y.size:.2f}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}