{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"provenance":[]},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Implementating Neural Network from scratch using Numpy and Maths","metadata":{}},{"cell_type":"markdown","source":"\n## A simple two-layer neural network trained on the MNIST digit recognizer dataset.\n#### Dataset Link - https://www.kaggle.com/competitions/digit-recognizer/data\n\nThe NN has a simple two-layer architecture. Input layer $a^{[0]}$ has 784 units corresponding to the 784 pixels in each 28x28 input image. A hidden layer $a^{[1]}$ has 10 units with ReLU activation, and finally the output layer $a^{[2]}$ has 10 units corresponding to the ten digit classes with softmax activation.\n\n\n**Forward propagation**\n\n$$Z^{[1]} = W^{[1]} X + b^{[1]}$$\n\n$$A^{[1]} = g_{\\text{ReLU}}(Z^{[1]}))$$\n\n$$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$$\n\n$$A^{[2]} = g_{\\text{softmax}}(Z^{[2]})$$\n\n**Backward propagation**\n\n$$dZ^{[2]} = A^{[2]} - Y$$\n\n$$dW^{[2]} = \\frac{1}{m} dZ^{[2]} A^{[1]T}$$\n\n$$dB^{[2]} = \\frac{1}{m} \\Sigma {dZ^{[2]}}$$\n\n$$dZ^{[1]} = W^{[2]T} dZ^{[2]} .* g^{[1]\\prime} (z^{[1]})$$\n\n$$dW^{[1]} = \\frac{1}{m} dZ^{[1]} A^{[0]T}$$\n\n$$dB^{[1]} = \\frac{1}{m} \\Sigma {dZ^{[1]}}$$\n\n**Parameter updates**\n\n$$W^{[2]} := W^{[2]} - \\alpha dW^{[2]}$$\n\n$$b^{[2]} := b^{[2]} - \\alpha db^{[2]}$$\n\n$$W^{[1]} := W^{[1]} - \\alpha dW^{[1]}$$\n\n$$b^{[1]} := b^{[1]} - \\alpha db^{[1]}$$\n\n**Vars and shapes**\n\nForward prop\n\n- $A^{[0]} = X$: 784 x m\n- $Z^{[1]} \\sim A^{[1]}$: 10 x m\n- $W^{[1]}$: 10 x 784 (as $W^{[1]} A^{[0]} \\sim Z^{[1]}$)\n- $B^{[1]}$: 10 x 1\n- $Z^{[2]} \\sim A^{[2]}$: 10 x m\n- $W^{[1]}$: 10 x 10 (as $W^{[2]} A^{[1]} \\sim Z^{[2]}$)\n- $B^{[2]}$: 10 x 1\n\nBackprop\n\n- $dZ^{[2]}$: 10 x m ($~A^{[2]}$)\n- $dW^{[2]}$: 10 x 10\n- $dB^{[2]}$: 10 x 1\n- $dZ^{[1]}$: 10 x m ($~A^{[1]}$)\n- $dW^{[1]}$: 10 x 10\n- $dB^{[1]}$: 10 x 1\n","metadata":{}},{"cell_type":"markdown","source":"## Importing libraries and Dataset","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt","metadata":{"id":"6SnYBc4sv9mw","execution":{"iopub.status.busy":"2024-02-02T09:22:10.063935Z","iopub.execute_input":"2024-02-02T09:22:10.064448Z","iopub.status.idle":"2024-02-02T09:22:10.522070Z","shell.execute_reply.started":"2024-02-02T09:22:10.064392Z","shell.execute_reply":"2024-02-02T09:22:10.521206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.read_csv(\"https://raw.githubusercontent.com/kunalarora0930/neural-network/main/data/train.csv\")","metadata":{"id":"t9sMR8Oav9mx","execution":{"iopub.status.busy":"2024-02-02T09:22:10.525019Z","iopub.execute_input":"2024-02-02T09:22:10.525590Z","iopub.status.idle":"2024-02-02T09:22:16.065400Z","shell.execute_reply.started":"2024-02-02T09:22:10.525551Z","shell.execute_reply":"2024-02-02T09:22:16.064141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.head(12)","metadata":{"id":"0SB5GFQXv9my","outputId":"17c23790-ea24-48ad-94a0-5d0387d32857","execution":{"iopub.status.busy":"2024-02-02T09:22:16.066853Z","iopub.execute_input":"2024-02-02T09:22:16.067223Z","iopub.status.idle":"2024-02-02T09:22:16.096379Z","shell.execute_reply.started":"2024-02-02T09:22:16.067192Z","shell.execute_reply":"2024-02-02T09:22:16.095254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing Data","metadata":{}},{"cell_type":"code","source":"data = np.array(data)\nprint(data)\nprint(data.shape)\nm,n = data.shape\nnp.random.shuffle(data)\n\ndata_dev = data[0:1000].T\nY_dev = data_dev[0]\nX_dev = data_dev[1:n]\nX_dev = X_dev/255.\n\ndata_train = data[1000:m].T\nY_train = data_train[0]\nX_train = data_train[1:n]\nX_train = X_train/255.","metadata":{"id":"H3VxbL0bv9my","outputId":"11dfc80f-569d-4610-8be7-2f44849a546a","execution":{"iopub.status.busy":"2024-02-02T09:22:16.099282Z","iopub.execute_input":"2024-02-02T09:22:16.100242Z","iopub.status.idle":"2024-02-02T09:22:16.960544Z","shell.execute_reply.started":"2024-02-02T09:22:16.100203Z","shell.execute_reply":"2024-02-02T09:22:16.959516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_train","metadata":{"id":"3TefnTjVv9mz","outputId":"cf87879f-781d-40da-f2bb-38c8339bb735","execution":{"iopub.status.busy":"2024-02-02T09:22:16.961866Z","iopub.execute_input":"2024-02-02T09:22:16.962250Z","iopub.status.idle":"2024-02-02T09:22:16.970615Z","shell.execute_reply.started":"2024-02-02T09:22:16.962215Z","shell.execute_reply":"2024-02-02T09:22:16.969316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Implementing the Network","metadata":{}},{"cell_type":"markdown","source":"### Define Functions","metadata":{}},{"cell_type":"code","source":"def init_params():\n    W1 = np.random.rand(10,784) - 0.5\n    b1 = np.random.rand(10,1) - 0.5\n    W2 = np.random.rand(10,10) - 0.5\n    b2 = np.random.rand(10,1) - 0.5\n    return W1, b1, W2, b2\n\ndef ReLU(Z):\n    return np.maximum(0,Z)\n\ndef softmax(Z):\n    A = np.exp(Z) / sum(np.exp(Z))\n    return (A)\n\ndef one_hot(Y):\n    one_hot_Y = np.zeros((Y.size, Y.max() + 1))\n    one_hot_Y[np.arange(Y.size), Y] = 1\n    one_hot_Y = one_hot_Y.T\n    return one_hot_Y\n\ndef forward_propagation(W1, b1, W2, b2, X):\n    Z1 = W1.dot(X) + b1\n    A1 = ReLU(Z1)\n    Z2 = W2.dot(A1) + b2\n    A2 = softmax(Z2)\n    return Z1, A1, Z2, A2\n\ndef deriv_ReLU(Z):\n    return Z > 0\n\ndef back_propagation(Z1, A1, Z2, A2, W1, W2, X, Y):\n    one_hot_Y = one_hot(Y)\n    dZ2 = A2 - one_hot_Y\n    dW2 = 1 / m * dZ2.dot(A1.T)\n    db2 = 1 / m * np.sum(dZ2)\n    dZ1 = W2.T.dot(dZ2) * deriv_ReLU(Z1)\n    dW1 = 1 / m * dZ1.dot(X.T)\n    db1 = 1 / m * np.sum(dZ1)\n    return dW1, db1, dW2, db2\n\n\ndef update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):\n    W1 = W1 - alpha * dW1\n    b1 = b1 - alpha * db1\n    W2 = W2 - alpha * dW2\n    b2 = b2 - alpha * db2\n\n    return W1, b1, W2, b2\n\n\n","metadata":{"id":"GXwxdKHDv9mz","execution":{"iopub.status.busy":"2024-02-02T09:22:16.972206Z","iopub.execute_input":"2024-02-02T09:22:16.972554Z","iopub.status.idle":"2024-02-02T09:22:16.988364Z","shell.execute_reply.started":"2024-02-02T09:22:16.972521Z","shell.execute_reply":"2024-02-02T09:22:16.987160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Functions","metadata":{}},{"cell_type":"code","source":"def get_predictions(A2):\n    return np.argmax(A2, 0)\n\ndef get_accuracy(predictions, Y):\n    print(predictions, Y)\n    return (np.sum(predictions == Y)/Y.size)\n\ndef gradient_descent(X, Y, itertions, alpha):\n    W1, b1, W2, b2 = init_params()\n    for i in range(itertions):\n        Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X)\n        dW1, db1, dW2, db2 = back_propagation(Z1, A1, Z2, A2, W1, W2, X, Y)\n        W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)\n        if (i%50 == 0):\n            print(\"iteration \" , i , \":\")\n            print(\"Accuracy: \" , get_accuracy(get_predictions(A2), Y))\n    return W1, b1, W2, b2\n","metadata":{"id":"hyJn96RBv9mz","execution":{"iopub.status.busy":"2024-02-02T09:22:16.989779Z","iopub.execute_input":"2024-02-02T09:22:16.990122Z","iopub.status.idle":"2024-02-02T09:22:17.000510Z","shell.execute_reply.started":"2024-02-02T09:22:16.990095Z","shell.execute_reply":"2024-02-02T09:22:16.999591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training The Model","metadata":{}},{"cell_type":"code","source":"W1, b1, W2, b2 = gradient_descent(X_train, Y_train, 5000, 0.05)","metadata":{"id":"teeXm86jv9m0","outputId":"af807117-6513-4679-d08f-d480002a8a38","scrolled":true,"execution":{"iopub.status.busy":"2024-02-02T09:22:17.001883Z","iopub.execute_input":"2024-02-02T09:22:17.002584Z","iopub.status.idle":"2024-02-02T09:32:59.218302Z","shell.execute_reply.started":"2024-02-02T09:22:17.002553Z","shell.execute_reply":"2024-02-02T09:32:59.216719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define testing function","metadata":{}},{"cell_type":"code","source":"def make_predictions(X, W1, b1, W2, b2):\n    _, _, _, A2 = forward_propagation(W1, b1, W2, b2, X)\n    predictions = get_predictions(A2)\n    return predictions\n\ndef test_prediction(index, W1, b1, W2, b2):\n    current_image = X_train[:, index, None]\n    prediction = make_predictions(X_train[:, index, None], W1, b1, W2, b2)\n    label = Y_train[index]\n    print(\"Prediction: \", prediction)\n    print(\"Label: \", label)\n\n    current_image = current_image.reshape((28, 28)) * 255\n    plt.gray()\n    plt.imshow(current_image, interpolation='nearest')\n    plt.show()","metadata":{"id":"To3fuOTOv9m0","execution":{"iopub.status.busy":"2024-02-02T09:32:59.220862Z","iopub.execute_input":"2024-02-02T09:32:59.222031Z","iopub.status.idle":"2024-02-02T09:32:59.235608Z","shell.execute_reply.started":"2024-02-02T09:32:59.221974Z","shell.execute_reply":"2024-02-02T09:32:59.234202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test predictions by taking a random sample from dataset","metadata":{}},{"cell_type":"code","source":"test_prediction(5, W1, b1, W2, b2)","metadata":{"id":"BbKerJeev9m0","outputId":"32dcd520-078b-4193-8bd4-948fb0de628b","execution":{"iopub.status.busy":"2024-02-02T09:32:59.240879Z","iopub.execute_input":"2024-02-02T09:32:59.242003Z","iopub.status.idle":"2024-02-02T09:32:59.490598Z","shell.execute_reply.started":"2024-02-02T09:32:59.241948Z","shell.execute_reply":"2024-02-02T09:32:59.489531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dev_predictions = make_predictions(X_dev, W1, b1, W2, b2)\nget_accuracy(dev_predictions, Y_dev)\n","metadata":{"id":"80WrS4Xwv9m0","outputId":"bf8737bf-6c31-475c-9e27-7784a0b4a71f","scrolled":true,"execution":{"iopub.status.busy":"2024-02-02T09:32:59.491965Z","iopub.execute_input":"2024-02-02T09:32:59.492345Z","iopub.status.idle":"2024-02-02T09:32:59.529602Z","shell.execute_reply.started":"2024-02-02T09:32:59.492310Z","shell.execute_reply":"2024-02-02T09:32:59.525219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Accuracy of model: 90.3%","metadata":{"id":"Qi1--wSLzvgW"}}]}