{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Simple MNIST NN from scratch\n\nIn this notebook, I implemented a simple two-layer neural network and trained it on the MNIST digit recognizer dataset. It's meant to be an instructional example, through which you can understand the underlying math of neural networks better.\n\nHere's a video which I referred for coding this : https://youtu.be/w8yWXqWQYmU","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-05-13T12:04:13.715120Z","iopub.execute_input":"2023-05-13T12:04:13.715573Z","iopub.status.idle":"2023-05-13T12:04:13.720541Z","shell.execute_reply.started":"2023-05-13T12:04:13.715528Z","shell.execute_reply":"2023-05-13T12:04:13.719478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:14.124037Z","iopub.execute_input":"2023-05-13T12:04:14.124400Z","iopub.status.idle":"2023-05-13T12:04:16.773843Z","shell.execute_reply.started":"2023-05-13T12:04:14.124371Z","shell.execute_reply":"2023-05-13T12:04:16.772869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:16.776256Z","iopub.execute_input":"2023-05-13T12:04:16.776724Z","iopub.status.idle":"2023-05-13T12:04:16.797869Z","shell.execute_reply.started":"2023-05-13T12:04:16.776673Z","shell.execute_reply":"2023-05-13T12:04:16.796829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = np.array(data)\nm, n = data.shape\nnp.random.shuffle(data)\n\ndata_dev = data[0:1000].T\nY_dev = data_dev[0]\nX_dev = data_dev[1:n]\nX_dev = X_dev/255\n\ndata_train = data[1000: m].T\nY_train = data_dev[0]\nX_train = data_dev[1:n]\nX_train = X_train / 255\n_,m_train = X_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:17.911129Z","iopub.execute_input":"2023-05-13T12:04:17.911482Z","iopub.status.idle":"2023-05-13T12:04:18.579864Z","shell.execute_reply.started":"2023-05-13T12:04:17.911452Z","shell.execute_reply":"2023-05-13T12:04:18.578625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_train","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:19.140377Z","iopub.execute_input":"2023-05-13T12:04:19.140782Z","iopub.status.idle":"2023-05-13T12:04:19.150330Z","shell.execute_reply.started":"2023-05-13T12:04:19.140746Z","shell.execute_reply":"2023-05-13T12:04:19.149051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our NN will have a simple two-layer architecture. Input layer $a^{[0]}$ will have 784 units corresponding to the 784 pixels in each 28x28 input image. A hidden layer $a^{[1]}$ will have 10 units with ReLU activation, and finally our output layer $a^{[2]}$ will have 10 units corresponding to the ten digit classes with softmax activation.\n\n**Forward propagation**\n\n$$Z^{[1]} = W^{[1]} X + b^{[1]}$$\n$$A^{[1]} = g_{\\text{ReLU}}(Z^{[1]}))$$\n$$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$$\n$$A^{[2]} = g_{\\text{softmax}}(Z^{[2]})$$\n\n**Backward propagation**\n\n$$dZ^{[2]} = A^{[2]} - Y$$\n$$dW^{[2]} = \\frac{1}{m} dZ^{[2]} A^{[1]T}$$\n$$dB^{[2]} = \\frac{1}{m} \\Sigma {dZ^{[2]}}$$\n$$dZ^{[1]} = W^{[2]T} dZ^{[2]} .* g^{[1]\\prime} (z^{[1]})$$\n$$dW^{[1]} = \\frac{1}{m} dZ^{[1]} A^{[0]T}$$\n$$dB^{[1]} = \\frac{1}{m} \\Sigma {dZ^{[1]}}$$\n\n**Parameter updates**\n\n$$W^{[2]} := W^{[2]} - \\alpha dW^{[2]}$$\n$$b^{[2]} := b^{[2]} - \\alpha db^{[2]}$$\n$$W^{[1]} := W^{[1]} - \\alpha dW^{[1]}$$\n$$b^{[1]} := b^{[1]} - \\alpha db^{[1]}$$\n\n**Vars and shapes**\n\nForward prop\n\n- $A^{[0]} = X$: 784 x m\n- $Z^{[1]} \\sim A^{[1]}$: 10 x m\n- $W^{[1]}$: 10 x 784 (as $W^{[1]} A^{[0]} \\sim Z^{[1]}$)\n- $B^{[1]}$: 10 x 1\n- $Z^{[2]} \\sim A^{[2]}$: 10 x m\n- $W^{[1]}$: 10 x 10 (as $W^{[2]} A^{[1]} \\sim Z^{[2]}$)\n- $B^{[2]}$: 10 x 1\n\nBackprop\n\n- $dZ^{[2]}$: 10 x m ($~A^{[2]}$)\n- $dW^{[2]}$: 10 x 10\n- $dB^{[2]}$: 10 x 1\n- $dZ^{[1]}$: 10 x m ($~A^{[1]}$)\n- $dW^{[1]}$: 10 x 10\n- $dB^{[1]}$: 10 x 1","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:24:02.689104Z","iopub.execute_input":"2023-05-12T10:24:02.689644Z","iopub.status.idle":"2023-05-12T10:24:02.714765Z","shell.execute_reply.started":"2023-05-12T10:24:02.689602Z","shell.execute_reply":"2023-05-12T10:24:02.713201Z"}}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"def init_params():\n    W1 = np.random.rand(10,784) - 0.5\n    b1 = np.random.rand(10,1) - 0.5\n    W2 = np.random.rand(10,10) - 0.5\n    b2 = np.random.rand(10,1) - 0.5\n    return W1,b1, W2, b2\n\ndef ReLU(Z):\n    return np.maximum(Z,0)\n\ndef softmax(Z):\n    return np.exp(Z) / sum(np.exp(Z))\n\ndef forward_prop(W1, b1, W2, b2, X):\n    Z1 = W1.dot(X) + b1\n    A1 = ReLU(Z1)\n    Z2 = W2.dot(A1) + b2\n    A2 = softmax(Z2)\n    return Z1, A1, Z2, A2\n\ndef ReLU_deriv(Z):\n    return Z>0\n\ndef one_hot(Y):\n    one_hot_Y = np.zeros((Y.size, Y.max()+1))\n    one_hot_Y[np.arange(Y.size), Y] = 1\n    return one_hot_Y.T\n\ndef backward_prop(Z1, A1, Z2, A2, W1, W2, X, Y):\n    m = Y.size\n    one_hot_Y = one_hot(Y)\n    dZ2 = A2 - one_hot_Y\n    dW2 = 1 / m * dZ2.dot(A1.T) \n    db2 = 1 / m * np.sum(dZ2)\n    dZ1 = W2.T.dot(dZ2) * ReLU_deriv(Z1)\n    dW1 = 1/m * dZ1.dot(X.T)\n    db1 = 1/m * np.sum(dZ1)\n    return dW1, db1, dW2, db2\n\ndef update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):\n    W1 = W1 - alpha * dW1\n    b1 = b1 - alpha * db1\n    W2 = W2 - alpha * dW2\n    b2 = b2 - alpha * db2\n    \n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:33.272907Z","iopub.execute_input":"2023-05-13T12:04:33.273307Z","iopub.status.idle":"2023-05-13T12:04:33.290382Z","shell.execute_reply.started":"2023-05-13T12:04:33.273275Z","shell.execute_reply":"2023-05-13T12:04:33.288809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_predictions(A2):\n    return np.argmax(A2, 0)\n\ndef get_accuracy(predictions, Y):\n    print(predictions, Y)\n    return np.sum(predictions==Y) / Y.size * 100\n\ndef gradient_descent(X, Y, alpha, iterations):\n    W1, b1, W2, b2 = init_params()\n    for i in range(iterations):\n        Z1, A1, Z2, A2 = forward_prop(W1, b1, W2, b2, X)\n        dW1, db1,dW2, db2 = backward_prop(Z1, A1, Z2, A2,W1, W2, X, Y)\n        W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)\n        \n        if(i%5==0):\n            print('Iteration : ',i)\n            print('Accuracy : ', get_accuracy(get_predictions(A2),Y), \"%\")\n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:34.425966Z","iopub.execute_input":"2023-05-13T12:04:34.426334Z","iopub.status.idle":"2023-05-13T12:04:34.437009Z","shell.execute_reply.started":"2023-05-13T12:04:34.426302Z","shell.execute_reply":"2023-05-13T12:04:34.435821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"W1, b1, W2, b2 = gradient_descent(X_train, Y_train,0.10 , 1000)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:35.109508Z","iopub.execute_input":"2023-05-13T12:04:35.109913Z","iopub.status.idle":"2023-05-13T12:04:37.988837Z","shell.execute_reply.started":"2023-05-13T12:04:35.109879Z","shell.execute_reply":"2023-05-13T12:04:37.987636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"~94% accuracy on training set.","metadata":{}},{"cell_type":"code","source":"def make_predictions(X, W1, b1, W2, b2):\n    _, _, _, A2 = forward_prop(W1, b1, W2, b2, X)\n    predictions = get_predictions(A2)\n    return predictions\n\ndef test_prediction(index, W1, b1, W2, b2):\n    current_image = X_train[:, index, None]\n    prediction = make_predictions(X_train[:, index, None], W1, b1, W2, b2)\n    label = Y_train[index]\n    print(\"Prediction: \", prediction)\n    print(\"Label: \", label)\n    \n    current_image = current_image.reshape((28, 28)) * 255\n    plt.gray()\n    plt.imshow(current_image, interpolation='nearest')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:52.852994Z","iopub.execute_input":"2023-05-13T12:04:52.853364Z","iopub.status.idle":"2023-05-13T12:04:52.862385Z","shell.execute_reply.started":"2023-05-13T12:04:52.853329Z","shell.execute_reply":"2023-05-13T12:04:52.861080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at a couple of examples:","metadata":{}},{"cell_type":"code","source":"test_prediction(0, W1, b1, W2, b2)\ntest_prediction(1, W1, b1, W2, b2)\ntest_prediction(2, W1, b1, W2, b2)\ntest_prediction(9, W1, b1, W2, b2)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:56.206912Z","iopub.execute_input":"2023-05-13T12:04:56.207255Z","iopub.status.idle":"2023-05-13T12:04:56.790029Z","shell.execute_reply.started":"2023-05-13T12:04:56.207226Z","shell.execute_reply":"2023-05-13T12:04:56.788704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's find the accuracy on the dev set:","metadata":{}},{"cell_type":"code","source":"dev_predictions = make_predictions(X_dev, W1, b1, W2, b2)\nget_accuracy(dev_predictions, Y_dev)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:04:57.941347Z","iopub.execute_input":"2023-05-13T12:04:57.941754Z","iopub.status.idle":"2023-05-13T12:04:57.958543Z","shell.execute_reply.started":"2023-05-13T12:04:57.941721Z","shell.execute_reply":"2023-05-13T12:04:57.957762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Still 94% accuracy, so our model generalized from the training data pretty well.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}