{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Simple MNIST NN from scratch\n\nIn this notebook, I implemented a simple two-layer neural network and trained it on the MNIST digit recognizer dataset. It's meant to be an instructional example, through which you can understand the underlying math of neural networks better.\n\nHere's a video I made explaining all the math and showing my progress as I coded the network: https://youtu.be/w8yWXqWQYmU","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\ndata = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-06-01T05:26:41.438380Z","iopub.execute_input":"2023-06-01T05:26:41.438829Z","iopub.status.idle":"2023-06-01T05:26:44.908574Z","shell.execute_reply.started":"2023-06-01T05:26:41.438789Z","shell.execute_reply":"2023-06-01T05:26:44.907636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = np.array(data)\nm, n = data.shape\nnp.random.shuffle(data) # shuffle before splitting into dev and training sets\n\ndata_dev = data[0:1000].T\nY_dev = data_dev[0]\nX_dev = data_dev[1:n]\nX_dev = X_dev / 255.\n\ndata_train = data[1000:m].T\nY_train = data_train[0]\nX_train = data_train[1:n]\nX_train = X_train / 255.\n_,m_train = X_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:26:44.911258Z","iopub.execute_input":"2023-06-01T05:26:44.911799Z","iopub.status.idle":"2023-06-01T05:26:45.794545Z","shell.execute_reply.started":"2023-06-01T05:26:44.911745Z","shell.execute_reply":"2023-06-01T05:26:45.793394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"m","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:35:33.869083Z","iopub.execute_input":"2023-06-01T05:35:33.869811Z","iopub.status.idle":"2023-06-01T05:35:33.877358Z","shell.execute_reply.started":"2023-06-01T05:35:33.869750Z","shell.execute_reply":"2023-06-01T05:35:33.875888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_train","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:26:45.797053Z","iopub.execute_input":"2023-06-01T05:26:45.797594Z","iopub.status.idle":"2023-06-01T05:26:45.805434Z","shell.execute_reply.started":"2023-06-01T05:26:45.797508Z","shell.execute_reply":"2023-06-01T05:26:45.804268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our NN will have a simple two-layer architecture. Input layer $a^{[0]}$ will have 784 units corresponding to the 784 pixels in each 28x28 input image. A hidden layer $a^{[1]}$ will have 10 units with ReLU activation, and finally our output layer $a^{[2]}$ will have 10 units corresponding to the ten digit classes with softmax activation.\n\n**Forward propagation**\n\n$$Z^{[1]} = W^{[1]} X + b^{[1]}$$\n$$A^{[1]} = g_{\\text{ReLU}}(Z^{[1]}))$$\n$$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$$\n$$A^{[2]} = g_{\\text{softmax}}(Z^{[2]})$$\n\n**Backward propagation**\n\n$$dZ^{[2]} = A^{[2]} - Y$$\n$$dW^{[2]} = \\frac{1}{m} dZ^{[2]} A^{[1]T}$$\n$$dB^{[2]} = \\frac{1}{m} \\Sigma {dZ^{[2]}}$$\n$$dZ^{[1]} = W^{[2]T} dZ^{[2]} .* g^{[1]\\prime} (z^{[1]})$$\n$$dW^{[1]} = \\frac{1}{m} dZ^{[1]} A^{[0]T}$$\n$$dB^{[1]} = \\frac{1}{m} \\Sigma {dZ^{[1]}}$$\n\n**Parameter updates**\n\n$$W^{[2]} := W^{[2]} - \\alpha dW^{[2]}$$\n$$b^{[2]} := b^{[2]} - \\alpha db^{[2]}$$\n$$W^{[1]} := W^{[1]} - \\alpha dW^{[1]}$$\n$$b^{[1]} := b^{[1]} - \\alpha db^{[1]}$$\n\n**Vars and shapes**\n\nForward prop\n\n- $A^{[0]} = X$: 784 x m\n- $Z^{[1]} \\sim A^{[1]}$: 10 x m\n- $W^{[1]}$: 10 x 784 (as $W^{[1]} A^{[0]} \\sim Z^{[1]}$)\n- $B^{[1]}$: 10 x 1\n- $Z^{[2]} \\sim A^{[2]}$: 10 x m\n- $W^{[1]}$: 10 x 10 (as $W^{[2]} A^{[1]} \\sim Z^{[2]}$)\n- $B^{[2]}$: 10 x 1\n\nBackprop\n\n- $dZ^{[2]}$: 10 x m ($~A^{[2]}$)\n- $dW^{[2]}$: 10 x 10\n- $dB^{[2]}$: 10 x 1\n- $dZ^{[1]}$: 10 x m ($~A^{[1]}$)\n- $dW^{[1]}$: 10 x 10\n- $dB^{[1]}$: 10 x 1","metadata":{}},{"cell_type":"code","source":"def init_params():\n    W1 = np.random.rand(10, 784) - 0.5\n    b1 = np.random.rand(10, 1) - 0.5\n    W2 = np.random.rand(10, 10) - 0.5\n    b2 = np.random.rand(10, 1) - 0.5\n    return W1, b1, W2, b2\n\ndef ReLU(Z):\n    return np.maximum(Z, 0)\n\ndef softmax(Z):\n    A = np.exp(Z) / sum(np.exp(Z))\n    return A\n\ndef deriv_ReLU(Z):\n    return Z > 0","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:29:32.020750Z","iopub.execute_input":"2023-06-01T05:29:32.021368Z","iopub.status.idle":"2023-06-01T05:29:32.033211Z","shell.execute_reply.started":"2023-06-01T05:29:32.021302Z","shell.execute_reply":"2023-06-01T05:29:32.032104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndef forward_prop(W1, b1, W2, b2, X):\n    Z1 = W1.dot(X) + b1\n    A1 = ReLU(Z1)\n    Z2 = W2.dot(A1) + b2\n    A2 = softmax(Z2)\n    return Z1, A1, Z2, A2\n\n# ini apaan anjrt \ndef one_hot(Y):\n    one_hot_Y = np.zeros((Y.size, Y.max() + 1))\n    one_hot_Y[np.arange(Y.size), Y] = 1\n    one_hot_Y = one_hot_Y.T\n    return one_hot_Y\n     \n# Y is the labels \ndef back_prop(Z1, A1, Z2, A2, W1, W2, X, Y):\n    m = Y.size # how many labels we are trying to train\n    one_hot_Y = one_hot(Y)\n    dZ2 = A2 - one_hot_Y\n    dW2 = 1 / m * dZ2.dot(A1.T)\n    db2 = 1 / m * np.sum(dZ2)\n    dZ1 = W2.T.dot(dZ2) * deriv_ReLU(Z1)\n    dW1 = 1 / m * dZ2.dot(X.T)\n    db1 = 1 / m * np.sum(dZ1)\n    return dW1, db1, dW2, db2\n\ndef update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):\n    W1 = W1 - alpha * dW1\n    b1 = b1 - alpha * db1\n    \n    W2 = W2 - alpha * dW2 \n    b2 = b2 - alpha * db2\n    \n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:29:32.209988Z","iopub.execute_input":"2023-06-01T05:29:32.210439Z","iopub.status.idle":"2023-06-01T05:29:32.228443Z","shell.execute_reply.started":"2023-06-01T05:29:32.210401Z","shell.execute_reply":"2023-06-01T05:29:32.227211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_predictions(A2):\n    return np.argmax(A2, 0)\n\ndef get_accuracy(predictions, Y):\n    print(predictions, Y)\n    return np.sum(predictions == Y) / Y.size\n\ndef gradient_descent(X, Y, iterations, alpha):\n    W1, b1, W2, b2 = init_params()\n    \n    for i in range(iterations):\n        Z1, A1, Z2, A2 = forward_prop(W1, b1, W2, b2, X)\n        dW1, db1, dW2, db2 = back_prop(Z1, A1, Z2, A2, W1, W2, X, Y)\n        W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)\n        if i % 10 == 0:\n            print(\"Iteration: \", i)\n            predictions = get_predictions(A2)\n            print(\"Accuracy: \", get_accuracy(predictions, Y))\n    \n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:29:33.080871Z","iopub.execute_input":"2023-06-01T05:29:33.081292Z","iopub.status.idle":"2023-06-01T05:29:33.094274Z","shell.execute_reply.started":"2023-06-01T05:29:33.081250Z","shell.execute_reply":"2023-06-01T05:29:33.093150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"W1, b1, W2, b2 = gradient_descent(X_train, Y_train, 500, 0.5)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:29:34.993694Z","iopub.execute_input":"2023-06-01T05:29:34.994556Z","iopub.status.idle":"2023-06-01T05:30:20.495702Z","shell.execute_reply.started":"2023-06-01T05:29:34.994442Z","shell.execute_reply":"2023-06-01T05:30:20.494449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"code","source":"def init_params():\n    W1 = np.random.rand(10, 784) - 0.5\n    b1 = np.random.rand(10, 1) - 0.5\n    W2 = np.random.rand(10, 10) - 0.5\n    b2 = np.random.rand(10, 1) - 0.5\n    return W1, b1, W2, b2\n\ndef ReLU(Z):\n    return np.maximum(Z, 0)\n\ndef softmax(Z):\n    A = np.exp(Z) / sum(np.exp(Z))\n    return A\n    \ndef forward_prop(W1, b1, W2, b2, X):\n    Z1 = W1.dot(X) + b1\n    A1 = ReLU(Z1)\n    Z2 = W2.dot(A1) + b2\n    A2 = softmax(Z2)\n    return Z1, A1, Z2, A2\n\ndef ReLU_deriv(Z):\n    return Z > 0\n\ndef one_hot(Y):\n    one_hot_Y = np.zeros((Y.size, Y.max() + 1))\n    one_hot_Y[np.arange(Y.size), Y] = 1\n    one_hot_Y = one_hot_Y.T\n    return one_hot_Y\n\ndef backward_prop(Z1, A1, Z2, A2, W1, W2, X, Y):\n    one_hot_Y = one_hot(Y)\n    dZ2 = A2 - one_hot_Y\n    dW2 = 1 / m * dZ2.dot(A1.T)\n    db2 = 1 / m * np.sum(dZ2)\n    dZ1 = W2.T.dot(dZ2) * ReLU_deriv(Z1)\n    dW1 = 1 / m * dZ1.dot(X.T)\n    db1 = 1 / m * np.sum(dZ1)\n    return dW1, db1, dW2, db2\n\ndef update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):\n    W1 = W1 - alpha * dW1\n    b1 = b1 - alpha * db1    \n    W2 = W2 - alpha * dW2  \n    b2 = b2 - alpha * db2    \n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:11:04.632420Z","iopub.execute_input":"2023-06-01T05:11:04.632895Z","iopub.status.idle":"2023-06-01T05:11:04.655342Z","shell.execute_reply.started":"2023-06-01T05:11:04.632853Z","shell.execute_reply":"2023-06-01T05:11:04.653669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_predictions(A2):\n    return np.argmax(A2, 0)\n\ndef get_accuracy(predictions, Y):\n    print(predictions, Y)\n    return np.sum(predictions == Y) / Y.size\n\ndef gradient_descent(X, Y, alpha, iterations):\n    W1, b1, W2, b2 = init_params()\n    for i in range(iterations):\n        Z1, A1, Z2, A2 = forward_prop(W1, b1, W2, b2, X)\n        dW1, db1, dW2, db2 = backward_prop(Z1, A1, Z2, A2, W1, W2, X, Y)\n        W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)\n        if i % 10 == 0:\n            print(\"Iteration: \", i)\n            predictions = get_predictions(A2)\n            print(get_accuracy(predictions, Y))\n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:11:05.503723Z","iopub.execute_input":"2023-06-01T05:11:05.504202Z","iopub.status.idle":"2023-06-01T05:11:05.518875Z","shell.execute_reply.started":"2023-06-01T05:11:05.504162Z","shell.execute_reply":"2023-06-01T05:11:05.517089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"W1, b1, W2, b2 = gradient_descent(X_train, Y_train, 0.10, 500)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:11:09.364073Z","iopub.execute_input":"2023-06-01T05:11:09.364742Z","iopub.status.idle":"2023-06-01T05:11:55.883976Z","shell.execute_reply.started":"2023-06-01T05:11:09.364681Z","shell.execute_reply":"2023-06-01T05:11:55.883017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"~85% accuracy on training set.","metadata":{}},{"cell_type":"code","source":"def make_predictions(X, W1, b1, W2, b2):\n    _, _, _, A2 = forward_prop(W1, b1, W2, b2, X)\n    predictions = get_predictions(A2)\n    return predictions\n\ndef test_prediction(index, W1, b1, W2, b2):\n    current_image = X_train[:, index, None]\n    prediction = make_predictions(X_train[:, index, None], W1, b1, W2, b2)\n    label = Y_train[index]\n    print(\"Prediction: \", prediction)\n    print(\"Label: \", label)\n    \n    current_image = current_image.reshape((28, 28)) * 255\n    plt.gray()\n    plt.imshow(current_image, interpolation='nearest')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:13:40.086073Z","iopub.execute_input":"2023-06-01T05:13:40.086550Z","iopub.status.idle":"2023-06-01T05:13:40.103268Z","shell.execute_reply.started":"2023-06-01T05:13:40.086495Z","shell.execute_reply":"2023-06-01T05:13:40.101659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at a couple of examples:","metadata":{}},{"cell_type":"code","source":"test_prediction(0, W1, b1, W2, b2)\ntest_prediction(1, W1, b1, W2, b2)\ntest_prediction(2, W1, b1, W2, b2)\ntest_prediction(3, W1, b1, W2, b2)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:13:42.280856Z","iopub.execute_input":"2023-06-01T05:13:42.281273Z","iopub.status.idle":"2023-06-01T05:13:43.052093Z","shell.execute_reply.started":"2023-06-01T05:13:42.281238Z","shell.execute_reply":"2023-06-01T05:13:43.050550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's find the accuracy on the dev set:","metadata":{}},{"cell_type":"code","source":"dev_predictions = make_predictions(X_dev, W1, b1, W2, b2)\nget_accuracy(dev_predictions, Y_dev)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:13:58.356492Z","iopub.execute_input":"2023-06-01T05:13:58.356993Z","iopub.status.idle":"2023-06-01T05:13:58.385888Z","shell.execute_reply.started":"2023-06-01T05:13:58.356951Z","shell.execute_reply":"2023-06-01T05:13:58.384148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Still 84% accuracy, so our model generalized from the training data pretty well.","metadata":{}}]}