{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"duration":55.320944,"end_time":"2020-11-24T15:34:35.258832","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2020-11-24T15:33:39.937888","version":"2.1.0"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":31154,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Simple MNIST NN\n\nIn this notebook, I implemented a simple two-layer neural network and trained it on the MNIST digit recognizer dataset. It's meant to be an instructional example, through which you can understand the underlying math of neural networks better.\n","metadata":{"papermill":{"duration":0.012295,"end_time":"2020-11-24T15:33:44.585418","exception":false,"start_time":"2020-11-24T15:33:44.573123","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\ndata = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2025-10-09T22:56:03.930879Z","iopub.execute_input":"2025-10-09T22:56:03.931129Z","iopub.status.idle":"2025-10-09T22:56:08.583289Z","shell.execute_reply.started":"2025-10-09T22:56:03.931100Z","shell.execute_reply":"2025-10-09T22:56:08.582141Z"},"papermill":{"duration":3.947576,"end_time":"2020-11-24T15:33:48.546057","exception":false,"start_time":"2020-11-24T15:33:44.598481","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data = np.array(data)\nm, n = data.shape\nnp.random.shuffle(data) # shuffle before splitting into dev and training sets\n\ndata_dev = data[0:1000].T\nY_dev = data_dev[0]\nX_dev = data_dev[1:n]\nX_dev = X_dev / 255.\n\ndata_train = data[1000:m].T\nY_train = data_train[0]\nX_train = data_train[1:n]\nX_train = X_train / 255.\n_,m_train = X_train.shape","metadata":{"execution":{"iopub.status.busy":"2025-10-09T22:56:16.426618Z","iopub.execute_input":"2025-10-09T22:56:16.427560Z","iopub.status.idle":"2025-10-09T22:56:17.351514Z","shell.execute_reply.started":"2025-10-09T22:56:16.427520Z","shell.execute_reply":"2025-10-09T22:56:17.350457Z"},"papermill":{"duration":0.795759,"end_time":"2020-11-24T15:33:49.353603","exception":false,"start_time":"2020-11-24T15:33:48.557844","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Y_train","metadata":{"execution":{"iopub.status.busy":"2025-10-09T22:56:26.468533Z","iopub.execute_input":"2025-10-09T22:56:26.468831Z","iopub.status.idle":"2025-10-09T22:56:26.475545Z","shell.execute_reply.started":"2025-10-09T22:56:26.468812Z","shell.execute_reply":"2025-10-09T22:56:26.474788Z"},"papermill":{"duration":0.023743,"end_time":"2020-11-24T15:33:49.389054","exception":false,"start_time":"2020-11-24T15:33:49.365311","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Our NN will have a simple two-layer architecture. Input layer $a^{[0]}$ will have 784 units corresponding to the 784 pixels in each 28x28 input image. A hidden layer $a^{[1]}$ will have 10 units with ReLU activation, and finally our output layer $a^{[2]}$ will have 10 units corresponding to the ten digit classes with softmax activation.\n\n**Forward propagation**\n\n$$Z^{[1]} = W^{[1]} X + b^{[1]}$$\n$$A^{[1]} = g_{\\text{ReLU}}(Z^{[1]}))$$\n$$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$$\n$$A^{[2]} = g_{\\text{softmax}}(Z^{[2]})$$\n\n**Backward propagation**\n\n$$dZ^{[2]} = A^{[2]} - Y$$\n$$dW^{[2]} = \\frac{1}{m} dZ^{[2]} A^{[1]T}$$\n$$dB^{[2]} = \\frac{1}{m} \\Sigma {dZ^{[2]}}$$\n$$dZ^{[1]} = W^{[2]T} dZ^{[2]} .* g^{[1]\\prime} (z^{[1]})$$\n$$dW^{[1]} = \\frac{1}{m} dZ^{[1]} A^{[0]T}$$\n$$dB^{[1]} = \\frac{1}{m} \\Sigma {dZ^{[1]}}$$\n\n**Parameter updates**\n\n$$W^{[2]} := W^{[2]} - \\alpha dW^{[2]}$$\n$$b^{[2]} := b^{[2]} - \\alpha db^{[2]}$$\n$$W^{[1]} := W^{[1]} - \\alpha dW^{[1]}$$\n$$b^{[1]} := b^{[1]} - \\alpha db^{[1]}$$\n\n**Vars and shapes**\n\nForward prop\n\n- $A^{[0]} = X$: 784 x m\n- $Z^{[1]} \\sim A^{[1]}$: 10 x m\n- $W^{[1]}$: 10 x 784 (as $W^{[1]} A^{[0]} \\sim Z^{[1]}$)\n- $B^{[1]}$: 10 x 1\n- $Z^{[2]} \\sim A^{[2]}$: 10 x m\n- $W^{[1]}$: 10 x 10 (as $W^{[2]} A^{[1]} \\sim Z^{[2]}$)\n- $B^{[2]}$: 10 x 1\n\nBackprop\n\n- $dZ^{[2]}$: 10 x m ($~A^{[2]}$)\n- $dW^{[2]}$: 10 x 10\n- $dB^{[2]}$: 10 x 1\n- $dZ^{[1]}$: 10 x m ($~A^{[1]}$)\n- $dW^{[1]}$: 10 x 10\n- $dB^{[1]}$: 10 x 1","metadata":{"papermill":{"duration":0.011724,"end_time":"2020-11-24T15:33:49.413261","exception":false,"start_time":"2020-11-24T15:33:49.401537","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def init_params():\n    W1 = np.random.rand(10, 784) - 0.5\n    b1 = np.random.rand(10, 1) - 0.5\n    W2 = np.random.rand(10, 10) - 0.5\n    b2 = np.random.rand(10, 1) - 0.5\n    return W1, b1, W2, b2\n\ndef ReLU(Z):\n    return np.maximum(Z, 0)\n\ndef softmax(Z):\n    A = np.exp(Z) / sum(np.exp(Z))\n    return A\n    \ndef forward_prop(W1, b1, W2, b2, X):\n    Z1 = W1.dot(X) + b1\n    A1 = ReLU(Z1)\n    Z2 = W2.dot(A1) + b2\n    A2 = softmax(Z2)\n    return Z1, A1, Z2, A2\n\ndef ReLU_deriv(Z):\n    return Z > 0\n\ndef one_hot(Y):\n    one_hot_Y = np.zeros((Y.size, Y.max() + 1))\n    one_hot_Y[np.arange(Y.size), Y] = 1\n    one_hot_Y = one_hot_Y.T\n    return one_hot_Y\n\ndef backward_prop(Z1, A1, Z2, A2, W1, W2, X, Y):\n    one_hot_Y = one_hot(Y)\n    dZ2 = A2 - one_hot_Y\n    dW2 = 1 / m * dZ2.dot(A1.T)\n    db2 = 1 / m * np.sum(dZ2)\n    dZ1 = W2.T.dot(dZ2) * ReLU_deriv(Z1)\n    dW1 = 1 / m * dZ1.dot(X.T)\n    db1 = 1 / m * np.sum(dZ1)\n    return dW1, db1, dW2, db2\n\ndef update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):\n    W1 = W1 - alpha * dW1\n    b1 = b1 - alpha * db1    \n    W2 = W2 - alpha * dW2  \n    b2 = b2 - alpha * db2    \n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2025-10-09T22:56:34.881744Z","iopub.execute_input":"2025-10-09T22:56:34.882066Z","iopub.status.idle":"2025-10-09T22:56:34.892468Z","shell.execute_reply.started":"2025-10-09T22:56:34.882043Z","shell.execute_reply":"2025-10-09T22:56:34.891406Z"},"papermill":{"duration":0.035762,"end_time":"2020-11-24T15:33:49.460906","exception":false,"start_time":"2020-11-24T15:33:49.425144","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_predictions(A2):\n    return np.argmax(A2, 0)\n\ndef get_accuracy(predictions, Y):\n    print(predictions, Y)\n    return np.sum(predictions == Y) / Y.size\n\ndef gradient_descent(X, Y, alpha, iterations):\n    W1, b1, W2, b2 = init_params()\n    for i in range(iterations):\n        Z1, A1, Z2, A2 = forward_prop(W1, b1, W2, b2, X)\n        dW1, db1, dW2, db2 = backward_prop(Z1, A1, Z2, A2, W1, W2, X, Y)\n        W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)\n        if i % 10 == 0:\n            print(\"Iteration: \", i)\n            predictions = get_predictions(A2)\n            print(get_accuracy(predictions, Y))\n    return W1, b1, W2, b2","metadata":{"execution":{"iopub.status.busy":"2025-10-09T22:56:42.355029Z","iopub.execute_input":"2025-10-09T22:56:42.355401Z","iopub.status.idle":"2025-10-09T22:56:42.365822Z","shell.execute_reply.started":"2025-10-09T22:56:42.355377Z","shell.execute_reply":"2025-10-09T22:56:42.364338Z"},"papermill":{"duration":0.026744,"end_time":"2020-11-24T15:33:49.499906","exception":false,"start_time":"2020-11-24T15:33:49.473162","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Experiment with different learning rates\nW1, b1, W2, b2 = gradient_descent(X_train, Y_train, 0.2, 2000)","metadata":{"execution":{"iopub.status.busy":"2025-10-09T23:00:34.750434Z","iopub.execute_input":"2025-10-09T23:00:34.750839Z","iopub.status.idle":"2025-10-09T23:03:06.147974Z","shell.execute_reply.started":"2025-10-09T23:00:34.750812Z","shell.execute_reply":"2025-10-09T23:03:06.146781Z"},"papermill":{"duration":44.403186,"end_time":"2020-11-24T15:34:33.915380","exception":false,"start_time":"2020-11-24T15:33:49.512194","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"~90% accuracy on training set.","metadata":{"papermill":{"duration":0.03222,"end_time":"2020-11-24T15:34:33.980634","exception":false,"start_time":"2020-11-24T15:34:33.948414","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def make_predictions(X, W1, b1, W2, b2):\n    _, _, _, A2 = forward_prop(W1, b1, W2, b2, X)\n    predictions = get_predictions(A2)\n    return predictions\n\ndef test_prediction(index, W1, b1, W2, b2):\n    current_image = X_train[:, index, None]\n    prediction = make_predictions(X_train[:, index, None], W1, b1, W2, b2)\n    label = Y_train[index]\n    print(\"Prediction: \", prediction)\n    print(\"Label: \", label)\n    \n    current_image = current_image.reshape((28, 28)) * 255\n    plt.gray()\n    plt.imshow(current_image, interpolation='nearest')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2025-10-09T23:03:51.585043Z","iopub.execute_input":"2025-10-09T23:03:51.586106Z","iopub.status.idle":"2025-10-09T23:03:51.593010Z","shell.execute_reply.started":"2025-10-09T23:03:51.586043Z","shell.execute_reply":"2025-10-09T23:03:51.592053Z"},"papermill":{"duration":0.048197,"end_time":"2020-11-24T15:34:34.061656","exception":false,"start_time":"2020-11-24T15:34:34.013459","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's look at a couple of examples:","metadata":{"papermill":{"duration":0.032382,"end_time":"2020-11-24T15:34:34.126664","exception":false,"start_time":"2020-11-24T15:34:34.094282","status":"completed"},"tags":[]}},{"cell_type":"code","source":"test_prediction(0, W1, b1, W2, b2)\ntest_prediction(1, W1, b1, W2, b2)\ntest_prediction(2, W1, b1, W2, b2)\ntest_prediction(4, W1, b1, W2, b2)\ntest_prediction(5, W1, b1, W2, b2)\ntest_prediction(6, W1, b1, W2, b2)","metadata":{"execution":{"iopub.status.busy":"2025-10-09T22:59:35.321972Z","iopub.execute_input":"2025-10-09T22:59:35.322799Z","iopub.status.idle":"2025-10-09T22:59:36.104225Z","shell.execute_reply.started":"2025-10-09T22:59:35.322761Z","shell.execute_reply":"2025-10-09T22:59:36.103395Z"},"papermill":{"duration":0.699008,"end_time":"2020-11-24T15:34:34.858375","exception":false,"start_time":"2020-11-24T15:34:34.159367","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Find all indices in Y_train where the label is 1\nindices_of_1s = np.where(Y_train == 5)[0]\n\n# Get the first index from the list of 1s\nindex_to_test = indices_of_1s[0]\n\n# Use that index to make a prediction\nprint(f\"Testing the image at index {index_to_test}, which is a '1'.\")\ntest_prediction(index_to_test, W1, b1, W2, b2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-09T23:07:01.723426Z","iopub.execute_input":"2025-10-09T23:07:01.723774Z","iopub.status.idle":"2025-10-09T23:07:01.976618Z","shell.execute_reply.started":"2025-10-09T23:07:01.723747Z","shell.execute_reply":"2025-10-09T23:07:01.975542Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def show_hidden_layer_activations(index, W1, b1, W2, b2):\n    \"\"\"\n    Shows an input image and a bar chart of the activations in the first hidden layer.\n    \"\"\"\n    # Select the image and its true label\n    current_image = X_train[:, index, None]\n    label = Y_train[index]\n    \n    # Perform the first step of forward propagation to get hidden layer activations\n    Z1 = W1.dot(current_image) + b1\n    A1 = ReLU(Z1) # A1 is the array of activations from our hidden layer\n    \n    # --- Plotting ---\n    fig, axes = plt.subplots(1, 2, figsize=(12, 5))\n    \n    # Plot the input image\n    axes[0].imshow(current_image.reshape((28, 28)) * 255, cmap='gray')\n    axes[0].set_title(f\"Input Image (Label: {label})\")\n    \n    # Plot the hidden layer activations as a bar chart\n    axes[1].bar(range(len(A1)), A1.flatten())\n    axes[1].set_title(\"Hidden Layer Activations\")\n    axes[1].set_xlabel(\"Neuron Index\")\n    axes[1].set_ylabel(\"Activation Value\")\n    \n    plt.tight_layout()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-09T23:09:02.436633Z","iopub.execute_input":"2025-10-09T23:09:02.437002Z","iopub.status.idle":"2025-10-09T23:09:02.447114Z","shell.execute_reply.started":"2025-10-09T23:09:02.436972Z","shell.execute_reply":"2025-10-09T23:09:02.445853Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# First, find the index of a '7'\nindex_of_a_7 = np.where(Y_train == 7)[0][0]\n\n# Now, visualize its activations\nshow_hidden_layer_activations(index_of_a_7, W1, b1, W2, b2)\n\n# Try it with another digit, like a '4'\nindex_of_a_4 = np.where(Y_train == 4)[0][0]\nshow_hidden_layer_activations(index_of_a_4, W1, b1, W2, b2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-09T23:09:17.548208Z","iopub.execute_input":"2025-10-09T23:09:17.548552Z","iopub.status.idle":"2025-10-09T23:09:18.254047Z","shell.execute_reply.started":"2025-10-09T23:09:17.548527Z","shell.execute_reply":"2025-10-09T23:09:18.253139Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Finally, let's find the accuracy on the dev set:","metadata":{"papermill":{"duration":0.037333,"end_time":"2020-11-24T15:34:34.933186","exception":false,"start_time":"2020-11-24T15:34:34.895853","status":"completed"},"tags":[]}},{"cell_type":"code","source":"dev_predictions = make_predictions(X_dev, W1, b1, W2, b2)\nget_accuracy(dev_predictions, Y_dev)","metadata":{"execution":{"iopub.status.busy":"2025-10-09T23:09:55.524154Z","iopub.execute_input":"2025-10-09T23:09:55.524530Z","iopub.status.idle":"2025-10-09T23:09:55.536532Z","shell.execute_reply.started":"2025-10-09T23:09:55.524505Z","shell.execute_reply":"2025-10-09T23:09:55.535720Z"},"papermill":{"duration":0.062985,"end_time":"2020-11-24T15:34:35.033457","exception":false,"start_time":"2020-11-24T15:34:34.970472","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Still aprox 92% accuracy, so our model generalized from the training data pretty well.","metadata":{"papermill":{"duration":0.038287,"end_time":"2020-11-24T15:34:35.112466","exception":false,"start_time":"2020-11-24T15:34:35.074179","status":"completed"},"tags":[]}}]}