{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Classifying Images of Digits","metadata":{}},{"cell_type":"markdown","source":"First, let's start by importing some libraries.\n* Numpy: The library for managing arrays fast (in C++, accelerated processing), and using normal math notation to process arrays. \n* Pandas: Built on top of numpy, this helps in getting some more spreadsheet like behavior, managing CSV files, keeping column headings, etc.\n* Matplotlib: Plotting library, normal graphs and showing images, everything.\n* Tensorflow: Our big machine learning library, which has the keras library as a submodule.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom matplotlib import pyplot as plt\n\nimport tensorflow as tf","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:05:48.480334Z","iopub.execute_input":"2021-08-27T09:05:48.480643Z","iopub.status.idle":"2021-08-27T09:05:49.705663Z","shell.execute_reply.started":"2021-08-27T09:05:48.480584Z","shell.execute_reply":"2021-08-27T09:05:49.704952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"Pandas can read CSV files (comma separated values, these are files where there are many rows, and each row has all the columns separated by commas, so like a single sheet in a spreadsheet software). Having read it, it converts it into a format that is easy to make an array out of, and is easy to print and analyze and generally play around with.\n\nYou can see that by using the `df.head()` command, we can see the first 5 columns as an example of what the dataset looks like.","metadata":{}},{"cell_type":"code","source":"data_train = pd.read_csv('../input/train.csv')\ndata_test = pd.read_csv('../input/test.csv')\n\nprint(f\"There are {len(data_train)} example images.\")\ndata_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:05:49.707827Z","iopub.execute_input":"2021-08-27T09:05:49.708249Z","iopub.status.idle":"2021-08-27T09:05:57.513238Z","shell.execute_reply.started":"2021-08-27T09:05:49.708076Z","shell.execute_reply":"2021-08-27T09:05:57.512577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Given the data, it's time to split it into training and validation dataset. We also need to separate the X and y values and store them as arrays. With this, our data preprocessing step is done.\n\nThe `df.drop('column_name', axis=1)` command removes one column, so we will have the remaining columns which are the values of the pixels for the image.\nThe `df['column_name']` just gets that one label since we need to use it as the correct output value.\nThen we do indexing, the first 1000 examples are called the validation examples and all others are used as training.","metadata":{}},{"cell_type":"code","source":"x, y = data_train.drop('label', axis=1), data_train['label']\n\nx_train = x[1000:]\ny_train = y[1000:]\n\nx_validate = x[:1000]\ny_validate = y[:1000]\n\nx_test = data_test","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:07:04.391514Z","iopub.execute_input":"2021-08-27T09:07:04.391838Z","iopub.status.idle":"2021-08-27T09:07:04.558757Z","shell.execute_reply.started":"2021-08-27T09:07:04.391784Z","shell.execute_reply":"2021-08-27T09:07:04.557912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's just see our image, the first image in our training set. We are using `plt.imshow` to convert the array of numbers into an actual digit. Before we show the image, we convert the array of size `(784,)` to an image of size `(28, 28)` so that it actually looks like an image to matplotlib.","metadata":{}},{"cell_type":"code","source":"example_image, example_label = x_train.iloc[0], y_train.iloc[0]\npixels = int(example_image.size ** 0.5)\nplt.imshow(np.reshape(example_image.values, [pixels, pixels]))\nplt.title(f\"Image of the number {example_label}\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:07:39.779036Z","iopub.execute_input":"2021-08-27T09:07:39.779328Z","iopub.status.idle":"2021-08-27T09:07:39.965815Z","shell.execute_reply.started":"2021-08-27T09:07:39.779279Z","shell.execute_reply":"2021-08-27T09:07:39.965060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Defining the Model","metadata":{}},{"cell_type":"markdown","source":"This is how to make the model. Our model is very simple, it just has Dense (aka. Fully Connected) layers. The names are optional, just for our convinience. ReLU is used as activation inside and softmax at the output (since it's like applying a sigmoid to many outputs and then normalizing it, not exactly though), so we have our 10 outputs now.","metadata":{}},{"cell_type":"code","source":"# model = tf.keras.models.Sequential([\n#     tf.keras.layers.InputLayer(input_shape=(pixels * pixels,), name = 'Block0-Input'),\n#     tf.keras.layers.Reshape((pixels, pixels, 1), name = 'Block0-Reshape'),\n#     # Pad, Convolve, Batch Normalize, ReLU\n#     tf.keras.layers.ZeroPadding2D((3, 3), name = 'Block1-Padding'),\n#     tf.keras.layers.Conv2D(32, (7, 7), strides = (1, 1), name = 'Block1-Convolution'),\n#     tf.keras.layers.BatchNormalization(axis = 3, name = 'Block1-Normalization'),\n#     tf.keras.layers.Activation('relu', name = 'Block1-Activation'),\n#     # Convolve, Batch Normalize, ReLU, Pool    \n#     tf.keras.layers.Conv2D(32, (7, 7), strides = (1, 1), name = 'Block2-Convolution'),\n#     tf.keras.layers.BatchNormalization(axis = 3, name = 'Block2-Normalization'),\n#     tf.keras.layers.Activation('relu', name = 'Block2-Activation'),\n#     tf.keras.layers.MaxPooling2D((2, 2), name= 'Block2-MaxPool'),\n#     # Flatten, Fully Connected, Sigmoid\n#     tf.keras.layers.Flatten(name = 'Block3-Flatten'),\n#     tf.keras.layers.Dense(10, activation='softmax', name='Block3-Dense'),\n# ])\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.InputLayer(input_shape=(pixels * pixels,), name = 'Block0-Input'),\n    tf.keras.layers.Dense(1000, activation='relu', name='Block1-Dense'),\n    tf.keras.layers.Dense(2000, activation='relu', name='Block2-Dense'),\n    tf.keras.layers.Dense(1000, activation='relu', name='Block3-Dense'),\n    tf.keras.layers.Dense(500, activation='relu', name='Block4-Dense'),\n    tf.keras.layers.Dense(500, activation='relu', name='Block5-Dense'),\n    tf.keras.layers.Dense(10, activation='softmax', name='Block6-Output'),\n])\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:09:41.700938Z","iopub.execute_input":"2021-08-27T09:09:41.701245Z","iopub.status.idle":"2021-08-27T09:09:41.821628Z","shell.execute_reply.started":"2021-08-27T09:09:41.701192Z","shell.execute_reply":"2021-08-27T09:09:41.820801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We keep **Stochastic Gradient Descent** as out optimizer.\n**Sparse Categorical Entropy** as the loss function, since only one of the 10 logits will turn out to be correct.\nWe want to keep a track of **Percentage Accuracy**, that is just for our benefit, so that we can see it improve.","metadata":{}},{"cell_type":"code","source":"model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:09:54.207520Z","iopub.execute_input":"2021-08-27T09:09:54.207823Z","iopub.status.idle":"2021-08-27T09:09:54.253657Z","shell.execute_reply.started":"2021-08-27T09:09:54.207774Z","shell.execute_reply":"2021-08-27T09:09:54.252942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's actual fit, for 3 epochs (i.e. 3 times repetition) over all the 41000 training images, and keep testing over the 1000 validation images. The backward pass will not happen on the validation images.","metadata":{}},{"cell_type":"code","source":"model.fit(x = x_train, y = y_train, epochs=10, batch_size=20, validation_data=(x_validate, y_validate))","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:10:28.088430Z","iopub.execute_input":"2021-08-27T09:10:28.088766Z","iopub.status.idle":"2021-08-27T09:11:46.261700Z","shell.execute_reply.started":"2021-08-27T09:10:28.088714Z","shell.execute_reply":"2021-08-27T09:11:46.261078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Storing the Answers","metadata":{}},{"cell_type":"markdown","source":"We can just call `model.predict` to get all our predictions. Taking `np.argmax` gives us the value of the digit, since the output of the model was a vector of confidences. Eg. `[0, 0, 0.7, 0.3, 0, 0, 0, 0, 0, 0]` implies that the model thinks that an image has `70%` probability of being a 2 and a `30%` change of being a `3`. The `np.argmax` will return 2, the max probability number.","metadata":{}},{"cell_type":"code","source":"results = model.predict(x_test)\npredictions = np.argmax(results, axis=1)\nprint(x_test.shape, predictions.shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:06:18.519295Z","iopub.execute_input":"2021-08-27T09:06:18.519759Z","iopub.status.idle":"2021-08-27T09:06:19.647138Z","shell.execute_reply.started":"2021-08-27T09:06:18.519530Z","shell.execute_reply":"2021-08-27T09:06:19.646254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's write the answers that we have into the `submissions.csv` file.","metadata":{}},{"cell_type":"code","source":"with open('submission.csv', 'w') as file:\n    file.write('ImageId,Label\\n')\n    for i in range(len(predictions)):\n        file.write(str(i+1) + ',' + str(predictions[i]) + '\\n')","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:06:19.648449Z","iopub.execute_input":"2021-08-27T09:06:19.648933Z","iopub.status.idle":"2021-08-27T09:06:19.717755Z","shell.execute_reply.started":"2021-08-27T09:06:19.648866Z","shell.execute_reply":"2021-08-27T09:06:19.716810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# More Visualizations","metadata":{}},{"cell_type":"markdown","source":"Let's see a few images of the digits our model got wrong.","metadata":{}},{"cell_type":"code","source":"predictions_validate = np.argmax(model.predict(x_validate), axis=1)\nincorrects = np.reshape(np.nonzero(predictions_validate != y_validate), [-1])\nprint(\"You got\", len(incorrects), \"images wrongly classified in Validation.\")\nfigure, axes = plt.subplots(4, 4, figsize=(15, 15))\naxes = np.reshape(axes, (-1))\nfor i in range(min(len(incorrects), 16)):\n    axes[i].imshow(np.reshape(x_validate.iloc[incorrects[i]].values, [pixels, pixels]), cmap='hot')\n    axes[i].set_title(str(y_validate.iloc[incorrects[i]]) + \" was classified as \" + str(predictions_validate[incorrects[i]]))","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:12:21.584488Z","iopub.execute_input":"2021-08-27T09:12:21.584822Z","iopub.status.idle":"2021-08-27T09:12:24.531001Z","shell.execute_reply.started":"2021-08-27T09:12:21.584770Z","shell.execute_reply":"2021-08-27T09:12:24.529970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's see how our model learnt what it did.","metadata":{}},{"cell_type":"code","source":"# Plot training & validation accuracy values\nplt.plot(model.history.history['acc'], color='IndianRed')\nplt.plot(model.history.history['val_acc'], color='RoyalBlue')\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(model.history.history['loss'], color='IndianRed')\nplt.plot(model.history.history['val_loss'], color='RoyalBlue')\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-27T09:16:38.760051Z","iopub.execute_input":"2021-08-27T09:16:38.760363Z","iopub.status.idle":"2021-08-27T09:16:39.224254Z","shell.execute_reply.started":"2021-08-27T09:16:38.760310Z","shell.execute_reply":"2021-08-27T09:16:39.223450Z"},"trusted":true},"execution_count":null,"outputs":[]}]}