{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST digit recognition - plain network in Keras\n\nStart here if you're new to computer vision and want to get started in Keras. In this kernel we will use Keras to build a simple, fully connected network to handle the MINST handwritten digit recognition problem, achieving around 95% accuracy on the test set.\n\n[This post](https://anarthal.github.io/kernel/posts/neural-networks-keras/) uses some lines from this kernel to explain Keras basics in more depth.\n\nPlease upvote if you found it useful! Any comments or feedback are welcome."},{"metadata":{},"cell_type":"markdown","source":"# 1. Imports\n\nNote that modern Keras versions are part of Tensorflow, so we import `keras` from the `tensorflow` module."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\n\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Exploratory data analysis\n\nRecall that the MNIST dataset consists of a set of 28x28 grayscale images containing handwritten digits, from zero to nine.\n\nLet's load the data and have it a look!"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"dftrain = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndftest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dftrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dftest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dftrain['pixel256'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To make things easy for us, we are given images in CSV format. The `label` column is the ground-truth: `0` if the digit is actually a zero, `1` if it's a one, and so on. The rest of the columns, `pixelxxx`, represent the intensity of each image pixel, as a number from 0 to 255."},{"metadata":{},"cell_type":"markdown","source":"## 2.1. Visualizing the digits\n\nLet's represent some of the images to see how they look like. The small number above them represents the ground-true label. I've taken this representation technique from\n[this notebook](https://www.kaggle.com/allunia/how-to-attack-a-machine-learning-model):"},{"metadata":{"trusted":true},"cell_type":"code","source":"_, ax = plt.subplots(5, 5, figsize=(8, 8))\nax = ax.flatten()\nfor a, (_, pixels), label in zip(ax, dftrain.filter(regex='pixel.*').iterrows(), dftrain['label']):\n    a.imshow(pixels.values.reshape((28,28)), cmap=\"gray_r\")\n    a.axis('off')\n    a.set_title(label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2. Checking for NaNs\n\nHow many train and test examples do we have? Do they have any NaNs?"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Train size: {dftrain.shape[0]}')\nprint(f'Test size:  {dftest.shape[0]}')\nprint(f'NaNs in train: {dftrain.isna().sum().any()}')\nprint(f'NaNs in test:  {dftest.isna().sum().any()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3. Digit distribution\n\nFrom the following figure, we can see that the digit distribution is not uniform, but also not very skewed. We shouldn't have imbalanced dataset problems."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nsns.countplot(data=dftrain, x='label');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Preprocessing\n\nThere is not that much to preprocess. However, labels are numeric (i.e. a number 0 to 9). This is not the best representation for our network because it implies an ordering. The digits are independent of each other: predicting a 5 is not less than predicting a 9. They are just separate entities.\n\nFor this reason we will transform the numerical labels into a one-hot encoding scheme:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = dftrain.drop(columns='label').values\ny_train = keras.utils.to_categorical(dftrain['label'].values)\nX_test = dftest.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Modelling\n## 4.1. Model definition\n\nWe are going to create a very simple fully-connected neural network, with three hidden layers. As a picture is worth a thousand words, here is what we want to build:\n\n<img src=\"https://anarthal.github.io/kernel/assets/img/neural-networks-keras/network-architecture.png\" width=\"500\"/>"},{"metadata":{},"cell_type":"markdown","source":"We will feed the pixel intensities directly to the neural network, as if they were input features. As the images are not big, this is a feasible technique (for bigger images, a CNN approach would work better). The output layer has 10 units because there are 10 possible classes in our classification problem (there are 10 digits from 0 to 9).\n\nWe use Keras' Sequential API because our model is just a sequence of connected layers. `X_train.shape[1]` gives us the number of input features. Note that we don't specify how many training examples we have. We have also added some regularization to prevent overfitting. Refer to [the post](https://anarthal.github.io/kernel/posts/neural-networks-keras/#defining-the-architecture) for further details."},{"metadata":{"trusted":true},"cell_type":"code","source":"lambda_ = 1e-2\nmodel = keras.Sequential([\n    keras.Input(shape=(X_train.shape[1],)),\n    keras.layers.Dense(20, activation='relu', name='l1', kernel_regularizer=keras.regularizers.l2(lambda_)),\n    keras.layers.Dense(15, activation='relu', name='l2', kernel_regularizer=keras.regularizers.l2(lambda_)),\n    keras.layers.Dense(10, activation='softmax', name='output', kernel_regularizer=keras.regularizers.l2(lambda_))\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2. Compiling the model\n\nThe second step is to compile the model, where we specify:\n\n- The Adam optimizer (similar to gradient descent) as the method to find the best weights for our training set.\n- The categorical crossentropy as the loss function (the default loss function for multiclass classification problems).\n- Accuracy as a metric to be tracked while the network is being trained.\n\nRefer to [this section](https://anarthal.github.io/kernel/posts/neural-networks-keras/#compiling-the-model) of the post for further details."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    optimizer=keras.optimizers.Adam(),\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3. Fitting the model\n\nAlright, time for the heavy lifting! By calling `fit` we ask Keras to train the network and thus find the parameters that work best for our training set. By passing `validation_split=0.2` we are asking Keras to split the passed-in data into a training set and a validation set. The training set will contain 80% of the examples, with the other 20% reserved for validation. This way we can have unbiased estimates of our performance easily.\n\nWe will train the model for 50 epochs, meaning it will perform 50 iterations over the entire training set. For each of these epochs, Keras will record the value for the loss function and the requested accuracy metric, both for the training and validation sets. You will see it when you hit shift-enter. This will take a while!"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    X_train, \n    y_train, \n    validation_split=0.2, \n    batch_size=128, \n    epochs=50\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4. The history object\n\nThe `fit()` function returns a `history` object, which contains the tracked values for the loss function and the accuracy for every epoch. The values are stored in the `history.history` attribute, which is a Python `dict`. Values are recorded for both the training set (`loss` and `accuracy`) and the validation set (`val_loss` and `val_accuracy`):"},{"metadata":{"trusted":true},"cell_type":"code","source":"loss = history.history['loss']\nval_loss = history.history['val_loss']\nacc = history.history['accuracy']\nval_acc = history.history['val_accuracy']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can use these measures to visualize the training process in time. In the plots below we can see that accuracy increases rapidly at first, and then flattens in a plateau. We can conclude that training the network for more epochs wouldn't yield a significant improvement. We can also see that train and validation metrics are quite close, so the model is not overfitting the training set.\n\nI've skipped the values for the first epoch because it distorts the graphs. The loss is very high and the accuracy very low because the network has been trained for a very little period of time."},{"metadata":{"trusted":true},"cell_type":"code","source":"firstidx = 1\n_, ax = plt.subplots(2, 1, figsize=(15, 15))\nplt.sca(ax[0])\nplt.plot(loss[firstidx:], label='loss')\nplt.plot(val_loss[firstidx:], label='val_loss')\nplt.legend()\nplt.sca(ax[1])\nplt.plot(acc[firstidx:], label='accuracy')\nplt.plot(val_acc[firstidx:], label='val_accuracy')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Making predictions\n\nWe can use `predict()` to make a prediction for a set of samples. Note that we have been employing a one-hot encoding scheme to represent our data, so the output layer has 10 units. The matrix returned by `predict()` thus has 10 columns, each one containing the probability that the given sample belongs to a class. To transform back to numeric labels, we use `argmin` to find the element with the highest probability."},{"metadata":{"trusted":true},"cell_type":"code","source":"dftrain_pred = dftrain.copy()\ndftrain_pred['pred'] = model.predict(X_train).argmax(axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now visualize a correctly guessed example and an incorrectly predicted one:"},{"metadata":{"trusted":true},"cell_type":"code","source":"success_example = dftrain_pred[dftrain_pred['label'] == dftrain_pred['pred']].iloc[0]\nerr_example = dftrain_pred[dftrain_pred['label'] != dftrain_pred['pred']].iloc[0]\n\n_, ax = plt.subplots(1, 2, figsize=(6, 6))\nax = ax.flatten()\n\nfor a, example in zip(ax, (success_example, err_example)):\n    a.imshow(example.filter(regex='pixel*').values.reshape((28,28)), cmap=\"gray_r\")\n    a.axis('off')\n    a.set_title(f'Predicted: {example[\"pred\"]}, actual: {example[\"label\"]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's repeat this same process for the examples in the submission:"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_prob = model.predict(X_test)\npreds_label = preds_prob.argmax(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dfpreds = pd.DataFrame({\n    'ImageId': np.arange(len(preds_label)) + 1,\n    'Label': preds_label\n})\ndfpreds.to_csv('submissions.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's all! Thanks for reading this far. Please upvote if you find it useful! Any comments or feedback are always welcome!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}