{"cells":[{"metadata":{},"cell_type":"markdown","source":"This kernel is about building a neural network that can recognise handwritten digits. The supplied data consists of about >40000 images of different handwritten digits (~4000 each for digit 0 to 9). \n

Each image (28x28 pixel grayscale) were supplied by transforming them to a single row of 28x28 = 784 columns, that consist of the pixel value of the image. So you'll see that I've made transformation later to turn them to 28x28 pixels. "},{"metadata":{},"cell_type":"markdown","source":"This is the way that I approach this problem.\n


**Table of Content**\n

\n1. Import data & inspect if there is any missing value

\n2. Reshaping the matrix and then normalize them (kind of useful cause it will give a better neural net performance)

\n3. Splitting the training data into train_train and train_test

\n4. Importing & build model by adding layers of the neural net"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"




1. **Loading data & inspect**"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# First lets load the data\n\ntrain = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 1.1 Now check whether we have any null in both train and test sets"},{"metadata":{"trusted":true},"cell_type":"code","source":"a = train.isnull().sum().sum()\nprint('null of train: ',a)\nb = test.isnull().sum().sum()\nprint('null of test: ',b)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 1.2 In the training set, we have additional column that contains the label of te digits (this is the target/result). So Im separating them into df_x, df_y"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Put the label of the picture as y_train. The rest as x_train\ndf_y = train['label']\n\ndf_x = train.drop(columns='label')\n\nprint(df_y.shape)\nprint(df_x.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 1.3 Double check how many digits we have (eventhough we already know we have 10), then plot the number of samples of each digit"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n# check the value counts of y (surely should be 0-9)\ndf_y.value_counts()\n\n# yup it is. Lets see the plot\nsns.countplot(df_y)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 1.4 So Im making a one-hot encoder. This is because I found that most people on the internet says that its better and theres couple of proven results. Not sur why though, and if you do, please let me know in the comments.

One hot encoder means you store the label in a binary form that consist of the size of the labels (in this case 10 digits from 0 to 9). So digit 2 is store as (0010000000), digit 6 as (0000001000) and digit 9 as (0000000001). "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get one-hot of df_y\n\ndf_y_OH = pd.get_dummies(df_y)\ndf_y_OH.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"




2. ** Reshape matrix to 28x28 frame then normalize train & test sets from range(0,255) to range(-0.5,0.5). This is to improve accuracy when we have a more normalized/centered data**"},{"metadata":{},"cell_type":"markdown","source":"> 2.1 Reshaping matrix from 784 to 28,28 pixels.
-1 indicates perform this until the last row of the array (which is about 40000 images/rows),
1 indicates that this is a grayscale image so only 1 layer (RGB has 3 layers)"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_x = df_x.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)\n\nprint(df_x.shape)\nprint(test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 2.2 Now every problem will be better if we can visualise them right? So Im trying to get the first sample of each digit and put them in a subplot so we can get a feel about the data that we're playing with"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get the first index of each digit, store them in idxStore (this is just to plot and visualise the digits)\nidxStore =[]\nfor i in range(10):\n idxStore.append(df_y[df_y==i].index[0])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the first index of each digit. Just so that I can get a feel about what im working on\nfor i in range(10):\n plt.subplot(2,5,i+1)\n a=df_x[idxStore[i]]\n plt.imshow(np.squeeze(a))\n \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 2.3 This part here is similar to previous one (2.2), except now I'm looking at plotting 10 different handwriting of one digit. (Actually this came up when I was explaining to my wife about what I'm doing)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Im just trying to show my wife the different handwritting digits varies in order to explain how this\n# stuff works, so neglect this part\nidxDigit4 = df_y[df_y==7].index[:10]\nfor i in range(10):\n plt.subplot(2,5,i+1)\n a=df_x[idxDigit4[i]]\n plt.imshow(np.squeeze(a))\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> 2.4 Now I'm normalizing the range so that the pixels lies between -0.5 to 0.5 rather than 0 to 255. This is because soeone suggested that a more centerd pixel will be better to feed on to the neural nets (sorry I don't have any link to support this)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the images.\ntrain_images = (df_x / 255) - 0.5\ntest_images = (test / 255) - 0.5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"




3. ** Split train and test data**"},{"metadata":{},"cell_type":"markdown","source":"> Here what I'm doing is trying to split the train set into 2 sets: 1 for training the neural nets & 1 for validation. I call this X_train, X_test, y_train & y_test. (Not to be confused to the test set as that is the one that I'll use for submission)"},{"metadata":{},"cell_type":"markdown","source":"> > 3.1 You can split the data the classic way (which is just random splitting by how manay percentage (test_size) that you want.... Or see next subsection"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n# Split the train and the validation set for the fitting\n# X_train, X_val, Y_train, Y_val = train_test_split(df_x, df_y_OH, test_size = 0.1, random_state=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> > 3.2 As I have 10 digits that consist of 4000 samples each, I want to make sure that when I split the train & validation sets, they are split evenly. (ie: 3000 samples of each digit in train set & 1000 of each digit in test set)
The Stratified function will do this for me. (So im neglecting the part above)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Splitting the train set into train_train and train_test.\n# Im using StratifiedShuffleSplit cause I want the train_Train set to have \n# equal digit values to the train_Test set. \nfrom sklearn.model_selection import StratifiedShuffleSplit\n\nsss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=1)\n\nfor train_index, test_index in sss.split(df_x, df_y_OH):\n X_train, X_test = df_x[train_index], df_x[test_index]\n y_train, y_test = df_y_OH.loc[train_index], df_y_OH.loc[test_index]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"




4. ** Importing Model and add layers**"},{"metadata":{},"cell_type":"markdown","source":"> So everything down here is my first approach in building the neural nets. \n>

4.1 I start with 2 Conv2D layers cause I have a sense that it will definitely do better than 1. Maybe even better with 3? but I'm starting with 2 first.


\n> You'll also see that I made a for loop here to see what number of units should I proceed with."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import Sequential\nfrom keras.layers import Conv2D, Flatten, Dense\nfrom sklearn.metrics import accuracy_score\nstore_i = []\nstore_acc = []\n\nfor i in [5,10,15]:\n model = Sequential()\n model.add(Conv2D(i,kernel_size=2,activation='relu',input_shape=(28,28,1)))\n model.add(Conv2D(i,kernel_size=2,activation='relu'))\n model.add(Flatten())\n model.add(Dense(10,activation='softmax'))\n\n model.compile(optimizer='adam',\n loss='categorical_crossentropy',\n metrics=['accuracy'])\n training = model.fit(X_train,y_train,epochs=1,batch_size=86)\n preds = model.predict(X_test)\n y_test_ = np.argmax(y_test.to_numpy(), axis=1)\n preds_ = np.argmax(preds, axis=1)\n acc = accuracy_score(y_test_,preds_)\n \n store_i.append(i)\n store_acc.append(acc)\n\nprint('Units, Accuracy : ',store_i,store_acc)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> As you see above, I'm testing how many units should I put in both Conv2D layers. Now I know the accuracy reached 96.17% for unseen data with 10 units.

(EDIT UPDATE: I just realised that the result is randomly generated everytime I run this part. So 10 units are not necessarily the best. I found that 15 units can also be better sometimes, in this notebook, I proceeded with 10 as I've already done that) "},{"metadata":{},"cell_type":"markdown","source":"
\n> 4.2 Now I want to see if adding more filter layers like MaxPool2D or __ improves the accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import MaxPool2D\nacc_avg_3_runs = []\nfor i in range(3):\n model = Sequential()\n model.add(Conv2D(10,kernel_size=2,activation='relu',input_shape=(28,28,1)))\n# model.add(MaxPool2D(2))\n model.add(Conv2D(10,kernel_size=2,activation='relu'))\n model.add(Flatten())\n model.add(Dense(10,activation='softmax'))\n\n model.compile(optimizer='adam',\n loss='categorical_crossentropy',\n metrics=['accuracy'])\n training = model.fit(X_train,y_train,epochs=1,batch_size=86)\n preds = model.predict(X_test)\n y_test_ = np.argmax(y_test.to_numpy(), axis=1)\n preds_ = np.argmax(preds, axis=1)\n acc = accuracy_score(y_test_,preds_)\n acc_avg_3_runs.append(acc)\n \nprint(i,np.mean(acc_avg_3_runs))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Without Pool (average of 3 runs): 0.9527\n
With Pool (average of 3 runs): 0.9271

Keep in mind that this is randomly generated results, so it might change when you run them. As long as the difference is not too huge, then should be fine (roughly in 5-10% range)"},{"metadata":{},"cell_type":"markdown","source":"> 4.3 This part is not really that important. I was looking at how many epochs should I use"},{"metadata":{"trusted":true},"cell_type":"code","source":"training = model.fit(X_train,y_train,epochs=5,batch_size=86)\nhistory = training.history\n\nplt.plot(history['loss'])\n\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2 epochs should be good as its a good practice to pick the value of epochs just before steady state"},{"metadata":{},"cell_type":"markdown","source":"> 4.4 Looking at the confusion matrix"},{"metadata":{},"cell_type":"markdown","source":"> > Before running the confusion matrix, I'm just instatiating the model again with 10 units, no pooling and 2 epochs.

Im running this again just so that it wont use the histry of the training in section 4.2"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Basically this is the same code as what I've done above\n\n# The layers\nmodel = Sequential()\nmodel.add(Conv2D(10,kernel_size=2,activation='relu',input_shape=(28,28,1)))\nmodel.add(Conv2D(10,kernel_size=2,activation='relu'))\nmodel.add(Flatten())\nmodel.add(Dense(10,activation='softmax'))\n\n# Compilation\nmodel.compile(optimizer='adam',\n loss='categorical_crossentropy',\n metrics=['accuracy'])\n\n# Training\ntraining = model.fit(X_train,y_train,epochs=2,batch_size=86)\n\n#Predicting\npreds = model.predict(X_test)\n\n# Change results from one-hot encode to normal (ie: back to 1,2,3,4,5....)\ny_test_ = np.argmax(y_test.to_numpy(), axis=1)\npreds_ = np.argmax(preds, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import confusion matrix and get them\nfrom sklearn.metrics import confusion_matrix\n\nconf_mtx = confusion_matrix(y_test_,preds_)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> > So now that we have the confusion matrix, Im just gonna use this function that was shared on plotting the confusion matrix in a heatmap.

I then use this function to plot the confusion matrix "},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport itertools\n\n# Make the function \n\ndef plot_confusion_matrix(cm, classes,\n normalize=False,\n title='Confusion matrix',\n cmap=plt.cm.Blues):\n \"\"\"\n This function prints and plots the confusion matrix.\n Normalization can be applied by setting `normalize=True`.\n \"\"\"\n plt.imshow(cm, interpolation='nearest', cmap=cmap)\n plt.title(title)\n plt.colorbar()\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n if normalize:\n cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n thresh = cm.max() / 2.\n for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n plt.text(j, i, cm[i, j],\n horizontalalignment=\"center\",\n color=\"white\" if cm[i, j] > thresh else \"black\")\n\n plt.tight_layout()\n plt.ylabel('True label')\n plt.xlabel('Predicted label')\n\n\n# Use the confusion matrix and plot them using the function above\nplot_confusion_matrix(conf_mtx, classes = range(10)) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> > So this confusion matrix tells you the predicted label on the x-axis and what is the actual result on the y-axis. You can see that our model got about 96.86% just now and predict the results with good accuracy.

All the diagonal cell shows the correct predictions"},{"metadata":{},"cell_type":"markdown","source":"

\n** Conclusion ** \n

So thats basically a first approach on building a convolutional neural network (CNN). What I can do to improve the accuracy is to try adding other filters that are available in tf.keras.layers, add more Conv2D or Dense layers, or tune the parameters in the layers itself. There are many ways that we can go from here, but I guess as long as you have sufficient amount of data and good basic model, don't worry too much you're good to go! Happy coding!"},{"metadata":{},"cell_type":"markdown","source":"


\n \nOh I forgot if you want to see how I predict the test data and save to csv file, its down here"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_predict = model.predict(test)\ntest_predict = np.argmax(test_predict, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\nId = sample['ImageId']\n\nsubmission = pd.DataFrame({ 'ImageId': Id,\n 'Label': test_predict })\nsubmission.to_csv(path_or_buf =\"MNIST_Submission.csv\", index=False)\nprint(\"Submission file is formed\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" Download File "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}