{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Classification using CNN\n\n**This notebook is a basic introduction to image classification in TensorFlow. Here you can learn about:**\n* Reading CSV files into an Image\n* Visualizing the image\n* Reshaping method and handling Image color channels\n* Data Augmentation using Tensorflow ImageDataGenerator\n* Building a CNN model with an introduction to the layers used\n* Plotting the training and loss curve\n* Reusable helper function\n\nSo, go ahead and read the notebook for a quick and easy introduction into CNN Image classification"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn import model_selection\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Input, BatchNormalization, Dropout, Conv2D, MaxPooling2D","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**DATA EXPLORATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\ntrain.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = train.iloc[:,1:]\ny_train = train.iloc[:,0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After the inspecting the dataset given as a CSV file, I separate the Image inputs (784 features) and Labels ( 1 feature)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.barplot(x=y_train.unique(), y=y_train.value_counts())\nplt.xlabel('Digits')\nplt.ylabel('Number of image samples')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at this class distribution graph, I can see that the classes are almost evenly distributed. Therefore, it does not require any resampling to be applied"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_helper(index):\n    plt.imshow(np.array(x_train.iloc[index, :]).reshape((28,28)), cmap = 'binary_r')\n    plt.axis('off')\n    plt.show()\n\nvisualize_helper(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I just visualize the image here. Since the images are given in a csv format, I reshaped it into the image format (i.e 28x28 = 784). For this purpose, I wrote a helper function. You can use modify and use this function according to your needs as well - Just give reference credit for this notebook!"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = np.array(x_train).reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are reshaping the data into images.\n* -1 is given so that the number of rows are dynamically written as per the dataset\n* 28*28 is the size of the image\n* 1 is channel as Images are in Black and White (Note - If it is in RGB, then channel value should be 3)"},{"metadata":{},"cell_type":"markdown","source":"**DATA AUGMENTATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n    rescale = 1./255,\n    rotation_range=20,\n    zoom_range=0.2,\n    width_shift_range = 0.1,\n    height_shift_range = 0.1,\n    validation_split=0.2\n)\n\ntrain_generator = train_datagen.flow(x_train, y_train, batch_size=64, subset='training')\nvalidation_generator = train_datagen.flow(x_train, y_train, batch_size=64,subset='validation')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Data Augmentation is performed in order to introduce variations that we may encounter in the test set. These variations are domain specific. So Please pick and choose the variations according to your problem domain. Here I have used the following variations:\n* Rescale - rescales the image in the range 0 to 1\n* Rotation Range - Rotates for upto +/-20 degrees randomly\n* Zoom Range - Zooms into the image for upto 20% randomly\n* Height shift range - Shifts the height of image by +/-10%\n* Width Shift range - Shifts width of image by +/-10%\nAlso, I have reserved 20% of the training set as the validation set\n\nThe augmentations are applied via the generator function provided by TF. Check this out for more info - https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator"},{"metadata":{},"cell_type":"markdown","source":"**MODEL BUILDING**"},{"metadata":{"trusted":true},"cell_type":"code","source":"nn_model = Sequential([\n    \n    Input(shape=(28,28,1)),\n    Conv2D(16, 8, activation='relu', padding='same'),\n    MaxPooling2D(),\n    BatchNormalization(),\n    Dropout(0.4),\n    Conv2D(32, 6, activation='relu', padding='same'),\n    MaxPooling2D(),\n    BatchNormalization(),\n    Dropout(0.3),\n    Flatten(),\n    Dense(256, activation='relu'),\n    BatchNormalization(),\n    Dropout(0.2),\n    Dense(128, activation='relu'),\n    BatchNormalization(),\n    Dropout(0.1),\n    Dense(10, activation='softmax')\n])\n\nnn_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n\nnn_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The Input Layer is defined to read the images into the CNN\n\n*  Convolution Layer - These are important when nearby associations among the features matter. Here it is less likely that the pixels in opposite corner of the images will contribute to the classification. Therefore, I used the convolutional layer \n\n* Max Pooling Layer - It selects the maximum element from the region of the feature map covered by the filter. So, the output would be a feature map containing the most prominent features of the previous feature map.\n\n* Batch Normalization - It is technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.\n\n* Dropout - It is a way of cutting too much association among features by dropping a specific percentage of weights randomly.\n\n* Flatten -  It is used when we get multidimensional output and want to make it linear to pass it onto a Dense layer. It is equivalent to numpy.ravel (Look at the shape before and flatten in the model summary)\n\n* Dense - are used when association can exist among any feature to any other feature in data point\n\n**The output layer in a classification problem should always be a softmax layer with the number of output classes as its paramter**\n\nNote - The choice of parameters are not explained here as it would take a good deal of understanding/explanation. I will make a separate notebook for this purpose. Please refer the TensorFlow documentation for now - https://www.tensorflow.org/api_docs/python/tf/keras/layers"},{"metadata":{"trusted":true},"cell_type":"code","source":"early_stopping = tf.keras.callbacks.EarlyStopping(patience=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Early stopping is a method that allows you to specify an arbitrary large number of training epochs and stop training once the model performance stops improving on a hold out validation dataset. It helps to prevent overfitting of the model "},{"metadata":{"trusted":true},"cell_type":"code","source":"history=nn_model.fit(train_generator, validation_data = validation_generator, epochs=200, callbacks=[early_stopping])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def acc_loss_plot():\n    fig, ax = plt.subplots(1,2,figsize=(15,7))\n    ax[0].plot(history.history['accuracy'])\n    ax[0].plot(history.history['val_accuracy'])\n    ax[0].set_title('Accuracy')\n    ax[0].set_ylabel('Accuracy')\n    ax[0].set_xlabel('Epoch')\n    ax[0].legend(['train', 'validation'], loc='lower right')\n    ax[0].grid()\n    ax[1].plot(history.history['loss'])\n    ax[1].plot(history.history['val_loss'])\n    ax[1].set_title('Loss')\n    ax[1].set_ylabel('Loss')\n    ax[1].set_xlabel('Epoch')\n    ax[1].legend(['train', 'validation'], loc='upper right')\n    ax[1].grid()\n    plt.show()\n\nacc_loss_plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"An another helper function. This is to plot the training and loss curve to track the model's performance. Please cite the notebook before using the function"},{"metadata":{},"cell_type":"markdown","source":"**PREDICTION AND SUBMISSION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = np.array(test).reshape(-1, 28, 28 , 1) / 255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = nn_model.predict(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = [np.argmax(x) for x in preds]\nids = [x+1 for x in range(len(preds))]\n\nsub = pd.DataFrame()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['ImageId'] = ids\nsub['Label'] = labels\n\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}