{"cells":[{"metadata":{},"cell_type":"markdown","source":"#  MNIST Classification\n\nMNIST is a Multiclass Classification project involving image recognition. We have to classify handwritten digits as 0 to 9.\n\n\nFurther reading:\nLinked this? Try out how to get the computer to generate MNIST data (deep fake numbers in handwritten format)\n\nLink: https://www.kaggle.com/kmldas/mnist-generative-adverserial-networks-in-pytorch/\nNote this will be more advanced as it talks about GANs (generative adversrial networks) but good to know!\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport tensorflow as tf\nimport tensorflow.keras as keras\nfrom tensorflow.keras import layers\n\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nkeras.backend.set_image_data_format('channels_last')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Read Directories & Folders","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the Data\n\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"mnist_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nmnist_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\ndisplay(\" train data\",mnist_train )\ndisplay(\" test data\",mnist_test )\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# What is 784?\n\nEach image is 28 pixels wide and 28 pixles long..... 28x28=","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_size=28*28\nimage_size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Get train and test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert to train and test data; Preserve original dataset\nX_train = mnist_train.drop('label', axis=1).copy()\nX_test = mnist_test.copy()\nY_train = mnist_train['label'].copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize values\nX_train = X_train / 255.0\nX_test = X_test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Why 255?\n\n1 byte of information = 8 bits. each bit has 2 values 0 or 1; the color intensity is 2^8=256 possible value; \ni.e. goes from 0 to 255\n\nby dividing by 255, we are making maximum value 1. Now black will be 1 and 0 is white; with various shades of grey in between","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshape to 28 x 28 so that we can see the image ie. handwritten number\nX_train = X_train.values.reshape(-1, 28, 28, 1)\nX_test = X_test.values.reshape(-1, 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##  Display images\n\nTo check whether everything worked as expected, let's take a look at a few images from each folder.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport random\nno_images=len(X_train)\n\n# Display random Image\nfig, ax = plt.subplots(figsize=(10, 10))\n\nplt.imshow(X_train[random.randint(0,no_images), :, :, 0], cmap='Greys', interpolation='nearest') \n\n# replace random.randint(0,no_images) in code above with a number if you want to see specific image. \n#This dispalys a random image each time\n\nplt.title(\"Sample Image\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here 1 is black and 0 is white....after normalization\n\nSmaller size helps us visualise better?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Display random Image\nfig, ax = plt.subplots(figsize=(2,2)) # now fix size is 2 x 2\n\nplt.imshow(X_train[random.randint(0,no_images), :, :, 0], cmap='Greys', interpolation='nearest')\nplt.title(\"Sample Image\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Conversion\nWe have a validation set of 20%","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split between train and validation set\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get one hot encoding\nY_train = keras.utils.to_categorical(Y_train, num_classes=10)\nY_val = keras.utils.to_categorical(Y_val, num_classes=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Architecture\n\n\nDefining the Model (Convolutional Neural Network)\n\nThe 2D convolution is a fairly simple operation at heart: you start with a kernel, which is simply a small matrix of weights. This kernel “slides” over the 2D input data, performing an elementwise multiplication with the part of the input it is currently on, and then summing up the results into a single output pixel. - [SOURCE- read more click here](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1)\n\n![A standard convolution](https://miro.medium.com/max/535/1*Zx-ZMLKab7VOCQTxdZ1OAw.gif)\n\n\nBelow is a function defining a CNN model with 3 main blocks of Convolutional layers. The first two blocks follow the same structure:\n1. Apply a Conv2D layer with 3x3 kernel size and valid padding, then another Conv2D layer with 3x3 kernel but with same padding to keep the same dimensions\n2. Apply a BatchNormalization layer to avoid layers being too depended from one another and allowing each activation to have 0 mean\n3. A RELU activation and a MaxPooling2D layer with 2x2 kernel size and stride=2\n4. An element-wise Dropout layer applied to MaxPooling2D keeping 80% of the activation units\n\nThe final 2 layers consists of a Fully Connected Layer that uses a Softmax activation for classification.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build CNN Model\ndef CNN():\n    model = keras.Sequential()\n    # CONV > CONV > BN > RELU > MAXPOOLING > DROPOUT\n    model.add(layers.Conv2D(32, (3, 3), (1, 1), padding='valid', input_shape=(28, 28, 1), name='conv2d_1_1'))\n    model.add(layers.Conv2D(32, (3, 3), (1, 1), padding='same', name='conv2d_1_2'))\n    model.add(layers.BatchNormalization(name='bn_1'))\n    model.add(layers.Activation('relu', name='relu_1'))\n    model.add(layers.MaxPooling2D((2, 2), (2, 2), padding='valid', name='mp2d_1'))\n    model.add(layers.Dropout(0.2, name='drop_1'))\n    # CONV > CONV > BN > RELU > MAXPOOLING > DROPOUT\n    model.add(layers.Conv2D(64, (3, 3), (1, 1), padding='valid', name='conv2d_2_1'))\n    model.add(layers.Conv2D(64, (3, 3), (1, 1), padding='same', name='conv2d_2_2'))\n    model.add(layers.BatchNormalization(name='bn_2'))\n    model.add(layers.Activation('relu', name='relu_2'))\n    model.add(layers.MaxPooling2D((2, 2), (2, 2), padding='valid', name='mp2d_2'))\n    model.add(layers.Dropout(0.2, name='drop_2'))\n    # FLATTEN > DENSE > CLASSIFICATION\n    model.add(layers.Flatten())\n    model.add(layers.Dense(100, activation='relu'))\n    model.add(layers.Dense(10, activation='softmax'))\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = CNN()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can try different models- Resnet or Efficientnet and see how they work!\n\nHere is a link to my model on multiclass classification of Human Protiens which was in the top 4% in the in-class competition\nThis uses CNN,  Resnet34,  Resnet50 and Resnet101 : [Human Protein Classification (top 4%) : PyTorch](https://www.kaggle.com/kmldas/human-protein-classification-top-4-pytorch)\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Model Compilation\n\nHere we will use an Adam optimizer with a Cross entropy loss function.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam', loss='CategoricalCrossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training and Prediction\n\nWe will train the model for 50 epochs, with a batch size of 64.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(X_train, Y_train, validation_data=(X_val, Y_val), batch_size=64, epochs=50, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can make verbose=0, in the code above if you dod not  want to see each step in the process","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Graphing Accuracy\n\nLet us check how our model went by graphing accuracy with validation accuracy, and our training loss with validation loss.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.subplot(1, 2, 1)\nplt.plot(history.history['accuracy'], label='accuracy')\nplt.plot(history.history['val_accuracy'], label='val_accuracy')\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Accuracy\")\nplt.legend(loc='lower right')\n\nplt.subplot(1, 2, 2)\nplt.plot(history.history['loss'], label='loss')\nplt.plot(history.history['val_loss'], label='val_loss')\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend(loc='lower right')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see 20 epochs should have been fine. Not much improvement in scores after 20 epochs.\n\nWhat can you change to get better validation? Try our different parameter changes and see!!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Predictions\n\nBelow is a function to help see whether we have trained the model properly. \"imgs\" is a parameter to see the first x number of images in our test dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(model, X, imgs):\n    s = int(np.sqrt(imgs))\n    fig, ax = plt.subplots(s, s, sharex=True, sharey=True, figsize=(15, 15))\n    ax = ax.flatten()\n    preds = model.predict(X[:imgs])\n    for i in range(imgs):\n        y_pred = np.argmax(preds[i])\n        img = X[i].reshape(28, 28)\n        ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n        ax[i].set_title(f'p: {y_pred}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict(model, X_test, 25)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission\n\nWe create the full prediction and place the predictions into the requested format.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = model.predict(X_test)\ny_pred = np.argmax(y_pred, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"name=\"Arushi\" #Add your name here\n\nfile_name=name+\"_mnist_submission.csv\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = pd.Series(y_pred, name='Label')\nsub = pd.concat([pd.Series(range(1, 28001), name=\"ImageId\"), y_pred], axis=1)\nsub.to_csv(file_name, index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Download the predictions from folder \"output\" \n\nrefresh  if not visible\n\nSUBMIT the predictions here: https://www.kaggle.com/c/digit-recognizer","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This code gets you into the top 35% of the competition. \n\nWhat more can you do? try and experiment. Happy learning!\n\n\nLook below for other attempts and read more kernels on this competition","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Acknowledgement, Sources and Suggestions\n\n\n\n\nkernel by Chris: https://www.kaggle.com/christianwallenwein/beginners-guide-to-mnist-with-fast-ai\n\nkernel by Timothy: https://www.kaggle.com/susantotm/digit-recognizer\n\nkernel by Yassine: https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n\ncan you use FASTAI to do this faster?\nhttps://www.fast.ai\n\nLook at my colab notenook on using 5 lines of code in FASTAI to get similar results! \nhttps://colab.research.google.com/drive/1tuKzXuWgYuJVa83k6NiL0GVKy_hyJJni?usp=sharing (link updated for viewing only- Pls copy to own colab/download to run. This does not have edit access)\n \n \nTry out how to get the computer to generate MNIST data (deep fake numbers in handwritten format)\n\nLink: https://www.kaggle.com/kmldas/mnist-generative-adverserial-networks-in-pytorch/\n\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}