{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30665,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This note book is made by me **Bibek Bhusal** just for the purpose of leanring `Deep Learning` and `Convolutional Neural Network`. Now, I have Descided This Notebook should be public and well documented for anyone to use and learn from this notebook. This notebook is also the part of Kaggle Compitation **Digit Recognizer** in which it has scored 0.99978. I will use Kears for this project, since kears is one of the most popular package for deep learning and the most easy to use.\n\nThis is my first porject in `Keras`, `Deep Learning` so the metod I choose to do certain task may not be the best way to do it and I am just getting started, I will appreciate if you give me some feedback and sugessions for improvement. \n\n\nIn Deep Learning and Computer vision Digit recognization is considered as **Hello world**. \n\nSo, let's get started.","metadata":{}},{"cell_type":"markdown","source":"# Importing Libraries and Dataset","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport pandas as pd  # generally used for data analysis and manipulation\nimport numpy as np # used for mathematical operations\nimport matplotlib.pyplot as plt # used create plots and graphs here we will use matplotlib to see images  \nimport seaborn as sns # Fast tool for data visualization\nfrom sklearn.metrics import confusion_matrix\n\nfrom keras import Sequential, layers, Model\nfrom keras.utils import to_categorical, plot_model\nfrom keras.datasets import mnist\nfrom keras.models import Model\n\nimport random ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I have imported All important Libaries. I will use Keras for this project and Keras also has dataset for this porjcet which is MNIST dataset","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest =  np.array(pd.read_csv('/kaggle/input/digit-recognizer/test.csv'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets look at the shape of the dataset\n'training data shape:', train.shape, 'testing dataset shape:', test.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Training datset contains one extra columns which is **label**. it is the target variable. Other 784 columns are pixel for the image. let's have a look at the data and I will also seperate the data into X and y.","metadata":{}},{"cell_type":"code","source":"train.head() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.label.value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img =  train.iloc[0,1:] # this is the first image of the dataset let's have a look at it \nlabel = train.iloc[0,0]\n\nimg = img.values.reshape(28,28 ) # we need to reshape the image because it was in 1D series \n\nplt.imshow(img, cmap='gray')\nplt.title(label)\nplt.axis('off')\n\nplt.show() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_v = train.label\nX_v  = train.drop(columns = 'label')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"MNIST dataset is large collenction of handwirtten digit images which are in grayscale. Keras also have inbuilt dataset for this. Which contains 60,000 images for training set and 10,000 images in test set. For this project I will used Both sets from Keras as traning set and for validation set I wll use training set porvided by kaggle. ","metadata":{}},{"cell_type":"code","source":"(X,y ) , (X_, y_) = mnist.load_data()\nX,y = np.concatenate((X,X_), axis=0), np.concatenate((y,y_), axis=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sneek Peek to the Dataset\nlets look at some random of the images in keras dataset","metadata":{}},{"cell_type":"code","source":"n = np.random.randint(len(X), size=6)\n\nplt.figure(figsize = (7,7))\nfor i,j in  enumerate(n):\n    ax = plt.subplot(3,3,i+1)\n    ax.imshow(X[j], cmap='gray')\n    ax.set_title(y[j], fontsize=20)\n    ax.set_xticks([])\n    ax.set_yticks([])\n    ax.margins(0.9)\nplt.show()","metadata":{"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets also look at data in kaggle dataset \nrows = 5\nfig, axes = plt.subplots(rows, 10, figsize=(10 ,rows))\n\nfor i in range (10):\n    ds  = train[train.label == i]\n    for j in range(rows):\n        ax = axes[j,i]\n        ax.imshow(ds.iloc[j,1:].values.reshape(28,28), cmap='gray')\n        ax.axis('off')\n        \nplt.tight_layout()\nplt.show() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the above images, the dataset contains images of numbers from `0` to `9` in different orientation, style and stroke.\n\n# Designing the model\nI will rehspe each image to (28,28,1) where 28 x 28 is size of the image and 1 is the number of color channels (grayscale).If images was in color then it's shale would be (28,28,3). \n\nI have also truned labels to categorical values which is required for classification this step is called`one-hot encoding`.","metadata":{}},{"cell_type":"code","source":"def reshape(arr):\n    return arr.reshape(arr.shape[0],28,28,1 )\n\nX = reshape(X)\nX_v = reshape(X_v.values)\nX_test = reshape(test)\ny = to_categorical(y)\ny_v = to_categorical(y_v)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I have added `Input` layer of shape same as image, \nrescaling layer will convert each color value from **0-255 to 0-1**,\n`Convolutional` Layer will add filters to images which are used to **extract features** from images,\n`MaxPooling` will **downsample** the input data for better performance,\n`Flatten` will flatten the data into 1D vector which is suitable for dense layer,\n`Dense` layer will add **fully connected** layers,\n\nI have used various layers of `Convolutional` layers followed by `MaxPooling` layer .\nOutput layer is with softmax activation function which will return probability for each of the 10 classes.","metadata":{}},{"cell_type":"code","source":"model = Sequential([\n    layers.Input(shape=(28,28,1)),\n    layers.Rescaling(1/255),\n    layers.Conv2D(32, (3,3), activation='relu'),\n    layers.MaxPooling2D((2,2)),\n    layers.Conv2D(64, (3,3), activation='relu'),\n    layers.MaxPooling2D((2,2)),\n    layers.Conv2D(64, (3,3), activation='relu'),\n    layers.Flatten(),\n    layers.Dense(64, activation='relu'),\n    layers.Dense(10, activation='softmax'),\n    layers.Softmax()\n])\n\nplot_model(model, show_shapes=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Expand the above output to see model architecture.\n\nI have combiled the model using `Adam` optimizer and `Categorical Crossentropy` loss function.\n40 epochs means that model will be trained for 200 epochs, each epoch accuracy will be imporved(hopefully) and loss will be decreased. I have used h to get the history of the model and later i have also plotted it using seaborn.","metadata":{}},{"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss= 'categorical_crossentropy',\n    metrics=['accuracy']\n)\n\nh = model.fit(- \n    X, y ,\n    validation_data=(X_v,y_v),\n    epochs =40 ,\n    verbose=0,\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Looking at the model performance and predictions","metadata":{}},{"cell_type":"code","source":"history = pd.DataFrame(h.history)\nlosses = ['loss', 'val_loss']\nfig, axes = plt.subplots(1, 2, figsize=(15, 5))\n\nsns.lineplot(data=history[losses], ax=axes[0], linestyle='--')\naxes[0].set_title('Losses over Epochs')\naxes[0].set_xlabel('Epoch')\naxes[0].set_ylabel('Loss')\naxes[0].grid(True, alpha=0.5, linestyle='--')  \n\nmetrics = history.drop(columns=losses)\nsns.lineplot(data=metrics, ax=axes[1],)\naxes[1].set_title('Metrics over Epochs')\naxes[1].set_xlabel('Epoch')\naxes[1].set_ylabel('Value')\naxes[1].grid(True, alpha=0.5, linestyle='--') \n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generally validation loss is more than training loss, Because of overfitting. But in this case model is not overfitting I guess it is due to large data. And validation accuracy is higher than training accuracy maybe because the data in validation set is more simpler. \n\nLet's look at the metrics for the last epoch.","metadata":{}},{"cell_type":"code","source":"history.iloc[-1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Validation Accuracy for last epoch is pretty high (getting over 99.8% accuracy each times). I will try to predict some of the random images form the validation set. To check on the model performance and plot it, the wrong prediction are in red(ish) color.There may be less or no wrong predictions, since model accuracy is high. ","metadata":{}},{"cell_type":"code","source":"def random_test(rows= 1, cols =1 ):\n    total = rows * cols\n    correct = total\n    random_indices = np.random.randint(len(X_v), size=total)\n    fig, axes = plt.subplots(rows, cols, figsize=(cols * 1.3  ,1.8* rows) )\n    random_images = X_v[random_indices]\n    labels = y_v[random_indices]\n    predictions_p = model.predict(random_images, verbose=0)\n    random_images.reshape(total, 28, 28)\n    \n    for i in range(rows):\n        for j in range(cols):\n            index = i * cols + j\n            image = random_images[index]\n            label =np.argmax( labels[index])\n            prediction = predictions_p[index]\n            prediction = np.argmax(prediction)\n            if rows == 1 and cols ==1:\n                ax = axes\n            elif rows ==1 :\n                ax = axes[j]\n            elif cols == 1:\n                ax = axes[i]\n            else:\n                ax = axes[i][j]\n            cmap = 'gray'\n            if label != prediction:\n                correct -= 1\n                cmap = 'OrRd'\n            ax.imshow(image, cmap=cmap)\n            ax.set_title(f'L: {label}, P: {prediction}')\n            ax.axis(False)\n        fig.suptitle(f'correct:{correct}/{total}', fontsize=16)\n        fig.tight_layout(rect=[0, 0.03, 1, 0.95])\n    plt.show()\n\nrandom_test(3,3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_test(10,10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I have run this above cells many times with even more images, there is less or no wrong predictions.Which means our model is not overfitting. So to understand the model, I will plot the confussion matrix with entire validation set. \n\nI have made `reverse_encoding` function to turn the model prediction (which is in probability) into labes by using `np.argmax` function.","metadata":{}},{"cell_type":"code","source":"def reverse_encoding(prediction_prob):\n    prediction =[]\n    for i in prediction_prob:\n        predicted_val = np.argmax(i)\n        prediction = np.append(prediction,predicted_val)\n        \n    return prediction\ny_hat = reverse_encoding(model.predict(X_v))\ncm  = confusion_matrix(y_v.argmax(axis=1), y_hat)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,8))\nsns.heatmap(cm, annot=True, cmap='Blues', fmt='d', linewidths=.5, cbar=False)\nplt.xlabel('Predicted labels')\nplt.ylabel('True labels')\nplt.title('Confusion Matrix')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Model had make very samll errors compared to the correct predictions it had made.","metadata":{}},{"cell_type":"markdown","source":"# Subimssion","metadata":{}},{"cell_type":"code","source":"prediction_prob = model.predict(X_test)\nprediction = reverse_encoding(prediction_prob)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output = pd.DataFrame(prediction, columns=['Label'])\noutput[\"Label\"] = output['Label'].astype(\"int16\")\noutput.index.name = 'ImageId'\noutput.index = output.index + 1\noutput.to_csv('submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thank you for reading this notebook till the end I will appreciate if you provide any feedback even if negative it will help me learn. I am also learning Pytorch with same dataset. I will also make the notebook I used to learn Pytorch public.","metadata":{}}]}