{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Simplest CNN for MNIST Digit-Rcognizer\n\nWhat is MNIST Dataset?\nMNIST Database is a collection of Images of Hand written Numbers (0-9). Each image is of size 28x28 pixels i.e 28 pixels in length and 28 pixels in height with only 1 axis to represent color gradient which is Black and White. Original Training set have 60,000 images for training and 10,000 images for test. But here in Kaggle we are given 42,000 images with label which will be used for training and 28,000 images to test/ predict results on.\n\nEntire Notebook has 4 major segments:\n1st - DATA EXTRACTION\n2ND - DATA PREPROCESSING\n3RD - MODELLING\n4TH - DATA POST PROCESS"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA EXTRACTION\nData extraction from Given CSV files"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ntrain_x_orig = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_x_orig = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\nprint(train_x_orig.shape)\nprint(test_x_orig.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Raw data:\ntrain.csv and test.csv have Different shapes as Train.csv contains one extra column for actual labels of images 785-784. and 42,000 , 28,000 represent the number of images.\n\nThe difference between them can be seen by calling there HEAD for 5 instances."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('traiing dataset columns',train_x_orig.head())\nprint('\\n test set columns',test_x_orig.head())\n\nprint(type(train_x_orig))\nprint(type(test_x_orig))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Extracting Lables and pixels from our train_x_orig dataset.\nAs we will preprocess the pixels values before fitting the model."},{"metadata":{},"cell_type":"markdown","source":"DATA extracted is in DATAFRAME FORMAT as seen by 'type(test_x_orig)' command. "},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_train = (train_x_orig['label'].values).reshape(-1,1)\n\npixels_train = train_x_orig.drop('label', axis = 1)\n\nprint('Shape of label_train ',labels_train.shape)\nprint('\\n Shape of pixels train',pixels_train.shape)\n\nprint('\\n First 10 values of Labels_train',labels_train[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above steps shows how we have seperated Labels and Pixels from our original Dataset for all 42,000 images.\n\nNext Step is DATA PREPROCESSING."},{"metadata":{},"cell_type":"markdown","source":"# DATA PREPROCESSING\n\nWhy we need DATA preprocessing?\n\n![](https://ibb.co/JtLdz6f)\nAs its clearly visible from the Image above, the pixels values follows 8-bit color system where maximum number for a color shade can be 256. including zero the scale ranges from 0-255 , 0 being full black and 255 being full white. Numbers in between are just shades of White and grey.\n\nLets Visualize one Image from our dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# 1st to convert DATAFRAME to uint8 datatype as array\n\nim = np.array(pixels_train, dtype = 'uint8')\nprint('IMAGE DATA TYPE ',type(im))\n\n# taking 2nd image from dataset of 42,000, python 0 = real world 1, python 1 = real world 2\n\nim1 = im[1] # changing [this number] for im will give you any index of image you want to visualize\nprint('\\n shape of 2nd image pixels = ', im1.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see the 2nd image seperated is having length of 784 and plotting that would'nt make any sense, so we reshape it as 28x28 which is our image Dimension 28 x 28 = 784"},{"metadata":{"trusted":true},"cell_type":"code","source":"im1 = im1.reshape((28,28))\nplt.imshow(im1, cmap = 'gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Printing Label for correspoing image from Labels_train dataset = labels_train[1]\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(labels_train[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before continuing further we would be diving our dataset into 3 groups \n\n1- Training data- model will train itself on this data.\n\n2- Validation Data- model will only checks its validity on this data, \n   no direct training is done on this dataset.\n   \n3- Test dataset- this is being provided for aking predictions on. this dataset doesnt have labels ,      it only have pixel values which our model will provide labels for once trained. \n\nratio between Train / validation dataset will be 70/30"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\npix_train, pix_valid, label_train, label_valid = train_test_split(pixels_train, labels_train, test_size = 0.30)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# data types of all variables\nprint(' Data type for pix_train = ',type(pix_train))\nprint(' Data type for label_train = ',type(label_train))\nprint(' Data type for pix_valid = ',type(pix_valid))\nprint(' Data type for label_valid = ',type(label_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Variable names can be confusing at times but upon reading and writing the code 2-3 times it becomes more clear:\n\npix_train ::: pixels for training set\n\nlabels_train ::: Labels for training set \n\npix_valid ::: pixels for validation set\n\nlabel_valid ::: labels for validation set\n\nAlso pixels_data is in still DATAFRAME format, we will conver this into 'float32' for mathematical operations."},{"metadata":{"trusted":true},"cell_type":"code","source":"pix_train = (pix_train.values).astype('float32')\npix_test = (test_x_orig.values).astype('float32')\npix_valid = (pix_valid.values).astype('float32')\n\n\nprint(' Data type for pix_train = ',type(pix_train))\nprint(' Data type for pix_valid = ',type(pix_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have 3 set of pixel data stored in pix_train, pix_valid, and pix_test.\n\nas the color encoding of images are in 8 bit they have maximum values of 255 to represent White color. We will be scaling this data by diving all the pixel values by 255.0.\n\nThis step will ensure none of the pixel value is greater than 1.0 and less then 0.0."},{"metadata":{"trusted":true},"cell_type":"code","source":"pix_train /= 255.0\npix_test  /= 255.0\npix_valid /= 255.0\n\nprint('Maximum value in pix_train =', np.max(pix_test))\nprint('Minimum value in pix_train =', np.min(pix_train))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Data Reshaping: in order to fit our data into the conv model we must reshape our data as:\n\n(number of images, 28,28,1)\n\ncurrently our data is shaped as (number of images,784)"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Shape of pix_train =',pix_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# reshaping\npix_train = pix_train.reshape(pix_train.shape[0], 28,28,1)\npix_test = pix_test.reshape(pix_test.shape[0], 28, 28,1)\npix_valid = pix_valid.reshape(pix_valid.shape[0], 28, 28,1)\n\nprint('Shape of pix_train after reshaping=',pix_train.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Preprocessing LABELS\n\ntill now we were busy to make our pixel data ready for our model.\n\nNow we need to process our labels, here we will use ONE HOT ENCODING.\n\nthe best article for understanding ONE HOT ENCODING - [https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/](httpS://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/)\n\nOur Digits/ labels ranges from 0-9 i.e 10 classes, it is required to input this number in our model last layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"num_classes = len(np.unique(labels_train))\n\nprint('Number of classes', num_classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One Hot Encoding will categorise our labels into 0-1 instead of 0-9, to represent number between 0-9 using numbers 0-9 is easy. \n\nExample: 9 is a 9 and we understand it as 9.\nbut our model require more simple version of this and the only vocabulary it has of digits are 0,1.\n\nwe have to represent number 9 in form of 0 and 1.\n\nthis is done via hot hot encoding which do the following conversion to entire labele_data.\n\n0 = [1, 0, 0, 0, 0, 0, 0 , 0, 0, 0]\n\n1 = [0, 1, 0, 0, 0, 0, 0 , 0, 0, 0]\n\n2 = [1, 0, 1, 0, 0, 0, 0 , 0, 0, 0]\n\n3 = [1, 0, 0, 1, 0, 0, 0 , 0, 0, 0]\n\n4 = [1, 0, 0, 0, 1, 0, 0 , 0, 0, 0]\n\n5 = [1, 0, 0, 0, 0, 1, 0 , 0, 0, 0]\n\n6 = [1, 0, 0, 0, 0, 0, 1 , 0, 0, 0]\n\n7 = [1, 0, 0, 0, 0, 0, 0 , 1, 0, 0]\n\n8 = [1, 0, 0, 0, 0, 0, 0 , 0, 1, 0]\n\n9 = [1, 0, 0, 0, 0, 0, 0 , 0, 0, 1]"},{"metadata":{"trusted":true},"cell_type":"code","source":"# using ONE HOT ENCODER\nfrom keras.utils import to_categorical\n\nlabel_train = to_categorical(label_train)\nlabel_valid = to_categorical(label_valid)\n\nprint('after one hot encoder',label_valid[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we can manually count the number and say the number represented by this vector is 6."},{"metadata":{},"cell_type":"markdown","source":"# MODELLING\n\nKeras Convolution Model\n\nSince this is the core of this code I assume that you know the basics.\n\nSome of the important terms are: PADDING, KERNEL_SIZE, STRIDE, number of neurons in any dimension\n\n[assuming dimension in both Height, width are equal]\n\nformula to calculate number of pixels after 1 conv layer is applied.\n\nnumber of pixel in next layer = [(number of pixel in last layer + 2 x padding - kernel_size) / stride] + 1\n\n\nfor our model we will follow the below mentioned model structure::\n\n1st conv layer : number of filter 64, kernel_size = (3,3), strides = 1, padding = 0\n\n2nd conv layer : number of filter 64, kernel_size = (3,3), strides = 1, padding = 0\n\n1st max Pooling layer : kernel_size = (2,2)\n\n3rd conv layer : number of filter 128, kernel_size = (3,3), strides = 1, padding = 0\n\n4th conv layer : number of filter 128, kernel_size = (3,3), strides = 1, padding = 0\n\n2nd max Pooling layer : kernel_size = (2,2)\n\n####################################################################\n\nusing formula to get output dim for 1st conv layer:\n\npadding = 0 = padding = 'valid'\n\n1st Conv layer output = [(28 + 0 - 3) / 1] + 1  =  26\n\n1st Conv layer output = 26 x 26 x 64\n\n2nd Conv layer output = [(26 + 0 - 3)  /  1] + 1 = 24\n\n2nd Conv layer output = 24 x 24 x 64\n\n1st Max Pool layer output = 24 x 24 x 64 -->> 12 x 12 x 64\n\n3rd Conv layer output = [(12 + 0 -3)  / 1] + 1  = 10\n\n3rd Conv layer output = 10 x 10 x 128\n\n4th Conv layer output = [(10 + 0 -3)  / 1] + 1 = 8\n\n4th Conv layer output = 8 x 8 x 128\n\n2nd Max Pool layer output = 8 x 8 x 64 -->> 4 x 4 x 128\n\n\n\nNow as our Image has reduced to 4 x 4 x 128 we will now be getting back to traditional Neural network,\nby using 4 x 4 x 64 neurons all in one layer, instead of 4 x 4 in 128.\n\nTotal number of neurons in layer = 4 x 4 x 128 = 2048\n\nwe will be having 2 layers of 256 nuerons and last output layer of 10 neurons for 10 classes that we have.\n\n2048 --> 512 --> 256 --> 10\n\n\nActivations Functions used - ReLu for all layers except last,\nIt will have Softmax."},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom keras.layers import Dense, Conv2D, Flatten, Dropout, BatchNormalization, MaxPooling2D\nfrom keras.models import Sequential\n\nmodel = Sequential()\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3), strides = 1, padding = 'valid', activation = 'relu', input_shape = pix_train.shape[1:] ))\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3), strides = 1, padding = 'valid', activation = 'relu'))\nmodel.add(MaxPooling2D(pool_size = (2,2)))\n\nmodel.add(Conv2D(filters = 128, kernel_size = (3,3), strides = 1, padding = 'valid', activation = 'relu'))\nmodel.add(Conv2D(filters = 128, kernel_size = (3,3), strides = 1, padding = 'valid', activation = 'relu'))\nmodel.add(MaxPooling2D(pool_size = (2,2)))\n\nmodel.add(Flatten())\n\nmodel.add(Dense(512, activation = 'relu'))\nmodel.add(Dropout(0.3))\nmodel.add(BatchNormalization())\n\nmodel.add(Dense(256, activation = 'relu'))\nmodel.add(Dropout(0.3))\nmodel.add(BatchNormalization())\n\nmodel.add(Dense(num_classes, activation = 'softmax'))\n\nmodel.compile(loss = 'categorical_crossentropy', optimizer = 'SGD', metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the model is now compiled we can match our calculated dimensions with the models structure.\nUse model.summary() to get  the results."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the last step for Modelling, here we will fit the model with the data we have pre-processed and pass on the number of epochs, batch_size and validation data."},{"metadata":{"trusted":true},"cell_type":"code","source":"hist = model.fit(pix_train, label_train, epochs = 100, batch_size = 64, validation_data = (pix_valid, label_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Number of epochs can be changed to understand the learning curve of the model.\nParameters that can regulated by users are:\n\n[PADDING, KERNEL_SIZE, STRIDE, POOLING](https://keras.io/layers/convolutional/), [NUMBER OF NEURON IN DENSE LAYER(256 --> 1024, ANYTHING)](https://keras.io/layers/core/#dense), [ACTIVATION FUNCTIONS](https://keras.io/activations/), [OPTIMIZERS](https://keras.io/optimizers/),[ LOSS FUNCTIONS](https://keras.io/losses/), [EPOCHS, BATCH-SIZE.](https://keras.io/getting-started/faq/#what-does-sample-batch-epoch-mean)\n\n\nplotting the Loss and Accuracy curve with number of epochs on y axis."},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot loss during training\nplt.subplot(211)\nplt.title('Loss')\nplt.plot(hist.history['loss'], label='train')\nplt.plot(hist.history['val_loss'], label='test')\nplt.legend()\n# plot accuracy during training\nplt.subplot(212)\nplt.title('Accuracy')\nplt.plot(hist.history['acc'], label='train')\nplt.plot(hist.history['val_acc'], label='test')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA POST PROCESSING\n\nAs our model is now trained we will use it for predicting labels for TEST set provided and then converting that output to the format required by KAGGLE to submit."},{"metadata":{"trusted":true},"cell_type":"code","source":"results = model.predict(pix_test)\n\nresults = np.argmax(results, axis = 1)\n\nresults = pd.Series(results, name = 'Label')\n\nsub = pd.concat([pd.Series(range(1,28001), name = 'ImageId'), results], axis = 1)\n\nsub.to_csv('csv_to_submit.csv', index = False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our result file will be saved names as fin_01.csv and can be directly uploaded for evaluation.\n\nThere are Lot more options avalable to tweak with CONV neural Network,\nlike [xavier initialization](https://keras.io/initializers/), [Dropout](https://keras.io/layers/core/#dropout), [BatchNormalization](https://keras.io/backend/#batch_normalization), [ImagedataGenerator](https://keras.io/preprocessing/image/) "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}