{"cells":[{"metadata":{},"cell_type":"markdown","source":"Here's my attempt at making a simple CNN to train and make predictions on the MINST digits dataset.  We'll start off by importing the libraries and functions we'll be using"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"#Imports\nimport numpy as np \nimport pandas as pd \nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPool2D, Activation, Dropout, Flatten, Dense, BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nfrom keras.optimizers import Adam","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we gotta import the data.  Kaggle stores it in the folders below, from which we import it."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Reads in the training and testing sets.  \ndf = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest =  pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\") #Won't see this again until later\n\nprint(df.shape)\nprint(test.shape)\nprint(df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, these are datasets of pixels by image, with each row representing a seperate image of 784 pixels (likely 28 high X 28 wide). The training set contains 42000 images, while the testing set contains 28000. This appears to be monohromatic, as there is only one layer. Were they not black and white, we could expect three different sets of values for each pixel, representing red, green, and blue.\n\nThe training set has an extra column relating to the actual digit drawn for each image (the labels). Going forward, that is going to be very important information but we need to seperate it out so we only have pixel data in the training set. Below, the labels are sectioned off into their own list, and the corrisponding column is dropped from the training set.  We also get the counts of each different label, which will be helpful to know."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Makes a list of the actual digits drawn for each image, then get's rid of that column from the training set\nlabels = df[\"label\"]\nX = df.drop('label', axis = 1)\nprint(labels.value_counts())\n\nprint(\"Baseline Accuracy: \" + str(round(labels.value_counts().max()/labels.value_counts().sum(),3)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above are the value counts for each label.  As you can see, it's fairly uniformly distributed, although there are some differences between categories.  Having a uniform distribution makes categorical classification a bit easier.  We also now know the base accuracy we need to achieve with the model to do better than just guessing the most common category every time (the number 1 in this case).  If the model does better than that, then it's a fair bet it's actually learnng from the imagery.\n\nNow let's find the range of potential values for each pixel"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(X.values.max())\nprint(X.values.min())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pretty standard configuration: 0 is black, 255 is white.  We should adjust the values so they fall between 0 and 1, which makes it easier for the model to handle later.  We should also reshape and re-type the data to the required format.  Let's wrap it all up in a function.\n\nNote: Keras has built-in ways to do this stuff, but here it is anyway."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Normalizes values in dataframe to 0-1, and reshapes it to the required dimensions \ndef normalizeANDreshape(df, minimum, maximum): \n    diff = maximum - minimum\n    zero_min = df - minimum\n    adjusted = zero_min/diff\n    \n    shaped = adjusted.values.reshape(-1,28,28,1) #the width and height of the images, 1 layer deep because it's monochromatic (would be 3 if it contained RGB values)\n    \n    return shaped\n\nprint(np.max(normalizeANDreshape(X, 0, 255)))\nprint(np.min(normalizeANDreshape(X, 0, 255)))\nprint(type(normalizeANDreshape(X, 0, 255)))\nprint(normalizeANDreshape(X, 0, 255).shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, we now have a function that will get the image data ready for us.\n\nNow we need to process the labels, i.e. get dummy variables, to train the model with."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Dummy lables in a dataframe\ny = pd.get_dummies(labels)\ny.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now to train this model well (and help avoid overfitting) we need to section off some of the training data into a validation set.  This way the model has some way to evaluate itself independent of the data it's using to train as it trains. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Splits into training and validation sets.  Uses 20% of data as validation\nX_train, X_val, y_train, y_val = train_test_split(normalizeANDreshape(X, 0, 255), y, test_size = 0.20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now this dataset is small enough (~122 MB) to fit into working memory, so we don't need to bother with configuring up directories to process it in chunks.  We will however want to augment it a bit.  \n\nThese images are very small.  Also, if you go look at images of what these numbers look like (sorry for not including examples in here), you'll see that all of the numbers are fairly standard: right side up, not mirrored, not too tilted, etc.  Thus, we shouldn't need to do too much to avoid overfitting; we'll just shift things around, rotate slightly, and zoom a bit.\n\nNote: we do not want to augment the testing or the validation sets."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Augments the data with some rotated, shifted, and zoomed images\naugment = ImageDataGenerator(rotation_range = 15,\n                                  width_shift_range = 0.35,\n                                  height_shift_range = 0.35,\n                                  zoom_range = 0.2,\n                                  )\naugment.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we need to make the model.  As hinted at in the title, it's going to be a simple convoluted neural network (CNN).  Below is a function that will create the framework.  Lot's of ways to do this, and the images are fairly simple and small, so we shouldn't need too complex a network to get good results; below is one possiblity."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Sets the model architecture\n#Won't into how this works heree\ndef makeCNN(shape):\n    \n    model = Sequential()\n\n    model.add(Conv2D(filters = 64, kernel_size = (4,4), padding = 'same', activation = 'relu', input_shape = shape))\n    model.add(MaxPool2D(pool_size=(2,2)))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25))\n\n    model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu'))\n    model.add(MaxPool2D(pool_size=(2,2)))\n\n    model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu'))\n\n    model.add(Conv2D(filters = 128, kernel_size = (2,2), padding = 'same', activation = 'relu'))\n    model.add(MaxPool2D(pool_size=(2,2)))\n\n    model.add(Flatten())\n\n    model.add(Dense(64, activation = 'relu'))\n    model.add(Dropout(0.25))\n\n    model.add(Dense(10, activation = 'softmax'))\n\n    return model\n    \n#Displays a summary of the model\nmodel = makeCNN((28, 28, 1))\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now have a model framework with about 182k parameters.  Next, we need to compile it.  We'll try using the 'Adam' optomizer function, which will adapt the learning rate, along with categorical crossentropy for loss (becuase we are looking for categorical classification), and measure accuracy."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Compiles the model\nmodel.compile(optimizer = Adam(), #Default learning rate is 0.001\n             loss = 'categorical_crossentropy',\n             metrics = 'accuracy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All that's left is to train it.  We'll choose 50 epochs and a batch size of 64.  The images are pretty small, so this shouldn't take too long.  Could probably even do K-fold validation in a reasonable time.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"#This isn't likely to take long, so we'll use a large epoch.  Also, fitting things into memory isn't gonna be a problem, so batch size of 64 should be fine\nepochs = 50\nbatch_size = 64\n\n#Trains the model\nhistory = model.fit_generator(augment.flow(X_train,y_train, batch_size=batch_size),  #Retrieves the augmented images from the ImageDataGenerator\n                              epochs = epochs, \n                              steps_per_epoch=len(X_train) // batch_size, #So we get the entire training set per epoch\n                              validation_data = (X_val,y_val),\n                              verbose = 1, \n                              use_multiprocessing = True, #Use multiple CPUs. \n                              workers = 2                 #Kaggle gives you two CPU cores when you use GPU\n                             )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Its getting high accuracy; definitely some learning going on.  Another interesting facet is that the validation loss is much less than the training loss.  This is because we did not augment the validation dataset, and as a result the validation images are much easier to predict than the augmented training images.  \n\nLet's quickly visualize how it did over time."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plots accuracy and loss across epochs\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_accuracy'])\nplt.plot(history.history['val_loss'])\nplt.legend(['Accuracy', 'Loss', \"Validation Accuracy\", \"Validation Loss\"])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see it improved really quickly over a few epochs, then slowed down.  Ends up with a pretty high accuracy.  Let's see if there are any numbers in particular that it has trouble with (at least on the validation set)."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Makes predictions on the validation set \nvalidation_predictions = model.predict_classes(X_val)\n\n#Sets up a confusion matrix\nconfusion = confusion_matrix(validation_predictions,y_val.idxmax(axis=1))\n\nprint(confusion)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks pretty good.  No particular false classification really stands out.  Now, we'll make predictions on the testing data."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Pre-process the testing data\ntest = normalizeANDreshape(test, 0, 255)\n\n#Makes class predictions on the test set\npredictions = model.predict_classes(test)\n\nprint(predictions[0:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Quick conclusion:\n\nThis is an example of a simple CNN.  These images are pretty basic and don't have too much noise/other things going on aside from the numbers, which makes for a pretty easy classification target.  99% accuracy for this dataset is pretty common, especially with CNNs for which is this pretty much an ideal dataset.  \n\nThat said, we still need to make and sumbit the predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Makes an Id column for the submission\n#Should just be int's between 1 and the length of the testing set\nId = []\nfor i in range(len(test)):\n    Id.append(i+1)\n\n    \n#Outputs the submission\noutput = pd.DataFrame({'ImageID': Id, 'Label': predictions})\noutput.to_csv('predictions.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}