{"cells":[{"metadata":{},"cell_type":"markdown","source":"# My Intro to Convolutional Nueral Networks \nThis is a Beginners attempt at using Nueral Networks on the \"Hello World\" of Deep Learning datasets. The Methodology I use will be from the most popular notebooks for this competition (which I will link below) but I will explain my understanding of each step and will Include any recourses that helped me grasp the concepts.\n\n### What are we doing?\nAiming to identify images using Covalent Nueral Networks (CNNs). CNNs are an artificial nueral network that peforms image recognition through the following steps: \n* Extracting the most important parts of an image (Convolution) \n* Introducing an element of Non-Linearity (ReLu) \n* Reducing the dimensions of the image (Pooling) \n* Finally passing it through a Fully Connected Layer for Classification\n\nThe process is explained clearly with helpful gifs [here](https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/) and [here](https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53). It's useful for understanding to read about these concepts in different words so I recommend reading both articles thoroughly.\n\n### So how are we gonna to do it?\nWe will be using the Keras Deep Learning API to create our nueral network and we will follow two established notebooks as our guiding stars:\n* [Introduction to CNN Keras](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6)\n* [Convolutional Nueral Network Tutorial](https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial)\n\nLet's get to copying."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"#All the classic imports along with the necessary Keras packages\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\n\nnp.random.seed(2)\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.optimizers import RMSprop\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\n\nsns.set(style='white', context='notebook', palette='deep')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Download the code\ntrain = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Seperate the training set\nY_train = train['label']\nX_train = train.drop(labels=['label'], axis=1)\n\n#Display the data\ng = sns.countplot(Y_train)\n\nY_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Preparing the data\nNow we will normalize the pixel values and convert them into a 3d Matrix (The required form for Keras). The first 2 dimensions will represent the horizontal and verticle axis for the image (28 by 28 pixels) with the third dimension representing potential colour layers (for example a RGB image would have 3 layers) as this is a greyscale image we will set this dimension to 1.\n\nWe will also one-hot encode the Y values (replace each label with a vector of 10 elements which will have a 1 in a certain place representing the labelled number and 0's at every other point). We will also split the data into a train and validation set. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the data. This operation takes the pixel values from the 1-255 integer range and reduces them to the 0-1 range\n#The Keras model can work more efficiently in this range.\nX_train = X_train / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshape as described\n#The first -1 in the brackets is to maintain the first dimension the same. In this case the amount of images.\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])\nY_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at our images"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some examples\ng = plt.imshow(X_train[0][:,:,0], cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"g = plt.imshow(X_train[1][:,:,0], cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some examples\ng = plt.imshow(X_train[20][:,:,0], cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split the train and the validation set using random seed\nrandom_seed = 2\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=random_seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Making The Model\nWe will be using Keras sequencial API. This creates the CNN layer by layer as you write it out. The layers were discussed in the introduction of this notebook, if anything is not clear after looking at the code below it could be worth re-reading the linked articles. We will be using the model from [this](1) notebook. Some other things to note are:\n\n* From what I understand Nueral Networks typically get better as 'more' is added to them (with exceptions) the more filters, the more layers, the more epochs, the more data, the better results you will get. However, this comes at the cost of processing power/time. Playing around with some of the values in each layer of the model will demonstrate this tradeoff.\n\n* We will be using Dropouts. This is a regularisation technique in which random nuerons in the nueral Network are ignored on each pass. This works to adress overfitting as it prevents the NN from generating high weightings as on each pass the 'active' nuerons will have to adapt and account for the missing nuerons rather than constantly relying on eachother.\n\n* The Model will make a specified amount of general passes over every step (epochs) and within each epoch it will run a certain amount of iterations which will be determined by the batch size, an iteration will be peformed for each batch of the data until every data point has gone through.\n\n* The Nueral Network will peform a forward pass and then back propogation for every iteration, assigning and updating weightings along the way. For more information on this let me be the 1000th person to tell you to check out Andrew NG's Machine Learning course on Coursera.\n\n* We will be using 'Softmax' as the final layer, this will take the predictions generated by the NN, which look weird, and put them in terms we understand. Softmax isn't the only option here however it's what the top-rated notebook used so it's what we're using.\n\n[1]: (https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# The CNN architechture is In -> [[Conv2D->relu]*2 -> MaxPool2D -> Dropout]*2 -> Flatten -> Dense -> Dropout -> Out\n\nmodel = Sequential()\n\nmodel.add(Conv2D(filters = 8, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 8, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\")) #I don't know why this is here tbh\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation = \"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Choosing an optimiser\nSo I just learned that all the equations and calculations I used in the Coursera Machine Learning Course to minimise the Cost Function (Loss) are called optimisers. Huh. \n\nTherefore an optimiser is not in fact a finetuner to make your algorithm extra snazzy but rather an integral part that every Nueral Network needs in order to minimise loss by changing the network's assorted weightings. You have a few choices here, from the very simple gradient descent to more confusing alogorithms with human names. You can find explanations of some of the different options [here](1) and [here](2).\n\nWe're going to use RMSProp, a relatively new optimiser, which aims to achieve the best of all worlds by using a changing learning rate that is determined by a group of the gradients around the current postition. I think. This process allows it to move quickly through steep changes and slowly through shallow changes whilst not falling victim to the pitfall of a constantly decaying learning rate that other algorithms suffer from. \n\nDisclaimer: Take all this with a heavy pinch of salt, this is not something I totally understand and definitely deserves reading around. What I've written might be a nice summation though, maybe.\n\n[1]: https://www.dlology.com/blog/quick-notes-on-how-to-choose-optimizer-in-keras/\n[2]: https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6#:~:text=Jan%2013%2C%202019%C2%B77%20min,order%20to%20reduce%20the%20losses."},{"metadata":{"trusted":true},"cell_type":"code","source":"#The optimiser\noptimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Setting epochs and batch size\nepochs = 10 \nbatch_size = 86","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Compiling the Model\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Augmenting the data\nWe can create more data artificially to give more examples for the NN to train itself from. We'll do this by transforming the existing images in various ways and adding them to the training set. This helps with overfitting as well as representing possible realistic changes in written digits (slight rotation, smudging etc)."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we are ready to run the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(datagen.flow(X_train,Y_train, batch_size=batch_size),\n                              epochs = epochs, validation_data = (X_val,Y_val),\n                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n                              )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluating The Data\nNow That we've run our model with a final value loss of 0.9905, Yay! (Even though I had nothing to do with creating the theory for this project it's still weirdly satisifying to get good results). It's time to evaluate how it all went.\n\nWe're gonna do this with 2 handy charts: A Test Loss against epoch graph as well as a confusion matrix. The Test Loss graph will show improvement with each epoch and the confusion matrix will show which numbers were typically miscategorised as other numbers."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Test Loss Graph\nplt.plot(history.history['val_loss'], color='b', label=\"validation loss\")\nplt.title(\"Test Loss\")\nplt.xlabel(\"Number of Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Confusion matrix \n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seems It had a hard time with predicting 9's as 4's. At this point the notebook I was following did some very fancy tricks to take a look at some of the values which were predicted incorrectly and found that a lot of those mistakes even a human might make (the 9's did look like 4s). So we can let our Network off for that."},{"metadata":{},"cell_type":"markdown","source":"## The End!\nI have sucessfully copied a notebook haha. This was an excericse for me to run through the theory behind Nueral Networks by following along from much smarter people. Once again, [Here](1) and [Here](2) are the notebooks I used. I've tried my best to explain some of the theory and provided links to articles that I found helpful, hopefully some people might find it useful to hear things from a beginners perspective rather than a experts. Alright, time to submit.\n\n[1]:(https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6)\n[2]:(https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial)"},{"metadata":{"trusted":true},"cell_type":"code","source":"results = model.predict(test)\n\n# select the index with the maximum probability\nresults = np.argmax(results,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"Digit_predictions.csv\",index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}