{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom sklearn.model_selection import train_test_split\n\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.utils import to_categorical\n\nfrom keras.models import Sequential, load_model\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D, BatchNormalization\nfrom keras.optimizers import Adam\n\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.callbacks import EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This notebook is a blend of what I have learned from other Kaggle Notebooks and external links.\n\nAppropriate external links are placed throughout the notebook and the following notebooks I found helpful are in the list below. \n\nChinmay Rane -> https://www.kaggle.com/fuzzywizard/beginners-guide-to-cnn-accuracy-99-7\n\nDATAI -> https://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\n\nChris Deotte -> https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist"},{"metadata":{},"cell_type":"markdown","source":"# Loading Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#loading data\ntrain = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')\ndf = train.copy()\ndf_test = test.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Just having a look at the data\ndf.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see there is one column that has the label and the rest are the image pixels!"},{"metadata":{"trusted":true},"cell_type":"code","source":"#check for any missing values in train data\ndf.isnull().any().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#check for any missing values in test data\ndf_test.isnull().any().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data PreProcessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Setting the seeds for Reproducibility.\nseed = 3141\nnp.random.seed(seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Splitting Train and Test Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# X is the pixels and Y is the image labels\nX = train.iloc[:,1:]\nY = train.iloc[:,0]\n\n#splitting dataframe using train_test_split\nx_train , x_test , y_train , y_test = train_test_split(X, Y , test_size=0.1, random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reshaping Images\n\n- We have a 1D vector with 784 pixels and we have to reshape it to (28x28x1) before passing it to the CNN. \n\n- This is because Keras wants an Extra Dimension in the end for channels. If this had been RGB images, there would have been 3 channels, but as MNIST is gray scale it only uses 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"#first param in reshape is number of examples. We can pass -1 here as we want numpy to figure that out by itself\n\n#reshape(examples, height, width, channels)\nx_train = x_train.values.reshape(-1, 28, 28, 1)\nx_test = x_test.values.reshape(-1, 28, 28, 1)\ndf_test=df_test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Augmentation"},{"metadata":{},"cell_type":"markdown","source":"Data augmentation is super important.  In terms of Images it means we can increase the number of images our model sees. \n\nThis can be acheived by Rotating the Image, Flipping the Image, Zooming the Image, Changing light conditions, Cropping it etc.\n\nKeep in mind doing all these things will not always help the model. For example in our situation a vertical_flip would not be wise as 6's would become 9's and vice-versa."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n            featurewise_center=False,  # set input mean to 0 over the dataset\n            samplewise_center=False,  # set each sample mean to 0\n            featurewise_std_normalization=False,  # divide inputs by std of the dataset\n            samplewise_std_normalization=False,  # divide each input by its std\n            zca_whitening=False,  # apply ZCA whitening\n            rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n            zoom_range = 0.1, # Randomly zoom image \n            width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n            height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n            horizontal_flip=False,  # randomly flip images\n            vertical_flip=False)  # randomly flip images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Normalization"},{"metadata":{},"cell_type":"markdown","source":"Pixel values are often stored as integers in the range 0 to 255, which is the range that int-8 can offer.\n\n\nNormalization:\n\n1. Sets the value of inputs between 0-1\n2. Helps Gradient Descent Converge much faster\n3. Brings features to equal level and weightage\n4. Helps remove distortians in an image caused by light and shadows "},{"metadata":{"trusted":true},"cell_type":"code","source":"#convert values to float as result will be a float. If not done vals are set to zero\n\nx_train = x_train.astype(\"float32\")/255\nx_test = x_test.astype(\"float32\")/255\ndf_test = df_test.astype(\"float32\")/255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fitting the ImageDataGenerator we defined above\ndatagen.fit(x_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### One hot encoding labels\n\nThe labels are given as integers between 0-9. We need to one hot encode them.\n\nFor example 4 looks like this: [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]\n\nThis is done so that we have labels for all the classes, and we can easily carry out the Error/Cost during BackPropogation.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#notice num_classes is set to 10 as we have 10 different labels\ny_train = to_categorical(y_train, num_classes=10)\ny_test = to_categorical(y_test, num_classes=10)\n\nprint(y_train[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Building CNN Model\n\nFor image classification CNN's are the best\n\nNOTE: the hardest part is picking right model by understanding the data rather than by tuning hyperparameters\n\nA larger training dataset will really help CNN accuracy"},{"metadata":{},"cell_type":"markdown","source":"Steps:\n\n1. Use Sequential Keras API \n2. Add Convolutional Layers - Building blocks of ConvNets and what do the heavy computation\n3. Add Pooling Layers - Steps along image - reduces params and decreases likelihood of overfitting\n4. Add Batch Normalization Layer - Scales down outliers, and forces NN to not relying too much on a Particular Weight\n5. Add Dropout Layer -  Regularization Technique that randomly drops a percentage of neurons to avoid overfitting (usually 20% - 50%)\n6. Add Flatten Layer - Flattens the input as a 1D vector\n7. Add Output Layer - Units equals number of classes. Sigmoid for Binary Classification, Softmax in case of Multi-Class Classification.\n8. Add Dense Layer - Fully connected layer which performs a linear operation on the layer's input"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Conv2d data_format parameter we use 'channel_last' for imgs\n\nmodel = Sequential()\n\nmodel.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', strides=1, padding='same', data_format='channels_last',\n                 input_shape=(28,28,1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', strides=1, padding='same', data_format='channels_last'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2, 2), strides=2, padding='valid' ))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu', strides=1, padding='same', data_format='channels_last'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters=64, kernel_size=(3, 3), strides=1, padding='same', activation='relu', data_format='channels_last'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2, 2), padding='valid', strides=2))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.25))\nmodel.add(Dense(1024, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Compiling Model (configure learning process)\n\nBefore training the model we need to make sure we specify how the model will \"learn\"\n\n1. Specify the Optimizer - The optimizer help us minimize the error function. Examples - RMSprop, Adam, AdaGrad, AdaDelta\n2. Specify Loss Function -  For Binary Classification use \"binary_crossentropy\" and for Multi-class Classification use \"categorical_crossentropy\"\n3. Specify the metrics to evaluate model performance\n\n"},{"metadata":{},"cell_type":"markdown","source":"Great article here to understand the ADAM optimizer -> https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Optimizer\noptimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Compiling the model\nmodel.compile(optimizer=optimizer, loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning Rate Decay\n\nMany optimization algorithms have a constant learning rate, which will often not reach a local minima.\n\nTo implement Learning Rate Decay we can use either LearningRateScheduler or ReduceLRonPlateau.\n\n- LearningRateScheduler -  takes the step decay function as argument and returns updated learning rates for use in optimzer at every epoch stage.\n\n- ReduceLRonPlateau  - monitors a quantity and if no improvement is seen for a 'patience' number of epochs, then the learning rate is reduced by a factor specified manually."},{"metadata":{"trusted":true},"cell_type":"code","source":"#for our case LearningRateScheduler will work great\nreduce_lr = LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"#visualizing what the learning rate decay will do to the learning rate through every epoch\ndecays = [(lambda x: 1e-3 * 0.9 ** x)(x) for x in range(10)]\ni=1\n\nfor lr in decays:\n    print(\"Epoch \" + str(i) +\" Learning Rate: \" + str(lr))\n    i+=1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Early Stopping Rounds\n\n- I would still like to define an early stopping parameter to ensure that my model stops training once I have reached a point where it is no longer necessary to continue training. This is another way to control overfitting.\n\n- Its important to note that we need to specify a validation dataset in the model to use an early_stopping callback"},{"metadata":{"trusted":true},"cell_type":"code","source":"#by default this is evaluated on 'val_loss'\nearly_stopping = EarlyStopping(\n    min_delta=0.001, # minimium amount of change to count as an improvement\n    patience=20, # how many epochs to wait before stopping\n    restore_best_weights=True,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we have the LearningRateScheduler, if we dont use the early stopping callback in this model we see an improvement from 96.4% to about 96.6% accuracy as we get closer and closer to the local minima."},{"metadata":{},"cell_type":"markdown","source":"# Fitting the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"#defining these prior to model to increase readability and debugging\nbatch_size = 64\nepochs = 50","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit the Model\nhistory = model.fit_generator(datagen.flow(x_train, y_train, batch_size = batch_size), epochs = epochs, \n                              validation_data = (x_test, y_test), verbose=1, \n                              steps_per_epoch=x_train.shape[0] // batch_size,\n                              callbacks = [reduce_lr]) #left out early_stopping parameter as it gets better accuracy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluating the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(13, 5))\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title(\"Model Loss\")\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend(['Train', 'Test'])\nplt.grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(13, 5))\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('Model Accuracy')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\nplt.legend(['Train','Test'])\nplt.grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the validation_accuracy and the accuracy are really close together we can conclude that we are not overfitting the data."},{"metadata":{},"cell_type":"markdown","source":"# Output prediction to CSV file"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_digits_test = np.argmax(model.predict(df_test),axis=1)\nimage_id_test = []\n\nfor i in range (len(pred_digits_test)):\n    image_id_test.append(i+1)\n    \nd = {'ImageId':image_id_test,'Label':pred_digits_test}\nanswer = pd.DataFrame(d)\n\nanswer.to_csv('answer.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}