{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n    This is a very beginner friendly notebook wherein I explore ANNs and CNNs along with some optimizations.\n    \n**If you see any problems with my code, comments or interpretations, feel free to correct me. I'm open to any feedbacks and/or constructive criticism. Also, drop an upvote if you found this notebook infomative.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport keras\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom keras.utils import to_categorical\nfrom keras.activations import relu, softmax\nfrom keras.regularizers import l1, l2, l1_l2\nfrom keras.initializers import glorot_normal\nfrom keras.optimizers import RMSprop, Adam, SGD\nfrom keras.models import Sequential, load_model\nfrom sklearn.preprocessing import StandardScaler\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split as tts\nfrom sklearn.metrics import classification_report, confusion_matrix\nfrom keras.callbacks import EarlyStopping, LearningRateScheduler, ReduceLROnPlateau\nfrom keras.layers import Dense, BatchNormalization,Dropout, BatchNormalization, Conv2D, MaxPooling2D, AveragePooling2D, Flatten, Activation\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading and Visualizing Train, Test and Sample Submission data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"files = []\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n        files.append(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(files[2])\ntest_data = pd.read_csv(files[0])\nsubmission = pd.read_csv(files[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#checking number of training and testing examples in hand.\ntrain_data.shape, test_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samples = train_data.drop(['label'], 1)\ntrain_labels = train_data.label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting 50 images from the training set\nfig,ax = plt.subplots(10, 5, figsize=(20,20))\naxes = ax.ravel()\nfor i in range(50):\n    image = np.array(train_samples.iloc[i+16,:]).reshape(28,28)\n    axes[i].imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting 50 images from the test set\nfig,ax = plt.subplots(10, 5, figsize=(20,20))\naxes = ax.ravel()\nfor i in range(50):\n    image = np.array(test_data.iloc[i+16,:]).reshape(28,28)\n    axes[i].imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Visualizing number of samples per class in the training set\nsns.countplot(train_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Normalizing Pixel Values to a range of [0,1]\ntrain_samples_scaled = train_samples.astype(float)/255\ntest_samples_scaled = test_data.astype(float)/255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#One-Hot Encoding target variables in the train data\nencoded_train_labels = to_categorical(train_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = train_samples_scaled\ny_train = encoded_train_labels\n\nX_test = test_samples_scaled","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Building ANN Model\n\n    This Artificial Neural Network takes in an input vector of shape (784,)\n    It is then passed onto a few Dense Layers with Relu activation function and finally going into an output layer with 10 neurons and a Softmax activation function.\n    Finally we compile the model with an Adam Optimizer, Categorical Crossentropy as our loss function and Accuracy as a metric for evauation.\n    We run this model for a total of 40 epochs and batch size of 64.\n    I'll be discussing the results of the model after it runs.\n    ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Dense(128, activation = 'relu', input_shape = (784,)))\nmodel.add(Dense(128))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Dense(64, activation='relu'))\nmodel.add(Dense(64))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Dense(32, activation='relu'))\nmodel.add(Dense(32))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Dense(10, activation = 'softmax'))\n\nopt = Adam(0.01)\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])\n\nmodel.summary()\n\nhistory = model.fit(X_train, y_train, validation_split=0.05, epochs = 40, batch_size = 64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.xlabel('accuracy/loss')\nplt.ylabel('epochs')\nplt.legend()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Interpreting ANN model 1\n\n    Considering the accuracy, I'd say the model did a pretty good job. Training accuracy of 99.72% and Validation Accuracy of 97.62%.\n    But notice how towards the last few epochs the loss doesn't decrease beyond a certain threshold. Turns out that's a problem with our learning rate. The thing is, the model is failing to reach its global minima and infact overshooting the same, thus revolving around the same set of values. Thus in the next model we'll be using a callback called Reduce Learning Rate on Plateau. What that does it. It'll reduce the learning rate by a factor we provide so that it can learn more from the data. \n    Just incase the issue still persists we're going to use another callback called early stopping that'll stop training if the loss revolves around the same values for a long time.\n  ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"es = EarlyStopping(monitor='val_loss', mode = 'min', patience = 20)\nrlrop = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5)\n\n\nmodel = Sequential()\n\nmodel.add(Dense(128, activation='relu', input_shape = (784,)))\nmodel.add(Dense(128))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Dense(64,activation='relu'))\nmodel.add(Dense(64))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Dense(32,activation='relu'))\nmodel.add(Dense(32))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Dense(10, activation = 'softmax'))\n\nopt = Adam(0.01)\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])\n\nmodel.summary()\n\nhistory = model.fit(X_train, y_train, validation_split=0.05, epochs = 40, batch_size = 64, callbacks = [es, rlrop])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.xlabel('accuracy/loss')\nplt.ylabel('epochs')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"    Saving the model as an hdf5 file so that we don't have to re-train the same model.\n    We also run predictions on the test set for submission.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('ann_model.hdf5')\npredictions = model.predict_classes(X_test)\nsubmission.Label = predictions\nsubmission.to_csv('/kaggle/working/ANN_submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Interpreting ANN Model 2\n\n    You see, when you reduce the learning rate on plateau, the loss drops to an extent where the training accuracy is as high as 99.97%.\n    Next we're going to see Covolutional Neural Network, so that our model learns features in a much efficient way whilst increasing the validation accuracy.\n    ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# ANN gave an accuracy of 98.103% on the test set on submission.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# CNN\n\n    CNNs unlike ANNs do not take input as a vector, rather they take the image in their true form i.e as matrices of order (nxn; (28,28) in our case).\n    We thus prepare our train and test sets in order to feed it into our CNN network.\n    The model's architecture is inspired from the VGG-16 model, although it's not exactly the same.\n    *Tip: Instead of trying to build your own architecture from scratch, try well-known architectures that have worked for other researchers because there's a high chance that it might work for you too. Obviously you need to change it up a bit based on your data/problem. You can then change the architecture perhaps by adding extra Conv2D/Dropout Layers based on the results, if required.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cnn_train = np.array(train_samples_scaled).reshape(-1,28,28,1)\ncnn_test = np.array(test_samples_scaled).reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"es = EarlyStopping(monitor='val_loss', mode = 'min', patience = 12)\nrlrop = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, min_lr=0.000001)\n\nmodel = Sequential()\n\nmodel.add(Conv2D(64, (3,3), activation = 'relu', padding = 'same', input_shape=(28,28,1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(64, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D((2,2), strides = ((2,2))))\n\nmodel.add(Conv2D(128, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D((2,2), strides = (2,2)))\n\nmodel.add(Conv2D(256, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(256, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(512, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D((2,2), strides = (2,2)))\nmodel.add(Flatten())\n\nmodel.add(Dense(4096, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dense(4096,activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])\nmodel.summary()\nhistory = model.fit(cnn_train, y_train, validation_split=0.01, epochs = 50, batch_size=256, callbacks = [es, rlrop])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize = (20,10))\n\nax[0].plot(history.history['accuracy'], label = 'Training Accuracy', color='r')\nax[0].plot(history.history['val_accuracy'],label = 'Validation Accuracy', color='b')\nax[0].legend(loc='lower right')\n\nax[1].plot(history.history['loss'], label = 'Training Loss', color='r')\nax[1].plot(history.history['val_loss'], label = 'Validation Loss', color = 'b')\nax[1].legend(loc='upper right')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Interpreting CNN Model 1\n    As compared to our ANN model, this did pretty good in terms of the accuracy numbers.\n    The training accuracy is a perfect 1.00 while the validation accuracy is 0.9952.\n    Considering the fact that Humans do pretty well with recognizing handwritten digits, we assume human error would be approximately 0%, we can say that our model did pretty well learning from the training data since training error is 0% ((1-accuracy)*100).\n    But what about our validation error? It is almost 0.0048.\n    Thus, this is a case of low bias (human error - training error) and high variance(training_error-validation error). This means our model has overfitted on the training data. I wouldn't say this is very high variance, but just for the sake of argument let's say relative to bias it is and thus we need to try and solve this problem of high variance. \n    A solution for this is regularization. We're going to use Dropout regularization in our next model. Stay 'Tuned'.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"es = EarlyStopping(monitor='val_loss', mode = 'min', patience = 12, restore_best_weights=True)\nrlrop = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, min_lr=0.0000001)\n\nmodel = Sequential()\n\nmodel.add(Conv2D(64, (3,3), activation = 'relu', padding = 'same', input_shape=(28,28,1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(64, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D((2,2), strides = ((2,2))))\n\nmodel.add(Conv2D(128, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D((2,2), strides = (2,2)))\n\nmodel.add(Conv2D(256, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(256, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(512, (3,3), activation = 'relu', padding = 'same'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D((2,2), strides = (2,2)))\nmodel.add(Flatten())\n\nmodel.add(Dense(4096, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.5))\nmodel.add(Dense(4096,activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])\nmodel.summary()\nhistory = model.fit(cnn_train, y_train, validation_split=0.01, epochs = 50, batch_size=256, callbacks = [es, rlrop])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize = (20,10))\n\nax[0].plot(history.history['accuracy'], label = 'Training Accuracy', color='r')\nax[0].plot(history.history['val_accuracy'],label = 'Validation Accuracy', color='b')\nax[0].legend(loc='lower right')\n\nax[1].plot(history.history['loss'], label = 'Training Loss', color='r')\nax[1].plot(history.history['val_loss'], label = 'Validation Loss', color = 'b')\nax[1].legend(loc='upper right')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Interpreting CNN Model 2\n    As you saw, our CNN model after adding a few Dropout Layers improved the validation accuracy to 0.9976.\n    Usually this comes at a small cost. A slight reduction in the training accuracy. That's due to regularization to prevent overfitting on the training data. But I lucky that didn't happen in this case because we had lots of training data in hand.\n    We now save our model and submit our predictions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('cnn_model.hdf5')\npredictions = model.predict_classes(cnn_test)\nsubmission.Label = predictions\nsubmission.to_csv('/kaggle/working/CNN_submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CNN gave an accuracy of 99.507% on the test set on submission. I rank 421 on the Leaderboard right now.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Thank You\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}