{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **The MNIST dataset is often being referred to as the \"Hello World!\" of Machine Learning**\n\n**In this kernel I have demonstrated how to step by step design a model to tackle the MNIST challenge**\n\nA special regard to Francis Chollet (Author of Keras) for writing an amazing book, \"Deep Learning with Python\".\n\nLet us begin!\n\nI have used the following techniques which will get you a step by step increment on the test set accuracy\n\n**1. Random Forest Algorithm (93.5% on the Test set)**\n\n> A limited parameter (max_depth = 20) ensemble of only 20 Decision Trees (default value is 100) which prevents overfitting and provides satisfactory accuracy on the validation set (around 95%) \n\n**2. A Simple Convolutional Neural Network (98.5% on the Test set)**\n\n> A simple CNN with minimal parameters. No advanced techniques like Batch Normalization, Learning Rate Annealer, or Data Augmentation. The validation accuracy will be around 99.1% (see previous versions of the kernel)\n\n**3. CNN with Data Augmentation (99.35% on the Test set)**\n\n> Apply Data Augmentation on the CNN along with learning rate annealer and Nadam optimizer. The validation accuracy will be around 99.48% at 20 epochs\n\n**4. An ensemble of CNNs (99.67% on the Test set)**\n\n> Combine 7 previously designed CNNs to work on random shuffled subsets of training and validation data. The ensemble technique used is Bagging. The validation accuracy achieved is 99.55%. To touch the test set accuracy mark of 99.7% you just have to increase the number of CNNs in the ensemble\n\n**5. We can use some regularization techniques to achieve 99.7% which seems to be accepted limit for the competition**\n\n> Increase the number of CNNs in the ensemble to 10 and apply regularization techniques to achive 99.7%. One can also aggregate the predictions of 5 different CNN architectures instead of using multiple same CNN implementations.\n\n**6. We have used multiple ML and DL algorithms & techniques to continuously increase our test set accuracy. Some more architectures can be experimented with, to touch the golden mark of 99.7%. But the problem is you still would not win the competition. Because some kagglers have been cheating to get a perfect test score of 100%.**\n\n**I first read about this in Aurelion Geron's book \"Hands on ML\". The book mentions this article to elaborate on how someone can achive a perfect 100% accuracy on the test set. **\n\nhttps://www.kaggle.com/c/digit-recognizer/discussion/61480\n\n> **So how can you cheat and hit the bullseye**\n> \n> **Train a model on the entire MNIST dataset (70K images) and then use it to make predictions**\n> \n> **A lot of notebooks use an ensemble CNN or simple CNN with the full MNIST dataset for a perfect score.**\n> \n> **But wait! If cheating is what you want to do, maybe do it the smart way. Simply use a Decision Tree Algorithm with (max_depth = None). It is in the nature of Decision Trees to overfit the training data.**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Importing the libraries\n\nimport numpy as np \nimport matplotlib.pyplot as plt\nimport pandas as pd \n\nimport tensorflow as tf\nfrom tensorflow import keras\n\nfrom sklearn.model_selection import train_test_split\n\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import LearningRateScheduler,ReduceLROnPlateau\n\nfrom tensorflow import keras  \nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.callbacks import LearningRateScheduler,ReduceLROnPlateau\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dense, MaxPool2D, Dropout, Flatten\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Importing the training dataset\n\ndataset = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Inspecting the dataset\n\ndataset.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking for missing values\n\nnp.any(dataset.isnull().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Slicing the dataset to separate feature matrix 'X' and the vector of predictions 'y'\n\nX = dataset.iloc[:, 1:].values\ny = dataset.iloc[:, 0].values\n\nX.shape, y.shape, X.dtype, y.dtype","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing the dataset by reshaping an image into the original format i.e. 28 * 28\n\nfirst_image = X[0]\nfirst_image = first_image.reshape((28, 28))\n\nplt.imshow(first_image)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Grayscaling the image, dropping the axis and printing the label\n\nplt.imshow(first_image, \"binary\")\nplt.title('label : {}'.format(y[0]))\nplt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting multiple randomly chosen images from the dataset for insights\n\nrandom_indexes = np.random.choice(range(len(X)), 25)\nprint(\"Random Indexes : \", random_indexes)\n\nX_random = X[random_indexes]\ny_random = y[random_indexes]\n\nprint(\"\\n Shape : \", X_random.shape)\n\nplt.figure(figsize = (16, 10))\nfor i in range(25):\n    image = X_random[i]\n    image = image.reshape((28, 28))\n    plt.subplot(5, 5, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.imshow(image, \"binary\")\n    plt.title('label : {}'.format(y_random[i]))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Applying feature scaling for faster convergence\n\nfrom sklearn.preprocessing import StandardScaler\nscaler = StandardScaler()\nX = scaler.fit_transform(X)\nX","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing the test set\n\ntest_data = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nX_test = test_data.values\nX_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Scaling the test set\n\nX_test_scaled = scaler.fit_transform(X_test)\nX_test_scaled","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshaping the data for Image Generator (Rank = 4)\n\nX = X.reshape((42000, 28, 28, 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Applying Data Augmentation\n\ndatagen = ImageDataGenerator(rotation_range=10,  zoom_range = 0.1,  width_shift_range=0.1,  height_shift_range=0.1)\n# datagen.fit(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# An empty list to store the ensemble of 10 CNNs\n\nmodel_list = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Learning rate annealer\n\nreduce_lr = ReduceLROnPlateau(monitor='val_accuracy', \n                                patience=3, \n                                verbose=1, \n                                factor=0.2, \n                                min_lr=1e-6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating 7 objects of the same CNN architecture and saving in model_list\n\nfor i in range(7):\n    model = keras.models.Sequential([\n    keras.layers.Conv2D(32, kernel_size = 3, activation = 'relu', padding = 'same', input_shape = [28, 28, 1]),\n    keras.layers.BatchNormalization(),\n    keras.layers.Conv2D(64, kernel_size = 3, activation = 'relu', padding = 'same'),\n    keras.layers.MaxPool2D(),\n    keras.layers.Conv2D(32, kernel_size = 3, activation = 'relu', padding = 'same'),\n    keras.layers.BatchNormalization(),\n    keras.layers.Conv2D(64, kernel_size = 3, activation = 'relu', padding = 'same'),\n    keras.layers.MaxPool2D(),\n    keras.layers.Flatten(),\n    keras.layers.Dropout(0.25),\n    keras.layers.Dense(256, activation = 'relu'),\n    keras.layers.Dropout(0.5),\n    keras.layers.Dense(10, activation = 'softmax')])\n\n    model.compile(loss = 'sparse_categorical_crossentropy', optimizer = 'nadam', metrics = ['accuracy'])\n    model_list.append(model)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Verifying the hashcodes\n\nmodel_list","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshaping to a tensor of rank 4\n\n# X = X.reshape((42000, 28, 28, 1))\nX.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training all the 7 CNNs in the ensemble together\n\nhistory = [0] * 7\n\nfor i in range(7):\n    X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size = 0.08)\n    history[i] = model_list[i].fit_generator(datagen.flow(X, y, batch_size = 64), \n                                             epochs = 20, validation_data = (X_valid, y_valid), callbacks = [reduce_lr])\n    print(\"CNN : {} Maximum Train Accuracy : {} Maximum Validation Accuracy : {}\".format(i+1, max(history[i].history['accuracy']), max(history[i].history['val_accuracy'])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating a prediction tensor for applying the bagging technique\n\nensemble_cnn_pred = np.zeros((X_test.shape[0], 10))\nensemble_cnn_pred.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reshaping the test set in a Rank 4 tensor\n\nX_test_scaled = X_test_scaled.reshape((28000, 28, 28, 1))\nX_test_scaled.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generating and aggregating predictions\n\nfor i in range(7):\n  ensemble_cnn_pred = ensemble_cnn_pred + model_list[i].predict(X_test_scaled)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Verifying the predictions of the ensemble\n\nnp.sum(ensemble_cnn_pred[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Aggregating the predictions\n\nensemble_cnn_pred = np.argmax(ensemble_cnn_pred, axis = 1)\nensemble_cnn_pred[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Verifying the shape\n\nensemble_cnn_pred.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Saving the predictions in a dataframe\n\npred_df_ensemble_cnn = pd.DataFrame(columns = ['ImageId', 'Label'])\npred_df_ensemble_cnn['ImageId'] = np.arange(1, 28001)\npred_df_ensemble_cnn['Label'] = ensemble_cnn_pred\npred_df_ensemble_cnn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Writing the predictions in a csv file\n\npred_df_ensemble_cnn.to_csv('ens_cnn_with_aug_sub.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> I sincerely hope you found something useful by reading this kernel!\n> \n> Happy Kaggling!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}