{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST: Simple CNN with Keras (top 9%)\n### A top 9% submission with an accuracy of 0.99685\n\n* **1. Import**\n* **2. Data preparation**\n    * 2.1 Load data\n    * 2.2 Inspect data\n    * 2.3 Visualize data\n    * 2.4 Normalise and reshape data\n    * 2.5 Split training and valdiation set\n* **3. Convolutional Neural Network**\n    * 3.1 Define the model\n    * 3.2 Data augmentation\n    * 3.3 Fit model\n* **4. Evaluate model**\n    * 4.1 Confusion matrix\n    * 4.2 Training and validation curves\n* **5. Prediction and submission**\n    * 5.1 Prediction \n    * 5.2 Submission\n    * 5.3 Save model\n* **6. References**\n\n"},{"metadata":{},"cell_type":"markdown","source":"## 1. Import\nImport necessary libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt #for plotting\nfrom collections import Counter\nfrom sklearn.metrics import confusion_matrix\nimport itertools\nimport seaborn as sns\n\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Data preparation\n### 2.1 Load data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2 Inspect data"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape)\nprint(test.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.3 Visualize data\nNumber of observations per label"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Counter(train['label']))\nsns.countplot(train['label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Properly define pixels and labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = (train.iloc[:,1:].values).astype('float32')\ny_train = train.iloc[:,0].values.astype('int32')\n\nx_test = test.values.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observe some digit examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nplt.figure(figsize=(12,6))\nx, y = 10, 4\nfor i in range(40):\n    plt.subplot(y, x, i+1)\n    plt.imshow(x_train[i].reshape((28,28)), interpolation='nearest')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"See an example of how pixel values are defined on the gray scale"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_input(img, ax):\n    ax.imshow(img, cmap='gray')\n    width, height = img.shape\n    thresh = img.max()/2.5\n    for x in range(width):\n        for y in range(height):\n            ax.annotate(str(round(img[x][y],2)), xy=(y,x),\n                        horizontalalignment='center',\n                        verticalalignment='center',\n                        color='white' if img[x][y]<thresh else 'black')\n\nx_train=(train.iloc[:,1:].values).astype('int32')\nfig = plt.figure(figsize = (12,12)) \nax = fig.add_subplot(111)\nvisualize_input(x_train[1].reshape(28,28), ax)\nx_train=(train.iloc[:,1:].values).astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.4 Normalise and reshape data"},{"metadata":{},"cell_type":"markdown","source":"In general, neural nets perform better if we normalise data"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train/255.0\nx_test = x_test/255.0\ny_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reshape data to match Keras' expectations"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('x_train shape:', x_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = x_train.reshape(x_train.shape[0], 28, 28, 1)\nX_test = x_test.reshape(x_test.shape[0], 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.5 Split training and validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"num_classes = 10\n\ny_train = keras.utils.to_categorical(y_train, num_classes)\n\nX_train, X_val, Y_train, Y_val= train_test_split(\n    X_train, y_train, test_size = 0.1, random_state = 42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Convolutional Neural Network\n### 3.1 Define the model\nThere is no single right way to define your CNN - number of epochs, batch size and other hyperparameters vary from problem to problem, though ReLU seems to be chosen as the standard activation function now, as the ReLU function avoids the vanishing gradients problem in neural networks. Furthermore, the Adam optimizer seems to be the most widely used optimizer at the moment."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 64\nepochs = 20\ninput_shape = (28, 28, 1)\n\nmodel = Sequential()\n\nmodel.add(Conv2D(32, kernel_size = (3,3), activation = 'relu', \n                kernel_initializer = 'he_normal', input_shape = input_shape))\nmodel.add(Conv2D(32, kernel_size = (3,3), activation = 'relu', \n                kernel_initializer = 'he_normal'))\nmodel.add(MaxPool2D((2, 2)))\nmodel.add(Dropout(0.20))\nmodel.add(Conv2D(64, (3, 3), activation='relu',\n                 padding='same', kernel_initializer='he_normal'))\nmodel.add(Conv2D(64, (3, 3), activation='relu',\n                 padding='same',kernel_initializer='he_normal'))\nmodel.add(MaxPool2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\nmodel.add(Conv2D(128, (3, 3), activation='relu',\n                 padding='same',kernel_initializer='he_normal'))\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\nmodel.add(Dense(128, activation = 'relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.25))\nmodel.add(Dense(num_classes, activation = 'softmax'))\n          \nmodel.compile(loss=keras.losses.categorical_crossentropy,\n                       optimizer = keras.optimizers.Adam(),\n                         metrics = ['accuracy'])\n\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc',\n                                           patience = 3,\n                                           verbose = 1,\n                                           factor = 0.5,\n                                           min_lr = 0.0001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2 Data augmentation\n\nWe augment data e.g. by randomy rotating the pictures. This expands our dataset and adds noise which helps us avoid overfitting."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(featurewise_center=False, # set input mean to 0 over the dataset\n                            samplewise_center = False, # set each sample mean to 0\n                            featurewise_std_normalization = False, # divide inputs by std of the dataset\n                            samplewise_std_normalization = False, # divide each input by its std\n                            zca_whitening = False, # apply ZCA whitening\n                            rotation_range = 15, # randomly rotate images in the range (degrees, 0 to 180)\n                            zoom_range = 0.1, # Randomly zoom image \n                            width_shift_range = 0.1, # randomly shift images horizontally (fraction of total width)\n                            height_shift_range = 0.1, # randomly shift images vertically (fraction of total height)\n                            horizontal_flip = False, # randomly flip images\n                            vertical_flip = False) # randomly flip images - we do not want this as it e.g. messes up the digits 6 and 9","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.3 Fit model"},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen.fit(X_train)\nh = model.fit_generator(datagen.flow(X_train, Y_train, batch_size = batch_size),\n                       epochs = epochs,\n                       validation_data = (X_val, Y_val),\n                       verbose = 1,\n                       steps_per_epoch = X_train.shape[0] // batch_size,\n                       callbacks = [learning_rate_reduction],)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_loss, final_acc = model.evaluate(X_val, Y_val, verbose=0)\nprint(\"validation loss: {0:.6f}, validation accuracy: {1:.6f}\".format(final_loss, final_acc))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Evaluate model\n### 4.1 Confusion matrix\nCode taken from the sklearn website: https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_confusion_matrix(cm, classes,\n                         normalize = False,\n                         title = 'Confusion matrix',\n                         cmap = plt.cm.Blues):\n    plt.imshow(cm, interpolation = 'nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n    \n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n        \n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred, axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val, axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(num_classes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Training and validation curves"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(h.history.keys())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy = h.history['acc']\nval_accuracy = h.history['val_acc']\nloss = h.history['loss']\nval_loss = h.history['val_loss']\n\nepochs = range(len(accuracy))\n\nplt.plot(epochs, accuracy, 'bo', label = 'training accuracy')\nplt.plot(epochs, val_accuracy, 'b', label = 'validation accuracy')\n\nplt.title('training and validation accuracy')\n\nplt.figure()\nplt.plot(epochs, loss, 'bo', label = 'training loss')\nplt.plot(epochs, val_loss, 'b', label = 'validation loss')\nplt.title('training and validation loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Prediction and submission\n### 5.1 Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_classes = model.predict_classes(X_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2 Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submissions = pd.DataFrame({\"ImageId\": list(range(1, len(predicted_classes)+1)),\n                           \"Label\": predicted_classes})\nsubmissions.to_csv(\"mnistSubmission.csv\", index = False, header = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.3 Save model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('my_model_1.h5')\n\njson_string = model.to_json()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. References\n - A great deal of inspiration taken from: https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n - Sklearn confusion matrix: https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}