{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Digit Recognizer\n## Description\n\n* data files contain gray-scale images of hand-drawn digits\n* Digits range from zero through nine.\n* Each image is 28x28\n* The training data set has 785 columns\n* First column (label) is the target\n* Each pixel column in the training is named pixelx, where x is an integer between 0 and 783, inclusive\n* The test data set contains 28000 images\n* Output columns should be ImageId and Label\n* Evaluation metric is the categorization accuracy","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import itertools\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport warnings\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.utils.np_utils import to_categorical\nfrom keras.optimizers import RMSprop\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nSEED = 7\nSPLIT = 0.1\nEPOCHS = 40\nBATCH_SIZE = 100\nnp.random.seed(SEED)\nwarnings.filterwarnings('ignore')\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Data Preparation\n### Load the Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read the CSV\nmnist_train_path = '../input/digit-recognizer/train.csv'\nmnist_test_path = '../input/digit-recognizer/test.csv'\n\ntrain_original = pd.read_csv(mnist_train_path)\ntest_original = pd.read_csv(mnist_test_path)\n\n# Preserve original\nX_train = train_original.drop('label', axis=1).copy()\nX_test = test_original.copy()\ny_train = train_original['label'].copy()\nprint('Train data shape: {}'.format(X_train.shape))\nprint('Test data shape: {}'.format(X_test.shape))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Label Distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots()\nfig.set_size_inches(11, 8)\nfig=sns.countplot(y_train)\nplt.xlabel('Labels')\nplt.ylabel('Count')\nplt.title('Distribution of Labels')\nplt.show(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Normalization\n\nI´m going to perform a grayscale normalization to reduce color intensity differences, and to scale the data in the 0-1 range.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train / 255.0\nX_test = X_test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reshape to 28x28","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Sample Image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots()\nfig.set_size_inches(11, 8)\nfig = plt.imshow(X_train[1][:,:,0])\nplt.title('Sample Image')\nplt.show(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Make Train-Validation Split","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X_train, y_train,\n                                                  test_size = SPLIT,\n                                                  random_state=SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encode the Labels","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train, num_classes=10)\ny_val = to_categorical(y_val, num_classes=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build the CNN Model\n\nHere, I´m using the Keras Sequential API, building one layer at a time from input to output.\n\nThe layers are:\n    * A pair of Conv2D layers with 32 filters\n    * A MaxPool2D layer\n    * A Dropout layer\n    * Two more Conv2D layers with 64 filters\n    * A MaxPool2D layer\n    * A Dropout layer\n    * A Flatten layer\n    * A RELU activation layer\n    * A SoftMax output layer\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation = \"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get optimizer\noptimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)\n\n# Compile the model\nmodel.compile(optimizer = optimizer , loss = 'categorical_crossentropy', metrics=['accuracy'])\n\n# Make learning rate annealer\nlrr = ReduceLROnPlateau(monitor='val_acc', \n                        patience=3, \n                        verbose=1, \n                        factor=0.5, \n                        min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augment the Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,\n        samplewise_center=False,\n        featurewise_std_normalization=False,\n        samplewise_std_normalization=False,\n        zca_whitening=False,\n        rotation_range=10,\n        zoom_range = 0.12, \n        width_shift_range=0.12,\n        height_shift_range=0.12,\n        horizontal_flip=False,\n        vertical_flip=False)\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fit the Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(datagen.flow(X_train,y_train, batch_size=BATCH_SIZE),\n                              epochs = EPOCHS,\n                              validation_data = (X_val,y_val),\n                              verbose = 2,\n                              steps_per_epoch=X_train.shape[0] // BATCH_SIZE,\n                              callbacks=[lrr])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate the Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1)\nfig.set_size_inches(11, 8)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"Validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Predict and Plot Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Helper method for confusion matrix\ndef plot_confusion_matrix(confusion_mat, classes):\n    fig, ax = plt.subplots()\n    fig.set_size_inches(11, 8)\n    fig = plt.imshow(confusion_mat, interpolation='nearest')\n    plt.title('Confusion matrix')\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes)\n    plt.yticks(tick_marks, classes)\n    plt.ylabel('Actual label')\n    plt.xlabel('Predicted label')\n    plt.show(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = model.predict(X_val)\ny_pred_classes = np.argmax(y_pred,axis = 1) \ny_actual = np.argmax(y_val,axis = 1) \nconfusion_mat = confusion_matrix(y_actual, y_pred_classes) \nplot_confusion_matrix(confusion_mat, classes = range(10)) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict and Submit","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"results = model.predict(X_test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name='Label')\nsub = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsub.to_csv('mnist_submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}