{"cells":[{"metadata":{"_uuid":"923587cb73e710b5c197c1b7bfd122655e7d67eb"},"cell_type":"markdown","source":"# Welcome!\nIn this kernel I will walk you through some basic exploratory data analysis (EDA) and how to set up a simple convolutional neural network to perform image classification. Hope you find this helpful. Please feel free to comment below if you have any questions."},{"metadata":{"_uuid":"bfe6413bbc20470fa3695f4264e97bcf9db91d99"},"cell_type":"markdown","source":"## 1. Setting up our environment\n\nLet's get started by setting up our environment. We will be using the Keras environment to set up our convolutional neural network (CNN). In addition to Keras we will be using numpy and pandas to help us manipulate and set up our data for our model. Finally, we will be using matplotlib and seaborn for plotting and visualization If you are unfamiliar with any of these, please take advantage of the links below.\n\n* References:\n    * Keras: https://keras.io/\n    * Numpy: https://docs.scipy.org/doc/\n    * Pandas: http://pandas.pydata.org/pandas-docs/stable/\n    * Matplotlib: https://matplotlib.org/index.html\n    * Seaborn: https://seaborn.pydata.org/"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Import numpy and pandas\nimport numpy as np \nimport pandas as pd\n\n# We'll use this later for splitting our data up into training and validation sets\nfrom sklearn.model_selection import train_test_split\n\n# Import matplotlib and sns \nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\n\n# Get everything we need from Keras\nfrom keras import layers\nfrom keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, MaxPooling2D, AveragePooling2D, Dropout\nfrom keras.models import Model\nfrom keras.initializers import glorot_uniform\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.preprocessing.image import ImageDataGenerator\nimport keras.backend as K\nK.set_image_data_format('channels_last')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e09162d1d8fd118ee8239cb9d9db583277da2893"},"cell_type":"markdown","source":"## 2. Loading our data and exploring it\nSo now that we have our environment set up, let's load our data and start exploring it. First we use pandas to read in the data. The data provided here is stored in the form of a CSV file. Each row consists of a label (0, 1, 2, 3, 5, ..., 9) and 748 pixel intensity values. Run the code in the cell below this to load the data."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"f733d66eccecee507de2efe1457b9ee5df1bd376"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a10da4bf5b64ffbad438923bba2a60f693ada8ef"},"cell_type":"markdown","source":"Our data is now stored as a pandas 'DataFrame'. We can take advantage of the 'head' attribute to get a glimpse at what our raw data looks like."},{"metadata":{"trusted":true,"_uuid":"503087e1c24b79f3c881f4053b41d6a947be090c"},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2327dea75b7abac22684befdd4d58506a616db4e"},"cell_type":"markdown","source":"Next on the adgenda is to see how many examples of each class we have. For this we use the 'countplot' function available through Seaborn. Go ahead and run the code below. We can see that we have roughly the same amount of training examples for each class. This is good! When training a model you want to have a uniform amount of examples from each class. If this is not the case, then you have a class imbalance problem. There are several methods available for fixing that issue. See the link below for more on that.\n\n* References:\n    * Countplot: https://seaborn.pydata.org/generated/seaborn.countplot.html?highlight=countplot#seaborn.countplot\n    * Class imbalance problem: https://towardsdatascience.com/dealing-with-imbalanced-classes-in-machine-learning-d43d6fa19d2\n\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Get number of training examples\nprint('Number of training examples: ' + str(train.shape[0]))\n\n# Plot the number of examples per class in our training set\nsns.countplot(train['label'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"283c6b67fd55881122d50eb72b724dc5e4536e08"},"cell_type":"markdown","source":"We have our data loaded and we know that it is evenly distributed among our different classes, but what does each example actually look like? Let's find out! The code below generates a set of 25 random examples, reshapes those examples from a 1 x 748 array to a 28 x 28 array, and then displays that array using matplotlib's 'imshow' function. \n\n* References:\n    * np.reshape: https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html\n    * imshow: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.imshow.html"},{"metadata":{"trusted":true,"_uuid":"c10bcb230140f50dd93b6b754e180ee92cbdebcd"},"cell_type":"code","source":"# Get a list of 25 random examples\nexample_ids = np.random.randint(28000, size = 25)\n\n# Set up your figure\ncnt = 1\nfig = plt.figure()\ntitle = fig.suptitle('MNIST Handwritten Digit Examples', fontsize = 20)\ntitle.set_position([.5, .92])\nfig.set_figheight(12)\nfig.set_figwidth(10)\nfor id in example_ids:\n    # Get the data associated with each example\n    example_data = train.iloc[[id]]\n    \n    # Get the label associated with each example\n    example_truth = example_data['label'].unique()[0]\n    \n    # Drop the label so we are only left with the image data\n    example_data = example_data.drop(['label'], axis = 1)\n    \n    # Convert the data to a numpy array so that it's compatible with imshow\n    example_data = np.array(example_data)\n    \n    # Reshape the example to a 28 x 28 array\n    example_data = np.reshape(example_data, [28, 28])\n    \n    # Plot that array in a subplot\n    plt.subplot(5, 5, cnt)\n    plt.title('Label = ' + str(np.array(example_truth)))\n    plt.axis('off')\n    plt.imshow(example_data)\n    cnt += 1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"092876bd86204890d1d90d8dabad404c1a6aa904"},"cell_type":"markdown","source":"## 3. Setting up our CNN\nNow for the fun part! Let's go ahead and set up the CNN that we will be using to perform image classification. Our CNN will have the following structure:\n\nINPUT - > CONV -> BN -> RELU -> CONV -> BN -> RELU -> MAXPOOL ->  DROPOUT -> CONV -> BN -> RELU -> CONV -> BN -> RELU -> MAXPOOL -> DROPOUT -> FULLY CONNECTED -> DROPOUT -> FULLY CONNECTED -> DROPOUT -> SOFTMAX\n\nIf you are unfamiliar with concepts such as convolutional layers, batch normalization, relu, and pooling, I would recommend taking the following course on CNNs.\n\n* References:\n    * Convolutional Neural Networks: https://www.coursera.org/learn/convolutional-neural-networks"},{"metadata":{"trusted":true,"_uuid":"465574a2c280c6cda2f94035221cb22a8b9038cc","collapsed":true},"cell_type":"code","source":"def digit_conv_net(input_shape, num_classes):\n    \n    # Define the input placeholder as a tensor with shape input_shape. Think of this as your input image!\n    X_input = Input(input_shape)\n\n    # Zero-Padding: pads the border of X_input with zeroes\n    X = ZeroPadding2D((2, 2))(X_input)\n\n    # CONV -> BN -> RELU Block applied to X\n    X = Conv2D(32, (5, 5), strides = (1, 1), name = 'conv0', padding = 'same', kernel_initializer = glorot_uniform(seed = 0))(X)\n    X = BatchNormalization(axis = 3, name = 'bn0')(X)\n    X = Activation('relu')(X)\n    \n    # CONV -> BN -> RELU Block applied to X\n    X = Conv2D(32, (5, 5), strides = (1, 1), name = 'conv1', padding = 'same', kernel_initializer = glorot_uniform(seed = 0))(X)\n    X = BatchNormalization(axis = 3, name = 'bn1')(X)\n    X = Activation('relu')(X)\n\n    # MAXPOOL\n    X = MaxPooling2D(pool_size = (2, 2), name = 'max_pool0')(X)\n    \n    # Add dropout to regularize\n    X = Dropout(0.25)(X)\n    \n    # CONV -> BN -> RELU Block applied to X\n    X = Conv2D(64, (5, 5), strides = (1, 1), name = 'conv2', padding = 'same', kernel_initializer = glorot_uniform(seed = 0))(X)\n    X = BatchNormalization(axis = 3, name = 'bn2')(X)\n    X = Activation('relu')(X)\n    \n    # CONV -> BN -> RELU Block applied to X\n    X = Conv2D(64, (5, 5), strides = (1, 1), name = 'conv3', padding = 'same', kernel_initializer = glorot_uniform(seed = 0))(X)\n    X = BatchNormalization(axis = 3, name = 'bn3')(X)\n    X = Activation('relu')(X)\n\n    # MAXPOOL\n    X = MaxPooling2D(pool_size = (2, 2), name = 'max_pool1')(X)\n    \n    # Add dropout to regularize\n    X = Dropout(0.25)(X)\n    \n    # FLATTEN X -> FULLY CONNECTED\n    X = Flatten()(X)\n    X = Dense(128, activation = 'relu', name = 'fc0')(X)\n    X = Dropout(0.25)(X)\n    X = Dense(128, activation = 'relu', name = 'fc1')(X)\n    X = Dropout(0.25)(X)\n    X = Dense(num_classes, activation = 'softmax', name = 'fc2')(X)\n\n    # Create model. This creates your Keras model instance, you'll use this instance to train/test the model.\n    model = Model(inputs = X_input, outputs = X, name = 'LeNet5')\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be213cb5cbf1a2ddab35ff43f6d6f7b5aaedd5b1"},"cell_type":"markdown","source":"We've defined the basic structure of our CNN. Now let's initilize it with the input size and number of classes. Once that is done, we complie the model.\n\n* References\n    * Compile: https://keras.io/models/model/"},{"metadata":{"trusted":true,"_uuid":"3ca5898f7777edab6d6b108046782f43468c7418","collapsed":true},"cell_type":"code","source":"# Initialize model with input size and number of classes\nDigitModel = digit_conv_net([28, 28, 1], 10)\n\n# Complie the model. Use Adam optimizer.\nDigitModel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b82fe46421ef052c9eb6ee1b589163a86327b970"},"cell_type":"markdown","source":"You can use the 'summary' attribute to see the structure of your model and get the number of parameters you will be training.\n\n* References:\n    * Summary: https://keras.io/models/about-keras-models/#about-keras-models"},{"metadata":{"trusted":true,"_uuid":"b0c834d924edcd5a22d27d6bd94c63898522d5a6"},"cell_type":"code","source":"DigitModel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f50fcb05a48a91f232eeb80653d377cc37850fd"},"cell_type":"markdown","source":"### 3.1. Preprocessing our data\n\nOur model has now been initialized and compiled. Now we need to set up our data so that we can feed it into the model. Do do this we separate our labels from our pixel intensity values. Our labels will become our target (stored as y) and our intensity values will be reshaped into 28 x 28 images. Before that happens, however, we will use the 'train_test_split' function to break our data into a training set and a validation set. Finally, we divide our pixel intensities by 255 to normalize them. This last step scales the data so that every pixel value is between 0 and 1. This speeds up optimization.\n\n* References:\n    * train_test_split: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"},{"metadata":{"trusted":true,"_uuid":"cf6d127974e447e996838979cee9bbe3f774993a","collapsed":true},"cell_type":"code","source":"train_labels = train['label']\ntrain_labels = pd.get_dummies(train_labels)\ntrain_features = train.drop(['label'], axis = 1)\nX_train, X_val, y_train, y_val = train_test_split(train_features,\n                                                   train_labels,\n                                                   train_size = 0.98,\n                                                   test_size = 0.02,\n                                                   random_state = 0)\nX_train = np.array(X_train)\nX_train = np.reshape(X_train, [X_train.shape[0], 28, 28, 1]) / 255.\n\nX_val = np.array(X_val)\nX_val = np.reshape(X_val, [X_val.shape[0], 28, 28, 1]) / 255.\n\ntest = np.array(test)\ntest = np.reshape(test, [test.shape[0], 28, 28, 1]) / 255.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d17c008827ed0f3cf8ebd30a9782251249372aac"},"cell_type":"markdown","source":"### 3.2. Some finishing touches...callbacks and data augmentation\n\nAt this point, our data and our model are ready to go. However, before we go any further, I'd like to showcase two particularly nice features in Keras: Callbacks and real-time data augmentation. A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. Here we use the 'ReduceLROnPlateau callback. In a nutshell, this monitors a parameter of our choice (i.e. the validation accuracy) and reduces the learning rate for our optimizer if our parameter starts to plateau. In this instance, the following inputs are given:\n\n* monitor - What parameter our model will monitor\n* factor - What we multiply the learning rate by in the event of a plateau\n* patience - How many epochs we are willing to wait before we reduce the learning rate\n* min_lr - How low we are willing to let the learning rate go. Do not set this too low. It will slow down your optimizer.\n\n* Refernces:\n    * Callbacks: https://keras.io/callbacks/"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"dffb7a01b5d095acf432079bd79021fec4e94077"},"cell_type":"code","source":"ReduceLearningRate = ReduceLROnPlateau(monitor = 'val_acc', \n                                       factor = 0.5, \n                                       patience = 3, \n                                       min_lr = 0.00001)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f065c168aef7e4446013ed2ce25c2e805bc3ba8"},"cell_type":"markdown","source":"Data augmentation is the process of applying small changes to our existing data set to generate more data. Take the kitten example below. From the original image, we can change the height and width, rotate, crop, and flip it to generate more training examples. Keras has a function called 'ImageDataGenerator' that applies these transformations for us in real time. These new examples are fed into our model during training. I would highly recommend going to the link below to see what options are available on Keras.\n\n* References:\n    * ImageDataGenerator: https://keras.io/preprocessing/image/\n\n![](https://cdn-images-1.medium.com/max/1600/1*C8hNiOqur4OJyEZmC7OnzQ.png)\n\nImage source: https://medium.com/nanonets/how-to-use-deep-learning-when-you-have-limited-data-part-2-data-augmentation-c26971dc8ced"},{"metadata":{"trusted":true,"_uuid":"8ba92f9c7b30eb104209479654f1f6e2bc209235","collapsed":true},"cell_type":"code","source":"DataGenerator = ImageDataGenerator(rotation_range = 10,\n                                   zoom_range = 0.1,\n                                   width_shift_range = 0.1,\n                                   height_shift_range = 0.1)\n\n# Fits the model on batches with real-time data augmentation\nDataGenerator.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8fab53c3489af5f53ae37f215884be0c9cceff4"},"cell_type":"markdown","source":"## 4. Fire it all up!\n\nNow we are ready to go. We've defined our model, precprocessed our data, defined a callback to monitor the progress of our model, and we've set our model up with data augmentation. Run the block of code below and enjoy the show. For better performance, increase the number of epochs (25-30 epochs should do the trick)."},{"metadata":{"trusted":true,"_uuid":"4d54a541d0e37f237c760d3338a2f47abdd84c7c"},"cell_type":"code","source":"# Store this in a variable called mdl which we will use later to visualize the model performance\nmdl = DigitModel.fit_generator(DataGenerator.flow(X_train, y_train, batch_size = 32), \n                               epochs = 15,\n                               validation_data = (X_val, y_val),\n                               callbacks = [ReduceLearningRate])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8bfb753c292c82930a35b9adf38cb36e1475a00"},"cell_type":"markdown","source":"## 5. Checking our work\n\nOur model is fully trained and we are ready to run our test data through it. However, before we do that, we can use the block of code below to check for any underfitting or overfitting. Earlier we split the data up into a training and validation sets. Below you have two plots. One shows training and validation accuracy vs epochs, and the other shows training and validation loss vs epochs. Ideally you want to see your training and validation accuracy go up AND remain close to each other. Similarly, you want to see your training and validation loss go down AND remain close to each other. This lets you know that your model is generalizing well to data that it's never seen before.\n"},{"metadata":{"trusted":true,"_uuid":"62f15bd7b893346248411ae62ea6d15c1386336f"},"cell_type":"code","source":"fig = plt.figure()\ntitle = fig.suptitle('Model Accuracy/Loss Summary', fontsize = 16)\ntitle.set_position([.5, 1.0])\nfig.set_figheight(5)\nfig.set_figwidth(10)\n\n# Summarize history for accuracy\nplt.subplot(1, 2, 1)\nplt.plot(mdl.history['acc'])\nplt.plot(mdl.history['val_acc'])\nplt.title('Model Accuracy', fontsize = 10)\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='lower right')\n\n# Summarize history for loss\nplt.subplot(1, 2, 2)\nplt.plot(mdl.history['loss'])\nplt.plot(mdl.history['val_loss'])\nplt.title('Model Loss', fontsize = 10)\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='upper right')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"061b5b4e146911c4d70e1e054f6fbc0dc8061d95"},"cell_type":"markdown","source":"## 6. Submit your work!\n\nYou can see above that our model is doing very well! Both the loss and and accuracy for our training and validation sets are similar. Feel free to use this code or play around with it even more. Hope this helps, and please feel free to leave any questions in the comments section."},{"metadata":{"trusted":true,"_uuid":"70170687fea28f406583258f4eeab706f84b93a8","collapsed":true},"cell_type":"code","source":"# predict results\nresults = DigitModel.predict(test)\n\n# select the index with the maximum probability\nresults = np.argmax(results, axis = 1)\nresults = pd.Series(results, name = \"Label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a87cc14419ae829477f4d48128ffffdb0c6303ed","collapsed":true},"cell_type":"code","source":"my_submission = pd.concat([pd.Series(range(1,28001), name = \"ImageId\"), results], axis = 1)\nmy_submission.to_csv(\"my_submission.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"df1dc65903566957a8c71103a8d842f13e580199"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}