{"cells":[{"metadata":{"_cell_guid":"b8f23803-f7c9-c890-22dd-eab6672e749c","_uuid":"d05b725c04e3c2351a39b5ebb1f38ee56473e4cd"},"cell_type":"markdown","source":"## Convolutional Neural Networks\nIf you want to apply machine learning to image recognition, convolutional neural networks (CNN) is the way to go. It has been sweeping the board in competitions for the last several years, but perhaps its first big success came in the late 90's when [Yann LeCun][1] used it to solve MNIST with 99.5% accuracy. I will show you how it is done in Keras, which is a user-friendly neural network library for python.\n\nMany other notebooks here use a simple fully-connected network (no convolution) to achieve 96-97%, which is a poor result on this dataset. In contrast, what I will show you here is nearly state-of-the-art. In the Kernel (<20 minutes training) we will achieve 99%, but if you train it overnight (or with a GPU) you should reach 99.5. If you then ensemble over several runs, you should get close to the best published accuracy of 99.77% . (Ignore the 100% results on the leaderboard; they were created by learning the test set through repeat submissions)\n\nHere goes:\n\n\n  [1]: http://yann.lecun.com/exdb/lenet/"},{"metadata":{"_cell_guid":"19d8eb6b-9b71-029c-06e1-7e74975ae669","_uuid":"e92d6f1944cc42c8d04a307760416650c47771fe","trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"8e18fecf-bc7e-dd8d-0894-2601c3f69e97","_uuid":"ed9be9a8d77a3318651ea5c779aa9cb01daac961"},"cell_type":"markdown","source":"If you don't already have [Keras][1], you can easily install it through conda or pip. It relies on either tensorflow or theano, so you should have these installed first. Keras is already available here in the kernel and on Amazon deep learning AMI.\n\n  [1]: https://keras.io/"},{"metadata":{"_cell_guid":"76adcf6f-d109-e47d-d8f0-c7a26e986df5","_uuid":"98744a16c6980b532d876a1e8f219219c2bc10eb","trusted":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.optimizers import Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"249daad0-7a0d-ad43-2bf9-4861ffac0d90","_uuid":"79226bfbd2735f381a96b0f569a4dceeafa63cef","trusted":true,"collapsed":true},"cell_type":"code","source":"train_file = \"../input/train.csv\"\ntest_file = \"../input/test.csv\"\noutput_file = \"submission.csv\"","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"b3a8472f-d3a1-949d-3bb4-2935b53c4f17","_uuid":"a40dc28dfe38b703a7df300a9fe22fce70e04a0f"},"cell_type":"markdown","source":"## Load the data"},{"metadata":{"_cell_guid":"fc8110b9-1087-2221-1399-6d218a494319","_uuid":"a34bbcb824c3fc85d0b26578eb5ed06f20854343"},"cell_type":"markdown","source":"As always, we split the data into a training set and a validation set, so that we can evaluate the performance of our model."},{"metadata":{"_cell_guid":"95cd483d-23ae-fa98-3823-724e41ed4b5b","_uuid":"c6fc3548ab2088ed5b66416deac2d20510282e87","trusted":true,"collapsed":true},"cell_type":"code","source":"raw_data = np.loadtxt(train_file, skiprows=1, dtype='int', delimiter=',')\nx_train, x_val, y_train, y_val = train_test_split(\n    raw_data[:,1:], raw_data[:,0], test_size=0.1)","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"ffbcd0ae-c1a0-b52c-ea4e-5c691e893333","_uuid":"2b47dff539265778c73e36a1caaef3b882940931"},"cell_type":"markdown","source":"Each data point consists of 784 values. A fully connected net just treats all these values the same, but a CNN treats it as a 28x28 square. Thes two graphs explain the difference: It's easy to understand why a CNN can get better results."},{"metadata":{"_cell_guid":"77036419-61ff-a6a5-6b67-fa39f1147d11","_uuid":"f19476ba010c4f0498c283fe54d2c24db91488dc","trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2, 1, figsize=(12,6))\nax[0].plot(x_train[0])\nax[0].set_title('784x1 data')\nax[1].imshow(x_train[0].reshape(28,28), cmap='gray')\nax[1].set_title('28x28 data')","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"dcee4998-a863-3dd9-57f9-0bde1b2f5099","_uuid":"96fb1d0cedcfbf90d20499791725c35e04ab49e4"},"cell_type":"markdown","source":"We now reshape all data this way. Keras wants an extra dimension in the end, for channels. If this had been RGB images, there would have been 3 channels, but as MNIST is gray scale it only uses one.\n\nThis notebook is written for the tensorflow channel ordering. If you have Keras installed for Theano backend, you might start seeing some error message soon related to channel ordering. This can easily be [solved][1].\n\n\n  [1]: https://keras.io/backend/#set_image_dim_ordering"},{"metadata":{"_cell_guid":"c32f7260-7973-31a4-3991-1bbac6c36b47","_uuid":"921b83179eabb35a96c7980f51ecadf53303be3a","trusted":true,"collapsed":true},"cell_type":"code","source":"x_train = x_train.reshape(-1, 28, 28, 1)\nx_val = x_val.reshape(-1, 28, 28, 1)","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"a9604fb0-79b2-eba6-92c3-b1b2b2d59a03","_uuid":"f7744260226cb1b9a5bf8bada5a022f5c9f7df2e"},"cell_type":"markdown","source":"It would be possible to train the net on the original data, with pixel values 0 to 255. If we use the standard initialization methods for weights, however, data between 0 and 1 should make the net converge faster. "},{"metadata":{"_cell_guid":"c291c473-5e0e-a947-bea6-22d5feb81797","_uuid":"920a27df4b97631cf5c79963b539e493f9df8dc3","trusted":true,"collapsed":true},"cell_type":"code","source":"x_train = x_train.astype(\"float32\")/255.\nx_val = x_val.astype(\"float32\")/255.","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"7a60023f-8844-a721-0ff9-39dfd82e0dd5","_uuid":"d6686c10fa43b213ab1a83364e1628f772ffa078"},"cell_type":"markdown","source":"The labels were given as integers between 0 and 9. We need to convert these to one-hot encoding, i.e. a 10x1 array with one 1 and nine 0:s, with the position of the 1 showing us the value. See the example, with the position of the 1 showing the correct value for the digit in the graph above."},{"metadata":{"_cell_guid":"fdc43c64-0655-e91d-0659-ceb97a6f3532","_uuid":"7dac701773ab4ff4988e9ce6b67b7b10dc224cca","trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train)\ny_val = to_categorical(y_val)\n#example:\nprint(y_train[0])","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"71f04810-96c8-bc4d-1938-381e2e4b4ec8","_uuid":"dd6b78e9638684545fbac86e10b219a37ee91a73"},"cell_type":"markdown","source":"## Train the model\n\nKeras offers two different ways of defining a network. We will the Sequential API, where you just add on one layer at a time, starting from the input.\n\nThe most important part are the convolutional layers Conv2D. Here they have 16-32 filters that use nine weights each to transform a pixel to a weighted average of itself and its eight neighbors. As the same nine weights are used over the whole image, the net will pick up features that are useful everywhere. As it is only nine weights, we can stack many convolutional layers on top of each other without running out of memory/time. \n\nThe MaxPooling layers just look at four neighboring pixels and picks the maximal value. This reduces the size of the image by half, and by combining convolutional and pooling layers, the net be able to combine its features to learn more global features of the image. In the end we use the features in two fully-connected (Dense) layers.\n\nBatch Normalization is a technical trick to make training faster. Dropout is a regularization method, where the layer randomly replaces  a proportion of its weights to zero for each training sample. This forces the net to learn features in a distributed way, not relying to much on a particular weight, and therefore improves generalization. 'relu' is the activation function x -> max(x,0)."},{"metadata":{"_cell_guid":"363e8bc8-a731-01da-9a39-d3914ea7a5c8","_uuid":"ef0eab9b9032f8f770b3c47cb70693df4dfe2683","trusted":true,"collapsed":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(filters = 16, kernel_size = (3, 3), activation='relu',\n                 input_shape = (28, 28, 1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 16, kernel_size = (3, 3), activation='relu'))\nmodel.add(BatchNormalization())\n#model.add(Conv2D(filters = 16, kernel_size = (3, 3), activation='relu'))\n#model.add(BatchNormalization())\nmodel.add(MaxPool2D(strides=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 32, kernel_size = (3, 3), activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(filters = 32, kernel_size = (3, 3), activation='relu'))\nmodel.add(BatchNormalization())\n#model.add(Conv2D(filters = 32, kernel_size = (3, 3), activation='relu'))\n#model.add(BatchNormalization())\nmodel.add(MaxPool2D(strides=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(1024, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"bb0658c9-59f1-5336-9e5d-abcd82010238","_uuid":"0b89d45a6990ead7a09bf7118fd9a5abc6d4d0c5"},"cell_type":"markdown","source":"Another important method to improve generalization is augmentation. This means generating more training data by randomly perturbing the images. If done in the right way, it can force the net to only learn translation-invariant features. If you train this model over hundreds of epochs, augmentation will definitely improve your performance. Here in the Kernel, we will only look at each image 4-5 times, so the difference is smaller. We use a Keras function for augmentation."},{"metadata":{"_cell_guid":"c34d0682-2777-382d-ab1c-fa0186fc0ae7","_uuid":"c29c4b36e76fc053ad1fef0bde62b39a63ecd430","trusted":true,"collapsed":true},"cell_type":"code","source":"datagen = ImageDataGenerator(zoom_range = 0.1,\n                            height_shift_range = 0.1,\n                            width_shift_range = 0.1,\n                            rotation_range = 10)","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"21ab56fa-c6fb-830a-47d3-3bc40c447ec9","_uuid":"1ba11b33b59f9bb6c87deb4850f9adbf08ec0f0f"},"cell_type":"markdown","source":"The model needs to be compiled before training can start. As our loss function, we use logloss which is called ''categorical_crossentropy\" in Keras. Metrics is only used for evaluation. As optimizer, we could have used ordinary stochastic gradient descent (SGD), but Adam is faster."},{"metadata":{"_cell_guid":"d0acbcf9-0be3-712f-222d-58e6670dac58","_uuid":"a0ad7d5a0a3d6dd41455f2f12e2bc172a251327f","trusted":true,"collapsed":true},"cell_type":"code","source":"model.compile(loss='categorical_crossentropy', optimizer = Adam(lr=1e-4), metrics=[\"accuracy\"])","execution_count":11,"outputs":[]},{"metadata":{"_cell_guid":"f8c4ffdf-abb0-7c28-b10b-16c5714b92d0","_uuid":"4ea17cc05258fca94d710eef85f7ee59aa928748"},"cell_type":"markdown","source":"We train once with a smaller learning rate to ensure convergence. We then speed things up, only to reduce the learning rate by 10% every epoch. Keras has a function for this: "},{"metadata":{"_cell_guid":"856a2d69-b0eb-6962-5323-fd2819d0be24","_uuid":"d0d572ed5c653d106eaae035fd21d3d3ccb9a92a","trusted":true,"collapsed":true},"cell_type":"code","source":"annealer = LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)","execution_count":12,"outputs":[]},{"metadata":{"_cell_guid":"e87ab385-9290-9b87-3b78-716188a140cc","_uuid":"04983b7d6ba3501d6a20295f6cac302c2994af99"},"cell_type":"markdown","source":"We will use a very small validation set during training to save time in the kernel."},{"metadata":{"_cell_guid":"b8d960a3-09ed-0802-2b23-4ed583f7492b","_uuid":"63fd8c108a239f3ab93a6c1b83a41249b0c5bc84","trusted":true},"cell_type":"code","source":"hist = model.fit_generator(datagen.flow(x_train, y_train, batch_size=16),\n                           steps_per_epoch=500,\n                           epochs=20, #Increase this when not on Kaggle kernel\n                           verbose=2,  #1 for ETA, 0 for silent\n                           validation_data=(x_val[:400,:], y_val[:400,:]), #For speed\n                           callbacks=[annealer])","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"9ceb04e7-b196-b2ca-ce2b-abcdc8870bcc","_uuid":"893dde0eff8ad594a20ad43c12d29b93afcef55e"},"cell_type":"markdown","source":"## Evaluate"},{"metadata":{"_cell_guid":"c64f7ec3-6af5-d3e6-e0e4-21f8e29d969c","_uuid":"f15de17bcf2bed4e0e57b86d27418c864fb97395"},"cell_type":"markdown","source":"We only used a subset of the validation set during training, to save time. Now let's check performance on the whole validation set."},{"metadata":{"_cell_guid":"d3a2a555-ef78-1cce-3971-0ea4f7bb0c1d","_uuid":"8f11329bf1f965e9d83190f30013a5e5353980a3","trusted":true},"cell_type":"code","source":"final_loss, final_acc = model.evaluate(x_val, y_val, verbose=0)\nprint(\"Final loss: {0:.4f}, final accuracy: {1:.4f}\".format(final_loss, final_acc))","execution_count":14,"outputs":[]},{"metadata":{"_cell_guid":"49fd9696-826d-f95c-3fe6-94fe4d1f0a6c","_uuid":"7472328229c4c118f769cd18ed23046175badb29","trusted":true},"cell_type":"code","source":"plt.plot(hist.history['loss'], color='b')\nplt.plot(hist.history['val_loss'], color='r')\nplt.show()\nplt.plot(hist.history['acc'], color='b')\nplt.plot(hist.history['val_acc'], color='r')\nplt.show()","execution_count":15,"outputs":[]},{"metadata":{"_cell_guid":"0783efce-d6b9-6d98-3824-0b45260a0f23","_uuid":"699cd31305f28b61694f89faf868799805b55cb7","trusted":true},"cell_type":"code","source":"y_hat = model.predict(x_val)\ny_pred = np.argmax(y_hat, axis=1)\ny_true = np.argmax(y_val, axis=1)\ncm = confusion_matrix(y_true, y_pred)\nprint(cm)","execution_count":16,"outputs":[]},{"metadata":{"_cell_guid":"5375f3d0-51fd-7199-4109-b32bcb4a4e63","_uuid":"494d59f1e02c030d783f23a2d3a22f6d67c9d7ce"},"cell_type":"markdown","source":"Not too bad, considering the minimal amount of training so far. In fact, we have only gone through the training data approximately five times. With proper training we should get really good results.\n\nAs you can see there are quite a few parameters that could be tweaked (number of layers, number of filters, Dropout parameters, learning rate, augmentation settings). This is often done with trial and error, and there is no easy shortcut. \n\nGetting convergence should not be a problem, unless you use an extremely large learning rate. It's easy, however, to create a net that overfits, with perfect results on the training set and very poor results on the validation data. If this happens, you could try increasing the Dropout parameters, increase augmentation, or perhaps stop training earlier. If you instead wants to increase accuracy, try adding on two more layers, or increase the number of filters."},{"metadata":{"_cell_guid":"0bd63110-f8f6-6f39-3225-c2f941184f3b","_uuid":"0d8a439bfa22b23832acaac981d8c41a1ffde0aa"},"cell_type":"markdown","source":"## Submit"},{"metadata":{"_cell_guid":"86d7a441-e1c9-9ad0-f305-25d0dcedaaee","_uuid":"1dc83423305d3a5a9b254a887d16d96bd9506789"},"cell_type":"markdown","source":"To easily get to the top half of the leaderboard, just follow these steps, go to the Kernel's output, and submit \"submission.csv\""},{"metadata":{"_cell_guid":"002ce4ad-9a8f-8a1e-c658-40f9c371f91d","_uuid":"f5cf6ab5f73d23621024439d8523bfc6831c6360","trusted":true,"collapsed":true},"cell_type":"code","source":"mnist_testset = np.loadtxt(test_file, skiprows=1, dtype='int', delimiter=',')\nx_test = mnist_testset.astype(\"float32\")\nx_test = x_test.reshape(-1, 28, 28, 1)/255.","execution_count":17,"outputs":[]},{"metadata":{"_cell_guid":"7413800d-21c3-654f-8b39-27e3df9c4391","_uuid":"4d60aeb07bf8ac2ca2a2939bfd39fb3fd6f17df4","trusted":true,"collapsed":true},"cell_type":"code","source":"y_hat = model.predict(x_test, batch_size=64)","execution_count":18,"outputs":[]},{"metadata":{"_cell_guid":"074aabf5-4a9a-bfdb-cbdd-0a8f06dc5fae","_uuid":"ad6f588e4967b2cc9bab90d97810270fcaf9a095"},"cell_type":"markdown","source":"y_hat consists of class probabilities (corresponding to the one-hot encoding of the training labels). I now select the class with highest probability"},{"metadata":{"_cell_guid":"575d9348-99ec-9b8c-f5a5-ceac44c5308d","_uuid":"3d3d11b9a3b905fd248ddab282b45e9232df3c61","trusted":false,"collapsed":true},"cell_type":"code","source":"y_pred = np.argmax(y_hat,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cbf31eb1-2ffd-efb5-e2b2-b4dda99fa1da","_uuid":"241933a29d8820944b64f80819c826c6c00e1670","trusted":true,"collapsed":true},"cell_type":"code","source":"with open(output_file, 'w') as f :\n    f.write('ImageId,Label\\n')\n    for i in range(len(y_pred)) :\n        f.write(\"\".join([str(i+1),',',str(y_pred[i]),'\\n']))","execution_count":19,"outputs":[]},{"metadata":{"_cell_guid":"580a57a8-c8b2-cb5b-9962-a743f50eb5a4","_uuid":"be07c1ddfca60f24f2f4326bbf2a4c207fc6a82b"},"cell_type":"markdown","source":"Submitting from this notebook usually gives you a result around 99%, with some randomness depending on weight initialization and test/train data split. I achieved 99.3% by averaging over 5 good runs, and you can get higher than that if you train overnight.\n\nIf you've successfully come this far, you can now create similar CNN for all kinds of image recognition problems. Good luck!"},{"metadata":{"_cell_guid":"5591673a-99e2-2269-8cc9-55573d49e029","_uuid":"fd6630d64e6edb52d52bbc675f8a6ae13954d409","trusted":false,"collapsed":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"_change_revision":0,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"_is_fork":false,"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}