{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"d2b53a9a-2f19-4819-0705-aa59fa93fc7e"},"source":"Note: it is recommended you download this notebook to your own PC if you want to train the model, as Kaggle's servers will be slow and have a timeout. \n\n### Load data and do preprocessing \nFirst we load our data into Pandas dataframes and convert them to NumPy arrays using *.values*. We further convert the datatype from *float64* to *float32* for speed. \n\nSince the training examples are 1D vectors, and we wish to do convolutions on the 2D images, we reshape the input data from (n_train x 784) to (n_train x 28 x 28). We also normalize the data to the interval [0,1], while this is not really necessary here as all the pixel values already lie in the same range [0, 255], it is a good procedure to follow in general. \n\nWe used *to_categorical* to transform the target data (which lies in the set [0,1,2,3,4,5,6,7,8,9]) to one hot vectors. "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1a6ad007-48e1-9943-a831-7f4bf749729f"},"outputs":[],"source":"import pandas as pd\nimport numpy as np\nfrom keras.utils.np_utils import to_categorical\nfrom keras import backend as K\n\nK.set_image_dim_ordering('th') #input shape: (channels, height, width)\n\ntrain_df = pd.read_csv(\"../input/train.csv\")\nvalid_df = pd.read_csv(\"../input/test.csv\")\n\nx_train = train_df.drop(['label'], axis=1).values.astype('float32')\nY_train = train_df['label'].values.astype('float32')\nx_valid = valid_df.values.astype('float32')\n\nimg_width, img_height = 28, 28\n\nn_train = x_train.shape[0]\nn_valid = x_valid.shape[0]\n\nn_classes = 10 \n\nx_train = x_train.reshape(n_train,1,img_width,img_height)\nx_valid = x_valid.reshape(n_valid,1,img_width,img_height)\n\nx_train = x_train/255 #normalize from [0,255] to [0,1]\nx_valid = x_valid/255 \n\ny_train = to_categorical(Y_train)"},{"cell_type":"markdown","metadata":{"_cell_guid":"fb34d450-7de9-f245-1201-e25a9363bbda"},"source":"### View an image to make sure everything is OK\n(The images are not color, but *imshow()* applies a colormap by default, and I'm not sure how to disable it)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0d5415bc-bced-8800-de41-5ad93637bb32"},"outputs":[],"source":"%matplotlib inline\nimport matplotlib.pyplot as plt\n\nimgplot = plt.imshow(x_train[4,0,:,:,],cmap='gray')"},{"cell_type":"markdown","metadata":{"_cell_guid":"6a8f5942-fafa-85e0-fbb7-2be7842bfdc3"},"source":"### Build Model \n\n** Handling edges/borders **  \nOne thing we have to decide is how to deal with the edges. To allow convolution of the data at the edges, one can first 'zero pad' the input array, by adding zeros to the left, right, top, and bottom. ie:\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00000<br>\n123&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;01230<br>\n456&nbsp;-->&nbsp;04560<br>\n789&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;07890<br>\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00000<br> \n\n\nThis can be done with the [ZeroPadding2D()](https://keras.io/layers/convolutional/#ZeroPadding2D) function in Keras. One must make sure to zero pad with enough zeros -- one needs *filter_size/2* zeros. \n\nAlternatively, a simpler solution is to set *border_mode='same'*, which returns a filter map of the same size and automatically appends zeros. \n\nThe other option available in Keras is *border_mode='valid'* which only does convolutions where the filter fits inside the image (also called narrow convolution). With this option set, the filter map has smaller dimensions than the input image. \n\n**[2D Convolution](https://keras.io/layers/convolutional/#convolution2d)**  \nThe main operation in the CNN. \n\n\n**[Max Pooling](https://keras.io/layers/pooling/#maxpooling2d)**  \nMax pooling reduces the size of the filter maps, by applying a *max filter* to non-overlapping subregions. A max pooling layer with pooling_size=2 (ie using 2x2 max filters) will reduce the number total number of parameters in the filter map by a factor of 4.\n\n**[Dropout](https://keras.io/layers/core/#dropout)**  \nThis is a technique for preventing overfitting. The dropout layer in Keras randomly drops a certain fraction of the neurons (units) with a probability p in each training round. This forces the network to learn redundant representations, and effectively lowers the number of paramters while maintaining a wide degree of flexibility.\n\n**[Flattening](https://keras.io/layers/core/#flaten)**  \nFlattening converts the input activations, which are in an array of shape (n_filters, filter_size_x, filter_size_y) into a 1D array. \n\n**[Dense layer](https://keras.io/layers/core/#dense)**  \nThis is a fully connected layer of neurons which works on the 1D input and gives a 1D output. \n\n**[Softmax activation](https://en.wikipedia.org/wiki/Softmax_function#Artificial_neural_networks)**  \nSoftmax converts the input  \n\n### Hyperparameters for this model\n\n* Number of filters (n_filters) \n\n* Size of convolution filters (filter_size1, filter_size2)  \n\n* Size of pooling windows (pool_size1, pool_size2)\n\n* Size of the dense layer (n_dense)\n\n### Compilation  \nThe 'compilation' step is where you specify your loss function in Keras. In this case, we use categorical crossentropy. "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"95dc8409-4741-1b35-677c-77589b616b44"},"outputs":[],"source":"from keras.models import Sequential\nfrom keras.layers.convolutional import *\nfrom keras.layers.core import Dropout, Dense, Flatten, Activation\n\nn_filters = 64\nfilter_size1 = 3\nfilter_size2 = 2\npool_size1 = 3\npool_size2 = 1\nn_dense = 128\n\nmodel = Sequential()\n\nmodel.add(Convolution2D(n_filters, filter_size1, filter_size1, batch_input_shape=(None, 1, img_width, img_height), activation='relu', border_mode='valid'))\n\nmodel.add(MaxPooling2D(pool_size=(pool_size1, pool_size1)))\n\nmodel.add(Convolution2D(n_filters, filter_size2, filter_size2, activation='relu', border_mode='valid'))\n\nmodel.add(MaxPooling2D(pool_size=(pool_size2, pool_size2)))\n\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\n\nmodel.add(Dense(n_dense))\n\nmodel.add(Activation('relu'))\n\nmodel.add(Dropout(0.5))\n\nmodel.add(Dense(n_classes))\n\nmodel.add(Activation('softmax'))\n\nmodel.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])"},{"cell_type":"markdown","metadata":{"_cell_guid":"5bb63964-a49f-7d80-4978-d3cf334d136d"},"source":"### Fit the model\nsee *[fit()](https://keras.io/models/sequential/#fit)* documentation  "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"aa355513-4b10-6650-6f89-cc94c24fde76"},"outputs":[],"source":"batch_size = 128\nn_epochs = 1\n\nmodel.fit(x_train,\n          y_train,\n          batch_size=batch_size,\n          nb_epoch=n_epochs,verbose=2,\n          validation_split=.2)"},{"cell_type":"markdown","metadata":{"_cell_guid":"1eb906ee-61c2-40aa-b54c-ba5209a1a6dd"},"source":"### Run model on validation data and save output\nsee *[predict_classes()](https://keras.io/models/sequential/#predict_classes)* documentation. \n\n(By contrast, *[predict()](https://keras.io/models/sequential/#predict)*  would return an array with shape (n_examples, n_classes), where each number represents a probability for the class in question.)\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0297c706-d7b7-d9b4-e213-1ef4dca5296c"},"outputs":[],"source":"yPred = model.predict_classes(x_valid,batch_size=32,verbose=1)\n\nnp.savetxt('mnist_output.csv', np.c_[range(1,len(yPred)+1),yPred], delimiter=',', header = 'ImageId,Label', comments = '', fmt='%d')"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}