{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"c36bd9cd-77cc-a897-5ede-d37ab29d3634"},"source":"With a very simple Convolutional Nueral Network using Keras I was able to score *0.995* accuracy on my training set and *0.9945* accuracy on my validation set. My best results were found splitting the first 2000 images of the training data into my validation set. I used a very straightforward archiecture:\n\n- Conv -> Conv ->Pool -> Dropout -> Flatten -> Hidden -> Dropout -> Output. \n\nThe best results were rendered with the Adadelta optimizer https://arxiv.org/pdf/1212.5701v1.pdf\n\n**NOTE:** With this network each epoch takes roughly 180-200 seconds, and due to Kaggle holding a restriction of 1200 seconds, *20 minutes*, I only ran this kernel with 6 epochs to avoid crashing. My Kaggle submission scored **0.98657**  with 25 epochs which took about an hour to run (140s/epoch) on a standard Macbook Pro (intel Core i5)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b80cfeed-f811-a0ad-01a5-7918347b39d9"},"outputs":[],"source":"import pandas as pd\nfrom matplotlib import pyplot as plt\nfrom matplotlib import cm\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Convolution2D, MaxPooling2D\nfrom keras.optimizers import SGD\nfrom keras.constraints import maxnorm\nfrom keras.utils import np_utils\nfrom sklearn.preprocessing import StandardScaler"},{"cell_type":"markdown","metadata":{"_cell_guid":"53a85a56-7b66-431e-cd89-88ad1a71dff0"},"source":"**Data:** *42,000* input vectors each of size *784* represented with its own grayscale intensity\n\nSplit the first *2000* examples, *about 5% of our data*, to be our validation set and everything after will be used for training. \n\nCall Sklearns *StandardScaler()* for zero mean and unit variance scaling to transform the data."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d0bf40ec-f45d-3177-084a-eae26dfd487c"},"outputs":[],"source":"# load data\ndata = pd.read_csv('../input/train.csv') # 42000x784\ntarget = data.pop('label').values # (42000,)\ny_train = target[2000:] # (37800,)\ny_valid = target[:2000] # (8000,)\nX_train = data[2000:].values.astype('float32') # (34000,784)\nX_valid = data[:2000].values.astype('float32') # (8000, 784)\n# preprocessing\nX_train = StandardScaler().fit(X_train).transform(X_train)\nX_valid = StandardScaler().fit(X_valid).transform(X_valid)"},{"cell_type":"markdown","metadata":{"_cell_guid":"15ed1d34-b086-3c20-dc6a-21b3f59ee5d4"},"source":"Reshape the training and validation inputs from *756* input vectors to *28x28* matrices for compatibility with Keras. \n\nCall *to_categorical()* to transform each target into one hot vector notation for classification"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d9af0396-8806-534f-3ba1-46d1d2fd85ab"},"outputs":[],"source":"X_train = X_train.reshape(-1, 28, 28, 1) # (40000, 28, 28, 1)\nX_valid = X_valid.reshape(-1, 28, 28, 1) # (2000, 28, 28, 1)\n\n# one hot vector utility\ny_train = np_utils.to_categorical(y_train, 10)\ny_valid = np_utils.to_categorical(y_valid, 10)"},{"cell_type":"markdown","metadata":{"_cell_guid":"49388ea4-f0ef-43d2-3bd8-a36c0806782b"},"source":"Visualize the first 9 images from our training set and their corresponding labels with their grayscale intensities\n\n*Note:* When plotting we have to reshape the dimensions of our training images to 1x28x28 matrices for them to be represented in their proper greyscale format. We have to initialize the images to 28x28x1 to match the channel dimensions for Keras' theano backend."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8564db03-09e2-0b41-ad96-3c31907388bb"},"outputs":[],"source":"for i in range(9):\n    plt.subplot(331+i)\n    plt.imshow(X_train.reshape(-1,1,28,28)[i][0], cmap=cm.binary)\nplt.show()\nprint(target[2000:2009])"},{"cell_type":"markdown","metadata":{"_cell_guid":"641e96d2-2313-578a-980b-6a789b570b28"},"source":"**Convolutional architecture:** Conv -> Conv ->Pool -> Dropout -> Flatten -> Hidden -> Dropout -> Output. "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1f1a55ae-316b-8151-e25a-c923379a8407"},"outputs":[],"source":"# Convolutional architecture ~conv, conv, pool, drop, flatten, dense, drop, dense~\nmodel = Sequential()\nmodel.add(Convolution2D(32, 3, 3, input_shape=(28,28,1), activation='relu',\n                       border_mode = 'valid'))\nmodel.add(Convolution2D(32, 3, 3, border_mode='same'))\nmodel.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))\nmodel.add(Dropout(0.3))\nmodel.add(Flatten())\nmodel.add(Dense(128, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))"},{"cell_type":"markdown","metadata":{"_cell_guid":"4f52787a-a496-d752-c9f5-3b1caa750f1f"},"source":"among *RMSprop*, *Adam*, *Adagrad*, *SGD*, *adadelta*, the best optimizer for this particular model was ***Adadelta***"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"79f7414f-8b58-668e-4443-8d5e9ae61dfe"},"outputs":[],"source":"num_epochs = 6\nmodel.compile(optimizer='adadelta', loss='categorical_crossentropy', metrics=['accuracy'])"},{"cell_type":"markdown","metadata":{"_cell_guid":"1df5e73a-2961-90f3-494f-0dea3d574ffa"},"source":"**mini batches:** *128*\n\n**epochs:** *6*"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"45825e95-d4a2-364c-5457-7e3ecf459a03"},"outputs":[],"source":"hist = model.fit(X_train,y_train, batch_size=128, nb_epoch=6, \n                validation_data=(X_valid,y_valid))\nscores = model.evaluate(X_valid, y_valid, verbose=0)\n\n\nplt.plot(hist.history['loss'])\nplt.plot(hist.history['val_loss'])\nplt.title('Loss Rate')\nplt.ylabel('Loss')\nplt.xlabel('Training interations')\nplt.legend(['Training', 'Testing'], loc='upper left')\nplt.show()\n#plt.savefig('MNIST_loss_plot1.png')\n\nplt.plot(hist.history['val_acc'])\nplt.title('Accuracy Rate')\nplt.ylabel('Accuracy %')\nplt.xlabel('Training iterations')\nplt.legend(['Testing'], loc='upper left')\nplt.show()\n#plt.savefig('MNIST_acc_plot1.png')"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"fbfd789e-5f16-af3c-3685-445a5fe4f860"},"outputs":[],"source":"# load test data\ntest_data = pd.read_csv('../input/test.csv').values.astype('float32')\ntest_data = StandardScaler().fit(test_data).transform(test_data)\ntest_data = test_data.reshape(-1,28,28,1)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e2d73faa-6dbb-f02c-f350-277846a3e7fb"},"outputs":[],"source":"# predictive model\ntest_submission = model.predict_classes(test_data, verbose=2)"},{"cell_type":"markdown","metadata":{"_cell_guid":"0dd5cae5-b72d-8151-a7e9-cc07891786f1"},"source":"We can print some of images from the test set and print the corresponding predicttions to get a sense of how good our model really wasWe can print some of images from the test set and print the corresponding predicttions to get a sense of how good our model really was"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b4130607-7021-afa9-3864-1b791c9d7e98"},"outputs":[],"source":"for i in range(9):\n    plt.subplot(330+i+1)\n    plt.imshow(test_data.reshape(-1,1,28,28)[i][0],cmap=cm.binary)\nplt.show()\nprint(test_submission[:9])"},{"cell_type":"markdown","metadata":{"_cell_guid":"231d9005-26fb-bf7f-3a24-5578eb94c108"},"source":"Results from my submission on 25 epochs: **0.98657** "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5d3d752c-c3df-7e71-9da4-73b89fc080ae"},"outputs":[],"source":"# save submission to csv\npd.DataFrame({\"ImageId\": list(range(1,len(test_data)+1)), \n              \"Label\": test_submission}).to_csv('MNIST-submission_1-3-2017.csv', index=False,header=True)"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}