{"cells": [{"cell_type": "markdown", "metadata": {"_cell_guid": "6b4b67cd-a639-4292-b0a1-fa05c1b87a32", "_uuid": "3b93993da452edca18514aa62f7b3a441451f562"}, "source": ["Now with test-time augmentation."]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "d9b88e17-3b65-4119-ad81-3dfa913a9aa2", "_uuid": "903ab7a05f78273100c3f4846a8a87ea99ac2aea"}, "source": ["This notebook is based on Peter Grenholm's MNIST kernel, but this version does not do validation.  (I presume you have already done validation to choose the number of epochs, so you can now run it with the whole training data set.  But you probably need more epochs than what seemed optimal when you were running with a partial training set in the validaiton run.  I'm too lazy to look up the formula right now, so I just did a seat-of-the-pants guess.  Really the optimal number of epochs is only a guess anyhow.)  To get good results you need to run it for a whole bunch of epochs (say 72) with a low learning rate (say 2e-4), which will be a very slow process unless you have a fast GPU, and which will not work at all on Kaggle.  Hence the <code>RUNNING_AS_KAGGLE_KERNEL</code> option for demonstration purposes.\n", "\n", "Note that the <code>RANDOM_SEED</code> does NOT make results reproducible, at least not on my setup at home.  However, in theory, you would get better results on average by running it with multiple alternative random seeds and taking a majority vote of the results.  (I haven't tried that.)  Even better perhaps, write out the cross-entropy results from each run and take an average, and then classify based on that average rather than on individual cross-entropy results.  (I haven't tried that either yet.)"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "11f95f95-cbae-418c-bb17-24bde6324360", "_execution_state": "idle", "collapsed": true, "_uuid": "a3fcf4d6d55e087e20b7e720f3bd1d4e283dbd85"}, "source": ["# Parameters\n", "\n", "RUNNING_AS_KERAS_KERNEL = True\n", "\n", "RANDOM_SEED = 0\n", "\n", "if RUNNING_AS_KERAS_KERNEL:\n", "   LEARNING_RATE = 1e-2\n", "   N_EPOCHS = 3\n", "   OUTPUT_FILE = \"mnistGrenholm-nocv-1ep01seed\" + str(RANDOM_SEED) + \".csv\"\n", "else:\n", "   LEARNING_RATE = 1.6e-4\n", "   N_EPOCHS = 200\n", "   OUTPUT_FILE = \"mnistGrenholm-nocv-72ep0002seed\" + str(RANDOM_SEED) + \".csv\"\n", "    \n"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "f3251b94-4894-41eb-b0a3-2646d8c2c9bf", "_uuid": "6f39fda4ff92be9129651091e104ffd17fc85e9f"}, "source": ["Peter Grenholm's Keras notebook for MNIST, with whatever changes I may have made"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "0146e5fd-7201-4f45-847d-9f87702d4ad2", "_execution_state": "idle", "collapsed": true, "_uuid": "3d2022e8232ec18d0c1984885f9728488e536b93"}, "source": ["import keras"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "635ff1dc-8767-4fcb-b3f5-d465abbd595b", "_execution_state": "idle", "collapsed": true, "_uuid": "ccf65e724a8a17ccb8a163fce15fae456db56e48"}, "source": ["import tensorflow as tf\n", "sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "99747ceb-dd4a-4e72-bfb6-1d9ead8713dc", "_execution_state": "idle", "collapsed": true, "_uuid": "b63b01b3dc337e17f859b0e810e75c23efd569a8"}, "source": ["import numpy as np # linear algebra\n", "\n", "keras.__version__\n", "from keras import backend as K\n", "K.set_image_dim_ordering('tf')\n"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "6bdcf0a8-c3ec-451f-9c2f-63dcd8754b2b", "_execution_state": "idle", "collapsed": true, "_uuid": "6b6e2b61467cad327c1de789ed5d6be6e092a155"}, "source": ["from keras.utils import np_utils\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Dropout, Flatten, Activation, Conv2D, MaxPooling2D\n", "from keras.optimizers import Adam\n", "from keras.callbacks import EarlyStopping, ModelCheckpoint\n", "from keras.preprocessing.image import ImageDataGenerator"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "547677f3-6b48-4423-ab27-5dfe1320952f", "collapsed": true, "_uuid": "669e6d4bf6190e8671c18524cda32acbb0cdb0e8"}, "source": ["from tqdm import tqdm"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "2658f455-c9e6-4334-9928-fe425794c455", "_execution_state": "idle", "collapsed": true, "_uuid": "9da290d247667bdaa14a625af375ef275f4db9a4"}, "source": ["train_file = \"../input/train.csv\"\n", "test_file = \"../input/test.csv\"\n", "output_file = OUTPUT_FILE"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "9d0db817-530b-4d8c-b652-31a961b05853", "_execution_state": "idle", "collapsed": true, "_uuid": "0b56db9c439729aa819fcc885f0fe914c4c4bc5c"}, "source": ["mnist_dataset = np.loadtxt(train_file, skiprows=1, dtype='int', delimiter=',')"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "88df585b-8949-409c-bb58-701c78381869", "_execution_state": "idle", "collapsed": true, "_uuid": "3db646bd41e04689b629fceaf82199e7d71ce5e5"}, "source": ["n_train = mnist_dataset.shape[0]\n", "\n", "np.random.seed(RANDOM_SEED)\n", "np.random.shuffle(mnist_dataset)\n", "x_train = mnist_dataset[:,1:]\n", "y_train = mnist_dataset[:,0]\n", "\n", "x_train = x_train.astype(\"float32\")/255.0\n", "y_train = np_utils.to_categorical(y_train)\n", "\n", "n_classes = y_train.shape[1]\n", "x_train = x_train.reshape(n_train, 28, 28, 1)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "7f568ad0-bbac-4ef1-baef-0da433715efb", "_uuid": "fdf15e264b9602682a4ae3a922220c7b45aa12cd"}, "source": ["Peter Grenholm copied the original net from several examples he found online.  I made some changes.  At least I increased the dropout rates from .25 to .3.  May have made other changes; I don't remember."]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "85296f91-a7f5-4d87-af0a-6448385e5b98", "_execution_state": "idle", "collapsed": true, "_uuid": "a5fb777a66ac3dd5ab244eff96a8163b15407584"}, "source": ["model = Sequential()\n", "\n", "model.add(Conv2D(filters = 32, kernel_size = (3, 3), input_shape = (28, 28, 1), padding='same'))\n", "model.add(Activation('relu'))\n", "model.add(Conv2D(filters = 32, kernel_size = (3, 3)))\n", "model.add(Activation('relu'))\n", "model.add(MaxPooling2D(pool_size=(2,2)))\n", "model.add(Dropout(0.3))\n", "\n", "model.add(Conv2D(filters = 64, kernel_size = (3, 3)))\n", "model.add(Activation('relu'))\n", "model.add(Conv2D(filters = 64, kernel_size = (3, 3)))\n", "model.add(Activation('relu'))\n", "model.add(MaxPooling2D(pool_size=(2,2)))\n", "model.add(Dropout(0.3))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(128))\n", "model.add(Activation('relu'))\n", "model.add(Dropout(0.5))\n", "model.add(Dense(10))\n", "model.add(Activation('softmax'))"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "194d94c6-a21f-4981-8cf9-0e358918da7b", "_uuid": "aa61072b31ffe935cb02b96c8994ff9133d2ef4c"}, "source": ["This Keras function simplifies augmentation, i.e. it randomly modifies the input for training to prevent overfitting. You can also normalize input with this function, just remember to adjust the validation and test sets accordingly in that case."]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "9365b06a-13d5-4994-ae85-e9821d26ed86", "_execution_state": "idle", "collapsed": true, "_uuid": "7beda9d81824953672d16069d1b884188cdba90b"}, "source": ["datagen = ImageDataGenerator(zoom_range = 0.1,\n", "                            height_shift_range = 0.1,\n", "                            width_shift_range = 0.1,\n", "                            rotation_range = 20)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "34763366-1bf4-4ddd-8b75-0fc68c0517ff", "_uuid": "79bfd7e9219f7733841056773e733b82adcf7146"}, "source": ["Certain parameters were chosen for the Kaggle kernel. For better performance, try reducing the learning rate and increase the number of epochs. Peter was able to reach 99.5% which put him at place 75 on the leaderboard. With a GPU, it still takes less than an hour to train."]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "30fad588-2d6a-4ec2-a849-2efea657258c", "_execution_state": "idle", "collapsed": true, "_uuid": "3a10651818290a9dfdd8eb50fb22468aca938f94"}, "source": ["model.compile(loss='categorical_crossentropy', optimizer = Adam(lr=LEARNING_RATE), metrics = [\"accuracy\"])"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "a63813e2-a108-4d69-90d7-8b80e4535b4a", "_uuid": "c24ec5ef49a1489f1ddee70da8719b4d05941811"}, "source": ["Model training. Note that the training loss is initially quite large, because of Dropout."]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "ed4641dc-9813-4675-bf75-abf886595828", "_execution_state": "idle", "collapsed": true, "_uuid": "baf0c3ef85ee03f40a7aae384b19f4a4f7ef138a"}, "source": ["hist = model.fit_generator(datagen.flow(x_train, y_train, batch_size = 64),\n", "                           steps_per_epoch = n_train/20,\n", "                           epochs = N_EPOCHS, \n", "                           verbose = 2  #verbose=1 outputs ETA, but doesn't work well in the cloud\n", "                          )"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "5c9c72e5-e116-4ca9-a7fe-617800aed86e", "_execution_state": "idle", "collapsed": true, "_uuid": "e88b27dfc41b89a3eb3c6c87c80903d528aa7662"}, "source": ["mnist_testset = np.loadtxt(test_file, skiprows=1, dtype='int', delimiter=',')\n", "x_test = mnist_testset.astype(\"float32\")/255.0\n", "n_test = x_test.shape[0]"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "92ae905c-7e38-45e6-a3f1-ef508d312e42", "_uuid": "58c6ce8fc7de14584813789123523d95d93728d3"}, "source": ["## Test Time Augmentation"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "eb0aa1f5-cd00-40bd-907e-cb2381bc7fdc", "collapsed": true, "_uuid": "8001ba6a3e8cb3fa0983b168dbc8c79ad3709e05"}, "source": ["x_test0 = np.ndarray(shape=(0, 28, 28, 1), dtype=\"float32\")\n", "x_test1 = np.ndarray(shape=(0, 28, 28, 1), dtype=\"float32\")\n", "x_test2 = np.ndarray(shape=(0, 28, 28, 1), dtype=\"float32\")\n", "x_test3 = np.ndarray(shape=(0, 28, 28, 1), dtype=\"float32\")"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "0f57ab37-4091-416c-b1b0-7f4c5945fe53", "collapsed": true, "_uuid": "ab97ab0f83a548a14421a39100db1a45e69c9900"}, "source": ["for i in tqdm(range(0,n_test)) :    \n", "    test_case = x_test[i,:].reshape(1,28,28,1)    \n", "    x_test0 = np.concatenate(   ( x_test0, test_case ),  axis=0  )\n", "    j = 0\n", "    for x_case, garbage in datagen.flow(test_case, [0], batch_size=1):\n", "        j += 1\n", "        if j == 1:\n", "            x_test1 = np.concatenate(   ( x_test1, x_case ),  axis=0  )\n", "        elif j == 2:\n", "            x_test2 = np.concatenate(   ( x_test2, x_case ),  axis=0  )\n", "        elif j == 3:\n", "            x_test3 = np.concatenate(   ( x_test3, x_case ),  axis=0  )\n", "        else:\n", "            break"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "2a0d43d0-5592-4f85-88a9-c6c75ba049bc", "collapsed": true, "_uuid": "3c25c270cbe068210b6a652570b60230a85ed1f7"}, "source": ["y_test0 = model.predict(x_test0, batch_size=64)\n", "y_test1 = model.predict(x_test1, batch_size=64)\n", "y_test2 = model.predict(x_test2, batch_size=64)\n", "y_test3 = model.predict(x_test3, batch_size=64)\n", "y_test = .4*y_test0 + .2*y_test1 + .2*y_test2 + .2*y_test3"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "c7b882c0-3123-4203-a5ad-5b20ef23dda1", "_uuid": "00e7ec542972b8d2bb61c85e5ab83c9dc732f13d"}, "source": ["y_test consists of class probabilities, I now select the class with highest probability"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "0c2de215-16c6-44e2-88f4-2a5e065bec55", "_execution_state": "idle", "collapsed": true, "_uuid": "6ad2362a10f3f930c47b3fb4115f9f85e76a8652"}, "source": ["y_index = np.argmax(y_test,axis=1)"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "7e8ecc1b-f9d2-40ed-a927-4a07c8144066", "_execution_state": "idle", "collapsed": true, "_uuid": "81a567b4649256483f98a04c211c3f969d72bc06"}, "source": ["with open(output_file, 'w') as f :\n", "    f.write('ImageId,Label\\n')\n", "    for i in range(0,n_test) :\n", "        f.write(\"\".join([str(i+1),',',str(y_index[i]),'\\n']))"]}, {"execution_count": null, "cell_type": "code", "outputs": [], "metadata": {"_cell_guid": "9e64d532-1370-4d03-852a-731860b623db", "_execution_state": "idle", "collapsed": true, "_uuid": "385b00dad26bd3fdc86c35983d9881d19f57c482"}, "source": []}], "nbformat_minor": 1, "nbformat": 4, "metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}, "language_info": {"mimetype": "text/x-python", "name": "python", "codemirror_mode": {"version": 3, "name": "ipython"}, "version": "3.6.1", "file_extension": ".py", "pygments_lexer": "ipython3", "nbconvert_exporter": "python"}}}