{"metadata": {"language_info": {"codemirror_mode": {"version": 3, "name": "ipython"}, "nbconvert_exporter": "python", "file_extension": ".py", "version": "3.6.1", "name": "python", "pygments_lexer": "ipython3", "mimetype": "text/x-python"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat_minor": 1, "cells": [{"metadata": {"_uuid": "903ab7a05f78273100c3f4846a8a87ea99ac2aea", "_cell_guid": "d9b88e17-3b65-4119-ad81-3dfa913a9aa2"}, "cell_type": "markdown", "source": ["This notebook is based on Peter Grenholm's MNIST kernel.  I've made a number of changes, including using longer epochs, which (hopefully) makes it easier to see if there is an improivng trend from one epoch to the next.  (Note that epoch length is, in principle, arbitrary, since we are generating random training data based on the input data, so we can never run out of data.)\n", "\n", "To run this on Kaggle or on any CPU or on a slow GPU or even on a fast GPU if you are going to sit there and watch the results, you will probably want to use <code>DEMO_MODE</code> so as to limit the fitting to a few epochs.  The full 300 epochs take several hours to run even on my GTX1080.\n", "\n", "Note that the <code>RANDOM_SEED</code> does NOT make results reproducible, at least not on my setup at home.  However, you should get better results on average by running it with multiple alternative random seeds and taking a majority vote of the results.  Or you could keep track of the cross-entropy results from each run and take an average, and then classify based on that average.  Maybe in the next version I will do something like that inside the code.  Then you can start it one day and come back the next day for a result."]}, {"metadata": {"_uuid": "a3fcf4d6d55e087e20b7e720f3bd1d4e283dbd85", "_execution_state": "idle", "collapsed": true, "_cell_guid": "11f95f95-cbae-418c-bb17-24bde6324360"}, "outputs": [], "cell_type": "code", "source": ["# Parameters\n", "\n", "DEMO_MODE = True\n", "\n", "RANDOM_SEED = 1\n", "\n", "if DEMO_MODE:\n", "   LEARNING_RATE = 3e-2\n", "   N_EPOCHS = 6\n", "   EPOCH_LENGTH = 2048\n", "   OUTPUT_FILE = \"mnist6longEpochs123155seed\" + str(RANDOM_SEED) + \".csv\"\n", "else:\n", "   LEARNING_RATE = 1.5e-4\n", "   N_EPOCHS = 300\n", "   EPOCH_LENGTH = 8192\n", "   OUTPUT_FILE = \"mnist300longEpochs123155seed\" + str(RANDOM_SEED) + \".csv\"\n", "    \n"], "execution_count": null}, {"metadata": {"_uuid": "5d2913f86569a6072466332ef1d6dc646b56b42f", "collapsed": true, "_cell_guid": "10236af2-5704-40be-a46f-c133940db34a"}, "outputs": [], "cell_type": "code", "source": ["import numpy as np\n", "import keras\n", "import tensorflow as tf\n", "from keras import backend as K\n", "from keras.utils import np_utils\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Dropout, Flatten, Activation, Conv2D, MaxPooling2D\n", "from keras.optimizers import Adam\n", "from keras.callbacks import EarlyStopping, ModelCheckpoint\n", "from keras.preprocessing.image import ImageDataGenerator\n", "from keras import regularizers\n", "from tqdm import tqdm"], "execution_count": null}, {"metadata": {"_uuid": "9bc64d4672fbdf3e820c1c676cbae13324322c7b", "collapsed": true, "_cell_guid": "92e55d07-9b24-47d3-a9c8-e729df1e0acd"}, "outputs": [], "cell_type": "code", "source": ["sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\n", "K.set_image_dim_ordering('tf')\n", "keras.__version__"], "execution_count": null}, {"metadata": {"_uuid": "9da290d247667bdaa14a625af375ef275f4db9a4", "_execution_state": "idle", "collapsed": true, "_cell_guid": "2658f455-c9e6-4334-9928-fe425794c455"}, "outputs": [], "cell_type": "code", "source": ["train_file = \"../input/train.csv\"\n", "test_file = \"../input/test.csv\"\n", "output_file = OUTPUT_FILE"], "execution_count": null}, {"metadata": {"_uuid": "78b3a6227282c3758f13633cc19a09e0bfd5d550", "_cell_guid": "9134068f-c4db-4ad9-90e1-0b258ca602ab"}, "cell_type": "markdown", "source": ["### Set up the model\n", "\n", "Model is based on Peter Grehholm's model, but I put in a bunch more dropout"]}, {"metadata": {"_uuid": "a5fb777a66ac3dd5ab244eff96a8163b15407584", "_execution_state": "idle", "collapsed": true, "_cell_guid": "85296f91-a7f5-4d87-af0a-6448385e5b98"}, "outputs": [], "cell_type": "code", "source": ["model = Sequential()\n", "\n", "model.add(Conv2D(filters = 32, kernel_size = (3, 3), input_shape = (28, 28, 1), padding='same' ))\n", "model.add(Activation('relu'))\n", "model.add(Dropout(0.12))\n", "model.add(Conv2D(filters = 32, kernel_size = (3, 3) ))\n", "model.add(Activation('relu'))\n", "model.add(MaxPooling2D(pool_size=(2,2)))\n", "model.add(Dropout(0.31))\n", "\n", "model.add(Conv2D(filters = 64, kernel_size = (3, 3) ))\n", "model.add(Activation('relu'))\n", "model.add(Dropout(0.12))\n", "model.add(Conv2D(filters = 64, kernel_size = (3, 3) ))\n", "model.add(Activation('relu'))\n", "model.add(MaxPooling2D(pool_size=(2,2)))\n", "model.add(Dropout(0.31))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(128))\n", "model.add(Activation('relu'))\n", "model.add(Dropout(0.55))\n", "model.add(Dense(10))\n", "model.add(Activation('softmax'))"], "execution_count": null}, {"metadata": {"_uuid": "3a10651818290a9dfdd8eb50fb22468aca938f94", "_execution_state": "idle", "collapsed": true, "_cell_guid": "30fad588-2d6a-4ec2-a849-2efea657258c"}, "outputs": [], "cell_type": "code", "source": ["model.compile(loss='categorical_crossentropy', optimizer = Adam(lr=1.6e-4, decay=8e-6), metrics = [\"accuracy\"])\n"], "execution_count": null}, {"metadata": {"_uuid": "aa61072b31ffe935cb02b96c8994ff9133d2ef4c", "_cell_guid": "194d94c6-a21f-4981-8cf9-0e358918da7b"}, "cell_type": "markdown", "source": ["This Keras function simplifies augmentation, i.e. it randomly modifies the input for training to prevent overfitting. You can also normalize input with this function, just remember to adjust the validation and test sets accordingly in that case."]}, {"metadata": {"_uuid": "7beda9d81824953672d16069d1b884188cdba90b", "_execution_state": "idle", "collapsed": true, "_cell_guid": "9365b06a-13d5-4994-ae85-e9821d26ed86"}, "outputs": [], "cell_type": "code", "source": ["datagen = ImageDataGenerator(zoom_range = 0.1,\n", "                            height_shift_range = 0.1,\n", "                            width_shift_range = 0.1,\n", "                            rotation_range = 20)"], "execution_count": null}, {"metadata": {"_uuid": "e6e3cc668e73c12fe5f273a6795d178f43a2aa5e", "_cell_guid": "2d2a5ab3-ef9d-4caf-8014-aa64dd1fa04c"}, "cell_type": "markdown", "source": ["### Read the data"]}, {"metadata": {"_uuid": "0b56db9c439729aa819fcc885f0fe914c4c4bc5c", "_execution_state": "idle", "collapsed": true, "_cell_guid": "9d0db817-530b-4d8c-b652-31a961b05853"}, "outputs": [], "cell_type": "code", "source": ["mnist_dataset = np.loadtxt(train_file, skiprows=1, dtype='int', delimiter=',')"], "execution_count": null}, {"metadata": {"_uuid": "ab521e51001ba5492d811c11b7157aea7441514e", "collapsed": true, "_cell_guid": "32eab952-afbd-46ea-a72e-90b6730de585"}, "outputs": [], "cell_type": "code", "source": ["val_split = 0.125\n", "n_raw = mnist_dataset.shape[0]\n", "n_val = int(n_raw * val_split + 0.5)\n", "n_train = n_raw - n_val\n", "\n", "np.random.seed(RANDOM_SEED)"], "execution_count": null}, {"metadata": {"_uuid": "a8417ac6323cb13b5059ade818696ac2aa9972ec", "_cell_guid": "64958d03-a4ff-40a2-b191-cf1a7b4e5571"}, "cell_type": "markdown", "source": ["### Fit the model\n", "\n", "Next 3 code cells could go inside a loop and be ensembled"]}, {"metadata": {"_uuid": "3db646bd41e04689b629fceaf82199e7d71ce5e5", "_execution_state": "idle", "collapsed": true, "_cell_guid": "88df585b-8949-409c-bb58-701c78381869"}, "outputs": [], "cell_type": "code", "source": ["np.random.shuffle(mnist_dataset)\n", "x_val, x_train = mnist_dataset[:n_val,1:], mnist_dataset[n_val:,1:]\n", "y_val, y_train = mnist_dataset[:n_val,0], mnist_dataset[n_val:,0]\n", "\n", "x_train = x_train.astype(\"float32\")/255.0\n", "x_val = x_val.astype(\"float32\")/255.0\n", "y_train = np_utils.to_categorical(y_train)\n", "y_val = np_utils.to_categorical(y_val)\n", "\n", "n_classes = y_train.shape[1]\n", "x_train = x_train.reshape(n_train, 28, 28, 1)\n", "x_val = x_val.reshape(n_val, 28, 28, 1)"], "execution_count": null}, {"metadata": {"_uuid": "c24ec5ef49a1489f1ddee70da8719b4d05941811", "_cell_guid": "a63813e2-a108-4d69-90d7-8b80e4535b4a"}, "cell_type": "markdown", "source": ["Model training. Note that the training loss is initially quite large, because of Dropout."]}, {"metadata": {"_uuid": "baf0c3ef85ee03f40a7aae384b19f4a4f7ef138a", "_execution_state": "idle", "scrolled": true, "collapsed": true, "_cell_guid": "ed4641dc-9813-4675-bf75-abf886595828"}, "outputs": [], "cell_type": "code", "source": ["hist = model.fit_generator(datagen.flow(x_train, y_train, batch_size = 64),\n", "                           steps_per_epoch = EPOCH_LENGTH,\n", "                           epochs = N_EPOCHS, \n", "                           verbose = 2,  #verbose=1 outputs ETA, but doesn't work well in the cloud\n", "                           validation_data = datagen.flow(x_val, y_val, batch_size=64),\n", "                           validation_steps = 256\n", "                          )"], "execution_count": null}, {"metadata": {"_uuid": "02b92fec2019d2cc621672d6e6e64632cfb212c5", "collapsed": true, "_cell_guid": "e7c98a23-c898-4384-86ae-0e831cb93ec3"}, "outputs": [], "cell_type": "code", "source": ["mnist_testset = np.loadtxt(test_file, skiprows=1, dtype='int', delimiter=',')\n", "x_test = mnist_testset.astype(\"float32\")/255.0\n", "n_test = x_test.shape[0]\n", "\n", "x_test = x_test.reshape(n_test, 28, 28, 1)\n", "y_test = model.predict(x_test, batch_size=64)\n", "\n", "y_index = np.argmax(y_test,axis=1)"], "execution_count": null}, {"metadata": {"_uuid": "5227456b1f9e5379f8226d631c64f6d975d86af1", "_cell_guid": "d80c60e6-f6a8-4d30-a416-f2c70a429ffb"}, "cell_type": "markdown", "source": ["### Write results to output file"]}, {"metadata": {"_uuid": "fd7ae104d01bffb86cd8bc7f839f2e5a1ead0b8e", "collapsed": true, "_cell_guid": "e6907682-2cfc-4e84-ad2d-0335c9db2a5e"}, "outputs": [], "cell_type": "code", "source": ["with open(OUTPUT_FILE, 'w') as f :\n", "    f.write('ImageId,Label\\n')\n", "    for i in range(0,n_test) :\n", "        f.write(\"\".join([str(i+1),',',str(y_index[i]),'\\n']))"], "execution_count": null}, {"metadata": {"_uuid": "0bef170e1433464d1460ec5e3804e94565a23f3c", "collapsed": true, "_cell_guid": "f8fff919-1aa9-4196-adac-084168c3be8d"}, "outputs": [], "cell_type": "code", "source": [], "execution_count": null}], "nbformat": 4}