{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"ebbe6bba-c88c-0d49-502a-573d19476310"},"source":"A Convolutional Neural Network for MNIST. This solution got me a score of 0.98929 on the leaderboard.\n\nNote: this solution is heavily based on the tensorflow tutorial found at [https://www.tensorflow.org/tutorials/mnist/pros/](https://www.tensorflow.org/tutorials/mnist/pros/)."},{"cell_type":"markdown","metadata":{"_cell_guid":"6f552ccc-f472-107c-e20a-1c03a9362532"},"source":"## Imports and Settings ##"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"15754c6a-4ee3-0ac0-3d01-9ec4831370e3"},"outputs":[],"source":"# These are all the modules we'll be using later. Make sure you can import them\n# before proceeding further.\nimport numpy as np\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom pandas import DataFrame, read_csv\nplt.rcParams['image.cmap'] = 'Greys'\nplt.rcParams['image.interpolation'] = 'nearest'\nplt.rcParams['figure.figsize'] = (2,2)"},{"cell_type":"markdown","metadata":{"_cell_guid":"d3088dc8-906f-32e9-db54-59a4c075bc31"},"source":"## Simulation Constants ##\n\nDownload notebook and use commented out values for better performance."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"50083ab2-7752-56fb-54e8-249a4017bd29"},"outputs":[],"source":"LABELS = 10 # Number of different types of labels (1-10)\nPIXELS = 28 # width / height of the image\nCHANNELS = 1 # Number of colors in the image (greyscale)\n\nTRAIN = 21000  #40000 # Train data size\nVALID = 42000 - TRAIN # Validation data size\n\nSTEPS = 5000 #20001   # Number of steps to run\nBATCH = 100 # Stochastic Gradient Descent batch size\nPATCH = 5 # Convolutional Kernel size\nDEPTH = 12 #32 # Convolutional Kernel depth size == Number of Convolutional Kernels\nHIDDEN = 100 #1024 # Number of hidden neurons in the fully connected layer\n\nLEARNING_RATE = 0.003 # Initial Learning rate\nDECAY_FACTOR = 0.95 # Continuous Learning Rate Decay Factor (per 1000 steps)"},{"cell_type":"markdown","metadata":{"_cell_guid":"4572603b-cc72-031c-f710-6196172efea4"},"source":"Define an accuracy metric"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1c493cc2-c127-b9e3-96fa-b3d8a523e202"},"outputs":[],"source":"def acc(pred, labels):\n    return 100.0 * np.mean(np.float32(np.argmax(pred, axis=1) == np.argmax(labels, axis=1)), axis=0)"},{"cell_type":"markdown","metadata":{"_cell_guid":"b734dc0f-50d6-37c7-7906-409d5b49f3cb"},"source":"To shuffle data and labels:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4cd92946-fe19-5590-7946-1f43e2a68e48"},"outputs":[],"source":"def shuffle(data, labels):\n    rnd = np.random.get_state()\n    np.random.shuffle(data)\n    np.random.set_state(rnd)\n    np.random.shuffle(labels)"},{"cell_type":"markdown","metadata":{"_cell_guid":"616030e9-9c6e-c369-c6f2-c19cd76a542b"},"source":"## Import Data ##\nImport and reformat into a TensorFlow-accepted shape:\n\n - image data shape: `(# images, # vertical pixels, # horizontal pixels, # colors)`\n - labels need 1-hot encoding: `1 = [1,0,0...0], 2 = [0,1,0...0] ...`"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f7b2abd6-d89d-1149-751f-3697f9afb52c"},"outputs":[],"source":"data = read_csv('../input/train.csv') # Read csv file in pandas dataframe\nlabels = np.array(data.pop('label')) # Remove the labels as a numpy array from the dataframe\nlabels = np.array([np.arange(LABELS) == label for label in labels])\ndata = np.array(data, dtype=np.float32)/255.0-1.0# Convert the dataframe to a numpy array\ndata = data.reshape(len(labels), PIXELS, PIXELS, CHANNELS) # Reshape the data into 42000 2d images\ntrain_data = data[:TRAIN]\ntrain_labels = labels[:TRAIN]\nvalid_data = data[TRAIN:]\nvalid_labels = labels[TRAIN:]\ntest_data = np.array(read_csv('../input/test.csv'), dtype=np.float32)/255.0-1.0\ntest_data = test_data.reshape(test_data.shape[0], PIXELS, PIXELS, CHANNELS)\n\nshuffle(train_data, train_labels) # Randomly shuffle the training data\n\nprint('train data shape = ' + str(train_data.shape) + ' = (TRAIN, PIXELS, PIXELS, CHANNELS)')\nprint('labels shape = ' + str(labels.shape) + ' = (TRAIN, LABELS)')"},{"cell_type":"markdown","metadata":{"_cell_guid":"fa565617-eaf9-af98-d4f6-b7a513f019f8"},"source":"Let's build a small network with two convolutional layers, followed by one fully connected layer. Convolutional networks are more expensive computationally, so we'll limit its depth and number of fully connected nodes."},{"cell_type":"markdown","metadata":{"_cell_guid":"254eff4d-6748-a01c-bbe6-dd5ce7510ad7"},"source":"## Tensorflow Graph ##"},{"cell_type":"markdown","metadata":{"_cell_guid":"051aeae2-3c32-6eb6-f25b-89579e30f762"},"source":"### Input Data ###\nWe use placeholders for the training data, since the training data is different for every batch."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8c23848b-e5df-548c-698a-d30313129b23"},"outputs":[],"source":"tf_train_data = tf.placeholder(tf.float32, shape=(BATCH, PIXELS, PIXELS, CHANNELS))\ntf_train_labels = tf.placeholder(tf.float32, shape=(BATCH, LABELS))\ntf_valid_data = tf.constant(valid_data)\ntf_test_data = tf.constant(test_data)"},{"cell_type":"markdown","metadata":{"_cell_guid":"bc95200a-61a2-5f5a-0dc0-71009c5601dc"},"source":"### Variables\nThe `global_step` variable is used for learning rate decay.\nFor the rest, we use a 4 layered network consisting of 2 convolutional layers (`w1`, `b1`) and (`w2`,`b2`) for which the depth of the second layer is twice the depth of the first layer (`DEPTH`), a fully connected hidden layer (`w3`,`b3`) with # `HIDDEN` hidden neurons and an output layer (`w4`, `b4`) with `10` output nodes (one-hot encoding)."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"429d6634-ba23-9644-84f0-9173627d1f96"},"outputs":[],"source":"global_step = tf.Variable(0)\nw1 = tf.Variable(tf.truncated_normal([PATCH, PATCH, CHANNELS, DEPTH], stddev=0.1))\nb1 = tf.Variable(tf.zeros([DEPTH]))\nw2 = tf.Variable(tf.truncated_normal([PATCH, PATCH, DEPTH, 2*DEPTH], stddev=0.1))\nb2 = tf.Variable(tf.constant(1.0, shape=[2*DEPTH]))\nw3 = tf.Variable(tf.truncated_normal([PIXELS // 4 * PIXELS // 4 * 2*DEPTH, HIDDEN], stddev=0.1))\nb3 = tf.Variable(tf.constant(1.0, shape=[HIDDEN]))\nw4 = tf.Variable(tf.truncated_normal([HIDDEN, LABELS], stddev=0.1))\nb4 = tf.Variable(tf.constant(1.0, shape=[LABELS]))"},{"cell_type":"markdown","metadata":{"_cell_guid":"6dde9903-2b67-6d57-0200-3a2041a786b6"},"source":"### Model\nAs discussed during the initialization of the variables, the output `logits` are obtained after two convolutional layers and a hidden fully connected layer."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3a183269-2293-7a6b-0cc9-ba51d3fc9428"},"outputs":[],"source":"def logits(data):\n    conv = tf.nn.conv2d(data, w1, [1, 1, 1, 1], padding='SAME')\n    pool = tf.nn.max_pool(conv, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')\n    hidden = tf.nn.relu(pool + b1)\n    conv = tf.nn.conv2d(hidden, w2, [1, 1, 1, 1], padding='SAME')\n    pool = tf.nn.max_pool(conv, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')\n    hidden = tf.nn.relu(pool + b2)\n    reshape = tf.reshape(hidden, (-1, PIXELS // 4 * PIXELS // 4 * 2*DEPTH))\n    hidden = tf.nn.relu(tf.matmul(reshape, w3) + b3)\n    return tf.matmul(hidden, w4) + b4"},{"cell_type":"markdown","metadata":{"_cell_guid":"12e19722-2713-afc0-5665-1963af498966"},"source":"### Loss\nWe use the cross entropy loss as our cost metric."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"555fd8d0-7716-7cc8-17c3-7427cb9276b6"},"outputs":[],"source":"loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits(tf_train_data), tf_train_labels))"},{"cell_type":"markdown","metadata":{"_cell_guid":"9f83f5cc-dd67-1e27-d46f-02a0d6b08fce"},"source":"### Optimizer\nAs optimizer we can use a Gradient Descent Optimizer with decaying learning rate or the more sophisticated (and easier to optimize!) Adam Optimizer."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"bf9c6a09-58e8-ff57-9002-ebfa8ca41175"},"outputs":[],"source":"#learning_rate = tf.train.exponential_decay(LEARNING_RATE, global_step, 1000, DECAY_FACTOR, staircase=False)\n#optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\noptimizer = tf.train.AdamOptimizer(1e-4).minimize(loss)"},{"cell_type":"markdown","metadata":{"_cell_guid":"ec8208b4-6a7d-f6f1-d23b-7164a2b455ed"},"source":"### Outputs"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"31a957e0-5883-9f44-53c4-255777b16ebe"},"outputs":[],"source":"train_prediction = tf.nn.softmax(logits(tf_train_data))\nvalid_prediction = tf.nn.softmax(logits(tf_valid_data))\ntest_prediction = tf.nn.softmax(logits(tf_test_data))"},{"cell_type":"markdown","metadata":{"_cell_guid":"769077b0-eff8-ceee-723a-0cbfe1fa87e0"},"source":"## Run Session"},{"cell_type":"markdown","metadata":{"_cell_guid":"35244d22-1ada-54e5-2a83-92debc3ede2f"},"source":"open the session"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f3d8bc95-1efe-8429-8eda-b7fcc78052be"},"outputs":[],"source":"session = tf.Session()\ntf.global_variables_initializer().run(session=session)"},{"cell_type":"markdown","metadata":{"_cell_guid":"3830ec55-9de8-8936-de65-8ddae63fde92"},"source":"Run the session (Run this cell again if the desired accuracy is not yet reached)."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0faca4c6-f71e-60be-ffca-66441fed07ec"},"outputs":[],"source":"_step = 0\nfor step in np.arange(STEPS):\n    _step += 1\n    if _step*BATCH > TRAIN: # Reshuffle data\n        shuffle(train_data, train_labels)\n        _step = 0\n    start = (step * BATCH) % (TRAIN - BATCH); stop = start + BATCH\n    batch_data = train_data[start:stop]\n    batch_labels = train_labels[start:stop, :]\n\n    feed_dict = {tf_train_data:batch_data, tf_train_labels:batch_labels}\n    opt, batch_loss, batch_prediction = session.run([optimizer, loss, train_prediction], feed_dict=feed_dict)\n\n    if (step % 1000 == 0):\n        b_acc = acc(batch_prediction, batch_labels) # Batch Accuracy\n        v_acc = acc(valid_prediction.eval(session=session), valid_labels) # Valid Accuracy\n        print('Step %i'%step, end='\\t')\n        print('Loss = %.2f'%batch_loss, end='\\t')\n        print('Batch Acc. = %.1f'%b_acc, end='\\t\\t')\n        print('Valid. Acc. = %.1f'%v_acc, end='\\n')\n        #print('learning rate = %.4f'%learning_rate.eval())"},{"cell_type":"markdown","metadata":{"_cell_guid":"bed7ce11-5a18-56d7-f526-8ebc103b0ce7"},"source":"## Results"},{"cell_type":"markdown","metadata":{"_cell_guid":"8e2b5cb1-2cb0-2873-22a6-11d02fc7c267"},"source":"Make a prediction about the test labels"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3d9691dd-1bd9-b6d5-9920-c619fd7443fc"},"outputs":[],"source":"test_labels = np.argmax(test_prediction.eval(session=session), axis=1)"},{"cell_type":"markdown","metadata":{"_cell_guid":"d6649ff2-2787-a737-9ffb-d8d55a0799e7"},"source":"Plot an example"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ea9f1fea-a6fd-6e63-2510-c98161146eb9"},"outputs":[],"source":"k = 0 # Try different images indices k\nplt.imshow(test_data[k,:,:,0])\nplt.axis('off')\nplt.show()\nprint(\"Label Prediction: %i\"%test_labels[k])"},{"cell_type":"markdown","metadata":{"_cell_guid":"83654d47-11bc-acc7-4761-d443efd44474"},"source":"## Submission"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3db12c10-bb63-22b8-5658-9f6bd740cc6c"},"outputs":[],"source":"submission = DataFrame(data={'ImageId':(np.arange(test_labels.shape[0])+1), 'Label':test_labels})\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"982a581a-ef18-fd33-848c-e53afcdde40d"},"source":"## Close Session\n\n(note: once the session is closed, one cannot rerun the cell where the training is performed...)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"53d600e1-cae8-a8fb-29e7-dcafae3d7979"},"outputs":[],"source":"#session.close()"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}