{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"ebbe6bba-c88c-0d49-502a-573d19476310"},"source":"A Convolutional Neural Network for MNIST Classification. This solution got me a score of 0.98929 on the leaderboard.\n\nNote: this solution is loosely based on the official [tensorflow tutorial](https://www.tensorflow.org/tutorials/mnist/pros/)."},{"cell_type":"markdown","metadata":{"_cell_guid":"6f552ccc-f472-107c-e20a-1c03a9362532"},"source":"## Packages and Imports"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"15754c6a-4ee3-0ac0-3d01-9ec4831370e3"},"outputs":[],"source":"%matplotlib inline\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import ShuffleSplit\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder"},{"cell_type":"markdown","metadata":{"_cell_guid":"d3088dc8-906f-32e9-db54-59a4c075bc31"},"source":"## Simulation Constants ##\n\nDownload notebook and use commented out values for better performance."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"50083ab2-7752-56fb-54e8-249a4017bd29"},"outputs":[],"source":"LABELS = 10 # Number of different types of labels (1-10)\nWIDTH = 28 # width / height of the image\nCHANNELS = 1 # Number of colors in the image (greyscale)\n\nVALID = 10000 # Validation data size\n\nSTEPS = 3500 #20000   # Number of steps to run\nBATCH = 100 # Stochastic Gradient Descent batch size\nPATCH = 5 # Convolutional Kernel size\nDEPTH = 8 #32 # Convolutional Kernel depth size == Number of Convolutional Kernels\nHIDDEN = 100 #1024 # Number of hidden neurons in the fully connected layer\n\nLR = 0.001 # Learning rate"},{"cell_type":"markdown","metadata":{"_cell_guid":"4572603b-cc72-031c-f710-6196172efea4"},"source":"## Import Data"},{"cell_type":"markdown","metadata":{"_cell_guid":"b734dc0f-50d6-37c7-7906-409d5b49f3cb"},"source":"prepare data by \n\n- applying 1-hot encoding: `1 = [1,0,0...0], 2 = [0,1,0...0] ...`\n- reshaping into image shape: `(# images, # vertical height, # horizontal width, # colors)`\n- splitting data into train and validation set."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4cd92946-fe19-5590-7946-1f43e2a68e48"},"outputs":[],"source":"data = pd.read_csv('../input/train.csv') # Read csv file in pandas dataframe\nlabels = np.array(data.pop('label')) # Remove the labels as a numpy array from the dataframe\nlabels = LabelEncoder().fit_transform(labels)[:, None]\nlabels = OneHotEncoder().fit_transform(labels).todense()\ndata = StandardScaler().fit_transform(np.float32(data.values)) # Convert the dataframe to a numpy array\ndata = data.reshape(-1, WIDTH, WIDTH, CHANNELS) # Reshape the data into 42000 2d images\ntrain_data, valid_data = data[:-VALID], data[-VALID:]\ntrain_labels, valid_labels = labels[:-VALID], labels[-VALID:]\n\nprint('train data shape = ' + str(train_data.shape) + ' = (TRAIN, WIDTH, WIDTH, CHANNELS)')\nprint('labels shape = ' + str(labels.shape) + ' = (TRAIN, LABELS)')"},{"cell_type":"markdown","metadata":{"_cell_guid":"616030e9-9c6e-c369-c6f2-c19cd76a542b"},"source":"## Model\n\nLet's now build a network with two convolutional layers, followed by one fully connected layer. Since this is computationally pretty expensive, we'll limit the depth and number of fully connected nodes for this online notebook.\n\nWe initialize the input data with placeholders"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f7b2abd6-d89d-1149-751f-3697f9afb52c"},"outputs":[],"source":"tf_data = tf.placeholder(tf.float32, shape=(None, WIDTH, WIDTH, CHANNELS))\ntf_labels = tf.placeholder(tf.float32, shape=(None, LABELS))"},{"cell_type":"markdown","metadata":{"_cell_guid":"fa565617-eaf9-af98-d4f6-b7a513f019f8"},"source":"We choose a 4 layered network consisting of 2 convolutional layers with weights and biases `(w1, b1)` and `(w2,b2)`, followed by a fully connected hidden layer `(w3,b3)` with #`HIDDEN` hidden neurons and an output layer `(w4, b4)` with `10` output nodes (one-hot encoding).\n\nWe initialize the weights and biases such that the kernel has a patch size of `PATCH` and the depth of the second convolutional layer is twice the depth of the first convolutional layer `(DEPTH)`. For the rest, the fully connected hidden layer has `HIDDEN` hidden neurons."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8c23848b-e5df-548c-698a-d30313129b23"},"outputs":[],"source":"w1 = tf.Variable(tf.truncated_normal([PATCH, PATCH, CHANNELS, DEPTH], stddev=0.1))\nb1 = tf.Variable(tf.zeros([DEPTH]))\nw2 = tf.Variable(tf.truncated_normal([PATCH, PATCH, DEPTH, 2*DEPTH], stddev=0.1))\nb2 = tf.Variable(tf.constant(1.0, shape=[2*DEPTH]))\nw3 = tf.Variable(tf.truncated_normal([WIDTH // 4 * WIDTH // 4 * 2*DEPTH, HIDDEN], stddev=0.1))\nb3 = tf.Variable(tf.constant(1.0, shape=[HIDDEN]))\nw4 = tf.Variable(tf.truncated_normal([HIDDEN, LABELS], stddev=0.1))\nb4 = tf.Variable(tf.constant(1.0, shape=[LABELS]))\n\ndef logits(data):\n    # Convolutional layer 1\n    x = tf.nn.conv2d(data, w1, [1, 1, 1, 1], padding='SAME')\n    x = tf.nn.max_pool(x, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')\n    x = tf.nn.relu(x + b1)\n    # Convolutional layer 2\n    x = tf.nn.conv2d(x, w2, [1, 1, 1, 1], padding='SAME')\n    x = tf.nn.max_pool(x, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')\n    x = tf.nn.relu(x + b2)\n    # Fully connected layer\n    x = tf.reshape(x, (-1, WIDTH // 4 * WIDTH // 4 * 2*DEPTH))\n    x = tf.nn.relu(tf.matmul(x, w3) + b3)\n    return tf.matmul(x, w4) + b4\n\n# Prediction:\ntf_pred = tf.nn.softmax(logits(tf_data))"},{"cell_type":"markdown","metadata":{"_cell_guid":"bc95200a-61a2-5f5a-0dc0-71009c5601dc"},"source":"We use the categorical cross entropy loss for training the model.\n\nAs optimizer we could use a Gradient Descent optimizer [with or without decaying learning rate] or one of the more sophisticated (and easier to optimize) optimizers like `Adam` or `RMSProp`"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"429d6634-ba23-9644-84f0-9173627d1f96"},"outputs":[],"source":"tf_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits(tf_data), tf_labels))\ntf_acc = 100*tf.reduce_mean(tf.to_float(tf.equal(tf.argmax(tf_pred, 1), tf.argmax(tf_labels, 1))))\n\n#tf_opt = tf.train.GradientDescentOptimizer(LR)\n#tf_opt = tf.train.AdamOptimizer(LR)\ntf_opt = tf.train.RMSPropOptimizer(LR)\ntf_step = tf_opt.minimize(tf_loss)"},{"cell_type":"markdown","metadata":{"_cell_guid":"6dde9903-2b67-6d57-0200-3a2041a786b6"},"source":"## Train"},{"cell_type":"markdown","metadata":{"_cell_guid":"3a183269-2293-7a6b-0cc9-ba51d3fc9428"},"source":"open the session"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"555fd8d0-7716-7cc8-17c3-7427cb9276b6"},"outputs":[],"source":"init = tf.global_variables_initializer()\nsession = tf.Session()\nsession.run(init)"},{"cell_type":"markdown","metadata":{"_cell_guid":"9f83f5cc-dd67-1e27-d46f-02a0d6b08fce"},"source":"Run the session (Run this cell again if the desired accuracy is not yet reached)."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"bf9c6a09-58e8-ff57-9002-ebfa8ca41175"},"outputs":[],"source":"ss = ShuffleSplit(n_splits=STEPS, train_size=BATCH)\nss.get_n_splits(train_data, train_labels)\nhistory = [(0, np.nan, 10)] # Initial Error Measures\nfor step, (idx, _) in enumerate(ss.split(train_data,train_labels), start=1):\n    fd = {tf_data:train_data[idx], tf_labels:train_labels[idx]}\n    session.run(tf_step, feed_dict=fd)\n    if step%500 == 0:\n        fd = {tf_data:valid_data, tf_labels:valid_labels}\n        valid_loss, valid_accuracy = session.run([tf_loss, tf_acc], feed_dict=fd)\n        history.append((step, valid_loss, valid_accuracy))\n        print('Step %i \\t Valid. Acc. = %f'%(step, valid_accuracy), end='\\n')"},{"cell_type":"markdown","metadata":{"_cell_guid":"ec8208b4-6a7d-f6f1-d23b-7164a2b455ed"},"source":"Visualize the training history:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"31a957e0-5883-9f44-53c4-255777b16ebe"},"outputs":[],"source":"steps, loss, acc = zip(*history)\n\nfig = plt.figure()\nplt.title('Validation Loss / Accuracy')\nax_loss = fig.add_subplot(111)\nax_acc = ax_loss.twinx()\nplt.xlabel('Training Steps')\nplt.xlim(0, max(steps))\n\nax_loss.plot(steps, loss, '-o', color='C0')\nax_loss.set_ylabel('Log Loss', color='C0');\nax_loss.tick_params('y', colors='C0')\nax_loss.set_ylim(0.01, 0.5)\n\nax_acc.plot(steps, acc, '-o', color='C1')\nax_acc.set_ylabel('Accuracy [%]', color='C1');\nax_acc.tick_params('y', colors='C1')\nax_acc.set_ylim(1,100)\n\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"769077b0-eff8-ceee-723a-0cbfe1fa87e0"},"source":"## Results"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"35244d22-1ada-54e5-2a83-92debc3ede2f"},"outputs":[],"source":"test = pd.read_csv('../input/test.csv') # Read csv file in pandas dataframe\ntest_data = StandardScaler().fit_transform(np.float32(test.values)) # Convert the dataframe to a numpy array\ntest_data = test_data.reshape(-1, WIDTH, WIDTH, CHANNELS) # Reshape the data into 42000 2d images"},{"cell_type":"markdown","metadata":{"_cell_guid":"f3d8bc95-1efe-8429-8eda-b7fcc78052be"},"source":"Make a prediction about the test labels"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0faca4c6-f71e-60be-ffca-66441fed07ec"},"outputs":[],"source":"test_pred = session.run(tf_pred, feed_dict={tf_data:test_data})\ntest_labels = np.argmax(test_pred, axis=1)"},{"cell_type":"markdown","metadata":{"_cell_guid":"bed7ce11-5a18-56d7-f526-8ebc103b0ce7"},"source":"Plot an example:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3d9691dd-1bd9-b6d5-9920-c619fd7443fc"},"outputs":[],"source":"k = 0 # Try different image indices k\nprint(\"Label Prediction: %i\"%test_labels[k])\nfig = plt.figure(figsize=(2,2)); plt.axis('off')\nplt.imshow(test_data[k,:,:,0]); plt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"83654d47-11bc-acc7-4761-d443efd44474"},"source":"## Submission"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3db12c10-bb63-22b8-5658-9f6bd740cc6c"},"outputs":[],"source":"submission = pd.DataFrame(data={'ImageId':(np.arange(test_labels.shape[0])+1), 'Label':test_labels})\nsubmission.to_csv('submission.csv', index=False)\nsubmission.tail()"},{"cell_type":"markdown","metadata":{"_cell_guid":"982a581a-ef18-fd33-848c-e53afcdde40d"},"source":"## Close Session\n\n(note: once the session is closed, the training cell cannot be run again...)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"53d600e1-cae8-a8fb-29e7-dcafae3d7979"},"outputs":[],"source":"#session.close()"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}