{"cells": [{"metadata": {"_uuid": "af6b6858618f056b490c1e14bc09313c45787e75", "_cell_guid": "3ec95489-a865-4d21-a4b6-12b124abb3b3"}, "cell_type": "markdown", "source": ["

TensorFlow Notebook for digit recognition with ~97% accuracy

\n", "\n", "

We start by importing the need libraries

"]}, {"metadata": {"_uuid": "dc9f54cce65dc6f1793af3d93f9b6b3d908e0165", "collapsed": true, "_cell_guid": "d84089a9-3aa9-4028-be06-5cc189404a43"}, "outputs": [], "cell_type": "code", "source": ["import numpy as np\n", "import tensorflow as tf\n", "import matplotlib.pyplot as plt\n", "import csv\n", "import matplotlib.cm as cm\n", "from tensorflow.python.framework import ops\n", "import math\n", "from random import randint\n", "%matplotlib inline"], "execution_count": null}, {"metadata": {"_uuid": "9e24035b8d1fcbee65c4662a32fa08534f9dadec", "_cell_guid": "311c065e-bfe0-45f1-9e2b-228e18776f69"}, "cell_type": "markdown", "source": ["

Define Helper functions

\n", "\n", "
dense_to_1hot function
\n", "[1,2,4,9,0] \n", "\n", "will become: \n", "\n", "[[ 0. 0. 0. 0. 1.]\n", " [ 1. 0. 0. 0. 0.]\n", " [ 0. 1. 0. 0. 0.]\n", " [ 0. 0. 0. 0. 0.]\n", " [ 0. 0. 1. 0. 0.]\n", " [ 0. 0. 0. 0. 0.]\n", " [ 0. 0. 0. 0. 0.]\n", " [ 0. 0. 0. 0. 0.]\n", " [ 0. 0. 0. 0. 0.]\n", " [ 0. 0. 0. 1. 0.]]"]}, {"metadata": {"_uuid": "18e8bbe3d602f47f26eac0d71ba1e09ca8da2cfe", "collapsed": true, "_cell_guid": "a360dc3d-b104-4f80-8d30-99ac67c5ebb4"}, "outputs": [], "cell_type": "code", "source": ["def dense_to_1hot(labels, depth):\n", " C = tf.constant(depth, dtype=tf.int32)\n", " one_hot_matrix = tf.one_hot(labels, C, axis=0)\n", " one_hot = sess.run(one_hot_matrix)\n", " return one_hot"], "execution_count": null}, {"metadata": {"_uuid": "fcd8f6f71ae3930d8b3e75ec4cca97ac0ccf9bcb", "_cell_guid": "9452d65d-5f51-48d3-97b2-861c0589cd19"}, "cell_type": "markdown", "source": ["
random_mini_batches function
\n", "Takes our training data, shuffles it, and divides into mini batches to run a mini batch adam optimizer later."]}, {"metadata": {"_uuid": "e2317e75c6e5cb042bf93efe09a7713f83e79b01", "collapsed": true, "_cell_guid": "0c8ac8f1-a152-4859-a176-d48382064174"}, "outputs": [], "cell_type": "code", "source": ["def random_mini_batches(X, Y, mini_batch_size = 128, seed = 0):\n", " np.random.seed(seed)\n", " m = X.shape[1] # number of training examples\n", " mini_batches = []\n", " \n", " # Step 1: Shuffle (X, Y)\n", " permutation = list(np.random.permutation(m))\n", " shuffled_X = X[:, permutation]\n", " shuffled_Y = Y[:, permutation].reshape((10,m))\n", "\n", " # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.\n", " num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning\n", " for k in range(0, num_complete_minibatches):\n", " mini_batch_X = shuffled_X[:, k * mini_batch_size: (k+1) * mini_batch_size]\n", " mini_batch_Y = shuffled_Y[:, k * mini_batch_size: (k+1) * mini_batch_size]\n", " mini_batch = (mini_batch_X, mini_batch_Y)\n", " mini_batches.append(mini_batch)\n", " \n", " # Handling the end case (last mini-batch < mini_batch_size)\n", " if m % mini_batch_size != 0:\n", " mini_batch_X = shuffled_X[:, (k+1) * mini_batch_size: ]\n", " mini_batch_Y = shuffled_Y[:, (k+1) * mini_batch_size: ]\n", " mini_batch = (mini_batch_X, mini_batch_Y)\n", " mini_batches.append(mini_batch)\n", " \n", " return mini_batches"], "execution_count": null}, {"metadata": {"_uuid": "523afdd2e2c0489156fb13019c4871cf919eec2b", "_cell_guid": "9d3a2115-9129-4f4f-b6f5-9957742bc9c2"}, "cell_type": "markdown", "source": ["
create_placeholders function
\n", "function that we will call in our tensorflow graph to create the needed placeholders"]}, {"metadata": {"_uuid": "aff3cf0de8840689aabda5b0d7c9f5cd9af89dad", "collapsed": true, "_cell_guid": "0081b7ca-8efa-449f-85ef-98ffb719c754"}, "outputs": [], "cell_type": "code", "source": ["def create_placeholders(n_x, n_y):\n", " X = tf.placeholder(dtype=tf.float32, shape=(n_x, None))\n", " Y = tf.placeholder(dtype=tf.float32, shape=(n_y, None))\n", " keep_prob = tf.placeholder(dtype=tf.float32)\n", " return X, Y, keep_prob"], "execution_count": null}, {"metadata": {"_uuid": "2ece076dc5345599788508e61a6d10d254ffb444", "_cell_guid": "218e5045-bf6f-40e4-b52b-8d51d749864d"}, "cell_type": "markdown", "source": ["
initialize_parameters function
\n", "function that will initialize our parameters for the neural network with 2 hidden layers, 50 units in the first, 25 in the second, and 10 for the output layer"]}, {"metadata": {"_uuid": "859fd1bd93ebd6923d4e7a74f3db2058edb232ca", "collapsed": true, "_cell_guid": "aa6c69a0-7cc3-47fb-a05c-ed7b67629c7f"}, "outputs": [], "cell_type": "code", "source": ["def initialize_parameters():\n", " W1 = tf.get_variable(\"W1\", [50,784], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n", " b1 = tf.get_variable(\"b1\", [50,1], initializer = tf.zeros_initializer())\n", " W2 = tf.get_variable(\"W2\", [25,50], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n", " b2 = tf.get_variable(\"b2\", [25,1], initializer = tf.zeros_initializer())\n", " W3 = tf.get_variable(\"W3\", [10, 25], initializer = tf.contrib.layers.xavier_initializer(seed=1))\n", " b3 = tf.get_variable(\"b3\", [10,1], initializer = tf.zeros_initializer())\n", " parameters = {\"W1\": W1,\n", " \"b1\": b1,\n", " \"W2\": W2,\n", " \"b2\": b2,\n", " \"W3\": W3,\n", " \"b3\": b3}\n", " \n", " return parameters"], "execution_count": null}, {"metadata": {"_uuid": "fccdcd3f301e34fa74a734c4e2d78e3a4183fe04", "_cell_guid": "818e3141-cbab-4db6-b024-d80607b4e960"}, "cell_type": "markdown", "source": ["
forward_propagation function
\n", "forward prop function with dropout regularization. We pass parameter keep_prob as 1 if we don't want to regularize with dropout or at prediciton time."]}, {"metadata": {"_uuid": "f46d94182b4003b5a478f219a7202a861ca506b7", "collapsed": true, "_cell_guid": "fefae35b-71f4-4b38-bae5-cf555a13afe6"}, "outputs": [], "cell_type": "code", "source": ["def forward_propagation(X, parameters, keep_prob):\n", " W1 = parameters['W1']\n", " b1 = parameters['b1']\n", " W2 = parameters['W2']\n", " b2 = parameters['b2']\n", " W3 = parameters['W3']\n", " b3 = parameters['b3']\n", " \n", " Z1 = tf.add(tf.matmul(W1, X), b1) \n", " Z1_drop = tf.nn.dropout(Z1, keep_prob)\n", " A1 = tf.nn.relu(Z1_drop) \n", " Z2 = tf.add(tf.matmul(W2, A1), b2) \n", " Z2_drop = tf.nn.dropout(Z2, keep_prob)\n", " A2 = tf.nn.relu(Z2_drop) \n", " Z3 = tf.add(tf.matmul(W3, A2), b3) \n", " Z3_drop = tf.nn.dropout(Z3, keep_prob)\n", " \n", " return Z3_drop"], "execution_count": 2}, {"metadata": {"_uuid": "f89d3aee8df5f9cb9cb51431d3ee0fcdcba8657e", "_cell_guid": "0cea47c1-d288-4ce6-a975-24b943302988"}, "cell_type": "markdown", "source": ["
compute_cost function
\n", "Self explanatory..."]}, {"metadata": {"_uuid": "b4deb77ce21558ee3a563e8cd2ead21797717449", "collapsed": true, "_cell_guid": "67b87baa-d3ae-461e-959a-b55295260093"}, "outputs": [], "cell_type": "code", "source": ["def compute_cost(Z3, Y):\n", " #transpose Z3 and Y for the softmax_cross_entropy function\n", " logits = tf.transpose(Z3)\n", " labels = tf.transpose(Y)\n", " cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = logits, labels = labels)) \n", " return cost"], "execution_count": null}, {"metadata": {"_uuid": "67e0b6da931d5b7be022b49402e443536663583f", "_cell_guid": "23d7bc81-eaec-4cac-94ae-a1347c573f53"}, "cell_type": "markdown", "source": ["
model function
\n", "This funciton will contain the whole of our tensorflow graph linked together."]}, {"metadata": {"_uuid": "31824413cb8d94a454558ec3371150c3e1578037", "collapsed": true, "_cell_guid": "68b35c9c-a3e5-4f89-8742-0b3325e95767"}, "outputs": [], "cell_type": "code", "source": ["def model(X_train, Y_train, X_test, Y_test, learning_rate = 0.0001,\n", " num_epochs = 200, minibatch_size = 32, print_cost = True, keep_prob=0.5):\n", " \n", " ops.reset_default_graph() \n", " seed = 1 \n", " (n_x, m) = X_train.shape \n", " n_y = Y_train.shape[0] \n", " costs = [] \n", " \n", " # Create Placeholders of shape (n_x, n_y)\n", " X, Y, keep_prob_ph = create_placeholders(X_train.shape[0], Y_train.shape[0])\n", " \n", " # Initialize parameters\n", " parameters = initialize_parameters()\n", " \n", " # Forward propagation: Build the forward propagation in the tensorflow graph\n", " Z3 = forward_propagation(X, parameters, keep_prob_ph)\n", " \n", " # Cost function: Add cost function to tensorflow graph\n", " cost = compute_cost(Z3, Y)\n", " \n", " # Backpropagation: Define the tensorflow optimizer.\n", " optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost)\n", " \n", " # Initialize all the variables\n", " init = tf.global_variables_initializer()\n", "\n", " # Start the session to compute the tensorflow graph\n", " with tf.Session() as sess:\n", " \n", " # Run the initialization\n", " sess.run(init)\n", " \n", " # Do the training loop\n", " for epoch in range(num_epochs+1):\n", "\n", " epoch_cost = 0. \n", " num_minibatches = int(m / minibatch_size) \n", " seed = seed + 1\n", " minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)\n", "\n", " for minibatch in minibatches:\n", "\n", " # Select a minibatch\n", " (minibatch_X, minibatch_Y) = minibatch\n", " \n", " _ , minibatch_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y, keep_prob_ph: keep_prob})\n", " \n", " epoch_cost += minibatch_cost / num_minibatches\n", "\n", " # Print the cost every 100 epoch\n", " if print_cost == True and epoch % 100 == 0:\n", " print (\"Cost after epoch %i: %f\" % (epoch, epoch_cost))\n", " if print_cost == True and epoch % 5 == 0:\n", " costs.append(epoch_cost)\n", " #Print the last epoch cost if it is not a multiple of 100\n", " if epoch % 100 != 0: \n", " print (\"Cost after epoch %i: %f\" % (epoch, epoch_cost)) \n", " \n", " # plot the cost\n", " plt.plot(np.squeeze(costs))\n", " plt.ylabel('cost')\n", " plt.xlabel('iterations (per tens)')\n", " plt.title(\"Learning rate =\" + str(learning_rate))\n", " plt.show()\n", "\n", " # save the parameters\n", " parameters = sess.run(parameters)\n", " print (\"Parameters have been trained!\")\n", "\n", " # Calculate the correct predictions\n", " correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y))\n", "\n", " # Calculate accuracy on the test set\n", " accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n", "\n", " print (\"Train Accuracy:\", accuracy.eval({X: X_train, Y: Y_train, keep_prob_ph: 1}))\n", " print (\"Test Accuracy:\", accuracy.eval({X: X_test, Y: Y_test, keep_prob_ph: 1}))\n", " \n", " return parameters"], "execution_count": null}, {"metadata": {"_uuid": "1832901c1e0433ffeb9940a08e9cae0f85e56ef6", "_cell_guid": "ef2df5e2-ba80-4995-b45c-3d78a2cb36e2"}, "cell_type": "markdown", "source": ["
predict function
\n", "Our predict function to run when we're done training, and we want to predict on our test set."]}, {"metadata": {"_uuid": "5f416a54a051dd8bc3f2b2006cec5448a888f1bf", "collapsed": true, "_cell_guid": "22ff45a4-58ba-4a3d-9c83-703d2b173107"}, "outputs": [], "cell_type": "code", "source": ["def predict(X, parameters):\n", " W1 = tf.convert_to_tensor(parameters[\"W1\"])\n", " b1 = tf.convert_to_tensor(parameters[\"b1\"])\n", " W2 = tf.convert_to_tensor(parameters[\"W2\"])\n", " b2 = tf.convert_to_tensor(parameters[\"b2\"])\n", " W3 = tf.convert_to_tensor(parameters[\"W3\"])\n", " b3 = tf.convert_to_tensor(parameters[\"b3\"])\n", " \n", " params = {\"W1\": W1,\n", " \"b1\": b1,\n", " \"W2\": W2,\n", " \"b2\": b2,\n", " \"W3\": W3,\n", " \"b3\": b3}\n", " \n", " x = tf.placeholder(\"float\", [784, None])\n", " \n", " z3 = forward_propagation(x, params, 1)\n", " p = tf.argmax(z3)\n", " \n", " sess = tf.Session()\n", " prediction = sess.run(p, feed_dict = {x: X})\n", " \n", " return prediction"], "execution_count": 1}, {"metadata": {"_uuid": "bd18982d6cfdb16dad81a8637bafdf42b6f81e76", "_cell_guid": "839d8ea2-1ae2-4a85-af9e-4c6468604414"}, "cell_type": "markdown", "source": ["
write_predictions function
\n", "Write our predictions to file"]}, {"metadata": {"_uuid": "c0c15d4e268f8444e1640c12711be3a6016ecf09", "collapsed": true, "_cell_guid": "288d46ef-4148-4203-99b2-8ef997947d59"}, "outputs": [], "cell_type": "code", "source": ["def write_predictions(predictions):\n", " with open('kaggle/submission_nn_tf.csv', 'w') as subs:\n", " subs.write(\"ImageId,Label\\n\")\n", " for i, pred in enumerate(predictions):\n", " subs.write(str(i+1)+','+str(pred)+'\\n')"], "execution_count": null}, {"metadata": {"_uuid": "4537da32b9c90d0d7db1c26749016be7f3da65d7", "_cell_guid": "23d563b3-f7a7-4f0b-9583-219a53106575"}, "cell_type": "markdown", "source": ["

Prepare Data

\n", "Prepare our training set, validation set, and test set to train our model, validate it and predict on the test set."]}, {"metadata": {"_uuid": "dd2355c1fbe91f657895e514bbc1de3b72718a27", "scrolled": true, "collapsed": true, "_cell_guid": "6c2cb22f-3216-4d67-9db1-e277b6f5b4db"}, "outputs": [], "cell_type": "code", "source": ["totalTrain = 42000\n", "tainPart = 40000\n", "testPart = 2000\n", "data = []\n", "sess = tf.Session()\n", "with open('../input/train.csv', 'r') as train:\n", " pixelReader = csv.reader(train, delimiter=',')\n", " next(pixelReader, None)\n", " for row in pixelReader:\n", " data.append(row[0:])\n", " \n", "data = np.array(data).astype(int)\n", "np.random.shuffle(data)\n", "data_train, data_test = data[:tainPart,:], data[tainPart:,:]\n", "labels_train_dense = data_train[:, 0]\n", "labels_test_dense = data_test[:, 0]\n", "\n", "data_train = np.multiply(data_train[:, 1:].T, 1/255)\n", "data_test = np.multiply(data_test[:, 1:].T, 1/255)\n", "assert(data_train.shape == (784, tainPart))\n", "assert(data_test.shape == (784, testPart))\n", "\n", "labels_train = dense_to_1hot(labels_train_dense, 10)\n", "labels_test = dense_to_1hot(labels_test_dense, 10)\n", "assert(labels_train.shape == (10, tainPart))\n", "assert(labels_test.shape == (10, testPart))\n", "\n", "image_to_show = 10\n", "plt.axis('off')\n", "plt.imshow(data_train[:, image_to_show].reshape(28, 28), cmap=cm.binary)\n", "sess.close()\n", "\n", "#read test data\n", "test = []\n", "with open('../input/test.csv', 'r') as train:\n", " pixelReader = csv.reader(train, delimiter=',')\n", " next(pixelReader, None)\n", " for row in pixelReader:\n", " test.append(row[0:])\n", "\n", "test = np.array(test).astype(int)\n", "test = np.multiply(test, 1.0 / 255.0)\n", "test = test.T"], "execution_count": null}, {"metadata": {"_uuid": "7fc928805ec2cfa8f101dc777cb0db2a78a3ee57", "_cell_guid": "5c13442e-6085-422a-91b6-fe0e0643b091"}, "cell_type": "markdown", "source": ["

Train Model

"]}, {"metadata": {"_uuid": "731ce60ba6a19558f8292fcd190a82389e315073", "collapsed": true, "_cell_guid": "ecfceee3-6d8f-4678-99f7-fdfddb7afcf1"}, "outputs": [], "cell_type": "code", "source": ["parameters = model(data_train, labels_train, data_test, labels_test, learning_rate = 0.0007, num_epochs = 750, keep_prob=0.99, minibatch_size = 512)"], "execution_count": null}, {"metadata": {"_uuid": "3be1ed156ca26850f58b26527ee5bd0dfa8bba34", "_cell_guid": "ee950b72-42cf-4f94-aed7-412bfc084252"}, "cell_type": "markdown", "source": ["

Predict And Write Predictions

"]}, {"metadata": {"_uuid": "433e81890cee235e765dd29f78c559c3e2a63502", "collapsed": true, "_cell_guid": "cfd5d583-129c-42b3-9d3f-d09458864a41"}, "outputs": [], "cell_type": "code", "source": ["predictions = predict(test, parameters)\n", "#write_predictions(predictions)"], "execution_count": null}, {"metadata": {"_uuid": "fde58950435ce811d2e5824fa4ba9e1498a1dcce", "collapsed": true, "_cell_guid": "b95f4017-98c8-4ac3-a292-8922239594d4"}, "outputs": [], "cell_type": "code", "source": [], "execution_count": null}, {"metadata": {"_uuid": "a72d61f752bb3da96b01e7ea511a5927a0a46453", "collapsed": true, "_cell_guid": "22491136-b521-485d-a6aa-31a1c9341c11"}, "outputs": [], "cell_type": "code", "source": [], "execution_count": null}], "nbformat": 4, "metadata": {"language_info": {"pygments_lexer": "ipython3", "codemirror_mode": {"name": "ipython", "version": 3}, "version": "3.6.3", "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python"}, "kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}}, "nbformat_minor": 1}