{"cells":[{"metadata":{"_uuid":"9d5ffc0ddafb37438617d8fda7213edf1958fc95"},"cell_type":"markdown","source":"This notebook covers a standard implementation of a convolutional neural network (convnet) for recognising 28x28 pixel greyscale images of digits. The network is implemented in tensorflow which requires we build the model first as a graph, then train it on the data. There are roughly three steps, processing the data so that it is in the correct format for the model, building the model,  and training/evaluating the model.\n\nThe processing step is mainly onehot encoding the labels from the training data, and scaling the pixel intensities from [0-255] to [0-1]. \n\nThe model building step is where we can be creative. Each layer consists of a set of weights and biases, of arbitrary dimensions. We can create convolutional layers that pass a filter over the data at each stage to produce a new feature (the idea being that this allows us to use the 2 dimensional structure of the image). We can also use pooling to reduce the amount of new features by taking the maximum over some window. Finally we can use dropout to help minimise overfitting to the training data. \n\nThe training step involves feeding stochastic mini-batches of the training data to the model. We can evaluate the performance by first splitting the training set into a train and validation set, and comparing the miscalssification error rates on this validation set for different models/parameters.","outputs":[],"execution_count":null},{"metadata":{"_uuid":"ecfdaa581e4f4656db759461b1b0c75ae347a8b3"},"cell_type":"markdown","source":" ### Step 1 - Processing Data","outputs":[],"execution_count":null},{"metadata":{"_cell_guid":"fab6c4f0-fcbd-4a1a-b1c0-d77c1bd33571","collapsed":true,"_uuid":"cd6cbf5a1f70a6690424dd5099da66360b7c48db","trusted":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport tensorflow as tf\n\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4846d355d00218817332b0af4b951315aefea756"},"cell_type":"code","source":"X = train.drop('label', axis=1).values \nwidth = int(np.sqrt(X.shape[1])) # this os only true if we know the images are square.\nheight = width # useful to be able to think of height and width separately, despite being the same here.\nchannels = 1 # number of colour channels in this case. ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b394c3ef1371a0cccf82b5ff87d379777e9c882d"},"cell_type":"code","source":"# helper function for formatting the data for use with tensorflow (which prefers np.float32 inputs).\n# the default values are for this data set.\ndef format_data(X, height=28, width=28, channels=1, normalisation=255.0):\n    # Reshape each image from a single array/vector into a 3 dimensional tensor.\n    # i.e. each channel of each image is represented as a matrix.\n    x = np.reshape(X, (X.shape[0], height, width, channels))\n    x = x.astype(np.float32)\n    return x/normalisation\nX_train = format_data(X)\nX_test = format_data(test.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"fa889f6339f87475bf02f8a098cb353cbfc07109"},"cell_type":"code","source":"# helper to onehot encode the labels\n# this requires that the labels are 0, 1, ...\ndef onehot_encode(y,num_classes=10):\n    onehot = np.arange(num_classes)==y[:,None]\n    return onehot.astype(np.float32)\ny_train = onehot_encode(train['label'].values)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f2fd1df287550df35ace818a19d26396bb5499f"},"cell_type":"markdown","source":"### Step 2 - Building the model","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d983fd7065e1bd3453963dbd5a26bedd04a303f9"},"cell_type":"code","source":"# helper functions for creating weights (W), biases (b), convolutions (conv) and pooling (pool).\ndef W(shape):\n    return tf.Variable(tf.truncated_normal(shape, stddev = 0.1))\ndef b(shape):\n    return tf.Variable(tf.constant(0.0, shape = shape))\n\n# we fix padding as SAME here. \n# Can also be VALID, but you need to be careful with how this affects dimensions between layers.\n# Strides are only in the width and height dimensions.\n# we don't want to convolve between multiple images or channels (there's only one channel here anyway).\ndef conv(x,W, stride=1):\n    return tf.nn.conv2d(x,W,strides = [1,stride,stride,1], padding = \"SAME\")\n\n# Similarly the window and strides here are 1 in the image-index and channels dimensions as we want these separate.\ndef pool(x, window = 2, stride = 2):\n    return tf.nn.max_pool(x,ksize = [1,window,window,1], strides = [1,stride,stride,1], \n                          padding = 'SAME')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"91bc1075bd66600cd553f42d00a18b510e1e3846"},"cell_type":"code","source":"# The model has 2 convolutional layers, followed by a fully connected layer and an output layer.\n# Most the parameters can be edited. The strides in the convolutions and pooling, \n# and the pooling window are fixed, but these could be made paramters too.\n\ndef model(X, height = 28, width = 28, channels = 1, \n          first_layer_patch_size = 5, first_layer_depth = 32,\n          second_layer_patch_size = 5, second_layer_depth = 64, \n          activation = tf.nn.relu, hidden_layer_depth = 1024, dropout_percent = 0.5, output_depth = 10):\n    # First convolutional layer\n    W1 = W([first_layer_patch_size,first_layer_patch_size,channels,first_layer_depth])\n    b1 = b([first_layer_depth])\n    h1 = activation(conv(X,W1) + b1)\n    hp1 = pool(h1, stride=2)\n    \n    # Second convolutional layer\n    W2 = W([second_layer_patch_size, second_layer_patch_size, first_layer_depth, second_layer_depth])\n    b2 = b([second_layer_depth])\n    h2 = activation(conv(hp1, W2) + b2)\n    hp2 = pool(h2, stride=2)\n\n    # Due to the strides in a pooling step, we shrink the image by 2 in each dimension. This is done twice.\n    flat_dim = (height//(2*2)) * (width//(2*2))\n    hp2_flat = tf.reshape(hp2, [-1, flat_dim * second_layer_depth])\n    \n    # fully connected layer\n    W_fc = W([flat_dim*second_layer_depth, hidden_layer_depth])\n    b_fc = b([hidden_layer_depth])\n    h_fc = tf.nn.relu(tf.matmul(hp2_flat, W_fc) + b_fc)\n    h_drop = tf.nn.dropout(h_fc, dropout_percent)\n    W_o = W([hidden_layer_depth, output_depth])\n    b_o = b([output_depth])\n    return tf.matmul(h_drop, W_o) + b_o","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8e1ff919265eacd746f0946ac3533b8f500bd5f"},"cell_type":"code","source":"# set up tensorflow graph\ngraph = tf.Graph()\nwith graph.as_default():\n    # placeholders for inputting minibatch data, and a dropout percent.\n    # Can use placeholders to feed in other parameters also.\n    X = tf.placeholder(tf.float32, shape = (None, height, width, channels))\n    y = tf.placeholder(tf.float32, shape = (None, 10))\n    dropout = tf.placeholder(tf.float32, shape = ())\n    \n    # Use the model.\n    logits = model(X, dropout_percent = dropout)\n    # Cross entropy loss.\n    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels = y, logits = logits))\n    \n    # Learning rate decay. \n    # The idea is as the model learns, we want the gradient descent to take finer steps.\n    # Too larger steps might always overshoot a minimum loss. \n    # However too smaller steps take a long time to converge, \n    # and could also get stuck in a sub-optimal loss.\n    # Decaying the step size over time should help the optimizer get near to a minimum in few steps, \n    # then fine tune around this minimum.\n    global_step = tf.Variable(0, trainable=False)\n    starter_learning_rate = 0.0004\n    learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step,\n                                           1000, 0.98, staircase=True)\n    \n    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)\n    \n    prediction = tf.nn.softmax(logits)\n    correct = tf.equal(tf.argmax(prediction,1), tf.argmax(y,1))\n    accuracy = 100*tf.reduce_mean(tf.cast(correct, 'float'))\n    predict = tf.argmax(prediction, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"761f75ffea5c70449b8a0ddba451801653a4fd89"},"cell_type":"markdown","source":"### Step 3 - Model Training, and Validation/Evaluation.","outputs":[],"execution_count":null},{"metadata":{"trusted":true,"_uuid":"a2027733e52771fc4d43726cf52e1d31c9980164"},"cell_type":"code","source":"# Need to split off a validation set from the train data.\n# Reset X_train here as I've reused X_train as the variable name after the split.\nX_train = format_data(train.drop('label', axis=1).values)\nfrom sklearn.model_selection import train_test_split\nX_train, X_valid, y_train, y_valid = train_test_split(X_train,y_train, test_size = 0.2, random_state = 42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"43c8c9c7094cc2e2f8aac6624d7365d9b17a7fa0"},"cell_type":"code","source":"# validation cycle\nnum_steps = 501\nbatch_size = 50\ntrain_accuracies = []\nvalid_accuracies = []\nwith tf.Session(graph=graph) as session:\n    tf.global_variables_initializer().run()\n    for step in range(num_steps):\n        # Semi-stochastic mini-batch selection, as used in the Udacity deep learning course.\n        offset = (step * batch_size) % (y_train.shape[0] - batch_size)\n        batch_data = X_train[offset:(offset + batch_size), :, :, :]\n        batch_labels = y_train[offset:(offset + batch_size), :]\n        feed_dict = {X : batch_data, y : batch_labels, dropout:0.75}\n        \n        # Note you seem to have to pass the optimizer out at each step.\n        _  = session.run([optimizer], feed_dict=feed_dict)\n        if (step % 100 == 0):\n            print(\"Step {}\".format(step))\n            train_accuracy = accuracy.eval(feed_dict = feed_dict)\n            train_accuracies.append(train_accuracy)\n            print('Minibatch training accuracy: {}'.format(train_accuracy))\n            valid_offset = (step*batch_size)%(y_valid.shape[0] - batch_size)\n            valid_batch_data = X_valid[valid_offset:(valid_offset+batch_size), :,:,:]\n            valid_batch_labels = y_valid[valid_offset:(valid_offset + batch_size),:]\n            valid_accuracy = accuracy.eval(feed_dict= {X:valid_batch_data, y:valid_batch_labels, dropout:1})\n            valid_accuracies.append(valid_accuracy)\n            print('Minibatch validation accuracy: {}'.format(valid_accuracy))\n    # Compute accuracy on the full validation set at the last step.\n    full_validation_accuracy = accuracy.eval(feed_dict = {X:X_valid, y:y_valid, dropout:1})\n    print(\"Full Validation Accuracy: \", full_validation_accuracy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d250f8b3e6e03c992d735dcf95fb004ddec5174a"},"cell_type":"code","source":"# Plots of the accuracies on minibatches.\nimport matplotlib.pyplot as plt\nsteps = np.arange(num_steps/100)*100\nplt.plot(steps, train_accuracies, c = 'b', label = \"Train\")\nplt.plot(steps, valid_accuracies, c='r', label = \"Valid\")\nplt.legend(loc=4)\nplt.ylabel(\"Accuracy\")\nplt.xlabel(\"Steps\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb4fcaae6e5ce2e0c84cd087c359c5e6d143965c"},"cell_type":"code","source":"# full cycle for the test data\nnum_steps = 5001\nbatch_size = 100\ntrain_accuracies = []\n\nX_full_train = train.drop('label', axis = 1).values\ny_full_train = onehot_encode(train['label'].values)\nX_full_train = format_data(X_full_train)\n\nwith tf.Session(graph=graph) as session:\n    tf.global_variables_initializer().run()\n    for step in range(num_steps):\n        offset = (step * batch_size) % (y_full_train.shape[0] - batch_size)\n        batch_data = X_full_train[offset:(offset + batch_size), :, :, :]\n        batch_labels = y_full_train[offset:(offset + batch_size), :]\n        feed_dict = {X : batch_data, y : batch_labels, dropout:0.75}\n        _ = session.run(\n            [optimizer], feed_dict=feed_dict)\n        if (step % 500 == 0):\n            print(\"Step {}\".format(step))\n            train_accuracy = accuracy.eval(feed_dict = feed_dict)\n            train_accuracies.append(train_accuracy)\n            print('Minibatch accuracy: {}'.format(train_accuracy))\n    test_size = X_test.shape[0]\n    test_predictions = np.zeros(test_size)\n    # Performance is better to predict in batches.\n    for i in range(0, test_size//batch_size):\n        test_predictions[i*batch_size:(i+1)*batch_size] = predict.eval(feed_dict = {X:X_test[i*batch_size:(i+1)*batch_size], dropout : 1})\n        \nsubmission_df = pd.DataFrame(test_predictions.astype(int))\nsubmission_df.columns = ['Label']\nsubmission_df['ImageId'] = submission_df.index + 1\nsubmission_df = submission_df.set_index('ImageId')\nsubmission_df.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"83b59fd4e6b2ac1951d145c23067b5fbb9d72454"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}