{"nbformat": 4, "cells": [{"source": ["# Introduction\n", "\n", "These are my notes while learning how to implement a Convolutional Neural Network (ConvNet or CNN) to solve the problem of digit recognition. I am using the [MNIST data set from Kaggle](https://www.kaggle.com/c/digit-recognizer). \n", "\n", "I wrote these notes by following the method described in the [Deep MNIST for Experts tutorial](https://www.tensorflow.org/get_started/mnist/pros) on Tensorflow.org. I also took some ideas from [Kirill Kliavin's notebook](https://www.kaggle.com/kakauandme/tensorflow-deep-nn) and a post by Imanol Schlag on [the Tensorflow input pipeline](http://ischlag.github.io/2016/06/19/tensorflow-input-pipeline-example/). "], "metadata": {"_cell_guid": "c33b4aa3-8ca5-4b6f-93b6-3d2e8cdcb2a5", "_uuid": "67fb2979c4fc63306a97aa589ab94f05a9e0a28a"}, "cell_type": "markdown"}, {"source": ["# Convolutional Neural Network\n", "\n", "ConvNets are similar to Neural Networks (NN) in that they have an input layer connected to hidden layers and an output layer. They are also trained using backprop. ConvNets are different in that their architecture is designed to take advantage of the assumption that the input layer is visual information. The difference lies in how the different layers are connected to each other: essentially, given a certain pixel or area in a picture, its most important connections are in the region surrounding it. This means that an input unit does not need to be fully connected to the hidden layer, reducing the number of parameters that are needed.   \n", "\n", "From [Stanford CS231](http://cs231n.github.io/convolutional-networks/):\n", "\n", ">Convolutional Neural Networks are very similar to ordinary Neural Networks [...]: they are made up of neurons that have learnable weights and biases. Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity. The whole network still expresses a single differentiable score function: from the raw image pixels on one end to class scores at the other. And they still have a loss function (e.g. SVM/Softmax) on the last (fully-connected) layer and all the tips/tricks we developed for learning regular Neural Networks still apply.\n", "\n", ">So what does change? ConvNet architectures make the explicit assumption that the inputs are images, which allows us to encode certain properties into the architecture. These then make the forward function more efficient to implement and vastly reduce the amount of parameters in the network.\n", "\n", "## CNN Architecture\n", "\n", "A CNN is a sequence of layers. Every layer of a CNN transforms one set of activations to another through a (differentiable) function. The set of activations is typically a volume of activations: each volume has height, width and channels (RGB, for example). The three main types of layers are Convolutional Layer, Pooling Layer and Fully-Connected Layer.\n", "\n", "The types of layers include:\n", "\n", "+ The INPUT layer holds the values of raw pixels in the image. In this case, we have 28 x 28 x 1 (h x w x channel) images\n", "+ The convolutional (CONV) layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected in the input volume. In this case, we will calculate 32 features/filters out of each 5x5 patch\n", "+ The output of the convolutional layer is fed to the ReLU layer, which will apply an element-wise activation such as max(0, x) thresholding at zero, so CONV-RELU can be abbreviated CONV\n", "+ The pooling layer (POOL) performs a downsampling operation. In this case, we will use max() over a 2x2 patch\n", "+ The fully-connected layers (FC), which will compute the class scores. Every neuron in this layer is connected to all numbers in previous volume\n", "\n", "A great resource on CNN are [Stanford's CS231 lecture notes](http://cs231n.github.io/convolutional-networks/#layers), from where I took most of the ideas above.\n", "\n", "Following the [Deep MNIST for Experts tutorial](https://www.tensorflow.org/get_started/mnist/pros), we will implement the following CNN:\n", "\n", "INPUT - CONV1 - POOL1 - CONV2 - POOL2 - FC1 - Dropout - FC2\n", "\n", "### Additional resources\n", "\n", "+ [Yann LeCun's website](http://yann.lecun.com/exdb/lenet/)\n", "+ [A post in StackExchange.com](https://datascience.stackexchange.com/questions/11619/rnn-vs-cnn-at-a-high-level) discussing the difference between a Convolutional NN and a Recurrent NN"], "metadata": {"_cell_guid": "2ad76b7f-1f7c-409a-93c1-acace40b4c2c", "_uuid": "4b7b249fed3da516fb8d1bb6508c5948b2fd5b52"}, "cell_type": "markdown"}, {"source": ["# The Tensorflow framework\n", "\n", "I will implement a ConvNet using Tensorflow. Tensors are the building blocks of Tensorflow; fundamentally, tensors are [abstractions of the matrix in higher dimensions](https://en.wikipedia.org/wiki/Tensor). In Tensorflow, tensors are used as inputs to operations which produce outputs that are tensors themselves and can serve as inputs to other operations. \n", "\n", "In Tensorflows, the operations are not run all at once. Instead, one defines a set of tensors, as well as a series of operations that will be applied on them and that will generate other tensors. Once all these relations are set, Tensorflow will run them in a Session outside of Python. Conceptually, operations can be thought as nodes in a graph, while vertices between nodes are tensors. \n", "\n", "From the description in the [Porgrammer's Guide section on Graphs and Sessions](https://www.tensorflow.org/programmers_guide/graphs):\n", "\n", ">TensorFlow uses a dataflow graph to represent your computation in terms of the dependencies between individual operations. This leads to a low-level programming model in which you first define the dataflow graph, then create a TensorFlow session to run parts of the graph across a set of local and remote devices.\n", "\n", "### Additional resources\n", "\n", "+ The [Programmer's Guide](https://www.tensorflow.org/programmers_guide/) is a good place to start\n", "+ [Building Machine Learning Estimator in TensorFlow](https://terrytangyuan.github.io/2016/07/08/understand-and-build-tensorflow-estimator/)\n", "+ [tf.contrib.learn Quickstart](https://www.tensorflow.org/versions/r0.12/tutorials/tflearn/)"], "metadata": {"_cell_guid": "8b70cd7d-46f7-4bfd-9c02-75712316ff2d", "_uuid": "a5d00581ad7004e59b7e7f84d69eea6b865a5a01"}, "cell_type": "markdown"}, {"source": ["# Creating the Tensorflow Graph\n", "\n", "## INPUT\n", "\n", "You are here:\n", "\n", "**INPUT** - CONV1 - POOL1 - CONV2 - POOL2 - FC1 - Dropout - FC2\n", "\n", "\n", "Let's start by loading the pandas and numpy libraries, as well as the [training and test data from Kaggle](https://www.kaggle.com/c/digit-recognizer/data). "], "metadata": {"_cell_guid": "e2ee22e7-8b1e-48b9-883f-39af43545e1c", "_uuid": "178df6e8d65be3d4815891b97d0ab2aa33783a90"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["import pandas as pd\n", "import numpy as np"], "outputs": [], "metadata": {"_cell_guid": "e818c285-bb02-4c97-b7a3-8d1c14593302", "collapsed": true, "_uuid": "66e07c552ea1edc27f27292562db880ff9db3cfc"}, "cell_type": "code"}, {"execution_count": null, "source": ["data = pd.read_csv('../input/train.csv')\n", "print('data.shape: ({0[0]}, {0[1]})'.format(data.shape))"], "outputs": [], "metadata": {"_cell_guid": "50e68da9-4d8c-437f-81cb-e853240c6548", "_uuid": "66c9bb28bc9a9a165f21a60615c5740f5737e96c"}, "cell_type": "code"}, {"execution_count": null, "source": ["test = pd.read_csv('../input/test.csv')\n", "print('test.shape: ({0[0]}, {0[1]})'.format(test.shape))"], "outputs": [], "metadata": {"_cell_guid": "3d904102-8c0e-4b6e-8685-36e37efc537d", "_uuid": "3d24f2ce37c3207db0f70d980a8c0bdd7b533de9"}, "cell_type": "code"}, {"source": ["The test data is already in a useable form, but the training data needs some prep work. In the following sections I will separate the image data from the labels. I will then go into tensors, placeholders and variables."], "metadata": {"_cell_guid": "05a7267c-1988-420d-82b9-be44e6c6b1e7", "_uuid": "468e036708cea301e04b4d22945f0b313027a18e"}, "cell_type": "markdown"}, {"source": ["### Images\n", "\n", "From the [data description in Kaggle](https://www.kaggle.com/c/digit-recognizer/data):\n", "\n", ">The training data set, (train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n", "\n", ">Each pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n", "\n", "Separate the images from labels to prep them separately. "], "metadata": {"_cell_guid": "02480610-83e0-45f6-86e8-622759bf1ad9", "_uuid": "56c765a0d317708a46db6a1c841436ce563a87a1"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["images = data.iloc[:, 1:].values\n", "print('images.shape: ({0[0]}, {0[1]})'.format(images.shape))"], "outputs": [], "metadata": {"_cell_guid": "47fd7015-924e-4859-bc2d-12b979725be6", "_uuid": "523202547690416127ff302379282972c65f82bf"}, "cell_type": "code"}, {"source": ["Rescale the image data: the previous scale had values between 0 and 254, the new one will be between 0 and 1."], "metadata": {"_cell_guid": "08586f69-a23a-4229-9a21-865119a5a46c", "_uuid": "e15889ddc3930855aff9e16f10e30eee6af4fa99"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["images = images.astype(np.float32)\n", "images = np.multiply(images, 1.0 / 255.0)"], "outputs": [], "metadata": {"_cell_guid": "b89781cb-9cd7-4ed6-be74-fd5ebc019752", "collapsed": true, "_uuid": "6b0dc776db38ae65f1dc3937f020db8a79fc7772"}, "cell_type": "code"}, {"execution_count": null, "source": ["test = test.astype(np.float32)\n", "test = np.multiply(test.as_matrix(), 1.0 / 255.0)"], "outputs": [], "metadata": {"_cell_guid": "b10d9c58-da7e-4b5b-adce-91927414f183", "collapsed": true, "_uuid": "d696d4ad0e9748c0ca437ba01307750316540c1d"}, "cell_type": "code"}, {"source": ["### Labels\n", "\n", "Convert the labels to one-hot variables, which are like dummy variables, but have one variable per value (as opposed to dummies, which would typically omit one value/variable). Some information about the difference between one-hot encoding and dummy variables can be found in [this post]((https://stats.stackexchange.com/questions/224051/one-hot-vs-dummy-encoding-in-scikit-learn). "], "metadata": {"_cell_guid": "97a29870-aba5-4b2c-b746-4f31cfbb8589", "_uuid": "d0d207c3832f4fcf2532b84700ebe7630ea9f563"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["labels_flat = data.iloc[:, 0].values.ravel()\n", "print('labels_flat.len: {0}'.format(len(labels_flat)))"], "outputs": [], "metadata": {"_cell_guid": "8a2e8410-700f-403b-aabf-839d1adb242c", "_uuid": "b2fc81602ed6833b7e57cea2ad65476277ff68f4"}, "cell_type": "code"}, {"execution_count": null, "source": ["labels_count = np.unique(labels_flat).shape[0]\n", "print('labels_count: {0}'.format(labels_count))"], "outputs": [], "metadata": {"_cell_guid": "adbf33b3-bcf0-4e4c-9859-db738d5c5003", "_uuid": "f9725b3d529758dba26f1d7c23dafe23ebfc216a"}, "cell_type": "code"}, {"source": ["The function to create the one-hot variables is from [Kirill's notebook](https://www.kaggle.com/kakauandme/tensorflow-deep-nn)."], "metadata": {"_cell_guid": "33f434a8-4fae-4f6d-bcfe-a6042fe4fdd2", "_uuid": "82b9cb4ebc490dd225a2e94d91ef734a4a017c36"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["def dense_to_one_hot(labels_dense, num_classes):\n", "    num_labels = labels_dense.shape[0]\n", "    index_offset = np.arange(num_labels) * num_classes\n", "    labels_one_hot = np.zeros((num_labels, num_classes))\n", "    labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1\n", "    return labels_one_hot"], "outputs": [], "metadata": {"_cell_guid": "6ed1286a-ccf2-420b-ab6e-8efa52b5a50b", "collapsed": true, "_uuid": "52197ee059a9280efd26ba8923ef180d580ae788"}, "cell_type": "code"}, {"execution_count": null, "source": ["labels = dense_to_one_hot(labels_flat, labels_count)\n", "labels = labels.astype(np.uint8)\n", "print('labels.shape: ({0[0]}, {0[1]})'.format(labels.shape))"], "outputs": [], "metadata": {"_cell_guid": "2436fe5a-254b-4986-b1ff-24fbe76c3c39", "_uuid": "e38f944abed946d1321f151c87c4e59561099758"}, "cell_type": "code"}, {"source": ["### Tensors, placeholders and variables\n", "\n", "The images and labels have been separated and converted to useful shapes. I will now create placeholders, which are tensors that will be initialized at runtime with values for training and validation further below. \n", "\n", "#### Placeholders\n", "\n", "As I mentioned above, tensors are the building blocks of Tensorflow. Placeholders, constants and variables are subclasses of tensors. As this [post discusses](https://stackoverflow.com/questions/36693740/whats-the-difference-between-tf-placeholder-and-tf-variable), the idea is to use placeholders for inputs and variables for trainable parameters such as weights and biases.  \n", "\n", "Placeholders need to be initialized by feeding its value using the `feed_dict`, which is an optional argument to methods such as `Session.run()`, `Tensor.eval()` or `Operation.run()`. "], "metadata": {"_cell_guid": "9ce5792d-c24b-44d6-9edb-043e47efaec5", "_uuid": "5e2c344bc5951ae257499c4e53b83e010c58051e"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["import tensorflow as tf\n", "\n", "x  = tf.placeholder(tf.float32, shape=[None, 784])\n", "y_ = tf.placeholder(tf.float32, shape=[None, 10])"], "outputs": [], "metadata": {"_cell_guid": "5269a600-18ca-4413-b44e-6a3fda9cae12", "_uuid": "c8873f51d9b2f636907e2927af869d19524425bc"}, "cell_type": "code"}, {"source": ["#### Variables\n", "\n", "Weights and biases are trainable parameters and, therefore, they will be variables. Weights are initialized randomly, using a value from a truncated normal distribution (all values more than 2 standard deviations away from the mean are repalced). In short, each weight gets a random value, most of them non-zero, but also not too far from zero."], "metadata": {"_cell_guid": "5c35e413-fb41-45c7-87a4-8d83bf45ab25", "_uuid": "11e8a038b776a48590485764e32a21eaeb82233f"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["def weight_variables(shape):\n", "    initial = tf.truncated_normal(shape, stddev = 0.1)\n", "    return tf.Variable(initial)\n", "\n", "def bias_variable(shape):\n", "    initial = tf.constant(0.1, shape = shape)\n", "    return tf.Variable(initial)"], "outputs": [], "metadata": {"_cell_guid": "5004e348-8942-4aa3-9fb6-1e0d1c8ef9fe", "collapsed": true, "_uuid": "f2bd7c31457d0ac333a9001b113593e1c792c6a9"}, "cell_type": "code"}, {"source": ["### Additional references\n", "\n", "+ The Programmer's Guide sections on [Tensors](https://www.tensorflow.org/programmers_guide/tensors) and [Variables](https://www.tensorflow.org/programmers_guide/variables) \n", "+ [tf.Placeholer](https://www.tensorflow.org/api_docs/python/tf/placeholder): Inserts a placeholder for a tensor that will be always fed. \n", "+ [tf.Variable](https://www.tensorflow.org/api_docs/python/tf/Variable): A variable maintains state in the graph across calls to run(). You add a variable to the graph by constructing an instance of the class Variable.\n", "+ [tf.truncated_normal](https://www.tensorflow.org/api_docs/python/tf/truncated_normal): Outputs random values from a truncated normal distribution. The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked. "], "metadata": {"_cell_guid": "3271a9ca-f3e7-4526-bf97-ef8d845489ed", "_uuid": "4a6afe969ecf48897836197c446e5ffe41817b3c"}, "cell_type": "markdown"}, {"source": ["## Convolution and Pooling\n", "\n", "\n", "### Convolution \n", "\n", "The convolution step consists of a set of learnable filters (weights), where every filter covers a small area (width x height), but goes through the entire depth of the training volume ([a \"depth column\"](http://xrds.acm.org/blog/2016/06/convolutional-neural-networks-cnns-illustrated-explanation/)). In this example this is not very relevant because we only have one channel or a depth of 1, but you can imagine this in three channels that represent something like RGB or CNY. There can be several filters and their activations are stored in a tensor, one per channel. These activations can be taken as inputs into later layers.\n", "\n", "There are a few features that are worth highlighting:\n", "\n", "1. One can implement a series of filters that apply over small areas (but through the depth of the tensor). After applying these filters in a convolutional layer, one can then collect their activations in later layers. The total number of parameters is reduced by using this approach as there is no layer that connects all pixels in the image to all the nodes in the layer.\n", "2. The output tensor will have a depth equal to the number of filters that are applied in the convolutional layer, one per filter. \n", "3. The stride of the filters is the number of pixels by which we move the filter (one/two/three/etc at a time).\n", "4. Padding the depth column with zeros is commonly used to preserve the area of the original tensor. \n", "\n", "A much better explanation can be found in the [CS231 site](http://cs231n.github.io/convolutional-networks/#conv).\n", "\n", "\n", "### ReLU functions\n", "\n", "The activation function can also be written as a layer. In this case the activation function after CONV will be a [rectifier ](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) `f(x) = max(0, x)`. \n", "\n", "### Pooling\n", "\n", "From [CS231](http://cs231n.github.io/convolutional-networks/#pool):\n", "\n", ">It is common to periodically insert a Pooling layer in-between successive Conv layers in a ConvNet architecture. Its function is to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting. The Pooling Layer operates independently on every depth slice of the input and resizes it spatially, using the MAX operation. The most common form is a pooling layer with filters of size 2x2 applied with a stride of 2 downsamples every depth slice in the input by 2 along both width and height, discarding 75% of the activations. Every MAX operation would in this case be taking a max over 4 numbers (little 2x2 region in some depth slice). The depth dimension remains unchanged. \n", "\n", "\n", "\n", "### Additional references\n", "\n", "+ [tf.nn.conv2d](https://www.tensorflow.org/api_docs/python/tf/nn/conv2d): Computes a 2-D convolution given 4-D input and filter tensors. \n", "+ [tf.nn.relu documentation](https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/relu): Computes rectified linear: max(features, 0).\n", "+ [tf.nn.max_pool](https://www.tensorflow.org/api_docs/python/tf/nn/max_pool): Performs the max pooling on the input.\n", "+ [Pooling](https://en.wikipedia.org/wiki/Convolutional_neural_network#Pooling_layer) in wiki.\n", "+ [Pooling](https://www.tensorflow.org/api_guides/python/nn#Pooling) in Tensorflow."], "metadata": {"_cell_guid": "b264e81c-8858-4e29-ad92-d280e54bb9aa", "_uuid": "c2599b06b72da84f7e34096787391b8fe57e595b"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["def conv2d(x, W):\n", "        return tf.nn.conv2d(x, W, strides =[1, 1, 1, 1], padding = 'SAME')\n", "    \n", "def max_pool_2x2(x):\n", "    return tf.nn.max_pool(x, ksize = [1, 2, 2, 1], \n", "                         strides = [1, 2, 2, 1], padding = 'SAME')"], "outputs": [], "metadata": {"_cell_guid": "3b1b7b6d-eaab-47a2-8569-09ab59344c2f", "collapsed": true, "_uuid": "03fac49e2bc91a0bf54f9accd381bb04a838a835"}, "cell_type": "code"}, {"source": ["## CONV1 - POOL1\n", "\n", "You are here:\n", "\n", "INPUT - **CONV1 - POOL1** - CONV2 - POOL2 - FC1 - Dropout - FC2\n", "\n", "In this CNN there will be two layers of CONV-RELU-POOL. The first convolutional layer will cover a 5x5 area or patch, it will have a depth of 1 (the images are greyscale) and it will apply 32 filters. We will need a weight vector with shape [5, 5, 1, 32], given by the size of the area (5x5), the depth (1) and the number of filters (32); it will also require a bias for every filter.  "], "metadata": {"_cell_guid": "a997a47b-0837-4666-a386-db46edb90869", "_uuid": "97d83f75e0fa393bd9340575b44b174061b90018"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["W_conv1 = weight_variables([5, 5, 1, 32])\n", "b_conv1 = bias_variable([32])"], "outputs": [], "metadata": {"_cell_guid": "36df9eb7-9767-416c-83e2-7580b945e46e", "collapsed": true, "_uuid": "8a5fb0bc8266d3ebfe1fdbd3ab9886eabd41d79f"}, "cell_type": "code"}, {"source": ["To apply the filter, we will need to reshape the image from a single 784-element array to a 28x28x1 image (where the last x1 is the number of channels). The -1 in the shape argument is a wildcard, in this case it will adjust for the number of images that are being processed in each batch. From the [tf.reshape documentation](https://www.tensorflow.org/api_docs/python/tf/reshape):   \n", "\n", ">If one component of shape is the special value -1, the size of that dimension is computed so that the total size remains constant."], "metadata": {"_cell_guid": "1ad6e40a-9f84-4ca2-a3bf-d4227d07d450", "_uuid": "26cd78424ccd9724b7b943f02fb4e670197b70a3"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["x_image = tf.reshape(x, [-1, 28, 28, 1])\n", "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)"], "outputs": [], "metadata": {"_cell_guid": "ad75724c-fe40-4a55-a680-c3678788261f", "collapsed": true, "_uuid": "8aefbca076bf0d07862d25de94b5692d72d43234"}, "cell_type": "code"}, {"execution_count": null, "source": ["print('h_conv1.shape: %s' % h_conv1.shape)"], "outputs": [], "metadata": {"_cell_guid": "68db9200-340d-4eda-a4c6-b4c69db7a5d7", "_uuid": "0d3db019c01c8956296d1af1b3ee86fc36af4252"}, "cell_type": "code"}, {"source": ["After convolution, the pooling layer reduces the area of the output volume from 28x28 to 14x14."], "metadata": {"_cell_guid": "29246a94-6f49-4bd5-bbbb-e0eebb5cb602", "_uuid": "4fc9fff3a59f6c9fa305b15f4a914ce7c71a9672"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["h_pool1 = max_pool_2x2(h_conv1)"], "outputs": [], "metadata": {"_cell_guid": "1e0c54e0-2945-4d0a-b1e9-d56354993817", "collapsed": true, "_uuid": "9ead8fc679a73dea70bb7e79d8752896888f8e80"}, "cell_type": "code"}, {"execution_count": null, "source": ["print('h_pool.shape: %s' % h_pool1.shape)"], "outputs": [], "metadata": {"_cell_guid": "73943d2d-608c-4155-aef5-4c97cd7ce091", "_uuid": "905c4ab3bb8597d03e60ca8ce3341cb720ec1e22"}, "cell_type": "code"}, {"source": ["## CONV2 - POOL2\n", "\n", "You are here:\n", "\n", "INPUT - CONV1 - POOL1 - **CONV2 - POOL2** - FC1 - Dropout - FC2\n", "\n", "The second layer takes 5x5 patches, with a depth of 32 and applies 64 filters, therefore the weights have shape [5, 5, 32, 64] with the respective 64 biases. \n", "\n", "The input to CONV2 is the output of POOL1, which is a *reduced* (from 28x28 to 14x14) version of the image after convolution. CONV2 will capture more general features of the image, while CONV1 would have captured finer details. After CONV2, POOL2 reduces the dimensions of the volume from 14x14 depth columns to 7x7. "], "metadata": {"_cell_guid": "825eadb2-450c-4907-af50-e5fe0c9769a7", "_uuid": "f1ca8f5c9539fa9a2ebee215526795a0d80f8450"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["W_conv2 = weight_variables([5, 5, 32, 64])\n", "b_conv2 = bias_variable([64])\n", "\n", "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n", "h_pool2 = max_pool_2x2(h_conv2)"], "outputs": [], "metadata": {"_cell_guid": "45c18b99-b612-4079-9249-3a9e2c0f31e6", "collapsed": true, "_uuid": "cb8709b1158171e963638a3a44c5674b9deb602b"}, "cell_type": "code"}, {"execution_count": null, "source": ["print('h_conv2.shape: %s' % h_conv2.shape)\n", "print('h_pool2.shape: %s' % h_pool2.shape)"], "outputs": [], "metadata": {"_cell_guid": "e9b916bd-59a7-44bb-a8ba-6771a50acd08", "_uuid": "db85566c488396d900596af88a1bcfc3391504ce"}, "cell_type": "code"}, {"source": ["## FC1: Fully-Connected Layer\n", "\n", "You are here:\n", "\n", "INPUT - CONV1 - POOL1 - CONV2 - POOL2 - **FC1** - Dropout - FC2\n", "\n", "After POOL2, the image has an area of 7x7 and a depth of 64. The first Fully Connected layer is added with 1024 nodes. To do this, first flatten the output of POOL2 to an array of 3136 (= 7\\*7\\*64) elements, multiply by their weights (3136x1024 matrix) and add a bias for each node."], "metadata": {"_cell_guid": "0dfff682-048e-4b76-9be2-df879aedc5c9", "_uuid": "fec3638dedcef595b262242e660902c243f4c4f3"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["W_fc1 = weight_variables([7*7*64, 1024])\n", "b_fc1 = bias_variable([1024])\n", "\n", "h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n", "h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)"], "outputs": [], "metadata": {"_cell_guid": "229c9d83-184d-4dd3-9dc3-e053e363c670", "collapsed": true, "_uuid": "6c8ad0af31f53dabb2c2845674e4508466a90eca"}, "cell_type": "code"}, {"execution_count": null, "source": ["print('h_pool2_flat.shape: %s' % h_pool2_flat.shape)"], "outputs": [], "metadata": {"_cell_guid": "0c7b327c-dc47-4bf0-bfec-2dea2a1becfd", "_uuid": "cff2d951a5cabcb30e447f858bf87765829f74d9"}, "cell_type": "code"}, {"source": ["## Dropout\n", "\n", "\n", "You are here:\n", "\n", "INPUT - CONV1 - POOL1 - CONV2 - POOL2 - FC1 - **Dropout** - FC2\n", "\n", "To prevent overfitting, Dropout is applied. From the [paper in which Dropout was first described](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf):\n", "\n", ">The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different \u201cthinned\u201d networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods.\n", "\n", "Notice that we specify `keep_prob` as a placeholder, since we will want to use different `keep_prob`s depending on what we are doing (say training vs evaluating).\n", "\n", "### Additional references\n", "\n", "+ From the [tf.nn.dropout documentation](https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/dropout): Computes dropout. With probability keep_prob, outputs the input element scaled up by 1 / keep_prob, otherwise outputs 0. The scaling is so that the expected sum is unchanged. "], "metadata": {"_cell_guid": "dd04da8a-13fe-4187-ad83-0a06ff7135c0", "_uuid": "1d272d81997040bf57cb702d86096d19e7534044"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["keep_prob = tf.placeholder(tf.float32)\n", "h_fcl_drop = tf.nn.dropout(h_fc1, keep_prob)"], "outputs": [], "metadata": {"_cell_guid": "df7b100f-99e0-4900-a1b3-476ff90b4025", "collapsed": true, "_uuid": "7b96fa3ac301e27e519402e29e9c2631527b8794"}, "cell_type": "code"}, {"source": ["## FC2: Readout Layer\n", "\n", "\n", "You are here:\n", "\n", "INPUT - CONV1 - POOL1 - CONV2 - POOL2 - FC1 - Dropout - **FC2**\n", "\n", "We add a readout layer which will be fed to a a [softmax layer](https://en.wikipedia.org/wiki/Softmax_function) in the next section. As the Wiki article states, a softmax function \"is a generalization of the logistic function that *squashes* a K-dimensional vector of arbitrary real values to a K-dimensional vector  of real values in the range [0, 1] that add up to 1\"."], "metadata": {"_cell_guid": "3b53709d-e55d-4fb6-a417-12a714c9dbfa", "_uuid": "4f1da2360457d24eeda6e59477b1a16ab9fc5593"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["W_fc2 = weight_variables([1024, 10])\n", "b_fc2 = bias_variable([10])\n", "\n", "y_conv = tf.matmul(h_fcl_drop, W_fc2) + b_fc2\n", "prediction = tf.argmax(y_conv, 1)"], "outputs": [], "metadata": {"_cell_guid": "807eb107-6624-447b-9ebb-b98c2f7b63bb", "collapsed": true, "_uuid": "67b68fd72b3938b8a888c5671295ab48ab39e58a"}, "cell_type": "code"}, {"source": ["# Evaluating the graph (model fitting)\n", "\n", "At this point, the graph has been specified and what remains is actually evaluating it. \n", "\n", "## Evaluation and Optimisation\n", "\n", "The ConvNet will be optimized using the Adam Optimizer using a cross-entropy cost function. Adam implements stochastic gradient descent. Its motivation is stated in the introduction to the paper where it was presented:\n", "\n", "> If the function is differentiable w.r.t. its parameters, gradient descent is a relatively efficient optimization method, since the computation of first-order partial derivatives w.r.t. all the parameters is of the same computational complexity as just evaluating the function. Often, objective functions are stochastic. For example, many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking gradient steps w.r.t. individual subfunctions, i.e. stochastic gradient descent (SGD) or ascent. SGD proved itself as an efficient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013). Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization. For all such noisy objectives, efficient stochastic optimization techniques are required. \n", "\n", "## Hyperparameters\n", "\n", "To keep things cleaner, the code chunk below sets the values for a number of hyperparameters.\n", "\n", "+ `VALIDATION_SET_SIZE` controls how many observations will be (randomly) selected to create a validation data set\n", "+ `BATCH_SIZE` is the number of observations that will be fed to the graph in every training iteration\n", "+ `TRAINING_ITERATIONS` is the number of training iterations. Every time an epoch passes, the training data is shuffled so that the next epoch is presented in a different order\n", "+ `LEARNING_RATE` and `EPSILON` are parameters of the Adam optimizer, the former is the upper bound of the step size (in most cases), while the latter is... beyond what I could learn while writing this (it appears to modify the SNR, which diminishes the update size of the parameters). For epsilon, the authors of the paper suggest a vale of 1e-8, while [tf.train.AdamOptimizer's documentation](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer) states that \n", "\n", ">The default value of 1e-8 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1.\n", "\n", "### Additional references\n", "\n", "+ [Adam Optimizer](https://arxiv.org/pdf/1412.6980.pdf)\n", "+ [Cross-entropy cost function](http://peterroelants.github.io/posts/neural_network_implementation_intermezzo01/#Cross-entropy-cost-function-for-the-logistic-function)"], "metadata": {"_cell_guid": "f3fe5490-6b0b-41aa-a854-03f4d547bfa0", "_uuid": "d9d260112d87af873e77b13b1cf5d7487c51db8c"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["VALIDATION_SET_SIZE = 4000\n", "\n", "BATCH_SIZE = 100\n", "TRAINING_ITERATIONS = 12000\n", "\n", "LEARNING_RATE = 1e-3\n", "EPSILON = 1e-8\n", "\n", "KEEP_RATE = 0.5"], "outputs": [], "metadata": {"_cell_guid": "329c37b4-90a9-4215-8cfc-02128b2abef4", "collapsed": true, "_uuid": "eda0afec69b5447e374c10efbee38ce8230d32e5"}, "cell_type": "code"}, {"source": ["With the hyperparameters set, now define the cost function and the training step."], "metadata": {"_cell_guid": "b34e8894-dc4c-4274-aa62-caddcb9e73fd", "_uuid": "da020b3210df9e03673e5a6ff716a466c4fe5258"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels= y_, logits = y_conv))\n", "train_step = tf.train.AdamOptimizer(learning_rate = LEARNING_RATE, epsilon = EPSILON).minimize(cross_entropy)"], "outputs": [], "metadata": {"_cell_guid": "d27a0fd9-2e36-479d-abc6-0454219effc4", "collapsed": true, "_uuid": "a72da892131d86292debb1781edd60728f62bbcc"}, "cell_type": "code"}, {"execution_count": null, "source": ["correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))\n", "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"], "outputs": [], "metadata": {"_cell_guid": "d2776600-fd04-4358-b2b6-87bd2cf562b5", "collapsed": true, "_uuid": "287b348f885941a696ec505a41659815ebe981ac"}, "cell_type": "code"}, {"source": ["## Training and Validation datasets\n", "\n", "The training and validation datasets are defined by random selection following [Imanol Schlag's post](http://ischlag.github.io/2016/06/19/tensorflow-input-pipeline-example/). Given a validation test size, m, first define an array of length n, the number of images, that contains m ones and n-m zeros. Shuffle it. "], "metadata": {"_cell_guid": "4d1e8fcb-a6c9-45e4-8dde-c957c924d557", "_uuid": "7ec4ce0fc26395bdbeda0303589afbf87ce78532"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["import random\n", "partitions = [0] * data.shape[0]\n", "partitions[:VALIDATION_SET_SIZE] = [1] * VALIDATION_SET_SIZE\n", "random.shuffle(partitions)"], "outputs": [], "metadata": {"_cell_guid": "e5be8d55-430b-4ffc-9644-06fae45b92b9", "collapsed": true, "_uuid": "952f03a8ac3aefe693dc87b2ab4ae748d35bc5bd"}, "cell_type": "code"}, {"source": ["Create placeholders for the image and labels data, which will take on the values of the training, validation and testing dataset, as applicable. Each of the placeholders are passed to [tf.dynamic_partition](https://www.tensorflow.org/versions/master/api_docs/python/tf/dynamic_partition), which will create two tensors based on the partition above."], "metadata": {"_cell_guid": "f4cd976c-ec48-484a-8570-a4199299661b", "_uuid": "306f4d52e1bdc2204f7816b8adbe66c9c683c78c"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["images_placeholder = tf.placeholder(images.dtype, images.shape)\n", "labels_placeholder = tf.placeholder(labels.dtype, labels.shape)"], "outputs": [], "metadata": {"_cell_guid": "36acd157-0a2a-4bb5-9882-b80f115e4ce5", "collapsed": true, "_uuid": "e9c6649a8d5a60afe343bcbcd763676b0a62b784"}, "cell_type": "code"}, {"execution_count": null, "source": ["training_images, validation_images = tf.dynamic_partition(images_placeholder, partitions = partitions, num_partitions = 2)\n", "training_labels, validation_labels = tf.dynamic_partition(labels_placeholder, partitions = partitions, num_partitions = 2)"], "outputs": [], "metadata": {"_cell_guid": "12ec5a00-dba6-41f4-a380-9f2db2a06c24", "collapsed": true, "_uuid": "d572f994b5dc517484739948db346b05e60ddbd7"}, "cell_type": "code"}, {"source": ["Now put the placeholders in datasets to take advantage of Tensorflow's iterators. Through iterators one can generate batches of a given size.\n", "\n", "### Additional resources\n", "\n", "+ [tf.iterator](https://www.tensorflow.org/versions/master/api_docs/python/tf/data/Iterator)"], "metadata": {"_cell_guid": "dcbd6316-c027-4c1b-88a6-9e44b54ecef4", "_uuid": "2621861dbeb1785e297396a66e37f295f518a1ca"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["training_dataset = tf.data.Dataset.from_tensor_slices((training_images, training_labels))\n", "training_dataset = training_dataset.shuffle(buffer_size = 10000)\n", "training_dataset = training_dataset.batch(BATCH_SIZE)\n", "training_iterator = training_dataset.make_initializable_iterator()\n", "next_batch = training_iterator.get_next() "], "outputs": [], "metadata": {"_cell_guid": "9e6e9d94-44cf-4565-80cb-07326db38d18", "collapsed": true, "_uuid": "dfb66b37581968c36781b38d6dbe79d1df62c205"}, "cell_type": "code"}, {"execution_count": null, "source": ["validation_dataset = tf.data.Dataset.from_tensor_slices((validation_images, validation_labels))\n", "validation_dataset = validation_dataset.batch(VALIDATION_SET_SIZE)\n", "validation_iterator = validation_dataset.make_initializable_iterator()\n", "validation_data = validation_iterator.get_next()"], "outputs": [], "metadata": {"_cell_guid": "7457fbbb-6ee1-4359-b9fb-ddd50ae1e197", "collapsed": true, "_uuid": "c31000413f5a1928e956be7eb7edcd3476e12e70"}, "cell_type": "code"}, {"source": ["# Train and validate\n", "\n", "This is where the magic happens. From Python you start a Session in Tensorflow. The session is run in a `with` block so that the session is closed at the end. Also, notice how a `feed_dict` is passed to initializers, runs and evals with the values of images, labels, etc. The training and validation sets are created within Tensorflow.  "], "metadata": {"_cell_guid": "430b796d-ecce-435e-bbf8-e4f91c8e1089", "_uuid": "c1c7fcbf386246a13eb46fedd8ed0834f7646ac6"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["display_step = 1\n", "\n", "with tf.Session() as sess:\n", "    sess.run(training_iterator.initializer, feed_dict = {images_placeholder: images,\n", "                                                         labels_placeholder: labels})\n", "    sess.run(tf.global_variables_initializer())\n", "    for i in range(TRAINING_ITERATIONS):\n", "        try:\n", "            batch = sess.run(next_batch)\n", "            if i % display_step == 0 or (i + 1) == TRAINING_ITERATIONS:\n", "                train_accuracy = accuracy.eval(feed_dict={x: batch[0],\n", "                                                          y_: batch[1], \n", "                                                          keep_prob: 1.0 })\n", "                sess.run(validation_iterator.initializer, feed_dict = {images_placeholder: images, \n", "                                                                       labels_placeholder: labels})\n", "                validation = sess.run(validation_data)\n", "                validation_accuracy = accuracy.eval(feed_dict = {x: validation[0], \n", "                                                                 y_: validation[1], \n", "                                                                 keep_prob: 1.0})\n", "                print('step %d. Training accuracy %g. Validation accuracy %g.' % (i, train_accuracy, validation_accuracy))\n", "            train_step.run(feed_dict = {x: batch[0], \n", "                                        y_: batch[1],\n", "                                        keep_prob: KEEP_RATE})\n", "        \n", "            # increase display_step\n", "            if i%(display_step*2) == 0 and i < 512:\n", "                display_step *= 2\n", "        \n", "        except tf.errors.OutOfRangeError:\n", "            print('End of Epoch. Initializing iterator.')\n", "            sess.run(training_iterator.initializer, feed_dict = {images_placeholder: images,\n", "                                                         labels_placeholder: labels})"], "outputs": [], "metadata": {"_cell_guid": "9c4d62ec-f421-44d5-98e8-08e072bbcbec", "_uuid": "ec53d568ba15efd4e6089c602cbe2774760628c1"}, "cell_type": "code"}], "nbformat_minor": 1, "metadata": {"language_info": {"mimetype": "text/x-python", "pygments_lexer": "ipython3", "name": "python", "nbconvert_exporter": "python", "version": "3.6.3", "file_extension": ".py", "codemirror_mode": {"name": "ipython", "version": 3}}, "celltoolbar": "Raw Cell Format", "kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3"}}}