{"metadata": {"language_info": {"name": "python", "mimetype": "text/x-python", "file_extension": ".py", "nbconvert_exporter": "python", "version": "3.6.4", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3"}, "kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3"}}, "nbformat": 4, "nbformat_minor": 1, "cells": [{"metadata": {"_cell_guid": "da7b5fc0-f3f6-4072-98bb-5a3172674472", "_uuid": "cbe00a7dfeed45b1db4ba9825501ab52cf52a244"}, "source": ["After implementing deep learning algorithm (only fully-connected layers introduced) and data augmentation, now let's try whether CNN could give us better output or not."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "bf6678e8-3782-4a93-ba2c-0756d15cb4ae", "_uuid": "6827cf66adf54f02c47aa97646a7698e626de6f9"}, "source": ["**LeNet-5 Structure**\n", "    \n", "    input: 28 * 28 images\n", "    zeropadding: (2, 2) => 32 * 32\n", "    conv_1: 5 * 5 and 6 filters => 28 * 28 * 6\n", "    act_1: activation\n", "    avgpool_2: 2 * 2 and non-overlapping => 14 * 14 * 6\n", "    conv_3: 5 * 5 and 16 filters => 10 * 10 * 16\n", "    act_2: activation\n", "    avgpool_4: 2 * 2 and non-overlapping => 5 * 5 * 16\n", "    conv_5: 5 * 5 and 120 filters => fully-connected 120 * 1\n", "    act_3: activation\n", "    fc_6: fully-connected 84 * 1\n", "    act_4: activation\n", "    output_layer(act_4): 10 * 1\n", "\n", "Note: not implementing bias in conv"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "5fef1e6d-440b-40fb-bea5-45545caa8ea8", "collapsed": true, "_uuid": "14a5d7b83eb2747868c785bf40aa03acd4ef8bbb"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "#import necessary modules\n", "import pandas as pd\n", "import tensorflow as tf\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.model_selection import train_test_split\n", "from tensorflow.python.framework import ops\n", "\n", "%matplotlib inline"]}, {"metadata": {"_cell_guid": "358da2cf-cdaa-4a08-bcd5-6d262436c09e", "_uuid": "88cad66760c8b7ed5318a1a5fbb3291b3caa821b"}, "source": ["**1 Preparing the data (Preprocessing)**\n", "\n", "1.1 Loading datasets\u00b6"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "0cf0ca88-77c7-417b-b67a-a099033ff571", "collapsed": true, "_uuid": "49bbf84b02c10fa1c54f34c91ca390856736b8f2"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["#read training and test sets\n", "data = pd.read_csv('data/train.csv')\n", "test = pd.read_csv('data/test.csv')"]}, {"metadata": {"_cell_guid": "662bd649-ba55-4d79-a802-9f48a22d7d7e", "collapsed": true, "_uuid": "dcc96dd03855f4ed8049c9ef9ef4b5bec8fff4ee"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["#set features and labels (arrays)\n", "features = data.iloc[:, 1:].values.reshape((42000,28,28,1))\n", "labels = data.iloc[:, 0].values.reshape((42000,1))\n", "test = test.values.reshape((28000,28,28,1))"]}, {"metadata": {"_cell_guid": "9b52a8b2-bee8-4cfa-98e6-3a26fde25d91", "_uuid": "46eb1eb97e0092cfb090a28831490e14e402851c"}, "source": ["1.2 Zero-padding to 32 * 32"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "cd819165-cd04-42d1-80cf-e758de2895d5", "collapsed": true, "_uuid": "53e50c00d5ec3719872d25868de8eebf2a46ebad"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def zero_pad(X, pad):\n", "    \"\"\"\n", "    Pad with zeros all images of the dataset X. The padding is applied to the height and width of an image, \n", "    as illustrated in Figure 1.\n", "    \n", "    Argument:\n", "    X -- python numpy array of shape (m, n_H, n_W, n_C) representing a batch of m images\n", "    pad -- integer, amount of padding around each image on vertical and horizontal dimensions\n", "    \n", "    Returns:\n", "    X_pad -- padded image of shape (m, n_H + 2*pad, n_W + 2*pad, n_C)\n", "    \"\"\"\n", "    \n", "    X_pad = np.pad(X, ((0,0), (pad,pad), (pad,pad), (0,0)), 'constant', constant_values = (0,0))\n", "    \n", "    return X_pad"]}, {"metadata": {"_cell_guid": "05e20c5c-9407-404d-9afa-3fe6f39975fe", "collapsed": true, "_uuid": "d0edb1e4f1a4afb49805c57332aac2ff237be68a"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["features = zero_pad(features, 2)\n", "test = zero_pad(test, 2)\n", "features.shape"]}, {"metadata": {"_cell_guid": "3a180009-a726-4c14-8b49-f1c9d23df7c3", "_uuid": "e8e49d8819d22b0bdbba69dbe05bd846329be390"}, "source": ["1.3 Split training and test set\u00b6"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "9cf43d6d-5c8d-48e1-bc54-f66b3151bbaa", "collapsed": true, "_uuid": "2e10d97bdfe1fecd4d0e9088169015620c5a3f6a"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["#train test split\n", "X_train, X_test, Y_train, Y_test = train_test_split(features,\n", "                                                   labels,\n", "                                                   test_size = 0.025,\n", "                                                   random_state = 0,\n", "                                                   stratify = labels)\n", "#no dev needed"]}, {"metadata": {"_cell_guid": "c9578408-a334-4ce0-96d7-9ca6ded7e91d", "_uuid": "4f05e32ad2193ef7f42564b0e27e08362a9aed5e"}, "source": ["1.4 Normalizing inputs and one-hot-coding labels"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "7899a34a-e7bc-43cf-a594-46bcb2e6f46e", "collapsed": true, "_uuid": "609f0a5bc7a19e3b3a81677f1575d05ebb6345c1"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["#Normalize image inputs\n", "def feature_norm(x):\n", "    '''\n", "    Normalize the feature values of x\n", "    Arguments:\n", "    x -- A scalar or numpy array of any size.\n", "\n", "    Return:\n", "    X_norm -- normalized x\n", "    '''\n", "    X_norm = x / 255\n", "    return X_norm\n", "\n", "X_train, X_test = feature_norm(X_train), feature_norm(X_test)"]}, {"metadata": {"_cell_guid": "0e811a91-8228-4499-a2e1-49b5271b0abe", "collapsed": true, "_uuid": "c8b496165bd13b8a768c4d6904650995898966de"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["#One-hot-code y values\n", "def convert_to_one_hot(Y, C):\n", "    Y = np.eye(C)[Y.reshape(-1)]\n", "    return Y\n", "\n", "Y_train, Y_test = convert_to_one_hot(Y_train, 10), convert_to_one_hot(Y_test, 10)"]}, {"metadata": {"_cell_guid": "b5b35206-f070-4f86-bfe1-f3b81be43c5d", "_uuid": "9ba729bffe86301233442cd183916f315f3a3a07"}, "source": ["**2 Build forward propagation**\n", "\n", "2.1 Loading input matrix, creating input layer"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "54e0c1ae-776e-495a-a3d3-bc9eff185f78", "collapsed": true, "_uuid": "34a4a2b46a483eb7754007865668fefadcbe74b3"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def create_placeholders(n_H0, n_W0, n_C0, n_y):\n", "    \"\"\"\n", "    Creates the placeholders for the tensorflow session.\n", "    \n", "    Arguments:\n", "    n_H0 -- scalar, height of an input image\n", "    n_W0 -- scalar, width of an input image\n", "    n_C0 -- scalar, number of channels of the input\n", "    n_y -- scalar, number of classes\n", "        \n", "    Returns:\n", "    X -- placeholder for the data input, of shape [None, n_H0, n_W0, n_C0] and dtype \"float\"\n", "    Y -- placeholder for the input labels, of shape [None, n_y] and dtype \"float\"\n", "    \"\"\"\n", "\n", "    X = tf.placeholder(tf.float32, shape = (None, n_H0, n_W0, n_C0))\n", "    Y = tf.placeholder(tf.float32, shape = (None, n_y))\n", "    \n", "    return X, Y"]}, {"metadata": {"_cell_guid": "cf35e6ae-4d1c-4bb5-9a01-93ff0d4b0638", "collapsed": true, "_uuid": "a122dd9a4192c15e17d6d355c42d09ea6f71ca2a"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["X, Y = create_placeholders(32, 32, 1, 10)"]}, {"metadata": {"_cell_guid": "24638fe0-9e86-4267-a27a-911fc91c0625", "_uuid": "99184b7661db26cb914a8bdf4a32a6b538eab354"}, "source": ["2.2 Setting conv parameter matrix"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "b262fb5f-9256-4fae-ab1e-da937f2c860f", "collapsed": true, "_uuid": "ca130a69ace8ae1b1399759abd2e70fb7132465c"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def initialize_parameters(C_F1 = 6, C_F3 = 16):\n", "    \"\"\"\n", "    Initializes weight parameters to build a neural network with tensorflow. The shapes are:\n", "                        C_W1 : [5, 5, 1, 16]\n", "                        C_W3 : [5, 5, 6, 32]\n", "                        C_W5 : [5, 5, 16, 120]\n", "                        FC => shape    [1, 120]\n", "                        FC_W6: [120, 84]\n", "                        FC_W7: [84, 10]\n", "                        \n", "    Returns:\n", "    parameters -- a dictionary of tensors containing W1, W2\n", "    \"\"\"\n", "    \n", "    tf.set_random_seed(1)                              # so that your \"random\" numbers match ours\n", "        \n", "    C_W1 = tf.get_variable(\"C_W1\", [5, 5, 1, C_F1], initializer = tf.contrib.layers.xavier_initializer(seed = 0))\n", "    C_W3 = tf.get_variable(\"C_W3\", [5, 5, C_F1, C_F3], initializer = tf.contrib.layers.xavier_initializer(seed = 0))\n", "    C_W5 = tf.get_variable(\"C_W5\", [5, 5, C_F3, 120], initializer = tf.contrib.layers.xavier_initializer(seed = 0))\n", "    FC_W6 = tf.get_variable(\"FC_W6\", [120, 84], initializer = tf.contrib.layers.xavier_initializer(seed = 0))\n", "    bias_6 = tf.Variable(tf.zeros([84]))\n", "    FC_W7 = tf.get_variable(\"FC_W7\", [84, 10], initializer = tf.contrib.layers.xavier_initializer(seed = 0))\n", "    bias_7 = tf.Variable(tf.zeros([10]))\n", "\n", "    parameters = {\"C_W1\": C_W1,\n", "                  \"C_W3\": C_W3,\n", "                  \"C_W5\": C_W5,\n", "                  \"FC_W6\": FC_W6,\n", "                  \"bias_6\": bias_6,\n", "                  \"FC_W7\": FC_W7,\n", "                  \"bias_7\": bias_7}\n", "    \n", "    return parameters"]}, {"metadata": {"_cell_guid": "fe25d8d4-ca17-4f32-8f98-a6f1df064959", "_uuid": "9b3025f42e97aae70c7dcdd903496fab0e97eb55"}, "source": ["2.2 Building forward propagation process\n", "\n", "When implementing conv2d, there's no need to set bias variables bcz the function sets bias computation automatically.\n", "\n", "    conv2d(\n", "        use_bias=True,\n", "        bias_initializer=tf.zeros_initializer())"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "ca60f865-7668-4ca3-9316-56fa24da6764", "collapsed": true, "_uuid": "1fc49a2b4a6dfb59c21603e9edd398085a59f478"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def forward_propagation(X, parameters, pooling = 'max'):\n", "    \"\"\"\n", "    Implements the forward propagation for the model:\n", "    CONV2D_1 -> RELU -> MAX/AVGPOOL_2 -> CONV2D_3 -> RELU -> MAX/AVGPOOL_4 -> CONV2D_5 -> RELU -> FLATTEN -> \n", "    FULLYCONNECTED_6 -> FULLYCONNECTED_7\n", "    \n", "    Arguments:\n", "    X -- input dataset placeholder, of shape (input size, number of examples)\n", "    parameters -- python dictionary containing your parameters \"W1\", \"W2\"\n", "                  the shapes are given in initialize_parameters\n", "\n", "    Returns:\n", "    Z7 -- the output of the last LINEAR unit\n", "    \"\"\"\n", "    \n", "    # Retrieve the parameters from the dictionary \"parameters\" \n", "    C_W1 = parameters['C_W1']\n", "    C_W3 = parameters['C_W3']\n", "    C_W5 = parameters['C_W5']\n", "    FC_W6 = parameters['FC_W6']\n", "    bias_6 = parameters['bias_6']\n", "    FC_W7 = parameters['FC_W7']\n", "    bias_7 = parameters['bias_7']\n", "    \n", "    # CONV2D: filters C_W1, stride of 1, padding 'SAME'\n", "    conv_1 = tf.nn.conv2d(X, C_W1, strides = [1,1,1,1], padding = 'VALID') #[m,h,w,c]\n", "    #print(conv_1)\n", "    \n", "    # padding arguement refer to \n", "    # https://stackoverflow.com/questions/37674306/what-is-the-difference-between-same-and-valid-padding-in-tf-nn-max-pool-of-t\n", "    # RELU\n", "    A1 = tf.nn.relu(conv_1)\n", "\n", "    if pooling == 'max':\n", "        # MAXPOOL: window 2x2, stride 2, padding 'SAME'\n", "        maxpool_2 = tf.nn.max_pool(A1, ksize = [1,2,2,1], strides = [1,2,2,1], padding = 'VALID')\n", "        #print(maxpool_2)\n", "    \n", "        # CONV2D: filters C_W3, stride 1, padding 'SAME'\n", "        conv_3 = tf.nn.conv2d(maxpool_2, C_W3, strides = [1,1,1,1], padding = 'VALID')\n", "        #print(conv_3)\n", "    \n", "        # RELU\n", "        A2 = tf.nn.relu(conv_3)\n", "        # MAXPOOL: window 2x2, stride 2, padding 'SAME'\n", "        maxpool_4 = tf.nn.max_pool(A2, ksize = [1,2,2,1], strides = [1,2,2,1], padding = 'VALID')\n", "        #print(maxpool_4)\n", "    \n", "        # CONV2D: filters C_W5, stride 1, padding 'SAME'\n", "        conv_5 = tf.nn.conv2d(maxpool_4, C_W5, strides = [1,1,1,1], padding = 'VALID')\n", "        #print(conv_5)\n", "        \n", "    elif pooling == 'avg':\n", "        # AVGPOOL: window 2x2, stride 2, padding 'SAME'\n", "        avgpool_2 = tf.nn.avg_pool(A1, ksize = [1,2,2,1], strides = [1,2,2,1], padding = 'VALID')\n", "        #print(maxpool_2)\n", "    \n", "        # CONV2D: filters C_W3, stride 1, padding 'SAME'\n", "        conv_3 = tf.nn.conv2d(avgpool_2, C_W3, strides = [1,1,1,1], padding = 'VALID')\n", "        #print(conv_3)\n", "    \n", "        # RELU\n", "        A2 = tf.nn.relu(conv_3)\n", "        # AVGPOOL: window 2x2, stride 2, padding 'SAME'\n", "        avgpool_4 = tf.nn.avg_pool(A2, ksize = [1,2,2,1], strides = [1,2,2,1], padding = 'VALID')\n", "        #print(maxpool_4)\n", "    \n", "        # CONV2D: filters C_W5, stride 1, padding 'SAME'\n", "        conv_5 = tf.nn.conv2d(avgpool_4, C_W5, strides = [1,1,1,1], padding = 'VALID')\n", "        #print(conv_5)\n", "    \n", "    # RELU\n", "    A3 = tf.nn.relu(conv_5)\n", "    # FLATTEN\n", "    fc_5 = tf.reshape(A3, [-1, 120]) #output: [batch_size, k]\n", "    # FULLY-CONNECTED\n", "    fc_6 = tf.nn.bias_add(tf.matmul(fc_5, FC_W6), bias_6)\n", "    # RELU\n", "    A4 = tf.nn.relu(fc_6)\n", "    # FULLY-CONNECTED without non-linear activation function (not call softmax).\n", "    # 6 neurons in output layer. Hint: one of the arguments should be \"activation_fn=None\" \n", "    fc_7 = tf.nn.bias_add(tf.matmul(A4, FC_W7), bias_7)\n", "    ### END CODE HERE ###\n", "\n", "    return fc_7"]}, {"metadata": {"_cell_guid": "f43decf2-6053-4dec-8056-3ed62c995613", "_uuid": "c7f94df38c7b6551b7169e9b36e1a151b8469dfb"}, "source": ["2.3 Build cost function"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "4a12db13-0779-400b-9b0b-631a07ab1d1f", "collapsed": true, "_uuid": "9a05fb3f5e046af02c0f30450e295646d3a998f7"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# GRADED FUNCTION: compute_cost \n", "\n", "def compute_cost(fc_7, Y):\n", "    \"\"\"\n", "    Computes the cost\n", "    \n", "    Arguments:\n", "    fc_7 -- output of forward propagation (output of the last LINEAR unit), of shape (number of examples, 10)\n", "    Y -- \"true\" labels vector placeholder, same shape as Z3\n", "    \n", "    Returns:\n", "    cost - Tensor of the cost function\n", "    \"\"\"\n", "    \n", "    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = fc_7, labels = Y)) #squeeze and compute mean\n", "    \n", "    return cost"]}, {"metadata": {"_cell_guid": "66b2c7ca-1a28-4c1f-9274-5edfac4f8460", "_uuid": "bcef8dcbead10d22ad073a3775e61557091334a6"}, "source": ["**3 Run CNN**\n", "\n", "3.1 Build model"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "e2ffee13-f5be-4d54-b9e6-84212ac42ce1", "collapsed": true, "_uuid": "34e527a2c50b35b99fbe9ff42ce446786e231047"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):\n", "    \"\"\"\n", "    Creates a list of random minibatches from (X, Y)\n", "    \n", "    Arguments:\n", "    X -- input data, of shape (input size, number of examples) (m, Hi, Wi, Ci)\n", "    Y -- true \"label\" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples) (m, n_y)\n", "    mini_batch_size - size of the mini-batches, integer\n", "    seed -- this is only for the purpose of grading, so that you're \"random minibatches are the same as ours.\n", "    \n", "    Returns:\n", "    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)\n", "    \"\"\"\n", "    \n", "    m = X.shape[0]                  # number of training examples\n", "    mini_batches = []\n", "    np.random.seed(seed)\n", "    \n", "    # Step 1: Shuffle (X, Y)\n", "    permutation = list(np.random.permutation(m))\n", "    shuffled_X = X[permutation,:,:,:]\n", "    shuffled_Y = Y[permutation,:]\n", "\n", "    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.\n", "    num_complete_minibatches = int(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning\n", "    for k in range(0, num_complete_minibatches):\n", "        mini_batch_X = shuffled_X[k * mini_batch_size : k * mini_batch_size + mini_batch_size,:,:,:]\n", "        mini_batch_Y = shuffled_Y[k * mini_batch_size : k * mini_batch_size + mini_batch_size,:]\n", "        mini_batch = (mini_batch_X, mini_batch_Y)\n", "        mini_batches.append(mini_batch)\n", "    \n", "    # Handling the end case (last mini-batch < mini_batch_size)\n", "    if m % mini_batch_size != 0:\n", "        mini_batch_X = shuffled_X[num_complete_minibatches * mini_batch_size : m,:,:,:]\n", "        mini_batch_Y = shuffled_Y[num_complete_minibatches * mini_batch_size : m,:]\n", "        mini_batch = (mini_batch_X, mini_batch_Y)\n", "        mini_batches.append(mini_batch)\n", "    \n", "    return mini_batches"]}, {"metadata": {"_cell_guid": "658ffb87-77eb-4e56-813a-e853512b0a80", "collapsed": true, "_uuid": "d9d676ad573199255de721cc524caf0bf84006b8"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["def model(X_train, Y_train, X_test, Y_test, test, learning_rate = 0.0005,\n", "          num_epochs = 30, minibatch_size = 64, pooling = 'max',\n", "          C_F1 = 6, C_F3 = 16,\n", "          print_cost = True, learning_decay = False):\n", "    \"\"\"\n", "    Implements a three-layer ConvNet in Tensorflow:\n", "    CONV2D_1 -> RELU -> MAXPOOL_2 -> CONV2D_3 -> RELU -> MAXPOOL_4 -> CONV2D_5 -> RELU -> FLATTEN -> \n", "    FULLYCONNECTED_6 -> FULLYCONNECTED_7\n", "    \n", "    Arguments:\n", "    X_train -- training set, of shape (None, 28, 28, 1)\n", "    Y_train -- test set, of shape (None, n_y = 10)\n", "    X_test -- training set, of shape (None, 28, 28, 1)\n", "    Y_test -- test set, of shape (None, n_y = 10)\n", "    test -- the test set (to be predicted), of shape (None, 28, 28, 1)\n", "    learning_rate -- learning rate of the optimization\n", "    num_epochs -- number of epochs of the optimization loop\n", "    minibatch_size -- size of a minibatch\n", "    pooling -- pooling method, avg or max\n", "    C_F1 -- filters number in conv_1\n", "    C_F3 -- filters number in conv_3\n", "    print_cost -- True to print the cost every 100 epochs\n", "    \n", "    Returns:\n", "    train_accuracy -- real number, accuracy on the train set (X_train)\n", "    test_accuracy -- real number, testing accuracy on the test set (X_test)\n", "    parameters -- parameters learnt by the model. They can then be used to predict.\n", "    \"\"\"\n", "    \n", "    ops.reset_default_graph()                         # to be able to rerun the model without overwriting tf variables\n", "    tf.set_random_seed(5)                             # to keep results consistent (tensorflow seed)\n", "    seed = 7                                          # to keep results consistent (numpy seed)\n", "    (m, n_H0, n_W0, n_C0) = X_train.shape\n", "    n_y = Y_train.shape[1]                            \n", "    costs = []                                        # To keep track of the cost\n", "    \n", "    # Create Placeholders of the correct shape\n", "    X, Y = create_placeholders(n_H0, n_W0, n_C0, n_y)\n", "\n", "    # Initialize parameters\n", "    parameters = initialize_parameters(C_F1 = C_F1, C_F3 = C_F3)\n", "    \n", "    # Forward propagation: Build the forward propagation in the tensorflow graph\n", "    fc_7 = forward_propagation(X, parameters, pooling = pooling)\n", "    # Cost function: Add cost function to tensorflow graph\n", "    cost = compute_cost(fc_7, Y)\n", "    \n", "    # Backpropagation: Define the tensorflow optimizer. Use an AdamOptimizer that minimizes the cost.\n", "    optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost)\n", "    \n", "    # Initialize all the variables globally\n", "    init = tf.global_variables_initializer()\n", "     \n", "    # Start the session to compute the tensorflow graph\n", "    with tf.Session() as sess:\n", "        \n", "        # Run the initialization\n", "        sess.run(init)\n", "        \n", "        # Do the training loop\n", "        for epoch in range(num_epochs):\n", "\n", "            minibatch_cost = 0.\n", "            num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set\n", "            seed = seed + 1\n", "            minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)\n", "\n", "            for minibatch in minibatches:\n", "\n", "                # Select a minibatch\n", "                (minibatch_X, minibatch_Y) = minibatch\n", "                # IMPORTANT: The line that runs the graph on a minibatch.\n", "                # Run the session to execute the optimizer and the cost, the feedict should contain a minibatch for (X,Y).\n", "                _ , temp_cost = sess.run([optimizer, cost], feed_dict={X:minibatch_X, Y:minibatch_Y})\n", "                #print(temp_cost)\n", "                \n", "                minibatch_cost += temp_cost / num_minibatches\n", "                \n", "\n", "            # Print the cost every epoch\n", "            if print_cost == True and epoch % 5 == 0 and epoch / 5 > 0:\n", "                print (\"Cost after epoch %i: %f\" % (epoch, minibatch_cost))\n", "                if learning_decay:\n", "                    learning_rate /= 2\n", "            if print_cost == True and epoch % 1 == 0:\n", "                costs.append(minibatch_cost)\n", "        \n", "        \n", "        # plot the cost\n", "        plt.plot(np.squeeze(costs))\n", "        plt.ylabel('cost')\n", "        plt.xlabel('iterations (per tens)')\n", "        plt.title(\"Learning rate =\" + str(learning_rate))\n", "        plt.show()\n", "\n", "        # Calculate the correct predictions\n", "        predict_op = tf.argmax(fc_7, 1)\n", "        correct_prediction = tf.equal(predict_op, tf.argmax(Y, 1))\n", "        \n", "        # Calculate accuracy on the test set\n", "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n", "        print(accuracy)\n", "        train_accuracy = accuracy.eval({X: X_train, Y: Y_train})\n", "        test_accuracy = accuracy.eval({X: X_test, Y: Y_test})\n", "        print(\"Train Accuracy:\", train_accuracy)\n", "        print(\"Test Accuracy:\", test_accuracy)\n", "        \n", "        #output the prediction for kaggle\n", "        predicted_labels = np.zeros(test.shape[0])\n", "        predicted_labels = sess.run(predict_op, \n", "                                    feed_dict={X: test})\n", "                \n", "        return train_accuracy, test_accuracy, parameters, predicted_labels"]}, {"metadata": {"_cell_guid": "ce137bf6-93e8-4d68-9ea0-53aef3d46784", "collapsed": true, "_uuid": "2f5cb07c4f21c665b647231990e68c4e74034224"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["_, _, parameters, predicted_labels = model(X_train, Y_train, X_test, Y_test, test, \n", "                                          learning_rate = 0.0005, mini_batch = 64,\n", "                                          learning_decay = True)"]}, {"metadata": {"_cell_guid": "61a78f5a-f9c2-4cc6-a235-928d6cc0ca2d", "collapsed": true, "_uuid": "f669cba3402c752c2dd6decd28b32d6ace7233e2"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["submissions = pd.DataFrame({'ImageId': np.arange(1 , 1 + test.shape[0]), 'Label': predicted_labels.astype(int)})\n", "submissions.to_csv('./tf_CNN.csv', index=False)\n", "#0.98642"]}, {"metadata": {"_cell_guid": "c24467e8-4073-44f7-9b3e-79c08c382c59", "_uuid": "dc1f229cf8042e234096b5a7cc157f9ff82e897e"}, "source": ["**4 Other tries**\n", "\n", "4.1 Average pooling"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "544955d6-35f3-4091-a64a-efff70b7063e", "collapsed": true, "_uuid": "0554a2c455e4a70fdf902bf47fdee9b15e479414"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["_, _, parameters, predicted_labels = model(X_train, Y_train, X_test, Y_test, test, learning_rate = 0.0005,\n", "                                          num_epochs = 35, minibatch_size = 64, pooling = 'avg',\n", "                                          print_cost = True, learning_decay = False)"]}, {"metadata": {"_cell_guid": "e5a62830-9530-406f-aaf4-1bf7d3cba2cd", "collapsed": true, "_uuid": "b671eacf2a72b36a241eade9e28f0467d6618fa3"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["submissions = pd.DataFrame({'ImageId': np.arange(1 , 1 + test.shape[0]), 'Label': predicted_labels.astype(int)})\n", "submissions.to_csv('./tf_CNN(avg).csv', index=False)\n", "#0.98271"]}, {"metadata": {"_cell_guid": "67e58b53-3df7-46bb-b644-75b9a91510e0", "_uuid": "6111dd1266c18725ae16482a91ecab23859e6e1b"}, "source": ["4.2 Run CNN with more filters"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "97c55af8-f9a8-496d-8bb8-2cb33b586e74", "collapsed": true, "_uuid": "ecc1c76218b3a5329cb9e1ddd5b5ddb67248b4f5"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["_, _, parameters, predicted_labels = model(X_train, Y_train, X_test, Y_test, test, learning_rate = 0.0004,\n", "                                          num_epochs = 30, minibatch_size = 64, pooling = 'max',\n", "                                          C_F1 = 16, C_F3 = 64,\n", "                                          print_cost = True, learning_decay = True)"]}, {"metadata": {"_cell_guid": "146e208e-b0f6-402f-a788-a7ec0e84b153", "collapsed": true, "_uuid": "c717e6870ffe9450a71a7cef214e726032cadf98"}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["submissions = pd.DataFrame({'ImageId': np.arange(1 , 1 + test.shape[0]), 'Label': predicted_labels.astype(int)})\n", "submissions.to_csv('./tf_CNN(more_filters).csv', index=False)\n", "#0.99028"]}, {"metadata": {"_cell_guid": "a0714aa9-1d4f-40e4-bb9e-7fa31f52b079", "_uuid": "34e9caaf2f427900768c79617a5340bcd7737557"}, "source": ["4.3 Data Augmentation\n", "\n", "Get 0.99242 with data augmentation."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "042e4394-b7c0-4afe-a709-8aeb1ea9b130", "_uuid": "2578ecd4a583c9a7df814674729ba6684fe01a06"}, "source": ["References:\n", "\n", "Deep learning from Andrew Ng: Convolution model Application\n", "\n", "Deep learning from Andrew Ng: Keras Tutorial - Happy House\n", "\n", "LeNet-5 by Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.\n", "\n", "http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf"], "cell_type": "markdown"}]}