{"nbformat_minor": 1, "cells": [{"cell_type": "markdown", "metadata": {}, "source": ["### Table of Contents\n", "\n", "1. Overview of the Problem set\n", "\n", "2. Load and prep the data\n", "\n", "3. Building the parts of our algorithm.\n", "    - 3.3 Forward propagation with dropout\n", "    - 3.5 Backward propagation with dropout\n", "    \n", "4. Merge all functions into a model\n", "\n", "5. Visualizing and Analysis\n", "\n", "6. Lab"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["import os\n", "import random\n", "import sys\n", "import datetime\n", "## pip3 install opencv-python\n", "import cv2\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline"], "outputs": []}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["## constants\n", "TRAIN_DIR = \"./input/train/\"\n", "TEST_DIR = \"./input/test/\"\n", "TRAIN_SIZE = 25000\n", "TEST_SIZE = 12500\n", "DEV_RATIO = 0.1\n", "IMAGE_HEIGHT = IMAGE_WIDTH = 128\n", "\n", "LEARNING_RATE = 0.0001\n", "MINIBATCH_SIZE = 32\n", "INPUT_SIZE = IMAGE_HEIGHT * IMAGE_WIDTH * 3\n", "OUTPUT_SIZE = 2"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["### 1. Overview of the Problem set\n", "A fun project to differentiate dogs from cats. Dataset is from Kaggle: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition.\n", "\n", "The ./input/train/ dir contains 12500 cat images and 12500 dog images. Each filename contains \"cat\" or \"dog\" as label.\n", "\n", "The ./input/test/ dir contains 12500 images to classify\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### 2. Load and prep the data"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["## data utility functions\n", "def split_data(two_dims_datas, split_ratio=DEV_RATIO):\n", "    left_count = int(two_dims_datas.shape[1] * split_ratio)\n", "    left_datas = two_dims_datas[:, :left_count]\n", "    right_datas = two_dims_datas[:, left_count:]\n", "    print(\"input datas shape: {}, left datas shape:{}, \\\n", "    right datas shape: {}\".format(two_dims_datas.shape, left_datas.shape, right_datas.shape))\n", "    return left_datas, right_datas"], "outputs": []}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["def load_data(dirname=TRAIN_DIR, file_count=1000, shuffle=True):\n", "    all_filenames = os.listdir(dirname)\n", "    random.shuffle(all_filenames)\n", "    filenames = all_filenames[:file_count]\n", "    \n", "    ## images\n", "    images = np.zeros((file_count, IMAGE_HEIGHT*IMAGE_WIDTH*3))\n", "    for i in range(file_count):\n", "        imgnd_origin = cv2.imread(dirname+filenames[i])\n", "        imgnd_resized = cv2.resize(imgnd_origin, (IMAGE_HEIGHT, IMAGE_WIDTH), interpolation=cv2.INTER_CUBIC)\n", "        imgnd_flatten = imgnd_resized.reshape(1,-1)\n", "        images[i] = imgnd_flatten\n", "    \n", "    ## labels from filenames\n", "    labels_list = [\"dog\" in filename for filename in filenames]\n", "    labels = np.array(labels_list, dtype='int8').reshape(file_count, 1)\n", "    \n", "    ## shuffle\n", "    if shuffle:\n", "        permutation = list(np.random.permutation(labels.shape[0]))\n", "        labels = labels[permutation, :]\n", "        images = images[permutation, :]\n", "\n", "    ## normalization\n", "    images = images/255.0\n", "    \n", "    return images.T, labels.T"], "outputs": []}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["images, labels = load_data(file_count=200)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["### 3. Building the parts of our algorithm.\n", "\n", "**Points**\n", "\n", "- Implement a 2-class classification neural network with a single hidden layer\n", "- Use units with a non-linear activation function, such as tanh \n", "- Compute the cross entropy loss \n", "- Implement forward and backward propagation\n", "\n", "**Here is our model**:\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/classification_kiank.png\" style=\"width:600px;height:300px;\">\n", "\n", "**Mathematically**:\n", "\n", "For one example $x^{(i)}$:\n", "$$z^{[1] (i)} =  W^{[1]} x^{(i)} + b^{[1] (i)}\\tag{1}$$ \n", "$$a^{[1] (i)} = \\tanh(z^{[1] (i)})\\tag{2}$$\n", "$$z^{[2] (i)} = W^{[2]} a^{[1] (i)} + b^{[2] (i)}\\tag{3}$$\n", "$$\\hat{y}^{(i)} = a^{[2] (i)} = \\sigma(z^{ [2] (i)})\\tag{4}$$\n", "$$y^{(i)}_{prediction} = \\begin{cases} 1 & \\mbox{if } a^{[2](i)} > 0.5 \\\\ 0 & \\mbox{otherwise } \\end{cases}\\tag{5}$$\n", "\n", "Given the predictions on all the examples, you can also compute the cost $J$ as follows: \n", "$$J = - \\frac{1}{m} \\sum\\limits_{i = 0}^{m} \\large\\left(\\small y^{(i)}\\log\\left(a^{[2] (i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[2] (i)}\\right)  \\large  \\right) \\small \\tag{6}$$\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["** 3.1 Defining the neural network structure **"]}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["n_x = train_images.shape[0] # size of input layer\n", "n_h = 4          # hard code the hidden layer size to be 4\n", "n_y = train_labels.shape[0] # size of output layer\n", "\n", "layer_sizes = (n_x, n_h, n_y)\n", "print(layer_sizes)"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["** 3.2 - Initialize the model's parameters **"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["def initialize_parameters(n_x, n_h, n_y):\n", "    \"\"\"\n", "    Argument:\n", "    n_x -- size of the input layer\n", "    n_h -- size of the hidden layer\n", "    n_y -- size of the output layer\n", "    \n", "    Returns:\n", "    params -- python dictionary containing your parameters:\n", "                    W1 -- weight matrix of shape (n_h, n_x)\n", "                    b1 -- bias vector of shape (n_h, 1)\n", "                    W2 -- weight matrix of shape (n_y, n_h)\n", "                    b2 -- bias vector of shape (n_y, 1)\n", "    \"\"\"\n", "    \n", "    W1 = np.random.randn(n_h, n_x) * 0.01\n", "    b1 = np.zeros((n_h, 1))\n", "    W2 = np.random.randn(n_y, n_h) * 0.01\n", "    b2 = np.zeros((n_y, 1))\n", "    \n", "    parameters = {\"W1\": W1,\n", "                  \"b1\": b1,\n", "                  \"W2\": W2,\n", "                  \"b2\": b2}\n", "    \n", "    return parameters"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["** 3.3 Forward propagation with dropout **\n", "\n", "Dropout is a widely used regularization technique that is specific to deep learning. It randomly shuts down some neurons in each iteration. \n", "\n", "**Instructions**:\n", "You would like to shut down some neurons in the first and second layers. To do that, you are going to carry out 4 Steps:\n", "1. In lecture, we dicussed creating a variable $d^{[1]}$ with the same shape as $a^{[1]}$ using `np.random.rand()` to randomly get numbers between 0 and 1. Here, you will use a vectorized implementation, so create a random matrix $D^{[1]} = [d^{[1](1)} d^{[1](2)} ... d^{[1](m)}] $ of the same dimension as $A^{[1]}$.\n", "2. Set each entry of $D^{[1]}$ to be 0 with probability (`1-keep_prob`) or 1 with probability (`keep_prob`), by thresholding values in $D^{[1]}$ appropriately. Hint: to set all the entries of a matrix X to 0 (if entry is less than 0.5) or 1 (if entry is more than 0.5) you would do: `X = (X < 0.5)`. Note that 0 and 1 are respectively equivalent to False and True.\n", "3. Set $A^{[1]}$ to $A^{[1]} * D^{[1]}$. (You are shutting down some neurons). You can think of $D^{[1]}$ as a mask, so that when it is multiplied with another matrix, it shuts down some of the values.\n", "4. Divide $A^{[1]}$ by `keep_prob`. By doing this you are assuring that the result of the cost will still have the same expected value as without drop-out. (This technique is also called inverted dropout.)\n", "\n", "Values needed in the backpropagation are stored in \"cache\". The cache will be given as an input to the backpropagation function."]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["## SIGMOID\n", "\n", "def sigmoid(z):\n", "    \"\"\"\n", "    Compute the sigmoid of z\n", "\n", "    Arguments:\n", "    z -- A scalar or numpy array of any size.\n", "\n", "    Return:\n", "    s -- sigmoid(z)\n", "    \"\"\"\n", "    \n", "    s = 1.0/(1.0 + np.exp(-1.0 * z))\n", "    \n", "    return s"], "outputs": []}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["# forward_propagation\n", "# forward_propagation\n", "def forward_propagation(X, parameters):\n", "    \"\"\"\n", "    Argument:\n", "    X -- input data of size (n_x, m)\n", "    parameters -- python dictionary containing your parameters (output of initialization function)\n", "            W1 -- weight matrix of shape (n_h, n_x)\n", "            b1 -- bias vector of shape (n_h, 1)\n", "            W2 -- weight matrix of shape (n_y, n_h)\n", "            b2 -- bias vector of shape (n_y, 1)\n", "    \n", "    Returns:\n", "    A2 -- The sigmoid output of the second activation\n", "    cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\"\n", "    \"\"\"\n", "    # Retrieve each parameter from the dictionary \"parameters\"\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Implement Forward Propagation to calculate A2 (probabilities)\n", "    #print(W1.shape, X.shape)\n", "    #print(np.matmul(W1, X).shape, b1.shape)\n", "    Z1 = np.add(np.matmul(W1, X), b1)\n", "    A1 = np.tanh(Z1)\n", "    Z2 = np.add(np.matmul(W2, A1), b2)\n", "    A2 = sigmoid(Z2)\n", "    \n", "    assert(A2.shape == (1, X.shape[1]))\n", "    \n", "    cache = {\"Z1\": Z1,\n", "             \"A1\": A1,\n", "             \"Z2\": Z2,\n", "             \"A2\": A2}\n", "    \n", "    return A2, cache\n", "\n", "def forward_propagation_with_dropout(X, parameters, keep_prob = 0.5):\n", "    \"\"\"\n", "    Argument:\n", "    X -- input data of size (n_x, m)\n", "    parameters -- python dictionary containing your parameters (output of initialization function)\n", "            W1 -- weight matrix of shape (n_h, n_x)\n", "            b1 -- bias vector of shape (n_h, 1)\n", "            W2 -- weight matrix of shape (n_y, n_h)\n", "            b2 -- bias vector of shape (n_y, 1)\n", "    keep_prob -- probability of keeping a neuron active during drop-out, scalar\n", "    \n", "    Returns:\n", "    A2 -- The sigmoid output of the second activation\n", "    cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\"\n", "    \"\"\"\n", "    # Retrieve each parameter from the dictionary \"parameters\"\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Implement Forward Propagation to calculate A2 (probabilities)\n", "    Z1 = np.add(np.matmul(W1, X), b1)\n", "    A1 = np.tanh(Z1)\n", "    ## Dropout\n", "    D1 = np.random.rand(A1.shape[0], A1.shape[1])                               # Step 1: initialize matrix D1 = np.random.rand(..., ...)\n", "    D1 = D1 < keep_prob                                         # Step 2: convert entries of D1 to 0 or 1 (using keep_prob as the threshold)\n", "    A1 = np.multiply(A1, D1)                                         # Step 3: shut down some neurons of A1\n", "    A1 = A1/keep_prob                                         # Step 4: scale the value of neurons that haven't been shut down\n", "\n", "    Z2 = np.add(np.matmul(W2, A1), b2)\n", "    A2 = sigmoid(Z2)\n", "    \n", "    assert(A2.shape == (1, X.shape[1]))\n", "    \n", "    cache = {\"Z1\": Z1,\n", "             \"A1\": A1,\n", "             \"D1\": D1,\n", "             \"Z2\": Z2,\n", "             \"A2\": A2}\n", "    \n", "    return A2, cache"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["** 3.4 compute_cost() to compute the value of the cost  JJ **"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["# compute_cost\n", "\n", "def compute_cost(A2, Y, parameters):\n", "    \"\"\"\n", "    Computes the cross-entropy cost given in equation (13)\n", "    \n", "    Arguments:\n", "    A2 -- The sigmoid output of the second activation, of shape (1, number of examples)\n", "    Y -- \"true\" labels vector of shape (1, number of examples)\n", "    parameters -- python dictionary containing your parameters W1, b1, W2 and b2\n", "    \n", "    Returns:\n", "    cost -- cross-entropy cost given equation (13)\n", "    \"\"\"\n", "    \n", "    m = Y.shape[1] # number of example\n", "\n", "    # Compute the cross-entropy cost\n", "    logprobs = np.multiply(Y, np.log(A2)) + np.multiply((1 - Y), np.log(1 - A2))\n", "    #print(logprobs.shape)\n", "    cost = (-1.0/m) * np.sum(logprobs)\n", "    \n", "    cost = np.squeeze(cost)     # makes sure cost is the dimension we expect. \n", "                                # E.g., turns [[17]] into 17 \n", "    assert(isinstance(cost, float))\n", "    \n", "    return cost"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["** 3.5 Backward propagation with dropout **\n", "\n", "**Instruction**:\n", "Backpropagation with dropout is actually quite easy. You will have to carry out 2 Steps:\n", "1. You had previously shut down some neurons during forward propagation, by applying a mask $D^{[1]}$ to `A1`. In backpropagation, you will have to shut down the same neurons, by reapplying the same mask $D^{[1]}$ to `dA1`. \n", "2. During forward propagation, you had divided `A1` by `keep_prob`. In backpropagation, you'll therefore have to divide `dA1` by `keep_prob` again (the calculus interpretation is that if $A^{[1]}$ is scaled by `keep_prob`, then its derivative $dA^{[1]}$ is also scaled by the same `keep_prob`).\n", "\n", "\n", "**Instructions**:\n", "Backpropagation is usually the hardest (most mathematical) part in deep learning. To help you, here again is the slide from the lecture on backpropagation. You'll want to use the six equations on the right of this slide, since you are building a vectorized implementation.  \n", "\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/grad_summary.png\" style=\"width:600px;height:300px;\">\n", "\n", "- Tips:\n", "    - To compute dZ1 you'll need to compute $g^{[1]'}(Z^{[1]})$. Since $g^{[1]}(.)$ is the tanh activation function, if $a = g^{[1]}(z)$ then $g^{[1]'}(z) = 1-a^2$. So you can compute \n", "    $g^{[1]'}(Z^{[1]})$ using `(1 - np.power(A1, 2))`."]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["# backward_propagation\n", "\n", "def backward_propagation_with_dropout(parameters, cache, X, Y, keep_prob):\n", "    \"\"\"\n", "    Implement the backward propagation using the instructions above.\n", "    \n", "    Arguments:\n", "    parameters -- python dictionary containing our parameters \n", "    cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\".\n", "    X -- input data of shape (2, number of examples)\n", "    Y -- \"true\" labels vector of shape (1, number of examples)\n", "    \n", "    Returns:\n", "    grads -- python dictionary containing your gradients with respect to different parameters\n", "    \"\"\"\n", "    m = X.shape[1]\n", "    \n", "    # First, retrieve W1 and W2 from the dictionary \"parameters\".\n", "    W1 = parameters[\"W1\"]\n", "    W2 = parameters[\"W2\"]\n", "        \n", "    # Retrieve also A1 and A2 from dictionary \"cache\".\n", "    A1 = cache[\"A1\"]\n", "    A2 = cache[\"A2\"]\n", "    D1 = cache[\"D1\"]\n", "    \n", "    # Backward propagation: calculate dW1, db1, dW2, db2. \n", "    dZ2 = A2 - Y\n", "    dW2 = (1.0/m) * np.matmul(dZ2, np.transpose(A1))\n", "    db2 = (1.0/m) * np.sum(dZ2, axis=1, keepdims=True)\n", "    dA1 = np.dot(W2.T, dZ2)\n", "    ## Dropout\n", "    dA1 = np.multiply(D1, dA1)        # Step 1: Apply mask D2 to shut down the same neurons as during the forward propagation\n", "    dA1 = dA1/keep_prob              # Step 2: Scale the value of neurons that haven't been shut down\n", "    \n", "    dZ1 = np.multiply(dA1, np.int64(A1 > 0))\n", "    dW1 = (1.0/m) * np.matmul(dZ1, np.transpose(X))\n", "    db1 = (1.0/m) * np.sum(dZ1, axis=1, keepdims=True)\n", "    \n", "    grads = {\"dW1\": dW1,\n", "             \"db1\": db1,\n", "             \"dW2\": dW2,\n", "             \"db2\": db2}\n", "    \n", "    return grads"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["** 3.6 Implement the update rule.**\n", "\n", "Use gradient descent. We need (dW1, db1, dW2, db2) in order to update (W1, b1, W2, b2).\n", "\n", "**General gradient descent rule**: $ \\theta = \\theta - \\alpha \\frac{\\partial J }{ \\partial \\theta }$ where $\\alpha$ is the learning rate and $\\theta$ represents a parameter.\n", "\n", "**Illustration**: The gradient descent algorithm with a good learning rate (converging) and a bad learning rate (diverging). Images courtesy of Adam Harley.\n", "\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/sgd.gif\" style=\"width:400;height:400;\">\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/sgd_bad.gif\" style=\"width:400;height:400;\">\n"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["# update_parameters\n", "\n", "def update_parameters(parameters, grads, learning_rate = 1.2):\n", "    \"\"\"\n", "    Updates parameters using the gradient descent update rule given above\n", "    \n", "    Arguments:\n", "    parameters -- python dictionary containing your parameters \n", "    grads -- python dictionary containing your gradients \n", "    \n", "    Returns:\n", "    parameters -- python dictionary containing your updated parameters \n", "    \"\"\"\n", "    # Retrieve each parameter from the dictionary \"parameters\"\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Retrieve each gradient from the dictionary \"grads\"\n", "    dW1 = grads[\"dW1\"]\n", "    db1 = grads[\"db1\"]\n", "    dW2 = grads[\"dW2\"]\n", "    db2 = grads[\"db2\"]\n", "    \n", "    # Update rule for each parameter\n", "    W1 = W1 - learning_rate * dW1\n", "    b1 = b1 - learning_rate * db1\n", "    W2 = W2 - learning_rate * dW2\n", "    b2 = b2 - learning_rate * db2\n", "    \n", "    parameters = {\"W1\": W1,\n", "                  \"b1\": b1,\n", "                  \"W2\": W2,\n", "                  \"b2\": b2}\n", "    \n", "    return parameters"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["### 4. Merge all functions into a model"]}, {"cell_type": "markdown", "metadata": {}, "source": ["** 4.1 nn_model**: The general methodology to build a Neural Network is to:\n", "    1. Define the neural network structure ( # of input units,  # of hidden units, etc). \n", "    2. Initialize the model's parameters\n", "    3. Loop:\n", "        - Implement forward propagation\n", "        - Compute loss\n", "        - Implement backward propagation to get the gradients\n", "        - Update parameters (gradient descent)"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["# nn_model\n", "\n", "def nn_model(X, Y, n_h, num_iterations = 10000, learning_rate=1.2, keep_prob=0.5, print_cost=False):\n", "    \"\"\"\n", "    Arguments:\n", "    X -- dataset of shape (n_x, number of examples)\n", "    Y -- labels of shape (n_y, number of examples)\n", "    n_h -- size of the hidden layer\n", "    num_iterations -- Number of iterations in gradient descent loop\n", "    print_cost -- if True, print the cost every 1000 iterations\n", "    \n", "    Returns:\n", "    parameters -- parameters learnt by the model. They can then be used to predict.\n", "    \"\"\"\n", "    \n", "    n_x = X.shape[0]\n", "    n_y = Y.shape[0]\n", "    \n", "    # Initialize parameters, then retrieve W1, b1, W2, b2. Inputs: \"n_x, n_h, n_y\". Outputs = \"W1, b1, W2, b2, parameters\".\n", "    parameters = initialize_parameters(n_x, n_h, n_y)\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Loop (gradient descent)\n", "    costs = []\n", "    for i in range(0, num_iterations):\n", "         \n", "        # Forward propagation. Inputs: \"X, parameters\". Outputs: \"A2, cache\".\n", "        A2, cache = forward_propagation_with_dropout(X, parameters, keep_prob = keep_prob)\n", "        \n", "        # Cost function. Inputs: \"A2, Y, parameters\". Outputs: \"cost\".\n", "        cost = compute_cost(A2, Y, parameters)\n", "        costs.append(cost)\n", " \n", "        # Backpropagation. Inputs: \"parameters, cache, X, Y\". Outputs: \"grads\".\n", "        grads = backward_propagation_with_dropout(parameters, cache, X, Y, keep_prob)\n", " \n", "        # Gradient descent parameter update. Inputs: \"parameters, grads\". Outputs: \"parameters\".\n", "        parameters = update_parameters(parameters, grads, learning_rate=learning_rate)\n", "        \n", "        # cache all about model\n", "        trained_model = {\n", "            \"layer_sizes\": (n_x, n_h, n_y),\n", "            \"learning_rate\": learning_rate,\n", "            \"costs\": costs,\n", "            \"parameters\": parameters\n", "        }\n", "\n", "        # Print the cost every 100 iterations\n", "        if print_cost and i % 100 == 0:\n", "            print (\"Cost after iteration %i: %f\" %(i, cost))\n", "\n", "    return trained_model"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["** 4.2 Predictions **\n", "\n", "Use forward propagation to predict results\n", "\n", "Reminder: predictions = $y_{prediction} = \\mathbb 1 \\text{{activation > 0.5}} = \\begin{cases}\n", "      1 & \\text{if}\\ activation > 0.5 \\\\\n", "      0 & \\text{otherwise}\n", "    \\end{cases}$  "]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["# predict\n", "\n", "def predict(parameters, X):\n", "    \"\"\"\n", "    Using the learned parameters, predicts a class for each example in X\n", "    \n", "    Arguments:\n", "    parameters -- python dictionary containing your parameters \n", "    X -- input data of size (n_x, m)\n", "    \n", "    Returns\n", "    predictions -- vector of predictions of our model (red: 0 / blue: 1)\n", "    \"\"\"\n", "    \n", "    # Computes probabilities using forward propagation, and classifies to 0/1 using 0.5 as the threshold.\n", "    A2, cache = forward_propagation(X, parameters)\n", "    predictions = A2 > 0.5\n", "    \n", "    return predictions"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["### 5. Visualizing and Analysis"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["def plot_costs(trained_model):\n", "    # Plot learning curve (with costs)\n", "    costs = np.squeeze(trained_model[\"costs\"])\n", "    plt.plot(costs)\n", "    plt.ylabel('cost')\n", "    plt.xlabel('iterations (per hundreds)')\n", "    plt.title(\"Learning rate ={}\\n, layer_sizes={}\\n, accuracy:{}, m:{}\\n\".format(\n", "        trained_model[\"learning_rate\"], trained_model[\"layer_sizes\"],\n", "        trained_model.get(\"accuracy\"), trained_model.get(\"m\"), trained_model.get(\"keep_probs\")\n", "    ))\n", "    #plt.show()\n", "    "], "outputs": []}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["# Example of a picture that was wrongly classified.\n", "trained_model = nn_model(train_images, train_labels, n_h, num_iterations = 50)\n", "\n", "index = 11\n", "plt.imshow(dev_images[:,index].reshape((IMAGE_HEIGHT, IMAGE_HEIGHT, 3)))\n", "predictions = predict(trained_model[\"parameters\"], dev_images[:,index:index+1])\n", "print (\"y = {}, you predicted that it is a {}\".format(dev_labels[0,index],int(predictions)))"], "outputs": []}, {"cell_type": "markdown", "metadata": {}, "source": ["<font color='blue'>\n", "**What you should remember about dropout:**\n", "- Dropout is a regularization technique.\n", "- You only use dropout during training. Don't use dropout (randomly eliminate nodes) during test time.\n", "- Apply dropout both during forward and backward propagation.\n", "- During training time, divide each dropout layer by keep_prob to keep the same expected value for the activations. For example, if keep_prob is 0.5, then we will on average shut down half the nodes, so the output will be scaled by 0.5 since only the remaining half are contributing to the solution. Dividing by 0.5 is equivalent to multiplying by 2. Hence, the output now has the same expected value. You can check that this works even when keep_prob is other values than 0.5."]}, {"cell_type": "markdown", "metadata": {}, "source": ["### 6. Lab"]}, {"cell_type": "markdown", "metadata": {}, "source": ["** 6.1 More Training data **\n", "\n", "set m = 9800, hidden_layer_sizes = [5, 20, 50]"]}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["# This may take about 1 minutes to run\n", "\n", "images, labels = load_data(file_count=10000)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "outputs": []}, {"cell_type": "code", "metadata": {"scrolled": true}, "execution_count": null, "source": ["# This may take about 2 minutes to run\n", "m = 9800\n", "keep_probs = [1.0, 0.7, 0.5, 0.3]\n", "num_iterations=1001\n", "learning_rate=0.05\n", "trained_models = []\n", "\n", "print(datetime.datetime.now())\n", "for prob in keep_probs:\n", "    trained_model = nn_model(train_images, train_labels, 20,\n", "                             num_iterations = num_iterations, learning_rate=learning_rate,\n", "                             keep_prob=prob, print_cost=True\n", "                            )\n", "    predictions = predict(trained_model[\"parameters\"], dev_images)\n", "    accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", "    print (\"Accuracy for {} keep_prob : {} %\".format(prob, accuracy))\n", "    ## cache trained_model\n", "    trained_model[\"accuracy\"] = accuracy\n", "    trained_model[\"m\"] = m\n", "    trained_model[\"num_iterations\"] = num_iterations\n", "    trained_model[\"keep_probs\"] = prob\n", "    trained_models.append(trained_model)\n", "    ## plot costs\n", "    plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", "    plot_costs(trained_model)\n", "\n", "    print(datetime.datetime.now())"], "outputs": []}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": ["import pandas as pd\n", "df = pd.DataFrame(trained_models)"], "outputs": []}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["df.head(10)"], "outputs": []}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["len(trained_models)"], "outputs": []}, {"cell_type": "code", "metadata": {}, "execution_count": null, "source": ["plt.figure(num=None, figsize=(20, 20), dpi=200, facecolor='w', edgecolor='k')\n", "for i in range(len(trained_models)):\n", "    plt.subplot(921+i)\n", "    plot_costs(trained_models[i])"], "outputs": []}, {"cell_type": "code", "metadata": {"collapsed": true}, "execution_count": null, "source": [], "outputs": []}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"file_extension": ".py", "codemirror_mode": {"version": 3, "name": "ipython"}, "mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "3.5.2", "name": "python", "pygments_lexer": "ipython3"}}, "nbformat": 4}