{"cells": [{"metadata": {}, "cell_type": "markdown", "source": ["Regularization will help you reduce overfitting.\n", "    \n", "Regularization will drive your weights to lower values.\n", "\n", "L2 regularization and Dropout are two very effective regularization techniques.\n", "\n", "### Table of Contents\n", "\n", "1. Overview of the Problem set\n", "\n", "2. Load and prep the data\n", "\n", "3. Building the parts of our algorithm.\n", "    - ** 3.4 compute_cost_with_regularization() to compute the value of the cost  J_regularized instead of compute_cost()**\n", "    - ** 3.5 backward_propagation_with_regularization() to compute grads with L2 regularization **\n", "\n", "4. Merge all functions into a model\n", "    - ** 4.1 nn_model with L2 regularization**\n", "\n", "5. Visualizing and Analysis\n", "\n", "6. Lab"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["import os\n", "import random\n", "import sys\n", "import datetime\n", "## pip3 install opencv-python\n", "import cv2\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline"], "outputs": []}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["## constants\n", "TRAIN_DIR = \"./input/train/\"\n", "TEST_DIR = \"./input/test/\"\n", "TRAIN_SIZE = 25000\n", "TEST_SIZE = 12500\n", "DEV_RATIO = 0.1\n", "IMAGE_HEIGHT = IMAGE_WIDTH = 128\n", "\n", "LEARNING_RATE = 0.0001\n", "MINIBATCH_SIZE = 32\n", "INPUT_SIZE = IMAGE_HEIGHT * IMAGE_WIDTH * 3\n", "OUTPUT_SIZE = 2"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["### 1. Overview of the Problem set\n", "A fun project to differentiate dogs from cats. Dataset is from Kaggle: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition.\n", "\n", "The ./input/train/ dir contains 12500 cat images and 12500 dog images. Each filename contains \"cat\" or \"dog\" as label.\n", "\n", "The ./input/test/ dir contains 12500 images to classify\n"]}, {"metadata": {}, "cell_type": "markdown", "source": ["### 2. Load and prep the data"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["## data utility functions\n", "def split_data(two_dims_datas, split_ratio=DEV_RATIO):\n", "    left_count = int(two_dims_datas.shape[1] * split_ratio)\n", "    left_datas = two_dims_datas[:, :left_count]\n", "    right_datas = two_dims_datas[:, left_count:]\n", "    print(\"input datas shape: {}, left datas shape:{}, \\\n", "    right datas shape: {}\".format(two_dims_datas.shape, left_datas.shape, right_datas.shape))\n", "    return left_datas, right_datas"], "outputs": []}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["def load_data(dirname=TRAIN_DIR, file_count=1000, shuffle=True):\n", "    all_filenames = os.listdir(dirname)\n", "    random.shuffle(all_filenames)\n", "    filenames = all_filenames[:file_count]\n", "    \n", "    ## images\n", "    images = np.zeros((file_count, IMAGE_HEIGHT*IMAGE_WIDTH*3))\n", "    for i in range(file_count):\n", "        imgnd_origin = cv2.imread(dirname+filenames[i])\n", "        imgnd_resized = cv2.resize(imgnd_origin, (IMAGE_HEIGHT, IMAGE_WIDTH), interpolation=cv2.INTER_CUBIC)\n", "        imgnd_flatten = imgnd_resized.reshape(1,-1)\n", "        images[i] = imgnd_flatten\n", "    \n", "    ## labels from filenames\n", "    labels_list = [\"dog\" in filename for filename in filenames]\n", "    labels = np.array(labels_list, dtype='int8').reshape(file_count, 1)\n", "    \n", "    ## shuffle\n", "    if shuffle:\n", "        permutation = list(np.random.permutation(labels.shape[0]))\n", "        labels = labels[permutation, :]\n", "        images = images[permutation, :]\n", "\n", "    ## normalization\n", "    images = images/255.0\n", "    \n", "    return images.T, labels.T"], "outputs": []}, {"metadata": {}, "cell_type": "code", "execution_count": null, "source": ["images, labels = load_data(file_count=200)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["### 3. Building the parts of our algorithm.\n", "\n", "**Points**\n", "\n", "- Implement a 2-class classification neural network with a single hidden layer\n", "- Use units with a non-linear activation function, such as tanh \n", "- Compute the cross entropy loss \n", "- Implement forward and backward propagation\n", "\n", "**Here is our model**:\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/classification_kiank.png\" style=\"width:600px;height:300px;\">\n", "\n", "**Mathematically**:\n", "\n", "For one example $x^{(i)}$:\n", "$$z^{[1] (i)} =  W^{[1]} x^{(i)} + b^{[1] (i)}\\tag{1}$$ \n", "$$a^{[1] (i)} = \\tanh(z^{[1] (i)})\\tag{2}$$\n", "$$z^{[2] (i)} = W^{[2]} a^{[1] (i)} + b^{[2] (i)}\\tag{3}$$\n", "$$\\hat{y}^{(i)} = a^{[2] (i)} = \\sigma(z^{ [2] (i)})\\tag{4}$$\n", "$$y^{(i)}_{prediction} = \\begin{cases} 1 & \\mbox{if } a^{[2](i)} > 0.5 \\\\ 0 & \\mbox{otherwise } \\end{cases}\\tag{5}$$\n", "\n", "Given the predictions on all the examples, you can also compute the cost $J$ as follows: \n", "$$J = - \\frac{1}{m} \\sum\\limits_{i = 0}^{m} \\large\\left(\\small y^{(i)}\\log\\left(a^{[2] (i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[2] (i)}\\right)  \\large  \\right) \\small \\tag{6}$$\n"]}, {"metadata": {}, "cell_type": "markdown", "source": ["** 3.1 Defining the neural network structure **"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "source": ["n_x = train_images.shape[0] # size of input layer\n", "n_h = 4          # hard code the hidden layer size to be 4\n", "n_y = train_labels.shape[0] # size of output layer\n", "\n", "layer_sizes = (n_x, n_h, n_y)\n", "print(layer_sizes)"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["** 3.2 - Initialize the model's parameters **"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["def initialize_parameters(n_x, n_h, n_y):\n", "    \"\"\"\n", "    Argument:\n", "    n_x -- size of the input layer\n", "    n_h -- size of the hidden layer\n", "    n_y -- size of the output layer\n", "    \n", "    Returns:\n", "    params -- python dictionary containing your parameters:\n", "                    W1 -- weight matrix of shape (n_h, n_x)\n", "                    b1 -- bias vector of shape (n_h, 1)\n", "                    W2 -- weight matrix of shape (n_y, n_h)\n", "                    b2 -- bias vector of shape (n_y, 1)\n", "    \"\"\"\n", "    \n", "    W1 = np.random.randn(n_h, n_x) * 0.01\n", "    b1 = np.zeros((n_h, 1))\n", "    W2 = np.random.randn(n_y, n_h) * 0.01\n", "    b2 = np.zeros((n_y, 1))\n", "    \n", "    parameters = {\"W1\": W1,\n", "                  \"b1\": b1,\n", "                  \"W2\": W2,\n", "                  \"b2\": b2}\n", "    \n", "    return parameters"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["** 3.3 forward_propagation **\n", "\n", "Values needed in the backpropagation are stored in \"cache\". The cache will be given as an input to the backpropagation function."]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["## SIGMOID\n", "\n", "def sigmoid(z):\n", "    \"\"\"\n", "    Compute the sigmoid of z\n", "\n", "    Arguments:\n", "    z -- A scalar or numpy array of any size.\n", "\n", "    Return:\n", "    s -- sigmoid(z)\n", "    \"\"\"\n", "    \n", "    s = 1.0/(1.0 + np.exp(-1.0 * z))\n", "    \n", "    return s"], "outputs": []}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["# forward_propagation\n", "\n", "def forward_propagation(X, parameters):\n", "    \"\"\"\n", "    Argument:\n", "    X -- input data of size (n_x, m)\n", "    parameters -- python dictionary containing your parameters (output of initialization function)\n", "            W1 -- weight matrix of shape (n_h, n_x)\n", "            b1 -- bias vector of shape (n_h, 1)\n", "            W2 -- weight matrix of shape (n_y, n_h)\n", "            b2 -- bias vector of shape (n_y, 1)\n", "    \n", "    Returns:\n", "    A2 -- The sigmoid output of the second activation\n", "    cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\"\n", "    \"\"\"\n", "    # Retrieve each parameter from the dictionary \"parameters\"\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Implement Forward Propagation to calculate A2 (probabilities)\n", "    #print(W1.shape, X.shape)\n", "    #print(np.matmul(W1, X).shape, b1.shape)\n", "    Z1 = np.add(np.matmul(W1, X), b1)\n", "    A1 = np.tanh(Z1)\n", "    Z2 = np.add(np.matmul(W2, A1), b2)\n", "    A2 = sigmoid(Z2)\n", "    \n", "    assert(A2.shape == (1, X.shape[1]))\n", "    \n", "    cache = {\"Z1\": Z1,\n", "             \"A1\": A1,\n", "             \"Z2\": Z2,\n", "             \"A2\": A2}\n", "    \n", "    return A2, cache"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["** 3.4 compute_cost_with_regularization() to compute the value of the cost  J_regularized **"]}, {"metadata": {}, "cell_type": "markdown", "source": [" L2 Regularization\n", "\n", "The standard way to avoid overfitting is called **L2 regularization**. It consists of appropriately modifying your cost function, from:\n", "$$J = -\\frac{1}{m} \\sum\\limits_{i = 1}^{m} \\large{(}\\small  y^{(i)}\\log\\left(a^{[L](i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[L](i)}\\right) \\large{)} \\tag{1}$$\n", "To:\n", "$$J_{regularized} = \\small \\underbrace{-\\frac{1}{m} \\sum\\limits_{i = 1}^{m} \\large{(}\\small y^{(i)}\\log\\left(a^{[L](i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[L](i)}\\right) \\large{)} }_\\text{cross-entropy cost} + \\underbrace{\\frac{1}{m} \\frac{\\lambda}{2} \\sum\\limits_l\\sum\\limits_k\\sum\\limits_j W_{k,j}^{[l]2} }_\\text{L2 regularization cost} \\tag{2}$$\n"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["# compute_cost\n", "def compute_cost_with_regularization(A2, Y, parameters, lambd):\n", "    \"\"\"\n", "    Computes the cross-entropy cost given in equation (13)\n", "    \n", "    Arguments:\n", "    A2 -- The sigmoid output of the second activation, of shape (1, number of examples)\n", "    Y -- \"true\" labels vector of shape (1, number of examples)\n", "    parameters -- python dictionary containing your parameters W1, b1, W2 and b2\n", "    lambd -- regularization hyperparameter, scalar\n", "    \n", "    Returns:\n", "    cost - value of the regularized loss function (formula (2))\n", "    \"\"\"\n", "    \n", "    m = Y.shape[1] # number of example\n", "    W1 = parameters[\"W1\"]\n", "    W2 = parameters[\"W2\"]\n", "\n", "    # Compute the cross-entropy cost\n", "    logprobs = np.multiply(Y, np.log(A2)) + np.multiply((1 - Y), np.log(1 - A2))\n", "    cross_entropy_cost = (-1.0/m) * np.sum(logprobs)\n", "    \n", "    # Compute L2 regularization cost\n", "    L2_regularization_cost = (np.sum(np.square(W1)) + np.sum(np.square(W2)))*(lambd/(2*m))\n", "    \n", "    # add cross_entropy_cost and L2_regularization_cost\n", "    cost = cross_entropy_cost + L2_regularization_cost\n", "    \n", "    # makes sure cost is the dimension we expect. \n", "    cost = np.squeeze(cost)\n", "    assert(isinstance(cost, float))\n", "    \n", "    return cost"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["** 3.5 backward_propagation_with_regularization() to compute grads with L2 regularization **\n", "\n", "**Instructions**:\n", "Backpropagation is usually the hardest (most mathematical) part in deep learning. To help you, here again is the slide from the lecture on backpropagation. You'll want to use the six equations on the right of this slide, since you are building a vectorized implementation.  \n", "\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/grad_summary.png\" style=\"width:600px;height:300px;\">\n", "\n", "- Tips:\n", "    - To compute dZ1 you'll need to compute $g^{[1]'}(Z^{[1]})$. Since $g^{[1]}(.)$ is the tanh activation function, if $a = g^{[1]}(z)$ then $g^{[1]'}(z) = 1-a^2$. So you can compute \n", "    $g^{[1]'}(Z^{[1]})$ using `(1 - np.power(A1, 2))`."]}, {"metadata": {}, "cell_type": "markdown", "source": ["Of course, because you changed the cost, you have to change backward propagation as well! All the gradients have to be computed with respect to this new cost. \n", "\n", "Implement the changes needed in backward propagation to take into account regularization. \n", "\n", "The changes only concern dW1, dW2 and dW3. For each, you have to add the regularization term's gradient ($\\frac{d}{dW} ( \\frac{1}{2}\\frac{\\lambda}{m}  W^2) = \\frac{\\lambda}{m} W$)."]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["# backward_propagation\n", "\n", "def backward_propagation_with_regularization(X, Y, cache, parameters, lambd):\n", "    \"\"\"\n", "    Implement the backward propagation using the instructions above.\n", "    \n", "    Arguments:\n", "    parameters -- python dictionary containing our parameters \n", "    cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\".cache output from forward_propagation()\n", "    X -- input data of shape (2, number of examples)\n", "    Y -- \"true\" labels vector of shape (1, number of examples)\n", "    lambd -- regularization hyperparameter, scalar\n", "    \n", "    Returns:\n", "    grads -- python dictionary containing your gradients with respect to different parameters\n", "    \"\"\"\n", "    m = X.shape[1]\n", "    \n", "    # First, retrieve W1 and W2 from the dictionary \"parameters\".\n", "    W1 = parameters[\"W1\"]\n", "    W2 = parameters[\"W2\"]\n", "        \n", "    # Retrieve also A1 and A2 from dictionary \"cache\".\n", "    A1 = cache[\"A1\"]\n", "    A2 = cache[\"A2\"]\n", "    \n", "    # Backward propagation: calculate dW1, db1, dW2, db2. Add the regularization term to dW2,dW1\n", "    dZ2 = A2 - Y\n", "    dW2 = (1.0/m) * np.matmul(dZ2, np.transpose(A1)) + (lambd/m)*W2 ## add the regularization term\n", "    db2 = (1.0/m) * np.sum(dZ2, axis=1, keepdims=True)\n", "    \n", "    dZ1 = np.matmul(np.transpose(W2), dZ2) * (1 - np.power(A1, 2))\n", "    dW1 = (1.0/m) * np.matmul(dZ1, np.transpose(X)) + (lambd/m)*W1 ## add the regularization\n", "    db1 = (1.0/m) * np.sum(dZ1, axis=1, keepdims=True)\n", "    \n", "    grads = {\"dW1\": dW1,\n", "             \"db1\": db1,\n", "             \"dW2\": dW2,\n", "             \"db2\": db2}\n", "    \n", "    return grads"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["** 3.6 Implement the update rule.**\n", "\n", "Use gradient descent. We need (dW1, db1, dW2, db2) in order to update (W1, b1, W2, b2).\n", "\n", "**General gradient descent rule**: $ \\theta = \\theta - \\alpha \\frac{\\partial J }{ \\partial \\theta }$ where $\\alpha$ is the learning rate and $\\theta$ represents a parameter.\n", "\n", "**Illustration**: The gradient descent algorithm with a good learning rate (converging) and a bad learning rate (diverging). Images courtesy of Adam Harley.\n", "\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/sgd.gif\" style=\"width:400;height:400;\">\n", "<img src=\"http://p1plx6n23.bkt.clouddn.com/sgd_bad.gif\" style=\"width:400;height:400;\">\n"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["# update_parameters\n", "\n", "def update_parameters(parameters, grads, learning_rate = 1.2):\n", "    \"\"\"\n", "    Updates parameters using the gradient descent update rule given above\n", "    \n", "    Arguments:\n", "    parameters -- python dictionary containing your parameters \n", "    grads -- python dictionary containing your gradients \n", "    \n", "    Returns:\n", "    parameters -- python dictionary containing your updated parameters \n", "    \"\"\"\n", "    # Retrieve each parameter from the dictionary \"parameters\"\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Retrieve each gradient from the dictionary \"grads\"\n", "    dW1 = grads[\"dW1\"]\n", "    db1 = grads[\"db1\"]\n", "    dW2 = grads[\"dW2\"]\n", "    db2 = grads[\"db2\"]\n", "    \n", "    # Update rule for each parameter\n", "    W1 = W1 - learning_rate * dW1\n", "    b1 = b1 - learning_rate * db1\n", "    W2 = W2 - learning_rate * dW2\n", "    b2 = b2 - learning_rate * db2\n", "    \n", "    parameters = {\"W1\": W1,\n", "                  \"b1\": b1,\n", "                  \"W2\": W2,\n", "                  \"b2\": b2}\n", "    \n", "    return parameters"], "outputs": []}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": [], "outputs": []}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": [], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["### 4. Merge all functions into a model"]}, {"metadata": {}, "cell_type": "markdown", "source": ["** 4.1 nn_model with L2 regularization**: The general methodology to build a Neural Network is to:\n", "    1. Define the neural network structure ( # of input units,  # of hidden units, etc). \n", "    2. Initialize the model's parameters\n", "    3. Loop:\n", "        - Implement forward propagation\n", "        - Compute loss\n", "        - Implement backward propagation to get the gradients\n", "        - Update parameters (gradient descent)"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["# nn_model\n", "\n", "def nn_model(X, Y, n_h, num_iterations = 10000, learning_rate=1.2, print_cost=False, lambd = 0.7):\n", "    \"\"\"\n", "    Arguments:\n", "    X -- dataset of shape (n_x, number of examples)\n", "    Y -- labels of shape (n_y, number of examples)\n", "    n_h -- size of the hidden layer\n", "    num_iterations -- Number of iterations in gradient descent loop\n", "    print_cost -- if True, print the cost every 1000 iterations\n", "    \n", "    Returns:\n", "    parameters -- parameters learnt by the model. They can then be used to predict.\n", "    \"\"\"\n", "    \n", "    n_x = X.shape[0]\n", "    n_y = Y.shape[0]\n", "    \n", "    # Initialize parameters, then retrieve W1, b1, W2, b2. Inputs: \"n_x, n_h, n_y\". Outputs = \"W1, b1, W2, b2, parameters\".\n", "    parameters = initialize_parameters(n_x, n_h, n_y)\n", "    W1 = parameters[\"W1\"]\n", "    b1 = parameters[\"b1\"]\n", "    W2 = parameters[\"W2\"]\n", "    b2 = parameters[\"b2\"]\n", "    \n", "    # Loop (gradient descent)\n", "    costs = []\n", "    for i in range(0, num_iterations):\n", "         \n", "        # Forward propagation. Inputs: \"X, parameters\". Outputs: \"A2, cache\".\n", "        A2, cache = forward_propagation(X, parameters)\n", "        \n", "        # Cost function. Inputs: \"A2, Y, parameters\". Outputs: \"cost\".\n", "        cost = compute_cost_with_regularization(A2, Y, parameters, lambd)\n", "        costs.append(cost)\n", " \n", "        # Backpropagation. Inputs: \"parameters, cache, X, Y\". Outputs: \"grads\".\n", "        grads = backward_propagation_with_regularization(X, Y, cache, parameters, lambd)\n", " \n", "        # Gradient descent parameter update. Inputs: \"parameters, grads\". Outputs: \"parameters\".\n", "        parameters = update_parameters(parameters, grads, learning_rate=learning_rate)\n", "        \n", "        # cache all about model\n", "        trained_model = {\n", "            \"layer_sizes\": (n_x, n_h, n_y),\n", "            \"learning_rate\": learning_rate,\n", "            \"costs\": costs,\n", "            \"parameters\": parameters\n", "        }\n", "\n", "        # Print the cost every 1000 iterations\n", "        if print_cost and i % 1000 == 0:\n", "            print (\"Cost after iteration %i: %f\" %(i, cost))\n", "\n", "    return trained_model"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["** 4.2 Predictions **\n", "\n", "Use forward propagation to predict results\n", "\n", "Reminder: predictions = $y_{prediction} = \\mathbb 1 \\text{{activation > 0.5}} = \\begin{cases}\n", "      1 & \\text{if}\\ activation > 0.5 \\\\\n", "      0 & \\text{otherwise}\n", "    \\end{cases}$  "]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["# predict\n", "\n", "def predict(parameters, X):\n", "    \"\"\"\n", "    Using the learned parameters, predicts a class for each example in X\n", "    \n", "    Arguments:\n", "    parameters -- python dictionary containing your parameters \n", "    X -- input data of size (n_x, m)\n", "    \n", "    Returns\n", "    predictions -- vector of predictions of our model (red: 0 / blue: 1)\n", "    \"\"\"\n", "    \n", "    # Computes probabilities using forward propagation, and classifies to 0/1 using 0.5 as the threshold.\n", "    A2, cache = forward_propagation(X, parameters)\n", "    predictions = A2 > 0.5\n", "    \n", "    return predictions"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["### 5. Visualizing and Analysis"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "source": ["def plot_costs(trained_model):\n", "    # Plot learning curve (with costs)\n", "    costs = np.squeeze(trained_model[\"costs\"])\n", "    plt.plot(costs)\n", "    plt.ylabel('cost')\n", "    plt.xlabel('iterations (per hundreds)')\n", "    plt.title(\"Learning rate ={}\\n, layer_sizes={}\\n, accuracy:{}, m:{}\\n\".format(\n", "        trained_model[\"learning_rate\"], trained_model[\"layer_sizes\"],\n", "        trained_model.get(\"accuracy\"), trained_model.get(\"m\")\n", "    ))\n", "    #plt.show()\n", "    "], "outputs": []}, {"metadata": {}, "cell_type": "code", "execution_count": null, "source": ["# Example of a picture that was wrongly classified.\n", "trained_model = nn_model(train_images, train_labels, 5, num_iterations = 50)\n", "\n", "index = 11\n", "plt.imshow(dev_images[:,index].reshape((IMAGE_HEIGHT, IMAGE_HEIGHT, 3)))\n", "predictions = predict(trained_model[\"parameters\"], dev_images[:,index:index+1])\n", "print (\"y = {}, you predicted that it is a {}\".format(dev_labels[0,index],int(predictions)))"], "outputs": []}, {"metadata": {}, "cell_type": "markdown", "source": ["<font color='blue'>\n", "**What you should remember** -- the implications of L2-regularization on:\n", "- The cost computation:\n", "    - A regularization term is added to the cost\n", "- The backpropagation function:\n", "    - There are extra terms in the gradients with respect to weight matrices\n", "- Weights end up smaller (\"weight decay\"): \n", "    - Weights are pushed to smaller values."]}, {"metadata": {}, "cell_type": "markdown", "source": ["### 6. Lab"]}], "metadata": {"language_info": {"mimetype": "text/x-python", "name": "python", "version": "3.5.2", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py"}, "kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3"}}, "nbformat": 4, "nbformat_minor": 1}