{"nbformat": 4, "cells": [{"source": ["### Table of Contents\n", "\n", "1. Overview of the Problem set\n", "\n", "2. Load and prep the data\n", "\n", "3. Building the parts of our algorithm.\n", "\n", "4. Merge all functions into a model\n", "\n", "5. Visualizing and Analysis\n", "\n", "6. Lab"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["import os\n", "import random\n", "import sys\n", "import datetime\n", "## pip3 install opencv-python\n", "import cv2\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["## constants\n", "TRAIN_DIR = \"../input/train/\"\n", "TEST_DIR = \"../input/test/\"\n", "TRAIN_SIZE = 25000\n", "TEST_SIZE = 12500\n", "DEV_RATIO = 0.1\n", "IMAGE_HEIGHT = IMAGE_WIDTH = 128\n", "\n", "LEARNING_RATE = 0.0001\n", "MINIBATCH_SIZE = 32\n", "INPUT_SIZE = IMAGE_HEIGHT * IMAGE_WIDTH * 3\n", "OUTPUT_SIZE = 2"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["### 1. Overview of the Problem set\n", "A fun project to differentiate dogs from cats. Dataset is from Kaggle: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition.\n", "\n", "The ./input/train/ dir contains 12500 cat images and 12500 dog images. Each filename contains \"cat\" or \"dog\" as label.\n", "\n", "The ./input/test/ dir contains 12500 images to classify\n"], "cell_type": "markdown", "metadata": {}}, {"source": ["### 2. Load and prep the data"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["## data utility functions\n", "def split_data(two_dims_datas, split_ratio=DEV_RATIO):\n", " left_count = int(two_dims_datas.shape[1] * split_ratio)\n", " left_datas = two_dims_datas[:, :left_count]\n", " right_datas = two_dims_datas[:, left_count:]\n", " print(\"input datas shape: {}, left datas shape:{}, \\\n", " right datas shape: {}\".format(two_dims_datas.shape, left_datas.shape, right_datas.shape))\n", " return left_datas, right_datas"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["def load_data(dirname=TRAIN_DIR, file_count=1000, shuffle=True):\n", " all_filenames = os.listdir(dirname)\n", " random.shuffle(all_filenames)\n", " filenames = all_filenames[:file_count]\n", " \n", " ## images\n", " images = np.zeros((file_count, IMAGE_HEIGHT*IMAGE_WIDTH*3))\n", " for i in range(file_count):\n", " imgnd_origin = cv2.imread(dirname+filenames[i])\n", " imgnd_resized = cv2.resize(imgnd_origin, (IMAGE_HEIGHT, IMAGE_WIDTH), interpolation=cv2.INTER_CUBIC)\n", " imgnd_flatten = imgnd_resized.reshape(1,-1)\n", " images[i] = imgnd_flatten\n", " \n", " ## labels from filenames\n", " labels_list = [\"dog\" in filename for filename in filenames]\n", " labels = np.array(labels_list, dtype='int8').reshape(file_count, 1)\n", " \n", " ## shuffle\n", " if shuffle:\n", " permutation = list(np.random.permutation(labels.shape[0]))\n", " labels = labels[permutation, :]\n", " images = images[permutation, :]\n", "\n", " ## normalization\n", " images = images/255.0\n", " \n", " return images.T, labels.T"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["images, labels = load_data(file_count=200)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "cell_type": "code", "metadata": {}}, {"source": ["### 3. Building the parts of our algorithm.\n", "\n", "**Points**\n", "\n", "- Implement a 2-class classification neural network with a single hidden layer\n", "- Use units with a non-linear activation function, such as tanh \n", "- Compute the cross entropy loss \n", "- Implement forward and backward propagation\n", "\n", "**Here is our model**:\n", "\n", "\n", "**Mathematically**:\n", "\n", "For one example $x^{(i)}$:\n", "$$z^{[1] (i)} = W^{[1]} x^{(i)} + b^{[1] (i)}\\tag{1}$$ \n", "$$a^{[1] (i)} = \\tanh(z^{[1] (i)})\\tag{2}$$\n", "$$z^{[2] (i)} = W^{[2]} a^{[1] (i)} + b^{[2] (i)}\\tag{3}$$\n", "$$\\hat{y}^{(i)} = a^{[2] (i)} = \\sigma(z^{ [2] (i)})\\tag{4}$$\n", "$$y^{(i)}_{prediction} = \\begin{cases} 1 & \\mbox{if } a^{[2](i)} > 0.5 \\\\ 0 & \\mbox{otherwise } \\end{cases}\\tag{5}$$\n", "\n", "Given the predictions on all the examples, you can also compute the cost $J$ as follows: \n", "$$J = - \\frac{1}{m} \\sum\\limits_{i = 0}^{m} \\large\\left(\\small y^{(i)}\\log\\left(a^{[2] (i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[2] (i)}\\right) \\large \\right) \\small \\tag{6}$$\n"], "cell_type": "markdown", "metadata": {}}, {"source": ["** 3.1 Defining the neural network structure **"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["n_x = train_images.shape[0] # size of input layer\n", "n_h = 4 # hard code the hidden layer size to be 4\n", "n_y = train_labels.shape[0] # size of output layer\n", "\n", "layer_sizes = (n_x, n_h, n_y)\n", "print(layer_sizes)"], "cell_type": "code", "metadata": {}}, {"source": ["** 3.2 - Initialize the model's parameters **"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["def initialize_parameters(n_x, n_h, n_y):\n", " \"\"\"\n", " Argument:\n", " n_x -- size of the input layer\n", " n_h -- size of the hidden layer\n", " n_y -- size of the output layer\n", " \n", " Returns:\n", " params -- python dictionary containing your parameters:\n", " W1 -- weight matrix of shape (n_h, n_x)\n", " b1 -- bias vector of shape (n_h, 1)\n", " W2 -- weight matrix of shape (n_y, n_h)\n", " b2 -- bias vector of shape (n_y, 1)\n", " \"\"\"\n", " \n", " W1 = np.random.randn(n_h, n_x) * 0.01\n", " b1 = np.zeros((n_h, 1))\n", " W2 = np.random.randn(n_y, n_h) * 0.01\n", " b2 = np.zeros((n_y, 1))\n", " \n", " parameters = {\"W1\": W1,\n", " \"b1\": b1,\n", " \"W2\": W2,\n", " \"b2\": b2}\n", " \n", " return parameters"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 3.3 forward_propagation **\n", "\n", "Values needed in the backpropagation are stored in \"cache\". The cache will be given as an input to the backpropagation function."], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["## SIGMOID\n", "\n", "def sigmoid(z):\n", " \"\"\"\n", " Compute the sigmoid of z\n", "\n", " Arguments:\n", " z -- A scalar or numpy array of any size.\n", "\n", " Return:\n", " s -- sigmoid(z)\n", " \"\"\"\n", " \n", " s = 1.0/(1.0 + np.exp(-1.0 * z))\n", " \n", " return s"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["# forward_propagation\n", "\n", "def forward_propagation(X, parameters):\n", " \"\"\"\n", " Argument:\n", " X -- input data of size (n_x, m)\n", " parameters -- python dictionary containing your parameters (output of initialization function)\n", " W1 -- weight matrix of shape (n_h, n_x)\n", " b1 -- bias vector of shape (n_h, 1)\n", " W2 -- weight matrix of shape (n_y, n_h)\n", " b2 -- bias vector of shape (n_y, 1)\n", " \n", " Returns:\n", " A2 -- The sigmoid output of the second activation\n", " cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\"\n", " \"\"\"\n", " # Retrieve each parameter from the dictionary \"parameters\"\n", " W1 = parameters[\"W1\"]\n", " b1 = parameters[\"b1\"]\n", " W2 = parameters[\"W2\"]\n", " b2 = parameters[\"b2\"]\n", " \n", " # Implement Forward Propagation to calculate A2 (probabilities)\n", " #print(W1.shape, X.shape)\n", " #print(np.matmul(W1, X).shape, b1.shape)\n", " Z1 = np.add(np.matmul(W1, X), b1)\n", " A1 = np.tanh(Z1)\n", " Z2 = np.add(np.matmul(W2, A1), b2)\n", " A2 = sigmoid(Z2)\n", " \n", " assert(A2.shape == (1, X.shape[1]))\n", " \n", " cache = {\"Z1\": Z1,\n", " \"A1\": A1,\n", " \"Z2\": Z2,\n", " \"A2\": A2}\n", " \n", " return A2, cache"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 3.4 compute_cost() to compute the value of the cost JJ **"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# compute_cost\n", "\n", "def compute_cost(A2, Y, parameters):\n", " \"\"\"\n", " Computes the cross-entropy cost given in equation (13)\n", " \n", " Arguments:\n", " A2 -- The sigmoid output of the second activation, of shape (1, number of examples)\n", " Y -- \"true\" labels vector of shape (1, number of examples)\n", " parameters -- python dictionary containing your parameters W1, b1, W2 and b2\n", " \n", " Returns:\n", " cost -- cross-entropy cost given equation (13)\n", " \"\"\"\n", " \n", " m = Y.shape[1] # number of example\n", "\n", " # Compute the cross-entropy cost\n", " logprobs = np.multiply(Y, np.log(A2)) + np.multiply((1 - Y), np.log(1 - A2))\n", " #print(logprobs.shape)\n", " cost = (-1.0/m) * np.sum(logprobs)\n", " \n", " cost = np.squeeze(cost) # makes sure cost is the dimension we expect. \n", " # E.g., turns [[17]] into 17 \n", " assert(isinstance(cost, float))\n", " \n", " return cost"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 3.5 backward_propagation() **\n", "\n", "**Instructions**:\n", "Backpropagation is usually the hardest (most mathematical) part in deep learning. To help you, here again is the slide from the lecture on backpropagation. You'll want to use the six equations on the right of this slide, since you are building a vectorized implementation. \n", "\n", "\n", "\n", "- Tips:\n", " - To compute dZ1 you'll need to compute $g^{[1]'}(Z^{[1]})$. Since $g^{[1]}(.)$ is the tanh activation function, if $a = g^{[1]}(z)$ then $g^{[1]'}(z) = 1-a^2$. So you can compute \n", " $g^{[1]'}(Z^{[1]})$ using `(1 - np.power(A1, 2))`."], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# backward_propagation\n", "\n", "def backward_propagation(parameters, cache, X, Y):\n", " \"\"\"\n", " Implement the backward propagation using the instructions above.\n", " \n", " Arguments:\n", " parameters -- python dictionary containing our parameters \n", " cache -- a dictionary containing \"Z1\", \"A1\", \"Z2\" and \"A2\".\n", " X -- input data of shape (2, number of examples)\n", " Y -- \"true\" labels vector of shape (1, number of examples)\n", " \n", " Returns:\n", " grads -- python dictionary containing your gradients with respect to different parameters\n", " \"\"\"\n", " m = X.shape[1]\n", " \n", " # First, retrieve W1 and W2 from the dictionary \"parameters\".\n", " W1 = parameters[\"W1\"]\n", " W2 = parameters[\"W2\"]\n", " \n", " # Retrieve also A1 and A2 from dictionary \"cache\".\n", " A1 = cache[\"A1\"]\n", " A2 = cache[\"A2\"]\n", " \n", " # Backward propagation: calculate dW1, db1, dW2, db2. \n", " dZ2 = A2 - Y\n", " dW2 = (1.0/m) * np.matmul(dZ2, np.transpose(A1))\n", " db2 = (1.0/m) * np.sum(dZ2, axis=1, keepdims=True)\n", " dZ1 = np.matmul(np.transpose(W2), dZ2) * (1 - np.power(A1, 2))\n", " dW1 = (1.0/m) * np.matmul(dZ1, np.transpose(X))\n", " db1 = (1.0/m) * np.sum(dZ1, axis=1, keepdims=True)\n", " \n", " grads = {\"dW1\": dW1,\n", " \"db1\": db1,\n", " \"dW2\": dW2,\n", " \"db2\": db2}\n", " \n", " return grads"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 3.6 Implement the update rule.**\n", "\n", "Use gradient descent. We need (dW1, db1, dW2, db2) in order to update (W1, b1, W2, b2).\n", "\n", "**General gradient descent rule**: $ \\theta = \\theta - \\alpha \\frac{\\partial J }{ \\partial \\theta }$ where $\\alpha$ is the learning rate and $\\theta$ represents a parameter.\n", "\n", "**Illustration**: The gradient descent algorithm with a good learning rate (converging) and a bad learning rate (diverging). Images courtesy of Adam Harley.\n", "\n", "\n", "\n"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# update_parameters\n", "\n", "def update_parameters(parameters, grads, learning_rate = 1.2):\n", " \"\"\"\n", " Updates parameters using the gradient descent update rule given above\n", " \n", " Arguments:\n", " parameters -- python dictionary containing your parameters \n", " grads -- python dictionary containing your gradients \n", " \n", " Returns:\n", " parameters -- python dictionary containing your updated parameters \n", " \"\"\"\n", " # Retrieve each parameter from the dictionary \"parameters\"\n", " W1 = parameters[\"W1\"]\n", " b1 = parameters[\"b1\"]\n", " W2 = parameters[\"W2\"]\n", " b2 = parameters[\"b2\"]\n", " \n", " # Retrieve each gradient from the dictionary \"grads\"\n", " dW1 = grads[\"dW1\"]\n", " db1 = grads[\"db1\"]\n", " dW2 = grads[\"dW2\"]\n", " db2 = grads[\"db2\"]\n", " \n", " # Update rule for each parameter\n", " W1 = W1 - learning_rate * dW1\n", " b1 = b1 - learning_rate * db1\n", " W2 = W2 - learning_rate * dW2\n", " b2 = b2 - learning_rate * db2\n", " \n", " parameters = {\"W1\": W1,\n", " \"b1\": b1,\n", " \"W2\": W2,\n", " \"b2\": b2}\n", " \n", " return parameters"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": [], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": [], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["### 4. Merge all functions into a model"], "cell_type": "markdown", "metadata": {}}, {"source": ["** 4.1 nn_model**: The general methodology to build a Neural Network is to:\n", " 1. Define the neural network structure ( # of input units, # of hidden units, etc). \n", " 2. Initialize the model's parameters\n", " 3. Loop:\n", " - Implement forward propagation\n", " - Compute loss\n", " - Implement backward propagation to get the gradients\n", " - Update parameters (gradient descent)"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# nn_model\n", "\n", "def nn_model(X, Y, n_h, num_iterations = 10000, learning_rate=1.2, print_cost=False):\n", " \"\"\"\n", " Arguments:\n", " X -- dataset of shape (n_x, number of examples)\n", " Y -- labels of shape (n_y, number of examples)\n", " n_h -- size of the hidden layer\n", " num_iterations -- Number of iterations in gradient descent loop\n", " print_cost -- if True, print the cost every 1000 iterations\n", " \n", " Returns:\n", " parameters -- parameters learnt by the model. They can then be used to predict.\n", " \"\"\"\n", " \n", " n_x = X.shape[0]\n", " n_y = Y.shape[0]\n", " \n", " # Initialize parameters, then retrieve W1, b1, W2, b2. Inputs: \"n_x, n_h, n_y\". Outputs = \"W1, b1, W2, b2, parameters\".\n", " parameters = initialize_parameters(n_x, n_h, n_y)\n", " W1 = parameters[\"W1\"]\n", " b1 = parameters[\"b1\"]\n", " W2 = parameters[\"W2\"]\n", " b2 = parameters[\"b2\"]\n", " \n", " # Loop (gradient descent)\n", " costs = []\n", " for i in range(0, num_iterations):\n", " \n", " # Forward propagation. Inputs: \"X, parameters\". Outputs: \"A2, cache\".\n", " A2, cache = forward_propagation(X, parameters)\n", " \n", " # Cost function. Inputs: \"A2, Y, parameters\". Outputs: \"cost\".\n", " cost = compute_cost(A2, Y, parameters)\n", " costs.append(cost)\n", " \n", " # Backpropagation. Inputs: \"parameters, cache, X, Y\". Outputs: \"grads\".\n", " grads = backward_propagation(parameters, cache, X, Y)\n", " \n", " # Gradient descent parameter update. Inputs: \"parameters, grads\". Outputs: \"parameters\".\n", " parameters = update_parameters(parameters, grads, learning_rate=learning_rate)\n", " \n", " # cache all about model\n", " trained_model = {\n", " \"layer_sizes\": (n_x, n_h, n_y),\n", " \"learning_rate\": learning_rate,\n", " \"costs\": costs,\n", " \"parameters\": parameters\n", " }\n", "\n", " # Print the cost every 1000 iterations\n", " if print_cost and i % 1000 == 0:\n", " print (\"Cost after iteration %i: %f\" %(i, cost))\n", "\n", " return trained_model"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 4.2 Predictions **\n", "\n", "Use forward propagation to predict results\n", "\n", "Reminder: predictions = $y_{prediction} = \\mathbb 1 \\text{{activation > 0.5}} = \\begin{cases}\n", " 1 & \\text{if}\\ activation > 0.5 \\\\\n", " 0 & \\text{otherwise}\n", " \\end{cases}$ "], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# predict\n", "\n", "def predict(parameters, X):\n", " \"\"\"\n", " Using the learned parameters, predicts a class for each example in X\n", " \n", " Arguments:\n", " parameters -- python dictionary containing your parameters \n", " X -- input data of size (n_x, m)\n", " \n", " Returns\n", " predictions -- vector of predictions of our model (red: 0 / blue: 1)\n", " \"\"\"\n", " \n", " # Computes probabilities using forward propagation, and classifies to 0/1 using 0.5 as the threshold.\n", " A2, cache = forward_propagation(X, parameters)\n", " predictions = A2 > 0.5\n", " \n", " return predictions"], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["### 5. Visualizing and Analysis"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["def plot_costs(trained_model):\n", " # Plot learning curve (with costs)\n", " costs = np.squeeze(trained_model[\"costs\"])\n", " plt.plot(costs)\n", " plt.ylabel('cost')\n", " plt.xlabel('iterations (per hundreds)')\n", " plt.title(\"Learning rate ={}\\n, layer_sizes={}\\n, accuracy:{}, m:{}\\n\".format(\n", " trained_model[\"learning_rate\"], trained_model[\"layer_sizes\"],\n", " trained_model.get(\"accuracy\"), trained_model.get(\"m\")\n", " ))\n", " #plt.show()\n", " "], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["# Example of a picture that was wrongly classified.\n", "trained_model = nn_model(train_images, train_labels, n_h, num_iterations = 50)\n", "\n", "index = 11\n", "plt.imshow(dev_images[:,index].reshape((IMAGE_HEIGHT, IMAGE_HEIGHT, 3)))\n", "predictions = predict(trained_model[\"parameters\"], dev_images[:,index:index+1])\n", "print (\"y = {}, you predicted that it is a {}\".format(dev_labels[0,index],int(predictions)))"], "cell_type": "code", "metadata": {}}, {"source": ["### 6. Lab"], "cell_type": "markdown", "metadata": {}}, {"source": ["** 6.1 Tuning hidden layer size **\n", "\n", "Run the following code. It may take 1-2 minutes. You will observe different behaviors of the model for various hidden layer sizes."], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["print(\"num of training data:{}\".format(train_images.shape[1]))"], "cell_type": "code", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 2 minutes to run\n", "m = 180\n", "hidden_layer_sizes = [1, 2, 3, 4, 5, 20, 50]\n", "learning_rate=1.2\n", "num_iterations=50\n", "trained_models = []\n", "\n", "for n_h in hidden_layer_sizes:\n", " trained_model = nn_model(train_images, train_labels, n_h, num_iterations=num_iterations, learning_rate=learning_rate)\n", " predictions = predict(trained_model[\"parameters\"], dev_images)\n", " accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", " print (\"Accuracy for {} hidden units: {} %\".format(n_h, accuracy))\n", " ## cache trained_model\n", " trained_model[\"accuracy\"] = accuracy\n", " trained_model[\"m\"] = m\n", " trained_model[\"num_iterations\"] = num_iterations\n", " trained_models.append(trained_model)\n", " ## plot costs\n", " plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", " plot_costs(trained_model)\n"], "cell_type": "code", "metadata": {"scrolled": false}}, {"source": ["** 6.2 More Training data **\n", "\n", "set m = 9800, hidden_layer_sizes = [5, 20, 50]"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 1 minutes to run\n", "\n", "images, labels = load_data(file_count=25000)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "cell_type": "code", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 2 minutes to run\n", "m = 22500\n", "hidden_layer_sizes = [5, 20, 50]\n", "num_iterations=50\n", "learning_rate=1.2\n", "\n", "print(datetime.datetime.now())\n", "hidden_layer_sizes = [5, 20, 50]\n", "for n_h in hidden_layer_sizes:\n", " trained_model = nn_model(train_images, train_labels, 5, num_iterations = num_iterations, learning_rate=learning_rate)\n", " predictions = predict(trained_model[\"parameters\"], dev_images)\n", " accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", " print (\"Accuracy for {} hidden units: {} %\".format(n_h, accuracy))\n", " ## cache trained_model\n", " trained_model[\"accuracy\"] = accuracy\n", " trained_model[\"m\"] = m\n", " trained_model[\"num_iterations\"] = num_iterations\n", " trained_models.append(trained_model)\n", " ## plot costs\n", " plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", " plot_costs(trained_model)\n", "\n", " print(datetime.datetime.now())"], "cell_type": "code", "metadata": {"scrolled": true}}, {"source": ["** 6.3 Less learning_rate **\n", "\n", "set m = 9800, hidden_layer_sizes = 5, learning_rates = [1.2, 0.5, 0.1, 0.05, 0.01]"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 1 minutes to run\n", "\n", "images, labels = load_data(file_count=10000)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "cell_type": "code", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 2 minutes to run\n", "m = 22500\n", "hidden_layer_size = 5\n", "learning_rates = [1.2, 0.5, 0.1, 0.05, 0.01]\n", "num_iterations=50\n", "\n", "print(datetime.datetime.now())\n", "for learning_rate in learning_rates:\n", " trained_model = nn_model(train_images, train_labels, hidden_layer_size, num_iterations = num_iterations, learning_rate=learning_rate, print_cost=True)\n", " predictions = predict(trained_model[\"parameters\"], dev_images)\n", " accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", " print (\"Accuracy for {} learning_rate: {} %\".format(learning_rate, accuracy))\n", " ## cache trained_model\n", " trained_model[\"accuracy\"] = accuracy\n", " trained_model[\"m\"] = m\n", " trained_model[\"num_iterations\"] = num_iterations\n", " trained_models.append(trained_model)\n", " ## plot costs\n", " plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", " plot_costs(trained_model)\n", "\n", " print(datetime.datetime.now())"], "cell_type": "code", "metadata": {"scrolled": true}}, {"execution_count": null, "outputs": [], "source": [], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 6.4 More num_iterations **\n", "\n", "set m = 9800, hidden_layer_sizes = 5, learning_rates = 0.05, num_iterations = [50, 100, 500, 2000] "], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 1 minutes to run\n", "\n", "images, labels = load_data(file_count=10000)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 20 minutes to run\n", "m = 9800\n", "hidden_layer_sizes = 5\n", "num_iterations = [50, 100, 500, 1000]\n", "learning_rate = 0.05 \n", "\n", "print(datetime.datetime.now())\n", "for num in num_iterations:\n", " trained_model = nn_model(train_images, train_labels, hidden_layer_sizes, num_iterations = num, learning_rate=learning_rate)\n", " predictions = predict(trained_model[\"parameters\"], dev_images)\n", " accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", " print (\"Accuracy for {} iterations: {} %\".format(num, accuracy))\n", " \n", " ## cache trained_model\n", " trained_model[\"accuracy\"] = accuracy\n", " trained_model[\"m\"] = m\n", " trained_model[\"num_iterations\"] = num\n", " trained_models.append(trained_model)\n", " ## plot costs\n", " plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", " plot_costs(trained_model)\n", "\n", " print(datetime.datetime.now())"], "cell_type": "code", "metadata": {"scrolled": true}}, {"execution_count": null, "outputs": [], "source": [], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 6.5 Less learning_rate **\n", "\n", "set m = 9800, hidden_layer_sizes = 5, num_iterations=200, learning_rates = [0.05, 0.01, 0.005]"], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 1 minutes to run\n", "\n", "images, labels = load_data(file_count=10000)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 2 minutes to run\n", "m = 9800\n", "hidden_layer_sizes = 5\n", "num_iterations=200\n", "learning_rates = [0.05, 0.01, 0.005]\n", "\n", "print(datetime.datetime.now())\n", "for learning_rate in learning_rates:\n", " trained_model = nn_model(train_images, train_labels, hidden_layer_sizes, num_iterations = num_iterations, learning_rate=learning_rate)\n", " predictions = predict(trained_model[\"parameters\"], dev_images)\n", " accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", " print (\"Accuracy for {} learning_rate: {} %\".format(learning_rate, accuracy))\n", " \n", " ## cache trained_model\n", " trained_model[\"m\"] = m\n", " trained_model[\"accuracy\"] = accuracy\n", " trained_model[\"num_iterations\"] = num_iterations\n", " ## plot costs\n", " plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", " plot_costs(trained_model)\n", " \n", "\n", " print(datetime.datetime.now())"], "cell_type": "code", "metadata": {"scrolled": true}}, {"execution_count": null, "outputs": [], "source": [], "cell_type": "code", "metadata": {"collapsed": true}}, {"source": ["** 6.6 More num_iterations **\n", "\n", "set m = 9800, hidden_layer_sizes = 5, learning_rates = 0.01, num_iterations = [200, 500] "], "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 1 minutes to run\n", "\n", "images, labels = load_data(file_count=10000)\n", "dev_images, train_images = split_data(images)\n", "dev_labels, train_labels = split_data(labels)"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["# This may take about 20 minutes to run\n", "m = 22500\n", "hidden_layer_sizes = 5\n", "num_iterations = [200, 500]\n", "learning_rate = 0.01\n", "\n", "\n", "print(datetime.datetime.now())\n", "for num in num_iterations:\n", " trained_model = nn_model(train_images, train_labels, hidden_layer_sizes, num_iterations = num, learning_rate=learning_rate)\n", " predictions = predict(trained_model[\"parameters\"], dev_images)\n", " accuracy = float((np.dot(dev_labels,predictions.T) + np.dot(1-dev_labels,1-predictions.T))/float(dev_labels.size)*100)\n", " print (\"Accuracy for {} iterations: {} %\".format(num, accuracy))\n", " \n", " ## cache trained_model\n", " trained_model[\"accuracy\"] = accuracy\n", " trained_model[\"m\"] = m\n", " trained_model[\"num_iterations\"] = num\n", " trained_models.append(trained_model)\n", " ## plot costs\n", " plt.figure(num=None, figsize=(15, 6), dpi=50, facecolor='w', edgecolor='k')\n", " plot_costs(trained_model)\n", "\n", " print(datetime.datetime.now())"], "cell_type": "code", "metadata": {"scrolled": true}}, {"execution_count": null, "outputs": [], "source": ["import pandas as pd\n", "df = pd.DataFrame(trained_models)"], "cell_type": "code", "metadata": {"collapsed": true}}, {"execution_count": null, "outputs": [], "source": ["df.head(10)"], "cell_type": "code", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["len(trained_models)"], "cell_type": "code", "metadata": {}}, {"execution_count": null, "outputs": [], "source": ["plt.figure(num=None, figsize=(20, 20), dpi=200, facecolor='w', edgecolor='k')\n", "for i in range(len(trained_models)):\n", " plt.subplot(921+i)\n", " plot_costs(trained_models[i])"], "cell_type": "code", "metadata": {}}, {"execution_count": null, "outputs": [], "source": [], "cell_type": "code", "metadata": {"collapsed": true}}], "metadata": {"kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3"}, "language_info": {"codemirror_mode": {"version": 3, "name": "ipython"}, "mimetype": "text/x-python", "version": "3.6.3", "pygments_lexer": "ipython3", "file_extension": ".py", "nbconvert_exporter": "python", "name": "python"}}, "nbformat_minor": 1}