{"cells":[{"metadata":{"_uuid":"a0e35e922f975bc76978f5627a1979afeac4236e","_cell_guid":"1634cef2-bce6-439d-bcab-513c2df2baf9"},"cell_type":"markdown","source":"**In this notebook, we will implement a Neural Network using fully connected layers. Such a neural network is also known as standard neural network. Here, we will get down to the basics of Neural Network implementation inspired by Andrew Ng's Deep Learning Specialization on Coursera. **\n\n**The purpose of this notebook is only to revisit the basics and we are not looking to have a highly efficient model here. But we will try our best.\nYou can find a model that works more efficiently here: https://www.kaggle.com/sarvajna/dogs-vs-cats-keras-solution. However, this does not cover the basics.**\n\n**You will see later on that this model doesn't perform so well. This showcases the fact that Standard Neural Nets are not well suited for classifying images. Convolutional Neural Nets perform better**"},{"metadata":{"_uuid":"44a4e7ece8026c58ce3b8b49589db4ef156114b3","_cell_guid":"8e8a337b-4ac3-4ecf-9773-dbf4789d0961"},"cell_type":"markdown","source":"**Just a headsup, we will be printing the dimensions of the matrices in many places to make sure we have got it right.**"},{"metadata":{"_uuid":"577859313237b290d07696b1d28c6d88d747bc66","_cell_guid":"fc7a4f4b-2c4e-4941-a620-7e5f86cdd2b9"},"cell_type":"markdown","source":"**Perform the necessary imports**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":false,"collapsed":true},"cell_type":"code","source":"import numpy as np\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nimport scipy\nimport re\nimport os\nimport cv2\nimport math","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12cc13dd03515bae55b3c7000da09e6330f8f7c4","_cell_guid":"23a0872d-280d-46d1-bac2-1c8a07788abf"},"cell_type":"markdown","source":"**Self-explanatory globals**"},{"metadata":{"collapsed":true,"_cell_guid":"0fbee00f-6294-43f9-a204-35f05c56d273","_uuid":"703b8ea8a9bf8e4a4ea472fd025bd39feec73c0c","trusted":false},"cell_type":"code","source":"image_width = 150\nimage_height = 150\nnumber_of_color_channels = 3 #RGB\ndata = \"../input/train/\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d02a983b87bf0be7b257f5d324c2ed08190fcbb","_cell_guid":"9bbf1eeb-4f57-4ab3-8b55-ec273bc8ad6c"},"cell_type":"markdown","source":"**We will not use \"../input/test/\" in this notebook as we are not intending to submit. We will split the entire training set into train, dev(cross-validation) and test  sets later**"},{"metadata":{"collapsed":true,"_cell_guid":"ffb13809-a0ae-451a-8549-e4b95842ae9a","_uuid":"4abd968c8cc803413028e5c9cd8a53252bfc8684","trusted":false},"cell_type":"code","source":"data_set = [data+idx for idx in os.listdir(path=data)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75fe57aa8b0c8076c4db5bd4f284302320e4f981","_cell_guid":"7bb1da6f-8816-4734-a027-75069d56dd48"},"cell_type":"markdown","source":"**Helper functions to sort the image files based on the numeric value in each file name.**"},{"metadata":{"collapsed":true,"_cell_guid":"93020a3b-dae3-455e-a206-197a6f855336","_uuid":"42c5df6795f469864db10402855bd00aa3044a4c","trusted":false},"cell_type":"code","source":"def atoi(text):\n    if text.isdigit() == True: #text contains digits only\n        return int(text)\n    else:\n        return text\n    \ndef natural_keys(text):\n    return [atoi(idx) for idx in re.split(pattern='(\\d+)', string=text)]","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"f95ccddb-e13b-491a-bbfe-67603baeca63","_uuid":"30d0ac642f961bd95e32300db14a10f2aa546b96","trusted":false},"cell_type":"code","source":"data_set.sort(key=natural_keys)\n#data_set = data_set[0:5000] + data_set[12500:17500]\ndata_set = data_set[0:2500] + data_set[12500:15000]\n#data_set = data_set[0:3000] + data_set[12500:15500]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"21eafcd146773828450faf5a20ea43e862a2106b","_cell_guid":"671ba1ba-42a7-4810-b323-e48faaf40f80"},"cell_type":"markdown","source":"**Function to represent images as real numbers **"},{"metadata":{"collapsed":true,"_cell_guid":"11dc2494-f0fa-4628-b63e-c3b01d0dd911","_uuid":"d093f18654861f7343b18ddbe884d07dd4694d3d","trusted":false},"cell_type":"code","source":"def prepare_data(list_of_images):\n    \"\"\"\n    Returns two arrays: \n        x is an array of resized images\n        y is an array of labels\n    \"\"\"\n    x = np.zeros(shape=(len(list_of_images), image_width, image_height, number_of_color_channels))\n    y = np.zeros(shape=(1, len(list_of_images)))\n    \n    for (index, image) in enumerate(list_of_images):\n        #read_image = np.array(ndimage.imread(image, flatten=False)) #deprecated in SciPy 1.0.0\n        #my_image = scipy.misc.imresize(read_image, size=(150,150)) #deprecated in Scipy 1.0.0, will be removed in SciPy 1.2.0\n        read_image = np.array(cv2.imread(image))\n        my_image = cv2.resize(src=read_image, dsize=(image_width, image_height))\n        x[index] = my_image\n    \n    for (index, image) in enumerate(list_of_images):\n        if 'dog' in image:\n            y[:, index] = 1\n        elif 'cat' in image:\n            y[:, index] = 0\n            \n    return x, y","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"516709f9-7db6-48e6-9b88-51f906162837","_uuid":"edf45464a229b99348afcfacb18f61c4ae558f27","trusted":false},"cell_type":"code","source":"x_all, y_all = prepare_data(data_set)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41d89386b50cca624ac3a0ae92b52155c6b68b66","_cell_guid":"4db22108-f347-4f1b-8932-79fb9b767179"},"cell_type":"markdown","source":"**Helper function to check out the properties of numpy arrays. We will be using this to validate dimensions of matrices throughout the rest of the code.**"},{"metadata":{"collapsed":true,"_cell_guid":"01f589a6-edf0-43d8-9c82-accb102c97dc","_uuid":"a88ec021896923b1b2095c7dacd5256aa31b9f85","trusted":false},"cell_type":"code","source":"def numpy_array_properties(array):\n    print(\"type:{}, shape:{}, dimensions:{}, size:{}, datatype:{}\".format(type(array), array.shape, array.ndim, array.size, array.dtype))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f5a109675dbb9ebf5f6822c903e6aaff2e3f9e6","_cell_guid":"d678b3a2-2d36-4b9b-9aab-bf146dc8f9eb","trusted":false,"collapsed":true},"cell_type":"code","source":"numpy_array_properties(x_all)\nnumpy_array_properties(y_all)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8aa61634b377c4a40c23e5395ca9139b7f9d7c4a","_cell_guid":"19fc6383-2429-483f-ac90-3a89d86c9196"},"cell_type":"markdown","source":"A side-by-side plot of:\n\n**Image on the left**: Read and Resized Image \n\n**Image on the right**: Original Image"},{"metadata":{"_uuid":"3d9c930b56602514a34d2b7ec32b8f58e23e6052","_cell_guid":"517a09f1-3039-4a3f-8558-c5c4c5d857c0","trusted":false,"collapsed":true},"cell_type":"code","source":"fig = plt.figure()\nimg1 = fig.add_subplot(1,2,1) #1 row, 2 coulmns, image fills the 1st column\nimg1.imshow(x_all[50])\nimg2 = fig.add_subplot(1,2,2) #1 row, 2 columns, image fills the 2nd column\nimg2.imshow(np.array(cv2.imread(data_set[50])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d288de4e71dd3ed2be823a1190f882b88e0afb0","_cell_guid":"fd64f5e4-bfb5-4a2b-9537-b53eb282a117","trusted":false,"collapsed":true},"cell_type":"code","source":"fig = plt.figure()\nimg1 = fig.add_subplot(1,2,1) #1 row, 2 coulmns, image fills the 1st column\nimg1.imshow(x_all[4305]) \nimg2 = fig.add_subplot(1,2,2)\nimg2.imshow(np.array(cv2.imread(data_set[4305]))) #1 row, 2 columns, image fills the 2nd column","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e52d5e8cd0c60a104cdd7e9585fab5fe7668788f","_cell_guid":"53612321-e7dc-4ad5-a068-bf786912f074"},"cell_type":"markdown","source":"**Reshape the input x_train_dev to a vector. i.e., currently x_train_dev is of shape (number_of_examples, image_width, image_height, number_of_color_channels), **\n\n**we now want it to be of shape (image_width\\*image_height\\*number_of_color_channels, number_of_examples)**"},{"metadata":{"collapsed":true,"_cell_guid":"049b9afc-e6e6-4c12-a7e3-37365d5c6120","_uuid":"577b0602e2b855ea0c5fc76fee6fd42181be612a","trusted":false},"cell_type":"code","source":"x_all_flatten = x_all.reshape(x_all.shape[0], -1).T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc05a447030a1a7813c2b35732d47c946a12d575","_cell_guid":"0cbb10d3-4b78-4e39-94e7-2f683941aa6c"},"cell_type":"markdown","source":"**Normalize the flattened array by 255 since RGB values are between 0 and 255**"},{"metadata":{"collapsed":true,"_cell_guid":"b1562d02-0efd-43db-ad39-2c931a890218","_uuid":"9211cd8f7f138eac653a2a0f1eda5a618df0f466","trusted":false},"cell_type":"code","source":"x_all_flatten_normalized = x_all_flatten/255.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"935d9fabb02b54e21769480ef64609d7f3d159ba","_cell_guid":"3cae827d-ccb5-4be9-8e1d-38f89853ceb0","trusted":false,"collapsed":true},"cell_type":"code","source":"numpy_array_properties(x_all_flatten_normalized)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f25491a83557c2619d33a08f0faa296cf95adca7","_cell_guid":"4545908a-3151-4097-b640-90ae1704a306"},"cell_type":"markdown","source":"**Let us now shuffle the data set randomly. We also have to make sure that x(input in general sense) and y(output labels in general sense) remain in sync during the shuffling. **\n\n**np.random.permutation(..) helps us with that.**"},{"metadata":{"collapsed":true,"_cell_guid":"ad66a79e-e635-4b14-a69e-2a617f538936","_uuid":"f3b7be70e1f17a73368e221006760c685ba73657","trusted":false},"cell_type":"code","source":"def synch_shuffle_data(x, y):\n    np.random.seed(1)\n    permutation = list(np.random.permutation(x.shape[1]))\n    x_shuffled = x[:, permutation]\n    y_shuffled = y[:, permutation]\n    return x_shuffled, y_shuffled","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"6ba90e39-1324-403b-a143-18c3b338c4e8","_uuid":"3fdc198ac5ae6083ba8afc7a99bd75fd6b323882","trusted":false},"cell_type":"code","source":"x_all_flatten_normalized_shuffled, y_all_shuffled = synch_shuffle_data(x_all_flatten_normalized, y_all)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbb8b3b338adb2d127d4241ffdb80c940c945f7e","_cell_guid":"877abe21-fe55-49f8-ade4-d8857d945523","trusted":false,"collapsed":true},"cell_type":"code","source":"numpy_array_properties(x_all_flatten_normalized_shuffled)\nnumpy_array_properties(y_all_shuffled)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"588ae91d0158200abf8c7242a2be33c41f81cdca","_cell_guid":"b3fb4257-ff17-483a-99eb-bcd6e463aa98"},"cell_type":"markdown","source":"**Split the data into  60% as train data, 20% as dev_set and the rest 20% as test set. Instead, sklearn.model_selection.train_test_split can be used**"},{"metadata":{"collapsed":true,"_cell_guid":"524916e5-f9f3-4d9f-a98e-a76f828f81ae","_uuid":"8760782d06638b8b3b6280a13421cd956494fab8","trusted":false},"cell_type":"code","source":"x_train = x_all_flatten_normalized_shuffled[:, 0:3000]\nx_dev = x_all_flatten_normalized_shuffled[:, 3000:4000]\nx_test = x_all_flatten_normalized_shuffled[:, 4000:5000]\ny_train = y_all_shuffled[:, 0:3000]\ny_dev = y_all_shuffled[:, 3000:4000]\ny_test = y_all_shuffled[:, 4000:5000]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b3f06d8b89e223653a8f0695c712b9dddd7160a","_cell_guid":"4996521f-3e4d-4a5b-a724-9aa1aa5caf17","trusted":false,"collapsed":true},"cell_type":"code","source":"numpy_array_properties(x_train)\nnumpy_array_properties(y_train)\nnumpy_array_properties(x_dev)\nnumpy_array_properties(y_dev)\nnumpy_array_properties(x_test)\nnumpy_array_properties(y_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"234a80478b375f46b2c7d44ed3e8038278886ffa","_cell_guid":"88b70ee0-ab54-4469-933c-6c9ea6ba01aa","trusted":false,"collapsed":true},"cell_type":"code","source":"print(x_train)\nprint(\"*\"*100)\nprint(y_train)\nprint(\"*\"*100)\nprint(x_dev)\nprint(\"*\"*100)\nprint(y_dev)\nprint(\"*\"*100)\nprint(x_test)\nprint(\"*\"*100)\nprint(y_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6421c6611ee2cb3b7066f2f284b4d599346450bd","_cell_guid":"824523c8-cdb4-4a08-8bb3-2727874f4ffd"},"cell_type":"markdown","source":"**Before proceeding, plot the images and verify the labels(to validate shuffling has happened in synch between x and y, also for fun ;))**"},{"metadata":{"collapsed":true,"_cell_guid":"8a9f2d31-6088-413a-bf4c-2f0464e2677b","_uuid":"639dbb539985a9ad66fe1154f0bd2eeddb9a53f6","trusted":false},"cell_type":"code","source":"def recreate_image_from_numpy_array(x, y, idx):\n    x = x*255. #undo normalize\n    x = x.T.reshape(x.shape[1], 150, 150, 3) #undo flatten ex: x_dev.shape = (400, 150, 150, 3)\n    print(\"Label for the below image is {}\".format(int(np.squeeze(y[:, idx]))))\n    plt.imshow(x[idx])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4f92d35fd2c7191f617e7aacb9d1c007231599c","_cell_guid":"b25d12d7-8abe-4fb3-be1a-abfb9f919578","trusted":false,"collapsed":true},"cell_type":"code","source":"recreate_image_from_numpy_array(x_dev, y_dev, 397)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74b1b20f9d95b5fc2f311c383770b0932af639de","_cell_guid":"aee3b9e4-0f91-4ce9-85c3-a705a4560061","trusted":false,"collapsed":true},"cell_type":"code","source":"recreate_image_from_numpy_array(x_dev, y_dev, 394)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77b3a53bdce8a949558af00981e2212b6ae60ded","_cell_guid":"80a3b8bc-0214-4aa5-a1d6-9b4d5b383b01"},"cell_type":"markdown","source":"**Data preprocessing and analysis ends. Deep learning begins!**"},{"metadata":{"_uuid":"484cb5dfee6c467eab652fc0cde8f66e1ea41076","_cell_guid":"334916be-d9d7-4231-b22f-a066a551afea"},"cell_type":"markdown","source":"**Create placeholders to pass training data when you run the tensorflow session**"},{"metadata":{"collapsed":true,"_cell_guid":"648ce61e-1240-42fa-b64e-e4d9e3d4f532","_uuid":"aac5497dd0cbeee715e59222d8ebf32c2e4826d0","trusted":false},"cell_type":"code","source":"def create_placeholders(n_x, n_y, m):\n    X = tf.placeholder(name=\"X\", shape=(n_x, None), dtype=tf.float32)\n    Y = tf.placeholder(name=\"Y\", shape=(n_y, None), dtype=tf.float32)\n    #the below two lines of code will cause problems. \n    #Firstly, there is no freedom in varying number of examples(m).\n    #When using minibatches, the last mininbatch size may differ from the rest of the mininbatches\n    #Secondly, it will need training and dev sets to consist of same number of examples.\n    #X = tf.placeholder(name=\"X\", shape=(n_x, m), dtype=tf.float32)\n    #Y = tf.placeholder(name=\"Y\", shape=(n_y, m), dtype=tf.float32)\n    \n    return X, Y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33f30a6a6548e95a1bafbe4d6d33602ff1670474","_cell_guid":"f8cd5007-c2a9-4293-8b2f-06ca3ee5fafe"},"cell_type":"markdown","source":"**Initializes parameters to build a neural network with tensorflow**\n\n**Our architecture will be a follows:**\n\n**Layer 0 : input layer**\n\n**Layer 1(hidden layer): shape of W1 is (125, 67500) and shape of b1 is (125, 1)**\n\n**Layer 2(hidden layer): shape of W2 is (50, 125) and shape of b2 is (50, 1)**\n\n**Layer 3(hidden layer): shape of W3 is (50, 50) and shape of b3 is (50, 1)**\n\n**Layer 4(output layer): shape of W4 is (1, 50) and shape of b4 is (1, 1)**"},{"metadata":{"collapsed":true,"_cell_guid":"a83dc1e6-b06e-4de6-b66c-8d662b4f8537","_uuid":"7b1dd67e800d5dd43519299b791298ab0ed2f297","trusted":false},"cell_type":"code","source":"def initialize_parameters():\n    #just so that initialization remains consistent. can be removed.\n    #In fact, its better to not use seeding becaus maybe it will lead to a better performance(reach global minimum instead of local minimum).\n    tf.set_random_seed(1) \n    W1 = tf.get_variable(name=\"W1\", shape=(125, 67500), \n                         initializer=tf.contrib.layers.xavier_initializer(seed=1))\n    b1 = tf.get_variable(name=\"b1\", shape=(125, 1), \n                         initializer=tf.zeros_initializer())\n    W2 = tf.get_variable(name=\"W2\", shape=(50, 125), \n                         initializer=tf.contrib.layers.xavier_initializer(seed=1))\n    b2 = tf.get_variable(name=\"b2\", shape=(50, 1), \n                         initializer=tf.zeros_initializer())\n    W3 = tf.get_variable(name=\"W3\", shape=(50, 50), \n                         initializer=tf.contrib.layers.xavier_initializer(seed=1))\n    b3 = tf.get_variable(name=\"b3\", shape=(50, 1), \n                         initializer=tf.zeros_initializer())\n    W4 = tf.get_variable(name=\"W4\", shape=(1, 50), \n                         initializer=tf.contrib.layers.xavier_initializer(seed=1))\n    b4 = tf.get_variable(name=\"b4\", shape=(1, 1), \n                         initializer=tf.zeros_initializer())\n    parameters = {\"W1\":W1, \"b1\":b1, \"W2\":W2, \"b2\":b2, \"W3\":W3, \"b3\":b3, \"W4\":W4, \"b4\":b4}\n    return parameters","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cefd6493ac6e119dd16682c1f2b2c6933acf741e","_cell_guid":"289a7f11-f998-43f4-8200-826281c89526"},"cell_type":"markdown","source":"**Implement the forward propagation for the model: LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID**\n\n**SIGMOID not implemented here because it is calculated in the cost function**"},{"metadata":{"collapsed":true,"_cell_guid":"a66b2a7e-dd57-4fb7-8fb7-d6b12089d708","_uuid":"924d634271180b8e9fa54004e4cfade997d5ec44","trusted":false},"cell_type":"code","source":"def forward_propagation(X, parameters):\n    W1 = parameters[\"W1\"]\n    b1 = parameters[\"b1\"]\n    W2 = parameters[\"W2\"]\n    b2 = parameters[\"b2\"]\n    W3 = parameters[\"W3\"]\n    b3 = parameters[\"b3\"]\n    W4 = parameters[\"W4\"]\n    b4 = parameters[\"b4\"]\n    \n    Z1 = tf.add(tf.matmul(W1, X), b1)\n    A1 = tf.nn.relu(Z1)\n    Z2 = tf.add(tf.matmul(W2, A1), b2)\n    A2 = tf.nn.relu(Z2)\n    Z3 = tf.add(tf.matmul(W3, A2), b3)\n    A3 = tf.nn.relu(Z3)\n    Z4 = tf.add(tf.matmul(W4, A3), b4)\n    \n    return Z4","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52ddc93c2ff5f830e6a16bb889b7b9cb23b9b1b4","_cell_guid":"773b8cac-8a3b-4382-8d95-640a1d2145a5"},"cell_type":"markdown","source":"**Implement cost function with SIGMOID activation for the last layer**"},{"metadata":{"collapsed":true,"_cell_guid":"1bec8a45-3b7f-4f36-be64-07bfb9296510","_uuid":"78ab13fdfe7c61249dd917fa92f5c1f13d8c9bb1","trusted":false},"cell_type":"code","source":"def compute_cost(Z4, Y, regularize, parameters):\n    logits = tf.transpose(Z4)\n    labels = tf.transpose(Y)\n    lambd = 0.01\n    \n    if regularize == 'no':\n        cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(name=\"cost\", logits=logits, labels=labels))\n    elif regularize == 'yes':\n        cost = (tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(name=\"cost\", logits=logits, labels=labels)) +\n                lambd*tf.nn.l2_loss(parameters[\"W1\"]) + \n                lambd*tf.nn.l2_loss(parameters[\"W2\"]) + \n                lambd*tf.nn.l2_loss(parameters[\"W3\"]) + \n                lambd*tf.nn.l2_loss(parameters[\"W4\"]))\n    return cost","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"993dcda5f137b29d6f671c621b7b7e8735b4c1ec","_cell_guid":"74e95210-af0d-4322-a6d6-39bcce685722"},"cell_type":"markdown","source":"**Function to generate random minibatches of X and Y in synch**"},{"metadata":{"collapsed":true,"_cell_guid":"d975ca3b-a834-4f78-82c0-ee7490cf59bd","_uuid":"496f8c21e99516104293173dee01fc7b85c4f16f","trusted":false},"cell_type":"code","source":"def random_mini_batches(X, Y, mini_batch_size, seed):\n    \"\"\"\n    Creates a list of random minibatches from (X, Y)\n    \n    Arguments:\n    X -- input data, of shape (input size, number of examples)\n    Y -- true \"label\" vector (1 for dog / 0 for cat), of shape (1, number of examples)\n    mini_batch_size -- size of the mini-batches, integer\n    \n    Returns:\n    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)\n    \"\"\"\n    np.random.seed(seed)           \n    m = X.shape[1]                  # number of training examples\n    mini_batches = []\n        \n    # Step 1: Shuffle (X, Y)\n    permutation = list(np.random.permutation(m))\n    shuffled_X = X[:, permutation]\n    shuffled_Y = Y[:, permutation]\n    \n    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.\n    num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning\n    for k in range(0, num_complete_minibatches):\n        mini_batch_X = shuffled_X[:, mini_batch_size*k:mini_batch_size*(k+1)]\n        mini_batch_Y = shuffled_Y[:, mini_batch_size*k:mini_batch_size*(k+1)]\n        mini_batch = (mini_batch_X, mini_batch_Y)\n        mini_batches.append(mini_batch)\n    \n    # Handling the end case (last mini-batch < mini_batch_size)\n    if m % mini_batch_size != 0:\n        mini_batch_X = shuffled_X[:, mini_batch_size*num_complete_minibatches:m]\n        mini_batch_Y = shuffled_Y[:, mini_batch_size*num_complete_minibatches:m]\n        mini_batch = (mini_batch_X, mini_batch_Y)\n        mini_batches.append(mini_batch)\n    \n    return mini_batches","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0fd24280118db5903700eeb73062358e7080486a","_cell_guid":"0adaaed6-5d06-4c03-b496-f9a3e054c1b6"},"cell_type":"markdown","source":"**Build the model and run the session!**"},{"metadata":{"collapsed":true,"_cell_guid":"b3a0cc19-5bbb-4779-b278-790a9f12ad8a","_uuid":"199f0e5bbcefdf6bd0a4a4802bce135a1fe93882","trusted":false},"cell_type":"code","source":"def model(X_train, Y_train, X_dev, Y_dev, X_test, Y_test, learning_rate = 0.01, \n          minibatch_size = 64, num_epochs = 250, print_cost = True):\n    \n    (n_x, m) = X_train.shape\n    n_y = Y_train.shape[0]\n    costs = []\n    seed = 0\n    \n    X, Y = create_placeholders(n_x=n_x, n_y=n_y, m=m)\n    parameters = initialize_parameters()\n    Z4 = forward_propagation(parameters=parameters, X=X)\n    cost = compute_cost(Y=Y, Z4=Z4, regularize='no', parameters=parameters)\n        \n    #optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)\n    \n    init = tf.global_variables_initializer()\n    \n    with tf.Session() as sess:\n        sess.run(init)\n        for epoch in range(num_epochs):\n            epoch_cost = 0.                           # Defines a cost related to an epoch\n            num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set\n            seed = seed + 1\n            minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)\n            for minibatch in minibatches:\n                # Select a minibatch\n                (minibatch_X, minibatch_Y) = minibatch               \n                # Run the session to execute the \"optimizer\" and the \"cost\", the feedict should contain a minibatch for (X,Y).\n                _ , minibatch_cost = sess.run(fetches=[optimizer, cost], feed_dict={X:minibatch_X, Y:minibatch_Y})\n                epoch_cost += minibatch_cost / num_minibatches\n            costs.append(epoch_cost)\n            if print_cost == True and epoch%10 == 0:\n                print(\"Cost after epoch {} is : {}\".format(epoch, epoch_cost))\n                \n        plt.plot(np.squeeze(costs))\n        plt.ylabel(\"Cost\")\n        plt.xlabel(\"Iterations\")\n        plt.title(\"Learning rate: {}\".format(learning_rate))\n        plt.show()\n        \n        #lets save the parameters in a variable\n        parameters = sess.run(parameters)\n        print(\"Parameters have been trained!\")\n        \n        correct_prediction = tf.equal(x=tf.round(x=tf.sigmoid(Z4)), y=Y)\n        accuracy = tf.reduce_mean(tf.cast(x=correct_prediction, dtype=\"float\"))\n        \n        print(\"Train accuracy:\", accuracy.eval({X:X_train, Y:Y_train}))\n        print(\"Dev set accuracy:\", accuracy.eval({X:X_dev, Y:Y_dev}))\n        print(\"Test set accuracy:\", accuracy.eval({X:X_test, Y:Y_test}))\n        \n        return parameters","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"505ea10f64395e7962a874b3605bf278c2289756","_cell_guid":"02d7ded7-fb88-4d3b-b6af-e365e8e5fb44","trusted":false,"collapsed":true},"cell_type":"code","source":"parameters = model(X_train=x_train, Y_train=y_train, X_dev=x_dev, Y_dev=y_dev, X_test=x_test, Y_test=y_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9b6a42e9e0db7a390c208965f2ea4716aeb7816","_cell_guid":"8dc1f018-8a90-40ad-bd5d-0a74c8907c88"},"cell_type":"markdown","source":"**Bear in mind that :**\n\n**1.#hidden layers and  #hidden units in each layer didn't change. Human accuracy considered is ~100% to estimate avoidable bias and variance.**\n\n**2.m_train = #training examples, m_dev = #dev_set examples and m_test = #test set examples. Could not increase them(to reduce overfitting/high variance) across trials as kaggle kernel kept crashing. **\n\n**In brief, the following iterations were carried out**"},{"metadata":{"_uuid":"c2f5397387b5b3cacd484e8bf2888a4f5ebbff5b","_cell_guid":"5ef7f0d3-f3c6-4bb1-ab6b-7b4664fc80c8"},"cell_type":"markdown","source":"**Trial #1**\n\n1. m_train=3000\n\n2.m_dev=1000\n\n3.m_test=1000\n\n4.Regularization=no\n\n5.learning_rate=0.01\n\n6.optimizer=Gradient Descent\n\n7.minibatchsize=64\n\n8.Activation=tanh\n\n9.#epoch=200\n\n10.TrainAccuracy=92.83\n\n11.DevAccuracy=57.1\n\n12.TestAccuravy=62.7\n\n13.Comments=Avoidable bias. Try ReLU."},{"metadata":{"_uuid":"4c814032f58b174866e722fd6f9c9bd2fa1d420d","_cell_guid":"fcc408fb-6b4c-4387-a9fa-eccdea753464"},"cell_type":"markdown","source":"**Trial #2**\n\n1. m_train=3000\n\n2.m_dev=1000\n\n3.m_test=1000\n\n4.Regularization=no\n\n5.learning_rate=0.01\n\n6.optimizer=Gradient Descent\n\n7.minibatchsize=64\n\n8.Activation=ReLU\n\n9.#epoch=200\n\n10.TrainAccuracy=99.93\n\n11.DevAccuracy=60.1\n\n12.TestAccuravy=62.1\n\n13.Comments=High variance. Try regularization."},{"metadata":{"_uuid":"3022fe0f83b2f6aa8cb01b449ff080323d23bf21","_cell_guid":"1c2a132b-8962-46bb-a0a5-ff83e215f7ca"},"cell_type":"markdown","source":"**Trial #3**\n\n1. m_train=3000\n\n2.m_dev=1000\n\n3.m_test=1000\n\n4.Regularization=L2 ; regularization parameter lambd=0.01\n\n5.learning_rate=0.01\n\n6.optimizer=Gradient Descent\n\n7.minibatchsize=64\n\n8.Activation=ReLU\n\n9.#epoch=200\n\n10.TrainAccuracy=94.17\n\n11.DevAccuracy=59.2\n\n12.TestAccuravy=60.4\n\n13.Comments=No improvement. Try more training data(couldn't). Train longer(avoidable bias surfaced again)."},{"metadata":{"_uuid":"17d799d3d1daa77b13aea47f8e8575d40a4f1178","_cell_guid":"f8921f62-5df1-4cf7-a767-b89675dae187"},"cell_type":"markdown","source":"**Trial #4**\n\n1. m_train=3000\n\n2.m_dev=1000\n\n3.m_test=1000\n\n4.Regularization=L2 ; regularization parameter lambd=0.01\n\n5.learning_rate=0.01\n\n6.optimizer=Gradient Descent\n\n7.minibatchsize=64\n\n8.Activation=ReLU\n\n9.#epoch=300\n\n10.TrainAccuracy=99.47\n\n11.DevAccuracy=59\n\n12.TestAccuravy=61.5\n\n13.Comments=Avoidable bias solved. High variance, try Adam optimizer."},{"metadata":{"_uuid":"b474362ae55fbcb6c2884de2f6ec974e52300d69","_cell_guid":"857ee6bd-3eb2-4620-a529-53203875f59e"},"cell_type":"markdown","source":"**Trial #5**\n\n1. m_train=3000\n\n2.m_dev=1000\n\n3.m_test=1000\n\n4.Regularization=L2 ; regularization parameter lambd=0.01\n\n5.learning_rate=0.001\n\n6.optimizer=Adam\n\n7.minibatchsize=64\n\n8.Activation=ReLU\n\n9.#epoch=300\n\n10.TrainAccuracy=50.6\n\n11.DevAccuracy=47.8\n\n12.TestAccuravy=50.4\n\n13.Comments=Why isn't Adam working? Back to Gradient Descent."},{"metadata":{"_uuid":"5f3f85f96cc27d20f7e46b231fa31889a38eb2c8","_cell_guid":"dcc32c96-d590-4c60-926b-905c57f139cf"},"cell_type":"markdown","source":"**Trial #6**\n\n1. m_train=3000\n\n2.m_dev=1000\n\n3.m_test=1000\n\n4.Regularization=L2 with different regularization parameters(lambd) for different layers(lambd=0.01 for layer1, lambd =0.001 for layer2, lambd=0.0001 for layer3, lambd=0.00001 for layer4)\n\n5.learning_rate=0.001\n\n6.optimizer=Gradient Descent\n\n7.minibatchsize=64\n\n8.Activation=ReLU\n\n9.#epoch=200\n\n10.TrainAccuracy=93.4\n\n11.DevAccuracy=58.4\n\n12.TestAccuravy=59\n\n13.Comments=Not helping."},{"metadata":{"_uuid":"76a988037d4b800fbdf73a21e5fecea419b5772d","_cell_guid":"6b7f162b-9c4d-4cad-937b-130fbc705008"},"cell_type":"markdown","source":"**As can be seen, high variance problem is not going away. So, let's stick to the better configuration(Trial #2) for the time being **\n\n**Maybe you could  try experimenting more with these settings and see if the model performs better.  Let me know in the comments! **\n\n**Please upvote if you find this notebook useful**"},{"metadata":{"collapsed":true,"_cell_guid":"4ee8dd42-63e3-4e00-beca-78f0deca96af","_uuid":"bf38bc9a683935751e5b006200ae4683be638be3","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}