{"cells":[{"metadata":{"_uuid":"bff92b0cf1761266783a68c2c0bdf650ffcc7305"},"cell_type":"markdown","source":"# NNNN - a Naive Neutral Network iNtroduction\n\nI've just completed Andrew Ng's Deep Learning specialization on Coursera, and I want to cement my knowledge via reimplementing main concepts using the dataset - the \"Hello World1!\" of image recognition - and the grader of this competition. \n\nI used to be a software developer in a previous life, but I haven't seen a blank code page before me for 15+ years :) Completing this notebook took 3-4 longer than I first estimated. Honestly speaking, I wanted to give up two or three times in the process when problems seemed insurmountable and bugs unsquashable.\n\nThe basic idea was to go from the simplest possible model - one layer of softmax activations - to some degree of model complexity, that would give a reasonable accuracy, and to try out different optimization techniques. I tried to create a mini framework that can be easily configured by tweaking a few parameters - it is a fantastic power of the software systems. However, it gets painful to train the model beyond three layers - apparently, numpy isn't accelerated by GPU. \n\nI've hidden most of supporting code from the published version. However, I hope it is still can be accessed if you fork the kernel (it is my first kernel and I'm only learning Kaggle). \n\nIf you see a bug, please let me know at alukashenkov@gmail.com. If you find this piece of use and interest, please upvote."},{"metadata":{"_uuid":"af4303cd916337ea5b9e1984d7c99866b7d15558"},"cell_type":"markdown","source":"## Acknowledgments\nI took much inspiration and borrowed heavily from programming assignments of the first three courses in the [Deep Learning Specialization](https://www.deeplearning.ai/courses/).\n\nI also borrowed many ideas from [Introduction to CNN Keras - 0.997 (top 6%)](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6) kernel. It also was beneficial as 101 on Kaggle submissions.\n\n[Softmax Regression](http://saitcelebi.com/tut/output/part2.html) by Sait Celebi was very helpful to make sense of softmax function derivatives."},{"metadata":{"_uuid":"1da37351e74307cc6ba60c567eef9d62762f62cd"},"cell_type":"markdown","source":"## Getting Data and Preparing Train/CV Sets\nImporting libraries."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"5546a53f990d89af64e667221e433cbf4749da91"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport seaborn as sns\nsns.set(style='white', context='notebook', palette='deep')\n\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nplt.rcParams['image.interpolation'] = 'nearest'\nplt.rcParams['image.cmap'] = 'gray'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4ee0289c3ed476e4fb4e8d2ab2895198c6e0ef5","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport math\nimport matplotlib.pyplot as plt\n\ndef sigmoid(Z):\n    \n    A = 1/(1 + np.exp(-Z))\n    \n    cache = Z\n    return A, cache\n\ndef relu(Z):\n    \n    A = np.maximum(0,Z)\n    \n    cache = Z \n    return A, cache\n\ndef softmax(Z):\n    \n    e_z = np.exp(Z - np.max(Z))\n    A = e_z / e_z.sum(axis = 0)\n    \n    cache = Z\n    \n    return A, cache\n\ndef softmax_backward(A, dA, cache):\n    \n    Z = cache\n    c, m = A.shape\n    \n    dZ = np.zeros(A.shape)\n\n    for i in range(m):\n        matrix = np.matmul(A[:, i].reshape(c,1), np.ones((1, c))) * (np.identity(c) - np.matmul(np.ones((c, 1)), A[:, i].reshape(c,1).T))    \n        dZ[:, i] = np.matmul(matrix, dA[:, i])\n    \n    assert(dZ.shape == dA.shape)\n\n    return dZ\n\ndef relu_backward(dA, cache):\n    \n    Z = cache\n    dZ = np.array(dA, copy=True) # just converting dz to a correct object.\n    \n    # When z <= 0, you should set dz to 0 as well. \n    dZ[Z <= 0] = 0\n    \n    return dZ\n\ndef sigmoid_backward(dA, cache):\n    \n    Z = cache\n    \n    s = 1/(1+np.exp(-Z))\n    dZ = dA * s * (1-s)\n    \n    return dZ\n\ndef initialize_parameters_deep(layer_dims):\n    \n    np.random.seed(3)\n    parameters = {}\n    L = len(layer_dims)            # number of layers in the network\n\n    for l in range(1, L):\n        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l-1]) * np.sqrt(2 / layer_dims[l-1])\n        parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))\n        \n    return parameters\n\ndef initialize_adam(parameters) :\n    \n    L = len(parameters) // 2 # number of layers in the neural networks\n    v = {}\n    s = {}\n    \n    # Initialize v, s. Input: \"parameters\". Outputs: \"v, s\".\n    for l in range(L):\n        v[\"dW\" + str(l+1)] = np.zeros(parameters[\"W\" + str(l+1)].shape)\n        v[\"db\" + str(l+1)] = np.zeros(parameters[\"b\" + str(l+1)].shape)\n        s[\"dW\" + str(l+1)] = np.zeros(parameters[\"W\" + str(l+1)].shape)\n        s[\"db\" + str(l+1)] = np.zeros(parameters[\"b\" + str(l+1)].shape)\n    \n    return v, s\n\ndef linear_forward(A, W, b):\n    \n    Z = np.dot(W, A) + b     # Compute linear part of the forfard propagation\n    assert(Z.shape == (W.shape[0], A.shape[1]))\n    \n    cache = (A, W, b)        # Return the product and the params\n    \n    return Z, cache\n\ndef linear_activation_forward(A_prev, W, b, activation, keep_prob = 1):\n    \n    Z, linear_cache = linear_forward(A_prev, W, b)\n\n    if activation == \"sigmoid\":\n        A, activation_cache = sigmoid(Z)   \n    elif activation == \"relu\":\n        A, activation_cache = relu(Z)\n    elif activation == \"softmax\":\n        A, activation_cache = softmax(Z)\n    \n    D = np.random.rand(np.shape(A)[0], np.shape(A)[1])   # Initialize matrix D = np.random.rand(..., ...)\n    if keep_prob != 1:\n        D = D < keep_prob                                # Convert entries of D to 0 or 1 (using keep_prob as the threshold)\n        A = A * D                                        # Shut down some neurons of A1\n        A = A / keep_prob                                # Scale the value of neurons that haven't been shut down\n        \n    cache = (linear_cache, activation_cache, D)\n\n    return A, cache\n\ndef L_model_forward(X, parameters, keep_prob = 1):\n    \n    caches = []\n    A = X\n    L = len(parameters) // 2            # number of layers in the neural network\n    \n    for l in range(1, L):\n        A_prev = A \n        A, cache = linear_activation_forward(A_prev, parameters['W'+str(l)], parameters['b'+str(l)], \"relu\", \n                                             keep_prob)\n        caches.append(cache)\n    \n    AL, cache = linear_activation_forward(A, parameters['W'+str(L)], parameters['b'+str(L)], \"softmax\")\n    caches.append(cache)\n            \n    return AL, caches\n\ndef compute_cost(AL, Y):\n    \n    m = Y.shape[1]\n\n    # Compute loss from aL and y. Hardcoding for softmax\n    cost = (-1./m) * np.sum(Y * np.log(AL))\n\n    cost = np.squeeze(cost)      # To make sure the cost's shape is what we expect (e.g. this turns [[17]] into 17).\n    assert(cost.shape == ())\n    \n    return cost\n\ndef compute_cost_with_regularization(AL, Y, parameters, lambd):\n\n    m = Y.shape[1]\n    L = len(parameters) // 2   # number of layers in the neural network\n\n    cross_entropy_cost = compute_cost(AL, Y) # This gives you the cross-entropy part of the cost\n    L2_regularization_cost = 0\n\n    # Update rule for each parameter.\n    for l in range(L):\n        W = parameters[\"W\" + str(l + 1)]\n        L2_regularization_cost = L2_regularization_cost + np.sum(np.square(W))\n    \n    cost = cross_entropy_cost + lambd / (2 * m) * L2_regularization_cost\n    \n    return cost\n\ndef linear_backward(dZ, cache, lambd):\n\n    A_prev, W, b = cache\n    m = A_prev.shape[1]\n    \n    dW = np.dot(dZ, A_prev.T) / m + lambd / m * W\n    db = np.sum(dZ, axis = 1, keepdims = True) / m\n    dA_prev = np.dot(W.T, dZ)\n\n    assert (dA_prev.shape == A_prev.shape)\n    assert (dW.shape == W.shape)\n    assert (db.shape == b.shape)\n    \n    return dA_prev, dW, db\n\ndef linear_activation_backward(A, dA, cache, lambd, activation, keep_prob = 1):\n\n    linear_cache, activation_cache, D = cache\n    \n    if activation == \"relu\":\n        dZ = relu_backward(dA, activation_cache)\n    elif activation == \"sigmoid\":\n        dZ = sigmoid_backward(dA, activation_cache)\n    elif activation == \"softmax\":\n        dZ = softmax_backward(A, dA, activation_cache)\n\n    if keep_prob != 1:\n        dA = dA * D           # Apply mask D to shut down the same neurons as during the forward propagation\n        dA = dA / keep_prob   # Scale the value of neurons that haven't been shut down\n        \n    dA_prev, dW, db = linear_backward(dZ, linear_cache, lambd)\n        \n    return dA_prev, dW, db\n\ndef L_model_backward_with_regularization(AL, Y, caches, lambd, keep_prob):\n    \n    grads = {}\n    L = len(caches)  # the number of layers\n    m = AL.shape[1]\n\n    # Initializing the backpropagation    \n    dAL = np.divide(- Y, AL)            # Hardcoding for softmax\n    assert(dAL.shape == AL.shape)\n    \n    current_cache = caches[L - 1]\n    grads[\"dA\"+str(L-1)], grads[\"dW\"+str(L)], grads[\"db\"+str(L)] = linear_activation_backward(AL, dAL, current_cache, lambd,\n                                                                                              \"softmax\")\n    \n    # Loop from l=L-2 to l=0\n    for l in reversed(range(L-1)):\n        # lth layer: (RELU -> LINEAR) gradients.\n        current_cache = caches[l]\n        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(AL, grads[\"dA\"+str(l+1)], current_cache, lambd,\n                                                                    \"relu\", keep_prob)\n        grads[\"dA\" + str(l)] = dA_prev_temp\n        grads[\"dW\" + str(l + 1)] = dW_temp\n        grads[\"db\" + str(l + 1)] = db_temp\n\n    return grads\n\ndef update_parameters_with_adam(parameters, grads, v, s, t, learning_rate,\n                                beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8):\n    \n    L = len(parameters) // 2                 # number of layers in the neural networks\n    v_corrected = {}                         # Initializing first moment estimate, python dictionary\n    s_corrected = {}                         # Initializing second moment estimate, python dictionary\n    \n    # Perform Adam update on all parameters\n    for l in range(L):\n        # Moving average of the gradients. Inputs: \"v, grads, beta1\". Output: \"v\".\n        v[\"dW\" + str(l+1)] = beta1 * v[\"dW\" + str(l+1)] + (1 - beta1) * grads[\"dW\" + str(l+1)]\n        v[\"db\" + str(l+1)] = beta1 * v[\"db\" + str(l+1)] + (1 - beta1) * grads[\"db\" + str(l+1)]\n\n        # Compute bias-corrected first moment estimate. Inputs: \"v, beta1, t\". Output: \"v_corrected\".\n        v_corrected[\"dW\" + str(l+1)] = v[\"dW\" + str(l+1)] / (1 - math.pow(beta1, t))\n        v_corrected[\"db\" + str(l+1)] = v[\"db\" + str(l+1)] / (1 - math.pow(beta1, t))\n\n        # Moving average of the squared gradients. Inputs: \"s, grads, beta2\". Output: \"s\".\n        s[\"dW\" + str(l+1)] = beta2 * s[\"dW\" + str(l+1)] + (1 - beta2) * grads[\"dW\" + str(l+1)] * grads[\"dW\" + str(l+1)]\n        s[\"db\" + str(l+1)] = beta2 * s[\"db\" + str(l+1)] + (1 - beta2) * grads[\"db\" + str(l+1)] * grads[\"db\" + str(l+1)]\n\n        # Compute bias-corrected second raw moment estimate. Inputs: \"s, beta2, t\". Output: \"s_corrected\".\n        s_corrected[\"dW\" + str(l+1)] = s[\"dW\" + str(l+1)] / (1 - math.pow(beta2, t))\n        s_corrected[\"db\" + str(l+1)] = s[\"db\" + str(l+1)] / (1 - math.pow(beta2, t))\n        \n        # Update parameters. Inputs: \"parameters, learning_rate, v_corrected, s_corrected, epsilon\". Output: \"parameters\".\n        parameters[\"W\" + str(l+1)] = parameters[\"W\" + str(l+1)] - learning_rate * v_corrected[\"dW\" + str(l+1)] / (epsilon + np.sqrt(s_corrected[\"dW\" + str(l+1)]))\n        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - learning_rate * v_corrected[\"db\" + str(l+1)] / (epsilon + np.sqrt(s_corrected[\"db\" + str(l+1)]))\n\n    return parameters, v, s\n\ndef predict(X, y, parameters):\n    \n    m = X.shape[1]\n    n = len(parameters) # number of layers in the neural network\n    p = np.zeros((1,m))\n    \n    # Forward propagation\n    probas, caches = L_model_forward(X, parameters)\n\n    # Convert probabilities into labels\n    p = probas.argmax(axis = 0).reshape(1,-1)\n    \n    assert(p.shape == y.shape)\n\n    print(\"Accuracy: \" + str(np.sum((p == y)/m)))\n        \n    return p\n\ndef print_10_mislabeled_images(X, y, p):\n    \n    img_size = 28 \n    num_images_to_show = 10\n    \n    a = p - y\n    mislabeled_indices = np.asarray(np.where(a != 0))\n\n    for i in range(num_images_to_show):\n        sample = np.random.randint(0, mislabeled_indices.shape[1])\n        index = mislabeled_indices[1][sample]\n        \n        plt.subplot(2, num_images_to_show, i + 1)\n        plt.imshow(X[:, index].reshape(img_size, img_size), interpolation = 'nearest')\n        plt.rcParams['figure.figsize'] = (50.0, 50.0) # set default size of plots\n        plt.axis('off')\n        plt.title(\"Prediction: \" + str(p[0, index]) + \" \\n Label: \" + str(y[0, index]))\n\ndef print_10_images(X, y):\n    \n    img_size = 28 \n    num_images_to_show = 10\n    m = X.shape[1]\n    \n    for i in range(num_images_to_show):\n        sample = np.random.randint(0, m - 1)\n        \n        plt.subplot(2, num_images_to_show, i + 1)\n        plt.imshow(X[:, sample].reshape(img_size, img_size), interpolation = 'nearest')\n        plt.rcParams['figure.figsize'] = (50.0, 50.0) # set default size of plots\n        plt.axis('off')\n        plt.title(\"Index: \" + str(sample) + \" \\n Label: \" + str(y[0, sample]))\n\ndef random_mini_batches(X, Y, mini_batch_size = 64):\n    \n    m = X.shape[1]                  # number of training examples\n    mini_batches = []\n        \n    # Shuffle (X, Y)\n    permutation = list(np.random.permutation(m))\n    shuffled_X = X[:, permutation]\n    shuffled_Y = Y[:, permutation].reshape((-1,m))\n\n    # Partition (shuffled_X, shuffled_Y). Minus the end case.\n    \n    # number of mini batches of size mini_batch_size in your partitionning\n    num_complete_minibatches = math.floor(m/mini_batch_size) \n    \n    for k in range(0, num_complete_minibatches):\n        mini_batch_X = shuffled_X[:, k * mini_batch_size : (k + 1) * mini_batch_size]\n        mini_batch_Y = shuffled_Y[:, k * mini_batch_size : (k + 1) * mini_batch_size]\n\n        mini_batch = (mini_batch_X, mini_batch_Y)\n        mini_batches.append(mini_batch)\n    \n    # Handling the end case (last mini-batch < mini_batch_size)\n    if m % mini_batch_size != 0:\n        mini_batch_X = shuffled_X[:, num_complete_minibatches * mini_batch_size : m]\n        mini_batch_Y = shuffled_Y[:, num_complete_minibatches * mini_batch_size : m]\n\n        mini_batch = (mini_batch_X, mini_batch_Y)\n        mini_batches.append(mini_batch)\n    \n    return mini_batches","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68037264bfe209db62bc1c8c72a0e5d1f405cfd2"},"cell_type":"markdown","source":"Definig some useful constants."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"1d50bfde13b9e4a68520fc31c7fa22eb08ecdf76"},"cell_type":"code","source":"img_size = 28               # Known from dataset description\nn_classes = 10              # We have 10 digits represented in dataset","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13704c1d62a35bed7086554adcfc0085e9694435"},"cell_type":"markdown","source":"Load train data."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"13c35d4741797073fcc3d88e24b5a26af3fd0b4c"},"cell_type":"code","source":"# Getting data as Numpy object and transpose it to have training examples as columns of the matrix\ntrain_data = pd.read_csv('../input/train.csv').values.T\n\n# Train data shape\nm = train_data.shape[1]\n\n# Labels\nY_train_data = train_data[0, :]\nY_train_data = Y_train_data.reshape(1, m)\n\n#Pixels\nX_train_data = train_data[1:, :]\nX_train_data = X_train_data/255.0     #Normalising values\n\n# Little clean up\ndel train_data","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c46b92082e8c74bad720e14583d41ce2c0a7433"},"cell_type":"markdown","source":"Let's look at what we've got to work with."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"eb208fee5a0ec90d17009c96bb28bea3765e79e8"},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (50.0, 50.0) # set default size of plots\n    \nprint(\"Number of samples in training data is:\", m)\n\nprint(\"Ten random images with their labels:\")\nprint_10_images(X_train_data, Y_train_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f5863c38fb08300c80d5d3e2d996986c75086cf1"},"cell_type":"markdown","source":"Les's properly split train data into training ant cross-validation sets."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"3251e0258e2240605d4c7e74047d0d06c018554f"},"cell_type":"code","source":"train_proportion = 0.05      # Proportion of CV set to be extracted from training data\n\n# Initializing arrays\ntrain_inds = np.zeros((1, m), dtype=bool)\n\n#Getting unique values of the labels\nvalues = np.unique(Y_train_data)\n\n# Creating indices for trainig and CV samples\nfor value in values:\n    value_inds = np.nonzero(Y_train_data == value)[1]\n    np.random.shuffle(value_inds)\n    n = int(train_proportion * len(value_inds))\n\n    train_inds[:, value_inds[n:]] = True\n\n# Now split provided dataset\nX_train = X_train_data[:, train_inds[0,:]]\nX_cv = X_train_data[:, ~train_inds[0,:]]\nY_train = Y_train_data[:, train_inds[0,:]]\nY_cv = Y_train_data[:, ~train_inds[0,:]]\n\n# And create one-hot representations for labels\nY_train_oh = np.eye(n_classes)[Y_train.reshape(-1)].T\nY_cv_oh = np.eye(n_classes)[Y_cv.reshape(-1)].T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"483b7fab09bf1c9f67efba7db421cc2f6fb3ada6"},"cell_type":"markdown","source":"Now visually checking training and CV sets labels distribution. They are expected to be fairly equal."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"90fbf7cfca27fe2b8390bd29c2b644263251a7d8"},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots\nsns.countplot(Y_train[0])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"d37fbf2d9245abc2885b4b722b7871ed70de534a"},"cell_type":"code","source":"sns.countplot(Y_cv[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff52e0dcddb26ba6d1a04b72ffd78613967a42bd"},"cell_type":"markdown","source":"## The Model\nHere comes the core `L_layer_model()` function that iterates through epoches and minibatches."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"a1d1cfbe5b8da9f9bccba9828f0d4fff373bd956"},"cell_type":"code","source":"def L_layer_model(X, Y, layers_dims, learning_rate, lr_decay_base, batch_size, num_epoch, lambd = 0, keep_prob = 1, print_cost = False):\n    \n    np.random.seed(1)\n    costs = []                      # keep track of cost\n    t = 0                           # initializing the counter required for Adam update\n    lr = learning_rate\n    batch_num = 0\n    \n    # Parameters initialization.\n    parameters = initialize_parameters_deep(layers_dims)\n    v, s = initialize_adam(parameters)\n    \n    # Loop (gradient descent)\n    for i in range(0, num_epoch):\n        \n        minibatches = random_mini_batches(X, Y, batch_size)\n\n        for minibatch in minibatches:\n\n            # Forward propagation: [LINEAR -> RELU]*(L-1) -> LINEAR -> SOFTMAX.\n            AL, caches = L_model_forward(X, parameters, keep_prob)\n        \n            # Compute cost. As the last activation is SOFTMAX use appropriate formula\n            cost = compute_cost_with_regularization(AL, Y, parameters, lambd)\n            costs.append(cost)\n    \n            # Backward propagation.\n            grads = L_model_backward_with_regularization(AL, Y, caches, lambd, keep_prob)\n            \n            # Update parameters.\n            t = t + 1 # Adam counter\n            parameters, v, s = update_parameters_with_adam(parameters, grads, v, s, t, lr)\n            \n            if print_cost:\n                print (\"Cost in epoch %i after batch %i: %f\" %(i, batch_num, cost))\n                batch_num = batch_num + 1\n        \n        # Update learning rate\n        lr = lr * (1 / (1 + lr_decay_base / num_epoch))\n        \n        # Reset batch count\n        batch_num = 0\n\n    return parameters, costs","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8f546a666be15ef3daaa0637d97a878c71b84a8"},"cell_type":"markdown","source":"Specifying model parameters."},{"metadata":{"trusted":true,"_uuid":"47b5b1abb4fb23a42764190f8cf6e6cb4c67853c"},"cell_type":"code","source":"# Define a model\nn_x = img_size * img_size          # Square image with one channel \nn_y = n_classes                    # Ten output classes\n\n# Specify model configuration - add number of hidden layers' activations beween n_x and n_y\n# The model would use relu activation for hidden layers and softmax for the output one\nlayers_dims = [n_x, 1024, 512, 256, n_y]\n    \n# Define learning parameters\nlearning_rate = 0.003               \nlr_decay_base = 0          \nbatch_size = m          \nnum_epoch = 400\nlambd = 0.7                # Weight decay\nkeep_prob = 0.8            # Dropout ratio","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25988e49a0be6d60ee24785cd9ecc18b69806bbe"},"cell_type":"markdown","source":"Running the model."},{"metadata":{"trusted":true,"_uuid":"6d7641972e16c95f5d219b1982b517a02aefa7a0","_kg_hide-output":true,"scrolled":false},"cell_type":"code","source":"%time parameters, costs = L_layer_model(X_train, Y_train_oh, layers_dims, learning_rate, lr_decay_base, batch_size, num_epoch, lambd, keep_prob, True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5390b7ff113510316c2d6cb5b16ee386965a22d"},"cell_type":"markdown","source":"Let's look at the cost graph."},{"metadata":{"trusted":true,"_uuid":"e16039cd62cb89a313ff6e958189c7ed6c85b59e"},"cell_type":"code","source":"# plot the cost\nplt.plot(np.squeeze(costs))\nplt.ylabel('cost')\nplt.xlabel('iterations (per tens)')\nplt.title(\"Learning rate = \" + str(learning_rate))\nplt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"506405d5cb042088a1dd7d2a940639fa503d5232"},"cell_type":"markdown","source":"Predict on the training set."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"48eaf1f8be7de2775bad97cd1e70b3d9f22009aa"},"cell_type":"code","source":"pred_train = predict(X_train, Y_train, parameters)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2364ed6f7e138f895ad5aaf980e7d7860e49e775"},"cell_type":"markdown","source":"Sanity check - what images the model gets wrong? In my case, most of the images I see show weird digits representations."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"14fc198a0186951962fb7a82905d8a9e9db3502f"},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (50.0, 50.0) # set default size of plots\n\nprint_10_mislabeled_images(X_train, Y_train, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50d66cc597f4e098a4f9470162b76608cfbb29ac"},"cell_type":"markdown","source":"Predict on the cross-validation set."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"933a3a7fe638d2f116a696312ad65a236980ba2e"},"cell_type":"code","source":"pred_cv = predict(X_cv, Y_cv, parameters)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"214aca4bf97577ce8e6a438584fac3dd04353bec"},"cell_type":"code","source":"print_10_mislabeled_images(X_cv, Y_cv, pred_cv)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30796f1ba7a0aebeddae99e1d8d3f795b3f396a1"},"cell_type":"markdown","source":"## Results Overview\nMy quick and rather impatient testing shows the following results:\n\n* A model with one output layer (inputs pixels are directly mapped to softmax activations) would give accuracy at around 90%. \n* Adding one hidden layer (1024 neurons) with dropout regularisation moves accuracy up to 95%.\n* With two hidden layers (1024 and 512 neurons) trained for 5 epochs the model gives 97,5% accuracy. Further training only makes the model to overfit training data - I've seen accuracy around 99.93% on the training set, but still only 97,5% on the CV set.\n* The third hidden layer (1024, 512, and 256 neurons) improves accuracy just a little bit up to 98,1%. Training longer wouldn't help much to improve results and the model starts suffering from gradient explosion (or there is a bug in my code)."},{"metadata":{"_uuid":"c49a65b2e1cfff25b8f0871808e33b1852d9c582"},"cell_type":"markdown","source":"## Testing and Submitting"},{"metadata":{"trusted":true,"_uuid":"cc26d819754f62f36691baef361e3c211c6e93a3"},"cell_type":"code","source":"# Getting data as Numpy object and transpose it to have training examples as columns of the matrix\ntest_data = pd.read_csv('../input/test.csv').values.T\n\n# Train data shape\nm_test = test_data.shape[1]\n\n#Pixels\nX_test_data = test_data[:, :]\nX_test_data = X_test_data/255.0     #Normalising values\n\n# Little clean up\ndel test_data","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"4ef3ea6df665713576a62bc6f050016408752272"},"cell_type":"code","source":"# Forward propagation\nprobs_test, _ = L_model_forward(X_test_data, parameters)\n\n# Convert probabilities into labels\np = probs_test.argmax(axis = 0)\n\nresults = pd.Series(p, name = \"Label\")\n\nsubmission = pd.concat([pd.Series(range(1,28001), name = \"ImageId\"), results], axis = 1)\n\nsubmission.to_csv(\"submission.csv\", index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}