{"cells":[{"metadata":{"trusted":true,"_uuid":"a8db3f94f2110d7a147b101aff6fc31387dd02cf"},"cell_type":"markdown","source":"## **Abstract**\n\nIn the code below training on MNIST dataset is done using neural networks. Implementation has been done with minimum use of libraries to get a better understanding of the concept and working on neural nets. Functions for initialization, activation, forward propagation, backward propagation, cost have been written separately.  The training labeled dataset consists of 42000 images, each of size 28x28 = 784 pixels. Labels are from 0 to 9. In this we are not going to use tensorflow or any other such module.\n\n\n[1.Analyse the input data](#1.Analyse the input data)  \n[2.Activation functions](#Activation functions)  \n[3.Derivative of activation function](#Derivative of activation function)  \n[4.Initialization](#Initialization)  \n[5.Forward propagation](#Forward propagation)  \n[6.Cost calculation](#Cost calculation)   \n[7.Back propagation](#Back propagation)   \n[8.Update parameters](#Update parameters)  \n[9.Plot graph of cost](#.Plot graph of cost)  \n[10.Define architecture](#Define architecture)  \n[11.Conclusion](#Conclusion)  \n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport math\nimport matplotlib.cm as cm \nimport matplotlib.pyplot as plt\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"062c382401f859f33a1111bd56b6edd464baa258"},"cell_type":"code","source":"train_data = pd.read_csv(\"../input/train.csv\")\ntest_data= pd.read_csv(\"../input/test.csv\")\n#separating labels and pixels\ntrain_labels=np.array(train_data.loc[:,'label'])\ntrain_data=np.array(train_data.loc[:,train_data.columns!='label'])\n#train_data=train_data/train_data.max()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37c4e47045f16498e0e68362aba14dd7df7220ad"},"cell_type":"markdown","source":"<a id='1.Analyse the input data'></a>\n ## 1.Analyse the input data\n* Visualize the data at particular index position. Change the index to visualise other element.\n* Graph is plotted for the number of occurance of particular element in dataset."},{"metadata":{"trusted":true,"_uuid":"c8b4f3e21900f2b9a63ba02257dc50da13e5652f"},"cell_type":"code","source":"#Visualize the input data. Change the index value to visualize the particular index data.\nindex=7;\nplt.title((train_labels[index]))\nplt.imshow(train_data[index].reshape(28,28), cmap=cm.binary)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f7e575bd28e367fc72a044179d05104064e403c"},"cell_type":"code","source":"print(\"train data\")\ny_value=np.zeros((1,10))\nfor i in range (10):\n    print(\"occurance of \",i,\"=\",np.count_nonzero(train_labels==i))\n    y_value[0,i-1]= np.count_nonzero(train_labels==i)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d1ebd82d2eb8bc8e59bd4fced4bf9c1efb3c8175","scrolled":true},"cell_type":"code","source":"y_value=y_value.ravel()\nx_value=[0,1,2,3,4,5,6,7,8,9]\nplt.xlabel('label')\nplt.ylabel('count')\nplt.bar(x_value,y_value,0.7,color='g')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b133c7e8fe8c3f399ce2aa7bdc29b5c6c81539ad"},"cell_type":"code","source":"#converting train_label in one hot encoder representation \ntrain_data=np.reshape(train_data,[784,42000])\ntrain_label=np.zeros((10,42000))\nfor col in range (42000):\n    val=train_labels[col]\n    for row in range (10):\n        if (val==row):\n            train_label[val,col]=1\nprint(\"train_data shape=\"+str(np.shape(train_data)))\nprint(\"train_label shape=\"+str(np.shape(train_label)))\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7d1d824083076ef8e55b5a4eb24b07e326e0a6c"},"cell_type":"markdown","source":"<a id='Activation functions'></a>\n ## 2.Activation functions\n* Relu\n* Softmax\n* Sigmoid(not used in this code)"},{"metadata":{"trusted":true,"_uuid":"457ba67e9ac9457de158a51dae78a58062082720"},"cell_type":"code","source":"#activation functions sigmoid relu and softmax\ndef sigmoid(Z):\n    A = 1/(1+np.exp(-Z))\n    cache = Z\n    return A, cache\n\ndef relu(Z):\n    A = np.maximum(0,Z)    \n    cache = Z \n    return A, cache\n\ndef softmax(Z):\n    e_x = np.exp(Z)\n    A= e_x / np.sum(np.exp(Z))  \n    cache=Z\n    return A,cache   ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92d33761468ef42d99730b129fd2d54634045f49"},"cell_type":"markdown","source":"<a id='Derivative of activation function'></a>\n ## 3.Derivative of activation function\n* These derivative of activation function are used in back propagation."},{"metadata":{"trusted":true,"_uuid":"af371709dc246bdb5fcbb1de97a0671d03deff1f"},"cell_type":"code","source":"#derivative of activation function\ndef relu_backward(dA, cache):\n    Z = cache\n    dZ = np.array(dA, copy=True)\n    dZ[Z <= 0] = 0\n    assert (dZ.shape == Z.shape)\n    return dZ\n\ndef sigmoid_backward(dA, cache):    \n    Z = cache\n    s = 1/(1+np.exp(-Z))\n    dZ = dA * s * (1-s)\n    assert (dZ.shape == Z.shape)\n    return dZ\n\ndef softmax_backward(Z,cache):\n    Z=cache\n    length=10  \n    dZ=np.zeros((42000,10))\n    Z=np.transpose(Z)\n    for row in range (0,42000):\n            den=(np.sum(np.exp(Z[row,:])))*(np.sum(np.exp(Z[row,:])))\n            for col in range (0,10):\n                sums=0\n                for j in range (0,10):\n                    if (j!=col):\n                        sums=sums+(math.exp(Z[row,j]))\n                \n                dZ[row,col]=(math.exp(Z[row,col])*sums)/den           \n    dZ=np.transpose(dZ)\n    Z=np.transpose(Z)\n\n    assert (dZ.shape == Z.shape)\n    return dZ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cade33a5af8a1eb1f40a6728b95dfdaf16f17e84"},"cell_type":"markdown","source":"<a id='Initialization'></a>\n ## 4.Initializaion\n* Weights\n* Bias"},{"metadata":{"trusted":true,"_uuid":"2fd14dce4aa75ceceb2a84ccae4721da7223fcba"},"cell_type":"code","source":"#initializing the parameters weights and bias\ndef initialize_parameters_deep(layer_dims):\n    #np.random.seed(1)\n    parameters = {}\n    L = len(layer_dims)            # number of layers in the network\n    \n    for l in range(1, L):\n        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l-1]) / np.sqrt(layer_dims[l-1]) #*0.01\n        parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))\n    \n    return parameters","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e71846b960404a3562de3a0d75f38fe28e2ec4b4"},"cell_type":"markdown","source":"<a id='Forward propagation'></a>\n ## 5.Forward propagation\nForward propagation is used to calculate the activated output of particular node. Function linear_forward is used to calculate Z value(z=wa+b). Then it is passed through activation (g(z)) to get the activated output or input for the next layer. The N hidden layers are using 'relu' while the output layer is using 'softmax' to generate out[ut in 10 classes(0 to 9).\n"},{"metadata":{"trusted":true,"_uuid":"5ee72f1b42d0fb5db23119b347fa33cd39d8875e"},"cell_type":"code","source":"#forward propagation\ndef linear_forward(A, W, b):\n    Z = np.dot(W,A) +b\n    cache = (A, W, b)\n    assert(Z.shape == (W.shape[0], A.shape[1]))\n    return Z, cache\n\ndef linear_activation_forward(A_prev, W, b, activation):\n    if activation == \"sigmoid\":\n        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n        Z, linear_cache = linear_forward(A_prev, W, b)\n        A, activation_cache = sigmoid(Z)\n    elif activation == \"relu\":\n        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n        Z, linear_cache = linear_forward(A_prev, W, b)\n        #print(\"Z=\"+str(Z))\n        A, activation_cache = relu(Z) \n    elif activation == \"softmax\":\n        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n        Z, linear_cache = linear_forward(A_prev, W, b)\n        A, activation_cache = softmax(Z)\n    cache = (linear_cache, activation_cache)\n    return A, cache\n\ndef L_model_forward(X, parameters):\n    caches = []\n    A = X\n    L = len(parameters) // 2                  # number of layers in the neural network\n    for l in range(1, L):\n        A_prev = A \n        A, cache = linear_activation_forward(A_prev, parameters['W' + str(l)], parameters['b' + str(l)], activation = \"relu\")\n        caches.append(cache)\n    AL, cache = linear_activation_forward(A, parameters['W' + str(L)], parameters['b' + str(L)], activation = \"softmax\")\n    caches.append(cache)               \n    return AL, caches","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1979ec3c23cb8e48ef5d472c529510d55cd9b78"},"cell_type":"markdown","source":"<a id='Cost calculation'></a>\n ## 6.Cost calculation\n\nCost is calculated usign output of forward propagaion(softmax layer) and train_label. "},{"metadata":{"trusted":true,"_uuid":"ea90857ffd7c47f3162565d5d091c5d765ad3d03"},"cell_type":"code","source":"#cost function\ndef compute_cost(AL, Y):\n    \n    m = Y.shape[1]\n    cost = (-1 / m) * np.sum(np.multiply(Y, np.log(AL)) + np.multiply(1 - Y, np.log(1 - AL)))\n    #print(\"cost=\"+str(cost))\n    return cost","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27fc6359061cfa7f0f1737c7363f6322f7db99b2"},"cell_type":"markdown","source":"<a id='Back propagation'></a>\n ## 7.Back propagation"},{"metadata":{"trusted":true,"_uuid":"3689b433fe672644e57e46e03c94c52891af123e"},"cell_type":"code","source":"#backward propagation\ndef linear_backward(dZ, cache):\n    A_prev, W, b = cache\n    m = A_prev.shape[1]\n    dW = 1./m * np.dot(dZ,A_prev.T)  \n    db = (1/m)*np.sum(dZ, axis=1, keepdims=True);\n    dA_prev = np.dot(W.T,dZ)\n    return dA_prev, dW, db\n\ndef linear_activation_backward(dA, cache, activation):\n    linear_cache, activation_cache = cache\n    if activation == \"relu\":\n        dZ = relu_backward(dA, activation_cache)\n        dA_prev, dW, db = linear_backward(dZ, linear_cache)  \n    elif activation == \"sigmoid\":\n        dZ = sigmoid_backward(dA, activation_cache)\n        dA_prev, dW, db = linear_backward(dZ, linear_cache)\n    elif activation == \"softmax\":\n        dZ = softmax_backward(dA, activation_cache)\n        dA_prev, dW, db = linear_backward(dZ, linear_cache)\n    return dA_prev, dW, db\n\ndef L_model_backward(AL, Y, caches):\n    grads = {}\n    L = len(caches) # the number of layers\n    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL))\n    M=len(layers_dims)\n    current_cache = caches[M-2]\n    grads[\"dA\"+str(M-1)], grads[\"dW\"+str(M-1)], grads[\"db\"+str(M-1)] = linear_activation_backward(dAL, current_cache, activation = \"softmax\")#M-1\n    for l in reversed(range(L-1)):\n        current_cache = caches[l]\n        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads[\"dA\" + str(l + 2)], current_cache, activation = \"relu\")\n        grads[\"dA\" + str(l + 1)] = dA_prev_temp\n        grads[\"dW\" + str(l + 1)] = dW_temp\n        grads[\"db\" + str(l + 1)] = db_temp\n    \n    return grads","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d48ad4a9b4bf69d6918b9a266760ba66cafca12"},"cell_type":"markdown","source":"<a id='Update parameters'></a>\n ## 8.Update parameters\n* Updated Weights    \nW=W-(learning_rate* dW)\n* Updated Bias     \nB=B-(learning_rate* db)"},{"metadata":{"trusted":true,"_uuid":"81cb4e11c50150194bb6e340f748d4bdb3da0ed3"},"cell_type":"code","source":"#upgrade function for weights and bias\ndef update_parameters(parameters, grads, learning_rate):\n    for l in range(len_update-1):\n        parameters[\"W\" + str(l+1)] =parameters[\"W\" + str(l+1)] - (learning_rate*grads[\"dW\" + str(l+1)])\n        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - (learning_rate*grads[\"db\" + str(l+1)])\n    return parameters","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86a631b246a2777fb131ddf4c4239663ef912c07"},"cell_type":"markdown","source":"<a id='.Plot graph of cost'></a>\n ## 9.Plot graph of cost"},{"metadata":{"trusted":true,"_uuid":"b942f22a4c6e525e14c81cc0856d0edb7e4af42d"},"cell_type":"code","source":"def plot_graph(cost_plot):\n       \n    x_value=list(range(1,len(cost_plot)+1))\n    #print(x_value)\n    #print(cost_plot)\n    plt.xlabel('iteration')\n    plt.ylabel('cost')\n    plt.plot(x_value,cost_plot,0.,color='g')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12ca6ff8b508580564d9bce4ddcbd73bfbe7b285"},"cell_type":"markdown","source":"<a id='Define architecture'></a>\n ## 10.Define architecture \n\nDefine layers_dim for required architecture of neural nets. First element is the input layer of 28* 28=784 pixel value. Last element is the output layer of 10 classes(0 to 9). Other elements are hidden layer with number of nodes.(Example First hidden layer has 500 nodes, second hiden layer has 400 nodes.....).\nFeel free to change the architecture and choose the best optimized one!!"},{"metadata":{"trusted":true,"_uuid":"60bb0e22b781b6b5c3a4f5c5457d6186e3180610"},"cell_type":"code","source":"#defining structure of neural network\nlayers_dims = [784,500,400,300,100,10] #  n-layer model (n=6 including input and output layer)\nlen_update=len(layers_dims)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1eae0283381ed4845b3e31ad02d8a00dd8af6237"},"cell_type":"code","source":"#function to call sub_functions\ndef L_layer_model(X, Y, layers_dims, learning_rate , num_iterations , print_cost=False):#lr was 0.009\n    print(\"training...\")\n    costs = []  \n    cost_plot=np.zeros(num_iterations)\n    parameters = initialize_parameters_deep(layers_dims)\n    for i in range(0, num_iterations):\n        AL, caches = L_model_forward(X, parameters)\n        cost =compute_cost(AL, Y)\n        grads = L_model_backward(AL, Y, caches)\n        parameters = update_parameters(parameters, grads, learning_rate) \n        cost_plot[i]=cost;\n    \n    plot_graph(cost_plot)\n    return parameters","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b560c8e376f731e4275596e49cf2acba5a3bbea7","scrolled":false},"cell_type":"code","source":"#variable parameter in network learning_rate, iterationd \nparameters = L_layer_model(train_data, train_label, layers_dims,learning_rate = 0.0005, num_iterations =35 , print_cost = True) \nprint(\"training done\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b566172c2fd1ce07a0036279724bb2d5e6a16461"},"cell_type":"markdown","source":"<a id='Conclusion'></a>\n ## 11.Conclusion \n This code should be sufficient to understand the working of a simple neural network. Understanding the working of activation function, forward and backward propagation will give the user more flexibility and a better understanding of the concept. Although the efficiency of the code is less as compared to the same problem solved through tensorflow but it gives more insight about the network.\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}