{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Getting the data and processing it\n\n### Creating train and test data","metadata":{}},{"cell_type":"code","source":"'''\nInput functionality\n'''\n\nimport pandas as pd\nimport numpy as np\ndata = pd.read_csv('../input/Kannada-MNIST/train.csv')\n\ndata = data.sample(frac=1)\n#print(int(data.shape[0]/2))\n\ndata_first_half = data.head(30000)\ndata_second_half = data.tail(30000)\n\n### get 100 data points \n### making sure that the data  is balanced\n\ntmp = pd.DataFrame()\nfor label in range(10):\n    if label==0:\n        tmp = data_first_half[data_first_half['label']==label].head(600)\n    else:\n        temp = data_first_half[data_first_half['label']==label].head(600)\n        tmp = pd.concat([tmp,temp])\ndata_balanced = tmp\n\ntmp = pd.DataFrame()\nfor label in range(10):\n    if label==0:\n        tmp = data_second_half[data_second_half['label']==label].head(100)\n    else:\n        temp = data_second_half[data_second_half['label']==label].head(100)\n        tmp = pd.concat([tmp,temp])\ndata_test = tmp\n\nprint(data_test.shape)\nprint(data_balanced.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### De- flattening the data\n* The data frame is de-flattened so that it resembles the shape of an image\n* The shape of the data will look like (N,1,28,28).\n* N is the number of data points.\n* (1,28,28) is the image with one channel.","metadata":{}},{"cell_type":"code","source":"\n### Convert flattened input train data to image data\ndata_balanced = data_balanced.sample(frac=1)\ndata_array = np.zeros((data_balanced.shape[0],1,28,28))\nimage_data = np.array(data_balanced.drop('label',axis=1))\n\nfor i in range(data_balanced.shape[0]):\n    single_image = image_data[i,:].reshape(1,-1)\n    single_image = single_image.reshape(-1,28)\n    data_array[i,0,:,:] = single_image\n\ndata_array = data_array/255.\n\n### Convert flattened input test data to image data\ndata_test = data_test.sample(frac=1)\ndata_test_input = np.zeros((data_test.shape[0],1,28,28))\nimage_data = np.array(data_test.drop('label',axis=1))\n\nfor i in range(data_test.shape[0]):\n    single_image = image_data[i,:].reshape(1,-1)\n    single_image = single_image.reshape(-1,28)\n    data_test_input[i,0,:,:] = single_image\ndata_test_input = data_test_input/255.\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Convert the label data to one hot encoding\n* One hot encoding is a way to encode categorical data.\n* Here, we start by  converting each label into a vector of zeros. The length of the vector will be equal to the total categories available.\n* For each category, we choose a unique position in this vector and flip the 0 at this position to 1.\n* For example , if we are trying to classify and image into a cat, dog and flower, we would have the following one hot encoded vectors\n<br>\n cat -> [0,0,1]<br>\n dog -> [0,1,0]<br>\n flower -> [0,0,1]<br>\n* This is important when we are doing multiclass classification problems. The one hot encoded vectors enable us to calculate cross entropy loss, which we have to reduce.\n\nIn the below code, we convert the labels in the train data into one hot encodings.\n ","metadata":{}},{"cell_type":"code","source":"### Convert Labels to one hot encoding\n\nlabel = data_balanced['label'].tolist()\none_hot_encoding = np.zeros((data_balanced.shape[0],10))\nfor i in range(data_balanced.shape[0]):\n    position = label[i]\n    one_hot_encoding[i,position] = 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Printing the image data\n\n* The following code will print the image data.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nfor i in range(10):\n    plt.imshow(data_array[i,0,:,:])\n    plt.pause(0.5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CNN architecture","metadata":{}},{"cell_type":"markdown","source":"#### The architecture that is used in this classification problem is as follows\n* Input Image (shape : 1X1X28X28)\n* Convolutional Filter of shape 2X1X5X5\n* ReLU function\n* 2X2 Maxpooling layer\n* 1st Fully Connected Layer with 288 units\n* 2nd Fully Connected Layer of 60 units activated with ReLU function\n* Output Layer with 10 units normalized by a softmax function\n\nIn the following sections, I will briefly try to explain and implement convolution, ReLU ,softmax and maxpool functions\n\n##### Convolutional Function\n* The convolution operation helps in selectively increasing/decreasing the pixel values of an image to uncover useful patterns/features for classification.\n* This is done my using a fixed matrix (convolutional filter matrix) and taking the dot product with  chunks of images of the same size. These chunks are chosen recursively based on two parameters ; stride and filter height/width. Stride is how much the convolutional filter moves after each dot product operation.\n* Doing convolution the naive way will result in longer processing times as this would involve nested for loops.\n* One way to combat this is to use the im2col method. The im2col method converts an image to a 2D array. The column of each array will be the elements that make up each stride of the convolution operation.\n* After the image is converted to its im2col form, the convolutional filter is reshaped to a 2D array. Each row of this 2D array will contain a flattened version of each convolutional filter.For example, if we have a convolutional filter of size (2,1,5,5), this means that there are 2 filters of size (1,5,5) each. So after the reshaping, the size of the filter will become (2,25).\n* The code snippet below introduces the im2col function.","metadata":{}},{"cell_type":"code","source":"def im2col(X,conv1, stride, pad):\n    \"\"\"\n        Transforms our input image into a matrix.\n\n        Parameters:\n        - X: input image.\n        - HF: filter height.\n        - WF: filter width.\n        - stride: stride value.\n        - pad: padding value.\n\n        Returns:\n        -cols: output matrix.\n    \"\"\"\n    # Padding\n    X_padded = np.pad(X, ((0,0), (0,0), (pad, pad), (pad, pad)), mode='constant')\n    X = X_padded\n    new_height = int((X.shape[2]+(2*pad)-(conv1.shape[2]))/stride)+1\n    new_width =  int((X.shape[3]+(2*pad)-(conv1.shape[3]))/stride)+1\n    im2col_vector = np.zeros((X.shape[1]*conv1.shape[2]*conv1.shape[3],new_width*new_height*X.shape[0]))\n    c = 0\n    for position in range(X.shape[0]):\n\n        image_position = X[position,:,:,:]\n        for height in range(0,image_position.shape[1],stride):\n            image_rectangle = image_position[:,height:height+conv1.shape[2],:]\n            if image_rectangle.shape[1]<conv1.shape[2]:\n                continue\n            else:\n                for width in range(0,image_rectangle.shape[2],stride):\n                    image_square = image_rectangle[:,:,width:width+conv1.shape[3]]\n                    if image_square.shape[2]<conv1.shape[3]:\n                        continue\n                    else:\n                        im2col_vector[:,c:c+1]=image_square.reshape(-1,1)\n                        c = c+1         \n            \n    return(im2col_vector)\n\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The code snippet below uses im2col to do convolution.","metadata":{}},{"cell_type":"code","source":"X_batch = data_array[0:10,:,:,:]\nconv1 = np.random.randn(2,1,5,5)\n\nX_im2col = im2col(X=X_batch,conv1=conv1,stride=1,pad=0)\nconv1_reshaped = conv1.reshape(conv1.shape[0],-1)\nX_conv = conv1_reshaped@X_im2col\nX_conv = np.array(np.hsplit(X_conv,X_batch.shape[0])).reshape((X_batch.shape[0],conv1.shape[0],24,24))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### MaxPool\nWe use a similar principal for maxpool\n","metadata":{}},{"cell_type":"code","source":"def maxpool_multiple(input_image,stride=2):\n    input_width = input_image.shape[3]\n    input_height = input_image.shape[2]\n    filter_width = 2\n    filter_height = 2\n    \n    output_width = int((input_width-filter_width)/stride)+1\n    output_height = int((input_height-filter_height)/stride)+1\n    \n    output_image = np.zeros((input_image.shape[0],input_image.shape[1],output_width,output_height))\n    for i in range(output_image.shape[0]):\n        output_image[i:i+1,:,:,:] = maxpool(input_image[i:i+1,:,:,:],stride=2)\n    return output_image\n\ndef maxpool(input_image,stride=2):\n    input_width = input_image.shape[3]\n    input_height = input_image.shape[2]\n    filter_width = 2\n    filter_height = 2\n    n_channels = input_image.shape[1]\n    num_images = input_image.shape[0] \n    \n    output_width = int((input_width-filter_width)/stride)+1\n    output_height = int((input_height-filter_height)/stride)+1\n    output = np.zeros((n_channels,output_width*output_height))\n    c=0\n    for height in range(0,input_height,stride):\n        if height+filter_height<=input_height:\n            image_rectangle = input_image[0,:,height:height+filter_height,:]\n            for width in range(0,input_width,stride):\n                if width+filter_width<=input_width:\n                    image_square = image_rectangle[:,:,width:width+filter_width]\n                    image_flatten = image_square.reshape(-1,1)\n#                     print(image_flatten)\n#                     print('----')\n                    output[:,c:c+1] = np.array([float(max(i)) for i in np.split(image_flatten,n_channels)]).reshape(-1,1)\n                    c+=1\n   \n            \n    final_output = np.array(np.hsplit(output,1)).reshape((1,n_channels,output_height,output_width))\n        \n    return final_output\n                    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### ReLU\n* The ReLU function filters out all values less than 0.","metadata":{}},{"cell_type":"code","source":"def ReLU(x):\n    return (x>0)*x","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Softmax\n* The softmax() function normalizes the last layer to a probability distribution.\n* The predicted target class, will be the one with the highest value after softmax.\n* Using softmax helps make the derivative of the last layer with respect to the output, much more easier to calculate.","metadata":{}},{"cell_type":"code","source":"def softmax(x):\n    \n    x_exp = np.exp(x-np.max(x))\n    \n    return x_exp/np.sum(x_exp,axis=0)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Functions for backpropagation.\n* After the signal from the input is passed and processed all the way to the output, it is likely that the estimation of the output is not close to the actual outputs.\n* This calls for a mechanism that minimizes this difference; this is called backpropagation.\n* The error with respect to the output is calculated at each phase of the CNN from the back to the front.\nThe following code snippets will include functions for ReLU Derivatives, storing indices before maxpooling, and calculating error of convolutional filter with respect to the output.","metadata":{}},{"cell_type":"markdown","source":"##### Derivative of ReLU \n* The derivative of ReLU makes sure that the values are 1 for all values > 0.\n* This function will be used during backpropagation for layers that have the ReLU activation function.\n","metadata":{}},{"cell_type":"code","source":"def dReLU(x):\n    return (x>0)*1.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Storing indices during maxpooling\n* During forward propagation, the maxpooling filter, chooses the max element in a defined window and passes it on forward to the subsequent layer.\n* This process is repeated across the image based on stride and filter size.\n* For this example, I have used stride = 2 and filter height/width = 2.\n* During backpropagation, the errors in the subsequent layer is then passed back to the positions where the max values were sourced. So , it is important to store the indices of these positions.","metadata":{}},{"cell_type":"code","source":"\ndef maxpool_indices(input_image,stride=2,filter_height=2, filter_width=2):\n    positional_vector = []\n\n    for channel in range(input_image.shape[1]):\n        x = -1\n\n        chosen_image_channel = input_image[:,channel,:,:]\n        for height in range(0,chosen_image_channel.shape[1],stride):\n            if height+stride<=chosen_image_channel.shape[1]:\n                image_rectangle = chosen_image_channel[:,height:height+filter_height,:]\n                x = x+1\n                y = -1\n                #print('Value of x:',x)\n                for width in range(0,image_rectangle.shape[2],stride):\n                    if width+stride<= image_rectangle.shape[2]:\n                        y = y+1\n                        #print('Value of y:',y)\n                        image_square = image_rectangle[:,:,width:width+filter_width]\n                        \n                        a,b,c = np.unravel_index(image_square.argmax(),image_square.shape)\n\n                        \n                        positional_vector.append([0,channel,int(b)+height,int(c)+width,0,channel,x,y])\n    return positional_vector\n\ndef maxpool_indices_multiple(input_image,stride=2,filter_height=2, filter_width=2):\n    positional_vector =[]\n    for i in range(input_image.shape[0]):\n        positional_vector.append(maxpool_indices(input_image[i:i+1,:,:,:],stride=2,filter_height=2,filter_width=2))\n    return positional_vector\n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Function to calculate the error of convolutional filter wrt output\n* To calculate to error with respect to a convolutional filter, we need two inputs.\n* One input is the layer before the convolution.\n* The other input is the error calculated at the layer after convolution.\n* Each element in the error matrix, multiplied with filter sized chunks of the layer before the convolution. These chunks are then added together to get the error for the convolution.\n* Running this using the naive way, will result in longer processing.\n* To combat this, we change the layer before convolution to im2col format.And then, we change the error of the layer that is calculated after convolution into a 2d array. In this 2D array, each row is a flattend vector of each channel.\n* We then multiply these two and reshape the result.\n\nThe code snippet below shows the error reshape function.","metadata":{}},{"cell_type":"code","source":"def error_layer_reshape(error_layer):\n    test_array = error_layer\n    test_array_new = np.zeros((test_array.shape[1],test_array.shape[0]*test_array.shape[2]*test_array.shape[3]))\n    for i in range(test_array_new.shape[0]):\n        test_array_new[i:i+1,:] = test_array[:,i:i+1,:,:].ravel()\n    return test_array_new","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The code snippet below, gives a brief idea of how to calculate the error of a convolutional layer. Let delta_conv be the error layer of shape (10,2,24,24) and let the layer before convolution be X_batch. We are dealing with a batch of 10 images.","metadata":{}},{"cell_type":"code","source":"X_batch_im2col = im2col(X=X_batch,conv1=conv1, stride=1, pad=0)\ndelta_conv = np.random.rand(10,2,24,24)\ndelta_conv_reshape = error_layer_reshape(delta_conv)\nconv1_delta = (delta_conv_reshape@X_batch_im2col.T).reshape(2,1,5,5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Combining all of these together.\n* We will use the above functions to creating the learning process.\n* The update process for our modeling parameters will happen through Adam Optimizer.\n* We will use a batch size of 128 and with 50 epochs. Batch size is the amount of data points we will train at a time and epoch is when we have been through the entire dataset once.\n* We will also initialize all the modeling parameters once.\n* The accuracy of our model is printed every 10 epochs.","metadata":{}},{"cell_type":"code","source":"'''Simple Architecture for Digit Recognition\n\n1) (1,1,28,28)\n2) Convolution filter (2,1,5,5)\n3) (Max Pool 2x2)\n4) Fc layer (1,288)\n5)Second FC (1,60)\n6) Output Layer(1,10)\n\n'''\n\nepochs = 50\nbatch_size = 128\nbatches = int(data_array.shape[0]/batch_size)\n\nconv1 = np.random.randn(2,1,5,5)*np.sqrt(1./5.)\nW1 = np.random.rand(60,288)/np.sqrt(288)\nB0 = np.zeros((60,1))/np.sqrt(288)\nW2 = np.random.rand(10,60)/np.sqrt(60)\nB1 = np.zeros((10,1))/np.sqrt(60)\nlearning_rate = 0.001\n## Implementing Adam Optimizer\n\nbeta1 = 0.9\nbeta2 = 0.995\nmomentum_w1 = 0\nmomentum_w2 = 0\nmomentum_b0 = 0\nmomentum_b1 = 0\nmomentum_conv1 = 0\nvelocity_w1 = 0\nvelocity_w2 = 0\nvelocity_b0 = 0\nvelocity_b1 = 0\nvelocity_conv1 = 0\nfor epoch_num in range(epochs):\n    \n    '''\n    Choose chunks of data based on batch size  \n    '''\n    i = 0\n    permutation = np.random.permutation(data_array.shape[0])\n    data_array_train = data_array[permutation,:,:,:]\n    one_hot_encoding_train = one_hot_encoding[permutation,:]\n    for i in range(batches):\n        start = i*batch_size\n        end = min(start+batch_size,data_array.shape[0]-1)\n        X_batch = data_array_train[start:end,:,:,:]\n        y_batch = one_hot_encoding_train[start:end,:].T\n        ### First Convolutional Layer\n        #X_conv = conv2dim2col_multiple(input_image=X_batch,conv_filter=conv1,stride=1)\n        X_im2col = im2col(X=X_batch,conv1=conv1,stride=1,pad=0)\n        conv1_reshaped = conv1.reshape(conv1.shape[0],-1)\n        X_conv = conv1_reshaped@X_im2col\n        X_conv = np.array(np.hsplit(X_conv,X_batch.shape[0])).reshape((X_batch.shape[0],conv1.shape[0],24,24))\n        \n        \n        ### Pass through ReLU\n        \n        X_relu = ReLU(X_conv)\n        \n        ### Pass Through Max Pool\n        \n        X_maxpool = maxpool_multiple(X_relu,stride=2)\n        \n        \n        \n        \n        \n        ### Get the indices of maxpool\n        \n        max_indices = maxpool_indices_multiple(X_relu,stride=2,filter_height=2, filter_width=2)\n        \n        ### Flatten the maxpool output\n        input_shape = X_maxpool.shape[0]\n        num_channels = X_maxpool.shape[1]\n        input_width = X_maxpool.shape[2]\n        input_height = X_maxpool.shape[3]\n        X_maxpool_flatten= np.zeros((input_width*input_height*num_channels,input_shape))\n        for image in range(input_shape):\n            X_maxpool_flatten[:,image:image+1] = X_maxpool[image:image+1,:,:,:].ravel().reshape(-1,1)\n        \n        \n        \n        ### Getting into fully connected layers\n        fc1 = ReLU(W1@X_maxpool_flatten+B0)\n        final_fc = softmax(W2@fc1+B1)\n#         print('Sum of Final FC')\n#         print(np.sum(final_fc))\n#         print(final_fc)\n#         break\n#         print('Loss:')\n#         print(cross_entropy(y=y_batch,y_hat=final_fc))\n\n        ### Calculating Loss Through Backprop\n        \n        delta_2 = (final_fc-y_batch)\n        delta_1 = np.multiply(W2.T@delta_2,dReLU(W1@X_maxpool_flatten+B0))\n        delta_0 = np.multiply(W1.T@delta_1,1.0)\n        \n        dW1 = delta_1@X_maxpool_flatten.T\n        dW2 = delta_2@fc1.T\n        dB0 = np.sum(delta_1,axis=1,keepdims=True)\n        dB1 = np.sum(delta_2,axis=1,keepdims=True)\n#         print('Delta 2')\n#         print(delta_2)\n        \n        ### Calculating Error for Last Layer before flattening\n        \n        delta_maxpool = delta_0.reshape(X_maxpool.shape)\n        \n        ### Calculating Error for previous convolutional layer\n        \n        delta_conv = np.zeros(X_conv.shape)\n        for image in range(len(max_indices)):\n            indices = max_indices[image]\n            for p in indices:\n                delta_conv[image:image+1,p[1],p[2],p[3]] = delta_maxpool[image:image+1,p[5],p[6],p[7]]\n        delta_conv = np.multiply(delta_conv,dReLU(X_conv))\n        \n        ### using Im2col\n        X_batch_im2col = im2col(X=X_batch,conv1=conv1, stride=1, pad=0)\n        delta_conv_reshape = error_layer_reshape(delta_conv)\n        conv1_delta = (delta_conv_reshape@X_batch_im2col.T).reshape(2,1,5,5)\n        \n        momentum_w1 = beta1*momentum_w1 + ((1-beta1)*dW1)\n        momentum_w2 = beta1*momentum_w2 + ((1-beta1)*dW2)\n        momentum_b0 = beta1*momentum_b0 + ((1-beta1)*dB0)\n        momentum_b1 = beta1*momentum_b1 + ((1-beta1)*dB1)\n        momentum_conv1 = beta1*momentum_conv1 + ((1-beta1)*conv1_delta)\n        velocity_w1 = beta2*velocity_w1 + ((1-beta2)*dW1**2)\n        velocity_w2 = beta2*velocity_w2 + ((1-beta2)*dW2**2)\n        velocity_b0 = beta2*velocity_b0 + ((1-beta2)*dB0**2)\n        velocity_b1 = beta2*velocity_b1 + ((1-beta2)*dB1**2)\n        velocity_conv1 = beta2*velocity_conv1 + ((1-beta2)*conv1_delta**2)\n        \n        \n        #conv1_delta = conv_filter_error_multiple(input_image=X_batch,error_layer=delta_conv,conv_filter=conv1,stride=1)\n        #print('conv1 delta done')\n        ## Update Weights\n        conv1 = conv1 - learning_rate * momentum_conv1/np.sqrt(velocity_conv1+0.0000001)\n        W1 = W1 - learning_rate*momentum_w1/np.sqrt(velocity_w1+0.0000001)\n        W2 = W2 - learning_rate*momentum_w2/np.sqrt(velocity_w2+0.0000001)\n        B0 = B0 - learning_rate*momentum_b0/np.sqrt(velocity_b0+0.0000001)\n        B1 = B1 - learning_rate*momentum_b1/np.sqrt(velocity_b1+0.0000001)\n        #print('Back Prop Done!')\n        #i+=1\n    \n    X = data_array\n    y = one_hot_encoding.T\n    X_im2col = im2col(X=X,conv1=conv1,stride=1,pad=0)\n    conv1_reshaped = conv1.reshape(conv1.shape[0],-1)\n    X_conv = conv1_reshaped@X_im2col\n    X_conv = np.array(np.hsplit(X_conv,X.shape[0])).reshape((X.shape[0],conv1.shape[0],24,24))\n        \n        \n    ### Pass through ReLU\n        \n    X_relu = ReLU(X_conv)\n        \n    ### Pass Through Max Pool\n        \n    X_maxpool = maxpool_multiple(X_relu,stride=2)\n    input_shape = X_maxpool.shape[0]\n    num_channels = X_maxpool.shape[1]\n    input_width = X_maxpool.shape[2]\n    input_height = X_maxpool.shape[3]\n    X_maxpool_flatten= np.zeros((input_width*input_height*num_channels,input_shape))\n    for image in range(input_shape):\n        X_maxpool_flatten[:,image:image+1] = X_maxpool[image:image+1,:,:,:].reshape(-1,1)  \n    ### Getting into fully connected layers\n    fc1 = ReLU(W1@X_maxpool_flatten+B0)\n    final_fc = softmax(W2@fc1+B1)\n    \n    #### Test Data\n    X = data_test_input\n    #y = one_hot_encoding.T\n    X_im2col = im2col(X=X,conv1=conv1,stride=1,pad=0)\n    conv1_reshaped = conv1.reshape(conv1.shape[0],-1)\n    X_conv = conv1_reshaped@X_im2col\n    X_conv = np.array(np.hsplit(X_conv,X.shape[0])).reshape((X.shape[0],conv1.shape[0],24,24))\n        \n        \n    ### Pass through ReLU\n        \n    X_relu = ReLU(X_conv)\n        \n    ### Pass Through Max Pool\n        \n    X_maxpool = maxpool_multiple(X_relu,stride=2)\n    input_shape = X_maxpool.shape[0]\n    num_channels = X_maxpool.shape[1]\n    input_width = X_maxpool.shape[2]\n    input_height = X_maxpool.shape[3]\n    X_maxpool_flatten= np.zeros((input_width*input_height*num_channels,input_shape))\n    for image in range(input_shape):\n        X_maxpool_flatten[:,image:image+1] = X_maxpool[image:image+1,:,:,:].reshape(-1,1)  \n    ### Getting into fully connected layers\n    fc1 = ReLU(W1@X_maxpool_flatten+B0)\n    final_fc_test = softmax(W2@fc1+B1)\n    \n    \n    \n    \n    if epoch_num % 5  == 0:\n        ### Getting accuracy\n        print('Epoch :', epoch_num)\n        labels_predict = np.argmax(final_fc,axis=0)\n        labels_df  = data_balanced[['label']]\n        labels_predict = labels_predict.tolist()\n        labels_predict = [int(value) for value in labels_predict]\n        #labels_df.loc[:,'label_predict'] = labels_predict\n        labels_df.insert(1,'label_predict',labels_predict)\n        accuracy = np.sum(labels_df['label']==labels_df['label_predict'])/labels_df.shape[0]\n        print('Train Accuracy')\n        print(round(accuracy*100,2),\"%\")\n        \n        ### Test Accuracy\n        \n        labels_predict = np.argmax(final_fc_test,axis=0)\n        labels_df  = data_test[['label']]\n        labels_predict = labels_predict.tolist()\n        labels_predict = [int(value) for value in labels_predict]\n        labels_df.insert(1,'label_predict',labels_predict)\n        #labels_df.loc[:,'label_predict'] = labels_predict\n        accuracy = np.sum(labels_df['label']==labels_df['label_predict'])/labels_df.shape[0]\n        print('Test Accuracy')\n        print(round(accuracy*100,2),\"%\")\n        print('-------------------------')\n        \n#       print(cross_entropy(y=y,y_hat=final_fc))\n        \n\n\n        \n    \n    \nprint('Done!')      ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Saving the weights","metadata":{}},{"cell_type":"code","source":"weight_dict = {\n    \n    'W1' : W1,\n    'B0' : B0,\n    'W2' : W2,\n    'B1': B1,\n    'conv1' : conv1\n}\nimport pickle as pkl\nwith open(\"weights.pkl\", \"wb\") as tf:\n    pkl.dump(weight_dict,tf)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}