{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load Libraries and Import Data\n\nIn this code section, the necessary libraries are loaded. The training data set is imported and converted to a  numpy array. The outputs are one hot encoded using the pandas function `get_dummies()`"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\ndata = pd.read_csv('../input/digit-recognizer/train.csv')\ndata = data.sample(frac=1)\nprint(data[['label']].groupby('label').size().reset_index())\n\none_hot = pd.get_dummies(data['label'].unique())\none_hot['label'] = one_hot.index\n\ndata = pd.merge(data,one_hot)\n#data = data.drop('label',axis=1)\ndata = data.sample(frac=1)\n\ndata_train = data.iloc[0:40000]\ndata_test = data.iloc[40000:42000]\ndata_train.drop('label',axis=1,inplace=True)\n\ndata_test.drop('label',axis=1,inplace=True)\n\n## Create the train and test set\nX_train = np.array(data_train.drop([0,1,2,3,4,5,6,7,8,9],axis=1).values)/255\ny_train = np.array(data_train[[0,1,2,3,4,5,6,7,8,9]].values)\nX_test = np.array(data_test.drop([0,1,2,3,4,5,6,7,8,9],axis=1).values)/255\ny_test = np.array(data_test[[0,1,2,3,4,5,6,7,8,9]].values)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The one hot encoded vectors is as follows."},{"metadata":{"trusted":true},"cell_type":"code","source":"one_hot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating a simple Neural Network\n\nThe neural network I have chosen has 3 layers. One input layer, one hidden layer, and one output layer.\n\n* The input layer has 784 units as the data has 784 columns\n* The hidden layer has 100 units and is activated by the sigmoid(x) function so as to take care of non - linear relationships amongst the columns.\n* The output layer has 10 outputs , which signifies the 10 digits that have to be estimated. This layer has a softmax activation.\n\n\n### Creating the weights and bias units\n\nThe following code snippet initializes  two weight matrices namely w1 and w2 and two bias units namely,b0 and b1. Their functions and dimensions are as follows\n\n* Let $X$ be the input matrix that has the dimensions 784 into 40,000. As there are 40,000 observations and 784 features\n* The weight matrix w1 maps the input to the first hidden layer. It has the dimensions 100 into 784. The mapping happens through the matrix multiplication tanh(w1.X+b0). Here b0 is the bias unit and has the dimensions 100 into 1\n* The weight matrix w2 maps the first hidden layer to the output layer. It has the dimensions 10 into 100. The mapping happens through the matrix multiplication softmax(w2.x1+b1). Here b1 is the bias unit and has the dimensions 10 into 1. x1 is the output from the hidden layer\n\nThe weight matrices and the bias units are initialized below. The command `np.random.seed(42)` ensures that the results here are reproducible. A function for the softmax is also declared.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.T\ny_train = y_train.T\nprint(X_train.shape)\nprint(y_train.shape)\nX_test = X_test.T\ny_test = y_test.T\n\ndef sigmoid(x):\n    return(1./(1+np.exp(-x)))\n\ndef softmax(x): \n    \"\"\"Compute softmax values for each sets of scores in x.\"\"\" \n\n    e_x = np.exp(x - np.max(x)) \n\n    return (e_x / e_x.sum(axis=0)) \n\nimport random\nrandom.seed(42)\nw1 = np.random.rand(100,784)/np.sqrt(784)\nb0 = np.zeros((100,1))/np.sqrt(784)\nw2 = np.random.rand(10,100)/np.sqrt(100)\nb1 = np.zeros((10,1))/np.sqrt(100)\nloss=[]\nbatches = 1000\n\nlr = 0.5\nbatch_size = 200\nbeta = 0.9\ncount = 0\nepochs = 500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Forward Pass and Backpropagation\n\n**Forward Pass**\n\n*The first step of estimating the actual results, is to run the algorithms with the parameters we already have.*\n\nFirst Hidden Layer:\n\nX1 = sigmoid(W1X+B0)\n\nOutput Layer:\n\nX2 = softmax(W1X1+B1)\n\n**Calculate Loss**\n\nAs this is a multi-class classification problem, I will be using the cross entropy loss that is given by the following equation\n\n`-np.mean(np.sum(np.multiply(y,np.log(x2))))` \nwhere y is the actual result and x2 is the estimated one. This function penalizes the classifier for making the wrong predictions with high confidence. For example, if y=1 but the classifier predicts x2=0, then the cross entropy loss converges to infinity.\n\n\n**Change Weights Using Gradient Descent**\n\nIn order to find the minima of a loss function, an optimization procedure is used. This procedure is called gradient descent. In the context of neural networks, we need to first calculate the differentials with respect to the output of the layers as well. This process is incapsulated using the following screenshot from http://neuralnetworksanddeeplearning.com/chap2.html\n\n![image](https://raw.githubusercontent.com/adhok/ML-Course-Andrew-Ng-python/master/neural_net.png)\n\n\n**Error Terms**\n\n*Differential With respect to output layer*\n\n`delta_3 = np.multiply((x2 - y)/42000,1)`\n\n*Differential With respect to hidden layer*\n\n`delta_1 = np.multiply(w2.T@delta_2,1,np.multiply(x1,1-x1))`\n\n*Gradient Descent*\n\n`w1 = w1 - alpha *(delta_1@X.T)\n b0 = b0 - (alpha * delta_1)\n w2 = w2 - alpha *(delta_2@x1.T)\n b1 = b1 - (alpha * delta_2)`\n \n where `alpha` is the learning step size. A larger step size can ensure faster learning speeds.\n \n \n ### Code for Neural Net\n \n The code for the forward pass and backpropagation is as follows. The forward pass and backpropagation step is run for 50 epochs.\n\n\n\n\n\n"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"loss_weight_dict = {\n    \n}\n### Forward Pass\nfor i in range(epochs):\n    if i%100==0:\n        print('Epoch :',i)\n    permutation = np.random.permutation(X_train.shape[1])\n    X_train_shuffled = X_train[:, permutation]\n    Y_train_shuffled = y_train[:, permutation]\n    \n    for j in range(batches):\n        \n        begin = j * batch_size\n        end = min(begin + batch_size, X_train.shape[1] - 1)\n        if begin>end:\n            continue\n        X = X_train_shuffled[:, begin:end]\n        Y = Y_train_shuffled[:, begin:end]\n        m_batch = end - begin\n        x1 = sigmoid(w1@X+b0)\n        x2 = softmax(w2@x1+b1)\n        delta_2 = (x2-Y)\n        delta_1 = np.multiply(w2.T@delta_2, np.multiply(x1,1-x1))\n        if i==0 :\n            dW1 = delta_1@X.T\n            dW2 = delta_2@x1.T\n            db0 = np.sum(delta_1,axis=1,keepdims=True)\n            db1 = np.sum(delta_2,axis=1,keepdims=True)\n        else:\n            dW1_old = dW1\n            dW2_old = dW2\n            db0_old = db0\n            db1_old = db1\n            dW1 = delta_1@X.T\n            dW2 = delta_2@x1.T\n            db0 = np.sum(delta_1,axis=1,keepdims=True)\n            db1 = np.sum(delta_2,axis=1,keepdims=True)\n            dW1 = (beta * dW1_old + (1. - beta) * dW1)\n            db0 = (beta * db0_old + (1. - beta) * db0)\n            dW2 = (beta * dW2_old + (1. - beta) * dW2)\n            db1 = (beta * db1_old + (1. - beta) * db1)\n\n\n        w1 = w1 - (1./m_batch)*(dW1)*lr\n        b0 = b0 - (1./m_batch)*(db0)*(lr)\n        w2 = w2 - (1./m_batch)*(dW2)*lr\n        b1 = b1 - (1./m_batch)*(db1)*(lr)\n    \n    x1 = sigmoid(w1@X_train+b0)\n    x2_train = softmax(w2@x1+b1)\n    \n#     print('Training Loss...')\n#     print(-np.mean(np.multiply(y_train,np.log(x2))))\n    add_loss = {\n        'loss' : -np.mean(np.multiply(y_train,np.log(x2_train))),\n        'weight_1' : w1,\n        'weight_2':w2,\n        'b0' : b0,\n        'b1': b1\n    }\n    \n    \n    \n    \n    \n    x1 = sigmoid(w1@X_test+b0)\n    x2_test = softmax(w2@x1+b1)\n#     print('Testing Loss...')\n#     print(-np.mean(np.multiply(y_test,np.log(x2))))\n    \n    add_loss['testing_loss'] = -np.mean(np.multiply(y_test,np.log(x2_test)))\n    loss_weight_dict[count] = add_loss\n    count = count + 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Testing Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_loss = []\n\nfor i in range(len(loss_weight_dict)):\n    test_loss.append(loss_weight_dict[i]['testing_loss'])\nimport matplotlib.pyplot as plt\nplt.plot(test_loss)\nplt.xlabel('Epochs')\nplt.ylabel('Testing Loss')\nplt.show()\n\nprint('Index where test loss is minimum :',test_loss.index(min(test_loss)))\nprint('Minimum Test Loss is :',min(test_loss))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loss = []\n\nfor i in range(len(loss_weight_dict)):\n    train_loss.append(loss_weight_dict[i]['loss'])\nimport matplotlib.pyplot as plt\nplt.plot(train_loss)\nplt.xlabel('Epochs')\nplt.ylabel('Training Loss')\nplt.show()\n\nprint('Training Loss at index where test loss is minimum :', train_loss[test_loss.index(min(test_loss))])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Trying the weights on the test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"testing_data = pd.read_csv('../input/digit-recognizer/test.csv')\ntest_data = np.array(testing_data.values)\nprint(test_data.shape)\ntest_data = test_data.T\n\ntest_data = test_data/255\n\nx1_test = sigmoid(w1@test_data+b0)\ny_test = softmax(w2@x1_test+b1)\n\ny_test_df = pd.DataFrame(y_test)\ny_test_df = (y_test_df == y_test_df.max()).astype(int)\n\ny_test_df = y_test_df.transpose().merge(one_hot)[['label']] \n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test_df = y_test_df.rename(columns = {'label':'Label'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test_df['ImageId'] = y_test_df.index+1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test_df.groupby('Label').size().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test_df.to_csv('sample_submission.csv',mode = 'w', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}