{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Data preprocessing\n\n## Importing the relevant libraries and datasets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=pd.read_csv('../input/digit-recognizer/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df=pd.read_csv('../input/digit-recognizer/test.csv')\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the above data is divided into training and testing datasets. We will also create a separate validation dataset. The label section in the training dataset tells us the actual digit that the model must accurately determine.\n\n\nSince we are dealing with image data in the form of pixel values, we will need to preprocess our data for it to be fed into the neural network.\n\n\n## Visualising the data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let us check how the data actually can be visualised. For this purpose, we need to reshape our 1D data into 2D data of matrix of 28X28. This will give us an idea how actually the data looks like.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df=train_df.copy()\ntemp_df.drop('label',axis=1,inplace=True)\ntemp_df=np.array(temp_df).reshape(-1,28,28,1)\ntemp_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels=train_df['label']\nplt.imshow(temp_df[9][:,:,0])\nprint('The number is:{}'.format(labels[9]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extracting data from the csv files","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"After glancing through the training dataframe, we can realise that the first column named as label is the target column while everything else are inputs for the nueral net.\n\nLet us try to separate this data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"unscaled_inputs=train_df.iloc[:,1:].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"targets=train_df.iloc[:,0].values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Standardize the inputs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"scaled_inputs=preprocessing.scale(unscaled_inputs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Shuffling the data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In case the data was arranged in some particular order, we would want to remove any bias by shuffling the data completely. This will make the dataset more homogeneous in nature and prevent any undue bias in the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"total_indices=scaled_inputs.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total amount of data in the training dataset: {}'.format(total_indices))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us now shuffle all these 42000 indices to make the data homogeneous in nature.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"shuffled_indices=np.arange(total_indices)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.shuffle(shuffled_indices)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shuffled_indices","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the indices have now been all shuffled.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"shuffled_inputs=scaled_inputs[shuffled_indices]\nshuffled_targets=targets[shuffled_indices]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Splitting the dataset into train,validation and test sets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_count=total_indices\n\ntrain_samples_count=int(0.8*samples_count)\nvalidation_samples_count=int(0.1*samples_count)\ntest_samples_count=samples_count-train_samples_count-validation_samples_count","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see from above few codes, we have allocated **80%** of the dataset for **training** , **10%** for **cross validation** and the remaining **10%** for **testing purpose**.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_inputs=shuffled_inputs[:train_samples_count]\ntrain_targets=shuffled_targets[:train_samples_count]\n\nvalidation_inputs=shuffled_inputs[train_samples_count:train_samples_count+validation_samples_count]\nvalidation_targets=shuffled_targets[train_samples_count:train_samples_count+validation_samples_count]\n\ntest_inputs=shuffled_inputs[train_samples_count+validation_samples_count:]\ntest_targets=shuffled_targets[train_samples_count+validation_samples_count:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above code, we have separated all the train, validation and test data and separated the inputs from the targets aswell.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Saving the three datasets into .npz form to be used in further neural network","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.savez('MNIST_train',inputs=train_inputs,target=train_targets)\nnp.savez('MNIST_validation',inputs=validation_inputs,target=validation_targets)\nnp.savez('MNIST_test',inputs=test_inputs,target=test_targets)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the NPZ files","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"npz=np.load('MNIST_train.npz')\ntrain_inputs=npz['inputs'].astype(np.float)\ntrain_targets=npz['target'].astype(np.int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"npz=np.load('MNIST_test.npz')\ntest_inputs=npz['inputs'].astype(np.float)\ntest_targets=npz['target'].astype(np.int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"npz=np.load('MNIST_validation.npz')\nvalidation_inputs=npz['inputs'].astype(np.float)\nvalidation_targets=npz['target'].astype(np.int)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Deep learning model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Creating the neural network model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"From the .CSV files, it is clear that we have the values for a total of **784 pixels** for each digit. This means,it is in the form of a rank 3 tensor as **28 X 28 X 1** . \n\nThe above situation is a problem because it is not possible to feed these values as input in simple neural networks. For convolutional neural networks, there is no issue with such a tensor input. In this case however, we need to apply the layer flattening option provided by Keras.\n\n\nAs we have 784 pixels for each digit, so, out input nodes (or values) will be 784.\n\nLet us take the number of hidden layers as 50\n\nThe digits may range from 0-9. Hence, the number of output values is taken as 10\n\n\nWe are implementing **three sets of hidden layers** initially.\nThe activation function we plan to use for the hidden layer is **'Relu'**\n\n\nFor backpropogation of the output layer, the activation function used is **Softmax** ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"input_size=784\noutput_size=10\nhidden_layer_size=50\n\nmodel=tf.keras.Sequential([\n    #Input layer\n    tf.keras.layers.Dense(input_size),\n    \n    #Hidden layer 1\n    tf.keras.layers.Dense(hidden_layer_size,activation='relu'),\n    #Hidden layer 2\n    tf.keras.layers.Dense(hidden_layer_size,activation='relu'),\n    #Hidden layer 3\n    tf.keras.layers.Dense(hidden_layer_size,activation='relu'),\n    \n    #Output layer\n    tf.keras.layers.Dense(output_size,activation='softmax')\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Choosing the optimizer and the loss function","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam',\n              loss='sparse_categorical_crossentropy',\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training the dataset\n\n\nWe have an option of setting an early stopping criteria which checks the steps where the validation loss increases in the subsequent steps. We can set it to any value less than the numer of epochs. This helps to control the overfit issue. However, since we have already used the validation datasets, we will comment out the code in the model.\n\n\n","execution_count":null},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"NUM_EPOCHS=50\nBATCH_SIZE=100\n\nearly_stopping=tf.keras.callbacks.EarlyStopping(patience=20)\n\nmodel.fit(train_inputs,train_targets,\n          batch_size=BATCH_SIZE,\n          epochs=NUM_EPOCHS,\n          callbacks=[early_stopping],\n          validation_data=(validation_inputs,validation_targets),\n          verbose=2,validation_steps=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Testing the model\n\nInitial testing on a part of the training data will be first done to check how the neural net performs.","execution_count":null},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"test_loss,test_accuracy=model.evaluate(test_inputs,test_targets)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('\\n Test loss:{0:.2f} Test accuracy: {1:.2f} %'.format(test_loss,test_accuracy*100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values=model.predict(test_inputs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(values).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the entries which have 1 are corresponding to the digits of their column name. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Final testing on new dataset","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Once the model has been completely trained, we import the test dataset provided to us.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unscaled_inputs_test=test_df.values\nscaled_inputs=preprocessing.scale(unscaled_inputs_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unlike the training dataset, we shall not shuffle the testing  dataset since we need to preserve the order for submission purpose.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_inputs=scaled_inputs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_inputs.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, we have 28000 different images with their pixel intensities.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.savez('Final_test',inputs=test_inputs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"npz_test=np.load('Final_test.npz')\ntest_inputs=npz_test['inputs'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df=pd.DataFrame(model.predict(test_inputs))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df=values_df[values_df>0.5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We need to sort out the labels for each entry now. We can convert every element in the dataframe into int datatype such that we have only 1s and 0s to make it more readable.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df[values_df>0.5]=1\nvalues_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df.fillna(0,inplace=True)\nvalues_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values_df.size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df=pd.DataFrame(values_df[values_df==1].stack())\npredictions_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df[0].isna().value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df.drop(0,axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The neural net could predict for the above number of cases. Rest could not be identified.\n\nLet us organise the dataframe properly.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df.rename(columns={'level_1':'Label'},inplace=True)\npredictions_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_id=pd.DataFrame(np.arange(0,28000),columns=['ImageId'])\nimage_id['ID']=image_id['ImageId']\nimage_id.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\npredictions_df.rename(columns={'level_0':'ImageId'},inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds=predictions_df.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds=pd.merge_ordered(final_preds,image_id,on='ImageId',fill_method='None')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds['Label'].isna().value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sadly, we could not recognize about few test cases. Let us see the various results amongst the recognized numbers.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds['Label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In a brute and inaccurate manner, we will fill the null values with mode of the Label column.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds['Label'].fillna(final_preds['Label'].mode()[0],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds=final_preds.astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds['ImageId']=final_preds['ImageId']+1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds.drop('ID',axis=1,inplace=True)\nfinal_preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds['Label'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub=pd.read_csv('../input/digit-recognizer/sample_submission.csv')\nsample_sub.dtypes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds['ImageId']=sample_sub['ImageId']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds.to_csv('Final_submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}