{"cells":[{"metadata":{},"cell_type":"markdown","source":"## A simple Neural Net model"},{"metadata":{},"cell_type":"markdown","source":"Classification of hand-written digits using a simple Neural Network Model on the MNIST dataset."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport keras\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport os\nprint(os.listdir(\"../input\"))\n%matplotlib inline\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = train.drop(columns=['label'])\ny_train = train['label']\ng = sns.countplot(y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have similar counts for each digit. The data is not skewed towards any single digit."},{"metadata":{},"cell_type":"markdown","source":"#### Normalisation\nNormalisation has a very big impact on the overall performance of the network. Non-normalised nets do not perform well.\nSince the pixel values range from 0-255, this forms a very large variation in pixel intensity values. By dividing it with 255, the values are normalised in the range 0-1."},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train / 255.0\ntest = test / 255.0\nx_train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> #### Creating the Network"},{"metadata":{},"cell_type":"markdown","source":"I have used Keras to create the network. Keras is a high-level API which is capable of running on tensorflow, theano, CNTK etc backend.\nHere I have used tensorflow backend.\n"},{"metadata":{},"cell_type":"markdown","source":"I will be using a sequential feed-forward model i.e the data flows from left to right, there are no feedback connections.<br>\nFor activation function, i will be using the Rectified Linear Unit (ReLU) for the hidden layers and SoftMax function for the output layer.\nThe softmax function creates a probability distribution from 0 to 1 hence makes it convinient to predict classes.<br>\nDropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the network\n\nmodel = keras.models.Sequential() # Using the Sequentioal feed-forward model\nmodel.add(keras.layers.Dense(256, activation='relu', input_shape=(x_train.shape[1],))) # 1st hidden layer with 128 neurons\n#model.add(keras.layers.Dense(128, activation='relu')) # 2nd hidden layer\nmodel.add(keras.layers.Dropout(0.5)) # Randomly deactivates some neurons. (for 0.5, deactivates 50% neurons) Prevents overfitting.\nmodel.add(keras.layers.Dense(128, activation='relu')) # 3rd hidden layer\nmodel.add(keras.layers.Dense(10, activation='softmax')) # Output layer (using softmax activation function as we require categorical values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Initiailising parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam', loss='sparse_categorical_crossentropy',\n             metrics=['accuracy'])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the loss function used is *sparse_categorical_crossentropy* we need not convert our labels to one-hot encodings."},{"metadata":{},"cell_type":"markdown","source":"### Splitting in training and validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_val, Y_train, Y_val = train_test_split(x_train, y_train, test_size = 0.1, random_state=np.random.seed(2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = X_train\ny_train = Y_train\nhistory = model.fit(x_train, y_train, epochs=30, validation_data = (X_val,Y_val)) # 30 epochs gave around 98.05% accuracy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training and Validation plots"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nfig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['acc'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_acc'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observing the above plots, we can see that model is performing pretty well, although there is a slight problem of overfitting after 15 epochs.\nValidation loss increases after around 15 epochs."},{"metadata":{},"cell_type":"markdown","source":"### Confusion Matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport itertools\n\n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Predict the values from the validation dataset\nY_pred = model.predict(x_train)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = y_train\n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10)) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Confusion matrix is a great way of visualising the performance or rather the shortcomings of our model.\nHere we can see that the moedel is having problems classifying the digits 9,8,6. On further observation 9 is most confused with 7, 8 and 6 are most confused as 0. These mistakes are common and though can be improved upon by using a CNN, these mistakes can are also frequent by humans."},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict([test])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Checking validation loss and acc"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_loss, val_acc = model.evaluate(X_val, Y_val)\nprint(val_loss, val_acc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We have an accuracy of 98.21%"},{"metadata":{"trusted":true},"cell_type":"code","source":"results = np.argmax(predictions,axis = 1)  # Since each prediction is a 1-hot array\n\nresults = pd.Series(results,name=\"Label\")\n#results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n#print(submission)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Writing predictions to csv file"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"More accurate results can be achieved using:\n*  a CNN model for training\n*  Data-augmentation to enhance the quality of the dataset\n*  Using a modified and tuned optimiser instead of default 'adam'\n*  Using Annealers for tweaking learning rate during training."},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}