{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Table of Contents:\n1. [Introduction](#section-one)\n\n2. [Exploratory Data Analysis](#section-two)\n\n3. [Data Preprocessing](#section-three)\n\n4. [Building ANN](#section-four)\n\n5. [Evaluating the model](#section-five)\n\n6. [Hyperparameter tuning ANN](#section-six)\n\n7. [Conclusion](#section-seven)"},{"metadata":{},"cell_type":"markdown","source":"\n# Introduction\n\nThe deep learning have open the new era to increase the computational power of processing the dataset. Also it has paved way to process unstructured data such as image, audio and video data which normal machine learning models takes longer hours to train. With the help of neural networks and back propogation we can minimize the loss in our prediction and be more accurate.\n\n## What is Deep Learning ?\n\nDeep learning is a part of Machine Learning where the model learns with the help of deep neural networks which resembles human brain. The complex data problems can be solved with the help of Deep learning soon.\n\n## Image Classification\n\nIf you give a human apple, he/she easily identifies that apple as a red fruit and names it 'Apple'. Internally the person's brain captures the image of apple and compares it with the historical images(training data) which the person has seen before and someone telling him apple(label). With the help of this data and label his brain has trained to classify any fruits given to him. The same can be fed to trained neural networks and it shall output us what is the name of the image.\n\n## Acknowledgements\n1. [Keras Documentation](https://keras.io/)\n2. [A Gentle Introduction to ANN](https://www.kaggle.com/nareshbhat/fashion-mnist-a-gentle-introduction-to-ann)- Naresh Bhat\n3. [Comprehensive Guide to ANN with Keras](https://www.kaggle.com/prashant111/comprehensive-guide-to-ann-with-keras)- Prashant Banerjee\n\n## Problem statement\nHere we have two datasets , one MNIST (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems and the other one is fashion MNIST which is also a large database on clothing apparels.Our work is to built an efficient deep learning model to identify handwritten digits and clothing accessories.\n\n## Objectives of Project\nThe Objective of project involves-\n1. Exploratory Data Analysis of MNIST\n2. Data preprocessing\n3. Building Deep learning model(ANN)\n4. Evaluation of model\n\n\n**Note:** This is my first deep learning project and I have considered Fashion MNIST and Digit MNIST dataset for practise.\n\n## Importing Libraries"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Basic Libraries\nimport numpy as np\nimport pandas as pd\n\n# Visualization libraries\nimport matplotlib.pyplot as plt\nimport pydot\nimport seaborn as sns\n\n#Evaluation library\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import GridSearchCV\n\n# Deep Learning libraries\nimport tensorflow as tf\nfrom tensorflow.keras import layers\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers.core import Dense,Activation,Dropout\nfrom keras.datasets import mnist\nfrom keras.utils import to_categorical\nfrom keras.wrappers.scikit_learn import KerasClassifier\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading both datasets\nLet's welcome our MNIST datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Digit MNIST dataset\n(X_train_digit, y_train_digit), (X_test_digit, y_test_digit) = mnist.load_data()\n\n#Fashion MNIST dataset\nfashion_train=pd.read_csv(\"../input/fashionmnist/fashion-mnist_train.csv\")\nfashion_test=pd.read_csv(\"../input/fashionmnist/fashion-mnist_test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How does the image dataset look like?\nYou might think of dataset with only images in the train and test dataset, But computer doesn't understand images but numbers. Let's see how does it look"},{"metadata":{"trusted":true},"cell_type":"code","source":"fashion_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In fashion mnist dataset, the label number doesn't mean the number itself but the id for the clothing accessory.We can get that image from the pixedl values given in the record. Each pixel values vary between 0 to 255. The higher intensity value(255) it resembles a color and lower intensity value(0) is white. There are many shades in between."},{"metadata":{},"cell_type":"markdown","source":"## Train test split-Fashion MNIST\nWe need to split our fashion MNIST dataset into input and label data. Our MNIST has already been extracted as X-y train test data, so no need to split that"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_fashion = fashion_train.drop('label',axis = 1)\ny_train_fashion = fashion_train['label']\nX_test_fashion = fashion_test.drop('label',axis = 1)\ny_test_fashion = fashion_test['label']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Exploratory Data Analysis\n\n\n## Visualizing the numbers\nLet's take a look at the images in each dataset. But before splitting in our fashion dataset the data format is in a dataframe and in such cases we can't view the images,so before that we are reshaping our dataframe and make them into array to get each images in the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Reshaping the dataset\nx_train_reshape = X_train_fashion.values.reshape(-1,28,28)\nx_test_reshape = X_test_fashion.values.reshape(-1,28,28)\n\n#Names of clothing accessories in order \ncol_names = ['T-shirt/top','Trouser','Pullover','Dress','Coat','Sandal','Shirt','Sneaker','Bag','Ankle boot']\n\n#Visualizing the images\nplt.figure(figsize=(10,10))\nfor i in range(15):\n plt.subplot(5,5,i+1)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(x_train_reshape[i], cmap='gray')\n plt.xlabel(col_names[y_train_fashion[i]])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we can see the different types of clothing accessories for both men and women in fashion mnist"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Names of numbers in the dataset in order\ncol_names = ['Zero','One','Two','Three','Four','Five','Six','Seven','Eight','Nine']\n\n#Visualizing the digits\nplt.figure(figsize=(10,10))\nfor i in range(15):\n plt.subplot(5,5,i+1)\n plt.xticks([])\n plt.yticks([])\n plt.imshow(X_train_digit[i], cmap='gray')\n plt.xlabel(col_names[y_train_digit[i]])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we can see the handwrittten digits from the mnist dataset. If you notice all the handwritten records are different from each other which makes it challenging for the computer to predict, but neural network does it with ease"},{"metadata":{},"cell_type":"markdown","source":"## Pixel intensity of images\nWe know the RGB will have values between 0 to 255 where 0 being the lowest intensity(black) and 255 being the highest(white). Let's check out the pixel intensity of each pixel with a help of amazing function taken from [Naresh Bhat's notebook](https://www.kaggle.com/nareshbhat/fashion-mnist-a-gentle-introduction-to-ann)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_input(img, ax):\n ax.imshow(img, cmap='summer')\n width, height = img.shape\n thresh = img.max()/2.5\n for x in range(width):\n for y in range(height):\n ax.annotate(str(round(img[x][y],2)), xy=(y,x),\n horizontalalignment='center',\n verticalalignment='center',\n color='white' if img[x][y]\n# Data processing \n\n## Reshaping Digit MNIST\nThe shape of digit MNIST is extracted in 2D data which can't be fed to a neural network as it allows only 1D data, so we convert them with the help of reshape function. Lets cofirm by checking the dimensions of training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_digit.ndim","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here the dimensions are showing as 3, where the first belongs to records followed by the 2D data. Let's check the shape"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_digit.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test_digit.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the train shape we can see that we have (6000,28,28). Here 6000 is the number of records we have and 28X28 is the dimension of 2D data. Now it can be represented as 784(28X28) which is a 1 dimensional data. By converting into 1D we can feed the data to neural network for training. Now using reshape function,"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_digit = X_train_digit.reshape(60000, 784)\nX_test_digit = X_test_digit.reshape(10000, 784)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_digit.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Encoding the labels\n\nOur MNIST datasets have 10 classes each in both the datasets. Now, let's encode the label classes in the dataset with the help of to_categorical() function from Keras utils library. If the label is '5' it will encode to one in the fifth position of vector and so on for all the class labels. Let's see it visually"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_fashion[0]#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Encoding Digit MNIST Labels\ny_train_digit = to_categorical(y_train_digit, num_classes=10)\n\ny_test_digit = to_categorical(y_test_digit, num_classes=10)\n\n\n#Encoding Fashion MNIST Labels\ny_train_fashion = to_categorical(y_train_fashion, num_classes=10)\n\ny_test_fashion = to_categorical(y_test_fashion, num_classes=10)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_fashion[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the result we can see that the number 2 is in activated in the second position of vector and thus it is encoded. This is similar to one hot encoding."},{"metadata":{},"cell_type":"markdown","source":"\n# Building Deep Learning model- Artificial Neural Network\n\nArtificial Neural network resembles the brain's neural network with densely connected neurons in between input and output layers. It has a hidden layers where the internal processing happens in ANN. The neural network's objective is to minimise the loss(actual-predicted) by using the learning method called as back propogation where the weights get re-initialized in each connecting layer for many epochs through which the loss is minimised.\n\nFirst lets build and compile an ANN model without any hyperparameter tuning and then we apply hyperparameter tuning to understand how the model accuracy improves\n\nIn this stage we follow 3 steps\n1. Defining the model\n2. Compile the model with loss function\n3. Fitting the model to our data\n\n## Defining the model\nTo define the model we need the Sequential() function which helps us to build the base neural network on that we have to decide the dense layers and neurons."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating base neural network\nmodel = keras.Sequential([\n layers.Dense(128, activation='relu', input_shape=(784,)),\n layers.Dropout(0.3),\n layers.BatchNormalization(),\n layers.Dense(24, activation='relu'),\n layers.Dropout(0.3),\n layers.BatchNormalization(),\n layers.Dense(24, activation='relu'),\n layers.Dropout(0.3),\n layers.BatchNormalization(),\n layers.Dense(10,activation='sigmoid'),\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting the schema of neural network\nkeras.utils.plot_model(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We have used relu activation function for the hidden layers and sigmoid for the output layer\n* Since we didn't normalize our dataset we are using BatchNormalization() function to normalize in the neural network\n* We are also considering the drop out layer in each hidden layers to reduce the chances of overfitting"},{"metadata":{},"cell_type":"markdown","source":"## Compiling the model\nThe base model of neural network is ready. It's time to connect the brain for the neural network. In this part we tell the neural network on how to learn the model where we signify the type of loss function and which optimizer and metrics to use.\n\n* Optimizer:\nAdam is an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data. It doesn't take a constand learning rate like SGD, it adapts in changing learning rate in each cycle\n\n* Loss function:\nCategorical crossentropy is a loss function that is used in multi-class classification tasks. These are tasks where an example can only belong to one out of many possible categories, and the model must decide which one.Formally, it is designed to quantify the difference between two probability distributions.\n\n* Metricss:\nAccuracy is the ratio of number of correct predictions to the total number of input samples."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Compiling the model\nmodel.compile(loss=\"categorical_crossentropy\",\n optimizer=\"adam\",\n metrics = ['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fitting the model\nNow, its time to train our neural network. Since we didn't use any hyperparameter training here. I'm just giving value for batch size and epochs myself which peformed better in this case"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(X_train_digit, y_train_digit, batch_size=100, epochs=30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the training results we can see that after just 30 epochs the accuracy has gained to approximately 95% and the loss value is 0.19 which is very good. We can expect a much better accuracy with hyperparameter tuning"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating base neural network\nmodel2 = keras.Sequential([\n layers.Dense(128, activation='relu', input_shape=(784,)),\n layers.Dropout(0.3),\n layers.BatchNormalization(),\n layers.Dense(24, activation='relu'),\n layers.Dropout(0.3),\n layers.BatchNormalization(),\n layers.Dense(24, activation='relu'),\n layers.Dropout(0.3),\n layers.BatchNormalization(),\n layers.Dense(10,activation='softmax'),\n])\n\n#Compiling the model\nmodel2.compile(loss=\"categorical_crossentropy\",\n optimizer=\"adam\",\n metrics = ['accuracy'])\n\n#Fitting the model\nmodel2.fit(X_train_fashion, y_train_fashion, batch_size=100, epochs=30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the fashion MNIST dataset we got the accuracy of 83% and the loss is around 0.5 which is good but can be made better with hyperparameter tuning or CNN network. Let's evaluate both the model with test data"},{"metadata":{},"cell_type":"markdown","source":"\n# Evaluation of Model\n\n## Test accuracy\nNow it's time to check how our model performs when it gets unseen data. We have evaluate() function from keras for evaluating our trained model, let's use that to get the test accuracy of both the datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Evaluating digit MNIST test accuracy\ntest_loss_digit, test_acc_digit = model.evaluate(X_test_digit, y_test_digit)\n\n#Evaluating fashion MNIST test accuracy\ntest_loss_fashion, test_acc_fashion = model2.evaluate(X_test_fashion, y_test_fashion)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Printing the test accuracy results\nprint('Digit MNIST Test accuracy:', round(test_acc_digit,4))\n\nprint('Fashion MNIST Test accuracy:', round(test_acc_fashion,4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the results we can see that digit MNIST data has performed better(97%) on test data compared to the fashion MNIST data(87%). But the situation may change after using hyperparameter tuning. Let's display the confusion matrix\n\n## Confusion Matrix\n\nLet's see how many labels where classified right and how many were misclassified"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Predicting the labels-DIGIT\ny_predict = model.predict(X_test_digit)\ny_predict=np.argmax(y_predict, axis=1) # Here we get the index of maximum value in the encoded vector\ny_test_digit_eval=np.argmax(y_test_digit, axis=1)\n\n#Predicting the labels-Fashion\ny_predict_fash = model2.predict(X_test_fashion)\ny_predict_fash=np.argmax(y_predict_fash, axis=1)\ny_test_fash_eval=np.argmax(y_test_fashion, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Confusion matrix for Digit MNIST\ncon_mat=confusion_matrix(y_test_digit_eval,y_predict)\nplt.style.use('seaborn-deep')\nplt.figure(figsize=(10,10))\nsns.heatmap(con_mat,annot=True,annot_kws={'size': 15},linewidths=0.5,fmt=\"d\",cmap=\"gray\")\nplt.title('True or False predicted digit MNIST\\n',fontweight='bold',fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Confusion matrix for Fashion MNIST\ncon_mat=confusion_matrix(y_test_fash_eval,y_predict_fash)\nplt.style.use('seaborn-deep')\nplt.figure(figsize=(10,10))\nsns.heatmap(con_mat,annot=True,annot_kws={'size': 15},linewidths=0.5,fmt=\"d\",cmap=\"gray\")\nplt.title('True or False predicted Fashion MNIST\\n',fontweight='bold',fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the results we can see that both the matrix show a positive result of having most of the labels classified right and there were very few labels which has been misclassified(numbers which are not in the diagonal part).\n\nIf you notice digit mnist model has done an impeccable work compared to fashion mnist. In fashion mnist label 6 product was classified incorrectly when compared to the rest.\n\nLet's now try hyperparameter tuning and see whether it gets improved."},{"metadata":{},"cell_type":"markdown","source":"\n# Hyperparameter tuning in ANN\nHyperparameters are the variables which determines the network structure(Eg: Number of Hidden Units) and the variables which determine how the network is trained(Eg: Learning Rate).Hyperparameters are set before training(before optimizing the weights and bias). We have a lot of parameters when it comes to \n\n1. Number of layers\n2. Number of neurons in each layers\n3. Batch Size\n4. Epochs\n5. Optimizer\n6. Loss function \n7. Activation\n\nNote : I'm not covering hyperparameter tuning for all the params since it will have high computation and it requires a good device with better workstation to run it smoothly. \n\n## Digit MNIST- Hyperparameter tuning\n\nNow, lets set hyperparameter for Digit MNIST model and repeat the same for Fashion MNIST"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Function for the base neural network\ndef create_model(layers, activation):\n model = Sequential()\n for i, nodes in enumerate(layers):\n if i==0:\n model.add(Dense(nodes,input_dim=X_train_digit.shape[1]))\n model.add(Activation(activation))\n model.add(Dropout(0.3))\n else:\n model.add(Dense(nodes))\n model.add(Activation(activation))\n model.add(Dropout(0.3))\n \n model.add(Dense(units = 10, kernel_initializer= 'glorot_uniform', activation = 'softmax')) \n \n model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Using Keras classifier to apply the function\nmodel3 = KerasClassifier(build_fn=create_model, verbose=0)\n\n#Tuning the layers, activation function and batch sizes\nlayers = [(20,), (40, 20), (45, 30, 15)]\nactivations = ['sigmoid', 'relu','softmax']\nparam_grid = dict(layers=layers, activation=activations, batch_size = [128, 256], epochs=[30])\n\n#Using GridSearchCV to fit the param dictionary\ngrid = GridSearchCV(estimator=model3, param_grid=param_grid,cv=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fitting the params with the training data to figure out the best params and accuracy score\ngrid_result = grid.fit(X_train_digit, y_train_digit)\n\nprint(grid_result.best_score_,grid_result.best_params_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have got the best score as 91% and params where we select sigmoid as activation function with 256 batch size and 40,20 hidden layers"},{"metadata":{},"cell_type":"markdown","source":"## Evaluation of model after Hyperparameter tuning\n\nWe have got a pretty good result. let's evaluate the tuned model with our test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Predicting from the params we got from grid search cv\npred_y = grid.predict(X_test_digit)\n\ny_test_digit=np.argmax(y_test_digit, axis=1)\n\n#Confusion matrix\ncon_mat=confusion_matrix(y_test_digit,pred_y)\nplt.style.use('seaborn-deep')\nplt.figure(figsize=(10,10))\nsns.heatmap(con_mat,annot=True,annot_kws={'size': 15},linewidths=0.5,fmt=\"d\",cmap=\"gray\")\nplt.title('True or False predicted digit MNIST\\n',fontweight='bold',fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Accuracy score \nscore=accuracy_score(pred_y,y_test_digit)\nprint(score)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have got 91% accuracy which is quite poor when compared to the one without hyperparamter tuning. It could be due to the fact that I havent dropout layers here."},{"metadata":{},"cell_type":"markdown","source":"### Submitting for competition"},{"metadata":{"trusted":true},"cell_type":"code","source":"results = model.predict_classes(X_test_digit,verbose = 0)\n\nresults = pd.Series(results, name ='Label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"ANN_mnist.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fashion MNIST- Hyperparameter tuning\n\nIt's time to tune our fashion MNIST model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model_fash(layers, activation):\n model = Sequential()\n for i, nodes in enumerate(layers):\n if i==0:\n model.add(Dense(nodes,input_dim=X_train_fashion.shape[1]))\n model.add(Activation(activation))\n model.add(Dropout(0.3))\n else:\n model.add(Dense(nodes))\n model.add(Activation(activation))\n model.add(Dropout(0.3))\n \n model.add(Dense(units = 10, kernel_initializer= 'glorot_uniform', activation = 'softmax')) \n \n model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Using Keras classifier to apply the function\nmodel4 = KerasClassifier(build_fn=create_model_fash, verbose=0)\n\n#Tuning the layers, activation function and batch sizes\nlayers = [(20,), (40, 20), (45, 30, 15)]\nactivations = ['sigmoid', 'relu','softmax']\nparam_grid = dict(layers=layers, activation=activations, batch_size = [128, 256], epochs=[30])\n\n#Using GridSearchCV to fit the param dictionary\ngrid = GridSearchCV(estimator=model4, param_grid=param_grid,cv=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fitting the params with the training data to figure out the best params and accuracy score\ngrid_result = grid.fit(X_train_fashion, y_train_fashion)\n\nprint(grid_result.best_score_,grid_result.best_params_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have got onlu 78% on accuracy which is poor compared to the one without hyperparameter tuning. There is only one hidden layer which was chosen with the sigmoid activation and there is no drop out layer here as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Predicting from the params we got from grid search cv\npred_y = grid.predict(X_test_fashion)\n\ny_test_fashion=np.argmax(y_test_fashion, axis=1)\n\n#Confusion matrix\ncon_mat=confusion_matrix(y_test_fashion,pred_y)\nplt.style.use('seaborn-deep')\nplt.figure(figsize=(10,10))\nsns.heatmap(con_mat,annot=True,annot_kws={'size': 15},linewidths=0.5,fmt=\"d\",cmap=\"gray\")\nplt.title('True or False predicted fashion MNIST\\n',fontweight='bold',fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Accuracy score \nscore=accuracy_score(pred_y,y_test_fashion)\n\nprint(score)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have got a 73% test accuracy from Fashion MNIST model after tuning . You can notice that label 6 and 4 products have been misclassified alot.You can try by tuning with more adding more in parameter dictionary and get better results than this"},{"metadata":{},"cell_type":"markdown","source":"\n# Conclusion\n\nWe have arrived at the conclusion where I would like to recap what we did in our project. We took two popular MNIST datasets and preprocessed it. We later fed into Artificial Neural network by creating one. Also we performed hyperparameter tuning in ANN and got the best accuracy. Before signing off, I would like to point out the stuff which can be done further\n\n1. More parameters can be tuned \n2. Hyperparameter tuning can also be done on selecting the number of neurons\n3. Can perform techniques such as early stopping and batch normalization\n\nI thank everyone for reading the whole notebook. If you have any critical feedback or suggestions to my work, please drop them in the comments. \n\n\n## You can find my [other notebooks here ](https://www.kaggle.com/benroshan/notebooks)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}