{"cells":[{"metadata":{"_uuid":"972f90217a185b93981aef489046a0aefbc64dc8"},"cell_type":"markdown","source":"# **MNIST: Sklearn and Keras** \n**Comparing the performance of ML and DL classification models on**  \n**the Digit Recognizer (MNIST) competition**  \n\n\n*from Kaggle:*   \nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision.  \nIn this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images."},{"metadata":{"_uuid":"a8840cfb479892df49d74122e7281a1b949d4036"},"cell_type":"markdown","source":"**My first kernel for the MNIST Digit Recognizer Competition**  \n\nThe kernel includes an **EDA part** on the distribution of labels and some sample images for each digit.  \nWe explore how the images of handwritten digits are stored as 784 values (pixels).  \nWe transform these 784 pixels back to images of 28 x 28 (width x heigth) and have a look at possible difficulties in identifying and distinguishing certain digits.  \nThe part on **prediction modelling** starts with **classifiers from sklearn**:  \nPerceptron, Logistic Regression, Random Forest, SVM, Multi Layer Perceptron  \nand then continues to **Neural Networks with Keras**:   \nDense (1,2 and 3 layers) and CNN (Conv2D, MaxPooling, Dropout)"},{"metadata":{"_uuid":"5a15f0810640d7baad9c31c5b530814343d65dc6"},"cell_type":"markdown","source":"**Outline of the kernel:**\n\n**Part 0: Imports, functions**\n\n[some useful functions](#some-useful-functions)\n\n\n**Part 1: Exploring the Data**\n\n[Distribution of labels](#Distribution-of-labels)  \n[first 10 samples for each digit](#first-10-image-samples-for-each-digit)  \n[Conclusions on EDA](#Conclusions-on-EDA) \n\n\n**Part 2: Sklearn Classifiers** \n\n[Perceptron](#Perceptron)  \n[Logistic Regression](#Logistic-Regression)  \n[KNN](#KNN)  \n[Random Forest Classifier](#Random-Forest-Classifier)  \n[Support Vector Machine Classifier](#Support-Vector-Machine-Classifier)  \n[Multi Layer Perceptron](#Multi-Layer-Perceptron)\n\n\n**Part 3: NN and CNN Classifiers with Keras**\n\n3.1 Fully-Connected Neural Networks  \n[Keras: dense, 1 hidden layer](#Keras:-1-hidden-layer)  \n[Keras: dense, 2 hidden layers](#Keras:-2-hidden-layers)  \n[Keras: dense, 3 hidden layers](#Keras:-3-hidden-layers)  \n3.2 Convolutional Neural Networks, CNN  \n[features : reshaping 1d vector to 2d images](#features-:-reshaping-1d-vector-to-2d-images)  \n[Keras: CNN, model 1](#Keras:-CNN-model-1)  \n[Keras: CNN, model 2](#Keras:-CNN-model-2)\n\n**Part 4: Predictions for test data**  \n[Fitting on all training data](#Fitting-on-all-training-data)  \n\n**Part 5: Comparing classifier performance**  \n[Validation scores](#Validation-scores)  \n[Test scores](#Test-scores)  \n\n\n**Part 6: Different optimizers**  \nadadelta  \nrmsprop  \nadam  \n\n**Part 7: Investigating false predictions**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f001988cbe36bd9d1ce9dcc712c7b06635fd8a7d"},"cell_type":"markdown","source":"# Part 0: Imports, Functions"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \n\nimport random as rn\n\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\n\n# plotly library\nimport plotly.plotly as py\nimport plotly.graph_objs as go\nfrom plotly import tools\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import confusion_matrix, classification_report, accuracy_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import cross_val_score, KFold \nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\nfrom sklearn.pipeline import Pipeline\nfrom scipy.stats import uniform\n\nimport itertools\n\nimport warnings\nfrom sklearn.exceptions import ConvergenceWarning\nwarnings.filterwarnings(\"ignore\", category=ConvergenceWarning)\n\nfrom keras.utils.np_utils import to_categorical\nfrom keras.utils import np_utils\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D, MaxPool2D\n#from keras.layers import AvgPool2D, BatchNormalization, Reshape\nfrom keras.optimizers import Adadelta, RMSprop, Adam\nfrom keras.losses import categorical_crossentropy\nfrom keras.wrappers.scikit_learn import KerasClassifier\n\nimport tensorflow as tf\n\nimport os\nprint(os.listdir(\"../input\"))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"613c5dcdfdde0b99ef2292da1d59c22edfc70fcf"},"cell_type":"markdown","source":"**some global variables**"},{"metadata":{"trusted":true,"_uuid":"d20ddc5ea49a5ddd0cb251a32c1fafe257afb124"},"cell_type":"code","source":"img_rows, img_cols = 28, 28\n\nnp.random.seed(5)\n#rn.seed(5)\n#tf.set_random_seed(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e76e22587309810772340c55880e3c721772257a"},"cell_type":"markdown","source":"### **some useful functions**"},{"metadata":{"_uuid":"67b0d192f7523e0bae49f6fc5498d83418f6d97b"},"cell_type":"markdown","source":"### get_best_score for GridSearchCV"},{"metadata":{"trusted":true,"_uuid":"3796d134b019785d4b60357db28151a4b32dc7ab"},"cell_type":"code","source":"def get_best_score(model):\n    \n    print(model.best_score_)    \n    print(model.best_params_)\n    print(model.best_estimator_)\n    \n    return model.best_score_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b48ff13fda2010e77067f7f9e0a312774f7c6b7"},"cell_type":"markdown","source":"### print Classification Report and Accuracy"},{"metadata":{"trusted":true,"_uuid":"6b2924e8055b1cd5c87db52dc75c5d18fb2f932f"},"cell_type":"code","source":"def print_validation_report(y_true, y_pred):\n    print(\"Classification Report\")\n    print(classification_report(y_true, y_pred))\n    acc_sc = accuracy_score(y_true, y_pred)\n    print(\"Accuracy : \"+ str(acc_sc))\n    \n    return acc_sc","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be3fac7d609a819d441a1799b57891381d21253c"},"cell_type":"markdown","source":"### plot_confusion_matrix"},{"metadata":{"trusted":true,"_uuid":"be399813086badb7465791bc904ff95970687f3b"},"cell_type":"code","source":"def plot_confusion_matrix(y_true, y_pred):\n    mtx = confusion_matrix(y_true, y_pred)\n    fig, ax = plt.subplots(figsize=(8,8))\n    sns.heatmap(mtx, annot=True, fmt='d', linewidths=.5,  cbar=False, ax=ax)\n    #  square=True,\n    plt.ylabel('true label')\n    plt.xlabel('predicted label')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"95c1ee02cd8ecc860c901bb5d91a34544dd7e3ad"},"cell_type":"markdown","source":"### plot_history_loss_and_acc"},{"metadata":{"trusted":true,"_uuid":"678cc8ca95af1cc66bf9b365e996225751156242"},"cell_type":"code","source":"def plot_history_loss_and_acc(history_keras_nn):\n\n    fig, axs = plt.subplots(1,2, figsize=(12,4))\n\n    axs[0].plot(history_keras_nn.history['loss'])\n    axs[0].plot(history_keras_nn.history['val_loss'])\n    axs[0].set_title('model loss')\n    axs[0].set_ylabel('loss')\n    axs[0].set_xlabel('epoch')\n    axs[0].legend(['train', 'validation'], loc='upper left')\n\n    axs[1].plot(history_keras_nn.history['acc'])\n    axs[1].plot(history_keras_nn.history['val_acc'])\n    axs[1].set_title('model accuracy')\n    axs[1].set_ylabel('accuracy')\n    axs[1].set_xlabel('epoch')\n    axs[1].legend(['train', 'validation'], loc='upper left')\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c709b6bd14531e7b0248d7e0ef3abb378ae2ae3"},"cell_type":"markdown","source":"### **Reading the data**"},{"metadata":{"trusted":true,"_uuid":"a5ecb7548864377c6d794e0f61b79884e45602fe"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02f31d6447f9f2f554f6055314a4d7d60ffa8d2a"},"cell_type":"code","source":"y = train[\"label\"]\nX = train.drop([\"label\"],axis = 1)\nX_test = test","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81951a034d3cbe377d8d6ab3ca6386e3fc6ba278"},"cell_type":"markdown","source":"### Normalization"},{"metadata":{"trusted":true,"_uuid":"865c65cb54e8c0f619914a445133800453620fd1"},"cell_type":"code","source":"X = X/255.0\nX_test = X_test/255.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"486e5a1ec5ea278cf8404cdc289bf252c453b8ed"},"cell_type":"markdown","source":"### setting train and validation data"},{"metadata":{"trusted":true,"_uuid":"551c7ff8551d3c080b22d7f014cb4b9f0d64ae23"},"cell_type":"code","source":"# for best performance, especially of the NN classfiers,\n# set mode = \"commit\"\nmode = \"edit\"\nmode = \"commit\"\n#\n\nif mode == \"edit\" :\n    nr_samples = 1200\n\nif mode == \"commit\" :    \n    nr_samples = 30000\n\ny_train=y[:nr_samples]\nX_train=X[:nr_samples]\nstart_ix_val = nr_samples \nend_ix_val = nr_samples + int(nr_samples/3)\ny_val=y[start_ix_val:end_ix_val]\nX_val=X[start_ix_val:end_ix_val]\n    \nprint(\"nr_samples train data:\", nr_samples)\nprint(\"start_ix_val:\", start_ix_val)\nprint(\"end_ix_val:\", end_ix_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4fe3baf442262e4e1a6b7e79595ea0651c043754"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc36e607a9b91f6b3629a69b04fcd4c63a9d00a2"},"cell_type":"markdown","source":"# Part 1 : Exploring the Data"},{"metadata":{"trusted":true,"_uuid":"3a29bc0afc5386409cbc482d99e5907dc45c4771"},"cell_type":"code","source":"print(\"X:\")\nprint(X.info())\nprint(\"*\"*50)\nprint(\"X_test:\")\nprint(X_test.info())\nprint(\"*\"*50)\nprint(\"y:\")\nprint(y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23b83df6872036ce3777486826a6752db65489e2"},"cell_type":"code","source":"X.iloc[0:5,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b6fe345d6f690bffcfbdb2bd1484c56922cb8a78"},"cell_type":"code","source":"y.iloc[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"015ed5a417dcb5af2d4ade65726e9eeb5db58bc1"},"cell_type":"code","source":"fig, axs = plt.subplots(1, 5, sharex=True, sharey=True, figsize=(10,6))\naxs = axs.flatten()\nfor i in range(0,5):\n    im = X.iloc[i]\n    im = im.values.reshape(-1,28,28,1)\n    axs[i].imshow(im[0,:,:,0], cmap=plt.get_cmap('gray'))\n    axs[i].set_title(y[i])\nplt.tight_layout()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec0b8d7effc2646021d3774443b14051986bccb1"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba4253f25dc25b8e0f8bb6e95669d28d5e253161"},"cell_type":"markdown","source":"### Distribution of labels \n**(in train set)**"},{"metadata":{"trusted":true,"_uuid":"2bd17d6ddc16abc28916fbffba8d8f1dc85d5889"},"cell_type":"code","source":"y.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d6341bf5524ea7fb274360332ad5823c4982fd0"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8,5))\ng = sns.countplot(y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49a93e3fb04db8f2bdf2cc3571025becfabf0c5a"},"cell_type":"markdown","source":"### Check some images"},{"metadata":{"_uuid":"1b37d40c34f0764b5b380a387b85c0e375c7ca33"},"cell_type":"markdown","source":"**get indexes of first 10 occurences for each number**"},{"metadata":{"trusted":true,"_uuid":"2ca8bc2ba4dbce27182114deb56570021b978320"},"cell_type":"code","source":"li_idxs = []\nfor i in range(10):\n    for nr in range(10):\n        ix = y[y==nr].index[i]\n        li_idxs.append(ix) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc04c8a1b0483378d761d1f750e4543d33136638"},"cell_type":"markdown","source":"### first 10 image samples for each digit"},{"metadata":{"trusted":true,"_uuid":"248fd67a691c82e7cb152b84e4d0d85514c22856"},"cell_type":"code","source":"fig, axs = plt.subplots(10, 10, sharex=True, sharey=True, figsize=(10,12))\naxs = axs.flatten()\nfor n, i in enumerate(li_idxs):\n    im = X.iloc[i]\n    im = im.values.reshape(-1,28,28,1)\n    axs[n].imshow(im[0,:,:,0], cmap=plt.get_cmap('gray'))\n    axs[n].set_title(y[i])\nplt.tight_layout()    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38b7b8cdbd0f70e3829e42ae638811c696e6579b"},"cell_type":"markdown","source":"### **Conclusions on EDA**"},{"metadata":{"_uuid":"c7b46c80bd0cc204f81dcb993aa9bd00bfa214f7"},"cell_type":"markdown","source":"1. **We can see that there is a lot of variety in the look of the numbers**  \n2. **The digits are obviously written by many different people**\n3. **And it seems that there might be difficulties identifying some of the digits correctly also for humans:**\n\n\n"},{"metadata":{"trusted":true,"_uuid":"cd5d6a3309f095b03b1247157793d10a92f99c82"},"cell_type":"markdown","source":"* The 6s in row 0 1 and 6 look close to a 4  \n* The 5 in row 1 looks close to a 6 and the 5 in row 8 looks close to a 8  \n* The 4 in row 4 is wriiten differently as the other 4s and the 4 in row 7 almost looks like a 7  \n* The 8 in row 8 is wriiten differently as the other 8s  "},{"metadata":{"trusted":true,"_uuid":"d9f9f7fd24e590abaca77f1ffded6c7af50427ea"},"cell_type":"markdown","source":"# Part 2 : Sklearn Classifiers"},{"metadata":{"trusted":true,"_uuid":"b8bb8350dbc4f58052cfce448e998b3302690ea1"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77ee05aabb4dfd5eb1eceabc1a98b21e020ee2a9"},"cell_type":"markdown","source":"### Perceptron"},{"metadata":{"trusted":true,"_uuid":"1ae5cd03701f0c0555927f24d469123113a436cd"},"cell_type":"code","source":"from sklearn.linear_model import Perceptron\nclf_Perceptron = Perceptron(random_state=0)\nparam_grid = { 'penalty': ['l1','l2'], 'tol': [0.05, 0.1] }\nGridCV_Perceptron = GridSearchCV(clf_Perceptron, param_grid, verbose=1, cv=5)\nGridCV_Perceptron.fit(X_train,y_train)\nscore_grid_Perceptron = get_best_score(GridCV_Perceptron)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a837cd6234557015c8326cd281107d2209ca8a40"},"cell_type":"code","source":"pred_val_perc = GridCV_Perceptron.predict(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e8fcea4c58c1467b7e6f9d1990e4b924180786e2"},"cell_type":"code","source":"acc_perc = print_validation_report(y_val, pred_val_perc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8290c219c3d81cb6856231e6360f70a5f65481c9"},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_perc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0c06780cfac00efcde0a07784785b9b68e55c14"},"cell_type":"markdown","source":"### Logistic Regression"},{"metadata":{"trusted":true,"_uuid":"6509cf8da9e4666006dc7b72f55bb510ef29309c"},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nclf_LR = LogisticRegression(random_state=0)\nparam_grid = {'C': [0.014,0.012], 'multi_class': ['multinomial'],  \n              'penalty': ['l1'],'solver': ['saga'], 'tol': [0.1] }\nGridCV_LR = GridSearchCV(clf_LR, param_grid, verbose=1, cv=5)\nGridCV_LR.fit(X_train,y_train)\nscore_grid_LR = get_best_score(GridCV_LR)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47ff8a03c200fe2984be94514008591c5ba37c47"},"cell_type":"code","source":"pred_val_lr = GridCV_LR.predict(X_val)\nacc_lr = print_validation_report(y_val, pred_val_lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ef5fd871318e6d17943399294ac149c5a10680af"},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### KNN"},{"metadata":{},"cell_type":"markdown","source":"Using GridSearchCV with KNN takes very long for this dataset.  \nTherefore I fit the data with one parameter: neighbors = 10  \nThe resulting accuracy is already quite good."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier\nclf_knn = KNeighborsClassifier(n_neighbors=10)\nclf_knn.fit(X_train,y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_val_knn = clf_knn.predict(X_val)\nacc_knn = print_validation_report(y_val, pred_val_knn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_knn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd8341887be07bc821a1a9033c77543d9e788f9e"},"cell_type":"markdown","source":"### Random Forest Classifier"},{"metadata":{"trusted":true,"_uuid":"b93fae420be63286ee5048491f5db002dd081b7a"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nclf_RF = RandomForestClassifier(random_state=0)\nparam_grid = {'max_depth': [15], 'max_features': [100],  \n              'min_samples_split': [5],'n_estimators' : [50] }\nGridCV_RF = GridSearchCV(clf_RF, param_grid, verbose=1, cv=5)\nGridCV_RF.fit(X_train,y_train)\nscore_grid_RF = get_best_score(GridCV_RF)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd62903ef129eb9942ef3e39a943cacbc4d67482"},"cell_type":"markdown","source":"'RandomForestClassifier' object has no attribute 'loss_curve_'"},{"metadata":{"trusted":true,"_uuid":"d42a0976e9aa538de2b8420c2a288f5dc700bff6"},"cell_type":"code","source":"pred_val_rf = GridCV_RF.predict(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afb30f89aa77e995e817dd01df9763379d10178f"},"cell_type":"code","source":"acc_rf = print_validation_report(y_val, pred_val_rf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"070d70f7866ba9f35d1a38f37365f38439db5b88"},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_rf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Support Vector Machine Classifier"},{"metadata":{},"cell_type":"markdown","source":"Like for KNN, GridSearchCV for SVM takes very long, so I only fit one good set of parameters here."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.svm import SVC\nclf_svm = SVC(C=5, gamma=0.05, kernel='rbf', random_state=0)\nclf_svm.fit(X_train,y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_val_svm = clf_svm.predict(X_val)\nacc_svm = print_validation_report(y_val, pred_val_svm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_svm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"716f78e7643e6ec7053c036588f2d93a85ac6d46"},"cell_type":"markdown","source":"### Multi Layer Perceptron"},{"metadata":{"trusted":true,"_uuid":"3658d5421e64553a02fa77736f671dd96adce9e5"},"cell_type":"code","source":"batchsize = int(nr_samples/15) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e15b9d68379a260c295e8b957669d3440af34a48"},"cell_type":"code","source":"from sklearn.neural_network import MLPClassifier\n\nclf_mlp = MLPClassifier(activation = \"logistic\", hidden_layer_sizes=(200,), random_state=0)\nparam_grid = { 'batch_size' : [batchsize] , 'max_iter': [600], 'alpha': [1e-4], \n               'solver': ['sgd'], 'learning_rate_init': [0.05,0.06],'tol': [1e-4] }\n    \nGridCV_MLP = GridSearchCV(clf_mlp, param_grid, verbose=1, cv=3)\nGridCV_MLP.fit(X_train,y_train)\nscore_grid_MLP = get_best_score(GridCV_MLP)   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4ee434b45062ad02a390b2166cc5965b832eca45"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(6,3))\nax.plot(GridCV_MLP.best_estimator_.loss_curve_)\n\nplt.xlabel(\"number of steps\") \nplt.ylabel(\"Loss During GD\")\nplt.title(\"loss function\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f459f1891db4014d53e8b3095b194257c67a35fe"},"cell_type":"code","source":"pred_val_mlp = GridCV_MLP.predict(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7d06686ac5bfe2d4564f340bb2e65fba84c91ef"},"cell_type":"code","source":"acc_mlp = print_validation_report(y_val, pred_val_mlp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a535848bbfce597e6d669765a96ca51e42983338"},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_mlp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"097ae27f3bbfd09c5b1ba7d62f59282fca453087"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4a55bacaddeed8be4b5ebdb6f6144558a1ab478"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"143c1fb5c6bc3717dc777320e3f7dcce7d4c0ac3"},"cell_type":"markdown","source":"# Part 3: NN Classifiers with Keras"},{"metadata":{"trusted":true,"_uuid":"470be43f625218e38f19c27bb36e8b05f3e12eef"},"cell_type":"code","source":"y_train = to_categorical(y_train, 10)\ny_val_10 = to_categorical(y_val, 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"16450c1055090761ba1bf1ab7f15f4fbbee6322b"},"cell_type":"markdown","source":"## 3.1 Fully-Connected Neural Networks  \ndense layers : every node is connected to every other node in the next layer"},{"metadata":{"_uuid":"3c484e94988c360c58c7b740c84f6cd679a70f9d"},"cell_type":"markdown","source":"### 3.1.1 Keras: only input and output layer"},{"metadata":{"trusted":true,"_uuid":"fc358789c6fbf0c32509647e895fa6d52d2c23d6"},"cell_type":"code","source":"def dense_model_0():\n    model = Sequential()\n    model.add(Dense(10, input_dim=784, activation='softmax'))\n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48a5a62896ef961e959086b8450c3f9212db1302"},"cell_type":"code","source":"model_dense_0 = dense_model_0()\nmodel_dense_0.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffacc9b33d66d05f52dd2f45889ebe9b0478ad35","_kg_hide-output":true},"cell_type":"code","source":"model_dense_0.fit(X_train, y_train, epochs=50, batch_size=batchsize)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d6ff3ada3417c10d34f1ededd39767e917981c3"},"cell_type":"code","source":"pred_val_dense0 = model_dense_0.predict_classes(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f55b909222491635ae3d91fb5b7900bbc2bd1e04"},"cell_type":"code","source":"acc_fc0 = print_validation_report(y_val, pred_val_dense0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75140ffcb2a90a14caf9c1c5dd280dca80a12a24"},"cell_type":"code","source":"plot_confusion_matrix(y_val, pred_val_dense0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c932a17421c381b2457aef0799fd19a57da1e6a5","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b01e3e51fba907baa34d4aafbde9f8715b9e201e"},"cell_type":"markdown","source":"### Keras: 1 hidden layer"},{"metadata":{"trusted":true,"_uuid":"73b54c77ec3ec628e4ccf9e6c44754a00b16cfea"},"cell_type":"code","source":"def dense_model_1():\n    model = Sequential()\n    model.add(Dense(100, input_dim=784, activation='relu'))\n    model.add(Dense(10, activation='softmax'))\n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a87ca19fa6dc47a415743eb17e42612d2fcfc5de"},"cell_type":"code","source":"model_dense_1 = dense_model_1()\nmodel_dense_1.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2587a85de2ebf23a894c8e32fc108dc6e1544fc","_kg_hide-output":true},"cell_type":"code","source":"history_dense_1 = model_dense_1.fit(X_train, y_train, validation_data=(X_val,y_val_10), \n                                    epochs=50, batch_size=batchsize)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d359f79d899cb8c42cd5c39d045846bfbc233c6"},"cell_type":"code","source":"plot_history_loss_and_acc(history_dense_1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c90f70cc1503464de85adf5a93393e2e69ea687"},"cell_type":"code","source":"pred_val_dense1 = model_dense_1.predict_classes(X_val)\nplot_confusion_matrix(y_val, pred_val_dense1)\nprint(classification_report(y_val, pred_val_dense1))\nacc_fc1 = accuracy_score(y_val, pred_val_dense1)\nprint(acc_fc1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e1eab41f877a4589e58bb668f7e421d9240f352"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"811d096ef0c9ff6a9f43dd6efd456999cb3bc1a6"},"cell_type":"markdown","source":"### Keras: 2 hidden layers"},{"metadata":{"trusted":true,"_uuid":"77cad84ecaf24475533a22c046ff7530da157337"},"cell_type":"code","source":"def dense_model_2():\n    model = Sequential()\n    model.add(Dense(100, input_dim=784, activation='relu'))\n    model.add(Dense(200, activation='relu'))\n    model.add(Dense(10, activation='softmax'))\n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"576a5bc338ac468bb363a0f0d3c9a153cd0c733e"},"cell_type":"code","source":"model_dense_2 = dense_model_2()\nmodel_dense_2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8647769ac840d15de38114d9820586277591b2f1","scrolled":false,"_kg_hide-output":true},"cell_type":"code","source":"history_dense_2 = model_dense_2.fit(X_train, y_train, validation_data=(X_val,y_val_10), \n                                    epochs=50, batch_size=batchsize)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"06588fc9807fff9f3052853a356ef705ca33d4ca"},"cell_type":"code","source":"plot_history_loss_and_acc(history_dense_2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8fc16105c3463f70b16ede4ce2baa656ae066f5b"},"cell_type":"code","source":"pred_val_dense2 = model_dense_2.predict_classes(X_val)\nplot_confusion_matrix(y_val, pred_val_dense2)\nprint(classification_report(y_val, pred_val_dense2))\nacc_fc2 = accuracy_score(y_val, pred_val_dense2)\nprint(acc_fc2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6114bf8e74a07dd1ee491cec48c4a2108baec72b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df592740e70fa6b82ddd044002ed407016b9fe97"},"cell_type":"markdown","source":"### Keras: 3 hidden layers"},{"metadata":{"trusted":true,"_uuid":"5a7e7c7b7b79894988a3955eefc60a4610dff5a3"},"cell_type":"code","source":"def dense_model_3():\n    \n    model = Sequential()  \n    model.add(Dense(100, activation='relu', input_dim=784))\n    model.add(Dense(200, activation='relu')) \n    model.add(Dense(100, activation='relu')) \n    model.add(Dense(10, activation='softmax'))\n         \n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    \n    #model.compile(optimizer=RMSprop(lr=0.001),\n    #         loss='categorical_crossentropy',\n    #         metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"074df37093573fe1d0b82902608c1cdc60f46fbb"},"cell_type":"code","source":"model_dense_3 = dense_model_3()\nmodel_dense_3.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c84be17854c5a36bdaab946a6e78ad978f6733fa","_kg_hide-output":true},"cell_type":"code","source":"history_dense_3 = model_dense_3.fit(X_train, y_train, validation_data=(X_val,y_val_10), \n                                    epochs=50, batch_size=batchsize)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"abcb580d09c4f6df6b592a9eaf8c02306033fe73"},"cell_type":"code","source":"plot_history_loss_and_acc(history_dense_3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15639f762176cbb6d6d1ac69836355491ccb4b9d"},"cell_type":"code","source":"pred_val_dense3 = model_dense_3.predict_classes(X_val)\nplot_confusion_matrix(y_val, pred_val_dense3)\nprint(classification_report(y_val, pred_val_dense3))\nacc_fc3 = accuracy_score(y_val, pred_val_dense3)\nprint(acc_fc3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"738eaed5f252286a9369896e371683ec42b86813"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a38a7511b03011c453cc5f0f7ba35a77dfb04f0"},"cell_type":"markdown","source":"## 3.2 Keras : Convolutional Neural Networks, CNN"},{"metadata":{"trusted":true,"_uuid":"fe26d5c4ab3661208005e023026c623557df9e13"},"cell_type":"markdown","source":"Convnets:  \n[Udacity](https://www.youtube.com/watch?v=jajksuQW4mc) "},{"metadata":{"trusted":true,"_uuid":"373be8132cf3e4b40c3e9e9a42b10c69caa9c472"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5233ad4dea6af705b60075c491ef21cf2bf7403"},"cell_type":"markdown","source":"### **features : reshaping 1d vector to 2d images**\n\n(784) --> (28,28,1)"},{"metadata":{"trusted":true,"_uuid":"1f740d4ed4a06f9e1e2a13443e4d4075ccbcccb6"},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1426af50bb6ed93408b9350ebd3acdf98c52bc36"},"cell_type":"code","source":"X_train = X_train.values.reshape(X_train.shape[0], img_rows, img_cols, 1)\nX_val = X_val.values.reshape(X_val.shape[0], img_rows, img_cols, 1)\n\ninput_shape = (img_rows, img_cols, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5b6926f5a25cedfc35862367b608524dad7c05c"},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a76db0b3da04ff3b7358cca60a6eeb155a81a2c3"},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26cd252615c013d2d560ea54a277504a0f59b42f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c64474f903a5389ea7c385156076ae532ca73105"},"cell_type":"markdown","source":"## Keras: CNN model 1\nConv2D (32, (3, 3))  \nConv2D (64, (3, 3))  \nPooling2D (2,2)  \nDropout (0.25)\nFlatten  \nDense(128, relu)  \nDropout (0.5)  \nDense(10, softmax)"},{"metadata":{"trusted":true,"_uuid":"4d4074066d9b6f38380dfaded0b43151f75406c8"},"cell_type":"code","source":"batchsize = 128\nepochs = 12","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"df839fc323bb17df0815fc012fec140db1689059"},"cell_type":"code","source":"activation = 'relu'\nadadelta = Adadelta()\nloss = categorical_crossentropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad9d406a087d879d27fcf33dc1a7412beb5044d6"},"cell_type":"code","source":"def cnn_model_1(activation):\n    \n    model = Sequential()\n    \n    model.add(Conv2D(32, kernel_size=(3, 3), activation=activation, input_shape=input_shape)) \n    \n    model.add(Conv2D(64, (3, 3), activation=activation))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(Dropout(0.25))\n \n    model.add(Flatten())\n\n    model.add(Dense(128, activation=activation))\n    model.add(Dropout(0.5))\n\n    model.add(Dense(10, activation='softmax'))\n\n    model.compile(loss=loss, optimizer=adadelta, metrics=['accuracy'])\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4a6637f192b18d3fc1b887f88d94398a4dca431"},"cell_type":"code","source":"model_cnn_1 = cnn_model_1(activation)\nmodel_cnn_1.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"753a494ae46339d1f9f1973e04cd46dbca3026a9","_kg_hide-output":true},"cell_type":"code","source":"#model_cnn_1.fit(X_train, y_train, batch_size=batchsize, epochs=epochs, verbose=1)\nhistory_cnn_1 = model_cnn_1.fit(X_train, y_train, validation_data=(X_val,y_val_10), \n                                   epochs=epochs, batch_size=batchsize, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ba98c29b29be32ee0370797c8ac77366228bb87"},"cell_type":"code","source":"plot_history_loss_and_acc(history_cnn_1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d31fbbbe5406d375029051c857fc903b101fcf4"},"cell_type":"code","source":"pred_val_cnn1 = model_cnn_1.predict_classes(X_val)\nplot_confusion_matrix(y_val, pred_val_cnn1)\nprint(classification_report(y_val, pred_val_cnn1))\nacc_cnn1 = accuracy_score(y_val, pred_val_cnn1)\nprint(acc_cnn1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3e41806cc20e1cf1eea40f0c75422888a393f50"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37dfe5415315db2a03f2225bff9386aec5238cfb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de19d7786fd1cff44b3122da8bb283b62ba49f16"},"cell_type":"markdown","source":"### Keras: CNN model 2  \nConv2D (32, (3, 3))  \nPooling2D (2,2)  \nConv2D (32, (3, 3))  \nPooling2D (2,2)  \nFlatten  \nDense(256, relu)  \nDense(10, softmax)  "},{"metadata":{"trusted":true,"_uuid":"7ccfc62ebf08b287f8a49515bcd015f7bc91c4ea"},"cell_type":"code","source":"batch_size=90\nepochs=30\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c79037260e74aba99f1676e64147d83c0357413"},"cell_type":"code","source":"def cnn_model_2(optimizer,loss):\n\n    model = Sequential()\n\n    model.add(Conv2D(32, (3, 3), padding = 'Same', activation=\"relu\", input_shape=input_shape ))\n    model.add(MaxPooling2D(pool_size = (2, 2)))\n\n    model.add(Conv2D(32, (3, 3), activation=\"relu\"))\n    model.add(MaxPooling2D(pool_size = (2, 2)))\n\n    model.add(Flatten())\n\n    model.add(Dense(256, activation=activation))\n    model.add(Dense(10, activation='softmax'))\n\n    model.compile(optimizer = optimizer, loss = loss, metrics = ['accuracy']) \n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"118c6e0a388e7ed8ea0d856675a54d71c313276e"},"cell_type":"code","source":"model_cnn_2 = cnn_model_2(adadelta, categorical_crossentropy)\nmodel_cnn_2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe6b203292cc967c3ad35e1199694f0ca3755312","_kg_hide-output":true},"cell_type":"code","source":"#model_cnn_2.fit(X_train, y_train, batch_size=batch_size, epochs=epochs)\nhistory_cnn_2 = model_cnn_2.fit(X_train, y_train, validation_data=(X_val,y_val_10), \n                                epochs=epochs, batch_size=batchsize, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"adbe3d7a391ec4eedfcba6c64410dcd24b94c74f"},"cell_type":"code","source":"plot_history_loss_and_acc(history_cnn_2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7060a5a43d925c3c66dd9b7a0ee2ca8215543c28"},"cell_type":"code","source":"pred_val_cnn2 = model_cnn_2.predict_classes(X_val)\nplot_confusion_matrix(y_val, pred_val_cnn2)\nprint(classification_report(y_val, pred_val_cnn2))\nacc_cnn2 = accuracy_score(y_val, pred_val_cnn2)\nprint(acc_cnn2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e6f3b90dccee24757d85e13cae01ea4ec774ac2c"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd46c1843dc3685487221202fcaa01187450cce9"},"cell_type":"markdown","source":"# Part 4: Predictions for test data"},{"metadata":{"trusted":true,"_uuid":"e5a965e3f8dfc40cd7a0893fb0bf93432cc7dd8d"},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/sample_submission.csv')\nif mode == \"edit\" :\n    X = X[:nr_samples//2]\n    y = y[:nr_samples//2]\n    X_test = X_test[:nr_samples//2]\n    sample_submission = sample_submission[:nr_samples//2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c2ab7b1b478d55cde8e72579d45a9f371cb7891"},"cell_type":"code","source":"print(X.shape)\nprint(y.shape)\nprint(X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bfc8459af6688c2adbc35245c476ba1addcab8ca"},"cell_type":"markdown","source":"## Fitting on all training data"},{"metadata":{"_uuid":"570c7aadcc59b2fb88c759a30bb9865a9f34794a"},"cell_type":"markdown","source":"**For the GridSearchCV studies on finding the best model parameters we fit the classifiers using cross validation.**  \n**This reduces the number of training examples because a portion of the data is used for validation.**  \n**We now fit the classifiers on the complete training set (42000 samples).**  \n**Then we use this new fit to make predictions for the test dataset (28000 samples).**"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Perceptron**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(GridCV_Perceptron.best_params_)\nGridCV_Perceptron.best_estimator_.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test_perc = GridCV_Perceptron.best_estimator_.predict(X_test)\nresult_perc = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_perc})\nresult_perc.to_csv(\"subm_perc.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Logistic Regression**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(GridCV_LR.best_params_)\nGridCV_LR.best_estimator_.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test_lr = GridCV_LR.best_estimator_.predict(X_test)\nresult_lr = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_lr})\nresult_lr.to_csv(\"subm_lr.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**KNN**"},{"metadata":{"trusted":true},"cell_type":"code","source":"clf_knn.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test_knn = clf_knn.predict(X_test)\nresult_knn = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_knn})\nresult_knn.to_csv(\"subm_knn.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06692ecfb698bee2fe81bededdb0c43ff0af9cd4"},"cell_type":"markdown","source":"**Random Forest**"},{"metadata":{"trusted":true,"_uuid":"f4b09aeeb55110c1a1613c5574d2e1e4965e4e49"},"cell_type":"code","source":"print(GridCV_RF.best_params_)\nGridCV_RF.best_estimator_.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b9424506c96c1489e6a6edc398a1a4d05138ea4"},"cell_type":"code","source":"pred_test_rf = GridCV_RF.best_estimator_.predict(X_test)\nresult_rf = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_rf})\nresult_rf.to_csv(\"subm_rf.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**SVM**"},{"metadata":{"trusted":true},"cell_type":"code","source":"clf_svm.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test_svm = clf_svm.predict(X_test)\nresult_svm = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_svm})\nresult_svm.to_csv(\"subm_svm.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1bdcfe075e2caea29dc6714161f8e7306dbea8e9"},"cell_type":"markdown","source":"**MLP**"},{"metadata":{"trusted":true,"_uuid":"6e75e8a0c00fd9bc664fc7a10721d51b483ee36e"},"cell_type":"code","source":"print(GridCV_MLP.best_params_)\nGridCV_MLP.best_estimator_.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"df1c0ac5208d098578e4341402c1c4cca459c9a2"},"cell_type":"code","source":"pred_test_mlp = GridCV_MLP.best_estimator_.predict(X_test)\nresult_mlp = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_mlp})\nresult_mlp.to_csv(\"subm_mlp.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba68604039f0621a649cc30b0762a20e2c568577"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = to_categorical(y, 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**dense_1**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_dense_1.fit(X,y)\npred_test_fc1 = model_dense_1.predict_classes(X_test)\nresult_fc1 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_fc1})\nresult_fc1.to_csv(\"dense_1.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**dense_2**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_dense_2.fit(X,y)\npred_test_fc2 = model_dense_2.predict_classes(X_test)\nresult_fc2 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_fc2})\nresult_fc2.to_csv(\"dense_2.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**dense_3**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_dense_3.fit(X,y)\npred_test_fc3 = model_dense_3.predict_classes(X_test)\nresult_fc3 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_fc3})\nresult_fc3.to_csv(\"dense_3.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**reshape for CNN**"},{"metadata":{"trusted":true,"_uuid":"f837e052e9122a35c40f234537a7e78c326141ee"},"cell_type":"code","source":"X = X.values.reshape(X.shape[0], img_rows, img_cols, 1)\nX_test = X_test.values.reshape(X_test.shape[0], img_rows, img_cols, 1)\n#y = to_categorical(y, 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d707dbea94924b801baec99f68f4425855a0835"},"cell_type":"markdown","source":"**CNN 1**"},{"metadata":{"trusted":true,"_uuid":"84ce81d5aecc157d902bd81f715dea44a32ad8dd"},"cell_type":"code","source":"batchsize = 128\nepochs = 12\nmodel_cnn_1 = cnn_model_1('relu')\nmodel_cnn_1.fit(X, y, epochs=epochs, batch_size=batchsize, verbose=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5713f7ef07f85800b6869769c4f44f96cc917f4f"},"cell_type":"code","source":"pred_test_cnn_1 = model_cnn_1.predict(X_test)\npred_test_cnn_1 = np.argmax(pred_test_cnn_1,axis=1)\nresult_cnn_1 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_cnn_1})\nresult_cnn_1.to_csv(\"subm_cnn_1.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c67a69ccc0b8ae7036c90bafbd83acc4521b92a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39798af2742ea7a448937b104d4f2574a5d6929e"},"cell_type":"markdown","source":"**CNN 2**"},{"metadata":{"trusted":true,"_uuid":"d592f348e38a7f1fd73b54b03cae5a9988440452"},"cell_type":"code","source":"batch_size=90\nepochs=30\nmodel_cnn_2 = cnn_model_2(adadelta, categorical_crossentropy)\nmodel_cnn_2.fit(X, y, epochs=epochs, batch_size=batchsize, verbose=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a398cc85d02a18befc6761cbb12ed7b6aaa7c508"},"cell_type":"code","source":"pred_test_cnn_2 = model_cnn_2.predict(X_test)\npred_test_cnn_2 = np.argmax(pred_test_cnn_2,axis=1)\nresult_cnn_2 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_cnn_2})\nresult_cnn_2.to_csv(\"subm_cnn_2_adadelta.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7300c21c094dfa58536e8a2edd919b3629bb941d"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Correlation of prediction results"},{"metadata":{"trusted":true,"_uuid":"cdb3d41e7a3b1478e048fe8434ca0661e44bc28a"},"cell_type":"code","source":"predictions = {'PERC': pred_test_perc, 'LR': pred_test_lr, 'KNN': pred_test_knn, \n               'RF': pred_test_rf, 'SVM': pred_test_svm, 'MLP': pred_test_mlp, \n               'DENSE1': pred_test_fc1, 'DENSE2': pred_test_fc2, 'DENSE3': pred_test_fc3, \n               'CNN1': pred_test_cnn_1, 'CNN2': pred_test_cnn_2}\ndf_predictions = pd.DataFrame(data=predictions) \ndf_predictions.corr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc7d2ec9be26dfcab932eecf34cb6351c84dd85a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 5: Comparing classifier performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"list_classifiers = ['PERC','LR','KNN','RF','SVM',\n                    'MLP','DENSE1','DENSE2','DENSE3',\n                    'CNN1','CNN2']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validation scores"},{"metadata":{},"cell_type":"markdown","source":"from validation data"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"11f2b5a8c4d13361b317a39b2187ac18965397a8"},"cell_type":"code","source":"val_scores = [acc_perc, acc_lr, acc_knn, acc_rf, \n               acc_svm, acc_mlp, acc_fc1, acc_fc2, \n               acc_fc3, acc_cnn1, acc_cnn2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5fbb8af1cfba8eccd1338e9befa6e61e6ca355d"},"cell_type":"markdown","source":"## Test scores"},{"metadata":{},"cell_type":"markdown","source":"from submission "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"score_perc  = 0.88057\nscore_lr    = 0.88700\nscore_knn   = 0.96557\nscore_rf    = 0.96028\nscore_svm   = 0.98100\nscore_mlp   = 0.96985\n\nscore_dns_1  = 0.95971 \nscore_dns_2  = 0.96228      \nscore_dns_3  = 0.96128\nscore_cnn_1  = 0.98928\nscore_cnn_2  = 0.99028","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41dc9e6e79e4aaf747f429267212bf566d57ae9e"},"cell_type":"code","source":"test_scores = [score_perc, score_lr, score_knn, score_rf, score_svm, score_mlp,\n               score_dns_1, score_dns_2, score_dns_3, score_cnn_1, score_cnn_2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff293aebece64b3962a973cc5f2155995d99abf7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2ef32f348371da4c987e5812d08ca41144930d4"},"cell_type":"code","source":"trace1 = go.Scatter(x = list_classifiers, y = val_scores,\n                   name=\"Validation\", text = list_classifiers)\ntrace2 = go.Scatter(x = list_classifiers, y = test_scores,\n                   name=\"Submission\", text = list_classifiers)\n\ndata = [trace1, trace2]\n\nlayout = dict(title = \"Validation and Submission Scores\", \n              xaxis=dict(ticklen=10, zeroline= False),\n              yaxis=dict(title = \"Accuracy\", side='left', ticklen=10,),                                  \n              legend=dict(orientation=\"v\", x=1.05, y=1.0),\n              autosize=False, width=750, height=500,\n              )\n\nfig = dict(data = data, layout = layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusions on classifier performance"},{"metadata":{},"cell_type":"markdown","source":"* Best accuracy around 99% is obtained with CNN1 and CNN2.  \n* SVC also has very good accuracy with 98%  \n* Except for Perceptron and Logistic regression, all classifiers get accuarcy similar to humans (above 95%)  \n* Fully connected NNs (dense 1,2,3) show most overfitting (validation score much larger than test score)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Part 6: Different Optimizers"},{"metadata":{},"cell_type":"markdown","source":"Adadelta"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn_2.optimizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"RMSprop"},{"metadata":{"trusted":true,"_uuid":"720d7a152db9aa4c0bac2bcbd829d3b4f6cb27ec"},"cell_type":"code","source":"model_cnn_2_rmsprop = cnn_model_2(RMSprop(), categorical_crossentropy)\nmodel_cnn_2_rmsprop.optimizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn_2.fit(X, y, epochs=epochs, batch_size=batchsize, verbose=0)\npred_test_cnn_2 = model_cnn_2.predict(X_test)\npred_test_cnn_2 = np.argmax(pred_test_cnn_2,axis=1)\nresult_cnn_2 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_cnn_2})\nresult_cnn_2.to_csv(\"subm_cnn_2_rmsprop.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Adam"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn_2_adam = cnn_model_2(Adam(), categorical_crossentropy)\nmodel_cnn_2_adam.optimizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn_2.fit(X, y, epochs=epochs, batch_size=batchsize, verbose=0)\npred_test_cnn_2 = model_cnn_2.predict(X_test)\npred_test_cnn_2 = np.argmax(pred_test_cnn_2,axis=1)\nresult_cnn_2 = pd.DataFrame({'ImageId':sample_submission.ImageId, 'Label':pred_test_cnn_2})\nresult_cnn_2.to_csv(\"subm_cnn_2_adam.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 7: Investigating false predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"arr_y_val = y_val.values\nfalse_cnn2 = pred_val_cnn2 != arr_y_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(5, 5, sharex=True, sharey=True, figsize=(10,12))\naxs = axs.flatten()\nfor i, n  in enumerate(false_cnn2[:25]):\n    im = X_val[false_cnn2][i,:,:,0]\n    axs[i].imshow(im, cmap=plt.get_cmap('gray'))\n    title = (\"predicted: \" + str(pred_val_cnn2[false_cnn2][i]) + \n            \"\\n\" + \"true: \" + str(arr_y_val[false_cnn2][i]) )\n    axs[i].set_title(title)\nplt.tight_layout()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}