{"cells":[{"metadata":{},"cell_type":"markdown","source":" - [Introduction](#section-one)\n - [Data exploration](#section-two)\n - [The model](#section-three)\n - [What went wrong/right](#section-four)\n\n\n## Please upvote if you found this interesting, and comment if you have any improvements :)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-one\"></a>\n## Introduction\n\nI am very much a begginer when it comes to data science, but I am really enjoying the learning process - so far anyway. I have been through many tutorials and I really need to start doing some competitions so I can actually learn. So here I am!\n\nOften I find sarting the hardest part; this is especially true with deep learning I am finding. Classification and regression competitions seem much easier. \n\nSo, here I am giving it a go in what I hope is a good starting point for a deep learning competition.\n\n### Progression\n1. achieved 0.95120\n2. added learning rate annealer and increased epochs by 10, achieved 0.97380\n3. Changed from Sequential to functional - no change to score\n    - Decreased dropout slightly.\n    - Increased image zoom and height and width_shift_range from 0.1 to 0.25\n4. added another dense layer and another convolutional layer, reduced the kernel size in the 1st layer. Also increased the test size by 10% - achieved 0.97620\n5. Completely changed the model to this specification https://www.kaggle.com/anshumandec94/6-layer-conv-nn-using-adam. I still have no idea where people get the ideas for the design of these models, if you know please comment below. achieved 0.97780\n6. changed optimizer from RMSprop to Adam 0.975\n7. Edited Adam optimizer; increased epochs; changed learning rate to 0.33 from 0.5; reduced test size. \n8. incorporated extra val set into training\n9. Increased epochs from 50 to 65\n10. changed learning rate and batch size\n11. removed digits dataset from training.\n12. added model checkpoint\n13. changed batch size to 100\n#### Completely changed my approach"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Import Packages\n\nimport random\nimport keras\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport matplotlib.patches as mpatches\n\nfrom skimage.filters import threshold_otsu\nfrom skimage.segmentation import clear_border\nfrom skimage.measure import label, regionprops\nfrom skimage.morphology import closing, square\nfrom skimage.color import label2rgb\nfrom math import sqrt\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential, Model\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, Input, BatchNormalization\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.callbacks import ModelCheckpoint","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-two\"></a>\n## Data Exploration"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('/kaggle/input/Kannada-MNIST/test.csv')\n#val_set = pd.read_csv(\"../input/Kannada-MNIST/Dig-MNIST.csv\")\n#train = pd.concat([train, val_set], axis=0)\ny_train = train['label']\nX = train.drop(['label'],axis=1)\ndel train\nId = test['id'] # save this for the submission\ntest = test.drop(['id'],axis=1) #","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Much fewer samples in the test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X.values[3].reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting, I cannot distinguish what this is, I hope a machine can."},{"metadata":{"trusted":true},"cell_type":"code","source":"# code for this plot taken from https://www.kaggle.com/josephvm/kannada-with-pytorch\n\nfig, ax = plt.subplots(nrows=10, ncols=10, figsize=(15,15))\n\n# I know these for loops look weird, but this way num_i is only computed once for each class\nfor i in range(10): # Column by column\n    num_i = X[y_train == i]\n    ax[0][i].set_title(i)\n    for j in range(10): # Row by row\n        ax[j][i].axis('off')\n        ax[j][i].imshow(num_i.iloc[j, :].to_numpy().astype(np.uint8).reshape(28, 28))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In hindsight, if I were developing a number system, I would make sure that the symbols are as different as possible - 3 and 7 are basically the same, as are 6 and 9! Equations in kannada are probably quite interesting"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nlab, val = np.unique(y_train,return_counts=True)\nfig, ax = plt.subplots(nrows=1, ncols=1, figsize=(15,7))\nsns.barplot(lab,val)\nplt.title('Equal number of samples in all classes in the training data')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hopefully the test set is as equally spread."},{"metadata":{"trusted":true},"cell_type":"code","source":"first_1000 = y_train[:10]\nnp.unique(first_1000, return_counts=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the numbers repeat every 10, that is usefull! let's see if the number of pixels that are non '0' are informative to the label? I suspect some numbers require more 'ink' than others"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels =[]\nfrequencies = []\n\n\nfor i in range(len(y_train)):\n    lab, freq = str(y_train[i]), len([n for n in X.values[i] if n > 0])\n    labels.append(lab)\n    frequencies.append(freq)\n    \n    \ndata = {'Labels':labels, 'Frequencies':frequencies}\n\ndf = pd.DataFrame(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# mean number of pixels per label\ndf.groupby('Labels').mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(15,10))\nsns.boxplot(x = 'Labels', y = 'Frequencies', data = df)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks like some numbers generally require more pixels e.g 8, but the variance is high as peoples handwriting varies quite a lot I suspect. Oh, well it was worth a try"},{"metadata":{"trusted":true},"cell_type":"code","source":"# apply threshold\nimage = X.values[3].reshape(28,28)\nthresh = threshold_otsu(image)\nbw = closing(image > thresh, square(3))\n\n# label image regions\nlabel_image = label(bw)\nimage_label_overlay = label2rgb(label_image, image=image)\n\nfig, ax = plt.subplots(figsize=(10, 6))\nax.imshow(image_label_overlay)\n\nfor region in regionprops(label_image):\n    # take regions with large enough areas\n    if region.area >= 30:\n        # draw rectangle around segmented coins\n        minr, minc, maxr, maxc = region.bbox\n        rect = mpatches.Rectangle((minc, minr), maxc - minc, maxr - minr,\n                                  fill=False, edgecolor='green', linewidth=2)\n        ax.add_patch(rect)\nplt.text(1,1, f'Width: {maxc-minc} Height: {maxr -minr} Diagonal: {round(sqrt((((maxc-minc)**2) + (maxr-minr)**2)),2)}', color = 'w')\nax.set_axis_off()\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def measurements(images):\n    \n    widths = []\n    heights = []\n    diags = []\n    \n    for i in range(len(images)):\n        # apply threshold\n        image = images[i].reshape(28,28)\n        thresh = threshold_otsu(image)\n        bw = closing(image > thresh, square(3))\n\n        # label image regions\n        label_image = label(bw)\n        image_label_overlay = label2rgb(label_image, image=image)\n\n        for region in regionprops(label_image):\n            # take regions with large enough areas\n            if region.area >= 30:\n                # draw rectangle around segmented coins\n                minr, minc, maxr, maxc = region.bbox\n                \n        widths.append(maxc-minc)\n        heights.append(maxr -minr)\n        diags.append(sqrt((((maxc-minc)**2) + (maxr-minr)**2)))\n                     \n    return widths, heights, diags\n                     \n\n\nwidths, heights, diags = measurements(X.values)\n                     \ndata = {'Labels':labels, 'diagonals':diags, 'widths':widths, 'heights':heights, 'Area':frequencies} #area is not actually area, but the number of pixels\n\ndf = pd.DataFrame(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(15,10))\nsns.boxplot(x = 'Labels', y = 'diagonals', data = df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(15,10))\nsns.boxplot(x = 'Labels', y = 'heights', data = df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(15,10))\nsns.boxplot(x = 'Labels', y = 'widths', data = df)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nSo, what have I learnt? Well, the height is very similar for each digit. This is strange to me as I would have though handwriting would vary alot more than that. Perhaps there is a logic error in my code? This is also true in the 'digits' dataset.\n\nthe width of the digit is much more variable but still, its not a huge difference.\n\nLet's see if a decision tree can tell the difference given these data\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nfrom sklearn.ensemble import RandomForestClassifier\ntest_y = df['Labels']\n\ntest_x = df\ntest_x.drop('Labels', axis =1, inplace = True)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train1, X_val1, y_train1, y_val1 = train_test_split(test_x, test_y, test_size=0.1, random_state=1337)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf_rfc = RandomForestClassifier(n_estimators = 100)\nclf_rfc.fit(X_train1,y_train1)\n\n\n\nfrom sklearn.metrics import accuracy_score\n\naccuracy_score(clf_rfc.predict(X_val1), y_val1)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Better than that of the mnist data!, but still terrible. To see the comparison look here https://www.kaggle.com/jakelj/mnist-digits-novice-to-master"},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-three\"></a>\n## Building the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# re-shaping the data so that keras can use it, this is something that trips me up every time\n\nX = X.values.reshape(X.shape[0], 28, 28,1)\ntest = test.values.reshape(test.shape[0], 28, 28,1)\n\n# Scale the data\nX = X / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This modifies some images slightly, I have seen this in a few tutorials and it usually makes the model more accurate. As a beginner, it goes without saying I don't fully understand all the parameters\n\ndatagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.25, # Randomly zoom image \n        width_shift_range=0.25,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.25,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\n\nvalid_datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.25, # Randomly zoom image \n        width_shift_range=0.25,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.25,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train,num_classes=10) # the labels need to be one-hot encoded, this is something else I usually forget","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# same as https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits\n\nlearning_rate_reduction = ReduceLROnPlateau(monitor='accuracy', \n                                            patience=3, \n                                            verbose=10, \n                                            factor=0.33, \n                                            min_lr=0.00001)\ncheckpoint=ModelCheckpoint('bestweights.md5', monitor='accuracy', verbose=1, save_best_only=True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have decided that I will try and make as simple model as possible, as it is all well and good modifying someone elses model, but you do not learn as much that way, so from now on I will aim for as few layers as possible"},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 40\ninitial_learningrate=2e-3\nbatch_size = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X, y_train, test_size=0.1, random_state=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\ndef build_model1(input_shape=(28, 28, 1), classes = 10):\n    \n    activation = 'relu'\n    padding = 'same'\n    gamma_initializer = 'uniform'\n    \n    input_layer = Input(shape=input_shape)\n    \n    hidden=Conv2D(32, (11,11), strides=1, padding =padding, activation = activation, name = 'conv1')(input_layer)\n    hidden=BatchNormalization()(hidden)\n    hidden=Conv2D(32, (11,11), strides=1, padding =padding, activation = activation, name = 'conv2')(hidden)\n    hidden=BatchNormalization()(hidden)\n    hidden=Conv2D(32, (5,5), strides=1, padding =padding, activation = activation, name = 'conv3')(hidden)\n    hidden=BatchNormalization()(hidden)\n    hidden=MaxPool2D()(hidden)\n    hidden=Dropout(0.35)(hidden)\n    \n    hidden=Conv2D(64, (3,3), strides=1, padding =padding, activation = activation, name = 'conv4')(input_layer)\n    hidden=BatchNormalization()(hidden)\n    hidden=Conv2D(64, (3,3), strides=1, padding =padding, activation = activation, name = 'conv5')(hidden)\n    hidden=BatchNormalization()(hidden)\n    hidden=Conv2D(64, (5,5), strides=1, padding =padding, activation = activation, name = 'conv6')(hidden)\n    hidden=BatchNormalization()(hidden)\n    hidden=MaxPool2D()(hidden)\n    hidden=Dropout(0.3)(hidden)\n    \n\n    hidden=Flatten()(hidden)\n    hidden=Dense(250,activation = activation, name=\"Dense1\")(hidden)\n    hidden=Dropout(0.3)(hidden)\n    \n    output = Dense(classes, activation = \"softmax\")(hidden)\n    \n    model = Model(inputs=input_layer, outputs=output)\n    \n    return model\n\noptimizer = Adam(learning_rate=initial_learningrate, beta_1=0.9, beta_2=0.999, amsgrad=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The main change to this model, other than the amount of layers, is batch normalization.\n\nFrom what I have read and, let's face it, watched on YouTube, the reason why this is done is to keep the data on a similar scale and prevent gradient explosions. Apparently, non-normalised data can decrease training speed. It helps to stop the weights of the model becoming inbalanced?\n\nhere is a usefull if you're interested https://www.youtube.com/watch?v=dXB-KQYkzNU\n\nOne other thing I noticed about this model is that the dropout is smaller than I have seen in others. My guess is that becuase there are more layers the dropout is smaller in each layer, but overall it is similar? The combined number in this model is 0.75\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n# Compile the model\n\n\nmodel = build_model1(input_shape=(28, 28, 1), classes = 10)\n\n\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen.fit(X_train)\ndatagen.fit(X_val)\n\ncallbacks = [checkpoint,learning_rate_reduction]\nhistory = model.fit_generator(datagen.flow(X_train,y_train, batch_size=batch_size ),\n                              epochs = epochs,\n                              validation_data=valid_datagen.flow(X_val,y_val),\n                              verbose = 1,\n                            callbacks = callbacks)\n\n# On the first attempt I forgot to add the 'learning_rate_reduction'\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Keeping score\n\nIn this area I am going t keep rack of the scores I got for various models over ten epochs.\nThe way of explaining the model is as in https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist:\n\nBasically model 1 means convolutional layer with 32 filters of size (3,3), then batch normalization, then dropout, then a pooling layer, then dense layer with 100 neurons, the 10 for out put. and finally the 'Accuracy and Val accuracy\n\n- model 1: 32C3 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9141,0.9522]\n- model 2: 32C5 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9378,0.9622]\n- model 3: 64C3 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9196, 0.9552]\n- model 4: 64C5 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9404, 0.9692]\n- model 5: 128C3 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9177, 0.9532]\n- model 6: 128C5 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9401, 0.9700] \n\nit looks like model 4 is the winner lets take this one forwards and develop it, and a 5x5 is better than 3x3 in this case. but what of 7x7?\n\n- model 7: 64C7 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9476, 0.9682]\n\nInteresting 9x9?\n- model 8: 64C9 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9506, 0.9708]\n\n\nummm...11x11\n\n- model 9: 64C11 - B - P2 - D(0.2)- 100 - D(0.2) - 10 [0.9516, 0.9733]\nwasn't much change after this\n- model 10: 64C11 - B - P2  - 32C9 - D(0.2)- 100 - D(0.2) - 10 [0.9788, 0.9802]\n\n- model 11: 64C11 - B - P2  - 32C9 - B - P2 16C5 - D(0.2)- 100 - D(0.2) - 10 [0.9819, 0.9827]\n- model 12: 64C11 - B - P2  - 32C9 - B - P2 32C5 - D(0.2)- 100 - D(0.2) - 10 [0.9834, 0.9817]\n\nFrom this point I suspect we are squeezing the last few % from the model and the differences over 10 epochs may not be enough to discrimate so I am goin to switch to 20\n\n- model 13: 64C11 - B - P2  - 32C9 - B - P2 32C5 - D(0.2)- 100 - D(0.2) - 10 [0.9870, 0.9882]. I like how similar the validation accuracy is to the accuracy. I will check how this compares to the eaderboard at this point. On the leaderboard this achieved 0.9770\n\n- model 14: 64C11 - B - P2  - 32C9 - B - P2 65C5 - D(0.2)- 250 - D(0.2) - 10 [0.9862, 9870]\n\n- model 15: 64C11 - B - P2  - 32C9 - B - P2 32C5 - D(0.2) - B - P2 32C3 - D(0.2) - 250 - D(0.2) - 10 [0.9870, 0.9882]\n \nformula [(W−K+2P)/S]+1 for calculating image size after convolution\n\n    W is the input volume \n    K is the Kernel size \n    P is the padding \n    S is the stride \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('bestweights.md5')\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-four\"></a>\n## How did we do?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1)\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best')\n\nax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\n\n# used the code from https://www.kaggle.com/shahules/indian-way-to-learn-cnn to create this\n\ny_pre_test=model.predict(X_val)\ny_pre_test=np.argmax(y_pre_test,axis=1)\ny_test=np.argmax(y_val,axis=1)\n\nconf=confusion_matrix(y_test,y_pre_test)\nconf=pd.DataFrame(conf,index=range(0,10),columns=range(0,10))\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.heatmap(conf, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('out of {} samples, we got {} incorrect'.format(len(X_train), round(len(X_train) - history.history['accuracy'][-1] * len(X_train))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#code taken from here\n\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(y_val,axis = 1) \n\n# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\ndef display_errors(errors_index,img_errors,pred_errors, obs_errors):\n    \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n    n = 0\n    nrows = 3\n    ncols = 3\n    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True,figsize = (10,10))\n    fig.tight_layout()\n    for row in range(nrows):\n        for col in range(ncols):\n            error = errors_index[n]\n            ax[row,col].imshow((img_errors[error]).reshape((28,28)))\n            ax[row,col].set_title(\"Predicted label :{}\\nTrue label :{}\".format(pred_errors[error],obs_errors[error]))\n            n += 1\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[-18:-9]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now to make our predictions and submit!"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### This made me stumble when I changed\n\nWith sequential there is a function to get the classes, but this is not true with the functional API. You get an array with the models probabilities for each class. Therefore you have to take the highest value from the array and get the corresponding class."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(test[0].reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions[0] # looks like the 4 element is what the model thinks it is and it is over 99.99 % sure","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\npredictions = predictions.argmax(axis = -1)\npredictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({ 'id': Id,\n                            'label': predictions })\nsubmission.to_csv(path_or_buf =\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#keras.backend.clear_session()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}