{"cells":[{"metadata":{},"cell_type":"markdown","source":" - [Introduction](#section-one)\n - [Data exploration](#section-two)\n - [The model](#section-three)\n - [What went wrong/right](#section-four)\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-one\"></a>\n## Introduction\n\nI am very much a begginer when it comes to data science, but I am really enjoying the learning process - so far anyway. I have been through many tutorials and I really need to start doing some competitions so I can actually learn. So here I am!\n\nOften I find sarting the hardest part; this is especially true with deep learning I am finding. Classification and regression competitions seem much easier. \n\nSo, here I am giving it a go in what I hope is a good starting point for a deep learning competition.\n\n### Progression\n1. achieved 0.95120\n2. added learning rate annealer and increased epochs by 10, achieved 0.97380\n3. Changed from Sequential to functional - no change to score\n    - Decreased dropout slightly.\n    - Increased image zoom and height and width_shift_range from 0.1 to 0.25\n4. added another dense layer and another convolutional layer, reduced the kernel size in the 1st layer. Also increased the test size by 10% - achieved 0.97620\n5. Completely changed the model to this specification https://www.kaggle.com/anshumandec94/6-layer-conv-nn-using-adam. I still have no idea where people get the ideas for the design of these models, if you know please comment below. achieved 0.97780\n6. changed optimizer from RMSprop to Adam 0.975\n7. Edited Adam optimizer; increased epochs; changed learning rate to 0.33 from 0.5; reduced test size. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Import Packages\n\nimport random\nimport keras\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential, Model\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, Input, BatchNormalization\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-two\"></a>\n## Data Exploration"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('/kaggle/input/Kannada-MNIST/test.csv')\ny_train = train['label']\nX = train.drop(['label'],axis=1)\ndel train\nId = test['id'] # save this for the submission\ntest = test.drop(['id'],axis=1) #","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Much fewer samples in the test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X.values[3].reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting, I cannot distinguish what this is, I hope a machine can."},{"metadata":{"trusted":true},"cell_type":"code","source":"# code for this plot taken from https://www.kaggle.com/josephvm/kannada-with-pytorch\n\nfig, ax = plt.subplots(nrows=10, ncols=10, figsize=(15,15))\n\n# I know these for loops look weird, but this way num_i is only computed once for each class\nfor i in range(10): # Column by column\n    num_i = X[y_train == i]\n    ax[0][i].set_title(i)\n    for j in range(10): # Row by row\n        ax[j][i].axis('off')\n        ax[j][i].imshow(num_i.iloc[j, :].to_numpy().astype(np.uint8).reshape(28, 28))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In hindsight, if I were developing a number system, I would make sure that the symbols are as different as possible - 3 and 7 are basically the same, as are 6 and 9! Equations in kannada are probably quite interesting"},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-three\"></a>\n## Building the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# re-shaping the data so that keras can use it, this is something that trips me up every time\n\nX = X.values.reshape(X.shape[0], 28, 28,1)\ntest = test.values.reshape(test.shape[0], 28, 28,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This modifies some images slightly, I have seen this in a few tutorials and it usually makes the model more accurate. As a beginner, it goes without saying I don't fully understand all the parameters\n\ndatagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.25, # Randomly zoom image \n        width_shift_range=0.25,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.25,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train,num_classes=10) # the labels need to be one-hot encoded, this is something else I usually forget","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# this model was taken from https://www.kaggle.com/anshumandec94/6-layer-conv-nn-using-adam\n\ndef build_model(input_shape=(28, 28, 1), classes = 10):\n    \n    activation = 'relu'\n    padding = 'same'\n    gamma_initializer = 'uniform'\n    \n    input_layer = Input(shape=input_shape)\n    hidden=Conv2D(16, (3,3), strides=1, padding=padding, activation = activation, name=\"conv1\")(input_layer)\n    hidden=BatchNormalization(axis =1, momentum=0.1, epsilon=1e-5, gamma_initializer=gamma_initializer, name=\"batch1\")(hidden)\n    hidden=Dropout(0.1)(hidden)\n    \n    hidden=Conv2D(32, (3,3), strides=1, padding=padding,activation = activation, name=\"conv2\")(hidden)\n    hidden=BatchNormalization(axis =1,momentum=0.15, epsilon=1e-5, gamma_initializer=gamma_initializer, name=\"batch2\")(hidden)\n    hidden=Dropout(0.15)(hidden)\n    hidden=MaxPool2D(pool_size=2, strides=2, padding=padding, name=\"max2\")(hidden)\n    \n    hidden=Conv2D(64, (5,5), strides=1, padding =padding, activation = activation,  name=\"conv3\")(hidden)\n    hidden=BatchNormalization(axis =1,momentum=0.17, epsilon=1e-5, gamma_initializer=gamma_initializer, name=\"batch3\")(hidden)\n    hidden=MaxPool2D(pool_size=2, strides=2, padding=\"same\", name=\"max3\")(hidden)\n    \n    hidden=Conv2D(128, (5,5), strides=1, padding=padding, activation = activation, name=\"conv4\")(hidden)\n    hidden=BatchNormalization(axis =1,momentum=0.15, epsilon=1e-5, gamma_initializer=gamma_initializer, name=\"batch4\")(hidden)\n    hidden=Dropout(0.17)(hidden)\n    \n    hidden=Conv2D(64, (3,3), strides=1, padding=padding, activation = activation, name=\"conv5\")(hidden)\n    hidden=BatchNormalization(axis =1,momentum=0.15, epsilon=1e-5, gamma_initializer=gamma_initializer, name=\"batch5\")(hidden)\n    hidden=Dropout(0.2)(hidden)\n    \n    hidden=Conv2D(32, (3,3), strides=1, padding=padding, activation = activation, name=\"conv6\")(hidden)\n    hidden=BatchNormalization(axis =1,momentum=0.15, epsilon=1e-5, gamma_initializer=gamma_initializer, name=\"batch6\" )(hidden)\n    hidden=Dropout(0.05)(hidden)\n\n    hidden=Flatten()(hidden)\n    hidden=Dense(50,activation = activation, name=\"Dense1\")(hidden)\n    hidden=Dropout(0.05)(hidden)\n    hidden=Dense(25, activation = activation, name=\"Dense2\")(hidden)\n    hidden=Dropout(0.03)(hidden)\n    output = Dense(classes, activation = \"softmax\")(hidden)\n    \n    model = Model(inputs=input_layer, outputs=output)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# this model was taken from hans_on of the RNP4CV \n\ndef build_model_rubens(input_shape=(28, 28, 1), classes = 10):\n\n    model = Sequential()\n\n    model.add(Conv2D(input_shape=input_shape, \n                     filters=10, \n                     kernel_size=(5, 5), \n                     strides=(1, 1), \n                     padding='valid', \n                     activation='relu', \n                     kernel_initializer='glorot_uniform'))\n\n    model.add(MaxPool2D(pool_size=(2, 2), \n                           padding='valid'))\n\n    model.add(Dropout(rate=0.3))\n\n    model.add(Conv2D(filters=2, \n                     kernel_size=(10, 10), \n                     strides=(1, 1), \n                     padding='valid',\n                     activation='relu',\n                     kernel_initializer='glorot_uniform'))\n\n    model.add(MaxPool2D(pool_size=(2, 2), \n                           padding='valid'))\n\n    model.add(Dropout(rate=0.3))\n\n    model.add(Flatten())\n\n    model.add(Dense(units=4, \n                    activation='relu',\n                    use_bias=True,\n                    kernel_initializer='glorot_uniform', \n                    bias_initializer='zeros'))\n\n    model.add(Dense(units=classes, \n                    activation='softmax',\n                    use_bias=True, \n                    kernel_initializer='glorot_uniform', \n                    bias_initializer='zeros'))\n\n    model.summary()\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The main change to this model, other than the amount of layers, is batch normalization.\n\nFrom what I have read and, let's face it, watched on YouTube, the reason why this is done is to keep the data on a similar scale and prevent gradient explosions. Apparently, non-normalised data can decrease training speed. It helps to stop the weights of the model becoming inbalanced?\n\nhere is a usefull if you're interested https://www.youtube.com/watch?v=dXB-KQYkzNU\n\nOne other thing I noticed about this model is that the dropout is smaller than I have seen in others. My guess is that becuase there are more layers the dropout is smaller in each layer, but overall it is similar? The combined number in this model is 0.75\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the optimizer\noptimizer = Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)\n\n# Compile the model\nmodel = build_model(input_shape=(28, 28, 1), classes = 10)\n#model = build_model_rubens(input_shape=(28, 28, 1), classes = 10)\n\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy', \n                                            patience=3, \n                                            verbose=10, \n                                            factor=0.33, \n                                            min_lr=0.00001)\n# this could use some tuning, i don't think it needs to drop off so sharply.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 50 \nbatch_size = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X, y_train, test_size=0.1, random_state=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(datagen.flow(X_train,y_train, batch_size=batch_size ),\n                              epochs = epochs,\n                              validation_data = (X_val,y_val),\n                              verbose = 1,\n                              steps_per_epoch = X_train.shape[0] // batch_size,\n                             callbacks = [learning_rate_reduction])\n\n# On the first attempt I forgot to add the 'learning_rate_reduction'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()\nfrom keras.utils.vis_utils import plot_model\nplot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"testing dropout\n\n- 10 epochs o.g acc = 0.9765\n- all dropout at the end = 0.7684, but val accuracy was still 0.989\n- with 0.4 on each layer apart from the last 0.9685 and val of 0.9899\n- 0.1 on each layer with 0.4 on the two dense layers 0.9535 and val of 0.9903\n\nin summary -- is there any point on the valdiation set?"},{"metadata":{},"cell_type":"markdown","source":"<a name=\"section-four\"></a>\n## How did we do?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1)\n\nax[0].set_title('Loss function') # adding title to subplot\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best')\n\nax[1].set_title('Accuracy')  # adding title to subplot\nax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nlegend = ax[1].legend(loc='best')\n\nfig.tight_layout()   # adjust space between subplots\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\n\n# used the code from https://www.kaggle.com/shahules/indian-way-to-learn-cnn to create this\n\ny_pre_test=model.predict(X_val)\ny_pre_test=np.argmax(y_pre_test,axis=1)\ny_test=np.argmax(y_val,axis=1)\n\nconf=confusion_matrix(y_test,y_pre_test)\nconf=pd.DataFrame(conf,index=range(0,10),columns=range(0,10))\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.heatmap(conf, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('out of {} samples, we got {} incorrect'.format(len(X_train), round(len(X_train) - history.history['accuracy'][-1] * len(X_train))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# code taken from https://stackoverflow.com/questions/55074681/how-to-find-the-wrong-predictions-in-keras - haven't yet figured out why my labels are onehot encoded still\n\ny_pre_test=model.predict(X_val)\ny_pre_test=np.argmax(y_pre_test,axis=1)\ny_test=np.argmax(y_val,axis=1)\n\nx=(y_pre_test-y_test!=0).tolist()\nx=[i for i,l in enumerate(x) if l!=False]\n\n\n\nfig,ax=plt.subplots(1,4,sharey=False,figsize=(15,15))\n\nfor i in range(4):\n    ax[i].imshow(X_val[x[i]][:,:,0])\n    ax[i].set_xlabel('Real {}, Predicted {}'.format(y_val[x[i]],y_pre_test[x[i]]))\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now to make our predictions and submit!"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### This made me stumble when I changed\n\nWith sequential there is a function to get the classes, but this is not true with the functional API. You get an array with the models probabilities for each class. Therefore you have to take the highest value from the array and get the corresponding class."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(test[0].reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions[0] # looks like the 4 element is what the model thinks it is and it is over 99.99 % sure","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\npredictions = predictions.argmax(axis = -1)\npredictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({ 'id': Id,\n                            'label': predictions })\nsubmission.to_csv(path_or_buf =\"Kannada_mnist.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}