{"cells":[{"metadata":{},"cell_type":"markdown","source":" # Kannada Digit Recognition\n \n In modeling this data, I'll be following a three step training process, to create the best possible senario for creating my final predictions.  \n \n**First Training** In the first training of the model I'll be using a traditional train/test split of the training data alone, to both training and validate the model. If I notice the model overfits notably, I'll implement different techniques to reduce/correct this issue. To prevent overfitting I will either be increasing the dropout rate in the various Dropout layers in the model, by increasing the amount of pooling layers, or by implementing som eform of regulation.  \n \n**Second Training** In the second training of the model, I'll be using the entirity of the training data set to train the model, and then wil validate the model on the validation set, `Dig-MNIST.csv`. It may be easier to notice the effects of any overfitting in the model at this stage than it is in the original model, as I'll have the benefit of a much larger dataset, increasing the effect of overfitting if such an issue previously existed. If I notice such an issue, I'll again implement different techniques to reduce/correct this issue.  \n \n**Third Training** In the third and final training stage of the model, I will concatenate the training and validation data sets to create a new training set, `train_total`. I'll then use this uber-set to train the final model which will be used to predict the labels of the test set.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Imports\nimport numpy as np\nimport pandas as pd\n\n#Visualizations\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nget_ipython().run_line_magic('matplotlib', 'inline')\nsns.set_style(\"whitegrid\")\n\n#Modeling\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, accuracy_score\n\n#Keras\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense, MaxPool2D, Conv2D, Flatten, Dropout, BatchNormalization\nfrom keras.losses import categorical_crossentropy\nfrom keras.utils import to_categorical\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data\n### Exploratory Analysis\nThe data for this challenge can be found on [the kaggle competition page](). It consists of three seperate files: the training set, `train.csv`, the validation set, `Dig-MNIST.csv`, and the testing set, `test.csv`. These three sets each contain image data, stored in row/column styles. As such, each row of the data sets reflects a specific image, while the rows describe the information for the image. The first row contains the image label (or an id in the case of the test set), a number between 0-9, and the last 784 columns each show the value of the individual pixels as an integer between 0 and 255, inclusive, for the 28 x 28 pixel images.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Data import\ntrain = pd.read_csv(\"/kaggle/input/Kannada-MNIST/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/Kannada-MNIST/test.csv\")\nvalidation = pd.read_csv(\"/kaggle/input/Kannada-MNIST/Dig-MNIST.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"To verify the construction of the three dataframes, we will quickly review the first few rowsof the data.\n\n**Training Data**  \nThis data will be used through the training of all models.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In grouping the training data by their labels we can verify the number of classes in our data, as well as the distribution of the digits in the data set. Ideally the sampling distribution of the digits should be uniform.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train[\"label\"])\nplt.title(\"Distribution of Digit Samples in the Training Set\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(train[\"label\"]).size()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Validation Data**  \nThis data will be used as the validation data in the model witch trains on the full training set, and will be concatenated with the training data for the final submission model.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"validation.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although we don't usually need to be as highly concerned with the distribution of the test set, I will be, at some point in the analysis, combining the training and validation data, to fully train the model I'll be using for my submission, in order to provide the most training data as possible for the final model.  \nAs such, I'm going to review the distribution of the validation set as well.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(validation[\"label\"])\nplt.title(\"Distribution of Digit Samples in Validation Set\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation.groupby(validation[\"label\"]).size()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Testing Data**  \nThis data will be used to generate the class predictions for the competition submissions."},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Treatment and Augmentation\n#### Data Treatment\nAs the data is presented in row/column format, I need to transform it into matrix format both to visualize the images and for the keras CNN model I will be building.\n \nFirst I'll be spliting the label and id columns off of the three data sets, and storing them in seperate variables. I'll then be creating array verisions of the pixel information for the images. The labels will be sent to binary matrices made up of the class vectors for the labels, while the pixel information will be reshapped into the 28 x 28 pixel images. The pixel information will also be normalized, such the maxium pixel value will be 1, with a minimum of zero, to improve the accuracy of the model.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Spliting off labels/Ids\ntrain_labels = to_categorical(train.iloc[:,0])\ntrain = train.iloc[:, 1:].values\n\nX_validation = validation.iloc[:, 1:].values\ny_validation = to_categorical(validation.iloc[:,0])\n\ntest_id = test.iloc[:, 0]\ntest = test.iloc[:, 1:].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Normalizing the data\ntrain = train/255\nX_validation = X_validation/255\n\ntest = test/255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Reshaping data\ntrain = train.reshape(train.shape[0], 28, 28, 1)\n\nX_validation = X_validation.reshape(validation.shape[0], 28, 28, 1)\n\ntest = test.reshape(test.shape[0], 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With our data now in the correct shape, lets visualize the training and the validation sets."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Visualizing the Training Data\nfig, ax = plt.subplots(5, 10)\nfor i in range(5):\n    for j in range(10):\n        ax[i][j].imshow(train[np.random.randint(0, train.shape[0]), :, :, 0], cmap = plt.cm.binary)\n        ax[i][j].axis(\"off\")\nplt.subplots_adjust(wspace=0, hspace=0)\nfig.set_figwidth(15)\nfig.set_figheight(7)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above is a random selection of 50 different diits in the training set. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Visualizing the Validation Data\nfig, ax = plt.subplots(5, 10)\nfor i in range(5):\n    for j in range(10):\n        ax[i][j].imshow(X_validation[np.random.randint(0, X_validation.shape[0]), :, :, 0], cmap = plt.cm.binary)\n        ax[i][j].axis(\"off\")\nplt.subplots_adjust(wspace=0, hspace=0)\nfig.set_figwidth(15)\nfig.set_figheight(7)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above is a random selection of 50 different digits in the validation set."},{"metadata":{},"cell_type":"markdown","source":"#### Data Augmentation\nThe more data I can provide to the model, the more accurately it should be able to predict on new data. As I'm ~~unwilling~~ unable to draw another 60,000 digit samples to add to my training data, I'll be augmenting the training data using the keras utility `ImageDataGenerator()`. This will implement random alterations on various images throughout the data set, and then supplement the original training data with these new, slightly augmented images. My generator will,\n+ preform a random rotation of no more than 12 degrees\n+ modify the width of the image up to no more than 25% of the original width\n+ modify the height similarly to, but independantly from, the width\n+ shear the images (rotate the top of the image, bu not the bottom, imagine a rhombus compared to a rectagle) up to no more than 12 degrees\n+ preform a random zoom of the image up to 25% of the original image\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Augmenting data\ntrain_datagen = ImageDataGenerator(\n    rotation_range=12,\n    width_shift_range=0.25,\n    height_shift_range=0.25,\n    shear_range=12,\n    zoom_range=0.25\n)\n\nvalid_datagen = ImageDataGenerator(    \n    rotation_range=12,\n    width_shift_range=0.25,\n    height_shift_range=0.25,\n    shear_range=12,\n    zoom_range=0.25)\n\nvalid_datagen_simple = ImageDataGenerator()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* * *"},{"metadata":{},"cell_type":"markdown","source":"## Modeling\nI'll be creating one model for this project, and then training the optimal fit of said  model three seperate times. In the process of optimizing the model, I'll first be overfitting the model, and then reducing/correcting for the overfitting in optimization. \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Splitting train/test sets\nX_train, X_test, y_train, y_test = train_test_split(train, train_labels, test_size = 0.2, random_state = 84)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below I create the model function."},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model():\n    model = Sequential()\n    \n    #First set of covolutional layers\n    #Each with 32 unit output\n    model.add(Conv2D(32, (3,3), activation = \"relu\", input_shape = (28,28,1), padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Conv2D(32, (5,5), strides = (2,2) ,activation = \"relu\", padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n    \n    #Second set of covolution layers\n    #Each with a 64 unit output\n    model.add(Conv2D(64, (3,3), activation = \"relu\", input_shape = (28,28,1), padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, (5,5), strides = (2,2) ,activation = \"relu\", padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n    \n    #Third set of covolution layers\n    #Each with a 128 unit output    \n    model.add(Conv2D(128, (3,3), activation = \"relu\", padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Conv2D(128, (5,5), strides = (2,2), activation = \"relu\", padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25))\n    \n    #Pooling Layer\n    model.add(MaxPool2D(2,2))\n    model.add(Dropout(0.2))\n    \n    #Fourth and final covolution layer\n    #Output of 256 units\n    model.add(Conv2D(256, (3,3), activation = \"relu\", padding = \"same\"))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n    \n    #Flatteing model to pass to dense layer\n    model.add(Flatten())\n    \n    #First Dense Layer    \n    model.add(Dense(128, activation = \"relu\"))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n    \n    #Final Dense layer\n    #Used to predict the ten label classes    \n    model.add(Dense(10, activation = \"softmax\"))\n    \n    #Compile the model\n    model.compile(optimizer = \"adam\", loss = categorical_crossentropy, metrics = [\"accuracy\"])\n    \n    return(model)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below I create the model function."},{"metadata":{"trusted":true},"cell_type":"code","source":"#model\nmodel = build_model()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**First Training Step**  \nThis is the first training of the model, in which the model is trained and validated on different splits of the training data set. The original training of this model showed an average accuracy of ! and a validation accuracy of !. Overfitting was considered at this stage, but I decided to wait and review the second training stage for issues which would stem from overfitting.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#fitting model on training split\nhistory = model.fit_generator(\n    train_datagen.flow(X_train, y_train, batch_size = 1024), \n    epochs = 50,\n    steps_per_epoch=50,\n    #shuffle = True, this only has effect when steps_per_epoch = \"None\" \n    validation_data = (X_test, y_test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Performance**\n\nThe model is now trained! Let's review the performance of the first training step.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(history.history).describe().iloc[1:,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test_labels = []\nfor i in y_test:\n    for j, val in enumerate(i):\n        if val == 0.:\n            pass\n        else:\n            y_test_labels.append(j)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#predicting on the test split\npreds = history.model.predict_classes(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#accuracy of the predictions\naccuracy_score(preds, np.array(y_test_labels))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(9,7))\nplt.plot(history.history[\"accuracy\"])\nplt.plot(history.history[\"val_accuracy\"])\nplt.title(\"Accuracy on Training Data vs. Accuracy on Validation Data\\nTraining Step 1\")\nplt.legend([\"Train\", \"Validation\"], loc = \"lower right\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.savefig(\"Accuracy_TrainS1.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (9,7))\nplt.plot(history.history[\"loss\"])\nplt.plot(history.history[\"val_loss\"])\nplt.title(\"Loss on Training Data vs. Loss on Validation Data\\nTraining Step 1\")\nplt.legend([\"Train\", \"Validation\"], loc = \"upper right\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.savefig(\"Loss_TrainS1.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* * *"},{"metadata":{},"cell_type":"markdown","source":"**Second Training Step**  \nThe model will be trained on the full training set in this step, and will be validated against the validation set.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#reinitialize model\nmodel = build_model()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fitting on full training set\n#validation set used for validation\nhistory_full = model.fit_generator(\n    train_datagen.flow(train, train_labels, batch_size = 1024), \n    epochs = 50,\n    steps_per_epoch=train.shape[0]//1024,\n    validation_data = valid_datagen.flow(X_validation, y_validation))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training for the second training step is complete! Lets review the history to see how well the model performed. \n"},{"metadata":{},"cell_type":"markdown","source":"**Performance**"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(history_full.history).describe().iloc[1:,:]#\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = history_full.model.predict_classes(X_validation)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy_score(preds, np.argmax(y_validation, axis = 1))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.crosstab(preds, validation.iloc[:,0])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(9,7))\nplt.plot(history_full.history[\"accuracy\"])\nplt.plot(history_full.history[\"val_accuracy\"])\nplt.title(\"Accuracy on Training Data vs. Accuracy on Validation Data\\nTraining Step 2\")\nplt.legend([\"Train\", \"Validation\"], loc = \"lower right\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.savefig(\"Accuracy_TrainS2.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (9,7))\nplt.plot(history_full.history[\"loss\"])\nplt.plot(history_full.history[\"val_loss\"])\nplt.title(\"Loss on Training Data vs. Loss on Validation Data\\nTraining Step 2\")\nplt.legend([\"Train\", \"Validation\"], loc = \"upper right\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.savefig(\"Loss_TrainS2.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* * *"},{"metadata":{},"cell_type":"markdown","source":"**Third Training Step**  \nIn this training step I'll be concatenating the training and validation sets, to provide the most data possible for the training og the model.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#concatenating data\nX_train_total = np.concatenate((train, X_validation))\ny_train_total = np.concatenate((train_labels, y_validation))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_total.shape\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_total.shape\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Reinitializing model\nmodel = build_model()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history_final = model.fit_generator(train_datagen.flow(\n    X_train_total, y_train_total, batch_size=1024),\n    epochs = 50, \n    steps_per_epoch = X_train_total.shape[0]//1024,\n    validation_data = valid_datagen.flow(X_validation, y_validation))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizations**"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nplt.figure(figsize=(9,7))\nplt.plot(history_final.history[\"accuracy\"])\nplt.plot(history_final.history[\"val_accuracy\"])\nplt.title(\"Accuracy on Training Data vs. Accuracy on Validation Data\\nTraining Step 3\")\nplt.legend([\"Train\", \"Validation\"], loc = \"lower right\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.savefig(\"Accuracy_TrainS3.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (9,7))\nplt.plot(history_final.history[\"loss\"])\nplt.plot(history_final.history[\"val_loss\"])\nplt.title(\"Loss on Training Data vs. Loss on Validation Data\\nTraining Step 3\")\nplt.legend([\"Train\", \"Validation\"], loc = \"upper right\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.savefig(\"Loss_TrainS3.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* * *"},{"metadata":{},"cell_type":"markdown","source":"## Submissions"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = history_final.model.predict_classes(test)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(data = [pd.Series(test_id, name = \"id\"), pd.Series(preds, name = \"label\")], ).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index = False)\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}