{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **ECE668M - MIDTERMS PROJECT - BARRION, Marck Herzon**","metadata":{}},{"cell_type":"markdown","source":"This Kaggle notebook presents the ten (10) experiments that involves various hypotheses on a dataset adapted from a Kaggle competition, known as the [\"Kannada MNIST\"](https://www.kaggle.com/competitions/Kannada-MNIST/data). Here, the author examines the dataset and tests various variations on either the methodology or other settings and configurations that may affect the output.","metadata":{}},{"cell_type":"markdown","source":"The following sections presented in the table of contents below provide an overview on what to expect on this midterm project.\n\n* *Table of Contents:*\n    1. Overview of Experiments \n    2. About the Competition\n        * A. Description of Competition\n        * B. Data and Dataset Description\n    3. Experiments\n        * A. Experiment 1\n        * B. Experiment 2\n        * C. Experiment 3\n        * D. Experiment 4\n        * E. Experiment 5\n        * F. Experiment 6\n        * G. Experiment 7\n        * H. Experiment 8\n        * I. Experiment 9\n        * J. Experiment 10\n    4. Insights and Conclusion\n    5. References","metadata":{}},{"cell_type":"markdown","source":"# ***1. OVERVIEW OF EXPERIMENTS***","metadata":{}},{"cell_type":"markdown","source":"The experiments included in this notebook falls under various categories. Given that the nature of the dataset available in the experiment falls under images, one of the possible ways at which hypotheses were made includes on the image preprocessing techniques. Additionally, since these are images, various machine learning and deep learning techniques were also tested and experimented on. This includes using different settings and parameters that affect and ultimately impact the results. \n\nIn each of the experiments, the hypothesis made is then tested through experimentation, and are evaluated using either qualitative or quantitative techniques. ","metadata":{}},{"cell_type":"markdown","source":"# ***2. ABOUT THE COMPETITION***","metadata":{}},{"cell_type":"markdown","source":"This section presents the details about the competition selected, together with a description of the dataset that were used throughout the experiments. This ","metadata":{}},{"cell_type":"markdown","source":"# **2.1. Description of Competition**","metadata":{}},{"cell_type":"markdown","source":"![Cover Photo](https://storage.googleapis.com/kaggle-media/competitions/Kannada-MNIST/kannada.png)\n\nThe Kaggle Competition, which is entitled as the [\"Kannada MNIST\"](https://www.kaggle.com/competitions/Kannada-MNIST/overview), is spearheaded by the Vinay Prabhu, who provided the dataset for the competition. This is a \"Research Code Competition\" in Kaggle which officially concluded last December 18, 2019, but still accepts late submissions for grading and scoring. \n\nAs presented in the competition, the traditional MNIST for the Arabic numerals have been overused and various solutions have been already presented that perform well in the traditional MNIST dataset. Additionally, the MNIST digits have been considered as the \"Hello World\" for those who are on their initial journey in learning about machine learning and deep learning. Hence, for this competition, a modification and extension of which is how it uses the Kannada digits. \n\nKannada is a language that is spoken predominantly by those who are in the southwestern India. What makes it unique is on how this language has around 45 million native speakers who writes with tehe Kanna Script. As such, the competition will be classifying 10 digits writtein in Kannada and the task is to correctly classify them.\n\nAs for evaluating the performance of the solution, the categorization accuracy is used wherein the proportion of the test images correctly classified are identified and measured.","metadata":{}},{"cell_type":"markdown","source":"# **2.2. Data and Dataset Description**","metadata":{}},{"cell_type":"markdown","source":"Given the description of the competition presented above, the competition mainly delves into how one can classify the Kannada Digits which are hand written. The data files include the training and the testing which has gray-scale images of the aforementioned hand-drawn digits. These include the numbers from 0 to 9 in the Kannada Script.\n\nEach image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel value that describes its lightness or darkness, with larger numbers representing darker pixels. An integer between 0 and 255, inclusive, makes up this pixel value.\n\nAs for the data set for training, it includes the labels for the first column and an overall number of 785 columns for the pixel values that are associated with the image. Additionally, it must also be noted that each of the pixels in the training set is named as pixel{x}, wherein the value of x is the integer going from 0 all the way to 783 inclusive of the ends. The same is said for the testing dataset, but this time the labels are omitted. The rest of the columns correspond with the training dataset.\n\nThe files for the competition includes the following:\n* train.csv - training set\n* test.csv - test set\n* sample_submission.csv - a sample submission file following correct format\n* Dig-MNIST.csv - additional labeled set of characters for further validation\n\nHere, there is a total of 60,000 images in the training set, which are flattened to have 784 pixels each.","metadata":{}},{"cell_type":"markdown","source":"# ***3. EXPERIMENTS***","metadata":{}},{"cell_type":"markdown","source":"# **3.1. Experiment 1 (E1): Analysis of the Variations in the Number of Epochs on the Model Performance** ","metadata":{}},{"cell_type":"markdown","source":"This first experiment establishes the baseline or the controlled variable which will be further referred to in the succeeding experiments. As such, this experiment will present the basic concepts of convolutional neural networks, focusing on its operation, use of filters, padding, pooling, batch normalization, and the drop out.","metadata":{}},{"cell_type":"markdown","source":"# 3.1.1. E1 Research Question","metadata":{}},{"cell_type":"markdown","source":"With the goal of this first experiment to establish the baseline or point of comparison and reference for the succeeding experiments, this section will present a simple question that may be answered through experimentation and hypothesis testing. The question goes as follows:\n* *Does the number of epochs in the training affect the validation accuracy of the model?*\n\nFor this simple question, it will establish the number of epochs to be used in the succeeding experiments once determined.","metadata":{}},{"cell_type":"markdown","source":"# 3.1.2. E1 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"With this in mind, one of the possible factors that may come in for the hypothesis to be tested includes the concept of overfitting. Overfitting occurs whenever one models the training data too well [1]. What this infers is that it learns the details and the noise present in the training data, which negatively impacts the performance when tested on new data the model has not seen yet. This usually is seen when having very big epoch size as the model will begin to overfit the data [2]. As such the hypothesis formulated based from this concept goes as folloes:\n* *There is no significant evidence that the validation accuracy of a model is improved when the number of epochs in training is increased.*","metadata":{}},{"cell_type":"markdown","source":"# 3.1.3. E1 Methodology","metadata":{}},{"cell_type":"markdown","source":"For this section, the implementation of the tests conducted are seen and discussed. The entire code itself is in this section from the very beginning to the creation of the submission file. Shown below is the summary of the general steps taken:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Assessing data \n    D. Data Preprocessing\n        D.1. Normalizing Data\n        D.2. Training and Validation Split\n        D.3. Reshaping\n    E. Building Model\n    F. Data Augmentation\n    G. Training the Model\n        G.1. Using Epoch = 5\n        G.2. Using Epoch = 10\n        G.3. Using Epoch = 20\n        G.4. Using Epoch = 50\n    H. Evaluating the model\n    I. Submission\n\nAs such, below are the specific steps undertaken in completing the experiment.\n\nAdditionally, it is necessary to note that the solutions provided by the following authors were used as base models and references to develop the solution for this notebook. These are the following for citation and acknowledgement:\n* [Benan Akca](https://www.kaggle.com/code/benanakca/kannada-mnist-cnn-tutorial-with-app-top-2)\n* [Amal Nair](https://analyticsindiamag.com/kannada-makes-it-to-the-machine-learning-world-with-a-full-fledged-mnist-dataset/)\n* [Andrew Gao](https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db)\n","metadata":{}},{"cell_type":"code","source":"##### 3.1.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:12.636675Z","iopub.execute_input":"2023-03-16T14:29:12.637066Z","iopub.status.idle":"2023-03-16T14:29:12.647133Z","shell.execute_reply.started":"2023-03-16T14:29:12.637033Z","shell.execute_reply":"2023-03-16T14:29:12.645989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T03:50:19.538489Z","iopub.execute_input":"2023-03-16T03:50:19.539193Z","iopub.status.idle":"2023-03-16T03:50:20.584669Z","shell.execute_reply.started":"2023-03-16T03:50:19.539154Z","shell.execute_reply":"2023-03-16T03:50:20.582266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')\n\n# Here, the first and last values are shown\nraw_train1.iloc[[0,-1],[1,-1]]","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:16.745945Z","iopub.execute_input":"2023-03-16T14:29:16.747010Z","iopub.status.idle":"2023-03-16T14:29:20.354910Z","shell.execute_reply.started":"2023-03-16T14:29:16.746947Z","shell.execute_reply":"2023-03-16T14:29:20.353816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.C. Assessing data \n# The distribution of the data according to the labels may be visualized\nnum = raw_train1.label.value_counts()\nsns.barplot(x=num.index, y=num)\nnumbers = num.index.values","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:21.878114Z","iopub.execute_input":"2023-03-16T14:29:21.879148Z","iopub.status.idle":"2023-03-16T14:29:22.136782Z","shell.execute_reply.started":"2023-03-16T14:29:21.879098Z","shell.execute_reply":"2023-03-16T14:29:22.135810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# It can be seen above that the data is distributed evenly across the different \n# classes. Hence, there is no need to adjust any classes that are lacking in labels.\n\n# A sample image is also taken and displayed as an overview:\nsample = 3 \nnumber = raw_train1.iloc[sample,1:].values.reshape(28,28)\nprint(\"Sample Image of \"+ str(sample) + \" hand -written Kannada style\")\nplt.imshow(number, cmap=plt.get_cmap('gray'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:26.212124Z","iopub.execute_input":"2023-03-16T14:29:26.212956Z","iopub.status.idle":"2023-03-16T14:29:26.408405Z","shell.execute_reply.started":"2023-03-16T14:29:26.212911Z","shell.execute_reply":"2023-03-16T14:29:26.407420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.D. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 80% of the data are set to training, whereas 20% are for validation.\n# Also, a random state of 42 is used for initializing the itnernal random number generator\n# that decides the  splitting of the data. It is set to 42 as it will output the same groupings \n# every time the split was made for different runs.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.2, random_state=42) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:30.402341Z","iopub.execute_input":"2023-03-16T14:29:30.403033Z","iopub.status.idle":"2023-03-16T14:29:31.406005Z","shell.execute_reply.started":"2023-03-16T14:29:30.402995Z","shell.execute_reply":"2023-03-16T14:29:31.404785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:37.004710Z","iopub.execute_input":"2023-03-16T14:29:37.005459Z","iopub.status.idle":"2023-03-16T14:29:37.011963Z","shell.execute_reply.started":"2023-03-16T14:29:37.005420Z","shell.execute_reply":"2023-03-16T14:29:37.010938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:39.296834Z","iopub.execute_input":"2023-03-16T14:29:39.297490Z","iopub.status.idle":"2023-03-16T14:29:39.308993Z","shell.execute_reply.started":"2023-03-16T14:29:39.297437Z","shell.execute_reply":"2023-03-16T14:29:39.307606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.E. Building Model\n# The first part of working with the Keras Sequential network is to build the model.\n# As mentioned earlier, this experiment contains the filter, padding, convolution, \n# pooling, batch normalization, dropout, and also the activation function layers.\n# This model is loosely based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:46.552374Z","iopub.execute_input":"2023-03-16T14:29:46.553415Z","iopub.status.idle":"2023-03-16T14:29:46.582150Z","shell.execute_reply.started":"2023-03-16T14:29:46.553357Z","shell.execute_reply":"2023-03-16T14:29:46.581002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,###########\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:49.033324Z","iopub.execute_input":"2023-03-16T14:29:49.034308Z","iopub.status.idle":"2023-03-16T14:29:49.048310Z","shell.execute_reply.started":"2023-03-16T14:29:49.034252Z","shell.execute_reply":"2023-03-16T14:29:49.047134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes and the batch size were also indicated.\nnum_classes = 10\nbatch_size = 1024\n\n## NUMBER OF EPOCHS MODIFICATION\n# Four (4) epoch values were determined for this first experiment. \n# These include the following: 5, 15, 30, 50\n\nepoch1 = 5\nepoch2 = 15\nepoch3 = 30\nepoch4 = 50","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:55.679019Z","iopub.execute_input":"2023-03-16T14:29:55.679715Z","iopub.status.idle":"2023-03-16T14:29:55.684884Z","shell.execute_reply.started":"2023-03-16T14:29:55.679676Z","shell.execute_reply":"2023-03-16T14:29:55.683778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.F. Data Augmentation\n# Data Augmentation is done for this first experiment.\ndatagen_try = ImageDataGenerator(rotation_range=10,\n                             width_shift_range = 0.15,\n                             height_shift_range = 0.15,\n                             shear_range = 20,\n                             zoom_range = 0.4,)\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:29:59.379001Z","iopub.execute_input":"2023-03-16T14:29:59.379364Z","iopub.status.idle":"2023-03-16T14:30:00.067893Z","shell.execute_reply.started":"2023-03-16T14:29:59.379332Z","shell.execute_reply":"2023-03-16T14:30:00.066828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(rotation_range = 10,\n                                   width_shift_range = 0.15,\n                                   height_shift_range = 0.15,\n                                   shear_range = 20,\n                                   zoom_range = 0.4,\n                                   horizontal_flip = False)\ndatagen_val = ImageDataGenerator() \nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:30:02.577632Z","iopub.execute_input":"2023-03-16T14:30:02.578019Z","iopub.status.idle":"2023-03-16T14:30:02.584278Z","shell.execute_reply.started":"2023-03-16T14:30:02.577986Z","shell.execute_reply":"2023-03-16T14:30:02.583167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:30:07.196511Z","iopub.execute_input":"2023-03-16T14:30:07.196905Z","iopub.status.idle":"2023-03-16T14:30:07.205201Z","shell.execute_reply.started":"2023-03-16T14:30:07.196849Z","shell.execute_reply":"2023-03-16T14:30:07.203852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.G. Training the Model\n# Using Epoch = 5\nhistory1 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epoch1,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T03:52:21.116918Z","iopub.execute_input":"2023-03-16T03:52:21.117767Z","iopub.status.idle":"2023-03-16T03:54:23.155776Z","shell.execute_reply.started":"2023-03-16T03:52:21.117730Z","shell.execute_reply":"2023-03-16T03:54:23.154422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Using Epoch = 15\nhistory2 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epoch2,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:41:55.894109Z","iopub.execute_input":"2023-03-16T14:41:55.894495Z","iopub.status.idle":"2023-03-16T14:47:17.664063Z","shell.execute_reply.started":"2023-03-16T14:41:55.894461Z","shell.execute_reply":"2023-03-16T14:47:17.662624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Using Epoch = 30\nhistory3 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epoch3,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T04:07:12.907595Z","iopub.execute_input":"2023-03-16T04:07:12.907984Z","iopub.status.idle":"2023-03-16T04:17:06.891231Z","shell.execute_reply.started":"2023-03-16T04:07:12.907951Z","shell.execute_reply":"2023-03-16T04:17:06.890082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Using Epoch = 50\nhistory4 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epoch4,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T04:19:33.647084Z","iopub.execute_input":"2023-03-16T04:19:33.648012Z","iopub.status.idle":"2023-03-16T04:35:36.783745Z","shell.execute_reply.started":"2023-03-16T04:19:33.647961Z","shell.execute_reply":"2023-03-16T04:35:36.782664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# From these trainings, the last value of the validation accuracy was used as the basis \n# for the comparisons made. These are as follows:\n\nprint('Validation Accuracy when Epoch = 5: ', history1.history['val_accuracy'][4])\nprint('\\nValidation Accuracy when Epoch = 15: ', history2.history['val_accuracy'][14])\nprint('\\nValidation Accuracy when Epoch = 30: ', history3.history['val_accuracy'][29])\nprint('\\nValidation Accuracy when Epoch = 50: ', history4.history['val_accuracy'][49])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:26:23.325198Z","iopub.execute_input":"2023-03-16T09:26:23.326239Z","iopub.status.idle":"2023-03-16T09:26:23.333530Z","shell.execute_reply.started":"2023-03-16T09:26:23.326199Z","shell.execute_reply":"2023-03-16T09:26:23.332302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.H. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n# USING EPOCH = 5\nplt.plot(history1.history['accuracy'])\nplt.plot(history1.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Epoch = 5')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history1.history['loss'])\nplt.plot(history1.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Epoch = 5')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,2.2)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T04:55:01.328474Z","iopub.execute_input":"2023-03-16T04:55:01.329171Z","iopub.status.idle":"2023-03-16T04:55:01.778701Z","shell.execute_reply.started":"2023-03-16T04:55:01.329133Z","shell.execute_reply":"2023-03-16T04:55:01.777582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USING EPOCH = 15\nplt.plot(history2.history['accuracy'])\nplt.plot(history2.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Epoch = 15')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history2.history['loss'])\nplt.plot(history2.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Epoch = 15')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:26:35.392564Z","iopub.execute_input":"2023-03-16T09:26:35.393367Z","iopub.status.idle":"2023-03-16T09:26:35.828643Z","shell.execute_reply.started":"2023-03-16T09:26:35.393324Z","shell.execute_reply":"2023-03-16T09:26:35.827598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USING EPOCH = 30\nplt.plot(history3.history['accuracy'])\nplt.plot(history3.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Epoch = 30')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history3.history['loss'])\nplt.plot(history3.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Epoch = 30')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T04:55:42.317414Z","iopub.execute_input":"2023-03-16T04:55:42.317877Z","iopub.status.idle":"2023-03-16T04:55:42.865739Z","shell.execute_reply.started":"2023-03-16T04:55:42.317836Z","shell.execute_reply":"2023-03-16T04:55:42.864546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USING EPOCH = 50\nplt.plot(history4.history['accuracy'])\nplt.plot(history4.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Epoch = 50')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history4.history['loss'])\nplt.plot(history4.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Epoch = 50')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T04:56:10.839144Z","iopub.execute_input":"2023-03-16T04:56:10.839598Z","iopub.status.idle":"2023-03-16T04:56:11.360971Z","shell.execute_reply.started":"2023-03-16T04:56:10.839555Z","shell.execute_reply":"2023-03-16T04:56:11.359940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each of the trials, the submission file was generated for each of the experiments as these will be submitted for further scoring in the competition.","metadata":{}},{"cell_type":"code","source":"##### 3.1.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:26:47.690317Z","iopub.execute_input":"2023-03-16T09:26:47.691076Z","iopub.status.idle":"2023-03-16T09:26:47.732515Z","shell.execute_reply.started":"2023-03-16T09:26:47.691036Z","shell.execute_reply":"2023-03-16T09:26:47.731521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission1.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:26:51.124514Z","iopub.execute_input":"2023-03-16T09:26:51.125285Z","iopub.status.idle":"2023-03-16T09:26:51.975981Z","shell.execute_reply.started":"2023-03-16T09:26:51.125247Z","shell.execute_reply":"2023-03-16T09:26:51.974870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.1.4. E1 Analysis\n","metadata":{}},{"cell_type":"markdown","source":"Looking at the results obtained, through direct observation and comparison of the values, it can be said that for the first three epoch numbers used, particularly those from 5, 15, and 30, the last validation accuracy have all improved as the number of iterations are increased. Specifically, when epoch = 5, it obtained a value of 0.95008, and then when epoch = 15, it increased to 0.99383, and then when epoch = 30 it slightly decreased to 0.98758. Lastly, when the number of epochs are increased epoch = 50, then it increased to 0.99183. These values show that there is indeed an increase of validation accuracy as the epoch are increased, but it is not all throughout as observed when epoch = 30.\n\nIn analyzing why does this result make sense, we can refer back to one of the theoretical considerations presented earlier. Here, since the epochs represent one entire passing of training data through the algorithm, then it is expected that for certain increases of the epoch number, then there will be more times that the weights are changed. This allows for the model to better learn and fit the model to the classification problem desired. Hence, from the concept of underfitting, then there will be a time where it will be at its optimal performance. However, if the number of epochs are further increased, then a problem that may occur is on overfitting,wherein the model learns the training data too well that it might memorize the training data. This can be a problem when new images are being classified which will make it perform worse. As such, looking at the initial results, the validation accuracy may be on of the indicators that can facilitate in the analysis and evaluation. \n\nThere may be other alternative methods on how to test these observations and hypothesis. One is through further increasing the number of epochs to go to extreme cases where the number of epochs may reach > 100. However, given the limitations of the equipment and tools used by the author, the four different epochs would suffice in providing a general overview of the problem. Lastly, the quality of data may be taken note of as these were directly taken from the Kannada MNIST. Hence, the labels and the headings are directly from the competition itself.\n\nFurthermore, confounding variables in this experiment were also taken in to account, particularly looking into the limitations set by the computer (hardware) as well as software specifications used. This may ultimately affect the training of the training of the model, and hence the performance after training. Here, the author only made use of the same device across the experiments to ensure that setup is kept controlled as possible. Additionally, the succeeding experiments ensure that changes in the settings are kept as minimal as possible to isolate and experiment on the intended variables.","metadata":{}},{"cell_type":"markdown","source":"# 3.1.5. E1 Evaluation","metadata":{}},{"cell_type":"markdown","source":"Looking at the initial results obtained, it can be said that the accuracy of the test after evaluation may be used as direct metrics in evaluating the model. As such, this may be better visualized through a bar chart showing the results of the three different configurations for when the epoch is set at 5, 15, 30, and 50.\n\nIt is also worth noting that only the qualitative method was applied in this experiment, given that the actual samples of the  were not directly compared and only a direct measure was compared side-by-side. Hence, for this evaluation, four different analysis were done, the first of which is for the bar chart, followed by the subplots of the training and validation accuracies, then the confusion matrix, and lastly the accuracy score.","metadata":{}},{"cell_type":"code","source":"# Bar Chart for Scores\nx_axis = ['5', '15', '30', '50']\ny_axis = [history1.history['val_accuracy'][4], history2.history['val_accuracy'][14],\n         history3.history['val_accuracy'][29], history4.history['val_accuracy'][49]]\n\nplt.bar(x_axis, y_axis)\nplt.title('Last Validation Accuracy Scores of when Epoch = 5, 15, 30, 50')\nplt.xlabel('x_axis name')\nplt.ylabel('y_axis name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T05:59:06.088770Z","iopub.execute_input":"2023-03-16T05:59:06.089471Z","iopub.status.idle":"2023-03-16T05:59:06.270598Z","shell.execute_reply.started":"2023-03-16T05:59:06.089434Z","shell.execute_reply":"2023-03-16T05:59:06.269583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Additionally, since the qualitative approach is used, the plots of the different training and validation accuracies are shown below for a more side-by-side comparison.","metadata":{}},{"cell_type":"code","source":"# Subplots\nfig, axes = plt.subplots(2,2,  sharey=True, figsize=(8,8))\n\nax1, ax2, ax3, ax4 = axes.flatten()\nax1.plot(history1.history['val_accuracy'])\nax1.plot(history1.history['accuracy'])\nax2.plot(history2.history['val_accuracy'])\nax2.plot(history2.history['accuracy'])\nax3.plot(history3.history['val_accuracy'])\nax3.plot(history3.history['accuracy'])\nax4.plot(history4.history['val_accuracy'])\nax4.plot(history4.history['accuracy'])\n# Legends\nax1.legend(['Training', 'Validation'], loc='lower right')\nax2.legend(['Training', 'Validation'], loc='lower right')\nax3.legend(['Training', 'Validation'], loc='lower right')\nax4.legend(['Training', 'Validation'], loc='lower right')\n\n# Label\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n\n# Title\nax1.title.set_text('Epochs = 5')\nax2.title.set_text('Epochs = 15')\nax3.title.set_text('Epochs = 30')\nax4.title.set_text('Epochs = 50')\n\nplt.show","metadata":{"execution":{"iopub.status.busy":"2023-03-16T05:35:19.113582Z","iopub.execute_input":"2023-03-16T05:35:19.114309Z","iopub.status.idle":"2023-03-16T05:35:19.954699Z","shell.execute_reply.started":"2023-03-16T05:35:19.114270Z","shell.execute_reply":"2023-03-16T05:35:19.953539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Referring to the plots presented above, one of the noticeable details observed includes how the plots of the training and validation accuracies vary from the lower number of epoch numbers, particularly that of the 5, 15, and also at 30. This is as expected as mentioned earlier wherein the model tries to learn and update the weight as it is further increased. However, looking at the number of epochs when it is set to 50, then it can be seen that no further improvements are observed. Hence, it can be said that the model has now suffered from overfitting as the training and validation plots are distant from each other. As such, it can be inferred that if the number of epochs are increased further than either 15-30, then no improvements are expected.\n\nDespite having the validation accuracy increase slightly, the plots actually present more information regarding the performance of the model as it gives the trend and the status of overfitting. Furthermore, this is the tool utilized in evaluating the validation accuracy metric posed in the hypothesis stated earlier.","metadata":{}},{"cell_type":"markdown","source":"The confusion matrix is also another qualititative analysis technique that may be used in looking at the results. However, given that the model was not isolated for each of the parameters, this only assesses the performance of the model when it was trained using 50 epochs. Given that we have identified how no further improvements were observed, then the tests below will be used as a point of discussion.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Ground Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T06:26:44.784277Z","iopub.execute_input":"2023-03-16T06:26:44.784665Z","iopub.status.idle":"2023-03-16T06:26:48.013670Z","shell.execute_reply.started":"2023-03-16T06:26:44.784608Z","shell.execute_reply":"2023-03-16T06:26:48.012599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the confusion matrix, it can be seen that the model is able to perform well, as the diagonals comprise of the majority of the data points across the validation set. This infers that the model is able to correctly predict the digits being classified. Regardless, one of the digits that it struggled with is when 9 was classified as 6, as 16 instances of this mistake was observed. Additionally, this is the tool that may be used to evaluate one portion of the hypothesis presented on the classification capabilities.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore1 = model.evaluate(x_val, y_val, verbose=2);\n\nprint('Accuracy Score for Experiment 1: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:40:12.815764Z","iopub.execute_input":"2023-03-16T09:40:12.816160Z","iopub.status.idle":"2023-03-16T09:40:15.455408Z","shell.execute_reply.started":"2023-03-16T09:40:12.816128Z","shell.execute_reply":"2023-03-16T09:40:15.454296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Shown above is the accuracy score, which presented a single output that is able to evaluate the model when it was trained using the number of epochs = 50. \n\nGiven all of these, one of the conclusions that may be made is that the effect of varying the number of epochs depend on how the model is able to improve on the validation accuracy as it learns. Hence, the validation accuracy may be improved by increasing the epoch number, but only to a certain extent, as overfitting may already occur where the model does not learn anything new apart from possibly memorizing the training dataset. As such, it is necessary to take note of when the model stops improving its validation accuracy to identify the optimal number of epochs. Overfitting already occurred at the plot when epochs = 50 and starts to occur when epochs = 30.\n\nFor the succeeding experiments, the epoch number will be set at around 15 since improvements are still observed at this value and it had a better plot for the training and validation accuracies and losses.","metadata":{}},{"cell_type":"markdown","source":"# **3.2. Experiment 2 (E2): Effect of Data Augmentation on Model Performance**","metadata":{}},{"cell_type":"markdown","source":"For this next experiment, the same competition, dataset is used. However, the number of epochs is now fixed at 15 as determined and explained previously.","metadata":{}},{"cell_type":"markdown","source":"# 3.2.1. E2 Research Question","metadata":{}},{"cell_type":"markdown","source":"In formulating the research question for this next experiment, one of the concepts that is focused for this section is the principle of data augmentation. Data augmentation is the process of increasing the amount of data either by generating new data points from existing data [4]. The reason for employing data augmentation may be attributed to various points, one of which includes having limited data set for training and validation, and another point is to enable the model to be exposed to more variations in the training. These points indicate that the model will be exposed to more data points as well as more variability in how it is trained. These can be done by various ways, either by making small changes, such as transformations. Given this, one research question in mind goes as follows:\n\n* *Will augmenting the data further improve the performance of the model in terms of its classification capabilities and validation accuracy?*","metadata":{}},{"cell_type":"markdown","source":"# 3.2.2. E2 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"Given the preliminary information stated above, this can be further expounded on. Provided  that augmented data is simply the original data with minor changes [5], it can be said that this will increase the amount of data which will lead to more variations in the training of the model. In line with the earlier discussions, it can also be said that exposing the model with these conditions in training will pave the way for it to perform better when a new set of data is classified and tested. As such, a hypothesis that will be tested in this experiment goes as follows:\n* *Applying data augmentation techniques will increase the performance of the model and its classification capabilities and validation accuracy.*\n\nThe article by [5] also mentions that data augmentation prevents overfitting, which may be an added point of analysis when testing the hypothesis.","metadata":{}},{"cell_type":"markdown","source":"# 3.2.3. E2 Methodology","metadata":{}},{"cell_type":"markdown","source":"It must be noted that the previous experiment already had the data augmentation done. As such, for this experiment, it will train the model without any data augmentation processes and compare the last validation accuracy with the ones obtained above. As a recall, the following is the final validation accuracy when epochs = 15, with data augmentation.","metadata":{}},{"cell_type":"code","source":"print('Last Validation Accuracy with Data Augmentation when Epoch = 15: ', history2.history['val_accuracy'][14])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:28:37.164243Z","iopub.execute_input":"2023-03-16T09:28:37.164670Z","iopub.status.idle":"2023-03-16T09:28:37.171283Z","shell.execute_reply.started":"2023-03-16T09:28:37.164613Z","shell.execute_reply":"2023-03-16T09:28:37.170067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Similarly, the entire code itself is in this section from the very beginning to the creation of the submission file. Shown below is the summary of the general steps taken:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    c. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Training the Model\n    F. Evaluating the model\n    G. Submission\n\nHere, no data augmentation is done.","metadata":{}},{"cell_type":"code","source":"##### 3.2.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:00:59.937309Z","iopub.execute_input":"2023-03-16T15:00:59.938186Z","iopub.status.idle":"2023-03-16T15:00:59.950432Z","shell.execute_reply.started":"2023-03-16T15:00:59.938146Z","shell.execute_reply":"2023-03-16T15:00:59.948655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:03.970911Z","iopub.execute_input":"2023-03-16T15:01:03.971839Z","iopub.status.idle":"2023-03-16T15:01:03.981642Z","shell.execute_reply.started":"2023-03-16T15:01:03.971785Z","shell.execute_reply":"2023-03-16T15:01:03.980502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.2.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:06.071703Z","iopub.execute_input":"2023-03-16T15:01:06.072624Z","iopub.status.idle":"2023-03-16T15:01:08.777880Z","shell.execute_reply.started":"2023-03-16T15:01:06.072570Z","shell.execute_reply":"2023-03-16T15:01:08.776758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.2.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 80% of the data are set to training, whereas 20% are for validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.2) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:10.876599Z","iopub.execute_input":"2023-03-16T15:01:10.876996Z","iopub.status.idle":"2023-03-16T15:01:11.471496Z","shell.execute_reply.started":"2023-03-16T15:01:10.876962Z","shell.execute_reply":"2023-03-16T15:01:11.470426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:13.091205Z","iopub.execute_input":"2023-03-16T15:01:13.091739Z","iopub.status.idle":"2023-03-16T15:01:13.103634Z","shell.execute_reply.started":"2023-03-16T15:01:13.091701Z","shell.execute_reply":"2023-03-16T15:01:13.101485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:16.284071Z","iopub.execute_input":"2023-03-16T15:01:16.284654Z","iopub.status.idle":"2023-03-16T15:01:16.295348Z","shell.execute_reply.started":"2023-03-16T15:01:16.284616Z","shell.execute_reply":"2023-03-16T15:01:16.293933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.2.3.D. Building Model\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:21.210718Z","iopub.execute_input":"2023-03-16T15:01:21.211429Z","iopub.status.idle":"2023-03-16T15:01:21.235213Z","shell.execute_reply.started":"2023-03-16T15:01:21.211389Z","shell.execute_reply":"2023-03-16T15:01:21.234059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:28.620551Z","iopub.execute_input":"2023-03-16T15:01:28.620958Z","iopub.status.idle":"2023-03-16T15:01:28.635797Z","shell.execute_reply.started":"2023-03-16T15:01:28.620924Z","shell.execute_reply":"2023-03-16T15:01:28.634597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:31.875304Z","iopub.execute_input":"2023-03-16T15:01:31.876112Z","iopub.status.idle":"2023-03-16T15:01:31.881496Z","shell.execute_reply.started":"2023-03-16T15:01:31.876072Z","shell.execute_reply":"2023-03-16T15:01:31.880368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.2.3.E. Training the Model\nhistory5 = model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_val, y_val))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:01:35.123781Z","iopub.execute_input":"2023-03-16T15:01:35.124398Z","iopub.status.idle":"2023-03-16T15:05:19.822371Z","shell.execute_reply.started":"2023-03-16T15:01:35.124360Z","shell.execute_reply":"2023-03-16T15:05:19.821394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# From this trainings, the last value of the validation accuracy was used as the basis \n# for the comparisons made. These are as follows:\n\nprint('Validation Accuracy without Data Augmentation: ', history5.history['val_accuracy'][14])\n\nprint('\\nValidation Accuracy with Data Augmentation: ', history2.history['val_accuracy'][14])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:28:46.081639Z","iopub.execute_input":"2023-03-16T09:28:46.082585Z","iopub.status.idle":"2023-03-16T09:28:46.089881Z","shell.execute_reply.started":"2023-03-16T09:28:46.082533Z","shell.execute_reply":"2023-03-16T09:28:46.088692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.2.3.F. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\n# USING EPOCH = 15\nplt.plot(history5.history['accuracy'])\nplt.plot(history5.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy without Data Augmentation')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0.8,1)\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history5.history['loss'])\nplt.plot(history5.history['val_loss'])\nplt.title('Plot Training and Validation Loss without Data Augmentation')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T08:40:46.108362Z","iopub.execute_input":"2023-03-16T08:40:46.108805Z","iopub.status.idle":"2023-03-16T08:40:46.572986Z","shell.execute_reply.started":"2023-03-16T08:40:46.108767Z","shell.execute_reply":"2023-03-16T08:40:46.572040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.2.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T08:40:55.123298Z","iopub.execute_input":"2023-03-16T08:40:55.123684Z","iopub.status.idle":"2023-03-16T08:40:55.159832Z","shell.execute_reply.started":"2023-03-16T08:40:55.123648Z","shell.execute_reply":"2023-03-16T08:40:55.158873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission2.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T08:40:57.683189Z","iopub.execute_input":"2023-03-16T08:40:57.683901Z","iopub.status.idle":"2023-03-16T08:40:59.102217Z","shell.execute_reply.started":"2023-03-16T08:40:57.683863Z","shell.execute_reply":"2023-03-16T08:40:59.101073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.2.4. E2 Analysis","metadata":{}},{"cell_type":"markdown","source":"Similarly, since only two values are available for this experiment, a direct comparison may suffice for this section. It was observed that the difference between the two values when data augmentation was used versus when none was used is insignificant only for the validation accuracy metric. This may be attributed to the limited number of data points available or the nature of the problem itself. This looks at the validation accuracy as one of the metrics of evaluation. However, for the plots, it can be seen that the model without data augmentation performed better given the plots of the validation and training accuracies.\n\nThe value obtained with data augmentation was equal to 0.99383, whereas it increased slightly to a value of 0.99500. While these changes may be insignificant, one of the reasons at which theses results may still be considered valid may be on the nature of the Kannada MNIST itself. Looking at these, we can say that these are digits that are supposed to be written in a specific way. That is, they have to be oriented properly, using the write strokes in order for others to understand. Through data augmentation, it was seen that various techniques were applied which includes random rotations, shearing, horizontal and vertical shifts, among others. This results in a lot of possible points at which the Kannada Digit may not be understood properly anymore. Hence, the small increase in the validation accuracy supports this observation.\n\nAnother point of analysis is on the training time. Without data augmentation, the training went drastically quicker than when there was data augmentation applied. With data augmentation, the average time to train an epoch is equal to around 18.07s. On the other hand, the average time to train an epoch for the model without data augmentation resulted in a value of 15.13s only. This may be attributed to the increase in the amount of data when data augmentation is used which causes the need for more time to train.\n\nFurthermore, confounding variables in this experiment were considered, particularly looking into the limitations set by the computer (hardware) as well as software specifications used. Here, the author only made use of the same device across the experiments to ensure that setup is kept controlled as possible.\n\nIn the next section, the plots for the training and validation accuracies are displayed for a qualitative evaluation, together with the confusion matrix and the classification score.","metadata":{}},{"cell_type":"markdown","source":"# 3.2.5. E2 Evaluation","metadata":{}},{"cell_type":"markdown","source":"As mentioned, one of the ways to further evaluate the performance apart from the metrics may be through the training and validation accuracy plots. The plots for the model trained with data augmentation and without data augmentation is shown as follows:","metadata":{}},{"cell_type":"code","source":"# Subplots\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax1, ax2 = axes.flatten()\n\nax1.plot(history2.history['val_accuracy'])\nax1.plot(history2.history['accuracy'])\nax2.plot(history5.history['val_accuracy'])\nax2.plot(history5.history['accuracy'])\n\n#Legends\nax1.legend(['Training', 'Validation'], loc='lower right')\nax2.legend(['Training', 'Validation'], loc='lower right')\n\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n\n# Titles\nax1.title.set_text('with Data Augmentation')\nax2.title.set_text('without Data Augmentation')\n\nplt.show","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:30:04.024126Z","iopub.execute_input":"2023-03-16T09:30:04.025191Z","iopub.status.idle":"2023-03-16T09:30:04.428743Z","shell.execute_reply.started":"2023-03-16T09:30:04.025141Z","shell.execute_reply":"2023-03-16T09:30:04.427649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, it can be said that despite having small differences between the validation accuracy metric shown earlier, the validation and trainign plots of the two greatly varies with each other. Here, the model with the data augmentation was not able to perform greatly as there is some form of overfitting that has occured. On the other hand, the model without data augmentation was able to perform better by some margin. Relating these again to the reasons stated earlier, these results may seem valid given the nature of the Kannada MNIST digits, which must be written in a more specific form and orientation. When data augmentation is done, a lot of these information are distorted.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Ground Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:36:16.368450Z","iopub.execute_input":"2023-03-16T09:36:16.369517Z","iopub.status.idle":"2023-03-16T09:36:19.571684Z","shell.execute_reply.started":"2023-03-16T09:36:16.369471Z","shell.execute_reply":"2023-03-16T09:36:19.570606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In analyzing the classification performance of the model, it can be seen above that it was ble to perform the prediction relatively well as majority of the instances lie along the diagonal that indicates correct predictions made. However, it was noticed that for this model, a lot of the instances of the digit \"0\" was incorrectly predicted to be \"1\". The same case is for when the model predicted a \"7\" instead of a \"3\" for this example. Regarldess, the model was able to perform the classification problem well despite not having any data augmentation techniques done.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore2= model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 2: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:06:29.955775Z","iopub.execute_input":"2023-03-16T15:06:29.956409Z","iopub.status.idle":"2023-03-16T15:06:32.673547Z","shell.execute_reply.started":"2023-03-16T15:06:29.956370Z","shell.execute_reply":"2023-03-16T15:06:32.672516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, the accuract score is still relatively high for this model. This further proves the observations presented earlier, wherein the differences may be deemed insignificant.\n\nWhile the training and validation plots show some differences, it can be said that that hypothesis made earlier is rejected. That is, applying data augmentation techniques to the model does not necessarily increase the performance of the model for both of its classification capabilities and its validation accuracy. This can also be attributed to the nature of the data set itself. Here, it is expected that people writing this will follow specific orientation, strokes, and shapes. Having data augmentation done will result in a lot of the images to be rotated, shifted, zoomed in or out, and in general, have significant information cut out or not understandable. Hence, it can be concluded that the experiment shows how applying data augmentation to a set of data may not always be favorable. Additionally, the increase in training time may also be a factor when such techniques are employed.","metadata":{}},{"cell_type":"markdown","source":"# **3.3. Experiment 3 (E3): Analysis of the Use of Optimizers on the Model Performance**","metadata":{}},{"cell_type":"markdown","source":"# 3.3.1. E3 Research Question","metadata":{}},{"cell_type":"markdown","source":"This next experiment delves into another configuration that may be modified, which is for this case, the optimizer that is used. Optimizers are considered as algorithms or functions that is able to adapt the neural network's attributes that include learning rate and weights [6]. These further helps the deep learning model to improve its performance as it is trained. Various optimizers exist, which has differences on how it modifies the weights and the learning. For the first experiment conducted, it utilized the RMSprop optimizer, which is similar to the traditional gradient descent algorithm with momentum, nut it restricts oscillations in the vertical direction [7]. Training times are also said to decrease, as using the right optimization algorithm can allow the model to converge to the local or global solution [8].\n\nGiven all of these, the research question to be answered in this experiment goes as follows:\n* *Do optimizers aid in improving the training of the data in terms of accuracy and training time?*","metadata":{}},{"cell_type":"markdown","source":"# 3.3.2. E3 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"With the necessary information stated above, it can be further expounded that optimizers have been developed to allow us to perform training in a more efficient manner. Given that a lot of these algorithms have come up and have been developed, it can be said that utilization of such continues to be a research interest in machine learning. Hence, one of the possible hypotheses formulated, is shown as follows:\n* *There is significant differences that optimizers improve model performance in terms of its classification capabilities or validation accuracy, together with the reduction in training time.*\n\nSince the first experiment made use of the RMSProp optimizer algorithm, this experiment will use the same model, but this time no optimizer will be used. Similarly, the number of epochs is set to 15 for this experiment as determined previously.","metadata":{}},{"cell_type":"markdown","source":"# 3.3.3. E3 Methodology","metadata":{}},{"cell_type":"markdown","source":"As for this section, the implementation of the tests conducted are seen and discussed. The entire code itself is in this section from the very beginning to the creation of the submission file. Shown below is the summary of the general steps taken:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nHence, below are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.3.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:49:54.114982Z","iopub.execute_input":"2023-03-16T09:49:54.115820Z","iopub.status.idle":"2023-03-16T09:49:54.157734Z","shell.execute_reply.started":"2023-03-16T09:49:54.115771Z","shell.execute_reply":"2023-03-16T09:49:54.156605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:51:54.237738Z","iopub.execute_input":"2023-03-16T09:51:54.238450Z","iopub.status.idle":"2023-03-16T09:51:54.247928Z","shell.execute_reply.started":"2023-03-16T09:51:54.238414Z","shell.execute_reply":"2023-03-16T09:51:54.246909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:52:33.063187Z","iopub.execute_input":"2023-03-16T09:52:33.063582Z","iopub.status.idle":"2023-03-16T09:52:35.909114Z","shell.execute_reply.started":"2023-03-16T09:52:33.063549Z","shell.execute_reply":"2023-03-16T09:52:35.908018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 80% of the data are set to training, whereas 20% are for validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.2) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:52:39.651000Z","iopub.execute_input":"2023-03-16T09:52:39.652167Z","iopub.status.idle":"2023-03-16T09:52:40.208443Z","shell.execute_reply.started":"2023-03-16T09:52:39.652117Z","shell.execute_reply":"2023-03-16T09:52:40.207305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:53:40.408175Z","iopub.execute_input":"2023-03-16T09:53:40.408889Z","iopub.status.idle":"2023-03-16T09:53:40.419115Z","shell.execute_reply.started":"2023-03-16T09:53:40.408850Z","shell.execute_reply":"2023-03-16T09:53:40.418036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:53:46.213064Z","iopub.execute_input":"2023-03-16T09:53:46.213434Z","iopub.status.idle":"2023-03-16T09:53:46.221322Z","shell.execute_reply.started":"2023-03-16T09:53:46.213402Z","shell.execute_reply":"2023-03-16T09:53:46.220100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.D. Building Model\n# The first part of working with the Keras Sequential network is to build the model.\n# As mentioned earlier, this experiment contains the filter, padding, convolution, \n# pooling, batch normalization, dropout, and also the activation function layers.\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:54:27.852383Z","iopub.execute_input":"2023-03-16T09:54:27.852802Z","iopub.status.idle":"2023-03-16T09:54:27.879449Z","shell.execute_reply.started":"2023-03-16T09:54:27.852768Z","shell.execute_reply":"2023-03-16T09:54:27.878513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, no optimizer was used\n\nmodel.compile(loss='categorical_crossentropy',\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:54:55.083181Z","iopub.execute_input":"2023-03-16T09:54:55.084177Z","iopub.status.idle":"2023-03-16T09:54:55.097662Z","shell.execute_reply.started":"2023-03-16T09:54:55.084121Z","shell.execute_reply":"2023-03-16T09:54:55.096558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:54:59.548431Z","iopub.execute_input":"2023-03-16T09:54:59.549154Z","iopub.status.idle":"2023-03-16T09:54:59.553652Z","shell.execute_reply.started":"2023-03-16T09:54:59.549116Z","shell.execute_reply":"2023-03-16T09:54:59.552582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.E. Data Augmentation\n# Data Augmentation is done for this first experiment.\ndatagen_try = ImageDataGenerator(rotation_range=10,\n                             width_shift_range = 0.15,\n                             height_shift_range = 0.15,\n                             shear_range = 20,\n                             zoom_range = 0.4,)\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:55:40.847146Z","iopub.execute_input":"2023-03-16T09:55:40.847522Z","iopub.status.idle":"2023-03-16T09:55:41.541706Z","shell.execute_reply.started":"2023-03-16T09:55:40.847489Z","shell.execute_reply":"2023-03-16T09:55:41.540586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(rotation_range = 20,\n                                   width_shift_range = 0.35,\n                                   height_shift_range = 0.35,\n                                   shear_range = 30,\n                                   zoom_range = 0.4,\n                                   horizontal_flip = False)\ndatagen_val = ImageDataGenerator() \nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:55:44.804367Z","iopub.execute_input":"2023-03-16T09:55:44.804761Z","iopub.status.idle":"2023-03-16T09:55:44.810355Z","shell.execute_reply.started":"2023-03-16T09:55:44.804726Z","shell.execute_reply":"2023-03-16T09:55:44.809316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:56:10.533014Z","iopub.execute_input":"2023-03-16T09:56:10.533446Z","iopub.status.idle":"2023-03-16T09:56:10.545445Z","shell.execute_reply.started":"2023-03-16T09:56:10.533408Z","shell.execute_reply":"2023-03-16T09:56:10.544436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.F. Training the Model\nhistory6 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T09:56:24.778717Z","iopub.execute_input":"2023-03-16T09:56:24.779455Z","iopub.status.idle":"2023-03-16T10:01:17.848953Z","shell.execute_reply.started":"2023-03-16T09:56:24.779417Z","shell.execute_reply":"2023-03-16T10:01:17.847948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy obtained in this experiment may be compared from \n# the validation accuracy obtained in experiment 1 with the RMSProp optimizer.\n\nprint('Validation Accuracy without optimizer: ', history6.history['val_accuracy'][14])\nprint('\\nValidation Accuracy with optimizer: ', history2.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:02:09.876644Z","iopub.execute_input":"2023-03-16T10:02:09.877080Z","iopub.status.idle":"2023-03-16T10:02:09.885826Z","shell.execute_reply.started":"2023-03-16T10:02:09.877044Z","shell.execute_reply":"2023-03-16T10:02:09.884581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.G. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\nplt.plot(history6.history['accuracy'])\nplt.plot(history6.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy without optimizer')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.ylim(0,1)\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history6.history['loss'])\nplt.plot(history6.history['val_loss'])\nplt.title('Plot Training and Validation Loss without optimizer')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:02:58.101589Z","iopub.execute_input":"2023-03-16T10:02:58.102326Z","iopub.status.idle":"2023-03-16T10:02:58.525588Z","shell.execute_reply.started":"2023-03-16T10:02:58.102287Z","shell.execute_reply":"2023-03-16T10:02:58.524562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.3.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:04:37.424751Z","iopub.execute_input":"2023-03-16T10:04:37.425735Z","iopub.status.idle":"2023-03-16T10:04:37.465639Z","shell.execute_reply.started":"2023-03-16T10:04:37.425695Z","shell.execute_reply":"2023-03-16T10:04:37.464664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission3.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:04:51.631480Z","iopub.execute_input":"2023-03-16T10:04:51.632204Z","iopub.status.idle":"2023-03-16T10:04:53.049690Z","shell.execute_reply.started":"2023-03-16T10:04:51.632164Z","shell.execute_reply":"2023-03-16T10:04:53.048653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.3.4. E3 Analysis","metadata":{}},{"cell_type":"markdown","source":"Looking at the results above, it can be said that the RMSProp optimizer aids in improving the overall performance of the mode in terms of its validation accuracy after training. From the model with the optimizer used, the validation accuracy reaced a value of 0.99383. However, when the optimizer was removed, it resulted in a value of 0.97617. While these changes may seem small, it can still be noticed that removing the optimizer affected the training process of the model. \n\nFurther into this, the average training time per epoch for this configuration without any optimizer resulted in a value of 18.13s. This can be seen to have increased from the initial value of 18.07s from the model trained with the RMSProp. Similarly, although the differences are not that significantly large, it has shown that it still has affected how long the model was trained. Further metrics for analysis are shown in the next subsection. \n\nLooking at this experiment, it can be said that one of the confounding variables include the limitations presented by tools and equipment use. Since the training relies a lot in the hardware and software specifications available, then variations in the training time as well as the performance of the model after training may be attributed to these variables that were not considered. Regardless, the experiment was designed in such a way as to minimize all the possible sources of deviations or errors. That is, different factors were kept constant which includes the model used, the configurations, the layers in the model, the data set, among others. As such, it can be said that the results conform to the theoretical considerations presented.","metadata":{}},{"cell_type":"markdown","source":"# 3.3.5. E3 Evaluation","metadata":{}},{"cell_type":"markdown","source":"To better evaluate this, since two single-valued parameters are again compared, the use of qualitative assessment is again used. The plots fo the validation and training accuracies are taken and shown below and compared with that of the first experiment that makes use of the RMSProp optimizer. Furthermore, the confusion matrix and the accuracy score were taken as additional metrics to the evaluation of the model trained without optimizers.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax3, ax4 = axes.flatten()\n\nax3.plot(history2.history['val_accuracy'])\nax3.plot(history2.history['accuracy'])\nax4.plot(history6.history['val_accuracy'])\nax4.plot(history6.history['accuracy'])\n\n# Legends\nax3.legend(['Training', 'Validation'], loc='lower right')\nax4.legend(['Training', 'Validation'], loc='lower right')\n\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n\n# Titles\nax3.title.set_text('with RMSProp Optimizer')\nax4.title.set_text('without any Optimizer')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:27:09.061228Z","iopub.execute_input":"2023-03-16T10:27:09.061603Z","iopub.status.idle":"2023-03-16T10:27:31.786821Z","shell.execute_reply.started":"2023-03-16T10:27:09.061569Z","shell.execute_reply":"2023-03-16T10:27:31.785625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the plots above, qualitatively, it can be said that there are significant differences when it comes to the training and the validation of the two configurations. The most noticeable difference between the two involves the variations in how overfitting is more prominent in the model without any optimizer. The model using the RMSProp optimizer was able to obtain better plots. This can be again attributed to how optimizers are designed to minimize the error or the cost function of a model. Hence, the effects are more noticeable when the two plots are compared side by side. From this, it can be inferred that optimizers do aid in improving the training of the data as it adjusts the model's parameters.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Grand Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:41:51.561526Z","iopub.execute_input":"2023-03-16T10:41:51.561945Z","iopub.status.idle":"2023-03-16T10:41:53.929655Z","shell.execute_reply.started":"2023-03-16T10:41:51.561912Z","shell.execute_reply":"2023-03-16T10:41:53.928607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Referring to the results above, it can be seent that there has been a drastic drop in the classification capabilities of the model without the optimizer. For instance, the model predicted the digit \"6\" for the digits that are supposed to be \"7\" for 61 instances, and also \"9\" for 26 instances. This is observed throughout the rest of the instances where several mistakes have been committed by the model. Direct comparison with the confusion matrix of the model with optimizers show that this change does affect how the model performs.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore3 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 3: ', score3[1])\n\nprint('Accuracy Score for Experiment 1: ', score1[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T10:46:25.173033Z","iopub.execute_input":"2023-03-16T10:46:25.174021Z","iopub.status.idle":"2023-03-16T10:46:27.310786Z","shell.execute_reply.started":"2023-03-16T10:46:25.173969Z","shell.execute_reply":"2023-03-16T10:46:27.309761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the results above, it can be said that the accuracy score dropped when no optimizers were used. \n\nReferring to the observations made, it can be said that through the training and validation plots, the validation accuracy was said to drop when no optimizers were implemented. On the other hand, through the confusion matrix results, several instances of incorrect predictions have shown the poorer classification performance of the model. This was supported when the accuracy score metric was compared between the two. Also, training time has increased by some small value when the optimizer was removed. As such, it can be concluded that the hypothesis can be accepted, wherein there improvements, although not as significant, for the model performance in terms of its classification capabilities, validation accuracy, and reduction in the training time. Although the confounding variables mentioned above may have affected the results, it can still be said that sufficient evidence proves how optimizers aid in the aforementioned objectives.","metadata":{}},{"cell_type":"markdown","source":"# **3.4. Experiment 4 (E4): Variations on the Amount of Data allotted for Training and Validation - Part 1**","metadata":{}},{"cell_type":"markdown","source":"For the next three experiments, the concept of varying the amounts of data will be further explored and presented. These consits of different configurations that alter the values of the training data set and the validation data set; more specifically, the ratio between the amount of data that will be allocated for the two functions are varied and experimented on. ","metadata":{}},{"cell_type":"markdown","source":"# 3.4.1. E4 Research Question","metadata":{}},{"cell_type":"markdown","source":"The next experiment delves into the splitting of the dataset between the training and the validation. Previously in the first experiment, 80% were alloted for the training, whereas 20% of the data was set for the validation. The amount of data that the model is exposed at is one of the necessary considerations that must be made. Given that these models are dependent on the data that will be inputted, then the size and amount of which influences how the model is trained and how it performs afterwards. This experiment will ask the following question:\n* *Will there be significant differences in the performance of the model in its classification capabilities and accuracies when the amount of data for training and for validation is altered, particularly increasing the validation data set and decreasing the training set?*","metadata":{}},{"cell_type":"markdown","source":"# 3.4.2. E4 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"In formulating the hypothesis of the research question presented above, one may refer to what the article of [9] states. Here, the author mentions that the training set should not be set too small, as it can lead to not having enough data for the model to learn. On the other hand, if the validation set is set too small, then evaluation metrics may entail large variances, particularly on the accuracy, precision, and F1 score. As such, if this experiment sets the training validation small, a hypothesis may be stated as follows:\n* *Small amount of training data will result in poor model performance in terms of its classification capabilities and validation accuracy.*\n\nSince the previous experiments, particularly experiment 1 makes use of the 80% for training and 20% for validation, this experiment will delve into a reverse situation that only has 20% training data and 80% validation.","metadata":{}},{"cell_type":"markdown","source":"# 3.4.3. E4 Methodology","metadata":{}},{"cell_type":"markdown","source":"Shown below is the summary of the general steps taken, which makes use of the same general steps and processes in order to keep the experiment as controlled as possible. Additionally, different variables are kept the same in order to ensure that the only changes made are the ones that are desired tobe experimented on, particularly the values for the training and validation datasets.:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nAs such, below are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.4.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:02:14.728959Z","iopub.execute_input":"2023-03-16T11:02:14.730092Z","iopub.status.idle":"2023-03-16T11:02:14.749767Z","shell.execute_reply.started":"2023-03-16T11:02:14.730043Z","shell.execute_reply":"2023-03-16T11:02:14.748791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:04:28.685555Z","iopub.execute_input":"2023-03-16T11:04:28.685958Z","iopub.status.idle":"2023-03-16T11:04:28.696975Z","shell.execute_reply.started":"2023-03-16T11:04:28.685924Z","shell.execute_reply":"2023-03-16T11:04:28.695792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:04:38.377385Z","iopub.execute_input":"2023-03-16T11:04:38.377797Z","iopub.status.idle":"2023-03-16T11:04:40.745742Z","shell.execute_reply.started":"2023-03-16T11:04:38.377762Z","shell.execute_reply":"2023-03-16T11:04:40.744687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 20% was set to the validation, whereas 80% was split towards the testing set.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.8) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:04:49.680392Z","iopub.execute_input":"2023-03-16T11:04:49.680807Z","iopub.status.idle":"2023-03-16T11:04:50.237945Z","shell.execute_reply.started":"2023-03-16T11:04:49.680771Z","shell.execute_reply":"2023-03-16T11:04:50.236881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:04:53.463881Z","iopub.execute_input":"2023-03-16T11:04:53.464294Z","iopub.status.idle":"2023-03-16T11:04:53.471610Z","shell.execute_reply.started":"2023-03-16T11:04:53.464257Z","shell.execute_reply":"2023-03-16T11:04:53.470369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:05:02.477816Z","iopub.execute_input":"2023-03-16T11:05:02.478201Z","iopub.status.idle":"2023-03-16T11:05:02.485829Z","shell.execute_reply.started":"2023-03-16T11:05:02.478167Z","shell.execute_reply":"2023-03-16T11:05:02.484543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.D. Building Model\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:06:16.862186Z","iopub.execute_input":"2023-03-16T11:06:16.862765Z","iopub.status.idle":"2023-03-16T11:06:16.886487Z","shell.execute_reply.started":"2023-03-16T11:06:16.862736Z","shell.execute_reply":"2023-03-16T11:06:16.885551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:06:57.026698Z","iopub.execute_input":"2023-03-16T11:06:57.027161Z","iopub.status.idle":"2023-03-16T11:06:57.048418Z","shell.execute_reply.started":"2023-03-16T11:06:57.027107Z","shell.execute_reply":"2023-03-16T11:06:57.045849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and the epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:07:02.262246Z","iopub.execute_input":"2023-03-16T11:07:02.263284Z","iopub.status.idle":"2023-03-16T11:07:02.268552Z","shell.execute_reply.started":"2023-03-16T11:07:02.263233Z","shell.execute_reply":"2023-03-16T11:07:02.267359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.F. Data Augmentation\n# Data Augmentation is done for this first experiment.\ndatagen_try = ImageDataGenerator(rotation_range=10,\n                             width_shift_range = 0.15,\n                             height_shift_range = 0.15,\n                             shear_range = 20,\n                             zoom_range = 0.4,)\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:08:03.044436Z","iopub.execute_input":"2023-03-16T11:08:03.045036Z","iopub.status.idle":"2023-03-16T11:08:03.704820Z","shell.execute_reply.started":"2023-03-16T11:08:03.044990Z","shell.execute_reply":"2023-03-16T11:08:03.703664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(rotation_range = 10,\n                                   width_shift_range = 0.15,\n                                   height_shift_range = 0.15,\n                                   shear_range = 20,\n                                   zoom_range = 0.4,\n                                   horizontal_flip = False)\ndatagen_val = ImageDataGenerator() \nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:08:12.374494Z","iopub.execute_input":"2023-03-16T11:08:12.375490Z","iopub.status.idle":"2023-03-16T11:08:12.382169Z","shell.execute_reply.started":"2023-03-16T11:08:12.375452Z","shell.execute_reply":"2023-03-16T11:08:12.380965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:08:18.136613Z","iopub.execute_input":"2023-03-16T11:08:18.137337Z","iopub.status.idle":"2023-03-16T11:08:18.144792Z","shell.execute_reply.started":"2023-03-16T11:08:18.137300Z","shell.execute_reply":"2023-03-16T11:08:18.143695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.F. Training the Model\nhistory7 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:08:24.437382Z","iopub.execute_input":"2023-03-16T11:08:24.437977Z","iopub.status.idle":"2023-03-16T11:10:58.635205Z","shell.execute_reply.started":"2023-03-16T11:08:24.437925Z","shell.execute_reply":"2023-03-16T11:10:58.634082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with the first one as shown below.\n\nprint('Validation Accuracy with 20% of Dataset alloted to Training: ', history7.history['val_accuracy'][14])\nprint('\\nValidation Accuracy with 80% of Dataset alloted to Training: ', history2.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:12:43.656862Z","iopub.execute_input":"2023-03-16T11:12:43.657310Z","iopub.status.idle":"2023-03-16T11:12:43.665054Z","shell.execute_reply.started":"2023-03-16T11:12:43.657263Z","shell.execute_reply":"2023-03-16T11:12:43.663860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.G. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\nplt.plot(history7.history['accuracy'])\nplt.plot(history7.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with 20% of Dataset alloted to Training')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history7.history['loss'])\nplt.plot(history7.history['val_loss'])\nplt.title('Plot Training and Validation Loss with 20% of Dataset alloted to Training')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:12:50.632377Z","iopub.execute_input":"2023-03-16T11:12:50.635772Z","iopub.status.idle":"2023-03-16T11:12:51.149675Z","shell.execute_reply.started":"2023-03-16T11:12:50.635700Z","shell.execute_reply":"2023-03-16T11:12:51.148534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:12:59.050124Z","iopub.execute_input":"2023-03-16T11:12:59.051228Z","iopub.status.idle":"2023-03-16T11:12:59.108469Z","shell.execute_reply.started":"2023-03-16T11:12:59.051183Z","shell.execute_reply":"2023-03-16T11:12:59.107704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission4.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:13:01.948727Z","iopub.execute_input":"2023-03-16T11:13:01.949109Z","iopub.status.idle":"2023-03-16T11:13:02.768752Z","shell.execute_reply.started":"2023-03-16T11:13:01.949077Z","shell.execute_reply":"2023-03-16T11:13:02.767781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.4.4. E4 Analysis","metadata":{}},{"cell_type":"markdown","source":"In comparing the two results, it can be said that there is a small difference when the model used only 20% of the data for training as opposed to the previous one with 80% of the data set to the training. Specifically 0.98694 was the validation accuracy obtained when only 20% was set, and it was  higher with a value of 0.99383 for when 80% of the data was set. This can be attributed to how limited the data is for the model to learn and adjust the necessary weights in training. As opposed to when the model was exposed to more data to learn, which allowed for a better performance. Going back to the theoretical considerations presented earlier, it can be said that the small amount of training data resulted in the model to now have enough or sufficient data for it to have the weights of the model update as needed. Furthermore, classification capabilities will be evaluated in the next section, together with the observations in the plots for the training and validation.\n\nFurther into this, it can be said that the experimentw as kept at the most controlled setup as possible, wherein only the training and validation data set split ratio was used. The rest of the configurations for the model was kept constant and the same. This allows for the confounding variables mentioned earlier to be minimized, particularly on its effects to the training and performance of the model. The results obtained has shown how it is necessary to consider how much the author will allocate the dataset into training and into validation. ","metadata":{}},{"cell_type":"markdown","source":"# 3.4.5. E4 Evaluation","metadata":{}},{"cell_type":"markdown","source":"Qualitative analysis may also be done for this experiment, directly comparing the validation and training plots as shown below. Not limited to this, but the use of confusion matrix plots and the accuracy scores were also implemented for further considerations and qualitative analysis. These methods will aid in arriving at the conclusion of whether to reject or accept the hypothesis.","metadata":{}},{"cell_type":"code","source":"\nfig, axes = plt.subplots(1,2, sharey=True,figsize=(12,8) )\nax5, ax6 = axes.flatten()\n\nax5.plot(history2.history['val_accuracy'])\nax5.plot(history2.history['accuracy'])\nax6.plot(history7.history['val_accuracy'])\nax6.plot(history7.history['accuracy'])\n# Legends\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n\n# Title\nax5.title.set_text('with 80% of Data for Training, 20% for Validation')\nax6.title.set_text('with 20% of Data for Training, 80% for Validation')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:25:11.965235Z","iopub.execute_input":"2023-03-16T11:25:11.965701Z","iopub.status.idle":"2023-03-16T11:25:12.365733Z","shell.execute_reply.started":"2023-03-16T11:25:11.965651Z","shell.execute_reply":"2023-03-16T11:25:12.364602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With reference to the plots obtained above, it can be seen that the the model with 20% of the data used for training was not able to perform too well as opposed to when 80% of the data was implemented. This is better seen when the plot contained several fluctiations in its training and also when the plot started to overfit. Similar to the discussions above, this can be attributed to how limited the available data is for allowing the model to learn the and adjust the weights. Further into this, it was also observed that the model with 80% of the data resulted in less errors in terms of the training and validation accuracies as supported by the plots above.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Ground Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:32:19.900832Z","iopub.execute_input":"2023-03-16T11:32:19.901546Z","iopub.status.idle":"2023-03-16T11:32:31.048574Z","shell.execute_reply.started":"2023-03-16T11:32:19.901507Z","shell.execute_reply":"2023-03-16T11:32:31.047554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The confusion matrix presented above has further supported the initial observations for this experiment. Here, a lot of instances were noticed where the model that used 20% of the data for training was not able to correctly predict the results. Taking the digit \"1\" for instance, here it was seen that 230 instances were seen where it was not able to correctly predict the digit \"1\" and it predicted it as \"0\" instead. This was observed throughout the rest of the digits, where several instances of incorrectly predicted values were obtained. Additionally, since these make use of the validation data set, it must also be noted that more data are used in conducting and constructing this confusion matrix. Regardless, the observations above supported the initial findings on how the results have varied for the experiment.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore4 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 4: ', score4[1])\nprint('Accuracy Score for Experiment 1: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:36:23.155778Z","iopub.execute_input":"2023-03-16T11:36:23.156764Z","iopub.status.idle":"2023-03-16T11:36:33.891704Z","shell.execute_reply.started":"2023-03-16T11:36:23.156724Z","shell.execute_reply":"2023-03-16T11:36:33.890319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The accuracy scores above provide another metric for comparing the two models. Here, it can be seen that the model from experiment 1 which made use of 80% of the data for training performed better as opposed to when 20% were used.\n\nOverall, it can be said that the hypothesis may be accepted wherein using small amounts of training data, particularly only 20% used, have resulted in a poorer model performance in terms of its classification capabilities as seen in the confusion matrix and the validation accuracies in terms of its plots and curves. Other variables that may have affected the results were kept at a minimum through having this controlled setup.","metadata":{}},{"cell_type":"markdown","source":"# **3.5. Experiment 5 (E5): Variations on the Amount of Data allotted for Training and Validation - Part 2**","metadata":{}},{"cell_type":"markdown","source":"This experiment is part of the three-part series of experimentation that further explores the concept of distributing and allocating values for the training and validation of the data set available. As such, similar theoretical backgrounds and considerations are applied in this section.","metadata":{}},{"cell_type":"markdown","source":"# 3.5.1. E5 Research Question","metadata":{}},{"cell_type":"markdown","source":"In relation to the number of data alloted for training, this next experiment will now allot more data for training. Similar theoretical background and principles are applied in this section. As such the research question for this is as follows:\n* *Will increasing the data alloted for training improve the performance of the model in terms of its classification capabilities and validation accuracy?*","metadata":{}},{"cell_type":"markdown","source":"# 3.5.2. E5 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"Given the results obtained in the earlier experiment and the theoretical principles aligning with this, another hypothesis to be tested is on the increase of the training data set and further reduction of the validation data set. Applying the same set of theories and concepts as shown in the previous experiments and in the results as well, then the hypothesis for this experiment is as follows:\n* *Having more data alloted for training will lead to better performance of the model particularly looking at its classification capabilities and the training and validation plots.*","metadata":{}},{"cell_type":"markdown","source":"# 3.5.3. E5 Methodology","metadata":{}},{"cell_type":"markdown","source":"Similarly, the steps taken here are as follows:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nTherefore, below are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.5.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:45:21.280682Z","iopub.execute_input":"2023-03-16T11:45:21.281068Z","iopub.status.idle":"2023-03-16T11:45:21.299262Z","shell.execute_reply.started":"2023-03-16T11:45:21.281036Z","shell.execute_reply":"2023-03-16T11:45:21.298285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:45:24.315690Z","iopub.execute_input":"2023-03-16T11:45:24.316741Z","iopub.status.idle":"2023-03-16T11:45:24.327412Z","shell.execute_reply.started":"2023-03-16T11:45:24.316691Z","shell.execute_reply":"2023-03-16T11:45:24.326193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:45:29.819880Z","iopub.execute_input":"2023-03-16T11:45:29.820595Z","iopub.status.idle":"2023-03-16T11:45:32.406060Z","shell.execute_reply.started":"2023-03-16T11:45:29.820555Z","shell.execute_reply":"2023-03-16T11:45:32.404967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 90% of the data is now alloted to the training and 10% to the validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.9) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:45:35.942879Z","iopub.execute_input":"2023-03-16T11:45:35.943919Z","iopub.status.idle":"2023-03-16T11:45:36.509641Z","shell.execute_reply.started":"2023-03-16T11:45:35.943878Z","shell.execute_reply":"2023-03-16T11:45:36.508588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:45:40.006145Z","iopub.execute_input":"2023-03-16T11:45:40.006863Z","iopub.status.idle":"2023-03-16T11:45:40.014592Z","shell.execute_reply.started":"2023-03-16T11:45:40.006822Z","shell.execute_reply":"2023-03-16T11:45:40.013459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.D. Building Model\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:45:51.300798Z","iopub.execute_input":"2023-03-16T11:45:51.301179Z","iopub.status.idle":"2023-03-16T11:45:51.326238Z","shell.execute_reply.started":"2023-03-16T11:45:51.301145Z","shell.execute_reply":"2023-03-16T11:45:51.325113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:46:01.201314Z","iopub.execute_input":"2023-03-16T11:46:01.201736Z","iopub.status.idle":"2023-03-16T11:46:01.215916Z","shell.execute_reply.started":"2023-03-16T11:46:01.201700Z","shell.execute_reply":"2023-03-16T11:46:01.214859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size,and the  epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:46:04.663878Z","iopub.execute_input":"2023-03-16T11:46:04.664970Z","iopub.status.idle":"2023-03-16T11:46:04.671551Z","shell.execute_reply.started":"2023-03-16T11:46:04.664922Z","shell.execute_reply":"2023-03-16T11:46:04.670454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.E. Data Augmentation\ndatagen_try = ImageDataGenerator(rotation_range=10,\n                             width_shift_range = 0.15,\n                             height_shift_range = 0.15,\n                             shear_range = 20,\n                             zoom_range = 0.4,)\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:46:33.585339Z","iopub.execute_input":"2023-03-16T11:46:33.585921Z","iopub.status.idle":"2023-03-16T11:46:34.244733Z","shell.execute_reply.started":"2023-03-16T11:46:33.585882Z","shell.execute_reply":"2023-03-16T11:46:34.243586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(rotation_range = 10,\n                                   width_shift_range = 0.15,\n                                   height_shift_range = 0.15,\n                                   shear_range = 20,\n                                   zoom_range = 0.4,\n                                   horizontal_flip = False)\ndatagen_val = ImageDataGenerator() \nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:46:40.428948Z","iopub.execute_input":"2023-03-16T11:46:40.429691Z","iopub.status.idle":"2023-03-16T11:46:40.435244Z","shell.execute_reply.started":"2023-03-16T11:46:40.429647Z","shell.execute_reply":"2023-03-16T11:46:40.434094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:46:44.282395Z","iopub.execute_input":"2023-03-16T11:46:44.283125Z","iopub.status.idle":"2023-03-16T11:46:44.290963Z","shell.execute_reply.started":"2023-03-16T11:46:44.283084Z","shell.execute_reply":"2023-03-16T11:46:44.289906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.G. Training the Model\nhistory8 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:46:51.488364Z","iopub.execute_input":"2023-03-16T11:46:51.488766Z","iopub.status.idle":"2023-03-16T11:49:09.886927Z","shell.execute_reply.started":"2023-03-16T11:46:51.488732Z","shell.execute_reply":"2023-03-16T11:49:09.885819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with the first one as shown below.\n\nprint('Validation Accuracy with 90% of Dataset alloted to Training: ', history8.history['val_accuracy'][14])\nprint('\\nValidation Accuracy with 80% of Dataset alloted to Training: ', history2.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:50:35.165613Z","iopub.execute_input":"2023-03-16T11:50:35.166367Z","iopub.status.idle":"2023-03-16T11:50:35.174438Z","shell.execute_reply.started":"2023-03-16T11:50:35.166328Z","shell.execute_reply":"2023-03-16T11:50:35.172294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.H. Evaluating the model\nplt.plot(history8.history['accuracy'])\nplt.plot(history8.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with 90% Training Data')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history8.history['loss'])\nplt.plot(history8.history['val_loss'])\nplt.title('Plot Training and Validation Loss with 90% Training Data')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:50:41.342660Z","iopub.execute_input":"2023-03-16T11:50:41.343398Z","iopub.status.idle":"2023-03-16T11:50:41.782572Z","shell.execute_reply.started":"2023-03-16T11:50:41.343360Z","shell.execute_reply":"2023-03-16T11:50:41.781578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:50:47.142736Z","iopub.execute_input":"2023-03-16T11:50:47.143699Z","iopub.status.idle":"2023-03-16T11:50:47.175551Z","shell.execute_reply.started":"2023-03-16T11:50:47.143645Z","shell.execute_reply":"2023-03-16T11:50:47.174583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission5.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T11:50:49.404205Z","iopub.execute_input":"2023-03-16T11:50:49.405323Z","iopub.status.idle":"2023-03-16T11:50:50.821857Z","shell.execute_reply.started":"2023-03-16T11:50:49.405281Z","shell.execute_reply":"2023-03-16T11:50:50.820806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.5.4. E5 Analysis","metadata":{}},{"cell_type":"markdown","source":"In analyzing this situation, it can be said there has been a drop in the validation accuracy for when the training was done with 90% of the data was used versus when only 80% was used. Specifically, the experiment resulted in 0.97817 which is significantly low as opposed to when 80% of the data was used which resulted in a validation accuracy of 0.99383. This can be attributed to how there is limited data for validation, which results in large variances. Similarly, even if there are alot of training data, the validation data was too limited to assess the performance of the model. Adding into this, it can also be said that this goes against  the formulated hypothesis, which states that the model performance will improve, particularly on the validation accuracy. One of the possible considerations for this may be because of how little validation dataset there is to test the capabilities of the model. \n\nAdditionally, confounding variables may also be attributed to have affected the results of the experiment, particularly on the limitations on the hardware and software specifications. Regardless, the experiment was kept as controlled as possible, trying to minimize the results and impact of the other possible factors.","metadata":{}},{"cell_type":"markdown","source":"# 3.5.5. E5 Evaluation","metadata":{}},{"cell_type":"markdown","source":"In the same sense, the plots for the validation and training were used as qualitative assessments and evaluation for the two configurations done. Additionally, methods included taking in the confusion matrix to check the classification capabilities, together with the accuracy score. ","metadata":{}},{"cell_type":"code","source":"# Subplots\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax5, ax6 = axes.flatten()\n\nax5.plot(history2.history['val_accuracy'])\nax5.plot(history2.history['accuracy'])\nax6.plot(history8.history['val_accuracy'])\nax6.plot(history8.history['accuracy'])\n# Legends\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n\n# Titles\nax5.title.set_text('with 80% of Data for Training, 20% for Validation')\nax6.title.set_text('with 90% of Data for Training, 10% for Validation')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T12:00:03.350670Z","iopub.execute_input":"2023-03-16T12:00:03.351386Z","iopub.status.idle":"2023-03-16T12:00:03.725568Z","shell.execute_reply.started":"2023-03-16T12:00:03.351347Z","shell.execute_reply":"2023-03-16T12:00:03.724570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As for the training and validation plots shown above, it can be seen that the model with the 80% of the data alloted to training has performed better than when 90% of the data was alloted to training. One of the possible reasons for this result may be attributed to how the model at the left is limited on the amount of data it can use to validate the model performance. Despite having more data points for training, a problem that was observed in this configuration is on how the training plot fluctuated a lot of times. This is as opposed to the previous configuration when 80% of the data was implemented for training where a smoother plot was obtained. As such, the hypothesis that states that the model will perform better in terms of its validation accuracy may already be rejected. However, further evaluations are done in the next subsections.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Grand Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T12:03:16.572468Z","iopub.execute_input":"2023-03-16T12:03:16.573310Z","iopub.status.idle":"2023-03-16T12:03:25.041477Z","shell.execute_reply.started":"2023-03-16T12:03:16.573266Z","shell.execute_reply":"2023-03-16T12:03:25.040313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As seen in the confusion matrix, it can be said that there is a limited amount of data that was used in validating the model performance. As such, a lot of instances were observed to have been incorrectly predicted. Taking one data point as an example, it can be seen that 391 instances of the digit \"1\" was incorrectly predicted as \"0\". Another significant situatin is when the digit \"7\" was incorrectly identified as \"6\". Here, it can be seen that the model's classification capabilities have definitely dropped in its performance. This again rejects the hypothesis stated earlier.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore5 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 4: ', score5[1])\nprint('Accuracy Score for Experiment 1: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T12:05:53.776034Z","iopub.execute_input":"2023-03-16T12:05:53.776446Z","iopub.status.idle":"2023-03-16T12:06:02.555527Z","shell.execute_reply.started":"2023-03-16T12:05:53.776409Z","shell.execute_reply":"2023-03-16T12:06:02.554474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking further at its accuracy score, this metric further confirms the observations and insights gained in the experiment. Taking everything into account, it can be said that the hypothesis may be rejected, as the model trained with 90% of the data allocated for training, wherein 10% of the data is allocated for validation, performed worse than the initial setup with 80% for training amd 20% for validation. Its classification capabilities have reduced and dropped drastically given the confusion matrix, as well as the decrease in its validation accuracy as shown in the training and validation plots. These results were obtained as the experiment was tried to kept as controlled as possible.","metadata":{}},{"cell_type":"markdown","source":"# **3.6. Experiment 6 (E6): Variations on the Amount of Data allotted for Training and Validation - Part 3**","metadata":{}},{"cell_type":"markdown","source":"This next experiment is the last one for the three-part series on the exploration regarding the amounf of data allotment for training and validation. As such this further explores on the training and testing dataset, but this time having a balanced split, being 50% for training and 50% for testing. The same theoretical principles and concepts are applied for this experiment, which were used as basis for the research question and the formulation of the hypothesis.","metadata":{}},{"cell_type":"markdown","source":"# 3.6.1. E6 Research Question","metadata":{}},{"cell_type":"markdown","source":"Having the same principles as discussed above when it comes to the concept of splitting the dataset to training and testing, another thought that have emerged from this is when the same amount is now used for both the training and validation. This poses the thought of whether or not it is necessary to allot more to one than the other. As such, the research question to be answered is as follows:\n* *Is having a balanced split between training and validation result in better performance of the model with regards to its classification capabilities and validation accuracy?*","metadata":{}},{"cell_type":"markdown","source":"# 3.6.2. E6 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"For the hypothesis of the experiment, it can be said that same principles apply to this problem, wherein the idea of having a certain amount for training and validation may influence the entire process of training the model. Since a split will be done, then the model will have a good range of data in which it can learn and update the parameters, but also validate and test these learnings. Additionally, having a balanced of both allows the model to have ample data points which is able to not just perform well in training, but also in validation. Given this, the hypothesis goes as follows:\n* *Obtaining a balanced split of data for training and validation can improve model performance, particularly looking at its classification capabitilies and the training and validation plots.*","metadata":{}},{"cell_type":"markdown","source":"# 3.6.3. E6 Methodology","metadata":{}},{"cell_type":"markdown","source":"Same with before, the steps taken here are as follows. Here, the experiment is kept as controlled as possible, in order to ensure that all of the other factors are kept as a constant, while those that are desired to be analyzed are changed.\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nHence, below are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.6.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:20:56.623845Z","iopub.execute_input":"2023-03-16T14:20:56.626036Z","iopub.status.idle":"2023-03-16T14:21:07.632643Z","shell.execute_reply.started":"2023-03-16T14:20:56.626004Z","shell.execute_reply":"2023-03-16T14:21:07.631089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:21:42.119209Z","iopub.execute_input":"2023-03-16T14:21:42.120169Z","iopub.status.idle":"2023-03-16T14:21:44.778310Z","shell.execute_reply.started":"2023-03-16T14:21:42.120117Z","shell.execute_reply":"2023-03-16T14:21:44.776338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.6.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:21:48.620677Z","iopub.execute_input":"2023-03-16T14:21:48.621389Z","iopub.status.idle":"2023-03-16T14:21:53.827966Z","shell.execute_reply.started":"2023-03-16T14:21:48.621351Z","shell.execute_reply":"2023-03-16T14:21:53.826925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.6.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 50% of the data is  alloted to the training and 50% to the validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.5) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:22:06.911870Z","iopub.execute_input":"2023-03-16T14:22:06.912521Z","iopub.status.idle":"2023-03-16T14:22:07.486126Z","shell.execute_reply.started":"2023-03-16T14:22:06.912482Z","shell.execute_reply":"2023-03-16T14:22:07.485053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:22:43.144450Z","iopub.execute_input":"2023-03-16T14:22:43.144833Z","iopub.status.idle":"2023-03-16T14:22:43.152654Z","shell.execute_reply.started":"2023-03-16T14:22:43.144799Z","shell.execute_reply":"2023-03-16T14:22:43.151510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.6.3.D. Building Model\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:22:51.289968Z","iopub.execute_input":"2023-03-16T14:22:51.290952Z","iopub.status.idle":"2023-03-16T14:22:51.922707Z","shell.execute_reply.started":"2023-03-16T14:22:51.290886Z","shell.execute_reply":"2023-03-16T14:22:51.921684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:23:05.646800Z","iopub.execute_input":"2023-03-16T14:23:05.647725Z","iopub.status.idle":"2023-03-16T14:23:05.676327Z","shell.execute_reply.started":"2023-03-16T14:23:05.647686Z","shell.execute_reply":"2023-03-16T14:23:05.675370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size,and the  epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:23:10.563565Z","iopub.execute_input":"2023-03-16T14:23:10.563960Z","iopub.status.idle":"2023-03-16T14:23:10.569958Z","shell.execute_reply.started":"2023-03-16T14:23:10.563925Z","shell.execute_reply":"2023-03-16T14:23:10.568644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.F. Data Augmentation\ndatagen_try = ImageDataGenerator(rotation_range=10,\n                             width_shift_range = 0.15,\n                             height_shift_range = 0.15,\n                             shear_range = 20,\n                             zoom_range = 0.4,)\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:23:21.267380Z","iopub.execute_input":"2023-03-16T14:23:21.267739Z","iopub.status.idle":"2023-03-16T14:23:21.980913Z","shell.execute_reply.started":"2023-03-16T14:23:21.267707Z","shell.execute_reply":"2023-03-16T14:23:21.979837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(rotation_range = 10,\n                                   width_shift_range = 0.15,\n                                   height_shift_range = 0.15,\n                                   shear_range = 20,\n                                   zoom_range = 0.4,\n                                   horizontal_flip = False)\ndatagen_val = ImageDataGenerator() \nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:23:29.113835Z","iopub.execute_input":"2023-03-16T14:23:29.114575Z","iopub.status.idle":"2023-03-16T14:23:29.120960Z","shell.execute_reply.started":"2023-03-16T14:23:29.114536Z","shell.execute_reply":"2023-03-16T14:23:29.119426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:23:42.332357Z","iopub.execute_input":"2023-03-16T14:23:42.333341Z","iopub.status.idle":"2023-03-16T14:23:42.340472Z","shell.execute_reply.started":"2023-03-16T14:23:42.333286Z","shell.execute_reply":"2023-03-16T14:23:42.339249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.6.3.G. Training the Model\nhistory9 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:07:33.131111Z","iopub.execute_input":"2023-03-16T15:07:33.132066Z","iopub.status.idle":"2023-03-16T15:12:22.872980Z","shell.execute_reply.started":"2023-03-16T15:07:33.132013Z","shell.execute_reply":"2023-03-16T15:12:22.871917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with the first one as shown below.\n\nprint('Validation Accuracy with 50% of Dataset alloted to Training: ', history9.history['val_accuracy'][14])\nprint('\\nValidation Accuracy with 80% of Dataset alloted to Training: ', history2.history['val_accuracy'][14])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:48:43.936705Z","iopub.execute_input":"2023-03-16T14:48:43.937147Z","iopub.status.idle":"2023-03-16T14:48:43.946516Z","shell.execute_reply.started":"2023-03-16T14:48:43.937110Z","shell.execute_reply":"2023-03-16T14:48:43.944961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.6.3.G. Evaluating the model\nplt.plot(history9.history['accuracy'])\nplt.plot(history9.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with 50% Training Data')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history9.history['loss'])\nplt.plot(history9.history['val_loss'])\nplt.title('Plot Training and Validation Loss with 50% Training Data')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:48:52.982774Z","iopub.execute_input":"2023-03-16T14:48:52.983753Z","iopub.status.idle":"2023-03-16T14:48:53.418765Z","shell.execute_reply.started":"2023-03-16T14:48:52.983706Z","shell.execute_reply":"2023-03-16T14:48:53.417795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.5.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:48:58.067236Z","iopub.execute_input":"2023-03-16T14:48:58.068391Z","iopub.status.idle":"2023-03-16T14:48:58.108895Z","shell.execute_reply.started":"2023-03-16T14:48:58.068344Z","shell.execute_reply":"2023-03-16T14:48:58.107909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission6.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:49:00.042672Z","iopub.execute_input":"2023-03-16T14:49:00.043067Z","iopub.status.idle":"2023-03-16T14:49:01.003623Z","shell.execute_reply.started":"2023-03-16T14:49:00.043034Z","shell.execute_reply":"2023-03-16T14:49:01.002585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.6.4. E6 Analysis","metadata":{}},{"cell_type":"markdown","source":"As for this experiment, it can be noticed that when the data has been split equally to the training and validation set, the results show that the model performance did not improve further, but is relatively close as compared to when the data was split to 80% allotted to the training data set. Theses initial results can be attributed to how there is enough data for the training to learn and adjust the weights, together with how there is sufficient data to avoid having too large variances in the validation process as compared with the previous experiments. Hence, this setup resulted in a better performance for the model. Even if the model trained with the 50% of the data alloted to the training data set did not perform as well as the one wiht the 80% of the data for training, it can still be said that it still obtained a relatively high value for the validation accuracy.\n\nThese results can be seen to make sense as the amount of data for the model to learn the parameters and update the weights are more than for the 80% of the dataset for training. This allows the model to have ample data points for the aforesaid purpose of training. Additionally, this experiment continued the trend of having the setup as controlled as possible to ensure that only the change in the amount of data allotment is the one variable. Although other possible factors for deviation can still be found, this still enabled the results to conform with the expected theoretical behavior.","metadata":{}},{"cell_type":"markdown","source":"# 3.6.5. E6 Evaluation\n","metadata":{}},{"cell_type":"markdown","source":"Similarly, the plots of the training and validation plots were taken and used as a qualitative point of analysis and evaluation. Not limited to this, but the confusion matrix and the accuracy score were all taken as metrics to further extend the evaluation of the experiment.","metadata":{}},{"cell_type":"code","source":"# Subplots\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax5, ax6 = axes.flatten()\nax5.plot(history2.history['val_accuracy'])\nax5.plot(history2.history['accuracy'])\nax6.plot(history9.history['val_accuracy'])\nax6.plot(history9.history['accuracy'])\n# Legends\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n# Label\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n# Title\nax5.title.set_text('with 80% of Dataset alloted to Training')\nax6.title.set_text('with 50% of Dataset alloted to Training')\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:56:40.553692Z","iopub.execute_input":"2023-03-16T14:56:40.554674Z","iopub.status.idle":"2023-03-16T14:56:41.226834Z","shell.execute_reply.started":"2023-03-16T14:56:40.554633Z","shell.execute_reply":"2023-03-16T14:56:41.225829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One of the noticeable differences lie on how there are more fluctuations in the training plot of the model with only 50% of the data alloted to training. This can be attributed to how there are less amount of data for the model to learn and update the weights and parameters. Also, towards the end of the plot, the curve for the validation is seen to start to further separate and deviate from the training curve. This can indicate overfitting if the model is further trained with more epochs. This is as opposed to the model with 80% of the data for training, where less errors are present. This shows that the hypothesis of having improved validation and training plots may be rejected as shown in the figure above.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Ground Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:12:46.869164Z","iopub.execute_input":"2023-03-16T15:12:46.869770Z","iopub.status.idle":"2023-03-16T15:12:49.200042Z","shell.execute_reply.started":"2023-03-16T15:12:46.869733Z","shell.execute_reply":"2023-03-16T15:12:49.198926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As for this confusion matrix, it can be said that now, 50% of the data is set for validation. One of the noticeable observations here is that not much difference is seen for the classification of the different classes for the Kannada MNIST. Here, it can be seen that it was able to perform well despite not having the validation accuracy plot be the same as that when 80% of the data is alloted for training. The diagonal indicates that more instances are properly predicted and labeled. This also proves that obtaining an equal balance may not be necessary as the 80-20% allotment may be enough for the training and validation, respectively. ","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore6 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 6: ', score6[1])\nprint('Accuracy Score for Experiment 1: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:12:36.945019Z","iopub.execute_input":"2023-03-16T15:12:36.945503Z","iopub.status.idle":"2023-03-16T15:12:39.043521Z","shell.execute_reply.started":"2023-03-16T15:12:36.945461Z","shell.execute_reply":"2023-03-16T15:12:39.042527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Taking everthing into consideration, together with the accuracy score above, it can be said that the values still conform with the theoretical considerations. However, the hypothesis formulated may be rejected, as no significant improvement is seen. This is supported by the qualitative analysis made for the training and validation plots in assessing the validation accuracies for this model. The same can be said for the confusion matrix analysis which shows no significant improvement. Regardless, the model still shows that it is able to perform well despite the changes in the allotment of data. Although for the 80% allotment for training, the model was exposed to more variability in updating and adjusting the weights. Regardless, this experiment ends the series of exploration in the effect of having various amounts of data allotted for training and for validation. It can be concluded that selecting a good balance, with having more in training but not to the point that not enough is alloted for validation, must be considered.","metadata":{}},{"cell_type":"markdown","source":"# **3.7. Experiment 7 (E7):  Analysis on the Techniques of Data Augmentation Part 1**","metadata":{}},{"cell_type":"markdown","source":"This next set of experimentations done involves further manipulating the data augmentation process. Given that the transformations included in the data augmentation done in the first experiment includes the rotation, vertical shift, horizontal shift, shear, and zoom, various points of changes may be done for further experimentation. As such, these will be presented in the succeeding experiments.","metadata":{}},{"cell_type":"markdown","source":"# 3.7.1. E7 Research Question","metadata":{}},{"cell_type":"markdown","source":"For the first research question presented delves into the rotation technique used for data augmentation. As what [11] mentions, the use of random rotation allows for a source image to be randomly rotated either clockwise or counterclockwise given a degree. Additionally, it can be said that this is used in times where the object being classified may be oriented in various ways. For instance, for objects that are expected to be moving in some ways, if the source image only shows one orientation, then random rotation may help the model learn the object being classified across various configurations. Another point of consideration is when there are necessary information in the cornvers or edges of the image. If there are, then random rotation may not be the most ideal. This is further expounded on the next subsection. \n\nGiven these considerations and and theoretical concepts, the research question to answer in this experiment is as follows:\n\n* *Will solely using the random rotation technique as the data augmentation process improve the performance of the model in terms of its validation accuracy and classification capabilities?*","metadata":{}},{"cell_type":"markdown","source":"# 3.7.2. E7 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"As a hypothesis to the research question on hand, it can be said that:\n* *The use of random rotation for the classification of Kannada MNIST will not improve the performance of the model.*\n\nThe reason for setting this hypothesis may be attributed to what was stated earlier in the research question portion. That is, random rotation is best for those objects to be classified which involve some motion and being seen and read in various orientations. Given the nature of the Kannada MNIST where these are written in a stationary state, then having the rotations may not seem favorable. Additionally, the entire image of the Kannada MNIST contains necessary information, particularly if the digit has features that must be cut out for it to be understood. Hence, it was hypothesized that if further rotations were made, then portions of the Kannada digit may not be readable already. This may result in further errors in the training of the model.","metadata":{}},{"cell_type":"markdown","source":"# 3.7.3. E7 Methodology","metadata":{}},{"cell_type":"markdown","source":"Shown below is the summary of the general steps taken:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation - Random Rotation\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nBelow are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.7.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:14:36.804007Z","iopub.execute_input":"2023-03-16T15:14:36.804739Z","iopub.status.idle":"2023-03-16T15:14:36.815269Z","shell.execute_reply.started":"2023-03-16T15:14:36.804699Z","shell.execute_reply":"2023-03-16T15:14:36.813928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:14:44.973834Z","iopub.execute_input":"2023-03-16T15:14:44.974840Z","iopub.status.idle":"2023-03-16T15:14:44.985353Z","shell.execute_reply.started":"2023-03-16T15:14:44.974799Z","shell.execute_reply":"2023-03-16T15:14:44.984253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:14:54.921451Z","iopub.execute_input":"2023-03-16T15:14:54.921974Z","iopub.status.idle":"2023-03-16T15:14:59.974285Z","shell.execute_reply.started":"2023-03-16T15:14:54.921927Z","shell.execute_reply":"2023-03-16T15:14:59.969059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 20% was set to the validation, whereas 80% was split towards the testing set.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.8) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:05.277467Z","iopub.execute_input":"2023-03-16T15:15:05.277838Z","iopub.status.idle":"2023-03-16T15:15:05.857154Z","shell.execute_reply.started":"2023-03-16T15:15:05.277806Z","shell.execute_reply":"2023-03-16T15:15:05.856042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:08.747723Z","iopub.execute_input":"2023-03-16T15:15:08.748314Z","iopub.status.idle":"2023-03-16T15:15:08.755016Z","shell.execute_reply.started":"2023-03-16T15:15:08.748264Z","shell.execute_reply":"2023-03-16T15:15:08.753916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:11.923813Z","iopub.execute_input":"2023-03-16T15:15:11.924535Z","iopub.status.idle":"2023-03-16T15:15:11.932845Z","shell.execute_reply.started":"2023-03-16T15:15:11.924497Z","shell.execute_reply":"2023-03-16T15:15:11.931619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.D. Building Model\n# The first part of working with the Keras Sequential network is to build the model.\n# As mentioned earlier, this experiment contains the filter, padding, convolution, \n# pooling, batch normalization, dropout, and also the activation function layers.\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:16.808377Z","iopub.execute_input":"2023-03-16T15:15:16.808735Z","iopub.status.idle":"2023-03-16T15:15:16.835527Z","shell.execute_reply.started":"2023-03-16T15:15:16.808703Z","shell.execute_reply":"2023-03-16T15:15:16.834430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:19.984346Z","iopub.execute_input":"2023-03-16T15:15:19.984701Z","iopub.status.idle":"2023-03-16T15:15:19.998639Z","shell.execute_reply.started":"2023-03-16T15:15:19.984669Z","shell.execute_reply":"2023-03-16T15:15:19.997416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and the epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:22.500185Z","iopub.execute_input":"2023-03-16T15:15:22.500565Z","iopub.status.idle":"2023-03-16T15:15:22.505974Z","shell.execute_reply.started":"2023-03-16T15:15:22.500533Z","shell.execute_reply":"2023-03-16T15:15:22.504644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.4.3.E. Data Augmentation\n# Data Augmentation is done but this time, only the random rotation technique is used.\n# Rotation range had been increased to 360 degrees\n# The fill mode has been set to nearest, which simply replaces\n# the empty area with the nearest pixel values.\ndatagen_try = ImageDataGenerator(rotation_range=360, fill_mode ='nearest')\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:27.091518Z","iopub.execute_input":"2023-03-16T15:15:27.091978Z","iopub.status.idle":"2023-03-16T15:15:27.768812Z","shell.execute_reply.started":"2023-03-16T15:15:27.091936Z","shell.execute_reply":"2023-03-16T15:15:27.767811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(rotation_range = 360)\n\ndatagen_val = ImageDataGenerator() \n\nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:39.658674Z","iopub.execute_input":"2023-03-16T15:15:39.659069Z","iopub.status.idle":"2023-03-16T15:15:39.664754Z","shell.execute_reply.started":"2023-03-16T15:15:39.659034Z","shell.execute_reply":"2023-03-16T15:15:39.663496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:45.747685Z","iopub.execute_input":"2023-03-16T15:15:45.748062Z","iopub.status.idle":"2023-03-16T15:15:45.754726Z","shell.execute_reply.started":"2023-03-16T15:15:45.748029Z","shell.execute_reply":"2023-03-16T15:15:45.753594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.G. Training the Model\nhistory10 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:15:56.534919Z","iopub.execute_input":"2023-03-16T15:15:56.535940Z","iopub.status.idle":"2023-03-16T15:18:08.928560Z","shell.execute_reply.started":"2023-03-16T15:15:56.535861Z","shell.execute_reply":"2023-03-16T15:18:08.927490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with one without data augmentation.\n\nprint('Validation Accuracy with Random Rotations for Data Augmentation: ', history10.history['val_accuracy'][14])\nprint('\\nValidation Accuracy without Data Augmentation: ', history5.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:19:22.580609Z","iopub.execute_input":"2023-03-16T15:19:22.581112Z","iopub.status.idle":"2023-03-16T15:19:22.588941Z","shell.execute_reply.started":"2023-03-16T15:19:22.581073Z","shell.execute_reply":"2023-03-16T15:19:22.586754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.H. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\nplt.plot(history10.history['accuracy'])\nplt.plot(history10.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy for Random Rotations for Data Augmentation')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history10.history['loss'])\nplt.plot(history10.history['val_loss'])\nplt.title('Plot Training and Validation Loss for Random Rotations used for Data Augmentation')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:19:49.042655Z","iopub.execute_input":"2023-03-16T15:19:49.043419Z","iopub.status.idle":"2023-03-16T15:19:49.588339Z","shell.execute_reply.started":"2023-03-16T15:19:49.043371Z","shell.execute_reply":"2023-03-16T15:19:49.586539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:19:53.744703Z","iopub.execute_input":"2023-03-16T15:19:53.745618Z","iopub.status.idle":"2023-03-16T15:19:53.775938Z","shell.execute_reply.started":"2023-03-16T15:19:53.745564Z","shell.execute_reply":"2023-03-16T15:19:53.774985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission7.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:19:56.987110Z","iopub.execute_input":"2023-03-16T15:19:56.988049Z","iopub.status.idle":"2023-03-16T15:19:58.408478Z","shell.execute_reply.started":"2023-03-16T15:19:56.987996Z","shell.execute_reply":"2023-03-16T15:19:58.407474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.7.4. E7 Analysis","metadata":{}},{"cell_type":"markdown","source":"In analyzing the results of the experiment, it can be seen that the results when only the random rotations were done have shown a drop in the validation accuracy. Specifically, when no rotation or any data augmentation is done, the validation accuracy reached a value of 0.99450. However, when the random rotation was done as the sole process of data augmentation, then the validation accuracy dropped significantly to a value of 0.93152. This can be clearly attributed to how the random rotations have distorted the pertinent information necessary for the classification of the Kannada MNIST digits. Here, since the digits must be written in a certain orientation for it to be read, the model had a hard time trying to learn those that are completely rotated. This may also be attributed to how the rotation goes from 0 to 360 degrees, which can pose extreme cases.\n\n\nAdditionally these insights may be attributed to how the data augmentation may affect the characteristics of the image greatly, to an extent where the labels will no longer match the correct image. For instance, if the image is rotated 180 degrees, then the characteristics of the Kannada digits those not directly correspond to the appropriate label for the class. \n\nThese results conform to what is expected, as indicated in the hypothesis, as significant information will be loss or not be relevant as extreme cases of rotations are implemented. Also, other confounding factors were also noted such as the limitations of the hardware and software specifications, but its effects were kept to a minimum to ensure that a controlled setup is achieved.","metadata":{}},{"cell_type":"markdown","source":"# 3.7.5. E7 Evaluation","metadata":{}},{"cell_type":"markdown","source":"As for evaluation, direct comparisons of the two may further be seen through qualitative measures as presented below. The training and validation plots are displayed for further analysis. Apart from this, the confusion matrix and their corresponding accuracy scores were taken into account and consideration.","metadata":{}},{"cell_type":"code","source":"# Subplots\n\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax5, ax6 = axes.flatten()\n\nax5.plot(history5.history['val_accuracy'])\nax5.plot(history5.history['accuracy'])\nax6.plot(history10.history['val_accuracy'])\nax6.plot(history10.history['accuracy'])\n# Legends\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n# Title\nax5.title.set_text('without any Data Augmentation')\nax6.title.set_text('with Random Rotations for Data Augmentation')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:27:34.732025Z","iopub.execute_input":"2023-03-16T15:27:34.733041Z","iopub.status.idle":"2023-03-16T15:27:35.106941Z","shell.execute_reply.started":"2023-03-16T15:27:34.733001Z","shell.execute_reply":"2023-03-16T15:27:35.105905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The plot for training and validation accuracy has shown that there is some overfitting that occurs, apart from the fluctuations that are more prominent in the training curve of the accuracy. This can be said to be due to the details presented earlier, wherein the Kannada MNIST digits are presented in a way that must not be so, especially those whose orientations do not correspond to the write label for the class. Without any data augmentation, the results have shown that it performs better, which is also expected as all the digits are properly oriented with the classes corresponding to the appropriate images for the digits of the Kannada script.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Grand Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:30:04.209744Z","iopub.execute_input":"2023-03-16T15:30:04.210454Z","iopub.status.idle":"2023-03-16T15:30:15.370615Z","shell.execute_reply.started":"2023-03-16T15:30:04.210416Z","shell.execute_reply":"2023-03-16T15:30:15.369898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The confusion matrix was obtained for added qualitative analysis and it shows above that the model performs worse than when no data augmentation is done. Taking specific points for discussion, it can be seen that the model struggled to predict the digit \"8\", as it predicted it as the digit \"5\" for 519 instances. This is understandable as the strokes for the two are similar especially when one is rotated certain degrees over which may result in the same figure and shape as the other. These kinds of mistakes are seen to be spread across the various classes, which make the model's classification capability to drop drastically. Hence, this supports the conclusion stated earlier where a drop in its performance is expected.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore7 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 7: ', score7[1])\nprint('Accuracy Score for Experiment 2: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:36:23.138340Z","iopub.execute_input":"2023-03-16T15:36:23.138759Z","iopub.status.idle":"2023-03-16T15:36:31.120297Z","shell.execute_reply.started":"2023-03-16T15:36:23.138725Z","shell.execute_reply":"2023-03-16T15:36:31.119313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The accuracy score further supports the observations and insights stated above as there is a drastic drop in the accuracy of the model when evaluated. Overall, it can be said that the hypothesis presented is accepted, as there was a general drop and decrease in the performance of the model as seen in its validation accuracy as the plots of the training and the validation graphs are examined qualitatively. Not limited to this, but the confusion matrix also supported the hypothesis as the classification capabilities of the model dropped drastically given the large amounts of incorrect predictions. In conclusion, using random rotations to a great extent can make the labels be inappropriately assigned to the image that has been rotated greatly. This causes confusion and problems when the model is trained with the said images.","metadata":{}},{"cell_type":"markdown","source":"# **3.8. Experiment 8 (E8): Analysis on the Techniques of Data Augmentation Part 2**","metadata":{}},{"cell_type":"markdown","source":"Continuing this experiment series for data augmentation, the next part of the series is presented in this section which tackles on the concept and technique of using random zoom in and out processes. The same principles and concepts presented and discussed in the earlier experiment applies in this section. ","metadata":{}},{"cell_type":"markdown","source":"# 3.8.1. E8 Research Question","metadata":{}},{"cell_type":"markdown","source":"Given the theoretical considerations presented in the previous experiment, further changes in the dataset may improve the performance of the model. However, it was determined that this varies on the specific technique done for the data augmentation. For this experiment, the random zoom in or out is tested. The research question that is desired to be answered goes as follows:\n* *Will solely using the random zoom in/out technique as the data augmentation process improve the performance of the model in terms of its validation accuracy and classification capabilities?*","metadata":{}},{"cell_type":"markdown","source":"# 3.8.2. E8 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"The process of zooming in and out the image may seem to greatly affect the nature of the Kannada MNIST as the digits are expected to be written in a given shape and stroke. With zooms, the aspect ratios and the amount of pixels with relevant information are greatly altered. As what [12] mentions, it is necessary to select which technique suits the nature of the images being classified as it can also lead to issues if applied incorrectly. As for zooming in, significant information may be cut-off or removed if zoomed too close. On the other hand, if it was zoomed out further, classifying the digit may also be a challenge to distinguish the pixels properly. Having said these, the hypothesis formulated are as follows:\n* *Increasing the random zoom in/out factors for data augmentation will result in the model to perform poorly in terms of its classification capabilities and validation accuracy.*","metadata":{}},{"cell_type":"markdown","source":"# 3.8.3. E8 Methodology","metadata":{}},{"cell_type":"markdown","source":"Presented below is the summary of the general steps taken for this experiment:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation - Random Zoom In/Out\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nBelow are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.8.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:40:22.490640Z","iopub.execute_input":"2023-03-16T15:40:22.491048Z","iopub.status.idle":"2023-03-16T15:40:22.513394Z","shell.execute_reply.started":"2023-03-16T15:40:22.491014Z","shell.execute_reply":"2023-03-16T15:40:22.512429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:40:35.307249Z","iopub.execute_input":"2023-03-16T15:40:35.307630Z","iopub.status.idle":"2023-03-16T15:40:35.319505Z","shell.execute_reply.started":"2023-03-16T15:40:35.307596Z","shell.execute_reply":"2023-03-16T15:40:35.318287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:40:52.857318Z","iopub.execute_input":"2023-03-16T15:40:52.858381Z","iopub.status.idle":"2023-03-16T15:40:55.913215Z","shell.execute_reply.started":"2023-03-16T15:40:52.858325Z","shell.execute_reply":"2023-03-16T15:40:55.912169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 80% of the data are set to training, whereas 20% are for validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.2) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:41:17.583839Z","iopub.execute_input":"2023-03-16T15:41:17.584874Z","iopub.status.idle":"2023-03-16T15:41:18.131912Z","shell.execute_reply.started":"2023-03-16T15:41:17.584814Z","shell.execute_reply":"2023-03-16T15:41:18.130919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:41:22.786370Z","iopub.execute_input":"2023-03-16T15:41:22.787751Z","iopub.status.idle":"2023-03-16T15:41:22.795576Z","shell.execute_reply.started":"2023-03-16T15:41:22.787704Z","shell.execute_reply":"2023-03-16T15:41:22.794474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:41:27.023159Z","iopub.execute_input":"2023-03-16T15:41:27.023909Z","iopub.status.idle":"2023-03-16T15:41:27.031706Z","shell.execute_reply.started":"2023-03-16T15:41:27.023845Z","shell.execute_reply":"2023-03-16T15:41:27.030537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.D. Building Model\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:41:31.555693Z","iopub.execute_input":"2023-03-16T15:41:31.556412Z","iopub.status.idle":"2023-03-16T15:41:31.605176Z","shell.execute_reply.started":"2023-03-16T15:41:31.556374Z","shell.execute_reply":"2023-03-16T15:41:31.604216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:42:01.036033Z","iopub.execute_input":"2023-03-16T15:42:01.036407Z","iopub.status.idle":"2023-03-16T15:42:01.050749Z","shell.execute_reply.started":"2023-03-16T15:42:01.036373Z","shell.execute_reply":"2023-03-16T15:42:01.049492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:42:10.706001Z","iopub.execute_input":"2023-03-16T15:42:10.707008Z","iopub.status.idle":"2023-03-16T15:42:10.712645Z","shell.execute_reply.started":"2023-03-16T15:42:10.706967Z","shell.execute_reply":"2023-03-16T15:42:10.711554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.E. Data Augmentation\n# Data Augmentation is done but solely the random zoom in and zoom out is done.\n# Here, the zoom range was used wherein it was set to [1-value, 1+value], where the value\n# is equal to 0.9. This means that it will be between 10% for the zoom in, and then\n# 190% for the zoom out.\ndatagen_try = ImageDataGenerator(zoom_range = [0.1,1.9])\n\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:42:23.784931Z","iopub.execute_input":"2023-03-16T15:42:23.785642Z","iopub.status.idle":"2023-03-16T15:42:24.466875Z","shell.execute_reply.started":"2023-03-16T15:42:23.785603Z","shell.execute_reply":"2023-03-16T15:42:24.465815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(zoom_range = [0.1,1.9])\n\ndatagen_val = ImageDataGenerator() \n\nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:42:28.561305Z","iopub.execute_input":"2023-03-16T15:42:28.562028Z","iopub.status.idle":"2023-03-16T15:42:28.568071Z","shell.execute_reply.started":"2023-03-16T15:42:28.561986Z","shell.execute_reply":"2023-03-16T15:42:28.566926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:42:34.251172Z","iopub.execute_input":"2023-03-16T15:42:34.251542Z","iopub.status.idle":"2023-03-16T15:42:34.258651Z","shell.execute_reply.started":"2023-03-16T15:42:34.251508Z","shell.execute_reply":"2023-03-16T15:42:34.257565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.F. Training the Model\nhistory11 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:49:41.119403Z","iopub.execute_input":"2023-03-16T15:49:41.120446Z","iopub.status.idle":"2023-03-16T15:53:53.478140Z","shell.execute_reply.started":"2023-03-16T15:49:41.120405Z","shell.execute_reply":"2023-03-16T15:53:53.477096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with one without data augmentation.\n\nprint('Validation Accuracy with only Zoom In/Out Data Augmentation: ', history11.history['val_accuracy'][14])\nprint('\\nValidation Accuracy without Data Augmentation: ', history5.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:57:06.247456Z","iopub.execute_input":"2023-03-16T15:57:06.248527Z","iopub.status.idle":"2023-03-16T15:57:06.255077Z","shell.execute_reply.started":"2023-03-16T15:57:06.248473Z","shell.execute_reply":"2023-03-16T15:57:06.253961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.G. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\nplt.plot(history11.history['accuracy'])\nplt.plot(history11.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Zoom In/Out Data Augmentation')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history11.history['loss'])\nplt.plot(history11.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Zoom In/Out Data Augmentation')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:57:13.516387Z","iopub.execute_input":"2023-03-16T15:57:13.517091Z","iopub.status.idle":"2023-03-16T15:57:13.967811Z","shell.execute_reply.started":"2023-03-16T15:57:13.517053Z","shell.execute_reply":"2023-03-16T15:57:13.966798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.7.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:57:33.667925Z","iopub.execute_input":"2023-03-16T15:57:33.668991Z","iopub.status.idle":"2023-03-16T15:57:33.706960Z","shell.execute_reply.started":"2023-03-16T15:57:33.668947Z","shell.execute_reply":"2023-03-16T15:57:33.705809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission8.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:57:35.623581Z","iopub.execute_input":"2023-03-16T15:57:35.624070Z","iopub.status.idle":"2023-03-16T15:57:36.966471Z","shell.execute_reply.started":"2023-03-16T15:57:35.624035Z","shell.execute_reply":"2023-03-16T15:57:36.965377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.8.4. E8 Analysis","metadata":{}},{"cell_type":"markdown","source":"In looking at the validation accuracy to analyze the result and relate it to the hypothesis, it can be said that it surprising was able to achieve a high validation accuracy equal to 0.99483, as opposed to the validation accuracy of the model without data augmentaiton equal to 0.99450. This relatively high values for the accuracy may appear favorable, but the plots shown for the validation and training accuracies presents otherwise. Here, it can be said that the model overfits greatly. This makes more sense as the model is able to memorize the data set for training, but when validated, it performs worse as it overfits. As such, it can be said that while the accuracy in the training is high, when the validation plot is obtained, overfitting has become more evident. This is again attributed to how some significant information are lost when the image is zoomed in or zoomed out. This causes the labels to be incorrectly placed across the images with some pertinent information lost.\n\nLooking at this, the variables that are desired to be analyzed were isolated in a way that a more controlled setup was done by keeping the rest of the possible factors as constant as possible. This includes using the same model, layers, configurations, among others.","metadata":{}},{"cell_type":"markdown","source":"\n# 3.8.5. E8 Evaluation","metadata":{}},{"cell_type":"markdown","source":"To further assess and evaluate the results, the qualitative method was applied through analyzing the plots for the training and the validation accuracies, together with the implementation of the confusion matrix, and their corresponding accuracy scores. These are seen and discussed in this section.","metadata":{}},{"cell_type":"code","source":"# Subplots\n\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax5, ax6 = axes.flatten()\nax5.plot(history5.history['val_accuracy'])\nax5.plot(history5.history['accuracy'])\nax6.plot(history11.history['val_accuracy'])\nax6.plot(history11.history['accuracy'])\n# Legends\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n# Titles\nax5.title.set_text('without any Data Augmentation')\nax6.title.set_text('with Zoom In/Out Data Augmentation')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:59:51.350244Z","iopub.execute_input":"2023-03-16T15:59:51.351199Z","iopub.status.idle":"2023-03-16T15:59:51.756741Z","shell.execute_reply.started":"2023-03-16T15:59:51.351148Z","shell.execute_reply":"2023-03-16T15:59:51.755679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Despite having a relatively close value for the validation accuracy in the training shown above, the plots of the model with the zoom in and zoom out data augmentation have resulted in overfitting problems as shown in the model. While the accuracies are high, it can still be said that the model was not able to properly validate those that are not part of the training data set. However, these changes are still not that significant. As such, the hypothesis which states that the model will have a drop in performance may still be conditionally accepted as overfitting is observed. However, the statement which says that the validation accuracy drops may be questioned further as only small and insignificant differences are observed.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Grand Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:08:16.478895Z","iopub.execute_input":"2023-03-16T16:08:16.479611Z","iopub.status.idle":"2023-03-16T16:08:19.663901Z","shell.execute_reply.started":"2023-03-16T16:08:16.479575Z","shell.execute_reply":"2023-03-16T16:08:19.662840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The confusion matrix presented above show that the model is still able to perform relatively well despite some small overfitting observed in the plots. Here, only small instances of incorrect predictions are observed which shows that the model is still relatively accurate for its classification capabilities. Furthermore, it can be assessed that one reason for this performance may be on how the orientation did not change much, as compared to when rotation was done. Here, the same strokes and form are still observed and are only zoomed in or out.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore8 =model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 8: ', score8[1])\nprint('Accuracy Score for Experiment 2: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:10:25.882553Z","iopub.execute_input":"2023-03-16T16:10:25.883275Z","iopub.status.idle":"2023-03-16T16:10:28.007902Z","shell.execute_reply.started":"2023-03-16T16:10:25.883235Z","shell.execute_reply":"2023-03-16T16:10:28.006830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As shown in the accuracy score, not much difference is observed for when the model used random zoom in and out versus when no data augmentation is used. Overall, it can  be said that the hypothesis presented is rejected, as the model performance did not drop when evaluated qualitatively using the training and validation plots, together with saying that it was still able to relatively do well in its classification tasks as supported by the confusion matrix and the accuracy score metric. This can be attributed to how the digit is still oriented properly and the strokes and overall shape is not distorted as much as when the data augmentation done is the random rotation as shown previously.\n\nIt can also be said that the performance is maintained since more data points are used allowing the model to be more robust versus the limited amount of data when no augmentation is done.","metadata":{}},{"cell_type":"markdown","source":"# **3.9. Experiment 9 (E9): Analysis on the Techniques of Data Augmentation Part 3**","metadata":{}},{"cell_type":"markdown","source":"Moving on to the next part of the series of experiments for Data Augmentation techniques, this experiment further explores the effect of shear technique.","metadata":{}},{"cell_type":"markdown","source":"# 3.9.1. E9 Research Question","metadata":{}},{"cell_type":"markdown","source":"Continuing the trend of experimentations done on data augmentation, so far, it can be noted that varying the techniques done may affect the performance of the model. Given this, other data augmentation steps may be utilized in order to test what would be appropriate for the nature of the problem being solved in this notebook. As for this experiment, the process of applying the shear technique is further explored. As what [13] mentions, the process of shearing is a bounding box transformation that usually turns rectangular images into a more parallelogram-like figure as it is sheared. With this, the research question for this experiment goes as follows:\n* *If the shearing technique is used as the data augmentation process, will it further improve the performance of the model in terms of its validation accuracy and classification capabilities?*","metadata":{}},{"cell_type":"markdown","source":"\n# 3.9.2. E9 Formulation of Hypothesis","metadata":{}},{"cell_type":"markdown","source":"As far as the utilization of shearing is concerned as for a data augmentation technique, it can be said that shearing may distort the image and the object being classified. Similar to most of what is followed across different languages, scripts, and writing in general, distorting these things may result to unintelligible handwritings that can lead to confusion. Since, shearing shifts one part of the image like a parallelogram [14], the Kannada script may be deformed and misshapen which can cause issues in classification. In contrast with the previus data augmentation techniques, it is expected that not much information will be cut-off, but the way the digits appear will vary greatly. Given all of these, the hypothesis to be tested goes as follows:\n\n* *Greatly shearing the images in the data set will result in the model to misinterpret the Kannada digits, which will lead to poorer model perfomance and lower accuracies.*","metadata":{}},{"cell_type":"markdown","source":"# 3.9.3. E9 Methodology","metadata":{}},{"cell_type":"markdown","source":"For this section, the methods and steps taken for the experiment are shown below, presented in a summarized form:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation - Shearing\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nBelow are the specific steps undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.9.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:13:14.126622Z","iopub.execute_input":"2023-03-16T16:13:14.127134Z","iopub.status.idle":"2023-03-16T16:13:14.144018Z","shell.execute_reply.started":"2023-03-16T16:13:14.127095Z","shell.execute_reply":"2023-03-16T16:13:14.142703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:13:24.250144Z","iopub.execute_input":"2023-03-16T16:13:24.250913Z","iopub.status.idle":"2023-03-16T16:13:24.260128Z","shell.execute_reply.started":"2023-03-16T16:13:24.250849Z","shell.execute_reply":"2023-03-16T16:13:24.258979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.9.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:15:30.218490Z","iopub.execute_input":"2023-03-16T16:15:30.219521Z","iopub.status.idle":"2023-03-16T16:15:32.499379Z","shell.execute_reply.started":"2023-03-16T16:15:30.219469Z","shell.execute_reply":"2023-03-16T16:15:32.498337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.9.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 80% of the data are set to training, whereas 20% are for validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.2) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:16:22.932906Z","iopub.execute_input":"2023-03-16T16:16:22.933277Z","iopub.status.idle":"2023-03-16T16:16:23.476035Z","shell.execute_reply.started":"2023-03-16T16:16:22.933243Z","shell.execute_reply":"2023-03-16T16:16:23.474972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:16:57.803255Z","iopub.execute_input":"2023-03-16T16:16:57.804215Z","iopub.status.idle":"2023-03-16T16:16:57.811385Z","shell.execute_reply.started":"2023-03-16T16:16:57.804164Z","shell.execute_reply":"2023-03-16T16:16:57.810152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:17:20.907351Z","iopub.execute_input":"2023-03-16T16:17:20.908086Z","iopub.status.idle":"2023-03-16T16:17:20.916629Z","shell.execute_reply.started":"2023-03-16T16:17:20.908045Z","shell.execute_reply":"2023-03-16T16:17:20.915528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.8.3.D. Building Model\n# The first part of working with the Keras Sequential network is to build the model.\n# As mentioned earlier, this experiment contains the filter, padding, convolution, \n# pooling, batch normalization, dropout, and also the activation function layers.\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:17:24.131956Z","iopub.execute_input":"2023-03-16T16:17:24.132699Z","iopub.status.idle":"2023-03-16T16:17:24.157883Z","shell.execute_reply.started":"2023-03-16T16:17:24.132659Z","shell.execute_reply":"2023-03-16T16:17:24.156792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:17:28.461633Z","iopub.execute_input":"2023-03-16T16:17:28.462095Z","iopub.status.idle":"2023-03-16T16:17:28.483161Z","shell.execute_reply.started":"2023-03-16T16:17:28.462057Z","shell.execute_reply":"2023-03-16T16:17:28.482095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:17:32.061648Z","iopub.execute_input":"2023-03-16T16:17:32.062040Z","iopub.status.idle":"2023-03-16T16:17:32.067503Z","shell.execute_reply.started":"2023-03-16T16:17:32.062006Z","shell.execute_reply":"2023-03-16T16:17:32.066221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.9.3.E. Data Augmentation\n# Data Augmentation is done but solely the random zoom in and zoom out is done.\n# Here, the shear range was used wherein one axis is fixed and the other is stretched at a\n# given angle which is equal to the one shown below.\n# The value of the shear_range is a float that represents the shear angle\n# in a countr-clockwise direction. It is set at 85 degrees\ndatagen_try = ImageDataGenerator(shear_range = 85)\n\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:17:35.862819Z","iopub.execute_input":"2023-03-16T16:17:35.864004Z","iopub.status.idle":"2023-03-16T16:17:36.552097Z","shell.execute_reply.started":"2023-03-16T16:17:35.863953Z","shell.execute_reply":"2023-03-16T16:17:36.550871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(shear_range = 85)\n\ndatagen_val = ImageDataGenerator() \n\nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:17:47.550936Z","iopub.execute_input":"2023-03-16T16:17:47.551989Z","iopub.status.idle":"2023-03-16T16:17:47.557292Z","shell.execute_reply.started":"2023-03-16T16:17:47.551951Z","shell.execute_reply":"2023-03-16T16:17:47.556135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:18:20.397989Z","iopub.execute_input":"2023-03-16T16:18:20.398428Z","iopub.status.idle":"2023-03-16T16:18:20.407768Z","shell.execute_reply.started":"2023-03-16T16:18:20.398389Z","shell.execute_reply":"2023-03-16T16:18:20.406589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.9.3.F. Training the Model\nhistory12 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:19:23.386437Z","iopub.execute_input":"2023-03-16T16:19:23.387416Z","iopub.status.idle":"2023-03-16T16:23:52.596459Z","shell.execute_reply.started":"2023-03-16T16:19:23.387359Z","shell.execute_reply":"2023-03-16T16:23:52.595367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with one without data augmentation.\n\nprint('Validation Accuracy with only Zoom In/Out Data Augmentation: ', history12.history['val_accuracy'][14])\nprint('\\nValidation Accuracy without Data Augmentation: ', history5.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:24:51.292967Z","iopub.execute_input":"2023-03-16T16:24:51.294061Z","iopub.status.idle":"2023-03-16T16:24:51.300591Z","shell.execute_reply.started":"2023-03-16T16:24:51.294010Z","shell.execute_reply":"2023-03-16T16:24:51.299349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.9.3.G. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\nplt.plot(history12.history['accuracy'])\nplt.plot(history12.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Shearing Data Augmentation')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history12.history['loss'])\nplt.plot(history12.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Shearing Data Augmentation')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:24:55.042446Z","iopub.execute_input":"2023-03-16T16:24:55.043160Z","iopub.status.idle":"2023-03-16T16:24:55.483343Z","shell.execute_reply.started":"2023-03-16T16:24:55.043123Z","shell.execute_reply":"2023-03-16T16:24:55.482379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:25:03.626693Z","iopub.execute_input":"2023-03-16T16:25:03.627907Z","iopub.status.idle":"2023-03-16T16:25:03.663183Z","shell.execute_reply.started":"2023-03-16T16:25:03.627839Z","shell.execute_reply":"2023-03-16T16:25:03.662144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission9.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:25:08.886391Z","iopub.execute_input":"2023-03-16T16:25:08.887418Z","iopub.status.idle":"2023-03-16T16:25:09.905660Z","shell.execute_reply.started":"2023-03-16T16:25:08.887371Z","shell.execute_reply":"2023-03-16T16:25:09.904600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 3.9.4. E9 Analysis","metadata":{}},{"cell_type":"markdown","source":"For this result, similar observations were made and seen. Particularly looking at the validation accuracies, not much significant differences were observed for the two configurations. Here, the model was still able to perform almost as well as the model without any data augmentation. However, when the plots are now obtained for the training and validation, this is when the discrepancies begin to show. It can be seen that the model now overfits, which infers that it is only able to do well in the training data, but for the validation, it tends to achieve a lower value relative to the training value. Regardless, it can still be argued that the differences are still not as significant as what the hypothesis presents in the earlier parts of the experiment. Hence, it is recommended that further tests be made to assess this. Additionally, another point on the degree at which the image is sheared may also be in question. Here, it can be further increased to extreme levels to test its more prominent effects.\n\nAdditionally, it can also be said that despite losing some information, the degree to which these are lost may not be as significant to be considered as an issue. Hence, the hypothesis of saying that the peformance of the model will drop may be rejected if the sole basis is the validation accuracy. However, as shown, the plots show the issue of overfitting when examined further. Regardless, the experiment attempted to keep the setup as controlled as possible to ensure accurate and representative results.","metadata":{}},{"cell_type":"markdown","source":"# 3.9.5. E9 Evaluation","metadata":{}},{"cell_type":"markdown","source":"To proceed with the evaluation, the plots of the training and validation accuracy was taken and compare qualitatively. Additionally, the confusion matrix and the accuracy score were taken as added metrics to evaluate and further asses the results.","metadata":{}},{"cell_type":"code","source":"# Subplots\n\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax5, ax6 = axes.flatten()\n\nax5.plot(history5.history['val_accuracy'])\nax5.plot(history5.history['accuracy'])\nax6.plot(history12.history['val_accuracy'])\nax6.plot(history12.history['accuracy'])\n# Legend\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n# Label\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n# Title\nax5.title.set_text('without any Data Augmentation')\nax6.title.set_text('with Shearing Technique for Data Augmentation')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:28:35.490501Z","iopub.execute_input":"2023-03-16T16:28:35.491550Z","iopub.status.idle":"2023-03-16T16:28:35.864913Z","shell.execute_reply.started":"2023-03-16T16:28:35.491500Z","shell.execute_reply":"2023-03-16T16:28:35.863847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The plots of the validation and training accuracies better represent the changes and differences between the model without any data augmentation, versus when there was some used but with shearing technique. This means that the model is only able to perform well when being trained but is not able to deliver the same performance when it is now exposed to the validation data set. This portion conforms with one of the sections of the hypothesis formulated, where there is some small drops in the training of the model as shown in the plots above.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Ground Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:40:45.777060Z","iopub.execute_input":"2023-03-16T16:40:45.777441Z","iopub.status.idle":"2023-03-16T16:40:47.995005Z","shell.execute_reply.started":"2023-03-16T16:40:45.777409Z","shell.execute_reply":"2023-03-16T16:40:47.993925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, the model had incurred some errors in some of the digits, but it can still be said that these are not as significant as compared to other models in the experiment. This effect may be attributed to how the amount of data has been increased when the data augmentation process was done. As presented theoretically, increases of these allow for the model to be exposed to more variability, which further paves the way for better updates in the weights and parameters. ","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore9 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 9: ', score9[1])\nprint('Accuracy Score for Experiment 2: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:41:04.457961Z","iopub.execute_input":"2023-03-16T16:41:04.458664Z","iopub.status.idle":"2023-03-16T16:41:06.572851Z","shell.execute_reply.started":"2023-03-16T16:41:04.458627Z","shell.execute_reply":"2023-03-16T16:41:06.571921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Overall, it can be said that the hypothesis may be rejected as the overall accuracy of the model did not drop significantly for its performance to be altered greatly. Rather, this may be attributed to how more data points are added to the training of the model, which allowed for better updates on the weights and the parameters for the model to perform well. The confusion matrix supports this as it stil was able to have classification capabilities that are relatively well. However, overfitting was an issue to some extent as shown in the plots above. ","metadata":{}},{"cell_type":"markdown","source":"# **3.10. Experiment 10 (E10): Analysis on the Techniques of Data Augmentation Part 4**","metadata":{}},{"cell_type":"markdown","source":"Moving on to the last part of not just the data augmentation series, but also for the entire notebook as well, the exploration and testing of the use of vertical and horizontal shifts were then analyzed.","metadata":{}},{"cell_type":"markdown","source":"# 3.10.1. E10 Research Question","metadata":{}},{"cell_type":"markdown","source":"As for the last experiment of the series and of the notebook, the process of horizontal and vertical shifts are analyzed and examined. Looking at [15], horizontal shifts are done when pixels are shifted horizontally, either left or right, without altering the image's aspect ratio or dimension. The same is true for vertical shifts, wherein pixels are translated vertically, either up or down, and the dimensions are still kept as is. Possible sections or edges of the image may be cut off and replaced in order to keep the same aspect ratio. With this, the research question for this experiment goes as follows:\n* *Will horizontally and vertically shifting the image as a data augmentation technique help improve the performance in terms of its validation accuracy and classification capabilities?*","metadata":{}},{"cell_type":"markdown","source":"# 3.10.2. E10 Formulation of Hypothesis\n","metadata":{}},{"cell_type":"markdown","source":"In contrast with the other data augmentation techniques previously implemented, horizontal and vertical shifts do not directly distort the digits. Here, it can be said that the risk of having portions cut-off and clipped are the only main concern when these two shifts are implemented. Additionally, by ensuring that these shifts do not extremely move and cut the Kannada digits themselves, then the following hypothesis is formulated:\n\n* *Applying horizontal and vertical shifts to images will improve the model's performance in terms of classification capabilities and validation accuracy.*","metadata":{}},{"cell_type":"markdown","source":"# 3.10.3. E10 Methodology","metadata":{}},{"cell_type":"markdown","source":"In order to keep things as controlled as possible, the same steps were undertaken as summarized in the list below:\n\n    A. Preparatory steps which include importing necessary libraries.\n    B. Loading of the dataset for the training and testing.\n    C. Data Preprocessing\n        C.1. Normalizing Data\n        C.2. Training and Validation Split\n        C.3. Reshaping\n    D. Building Model\n    E. Data Augmentation - Horizontal and Vertical Shifts\n    F. Training the Model\n    G. Evaluating the model\n    H. Submission\n\nBelow are the specific steps and codes undertaken in completing the experiment.","metadata":{}},{"cell_type":"code","source":"##### 3.10.3.A. Preparatory Steps\n\n# The libraries and modules are first imported\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # data visualization\nimport seaborn as sns # data visualization\n\n# Libraries to prepare the data\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\n\n# Tensor flow Keras is used since images are concerned\nimport tensorflow as tf\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D,LeakyReLU\nfrom tensorflow.keras.optimizers import RMSprop,Nadam,Adadelta\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.regularizers import l2\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:46:35.295311Z","iopub.execute_input":"2023-03-16T16:46:35.296354Z","iopub.status.idle":"2023-03-16T16:46:35.318329Z","shell.execute_reply.started":"2023-03-16T16:46:35.296312Z","shell.execute_reply":"2023-03-16T16:46:35.316530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.test.gpu_device_name()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:48:22.601586Z","iopub.execute_input":"2023-03-16T16:48:22.602344Z","iopub.status.idle":"2023-03-16T16:48:22.612263Z","shell.execute_reply.started":"2023-03-16T16:48:22.602304Z","shell.execute_reply":"2023-03-16T16:48:22.611152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.10.3.B. Loading of the dataset for the training and testing\nraw_train1 = pd.read_csv('../input/Kannada-MNIST/train.csv')\nraw_test1 = pd.read_csv('../input/Kannada-MNIST/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:48:33.740313Z","iopub.execute_input":"2023-03-16T16:48:33.741129Z","iopub.status.idle":"2023-03-16T16:48:36.066001Z","shell.execute_reply.started":"2023-03-16T16:48:33.741087Z","shell.execute_reply":"2023-03-16T16:48:36.064914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.10.3.C. Data Preprocessing\n# Normalization\n# The first preprocessing technique done was to normalize the data to adjust the \n# values on a different scale. Particularly going from 0 to 1. This can be done\n# by dividing the data by the highest value, which is equal to 255 for the pixel.\n\nx = raw_train1.iloc[:, 1:].values.astype('float32')/ 255\ny = raw_train1.iloc[:, 0] \n\n# Training and Validation Split\n# This next part splits the data for training and also validation.\n# Here, 80% of the data are set to training, whereas 20% are for validation.\nx_train, x_val, y_train, y_val = train_test_split(x, y, test_size = 0.2) ","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:48:55.694642Z","iopub.execute_input":"2023-03-16T16:48:55.695385Z","iopub.status.idle":"2023-03-16T16:48:56.259845Z","shell.execute_reply.started":"2023-03-16T16:48:55.695347Z","shell.execute_reply":"2023-03-16T16:48:56.258789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshaping\n# To ensure that the data input fits the model, the reshaping of the data is done as follows\n# Initial size:\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:48:58.628198Z","iopub.execute_input":"2023-03-16T16:48:58.628612Z","iopub.status.idle":"2023-03-16T16:48:58.637001Z","shell.execute_reply.started":"2023-03-16T16:48:58.628563Z","shell.execute_reply":"2023-03-16T16:48:58.633929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# performing the reshape and ensure it is categorical\nx_train = x_train.reshape(-1, 28, 28,1)\nx_val = x_val.reshape(-1, 28, 28,1)\ny_train = to_categorical(y_train)\ny_val = to_categorical(y_val)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:49:02.138604Z","iopub.execute_input":"2023-03-16T16:49:02.139305Z","iopub.status.idle":"2023-03-16T16:49:02.146215Z","shell.execute_reply.started":"2023-03-16T16:49:02.139267Z","shell.execute_reply":"2023-03-16T16:49:02.145131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.10.3.D. Building Model\n# This model is based on the solutions provided by Andrew Gao [3],with some modifications done\n# LINK: https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db\n\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    tf.keras.layers.Conv2D(256, (3,3), padding='same'),\n    tf.keras.layers.BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"),##\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n\n    tf.keras.layers.MaxPooling2D(2,2),\n    tf.keras.layers.Dropout(0.2),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256),\n    tf.keras.layers.LeakyReLU(alpha=0.1),\n    \n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.Dense(10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:49:36.224894Z","iopub.execute_input":"2023-03-16T16:49:36.225947Z","iopub.status.idle":"2023-03-16T16:49:36.251197Z","shell.execute_reply.started":"2023-03-16T16:49:36.225899Z","shell.execute_reply":"2023-03-16T16:49:36.250246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compilation \n# For this model, the optimizer used was from the RMSprop function as adapted from [3]\noptimizer = RMSprop(learning_rate=0.01,\n    rho=0.6,\n    momentum=0.01,\n    epsilon=1e-07,\n    centered=True,\n    name='RMSprop')\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:49:43.354749Z","iopub.execute_input":"2023-03-16T16:49:43.355764Z","iopub.status.idle":"2023-03-16T16:49:43.369700Z","shell.execute_reply.started":"2023-03-16T16:49:43.355716Z","shell.execute_reply":"2023-03-16T16:49:43.368718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of classes, batch size, and epochs were also indicated.\nnum_classes = 10\nbatch_size = 1024\nepochs = 15","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:49:46.628085Z","iopub.execute_input":"2023-03-16T16:49:46.629089Z","iopub.status.idle":"2023-03-16T16:49:46.636194Z","shell.execute_reply.started":"2023-03-16T16:49:46.629048Z","shell.execute_reply":"2023-03-16T16:49:46.635059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.10.3.E. Data Augmentation\n# Data Augmentation is done but solely the random zoom in and zoom out is done.\n# Here, the horizontal and vertical shifts are done using the arguments below\n# vertical shifts are done using the height_shift_range\n# horizontal shifts are done using the width_shift_range\n# both are set to 1 to indicate -90% to +90% shifts \ndatagen_try = ImageDataGenerator(width_shift_range = 0.90, height_shift_range =0.90)\n\n# fit parameters from data\ndatagen_try.fit(x_train)\n# configure batch size and retrieve one batch of images\nfor x_batch, y_batch in datagen_try.flow(x_train, y_train, batch_size=9):\n\t# create a grid of 3x3 images\n\tfor i in range(0, 9):\n\t\tplt.subplot(330 + 1 + i)\n\t\tplt.imshow(x_batch[i].reshape(28, 28), cmap=plt.get_cmap('gray'))\n\t# show the plot\n\tplt.show()\n\tbreak","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:52:58.871066Z","iopub.execute_input":"2023-03-16T16:52:58.872049Z","iopub.status.idle":"2023-03-16T16:52:59.562885Z","shell.execute_reply.started":"2023-03-16T16:52:58.871993Z","shell.execute_reply":"2023-03-16T16:52:59.561875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datagen_train = ImageDataGenerator(width_shift_range = 0.90, height_shift_range =0.90)\n\ndatagen_val = ImageDataGenerator() \n\nstep_train = x_train.shape[0] // batch_size\nstep_val = x_val.shape[0] // batch_size","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:53:19.910383Z","iopub.execute_input":"2023-03-16T16:53:19.910757Z","iopub.status.idle":"2023-03-16T16:53:19.916502Z","shell.execute_reply.started":"2023-03-16T16:53:19.910723Z","shell.execute_reply":"2023-03-16T16:53:19.915263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Callbacks have been declared\nlearning_rate_reduction = tf.keras.callbacks.ReduceLROnPlateau( \n    monitor='loss',    # Quantity to be monitored.\n    factor=0.25,       # Factor by which the learning rate will be reduced. new_lr = lr * factor\n    patience=2,        # The number of epochs with no improvement after which learning rate will be reduced.\n    verbose=1,         # 0: quiet - 1: update messages.\n    mode=\"auto\",       # {auto, min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing; \n                       # in the max mode it will be reduced when the quantity monitored has stopped increasing; \n                       # in auto mode, the direction is automatically inferred from the name of the monitored quantity.\n    min_delta=0.0001,  # threshold for measuring the new optimum, to only focus on significant changes.\n    cooldown=0,        # number of epochs to wait before resuming normal operation after learning rate (lr) has been reduced.\n    min_lr=0.00001     # lower bound on the learning rate.\n    )\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=300, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:53:24.214987Z","iopub.execute_input":"2023-03-16T16:53:24.215524Z","iopub.status.idle":"2023-03-16T16:53:24.227277Z","shell.execute_reply.started":"2023-03-16T16:53:24.215477Z","shell.execute_reply":"2023-03-16T16:53:24.225501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.10.3.F. Training the Model\nhistory13 = model.fit_generator(datagen_train.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=len(x_train)//batch_size,\n                              epochs=epochs,\n                              validation_data=(x_val, y_val),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es],\n                              verbose=2)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:53:41.253873Z","iopub.execute_input":"2023-03-16T16:53:41.254252Z","iopub.status.idle":"2023-03-16T16:58:10.699238Z","shell.execute_reply.started":"2023-03-16T16:53:41.254218Z","shell.execute_reply":"2023-03-16T16:58:10.698127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The last validation accuracy from this experiment is compared with one without data augmentation.\n\nprint('Validation Accuracy with only Horizontal and Vertical Shifts for Data Augmentation: ', history13.history['val_accuracy'][14])\nprint('\\nValidation Accuracy without Data Augmentation: ', history5.history['val_accuracy'][14])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-16T17:00:03.257711Z","iopub.execute_input":"2023-03-16T17:00:03.258436Z","iopub.status.idle":"2023-03-16T17:00:03.266823Z","shell.execute_reply.started":"2023-03-16T17:00:03.258397Z","shell.execute_reply":"2023-03-16T17:00:03.264551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.10.3.G. Evaluating the model\n# One of the ways at which the model may be evaluated is on the plots of the accuracies and the losses\n\nplt.plot(history13.history['accuracy'])\nplt.plot(history13.history['val_accuracy'])\nplt.title('Plot Training and Validation Accuracy with Horizontal and Vertical Shift Data Augmentation')\nplt.ylabel('Accuracy')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history13.history['loss'])\nplt.plot(history13.history['val_loss'])\nplt.title('Plot Training and Validation Loss with Horizontal and Vertical Shift Data Augmentation')\nplt.ylabel('Loss')\nplt.xlabel('Number of Epochs')\nplt.legend(['Training', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:59:10.946610Z","iopub.execute_input":"2023-03-16T16:59:10.947105Z","iopub.status.idle":"2023-03-16T16:59:11.836675Z","shell.execute_reply.started":"2023-03-16T16:59:10.947066Z","shell.execute_reply":"2023-03-16T16:59:11.835739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### 3.1.3.H. Submission \n# Get the ID\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nraw_test_id=raw_test1.id\nraw_test=raw_test1.drop(\"id\",axis=\"columns\")\nraw_test=raw_test / 255\ntest=raw_test.values.reshape(-1,28,28,1)\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:59:15.791848Z","iopub.execute_input":"2023-03-16T16:59:15.792458Z","iopub.status.idle":"2023-03-16T16:59:15.826457Z","shell.execute_reply.started":"2023-03-16T16:59:15.792422Z","shell.execute_reply":"2023-03-16T16:59:15.825498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making the Predictions\nsub=model.predict(test)     #making prediction\n\n# Changing to labels\nsub=np.argmax(sub,axis=1) ##changing the prediction intro labels\n\nsample_sub['label']=sub\nsample_sub.to_csv('submission10.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T16:59:18.657244Z","iopub.execute_input":"2023-03-16T16:59:18.658029Z","iopub.status.idle":"2023-03-16T16:59:19.509286Z","shell.execute_reply.started":"2023-03-16T16:59:18.657989Z","shell.execute_reply":"2023-03-16T16:59:19.508248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.10.4. E10 Analysis","metadata":{}},{"cell_type":"markdown","source":"In analyzing the results, the validation accuracy metric was first observed and it can be seen that there is significant drop from the model that did not have any data augmentation, to the model trained with the horizontal and vertical shifts. Specifically, the model without data augmentation resulted in a value equal to 0.99450, whereas the model with the horizontal and vertical shifts results in a value for the validationa accuracy equal to 0.91425. It can be seen that this goes against the hypothesis presented, which states that there will be some improvement to the model's validatin accuracy and performance. Further analyzing this, one of the points that were not considered in formulating the hypothesis was the amount of shifts that were made. Shown above, it can be said that there are already images that have been shifted greatly where the significant information and details have already been cut off.\n\nThe results also make it clear that having significant information that is loss will result in incorrect labels for the classes and categories for the images that are being learned by the model while training. Furthermore, it has shown that when pertinent information are removed for the image, then it will result in issues and problems where weights are incorrectly updated as shown in the plots, to be further analyzed in the evaluation portion. Regardless, the experiment setup was kept as controlled as possible to ensure accurate represntation on the changes made.","metadata":{}},{"cell_type":"markdown","source":"# 3.10.5. E10 Evaluation","metadata":{}},{"cell_type":"markdown","source":"The evaluation was done qualitatively, wherein the plots of the training and validation accuracy were compared side by side, together with added metrics from the confusion matrix, and the overall accuracy score of the model trained.","metadata":{}},{"cell_type":"code","source":"# Subplots\n\nfig, axes = plt.subplots(1,2, sharey=True, figsize=(12,8))\nax5, ax6 = axes.flatten()\n\nax5.plot(history5.history['val_accuracy'])\nax5.plot(history5.history['accuracy'])\nax6.plot(history13.history['val_accuracy'])\nax6.plot(history13.history['accuracy'])\n# Legends\nax5.legend(['Training', 'Validation'], loc='lower right')\nax6.legend(['Training', 'Validation'], loc='lower right')\n# Labels\nplt.xlabel('Number of Epochs')\nplt.ylabel('Accuracy')\n# Titles\nax5.title.set_text('without Data Augmentation')\nax6.title.set_text('with Horizontal and Vertical Shifts')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T17:06:23.187745Z","iopub.execute_input":"2023-03-16T17:06:23.188463Z","iopub.status.idle":"2023-03-16T17:06:23.567833Z","shell.execute_reply.started":"2023-03-16T17:06:23.188425Z","shell.execute_reply":"2023-03-16T17:06:23.566837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The plots shown above supports the observations and insights generated in the previous section. Here, the model trained with horizontal and vertical shifts as its data augmentation resulted in significant drops in its performance levels. It was observed that there is a great degree of overfitting for this plot as the discrepancies between the training and validation curves are significant. This can again be attributed to the loss of information and details when the images are shifted to the right/left and up/down. Further into this, it can also be said that the shifts were done in amore extreme case, going from -90% to 90%, which already causes issues. As such, it can be said that when this data augmentation technique is to be used, then appropriate ranges must be set so as not to distort and compromise the quality of the data.","metadata":{}},{"cell_type":"code","source":"## CONFUSION MATRIX\ny_predicted = model.predict(x_val)\ny_grand_truth = y_val\ny_predicted = np.argmax(y_predicted,axis=1)\ny_grand_truth = np.argmax(y_grand_truth,axis=1)\nfrom sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y_grand_truth, y_predicted)\n\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(cm,fmt=\".0f\", annot=True,linewidths=0.1, linecolor=\"purple\", ax=ax)\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Ground Truth\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T17:11:50.027727Z","iopub.execute_input":"2023-03-16T17:11:50.028450Z","iopub.status.idle":"2023-03-16T17:11:53.217087Z","shell.execute_reply.started":"2023-03-16T17:11:50.028410Z","shell.execute_reply":"2023-03-16T17:11:53.215981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The errors are carried over the classification capabilities of the model as seen in the confusion matrix shown above. Here, it can be said that there are multiple instances where the model was not able to correctly classify what the digit is. For instance, the digit \n'0' was mistakenly predicted as '4' for 290 instances. This is further seen across various classes such as when '8' was predicted as '4' for 103 instances. As such, it can be said that the model performance dropped drastically, which rejects the hypothesis stated earlier.","metadata":{}},{"cell_type":"code","source":"### ACCURACY SCORE\n# The accuracy score of the entire model may be checked and tested using the evaluate function\nscore10 = model.evaluate(x_val, y_val, verbose=2);\nprint('Accuracy Score for Experiment 10: ', score10[1])\nprint('Accuracy Score for Experiment 2: ', score2[1])","metadata":{"execution":{"iopub.status.busy":"2023-03-16T17:13:40.714272Z","iopub.execute_input":"2023-03-16T17:13:40.714664Z","iopub.status.idle":"2023-03-16T17:13:43.489064Z","shell.execute_reply.started":"2023-03-16T17:13:40.714630Z","shell.execute_reply":"2023-03-16T17:13:43.487035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The accuracy score supplements the insights generated and discussed above. As such, it can be concluded that the hypothesis for this experiment is rejected, as sufficient qualitative evidences show that the model is not able to perform well already as shown in the drastic drops in the validation accuracy as assessed in the validation and training plots. This is also shown in the decrease in the classification capabilities as the confusion matrix presented several instances of the model not being able to predict the correct classes. Furthermore, the results have shown that when all other factors are kept constant and controlled, it would be relatively easier to determine and assess the effects of those variables that were changed.","metadata":{}},{"cell_type":"markdown","source":"# ***4. INSIGHTS AND CONCLUSION***","metadata":{}},{"cell_type":"markdown","source":"Throughout the conduct of the experiments, one of the recurring themes that were realized by the author is on the importance of taking things step by step. That is, each of the steps that are usually performed in solving and tackling problems in machine learning or artificial intelligence has a purpose and that changing them or manipulating such may result in significant changes in the results and the output. This was observed as various small changes and experimentations were conducted that seems to not be that thought of whenever proposes solutions using machine learning; however, when they were changed and altered, the performance of the model or the solution may drastically be affected.\n\nFurthermore, this activity paved the way for a deeper appreciation in ensuring that experiments are as scientific as possible. Given how there may be a lot of variables, particularly confounding variables that can influence the experimentation, it is necessary to keep them as controlled as possible. That is, various measures are taken  to minimize the possible effects they may bring. One of the main goals of the author in this notebook is to ensure that the experiments are done in such a way that the setups are as controlled as possible, with regards to the possible sources of deviations and errors. Hence, the author was able to realize how it can be a challenge in trying to produce a reproducible, systematic, and organized way of testing hypotheses and evaluating the results. Despite the challenges, results have shown that the scientific process empowers the author as it presents a guide in trying to arrive at solutions to the problems that may either seem daunting, simple, or even complex ones.\n\nOverall, the ten experiments conducted facilitated and paved the way for the development of a more scientific approach to testing various hypotheses, and also analyzing and assessing the best model in proposing a solution to a practical problem.","metadata":{}},{"cell_type":"markdown","source":"# ***5. REFERENCES***","metadata":{}},{"cell_type":"markdown","source":"[1]J. Brownlee, “Overfitting and Underfitting With Machine Learning Algorithms,” Machine Learning Mastery, Mar. 20, 2016. https://machinelearningmastery.com/overfitting-and-underfitting-with-machine-learning-algorithms/\n\n[2]M. Kayley, “How does epoch affect accuracy?,” Deepchecks. https://deepchecks.com/question/how-does-epoch-affect-accuracy/#:~:text=A%20very%20big%20epoch%20size (accessed Mar. 05, 2023).\n\n[3]A. Gao, “Kannada MNIST:CNN Tutorial with App.(Top %2 cce7db,” kaggle.com, Dec. 18, 2019. https://www.kaggle.com/code/andrewgao/kannada-mnist-cnn-tutorial-with-app-top-2-cce7db (accessed Mar. 06, 2023).\n\n[4]A. Takimoglu, “What is Data Augmentation? Techniques, Benefit and Examples,” research.aimultiple.com, Apr. 30, 2021. https://research.aimultiple.com/data-augmentation/\n\n[5]A. A. Awan, “A Complete Guide to Data Augmentation,” www.datacamp.com, Nov. 2022. https://www.datacamp.com/tutorial/complete-guide-data-augmentation\n\n[6]P. Vadapalli, “Types of Optimizers in Deep Learning Every AI Engineer Should Know,” upGrad blog, Oct. 06, 2022. https://www.upgrad.com/blog/types-of-optimizers-in-deep-learning/#:~:text=An%20optimizer%20is%20an%20algorithm (accessed Mar. 06, 2023).\n\n[7]R. Gandhi, “A Look at Gradient Descent and RMSprop Optimizers,” Medium, Jun. 19, 2018. https://towardsdatascience.com/a-look-at-gradient-descent-and-rmsprop-optimizers-f77d483ef08b#:~:text=RMSprop%20Optimizer (accessed Mar. 06, 2023).\n\n[8]S. Doshi, “Various Optimization Algorithms For Training Neural Network,” Medium, Aug. 03, 2020. https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6#:~:text=The%20right%20optimization%20algorithm%20can (accessed Mar. 07, 2023).\n\n[9]P. Baheti, “Train Test Validation Split: How To & Best Practices [2023],” www.v7labs.com, Mar. 02, 2023. https://www.v7labs.com/blog/train-validation-test-set#:~:text=The%20training%20set%20should%20not (accessed Mar. 08, 2023).\n\n[10]W.-M. Lee, “Image Data Augmentation for Deep Learning,” Medium, Dec. 22, 2022. https://towardsdatascience.com/image-data-augmentation-for-deep-learning-77a87fabd2bf#:~:text=The%20ImageDataGenerator%20class%20allows%20your (accessed Mar. 08, 2023).\n\n[11]J. S. Jun 24 and Read    2020 9 M., “Why and How to Implement Random Rotate Data Augmentation,” Roboflow Blog, Jun. 24, 2020. https://blog.roboflow.com/why-and-how-to-implement-random-rotate-data-augmentation/\n\n[12]N. Balla, “Five Popular Data Augmentation Techniques In Deep Learning,” Dataaspirant, Aug. 31, 2020. https://dataaspirant.com/data-augmentation-techniques-deep-learning/#t-1598844713774 (accessed Mar. 9, 2023).\n\n[13]A. Kathuria, “Data Augmentation for object detection: How to Rotate Bounding Boxes,” Paperspace Blog, Sep. 12, 2018. https://blog.paperspace.com/data-augmentation-for-object-detection-rotation-and-shearing/ (accessed Mar. 09, 2023).\n\n[14]R. Khandelwal, “Data Augmentation techniques in Python,” Medium, Dec. 11, 2019. https://towardsdatascience.com/data-augmentation-techniques-in-python-f216ef5eed69 (accessed Mar. 09, 2023).\n\n[15]“Horizontal And Vertical Shift Data Augmentation - Studytonight,” www.studytonight.com. https://www.studytonight.com/post/horizontal-and-vertical-shift-data-augmentation (accessed Mar. 09, 2023).","metadata":{}},{"cell_type":"markdown","source":"\n","metadata":{}}]}