{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **Understanding CNNs with Kannada-MNIST**\n\n\n# Imports\nThese are the imports we need to get started with machine leraning\n- Pandas - Used for handling csv data\n- MatPlotLib - Used for plotting graphs\n- Keras - A popular deep learning library we will use to create our CNN\n- SKLearn - Popular machine learning library we will use to split our data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, MaxPooling2D\nfrom keras.optimizers import RMSprop\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom keras.utils.np_utils import to_categorical\n\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data\nWe will load the data, split it, artificially generate more data to make it more diverse and then make it ready for training"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the data\nDig_MNIST = pd.read_csv(\"../input/Kannada-MNIST/Dig-MNIST.csv\")\nsample_submission = pd.read_csv(\"../input/Kannada-MNIST/sample_submission.csv\")\ntest = pd.read_csv(\"../input/Kannada-MNIST/test.csv\")\ntrain = pd.read_csv(\"../input/Kannada-MNIST/train.csv\")\n\n# Split the data\nx = train.iloc[:,1:].values\ny = train.iloc[:,0].values\ny[:10]\nx_test = test.drop('id', axis=1).iloc[:,:].values\nx_dig = Dig_MNIST.drop('label', axis=1).iloc[:,:].values\ny_dig = Dig_MNIST.label\n\n# Reshape the data\nx = x.reshape(x.shape[0], 28, 28, 1)\ny = to_categorical(y, 10)\nx_test = x_test.reshape(x_test.shape[0], 28, 28, 1)\nx_dig = x_dig.reshape(x_dig.shape[0], 28, 28, 1)\n\n# Split the data between train and test\nx_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.10, random_state=42) \n\n# Artificially increase training set\ntrain_datagen = ImageDataGenerator(rescale=1./255.,\n                                   rotation_range=10,\n                                   width_shift_range=0.25,\n                                   height_shift_range=0.25,\n                                   shear_range=0.1,\n                                   zoom_range=0.25,\n                                   horizontal_flip=False)\n\nvalid_datagen = ImageDataGenerator(rescale=1./255.)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The CNN\n\n## Simple explanation of what a CNN is\n![](https://i.imgur.com/Qd9JpWj.png)\nConvolutional neural network (CNN) is a type of neural network architecture specially made to deal with visual data. In this article we will discuss the architecture of CNN and implement it on CIFAR-10 dataset in part-2. The main benefit of using a CNN over simple ANN on visual data is that CNN’s are constrained to deal with image data exclusively. Two main features of CNNs are\n\n- Weight sharing\n- Feature extractors\n\n![](https://i.imgur.com/RYMoJpL.png)\n\nAs we described above, a simple ConvNet is a sequence of layers, and every layer of a ConvNet transforms one volume of activations to another through a differentiable function. We use three main types of layers to build ConvNet architectures: Convolutional Layer, Pooling Layer, and Fully-Connected Layer (exactly as seen in regular Neural Networks). We will stack these layers to form a full ConvNet architecture.\n\nSeveral new layers are introduced in CNNs to extract the useful features from our image or reducing the size of image without using the original representation.\n\n### Convolution Layer\nConvolutional layer apply convolution operation on the input layer, passing the results to next layer. A convolution operation is basically computing a dot product between their weights and a small region they are connected(currently overlapping) to in the input volume. This will change the dimensions depending on the filter size used and number of filters used.\n\nWe can compute the spatial size of the output volume as a function of the input volume size (W), the receptive field size of the Conv Layer neurons (F), the stride with which they are applied (S), and the amount of zero padding used (P) on the border. You can convince yourself that the correct formula for calculating how many neurons “fit” is given by (W−F+2P)/S+1. For example for a 7x7 input and a 3x3 filter with stride 1 and pad 0 we would get a 5x5 output. With stride 2 we would get a 3x3 output.\n\n### ReLU Layer\nRectifying Linear Unit (ReLU) layer applies the relu activation element-wise. It is a mathematical function, which returns a positive value or 0 in place of previous negative values, It does not change the dimensions of the previous layer.\n\n### Pooling Layer\nPooling layer will perform a down-sampling operation along the width and resulting in the reduction of the dimensions. The sole purpose of pooling is to reduce spatial dimensions. There are various types of pooling in which the most common is Max Pooling, i.e taking the maximum element from the window.\n![](https://i.imgur.com/5Q54piT.png)\n\n### Stride\nStride decides by how much we move our window ,when we have a stride of one we move across and down a single pixel. With higher stride values, we move large number of pixels at a time and hence produce smaller output volumes.\n\n### Padding\nPadding is used to preserve the boundary information , since without padding they are only traversed once.\n![](https://i.imgur.com/MWorvXh.png)\n\n### Flattening Layer\nThis layer will convert the 3-dimensions (height,width,depth) into a single long vector to feed it to the fully connected layer or Dense layer. It connects every neuron in one layer to every neuron in another layer.\n\nFully Connected Layer and Output Layer\nFully connected layers or dense layers are the same hidden layers consisting of defined number of neurons connected with elements of another layer that we discussed in simple ANN. However the output layer is also the same but the number of neurons depend on our task. For instance in CIFAR-10 dataset we have 10 classes hence we will have 10 neurons in the outer layer.\n\n### Summary\n![](https://i.imgur.com/x0DIK9L.jpg)\n\nIn summary, the architecture of CNN , we can simply understand that it consist of an input layer followed by a Conv layer. The dimensions of conv layer depends on the data and problem, hence changing the dimensions accordingly. After the Conv Layer there is a activation layer , usually ReLU since it gives better results. After some conv and relu combination , pooling layer is used to reduce the size. Then after some combination of previously defined architecture , flattening layer is used to flatten the input for fully connected layer. Next to these layer, the last layer is the output layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 1024\nnum_classes = 10\nepochs = 50\nlearning_rate = 0.001\nmodel_name = 'k-mnist_trained_model.h5'\n\nmodel = Sequential()\n\nmodel.add(Conv2D(64, kernel_size=3, activation='relu', input_shape=(28, 28, 1)))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(64, kernel_size=3, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(64, kernel_size=5, padding='same', activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(128, kernel_size=3, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128, kernel_size=3, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128, kernel_size=5, padding='same', activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Conv2D(256, kernel_size=3, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.2))\n\nmodel.add(Flatten())\nmodel.add(Dense(256))\nmodel.add(BatchNormalization())\nmodel.add(Dense(128))\nmodel.add(BatchNormalization())\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.summary()\n\noptimizer = RMSprop(lr=learning_rate)\n\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=optimizer,\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fitting the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate_reduction = ReduceLROnPlateau(monitor='val_loss', \n                                            patience=200,\n                                            verbose=1,\n                                            factor=0.2)\n\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)\n\nhistory = model.fit_generator(train_datagen.flow(x_train, y_train, batch_size=batch_size),\n                              steps_per_epoch=100,\n                              epochs=epochs,\n                              validation_data=valid_datagen.flow(x_valid, y_valid),\n                              validation_steps=50,\n                              callbacks=[learning_rate_reduction, es])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualising accuracy and loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot training & validation accuracy values\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make the submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict_classes(x_test/255.)\nsubmission = pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nsubmission['label'] = predictions\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}