{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Kannada MNIST with Keras CNN"},{"metadata":{},"cell_type":"markdown","source":"# Table of contents\n\n1. [Context](#context)  \n2. [Importations](#importations)  \n3. [Informations](#informations)\n4. [Set parameters](#set_parameters)\n5. [Data exploration](#data_exploration)  \n    5.1 [Import data](#import_data)  \n    5.2 [General analysis](#general_analysis)   \n    5.3 [Visualization](#visualization)\n6. [Modelisation](#modelisation)  \n    6.1 [Learning](#learning)    \n    6.2 [Learning curves](#learning_curves)  \n    6.3 [Learning rate](#learning_rate)  \n    6.4 [Results](#results)  \n    6.5 [Visualize](#visualize)  \n7. [Submission](#submission)\n8. [Conclusion](#conclusion)\n9. [References](#references)"},{"metadata":{},"cell_type":"markdown","source":"# 1. Context <a id=\"context\"></a>"},{"metadata":{},"cell_type":"markdown","source":"<p style=\"text-align:center;\">\n    <img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/16017/logos/header.png?t=2019-09-11-17-07-02\" style=\"height:100%; width:100%\"/>\n</p>\n\nThe goal of this competition is to provide a simple extension to the classic MNIST competition we're all familiar with. Instead of using Arabic numerals, it uses a recently-released dataset of Kannada digits."},{"metadata":{},"cell_type":"markdown","source":"# 2. Importations <a id=\"importations\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Linear algebra and data processing\nimport numpy as np\nimport pandas as pd\nimport math\nimport random\n\n# Get version python/keras/tensorflow/sklearn\nfrom platform import python_version\nimport sklearn\nimport keras\nimport tensorflow as tf\n\n# Folder manipulation\nimport os\n\n# Spliting data\nfrom sklearn.model_selection import train_test_split\n\n# Keras importation\nfrom keras.utils import to_categorical\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau\nfrom keras.optimizers import Adam\nfrom keras.models import Model, Sequential\nfrom keras.layers import Input, Conv2D, Flatten, MaxPooling2D\nfrom keras.layers.core import Dense, Dropout, Activation\nfrom keras.layers.normalization import BatchNormalization\n\n# For images augmentations\nimport albumentations as albu\n\n# Visualizaton\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Informations <a id=\"informations\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir(\"../input\"))\nprint(\"Keras version : \" + keras.__version__)\nprint(\"Tensorflow version : \" + tf.__version__)\nprint(\"Python version : \" + python_version())\nprint(\"Sklearn version : \" + sklearn.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Set parameters <a id=\"set_parameters\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"MAIN_DIR = \"../input/Kannada-MNIST/\"\n\nDATA_TRAIN = MAIN_DIR + \"train.csv\"\nDATA_TEST = MAIN_DIR + \"test.csv\"\n\nIMG_HEIGHT = 28\nIMG_WIDTH = 28\nCHANNELS = 1\nIMG_SHAPE = (IMG_WIDTH, IMG_HEIGHT, CHANNELS)\n\n# Set graph font size\nsns.set(font_scale=1.3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# 5. Data exploration <a id=\"data_exploration\"></a>"},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Import data <a id=\"import_data\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_data():\n    df_train =  pd.read_csv(DATA_TRAIN)\n    df_test =  pd.read_csv(DATA_TEST)\n    return df_train, df_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_train, data_test = load_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Training data has shape : {data_train.shape}\")\nprint(f\"Test data has shape : {data_test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 General analysis <a id=\"general_analysis\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Data train has {data_train.isna().sum().sum(axis=0)} Nan values\")\nprint(f\"Data test has {data_test.isna().sum().sum(axis=0)} Nan values\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(6, 4))\nsns.countplot(x='label', data=data_train, ax=ax)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.3 Visualization <a id=\"visualization\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_images(data, nb_rows=6, nb_cols=6, figsize=(14, 14)):\n    # Get data\n    df = data.copy()\n    X_raw = df.drop(['label'], axis=1).values\n    X = np.reshape(X_raw, (X_raw.shape[0], IMG_WIDTH, IMG_HEIGHT))\n    y = df['label'].values\n        \n    # Set up the grid\n    fig, ax = plt.subplots(nb_rows, nb_cols, figsize=figsize, gridspec_kw=None)\n    fig.subplots_adjust(wspace=0.4, hspace=0.4)\n\n    for i in range(0, nb_rows):\n        for j in range(0, nb_cols):\n            index = np.random.randint(0, X.shape[0])\n    \n            # Hide grid\n            ax[i, j].grid(False)\n            ax[i, j].axis('off')\n            \n            # Plot picture on grid\n            ax[i, j].imshow(X[index].astype(np.int), cmap='gray')\n            ax[i, j].set_title(f\"Label : {y[index]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images(data_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# 6. Modelisation <a id=\"modelisation\"></a>"},{"metadata":{},"cell_type":"markdown","source":"For more information about fine-tunning a neural networks see more on [[1]](https://karpathy.github.io/2019/04/25/recipe/).\n\nFor creating your Keras data generator see more on [[2]](https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly).\n\nFor more information about data augmentation see more on [[3]](https://github.com/albu/albumentations)."},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self,\n                 data,\n                 list_IDs, \n                 labels=None,\n                 batch_size=32,\n                 dim=IMG_SHAPE,\n                 n_channels=1,\n                 augment=False,\n                 n_classes=10,\n                 mode='fit',\n                 shuffle=True,\n                 random_state=42):\n        \n        'Initialization'\n        self.data = data\n        self.dim = dim\n        self.batch_size = batch_size\n        self.labels = labels\n        self.list_IDs = list_IDs\n        self.n_channels = n_channels\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.augment = augment\n        self.mode = mode\n        self.random_state = random_state\n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        if(self.mode == 'fit'):\n            return int(np.floor(len(self.list_IDs) / self.batch_size))\n        else:\n            return int(np.ceil(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Augment 1 batch over 2\n        augment_batch = random.choice([True, False])\n        \n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_batch = [self.list_IDs[k] for k in indexes]\n\n        # Generate input\n        X = self.__generate_X(list_IDs_batch)\n        \n        # Generate target or predict        \n        if(self.mode == 'fit'):\n            y = self.__generate_y(list_IDs_batch)\n            \n            if(self.augment and augment_batch):\n                X = self.__augment_batch(X)\n            \n            return X, y\n        \n        elif self.mode == 'predict':\n            return X\n        else:\n            raise AttributeError('The mode parameter should be set to \"fit\" or \"predict\".')\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indexes)\n\n    def __generate_X(self, list_IDs_batch):\n        'Generates data containing batch_size samples'\n        # Initialization\n        X = np.empty((len(list_IDs_batch), *self.dim))\n        \n        # Generate data\n        for i, ID in enumerate(list_IDs_batch):\n            # Store sample\n            X[i,] = np.reshape(self.data[ID], self.dim)\n\n        return X\n    \n    def __generate_y(self, list_IDs_batch):\n        y = np.empty((self.batch_size), dtype=int)\n        \n        for i, ID in enumerate(list_IDs_batch):\n            # Generate data\n            y[i, ] = self.labels[ID]\n\n        return keras.utils.to_categorical(y, num_classes=self.n_classes)\n    \n    def __random_transform(self, img):\n        composition = albu.Compose([\n            albu.ShiftScaleRotate(rotate_limit=10, shift_limit=0.15, scale_limit=0.1)\n        ])\n        \n        composed = composition(image=img)        \n        aug_img = composed['image']\n        \n        return aug_img\n    \n    def __augment_batch(self, img_batch):\n        for i in range(img_batch.shape[0]):\n            img_batch[i, ] = self.__random_transform(img_batch[i, ])\n        \n        return img_batch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model():\n    input_layer = Input(shape=IMG_SHAPE)\n    \n    x = Conv2D(32, (3,3), strides=1, padding=\"same\", name=\"conv1\")(input_layer)\n    x = BatchNormalization(name=\"batch1\")(x)\n    x = Activation('relu', name='relu1')(x)\n    x = MaxPooling2D(pool_size=2, strides=2, padding=\"valid\", name=\"max2\")(x)\n    x = Conv2D(32, (3, 3), padding='same', name=\"conv2\")(x)\n    x = BatchNormalization(name=\"batch2\")(x)\n    x = Activation('relu', name='relu2')(x)\n    x = Dropout(0.4, name='dropout1')(x)\n    \n    x = Conv2D(64, (3,3), strides=1, padding=\"same\", name=\"conv3\")(x)\n    x = BatchNormalization(name=\"batch3\")(x)\n    x = Activation('relu', name='relu3')(x)\n    x = MaxPooling2D(pool_size=2, strides=2, padding=\"valid\", name=\"max3\")(x)\n    x = Conv2D(64, (3, 3), padding='same', name=\"conv4\")(x)\n    x = BatchNormalization(name=\"batch4\")(x)\n    x = Activation('relu', name='relu4')(x)\n    x = Dropout(0.4, name='dropout2')(x)\n    \n    x = Conv2D(128, (3,3), strides=1, padding=\"same\", name=\"conv5\")(x)\n    x = BatchNormalization(name=\"batch5\")(x)\n    x = Activation('relu', name='relu5')(x)\n    x = MaxPooling2D(pool_size=2, strides=2, padding=\"valid\", name=\"max4\")(x)\n    x = Conv2D(128, (3, 3), padding='same', name=\"conv6\")(x)\n    x = BatchNormalization(name=\"batch6\")(x)\n    x = Activation('relu', name='relu6')(x)\n    x = Dropout(0.4, name='dropout3')(x)\n    \n    x = Flatten(name='flatten')(x)\n    \n    x = Dense(128, name='dense1')(x)\n    x = BatchNormalization(name=\"batch7\")(x)\n    x = Activation('relu', name=\"relu7\")(x)\n    x = Dropout(0.45, name=\"dropout4\")(x)\n    \n    x = Dense(10, activation='softmax', name=\"dense2\")(x)\n    \n    model = Model(inputs=input_layer, outputs=x)\n    model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=1e-3), metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.1 Learning <a id=\"learning\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_generators(data_train, data_test):\n    # Set variables\n    df_train = data_train.copy()\n    df_test = data_test.copy()\n\n    # Spliting data by gettig index\n    train_idx, val_idx = train_test_split(\n        df_train.index, random_state=42, test_size=0.1, shuffle=True\n    )\n    \n    # Parameters for generator # Put outside of the generator\n    params = {'dim': IMG_SHAPE,\n              'batch_size': 32,\n              'n_classes': 10,\n              'n_channels': 1}\n    \n    # Generators train/val\n    X = df_train.drop(['label'], axis=1).values\n    y = df_train['label'].values\n    \n    training_generator = DataGenerator(data=X, \n                                       labels=y,\n                                       list_IDs=train_idx,\n                                       **params,\n                                       mode='fit',\n                                       shuffle=True, augment=False)\n    validation_generator = DataGenerator(data=X,\n                                         labels=y,\n                                         list_IDs=val_idx,\n                                         **params,\n                                         mode='fit',\n                                         shuffle=True, augment=False)\n    \n    # Generator test\n    X_test = df_test.drop(['id'], axis=1).values\n    test_generator = DataGenerator(data=X_test,\n                                   list_IDs=df_test.index.values,\n                                   **params,\n                                   mode='predict',\n                                   shuffle=False)\n    \n    return training_generator, validation_generator, test_generator","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(gen_train, gen_val):\n    model = build_model()\n\n    cbs = [ReduceLROnPlateau(monitor='loss',\n                             factor=0.5,\n                             patience=1,\n                             min_lr=1e-5,\n                             verbose=0,\n                             skip_mismatch=True),\n           EarlyStopping(monitor='val_loss',\n                         min_delta=1e-6,\n                         patience=10,\n                         verbose=1,\n                         mode='auto',\n                         restore_best_weights=True)]\n    model.summary()\n    history = model.fit_generator(gen_train, \n                        epochs=50,\n                        validation_data=gen_val, \n                        validation_steps=len(gen_val), \n                        shuffle=True, \n                        callbacks=cbs, \n                        verbose=1)\n    return model, history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_gen, val_gen, test_gen = get_generators(data_train, data_test)\nmodel, history = train_model(train_gen, val_gen)\ny_pred = model.predict_generator(test_gen, use_multiprocessing=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2 Learning curves <a id=\"learning_curves\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_loss(history):\n    fig, ax = plt.subplots(1, 2, figsize=(15, 5))\n    \n    # Plot train/val accuracy\n    ax[0].plot(history.history['accuracy'])\n    ax[0].plot(history.history['val_accuracy'])\n    ax[0].set_title('Model accuracy')\n    ax[0].set_ylabel('Accuracy')\n    ax[0].set_xlabel('Epochs')\n    ax[0].legend(['Train', 'Test'], loc='lower right')\n    ax[0].set_ylim(0, 1.05)\n    \n    # Plot train/val loss\n    ax[1].plot(history.history['loss'])\n    ax[1].plot(history.history['val_loss'])\n    ax[1].set_title('Model Loss')\n    ax[1].set_ylabel('Loss')\n    ax[1].set_xlabel('Epochs')\n    ax[1].legend(['Train', 'Test'], loc='upper right')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.3 Learning rates <a id=\"learning_rates\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_lr(history):\n    fig, ax = plt.subplots(figsize=(7, 5))\n    \n    # Plot learning rate\n    ax.plot(history.history['lr'])\n    ax.set_title('Learning rate evolution')\n    ax.set_ylabel('Learning rate value')\n    ax.set_xlabel('Epochs')\n    ax.legend(['Train'], loc='upper right')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_lr(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.4 Results <a id=\"results\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_results(history):\n    print(\"ACCURACY :\")\n    print(f\"Training accuracy : {history.history['accuracy'][-1]}\")\n    print(f\"Validation accuracy : {history.history['val_accuracy'][-1]}\")\n    \n    print(\"\\nLOSS :\")\n    print(f\"Training categorical crossentropy loss : {history.history['loss'][-1]}\")\n    print(f\"Validation categorical crossentropy loss : {history.history['val_loss'][-1]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_results(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.5 Visualize <a id=\"visualize\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_test_label = data_test.copy() \ny_pred_label = np.argmax(y_pred, axis=1)\ndata_test_label['label'] = y_pred_label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images(data_test_label.drop(['id'], axis=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Submission <a id=\"submission\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_submission(data):\n    df = pd.DataFrame()\n    df['id'] = data['id']\n    df['label'] = data['label']\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = create_submission(data_test_label)\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8. Conclusion <a id=\"conclusion\"></a>"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"This very simple model seems to have good performances on Kannada digits. A way of improvement could be to change the hyperparameters of **BatchNormalization** layers or to try a different architectures."},{"metadata":{},"cell_type":"markdown","source":"# 9. References <a id=\"references\"></a>"},{"metadata":{},"cell_type":"markdown","source":"[[1]](https://karpathy.github.io/2019/04/25/recipe/) How to fine tune a neural network.  \n[[2]](https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly) Create your Keras generator from scratch.  \n[[3]](https://github.com/albu/albumentations) Data augmentation library."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}