{"cells":[{"metadata":{},"cell_type":"markdown","source":"1. Introduction\n2. Data preparation\n3. Data augmentation\n4. CNN + Batch normalization\n5. CNN + Independent-Component layer\n6. Submission"},{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction"},{"metadata":{},"cell_type":"markdown","source":"Batch Normalization and Dropout are among the most used techniques in DNN because of their ease of implementation and beneficial returns. \n\nIn a few words, [Dropout](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) is a regularization technique that essentially consists in introducing some noise into a network by randomly setting some of the dimensions of the input vector to be zero with probability p. It helps reduce overfitting, and force the network to learn to generalize better. \nOn another side, [Batch Normaliaztion](https://arxiv.org/pdf/1502.03167.pdf) is an optimization technique that dynamically \"normalize\" the inputs by both mean and variance reference on a per mini-batch basis. It usually results in a network that converges faster and generalizes well.\n\nHowever, it is known that BatchNormalization and Dropout don't play well together [(Li et al. (2018))](https://arxiv.org/abs/1801.05134). The aim of this notebook is to try the implementation of the Independent-Component layer (IC) as defined in [Chen et al. (2019)](http://arxiv.org/abs/1905.05928), which offers a simple solution on how to reconciliate Batch Normalization with Dropout units.\n\nCheck [here](https://www.kaggle.com/schateau/maxout-dropout) for an implementation of a Maxout network [(Goodfellow et al. (2013))](https://arxiv.org/pdf/1302.4389.pdf)."},{"metadata":{},"cell_type":"markdown","source":"# 2. Data preparation"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Loading librairies:\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport random\n\n# Input data files are available in the \"../input/\" directory.\nimport os\nprint(os.listdir(\"../input/dogs-vs-cats\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Prepare training data\nfilenames = os.listdir(\"../input/dogs-vs-cats/train/train\")\ncategories = []\nfor filename in filenames:\n    category = filename.split('.')[0]\n    if category == 'dog':\n        categories.append(1)\n    else:\n        categories.append(0)\n\nprint('1=dog; 0=cat')\ndf = pd.DataFrame({\n    'filename': filenames,\n    'category': categories\n})\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#How are the pictures distributed ?\ndf['category'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"427300c6-b035-4c35-9573-abae8d379f74","_cell_guid":"208f79d7-15b9-4a96-b710-c938dd81ba7d","trusted":true},"cell_type":"code","source":"#See a random sample:\nfrom keras.preprocessing.image import load_img\n\nsample = random.choice(filenames)\nimage = load_img(\"../input/dogs-vs-cats/train/train/\"+sample)\nplt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#define image_shape:\nheight = 150\nwidth = 150\nchannels = 3\nimage_shape = (height, width, channels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Because we'll use a generator with binary classification for the training set, we must pass from 'int' to 'string' for the y_col=\"category\" column\ndf[\"category\"] = df[\"category\"].replace({0: 'cat', 1: 'dog'}) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Because we want to use the \".evaluate_generator\" method that will evaluate our models, we have to reserve a small subset of df as a test set (we need the labels)."},{"metadata":{"trusted":true},"cell_type":"code","source":"#splitting into training, validation, and test sets:\nfrom sklearn.model_selection import train_test_split\n\n#validation set:\ntrain_df_tmp, validate_df = train_test_split(df, test_size=0.1, random_state=2)\n#training and test sets:\ntrain_df, test_df = train_test_split(train_df_tmp, test_size=0.1, random_state=2)\n\n#print the number of samples in each set:\nprint('Number of samples in train_df:', len(train_df), \n      '\\nNumber of samples in validate_df:', len(validate_df)\n     ,'\\nNumber of samples in test_df:', len(test_df))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Since i will train and test different models, i choose to reduce the sizes of the training set to save some GPU time.\n#Here, i cut train_df in half\ntrain_df = train_df.sample(n=10000).reset_index() # use for fast testing code purpose\nvalidate_df = validate_df.reset_index()\ntest_df = test_df.reset_index()\n\n#If you want to train on the full set, uncomment the following line:\n#train_df = train_df.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 64\ntotal_train = train_df.shape[0]\ntotal_validate = validate_df.shape[0]\nepochs = 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Data augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Data preprocessing:\nfrom keras.preprocessing.image import ImageDataGenerator\n\n#ImageDataGenerator with data augmentation:\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest')\n\n#Generator for training:\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_df,\n    directory='../input/dogs-vs-cats/train/train/', \n    x_col='filename',\n    y_col='category',\n    class_mode='binary',\n    target_size=(height, width),\n    batch_size=batch_size\n)\n\n#The images outputed by the validation generator should not be augmented:\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validate_df, \n    directory='../input/dogs-vs-cats/train/train/',\n    x_col='filename',\n    y_col='category',\n    class_mode='binary',\n    target_size=(height, width),\n    batch_size=batch_size\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Displaying some randomly augmented training images\nexample_df = train_df.sample(n=1).reset_index(drop=True)\nexample_generator = train_datagen.flow_from_dataframe(\n    dataframe=example_df,\n    directory='../input/dogs-vs-cats/train/train/', \n    x_col='filename',\n    y_col='category',\n    class_mode='categorical'\n)\nplt.figure(figsize=(12, 12))\nfor i in range(0, 9):\n    plt.subplot(3, 3, i+1)\n    for X_batch, Y_batch in example_generator:\n        image = X_batch[0]\n        plt.imshow(image)\n        break\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. CNN + Batch normalization"},{"metadata":{},"cell_type":"markdown","source":"The benefits from Batch Normalization are multiple:\n1. Allows higher learning rates, resulting in networks that train faster and converge much more quickly.\n2. Makes the weights easier to initialize.\n3. Usually gives better results\n4. Makes more activation functions viable.\n\n[Click here for more infos](https://www.dlology.com/blog/one-simple-trick-to-train-keras-model-faster-with-batch-normalization/)"},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Model definition"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Building a network using Batch_normalization for fast training\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Activation, BatchNormalization\nfrom keras import optimizers\n\nBN_model = Sequential(name='BN_model')\n\n#Note: a common practice is to place BN before the activation if RelU is employed. This is questionned by https://arxiv.org/pdf/1905.05928.pdf\nBN_model.add(Conv2D(32, (3, 3), input_shape=image_shape, use_bias=False)) #Batchnorm layers already include a bias term.\nBN_model.add(BatchNormalization()) \nBN_model.add(Activation(\"relu\"))\nBN_model.add(MaxPooling2D((2, 2)))\n\nBN_model.add(Conv2D(64, (3, 3), use_bias=False))\nBN_model.add(BatchNormalization())\nBN_model.add(Activation(\"relu\"))\nBN_model.add(MaxPooling2D((2, 2)))\n\nBN_model.add(Conv2D(128, (3, 3), use_bias=False))\nBN_model.add(BatchNormalization())\nBN_model.add(Activation(\"relu\"))\nBN_model.add(MaxPooling2D((2, 2)))\n\nBN_model.add(Conv2D(128, (3, 3), use_bias=False))\nBN_model.add(BatchNormalization())\nBN_model.add(Activation(\"relu\"))\nBN_model.add(MaxPooling2D((2, 2)))\n\nBN_model.add(Flatten())\n\nBN_model.add(Dense(512, use_bias=False))\nBN_model.add(BatchNormalization())\nBN_model.add(Activation(\"relu\"))\n\nBN_model.add(Dense(1, use_bias=False))\nBN_model.add(BatchNormalization())\nBN_model.add(Activation(\"sigmoid\"))\n\nmodel_name = BN_model.name\nBN_model.compile(loss='binary_crossentropy',\n                 optimizer=optimizers.RMSprop(lr=1e-3), #Because of BN, we can use larger learning rate\n                 metrics=['acc'])\n\nBN_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Callbacks"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\n\npatience_earlystop = 20\npatience_ReduceLROnPlateau = 10\n\nfilepath = 'best_{0}'.format(model_name) + '-{epoch:03d}-{val_acc:03f}.h5'\nmcp = ModelCheckpoint(filepath, monitor='val_loss', mode='min', save_best_only=True, verbose=1)\nearlystop = EarlyStopping(monitor='val_loss',\n                          mode='min',\n                          patience=patience_earlystop,\n                          verbose=1)\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=patience_ReduceLROnPlateau, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=1e-5)\ncallbacks = [mcp, earlystop, learning_rate_reduction]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fitting the model using a batch generator:\nhistory = BN_model.fit_generator(\n    train_generator,\n    steps_per_epoch=total_train//batch_size,\n    epochs=epochs,\n    validation_data=validation_generator,\n    validation_steps=total_validate//batch_size,\n    callbacks=callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4 Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define a smooth function to display the training and validation curves\ndef plot_smoothed_learning_curves(history):\n    val_loss = history.history['val_loss']#[-30:-1] #Uncomment if you want to see only the last epochs\n    loss = history.history['loss']#[-30:-1]\n    acc = history.history['acc']#[-30:-1]\n    val_acc = history.history['val_acc']#[-30:-1]\n    \n    epochs = range(1, len(acc)+1 )\n    \n    # Plot the loss and accuracy curves for training and validation \n    fig, ax = plt.subplots(2,1, figsize=(12, 12))\n    ax[0].plot(epochs, smooth_curve(loss), 'bo', label=\"Smoothed training loss\")\n    ax[0].plot(epochs, smooth_curve(val_loss), 'b', label=\"Smoothed validation loss\",axes =ax[0])\n    legend = ax[0].legend(loc='best', shadow=True)\n    ax[0].set_xlabel('Epochs')\n    ax[0].set_ylabel('Loss')\n\n    ax[1].plot(epochs, smooth_curve(acc), 'bo', label=\"Smoothed training accuracy\")\n    ax[1].plot(epochs, smooth_curve(val_acc), 'b',label=\"Smoothed validation accuracy\")\n    legend = ax[1].legend(loc='best', shadow=True)\n    ax[1].set_xlabel('Epochs')\n    ax[1].set_ylabel('Accuracy')\n    return\n\ndef smooth_curve(points, factor=0.8):\n    smoothed_points = []\n    for point in points:\n        if smoothed_points:\n            previous = smoothed_points[-1]\n            smoothed_points.append(previous*factor + point*(1-factor))\n        else:\n            smoothed_points.append(point)\n    return smoothed_points","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualisation:\nplot_smoothed_learning_curves(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.5 Evaluation of the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Define test_generator:\ntest_datagen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_datagen.flow_from_dataframe(\n    dataframe = test_df, \n    directory=\"../input/dogs-vs-cats/train/train/\", \n    x_col='filename',\n    y_col='category',\n    class_mode='binary',\n    target_size=(height, width),\n    batch_size=batch_size,\n    shuffle=False\n)\n\nnb_samples = test_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Evaluate the model:\ntest_loss, test_acc = BN_model.evaluate_generator(test_generator, steps=np.ceil(nb_samples/batch_size))\nprint('test acc:', test_acc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.6 Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Load test data:\ntest_filenames = os.listdir(\"../input/dogs-vs-cats/test1/test1\")\nreal_test_df = pd.DataFrame({\n    'filename': test_filenames\n})\n\nreal_test_df = real_test_df\nnb_samples = real_test_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Generator:\nreal_test_generator = test_datagen.flow_from_dataframe(\n    dataframe = real_test_df, \n    directory=\"../input/dogs-vs-cats/test1/test1/\", \n    x_col='filename',\n    y_col=None,\n    class_mode=None,\n    target_size=(height, width),\n    batch_size=batch_size,\n    shuffle=False\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Prediction:\npredict = BN_model.predict_generator(real_test_generator, steps=np.ceil(nb_samples/batch_size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"real_test_df['category'] = predict.round().astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"real_test_df['category'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.7 Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = real_test_df.copy()\nsubmission_df['id'] = submission_df['filename'].str.split('.').str[0]\nsubmission_df['label'] = submission_df['category']\nsubmission_df.drop(['filename', 'category'], axis=1, inplace=True)\nsubmission_df.to_csv('submission_BN.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. CNN + Independent-Component layer"},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Model definition"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Building a networl with IC layers:\nfrom keras import Input\nfrom keras.models import Model\nfrom keras.constraints import max_norm\nfrom keras.layers import Dropout\n\nmax_norm4 = max_norm(max_value=4, axis=[0, 1, 2])\n\ndef IC(inputs, p):\n    x = BatchNormalization()(inputs)\n    x = Dropout(p)(x)\n    return x\n\ninput_tensor = Input(shape=image_shape)\n\nx = Activation(\"relu\")(input_tensor)\nx = IC(x, 0.03) #We use a very small value of dropout, as advised in Chen et al. (2019)\nx = Conv2D(32, (3, 3), use_bias=False, kernel_constraint=max_norm4)(x)\nx = MaxPooling2D((2, 2))(x)\n\nx = Activation(\"relu\")(x)\nx = IC(x, 0.03)\nx = Conv2D(64, (3, 3), use_bias=False, kernel_constraint=max_norm4)(x)\nx = MaxPooling2D((2, 2))(x)\n\nx = Activation(\"relu\")(x)\nx = IC(x, 0.03)\nx = Conv2D(64, (3, 3), use_bias=False, kernel_constraint=max_norm4)(x)\nx = MaxPooling2D((2, 2))(x)\n\nx = Activation(\"relu\")(x)\nx = IC(x, 0.03)\nx = Conv2D(128, (3, 3), use_bias=False, kernel_constraint=max_norm4)(x)\nx = MaxPooling2D((2, 2))(x)\n\nx = Flatten()(x)\n\nx = Activation(\"relu\")(x)\nx = IC(x, 0.03)\nx = Dense(512, use_bias=False, kernel_constraint=max_norm(4))(x)\n\noutput_tensor = Dense(1, activation=\"sigmoid\")(x)\n\nIC_model = Model(input_tensor, output_tensor)\nmodel_name = IC_model.name\n\nIC_model.compile(loss='binary_crossentropy',\n                 optimizer=optimizers.rmsprop(lr=1e-3),\n                 metrics=['acc'])\n\nIC_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Callbacks"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Here, we just modify the patience value:\npatience_earlystop = 30\npatience_ReduceLROnPlateau = 10\n\nearlystop = EarlyStopping(monitor='val_loss',\n                          mode='min',\n                          patience=patience_earlystop,\n                          verbose=1)\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=patience_ReduceLROnPlateau, \n                                            verbose=1, \n                                            factor=0.1, \n                                            min_lr=1e-5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.3 Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fitting the model using a batch generator:\nhistory = IC_model.fit_generator(\n    train_generator,\n    steps_per_epoch=total_train//batch_size,\n    epochs=epochs,\n    validation_data=validation_generator,\n    validation_steps=total_validate//batch_size,\n    callbacks=callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.4 Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Visualization:\nplot_smoothed_learning_curves(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.5 Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Evaluate the model:\ntest_loss, test_acc = IC_model.evaluate_generator(test_generator, steps=np.ceil(nb_samples/batch_size))\nprint('test acc:', test_acc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.6 Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Prediction:\npredict = IC_model.predict_generator(real_test_generator, steps=np.ceil(nb_samples/batch_size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"real_test_df['category'] = predict.round().astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"real_test_df['category'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.7 Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = real_test_df.copy()\nsubmission_df['id'] = submission_df['filename'].str.split('.').str[0]\nsubmission_df['label'] = submission_df['category']\nsubmission_df.drop(['filename', 'category'], axis=1, inplace=True)\nsubmission_df.to_csv('submission_IC.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}