{"cells":[{"metadata":{},"cell_type":"markdown","source":"# The dataset \n\nThe CIFAR-10 dataset is a \"standard\" collection of 60.000 32x32 colour images in 10 classes, each containing 6,000 images per class. The classes are:\n\n- airplane\n- automobile\n- bird\n- cat\n- deer\n- dog\n- frog \n- horse\n- ship\n- truck\n\nLuckily, the classes are mutually exclusive, meaning there will be no overlap of images belonging to two classes (e.g. pickup trucks belonging to automobile and truck). \n\nYou can find more information about this dataset at the [www.cs.toronto.edu homepage](https://www.cs.toronto.edu/~kriz/cifar.html)\n\nRemember, it would be really rare to have this kind of dataset in the real world. Some challenges include:\n\n- Having mutually exclusive images -> the image \"should\" only show the object to be classified\n- For MLP and CNNs to work we need them all to be the same size. This means we need to resize all our pictures, which can crop or transform them if we do not pay attention\n- Additionally to the previous point pictures are usually not in the format of 32 x 32, but rather 800x600 or similar, meaning they will take way more space before resizing them resulting in a huge storage demand\n- Lighting conditions and the colour scheme of the images can widely vary\n\n## The challenge\n\nNow our task is to divide the dataset into a train and test set, where we train both an MLP and CNN to predict the right labels for the test dataset.\n\nWith previous knowledge we know that CNNs will perform way better than MLPs, but why?\nBecause CNNs also take the spatial information into account, and are not fully connected. A classic MLP will have every node of the neuronal network attached to each node, resulting in a huge calculational cost. Furthermore, MLPs will transform the two-dimensional image into a single array to process it, which means that we will have one huge one-dimensional array where all our two-dimensional information will be lost. But still - for science, let us try it out!\n\n## My contact\n\nJustin Güse\n\nguese.justin@gmail.com\n\n[https://www.linkedin.com/in/justin-guese/](https://www.linkedin.com/in/justin-guese/)\n\n### Sources\n\nSome code taken from the udacity github library: https://github.com/udacity/aind2-cnn/tree/master/cifar10-classification\n"},{"metadata":{},"cell_type":"markdown","source":"# Part 1: Load the data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\ndata = np.load(\"/kaggle/input/cifar10-keras-files-cifar10load-data/cifar-10.npz\")\nfilenames = [\"x_train\",\"y_train\",\"x_test\",\"y_test\"]\nnps = []\nfor filename in filenames:\n    nps.append(data[filename])\nx_train,y_train,x_test,y_test = nps\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape,x_test.shape,y_train.shape,y_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 2: Let's explore some images!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# the images are matrices itself, meaning we will use numpy to read them in and matplotlib to \"plot\" them. \nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\ndef plot_images(x_train):\n    fig = plt.figure(figsize=(20,5))\n    for i in range(36):\n        ax = fig.add_subplot(3, 12, i + 1, xticks=[], yticks=[])\n        ax.imshow(np.squeeze(x_train[i]))\nplot_images(x_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape,x_test.shape,y_train.shape,y_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# rescale to bigger images by dividing through 256\n# rescale [0,255] --> [0,1]\n#x_train = x_train.astype('float32')/255\n#x_test = x_test.astype('float32')/255 \n#plot_images(x_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 3: Split into test and train\nnow for our train data we would like to use 5000 samples, with 1000 for test****"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import np_utils\nimport keras\n# one-hot encode the labels\nnum_classes = len(np.unique(y_train))\ny_train = keras.utils.to_categorical(y_train, num_classes)\ny_test = keras.utils.to_categorical(y_test, num_classes)\n\nx_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\nx_train /= 255\nx_test /= 255\n\n# break training set into training and validation sets\n(x_train, x_valid) = x_train[5000:], x_train[:5000]\n(y_train, y_valid) = y_train[5000:], y_train[:5000]\n\n# print shape of training set\nprint('x_train shape:', x_train.shape)\n\n# print number of training, validation, and test images\nprint(x_train.shape[0], 'train samples')\nprint(x_test.shape[0], 'test samples')\nprint(x_valid.shape[0], 'validation samples')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape,x_test.shape,y_train.shape,y_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 4: Model Training\n\n## 4.1 MLP "},{"metadata":{"trusted":true},"cell_type":"code","source":"# let us first construct a basic mlp model, feel free to play around!\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\n\n# define the model\nmodel = Sequential()\nmodel.add(Flatten(input_shape = x_train.shape[1:]))\nmodel.add(Dense(1000, activation='relu'))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(num_classes, activation='softmax'))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compile the model\nmodel.compile(loss='categorical_crossentropy', optimizer='rmsprop', \n                  metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now as you can see above, the MLP is a dense neuronal network (all nodes are connected with all input nodes), which means we will have  3,586,538 nodes in our model, which is crazy computationally expensive! This will take some time, so feel free to skip ahead to the CNN  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# to speed up you should consider using a GPU\nfrom tensorflow.python.client import device_lib\nprint(device_lib.list_local_devices()) # <- if it shows gpu here we're good to go!","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom keras.callbacks import ModelCheckpoint   \nimport time\n# train our model and save the best results in the file: MLP.best_weights\n# additionally count the time how long it took\n\nmlp_start = time.time() # for stopwatch\n\ncheckpointer = ModelCheckpoint(filepath='MLP.best_weights.hdf5', verbose=1, \n                               save_best_only=True)\nhist = model.fit(x_train, y_train, batch_size=32, epochs=20,\n          validation_data=(x_valid, y_valid), callbacks=[checkpointer], \n          verbose=2, shuffle=True)\n\nmlp_end = time.time()\nmlp_took = mlp_end -mlp_start\nprint(\"took %s seconds\"%(mlp_took))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation of the MLP model\n\nNow we will load the weights of the best MLP model and try to get an accuracy on the validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the weights that yielded the best validation accuracy\nmodel.load_weights('MLP.best_weights.hdf5')\n# evaluate and print test accuracy\nmlp_score = model.evaluate(x_test, y_test, verbose=0)\nprint('\\n', 'Test accuracy:', mlp_score[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 CNN\n\nConvolutional neuronal networks perform way better on image data, and are way quicker for the aforementioned reasons. Let us give it a shot - first with a basic model, and then something more complex for optimal results"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom keras.layers import Conv2D, MaxPooling2D\n\nmodel = Sequential()\nmodel.add(Conv2D(filters=16, kernel_size=2, padding='same', activation='relu', \n                        input_shape=(32, 32, 3)))\nmodel.add(MaxPooling2D(pool_size=2))\nmodel.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=2))\nmodel.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=2))\nmodel.add(Dropout(0.3))\nmodel.add(Flatten())\nmodel.add(Dense(500, activation='relu'))\nmodel.add(Dropout(0.4))\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compile the model\nmodel.compile(loss='categorical_crossentropy', optimizer='rmsprop', \n                  metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint   \ncnn_start = time.time() # for stopwatch\n# train the model\ncheckpointer = ModelCheckpoint(filepath='CNN.best_weights.hdf5', verbose=1, \n                               save_best_only=True)\nhist = model.fit(x_train, y_train, batch_size=32, epochs=100,\n          validation_data=(x_valid, y_valid), callbacks=[checkpointer], \n          verbose=2, shuffle=True)\ncnn_end = time.time()\ncnn_took = cnn_end -cnn_start\nprint(\"took %s seconds\"%(cnn_took))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CNN evaluation\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('CNN.best_weights.hdf5')\n# evaluate and print test accuracy\ncnn_score = model.evaluate(x_test, y_test, verbose=0)\nprint('\\n', 'Test accuracy:', cnn_score[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Comparing MLP and CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Time it took for the MLP to train: %f minutes. Accuracy %f \" % (int(mlp_took/60),mlp_score[1]))\nprint(\"Time it took for the CNN to train: %f minutes. Accuracy %f \" % (int(cnn_took/60),cnn_score[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization of some results\nNow let us have a look at some predictions that were done by the CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get predictions on the test set\ny_hat = model.predict(x_test)\n\ncifar10_labels = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n# plot a random sample of test images, their predicted labels, and ground truth\n\nprint(\"Green labels are right predictions, red are wrong predictions. The one in the paranthesis is the ground truth(real), \\\n      the first label is the prediction by the CNN\")\nfig = plt.figure(figsize=(20, 8))\nfor i, idx in enumerate(np.random.choice(x_test.shape[0], size=32, replace=False)):\n    ax = fig.add_subplot(4, 8, i + 1, xticks=[], yticks=[])\n    ax.imshow(np.squeeze(x_test[idx]))\n    pred_idx = np.argmax(y_hat[idx])\n    true_idx = np.argmax(y_test[idx])\n    ax.set_title(\"{} ({})\".format(cifar10_labels[pred_idx], cifar10_labels[true_idx]),\n                 color=(\"green\" if pred_idx == true_idx else \"red\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Advanced: Using state of the art models for object recognition\n\nOf course there are way more possibilities to play around with the different parameters in the CNN, and I guess you are able to reach 92% accuracy using only CNNs. But for now let us have a look at different models and how they compared in the CIFAR-10 challenge:\nhttps://benchmarks.ai/cifar-10\n\nThe GPipe model seems to perform best at the moment, but it uses many GPUs for this. This is why in this case I will not try to implement it, but instead have a look at the other candidate, the [\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (May 2019, arXiv 2019)\"](https://arxiv.org/pdf/1905.11946.pdf) by \tMingxing Tan, Quoc V. Le\nusing github code found in: https://github.com/qubvel/efficientnet\n\n## Implementing EfficientNet for CIFAR10\n\nFor this we will take the base model, add a custom layer and return our prediction with a softmax in the end like we did before"},{"metadata":{"trusted":true},"cell_type":"code","source":"# first install it using pip --> only if notebook has internet. file will be at\n# /kaggle/working/efficientnet-1.0.0-py3-none-any.whl\n#!wget https://files.pythonhosted.org/packages/97/82/f3ae07316f0461417dc54affab6e86ab188a5a22f33176d35271628b96e0/efficientnet-1.0.0-py3-none-any.whl\n!pip install -U efficientnet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Try out efficientnet B0"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Swish defination\nfrom keras.backend import sigmoid\nfrom keras.layers import Activation\nclass SwishActivation(Activation):\n    \n    def __init__(self, activation, **kwargs):\n        super(SwishActivation, self).__init__(activation, **kwargs)\n        self.__name__ = 'swish_act'\n\ndef swish_act(x, beta = 1):\n    return (x * sigmoid(beta * x))\n\nfrom keras.utils.generic_utils import get_custom_objects\nget_custom_objects().update({'swish_act': SwishActivation(swish_act)})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import efficientnet.keras as efn\n# loading B0 without the final layer, we will change it\nmodel = efn.EfficientNetB0(include_top=False, input_shape=(32,32,3), pooling='avg', weights='imagenet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import Dense, Dropout, Activation, BatchNormalization, Flatten\nfrom keras.models import Model\n# add some layers with dropout, dense, batchnormalization\n# idea from https://colab.research.google.com/drive/1ASAt0K3LcaTBx7FoB0ydPJNA63DeyPdG#forceEdit=true&sandboxMode=true&scrollTo=MWILUdR9cub7\nx = model.output\n\nx = BatchNormalization()(x)\nx = Dropout(0.7)(x)\n\nx = Dense(512)(x)\nx = BatchNormalization()(x)\nx = Activation(swish_act)(x)\nx = Dropout(0.5)(x)\n\nx = Dense(128)(x)\nx = BatchNormalization()(x)\nx = Activation(swish_act)(x)\n\n# output layer\npredictions = Dense(10, activation=\"softmax\")(x)\n\nmodel_final = Model(inputs = model.input, outputs = predictions)\n\nmodel_final.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\nplot_model(model_final, to_file='EN0_model_plot.png', show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.optimizers import Adam\nfrom keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\nimport time # should be imported from above\n# model compilation\nmodel_final.compile(loss='categorical_crossentropy',\n              optimizer=Adam(0.0001),\n              metrics=['accuracy'])\n\nmcp_save = ModelCheckpoint('EnetB0_CIFAR10_TL.h5', save_best_only=True, monitor='val_acc')\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.5, patience=2, verbose=1,)\n\nstart = time.time()\nprint(\"Training....\")\nmodel_final.fit(x_train, y_train,\n              batch_size=32,\n              epochs=20,\n              validation_split=0.1,\n              callbacks=[mcp_save, reduce_lr],\n              shuffle=True,\n              verbose=1)\nend = time.time()\nenb0_time = int((end - start)/60)\nprint(\"Efficientnet Training took %f minutes\"%(enb0_time))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# evaluate new model\nen_score = model_final.evaluate(x_test, y_test)\nprint('EfficientNetB0 Test accuracy: %f  , took %d minutes' %(en_score[1],enb0_time))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time # should be imported from above\n# from https://keras.io/preprocessing/image/\nfrom keras.preprocessing.image import ImageDataGenerator\ndatagen = ImageDataGenerator(\n    featurewise_center=True,\n    featurewise_std_normalization=True,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    horizontal_flip=True)\n\n# compute quantities required for featurewise normalization\n# (std, mean, and principal components if ZCA whitening is applied)\ndatagen.fit(x_train)\n\nen_start = time.time() # for stopwatch\n# fits the model on batches with real-time data augmentation:\n#model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),\n#                    steps_per_epoch=round(len(x_train) / 32), epochs=nb_epochs, callbacks = [checkpointer, early], verbose= 2)\n\nen_end = time.time()\nen_took = en_end -en_start\nprint(\"took %s minutes\"%(int(en_took/60)))\n# - 388s - loss: 0.6374 - accuracy: 0.7898\n#took 65 minutes","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}