{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this notebook I'll be trying out few different ways of building a CNN model, building the model from a scratch, adding regularization and data augmentation, and then transfer learning using a pre-trained model, and comparing both prediction accuracy and training time from these in the end. But there's a twist: Instead of doing the proper thing and testing the model performance on the competition data and treating the task at hand as a multiclass classification problem, the models will be tested using 50 pictures of our dog: 'Ace' the corgi! The model performance will be evaluated only on how well it can predict that 'Ace' is a corgi. \n\nI've been reading [Chollet's Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python) so this notebook will be quite heavily influenced by that. I'll try to share my reasoning, but as this notebook is created from my learning and \"for fun\" purposes don't expect best practices here!\n\n# Preparations\n\n## Loading Libraries and Reading Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport shutil\nimport random\nimport time\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom skimage.transform import resize\n\nfrom keras import models\nfrom keras import layers\nfrom keras import optimizers\nfrom keras.preprocessing import image\nfrom keras.callbacks import EarlyStopping\n\nrandom.seed(99)  # for reproducibility","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first thing needs to be done is sorting the pictures in the training and test data to corresponding folders to be read from there by `Keras`. The `input` folder in Kaggle is read-only. Because of this, one needs to use the `working` folder to arrange the images from the training data to subfolders by the dog breed:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_dir = '/kaggle/input/dog-breed-identification/train'\ntest_dataset_dir = '/kaggle/input/50-corgi-pictures/'\ntrain_labels = pd.read_csv('/kaggle/input/dog-breed-identification/labels.csv')\ntest_labels = pd.DataFrame({'id': os.listdir(test_dataset_dir), 'breed': 'pembroke'})\n\n# helper function to create directory for the script not throwing an error if the \ndef make_dir(x):\n    if not os.path.exists(x):\n        os.makedirs(x)\n\n# directory where we’ll store our dataset subset for keras generators to read from\nbase_dir = '/kaggle/working/dog-breed-identification/subsets/'\nmake_dir(base_dir)\n\nbreeds = list(train_labels.breed.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `train_labels` are in the form of a data frame, with one column for the image name, and the other for the corresponding label:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_labels.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The pictures not used for training are used for validation. This could also be specified as an argument in Keras in `model.fit()`, but here I wanted to be able to inspect the pictures by hand that is used as validation from the folders on my computer."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_frac = 0.8","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As mentioned above, the data needs to be arranged so that pictures for each breed are in their folder. First, we'll arrange the test and validation data provided by the competition to the correct folders:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_img_fnames = []\nvalidation_img_fnames = []\ntest_img_fnames = []\n\n# directories for the training, validation, and test images\ntrain_dir = os.path.join(base_dir, 'train')\nmake_dir(train_dir)\nvalidation_dir = os.path.join(base_dir, 'validation')\nmake_dir(validation_dir)\ntest_dir = os.path.join(base_dir, 'test')\nmake_dir(test_dir)\n\nbreed_counts = train_labels.breed.value_counts()\n\n# loop through breeds for training and validation\nfor breed in breeds:\n    # make a directory for each breed\n    train_breed_dir = os.path.join(train_dir, breed)\n    validation_breed_dir = os.path.join(validation_dir, breed)\n    \n    make_dir(train_breed_dir)\n    make_dir(validation_breed_dir)\n    \n    # get training count\n    n_train = int(breed_counts[breed] * train_frac)\n    i = 0\n    \n    # get ids for training and validation by breeds\n    breed_ids = train_labels[train_labels.breed == breed].id\n    breed_train = breed_ids.sample(n=n_train, random_state=57)\n    breed_validation = breed_ids[~breed_ids.isin(breed_train)]\n\n    # transfer doggo images to these folders accordingly\n    for dog in breed_train:\n        i+=1\n        src = os.path.join(dataset_dir, dog + '.jpg')\n        dst = os.path.join(train_breed_dir, dog + '.jpg')\n        shutil.copyfile(src, dst)\n        train_img_fnames.append(dog)\n\n    for dog in breed_validation:\n        i+=1\n        src = os.path.join(dataset_dir, dog + '.jpg')\n        dst = os.path.join(validation_breed_dir, dog + '.jpg')\n        shutil.copyfile(src, dst)\n        validation_img_fnames.append(dog)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For clarity, let's check the paths and filenames from the last iteration of the for loop:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Source: {src}\\nDestination: {dst}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's verify that the last dog is a rottweiler:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.loc[train_labels.id == os.path.splitext(src)[0].split('/')[-1]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After dealing with the training and validation data comes the test data, containing only pictures of 'Ace': "},{"metadata":{"trusted":true},"cell_type":"code","source":"# loop through 'breeds' for testing\ntest_breed_dir = os.path.join(test_dir, 'pembroke')\nmake_dir(test_breed_dir)\nfor corgi_img in test_labels.id:\n    src = os.path.join(test_dataset_dir, corgi_img)\n    dst = os.path.join(test_breed_dir, corgi_img)\n    shutil.copyfile(src, dst)\n    test_img_fnames.append(os.path.splitext(corgi_img)[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And for the test data the equivalent paths look like:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Source: {src}\\nDestination: {dst}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the Data\n\n\"Data exploration\" here is pretty much just looking at the corgi pictures. Also, an interesting question at this point is that how represented 'pembroke' is in the training data?"},{"metadata":{"trusted":true},"cell_type":"code","source":"breed_color = [['red' if (x == 'pembroke') else 'lightgrey' for x in breed_counts.index]] # attention to double []\npd.DataFrame(breed_counts).plot.bar(color=breed_color, width=0.8, figsize=(21, 5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.loc[train_labels.breed == 'pembroke', 'id']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's look at some random corgis from the training data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# pick 9 random pembroke images from training data\npembrokes = train_labels.loc[train_labels.breed == 'pembroke', 'id']\n\ntrain_img_sample = random.sample(pembrokes.tolist(), 9)\nread_train_imgs = [mpimg.imread(os.path.join(dataset_dir, x + '.jpg')) for x in train_img_sample]\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(15,15))\nfor i, v in enumerate(fig.axes):\n    v.imshow(read_train_imgs[i])\n    v.text(x=read_train_imgs[i].shape[1]/2, y=read_train_imgs[i].shape[1]/40, s=train_img_sample[i], bbox=dict(facecolor='white', alpha=0.9), ha='center', va='top', size=11)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And compare them to the pictures of 'Ace':"},{"metadata":{"trusted":true},"cell_type":"code","source":"# take a look at the pictures of Ace\ndef resize_pic(x):\n    return resize(x, (x.shape[0] // 8, x.shape[1] // 8), anti_aliasing=True)\n\ntest_img_sample = random.sample(test_img_fnames, 9)\nread_test_imgs = [resize_pic(mpimg.imread(os.path.join(test_dataset_dir, x + '.jpg'))) for x in test_img_sample]\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(15,15))\nfor i, v in enumerate(fig.axes):\n    v.imshow(read_test_imgs[i])\n    v.text(x=read_test_imgs[i].shape[1]/2, y=read_test_imgs[i].shape[1]/40, s=test_img_sample[i], bbox=dict(facecolor='white', alpha=0.9), ha='center', va='top', size=14)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training data includes pictures of both puppies and adult dogs, which is nice because test data does this also.\n\n## Keras Callbacks, Helper Functions, and Other Parameters\n\nHere, we want to define some common parameters for the models, e.g. `batch_size`, early stopping (when the validation loss doesn't improve anymore), and helper functions for visualizing the model performance over the epochs:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# variables used by all models\nbatch_size = 20\n# early stopping to stop training after validation loss stops improving\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)\n\n# helper function with the visualization\n\ndef plot_history(history):\n    acc = history.history['acc']\n    val_acc = history.history['val_acc']\n    loss = history.history['loss']\n    val_loss = history.history['val_loss']\n    epochs = range(1, len(acc) + 1)\n    plt.plot(epochs, acc,  color='#008080', mec='k', label='Training accuracy')\n    plt.plot(epochs, val_acc, color='#FFA500', label='Validation accuracy')\n    plt.title('Training and validation accuracy')\n    plt.legend()\n    plt.figure()\n    plt.plot(epochs, loss, color='#008080', label='Training loss')\n    plt.plot(epochs, val_loss, color='#FFA500', label='Validation loss')\n    plt.title('Training and validation loss')\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `test_model` function is used to make predictions and store them in a table for easy comparison between model performance at the end of the notebook. Some models use generators and others (transfer learning) use features extracted by the pre-trained model, and the `test_model`-function needs to be able to work with both."},{"metadata":{"trusted":true},"cell_type":"code","source":"results = []\n\ndef test_model(model_name, model, train_time, history, features=None, generator=None):\n    if generator is None:\n        predictions = model.predict(features)\n    else:\n        predictions = model.predict_generator(generator, steps = generator.n)\n    pred = []\n    truth = [86 for i in range(50)]\n    for i in range(0, len(predictions)):\n        pred.append(np.argmax(predictions[i]))\n    acc_bool = [x == y for x, y in zip(pred, truth)]\n    acc = round(sum(acc_bool) / len(acc_bool) * 100, 1)\n    valid_loss = min(history.history['val_loss'])\n    print(f'TEST ACCURACY\\n\\t{acc}% ({sum(acc_bool)}/{len(acc_bool)} correct)')\n    train_time = round(train_time / 60, 1)\n    return {'Model Name': model_name, 'Test Accuracy %': acc, 'Validation Loss': valid_loss, 'Training Time (minutes)': train_time}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Own Models\n\n## ConvNet from a Scratch\n\nThe training data is quite limited and there are not too many pictures on average per breed, so the data is probably not enough to learn a CNN from a scratch. Anyway, we'll try to do so, and at least will get to use the pre-trained model(s) as a baseline to compare the pre-trained networks with.\n\n__CNN architecture__<br>\nI used principles mentioned [here](https://towardsdatascience.com/a-guide-to-an-efficient-way-to-build-neural-network-architectures-part-ii-hyper-parameter-42efca01e5d7), \"dogs vs cats\" from Chollet's book as well as VGG-16 as an inspiration. I also tested some different values for the number of hidden layers and the dropout, and selected values that seemed to work reasonably well while keeping the network size moderate.\n\n__Optimizer__<br>\nAs an optimizer, we'll use `Adam`, which is short for Adaptive Moment Estimation. Adam is a popular algorithm in the field of deep learning because it achieves good results fast. Andrej Karpathy (Tesla AI guy) suggested it as the default optimization method for deep learning applications, so it should be good for a simple dog picture problem. Metric is accuracy, as we're only interested in how many corgi pictures the model classifies correctly.\n\nBecause the targets are not one-hot encoded but instead stored integers, I'll use `sparse_categorical_crossentropy` instead of `categorical_crossentropy` loss while compiling the model. We'll use a generator built with `flow_from_directory()`-function, which reads through the training and validation folders and feeds the pictures in batches for calculating the weights."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = models.Sequential()\nmodel.add(layers.Conv2D(32, (3, 3), activation='relu',\ninput_shape=(224, 224, 3)))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(64, (3, 3), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(128, (3, 3), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(128, (3, 3), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dense(120, activation='softmax')) # 120 different breeds\n\nmodel.summary()\n\noptimizer = optimizers.Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, amsgrad=False)\nmodel.compile(optimizer=optimizer,\n    loss='sparse_categorical_crossentropy',\n    metrics=['acc'])\n\ndatagen = image.ImageDataGenerator(rescale=1./255) # rescale pixel values to [0, 1] interval\n\ntrain_generator = datagen.flow_from_directory(\n    train_dir,\n    target_size=(224, 224),\n    batch_size=batch_size,\n    class_mode='sparse')\n        \nvalidation_generator = datagen.flow_from_directory(\n    validation_dir,\n    target_size=(224, 224),\n    batch_size=batch_size,\n    class_mode='sparse',\n    classes=train_generator.class_indices)\n\ntest_generator = datagen.flow_from_directory(\n    test_dir,\n    target_size=(224, 224),\n    batch_size=1, # predict one picture at a time\n    class_mode='sparse',\n    classes=train_generator.class_indices)\n\n# dummy check if all class indices are the same in all generators\nprint(f'\\nAll classes same: {train_generator.class_indices == test_generator.class_indices == validation_generator.class_indices}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If the model initializes correctly, the loss at the beginning of the training process should be -ln(1/120), which is approximately 4.787. As a baseline accuracy, we can use a random guess, which would result in an accuracy of 0.8%."},{"metadata":{"trusted":true},"cell_type":"code","source":"start_clock = time.clock()\nhistory = model.fit_generator(\n        train_generator,\n        steps_per_epoch=int(train_generator.n / train_generator.batch_size), # matching the number of samples, 8127/20  \n        epochs=100,\n        validation_data=validation_generator,\n        validation_steps=int(validation_generator.n / validation_generator.batch_size),\n        callbacks=[es],\n        verbose=0)\nend_clock = time.clock()\ntrain_time = end_clock - start_clock","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can inspect the learning process, and store the results from the trained model for later use using the helper functions defined earlier:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history)\nresults.append(test_model('CNN', model, train_time, history, generator=test_generator))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model learns the features of the training data but heavily overfits, as the training error $J_{train}(\\Theta)$ keeps decreasing towards 0 and is very low compared to the validation error $J_{CV}(\\Theta)$ when both are plotted over the epochs. In the contrast, if both errors were high, we would observe high bias (underfitting). In this case, the next steps would be to take a step back and to modify the architecture of the model by adding regularization and \"more data\" through the data augmentation.\n\n## Adding Data Augmentation and Regularization to ConvNet\n\nAs the first results were not too encouraging (though beating a random guess), we can try to improve the model performance using data augmentation and regularization. For data augmentation, we'll define a new data generator performing transformations to the input images. Test and validation images should not be augmented.\n\nAlso, it's to be noted, that the `ImageDataGenerator` takes in the original data, applies the transformations, and returns the new augmented data and disregards the original data! To use data augmentation, we need a new generator:"},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_datagen = image.ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=30,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    vertical_flip=False,\n    brightness_range=[0.2, 1.0],\n    fill_mode='nearest')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at a randomly chosen training image how it looks augmented:"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_img = os.path.join(dataset_dir, random.sample(train_labels[train_labels['breed'] == 'pembroke'].id.tolist(), 1)[0] + '.jpg')\nmy_img = image.load_img(my_img, target_size=(224, 224))\nmy_img = image.img_to_array(my_img)\n\nmy_img = my_img.reshape((1,) + my_img.shape)\nfig, axes = plt.subplots(nrows=1, ncols=4, figsize=(15,5))\nfor batch, (i, v) in zip(aug_datagen.flow(my_img, batch_size=1), enumerate(fig.axes)):\n    v.imshow(image.array_to_img(batch[0]))\n    if i == 3:\n        break\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now training a model using data augmentation and regularization:"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = models.Sequential()\nmodel.add(layers.Conv2D(32, (3, 3), activation='relu',\ninput_shape=(224, 224, 3)))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(64, (3, 3), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(128, (3, 3), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(128, (3, 3), activation='relu'))\nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(512, activation='relu', input_dim=7 * 7 * 512))\nmodel.add(layers.Dropout(0.2)) # it's now drop-rate instead of keep-rate\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(120, activation='softmax'))\n\nmodel.summary()\n\noptimizer = optimizers.Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, amsgrad=False)\nmodel.compile(optimizer=optimizer,\n    loss='sparse_categorical_crossentropy',\n    metrics=['acc'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator_aug = aug_datagen.flow_from_directory(\n    train_dir,\n    target_size=(224, 224),\n    batch_size=batch_size,\n    class_mode='sparse')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start_clock = time.clock()\nhistory = model.fit_generator(\n        train_generator_aug,\n        steps_per_epoch=int(train_generator_aug.n / train_generator_aug.batch_size) * 3, # every image augmented three times\n        epochs=100,\n        validation_data=validation_generator,\n        validation_steps=int(validation_generator.n / validation_generator.batch_size),\n        callbacks=[es],\n        verbose=0)\nend_clock = time.clock()\ntrain_time = end_clock - start_clock","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history)\nresults.append(test_model('CNN+reg+aug', model, train_time, history, generator=test_generator))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The results improved, but the overall performance of the model is still not very good. As a next step in the attempt to make better predictions we'll try a pre-trained model.\n\n# Pretrained Models\n\nAs expected, training a model from a scratch didn't work that well for the problem at hand as the training data was quite limited for a neural network classifier that can learn complex functions. Therefore, it's expected to be beneficial for the model to be exposed to a higher amount of data. We can achieve this in a computationally effective way by using a pre-trained network to extract the features, which is then used as input to our dense classifier, which makes the final predictions. First, we define helper functions to extract the features and count the files:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_features(directory, sample_count, x, y, z, target_size):\n    global class_dictionary\n    global filenames\n    features = np.zeros(shape=(sample_count, x, y, z))\n    labels = np.zeros(shape=(sample_count))\n    generator = datagen.flow_from_directory(\n        directory,\n        target_size=target_size,\n        batch_size=batch_size,\n        class_mode='sparse',\n        shuffle=False,\n        classes=train_generator.class_indices)\n    i = 0\n    for inputs_batch, labels_batch in generator:\n        features_batch = conv_base.predict(inputs_batch)\n        features[i * batch_size : (i + 1) * batch_size] = features_batch\n        labels[i * batch_size : (i + 1) * batch_size] = labels_batch\n        i += 1\n        #print(i)\n        if i * batch_size >= sample_count:\n            break\n    class_dictionary = generator.class_indices\n    filenames = generator.filenames\n    return features, labels\n\ndef count_files(input_dir):  # counts files from subdirs\n    path = input_dir\n    n = 0\n    folders = ([name for name in os.listdir(path) \n                if os.path.isdir(os.path.join(path, name))])  # get all directories \n    for folder in folders:\n        contents = os.listdir(os.path.join(path,folder))  # get list of contents\n        \n        n += len(contents)\n    return(n)\n\ntrain_n = count_files(train_dir)\nvalidation_n = count_files(validation_dir)\ntest_n = count_files(test_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## VGG19\n\nAs a first model lets try out VGG19 which takes a default input image size of 224x224 pixels:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications import VGG19\nconv_base = VGG19(weights='imagenet', include_top=False, input_shape=(224, 224, 3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, checking the shape of the output of the last layer of the convolutional base to get the shape of the features that will be fed to the dense classifier:"},{"metadata":{"trusted":true},"cell_type":"code","source":"conv_base.layers[-1].output_shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hw = conv_base.layers[-1].output_shape[1]  # height/width\nde = conv_base.layers[-1].output_shape[-1]  # depth\nts = 224  # size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'NUMBER OF IMAGE FILES\\nTrain: {train_n}\\nValidation: {validation_n}\\nTest: {test_n}')\n\ntrain_features, train_labels_arr = extract_features(train_dir, train_n, hw, hw, de, target_size=(ts, ts))\nvalidation_features, validation_labels_arr = extract_features(validation_dir, validation_n, hw, hw, de, target_size=(ts, ts))\ntest_features, test_labels_arr = extract_features(test_dir, test_n, hw, hw, de, target_size=(ts, ts))\n\ntrain_features = np.reshape(train_features, (train_n, hw * hw * de))\nvalidation_features = np.reshape(validation_features, (validation_n, hw * hw * de))\ntest_features = np.reshape(test_features, (test_n, hw * hw * de))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = models.Sequential()\nmodel.add(layers.Dense(512, activation='relu', input_dim=hw * hw * de))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(120, activation='softmax'))\noptimizer = optimizers.Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, amsgrad=False)\nmodel.compile(optimizer=optimizer,\n    loss='sparse_categorical_crossentropy',\n    metrics=['acc'])\n\nstart_clock = time.clock()\nhistory = model.fit(train_features, train_labels_arr,\n    epochs=100,\n    batch_size=batch_size,\n    validation_data=(validation_features, validation_labels_arr),\n    callbacks=[es],\n    verbose=0)\nend_clock = time.clock()\ntrain_time = end_clock - start_clock","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history)\nresults.append(test_model('VGG19', model, train_time, history, features=test_features))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The accuracy is well improved over the self-made CNN, so using a pre-trained model is taking things in the right direction.\n\n## InceptionResNetV2\n\nAnother pre-trained model that should work well with the problem at hand is InceptionResNetV2. The default input size for Inception-ResNet-V2 is 299x299. Again, we load a pre-trained model without its top:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications import InceptionResNetV2\nconv_base = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(299, 299, 3))\nconv_base.layers[-1].output_shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hw = conv_base.layers[-1].output_shape[1]  # height/width\nde = conv_base.layers[-1].output_shape[-1]  # depth\nts = 299  # size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'NUMBER OF IMAGE FILES\\nTrain: {train_n}\\nValidation: {validation_n}\\nTest: {test_n}')\n\ntrain_features, train_labels_arr = extract_features(train_dir, train_n, hw, hw, de, target_size=(ts, ts))\nvalidation_features, validation_labels_arr = extract_features(validation_dir, validation_n, hw, hw, de, target_size=(ts, ts))\ntest_features, test_labels_arr = extract_features(test_dir, test_n, hw, hw, de, target_size=(ts, ts))\n\ntrain_features = np.reshape(train_features, (train_n, hw * hw * de))\nvalidation_features = np.reshape(validation_features, (validation_n, hw * hw * de))\ntest_features = np.reshape(test_features, (test_n, hw * hw * de))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Building the dense classifier which takes the right input size:"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = models.Sequential()\nmodel.add(layers.Dense(512, activation='relu', input_dim=hw * hw * de))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(512, activation='relu'))\nmodel.add(layers.Dropout(0.2))\nmodel.add(layers.Dense(120, activation='softmax'))\noptimizer = optimizers.Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, amsgrad=False)\nmodel.compile(optimizer=optimizer,\n              loss='sparse_categorical_crossentropy',\n              metrics=['acc'])\n\nstart_clock = time.clock()\nhistory = model.fit(train_features, train_labels_arr,\n    epochs=100,\n    batch_size=batch_size,\n    validation_data=(validation_features, validation_labels_arr),\n    callbacks=[es],\n    verbose=0)\nend_clock = time.clock()\ntrain_time = end_clock - start_clock","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history)\nresults.append(test_model('Inception-ResNet-V2', model, train_time, history, features=test_features))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Huge improvement over the own model in both validation and test accuracy! Let's now make predictions using this model using the whole test data and see how the classifier performs.\n\n# Results\n\n## Visualizing Predictions\n\nNow when we have a model with good enough accuracy, let's make some predictions on the test data, and see what kind of dog it thinks our test subject is."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_predictions = model.predict(test_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we'll build the data for the visualization, for all the pictures we get to extract the top 5 predictions for a visualization next to the image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"all_labels = dict((v,k) for k,v in test_generator.class_indices.items())\n\ntest_ind_labels = np.argsort(-test_predictions, axis=1)[:, :5]\npred_labels = []\nfor i, v in enumerate(test_ind_labels):\n    pred_labels.append([all_labels[x] for x in v])\npred_probs = [test_predictions[i][test_ind_labels[i]] for i in range(0, test_n)]\n\nread_test_imgs = []\nfor i, v in enumerate(test_img_fnames[0:20]):\n    read_test_imgs.append(resize_pic(mpimg.imread(os.path.join(test_dataset_dir, test_img_fnames[i] + '.jpg'))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now plotting the images with the predictions as bar charts next to them, coloring the truth bar with distinguishable color:"},{"metadata":{"trusted":true},"cell_type":"code","source":"pics_per_row = 2\npred_n = len(read_test_imgs)\nfig, axes = plt.subplots(nrows=pred_n // pics_per_row, ncols=pics_per_row * 2, figsize=(pics_per_row * 9, pred_n * 1.5))\nr = -1\nc = 0\nfor i, v in enumerate(read_test_imgs):\n    if i % pics_per_row*2 == 0:\n        r += 1\n        c = 0\n    axes[r,c].barh([str(x) for x in pred_labels[i]], pred_probs[i],\n    color=['royalblue' if x == 'pembroke' else 'darkgrey' for x in pred_labels[i]], edgecolor='black')\n    axes[r,c].set(xlim=(0,1))\n    axes[r,c].set_aspect(0.2, anchor='W')\n    axes[r,c].invert_yaxis()\n    for a in range(len(pred_probs[i])):\n        axes[r,c].text(x=pred_probs[i][a], y=a,\n            s=str(round(pred_probs[i][a] * 100, 1)) + '%',\n            verticalalignment='center',\n            horizontalalignment='right' if pred_probs[i][a] > 0.25 else 'left', size = 16)\n    axes[r,c].set_title('Predicted')\n    axes[r,c].set_anchor('E')\n    axes[r,c].tick_params(axis='y', labelsize=16)\n    c += 1\n    axes[r,c].imshow(v)\n    axes[r,c].set_title(test_img_fnames[i])\n    axes[r,c].set_anchor('W')\n    c += 1\nplt.tight_layout()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Comparison and Conclusions"},{"metadata":{"trusted":true},"cell_type":"code","source":"result_table = pd.DataFrame(results)\nprint(result_table)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# delete all subdirs (needed for not throwing an error for too many files)\nshutil.rmtree(base_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If training up from a scratch, adding augmentation improved the model performance but came with a cost regarding the training time. Inception-ResNet-v2 worked well with the problem, and with further tuning of the dense classifier a top of it would have most likely to perform even better. Also it's mentioned in [Google's AI blog](https://ai.googleblog.com/2016/08/improving-inception-and-image.html) that the architecture excels at identifying individual dog breeds, which is quite neat. With the limited data building up the classifier from the scratch did not work very well. Nonetheless, it's good to keep in mind that the dog breed classification problem is quite tough: because of the 120 classes present in the data set, taking random guesses would result in accuracy of only 0.8%."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}