{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"In this kernel we will explore the concept of Test Time Augmentation (TTA) and will run an experiment on **Dogs vs. Cats** competition. Compare the results with and without TTA."},{"metadata":{"_uuid":"bb48e605fd4619e935568e2abfd7bdc0b2e130cc"},"cell_type":"markdown","source":"## What is Test Time Augmentation ?"},{"metadata":{"_uuid":"cc9a2018d535b41e491e166914ad84de2f2c1a6a"},"cell_type":"markdown","source":"TTA is simply to apply different transformations to **test** image like: rotations, flipping and translations. Then feed these different transformed images to the trained model and average the results to get more confident answer. For example, the image shown below applies two transformations (Left-Right flipping and Contrast change) together with the original image. All of these images are passed to the same model and the results are averaged."},{"metadata":{"_uuid":"0060819f8b9de911d405c3b51b214e570aa8f6ba"},"cell_type":"markdown","source":"![pipeline](https://preview.ibb.co/kH61v0/pipeline.png)"},{"metadata":{"_uuid":"c9bff24026e80a594394b3de8d82ba0504f67fcc"},"cell_type":"markdown","source":"[Edafa](https://github.com/andrewekhalel/edafa) is a ready-to-use package for TTA which can be used directly as we will show in the example"},{"metadata":{"_uuid":"10a09ed39610856f9df7925729eba9e0f9602baf"},"cell_type":"markdown","source":"## A proof of concept on Dogs vs. Cats competition"},{"metadata":{"trusted":true,"_uuid":"faca7e867d8f803ef27ccb0c0942c6265a44954a"},"cell_type":"code","source":"# important dependencies\nfrom sklearn.model_selection import train_test_split\nimport keras\nfrom keras.applications.vgg16 import VGG16,preprocess_input,decode_predictions\nfrom keras.preprocessing import image\nfrom keras.layers import Flatten,Dropout,Dense\nfrom keras.optimizers import Adam,SGD\nfrom keras import Model\nimport numpy as np\nimport random\nimport os\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b8de49655bf9902426ddca630db83ba4d24fdbf"},"cell_type":"markdown","source":"### Let's explore the dataset!"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"25fdb75a22c926fa739e48e26914771fd5e7ce41"},"cell_type":"code","source":"TRAIN_DIR = '../input/train/train/'\nprint ('This is a sample of the dataset file names')\nprint( os.listdir(TRAIN_DIR)[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5edae07d23e9e321f77675956a4c0a417a12821e"},"cell_type":"code","source":"sample = plt.imread(os.path.join(TRAIN_DIR,random.choice(os.listdir(TRAIN_DIR))))\nprint ('Visualize a sample of the image')\nprint ('Image shape:',sample.shape)\nplt.imshow(sample)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4390e7a2a77aa638c0ab6eddd02e164d44cc8c22"},"cell_type":"markdown","source":"Split data into train and validation sets (70% and 30% respectively)"},{"metadata":{"trusted":true,"_uuid":"46e37e9120c3b3be8c9bd843e4f414a24757da5c"},"cell_type":"code","source":"f_train, f_valid = train_test_split(os.listdir(TRAIN_DIR), test_size=0.7, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c1d93f734ade8b4aa030f461c1503c2d2d9f3b5"},"cell_type":"code","source":"# Network input size\nPATCH_DIM = 32","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"00425649fd625cfb0f31dd6675dc54c54f4055af"},"cell_type":"markdown","source":"### Build data generator that reads batch by batch from disk when needed"},{"metadata":{"trusted":true,"_uuid":"a9fbbec1a9be064ca71f815f8bd9e6e267f4fd47"},"cell_type":"code","source":"# src: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly\nclass DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, files, batch_size=32, dim=(224,224), n_channels=3,n_classes=2, shuffle=True):\n        'Initialization'\n        self.dim = dim\n        self.batch_size = batch_size\n        self.files = files\n        self.n_channels = n_channels\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.files) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = np.random.choice(len(self.files), self.batch_size)\n        \n        # Find files of IDs\n        batch_files = self.files[indexes]\n\n        # Generate data\n        X, y = self.__data_generation(batch_files)\n\n        return X, y\n\n    def __data_generation(self, batch_files):\n        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)\n        # Initialization\n        X = np.empty((self.batch_size, *self.dim, self.n_channels))\n        y = np.empty((self.batch_size), dtype=int)\n\n        # Generate data\n        for i, f in enumerate(batch_files):\n            # Store sample\n            img_path = os.path.join(TRAIN_DIR,f)\n            img = image.load_img(img_path, target_size=self.dim)\n            x = image.img_to_array(img)\n            x = np.expand_dims(x, axis=0)\n            x = preprocess_input(x)\n            x = np.squeeze(x)\n            X[i,:,:,:] = x\n\n            # Store class\n            if 'dog' in f:\n                y[i]=1\n            else:\n                y[i]=0\n                \n        return X, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e1f19e802cc1a49e1e9f9a6443d9a455a83d5f1a"},"cell_type":"code","source":"training_generator = DataGenerator(np.array(f_train),dim=(PATCH_DIM,PATCH_DIM))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7b0b160e1b6c5f7a040a922434add5962b6dcea"},"cell_type":"markdown","source":"### Build and train the model\nOur model is reusing VGG16 architecture without the fully connected layers. So we used the weights from imagenet and add our head as shown"},{"metadata":{"trusted":true,"_uuid":"2ff195a3cc4604fbe2f37aff1bffd7e476e0f02c"},"cell_type":"code","source":"initial_model = VGG16(weights=\"imagenet\", include_top=False ,input_shape = (PATCH_DIM,PATCH_DIM,3))\nlast = initial_model.output\n\nx = Flatten()(last)\nx = Dense(4096, activation='relu')(x)\nx = Dropout(0.4)(x)\nx = Dense(4096, activation='relu')(x)\nx = Dropout(0.4)(x)\npreds = Dense(1, activation='sigmoid')(x)\n\nmodel = Model(initial_model.input, preds)\nmodel.compile(Adam(1e-5), loss='binary_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ecbe29263dfc6ffc07f514729600b1bfa5a3180e"},"cell_type":"markdown","source":"Now we train the model"},{"metadata":{"trusted":true,"_uuid":"f80a52d5bbd04cb71576f3c013dcd98394d3e9ad","scrolled":true},"cell_type":"code","source":"model.fit_generator(generator=training_generator,\n                    use_multiprocessing=True,\n                    epochs=4,\n                    workers=8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"525d6df7b41aef9040dbe9e5943306b05d9bef82"},"cell_type":"markdown","source":"### Read validation images and evaluate the model"},{"metadata":{"trusted":true,"_uuid":"49c0256e585ee672fa9766a25a983f48fb35d9a2"},"cell_type":"code","source":"X_val = np.empty((len(f_valid), PATCH_DIM, PATCH_DIM ,3))\ny_val = np.empty((len(f_valid)), dtype=int)\n\nfor i, f in enumerate(f_valid):\n    # Store sample\n    img_path = os.path.join(TRAIN_DIR,f)\n    img = image.load_img(img_path, target_size=(PATCH_DIM,PATCH_DIM))\n    x = image.img_to_array(img)\n    x = np.expand_dims(x, axis=0)\n    x = preprocess_input(x)\n    x = np.squeeze(x)\n    X_val[i,:,:,:] = x\n\n    # Store class\n    if 'dog' in f:\n        y_val[i]=1\n    else:\n        y_val[i]=0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"94ce7c30927ca0b9b06fa5f1a8940346dfea053c"},"cell_type":"code","source":"y_pred = model.predict(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3974de22a668b25530a9b37bc92ded1be1575185"},"cell_type":"code","source":"y_pred = [(y[0]>=0.5).astype(np.uint8) for y in y_pred]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f4355ee9d6476a4084e2429343a18b661b10c14"},"cell_type":"code","source":"print('Accuracy without TTA:',np.mean((y_val==y_pred)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"71cbc407b6bb616b3d034f97409649c9d480c5b8"},"cell_type":"markdown","source":"### Now we use edafa (TTA package)"},{"metadata":{"_uuid":"40d40b8abed4d502d19f0a3f303e2c5a8f4edbc5"},"cell_type":"markdown","source":"Step 1: Import the predictor suitable for your problem (`ClassPredictor` for Classification and `SegPredictor` for Segmentation)"},{"metadata":{"trusted":true,"_uuid":"769cd16628df2b3df849c966c0279660eacabbbb"},"cell_type":"code","source":"from edafa import ClassPredictor","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"678fd37ac51fbf9fc59661df39082c161213589f"},"cell_type":"markdown","source":"Step 2: Inherit predictor class and implement the main function `predict_patches(self,patches)`"},{"metadata":{"trusted":true,"_uuid":"f08aba7be172ecc07edfc267f50de9851f99dcd5"},"cell_type":"code","source":"class myPredictor(ClassPredictor):\n    def __init__(self,model,*args,**kwargs):\n        super().__init__(*args,**kwargs)\n        self.model = model\n\n    def predict_patches(self,patches):\n        return self.model.predict(patches)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4229049d028bac333884f57a77389c0cf8dccf2"},"cell_type":"markdown","source":"Step 3: Instantiate your class with configuration and whatever parameters needed"},{"metadata":{"trusted":true,"_uuid":"133b4ca442e615e959f562333f811f35990a7d18"},"cell_type":"code","source":"# use orignal image and flipped Left-Right images\n# use arithmetic mean for averaging\nconf = '{\"augs\":[\"NO\",\\\n                \"FLIP_LR\"],\\\n        \"mean\":\"ARITH\"}'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dadb6d2924e8497c2c26a3ba9283a910b4671520"},"cell_type":"code","source":"p = myPredictor(model,conf)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0441e5792184a26ac6654a1ab0d032ec80bf284e"},"cell_type":"markdown","source":"Step 4: Predict images"},{"metadata":{"trusted":true,"_uuid":"ef0b22aa493cc6adacf7a14df0f71d7645bdfd49"},"cell_type":"code","source":"y_pred_aug = p.predict_images(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04111c3c705ff0d210f0b8178249e941b1febae5"},"cell_type":"code","source":"y_pred_aug = [(y[0]>=0.5).astype(np.uint8) for y in y_pred_aug ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dcf2b52923859f6e43164e69b3204e9528850246"},"cell_type":"code","source":"print('Accuracy with TTA:',np.mean((y_val==y_pred_aug)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61637da70de4efe3457290e6f925d4d6b2fbe9d6"},"cell_type":"markdown","source":"## Conclusion\nWe can see that the accuracy improved using the exact same model. Thanks to TTA!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}