{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <a id='0'>[3rd ML] Keras EfficientNet + TTA solution</a>"},{"metadata":{},"cell_type":"markdown","source":"## Record\n\nBaseline 코드는 [@태진님의 커널](https://www.kaggle.com/fulrose/3rd-ml-month-car-model-classification-baseline)과 함께 APTOS competition의 커널인 [resnet50 baseline](https://www.kaggle.com/mathormad/aptos-resnet50-baseline) 코드를 사용했습니다.\n해당 커널를 필사하면서 Generator와 Callback을 custom하는 부분이 많이 도움되었습니다. 이번 커널에서는 결과를 정리하고, 모델의 weight를 불러와 TTA로 결과값을 계산하는 과정과 여기서 필요한 Test Generator를 직접 설계했던 과정을 올렸습니다. 좋은 대회 열어주신 캐글 코리아 운영진께 감사합니다.\n\n\n### 1. Model\n* Xception과 Resnet, Moblienet, EfficientNet(B0, B3, B4)를 실험하였으며, 최종적으로 EfficientNetB4로 총 8개 모델 weight를 생성했습니다. \n\n```\ntrain(0.85), val(0.15) 데이터로 학습(50 epoch) : 0.93400\ntrain(0.85), val(0.15) 데이터로 학습 + 추가 20 epoch 학습: 0.92826\ntrain(0.85), val(0.15) 데이터로 학습 + 전체 데이터 추가학습 : 0.93591\n\n@saewonYang 님의 How to split folds and train의 fold.csv를 이용해서 학습\n1 fold : 0.93208\n2 fold : 0.93161\n3 fold : 0.93591\n4 fold : 0.93298\n5 fold : 0.94021\n```\n### 2. Image & Augmentation\n* Data는 허태명님이 제공해주신 [Crop image](https://www.kaggle.com/tmheo74/3rd-ml-month-car-image-cropping-updated-7-10)를 사용했습니다.\n* Size 는 300에서 380으로 올렸을 때, EfficientNet 기준 3%가 올라서 380으로 결정하였습니다.\n* Augmentation은 초기에는 ImageGenerator에서 설정하는 값만 적용하다가, [Autoaugmentation](https://github.com/DeepVoltaire/AutoAugment)을 적용했습니다(약 0.8%의 상향).\n\n\n### 3. Optimizer\n* Adam과 AdamAccumulate_v1, AdamBound 세가지를 실험했습니다. Warm up 학습에는 Adam를 쓰고 이후  AdamBound와 AdamAccumulate_v1 중, AdamAccumulate_v1가 0.2% 높게 나왔습니다.\n\n\n### 4. TTA\n* TTA를 적용하는 것만으로도 약 0.5% 정확도가 상승했습니다. 원본 이미지 output + 9번의 TTA output, 총 10개의 값을 평균내서 최종 output을 구했습니다. NN도 적용해봤는데 오히려 평균한 것보다 score가 떨어지더군요. Augmentation 방법은 학습했던 것과 동일하게 Autoaugmentation를 사용했습니다.\n\n### 5. Ensemble\n\n* 최종 8개 값으로 Ensemble 방법 중 몇 가지를 실험해보았습니다. \n\n    1. 단순 평균\n    2. 모델의 public score로 가중치를 부여하는 Weight 평균\n    3. 8개의 모델 output(8, 6150, 196)을 NN을 통과시켜 계산한 결과값.\n\n* 1번과 2번는 public score 기준 0.95265, 3번은 0.95361을 보여주었습니다.\n\n### 6. Etc....\n* `categorical_crossentropy` 대신 Focal loss을 적용해보았습니다. 오히려 정확도가 떨어져서 적용하지 않았습니다. focal loss는 아래 코드를 사용했는데 혹시 다른 코드가 있으면 comment에 남겨주시면 감사하겠습니다.\n\n```\ngamma = 2.0\nepsilon = K.epsilon() #1e-7\ndef focal_loss(y_true, y_pred):\n    pt = y_pred * y_true + (1-y_pred) * (1-y_true)\n    pt = K.clip(pt, epsilon, 1-epsilon)\n    CE = -K.log(pt)\n    FL = K.pow(1-pt, gamma) * CE\n    loss = K.mean(FL, axis=1)\n    return loss\n```\n* 자동차의 종류와 브랜드를 결합해서 모델을 만들어보았습니다. class.csv 파일을 살펴보면, 자동차의 브랜드와 종류가 잘 나눠져 있어, 49개의 브랜드, 9개의 차종으로 분류하는 CNN 모델을 학습시키고 (196,), (49, ), (9, )를 concat하여 Dense로 연결시킨 모델을 구성했습니다. Moblienet 기준, 2% 상승했으며 EfficientNetB4에도 적용하려고 했으나 커널 메모리가 터져버리는 바람에 학습에 실패했네요ㅠ "},{"metadata":{},"cell_type":"markdown","source":"아래부터는 모델 weight를 load해서 TTA하는 과정입니다."},{"metadata":{},"cell_type":"markdown","source":"## Set up "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport numpy as np \nimport pandas as pd \nimport os\nimport pickle\nprint(os.listdir(\"../input\"))\n\nimport matplotlib.pyplot as plt\nimport skimage.io\nfrom skimage.transform import resize\nfrom tqdm import tqdm\nfrom tqdm import tqdm_notebook\nimport random\nimport PIL\nfrom PIL import Image, ImageOps\nimport cv2\nfrom sklearn.utils import class_weight, shuffle\nfrom keras.losses import binary_crossentropy, categorical_crossentropy\nfrom keras.applications.xception import preprocess_input\nimport keras.backend as K\nimport tensorflow as tf\nfrom sklearn.metrics import f1_score, fbeta_score, cohen_kappa_score, accuracy_score\nfrom keras.utils import Sequence\nfrom keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Install efficientNet\n\nefficientNet input size by https://github.com/lukemelas/EfficientNet-PyTorch/issues/42\n\n        # Coefficients:   width,depth,res,dropout\n        'efficientnet-b0': (1.0, 1.0, 224, 0.2),\n        'efficientnet-b1': (1.0, 1.1, 240, 0.2),\n        'efficientnet-b2': (1.1, 1.2, 260, 0.3),\n        'efficientnet-b3': (1.2, 1.4, 300, 0.3),\n        'efficientnet-b4': (1.4, 1.8, 380, 0.4),\n        'efficientnet-b5': (1.6, 2.2, 456, 0.4),\n        'efficientnet-b6': (1.8, 2.6, 528, 0.5),\n        'efficientnet-b7': (2.0, 3.1, 600, 0.5),"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -U efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from efficientnet import EfficientNetB4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Hyper parameters\nWORKERS = 2\nCHANNEL = 3\nSIZE = 380\nNUM_CLASSES = 196\nepoch = 20\nbatch_size = 16","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#seed 고정\n#Fix seed number.\nseed = 200\n\nnp.random.seed(seed)\ntf.set_random_seed(seed)\nos.environ['PYTHONHASHSEED']=str(seed)\nrandom.seed(seed)\nsession_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\nsess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\nK.set_session(sess)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## data augmentation\n\n* Autoaugmentation by ImageNetPolicy, https://github.com/DeepVoltaire/AutoAugment"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from PIL import Image, ImageEnhance, ImageOps\nimport numpy as np\nimport random\n\nclass ImageNetPolicy(object):\n    \"\"\" Randomly choose one of the best 24 Sub-policies on ImageNet.\n        Example:\n        >>> policy = ImageNetPolicy()\n        >>> transformed = policy(image)\n        Example as a PyTorch Transform:\n        >>> transform=transforms.Compose([\n        >>>     transforms.Resize(256),\n        >>>     ImageNetPolicy(),\n        >>>     transforms.ToTensor()])\n    \"\"\"\n    def __init__(self, fillcolor=(128, 128, 128)):\n        self.policies = [\n            SubPolicy(0.4, \"posterize\", 8, 0.6, \"rotate\", 9, fillcolor),\n            SubPolicy(0.6, \"solarize\", 5, 0.6, \"autocontrast\", 5, fillcolor),\n            SubPolicy(0.8, \"equalize\", 8, 0.6, \"equalize\", 3, fillcolor),\n            SubPolicy(0.6, \"posterize\", 7, 0.6, \"posterize\", 6, fillcolor),\n            SubPolicy(0.4, \"equalize\", 7, 0.2, \"solarize\", 4, fillcolor),\n\n            SubPolicy(0.4, \"equalize\", 4, 0.8, \"rotate\", 8, fillcolor),\n            SubPolicy(0.6, \"solarize\", 3, 0.6, \"equalize\", 7, fillcolor),\n            SubPolicy(0.8, \"posterize\", 5, 1.0, \"equalize\", 2, fillcolor),\n            SubPolicy(0.2, \"rotate\", 3, 0.6, \"solarize\", 8, fillcolor),\n            SubPolicy(0.6, \"equalize\", 8, 0.4, \"posterize\", 6, fillcolor),\n\n            SubPolicy(0.8, \"rotate\", 8, 0.4, \"color\", 0, fillcolor),\n            SubPolicy(0.4, \"rotate\", 9, 0.6, \"equalize\", 2, fillcolor),\n            SubPolicy(0.0, \"equalize\", 7, 0.8, \"equalize\", 8, fillcolor),\n            SubPolicy(0.6, \"invert\", 4, 1.0, \"equalize\", 8, fillcolor),\n            SubPolicy(0.6, \"color\", 4, 1.0, \"contrast\", 8, fillcolor),\n\n            SubPolicy(0.8, \"rotate\", 8, 1.0, \"color\", 2, fillcolor),\n            SubPolicy(0.8, \"color\", 8, 0.8, \"solarize\", 7, fillcolor),\n            SubPolicy(0.4, \"sharpness\", 7, 0.6, \"invert\", 8, fillcolor),\n            SubPolicy(0.6, \"shearX\", 5, 1.0, \"equalize\", 9, fillcolor),\n            SubPolicy(0.4, \"color\", 0, 0.6, \"equalize\", 3, fillcolor),\n\n            SubPolicy(0.4, \"equalize\", 7, 0.2, \"solarize\", 4, fillcolor),\n            SubPolicy(0.6, \"solarize\", 5, 0.6, \"autocontrast\", 5, fillcolor),\n            SubPolicy(0.6, \"invert\", 4, 1.0, \"equalize\", 8, fillcolor),\n            SubPolicy(0.6, \"color\", 4, 1.0, \"contrast\", 8, fillcolor),\n            SubPolicy(0.8, \"equalize\", 8, 0.6, \"equalize\", 3, fillcolor)\n        ]\n\n\n    def __call__(self, img):\n        policy_idx = random.randint(0, len(self.policies) - 1)\n        return self.policies[policy_idx](img)\n\n    def __repr__(self):\n        return \"AutoAugment ImageNet Policy\"\nclass SubPolicy(object):\n    def __init__(self, p1, operation1, magnitude_idx1, p2, operation2, magnitude_idx2, fillcolor=(128, 128, 128)):\n        ranges = {\n            \"shearX\": np.linspace(0, 0.3, 10),\n            \"shearY\": np.linspace(0, 0.3, 10),\n            \"translateX\": np.linspace(0, 150 / 331, 10),\n            \"translateY\": np.linspace(0, 150 / 331, 10),\n            \"rotate\": np.linspace(0, 30, 10),\n            \"color\": np.linspace(0.0, 0.9, 10),\n            \"posterize\": np.round(np.linspace(8, 4, 10), 0).astype(np.int),\n            \"solarize\": np.linspace(256, 0, 10),\n            \"contrast\": np.linspace(0.0, 0.9, 10),\n            \"sharpness\": np.linspace(0.0, 0.9, 10),\n            \"brightness\": np.linspace(0.0, 0.9, 10),\n            \"autocontrast\": [0] * 10,\n            \"equalize\": [0] * 10,\n            \"invert\": [0] * 10\n        }\n\n        # from https://stackoverflow.com/questions/5252170/specify-image-filling-color-when-rotating-in-python-with-pil-and-setting-expand\n        def rotate_with_fill(img, magnitude):\n            rot = img.convert(\"RGBA\").rotate(magnitude)\n            return Image.composite(rot, Image.new(\"RGBA\", rot.size, (128,) * 4), rot).convert(img.mode)\n\n        func = {\n            \"shearX\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, magnitude * random.choice([-1, 1]), 0, 0, 1, 0),\n                Image.BICUBIC, fillcolor=fillcolor),\n            \"shearY\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, 0, 0, magnitude * random.choice([-1, 1]), 1, 0),\n                Image.BICUBIC, fillcolor=fillcolor),\n            \"translateX\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, 0, magnitude * img.size[0] * random.choice([-1, 1]), 0, 1, 0),\n                fillcolor=fillcolor),\n            \"translateY\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, 0, 0, 0, 1, magnitude * img.size[1] * random.choice([-1, 1])),\n                fillcolor=fillcolor),\n            \"rotate\": lambda img, magnitude: rotate_with_fill(img, magnitude),\n            # \"rotate\": lambda img, magnitude: img.rotate(magnitude * random.choice([-1, 1])),\n            \"color\": lambda img, magnitude: ImageEnhance.Color(img).enhance(1 + magnitude * random.choice([-1, 1])),\n            \"posterize\": lambda img, magnitude: ImageOps.posterize(img, magnitude),\n            \"solarize\": lambda img, magnitude: ImageOps.solarize(img, magnitude),\n            \"contrast\": lambda img, magnitude: ImageEnhance.Contrast(img).enhance(\n                1 + magnitude * random.choice([-1, 1])),\n            \"sharpness\": lambda img, magnitude: ImageEnhance.Sharpness(img).enhance(\n                1 + magnitude * random.choice([-1, 1])),\n            \"brightness\": lambda img, magnitude: ImageEnhance.Brightness(img).enhance(\n                1 + magnitude * random.choice([-1, 1])),\n            \"autocontrast\": lambda img, magnitude: ImageOps.autocontrast(img),\n            \"equalize\": lambda img, magnitude: ImageOps.equalize(img),\n            \"invert\": lambda img, magnitude: ImageOps.invert(img)\n        }\n\n        # self.name = \"{}_{:.2f}_and_{}_{:.2f}\".format(\n        #     operation1, ranges[operation1][magnitude_idx1],\n        #     operation2, ranges[operation2][magnitude_idx2])\n        self.p1 = p1\n        self.operation1 = func[operation1]\n        self.magnitude1 = ranges[operation1][magnitude_idx1]\n        self.p2 = p2\n        self.operation2 = func[operation2]\n        self.magnitude2 = ranges[operation2][magnitude_idx2]\n\n\n    def __call__(self, img):\n        if random.random() < self.p1: img = self.operation1(img, self.magnitude1)\n        if random.random() < self.p2: img = self.operation2(img, self.magnitude2)\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trans = ImageNetPolicy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load csv data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#데이터 path\npath = '../input/2019-3rd-ml-month-with-kakr/'\n#train_img_path = os.path.join(path, 'train')\n#test_img_path = os.path.join(path, 'test')\ncrop_train_path = '../input/2019-3rd-ml-month-with-kakr/train/'\ncrop_test_path = '../input/2019-3rd-ml-month-with-kakr/test/'\nweight_path = '../input/model-weight/'\n\n#Load csv files\ndf_train = pd.read_csv(os.path.join(path, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(path, 'test.csv'))\ndf_class = pd.read_csv(os.path.join(path, 'class.csv'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"여기서 class에 1을 빼주는 이유는 `df_train['class']`0이 아닌 1부터 클래스 부여가 되어있어 따로 `to_categorical`을 적용하면 에러가 발생합니다. 이부분을 해결하기 위해 1을 뺐습니다. 마지막 부분에서 다시 1을 더해줘야합니다."},{"metadata":{"trusted":true},"cell_type":"code","source":"#To use to_categorical function, add -1\ndf_train['class'] = df_train['class'].values.astype(int) -1\ndf_train['class'] = df_train['class'].astype('str')\n\nx = df_train['img_file']\ny = df_train['class']\ny = to_categorical(y, num_classes=NUM_CLASSES)\n\ntrain_x, valid_x, train_y, valid_y = train_test_split(x, y, test_size=0.15,\n                                                      stratify=y, random_state=8)\nprint(train_x.shape)\nprint(train_y.shape)\nprint(valid_x.shape)\nprint(valid_y.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Generator\n\nCustom Generator를 생성합니다. `augmen` 로 augmentation을 적용할지 선택합니다."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from keras.utils import Sequence\n\nclass My_Generator(Sequence):\n\n    def __init__(self, image_filenames, labels,\n                 batch_size, is_train=False, is_test=False,\n                 mix=False, augment=False):\n        self.image_filenames, self.labels = image_filenames, labels\n        self.batch_size = batch_size\n        self.is_train = is_train\n        self.is_test = is_test\n        self.is_augment = augment\n        if(self.is_train):\n            self.on_epoch_end()\n        self.is_mix = mix\n\n    def __len__(self):\n        return int(np.ceil(len(self.image_filenames) / float(self.batch_size)))\n\n    def __getitem__(self, idx):\n        batch_x = self.image_filenames[idx * self.batch_size:(idx + 1) * self.batch_size]\n        batch_y = self.labels[idx * self.batch_size:(idx + 1) * self.batch_size]\n\n        if(self.is_train):\n            return self.train_generate(batch_x, batch_y)\n        if(self.is_test):\n            return self.test_generate(batch_x)\n        return self.valid_generate(batch_x, batch_y)\n\n    def on_epoch_end(self):\n        if(self.is_train):\n            self.image_filenames, self.labels = shuffle(self.image_filenames, self.labels)\n    \n    def mix_up(self, x, y):\n        lam = np.random.beta(0.2, 0.4)\n        ori_index = np.arange(int(len(x)))\n        index_array = np.arange(int(len(x)))\n        np.random.shuffle(index_array)        \n        \n        mixed_x = lam * x[ori_index] + (1 - lam) * x[index_array]\n        mixed_y = lam * y[ori_index] + (1 - lam) * y[index_array]\n        \n        return mixed_x, mixed_y\n\n    def train_generate(self, batch_x, batch_y):\n        batch_images = []\n        for (sample, label) in zip(batch_x, batch_y):\n            #img = cv2.imread(os.path.join(crop_train_path, sample))\n            #img = cv2.resize(img, (SIZE, SIZE))\n            img = PIL.Image.open(os.path.join(crop_train_path, sample)).convert('RGB')\n            img = img.resize((SIZE, SIZE))\n            \n            if(self.is_augment):\n                #img = seq.augment_image(img)\n                img = trans(img)\n            img_np  = np.asarray(img)\n            if img_np.shape != (240, 240, 3):\n                img = img.convert('RGB')\n                img_np  = np.asarray(img)   \n            batch_images.append(img_np)\n        batch_images = np.array(batch_images, np.float32) / 255\n        batch_y = np.array(batch_y, np.float32)\n        if(self.is_mix):\n            batch_images, batch_y = self.mix_up(batch_images, batch_y)\n        return batch_images, batch_y\n\n    def valid_generate(self, batch_x, batch_y):\n        batch_images = []\n        for (sample, label) in zip(batch_x, batch_y):\n            #img = cv2.imread(os.path.join(crop_train_path, sample))\n            #img = cv2.resize(img, (SIZE, SIZE))\n            img = PIL.Image.open(os.path.join(crop_train_path, sample)).convert('RGB')\n            img = img.resize((SIZE, SIZE))\n            img  = np.asarray(img)\n            batch_images.append(img)\n        batch_images = np.array(batch_images, np.float32) / 255\n        batch_y = np.array(batch_y, np.float32)\n        return batch_images, batch_y\n    \n    def test_generate(self, batch_x):\n        batch_images = []\n        for sample in batch_x:\n            #img = cv2.imread(os.path.join(crop_train_path, sample))\n            #img = cv2.resize(img, (SIZE, SIZE))\n            img = PIL.Image.open(os.path.join(crop_test_path, sample)).convert('RGB')\n            img = img.resize((SIZE, SIZE))\n            \n            if (self.is_augment):\n                img = trans(img)\n            img_np  = np.asarray(img)\n            if img_np.shape != (380, 380, 3): #위의 Auto Augmentation을 적용하면 드물게 grayscale한 결과를 주게 되는데, 이때 에러를 방지하기 위함.\n                img = img.convert('RGB')\n                img_np  = np.asarray(img)\n            batch_images.append(img_np)\n        batch_images = np.array(batch_images, np.float32) / 255\n        return batch_images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submit = pd.read_csv(os.path.join(path, 'sample_submission.csv'))\ntest_x = submit['img_file']\ntest_y = submit['class']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_step = np.ceil(len(test_x) / batch_size)\ntest_generator = My_Generator(test_x, test_y, batch_size=batch_size, is_train=False, is_test=True)\ntest_aug_generator = My_Generator(test_x, test_y, batch_size=batch_size, is_train=False, is_test=True, augment=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create model"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential, load_model\nfrom keras.layers import (Activation, Dropout, Flatten, Dense, GlobalMaxPooling2D,\n                          BatchNormalization, Input, Conv2D, GlobalAveragePooling2D, Concatenate)\n#from keras.applications.xception import Xception\nfrom keras.callbacks import ModelCheckpoint\nfrom keras import metrics\nfrom keras.optimizers import Adam \nfrom keras import backend as K\nimport keras\nfrom keras.models import Model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"function = \"softmax\"\ndef create_model(input_shape, n_out):\n    input_tensor = Input(shape=input_shape)\n    base_model = EfficientNetB4(include_top=False,\n                   #weights='imagenet',\n                    weights=None,\n                   input_tensor=input_tensor)\n    #base_model.load_weights('../input/model-weight/xception_weights_tf_dim_ordering_tf_kernels_notop.h5')\n    x = GlobalAveragePooling2D()(base_model.output)\n    x = Dropout(0.5)(x)\n    x = Dense(1024, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    final_output = Dense(n_out, activation=function, name='final_output')(x)\n    model = Model(input_tensor, final_output)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict by Test Time augmentation "},{"metadata":{"trusted":true},"cell_type":"code","source":"def generator_predict(model, filename=None, test_generator=test_generator, test_aug_generator=test_aug_generator,batch_size=batch_size):\n    '''\n        input param\n            model : 사용할 모델\n            filename : output을 저장할 pickle 파일명, 만약 입력하지 않으면 저정하지 않음. \n            test_generator : 원본 image generator\n            test_aug_generator : Augmentation을 적용한 image generator\n        return \n            TTA한 값을 평균한 output\n    '''\n    pred_list = []\n    #org predict\n    pred_list.append(model.predict_generator(test_generator, \n                                             steps=train_step, \n                                             #steps=5,\n                                             verbose=1\n                                            ))\n    #Add TTA predict\n    for i in range(9):\n        pred_list.append(model.predict_generator(test_aug_generator, \n                                                 steps=train_step, \n                                                 #steps=5,\n                                                 #verbose=1\n                                                ))\n\n    #Store org predict score\n    if filename:\n        filename = filename + '.pickle'\n        with open(filename, 'wb') as f:\n                pickle.dump(pred_list, f, pickle.HIGHEST_PROTOCOL)\n\n    #Predict by score avg\n    predicted = np.mean(pred_list, axis=0)\n    return predicted #(Num of image, NUM of classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_list = ['effi_basic.h5', \n              'effi_basic_add_4f.h5', \n              'effi_basic_add.h5', \n              'effi_f1_bestqwk.h5', \n              'effi_f2_bestqwk.h5', \n              'effi_f3_bestqwk.h5', \n              'effi_f4_bestqwk.h5',\n              'effi_f5_bestqwk.h5']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_pred = []\nfor i, name in enumerate(tqdm_notebook(model_list)):\n    test_generator = My_Generator(test_x,test_y, batch_size=batch_size, is_train=False, is_test=True)\n    test_aug_generator = My_Generator(test_x, test_y, batch_size=batch_size, is_train=False, is_test=True, augment=True)\n    \n    print('{} model predict'.format(name))\n    model = create_model(\n        input_shape=(SIZE,SIZE,3), \n        n_out=NUM_CLASSES)\n    model.load_weights(os.path.join(weight_path, name))\n    model_pred.append(generator_predict(model=model, filename=name))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"모든 결과값을 평균내서 최종 결과값을 계산합니다."},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"predict = np.mean(model_pred, axis=0)\npredict = np.argmax(predict, axis=1) + 1\nsubmit['class'] = predict\nprint(submit.head())","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"100e284c8eec4bf1be15ca4bb2c737a9":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"1ac67b62fef84332ad2d541bd10978eb":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"288a01aed9f54f77b984d55fe02c730f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"","description":"","description_tooltip":null,"layout":"IPY_MODEL_cb06fb02e20c41a483ef90382e9b2da8","max":6150,"min":0,"orientation":"horizontal","style":"IPY_MODEL_1ac67b62fef84332ad2d541bd10978eb","value":3212}},"3c059aed6d7445dab1a9e7208289bfcf":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_8cf005e268bd4958b083d6b918bab074","placeholder":"​","style":"IPY_MODEL_66be30b28d754f1bb86cdf9e229bb41a","value":" 52% 3212/6150 [20:39&lt;18:52,  2.60it/s]"}},"6466f3dd8e764e5a88e75d454919cef0":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_288a01aed9f54f77b984d55fe02c730f","IPY_MODEL_3c059aed6d7445dab1a9e7208289bfcf"],"layout":"IPY_MODEL_100e284c8eec4bf1be15ca4bb2c737a9"}},"66be30b28d754f1bb86cdf9e229bb41a":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"8cf005e268bd4958b083d6b918bab074":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"cb06fb02e20c41a483ef90382e9b2da8":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":1}