{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Kaggle Korea 3rd Competition - Car Image Classification\n\n안녕하세요. 이전에 딥러닝에 대해 공부하긴 했지만 실제 프로젝트를 하거나 문제를 풀어본 건 이번이 처음이었는데요.\n\n대회 참여하면서 많은 분들이 훌륭한 커널을 공개해주신 덕에 실제 이미지 분류 문제를 어떻게 풀어야 하는지 여러가지 테크닉을 많이 배우고 적용해 볼 수 있었습니다.\n\n좋은 기회를 마련해주신 캐글 코리아 운영진에게 진심으로 감사드립니다.\n\n저는 최종 스코어 프라이빗 리더보드 기준으로 0.95491이 나왔고 순위는 12위를 했는데요.\n\n솔루션 공유를 위해 커널을 다시 재현해보니 0.95565로 최종 성적보다 좀 더 높게 나오네요.\n\n최종 커널은 양세원님의 https://www.kaggle.com/yangsaewon/pytorch-baseline-updated-7-10 를 기반으로 만들었습니다.\n\n트레이닝까지 하면 시간이 너무 오래 걸려서 트레이닝 부분은 fold 1만 하게 했고, 실제 트레이닝도 flag 변수로 스킵하게 했습니다.\n\n자세한 설명을 쓰지는 못했는데, 혹시 궁금한 점 문의주시면 답변 드리겠습니다.\n\n보시고 도움이 됐다면 제 Kernel에 Vote 해주시면 감사하겠습니다!"},{"metadata":{},"cell_type":"markdown","source":"# Deep Learning Framework\n\n## Fast.ai\n\n처음 대회 참여했을 때는 이호민님이 공유해주신 Fast.ai 기반의 커널을 참고로 해서 퍼블릭 스코어 기준으로 0.89까지 점수를 낼 수 있었습니다.\n\n## Keras\n\nFast.ai가 쓰기는 편하지만 커스터마이징 하기에는 어려워서 가장 많이 쓰이는 Keras에 대해 공부를 하고,\n\n장은동님이 공유해주신 커널을 기반으로 5 fold ensemble까지 해서 퍼블릭 스코어 기준 0.95까지 점수를 올렸습니다.\n\n## PyTorch\n\n이후에 더 이상 점수가 오르지 않고 정체기를 겪다가, 새로운 프레임워크를 공부해볼겸 양세원님이 공유해주신 PyTorch 기반의 커널을 기반으로 진행을 했고 여기서 가장 성능이 잘 나오는 모델을 만들 수 있었습니다."},{"metadata":{},"cell_type":"markdown","source":"# Approach"},{"metadata":{},"cell_type":"markdown","source":"# Image Preprocessing\n\n* Bounding box crop\n* Bruce Kim님이 공유해주신 커널의 Histogram Equalization 적용\n\n# Image Augmentation\n\n* 이호민님이 공유해주신 AutoAugmentation의 Cifar10 Policy\n* 김찬란님이 공유해주신 Cutout\n\n# CNN Architecture\n\n* SE-ResNeXt50 32x4d \n* EfficientNet B3\n\n# Learning Rate Scheduling\n\n* 양세원님이 공유해주신 SGDR에 파라미터만 수정\n\n# Loss Function\n\n* Class 분포에 불균형한 데이터가 있어서 Focal Loss 사용 \n\n# Ensemble\n\n* SE-ResNeXt50 32x4d 5 fold + 3 tta\n* EfficientNet B3 4 fold + 3 tta\n* SE-ResNeXt50 32x4d, EfficientNet B3의 weighted average ensemble"},{"metadata":{},"cell_type":"markdown","source":"# Reference Kernel\n\n* https://www.kaggle.com/yangsaewon/pytorch-baseline-updated-7-10\n* https://www.kaggle.com/janged/3rd-ml-month-xception-stratifiedkfold-ensemble\n* https://www.kaggle.com/seriousran/cutout-augmentation-on-keras-efficientnet\n* https://www.kaggle.com/hominlee/ka-kr-sillim-fast-ai-high-level-framework\n* https://www.kaggle.com/yangsaewon/pytorch-inference-tta\n* https://www.kaggle.com/hominlee/ka-kr-sillim-autoaugmentation\n* https://www.kaggle.com/devbruce/kakr-2019-3rd-eda-imageprep-mixup-cv-keras"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pretrainedmodels==0.7.4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet-pytorch==0.4.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport math\nimport random\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path\nimport glob\nimport matplotlib.pyplot as plt\nfrom PIL import Image, ImageEnhance, ImageOps\n\nfrom tqdm import tqdm, tqdm_notebook\n\nimport torch\nfrom torch import nn, cuda\nfrom torch.autograd import Variable \nimport torch.nn.functional as F\nimport torchvision as vision\nimport torchvision.models as models\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.optim import Adam, SGD, Optimizer\nfrom torch.optim.lr_scheduler import _LRScheduler, CosineAnnealingLR, ReduceLROnPlateau\nimport pretrainedmodels\nfrom efficientnet_pytorch import EfficientNet\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import f1_score\n\nimport warnings\nwarnings.filterwarnings(action='ignore')\n\npd.set_option('display.max_columns', 200)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class AdamW(Optimizer):\n    \"\"\"Implements AdamW algorithm.\n\n    It has been proposed in `Fixing Weight Decay Regularization in Adam`_.\n\n    Arguments:\n        params (iterable): iterable of parameters to optimize or dicts defining\n            parameter groups\n        lr (float, optional): learning rate (default: 1e-3)\n        betas (Tuple[float, float], optional): coefficients used for computing\n            running averages of gradient and its square (default: (0.9, 0.999))\n        eps (float, optional): term added to the denominator to improve\n            numerical stability (default: 1e-8)\n        weight_decay (float, optional): weight decay (L2 penalty) (default: 0)\n\n    .. Fixing Weight Decay Regularization in Adam:\n    https://arxiv.org/abs/1711.05101\n    \"\"\"\n\n    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8,\n                 weight_decay=0):\n        defaults = dict(lr=lr, betas=betas, eps=eps,\n                        weight_decay=weight_decay)\n        super(AdamW, self).__init__(params, defaults)\n\n    def step(self, closure=None):\n        \"\"\"Performs a single optimization step.\n\n        Arguments:\n            closure (callable, optional): A closure that reevaluates the model\n                and returns the loss.\n        \"\"\"\n        loss = None\n        if closure is not None:\n            loss = closure()\n\n        for group in self.param_groups:\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad.data\n                if grad.is_sparse:\n                    raise RuntimeError('AdamW does not support sparse gradients, please consider SparseAdam instead')\n\n                state = self.state[p]\n\n                # State initialization\n                if len(state) == 0:\n                    state['step'] = 0\n                    # Exponential moving average of gradient values\n                    state['exp_avg'] = torch.zeros_like(p.data)\n                    # Exponential moving average of squared gradient values\n                    state['exp_avg_sq'] = torch.zeros_like(p.data)\n\n                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']\n                beta1, beta2 = group['betas']\n\n                state['step'] += 1\n\n                # according to the paper, this penalty should come after the bias correction\n                # if group['weight_decay'] != 0:\n                #     grad = grad.add(group['weight_decay'], p.data)\n\n                # Decay the first and second moment running average coefficient\n                exp_avg.mul_(beta1).add_(1 - beta1, grad)\n                exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)\n\n                denom = exp_avg_sq.sqrt().add_(group['eps'])\n\n                bias_correction1 = 1 - beta1 ** state['step']\n                bias_correction2 = 1 - beta2 ** state['step']\n                step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1\n\n                p.data.addcdiv_(-step_size, exp_avg, denom)\n\n                if group['weight_decay'] != 0:\n                    p.data.add_(-group['weight_decay'], p.data)\n\n        return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CosineAnnealingWithRestartsLR(_LRScheduler):\n    '''\n    SGDR\\: Stochastic Gradient Descent with Warm Restarts: https://arxiv.org/abs/1608.03983\n    code: https://github.com/gurucharanmk/PyTorch_CosineAnnealingWithRestartsLR/blob/master/CosineAnnealingWithRestartsLR.py\n    added restart_decay value to decrease lr for every restarts\n    '''\n    def __init__(self, optimizer, T_max, eta_min=0, last_epoch=-1, T_mult=1, restart_decay=0.95):\n        self.T_max = T_max\n        self.T_mult = T_mult\n        self.next_restart = T_max\n        self.eta_min = eta_min\n        self.restarts = 0\n        self.last_restart = 0\n        self.T_num = 0\n        self.restart_decay = restart_decay\n        super(CosineAnnealingWithRestartsLR,self).__init__(optimizer, last_epoch)\n\n    def get_lr(self):\n        self.Tcur = self.last_epoch - self.last_restart\n        if self.Tcur >= self.next_restart:\n            self.next_restart *= self.T_mult\n            self.last_restart = self.last_epoch\n            self.T_num += 1\n        learning_rate = [(self.eta_min + ((base_lr)*self.restart_decay**self.T_num - self.eta_min) * (1 + math.cos(math.pi * self.Tcur / self.next_restart)) / 2) for base_lr in self.base_lrs]\n        return learning_rate","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n유명한 mixup 논문. 타대회에서도 많이 씁니다.\nhttps://arxiv.org/abs/1710.09412\n'''\ndef mixup_criterion(criterion, pred, y_a, y_b, lam):\n    return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)\n\ndef mixup_data(x, y, alpha=1.0, use_cuda=True):\n    '''Returns mixed inputs, pairs of targets, and lambda'''\n    if alpha > 0:\n        lam = np.random.beta(alpha, alpha)\n    else:\n        lam = 1\n\n    batch_size = x.size()[0]\n    if use_cuda:\n        index = torch.randperm(batch_size).cuda()\n    else:\n        index = torch.randperm(batch_size)\n\n    mixed_x = lam * x + (1 - lam) * x[index, :]\n    y_a, y_b = y, y[index]\n    return mixed_x, y_a, y_b, lam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ImageNetPolicy(object):\n    \"\"\" Randomly choose one of the best 24 Sub-policies on ImageNet.\n        Example:\n        >>> policy = ImageNetPolicy()\n        >>> transformed = policy(image)\n        Example as a PyTorch Transform:\n        >>> transform=transforms.Compose([\n        >>>     transforms.Resize(256),\n        >>>     ImageNetPolicy(),\n        >>>     transforms.ToTensor()])\n    \"\"\"\n    def __init__(self, fillcolor=(128, 128, 128)):\n        self.policies = [\n            SubPolicy(0.4, \"posterize\", 8, 0.6, \"rotate\", 9, fillcolor),\n            SubPolicy(0.6, \"solarize\", 5, 0.6, \"autocontrast\", 5, fillcolor),\n            SubPolicy(0.8, \"equalize\", 8, 0.6, \"equalize\", 3, fillcolor),\n            SubPolicy(0.6, \"posterize\", 7, 0.6, \"posterize\", 6, fillcolor),\n            SubPolicy(0.4, \"equalize\", 7, 0.2, \"solarize\", 4, fillcolor),\n\n            SubPolicy(0.4, \"equalize\", 4, 0.8, \"rotate\", 8, fillcolor),\n            SubPolicy(0.6, \"solarize\", 3, 0.6, \"equalize\", 7, fillcolor),\n            SubPolicy(0.8, \"posterize\", 5, 1.0, \"equalize\", 2, fillcolor),\n            SubPolicy(0.2, \"rotate\", 3, 0.6, \"solarize\", 8, fillcolor),\n            SubPolicy(0.6, \"equalize\", 8, 0.4, \"posterize\", 6, fillcolor),\n\n            SubPolicy(0.8, \"rotate\", 8, 0.4, \"color\", 0, fillcolor),\n            SubPolicy(0.4, \"rotate\", 9, 0.6, \"equalize\", 2, fillcolor),\n            SubPolicy(0.0, \"equalize\", 7, 0.8, \"equalize\", 8, fillcolor),\n            SubPolicy(0.6, \"invert\", 4, 1.0, \"equalize\", 8, fillcolor),\n            SubPolicy(0.6, \"color\", 4, 1.0, \"contrast\", 8, fillcolor),\n\n            SubPolicy(0.8, \"rotate\", 8, 1.0, \"color\", 2, fillcolor),\n            SubPolicy(0.8, \"color\", 8, 0.8, \"solarize\", 7, fillcolor),\n            SubPolicy(0.4, \"sharpness\", 7, 0.6, \"invert\", 8, fillcolor),\n            SubPolicy(0.6, \"shearX\", 5, 1.0, \"equalize\", 9, fillcolor),\n            SubPolicy(0.4, \"color\", 0, 0.6, \"equalize\", 3, fillcolor),\n\n            SubPolicy(0.4, \"equalize\", 7, 0.2, \"solarize\", 4, fillcolor),\n            SubPolicy(0.6, \"solarize\", 5, 0.6, \"autocontrast\", 5, fillcolor),\n            SubPolicy(0.6, \"invert\", 4, 1.0, \"equalize\", 8, fillcolor),\n            SubPolicy(0.6, \"color\", 4, 1.0, \"contrast\", 8, fillcolor),\n            SubPolicy(0.8, \"equalize\", 8, 0.6, \"equalize\", 3, fillcolor)\n        ]\n\n\n    def __call__(self, img):\n        policy_idx = random.randint(0, len(self.policies) - 1)\n        return self.policies[policy_idx](img)\n\n    def __repr__(self):\n        return \"AutoAugment ImageNet Policy\"\n\n\nclass CIFAR10Policy(object):\n    \"\"\" Randomly choose one of the best 25 Sub-policies on CIFAR10.\n        Example:\n        >>> policy = CIFAR10Policy()\n        >>> transformed = policy(image)\n        Example as a PyTorch Transform:\n        >>> transform=transforms.Compose([\n        >>>     transforms.Resize(256),\n        >>>     CIFAR10Policy(),\n        >>>     transforms.ToTensor()])\n    \"\"\"\n    def __init__(self, fillcolor=(128, 128, 128)):\n        self.policies = [\n            SubPolicy(0.1, \"invert\", 7, 0.2, \"contrast\", 6, fillcolor),\n            SubPolicy(0.7, \"rotate\", 2, 0.3, \"translateX\", 9, fillcolor),\n            SubPolicy(0.8, \"sharpness\", 1, 0.9, \"sharpness\", 3, fillcolor),\n            SubPolicy(0.5, \"shearY\", 8, 0.7, \"translateY\", 9, fillcolor),\n            SubPolicy(0.5, \"autocontrast\", 8, 0.9, \"equalize\", 2, fillcolor),\n\n            SubPolicy(0.2, \"shearY\", 7, 0.3, \"posterize\", 7, fillcolor),\n            SubPolicy(0.4, \"color\", 3, 0.6, \"brightness\", 7, fillcolor),\n            SubPolicy(0.3, \"sharpness\", 9, 0.7, \"brightness\", 9, fillcolor),\n            SubPolicy(0.6, \"equalize\", 5, 0.5, \"equalize\", 1, fillcolor),\n            SubPolicy(0.6, \"contrast\", 7, 0.6, \"sharpness\", 5, fillcolor),\n\n            SubPolicy(0.7, \"color\", 7, 0.5, \"translateX\", 8, fillcolor),\n            SubPolicy(0.3, \"equalize\", 7, 0.4, \"autocontrast\", 8, fillcolor),\n            SubPolicy(0.4, \"translateY\", 3, 0.2, \"sharpness\", 6, fillcolor),\n            SubPolicy(0.9, \"brightness\", 6, 0.2, \"color\", 8, fillcolor),\n            SubPolicy(0.5, \"solarize\", 2, 0.0, \"invert\", 3, fillcolor),\n\n            SubPolicy(0.2, \"equalize\", 0, 0.6, \"autocontrast\", 0, fillcolor),\n            SubPolicy(0.2, \"equalize\", 8, 0.8, \"equalize\", 4, fillcolor),\n            SubPolicy(0.9, \"color\", 9, 0.6, \"equalize\", 6, fillcolor),\n            SubPolicy(0.8, \"autocontrast\", 4, 0.2, \"solarize\", 8, fillcolor),\n            SubPolicy(0.1, \"brightness\", 3, 0.7, \"color\", 0, fillcolor),\n\n            SubPolicy(0.4, \"solarize\", 5, 0.9, \"autocontrast\", 3, fillcolor),\n            SubPolicy(0.9, \"translateY\", 9, 0.7, \"translateY\", 9, fillcolor),\n            SubPolicy(0.9, \"autocontrast\", 2, 0.8, \"solarize\", 3, fillcolor),\n            SubPolicy(0.8, \"equalize\", 8, 0.1, \"invert\", 3, fillcolor),\n            SubPolicy(0.7, \"translateY\", 9, 0.9, \"autocontrast\", 1, fillcolor)\n        ]\n\n\n    def __call__(self, img):\n        policy_idx = random.randint(0, len(self.policies) - 1)\n        return self.policies[policy_idx](img)\n\n    def __repr__(self):\n        return \"AutoAugment CIFAR10 Policy\"\n\n\nclass SVHNPolicy(object):\n    \"\"\" Randomly choose one of the best 25 Sub-policies on SVHN.\n        Example:\n        >>> policy = SVHNPolicy()\n        >>> transformed = policy(image)\n        Example as a PyTorch Transform:\n        >>> transform=transforms.Compose([\n        >>>     transforms.Resize(256),\n        >>>     SVHNPolicy(),\n        >>>     transforms.ToTensor()])\n    \"\"\"\n    def __init__(self, fillcolor=(128, 128, 128)):\n        self.policies = [\n            SubPolicy(0.9, \"shearX\", 4, 0.2, \"invert\", 3, fillcolor),\n            SubPolicy(0.9, \"shearY\", 8, 0.7, \"invert\", 5, fillcolor),\n            SubPolicy(0.6, \"equalize\", 5, 0.6, \"solarize\", 6, fillcolor),\n            SubPolicy(0.9, \"invert\", 3, 0.6, \"equalize\", 3, fillcolor),\n            SubPolicy(0.6, \"equalize\", 1, 0.9, \"rotate\", 3, fillcolor),\n\n            SubPolicy(0.9, \"shearX\", 4, 0.8, \"autocontrast\", 3, fillcolor),\n            SubPolicy(0.9, \"shearY\", 8, 0.4, \"invert\", 5, fillcolor),\n            SubPolicy(0.9, \"shearY\", 5, 0.2, \"solarize\", 6, fillcolor),\n            SubPolicy(0.9, \"invert\", 6, 0.8, \"autocontrast\", 1, fillcolor),\n            SubPolicy(0.6, \"equalize\", 3, 0.9, \"rotate\", 3, fillcolor),\n\n            SubPolicy(0.9, \"shearX\", 4, 0.3, \"solarize\", 3, fillcolor),\n            SubPolicy(0.8, \"shearY\", 8, 0.7, \"invert\", 4, fillcolor),\n            SubPolicy(0.9, \"equalize\", 5, 0.6, \"translateY\", 6, fillcolor),\n            SubPolicy(0.9, \"invert\", 4, 0.6, \"equalize\", 7, fillcolor),\n            SubPolicy(0.3, \"contrast\", 3, 0.8, \"rotate\", 4, fillcolor),\n\n            SubPolicy(0.8, \"invert\", 5, 0.0, \"translateY\", 2, fillcolor),\n            SubPolicy(0.7, \"shearY\", 6, 0.4, \"solarize\", 8, fillcolor),\n            SubPolicy(0.6, \"invert\", 4, 0.8, \"rotate\", 4, fillcolor),\n            SubPolicy(0.3, \"shearY\", 7, 0.9, \"translateX\", 3, fillcolor),\n            SubPolicy(0.1, \"shearX\", 6, 0.6, \"invert\", 5, fillcolor),\n\n            SubPolicy(0.7, \"solarize\", 2, 0.6, \"translateY\", 7, fillcolor),\n            SubPolicy(0.8, \"shearY\", 4, 0.8, \"invert\", 8, fillcolor),\n            SubPolicy(0.7, \"shearX\", 9, 0.8, \"translateY\", 3, fillcolor),\n            SubPolicy(0.8, \"shearY\", 5, 0.7, \"autocontrast\", 3, fillcolor),\n            SubPolicy(0.7, \"shearX\", 2, 0.1, \"invert\", 5, fillcolor)\n        ]\n\n\n    def __call__(self, img):\n        policy_idx = random.randint(0, len(self.policies) - 1)\n        return self.policies[policy_idx](img)\n\n    def __repr__(self):\n        return \"AutoAugment SVHN Policy\"\n\n\nclass SubPolicy(object):\n    def __init__(self, p1, operation1, magnitude_idx1, p2, operation2, magnitude_idx2, fillcolor=(128, 128, 128)):\n        ranges = {\n            \"shearX\": np.linspace(0, 0.3, 10),\n            \"shearY\": np.linspace(0, 0.3, 10),\n            \"translateX\": np.linspace(0, 150 / 331, 10),\n            \"translateY\": np.linspace(0, 150 / 331, 10),\n            \"rotate\": np.linspace(0, 30, 10),\n            \"color\": np.linspace(0.0, 0.9, 10),\n            \"posterize\": np.round(np.linspace(8, 4, 10), 0).astype(np.int),\n            \"solarize\": np.linspace(256, 0, 10),\n            \"contrast\": np.linspace(0.0, 0.9, 10),\n            \"sharpness\": np.linspace(0.0, 0.9, 10),\n            \"brightness\": np.linspace(0.0, 0.9, 10),\n            \"autocontrast\": [0] * 10,\n            \"equalize\": [0] * 10,\n            \"invert\": [0] * 10\n        }\n\n        # from https://stackoverflow.com/questions/5252170/specify-image-filling-color-when-rotating-in-python-with-pil-and-setting-expand\n        def rotate_with_fill(img, magnitude):\n            rot = img.convert(\"RGBA\").rotate(magnitude)\n            return Image.composite(rot, Image.new(\"RGBA\", rot.size, (128,) * 4), rot).convert(img.mode)\n\n        func = {\n            \"shearX\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, magnitude * random.choice([-1, 1]), 0, 0, 1, 0),\n                Image.BICUBIC, fillcolor=fillcolor),\n            \"shearY\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, 0, 0, magnitude * random.choice([-1, 1]), 1, 0),\n                Image.BICUBIC, fillcolor=fillcolor),\n            \"translateX\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, 0, magnitude * img.size[0] * random.choice([-1, 1]), 0, 1, 0),\n                fillcolor=fillcolor),\n            \"translateY\": lambda img, magnitude: img.transform(\n                img.size, Image.AFFINE, (1, 0, 0, 0, 1, magnitude * img.size[1] * random.choice([-1, 1])),\n                fillcolor=fillcolor),\n            \"rotate\": lambda img, magnitude: rotate_with_fill(img, magnitude),\n            # \"rotate\": lambda img, magnitude: img.rotate(magnitude * random.choice([-1, 1])),\n            \"color\": lambda img, magnitude: ImageEnhance.Color(img).enhance(1 + magnitude * random.choice([-1, 1])),\n            \"posterize\": lambda img, magnitude: ImageOps.posterize(img, magnitude),\n            \"solarize\": lambda img, magnitude: ImageOps.solarize(img, magnitude),\n            \"contrast\": lambda img, magnitude: ImageEnhance.Contrast(img).enhance(\n                1 + magnitude * random.choice([-1, 1])),\n            \"sharpness\": lambda img, magnitude: ImageEnhance.Sharpness(img).enhance(\n                1 + magnitude * random.choice([-1, 1])),\n            \"brightness\": lambda img, magnitude: ImageEnhance.Brightness(img).enhance(\n                1 + magnitude * random.choice([-1, 1])),\n            \"autocontrast\": lambda img, magnitude: ImageOps.autocontrast(img),\n            \"equalize\": lambda img, magnitude: ImageOps.equalize(img),\n            \"invert\": lambda img, magnitude: ImageOps.invert(img)\n        }\n\n        # self.name = \"{}_{:.2f}_and_{}_{:.2f}\".format(\n        #     operation1, ranges[operation1][magnitude_idx1],\n        #     operation2, ranges[operation2][magnitude_idx2])\n        self.p1 = p1\n        self.operation1 = func[operation1]\n        self.magnitude1 = ranges[operation1][magnitude_idx1]\n        self.p2 = p2\n        self.operation2 = func[operation2]\n        self.magnitude2 = ranges[operation2][magnitude_idx2]\n\n\n    def __call__(self, img):\n        if random.random() < self.p1: img = self.operation1(img, self.magnitude1)\n        if random.random() < self.p2: img = self.operation2(img, self.magnitude2)\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Cutout(object):\n    \"\"\"Randomly mask out one or more patches from an image.\n    Args:\n        n_holes (int): Number of patches to cut out of each image.\n        length (int): The length (in pixels) of each square patch.\n    \"\"\"\n    def __init__(self, n_holes, length):\n        self.n_holes = n_holes\n        self.length = length\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (Tensor): Tensor image of size (C, H, W).\n        Returns:\n            Tensor: Image with n_holes of dimension length x length cut out of it.\n        \"\"\"\n        h = img.size(1)\n        w = img.size(2)\n\n        mask = np.ones((h, w), np.float32)\n\n        for n in range(self.n_holes):\n            y = np.random.randint(h)\n            x = np.random.randint(w)\n\n            y1 = np.clip(y - self.length // 2, 0, h)\n            y2 = np.clip(y + self.length // 2, 0, h)\n            x1 = np.clip(x - self.length // 2, 0, w)\n            x2 = np.clip(x + self.length // 2, 0, w)\n\n            mask[y1: y2, x1: x2] = 0.\n\n        mask = torch.from_numpy(mask)\n        mask = mask.expand_as(img)\n        img = img * mask\n\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# seed value fix\n# seed 값을 고정해야 hyper parameter 바꿀 때마다 결과를 비교할 수 있습니다.\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nSEED = 42\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"use_cuda = cuda.is_available()\nuse_cuda","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_IMAGE_PATH = Path('../input/car-image-preprocessing/train_crop_he/')\nTEST_IMAGE_PATH = Path('../input/car-image-preprocessing/test_crop_he/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class TrainDataset(Dataset):\n    def __init__(self, df, mode='train', transforms=None):\n        self.df = df\n        self.mode = mode\n        self.transform = transforms[self.mode]\n        \n    def __len__(self):\n        return len(self.df)\n            \n    def __getitem__(self, idx):\n        \n        image = Image.open(TRAIN_IMAGE_PATH / self.df['img_file'][idx]).convert(\"RGB\")\n\n        if self.transform:\n            image = self.transform(image)\n\n        label = self.df['class'][idx]\n\n        return image, label\n\n    \nclass TestDataset(Dataset):\n    def __init__(self, df, mode='test', transforms=None):\n        self.df = df\n        self.mode = mode\n        self.transform = transforms[self.mode]\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        \n        image = Image.open(TEST_IMAGE_PATH / self.df['img_file'][idx]).convert(\"RGB\")\n        \n        if self.transform:\n            image = self.transform(image)\n            \n        return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class FocalLoss(nn.Module):\n    def __init__(self, alpha=1., gamma=2.):\n        super().__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n\n    def forward(self, inputs, targets, **kwargs):\n        CE_loss = nn.CrossEntropyLoss(reduction='none')(inputs, targets)\n        pt = torch.exp(-CE_loss)\n        F_loss = self.alpha * ((1-pt)**self.gamma) * CE_loss\n        return F_loss.mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_size = 224\ntarget_size = (image_size, image_size)\n\ndata_transforms = {\n    'train': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomHorizontalFlip(),\n        vision.transforms.RandomRotation(20),\n        CIFAR10Policy(),\n        vision.transforms.ToTensor(),\n        Cutout(n_holes=1, length=image_size//5),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n    'valid': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomResizedCrop(target_size, scale=(0.8,1.0)),\n        vision.transforms.RandomHorizontalFlip(),\n        vision.transforms.ToTensor(),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n    'test': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomResizedCrop(target_size, scale=(0.8,1.0)),\n        vision.transforms.ToTensor(),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n    'tta': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomHorizontalFlip(),\n        vision.transforms.RandomRotation(20),\n        CIFAR10Policy(),\n        vision.transforms.ToTensor(),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = '../input/2019-3rd-ml-month-with-kakr'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(DATA_PATH, 'test.csv'))\ndf_class = pd.read_csv(os.path.join(DATA_PATH, 'class.csv'))\n\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['class'] = df_train['class'] - 1\ndf_train = df_train[['img_file', 'class']]\ndf_test = df_test[['img_file']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_one_epoch(model, criterion, train_loader, optimizer, mixup_loss, accumulation_step=2):\n    \n    model.train()\n    train_loss = 0.\n    optimizer.zero_grad()\n\n    for i, (inputs, targets) in enumerate(train_loader):\n            \n        inputs, targets = inputs.cuda(), targets.cuda()\n\n        if mixup_loss:\n            inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, alpha=0.5, use_cuda = use_cuda) # alpha in [0.4, 1.0] 선택 가능\n            inputs, targets_a, targets_b = map(Variable, (inputs, targets_a, targets_b))\n            outputs = model(inputs)\n            loss = mixup_criterion(criterion, outputs.cuda(), targets_a.cuda(), targets_b.cuda(), lam)\n            \n        else:\n            outputs = model(inputs)\n            loss = criterion(outputs, targets)\n\n        loss.backward()\n        \n        if accumulation_step:\n            if (i+1) % accumulation_step == 0:  \n                optimizer.step()\n                optimizer.zero_grad()\n        else:\n            optimizer.step()\n            optimizer.zero_grad()\n        \n\n        train_loss += loss.item() / len(train_loader)\n        \n    return train_loss\n\n\ndef validation(model, criterion, valid_loader, y_true):\n    \n    model.eval()\n    valid_preds = np.zeros((len(valid_dataset), num_classes))\n    val_loss = 0.\n    \n    with torch.no_grad():\n        for i, (inputs, targets) in enumerate(valid_loader):\n\n            inputs, targets = inputs.cuda(), targets.cuda()\n            \n            outputs = model(inputs).detach()\n            loss = criterion(outputs, targets)\n            valid_preds[i * batch_size: (i+1) * batch_size] = outputs.cpu().numpy()\n            \n            val_loss += loss.item() / len(valid_loader)\n            \n        y_pred = np.argmax(valid_preds, axis=1)\n        val_score = f1_score(y_true, y_pred, average='micro')  \n        \n    return val_loss, val_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 스코어 기준과 loss 기준. lb 점수가 cv score와 비교했을 때 굉장히\n# consistent해서 cv score를 기준으로 합니다.\ndef pick_best_score(result1, result2):\n    if result1['best_score'] < result2['best_score']:\n        return result2\n    else:\n        return result1\n    \ndef pick_best_loss(result1, result2):\n    if result1['best_loss'] < result2['best_loss']:\n        return result1\n    else:\n        return result2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(num_epochs=60, accumulation_step=4, mixup_loss=False, cv_checkpoint=False, fine_tune=False,\n                weight_file_name='weight_best.pt', y_true=None, **train_kwargs):\n    \n    # choose scheduler\n    if fine_tune:\n        lr = 0.00001\n        optimizer = AdamW(model.parameters(), lr=lr, weight_decay=0.000025)   \n        scheduler = ReduceLROnPlateau(optimizer, 'min', patience=10, factor=0.1)\n    else:    \n        lr = 0.01\n        optimizer = SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=0.025)\n        eta_min = 1e-6\n        T_max = 5\n        T_mult = 2\n        restart_decay = 1.0\n        scheduler = CosineAnnealingWithRestartsLR(optimizer,T_max=T_max, eta_min=eta_min, T_mult=T_mult, restart_decay=restart_decay)\n\n    train_result = {}\n    train_result['weight_file_name'] = weight_file_name\n    best_epoch = -1\n    best_score = 0.\n    lrs = []\n    score = []\n    \n    for epoch in range(num_epochs):\n        \n        start_time = time.time()\n\n        train_loss = train_one_epoch(model, criterion, train_loader, optimizer, mixup_loss, accumulation_step)\n        val_loss, val_score = validation(model, criterion, valid_loader, y_true)\n        score.append(val_score)\n    \n        # model save (score or loss?)\n        if cv_checkpoint:\n            if val_score > best_score:\n                best_score = val_score\n                train_result['best_epoch'] = epoch + 1\n                train_result['best_score'] = round(best_score, 5)\n                torch.save(model.state_dict(), weight_file_name)\n        else:\n            if val_loss < best_loss:\n                best_loss = val_loss\n                train_result['best_epoch'] = epoch + 1\n                train_result['best_loss'] = round(best_loss, 5)\n                torch.save(model.state_dict(), weight_file_name)\n        \n        elapsed = time.time() - start_time\n        \n        lr = [_['lr'] for _ in optimizer.param_groups]\n        print(\"Epoch {} - train_loss: {:.4f}  val_loss: {:.4f}  cv_score: {:.4f}  lr: {:.6f}  time: {:.0f}s\".format(\n                epoch+1, train_loss, val_loss, val_score, lr[0], elapsed))\n        \n        for param_group in optimizer.param_groups:\n            lrs.append(param_group['lr'])\n        \n        # scheduler update\n        if fine_tune:\n            if cv_checkpoint:\n                scheduler.step(val_score)\n            else:\n                scheduler.step(val_loss)\n        else:\n            scheduler.step()\n     \n    return train_result, lrs, score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SE-ResNeXt50 32x4d Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_seresnext = False\n\nif train_seresnext:\n    k_folds = 5\n    num_classes = 196\n    skf = StratifiedKFold(n_splits=k_folds, random_state=SEED)\n    start_fold = 1\n    end_fold = 1\n    result_arr = []\n\n    for i, (train_index, valid_index) in enumerate(skf.split(df_train['img_file'], df_train['class'])):\n        fold = i + 1\n        train_df = df_train.iloc[train_index, :].reset_index()\n        valid_df = df_train.iloc[valid_index, :].reset_index()\n        y_true = valid_df['class'].values\n\n        print(\"===========================================\")\n        print(\"====== K Fold Validation step => %d/%d ======\" % ((fold),k_folds))\n        print(\"===========================================\")\n\n        batch_size = 32 * torch.cuda.device_count()\n\n        train_dataset = TrainDataset(train_df, mode='train', transforms=data_transforms)\n        valid_dataset = TrainDataset(valid_df, mode='valid', transforms=data_transforms)\n\n        train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n        valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)\n\n        if fold >= start_fold and fold <= end_fold:\n            torch.cuda.empty_cache()\n\n            # baseline이기 때문에 resnet50 사용합니다. 바꿔보세요!\n            #model = models.resnet50(pretrained=True)\n            model = pretrainedmodels.se_resnext50_32x4d(pretrained='imagenet')\n            model.last_linear = nn.Linear(2048, num_classes)\n            #model = models.resnet101(pretrained=True)\n            #model = models.resnext101_32x8d(pretrained=True)\n\n            if torch.cuda.device_count() > 1:\n                print(f'use multi gpu : {torch.cuda.device_count()}')\n                model = nn.DataParallel(model)\n            model.cuda()\n\n            criterion = FocalLoss()\n\n            train_kwargs = dict(\n                train_loader=train_loader,\n                valid_loader=valid_loader,\n                model=model,\n                criterion=criterion,\n            )\n\n            num_epochs = 75\n            result, lrs, score = train_model(num_epochs=num_epochs, accumulation_step=16, mixup_loss=False,\n                                             cv_checkpoint=True, fine_tune=False,\n                                             weight_file_name=f'seresnext50_fold_{fold}.pt',\n                                             y_true=y_true, **train_kwargs)\n            result_arr.append(result)\n            print(result)\n\n            # learning rate plot\n            plt.figure(figsize=(18,4))\n            plt.subplot(1,2,1)\n            plt.plot(lrs, 'b')\n            plt.xlabel('Epochs', fontsize=12, fontweight='bold')\n            plt.ylabel('Learning rate', fontsize=14, fontweight='bold')\n            plt.title(f'Fold {fold} Learning rate schedule', fontsize=15, fontweight='bold')\n\n            x = [x for x in range(0, num_epochs, 10)]\n            y = [0.01, 0.005, 0.000001]\n            ylabel = ['1e-2', '1e-4', '1e-6']\n            plt.xticks(x)\n            plt.yticks(y, ylabel)\n\n            plt.subplot(1,2,2)\n            plt.plot(score, 'r')\n            plt.xlabel('Epochs', fontsize=12, fontweight='bold')\n            plt.ylabel('Valid score', fontsize=14, fontweight='bold')\n            plt.title(f'Fold {fold} F1 Score', fontsize=15, fontweight='bold')\n\n            x = [x for x in range(0, num_epochs, 10)]\n\n            plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SE-ResNeXt50 32x4d Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"k_folds = 5\nnum_classes = 196\n\nbatch_size = 1\ntest_dataset = TestDataset(df_test, mode='test', transforms=data_transforms)\ntest_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\ntotal_num_models = k_folds\n\nmodel = pretrainedmodels.se_resnext50_32x4d(pretrained=None)\nmodel.last_linear = nn.Linear(2048, num_classes)\nmodel.cuda()\n\nall_prediction = np.zeros((len(test_dataset), num_classes))\n\nfor f in range(k_folds):\n    fold = f + 1\n    print(f'fold {fold} prediction starts')\n    \n    weight_path = f'../input/seresnext50-weight/seresnext50_fold_{fold}.pt'\n    model.load_state_dict(torch.load(weight_path))\n\n    model.eval()\n\n    prediction = np.zeros((len(test_dataset), num_classes)) # num_classes=196\n    with torch.no_grad():\n        for i, images in enumerate(test_loader):\n            images = images.cuda()\n\n            preds = model(images).detach()\n            preds = F.softmax(preds, dim=1) # convert output to probability\n            prediction[i * batch_size: (i+1) * batch_size] = preds.cpu().numpy()\n    all_prediction = all_prediction + prediction\n    \nall_prediction /= total_num_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"seresnext50_pred = pd.DataFrame(all_prediction)\nseresnext50_pred.to_csv('seresnext50_pred.csv', index=False)\n\nseresnext50_pred.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SE-ResNeXt50 32x4d TTA Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"k_folds = 5\nnum_classes = 196\n\nbatch_size = 1\ntta = 3\ntta_dataset = TestDataset(df_test, mode='tta', transforms=data_transforms)\ntta_loader = DataLoader(tta_dataset, batch_size=batch_size, shuffle=False)\ntotal_num_models = k_folds*tta\n\nmodel = pretrainedmodels.se_resnext50_32x4d(pretrained=None)\nmodel.last_linear = nn.Linear(2048, num_classes)\nmodel.cuda()\n\nall_prediction_tta = np.zeros((len(tta_dataset), num_classes))\n\nfor f in range(k_folds):\n    fold = f + 1\n    print(f'fold {fold} prediction starts')\n    \n    for _ in range(tta):\n        print(\"tta {}\".format(_+1))\n\n        weight_path = f'../input/seresnext50-weight/seresnext50_fold_{fold}.pt'\n        model.load_state_dict(torch.load(weight_path))\n\n        model.eval()\n        \n        prediction = np.zeros((len(tta_dataset), num_classes)) # num_classes=196\n        with torch.no_grad():\n            for i, images in enumerate(tta_loader):\n                images = images.cuda()\n\n                preds = model(images).detach()\n                preds = F.softmax(preds, dim=1) # convert output to probability\n                prediction[i * batch_size: (i+1) * batch_size] = preds.cpu().numpy()\n        all_prediction_tta = all_prediction_tta + prediction\n    \nall_prediction_tta /= total_num_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"seresnext50_pred_tta = pd.DataFrame(all_prediction_tta)\nseresnext50_pred_tta.to_csv('seresnext50_pred_tta.csv', index=False)\n\nseresnext50_pred_tta.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EfficientNet-B3 Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_name = 'efficientnet-b3'\nimage_size = EfficientNet.get_image_size(model_name)\nprint(image_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_size = (image_size, image_size)\n\ndata_transforms_2 = {\n    'train': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomHorizontalFlip(),\n        vision.transforms.RandomRotation(20),\n        CIFAR10Policy(),\n        vision.transforms.ToTensor(),\n        Cutout(n_holes=1, length=image_size//4),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n    'valid': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomResizedCrop(target_size, scale=(0.8,1.0)),\n        vision.transforms.RandomHorizontalFlip(),\n        vision.transforms.ToTensor(),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n    'test': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomResizedCrop(target_size, scale=(0.8,1.0)),\n        vision.transforms.ToTensor(),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n    'tta': vision.transforms.Compose([\n        vision.transforms.Resize(target_size),\n        vision.transforms.RandomHorizontalFlip(),\n        vision.transforms.RandomRotation(20),\n        CIFAR10Policy(),\n        vision.transforms.ToTensor(),\n        vision.transforms.Normalize(\n            [0.485, 0.456, 0.406], \n            [0.229, 0.224, 0.225])\n    ]),\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_efficientnet = False\n\nif train_efficientnet:\n    k_folds = 4\n    num_classes = 196\n    skf = StratifiedKFold(n_splits=k_folds, random_state=SEED)\n    start_fold = 1\n    end_fold = 1\n    result_arr = []\n\n    for i, (train_index, valid_index) in enumerate(skf.split(df_train['img_file'], df_train['class'])):\n        fold = i + 1\n        train_df = df_train.iloc[train_index, :].reset_index()\n        valid_df = df_train.iloc[valid_index, :].reset_index()\n        y_true = valid_df['class'].values\n\n        print(\"===========================================\")\n        print(\"====== K Fold Validation step => %d/%d ======\" % ((fold),k_folds))\n        print(\"===========================================\")\n\n        batch_size = 32 * torch.cuda.device_count()\n\n        train_dataset = TrainDataset(train_df, mode='train', transforms=data_transforms_2)\n        valid_dataset = TrainDataset(valid_df, mode='valid', transforms=data_transforms_2)\n\n        train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n        valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)\n\n        if fold >= start_fold and fold <= end_fold:\n            torch.cuda.empty_cache()\n\n            # baseline이기 때문에 resnet50 사용합니다. 바꿔보세요!\n            #model = models.resnet50(pretrained=True)\n            model = EfficientNet.from_pretrained(model_name, num_classes=num_classes)\n            #model = models.resnet101(pretrained=True)\n            #model = models.resnext101_32x8d(pretrained=True)\n            #model.fc = nn.Linear(2048, num_classes)\n\n            if torch.cuda.device_count() > 1:\n                print(f'use multi gpu : {torch.cuda.device_count()}')\n                model = nn.DataParallel(model)\n            model.cuda()\n\n            criterion = nn.CrossEntropyLoss()\n\n            train_kwargs = dict(\n                train_loader=train_loader,\n                valid_loader=valid_loader,\n                model=model,\n                criterion=criterion,\n            )\n\n            num_epochs = 75\n            result, lrs, score = train_model(num_epochs=num_epochs, accumulation_step=16, mixup_loss=False,\n                                             cv_checkpoint=True, fine_tune=False, weight_file_name=f'efficientnetb3_fold_{fold}.pt',\n                                             y_true=y_true, **train_kwargs)\n            result_arr.append(result)\n            print(result)\n\n            # learning rate plot\n            plt.figure(figsize=(18,4))\n            plt.subplot(1,2,1)\n            plt.plot(lrs, 'b')\n            plt.xlabel('Epochs', fontsize=12, fontweight='bold')\n            plt.ylabel('Learning rate', fontsize=14, fontweight='bold')\n            plt.title(f'Fold {fold} Learning rate schedule', fontsize=15, fontweight='bold')\n\n            x = [x for x in range(0, num_epochs, 10)]\n            y = [0.01, 0.005, 0.000001]\n            ylabel = ['1e-2', '1e-4', '1e-6']\n            plt.xticks(x)\n            plt.yticks(y, ylabel)\n\n            plt.subplot(1,2,2)\n            plt.plot(score, 'r')\n            plt.xlabel('Epochs', fontsize=12, fontweight='bold')\n            plt.ylabel('Valid score', fontsize=14, fontweight='bold')\n            plt.title(f'Fold {fold} F1 Score', fontsize=15, fontweight='bold')\n\n            x = [x for x in range(0, num_epochs, 10)]\n\n            plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EfficientNet-B3 Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"k_folds = 4\nnum_classes = 196\n\nbatch_size = 1\ntest_dataset = TestDataset(df_test, mode='test', transforms=data_transforms_2)\ntest_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\ntotal_num_models = k_folds\n\nmodel = EfficientNet.from_pretrained(model_name, num_classes=num_classes)\nmodel.cuda()\n\nall_prediction = np.zeros((len(test_dataset), num_classes))\n\nfor f in range(k_folds):\n    fold = f + 1\n    print(f'fold {fold} prediction starts')\n    \n    weight_path = f'../input/efficientnetb3-weight/efficientnetb3_fold_{fold}.pt'\n    model.load_state_dict(torch.load(weight_path))\n\n    model.eval()\n\n    prediction = np.zeros((len(test_dataset), num_classes)) # num_classes=196\n    with torch.no_grad():\n        for i, images in enumerate(test_loader):\n            images = images.cuda()\n\n            preds = model(images).detach()\n            preds = F.softmax(preds, dim=1) # convert output to probability\n            prediction[i * batch_size: (i+1) * batch_size] = preds.cpu().numpy()\n    all_prediction = all_prediction + prediction\n    \nall_prediction /= total_num_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"efficientnetb3_pred = pd.DataFrame(all_prediction)\nefficientnetb3_pred.to_csv('efficientnetb3_pred.csv', index=False)\n\nefficientnetb3_pred.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EfficientNet-B3 TTA Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"k_folds = 4\nnum_classes = 196\n\nbatch_size = 1\ntta = 3\ntta_dataset = TestDataset(df_test, mode='tta', transforms=data_transforms_2)\ntta_loader = DataLoader(tta_dataset, batch_size=batch_size, shuffle=False)\ntotal_num_models = k_folds*tta\n\nmodel = EfficientNet.from_pretrained(model_name, num_classes=num_classes)\nmodel.cuda()\n\nall_prediction_tta = np.zeros((len(tta_dataset), num_classes))\n\nfor f in range(k_folds):\n    fold = f + 1\n    print(f'fold {fold} prediction starts')\n    \n    for _ in range(tta):\n        print(\"tta {}\".format(_+1))\n\n        weight_path = f'../input/efficientnetb3-weight/efficientnetb3_fold_{fold}.pt'\n        model.load_state_dict(torch.load(weight_path))\n\n        model.eval()\n        \n        prediction = np.zeros((len(tta_dataset), num_classes)) # num_classes=196\n        with torch.no_grad():\n            for i, images in enumerate(tta_loader):\n                images = images.cuda()\n\n                preds = model(images).detach()\n                preds = F.softmax(preds, dim=1) # convert output to probability\n                prediction[i * batch_size: (i+1) * batch_size] = preds.cpu().numpy()\n        all_prediction_tta = all_prediction_tta + prediction\n    \nall_prediction_tta /= total_num_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"efficientnetb3_pred_tta = pd.DataFrame(all_prediction_tta)\nefficientnetb3_pred_tta.to_csv('efficientnetb3_pred_tta.csv', index=False)\n\nefficientnetb3_pred_tta.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble"},{"metadata":{"trusted":true},"cell_type":"code","source":"seresnext50_ensemble = 0.25*seresnext50_pred.values + 0.75*seresnext50_pred_tta.values\nefficientnetb3_ensemble = 0.25*efficientnetb3_pred.values + 0.75*efficientnetb3_pred_tta.values\nfinal_ensemble = 0.3*efficientnetb3_ensemble + 0.7*seresnext50_ensemble\n\nresult_ensemble = np.argmax(final_ensemble, axis=1)\nresult_ensemble = result_ensemble + 1\n\nsubmission_ensemble = pd.read_csv('../input/2019-3rd-ml-month-with-kakr/sample_submission.csv')\nsubmission_ensemble[\"class\"] = result_ensemble\nsubmission_ensemble.to_csv(\"submission_ensemble.csv\", index=False)\nsubmission_ensemble.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}