{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EfficientNet: small step for an optimizer gient giant leap datascientists\n### **Data mining** | **Object detection** | **PyTorch** | **EfficientNet initializing** | **EfficientNet training** ","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"}},{"cell_type":"markdown","source":"EfficientNet paper: [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\n](https://arxiv.org/pdf/1905.11946.pdf)\n\n![graphics](https://i.ibb.co/k8TV0MM/image3.png)","metadata":{}},{"cell_type":"markdown","source":"https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html\nhttps://colab.research.google.com/drive/1ByRi9d6_Yzu0nrEKArmLMLuMaZjYfygO#scrollTo=mub8GJMBVluA\nhttps://www.kaggle.com/shonenkov/inference-efficientdet\nhttps://www.kaggle.com/shonenkov/training-efficientdet\n\n**Huge thanks to [Alex Shonenkov](https://www.kaggle.com/shonenkov) for making so well-explaining notebooks**","metadata":{}},{"cell_type":"markdown","source":"### Hello!\nIn this notebook we're going to work through one of the most popular architectures for multiple object detection: EfficientNet. One must have read lots of notebooks, papers about this quite new and modern DL thing, but there are really few notebooks about **the whole** process of creating, training and using yolov3 in your projects. Due to the very complex architecture, EfficientNet (lots of convolution, maxpooling layers and huge output vectors) is very difficult even for modern hardware to be trained. One needs GPU accelerators, or even TPU so as to make training faster. But as far as you know, **transfer** learning is a widely-used technique, so here we are free to use treansfer learning too.","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"}},{"cell_type":"markdown","source":"Lets divide our complex task into some subtasks, which must be easier to be solved:\n1. *Data mining*. Here we need to find data(images) for training an EfficientNet of yours, but the data must be 'preprocessed'. So before using the data, you have to make sure, that all the images have bounding boxes and an appropriate class for all the boxes. [LabelImg](https://github.com/tzutalin/labelImg) is a useful open source project for it.\n![](https://lh3.googleusercontent.com/proxy/OVFxl--wnKLg3QTcpb1wvH9twmERGcrQ2I7DZ3voTN3nKmOBvvzBSQWWVanPPnWJETahZV2LjpaevdidRJ65eWo)\n<br>\n2. *Builiding*. Having a nice dataset for object detection, you have to build an EfficientNet.\n3. *Training*. Fit your specific dataset into the model.\n4. (OPTIONAL)*User friendly*. Build a simple in->out functionality between user and your EfficientNet trained model, so that it'd be easy for you and someone else to just simply give an image to your model and get a bounding box(es) + class names.","metadata":{}},{"cell_type":"markdown","source":"## Data mining","metadata":{}},{"cell_type":"markdown","source":"I already have an idea to build an application for **OCR** [optical character recogniton](https://en.wikipedia.org/wiki/Optical_character_recognition) problem with my own huge dataset. You may have taken another problem. So if anyone asks for deeper explanations, I'll be pleased to share with you the whole process.","metadata":{}},{"cell_type":"markdown","source":"*** Import appropriate libraries for data**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nfrom random import sample\nfrom colorama import Fore, Style","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:25.440854Z","iopub.execute_input":"2023-05-27T16:26:25.441347Z","iopub.status.idle":"2023-05-27T16:26:25.446511Z","shell.execute_reply.started":"2023-05-27T16:26:25.441304Z","shell.execute_reply":"2023-05-27T16:26:25.445622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_FOLDER = '../input/charaters-for-detection-classification/csvdata/'\n\nmarking = pd.read_csv(os.path.join(TRAIN_FOLDER, 'train.csv'))\n#DEBUG#\nmarking['source'] = ['' for i in range(len(marking['image_id']))]\n#DEBUG#\nTRAIN_IMGS_FOLDER = os.path.join(TRAIN_FOLDER, 'images')\nprint(f'NUMBER OF TRAINING EXAMPLES{Fore.BLUE}', len(os.listdir(TRAIN_IMGS_FOLDER)),f'{Style.RESET_ALL}')","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:25.448196Z","iopub.execute_input":"2023-05-27T16:26:25.448702Z","iopub.status.idle":"2023-05-27T16:26:25.804563Z","shell.execute_reply.started":"2023-05-27T16:26:25.448665Z","shell.execute_reply":"2023-05-27T16:26:25.803811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"######################### BBOXS ERROR ###############################\nimport numpy as np\nbboxs = np.stack(marking['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=' ')))\nbad_idxs = []\nprint('before:', len(marking))\nprint(marking.shape)\nfor i, bbox in enumerate(bboxs):\n    if bbox[0] + bbox[2] > 255 or bbox[1] + bbox[3] > 255:\n        bad_idxs.append(i)\nmarking = marking.drop(bad_idxs)\nprint('after:', len(marking))\nprint(marking.shape)\n#####################################################################","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:25.806037Z","iopub.execute_input":"2023-05-27T16:26:25.806554Z","iopub.status.idle":"2023-05-27T16:26:26.176321Z","shell.execute_reply.started":"2023-05-27T16:26:25.806510Z","shell.execute_reply":"2023-05-27T16:26:26.175289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For my problem, I made a program for generating background and a random character with different colours so as to avoid anomalies in dataset. Let me show you what it looks like...","metadata":{}},{"cell_type":"code","source":"num_examples = 6\nexample_images = [os.path.join(TRAIN_IMGS_FOLDER, file) for file in sample(os.listdir(TRAIN_IMGS_FOLDER), num_examples)]","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:26.179145Z","iopub.execute_input":"2023-05-27T16:26:26.179517Z","iopub.status.idle":"2023-05-27T16:26:26.190014Z","shell.execute_reply.started":"2023-05-27T16:26:26.179478Z","shell.execute_reply":"2023-05-27T16:26:26.189231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_for_shape = cv2.imread(example_images[0])\nIMG_WIDTH, IMG_HEIGHT = img_for_shape.shape[0],img_for_shape.shape[1]\nprint('image width=', IMG_WIDTH,'; image height=', IMG_HEIGHT)","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:26.193238Z","iopub.execute_input":"2023-05-27T16:26:26.193677Z","iopub.status.idle":"2023-05-27T16:26:26.217332Z","shell.execute_reply.started":"2023-05-27T16:26:26.193638Z","shell.execute_reply":"2023-05-27T16:26:26.216534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raws, columns = 2, 3\nfig, axis = plt.subplots(raws, columns, figsize=(12, 12))\nfor i, img_path in enumerate(example_images):\n    img_data = cv2.imread(img_path)\n    axis[i % raws, i % columns].imshow(img_data)","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:26.220239Z","iopub.execute_input":"2023-05-27T16:26:26.220502Z","iopub.status.idle":"2023-05-27T16:26:27.086740Z","shell.execute_reply.started":"2023-05-27T16:26:26.220476Z","shell.execute_reply":"2023-05-27T16:26:27.085809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building","metadata":{}},{"cell_type":"markdown","source":"Lets build our EfficientNet model. First we need to download all dependencies for it.","metadata":{}},{"cell_type":"code","source":"!pip install --no-deps '../input/timm-package/timm-0.1.26-py3-none-any.whl' > /dev/null\n!pip install --no-deps '../input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl' > /dev/null","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:27.087950Z","iopub.execute_input":"2023-05-27T16:26:27.088328Z","iopub.status.idle":"2023-05-27T16:26:33.700806Z","shell.execute_reply.started":"2023-05-27T16:26:27.088290Z","shell.execute_reply":"2023-05-27T16:26:33.699784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.insert(0, \"../input/timm-efficientdet-pytorch\")\nsys.path.insert(0, \"../input/omegaconf\")\n\nimport torch\nfrom datetime import datetime\nimport time\nimport random\nimport numpy as np\nimport albumentations as A\nimport matplotlib.pyplot as plt\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nfrom glob import glob\n\nSEED = 42\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n\nseed_everything(SEED)\n\nprint(f'{Fore.GREEN}everything was setup succsefully{Style.RESET_ALL}')","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:33.702546Z","iopub.execute_input":"2023-05-27T16:26:33.702882Z","iopub.status.idle":"2023-05-27T16:26:36.548406Z","shell.execute_reply.started":"2023-05-27T16:26:33.702846Z","shell.execute_reply":"2023-05-27T16:26:36.547450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bboxs = np.stack(marking['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=' ')))\nfor i, column in enumerate(['x', 'y', 'w', 'h']):\n    marking[column] = bboxs[:,i]\nmarking.drop(columns=['bbox'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:36.549816Z","iopub.execute_input":"2023-05-27T16:26:36.550367Z","iopub.status.idle":"2023-05-27T16:26:36.785590Z","shell.execute_reply.started":"2023-05-27T16:26:36.550326Z","shell.execute_reply":"2023-05-27T16:26:36.784763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\ndf_folds = marking[['image_id']].copy()\ndf_folds.loc[:, 'bbox_count'] = 1\ndf_folds = df_folds.groupby('image_id').count()\ndf_folds.loc[:, 'source'] = marking[['image_id', 'source']].groupby('image_id').min()['source']\ndf_folds.loc[:, 'stratify_group'] = np.char.add(\n    df_folds['source'].values.astype(str),\n    df_folds['bbox_count'].apply(lambda x: f'_{x // 15}').values.astype(str)\n)\ndf_folds.loc[:, 'fold'] = 0\n\nfor fold_number, (train_index, val_index) in enumerate(skf.split(X=df_folds.index, y=df_folds['stratify_group'])):\n    df_folds.loc[df_folds.iloc[val_index].index, 'fold'] = fold_number","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:36.789193Z","iopub.execute_input":"2023-05-27T16:26:36.789482Z","iopub.status.idle":"2023-05-27T16:26:38.195748Z","shell.execute_reply.started":"2023-05-27T16:26:36.789453Z","shell.execute_reply":"2023-05-27T16:26:38.194794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_transforms():\n    return A.Compose(\n        [A.OneOf([\n                    A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n                                         val_shift_limit=0.2, p=0.9),\n                    A.RandomBrightnessContrast(brightness_limit=0.2, \n                                               contrast_limit=0.2, p=0.9),\n                    A.RandomGamma(p=0.9),\n            ],p=0.25),\n            A.OneOf([\n                A.IAASharpen(alpha=(0.1, 0.3), p=0.5),\n                A.CLAHE(p=0.8),\n                #A.GaussNoise(var_limit=(10.0, 50.0), p=0.5),\n                #A.GaussianBlur(blur_limit=3, p=0.5),\n                A.ISONoise(color_shift=(0.01, 0.05), intensity=(0.1, 0.5), p=0.3),\n            ], p=0.0),\n            ToTensorV2(p=1.0),\n        ], \n        p=1.0, \n        bbox_params=A.BboxParams(\n            format='pascal_voc',\n            min_area=0, \n            min_visibility=0,\n            label_fields=['labels']\n        )\n    )\n\n\ndef get_valid_transforms():\n    return A.Compose(\n        [\n            A.Resize(height=256, width=256, p=1.0),\n            ToTensorV2(p=1.0),\n        ], \n        p=1.0, \n        bbox_params=A.BboxParams(\n            format='pascal_voc',\n            min_area=0, \n            min_visibility=0,\n            label_fields=['labels']\n        )\n    )","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.197188Z","iopub.execute_input":"2023-05-27T16:26:38.197566Z","iopub.status.idle":"2023-05-27T16:26:38.210767Z","shell.execute_reply.started":"2023-05-27T16:26:38.197524Z","shell.execute_reply":"2023-05-27T16:26:38.209424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_ROOT_PATH = TRAIN_IMGS_FOLDER\nIMAGENET_DEFAULT_MEAN = [x * 255 for x in (0.485, 0.456, 0.406)]\nIMAGENET_DEFAULT_STD = [x * 255 for x in (0.229, 0.224, 0.225)]\n\nclass DatasetRetriever(Dataset):\n\n    def __init__(self, marking, image_ids, transforms=None, test=False):\n        super().__init__()\n\n        self.image_ids = image_ids\n        self.marking = marking\n        self.transforms = transforms\n        self.test = test\n\n    def __getitem__(self, index: int):\n        image_id = self.image_ids[index]\n        \n        r = random.random()\n        image, boxes = self.load_image_and_boxes(index)\n        # there is only one class\n        labels = torch.ones((boxes.shape[0],), dtype=torch.int64)\n        \n        target = {}\n        target['boxes'] = boxes\n        target['labels'] = labels\n        target['image_id'] = torch.tensor([index])\n        image = image.astype(np.uint8)\n\n        if self.transforms:\n            for i in range(10):\n                sample = self.transforms(**{\n                    'image': image,\n                    'bboxes': target['boxes'],\n                    'labels': labels\n                })\n                if len(sample['bboxes']) > 0:\n                    image = sample['image']\n                    target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n                    target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]]  #yxyx: be warning\n                    break\n        image = image.float() / 255.0\n\n        return image, target, image_id\n\n    def __len__(self) -> int:\n        return self.image_ids.shape[0]\n\n    def load_image_and_boxes(self, index):\n        image_id = self.image_ids[index]\n        image = cv2.imread(f'{TRAIN_ROOT_PATH}/{image_id}.png', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.uint8)\n        #image = cv2.addWeighted(image, 4, cv2.GaussianBlur(image, (0,0) , 10), -4 ,128)\n        #image = (image - IMAGENET_DEFAULT_MEAN) / IMAGENET_DEFAULT_STD\n        records = self.marking[self.marking['image_id'] == image_id]\n        boxes = records[['x', 'y', 'w', 'h']].values\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        return image, boxes\n\n    def load_mixup_iamge_and_boxes(self, index):\n        image, boxes = self.load_image_and_boxes(index)\n        r_image, r_boxes = self.load_image_and_boxes(random.randint(0, self.image_ids.shape[0] - 1))\n        mixup_image = (image + r_image) / 2\n        mixup_boxes = np.concatenate([boxes, r_boxes], axis=0)\n        return mixup_image, mixup_boxes\n        \n    \n    def load_cutmix_image_and_boxes(self, index, imsize=256):\n        \"\"\" \n        This implementation of cutmix author:  https://www.kaggle.com/nvnnghia \n        Refactoring and adaptation: https://www.kaggle.com/shonenkov\n        \"\"\"\n        w, h = imsize, imsize\n        s = imsize // 2\n    \n        xc, yc = [int(random.uniform(imsize * 0.25, imsize * 0.75)) for _ in range(2)]  # center x, y\n        indexes = [index] + [random.randint(0, self.image_ids.shape[0] - 1) for _ in range(3)]\n\n        result_image = np.full((imsize, imsize, 3), 1, dtype=np.float32)\n        result_boxes = []\n\n        for i, index in enumerate(indexes):\n            image, boxes = self.load_image_and_boxes(index)\n            if i == 0:\n                x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc  # xmin, ymin, xmax, ymax (large image)\n                x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h  # xmin, ymin, xmax, ymax (small image)\n            elif i == 1:  # top right\n                x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc\n                x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h\n            elif i == 2:  # bottom left\n                x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)\n                x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, max(xc, w), min(y2a - y1a, h)\n            elif i == 3:  # bottom right\n                x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)\n                x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)\n            result_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]\n            padw = x1a - x1b\n            padh = y1a - y1b\n\n            boxes[:, 0] += padw\n            boxes[:, 1] += padh\n            boxes[:, 2] += padw\n            boxes[:, 3] += padh\n\n            result_boxes.append(boxes)\n\n        result_boxes = np.concatenate(result_boxes, 0)\n        np.clip(result_boxes[:, 0:], 0, 2 * s, out=result_boxes[:, 0:])\n        result_boxes = result_boxes.astype(np.int32)\n        result_boxes = result_boxes[np.where((result_boxes[:,2]-result_boxes[:,0])*(result_boxes[:,3]-result_boxes[:,1]) > 0)]\n        return result_image, result_boxes","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.212310Z","iopub.execute_input":"2023-05-27T16:26:38.212899Z","iopub.status.idle":"2023-05-27T16:26:38.256920Z","shell.execute_reply.started":"2023-05-27T16:26:38.212860Z","shell.execute_reply":"2023-05-27T16:26:38.256130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fold_number = 0\n\ntrain_dataset = DatasetRetriever(\n    image_ids=df_folds[df_folds['fold'] != fold_number].index.values,\n    marking=marking,\n    transforms=get_train_transforms(),\n    test=False,\n)\n\nvalidation_dataset = DatasetRetriever(\n    image_ids=df_folds[df_folds['fold'] == fold_number].index.values,\n    marking=marking,\n    transforms=get_valid_transforms(),\n    test=True,\n)","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.258361Z","iopub.execute_input":"2023-05-27T16:26:38.258728Z","iopub.status.idle":"2023-05-27T16:26:38.289235Z","shell.execute_reply.started":"2023-05-27T16:26:38.258690Z","shell.execute_reply":"2023-05-27T16:26:38.288544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.292360Z","iopub.execute_input":"2023-05-27T16:26:38.292626Z","iopub.status.idle":"2023-05-27T16:26:38.301408Z","shell.execute_reply.started":"2023-05-27T16:26:38.292599Z","shell.execute_reply":"2023-05-27T16:26:38.300592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\n\nwarnings.filterwarnings(\"ignore\")\n\nclass Fitter:\n    \n    def __init__(self, model, device, config):\n        self.config = config\n        self.epoch = 0\n\n        self.base_dir = f'./{config.folder}'\n        if not os.path.exists(self.base_dir):\n            os.makedirs(self.base_dir)\n        \n        self.log_path = f'{self.base_dir}/log.txt'\n        self.best_summary_loss = 10**5\n\n        self.model = model\n        self.device = device\n\n        param_optimizer = list(self.model.named_parameters())\n        no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n        optimizer_grouped_parameters = [\n            {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},\n            {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n        ] \n\n        self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=config.lr)\n        self.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\n        self.log(f'Fitter prepared. Device is {self.device}')\n\n    def fit(self, train_loader, validation_loader):\n        for e in range(self.config.n_epochs):\n            if self.config.verbose:\n                lr = self.optimizer.param_groups[0]['lr']\n                timestamp = datetime.utcnow().isoformat()\n                self.log(f'\\n{timestamp}\\nLR: {lr}')\n\n            t = time.time()\n            summary_loss = self.train_one_epoch(train_loader)\n\n            self.log(f'[RESULT]: Train. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n            self.save(f'{self.base_dir}/last-checkpoint.bin')\n\n            t = time.time()\n            summary_loss = self.validation(validation_loader)\n\n            self.log(f'[RESULT]: Val. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n            if summary_loss.avg < self.best_summary_loss:\n                self.best_summary_loss = summary_loss.avg\n                self.model.eval()\n                self.save(f'{self.base_dir}/best-checkpoint-{str(self.epoch).zfill(3)}epoch.bin')\n                for path in sorted(glob(f'{self.base_dir}/best-checkpoint-*epoch.bin'))[:-3]:\n                    os.remove(path)\n\n            if self.config.validation_scheduler:\n                self.scheduler.step(metrics=summary_loss.avg)\n\n            self.epoch += 1\n\n    def validation(self, val_loader):\n        self.model.eval()\n        summary_loss = AverageMeter()\n        t = time.time()\n        for step, (images, targets, image_ids) in enumerate(val_loader):\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    print(\n                        f'Val Step {step}/{len(val_loader)}, ' + \\\n                        f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n                        f'time: {(time.time() - t):.5f}', end='\\r'\n                    )\n            with torch.no_grad():\n                images = torch.stack(images)\n                batch_size = images.shape[0]\n                images = images.to(self.device).float()\n                boxes = [target['boxes'].to(self.device).float() for target in targets]\n                labels = [target['labels'].to(self.device).float() for target in targets]\n\n                loss, _, _ = self.model(images, boxes, labels)\n                summary_loss.update(loss.detach().item(), batch_size)\n\n        return summary_loss\n        \n    def train_one_epoch(self, train_loader):\n        self.model.train()\n        summary_loss = AverageMeter()\n        t = time.time()\n        for step, (images, targets, image_ids) in enumerate(train_loader):\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    print(\n                        f'Train Step {step}/{len(train_loader)}, ' + \\\n                        f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n                        f'time: {(time.time() - t):.5f}', end='\\r'\n                    )\n            \n            images = torch.stack(images)\n            images = images.to(self.device).float()\n            batch_size = images.shape[0]\n            boxes = [target['boxes'].to(self.device).float() for target in targets]\n            labels = [target['labels'].to(self.device).float() for target in targets]\n            \n            loss, _, _ = self.model(images, boxes, labels)\n            loss = loss / float(self.config.grad_accumulation_steps)\n            loss.backward()\n            summary_loss.update(loss.detach().item(), batch_size)\n            \n            # Gradient accumulation\n            if (step + 1) % self.config.grad_accumulation_steps == 0:\n                self.optimizer.step()\n                self.optimizer.zero_grad()\n                if self.config.step_scheduler:\n                    self.scheduler.step()\n\n        return summary_loss\n    \n    def save(self, path):\n        self.model.eval()\n        torch.save({\n            'model_state_dict': self.model.model.state_dict(),\n            'optimizer_state_dict': self.optimizer.state_dict(),\n            'scheduler_state_dict': self.scheduler.state_dict(),\n            'best_summary_loss': self.best_summary_loss,\n            'epoch': self.epoch,\n        }, path)\n\n    def load(self, path):\n        checkpoint = torch.load(path)\n        self.model.model.load_state_dict(checkpoint['model_state_dict'])\n        self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n        self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n        self.best_summary_loss = checkpoint['best_summary_loss']\n        self.epoch = checkpoint['epoch'] + 1\n        \n    def log(self, message):\n        if self.config.verbose:\n            print(message)\n        with open(self.log_path, 'a+') as logger:\n            logger.write(f'{message}\\n')","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.302978Z","iopub.execute_input":"2023-05-27T16:26:38.303633Z","iopub.status.idle":"2023-05-27T16:26:38.346553Z","shell.execute_reply.started":"2023-05-27T16:26:38.303593Z","shell.execute_reply":"2023-05-27T16:26:38.345642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TrainGlobalConfig:\n    num_workers = 2\n    batch_size = 1\n    n_epochs = 8 # n_epochs = 40\n    lr = 0.001 # lr = 0.0002 # 0.001 recommended for Adam\n    grad_accumulation_steps = 64\n\n    folder = 'effdet6-cutmix-gradient-fold0'\n\n    # -------------------\n    verbose = True\n    verbose_step = 1\n    # -------------------\n\n    # --------------------\n    step_scheduler = False  # do scheduler.step after optimizer.step\n    validation_scheduler = True  # do scheduler.step after validation stage loss\n\n#     SchedulerClass = torch.optim.lr_scheduler.OneCycleLR\n#     scheduler_params = dict(\n#         max_lr=0.001,\n#         epochs=n_epochs,\n#         steps_per_epoch=int(len(train_dataset) / batch_size),\n#         pct_start=0.1,\n#         anneal_strategy='cos', \n#         final_div_factor=10**5\n#     )\n    \n    SchedulerClass = torch.optim.lr_scheduler.ReduceLROnPlateau\n    scheduler_params = dict(\n        mode='min',\n        factor=0.5,\n        patience=2,\n        verbose=False, \n        threshold=0.0001,\n        threshold_mode='abs',\n        cooldown=0, \n        min_lr=1e-8,\n        eps=1e-08\n    )\n    # --------------------","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.348941Z","iopub.execute_input":"2023-05-27T16:26:38.349483Z","iopub.status.idle":"2023-05-27T16:26:38.361404Z","shell.execute_reply.started":"2023-05-27T16:26:38.349444Z","shell.execute_reply":"2023-05-27T16:26:38.360687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def collate_fn(batch):\n    return tuple(zip(*batch))\n\ndef run_training(path=''):\n    device = torch.device('cuda:0')\n    net.to(device)\n\n    train_loader = torch.utils.data.DataLoader(\n        train_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=RandomSampler(train_dataset),\n        pin_memory=False,\n        drop_last=True,\n        num_workers=TrainGlobalConfig.num_workers,\n        collate_fn=collate_fn,\n    )\n    val_loader = torch.utils.data.DataLoader(\n        validation_dataset, \n        batch_size=TrainGlobalConfig.batch_size,\n        num_workers=TrainGlobalConfig.num_workers,\n        shuffle=False,\n        sampler=SequentialSampler(validation_dataset),\n        pin_memory=False,\n        collate_fn=collate_fn,\n    )\n\n    fitter = Fitter(model=net, device=device, config=TrainGlobalConfig)\n    if path:\n        fitter.load(path=path)\n    fitter.fit(train_loader, val_loader)","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.362980Z","iopub.execute_input":"2023-05-27T16:26:38.363385Z","iopub.status.idle":"2023-05-27T16:26:38.376164Z","shell.execute_reply.started":"2023-05-27T16:26:38.363347Z","shell.execute_reply":"2023-05-27T16:26:38.375507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\nimport gc \n\ndef get_net():\n    config = get_efficientdet_config('tf_efficientdet_d5')\n    net = EfficientDet(config, pretrained_backbone=False)\n    config.num_classes = 1\n    config.image_size = 256\n    checkpoint = torch.load('../input/efficientdet/efficientdet_d5-ef44aea8.pth')\n    net.load_state_dict(checkpoint)\n    net.class_net = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=.001, momentum=.01))\n    \n    #optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n    del checkpoint\n    gc.collect()\n    \n    return DetBenchTrain(net, config)\n\nnet = get_net()","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:38.377925Z","iopub.execute_input":"2023-05-27T16:26:38.378360Z","iopub.status.idle":"2023-05-27T16:26:42.824318Z","shell.execute_reply.started":"2023-05-27T16:26:38.378315Z","shell.execute_reply":"2023-05-27T16:26:42.823310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run_training()","metadata":{"execution":{"iopub.status.busy":"2023-05-27T16:26:42.827268Z","iopub.execute_input":"2023-05-27T16:26:42.827630Z"},"trusted":true},"execution_count":null,"outputs":[]}]}