{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Really good training pipeline for pytorch EfficientDet \n\nHi everyone!\n\nMy name is Alex Shonenkov, I am DL/NLP/CV/TS research engineer. Especially I am in Love with NLP & DL.\n\nRecently I have created kernel for this competition about Weighted Boxes Fusion:\n- [WBF approach for ensemble](https://www.kaggle.com/shonenkov/wbf-approach-for-ensemble)\n\n\nI hope it is useful for you, my friends! If you didn't read this kernel, don't forget to do it! :)\n\n\nToday I would like to share really good training pipeline for this competition using SOTA [EfficientDet: Scalable and Efficient Object Detection](https://arxiv.org/pdf/1911.09070.pdf)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Main Idea\n \nI read [all public kernels about EfficientDet in kaggle community](https://www.kaggle.com/search?q=efficientdet+in%3Anotebooks) and understand that kaggle don't have really good working public kernels with good score. Why? You can see below picture about COCO AP for different architectures, I think everyone should be able to use such a strong tools EfficientDet for own research, lets do it!\n\n \n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Dependencies and imports","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install --no-deps '../input/timm-package/timm-0.1.26-py3-none-any.whl' > /dev/null\n!pip install --no-deps '../input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl' > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import sys\nsys.path.insert(0, \"../input/timm-efficientdet-pytorch\")\nsys.path.insert(0, \"../input/omegaconf\")\n\nimport torch\nimport os\nfrom datetime import datetime\nimport time\nimport random\nimport cv2\nimport pandas as pd\nimport numpy as np\nimport albumentations as A\nimport matplotlib.pyplot as plt\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nfrom glob import glob\n\nSEED = 42\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"marking = pd.read_csv('../input/global-wheat-detection/train.csv')\n\nbboxs = np.stack(marking['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=',')))\nfor i, column in enumerate(['x', 'y', 'w', 'h']):\n marking[column] = bboxs[:,i]\nmarking.drop(columns=['bbox'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"About data splitting you can read [here](https://www.kaggle.com/shonenkov/wbf-approach-for-ensemble):","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\ndf_folds = marking[['image_id']].copy()\ndf_folds.loc[:, 'bbox_count'] = 1\ndf_folds = df_folds.groupby('image_id').count()\ndf_folds.loc[:, 'source'] = marking[['image_id', 'source']].groupby('image_id').min()['source']\ndf_folds.loc[:, 'stratify_group'] = np.char.add(\n df_folds['source'].values.astype(str),\n df_folds['bbox_count'].apply(lambda x: f'_{x // 15}').values.astype(str)\n)\ndf_folds.loc[:, 'fold'] = 0\n\nfor fold_number, (train_index, val_index) in enumerate(skf.split(X=df_folds.index, y=df_folds['stratify_group'])):\n df_folds.loc[df_folds.iloc[val_index].index, 'fold'] = fold_number","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Albumentations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_transforms():\n return A.Compose(\n [\n A.RandomSizedCrop(min_max_height=(800, 800), height=1024, width=1024, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.ToGray(p=0.01),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.Resize(height=512, width=512, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\n\ndef get_valid_transforms():\n return A.Compose(\n [\n A.Resize(height=512, width=512, p=1.0),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_ROOT_PATH = '../input/global-wheat-detection/train'\n\nclass DatasetRetriever(Dataset):\n\n def __init__(self, marking, image_ids, transforms=None, test=False):\n super().__init__()\n\n self.image_ids = image_ids\n self.marking = marking\n self.transforms = transforms\n self.test = test\n\n def __getitem__(self, index: int):\n image_id = self.image_ids[index]\n \n if self.test or random.random() > 0.5:\n image, boxes = self.load_image_and_boxes(index)\n else:\n image, boxes = self.load_cutmix_image_and_boxes(index)\n\n # there is only one class\n labels = torch.ones((boxes.shape[0],), dtype=torch.int64)\n \n target = {}\n target['boxes'] = boxes\n target['labels'] = labels\n target['image_id'] = torch.tensor([index])\n\n if self.transforms:\n for i in range(10):\n sample = self.transforms(**{\n 'image': image,\n 'bboxes': target['boxes'],\n 'labels': labels\n })\n if len(sample['bboxes']) > 0:\n image = sample['image']\n target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]] #yxyx: be warning\n break\n\n return image, target, image_id\n\n def __len__(self) -> int:\n return self.image_ids.shape[0]\n\n def load_image_and_boxes(self, index):\n image_id = self.image_ids[index]\n image = cv2.imread(f'{TRAIN_ROOT_PATH}/{image_id}.jpg', cv2.IMREAD_COLOR)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n records = self.marking[self.marking['image_id'] == image_id]\n boxes = records[['x', 'y', 'w', 'h']].values\n boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n return image, boxes\n\n def load_cutmix_image_and_boxes(self, index, imsize=1024):\n \"\"\" \n This implementation of cutmix author: https://www.kaggle.com/nvnnghia \n Refactoring and adaptation: https://www.kaggle.com/shonenkov\n \"\"\"\n w, h = imsize, imsize\n s = imsize // 2\n \n xc, yc = [int(random.uniform(imsize * 0.25, imsize * 0.75)) for _ in range(2)] # center x, y\n indexes = [index] + [random.randint(0, self.image_ids.shape[0] - 1) for _ in range(3)]\n\n result_image = np.full((imsize, imsize, 3), 1, dtype=np.float32)\n result_boxes = []\n\n for i, index in enumerate(indexes):\n image, boxes = self.load_image_and_boxes(index)\n if i == 0:\n x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc # xmin, ymin, xmax, ymax (large image)\n x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h # xmin, ymin, xmax, ymax (small image)\n elif i == 1: # top right\n x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc\n x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h\n elif i == 2: # bottom left\n x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)\n x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, max(xc, w), min(y2a - y1a, h)\n elif i == 3: # bottom right\n x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)\n x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)\n result_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]\n padw = x1a - x1b\n padh = y1a - y1b\n\n boxes[:, 0] += padw\n boxes[:, 1] += padh\n boxes[:, 2] += padw\n boxes[:, 3] += padh\n\n result_boxes.append(boxes)\n\n result_boxes = np.concatenate(result_boxes, 0)\n np.clip(result_boxes[:, 0:], 0, 2 * s, out=result_boxes[:, 0:])\n result_boxes = result_boxes.astype(np.int32)\n result_boxes = result_boxes[np.where((result_boxes[:,2]-result_boxes[:,0])*(result_boxes[:,3]-result_boxes[:,1]) > 0)]\n return result_image, result_boxes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_number = 0\n\ntrain_dataset = DatasetRetriever(\n image_ids=df_folds[df_folds['fold'] != fold_number].index.values,\n marking=marking,\n transforms=get_train_transforms(),\n test=False,\n)\n\nvalidation_dataset = DatasetRetriever(\n image_ids=df_folds[df_folds['fold'] == fold_number].index.values,\n marking=marking,\n transforms=get_valid_transforms(),\n test=True,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image, target, image_id = train_dataset[1]\nboxes = target['boxes'].cpu().numpy().astype(np.int32)\n\nnumpy_image = image.permute(1,2,0).cpu().numpy()\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nfor box in boxes:\n cv2.rectangle(numpy_image, (box[1], box[0]), (box[3], box[2]), (0, 1, 0), 2)\n \nax.set_axis_off()\nax.imshow(numpy_image);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fitter","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n\n def update(self, val, n=1):\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\n\nwarnings.filterwarnings(\"ignore\")\n\nclass Fitter:\n \n def __init__(self, model, device, config):\n self.config = config\n self.epoch = 0\n\n self.base_dir = f'./{config.folder}'\n if not os.path.exists(self.base_dir):\n os.makedirs(self.base_dir)\n \n self.log_path = f'{self.base_dir}/log.txt'\n self.best_summary_loss = 10**5\n\n self.model = model\n self.device = device\n\n param_optimizer = list(self.model.named_parameters())\n no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n optimizer_grouped_parameters = [\n {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},\n {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n ] \n\n self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=config.lr)\n self.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\n self.log(f'Fitter prepared. Device is {self.device}')\n\n def fit(self, train_loader, validation_loader):\n for epoch in range(self.config.n_epochs):\n if epoch < self.config.warmup:\n lr_scale = min(1., float(epoch+1) / float(self.config.warmup))\n for pg in self.optimizer.param_groups:\n pg['lr'] = lr_scale * self.config.lr\n if self.config.verbose:\n lr = self.optimizer.param_groups[0]['lr']\n timestamp = datetime.utcnow().isoformat()\n self.log(f'\\n{timestamp}\\nLR: {lr}')\n\n t = time.time()\n summary_loss = self.train_one_epoch(train_loader)\n\n self.log(f'[RESULT]: Train. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n self.save(f'{self.base_dir}/last-checkpoint.bin')\n\n t = time.time()\n summary_loss = self.validation(validation_loader)\n\n self.log(f'[RESULT]: Val. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n if summary_loss.avg < self.best_summary_loss:\n self.best_summary_loss = summary_loss.avg\n self.model.eval()\n self.save(f'{self.base_dir}/best-checkpoint-{str(self.epoch).zfill(3)}epoch.bin')\n for path in sorted(glob(f'{self.base_dir}/best-checkpoint-*epoch.bin'))[:-3]:\n os.remove(path)\n\n if self.config.validation_scheduler:\n self.scheduler.step(metrics=summary_loss.avg)\n\n self.epoch += 1\n\n def validation(self, val_loader):\n self.model.eval()\n summary_loss = AverageMeter()\n t = time.time()\n for step, (images, targets, image_ids) in enumerate(val_loader):\n if self.config.verbose:\n if step % self.config.verbose_step == 0:\n print(\n f'Val Step {step}/{len(val_loader)}, ' + \\\n f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n f'time: {(time.time() - t):.5f}', end='\\r'\n )\n with torch.no_grad():\n images = torch.stack(images)\n batch_size = images.shape[0]\n images = images.to(self.device).float()\n boxes = [target['boxes'].to(self.device).float() for target in targets]\n labels = [target['labels'].to(self.device).float() for target in targets]\n\n loss, _, _ = self.model(images, boxes, labels)\n summary_loss.update(loss.detach().item(), batch_size)\n\n return summary_loss\n\n def train_one_epoch(self, train_loader):\n self.model.train()\n summary_loss = AverageMeter()\n t = time.time()\n for step, (images, targets, image_ids) in enumerate(train_loader):\n if self.config.verbose:\n if step % self.config.verbose_step == 0:\n print(\n f'Train Step {step}/{len(train_loader)}, ' + \\\n f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n f'time: {(time.time() - t):.5f}', end='\\r'\n )\n \n images = torch.stack(images)\n images = images.to(self.device).float()\n batch_size = images.shape[0]\n boxes = [target['boxes'].to(self.device).float() for target in targets]\n labels = [target['labels'].to(self.device).float() for target in targets]\n\n self.optimizer.zero_grad()\n \n loss, _, _ = self.model(images, boxes, labels)\n \n loss.backward()\n\n summary_loss.update(loss.detach().item(), batch_size)\n\n self.optimizer.step()\n\n if self.config.step_scheduler:\n self.scheduler.step()\n\n return summary_loss\n \n def save(self, path):\n self.model.eval()\n torch.save({\n 'model_state_dict': self.model.model.state_dict(),\n 'optimizer_state_dict': self.optimizer.state_dict(),\n 'scheduler_state_dict': self.scheduler.state_dict(),\n 'best_summary_loss': self.best_summary_loss,\n 'epoch': self.epoch,\n }, path)\n\n def load(self, path):\n checkpoint = torch.load(path)\n self.model.model.load_state_dict(checkpoint['model_state_dict'])\n self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n self.best_summary_loss = checkpoint['best_summary_loss']\n self.epoch = checkpoint['epoch'] + 1\n \n def log(self, message):\n if self.config.verbose:\n print(message)\n with open(self.log_path, 'a+') as logger:\n logger.write(f'{message}\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class TrainGlobalConfig:\n num_workers = 2\n batch_size = 4 \n n_epochs = 40 # n_epochs = 40\n lr = 0.0002\n warmup = 5\n folder = 'effdet5-cutmix-augmix'\n\n # -------------------\n verbose = True\n verbose_step = 1\n # -------------------\n\n # --------------------\n step_scheduler = False # do scheduler.step after optimizer.step\n validation_scheduler = True # do scheduler.step after validation stage loss\n\n# SchedulerClass = torch.optim.lr_scheduler.OneCycleLR\n# scheduler_params = dict(\n# max_lr=0.001,\n# epochs=n_epochs,\n# steps_per_epoch=int(len(train_dataset) / batch_size),\n# pct_start=0.1,\n# anneal_strategy='cos', \n# final_div_factor=10**5\n# )\n \n SchedulerClass = torch.optim.lr_scheduler.ReduceLROnPlateau\n scheduler_params = dict(\n mode='min',\n factor=0.5,\n patience=1,\n verbose=False, \n threshold=0.0001,\n threshold_mode='abs',\n cooldown=0, \n min_lr=1e-8,\n eps=1e-08\n )\n # --------------------","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def collate_fn(batch):\n return tuple(zip(*batch))\n\ndef run_training():\n device = torch.device('cuda:0')\n net.to(device)\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=TrainGlobalConfig.batch_size,\n sampler=RandomSampler(train_dataset),\n pin_memory=False,\n drop_last=True,\n num_workers=TrainGlobalConfig.num_workers,\n collate_fn=collate_fn,\n )\n val_loader = torch.utils.data.DataLoader(\n validation_dataset, \n batch_size=TrainGlobalConfig.batch_size,\n num_workers=TrainGlobalConfig.num_workers,\n shuffle=False,\n sampler=SequentialSampler(validation_dataset),\n pin_memory=False,\n collate_fn=collate_fn,\n )\n\n fitter = Fitter(model=net, device=device, config=TrainGlobalConfig)\n fitter.fit(train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\n\ndef get_net():\n config = get_efficientdet_config('tf_efficientdet_d7')\n net = EfficientDet(config, pretrained_backbone=False)\n checkpoint = torch.load('../input/efficientdet/efficientdet_d7-f05bf714.pth')\n net.load_state_dict(checkpoint)\n config.num_classes = 1\n config.image_size = 512\n net.class_net = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=.001, momentum=.01))\n return DetBenchTrain(net, config)\n\nnet = get_net()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run_training()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Thank you for reading my kernel!\n\nSo, I have prepared good training SOTA-model baseline for you, my friends! I have used n_epochs = 40 and have got best checkpoint single model that gives 0.7176 LB. You can see [here](https://www.kaggle.com/shonenkov/inference-efficientdet) inference kernel.\n\nJust recently I have started publishing my works, if you like this format of notebooks I would like continue to make kernels.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}