{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"

[Pytorch] - Global Wheat Detection FASTER R-CNN [EDA - AUGMENTATION - COMPARE MODELS]

\n\n***","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Change Log\n\n#### V2\n\n> - add new model (LB score 0.663 --> 0.673)\n\n#### V1\n\n> - add inference\n> - add another model to compare\n> - change code for train and test","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Import necessary libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport time\nimport shutil\n\nimport numpy as np\nimport pandas as pd\n\nimport torch\nimport torchvision\nfrom torchvision import transforms\nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor, FasterRCNN\nfrom torchvision.models.detection.backbone_utils import resnet_fpn_backbone\nfrom tqdm.notebook import tqdm\n\nimport albumentations\nfrom albumentations import *\nfrom albumentations.pytorch.transforms import ToTensorV2\nimport cv2\nimport matplotlib.pyplot as plt\nfrom matplotlib import patches\nimport seaborn as sns\n%matplotlib inline\nprint('Ready...')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data and Simple EDA","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = '../input/global-wheat-detection/train/'\nTEST_DIR = '../input/global-wheat-detection/test/'\ntrain_df_path = '../input/global-wheat-detection/train.csv'\ntest_df_path = '../input/global-wheat-detection/sample_submission.csv'\nList_Data_dir = os.listdir(DATA_DIR)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"raw = pd.read_csv(train_df_path)\nraw","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"raw.describe()\n\n# all images have resolution 1024 x 1024","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Total number of train images: {raw.image_id.nunique()}')\nprint(f'Total number of test images: {len(os.listdir(TEST_DIR))}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,8))\nplt.title('Wheat Distribution', fontsize= 20)\nsns.countplot(x=\"source\", data=raw)\n\n# based on the chart, there are seven types of wheat from images data, with the most types 'ethz_1' and the least is 'inrae_1'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract bbox column to xmin, ymin, width, height, then create xmax, ymax, and area columns\n\nraw[['xmin','ymin','w','h']] = pd.DataFrame(raw.bbox.str.strip('[]').str.split(',').tolist()).astype(float)\nraw['xmax'], raw['ymax'], raw['area'] = raw['xmin'] + raw['w'], raw['ymin'] + raw['h'], raw['w'] * raw['h']\nraw","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's look at some random images with boundary boxes**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_image(image_id):\n \n fig, ax = plt.subplots(1, 2, figsize = (24, 24))\n ax = ax.flatten()\n \n bbox = raw[raw['image_id'] == image_id ]\n img_path = os.path.join(DATA_DIR, image_id + '.jpg')\n \n image = cv2.imread(img_path, cv2.IMREAD_COLOR)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n image2 = image\n \n ax[0].set_title('Original Image')\n ax[0].imshow(image)\n \n for idx, row in bbox.iterrows():\n x1 = row['xmin']\n y1 = row['ymin']\n x2 = row['xmax']\n y2 = row['ymax']\n label = row['source']\n \n cv2.rectangle(image2, (int(x1),int(y1)), (int(x2),int(y2)), (255,255,255), 2)\n font = cv2.FONT_HERSHEY_SIMPLEX\n cv2.putText(image2, label, (int(x1),int(y1-10)), font, 1, (255,255,255), 2)\n \n ax[1].set_title('Image with Bondary Box')\n ax[1].imshow(image2)\n\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image(raw.image_id.unique()[91])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image(raw.image_id.unique()[1231])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image(raw.image_id.unique()[3121])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentations\n\n\n

Data augmentation is a strategy that enables practitioners to significantly increase the diversity of data available for training models, without actually collecting new data. \nIn image data, augmentation can range from basic image manipulation like color variation, Fliping, resize, or rotate image, and data augmentation can also reduce overfitting.

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_bboxes(bboxes, col, bbox_format = 'pascal_voc', color='white'):\n for i in range(len(bboxes)):\n x_min = bboxes[i][0]\n y_min = bboxes[i][1]\n x_max = bboxes[i][2]\n y_max = bboxes[i][3]\n width = x_max - x_min\n height = y_max - y_min\n rect = patches.Rectangle((x_min, y_min), \n width, height, \n linewidth=2, \n edgecolor=color, \n facecolor='none')\n col.add_patch(rect)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def augmented_images(image, augment):\n \n fig, ax = plt.subplots(1, 2, figsize = (24, 24))\n ax = ax.flatten()\n \n image_data = raw[raw['image_id'] == image]\n bbox = image_data[['xmin', 'ymin', 'xmax', 'ymax']].astype(np.int32).values\n labels = np.ones((len(bbox), ))\n\n image = cv2.imread(os.path.join(DATA_DIR + '/{}.jpg').format(image), cv2.IMREAD_COLOR)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n \n get_bboxes(bbox, ax[0], color='white')\n \n ax[0].set_title('Original Image with Bounding Boxes')\n ax[0].imshow(image)\n \n aug = albumentations.Compose([augment], \n bbox_params={'format': 'pascal_voc', 'label_fields':['labels']})\n\n \n aug_result = aug(image=image, bboxes=bbox, labels=labels)\n\n aug_image = aug_result['image']\n get_bboxes(aug_result['bboxes'], ax[1], color='red')\n \n ax[1].set_title('Augmented Image with Bounding Boxes')\n ax[1].imshow(aug_image)\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# HorizontalFlip Augmentation\naugmented_images(raw.image_id.unique()[1230], albumentations.HorizontalFlip(p=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# VerticalFlip Augmentation\naugmented_images(raw.image_id.unique()[2110], albumentations.VerticalFlip(p=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Change Color to gray\naugmented_images(raw.image_id.unique()[1212], albumentations.ToGray(p=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Random Change Brightness Contrast\naugmented_images(raw.image_id.unique()[1230], albumentations.RandomBrightnessContrast(p=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Preprocessing (Train Data)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class wheatdataset_train(Dataset):\n \n def __init__(self, dataframe, data_dir, transforms=None):\n super().__init__()\n self.df = dataframe \n self.image_list = list(self.df['image_id'].unique())\n self.image_dir = data_dir\n self.transforms = transforms\n \n def __len__(self):\n return len(self.image_list)\n \n def __getitem__(self, idx):\n \n image_id = self.image_list[idx]\n image_data = self.df.loc[self.df['image_id'] == image_id]\n boxes = torch.as_tensor(np.array(image_data[['xmin','ymin','xmax','ymax']]), \n dtype=torch.float32)\n area = torch.tensor(np.array(image_data['area']), dtype=torch.int64) \n labels = torch.ones((image_data.shape[0],), dtype=torch.int64)\n iscrowd = torch.zeros((image_data.shape[0],), dtype=torch.uint8)\n \n target = {}\n target['boxes'] = boxes\n target['area'] = area\n target['labels'] = labels\n target['iscrowd'] = iscrowd\n \n image = cv2.imread((self.image_dir + '/' + image_id + '.jpg'), cv2.IMREAD_COLOR)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n \n if self.transforms:\n \n image_transforms = {\n 'image': image,\n 'bboxes': target['boxes'],\n 'labels': labels\n }\n \n image_transforms = self.transforms(**image_transforms)\n image = image_transforms['image']\n \n target['boxes'] = torch.as_tensor(image_transforms['bboxes'], dtype=torch.float32)\n \n return image, target","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Albumentations\n\ndef get_train_transform():\n return albumentations.Compose([\n #albumentations.Resize(p=1, height=512, width=512),\n albumentations.ToGray(p=0.5),\n albumentations.Flip(p=0.5),\n albumentations.RandomBrightnessContrast(p=0.5),\n ToTensorV2(p=1.0)\n ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\n\ndef get_test_transform():\n return albumentations.Compose([\n ToTensorV2(p=1.0)\n ])\n\n\ndef collate_fn(batch):\n return tuple(zip(*batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = wheatdataset_train(raw, DATA_DIR, get_train_transform())\ntrain_dataloader = DataLoader(train_data, batch_size=16,shuffle=True, num_workers=4,collate_fn=collate_fn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create a model and training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\ntorch.cuda.empty_cache()\nprint(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def train_model():\n \n model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n num_classes = 2 \n in_features = model.roi_heads.box_predictor.cls_score.in_features\n model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\n \n return model\n\ndef train(data_loader, epoch):\n \n model = train_model()\n model.to(device)\n params = [p for p in model.parameters() if p.requires_grad]\n optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)\n model.parameters\n\n\n total_train_loss = []\n itr = 1\n\n for epoch in tqdm(range(epoch)):\n \n print(f'Epoch :{epoch + 1}')\n start_time = time.time()\n train_loss = []\n model.train()\n for images, targets in tqdm(data_loader):\n \n images = list(image.to(device) for image in images)\n targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n loss_dict = model(images, targets)\n\n losses = sum(loss for loss in loss_dict.values())\n \n loss_value = losses.item()\n \n train_loss.append(losses.item())\n optimizer.zero_grad()\n losses.backward()\n optimizer.step()\n \n if itr % 50 == 0:\n print(f\"Iteration #{itr} loss: {loss_value:.4f}\")\n\n itr += 1\n \n \n epoch_train_loss = np.mean(train_loss)\n total_train_loss.append(epoch_train_loss)\n print(f'Epoch train loss is {epoch_train_loss:.4f}')\n time_elapsed = time.time() - start_time\n print('{:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n\n torch.save(model.state_dict(), 'resnet50_GWD.pth')\n\n #visualize\n plt.figure(figsize=(12,6))\n plt.title('Train Loss', fontsize= 20)\n plt.plot(total_train_loss)\n plt.xlabel('epochs')\n plt.ylabel('loss')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"num_epochs=3\ntrain(train_dataloader, num_epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference and Compare Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Load test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(test_df_path)\ntest_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Data Preprocessing (Test Data)

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class wheatdataset_test(Dataset):\n \n def __init__(self, dataframe, data_dir, transforms=None):\n super().__init__()\n self.df = dataframe \n self.image_list = list(self.df['image_id'].unique())\n self.image_dir = data_dir\n self.transforms = transforms\n \n def __len__(self):\n return len(self.image_list)\n \n def __getitem__(self, idx):\n \n image_id = self.image_list[idx]\n image_data = self.df.loc[self.df['image_id'] == image_id] \n image = cv2.imread((self.image_dir + '/' + image_id + '.jpg'), cv2.IMREAD_COLOR)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n \n if self.transforms:\n \n image_transforms = {\n 'image': image\n }\n \n image_transforms = self.transforms(**image_transforms)\n image = image_transforms['image']\n \n return image, image_id","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = wheatdataset_test(test_df, TEST_DIR, get_test_transform())\n\ntest_data_loader = DataLoader(\n test_dataset,\n batch_size=4,\n shuffle=False,\n num_workers=4,\n drop_last=False,\n collate_fn=collate_fn\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Inference

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"detection_threshold = 0.3\nresults = []\ncolor = (255,255,255)\nthickness = 2\nfont = cv2.FONT_HERSHEY_SIMPLEX \n\ndef save_model(model_number):\n\n MODEL_USE = model_number\n if MODEL_USE == 1:\n MODEL_NAME = 'Faster_RCNN_1'\n WEIGHT_PATH = '../input/global-wheat-detection-model/Faster_RCNN/resnet50_GWD_3.pth'\n elif MODEL_USE == 2:\n MODEL_NAME = 'Faster_RCNN_2'\n WEIGHT_PATH = '../input/global-wheat-detection-model/Faster_RCNN/resnet50_GWD_4.pth'\n elif MODEL_USE == 3:\n MODEL_NAME = 'Faster_RCNN_3'\n WEIGHT_PATH = '../input/global-wheat-detection-model/Faster_RCNN/resnet50_GWD_6.pth'\n elif MODEL_USE == 4:\n MODEL_NAME = 'Faster_RCNN_4'\n WEIGHT_PATH = '../input/global-wheat-detection-model/Faster_RCNN/resnet50_GWD_5.pth'\n elif MODEL_USE == 5:\n MODEL_NAME = 'Faster_RCNN_5'\n WEIGHT_PATH = '../input/global-wheat-detection-model/Faster_RCNN_v2/resnet50_GWD_7.pth' \n\n return MODEL_NAME, WEIGHT_PATH\n\ndef test_model():\n \n model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False,\n pretrained_backbone=False)\n num_classes = 2\n in_features = model.roi_heads.box_predictor.cls_score.in_features\n model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\n\n return model\n\ndef format_prediction_string(boxes, scores):\n pred_strings = []\n for j in zip(scores, boxes):\n pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], \n j[1][2], j[1][3]))\n\n return \" \".join(pred_strings)\n\ndef show_result(data_loader, model_number, color, subs = False):\n results = []\n MODEL_NAME, WEIGHT_PATH = save_model(model_number)\n model = test_model()\n model.load_state_dict(torch.load(WEIGHT_PATH))\n model.to(device)\n model.eval()\n print(f'Model Name : {MODEL_NAME}')\n\n for images, image_ids in data_loader:\n\n images = list(image.to(device) for image in images)\n outputs = model(images)\n\n for i, image in enumerate(images):\n\n boxes = outputs[i]['boxes'].data.cpu().numpy()\n scores = outputs[i]['scores'].data.cpu().numpy()\n \n boxes = boxes[scores >= detection_threshold].astype(np.int32)\n scores = scores[scores >= detection_threshold]\n image_id = image_ids[i]\n \n boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n\n if subs is True:\n result = {\n 'image_id': image_id,\n 'PredictionString': format_prediction_string(boxes, scores)\n }\n results.append(result)\n\n im = cv2.imread('{}/{}.jpg'.format(TEST_DIR, image_id))\n im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB).astype(np.float32)\n im /= 255.0\n\n for b,s in zip(boxes,scores):\n cv2.rectangle(im, (b[0],b[1]), (b[0]+b[2],b[1]+b[3]), color, thickness)\n cv2.putText(im, '{:.2}'.format(s), (b[0],b[1]), font, 1, color, thickness)\n \n plt.figure(figsize=(12,12))\n plt.imshow(im)\n \n return results,model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results, model = show_result(test_data_loader, model_number= 3, color = (0,255,0), subs = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compare Model\n\n****","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame(a, columns=['image_id', 'PredictionString'])\ntest_df.to_csv('submission.csv', index=False)\ntest_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), 'fasterrcnn_resnet50_fpn.pth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References\n\n\nEDA - Augmentations\n\n* https://github.com/albumentations-team/albumentations_examples\n\n* https://link.springer.com/article/10.1186/s40537-019-0197-0\n\n\nPytorch - Model\n\n* https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html\n \n* https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train\n \n* https://www.kaggle.com/arunmohan003/fasterrcnn-using-pytorch-baseline","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}