{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Faster RCNN for Global Wheat Detection\n\n\n### Understanding Task\n\nThe task of this competition is to predict the bounding boxes of wheat heads in different images. The images have a varying number of wheat heads, colors, orientations, and so on making the task more challenging. These images are used to estimate the density and size of wheat heads in different varieties. Farmers can use the data to assess health and maturity when making management decisions in their fields.\n\n\n### About Dataset\n\n- There is a toal of 3422 unique train images. The code to get this number:\n- Thus some images don't have any masks\n- There is a toal of 147793 masks\n- Thus, on average, there are 43.8 masks per image\n- The image with the most masks contains 116. It is the image with id 35b935b6c.\n- All the train images have the same size: 1024 x 1024.\n- There are 3 channels: R, G, B.\n- train.csv: each row show coordinates of a wheat bounding box. Information fields is: image_id, image_width, image_height, bbox(x, y, w, h)","metadata":{}},{"cell_type":"markdown","source":"## 1. Visualize GWD dataset","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport cv2\nimport os\nimport re\n\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport matplotlib.image as mpimg","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-08-04T13:01:53.059964Z","iopub.execute_input":"2021-08-04T13:01:53.060347Z","iopub.status.idle":"2021-08-04T13:01:53.065925Z","shell.execute_reply.started":"2021-08-04T13:01:53.060316Z","shell.execute_reply":"2021-08-04T13:01:53.064799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INPUT_DIR = '/kaggle/input/global-wheat-detection'\nTRAIN_DIR = f'{INPUT_DIR}/train'\nTEST_DIR = f'{INPUT_DIR}/test'\nTRAIN_CSV_PATH = f'{INPUT_DIR}/train.csv'","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:53.067654Z","iopub.execute_input":"2021-08-04T13:01:53.067947Z","iopub.status.idle":"2021-08-04T13:01:53.078901Z","shell.execute_reply.started":"2021-08-04T13:01:53.067921Z","shell.execute_reply":"2021-08-04T13:01:53.077798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_CSV_PATH)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:53.081162Z","iopub.execute_input":"2021-08-04T13:01:53.081602Z","iopub.status.idle":"2021-08-04T13:01:53.436262Z","shell.execute_reply.started":"2021-08-04T13:01:53.081559Z","shell.execute_reply":"2021-08-04T13:01:53.435541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(train_df['source'])","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:53.437463Z","iopub.execute_input":"2021-08-04T13:01:53.437853Z","iopub.status.idle":"2021-08-04T13:01:53.803467Z","shell.execute_reply.started":"2021-08-04T13:01:53.437826Z","shell.execute_reply":"2021-08-04T13:01:53.802477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Separating x,y,w,h into separate columns for convenience","metadata":{}},{"cell_type":"code","source":"bboxes = np.stack(train_df['bbox'].apply(lambda x:np.fromstring(x[1:-1], sep=',')))\n\nfor i, col in enumerate(['x', 'y', 'w', 'h']):\n    train_df[col] = bboxes[:, i]","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:53.804601Z","iopub.execute_input":"2021-08-04T13:01:53.804902Z","iopub.status.idle":"2021-08-04T13:01:54.561147Z","shell.execute_reply.started":"2021-08-04T13:01:53.804875Z","shell.execute_reply":"2021-08-04T13:01:54.560204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dropping the bbox column as it is not needed now","metadata":{}},{"cell_type":"code","source":"\ntrain_df['box_area'] = train_df['w']*train_df['h']","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:54.562732Z","iopub.execute_input":"2021-08-04T13:01:54.563122Z","iopub.status.idle":"2021-08-04T13:01:54.587742Z","shell.execute_reply.started":"2021-08-04T13:01:54.563083Z","shell.execute_reply":"2021-08-04T13:01:54.58677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:54.58892Z","iopub.execute_input":"2021-08-04T13:01:54.58922Z","iopub.status.idle":"2021-08-04T13:01:54.610273Z","shell.execute_reply.started":"2021-08-04T13:01:54.589193Z","shell.execute_reply":"2021-08-04T13:01:54.609243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#number of unique images in the dataframe\nlen(train_df['image_id'].unique())","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:54.611698Z","iopub.execute_input":"2021-08-04T13:01:54.612044Z","iopub.status.idle":"2021-08-04T13:01:54.640021Z","shell.execute_reply.started":"2021-08-04T13:01:54.611969Z","shell.execute_reply":"2021-08-04T13:01:54.639021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#number of images in the training directory\nlen(os.listdir(TRAIN_DIR))","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:54.643772Z","iopub.execute_input":"2021-08-04T13:01:54.644116Z","iopub.status.idle":"2021-08-04T13:01:54.912405Z","shell.execute_reply.started":"2021-08-04T13:01:54.644087Z","shell.execute_reply":"2021-08-04T13:01:54.911455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#append .jpg to image ids for easier handling\ntrain_df['image_id'] = train_df['image_id'].apply(lambda x: str(x) + '.jpg')\ntrain_df['image_id'].head()","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:54.914094Z","iopub.execute_input":"2021-08-04T13:01:54.914373Z","iopub.status.idle":"2021-08-04T13:01:54.992381Z","shell.execute_reply.started":"2021-08-04T13:01:54.914347Z","shell.execute_reply":"2021-08-04T13:01:54.991334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#obtaining a list of all images which have no wheat heads in them\nunique_imgs_wbox = list(train_df['image_id'].unique())\nall_unique_imgs = os.listdir(TRAIN_DIR)\nno_wheat_imgs = [img_id for img_id in all_unique_imgs if img_id not in unique_imgs_wbox]\nlen(no_wheat_imgs)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:54.993635Z","iopub.execute_input":"2021-08-04T13:01:54.994224Z","iopub.status.idle":"2021-08-04T13:01:55.184709Z","shell.execute_reply.started":"2021-08-04T13:01:54.994181Z","shell.execute_reply":"2021-08-04T13:01:55.183585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_all_bboxes(df, image_id, count=False):\n    bboxes = []\n    \n    for _,row in df[df.image_id == image_id].iterrows():\n        bboxes.append([row.x, row.y, row.w, row.h])\n    if count:\n        return bboxes, len(bboxes)\n    else:\n        return bboxes\n    \n\ndef select_img(df, n, wheat=True):\n    \n    if wheat:\n        img_ids = df.sample(n=n, random_state=0)['image_id']\n        return list(img_ids)\n    else:\n        img_ids = np.random.choice(no_wheat_imgs, n)\n        return list(img_ids)\n        \n\ndef plot_image(df, ids, bbox=False):\n    n = len(ids)\n    fig, ax = plt.subplots(2, n//2, figsize=(40,30))\n    \n    for i, img_id in enumerate(ids):\n        img = mpimg.imread(os.path.join(TRAIN_DIR, img_id))\n        ax[i//(n//2)][i%(n//2)].imshow(img)\n        ax[i//(n//2)][i%(n//2)].axis('off')\n        \n        if bbox:\n            bboxes = get_all_bboxes(df, img_id)\n            for bbox in bboxes:\n                rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=2,edgecolor='r',facecolor='none')\n                ax[i//(n//2)][i%(n//2)].add_patch(rect)\n        else:\n            pass\n        \n    plt.tight_layout()\n    plt.show()\n                ","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:55.186163Z","iopub.execute_input":"2021-08-04T13:01:55.186525Z","iopub.status.idle":"2021-08-04T13:01:55.199659Z","shell.execute_reply.started":"2021-08-04T13:01:55.186494Z","shell.execute_reply":"2021-08-04T13:01:55.198569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image(train_df, select_img(train_df,6))\n","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:01:55.201032Z","iopub.execute_input":"2021-08-04T13:01:55.201336Z","iopub.status.idle":"2021-08-04T13:02:00.168864Z","shell.execute_reply.started":"2021-08-04T13:01:55.201307Z","shell.execute_reply":"2021-08-04T13:02:00.166481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image(train_df, select_img(train_df, 6, wheat=False))","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:00.17051Z","iopub.execute_input":"2021-08-04T13:02:00.171004Z","iopub.status.idle":"2021-08-04T13:02:04.928115Z","shell.execute_reply.started":"2021-08-04T13:02:00.170957Z","shell.execute_reply":"2021-08-04T13:02:04.927068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image(train_df, select_img(train_df,6), bbox=True)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:04.929436Z","iopub.execute_input":"2021-08-04T13:02:04.929756Z","iopub.status.idle":"2021-08-04T13:02:10.607098Z","shell.execute_reply.started":"2021-08-04T13:02:04.929727Z","shell.execute_reply":"2021-08-04T13:02:10.604778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Create DataLoader","metadata":{}},{"cell_type":"markdown","source":"### Split train-valid dataframe","metadata":{}},{"cell_type":"code","source":"image_ids = train_df['image_id'].unique()\nvalid_ids = image_ids[-665:]\ntrain_ids = image_ids[:-665]\n\nvalid_df = train_df[train_df['image_id'].isin(valid_ids)]\ntrain_df = train_df[train_df['image_id'].isin(train_ids)]","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:10.608753Z","iopub.execute_input":"2021-08-04T13:02:10.609183Z","iopub.status.idle":"2021-08-04T13:02:10.670834Z","shell.execute_reply.started":"2021-08-04T13:02:10.609139Z","shell.execute_reply":"2021-08-04T13:02:10.670068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:10.67185Z","iopub.execute_input":"2021-08-04T13:02:10.672252Z","iopub.status.idle":"2021-08-04T13:02:10.67801Z","shell.execute_reply.started":"2021-08-04T13:02:10.672216Z","shell.execute_reply":"2021-08-04T13:02:10.676583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:10.679328Z","iopub.execute_input":"2021-08-04T13:02:10.679626Z","iopub.status.idle":"2021-08-04T13:02:10.694781Z","shell.execute_reply.started":"2021-08-04T13:02:10.679597Z","shell.execute_reply":"2021-08-04T13:02:10.69389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create DataLoader","metadata":{}},{"cell_type":"code","source":"import albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nimport torch\nimport torchvision\n\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection import FasterRCNN\nfrom torchvision.models.detection.rpn import AnchorGenerator\n\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.utils.data.sampler import SequentialSampler","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:10.696329Z","iopub.execute_input":"2021-08-04T13:02:10.6967Z","iopub.status.idle":"2021-08-04T13:02:13.053193Z","shell.execute_reply.started":"2021-08-04T13:02:10.69666Z","shell.execute_reply":"2021-08-04T13:02:13.052109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class WheatDataset(Dataset):\n    \n    def __init__(self, dataframe, image_dir, transforms=None):\n        super().__init__()\n        \n        self.image_ids = dataframe['image_id'].unique()\n        self.df = dataframe\n        self.image_dir = image_dir\n        self.transforms = transforms\n        \n    def __getitem__(self, index: int):\n        image_id = self.image_ids[index]\n        records = self.df[self.df['image_id'] == image_id]\n        \n        image = cv2.imread(os.path.join(self.image_dir, image_id), cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB). astype(np.float32)\n        image /= 255.0\n        \n        boxes = records[['x', 'y', 'w', 'h']].values\n        boxes[:, 2] = boxes[:,0] + boxes[:,2] #x_max = x_min(x) + w\n        boxes[:, 3] = boxes[:,1] + boxes[:,3] #y_max = y_min(y) + h\n        \n        area = records['box_area'].values\n        area = torch.as_tensor(area, dtype=torch.float32)\n        \n        #there is only one class\n        labels = torch.ones((records.shape[0],), dtype=torch.int64)\n        \n        #suppose all instances are not crowd\n        iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)\n        \n        target = {}\n        target['boxes'] = boxes\n        target['labels'] = labels\n        target['image_id'] = torch.tensor([index])\n        target['area'] = area\n        target['iscrowd'] = iscrowd\n        \n        if self.transforms:\n            sample = {\n                'image': image,\n                'bboxes': target['boxes'],\n                'labels': labels\n            }\n            sample = self.transforms(**sample)\n            image = sample['image']\n            \n            target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1,0)\n            \n        return image, target, image_id\n    \n    def __len__(self)->int:\n        return self.image_ids.shape[0]","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.054491Z","iopub.execute_input":"2021-08-04T13:02:13.054821Z","iopub.status.idle":"2021-08-04T13:02:13.070222Z","shell.execute_reply.started":"2021-08-04T13:02:13.054789Z","shell.execute_reply":"2021-08-04T13:02:13.069146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Albumentations\ndef get_train_transforms():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format':'pascal_voc', 'label_fields':['labels']})\n\ndef get_valid_transforms():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format':'pascal_voc', 'label_fields':['labels']})\n\n# Data Loader\ndef collate_fn(batch):\n    return tuple(zip(*batch))","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.071591Z","iopub.execute_input":"2021-08-04T13:02:13.072047Z","iopub.status.idle":"2021-08-04T13:02:13.084844Z","shell.execute_reply.started":"2021-08-04T13:02:13.072015Z","shell.execute_reply":"2021-08-04T13:02:13.084067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = WheatDataset(train_df, TRAIN_DIR, get_train_transforms())\nvalid_dataset = WheatDataset(train_df, TRAIN_DIR, get_valid_transforms())","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.085841Z","iopub.execute_input":"2021-08-04T13:02:13.086246Z","iopub.status.idle":"2021-08-04T13:02:13.122398Z","shell.execute_reply.started":"2021-08-04T13:02:13.086217Z","shell.execute_reply":"2021-08-04T13:02:13.121393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# split the dataset in train and test set\nindices = torch.randperm(len(train_dataset)).tolist","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.12358Z","iopub.execute_input":"2021-08-04T13:02:13.123926Z","iopub.status.idle":"2021-08-04T13:02:13.142977Z","shell.execute_reply.started":"2021-08-04T13:02:13.123895Z","shell.execute_reply":"2021-08-04T13:02:13.142233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_loader = DataLoader(\n    train_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.144508Z","iopub.execute_input":"2021-08-04T13:02:13.145236Z","iopub.status.idle":"2021-08-04T13:02:13.150889Z","shell.execute_reply.started":"2021-08-04T13:02:13.145187Z","shell.execute_reply":"2021-08-04T13:02:13.149806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_data_loader = DataLoader(\n    valid_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.152413Z","iopub.execute_input":"2021-08-04T13:02:13.153206Z","iopub.status.idle":"2021-08-04T13:02:13.161783Z","shell.execute_reply.started":"2021-08-04T13:02:13.153134Z","shell.execute_reply":"2021-08-04T13:02:13.160901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Create Model","metadata":{}},{"cell_type":"code","source":"# load a model; pre-trained on COCO\nmodel = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:13.163084Z","iopub.execute_input":"2021-08-04T13:02:13.163657Z","iopub.status.idle":"2021-08-04T13:02:20.536258Z","shell.execute_reply.started":"2021-08-04T13:02:13.163612Z","shell.execute_reply":"2021-08-04T13:02:20.53543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_classes = 2 # wheat + no_wheat(background)\n\n#get number of input features for the  classificatier\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n\n#replace the pre-trained head with a new one\nmodel.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:20.542989Z","iopub.execute_input":"2021-08-04T13:02:20.543324Z","iopub.status.idle":"2021-08-04T13:02:20.550904Z","shell.execute_reply.started":"2021-08-04T13:02:20.54329Z","shell.execute_reply":"2021-08-04T13:02:20.549547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Averager:\n    def __init__(self):\n        self.current_total = 0.0\n        self.iterations = 0.0\n        \n    def send(self, value):\n        self.current_total += value\n        self.iterations += 1\n        \n    @property\n    def value(self):\n        if self.iterations == 0:\n            return 0\n        else:\n            return 1.0 * self.current_total/self.iterations\n        \n    def reset(self):\n        self.current_total = 0.0\n        self.iterations = 0.0","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:20.555144Z","iopub.execute_input":"2021-08-04T13:02:20.556912Z","iopub.status.idle":"2021-08-04T13:02:20.747323Z","shell.execute_reply.started":"2021-08-04T13:02:20.556862Z","shell.execute_reply":"2021-08-04T13:02:20.746509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Training Model","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:20.750836Z","iopub.execute_input":"2021-08-04T13:02:20.751164Z","iopub.status.idle":"2021-08-04T13:02:20.761575Z","shell.execute_reply.started":"2021-08-04T13:02:20.751131Z","shell.execute_reply":"2021-08-04T13:02:20.760799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.to(device)\nparams = [p for p in model.parameters() if  p.requires_grad]\noptimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)\nlr_scheduler = None\n# lr_scheduler = torch.optim.lr_scheduler.StepLR(opotimizer, step_size=3, gamma=0.1)\nnum_epochs = 2\n\nloss_hist = Averager()\n\nitr = 1","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:20.762699Z","iopub.execute_input":"2021-08-04T13:02:20.763125Z","iopub.status.idle":"2021-08-04T13:02:20.784226Z","shell.execute_reply.started":"2021-08-04T13:02:20.763094Z","shell.execute_reply":"2021-08-04T13:02:20.783078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for epoch in range(num_epochs):\n    loss_hist.reset()\n    \n    for images, targets, image_ids in train_data_loader:\n        images = list(image.to(device) for image in images)\n        targets = [{k: v.to(device) for k,v in t.items()} for t in targets]\n        \n        loss_dict = model(images, targets)\n        \n        losses = sum(loss for loss in loss_dict.values())\n        loss_value = losses.item()\n        \n        loss_hist.send(loss_value)\n        \n        optimizer.zero_grad()\n        losses.backward()\n        optimizer.step()\n        \n        if itr % 50 == 0:\n            print(f\"Iteration #{itr} loss: {loss_value}\")\n            \n        itr += 1\n        \n    # update the learning rate\n    if lr_scheduler is not None:\n        lr_scheduler.step()\n        \n    print(f\"Epoch #{epoch} loss: {loss_hist.value}\")","metadata":{"execution":{"iopub.status.busy":"2021-08-04T13:02:20.785501Z","iopub.execute_input":"2021-08-04T13:02:20.78581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images, targets, image_ids = next(iter(valid_data_loader))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images = list(img.to(device) for img in images)\ntargets = [{k: v.to(device) for k,v in t.items()} for t in targets]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxes = targets[1]['boxes'].cpu().numpy().astype(np.int32)\nsample = images[1].permute(1,2,0).cpu().numpy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.eval()\ncpu_device = torch.device(\"cpu\")\n\noutputs = model(images)\noutputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nfor box in boxes:\n    cv2.rectangle(sample,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (220, 0, 0), 3)\n    \nax.set_axis_off()\nax.imshow(sample)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.save(model.state_dict(), 'fasterrcnn_resnet50_fpn.pth')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. Object Detection with [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)","metadata":{}},{"cell_type":"markdown","source":"### 5.1. Import libraries","metadata":{}},{"cell_type":"code","source":"!pip uninstall pycocotools -y\n!pip install -q git+https://github.com/waleedka/coco.git#subdirectory=PythonAPI","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install hydra-core\n!pip install pytorch-lightning==0.8.1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import DataLoader, Dataset\nimport torch\nfrom PIL import Image\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMG_SIZE = 256","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_transforms():\n    return A.Compose([\n        A.RandomSizedCrop(min_max_height=(800,800), height=IMG_SIZE, width=IMG_SIZE, p=0.5),\n        A.OneOf([\n            A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2,\n                                val_shift_limit=0.2, p=0.9),\n            A.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.9),\n        ], p=0.9),\n        A.ToGray(p=0.01),\n        A.HorizontalFlip(p=0.5),\n        A.VerticalFlip(p=0.5),\n        A.Resize(height=256, width=256, p=1),\n        A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n        ToTensorV2(p=1.0),\n    ],\n    p=1.0,\n    bbox_params=A.BboxParams(\n        format='pascal_voc',\n        min_area=0,\n        min_visibility=0,\n        label_fields=['labels']\n        )\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_valid_transforms():\n    return A.Compose([\n        A.Resize(height=IMG_SIZE, width=IMG_SIZE, p=1.0),\n        ToTensorV2(p=1.0),\n        ],\n        p=1.0,\n        bbox_params=A.BboxParams(\n            format='pascal_voc',\n            min_area=0,\n            min_visibility=0,\n            label_fields=['labels']\n        )\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_test_transforms():\n    return A.Compose([\n        A.Resize(height=IMG_SIZE, width=IMG_SIZE, p=1.0),\n        ToTensorV2(p=1.0),\n    ], p=1.0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class WheatDataset(Dataset):\n    \n    def __init__(self, df=None, mode=\"train\", image_dir=\"\", transforms=None):\n        super().__init__()\n        if df is not None:\n            self.df = df.copy()\n            self.image_ids = df['image_id'].unique()\n        else:\n            # test case\n            self.df = None\n            self.image_ids = [p.stem for p in Path(image_dir).glob(\"*.jpg\")]\n        \n        self.image_dir = image_dir\n        self.transforms = transforms\n        self.mode = mode\n        \n        \n    def __getitem__(self, index:int):\n        image_id = self.image_ids[index]\n        \n        image = Image.open(f'{self.image_dir}/{image_id}').convert(\"RGB\")\n        image = np.array(image)\n        image = image/255.\n        image = image.astype(np.float32)\n        \n        if self.mode != 'test':\n            records = self.df[self.df['image_id'] == image_id]\n            \n            area = records['box_area'].values\n            area = torch.as_tensor(area, dtype=torch.float32)\n            \n            boxes = records[['x', 'y', 'w', 'h']].values\n            boxes[:, 2] = boxes[:,0] + boxes[:,2] #x_max = x_min(x) + w\n            boxes[:, 3] = boxes[:,1] + boxes[:,3] #y_max = y_min(y) + h\n            \n            labels = torch.ones((records.shape[0],), dtype=torch.int64)\n            \n            iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)\n            \n            target = {}\n            target['boxes'] = boxes\n            target['labels'] = labels\n            # target['masks'] = None\n            target['image_id'] = torch.tensor([index])\n            target['area'] = area\n            target['iscrowd'] = iscrowd\n            # These are needed as well by the efficientdet model.\n            target['img_size'] = torch.tensor([(IMG_SIZE, IMG_SIZE)])\n            target['img_scale'] = torch.tensor([1.])\n            \n        else:\n            target = {'cls': torch.as_tensor([[0]], dtype=torch.float32),\n                      'bbox': torch.as_tensor([[0,0,0,0]], dtype=torch.float32),\n                      'img_size': torch.tensor([(IMG_SIZE, IMG_SIZE)]),\n                      'img_scale': torch.tensor([1.])\n                     }\n            \n        \n        if self.mode != 'test':\n            \n            if self.transforms:\n                sample = {\n                    'image': image,\n                    'bboxes': target['boxes'],\n                    'labels': labels\n                }\n                if len(sample['bboxes']) > 0:\n                    #apply augmentation on the fly\n                    sample = self.transforms(**sample)\n                    image = sample['image']\n                    boxes = sample['bboxes']\n                    target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*boxes)))).permute(1,0)\n            \n            else:\n                sample = {\n                    'image': image,\n                    'bbox': target['bbox'],\n                    'cls': target['cls']\n                }\n                image = self.transforms(**sample)['image']\n        \n            return image, target\n        \n    def __len__(self) -> int:\n        return len(self.image_ids)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_train_labels_df = train_df.copy()\nprocessed_train_labels_df[\"x2\"] = processed_train_labels_df[\"x\"] + processed_train_labels_df[\"w\"]\nprocessed_train_labels_df[\"y2\"] = processed_train_labels_df[\"y\"] + processed_train_labels_df[\"h\"]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_train_labels_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create stratified folds, here using the source.\n# This isn't the most optimal way to do it but I will leave it to you \n# to find a better one. \nfrom sklearn.model_selection import StratifiedKFold\nskf = StratifiedKFold(n_splits=5)\nfor fold, (train_index, valid_index) in enumerate(skf.split(processed_train_labels_df,\n                                                            y=processed_train_labels_df['source'])):\n    processed_train_labels_df.loc[valid_index, 'fold'] = fold","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_train_labels_df.sample(2).T","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_transforms = get_train_transforms()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = WheatDataset(processed_train_labels_df, mode='train',\n                            image_dir=TRAIN_DIR, transforms=train_transforms)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image, target = train_dataset[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_train_labels_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom PIL import Image, ImageDraw\nimport pandas as pd\nfrom pathlib import Path\n\ndef plot_image_with_bboxes(img_id, df):\n    img_path= Path(f'{TRAIN_DIR}/{img_id}')\n    img = Image.open(img_path)\n    draw = ImageDraw.Draw(img)\n    \n    bboxes=[]\n    for _,row in df[df.image_id == img_id].iterrows():\n        bboxes.append([row.x, row.y, row.w, row.h])\n        \n    for bbox in bboxes:\n        x, y, w, h = bbox\n#         print(x, y, w, h)\n        transformed_bbox = [x, y, x + w, y + h]\n        draw.rectangle(transformed_bbox, outline=\"red\", width=3)\n        \n    return img","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_with_bboxes('b6ab77fd7.jpg', train_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.2 Pytorch Lightning Complete Pipline\n\n**Pytorch Lightning**: we will build a model, start with the **model building block** then addd the **processing** step.\n\nFisetly, we install the nesscessary library:","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade pip\n!pip install pytorch_lightning\n!pip install effdet --upgrade\n!pip install timm\n!pip install omegaconf\n!pip install pycocotools","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create the EfficientDet model using the code snippet presented above: the new `create_model` code snippet instead of the `get_train_efficientdet` function since we are using the latest effdet version","metadata":{}},{"cell_type":"code","source":"import torch\nfrom effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\n\n\ndef get_train_efficientdet():\n    config = get_efficientdet_config('tf_efficientdet_d5')\n    net = EfficientDet(config, pretrained_backbone=False)\n    checkpoint = torch.load('../input/efficientdet/efficientdet_d5-ef44aea8.pth')\n    net.load_state_dict(checkpoint)\n    config.num_classes = 1 \n    config.image_size = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=0.001, momentum=0.01))\n    return DetBenchTrain(net, config)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# new way\nfrom effdet import create_model\n\ndef get_train_efficientdet():\n    return create_model('tf_efficientdet_d5', bench_task='train', \n                        num_classes=2, bench_labeler=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pytorch_lightning import LightningModule\n\nclass WheatModel(LightningModule):\n    \n    def __init__(self,  df, fold):\n        super().__init__()\n        self.df = df\n        self.train_df = self.df.loc[lambda df: df[\"fold\"] != fold]\n        self.valid_df = self.df.loc[lambda df: df[\"fold\"] == fold]\n        self.image_dir = TRAIN_DIR\n        self.model = get_train_efficientdet()\n        self.num_workers = 4\n        self.batch_size = 8\n    \n    def forward(self, image, target):\n        return self.model(image, target)\n    \n#Create model for one fold\nmodel = WheatModel(processed_train_labels_df, fold=0)\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n# add the train and validation data loaders\ndef train_dataloader(self):\n    train_transforms = get_train_transforms()\n    train_dataset = WheatDataset(self.train_df, image_dir=self.image_dir,\n                                transforms=train_transforms)\n    return DataLoader(\n        train_dataset,\n        batch_size=self.batch_size,\n        sampler=RandomSampler(train_dataset),\n        pin_memory=False,\n        drop_last=True,\n        collate_fn=collate_fn,\n        num_workers=self.num_workers,\n    )\n\ndef val_dataloader(self):\n    valid_transforms = get_train_transforms()\n    valid_dataset = WheatDataset(self.valid_df, image_dir=self.image_dir,\n                                transforms=valid_transforms)\n    valid_dataloader = DataLoader(\n        valid_dataset,\n        batch_size=self.batch_size,\n        sampler=SequentialSampler(valid_dataset),\n        pin_memory=False,\n        shuffle=False,\n        collate_fn=collate_fn,\n        num_workers=self.num_workers,\n    )\n    \n    iou_types = [\"bbox\"]\n    \n    return valid_dataloader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"WheatModel.train_dataloader = train_dataloader\nWheatModel.val_dataloader = val_dataloader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def training_step(self, batch, batch_idx):\n    images, targets = batch\n    targets = [{k: v for k, v in t.items()} for t in targets]\n    \n    #separate losses\n    images = torch.stack(images).float()\n    targets2 = {}\n    targets2[\"bbox\"] = [target[\"boxes\"].float() for target in targets]\n    #variable nu,ber of instances, so the entire structure can be forced to tensor\n    targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n    \"\"\"\n    targets2[\"image_id\"] = torch.tensor(\n        [target[\"image_id\"] for target in targets]\n    ).float()\n    targets2[\"img_scale\"] = torch.tensor(\n        [target[\"img_scale\"] for target in targets], device=\"cuda\"\n    ).float()\n    targets2[\"img_size\"] = torch.tensor(\n        [(IMG_SIZE, IMG_SIZE) for target in targets], device=\"cuda\"\n    ).float()\n    \"\"\"\n    loses_dict = self.model(images, targets2)\n    \n    return {\"loss\": losses_dict[\"loss\"], \"log\": losses_dict}\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def validation_step(self, batch, batch_idx):\n    images, targets = batch\n    targets = [{k: v for k, v in t.items()} for t in targets]\n    \n    #separate losses\n    images = torch.stack(images).float()\n    targets2 = {}\n    targets2[\"bbox\"] = [target[\"boxes\"].float() for target in targets]\n    #variable nu,ber of instances, so the entire structure can be forced to tensor\n    targets2[\"cls\"] = [target[\"labels\"].float() for target in targets]\n    \"\"\"\n    targets2[\"image_id\"] = torch.tensor(\n        [target[\"image_id\"] for target in targets]\n    ).float()\n    targets2[\"img_scale\"] = torch.tensor(\n        [target[\"img_scale\"] for target in targets], device=\"cuda\"\n    ).float()\n    targets2[\"img_size\"] = torch.tensor(\n        [(IMG_SIZE, IMG_SIZE) for target in targets], device=\"cuda\"\n    ).float()\n    \"\"\"\n    loses_dict = self.model(images, targets2)\n    loss_val = losses_dict[\"loss\"]\n    detections = losses_dict[\"detections\"]\n    #Back to x, y, x, y format\n    detections[:,:,[1,0,3,2]] = detections[:, :, [0,1,2,3]]\n    \n    res = {target[\"image_id\"].item(): {\n                'boxes': output[:, 0:4],\n                'scores': output[:, 4],\n                'labels': output[:, 5]      \n    }for target, output in zip(targets, detections)}\n    \n    # iou = self._calculate_iou(targets, res, IMG_SIZE)\n    # iou = torch.as_tensor(iou)\n    # self.coco_evaluator.update(res)\n    \n    return {\"loss\": losses_dict[\"loss\"], \"log\": losses_dict}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def validation_epoch_end(self, outputs):\n    # self.coco_evaluator.accumulate()\n    # self.coco_evaluator.summarize()\n    # coco main metric\n    # metric = self.coco_evaluator.coco_eval[\"bbox\"].stats[0]\n    # metric = torch.as_tensor(metric)\n    # tensorboard_logs = {\"main_score\": metric}\n    # return {\n    #     \"val_loss\": metric,\n    #     \"log\": tensorboard_logs,\n    #     \"progress_bar\": tensorboard_logs,\n    # }\n    pass\n\ndef configure_optimizers(self):\n    return torch.optim.AdamW(self.model.parameters(), lr=1e-4)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"WheatModel.training_step = training_step\n# WheatModel.validation_step = validation_step\n# WheatModel.validation_epoch_end = validation_epoch_end\nWheatModel.configure_optimizers = configure_optimizers","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`WheatModel` is ready now. Let's train it. For that, we will create a `Trainer` and set it to `fast_dev_run=True` for a quicker demo. Also, since it is in this mode, the Trainer doesn't automatically save the weights at the end (correct me if I am wrong of course) so we need to add a torch.save call at the end.","metadata":{}},{"cell_type":"code","source":"from pytorch_lightning import Trainer, seed_everything, loggers\n\nseed_everything(314)\n\n#create model for one fold\nmodel = WheatModel(processed_train_labels_df, fold=0)\nlogger = loggers.TensorBoardLogger(\"logs\", name=\"effdet-b5\", version=\"fold_0\")\ntrainer = Trainer(gpus=1, logger=logger, fast_dev_run=True)\ntrainer.fit(model)\ntorch.save(model.model.state_dict(), \"wheatdet.pth\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}