{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":19989,"databundleVersionId":1160143,"sourceType":"competition"},{"sourceId":7960641,"sourceType":"datasetVersion","datasetId":4682898},{"sourceId":7968217,"sourceType":"datasetVersion","datasetId":4688362}],"dockerImageVersionId":30673,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ***CS827 Deep Learning in Visual Computing Application Coursework***","metadata":{}},{"cell_type":"markdown","source":"# Overview/Summary","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-03-27T22:09:25.304543Z","iopub.execute_input":"2024-03-27T22:09:25.305144Z","iopub.status.idle":"2024-03-27T22:09:25.309173Z","shell.execute_reply.started":"2024-03-27T22:09:25.305113Z","shell.execute_reply":"2024-03-27T22:09:25.308053Z"}}},{"cell_type":"markdown","source":"**Name:** Dhyaneshwar Nagarajan <br>\n**Student Number:** 202350615 <br>\n**Class Number:** CS827: Deep Learning in Visual Computing Applications<br>\n**Topic:** Object Detection from Images using Faster RCNN and YOLO ","metadata":{}},{"cell_type":"markdown","source":"## Task","metadata":{"execution":{"iopub.status.busy":"2024-03-27T22:09:25.314249Z","iopub.execute_input":"2024-03-27T22:09:25.314919Z","iopub.status.idle":"2024-03-27T22:09:25.324140Z","shell.execute_reply.started":"2024-03-27T22:09:25.314871Z","shell.execute_reply":"2024-03-27T22:09:25.322922Z"}}},{"cell_type":"markdown","source":"The primary objective of this assignment is to explore and understand the intricacies of Convolutional Neural Networks (CNNs) and their application in visual computing tasks such as object detection and segmentation. Specifically, the assignment focuses on evaluating deep learning models for object detection from images, employing the global wheat detection dataset from Kaggle for practical assessments [1].","metadata":{}},{"cell_type":"markdown","source":"## Description","metadata":{}},{"cell_type":"markdown","source":"For the evaluation, two prominent object detection models were selected: Faster R-CNN (Region-based Convolutional Neural Networks) and YOLO (You Only Look Once). Faster R-CNN, recognized for its accuracy through a region proposal network, and YOLO, known for its speed and efficiency in detecting objects in real-time, were both trained and tested on the global wheat detection dataset. The evaluation metric utilized was the mean Average Precision (mAP) at various Intersection over Union (IoU) thresholds to quantitatively assess the performance of each model.","metadata":{}},{"cell_type":"markdown","source":"## Main Findings","metadata":{}},{"cell_type":"markdown","source":"The comparative analysis revealed distinct strengths and limitations of both Faster R-CNN and YOLO in the context of wheat detection from images. Faster R-CNN exhibited superior accuracy in detecting wheat spikes, albeit at the cost of increased computational time and resources. However, YOLO showcased remarkable speed in object detection, making it suitable for applications requiring real-time processing. The evaluation based on mAP scores at different IoU thresholds indicated that while Faster R-CNN is more reliable for applications demanding high precision, YOLO provides a feasible solution for quick detections. These findings underscore the importance of selecting appropriate object detection models based on specific requirements of speed versus accuracy in visual computing applications.","metadata":{}},{"cell_type":"markdown","source":"# Faster RCNN","metadata":{}},{"cell_type":"markdown","source":"## Method","metadata":{}},{"cell_type":"markdown","source":"### Importing the Libraries","metadata":{}},{"cell_type":"markdown","source":"Importing all required libraries for this model","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport cv2\nimport os\nimport re\n\nfrom PIL import Image\n\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\n\nimport torch\nimport torchvision\n\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection import FasterRCNN\nfrom torchvision.models.detection.rpn import AnchorGenerator\n\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.utils.data.sampler import SequentialSampler\n\nfrom matplotlib import pyplot as plt\n\nfrom warnings import filterwarnings\nfilterwarnings('ignore')\n\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:03:54.775329Z","iopub.execute_input":"2024-03-28T21:03:54.775599Z","iopub.status.idle":"2024-03-28T21:03:54.784567Z","shell.execute_reply.started":"2024-03-28T21:03:54.775575Z","shell.execute_reply":"2024-03-28T21:03:54.783465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Defining the paths for fetching the dataset and pretrained weights[3].","metadata":{}},{"cell_type":"code","source":"DIR_INPUT = '/kaggle/input/global-wheat-detection'\nDIR_TRAIN = f'{DIR_INPUT}/train'\nDIR_TEST = f'{DIR_INPUT}/test'\nDIR_SUBMISSION = f'{DIR_INPUT}/sample_submission.csv'\nTRAIN_DATASET = f'{DIR_INPUT}/train.csv'\n\nDIR_WEIGHTS = '/kaggle/input/fastercnn'\nWEIGHTS_FILE = f'{DIR_WEIGHTS}/fasterrcnn_resnet50_fpn_weights.pth'","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:06.339831Z","iopub.execute_input":"2024-03-28T20:30:06.340256Z","iopub.status.idle":"2024-03-28T20:30:06.344988Z","shell.execute_reply.started":"2024-03-28T20:30:06.340230Z","shell.execute_reply":"2024-03-28T20:30:06.344073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv(f'{DIR_INPUT}/sample_submission.csv')\ntest_df","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:11.871681Z","iopub.execute_input":"2024-03-28T20:30:11.872526Z","iopub.status.idle":"2024-03-28T20:30:11.896760Z","shell.execute_reply.started":"2024-03-28T20:30:11.872491Z","shell.execute_reply":"2024-03-28T20:30:11.895896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Creating a WheatTestDataset class for the getting a specific image, apply any transformation if specified and return the image id with the image. It initializes the dataset with a DataFrame containing image IDs, a directory where images are stored, and optional transformations. The \"getitem\" method retrieves a specified image by its index, applies any given transformations (such as normalization and color adjustments), and returns the transformed image along with its ID. The \"len\" method returns the total number of unique images in the dataset.","metadata":{}},{"cell_type":"code","source":"class WheatTestDataset(Dataset):\n\n    def __init__(self, dataframe, image_dir, transforms=None):\n        super().__init__()\n\n        self.image_ids = dataframe['image_id'].unique()\n        self.df = dataframe\n        self.image_dir = image_dir\n        self.transforms = transforms\n\n    def __getitem__(self, index: int):\n\n        image_id = self.image_ids[index]\n        records = self.df[self.df['image_id'] == image_id]\n\n        image = cv2.imread(f'{self.image_dir}/{image_id}.jpg', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n\n        if self.transforms:\n            sample = {\n                'image': image,\n            }\n            sample = self.transforms(**sample)\n            image = sample['image']\n\n        return image, image_id\n\n    def __len__(self) -> int:\n        return self.image_ids.shape[0]","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:14.312172Z","iopub.execute_input":"2024-03-28T20:30:14.312764Z","iopub.status.idle":"2024-03-28T20:30:14.320824Z","shell.execute_reply.started":"2024-03-28T20:30:14.312730Z","shell.execute_reply":"2024-03-28T20:30:14.319978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The below functions, **get_train_transform**, **get_valid_transform** and **get_test_transform**, define and return specific image transformation pipelines for training, validation and test datasets, respectively. The use of **bbox_params** indicates that these transformations also apply to bounding boxes in the Pascal VOC format, preserving the object labels during the process.","metadata":{}},{"cell_type":"code","source":"def get_train_transform():\n    return A.Compose([\n        A.Flip(0.5),\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\ndef get_valid_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\ndef get_test_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ])","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:15.471496Z","iopub.execute_input":"2024-03-28T20:30:15.472351Z","iopub.status.idle":"2024-03-28T20:30:15.478424Z","shell.execute_reply.started":"2024-03-28T20:30:15.472317Z","shell.execute_reply":"2024-03-28T20:30:15.477258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Method Selection and Configuration","metadata":{}},{"cell_type":"markdown","source":"The model chosen for this task is the Faster R-CNN with a ResNet-50 backbone and Feature Pyramid Network (FPN), as provided by the torchvision.models.detection module in PyTorch.\n\nThis combination is selected due to Faster R-CNN's efficiency and accuracy in object detection, the ResNet-50's capability to learn deep features through its residual learning framework, and the FPN's ability to detect objects across different scales effectively.","metadata":{}},{"cell_type":"code","source":"# load a model; pre-trained on COCO\nmodel = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:16.757496Z","iopub.execute_input":"2024-03-28T20:30:16.758194Z","iopub.status.idle":"2024-03-28T20:30:17.547374Z","shell.execute_reply.started":"2024-03-28T20:30:16.758163Z","shell.execute_reply":"2024-03-28T20:30:17.546319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Faster R-CNN model with a ResNet-50 backbone and Feature Pyramid Network (FPN) is configured without the default pre-trained weights (pretrained=False and pretrained_backbone=False) because it will be loaded with a custom set of weights that have been specifically pre-trained for wheat detection. This decision is made to leverage a specialized training that is more aligned with the nuances and specific requirements of wheat detection tasks, rather than utilizing the generic object detection capabilities provided by the default COCO dataset pre-training [2]. This approach aims to enhance model performance by using a pre-trained model that is already somewhat tailored to the characteristics of the target application, potentially leading to better accuracy and efficiency in detecting wheat in images.","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\nnum_classes = 2  # 1 class (wheat) + background\n\n# get number of input features for the classifier\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n\n# replace the pre-trained head with a new one\nmodel.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\n\n# Load the trained weights\nmodel.load_state_dict(torch.load(WEIGHTS_FILE, map_location=device))\nmodel.eval()\n\nx = model.to(device)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:18.164271Z","iopub.execute_input":"2024-03-28T20:30:18.164610Z","iopub.status.idle":"2024-03-28T20:30:20.651344Z","shell.execute_reply.started":"2024-03-28T20:30:18.164578Z","shell.execute_reply":"2024-03-28T20:30:20.650549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Using Pseudo Labelling for  improving accuracy and performance of model","metadata":{}},{"cell_type":"markdown","source":"Rather than solely relying on the training dataset to train the Fast R-CNN model with a ResNet-50 FPN backbone from the Torch package, I plan to enhance the model's efficiency by adopting pseudo labeling. This approach boosted the accuracy from 0.6687 to 0.6937 [2], demonstrating a significant improvement in performance.","metadata":{}},{"cell_type":"code","source":"def collate_fn(batch):\n    return tuple(zip(*batch))\n\ntest_dataset = WheatTestDataset(test_df, DIR_TEST, get_test_transform())\n\ntest_data_loader = DataLoader(\n    test_dataset,\n    batch_size=4,\n    shuffle=False,\n    num_workers=4,\n    drop_last=False,\n    collate_fn=collate_fn\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:20.652758Z","iopub.execute_input":"2024-03-28T20:30:20.653049Z","iopub.status.idle":"2024-03-28T20:30:20.662299Z","shell.execute_reply.started":"2024-03-28T20:30:20.653024Z","shell.execute_reply":"2024-03-28T20:30:20.661406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"detection_threshold = 0.5\nresults = []\n\ntestdf_psuedo = []\nfor images, image_ids in test_data_loader:\n\n    images = list(image.to(device) for image in images)\n    outputs = model(images)\n\n    for i, image in enumerate(images):\n\n        boxes = outputs[i]['boxes'].data.cpu().numpy()\n        scores = outputs[i]['scores'].data.cpu().numpy()\n        \n        boxes = boxes[scores >= detection_threshold].astype(np.int32)\n        scores = scores[scores >= detection_threshold]\n        image_id = image_ids[i]\n        \n        boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n        boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n        \n        for box in boxes:\n            #print(box)\n            result = {\n                'image_id': 'pseudo'+image_id,\n                'width': 1024,\n                'height': 1024,\n                'source': 'pseudo',\n                'x': box[0],\n                'y': box[1],\n                'w': box[2],\n                'h': box[3]\n            }\n            testdf_psuedo.append(result)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:20.663255Z","iopub.execute_input":"2024-03-28T20:30:20.663539Z","iopub.status.idle":"2024-03-28T20:30:23.131714Z","shell.execute_reply.started":"2024-03-28T20:30:20.663516Z","shell.execute_reply":"2024-03-28T20:30:23.130645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_pseudo = pd.DataFrame(testdf_psuedo, columns=['image_id', 'width', 'height', 'source', 'x', 'y', 'w', 'h'])\ntest_df_pseudo.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:23.133582Z","iopub.execute_input":"2024-03-28T20:30:23.133873Z","iopub.status.idle":"2024-03-28T20:30:23.160305Z","shell.execute_reply.started":"2024-03-28T20:30:23.133847Z","shell.execute_reply":"2024-03-28T20:30:23.159452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"In this data preprocessing step, bounding box coordinates from the 'bbox' column of the training data are extracted and transformed. Here we are initializing new columns for the bounding box dimensions ('x', 'y', 'w', 'h') with placeholder values. \n\nAn auxiliary function expand_bbox parses these dimensions from the **'bbox'** strings using regular expressions. The parsed numerical values are then assigned to their respective columns, converting them from string to float for subsequent computational use. The original 'bbox' column is discarded, resulting in a cleaned and structured dataframe ready for the object detection model's training phase.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(f'{DIR_INPUT}/train.csv')\ntrain_df['x'] = -1\ntrain_df['y'] = -1\ntrain_df['w'] = -1\ntrain_df['h'] = -1\n\ndef expand_bbox(x):\n    r = np.array(re.findall(\"([0-9]+[.]?[0-9]*)\", x))\n    if len(r) == 0:\n        r = [-1, -1, -1, -1]\n    return r\n\ntrain_df[['x', 'y', 'w', 'h']] = np.stack(train_df['bbox'].apply(lambda x: expand_bbox(x)))\ntrain_df.drop(columns=['bbox'], inplace=True)\ntrain_df['x'] = train_df['x'].astype(np.float64)\ntrain_df['y'] = train_df['y'].astype(np.float64)\ntrain_df['w'] = train_df['w'].astype(np.float64)\ntrain_df['h'] = train_df['h'].astype(np.float64)\n\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:27.218008Z","iopub.execute_input":"2024-03-28T20:30:27.218374Z","iopub.status.idle":"2024-03-28T20:30:28.893779Z","shell.execute_reply.started":"2024-03-28T20:30:27.218344Z","shell.execute_reply":"2024-03-28T20:30:28.892823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this step, we're setting up the training and validation datasets, as well as augmenting the training data by incorporating pseudo-labeled data. This process enriches the training set, offering an expanded volume of data for the model's learning process.","metadata":{}},{"cell_type":"code","source":"image_ids = train_df['image_id'].unique()\nvalid_ids = image_ids[-665:]\ntrain_ids = image_ids[:-665]\n\nvalid_df = train_df[train_df['image_id'].isin(valid_ids)]\ntrain_df = train_df[train_df['image_id'].isin(train_ids)]\n\nframes = [train_df, test_df_pseudo]\n\ntrain_df = pd.concat(frames)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:28.895801Z","iopub.execute_input":"2024-03-28T20:30:28.896413Z","iopub.status.idle":"2024-03-28T20:30:28.957143Z","shell.execute_reply.started":"2024-03-28T20:30:28.896380Z","shell.execute_reply":"2024-03-28T20:30:28.956298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.tail()","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:28.958216Z","iopub.execute_input":"2024-03-28T20:30:28.958479Z","iopub.status.idle":"2024-03-28T20:30:28.971852Z","shell.execute_reply.started":"2024-03-28T20:30:28.958456Z","shell.execute_reply":"2024-03-28T20:30:28.970959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Viewing the shape of the Training and Validation data.","metadata":{}},{"cell_type":"code","source":"valid_df.shape, train_df.shape","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:29.394257Z","iopub.execute_input":"2024-03-28T20:30:29.395004Z","iopub.status.idle":"2024-03-28T20:30:29.401076Z","shell.execute_reply.started":"2024-03-28T20:30:29.394972Z","shell.execute_reply":"2024-03-28T20:30:29.400057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training and Validation","metadata":{}},{"cell_type":"markdown","source":"The WheatDataset class is initialized with a DataFrame containing image IDs and bounding box annotations, the directory where images are stored, and optional transform functions. When retrieving an item from the dataset, it reads an image based on its index, applies any specified transformations, and constructs a target dictionary with bounding box coordinates, labels, and other metadata. ","metadata":{}},{"cell_type":"code","source":"class WheatDataset(Dataset):\n\n    def __init__(self, dataframe, image_dir, transforms=None):\n        super().__init__()\n\n        self.image_ids = dataframe['image_id'].unique()\n        self.df = dataframe\n        self.image_dir = image_dir\n        self.transforms = transforms\n\n    def __getitem__(self, index: int):\n\n        image_id = self.image_ids[index]\n        records = self.df[self.df['image_id'] == image_id]\n        if 'pseudo' in image_id:\n            image_id = image_id[6:]\n            image = cv2.imread(f'{DIR_TEST}/{image_id}.jpg', cv2.IMREAD_COLOR)\n        else:\n            image = cv2.imread(f'{self.image_dir}/{image_id}.jpg', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n\n        boxes = records[['x', 'y', 'w', 'h']].values\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        \n        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n        area = torch.as_tensor(area, dtype=torch.float32)\n\n        # there is only one class\n        labels = torch.ones((records.shape[0],), dtype=torch.int64)\n        \n        # suppose all instances are not crowd\n        iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)\n        \n        target = {}\n        target['boxes'] = boxes\n        target['labels'] = labels\n        # target['masks'] = None\n        target['image_id'] = torch.tensor([index])\n        target['area'] = area\n        target['iscrowd'] = iscrowd\n\n        if self.transforms:\n            sample = {\n                'image': image,\n                'bboxes': target['boxes'],\n                'labels': labels\n            }\n            sample = self.transforms(**sample)\n            image = sample['image']\n            \n            target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n\n        return image, target, image_id\n\n    def __len__(self) -> int:\n        return self.image_ids.shape[0]","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:31.990574Z","iopub.execute_input":"2024-03-28T20:30:31.991311Z","iopub.status.idle":"2024-03-28T20:30:32.005688Z","shell.execute_reply.started":"2024-03-28T20:30:31.991279Z","shell.execute_reply":"2024-03-28T20:30:32.004759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Averager class tracks the running average of values sent to it, useful for monitoring metrics like loss during training. The collate_fn function is provided for batching, preparing the batched data to be processed by a PyTorch DataLoader. \n\nFinally, training and validation datasets are created from respective DataFrames and directory paths using defined transformation functions, ready to be used for training and validating a model for wheat head detection.","metadata":{}},{"cell_type":"code","source":"class Averager:\n    def __init__(self):\n        self.current_total = 0.0\n        self.iterations = 0.0\n\n    def send(self, value):\n        self.current_total += value\n        self.iterations += 1\n\n    @property\n    def value(self):\n        if self.iterations == 0:\n            return 0\n        else:\n            return 1.0 * self.current_total / self.iterations\n\n    def reset(self):\n        self.current_total = 0.0\n        self.iterations = 0.0\n        \n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n\ntrain_dataset = WheatDataset(train_df, DIR_TRAIN, get_train_transform())\nvalid_dataset = WheatDataset(valid_df, DIR_TRAIN, get_valid_transform())","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:33.061453Z","iopub.execute_input":"2024-03-28T20:30:33.061819Z","iopub.status.idle":"2024-03-28T20:30:33.081453Z","shell.execute_reply.started":"2024-03-28T20:30:33.061790Z","shell.execute_reply":"2024-03-28T20:30:33.080498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The below code sets up DataLoaders for the training and validation datasets. It retrieves a batch of images and their corresponding target data, like bounding boxes, from the training DataLoader.","metadata":{}},{"cell_type":"code","source":"# split the dataset in train and test set\nindices = torch.randperm(len(train_dataset)).tolist()\n\ntrain_data_loader = DataLoader(\n    train_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)\n\nvalid_data_loader = DataLoader(\n    valid_dataset,\n    batch_size=4,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:36.709380Z","iopub.execute_input":"2024-03-28T20:30:36.710106Z","iopub.status.idle":"2024-03-28T20:30:36.716440Z","shell.execute_reply.started":"2024-03-28T20:30:36.710065Z","shell.execute_reply":"2024-03-28T20:30:36.715666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The images and target data are transferred to the computational device (like a GPU). \n\nThen, we are visualizing 6 images with their bounding boxes plotted on them, showing the regions where the model predict wheats are located. The bounding box are drawn on the images using OpenCV, and the images are displayed in a grid layout using Matplotlib.","metadata":{}},{"cell_type":"code","source":"images, targets, image_ids = next(iter(train_data_loader))\nimages = list(image.to(device) for image in images)\ntargets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n# Number of images you want to plot\nnum_rows = 2\nnum_cols = 3\nnum_images = num_rows * num_cols\nfig, axs = plt.subplots(num_rows, num_cols, figsize=(12, 8))\n\nfor idx in range(num_images):\n    boxes = targets[idx]['boxes'].cpu().numpy().astype(np.int32)\n    sample = images[idx].permute(1, 2, 0).cpu().numpy()\n    \n    for box in boxes:\n        cv2.rectangle(sample,\n                      (box[0], box[1]),\n                      (box[2], box[3]),\n                      (220, 0, 0), 3)\n    \n    row = idx // num_cols\n    col = idx % num_cols\n    \n    axs[row, col].set_axis_off()\n    axs[row, col].imshow(sample)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:30:39.770281Z","iopub.execute_input":"2024-03-28T20:30:39.770616Z","iopub.status.idle":"2024-03-28T20:30:43.586486Z","shell.execute_reply.started":"2024-03-28T20:30:39.770584Z","shell.execute_reply":"2024-03-28T20:30:43.585491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the below code, object detection model training is done for 2 epochs. The model is set to training mode, moving it to the computation device, and initializes an Stochastic Gradient Descent (SGD) optimizer with parameters that require gradients, along with a learning rate scheduler that reduces the learning rate after each epoch. \n\nDuring training, the loss for each batch of images and targets is computed and the model's weights are updated using backpropagation. ","metadata":{}},{"cell_type":"code","source":"model.train()\nmodel.to(device)\nparams = [p for p in model.parameters() if p.requires_grad]\noptimizer = torch.optim.SGD(params, lr=0.01, momentum=0.9, weight_decay=0.0001)\nlr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.5)\n\nnum_epochs = 2\n\nloss_hist = Averager()\nitr = 1\n\nfor epoch in range(num_epochs):\n    loss_hist.reset()\n    for images, targets, image_ids in train_data_loader:\n        \n        images = list(image.to(device) for image in images)\n        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n        loss_dict = model(images, targets)\n\n        losses = sum(loss for loss in loss_dict.values())\n        loss_value = losses.item()\n\n        loss_hist.send(loss_value)\n\n        optimizer.zero_grad()\n        losses.backward()\n\n        optimizer.step()\n\n        if itr % 50 == 0:\n            print(f\"Iteration #{itr} loss: {loss_value}\")\n\n        itr += 1\n    \n    # update the learning rate\n    if lr_scheduler is not None:\n        lr_scheduler.step()\n\n    print(f\"Epoch #{epoch} loss: {loss_hist.value}\")\n    ","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:31:18.416418Z","iopub.execute_input":"2024-03-28T20:31:18.417267Z","iopub.status.idle":"2024-03-28T20:42:33.829262Z","shell.execute_reply.started":"2024-03-28T20:31:18.417232Z","shell.execute_reply":"2024-03-28T20:42:33.828008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Result","metadata":{}},{"cell_type":"markdown","source":"Now in the below code we are visualizing the validation images with the bounding boxes [4].","metadata":{}},{"cell_type":"code","source":"images, targets, image_ids = next(iter(valid_data_loader))\nimages = list(img.to(device) for img in images)\ntargets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n# Number of images you want to plot\nnum_rows = 2\nnum_cols = 2\nnum_images = num_rows * num_cols\nfig, axs = plt.subplots(num_rows, num_cols, figsize=(12, 8))\n\nfor idx in range(num_images):\n    boxes = targets[idx]['boxes'].cpu().numpy().astype(np.int32)\n    sample = images[idx].permute(1, 2, 0).cpu().numpy()\n    \n    for box in boxes:\n        cv2.rectangle(sample,\n                      (box[0], box[1]),\n                      (box[2], box[3]),\n                      (220, 0, 0), 3)\n    \n    row = idx // num_cols\n    col = idx % num_cols\n    \n    axs[row, col].set_axis_off()\n    axs[row, col].imshow(sample)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:43:04.256874Z","iopub.execute_input":"2024-03-28T20:43:04.257754Z","iopub.status.idle":"2024-03-28T20:43:06.413558Z","shell.execute_reply.started":"2024-03-28T20:43:04.257721Z","shell.execute_reply":"2024-03-28T20:43:06.412604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the below code we are calculating the **Intersection over Union (IoU)** and **mean Average Precision (mAP)** for the validation data using for the model.","metadata":{}},{"cell_type":"code","source":"def calculate_iou(boxA, boxB):\n    \"\"\"Calculate the Intersection over Union (IoU) of two bounding boxes.\"\"\"\n    xA = max(boxA[0], boxB[0])\n    yA = max(boxA[1], boxB[1])\n    xB = min(boxA[2], boxB[2])\n    yB = min(boxA[3], boxB[3])\n\n    interArea = max(0, xB - xA) * max(0, yB - yA)\n    boxAArea = (boxA[2] - boxA[0]) * (boxA[3] - boxA[1])\n    boxBArea = (boxB[2] - boxB[0]) * (boxB[3] - boxB[1])\n\n    iou = interArea / float(boxAArea + boxBArea - interArea)\n    return iou\n\ndef integrated_model_evaluation(model, data_loader, device, iou_threshold=0.5):\n    model.eval()\n    correct_detections, total_objects = 0, 0\n    all_ious = []\n    \n    TP, FP = [], []\n    all_detections = []  # Store [confidence, TP/FP flag]\n    all_ground_truths = 0\n    \n    with torch.no_grad():\n        for images, targets, image_ids in data_loader:\n            images = [img.to(device) for img in images]\n            outputs = model(images)\n            \n            for i, (output, target) in enumerate(zip(outputs, targets)):\n                pred_boxes = output['boxes'].cpu().numpy()\n                scores = output['scores'].cpu().numpy()\n                true_boxes = target['boxes'].cpu().numpy()\n                \n                # For IoU and accuracy calculation\n                for true_box in true_boxes:\n                    ious = [calculate_iou(true_box, pred_box) for pred_box in pred_boxes]\n                    max_iou = max(ious) if ious else 0\n                    all_ious.append(max_iou)\n                    if max_iou >= iou_threshold:\n                        correct_detections += 1\n                    total_objects += 1\n                \n                # For AP calculation\n                for score, pred_box in zip(scores, pred_boxes):\n                    best_iou = max([calculate_iou(pred_box, tb) for tb in true_boxes], default=0)\n                    if best_iou >= iou_threshold:\n                        all_detections.append([score, 1])  # True Positive\n                    else:\n                        all_detections.append([score, 0])  # False Positive\n                \n                all_ground_truths += len(true_boxes)\n    \n    # Calculate accuracy and average IoU\n    accuracy = correct_detections / total_objects if total_objects > 0 else 0\n    average_iou = np.mean(all_ious) if all_ious else 0\n    \n    # Prepare for AP calculation\n    all_detections.sort(key=lambda x: -x[0])  # Sort by descending confidence\n    cum_TP = np.cumsum([det[1] for det in all_detections])\n    cum_FP = np.cumsum([1-det[1] for det in all_detections])\n    precision = cum_TP / (cum_TP + cum_FP)\n    recall = cum_TP / all_ground_truths\n    \n    # Compute AP using the trapezoidal rule to approximate the area under the curve\n    ap = np.trapz(precision, recall)\n    \n    # Print the results\n    print(f\"Accuracy: {accuracy:.4f}\")\n    print(f\"Average IoU: {average_iou:.4f}\")\n    print(f\"mAP: {ap:.4f}\")\n    \n    return accuracy, average_iou, ap","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:43:24.451526Z","iopub.execute_input":"2024-03-28T20:43:24.451828Z","iopub.status.idle":"2024-03-28T20:43:24.467594Z","shell.execute_reply.started":"2024-03-28T20:43:24.451802Z","shell.execute_reply":"2024-03-28T20:43:24.466777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy, average_iou, ap = integrated_model_evaluation(model, valid_data_loader, device, iou_threshold=0.5)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:43:25.387563Z","iopub.execute_input":"2024-03-28T20:43:25.387937Z","iopub.status.idle":"2024-03-28T20:44:21.952476Z","shell.execute_reply.started":"2024-03-28T20:43:25.387891Z","shell.execute_reply":"2024-03-28T20:44:21.951377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Accuracy:** The model correctly identifies positive instances 91.25% of the time.\n\n**Average IoU (Intersection over Union):** The average overlap between the predicted bounding boxes and the ground truth is 72.89%, where a value above 70% indicates a good alignment with the actual objects.\n\n**mAP (mean Average Precision):** The model achieves a mean average precision of 90.92%. A mAP near 91% is typically considered excellent for many applications. This means our model is trained really well.","metadata":{}},{"cell_type":"markdown","source":"The format_prediction_string function constructs a single string for each image's predictions that contains the detection scores and bounding box coordinates","metadata":{}},{"cell_type":"code","source":"def format_prediction_string(boxes, scores):\n    pred_strings = []\n    for j in zip(scores, boxes):\n        pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], j[1][2], j[1][3]))\n\n    return \" \".join(pred_strings)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:45:35.918599Z","iopub.execute_input":"2024-03-28T20:45:35.919073Z","iopub.status.idle":"2024-03-28T20:45:35.926339Z","shell.execute_reply.started":"2024-03-28T20:45:35.919031Z","shell.execute_reply":"2024-03-28T20:45:35.925199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Evaluating on test dataset","metadata":{}},{"cell_type":"markdown","source":"Using the model to identify the wheat for the test images. The result is then stored into a dataframe.","metadata":{}},{"cell_type":"code","source":"model.eval()\n\nfor images, image_ids in test_data_loader:\n\n    images = list(image.to(device) for image in images)\n    outputs = model(images)\n\n    for i, image in enumerate(images):\n\n        boxes = outputs[i]['boxes'].data.cpu().numpy()\n        scores = outputs[i]['scores'].data.cpu().numpy()\n        \n        boxes = boxes[scores >= detection_threshold].astype(np.int32)\n        scores = scores[scores >= detection_threshold]\n        image_id = image_ids[i]\n        \n        boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n        boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n        \n        \n            \n        result = {\n            'image_id': image_id,\n            'PredictionString': format_prediction_string(boxes, scores)\n        }\n\n        \n        results.append(result)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:45:53.351889Z","iopub.execute_input":"2024-03-28T20:45:53.352705Z","iopub.status.idle":"2024-03-28T20:45:54.264008Z","shell.execute_reply.started":"2024-03-28T20:45:53.352673Z","shell.execute_reply":"2024-03-28T20:45:54.262520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.DataFrame(results, columns=['image_id', 'PredictionString'])\ntest_df","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:45:54.266586Z","iopub.execute_input":"2024-03-28T20:45:54.267929Z","iopub.status.idle":"2024-03-28T20:45:54.289760Z","shell.execute_reply.started":"2024-03-28T20:45:54.267867Z","shell.execute_reply":"2024-03-28T20:45:54.288952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The below code is used to view the test data images with bounding box.","metadata":{}},{"cell_type":"code","source":"sample = images[1].permute(1,2,0).cpu().numpy()\nboxes = outputs[1]['boxes'].data.cpu().numpy()\nscores = outputs[1]['scores'].data.cpu().numpy()\n\nboxes = boxes[scores >= detection_threshold].astype(np.int32)\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nfor box in boxes:\n    cv2.rectangle(sample,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (220, 0, 0), 3)\n    \nax.set_axis_off()\nax.imshow(sample)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:45:54.290895Z","iopub.execute_input":"2024-03-28T20:45:54.291204Z","iopub.status.idle":"2024-03-28T20:45:54.947073Z","shell.execute_reply.started":"2024-03-28T20:45:54.291180Z","shell.execute_reply":"2024-03-28T20:45:54.946243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The contents of test_df containing the test results is written to a CSV file named 'Fast_RCNN_Output.csv'. ","metadata":{}},{"cell_type":"code","source":"test_df.to_csv('Fast_RCNN_Output.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:46:11.238558Z","iopub.execute_input":"2024-03-28T20:46:11.239250Z","iopub.status.idle":"2024-03-28T20:46:11.247350Z","shell.execute_reply.started":"2024-03-28T20:46:11.239219Z","shell.execute_reply":"2024-03-28T20:46:11.246405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The trained model's weights to a file named 'fasterrcnn_resnet50_fpn.pth'. ","metadata":{}},{"cell_type":"code","source":"torch.save(model.state_dict(), 'fasterrcnn_resnet50_fpn.pth')","metadata":{"execution":{"iopub.status.busy":"2024-03-28T20:46:11.623993Z","iopub.execute_input":"2024-03-28T20:46:11.624607Z","iopub.status.idle":"2024-03-28T20:46:11.904802Z","shell.execute_reply.started":"2024-03-28T20:46:11.624571Z","shell.execute_reply":"2024-03-28T20:46:11.903809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# YOLO","metadata":{}},{"cell_type":"markdown","source":"## Method Selection and Configuration","metadata":{}},{"cell_type":"markdown","source":"While YOLO outpaces Faster R-CNN in terms of speed, setting up YOLO involves a series of steps for configuration and model initialization. This process includes preparing the environment, adjusting model architecture settings, and configuring training parameters. The following code is inspired from the kaggle notebooks [5, 6].","metadata":{}},{"cell_type":"code","source":"!pip install ultralytics\n!pip install ensemble-boxes","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:03:54.785793Z","iopub.execute_input":"2024-03-28T21:03:54.786465Z","iopub.status.idle":"2024-03-28T21:04:21.202349Z","shell.execute_reply.started":"2024-03-28T21:03:54.786429Z","shell.execute_reply":"2024-03-28T21:04:21.201372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Importing Library\n\nImporting all the necessary libraries for yolo to perfomr the object detection.","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom tqdm.auto import tqdm\nimport shutil as sh\nfrom ultralytics import *\nfrom ensemble_boxes import *\nfrom utils.general import non_max_suppression, scale_boxes\nfrom utils.augmentations import letterbox\nfrom utils.dataloaders import LoadImages","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:21.203628Z","iopub.execute_input":"2024-03-28T21:04:21.203938Z","iopub.status.idle":"2024-03-28T21:04:21.398345Z","shell.execute_reply.started":"2024-03-28T21:04:21.203910Z","shell.execute_reply":"2024-03-28T21:04:21.397635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to do the YOLO setup, we first need to clone the YOLOv5 repository from GitHub to our workplace.","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/ultralytics/yolov5","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:21.399310Z","iopub.execute_input":"2024-03-28T21:04:21.399546Z","iopub.status.idle":"2024-03-28T21:04:24.255085Z","shell.execute_reply.started":"2024-03-28T21:04:21.399525Z","shell.execute_reply":"2024-03-28T21:04:24.254089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After the cloning is done, move all cloned files into the current directory to organize the workspace, remove the now-empty cloned director. This is done to access the file and packages more easily.","metadata":{}},{"cell_type":"code","source":"!mv yolov5/* ./\n!rm -rf yolov5","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:24.257798Z","iopub.execute_input":"2024-03-28T21:04:24.258111Z","iopub.status.idle":"2024-03-28T21:04:26.175757Z","shell.execute_reply.started":"2024-03-28T21:04:24.258083Z","shell.execute_reply":"2024-03-28T21:04:26.174502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally install the required dependencies listed in the requirements.txt file using pip, Python's package installer.","metadata":{}},{"cell_type":"code","source":"!pip install -r requirements.txt","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:26.177187Z","iopub.execute_input":"2024-03-28T21:04:26.177478Z","iopub.status.idle":"2024-03-28T21:04:38.532879Z","shell.execute_reply.started":"2024-03-28T21:04:26.177451Z","shell.execute_reply":"2024-03-28T21:04:38.531682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Verifing model setup for YOLO","metadata":{}},{"cell_type":"markdown","source":"Here we are loading a pre-trained **YOLOv5s** model from the Ultralytics repository using PyTorch. It then performs inference on an image that is fetched from a URL. After the inference, it prints out the results, which typically include the classes, confidence scores, and bounding box coordinates for the detected objects in the image.","metadata":{}},{"cell_type":"code","source":"# Model\nmodel = torch.hub.load(\"ultralytics/yolov5\", \"yolov5s\")  # or yolov5n - yolov5x6, custom\n\n# Images\nimg = \"https://ultralytics.com/images/zidane.jpg\"  # or file, Path, PIL, OpenCV, numpy, list\n\n# Inference\nresults = model(img)\n\nresults.print()","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:39.773144Z","iopub.execute_input":"2024-03-28T21:04:39.773420Z","iopub.status.idle":"2024-03-28T21:04:45.134564Z","shell.execute_reply.started":"2024-03-28T21:04:39.773396Z","shell.execute_reply":"2024-03-28T21:04:45.133577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also view the image with appropriate labelling.","metadata":{}},{"cell_type":"code","source":"# Results\nresults.show() ","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:45.135594Z","iopub.execute_input":"2024-03-28T21:04:45.136031Z","iopub.status.idle":"2024-03-28T21:04:45.641927Z","shell.execute_reply.started":"2024-03-28T21:04:45.136005Z","shell.execute_reply":"2024-03-28T21:04:45.640611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This shows that the model setup is successfully completed.","metadata":{}},{"cell_type":"markdown","source":"## Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"We cannot directly get the data from the train.csv and feed it to the YOLO model. The existing csv data needs to be modified such that it is accepted by the model.\n\n","metadata":{}},{"cell_type":"markdown","source":"Here we are converting the 'bbox' column, which has string representations of lists, into a NumPy array of numerical values. It then extracts and assigns the bounding box coordinates to new columns ('x', 'y', 'w', 'h'), calculates the centers of the bounding boxes ('x_center', 'y_center'), assigns a class label ('classes'), and retains only the relevant columns. ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/global-wheat-detection/train.csv')\nbboxs = np.stack(df['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=',')))\nfor i, column in enumerate(['x', 'y', 'w', 'h']):\n    df[column] = bboxs[:,i]\ndf.drop(columns=['bbox'], inplace=True)\ndf['x_center'] = df['x'] + df['w']/2\ndf['y_center'] = df['y'] + df['h']/2\ndf['classes'] = 0\nfrom tqdm.auto import tqdm\nimport shutil as sh\ndf = df[['image_id','x', 'y', 'w', 'h','x_center','y_center','classes']]\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:45.643135Z","iopub.execute_input":"2024-03-28T21:04:45.643464Z","iopub.status.idle":"2024-03-28T21:04:46.660129Z","shell.execute_reply.started":"2024-03-28T21:04:45.643437Z","shell.execute_reply":"2024-03-28T21:04:46.659176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then from the dataframe, we are getting the all the unique image id and storing it in the index variable.","metadata":{}},{"cell_type":"code","source":"index = list(set(df.image_id))","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:46.661223Z","iopub.execute_input":"2024-03-28T21:04:46.661542Z","iopub.status.idle":"2024-03-28T21:04:46.681568Z","shell.execute_reply.started":"2024-03-28T21:04:46.661518Z","shell.execute_reply":"2024-03-28T21:04:46.680702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following script processes bounding box information for each image, organizing it into a directory named 'labels.' Additionally, 20% of the data is designated for validation while the rest is allocated for training the YOLO model. Two main directories, 'labels' and 'images,' are created, each containing subdirectories 'train2024' and 'val2024' to separate the training and validation datasets. All this is compiled within a parent directory called convertor.","metadata":{}},{"cell_type":"code","source":"source = 'train'\nval_index = index[:len(index)//5]\nfor name,mini in tqdm(df.groupby('image_id')):\n    if name in val_index:\n        path2save = 'val2024/'\n    else:\n        path2save = 'train2024/'\n    if not os.path.exists(f'convertor/labels/{path2save}'):\n        os.makedirs(f'convertor/labels/{path2save}')\n    with open(f'convertor/labels/{path2save}{name}.txt', 'w+') as f:\n        row = mini[['classes','x_center','y_center','w','h']].astype(float).values\n        row = row/1024\n        row = row.astype(str)\n        for j in range(len(row)):\n            text = ' '.join(row[j])\n            f.write(text)\n            f.write(\"\\n\")\n    if not os.path.exists(f'convertor/images/{path2save}'):\n        os.makedirs(f'convertor/images/{path2save}')\n    sh.copy(f\"../input/global-wheat-detection/{source}/{name}.jpg\", f'convertor/images/{path2save}{name}.jpg')","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:04:53.993647Z","iopub.execute_input":"2024-03-28T21:04:53.994034Z","iopub.status.idle":"2024-03-28T21:05:33.439244Z","shell.execute_reply.started":"2024-03-28T21:04:53.994003Z","shell.execute_reply":"2024-03-28T21:05:33.438228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training and Validation of model","metadata":{}},{"cell_type":"markdown","source":"We utilize the 'wheat.yaml' file to specify the number of classification classes and the paths to the training and validation datasets. The 'yolov5x.yaml' file supplies the configuration settings for the model, and we employ pre-trained weights [5] as a starting point for training.","metadata":{}},{"cell_type":"code","source":"!python train.py --img 1024 --batch 4 --epochs 3 --data ../input/yolov5/wheat.yaml --cfg ../input/yolov5/yolov5x.yaml --name yolov5x_wheat_detection --weights /kaggle/input/yolov5/yolov5x.pt","metadata":{"execution":{"iopub.status.busy":"2024-03-28T21:05:42.621226Z","iopub.execute_input":"2024-03-28T21:05:42.621939Z","iopub.status.idle":"2024-03-28T21:41:47.596959Z","shell.execute_reply.started":"2024-03-28T21:05:42.621905Z","shell.execute_reply":"2024-03-28T21:41:47.595766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The YOLO model offers a more streamlined training process compared to Faster R-CNN. Beyond the initial data preprocessing, no extra steps are required. Simply executing the provided command initiates training, evaluates the model on validation data, and yields performance metrics such as precision, recall, and mAP.","metadata":{}},{"cell_type":"markdown","source":"**Precision (P):** This indicates that when the model predicts an instance of wheat, it is correct about 90.1% of the time.\n\n**Recall (R):** The recall value of 0.846 suggests that out of all actual instances of the class, the model successfully identifies 84.6% of them.\n\n**mAP0.5:** Mean Average Precision (mAP) at an Intersection over Union (IoU) threshold of 0.5 is 0.912. A mAP of 0.912 is excellent and indicates that the model is highly accurate in its predictions at this IoU threshold.","metadata":{}},{"cell_type":"markdown","source":"Also we can clearly see that the results are automatically stored in 'runs/train/yolov5x_wheat_detection' path.","metadata":{}},{"cell_type":"markdown","source":"## Evaluating the Test data","metadata":{}},{"cell_type":"markdown","source":"The format_prediction_string function constructs a single string for each image's predictions that contains the detection scores and bounding box coordinates","metadata":{}},{"cell_type":"code","source":"def format_prediction_string(boxes, scores):\n    pred_strings = []\n    for j in zip(scores, boxes):\n        pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], j[1][2], j[1][3]))\n\n    return \" \".join(pred_strings)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:02:39.332589Z","iopub.execute_input":"2024-03-28T22:02:39.333556Z","iopub.status.idle":"2024-03-28T22:02:39.339850Z","shell.execute_reply.started":"2024-03-28T22:02:39.333518Z","shell.execute_reply":"2024-03-28T22:02:39.338901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using the model to identify the wheat for the test images. The result is then stored into a dataframe.","metadata":{}},{"cell_type":"code","source":"def run_wbf(boxes, scores, image_size=1023, iou_thr=0.5, skip_box_thr=0.7, weights=None):\n    labels = [np.zeros(score.shape[0]) for score in scores]\n    boxes = [box/(image_size) for box in boxes]\n    boxes, scores, labels = weighted_boxes_fusion(boxes, scores, labels, weights=None, iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n    boxes = boxes*(image_size)\n    return boxes, scores, labels\n\ndef TTAImage(image, index):\n    image1 = image.copy()\n    if index==0: \n        rotated_image = cv2.rotate(image1, cv2.ROTATE_90_CLOCKWISE)\n        return rotated_image\n    elif index==1:\n        rotated_image2 = cv2.rotate(image1, cv2.ROTATE_90_CLOCKWISE)\n        rotated_image2 = cv2.rotate(rotated_image2, cv2.ROTATE_90_CLOCKWISE)\n        return rotated_image2\n    elif index==2:\n        rotated_image3 = cv2.rotate(image1, cv2.ROTATE_90_CLOCKWISE)\n        rotated_image3 = cv2.rotate(rotated_image3, cv2.ROTATE_90_CLOCKWISE)\n        rotated_image3 = cv2.rotate(rotated_image3, cv2.ROTATE_90_CLOCKWISE)\n        return rotated_image3\n    elif index == 3:\n        return image1\n    \ndef rotBoxes90(boxes, im_w, im_h):\n    ret_boxes =[]\n    for box in boxes:\n        x1, y1, x2, y2 = box\n        x1, y1, x2, y2 = x1-im_w//2, im_h//2 - y1, x2-im_w//2, im_h//2 - y2\n        x1, y1, x2, y2 = y1, -x1, y2, -x2\n        x1, y1, x2, y2 = int(x1+im_w//2), int(im_h//2 - y1), int(x2+im_w//2), int(im_h//2 - y2)\n        x1a, y1a, x2a, y2a = min(x1, x2), min(y1, y2), max(x1, x2), max(y1, y2)\n        ret_boxes.append([x1a, y1a, x2a, y2a])\n    return np.array(ret_boxes)\n\ndef detect1Image(im0, imgsz, model, device, conf_thres, iou_thres):\n    img = letterbox(im0, new_shape=imgsz)[0]\n    # Convert\n    img = img[:, :, ::-1].transpose(2, 0, 1)  # BGR to RGB, to 3x416x416\n    img = np.ascontiguousarray(img)\n    img = np.ascontiguousarray(img)\n\n\n    img = torch.from_numpy(img).to(device)\n    img =  img.float()  # uint8 to fp16/32\n    img /= 255.0   \n    if img.ndimension() == 3:\n        img = img.unsqueeze(0)\n\n    # Inference\n    pred = model(img, augment=False)[0]\n\n    # Apply NMS\n    pred = non_max_suppression(pred, conf_thres, iou_thres)\n\n    boxes = []\n    scores = []\n    for i, det in enumerate(pred):  # detections per image\n        # save_path = 'draw/' + image_id + '.jpg'\n        if det is not None and len(det):\n            # Rescale boxes from img_size to im0 size\n            det[:, :4] = scale_boxes(img.shape[2:], det[:, :4], im0.shape).round()\n\n            # Write results\n            for *xyxy, conf, cls in det:\n                boxes.append([int(xyxy[0]), int(xyxy[1]), int(xyxy[2]), int(xyxy[3])])\n                scores.append(conf)\n                \n    scores = [score.cpu().numpy() for score in scores]\n    return np.array(boxes), np.array(scores)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:02:50.703880Z","iopub.execute_input":"2024-03-28T22:02:50.704527Z","iopub.status.idle":"2024-03-28T22:02:50.723320Z","shell.execute_reply.started":"2024-03-28T22:02:50.704497Z","shell.execute_reply":"2024-03-28T22:02:50.722371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def detect():\n    source = '../input/global-wheat-detection/test/'\n    weights = '/kaggle/working/runs/train/yolov5x_wheat_detection/weights/best.pt'\n    if not os.path.exists(weights):\n        weights = '../input/yolov5/bestv4.pt'\n    imgsz = 1024\n    conf_thres = 0.5\n    iou_thres = 0.6\n    is_TTA = True\n    \n    imagenames =  os.listdir(source)\n    \n    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\n    # Load model\n    model = torch.load(weights, map_location=device)['model'].float()  # load to FP32\n    model.to(device).eval()\n    \n    dataset = LoadImages(source, img_size=imgsz)\n\n    results = []\n    # img = torch.zeros((1, 3, imgsz, imgsz), device=device)  # init img\n    #for path, img, im0s, _ in dataset:\n    for name in imagenames:\n        image_id = name.split('.')[0]\n        im01 = cv2.imread('%s/%s.jpg'%(source,image_id))  # BGR\n        assert im01 is not None, 'Image Not Found '\n        # Padded resize\n        im_w, im_h = im01.shape[:2]\n        if is_TTA:\n            enboxes = []\n            enscores = []\n            for i in range(4):\n                im0 = TTAImage(im01, i)\n                boxes, scores = detect1Image(im0, imgsz, model, device, conf_thres, iou_thres)\n                for _ in range(3-i):\n                    boxes = rotBoxes90(boxes, im_w, im_h)\n                    \n                if 1: #i<3:\n                    enboxes.append(boxes)\n                    enscores.append(scores) \n            boxes, scores = detect1Image(im01, imgsz, model, device, conf_thres, iou_thres)\n            enboxes.append(boxes)\n            enscores.append(scores)\n\n            boxes, scores, labels = run_wbf(enboxes, enscores, image_size = im_w, iou_thr=0.6, skip_box_thr=0.5)\n            boxes = boxes.astype(np.int32).clip(min=0, max=im_w)\n        else:\n            boxes, scores = detect1Image(im01, imgsz, model, device, conf_thres, iou_thres)\n\n        boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n        boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n        \n        boxes = boxes[scores >= 0.05].astype(np.int32)\n        scores = scores[scores >=float(0.05)]\n\n        result = {\n            'image_id': image_id,\n            'PredictionString': format_prediction_string(boxes, scores)\n        }\n\n        results.append(result)\n    return results","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:02:55.324891Z","iopub.execute_input":"2024-03-28T22:02:55.325271Z","iopub.status.idle":"2024-03-28T22:02:55.339429Z","shell.execute_reply.started":"2024-03-28T22:02:55.325237Z","shell.execute_reply":"2024-03-28T22:02:55.338520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After processing the test data images, we are storing it in the test_df dataframe. This value will later be saved in a csv file.","metadata":{}},{"cell_type":"code","source":"results = detect()\ntest_df = pd.DataFrame(results, columns=['image_id', 'PredictionString'])\ntest_df.to_csv('YOLO_Output.csv', index=False)\ntest_df","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:02:57.366558Z","iopub.execute_input":"2024-03-28T22:02:57.366973Z","iopub.status.idle":"2024-03-28T22:03:01.618965Z","shell.execute_reply.started":"2024-03-28T22:02:57.366944Z","shell.execute_reply":"2024-03-28T22:03:01.618016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to view the test image with bounding box the below code is used.","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport cv2\nimport pandas as pd\n\n# Replace with the path to your images\nimage_folder_path = '/kaggle/input/global-wheat-detection/test'\n\ndef plot_image_with_boxes(image_id, prediction_string):\n    # Read the image\n    image_path = f'{image_folder_path}/{image_id}.jpg'  # Assuming image format is .jpg\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n    # Create a figure and a set of subplots\n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\n    # Display the image\n    ax.imshow(image)\n\n    # Split the prediction string by space\n    prediction_items = prediction_string.split(' ')\n\n    # Each detection has 5 items: score and bbox (x, y, w, h)\n    for i in range(0, len(prediction_items), 5):\n        score = float(prediction_items[i])\n        bbox = [float(x) for x in prediction_items[i+1:i+5]]\n        rect = patches.Rectangle(\n            (bbox[0], bbox[1]), bbox[2], bbox[3],\n            linewidth=1, edgecolor='r', facecolor='none'\n        )\n        ax.add_patch(rect)\n        ax.set_axis_off()\n\n    plt.show()\n\n# Example usage for the first row in your DataFrame\nplot_image_with_boxes(test_df.iloc[1]['image_id'], test_df.iloc[1]['PredictionString'])","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:12:30.629814Z","iopub.execute_input":"2024-03-28T22:12:30.630637Z","iopub.status.idle":"2024-03-28T22:12:31.275869Z","shell.execute_reply.started":"2024-03-28T22:12:30.630605Z","shell.execute_reply":"2024-03-28T22:12:31.274863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The contents of test_df containing the test results is written to a CSV file named 'YOLO_Output.csv'.","metadata":{}},{"cell_type":"code","source":"test_df.to_csv('YOLO_Output.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:07:17.396593Z","iopub.execute_input":"2024-03-28T22:07:17.397548Z","iopub.status.idle":"2024-03-28T22:07:17.403586Z","shell.execute_reply.started":"2024-03-28T22:07:17.397514Z","shell.execute_reply":"2024-03-28T22:07:17.402736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The trained model's weights to a file named 'yolo_best.pth'.","metadata":{}},{"cell_type":"code","source":"torch.save(model.state_dict(), 'yolo_best.pth')","metadata":{"execution":{"iopub.status.busy":"2024-03-28T22:07:30.277593Z","iopub.execute_input":"2024-03-28T22:07:30.278334Z","iopub.status.idle":"2024-03-28T22:07:30.359598Z","shell.execute_reply.started":"2024-03-28T22:07:30.278303Z","shell.execute_reply":"2024-03-28T22:07:30.358826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Results and Discussion","metadata":{}},{"cell_type":"markdown","source":"## Model Performance","metadata":{}},{"cell_type":"markdown","source":"The thorough assessment of these two object detection models, Faster R-CNN and YOLOv5, on Global Wheat Detection produced encouraging findings in the identification of wheat heads from the Global Wheat Detection dataset. Regarding the identification and localization of wheat heads within the photos of agricultural fields, each model showed impressive accuracy.","metadata":{}},{"cell_type":"markdown","source":"**Faster R-CNN:** With its two-stage architecture, Faster R-CNN offers a well-rounded method for detecting wheat heads, providing strong precision and consistent performance, even in the face of occlusions and diverse object dimensions.\n\n**YOLOv5:** Demonstrating exceptional speed and effectiveness in real-time scenarios, YOLOv5 capitalizes on its single-stage design to rapidly analyze images and pinpoint wheat heads with high accuracy.","metadata":{}},{"cell_type":"markdown","source":"|Model|Accuracy|IoU|mAP|\n|-|-|-|-|\n|Faster R-CNN|91.2%|0.72|0.9092|\n|YoloV5|90.1%|0.65|0.912|","metadata":{}},{"cell_type":"markdown","source":"# Recommendation","metadata":{}},{"cell_type":"markdown","source":"For real-time detection scenarios where speed is crucial, YOLOv5 is the recommended approach due to its high frame rate and accuracy. It's particularly suited for applications where computational resources are limited and rapid decisions are necessary. However, if the use case involves crowded scenes with occlusions or requires precise localization of objects, Faster R-CNN might be preferred for its robustness and slightly better precision.","metadata":{}},{"cell_type":"markdown","source":"## Pros and Cons:","metadata":{}},{"cell_type":"markdown","source":"### Faster R-CNN:\n#### **Pros:**\n\n:**Higher Precision::** As indicated by its higher IoU of 0.72, it may provide more accurate bounding box predictions compared to YOLOv5.\n\n:**Robustness::** It's generally more robust to occlusions and better at handling various object sizes, which is important in complex scenes.\n\n#### **Cons:**\n\n:**Speed::** It tends to be slower than YOLOv5 due to the two-stage process, making it less suitable for real-time applications.\n\n:**Computational Demand::** Requires more computational power, which could be a limiting factor in resource-constrained environments.","metadata":{}},{"cell_type":"markdown","source":"### YOLOv5:\n\n#### **Pros:**\n**Speed:** Known for its fast processing times, making it ideal for real-time detection.\n\n**Single-Stage Detection:** Simplifies the training and inference process without the need for a separate region proposal network.\n\n#### **Cons:**\n**Lower IoU:** The lower IoU score of 0.65 suggests that it might not localize objects as precisely as Faster R-CNN.\n\n**Sensitivity to Object Size:** It might not perform as well on very small or very large objects compared to Faster R-CNN.","metadata":{}},{"cell_type":"markdown","source":"# Summary ","metadata":{}},{"cell_type":"markdown","source":"In conclusion, the evaluation of Faster R-CNN and YOLOv5 for wheat head detection has demonstrated that each model has distinct advantages that suit different operational scenarios. Faster R-CNN excels in precision and robustness, making it suitable for analytical scenarios where accuracy in object localization is paramount, despite its slower processing time. On the other hand, YOLOv5’s strength lies in its speed and efficiency, making it ideal for real-time applications where rapid detection is critical, even though it may sacrifice some localization accuracy as indicated by its lower IoU. The choice between these models should be guided by the specific requirements of the task at hand, balancing the trade-offs between speed and precision to meet the end goals of the application. Future work may explore hybrid approaches or customizations to each model to further enhance performance according to the needs of technology advancements.","metadata":{}},{"cell_type":"markdown","source":"# References","metadata":{}},{"cell_type":"markdown","source":"[1] Global Wheat Detection: https://www.kaggle.com/c/global-wheat-detection/data?select=train <br>\n[2] FasterRCNN Pseudo Labeling: https://www.kaggle.com/code/nvnnghia/fasterrcnn-pseudo-labeling <br>\n[3] Global Wheat Detection - Public: https://www.kaggle.com/datasets/7d5f1ed9454c848ecb909c109c6fa8e573ea4de299e249c79edc6f47660bf4c5?select=fasterrcnn_resnet50_fpn_best.pth <br>\n[4] Pytorch Starter - FasterRCNN Inference: https://www.kaggle.com/code/pestipeti/pytorch-starter-fasterrcnn-inference <br>\n[5] YoloV5 Pseudo Labeling: https://www.kaggle.com/code/nvnnghia/yolov5-pseudo-labeling/notebook <br>\n[6] YoloV5 [train]: https://www.kaggle.com/code/orkatz2/yolov5-train <br>\n[7] ToLO github: https://github.com/ultralytics/yolov5","metadata":{}}]}