{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom matplotlib import pyplot as plt\n\n#from tqdm import tqdm_notebook as tqdm\n#from tqdm import tqdm \nfrom tqdm.notebook import tqdm as tqdm\n\nimport cv2\nimport os\nimport re\n\nimport random\n\nfrom PIL import Image\n\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\n\nimport torch\nimport torchvision\n\nimport ast\n\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection import FasterRCNN\nfrom torchvision.models.detection.rpn import AnchorGenerator\n\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.utils.data.sampler import SequentialSampler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Deep Learning Detection and Classification of Wheat Heads to Estimate Yield"},{"metadata":{},"cell_type":"markdown","source":"Please find notebooks for data EDA and for both models on my kaggle page: https://www.kaggle.com/ottiliemitchell/notebooks"},{"metadata":{},"cell_type":"markdown","source":"## Background\nWheat is a core cereal that is relied on around the world in our diet. In the 1920s wheat breeding was introduced to increase yields and to build immunity against disease (David et al., 2020). \n\nWheat's popularity as a staple food product, has meant that farmers globally have had to grow different varieties to suit their growing conditions. This is  vital for food security to ensure successful harvests. Breeding has significantly increased yields, however, this rate of increase has plateaued since the early 1990s (Brisson et al., 2010). Farmers now need to look at other ways to increase and estimate wheat yields. \n\nDemand for high yielding wheat has initiated wide research, with the creation of a large database, including a set of images which will be studied in this investigation. The images are made up of \"wheat heads\" which are the spikes at the top of the plant's stem. The wheat head holds the plant's nutrition and is harvested and processed into cereal for food consumption. It would be of benefit to farmers if they were able to estimate the density and size of the wheat heads to predict wheat yields.\n\nWith the advance in GPU performance, deep learning has become more accessable to individuals as well as institutions to perform object detection on large datasets (Alom et al., 2019). Although deep learning can be an accurate way to look at large datasets, it can come with limitations. As the images are taken of wheat fields outside, images are often distorted or blury, with movement in the wind. Also, different varieties, at different growth stages and the orientation of wheat heads can appear differently. This can make it more challenging to identify single wheat heads.\n\nThe Global Wheat Head data is led by 9 research institutions from across the world, with the joint ambition of accurately detecting wheat heads. These institutions include The University of Tokyo, Institut National de Researche pour L'Agriculture, L'Alimentation et L'Environment, Arvalis, ETHZ, University of Saskatchewan, University of Queensland, Nanjing Agricultural University and Rothamsted Research.\n\n### Problem outline\nThe accurate classification and detection of wheat heads would give farmers an invaluable tool in the prediction of wheat yields. Using the Global Wheat Head data, this study will attempt to successfully identify the location of wheat heads within provided images."},{"metadata":{},"cell_type":"markdown","source":"## Methods\n\n### Data\nThe data set was found on Kaggle (https://www.kaggle.com/c/global-wheat-detection).\n\nThe data includes 2 folders of images, test and train and metadata giving further information of the bounding boxes.\n\nThere are 3422 images in the training folder and 10 images in the test folder. Each image is assigned a unique ID number, have the dimension 1024 x 1024 and are made up of 3 layers.\n\nThe metadata is made up of 5 columns and 147794 rows. The columns include:\n* Image_id – the unique ID number given to each image\n* Width – Width of each image\n* Height – Height of each images\n* Bbox – a bounding box including the xmin, ymin, width and height\n* Source – where the images came from."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"DATA_PATH = \"../input/global-wheat-detection/\"\nTRAIN_DIR = \"../input/global-wheat-detection/train\"\nTEST_DIR = \"../input/global-wheat-detection/test\"","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"df = pd.read_csv(os.path.join(DATA_PATH, \"train.csv\"))\ndf.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Detection and Classification\nClassification is simply categorising a stimuli into a finite set of classes. This involves recognising the dominant content in a scene, which is given the strongest confidence score, irrespective of the location, scaling or rotation.\n\nAn example of this would be recognising the dominant feature within an image, such as cat or dog. Classification does not consider where the dominant feature is.\n\nDetection differs from classification, as it involves the classification and localisation of an object (Tuzel et al., 2006). For this study, we would like to be able to determine the wheat heads and their location, thus it is a detection problem.\n\n\n### Data Preprocessing\nThe data needs to be formatted into appropriately pre-processed floating point tensors before being run through the model. \n\nFirstly, the bounding box coordinates must be extracted to create separate columns."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def extract_bbox(DataFrame):\n    DataFrame[\"x\"] = [np.float(ast.literal_eval(i)[0]) for i in DataFrame[\"bbox\"]]\n    DataFrame[\"y\"] = [np.float(ast.literal_eval(i)[1]) for i in DataFrame[\"bbox\"]]\n    DataFrame[\"w\"] = [np.float(ast.literal_eval(i)[2]) for i in DataFrame[\"bbox\"]]\n    DataFrame[\"h\"] = [np.float(ast.literal_eval(i)[3]) for i in DataFrame[\"bbox\"]]\n    \nextract_bbox(df)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images from the training data is then split into train and validation sets using a 80:20 split. Once the data has been split, it is integrated back with the metadata. This can then show how many bounding boxes are in each data set, training and validation.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_split = 0.8\nimages_id   = df[\"image_id\"].unique() \ntrain_ids   = images_id[:int(len(images_id)*train_split)]\nvalid_ids   = images_id[int(len(images_id)*train_split):]\n\nprint(f'Total Images Number: {len(images_id)}')\nprint(f'Number of Training images: {len(train_ids)}')\nprint(f'Number of Validation images: {len(valid_ids)}')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train_df = df[df[\"image_id\"].isin(train_ids)]\nvalid_df = df[df[\"image_id\"].isin(valid_ids)]\n\nprint(f'Shape of train_df: {train_df.shape}')\nprint(f'Shape of valid_df: {valid_df.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#### Image Augmentation\nImage augmentation is where the images are distorted, making it more difficult for the model to detect the wheat heads. It is used in deep learning and computer vision to increase the quality of trained models. In this study a Python library Albumentation was used for image augmentation. The model is first run without image augmentations and to improve the model it was then run again using the albumentation arguments \"HorizontalFlip\", \"RandomBrightnessContrast\" and \"Blur\"."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Data Transform - Albumentation\ndef get_train_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\ndef get_valid_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class WheatDataset(Dataset):\n    def __init__(self, dataframe, image_dir, transform=None):\n        super().__init__()\n        self.dataframe = dataframe\n        self.image_dir = image_dir\n        self.transform = transform\n        self.image_ids = dataframe[\"image_id\"].unique()\n        \n    def __getitem__(self, idx):\n        #Load images and details\n        image_id = self.image_ids[idx]\n        details = self.dataframe[self.dataframe[\"image_id\"]==image_id]\n        img_path = os.path.join(TRAIN_DIR, image_id)+\".jpg\"\n        image = cv2.imread(img_path, cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        \n        #Row of Dataframe of a particular index.\n        boxes = details[['x', 'y', 'w', 'h']].values\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        \n        #To find area\n        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n        \n        #Convert it into tensor dataType\n        area = torch.as_tensor(area, dtype=torch.float32)\n        \n        # there is only one class\n        labels = torch.ones((details.shape[0],), dtype=torch.int64)\n        \n        # suppose all instances are not crowd\n        iscrowd = torch.zeros((details.shape[0],), dtype=torch.int64)\n        \n        target = {}\n        target['boxes'] = boxes\n        target['labels'] = labels\n        target['image_id'] = torch.tensor(idx) \n        target['area'] = area\n        target['iscrowd'] = iscrowd\n        \n        if self.transform:\n            sample = {\n                'image': image,\n                'bboxes': target['boxes'],\n                'labels': labels\n            }\n            \n            sample = self.transform(**sample)\n            image = sample['image']\n            target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n            target[\"boxes\"] = torch.as_tensor(target[\"boxes\"], dtype=torch.long)\n        \n        return image, target     #, image_id\n    \n    def __len__(self) -> int:\n        return len(self.image_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def collate_fn(batch):\n    return tuple(zip(*batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_dataset = WheatDataset(train_df, TRAIN_DIR, get_train_transform())\nvalid_dataset = WheatDataset(valid_df, TRAIN_DIR, get_valid_transform())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"indices = torch.randperm(len(train_dataset)).tolist()\n\ntrain_data_loader = DataLoader(\n    train_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)\n\nvalid_data_loader = DataLoader(\n    valid_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_images(n_num, random_selection=True):\n\n    if random_selection:\n        index = random.sample(range(0, len(train_df[\"image_id\"].unique())-1), n_num)\n    else:\n        index = range(0, n_num)\n    plt.figure(figsize=(15,15))\n    fig_no = 1\n    \n    for i in index:\n        images, targets = train_dataset.__getitem__(i)\n        sample = np.array(np.transpose(images, (1,2,0)))\n        boxes = targets[\"boxes\"].numpy().astype(np.int32)\n    \n        #Plot figure/image\n\n        for box in boxes:\n            cv2.rectangle(sample,(box[0], box[1]),(box[2], box[3]),(255,223,0), 2)\n        plt.subplot(n_num/2, n_num/2, fig_no)\n        plt.imshow(sample)\n        fig_no+=1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that the data is in the correct form, the data can be run through the model. Here is an example of four of the training images with bounding boxes."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#train images\nplot_images(4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model\nThe model used for this study was a pretrained Faster R-CNN ResNet50. This model extends the Faster R-CNN, whilst only adding a small overhead, running at 5 fps. Additionally, this model outperforms all existing single-model entries for segmentation, bounding-box object detection and person keypoint detection.\n\nThe model was firstly trained on a model with no image augmentation and 20 epochs. This was then improved by adding several image augmentations and reducing the number of epochs.\n\nThe model needs to be downloaded and the head of the model changed to a more appropriate predictor. The number of classes, input features, device, parameters, optimisers and epochs are also defined. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)\n\nnum_classes = 2  # wheat + background\n# get number of input features for the classifier\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n# replace the pre-trained head with a new one\nmodel.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\ndevice = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nmodel.to(device)\nparams = [p for p in model.parameters() if p.requires_grad]\noptimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)\n# lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)\nlr_scheduler = None\n\nnum_epochs = 20","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model is now ready to run on the training and validation data."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import time\n\nitr=1\n\ntotal_train_loss = []\ntotal_valid_loss = []\n\nlosses_value = 0\nfor epoch in range(num_epochs):\n  \n    start_time = time.time()\n    train_loss = []\n    model.train()\n    \n #<-----------Training Loop---------------------------->\n    pbar = tqdm(train_data_loader, desc = 'description')\n    for images, targets in pbar:\n        \n        images = list(image.to(device) for image in images)\n        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n        loss_dict = model(images, targets)\n        \n        losses = sum(loss for loss in loss_dict.values())\n        losses_value = losses.item()\n        train_loss.append(losses_value)        \n        optimizer.zero_grad()\n        losses.backward()\n        optimizer.step()\n        \n        pbar.set_description(f\"Epoch: {epoch+1}, Batch: {itr}, loss: {losses_value}\")\n        itr+=1\n\n    epoch_train_loss = np.mean(train_loss)\n    total_train_loss.append(epoch_train_loss)\n\n #<---------------Validation Loop---------------------->\n    with torch.no_grad():\n        valid_loss = []\n\n        for images, targets in valid_data_loader:\n            images = list(image.to(device) for image in images)\n            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n            \n            \n            # If you need validation losses\n            model.train()\n            # Calculate validation losses\n            loss_dict = model(images, targets)\n            losses = sum(loss for loss in loss_dict.values())\n            loss_value = losses.item()\n            valid_loss.append(loss_value)\n            \n    epoch_valid_loss = np.mean(valid_loss)\n    total_valid_loss.append(epoch_valid_loss)    \n    \n    print(f\"Epoch Completed: {epoch+1}/{num_epochs}, Time: {time.time()-start_time},\\\n    Train Loss: {epoch_train_loss}, Valid Loss: {epoch_valid_loss}\")  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results "},{"metadata":{},"cell_type":"markdown","source":"The graph shows that the validation loss and training loss diverge exponentially after aproximately 10 epochs. This tells us that the model is overfitting and that using 20 epochs was far too many. The training loss has continued to decrease from 0.94 to 0.62, whilst the validation loss starts to decrease from 0.86 to 0.81, but then increase to 1.03.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(8,5))\nsns.set_style(style=\"whitegrid\")\nsns.lineplot(range(1, len(total_train_loss)+1), total_train_loss, label=\"Training Loss\")\nsns.lineplot(range(1, len(total_train_loss)+1), total_valid_loss, label=\"Valid Loss\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Loss\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the first model was clearly overfitting the data, I reduced the number of epochs to 9 and introduced image augmentation to try and decrease the validation loss further. "},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"DATA_PATH = \"../input/global-wheat-detection/\"\nTRAIN_DIR = \"../input/global-wheat-detection/train\"\nTEST_DIR = \"../input/global-wheat-detection/test\"\ndf = pd.read_csv(os.path.join(DATA_PATH, \"train.csv\"))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def extract_bbox(DataFrame):\n    DataFrame[\"x\"] = [np.float(ast.literal_eval(i)[0]) for i in DataFrame[\"bbox\"]]\n    DataFrame[\"y\"] = [np.float(ast.literal_eval(i)[1]) for i in DataFrame[\"bbox\"]]\n    DataFrame[\"w\"] = [np.float(ast.literal_eval(i)[2]) for i in DataFrame[\"bbox\"]]\n    DataFrame[\"h\"] = [np.float(ast.literal_eval(i)[3]) for i in DataFrame[\"bbox\"]]\n    \nextract_bbox(df)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_split = 0.8\nimages_id   = df[\"image_id\"].unique() \ntrain_ids   = images_id[:int(len(images_id)*train_split)]\nvalid_ids   = images_id[int(len(images_id)*train_split):]\n\ntrain_df = df[df[\"image_id\"].isin(train_ids)]\nvalid_df = df[df[\"image_id\"].isin(valid_ids)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Data Transform - Albumentation\ndef get_train_transform():\n    return A.Compose([\n        A.HorizontalFlip(p=0.5),\n        A.RandomBrightnessContrast(p=0.2),\n        A.Blur(p=1),\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\ndef get_valid_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class WheatDataset(Dataset):\n    def __init__(self, dataframe, image_dir, transform=None):\n        super().__init__()\n        self.dataframe = dataframe\n        self.image_dir = image_dir\n        self.transform = transform\n        self.image_ids = dataframe[\"image_id\"].unique()\n        \n    def __getitem__(self, idx):\n        #Load images and details\n        image_id = self.image_ids[idx]\n        details = self.dataframe[self.dataframe[\"image_id\"]==image_id]\n        img_path = os.path.join(TRAIN_DIR, image_id)+\".jpg\"\n        image = cv2.imread(img_path, cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        \n        #Row of Dataframe of a particular index.\n        boxes = details[['x', 'y', 'w', 'h']].values\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        \n        #To find area\n        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n               \n        #Convert it into tensor dataType\n        area = torch.as_tensor(area, dtype=torch.float32)\n        \n        # there is only one class\n        labels = torch.ones((details.shape[0],), dtype=torch.int64)\n        \n        # suppose all instances are not crowd\n        iscrowd = torch.zeros((details.shape[0],), dtype=torch.int64)\n        \n        target = {}\n        target['boxes'] = boxes\n        target['labels'] = labels\n        target['image_id'] = torch.tensor(idx) ### <------------ New change list has been removed\n        target['area'] = area\n        target['iscrowd'] = iscrowd\n        \n        if self.transform:\n            sample = {\n                'image': image,\n                'bboxes': target['boxes'],\n                'labels': labels\n            }\n            \n            sample = self.transform(**sample)\n            image = sample['image']\n            target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n            target[\"boxes\"] = torch.as_tensor(target[\"boxes\"], dtype=torch.long)\n        \n        return image, target     #, image_id\n    \n    def __len__(self) -> int:\n        return len(self.image_ids)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def collate_fn(batch):\n    return tuple(zip(*batch))\n\ntrain_dataset = WheatDataset(train_df, TRAIN_DIR, get_train_transform())\nvalid_dataset = WheatDataset(valid_df, TRAIN_DIR, get_valid_transform())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"indices = torch.randperm(len(train_dataset)).tolist()\n\ntrain_data_loader = DataLoader(\n    train_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)\n\nvalid_data_loader = DataLoader(\n    valid_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)\nnum_classes = 2  # wheat + background\n# get number of input features for the classifier\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n# replace the pre-trained head with a new one\nmodel.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\ndevice = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nmodel.to(device)\nparams = [p for p in model.parameters() if p.requires_grad]\noptimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)\n# lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)\nlr_scheduler = None\n\nnum_epochs = 9","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import time\n\nitr=1\n\ntotal_train_loss = []\ntotal_valid_loss = []\n\nlosses_value = 0\nfor epoch in range(num_epochs):\n  \n    start_time = time.time()\n    train_loss = []\n    model.train()\n    \n #<-----------Training Loop---------------------------->\n    pbar = tqdm(train_data_loader, desc = 'description')\n    for images, targets in pbar:\n        \n        images = list(image.to(device) for image in images)\n        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n        loss_dict = model(images, targets)\n        \n        losses = sum(loss for loss in loss_dict.values())\n        losses_value = losses.item()\n        train_loss.append(losses_value)        \n        optimizer.zero_grad()\n        losses.backward()\n        optimizer.step()\n        \n        pbar.set_description(f\"Epoch: {epoch+1}, Batch: {itr}, loss: {losses_value}\")\n        itr+=1\n\n    epoch_train_loss = np.mean(train_loss)\n    total_train_loss.append(epoch_train_loss)\n    \n    \n    #<---------------Validation Loop---------------------->\n    with torch.no_grad():\n        valid_loss = []\n\n        for images, targets in valid_data_loader:\n            images = list(image.to(device) for image in images)\n            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n            \n            \n            # If you need validation losses\n            model.train()\n            # Calculate validation losses\n            loss_dict = model(images, targets)\n            losses = sum(loss for loss in loss_dict.values())\n            loss_value = losses.item()\n            valid_loss.append(loss_value)\n            \n    epoch_valid_loss = np.mean(valid_loss)\n    total_valid_loss.append(epoch_valid_loss)\n    \n    print(f\"Epoch Completed: {epoch+1}/{num_epochs}, Time: {time.time()-start_time},\\\n    Train Loss: {epoch_train_loss}, Valid Loss: {epoch_valid_loss}\")   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the graph below you can see that adding in the augmentations has significantly reduced the validation loss of the model to just below 0.825."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(8,5))\nsns.set_style(style=\"whitegrid\")\nsns.lineplot(range(1, len(total_train_loss)+1), total_train_loss, label=\"Training Loss\")\nsns.lineplot(range(1, len(total_train_loss)+1), total_valid_loss, label=\"Valid Loss\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Loss\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Data Transform - Test Albumentation\ndef get_test_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class WheatTestDatasetTest(Dataset):\n    def __init__(self, dataframe, image_dir, transform=None):\n        super().__init__()\n        self.dataframe = dataframe\n        self.image_dir = image_dir\n        self.transform = transform\n        self.image_ids = dataframe[\"image_id\"].unique()\n        \n    def __getitem__(self, idx):\n        image_id = self.image_ids[idx]\n        details = self.dataframe[self.dataframe[\"image_id\"]==image_id]\n        img_path = os.path.join(TEST_DIR, image_id)+\".jpg\"\n        image = cv2.imread(img_path, cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        \n        if self.transform:\n            sample = {\n                'image': image,\n            }\n            \n            sample = self.transform(**sample)\n            image = sample['image'] \n            \n        return image, image_id\n    \n    def __len__(self) -> int:\n        return len(self.image_ids)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv(os.path.join(DATA_PATH, \"sample_submission.csv\"))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def collate_fn(batch):\n    return tuple(zip(*batch))\n\ntest_dataset = WheatTestDatasetTest(df_test, TEST_DIR, get_test_transform())\nprint(f\"Length of test dataset: {len(test_dataset)}\")\n\ntest_data_loader = DataLoader(\n    test_dataset,\n    batch_size=4,\n    shuffle=False,\n    num_workers=4,\n    drop_last=False,\n    collate_fn=collate_fn\n)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model.eval()\nx = model.to(device)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"detection_threshold = 0.5\noutput_list = []\n\nfor images, image_ids in test_data_loader:\n\n    images = list(image.to(device) for image in images)\n    outputs = model(images)\n\n    for i, image in enumerate(images):\n\n        boxes = outputs[i]['boxes'].data.cpu().numpy()\n        scores = outputs[i]['scores'].data.cpu().numpy()\n        \n        boxes = boxes[scores >= detection_threshold].astype(np.int32)\n        scores = scores[scores >= detection_threshold]\n        image_id = image_ids[i]\n        \n        boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n        boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n        \n        output_dict = {\n            'image_id': image_ids[i],\n            'boxes': outputs[i]['boxes'].data.cpu().numpy(),\n            'scores': outputs[i]['scores'].data.cpu().numpy()\n        }\n        output_list.append(output_dict)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"## Plot image prediction\n\ndef predict_images(n_num, random_selection=True):\n    '''Plot N Number of Predicted Images'''\n    if random_selection:\n        index = random.sample(range(0, len(df_test[\"image_id\"].unique())), n_num)\n    else:\n        index = range(0, n_num)\n        \n    plt.figure(figsize=(15,15))\n    fig_no = 1\n    \n    for i in index:\n        images, image_id = test_dataset.__getitem__(i)\n        sample = images.permute(1,2,0).cpu().numpy()\n        boxes = output_list[i]['boxes']\n        scores = output_list[i]['scores']\n        boxes = boxes[scores >= detection_threshold].astype(np.int32)\n        #Plot figure/image\n        for box in boxes:\n            cv2.rectangle(sample,(box[0], box[1]),(box[2], box[3]),(255,223,0), 2)\n        plt.subplot(n_num/2, n_num/2, fig_no)\n        plt.imshow(sample)\n        fig_no+=1","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def print_image(i):\n  plt.figure(figsize=(6,6))\n  images, image_id = test_dataset.__getitem__(i)\n  sample = images.permute(1,2,0).cpu().numpy()\n  boxes = output_list[i]['boxes']\n  scores = output_list[i]['scores']\n  boxes = boxes[scores >= detection_threshold].astype(np.int32)\n  #Plot figure/image\n  for box in boxes:\n      cv2.rectangle(sample,(box[0], box[1]),(box[2], box[3]),(255,223,0), 2)\n  # plt.subplot(n_num/2, n_num/2, fig_no)\n  plt.imshow(sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The test images were run through the improved model to see how well it could detect the wheat heads. As there is no data for the bounding boxes of the test data, the study was unable to produce a numeric accuracy, however, it can be seen from the images below that the wheat heads were successfully detected. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"predict_images(4, True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Discussion\nAs wheat is such an important global food source, the ability to sucessfully identify wheat heads from a set of images, can help farmers predict their yield, to aid food security issues.\n\nThe outcome of this study shows that the use of augmentations are essential to running and improving the accuracy of a model. Although the model was successful in detecting wheat heads, further analysis of the wheat heads is needed to support farmers in predicting wheat yields.\n\nThe model used has the potential to be developed through further analysis of images taken throughout the growing period, to recognise the development of the crop through the growth stages. This would give farmers more information to help them with decision making of requirements such as irrigation and fertilisation. Accurate images could also indicate the health of the crop by detection of disease and pest infestation.\n\nThe global agri-tech sector is experiencing exponential growth, with government support and investment, to deal with challenges facing the global agricultural sector. The use of technology is slowly becoming cricial in the industry, to feed a growing global population with added constraints on land and farming inputs. With the demand of farmers to produce higher yields; if we continue with traditional farming methods, the security of the food chain could be put at risk.\n\nMany farmers have turned to intensive farming strategies to meet this demand, although this has increased productivity, it comes at a cost to the environment (Mennerat et al., 2010). Agri-tech improves yields, profitability and efficiency, whilst reducing the impact on the environment and by building sustainablity and resiliance across crop cultivation.\n\nFarm work is historically manual, however, more and more farmers have begun to consult data about essential variables such as soil, crops, irrigation, fertilisers and weather (Goedde et al., 2020). The benefits of adopting technologies would free up significant time for farmers to pursue work outside their core industry. Farmers must embrace change to a digital future, to overcome increased demand and disruptive forces.\n\nDrones use computer vision to analyse field conditions and are giving farmers the ability of surveying crops quicker over large areas, to relay real-time data (Goedde et al., 2020). The collected data could be used with machine learning to quickly analyse crop yields and to aid to farmers decisions.\n\nWith advances in the Agri-tech sector, the application of machine learning using imagery  will aid yield prediction and should be considered as an essential tool for farmers in the future."},{"metadata":{},"cell_type":"markdown","source":"## References \nAlom, M.Z., Taha, C., Yakopcic, C. et al., A state of the art survey on deep learning theory and architectures. *Electron*, 8:3, pg. 292.\n\nBrisson, N.P., Gate, P., Gouache, D., Charmet, G., Oury, F.X. & Huard, F. (2010). Why are wheat yields stagnating in Europe? A comprohensive data analysis for France. *Field Crops Research*, 119:1, pg. 201-212.\n\nDavid, E., Madec, S., Sadeghi-Tehran, P., Aasen, H., Zheng, B., Lui, S. (2020). Global wheat head detection (GWHD) dataset: A large and diverse dataset of high-resolution RGB-labelled images to develop and benchmark wheat heat detection methods. *Plant Phenomics*, 2020, pg. 12.\n\nGoedde, L., Katz, J., Menard, A. & Revellat, J. (2020). Agriculture’s connected future: How technology can yield new growth. Available at: https://www.mckinsey.com/industries/agriculture/our-insights/agricultures-connected-future-how-technology-can-yield-new-growth (Accessed: 6th Jan. 2021)\n\nKaggle. (2020). Global wheat detection. Available at: https://www.kaggle.com/c/global-wheat-detection (Accessed: 3rd Dec. 2020)\n\nMennerat, A., Nilsen, F., Ebert, D. et al. (2010) Intensive Farming: Evolutionary Implications for Parasites and Pathogens. *Evol Biol* 37, pg. 59–67.\n\nTuzel, O., Porikli, F. & Meer, P. (2020). Region Covariance: A fast detection and classifiaction. *Lecture notes in Computer Science*. 3952, pg. 589-600. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}