{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <span STYLE=\"text-decoration:underline\">Global Wheat Detection</span>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this competition, you’ll detect wheat heads from outdoor images of wheat plants, including wheat datasets from around the globe. Using worldwide data, you will focus on a generalized solution to estimate the number and size of wheat heads. To better gauge the performance for unseen genotypes, environments, and observational conditions, the training dataset covers multiple regions. You will use more than 3,000 images from Europe (France, UK, Switzerland) and North America (Canada). The test data includes about 1,000 images from Australia, Japan, and China. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<div align=\"center\"><img src=\"http://www.global-wheat.com/wp-content/uploads/2020/04/ILLU_01_EN.jpg\" width=\"800\"/></div>\n<span STYLE=\"text-decoration:underline\">\n\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1.2 Let's know more by answering few Questions","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Q1. Why are we solving this problem?\n\nFor several years, agricultural research has been using sensors to observe plants at key moments in their development. However, some important plant traits are still measured manually. One example of this is the manual counting of wheat ears from digital images – a long and tedious job. Factors that make it difficult to manually count wheat ears from digital images include the possibility of overlapping ears, variations in appearance according to maturity and genotype, the presence or absence of barbs, head orientation and even wind.  \n\nHowever, accurate wheat head detection in outdoor field images can be visually challenging. There is often overlap of dense wheat plants, and the wind can blur the photographs. Both make it difficult to identify single heads. Additionally, appearances vary due to maturity, color, genotype, and head orientation. Finally, because wheat is grown worldwide, different varieties, planting densities, patterns, and field conditions must be considered. Models developed for wheat phenotyping need to generalize between different growing environments. Current detection methods involve one- and two-stage detectors (Yolo-V3 and Faster-RCNN), but even when trained with a large dataset, a bias to the training region remains.\n\nThere is the need for a robust and accurate computer model that is capable of counting wheat ears from digital images. This model will benefit phenotyping research and help producers around the world assess ear density, health and maturity more effectively. Some work has already been done in deep learning, though it has resulted in too little data to have a generic model.  \n<br>\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Q2. How the dataset looks like?\n\nThe dataset contains following 4 important files/folders\n\n* `train.csv` - the training data\n* `sample_submission.csv` - a sample submission file in the correct format\n* `train.zip` - training images\n* `test.zip` - test images\n\n> **Note**: Most of the test set images are hidden. A small subset of test images has been included for your use in writing code.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Q3. What are the columns in the data\n\n* `image_id` - the unique image ID\n* `width` - the width of the images\n* `height` - the height of the images\n* `bbox` - a bounding box, formatted as a Python-style list of [xmin, ymin, width, height]\n* `source` - the source of the data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Q4. What am I predicting?\n\nYou are attempting to predict bounding boxes around each wheat head in images that have them. If there are no wheat heads, you must predict no bounding boxes.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Q5. How dataset is prepared\n\nThe [Global Wheat Head Dataset](http://www.global-wheat.com/2020-challenge/) is led by nine research institutes from seven countries: the University of Tokyo, Institut national de recherche pour l’agriculture, l’alimentation et l’environnement, Arvalis, ETHZ, University of Saskatchewan, University of Queensland, Nanjing Agricultural University, and Rothamsted Research. These institutions are joined by many in their pursuit of accurate wheat head detection, including the Global Institute for Food Security, DigitAg, Kubota, and Hiphen.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Q6 What is mAP(the metric used for evaluation)?\n\nThis competition is evaluated on the **mean average precision** at different intersection over union (IoU) thresholds.\n\n`MAP(mean average precision)`: **mAP (mean average precision)** is the average of AP. In some context, we compute the AP for each class and average them. But in some context, they mean the same thing. For example, under the COCO context, there is no difference between AP and mAP.\n\n\n![](https://i.stack.imgur.com/JlHnn.jpg)\n\n> Important note: if there are no ground truth objects at all for a given image, ANY number of predictions (false positives) will result in the image receiving a score of zero, and being included in the mean average precision.\n\n\n\n\nPlease visit following links to know more about MAP\n* https://www.kaggle.com/c/global-wheat-detection/overview/evaluation\n* https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation\n* https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52\n* https://datascience.stackexchange.com/questions/25119/how-to-calculate-map-for-detection-task-for-the-pascal-voc-challenge","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. Getting Data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Loading Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd # package for high-performance, easy-to-use data structures and data analysis\nimport numpy as np # fundamental package for scientific computing with Python\nimport matplotlib\nimport os\nfrom PIL import Image, ImageDraw\nfrom ast import literal_eval\nimport matplotlib.pyplot as plt # for plotting\nimport seaborn as sns # for making plots with seaborn\ncolor = sns.color_palette()\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.offline as offline\noffline.init_notebook_mode()\nimport imageio\nfrom skimage import data, io, filters, img_as_ubyte","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Reading data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_PATH = '../input/global-wheat-detection'\nTRAIN_DIR = f'{BASE_PATH}/train'\nTEST_DIR = f'{BASE_PATH}/test'\n\ntrain = pd.read_csv(f'{BASE_PATH}/train.csv')\nsubmission = pd.read_csv(f'{BASE_PATH}/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Size of train data', train.shape)\nprint('Size of submission file', submission.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Peek at Dataset\n\n* There are 147793 images in the train data\n* We need to predict bounding boxes around each wheat head in images that have them.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2.4 Table overview","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### train data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# display head of train data\ndisplay(train.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### number of unique images in train dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Number of unique images in train data is {len(list(np.unique(train.image_id)))}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's have a look at the describe function\ndisplay(train.describe())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### submission file","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"display(submission.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.5 Check for missing values\n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = train.isnull().sum().sort_values(ascending = False)\npercent = (train.isnull().sum()/train.isnull().count()*100).sort_values(ascending = False)\nmissing_train_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Exploratory Data Analysis (EDA)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.1 Checking for data `source` distribution\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Splitting the bboxes into x, y, w and h\nbboxs = np.stack(train['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=',')))\nfor i, column in enumerate(['x', 'y', 'w', 'h']):\n    train[column] = bboxs[:,i]\n\ntrain['x1'] = train['x'] + train['w']\ntrain['y1'] = train['y'] + train['h']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a dataframe with all train images\nall_train_images = pd.DataFrame([i.split('/')[-1][:-4] for i in train_dir])\nall_train_images.columns=['image_id']\n\n\n# Merge all train images with the bounding boxes dataframe\nall_train_images = all_train_images.merge(train, on='image_id', how='left')\n\n# replace nan values with zeros\nall_train_images['bbox'] = all_train_images.bbox.fillna('[0,0,0,0]')\n\n# split bbox column\nbbox_items = all_train_images.bbox.str.split(',', expand=True)\nall_train_images['bbox_xmin'] = bbox_items[0].str.strip('[ ').astype(float)\nall_train_images['bbox_ymin'] = bbox_items[1].str.strip(' ').astype(float)\nall_train_images['bbox_width'] = bbox_items[2].str.strip(' ').astype(float)\nall_train_images['bbox_height'] = bbox_items[3].str.strip(' ]').astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nobboxes_images = all_train_images[~all_train_images.image_id.isin(train_df.image_id)]\nprint('Number of images with no bounding boxes:', len(nobboxes_images))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images['image_id'].value_counts().iplot(kind='hist',bins=30,color='blue',xTitle='No. of bboxes per Image Id',yTitle='No. of Images')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_count(df, feature, title='', size=2):\n    f, ax = plt.subplots(1,1, figsize=(4*size,3*size))\n    total = float(len(df))\n    sns.countplot(df[feature],order = df[feature].value_counts().index, palette='Set2')\n    plt.title(title)\n    for p in ax.patches:\n        height = p.get_height()\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(100*height/total),\n                ha=\"center\") \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_count(df=train, feature='source', title = 'data source count and %age plot', size=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**inference**\n\n* `ethz_1` and `arvalis_1` are the 2 major data sources (contributing around 65% 0f total data).\n* Dataset is not balanced in terms of source provided.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Visualizing images with bounding boxes","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.2.1 train images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_images(images): \n    f, ax = plt.subplots(5,3, figsize=(18,22))\n    for i, image_id in enumerate(images):\n        image_path = os.path.join(TRAIN_DIR, f'{image_id}.jpg')\n        image = Image.open(image_path)\n        \n        # get all bboxes for given image in [xmin, ymin, width, height]\n        bboxes = [literal_eval(box) for box in train[train['image_id'] == image_id]['bbox']]\n        # draw rectangles on image\n        draw = ImageDraw.Draw(image)\n        for bbox in bboxes:    \n            draw.rectangle([bbox[0], bbox[1], bbox[0] + bbox[2], bbox[1] + bbox[3]], width=3)\n            \n        ax[i//3, i%3].imshow(image) \n        image.close()       \n        ax[i//3, i%3].axis('off')\n\n        source = train[train['image_id'] == image_id]['source'].values[0]\n        ax[i//3, i%3].set_title(f\"image_id: {image_id}\\nSource: {source}\")\n        \n\n    plt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"     def rescale_image(image, bboxes, factor):\n                '''\\n\",\n            image : image_path that needs to be rescaled\\n\",\n            bboxes : list of bounding boxes for that image\\n\",\n            factor : rescaling factor, preferably an integer. We scale both dimensions equally\\n\",\n        '''\n        img = []\n        for i in images:\n            img.append(i)\n        x_size = im.shape[0]\n        y_size = im.shape[1]\n    \n        resized_image = cv2.resize(img, dsize=( int(x_size / factor), int(y_size / factor)), interpolation=cv2.INTER_CUBIC)\\n\",\n        \n        fig,ax = plt.subplots(1)\n        fig.set_size_inches(15, 10)\n        ax.imshow(resized_image)\n        \n        new_bboxes = []\\n\n        for xmin, ymin, width, height in bboxes:\n            rect = patches.Rectangle( (int(xmin/factor) ,int(ymin/factor)), int(width/factor), int(height/factor), linewidth=1, edgecolor='r', facecolor='none')\n            ax.add_patch(rect)\n            new_bboxes.append([int(xmin/factor) ,int(ymin/factor), int(width/factor), int(height/factor)])\n        return resized_image, new_bboxes\n        \n  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = train.sample(n=15, random_state=42)['image_id'].values\ndisplay_images(images)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"###  Let's take a more closer look ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_images_large(images): \n    f, ax = plt.subplots(5,2, figsize=(20, 50))\n    for i, image_id in enumerate(images):\n        image_path = os.path.join(TRAIN_DIR, f'{image_id}.jpg')\n        image = Image.open(image_path)        \n        bboxes = [literal_eval(box) for box in train[train['image_id'] == image_id]['bbox']]\n        # draw rectangles on image\n        draw = ImageDraw.Draw(image)\n        for bbox in bboxes:    \n            draw.rectangle([bbox[0], bbox[1], bbox[0] + bbox[2], bbox[1] + bbox[3]], width=3)\n            \n        ax[i//2, i%2].imshow(image) \n        ax[i//2, i%2].axis('off')\n        source = train[train['image_id'] == image_id]['source'].values[0]\n        ax[i//2, i%2].set_title(f\"image_id: {image_id}\\nSource: {source}\")\n\n    plt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = train.sample(n=10, random_state=42)['image_id'].values\ndisplay_images_large(images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2.2 Visualizing test images\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_test_images(images): \n    f, ax = plt.subplots(5,2, figsize=(20, 50))\n    for i, image_id in enumerate(images):\n        image_path = os.path.join(TEST_DIR, f'{image_id}.jpg')\n        image = Image.open(image_path)        \n            \n        ax[i//2, i%2].imshow(image) \n        ax[i//2, i%2].axis('off')\n        ax[i//2, i%2].set_title(f\"image_id: {image_id}\")\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# since we need to predict bounding boxes for test images, hence below images do not have any bounding boxes\ntest_images = submission.image_id.values\ndisplay_test_images(test_images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plots the images as per the annotations format and applied augmentations\ndef plot_image_list(annotations_list, subtitle_list, cols=2, title='Image Examples'):\n    fig, axs = plt.subplots(nrows=1, ncols=cols, figsize=(16,12), squeeze=False)\n    for i, (annotations, title) in enumerate(zip(annotations_list, subtitle_list)):\n        axs[i // cols][i % cols].imshow(annotations['image'])\n        axs[i // cols][i % cols].set_title(title, fontsize=14)\n        for bbox in annotations['bboxes']:\n            rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=1,edgecolor='r',facecolor='none')\n            axs[i // cols][i % cols].add_patch(rect) \n    fig.suptitle(title, fontsize=18)\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_aug(aug, min_area=0., min_visibility=0.):\n    return A.Compose(aug, bbox_params=A.BboxParams(format='coco', min_area=min_area, \n                                               min_visibility=min_visibility, label_fields=['category_id']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Applying Vertical Flip on the Original Image\naug = get_aug([A.VerticalFlip(p=1)])\nverticalFlip = aug(**orig_annotations)\nannotations_list=[orig_annotations, verticalFlip]\nsubtitle_list=['Original Image', 'Vertically Flipped Image']\nplot_image_list(annotations_list, subtitle_list, title='Vertical Flip' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Applying Horizontal Flip on the Original Image\naug = get_aug([A.HorizontalFlip(p=1)])\nhorizontalFlip = aug(**orig_annotations)\nannotations_list=[orig_annotations, horizontalFlip]\nsubtitle_list=['Original Image', 'Horizontally Flipped Image']\nplot_image_list(annotations_list, subtitle_list, title='Horizontal Flip' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Applying Center Cropping the Original Image\naug = get_aug([A.CenterCrop(p=1, height=512, width=512)])\nCropped = aug(**orig_annotations)\nannotations_list=[orig_annotations, Cropped]\nsubtitle_list=['Original Image', 'Centre Cropped Image']\nplot_image_list(annotations_list, subtitle_list, title='Image Cropping' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Changing the Brightness of the image randomly\naug = get_aug([A.RandomBrightness(p=0.4)])\nbrightness = aug(**orig_annotations)\nannotations_list=[orig_annotations, brightness]\nsubtitle_list=['Original Image', 'Random Brightness']\nplot_image_list(annotations_list, subtitle_list, title='Change in Brightness' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Applying Random SunFlare\n\naug = get_aug([A.RandomSunFlare(p=1)])\nshadow = aug(**orig_annotations)\nannotations_list=[orig_annotations, shadow]\nsubtitle_list=['Original Image', 'Random Sun Flare']\nplot_image_list(annotations_list, subtitle_list, title='Random Sun Flare Results' )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Possible Approaches for Prediction\n1. We can directly create an object detection neural network for this problem. But it would try to find wheat in every picture even if it doesn't have any. This might result in false positives.\n2. Another approach is to create a classification + object detection ensemble model which would first classify whether the image has any wheat or not. The images that are classified as having wheats will be passed to the object detector for bounding box detection. This would reduce the problem of false positives but might lead to some false negatives.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Some insights about model selection and other tricks:\n* Till now, EfficentDet seems to outperform other model architectures.\n* Augmentation always helps improve accuracy.\n* Cutmix and mixup are specially useful types of augmentations.\n* 5 fold training with ensemble based on **WBF** seems to work great.\n* Training is a very very slow process and using kaggle for trainig is not a very good idea. Use colab with some tricks insted.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# webinar on how to improve wheat heads counting thanks to the Global Wheat Challenge ?\n\nfrom IPython.display import IFrame, YouTubeVideo\nYouTubeVideo('Wr44me5eyWY',width=600, height=400)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References:\n\n* image visualization help taken from https://www.kaggle.com/devvindan/wheat-detection-eda\n* http://www.global-wheat.com/2020-challenge/","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### <p><span style=\"color:green\">This Kernel is work in progress, will update soon </br></span></p>\n\n### <p><span style=\"color:red\">Ending note: <br>Please upvote this kernel if you like it . It motivates me to produce more quality content :)</br></span></p>","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}