{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# import useful tools\nfrom glob import glob\nfrom PIL import Image\nimport cv2\n\n# import data visualization\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport seaborn as sns\n\nfrom bokeh.plotting import figure\nfrom bokeh.io import output_notebook, show, output_file\nfrom bokeh.models import ColumnDataSource, HoverTool, Panel\nfrom bokeh.models.widgets import Tabs\n\n# import data augmentation\nimport albumentations as albu","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![image](https://raw.githubusercontent.com/Lexie88rus/GlobalWheatDetection/master/wheat_image_cropped.png)"},{"metadata":{},"cell_type":"markdown","source":"# Global Wheat Detection Competition EDA"},{"metadata":{},"cell_type":"markdown","source":"Detection of wheat spikes with computer vision opens a lot of opportunity for the farmers and breeders like:\n* controlling the growth stage of the plants on the field: the number oand the area of spikes is raising closer to the harvest date;\n* controlling the heatlth of the plants: unusually small number or small size of plants might be a signal of deceased plants;\n* spikes density characteristic and approximate yield estimation for different varieties of wheat.\n\nIn this notebook I am exploring the data and giving some thoughts on what to pay attention when making and validating the models."},{"metadata":{},"cell_type":"markdown","source":"## General Dataset Information"},{"metadata":{},"cell_type":"markdown","source":"Let's just look at the numbers first. Those numbers will give us a hint which images might be interesting to look at."},{"metadata":{},"cell_type":"markdown","source":"`1` The number of train and test images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setup the paths to train and test images\n# TRAIN_DIR = '../input/zalo-ai/za_traffic_2020/traffic_train/images/'\n# TEST_DIR = '../input/zalo-ai/za_traffic_2020/traffic_public_test/images/'\n# TRAIN_CSV_PATH = '../input/train-sign-csv/train_traffic_sign_dataset.csv'\n\nTEST_DIR = '../input/global-wheat-detection/test/'\nTRAIN_DIR = '../input/global-wheat-detection/train/'\nTRAIN_CSV_PATH = '../input/global-wheat-detection/train.csv'\n# Glob the directories and get the lists of train and test images\ntrain_fns = glob(TRAIN_DIR + '*')\n\ntest_fns = glob(TEST_DIR + '*')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compute at the number of train and test images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of train images is {}'.format(len(train_fns)))\nprint('Number of test images is {}'.format(len(test_fns)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setup the paths to train and test images\nTRAIN_DIR = '../input/zalo-ai/za_traffic_2020/traffic_train/images/'\nTEST_DIR = '../input/zalo-ai/za_traffic_2020/traffic_public_test/images/'\nTRAIN_CSV_PATH = '../input/train-sign-csv/train_traffic_sign_dataset.csv'\n\n# TEST_DIR = '../input/global-wheat-detection/test/'\n# TRAIN_DIR = '../input/global-wheat-detection/train/'\n# TRAIN_CSV_PATH = '../input/global-wheat-detection/train.csv'\n# Glob the directories and get the lists of train and test images\ntrain_fns = glob(TRAIN_DIR + '*')\n\ntest_fns = glob(TEST_DIR + '*')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of train images is {}'.format(len(train_fns)))\nprint('Number of test images is {}'.format(len(test_fns)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have only 10 test images here, other test images will be used to evaluate the prediction models during the submission.\n\nJust `3422` images for the training seems to be not much at all. Data augmentation techniques will be definetely required in this competition."},{"metadata":{},"cell_type":"markdown","source":"`2` The number of bounding boxes (wheat spikes) per image:"},{"metadata":{},"cell_type":"markdown","source":"Construct dataframe with all images (images with no bboxes will have nan values in all columns except `image_id`):"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Load the dataframe with the bounding boxes\ntrain = pd.read_csv(TRAIN_CSV_PATH)\n\n# Create a dataframe with all train images\nall_train_images = pd.DataFrame([fns.split('/')[-1][:-4] for fns in train_fns])\nall_train_images.columns=['image_id']\n\n# Merge all train images with the bounding boxes dataframe\nall_train_images = all_train_images.merge(train, on='image_id', how='left')\n\n# replace nan values with zeros\nall_train_images['bbox'] = all_train_images.bbox.fillna('[0,0,0,0]')\n\n# split bbox column\nbbox_items = all_train_images.bbox.str.split(',', expand=True)\nall_train_images['bbox_xmin'] = bbox_items[0].str.strip('[ ').astype(float)\nall_train_images['bbox_ymin'] = bbox_items[1].str.strip(' ').astype(float)\nall_train_images['bbox_width'] = bbox_items[2].str.strip(' ').astype(float)\nall_train_images['bbox_height'] = bbox_items[3].str.strip(' ]').astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('{} images without wheat heads.'.format(len(all_train_images) - len(train)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_coco_json_to_csv(filename):\n    import pandas as pd\n    import json\n    import os\n    s = json.load(open(filename, 'r'))\n    filename = os.path.split(filename)[-1][:-5]\n    print(filename)\n    out_file = filename + '_1.csv'\n    out = open(''+out_file, 'w')\n    out.write('image_id,width,height,x,y,w,h,label\\n')\n\n    all_ids = []\n    all_sizes = {}\n    for im in s['images']:\n        all_ids.append(im['id'])\n        all_sizes[im['id']] = [im['width'], im['height']]\n    all_sizes\n    all_ids_ann = []\n    for ann in s['annotations']:\n        image_id = ann['image_id']\n        all_ids_ann.append(image_id)\n        imgw = all_sizes[image_id][0]\n        imgh = all_sizes[image_id][1]\n        xmin = ann['bbox'][0]\n        ymin = ann['bbox'][1]\n        width = ann['bbox'][2]\n        height = ann['bbox'][3]\n        label = ann['category_id']\n        out.write('{},{},{},{},{},{},{},{}\\n'.format(image_id, imgw, imgh, xmin, ymin, width, height, label))\n\n    all_ids = set(all_ids)\n    all_ids_ann = set(all_ids_ann)\n    no_annotations = list(all_ids - all_ids_ann)\n    print(\"Number of images without annotations\", len(no_annotations))\n    # Output images without any annotations\n    for image_id in no_annotations:\n        out.write('{},{},{},{},{},{},{},{}\\n'.format(image_id, -1, -1, -1, -1, -1, -1, -1))\n    out.close()\n\n    # Sort file by image id\n    s1 = pd.read_csv(out_file)\n    s1.sort_values('image_id', inplace=True)\n    s1.to_csv(out_file, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"convert_coco_json_to_csv('../input/zalo-ai/za_traffic_2020/traffic_train/train_traffic_sign_dataset.json')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"trong train set không có ảnh nào không được đánh nhãn "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the dataframe with the bounding boxes\nall_train_images = pd.read_csv(TRAIN_CSV_PATH)\nall_train_images.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nall_train_images['x'] = all_train_images['x'].astype(float)\nall_train_images['y'] = all_train_images['y'].astype(float)\nall_train_images['w'] = all_train_images['w'].astype(float)\nall_train_images['h'] = all_train_images['h'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot some image examples:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_all_bboxes(df, image_id):\n    image_bboxes = df[df.image_id == image_id]\n    \n    bboxes = []\n    for _,row in image_bboxes.iterrows():\n        bboxes.append((row.x, row.y, row.w, row.h))\n    #print(bboxes)\n    return bboxes\n\ndef plot_image_examples(df, rows=3, cols=3, title='Image examples'):\n    fig, axs = plt.subplots(rows, cols, figsize=(20,20))\n    for row in range(rows):\n        for col in range(cols):\n            idx = np.random.randint(len(df), size=1)[0]\n            img_id = df.iloc[idx].image_id\n            #print(type(TRAIN_DIR))\n            img_id = img_id.astype(np.int64)\n            img = Image.open(TRAIN_DIR + str(img_id) + '.png')\n            axs[row, col].imshow(img)\n            \n            bboxes = get_all_bboxes(df, img_id)\n            print(bboxes)\n            for bbox in bboxes:\n                rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=1,edgecolor='r',facecolor='none')\n                axs[row, col].add_patch(rect)\n            \n            axs[row, col].axis('off')\n            \n    plt.suptitle(title)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image_examples(all_train_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see images taken at different lighting conditions and plant maturity stages!"},{"metadata":{},"cell_type":"markdown","source":"Count number of bounding boxes per image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.width","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compute the number of bounding boxes per train image\nall_train_images['count'] = all_train_images.apply(lambda row: 1 if np.isfinite(row.width) else 0, axis=1)\ntrain_images_count = all_train_images.groupby('image_id').sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# See this article on how to plot bar charts with Bokeh:\n# https://towardsdatascience.com/interactive-histograms-with-bokeh-202b522265f3\ndef hist_hover(dataframe, column, colors=[\"#94c8d8\", \"#ea5e51\"], bins=30, title=''):\n    hist, edges = np.histogram(dataframe[column], bins = bins)\n    \n    hist_df = pd.DataFrame({column: hist,\n                             \"left\": edges[:-1],\n                             \"right\": edges[1:]})\n    hist_df[\"interval\"] = [\"%d to %d\" % (left, right) for left, \n                           right in zip(hist_df[\"left\"], hist_df[\"right\"])]\n\n    src = ColumnDataSource(hist_df)\n    plot = figure(plot_height = 400, plot_width = 600,\n          title = title,\n          x_axis_label = column,\n          y_axis_label = \"Count\")    \n    plot.quad(bottom = 0, top = column,left = \"left\", \n        right = \"right\", source = src, fill_color = colors[0], \n        line_color = \"#35838d\", fill_alpha = 0.7,\n        hover_fill_alpha = 0.7, hover_fill_color = colors[1])\n        \n    hover = HoverTool(tooltips = [('Interval', '@interval'),\n                              ('Count', str(\"@\" + column))])\n    plot.add_tools(hover)\n    \n    output_notebook()\n    show(plot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_hover(train_images_count, 'count', title='Number of signs per image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the images have 1-2 signs on them."},{"metadata":{},"cell_type":"markdown","source":"Let's plot some examples with small number of spikes per image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"less_spikes_ids = train_images_count[train_images_count['count'] < 3].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(less_spikes_ids)], title='Example images with small number of spikes')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"less_sign_ids = train_images_count[train_images_count['count'] < 3].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(less_sign_ids)], title='Example images with small number of signs')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are some very strange examples:\n* on some of the images all we see is the ground;\n* some of the images are just zoomed in a lot."},{"metadata":{},"cell_type":"markdown","source":"Plot the images with many spikes:"},{"metadata":{},"cell_type":"markdown","source":"Kiểm tra lại 1 số sample."},{"metadata":{"trusted":true},"cell_type":"code","source":"many_spikes_ids = train_images_count[train_images_count['count'] > 100].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(many_spikes_ids)], title='Example images with large number of spikes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These actually look much better than those with small number of spikes."},{"metadata":{"trusted":true},"cell_type":"code","source":"more_sign_ids = train_images_count[train_images_count['count'] > 8].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(more_sign_ids)], title='Example images with large number of signs')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"có 1 vài ảnh lạ khi có nhiều box mà không thấy được vẽ lên ảnh -> maybe box nhỏ"},{"metadata":{},"cell_type":"markdown","source":"`3` Area of bounding boxes:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# compute bounding box areas\nall_train_images['bbox_area'] = all_train_images['w'] * all_train_images['h']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot a histogram of bounding box areas\nhist_hover(all_train_images, 'bbox_area', title='Area of a single bounding box')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"có vẻ là diện tích box khá bé, chủ yếu là các box bé -> maybe các skill trong bài Wheat Detection có thể dùng được","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"có vẻ là diện tích box khá bé, chủ yếu là các box bé -> maybe các skill trong bài Wheat Detection có thể dùng được"},{"metadata":{},"cell_type":"markdown","source":"The maximum area of bounding box:"},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.bbox_area.max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of individual areas of bounding boxes has a very long tail. It would be interesting to look at the images with those large bounding boxes."},{"metadata":{},"cell_type":"markdown","source":"Let's plot some examples of large bounding boxes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"large_boxes_ids = all_train_images[all_train_images['bbox_area'] > 8000].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(large_boxes_ids)], title='Example images with large bbox area')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"large_boxes_ids = all_train_images[all_train_images['bbox_area'] > 20000].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(large_boxes_ids)], title='Example images with large bbox area')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"các box to là ở gần -> có thể không cần remove cho training"},{"metadata":{"trusted":true},"cell_type":"code","source":"có 1 vài trường hợp biển bị cong vẹo","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__What are these anomally large bounding boxes?? I think those should be removed while training!!!__"},{"metadata":{},"cell_type":"markdown","source":"Similarly, let's look at very small bounding boxes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"min_area = all_train_images[all_train_images['bbox_area'] > 0].bbox_area.min()\nprint('The smallest bounding box area is {}'.format(min_area))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"small_boxes_ids = all_train_images[(all_train_images['bbox_area'] < 50) & (all_train_images['bbox_area'] > 0)].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(small_boxes_ids)], title='Example images with large bbox area')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"small_boxes_ids = all_train_images[(all_train_images['bbox_area'] < 50) & (all_train_images['bbox_area'] > 0)].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(small_boxes_ids)], title='Example images with large bbox area')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"có các boxes quá nhỏ < 10 pixel, maybe có thể remove ra khỏi training process"},{"metadata":{},"cell_type":"markdown","source":"If you look very close, you can probably see those tinyest bounding boxes near the corners and borders of the images. Probably, the boundries were drawn first, than the images were cut into several ones. That is why we see those strange small bounsing boxes in the corners.\n\nIt is not necessary to clean these, because they won't have much effect on the IOU metric."},{"metadata":{},"cell_type":"markdown","source":"`4` Area of bounding boxes per image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# compute the total bounding boxes area per image\narea_per_image = all_train_images.groupby(by='image_id').sum().reset_index()\n\n# compute the percentage of the image area covered by bounding boxes\narea_per_image_percentage = area_per_image.copy()\narea_per_image_percentage['bbox_area'] = area_per_image_percentage['bbox_area'] / (626*1626) * 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_hover(area_per_image_percentage, 'bbox_area', title='Percentage of image area covered by bounding boxes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This looks like a nice normal distribution! 20-40% of the image area is covered by the bounding boxes.\n\nThis observation can be used to validate the predictions of the resulting model. The percentage of predicted bounding boxes area should be normally distributed too.\n\nWe can also see that the maximum is actually greater than 100%. This means that the bounding boxes are overlapping."},{"metadata":{},"cell_type":"markdown","source":"Let's look at some examples of images with small areas covered by bounding boxes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"small_area_perc_ids = area_per_image_percentage[area_per_image_percentage['bbox_area'] < 7].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(small_area_perc_ids)], title='Example images with small percentage of area covered by bounding boxes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's plot the images with large areas covered by bounding boxes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"small_area_perc_ids = area_per_image_percentage[area_per_image_percentage['bbox_area'] < 1.5].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(small_area_perc_ids)], title='Example images with small percentage of area covered by bounding boxes')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"large_area_perc_ids = area_per_image_percentage[area_per_image_percentage['bbox_area'] > 3].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(large_area_perc_ids)], title='Example images with large percentage of area covered by bounding boxes')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"large_area_perc_ids = area_per_image_percentage[area_per_image_percentage['bbox_area'] > 7].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(large_area_perc_ids)], title='Example images with large percentage of area covered by bounding boxes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"chủ yếu là các box nhỏ có 1 vài trường hợp box còn chồng lấn vào nhau -> cần phải xóa ra khỏi training process"},{"metadata":{},"cell_type":"markdown","source":"`5` The brightness of the images:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_image_brightness(image):\n    # convert to grayscale\n    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n    \n    # get average brightness\n    return np.array(gray).mean()\n\ndef add_brightness(df):\n    brightness = []\n    for _, row in df.iterrows():\n        img_id = row.image_id\n        #print(img_id)\n        image = cv2.imread(TRAIN_DIR + str(img_id) + '.png')\n        brightness.append(get_image_brightness(image))\n        \n    brightness_df = pd.DataFrame(brightness)\n    brightness_df.columns = ['brightness']\n    df = pd.concat([df, brightness_df], ignore_index=True, axis=1)\n    df.columns = ['image_id', 'brightness']\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images_df = pd.DataFrame(all_train_images.image_id.unique())\nimages_df.columns = ['image_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# add brightness to the dataframe\nimages_df = pd.DataFrame(all_train_images.image_id.unique())\nimages_df.columns = ['image_id']\nbrightness_df = add_brightness(images_df)\n\nall_train_images = all_train_images.merge(brightness_df, on='image_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_hover(all_train_images, 'brightness', title='Images brightness distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot the darkest images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dark_ids = all_train_images[all_train_images['brightness'] < 30].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(dark_ids)], title='Darkest images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On some of the images, it is even hard for human to see the spikes!"},{"metadata":{"trusted":true},"cell_type":"code","source":"dark_ids = all_train_images[all_train_images['brightness'] < 40].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(dark_ids)], title='Darkest images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot the brightest images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"bright_ids = all_train_images[all_train_images['brightness'] > 150].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(bright_ids)], title='Brightest images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are very different from the dark ones. Some filters needed here to make the spikes more clear.\n\nI also see the missing boundaries."},{"metadata":{"trusted":true},"cell_type":"code","source":"bright_ids = all_train_images[all_train_images['brightness'] > 130].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(bright_ids)], title='Brightest images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`6` The most and the least green and yellow images:"},{"metadata":{},"cell_type":"markdown","source":"I would like to plot images with different dominant colors. The idea is that the most green images will represent healthy plants. The most yellow images will contain plants close to maturity. The most brown images will have ground on them."},{"metadata":{},"cell_type":"markdown","source":"không cần so sánh vàng với xanh lắm vì đây context phức tạp không phải chỉ có mỗi vàng và xanh như GWD"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_percentage_of_green_pixels(image):\n    # convert to HSV\n    hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n    \n    # get the green mask\n    hsv_lower = (40, 40, 40) \n    hsv_higher = (70, 255, 255)\n    green_mask = cv2.inRange(hsv, hsv_lower, hsv_higher)\n    \n    return float(np.sum(green_mask)) / 255 / (1024 * 1024)\n\ndef get_percentage_of_yellow_pixels(image):\n    # convert to HSV\n    hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n    \n    # get the green mask\n    hsv_lower = (25, 40, 40) \n    hsv_higher = (35, 255, 255)\n    yellow_mask = cv2.inRange(hsv, hsv_lower, hsv_higher)\n    \n    return float(np.sum(yellow_mask)) / 255 / (1024 * 1024)\n\ndef add_green_pixels_percentage(df):\n    green = []\n    for _, row in df.iterrows():\n        img_id = row.image_id  \n        image = cv2.imread(TRAIN_DIR + str(img_id) + '.png')\n        green.append(get_percentage_of_green_pixels(image))\n        \n    green_df = pd.DataFrame(green)\n    green_df.columns = ['green_pixels']\n    df = pd.concat([df, green_df], ignore_index=True, axis=1)\n    df.columns = ['image_id', 'green_pixels']\n    \n    return df\n\ndef add_yellow_pixels_percentage(df):\n    yellow = []\n    for _, row in df.iterrows():\n        img_id = row.image_id  \n        image = cv2.imread(TRAIN_DIR + str(img_id) + '.png')\n        yellow.append(get_percentage_of_yellow_pixels(image))\n        \n    yellow_df = pd.DataFrame(yellow)\n    yellow_df.columns = ['yellow_pixels']\n    df = pd.concat([df, yellow_df], ignore_index=True, axis=1)\n    df.columns = ['image_id', 'yellow_pixels']\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# add a column with the percentage of green pixels\ngreen_pixels_df = add_green_pixels_percentage(images_df)\nall_train_images = all_train_images.merge(green_pixels_df, on='image_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_hover(all_train_images, 'green_pixels', title='Percentage of green pixels distribution', colors=['#c3ea84', '#3e7a17'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Surprisingly, the most green images have only aoung 60% green pixels.\n\nThe majority of the images are not green at all! Most probably, they are more yellow and are the images of the plants close to harvest."},{"metadata":{},"cell_type":"markdown","source":"The most green images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"green_ids = all_train_images[all_train_images['green_pixels'] > 0.55].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(green_ids)], title='The most green images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The most green images mostly contain the plants with very small spikes, which are just starting to appear."},{"metadata":{"trusted":true},"cell_type":"code","source":"# add a column with the percentage of yellow pixels\nyellow_pixels_df = add_yellow_pixels_percentage(images_df)\nall_train_images = all_train_images.merge(yellow_pixels_df, on='image_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_hover(all_train_images, 'yellow_pixels', title='Percentage of yellow pixels distribution', colors=['#fffedb', '#fffeab'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the most yellow images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"yellow_ids = all_train_images[all_train_images['yellow_pixels'] > 0.55].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(yellow_ids)], title='The most yellow images')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"yellow_ids = all_train_images[all_train_images['yellow_pixels'] > 0.3].image_id\nplot_image_examples(all_train_images[all_train_images.image_id.isin(yellow_ids)], title='The most yellow images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Thoughts on Data Augmentation\n\nData augmentation is critical in this competition, because there is a relatively small training set. Data augmentation will allow to build robust models under given circumstances.\nWhat augmentations/filers might work:\n* flipping images horizontally and vertically, because the orientation of original images is different;\n* crop-resize, because we can see spikes at different zoom levels;\n* different filters to adjust the lighting conditions. I suggest looking at [this competition](https://www.kaggle.com/c/aptos2019-blindness-detection) for example.\n\nWhat to do with caution:\n* rotation might not work, because rotation messes up the bounding boxes."},{"metadata":{},"cell_type":"markdown","source":"Example augmentation pipeline:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# setup an example augmentation pipeline\n# be sure to use bbox safe functions for data augmentation\nexample_transforms = albu.Compose([\n    albu.RandomSizedBBoxSafeCrop(512, 512, erosion_rate=0.0, interpolation=1, p=1.0),\n    albu.HorizontalFlip(p=0.5),\n    albu.VerticalFlip(p=0.5),\n    albu.OneOf([albu.RandomContrast(),\n                albu.RandomGamma(),\n                albu.RandomBrightness()], p=1.0),\n    albu.CLAHE(p=1.0)], p=1.0, bbox_params=albu.BboxParams(format='coco', label_fields=['category_id']))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def apply_transforms(transforms, df, n_transforms=3):\n    idx = np.random.randint(len(df), size=1)[0]\n    \n    image_id = df.iloc[idx].image_id\n    image_id = image_id.astype(np.int64)\n    bboxes = []\n    for _, row in df[df.image_id == image_id].iterrows():\n        bboxes.append([row.x, row.y, row.w, row.h])\n        \n    image = Image.open(TRAIN_DIR + str(image_id) + '.png')\n    \n    fig, axs = plt.subplots(1, n_transforms+1, figsize=(15,7))\n    \n    # plot the original image\n    axs[0].imshow(image)\n    axs[0].set_title('original')\n    for bbox in bboxes:\n        rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=1,edgecolor='r',facecolor='none')\n        axs[0].add_patch(rect)\n    \n    # apply transforms n_transforms times\n    for i in range(n_transforms):\n        params = {'image': np.asarray(image),\n                  'bboxes': bboxes,\n                  'category_id': [1 for j in range(len(bboxes))]}\n        augmented_boxes = transforms(**params)\n        bboxes_aug = augmented_boxes['bboxes']\n        image_aug = augmented_boxes['image']\n\n        # plot the augmented image and augmented bounding boxes\n        axs[i+1].imshow(image_aug)\n        axs[i+1].set_title('augmented_' + str(i+1))\n        for bbox in bboxes_aug:\n            rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=1,edgecolor='r',facecolor='none')\n            axs[i+1].add_patch(rect)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_train_images.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"apply_transforms(example_transforms, all_train_images, n_transforms=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"apply_transforms(example_transforms, all_train_images, n_transforms=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note how CLAHE emphasizes the features in darker spots. It is a must have for this competition."},{"metadata":{},"cell_type":"markdown","source":"## Conclusions:\n\n1. Images are taken at different zoom levels. Crop and resize data augmentations to be used for model training.\n2. Images are taken at various lighting conditions. Special filters should be used to address that.\n3. Bounding boxes are messy! \n    * Giant bounding boxes should be filtered out by area and removed before model training.\n    * Micro bounding boxes. These can stay. They won't have much effect on the IOU metric.\n    * Some spikes are not surrounded by a bounding box (missing bounding boxes)."},{"metadata":{},"cell_type":"markdown","source":"## Related Papers and Links\nI found some interesting papers and articles:\n1. [Deep learning based banana plant detection and counting using high-resolution red-green-blue (RGB) images collected from unmanned aerial vehicle (UAV)](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0223906). The objective of this research is focused to automate the count of banana plants during early stage of growth, in which they are prone to lose maturity.\n2. [Fine-grained recognition of plants from images](https://plantmethods.biomedcentral.com/articles/10.1186/s13007-017-0265-4). Fine-grained recognition of plants from images is a challenging computer vision task, due to the diverse appearance and complex structure of plants, high intra-class variability and small inter-class differences. We review the state-of-the-art and discuss plant recognition tasks, from identification of plants from specific plant organs to general plant recognition “in the wild”.\n3. [Review of deep learning algorithms for object detection](https://medium.com/zylapp/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852)\n4. [Fast Detection Models](https://lilianweng.github.io/lil-log/2018/12/27/object-detection-part-4.html)\n5. [SSD Tutorial](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection), [SSD implementation in pytorch](https://github.com/qfgaohao/pytorch-ssd)\n6. [EfficientDet](https://arxiv.org/pdf/1911.09070.pdf)\n7. [CutMix paper](https://arxiv.org/pdf/1905.04899.pdf)"},{"metadata":{},"cell_type":"markdown","source":"## References & Credits\n1. [Article on how to plot bar charts with Bokeh](https://towardsdatascience.com/interactive-histograms-with-bokeh-202b522265f3)"},{"metadata":{},"cell_type":"markdown","source":"__This EDA is being updated. Your questions and comments are very welcome!__"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}