{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <span STYLE=\"text-decoration:underline\">Global Wheat Detection</span>\n<div align=\"center\"><img src=\"http://www.global-wheat.com/wp-content/uploads/2020/04/ILLU_01_EN.jpg\" width=\"800\"/></div>\n<span STYLE=\"text-decoration:underline\">\nThe Problem</span> \n<br>\nFor several years, agricultural research has been using sensors to observe plants at key moments in their development. However, some important plant traits are still measured manually. One example of this is the manual counting of wheat ears from digital images – a long and tedious job. Factors that make it difficult to manually count wheat ears from digital images include the possibility of overlapping ears, variations in appearance according to maturity and genotype, the presence or absence of barbs, head orientation and even wind.  \n \n<br>\n<span style=\"text-decoration:underline\">The Need</span> \n<br>\nThere is the need for a robust and accurate computer model that is capable of counting wheat ears from digital images. This model will benefit phenotyping research and help producers around the world assess ear density, health and maturity more effectively. Some work has already been done in deep learning, though it has resulted in too little data to have a generic model.  \n<br>\nRefer [this](http://www.global-wheat.com/) page for more details.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom pandas_profiling import ProfileReport\nfrom pandas_summary import DataFrameSummary\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport random\nimport math\nfrom tqdm import tqdm\nimport seaborn as sns\nfrom os import listdir\nfrom os.path import isfile, join\n%matplotlib inline\n\npath = '../input/global-wheat-detection/'\nTRAIN_IMAGES_PATH = path+'train/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train = pd.read_csv(path+'train.csv')\nsub = pd.read_csv(path+'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Pre-processing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Utility Functions","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_images(dataframe, show_bb = True, image_count = 16,idxs = [], image_label='image_id', bbox_label='bbox', n = 3):\n    \"\"\"This function can display any no of image in the grid size of 4 by no_of_images/4.\n        Usage:-\n            dataframe = The dataframe containing images and bounding boxes grouped into single list.\n            show_bb = Show bounding boxes or not.\n            imge_count = No of images to display (multiple of n).\n            idxs =  If u want to pass your own indexes for ploting use this else pass [] an empty array. It will automatically select random images.\n            image_label = Name of the column containing images in dataframe.\n            bbox_label = Name of the column containing list of all bounding boxes per image in dataframe.\n            n = Number of images per row.\n\n        [NOTE]: If you want to convert stock train dataframe to desired format use the clean_data method. \n    \"\"\"\n    size = len(dataframe)\n    image_count = image_count + (image_count % n)\n    if len(idxs)==0:\n        create_idx = True\n    else:\n        create_idx = False\n        \n    row_count = (int) (image_count / n)\n    fig, ax = plt.subplots(row_count, n, figsize=(20,10))\n    for i in range(image_count):\n        x = (int)(i/n)\n        y = i%n\n        if create_idx:\n            idx = random.randint(0, size-1)\n            idxs.append(idx)\n        else:\n            idx = idxs[i]\n        input_row = dataframe.iloc[idx]\n        tuple_index = (x,y) if row_count > 1 else y\n        ax[tuple_index].imshow(cv2.imread(TRAIN_IMAGES_PATH + input_row[image_label]))\n        ax[tuple_index].set_title(input_row['image_id'])\n        if show_bb:\n            try:\n                bbs = input_row[bbox_label]\n                for bbox in bbs:    \n                    rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=2,edgecolor='r',facecolor='none')\n                    ax[tuple_index].add_patch(rect)\n            except:\n                pass\n    fig.show()\n    return idxs\n\n\ndef enlarge_image(dataframe, idx = -1, show_bb = True,image_label='image_id', bbox_label='bbox'):\n    \"\"\"This function is used to enlarge single image with or without bounding boxes.\n        Usage:-\n            dataframe = The dataframe containing images and bounding boxes grouped into single list.\n            idx = The index of the image to be displayed. -1(default) means random selection.\n            show_bb = Show bounding boxes or not.\n            image_label = Name of the column containing images in dataframe.\n            bbox_label = Name of the column containing list of all bounding boxes per image in dataframe.\n        \n        [NOTE]: If you want to convert stock train dataframe to desired format use the clean_data method. \n    \"\"\"\n    fig, ax = plt.subplots(figsize=(15,15))\n    size = len(dataframe[dataframe['source']!='not_specified'])\n    if idx==-1:\n        idx = random.randint(0, size-1)\n    input_row = dataframe.iloc[idx]\n    tuple_index = (0,0)\n    ax.imshow(cv2.imread(TRAIN_IMAGES_PATH + input_row[image_label]))\n    ax.set_title(input_row[image_label])\n    if show_bb:\n        try:\n            bbs = input_row[bbox_label]\n            for bbox in bbs:    \n                rect = patches.Rectangle((bbox[0],bbox[1]),bbox[2],bbox[3],linewidth=2,edgecolor='r',facecolor='none')\n                ax.add_patch(rect)\n        except:\n            pass\n    fig.show()\n    \n\ndef clean_data(train):\n    dic = {}\n    imgs = []\n    bbs = []\n    srcs = []\n    tmp = []\n    tn='---'\n    for i in tqdm(train.iterrows()):\n        img,_,_,bb,s = i[1]\n        if tn=='---':\n            tn = img\n        elif tn == img:\n            tmp.append(list(map(math.floor, list(map(float,bb.replace('[',\"\").replace(']',\"\").split(','))))))\n        else:\n            imgs.append(tn+'.jpg')\n            bbs.append(tmp)\n            srcs.append(s)\n            tn = img\n            tmp=[]\n            tmp.append(list(map(math.floor, list(map(float,bb.replace('[',\"\").replace(']',\"\").split(','))))))\n    imgs.append(tn+'.jpg')\n    bbs.append(tmp)\n    srcs.append(s)\n    dic['image_id']=imgs\n    dic['bbox'] = bbs\n    dic['source'] = srcs\n    train_clean = pd.DataFrame(dic)\n    return train_clean","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dfs = DataFrameSummary(train)\n\ndfs.columns_stats","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train_clean = clean_data(train)\n\n\n\"\"\"Adding unlabeled images from train directory.\"\"\"\ndic = {}\nonlyfiles = [f for f in listdir(path+'train') if isfile(join(path+'train', f))]\nunlabeled = list(set(onlyfiles) - set(train_clean['image_id']))\ndic['image_id'] = unlabeled\ndic['bbox'] = [[] for i in range(len(unlabeled))]\ndic['source'] = ['not_specified' for i in range(len(unlabeled))]\ntemp_clean = pd.DataFrame(dic)\ntrain_clean = pd.concat([train_clean,temp_clean])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_clean.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA\n## About the Data Sources\n[ARVALIS - Plant Institute](https://www.english.arvalisinstitutduvegetal.fr/index.html). Institut du Végétal: ARVALIS - Institut du vegetal is an applied agricultural research organization dedicated to arable crops : cereals, maize, sorghum, potatoes, fodder crops, flax and tobacco. … It considers technological innovation as a major tool to enable producers and agri-companies to respond to societal challenges.\n<br>\n<br>\n[ETHZ- ETH Zurich](https://ethz.ch/en.html) trains true experts and prepares its students to carry out their tasks as critical members of their communities, making an important contribution to the sustainable development of science, the economy and society.\n<br>\n<br>\n[INRAE](https://www.inrae.fr/en) is France's new National Research Institute for Agriculture, Food and Environment, created on January 1, 2020, It was formed by the merger of INRA, the National Institute for Agricultural Research, and IRSTEA, the National Research Institute of Science and Technology for the Environment and Agriculture.\n<br>\n<br>\n[RRES 90003](https://tinyurl.com/y8q7gpht) is a global engineering specification for Identification Marking Methods and Controls. It is a Rolls-Royce global document that was compiled for new component designs and will be called out in place of the previous system of JES and EDI specifications.\n<br>\n<br>\n[University of Saskatchewan (USask)](https://www.usask.ca/) researchers played a key role in an international consortium that has sequenced the entire genome of durum wheat—the source of semolina for pasta, a food staple for the world's population, according to an article published today in Nature Genetics.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"counts = dict(train_clean['source'].value_counts())\n\nfig, ax = plt.subplots(figsize=(8,8));\nwedges, texts, autotexts = ax.pie(list(counts.values()), autopct='%1.1f%%',\n        shadow=True, startangle=90);\nax.legend(wedges, list(counts.keys()),\n          title=\"Sources\",\n          loc=\"center\",\n          bbox_to_anchor=(1, 0, 0.5, 1));\n\nplt.setp(autotexts, size=15);\n\nax.set_title(\"Data Distribution based on Sources\");\nplt.show();\nsns.countplot(x='source', data=train_clean);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"profile = ProfileReport(train, title='Report',progress_bar = False);\nprofile.to_widgets()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Close Analysis of a Single Image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"enlarge_image(train_clean)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting with and without bounding boxes for same images","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean, show_bb = False, image_count = 3);\nplot_images(train_clean, show_bb = True,idxs = idxs, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Examples of images without any wheat heads","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_images(train_clean[train_clean['source']=='not_specified'], idxs = [],show_bb = True, image_count = 6);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [ARVALIS - Plant Institute](https://www.english.arvalisinstitutduvegetal.fr/index.html)-1","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='arvalis_1'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='arvalis_1'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [ARVALIS - Plant Institute](https://www.english.arvalisinstitutduvegetal.fr/index.html)-2","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='arvalis_2'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='arvalis_2'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [ARVALIS - Plant Institute](https://www.english.arvalisinstitutduvegetal.fr/index.html)-3","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='arvalis_3'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='arvalis_3'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [ETH Zurich](https://ethz.ch/en.html)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='ethz_1'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='ethz_1'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [ROLLS-ROYCE ENGINEERING SPECIFICATION INDEX](https://tinyurl.com/y8q7gpht)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='rres_1'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='rres_1'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [University of Saskatchewan](https://www.usask.ca/)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='usask_1'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='usask_1'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from [INRAE](https://www.inrae.fr/en)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"idxs = plot_images(train_clean[train_clean['source']=='inrae_1'], show_bb = False,idxs = [], image_count = 3);\nplot_images(train_clean[train_clean['source']=='inrae_1'], idxs = idxs, show_bb = True, image_count = 3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is evident from the above image analysis that each type of source has specific types of wheat images. This is probably dependent on the region of data collection and time of the year when the particular data was collected. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Possible Approaches for Prediction\n1. We can directly create an object detection neural network for this problem. But it would try to find wheat in every picture even if it doesn't have any. This might result in false positives.\n2. Another approach is to create a classification + object detection ensemble model which would first classify whether the image has any wheat or not. The images that are classified as having wheats will be passed to the object detector for bounding box detection. This would reduce the problem of false positives but might lead to some false negatives.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Some insights about model selection and other tricks:\n* Till now, EfficentDet seems to outperform other model architectures.\n* Augmentation always helps improve accuracy.\n* Cutmix and mixup are specially useful types of augmentations.\n* 5 fold training with ensemble based on **WBF** seems to work great.\n* Training is a very very slow process and using kaggle for trainig is not a very good idea. Use colab with some tricks insted.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### <span style=\"text-decoration:underline\">TODO:</span>\n* <del>Image Augmentation</del><br>Cutmix and mixup are good. Other augmentations include cutout, random flip and rotate. Brightness changes and blurring doesn't seem to help that much. \n* <del>Training and Inference Code</del><br>Refer to [this notebook](https://www.kaggle.com/yashchoudhary/gwd-fasterrcnn-with-augmentation-train-inference) for <span style=\"text-decoration:underline; color:red\">FasterRCNN Training and Inference Code with Augmentation.</span>\n* <del>Final Model Selection</del><br> EfficientDet out-performs others.\n\n\nI will be updating this notebook with more analysis methodologies.\nIf you like my work please <span style=\"text-decoration:underline;color:red\">UPVOTE</span>. It really motivates me to create better notebooks.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}