{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n

Global Wheat Detection EDA for BBox

","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## About this EDA 📗\n\nIn this competition, there are some wrong bbox and unlabeled wheat heads.\n- Hug bbox\n- Too small bbox\n- Wrong target(e.g. ladybug)\n- etc.\n\nI cleaned data in my way. \n\nAlthough this competition is over,I think it will help someone. \nI decided to share this results.\nLater, you can just download **clean_df.csv** in my public dataset.\n\n**[Update]** Clean BBox Dataset for Wheat Detection is [this](http://www.kaggle.com/piantic/gwd-clean-train).\nAnd I added this dataset!\n\nIf you feel this was something new and fresh, and it added some value to you, please consider upvoting, it motivates to keep writing good kernels. 😄","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Importing the necessary libraries ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nimport math\nimport numpy as np\nimport scipy as sp\nimport pandas as pd\n\nimport glob \nfrom PIL import Image, ImageDraw\nimport matplotlib.pyplot as plt\n\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport plotly.figure_factory as ff","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading the train.csv \n- train_df : `train.csv`(original)\n- clean_df : `clean train.csv`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/gwd-clean-train/original_train.csv')\nclean_df = pd.read_csv('../input/gwd-clean-train/new_train_0805.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_path = \"../input/global-wheat-detection/train/\"\ntest_image_path = \"../input/global-wheat-detection/test/\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clean_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Unique image_ids","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['image_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clean_df['image_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing images","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_box(df, image_id, color='red'):\n df = df.where(df['image_id']== image_id)\n df = df.dropna(axis='rows')\n arr = df[\"bbox\"].to_numpy()\n\n image = cv2.imread(f'{train_image_path}/{image_id}.jpg')\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n for box in arr:\n box=box[1:-1]\n xmin,ymin,width,height= box.split(\",\")\n \n xmin = int(float(xmin))\n ymin= int(float(ymin)) \n width = int(float(width))\n height= int(float(height))\n\n xmax = xmin + width\n ymax = ymin + height\n \n color_tuple = (255,0,0)\n if color == 'blue':\n color_tuple = (0,0,255)\n \n image = cv2.rectangle(image,(xmin,ymin), (xmax,ymax),color_tuple,3)\n \n img = Image.fromarray(image)\n return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'm going to show images with bbox.\n\n`red`: train.csv\n
\n\n\n`blue`: clean train.csv\n\nYou can see wrong bbox. - Only visible to good people :)\n- I recommend viewing the image in a larger view.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n \nax.imshow(show_box(train_df, '41c0123cc'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this image, there is a hub bbox colored `red`.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"And I removed this bbox as below(`blue`). ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n \nax.imshow(show_box(clean_df, '41c0123cc', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This looks pretty good.\n\nBut there is a wheat head that is `not labeled`. A hub box that I removed is for this.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Cleaning the train.csv \nThere are several ways for cleaning. Generally,\n1. remove images have something wrong in train data : `simple`, `but inefficient`\n\n2. remove wrong bbox in images : `it seems good`, but `unmarked wheat heads is appeared.`\n\n3. remove wrong bbox and add new bbox in images : `best I think`, `but need doman knowledge & a lot of resources.`\n\n4. etc.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I decide to choose 2nd way mainly.\n\n\nAnd I removed image when unmarked wheat heads appeared too much.\n- This is very subjective. So, it may be different from your opinion.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"All explanations are over.\n\nLet's start visualization for clean bbox.\n- left side : `red` - train.csv\n- right side :`blue` - clean train.csv","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '41c0123cc'))\nax[1].imshow(show_box(clean_df, '41c0123cc', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(322.0, 626.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '2cc75e9f5'))\nax[1].imshow(show_box(clean_df, '2cc75e9f5', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(37.0, 84.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'c01a58fdb'))\nax[1].imshow(show_box(clean_df, 'c01a58fdb', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> You can see `a ladybug` in left side. about `(713.0, 634.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'c631c7fdb'))\nax[1].imshow(show_box(clean_df, 'c631c7fdb', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(545.0, 798.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '61ff5cdc2'))\nax[1].imshow(show_box(clean_df, '61ff5cdc2', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(526.0, 531.0)`","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'c6d94be4c'))\nax[1].imshow(show_box(clean_df, 'c6d94be4c', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong target bbox in left side. about `(633.0, 421.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '8bad19780'))\nax[1].imshow(show_box(clean_df, '8bad19780', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(149.0, 1.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '4cbb2b7bd'))\nax[1].imshow(show_box(clean_df, '4cbb2b7bd', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(322.0, 626.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'fade0e053'))\nax[1].imshow(show_box(clean_df, 'fade0e053', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(753.0, 768.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '9a30dd802'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'b53afdf5c'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'd96205316'))\nax[1].imshow(show_box(clean_df, 'd96205316', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(479.0, 813.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'dc7c60052'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '6106eefbc'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '83842ec14'))\nax[1].imshow(show_box(clean_df, '83842ec14', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(493.0, 533.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '42e6efaaa'))\nax[1].imshow(show_box(clean_df, '42e6efaaa', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(272.0, 0.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'fc6860020'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '9780d64f5'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '49c3e4f6e'))\nax[1].imshow(show_box(clean_df, '49c3e4f6e', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(253.0, 684.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '409a8490c'))\nax[1].imshow(show_box(clean_df, '409a8490c', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(102.0, 268.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'd7a02151d'))\nax[1].imshow(show_box(clean_df, 'd7a02151d', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(712.0, 217.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'b53afdf5c'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'a1321ca95'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'a36608629'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '5ec31deb1'))\nax[1].imshow(show_box(clean_df, '5ec31deb1', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(602.0, 131.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '834690a35'))\nax[1].imshow(show_box(clean_df, '834690a35', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> two wrong bbox in left side. about `(31.0, 393.0), (952.0, 729.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'f5f5a9d30'))\nax[1].imshow(show_box(clean_df, 'f5f5a9d30', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(499.0, 677.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'd8cae4d1b'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'd067ac2b1'))\nax[1].imshow(show_box(clean_df, 'd067ac2b1', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(0.0, 153.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '69e509038'))\nax[1].imshow(show_box(clean_df, '69e509038', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(590.0, 419.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'f24698e88'))\nax[1].imshow(show_box(clean_df, 'f24698e88', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(721.0, 768.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '4b4f6de9b'))\nax[1].imshow(show_box(clean_df, '4b4f6de9b', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(13.0, 431.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '91c7fb84e'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '7d5af5b74'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> remove images in clean data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '7c659d49a'))\nax[1].imshow(show_box(clean_df, '7c659d49a', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(243.0, 633.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, 'd60e832a5'))\nax[1].imshow(show_box(clean_df, 'd60e832a5', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(325.0, 62.0)`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '595709e55'))\nax[1].imshow(show_box(clean_df, '595709e55', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n \nax[0].imshow(show_box(train_df, '893938464'))\nax[1].imshow(show_box(clean_df, '893938464', 'blue'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> a wrong bbox in left side. about `(116.0, 177.0)`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## If this kernel is useful, please upvote!\n- I hope everyone has a good something in this competition!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}