{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Gloabl Wheat Detection - EDA\n\n![GWD](https://albumizr.com/ia/68bce2e2687097c61faee360dc1dce79.jpg)\n\n> In this competition, you’ll detect wheat heads from outdoor images of wheat plants, including wheat datasets from around the globe. Using worldwide data, you will focus on a generalized solution to estimate the number and size of wheat heads. To better gauge the performance for unseen genotypes, environments, and observational conditions, the training dataset covers multiple regions. You will use more than 3,000 images from Europe (France, UK, Switzerland) and North America (Canada). The test data includes about 1,000 images from Australia, Japan, and China.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport cv2\nimport re\nfrom tqdm.notebook import tqdm\nfrom PIL import Image\nimport hashlib\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\nsns.set()\n\nDIR_INPUT = '/kaggle/input/global-wheat-detection'\nDIR_TRAIN_IMAGES = f'{DIR_INPUT}/train'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(f'{DIR_INPUT}/train.csv')\ntrain_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image meta\n- We have 3373 unique images in the train dataset\n- It seems all of the images hav the same size (1024x1024)\n- Minimum number of bbox per image: 0\n- Maximum number of bbox per image: 116\n- Average number of bbox per image: 43.82\n- No duplicated images (in train)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['image_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['height'].value_counts(), train_df['width'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_hash(im):\n    md5 = hashlib.md5()\n    md5.update(np.array(im).tostring())\n    \n    return md5.hexdigest()\n    \ndef get_image_meta(image_id, image_src, dataset='train'):\n    im = Image.open(image_src)\n    extrema = im.getextrema()\n\n    meta = {\n        'image_id': image_id,\n        'dataset': dataset,\n        'hash': calculate_hash(im),\n        'r_min': extrema[0][0],\n        'r_max': extrema[0][1],\n        'g_min': extrema[1][0],\n        'g_max': extrema[1][1],\n        'b_min': extrema[2][0],\n        'b_max': extrema[2][1],\n        'height': im.size[0],\n        'width': im.size[1],\n        'format': im.format,\n        'mode': im.mode\n    }\n    return meta","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = []\n\nfor i, image_id in enumerate(tqdm(train_df['image_id'].unique(), total=train_df['image_id'].unique().shape[0])):\n    data.append(get_image_meta(image_id, DIR_TRAIN_IMAGES + '/{}.jpg'.format(image_id)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta_df = pd.DataFrame(data)\nmeta_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Duplications"},{"metadata":{"trusted":true},"cell_type":"code","source":"duplicates = meta_df.groupby(by='hash')[['image_id']].count().reset_index()\nduplicates = duplicates[duplicates['image_id'] > 1]\nduplicates.reset_index(drop=True, inplace=True)\n\nduplicates = duplicates.merge(meta_df[['image_id', 'hash']], on='hash')\n\nduplicates.head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great! No duplications!"},{"metadata":{},"cell_type":"markdown","source":"## Extract bounding box data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['x'] = -1\ntrain_df['y'] = -1\ntrain_df['w'] = -1\ntrain_df['h'] = -1\n\ndef expand_bbox(x):\n    r = np.array(re.findall(\"([0-9]+[.]?[0-9]*)\", x))\n    if len(r) == 0:\n        r = [-1, -1, -1, -1]\n    return r\n\ntrain_df[['x', 'y', 'w', 'h']] = np.stack(train_df['bbox'].apply(lambda x: expand_bbox(x)))\ntrain_df.drop(columns=['bbox'], inplace=True)\ntrain_df['x'] = train_df['x'].astype(np.float)\ntrain_df['y'] = train_df['y'].astype(np.float)\ntrain_df['w'] = train_df['w'].astype(np.float)\ntrain_df['h'] = train_df['h'].astype(np.float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.groupby(by='image_id')['source'].count().agg(['min', 'max', 'mean'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"source = train_df['source'].value_counts()\nsource","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure(data=[\n    go.Pie(labels=source.index, values=source.values)\n])\n\nfig.update_layout(title='Source distribution')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_images(image_ids):\n    \n    col = 5\n    row = min(len(image_ids) // col, 5)\n    \n    fig, ax = plt.subplots(row, col, figsize=(16, 8))\n    ax = ax.flatten()\n\n    for i, image_id in enumerate(image_ids):\n        image = cv2.imread(DIR_TRAIN_IMAGES + '/{}.jpg'.format(image_id))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        ax[i].set_axis_off()\n        ax[i].imshow(image)\n        ax[i].set_title(image_id)\n        \ndef show_image_bb(image_data):\n    \n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n    \n    image = cv2.imread(DIR_TRAIN_IMAGES + '/{}.jpg'.format(image_data.iloc[0]['image_id']))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n    for i, row in image_data.iterrows():\n        \n        cv2.rectangle(image,\n                      (int(row['x']), int(row['y'])),\n                      (int(row['x']) + int(row['w']), int(row['y']) + int(row['h'])),\n                      (220, 0, 0), 3)\n\n    ax.set_axis_off()\n    ax.imshow(image)\n    ax.set_title(image_id)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_images(train_df.sample(n=15)['image_id'].values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image_bb(train_df[train_df['image_id'] == '5e0747034'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image_bb(train_df[train_df['image_id'] == '5b13b8160'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image_bb(train_df[train_df['image_id'] == '1f2b1a759'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}