{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Importing Libraries","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:23:13.735111Z","start_time":"2020-05-10T15:23:13.139469Z"},"trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport csv\nimport glob\nimport pandas as pd\nimport numpy as np\nimport random\nimport itertools\nfrom collections import Counter\nfrom math import ceil\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this kernel, I present some utility functions to do some sanity check on images, as well as some functions that you can reuse for future projects when you want to plot multiple images in a grid. A sneak peek of how a multiple bounding box plot is as such:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![sample](https://i.ibb.co/9GXMpWT/img.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Utilities","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Utility functions are stored here, they are useful and feel free to add these into your arsenal.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:27:14.796817Z","start_time":"2020-05-10T15:27:14.785846Z"},"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def skip_csv_header(file):\n    has_header = csv.Sniffer().has_header(file.read(1024))\n    file.seek(0)\n    if has_header:\n        next(file)\n\n\ndef total_image_list(image_folder_path):\n    total_img_list = [os.path.basename(img_path_name) for img_path_name in glob.glob(os.path.join(image_folder_path, \"*.jpg\"))]\n    return total_img_list\n\ndef draw_rect(img, bboxes, color=None):\n    img = img.copy()\n    bboxes = bboxes[:, :4]\n    bboxes = bboxes.reshape(-1, 4)\n    for bbox in bboxes:\n        pt1, pt2 = (bbox[0], bbox[1]), (bbox[2], bbox[3])\n        pt1 = int(pt1[0]), int(pt1[1])\n        pt2 = int(pt2[0]), int(pt2[1])\n        img = cv2.rectangle(img.copy(), pt1, pt2, color, int(max(img.shape[:2]) / 200))\n    return img\n\ndef plot_multiple_img(img_matrix_list, title_list, ncols, main_title=\"\"):\n    fig, myaxes = plt.subplots(figsize=(20, 15), nrows=ceil(len(img_matrix_list) / ncols), ncols=ncols, squeeze=False)\n    fig.suptitle(main_title, fontsize = 30)\n    fig.subplots_adjust(wspace=0.3)\n    fig.subplots_adjust(hspace=0.3)\n    for i, (img, title) in enumerate(zip(img_matrix_list, title_list)):\n        myaxes[i // ncols][i % ncols].imshow(img)\n        myaxes[i // ncols][i % ncols].set_title(title, fontsize=15)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading and Loading the Dataset","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:23:18.556841Z","start_time":"2020-05-10T15:23:18.425088Z"},"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/global-wheat-detection/train.csv\")  \nimage_folder_path = \"/kaggle/input/global-wheat-detection/train/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I personally like to expand the bounding box coordinates into the form of **x_min, y_min, x_max, y_max**, but currently they are stored in a list of **[x_min,y_min, width of bbox, height of bbox]**. So the next portion will help to expand them out. **This is a personal preference, in actual fact you do not need to do this, it is easier for me to normalize the bboxes**.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:23:28.582189Z","start_time":"2020-05-10T15:23:20.780108Z"},"trusted":true},"cell_type":"code","source":"train['bbox'] = train['bbox'].apply(lambda x: x[1:-1].split(\",\"))\ntrain['x_min'] = train['bbox'].apply(lambda x: x[0]).astype('float32')\ntrain['y_min'] = train['bbox'].apply(lambda x: x[1]).astype('float32')\ntrain['width'] = train['bbox'].apply(lambda x: x[2]).astype('float32')\ntrain['height'] = train['bbox'].apply(lambda x: x[3]).astype('float32')\ntrain = train[['image_id','x_min', 'y_min', 'width', 'height']]\ntrain[\"x_max\"] = train.apply(lambda col: col.x_min + col.width, axis=1)\ntrain[\"y_max\"] = train.apply(lambda col: col.y_min + col.height, axis = 1)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Range Checking on Bounding Box Coordinates","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Furthermore, due to python's internal floating problems, there may be weird values like negative or values that adds up to be more than 1024 in `x_max, y_max`. We need to be careful here. \n\n**This is a serious problem that one can run into when you Normalize the bounding box, it may exceed 1 and this will cause an error especially if you decide to augment the images as well.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train[\"x_max\"] > 1024]\ntrain[train[\"y_max\"] > 1024]\ntrain[train[\"x_min\"] < 0]\ntrain[train[\"y_min\"] < 0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sole reason that for eg row 31785 has `x_max` more than 1024 is because of the original dataset's labelling. Let's look at the respective problematic rows. For example, in row 31785, the `x_min` provided is 873.200012, and when you add that to the width being 150.800003, it gives you 1024.000015, which exceeds the image size already. So you have to round down. And as far as I feel, bounding boxes, when de-normalized, should necessary be in integer. But this is just my opinion. Let's change these problematic values to 1024","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x_max = np.array(train[\"x_max\"].values.tolist())\ny_max = np.array(train[\"y_max\"].values.tolist())\ntrain[\"x_max\"] = np.where(x_max > 1024, 1024, x_max).tolist()\ntrain[\"y_max\"] = np.where(y_max > 1024, 1024, y_max).tolist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can delete width and height columns because we do not need them, it can be easily pulled out from the images itself.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:23:51.619723Z","start_time":"2020-05-10T15:23:51.608205Z"},"trusted":true},"cell_type":"code","source":"del train[\"width\"]\ndel train[\"height\"]\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I assign a class \"1\" which is the label wheat. It may be useful later on should we wish to add in images with no wheat inside the image.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:23:55.105905Z","start_time":"2020-05-10T15:23:55.101893Z"},"trusted":true},"cell_type":"code","source":"train[\"class\"] = \"1\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check if image extensions are all jpg","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First, we check if all images in the train folder are all in **.jpg** format. It is better to check because if there are a mixture of image type, we may face troubles later on.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:24:04.563766Z","start_time":"2020-05-10T15:24:04.552795Z"},"trusted":true},"cell_type":"code","source":"def check_file_type(image_folder_path):\n    extension_type = []\n    file_list = os.listdir(image_folder_path)\n    for file in file_list:\n        extension_type.append(file.rsplit(\".\", 1)[1].lower())\n    print(Counter(extension_type).keys())\n    print(Counter(extension_type).values())\n    \ncheck_file_type(image_folder_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Good, seems like all our images in the folder are of **.jpg** format. Next, it is better to append **.jpg** behind all the **image_id** in the dataframe. This will make us manipulate the data easier later.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:25:04.020873Z","start_time":"2020-05-10T15:25:03.951036Z"},"trusted":true},"cell_type":"code","source":"## replace image_id with .jpg behind the image_id\n# image_id_list = train[\"image_id\"].tolist()\n# image_id_append_jpg = []\n# for image_id in image_id_list:\n#     image_id_append_jpg.append(image_id + \".jpg\")\n# train[\"image_id\"] = image_id_append_jpg\n# train.head()\n\n\n## Alternatively like Rohit suggested, an one liner will do the trick\n\ntrain[\"image_id\"] = train[\"image_id\"].apply(lambda x: str(x) + \".jpg\")\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:25:04.965887Z","start_time":"2020-05-10T15:25:04.949901Z"},"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T11:45:33.459017Z","start_time":"2020-05-10T11:45:32.877548Z"},"trusted":true},"cell_type":"code","source":"train[\"image_id\"] = train[\"image_id\"].astype(\"str\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.to_csv(\"wheat.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check if there are corrupted images and all images are 1024 by 1024","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Most people will use `df['width'].unique() == df['height'].unique() == [1024]` to check if all images are of 1024x1024 resolution; But we will not be 100% sure if its true in the training folder. So we won't use the same way here.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T16:03:48.307185Z","start_time":"2020-05-10T16:03:48.302222Z"},"trusted":true},"cell_type":"code","source":"def check_image_size(image_folder_path):\n    total_img_list = glob.glob(os.path.join(image_folder_path,\"*\"))\n    counter = 0\n    for image in tqdm(total_img_list, desc = \"Checking in progress\"):\n        img = cv2.imread(image)\n        height, width = img.shape[1], img.shape[0]\n        if not (height == 1024 and width == 1024):\n            counter = counter + 1\n    return counter \n        \n        ","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T16:04:30.587206Z","start_time":"2020-05-10T16:03:48.465394Z"},"trusted":true},"cell_type":"code","source":"check_image_size(image_folder_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great, indeed all our images are of 1024x1024 in size. And the good thing is, this code also helps us to check for corrupted images as well, so if there is a corrupted image, it will definitely show up that the counter is non zero. And from there you can further check which image is the one causing problem.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Sanity Check between train csv and train images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We will write a function to check if the number of **unique image_ids** match the number of unique **images** in the folder.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:26:25.634413Z","start_time":"2020-05-10T15:26:25.548639Z"},"trusted":true},"cell_type":"code","source":"## our new dataset\nwheat = pd.read_csv(\"wheat.csv\") \nimage_folder_path = \"/kaggle/input/global-wheat-detection/train/\"\nimage_annotation_file = \"wheat.csv\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wheat.head()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:26:26.690066Z","start_time":"2020-05-10T15:26:26.684082Z"},"trusted":true},"cell_type":"code","source":"def sanity_tally(image_folder_path, image_annotation_file):\n    img_dict = {}\n    with open(image_annotation_file, \"r\") as file:\n        skip_csv_header(file)\n        for row in file:\n            try:\n                image_name, x_min, y_min, x_max, y_max, class_idx = row.split(\",\")\n                if image_name not in img_dict:\n                    img_dict[image_name] = list()\n                img_dict[image_name].append(\n                    [float(x_min), float(y_min), float(x_max), float(y_max), int(class_idx)]\n                )\n            except ValueError:\n                print(\"Could not convert float to string, likely that your data has empty values.\")\n        \n    img_annotation_list = [*img_dict]\n    total_img_list = total_image_list(image_folder_path)\n    if set(img_annotation_list) == set(total_img_list):\n        print(\"Sanity Check Status: True\")\n    else:\n        print(\"Sanity Check Status: Failed. \\nThe elements in wheat/train.csv but not in the train image folder is {}. \\nThe elements in train image folder but not in wheat/train.csv is {}\".format(\n                set(img_annotation_list) - set(total_img_list), set(total_img_list) - set(img_annotation_list)))\n        return list(set(img_annotation_list) - set(total_img_list)), list(set(total_img_list) - set(img_annotation_list))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:27:19.168619Z","start_time":"2020-05-10T15:27:18.826034Z"},"trusted":true},"cell_type":"code","source":"set_diff1, set_diff2 = sanity_tally(image_folder_path, image_annotation_file = image_annotation_file)\n\nprint(\"There are {} images without annotations in the train/wheat.csv\".format(len(set_diff2)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see from the above, there are 49 images without bounding box annotations because they do not have wheats in the image, and hence did not appear in the **train.csv**. It might be an idea that we can put these 49 images inside the train.csv and label them as 0.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Plotting Multiple Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here we define a nice function that is useful not only for this competition, but for similar project as well. Note that we used our utility function here to plot them. One can tune the parameters accordingly.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:04:54.304789Z","start_time":"2020-05-10T15:04:54.297782Z"},"trusted":true},"cell_type":"code","source":"def plot_random_images(image_folder_path, image_annotation_file, num = 12):\n    img_dict = {}\n    with open(image_annotation_file, \"r\") as file:\n        skip_csv_header(file)\n        for row in file:\n            try:\n                image_name, x_min, y_min, x_max, y_max, class_idx = row.split(\",\")\n                if image_name not in img_dict:\n                    img_dict[image_name] = list()\n                img_dict[image_name].append(\n                    [float(x_min), float(y_min), float(x_max), float(y_max), int(class_idx)]\n                )\n            except ValueError:\n                print(\"Could not convert float to string, likely that your data has empty values.\")\n\n    # randomly choose 12 images to plot\n    img_files_list = np.random.choice(list(img_dict.keys()), num)\n    print(\"The images' names are {}\".format(img_files_list))\n    img_matrix_list = []\n    \n    for img_file in img_files_list:\n        image_file_path = os.path.join(image_folder_path, img_file)\n        img = cv2.imread(image_file_path)[:,:,::-1]  \n        img_matrix_list.append(img)\n\n    \n    return plot_multiple_img(img_matrix_list, title_list = img_files_list, ncols = 4, main_title=\"Wheat Images\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we see a nice grid of 12 images plotted.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:04:56.958859Z","start_time":"2020-05-10T15:04:54.560035Z"},"trusted":true},"cell_type":"code","source":"plot_random_images(image_folder_path, image_annotation_file, num = 12)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plotting Multiples Images with Bounding Boxes","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In object detection with bounding boxes, it is always a good idea to randomly plot some images with their bounding boxes to check for any awry bounding box coordinates. Although I have to say that in this particular competition, there are quite a lot of images with many bounding boxes and hence you have to scrutinize clearly.","execution_count":null},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:05:03.610126Z","start_time":"2020-05-10T15:05:03.599127Z"},"trusted":true},"cell_type":"code","source":"def random_bbox_check(image_folder_path, image_annotation_file, num = 12):\n    img_dict = {}\n    labels = [\"wheat\", \"no wheat\"]\n    with open(image_annotation_file, \"r\") as file:\n        skip_csv_header(file)\n        for row in file:\n            try:\n                image_name, x_min, y_min, x_max, y_max, class_idx = row.split(\",\")\n                if image_name not in img_dict:\n                    img_dict[image_name] = list()\n                img_dict[image_name].append(\n                    [float(x_min), float(y_min), float(x_max), float(y_max), int(class_idx)]\n                )\n            except ValueError:\n                print(\"Could not convert float to string, likely that your data has empty values.\")\n\n    # randomly choose 12 image.\n    img_files_list = np.random.choice(list(img_dict.keys()), num)\n    print(\"The images' names are {}\".format(img_files_list))\n    image_file_path_list = []\n\n    bbox_list = []\n    img_matrix_list = []\n    random_image_matrix_list = []\n    \n    for img_file in img_files_list:\n        image_file_path = os.path.join(image_folder_path, img_file)\n        img = cv2.imread(image_file_path)[:,:,::-1]  \n        height, width, channels = img.shape\n        bbox_list.append(img_dict[img_file])\n        img_matrix_list.append(img)\n\n    \n    final_bbox_list = []\n    for bboxes, img in zip(bbox_list, img_matrix_list):\n        final_bbox_array = np.array([])\n        #bboxes is a 2d array [[...], [...]]\n        for bbox in bboxes:\n            bbox = np.array(bbox).reshape(1,5)\n            final_bbox_array = np.append(final_bbox_array, bbox)\n        final_bbox_array = final_bbox_array.reshape(-1,5)\n        random_image = draw_rect(img.copy(), final_bbox_array.copy(), color = (255,0,0))\n        random_image_matrix_list.append(random_image)\n    plot_multiple_img(random_image_matrix_list, title_list = img_files_list, ncols = 4, main_title=\"Bounding Box Wheat Images\")    \n    \n\n","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-05-10T15:05:06.915322Z","start_time":"2020-05-10T15:05:03.72127Z"},"trusted":true},"cell_type":"code","source":"random_bbox_check(image_folder_path, image_annotation_file)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentations","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Augmentation is an important technique to artifically boost your data size. In particular, when the dataset is small, augmentation prior to training the model will help the network to learn better.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Albumentations\nimport albumentations as A","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_folder_path = \"/kaggle/input/global-wheat-detection/train/\"\nchosen_image = cv2.imread(os.path.join(image_folder_path, \"aa6b16251.jpg\"))[:,:,::-1]\nplt.imshow(chosen_image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below are some snippets of augmentation types you can use, interestingly, Albumentation offers `RandomSunFlare` and `RandomFog`; although all the images seem to be taken in a very good lighting, but it might not be that bad an idea since in the real world, images of wheat may taken in different weather conditions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"albumentation_list = [A.RandomSunFlare(p=1),A.RandomFog(p=1), A.RandomBrightness(p=1),A.HorizontalFlip(p=1),A.VerticalFlip(p=1), A.RandomContrast(limit = 0.5,p = 1), A.RandomBrightness(p=1), \n                       A.HueSaturationValue(p=1,hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=50)]\n\nimg_matrix_list = []\nbboxes_list = []\nfor aug_type in albumentation_list:\n    img = aug_type(image = chosen_image)['image']\n    img_matrix_list.append(img)\n\nimg_matrix_list.insert(0,chosen_image)    \n\ntitles_list = [\"Original\",\"RandomSunFlare\",\"RandomFog\",\"RandomBrightness\",\"HorizontalFlip\", \"VerticalFlip\", \"RandomContrast\",\"HSV\"]\n\n##reminder of helper function\ndef plot_multiple_img(img_matrix_list, title_list, ncols, main_title=\"\"):\n    fig, myaxes = plt.subplots(figsize=(20, 15), nrows=2, ncols=ncols, squeeze=False)\n    fig.suptitle(main_title, fontsize = 30)\n    fig.subplots_adjust(wspace=0.3)\n    fig.subplots_adjust(hspace=0.3)\n    for i, (img, title) in enumerate(zip(img_matrix_list, title_list)):\n        myaxes[i // ncols][i % ncols].imshow(img)\n        myaxes[i // ncols][i % ncols].set_title(title, fontsize=15)\n    plt.show()\n    \nplot_multiple_img(img_matrix_list, titles_list, ncols = 4,main_title=\"Different Types of Augmentations\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" **If you liked the visualizations, please upvote to motivate me further and stay tuned for the next kernel which will be on training the models.**","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}