{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **In this notebook I am going to do some basic data exploration.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"*First of all let's figure out the size of the dataset that we have for polished analysis*","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport shutil\nimport random\n\ntrain_path = \"../input/global-wheat-detection/train.csv\"\nimages_path = \"../input/global-wheat-detection/train/\"\n\n# We will load the csv data in a pandas dataframe and find its shape\ndata = pd.read_csv(train_path)\nprint(data.shape)\n\nos.mkdir(\"un-annotated\")\n\n# We will count the number of images in the train folder\ni = 0\nfor image in os.listdir(images_path):\n    i += 1\nprint(\"Total number of images are %d\", i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*As we can see that there is 43 times more data in the csv file that contains the bounding boxes for the wheat heads. This clearly states that each images has multiple bounding boxes.*\n\n*Now lets have a look at the kind of data that we have in the csv file*","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The file contains the width and the height of the image. The **bbox** column in the dataframe represents the dimensions of the bounding boxes in the format of [**xmin, ymin, width, height**]\n\nThe problem here is that the values in the bbox column is a string nor a list. We will have to convert it into a list.\n\nBut, an even more considerable problem is that, not all of the images in the train set have bounding boxes, but they donot affect the functioning of the model as they have no entries in the CSV file.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Splitting the data in the train and test set","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**In order to split the data in the train and the test sets, we take 100 unique image ids in the train csv and pick all the entries for that image id to move that into another cv file\n**\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ids = random.sample(list(data[\"image_id\"].unique()), 650)\ntest = pd.DataFrame()\nfor id in test_ids:\n  test = test.append(data[data['image_id'] == id])\n  data.drop(data[data['image_id'] == id].index, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Saving the test data file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test.to_csv(\"/kaggle/working/un-annotated/test.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***Now its time to address the previously mentioned problem that states that the bbox data is of type string, we need it to be of type list and also we can change the format of the csv file, both train and test, to follow the genric format for object detection data files. This file then can also be directly consumed by tensorflow object detection API***","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Code for converting the train data file with columns [**image_id, height, width, xmin, ymin, xmax, ymax**]","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"values = []\nfilename = []\n\n# Extracting the bbox values from all the rows, removing the brackets and converting them to list and finally appending the to another list\nfor index, row in data.iterrows():\n  values.append(row.bbox.strip('[]').split(\", \"))\n  filename.append(row.image_id + '.jpg')\n\n# Type converting values from string to float\n# Calculating xmax and ymax by adding width and height\n# Saving the value in the corresponding list\nxmin = []\nymin = []\nxmax = []\nymax = []\nfor value in values:\n  xmin.append(float(value[0]))\n  ymin.append(float(value[1]))\n  xmax.append(float(value[2]) + float(value[0]))\n  ymax.append(float(value[3]) + float(value[1]))\n\n# Preparing anew dataframe in the required format\nprocessed_data = {}\nprocessed_data[\"filename\"] = filename\nprocessed_data[\"width\"] = data['width']\nprocessed_data[\"height\"] = data['height']\ndata['class'] = 'wheat'\nprocessed_data[\"class\"] = data['class']\nprocessed_data[\"xmin\"] = xmin\nprocessed_data[\"ymin\"] = ymin\nprocessed_data[\"xmax\"] = xmax\nprocessed_data[\"ymax\"] = ymax\nprocessed_data = pd.DataFrame(processed_data)\n\n# Saving the newly processed data in a file\nprocessed_data.to_csv(\"/kaggle/working/un-annotated/train_processed.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***Converting the test data in the required format***","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv(\"/kaggle/working/un-annotated/test.csv\")\nvalues = []\nfilename = []\n\n# Extracting the bbox values from all the rows, removing the brackets and converting them to list and finally appending the to another list\nfor index, row in data.iterrows():\n  values.append(row.bbox.strip('[]').split(\", \"))\n  filename.append(row.image_id + '.jpg')\n\n# Type converting values from string to float\n# Calculating xmax and ymax by adding width and height\n# Saving the value in the corresponding list\nxmin = []\nymin = []\nxmax = []\nymax = []\nfor value in values:\n  xmin.append(float(value[0]))\n  ymin.append(float(value[1]))\n  xmax.append(float(value[2]) + float(value[0]))\n  ymax.append(float(value[3]) + float(value[1]))\n\n# Preparing anew dataframe in the required format\nprocessed_data = {}\nprocessed_data[\"filename\"] = filename\nprocessed_data[\"width\"] = data['width']\nprocessed_data[\"height\"] = data['height']\ndata['class'] = 'wheat'\nprocessed_data[\"class\"] = data['class']\nprocessed_data[\"xmin\"] = xmin\nprocessed_data[\"ymin\"] = ymin\nprocessed_data[\"xmax\"] = xmax\nprocessed_data[\"ymax\"] = ymax\nprocessed_data = pd.DataFrame(processed_data)\n\n# Saving the newly processed data in a file\nprocessed_data.to_csv(\"/kaggle/working/un-annotated/test_processed.csv\", index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}