{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Wheat Head Detection using Pyspark, Python, VGGUnet\n\n### `Author : Deepika Sharma`                                           `Date : June 2020 - July 2020`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### `Folder structure`:\n1. Input(train) data 3422 images.\n2. Train csv : CSV containing information of bounding boxes around wheat spikes in each image.\n3. Test : Images on which prediction has to be made, prediction output has to be in the format present in submission csv.\n4. Submission csv : Format in which submission has to be made. It has 2 columns : 1. image name, 2. probability of bounding box containing wheat spike, correspnding co-orinates x,y,h,w ... prob and x,y,h,w for each predicted bbox.\n5. Intermediate folder (temporary folder to keep images to be processed batchwise). `Batchsize needs to be changed based on your system specifications.`\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### `Steps`:\n###### Step 1: In this kernal, I'll be using pyspark for data preparation as it requires manipulating 3000 images (train size). ~300 for kept for testing. Train csv does not contain information for all images present in Raw data, pls do not worry if numbers do not sum up.\n\nFor training, each image is converted in to binary image, where 0's represent background, 1's represent wheat spikes \nEncoding is done using bounding boxes coordinates present in the Train csv.\n\n\n######  Step 2: Model (Keras vgg_unet) training on ~3000 images with n_classes=2 ,  input_height=1024, input_width=1024. \nimages are resized to reduced computation on CPU.\n\n###### Step 3: Prediction on ~400 images to remove background (0) and only predict wheat spikes as foreground (1)\n\n###### Step 4: Blob detection using blob_dog (difference of Gaussian) to detect blobs in model predicted segments for wheat spikes.\n\n###### Step 5: IoU calculation, and computation of probability of each detection in each image.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Step 1: Data Preparation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Import libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install keras_segmentation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pyspark","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import cv2\nimport math\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n\n### Scikit learn libraries ### \nfrom skimage.color import rgb2gray\nfrom skimage.feature import blob_dog\nfrom keras.models import model_from_json\n\n\n### Pyspark libraries ###\nimport os,gc\nimport shutil\nimport pyspark\nimport itertools\nfrom operator import add\nfrom pyspark.sql.types import *\nfrom pyspark import SparkContext\nfrom pyspark.sql import DataFrame\nfrom pyspark.sql.functions import udf\nfrom pyspark.ml.image import ImageSchema","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"file = pd.read_csv(\"/kaggle/input/global-wheat-detection/train.csv\")  # This file contains bbox co-ordinates of wheat spikes for each image.\n\n# Train csv has multiple rows for each image depending upon number of wheat spikes in each image.\nids = [file.iloc[i][0] for i in range(file.shape[0])] \n\n# Factor by which we will resize each image as each raw image has size 1024*1024, which will be too large for network to train on to.\nfactor = 2; size = 512; batch_size = 500; x_axis_gaps = 5; coords_dist_th = 15; N = 3100; N_50 = N+50","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = os.listdir(\"/kaggle/input/global-wheat-detection/train\")[0:3]\n\nfor img_name in sample:\n    image = cv2.imread('/kaggle/input/global-wheat-detection/train/'+ img_name)\n    \n    # Get image ids for each image to access the segmentation co-ordinates\n    image_ids = [l for l,val in enumerate(ids) if val == str(img_name.split(\".\")[0])]\n    list_of_coords = [[int(float(val)) for val in file.iloc[v][3][1:len(file.iloc[v][3])-1].split(\",\")] for v in image_ids] \n            \n    for l in list_of_coords:        \n        image_annotated = cv2.rectangle(image,(int(l[0]),int(l[1])),(int((l[0]+l[2])),int((l[1]+l[3]))),(255,0,0),4)\n\n    fig, axes = plt.subplots(1, 2, figsize=(12, 6))#, sharex=True, sharey=True)\n    ax = axes.ravel()\n    image = cv2.imread('/kaggle/input/global-wheat-detection/train/'+ img_name)    \n    ax[0].imshow(cv2.cvtColor(image,cv2.COLOR_RGB2BGR))\n    ax[1].imshow(cv2.cvtColor(image_annotated,cv2.COLOR_RGB2BGR))\n        \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Converting images to binary for training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists('train_annotated'):\n        os.makedirs('train_annotated')\n        \ndef data_prep(img_name):\n    '''\n    img_name : input image name \n    \n    '''\n    \n    # Initialising a balck n image of zeros which we will later manipulate using annotation information.\n    binary_img = np.zeros((size, size, 3))\n    \n    image = cv2.imread('spark_temp/'+ img_name.split(\".\")[0] + \".png\") #reading actual image \".png\")\n    \n    # Get image ids for each image to access the segmentation co-ordinates\n    image_ids = [l for l,val in enumerate(ids) if val == str(img_name.split(\".\")[0])]\n    list_of_coords = [[int(float(val)) for val in file.iloc[v][3][1:len(file.iloc[v][3])-1].split(\",\")] for v in image_ids] \n    if len(list_of_coords) >= 0:#>1\n        for l in list_of_coords:\n            # Getting segment for wheat spike, co-ordinates adjusted \n            cropped = image[int(l[1]/factor):int((l[1]+l[3])/factor), int(l[0]/factor):int((l[0]+l[2])/factor)]   \n\n            for i,x in enumerate(range(int(l[1]/factor),int((l[1]+l[3])/factor))):\n                for j,y in enumerate(range(int(l[0]/factor),int((l[0]+l[2])/factor))):\n                    x = min(image.shape[0]-1,x)\n                    y = min(image.shape[0]-1,y)\n                    try:\n                        binary_img[x,y][0] = (image[x,y][0]/np.max(cropped[:,:,0]))*50\n                        # normalising the image area where wheat spike exists\n\n                    except:pass\n\n        binary_img[binary_img < np.mean(binary_img)] = 0\n        binary_img[binary_img >= np.mean(binary_img)] = 1\n#         binary_img = cv2.resize(binary_img, dsize=(1024, 1024), interpolation=cv2.INTER_NEAREST)\n        cv2.imwrite(\"train_annotated/\" + str(img_name.split(\".\")[0]) + \".png\",binary_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's Spark it up!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### `Setting up Spark Context`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_spark():   \n    sc=SparkContext(master=\"local[15]\") # number[15] can be changed based on your system specification.\n#     print(sc.binaryFiles(\"spark_temp/*.png\"))\n    image_df = sc.binaryFiles(\"spark_temp\") # Reading images into Spark context\n    \n#     pyspark.sql.udf.UDFRegistration.register(name=\"data_prep\", f = data_prep, returnType=StringType()) #registering UDF \n    # such that Spark context recongises the function used for data preparation.\n    \n    job = [data_prep(x[0].split(\"/\")[-1]) for idx,x in enumerate(image_df.take(batch_size))]\n    shutil.rmtree('spark_temp') #remove the folder once 'one batch' is complete to avoid Spark remembering \n    # indices for images it is done with.\n    sc.stop() # stop the spark context else Spark will have unnecessary information cached whcih we do not require anymore.\n    gc.collect() # remove any other cache which memory might have been holding.\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_list = os.listdir(\"/kaggle/input/global-wheat-detection/train/\")[0:N]\ninput_seq = sorted(list(set([val for val in range(0, len(input_list),batch_size)]+ [len(input_list)])))\n\ndef image_transfer(value):\n    if not os.path.exists('spark_temp'):\n        os.makedirs('spark_temp')\n    image = cv2.imread('/kaggle/input/global-wheat-detection/train/'+value)\n    cv2.imwrite(\"spark_temp/\" + value.split(\".\")[0]+\".png\",cv2.resize(image,(size,size)))#cv2.resize(image,(size,size))\n    \n    \nfor val in range(len(input_seq)-1):\n    img_batch = input_list[input_seq[val]:input_seq[val]+batch_size]\n    [image_transfer(val) for val in img_batch]\n  \n    run_spark()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Data Preparation is Done!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Step 2: Model Training ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"`Lets' do a bit checking on our data.`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists('train_temp'):\n        os.makedirs('train_temp')\ndef image_transfer_(value):      \n    cv2.imwrite(\"train_temp/\" + value.split(\".\")[0]+\".png\",cv2.resize(cv2.imread('/kaggle/input/global-wheat-detection/train/'+value.split(\".\")[0]+\".jpg\"),(size,size)))\n    \nfirst_3000 = os.listdir(\"train_annotated\")\ntask = [image_transfer_(val) for val in first_3000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"error_list = [val if np.max(cv2.imread('train_annotated/'+val)) > 1 else None for val in os.listdir(\"train_annotated\")]\nerror_list = [val for val in error_list if val != None]\nprint(error_list)\n[os.remove(\"train_temp/\"+val) for val in error_list]\n[os.remove(\"train_annotated/\"+val) for val in error_list]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras_segmentation.models.unet import vgg_unet\nmodel = vgg_unet(n_classes=2 ,  input_height=size, input_width=size)\nmodel.train(\n    train_images =  \"train_temp/\",#train\n    train_annotations = \"train_annotated/\",n_classes = 2,epochs=10,steps_per_epoch=5,#annotations_prepped_train_v3\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" #saving model to disk\n\nfrom keras.models import model_from_json\nmodel_json = model.to_json()\n\nwith open(\"model.json\", \"w\") as json_file:\n    json_file.write(model_json)\nmodel.save_weights(\"model.h5\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 3: Segmentation prediction \n## Step 4: Blob Detection\n## Step 5: IoU calculation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists('test_'):\n        os.makedirs('test_')\ndef test_image_transfer(value):      \n    cv2.imwrite(\"test_/\" + value.split(\".\")[0]+\".png\",cv2.resize(cv2.imread('/kaggle/input/global-wheat-detection/test/'+value.split(\".\")[0]+\".jpg\"),(size,size)))\n    \ntest_imgs = os.listdir(\"/kaggle/input/global-wheat-detection/test\")\ntask = [test_image_transfer(val) for val in test_imgs]\n\n\nif not os.path.exists('train_0'):\n        os.makedirs('train_0')\ndef train_image_transfer(value):      \n    cv2.imwrite(\"train_0/\" + value.split(\".\")[0]+\".png\",cv2.resize(cv2.imread('/kaggle/input/global-wheat-detection/train/'+value.split(\".\")[0]+\".jpg\"),(size,size)))\n    \ntrain_imgs = os.listdir(\"/kaggle/input/global-wheat-detection/train/\")[N:N_50]\ntask = [train_image_transfer(val) for val in train_imgs]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings,os\nwarnings.simplefilter(\"ignore\")\nsubmission = {\"image_id\" : [], \"width\" : [], \"height\" : [],\"bbox\" : [], \"Prob\":[]}\n\nfor idx,img_name in enumerate(os.listdir(\"test_\")[0:5]):\n    if not os.path.exists('segmentation'):\n        os.makedirs('segmentation')\n    if not os.path.exists('Output'):\n        os.makedirs('Output')\n        \n    \n    out = model.predict_segmentation(\n    inp=\"test_/\"+ img_name,\n    out_fname=\"segmentation/\" + img_name.split(\".\")[0] + \".png\" \n    )  \n    \n    out_image = cv2.imread('segmentation/'+ img_name.split(\".\")[0] + \".png\")\n    _image = cv2.imread('test_/'+ img_name.split(\".\")[0] + \".png\")\n#     print(out_image)\n#     out_image = cv2.resize(out_image,(512,512))\n\n    ## Step 4:Blob detection\n    image_gray = rgb2gray(out_image)\n    blobs_dog = blob_dog(image_gray, max_sigma=30, threshold=.10)\n    \n    try:\n        blobs_dog[:, 2] = blobs_dog[:, 2] * math.sqrt(2)\n        submission[\"image_id\"].append(img_name.split(\".\")[0])\n        submission[\"width\"].append(size); submission[\"height\"].append(size)\n        submission[\"bbox\"].append([str([int(blob[0]), int(blob[1]),int(blob[2]), int(blob[2])]) for blob in blobs_dog])\n        submission[\"Prob\"].append(.50)\n    except:\n        submission[\"image_id\"].append(img_name.split(\".\")[0])\n        submission[\"width\"].append(size); submission[\"height\"].append(size)\n        submission[\"bbox\"].append([])\n        submission[\"Prob\"].append(1)\n\n    \n    blob_list_sorted = sorted([[int(blob[0]), int(blob[1]),int(blob[2])] for blob in blobs_dog])\n    len_val = len(blob_list_sorted)\n    for i,val in enumerate(range(0,len(blob_list_sorted))):\n        for j in range(i+1,len(blob_list_sorted)-2):\n            if max(i,j) < len_val and max([abs(np.diff(x)) for x in zip(blob_list_sorted[j][0:2], blob_list_sorted[i][0:2])]) < coords_dist_th:\n                blob_list_sorted = blob_list_sorted+[[min(x) for x in zip(blob_list_sorted[j], blob_list_sorted[i])][0:2]+[blob_list_sorted[j][2]+blob_list_sorted[i][2]]]\n                del blob_list_sorted[i]\n                del blob_list_sorted[j]  \n                len_val = len(blob_list_sorted)\n   \n    try:\n        for blob in blobs_dog:                   \n            y, x, r = blob\n            Iou = cv2.rectangle(_image,(int(x),int(y)),(int((x+r)),int((y+r))),(255,0,0),2)         \n        \n        fig, axes = plt.subplots(1, 2, figsize=(12, 6))\n        ax = axes.ravel()\n\n        ax[0].imshow(cv2.cvtColor(cv2.imread('test_/'+ img_name.split(\".\")[0] + \".png\"),cv2.COLOR_RGB2BGR))\n        ax[1].imshow(Iou)\n\n        plt.show()\n        cv2.imwrite(\"Output/\" + img_name.split(\".\")[0] + \".png\",Iou)\n    except:pass\n   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(submission).to_csv(\"submission.csv\") ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"warnings.simplefilter(\"ignore\")\n\nfor idx,img_name in enumerate(os.listdir(\"train_0/\")):\n    if not os.path.exists('segmentation'):\n        os.makedirs('segmentation')\n    \n    \n    out = model.predict_segmentation(\n    inp=\"train_0/\"+ img_name,\n    out_fname=\"segmentation/\" + img_name.split(\".\")[0] + \".png\" \n    )  \n    \n    out_image = cv2.imread('segmentation/'+ img_name.split(\".\")[0] + \".png\")\n    _image = cv2.imread('train_0/'+ img_name.split(\".\")[0] + \".png\")\n\n\n    ## Step 4:Blob detection\n    image_gray = rgb2gray(out_image)\n    blobs_dog = blob_dog(image_gray, max_sigma=30, threshold=.10)\n    \n    blob_list_sorted = sorted([[int(blob[0]), int(blob[1]),int(blob[2])] for blob in blobs_dog])\n    len_val = len(blob_list_sorted)\n    for i,val in enumerate(range(0,len(blob_list_sorted))):\n        for j in range(i+1,len(blob_list_sorted)-2):\n            if max(i,j) < len_val and max([abs(np.diff(x)) for x in zip(blob_list_sorted[j][0:2], blob_list_sorted[i][0:2])]) < coords_dist_th:\n                blob_list_sorted = blob_list_sorted+[[min(x) for x in zip(blob_list_sorted[j], blob_list_sorted[i])][0:2]+[blob_list_sorted[j][2]+blob_list_sorted[i][2]]]\n                del blob_list_sorted[i]\n                del blob_list_sorted[j]  \n                len_val = len(blob_list_sorted)\n   \n    try:\n        for blob in blobs_dog:                   \n            y, x, r = blob\n            Iou = cv2.rectangle(_image,(int(x),int(y)),(int((x+r)),int((y+r))),(255,0,0),2)         \n        \n        fig, axes = plt.subplots(1, 2, figsize=(12, 6))\n        ax = axes.ravel()\n\n        ax[0].imshow(cv2.cvtColor(cv2.imread('train_0/'+ img_name.split(\".\")[0] + \".png\"),cv2.COLOR_RGB2BGR))\n        ax[1].imshow(Iou)\n\n        plt.show()\n        cv2.imwrite(\"Output/\" + img_name.split(\".\")[0] + \".png\",Iou)\n    except:pass\n   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}