{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Downloading Pretrained Weights","metadata":{"id":"0o2LLOGmOKhc"}},{"cell_type":"markdown","source":"## Downloading main data","metadata":{"id":"0N_5Ie4fOSI0"}},{"cell_type":"markdown","source":"## Loading all the necesssary Modules","metadata":{"id":"aqErMtkdvWYt"}},{"cell_type":"code","source":"from absl import logging\nimport tensorflow as tf\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.layers import Add,Concatenate,Conv2D,Input,Lambda,LeakyReLU,MaxPool2D,UpSampling2D,ZeroPadding2D,BatchNormalization\nfrom tensorflow.keras.regularizers import l2\nfrom tensorflow.keras.losses import binary_crossentropy, sparse_categorical_crossentropy\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau,EarlyStopping,ModelCheckpoint,TensorBoard\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping, LearningRateScheduler, CSVLogger\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport numpy as np\nfrom PIL import Image, ImageDraw, ImageFont\nfrom IPython.display import display\nfrom seaborn import color_palette\nimport cv2\nimport pandas as pd\nfrom tqdm import tqdm\nimport os\nimport tensorflow as tf\nimport time\nfrom tqdm import tqdm\nimport datetime\nimport ast","metadata":{"id":"UVq2ukj-u5wj","outputId":"13c3f7a7-8ddb-458b-8f45-8566a942021f","execution":{"iopub.status.busy":"2022-03-20T11:14:12.883133Z","iopub.execute_input":"2022-03-20T11:14:12.883866Z","iopub.status.idle":"2022-03-20T11:14:12.892327Z","shell.execute_reply.started":"2022-03-20T11:14:12.883826Z","shell.execute_reply":"2022-03-20T11:14:12.891352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading Data into Dataframe","metadata":{"id":"TTSiTM_8OrO_"}},{"cell_type":"code","source":"cd ../input/global-wheat-detection","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:14:12.897637Z","iopub.execute_input":"2022-03-20T11:14:12.898377Z","iopub.status.idle":"2022-03-20T11:14:12.906327Z","shell.execute_reply.started":"2022-03-20T11:14:12.898337Z","shell.execute_reply":"2022-03-20T11:14:12.905502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv('train.csv') #Load the CSV file\ntrain_df['image_id'] = 'train/' + train_df['image_id'].astype(str)+'.jpg' #Add the path to the images","metadata":{"id":"DbZPU1_gOxPb","execution":{"iopub.status.busy":"2022-03-20T11:14:12.908622Z","iopub.execute_input":"2022-03-20T11:14:12.909161Z","iopub.status.idle":"2022-03-20T11:14:13.104697Z","shell.execute_reply.started":"2022-03-20T11:14:12.909127Z","shell.execute_reply":"2022-03-20T11:14:13.103910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:14:13.106092Z","iopub.execute_input":"2022-03-20T11:14:13.106375Z","iopub.status.idle":"2022-03-20T11:14:13.118262Z","shell.execute_reply.started":"2022-03-20T11:14:13.106338Z","shell.execute_reply":"2022-03-20T11:14:13.117493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing the data","metadata":{"id":"f_AXWkDGO0Ft"}},{"cell_type":"code","source":"train_df[['x_min','y_min', 'width', 'height']] = pd.DataFrame([ast.literal_eval(x) for x in train_df.bbox.tolist()], index= train_df.index)\ntrain_df = train_df[['image_id', 'bbox', 'source', 'x_min', 'y_min', 'width', 'height']]\ntrain_df['area'] = train_df['width'] * train_df['height']\ntrain_df['x_max'] = train_df['x_min'] + train_df['width']\ntrain_df['y_max'] = train_df['y_min'] + train_df['height']\ntrain_df = train_df.drop(['bbox', 'source'], axis=1)\ntrain_df = train_df[['image_id', 'x_min', 'y_min', 'x_max', 'y_max', 'width', 'height', 'area']]\n\n# There are some buggy annonations in training images having huge bounding boxes. Let's remove those bboxes\ntrain_df = train_df[train_df['area'] < 100000]","metadata":{"id":"zURqRyzuxPCo","execution":{"iopub.status.busy":"2022-03-20T11:14:13.119424Z","iopub.execute_input":"2022-03-20T11:14:13.119936Z","iopub.status.idle":"2022-03-20T11:14:15.564853Z","shell.execute_reply.started":"2022-03-20T11:14:13.119896Z","shell.execute_reply":"2022-03-20T11:14:15.564150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:14:15.567198Z","iopub.execute_input":"2022-03-20T11:14:15.567460Z","iopub.status.idle":"2022-03-20T11:14:15.582185Z","shell.execute_reply.started":"2022-03-20T11:14:15.567424Z","shell.execute_reply":"2022-03-20T11:14:15.581425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:14:15.583527Z","iopub.execute_input":"2022-03-20T11:14:15.584035Z","iopub.status.idle":"2022-03-20T11:14:15.591014Z","shell.execute_reply.started":"2022-03-20T11:14:15.583999Z","shell.execute_reply":"2022-03-20T11:14:15.590316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = train_df.drop(['width','height','area'],axis = 1) #Dropping unwanted columns\nclass_list = ['Wheat']*train_df.shape[0]  \ntrain_df['category'] = class_list  #Adding category column\nlabel_list = [1]*train_df.shape[0] \ntrain_df['label'] = label_list #Adding class label to each row\ntrain_df.columns = ['img_path','min_w','min_h','max_w','max_h','category','label'] #Changing Column names\n\ndf = train_df  #Loading a copy into df\ndf = df[['max_h', 'max_w', 'min_h', 'min_w', 'category', 'img_path', 'label']] #Changing the order of columns\ndf.head()","metadata":{"id":"X3RNlcZFO7MT","outputId":"af5c18a9-f656-49df-f10f-0a65c0dbae20","execution":{"iopub.status.busy":"2022-03-20T11:14:15.592440Z","iopub.execute_input":"2022-03-20T11:14:15.592858Z","iopub.status.idle":"2022-03-20T11:14:15.683480Z","shell.execute_reply.started":"2022-03-20T11:14:15.592823Z","shell.execute_reply":"2022-03-20T11:14:15.682810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test Train Split","metadata":{"id":"wnO3WkhEPFTf"}},{"cell_type":"code","source":"max_boxes = df.groupby(['img_path']).count()['category'].max()\nprint('Max number of boxes in a given image are :',max_boxes) #Calculating maximum number of boxes in an image\n\nindex = list(set([i.split('/')[-1] for i in df['img_path'].values]))\nprint('Total number of unique images in the dataframe :',len(index))  #Taking out all the unique images in the df\n\n#Here for simplicity sake i am using 1000 images as train and 200 images as test.\nprint('Splitting into train and test data......')\ntrain_image=index[0:2500]\ntest_image =index[2500:3733]\nprint('Number of images in train data :',len(train_image))\nprint('Number of images in test data  :',len(test_image))","metadata":{"id":"bEtQGSrkxR6Y","outputId":"fccf9544-9440-4b89-dba7-91a2d730143c","execution":{"iopub.status.busy":"2022-03-20T11:14:15.684765Z","iopub.execute_input":"2022-03-20T11:14:15.685001Z","iopub.status.idle":"2022-03-20T11:14:15.782716Z","shell.execute_reply.started":"2022-03-20T11:14:15.684968Z","shell.execute_reply":"2022-03-20T11:14:15.781926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Initializing all the necessary variables","metadata":{"id":"LLc3w9e4o5Qa"}},{"cell_type":"code","source":"yolo_anchors = np.array([(10, 13), (16, 30), (33, 23), (30, 61), (62, 45),\n                         (59, 119), (116, 90), (156, 198), (373, 326)],\n                        np.float32) / 320\nyolo_anchor_masks = np.array([[6, 7, 8], [3, 4, 5], [0, 1, 2]])\n\nsize = 320 # size of resize image\nbatch_size = 8 #Batch size that we want to load when training\nyolo_max_boxes = 116 # maximum yolo boxes predicted per image.\nyolo_iou_threshold = 0.5 # IOU threshold score \nyolo_score_threshold = 0.4 # objectness threshold score\nlearning_rate = 1e-4 # learning rate\nnum_classes = 2 # num of category in our dataset\nepochs = 30 # epochs run to fine tune our model\nclass_dict = {'Wheat':1} # Mapping of category and its corresponding label","metadata":{"id":"5_RE30UXo2Zi","execution":{"iopub.status.busy":"2022-03-20T11:14:15.783930Z","iopub.execute_input":"2022-03-20T11:14:15.784232Z","iopub.status.idle":"2022-03-20T11:14:15.791819Z","shell.execute_reply.started":"2022-03-20T11:14:15.784196Z","shell.execute_reply":"2022-03-20T11:14:15.791139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# YOLOV3 architecture starts here","metadata":{"id":"GiogsVmOpJFf"}},{"cell_type":"code","source":"def DarknetConv(x, filters, size, strides=1, batch_norm=True):\n    if strides == 1: \n        padding = 'same' \n    else:\n        x = ZeroPadding2D(((1, 0), (1, 0)))(x)  # top left half-padding \n        padding = 'valid' \n    x = Conv2D(filters=filters, kernel_size=size,\n               strides=strides, padding=padding,\n               use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)\n    if batch_norm:\n        x = BatchNormalization()(x)\n        x = LeakyReLU(alpha=0.1)(x) # by default alpha is 0.3\n    return x","metadata":{"id":"oDBKh_S-o2n1","execution":{"iopub.status.busy":"2022-03-20T11:14:15.793029Z","iopub.execute_input":"2022-03-20T11:14:15.793423Z","iopub.status.idle":"2022-03-20T11:14:15.800933Z","shell.execute_reply.started":"2022-03-20T11:14:15.793386Z","shell.execute_reply":"2022-03-20T11:14:15.800289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# below is residual connection layer function\ndef DarknetResidual(x, filters):\n    prev = x # storing input in prev variable \n    x = DarknetConv(x, filters // 2, 1) \n    x = DarknetConv(x, filters, 3)\n    x = Add()([prev, x]) # residual connection\n    return x","metadata":{"id":"Jy9xkN0WQn7j","execution":{"iopub.status.busy":"2022-03-20T11:14:15.802277Z","iopub.execute_input":"2022-03-20T11:14:15.802667Z","iopub.status.idle":"2022-03-20T11:14:15.809645Z","shell.execute_reply.started":"2022-03-20T11:14:15.802631Z","shell.execute_reply":"2022-03-20T11:14:15.809053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this is our real Darknetblock function calling above 2 fucntions \ndef DarknetBlock(x, filters, blocks):\n    x = DarknetConv(x, filters, 3, strides=2) \n    for _ in range(blocks):\n        x = DarknetResidual(x, filters)\n    return x","metadata":{"id":"BtfFrgujQp3u","execution":{"iopub.status.busy":"2022-03-20T11:14:15.811092Z","iopub.execute_input":"2022-03-20T11:14:15.811366Z","iopub.status.idle":"2022-03-20T11:14:15.819444Z","shell.execute_reply.started":"2022-03-20T11:14:15.811333Z","shell.execute_reply":"2022-03-20T11:14:15.818780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Darknet(name=None):\n    x = inputs = Input([None, None, 3])\n    x = DarknetConv(x, 32, 3)\n    x = DarknetBlock(x, 64, 1)\n    x = DarknetBlock(x, 128, 2)  # skip connection\n    x = x_36 = DarknetBlock(x, 256, 8)  # skip connection\n    x = x_61 = DarknetBlock(x, 512, 8)\n    x = DarknetBlock(x, 1024, 4) # last layer detecting bounding box dimension (tx,ty,bx,by)\n    return tf.keras.Model(inputs, (x_36, x_61, x), name=name)","metadata":{"id":"tLpcCT7mQr7g","execution":{"iopub.status.busy":"2022-03-20T11:14:15.823526Z","iopub.execute_input":"2022-03-20T11:14:15.823740Z","iopub.status.idle":"2022-03-20T11:14:15.831630Z","shell.execute_reply.started":"2022-03-20T11:14:15.823684Z","shell.execute_reply":"2022-03-20T11:14:15.830967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def YoloConv(filters, name=None):\n    def yolo_conv(x_in):\n        if isinstance(x_in, tuple):\n            inputs = Input(x_in[0].shape[1:]), Input(x_in[1].shape[1:])\n            x, x_skip = inputs\n\n            # concat with skip connection\n            x = DarknetConv(x, filters, 1)\n            # upsampling of a layer\n            x = UpSampling2D(2)(x)\n            # concatenation of skip connection result and last output result\n            x = Concatenate()([x, x_skip])\n        else:\n            x = inputs = Input(x_in.shape[1:])\n        \n        x = DarknetConv(x, filters, 1)\n        x = DarknetConv(x, filters * 2, 3)\n        x = DarknetConv(x, filters, 1)\n        x = DarknetConv(x, filters * 2, 3)\n        x = DarknetConv(x, filters, 1)\n        return Model(inputs, x, name=name)(x_in)\n    return yolo_conv","metadata":{"id":"kJEMjhLbo2kw","execution":{"iopub.status.busy":"2022-03-20T11:14:15.832675Z","iopub.execute_input":"2022-03-20T11:14:15.832975Z","iopub.status.idle":"2022-03-20T11:14:15.840991Z","shell.execute_reply.started":"2022-03-20T11:14:15.832940Z","shell.execute_reply":"2022-03-20T11:14:15.840305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## YOLO output","metadata":{"id":"sTzcDo4TQ1AM"}},{"cell_type":"code","source":"def YoloOutput(filters, anchors, classes, name=None):\n    def yolo_output(x_in):\n        x = inputs = Input(x_in.shape[1:]) # this is an input shape excluded with batch size.\n        x = DarknetConv(x, filters * 2, 3) # Darkconv is a fn implemented above which is internally calling\n        x = DarknetConv(x, anchors * (classes + 5), 1, batch_norm=False)\n        x = Lambda(lambda x: tf.reshape(x, (-1, tf.shape(x)[1], tf.shape(x)[2],\n                                            anchors, classes + 5)))(x)\n        # x is reshaped into (None, grid_size, grid_size, anchors, (x,y,w,h,objectness score,..classes))\n        return tf.keras.Model(inputs, x, name=name)(x_in)\n    return yolo_output","metadata":{"id":"ZTWKeWT8Q25a","execution":{"iopub.status.busy":"2022-03-20T11:14:15.842080Z","iopub.execute_input":"2022-03-20T11:14:15.842895Z","iopub.status.idle":"2022-03-20T11:14:15.850970Z","shell.execute_reply.started":"2022-03-20T11:14:15.842857Z","shell.execute_reply":"2022-03-20T11:14:15.850322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Non Max Supression","metadata":{"id":"2F0_pnVnQ-a1"}},{"cell_type":"code","source":"def yolo_boxes(pred, anchors, classes):\n    # pred: (batch_size, grid, grid, anchors, (x, y, w, h, obj, ...classes))\n    grid_size = tf.shape(pred)[1]\n    box_xy, box_wh, objectness, class_probs = tf.split(\n        pred, (2, 2, 1, classes), axis=-1)\n    # objectness : it's means whether there is any object in a predicted box\n    # class_probs : it's a probability of a class given object is there i.e P(Pc|object)\n\n    box_xy = tf.sigmoid(box_xy) \n    objectness = tf.sigmoid(objectness)\n    class_probs = tf.sigmoid(class_probs)\n    pred_box = tf.concat((box_xy, box_wh), axis=-1)  # original xywh for loss\n\n    # !!! grid[x][y] == (y, x)\n    grid = tf.meshgrid(tf.range(grid_size), tf.range(grid_size))\n    grid = tf.expand_dims(tf.stack(grid, axis=-1), axis=2)  # [gx, gy, 1, 2]\n    box_xy = (box_xy + tf.cast(grid, tf.float32)) / \\\n        tf.cast(grid_size, tf.float32)\n    box_wh = tf.exp(box_wh) * anchors\n\n    box_x1y1 = box_xy - box_wh / 2 \n    box_x2y2 = box_xy + box_wh / 2\n    bbox = tf.concat([box_x1y1, box_x2y2], axis=-1)\n\n    return bbox, objectness, class_probs, pred_box\n\ndef yolo_nms(outputs, anchors, masks, classes):\n    # boxes, confidence scores(objectness scores), class probabilities\n\n    b, c, t = [], [], []\n\n    # iterating through each outputs predicted by model\n    for o in outputs:\n        b.append(tf.reshape(o[0], (tf.shape(o[0])[0], -1, tf.shape(o[0])[-1])))\n        c.append(tf.reshape(o[1], (tf.shape(o[1])[0], -1, tf.shape(o[1])[-1])))\n        t.append(tf.reshape(o[2], (tf.shape(o[2])[0], -1, tf.shape(o[2])[-1])))\n\n    bbox = tf.concat(b, axis=1)\n    confidence = tf.concat(c, axis=1)\n    class_probs = tf.concat(t, axis=1)\n\n    scores = confidence * class_probs # this is P(Pc|objectness score) value\n    boxes, scores, classes, valid_detections = tf.image.combined_non_max_suppression(\n        boxes=tf.reshape(bbox, (tf.shape(bbox)[0], -1, 1, 4)),\n        scores=tf.reshape(\n            scores, (tf.shape(scores)[0], -1, tf.shape(scores)[-1])),\n        max_output_size_per_class = yolo_max_boxes, # here it is 223, define above \n        max_total_size = yolo_max_boxes,\n        iou_threshold = yolo_iou_threshold, # threshold for filtering the boxes\n        score_threshold = yolo_score_threshold # threshold for objectness score below which we ignore that bounding box\n    )\n\n    return boxes, scores, classes, valid_detections","metadata":{"id":"KFPMAJ4YRAeY","execution":{"iopub.status.busy":"2022-03-20T11:14:15.852157Z","iopub.execute_input":"2022-03-20T11:14:15.852433Z","iopub.status.idle":"2022-03-20T11:14:15.869421Z","shell.execute_reply.started":"2022-03-20T11:14:15.852401Z","shell.execute_reply":"2022-03-20T11:14:15.868660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Backbone Darknet Architecture","metadata":{"id":"7eWEbU40RHoe"}},{"cell_type":"code","source":"def YoloV3(size=None, channels=3, anchors=yolo_anchors,\n           masks=yolo_anchor_masks, classes=80, training=False):\n    x = inputs = Input([size, size, channels], name='input') # input of an image\n\n    x_36, x_61, x = Darknet(name='yolo_darknet')(x) # backbone networks 3 outputs w.r.t to each grid size\n    # till here darknet network\n\n    # from below it's a Feature Pyramind Network with lateral connections\n    # for 13*13 grid size output\n    x = YoloConv(512, name='yolo_conv_0')(x) \n    output_0 = YoloOutput(512, len(masks[0]), classes, name='yolo_output_0')(x)\n    \n    # for 26*26 grid size output\n    x = YoloConv(256, name='yolo_conv_1')((x, x_61))\n    output_1 = YoloOutput(256, len(masks[1]), classes, name='yolo_output_1')(x)\n\n    # for 52*52 grid size output\n    x = YoloConv(128, name='yolo_conv_2')((x, x_36))\n    output_2 = YoloOutput(128, len(masks[2]), classes, name='yolo_output_2')(x)\n\n    if training:\n        return Model(inputs, (output_0, output_1, output_2), name='yolov3')\n\n    # for 13*13 grid size output\n    boxes_0 = Lambda(lambda x: yolo_boxes(x, anchors[masks[0]], classes),\n                     name='yolo_boxes_0')(output_0)\n    # for 26*26 grid size output\n    boxes_1 = Lambda(lambda x: yolo_boxes(x, anchors[masks[1]], classes),\n                     name='yolo_boxes_1')(output_1)\n    # for 52*52 grid size output\n    boxes_2 = Lambda(lambda x: yolo_boxes(x, anchors[masks[2]], classes),\n                     name='yolo_boxes_2')(output_2)\n    #  after combining boxes from various scales we have total 10,647 boxes which is too large\n    # so to remove invalid boxes we use non_maximum_suppression \n\n    outputs = Lambda(lambda x: yolo_nms(x, anchors, masks, classes),\n                     name='yolo_nms')((boxes_0[:3], boxes_1[:3], boxes_2[:3]))\n\n    return Model(inputs, outputs, name='yolov3')\n","metadata":{"id":"e9Uj5IQCo2Wo","execution":{"iopub.status.busy":"2022-03-20T11:14:15.870569Z","iopub.execute_input":"2022-03-20T11:14:15.871302Z","iopub.status.idle":"2022-03-20T11:14:15.884308Z","shell.execute_reply.started":"2022-03-20T11:14:15.871266Z","shell.execute_reply":"2022-03-20T11:14:15.883489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Custom Loss Function","metadata":{"id":"KptixCdtRNaE"}},{"cell_type":"code","source":"def YoloLoss(anchors, classes=80, ignore_thresh=0.5):\n    def yolo_loss(y_true, y_pred):\n        # 1. transform all pred outputs\n        # y_pred: (batch_size, grid, grid, anchors, (x, y, w, h, obj, ...cls))\n        pred_box, pred_obj, pred_class, pred_xywh = yolo_boxes(\n            y_pred, anchors, classes)\n        pred_xy = pred_xywh[..., 0:2]\n        pred_wh = pred_xywh[..., 2:4]\n\n        # 2. transform all true outputs\n        # y_true: (batch_size, grid, grid, anchors, (x1, y1, x2, y2, obj, cls))\n        \n        true_box, true_obj, true_class_idx = tf.split(\n            y_true, (4, 1, 1), axis=-1) \n\n        # the above split function split (x1,y1,x2...cls) into (x1,y1),(x2,y2),(obj),(cls)\n        # the 4,1,1 is a length at which it split\n\n        true_xy = (true_box[..., 0:2] + true_box[..., 2:4]) / 2 # finding center (Xcen,Ycen)\n        true_wh = true_box[..., 2:4] - true_box[..., 0:2] # width and height\n\n        # give higher weights to small boxes\n        box_loss_scale = 2 - true_wh[..., 0] * true_wh[..., 1]\n\n        # 3. inverting the pred box equations\n        grid_size = tf.shape(y_true)[1]\n        grid = tf.meshgrid(tf.range(grid_size), tf.range(grid_size))\n        grid = tf.expand_dims(tf.stack(grid, axis=-1), axis=2)\n        true_xy = true_xy * tf.cast(grid_size, tf.float32) - \\\n            tf.cast(grid, tf.float32) # this code snippet giving us at which point each cell is starting and ending\n            # in resize image of 416 * 416 \n            # suppose there 13*13 = 169 cells , so every cell we will have starting and ending point\n        true_wh = tf.math.log(true_wh / anchors) \n        # YOLO doesn’t predict the absolute coordinates of the bounding box’s center\n        true_wh = tf.where(tf.math.is_inf(true_wh),\n                           tf.zeros_like(true_wh), true_wh)\n\n        # 4. calculate all masks\n        obj_mask = tf.squeeze(true_obj, -1)\n        \n        # ignore false positive when iou is over threshold\n        best_iou = tf.map_fn(\n            lambda x: tf.reduce_max(broadcast_iou(x[0], tf.boolean_mask(\n                x[1], tf.cast(x[2], tf.bool))), axis=-1),\n            (pred_box, true_box, obj_mask),\n            tf.float32)\n        ignore_mask = tf.cast(best_iou < ignore_thresh, tf.float32)\n\n        # 5. calculate all losses\n        xy_loss = obj_mask * box_loss_scale * \\\n            tf.reduce_sum(tf.square(true_xy - pred_xy), axis=-1)\n        wh_loss = obj_mask * box_loss_scale * \\\n            tf.reduce_sum(tf.square(true_wh - pred_wh), axis=-1)\n        \n        obj_loss = binary_crossentropy(true_obj, pred_obj)\n        # obj_loss = obj_mask * obj_loss + \\\n        #     (1 - obj_mask) * ignore_mask * obj_loss\n        \n        alpha = 0.75 # focal loss hyperparameter\n        conf_focal = tf.pow(obj_mask-tf.squeeze(tf.sigmoid(pred_obj),-1),2)\n        obj_loss = conf_focal*((1-alpha)*obj_mask*obj_loss + alpha*(1-obj_mask)*ignore_mask*obj_loss)  # batch * grid * grid * anchors_per_scale\n\n        # TODO: use binary_crossentropy instead\n        class_loss = obj_mask * sparse_categorical_crossentropy(\n            true_class_idx, pred_class)\n\n        # 6. sum over (batch, gridx, gridy, anchors) => (batch, 1)\n        xy_loss = tf.reduce_sum(xy_loss, axis=(1, 2, 3))\n        wh_loss = tf.reduce_sum(wh_loss, axis=(1, 2, 3))\n        obj_loss = tf.reduce_sum(obj_loss, axis=(1, 2, 3))\n        class_loss = tf.reduce_sum(class_loss, axis=(1, 2, 3))\n\n        return xy_loss + wh_loss + obj_loss + class_loss\n    return yolo_loss","metadata":{"id":"VbJmFhlBo2Ub","execution":{"iopub.status.busy":"2022-03-20T11:14:15.887534Z","iopub.execute_input":"2022-03-20T11:14:15.888327Z","iopub.status.idle":"2022-03-20T11:14:15.906933Z","shell.execute_reply.started":"2022-03-20T11:14:15.888274Z","shell.execute_reply":"2022-03-20T11:14:15.906170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Some useful functions","metadata":{"id":"jMF4aGyio4HA"}},{"cell_type":"code","source":"YOLOV3_LAYER_LIST = ['yolo_darknet','yolo_conv_0','yolo_output_0','yolo_conv_1',\n                     'yolo_output_1','yolo_conv_2','yolo_output_2',]\n\ndef load_darknet_weights(model, weights_file, tiny=False):\n    wf = open(weights_file, 'rb') # reading weights file\n    major, minor, revision, seen, _ = np.fromfile(wf, dtype=np.int32, count=5)\n\n    layers = YOLOV3_LAYER_LIST\n\n    # iterating through all layers define in above yolov3_layers_list\n    for layer_name in layers:\n        # for eg if there is one layer darknet then there is many sub layers inside it's network\n        sub_model = model.get_layer(layer_name)\n        for i, layer in enumerate(sub_model.layers):\n            if not layer.name.startswith('conv2d'): \n                continue\n            batch_norm = None\n            if i + 1 < len(sub_model.layers) and \\\n                    sub_model.layers[i + 1].name.startswith('batch_norm'):\n                batch_norm = sub_model.layers[i + 1]\n\n            logging.info(\"{}/{} {}\".format(\n                sub_model.name, layer.name, 'bn' if batch_norm else 'bias'))\n\n            filters = layer.filters\n            size = layer.kernel_size[0]\n            in_dim = layer.input_shape[-1]\n\n            if batch_norm is None:\n                conv_bias = np.fromfile(wf, dtype=np.float32, count=filters)\n            else:\n                # darknet [beta, gamma, mean, variance]\n                bn_weights = np.fromfile(\n                    wf, dtype=np.float32, count=4 * filters)\n                # tf [gamma, beta, mean, variance]\n                bn_weights = bn_weights.reshape((4, filters))[[1, 0, 2, 3]]\n\n            # darknet shape (out_dim, in_dim, height, width)\n            conv_shape = (filters, in_dim, size, size)\n            conv_weights = np.fromfile(\n                wf, dtype=np.float32, count=np.product(conv_shape))\n            # tf shape (height, width, in_dim, out_dim)\n            conv_weights = conv_weights.reshape(\n                conv_shape).transpose([2, 3, 1, 0])\n\n            if batch_norm is None:\n                layer.set_weights([conv_weights, conv_bias])\n            else:\n                layer.set_weights([conv_weights])\n                batch_norm.set_weights(bn_weights)\n\n    assert len(wf.read()) == 0, 'failed to read all data'\n    wf.close()\n\n\n\ndef broadcast_iou(box_1, box_2):\n    # box_1: (..., (x1, y1, x2, y2))\n    # box_2: (N, (x1, y1, x2, y2))\n\n    # broadcast boxes\n    box_1 = tf.expand_dims(box_1, -2)\n    box_2 = tf.expand_dims(box_2, 0)\n    # new_shape: (..., N, (x1, y1, x2, y2))\n    new_shape = tf.broadcast_dynamic_shape(tf.shape(box_1), tf.shape(box_2))\n    box_1 = tf.broadcast_to(box_1, new_shape) # it will change the shape of box into new_shape given\n    box_2 = tf.broadcast_to(box_2, new_shape)\n\n    # in below code we are finding intersection box width and height through which we will find intersection area.\n    # and this we are finding all boxes \n    int_w = tf.maximum(tf.minimum(box_1[..., 2], box_2[..., 2]) -\n                       tf.maximum(box_1[..., 0], box_2[..., 0]), 0) \n    int_h = tf.maximum(tf.minimum(box_1[..., 3], box_2[..., 3]) -\n                       tf.maximum(box_1[..., 1], box_2[..., 1]), 0) \n    int_area = int_w * int_h  # area of intersection\n    box_1_area = (box_1[..., 2] - box_1[..., 0]) * \\\n        (box_1[..., 3] - box_1[..., 1]) # this box_1_area contains all boxes area predicted in an image\n    box_2_area = (box_2[..., 2] - box_2[..., 0]) * \\\n        (box_2[..., 3] - box_2[..., 1]) # this box2_area is our ground truth box area\n\n    # Formula: Union(A,B) = A + B - Inter(A,B)\n    return int_area / (box_1_area + box_2_area - int_area)\n\n\ndef freeze_all(model, frozen=True):\n    model.trainable = not frozen\n    if isinstance(model, tf.keras.Model):\n        for l in model.layers:\n            freeze_all(l, frozen)","metadata":{"id":"o5RVOsLZo3Wg","execution":{"iopub.status.busy":"2022-03-20T11:14:15.909536Z","iopub.execute_input":"2022-03-20T11:14:15.909737Z","iopub.status.idle":"2022-03-20T11:14:15.929736Z","shell.execute_reply.started":"2022-03-20T11:14:15.909714Z","shell.execute_reply":"2022-03-20T11:14:15.929122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Dataset Loader And Transformation","metadata":{"id":"EI9WjPsvfZs2"}},{"cell_type":"code","source":"@tf.function\ndef transform_targets_for_output(y_true, grid_size, anchor_idxs):\n    # y_true: (N, boxes, (x1, y1, x2, y2, class, best_anchor))\n    N = tf.shape(y_true)[0]\n\n    # y_true_out: (N, grid, grid, anchors, [x, y, w, h, obj, class])\n    y_true_out = tf.zeros(\n        (N, grid_size, grid_size, tf.shape(anchor_idxs)[0], 6))\n\n    anchor_idxs = tf.cast(anchor_idxs, tf.int32)\n\n    indexes = tf.TensorArray(tf.int32, 1, dynamic_size=True)\n    updates = tf.TensorArray(tf.float32, 1, dynamic_size=True)\n    idx = 0\n\n    # below iteration change the values and update it to the format which acceptable by yolov3.\n    for i in tf.range(N):\n        for j in tf.range(tf.shape(y_true)[1]):\n            if tf.equal(y_true[i][j][2], 0): \n                continue\n            anchor_eq = tf.equal(\n                anchor_idxs, tf.cast(y_true[i][j][5], tf.int32))\n\n            if tf.reduce_any(anchor_eq):\n                box = y_true[i][j][0:4] #(x1,y1,x2,y2)\n                box_xy = (y_true[i][j][0:2] + y_true[i][j][2:4]) / 2 # ((x1+x2)/2,(y1+y2)/2)\n\n                # which is (Xcenter,Ycenter)\n\n                anchor_idx = tf.cast(tf.where(anchor_eq), tf.int32)\n                grid_xy = tf.cast(box_xy // (1/grid_size), tf.int32) # multiplying it by grid_size\n\n                # grid[y][x][anchor] = (tx, ty, bw, bh, obj, class)\n                indexes = indexes.write(\n                    idx, [i, grid_xy[1], grid_xy[0], anchor_idx[0][0]]) \n                updates = updates.write(\n                    idx, [box[0], box[1], box[2], box[3], 1, y_true[i][j][4]])\n                idx += 1\n    return tf.tensor_scatter_nd_update(\n        y_true_out, indexes.stack(), updates.stack())\n","metadata":{"id":"ylcMnsTSVYtV","execution":{"iopub.status.busy":"2022-03-20T11:14:15.931195Z","iopub.execute_input":"2022-03-20T11:14:15.931649Z","iopub.status.idle":"2022-03-20T11:14:15.945808Z","shell.execute_reply.started":"2022-03-20T11:14:15.931608Z","shell.execute_reply":"2022-03-20T11:14:15.945162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def transform_targets(y_train, anchors, anchor_masks, size):\n    y_outs = []\n    grid_size = size // 32 # suppose we input 416 size then grid size is 416//32 = 13\n\n    # calculate anchor index for true boxes\n    anchors = tf.cast(anchors, tf.float32) # casting every anchors to float\n    anchor_area = anchors[..., 0] * anchors[..., 1] # calculating the area of anchors\n    box_wh = y_train[..., 2:4] - y_train[..., 0:2] # here we are peforming xmax-xmin,ymax-ymin using vectors\n    box_wh = tf.tile(tf.expand_dims(box_wh, -2),\n                     (1, 1, tf.shape(anchors)[0], 1))\n    box_area = box_wh[..., 0] * box_wh[..., 1] # these are our Ground Truth Box Area\n    intersection = tf.minimum(box_wh[..., 0], anchors[..., 0]) * \\\n        tf.minimum(box_wh[..., 1], anchors[..., 1]) # here we tring to get IOU area \n    iou = intersection / (box_area + anchor_area - intersection) # simple operation of Intersection/Union\n    anchor_idx = tf.cast(tf.argmax(iou, axis=-1), tf.float32) # storing those anchor index which has highest IOU number\n    anchor_idx = tf.expand_dims(anchor_idx, axis=-1)\n\n    y_train = tf.concat([y_train, anchor_idx], axis=-1)\n\n    for anchor_idxs in anchor_masks:\n        y_outs.append(transform_targets_for_output(\n            y_train, grid_size, anchor_idxs))\n        grid_size *= 2 # here we are calling the above function for 13*13 grid then, 26*26 grid then, 52*52 grid.\n\n    return tuple(y_outs)","metadata":{"id":"UXjrP_rXVYlb","execution":{"iopub.status.busy":"2022-03-20T11:14:15.947296Z","iopub.execute_input":"2022-03-20T11:14:15.947609Z","iopub.status.idle":"2022-03-20T11:14:15.959565Z","shell.execute_reply.started":"2022-03-20T11:14:15.947576Z","shell.execute_reply":"2022-03-20T11:14:15.958876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Loading Yolov3 weights into Model","metadata":{"id":"5_Z9P9tgx_me"}},{"cell_type":"code","source":"%%time\nyolo = YoloV3(classes=80)\nyolo.summary()","metadata":{"id":"B_GA8Ytb0KE-","outputId":"03313e25-b686-4220-ba08-6d33e67e5a76","execution":{"iopub.status.busy":"2022-03-20T11:14:15.960540Z","iopub.execute_input":"2022-03-20T11:14:15.960730Z","iopub.status.idle":"2022-03-20T11:14:19.992294Z","shell.execute_reply.started":"2022-03-20T11:14:15.960700Z","shell.execute_reply":"2022-03-20T11:14:19.991599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"load_darknet_weights(yolo,\"/kaggle/input/pre-trained-weights/yolov3.weights\", False)\nyolo.save_weights(\"/kaggle/working/yolov3.tf\")","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:14:19.994988Z","iopub.execute_input":"2022-03-20T11:14:19.995185Z","iopub.status.idle":"2022-03-20T11:14:24.550540Z","shell.execute_reply.started":"2022-03-20T11:14:19.995160Z","shell.execute_reply":"2022-03-20T11:14:24.549827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /kaggle/working","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:14:24.551641Z","iopub.execute_input":"2022-03-20T11:14:24.551877Z","iopub.status.idle":"2022-03-20T11:14:25.250994Z","shell.execute_reply.started":"2022-03-20T11:14:24.551845Z","shell.execute_reply":"2022-03-20T11:14:25.249282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Parsing Function","metadata":{"id":"NHrmlXC_R1x3"}},{"cell_type":"code","source":"def parse_dataset(data,class_dict,size,image,path,yolo_max_boxes,count=0):\n    X = []\n    Y = []\n    for img in tqdm(image):\n        x_train = Image.open(path+img) # reading image\n        width,height = x_train.size # storing actual width and height so that we can later scale it\n        x_train = x_train.resize((size,size)) # resizing \n        x_train = np.array(x_train)\n        temp_data = []\n        # ierating over dataset having info about objects in an image\n        for _,row in data[data['img_path']==path+img].iterrows():\n            xmin = row.min_w/width\n            xmax = row.max_w/width\n            ymin = row.min_h/height\n            ymax = row.max_h/height\n            cls = class_dict[row.category]\n            temp_data.append([xmin,ymin,xmax,ymax,cls])\n        temp_data = temp_data+[[0,0,0,0,0]]*(yolo_max_boxes-len(temp_data)) # it's like padding \n        #return(temp)\n        Y.append(temp_data)\n        X.append(x_train)\n    return(np.array(X),np.stack(np.array(Y)))\n\n# transforming each image and normalizing it in range [0,1]\ndef transform_images(x,size):\n    x = tf.image.resize(x,(size,size))\n    x = x/255.0\n    return(x)","metadata":{"id":"VsMdGHyGR0h9","execution":{"iopub.status.busy":"2022-03-20T11:14:25.253644Z","iopub.execute_input":"2022-03-20T11:14:25.254220Z","iopub.status.idle":"2022-03-20T11:14:25.271082Z","shell.execute_reply.started":"2022-03-20T11:14:25.254173Z","shell.execute_reply":"2022-03-20T11:14:25.270241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading Train  and Validation dataset","metadata":{"id":"8BG6RgO3SUMM"}},{"cell_type":"code","source":"x,y= parse_dataset(\n   df,class_dict,size,train_image[:],'/kaggle/input/global-wheat-detection/train/',116) \n# df = data\nx = x.astype(np.float32)\ny = y.astype(np.float32)\n\ntrain_dataset = tf.data.Dataset.from_tensor_slices((x,y))\n\ntrain_dataset = train_dataset.shuffle(buffer_size=64) # Randomizing the data\ntrain_dataset = train_dataset.batch(8) # Setting Batch size\ntrain_dataset = train_dataset.map(lambda x, y: (transform_images(x, size),\n                                  transform_targets(y, yolo_anchors, yolo_anchor_masks, size)))\ntrain_dataset = train_dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE) # Prefetching for faster performance","metadata":{"id":"2IA5kCAon86D","outputId":"20975c22-c524-4f30-8717-6d92b07213f9","execution":{"iopub.status.busy":"2022-03-20T11:14:25.272856Z","iopub.execute_input":"2022-03-20T11:14:25.273444Z","iopub.status.idle":"2022-03-20T11:16:50.648367Z","shell.execute_reply.started":"2022-03-20T11:14:25.273400Z","shell.execute_reply":"2022-03-20T11:16:50.647533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset","metadata":{"id":"lSEM6gQFk-7C","outputId":"fb7fbc3b-5cf6-4218-cf40-4ca00c0ca505","execution":{"iopub.status.busy":"2022-03-20T11:16:50.649793Z","iopub.execute_input":"2022-03-20T11:16:50.650069Z","iopub.status.idle":"2022-03-20T11:16:50.659191Z","shell.execute_reply.started":"2022-03-20T11:16:50.650032Z","shell.execute_reply":"2022-03-20T11:16:50.658124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# parsing valid dataset \nx,y= parse_dataset(df,class_dict,size,test_image[:],'/kaggle/input/global-wheat-detection/train/',116) \nx = x.astype(np.float32)\ny = y.astype(np.float32)\nval_dataset = tf.data.Dataset.from_tensor_slices((x,y))\n\nval_dataset = val_dataset.shuffle(buffer_size=16)\nval_dataset = val_dataset.batch(8)\nval_dataset = val_dataset.map(\n    lambda x, y: (transform_images(x, size),\n    transform_targets(y, yolo_anchors, yolo_anchor_masks, size)))\n\nval_dataset = val_dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)","metadata":{"id":"jV0oCnpHlSxz","outputId":"5c347615-606a-4cf3-e096-48f704549786","execution":{"iopub.status.busy":"2022-03-20T11:16:50.660767Z","iopub.execute_input":"2022-03-20T11:16:50.661615Z","iopub.status.idle":"2022-03-20T11:17:41.429635Z","shell.execute_reply.started":"2022-03-20T11:16:50.661575Z","shell.execute_reply":"2022-03-20T11:17:41.428929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_dataset","metadata":{"id":"db7xNBOLd73p","outputId":"f40cf3ea-7380-4afd-9808-e76c54e46fec","execution":{"iopub.status.busy":"2022-03-20T11:17:41.430734Z","iopub.execute_input":"2022-03-20T11:17:41.430988Z","iopub.status.idle":"2022-03-20T11:17:41.436987Z","shell.execute_reply.started":"2022-03-20T11:17:41.430954Z","shell.execute_reply":"2022-03-20T11:17:41.436241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Training","metadata":{"id":"0BPDSk7ilypC"}},{"cell_type":"code","source":"model = YoloV3(size, training=True, classes=num_classes)\nanchors = yolo_anchors\nanchor_masks = yolo_anchor_masks","metadata":{"id":"pqSs1z65Vkta","execution":{"iopub.status.busy":"2022-03-20T11:17:41.438146Z","iopub.execute_input":"2022-03-20T11:17:41.438730Z","iopub.status.idle":"2022-03-20T11:17:43.035650Z","shell.execute_reply.started":"2022-03-20T11:17:41.438694Z","shell.execute_reply":"2022-03-20T11:17:43.034943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading Pretrained model and freezing for transfer learning","metadata":{"id":"JG6q3NijdGhX"}},{"cell_type":"code","source":"%%time\nmodel_pretrained = YoloV3(size, training=True, classes=80)\nmodel_pretrained.load_weights(\"/kaggle/working/yolov3.tf\")\nmodel.get_layer('yolo_darknet').set_weights(\nmodel_pretrained.get_layer('yolo_darknet').get_weights())\nfreeze_all(model.get_layer('yolo_darknet'))","metadata":{"id":"KQZETfM8VYjy","outputId":"ac6705bc-ac5f-47ed-e612-0b1d27e6d88f","execution":{"iopub.status.busy":"2022-03-20T11:17:43.036843Z","iopub.execute_input":"2022-03-20T11:17:43.037620Z","iopub.status.idle":"2022-03-20T11:17:46.528328Z","shell.execute_reply.started":"2022-03-20T11:17:43.037578Z","shell.execute_reply":"2022-03-20T11:17:46.527573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Defining Optimizer and Loss for the YOLO model","metadata":{"id":"WYARhdfOdASu"}},{"cell_type":"code","source":" # we are using graph mode of tensorflow so that we can use our own Gradient Tape\noptimizer = tf.keras.optimizers.Adam(learning_rate=1e-4) # Adam optimizers\nloss = [YoloLoss(anchors[mask], classes=num_classes) # customized yolo loss define above in utils.\n            for mask in anchor_masks]","metadata":{"id":"C7SjghvhVYhJ","execution":{"iopub.status.busy":"2022-03-20T11:17:46.529672Z","iopub.execute_input":"2022-03-20T11:17:46.530078Z","iopub.status.idle":"2022-03-20T11:17:46.535716Z","shell.execute_reply.started":"2022-03-20T11:17:46.530039Z","shell.execute_reply":"2022-03-20T11:17:46.534991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Calculating Loss using Gradient Tape","metadata":{"id":"wOxjIUzNCDTf"}},{"cell_type":"code","source":"import datetime\ncurrent_time = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") \ntrain_log_dir = '/kaggle/working/gradient_tape/' + current_time + '/train' # train dir path\ntest_log_dir = '/kaggle/working/gradient_tape/' + current_time + '/test' # test dir path\ntrain_summary_writer = tf.summary.create_file_writer(train_log_dir) # train writer\ntest_summary_writer = tf.summary.create_file_writer(test_log_dir) # test writer","metadata":{"id":"f3rozfk5qzMe","execution":{"iopub.status.busy":"2022-03-20T11:17:46.537001Z","iopub.execute_input":"2022-03-20T11:17:46.537496Z","iopub.status.idle":"2022-03-20T11:17:46.548298Z","shell.execute_reply.started":"2022-03-20T11:17:46.537447Z","shell.execute_reply":"2022-03-20T11:17:46.547573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"avg_loss = tf.keras.metrics.Mean('loss', dtype=tf.float32)\navg_val_loss = tf.keras.metrics.Mean('val_loss', dtype=tf.float32)\nckpt = tf.train.Checkpoint(step=tf.Variable(1), optimizer=optimizer, model = model)\nmanager = tf.train.CheckpointManager(ckpt, '/kaggle/working/yolov3_train/tf_ckpts', max_to_keep=3)\nckpt.restore(manager.latest_checkpoint)\nif manager.latest_checkpoint:\n    print(\"Restored from {}\".format(manager.latest_checkpoint))\n    start = ckpt.step.numpy() \nelse:\n    print(\"Initializing from scratch.\")\n    start = 0\n\nepochs = 50\nfor epoch in range(start, epochs+1):\n    for batch, (images, labels) in enumerate(train_dataset):\n        with tf.GradientTape() as tape:\n            outputs = model(images, training=True)\n            regularization_loss = tf.reduce_sum(model.losses)\n            pred_loss = []\n            for output, label, loss_fn in zip(outputs, labels, loss):\n                pred_loss.append(loss_fn(label, output))\n            total_loss = tf.reduce_sum(pred_loss) + regularization_loss\n        # calculating grads over trainable parameters\n        grads = tape.gradient(total_loss, model.trainable_variables) # calculating loss after each batch \n        optimizer.apply_gradients(\n            zip(grads, model.trainable_variables)) # then appliying gradient optimization on the loss to fine tune the weights \n\n        # writing summary to train file(for tensorboard)\n\n        with train_summary_writer.as_default():\n            tf.summary.scalar('avg_loss', total_loss.numpy(), step=epoch)\n        # to update avg loss after each batch.\n        avg_loss.update_state(total_loss)\n    \n    # testing datasets\n    for batch, (images, labels) in enumerate(val_dataset):\n        outputs = model(images)\n        regularization_loss = tf.reduce_sum(model.losses)\n        pred_loss = []\n        for output, label, loss_fn in zip(outputs, labels, loss):\n            pred_loss.append(loss_fn(label, output))\n        total_loss = tf.reduce_sum(pred_loss) + regularization_loss\n\n        # writing summary to test file(for tensorboard)\n\n        with test_summary_writer.as_default():\n            tf.summary.scalar('avg_val_loss', total_loss.numpy(), step=epoch)\n        avg_val_loss.update_state(total_loss)\n\n    # print result \n    print(\"{}, train: {}, val: {}\".format(epoch,\n        avg_loss.result().numpy(),\n        avg_val_loss.result().numpy()))\n    \n    ckpt.step.assign_add(1)\n    if int(ckpt.step) % 5 == 0:\n        save_path = manager.save()\n        print(\"Saved checkpoint for step {}: {}\".format(int(ckpt.step), save_path))\n\n    avg_loss.reset_states()\n    avg_val_loss.reset_states()","metadata":{"id":"fM3EjboZfVq4","outputId":"7e4e65a4-bccc-4e9b-dbef-5823561a8485","execution":{"iopub.status.busy":"2022-03-20T11:17:46.549755Z","iopub.execute_input":"2022-03-20T11:17:46.550039Z","iopub.status.idle":"2022-03-20T11:28:25.798723Z","shell.execute_reply.started":"2022-03-20T11:17:46.550002Z","shell.execute_reply":"2022-03-20T11:28:25.797986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.save_weights('/kaggle/working/Model_Save')","metadata":{"id":"6CZSiBdFIs7J","execution":{"iopub.status.busy":"2022-03-20T11:37:32.025920Z","iopub.execute_input":"2022-03-20T11:37:32.026240Z","iopub.status.idle":"2022-03-20T11:37:32.913464Z","shell.execute_reply.started":"2022-03-20T11:37:32.026208Z","shell.execute_reply":"2022-03-20T11:37:32.912636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls","metadata":{"execution":{"iopub.status.busy":"2022-03-20T11:37:36.580238Z","iopub.execute_input":"2022-03-20T11:37:36.581024Z","iopub.status.idle":"2022-03-20T11:37:37.366750Z","shell.execute_reply.started":"2022-03-20T11:37:36.580987Z","shell.execute_reply":"2022-03-20T11:37:37.365700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Utility functions that help in visualizations","metadata":{"id":"dnpaGqfcciiR"}},{"cell_type":"markdown","source":"## Creating new DF with bbox counts","metadata":{"id":"mxtKcDXPDnOp"}},{"cell_type":"markdown","source":"# Final Pipeline","metadata":{"id":"0JRv2j6bDWOQ"}},{"cell_type":"markdown","source":"## Visualising Dense images","metadata":{"id":"gFxPFBy4D-zE"}},{"cell_type":"markdown","source":"## Visualising sparse images","metadata":{"id":"QhIfpBr6EsRj"}},{"cell_type":"markdown","source":"## Visualizing very sparse images","metadata":{"id":"aLMY_zptExSe"}},{"cell_type":"markdown","source":"# Step by Step Procedure","metadata":{"id":"mlxJNSoGE6Wj"}},{"cell_type":"markdown","source":"1. Download the gloabl_wheat_detection zip from kaggle and unzip the data.\n2. Download the pretrained yolov3 weights.\n3. Preprocess the given dataframe by expanding the bbox dimensions and calculate x_min,y_min,x_max,y_max and save it.\n4. Split the data into train and validation data.\n5. Import all the necessary modules that are needed.\n6. Initialize the necessary variables like yolo_anchors, yolo_masks, yolo_max_boxes, batch_size, image_Size, num_Classes, class_dict that are useful for training.\n7. Define yolov3 architecture that is implemented compleyetly in kears and tensorflow.\n8. Define your own custom loss along with the yolo loss for object detection. I used focal loss for my task.\n9. Define parse_Dataset function which will help us the map the data that is needed for trainng by converting it into the format that yolov3 expects.\n10. Load both train and validation datasets.\n11. Initialize the model and load pretrained weights and freeze the layers that are not required for fine tuning.\n12. Start the trainnig. I used gradient tape method to train the model as it gives control over nitty gritty details in the model.\n13. While trainng make sure that we are saving logs and model weights for future use.\n14. Once the model is trained save the model weights instead of checkpoints.\n15. For making predictions, define yolo architecute again and load the saved weights and make the predictions and visualizations.\n","metadata":{"id":"GpkjSFofE9AD"}},{"cell_type":"markdown","source":"# Observations","metadata":{"id":"Goli_sbVHLDS"}},{"cell_type":"markdown","source":"1. When using categorical **cross entropy loss** for negative samples i.e Bounding box for which there is no object is to high and loss for positive samples is too low for which our optimizers tries to lower negative samples loss as it's higher.Therefore, our model wasn't working well at detections.\n2. So to overcome the above hurdle we have tried **focal loss** which try to down weight **negative sample loss** and hence improve our model performace.\n3. For this model I have taken alpha parameter of **focal loss 0.85**, I have also tried 0.60,0.65,0.70,0.75,0.80,0.85 but at last 0.75 suited best for this model.\n\n","metadata":{"id":"CjB8MtlgHNsL"}},{"cell_type":"markdown","source":"# Future Scope or Work","metadata":{"id":"glwtL1h5HSGk"}},{"cell_type":"markdown","source":"1. In our model we have used YOLOv3 which is a really good object detection technique but at the time of making this case study we already have YOLOV5 which is state of the art model.\n2. So using YOLOV5 might get some better results as yolov5 is faster than yolov3 and its accuracy is reasonable and comparable too.\n3. We have only 3k images for training, and we all know that more the data more the model performance for deep learning models. So training to incorporate Augmentations might be helpful.\n4. There is a module called Albumenations that will help immensely while augmenting images with bounding boxes.\n5. Instead of yolo we can use other models like Faster RCNN which is build for object detection tasks.\n6. We can get some more images from internet and pseudo labelling them carefully and addding those images for training might help too.\n","metadata":{"id":"HfH_s8IHHVwW"}},{"cell_type":"markdown","source":"# What I Gained?","metadata":{"id":"-pjspHfPIV33"}},{"cell_type":"markdown","source":"1. By doing this case study I got to know every single thing that YOLOV3 does internally for object detections.\n2. By implementing YOLOV3 completely in keras or tensoflow helped me gain confidence in me so that anyone with good knowledge can build an research on their own using all the tools available. \n3. If loss fucntion is continous and differentiable we can always back porpagate the loss and minimize loss using the techniques we learned in deep learning.\n4. Finally I think i will be albe to implement my own versions of other models in keras or tensorflow.","metadata":{"id":"tSGIrtTBIa3o"}}]}