{"cells":[{"metadata":{},"cell_type":"markdown","source":"<center><img src='https://i.ytimg.com/vi/WP4F6aK1Ft8/maxresdefault.jpg' size ='400px'></center>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### ***PURPOSE***<br>\n<tab>So why do i am presenting this notebook.just to contribute to the comunity.Today i was studing about how does the yolov3 works and i was doing some of the PoCs on how to implement and use yolov3 on my project to complete certain task.So this notebook is one result of 3 PoCs that i have done today and i think that i should show and get your feedback on this. I also believe that this notebook might help you in some way or it might be a source from where anyone could take insperation and use it to increase their understanding......<br>\nif you get some insperation you can use it in [this competition](https://www.kaggle.com/c/global-wheat-detection)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<font color=\"red\" size=3>Please UPVOTE this kernel if you like it. It motivates me to produce more quality content :) <br><br>\n    Please do comment what your views are and what you understand from this.\n    <br><br> Dont't forget to give your suggestions in the comment section </font>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**So why yolov3 not any other**<br>\n### 1. RCNN\n\nfrom last 5 years we have found leaps and bound growth in the object detection since the famous rcnn paper(https://arxiv.org/pdf/1311.2524.pdf)\n![see](https://cv-tricks.com/wp-content/uploads/2017/12/RCNN-e1514378306435.jpg)\n#### Selective Search:\n    1. Generate initial sub-segmentation, we generate many candidate regions\n    2. Use greedy algorithm to recursively combine similar regions into larger ones \n    3. Use the generated regions to produce the final candidate region proposals <br>\nthese region proposals are then sent to a cnn model like vgg or resnet and than a svm is used to classify the category and a bounding box regeressor <br>\n#### Problems:\n    1. It cannot be implemented real time as it takes around 40 seconds for each test image.\n    2. no learning is happening at the selective search algorithm stage Therefore, no learning is happening at that stage. This could lead to the generation of bad candidate region proposals.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2. **Spatial Pyramid Pooling(SPP-net)**\n\n1. so to tackle the rcnn slow process in this SPP-Net we calculate the CNN representation for entire image only once and can use that to calculate the CNN representation for each patch generated by Selective Search. This can be done by performing a pooling type of operation on JUST that section of the feature maps of last conv layer that corresponds to the region. this leads to significant difference in the execution and training time \n2. generate the fixed size of input for the fully connected layers of the CNN so, SPP introduces one more trick. It uses spatial pooling after the last convolutional layer as opposed to traditionally used max-pooling.\n    \nWith SPP, we don’t need to crop the image to fixed size, like AlexNet, before going into CNN. Any image sizes can be inputted.\n![see](https://miro.medium.com/max/1400/1*n4LE9idyGJX_efOsS-FNvw.png)\nafter these features are passes to Three-Level Spatial Pyramid Pooling (SPP) in SPPNet with Pyramid {4×4, 2×2, 1×1}. which inturn leads to svm for classification and bounding box \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3. **Fast R-CNN**\n\n1. The reason “Fast R-CNN” is faster than R-CNN is because you don’t have to feed 2000 region proposals to the convolutional neural network every time. Instead, the model has a ROI layer this RoI pooling layer turn the regions into a fixed size so that it can be fed into a fully connected layer. From the RoI feature vector, we use a softmax layer to predict the class of the proposed region and also the offset values for the bounding box.\n2. instead of feeding the region proposals to the CNN, we feed the input image to the CNN to generate a convolutional feature map. From the convolutional feature map adapted from the spp-net this reduces the need of repeatedly proposing the features \n3. this created first model with end-to-end learning strategy\n    \n![](https://miro.medium.com/max/1400/1*TDOx63GthXwgXJb3hXUsUQ.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 4. **Faster RCNN**\n\n1. the above algorithms(R-CNN & Fast R-CNN) uses selective search to find out the region proposals. Selective search is a slow and time-consuming process affecting the performance of the network.\n2. Faster RCNN replaces selective search with a very small convolutional network called Region Proposal Network to generate regions of Interests.\n3. To handle the variations in aspect ratio and scale of objects, Faster R-CNN introduces the idea of anchor boxes. At each location, the original paper uses 3 kinds of anchor boxes for scale 128x 128, 256×256 and 512×512. Similarly, for aspect ratio, it uses three aspect ratios 1:1, 2:1 and 1:2. \n4. RPN gives out bounding boxes of various sizes with the corresponding probabilities of each class. The varying sizes of bounding boxes can be passed further by apply Spatial Pooling just like Fast-RCNN\n\n![](https://miro.medium.com/max/1400/1*pSnVmJCyQIRKHDPt3cfnXA.png) ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 5. **YOLO**\n1. In YOLO detection is a simple regression problem which takes an input image and learns the class probabilities and bounding box coordinates\n2. Though it is no longer the most accurate object detection algorithm, it is a very good choice when you need real-time detection, without loss of too much accuracy.\n3.  YOLO divides each image into a grid of S x S and each grid predicts N bounding boxes and confidence. The confidence reflects the accuracy of the bounding box and whether the bounding box actually contains an object(regardless of class). YOLO also predicts the classification score for each box for every class in training. You can combine both the classes to calculate the probability of each class being present in a predicted box.\n4. Notice that at runtime, we have run our image on CNN only once. Hence, YOLO is super fast and can be run real time. Another key difference is that YOLO sees the complete image at once as opposed to looking at only a generated region proposals in the previous methods\n\n![](https://cv-tricks.com/wp-content/uploads/2017/12/model2-1024x280.jpg)    ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### But still th e question remains why yolov3\n   1. For the task of detection,it has 106 layer fully convolutional underlying architecture making it slower but also improving the accuracy or (MAP) for the detection\n   2. The most salient feature of v3 is that it makes detections at three different scales.\n   3. Better at detecting smaller objects as The upsampled layers concatenated with the previous layers help preserve the fine grained features which help in detecting small objects.\n   4. It predicts boxes at 3 different scales. For the same image it predicts 10x the number of boxes predicted by YOLO v2\n   5. No more softmaxing the classes as its requirement is to have mutually exclusive classes but different objects might have some relation between them which could not be independent\n   \n![](https://image.slidesharecdn.com/pr12yolov3-191117141544/95/pr207-yolov3-an-incremental-improvement-13-638.jpg?cb=1574001871)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### ***STRATING THE GAME***","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import struct\nimport numpy as np\nfrom keras.layers import Conv2D\nfrom keras.layers import Input\nfrom keras.layers import BatchNormalization\nfrom keras.layers import LeakyReLU\nfrom keras.layers import ZeroPadding2D\nfrom keras.layers import UpSampling2D\nfrom keras.layers.merge import add, concatenate\nfrom keras.models import Model\nfrom keras import backend as K\nfrom keras.models import load_model\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array\nfrom matplotlib.patches import Rectangle\nimport matplotlib.pyplot as plt\nimport imageio\nimport sys\nfrom keras.utils import plot_model\nimport matplotlib.image as mpimg\nimport cv2\nimport math\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# create a YOLOv3 Keras model and save it to file\ndef _conv_block(inp, convs, skip=True):\n    x = inp\n    count = 0\n    for conv in convs:\n        if count == (len(convs) - 2) and skip:\n            skip_connection = x\n        count += 1\n        if conv['stride'] > 1: x = ZeroPadding2D(((1,0),(1,0)))(x) \n        x = Conv2D(conv['filter'],\n                    conv['kernel'],\n                    strides=conv['stride'],\n                    padding='valid' if conv['stride'] > 1 else 'same',\n                    name='conv_' + str(conv['layer_idx']),\n                   use_bias=False if conv['bnorm'] else True)(x)\n        if conv['bnorm']: x = BatchNormalization(epsilon=0.001, name='bnorm_' + str(conv['layer_idx']))(x)\n        if conv['leaky']: x = LeakyReLU(alpha=0.1, name='leaky_' + str(conv['layer_idx']))(x)\n    return add([skip_connection, x]) if skip else x\n\ndef make_yolov3_model():\n    input_image = Input(shape=(None, None, 3))\n    # Layer  0 => 4\n    x = _conv_block(input_image, [{'filter': 32, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 0},\n                                  {'filter': 64, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 1},\n                                  {'filter': 32, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 2},\n                                  {'filter': 64, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 3}])\n    # Layer  5 => 8\n    x = _conv_block(x, [{'filter': 128, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 5},\n                        {'filter':  64, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 6},\n                        {'filter': 128, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 7}])\n    # Layer  9 => 11\n    x = _conv_block(x, [{'filter':  64, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 9},\n                        {'filter': 128, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 10}])\n    # Layer 12 => 15\n    x = _conv_block(x, [{'filter': 256, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 12},\n                        {'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 13},\n                        {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 14}])\n    # Layer 16 => 36\n    for i in range(7):\n        x = _conv_block(x, [{'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 16+i*3},\n                            {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 17+i*3}])\n    skip_36 = x\n    # Layer 37 => 40\n    x = _conv_block(x, [{'filter': 512, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 37},\n                        {'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 38},\n                        {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 39}])\n    # Layer 41 => 61\n    for i in range(7):\n        x = _conv_block(x, [{'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 41+i*3},\n                            {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 42+i*3}])\n    skip_61 = x\n    # Layer 62 => 65\n    x = _conv_block(x, [{'filter': 1024, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 62},\n                        {'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 63},\n                        {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 64}])\n    # Layer 66 => 74\n    for i in range(3):\n        x = _conv_block(x, [{'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 66+i*3},\n                            {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 67+i*3}])\n    # Layer 75 => 79\n    x = _conv_block(x, [{'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 75},\n                        {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 76},\n                        {'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 77},\n                        {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 78},\n                        {'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 79}], skip=False)\n    # Layer 80 => 82\n    yolo_82 = _conv_block(x, [{'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 80},\n                              {'filter':  255, 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 81}], skip=False)\n    # Layer 83 => 86\n    x = _conv_block(x, [{'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 84}], skip=False)\n    x = UpSampling2D(2)(x)\n    x = concatenate([x, skip_61])\n    # Layer 87 => 91\n    x = _conv_block(x, [{'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 87},\n                        {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 88},\n                        {'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 89},\n                        {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 90},\n                        {'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 91}], skip=False)\n    # Layer 92 => 94\n    yolo_94 = _conv_block(x, [{'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 92},\n                              {'filter': 255, 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 93}], skip=False)\n    # Layer 95 => 98\n    x = _conv_block(x, [{'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True,   'layer_idx': 96}], skip=False)\n    x = UpSampling2D(2)(x)\n    x = concatenate([x, skip_36])\n    # Layer 99 => 106\n    yolo_106 = _conv_block(x, [{'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 99},\n                                 {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 100},\n                                 {'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 101},\n                                 {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 102},\n                                 {'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 103},\n                                {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 104},\n                               {'filter': 255, 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 105}], skip=False)\n    model = Model(input_image, [yolo_82, yolo_94, yolo_106])\n    return model\nmodel = make_yolov3_model()\nplot_model(model, to_file='model.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The above model approach clearly shows that how is the architecture of the yolov3 model  **<br>\nReading the weights of the v3 model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class WeightReader:\n    def __init__(self, weight_file):\n        with open(weight_file, 'rb') as w_f:\n            major,\t= struct.unpack('i', w_f.read(4))\n            minor,\t= struct.unpack('i', w_f.read(4))\n            revision, = struct.unpack('i', w_f.read(4))\n            if (major*10 + minor) >= 2 and major < 1000 and minor < 1000:\n                w_f.read(8)\n            else:\n                w_f.read(4)\n            transpose = (major > 1000) or (minor > 1000)\n            binary = w_f.read()\n        self.offset = 0\n        self.all_weights = np.frombuffer(binary, dtype='float32')\n\n    def read_bytes(self, size):\n        self.offset = self.offset + size\n        return self.all_weights[self.offset-size:self.offset]\n\n    def load_weights(self, model):\n        for i in range(106):\n            try:\n                conv_layer = model.get_layer('conv_' + str(i))\n                print(\"loading weights of convolution #\" + str(i))\n                if i not in [81, 93, 105]:\n                    norm_layer = model.get_layer('bnorm_' + str(i))\n                    size = np.prod(norm_layer.get_weights()[0].shape)\n                    beta  = self.read_bytes(size) \n                    gamma = self.read_bytes(size) \n                    mean  = self.read_bytes(size) \n                    var   = self.read_bytes(size) \n                    weights = norm_layer.set_weights([gamma, beta, mean, var])\n                if len(conv_layer.get_weights()) > 1:\n                    bias   = self.read_bytes(np.prod(conv_layer.get_weights()[1].shape))\n                    kernel = self.read_bytes(np.prod(conv_layer.get_weights()[0].shape))\n                    kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))\n                    kernel = kernel.transpose([2,3,1,0])\n                    conv_layer.set_weights([kernel, bias])\n                else:\n                    kernel = self.read_bytes(np.prod(conv_layer.get_weights()[0].shape))\n                    kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))\n                    kernel = kernel.transpose([2,3,1,0])\n                    conv_layer.set_weights([kernel])\n            except ValueError:\n                print(\"no convolution #\" + str(i))\n\n    def reset(self):\n        self.offset = 0\n\nmodel = make_yolov3_model()\nweight_reader = WeightReader('/kaggle/input/yolov3/yolov3.weights')\nweight_reader.load_weights(model)\nmodel.save('model.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i1=\"/kaggle/input/images2/solidWhiteCurve.jpg\"\nimg = mpimg.imread(i1)\nplt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Here I will show how i am detecting the lanes using cv2's canny edge and HoughLinesP**\n### 1. **Canny Edge detection**<br>\nThe Canny edge detector is an edge detection operator that uses a multi-stage algorithm to detect a wide range of edges in images. It was developed by John F. Canny in 1986. Canny also produced a computational theory of edge detection explaining why the technique works. (Wikipedia)\nfirst we will apply gaussian blur to remove the noise then we will use to find the magnitude and slope of the gradient with the help os sober filters we will have edges with high and low intensity . non max supresstion is then used on these intensities which supresses the small of low intensity edges leading us to have only high intensity edge . then the still remaining weak pixels are than converted to strong pixels only is they have strong pixels near them else they are finally supressed.\n\n![](https://scikit-image.org/docs/0.5/_images/plot_canny_1.png)\n### 2. ** HoughLinesProbablistic**<br>\nHough transform is a feature extraction method for detecting simple shapes such as circles, lines etc in an image. first we create an accumulater which we can think of a 2D array which then we will fill this array with the edges pixels no we incrementally select and update the accumulater with edge interaction pixels leading us to fill up the whole 2D matrix through which We can simply select the bins in the accumulator above a certain threshold to find the lines in the image. If the threshold is higher, you will find fewer strong lines, and if it is lower, you will find a large number of lines including some weak ones\n\n![](https://scikit-image.org/docs/0.11.x/_images/plot_line_hough_transform_2.png)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def region_of_interest(img, vertices):\n    mask = np.zeros_like(img)\n    match_mask_color = 255\n    cv2.fillPoly(mask, vertices, match_mask_color)\n    masked_image = cv2.bitwise_and(img, mask)\n    return masked_image\n\ndef draw_lines(img, lines, color=[255, 0, 0], thickness=3):\n    if lines is None:\n        return \n    img = np.copy(img) \n    line_img = np.zeros(\n        (\n            img.shape[0],\n            img.shape[1],\n            3\n        ),\n        dtype=np.uint8,\n    )    \n    for line in lines:\n        for x1, y1, x2, y2 in line:\n            cv2.line(line_img, (x1, y1), (x2, y2), color, thickness)    \n    img = cv2.addWeighted(img, 0.8, line_img, 1.0, 0.0) \n    return img\n\ndef pipeline(image):\n    \"\"\"\n    An image processing pipeline which will output\n    an image with the lane lines annotated.\n    \"\"\" \n    height = image.shape[0]\n    width = image.shape[1]\n    region_of_interest_vertices = [\n        (0, height),\n        (width / 2, height / 2),\n        (width, height),\n    ]    \n    gray_image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)    \n    cannyed_image = cv2.Canny(gray_image, 100, 200)\n \n    cropped_image = region_of_interest(\n        cannyed_image,\n        np.array(\n            [region_of_interest_vertices],\n            np.int32\n        ),\n    )\n \n    lines = cv2.HoughLinesP(\n        cropped_image,\n        rho=6,\n        theta=np.pi / 60,\n        threshold=160,\n        lines=np.array([]),\n        minLineLength=40,\n        maxLineGap=25\n    )\n \n    left_line_x = []\n    left_line_y = []\n    right_line_x = []\n    right_line_y = []\n \n    for line in lines:\n        for x1, y1, x2, y2 in line:\n            slope = (y2 - y1) / (x2 - x1)\n            if math.fabs(slope) < 0.5: \n                continue\n            if slope <= 0: \n                left_line_x.extend([x1, x2])\n                left_line_y.extend([y1, y2])\n            else: \n                right_line_x.extend([x1, x2])\n                right_line_y.extend([y1, y2]) \n    min_y = int(image.shape[0] * (3 / 5)) \n    max_y = int(image.shape[0])\n    poly_left = np.poly1d(np.polyfit(\n        left_line_y,\n        left_line_x,\n        deg=1\n    ))\n \n    left_x_start = int(poly_left(max_y))\n    left_x_end = int(poly_left(min_y))\n \n    poly_right = np.poly1d(np.polyfit(\n        right_line_y,\n        right_line_x,\n       deg=1\n    ))\n    right_x_start = int(poly_right(max_y))\n    right_x_end = int(poly_right(min_y)) \n    \n    line_image = draw_lines(\n        image,\n        [[\n            [left_x_start, max_y, left_x_end, min_y],\n            [right_x_start, max_y, right_x_end, min_y],\n        ]],\n        thickness=5,\n    )    \n    pts = np.array([[left_x_start, max_y],[left_x_end, min_y],[right_x_start, max_y],\n                    [right_x_end, min_y]], np.int32)\n    return line_image,pts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class BoundBox:\n    def __init__(self, xmin, ymin, xmax, ymax, objness = None, classes = None):\n        self.xmin = xmin\n        self.ymin = ymin\n        self.xmax = xmax\n        self.ymax = ymax\n        self.objness = objness\n        self.classes = classes\n        self.label = -1\n        self.score = -1\n\n    def get_label(self):\n        if self.label == -1:\n            self.label = np.argmax(self.classes)\n\n        return self.label\n\n    def get_score(self):\n        if self.score == -1:\n            self.score = self.classes[self.get_label()]\n\n        return self.score\n\ndef _sigmoid(x):\n    return 1. / (1. + np.exp(-x))\n\n\ndef decode_netout(netout, anchors, obj_thresh, net_h, net_w):\n    \"\"\" \n        This is the inference part of yolo outputs as is take predefined\n        anchors and netoutput which is a list of 106th,82th,86th layers\n        \n    \"\"\"\n    grid_h, grid_w = netout.shape[:2]\n    nb_box = 3\n    netout = netout.reshape((grid_h, grid_w, nb_box, -1))\n    nb_class = netout.shape[-1] - 5\n    boxes = []\n    netout[..., :2]  = _sigmoid(netout[..., :2])\n    netout[..., 4:]  = _sigmoid(netout[..., 4:])\n    netout[..., 5:]  = netout[..., 4][..., np.newaxis] * netout[..., 5:]\n    netout[..., 5:] *= netout[..., 5:] > obj_thresh\n\n    for i in range(grid_h*grid_w):\n        row = i / grid_w\n        col = i % grid_w\n        for b in range(nb_box):\n            objectness = netout[int(row)][int(col)][b][4]\n            if(objectness.all() <= obj_thresh): continue\n            x, y, w, h = netout[int(row)][int(col)][b][:4]\n            x = (col + x) / grid_w \n            y = (row + y) / grid_h \n            w = anchors[2 * b + 0] * np.exp(w) / net_w \n            h = anchors[2 * b + 1] * np.exp(h) / net_h \n            classes = netout[int(row)][col][b][5:]\n            box = BoundBox(x-w/2, y-h/2, x+w/2, y+h/2, objectness, classes)\n            boxes.append(box)\n    return boxes\n\ndef correct_yolo_boxes(boxes, image_h, image_w, net_h, net_w):\n    new_w, new_h = net_w, net_h\n    for i in range(len(boxes)):\n        x_offset, x_scale = (net_w - new_w)/2./net_w, float(new_w)/net_w\n        y_offset, y_scale = (net_h - new_h)/2./net_h, float(new_h)/net_h\n        boxes[i].xmin = int((boxes[i].xmin - x_offset) / x_scale * image_w)\n        boxes[i].xmax = int((boxes[i].xmax - x_offset) / x_scale * image_w)\n        boxes[i].ymin = int((boxes[i].ymin - y_offset) / y_scale * image_h)\n        boxes[i].ymax = int((boxes[i].ymax - y_offset) / y_scale * image_h)\n\ndef _interval_overlap(interval_a, interval_b):\n    x1, x2 = interval_a\n    x3, x4 = interval_b\n    if x3 < x1:\n        if x4 < x1:\n            return 0\n        else:\n            return min(x2,x4) - x1\n    else:\n        if x2 < x3:\n             return 0\n        else:\n            return min(x2,x4) - x3\n\ndef bbox_iou(box1, box2):\n    intersect_w = _interval_overlap([box1.xmin, box1.xmax], [box2.xmin, box2.xmax])\n    intersect_h = _interval_overlap([box1.ymin, box1.ymax], [box2.ymin, box2.ymax])\n    intersect = intersect_w * intersect_h\n    w1, h1 = box1.xmax-box1.xmin, box1.ymax-box1.ymin\n    w2, h2 = box2.xmax-box2.xmin, box2.ymax-box2.ymin\n    union = w1*h1 + w2*h2 - intersect\n    return float(intersect) / union\n\ndef do_nms(boxes, nms_thresh):\n    if len(boxes) > 0:\n        nb_class = len(boxes[0].classes)\n    else:\n        return\n    for c in range(nb_class):\n        sorted_indices = np.argsort([-box.classes[c] for box in boxes])\n        for i in range(len(sorted_indices)):\n            index_i = sorted_indices[i]\n            if boxes[index_i].classes[c] == 0: continue\n            for j in range(i+1, len(sorted_indices)):\n                index_j = sorted_indices[j]\n                if bbox_iou(boxes[index_i], boxes[index_j]) >= nms_thresh:\n                    boxes[index_j].classes[c] = 0\n\n                    \ndef load_image_pixels(filename, shape):\n    image = load_img(filename)\n    width, height = image.size\n    image = load_img(filename, target_size=shape)\n    image = img_to_array(image)\n    image = image.astype('float32')\n    image /= 255.0\n    image = np.expand_dims(image, 0)\n    return image, width, height\n\n\ndef get_boxes(boxes, labels, thresh):\n    v_boxes, v_labels, v_scores = list(), list(), list()\n    for box in boxes:\n        for i in range(len(labels)):\n            if box.classes[i] > thresh:\n                v_boxes.append(box)\n                v_labels.append(labels[i])\n                v_scores.append(box.classes[i]*100)\n    return v_boxes, v_labels, v_scores\n\ndef draw_boxes(filename, v_boxes, v_labels, v_scores,pshow,pipeline):\n    q,pts=pipeline(pshow)\n    pts = pts.reshape((-1,1,2))\n    cv2.fillPoly(q,[pts],(0,0,255))\n    plt.imshow(q)\n    ax = plt.gca()\n    for i in range(len(v_boxes)):\n        box = v_boxes[i]\n        y1, x1, y2, x2 = box.ymin, box.xmin, box.ymax, box.xmax\n        width, height = x2 - x1, y2 - y1\n        rect = Rectangle((x1, y1), width, height, fill=False, color='white')\n        ax.add_patch(rect)\n        label = \"%s\" % (v_labels[i])\n        plt.text(x1, y1, label, color='white')\n    return plt.show()\nmodel = load_model('model.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Non Max Supression**\n![](https://lilianweng.github.io/lil-log/assets/images/non-max-suppression.png)\n\nThis algorithms can be simply expained as first we select a proposal with highest confidence score, remove it from B and add it to the final proposal in to a new list D. Now compare this proposal with all the proposals and then we will calculate the IOU of this proposal with every other proposal. If the IOU is greater than the threshold N, remove that proposal from B.Again take the proposal with the highest confidence from the remaining proposals in B and remove it from B and add it to D.Once again calculate the IOU of this proposal with all the proposals in B and eliminate the boxes which have high IOU than threshold.This process is repeated until there are no more proposals left in B.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### **compute** == **\"Main Function\"**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def compute(photo_filename,img,pipeline):\n    \n    input_w, input_h = 416,416\n    anchors = [[116,90, 156,198, 373,326], [30,61, 62,45, 59,119], [10,13, 16,30, 33,23]]\n    class_threshold = 0.6\n    \n    labels = [\"person\", \"bicycle\", \"car\", \"motorbike\", \"aeroplane\", \"bus\", \"train\", \"truck\",\n    \"boat\", \"traffic light\", \"fire hydrant\", \"stop sign\", \"parking meter\", \"bench\",\n    \"bird\", \"cat\", \"dog\", \"horse\", \"sheep\", \"cow\", \"elephant\", \"bear\", \"zebra\", \"giraffe\",\n    \"backpack\", \"umbrella\", \"handbag\", \"tie\", \"suitcase\", \"frisbee\", \"skis\", \"snowboard\",\n    \"sports ball\", \"kite\", \"baseball bat\", \"baseball glove\", \"skateboard\", \"surfboard\",\n    \"tennis racket\", \"bottle\", \"wine glass\", \"cup\", \"fork\", \"knife\", \"spoon\", \"bowl\", \"banana\",\n    \"apple\", \"sandwich\", \"orange\", \"broccoli\", \"carrot\", \"hot dog\", \"pizza\", \"donut\", \"cake\",\n    \"chair\", \"sofa\", \"pottedplant\", \"bed\", \"diningtable\", \"toilet\", \"tvmonitor\", \"laptop\", \"mouse\",\n    \"remote\", \"keyboard\", \"cell phone\", \"microwave\", \"oven\", \"toaster\", \"sink\", \"refrigerator\",\n    \"book\", \"clock\", \"vase\", \"scissors\", \"teddy bear\", \"hair drier\", \"toothbrush\"]\n    \n    if type(photo_filename)==str:\n        image, image_w, image_h = load_image_pixels(photo_filename, (input_w, input_h))\n    else:\n        image= cv2.resize(photo_filename, (input_w, input_h))\n        image_w,image_h=image.shape[1],image.shape[0]\n        image=np.expand_dims(image,axis=0)\n        \n    yhat = model.predict(image)\n    boxes = list()\n    for i in range(len(yhat)):\n        boxes += decode_netout(yhat[i][0], anchors[i], class_threshold, input_h, input_w)\n    correct_yolo_boxes(boxes, image_h, image_w, input_h, input_w)\n    do_nms(boxes, 0.5)\n    \n    v_boxes, v_labels, v_scores = get_boxes(boxes, labels, class_threshold)\n    draw_boxes(photo_filename, v_boxes, v_labels, v_scores,img,pipeline)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **RUN IT ALL**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\nprint (time.asctime( time.localtime(time.time()) ))\ncap = cv2.VideoCapture('/kaggle/input/videos/solidWhiteRight.mp4')\nout = cv2.VideoWriter('outpy.avi',cv2.VideoWriter_fourcc(*'MJPG'),30.0, (640,480))\nwhile(cap.isOpened()):\n    ret,frame = cap.read()\n    cv2.imwrite('1.jpg',frame)\n    if ret==True:\n        a=compute('1.jpg',img,pipeline)\n        out.write(a)\n    else:\n        break\ncap.release()\nprint (time.asctime( time.localtime(time.time()) ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ***References***\n1. https://lilianweng.github.io this is very informative page in all terms seriously i have learn a lot of things it\n2. I might have left the concept of [mAP](https://datascience.stackexchange.com/questions/25119/how-to-calculate-map-for-detection-task-for-the-pascal-voc-challenge) so please visit this question and answer pair on stackoverflow\n3. If you want to implement yolo from scratch in pytorch you can visit https://blog.paperspace.com/how-to-implement-a-yolo-object-detector-in-pytorch/ man this is good \n4. Want to deeply analyze the benchmarks see https://medium.com/@jonathan_hui/object-detection-speed-and-accuracy-comparison-faster-r-cnn-r-fcn-ssd-and-yolo-5425656ae359 i don't know anything better than this post interms of object detection benchmarking\n5. Also i Have not touched the bounding box regression very deeply in Yolo so keen learners can visit http://christopher5106.github.io/object/detectors/2017/08/10/bounding-box-object-detectors-understanding-yolo.html\n6. https://navoshta.com/detecting-road-features/ goto blogpost if you are interested in self-driving or road-features\n7. In the end you can visit https://www.kaggle.com/ because i think this is also a \"\"goood\"\" site.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### ***ENDNOTE***\n\n\nI just want to say thankyou for other unknown kagglers and people that provide information they accuired from any where and share it keeping the free flow of knowledge alive respect++<br>\n\n\n**CHRONOLOGY samaj lijiye**\n![](https://pbs.twimg.com/media/DwJmqj1V4AEcLvA?format=jpg&name=small)\n\n\nIf you like this tutorial on how to use yolo with opencv please <font color=\"red\" size=3> UPVOTE</font>because\n![](https://media3.giphy.com/media/SnAXCQTYU4Awg/200.webp?cid=ecf05e47733219bb765a346445787ebe7b41c48d22cc7d8d&rid=200.webp)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}