{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **What Is R-CNN?**\n\n\nRegion-CNN (R-CNN), originally proposed in 2014 by Ross Girshik et. al., is a deep learning object detection algorithm that aims to find and classify multiple objects within an image.\n\nThere are two main problems R-CNN addresses:\n\n*     The algorithm doesn’t know in advance how many objects there will be in the image. This makes it difficult to use a Convolutional Neural Network (CNN), because the input is of variable length.\n*     There is a dilemma with regard to identifying objects in the image━you can arbitrarily choose a few regions and classify them, but then risk missing the important objects. Or check every possible region in the image, which would take too long to run.\n\n* R-CNN addresses the problems above using Selective Search. This involves sliding a window over the image to generate “region proposals”━areas where objects could possibly be found. The sliding window is in fact composed of several windows, each with different aspect ratios, to capture objects that appear in different sizes and are pictured from different angles.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](http://missinglink.ai/wp-content/uploads/2019/08/R-CNN.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Using this sliding window, R-CNN generates 2,000 region proposals. It uses a greedy algorithm to recursively combine similar regions into one. The remaining list of regions is fed into a CNN━solving the variable input problem, because the number of areas for classification is now known.\n\nThen, R-CNN may use one of several CNN architectures including AlexNet, VF, VGG, MobileNet or DenseNet to classify each of the candidate regions. Finally, it uses regression to predict the correct coordinates for the bounding box of each object (because the original Selective Search may not have accurately captured the entire object).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **What Is Faster R-CNN?**\n\nThe main problem with R-CNN is that it is very slow to run. It can take 47 seconds to process one image on a standard deep learning machine, making it unusable for real-time image processing scenarios.\n\nThe main thing that slows down R-CNN is the Selective Search mechanism that proposes many possible regions and requires classifying all of them. In addition, the region selection process is not “deep” and there is no learning involved, limiting its accuracy. In 2015 Girshik proposed an improved algorithm called Fast R-CNN, but it still relied on Selective Search, limiting its performance.\n\nShoqing Ren et. al. proposed an improved algorithm called Faster R-CNN, which does away with Selective Search altogether and lets the network learn the region proposals directly. Faster R-CNN takes the source image and inputs it to a CNN called a Region Prediction Network (RPN). It considers a large number of possible regions, even more than in the original R-CNN algorithm, and uses an efficient deep learning method to predict which regions are most likely to be objects of interest.\n\nThe predicted region proposals are then reshaped using a Region of Interest (RoI) pooling layer. This layer itself is used to classify the images within each region and predict the offset values for the bounding boxes.\n\nThe image below shows the huge performance gains that Faster R-CNN achieves compared to the original R-CNN and Fast R-CNN proposed by Girshik’s team.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](http://missinglink.ai/wp-content/uploads/2019/08/Faster-R-CNN.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# > **Object Detection with Faster R-CNN: How it Works**\n\n\n![](http://miro.medium.com/max/1282/1*WO3athE5rXRW76CGbEqk9w.jpeg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Step 1: Anchors\n\nFaster R-CNN uses a system of ‘anchors’, allowing the operator to define the possible regions that will be fed into the Region Prediction Network. An anchor is a box. The image below shows an image with size (600, 800) with nine anchors, reflecting three possible sizes and three aspect ratios━1:1, 1:2 and 2:1.\n\n![](http://missinglink.ai/wp-content/uploads/2019/08/Anchors.png)\n\nGiven a stride of 16, meaning each of the anchors will slide over the image skipping 16 pixels at a time, there will be almost 18,000 possible regions. It is possible to fine-tune the anchors to suit the object detection problem at hand━for example if you need to identify people or cars from a distance in a surveillance video, you may focus the anchor on smaller sizes and appropriate aspect ratios.\n\n## Step 2: Region Proposal Network (RPN)\n\n\nThe algorithm feeds the possible regions, generated by the anchors defined in the previous step, into the RPN, a special CNN used for predicting regions with objects of interest. The RPN predicts the possibility of an anchor being background or foreground and refines the anchor or bounding box.\n\nThe training data of the RPN is the anchors and a set of ground-truth boxes. Anchors that have a higher overlap with ground-truth boxes should be labeled as foreground, while others should be labeled as background. The RPN convolves the image into features and considers each feature using the 9 anchors, with two possible labels for each (background or foreground).\n\nFinally, the output is fed into a Softmax or logistic regression activation function, to predict the labels for each anchor. A similar process is used to refine the anchors and define the bounding boxes for the selected features. Anchors that are found to be foreground are passed to the next stage of the R-CNN algorithm.\n\n\n![](http://tryolabs.com/images/blog/post-images/2018-01-18-faster-rcnn/rpn-architecture.99b6c089.png)\n\n\n\n## Step 3: Region of Interest (RoI) pooling\n\nThe RPN provides proposed regions with different sizes. Each of these is a CNN feature map with a different size. Now the algorithm applies Region of Interest (RoI) pooling to reduce all the feature maps to the same size.\n\n\n![](https://missinglink.ai/wp-content/uploads/2019/08/Region-of-Interest-RoI-pooling-1.png)\n\n\nFaster R-CNN performs RoI pooling using the original R-CNN architecture. It takes the feature map for each region proposal, flattens it, and passes it through two fully-connected layers with ReLU activation. It then uses two different fully-connected layers to generate a prediction for each of the objects.\n\n![](https://missinglink.ai/wp-content/uploads/2019/08/Region-of-Interest-RoI-pooling-2.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![Faster-RCNN block diagram. The magenta colored blocks are active only during training. The numbers indicate size of the tensors.](https://miro.medium.com/max/2000/1*1Mj0C4wzi57Z6Z933gb6vA.png)\n\nFaster-RCNN block diagram. The magenta colored blocks are active only during training. The numbers indicate size of the tensors.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Do we need any special models?\n\n- Let's use SOTA\n\n\n# Facebook Research to the Rescue\n\n![](https://raw.githubusercontent.com/facebookresearch/detectron2/master/.github/Detectron2-Logo-Horz.svg?sanitize=true)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Detectron2 is Facebook AI Research's next generation software system that implements state-of-the-art object detection algorithms. It is a ground-up rewrite of the previous version, Detectron, and it originates from maskrcnn-benchmark.\n\n\n\n# **What's New**\n\n*     It is powered by the PyTorch deep learning framework.\n*    Includes more features such as panoptic segmentation, densepose, Cascade R-CNN, rotated bounding boxes, etc.\n*    Can be used as a library to support different projects on top of it. We'll open source more research projects in this way.\n*   It trains much faster.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](http://miro.medium.com/max/2000/1*5mz6xC1oLPVdu8CIqXio4w.png)\n\n\n Detailed architecture of Base-RCNN-FPN. Blue labels represent class names.\n ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"If you are comfortable with handling json file formats you might want to have a look at [Colab Notebook](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5)\n\nIn case you have difficulty understanding the above code don't worry the later sections of this notebook contains everything formatted just in case for you, all you gotta do is load the data, partition it, install dependencies and fit the data in the model.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Will be documenting this down even more so everyone understands the building blocks of Faster-RCNN and Detecteron 2\n\n\nDo Upvote if you liked my kernel __/\\__","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install '/kaggle/input/torch-15/torch-1.5.0cu101-cp37-cp37m-linux_x86_64.whl'\n!pip install '/kaggle/input/torch-15/torchvision-0.6.0cu101-cp37-cp37m-linux_x86_64.whl'\n!pip install '/kaggle/input/torch-15/yacs-0.1.7-py3-none-any.whl'\n!pip install '/kaggle/input/torch-15/fvcore-0.1.1.post200513-py3-none-any.whl'\n!pip install '/kaggle/input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl'\n!pip install '/kaggle/input/detectron2/detectron2-0.1.3cu101-cp37-cp37m-linux_x86_64.whl'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# install dependencies: (use cu101 because colab has CUDA 10.1)\n!pip install -U torch==1.5 torchvision==0.6 -f https://download.pytorch.org/whl/cu101/torch_stable.html \n!pip install cython pyyaml==5.1\n!pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'\nimport torch, torchvision\nprint(torch.__version__, torch.cuda.is_available())\n!gcc --version\n# opencv is pre-installed on colab","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# install detectron2:\n!pip install detectron2==0.1.3 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm.notebook import tqdm\nimport gc\nimport os\nfrom glob import glob\nimport cv2\n\nfrom PIL import Image\nimport random\nfrom collections import deque, defaultdict\nfrom multiprocessing import Pool, Process\nfrom functools import partial\n\nimport pycocotools\nimport detectron2\nfrom detectron2.config import get_cfg\nfrom detectron2 import model_zoo\nfrom detectron2.engine import DefaultPredictor, DefaultTrainer\nfrom detectron2.utils.visualizer import Visualizer, ColorMode\nfrom detectron2.structures import BoxMode\nfrom detectron2.data import datasets, DatasetCatalog, MetadataCatalog","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_feature(df, feature):\n    \n    plt.figure(figsize=(15,8))\n    ax = sns.countplot(y=feature, data=df, order=df[feature].value_counts().index)\n\n    for p in ax.patches:\n        ax.annotate('{:.2f}%'.format(100*p.get_width()/df.shape[0]), (p.get_x() + p.get_width() + 0.02, p.get_y() + p.get_height()/2))\n\n    plt.title(f'Distribution of {feature}', size=25, color='b')    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MAIN_PATH = '/kaggle/input/global-wheat-detection'\nTRAIN_IMAGE_PATH = os.path.join(MAIN_PATH, 'train/')\nTEST_IMAGE_PATH = os.path.join(MAIN_PATH, 'test/')\nTRAIN_PATH = os.path.join(MAIN_PATH, 'train.csv')\nSUB_PATH = os.path.join(MAIN_PATH, 'sample_submission.csv')\n\nSEED_COLOR = 37\nNUMBER_TRAIN_SAMPLE = -1\nMODEL_PATH = 'COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml'\nWEIGHT_PATH = \"COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml\"\nEPOCH = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_img = glob(f'{TRAIN_IMAGE_PATH}/*.jpg')\ntest_img = glob(f'{TEST_IMAGE_PATH}/*.jpg')\n\nprint(f'Number of train image:{len(train_img)}, test image:{len(test_img)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df = pd.read_csv(SUB_PATH)\nsub_df.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_PATH)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_source = train_df['source'].unique().tolist()\nprint(list_source)\ndisplay_feature(train_df, 'source')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nimage_unique = train_df['image_id'].unique()\nimage_unique_in_train_path = [i for i in image_unique if i + '.jpg' in os.listdir(TRAIN_IMAGE_PATH)]\n\nprint(f'Number of image unique: {len(image_unique)}, in train path: {len(image_unique_in_train_path)}')\n\ndel image_unique, image_unique_in_train_path\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def list_color(seed):\n    class_unique = sorted(train_df['source'].unique().tolist())\n    dict_color = dict()\n    random.seed(seed)\n    for classid in class_unique:\n        dict_color[classid] = random.sample(range(256), 3)\n    \n    return dict_color\n\n\ndef display_image(df, folder, num_img=3):\n    \n    if df is train_df:\n        dict_color = list_color(SEED_COLOR)\n        \n    for i in range(num_img):\n        fig, ax = plt.subplots(figsize=(15, 15))\n        img_random = random.choice(df['image_id'].unique())\n        assert (img_random + '.jpg') in os.listdir(folder)\n        \n        img_df = df[df['image_id']==img_random]\n        img_df.reset_index(drop=True, inplace=True)\n        \n        img = cv2.imread(os.path.join(folder, img_random + '.jpg'))\n        for row in range(len(img_df)):\n            source = img_df.loc[row, 'source']\n            box = img_df.loc[row, 'bbox'][1:-1]\n            box = list(map(float, box.split(', ')))\n            x, y, w, h = list(map(int, box))\n            if df is train_df:\n                cv2.rectangle(img, (x, y), (x+w, y+h), dict_color[source], 2)\n            else:\n                cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 255), 2)\n                \n        ax.set_title(f'{img_random} have {len(img_df)} bbox')\n        ax.imshow(img)   \n        \n    plt.show()        \n    plt.tight_layout()\n    \ndisplay_image(train_df, TRAIN_IMAGE_PATH)    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndef wheat_dataset(df, folder, is_train, img_unique):\n    img_id, img_name = img_unique\n    if is_train:\n        img_group = df[df['image_id']==img_name].reset_index(drop=True)\n        record = defaultdict()\n        img_path = os.path.join(folder, img_name+'.jpg')\n        \n        record['file_name'] = img_path\n        record['image_id'] = img_id\n        record['height'] = int(img_group.loc[0, 'height'])\n        record['width'] = int(img_group.loc[0, 'width'])\n        \n        annots = deque()\n        for _, ant in img_group.iterrows():\n            source = ant.source\n            annot = defaultdict()\n            box = ant.bbox[1:-1]\n            box = list(map(float, box.split(', ')))\n            x, y, w, h = list(map(int, box))\n            \n            annot['bbox'] = (x, y, x+w, y+h)\n            annot['bbox_mode'] = BoxMode.XYXY_ABS\n            annot['category_id'] = list_source.index(source)\n            \n            annots.append(dict(annot))\n            \n        record['annotations'] = list(annots)\n    \n    else:\n        img_group = df[df['image_id']==img_name].reset_index(drop=True)\n        record = defaultdict()\n        img_path = os.path.join(folder, img_name+'.jpg')\n        img = cv2.imread(img_path)\n        h, w = img.shape[:2]\n        \n        record['file_name'] = img_path\n        record['image_id'] = img_id\n        record['height'] = int(h)\n        record['width'] = int(w)\n    \n    return dict(record)\n\n\n\ndef wheat_parallel(df, folder, is_train):\n    \n    if is_train:\n        if NUMBER_TRAIN_SAMPLE != -1:\n            df = df[:NUMBER_TRAIN_SAMPLE]\n        \n    pool = Pool()\n    img_uniques = list(zip(range(df['image_id'].nunique()), df['image_id'].unique()))\n    func = partial(wheat_dataset, df, folder, is_train)\n    detaset_dict = pool.map(func, img_uniques)\n    pool.close()\n    pool.join()\n    \n    return detaset_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for d in ['train', 'test']:\n    DatasetCatalog.register(f'wheat_{d}', lambda d=d: wheat_parallel(train_df if d=='train' else sub_df, \n                                                                     TRAIN_IMAGE_PATH if d=='train' else TEST_IMAGE_PATH,\n                                                                     True if d=='train' else False))\n    MetadataCatalog.get(f'wheat_{d}').set(thing_classes=list_source)\n    \nmicro_metadata = MetadataCatalog.get('wheat_train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visual_train(dataset, n_sampler=10):\n    for sample in random.sample(dataset, n_sampler):\n        img = cv2.imread(sample['file_name'])\n        v = Visualizer(img[:, :, ::-1], metadata=micro_metadata, scale=0.5)\n        v = v.draw_dataset_dict(sample)\n        plt.figure(figsize = (14, 10))\n        plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))\n        plt.show()\n        \ntrain_dataset = wheat_parallel(train_df, TRAIN_IMAGE_PATH, True)        \nvisual_train(train_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ncfg = get_cfg()\ncfg.merge_from_file(model_zoo.get_config_file(MODEL_PATH))\ncfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml\")\ncfg.MODEL.ROI_HEADS.NUM_CLASSES = len(list_source)\ncfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256\n\ncfg.DATASETS.TRAIN = ('wheat_train',)\ncfg.DATASETS.TEST = ()\ncfg.DATALOADER.NUM_WORKERS = 4\n\ncfg.SOLVER.IMS_PER_BATCH = 2\ncfg.SOLVER.BASE_LS = 0.00025\ncfg.SOLVER.MAX_ITER = 5000\n\nos.makedirs(cfg.OUTPUT_DIR, exist_ok=True)\ntrainer = DefaultTrainer(cfg)\ntrainer.resume_or_load(resume=False)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\n#torch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrainer.train()\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg.DATASETS.TEST = ('wheat_test',)\ncfg.MODEL.ROI_HEADS.NUM_CLASSES = len(list_source)\ncfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, 'model_final.pth')\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5\npredict = DefaultPredictor(cfg)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndef visual_predict(dataset):\n    for sample in dataset:\n        img = cv2.imread(sample['file_name'])\n        output = predict(img)\n        \n        v = Visualizer(img[:, :, ::-1], metadata=micro_metadata, scale=0.5)\n        v = v.draw_instance_predictions(output['instances'].to('cpu'))\n        plt.figure(figsize = (14, 10))\n        plt.imshow(cv2.cvtColor(v.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB))\n        plt.show()\n\ntest_dataset = wheat_parallel(sub_df, TEST_IMAGE_PATH, False)\nvisual_predict(test_dataset)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def submit():\n    for idx, row in tqdm(sub_df.iterrows(), total=len(sub_df)):\n        img_path = os.path.join(TEST_IMAGE_PATH, row.image_id+'.jpg')\n        img = cv2.imread(img_path)\n        outputs = predict(img)['instances']\n        boxes = [i.cpu().detach().numpy() for i in outputs.pred_boxes]\n        scores = outputs.scores.cpu().detach().numpy()\n        list_str = []\n        for box, score in zip(boxes, scores):\n            box[3] -= box[1]\n            box[2] -= box[0]\n            box = list(map(int, box))\n            score = round(score, 4)\n            list_str.append(score) \n            list_str.extend(box)\n        sub_df.loc[idx, 'PredictionString'] = ' '.join(map(str, list_str))\n    \n    return sub_df\n\nsub_df = submit()    \nsub_df.to_csv('submission.csv', index=False)\nsub_df","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}