{"cells":[{"metadata":{"id":"aqv38Iz_ZJY6"},"cell_type":"markdown","source":"# README\n\n1. Install pytorch and detectron.\n2. After detectron2 installed, the runtime needs to be restarted. Thus you can see a code exit(0) to restart the runtime and an error message: \"你的工作階段因不明原因而異常終止。\" shown. This is a correct result. There is no problem to go on executing the program.\n3. The program will download the dataset, including `train_images.zip`, `pascal_train.json`, `test_images.zip` and `test.json`. And `train_images.zip` and `test_images.zip` will be automatically unzipped into `./train_images` and `./test_images`.\n4. You can see the length of train dataset is 1349 and the list of 20 categories by checking coco.cats.\n5. Two functions `datasetCateId_2_modelCateId` and `modelCateId_2_datasetCateId` are used to transfer between dataset category id and model category id. It is because dataset category id is 1 ~ 20 and model category id should be 0 ~ 19.\n6. In detectron2, the dataset should be registered, such as:\n<pre><code>DatasetCatalog.register('tiny_voc_train', lambda: get_tiny_voc_dicts('./train_images', './pascal_train.json', 'train'))\nMetadataCatalog.get('tiny_voc_train').thing_classes = tiny_voc_classes('./train_images', './pascal_train.json') </code></pre>\n7. In `get_tiny_voc_dicts`, we split the 1349 samples in train dataset into train dataset (1079 samples) and validate dataset (270 samples) and specify by the third parameter 'train' or 'valid'.\n8. In Inference & evaluation using the trained model section. I have uploaded 7 models I trained in my top mAP list: mAP 64.238% in epoch 48999, mAP 64.129% in epoch 41999, mAP 63.109% in epoch 33999, mAP 62.736% in epoch 61999, mAP 61.604% in epoch 56999, mAP 61.367% in epoch 52999 and mAP 60.899% in epoch 63999.\n9. In the colab, I use mAP 64.238% in epoch 48999, `model_0048999.pth`, as a demo model.\n10. In Prepare submission file section, each instance is recorded in the list of `image_id`, `score`, `category_id` and `segmentation`. The instance mask map is converted by the function `binary_mask_to_rle` and is put into the variable `segmentation`.\n11. The final submission file can be obtained from the file of `./output/submission.json`.\n12. In Train section, we will resume the train work from epoch 63999.\n13. In train phase, data augmentation of `RandomFlip`, `RandomBrightness`, `RandomContrast`, `RandomLighting` and `RandomRotation` are used.\n14. The 1349 samples of train dataset are used to validate the mAP and submission jason format as well."},{"metadata":{"id":"qR01xE4mvbMP"},"cell_type":"markdown","source":"# Import Packages"},{"metadata":{"trusted":true},"cell_type":"code","source":"complete_submission = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not complete_submission:\n    !/opt/conda/bin/python3.7 -m pip install --upgrade pip","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not complete_submission:\n    # !pip install -q -U torch torchvision -f https://download.pytorch.org/whl/torch_stable.html\n    !pip install torch===1.7.1 torchvision===0.8.2 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not complete_submission:\n    !pip install -q -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not complete_submission:\n    !pip install -q detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/index.html","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not complete_submission:\n    !pip install googledrivedownloader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not complete_submission:\n    !pip install bbox-visualizer","execution_count":null,"outputs":[]},{"metadata":{"id":"09VCEE0vvZsk","trusted":true},"cell_type":"code","source":"# import common utilities\nfrom contextlib import contextmanager\nimport cv2\nimport glob\nfrom itertools import groupby\nimport json\nimport matplotlib.pyplot as plt\nimport random\nimport numpy as np\nimport os\nimport pandas\nimport re\nimport sys\n\n# import coco python api\nfrom pycocotools import mask as maskutil\nfrom pycocotools.coco import COCO\nfrom pycocotools.cocoeval import COCOeval\n\nfrom google_drive_downloader import GoogleDriveDownloader as gdd\n\n# import pytorch utilities\nimport torch\nimport torchvision\nassert torch.__version__.startswith(\"1.7\")\n\n# import bbox visualizer\nimport bbox_visualizer as bbv","execution_count":null,"outputs":[]},{"metadata":{"id":"vM54r6jlKTII"},"cell_type":"markdown","source":"# Install detectron2"},{"metadata":{"id":"u4sCqbs05aiH","outputId":"84948fba-9bb5-4ce9-f143-7f546db9f790","trusted":true},"cell_type":"code","source":"# import detectron2 utilities\nimport detectron2\nfrom detectron2 import model_zoo\nfrom detectron2.config import get_cfg\nfrom detectron2.data import DatasetMapper, build_detection_train_loader, build_detection_test_loader\nfrom detectron2.data import MetadataCatalog, DatasetCatalog\nimport detectron2.data.transforms as T\nfrom detectron2.engine import DefaultPredictor\nfrom detectron2.engine import DefaultTrainer\nfrom detectron2.evaluation import COCOEvaluator, inference_on_dataset\nfrom detectron2.structures import BoxMode\nfrom detectron2.utils.logger import setup_logger\nfrom detectron2.utils.visualizer import ColorMode\nfrom detectron2.utils.visualizer import Visualizer","execution_count":null,"outputs":[]},{"metadata":{"id":"ZyAvNCJMmvFF","outputId":"d3c9d0fe-1135-407c-e7e3-4f223b944121","trusted":true},"cell_type":"code","source":"# Setup detectron2 logger\nsetup_logger()","execution_count":null,"outputs":[]},{"metadata":{"id":"x2UMN1KIWm0P"},"cell_type":"markdown","source":"# Prepare Kaggle Wheat Competition Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"id":"W7vxDiVyOQvi"},"cell_type":"markdown","source":"## Show Dataset Information"},{"metadata":{"id":"jmc5M5c2NUG-","outputId":"eff8e59b-c7f2-4c88-ed1d-575c20e4b728","trusted":true},"cell_type":"code","source":"i = glob.glob('/kaggle/input/global-wheat-detection/train/*.jpg')\nprint('number of train images: {}'.format(len(i)))\nprint('train images: {}'.format(i))\n\ni = glob.glob('/kaggle/input/global-wheat-detection/test/*.jpg')\nprint('number of test images: {}'.format(len(i)))\nprint('test images: {}'.format(i))","execution_count":null,"outputs":[]},{"metadata":{"id":"6MK5fgAqZpvz"},"cell_type":"markdown","source":"## Preprocess Kaggle Wheat Competition Dataset"},{"metadata":{"id":"_hOKi1inPPYP","outputId":"d7a8b669-696b-4fa6-ce5d-89fea87e01cb","trusted":true},"cell_type":"code","source":"df = pandas.read_csv('/kaggle/input/global-wheat-detection/train.csv')\nprint(df)","execution_count":null,"outputs":[]},{"metadata":{"id":"l5ni2m-DRjHR","outputId":"b0843494-b783-467e-9599-3621b9403c49","trusted":true},"cell_type":"code","source":"for i in df:\n    print(i)","execution_count":null,"outputs":[]},{"metadata":{"id":"LdKbLKM36bHu","outputId":"960336f1-741d-421d-cb28-94a1f94adf71","trusted":true},"cell_type":"code","source":"wheat_list = df.to_dict('records')\nprint('number of label records: {}'.format(len(wheat_list)))","execution_count":null,"outputs":[]},{"metadata":{"id":"B5onuojO8Ut6","outputId":"258984ef-a3ad-4c5f-8ec4-912d132426d2","trusted":true},"cell_type":"code","source":"for wheat in wheat_list[0:10]:\n    print(wheat)","execution_count":null,"outputs":[]},{"metadata":{"id":"m6N-E6eE9Zfb","outputId":"fceae7c1-5a71-4c31-dc1a-abb91f1add24","trusted":true},"cell_type":"code","source":"images = [wheat['image_id'] for wheat in wheat_list]\nimages = list(dict.fromkeys(images))\nprint('number of wheat images: {}'.format(len(images)))\n\nimage_name2id = {}\nfor i, img in enumerate(images):\n    # print(i, img)\n    image_name2id[img] = i + 1\nprint('Image Name to ID mapping: {}'.format(image_name2id))\n\nimage_id2name = {}\nfor i, img in enumerate(images):\n    # print(i, img)\n    image_id2name[i+1] = img\nprint('Classes ID to Name mapping: {}'.format(image_id2name))","execution_count":null,"outputs":[]},{"metadata":{"id":"fecqsMzJRMwT","outputId":"5570c501-da00-43be-a522-191a6300fe47","trusted":true},"cell_type":"code","source":"wheat_classes = [wheat['source'] for wheat in wheat_list]\nwheat_classes = list(dict.fromkeys(wheat_classes))\nwheat_classes.sort()\nprint('wheat classes: {}'.format(wheat_classes))\n\nnum_classes = len(wheat_classes)\nprint('num of classes: {}'.format(num_classes))\n\nclass_name2id = {}\nfor i, c in enumerate(wheat_classes):\n    # print(i, c)\n    class_name2id[c] = i + 1\nprint('Classes Name to ID mapping: {}'.format(class_name2id))\n\nclass_id2name = {}\nfor i, c in enumerate(wheat_classes):\n    # print(i, c)\n    class_id2name[i+1] = c\nprint('Classes ID to Name mapping: {}'.format(class_id2name))","execution_count":null,"outputs":[]},{"metadata":{"id":"RnkVdaJe_L3f","outputId":"57813906-8907-4556-e196-a4af0e57675a","trusted":true},"cell_type":"code","source":"wheat_dicts = {}\nfor idx, wheat in enumerate(wheat_list):\n    if not wheat['image_id'] in wheat_dicts:\n        image_id = wheat['image_id']\n        wheat_dicts[image_id] = {}\n        wheat_dicts[image_id]['file_name'] = os.path.join('./train', wheat['image_id']+'.jpg')\n        wheat_dicts[image_id]['image_id'] = image_name2id[wheat['image_id']]\n        wheat_dicts[image_id]['height'] = wheat['height']\n        wheat_dicts[image_id]['width'] = wheat['width']\n        wheat_dicts[image_id]['annotations'] = []\n        ann = {}\n        regex = re.compile(r'\\[(\\d*\\.?\\d*), (\\d*\\.?\\d*), (\\d*\\.?\\d*), (\\d*\\.?\\d*)\\]')\n        match = regex.search(wheat['bbox'])\n        ann['bbox'] = [float(match.group(1)), float(match.group(2)), float(match.group(3)), float(match.group(4))]\n        ann['bbox_mode'] = BoxMode.XYWH_ABS\n        ann['segmentation'] = []\n        ann['category_id'] = class_name2id[wheat['source']]\n        ann['id'] = idx + 1\n        wheat_dicts[image_id]['annotations'].append(ann)\n    else:\n        ann = {}\n        regex = re.compile(r'\\[(\\d*\\.?\\d*), (\\d*\\.?\\d*), (\\d*\\.?\\d*), (\\d*\\.?\\d*)\\]')\n        match = regex.search(wheat['bbox'])\n        ann['bbox'] = [float(match.group(1)), float(match.group(2)), float(match.group(3)), float(match.group(4))]\n        ann['bbox_mode'] = BoxMode.XYWH_ABS\n        ann['segmentation'] = []\n        ann['category_id'] = class_name2id[wheat['source']]\n        ann['id'] = idx + 1\n        wheat_dicts[image_id]['annotations'].append(ann)\n\nprint(len(wheat_dicts))","execution_count":null,"outputs":[]},{"metadata":{"id":"GSBg9h1w1wF4","outputId":"f5c768be-f4f0-46ba-e89b-43911cff8a5a","trusted":true},"cell_type":"code","source":"for wheat in wheat_dicts:\n    print(wheat_dicts[wheat])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_dir = './temp'\nos.makedirs(temp_dir, exist_ok=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"kd1o---Qz8kz","trusted":true},"cell_type":"code","source":"# if your dataset is in COCO format, this cell can be replaced by the following three lines:\n# from detectron2.data.datasets import register_coco_instances\n# register_coco_instances(\"my_dataset_train\", {}, \"json_annotation_train.json\", \"path/to/image/dir\")\n# register_coco_instances(\"my_dataset_val\", {}, \"json_annotation_val.json\", \"path/to/image/dir\")\n\ndef get_wheat_dicts(wheat_dicts, mode='train'):\n\n    dataset_dicts = []\n\n    train_valid_split = 5\n    for idx, wheat in enumerate(wheat_dicts):\n        if mode == 'train' and (idx % train_valid_split) == 0:\n            continue\n        if mode == 'valid' and not (idx % train_valid_split) == 0:\n            continue\n\n        dataset_dicts.append(wheat_dicts[wheat])\n\n    print('mode: {}, number of samples: {}'.format(mode, len(dataset_dicts)))\n    return dataset_dicts","execution_count":null,"outputs":[]},{"metadata":{"id":"QrRv-_Wfreqg","outputId":"301ef0d0-6c61-4a54-dcd0-ae86f74d609b","trusted":true},"cell_type":"code","source":"train_dataset_dicts = get_wheat_dicts(wheat_dicts, 'train')\nprint(train_dataset_dicts[0:10])\n\ntrain_dataset_coco_dicts = {}\ntrain_dataset_coco_dicts['annotations'] = [{'segmentation':[\n                        [ann['bbox'][0], ann['bbox'][1],\n                        ann['bbox'][0]+ann['bbox'][2], ann['bbox'][1],\n                        ann['bbox'][0]+ann['bbox'][2], ann['bbox'][1]+ann['bbox'][3],\n                        ann['bbox'][0], ann['bbox'][1]+ann['bbox'][3]]\n           ],\n           'area':ann['bbox'][2]*ann['bbox'][3],\n           'iscrowd':0,\n           'image_id':img['image_id'],\n           'bbox':ann['bbox'],\n           'category_id':ann['category_id'],\n           'id': ann['id']}\n           for img in train_dataset_dicts\n           for ann in img['annotations']]\ntrain_dataset_coco_dicts['images'] = [{'file_name':img['file_name'], 'id':img['image_id'], 'height':img['height'], 'width':img['width']} for img in train_dataset_dicts]\ntrain_dataset_coco_dicts['categories'] = [{'supercategory':c, 'name':c, 'id':class_name2id[c]} for c in wheat_classes]\n\nwith open(os.path.join(temp_dir, 'wheat_train_coco.json'), 'w') as outfile:\n    json.dump(train_dataset_coco_dicts, outfile)\noutfile.close()","execution_count":null,"outputs":[]},{"metadata":{"id":"Kf7gDTY07FSh","outputId":"4b8f84f1-6f10-4dc4-cab7-746f118be8a5","trusted":true},"cell_type":"code","source":"valid_dataset_dicts = get_wheat_dicts(wheat_dicts, 'valid')\nprint('number of valid samples: {}', len(valid_dataset_dicts))\nprint(valid_dataset_dicts[0:10])\n\nvalid_dataset_coco_dicts = {}\nvalid_dataset_coco_dicts['annotations'] = [{'segmentation':[\n                        [ann['bbox'][0], ann['bbox'][1],\n                        ann['bbox'][0]+ann['bbox'][2], ann['bbox'][1],\n                        ann['bbox'][0]+ann['bbox'][2], ann['bbox'][1]+ann['bbox'][3],\n                        ann['bbox'][0], ann['bbox'][1]+ann['bbox'][3]]\n           ],\n           'area':ann['bbox'][2]*ann['bbox'][3],\n           'iscrowd':0,\n           'image_id':img['image_id'],\n           'bbox':ann['bbox'],\n           'category_id':ann['category_id'],\n           'id': ann['id']}\n           for img in valid_dataset_dicts\n           for ann in img['annotations']]\nvalid_dataset_coco_dicts['images'] = [{'file_name':img['file_name'], 'id':img['image_id'], 'height':img['height'], 'width':img['width']} for img in valid_dataset_dicts]\nvalid_dataset_coco_dicts['categories'] = [{'supercategory':c, 'name':c, 'id':class_name2id[c]} for c in wheat_classes]\n\nwith open(os.path.join(temp_dir, 'wheat_valid_coco.json'), 'w') as outfile:\n    json.dump(valid_dataset_coco_dicts, outfile)\noutfile.close()","execution_count":null,"outputs":[]},{"metadata":{"id":"v4iQytxg-Gwp","outputId":"f2ec3175-04dd-42db-bce5-ea99dbefb54b","trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"id":"hDzuUmzA7f8e","trusted":true},"cell_type":"code","source":"DatasetCatalog.clear()\nfrom detectron2.data.datasets import register_coco_instances\n\nregister_coco_instances('wheat_train', {}, 'wheat_train_coco.json', temp_dir)\nregister_coco_instances('wheat_valid', {}, 'wheat_valid_coco.json', temp_dir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm ./temp/wheat_train_coco.json\n!rm ./temp/wheat_valid_coco.json","execution_count":null,"outputs":[]},{"metadata":{"id":"qTT3cFDhZmFj","trusted":true},"cell_type":"code","source":"cfg = get_cfg()\n# cfg.OUTPUT_DIR = '/content/drive/MyDrive/NCTU/基於深度學習之視覺辨識專論/HW/Final/output1'\ncfg.MODEL.DEVICE = 'cpu'\ncfg.OUTPUT_DIR = temp_dir\ncfg.merge_from_file(model_zoo.get_config_file(\"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\"))\ncfg.DATASETS.TRAIN = ('wheat_train',)\ncfg.DATASETS.TEST = ('wheat_valid',)\ncfg.TEST.EVAL_PERIOD = 2000\ncfg.SOLVER.CHECKPOINT_PERIOD = 1000\ncfg.DATALOADER.NUM_WORKERS = 2\n# cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\")  # Let training initialize from model zoo\ncfg.SOLVER.IMS_PER_BATCH = 2\ncfg.SOLVER.BASE_LR = 0.0000025  # 0.00025: 0~12999; 0.0001: 13000~\ncfg.SOLVER.MAX_ITER = 300000  # 300 iterations seems good enough for this toy dataset; you will need to train longer for a practical dataset\ncfg.SOLVER.GAMMA = 0.5\n# cfg.SOLVER.STEPS = (60000,100000,210000,250000) 0.5 gamma\ncfg.SOLVER.STEPS = (30000,)\n# cfg.SOLVER.LR_SCHEDULER_NAME = WarmupMultiStepLR\ncfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 0\ncfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512\ncfg.MODEL.ROI_HEADS.NUM_CLASSES = num_classes  # only has one class (ballon). (see https://detectron2.readthedocs.io/tutorials/datasets.html#update-the-config-for-new-datasets)","execution_count":null,"outputs":[]},{"metadata":{"id":"idDNBIgWZqY0","outputId":"14a946a2-3391-40c5-a1f1-cd2a6390b56a","trusted":true},"cell_type":"code","source":"print(cfg)","execution_count":null,"outputs":[]},{"metadata":{"id":"0e4vdDIOXyxF"},"cell_type":"markdown","source":"# Inference & evaluation using the trained model\nNow, let's run inference with the trained model on the balloon validation dataset. First, let's create a predictor using the model we just trained:"},{"metadata":{"id":"Ya5nEuMELeq8","trusted":true},"cell_type":"code","source":"# Inference should use the config with parameters that are used in training\n# cfg now already contains everything we've set previously. We changed it a little bit for inference:\n\nif not complete_submission:\n    gdd.download_file_from_google_drive(file_id='12T8tCt1n8uplDLjeMypkbqWNofLu0Ine',\n            dest_path=os.path.join(cfg.OUTPUT_DIR, 'model_0299999.pth'),\n            unzip=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, 'model_0299999.pth')  # path to the model we just trained\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7   # set a custom testing threshold\n\npredictor = DefaultPredictor(cfg)","execution_count":null,"outputs":[]},{"metadata":{"id":"qWq1XHfDWiXO"},"cell_type":"markdown","source":"## Prepare submission file"},{"metadata":{"id":"0CrM3qF6Kf54","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\ntestfiles = [f for f in os.listdir('/kaggle/input/global-wheat-detection/test') if os.path.isfile(os.path.join('/kaggle/input/global-wheat-detection/test', f))]\n# print(testfiles)\n\nfilenames = []\npredicts = []\n\nfor f in testfiles:\n    # print('image filename: {}'.format(f))\n    # print(os.path.join('/kaggle/input/global-wheat-detection/test', f))\n    image = cv2.imread(os.path.join('/kaggle/input/global-wheat-detection/test', f))\n\n    outputs = predictor(image)\n\n    result = {}\n\n    # result['image_id'] = f['id']\n    result['image_id'] = os.path.splitext(f)[0]\n    result['score'] = [[round(i.tolist(),2)] for i in outputs['instances'].scores]\n    result['category_id'] = [cate_id.numpy()+1 for cate_id in outputs[\"instances\"].pred_classes.to('cpu')]\n    result['bbox'] = [[round(box.tolist()[0]),\n                   round(box.tolist()[1]),\n                   round(box.tolist()[2]-box.tolist()[0]),\n                   round(box.tolist()[3]-box.tolist()[1])] for box in outputs[\"instances\"].pred_boxes]\n\n    pred = []\n    n_instances = len(outputs['instances'].scores)\n    # print(n_instances)\n    for i in range(n_instances):  # Loop all instances\n        # save information of the instance in a dictionary then append on coco_dt list\n        pred.extend(result['score'][i])\n        pred.extend(result['bbox'][i])\n\n    filenames.append(result['image_id'])\n    predicts.append(pred)\n    # print(pred)\n\nprint('filename list: {}'.format(filenames))\n# print('predict list: {}'.format(predicts))\npredicts = [' '.join(str(p) for p in pred) for pred in predicts]\nprint('predict list: {}'.format(predicts))\n\nsubmission_dict = {\n    'image_id': filenames,\n    'PredictionString': predicts\n}\nsubmission = pd.DataFrame(submission_dict)\n# submission.to_csv(os.path.join(cfg.OUTPUT_DIR, 'submission.csv'), encoding = 'utf-8',index = False)\n\n# submission.to_json(os.path.join('./', 'submission.json'), index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(os.path.join('./', 'submission.csv'), encoding = 'utf-8',index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}