{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThanks to great competition!! Our team divided our two submissions into two models (1) FRCNN-ResNest and (2) EffDet. The FRCNN-ResNest is able to reach 0.7702 Public LB at the end but not perform well on Private LB. While EffDet safe us to get relatively great Private position :D\n\nThis notebook is on FRCNN-ResNest and shows how to improve  by combining Pseudo labeling techinque with ensemble by the following steps:\n\n0. Train models on K Folds (see training notebook in [Colab here](https://colab.research.google.com/drive/1ckIi9A8npT3tazlixfQiHdhWWhLh1lQI?usp=sharing)\n1. Ensemble all K models for prediction of pseudo labeling\n2. Retrain each K models again with new pseudo labeling, together with each corresponding training fold\n3. Recombine K models prediction again after finish training for all models\n\n## Proper Credits\n- The optimized version of this notebook can reach 0.770+ thanks to my teammates @nitindatta and @datahobbit for dedicated efforts while @yashchoudhary complete the Kfold training. \n- @kyoshioka47 (or arutema47) provided us \"effective\" EffDet responsible for our final position and many insightful discussions which you can see in his own thread / kernel to be published soon :D\n- Original work of FRCNN-ResNest is due to amazing @whurobin (https://github.com/wuxinwang1997/wheatdetection) and @shonenkov who provided solid starter for everyone in this competition (needless to put links)\n\n## What's go wrong in Private LB for this model ?\nBy looking at the boxes in the sample 10 test images (see Version 12 for real pseudo labeling run). I believe FRCNN-ResNest is too good detecting every blur and small wheat near the edges, which is don't count as valid labels according to the original paper -- so it has many false-positive but in fact work fine in my opinion","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nLogs.\n- v6[7638] original Ensemble --> K-Models Pseudo --> final ensemble\n- v7[7636] add SatTTA again , with stricter post-wbf filter\n- v8[QUICK] scale ACCUM before loss.backward() / fix train-val loaders for KFolds\n- v9[QUICK] Disable SatTTA / identify bug when each checkpoints have different last epochs\n- v10[7652] Run on the real Fold2!! fix bug when each checkpoints have different last epochs\n- v12[7662] 4x4\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference Kernel on Test Data\n\n# 1. Prepare ResNest and Best Weights Data\nJung's already make ResNest repo as a dataset. We will move this to Kaggle's working directory so that we can modify (`/kaggle/input/` is read-only)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from object_detection_utils import show_Nimages","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp -rf /kaggle/input/wheatdetection-resnest-develop-branch-july9/wheatdetection/wheatdetection /kaggle/working/\n!ls /kaggle/working","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CODE_PATH = '/kaggle/working/wheatdetection/'\n!ls {CODE_PATH}\n%cd {CODE_PATH}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we define a path to our best checkpoint. Please modify the following cell to your own trained dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/input/5fold-68-clear/\n!ls /kaggle/input/best-models-frcnn/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport random\nimport numpy as np\nimport os\n\ndef seed_everything(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Inference on Test (to get Pseudo Labels)\n\nDefine hyperparameters here. If we don't use NMS, we simply filter out the boxes using `SCORE_THRESHOLD` ; If we use NMS, we will combine both `SCORE_THRESHOLD` and `IOU_THRESHOLD` to filter box according to NMS logic.\n\nIn the pseudo-labeling kernel, various parameters are added and commented.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# For simple implementations and best performance, \n# BEST_PATHS should be sorted from Fold0, Fold1, Fold2, ..., FoldK respectively [use this structure in Kfolds split]\nBEST_PATHS = [\"/kaggle/input/best-models-frcnn/F0_68_nofinetune_clear_best.bin\", \n              \"/kaggle/input/best-models-frcnn/F1_68_nofinetune_clear_best.bin\",\n              \"/kaggle/input/5fold-68-clear/F2_68_nofinetune_clear_best.bin\",\n              \"/kaggle/input/5fold-68-clear/F3_68_nofinetune_clear_best.bin\"]\n\n# print to see best weights information\nfor BEST_PATH in BEST_PATHS:\n    ckp = torch.load(BEST_PATH)\n    print(ckp.keys())\n    print(ckp['epoch'], ckp['best_valid_loss'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED=42\nseed_everything(SEED)\n\nAPEX = False # NOT WORK at the moment\nACCUM = 2 # accumulative gradient epochs, 1=do nothing\n\n# VALID_FOLD = 0 # Not specific on use-case of KFolds ensemble -- use corresponding valid fold for each fold model\nSWAP_VALID_AND_TRAIN = False\nN_VIZ = 10 # How many pictures you want to visualize (0-10)\n\n'''POST-PROCESS PARAMETERS'''\nUSE_NMS = False\nSCORE_THRESHOLD = 0.65\nNMS_IOU_THRESHOLD = 0.5\nIMG_SIZE = 1024\nWBF_IOU, WBF_SKIP_BOX = 0.44, 0.38\nPP_SHRINK = [-1,0] # [ Shrink for pseudo labeling, Shrink for the final prediction ]\n\n\nWBF_SCORE_THRESHOLD = 0.265 # 0=UNUSED=BAD_CV , 0.3=BEST_CV, 0.25=BEST_LB\n\nUSE_BOUNDS_FILTER = True \nLOWER_BOUND, UPPER_BOUND = 70, 175000 # observed bounds from Train data (400, 150000)\n\n'''PSEUDO-LABELING PARAMETERS'''\nPSEUDO_EPOCHS = 4 #\nPSEUDO_EPOCHS_COMMIT = 0 # if you just commit, not submit, it will run this #epochs (you can set to 0 or 1 for fast experiments)\n\nHSV_H = 0.03\nHSV_S = 0.68\nHSV_V = 0.36\nBC_B = 0.1\nBC_C  = 0.1\nBASE_LR = 0.00135\nBIAS_LR_FACTOR = 0.5 # DEFAULT is 1, I don't really know the true effect of this parameter\nMOMENTUM = 0.75\nWARMUP_EPOCHS = 200","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"if APEX:\n    ! pip install --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" /kaggle/input/nvidiaapex/.\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if APEX:\n    from apex import amp","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the repo, WheatDetector Class always need internet connection for `pretrained` weights, so below we rewrite the file for `pretrained=False`","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%writefile ./modeling/wheat_detector.py\n\nimport torch\nfrom torch import nn\nfrom layers import FasterRCNN\nfrom layers.backbone_utils import resnest_fpn_backbone\n\nclass WheatDetector(nn.Module):\n    def __init__(self, cfg, **kwargs):\n        super(WheatDetector, self).__init__()\n        self.backbone = resnest_fpn_backbone(pretrained=False) #change here\n        self.base = FasterRCNN(self.backbone, num_classes=cfg.MODEL.NUM_CLASSES, **kwargs)\n\n    def forward(self, images, targets=None):\n        return self.base(images, targets)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hides two cells of TTA and Tester codes","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport cv2\n\nimport warnings\n\nfrom tqdm import tqdm\nimport pandas as pd\nfrom itertools import product\nimport sys\nsys.path.insert(0, \"./external/wbf\")\n# sys.path.insert(0, \"/kaggle/input/weighted-boxes-fusion-104\")\nimport ensemble_boxes\nfrom torchvision.transforms import functional_tensor as TF\nfrom torchvision import transforms\nwarnings.filterwarnings(\"ignore\")\n\n\nclass BaseWheatTTA:\n    \"\"\" author: @shonenkov \"\"\"\n    image_size = IMG_SIZE\n\n    def augment(self, image):\n        raise NotImplementedError\n\n    def batch_augment(self, images):\n        raise NotImplementedError\n\n    def deaugment_boxes(self, boxes):\n        raise NotImplementedError\n\nclass TTAReduceSaturation(BaseWheatTTA):\n    \"\"\" author: @ratthachat \"\"\"\n\n    def augment(self, image):\n        image = TF.adjust_saturation(image, 0.9)\n        return image\n\n    def batch_augment(self, images):\n        images_new = []\n        for image in images:\n            images_new.append(self.augment(image))\n        return torch.stack(images_new)\n\n    def deaugment_boxes(self, boxes):\n        return boxes\n\nclass TTAHorizontalFlip(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n\n    def augment(self, image):\n        return image.flip(1)\n\n    def batch_augment(self, images):\n        return images.flip(2)\n\n    def deaugment_boxes(self, boxes):\n        boxes[:, [1, 3]] = self.image_size - boxes[:, [3, 1]]\n        return boxes\n\n\nclass TTAVerticalFlip(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n\n    def augment(self, image):\n        return image.flip(2)\n\n    def batch_augment(self, images):\n        return images.flip(3)\n\n    def deaugment_boxes(self, boxes):\n        boxes[:, [0, 2]] = self.image_size - boxes[:, [2, 0]]\n        return boxes\n\n\nclass TTARotate90(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n\n    def augment(self, image):\n        return torch.rot90(image, 1, (1, 2))\n\n    def batch_augment(self, images):\n        return torch.rot90(images, 1, (2, 3))\n\n    def deaugment_boxes(self, boxes):\n        res_boxes = boxes.copy()\n        res_boxes[:, [0, 2]] = self.image_size - boxes[:, [3, 1]]\n        res_boxes[:, [1, 3]] = boxes[:, [0, 2]]\n        return res_boxes\n\n\nclass TTACompose(BaseWheatTTA):\n    \"\"\" author: @shonenkov \"\"\"\n\n    def __init__(self, transforms):\n        self.transforms = transforms\n\n    def augment(self, image):\n        for transform in self.transforms:\n            image = transform.augment(image)\n        return image\n\n    def batch_augment(self, images):\n        for transform in self.transforms:\n            images = transform.batch_augment(images)\n        return images\n\n    def prepare_boxes(self, boxes):\n        result_boxes = boxes.copy()\n        result_boxes[:, 0] = np.min(boxes[:, [0, 2]], axis=1)\n        result_boxes[:, 2] = np.max(boxes[:, [0, 2]], axis=1)\n        result_boxes[:, 1] = np.min(boxes[:, [1, 3]], axis=1)\n        result_boxes[:, 3] = np.max(boxes[:, [1, 3]], axis=1)\n        return result_boxes\n\n    def deaugment_boxes(self, boxes):\n        for transform in self.transforms[::-1]:\n            boxes = transform.deaugment_boxes(boxes)\n        return self.prepare_boxes(boxes)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Tester is modified for multi-models\nclass Tester:\n    def __init__(self, models, device, cfg, test_loader, n_viz=N_VIZ):\n        self.config = cfg\n        self.test_loader = test_loader\n\n        self.base_dir = f'{self.config.OUTPUT_DIR}'\n        if not os.path.exists(self.base_dir):\n            os.makedirs(self.base_dir)\n\n        self.log_path = f'{self.base_dir}/log.txt'\n        self.score_threshold = SCORE_THRESHOLD\n        self.iou_threshold = NMS_IOU_THRESHOLD\n        self.use_nms = USE_NMS\n        self.n_viz = n_viz\n        \n        self.device = device\n        self.models = []\n        for i,model in enumerate(models):\n            self.models.append(model)\n            self.models[-1].eval()\n            self.models[-1].to(self.device)\n\n        self.log(f'Tester prepared. Device is {self.device}')\n\n    def test(self, pp_shrink=0):\n        results,all_predictions = self.infer(pp_shrink)\n        self.save_predictions(results)\n        return all_predictions\n    def process_det(self, index, outputs):\n        boxes = outputs[index]['boxes'].data.cpu().numpy()\n        scores = outputs[index]['scores'].data.cpu().numpy()\n        boxes = (boxes).clip(min=0, max=1023).astype(int)\n        indexes = np.where(scores > self.score_threshold)\n        boxes = boxes[indexes]\n        scores = scores[indexes]\n        return boxes, scores\n\n    def make_tta_predictions(self, tta_transforms, images, model_id):\n        with torch.no_grad():\n            images = torch.stack(images).float().cuda()\n            predictions = []\n            for tta_transform in tta_transforms:\n                result = []\n                outputs = self.models[model_id](tta_transform.batch_augment(images.clone()))\n                \n                \n                for i, image in enumerate(images):\n                    boxes = outputs[i]['boxes'].data.cpu().numpy()\n                    scores = outputs[i]['scores'].data.cpu().numpy()\n                    indexes = np.where(scores > self.score_threshold)[0]\n                    boxes = tta_transform.deaugment_boxes(boxes.copy())\n                    \n                    if self.use_nms: \n                        labels = np.ones(scores.shape[0]).astype(int).tolist()\n                        boxes, scores, labels = ensemble_boxes.ensemble_boxes_nms.nms_method([boxes], [scores], [labels], method=3,\n                                                                                        weights=None, iou_thr=self.iou_threshold,\n                                                                                        thresh=self.score_threshold)\n                    else: # not use NMS, just filter by confidence score\n                        boxes = boxes[indexes]\n                        scores = scores[indexes]\n                    result.append({\n                        'boxes': boxes,\n                        'scores': scores,\n                    })\n                predictions.append(result)\n        return predictions # [TTA_NUM , BATCH_IMG_NUM, DICT[BOXES, SCORES]]\n    \n    '''\n    run_wbf\n    - predictions is in [TTA_NUM , BATCH_IMG_NUM, DICT[BOXES, SCORES]] format\n    - BOXES are xyxy format \n    - BOXES are unnormalized -- and will get WRONG result for non-square non-1024 images!! (hidden BUG FOUND)\n    '''\n    def run_wbf(self, predictions, image_index, image_shape=(IMG_SIZE,IMG_SIZE), iou_thr=WBF_IOU, skip_box_thr=WBF_SKIP_BOX, weights=None):\n#         boxes = [(prediction[image_index]['boxes'] / (image_size - 1)).tolist() for prediction in predictions] # for each TTA\n        \n        new_boxes_multi_models = []\n        for prediction in predictions:\n            pred_boxes = prediction[image_index]['boxes'] # each (TTA) model, specific image_id --> np-shape[NUM_BOXES, 4] \n            pred_boxes[:, 0] = pred_boxes[:, 0]/(image_shape[0]-1)\n            pred_boxes[:, 1] = pred_boxes[:, 1]/(image_shape[1]-1)\n            pred_boxes[:, 2] = pred_boxes[:, 2]/(image_shape[0]-1)\n            pred_boxes[:, 3] = pred_boxes[:, 3]/(image_shape[1]-1)\n            \n            new_boxes_multi_models.append(pred_boxes.tolist())\n        \n        boxes = new_boxes_multi_models\n        scores = [prediction[image_index]['scores'].tolist() for prediction in predictions]\n        labels = [np.ones(prediction[image_index]['scores'].shape[0]).astype(int).tolist() for prediction in\n                  predictions]\n        boxes, scores, labels = ensemble_boxes.ensemble_boxes_wbf.weighted_boxes_fusion(boxes, scores, labels,\n                                                                                        weights=None, iou_thr=iou_thr,\n                                                                                        skip_box_thr=skip_box_thr)\n#         boxes = boxes * (image_size - 1)\n        boxes[:, 0] = boxes[:, 0]*(image_shape[0]-1)\n        boxes[:, 1] = boxes[:, 1]*(image_shape[1]-1)\n        boxes[:, 2] = boxes[:, 2]*(image_shape[0]-1)\n        boxes[:, 3] = boxes[:, 3]*(image_shape[1]-1)\n        return boxes, scores, labels\n\n    def format_prediction_string(self, boxes, scores):\n        pred_strings = []\n        for j in zip(scores, boxes):\n            pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], j[1][2], j[1][3]))\n        return \" \".join(pred_strings)\n\n    def infer(self, pp_shrink=0):\n        for i in range(len(self.models)):\n            self.models[i].eval()\n        torch.cuda.empty_cache()\n\n        tta_transforms = []\n        for tta_combination in product([TTAHorizontalFlip(), None],\n                                       [TTAVerticalFlip(), None],\n#                                        [TTAReduceSaturation(), None],\n                                       [TTARotate90(), None]):\n            tta_transforms.append(TTACompose([tta_transform for tta_transform in tta_combination if tta_transform]))\n        test_loader = tqdm(self.test_loader, total=len(self.test_loader), desc=\"Testing\")\n        results = []\n        all_predictions = []\n        boxes10 = []\n        \n        for images, image_ids in test_loader:\n            predictions=[]\n            for ii in range(len(self.models)):\n                predictions.append(self.make_tta_predictions(tta_transforms, images,model_id=ii)) # [TTA_NUM , BATCH_IMG_NUM, DICT[BOXES, SCORES]]\n            \n            predictions = np.vstack(predictions)\n            \n            for i, image in enumerate(images):\n#                 print('shape : ',image.shape)\n                boxes, scores, labels = self.run_wbf(predictions, image_index=i,image_shape=[image.shape[1],image.shape[2]])\n\n                #round and clip seems to be better before rather than after pp_shrink\n                boxes = boxes.round().astype(np.int32).clip(min=0, max=1023)\n                \n                image_id = image_ids[i]\n                \n                if len(boxes10) < self.n_viz:\n                    print('writing ... ',i,image_id)\n                    sample = image.permute(1,2,0).cpu().numpy()\n\n                    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n                    boxes10.append((sample, boxes))\n                    \n                    for box, score in zip(boxes,scores):\n                        cv2.rectangle(sample, (box[0], box[1]), (box[2], box[3]), (1, 0, 0), 5)\n                        cv2.putText(sample, '%.2f'%(score), (box[0], box[1]), cv2.FONT_HERSHEY_SIMPLEX ,  \n                   1, (255,255,255), 3, cv2.LINE_AA)\n                    \n                    ax.set_axis_off()\n                    ax.imshow(sample);\n                    plt.show()\n                                \n                #post-processing box size adjusting (host advised boxes in test are tight to image)\n                boxes[:, 0] = boxes[:, 0] + pp_shrink\n                boxes[:, 1] = boxes[:, 1] + pp_shrink\n                boxes[:, 2] = boxes[:, 2] - pp_shrink\n                boxes[:, 3] = boxes[:, 3] - pp_shrink\n                \n                img_shape = image.cpu().numpy().shape\n                boxes[:, 0] = [max(min(x, img_shape[1]-1), 0) for x in boxes[:, 0]]\n                boxes[:, 1] = [max(min(x, img_shape[2]-1), 0) for x in boxes[:, 1]]\n                boxes[:, 2] = [max(min(x, img_shape[1]-1), 0) for x in boxes[:, 2]]\n                boxes[:, 3] = [max(min(x, img_shape[2]-1), 0) for x in boxes[:, 3]]\n                \n                boxes[:, 2] = boxes[:, 2] - boxes[:, 0]\n                boxes[:, 3] = boxes[:, 3] - boxes[:, 1]\n                \n                areas = boxes[:, 2]*boxes[:, 3]\n                \n                if USE_BOUNDS_FILTER==True:\n                    #if boxes is filtered do we need to filter scores as well? Otherwise the score will not be aligned with the correct box?\n                    #boxes = [boxes[i] for i in range(len(boxes)) if areas[i] > LOWER_BOUND and areas[i] < UPPER_BOUND]\n                    print(\"Filter by area bounds\")\n                    print(\"Length of unfiltered boxes: \" + str(len(boxes)))\n                    print(\"Length of unfiltered scores: \" + str(len(scores)))\n                    filtered_boxes = []\n                    filtered_scores = []\n                    for i in range(len(boxes)):\n                        if areas[i] > LOWER_BOUND and areas[i] < UPPER_BOUND and scores[i] >= WBF_SCORE_THRESHOLD:\n                            filtered_boxes.append(boxes[i])\n                            filtered_scores.append(scores[i])\n                    print(\"Length of filtered boxes: \" + str(len(filtered_boxes)))\n                    print(\"Length of filtered scores: \" + str(len(filtered_scores)))\n                    boxes = filtered_boxes\n                    scores = filtered_scores\n                else:\n                    print(\"area filtering not applied\")\n                        \n                \n                result = {\n                    'image_id': image_id,\n                    'PredictionString': self.format_prediction_string(boxes, scores),\n                }\n                \n                all_prediction = {\n                    'image_id': image_id,\n                    'pred_boxes': boxes,\n                    'scores': scores,\n                    'img_shape' : img_shape\n                }\n                \n                results.append(result)\n                all_predictions.append(all_prediction)\n        return results, all_predictions\n\n    def format_prediction_string(self, boxes, scores):\n        pred_strings = []\n        for j in zip(scores, boxes):\n            pred_strings.append(\"{0:.4f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], j[1][2], j[1][3]))\n\n        return \" \".join(pred_strings)\n\n    def save_predictions(self, results):\n        test_df = pd.DataFrame(results, columns=['image_id', 'PredictionString'])\n        test_df.to_csv(f'{self.config.OUTPUT_DIR}/submission.csv', index=False)\n\n    def load(self, paths):\n        print(paths)\n        for i in range(len(paths)):\n            checkpoint = torch.load(paths[i])\n            self.models[i].load_state_dict(checkpoint['model_state_dict'])\n            print('finish loading ',paths[i])\n    def log(self, message):\n        if self.config.VERBOSE:\n            print(message)\n        with open(self.log_path, 'a+') as logger:\n            logger.write(f'{message}\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from config import cfg\ncfg['OUTPUT_DIR'] = \"/kaggle/working/\"\ncfg['DATASETS']['ROOT_DIR'] = \"/kaggle/input/global-wheat-detection\"\ncfg['TEST']['IMS_PER_BATCH'] = 1\ncfg['TEST']['WEIGHT'] = BEST_PATHS\n# cfg.DATASETS.VALID_FOLD = VALID_FOLD\ncfg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport sys\n\nfrom os import mkdir\nsys.path.append('.')\nfrom data import make_test_data_loader\nfrom modeling import build_model\nfrom utils.logger import setup_logger\n\n# start here!!\nif True:\n#     cfg.freeze()\n\n    output_dir = cfg.OUTPUT_DIR\n    if output_dir and not os.path.exists(output_dir):\n        print('creating ',cfg.OUTPUT_DIR)\n        mkdir(output_dir)\n    \n    models=[]\n    for i, best in enumerate(BEST_PATHS):\n        models.append(build_model(cfg)) # build_model uses cfg only on cfg.num_classes\n\n    test_loader = make_test_data_loader(cfg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Pseudo labeling and Training\n\n## 3.1 Create the new marking for Train+Test Pseudo Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = cfg.MODEL.DEVICE\ntester = Tester(models=models, device=device, cfg=cfg, test_loader=test_loader, n_viz=N_VIZ)\ntester.load(cfg['TEST']['WEIGHT'])\n    \nall_preds = tester.test(pp_shrink=PP_SHRINK[0])\n!ls /kaggle/working\n!rm -f /kaggle/working/submission.csv\n!ls -lh /kaggle/working","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### create new directory and test dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nNEW_INPUT_PATH = '/kaggle/working/imgs/'\n!mkdir {NEW_INPUT_PATH}\n!mkdir {NEW_INPUT_PATH}train\n!cp -rf /kaggle/input/global-wheat-detection/train/* {NEW_INPUT_PATH}train\n!cp -rf /kaggle/input/global-wheat-detection/test/* {NEW_INPUT_PATH}train\n\n!ls /kaggle/input/global-wheat-detection/train/ | wc\n!ls {NEW_INPUT_PATH}train | wc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_dict = {}\ndf_dict['x'],df_dict['y'],df_dict['w'],df_dict['h'],df_dict['image_id'] = [],[],[],[],[]\nfor i in range(len(all_preds)):\n    if len(all_preds[i]['pred_boxes']) == 0: # handle empty-box case\n        print('pass')\n        continue\n    \n#     print(all_preds[i]['img_shape'])\n    \n    if all_preds[i]['img_shape'][1] != 1024 or all_preds[i]['img_shape'][2] != 1024: # handle non-1024 cases\n        print('pass')\n        continue\n    \n    for box in all_preds[i]['pred_boxes']:\n        df_dict['image_id'].append(all_preds[i]['image_id'])\n        df_dict['x'].append(box[0])\n        df_dict['y'].append(box[1])\n        df_dict['w'].append(box[2])\n        df_dict['h'].append(box[3])\n        \ndf = pd.DataFrame(df_dict)\ndf['source'] = 'test'\ndf['width'] = 1024\ndf['height'] = 1024\ndf['area'] = df['w']*df['h']\n\nprint(df.shape)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from data.build import split_dataset\n\nmarking_list, train_ids_list, valid_ids_list = [], [], []\n\nfor ii in range(len(BEST_PATHS)):\n    print('\\n** weights -- #%d' % ii)\n    cfg.DATASETS.VALID_FOLD = ii # THIS is WHY we recommend SORTED-by-fold weights in BEST_PATHS\n    marking, train_ids0, valid_ids0 = split_dataset(cfg)\n\n    if SWAP_VALID_AND_TRAIN:\n        print('swap!!')\n        valid_ids, train_ids =train_ids0, valid_ids0\n    else:\n        train_ids, valid_ids =train_ids0, valid_ids0\n\n    marking.head(3)\n    print(ii, marking.shape, df.shape)\n    marking = pd.concat([marking,df])\n    marking = marking.reset_index()\n    marking = marking.drop(['index'], axis=1)\n    print(ii, marking.shape, df.shape)\n    marking.tail(3)\n\n    print('#train before adding test',len(train_ids))\n    train_ids = np.concatenate([train_ids, df.image_id.unique()])\n    print('#train after  adding test',len(train_ids))\n    marking_list.append(marking)\n    train_ids_list.append(train_ids)\n    valid_ids_list.append(valid_ids)\n\n# marking will always the same, so marking_list is in fact irrelevant!\n# the main important is train/val_ids_list\nprint(len(marking_list), len(train_ids_list), len(valid_ids_list))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for ii in range(len(BEST_PATHS)):\n    print(valid_ids_list[ii][:5], valid_ids_list[ii].shape)\nprint(675*5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n# NOTE: to my understanding, this train.csv only use by split_dataset , \nbut on cells above we already splitted it using the original train.csv to make K folds.\nTherefore, in fact, this train.csv is not used.\nI confirm this fact by searching for train.csv in original repo:\nhttps://github.com/wuxinwang1997/wheatdetection/search?q=train.csv&unscoped_q=train.csv\n'''\n\n# marking.to_csv(NEW_INPUT_PATH+'train.csv',index=False)\n# pd.read_csv(NEW_INPUT_PATH+'train.csv').shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Training with Pseudo Labels on Test\nIn the first hidden cell, I have to hack several functions :\n\n- re-define data loader for new marking / new ids\n- disable self.best_score_threshold , self.best_final_score\n- disable colab dependency\n- print out when best weights are updated\n\nWe also make the next two cells (DataLoader and Fitter) hidden.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%writefile ./data/transforms/build.py\n\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom .transforms import RandomErasing\n\ndef get_train_transforms(cfg):\n    return A.Compose(\n        [\n            A.Resize(1024, 1024, p=1.0),\n            A.RandomSizedCrop(min_max_height=cfg.INPUT.RSC_MIN_MAX_HEIGHT, height=cfg.INPUT.RSC_HEIGHT,\n                              width=cfg.INPUT.RSC_WIDTH, p=cfg.INPUT.RSC_PROB),\n            A.OneOf([\n                A.HueSaturationValue(hue_shift_limit=cfg.INPUT.HSV_H, sat_shift_limit=cfg.INPUT.HSV_S,\n                                     val_shift_limit=cfg.INPUT.HSV_V, p=cfg.INPUT.HSV_PROB),\n                A.RandomBrightnessContrast(brightness_limit=cfg.INPUT.BC_B,\n                                           contrast_limit=cfg.INPUT.BC_C, p=cfg.INPUT.BC_PROB),\n            ],p=cfg.INPUT.COLOR_PROB),\n            A.ToGray(p=cfg.INPUT.TOFGRAY_PROB),\n            A.HorizontalFlip(p=cfg.INPUT.HFLIP_PROB),\n            A.VerticalFlip(p=cfg.INPUT.VFLIP_PROB),\n            # A.Resize(height=512, width=512, p=1),\n            A.Cutout(num_holes=cfg.INPUT.COTOUT_NUM_HOLES, max_h_size=cfg.INPUT.COTOUT_MAX_H_SIZE,\n                     max_w_size=cfg.INPUT.COTOUT_MAX_W_SIZE, fill_value=cfg.INPUT.COTOUT_FILL_VALUE, p=cfg.INPUT.COTOUT_PROB),\n            ToTensorV2(p=1.0),\n        ],\n        p=1.0,\n        bbox_params=A.BboxParams(\n            format='pascal_voc',\n            min_area=0,\n            min_visibility=0,\n            label_fields=['labels']\n        )\n    )\n\ndef get_valid_transforms(cfg):\n    return A.Compose(\n        [\n            # A.Resize(height=512, width=512, p=1.0),\n            ToTensorV2(p=1.0),\n        ],\n        p=1.0,\n        bbox_params=A.BboxParams(\n            format='pascal_voc',\n            min_area=0,\n            min_visibility=0,\n            label_fields=['labels']\n        )\n    )\n\ndef get_test_transform():\n    return A.Compose([\n        # A.Resize(512, 512),\n        ToTensorV2(p=1.0)\n    ])\n\ndef build_transforms(cfg, is_train=True):\n    if is_train:\n        transform = get_train_transforms(cfg)\n    else:\n        transform = get_valid_transforms(cfg)\n\n    return transform\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"!ls -lh ./data/transforms/\n!cat ./data/transforms/build.py","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nimport time\nimport warnings\nfrom datetime import datetime\nfrom engine.average import AverageMeter\nfrom evaluate.inference import inference\nfrom evaluate.evaluate import evaluate\nimport data\nfrom data.transforms import build_transforms\nfrom solver.build import make_optimizer\nfrom solver.lr_scheduler import make_scheduler\nimport logging\nwarnings.filterwarnings(\"ignore\")\nfrom data.collate_batch import  collate_batch\nfrom data.datasets.train_wheat import train_wheat\n\ndef build_dataset(cfg, marking,train_ids, valid_ids):\n#     marking, train_ids, valid_ids = split_dataset(cfg)\n    train_dataset = train_wheat(\n        root = cfg.DATASETS.ROOT_DIR,\n        image_ids=train_ids,\n        marking=marking,\n        transforms=build_transforms(cfg, is_train=True),\n        test=False,\n    )\n\n    validation_dataset = train_wheat(\n        root=cfg.DATASETS.ROOT_DIR,\n        image_ids=valid_ids,\n        marking=marking,\n        transforms=build_transforms(cfg, is_train=False),\n        test=True,\n    )\n\n    return train_dataset, validation_dataset\n\ndef make_data_loader(cfg, marking,train_ids, valid_ids, is_train=True):\n    if is_train:\n        batch_size = cfg.SOLVER.IMS_PER_BATCH\n    else:\n        batch_size = cfg.TEST.IMS_PER_BATCH\n\n    train_dataset, validation_dataset = build_dataset(cfg, marking,train_ids, valid_ids)\n\n    num_workers = cfg.DATALOADER.NUM_WORKERS\n    train_loader = torch.utils.data.DataLoader(\n        train_dataset,\n        batch_size=batch_size,\n        sampler=torch.utils.data.sampler.RandomSampler(train_dataset),\n        pin_memory=False,\n        drop_last=True,\n        num_workers=num_workers,\n        collate_fn=collate_batch,\n    )\n    val_loader = torch.utils.data.DataLoader(\n        validation_dataset,\n        batch_size=batch_size,\n        num_workers=num_workers,\n        shuffle=False,\n        sampler=torch.utils.data.sampler.SequentialSampler(validation_dataset),\n        pin_memory=False,\n        collate_fn=collate_batch,\n    )\n\n    return train_loader, val_loader\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class Fitter:\n    def __init__(self, model, device, cfg, train_loader, val_loader, logger, mixed_precision=APEX, accum=ACCUM):\n        self.config = cfg\n        self.epoch = 0\n        self.train_loader = train_loader\n        self.val_loader = val_loader\n\n        self.base_dir = f'{self.config.OUTPUT_DIR}'\n        if not os.path.exists(self.base_dir):\n            os.makedirs(self.base_dir)\n\n        self.logger = logger\n        self.best_final_score = 0.0\n        self.best_score_threshold = SCORE_THRESHOLD\n        \n        self.mixed_precision = mixed_precision\n        self.accumulate = accum\n        \n        self.model = model\n        self.device = device\n        self.model.to(self.device)\n        \n        self.optimizer = make_optimizer(cfg, model)\n        \n        if self.mixed_precision:\n            self.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level=\"O1\", verbosity=0)\n        \n        self.scheduler = make_scheduler(cfg, self.optimizer, train_loader)\n\n        self.logger.info(f'Fitter prepared. Device is {self.device}')\n        self.all_predictions = []\n        self.early_stop_epochs = 0\n        self.early_stop_patience = self.config.SOLVER.EARLY_STOP_PATIENCE\n        self.do_scheduler = True\n        self.logger.info(\"Start training\")\n\n    def fit(self):\n        for epoch in range(self.epoch, self.config.SOLVER.MAX_EPOCHS ):\n            if epoch < self.config.SOLVER.WARMUP_EPOCHS:\n                lr_scale = min(1., float(epoch + 1) / float(self.config.SOLVER.WARMUP_EPOCHS))\n                for pg in self.optimizer.param_groups:\n                    pg['lr'] = lr_scale * self.config.SOLVER.BASE_LR\n                self.do_scheduler = False\n            else:\n                self.do_scheduler = True\n            if self.config.VERBOSE:\n                lr = self.optimizer.param_groups[0]['lr']\n                timestamp = datetime.utcnow().isoformat()\n                self.logger.info(f'\\n{timestamp}\\nLR: {lr}')\n\n            t = time.time()\n            summary_loss = self.train_one_epoch()\n\n            self.logger.info(f'[RESULT]: Train. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n            self.save(f'{self.base_dir}/last-checkpoint.bin')\n\n            t = time.time()\n            best_score_threshold, best_final_score = self.validation()\n\n            self.logger.info( f'[RESULT]: Val. Epoch: {self.epoch}, Best Score Threshold: {best_score_threshold:.2f}, Best Score: {best_final_score:.5f}, time: {(time.time() - t):.5f}')\n            if best_final_score > self.best_final_score:\n                self.logger.info('** UPDATE best weights **!')\n                self.best_final_score = best_final_score\n                self.best_score_threshold = best_score_threshold\n                self.model.eval()\n                self.save(f'{self.base_dir}/best-checkpoint.bin')\n                self.save_model(f'{self.base_dir}/best-model.bin')\n                self.save_predictions(f'{self.base_dir}/all_predictions.csv')\n\n            self.early_stop(best_final_score)\n            if self.early_stop_epochs > self.config.SOLVER.EARLY_STOP_PATIENCE:\n                self.logger.info('Early Stopping!')\n                break\n\n            if self.epoch % self.config.SOLVER.CLEAR_OUTPUT == 0:\n                pass # CHANGE ONLY ONE LINE\n#                 output.clear()\n\n            self.epoch += 1\n\n    def validation(self):\n        self.model.eval()\n        t = time.time()\n        self.all_predictions = []\n        torch.cuda.empty_cache()\n        valid_loader = tqdm(self.val_loader, total=len(self.val_loader), desc=\"Validating\")\n        with torch.no_grad():\n            for step, (images, targets, image_ids) in enumerate(valid_loader):\n                images = list(image.cuda() for image in images)\n                outputs = self.model(images)\n                inference(self.all_predictions, images, outputs, targets, image_ids)\n                valid_loader.set_description(f'Validate Step {step}/{len(self.val_loader)}, ' + \\\n                                             f'time: {(time.time() - t):.5f}')\n        best_score_threshold, best_final_score = evaluate(self.all_predictions)\n\n        return best_score_threshold, best_final_score\n\n    def train_one_epoch(self):\n        self.model.train()\n        summary_loss = AverageMeter()\n        loss_box_reg = AverageMeter()\n        loss_classifier = AverageMeter()\n        loss_objectness = AverageMeter()\n        loss_rpn_box_reg = AverageMeter()\n        t = time.time()\n        train_loader = tqdm(self.train_loader, total=len(self.train_loader), desc=\"Training\")\n        for step, (images, targets, image_ids) in enumerate(train_loader):\n            images = torch.stack(images)\n            \n            if self.mixed_precision == False:\n                images = images.to(self.device).float()\n            else:\n                images = images.to(self.device).half()\n            \n            batch_size = images.shape[0]\n            targets = [{k: v.to(self.device) for k, v in t.items()} for t in targets]\n            for i in range(len(targets)):\n                if self.mixed_precision == False:\n                    targets[i]['boxes'] = targets[i]['boxes'].float()\n                else:\n                    targets[i]['boxes'] = targets[i]['boxes'].half()\n                \n            loss_dict = self.model(images, targets)\n            loss = sum(loss for loss in loss_dict.values())\n            box_reg = loss_dict['loss_box_reg']\n            classifier = loss_dict['loss_classifier']\n            objectness = loss_dict['loss_objectness']\n            rpn_box_reg = loss_dict['loss_rpn_box_reg']\n\n            if self.mixed_precision:\n                with amp.scale_loss(loss, self.optimizer) as scaled_loss:\n                    scaled_loss = scaled_loss/ self.accumulate\n                    scaled_loss.backward()\n            else:\n                loss = loss/ self.accumulate\n                loss.backward()\n\n            summary_loss.update(loss.item(), batch_size)\n            loss_box_reg.update(box_reg.item(), batch_size)\n            loss_classifier.update(classifier.item(), batch_size)\n            loss_objectness.update(objectness.item(), batch_size)\n            loss_rpn_box_reg.update(rpn_box_reg.item(), batch_size)\n            \n            if step % self.accumulate == 0:\n                self.optimizer.step()\n                self.optimizer.zero_grad()\n            \n            if self.do_scheduler:\n                self.scheduler.step()\n            train_loader.set_description(f'Train Step {step}/{len(self.train_loader)}, ' + \\\n                                         f'Learning rate {self.optimizer.param_groups[0][\"lr\"]}, ' + \\\n                                         f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n                                         f'loss_box_reg: {loss_box_reg.avg:.5f}, ' + \\\n                                         f'loss_classifier: {loss_classifier.avg:.5f}, ' + \\\n                                         f'loss_objectness: {loss_objectness.avg:.5f}, ' + \\\n                                         f'loss_rpn_box_reg: {loss_rpn_box_reg.avg:.5f}, ' + \\\n                                         f'time: {(time.time() - t):.5f}')\n\n        return summary_loss\n\n    def save(self, path):\n        self.model.eval()\n        torch.save({\n            'model_state_dict': self.model.state_dict(),\n            'optimizer_state_dict': self.optimizer.state_dict(),\n            'scheduler_state_dict': self.scheduler.state_dict(),\n            'best_score_threshold': self.best_score_threshold,\n            'best_final_score': self.best_final_score,\n            'epoch': self.epoch,\n        }, path)\n\n    def save_model(self, path):\n        self.model.eval()\n        torch.save({\n            'model_state_dict': self.model.state_dict(),\n            'best_score_threshold': self.best_score_threshold,\n            'best_final_score': self.best_final_score,\n        }, path)\n\n    def save_predictions(self, path):\n        df = pd.DataFrame(self.all_predictions)\n        df.to_csv(path, index=False)\n\n    def load(self, path):\n        checkpoint = torch.load(path)\n        self.model.load_state_dict(checkpoint['model_state_dict'])\n        self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n        self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n        self.best_score_threshold = SCORE_THRESHOLD\n        self.best_final_score = 0.69 # adhoc\n        self.epoch = checkpoint['epoch'] + 1\n\n    def early_stop(self, score):\n        if score < self.best_final_score:\n            self.early_stop_epochs += 1\n        else:\n            self.early_stop_epochs = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_test = len(os.listdir('/kaggle/input/global-wheat-detection/test/'))\nprint(n_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg.defrost()\n\ncfg['DATASETS']['ROOT_DIR'] = NEW_INPUT_PATH\ncfg.INPUT.HSV_H = HSV_H\ncfg.INPUT.HSV_S = HSV_S\ncfg.INPUT.HSV_V = HSV_V\ncfg.INPUT.BC_B = BC_B\ncfg.INPUT.BC_C = BC_C\ncfg.INPUT.COTOUT_NUM_HOLES=0 \ncfg.SOLVER.BASE_LR = BASE_LR\ncfg.SOLVER.BIAS_LR_FACTOR = BIAS_LR_FACTOR\ncfg.SOLVER.MOMENTUM=MOMENTUM\ncfg.SOLVER.WARMUP_EPOCHS=WARMUP_EPOCHS\n\nOUTPUT_DIRS = []\nfor ii in range(len(models)):\n    OUTPUT_DIRS.append(\"/kaggle/working/weights_%d/\"%ii)\n    if os.path.exists(OUTPUT_DIRS[-1]) == False:\n        print('create ',OUTPUT_DIRS[-1])\n        !mkdir {OUTPUT_DIRS[-1]}\n\n!ls /kaggle/working/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from utils.logger import setup_logger\n\nfitters=[]\nfor ii,path in enumerate(cfg['TEST']['WEIGHT']):\n    \n    cfg['OUTPUT_DIR'] = OUTPUT_DIRS[ii]\n    \n    checkpoint = torch.load(path)\n    cfg.SOLVER.MAX_EPOCHS = checkpoint['epoch']+PSEUDO_EPOCHS+1 # \n    if n_test <11: #standard trick to save kernel committing time\n        cfg.SOLVER.MAX_EPOCHS = checkpoint['epoch']+PSEUDO_EPOCHS_COMMIT+1\n    print('epochs = %d+%d+%d'%(checkpoint['epoch'],PSEUDO_EPOCHS_COMMIT,1))\n    print(cfg)\n    \n    print('*** weight path ***', path)\n    model = build_model(cfg)\n    model.load_state_dict(checkpoint['model_state_dict'])\n    \n    train_loader, val_loader = make_data_loader(cfg,marking_list[ii],train_ids_list[ii], valid_ids_list[ii])\n    logger = setup_logger(\"logger\", cfg['OUTPUT_DIR'], 0)\n    \n    \n    \n    fitter = Fitter(model=model, device=\"cuda\", cfg=cfg, train_loader=train_loader, val_loader=val_loader, logger=logger)\n    fitter.load(path)\n    fitter.fit()\n    \n    !rm -f {OUTPUT_DIRS[ii]+'last-checkpoint.bin'} # remove last checkpoint\n    \n    seed_everything(ii+1) # change random states of each model\n    cfg['SEED'] = ii+1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Now we have a new best weights","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!date\n!ls -lh {cfg['OUTPUT_DIR']} # check the new weights, there will always be last-checkpoint.bin , but may not best-checkpoint.bin","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -lh {cfg['OUTPUT_DIR']+'/last-checkpoint.bin'}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_paths = []\n\nfor ii in range(len(OUTPUT_DIRS)):\n    if os.path.exists(OUTPUT_DIRS[ii]+'best-checkpoint.bin'):\n        best_path = OUTPUT_DIRS[ii]+'best-checkpoint.bin'\n    elif os.path.exists(OUTPUT_DIRS[ii]+'last-checkpoint.bin'):\n        best_path = OUTPUT_DIRS[ii]+'last-checkpoint.bin'\n    else:\n        best_path = BEST_PATHS[ii] # use the non-pseudo-labeling path\n    best_paths.append(best_path)\n    \n    !ls -lh {OUTPUT_DIRS[ii]}\n    print(' ---- \\n')\n    \nprint('best paths are ', best_paths)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 re-predict the test data with new weights","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if True:\n    cfg['OUTPUT_DIR'] = \"/kaggle/working/\"\n    cfg['DATASETS']['ROOT_DIR'] = \"/kaggle/input/global-wheat-detection\"\n    cfg['TEST']['WEIGHT'] = best_paths\n    cfg['TEST']['IMS_PER_BATCH'] = 1\n    print(cfg)\n    test_loader = make_test_data_loader(cfg)\n    \n    # I load checkpoints twice to ensure that correct weights are loaded\n#     checkpoint = torch.load(best_path)\n#     model.load_state_dict(checkpoint['model_state_dict']) \n    tester = Tester(models=models, device=device, cfg=cfg, test_loader=test_loader, n_viz=N_VIZ)\n    tester.load(best_paths)\n    print('success load weights!', best_paths)\n    \n    tester.test(pp_shrink=PP_SHRINK[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check whether submission.csv is re-created\n!date\n!ls -lh {cfg['OUTPUT_DIR']}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Delete the repo when finish","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n%cd ..\n!rm -rf {CODE_PATH}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf {NEW_INPUT_PATH}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/working\n!ls /kaggle/working/wheatdetection\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}