{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":43873,"databundleVersionId":5282756,"sourceType":"competition"},{"sourceId":1154184,"sourceType":"datasetVersion","datasetId":652413},{"sourceId":1164617,"sourceType":"datasetVersion","datasetId":659784},{"sourceId":5264647,"sourceType":"datasetVersion","datasetId":3064176},{"sourceId":5271916,"sourceType":"datasetVersion","datasetId":3067635},{"sourceId":5286884,"sourceType":"datasetVersion","datasetId":3071954}],"dockerImageVersionId":30445,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Train code: [Benetech | EfficientDet [Train]](https://www.kaggle.com/code/alejopaullier/benetech-object-detection-train)\n\n# | Benetech: EfficientDet [Inference] \n\n***\n\n\n### Table of Contents \n
\n
  • Introduction
    • \n
  • Install libraries
    • \n
  • Import Libraries
    • \n
  • Configuration
    • \n
  • Utils
    • \n
  • Load Data
    • \n
  • Data Augmentation
    • \n
  • Dataset
    • \n
  • Data Loader
    • \n
  • Model
    • \n
  • Create Model
    • \n
  • Inference Function
    • \n
  • Inference
    • \n
  • Predictions
    • \n
  • Plot Predictions
    • \n
  • Conclusion
    • \n
    ","metadata":{}},{"cell_type":"markdown","source":"# | Introduction [↑](#top) \n\n***\n\n### Goal \n\n1. You need this notebook to detect the bounding boxes for the x-axis and y-axis ticks and labels. \n2. Knowing these bounding boxes you can later interpolate the data series with cross multiplication. \n3. By knowing the precise location of a chart element you can pass it to an OCR network to detect the numbers and text in the bounding box.\n\nThis notebook follows the [How to tackle this competition](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/396773) discussion. Here, we will address Step 3. We will be detecting bounding boxes in the test images using the trained EfficientDet model.\n\n
    \n\n\n","metadata":{}},{"cell_type":"markdown","source":"# | Install Libraries [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"!pip install pycocotools","metadata":{"scrolled":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-01T14:18:52.881340Z","iopub.execute_input":"2023-04-01T14:18:52.881724Z","iopub.status.idle":"2023-04-01T14:19:25.288209Z","shell.execute_reply.started":"2023-04-01T14:18:52.881683Z","shell.execute_reply":"2023-04-01T14:19:25.286976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Import Libraries [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.insert(0, \"../input/timm-efficientdet-pytorch\")\nsys.path.insert(0, \"../input/omegaconf\")\nsys.path.insert(0, \"../input/albumentations\")\n\nimport albumentations as A\nimport copy\nimport cv2\nimport gc\nimport matplotlib.pyplot as plt\nimport multiprocessing\nimport numpy as np\nimport os\nimport pandas as pd\nimport PIL\nimport random\nimport time\nimport timm\nimport torch\nimport torch.nn as nn\n\n\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom colorama import Fore, Back, Style\nfrom datetime import datetime, timedelta\nfrom glob import glob\nfrom pprint import pprint\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nfrom tqdm import tqdm\n\n\nc_ = Fore.GREEN\nsr_ = Style.RESET_ALL\nprint(f\"There are {multiprocessing.cpu_count()} CPUs available\")\nprint()\n!mkdir logs\n!mkdir saved_models","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-01T14:19:25.291997Z","iopub.execute_input":"2023-04-01T14:19:25.292310Z","iopub.status.idle":"2023-04-01T14:19:31.991987Z","shell.execute_reply.started":"2023-04-01T14:19:25.292280Z","shell.execute_reply":"2023-04-01T14:19:31.990458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Configuration [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"class config:\n BATCH_SIZE_TEST = 4\n DEBUG = False\n DEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n NUM_CLASSES = 8\n NUM_WORKERS = multiprocessing.cpu_count()\n RESOLUTION = 512\n SEED = 42\n \n \nclass paths:\n BEST_MODEL = \"/kaggle/input/benetech-model/best_epoch_model_v2.bin\"\n TEST_IMAGES_FOLDER = \"/kaggle/input/benetech-making-graphs-accessible/test/images/\"","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:21:25.695912Z","iopub.execute_input":"2023-04-01T14:21:25.696612Z","iopub.status.idle":"2023-04-01T14:21:25.703043Z","shell.execute_reply.started":"2023-04-01T14:21:25.696573Z","shell.execute_reply":"2023-04-01T14:21:25.701473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Utils [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"stoi = {\n 'chart_title': 1, # red\n 'axis_title': 2, # pink\n 'tick_label': 3, # cyan\n 'x-axis-tick': 4, # blue\n 'y-axis-tick': 5, # green \n 'other': 6, # orange\n 'tick_grouping': 7, # yellow\n 'legend_label': 8 # black\n}\n\n\nstoi = {\n 'axis_title': 1,\n 'chart_title': 5,\n 'legend_label': 8,\n 'other': 6,\n 'tick_grouping': 7,\n 'tick_label': 2,\n 'x-axis-tick': 3,\n 'y-axis-tick': 4\n}\n\n\ncolor_map = {\n 1: (255, 0, 0), # red\n 2: (255, 0, 255), # pink\n 3: (0, 255, 255), # cyan\n 4: (0, 0, 255), # blue\n 5: (0, 255, 0), # green\n 6: (255, 127, 0), # orange\n 7: (255, 255, 0), # yellow\n 8: (0, 0, 0) # black\n}\n\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n\nseed_everything(config.SEED)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-01T14:21:03.976440Z","iopub.execute_input":"2023-04-01T14:21:03.977184Z","iopub.status.idle":"2023-04-01T14:21:03.986568Z","shell.execute_reply.started":"2023-04-01T14:21:03.977145Z","shell.execute_reply":"2023-04-01T14:21:03.985400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Load Data [↑](#top) \n\n***\n","metadata":{}},{"cell_type":"code","source":"df = pd.DataFrame(glob(paths.TEST_IMAGES_FOLDER+\"*.jpg\"))\ndf.columns = [\"image_path\"]\ndf[\"image_id\"] = df[\"image_path\"].apply(lambda x: x.split(\"/\")[-1].split(\".\")[0])\nprint(f\"Dataframe shape is: {df.shape}\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:21:07.114173Z","iopub.execute_input":"2023-04-01T14:21:07.115166Z","iopub.status.idle":"2023-04-01T14:21:07.131613Z","shell.execute_reply.started":"2023-04-01T14:21:07.115127Z","shell.execute_reply":"2023-04-01T14:21:07.130412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Data Augmentation [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"def get_test_transforms():\n return A.Compose(\n [\n A.Resize(height=config.RESOLUTION, width=config.RESOLUTION, p=1),\n A.Normalize(p=1),\n ToTensorV2(p=1.0),\n ]\n )","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:21:09.141255Z","iopub.execute_input":"2023-04-01T14:21:09.141631Z","iopub.status.idle":"2023-04-01T14:21:09.147934Z","shell.execute_reply.started":"2023-04-01T14:21:09.141597Z","shell.execute_reply":"2023-04-01T14:21:09.146507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Dataset [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"class CustomDataset(Dataset):\n\n def __init__(self, df, transforms=None):\n super().__init__()\n self.df = df # pandas dataframe\n self.transforms = transforms # albumentations transformations\n self.image_ids = self.df[\"image_id\"].unique().tolist() # list with unique image IDs\n\n def __getitem__(self, index: int):\n \"\"\"\n :return image: an augmented image as a numpy array.\n :return target: a dictionary containing a tensor with the bboxes for the image (torch.Tensor),\n a list of strings containing the bboxes labels and a tensor containing the image index.\n :return image_id: the image ID. A unique identifier for each image. \n \n \"\"\"\n image_id = self.df[\"image_id\"][index] # select one image\n image = cv2.imread(f'{paths.TEST_IMAGES_FOLDER}{image_id}.jpg', cv2.IMREAD_COLOR) # read image from path\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32) # convert to RGB \n transformed_image = self.transforms(**{\"image\":image})\n image = transformed_image['image']\n return image, image_id\n\n def __len__(self) -> int:\n return len(self.image_ids)","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:21:09.650423Z","iopub.execute_input":"2023-04-01T14:21:09.651556Z","iopub.status.idle":"2023-04-01T14:21:09.660674Z","shell.execute_reply.started":"2023-04-01T14:21:09.651509Z","shell.execute_reply":"2023-04-01T14:21:09.659491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plot one image","metadata":{}},{"cell_type":"code","source":"test_dataset = CustomDataset(\n df=df,\n transforms=get_test_transforms()\n)\nindex = random.choice(range(0,len(test_dataset))) # select random image index\nimage, image_id = test_dataset[index] # get one sample from dataset\nnumpy_image = image.permute(1,2,0).cpu().numpy() # get image\nprint(f\"Image ID: {image_id}\") \nfig, ax = plt.subplots(1, 1, figsize=(10, 8)) # create figure\nax.set_axis_off()\nax.imshow(numpy_image)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-01T14:19:32.134596Z","iopub.execute_input":"2023-04-01T14:19:32.135090Z","iopub.status.idle":"2023-04-01T14:19:32.511912Z","shell.execute_reply.started":"2023-04-01T14:19:32.135053Z","shell.execute_reply":"2023-04-01T14:19:32.510860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | DataLoaders [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"def collate_fn(batch):\n return tuple(zip(*batch))\n\n\ndef prepare_loaders(df):\n \"\"\"\n Creates a PyTorch dataset. Creates DataLoader from Dataset.\n :param df: test dataframe (pandas dataframe).\n :return test_loader: dataloaders for test stage.\n \"\"\"\n # === Select data for Train and Validation ===\n test_df = df.reset_index(drop=True) # Select all rows not from validation fold\n \n # === Build Datasets ===\n test_dataset = CustomDataset(test_df, transforms=get_test_transforms())\n \n # === Create DataLoaders for Train and Validation ===\n test_loader = DataLoader(\n test_dataset,\n batch_size=config.BATCH_SIZE_TEST, \n num_workers=config.NUM_WORKERS,\n sampler=SequentialSampler(test_dataset),\n shuffle=False, pin_memory=True,\n drop_last=False, collate_fn=collate_fn\n )\n \n return test_loader\n\ntest_loader = prepare_loaders(df=df)","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:19:32.513308Z","iopub.execute_input":"2023-04-01T14:19:32.514435Z","iopub.status.idle":"2023-04-01T14:19:32.523463Z","shell.execute_reply.started":"2023-04-01T14:19:32.514394Z","shell.execute_reply":"2023-04-01T14:19:32.522422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Model [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"from effdet.config.model_config import efficientdet_model_param_dict\nfrom effdet import get_efficientdet_config, EfficientDet, DetBenchTrain, DetBenchPredict\nfrom effdet.efficientdet import HeadNet\nfrom effdet.config.model_config import efficientdet_model_param_dict\n\ndef create_model(num_classes=config.NUM_CLASSES, image_size=512,\n architecture=\"tf_efficientnetv2_s\", verbose=False):\n efficientdet_model_param_dict['tf_efficientnetv2_s'] = dict(\n name='tf_efficientnetv2_s',\n backbone_name='tf_efficientnetv2_s',\n backbone_args=dict(drop_path_rate=0.2),\n num_classes=num_classes,\n url='', )\n \n cfg = get_efficientdet_config(architecture)\n cfg.update({'num_classes': num_classes})\n cfg.update({'image_size': (image_size, image_size)})\n \n if verbose:\n pprint(cfg)\n\n net = EfficientDet(cfg, pretrained_backbone=True)\n net.class_net = HeadNet(\n cfg,\n num_outputs=cfg.num_classes,\n )\n return DetBenchPredict(net)\n\n\ndef load_model(model_weights_path, model):\n \"\"\"\n Load model weights.\n \"\"\"\n model.load_state_dict(torch.load(model_weights_path))\n model.eval()\n return model","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:19:32.527692Z","iopub.execute_input":"2023-04-01T14:19:32.528385Z","iopub.status.idle":"2023-04-01T14:19:33.118445Z","shell.execute_reply.started":"2023-04-01T14:19:32.528357Z","shell.execute_reply":"2023-04-01T14:19:33.117243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Create Model [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"best_model = create_model(num_classes=config.NUM_CLASSES, verbose=True)\nbest_model = load_model(paths.BEST_MODEL, best_model)\nbest_model.to(config.DEVICE)","metadata":{"scrolled":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-01T14:21:30.452759Z","iopub.execute_input":"2023-04-01T14:21:30.453759Z","iopub.status.idle":"2023-04-01T14:21:31.630174Z","shell.execute_reply.started":"2023-04-01T14:21:30.453720Z","shell.execute_reply":"2023-04-01T14:21:31.629161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Inference Function [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"@torch.no_grad()\ndef inference_function(model, dataloader, device):\n model.eval() # Set model in evaluation mode\n pbar = tqdm(enumerate(dataloader), total=len(dataloader), desc='Test ') # Progress bar\n predictions = []\n ids = []\n for step, (images, image_ids) in pbar: \n # === Collate ===\n images = torch.stack(images).to(device).float() # Get images\n batch_size = images.shape[0] # Get batch size\n preds = model(images).cpu().numpy().tolist()\n predictions += preds # Forward pass\n ids += image_ids\n return predictions, ids","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:21:33.976543Z","iopub.execute_input":"2023-04-01T14:21:33.977057Z","iopub.status.idle":"2023-04-01T14:21:33.984383Z","shell.execute_reply.started":"2023-04-01T14:21:33.977021Z","shell.execute_reply":"2023-04-01T14:21:33.983001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Inference [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"test_loader = prepare_loaders(df=df)\npredictions, image_ids = inference_function(best_model, test_loader, config.DEVICE)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-04-01T14:21:35.860380Z","iopub.execute_input":"2023-04-01T14:21:35.861023Z","iopub.status.idle":"2023-04-01T14:21:40.736550Z","shell.execute_reply.started":"2023-04-01T14:21:35.860980Z","shell.execute_reply":"2023-04-01T14:21:40.734777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Predictions [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"preds_df = pd.DataFrame([image_ids, predictions]).T\npreds_df.columns = [\"image_id\", \"prediction\"]\npreds_df = preds_df.explode(\"prediction\")\ncols = ['x_min', 'y_min', 'x_max', 'y_max', 'prob', 'label']\npreds_df[cols] = pd.DataFrame(preds_df.prediction.tolist(), index= preds_df.index)\npreds_df = preds_df[preds_df[\"prob\"]>0.35]\npreds_df","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:35:55.570501Z","iopub.execute_input":"2023-04-01T14:35:55.570917Z","iopub.status.idle":"2023-04-01T14:35:55.602578Z","shell.execute_reply.started":"2023-04-01T14:35:55.570878Z","shell.execute_reply":"2023-04-01T14:35:55.601481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Plot Predictions [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"for image_id in preds_df.image_id.unique().tolist():\n aux = preds_df[preds_df[\"image_id\"]==image_id]\n image = cv2.imread(paths.TEST_IMAGES_FOLDER + image_id + \".jpg\")\n transform = A.Resize(height=config.RESOLUTION, width=config.RESOLUTION, p=1)\n transformed_image = transform(**{\"image\":image})\n image = transformed_image[\"image\"]\n print(f\"Image ID: {image_id}\")\n fig, ax = plt.subplots(1, 1, figsize=(12, 10)) # create figure\n for index, row in aux.iterrows():\n label = int(row[\"label\"])\n x_min, x_max, y_min, y_max = int(row[\"x_min\"]), int(row[\"x_max\"]), int(row[\"y_min\"]), int(row[\"y_max\"])\n # === Plot bounding boxes ====\n cv2.rectangle(image, (x_min, y_min), (x_max, y_max), color_map[label], 2)\n ax.set_axis_off()\n ax.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2023-04-01T14:35:56.168924Z","iopub.execute_input":"2023-04-01T14:35:56.169633Z","iopub.status.idle":"2023-04-01T14:35:57.765782Z","shell.execute_reply.started":"2023-04-01T14:35:56.169599Z","shell.execute_reply":"2023-04-01T14:35:57.764863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Conclusion [↑](#top) \n\n***\n\nAs we can see it does a great job!\n\nNow, knowing where the bounding boxes for the ticks are we can know the tick labels which are necessary for making predictions!\n\nIf you liked the notebook I would be grateful if you could upvote 🍀","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}