{ "id": 418705, "title": "28th Place Solution", "url": "/competitions/benetech-making-graphs-accessible/writeups/complex-1-28th-place-solution", "author_name": "", "post_date": "2023-07-30T01:02:34.157Z", "votes": 13, "comment_count": 5, "views": 0, "content": "

First of all, we would like to thank Kaggle & Benetech for hosting this very interesting competition. It was great fun to participate, and we learned a tremendous amount from it. ๐Ÿ˜€

\n

We used a combination of image classification, object detection, Donut, PaddleOCR, TrOCR, and lots of postprocessing! ๐Ÿ˜…

\n
\n

Models

\n

We trained models for the following tasks:

\n
    \n
  1. Plot type classification - ResNet34
    2. \n
  2. Rotated vs non-rotated X tick label classification - ResNet34
    3. \n
  3. Horizontal bar detection - FasterRCNN with ResNet50-FPN backbone
    4. \n
  4. Vertical bar detection (used in the histogram pipeline as well) - FasterRCNN with ResNet50-FPN backbone
    5. \n
  5. Line point detection - FasterRCNN with ResNet50-FPN backbone
    6. \n
  6. Scatter point & dot detection (a single model for both) - FasterRCNN with ResNet50-FPN backbone
    7. \n
  7. Tick mark detection - FasterRCNN with ResNet50-FPN backbone
    8. \n
  8. X tick label detection (used only for detecting non-rotated X tick labels) - FasterRCNN with ResNet50-FPN backbone
    9. \n
  9. Y tick label detection - FasterRCNN with ResNet50-FPN backbone
    10. \n
  10. Donut (used only for recognizing rotated X tick labels) - 'naver-clova-ix/donut-base'
    11. \n
\n

In addition, we used the following models as-is (i.e., without any fine-tuning):

\n
    \n
  1. PaddleOCR for multi-line text detection & conversion to single-line texts - PP-OCRv3
    2. \n
  2. TrOCR for OCR of single-line texts - 'microsoft/trocr-base-printed'
    3. \n
\n

Note: We used PaddleOCR because TrOCR can only read single-line texts. For single-line texts, we found TrOCR to be very accurate.

\n
\n

Inference Examples

\n

The following images are examples of inference using each of the above models:

\n

Plot type classification:

\n

\"\"

\n

Rotated vs non-rotated X tick label classification:

\n

\"\"

\n

Horizontal bar detection:

\n

\n

Vertical bar detection (used in the histogram pipeline as well):

\n

\n

Line point detection:

\n

\n

Scatter point & dot detection (a single model for both):

\n

\n

\n

Tick mark detection:

\n

\n

X tick label detection (used only for detecting non-rotated X tick labels):

\n

\n

Y tick label detection:

\n

\n

Donut (used only for recognizing rotated X tick labels):

\n

\"\"

\n

Note: We trained our Donut model only on image patches below the X axis which contain rotated text.

\n

PaddleOCR (for multi-line text detection & conversion to single-line texts):

\n

\"\"

\n

TrOCR (for OCR of single-line texts):

\n

Image snippet:

\n

\"\"

\n

Output: \"A. Total Genotoxic Potency\"

\n
\n

Deep Learning Frameworks

\n

We used:

\n\n
\n

Data

\n

In addition to the competition dataset, we used some additional images from the ICPR 2022 CHART dataset available at: https://chartinfo.github.io/toolsanddata.html

\n

Bounding box annotations for the chart elements were added using either (i) custom Python scripts or (ii) manual annotation tools, viz. Make Sense (https://www.makesense.ai/) and Roboflow. The annotations were created in the PASCAL VOC format.

\n

For line point detection, scatter point detection and tick mark detection, we had to convert the original point annotations to bounding box annotations. We noticed that the choice of bounding box size has a big impact on performance. Hence, we experimented with a few bounding box sizes, and found that the following sizes work well:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Line PointScatter PointTick Mark
18px16px16px
\n

For X tick label detection and Y tick label detection, we noticed that adding a 2px margin to the provided bounding boxes improves performance.

\n

Finally, we found that in a tiny fraction of images, one or more bounding boxes overstep the bounds of the image. We deleted these images from our dataset.

\n
\n

Model Validation Schemes

\n

Plot type classification:

\n

\"\"

\n

Rotated vs non-rotated X tick label classification:

\n

\"\"

\n

Horizontal bar detection:

\n

\"\"

\n

Vertical bar detection (used in the histogram pipeline as well):

\n

\"\"

\n

Line point detection:

\n

\"\"

\n

Scatter point & dot detection (a single model for both):

\n

\"\"

\n

Tick mark detection:

\n

\"\"

\n

X tick label detection (used only for detecting non-rotated X tick labels):

\n

\"\"

\n

Y tick label detection:

\n

\"\"

\n

Donut (used only for recognizing rotated X tick labels):

\n

\"\"

\n
\n

Pipelines

\n

Image Classification

\n

\"\"

\n

Horizontal Bar Pipeline

\n

\"\"

\n

Vertical Bar Pipeline

\n

\"\"

\n

Histogram Pipeline

\n

\"\"

\n

Line Pipeline

\n

\"\"

\n

Scatter Pipeline

\n

\"\"

\n

Dot Pipeline

\n

\"\"

\n
\n

Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
OverallScatterDotLineVertical BarHorizontal Bar
public0.660.060.000.260.330.01
private0.490.190.010.080.200.01
\n
", "messages": [ { "id": "2312747", "postDate": "06/22/2023 06:58:44", "content": "

First of all, we would like to thank Kaggle & Benetech for hosting this very interesting competition. It was great fun to participate, and we learned a tremendous amount from it. ๐Ÿ˜€

\n

We used a combination of image classification, object detection, Donut, PaddleOCR, TrOCR, and lots of postprocessing! ๐Ÿ˜…

\n
\n

Models

\n

We trained models for the following tasks:

\n
    \n
  1. Plot type classification - ResNet34
    2. \n
  2. Rotated vs non-rotated X tick label classification - ResNet34
    3. \n
  3. Horizontal bar detection - FasterRCNN with ResNet50-FPN backbone
    4. \n
  4. Vertical bar detection (used in the histogram pipeline as well) - FasterRCNN with ResNet50-FPN backbone
    5. \n
  5. Line point detection - FasterRCNN with ResNet50-FPN backbone
    6. \n
  6. Scatter point & dot detection (a single model for both) - FasterRCNN with ResNet50-FPN backbone
    7. \n
  7. Tick mark detection - FasterRCNN with ResNet50-FPN backbone
    8. \n
  8. X tick label detection (used only for detecting non-rotated X tick labels) - FasterRCNN with ResNet50-FPN backbone
    9. \n
  9. Y tick label detection - FasterRCNN with ResNet50-FPN backbone
    10. \n
  10. Donut (used only for recognizing rotated X tick labels) - 'naver-clova-ix/donut-base'
    11. \n
\n

In addition, we used the following models as-is (i.e., without any fine-tuning):

\n
    \n
  1. PaddleOCR for multi-line text detection & conversion to single-line texts - PP-OCRv3
    2. \n
  2. TrOCR for OCR of single-line texts - 'microsoft/trocr-base-printed'
    3. \n
\n

Note: We used PaddleOCR because TrOCR can only read single-line texts. For single-line texts, we found TrOCR to be very accurate.

\n
\n

Inference Examples

\n

The following images are examples of inference using each of the above models:

\n

Plot type classification:

\n

\"\"

\n

Rotated vs non-rotated X tick label classification:

\n

\"\"

\n

Horizontal bar detection:

\n

\n

Vertical bar detection (used in the histogram pipeline as well):

\n

\n

Line point detection:

\n

\n

Scatter point & dot detection (a single model for both):

\n

\n

\n

Tick mark detection:

\n

\n

X tick label detection (used only for detecting non-rotated X tick labels):

\n

\n

Y tick label detection:

\n

\n

Donut (used only for recognizing rotated X tick labels):

\n

\"\"

\n

Note: We trained our Donut model only on image patches below the X axis which contain rotated text.

\n

PaddleOCR (for multi-line text detection & conversion to single-line texts):

\n

\"\"

\n

TrOCR (for OCR of single-line texts):

\n

Image snippet:

\n

\"\"

\n

Output: \"A. Total Genotoxic Potency\"

\n
\n

Deep Learning Frameworks

\n

We used:

\n\n
\n

Data

\n

In addition to the competition dataset, we used some additional images from the ICPR 2022 CHART dataset available at: https://chartinfo.github.io/toolsanddata.html

\n

Bounding box annotations for the chart elements were added using either (i) custom Python scripts or (ii) manual annotation tools, viz. Make Sense (https://www.makesense.ai/) and Roboflow. The annotations were created in the PASCAL VOC format.

\n

For line point detection, scatter point detection and tick mark detection, we had to convert the original point annotations to bounding box annotations. We noticed that the choice of bounding box size has a big impact on performance. Hence, we experimented with a few bounding box sizes, and found that the following sizes work well:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Line PointScatter PointTick Mark
18px16px16px
\n

For X tick label detection and Y tick label detection, we noticed that adding a 2px margin to the provided bounding boxes improves performance.

\n

Finally, we found that in a tiny fraction of images, one or more bounding boxes overstep the bounds of the image. We deleted these images from our dataset.

\n
\n

Model Validation Schemes

\n

Plot type classification:

\n

\"\"

\n

Rotated vs non-rotated X tick label classification:

\n

\"\"

\n

Horizontal bar detection:

\n

\"\"

\n

Vertical bar detection (used in the histogram pipeline as well):

\n

\"\"

\n

Line point detection:

\n

\"\"

\n

Scatter point & dot detection (a single model for both):

\n

\"\"

\n

Tick mark detection:

\n

\"\"

\n

X tick label detection (used only for detecting non-rotated X tick labels):

\n

\"\"

\n

Y tick label detection:

\n

\"\"

\n

Donut (used only for recognizing rotated X tick labels):

\n

\"\"

\n
\n

Pipelines

\n

Image Classification

\n

\"\"

\n

Horizontal Bar Pipeline

\n

\"\"

\n

Vertical Bar Pipeline

\n

\"\"

\n

Histogram Pipeline

\n

\"\"

\n

Line Pipeline

\n

\"\"

\n

Scatter Pipeline

\n

\"\"

\n

Dot Pipeline

\n

\"\"

\n
\n

Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
OverallScatterDotLineVertical BarHorizontal Bar
public0.660.060.000.260.330.01
private0.490.190.010.080.200.01
\n
", "rawMarkdown": "First of all, we would like to thank Kaggle & Benetech for hosting this very interesting competition. It was great fun to participate, and we learned a tremendous amount from it. ๐Ÿ˜€\n\nWe used a combination of image classification, object detection, Donut, PaddleOCR, TrOCR, and lots of postprocessing! ๐Ÿ˜…\n\n---\n\n# Models\n\nWe trained models for the following tasks:\n\n1. Plot type classification - ResNet34\n2. Rotated vs non-rotated X tick label classification - ResNet34\n3. Horizontal bar detection - FasterRCNN with ResNet50-FPN backbone\n4. Vertical bar detection (used in the histogram pipeline as well) - FasterRCNN with ResNet50-FPN backbone\n5. Line point detection - FasterRCNN with ResNet50-FPN backbone\n6. Scatter point & dot detection (a single model for both) - FasterRCNN with ResNet50-FPN backbone\n7. Tick mark detection - FasterRCNN with ResNet50-FPN backbone\n8. X tick label detection (used only for detecting non-rotated X tick labels) - FasterRCNN with ResNet50-FPN backbone\n9. Y tick label detection - FasterRCNN with ResNet50-FPN backbone\n10. Donut (used only for recognizing rotated X tick labels) - `'naver-clova-ix/donut-base'`\n\nIn addition, we used the following models as-is (i.e., without any fine-tuning):\n\n1. PaddleOCR for multi-line text detection & conversion to single-line texts - PP-OCRv3\n2. TrOCR for OCR of single-line texts - `'microsoft/trocr-base-printed'`\n\n**Note:** We used PaddleOCR because TrOCR can only read single-line texts. For single-line texts, we found TrOCR to be very accurate.\n\n---\n\n# Inference Examples\n\nThe following images are examples of inference using each of the above models:\n\n**Plot type classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Faf30206384b2f090c6fcbd0cb59bf959%2Fplot_type_inference.png?generation=1687424929176720&alt=media)\n\n**Rotated vs non-rotated X tick label classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F037c35db3590c5c1b67f85ab57fb36e2%2Frotation_inference.png?generation=1687422645141712&alt=media)\n\n**Horizontal bar detection:**\n\n\n\n**Vertical bar detection (used in the histogram pipeline as well):**\n\n\n\n**Line point detection:**\n\n\n\n**Scatter point & dot detection (a single model for both):**\n\n\n\n\n\n**Tick mark detection:**\n\n\n\n**X tick label detection (used only for detecting non-rotated X tick labels):**\n\n\n\n**Y tick label detection:**\n\n\n\n**Donut (used only for recognizing rotated X tick labels):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ffbb5753a331e6cbb22fa19cc56a55a04%2Fdonut_inference.png?generation=1687429751410113&alt=media)\n\n**Note:** We trained our Donut model only on image patches below the X axis which contain rotated text.\n\n**PaddleOCR (for multi-line text detection & conversion to single-line texts):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ff3647d71073644c6033c7116e1e33da2%2Fpaddle.png?generation=1687440095811819&alt=media)\n\n**TrOCR (for OCR of single-line texts):**\n\nImage snippet:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F49734d8ecacdb174d1bfaf431d15be99%2Ftrocr_snippet.jpg?generation=1687429867440216&alt=media)\n\nOutput: `\"A. Total Genotoxic Potency\"`\n\n---\n\n# Deep Learning Frameworks\n\nWe used:\n\n- For image classification - fastai\n- For object detection - IceVision + fastai\n- For Donut - Hugging Face Transformers Trainer API\n\n---\n\n# Data\n\nIn addition to the competition dataset, we used some additional images from the *ICPR 2022 CHART* dataset available at: https://chartinfo.github.io/toolsanddata.html\n\nBounding box annotations for the chart elements were added using either (i) custom Python scripts or (ii) manual annotation tools, viz. Make Sense (https://www.makesense.ai/) and Roboflow. The annotations were created in the PASCAL VOC format.\n\nFor line point detection, scatter point detection and tick mark detection, we had to convert the original point annotations to bounding box annotations. We noticed that the choice of bounding box size has a big impact on performance. Hence, we experimented with a few bounding box sizes, and found that the following sizes work well:\n\n| Line Point | Scatter Point | Tick Mark |\n|---|---|---|\n| 18px | 16px | 16px |\n\nFor X tick label detection and Y tick label detection, we noticed that adding a 2px margin to the provided bounding boxes improves performance.\n\nFinally, we found that in a tiny fraction of images, one or more bounding boxes overstep the bounds of the image. We deleted these images from our dataset.\n\n---\n\n# Model Validation Schemes\n\n**Plot type classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fb0e8a053b3a9509a36b0a03b0935f248%2Fplot_type_val_scheme.png?generation=1689168901672895&alt=media)\n\n**Rotated vs non-rotated X tick label classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F7a1cdc221a5e10f875a76b9163888e21%2Frotation_val_scheme.png?generation=1689169392117974&alt=media)\n\n**Horizontal bar detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fa866016d76730c38864c90c317fb89b7%2Fhorizontal_bar_val_scheme.png?generation=1689168936143640&alt=media)\n\n**Vertical bar detection (used in the histogram pipeline as well):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fd27e058d0bb05846f5ea42eca69e101f%2Fvertical_bar_val_scheme.png?generation=1689168951683704&alt=media)\n\n**Line point detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ffc25de18f4386ad8711b7953ddffc904%2Fline_val_scheme.png?generation=1689168967592482&alt=media)\n\n**Scatter point & dot detection (a single model for both):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F2411ee14867dda5e3b7aba79d7d234cf%2Fscatter_dot_val_scheme.png?generation=1689170209501077&alt=media)\n\n**Tick mark detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fc3cc591f20cadb2cb59c53fa40ba9b0b%2Ftick_mark_val_scheme.png?generation=1689170225648108&alt=media)\n\n**X tick label detection (used only for detecting non-rotated X tick labels):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F2c1ee44e72bbea46fb8e3e514ad042d2%2Fx_tick_label_val_scheme.png?generation=1689170239312207&alt=media)\n\n**Y tick label detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F04a9bed0423c7c186b6b4a3a24b1fc8a%2Fy_tick_label_val_scheme.png?generation=1689170253004364&alt=media)\n\n**Donut (used only for recognizing rotated X tick labels):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fce3009c4f44c3d1126e268840097779b%2Fdonut_val_scheme.png?generation=1689170268258727&alt=media)\n\n---\n\n# Pipelines\n\n## Image Classification\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F65ff8a8d4344bd4c111c8a1d595b740a%2Fdiagram_image_classifier.png?generation=1687437455837261&alt=media)\n\n## Horizontal Bar Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ff1c8ac7ea8ce27d36a9c8f21ad5a3112%2Fdiagram_horizontal_bar.png?generation=1687695397682069&alt=media)\n\n## Vertical Bar Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F4b4ea6617dd3907c0a044474cc475fef%2Fdiagram_vertical_bar.png?generation=1687695428772800&alt=media)\n\n## Histogram Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F717f752077d0b61d3f277aba0ab66877%2Fdiagram_histogram.png?generation=1687696678498713&alt=media)\n\n## Line Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F2e1debd34468def631db044e287cb0f5%2Fdiagram_line.png?generation=1687696700722044&alt=media)\n\n## Scatter Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fc9b7be151091d9a3cc030f8341d988bf%2Fdiagram_scatter.png?generation=1687518134085611&alt=media)\n\n## Dot Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F8b79f1c390dff4b5c8a703767ee3a372%2Fdiagram_dot.png?generation=1687764390803909&alt=media)\n\n---\n\n# Results\n\n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n|---|---|---|---|---|---|---|\n| public | 0.66 | 0.06 | 0.00 | 0.26 | 0.33 | 0.01 |\n| private | 0.49 | 0.19 | 0.01 | 0.08 | 0.20 | 0.01 |\n\n---", "votes": null }, { "id": "2313984", "postDate": "06/23/2023 05:16:49", "content": "

Nice!! We did our best and tried everything we could think of with what we knew. I learned a lot from this competition!! @sambitmukherjee

", "rawMarkdown": "Nice!! We did our best and tried everything we could think of with what we knew. I learned a lot from this competition!! @sambitmukherjee", "votes": null }, { "id": "2317184", "postDate": "06/25/2023 13:53:56", "content": "

nice pipeline graphs

", "rawMarkdown": "nice pipeline graphs", "votes": null }, { "id": "2317300", "postDate": "06/25/2023 15:26:28", "content": "

Thanks. I used a free web-based tool called Excalidrawโ€ฆ

", "rawMarkdown": "Thanks. I used a free web-based tool called Excalidraw...", "votes": null }, { "id": "2317922", "postDate": "06/26/2023 05:04:15", "content": "

We (complex[::-1]) Team, thank competition host for conducting such a interesting completion. It is a very good learning for us. I thank all participants also for sharing your insights in discussion, which helped us a lot.

\n

Reading of various winning solutions giving me good knowledge and diversified approaches for the same problem.

\n

Finally, I thank my team @sambitmukherjee, @aravind36 and @wasimmadha for fantastic work..

", "rawMarkdown": "We (complex[::-1]) Team, thank competition host for conducting such a interesting completion. It is a very good learning for us. I thank all participants also for sharing your insights in discussion, which helped us a lot. \n\nReading of various winning solutions giving me good knowledge and diversified approaches for the same problem. \n\nFinally, I thank my team @sambitmukherjee, @aravind36 and @wasimmadha for fantastic work..", "votes": null }, { "id": "2319382", "postDate": "06/27/2023 05:24:44", "content": "

@sambitmukherjee thanks for sharing detailed topic! Very nice graphs!

", "rawMarkdown": "sambitmukherjee thanks for sharing detailed topic! Very nice graphs!", "votes": null } ], "comments": [ { "id": 2313984, "author_name": "aravind36", "author_url": "", "post_date": "06/23/2023 05:16:49", "content": "

Nice!! We did our best and tried everything we could think of with what we knew. I learned a lot from this competition!! @sambitmukherjee

", "votes": null, "replies": [] }, { "id": 2317184, "author_name": "joek47", "author_url": "", "post_date": "06/25/2023 13:53:56", "content": "

nice pipeline graphs

", "votes": null, "replies": [ { "id": 2317300, "author_name": "sambitmukherjee", "author_url": "", "post_date": "06/25/2023 15:26:28", "content": "

Thanks. I used a free web-based tool called Excalidrawโ€ฆ

", "votes": null, "replies": [] } ] }, { "id": 2317922, "author_name": "laskari", "author_url": "", "post_date": "06/26/2023 05:04:15", "content": "

We (complex[::-1]) Team, thank competition host for conducting such a interesting completion. It is a very good learning for us. I thank all participants also for sharing your insights in discussion, which helped us a lot.

\n

Reading of various winning solutions giving me good knowledge and diversified approaches for the same problem.

\n

Finally, I thank my team @sambitmukherjee, @aravind36 and @wasimmadha for fantastic work..

", "votes": null, "replies": [] }, { "id": 2319382, "author_name": "serangu", "author_url": "", "post_date": "06/27/2023 05:24:44", "content": "

@sambitmukherjee thanks for sharing detailed topic! Very nice graphs!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2312747": "First of all, we would like to thank Kaggle & Benetech for hosting this very interesting competition. It was great fun to participate, and we learned a tremendous amount from it. ๐Ÿ˜€\n\nWe used a combination of image classification, object detection, Donut, PaddleOCR, TrOCR, and lots of postprocessing! ๐Ÿ˜…\n\n---\n\n# Models\n\nWe trained models for the following tasks:\n\n1. Plot type classification - ResNet34\n2. Rotated vs non-rotated X tick label classification - ResNet34\n3. Horizontal bar detection - FasterRCNN with ResNet50-FPN backbone\n4. Vertical bar detection (used in the histogram pipeline as well) - FasterRCNN with ResNet50-FPN backbone\n5. Line point detection - FasterRCNN with ResNet50-FPN backbone\n6. Scatter point & dot detection (a single model for both) - FasterRCNN with ResNet50-FPN backbone\n7. Tick mark detection - FasterRCNN with ResNet50-FPN backbone\n8. X tick label detection (used only for detecting non-rotated X tick labels) - FasterRCNN with ResNet50-FPN backbone\n9. Y tick label detection - FasterRCNN with ResNet50-FPN backbone\n10. Donut (used only for recognizing rotated X tick labels) - `'naver-clova-ix/donut-base'`\n\nIn addition, we used the following models as-is (i.e., without any fine-tuning):\n\n1. PaddleOCR for multi-line text detection & conversion to single-line texts - PP-OCRv3\n2. TrOCR for OCR of single-line texts - `'microsoft/trocr-base-printed'`\n\n**Note:** We used PaddleOCR because TrOCR can only read single-line texts. For single-line texts, we found TrOCR to be very accurate.\n\n---\n\n# Inference Examples\n\nThe following images are examples of inference using each of the above models:\n\n**Plot type classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Faf30206384b2f090c6fcbd0cb59bf959%2Fplot_type_inference.png?generation=1687424929176720&alt=media)\n\n**Rotated vs non-rotated X tick label classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F037c35db3590c5c1b67f85ab57fb36e2%2Frotation_inference.png?generation=1687422645141712&alt=media)\n\n**Horizontal bar detection:**\n\n\n\n**Vertical bar detection (used in the histogram pipeline as well):**\n\n\n\n**Line point detection:**\n\n\n\n**Scatter point & dot detection (a single model for both):**\n\n\n\n\n\n**Tick mark detection:**\n\n\n\n**X tick label detection (used only for detecting non-rotated X tick labels):**\n\n\n\n**Y tick label detection:**\n\n\n\n**Donut (used only for recognizing rotated X tick labels):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ffbb5753a331e6cbb22fa19cc56a55a04%2Fdonut_inference.png?generation=1687429751410113&alt=media)\n\n**Note:** We trained our Donut model only on image patches below the X axis which contain rotated text.\n\n**PaddleOCR (for multi-line text detection & conversion to single-line texts):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ff3647d71073644c6033c7116e1e33da2%2Fpaddle.png?generation=1687440095811819&alt=media)\n\n**TrOCR (for OCR of single-line texts):**\n\nImage snippet:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F49734d8ecacdb174d1bfaf431d15be99%2Ftrocr_snippet.jpg?generation=1687429867440216&alt=media)\n\nOutput: `\"A. Total Genotoxic Potency\"`\n\n---\n\n# Deep Learning Frameworks\n\nWe used:\n\n- For image classification - fastai\n- For object detection - IceVision + fastai\n- For Donut - Hugging Face Transformers Trainer API\n\n---\n\n# Data\n\nIn addition to the competition dataset, we used some additional images from the *ICPR 2022 CHART* dataset available at: https://chartinfo.github.io/toolsanddata.html\n\nBounding box annotations for the chart elements were added using either (i) custom Python scripts or (ii) manual annotation tools, viz. Make Sense (https://www.makesense.ai/) and Roboflow. The annotations were created in the PASCAL VOC format.\n\nFor line point detection, scatter point detection and tick mark detection, we had to convert the original point annotations to bounding box annotations. We noticed that the choice of bounding box size has a big impact on performance. Hence, we experimented with a few bounding box sizes, and found that the following sizes work well:\n\n| Line Point | Scatter Point | Tick Mark |\n|---|---|---|\n| 18px | 16px | 16px |\n\nFor X tick label detection and Y tick label detection, we noticed that adding a 2px margin to the provided bounding boxes improves performance.\n\nFinally, we found that in a tiny fraction of images, one or more bounding boxes overstep the bounds of the image. We deleted these images from our dataset.\n\n---\n\n# Model Validation Schemes\n\n**Plot type classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fb0e8a053b3a9509a36b0a03b0935f248%2Fplot_type_val_scheme.png?generation=1689168901672895&alt=media)\n\n**Rotated vs non-rotated X tick label classification:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F7a1cdc221a5e10f875a76b9163888e21%2Frotation_val_scheme.png?generation=1689169392117974&alt=media)\n\n**Horizontal bar detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fa866016d76730c38864c90c317fb89b7%2Fhorizontal_bar_val_scheme.png?generation=1689168936143640&alt=media)\n\n**Vertical bar detection (used in the histogram pipeline as well):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fd27e058d0bb05846f5ea42eca69e101f%2Fvertical_bar_val_scheme.png?generation=1689168951683704&alt=media)\n\n**Line point detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ffc25de18f4386ad8711b7953ddffc904%2Fline_val_scheme.png?generation=1689168967592482&alt=media)\n\n**Scatter point & dot detection (a single model for both):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F2411ee14867dda5e3b7aba79d7d234cf%2Fscatter_dot_val_scheme.png?generation=1689170209501077&alt=media)\n\n**Tick mark detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fc3cc591f20cadb2cb59c53fa40ba9b0b%2Ftick_mark_val_scheme.png?generation=1689170225648108&alt=media)\n\n**X tick label detection (used only for detecting non-rotated X tick labels):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F2c1ee44e72bbea46fb8e3e514ad042d2%2Fx_tick_label_val_scheme.png?generation=1689170239312207&alt=media)\n\n**Y tick label detection:**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F04a9bed0423c7c186b6b4a3a24b1fc8a%2Fy_tick_label_val_scheme.png?generation=1689170253004364&alt=media)\n\n**Donut (used only for recognizing rotated X tick labels):**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fce3009c4f44c3d1126e268840097779b%2Fdonut_val_scheme.png?generation=1689170268258727&alt=media)\n\n---\n\n# Pipelines\n\n## Image Classification\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F65ff8a8d4344bd4c111c8a1d595b740a%2Fdiagram_image_classifier.png?generation=1687437455837261&alt=media)\n\n## Horizontal Bar Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Ff1c8ac7ea8ce27d36a9c8f21ad5a3112%2Fdiagram_horizontal_bar.png?generation=1687695397682069&alt=media)\n\n## Vertical Bar Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F4b4ea6617dd3907c0a044474cc475fef%2Fdiagram_vertical_bar.png?generation=1687695428772800&alt=media)\n\n## Histogram Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F717f752077d0b61d3f277aba0ab66877%2Fdiagram_histogram.png?generation=1687696678498713&alt=media)\n\n## Line Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F2e1debd34468def631db044e287cb0f5%2Fdiagram_line.png?generation=1687696700722044&alt=media)\n\n## Scatter Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2Fc9b7be151091d9a3cc030f8341d988bf%2Fdiagram_scatter.png?generation=1687518134085611&alt=media)\n\n## Dot Pipeline\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1191272%2F8b79f1c390dff4b5c8a703767ee3a372%2Fdiagram_dot.png?generation=1687764390803909&alt=media)\n\n---\n\n# Results\n\n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n|---|---|---|---|---|---|---|\n| public | 0.66 | 0.06 | 0.00 | 0.26 | 0.33 | 0.01 |\n| private | 0.49 | 0.19 | 0.01 | 0.08 | 0.20 | 0.01 |\n\n---", "2313984": "Nice!! We did our best and tried everything we could think of with what we knew. I learned a lot from this competition!! @sambitmukherjee", "2317184": "nice pipeline graphs", "2317300": "Thanks. I used a free web-based tool called Excalidraw...", "2317922": "We (complex[::-1]) Team, thank competition host for conducting such a interesting completion. It is a very good learning for us. I thank all participants also for sharing your insights in discussion, which helped us a lot. \n\nReading of various winning solutions giving me good knowledge and diversified approaches for the same problem. \n\nFinally, I thank my team @sambitmukherjee, @aravind36 and @wasimmadha for fantastic work..", "2319382": "sambitmukherjee thanks for sharing detailed topic! Very nice graphs!" }, "source": "meta" }