Thank Kaggle and the host to organize the competition. I want to do a quick write-up of my simple solution. Everything is basic, all steps can be done using Kaggle and Google Colab (Pro) computing resources.
\nFinal Submission notebook: https://www.kaggle.com/code/namgalielei/benetech-eval-and-infer-v12/notebook
\nChart type classification + Plot BBox prediction -> OCR -> Chart data instance detection / segmentation (line) -> Data association
\nSome thing not having time to try yet: External dataset, a wider range of instance segmentation / object detection models, Generative models
", "messages": [ { "id": "2309931", "postDate": "06/20/2023 02:02:15", "content": "Thank Kaggle and the host to organize the competition. I want to do a quick write-up of my simple solution. Everything is basic, all steps can be done using Kaggle and Google Colab (Pro) computing resources.
\nFinal Submission notebook: https://www.kaggle.com/code/namgalielei/benetech-eval-and-infer-v12/notebook
\nChart type classification + Plot BBox prediction -> OCR -> Chart data instance detection / segmentation (line) -> Data association
\nSome thing not having time to try yet: External dataset, a wider range of instance segmentation / object detection models, Generative models
", "rawMarkdown": "Thank Kaggle and the host to organize the competition. I want to do a quick write-up of my simple solution. Everything is basic, all steps can be done using Kaggle and Google Colab (Pro) computing resources.\n\nFinal Submission notebook: https://www.kaggle.com/code/namgalielei/benetech-eval-and-infer-v12/notebook\n\n## 1. Overview\nChart type classification + Plot BBox prediction -> OCR -> Chart data instance detection / segmentation (line) -> Data association \n\n## 2. Modules\n### a. Chart type classification + Plot BBox prediction:\n- Segmentation model pytorch Unet with backbone eca nfnet l1, auxilary classification head. I do binary segmentation for Plot BBox prediction, and softmax classifcation for Chart type classification \n- Ensemble 3 folds.\nTraining code: https://colab.research.google.com/drive/1WdlqUr1ONcntWfaIksDp6Z6BD4MXDJg4?usp=sharing\n### b. OCR:\n- Based on Paddle ppOCRv3\n+ Text detection: Fine tune the light weighted mobilenetv3-dbnet on the competition's data. The target is at word group level. The ground-truth for this is provided on the json file.\n+ Text recognition: Use the pretrained model. \n+ Modify some logic: First, the default post processing of Paddle TextDet (mask to quad) may result in some polygons with only 3 points (triangle instead of quadrilateral). I fixed this by finding the minimum bounding rotated rectangle. Second, when perspective-transform a cropped text, it might be rotated, so the text recognition is not able to read them well. I predict twice (no rotation and clock-wise 90deg rotation) and take the one with higher confidence score. \n### c. Chart data detection: \n- VBar, HBar, Scatter and Dot detector: Mask RCNN Resnet50 model. Training code: https://colab.research.google.com/drive/14X97mTwAU9kxS__xaehDU7SQbe_2ivP0?usp=sharing\n- Line detector: a single Unet++ model with backbone efficientnet b5. Training code: https://drive.google.com/file/d/1OFoogWWWP3vHAHe9g0UsY0cCfiHKOeMp/view?usp=sharing\nI annotated some images and train these models.\n### d. Data association:\n- Some rule-based logic to associate the chart data (bar, scatter, dot) to its corresponding ticks. \n- Project the pixel coordinates of ticks and chart data onto the axes-relative scale.\n\nSome thing not having time to try yet: External dataset, a wider range of instance segmentation / object detection models, Generative models", "votes": null }, { "id": "2309960", "postDate": "06/20/2023 02:46:38", "content": "@namgalielei Nice solution, can you tell me the reason why you use \"eca_nfnet_l1\" backbone?. It's not famous and popular.
", "rawMarkdown": "namgalielei Nice solution, can you tell me the reason why you use \"eca_nfnet_l1\" backbone?. It's not famous and popular.", "votes": null }, { "id": "2309974", "postDate": "06/20/2023 02:59:09", "content": "I have used it several times in the past and it showed good result. The architecture does not use batch norm which has some limitation. However, if you have time, you can try various model types in timm package and see what works with your data
", "rawMarkdown": "I have used it several times in the past and it showed good result. The architecture does not use batch norm which has some limitation. However, if you have time, you can try various model types in timm package and see what works with your data", "votes": null }, { "id": "2309992", "postDate": "06/20/2023 03:42:30", "content": "Thank you. when everyone use unet architecture, they often use se-resnext, efficientnet or some popular architectures,
\nI'm so surprised when you use an old and strange architecture backbone.
@namgalielei cool solution! Thanks for sharing notebook!
", "rawMarkdown": "namgalielei cool solution! Thanks for sharing notebook!", "votes": null }, { "id": "2310361", "postDate": "06/20/2023 10:03:53", "content": "Thank you for sharing your solution @namgalielei and congrats on your strong finish! That's a really good pipeline to study CV in general. A lot of things included. Also thank you for sharing your training code!
", "rawMarkdown": "Thank you for sharing your solution @namgalielei and congrats on your strong finish! That's a really good pipeline to study CV in general. A lot of things included. Also thank you for sharing your training code!", "votes": null }, { "id": "2315075", "postDate": "06/23/2023 19:45:16", "content": "Thanks for sharing.
\nHow did you generate segmentation mask for line plots?
i am surprised about that as well. I should also give these less popular models a try.
", "rawMarkdown": "i am surprised about that as well. I should also give these less popular models a try.", "votes": null }, { "id": "2317247", "postDate": "06/25/2023 14:32:56", "content": "I use semi-auto mask generation. First, I notice that by using Opencv image processing, I can group out those pixels belong to the line with some heuristic thresholds (binarization), then use cv2.contours to retrieve those polygons of that line. Second, there are some lines which cannot be extracted automatically, I manually annotate them.
", "rawMarkdown": "I use semi-auto mask generation. First, I notice that by using Opencv image processing, I can group out those pixels belong to the line with some heuristic thresholds (binarization), then use cv2.contours to retrieve those polygons of that line. Second, there are some lines which cannot be extracted automatically, I manually annotate them.", "votes": null } ], "comments": [ { "id": 2309960, "author_name": "hidngnguyna", "author_url": "", "post_date": "06/20/2023 02:46:38", "content": "@namgalielei Nice solution, can you tell me the reason why you use \"eca_nfnet_l1\" backbone?. It's not famous and popular.
", "votes": null, "replies": [ { "id": 2309974, "author_name": "namgalielei", "author_url": "", "post_date": "06/20/2023 02:59:09", "content": "I have used it several times in the past and it showed good result. The architecture does not use batch norm which has some limitation. However, if you have time, you can try various model types in timm package and see what works with your data
", "votes": null, "replies": [ { "id": 2309992, "author_name": "hidngnguyna", "author_url": "", "post_date": "06/20/2023 03:42:30", "content": "Thank you. when everyone use unet architecture, they often use se-resnext, efficientnet or some popular architectures,
\nI'm so surprised when you use an old and strange architecture backbone.
i am surprised about that as well. I should also give these less popular models a try.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2310097, "author_name": "serangu", "author_url": "", "post_date": "06/20/2023 05:32:21", "content": "@namgalielei cool solution! Thanks for sharing notebook!
", "votes": null, "replies": [] }, { "id": 2310361, "author_name": "snnclsr", "author_url": "", "post_date": "06/20/2023 10:03:53", "content": "Thank you for sharing your solution @namgalielei and congrats on your strong finish! That's a really good pipeline to study CV in general. A lot of things included. Also thank you for sharing your training code!
", "votes": null, "replies": [] }, { "id": 2315075, "author_name": "tanjiroll", "author_url": "", "post_date": "06/23/2023 19:45:16", "content": "Thanks for sharing.
\nHow did you generate segmentation mask for line plots?
I use semi-auto mask generation. First, I notice that by using Opencv image processing, I can group out those pixels belong to the line with some heuristic thresholds (binarization), then use cv2.contours to retrieve those polygons of that line. Second, there are some lines which cannot be extracted automatically, I manually annotate them.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2309931": "Thank Kaggle and the host to organize the competition. I want to do a quick write-up of my simple solution. Everything is basic, all steps can be done using Kaggle and Google Colab (Pro) computing resources.\n\nFinal Submission notebook: https://www.kaggle.com/code/namgalielei/benetech-eval-and-infer-v12/notebook\n\n## 1. Overview\nChart type classification + Plot BBox prediction -> OCR -> Chart data instance detection / segmentation (line) -> Data association \n\n## 2. Modules\n### a. Chart type classification + Plot BBox prediction:\n- Segmentation model pytorch Unet with backbone eca nfnet l1, auxilary classification head. I do binary segmentation for Plot BBox prediction, and softmax classifcation for Chart type classification \n- Ensemble 3 folds.\nTraining code: https://colab.research.google.com/drive/1WdlqUr1ONcntWfaIksDp6Z6BD4MXDJg4?usp=sharing\n### b. OCR:\n- Based on Paddle ppOCRv3\n+ Text detection: Fine tune the light weighted mobilenetv3-dbnet on the competition's data. The target is at word group level. The ground-truth for this is provided on the json file.\n+ Text recognition: Use the pretrained model. \n+ Modify some logic: First, the default post processing of Paddle TextDet (mask to quad) may result in some polygons with only 3 points (triangle instead of quadrilateral). I fixed this by finding the minimum bounding rotated rectangle. Second, when perspective-transform a cropped text, it might be rotated, so the text recognition is not able to read them well. I predict twice (no rotation and clock-wise 90deg rotation) and take the one with higher confidence score. \n### c. Chart data detection: \n- VBar, HBar, Scatter and Dot detector: Mask RCNN Resnet50 model. Training code: https://colab.research.google.com/drive/14X97mTwAU9kxS__xaehDU7SQbe_2ivP0?usp=sharing\n- Line detector: a single Unet++ model with backbone efficientnet b5. Training code: https://drive.google.com/file/d/1OFoogWWWP3vHAHe9g0UsY0cCfiHKOeMp/view?usp=sharing\nI annotated some images and train these models.\n### d. Data association:\n- Some rule-based logic to associate the chart data (bar, scatter, dot) to its corresponding ticks. \n- Project the pixel coordinates of ticks and chart data onto the axes-relative scale.\n\nSome thing not having time to try yet: External dataset, a wider range of instance segmentation / object detection models, Generative models", "2309960": "namgalielei Nice solution, can you tell me the reason why you use \"eca_nfnet_l1\" backbone?. It's not famous and popular.", "2309974": "I have used it several times in the past and it showed good result. The architecture does not use batch norm which has some limitation. However, if you have time, you can try various model types in timm package and see what works with your data", "2309992": "Thank you. when everyone use unet architecture, they often use se-resnext, efficientnet or some popular architectures,\nI'm so surprised when you use an old and strange architecture backbone.", "2310097": "namgalielei cool solution! Thanks for sharing notebook!", "2310361": "Thank you for sharing your solution @namgalielei and congrats on your strong finish! That's a really good pipeline to study CV in general. A lot of things included. Also thank you for sharing your training code!", "2315075": "Thanks for sharing.\nHow did you generate segmentation mask for line plots?", "2317199": "i am surprised about that as well. I should also give these less popular models a try.", "2317247": "I use semi-auto mask generation. First, I notice that by using Opencv image processing, I can group out those pixels belong to the line with some heuristic thresholds (binarization), then use cv2.contours to retrieve those polygons of that line. Second, there are some lines which cannot be extracted automatically, I manually annotate them." }, "source": "existing_discussion", "complete": true, "competition_id": 43873, "competition_slug": "benetech-making-graphs-accessible", "team_id": 10116840, "solution_url": "/competitions/benetech-making-graphs-accessible/writeups/the-nam-13th-place-solution" }