![\"cls\"](\"https://i.ibb.co/84T42Hv/cls.png\")
First of all, congrats to all the participants especially to those that managed to stick it to the end. This was overall a really interesting challenge, and we really enjoyed tackling it. Anyone in 0.86+ public scores could've taken the win so we're glad we dropped only 2 spots.
\nThanks to @crodoc for the great teaming experience, we had two different opinions on how to tackle the problem : he went for and end-to-end approach whereas I wanted to do object detection & OCR. This proved to be ideal for this competition as object detection compensated for end-to-end weaknesses.
\n
Our solution is a two-step pipeline, where the first step is a simple classification task and the second step solves the task for the different chart types. For scatter and dot plots, we used a detection approach. For line and bar charts, Matcha was really strong. Here are our LB scores :
\n| \n | Overall | \nScatter | \nDots | \nLine | \nHorizontal Bar | \nVertical Bar | \n
|---|---|---|---|---|---|---|
| Public | \n0.87 | \n0.09 | \n0 | \n0.33 | \n0.04 | \n0.39 | \n
| Private | \n0.71 | \n0.28 | \n0.01 | \n0.13 | \n0.01 | \n0.27 | \n
From the description page :
\n\n\nThe extracted figures in the training and public test sets are drawn from the same set of sources. The figures in the private test set are drawn from a distinct set of sources.
\n
Public LB is leaky, we need to create models that generalize to new data sources. Therefore, we use all the extracted data as validation, and train only on generated data. Data provided by the hosts is not diverse enough to allow for generalization so we generated our own. Matcha and classification models were fullfit on all the data after careful parameter optimization, but detection models were not trained on any extracted data.
\nFor validating on dots, we used a small curated dataset of 20 images found on google images. We estimated via probing that there are between 100 and 125 dots in the test set, all of them most likely being in the private test set. We did not probe for scatter though, this would've been helpful.
\nNothing too fancy here, we trained models on (benetech + theo + crodoc) generated data.
\nMain parameters:
\n3e-4 or 5e-4 (2 lrs for blend, we also used 2 seeds).![\"scatter-pipe\"](\"https://i.ibb.co/ct3TgvR/scatter-pipe.png\")
For scatter, we relied on YoloX to detect all markers. Cached is used to take care of the other useful elements. If all the points are detected correctly, it’s not that complicated to infer the target : detect the ticks & labels, read the labels, and interpolate !
\nMore details:
\n![\"dot-pipe\"](\"https://i.ibb.co/TwFzdjS/dot-pipe.png\")
Dot pipeline is similar to scatter, but a bit more simple. We detect the points and cluster them, and map them to the detected x-labels. Labels with no assigned cluster are given the target 0, others the number of detected points. Counting points is not really robust to detection mistakes, so instead we used the height of the uppermost point and interpolated.
\nMatcha was really powerful here. We used matcha-base and set is_vqa=False to avoid giving texts as input to the model.
We trained Matcha to predict chart type, xs and ys for an image. The ground truth looked the same as @nbroad used for his donut approach (except we dropped the prompt token). We tried other approaches, but this worked best :
\n = X_START + .join(list(map(str, xs))) + X_END\n = Y_START + .join(list(map(str, ys))) + Y_END\n = + chart_type + + x_str + y_str\ne.g: <line><x_start>0;2;4;6<x_end><y_start>2.7;2.2;3.6;5.2;<y_end>
The most valuable boost we got by generating additional charts using matplotlib. We reused the values and texts from the train dataset to generate ticks & values, with different styles/patterns/fonts/colors for diversity. The code for generating additional images has around 1000 lines and basically covers most cases where the model was failing when validated on the \"extracted\" dataset (e.g. negative values, line edges, missing bars, multiline text, text rotations).
\nOther things that helped :
\nnumber_of_decimals = max(0, round(np.log10(1/ (max(y_ticks) - min(y_ticks))) + 3)More details:
\nWe were greatly perturbed by the unexpected second rule change, especially considering the fact that I was on vacation. Matcha and classification models were retrained on ICDAR during the last 3 days, this gave a 0.01 public LB boost but nothing on private.
\nThanks for reading !
", "messages": [ { "id": "2310638", "postDate": "06/20/2023 14:15:42", "content": "First of all, congrats to all the participants especially to those that managed to stick it to the end. This was overall a really interesting challenge, and we really enjoyed tackling it. Anyone in 0.86+ public scores could've taken the win so we're glad we dropped only 2 spots.
\nThanks to @crodoc for the great teaming experience, we had two different opinions on how to tackle the problem : he went for and end-to-end approach whereas I wanted to do object detection & OCR. This proved to be ideal for this competition as object detection compensated for end-to-end weaknesses.
\n
Our solution is a two-step pipeline, where the first step is a simple classification task and the second step solves the task for the different chart types. For scatter and dot plots, we used a detection approach. For line and bar charts, Matcha was really strong. Here are our LB scores :
\n| \n | Overall | \nScatter | \nDots | \nLine | \nHorizontal Bar | \nVertical Bar | \n
|---|---|---|---|---|---|---|
| Public | \n0.87 | \n0.09 | \n0 | \n0.33 | \n0.04 | \n0.39 | \n
| Private | \n0.71 | \n0.28 | \n0.01 | \n0.13 | \n0.01 | \n0.27 | \n
From the description page :
\n\n\nThe extracted figures in the training and public test sets are drawn from the same set of sources. The figures in the private test set are drawn from a distinct set of sources.
\n
Public LB is leaky, we need to create models that generalize to new data sources. Therefore, we use all the extracted data as validation, and train only on generated data. Data provided by the hosts is not diverse enough to allow for generalization so we generated our own. Matcha and classification models were fullfit on all the data after careful parameter optimization, but detection models were not trained on any extracted data.
\nFor validating on dots, we used a small curated dataset of 20 images found on google images. We estimated via probing that there are between 100 and 125 dots in the test set, all of them most likely being in the private test set. We did not probe for scatter though, this would've been helpful.
\nNothing too fancy here, we trained models on (benetech + theo + crodoc) generated data.
\nMain parameters:
\n3e-4 or 5e-4 (2 lrs for blend, we also used 2 seeds).
For scatter, we relied on YoloX to detect all markers. Cached is used to take care of the other useful elements. If all the points are detected correctly, it’s not that complicated to infer the target : detect the ticks & labels, read the labels, and interpolate !
\nMore details:
\n
Dot pipeline is similar to scatter, but a bit more simple. We detect the points and cluster them, and map them to the detected x-labels. Labels with no assigned cluster are given the target 0, others the number of detected points. Counting points is not really robust to detection mistakes, so instead we used the height of the uppermost point and interpolated.
\nMatcha was really powerful here. We used matcha-base and set is_vqa=False to avoid giving texts as input to the model.
We trained Matcha to predict chart type, xs and ys for an image. The ground truth looked the same as @nbroad used for his donut approach (except we dropped the prompt token). We tried other approaches, but this worked best :
\n = X_START + .join(list(map(str, xs))) + X_END\n = Y_START + .join(list(map(str, ys))) + Y_END\n = + chart_type + + x_str + y_str\ne.g: <line><x_start>0;2;4;6<x_end><y_start>2.7;2.2;3.6;5.2;<y_end>
The most valuable boost we got by generating additional charts using matplotlib. We reused the values and texts from the train dataset to generate ticks & values, with different styles/patterns/fonts/colors for diversity. The code for generating additional images has around 1000 lines and basically covers most cases where the model was failing when validated on the \"extracted\" dataset (e.g. negative values, line edges, missing bars, multiline text, text rotations).
\nOther things that helped :
\nnumber_of_decimals = max(0, round(np.log10(1/ (max(y_ticks) - min(y_ticks))) + 3)More details:
\nWe were greatly perturbed by the unexpected second rule change, especially considering the fact that I was on vacation. Matcha and classification models were retrained on ICDAR during the last 3 days, this gave a 0.01 public LB boost but nothing on private.
\nThanks for reading !
", "rawMarkdown": "First of all, congrats to all the participants especially to those that managed to stick it to the end. This was overall a really interesting challenge, and we really enjoyed tackling it. Anyone in 0.86+ public scores could've taken the win so we're glad we dropped only 2 spots.\n\nThanks to @crodoc for the great teaming experience, we had two different opinions on how to tackle the problem : he went for and end-to-end approach whereas I wanted to do object detection & OCR. This proved to be ideal for this competition as object detection compensated for end-to-end weaknesses.\n\n## Overview\n\n\n\nOur solution is a two-step pipeline, where the first step is a simple classification task and the second step solves the task for the different chart types. For scatter and dot plots, we used a detection approach. For line and bar charts, Matcha was really strong. Here are our LB scores :\n\n| | Overall | Scatter | Dots | Line | Horizontal Bar | Vertical Bar|\n| --- |\n| **Public** | 0.87 | 0.09 | 0| 0.33 | 0.04| 0.39|\n| **Private** | 0.71 | 0.28 | 0.01| 0.13 | 0.01 | 0.27 |\n\n## Validation & Insights\n\nFrom the description page : \n> The extracted figures in the training and public test sets are drawn from the same set of sources. The figures in the private test set are drawn from a distinct set of sources.\n\nPublic LB is leaky, we need to create models that generalize to new data sources. Therefore, we use all the extracted data as validation, and train only on generated data. Data provided by the hosts is not diverse enough to allow for generalization so we generated our own. Matcha and classification models were fullfit on all the data after careful parameter optimization, but detection models were not trained on any extracted data.\n\nFor validating on dots, we used a small curated dataset of 20 images found on google images. We estimated via probing that there are between 100 and 125 dots in the test set, all of them most likely being in the private test set. We did not probe for scatter though, this would've been helpful. \n\n## Step 1 - Classification\n\nNothing too fancy here, we trained models on (benetech + theo + crodoc) generated data.\n\n**Main parameters:**\n- 2 epochs on 88k images. \n- lr `3e-4` or `5e-4` (2 lrs for blend, we also used 2 seeds).\n- Mixup and some color augmentations mostly.\n- 256x384 image size.\n- NfNet-l2 with 0.2 dropout.\n\n## Step 2.a - Scatter\n\n\n\nFor scatter, we relied on [YoloX](https://github.com/Megvii-BaseDetection/YOLOX) to detect all markers. Cached is used to take care of the other useful elements. If all the points are detected correctly, it’s not that complicated to infer the target : detect the ticks & labels, read the labels, and interpolate !\n\n**More details:**\n- Ensemble YoloX-m and YoloX-l with NMS, helps reduce the number of FNs.\n- Models are trained on scatter + dot generated benetech data + a bunch of plots I generated and pseudo labeled, for 10 epochs.\n- CV 0.67, public LB 0.09~, private 0.29 - performance drop almost exclusively comes from overlapping/too hard to detect markers.\n- Bunch of post-processing to make the pipeline more robust to OCR mistakes and detection mistakes.\n- We were initially using Yolo-v7, but had to switch to YoloX because of the first rule change. It took us a week to match Yolo-v7 performance with YoloX.\n\n## Step 2.b - Dots\n\n\n\nDot pipeline is similar to scatter, but a bit more simple. We detect the points and cluster them, and map them to the detected x-labels. Labels with no assigned cluster are given the target 0, others the number of detected points. Counting points is not really robust to detection mistakes, so instead we used the height of the uppermost point and interpolated.\n\n## Step 2.c - Bars & Lines\n\nMatcha was really powerful here. We used `matcha-base` and set `is_vqa=False` to avoid giving texts as input to the model.\n\nWe trained Matcha to predict chart type, xs and ys for an image. The ground truth looked the same as @nbroad used for his donut approach (except we dropped the prompt token). We tried other approaches, but this worked best :\n```\nx_str = X_START + \";\".join(list(map(str, xs))) + X_END\ny_str = Y_START + \";\".join(list(map(str, ys))) + Y_END\nground_truth = '<' + chart_type + '>' + x_str + y_str\n```\ne.g: `Our inference code is here : https://www.kaggle.com/code/crodoc/benetech-mit-ensemble?scriptVersionId=134055662
\nWe will be sharing and open-sourcing the rest of our solution in the upcoming days.
We are prize eligible, but I still have not understood which prize we will get :)
", "rawMarkdown": "Our inference code is here : https://www.kaggle.com/code/crodoc/benetech-mit-ensemble?scriptVersionId=134055662\nWe will be sharing and open-sourcing the rest of our solution in the upcoming days. \n\nWe are prize eligible, but I still have not understood which prize we will get :)", "votes": null }, { "id": "2310664", "postDate": "06/20/2023 14:29:14", "content": "This is really cool strategy, and thanks for the nice write-up as well.
\nCongratulations.
Great approach, definetly good ideas embedded. Congratulations becoming 3rd place!
", "rawMarkdown": "Great approach, definetly good ideas embedded. Congratulations becoming 3rd place!", "votes": null }, { "id": "2311031", "postDate": "06/20/2023 20:50:13", "content": "\n\nAnyone in 0.86+ public scores could've taken the win
\n
I second that.
\nGreat work and interesting solution @theoviel and @crodoc
", "rawMarkdown": ">Anyone in 0.86+ public scores could've taken the win\n\nI second that. \n\nGreat work and interesting solution @theoviel and @crodoc", "votes": null }, { "id": "2311039", "postDate": "06/20/2023 21:12:10", "content": "Yea. We all expected a shake-up in the last digits.
\nWhere we failed was not probing for the number of all chart types in the test set. We just probed dots 🤦♂️ then scatter slaps you on private 🤣
", "rawMarkdown": "Yea. We all expected a shake-up in the last digits.\n\nWhere we failed was not probing for the number of all chart types in the test set. We just probed dots 🤦♂️ then scatter slaps you on private 🤣", "votes": null }, { "id": "2311489", "postDate": "06/21/2023 07:29:30", "content": "Congrats, great work. How could you get the score per chart type in public and private?
", "rawMarkdown": "Congrats, great work. How could you get the score per chart type in public and private?", "votes": null }, { "id": "2311519", "postDate": "06/21/2023 07:55:06", "content": "Congrats on 3rd place, I learned a lot from reading your solution!
", "rawMarkdown": "Congrats on 3rd place, I learned a lot from reading your solution!", "votes": null }, { "id": "2311522", "postDate": "06/21/2023 07:56:27", "content": "Thanks @thedrcat. Nice to hear from an old teamate :)
", "rawMarkdown": "Thanks @thedrcat. Nice to hear from an old teamate :)", "votes": null }, { "id": "2311526", "postDate": "06/21/2023 07:59:56", "content": "To get score for scatter, replace all non-scatter rows with: ';'.join(['0']*200)
Ok, thanks. I got it.
", "rawMarkdown": "Ok, thanks. I got it.", "votes": null }, { "id": "2311542", "postDate": "06/21/2023 08:26:37", "content": "Congratulations for third place!!! @crodoc
", "rawMarkdown": "Congratulations for third place!!! @crodoc", "votes": null }, { "id": "2311577", "postDate": "06/21/2023 08:49:04", "content": "Thank you @akshitsharma1 :)
", "rawMarkdown": "Thank you @akshitsharma1 :)", "votes": null }, { "id": "2311830", "postDate": "06/21/2023 13:28:04", "content": "Congrats @theoviel and @crodoc !
\nAnd thanks for the overview (nice graphs as always) !
", "rawMarkdown": "Congrats @theoviel and @crodoc !\n\nAnd thanks for the overview (nice graphs as always) !", "votes": null }, { "id": "2311893", "postDate": "06/21/2023 14:11:46", "content": "Thanks Vincent :) Theo is the one making nice graphs :D
", "rawMarkdown": "Thanks Vincent :) Theo is the one making nice graphs :D", "votes": null }, { "id": "2312016", "postDate": "06/21/2023 15:52:11", "content": "The third-place solution showcases a thoughtful and effective approach, combining classification and object detection techniques to tackle the competition task. Their strategy for generalization and careful parameter optimization contributed to their impressive results. Well done!
", "rawMarkdown": "The third-place solution showcases a thoughtful and effective approach, combining classification and object detection techniques to tackle the competition task. Their strategy for generalization and careful parameter optimization contributed to their impressive results. Well done!", "votes": null }, { "id": "2315067", "postDate": "06/23/2023 19:37:10", "content": "Congrats on your gold.
\nDid you notice improvement using Augs? Also, what do you mean by \"model soup of all checkpoints\"?
Congrats @theoviel and @crodoc !
\nAnd thanks for this wonderful overview !
", "rawMarkdown": "Congrats @theoviel and @crodoc !\n\nAnd thanks for this wonderful overview !", "votes": null }, { "id": "2315868", "postDate": "06/24/2023 13:36:11", "content": "Great work :) I feel I learned a lot reading this through. I have a few questions. If you or someone else could respond, I'd be grateful. I'm a newbie though, so they might be trivial.
\nDid you guys have any issues with inference computation time using matcha? I tried it and a few models like it, and running a prediction took 10s of seconds for one single plot with kaggle GPUs. So I figured an inference notebook would exceed the time limit. Same question goes for training.
What is this \"cached\" that you ensembled with YoloX?
As the model gets better, inference time decreases. At the beginning of training, the model produces a lot of tokens (up to max length, which we set to 512). This is slow. It needs to learn to predict the end token and stop generating more tokens. This is why we always skipped doing validation in the first epoch.
\nCached is this. And this is a good notebook by @junkoda.
", "rawMarkdown": "As the model gets better, inference time decreases. At the beginning of training, the model produces a lot of tokens (up to max length, which we set to 512). This is slow. It needs to learn to predict the end token and stop generating more tokens. This is why we always skipped doing validation in the first epoch.\n\nCached is [this](https://arxiv.org/pdf/2305.04151.pdf). And [this](https://www.kaggle.com/code/junkoda/cached-chart-element-detection) is a good notebook by @junkoda.", "votes": null }, { "id": "2319402", "postDate": "06/27/2023 05:31:24", "content": "@theoviel congrats with 3 place! Nice graphics!
", "rawMarkdown": "theoviel congrats with 3 place! Nice graphics!", "votes": null } ], "comments": [ { "id": 2310655, "author_name": "theoviel", "author_url": "", "post_date": "06/20/2023 14:23:37", "content": "Our inference code is here : https://www.kaggle.com/code/crodoc/benetech-mit-ensemble?scriptVersionId=134055662
\nWe will be sharing and open-sourcing the rest of our solution in the upcoming days.
We are prize eligible, but I still have not understood which prize we will get :)
", "votes": null, "replies": [] }, { "id": 2310664, "author_name": "cooleel", "author_url": "", "post_date": "06/20/2023 14:29:14", "content": "This is really cool strategy, and thanks for the nice write-up as well.
\nCongratulations.
Great approach, definetly good ideas embedded. Congratulations becoming 3rd place!
", "votes": null, "replies": [] }, { "id": 2311031, "author_name": "christofhenkel", "author_url": "", "post_date": "06/20/2023 20:50:13", "content": "\n\nAnyone in 0.86+ public scores could've taken the win
\n
I second that.
\nGreat work and interesting solution @theoviel and @crodoc
", "votes": null, "replies": [ { "id": 2311039, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 21:12:10", "content": "Yea. We all expected a shake-up in the last digits.
\nWhere we failed was not probing for the number of all chart types in the test set. We just probed dots 🤦♂️ then scatter slaps you on private 🤣
", "votes": null, "replies": [] } ] }, { "id": 2311489, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/21/2023 07:29:30", "content": "Congrats, great work. How could you get the score per chart type in public and private?
", "votes": null, "replies": [ { "id": 2311526, "author_name": "crodoc", "author_url": "", "post_date": "06/21/2023 07:59:56", "content": "To get score for scatter, replace all non-scatter rows with: ';'.join(['0']*200)
Ok, thanks. I got it.
", "votes": null, "replies": [] } ] } ] }, { "id": 2311519, "author_name": "thedrcat", "author_url": "", "post_date": "06/21/2023 07:55:06", "content": "Congrats on 3rd place, I learned a lot from reading your solution!
", "votes": null, "replies": [ { "id": 2311522, "author_name": "crodoc", "author_url": "", "post_date": "06/21/2023 07:56:27", "content": "Thanks @thedrcat. Nice to hear from an old teamate :)
", "votes": null, "replies": [] } ] }, { "id": 2311542, "author_name": "akshitsharma1", "author_url": "", "post_date": "06/21/2023 08:26:37", "content": "Congratulations for third place!!! @crodoc
", "votes": null, "replies": [ { "id": 2311577, "author_name": "crodoc", "author_url": "", "post_date": "06/21/2023 08:49:04", "content": "Thank you @akshitsharma1 :)
", "votes": null, "replies": [] } ] }, { "id": 2311830, "author_name": "vincentschuler", "author_url": "", "post_date": "06/21/2023 13:28:04", "content": "Congrats @theoviel and @crodoc !
\nAnd thanks for the overview (nice graphs as always) !
", "votes": null, "replies": [ { "id": 2311893, "author_name": "crodoc", "author_url": "", "post_date": "06/21/2023 14:11:46", "content": "Thanks Vincent :) Theo is the one making nice graphs :D
", "votes": null, "replies": [] } ] }, { "id": 2312016, "author_name": "poojach7611", "author_url": "", "post_date": "06/21/2023 15:52:11", "content": "The third-place solution showcases a thoughtful and effective approach, combining classification and object detection techniques to tackle the competition task. Their strategy for generalization and careful parameter optimization contributed to their impressive results. Well done!
", "votes": null, "replies": [] }, { "id": 2315067, "author_name": "tanjiroll", "author_url": "", "post_date": "06/23/2023 19:37:10", "content": "Congrats on your gold.
\nDid you notice improvement using Augs? Also, what do you mean by \"model soup of all checkpoints\"?
Congrats @theoviel and @crodoc !
\nAnd thanks for this wonderful overview !
", "votes": null, "replies": [] }, { "id": 2315868, "author_name": "esedx12", "author_url": "", "post_date": "06/24/2023 13:36:11", "content": "Great work :) I feel I learned a lot reading this through. I have a few questions. If you or someone else could respond, I'd be grateful. I'm a newbie though, so they might be trivial.
\nDid you guys have any issues with inference computation time using matcha? I tried it and a few models like it, and running a prediction took 10s of seconds for one single plot with kaggle GPUs. So I figured an inference notebook would exceed the time limit. Same question goes for training.
What is this \"cached\" that you ensembled with YoloX?
As the model gets better, inference time decreases. At the beginning of training, the model produces a lot of tokens (up to max length, which we set to 512). This is slow. It needs to learn to predict the end token and stop generating more tokens. This is why we always skipped doing validation in the first epoch.
\nCached is this. And this is a good notebook by @junkoda.
", "votes": null, "replies": [] } ] }, { "id": 2319402, "author_name": "serangu", "author_url": "", "post_date": "06/27/2023 05:31:24", "content": "@theoviel congrats with 3 place! Nice graphics!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2310638": "First of all, congrats to all the participants especially to those that managed to stick it to the end. This was overall a really interesting challenge, and we really enjoyed tackling it. Anyone in 0.86+ public scores could've taken the win so we're glad we dropped only 2 spots.\n\nThanks to @crodoc for the great teaming experience, we had two different opinions on how to tackle the problem : he went for and end-to-end approach whereas I wanted to do object detection & OCR. This proved to be ideal for this competition as object detection compensated for end-to-end weaknesses.\n\n## Overview\n\n\n\nOur solution is a two-step pipeline, where the first step is a simple classification task and the second step solves the task for the different chart types. For scatter and dot plots, we used a detection approach. For line and bar charts, Matcha was really strong. Here are our LB scores :\n\n| | Overall | Scatter | Dots | Line | Horizontal Bar | Vertical Bar|\n| --- |\n| **Public** | 0.87 | 0.09 | 0| 0.33 | 0.04| 0.39|\n| **Private** | 0.71 | 0.28 | 0.01| 0.13 | 0.01 | 0.27 |\n\n## Validation & Insights\n\nFrom the description page : \n> The extracted figures in the training and public test sets are drawn from the same set of sources. The figures in the private test set are drawn from a distinct set of sources.\n\nPublic LB is leaky, we need to create models that generalize to new data sources. Therefore, we use all the extracted data as validation, and train only on generated data. Data provided by the hosts is not diverse enough to allow for generalization so we generated our own. Matcha and classification models were fullfit on all the data after careful parameter optimization, but detection models were not trained on any extracted data.\n\nFor validating on dots, we used a small curated dataset of 20 images found on google images. We estimated via probing that there are between 100 and 125 dots in the test set, all of them most likely being in the private test set. We did not probe for scatter though, this would've been helpful. \n\n## Step 1 - Classification\n\nNothing too fancy here, we trained models on (benetech + theo + crodoc) generated data.\n\n**Main parameters:**\n- 2 epochs on 88k images. \n- lr `3e-4` or `5e-4` (2 lrs for blend, we also used 2 seeds).\n- Mixup and some color augmentations mostly.\n- 256x384 image size.\n- NfNet-l2 with 0.2 dropout.\n\n## Step 2.a - Scatter\n\n\n\nFor scatter, we relied on [YoloX](https://github.com/Megvii-BaseDetection/YOLOX) to detect all markers. Cached is used to take care of the other useful elements. If all the points are detected correctly, it’s not that complicated to infer the target : detect the ticks & labels, read the labels, and interpolate !\n\n**More details:**\n- Ensemble YoloX-m and YoloX-l with NMS, helps reduce the number of FNs.\n- Models are trained on scatter + dot generated benetech data + a bunch of plots I generated and pseudo labeled, for 10 epochs.\n- CV 0.67, public LB 0.09~, private 0.29 - performance drop almost exclusively comes from overlapping/too hard to detect markers.\n- Bunch of post-processing to make the pipeline more robust to OCR mistakes and detection mistakes.\n- We were initially using Yolo-v7, but had to switch to YoloX because of the first rule change. It took us a week to match Yolo-v7 performance with YoloX.\n\n## Step 2.b - Dots\n\n\n\nDot pipeline is similar to scatter, but a bit more simple. We detect the points and cluster them, and map them to the detected x-labels. Labels with no assigned cluster are given the target 0, others the number of detected points. Counting points is not really robust to detection mistakes, so instead we used the height of the uppermost point and interpolated.\n\n## Step 2.c - Bars & Lines\n\nMatcha was really powerful here. We used `matcha-base` and set `is_vqa=False` to avoid giving texts as input to the model.\n\nWe trained Matcha to predict chart type, xs and ys for an image. The ground truth looked the same as @nbroad used for his donut approach (except we dropped the prompt token). We tried other approaches, but this worked best :\n```\nx_str = X_START + \";\".join(list(map(str, xs))) + X_END\ny_str = Y_START + \";\".join(list(map(str, ys))) + Y_END\nground_truth = '<' + chart_type + '>' + x_str + y_str\n```\ne.g: `