![](\"https://i.imgur.com/SpheoRP.png\")
I hope you find the training and inference codes neat and tidy, I did my best effort to not make a mess!
\n\nHere is a brief explanation of our team solution with @cody11null.
\nOur solution consisted of a pipeline of object detectors (EfficientDet) and image-encoders/text-decoders (Matcha and Donut):
\nvertical_bar, horizontal_bar, line and dot) we used a Matcha model.
We train different image-encoder/text-decoder architectures to detect the axis labels. I trained a Donut model (which surprisingly got the best results) and two other Matcha models which got worse performance. Despite training with different max_patches results were rather similar for all. Most complications were associated to shared origins in the axis labels, long sequences, floats with many decimal places, etc. In this step, the most important is to get the minimum and the maximum values right for each axis so the cross multiplication is done right.
This was one of the most time consuming parts for me. At the beggining of the competition until Nicholas shared his Donut model I didn't know that image-encoder/text-decoder models could perform that well in this competition so I focused on an object detector model which could detect well data series. To do so I lack bounding boxes training data, so I started manually labelling thousands of images. These took me a lot of days. My EfficientDet object detector did extremely well on generated images but not as well on extracted. I labeled as many images as I could. My OD has ~50% exact matches accuracy for scatter plots. It performs much better for other chart types, but since scatter plots can have a lot of points it's reasonable to be a tougher task.
\nHere you can see some of my ODs predictions:
\n![](\"https://i.imgur.com/PQ1L3az.png\")
As part of the pipeline I had to detect the chart's bounding box so I could map pixel coordinates to real numerical values from the axis labels. This step was rather easy and its basically the same code from part (2) applied on the bounding boxes of plot-bb from the annotations. We had this training data available inside the JSONs files and it's a simple task so the OD achieves high accuracy.
At the beggining of the competition I started making several changes to Nicholas' Donut model and soon discovered that you could achieve higher performance by perfoming some basic postprocessing. However, even after training more epochs, perfoming data augmentation and other tricks I realised I couldn't achieve much higher performances. I then tried implementing a Matcha model and with a lot of effort I made it work thanks (again) to Nicholas which raised the issue of the GitHub discussion. Once I got the Matcha model working I tried a ton of different stuff until I could squeeze as much performance as I could. So here is a little roadmap to achieveing 0.74 in the public LB:
\nmax_patches of Matcha from 512 to 1024.So many things that didn't work! I will write the ones I can remember:
\nstatista, chartqa, plotqa. Best performance came from matcha-base.RGBShift, RandomBrightness, ColorJitter, etc. max_patches parameter to 1536 (halfway between 1024 and 2048). So, as I saw an improvement by increasing max_patches from 512 to 1024 I thought, why don't I increase it a bit more and see what happens? Well, it didn't seem to improve the score and of course consumed more VRAM and computation time. Increasing max_patches increases a lot the VRAM consumption so the batch size had to be lowered and training took longer.OneCycleLR but before I tried some other, even constant LR, and never saw an increase in performance by modifying the LR scheduler.tf_efficientnetv2_s as my OD's backbone, but tried larger models without match success like tf_efficientnetv2_l.bfloat16). However, since you use less VRAM compared to float32 you can train with larger batch sizes which is a recommended practice (I heard that Karpathy said that it's always the best to train with as larger batches as you can, I may be wrong here).It's barely impossible to train models like Matcha and Donut with the P100's provided by Kaggle. We ended up paying Google Colab Pro+ which provides A100 GPUs with 40 GB of VRAM since neither of us had DL GPUs like an RTX 3090/4090. With the A100 we were able to train large models and run many experiments faster. I trained more than 30 different models!
\nEven though I would have liked to get inside the gold zone (always hoping for the best) I am grateful for the result we obtained and mainly for all the lessons learned throughout the competition. I firmly believe that if you want to learn something try changing it! Code something from scratch! Best way to learn by far. I am really looking forward to read other teams solutions. I would also like to know if anyone tried changing the Vision Model and the Text Model from Pix2Struct. I don't know if it's even possible or how its done so if you know please leave in the comments.
\nThanks all!
", "messages": [ { "id": "2310035", "postDate": "06/20/2023 04:36:28", "content": "I hope you find the training and inference codes neat and tidy, I did my best effort to not make a mess!
\n\nHere is a brief explanation of our team solution with @cody11null.
\nOur solution consisted of a pipeline of object detectors (EfficientDet) and image-encoders/text-decoders (Matcha and Donut):
\nvertical_bar, horizontal_bar, line and dot) we used a Matcha model.
We train different image-encoder/text-decoder architectures to detect the axis labels. I trained a Donut model (which surprisingly got the best results) and two other Matcha models which got worse performance. Despite training with different max_patches results were rather similar for all. Most complications were associated to shared origins in the axis labels, long sequences, floats with many decimal places, etc. In this step, the most important is to get the minimum and the maximum values right for each axis so the cross multiplication is done right.
This was one of the most time consuming parts for me. At the beggining of the competition until Nicholas shared his Donut model I didn't know that image-encoder/text-decoder models could perform that well in this competition so I focused on an object detector model which could detect well data series. To do so I lack bounding boxes training data, so I started manually labelling thousands of images. These took me a lot of days. My EfficientDet object detector did extremely well on generated images but not as well on extracted. I labeled as many images as I could. My OD has ~50% exact matches accuracy for scatter plots. It performs much better for other chart types, but since scatter plots can have a lot of points it's reasonable to be a tougher task.
\nHere you can see some of my ODs predictions:
\n
As part of the pipeline I had to detect the chart's bounding box so I could map pixel coordinates to real numerical values from the axis labels. This step was rather easy and its basically the same code from part (2) applied on the bounding boxes of plot-bb from the annotations. We had this training data available inside the JSONs files and it's a simple task so the OD achieves high accuracy.
At the beggining of the competition I started making several changes to Nicholas' Donut model and soon discovered that you could achieve higher performance by perfoming some basic postprocessing. However, even after training more epochs, perfoming data augmentation and other tricks I realised I couldn't achieve much higher performances. I then tried implementing a Matcha model and with a lot of effort I made it work thanks (again) to Nicholas which raised the issue of the GitHub discussion. Once I got the Matcha model working I tried a ton of different stuff until I could squeeze as much performance as I could. So here is a little roadmap to achieveing 0.74 in the public LB:
\nmax_patches of Matcha from 512 to 1024.So many things that didn't work! I will write the ones I can remember:
\nstatista, chartqa, plotqa. Best performance came from matcha-base.RGBShift, RandomBrightness, ColorJitter, etc. max_patches parameter to 1536 (halfway between 1024 and 2048). So, as I saw an improvement by increasing max_patches from 512 to 1024 I thought, why don't I increase it a bit more and see what happens? Well, it didn't seem to improve the score and of course consumed more VRAM and computation time. Increasing max_patches increases a lot the VRAM consumption so the batch size had to be lowered and training took longer.OneCycleLR but before I tried some other, even constant LR, and never saw an increase in performance by modifying the LR scheduler.tf_efficientnetv2_s as my OD's backbone, but tried larger models without match success like tf_efficientnetv2_l.bfloat16). However, since you use less VRAM compared to float32 you can train with larger batch sizes which is a recommended practice (I heard that Karpathy said that it's always the best to train with as larger batches as you can, I may be wrong here).It's barely impossible to train models like Matcha and Donut with the P100's provided by Kaggle. We ended up paying Google Colab Pro+ which provides A100 GPUs with 40 GB of VRAM since neither of us had DL GPUs like an RTX 3090/4090. With the A100 we were able to train large models and run many experiments faster. I trained more than 30 different models!
\nEven though I would have liked to get inside the gold zone (always hoping for the best) I am grateful for the result we obtained and mainly for all the lessons learned throughout the competition. I firmly believe that if you want to learn something try changing it! Code something from scratch! Best way to learn by far. I am really looking forward to read other teams solutions. I would also like to know if anyone tried changing the Vision Model and the Text Model from Pix2Struct. I don't know if it's even possible or how its done so if you know please leave in the comments.
\nThanks all!
", "rawMarkdown": "# Code\n\nI hope you find the training and inference codes neat and tidy, I did my best effort to not make a mess!\n- [Matcha Training Code](https://www.kaggle.com/code/alejopaullier/benetech-matcha-train-0-74)\n- [Matcha Inference Code](https://www.kaggle.com/code/alejopaullier/benetech-matcha-inference-0-74)\n\n# TL;DR\n\nHere is a brief explanation of our team solution with @cody11null.\n\nOur solution consisted of a pipeline of object detectors (EfficientDet) and image-encoders/text-decoders (Matcha and Donut):\n1. A Donut model which acts like an OCR (Object Character Recognition) model is used to extract the axis labels of the chart (red)\n2. An object detector (EfficientDet) detects the data series (green). In the end, we only used this for scatter plots.\n3. A second object detector (EfficientDet) detects the plot's chart area (area where the data points are)(purple).\n4. Knowing the chart's limits from (3), the bounding boxes of the data series from (2) and the axis numerical values from (1) you can with a simple cross multiplication get the data series values. This approach is essentially as described in my [discussion](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/396773).\n5. For the other chart types (`vertical_bar`, `horizontal_bar`, `line` and `dot`) we used a Matcha model.\n\n\n\n\n# Solution in detail\n\n### 1. Donut: Axis labels detection\n\nWe train different image-encoder/text-decoder architectures to detect the axis labels. I trained a Donut model (which surprisingly got the best results) and two other Matcha models which got worse performance. Despite training with different `max_patches` results were rather similar for all. Most complications were associated to shared origins in the axis labels, long sequences, floats with many decimal places, etc. In this step, the most important is to get the minimum and the maximum values right for each axis so the cross multiplication is done right.\n\n### 2. EfficientDet: Object Detector for Data Series\n\nThis was one of the most time consuming parts for me. At the beggining of the competition until Nicholas shared his Donut model I didn't know that image-encoder/text-decoder models could perform that well in this competition so I focused on an object detector model which could detect well data series. To do so I lack bounding boxes training data, so I started manually labelling thousands of images. These took me a lot of days. My EfficientDet object detector did extremely well on generated images but not as well on extracted. I labeled as many images as I could. My OD has ~50% exact matches accuracy for scatter plots. It performs much better for other chart types, but since scatter plots can have *a lot* of points it's reasonable to be a tougher task.\n\nHere you can see some of my ODs predictions:\n\n\n\n### 3. EfficientDet: Object Detector for Chart bounding box\n\nAs part of the pipeline I had to detect the chart's bounding box so I could map pixel coordinates to real numerical values from the axis labels. This step was rather easy and its basically the same code from part (2) applied on the bounding boxes of `plot-bb` from the annotations. We had this training data available inside the JSONs files and it's a simple task so the OD achieves high accuracy.\n\n### 4. Matcha and Donut models\n\nAt the beggining of the competition I started making several changes to Nicholas' Donut model and soon discovered that you could achieve higher performance by perfoming some basic postprocessing. However, even after training more epochs, perfoming data augmentation and other tricks I realised I couldn't achieve much higher performances. I then tried implementing a Matcha model and with a lot of effort I made it work thanks (again) to Nicholas which raised the issue of the GitHub discussion. Once I got the Matcha model working I tried a ton of different stuff until I could squeeze as much performance as I could. So here is a little roadmap to achieveing 0.74 in the public LB:\n- **0.20:** a model with only Object Detection and Donut for axis labels.\n- **0.47:** Nicholas model but just modifying post processing, like using max length of xy series instead of the min.\n- **0.48:** doing additional post processing like filling values with average of the data series mean values.\n- **0.49:** using a mixture of Donut and the Object Detector.\n- **0.50**: training Donut for 10 epochs instead of 5.\n- **0.56:** BOOM! big increase by making the Matcha model work. Super vanilla, no fancy stuff.\n- **0.61:** combining vanilla Matcha with my Object Detector for scatter plots.\n- **0.64:** training Matcha for more epochs (10 epochs).\n- **0.69:** training Matcha with Bartley's generated images from code. These improvement increase the performance a lot of all chart types except scatter with a relatively low number of additional images per chart type (+5k per image).\n- **0.71:** increase `max_patches` of Matcha from 512 to 1024.\n- **0.74:** trained Matcha on 100% of the extracted images instead of the 75% I usually used so I could get a 25% holdout set for validation.\n\n\n# Things that didn't work\n\nSo many things that didn't work! I will write the ones I can remember:\n1. I tried training at an early stage **one different image-encoder/text-decoder model per each chart type** instead of training a general model. I believe that this didnt work for two reasons: 1. I think I trained a donut model for this 2. I didnt have enough images (at that point I didnt use Bartley generated images).\n2. I tried **different pre-trained weights** for the Matcha model and ALL got worse results. These were: `statista`, `chartqa`, `plotqa`. Best performance came from `matcha-base`.\n3. Tried training **two Matcha models for each axis**. I really thought this idea could work and until today I don't know why it didnt! My reasoning is the following: it's harder to predict a longer sequence, so why don't I train a model to predict x-axis series and another for y-axis? In the end it gave worse results than a combined model.\n4. Tried **training with A LOT of generated images**. I tried training with additional 5k, 10k and 25k more images per chart type. So essentially I trained the model with the competitions 60k plus the additional generated images. So I trained some models with a total number of 180k images! What I soon realised was that if there is not that much variance in additional generated images no matter how many images you add the model will have reached its capacity. I discussed this [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/415470#2294381), which relates to the Law of Diminishing returns.\n5. Trained with **Balance Sampler/Oversampling**. I trained with a Balance Sampler, which guarantees that on every batch, at least one minority class image will be present. My minority class images are the extracted images. This has two effects: it oversamples the minority class and (in theory) makes the model converge faster. I didn't see any benefit from oversampling (no surprise here).\n6. **Data augmentation**. I tried augmenting data mostly by applying color-related augmentations like `RGBShift`, `RandomBrightness`, `ColorJitter`, etc. \n7. **Increased the `max_patches` parameter** to 1536 (halfway between 1024 and 2048). So, as I saw an improvement by increasing `max_patches` from 512 to 1024 I thought, why don't I increase it a bit more and see what happens? Well, it didn't seem to improve the score and of course consumed more VRAM and computation time. Increasing `max_patches` increases a lot the VRAM consumption so the batch size had to be lowered and training took longer.\n8. Trained with **different schedulers**. I ended up training with `OneCycleLR` but before I tried some other, even constant LR, and never saw an increase in performance by modifying the LR scheduler.\n9. For the Object Detector I tried using **Weighted Boxes Fusion** to increase the exact matches of the data series. One of the greatest challenges of this competition is to get the exact number of data series points right. My OD sometimes produced an excess of low confident bounding boxes. I thought that maybe if I could fuse them with higher confidence bounding boxes I could get higher accuracies but couldn't quite make it. I ended up tunning the probability threshold for keeping/discarding the bounding boxes. In the end, bounding boxes with p>0.22 where kept. Would be glad to know how anyone did this!\n10. Different Object Detector backbones. I used `tf_efficientnetv2_s` as my OD's backbone, but tried larger models without match success like `tf_efficientnetv2_l`.\n11. **Automatic Mix Precision**. Not that it didn't work but didn't provide any performance by training with mixed precision using brain floating point tensors (`bfloat16`). However, since you use less VRAM compared to `float32` you can train with larger batch sizes which is a recommended practice (I heard that Karpathy said that it's always the best to train with as larger batches as you can, I may be wrong here).\n12. Many other things that I don't remember right now.\n\n\n# Hardware \n\nIt's barely impossible to train models like Matcha and Donut with the P100's provided by Kaggle. We ended up paying Google Colab Pro+ which provides A100 GPUs with 40 GB of VRAM since neither of us had DL GPUs like an RTX 3090/4090. With the A100 we were able to train large models and run many experiments faster. I trained more than 30 different models!\n\n# Conclusions\n\nEven though I would have liked to get inside the gold zone (always hoping for the best) I am grateful for the result we obtained and mainly for all the lessons learned throughout the competition. I firmly believe that if you want to learn something try changing it! Code something from scratch! Best way to learn by far. I am really looking forward to read other teams solutions. I would also like to know if anyone tried changing the Vision Model and the Text Model from Pix2Struct. I don't know if it's even possible or how its done so if you know please leave in the comments.\n\nThanks all!",
"votes": null
},
{
"id": "2310040",
"postDate": "06/20/2023 04:43:07",
"content": "Well laid out! I really feel we did our best and tried every idea we reasonably had with the information we had. I learned so much from this competition and from @alejopaullier and I am great duo for the opportunity.
", "rawMarkdown": "Well laid out! I really feel we did our best and tried every idea we reasonably had with the information we had. I learned so much from this competition and from @alejopaullier and I am great duo for the opportunity.", "votes": null }, { "id": "2310093", "postDate": "06/20/2023 05:28:33", "content": "@alejopaullier thank you for detailed topic. Very interesting!
", "rawMarkdown": "alejopaullier thank you for detailed topic. Very interesting!", "votes": null }, { "id": "2310242", "postDate": "06/20/2023 08:27:11", "content": "@alejopaullier Thanks for sharing your approach!
", "rawMarkdown": "alejopaullier Thanks for sharing your approach!", "votes": null } ], "comments": [ { "id": 2310040, "author_name": "cody11null", "author_url": "", "post_date": "06/20/2023 04:43:07", "content": "Well laid out! I really feel we did our best and tried every idea we reasonably had with the information we had. I learned so much from this competition and from @alejopaullier and I am great duo for the opportunity.
", "votes": null, "replies": [] }, { "id": 2310093, "author_name": "serangu", "author_url": "", "post_date": "06/20/2023 05:28:33", "content": "@alejopaullier thank you for detailed topic. Very interesting!
", "votes": null, "replies": [] }, { "id": 2310242, "author_name": "robertolabadietamayo", "author_url": "", "post_date": "06/20/2023 08:27:11", "content": "@alejopaullier Thanks for sharing your approach!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2310035": "# Code\n\nI hope you find the training and inference codes neat and tidy, I did my best effort to not make a mess!\n- [Matcha Training Code](https://www.kaggle.com/code/alejopaullier/benetech-matcha-train-0-74)\n- [Matcha Inference Code](https://www.kaggle.com/code/alejopaullier/benetech-matcha-inference-0-74)\n\n# TL;DR\n\nHere is a brief explanation of our team solution with @cody11null.\n\nOur solution consisted of a pipeline of object detectors (EfficientDet) and image-encoders/text-decoders (Matcha and Donut):\n1. A Donut model which acts like an OCR (Object Character Recognition) model is used to extract the axis labels of the chart (red)\n2. An object detector (EfficientDet) detects the data series (green). In the end, we only used this for scatter plots.\n3. A second object detector (EfficientDet) detects the plot's chart area (area where the data points are)(purple).\n4. Knowing the chart's limits from (3), the bounding boxes of the data series from (2) and the axis numerical values from (1) you can with a simple cross multiplication get the data series values. This approach is essentially as described in my [discussion](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/396773).\n5. For the other chart types (`vertical_bar`, `horizontal_bar`, `line` and `dot`) we used a Matcha model.\n\n\n\n\n# Solution in detail\n\n### 1. Donut: Axis labels detection\n\nWe train different image-encoder/text-decoder architectures to detect the axis labels. I trained a Donut model (which surprisingly got the best results) and two other Matcha models which got worse performance. Despite training with different `max_patches` results were rather similar for all. Most complications were associated to shared origins in the axis labels, long sequences, floats with many decimal places, etc. In this step, the most important is to get the minimum and the maximum values right for each axis so the cross multiplication is done right.\n\n### 2. EfficientDet: Object Detector for Data Series\n\nThis was one of the most time consuming parts for me. At the beggining of the competition until Nicholas shared his Donut model I didn't know that image-encoder/text-decoder models could perform that well in this competition so I focused on an object detector model which could detect well data series. To do so I lack bounding boxes training data, so I started manually labelling thousands of images. These took me a lot of days. My EfficientDet object detector did extremely well on generated images but not as well on extracted. I labeled as many images as I could. My OD has ~50% exact matches accuracy for scatter plots. It performs much better for other chart types, but since scatter plots can have *a lot* of points it's reasonable to be a tougher task.\n\nHere you can see some of my ODs predictions:\n\n\n\n### 3. EfficientDet: Object Detector for Chart bounding box\n\nAs part of the pipeline I had to detect the chart's bounding box so I could map pixel coordinates to real numerical values from the axis labels. This step was rather easy and its basically the same code from part (2) applied on the bounding boxes of `plot-bb` from the annotations. We had this training data available inside the JSONs files and it's a simple task so the OD achieves high accuracy.\n\n### 4. Matcha and Donut models\n\nAt the beggining of the competition I started making several changes to Nicholas' Donut model and soon discovered that you could achieve higher performance by perfoming some basic postprocessing. However, even after training more epochs, perfoming data augmentation and other tricks I realised I couldn't achieve much higher performances. I then tried implementing a Matcha model and with a lot of effort I made it work thanks (again) to Nicholas which raised the issue of the GitHub discussion. Once I got the Matcha model working I tried a ton of different stuff until I could squeeze as much performance as I could. So here is a little roadmap to achieveing 0.74 in the public LB:\n- **0.20:** a model with only Object Detection and Donut for axis labels.\n- **0.47:** Nicholas model but just modifying post processing, like using max length of xy series instead of the min.\n- **0.48:** doing additional post processing like filling values with average of the data series mean values.\n- **0.49:** using a mixture of Donut and the Object Detector.\n- **0.50**: training Donut for 10 epochs instead of 5.\n- **0.56:** BOOM! big increase by making the Matcha model work. Super vanilla, no fancy stuff.\n- **0.61:** combining vanilla Matcha with my Object Detector for scatter plots.\n- **0.64:** training Matcha for more epochs (10 epochs).\n- **0.69:** training Matcha with Bartley's generated images from code. These improvement increase the performance a lot of all chart types except scatter with a relatively low number of additional images per chart type (+5k per image).\n- **0.71:** increase `max_patches` of Matcha from 512 to 1024.\n- **0.74:** trained Matcha on 100% of the extracted images instead of the 75% I usually used so I could get a 25% holdout set for validation.\n\n\n# Things that didn't work\n\nSo many things that didn't work! I will write the ones I can remember:\n1. I tried training at an early stage **one different image-encoder/text-decoder model per each chart type** instead of training a general model. I believe that this didnt work for two reasons: 1. I think I trained a donut model for this 2. I didnt have enough images (at that point I didnt use Bartley generated images).\n2. I tried **different pre-trained weights** for the Matcha model and ALL got worse results. These were: `statista`, `chartqa`, `plotqa`. Best performance came from `matcha-base`.\n3. Tried training **two Matcha models for each axis**. I really thought this idea could work and until today I don't know why it didnt! My reasoning is the following: it's harder to predict a longer sequence, so why don't I train a model to predict x-axis series and another for y-axis? In the end it gave worse results than a combined model.\n4. Tried **training with A LOT of generated images**. I tried training with additional 5k, 10k and 25k more images per chart type. So essentially I trained the model with the competitions 60k plus the additional generated images. So I trained some models with a total number of 180k images! What I soon realised was that if there is not that much variance in additional generated images no matter how many images you add the model will have reached its capacity. I discussed this [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/415470#2294381), which relates to the Law of Diminishing returns.\n5. Trained with **Balance Sampler/Oversampling**. I trained with a Balance Sampler, which guarantees that on every batch, at least one minority class image will be present. My minority class images are the extracted images. This has two effects: it oversamples the minority class and (in theory) makes the model converge faster. I didn't see any benefit from oversampling (no surprise here).\n6. **Data augmentation**. I tried augmenting data mostly by applying color-related augmentations like `RGBShift`, `RandomBrightness`, `ColorJitter`, etc. \n7. **Increased the `max_patches` parameter** to 1536 (halfway between 1024 and 2048). So, as I saw an improvement by increasing `max_patches` from 512 to 1024 I thought, why don't I increase it a bit more and see what happens? Well, it didn't seem to improve the score and of course consumed more VRAM and computation time. Increasing `max_patches` increases a lot the VRAM consumption so the batch size had to be lowered and training took longer.\n8. Trained with **different schedulers**. I ended up training with `OneCycleLR` but before I tried some other, even constant LR, and never saw an increase in performance by modifying the LR scheduler.\n9. For the Object Detector I tried using **Weighted Boxes Fusion** to increase the exact matches of the data series. One of the greatest challenges of this competition is to get the exact number of data series points right. My OD sometimes produced an excess of low confident bounding boxes. I thought that maybe if I could fuse them with higher confidence bounding boxes I could get higher accuracies but couldn't quite make it. I ended up tunning the probability threshold for keeping/discarding the bounding boxes. In the end, bounding boxes with p>0.22 where kept. Would be glad to know how anyone did this!\n10. Different Object Detector backbones. I used `tf_efficientnetv2_s` as my OD's backbone, but tried larger models without match success like `tf_efficientnetv2_l`.\n11. **Automatic Mix Precision**. Not that it didn't work but didn't provide any performance by training with mixed precision using brain floating point tensors (`bfloat16`). However, since you use less VRAM compared to `float32` you can train with larger batch sizes which is a recommended practice (I heard that Karpathy said that it's always the best to train with as larger batches as you can, I may be wrong here).\n12. Many other things that I don't remember right now.\n\n\n# Hardware \n\nIt's barely impossible to train models like Matcha and Donut with the P100's provided by Kaggle. We ended up paying Google Colab Pro+ which provides A100 GPUs with 40 GB of VRAM since neither of us had DL GPUs like an RTX 3090/4090. With the A100 we were able to train large models and run many experiments faster. I trained more than 30 different models!\n\n# Conclusions\n\nEven though I would have liked to get inside the gold zone (always hoping for the best) I am grateful for the result we obtained and mainly for all the lessons learned throughout the competition. I firmly believe that if you want to learn something try changing it! Code something from scratch! Best way to learn by far. I am really looking forward to read other teams solutions. I would also like to know if anyone tried changing the Vision Model and the Text Model from Pix2Struct. I don't know if it's even possible or how its done so if you know please leave in the comments.\n\nThanks all!",
"2310040": "Well laid out! I really feel we did our best and tried every idea we reasonably had with the information we had. I learned so much from this competition and from @alejopaullier and I am great duo for the opportunity.",
"2310093": "alejopaullier thank you for detailed topic. Very interesting!",
"2310242": "alejopaullier Thanks for sharing your approach!"
},
"source": "existing_discussion",
"complete": true,
"competition_id": 43873,
"competition_slug": "benetech-making-graphs-accessible",
"team_id": 10092008,
"solution_url": "/competitions/benetech-making-graphs-accessible/writeups/expensive-ac-40th-place-solution-code"
}