![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F649b0fe7f791cae2522d4153ec579b22%2FScreen%20Shot%202023-03-22%20at%2018.50.11.png?generation=1679522250175667&alt=media\")
In this competition you need to predict different values (data series) depending on the figure type. A very clear explanation can be found in the Graph Convention section of this competition. These conventions ensure that predictions can be made in a consistent manner across a variety of chart formats.
\nDifferent types of figures have different predictions conventions. So the first step is to predict which kind of chart is present on the image. You can use any CNN backbone (like EfficientNets, Resnets, etc) with a Linear layer at the end with the same amount of neurons as chart types.
\n
This step is relatively easy and you should get high accuracy since it's pretty easy for CNN to distinguish the different types of charts!
\nEdit: the initial dataset description stated there were 4 types of charts but in fact there is a distinction between vertical bars and horizontal bars, so in reality there are 5 different types, so there should be 5 neurons in the linear layer.
\nThis competition's annotations already provide bounding boxes for the different components of the charts, like the position of the title, x-axis and y-axis labels, x-ticks and y-ticks, etc. You can find how to plot these bounding boxes in this notebook.
\nHowever, you might need to add some custom markers for points of interest, like data series which are not marked in the original images. In the next image these markers (bounding boxes) are represented by yellow dots.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F2d3f1a5bfed0969dae77a74514e47ea4%2FScreen%20Shot%202023-03-22%20at%2018.50.26.png?generation=1679522437122001&alt=media\")
Note: it might not be necessary to mark every data series point in the charts. With enough training points the Object Detection network is still able to detect data series points. Also, the accuracy of your Object Detector depends on the accuracy on the input bounding boxes and these are not perfect! For example, just look at the vertical bar chart on the image and you will notice that the provided sky blue bounding boxes are not always very accurate!
\nWith the images, the annotations and the markers you can train or fine-tune an Object Detection neural network. Use this network later at inference time to predict bounding boxes for test images without annotations nor data series points.
\nThis step may be optional, since OCR networks (step 4) predict text and numbers and also output their respective bounding boxes. However, you would rely on the OCR's predictive precision (which is not that precise) and you would still need to find a way to detect the data series bounding boxes (yellow dots)!
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2Fb1f0ef1f03e6e0cb989b9b6288c3e4d8%2FScreen%20Shot%202023-03-22%20at%2019.04.29.png?generation=1679522686864694&alt=media\")
Finally, with the bounding boxes and the original image you can pass them to an OCR (Object Character Recognition) network and get the corresponding text and numbers. Format the numbers, text and data series according to the chart type.
\nKnowing the bounding boxes for the data series points, the minimum and maximum values for each axis and their corresponding bounding boxes coordinates you can get data series values by a simple cross multiplication.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F279d702c6ce5d659be5693e64196d99d%2FScreen%20Shot%202023-03-22%20at%2019.04.12.png?generation=1679522897128510&alt=media\")
In this competition you need to predict different values (data series) depending on the figure type. A very clear explanation can be found in the Graph Convention section of this competition. These conventions ensure that predictions can be made in a consistent manner across a variety of chart formats.
\nDifferent types of figures have different predictions conventions. So the first step is to predict which kind of chart is present on the image. You can use any CNN backbone (like EfficientNets, Resnets, etc) with a Linear layer at the end with the same amount of neurons as chart types.
\n
This step is relatively easy and you should get high accuracy since it's pretty easy for CNN to distinguish the different types of charts!
\nEdit: the initial dataset description stated there were 4 types of charts but in fact there is a distinction between vertical bars and horizontal bars, so in reality there are 5 different types, so there should be 5 neurons in the linear layer.
\nThis competition's annotations already provide bounding boxes for the different components of the charts, like the position of the title, x-axis and y-axis labels, x-ticks and y-ticks, etc. You can find how to plot these bounding boxes in this notebook.
\nHowever, you might need to add some custom markers for points of interest, like data series which are not marked in the original images. In the next image these markers (bounding boxes) are represented by yellow dots.
\n
Note: it might not be necessary to mark every data series point in the charts. With enough training points the Object Detection network is still able to detect data series points. Also, the accuracy of your Object Detector depends on the accuracy on the input bounding boxes and these are not perfect! For example, just look at the vertical bar chart on the image and you will notice that the provided sky blue bounding boxes are not always very accurate!
\nWith the images, the annotations and the markers you can train or fine-tune an Object Detection neural network. Use this network later at inference time to predict bounding boxes for test images without annotations nor data series points.
\nThis step may be optional, since OCR networks (step 4) predict text and numbers and also output their respective bounding boxes. However, you would rely on the OCR's predictive precision (which is not that precise) and you would still need to find a way to detect the data series bounding boxes (yellow dots)!
\n
Finally, with the bounding boxes and the original image you can pass them to an OCR (Object Character Recognition) network and get the corresponding text and numbers. Format the numbers, text and data series according to the chart type.
\nKnowing the bounding boxes for the data series points, the minimum and maximum values for each axis and their corresponding bounding boxes coordinates you can get data series values by a simple cross multiplication.
\n
I really believe that end to end and OCR free approaches would have great results in this competition. The DoNut model for instance is one of such models.
", "rawMarkdown": "I really believe that end to end and OCR free approaches would have great results in this competition. The [DoNut](https://huggingface.co/docs/transformers/model_doc/donut) model for instance is one of such models.", "votes": 8, "replies": [ { "id": 2198184, "postDate": "2023-03-26T18:34:36.390Z", "content": "@kneroma what do you mean by end to end? DoNut looks interesting, the use case would be mainly asking which lables and ticks are present in the chart? Idk how would it be able to retrieve information about the data series though
", "rawMarkdown": "@kneroma what do you mean by end to end? DoNut looks interesting, the use case would be mainly asking which lables and ticks are present in the chart? Idk how would it be able to retrieve information about the data series though ", "replies": [ { "id": 2201006, "postDate": "2023-03-29T01:10:14.607Z", "content": "By end-to-end, I mean a trainable pipeline whose output can be directly used to solve the problem at hand. An end-to-end approach is opposed to a multi-step approach (ex: OCR + Classifier + PostProcessing).
\nFor the DoNut model, since it is a text generator, if one manages to construct a correct representation of the input (an html or xml format for example), the output can be post-processed to solve this problem directly.
", "rawMarkdown": "By end-to-end, I mean a trainable pipeline whose output can be directly used to solve the problem at hand. An end-to-end approach is opposed to a multi-step approach (ex: OCR + Classifier + PostProcessing).\n\nFor the DoNut model, since it is a text generator, if one manages to construct a correct representation of the input (an html or xml format for example), the output can be post-processed to solve this problem directly.", "votes": 2 } ] }, { "id": 2199018, "postDate": "2023-03-27T12:42:41.883Z", "content": "I hope you're right, it would be really cool to build an E2E model that can crack it… my bet is still that multi step engineered approaches will be more successful on the LB here…
", "rawMarkdown": "I hope you're right, it would be really cool to build an E2E model that can crack it... my bet is still that multi step engineered approaches will be more successful on the LB here...", "votes": 3, "replies": [ { "id": 2201007, "postDate": "2023-03-29T01:10:49.817Z", "content": "@thedrcat , I would bet on multi-step approaches if one could have access to high quality OCR Engine like Azure Read or Google Vision … If Tesseract (or equivalent OCR engine) is only what we have, then the OCR engine will be a real bottleneck in any multi-step approach.
", "rawMarkdown": "@thedrcat , I would bet on multi-step approaches if one could have access to high quality OCR Engine like Azure Read or Google Vision … If Tesseract (or equivalent OCR engine) is only what we have, then the OCR engine will be a real bottleneck in any multi-step approach.", "votes": 2 } ] }, { "id": 2199295, "postDate": "2023-03-27T15:59:06.353Z", "content": "Or something like this - Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
\nPaper - https://arxiv.org/abs/2210.03347
\nDocs - https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct
\nModels - https://huggingface.co/models?search=pix2struct
\n![\"Screenshot-from-2023-03-27-12-06-11.png\"](\"https://i.postimg.cc/Pxh8kDKH/Screenshot-from-2023-03-27-12-06-11.png\")
This seems like a good approach. Thanks for sharing @alejopaullier! Also really nice visual explanations!
", "rawMarkdown": "This seems like a good approach. Thanks for sharing @alejopaullier! Also really nice visual explanations!", "votes": 1, "replies": [ { "id": 2193786, "postDate": "2023-03-23T13:35:12.547Z", "content": "Thanks @ravishah1 much appreciated
", "rawMarkdown": "Thanks @ravishah1 much appreciated", "votes": 1 }, { "id": 2193910, "postDate": "2023-03-23T15:32:48.240Z", "content": "Hi @alejopaullier,
\nI did notice one slight correction. You say there should be 4 heads on the linear layer of the first model, but I think there are actually 5 classes. Bar is split into horizontal and vertical. So the classes we are predicting are:
You are right @ravishah1 I followed the dataset description which states:
\n\n\nThis competition's dataset comprises about 65,000 comprehensively-annotated scientific figures of four kinds: bar graphs, dot plots, line graphs, and scatter plots.
\n
Maybe @benjiaa can confirm this and eventually update the description. But by looking at the dataset I see that some annotation differentiate vertical bars from horizontal bars so there should be 5 classes.
", "rawMarkdown": "You are right @ravishah1 I followed the [dataset description](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/data) which states:\n> This competition's dataset comprises about 65,000 comprehensively-annotated scientific figures of **four kinds**: bar graphs, dot plots, line graphs, and scatter plots.\n\nMaybe @benjiaa can confirm this and eventually update the description. But by looking at the dataset I see that some annotation differentiate vertical bars from horizontal bars so there should be 5 classes.", "votes": 1 }, { "id": 2193943, "postDate": "2023-03-23T15:57:07.310Z", "content": "There are five types of chart. I'll update the description. Thanks!
", "rawMarkdown": "There are five types of chart. I'll update the description. Thanks!", "votes": 3 } ] } ] }, { "id": 2197745, "postDate": "2023-03-26T11:33:50.483Z", "content": "So this approach will still use rule based answer( in last step we \"get data series values by a simple cross multiplication\"). In this case it may be very hard to write rules for all types of graphs. Should we try to make some seq2seq predictions ? We input x-axis values as a sequence of items, with an embeded vector for this graph, then try to make predict values of y axis ?
", "rawMarkdown": "So this approach will still use rule based answer( in last step we \"get data series values by a simple cross multiplication\"). In this case it may be very hard to write rules for all types of graphs. Should we try to make some seq2seq predictions ? We input x-axis values as a sequence of items, with an embeded vector for this graph, then try to make predict values of y axis ? ", "votes": 2, "replies": [ { "id": 2198189, "postDate": "2023-03-26T18:36:45.963Z", "content": "By cross multiplication I mean that if you know the tick labels and the tick positions (bounding box in pixels) you can interpolate any data series point values if you know the data series bounding boxes
", "rawMarkdown": "By cross multiplication I mean that if you know the tick labels and the tick positions (bounding box in pixels) you can interpolate any data series point values if you know the data series bounding boxes", "votes": 1 } ] }, { "id": 2255619, "postDate": "2023-05-11T21:28:29.890Z", "content": "Nice work!
", "rawMarkdown": "Nice work!" }, { "id": 2249754, "postDate": "2023-05-08T05:18:32.333Z", "content": "Did you use this method to reach the current ranking？
", "rawMarkdown": "Did you use this method to reach the current ranking？" }, { "id": 2229909, "postDate": "2023-04-21T20:08:19.463Z", "content": "Excellent approach. New kaglers or beginners here is a notebook with a possible implementation of Step 1. https://www.kaggle.com/montenegroj/step-1-of-benetech-competition
", "rawMarkdown": "Excellent approach. New kaglers or beginners here is a notebook with a possible implementation of **Step 1**. https://www.kaggle.com/montenegroj/step-1-of-benetech-competition" }, { "id": 2208479, "postDate": "2023-04-04T05:16:27.843Z", "content": "Hi,
\nThanks for putting this together!
\nIf you'll pardon a beginner question, why use a linear layer at the output of the graph classifier rather than a softmax layer? Softmax seems like the more obvious choice to me for a classification problem, and I'd think you'd use a linear layer at the end for regression, but I'm still learning.
\nCheers,
\nMax
Hello Max!
\nIn PyTorch, it is not necessary to add a softmax or sigmoid layer at the end of a CNN because these activation functions are already included in the loss functions that are commonly used for classification tasks.
\nFor example, if you are training a CNN for multi-class classification using cross-entropy loss (like in this competition), the PyTorch implementation of the loss function already includes a softmax activation. Similarly, if you are training a CNN for binary classification using binary cross-entropy loss, the PyTorch implementation of the loss function already includes a sigmoid activation.
\nWhen you call the loss function in PyTorch, it expects the network's output to be in the form of logits (unnormalized scores for each class), rather than probabilities. The loss function then applies the appropriate activation function (softmax or sigmoid) and calculates the loss based on the probabilities that result.
", "rawMarkdown": "Hello Max! \n\nIn PyTorch, it is not necessary to add a softmax or sigmoid layer at the end of a CNN because these activation functions are already included in the loss functions that are commonly used for classification tasks.\n\nFor example, if you are training a CNN for multi-class classification using cross-entropy loss (like in this competition), the PyTorch implementation of the loss function already includes a softmax activation. Similarly, if you are training a CNN for binary classification using binary cross-entropy loss, the PyTorch implementation of the loss function already includes a sigmoid activation.\n\nWhen you call the loss function in PyTorch, it expects the network's output to be in the form of logits (unnormalized scores for each class), rather than probabilities. The loss function then applies the appropriate activation function (softmax or sigmoid) and calculates the loss based on the probabilities that result.", "votes": 1 } ] }, { "id": 2205672, "postDate": "2023-04-01T19:16:02.973Z", "content": "They seem so nice .
", "rawMarkdown": "They seem so nice ." }, { "id": 2205166, "postDate": "2023-04-01T10:03:16.973Z", "content": "It would be best if we could generate a graph, increase the data, and solve the problem with an E2E model like DoNut.
\n@ryanholbrook
\nIs it allowed to generate data? Also, it needs to be clarified if models like yolov5 and yolov8 are allowed.
Yes, you can generate new data.
\nIn the rules the competition license that the host is looking for is MIT, which is not compatible with yolo. If there is a version of yolo that will work with MIT then you can use that.
", "rawMarkdown": "Yes, you can generate new data. \n\nIn the rules the competition license that the host is looking for is MIT, which is not compatible with yolo. If there is a version of yolo that will work with MIT then you can use that. \n", "votes": 3, "replies": [ { "id": 2213711, "postDate": "2023-04-07T19:37:21.980Z", "content": "what is MIT?
", "rawMarkdown": "what is MIT?\n" }, { "id": 2214136, "postDate": "2023-04-08T07:06:07.007Z", "content": "One type of permission opensource license. You can get the details here: https://fossa.com/blog/open-source-licenses-101-mit-license/
", "rawMarkdown": "One type of permission opensource license. You can get the details here: https://fossa.com/blog/open-source-licenses-101-mit-license/", "votes": 1 }, { "id": 2216894, "postDate": "2023-04-10T12:58:05.770Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2219040, "postDate": "2023-04-12T08:25:38.237Z", "content": "Hi
\nJust out of curiosity, a model or code that is licensed “Apache 2.0” can be used ? Since what I know that the apache license is compatible with MIT license ?
\n@maggiemd
\n@ryanholbrook
Thanks for sharing this comprehensive approach @alejopaullier 🙌🙏
", "rawMarkdown": "Thanks for sharing this comprehensive approach @alejopaullier 🙌🙏" }, { "id": 2274410, "postDate": "2023-05-26T01:23:24.923Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2200104, "postDate": "2023-03-28T09:39:16.247Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2226557, "postDate": "2023-04-19T03:23:30.973Z", "content": "Thanks. Nice analysis
", "rawMarkdown": "Thanks. Nice analysis", "votes": 2 } ], "comments": [ { "id": 2195324, "author_name": "kkiller", "author_url": "", "post_date": "2023-03-24T15:08:54.813000", "content": "I really believe that end to end and OCR free approaches would have great results in this competition. The DoNut model for instance is one of such models.
", "votes": 8, "replies": [ { "id": 2198184, "author_name": "moth", "author_url": "", "post_date": "2023-03-26T18:34:36.390000", "content": "@kneroma what do you mean by end to end? DoNut looks interesting, the use case would be mainly asking which lables and ticks are present in the chart? Idk how would it be able to retrieve information about the data series though
", "votes": 0, "replies": [ { "id": 2201006, "author_name": "kkiller", "author_url": "", "post_date": "2023-03-29T01:10:14.607000", "content": "By end-to-end, I mean a trainable pipeline whose output can be directly used to solve the problem at hand. An end-to-end approach is opposed to a multi-step approach (ex: OCR + Classifier + PostProcessing).
\nFor the DoNut model, since it is a text generator, if one manages to construct a correct representation of the input (an html or xml format for example), the output can be post-processed to solve this problem directly.
", "votes": 2, "replies": [] } ] }, { "id": 2199018, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2023-03-27T12:42:41.883000", "content": "I hope you're right, it would be really cool to build an E2E model that can crack it… my bet is still that multi step engineered approaches will be more successful on the LB here…
", "votes": 3, "replies": [ { "id": 2201007, "author_name": "kkiller", "author_url": "", "post_date": "2023-03-29T01:10:49.817000", "content": "@thedrcat , I would bet on multi-step approaches if one could have access to high quality OCR Engine like Azure Read or Google Vision … If Tesseract (or equivalent OCR engine) is only what we have, then the OCR engine will be a real bottleneck in any multi-step approach.
", "votes": 2, "replies": [] } ] }, { "id": 2199295, "author_name": "RB", "author_url": "", "post_date": "2023-03-27T15:59:06.353000", "content": "Or something like this - Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
\nPaper - https://arxiv.org/abs/2210.03347
\nDocs - https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct
\nModels - https://huggingface.co/models?search=pix2struct
\n
This seems like a good approach. Thanks for sharing @alejopaullier! Also really nice visual explanations!
", "votes": 1, "replies": [ { "id": 2193786, "author_name": "moth", "author_url": "", "post_date": "2023-03-23T13:35:12.547000", "content": "Thanks @ravishah1 much appreciated
", "votes": 1, "replies": [] }, { "id": 2193910, "author_name": "Ravi Shah", "author_url": "", "post_date": "2023-03-23T15:32:48.240000", "content": "Hi @alejopaullier,
\nI did notice one slight correction. You say there should be 4 heads on the linear layer of the first model, but I think there are actually 5 classes. Bar is split into horizontal and vertical. So the classes we are predicting are:
You are right @ravishah1 I followed the dataset description which states:
\n\n\nThis competition's dataset comprises about 65,000 comprehensively-annotated scientific figures of four kinds: bar graphs, dot plots, line graphs, and scatter plots.
\n
Maybe @benjiaa can confirm this and eventually update the description. But by looking at the dataset I see that some annotation differentiate vertical bars from horizontal bars so there should be 5 classes.
", "votes": 1, "replies": [] }, { "id": 2193943, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-03-23T15:57:07.310000", "content": "There are five types of chart. I'll update the description. Thanks!
", "votes": 3, "replies": [] } ] } ] }, { "id": 2197745, "author_name": "hutedu", "author_url": "", "post_date": "2023-03-26T11:33:50.483000", "content": "So this approach will still use rule based answer( in last step we \"get data series values by a simple cross multiplication\"). In this case it may be very hard to write rules for all types of graphs. Should we try to make some seq2seq predictions ? We input x-axis values as a sequence of items, with an embeded vector for this graph, then try to make predict values of y axis ?
", "votes": 2, "replies": [ { "id": 2198189, "author_name": "moth", "author_url": "", "post_date": "2023-03-26T18:36:45.963000", "content": "By cross multiplication I mean that if you know the tick labels and the tick positions (bounding box in pixels) you can interpolate any data series point values if you know the data series bounding boxes
", "votes": 1, "replies": [] } ] }, { "id": 2255619, "author_name": "Durval Slompo Junior", "author_url": "", "post_date": "2023-05-11T21:28:29.890000", "content": "Nice work!
", "votes": 0, "replies": [] }, { "id": 2249754, "author_name": "Chinartist", "author_url": "", "post_date": "2023-05-08T05:18:32.333000", "content": "Did you use this method to reach the current ranking？
", "votes": 0, "replies": [] }, { "id": 2229909, "author_name": "jdmm", "author_url": "", "post_date": "2023-04-21T20:08:19.463000", "content": "Excellent approach. New kaglers or beginners here is a notebook with a possible implementation of Step 1. https://www.kaggle.com/montenegroj/step-1-of-benetech-competition
", "votes": 0, "replies": [] }, { "id": 2208479, "author_name": "Max Bond ABE", "author_url": "", "post_date": "2023-04-04T05:16:27.843000", "content": "Hi,
\nThanks for putting this together!
\nIf you'll pardon a beginner question, why use a linear layer at the output of the graph classifier rather than a softmax layer? Softmax seems like the more obvious choice to me for a classification problem, and I'd think you'd use a linear layer at the end for regression, but I'm still learning.
\nCheers,
\nMax
Hello Max!
\nIn PyTorch, it is not necessary to add a softmax or sigmoid layer at the end of a CNN because these activation functions are already included in the loss functions that are commonly used for classification tasks.
\nFor example, if you are training a CNN for multi-class classification using cross-entropy loss (like in this competition), the PyTorch implementation of the loss function already includes a softmax activation. Similarly, if you are training a CNN for binary classification using binary cross-entropy loss, the PyTorch implementation of the loss function already includes a sigmoid activation.
\nWhen you call the loss function in PyTorch, it expects the network's output to be in the form of logits (unnormalized scores for each class), rather than probabilities. The loss function then applies the appropriate activation function (softmax or sigmoid) and calculates the loss based on the probabilities that result.
", "votes": 1, "replies": [] } ] }, { "id": 2205672, "author_name": "emine ildes egri", "author_url": "", "post_date": "2023-04-01T19:16:02.973000", "content": "They seem so nice .
", "votes": 0, "replies": [] }, { "id": 2205166, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2023-04-01T10:03:16.973000", "content": "It would be best if we could generate a graph, increase the data, and solve the problem with an E2E model like DoNut.
\n@ryanholbrook
\nIs it allowed to generate data? Also, it needs to be clarified if models like yolov5 and yolov8 are allowed.
Yes, you can generate new data.
\nIn the rules the competition license that the host is looking for is MIT, which is not compatible with yolo. If there is a version of yolo that will work with MIT then you can use that.
", "votes": 3, "replies": [ { "id": 2213711, "author_name": "YAZEED BADARO", "author_url": "", "post_date": "2023-04-07T19:37:21.980000", "content": "what is MIT?
", "votes": 0, "replies": [] }, { "id": 2214136, "author_name": "Abdullah Husein", "author_url": "", "post_date": "2023-04-08T07:06:07.007000", "content": "One type of permission opensource license. You can get the details here: https://fossa.com/blog/open-source-licenses-101-mit-license/
", "votes": 1, "replies": [] }, { "id": 2216894, "author_name": "", "author_url": "", "post_date": "2023-04-10T12:58:05.770000", "content": "", "votes": 0, "replies": [] }, { "id": 2219040, "author_name": "Ali", "author_url": "", "post_date": "2023-04-12T08:25:38.237000", "content": "Hi
\nJust out of curiosity, a model or code that is licensed “Apache 2.0” can be used ? Since what I know that the apache license is compatible with MIT license ?
\n@maggiemd
\n@ryanholbrook
Thanks for sharing this comprehensive approach @alejopaullier 🙌🙏
", "votes": 0, "replies": [] }, { "id": 2274410, "author_name": "", "author_url": "", "post_date": "2023-05-26T01:23:24.923000", "content": "", "votes": 0, "replies": [] }, { "id": 2200104, "author_name": "", "author_url": "", "post_date": "2023-03-28T09:39:16.247000", "content": "", "votes": 0, "replies": [] }, { "id": 2226557, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-04-19T03:23:30.973000", "content": "Thanks. Nice analysis
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2192751": "In this competition you need to predict different values (data series) depending on the figure type. A very clear explanation can be found in the [Graph Convention section](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/overview/graph-conventions) of this competition. These conventions ensure that predictions can be made in a consistent manner across a variety of chart formats.\n\n# Step 1: predict the figure type\n\nDifferent types of figures have different predictions conventions. So the first step is to predict which kind of chart is present on the image. You can use any CNN backbone (like EfficientNets, Resnets, etc) with a Linear layer at the end with the same amount of neurons as chart types.\n\n\n\nThis step is relatively easy and you should get high accuracy since it's pretty easy for CNN to distinguish the different types of charts!\n\n**Edit:** the initial dataset description stated there were 4 types of charts but in fact there is a distinction between vertical bars and horizontal bars, so in reality there are 5 different types, so there should be 5 neurons in the linear layer.\n\n# Step 2: add annotations and custom markers\n\nThis competition's annotations already provide bounding boxes for the different components of the charts, like the position of the title, x-axis and y-axis labels, x-ticks and y-ticks, etc. You can find how to plot these bounding boxes in this [notebook](https://www.kaggle.com/code/alejopaullier/benetech-simple-ocr-network).\n\nHowever, you might need to add some custom markers for points of interest, like data series which are not marked in the original images. In the next image these markers (bounding boxes) are represented by yellow dots.\n\n\n\n**Note:** it might not be necessary to mark every data series point in the charts. With enough training points the Object Detection network is still able to detect data series points. Also, the accuracy of your Object Detector depends on the accuracy on the input bounding boxes and these are not perfect! For example, just look at the vertical bar chart on the image and you will notice that the provided sky blue bounding boxes are not always very accurate!\n\n# Step 3: train an Object Detection Neural Network and predict bounding boxes\n\nWith the images, the annotations and the markers you can train or fine-tune an Object Detection neural network. Use this network later at inference time to predict bounding boxes for test images without annotations nor data series points.\n\nThis step may be optional, since OCR networks (step 4) predict text and numbers and also output their respective bounding boxes. However, you would rely on the OCR's predictive precision (which is not that precise) and you would still need to find a way to detect the data series bounding boxes (yellow dots)!\n\n\n\n# Step 4: get text and numbers from OCR\n\nFinally, with the bounding boxes and the original image you can pass them to an OCR (Object Character Recognition) network and get the corresponding text and numbers. Format the numbers, text and data series according to the chart type. \n\nKnowing the bounding boxes for the data series points, the minimum and maximum values for each axis and their corresponding bounding boxes coordinates you can get data series values by a simple cross multiplication.\n\n\n ", "2195324": "I really believe that end to end and OCR free approaches would have great results in this competition. The [DoNut](https://huggingface.co/docs/transformers/model_doc/donut) model for instance is one of such models.", "2193094": "This seems like a good approach. Thanks for sharing @alejopaullier! Also really nice visual explanations!", "2197745": "So this approach will still use rule based answer( in last step we \"get data series values by a simple cross multiplication\"). In this case it may be very hard to write rules for all types of graphs. Should we try to make some seq2seq predictions ? We input x-axis values as a sequence of items, with an embeded vector for this graph, then try to make predict values of y axis ? ", "2255619": "Nice work!", "2249754": "Did you use this method to reach the current ranking？", "2229909": "Excellent approach. New kaglers or beginners here is a notebook with a possible implementation of **Step 1**. https://www.kaggle.com/montenegroj/step-1-of-benetech-competition", "2208479": "Hi,\n\nThanks for putting this together!\n\nIf you'll pardon a beginner question, why use a linear layer at the output of the graph classifier rather than a softmax layer? Softmax seems like the more obvious choice to me for a classification problem, and I'd think you'd use a linear layer at the end for regression, but I'm still learning.\n\nCheers,\nMax", "2205672": "They seem so nice .", "2205166": "It would be best if we could generate a graph, increase the data, and solve the problem with an E2E model like DoNut.\n@ryanholbrook \nIs it allowed to generate data? Also, it needs to be clarified if models like yolov5 and yolov8 are allowed.", "2201190": "Thanks for sharing this comprehensive approach @alejopaullier 🙌🙏", "2274410": "", "2200104": "", "2226557": "Thanks. Nice analysis" } }