![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18102%2F93f4e85ac942a4dbcecb438651665bb1%2FScreenshot%202023-04-22%20at%2019.35.17.png?generation=1682185682003901&alt=media\")
\n\nOccasionally, a figure may contain annotations with value labels. While you may use these values if you wish, note they may differ from ground-truth values, which are calculated based on the positioning of graphical elements on the page.
\n
This section got me wondering what are we actually trying to model here? It feels like the host has already some automated way to create \"ground truth\" labels for 10K test charts.
", "messages": [ { "id": "2228456", "postDate": "04/20/2023 15:04:33", "content": "\n\nOccasionally, a figure may contain annotations with value labels. While you may use these values if you wish, note they may differ from ground-truth values, which are calculated based on the positioning of graphical elements on the page.
\n
This section got me wondering what are we actually trying to model here? It feels like the host has already some automated way to create \"ground truth\" labels for 10K test charts.
", "rawMarkdown": ">Occasionally, a figure may contain annotations with value labels. While you may use these values if you wish, note they may differ from ground-truth values, which are calculated based on the positioning of graphical elements on the page.\n\nThis section got me wondering what are we actually trying to model here? It feels like the host has already some automated way to create \"ground truth\" labels for 10K test charts.", "votes": null }, { "id": "2229731", "postDate": "04/21/2023 16:30:02", "content": "Hi @gaborfodor,
\nAs the passage mentions, the ground-truth series were derived from the graphical elements in the figure, like the tick positions, figure bbox, etc. All of these graphical elements had to be annotated by hand. The goal of the competition is to derive the data series for a chart directly from the figure itself without the need for manual annotation.
", "rawMarkdown": "Hi @gaborfodor,\n\nAs the passage mentions, the ground-truth series were derived from the graphical elements in the figure, like the tick positions, figure bbox, etc. All of these graphical elements had to be annotated by hand. The goal of the competition is to derive the data series for a chart directly from the figure itself without the need for manual annotation.", "votes": null }, { "id": "2229774", "postDate": "04/21/2023 17:05:36", "content": "Wow did you really manually annotate the 10K test images? If we have only label noise that is understandable. I am only a bit worried about systematic bias if for example automated labeling aid was used during the annotation process. Humans are usually lazy and it would introduce some bias that we would need to learn as well to optimize the LB.
\nAt this point there are huge gaps on the LB so we have lots of room for improvement.
", "rawMarkdown": "Wow did you really manually annotate the 10K test images? If we have only label noise that is understandable. I am only a bit worried about systematic bias if for example automated labeling aid was used during the annotation process. Humans are usually lazy and it would introduce some bias that we would need to learn as well to optimize the LB.\n\nAt this point there are huge gaps on the LB so we have lots of room for improvement.", "votes": null }, { "id": "2229791", "postDate": "04/21/2023 17:24:19", "content": "I believe the competition host employed a labeling service that used some custom software to assist the process. To my knowledge, there wasn't any kind of algorithmic assistance, however. @benjiaa might be able to speak to it more.
", "rawMarkdown": "I believe the competition host employed a labeling service that used some custom software to assist the process. To my knowledge, there wasn't any kind of algorithmic assistance, however. @benjiaa might be able to speak to it more.", "votes": null }, { "id": "2229945", "postDate": "04/21/2023 20:54:31", "content": "Yes, humans annotated all samples. The software used by our annotators has a feature to shrink existing bounding boxes to fit text and chart elements closely, but other than that I don't think there's anything you could call algorithmic assistance.
", "rawMarkdown": "Yes, humans annotated all samples. The software used by our annotators has a feature to shrink existing bounding boxes to fit text and chart elements closely, but other than that I don't think there's anything you could call algorithmic assistance.", "votes": null }, { "id": "2230740", "postDate": "04/22/2023 17:47:26", "content": "Thanks for the quick reply! Sorry for the nitpicking but it is still a bit confusing for me what did the human annotators actually annotate.
\nIn the provided JSONs we do not have bounding boxes or coordinates for the most important graphical elements (bars, scatterpoints) etc.
Do you have those annotations as well or did you use some automated processing to derive the relative pixel postitions?
\nI would like to understand the process a bit better how you came up with the target ground truth for the non-generated samples.
\n
Hi @gaborfodor ,
\nI updated the dataset to add annotations for the bars, scatterplots, etc. Hopefully this should make the conversion from visual element to data point more clear.
", "rawMarkdown": "Hi @gaborfodor ,\n\nI updated the dataset to add annotations for the bars, scatterplots, etc. Hopefully this should make the conversion from visual element to data point more clear.", "votes": null } ], "comments": [ { "id": 2229731, "author_name": "ryanholbrook", "author_url": "", "post_date": "04/21/2023 16:30:02", "content": "Hi @gaborfodor,
\nAs the passage mentions, the ground-truth series were derived from the graphical elements in the figure, like the tick positions, figure bbox, etc. All of these graphical elements had to be annotated by hand. The goal of the competition is to derive the data series for a chart directly from the figure itself without the need for manual annotation.
", "votes": null, "replies": [] }, { "id": 2229774, "author_name": "gaborfodor", "author_url": "", "post_date": "04/21/2023 17:05:36", "content": "Wow did you really manually annotate the 10K test images? If we have only label noise that is understandable. I am only a bit worried about systematic bias if for example automated labeling aid was used during the annotation process. Humans are usually lazy and it would introduce some bias that we would need to learn as well to optimize the LB.
\nAt this point there are huge gaps on the LB so we have lots of room for improvement.
", "votes": null, "replies": [ { "id": 2229791, "author_name": "ryanholbrook", "author_url": "", "post_date": "04/21/2023 17:24:19", "content": "I believe the competition host employed a labeling service that used some custom software to assist the process. To my knowledge, there wasn't any kind of algorithmic assistance, however. @benjiaa might be able to speak to it more.
", "votes": null, "replies": [ { "id": 2229945, "author_name": "benjiaa", "author_url": "", "post_date": "04/21/2023 20:54:31", "content": "Yes, humans annotated all samples. The software used by our annotators has a feature to shrink existing bounding boxes to fit text and chart elements closely, but other than that I don't think there's anything you could call algorithmic assistance.
", "votes": null, "replies": [ { "id": 2230740, "author_name": "gaborfodor", "author_url": "", "post_date": "04/22/2023 17:47:26", "content": "Thanks for the quick reply! Sorry for the nitpicking but it is still a bit confusing for me what did the human annotators actually annotate.
\nIn the provided JSONs we do not have bounding boxes or coordinates for the most important graphical elements (bars, scatterpoints) etc.
Do you have those annotations as well or did you use some automated processing to derive the relative pixel postitions?
\nI would like to understand the process a bit better how you came up with the target ground truth for the non-generated samples.
\n
Hi @gaborfodor ,
\nI updated the dataset to add annotations for the bars, scatterplots, etc. Hopefully this should make the conversion from visual element to data point more clear.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2228456": ">Occasionally, a figure may contain annotations with value labels. While you may use these values if you wish, note they may differ from ground-truth values, which are calculated based on the positioning of graphical elements on the page.\n\nThis section got me wondering what are we actually trying to model here? It feels like the host has already some automated way to create \"ground truth\" labels for 10K test charts.", "2229731": "Hi @gaborfodor,\n\nAs the passage mentions, the ground-truth series were derived from the graphical elements in the figure, like the tick positions, figure bbox, etc. All of these graphical elements had to be annotated by hand. The goal of the competition is to derive the data series for a chart directly from the figure itself without the need for manual annotation.", "2229774": "Wow did you really manually annotate the 10K test images? If we have only label noise that is understandable. I am only a bit worried about systematic bias if for example automated labeling aid was used during the annotation process. Humans are usually lazy and it would introduce some bias that we would need to learn as well to optimize the LB.\n\nAt this point there are huge gaps on the LB so we have lots of room for improvement.", "2229791": "I believe the competition host employed a labeling service that used some custom software to assist the process. To my knowledge, there wasn't any kind of algorithmic assistance, however. @benjiaa might be able to speak to it more.", "2229945": "Yes, humans annotated all samples. The software used by our annotators has a feature to shrink existing bounding boxes to fit text and chart elements closely, but other than that I don't think there's anything you could call algorithmic assistance.", "2230740": "Thanks for the quick reply! Sorry for the nitpicking but it is still a bit confusing for me what did the human annotators actually annotate. \nIn the provided JSONs we do not have bounding boxes or coordinates for the most important graphical elements (bars, scatterpoints) etc.\n\nDo you have those annotations as well or did you use some automated processing to derive the relative pixel postitions?\n\nI would like to understand the process a bit better how you came up with the target ground truth for the non-generated samples.\n\n", "2234044": "Hi @gaborfodor ,\n\nI updated the dataset to add annotations for the bars, scatterplots, etc. Hopefully this should make the conversion from visual element to data point more clear." }, "source": "meta" }