{ "id": 418350, "title": "[0.71] MIT compliant score & ICDAR boost", "url": "/competitions/benetech-making-graphs-accessible/discussion/418350", "author_name": "", "post_date": "2023-06-20T06:55:18.112164600Z", "votes": 15, "comment_count": 16, "views": 0, "content": "

Our MIT compliant submission scored 0.87 on public and 0.71 on private LB.

\n

Adding ICDAR boosted public by 0.01, but private stayed at 0.71. Since I was adding it the night they announced it was allowed, I wasn't able to label it perfectly (just filtered out 20k images with a GUI I made, parsed available labels and pseudolabel the rest).

\n

Did you get more out of it? I am wondering if I labeled it better, would it help more.
\nDid anything else got some boost out of non-MIT datasets & models?

\n

EDIT: I just realized that adding ICDAR actually dropped our score on private.

", "messages": [ { "id": "2310170", "postDate": "06/20/2023 06:55:18", "content": "

Our MIT compliant submission scored 0.87 on public and 0.71 on private LB.

\n

Adding ICDAR boosted public by 0.01, but private stayed at 0.71. Since I was adding it the night they announced it was allowed, I wasn't able to label it perfectly (just filtered out 20k images with a GUI I made, parsed available labels and pseudolabel the rest).

\n

Did you get more out of it? I am wondering if I labeled it better, would it help more.
\nDid anything else got some boost out of non-MIT datasets & models?

\n

EDIT: I just realized that adding ICDAR actually dropped our score on private.

", "rawMarkdown": "Our MIT compliant submission scored 0.87 on public and 0.71 on private LB.\n\nAdding ICDAR boosted public by 0.01, but private stayed at 0.71. Since I was adding it the night they announced it was allowed, I wasn't able to label it perfectly (just filtered out 20k images with a GUI I made, parsed available labels and pseudolabel the rest).\n\nDid you get more out of it? I am wondering if I labeled it better, would it help more.\nDid anything else got some boost out of non-MIT datasets & models?\n\nEDIT: I just realized that adding ICDAR actually dropped our score on private.", "votes": null }, { "id": "2310192", "postDate": "06/20/2023 07:30:03", "content": "

I used around 1100 images from ICDAR, only those having 1 XY series. I also did post-processing to ensure annotations match competition graph conventions (e.g. handling percentages, interpolation of line plot data to match tick labels etc). At the start of comp, I got ~0.04 boost in public LB with this data. For my final model, the impact was much smaller (~0.01 in private LB), as I did first-phase of training with a large number (~900k) of self-generated synthetic plots.

", "rawMarkdown": "I used around 1100 images from ICDAR, only those having 1 XY series. I also did post-processing to ensure annotations match competition graph conventions (e.g. handling percentages, interpolation of line plot data to match tick labels etc). At the start of comp, I got ~0.04 boost in public LB with this data. For my final model, the impact was much smaller (~0.01 in private LB), as I did first-phase of training with a large number (~900k) of self-generated synthetic plots.", "votes": null }, { "id": "2310296", "postDate": "06/20/2023 09:04:05", "content": "

The boost is not that big for you as well. You probably had a bit better labels for ICDAR compared to us.

", "rawMarkdown": "The boost is not that big for you as well. You probably had a bit better labels for ICDAR compared to us.", "votes": null }, { "id": "2310375", "postDate": "06/20/2023 10:15:13", "content": "

@crodoc good topic and good questions! Thank you! Can you share your code?

", "rawMarkdown": "crodoc good topic and good questions! Thank you! Can you share your code?", "votes": null }, { "id": "2310414", "postDate": "06/20/2023 10:47:23", "content": "

Planning to put it on git :)

", "rawMarkdown": "Planning to put it on git :)", "votes": null }, { "id": "2310444", "postDate": "06/20/2023 11:07:18", "content": "

Congrats! Looking forward to learning from your solution.

", "rawMarkdown": "Congrats! Looking forward to learning from your solution.", "votes": null }, { "id": "2310527", "postDate": "06/20/2023 12:34:49", "content": "

@conjuring92 did you use scatter charts from ICDAR?

", "rawMarkdown": "conjuring92 did you use scatter charts from ICDAR?", "votes": null }, { "id": "2310552", "postDate": "06/20/2023 13:03:18", "content": "

@crodoc in our case it made our score worse. However, I took all available images from scatter, vertical bars, lines and horizontal bars regardless if there were more than one data series in the chart or not. Btw, I remember you guys had a big jump from 0.79 to 0.87 like in one submission, what made you had such big increment?

", "rawMarkdown": "crodoc in our case it made our score worse. However, I took all available images from scatter, vertical bars, lines and horizontal bars regardless if there were more than one data series in the chart or not. Btw, I remember you guys had a big jump from 0.79 to 0.87 like in one submission, what made you had such big increment?", "votes": null }, { "id": "2310587", "postDate": "06/20/2023 13:47:37", "content": "

All of the subs below 0.8 were without scatter inference. We made our first full sub the night before and it crashed 😅 so I had to wait until 2AM of the last day since we were out of subs.

\n

I overwrote scatter with matcha predictions by accident. Luckily it was an easy bug to spot.

", "rawMarkdown": "All of the subs below 0.8 were without scatter inference. We made our first full sub the night before and it crashed 😅 so I had to wait until 2AM of the last day since we were out of subs.\n\nI overwrote scatter with matcha predictions by accident. Luckily it was an easy bug to spot.", "votes": null }, { "id": "2310590", "postDate": "06/20/2023 13:49:48", "content": "

Living life to the limit! 😂 Good you spotted that since scatter has a larger proportion in private LB than in public LB!

", "rawMarkdown": "Living life to the limit! 😂 Good you spotted that since scatter has a larger proportion in private LB than in public LB!", "votes": null }, { "id": "2310596", "postDate": "06/20/2023 13:52:26", "content": "

We should have probed for #charts for each chart type in the test set. I was surprised by the amount of scatter charts in the private test set.

", "rawMarkdown": "We should have probed for #charts for each chart type in the test set. I was surprised by the amount of scatter charts in the private test set.", "votes": null }, { "id": "2310603", "postDate": "06/20/2023 13:55:51", "content": "

is there a way to probe private LB? you could have never known there were going to be that many scatters

", "rawMarkdown": "is there a way to probe private LB? you could have never known there were going to be that many scatters", "votes": null }, { "id": "2310605", "postDate": "06/20/2023 13:57:37", "content": "

Use a classifier and raise exception of it predicts over X scatter plots.

", "rawMarkdown": "Use a classifier and raise exception of it predicts over X scatter plots.", "votes": null }, { "id": "2310608", "postDate": "06/20/2023 13:59:47", "content": "

good to know, I once tried probing LB using timers (freeze time 2 sec every time you find X chart), but couldn't get much sense out of it

", "rawMarkdown": "good to know, I once tried probing LB using timers (freeze time 2 sec every time you find X chart), but couldn't get much sense out of it", "votes": null }, { "id": "2310613", "postDate": "06/20/2023 14:02:11", "content": "

Inference time is not reliable. Exceptions are 😅

", "rawMarkdown": "Inference time is not reliable. Exceptions are 😅", "votes": null }, { "id": "2310762", "postDate": "06/20/2023 15:33:43", "content": "

yes, I had 150+ scatter from ICDAR

", "rawMarkdown": "yes, I had 150+ scatter from ICDAR", "votes": null }, { "id": "2310920", "postDate": "06/20/2023 17:32:50", "content": "

I think you were better on scatter by 0.01 :)

", "rawMarkdown": "I think you were better on scatter by 0.01 :)", "votes": null } ], "comments": [ { "id": 2310192, "author_name": "conjuring92", "author_url": "", "post_date": "06/20/2023 07:30:03", "content": "

I used around 1100 images from ICDAR, only those having 1 XY series. I also did post-processing to ensure annotations match competition graph conventions (e.g. handling percentages, interpolation of line plot data to match tick labels etc). At the start of comp, I got ~0.04 boost in public LB with this data. For my final model, the impact was much smaller (~0.01 in private LB), as I did first-phase of training with a large number (~900k) of self-generated synthetic plots.

", "votes": null, "replies": [ { "id": 2310296, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 09:04:05", "content": "

The boost is not that big for you as well. You probably had a bit better labels for ICDAR compared to us.

", "votes": null, "replies": [] }, { "id": 2310527, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 12:34:49", "content": "

@conjuring92 did you use scatter charts from ICDAR?

", "votes": null, "replies": [ { "id": 2310762, "author_name": "conjuring92", "author_url": "", "post_date": "06/20/2023 15:33:43", "content": "

yes, I had 150+ scatter from ICDAR

", "votes": null, "replies": [ { "id": 2310920, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 17:32:50", "content": "

I think you were better on scatter by 0.01 :)

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2310375, "author_name": "serangu", "author_url": "", "post_date": "06/20/2023 10:15:13", "content": "

@crodoc good topic and good questions! Thank you! Can you share your code?

", "votes": null, "replies": [ { "id": 2310414, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 10:47:23", "content": "

Planning to put it on git :)

", "votes": null, "replies": [] } ] }, { "id": 2310444, "author_name": "zhugds", "author_url": "", "post_date": "06/20/2023 11:07:18", "content": "

Congrats! Looking forward to learning from your solution.

", "votes": null, "replies": [] }, { "id": 2310552, "author_name": "alejopaullier", "author_url": "", "post_date": "06/20/2023 13:03:18", "content": "

@crodoc in our case it made our score worse. However, I took all available images from scatter, vertical bars, lines and horizontal bars regardless if there were more than one data series in the chart or not. Btw, I remember you guys had a big jump from 0.79 to 0.87 like in one submission, what made you had such big increment?

", "votes": null, "replies": [ { "id": 2310587, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 13:47:37", "content": "

All of the subs below 0.8 were without scatter inference. We made our first full sub the night before and it crashed 😅 so I had to wait until 2AM of the last day since we were out of subs.

\n

I overwrote scatter with matcha predictions by accident. Luckily it was an easy bug to spot.

", "votes": null, "replies": [ { "id": 2310590, "author_name": "alejopaullier", "author_url": "", "post_date": "06/20/2023 13:49:48", "content": "

Living life to the limit! 😂 Good you spotted that since scatter has a larger proportion in private LB than in public LB!

", "votes": null, "replies": [ { "id": 2310596, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 13:52:26", "content": "

We should have probed for #charts for each chart type in the test set. I was surprised by the amount of scatter charts in the private test set.

", "votes": null, "replies": [ { "id": 2310603, "author_name": "alejopaullier", "author_url": "", "post_date": "06/20/2023 13:55:51", "content": "

is there a way to probe private LB? you could have never known there were going to be that many scatters

", "votes": null, "replies": [ { "id": 2310605, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 13:57:37", "content": "

Use a classifier and raise exception of it predicts over X scatter plots.

", "votes": null, "replies": [ { "id": 2310608, "author_name": "alejopaullier", "author_url": "", "post_date": "06/20/2023 13:59:47", "content": "

good to know, I once tried probing LB using timers (freeze time 2 sec every time you find X chart), but couldn't get much sense out of it

", "votes": null, "replies": [ { "id": 2310613, "author_name": "crodoc", "author_url": "", "post_date": "06/20/2023 14:02:11", "content": "

Inference time is not reliable. Exceptions are 😅

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2310170": "Our MIT compliant submission scored 0.87 on public and 0.71 on private LB.\n\nAdding ICDAR boosted public by 0.01, but private stayed at 0.71. Since I was adding it the night they announced it was allowed, I wasn't able to label it perfectly (just filtered out 20k images with a GUI I made, parsed available labels and pseudolabel the rest).\n\nDid you get more out of it? I am wondering if I labeled it better, would it help more.\nDid anything else got some boost out of non-MIT datasets & models?\n\nEDIT: I just realized that adding ICDAR actually dropped our score on private.", "2310192": "I used around 1100 images from ICDAR, only those having 1 XY series. I also did post-processing to ensure annotations match competition graph conventions (e.g. handling percentages, interpolation of line plot data to match tick labels etc). At the start of comp, I got ~0.04 boost in public LB with this data. For my final model, the impact was much smaller (~0.01 in private LB), as I did first-phase of training with a large number (~900k) of self-generated synthetic plots.", "2310296": "The boost is not that big for you as well. You probably had a bit better labels for ICDAR compared to us.", "2310375": "crodoc good topic and good questions! Thank you! Can you share your code?", "2310414": "Planning to put it on git :)", "2310444": "Congrats! Looking forward to learning from your solution.", "2310527": "conjuring92 did you use scatter charts from ICDAR?", "2310552": "crodoc in our case it made our score worse. However, I took all available images from scatter, vertical bars, lines and horizontal bars regardless if there were more than one data series in the chart or not. Btw, I remember you guys had a big jump from 0.79 to 0.87 like in one submission, what made you had such big increment?", "2310587": "All of the subs below 0.8 were without scatter inference. We made our first full sub the night before and it crashed 😅 so I had to wait until 2AM of the last day since we were out of subs.\n\nI overwrote scatter with matcha predictions by accident. Luckily it was an easy bug to spot.", "2310590": "Living life to the limit! 😂 Good you spotted that since scatter has a larger proportion in private LB than in public LB!", "2310596": "We should have probed for #charts for each chart type in the test set. I was surprised by the amount of scatter charts in the private test set.", "2310603": "is there a way to probe private LB? you could have never known there were going to be that many scatters", "2310605": "Use a classifier and raise exception of it predicts over X scatter plots.", "2310608": "good to know, I once tried probing LB using timers (freeze time 2 sec every time you find X chart), but couldn't get much sense out of it", "2310613": "Inference time is not reliable. Exceptions are 😅", "2310762": "yes, I had 150+ scatter from ICDAR", "2310920": "I think you were better on scatter by 0.01 :)" }, "source": "meta" }