I have a model that is doing far, far worse on the test set than on the train set. But, I don't think it is due to overfitting since I only look at roughly 50 images of the training set and get a much better score on the remaining charts in the training set.
\nMy model is currently only focused on numeric y-values (or x-values for horizontal bar or x- and y- values for scatter). I also am not reading the axis labels at all and just assuming each axis goes from 0 to 100. Clearly that can be improved, but I am expecting better submission scores.
\nMy best submission score is 0.01 (currently only non-zero score on leaderboard). That score can be achieved by just guessing 'vertical_bar' each time with x labels of '1' repeated 10 times and y labels of 50 repeated 10 times. But, I get the same score (or sometimes 0) with a model that actually does something. On the train set I am getting overall average scores of roughly 0.16.
\nSome potential thoughts:
\nI have a model that is doing far, far worse on the test set than on the train set. But, I don't think it is due to overfitting since I only look at roughly 50 images of the training set and get a much better score on the remaining charts in the training set.
\nMy model is currently only focused on numeric y-values (or x-values for horizontal bar or x- and y- values for scatter). I also am not reading the axis labels at all and just assuming each axis goes from 0 to 100. Clearly that can be improved, but I am expecting better submission scores.
\nMy best submission score is 0.01 (currently only non-zero score on leaderboard). That score can be achieved by just guessing 'vertical_bar' each time with x labels of '1' repeated 10 times and y labels of 50 repeated 10 times. But, I get the same score (or sometimes 0) with a model that actually does something. On the train set I am getting overall average scores of roughly 0.16.
\nSome potential thoughts:
\nIf you look at the evaluation section, it says
\n\n\nWe first check the number of values and type of chart predicted for a series. If either of these is different than that in the ground-truth, the score for that series is 0.
\n
In your approach, all examples that are not 'vertical_bar' will score 0. All examples that are 'vertical_bar' that do not have 10 items will score 0. I would assume that there are very few examples in the test set that are vertical bar charts with 10 items
", "rawMarkdown": "If you look at the [evaluation section](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/overview/evaluation), it says\n\n>We first check the number of values and type of chart predicted for a series. If either of these is different than that in the ground-truth, the score for that series is 0. \n\n\nIn your approach, all examples that are not 'vertical_bar' will score 0. All examples that are 'vertical_bar' that do not have 10 items will score 0. I would assume that there are very few examples in the test set that are vertical bar charts with 10 items", "votes": null }, { "id": "2198056", "postDate": "03/26/2023 16:45:06", "content": "The all 'vertical_bar' is a dummy approach that gets a 0.01 score. What I am confused by is an approach that is doing some processing that includes roughly determining chart type and numerical values is getting the same score (or sometimes 0 depending on some parameters). The approach that is actually doing processing scores around 0.16 (with a 0 if chart type or number of values is incorrect) on the entirety of the train set.
", "rawMarkdown": "The all 'vertical_bar' is a dummy approach that gets a 0.01 score. What I am confused by is an approach that is doing some processing that includes roughly determining chart type and numerical values is getting the same score (or sometimes 0 depending on some parameters). The approach that is actually doing processing scores around 0.16 (with a 0 if chart type or number of values is incorrect) on the entirety of the train set.", "votes": null }, { "id": "2199010", "postDate": "03/27/2023 12:29:49", "content": "One thing to keep in mind is that the test set only has extracted graphs, not generated. The extracted graphs are generally more difficult. If you aren't already, you might try forming your validation set only from the extracted set of graphs.
Thanks. I am getting a score of about 0.013 on the extracted part of the train set. So, that could very well be why I am seeing what I am seeing.
", "rawMarkdown": "Thanks. I am getting a score of about 0.013 on the extracted part of the train set. So, that could very well be why I am seeing what I am seeing.", "votes": null } ], "comments": [ { "id": 2198007, "author_name": "nbroad", "author_url": "", "post_date": "03/26/2023 16:02:17", "content": "If you look at the evaluation section, it says
\n\n\nWe first check the number of values and type of chart predicted for a series. If either of these is different than that in the ground-truth, the score for that series is 0.
\n
In your approach, all examples that are not 'vertical_bar' will score 0. All examples that are 'vertical_bar' that do not have 10 items will score 0. I would assume that there are very few examples in the test set that are vertical bar charts with 10 items
", "votes": null, "replies": [ { "id": 2198056, "author_name": "pabyrnes", "author_url": "", "post_date": "03/26/2023 16:45:06", "content": "The all 'vertical_bar' is a dummy approach that gets a 0.01 score. What I am confused by is an approach that is doing some processing that includes roughly determining chart type and numerical values is getting the same score (or sometimes 0 depending on some parameters). The approach that is actually doing processing scores around 0.16 (with a 0 if chart type or number of values is incorrect) on the entirety of the train set.
", "votes": null, "replies": [] } ] }, { "id": 2199010, "author_name": "ryanholbrook", "author_url": "", "post_date": "03/27/2023 12:29:49", "content": "One thing to keep in mind is that the test set only has extracted graphs, not generated. The extracted graphs are generally more difficult. If you aren't already, you might try forming your validation set only from the extracted set of graphs.
Thanks. I am getting a score of about 0.013 on the extracted part of the train set. So, that could very well be why I am seeing what I am seeing.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2197869": "I have a model that is doing far, far worse on the test set than on the train set. But, I don't think it is due to overfitting since I only look at roughly 50 images of the training set and get a much better score on the remaining charts in the training set.\n\nMy model is currently only focused on numeric y-values (or x-values for horizontal bar or x- and y- values for scatter). I also am not reading the axis labels at all and just assuming each axis goes from 0 to 100. Clearly that can be improved, but I am expecting better submission scores.\n\nMy best submission score is 0.01 (currently only non-zero score on leaderboard). That score can be achieved by just guessing 'vertical_bar' each time with x labels of '1' repeated 10 times and y labels of 50 repeated 10 times. But, I get the same score (or sometimes 0) with a model that actually does something. On the train set I am getting overall average scores of roughly 0.16.\n\nSome potential thoughts:\n1. Something is wrong with the evaluation metric.\n2. The test set is very different from the train set in ways that matter to my model (I'm certainly expecting differences here, but not to the point of getting essentially nothing right on the test set).\n3. I'm doing something wrong when constructing submissions.csv.\n4. I'm doing something wrong inherent to a code competition (never really done one of those before).", "2198007": "If you look at the [evaluation section](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/overview/evaluation), it says\n\n>We first check the number of values and type of chart predicted for a series. If either of these is different than that in the ground-truth, the score for that series is 0. \n\n\nIn your approach, all examples that are not 'vertical_bar' will score 0. All examples that are 'vertical_bar' that do not have 10 items will score 0. I would assume that there are very few examples in the test set that are vertical bar charts with 10 items", "2198056": "The all 'vertical_bar' is a dummy approach that gets a 0.01 score. What I am confused by is an approach that is doing some processing that includes roughly determining chart type and numerical values is getting the same score (or sometimes 0 depending on some parameters). The approach that is actually doing processing scores around 0.16 (with a 0 if chart type or number of values is incorrect) on the entirety of the train set.", "2199010": "One thing to keep in mind is that the test set only has `extracted` graphs, not `generated`. The `extracted` graphs are generally more difficult. If you aren't already, you might try forming your validation set only from the `extracted` set of graphs.", "2199174": "Thanks. I am getting a score of about 0.013 on the extracted part of the train set. So, that could very well be why I am seeing what I am seeing." }, "source": "meta" }