I was wondering if there are any issues that could arise with the scoring metric. We can already see that predicting no events, leads to a submission scoring error. Is there any chance that e.g. predicting too many events could lead to such an error as well? Perhaps due to OOM errors on the server side when evaluating the submission? If that is the case it could break submissions on the private set, since it is several times larger than the public test set. It would be nice to have clarification on that @ryanholbrook . Thanks!
", "messages": [ { "id": "2538875", "postDate": "11/26/2023 14:05:32", "content": "I was wondering if there are any issues that could arise with the scoring metric. We can already see that predicting no events, leads to a submission scoring error. Is there any chance that e.g. predicting too many events could lead to such an error as well? Perhaps due to OOM errors on the server side when evaluating the submission? If that is the case it could break submissions on the private set, since it is several times larger than the public test set. It would be nice to have clarification on that @ryanholbrook . Thanks!
", "rawMarkdown": "I was wondering if there are any issues that could arise with the scoring metric. We can already see that predicting no events, leads to a submission scoring error. Is there any chance that e.g. predicting too many events could lead to such an error as well? Perhaps due to OOM errors on the server side when evaluating the submission? If that is the case it could break submissions on the private set, since it is several times larger than the public test set. It would be nice to have clarification on that @ryanholbrook . Thanks!", "votes": null }, { "id": "2538888", "postDate": "11/26/2023 14:11:18", "content": "The public and private test sets are scored at the same time, but you only see the public result. So if you get a result from the submission, then you can be sure that the private test set has a result as well!
", "rawMarkdown": "The public and private test sets are scored at the same time, but you only see the public result. So if you get a result from the submission, then you can be sure that the private test set has a result as well!", "votes": null }, { "id": "2538906", "postDate": "11/26/2023 14:24:06", "content": "I know that I already make predictions for the private set already, but I wasn't sure that the result on the private set is necessarily valid just because the displayed score is valid. If the metrics are calculated for the public and the private series, it would be possible that the public score is valid but the private score yields a submission scoring error.
", "rawMarkdown": "I know that I already make predictions for the private set already, but I wasn't sure that the result on the private set is necessarily valid just because the displayed score is valid. If the metrics are calculated for the public and the private series, it would be possible that the public score is valid but the private score yields a submission scoring error.", "votes": null }, { "id": "2538915", "postDate": "11/26/2023 14:32:16", "content": "Ah, I see. I've asked about that before, and if either the public or private set has a submission scoring error, then you will get that error on your submission - errors from both are propagated and displayed
", "rawMarkdown": "Ah, I see. I've asked about that before, and if either the public or private set has a submission scoring error, then you will get that error on your submission - errors from both are propagated and displayed", "votes": null }, { "id": "2538934", "postDate": "11/26/2023 14:49:43", "content": "I think another possible problem apart from errors is the noise of the labels. Not only are some series only partially annotated, some series have annotations that are quite bad (by manual inspection), and it's quite probable that some of the problems also happen in the private test set.
\nUsing the LB to tune hyperparameters/post processing parameters seem to be at risk of overfitting to both LB+CV. Probably have to think of a model with a good prior that generalises well to unseen data.
", "rawMarkdown": "I think another possible problem apart from errors is the noise of the labels. Not only are some series only partially annotated, some series have annotations that are quite bad (by manual inspection), and it's quite probable that some of the problems also happen in the private test set.\n\nUsing the LB to tune hyperparameters/post processing parameters seem to be at risk of overfitting to both LB+CV. Probably have to think of a model with a good prior that generalises well to unseen data.", "votes": null }, { "id": "2539043", "postDate": "11/26/2023 16:33:04", "content": "Ah thanks a lot, that is indeed helpful!
", "rawMarkdown": "Ah thanks a lot, that is indeed helpful!", "votes": null }, { "id": "2544230", "postDate": "11/30/2023 17:10:18", "content": "Hey @chris62, I am still a bit paranoid about this and couldn't find anything confirming this. Are you sure that is how it is handled/ do you perhaps have a link to the discussion thread where this was discussed? Or @ryanholbrook can you confirm this? Thanks a lot!
", "rawMarkdown": "Hey @chris62, I am still a bit paranoid about this and couldn't find anything confirming this. Are you sure that is how it is handled/ do you perhaps have a link to the discussion thread where this was discussed? Or @ryanholbrook can you confirm this? Thanks a lot!", "votes": null }, { "id": "2544243", "postDate": "11/30/2023 17:20:12", "content": "I have no link to share, but I can assure you he is correct. if you see a score, your submission is fine
", "rawMarkdown": "I have no link to share, but I can assure you he is correct. if you see a score, your submission is fine", "votes": null }, { "id": "2544576", "postDate": "11/30/2023 23:51:23", "content": "I've tried to detect and remove all those \"repeated patterns\". It increased score locally but dropped significantly on the leader board. For a simple UNet1D it was 0.41 -> 0.44 locally and 0.31 -> 0.1 on the leaderboard. The labels here look quite messy.
", "rawMarkdown": "I've tried to detect and remove all those \"repeated patterns\". It increased score locally but dropped significantly on the leader board. For a simple UNet1D it was 0.41 -> 0.44 locally and 0.31 -> 0.1 on the leaderboard. The labels here look quite messy.", "votes": null } ], "comments": [ { "id": 2538888, "author_name": "chris62", "author_url": "", "post_date": "11/26/2023 14:11:18", "content": "The public and private test sets are scored at the same time, but you only see the public result. So if you get a result from the submission, then you can be sure that the private test set has a result as well!
", "votes": null, "replies": [ { "id": 2538906, "author_name": "fritzcremer", "author_url": "", "post_date": "11/26/2023 14:24:06", "content": "I know that I already make predictions for the private set already, but I wasn't sure that the result on the private set is necessarily valid just because the displayed score is valid. If the metrics are calculated for the public and the private series, it would be possible that the public score is valid but the private score yields a submission scoring error.
", "votes": null, "replies": [ { "id": 2538915, "author_name": "chris62", "author_url": "", "post_date": "11/26/2023 14:32:16", "content": "Ah, I see. I've asked about that before, and if either the public or private set has a submission scoring error, then you will get that error on your submission - errors from both are propagated and displayed
", "votes": null, "replies": [ { "id": 2539043, "author_name": "fritzcremer", "author_url": "", "post_date": "11/26/2023 16:33:04", "content": "Ah thanks a lot, that is indeed helpful!
", "votes": null, "replies": [] }, { "id": 2544230, "author_name": "fritzcremer", "author_url": "", "post_date": "11/30/2023 17:10:18", "content": "Hey @chris62, I am still a bit paranoid about this and couldn't find anything confirming this. Are you sure that is how it is handled/ do you perhaps have a link to the discussion thread where this was discussed? Or @ryanholbrook can you confirm this? Thanks a lot!
", "votes": null, "replies": [ { "id": 2544243, "author_name": "nyleve", "author_url": "", "post_date": "11/30/2023 17:20:12", "content": "I have no link to share, but I can assure you he is correct. if you see a score, your submission is fine
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2538934, "author_name": "chauyh", "author_url": "", "post_date": "11/26/2023 14:49:43", "content": "I think another possible problem apart from errors is the noise of the labels. Not only are some series only partially annotated, some series have annotations that are quite bad (by manual inspection), and it's quite probable that some of the problems also happen in the private test set.
\nUsing the LB to tune hyperparameters/post processing parameters seem to be at risk of overfitting to both LB+CV. Probably have to think of a model with a good prior that generalises well to unseen data.
", "votes": null, "replies": [ { "id": 2544576, "author_name": "nhargan", "author_url": "", "post_date": "11/30/2023 23:51:23", "content": "I've tried to detect and remove all those \"repeated patterns\". It increased score locally but dropped significantly on the leader board. For a simple UNet1D it was 0.41 -> 0.44 locally and 0.31 -> 0.1 on the leaderboard. The labels here look quite messy.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2538875": "I was wondering if there are any issues that could arise with the scoring metric. We can already see that predicting no events, leads to a submission scoring error. Is there any chance that e.g. predicting too many events could lead to such an error as well? Perhaps due to OOM errors on the server side when evaluating the submission? If that is the case it could break submissions on the private set, since it is several times larger than the public test set. It would be nice to have clarification on that @ryanholbrook . Thanks!", "2538888": "The public and private test sets are scored at the same time, but you only see the public result. So if you get a result from the submission, then you can be sure that the private test set has a result as well!", "2538906": "I know that I already make predictions for the private set already, but I wasn't sure that the result on the private set is necessarily valid just because the displayed score is valid. If the metrics are calculated for the public and the private series, it would be possible that the public score is valid but the private score yields a submission scoring error.", "2538915": "Ah, I see. I've asked about that before, and if either the public or private set has a submission scoring error, then you will get that error on your submission - errors from both are propagated and displayed", "2538934": "I think another possible problem apart from errors is the noise of the labels. Not only are some series only partially annotated, some series have annotations that are quite bad (by manual inspection), and it's quite probable that some of the problems also happen in the private test set.\n\nUsing the LB to tune hyperparameters/post processing parameters seem to be at risk of overfitting to both LB+CV. Probably have to think of a model with a good prior that generalises well to unseen data.", "2539043": "Ah thanks a lot, that is indeed helpful!", "2544230": "Hey @chris62, I am still a bit paranoid about this and couldn't find anything confirming this. Are you sure that is how it is handled/ do you perhaps have a link to the discussion thread where this was discussed? Or @ryanholbrook can you confirm this? Thanks a lot!", "2544243": "I have no link to share, but I can assure you he is correct. if you see a score, your submission is fine", "2544576": "I've tried to detect and remove all those \"repeated patterns\". It increased score locally but dropped significantly on the leader board. For a simple UNet1D it was 0.41 -> 0.44 locally and 0.31 -> 0.1 on the leaderboard. The labels here look quite messy." }, "source": "meta" }