{ "id": 176066, "title": "Does changing dataset setting after the competition close affect the re-run?", "url": "/competitions/global-wheat-detection/discussion/176066", "author_name": "", "post_date": "2020-08-20T10:37:36.231953300Z", "votes": 8, "comment_count": 12, "views": 0, "content": "

Similar to @authman, the submissions of my team was mark as error with \"Failed final private rerun\". After an investigation inside my team, we do not think we break the rule in any cases. We now guess that the reason to error is similar to @authman's. He deleted the dataset after the competition. For our situation, we used a external dataset made from the Wheat Ear Dataset. After the competition was closed, we shared that dataset and our labels here. We used this in our two selected submissions for pseudo labelling. When I make that dataset public, I turned off the versioning because the first version was mislabeled and I want that dataset to be clean for publishing. The problem might be the dataset was unversioned when the notebook was re-run and our notebook used the version 2 (they are actually the identical version!), so the re-run program cannot find the correct dataset then we get an error. Now I just turn the setting back to enable versioning. If that is the problem, our notebook should be fine now.

\n

I think there are many people having the similar situation to ours. We need the Kaggle team to answer why we get the error. It is very frustrating if my team is disqualified because I share our solution after the competition closed. We obeyed the rule, waited a long time. No matter we get medals or not, we just want a result. @philculliton Could you please make any explanations?

\n

If your team have the same issue, please check if you made any modifications to your dataset or notebook. Or you can post your situation below, we can help you to infer your problems.

", "messages": [ { "id": "978706", "postDate": "08/20/2020 10:37:36", "content": "

Similar to @authman, the submissions of my team was mark as error with \"Failed final private rerun\". After an investigation inside my team, we do not think we break the rule in any cases. We now guess that the reason to error is similar to @authman's. He deleted the dataset after the competition. For our situation, we used a external dataset made from the Wheat Ear Dataset. After the competition was closed, we shared that dataset and our labels here. We used this in our two selected submissions for pseudo labelling. When I make that dataset public, I turned off the versioning because the first version was mislabeled and I want that dataset to be clean for publishing. The problem might be the dataset was unversioned when the notebook was re-run and our notebook used the version 2 (they are actually the identical version!), so the re-run program cannot find the correct dataset then we get an error. Now I just turn the setting back to enable versioning. If that is the problem, our notebook should be fine now.

\n

I think there are many people having the similar situation to ours. We need the Kaggle team to answer why we get the error. It is very frustrating if my team is disqualified because I share our solution after the competition closed. We obeyed the rule, waited a long time. No matter we get medals or not, we just want a result. @philculliton Could you please make any explanations?

\n

If your team have the same issue, please check if you made any modifications to your dataset or notebook. Or you can post your situation below, we can help you to infer your problems.

", "rawMarkdown": "Similar to @authman, the submissions of my team was mark as error with \"Failed final private rerun\". After an investigation inside my team, we do not think we break the rule in any cases. We now guess that the reason to error is similar to @authman's. He deleted the dataset after the competition. For our situation, we used a external dataset made from the [Wheat Ear Dataset](https://zenodo.org/record/1471626#.W9H_XFUzZjU). After the competition was closed, we shared that dataset and our labels [here](https://www.kaggle.com/qingyuanwang/globalwheatdetectionextend). We used this in our two selected submissions for pseudo labelling. When I make that dataset public, **I turned off the versioning** because the first version was mislabeled and I want that dataset to be clean for publishing. The problem might be **the dataset was unversioned when the notebook was re-run and our notebook used the version 2** (they are actually the identical version!), so the re-run program cannot find the correct dataset then we get an error. Now I just turn the setting back to enable versioning. If that is the problem, our notebook should be fine now.\n\nI think there are many people having the similar situation to ours. We need the Kaggle team to answer why we get the error. It is very frustrating if my team is disqualified because I share our solution after the competition closed. We obeyed the rule, waited a long time. No matter we get medals or not, we just want a result. @philculliton Could you please make any explanations?\n \nIf your team have the same issue, please check if you made any modifications to your dataset or notebook. Or you can post your situation below, we can help you to infer your problems.", "votes": null }, { "id": "979246", "postDate": "08/20/2020 17:58:33", "content": "

Best thing to do if you think your score was incorrectly removed is contact the compliance team here: https://www.kaggle.com/compliance

", "rawMarkdown": "Best thing to do if you think your score was incorrectly removed is contact the compliance team here: https://www.kaggle.com/compliance", "votes": null }, { "id": "984184", "postDate": "08/24/2020 23:16:37", "content": "

You were totally robbed and I feel for you. Your situation is even worse than mine because the cause of your issue is 100% a kaggle User Experience problem.

\n
\n

Separately:

\n

One interesting thing we have learned from this fiasco is that Kaggle actually re-runs the kernels upon competition end, potentially multiple times, as what happened with GWD. This is actually a vulnerability.

\n

Suppose I am a malicious user and I upload an unversioned model dataset and use it to infer with my final submission. The competition then ends. Then I update my unversioned dataset (I mean, the competition already ended right? Should be legal, normal, etc. for me to update, delete, or do whatever I want to the dataset, per Inversions comment on the other thread).

\n

If / when Kaggle reruns my model days later, it'll pull the latest edition of the model dataset which I could easily have updated after the completion of the competition, perhaps trained with knowledge gained after the comp's stated conclusion. From what we've heard of how Kaggle works, that's exactly what's going on currently, and I hope they fix it, especially with the launch today of a brand new code competition. I also hope you get your medal back.

", "rawMarkdown": "You were totally robbed and I feel for you. Your situation is even worse than mine because the cause of your issue is 100% a kaggle User Experience problem.\n\n---\n\nSeparately:\n\nOne interesting thing we have learned from this fiasco is that Kaggle _actually_ re-runs the kernels upon competition end, potentially multiple times, as what happened with GWD. **This is actually a vulnerability**.\n\nSuppose I am a malicious user and I upload an unversioned model dataset and use it to infer with my final submission. The competition then ends. Then I update my unversioned dataset (I mean, the competition already ended right? Should be legal, normal, etc. for me to update, delete, or do whatever I want to the dataset, per Inversions comment on the other thread).\n\nIf / when Kaggle reruns my model days later, it'll pull the latest edition of the model dataset which I could easily have updated after the completion of the competition, perhaps trained with knowledge gained after the comp's stated conclusion. From what we've heard of how Kaggle works, that's exactly what's going on currently, and I hope they fix it, especially with the launch today of a brand new code competition. I also hope you get your medal back.", "votes": null }, { "id": "984231", "postDate": "08/25/2020 01:07:53", "content": "

Update:
\nWe just got response from Kaggle Compliance,
\n\"\"

\n

If my understanding is not wrong, Kaggle says they re-runs again with the new dataset(settings), and it still doesn't work.

\n

This reply makes our teammates totally confused. Because we submitted the identical notebook version through the late submission and it gives the score of 0.6543 which deserves a silver medal.

\n

\"\"
\n\"\"

\n

Both are final notebook (version 1/4). From my knowledge, the committed things (should) cannot be changed, so we don't even need to prove they are the same notebook and version. Actually, we just clicked the submit button again. I don't know how Kaggle re-runs our submission or how complex the Kaggle backend is. At least I think they are able to use my account authority to \"click the submit button\".

\n

The strange thing is, Kaggle says they tried several times and still cannot re-run our submissions. But we easily re-run the same notebook version and get a score. I understand for other unlucky team, who deleted their dataset cannot prove (or maybe they can if Kaggle keeps any record)the late submission was identical to they submitted before the end of this competition. However, for our case, there is no reason to believe these two submissions are different. The version of notebook was committed. We know the term \"committed\" means something is \"stored and fixed\" and cannot be changed at least by us. (Or I have some misunderstandings with the term \"commit\"? Oh my college professor, I have to come to you again.)

\n

Interesting, very interesting.

", "rawMarkdown": "Update:\nWe just got response from Kaggle Compliance, \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4650554%2Fda016881f41b9d534a62df5a3881554f%2F2020-08-25%20082323.png?generation=1598315025752125&alt=media)\n\nIf my understanding is not wrong, Kaggle says they re-runs again with the new dataset(settings), and it still doesn't work.\n\nThis reply makes our teammates totally confused. Because we submitted the identical notebook version through the late submission and it gives the score of 0.6543 which deserves a silver medal.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4650554%2Fc89c6325e6b752ab52442f22e74f59f2%2F2020-08-25%20083203.png?generation=1598315563810618&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4650554%2F7be17d962272441da5d6f1594fc40099%2F2020-08-25%20083241.png?generation=1598315581430372&alt=media)\n\nBoth are final notebook (version 1/4). From my knowledge, the committed things (should) cannot be changed, so we don't even need to prove they are the same notebook and version. Actually, we just clicked the submit button again. I don't know how Kaggle re-runs our submission or how complex the Kaggle backend is. At least I think they are able to use my account authority to \"click the submit button\". \n\nThe strange thing is, Kaggle says they tried several times and still cannot re-run our submissions. But we easily re-run the same notebook version and get a score. I understand for other unlucky team, who deleted their dataset cannot prove (or maybe they can if Kaggle keeps any record)the late submission was identical to they submitted before the end of this competition. **However, for our case, there is no reason to believe these two submissions are different.** The version of notebook was committed. We know the term \"committed\" means something is \"stored and fixed\" and cannot be changed at least by us. (Or I have some misunderstandings with the term \"commit\"? Oh my college professor, I have to come to you again.)\n\nInteresting, very interesting.", "votes": null }, { "id": "984232", "postDate": "08/25/2020 01:11:52", "content": "

Hi @authman, thank you for your help, we just got a very confusing response from Kaggle (See my comment below). We don't know what we should do now. I hope you get your deserved medal as well.

", "rawMarkdown": "Hi @authman, thank you for your help, we just got a very confusing response from Kaggle (See my comment below). We don't know what we should do now. I hope you get your deserved medal as well.", "votes": null }, { "id": "984260", "postDate": "08/25/2020 02:06:18", "content": "

I also just now received the same email verbatim, sans the team name.

\n

āŒ˜+C, āŒ˜+V

", "rawMarkdown": "I also just now received the same email verbatim, sans the team name.\n\nāŒ˜+C, āŒ˜+V", "votes": null }, { "id": "984281", "postDate": "08/25/2020 02:38:10", "content": "

@authman Thank you very much for your reply and support. I hope we can all get satisfactory results.šŸ¤•šŸ¤•
\nThe kaggle team 's reply until now was so perfunctory that we were very disappointed.

", "rawMarkdown": "authman Thank you very much for your reply and support. I hope we can all get satisfactory results.šŸ¤•šŸ¤•\nThe kaggle team 's reply until now was so perfunctory that we were very disappointed.", "votes": null }, { "id": "984284", "postDate": "08/25/2020 02:41:25", "content": "

Hi, @matthewmasters .Do u still think best thing to do is contact the compliance team, after looking the reply we receive? I don't think they value our feedback.

", "rawMarkdown": "Hi, @matthewmasters .Do u still think best thing to do is contact the compliance team, after looking the reply we receive? I don't think they value our feedback.", "votes": null }, { "id": "984453", "postDate": "08/25/2020 05:23:34", "content": "

LOL, now I guess they didn't re-run my submissions at all.

", "rawMarkdown": "LOL, now I guess they didn't re-run my submissions at all.", "votes": null }, { "id": "996596", "postDate": "09/03/2020 12:20:29", "content": "

Hello,can you tell me the enhencement of the extend dataset?

", "rawMarkdown": "Hello,can you tell me the enhencement of the extend dataset?", "votes": null }, { "id": "996598", "postDate": "09/03/2020 12:21:22", "content": "

Hello,can you tell me the enhencement of the extend dataset?

", "rawMarkdown": "Hello,can you tell me the enhencement of the extend dataset?", "votes": null }, { "id": "996661", "postDate": "09/03/2020 13:12:49", "content": "

Please refer to the description in https://www.kaggle.com/qingyuanwang/globalwheatdetectionextend

", "rawMarkdown": "Please refer to the description in https://www.kaggle.com/qingyuanwang/globalwheatdetectionextend", "votes": null }, { "id": "996701", "postDate": "09/03/2020 13:54:27", "content": "

Thanks ļ¼I mistake it is not necessary in this game at the first time.

", "rawMarkdown": "Thanks ļ¼I mistake it is not necessary in this game at the first time.", "votes": null } ], "comments": [ { "id": 979246, "author_name": "matthewmasters", "author_url": "", "post_date": "08/20/2020 17:58:33", "content": "

Best thing to do if you think your score was incorrectly removed is contact the compliance team here: https://www.kaggle.com/compliance

", "votes": null, "replies": [ { "id": 984284, "author_name": "li325040229", "author_url": "", "post_date": "08/25/2020 02:41:25", "content": "

Hi, @matthewmasters .Do u still think best thing to do is contact the compliance team, after looking the reply we receive? I don't think they value our feedback.

", "votes": null, "replies": [] } ] }, { "id": 984184, "author_name": "authman", "author_url": "", "post_date": "08/24/2020 23:16:37", "content": "

You were totally robbed and I feel for you. Your situation is even worse than mine because the cause of your issue is 100% a kaggle User Experience problem.

\n
\n

Separately:

\n

One interesting thing we have learned from this fiasco is that Kaggle actually re-runs the kernels upon competition end, potentially multiple times, as what happened with GWD. This is actually a vulnerability.

\n

Suppose I am a malicious user and I upload an unversioned model dataset and use it to infer with my final submission. The competition then ends. Then I update my unversioned dataset (I mean, the competition already ended right? Should be legal, normal, etc. for me to update, delete, or do whatever I want to the dataset, per Inversions comment on the other thread).

\n

If / when Kaggle reruns my model days later, it'll pull the latest edition of the model dataset which I could easily have updated after the completion of the competition, perhaps trained with knowledge gained after the comp's stated conclusion. From what we've heard of how Kaggle works, that's exactly what's going on currently, and I hope they fix it, especially with the launch today of a brand new code competition. I also hope you get your medal back.

", "votes": null, "replies": [ { "id": 984232, "author_name": "qingyuanwang", "author_url": "", "post_date": "08/25/2020 01:11:52", "content": "

Hi @authman, thank you for your help, we just got a very confusing response from Kaggle (See my comment below). We don't know what we should do now. I hope you get your deserved medal as well.

", "votes": null, "replies": [] }, { "id": 984281, "author_name": "li325040229", "author_url": "", "post_date": "08/25/2020 02:38:10", "content": "

@authman Thank you very much for your reply and support. I hope we can all get satisfactory results.šŸ¤•šŸ¤•
\nThe kaggle team 's reply until now was so perfunctory that we were very disappointed.

", "votes": null, "replies": [] } ] }, { "id": 984231, "author_name": "qingyuanwang", "author_url": "", "post_date": "08/25/2020 01:07:53", "content": "

Update:
\nWe just got response from Kaggle Compliance,
\n\"\"

\n

If my understanding is not wrong, Kaggle says they re-runs again with the new dataset(settings), and it still doesn't work.

\n

This reply makes our teammates totally confused. Because we submitted the identical notebook version through the late submission and it gives the score of 0.6543 which deserves a silver medal.

\n

\"\"
\n\"\"

\n

Both are final notebook (version 1/4). From my knowledge, the committed things (should) cannot be changed, so we don't even need to prove they are the same notebook and version. Actually, we just clicked the submit button again. I don't know how Kaggle re-runs our submission or how complex the Kaggle backend is. At least I think they are able to use my account authority to \"click the submit button\".

\n

The strange thing is, Kaggle says they tried several times and still cannot re-run our submissions. But we easily re-run the same notebook version and get a score. I understand for other unlucky team, who deleted their dataset cannot prove (or maybe they can if Kaggle keeps any record)the late submission was identical to they submitted before the end of this competition. However, for our case, there is no reason to believe these two submissions are different. The version of notebook was committed. We know the term \"committed\" means something is \"stored and fixed\" and cannot be changed at least by us. (Or I have some misunderstandings with the term \"commit\"? Oh my college professor, I have to come to you again.)

\n

Interesting, very interesting.

", "votes": null, "replies": [ { "id": 984260, "author_name": "authman", "author_url": "", "post_date": "08/25/2020 02:06:18", "content": "

I also just now received the same email verbatim, sans the team name.

\n

āŒ˜+C, āŒ˜+V

", "votes": null, "replies": [] }, { "id": 984453, "author_name": "qingyuanwang", "author_url": "", "post_date": "08/25/2020 05:23:34", "content": "

LOL, now I guess they didn't re-run my submissions at all.

", "votes": null, "replies": [] }, { "id": 996598, "author_name": "qq1623620766", "author_url": "", "post_date": "09/03/2020 12:21:22", "content": "

Hello,can you tell me the enhencement of the extend dataset?

", "votes": null, "replies": [] } ] }, { "id": 996596, "author_name": "qq1623620766", "author_url": "", "post_date": "09/03/2020 12:20:29", "content": "

Hello,can you tell me the enhencement of the extend dataset?

", "votes": null, "replies": [ { "id": 996661, "author_name": "qingyuanwang", "author_url": "", "post_date": "09/03/2020 13:12:49", "content": "

Please refer to the description in https://www.kaggle.com/qingyuanwang/globalwheatdetectionextend

", "votes": null, "replies": [] }, { "id": 996701, "author_name": "qq1623620766", "author_url": "", "post_date": "09/03/2020 13:54:27", "content": "

Thanks ļ¼I mistake it is not necessary in this game at the first time.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "978706": "Similar to @authman, the submissions of my team was mark as error with \"Failed final private rerun\". After an investigation inside my team, we do not think we break the rule in any cases. We now guess that the reason to error is similar to @authman's. He deleted the dataset after the competition. For our situation, we used a external dataset made from the [Wheat Ear Dataset](https://zenodo.org/record/1471626#.W9H_XFUzZjU). After the competition was closed, we shared that dataset and our labels [here](https://www.kaggle.com/qingyuanwang/globalwheatdetectionextend). We used this in our two selected submissions for pseudo labelling. When I make that dataset public, **I turned off the versioning** because the first version was mislabeled and I want that dataset to be clean for publishing. The problem might be **the dataset was unversioned when the notebook was re-run and our notebook used the version 2** (they are actually the identical version!), so the re-run program cannot find the correct dataset then we get an error. Now I just turn the setting back to enable versioning. If that is the problem, our notebook should be fine now.\n\nI think there are many people having the similar situation to ours. We need the Kaggle team to answer why we get the error. It is very frustrating if my team is disqualified because I share our solution after the competition closed. We obeyed the rule, waited a long time. No matter we get medals or not, we just want a result. @philculliton Could you please make any explanations?\n \nIf your team have the same issue, please check if you made any modifications to your dataset or notebook. Or you can post your situation below, we can help you to infer your problems.", "979246": "Best thing to do if you think your score was incorrectly removed is contact the compliance team here: https://www.kaggle.com/compliance", "984184": "You were totally robbed and I feel for you. Your situation is even worse than mine because the cause of your issue is 100% a kaggle User Experience problem.\n\n---\n\nSeparately:\n\nOne interesting thing we have learned from this fiasco is that Kaggle _actually_ re-runs the kernels upon competition end, potentially multiple times, as what happened with GWD. **This is actually a vulnerability**.\n\nSuppose I am a malicious user and I upload an unversioned model dataset and use it to infer with my final submission. The competition then ends. Then I update my unversioned dataset (I mean, the competition already ended right? Should be legal, normal, etc. for me to update, delete, or do whatever I want to the dataset, per Inversions comment on the other thread).\n\nIf / when Kaggle reruns my model days later, it'll pull the latest edition of the model dataset which I could easily have updated after the completion of the competition, perhaps trained with knowledge gained after the comp's stated conclusion. From what we've heard of how Kaggle works, that's exactly what's going on currently, and I hope they fix it, especially with the launch today of a brand new code competition. I also hope you get your medal back.", "984231": "Update:\nWe just got response from Kaggle Compliance, \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4650554%2Fda016881f41b9d534a62df5a3881554f%2F2020-08-25%20082323.png?generation=1598315025752125&alt=media)\n\nIf my understanding is not wrong, Kaggle says they re-runs again with the new dataset(settings), and it still doesn't work.\n\nThis reply makes our teammates totally confused. Because we submitted the identical notebook version through the late submission and it gives the score of 0.6543 which deserves a silver medal.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4650554%2Fc89c6325e6b752ab52442f22e74f59f2%2F2020-08-25%20083203.png?generation=1598315563810618&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4650554%2F7be17d962272441da5d6f1594fc40099%2F2020-08-25%20083241.png?generation=1598315581430372&alt=media)\n\nBoth are final notebook (version 1/4). From my knowledge, the committed things (should) cannot be changed, so we don't even need to prove they are the same notebook and version. Actually, we just clicked the submit button again. I don't know how Kaggle re-runs our submission or how complex the Kaggle backend is. At least I think they are able to use my account authority to \"click the submit button\". \n\nThe strange thing is, Kaggle says they tried several times and still cannot re-run our submissions. But we easily re-run the same notebook version and get a score. I understand for other unlucky team, who deleted their dataset cannot prove (or maybe they can if Kaggle keeps any record)the late submission was identical to they submitted before the end of this competition. **However, for our case, there is no reason to believe these two submissions are different.** The version of notebook was committed. We know the term \"committed\" means something is \"stored and fixed\" and cannot be changed at least by us. (Or I have some misunderstandings with the term \"commit\"? Oh my college professor, I have to come to you again.)\n\nInteresting, very interesting.", "984232": "Hi @authman, thank you for your help, we just got a very confusing response from Kaggle (See my comment below). We don't know what we should do now. I hope you get your deserved medal as well.", "984260": "I also just now received the same email verbatim, sans the team name.\n\nāŒ˜+C, āŒ˜+V", "984281": "authman Thank you very much for your reply and support. I hope we can all get satisfactory results.šŸ¤•šŸ¤•\nThe kaggle team 's reply until now was so perfunctory that we were very disappointed.", "984284": "Hi, @matthewmasters .Do u still think best thing to do is contact the compliance team, after looking the reply we receive? I don't think they value our feedback.", "984453": "LOL, now I guess they didn't re-run my submissions at all.", "996596": "Hello,can you tell me the enhencement of the extend dataset?", "996598": "Hello,can you tell me the enhencement of the extend dataset?", "996661": "Please refer to the description in https://www.kaggle.com/qingyuanwang/globalwheatdetectionextend", "996701": "Thanks ļ¼I mistake it is not necessary in this game at the first time." }, "source": "meta" }