![\"fcplm\"](\"https://habrastorage.org/webt/qf/rp/lz/qfrplzpgalzv13zgni6aesrrzw4.jpeg\")
There have been a lot of posts about the removal of those who did not upload a model before the end of Round 1 for this competition. The spirit of 2-stage competitions is that we are getting access to a unique dataset but in limited quantities. For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.
\n\nWhile it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. We discussed this internally at Kaggle and decided that we are going to change our practice going forward. We will be updating our 2-Stage competition rules and will follow this practice going forward.
", "messages": [ { "id": 318213, "postDate": "2018-04-23T12:13:33.740Z", "content": "We here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions), based on the FAQ which stated this. We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!
", "rawMarkdown": "We here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions), based on the FAQ which stated this. We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!", "votes": 39, "replies": [ { "id": 318376, "postDate": "2018-04-23T17:42:52.530Z", "content": "Way to go, Broad team!
", "rawMarkdown": "Way to go, Broad team! ", "votes": -1 }, { "id": 318380, "postDate": "2018-04-23T17:51:53.913Z", "content": "It means a lot to hear you say that.
\n\nPeople spent a long time on your data.
", "rawMarkdown": "It means a lot to hear you say that. \n\nPeople spent a long time on your data.", "votes": 1 }, { "id": 318455, "postDate": "2018-04-23T21:04:02.383Z", "content": "Thanks, Board team for considering. Please understand that people with multiple submissions have put effort, time and money(running models on AWS and floydhub etc..costs) to prepare submissions and at least we deserve a mention that we did a project as such
", "rawMarkdown": "Thanks, Board team for considering. Please understand that people with multiple submissions have put effort, time and money(running models on AWS and floydhub etc..costs) to prepare submissions and at least we deserve a mention that we did a project as such" }, { "id": 318470, "postDate": "2018-04-23T22:21:18.237Z", "content": "Hi, Anne, while we are waiting for the teams of the hard working kagglers to come back to the LB, I just found out additional teams (including ours) got removed today...
", "rawMarkdown": "Hi, Anne, while we are waiting for the teams of the hard working kagglers to come back to the LB, I just found out additional teams (including ours) got removed today..." } ] }, { "id": 317523, "postDate": "2018-04-21T19:30:26.523Z", "content": "It was not included in the rules https://www.kaggle.com/c/data-science-bowl-2018/rules
\n\nIt was not included in the FAQ https://www.kaggle.com/two-stage-faq
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do\n submit in the second stage?
\n \nYes. However, if you do not upload a model and finish in a prize\n position, your team will be removed from the competition standings\n entirely.
\n
Ex post facto laws are expressly forbidden by the United States Constitution in Article 1, Section 9, Clause 3 (with respect to federal laws) and Article 1, Section 10 (with respect to state laws).
\n\nSo are you going to reinstate removed teams for this competition and change the rules from now on?\nNo-one is infallible. I think Kaggle is creating an unnecessary issue out of this. You could just accept that you didn't communicate this rule properly and on time, restore removed submissions and move on.
", "rawMarkdown": "It was not included in the rules [https://www.kaggle.com/c/data-science-bowl-2018/rules][1]\n\nIt was not included in the FAQ [https://www.kaggle.com/two-stage-faq][2]\n\n> Will I still be on the leaderboard if I don't upload a model but do\n> submit in the second stage?\n> \n> Yes. However, if you do not upload a model and finish in a prize\n> position, your team will be removed from the competition standings\n> entirely.\n\nEx post facto laws are expressly forbidden by the United States Constitution in Article 1, Section 9, Clause 3 (with respect to federal laws) and Article 1, Section 10 (with respect to state laws).\n\nSo are you going to reinstate removed teams for this competition and change the rules from now on?\nNo-one is infallible. I think Kaggle is creating an unnecessary issue out of this. You could just accept that you didn't communicate this rule properly and on time, restore removed submissions and move on.\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/rules\n [2]: https://www.kaggle.com/two-stage-faq", "votes": 29 }, { "id": 317539, "postDate": "2018-04-21T20:17:27.967Z", "content": "Maggie, can you please point to the clause in the rules that I signed which states or implies that not uploading a model before a certain deadline leads to a COMPLETE team disqualification with removal from LB and removal of all submissions history etc... even if the team is not a winner?
", "rawMarkdown": "Maggie, can you please point to the clause in the rules that I signed which states or implies that not uploading a model before a certain deadline leads to a COMPLETE team disqualification with removal from LB and removal of all submissions history etc... even if the team is not a winner?", "votes": 25 }, { "id": 317525, "postDate": "2018-04-21T19:31:29.737Z", "content": "Dear @Maggie I think it is not good for Kaggle when you remove half of your LB and tell to participants who have been removed that it is their fault. From my point of view it it's not quite honest. The second part of the competition was organized no so well and we have corresponding outcomes. First of all you mentioned the dataset was much smaller in Round 2. it is not true - the test data set for the 1 stage included around 65 images while for the second stage it included approximately 3000 images. Then, for example, in your rules:
\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set. PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points. \n
https://www.kaggle.com/c/data-science-bowl-2018#Timeline\nthere are no words that participant who did not upload their code will be automatically removed from the LB. PLEASE NOTE: says only about submitting prediction in order to be in the LB. In addition, based on my own experience it was not easy to find a place where I should upload the model. Only my teammates helped me to find it out.
", "rawMarkdown": "Dear @Maggie I think it is not good for Kaggle when you remove half of your LB and tell to participants who have been removed that it is their fault. From my point of view it it's not quite honest. The second part of the competition was organized no so well and we have corresponding outcomes. First of all you mentioned ` the dataset was much smaller in Round 2.` it is not true - the test data set for the 1 stage included around 65 images while for the second stage it included approximately 3000 images. Then, for example, in your rules: \n\n ```\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set. PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points. \n```\n\nhttps://www.kaggle.com/c/data-science-bowl-2018#Timeline\nthere are no words that participant who did not upload their code will be automatically removed from the LB. PLEASE NOTE: says only about submitting prediction in order to be in the LB. In addition, based on my own experience it was not easy to find a place where I should upload the model. Only my teammates helped me to find it out. ", "votes": 27, "replies": [ { "id": 317560, "postDate": "2018-04-21T21:24:53.197Z", "content": "Can't believe a Kaggle admin @Maggie would say the dataset was much smaller in Round 2.
Adding another comment:
\n\n\n\n\nWhile it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action.
\n
If @Maggie means significant increase of people who did not upload models, I think the reason is that: among the 800+ teams (twice that of dsb2017), many started doing Kaggle only recently. A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018. When the rules are not clearly written, newcomers are much more likely to be confused about whether they should upload models or not. So the significant increase of people who did not upload models is the result of more new people joining the competition and the rules being unclear.
Since many of us are new to Kaggle, removing 50%+ teams will strongly discourage new people from joining and participating in Kaggle competitions (even for people who did upload models, it's so sad to see so many peers being removed). And when @Maggie says felt we needed to take action based on non-existing rules only after the competition, this reflects badly on Kaggle's internal decision making.
Overall, I believe the rules should be changed to require model upload, but it should only be enforce in future competitions, not this one.
", "rawMarkdown": "Can't believe a Kaggle admin @Maggie would say `the dataset was much smaller in Round 2`. \n\nAdding another comment:\n\n> While it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. \n\nIf @Maggie means `significant increase of people who did not upload models`, I think the reason is that: among the 800+ teams (twice that of dsb2017), many started doing Kaggle only recently. A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018. When the rules are not clearly written, newcomers are much more likely to be confused about whether they should upload models or not. So the significant increase of people who did not upload models is the result of more new people joining the competition and the rules being unclear. \n\nSince many of us are new to Kaggle, removing 50%+ teams will strongly discourage new people from joining and participating in Kaggle competitions (even for people who did upload models, it's so sad to see so many peers being removed). And when @Maggie says `felt we needed to take action` based on non-existing rules only after the competition, this reflects badly on Kaggle's internal decision making. \n\nOverall, I believe the rules should be changed to require model upload, but it should only be enforce in future competitions, not this one. \n", "votes": 16 } ] }, { "id": 317645, "postDate": "2018-04-22T05:16:34.320Z", "content": "Obviously this post shows a lack of respect. Really disappointing.
", "rawMarkdown": "Obviously this post shows a lack of respect. Really disappointing.", "votes": 17 }, { "id": 317912, "postDate": "2018-04-22T20:24:15.040Z", "content": "I recently provided official stats about my Kaggle profile, among those were \"100 competitions\", lol...
\nThis being my first Kaggle disqualification in +3 years, now I am at 99.
Two (probably wrong) thoughts:
\n1) Whats the logic behind... allowing people to submit to Stage 2 if they would be disqualified anyway?
\n2) I am in Kaggle to learn, not to do business, I think submitting hello world as my model will be my approach in the future.
Hi, Maggie. \nYou can't just say \"felt we needed to take action\" without make any announcement before stage1 deadline.
\n\n\" the dataset was much smaller in Round 2 \" is wrong too. There are 670 images in stage1 training set, 65 images in stage1 test set and 3019 images in stage2 test set.
\n\nYou can't change the rules without notifying all the competitors. You can't change the rules after the competition, and then tell us \"You've broken the rules.\" And the rules you have changed are contrary to the previous rules.
\n\n“A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018.” This approach will disappoint those who have devoted much effort to this competition. This will strongly discourage new people from joining and participating in Kaggle competitions.
\n\nI think this explanation will make the contestants who have been removed very disappointed. All my efforts have been wasted. This explanation is unacceptable.
\n\nYou can't remove our result without any reasonable explanation. That is not fair! NOT FAIR!!!!
", "rawMarkdown": "Hi, Maggie. \nYou can't just say \"felt we needed to take action\" without make any announcement before stage1 deadline.\n\n\" the dataset was much smaller in Round 2 \" is wrong too. There are 670 images in stage1 training set, 65 images in stage1 test set and 3019 images in stage2 test set.\n\nYou can't change the rules without notifying all the competitors. You can't change the rules after the competition, and then tell us \"You've broken the rules.\" And the rules you have changed are contrary to the previous rules.\n\n“A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018.” This approach will disappoint those who have devoted much effort to this competition. This will strongly discourage new people from joining and participating in Kaggle competitions.\n\nI think this explanation will make the contestants who have been removed very disappointed. All my efforts have been wasted. This explanation is unacceptable.\n\nYou can't remove our result without any reasonable explanation. That is not fair! **NOT FAIR!!!!**", "votes": 16 }, { "id": 317499, "postDate": "2018-04-21T18:24:20.040Z", "content": "So the change is made after the competition ended. Is this acceptable? And the explanations are not reasonable at all. I posted my questions here.
\n\nFirst, as mentioned by someone else, the rules that we all agree before entering doens't have the mentioned requirement.
\n\nSecond, the base for this immediate change (for this competition) is basically not true.
\n\n\n\n\nFor this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.
\n
Here is the data , in terms of number of predictions we need to make:
\n\nLast year DSB: stage1 had about 200 samples, stage2 had about 500 samples
\n\nThis year DSB: stage1 has < 700 samples, stage2 has > 3000 samples
\n\nI agree that the change is a right step forward. But things should be clear from the beginning. Any changes should be applied for the future not something that is done.
\n\n@William, @Wendy
", "rawMarkdown": "So the change is made after the competition ended. Is this acceptable? And the explanations are not reasonable at all. I posted my questions [here][1]. \n\nFirst, as mentioned by someone else, the rules that we all agree before entering doens't have the mentioned requirement. \n\nSecond, the base for this immediate change (for this competition) is basically not true. \n>For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.\n\nHere is the data , in terms of number of predictions we need to make:\n\nLast year DSB: stage1 had about 200 samples, stage2 had about 500 samples\n\nThis year DSB: stage1 has < 700 samples, stage2 has > 3000 samples\n\nI agree that the change is a right step forward. But things should be clear from the beginning. Any changes should be applied for the future not something that is done. \n\n@William, @Wendy\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55034#317362", "votes": 16 }, { "id": 317690, "postDate": "2018-04-22T09:05:43.573Z", "content": "I would expect to see such notification in rare case scenario but it seems that you're sending this notification like WhatsApp message after every competition depending on your mood . Zillow is the most recent example.
\n\nOn a lighter note, I can imagine what will be the situation in upcoming/ ongoing competitions if this philosophy continues. Few examples below:
\n\nright now and throwing you out immediately. We will continue this practice and will suspend users who use language other than Python :)right now and reducing the prize money based on your country...I and hopefully, most participants see this as misuse of power and request proper escalation.
\n\nAll it takes is to spend sufficient time on writing clear rules before launching any competition.
", "rawMarkdown": "I would expect to see such notification in rare case scenario but it seems that you're sending this notification like WhatsApp message after every competition depending on your **mood** . Zillow is the most recent example. \n\nOn a lighter note, I can imagine what will be the situation in upcoming/ ongoing competitions if this philosophy continues. Few examples below:\n\n- We **felt that** most companies and most people are using Python but you have submitted your model in R so we are updating our rules `right now` and throwing you out immediately. We will continue this practice and will suspend users who use language other than Python :)\n- After seeing the results, we notice that all most winners are from developing country and **felt that** prize money would be too much for them compared to prize money for winners from developed countries so we are updating our rules `right now` and reducing the prize money based on your country...\n\nI and hopefully, most participants see this as **misuse of power** and request proper escalation. \n\n**All it takes is to spend sufficient time on writing clear rules before launching any competition.** ", "votes": 13 }, { "id": 317488, "postDate": "2018-04-21T18:02:26.397Z", "content": "I did not participate in this competition, nor have I participated in other 2-stage competitions, so I am not very familiar with the rules. From my understanding, reading Discussion posts, the rules stated people would not be removed from the rankings if they didn't upload the model (correct me if I'm wrong).
\n\nThen, from this post I get that the organizers decided after the fact that they would go ahead and remove all these people, essentially overriding the rules without notice.
\n\nI'm sorry, but if that is true, this is very unprofessional. I really am disappointed in Kaggle for this one. All these people worked hard for a placement and followed the rules, and now they get removed because \"a significant increase was noticed\"? I get that points and medals are not the end-all-be-all and the goal of the competition was to advance medical research, but scratching away all this work without reason or transparency is not cool.
\n\nI really hope I am misunderstanding this, since it is very upsetting Kaggle staff and organizers can change the rules as they feel without notifying participants. That's my two cents.
", "rawMarkdown": "I did not participate in this competition, nor have I participated in other 2-stage competitions, so I am not very familiar with the rules. From my understanding, reading Discussion posts, the rules stated people would not be removed from the rankings if they didn't upload the model (correct me if I'm wrong).\n\nThen, from this post I get that the organizers decided after the fact that they would go ahead and remove all these people, essentially overriding the rules without notice.\n\nI'm sorry, but if that is true, this is *very* unprofessional. I really am disappointed in Kaggle for this one. All these people worked hard for a placement and followed the rules, and now they get removed because \"a significant increase was noticed\"? I get that points and medals are not the end-all-be-all and the goal of the competition was to advance medical research, but scratching away all this work without reason or transparency is not cool.\n\nI really hope I am misunderstanding this, since it is very upsetting Kaggle staff and organizers can change the rules as they feel without notifying participants. That's my two cents.", "votes": 11 }, { "id": 317476, "postDate": "2018-04-21T17:30:24.960Z", "content": "My and many other teams uploaded model before the deadline, and still were removed from the LB.
Don't you think that it is better to clarify the rules firstly and then take action? Not vice versa
Agreed on 2. About 1, same here, removed with uploaded model.
\n\nA possible explanation of those removals commented on KaggleNoobs is that there can be a problem with aes-256 encripted zipped models (my case). No idea if that is really the issue but as I said, some other kagglers commented to have problems in the past with that.
", "rawMarkdown": "Agreed on 2. About 1, same here, removed with uploaded model.\n\nA possible explanation of those removals commented on KaggleNoobs is that there can be a problem with aes-256 encripted zipped models (my case). No idea if that is really the issue but as I said, some other kagglers commented to have problems in the past with that. \n" }, { "id": 317485, "postDate": "2018-04-21T17:54:41.893Z", "content": "@miguel did you encrypt a zip file or zipped an encrypted file?
", "rawMarkdown": "@miguel did you encrypt a zip file or zipped an encrypted file?" }, { "id": 317498, "postDate": "2018-04-21T18:21:16.027Z", "content": "@Oscar, used 7zip password with aes256 encryption when compressing the file. (Passwording zip files was ok in previous editions of 2 stage competitions as long as password was provided to Kaggle's requirement). Don't really know if that is the issue, or what kind of automated check has been made to uploaded models... hope will find out soon :-)
", "rawMarkdown": "@Oscar, used 7zip password with aes256 encryption when compressing the file. (Passwording zip files was ok in previous editions of 2 stage competitions as long as password was provided to Kaggle's requirement). Don't really know if that is the issue, or what kind of automated check has been made to uploaded models... hope will find out soon :-)" }, { "id": 317508, "postDate": "2018-04-21T18:40:46.683Z", "content": "@miguel Thanks, It should be okay then. I hope you get your case resolved.
", "rawMarkdown": "@miguel Thanks, It should be okay then. I hope you get your case resolved.", "votes": 1 }, { "id": 317515, "postDate": "2018-04-21T19:04:00.053Z", "content": "@miguel_perez, the post I mentioned on the KaggleNoob channel was here.\n https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#317272\n @bsp2020 uploaded sha256sum before the deadline of the first stage and got removed, not quite the same as you. Both of you definitely can appeal the decisions, don't worry about it.
Thanks @Nicole, yes, I see the difference but also similarity in that both ways assure the model is frozen from the end of stage 1 and that are -maybe- not what an automated check would expect. So I was grateful for your info, a reason to a bit more optimistic ! :-)
", "rawMarkdown": "Thanks @Nicole, yes, I see the difference but also similarity in that both ways assure the model is frozen from the end of stage 1 and that are -maybe- not what an automated check would expect. So I was grateful for your info, a reason to a bit more optimistic ! :-) ", "votes": 1 }, { "id": 317616, "postDate": "2018-04-22T02:07:10.673Z", "content": "Still removed with uploaded model before deadline of stage 1, which really confused me. However, I have not received any explanation about it.
", "rawMarkdown": "Still removed with uploaded model before deadline of stage 1, which really confused me. However, I have not received any explanation about it." } ] }, { "id": 317754, "postDate": "2018-04-22T13:23:24.610Z", "content": "Sorry but I'm afraid you're discouraging people from participating in such highly demanding competitions in the future .
\n\nAfter all, is the effort worth it, if one doesn't know how the rules evolve and if such critical decisions are taken after just \" internal discussions\" ?
", "rawMarkdown": "Sorry but I'm afraid you're discouraging people from participating in such highly demanding competitions in the future .\n\nAfter all, is the effort worth it, if one doesn't know how the rules evolve and if such critical decisions are taken after just \" internal discussions\" ? ", "votes": 10 }, { "id": 317788, "postDate": "2018-04-22T15:07:29.973Z", "content": "I won't be popular I guess but here it is. First, let me clear any doubt. I totally disagree with changing rules during a competition. In democracy law are not applied retrospectively. Here they are. Sad and wrong. People should not be removed based on a new policy decided after stage 1.
\n\nThis said, the issue was not the absence of a written statement about the deadline. The issue was a question in the FAQ whose answer could lead people to ignore the deadline and think they'll be safe.
\n\nThe deadline for uploading models was clearly stated in the Timeline page, as well as in Stage2 FAQ. The spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1. The fact that half of stage 2 teams did upload their model is a clear indication that this was quite visible, contrarily to many posts here that say the deadline was not written anywhere.
\n\nYes, something is wrong, but not what many here say is wrong.
", "rawMarkdown": "I won't be popular I guess but here it is. First, let me clear any doubt. I totally disagree with changing rules during a competition. In democracy law are not applied retrospectively. Here they are. Sad and wrong. People should not be removed based on a new policy decided after stage 1.\n\nThis said, the issue was not the absence of a written statement about the deadline. The issue was a question in the FAQ whose answer could lead people to ignore the deadline and think they'll be safe. \n\nThe deadline for uploading models was clearly stated in the Timeline page, as well as in Stage2 FAQ. The spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1. The fact that half of stage 2 teams did upload their model is a clear indication that this was quite visible, contrarily to many posts here that say the deadline was not written anywhere. \n\nYes, something is wrong, but not what many here say is wrong.\n", "votes": 14, "replies": [ { "id": 317796, "postDate": "2018-04-22T15:20:03.777Z", "content": "From my understanding, the rules stated that people didn't have to upload their model to be part of the leaderboard. Then, people were removed because they didn't upload a model, even though it was not stated in the rules.
\n\nI think that is what people are saying. Rules being added after the fact is not respectful to the participants.
\n\nAm I missing something? I didn't participate so I haven't read the rules, but from what I see here this is what must have happened. That is definitely wrong in my book.
", "rawMarkdown": "From my understanding, the rules stated that people didn't have to upload their model to be part of the leaderboard. Then, people were removed because they didn't upload a model, even though it was not stated in the rules.\n\nI think that is what people are saying. Rules being added after the fact is not respectful to the participants.\n\nAm I missing something? I didn't participate so I haven't read the rules, but from what I see here this is what must have happened. That is definitely wrong in my book.", "votes": 1 }, { "id": 317797, "postDate": "2018-04-22T15:20:37.573Z", "content": "\n\n\nThe spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1.
\n
Is reading discussion forum and finding potential new rules a must to compete in Kaggle?
", "rawMarkdown": "> The spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1. \n\nIs reading discussion forum and finding potential new rules a must to compete in Kaggle?" }, { "id": 317802, "postDate": "2018-04-22T15:30:21.870Z", "content": "\n\n\nthe rules stated that people didn't have to upload their model to be part of the leaderboard.
\n
Where did you read this in the Rules page?
\n\nAgain, the source of confusion comes from a FAQ, not the pages of the competition. The competition pages (esp Timeline page) are crystal clear: models must be finalized and uploaded by end of stage1.
", "rawMarkdown": "> the rules stated that people didn't have to upload their model to be part of the leaderboard. \n\nWhere did you read this in the Rules page?\n\nAgain, the source of confusion comes from a FAQ, not the pages of the competition. The competition pages (esp Timeline page) are crystal clear: models must be finalized and uploaded by end of stage1.", "votes": 2 }, { "id": 317803, "postDate": "2018-04-22T15:32:49.467Z", "content": "\n\n\nIs reading discussion forum [...] a must to compete in Kaggle?
\n
I think so, for many reasons. Reading all information available is good practice.
\n\nIf your question is : was the policy change announced at the right place and at the right time? then the answer is clearly no IMHO.
", "rawMarkdown": "> Is reading discussion forum [...] a must to compete in Kaggle?\n\nI think so, for many reasons. Reading all information available is good practice. \n\nIf your question is : was the policy change announced at the right place and at the right time? then the answer is clearly no IMHO.", "votes": 1 }, { "id": 317805, "postDate": "2018-04-22T15:35:35.703Z", "content": "@CMPM: As I said, I have not read the rules as I did not participate. However, people are quoting this:
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
Which clearly states people will not be removed from the LB regardless of uploading a model or not, unless they finish in prize positions.
", "rawMarkdown": "@CMPM: As I said, I have not read the rules as I did not participate. However, people are quoting this:\n\n> Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.\n\nWhich clearly states people will **not** be removed from the LB regardless of uploading a model or not, unless they finish in prize positions.", "votes": 2 }, { "id": 317807, "postDate": "2018-04-22T15:40:36.013Z", "content": "This is the timeline version I can see
\n\n\n\n\nCompetition Timeline
\n
\n Start Date: January 16, 2018
\n Entry deadline: April 9th, 2018
\n Team merger deadline: April 9th, 2018
\n Stage one deadline and stage two data release: April 11, 2018 - PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points.
\n End date: April 16, 2018 - 11:59PM UTC
It is crystal clear that no “model” mentioned.
", "rawMarkdown": "This is the timeline version I can see\n\n> Competition Timeline \nStart Date: January 16, 2018 \nEntry deadline: April 9th, 2018 \nTeam merger deadline: April 9th, 2018 \nStage one deadline and stage two data release: April 11, 2018 - PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points. \nEnd date: April 16, 2018 - 11:59PM UTC\n\nIt is crystal clear that no “model” mentioned.", "votes": 1 }, { "id": 317808, "postDate": "2018-04-22T15:40:44.843Z", "content": "The rules https://www.kaggle.com/c/data-science-bowl-2018/rules are the only binding agreement between Kaggle and participants. Every little detail should be included there. If not, then there is no reason for rule agreement before entering each competition. The forums are for discussion and decision making. After such an important decision, the admins should have communicated it better by updating the rules page, and forcing all participants to agree with the new rules as soon as this decision was made at the next login. The way this was handled is totally unprofessional. Noone is obligated to continuously read the forums and try to sneak every detail, and given the importance of the rule change (disqualification) the forums discussions is not sufficient. They didn't even send an email for God's sake! Besides if you read Maggie's announcement it is clear that such a decision was made only internally and was not communicated properly through the rules update. Let alone that they left the FAQ unchanged.
", "rawMarkdown": "The rules [https://www.kaggle.com/c/data-science-bowl-2018/rules][1] are the only binding agreement between Kaggle and participants. Every little detail should be included there. If not, then there is no reason for rule agreement before entering each competition. The forums are for discussion and decision making. After such an important decision, the admins should have communicated it better by updating the rules page, and forcing all participants to agree with the new rules as soon as this decision was made at the next login. The way this was handled is totally unprofessional. Noone is obligated to continuously read the forums and try to sneak every detail, and given the importance of the rule change (disqualification) the forums discussions is not sufficient. They didn't even send an email for God's sake! Besides if you read Maggie's announcement it is clear that such a decision was made only internally and was not communicated properly through the rules update. Let alone that they left the FAQ unchanged.\n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/rules", "votes": 8 }, { "id": 317809, "postDate": "2018-04-22T15:41:46.567Z", "content": "CPMP is right. Also the main point is FAQ writes:
\n\n\" Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?
\n\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.\"
\n\nNone of us \"finish in a prize posistion\" but we got removed beacause
\n\n\"While it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. \"
", "rawMarkdown": "CPMP is right. Also the main point is FAQ writes: \n\n\" Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\n\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.\"\n\nNone of us \"finish in a prize posistion\" but we got removed beacause \n\n\"While it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. \"", "votes": 6 }, { "id": 317816, "postDate": "2018-04-22T15:55:59.837Z", "rawMarkdown": "", "isDeleted": true }, { "id": 317819, "postDate": "2018-04-22T16:06:09.180Z", "content": "@Anthony Marakis, you say you did not read the rules yet you comment them. Weird.
\n\n@Shitian Ni, not sure how you manage to truncate the Timeline page quote, but here is a part that you seem to keep ignoring:
\n\n\n\n\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline
\n
@Zheng Li, thank you. I'm truly sorry your great solutionisn't rewarded, and I hope the decision will be reversed for you and many other.
\n\nAnd to close my contribution here, I am not the one making the rules, nor do I agree to the way they are modified and enforced after the fact. Don't kill the messenger ;)
", "rawMarkdown": "@Anthony Marakis, you say you did not read the rules yet you comment them. Weird.\n\n@Shitian Ni, not sure how you manage to truncate the Timeline page quote, but here is a part that you seem to keep ignoring: \n\n> April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline\n\n@Zheng Li, thank you. I'm truly sorry your great solutionisn't rewarded, and I hope the decision will be reversed for you and many other.\n\nAnd to close my contribution here, I am not the one making the rules, nor do I agree to the way they are modified and enforced after the fact. Don't kill the messenger ;)", "votes": 2 }, { "id": 317820, "postDate": "2018-04-22T16:09:03.037Z", "content": "Since the issue is becoming gradually very serious, and the admins are not responding to the increased community frustration (no response after the announcement), I am asking for Kaggle's legal department view on this https://www.kaggle.com/c/data-science-bowl-2018/discussion/55131
", "rawMarkdown": "Since the issue is becoming gradually very serious, and the admins are not responding to the increased community frustration (no response after the announcement), I am asking for Kaggle's legal department view on this [https://www.kaggle.com/c/data-science-bowl-2018/discussion/55131][1]\n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55131", "votes": 1 }, { "id": 317822, "postDate": "2018-04-22T16:15:13.757Z", "content": "@CPMP I just quoted the timeline from the Rules page. I just copy&paste that. I believe that the timeline in Rules is the only one we all read, agreed on, and have to obey. And I note that there is a PLEASE NOTE there, again, no words about “model”
", "rawMarkdown": "@CPMP I just quoted the timeline from the Rules page. I just copy&paste that. I believe that the timeline in Rules is the only one we all read, agreed on, and have to obey. And I note that there is a PLEASE NOTE there, again, no words about “model”" }, { "id": 317823, "postDate": "2018-04-22T16:15:20.983Z", "content": "@CPMP: Why is reading the rules a requirement for me to chime in? Pretty much everyone says that what I quoted is in the FAQ, so I take their word for it and add my perspective.
\n\nI'm sorry, but it's frankly childish to sweep what I say under the rug because I was not part of this. If what I'm saying is indeed true, it's true regardless of whether I participated.
", "rawMarkdown": "@CPMP: Why is reading the rules a requirement for me to chime in? Pretty much everyone says that what I quoted is in the FAQ, so I take their word for it and add my perspective.\n\nI'm sorry, but it's frankly childish to sweep what I say under the rug because I was not part of this. If what I'm saying is indeed true, it's true regardless of whether I participated.", "votes": 1 }, { "id": 317828, "postDate": "2018-04-22T16:40:22.600Z", "content": "\n\n\nCPMP wrote
\n \n\n\n \n> Is reading discussion forum and finding potential new rules a must to compete in Kaggle?
\nI think so, for many reasons. Reading all information available is good practice.
\n \nIf your question is : was the policy change announced at the right place and at the right time? then the answer is clearly no IMHO.
\n
@CPMP Can't believe your reply is pointing to a yes to his question. However I take it that your main position is \"Reading all information available is good practice\", and is not actually answering to his question.
\n\nI think what he is asking is, and from this point onward many of our concern should be,
\n\nAfter this episode, do we need to check the forum every now and then for a new rule cooked up that might disqualify us? Clearly the answer is no.
\n\nEven if we checked, its too late. The competition is already over!
\n\nGiving an analogy, will we want to go to the market every now and then to check if there is a new rule the police might send us to jail? Maybe they are saying we can't be wearing green clothes yesterday! Is it alright to breathe?
\n\nClearly the answer is no. This isn't a right way to live.
\n\nRules need to be official and binding, not changed at whims on the forum.
", "rawMarkdown": "\n> **CPMP wrote**\n> \n> > > Is reading discussion forum and finding potential new rules a must to compete in Kaggle?\n> \n> I think so, for many reasons. Reading all information available is good practice. \n> \n> If your question is : was the policy change announced at the right place and at the right time? then the answer is clearly no IMHO.\n\n@CPMP Can't believe your reply is pointing to a yes to his question. However I take it that your main position is \"Reading all information available is good practice\", and is not actually answering to his question. \n\nI think what he is asking is, and from this point onward many of our concern should be,\n\nAfter this episode, do we need to check the forum every now and then for a new rule cooked up that might disqualify us? Clearly the answer is no. \n\nEven if we checked, its too late. The competition is already over!\n\nGiving an analogy, will we want to go to the market every now and then to check if there is a new rule the police might send us to jail? Maybe they are saying we can't be wearing green clothes yesterday! Is it alright to breathe?\n\nClearly the answer is no. This isn't a right way to live. \n\nRules need to be official and binding, not changed at whims on the forum.", "votes": 6 }, { "id": 317876, "postDate": "2018-04-22T18:21:02.610Z", "content": "@Anthony, you said :
\n\n\n\n\nthe rules stated that people didn't have to upload their model to be part of the leaderboard.
\n
No, it is not part of the rules. The rules are in the page you did not read, hence my reaction. They do not contain what you say they contain.
\n\n@Shitian Ni, you are right, the Rules page does not contain the obligation to upload models by end of stage 1. It does not contain either anything saying you'll be on the leaderboard if you don't upload your model by end of stage 1. Itt does not contain anything about Stage 1 and Stage 2 data, content, and obligations altogether. This is part of the problem obviously. Seems we need to read beyond the rules page therefore. If reading the FAQ is within scope, then so is reading the Timeline page IMHO.
\n\n\n\n\nRules need to be official and binding, not changed at whims on the forum.
\n
@Mak, I agree, and I've written this many times. In this specific case, reading the forum would have helped people wonder if not uploading was a safe bet. But it would not have warned anyone that all those who did not would be removed, as it was decided after stage 1 end.
", "rawMarkdown": "@Anthony, you said :\n\n> the rules stated that people didn't have to upload their model to be part of the leaderboard. \n\nNo, it is not part of the rules. The rules are in the page you did not read, hence my reaction. They do not contain what you say they contain.\n\n@Shitian Ni, you are right, the Rules page does not contain the obligation to upload models by end of stage 1. It does not contain either anything saying you'll be on the leaderboard if you don't upload your model by end of stage 1. Itt does not contain anything about Stage 1 and Stage 2 data, content, and obligations altogether. This is part of the problem obviously. Seems we need to read beyond the rules page therefore. If reading the FAQ is within scope, then so is reading the Timeline page IMHO.\n\n> Rules need to be official and binding, not changed at whims on the forum.\n\n@Mak, I agree, and I've written this many times. In this specific case, reading the forum would have helped people wonder if not uploading was a safe bet. But it would not have warned anyone that all those who did not would be removed, as it was decided after stage 1 end.", "votes": -3 }, { "id": 317881, "postDate": "2018-04-22T18:38:30.217Z", "content": "@CPMP: Sure, I didn't say where the quote was from, but that's besides the point (since the FAQ quote and the rules mentioning nothing are thrown around here a lot, I assumed I didn't need to specify that the quote was not in the rules).
\n\nI did read the rules now because I got curious why you took such offense to what I said, and I found no mention of such disqualification (not even in Competition Timeline).
\n\nThe FAQ was wrong, the rules did not mention anything and organizers acted on their own. That is the issue and why people are dissatisfied. I am not sure what you mean by \"something is wrong, but not what many here say is wrong\".
", "rawMarkdown": "@CPMP: Sure, I didn't say where the quote was from, but that's besides the point (since the FAQ quote and the rules mentioning nothing are thrown around here a lot, I assumed I didn't need to specify that the quote was not in the rules).\n\nI did read the rules now because I got curious why you took such offense to what I said, and I found no mention of such disqualification (not even in Competition Timeline).\n\nThe FAQ was wrong, the rules did not mention anything and organizers acted on their own. That is the issue and why people are dissatisfied. I am not sure what you mean by \"something is wrong, but not what many here say is wrong\"." }, { "id": 317885, "postDate": "2018-04-22T18:50:04.560Z", "content": "@Anthony, seems you want to be right whatever the situation. Fine with me. Life is short.
", "rawMarkdown": "@Anthony, seems you want to be right whatever the situation. Fine with me. Life is short.", "votes": -1 }, { "id": 317895, "postDate": "2018-04-22T19:40:51.497Z", "content": "I just think what you said is another side of the same coin, but wanted to appear contrarian.
\n\nHave a good day and cograts on your positioning!
", "rawMarkdown": "I just think what you said is another side of the same coin, but wanted to appear contrarian.\n\nHave a good day and cograts on your positioning!", "votes": 2 } ] }, { "id": 317684, "postDate": "2018-04-22T08:40:41.827Z", "content": "
As implied, we here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions). We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!
", "rawMarkdown": "As implied, we here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions). We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!", "votes": 12 } ] }, { "id": 317526, "postDate": "2018-04-21T19:33:31.970Z", "content": "While I support making competitions better in any ways, punishing people over a rule that was made only after the competition is an extremely poor decision. It impacts half of the people on leaderboard. I think this is really doing a disservice to the community.
", "rawMarkdown": "While I support making competitions better in any ways, punishing people over a rule that was made only after the competition is an extremely poor decision. It impacts half of the people on leaderboard. I think this is really doing a disservice to the community.", "votes": 10 }, { "id": 317625, "postDate": "2018-04-22T03:11:47.717Z", "content": "You said \"We discussed this internally at Kaggle and decided that we are going to change our practice going forward.\"\n....\nwhy change the rule of submit model internally and didn't inform us immediately and formally?
\n\nWe didnt know about the rule of \"We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.\" \nAt the very start, we just know that :
\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n\nprize position means top5 in this competition.
\n\nI think you should give us a solution to solve this misleading.
", "rawMarkdown": "You said \"We discussed this **internally** at Kaggle and decided that we are going to change our practice going forward.\"\n....\nwhy change the rule of submit model internally and didn't inform us immediately and formally? \n\nWe didnt know about the rule of \"**We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model**.\" \nAt the very start, we just know that :\n\n:**Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.**\n\nprize position means top5 in this competition.\n\nI think you should give us a solution to solve this misleading.", "votes": 10 }, { "id": 317522, "postDate": "2018-04-21T19:29:34.117Z", "content": "Hi Maggie,
\n\nThank you for speaking on the issue. I am a beginner and I learned a lot from this competition. Unfortunately, I got removed. However, I am a bit disappointed with your explanation.
\n\n\n\n\nFor this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.
\n
I support your decision of uploading models, however this should be implemented for future competitions, not this one. Hope that the widespread dissatisfaction will be enough to overrule your decision for this competition. Thanks a lot anyways.
", "rawMarkdown": "Hi Maggie,\n\nThank you for speaking on the issue. I am a beginner and I learned a lot from this competition. Unfortunately, I got removed. However, I am a bit disappointed with your explanation. \n\n> For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.\n\n1. This rule is mentioned nowhere. You can't make a rule after a competition ends.\n2. You are wrong in mentioning that Round 2 dataset (which was greater than 3000 images) was less than Round 1 dataset (which was less than 700 images). I am sorry but this reflects very badly on Kaggle. \n\nI support your decision of uploading models, however this should be implemented for future competitions, not this one. Hope that the widespread dissatisfaction will be enough to overrule your decision for this competition. Thanks a lot anyways.", "votes": 9 }, { "id": 318186, "postDate": "2018-04-23T10:54:46.407Z", "content": "I was under the impression the model uploads were just for prize eligibility and am disappointed to be removed. It seems appropriate to have had a warning message of some kind after submission if removal from the competition were at risk.
\n\nThis is the email reminder sent out before the competition ended -- no mention of the upload requirement:
\n\n\n", "rawMarkdown": "I was under the impression the model uploads were just for prize eligibility and am disappointed to be removed. It seems appropriate to have had a warning message of some kind after submission if removal from the competition were at risk. \n\nThis is the email reminder sent out before the competition ended -- no mention of the upload requirement:\n\n> The \"data-science-bowl-2018\" competition closes in 7.0 days. You are currently in 595th place as part of the team \"Stephen Bailey.\" This competition requires you to choose 2 entries that count towards the final result. To choose 2 entries: \n> \n> 1. Login at https://www.kaggle.com\n> 2. Visit the submission page ( https://www.kaggle.com/c/data-science-bowl-2018/submissions ) \n> 3. Click the checkbox next to the relevant entry to select it. \n> 4. Click the \"Submit Selection Changes\" button.\n> \n> (If you do not choose any entries, your top 2 entries from the public leaderboard will be chosen by default.)\n>\n> Cheers, Kaggle", "votes": 7, "replies": [ { "id": 318197, "postDate": "2018-04-23T11:26:29.290Z", "content": "The \"data-science-bowl-2018\" competition closes in 7.0 days. You are currently in 595th place as part of the team \"Stephen Bailey.\" This competition requires you to choose 2 entries that count towards the final result. To choose 2 entries:
\n \n\n
\n \n- Login at https://www.kaggle.com
\n- Visit the submission page ( https://www.kaggle.com/c/data-science-bowl-2018/submissions )
\n- Click the checkbox next to the relevant entry to select it.
\n- Click the \"Submit Selection Changes\" button.
\n(If you do not choose any entries, your top 2 entries from the public leaderboard will be chosen by default.)
\n \nCheers, Kaggle
\n
If even people (like you, Raoul etc. ) who published great and very helpful kernels for the community (and the organizers ) are removed...^^
\n\nFrankly kaggle Admins, you are invited to reconsider you decision about LB removal... and to make your RULES (and possible sanctions ) crystal clear at the beginning of upcoming competitions
", "rawMarkdown": "If even people (like you, Raoul etc. ) who published great and very helpful kernels for the community (and the organizers ) are removed...^^\n\nFrankly kaggle Admins, you are invited to reconsider you decision about LB removal... and to make your RULES (and possible sanctions ) crystal clear at the beginning of upcoming competitions", "votes": 7 } ] }, { "id": 317611, "postDate": "2018-04-22T01:43:14.153Z", "content": "I disagree with the removal decision taken after stage 1， before updating the 2-Stage rule.
\n\nPlease respect competitors！
", "rawMarkdown": "I disagree with the removal decision taken after stage 1， before updating the 2-Stage rule.\n\nPlease respect competitors！", "votes": 7 }, { "id": 317815, "postDate": "2018-04-22T15:54:05.077Z", "content": "Hi Everyone,
\n\nThe purpose of my post is to try to bring everyone together and begin thinking about a solution that serves the best interest of Kaggle, participants, and the medical research community.
\n\nKaggle’s Perspective: Remove participants from the Final LB that haven’t uploaded a Model.
\n\nParticipants Perspective: The rules weren’t communicated effectively about whether the model should be submitted, evident by the fact that more than 50% of participants have been removed from the LB for not submitting a model.
\n\nThe Perspective of the Medical Research Community: They need the models to be submitted in order to review and advance medical\n research.
\n\nRecommended Solution: Give participants a second chance to upload their models and keep the original final LB that was published on April 16th, 2018.
\n\nAs I have described in another post, this research is valued at approximately $50+ million. Let’s try to manage it successfully!
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/55092
\n\nThank you,\nTEAMPDITI
", "rawMarkdown": "Hi Everyone, \n\nThe purpose of my post is to try to bring everyone together and **begin thinking about a solution** that serves the best interest of Kaggle, participants, and the medical research community. \n\n **Kaggle’s Perspective:** Remove participants from the Final LB that haven’t uploaded a Model. \n\n**Participants Perspective:** The rules weren’t communicated effectively about whether the model should be submitted, evident by the fact that more than 50% of participants have been removed from the LB for not submitting a model. \n \n**The Perspective of the Medical Research Community:** They need the models to be submitted in order to review and advance medical\n research.\n\n**Recommended Solution: Give participants a second chance to upload their models and keep the original final LB that was published on April 16th, 2018.** \n\nAs I have described in another post, this research is valued at approximately **$50+ million.** Let’s try to manage it successfully! \n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/55092\n\nThank you,\nTEAMPDITI \n", "votes": 6 }, { "id": 317617, "postDate": "2018-04-22T02:08:18.930Z", "content": "We submit our model but our lb also be removed.there are almost 452 team forgeting upload their model before deadline？we spend a lot of time and money in this competition,I don’t think we all make mistake like forgeting uploading model before deadline.we enjoy the process but the result is also very import.
", "rawMarkdown": "We submit our model but our lb also be removed.there are almost 452 team forgeting upload their model before deadline？we spend a lot of time and money in this competition,I don’t think we all make mistake like forgeting uploading model before deadline.we enjoy the process but the result is also very import.", "votes": 6 }, { "id": 317607, "postDate": "2018-04-22T01:11:46.797Z", "content": "Very poor decision by Kaggle admins. LB rankings for stage 2 should have been left alone (instead of being deleted) for those who did not upload code in stage 1. Those who did not upload code in stage 1 should just be labeled ineligible on the LB for stage 2.
\n\nMaggie and Kaggle need to admit they made the error that has been frequently cited by others in their FAQ . It is grossly incorrect to state that the ~300 teams that were deleted did not participate.
", "rawMarkdown": "Very poor decision by Kaggle admins. LB rankings for stage 2 should have been left alone (instead of being deleted) for those who did not upload code in stage 1. Those who did not upload code in stage 1 should just be labeled ineligible on the LB for stage 2.\n\n Maggie and Kaggle need to admit they made the error that has been frequently cited by others in their FAQ . It is grossly incorrect to state that the ~300 teams that were deleted did not participate.", "votes": 5 }, { "id": 317507, "postDate": "2018-04-21T18:39:09.727Z", "content": "What about teams who have uploaded the model before stage 1 deadline and still got removed from leadership board?
", "rawMarkdown": "What about teams who have uploaded the model before stage 1 deadline and still got removed from leadership board?", "votes": 6 }, { "id": 317463, "postDate": "2018-04-21T16:58:06.053Z", "content": "Can you clarify what you are doing for THIS competition?
\n\nAre you reinstating the 50% that were removed?
\n\nAnd what is it that you are changing going forwards?
\n\nMaking the rules clearer?
", "rawMarkdown": "Can you clarify what you are doing for THIS competition?\n\nAre you reinstating the 50% that were removed?\n\nAnd what is it that you are changing going forwards?\n\nMaking the rules clearer?", "votes": 6, "replies": [ { "id": 317492, "postDate": "2018-04-21T18:13:08.253Z", "content": "Can’t you see that all of these people entered the competition in good faith and you benefit nothing by their disqualification.\n50 people spending 20 hours each per entry is 1000 man hours! How whimsical and ill thought out was the meeting where this action was discussed?\nI think this needs to be escalated to someone within Kaggle who can see the big picture and accept that a mistake has been made.\nThe bad feeling this will generate is not in Kaggle’s Interest.
", "rawMarkdown": "Can’t you see that all of these people entered the competition in good faith and you benefit nothing by their disqualification.\n50 people spending 20 hours each per entry is 1000 man hours! How whimsical and ill thought out was the meeting where this action was discussed?\nI think this needs to be escalated to someone within Kaggle who can see the big picture and accept that a mistake has been made.\nThe bad feeling this will generate is not in Kaggle’s Interest.", "votes": 16 } ] }, { "id": 317726, "postDate": "2018-04-22T11:33:49.400Z", "content": "@Maggie I hope you meant 'Stage 1 (April 11, 11:59 PM UTC)' instead of Stage 1 (April 11, 11:59 UTC) as you mentioned above.\ncould this be the reason for filtering wrongly for the cases model uploaded after 11:59 AM and before 11:59 PM on april 11 ?
", "rawMarkdown": "@Maggie I hope you meant **'Stage 1 (April 11, 11:59 PM UTC)'** instead of **Stage 1 (April 11, 11:59 UTC)** as you mentioned above.\ncould this be the reason for filtering wrongly for the cases model uploaded after 11:59 AM and before 11:59 PM on april 11 ?", "votes": 4 }, { "id": 318477, "postDate": "2018-04-23T22:45:10.497Z", "content": "See update to this post here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55228
", "rawMarkdown": "See update to this post here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55228", "votes": 4 }, { "id": 317467, "postDate": "2018-04-21T17:08:23.497Z", "content": "Thank you for your feedback. The objective of the competition has always been from day one to advance medical research and recognize participants for their efforts. Accordingly, it is still valuable to request from the teams that they submit their models, especially because the decision was made after the competition ended. I advise we still focus on the objective here, which is advancing medical research, and still give the opportunity to teams to submit their models. Please note that I did submit the model in accordance with the rules and timeline.
", "rawMarkdown": "Thank you for your feedback. The objective of the competition has always been from day one to advance medical research and recognize participants for their efforts. Accordingly, it is still valuable to request from the teams that they submit their models, especially because the decision was made after the competition ended. I advise we still focus on the objective here, which is advancing medical research, and still give the opportunity to teams to submit their models. Please note that I did submit the model in accordance with the rules and timeline. ", "votes": 1, "replies": [ { "id": 317604, "postDate": "2018-04-22T01:05:22.057Z", "content": "Was there a submit model url described under Overview, at all?
", "rawMarkdown": "Was there a submit model url described under Overview, at all?" } ] }, { "id": 317713, "postDate": "2018-04-22T10:27:55.590Z", "content": "Please inform the participants the probable reason for removing ( like duplicate accounts, model not uploaded , code change suspected .. etc ) so that people(genuine cases) are given chance to defend themselves and can provide justification.
\n\nsometimes not providing this info puts the participants in wondering state thinking forever why this happened ?\nI had uploaded my code before the stage 1 deadline. but my entry still missing on LB.
\n\nAs I was wondering what could have gone wrong . I remember this ambiguity around the selecting the best model. \nlet us say on stage 1, I selected two model outputs and code for these two were uploaded .\nIn stage 2, with the same above code made the predictions on stage 2 data and uploaded them.\nNow comes the tricky part. Do I need to keep the stage 1 selected check box entries as is .\n or do I need to select stage 2 submission entries and uncheck stage1 earlier entries.\nSome how i got confused and unchecked previous ones, and selected stage2 submissions.\nCan this be a reason for deletion from leader board ? I am not sure ... still wondering why ??? ... :)
", "rawMarkdown": "Please inform the participants the probable reason for removing ( like duplicate accounts, model not uploaded , code change suspected .. etc ) so that people(genuine cases) are given chance to defend themselves and can provide justification. \n\nsometimes not providing this info puts the participants in wondering state thinking forever why this happened ?\nI had uploaded my code before the stage 1 deadline. but my entry still missing on LB. \n\nAs I was wondering what could have gone wrong . I remember this ambiguity around the selecting the best model. \nlet us say on stage 1, I selected two model outputs and code for these two were uploaded .\nIn stage 2, with the same above code made the predictions on stage 2 data and uploaded them.\nNow comes the tricky part. Do I need to keep the stage 1 selected check box entries as is .\n or do I need to select stage 2 submission entries and uncheck stage1 earlier entries.\nSome how i got confused and unchecked previous ones, and selected stage2 submissions.\nCan this be a reason for deletion from leader board ? I am not sure ... still wondering why ??? ... :)", "votes": 2 }, { "id": 317461, "postDate": "2018-04-21T16:45:52.393Z", "content": "There have been a lot of posts about the removal of those who did not upload a model before the end of Round 1 for this competition. The spirit of 2-stage competitions is that we are getting access to a unique dataset but in limited quantities. For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.
\n\nWhile it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. We discussed this internally at Kaggle and decided that we are going to change our practice going forward. We will be updating our 2-Stage competition rules and will follow this practice going forward.
", "rawMarkdown": "There have been a lot of posts about the removal of those who did not upload a model before the end of Round 1 for this competition. The spirit of 2-stage competitions is that we are getting access to a unique dataset but in limited quantities. For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2. \n\nWhile it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. We discussed this internally at Kaggle and decided that we are going to change our practice going forward. We will be updating our 2-Stage competition rules and will follow this practice going forward. ", "votes": -49 }, { "id": 317775, "postDate": "2018-04-22T14:30:35.733Z", "content": "@Maggie, Is this the final standings? Would the medals be updated because the Kaggle 2nd stage FAQ clearly states that medals would be given according to the number of participants in the first stage (over 3400 in this case). Please clarify.
", "rawMarkdown": "@Maggie, Is this the final standings? Would the medals be updated because the Kaggle 2nd stage FAQ clearly states that medals would be given according to the number of participants in the first stage (over 3400 in this case). Please clarify.", "votes": -3 }, { "id": 318468, "postDate": "2018-04-23T22:14:54.797Z", "content": "So just now we found our team was removed from LB.. Are you still doing the removal process? The thing is that we uploaded our model on time but why still got removed....
", "rawMarkdown": "So just now we found our team was removed from LB.. Are you still doing the removal process? The thing is that we uploaded our model on time but why still got removed...." }, { "id": 318394, "postDate": "2018-04-23T18:36:36.147Z", "content": "Our team member did upload a model for Stage 1. But our team (GeorgetownICBI) is not listed on the leaderboard anymore. I believe we should get re-instated.
", "rawMarkdown": "Our team member did upload a model for Stage 1. But our team (GeorgetownICBI) is not listed on the leaderboard anymore. I believe we should get re-instated." }, { "id": 318168, "postDate": "2018-04-23T09:46:24.843Z", "content": "Do we have to upload the weights as well? \nOur team uploaded the model before time, still we were removed from the competition.\nCan anyone throw some light on this.
", "rawMarkdown": "**Do we have to upload the weights as well?** \nOur team uploaded the model before time, still we were removed from the competition.\nCan anyone throw some light on this.", "replies": [ { "id": 318178, "postDate": "2018-04-23T10:30:08.703Z", "content": "You can/probably should do this https://www.kaggle.com/c/data-science-bowl-2018/discussion/55041
", "rawMarkdown": "You can/probably should do this https://www.kaggle.com/c/data-science-bowl-2018/discussion/55041" } ] }, { "id": 318068, "postDate": "2018-04-23T05:45:31.257Z", "content": "Sad for kaggle
", "rawMarkdown": "Sad for kaggle" }, { "id": 318028, "postDate": "2018-04-23T04:02:21.967Z", "content": "Where were we supposed to upload a model? I had models in my kernels data section and made sure my kernel could access and generate outputs. Then I ran the same code with different path names locally. This is my first competition and may not have understood what uploading the model meant. My placement was removed, what should I have done?
", "rawMarkdown": "Where were we supposed to upload a model? I had models in my kernels data section and made sure my kernel could access and generate outputs. Then I ran the same code with different path names locally. This is my first competition and may not have understood what uploading the model meant. My placement was removed, what should I have done? " }, { "id": 317685, "postDate": "2018-04-22T08:43:01.243Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 317660, "postDate": "2018-04-22T07:07:50.900Z", "rawMarkdown": "", "isDeleted": true }, { "id": 317576, "postDate": "2018-04-21T22:10:12.977Z", "rawMarkdown": "", "votes": 6, "isDeleted": true } ], "comments": [ { "id": 318213, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-23T12:13:33.740000", "content": "We here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions), based on the FAQ which stated this. We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!
", "votes": 39, "replies": [ { "id": 318376, "author_name": "Burhan ", "author_url": "", "post_date": "2018-04-23T17:42:52.530000", "content": "Way to go, Broad team!
", "votes": -1, "replies": [] }, { "id": 318380, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-04-23T17:51:53.913000", "content": "It means a lot to hear you say that.
\n\nPeople spent a long time on your data.
", "votes": 1, "replies": [] }, { "id": 318455, "author_name": "Sandeepanand", "author_url": "", "post_date": "2018-04-23T21:04:02.383000", "content": "Thanks, Board team for considering. Please understand that people with multiple submissions have put effort, time and money(running models on AWS and floydhub etc..costs) to prepare submissions and at least we deserve a mention that we did a project as such
", "votes": 0, "replies": [] }, { "id": 318470, "author_name": "Rui Li", "author_url": "", "post_date": "2018-04-23T22:21:18.237000", "content": "Hi, Anne, while we are waiting for the teams of the hard working kagglers to come back to the LB, I just found out additional teams (including ours) got removed today...
", "votes": 0, "replies": [] } ] }, { "id": 317523, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-21T19:30:26.523000", "content": "It was not included in the rules https://www.kaggle.com/c/data-science-bowl-2018/rules
\n\nIt was not included in the FAQ https://www.kaggle.com/two-stage-faq
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do\n submit in the second stage?
\n \nYes. However, if you do not upload a model and finish in a prize\n position, your team will be removed from the competition standings\n entirely.
\n
Ex post facto laws are expressly forbidden by the United States Constitution in Article 1, Section 9, Clause 3 (with respect to federal laws) and Article 1, Section 10 (with respect to state laws).
\n\nSo are you going to reinstate removed teams for this competition and change the rules from now on?\nNo-one is infallible. I think Kaggle is creating an unnecessary issue out of this. You could just accept that you didn't communicate this rule properly and on time, restore removed submissions and move on.
", "votes": 29, "replies": [] }, { "id": 317539, "author_name": "Alexander Popov", "author_url": "", "post_date": "2018-04-21T20:17:27.967000", "content": "Maggie, can you please point to the clause in the rules that I signed which states or implies that not uploading a model before a certain deadline leads to a COMPLETE team disqualification with removal from LB and removal of all submissions history etc... even if the team is not a winner?
", "votes": 25, "replies": [] }, { "id": 317525, "author_name": "Alex Shvets", "author_url": "", "post_date": "2018-04-21T19:31:29.737000", "content": "Dear @Maggie I think it is not good for Kaggle when you remove half of your LB and tell to participants who have been removed that it is their fault. From my point of view it it's not quite honest. The second part of the competition was organized no so well and we have corresponding outcomes. First of all you mentioned the dataset was much smaller in Round 2. it is not true - the test data set for the 1 stage included around 65 images while for the second stage it included approximately 3000 images. Then, for example, in your rules:
\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set. PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points. \n
https://www.kaggle.com/c/data-science-bowl-2018#Timeline\nthere are no words that participant who did not upload their code will be automatically removed from the LB. PLEASE NOTE: says only about submitting prediction in order to be in the LB. In addition, based on my own experience it was not easy to find a place where I should upload the model. Only my teammates helped me to find it out.
", "votes": 27, "replies": [ { "id": 317560, "author_name": "yueqi", "author_url": "", "post_date": "2018-04-21T21:24:53.197000", "content": "Can't believe a Kaggle admin @Maggie would say the dataset was much smaller in Round 2.
Adding another comment:
\n\n\n\n\nWhile it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action.
\n
If @Maggie means significant increase of people who did not upload models, I think the reason is that: among the 800+ teams (twice that of dsb2017), many started doing Kaggle only recently. A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018. When the rules are not clearly written, newcomers are much more likely to be confused about whether they should upload models or not. So the significant increase of people who did not upload models is the result of more new people joining the competition and the rules being unclear.
Since many of us are new to Kaggle, removing 50%+ teams will strongly discourage new people from joining and participating in Kaggle competitions (even for people who did upload models, it's so sad to see so many peers being removed). And when @Maggie says felt we needed to take action based on non-existing rules only after the competition, this reflects badly on Kaggle's internal decision making.
Overall, I believe the rules should be changed to require model upload, but it should only be enforce in future competitions, not this one.
", "votes": 16, "replies": [] } ] }, { "id": 317645, "author_name": "Ruobing", "author_url": "", "post_date": "2018-04-22T05:16:34.320000", "content": "Obviously this post shows a lack of respect. Really disappointing.
", "votes": 17, "replies": [] }, { "id": 317912, "author_name": "NxGTR", "author_url": "", "post_date": "2018-04-22T20:24:15.040000", "content": "I recently provided official stats about my Kaggle profile, among those were \"100 competitions\", lol...
\nThis being my first Kaggle disqualification in +3 years, now I am at 99.
Two (probably wrong) thoughts:
\n1) Whats the logic behind... allowing people to submit to Stage 2 if they would be disqualified anyway?
\n2) I am in Kaggle to learn, not to do business, I think submitting hello world as my model will be my approach in the future.
Hi, Maggie. \nYou can't just say \"felt we needed to take action\" without make any announcement before stage1 deadline.
\n\n\" the dataset was much smaller in Round 2 \" is wrong too. There are 670 images in stage1 training set, 65 images in stage1 test set and 3019 images in stage2 test set.
\n\nYou can't change the rules without notifying all the competitors. You can't change the rules after the competition, and then tell us \"You've broken the rules.\" And the rules you have changed are contrary to the previous rules.
\n\n“A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018.” This approach will disappoint those who have devoted much effort to this competition. This will strongly discourage new people from joining and participating in Kaggle competitions.
\n\nI think this explanation will make the contestants who have been removed very disappointed. All my efforts have been wasted. This explanation is unacceptable.
\n\nYou can't remove our result without any reasonable explanation. That is not fair! NOT FAIR!!!!
", "votes": 16, "replies": [] }, { "id": 317499, "author_name": "Plankton", "author_url": "", "post_date": "2018-04-21T18:24:20.040000", "content": "So the change is made after the competition ended. Is this acceptable? And the explanations are not reasonable at all. I posted my questions here.
\n\nFirst, as mentioned by someone else, the rules that we all agree before entering doens't have the mentioned requirement.
\n\nSecond, the base for this immediate change (for this competition) is basically not true.
\n\n\n\n\nFor this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.
\n
Here is the data , in terms of number of predictions we need to make:
\n\nLast year DSB: stage1 had about 200 samples, stage2 had about 500 samples
\n\nThis year DSB: stage1 has < 700 samples, stage2 has > 3000 samples
\n\nI agree that the change is a right step forward. But things should be clear from the beginning. Any changes should be applied for the future not something that is done.
\n\n@William, @Wendy
", "votes": 16, "replies": [] }, { "id": 317690, "author_name": "Pranav Pandya", "author_url": "", "post_date": "2018-04-22T09:05:43.573000", "content": "I would expect to see such notification in rare case scenario but it seems that you're sending this notification like WhatsApp message after every competition depending on your mood . Zillow is the most recent example.
\n\nOn a lighter note, I can imagine what will be the situation in upcoming/ ongoing competitions if this philosophy continues. Few examples below:
\n\nright now and throwing you out immediately. We will continue this practice and will suspend users who use language other than Python :)right now and reducing the prize money based on your country...I and hopefully, most participants see this as misuse of power and request proper escalation.
\n\nAll it takes is to spend sufficient time on writing clear rules before launching any competition.
", "votes": 13, "replies": [] }, { "id": 317488, "author_name": "Antonis Maronikolakis", "author_url": "", "post_date": "2018-04-21T18:02:26.397000", "content": "I did not participate in this competition, nor have I participated in other 2-stage competitions, so I am not very familiar with the rules. From my understanding, reading Discussion posts, the rules stated people would not be removed from the rankings if they didn't upload the model (correct me if I'm wrong).
\n\nThen, from this post I get that the organizers decided after the fact that they would go ahead and remove all these people, essentially overriding the rules without notice.
\n\nI'm sorry, but if that is true, this is very unprofessional. I really am disappointed in Kaggle for this one. All these people worked hard for a placement and followed the rules, and now they get removed because \"a significant increase was noticed\"? I get that points and medals are not the end-all-be-all and the goal of the competition was to advance medical research, but scratching away all this work without reason or transparency is not cool.
\n\nI really hope I am misunderstanding this, since it is very upsetting Kaggle staff and organizers can change the rules as they feel without notifying participants. That's my two cents.
", "votes": 11, "replies": [] }, { "id": 317476, "author_name": "Pavel Oreshin", "author_url": "", "post_date": "2018-04-21T17:30:24.960000", "content": "My and many other teams uploaded model before the deadline, and still were removed from the LB.
Don't you think that it is better to clarify the rules firstly and then take action? Not vice versa
Agreed on 2. About 1, same here, removed with uploaded model.
\n\nA possible explanation of those removals commented on KaggleNoobs is that there can be a problem with aes-256 encripted zipped models (my case). No idea if that is really the issue but as I said, some other kagglers commented to have problems in the past with that.
", "votes": 0, "replies": [] }, { "id": 317485, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-21T17:54:41.893000", "content": "@miguel did you encrypt a zip file or zipped an encrypted file?
", "votes": 0, "replies": [] }, { "id": 317498, "author_name": "miguel perez", "author_url": "", "post_date": "2018-04-21T18:21:16.027000", "content": "@Oscar, used 7zip password with aes256 encryption when compressing the file. (Passwording zip files was ok in previous editions of 2 stage competitions as long as password was provided to Kaggle's requirement). Don't really know if that is the issue, or what kind of automated check has been made to uploaded models... hope will find out soon :-)
", "votes": 0, "replies": [] }, { "id": 317508, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-21T18:40:46.683000", "content": "@miguel Thanks, It should be okay then. I hope you get your case resolved.
", "votes": 1, "replies": [] }, { "id": 317515, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-21T19:04:00.053000", "content": "@miguel_perez, the post I mentioned on the KaggleNoob channel was here.\n https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#317272\n @bsp2020 uploaded sha256sum before the deadline of the first stage and got removed, not quite the same as you. Both of you definitely can appeal the decisions, don't worry about it.
Thanks @Nicole, yes, I see the difference but also similarity in that both ways assure the model is frozen from the end of stage 1 and that are -maybe- not what an automated check would expect. So I was grateful for your info, a reason to a bit more optimistic ! :-)
", "votes": 1, "replies": [] }, { "id": 317616, "author_name": "Dexter Lei", "author_url": "", "post_date": "2018-04-22T02:07:10.673000", "content": "Still removed with uploaded model before deadline of stage 1, which really confused me. However, I have not received any explanation about it.
", "votes": 0, "replies": [] } ] }, { "id": 317754, "author_name": "Serigne ", "author_url": "", "post_date": "2018-04-22T13:23:24.610000", "content": "Sorry but I'm afraid you're discouraging people from participating in such highly demanding competitions in the future .
\n\nAfter all, is the effort worth it, if one doesn't know how the rules evolve and if such critical decisions are taken after just \" internal discussions\" ?
", "votes": 10, "replies": [] }, { "id": 317788, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-22T15:07:29.973000", "content": "I won't be popular I guess but here it is. First, let me clear any doubt. I totally disagree with changing rules during a competition. In democracy law are not applied retrospectively. Here they are. Sad and wrong. People should not be removed based on a new policy decided after stage 1.
\n\nThis said, the issue was not the absence of a written statement about the deadline. The issue was a question in the FAQ whose answer could lead people to ignore the deadline and think they'll be safe.
\n\nThe deadline for uploading models was clearly stated in the Timeline page, as well as in Stage2 FAQ. The spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1. The fact that half of stage 2 teams did upload their model is a clear indication that this was quite visible, contrarily to many posts here that say the deadline was not written anywhere.
\n\nYes, something is wrong, but not what many here say is wrong.
", "votes": 14, "replies": [ { "id": 317796, "author_name": "Antonis Maronikolakis", "author_url": "", "post_date": "2018-04-22T15:20:03.777000", "content": "From my understanding, the rules stated that people didn't have to upload their model to be part of the leaderboard. Then, people were removed because they didn't upload a model, even though it was not stated in the rules.
\n\nI think that is what people are saying. Rules being added after the fact is not respectful to the participants.
\n\nAm I missing something? I didn't participate so I haven't read the rules, but from what I see here this is what must have happened. That is definitely wrong in my book.
", "votes": 1, "replies": [] }, { "id": 317797, "author_name": "Shitian Ni", "author_url": "", "post_date": "2018-04-22T15:20:37.573000", "content": "\n\n\nThe spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1.
\n
Is reading discussion forum and finding potential new rules a must to compete in Kaggle?
", "votes": 0, "replies": [] }, { "id": 317802, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-22T15:30:21.870000", "content": "\n\n\nthe rules stated that people didn't have to upload their model to be part of the leaderboard.
\n
Where did you read this in the Rules page?
\n\nAgain, the source of confusion comes from a FAQ, not the pages of the competition. The competition pages (esp Timeline page) are crystal clear: models must be finalized and uploaded by end of stage1.
", "votes": 2, "replies": [] }, { "id": 317803, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-22T15:32:49.467000", "content": "\n\n\nIs reading discussion forum [...] a must to compete in Kaggle?
\n
I think so, for many reasons. Reading all information available is good practice.
\n\nIf your question is : was the policy change announced at the right place and at the right time? then the answer is clearly no IMHO.
", "votes": 1, "replies": [] }, { "id": 317805, "author_name": "Antonis Maronikolakis", "author_url": "", "post_date": "2018-04-22T15:35:35.703000", "content": "@CMPM: As I said, I have not read the rules as I did not participate. However, people are quoting this:
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
Which clearly states people will not be removed from the LB regardless of uploading a model or not, unless they finish in prize positions.
", "votes": 2, "replies": [] }, { "id": 317807, "author_name": "Shitian Ni", "author_url": "", "post_date": "2018-04-22T15:40:36.013000", "content": "This is the timeline version I can see
\n\n\n\n\nCompetition Timeline
\n
\n Start Date: January 16, 2018
\n Entry deadline: April 9th, 2018
\n Team merger deadline: April 9th, 2018
\n Stage one deadline and stage two data release: April 11, 2018 - PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points.
\n End date: April 16, 2018 - 11:59PM UTC
It is crystal clear that no “model” mentioned.
", "votes": 1, "replies": [] }, { "id": 317808, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-22T15:40:44.843000", "content": "The rules https://www.kaggle.com/c/data-science-bowl-2018/rules are the only binding agreement between Kaggle and participants. Every little detail should be included there. If not, then there is no reason for rule agreement before entering each competition. The forums are for discussion and decision making. After such an important decision, the admins should have communicated it better by updating the rules page, and forcing all participants to agree with the new rules as soon as this decision was made at the next login. The way this was handled is totally unprofessional. Noone is obligated to continuously read the forums and try to sneak every detail, and given the importance of the rule change (disqualification) the forums discussions is not sufficient. They didn't even send an email for God's sake! Besides if you read Maggie's announcement it is clear that such a decision was made only internally and was not communicated properly through the rules update. Let alone that they left the FAQ unchanged.
", "votes": 8, "replies": [] }, { "id": 317809, "author_name": "Zheng Li", "author_url": "", "post_date": "2018-04-22T15:41:46.567000", "content": "CPMP is right. Also the main point is FAQ writes:
\n\n\" Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?
\n\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.\"
\n\nNone of us \"finish in a prize posistion\" but we got removed beacause
\n\n\"While it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. \"
", "votes": 6, "replies": [] }, { "id": 317816, "author_name": "", "author_url": "", "post_date": "2018-04-22T15:55:59.837000", "content": "", "votes": 0, "replies": [] }, { "id": 317819, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-22T16:06:09.180000", "content": "@Anthony Marakis, you say you did not read the rules yet you comment them. Weird.
\n\n@Shitian Ni, not sure how you manage to truncate the Timeline page quote, but here is a part that you seem to keep ignoring:
\n\n\n\n\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline
\n
@Zheng Li, thank you. I'm truly sorry your great solutionisn't rewarded, and I hope the decision will be reversed for you and many other.
\n\nAnd to close my contribution here, I am not the one making the rules, nor do I agree to the way they are modified and enforced after the fact. Don't kill the messenger ;)
", "votes": 2, "replies": [] }, { "id": 317820, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-22T16:09:03.037000", "content": "Since the issue is becoming gradually very serious, and the admins are not responding to the increased community frustration (no response after the announcement), I am asking for Kaggle's legal department view on this https://www.kaggle.com/c/data-science-bowl-2018/discussion/55131
", "votes": 1, "replies": [] }, { "id": 317822, "author_name": "Shitian Ni", "author_url": "", "post_date": "2018-04-22T16:15:13.757000", "content": "@CPMP I just quoted the timeline from the Rules page. I just copy&paste that. I believe that the timeline in Rules is the only one we all read, agreed on, and have to obey. And I note that there is a PLEASE NOTE there, again, no words about “model”
", "votes": 0, "replies": [] }, { "id": 317823, "author_name": "Antonis Maronikolakis", "author_url": "", "post_date": "2018-04-22T16:15:20.983000", "content": "@CPMP: Why is reading the rules a requirement for me to chime in? Pretty much everyone says that what I quoted is in the FAQ, so I take their word for it and add my perspective.
\n\nI'm sorry, but it's frankly childish to sweep what I say under the rug because I was not part of this. If what I'm saying is indeed true, it's true regardless of whether I participated.
", "votes": 1, "replies": [] }, { "id": 317828, "author_name": "Hon Keat", "author_url": "", "post_date": "2018-04-22T16:40:22.600000", "content": "\n\n\nCPMP wrote
\n \n\n\n \n> Is reading discussion forum and finding potential new rules a must to compete in Kaggle?
\nI think so, for many reasons. Reading all information available is good practice.
\n \nIf your question is : was the policy change announced at the right place and at the right time? then the answer is clearly no IMHO.
\n
@CPMP Can't believe your reply is pointing to a yes to his question. However I take it that your main position is \"Reading all information available is good practice\", and is not actually answering to his question.
\n\nI think what he is asking is, and from this point onward many of our concern should be,
\n\nAfter this episode, do we need to check the forum every now and then for a new rule cooked up that might disqualify us? Clearly the answer is no.
\n\nEven if we checked, its too late. The competition is already over!
\n\nGiving an analogy, will we want to go to the market every now and then to check if there is a new rule the police might send us to jail? Maybe they are saying we can't be wearing green clothes yesterday! Is it alright to breathe?
\n\nClearly the answer is no. This isn't a right way to live.
\n\nRules need to be official and binding, not changed at whims on the forum.
", "votes": 6, "replies": [] }, { "id": 317876, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-22T18:21:02.610000", "content": "@Anthony, you said :
\n\n\n\n\nthe rules stated that people didn't have to upload their model to be part of the leaderboard.
\n
No, it is not part of the rules. The rules are in the page you did not read, hence my reaction. They do not contain what you say they contain.
\n\n@Shitian Ni, you are right, the Rules page does not contain the obligation to upload models by end of stage 1. It does not contain either anything saying you'll be on the leaderboard if you don't upload your model by end of stage 1. Itt does not contain anything about Stage 1 and Stage 2 data, content, and obligations altogether. This is part of the problem obviously. Seems we need to read beyond the rules page therefore. If reading the FAQ is within scope, then so is reading the Timeline page IMHO.
\n\n\n\n\nRules need to be official and binding, not changed at whims on the forum.
\n
@Mak, I agree, and I've written this many times. In this specific case, reading the forum would have helped people wonder if not uploading was a safe bet. But it would not have warned anyone that all those who did not would be removed, as it was decided after stage 1 end.
", "votes": -3, "replies": [] }, { "id": 317881, "author_name": "Antonis Maronikolakis", "author_url": "", "post_date": "2018-04-22T18:38:30.217000", "content": "@CPMP: Sure, I didn't say where the quote was from, but that's besides the point (since the FAQ quote and the rules mentioning nothing are thrown around here a lot, I assumed I didn't need to specify that the quote was not in the rules).
\n\nI did read the rules now because I got curious why you took such offense to what I said, and I found no mention of such disqualification (not even in Competition Timeline).
\n\nThe FAQ was wrong, the rules did not mention anything and organizers acted on their own. That is the issue and why people are dissatisfied. I am not sure what you mean by \"something is wrong, but not what many here say is wrong\".
", "votes": 0, "replies": [] }, { "id": 317885, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-22T18:50:04.560000", "content": "@Anthony, seems you want to be right whatever the situation. Fine with me. Life is short.
", "votes": -1, "replies": [] }, { "id": 317895, "author_name": "Antonis Maronikolakis", "author_url": "", "post_date": "2018-04-22T19:40:51.497000", "content": "I just think what you said is another side of the same coin, but wanted to appear contrarian.
\n\nHave a good day and cograts on your positioning!
", "votes": 2, "replies": [] } ] }, { "id": 317684, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2018-04-22T08:40:41.827000", "content": "
As implied, we here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions). We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!
", "votes": 12, "replies": [] } ] }, { "id": 317526, "author_name": "Hon Keat", "author_url": "", "post_date": "2018-04-21T19:33:31.970000", "content": "While I support making competitions better in any ways, punishing people over a rule that was made only after the competition is an extremely poor decision. It impacts half of the people on leaderboard. I think this is really doing a disservice to the community.
", "votes": 10, "replies": [] }, { "id": 317625, "author_name": "databatman", "author_url": "", "post_date": "2018-04-22T03:11:47.717000", "content": "You said \"We discussed this internally at Kaggle and decided that we are going to change our practice going forward.\"\n....\nwhy change the rule of submit model internally and didn't inform us immediately and formally?
\n\nWe didnt know about the rule of \"We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.\" \nAt the very start, we just know that :
\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n\nprize position means top5 in this competition.
\n\nI think you should give us a solution to solve this misleading.
", "votes": 10, "replies": [] }, { "id": 317522, "author_name": "DeepNet", "author_url": "", "post_date": "2018-04-21T19:29:34.117000", "content": "Hi Maggie,
\n\nThank you for speaking on the issue. I am a beginner and I learned a lot from this competition. Unfortunately, I got removed. However, I am a bit disappointed with your explanation.
\n\n\n\n\nFor this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.
\n
I support your decision of uploading models, however this should be implemented for future competitions, not this one. Hope that the widespread dissatisfaction will be enough to overrule your decision for this competition. Thanks a lot anyways.
", "votes": 9, "replies": [] }, { "id": 318186, "author_name": "Stephen Bailey", "author_url": "", "post_date": "2018-04-23T10:54:46.407000", "content": "I was under the impression the model uploads were just for prize eligibility and am disappointed to be removed. It seems appropriate to have had a warning message of some kind after submission if removal from the competition were at risk.
\n\nThis is the email reminder sent out before the competition ended -- no mention of the upload requirement:
\n\n\n", "votes": 7, "replies": [ { "id": 318197, "author_name": "Serigne ", "author_url": "", "post_date": "2018-04-23T11:26:29.290000", "content": "The \"data-science-bowl-2018\" competition closes in 7.0 days. You are currently in 595th place as part of the team \"Stephen Bailey.\" This competition requires you to choose 2 entries that count towards the final result. To choose 2 entries:
\n \n\n
\n \n- Login at https://www.kaggle.com
\n- Visit the submission page ( https://www.kaggle.com/c/data-science-bowl-2018/submissions )
\n- Click the checkbox next to the relevant entry to select it.
\n- Click the \"Submit Selection Changes\" button.
\n(If you do not choose any entries, your top 2 entries from the public leaderboard will be chosen by default.)
\n \nCheers, Kaggle
\n
If even people (like you, Raoul etc. ) who published great and very helpful kernels for the community (and the organizers ) are removed...^^
\n\nFrankly kaggle Admins, you are invited to reconsider you decision about LB removal... and to make your RULES (and possible sanctions ) crystal clear at the beginning of upcoming competitions
", "votes": 7, "replies": [] } ] }, { "id": 317611, "author_name": "yyll008", "author_url": "", "post_date": "2018-04-22T01:43:14.153000", "content": "I disagree with the removal decision taken after stage 1， before updating the 2-Stage rule.
\n\nPlease respect competitors！
", "votes": 7, "replies": [] }, { "id": 317815, "author_name": "Leith Akkawi", "author_url": "", "post_date": "2018-04-22T15:54:05.077000", "content": "Hi Everyone,
\n\nThe purpose of my post is to try to bring everyone together and begin thinking about a solution that serves the best interest of Kaggle, participants, and the medical research community.
\n\nKaggle’s Perspective: Remove participants from the Final LB that haven’t uploaded a Model.
\n\nParticipants Perspective: The rules weren’t communicated effectively about whether the model should be submitted, evident by the fact that more than 50% of participants have been removed from the LB for not submitting a model.
\n\nThe Perspective of the Medical Research Community: They need the models to be submitted in order to review and advance medical\n research.
\n\nRecommended Solution: Give participants a second chance to upload their models and keep the original final LB that was published on April 16th, 2018.
\n\nAs I have described in another post, this research is valued at approximately $50+ million. Let’s try to manage it successfully!
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/55092
\n\nThank you,\nTEAMPDITI
", "votes": 6, "replies": [] }, { "id": 317617, "author_name": "yaya", "author_url": "", "post_date": "2018-04-22T02:08:18.930000", "content": "We submit our model but our lb also be removed.there are almost 452 team forgeting upload their model before deadline？we spend a lot of time and money in this competition,I don’t think we all make mistake like forgeting uploading model before deadline.we enjoy the process but the result is also very import.
", "votes": 6, "replies": [] }, { "id": 317607, "author_name": "kickback", "author_url": "", "post_date": "2018-04-22T01:11:46.797000", "content": "Very poor decision by Kaggle admins. LB rankings for stage 2 should have been left alone (instead of being deleted) for those who did not upload code in stage 1. Those who did not upload code in stage 1 should just be labeled ineligible on the LB for stage 2.
\n\nMaggie and Kaggle need to admit they made the error that has been frequently cited by others in their FAQ . It is grossly incorrect to state that the ~300 teams that were deleted did not participate.
", "votes": 5, "replies": [] }, { "id": 317507, "author_name": "Nishant Singh", "author_url": "", "post_date": "2018-04-21T18:39:09.727000", "content": "What about teams who have uploaded the model before stage 1 deadline and still got removed from leadership board?
", "votes": 6, "replies": [] }, { "id": 317463, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-04-21T16:58:06.053000", "content": "Can you clarify what you are doing for THIS competition?
\n\nAre you reinstating the 50% that were removed?
\n\nAnd what is it that you are changing going forwards?
\n\nMaking the rules clearer?
", "votes": 6, "replies": [ { "id": 317492, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-04-21T18:13:08.253000", "content": "Can’t you see that all of these people entered the competition in good faith and you benefit nothing by their disqualification.\n50 people spending 20 hours each per entry is 1000 man hours! How whimsical and ill thought out was the meeting where this action was discussed?\nI think this needs to be escalated to someone within Kaggle who can see the big picture and accept that a mistake has been made.\nThe bad feeling this will generate is not in Kaggle’s Interest.
", "votes": 16, "replies": [] } ] }, { "id": 317726, "author_name": "Kishore M", "author_url": "", "post_date": "2018-04-22T11:33:49.400000", "content": "@Maggie I hope you meant 'Stage 1 (April 11, 11:59 PM UTC)' instead of Stage 1 (April 11, 11:59 UTC) as you mentioned above.\ncould this be the reason for filtering wrongly for the cases model uploaded after 11:59 AM and before 11:59 PM on april 11 ?
", "votes": 4, "replies": [] }, { "id": 318477, "author_name": "Maggie", "author_url": "", "post_date": "2018-04-23T22:45:10.497000", "content": "See update to this post here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55228
", "votes": 4, "replies": [] }, { "id": 317467, "author_name": "Leith Akkawi", "author_url": "", "post_date": "2018-04-21T17:08:23.497000", "content": "Thank you for your feedback. The objective of the competition has always been from day one to advance medical research and recognize participants for their efforts. Accordingly, it is still valuable to request from the teams that they submit their models, especially because the decision was made after the competition ended. I advise we still focus on the objective here, which is advancing medical research, and still give the opportunity to teams to submit their models. Please note that I did submit the model in accordance with the rules and timeline.
", "votes": 1, "replies": [ { "id": 317604, "author_name": "Kaggoo", "author_url": "", "post_date": "2018-04-22T01:05:22.057000", "content": "Was there a submit model url described under Overview, at all?
", "votes": 0, "replies": [] } ] }, { "id": 317713, "author_name": "Kishore M", "author_url": "", "post_date": "2018-04-22T10:27:55.590000", "content": "Please inform the participants the probable reason for removing ( like duplicate accounts, model not uploaded , code change suspected .. etc ) so that people(genuine cases) are given chance to defend themselves and can provide justification.
\n\nsometimes not providing this info puts the participants in wondering state thinking forever why this happened ?\nI had uploaded my code before the stage 1 deadline. but my entry still missing on LB.
\n\nAs I was wondering what could have gone wrong . I remember this ambiguity around the selecting the best model. \nlet us say on stage 1, I selected two model outputs and code for these two were uploaded .\nIn stage 2, with the same above code made the predictions on stage 2 data and uploaded them.\nNow comes the tricky part. Do I need to keep the stage 1 selected check box entries as is .\n or do I need to select stage 2 submission entries and uncheck stage1 earlier entries.\nSome how i got confused and unchecked previous ones, and selected stage2 submissions.\nCan this be a reason for deletion from leader board ? I am not sure ... still wondering why ??? ... :)
", "votes": 2, "replies": [] }, { "id": 317775, "author_name": "Burhan ", "author_url": "", "post_date": "2018-04-22T14:30:35.733000", "content": "@Maggie, Is this the final standings? Would the medals be updated because the Kaggle 2nd stage FAQ clearly states that medals would be given according to the number of participants in the first stage (over 3400 in this case). Please clarify.
", "votes": -3, "replies": [] }, { "id": 318468, "author_name": "", "author_url": "", "post_date": "2018-04-23T22:14:54.797000", "content": "", "votes": 0, "replies": [] }, { "id": 318394, "author_name": "", "author_url": "", "post_date": "2018-04-23T18:36:36.147000", "content": "", "votes": 0, "replies": [] }, { "id": 318168, "author_name": "", "author_url": "", "post_date": "2018-04-23T09:46:24.843000", "content": "", "votes": 0, "replies": [ { "id": 318178, "author_name": "", "author_url": "", "post_date": "2018-04-23T10:30:08.703000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 318068, "author_name": "", "author_url": "", "post_date": "2018-04-23T05:45:31.257000", "content": "", "votes": 0, "replies": [] }, { "id": 318028, "author_name": "", "author_url": "", "post_date": "2018-04-23T04:02:21.967000", "content": "", "votes": 0, "replies": [] }, { "id": 317685, "author_name": "", "author_url": "", "post_date": "2018-04-22T08:43:01.243000", "content": "", "votes": -2, "replies": [] }, { "id": 317660, "author_name": "", "author_url": "", "post_date": "2018-04-22T07:07:50.900000", "content": "", "votes": 0, "replies": [] }, { "id": 317576, "author_name": "", "author_url": "", "post_date": "2018-04-21T22:10:12.977000", "content": "", "votes": 6, "replies": [] } ], "raw_markdown_by_id": { "318213": "We here on the Broad team agree that model uploading should not be a requirement to appear on the final leaderboard (except for top 5 positions), based on the FAQ which stated this. We are anxiously awaiting a decision on this. Thanks to the community for remaining relatively calm and civil in the meantime!", "317523": "It was not included in the rules [https://www.kaggle.com/c/data-science-bowl-2018/rules][1]\n\nIt was not included in the FAQ [https://www.kaggle.com/two-stage-faq][2]\n\n> Will I still be on the leaderboard if I don't upload a model but do\n> submit in the second stage?\n> \n> Yes. However, if you do not upload a model and finish in a prize\n> position, your team will be removed from the competition standings\n> entirely.\n\nEx post facto laws are expressly forbidden by the United States Constitution in Article 1, Section 9, Clause 3 (with respect to federal laws) and Article 1, Section 10 (with respect to state laws).\n\nSo are you going to reinstate removed teams for this competition and change the rules from now on?\nNo-one is infallible. I think Kaggle is creating an unnecessary issue out of this. You could just accept that you didn't communicate this rule properly and on time, restore removed submissions and move on.\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/rules\n [2]: https://www.kaggle.com/two-stage-faq", "317539": "Maggie, can you please point to the clause in the rules that I signed which states or implies that not uploading a model before a certain deadline leads to a COMPLETE team disqualification with removal from LB and removal of all submissions history etc... even if the team is not a winner?", "317525": "Dear @Maggie I think it is not good for Kaggle when you remove half of your LB and tell to participants who have been removed that it is their fault. From my point of view it it's not quite honest. The second part of the competition was organized no so well and we have corresponding outcomes. First of all you mentioned ` the dataset was much smaller in Round 2.` it is not true - the test data set for the 1 stage included around 65 images while for the second stage it included approximately 3000 images. Then, for example, in your rules: \n\n ```\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline. After this deadline, the test set is released, the answers to the validation set are released, and participants make predictions on the test set. PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points. \n```\n\nhttps://www.kaggle.com/c/data-science-bowl-2018#Timeline\nthere are no words that participant who did not upload their code will be automatically removed from the LB. PLEASE NOTE: says only about submitting prediction in order to be in the LB. In addition, based on my own experience it was not easy to find a place where I should upload the model. Only my teammates helped me to find it out. ", "317645": "Obviously this post shows a lack of respect. Really disappointing.", "317912": "I recently provided official stats about my Kaggle profile, among those were \"100 competitions\", lol... \nThis being my first Kaggle disqualification in +3 years, now I am at 99.\n\nTwo (probably wrong) thoughts: \n1) Whats the logic behind... allowing people to submit to Stage 2 if they would be disqualified anyway? \n2) I am in Kaggle to learn, not to do business, I think submitting hello world as my model will be my approach in the future.", "317608": "Hi, Maggie. \nYou can't just say \"felt we needed to take action\" without make any announcement before stage1 deadline.\n\n\" the dataset was much smaller in Round 2 \" is wrong too. There are 670 images in stage1 training set, 65 images in stage1 test set and 3019 images in stage2 test set.\n\nYou can't change the rules without notifying all the competitors. You can't change the rules after the competition, and then tell us \"You've broken the rules.\" And the rules you have changed are contrary to the previous rules.\n\n“A lot of us here are first-time kagglers, and joined because of the spirits and scientific goals of DSB 2018.” This approach will disappoint those who have devoted much effort to this competition. This will strongly discourage new people from joining and participating in Kaggle competitions.\n\nI think this explanation will make the contestants who have been removed very disappointed. All my efforts have been wasted. This explanation is unacceptable.\n\nYou can't remove our result without any reasonable explanation. That is not fair! **NOT FAIR!!!!**", "317499": "So the change is made after the competition ended. Is this acceptable? And the explanations are not reasonable at all. I posted my questions [here][1]. \n\nFirst, as mentioned by someone else, the rules that we all agree before entering doens't have the mentioned requirement. \n\nSecond, the base for this immediate change (for this competition) is basically not true. \n>For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.\n\nHere is the data , in terms of number of predictions we need to make:\n\nLast year DSB: stage1 had about 200 samples, stage2 had about 500 samples\n\nThis year DSB: stage1 has < 700 samples, stage2 has > 3000 samples\n\nI agree that the change is a right step forward. But things should be clear from the beginning. Any changes should be applied for the future not something that is done. \n\n@William, @Wendy\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55034#317362", "317690": "I would expect to see such notification in rare case scenario but it seems that you're sending this notification like WhatsApp message after every competition depending on your **mood** . Zillow is the most recent example. \n\nOn a lighter note, I can imagine what will be the situation in upcoming/ ongoing competitions if this philosophy continues. Few examples below:\n\n- We **felt that** most companies and most people are using Python but you have submitted your model in R so we are updating our rules `right now` and throwing you out immediately. We will continue this practice and will suspend users who use language other than Python :)\n- After seeing the results, we notice that all most winners are from developing country and **felt that** prize money would be too much for them compared to prize money for winners from developed countries so we are updating our rules `right now` and reducing the prize money based on your country...\n\nI and hopefully, most participants see this as **misuse of power** and request proper escalation. \n\n**All it takes is to spend sufficient time on writing clear rules before launching any competition.** ", "317488": "I did not participate in this competition, nor have I participated in other 2-stage competitions, so I am not very familiar with the rules. From my understanding, reading Discussion posts, the rules stated people would not be removed from the rankings if they didn't upload the model (correct me if I'm wrong).\n\nThen, from this post I get that the organizers decided after the fact that they would go ahead and remove all these people, essentially overriding the rules without notice.\n\nI'm sorry, but if that is true, this is *very* unprofessional. I really am disappointed in Kaggle for this one. All these people worked hard for a placement and followed the rules, and now they get removed because \"a significant increase was noticed\"? I get that points and medals are not the end-all-be-all and the goal of the competition was to advance medical research, but scratching away all this work without reason or transparency is not cool.\n\nI really hope I am misunderstanding this, since it is very upsetting Kaggle staff and organizers can change the rules as they feel without notifying participants. That's my two cents.", "317476": "1. My and many other teams uploaded model before the deadline, and still were removed from the LB.\n\n2. Don't you think that it is better to clarify the rules firstly and then take action? Not vice versa", "317754": "Sorry but I'm afraid you're discouraging people from participating in such highly demanding competitions in the future .\n\nAfter all, is the effort worth it, if one doesn't know how the rules evolve and if such critical decisions are taken after just \" internal discussions\" ? ", "317788": "I won't be popular I guess but here it is. First, let me clear any doubt. I totally disagree with changing rules during a competition. In democracy law are not applied retrospectively. Here they are. Sad and wrong. People should not be removed based on a new policy decided after stage 1.\n\nThis said, the issue was not the absence of a written statement about the deadline. The issue was a question in the FAQ whose answer could lead people to ignore the deadline and think they'll be safe. \n\nThe deadline for uploading models was clearly stated in the Timeline page, as well as in Stage2 FAQ. The spirit of the competition (finalize model by end of stage1) was clearly stated, and discussed many times on the forum during stage 1. The fact that half of stage 2 teams did upload their model is a clear indication that this was quite visible, contrarily to many posts here that say the deadline was not written anywhere. \n\nYes, something is wrong, but not what many here say is wrong.\n", "317684": "![fcplm][1]\n\n\n [1]: https://habrastorage.org/webt/qf/rp/lz/qfrplzpgalzv13zgni6aesrrzw4.jpeg", "317526": "While I support making competitions better in any ways, punishing people over a rule that was made only after the competition is an extremely poor decision. It impacts half of the people on leaderboard. I think this is really doing a disservice to the community.", "317625": "You said \"We discussed this **internally** at Kaggle and decided that we are going to change our practice going forward.\"\n....\nwhy change the rule of submit model internally and didn't inform us immediately and formally? \n\nWe didnt know about the rule of \"**We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model**.\" \nAt the very start, we just know that :\n\n:**Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.**\n\nprize position means top5 in this competition.\n\nI think you should give us a solution to solve this misleading.", "317522": "Hi Maggie,\n\nThank you for speaking on the issue. I am a beginner and I learned a lot from this competition. Unfortunately, I got removed. However, I am a bit disappointed with your explanation. \n\n> For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2.\n\n1. This rule is mentioned nowhere. You can't make a rule after a competition ends.\n2. You are wrong in mentioning that Round 2 dataset (which was greater than 3000 images) was less than Round 1 dataset (which was less than 700 images). I am sorry but this reflects very badly on Kaggle. \n\nI support your decision of uploading models, however this should be implemented for future competitions, not this one. Hope that the widespread dissatisfaction will be enough to overrule your decision for this competition. Thanks a lot anyways.", "318186": "I was under the impression the model uploads were just for prize eligibility and am disappointed to be removed. It seems appropriate to have had a warning message of some kind after submission if removal from the competition were at risk. \n\nThis is the email reminder sent out before the competition ended -- no mention of the upload requirement:\n\n> The \"data-science-bowl-2018\" competition closes in 7.0 days. You are currently in 595th place as part of the team \"Stephen Bailey.\" This competition requires you to choose 2 entries that count towards the final result. To choose 2 entries: \n> \n> 1. Login at https://www.kaggle.com\n> 2. Visit the submission page ( https://www.kaggle.com/c/data-science-bowl-2018/submissions ) \n> 3. Click the checkbox next to the relevant entry to select it. \n> 4. Click the \"Submit Selection Changes\" button.\n> \n> (If you do not choose any entries, your top 2 entries from the public leaderboard will be chosen by default.)\n>\n> Cheers, Kaggle", "317611": "I disagree with the removal decision taken after stage 1， before updating the 2-Stage rule.\n\nPlease respect competitors！", "317815": "Hi Everyone, \n\nThe purpose of my post is to try to bring everyone together and **begin thinking about a solution** that serves the best interest of Kaggle, participants, and the medical research community. \n\n **Kaggle’s Perspective:** Remove participants from the Final LB that haven’t uploaded a Model. \n\n**Participants Perspective:** The rules weren’t communicated effectively about whether the model should be submitted, evident by the fact that more than 50% of participants have been removed from the LB for not submitting a model. \n \n**The Perspective of the Medical Research Community:** They need the models to be submitted in order to review and advance medical\n research.\n\n**Recommended Solution: Give participants a second chance to upload their models and keep the original final LB that was published on April 16th, 2018.** \n\nAs I have described in another post, this research is valued at approximately **$50+ million.** Let’s try to manage it successfully! \n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/55092\n\nThank you,\nTEAMPDITI \n", "317617": "We submit our model but our lb also be removed.there are almost 452 team forgeting upload their model before deadline？we spend a lot of time and money in this competition,I don’t think we all make mistake like forgeting uploading model before deadline.we enjoy the process but the result is also very import.", "317607": "Very poor decision by Kaggle admins. LB rankings for stage 2 should have been left alone (instead of being deleted) for those who did not upload code in stage 1. Those who did not upload code in stage 1 should just be labeled ineligible on the LB for stage 2.\n\n Maggie and Kaggle need to admit they made the error that has been frequently cited by others in their FAQ . It is grossly incorrect to state that the ~300 teams that were deleted did not participate.", "317507": "What about teams who have uploaded the model before stage 1 deadline and still got removed from leadership board?", "317463": "Can you clarify what you are doing for THIS competition?\n\nAre you reinstating the 50% that were removed?\n\nAnd what is it that you are changing going forwards?\n\nMaking the rules clearer?", "317726": "@Maggie I hope you meant **'Stage 1 (April 11, 11:59 PM UTC)'** instead of **Stage 1 (April 11, 11:59 UTC)** as you mentioned above.\ncould this be the reason for filtering wrongly for the cases model uploaded after 11:59 AM and before 11:59 PM on april 11 ?", "318477": "See update to this post here: https://www.kaggle.com/c/data-science-bowl-2018/discussion/55228", "317467": "Thank you for your feedback. The objective of the competition has always been from day one to advance medical research and recognize participants for their efforts. Accordingly, it is still valuable to request from the teams that they submit their models, especially because the decision was made after the competition ended. I advise we still focus on the objective here, which is advancing medical research, and still give the opportunity to teams to submit their models. Please note that I did submit the model in accordance with the rules and timeline. ", "317713": "Please inform the participants the probable reason for removing ( like duplicate accounts, model not uploaded , code change suspected .. etc ) so that people(genuine cases) are given chance to defend themselves and can provide justification. \n\nsometimes not providing this info puts the participants in wondering state thinking forever why this happened ?\nI had uploaded my code before the stage 1 deadline. but my entry still missing on LB. \n\nAs I was wondering what could have gone wrong . I remember this ambiguity around the selecting the best model. \nlet us say on stage 1, I selected two model outputs and code for these two were uploaded .\nIn stage 2, with the same above code made the predictions on stage 2 data and uploaded them.\nNow comes the tricky part. Do I need to keep the stage 1 selected check box entries as is .\n or do I need to select stage 2 submission entries and uncheck stage1 earlier entries.\nSome how i got confused and unchecked previous ones, and selected stage2 submissions.\nCan this be a reason for deletion from leader board ? I am not sure ... still wondering why ??? ... :)", "317461": "There have been a lot of posts about the removal of those who did not upload a model before the end of Round 1 for this competition. The spirit of 2-stage competitions is that we are getting access to a unique dataset but in limited quantities. For this competition the requirement was to upload your model before the end of Stage 1 (April 11, 11:59 UTC) in order to participate in Round 2 because the dataset was much smaller in Round 2. \n\nWhile it is true that we have not removed people who did not upload models in the past we noticed a significant increase this year and felt we needed to take action. We discussed this internally at Kaggle and decided that we are going to change our practice going forward. We will be updating our 2-Stage competition rules and will follow this practice going forward. ", "317775": "@Maggie, Is this the final standings? Would the medals be updated because the Kaggle 2nd stage FAQ clearly states that medals would be given according to the number of participants in the first stage (over 3400 in this case). Please clarify.", "318468": "So just now we found our team was removed from LB.. Are you still doing the removal process? The thing is that we uploaded our model on time but why still got removed....", "318394": "Our team member did upload a model for Stage 1. But our team (GeorgetownICBI) is not listed on the leaderboard anymore. I believe we should get re-instated.", "318168": "**Do we have to upload the weights as well?** \nOur team uploaded the model before time, still we were removed from the competition.\nCan anyone throw some light on this.", "318068": "Sad for kaggle", "318028": "Where were we supposed to upload a model? I had models in my kernels data section and made sure my kernel could access and generate outputs. Then I ran the same code with different path names locally. This is my first competition and may not have understood what uploading the model meant. My placement was removed, what should I have done? ", "317685": "", "317660": "", "317576": "" } }