To Kaggle staff.
\n\nYour 2 stage faq (https://www.kaggle.com/two-stage-faq) reads:
\n\n\n\n\nThe spirit of having a second stage is to prevent hand labeling and\n leaderboard probing of the test data. In order to achieve this, we ask\n that you upload your source code, including the correct parameters\n that you used for generating your submission files. This is for you to\n prove that you have written automated code to create your final\n submission(s).
\n
Yet, further down in the page it allows people to participate in Stage 2 if they have not uploaded their code. I don't get why you do this, given you can easily check that code has been uploaded in stage1.
\n\nPeople that have not uploaded code are free to modify their code and get a better rank. It is only if they are in the prize money (unlikely for most of us) that they can be caught.
\n\nThis is very unfair. It is very unfair to all who try to be in the spirit of the competition, i.e. reuse their stage 1 code with minimal, non scientific, change to cope with the new stage2 test data.
\n\nEdit on April 16:
\nSeems there is disagreement with calling this cheating. Here is why I used that word.
The timeline page of this competition reads:
\n\n\n\n\n\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.\n
Anyone not doing it is not complying with the competition description. I'm fine if some prefer to call it not complying rather than cheating.
", "messages": [ { "id": 314302, "postDate": "2018-04-15T07:37:56.230Z", "content": "To Kaggle staff.
\n\nYour 2 stage faq (https://www.kaggle.com/two-stage-faq) reads:
\n\n\n\n\nThe spirit of having a second stage is to prevent hand labeling and\n leaderboard probing of the test data. In order to achieve this, we ask\n that you upload your source code, including the correct parameters\n that you used for generating your submission files. This is for you to\n prove that you have written automated code to create your final\n submission(s).
\n
Yet, further down in the page it allows people to participate in Stage 2 if they have not uploaded their code. I don't get why you do this, given you can easily check that code has been uploaded in stage1.
\n\nPeople that have not uploaded code are free to modify their code and get a better rank. It is only if they are in the prize money (unlikely for most of us) that they can be caught.
\n\nThis is very unfair. It is very unfair to all who try to be in the spirit of the competition, i.e. reuse their stage 1 code with minimal, non scientific, change to cope with the new stage2 test data.
\n\nEdit on April 16:
\nSeems there is disagreement with calling this cheating. Here is why I used that word.
The timeline page of this competition reads:
\n\n\n\n\n\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.\n
Anyone not doing it is not complying with the competition description. I'm fine if some prefer to call it not complying rather than cheating.
", "rawMarkdown": "To Kaggle staff.\n\nYour 2 stage faq (https://www.kaggle.com/two-stage-faq) reads:\n\n> The spirit of having a second stage is to prevent hand labeling and\n> leaderboard probing of the test data. In order to achieve this, we ask\n> that you upload your source code, including the correct parameters\n> that you used for generating your submission files. This is for you to\n> prove that you have written automated code to create your final\n> submission(s).\n\nYet, further down in the page it allows people to participate in Stage 2 if they have not uploaded their code. I don't get why you do this, given you can easily check that code has been uploaded in stage1.\n\nPeople that have not uploaded code are free to modify their code and get a better rank. It is only if they are in the prize money (unlikely for most of us) that they can be caught.\n\nThis is very unfair. It is very unfair to all who try to be in the spirit of the competition, i.e. reuse their stage 1 code with minimal, non scientific, change to cope with the new stage2 test data.\n\nEdit on April 16: \nSeems there is disagreement with calling this cheating. Here is why I used that word.\n\nThe timeline page of this competition reads:\n\n> April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.\n\nAnyone not doing it is not complying with the competition description. I'm fine if some prefer to call it *not complying* rather than *cheating*. ", "votes": 38 }, { "id": 314945, "postDate": "2018-04-16T14:36:05.770Z", "content": "here are my views:
\n\nthere is no way to prevent \"cheating\" , even for 1 stage competition. e.g. the user can hand label test data in one-stage competition. if you don't win a prize, kaggle don't check your code
why do you want to \"cheat\"? for people who want to get good ranking for resume, it could be done like this:
\n\nsubmit legally for the competition, it is ok that you get poor rank
after competition, you still can do a post submission. at this point of time many top solutions would be made known. you can re-implement to get good results.
at the interview, you can say although my ranking was not good, but i am fast to learn. and my post-results ranked 10, up from 100 in the period of one week after studying the top solutions.
\n\nMaybe kaggle can made a third ranking, called post-competition ranking. Then no one would need to cheat becuase anyone can submit after the competition for good ranking
", "rawMarkdown": "here are my views:\n\n1. there is no way to prevent \"cheating\" , even for 1 stage competition. e.g. the user can hand label test data in one-stage competition. if you don't win a prize, kaggle don't check your code\n\n2. why do you want to \"cheat\"? for people who want to get good ranking for resume, it could be done like this:\n\n- submit legally for the competition, it is ok that you get poor rank\n\n- after competition, you still can do a post submission. at this point of time many top solutions would be made known. you can re-implement to get good results.\n\nat the interview, you can say although my ranking was not good, but i am fast to learn. and my post-results ranked 10, up from 100 in the period of one week after studying the top solutions.\n\n\nMaybe kaggle can made a third ranking, called post-competition ranking. Then no one would need to cheat becuase anyone can submit after the competition for good ranking\n\n\n\n\n\n\n", "votes": 13, "replies": [ { "id": 314955, "postDate": "2018-04-16T14:50:41.330Z", "content": "@Heng re:
\n\n\n\n\nwhy do you want to \"cheat\"? for people who want to get good ranking\n for resume, it could be done like this: submit legally for the\n competition, it is ok that you get poor rank after competition, you\n still can do a post submission. at this point of time many top\n solutions would be made known. you can re-implement to get good\n results. at the interview, you can say although my ranking was not\n good, but i am fast to learn. and my post results ranked 10, up from\n 100 in the period of one week after studying the top solution.
\n
That's true, if you're interested to participate for the sake of participating and to fuel your interest in learning something from the competition. But some people don't want that; they participate to get a good ranking on the LB which they perceive would grant them medals, recognition and automatic bragging rights.
", "rawMarkdown": "@Heng re:\n\n> why do you want to \"cheat\"? for people who want to get good ranking\n> for resume, it could be done like this: submit legally for the\n> competition, it is ok that you get poor rank after competition, you\n> still can do a post submission. at this point of time many top\n> solutions would be made known. you can re-implement to get good\n> results. at the interview, you can say although my ranking was not\n> good, but i am fast to learn. and my post results ranked 10, up from\n> 100 in the period of one week after studying the top solution.\n\nThat's true, if you're interested to participate for the sake of participating and to fuel your interest in learning something from the competition. But some people don't want that; they participate to get a good ranking on the LB which they perceive would grant them medals, recognition and automatic bragging rights.", "votes": 1 }, { "id": 316343, "postDate": "2018-04-18T19:26:41.203Z", "content": "I totally agree with your point of view!! :)
", "rawMarkdown": "I totally agree with your point of view!! :)" } ] }, { "id": 314810, "postDate": "2018-04-16T10:00:31.690Z", "content": "@CPMP To be honest, I was very worried. I finished stage 1 in 9th place. This is my first chance to get a gold medal. In stage 2, my prediction is very disastrous and I found it may be repaired by a simple modification. I thought about cheating to get a gold medal seriously. Thank you. Your words made me stop short of cheating.
", "rawMarkdown": "@CPMP To be honest, I was very worried. I finished stage 1 in 9th place. This is my first chance to get a gold medal. In stage 2, my prediction is very disastrous and I found it may be repaired by a simple modification. I thought about cheating to get a gold medal seriously. Thank you. Your words made me stop short of cheating.", "votes": 11, "replies": [ { "id": 314811, "postDate": "2018-04-16T10:08:45.580Z", "content": "I wish you will fare well still. And thank you for sharing what you shared during stage1. I wish I had more time to implement some of what you shared. If you don't get a gold here I'm sure you'll get one soon in another competition given how well you did in stage1.
\n\nFor me too my stage2 is messed and I can fix it with a simple change. But I'll eat my own food, keep the code as is, and praise the gods of overfitting!
", "rawMarkdown": "I wish you will fare well still. And thank you for sharing what you shared during stage1. I wish I had more time to implement some of what you shared. If you don't get a gold here I'm sure you'll get one soon in another competition given how well you did in stage1.\n\nFor me too my stage2 is messed and I can fix it with a simple change. But I'll eat my own food, keep the code as is, and praise the gods of overfitting!", "votes": 1 }, { "id": 314828, "postDate": "2018-04-16T10:36:27.190Z", "content": "Thank you. You inspire me so much too.
", "rawMarkdown": "Thank you. You inspire me so much too.", "votes": 1 }, { "id": 314830, "postDate": "2018-04-16T10:41:23.430Z", "content": "@OsciiArt & @CPMP: Kaggle has already clearly addressed this issue in their FAQ at https://www.kaggle.com/two-stage-faq. If you think that you could get a better score by making a modification in your model & you intend to make the modification, just be considerate enough to indicate that you have forfeited/withdrawn from the competition so that your model's actual standing will not have an impact on the rest of the contestants. Of course, if this means that you have gained a prize position by doing so, Kaggle will revoke your prize, which is a fair deal. On the other hand, if you are really interested in continuing to participate in the competition, then you should adhere to the competition rules, & not make such changes. It is of no use pretending to adhere strictly to the competition rules by highlighting this in a discussion thread (such as in this Discussion board), when (in reality) you resort to underhanded means by training your model on external datasets, and/or re-training on Stage 1 test data (and labels) to get a better score on the private LB, and subsequently pretending that you have gotten this far by using your existing models (supposedly solely trained on Stage 1 training data) on the Stage 2 test set.
", "rawMarkdown": "@OsciiArt & @CPMP: Kaggle has already clearly addressed this issue in their FAQ at https://www.kaggle.com/two-stage-faq. If you think that you could get a better score by making a modification in your model & you intend to make the modification, just be considerate enough to indicate that you have forfeited/withdrawn from the competition so that your *model's* actual standing will not have an impact on the rest of the contestants. Of course, if this means that you have gained a prize position by doing so, Kaggle will revoke your prize, which is a fair deal. On the other hand, if you are really interested in continuing to participate in the competition, then you should adhere to the competition rules, & not make such changes. It is of no use pretending to adhere strictly to the competition rules by highlighting this in a discussion thread (such as in this Discussion board), when (in reality) you resort to underhanded means by training your model on external datasets, and/or re-training on Stage 1 test data (and labels) to get a better score on the private LB, and subsequently ***pretending*** that you have gotten this far by using your existing models (supposedly *solely* trained on Stage 1 training data) on the Stage 2 test set.", "votes": 2 }, { "id": 314867, "postDate": "2018-04-16T11:53:04.803Z", "content": "@Shiraz, thank you I didn't thought of reading the FAQ I commented in my post. Glad you're here to remind me.
", "rawMarkdown": "@Shiraz, thank you I didn't thought of reading the FAQ I commented in my post. Glad you're here to remind me." }, { "id": 314887, "postDate": "2018-04-16T12:41:08.943Z", "content": "Is changing a magic number in your code considered as a scientifically relevant change ? :)
\n\nWhat happens if your code, for whatever reasons, crashes on some specific data ?
\n\nI mean, can the organizers expect us to upload a stable code ?
", "rawMarkdown": "Is changing a magic number in your code considered as a scientifically relevant change ? :)\n\nWhat happens if your code, for whatever reasons, crashes on some specific data ? \n\nI mean, can the organizers expect us to upload a stable code ? \n\n\n\n" }, { "id": 314909, "postDate": "2018-04-16T13:42:55.970Z", "content": "@OsciiArt respect for admitting it openly. You should still have a good chance to have a nice result at stage 2, not only your result is bad, other may be even worse.
", "rawMarkdown": "@OsciiArt respect for admitting it openly. You should still have a good chance to have a nice result at stage 2, not only your result is bad, other may be even worse.", "votes": 3 }, { "id": 314912, "postDate": "2018-04-16T13:47:18.027Z", "content": "\n\n\nIs changing a magic number in your code considered as a scientifically relevant change ? :)
\n
What is a magic number?
", "rawMarkdown": "> Is changing a magic number in your code considered as a scientifically relevant change ? :)\n\nWhat is a magic number?" }, { "id": 314932, "postDate": "2018-04-16T14:25:05.323Z", "content": "\"A magic number is a bare naked number used in source code. It's magic because no-one has a clue what it means including the author inside 3 months.\"
", "rawMarkdown": "\"A magic number is a bare naked number used in source code. It's magic because no-one has a clue what it means including the author inside 3 months.\"" }, { "id": 314937, "postDate": "2018-04-16T14:30:52.483Z", "content": "Given what you did in Stage1 I doubt there are magic numbers in your code. ;)
", "rawMarkdown": "Given what you did in Stage1 I doubt there are magic numbers in your code. ;)" }, { "id": 314951, "postDate": "2018-04-16T14:47:13.177Z", "content": "@Dmytro Poplavskiy Thank you. I hope so!
", "rawMarkdown": "@Dmytro Poplavskiy Thank you. I hope so!" }, { "id": 314962, "postDate": "2018-04-16T15:08:51.167Z", "content": "@CPMP re:
\n\n\n\n\n@Shiraz, thank you I didn't thought of reading the FAQ I commented in\n my post. Glad you're here to remind me.
\n
You're welcome.
", "rawMarkdown": "@CPMP re:\n\n> @Shiraz, thank you I didn't thought of reading the FAQ I commented in\n> my post. Glad you're here to remind me.\n\nYou're welcome." }, { "id": 314989, "postDate": "2018-04-16T15:55:36.667Z", "content": "I would not complain if somebody says, because of the huge amount of noise in the test data we decided to change the sigma of our gaussian filter. I mean this is a human decision based on a simple observation of the data. Nothing extraordinary. Unfortunately I don't have any preprocessing and my results are miserable... :(
", "rawMarkdown": "I would not complain if somebody says, because of the huge amount of noise in the test data we decided to change the sigma of our gaussian filter. I mean this is a human decision based on a simple observation of the data. Nothing extraordinary. Unfortunately I don't have any preprocessing and my results are miserable... :( " }, { "id": 314997, "postDate": "2018-04-16T16:07:26.990Z", "content": "> I mean this is a human decision based on a simple observation of the data. Nothing extraordinary.
\n\nPoint is that if the data you mention is stage2 test data then you are modifying your model to better perform on it, which is precisely what is not in the spirit of this competition.
\n\n\n\n\nUnfortunately I don't have any preprocessing and my results are miserable... :(
\n
I hope your results are not that bad, you deserve to do well given what you did in stage1.
\n\nI have yet to see someone happy about stage 2 results ;)
", "rawMarkdown": "> I mean this is a human decision based on a simple observation of the data. Nothing extraordinary. \n\nPoint is that if the data you mention is stage2 test data then you are modifying your model to better perform on it, which is precisely what is not in the spirit of this competition.\n\n> Unfortunately I don't have any preprocessing and my results are miserable... :( \n\nI hope your results are not that bad, you deserve to do well given what you did in stage1.\n\nI have yet to see someone happy about stage 2 results ;)", "votes": 1 } ] }, { "id": 314430, "postDate": "2018-04-15T15:01:13.230Z", "content": "I agree with CPMP, the rules in the two-stage-faq allow someone to circumvent the main reason for having a two stage competition in the first place. I had wanted to start a thread like this myself but then I thought that there really doesn't seem to be a good way to eliminate all \"cheating\" so I decided against it.
\n\nThe main reason for having a two stage competition is to ensure that contestants cannot see the final test set until they have finalized their code. So they cannot tailor their code to this final test set. However the two-stage-faq does not enforce this. Allowing contestants to make a submission and get a ranking without having uploaded a model means that they are not bound by the model upload rule. We then have a situation where people who have uploaded a model are only allowed to make submissions based on their uploaded model. This model has not been tailored to fit the final test data. On the other hand those people who have not uploaded a model in stage 1 do not have make sure that their submission comes from any specific model. They are free to look at the final test data, change their model, and then make their submission.
\n\nNow one might argue that not having uploaded a model means that one cannot win any of the prizes. But the prizes are not the only thing that matter to contestants. There are plenty of people who participate in these competitions because it affects their careers or their education. The kaggle ranking matters and has real world consequences, even if you don't win a prize.
\n\nI'm not sure that there is any sure-fire way of eliminating all \"cheating\" but it can be reduced if people are only allowed to make a submission when they have uploaded a model during stage 1.
\n\nAnother approach might be to allow everyone to make submissions without any model constraints. This would ensure that we have a level playing field when it comes to rankings, basically the same conditions as those for a regular single stage competition. But when it comes to the prizes, only submissions that can be verified with uploaded models would be considered, and submissions without models in prize-winning positions would be removed.
", "rawMarkdown": "I agree with CPMP, the rules in the two-stage-faq allow someone to circumvent the main reason for having a two stage competition in the first place. I had wanted to start a thread like this myself but then I thought that there really doesn't seem to be a good way to eliminate all \"cheating\" so I decided against it.\n\nThe main reason for having a two stage competition is to ensure that contestants cannot see the final test set until they have finalized their code. So they cannot tailor their code to this final test set. However the two-stage-faq does not enforce this. Allowing contestants to make a submission and get a ranking without having uploaded a model means that they are not bound by the model upload rule. We then have a situation where people who have uploaded a model are only allowed to make submissions based on their uploaded model. This model has not been tailored to fit the final test data. On the other hand those people who have not uploaded a model in stage 1 do not have make sure that their submission comes from any specific model. They are free to look at the final test data, change their model, and then make their submission.\n\nNow one might argue that not having uploaded a model means that one cannot win any of the prizes. But the prizes are not the only thing that matter to contestants. There are plenty of people who participate in these competitions because it affects their careers or their education. The kaggle ranking matters and has real world consequences, even if you don't win a prize.\n\nI'm not sure that there is any sure-fire way of eliminating all \"cheating\" but it can be reduced if people are only allowed to make a submission when they have uploaded a model during stage 1.\n\nAnother approach might be to allow everyone to make submissions without any model constraints. This would ensure that we have a level playing field when it comes to rankings, basically the same conditions as those for a regular single stage competition. But when it comes to the prizes, only submissions that can be verified with uploaded models would be considered, and submissions without models in prize-winning positions would be removed.", "votes": 7, "replies": [ { "id": 314437, "postDate": "2018-04-15T15:22:09.327Z", "content": "\n\n\nAnother approach might be to allow everyone to make submissions\n without any model constraints. This would ensure that we have a level\n playing field when it comes to rankings, basically the same conditions\n as those for a regular single stage competition. But when it comes to\n the prizes, only submissions that can be verified with uploaded models\n would be considered, and submissions without models in prize-winning\n positions would be removed.
\n
Yes, I think this would be a better option for this (and future 2-stage) competition/(s) as well. Model accuracy is evolutionary, and if this is the desired outcome (as is the case for this competition where the aim is to obtain robust and accurate models to predict the ROI of a cell's nucleus), then the restriction on compulsory model submission to be considered for Stage 2 participation should be lifted.
", "rawMarkdown": "> Another approach might be to allow everyone to make submissions\n> without any model constraints. This would ensure that we have a level\n> playing field when it comes to rankings, basically the same conditions\n> as those for a regular single stage competition. But when it comes to\n> the prizes, only submissions that can be verified with uploaded models\n> would be considered, and submissions without models in prize-winning\n> positions would be removed.\n\nYes, I think this would be a better option for this (and future 2-stage) competition/(s) as well. Model accuracy is evolutionary, and if this is the desired outcome (as is the case for this competition where the aim is to obtain robust and accurate models to predict the ROI of a cell's nucleus), then the restriction on compulsory model submission to be considered for Stage 2 participation should be lifted.", "votes": 1 } ] }, { "id": 314965, "postDate": "2018-04-16T15:11:08.960Z", "content": "I just love this topic. It seems that all of us forgot or some are not even aware what Paulo Pinto did in Fisheries competition https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/discussion/31607
\n\nAll the points made about 2stage not being \"fair\" or is a place to cheat here are valid, but imho, there is no better solution how to make it \"fair\"
", "rawMarkdown": "I just love this topic. It seems that all of us forgot or some are not even aware what Paulo Pinto did in Fisheries competition https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/discussion/31607\n\nAll the points made about 2stage not being \"fair\" or is a place to cheat here are valid, but imho, there is no better solution how to make it \"fair\"", "votes": 5, "replies": [ { "id": 314972, "postDate": "2018-04-16T15:31:09.647Z", "content": "2 of the top 5 teams were removed, right?
", "rawMarkdown": "2 of the top 5 teams were removed, right?" }, { "id": 314975, "postDate": "2018-04-16T15:34:50.783Z", "content": "Yes
", "rawMarkdown": "Yes", "votes": 2 } ] }, { "id": 314371, "postDate": "2018-04-15T11:40:00.480Z", "content": "The fact that I got down votes is a confirmation that some participants want to change their code for stage2.
", "rawMarkdown": "The fact that I got down votes is a confirmation that some participants want to change their code for stage2.", "votes": 5, "replies": [ { "id": 315103, "postDate": "2018-04-16T18:44:11.793Z", "content": "Hey CPMP, I think for some people 'cheating' would come across as too strong a word to digest, so never mind the down votes! I personally agree to your point of view, because this is one of the valid interpretations of the FAQ that are possible. For me, the real problem is this fact itself that it is open to interpretations and is not technically complete. In fact, I've not seen any kind of a service provider depending on an 'FAQ' for rules enforcement, this looks very unprofessional I am afraid. There should be dedicated page/s legally explaining all the rules for participation in single or two stage competitions and these pages should be linked to the rules in the competitions' sections where we actually agree to the terms.
\n\nRegarding the point you've brought up though, I think there should be a simple solution.
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage?\n Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
As far as I've understood, the above quote is the part where you think Kaggle is allowing the upload of models by participant users in the stage 2 even when no models have been uploaded by them in stage 1, and in the process, Kaggle is giving them an unfair advantage in the context of rankings right ? I don't know much about the rankings yet, but if it's some kind of privilege one would get, then surely it'd be unfair to the user who has submitted a model in stage 1 and is bound by rules not to modify code. Whereas, the other users who have not submitted in stage 1 are free to do anything, even use stage 2 data anyway they want to get ranked. This is what I understand the problem is from your point of view.
\n\nNow, lets try to understand another interpretation. The clarity with which Kaggle says 'Yes' in the above quote makes me think that Kaggle surely is intentionally allowing this with a definite purpose. They are removing the stage 2 only participant from the competition standings entirely too. So, taking everything in good faith (why would such a reputed organization abet cheating?), I'd like to think that there are good reasons to allow it. May be because they want to let users get the experience of a live competition and gauge themselves even when they've missed the opportunity otherwise ? This is good too as it encourages learning and it goes with the overall aims of the platform.
\n\nHere we've two point of views, and both are true at the same time I think. The real problem is, there is scope for cheating for the users who WANT to cheat! Neither Kaggle nor any genuine participant user would want to abet cheating I am sure. So, lets come to the real part of the problem which I think is allowing a place in the LB to users who are taking part only in stage 2. I am not sure, but I guess it helps stage 2 only participants to gauge their performance by looking at their position there ? This must be important too for a genuine stage 2 only participant, coming up on the LB that is.
\n\nSo, ultimately I am getting the following two key points:
\n\nI might be missing something, but as far as I understand the problem and given the above two key points are true, I'd suggest a simple solution for it as follows:
\n\nThis will keep everyone happy and things clean. From a software point of view, it is very much implementable as everything is already in place. Adding a special view should not be too much of a problem. As also, if something has always been like a particular way then that does not mean we can't look for betterment.
\n\nHopefully, at least something will be done about all this. I would love Kaggle to interact with us too and share their perspective. After all, the responsibility lies on each one of us to be genuine and put in efforts to make this platform better.
", "rawMarkdown": "Hey CPMP, I think for some people 'cheating' would come across as too strong a word to digest, so never mind the down votes! I personally agree to your point of view, because this is one of the valid interpretations of the FAQ that are possible. For me, the real problem is this fact itself that it is open to interpretations and is not technically complete. In fact, I've not seen any kind of a service provider depending on an 'FAQ' for rules enforcement, this looks very unprofessional I am afraid. There should be dedicated page/s legally explaining all the rules for participation in single or two stage competitions and these pages should be linked to the rules in the competitions' sections where we actually agree to the terms. \n\nRegarding the point you've brought up though, I think there should be a simple solution.\n\n> **Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?**\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.\n\n\nAs far as I've understood, the above quote is the part where you think Kaggle is allowing the upload of models by participant users in the stage 2 even when no models have been uploaded by them in stage 1, and in the process, Kaggle is giving them an unfair advantage in the context of rankings right ? I don't know much about the rankings yet, but if it's some kind of privilege one would get, then surely it'd be unfair to the user who has submitted a model in stage 1 and is bound by rules not to modify code. Whereas, the other users who have not submitted in stage 1 are free to do anything, even use stage 2 data anyway they want to get ranked. This is what I understand the problem is from your point of view. \n\n\nNow, lets try to understand another interpretation. The clarity with which Kaggle says 'Yes' in the above quote makes me think that Kaggle surely is intentionally allowing this with a definite purpose. They are removing the stage 2 only participant from the competition standings entirely too. So, taking everything in good faith (why would such a reputed organization abet cheating?), I'd like to think that there are good reasons to allow it. May be because they want to let users get the experience of a live competition and gauge themselves even when they've missed the opportunity otherwise ? This is good too as it encourages learning and it goes with the overall aims of the platform. \n\n\nHere we've two point of views, and both are true at the same time I think. The real problem is, there is scope for cheating for the users who WANT to cheat! Neither Kaggle nor any genuine participant user would want to abet cheating I am sure. So, lets come to the real part of the problem which I think is allowing a place in the LB to users who are taking part only in stage 2. I am not sure, but I guess it helps stage 2 only participants to gauge their performance by looking at their position there ? This must be important too for a genuine stage 2 only participant, coming up on the LB that is. \n\n\nSo, ultimately I am getting the following two key points:\n\n- It IS important to allow stage 2 only participants and putting them on the LB too\n- There should not be ANY scope for cheating whatsoever\n\n\nI might be missing something, but as far as I understand the problem and given the above two key points are true, I'd suggest a simple solution for it as follows:\n\n- Allow a participant user to upload code in stage 2 even if s/he has not uploaded a model in stage 1. Put them on the LB too, BUT, this LB should be a special 'Personal LB View' that is only visible to that particular user. It is just a view, so the user will not get ANY rank in actuality (if one wants to get ranked then they need to work hard and take part from the start). But they will have all the facility to gauge their performance by looking at this personal view of the LB. This scheme fulfills all the genuine requirements. \n\nThis will keep everyone happy and things clean. From a software point of view, it is very much implementable as everything is already in place. Adding a special view should not be too much of a problem. As also, if something has always been like a particular way then that does not mean we can't look for betterment. \n\n\nHopefully, at least something will be done about all this. I would love Kaggle to interact with us too and share their perspective. After all, the responsibility lies on each one of us to be genuine and put in efforts to make this platform better.\n" }, { "id": 315104, "postDate": "2018-04-16T18:48:07.490Z", "content": "\n\n\nFor me, the real problem is this fact itself that it is open to interpretations and is not technically complete.
\n
I agree, and I would add it is self contradicting. Saying in the same page that you have to upload your model by some deadline, and that you can still compete after deadline if you don't is a contradiction.
\n\nI also agree with what you write in general.
", "rawMarkdown": "> For me, the real problem is this fact itself that it is open to interpretations and is not technically complete. \n\nI agree, and I would add it is self contradicting. Saying in the same page that you have to upload your model by some deadline, and that you can still compete after deadline if you don't is a contradiction.\n\nI also agree with what you write in general.", "votes": 1 } ] }, { "id": 314970, "postDate": "2018-04-16T15:26:09.690Z", "content": "I am using this competition to learn about python and deep learning in general (I have learned a lot!). I know I have no chance of winning a prize based on my stage 1 score (top 30%) but I had to make some changes to get a valid .csv file for stage 2 submission (due to atypical data compared to stage 1). I can easily see there are many more things for me to learn.
\n\nI don't think continuous learning should be considered cheating, it should actually be encouraged!
\n\ntx
\n\nKickback
", "rawMarkdown": "I am using this competition to learn about python and deep learning in general (I have learned a lot!). I know I have no chance of winning a prize based on my stage 1 score (top 30%) but I had to make some changes to get a valid .csv file for stage 2 submission (due to atypical data compared to stage 1). I can easily see there are many more things for me to learn. \n\n I don't think continuous learning should be considered cheating, it should actually be encouraged!\n\ntx\n\nKickback", "votes": 4, "replies": [ { "id": 314977, "postDate": "2018-04-16T15:35:53.270Z", "content": "I agree with you on this.
", "rawMarkdown": "I agree with you on this." }, { "id": 315025, "postDate": "2018-04-16T16:40:59.760Z", "content": "@kickback Changes necessary just to make a valid submission csv are certainly ok..\nHere the issue is modifying preprocessing parameters during stage 2 etc.
", "rawMarkdown": "@kickback Changes necessary just to make a valid submission csv are certainly ok..\nHere the issue is modifying preprocessing parameters during stage 2 etc.", "votes": 1 } ] }, { "id": 314384, "postDate": "2018-04-15T13:07:09.290Z", "content": "@CPMP No offense but I think that it's always been like that in every other 2 stage competition hosted. If they want to enforce code lock then they choose to host kernel only competition. Besides, top 10-20 teams are going to abide by the code rule enforcement, it is too big a risk to lose the prize. As far as the others are concerned, it is pretty time consuming for Kaggle to manually crosscheck codes and submissions, given the different parameters and enviroment variety, since I repeat it is not a kernel only competition where all people use the same platform. It is difficult already to do so for the top winning ones. It is this variety they want, given that the problem difficulty is such that hosting a kernel only competition would limit the solution range. So please take it cool, and note that the term \"cheating\" you chose is exaggerated and offending to other Kagglers who know it was always like that in 2 stage competitions. That's the reason you are being downvoted.
", "rawMarkdown": "@CPMP No offense but I think that it's always been like that in every other 2 stage competition hosted. If they want to enforce code lock then they choose to host kernel only competition. Besides, top 10-20 teams are going to abide by the code rule enforcement, it is too big a risk to lose the prize. As far as the others are concerned, it is pretty time consuming for Kaggle to manually crosscheck codes and submissions, given the different parameters and enviroment variety, since I repeat it is not a kernel only competition where all people use the same platform. It is difficult already to do so for the top winning ones. It is this variety they want, given that the problem difficulty is such that hosting a kernel only competition would limit the solution range. So please take it cool, and note that the term \"cheating\" you chose is exaggerated and offending to other Kagglers who know it was always like that in 2 stage competitions. That's the reason you are being downvoted.", "votes": 4, "replies": [ { "id": 314386, "postDate": "2018-04-15T13:17:42.700Z", "content": "No offense taken ;)
\n\nI am not asking for checking manually that all teams did not change code. I'm asking that rules are enforced fairly. The quote I provide above says people have to upload code. People who do not comply should not be allowed to keep going.
\n\nI've lost a gold medal in a recent competition because someone I teamed with did not comply with some rule. Why would people not complying with rules be allowed to compete here?
\n\nAnyway, I don't want to argue with you or other competitors, I asked a question to Kaggle team. Let's see what the response will be, if any.
\n\nLet me add that I'm so cool now that it astonishes me ;) I'm just waiting for my retraining with stage1 test labels to complete and I'll be done with this competition. As I wrote elsewhere, except for the fuzziness about stage2 rules, it was a great learning experience.
", "rawMarkdown": "No offense taken ;)\n\nI am not asking for checking manually that all teams did not change code. I'm asking that rules are enforced fairly. The quote I provide above says people have to upload code. People who do not comply should not be allowed to keep going. \n\nI've lost a gold medal in a recent competition because someone I teamed with did not comply with some rule. Why would people not complying with rules be allowed to compete here? \n\nAnyway, I don't want to argue with you or other competitors, I asked a question to Kaggle team. Let's see what the response will be, if any.\n\nLet me add that I'm so cool now that it astonishes me ;) I'm just waiting for my retraining with stage1 test labels to complete and I'll be done with this competition. As I wrote elsewhere, except for the fuzziness about stage2 rules, it was a great learning experience.", "votes": 4 }, { "id": 314388, "postDate": "2018-04-15T13:22:03.957Z", "content": "Fair enough,however if they did that they would be inconsistent compared to previously held 2 stage competitions . Anyway, good luck to you and all other Kagglers!
", "rawMarkdown": "Fair enough,however if they did that they would be inconsistent compared to previously held 2 stage competitions . Anyway, good luck to you and all other Kagglers!", "votes": 2 }, { "id": 314394, "postDate": "2018-04-15T13:34:38.057Z", "content": "@gavrand, I think @CPMP suggested Kaggle changing the 2 stage rules without making extra effort to check if people are being honest at the 2nd stage. I know how much work for Kaggle admins to crosscheck the code already. As you suggested, the rules have always been like this, but it doesn't mean the rules can not be improved for the future competitions to save work on both sides:
\n\nMy 2 cents:
\n\nstage1_solution.csv because we expected the solution would have been released in the same format as stage1_train. And other participants hit problems like reading some images which were in jpeg/tf format and they had to add code to handle this. This could've been prevented. @Nicole Finnie What's the point? I mean if you don't crosscheck then anyone could have just submitted a dummy code?
", "rawMarkdown": "@Nicole Finnie What's the point? I mean if you don't crosscheck then anyone could have just submitted a dummy code?" }, { "id": 314397, "postDate": "2018-04-15T13:50:31.660Z", "content": "@gavrand, you made a valid point, no system will be bullet proof. It won't stop all of them, at least it will discourage some of them. I believe people who pursue DS career will use their Kaggle achievements for their resumes in the future. That's why an open discussion is always good to make the ranking system of this platform more reliable and trustworthy. I think Kaggle will appreciate different opinions too.
", "rawMarkdown": "@gavrand, you made a valid point, no system will be bullet proof. It won't stop all of them, at least it will discourage some of them. I believe people who pursue DS career will use their Kaggle achievements for their resumes in the future. That's why an open discussion is always good to make the ranking system of this platform more reliable and trustworthy. I think Kaggle will appreciate different opinions too.", "votes": 3 }, { "id": 314603, "postDate": "2018-04-15T23:21:04.603Z", "content": "@Nicole, I can not agree more. I believe the NIPS 2017 adversarial attack competitions' strategies are more appropriate for 2-stage competitions. Template code are provided to standardize input format and enable error checking (to make sure you code can run successfully in the final stage).
\n\nI believe the major concern here is allowing participants to modify (non scientific change) and retrain on their own, which opens the door of cheating. If templates are provided, Kaggle could run the retraining/inference script on their GPU clusters.
", "rawMarkdown": "@Nicole, I can not agree more. I believe the NIPS 2017 adversarial attack competitions' strategies are more appropriate for 2-stage competitions. Template code are provided to standardize input format and enable error checking (to make sure you code can run successfully in the final stage).\n\nI believe the major concern here is allowing participants to modify (non scientific change) and retrain on their own, which opens the door of cheating. If templates are provided, Kaggle could run the retraining/inference script on their GPU clusters. ", "votes": 4 }, { "id": 314752, "postDate": "2018-04-16T07:55:42.647Z", "content": "@Rand Xie, interesting. Let' s hope Kaggle team will use that approach for 2 stages competitions from now on: code template, and no retraining in stage2.
", "rawMarkdown": "@Rand Xie, interesting. Let' s hope Kaggle team will use that approach for 2 stages competitions from now on: code template, and no retraining in stage2." }, { "id": 314823, "postDate": "2018-04-16T10:22:48.997Z", "content": "Yes, I'm in for the idea on kernels being used to freeze code following completion of Stage 1 for future 2-stage competitions as well. However, the only issue I have with the kernels is that some of the commonly-used libraries are not available for download and testing in Kaggle's kernel environment (I use R by the way). And yes, I'm also aware that a number of contestants have resorted to training their models with external datasets downloaded from elsewhere (discussed in https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572), and even the Stage 1 test set (when the labels have been released), while this can also be seen as a form of \"cheating\", since by training your models with Stage 1 test data or external datasets, the training algorithm will compute the weights of all of the respective nodes differently as compared to the situation in which you were training it on the provided Stage 1 training data alone, resulting in a very different model from what you would have obtained if you had used Stage 1 training data alone (for the training & construction of your models).
", "rawMarkdown": "Yes, I'm in for the idea on kernels being used to freeze code following completion of Stage 1 for future 2-stage competitions as well. However, the only issue I have with the kernels is that some of the commonly-used libraries are not available for download and testing in Kaggle's kernel environment (I use R by the way). And yes, I'm also aware that a number of contestants have resorted to training their models with external datasets downloaded from elsewhere (discussed in https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572), and even the Stage 1 test set (when the labels have been released), while this can also be seen as a form of \"cheating\", since by training your models with Stage 1 test data or external datasets, the training algorithm will compute the weights of all of the respective nodes differently as compared to the situation in which you were training it on the provided Stage 1 training data alone, resulting in a very different model from what you would have obtained if you had used Stage 1 training data alone (for the training & construction of your models)." } ] }, { "id": 486136, "postDate": "2019-03-08T10:56:39.927Z", "content": "spam :(
", "rawMarkdown": "spam :(", "votes": -1 }, { "id": 314676, "postDate": "2018-04-16T04:54:18.327Z", "content": "True
", "rawMarkdown": "True", "votes": 1 }, { "id": 316023, "postDate": "2018-04-18T03:04:50.770Z", "content": "At the beginning of stage2, kaggle released the solution to stage1 which is an important supplement to the training data. I wonder if those got score more than 0.55 did not use those solutions to retrain their model, then how can they finalize their model before the solutions were released? \nAnd also, the test data in stage2 contained lots of unseen types, which means it is very hard to get a better score than stage1 without argument the training data to suit this situation. If not, how can the top 100 got 0.1 increase on an unfit data? Which means most of the players did not submit their prediction from the model in stage1 but did some further work on it. \nSo, the model uploaded in stage1 is just a 'sign in', no guarantee at all. If we only focus on the 'code', then we miss the effect of training data, one can just manipulate on training data without touch the code to re-train their model. \nTo fix this issue once for all, kaggle should require to submit the model file like .h5, then use the model to predict on the hidden data by themselves.
", "rawMarkdown": "At the beginning of stage2, kaggle released the solution to stage1 which is an important supplement to the training data. I wonder if those got score more than 0.55 did not use those solutions to retrain their model, then how can they finalize their model before the solutions were released? \nAnd also, the test data in stage2 contained lots of unseen types, which means it is very hard to get a better score than stage1 without argument the training data to suit this situation. If not, how can the top 100 got 0.1 increase on an unfit data? Which means most of the players did not submit their prediction from the model in stage1 but did some further work on it. \nSo, the model uploaded in stage1 is just a 'sign in', no guarantee at all. If we only focus on the 'code', then we miss the effect of training data, one can just manipulate on training data without touch the code to re-train their model. \nTo fix this issue once for all, kaggle should require to submit the model file like .h5, then use the model to predict on the hidden data by themselves.", "votes": 2, "replies": [ { "id": 316059, "postDate": "2018-04-18T05:34:23.273Z", "content": "I hope for the teams in prize money position that they did comply with the rules of the competition. If not then they'll be removed. In a previous 2 stages competition 2 of the top 5 teams were removed.
", "rawMarkdown": "I hope for the teams in prize money position that they did comply with the rules of the competition. If not then they'll be removed. In a previous 2 stages competition 2 of the top 5 teams were removed." }, { "id": 316077, "postDate": "2018-04-18T06:24:41.390Z", "content": "Then kaggle should emphasis the 'upload model', both in rule and the web page, code a specific button for it, for newbies like me who never noticed that thing. \nIt feels very unfair to be kickoff by such an incomplete even somehow useless law without even a button on the main page, after all we spent days on this competition.\nFinally, the law can not deal with the reproduce issue, if they can, there is no need to hold a stage2, just upload the model and kaggle should do the final prediction.
", "rawMarkdown": "Then kaggle should emphasis the 'upload model', both in rule and the web page, code a specific button for it, for newbies like me who never noticed that thing. \nIt feels very unfair to be kickoff by such an incomplete even somehow useless law without even a button on the main page, after all we spent days on this competition.\nFinally, the law can not deal with the reproduce issue, if they can, there is no need to hold a stage2, just upload the model and kaggle should do the final prediction." }, { "id": 316085, "postDate": "2018-04-18T07:04:56.673Z", "content": "To your other point, about good scores in stage2, our own score went from0.504 to 0.545 with the exact same code. It seems stage2 images were a bit easier to predict on average.
", "rawMarkdown": "To your other point, about good scores in stage2, our own score went from0.504 to 0.545 with the exact same code. It seems stage2 images were a bit easier to predict on average." }, { "id": 316093, "postDate": "2018-04-18T07:27:04.077Z", "content": "@CPMP: I came to the same conclusion that stage-2 was easier than stage-1. Using the exact same model in stage-1, I got 0.400 in stage 1, and 0.439 in stage-2. Adding the solutions images in stage-1 and retraining without touching parameters produced a private score of 0.470 for stage-2. But the number of test images in stage-1 was much smaller than stage-2 so I'm not sure how much statistical variance we have as a consequence.
", "rawMarkdown": "@CPMP: I came to the same conclusion that stage-2 was easier than stage-1. Using the exact same model in stage-1, I got 0.400 in stage 1, and 0.439 in stage-2. Adding the solutions images in stage-1 and retraining without touching parameters produced a private score of 0.470 for stage-2. But the number of test images in stage-1 was much smaller than stage-2 so I'm not sure how much statistical variance we have as a consequence." }, { "id": 316097, "postDate": "2018-04-18T07:32:22.723Z", "content": "That's the point, how can you prove you are using the same code and same training data to get a better score in a different distribution? \nThe only way is that using the same model, which means same architecture and same weights. To to this, it is not complete to just upload the code but also the model itsself, because even random seed can cause difference in the result.
", "rawMarkdown": "That's the point, how can you prove you are using the same code and same training data to get a better score in a different distribution? \nThe only way is that using the same model, which means same architecture and same weights. To to this, it is not complete to just upload the code but also the model itsself, because even random seed can cause difference in the result." }, { "id": 316098, "postDate": "2018-04-18T07:35:15.013Z", "content": "I can prove it because I uploaded my code before deadline. If you run it then you can reproduce my results. You are right about model weights, but my team mate Nicole asked the question, and we were told that we don't need to upload model weights. Moreover, organizers also wrote that it was fine to retrain with stage1 test labels added to the training data we used in stage 1. Maybe you missed that.
\n\nNot sure what you are after actually. My post was about a contradiction within the stage 2 FAQ that could encourage people to ignore the stage1 upload deadline. I don't know if you agree this is an issue or not. For the rest, please argue with organizers, not me. I'm not the one setting the rules here.
", "rawMarkdown": "I can prove it because I uploaded my code before deadline. If you run it then you can reproduce my results. You are right about model weights, but my team mate Nicole asked the question, and we were told that we don't need to upload model weights. Moreover, organizers also wrote that it was fine to retrain with stage1 test labels added to the training data we used in stage 1. Maybe you missed that.\n\nNot sure what you are after actually. My post was about a contradiction within the stage 2 FAQ that could encourage people to ignore the stage1 upload deadline. I don't know if you agree this is an issue or not. For the rest, please argue with organizers, not me. I'm not the one setting the rules here." }, { "id": 316099, "postDate": "2018-04-18T07:35:42.613Z", "content": "@Oscar Then you are using a different training data which is certain that you can not get it before you upload your model.\nAgain, training data is the part which the law does not cover.
", "rawMarkdown": "@Oscar Then you are using a different training data which is certain that you can not get it before you upload your model.\nAgain, training data is the part which the law does not cover." }, { "id": 316100, "postDate": "2018-04-18T07:41:12.720Z", "content": "@Yunfei, for this competition, training the stage 1 test data is allowed, the Kaggle admin gave us the green light in this post. It was vaguly addressed in the 2 stage FAQ, so we explicitly asked again.\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54299
\n\nAnd in this post, Wendy explicitly said you have to upload your code, not the model files, and you can retrain your model.\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/52385#298629
\n\nAnd you're right, since the rules are not crystal clear, there's a grey area for further discussions and improvement of rules.
", "rawMarkdown": "@Yunfei, for this competition, training the stage 1 test data is allowed, the Kaggle admin gave us the green light in this post. It was vaguly addressed in the 2 stage FAQ, so we explicitly asked again.\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54299\n\nAnd in this post, Wendy explicitly said you have to upload your code, not the model files, and you can retrain your model.\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/52385#298629\n\nAnd you're right, since the rules are not crystal clear, there's a grey area for further discussions and improvement of rules. ", "votes": 3 }, { "id": 316102, "postDate": "2018-04-18T07:44:21.217Z", "content": "Hi @Yunfei. I uploaded before the deadline instructions and code (SHA256 checksum) on how to create the model. I believe the rules permitted using the disclosed additional masks for re-training on the second stage as long as the code and parameters were left untouched.
", "rawMarkdown": "Hi @Yunfei. I uploaded before the deadline instructions and code (SHA256 checksum) on how to create the model. I believe the rules permitted using the disclosed additional masks for re-training on the second stage as long as the code and parameters were left untouched.", "votes": 1 }, { "id": 316106, "postDate": "2018-04-18T07:48:45.727Z", "content": "@CPMP \nThe law does not cover the training data which is vital to the result. And this makes the law incomplete. Anyone who really think about it can make this clear. As your title says, this law still \"encourage\" cheating. So I do not agree with it. It is just a 'sign in'.
", "rawMarkdown": "@CPMP \nThe law does not cover the training data which is vital to the result. And this makes the law incomplete. Anyone who really think about it can make this clear. As your title says, this law still \"encourage\" cheating. So I do not agree with it. It is just a 'sign in'." }, { "id": 316110, "postDate": "2018-04-18T07:57:48.213Z", "content": "Thank you for the information, I should visit forum more.
", "rawMarkdown": "Thank you for the information, I should visit forum more." }, { "id": 316112, "postDate": "2018-04-18T08:00:02.843Z", "content": "@Yunfei Duan: Precisely, this is the exact reason why this competition promotes double standards, which would be the exact definition of unfairness and inequality. How can the organizers encourage participants to use the Stage 1 test data for re-training, but regard those who did not upload their model before the Stage 1 deadline as cheating/not complying with the rules? By allowing the participants to re-train their models on the Stage 1 test set & its labels (when the solution to the Stage 1 test set was released), the organizers are in fact encouraging cheating, since re-training the model will definitely change the weights of the individual nodes in the model (if a type of neural network is being used, which was the most common type of model assimilated by most participants in this competition). This would indeed be equivalent to submitting dummy code in the first instance (just to meet the timeline of model submission), and then making the required changes (as & when the participants deem fit) so as to attain a better private LB score. So this competition is indeed rigged, as I suspected all along ...
", "rawMarkdown": "@Yunfei Duan: Precisely, this is the exact reason why this competition promotes double standards, which would be the exact definition of unfairness and inequality. How can the organizers encourage participants to use the Stage 1 test data for re-training, but regard those who did not upload their model before the Stage 1 deadline as cheating/not complying with the rules? By allowing the participants to re-train their models on the Stage 1 test set & its labels (when the solution to the Stage 1 test set was released), the organizers are in fact encouraging cheating, since re-training the model will definitely change the weights of the individual nodes in the model (if a type of neural network is being used, which was the most common type of model assimilated by most participants in this competition). This would indeed be equivalent to submitting dummy code in the first instance (just to meet the timeline of model submission), and then making the required changes (as & when the participants deem fit) so as to attain a better private LB score. So this competition is indeed rigged, as I suspected all along ...", "votes": -1 }, { "id": 316120, "postDate": "2018-04-18T08:10:16.737Z", "content": "\"NOTE: you can always upload just one file to Kaggle, so if you decide to go with the hash upload just the hash, otherwise upload a zip file (which could include code + hash of the weights file for example)
\n\nRegarding the 2 submissions - you can have multiple model / code variants (but all need to be published BEFORE the deadline), so you can produce any number of submissions for Stage 2; in the end you will have to choose only two of the submission (which in your opinion have the highest chance of winning)\"
\n\nThere are a lot of gray areas can be settled. \nKaggle should develop these rule into its own logics and web interfaces with obvious notices. The reason why Kaggle do not do the final prediction may be that they can not afford so much computational cost.
", "rawMarkdown": "\"NOTE: you can always upload just one file to Kaggle, so if you decide to go with the hash upload just the hash, otherwise upload a zip file (which could include code + hash of the weights file for example)\n\nRegarding the 2 submissions - you can have multiple model / code variants (but all need to be published BEFORE the deadline), so you can produce any number of submissions for Stage 2; in the end you will have to choose only two of the submission (which in your opinion have the highest chance of winning)\"\n\nThere are a lot of gray areas can be settled. \nKaggle should develop these rule into its own logics and web interfaces with obvious notices. The reason why Kaggle do not do the final prediction may be that they can not afford so much computational cost.\n" }, { "id": 316217, "postDate": "2018-04-18T13:00:09.427Z", "content": "Interesting, I missed this part, I always assumed you have to decide all the parameters for two submissions when uploading the model. Allowing to upload multiple code variants opens the door for more misuse, for example it's possible to upload huge number of possible submissions covering a grid of hyperparameters values.
\n\nIt's certainly worth to clarify and tune the rules. Would be much more straightforward if all the weights to be finalized before the second stage release and only prediction run on the second stage data, with only two predefined submissions.
", "rawMarkdown": "Interesting, I missed this part, I always assumed you have to decide all the parameters for two submissions when uploading the model. Allowing to upload multiple code variants opens the door for more misuse, for example it's possible to upload huge number of possible submissions covering a grid of hyperparameters values.\n\nIt's certainly worth to clarify and tune the rules. Would be much more straightforward if all the weights to be finalized before the second stage release and only prediction run on the second stage data, with only two predefined submissions.", "votes": 7 }, { "id": 316231, "postDate": "2018-04-18T13:45:21.970Z", "content": "Not only do you have to comply with the law, you should comply with the spirit of the law.
\n\nNot sure why you guys and gals are so hung up with this. The spirit of the rule is that you should not change your code or hyperparameter, unless your code tunes it automatically. You are also allowed to change code to handle new data.
\n\nSince they are not showing the score on second state leader board, I think it will be very tough to cheat anyway. I was socked to find out Heng's team scored so low in the public leaderboard. His model scored very high in stage 1 but it seems to have overfit to stage 1 test data. He also said that his submission was not the best scoring model. If an expert like him could not figure out the best scoring model, what change do you think us, mere mortals, have to train better scoring model without public leaderboard guiding us? I'm all for clear rules and communications. (I was shocked, see 0 score when I first submitted my state 2 answer) But I think they have done a good job, making his 2 stage competition as fair as they can and I'm sure the next 2 state kaggle holds will be better.
\n\nJust my 2 cents. :)
", "rawMarkdown": "Not only do you have to comply with the law, you should comply with the spirit of the law.\n\nNot sure why you guys and gals are so hung up with this. The spirit of the rule is that you should not change your code or hyperparameter, unless your code tunes it automatically. You are also allowed to change code to handle new data. \n\nSince they are not showing the score on second state leader board, I think it will be very tough to cheat anyway. I was socked to find out Heng's team scored so low in the public leaderboard. His model scored very high in stage 1 but it seems to have overfit to stage 1 test data. He also said that his submission was not the best scoring model. If an expert like him could not figure out the best scoring model, what change do you think us, mere mortals, have to train better scoring model without public leaderboard guiding us? I'm all for clear rules and communications. (I was shocked, see 0 score when I first submitted my state 2 answer) But I think they have done a good job, making his 2 stage competition as fair as they can and I'm sure the next 2 state kaggle holds will be better.\n\nJust my 2 cents. :)" }, { "id": 316234, "postDate": "2018-04-18T14:00:34.890Z", "content": "\n\n\nThe spirit of the rule is that you should not change your code or\n hyperparameter, unless your code tunes it automatically.
\n
The code doesn't tune anything automatically, unless you direct it to (possibly by injecting new training data, or varying certain values/hyperparameters such as the number of epochs, etc).
\n\n\n\n\nYou are also allowed to change code to handle new data.
\n
This is only allowed for non-scientific changes (e.g. file path) and not scientific ones, after you have submitted your model, since it promotes cheating.
\n\n\n\n\nSince they are not showing the score on second state leader board, I\n think it will be very tough to cheat anyway.
\n
A participant doesn't need to know his/her score on the LB to cheat. Rather, the participant can easily do a visual comparison between the predicted masks and the actual photomicrograph, to detect if there are any areas which might need to be included/removed from the masks, & subsequently include some code to do this post-processing of the predicted masks. This would already constitute a form of cheating, since the post-processing code was not in the original submitted model to begin with.
", "rawMarkdown": "> The spirit of the rule is that you should not change your code or\n> hyperparameter, unless your code tunes it automatically.\n\nThe code doesn't tune anything *automatically*, unless you direct it to (possibly by injecting new training data, or varying certain values/hyperparameters such as the number of epochs, etc).\n\n> You are also allowed to change code to handle new data.\n\nThis is only allowed for non-scientific changes (e.g. file path) and not scientific ones, after you have submitted your model, since it promotes cheating.\n\n> Since they are not showing the score on second state leader board, I\n> think it will be very tough to cheat anyway.\n\nA participant doesn't need to know his/her score on the LB to cheat. Rather, the participant can easily do a visual comparison between the predicted masks and the actual photomicrograph, to detect if there are any areas which might need to be included/removed from the masks, & subsequently include some code to do this post-processing of the predicted masks. This would already constitute a form of cheating, since the post-processing code was not in the original submitted model to begin with." }, { "id": 316249, "postDate": "2018-04-18T14:32:56.463Z", "content": "@Shiraz As you said, there are many ways to break the spirit but stay in the law. The final solution may be that kaggle release an API which calls prediction on the submitted model, all submission follow that framework.
", "rawMarkdown": "@Shiraz As you said, there are many ways to break the spirit but stay in the law. The final solution may be that kaggle release an API which calls prediction on the submitted model, all submission follow that framework." }, { "id": 316290, "postDate": "2018-04-18T16:28:13.937Z", "content": "\n\n\nA participant doesn't need to know his/her score on the LB to cheat. Rather, the participant can easily do a visual comparison ...
\n
I'm sure people who spent the time to visually inspect the output could have improved LB. Hopefully, it was the worth the time for them. If you are in the money and did something wrong, the organizer will look at their code/methodology and disqualify them. If you are not in the money and am getting away with it, shame on you.
\n\nHowever, I don't think it would affect me personally. I suspect that most people who are upset about this, are upset because they think those who did not comply with the rule may have surpassed them in LB? Due to the stochastic nature of machine learning training, I'm pretty sure more people lost their places in LB than gained if they tuned their model without the guidance of 2nd stage public LB score.
\n\nTo be honest, after I realized that they intentionally showed 0 for all 2nd stage submissions, I felt relieved and stop working on my model. I did not even retrain my model using the 1st stage test data. I just used the best weighs from the first stage training. That model weight got me 85th in 1st stage LB but my final standing is 44th.
\n\nI'm interested in machine learning/AI/computer vision. I'd rather spend my time learning the algorithm and training techniques and not worry about minute details of the competition. Though I was a bit shocked and distressed when I saw 0 score for second stage submission (my understanding is that this is the first time they did it), I think it worked out very well to keep the competition more honest and interesting. I'm sure the organizers/kaggle are learning about how to run 2 stage competition better and the next one will be even more awesome than this one.
", "rawMarkdown": "> A participant doesn't need to know his/her score on the LB to cheat. Rather, the participant can easily do a visual comparison ...\n\nI'm sure people who spent the time to visually inspect the output could have improved LB. Hopefully, it was the worth the time for them. If you are in the money and did something wrong, the organizer will look at their code/methodology and disqualify them. If you are not in the money and am getting away with it, shame on you.\n\nHowever, I don't think it would affect me personally. I suspect that most people who are upset about this, are upset because they think those who did not comply with the rule may have surpassed them in LB? Due to the stochastic nature of machine learning training, I'm pretty sure more people lost their places in LB than gained if they tuned their model without the guidance of 2nd stage public LB score. \n\nTo be honest, after I realized that they intentionally showed 0 for all 2nd stage submissions, I felt relieved and stop working on my model. I did not even retrain my model using the 1st stage test data. I just used the best weighs from the first stage training. That model weight got me 85th in 1st stage LB but my final standing is 44th. \n\nI'm interested in machine learning/AI/computer vision. I'd rather spend my time learning the algorithm and training techniques and not worry about minute details of the competition. Though I was a bit shocked and distressed when I saw 0 score for second stage submission (my understanding is that this is the first time they did it), I think it worked out very well to keep the competition more honest and interesting. I'm sure the organizers/kaggle are learning about how to run 2 stage competition better and the next one will be even more awesome than this one. ", "votes": 3 }, { "id": 316378, "postDate": "2018-04-18T21:23:45.060Z", "content": "We are talking about the detail and lack of the rules, not the big spirit. Big spirit is always right, like peace, honest. Nobody is against the spirit of kaggle, but there is no use to stick on the big topic while the rule is vulnerable. It is just different topic.
", "rawMarkdown": "We are talking about the detail and lack of the rules, not the big spirit. Big spirit is always right, like peace, honest. Nobody is against the spirit of kaggle, but there is no use to stick on the big topic while the rule is vulnerable. It is just different topic." }, { "id": 316431, "postDate": "2018-04-19T03:00:04.273Z", "content": "\n\n\nI'm sure people who spent the time to visually inspect the output\n could have improved LB. Hopefully, it was the worth the time for them.\n If you are in the money and did something wrong, the organizer will\n look at their code/methodology and disqualify them. If you are not in\n the money and am getting away with it, shame on you.
\n
That is essentially what this whole discussion is about - on what should be permitted and what shouldn't. Are people who inspect their code visually (after testing their model on the Stage 2 test data) and realizing that they can improve their LB ranking by making some changes to their existing code/model (e.g. incorporating new filters, etc) allowed to do so, or would this be violating the competition rules?
\n\n\n\n\nHowever, I don't think it would affect me personally. I suspect that\n most people who are upset about this, are upset because they think\n those who did not comply with the rule may have surpassed them in LB?
\n
Possibly so, that is after all the spirit of any competition, i.e. to have a friendly rivalry amongst the participants, thus it would be expected of most participants to aim for medals and prizes. However, there will undeniably be some individuals who couldn't care less about the LB rankings, since they participate to gain some knowledge/experience in this respect. Personally, I'd say that I am of the second group in this context, since I did not even use any ML algorithm/packages for this competition but wanted to evaluate the accuracy and capability of my self-devised algorithm. I expected the results to be poor for this current context, since I did not even have the time to run my model on >530 images in the Stage 2 test set, so I left those as blank entries instead, which caused my solution to land in 817th place ... ;)
\n\n\n\n\nDue to the stochastic nature of machine learning training, I'm pretty\n sure more people lost their places in LB than gained if they tuned\n their model without the guidance of 2nd stage public LB score.
\n
That's not necessarily so, since it's not exactly stochastic as some individuals would realize the importance of filters for the 2nd test set, when this might not be so apparent for Stage 1 predictions, leading some to make revisions to their existing model to evaluate the Stage 2 test data (these revisions may include code for these filters in their models, making changes to the Stage 2 test images to accomodate their current model setup, etc). This is where the issue of cheating comes in - these participants who have made such revisions are rightfully gaining an unfair advantage over other contestants.
\n\n\n\n\nI'm interested in machine learning/AI/computer vision. I'd rather\n spend my time learning the algorithm and training techniques and not\n worry about minute details of the competition.
\n
That is a pretty good perspective to take on this issue, and to be frank, I share your sentiments in this context as well. However, details like these are essential and should be clearly specified for the conduct of subsequent 2-stage competitions in Kaggle to ensure fairness amongst all participants & prevent further abuses of the system and/or debates of this nature arising in this context.
", "rawMarkdown": "> I'm sure people who spent the time to visually inspect the output\n> could have improved LB. Hopefully, it was the worth the time for them.\n> If you are in the money and did something wrong, the organizer will\n> look at their code/methodology and disqualify them. If you are not in\n> the money and am getting away with it, shame on you.\n\nThat is essentially what this whole discussion is about - on what should be permitted and what shouldn't. Are people who inspect their code visually (after testing their model on the Stage 2 test data) and realizing that they can improve their LB ranking by making some changes to their existing code/model (e.g. incorporating new filters, etc) ***allowed*** to do so, or would this be violating the competition rules?\n\n> However, I don't think it would affect me personally. I suspect that\n> most people who are upset about this, are upset because they think\n> those who did not comply with the rule may have surpassed them in LB?\n\nPossibly so, that is after all the spirit of any competition, i.e. to have a friendly rivalry amongst the participants, thus it would be expected of most participants to aim for medals and prizes. However, there will undeniably be some individuals who couldn't care less about the LB rankings, since they participate to gain some knowledge/experience in this respect. Personally, I'd say that I am of the second group in this context, since I did not even use any ML algorithm/packages for this competition but wanted to evaluate the accuracy and capability of my self-devised algorithm. I expected the results to be poor for this current context, since I did not even have the time to run my model on >530 images in the Stage 2 test set, so I left those as blank entries instead, which caused my solution to land in 817th place ... ;)\n\n> Due to the stochastic nature of machine learning training, I'm pretty\n> sure more people lost their places in LB than gained if they tuned\n> their model without the guidance of 2nd stage public LB score.\n\nThat's not necessarily so, since it's not *exactly* stochastic as some individuals would realize the importance of filters for the 2nd test set, when this might not be so apparent for Stage 1 predictions, leading some to make *revisions* to their existing model to evaluate the Stage 2 test data (these revisions may include code for these filters in their models, making changes to the Stage 2 test images to accomodate their current model setup, etc). This is where the issue of cheating comes in - these participants who have made such *revisions* are rightfully gaining an unfair advantage over other contestants.\n\n> I'm interested in machine learning/AI/computer vision. I'd rather\n> spend my time learning the algorithm and training techniques and not\n> worry about minute details of the competition.\n\nThat is a pretty good perspective to take on this issue, and to be frank, I share your sentiments in this context as well. However, details like these are essential and should be clearly specified for the conduct of subsequent 2-stage competitions in Kaggle to ensure fairness amongst all participants & prevent further abuses of the system and/or debates of this nature arising in this context." } ] }, { "id": 314985, "postDate": "2018-04-16T15:48:58.913Z", "content": "This is a machine learning problem on how you ranked the following words in order of importance:
\n\n\"integrity\", \"fame\", \"correct\", \"wrong\", \"rule\", \"results\", \"value\", ...
", "rawMarkdown": "This is a machine learning problem on how you ranked the following words in order of importance:\n\n\"integrity\", \"fame\", \"correct\", \"wrong\", \"rule\", \"results\", \"value\", ...", "votes": 2, "replies": [ { "id": 314990, "postDate": "2018-04-16T15:56:25.837Z", "content": "from my personal experience, rules are not #1 in kaggle. I had the case when I asked if I had to follow the rules, and was told that for that specific case it was not necessary by a kaggle team member :\\ but I did follow them in the end...
", "rawMarkdown": "from my personal experience, rules are not #1 in kaggle. I had the case when I asked if I had to follow the rules, and was told that for that specific case it was not necessary by a kaggle team member :\\ but I did follow them in the end..." }, { "id": 314993, "postDate": "2018-04-16T15:57:51.003Z", "content": "and the example of ods.ai internally sharing data in many segmentation competitions. this has been a thing nobody even cares to question these days.
", "rawMarkdown": "and the example of ods.ai internally sharing data in many segmentation competitions. this has been a thing nobody even cares to question these days.", "votes": 2 }, { "id": 314999, "postDate": "2018-04-16T16:12:02.093Z", "content": "@raddar do you have evidence of sharing the data on ods?
", "rawMarkdown": "@raddar do you have evidence of sharing the data on ods? ", "votes": 16 }, { "id": 315017, "postDate": "2018-04-16T16:32:36.167Z", "content": "I first internally shared my data as a child and it traumatized me so much.
", "rawMarkdown": "I first internally shared my data as a child and it traumatized me so much.", "votes": 7 }, { "id": 315023, "postDate": "2018-04-16T16:38:42.123Z", "content": "Hi @raddar, ods.ai community is extremely careful to follow the rules of no private code or data sharing outside already formed (on kaggle) teams. For camera competition specifically please see this post by kaggle admins who double-checked that: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49268#279846
", "rawMarkdown": "Hi @raddar, ods.ai community is extremely careful to follow the rules of no private code or data sharing outside already formed (on kaggle) teams. For camera competition specifically please see this post by kaggle admins who double-checked that: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49268#279846", "votes": 8 }, { "id": 315034, "postDate": "2018-04-16T16:57:18.203Z", "content": "@Konstantin, interesting discussion indeed. It says it is easy to join ods.ai, but I don't see how to join, neither when I go to https://opendatascience.slack.com/ nor to ods.ai site and download the slides. Can you help?
", "rawMarkdown": "@Konstantin, interesting discussion indeed. It says it is easy to join ods.ai, but I don't see how to join, neither when I go to https://opendatascience.slack.com/ nor to ods.ai site and download the slides. Can you help?", "votes": 1 }, { "id": 315036, "postDate": "2018-04-16T16:58:11.913Z", "content": "Thanks Konstantin. I probably feel that I have to apologize. It's kind of frustrating at the same time, because I was referring to the same competition. It is not so obvious that flickr was that good source to extend the given data. And it seems only ods.ai teams used it (my impression after the solutions posted in forums).
\n\nI don't have anything ods.ai, in fact I wanted to join it but couldn't for some reason. That is probably why I had been suspicious about it since then. Until now I really thought that it is a closed circle russian community.
\n\nNow as to a rainfall of downvotes - im pretty sure all of them come from ods.ai. I understand why you are not too happy about my opinion, but maybe given these facts you would not look at me as a \"traumatized child\".
", "rawMarkdown": "Thanks Konstantin. I probably feel that I have to apologize. It's kind of frustrating at the same time, because I was referring to the same competition. It is not so obvious that flickr was that good source to extend the given data. And it seems only ods.ai teams used it (my impression after the solutions posted in forums). \n\nI don't have anything ods.ai, in fact I wanted to join it but couldn't for some reason. That is probably why I had been suspicious about it since then. Until now I really thought that it is a closed circle russian community.\n\nNow as to a rainfall of downvotes - im pretty sure all of them come from ods.ai. I understand why you are not too happy about my opinion, but maybe given these facts you would not look at me as a \"traumatized child\". ", "votes": 13 }, { "id": 315037, "postDate": "2018-04-16T16:59:36.370Z", "content": "There is button \"Присоединиться\" (Join) on ods.ai, it will forward you to google form to request invite. Just fill it and you'll get there.
", "rawMarkdown": "There is button \"Присоединиться\" (Join) on ods.ai, it will forward you to google form to request invite. Just fill it and you'll get there. ", "votes": 7 }, { "id": 315038, "postDate": "2018-04-16T17:00:12.113Z", "content": "@raddar, ods.ai is an Open slack communtiy with 12500 users. Feel free to join :) \nThere are Kaggle Admins there as well (@inversion).
\n\nAs the community is biased towards people interested in Data Science there are a lot of users with experience in CV/DL/ML and competitions on Kaggle, Topcoder, DrivenData.
\n\nIt is insulting to presume that no one from 12500 data scientists across the world can take top places in CV competitions...
\n\nPeople usually use opensourced solutions, for example from 1st place Carvana https://github.com/asanakoy/kaggle_carvana_segmentation \nEveryone who read Kaggle blog about Carvana got that link... Everyone who has github can use it... Using public github repositories is not a violation of the rules, right?
\n\nWinning teams usually come up with new approaches that, of course, are not shared. I'm pretty sure you know it is not enough to just take some solution from github and win.
", "rawMarkdown": "@raddar, ods.ai is an Open slack communtiy with 12500 users. Feel free to join :) \nThere are Kaggle Admins there as well (@inversion). \n\nAs the community is biased towards people interested in Data Science there are a lot of users with experience in CV/DL/ML and competitions on Kaggle, Topcoder, DrivenData. \n\nIt is insulting to presume that no one from 12500 data scientists across the world can take top places in CV competitions... \n\nPeople usually use opensourced solutions, for example from 1st place Carvana https://github.com/asanakoy/kaggle_carvana_segmentation \nEveryone who read Kaggle blog about Carvana got that link... Everyone who has github can use it... Using public github repositories is not a violation of the rules, right? \n\nWinning teams usually come up with new approaches that, of course, are not shared. I'm pretty sure you know it is not enough to just take some solution from github and win. ", "votes": 3 }, { "id": 315039, "postDate": "2018-04-16T17:00:45.240Z", "content": "Thanks, didn't dare click on a button I did not understand!
", "rawMarkdown": "Thanks, didn't dare click on a button I did not understand!", "votes": 2 }, { "id": 315041, "postDate": "2018-04-16T17:05:36.143Z", "content": "I filled the form using google translate ;)
", "rawMarkdown": "I filled the form using google translate ;)", "votes": 2 }, { "id": 315043, "postDate": "2018-04-16T17:08:44.073Z", "content": "@raddar, I gave you an upvote (but I still perfectly understand my friends from ods giving you downvotes, mostly it is a group dynamics, you know). \nAs for camera competition, idea of using flickr data was on a surface and after someone posted huge archive of photos from flickr it was a matter of individual laziness (or maybe skills) to get more of the same kind. \nAlso, it wasn't the only factor to win, my team head this extremely experienced Grandmaster and it was his skill and magic that allowed us to prepare (and then select!) a winning submission with score much much lower on Public LB.
", "rawMarkdown": "@raddar, I gave you an upvote (but I still perfectly understand my friends from ods giving you downvotes, mostly it is a group dynamics, you know). \nAs for camera competition, idea of using flickr data was on a surface and after someone posted huge archive of photos from flickr it was a matter of individual laziness (or maybe skills) to get more of the same kind. \nAlso, it wasn't the only factor to win, my team head this extremely experienced Grandmaster and it was his skill and magic that allowed us to prepare (and then select!) a winning submission with score much much lower on Public LB. ", "votes": 1 }, { "id": 315061, "postDate": "2018-04-16T17:49:10.687Z", "content": "I got accepted on ods.ai, this is fast indeed. Now I just have to learn Russian ;)
", "rawMarkdown": "I got accepted on ods.ai, this is fast indeed. Now I just have to learn Russian ;)", "votes": 5 }, { "id": 315064, "postDate": "2018-04-16T17:53:36.240Z", "content": "@raddar @CPMP please write smth to #welcome channel, you will be very pleased with reaction
", "rawMarkdown": "@raddar @CPMP please write smth to #welcome channel, you will be very pleased with reaction", "votes": 3 }, { "id": 315606, "postDate": "2018-04-17T10:41:54.423Z", "content": "@CPMP, I just filled up the form using your method :)
", "rawMarkdown": "@CPMP, I just filled up the form using your method :)", "votes": 1 } ] }, { "id": 314405, "postDate": "2018-04-15T13:58:06.710Z", "content": "@CPMP: Although I was not one of those who downvoted you, I have to say that I do not agree that this could be considered cheating, unless the intention of the person who submitted the code was to win a prize. Otherwise, it would amount to nothing more than a playground competition for those who did not submit their code. I, for one, did not submit my code, and I have officially withdrawn from this competition (as stated in the thread posting at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54161) as well as explicitly indicated my team (which consists solely of myself) as such (i.e. Withdrawn - Not for award consideration) on the leaderboard. Nonetheless, I'd still like to make amendments to my current model, and upload the latest results to the leaderboard to see how my revised model would fare in terms of its accuracy in ROI prediction. For me, the accuracy & capability of my model is of utmost importance, as I am an avid optical microscopist who utilizes a vast range of light microscopy & imaging modalities, and unlike most Kagglers here, I did not use any existing machine learning packages for this competition (such as deep neural networks, mask-rcnn, etc), but rather created my own algorithm from scratch, hence I'd be interested to evaluate the accuracy, capability and robustness of my model & algorithm in this context.
", "rawMarkdown": "@CPMP: Although I was not one of those who downvoted you, I have to say that I do not agree that this could be considered cheating, unless the intention of the person who submitted the code was to win a prize. Otherwise, it would amount to nothing more than a playground competition for those who did not submit their code. I, for one, did not submit my code, and I have officially withdrawn from this competition (as stated in the thread posting at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54161) as well as explicitly indicated my team (which consists solely of myself) as such (i.e. **Withdrawn - Not for award consideration**) on the leaderboard. Nonetheless, I'd still like to make amendments to my current model, and upload the latest results to the leaderboard to see how my revised model would fare in terms of its accuracy in ROI prediction. For me, the accuracy & capability of my model is of utmost importance, as I am an avid optical microscopist who utilizes a vast range of light microscopy & imaging modalities, and unlike most Kagglers here, I did not use any existing machine learning packages for this competition (such as deep neural networks, mask-rcnn, etc), but rather created my own algorithm from scratch, hence I'd be interested to evaluate the accuracy, capability and robustness of my model & algorithm in this context.", "votes": 2, "replies": [ { "id": 314410, "postDate": "2018-04-15T14:17:13.577Z", "content": "@Shiraz: After stage 1 though, the LB will not give you feedback on how well you have improved your code. And, since they release the stage 1 expected results, you no longer need to submit any more, you can use the test set for local validation of whether your model has improved or not. However, @CPMP's comments are not addressed to you it seems, you seem like one of the honest ones by marking your team as ineligible for awards.
", "rawMarkdown": "@Shiraz: After stage 1 though, the LB will not give you feedback on how well you have improved your code. And, since they release the stage 1 expected results, you no longer need to submit any more, you can use the test set for local validation of whether your model has improved or not. However, @CPMP's comments are not addressed to you it seems, you seem like one of the honest ones by marking your team as ineligible for awards.", "votes": 3 }, { "id": 314418, "postDate": "2018-04-15T14:31:51.697Z", "content": "@Liam: Yes, indeed, but as everyone here knows, the Stage 1 data set is extremely 'clean', with no images having the expected anomalies of a typical amateur-ish photomicrograph (such as background noise, poor S/N ratio, improper lighting, over-exposure, etc). Most, if not all, of these conditions were present in the Stage 2 dataset, which makes it a pretty good test of resilience for any model or algorithm, hence my interest in using this dataset to stretch the limits of my current model under extreme conditions. And yes, you're correct that the public LB does not give any indication at present on the accuracy of the submitted models, although I do expect it to do so at the end of the competition (for the 2 models we choose for final consideration in the private LB). And I also hope that the organizers will make the actual results of the test set for Stage 2 public at the end of the competition, so that we can compare our own results against the proposed solutions, which would be very helpful for everyone here ... ;)
", "rawMarkdown": "@Liam: Yes, indeed, but as everyone here knows, the Stage 1 data set is extremely 'clean', with no images having the expected anomalies of a typical amateur-ish photomicrograph (such as background noise, poor S/N ratio, improper lighting, over-exposure, etc). Most, if not all, of these conditions were present in the Stage 2 dataset, which makes it a pretty good test of resilience for any model or algorithm, hence my interest in using this dataset to stretch the limits of my current model under extreme conditions. And yes, you're correct that the public LB does not give any indication at present on the accuracy of the submitted models, although I do expect it to do so at the end of the competition (for the 2 models we choose for final consideration in the private LB). And I also hope that the organizers will make the actual results of the test set for Stage 2 public at the end of the competition, so that we can compare our own results against the proposed solutions, which would be very helpful for everyone here ... ;)", "votes": 2 }, { "id": 314419, "postDate": "2018-04-15T14:32:26.347Z", "content": "Shiraz, you raise a good point, you have a genuine need to keep submitting.
\n\nWhat I do for testing ideas is I submit after competition end, as I get the private LB feedback on it. There is no need, nor advantage, to submit before competition end something not intended for final ranking of the competition, because there is not LB feedback. I guess this was one of the reason organizers selected to not provide LB feedback in stage 2.
", "rawMarkdown": "Shiraz, you raise a good point, you have a genuine need to keep submitting.\n\nWhat I do for testing ideas is I submit after competition end, as I get the private LB feedback on it. There is no need, nor advantage, to submit before competition end something not intended for final ranking of the competition, because there is not LB feedback. I guess this was one of the reason organizers selected to not provide LB feedback in stage 2.", "votes": 1 }, { "id": 314429, "postDate": "2018-04-15T14:56:37.930Z", "content": "@CPMP: Thanks for your feedback & suggestion. Yes, anyway, the competition draws to a close in less than a couple of days, and I have a couple of models which I am looking forward to submitting before then, as I'm not sure whether we can still keep submitting the results of our models to the competition to get the private LB feedback after the competition closes (that would depend entirely on the organizers). But if so, I'll likely keep submitting the results obtained from future revised versions of my model/(s) to gain the feedback from the private LB, even after the competition closes.
", "rawMarkdown": "@CPMP: Thanks for your feedback & suggestion. Yes, anyway, the competition draws to a close in less than a couple of days, and I have a couple of models which I am looking forward to submitting before then, as I'm not sure whether we can still keep submitting the results of our models to the competition to get the private LB feedback after the competition closes (that would depend entirely on the organizers). But if so, I'll likely keep submitting the results obtained from future revised versions of my model/(s) to gain the feedback from the private LB, even after the competition closes.", "votes": 2 }, { "id": 314435, "postDate": "2018-04-15T15:17:39.237Z", "content": "Maybe you can ask in a new topic if late submissions will be allowed.
", "rawMarkdown": "Maybe you can ask in a new topic if late submissions will be allowed. " }, { "id": 314451, "postDate": "2018-04-15T15:47:00.547Z", "content": "@CPMP: Since I have withdrawn from the Competition, I'm not submitting my code at this time, just the results of my models to obtain the private LB feedback when the competition closes. But I've started a separate thread to see if future submissions will be possible after the competition closes on 16 April 2018, 23:59h.
", "rawMarkdown": "@CPMP: Since I have withdrawn from the Competition, I'm not submitting my code at this time, just the results of my models to obtain the private LB feedback when the competition closes. But I've started a separate thread to see if future submissions will be possible after the competition closes on 16 April 2018, 23:59h." }, { "id": 314457, "postDate": "2018-04-15T15:57:32.353Z", "content": "Sure, but whether you want it or not you will appear in the leaderboard with the submissions you make before competition end. There is no such thing like withdrawal and submitting at the same time. Now, if Kaggle team wants to take time to handle you case...
", "rawMarkdown": "Sure, but whether you want it or not you will appear in the leaderboard with the submissions you make before competition end. There is no such thing like withdrawal and submitting at the same time. Now, if Kaggle team wants to take time to handle you case..." }, { "id": 314468, "postDate": "2018-04-15T16:14:43.690Z", "content": "Yes, and that is fine with me, since my intent is to submit the results of my model to evaluate its accuracy against the ground truth data (which may be gleaned from the LB scores), and not for competing, or pitting my model against the rest of the contestants.
", "rawMarkdown": "Yes, and that is fine with me, since my intent is to submit the results of my model to evaluate its accuracy against the ground truth data (which may be gleaned from the LB scores), and not for competing, or pitting my model against the rest of the contestants." }, { "id": 314483, "postDate": "2018-04-15T16:47:31.587Z", "content": "Sure, but if you get a medal that way then it is a medal less for others. I'm not saying you should not submit, I'm just discussing the assumptions you have with saying you withdraw yet you submit.
", "rawMarkdown": "Sure, but if you get a medal that way then it is a medal less for others. I'm not saying you should not submit, I'm just discussing the assumptions you have with saying you withdraw yet you submit. " }, { "id": 314834, "postDate": "2018-04-16T10:46:59.030Z", "content": "Sure, but then again, there are participants who train their models on Stage 1 test data & the provided labels (in the Stage 1 solution set) and/or external datasets, while this would also put them at an unfair advantage to the rest of us who don't do so. And these participants continue to submit the results of their trained models to the LB for scoring and getting medals. To the rest of us who don't do this (including myself), these practices are extremely unfair as well.
", "rawMarkdown": "Sure, but then again, there are participants who train their models on Stage 1 test data & the provided labels (in the Stage 1 solution set) and/or external datasets, while this would also put them at an unfair advantage to the rest of us who don't do so. And these participants continue to submit the results of their trained models to the LB for scoring and getting medals. To the rest of us who don't do this (including myself), these practices are extremely unfair as well.", "votes": 1 } ] }, { "id": 315024, "postDate": "2018-04-16T16:39:15.833Z", "content": "Hi there - Thanks for your comments. While people can compete in Stage 2 without uploading their model in Stage 1, it says that their submissions and standings will be removed after the competition is over.
", "rawMarkdown": "Hi there - Thanks for your comments. While people can compete in Stage 2 without uploading their model in Stage 1, it says that their submissions and standings will be removed after the competition is over. ", "votes": -3, "replies": [ { "id": 315029, "postDate": "2018-04-16T16:48:04.520Z", "content": "@Maggie, this would indeed fix the issue I had with your 2 stage FAQ. Rereading it, it still contains this:
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage?
\n \nYes. However, if you do not upload a model and finish in a prize\n position, your team will be removed from the competition standings\n entirely.
\n
Removal is only if you are in prize money according to your FAQ. This is why I wrote the post in the first place.
", "rawMarkdown": "@Maggie, this would indeed fix the issue I had with your 2 stage FAQ. Rereading it, it still contains this:\n\n> Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\n> \n> Yes. However, if you do not upload a model and finish in a prize\n> position, your team will be removed from the competition standings\n> entirely.\n\nRemoval is only if you are in prize money according to your FAQ. This is why I wrote the post in the first place.", "votes": 4 }, { "id": 315180, "postDate": "2018-04-16T20:25:12.107Z", "content": "@Maggie that's the inconsistency I was talking about. What you say stands only for those finishing in a prize position. And you can't suddenly change it just because some users posted their disagreement. If Kaggle thinks it should be changed then do so, but from the next 2 stage competition onwards and not in the middle of a 3500 participants competition.\nBesides, you admins should know the rules better than us, not the other way around.
\n\n\n", "rawMarkdown": "@Maggie that's the inconsistency I was talking about. What you say stands only for those finishing in a prize position. And you can't suddenly change it just because some users posted their disagreement. If Kaggle thinks it should be changed then do so, but from the next 2 stage competition onwards and not in the middle of a 3500 participants competition.\nBesides, you admins should know the rules better than us, not the other way around.\n\n> Will I still be on the leaderboard if I don't upload a model but do\n> submit in the second stage? \n> Yes. However, if you do not upload a\n> model and finish in a prize position, your team will be removed from\n> the competition standings entirely.\n\n", "votes": 4 }, { "id": 315966, "postDate": "2018-04-17T22:28:44.233Z", "content": "Will I still be on the leaderboard if I don't upload a model but do\n submit in the second stage?
\n
\n Yes. However, if you do not upload a\n model and finish in a prize position, your team will be removed from\n the competition standings entirely.
The rules for this competition are that the model needs to be uploaded before the deadline for Round 1, we will remove all who did not upload a model before the deadline.
", "rawMarkdown": "The rules for this competition are that the model needs to be uploaded before the deadline for Round 1, we will remove all who did not upload a model before the deadline. ", "votes": -2 }, { "id": 316014, "postDate": "2018-04-18T02:27:09.300Z", "content": "Hi Maggie, I just want to make things fair to all competitors, but I read and re-read the competition rules and there is nothing there saying that teams who didn't upload the model by stage 1 deadline will be removed from LB after stage 2, where is it written?
\n\nThe only place I found something about 2-stage competitions is here: [https://www.kaggle.com/two-stage-faq][1], it says:\n\"Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely. \"
\n\nSo maybe Kaggle needs only to edit the 2-Stage Competition Rules template to avoid that issue in future competitions, but never changing the rules after the competition's end.
\n\nAlso I agree with @CPMP that allowing people to submit in stage 2 without uploading the model can be a potential way of cheating for ranking.
", "rawMarkdown": "Hi Maggie, I just want to make things fair to all competitors, but I read and re-read the competition rules and there is nothing there saying that teams who didn't upload the model by stage 1 deadline will be removed from LB after stage 2, where is it written? \n\nThe only place I found something about 2-stage competitions is here: [https://www.kaggle.com/two-stage-faq][1], it says:\n\"Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely. \"\n\nSo maybe Kaggle needs only to edit the 2-Stage Competition Rules template to avoid that issue in future competitions, but never changing the rules after the competition's end.\n\nAlso I agree with @CPMP that allowing people to submit in stage 2 without uploading the model can be a potential way of cheating for ranking.", "votes": 10 }, { "id": 316017, "postDate": "2018-04-18T02:53:23.590Z", "content": "@Maggie: As mentioned by @Giba, nowhere is it stated in Kaggle's FAQs or on the competition website that the rankings for teams who did not submit their models at the end of Stage 1 would be removed. If Kaggle really wanted to ensure fairness amongst participants, you would also need to remove participants who have used external datasets other than what was provided by the organizers for this competition as Stage 1 training data (and this includes the Stage 1 test set) to train their models.
", "rawMarkdown": "@Maggie: As mentioned by @Giba, ***nowhere*** is it stated in Kaggle's FAQs or on the competition website that the rankings for teams who did not submit their models at the end of Stage 1 would be removed. If Kaggle *really* wanted to ensure fairness amongst participants, you would also need to remove participants who have used external datasets other than what was provided by the organizers for this competition as Stage 1 ***training*** data (and this includes the Stage 1 test set) to train their models.", "votes": 2 }, { "id": 316054, "postDate": "2018-04-18T05:20:25.757Z", "content": "\n\n\nthere is nothing there saying that teams who didn't upload the model by stage 1 deadline will be removed from LB after stage 2, where is it written?
\n
In the timeline page it says models must be uploaded before stage 1 deadline:
\n\n\n\n\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.
\n
We cannot say the obligation is not written anywhere, sorry. Now if people believe they can ignore an obligation and still get points/medals then something is not working as it should.
\n\nWhat is creating confusion is the part of the FAQ that says you can still enter stage2 if you did not upload your model.
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage?
\n \nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
It is because of this contradiction that I wrote the post. People who comply with the timeline page are at a disadvantage compared to people who don't. That's what is unfair. The FAQ must be rewritten to remove the contradiction before next Stage 2 competition IMHO.
", "rawMarkdown": "> there is nothing there saying that teams who didn't upload the model by stage 1 deadline will be removed from LB after stage 2, where is it written? \n\nIn the timeline page it says models must be uploaded before stage 1 deadline:\n\n> April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.\n\nWe cannot say the obligation is not written anywhere, sorry. Now if people believe they can ignore an obligation and still get points/medals then something is not working as it should.\n\nWhat is creating confusion is the part of the FAQ that says you can still enter stage2 if you did not upload your model.\n\n> Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\n>\n> Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.\n\nIt is because of this contradiction that I wrote the post. People who comply with the timeline page are at a disadvantage compared to people who don't. That's what is unfair. The FAQ must be rewritten to remove the contradiction before next Stage 2 competition IMHO.", "votes": 3 }, { "id": 316064, "postDate": "2018-04-18T05:39:17.553Z", "content": "For this competition, given the FAQ, it may be hard to remove people, as some may genuinely have been encouraged by the FAQ to not upload their model/code.
", "rawMarkdown": "For this competition, given the FAQ, it may be hard to remove people, as some may genuinely have been encouraged by the FAQ to not upload their model/code.", "votes": 2 }, { "id": 316069, "postDate": "2018-04-18T05:44:02.833Z", "content": "\n\n\nWe cannot say it is not written anywhere, sorry.
\n
The confusion here seems to be that it was not written specifically that models had to be uploaded for participants to continue participating during Stage 2 (while the FAQ seems to indicate otherwise, i.e. that this is not a necessity for participating in Stage 2), and in this respect, I do agree that this is an anomaly that needs to be addressed by Kaggle. However, by the uploading of models, it would also be essential to emphasize that the models should be viable and the same model is being used to obtain the results for Stage 2, without any re-training on the dataset. Otherwise, by just allowing participants to upload models to meet the Stage 2 participation requirement, a loophole is provided for participants to submit 'dummy code', with some participants continuing to change their model/re-train their model (using Stage 1 test data) even after the Stage 2 test data has been released.
\n\n\n\n\nPeople who comply with the timeline page are at a disadvantage\n compared to people who don't. That's what is unfair.
\n
It is not just people who comply with the timeline page, but also contestants who have limited themselves to using only the Stage 1 training data for training. So even if someone complies with the timeline (by submitting their model before Stage 1 closes), but then proceeds to change/re-train their model (after the Stage 2 dataset is revealed), this would be considered as being unfair to other participants.
", "rawMarkdown": "> We cannot say it is not written anywhere, sorry.\n\nThe confusion here seems to be that it was not written ***specifically*** that ***models*** had to be uploaded for participants to continue participating during Stage 2 (while the FAQ seems to indicate otherwise, i.e. that this is ***not*** a necessity for *participating* in Stage 2), and in this respect, I do agree that this is an anomaly that needs to be addressed by Kaggle. However, by the uploading of models, it would also be essential to emphasize that the models should be viable and the ***same*** model is being used to obtain the results for Stage 2, without any re-training on the dataset. Otherwise, by just allowing participants to upload models to meet the Stage 2 participation requirement, a loophole is provided for participants to submit 'dummy code', with some participants continuing to change their model/re-train their model (using Stage 1 test data) even after the Stage 2 test data has been released.\n\n> People who comply with the timeline page are at a disadvantage\n> compared to people who don't. That's what is unfair.\n\nIt is not just people who comply with the timeline page, but also contestants who have limited themselves to using only the Stage 1 training data for training. So even if someone complies with the timeline (by submitting their model before Stage 1 closes), but then proceeds to change/re-train their model (after the Stage 2 dataset is revealed), this would be considered as being unfair to other participants.", "votes": 2 }, { "id": 316073, "postDate": "2018-04-18T05:52:53.347Z", "content": "\n\n\nFor this competition, given the FAQ, it may be hard to remove people,\n as some may genuinely have been encouraged by the FAQ to not upload\n their model/code.
\n
This is indeed true, and in fact it gets very messy here, since the definition of what should be allowed (and what shouldn't) becomes grayed. But I partially think that the idea proposed by Kaggle of submitting the model's SHA256 hash prior to the close of Stage 1 and subsequently submitting the actual model (having the same SHA256 hash) after Stage 2 closes (when the participant lands in a prize position to win a prize) is the best alternative in this context. This is to protect the intellectual property (IP) rights of the participants (some participants may be using self-developed models which they are intending to patent after the competition ends), thus allowing the prize-winning participants the right to decide if they do indeed wish to submit their models for evaluation and to win a prize, or if they prefer to withhold their models in the event of other constraints (e.g. filing a patent, etc), in which case, they would have forsaken the award.
", "rawMarkdown": "> For this competition, given the FAQ, it may be hard to remove people,\n> as some may genuinely have been encouraged by the FAQ to not upload\n> their model/code.\n\nThis is indeed true, and in fact it gets very messy here, since the definition of what should be allowed (and what shouldn't) becomes grayed. But I partially think that the idea proposed by Kaggle of submitting the model's SHA256 hash prior to the close of Stage 1 and subsequently submitting the actual model (having the same SHA256 hash) after Stage 2 closes (when the participant lands in a prize position to win a prize) is the best alternative in this context. This is to protect the intellectual property (IP) rights of the participants (some participants may be using self-developed models which they are intending to patent after the competition ends), thus allowing the *prize-winning* participants the right to decide if they do indeed wish to submit their models for evaluation and to win a prize, or if they prefer to withhold their models in the event of other constraints (e.g. filing a patent, etc), in which case, they would have forsaken the award.", "votes": 1 }, { "id": 316081, "postDate": "2018-04-18T06:41:04.710Z", "content": "@Maggie like others said, you should have changed the https://www.kaggle.com/two-stage-faqin time. Now you can only remove models that finish in a prize position. That's the way all 2-stage competitions were run up until now. It's greatly inconsistent to change it after the competition's end. Besides if that was written in time, then anyone could have just submitted a dummy model or a public kernel code model, because I'm certain you are not going to run all submitted code and verify the submissions for all participants, that's why the rule is made that way.\nAnd if it's somebody's opinion that should be seriously accounted for, with all respect to other participants, that's Giba's, top Kaggler for long and respected by everyone in the community. Even the topic's starter agrees with this.
\n\nRemoving people just for not uploading any dummy model, when the rules stated that it is ok if you didn't compete for the prizes, by changing the rules retrospectively, that's what is unfair.
", "rawMarkdown": "@Maggie like others said, you should have changed the [https://www.kaggle.com/two-stage-faq][1]in time. Now you can only remove models that finish in a prize position. That's the way all 2-stage competitions were run up until now. It's greatly inconsistent to change it after the competition's end. Besides if that was written in time, then anyone could have just submitted a dummy model or a public kernel code model, because I'm certain you are not going to run all submitted code and verify the submissions for all participants, that's why the rule is made that way.\nAnd if it's somebody's opinion that should be seriously accounted for, with all respect to other participants, that's Giba's, top Kaggler for long and respected by everyone in the community. Even the topic's starter agrees with this.\n\nRemoving people just for not uploading any dummy model, when the rules stated that it is ok if you didn't compete for the prizes, by changing the rules retrospectively, that's what is unfair.\n\n\n [1]: https://www.kaggle.com/two-stage-faq", "votes": 4 }, { "id": 316293, "postDate": "2018-04-18T16:38:43.030Z", "content": "I think, when the rule of the specific competition (rules specified in the timeline) conflicts with the rule of the general Kaggle (2 stage faq) competitions, competition rule prevails. So, though I agree that there were confusions, it is within their rights to enforce the competition rule.
", "rawMarkdown": "I think, when the rule of the specific competition (rules specified in the timeline) conflicts with the rule of the general Kaggle (2 stage faq) competitions, competition rule prevails. So, though I agree that there were confusions, it is within their rights to enforce the competition rule." }, { "id": 316337, "postDate": "2018-04-18T19:13:19.720Z", "content": "@bsp2020 the code upload timeline rule is specified in every other 2-stage competition held so far, however in accordance with the faq page removal can only stand for prize winning positions. And the reason is simple. Given that Kaggle can't reproduce all uploaded models to verify the submissions, anyone could just upload a dummy code. If they wanted to enforce code restriction then they should have chosen Kernel only competition type and take care for the hardware requirements such a decision would incur. Since they chose 2-stage type and didn't change the faq clarification in time, it is unfair to remove participants.
", "rawMarkdown": "@bsp2020 the code upload timeline rule is specified in every other 2-stage competition held so far, however in accordance with the faq page removal can only stand for prize winning positions. And the reason is simple. Given that Kaggle can't reproduce all uploaded models to verify the submissions, anyone could just upload a dummy code. If they wanted to enforce code restriction then they should have chosen Kernel only competition type and take care for the hardware requirements such a decision would incur. Since they chose 2-stage type and didn't change the faq clarification in time, it is unfair to remove participants. ", "votes": 3 }, { "id": 316388, "postDate": "2018-04-18T22:17:20.917Z", "content": "@gavrand Not that I want to continue this unproductive discussion, but I'll respond since you specifically replied to me.
\n\nI think your argument is that, since they did not enforce the rule in the past, they can't do it for this competition. No?
\n\nI think the rule is pretty clear. The competition specific rule prevails. Sure, they have not been enforced in the past competition. Not sure why that matters. When I got a speeding ticket, I argued that the officer did not enforce the rule properly because he did not catch all the speeders and he should not give me a ticket either. You should have seen his face. I'll never argue in such a way next time I'm stopped by a police. :)
", "rawMarkdown": "@gavrand Not that I want to continue this unproductive discussion, but I'll respond since you specifically replied to me.\n\nI think your argument is that, since they did not enforce the rule in the past, they can't do it for this competition. No?\n\nI think the rule is pretty clear. The competition specific rule prevails. Sure, they have not been enforced in the past competition. Not sure why that matters. When I got a speeding ticket, I argued that the officer did not enforce the rule properly because he did not catch all the speeders and he should not give me a ticket either. You should have seen his face. I'll never argue in such a way next time I'm stopped by a police. :)\n\n\n" }, { "id": 316439, "postDate": "2018-04-19T03:19:18.603Z", "content": "\n\n\nI think the rule is pretty clear. The competition specific rule\n prevails.
\n
Nowhere was it explicitly mentioned in the competition rules (up to the competition closure on 16 April 2018, 23:59H) that individuals who did not submit their models in Stage 1 would be barred from participating in Stage 2 (& I have screenshots to prove this, so no further amendments after the said date would be acceptable). Yes, you may argue that there was a timeline for model submission for Stage 1, but that is not equivalent to enforcing that individuals have to submit their models in Stage 1 to participate in Stage 2. In contrast, the Kaggle FAQ (accessible at https://www.kaggle.com/two-stage-faq) specifically addresses this matter (and which I quote for your easy reference below):
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do\n submit in the second stage? Yes. However, if you do not upload a model\n and finish in a prize position, your team will be removed from the\n competition standings entirely.
\n
You have to be very clear and explicit in this context, else there would be multiple interpretations, which can't be generalized in this (or any other) context for that matter.\nOn a different note, it comes across as very intriguing to me as to why some participants in this competition are earnestly arguing on the unfairness of the lack of model submission but continued participation by some contestants, even highlighting the need for the FAQ to be re-written in this respect, but gloss over the fact that changes to the submitted model (e.g. by re-training on Stage 1 test data & labels) are acceptable, stating that this was allowed by the Kaggle admin/organizers. Shouldn't fairness be enforced equally in all respects, and not be skewed to suit one's actions, else this would undeniably be considered as a form of corruption? Also, there seems to be some people who are downvoting this post, suggesting that these individuals are indeed actively promoting corruption.
", "rawMarkdown": "> I think the rule is pretty clear. The competition specific rule\n> prevails.\n\nNowhere was it ***explicitly*** mentioned in the competition rules (up to the competition closure on 16 April 2018, 23:59H) that individuals who did **not** submit their models in Stage 1 would be barred from participating in Stage 2 (& I have screenshots to prove this, so no further amendments after the said date would be acceptable). Yes, you may argue that there was a timeline for model submission for Stage 1, but that is *not* equivalent to enforcing that individuals have to submit their models in Stage 1 to participate in Stage 2. In contrast, the Kaggle FAQ (accessible at https://www.kaggle.com/two-stage-faq) ***specifically*** addresses this matter (and which I quote for your easy reference below):\n\n> Will I still be on the leaderboard if I don't upload a model but do\n> submit in the second stage? Yes. However, if you do not upload a model\n> and finish in a prize position, your team will be removed from the\n> competition standings entirely.\n\nYou have to be very clear and explicit in this context, else there would be multiple interpretations, which can't be generalized in this (or any other) context for that matter.\nOn a different note, it comes across as very intriguing to me as to why some participants in this competition are earnestly arguing on the unfairness of the lack of model submission but continued participation by some contestants, even highlighting the need for the FAQ to be re-written in this respect, but gloss over the fact that changes to the submitted model (e.g. by re-training on Stage 1 test data & labels) are acceptable, stating that this was allowed by the Kaggle admin/organizers. Shouldn't fairness be enforced equally in all respects, and not be skewed to suit one's actions, else this would undeniably be considered as a form of corruption? Also, there seems to be some people who are downvoting this post, suggesting that these individuals are indeed actively promoting corruption.", "votes": 3 }, { "id": 316454, "postDate": "2018-04-19T04:49:17.063Z", "content": "@bsp2020 let me explain my point of view because there seems to be a misunderstanding. I have nothing personal, I respect every opinion, I just express my disagreement.
\n\nMy argument is that the rule consists of two parts:\nFirst part is the code upload timeline, that stands for all participants. \nSecond part is that in accordance with the FAQ if you don't upload code then you can still be on the leaderboard, except for the case that you finish in a prize winning position and your submission will be removed.
\n\nThe reason I mention the previous 2-stage competitions is because they are testament that the rule was enforced this way so far. Now it appears as if this rule is being enforced only partially, without previous notice and retrospectively. That's unfair. If the change was published in time then all participants would be discouraged to not upload their model and would almost certainly upload their code, because getting results in such a competition requires effort, time and money, and wasting all that for such a simple thing is irrational. I can see that you are relatively new in Kaggle (3 competitions). I have done much more and I just testify my experience about previous 2-stage competitions.
\n\nAnd since you mention speeding tickets, it is as if the signs restricts you at 65mph, and the police officer fines you for traveling at 50mph because, retrospectively and with no further notice, it was decided that the speed limit should be changed to 45mph.
", "rawMarkdown": "@bsp2020 let me explain my point of view because there seems to be a misunderstanding. I have nothing personal, I respect every opinion, I just express my disagreement.\n\nMy argument is that the rule consists of two parts:\nFirst part is the code upload timeline, that stands for all participants. \nSecond part is that in accordance with the FAQ if you don't upload code then you can still be on the leaderboard, except for the case that you finish in a prize winning position and your submission will be removed. \n\nThe reason I mention the previous 2-stage competitions is because they are testament that the rule was enforced this way so far. Now it appears as if this rule is being enforced only partially, without previous notice and retrospectively. That's unfair. If the change was published in time then all participants would be discouraged to not upload their model and would almost certainly upload their code, because getting results in such a competition requires effort, time and money, and wasting all that for such a simple thing is irrational. I can see that you are relatively new in Kaggle (3 competitions). I have done much more and I just testify my experience about previous 2-stage competitions. \n\nAnd since you mention speeding tickets, it is as if the signs restricts you at 65mph, and the police officer fines you for traveling at 50mph because, retrospectively and with no further notice, it was decided that the speed limit should be changed to 45mph.", "votes": 2 }, { "id": 316499, "postDate": "2018-04-19T08:01:55.553Z", "content": "@Shiraz S K
\n\n\n\n\nOn a different note, it comes across as very intriguing to me as to why some participants in this competition are earnestly arguing on the unfairness of the lack of model submission but continued participation by some contestants, even highlighting the need for the FAQ to be re-written in this respect, but gloss over the fact that changes to the submitted model (e.g. by re-training on Stage 1 test data & labels) are acceptable, stating that this was allowed by the Kaggle admin/organizers.
\n
Since you mentioned this point already a couple of times, I think there is a misunderstanding: a submitted model to Kaggle does not mean the weights of a neural network or other classifier but rather the complete algorithm and processing pipeline which can be run without human intervention (up to correcting errors) to produce the stage 2 submissions.
\n\nIn particular, this algorithm may contain a training phase (which can be run on stage1 train+test data) and a subsequent prediction phase. The key point is that no manual fiddling with any parameters can take place.
", "rawMarkdown": "@Shiraz S K\n\n> On a different note, it comes across as very intriguing to me as to why some participants in this competition are earnestly arguing on the unfairness of the lack of model submission but continued participation by some contestants, even highlighting the need for the FAQ to be re-written in this respect, but gloss over the fact that changes to the submitted model (e.g. by re-training on Stage 1 test data & labels) are acceptable, stating that this was allowed by the Kaggle admin/organizers.\n\nSince you mentioned this point already a couple of times, I think there is a misunderstanding: a submitted model to Kaggle **does not** mean the weights of a neural network or other classifier but rather the complete algorithm and processing pipeline which can be run without human intervention (up to correcting errors) to produce the stage 2 submissions.\n\nIn particular, this algorithm may contain a training phase (which can be run on stage1 train+test data) and a subsequent prediction phase. The key point is that no manual fiddling with any parameters can take place.\n" }, { "id": 316506, "postDate": "2018-04-19T08:24:59.047Z", "content": "@rmldj: That is precisely the point - allowing participants to re-train their model on Stage 1 test data and submit the new model's predictions on Stage 2 test data can also be considered as condoning cheating, since it creates a completely different model from what was originally used for Stage 1 test label predictions. This would defeat the purpose of code freezing in the first instance, since the model architecture can now be varied (& quite significantly too) by subjecting it to a different training set each time. In some instances, code can also be included in the Stage 1 script to allow the model to dynamically change the hyperparameters without manual intervention, for e.g. if the user defines the number of training epochs, width or depth of the neural network (which represent hyperparameters that should not be varied) to be proportional to the size of the training dataset, a larger training dataset would in turn change these values, which would (in turn) result in a totally different model architecture. This would be highly unfair to other individuals who do not resort to such practices, but use their models constructed from Stage 1 for Stage 2 test label prediction.
", "rawMarkdown": "@rmldj: That is precisely the point - allowing participants to re-train their model on Stage 1 test data and submit the new model's predictions on Stage 2 test data can also be considered as condoning cheating, since it creates a completely different model from what was originally used for Stage 1 test label predictions. This would defeat the purpose of code freezing in the first instance, since the model architecture can now be varied (& quite significantly too) by subjecting it to a different training set each time. In some instances, code can also be included in the Stage 1 script to allow the model to dynamically change the hyperparameters without manual intervention, for e.g. if the user defines the number of training epochs, width or depth of the neural network (which represent hyperparameters that should not be varied) to be proportional to the size of the training dataset, a larger training dataset would in turn change these values, which would (in turn) result in a totally different model architecture. This would be highly unfair to other individuals who do not resort to such practices, but use their models constructed from Stage 1 for Stage 2 test label prediction." } ] }, { "id": 314627, "postDate": "2018-04-16T00:56:53.533Z", "content": "I fully agree. It is simply unfair: different rules are being enforced for different people (this is the exact definition of unfair). While I am not a talented data scientist, I have never participated in a two-stage competition because of these rules.
", "rawMarkdown": "I fully agree. It is simply unfair: different rules are being enforced for different people (this is the exact definition of unfair). While I am not a talented data scientist, I have never participated in a two-stage competition because of these rules.", "votes": 1, "replies": [ { "id": 314794, "postDate": "2018-04-16T09:37:06.183Z", "content": "Different rules are enforced for different competitions as well - not every competition gives you a monetary prize & not every competition gives the same number of participants monetary awards (e.g. for this competition, you get a monetary award if you make it to the top 5 on the LB, but some competitions only award the top 3 participants). Would you consider this to be unfair as well?
", "rawMarkdown": "Different rules are enforced for different competitions as well - not every competition gives you a monetary prize & not every competition gives the *same* number of participants monetary awards (e.g. for this competition, you get a monetary award if you make it to the top 5 on the LB, but some competitions only award the top 3 participants). Would you consider this to be unfair as well?", "votes": -1 }, { "id": 315209, "postDate": "2018-04-16T22:02:47.833Z", "content": "I would not consider that unfair. To clarify, I meant unfair as in not equal for everyone. If competition rules hold everyone to the same regulations, it seems fair.
", "rawMarkdown": "I would not consider that unfair. To clarify, I meant unfair as in not equal for everyone. If competition rules hold everyone to the same regulations, it seems fair." }, { "id": 316455, "postDate": "2018-04-19T04:52:40.623Z", "content": "Yes, indeed, but this debate seems to picking on one such rule (namely the continued participation by some individuals despite not having previously submitted a model) & claiming that this is unfair, when in fact, it has been common practice for this to occur in multiple 2-stage competitions in the past, without such claims being made. This is just because of some people's grievances in this respect, and although justified, these same people overlook other occurrences (e.g. the re-training of already-submitted models), while these might also be considered to be unfair to other participants.
", "rawMarkdown": "Yes, indeed, but this debate seems to picking on one such rule (namely the continued participation by some individuals despite not having previously submitted a model) & claiming that this is unfair, when in fact, it has been common practice for this to occur in multiple 2-stage competitions in the past, without such claims being made. This is just because of some people's grievances in this respect, and although justified, these same people overlook other occurrences (e.g. the re-training of already-submitted models), while these might also be considered to be unfair to other participants." } ] }, { "id": 314552, "postDate": "2018-04-15T19:46:52.640Z", "content": "I completely agree with this. I have actually modified my code in a way that I think could significantly increase my score. Under the rules, it seems I could choose to submit a .csv based on this new code, as I am unlikely to be in top 5, just to increase my chance of a medal.
\n\nHowever, I see this as cheating, so I will use an upload based on my frozen model for my final submission.
\n\nI will upload a .csv with my new model just to see how well it would have done, but I will not do this until after the deadline, when it will be counted as late submission, and can simply be used to evaluate my model improvements.
\n\nI honestly think you should be disqualified from submitting in stage 2 if you have not uploaded a frozen model in stage 1. And although you couldn't manually assess if people have been honest, to make it clear that you will be removed from the leaderboard if you are discovered would discourage at least some people from cheating.
\n\nIt's annoying there isn't some sort of checksum that could be performed to ensure that each .csv has been generated by a specific uploaded model code. Maybe some genius can come up with something.
", "rawMarkdown": "I completely agree with this. I have actually modified my code in a way that I think could significantly increase my score. Under the rules, it seems I could choose to submit a .csv based on this new code, as I am unlikely to be in top 5, just to increase my chance of a medal. \n\nHowever, I see this as cheating, so I will use an upload based on my frozen model for my final submission.\n\nI will upload a .csv with my new model just to see how well it would have done, but I will not do this until after the deadline, when it will be counted as late submission, and can simply be used to evaluate my model improvements.\n\nI honestly think you should be disqualified from submitting in stage 2 if you have not uploaded a frozen model in stage 1. And although you couldn't manually assess if people have been honest, to make it clear that you **will** be removed from the leaderboard if you are discovered would discourage at least some people from cheating.\n\nIt's annoying there isn't some sort of checksum that could be performed to ensure that each .csv has been generated by a specific uploaded model code. Maybe some genius can come up with something.", "replies": [ { "id": 314564, "postDate": "2018-04-15T20:20:41.783Z", "content": "Your last paragraph reminds of Zero knowledge protocol. If you are interested in this topic, it is a fascinating concept.
", "rawMarkdown": "Your last paragraph reminds of [**Zero knowledge protocol**](https://en.wikipedia.org/wiki/Zero-knowledge_proof). If you are interested in this topic, it is a fascinating concept. ", "votes": 2 }, { "id": 314588, "postDate": "2018-04-15T21:54:33.630Z", "content": "Interesting stuff. Thanks for the info.
", "rawMarkdown": "Interesting stuff. Thanks for the info.", "votes": 1 }, { "id": 314591, "postDate": "2018-04-15T22:13:08.767Z", "content": "Glad that you liked it. It was sort of a hot topic when I was in grad school. I didn't work in that area but some of my colleagues did. The wikipedia article section 5.2 is more relevant to your earlier discussion about using ZNP to enforce ethical behavior.
", "rawMarkdown": "Glad that you liked it. It was sort of a hot topic when I was in grad school. I didn't work in that area but some of my colleagues did. The wikipedia article section 5.2 is more relevant to your earlier discussion about using ZNP to enforce [**ethical behavior**](https://en.wikipedia.org/wiki/Zero-knowledge_proof#Ethical_behavior). ", "votes": 1 }, { "id": 314881, "postDate": "2018-04-16T12:31:23.110Z", "content": "I wasn't involved in setting the overall design of the competition, being new to Kaggle myself, but it does indeed seem like there is no good way to guard against cheating other than manual checking.
\n\n@DStjhb: to me, it sounds sensible to not allow stage 2 submissions if you haven't uploaded code. On the other hand, it's easy for anyone to upload some nonsense code to meet that criterion. Maybe because it's such an easy workaround Kaggle has chosen not to enforce at that stage.
", "rawMarkdown": "I wasn't involved in setting the overall design of the competition, being new to Kaggle myself, but it does indeed seem like there is no good way to guard against cheating other than manual checking. \n\n@DStjhb: to me, it sounds sensible to not allow stage 2 submissions if you haven't uploaded code. On the other hand, it's easy for anyone to upload some nonsense code to meet that criterion. Maybe because it's such an easy workaround Kaggle has chosen not to enforce at that stage.", "votes": 2 }, { "id": 314888, "postDate": "2018-04-16T12:41:51.680Z", "content": "@Anne: I agree it is really difficult to guard against in practise. Especially in these big competitions with thousands of entries. I guess you could have some sort of Kaggle virtual environment that you run your code inside, which could log any changes; but that would limit what languages you can use etc.
\n\nThis is the first competition I have tried, so I may simply be projecting my fears about cheating, rather than speaking with any real experience of seeing it happen. It would be good for Kaggle to do some research into the prevalence of cheating on their platform. Maybe you could do an anonymous survey (making it explicit there will be no repercussions), and ask people some questions, whether they modified code etc., so we have an idea whether our fears are grounded in reality, or if we are all just making a big fuss about a negligible percentage of users!
", "rawMarkdown": "@Anne: I agree it is really difficult to guard against in practise. Especially in these big competitions with thousands of entries. I guess you could have some sort of Kaggle virtual environment that you run your code inside, which could log any changes; but that would limit what languages you can use etc.\n\nThis is the first competition I have tried, so I may simply be projecting my fears about cheating, rather than speaking with any real experience of seeing it happen. It would be good for Kaggle to do some research into the prevalence of cheating on their platform. Maybe you could do an anonymous survey (making it explicit there will be no repercussions), and ask people some questions, whether they modified code etc., so we have an idea whether our fears are grounded in reality, or if we are all just making a big fuss about a negligible percentage of users!", "votes": 1 }, { "id": 314921, "postDate": "2018-04-16T13:59:25.473Z", "content": "@Anne re:
\n\n\n\n\n@DStjhb: to me, it sounds sensible to not allow stage 2 submissions if\n you haven't uploaded code. On the other hand, it's easy for anyone to\n upload some nonsense code to meet that criterion. Maybe because it's\n such an easy workaround Kaggle has chosen not to enforce at that\n stage.
\n
Yes, it might be true that nonsense code uploaded into the system will easily meet the criterion. Then there's also the possibility of contestants using external datasets and test data from Stage 1 which would also be unfair to other participants and thus be considered as \"cheating\". So I still think the use of Kernels will be an apt way to counter this issue (as highlighted in my previous post) since it ensures that the code uploaded is the one used to create the model, and that the dataset used is the one supplied by the organizers for the competition, and not some other dataset. The only issue with Kernels would be the lack of libraries supported by the current Kernel environment, unless Kaggle decides to do something about this in future.
\n\n@DStjhb re:
\n\n\n\n\n@Anne: I agree it is really difficult to guard against in practise.\n Especially in these big competitions with thousands of entries. I\n guess you could have some sort of Kaggle virtual environment that you\n run your code inside, which could log any changes; but that would\n limit what languages you can use etc.
\n
Yes, that's why I suggested that organizers on Kaggle should only utilize Kernels for future 2-stage competitions.
", "rawMarkdown": "@Anne re:\n\n> @DStjhb: to me, it sounds sensible to not allow stage 2 submissions if\n> you haven't uploaded code. On the other hand, it's easy for anyone to\n> upload some nonsense code to meet that criterion. Maybe because it's\n> such an easy workaround Kaggle has chosen not to enforce at that\n> stage.\n\nYes, it might be true that nonsense code uploaded into the system will easily meet the criterion. Then there's also the possibility of contestants using external datasets and test data from Stage 1 which would also be unfair to other participants and thus be considered as \"cheating\". So I still think the use of Kernels will be an apt way to counter this issue (as highlighted in my previous post) since it ensures that the code uploaded is the one used to create the model, and that the dataset used is the one supplied by the organizers for the competition, and not some other dataset. The only issue with Kernels would be the lack of libraries supported by the current Kernel environment, unless Kaggle decides to do something about this in future.\n\n@DStjhb re:\n\n> @Anne: I agree it is really difficult to guard against in practise.\n> Especially in these big competitions with thousands of entries. I\n> guess you could have some sort of Kaggle virtual environment that you\n> run your code inside, which could log any changes; but that would\n> limit what languages you can use etc.\n\nYes, that's why I suggested that organizers on Kaggle should only utilize Kernels for future 2-stage competitions." } ] }, { "id": 314421, "postDate": "2018-04-15T14:36:09.620Z", "content": "Edit: moved to main post.
", "rawMarkdown": "Edit: moved to main post." }, { "id": 314487, "postDate": "2018-04-15T17:06:52.543Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 314489, "postDate": "2018-04-15T17:12:41.737Z", "content": "I think it is fair to say that some of the greatest advances in expert domains were done by experts in other domains. If you think you have an interesting idea you may want to go for it!
", "rawMarkdown": "I think it is fair to say that some of the greatest advances in expert domains were done by experts in other domains. If you think you have an interesting idea you may want to go for it!", "votes": 1 }, { "id": 314494, "postDate": "2018-04-15T17:26:34.743Z", "content": "@the1owl, I am not sure I understand what you write. Is it related to my post at all?
\n\nWhere I relate to it is that I am a beginner in image processing, and do not expect much from this competition. I came to learn, and I did learn a lot, from Heng Cher Keng, and from many others.
", "rawMarkdown": "@the1owl, I am not sure I understand what you write. Is it related to my post at all?\n\nWhere I relate to it is that I am a beginner in image processing, and do not expect much from this competition. I came to learn, and I did learn a lot, from Heng Cher Keng, and from many others." }, { "id": 314500, "postDate": "2018-04-15T17:34:48.573Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 314507, "postDate": "2018-04-15T17:46:05.140Z", "content": "@the1owl Agreed. I could only wrap my head around something simpler as UNet. I got a public score jump from 0.35 to 0.40 right on the deadline but it is dead on arrival for the second stage. It is also way short of what @CPMP and his team got with UNet too. I'd definitely would like to learn about it more.
", "rawMarkdown": "@the1owl Agreed. I could only wrap my head around something simpler as UNet. I got a public score jump from 0.35 to 0.40 right on the deadline but it is dead on arrival for the second stage. It is also way short of what @CPMP and his team got with UNet too. I'd definitely would like to learn about it more." }, { "id": 314518, "postDate": "2018-04-15T18:12:09.893Z", "content": "@the1owl I intend to share my code after competition end.
", "rawMarkdown": "@the1owl I intend to share my code after competition end.", "votes": 1 } ] } ], "comments": [ { "id": 314945, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-16T14:36:05.770000", "content": "here are my views:
\n\nthere is no way to prevent \"cheating\" , even for 1 stage competition. e.g. the user can hand label test data in one-stage competition. if you don't win a prize, kaggle don't check your code
why do you want to \"cheat\"? for people who want to get good ranking for resume, it could be done like this:
\n\nsubmit legally for the competition, it is ok that you get poor rank
after competition, you still can do a post submission. at this point of time many top solutions would be made known. you can re-implement to get good results.
at the interview, you can say although my ranking was not good, but i am fast to learn. and my post-results ranked 10, up from 100 in the period of one week after studying the top solutions.
\n\nMaybe kaggle can made a third ranking, called post-competition ranking. Then no one would need to cheat becuase anyone can submit after the competition for good ranking
", "votes": 13, "replies": [ { "id": 314955, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T14:50:41.330000", "content": "@Heng re:
\n\n\n\n\nwhy do you want to \"cheat\"? for people who want to get good ranking\n for resume, it could be done like this: submit legally for the\n competition, it is ok that you get poor rank after competition, you\n still can do a post submission. at this point of time many top\n solutions would be made known. you can re-implement to get good\n results. at the interview, you can say although my ranking was not\n good, but i am fast to learn. and my post results ranked 10, up from\n 100 in the period of one week after studying the top solution.
\n
That's true, if you're interested to participate for the sake of participating and to fuel your interest in learning something from the competition. But some people don't want that; they participate to get a good ranking on the LB which they perceive would grant them medals, recognition and automatic bragging rights.
", "votes": 1, "replies": [] }, { "id": 316343, "author_name": "villammartin", "author_url": "", "post_date": "2018-04-18T19:26:41.203000", "content": "I totally agree with your point of view!! :)
", "votes": 0, "replies": [] } ] }, { "id": 314810, "author_name": "OsciiArt", "author_url": "", "post_date": "2018-04-16T10:00:31.690000", "content": "@CPMP To be honest, I was very worried. I finished stage 1 in 9th place. This is my first chance to get a gold medal. In stage 2, my prediction is very disastrous and I found it may be repaired by a simple modification. I thought about cheating to get a gold medal seriously. Thank you. Your words made me stop short of cheating.
", "votes": 11, "replies": [ { "id": 314811, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T10:08:45.580000", "content": "I wish you will fare well still. And thank you for sharing what you shared during stage1. I wish I had more time to implement some of what you shared. If you don't get a gold here I'm sure you'll get one soon in another competition given how well you did in stage1.
\n\nFor me too my stage2 is messed and I can fix it with a simple change. But I'll eat my own food, keep the code as is, and praise the gods of overfitting!
", "votes": 1, "replies": [] }, { "id": 314828, "author_name": "OsciiArt", "author_url": "", "post_date": "2018-04-16T10:36:27.190000", "content": "Thank you. You inspire me so much too.
", "votes": 1, "replies": [] }, { "id": 314830, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T10:41:23.430000", "content": "@OsciiArt & @CPMP: Kaggle has already clearly addressed this issue in their FAQ at https://www.kaggle.com/two-stage-faq. If you think that you could get a better score by making a modification in your model & you intend to make the modification, just be considerate enough to indicate that you have forfeited/withdrawn from the competition so that your model's actual standing will not have an impact on the rest of the contestants. Of course, if this means that you have gained a prize position by doing so, Kaggle will revoke your prize, which is a fair deal. On the other hand, if you are really interested in continuing to participate in the competition, then you should adhere to the competition rules, & not make such changes. It is of no use pretending to adhere strictly to the competition rules by highlighting this in a discussion thread (such as in this Discussion board), when (in reality) you resort to underhanded means by training your model on external datasets, and/or re-training on Stage 1 test data (and labels) to get a better score on the private LB, and subsequently pretending that you have gotten this far by using your existing models (supposedly solely trained on Stage 1 training data) on the Stage 2 test set.
", "votes": 2, "replies": [] }, { "id": 314867, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T11:53:04.803000", "content": "@Shiraz, thank you I didn't thought of reading the FAQ I commented in my post. Glad you're here to remind me.
", "votes": 0, "replies": [] }, { "id": 314887, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-16T12:41:08.943000", "content": "Is changing a magic number in your code considered as a scientifically relevant change ? :)
\n\nWhat happens if your code, for whatever reasons, crashes on some specific data ?
\n\nI mean, can the organizers expect us to upload a stable code ?
", "votes": 0, "replies": [] }, { "id": 314909, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2018-04-16T13:42:55.970000", "content": "@OsciiArt respect for admitting it openly. You should still have a good chance to have a nice result at stage 2, not only your result is bad, other may be even worse.
", "votes": 3, "replies": [] }, { "id": 314912, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T13:47:18.027000", "content": "\n\n\nIs changing a magic number in your code considered as a scientifically relevant change ? :)
\n
What is a magic number?
", "votes": 0, "replies": [] }, { "id": 314932, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-16T14:25:05.323000", "content": "\"A magic number is a bare naked number used in source code. It's magic because no-one has a clue what it means including the author inside 3 months.\"
", "votes": 0, "replies": [] }, { "id": 314937, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T14:30:52.483000", "content": "Given what you did in Stage1 I doubt there are magic numbers in your code. ;)
", "votes": 0, "replies": [] }, { "id": 314951, "author_name": "OsciiArt", "author_url": "", "post_date": "2018-04-16T14:47:13.177000", "content": "@Dmytro Poplavskiy Thank you. I hope so!
", "votes": 0, "replies": [] }, { "id": 314962, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T15:08:51.167000", "content": "@CPMP re:
\n\n\n\n\n@Shiraz, thank you I didn't thought of reading the FAQ I commented in\n my post. Glad you're here to remind me.
\n
You're welcome.
", "votes": 0, "replies": [] }, { "id": 314989, "author_name": "jeandebleau", "author_url": "", "post_date": "2018-04-16T15:55:36.667000", "content": "I would not complain if somebody says, because of the huge amount of noise in the test data we decided to change the sigma of our gaussian filter. I mean this is a human decision based on a simple observation of the data. Nothing extraordinary. Unfortunately I don't have any preprocessing and my results are miserable... :(
", "votes": 0, "replies": [] }, { "id": 314997, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T16:07:26.990000", "content": "> I mean this is a human decision based on a simple observation of the data. Nothing extraordinary.
\n\nPoint is that if the data you mention is stage2 test data then you are modifying your model to better perform on it, which is precisely what is not in the spirit of this competition.
\n\n\n\n\nUnfortunately I don't have any preprocessing and my results are miserable... :(
\n
I hope your results are not that bad, you deserve to do well given what you did in stage1.
\n\nI have yet to see someone happy about stage 2 results ;)
", "votes": 1, "replies": [] } ] }, { "id": 314430, "author_name": "Robert", "author_url": "", "post_date": "2018-04-15T15:01:13.230000", "content": "I agree with CPMP, the rules in the two-stage-faq allow someone to circumvent the main reason for having a two stage competition in the first place. I had wanted to start a thread like this myself but then I thought that there really doesn't seem to be a good way to eliminate all \"cheating\" so I decided against it.
\n\nThe main reason for having a two stage competition is to ensure that contestants cannot see the final test set until they have finalized their code. So they cannot tailor their code to this final test set. However the two-stage-faq does not enforce this. Allowing contestants to make a submission and get a ranking without having uploaded a model means that they are not bound by the model upload rule. We then have a situation where people who have uploaded a model are only allowed to make submissions based on their uploaded model. This model has not been tailored to fit the final test data. On the other hand those people who have not uploaded a model in stage 1 do not have make sure that their submission comes from any specific model. They are free to look at the final test data, change their model, and then make their submission.
\n\nNow one might argue that not having uploaded a model means that one cannot win any of the prizes. But the prizes are not the only thing that matter to contestants. There are plenty of people who participate in these competitions because it affects their careers or their education. The kaggle ranking matters and has real world consequences, even if you don't win a prize.
\n\nI'm not sure that there is any sure-fire way of eliminating all \"cheating\" but it can be reduced if people are only allowed to make a submission when they have uploaded a model during stage 1.
\n\nAnother approach might be to allow everyone to make submissions without any model constraints. This would ensure that we have a level playing field when it comes to rankings, basically the same conditions as those for a regular single stage competition. But when it comes to the prizes, only submissions that can be verified with uploaded models would be considered, and submissions without models in prize-winning positions would be removed.
", "votes": 7, "replies": [ { "id": 314437, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-15T15:22:09.327000", "content": "\n\n\nAnother approach might be to allow everyone to make submissions\n without any model constraints. This would ensure that we have a level\n playing field when it comes to rankings, basically the same conditions\n as those for a regular single stage competition. But when it comes to\n the prizes, only submissions that can be verified with uploaded models\n would be considered, and submissions without models in prize-winning\n positions would be removed.
\n
Yes, I think this would be a better option for this (and future 2-stage) competition/(s) as well. Model accuracy is evolutionary, and if this is the desired outcome (as is the case for this competition where the aim is to obtain robust and accurate models to predict the ROI of a cell's nucleus), then the restriction on compulsory model submission to be considered for Stage 2 participation should be lifted.
", "votes": 1, "replies": [] } ] }, { "id": 314965, "author_name": "raddar", "author_url": "", "post_date": "2018-04-16T15:11:08.960000", "content": "I just love this topic. It seems that all of us forgot or some are not even aware what Paulo Pinto did in Fisheries competition https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/discussion/31607
\n\nAll the points made about 2stage not being \"fair\" or is a place to cheat here are valid, but imho, there is no better solution how to make it \"fair\"
", "votes": 5, "replies": [ { "id": 314972, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T15:31:09.647000", "content": "2 of the top 5 teams were removed, right?
", "votes": 0, "replies": [] }, { "id": 314975, "author_name": "raddar", "author_url": "", "post_date": "2018-04-16T15:34:50.783000", "content": "Yes
", "votes": 2, "replies": [] } ] }, { "id": 314371, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T11:40:00.480000", "content": "The fact that I got down votes is a confirmation that some participants want to change their code for stage2.
", "votes": 5, "replies": [ { "id": 315103, "author_name": "Kishore Badyakar", "author_url": "", "post_date": "2018-04-16T18:44:11.793000", "content": "Hey CPMP, I think for some people 'cheating' would come across as too strong a word to digest, so never mind the down votes! I personally agree to your point of view, because this is one of the valid interpretations of the FAQ that are possible. For me, the real problem is this fact itself that it is open to interpretations and is not technically complete. In fact, I've not seen any kind of a service provider depending on an 'FAQ' for rules enforcement, this looks very unprofessional I am afraid. There should be dedicated page/s legally explaining all the rules for participation in single or two stage competitions and these pages should be linked to the rules in the competitions' sections where we actually agree to the terms.
\n\nRegarding the point you've brought up though, I think there should be a simple solution.
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage?\n Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
As far as I've understood, the above quote is the part where you think Kaggle is allowing the upload of models by participant users in the stage 2 even when no models have been uploaded by them in stage 1, and in the process, Kaggle is giving them an unfair advantage in the context of rankings right ? I don't know much about the rankings yet, but if it's some kind of privilege one would get, then surely it'd be unfair to the user who has submitted a model in stage 1 and is bound by rules not to modify code. Whereas, the other users who have not submitted in stage 1 are free to do anything, even use stage 2 data anyway they want to get ranked. This is what I understand the problem is from your point of view.
\n\nNow, lets try to understand another interpretation. The clarity with which Kaggle says 'Yes' in the above quote makes me think that Kaggle surely is intentionally allowing this with a definite purpose. They are removing the stage 2 only participant from the competition standings entirely too. So, taking everything in good faith (why would such a reputed organization abet cheating?), I'd like to think that there are good reasons to allow it. May be because they want to let users get the experience of a live competition and gauge themselves even when they've missed the opportunity otherwise ? This is good too as it encourages learning and it goes with the overall aims of the platform.
\n\nHere we've two point of views, and both are true at the same time I think. The real problem is, there is scope for cheating for the users who WANT to cheat! Neither Kaggle nor any genuine participant user would want to abet cheating I am sure. So, lets come to the real part of the problem which I think is allowing a place in the LB to users who are taking part only in stage 2. I am not sure, but I guess it helps stage 2 only participants to gauge their performance by looking at their position there ? This must be important too for a genuine stage 2 only participant, coming up on the LB that is.
\n\nSo, ultimately I am getting the following two key points:
\n\nI might be missing something, but as far as I understand the problem and given the above two key points are true, I'd suggest a simple solution for it as follows:
\n\nThis will keep everyone happy and things clean. From a software point of view, it is very much implementable as everything is already in place. Adding a special view should not be too much of a problem. As also, if something has always been like a particular way then that does not mean we can't look for betterment.
\n\nHopefully, at least something will be done about all this. I would love Kaggle to interact with us too and share their perspective. After all, the responsibility lies on each one of us to be genuine and put in efforts to make this platform better.
", "votes": 0, "replies": [] }, { "id": 315104, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T18:48:07.490000", "content": "\n\n\nFor me, the real problem is this fact itself that it is open to interpretations and is not technically complete.
\n
I agree, and I would add it is self contradicting. Saying in the same page that you have to upload your model by some deadline, and that you can still compete after deadline if you don't is a contradiction.
\n\nI also agree with what you write in general.
", "votes": 1, "replies": [] } ] }, { "id": 314970, "author_name": "kickback", "author_url": "", "post_date": "2018-04-16T15:26:09.690000", "content": "I am using this competition to learn about python and deep learning in general (I have learned a lot!). I know I have no chance of winning a prize based on my stage 1 score (top 30%) but I had to make some changes to get a valid .csv file for stage 2 submission (due to atypical data compared to stage 1). I can easily see there are many more things for me to learn.
\n\nI don't think continuous learning should be considered cheating, it should actually be encouraged!
\n\ntx
\n\nKickback
", "votes": 4, "replies": [ { "id": 314977, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T15:35:53.270000", "content": "I agree with you on this.
", "votes": 0, "replies": [] }, { "id": 315025, "author_name": "rmldj", "author_url": "", "post_date": "2018-04-16T16:40:59.760000", "content": "@kickback Changes necessary just to make a valid submission csv are certainly ok..\nHere the issue is modifying preprocessing parameters during stage 2 etc.
", "votes": 1, "replies": [] } ] }, { "id": 314384, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-15T13:07:09.290000", "content": "@CPMP No offense but I think that it's always been like that in every other 2 stage competition hosted. If they want to enforce code lock then they choose to host kernel only competition. Besides, top 10-20 teams are going to abide by the code rule enforcement, it is too big a risk to lose the prize. As far as the others are concerned, it is pretty time consuming for Kaggle to manually crosscheck codes and submissions, given the different parameters and enviroment variety, since I repeat it is not a kernel only competition where all people use the same platform. It is difficult already to do so for the top winning ones. It is this variety they want, given that the problem difficulty is such that hosting a kernel only competition would limit the solution range. So please take it cool, and note that the term \"cheating\" you chose is exaggerated and offending to other Kagglers who know it was always like that in 2 stage competitions. That's the reason you are being downvoted.
", "votes": 4, "replies": [ { "id": 314386, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T13:17:42.700000", "content": "No offense taken ;)
\n\nI am not asking for checking manually that all teams did not change code. I'm asking that rules are enforced fairly. The quote I provide above says people have to upload code. People who do not comply should not be allowed to keep going.
\n\nI've lost a gold medal in a recent competition because someone I teamed with did not comply with some rule. Why would people not complying with rules be allowed to compete here?
\n\nAnyway, I don't want to argue with you or other competitors, I asked a question to Kaggle team. Let's see what the response will be, if any.
\n\nLet me add that I'm so cool now that it astonishes me ;) I'm just waiting for my retraining with stage1 test labels to complete and I'll be done with this competition. As I wrote elsewhere, except for the fuzziness about stage2 rules, it was a great learning experience.
", "votes": 4, "replies": [] }, { "id": 314388, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-15T13:22:03.957000", "content": "Fair enough,however if they did that they would be inconsistent compared to previously held 2 stage competitions . Anyway, good luck to you and all other Kagglers!
", "votes": 2, "replies": [] }, { "id": 314394, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-15T13:34:38.057000", "content": "@gavrand, I think @CPMP suggested Kaggle changing the 2 stage rules without making extra effort to check if people are being honest at the 2nd stage. I know how much work for Kaggle admins to crosscheck the code already. As you suggested, the rules have always been like this, but it doesn't mean the rules can not be improved for the future competitions to save work on both sides:
\n\nMy 2 cents:
\n\nstage1_solution.csv because we expected the solution would have been released in the same format as stage1_train. And other participants hit problems like reading some images which were in jpeg/tf format and they had to add code to handle this. This could've been prevented. @Nicole Finnie What's the point? I mean if you don't crosscheck then anyone could have just submitted a dummy code?
", "votes": 0, "replies": [] }, { "id": 314397, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-15T13:50:31.660000", "content": "@gavrand, you made a valid point, no system will be bullet proof. It won't stop all of them, at least it will discourage some of them. I believe people who pursue DS career will use their Kaggle achievements for their resumes in the future. That's why an open discussion is always good to make the ranking system of this platform more reliable and trustworthy. I think Kaggle will appreciate different opinions too.
", "votes": 3, "replies": [] }, { "id": 314603, "author_name": "Rand Xie", "author_url": "", "post_date": "2018-04-15T23:21:04.603000", "content": "@Nicole, I can not agree more. I believe the NIPS 2017 adversarial attack competitions' strategies are more appropriate for 2-stage competitions. Template code are provided to standardize input format and enable error checking (to make sure you code can run successfully in the final stage).
\n\nI believe the major concern here is allowing participants to modify (non scientific change) and retrain on their own, which opens the door of cheating. If templates are provided, Kaggle could run the retraining/inference script on their GPU clusters.
", "votes": 4, "replies": [] }, { "id": 314752, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T07:55:42.647000", "content": "@Rand Xie, interesting. Let' s hope Kaggle team will use that approach for 2 stages competitions from now on: code template, and no retraining in stage2.
", "votes": 0, "replies": [] }, { "id": 314823, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T10:22:48.997000", "content": "Yes, I'm in for the idea on kernels being used to freeze code following completion of Stage 1 for future 2-stage competitions as well. However, the only issue I have with the kernels is that some of the commonly-used libraries are not available for download and testing in Kaggle's kernel environment (I use R by the way). And yes, I'm also aware that a number of contestants have resorted to training their models with external datasets downloaded from elsewhere (discussed in https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572), and even the Stage 1 test set (when the labels have been released), while this can also be seen as a form of \"cheating\", since by training your models with Stage 1 test data or external datasets, the training algorithm will compute the weights of all of the respective nodes differently as compared to the situation in which you were training it on the provided Stage 1 training data alone, resulting in a very different model from what you would have obtained if you had used Stage 1 training data alone (for the training & construction of your models).
", "votes": 0, "replies": [] } ] }, { "id": 486136, "author_name": "", "author_url": "", "post_date": "2019-03-08T10:56:39.927000", "content": "spam :(
", "votes": -1, "replies": [] }, { "id": 314676, "author_name": "adityabhat", "author_url": "", "post_date": "2018-04-16T04:54:18.327000", "content": "True
", "votes": 1, "replies": [] }, { "id": 316023, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T03:04:50.770000", "content": "At the beginning of stage2, kaggle released the solution to stage1 which is an important supplement to the training data. I wonder if those got score more than 0.55 did not use those solutions to retrain their model, then how can they finalize their model before the solutions were released? \nAnd also, the test data in stage2 contained lots of unseen types, which means it is very hard to get a better score than stage1 without argument the training data to suit this situation. If not, how can the top 100 got 0.1 increase on an unfit data? Which means most of the players did not submit their prediction from the model in stage1 but did some further work on it. \nSo, the model uploaded in stage1 is just a 'sign in', no guarantee at all. If we only focus on the 'code', then we miss the effect of training data, one can just manipulate on training data without touch the code to re-train their model. \nTo fix this issue once for all, kaggle should require to submit the model file like .h5, then use the model to predict on the hidden data by themselves.
", "votes": 2, "replies": [ { "id": 316059, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T05:34:23.273000", "content": "I hope for the teams in prize money position that they did comply with the rules of the competition. If not then they'll be removed. In a previous 2 stages competition 2 of the top 5 teams were removed.
", "votes": 0, "replies": [] }, { "id": 316077, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T06:24:41.390000", "content": "Then kaggle should emphasis the 'upload model', both in rule and the web page, code a specific button for it, for newbies like me who never noticed that thing. \nIt feels very unfair to be kickoff by such an incomplete even somehow useless law without even a button on the main page, after all we spent days on this competition.\nFinally, the law can not deal with the reproduce issue, if they can, there is no need to hold a stage2, just upload the model and kaggle should do the final prediction.
", "votes": 0, "replies": [] }, { "id": 316085, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T07:04:56.673000", "content": "To your other point, about good scores in stage2, our own score went from0.504 to 0.545 with the exact same code. It seems stage2 images were a bit easier to predict on average.
", "votes": 0, "replies": [] }, { "id": 316093, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-18T07:27:04.077000", "content": "@CPMP: I came to the same conclusion that stage-2 was easier than stage-1. Using the exact same model in stage-1, I got 0.400 in stage 1, and 0.439 in stage-2. Adding the solutions images in stage-1 and retraining without touching parameters produced a private score of 0.470 for stage-2. But the number of test images in stage-1 was much smaller than stage-2 so I'm not sure how much statistical variance we have as a consequence.
", "votes": 0, "replies": [] }, { "id": 316097, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T07:32:22.723000", "content": "That's the point, how can you prove you are using the same code and same training data to get a better score in a different distribution? \nThe only way is that using the same model, which means same architecture and same weights. To to this, it is not complete to just upload the code but also the model itsself, because even random seed can cause difference in the result.
", "votes": 0, "replies": [] }, { "id": 316098, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T07:35:15.013000", "content": "I can prove it because I uploaded my code before deadline. If you run it then you can reproduce my results. You are right about model weights, but my team mate Nicole asked the question, and we were told that we don't need to upload model weights. Moreover, organizers also wrote that it was fine to retrain with stage1 test labels added to the training data we used in stage 1. Maybe you missed that.
\n\nNot sure what you are after actually. My post was about a contradiction within the stage 2 FAQ that could encourage people to ignore the stage1 upload deadline. I don't know if you agree this is an issue or not. For the rest, please argue with organizers, not me. I'm not the one setting the rules here.
", "votes": 0, "replies": [] }, { "id": 316099, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T07:35:42.613000", "content": "@Oscar Then you are using a different training data which is certain that you can not get it before you upload your model.\nAgain, training data is the part which the law does not cover.
", "votes": 0, "replies": [] }, { "id": 316100, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-04-18T07:41:12.720000", "content": "@Yunfei, for this competition, training the stage 1 test data is allowed, the Kaggle admin gave us the green light in this post. It was vaguly addressed in the 2 stage FAQ, so we explicitly asked again.\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/54299
\n\nAnd in this post, Wendy explicitly said you have to upload your code, not the model files, and you can retrain your model.\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/52385#298629
\n\nAnd you're right, since the rules are not crystal clear, there's a grey area for further discussions and improvement of rules.
", "votes": 3, "replies": [] }, { "id": 316102, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-18T07:44:21.217000", "content": "Hi @Yunfei. I uploaded before the deadline instructions and code (SHA256 checksum) on how to create the model. I believe the rules permitted using the disclosed additional masks for re-training on the second stage as long as the code and parameters were left untouched.
", "votes": 1, "replies": [] }, { "id": 316106, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T07:48:45.727000", "content": "@CPMP \nThe law does not cover the training data which is vital to the result. And this makes the law incomplete. Anyone who really think about it can make this clear. As your title says, this law still \"encourage\" cheating. So I do not agree with it. It is just a 'sign in'.
", "votes": 0, "replies": [] }, { "id": 316110, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T07:57:48.213000", "content": "Thank you for the information, I should visit forum more.
", "votes": 0, "replies": [] }, { "id": 316112, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-18T08:00:02.843000", "content": "@Yunfei Duan: Precisely, this is the exact reason why this competition promotes double standards, which would be the exact definition of unfairness and inequality. How can the organizers encourage participants to use the Stage 1 test data for re-training, but regard those who did not upload their model before the Stage 1 deadline as cheating/not complying with the rules? By allowing the participants to re-train their models on the Stage 1 test set & its labels (when the solution to the Stage 1 test set was released), the organizers are in fact encouraging cheating, since re-training the model will definitely change the weights of the individual nodes in the model (if a type of neural network is being used, which was the most common type of model assimilated by most participants in this competition). This would indeed be equivalent to submitting dummy code in the first instance (just to meet the timeline of model submission), and then making the required changes (as & when the participants deem fit) so as to attain a better private LB score. So this competition is indeed rigged, as I suspected all along ...
", "votes": -1, "replies": [] }, { "id": 316120, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T08:10:16.737000", "content": "\"NOTE: you can always upload just one file to Kaggle, so if you decide to go with the hash upload just the hash, otherwise upload a zip file (which could include code + hash of the weights file for example)
\n\nRegarding the 2 submissions - you can have multiple model / code variants (but all need to be published BEFORE the deadline), so you can produce any number of submissions for Stage 2; in the end you will have to choose only two of the submission (which in your opinion have the highest chance of winning)\"
\n\nThere are a lot of gray areas can be settled. \nKaggle should develop these rule into its own logics and web interfaces with obvious notices. The reason why Kaggle do not do the final prediction may be that they can not afford so much computational cost.
", "votes": 0, "replies": [] }, { "id": 316217, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2018-04-18T13:00:09.427000", "content": "Interesting, I missed this part, I always assumed you have to decide all the parameters for two submissions when uploading the model. Allowing to upload multiple code variants opens the door for more misuse, for example it's possible to upload huge number of possible submissions covering a grid of hyperparameters values.
\n\nIt's certainly worth to clarify and tune the rules. Would be much more straightforward if all the weights to be finalized before the second stage release and only prediction run on the second stage data, with only two predefined submissions.
", "votes": 7, "replies": [] }, { "id": 316231, "author_name": "bsp2020", "author_url": "", "post_date": "2018-04-18T13:45:21.970000", "content": "Not only do you have to comply with the law, you should comply with the spirit of the law.
\n\nNot sure why you guys and gals are so hung up with this. The spirit of the rule is that you should not change your code or hyperparameter, unless your code tunes it automatically. You are also allowed to change code to handle new data.
\n\nSince they are not showing the score on second state leader board, I think it will be very tough to cheat anyway. I was socked to find out Heng's team scored so low in the public leaderboard. His model scored very high in stage 1 but it seems to have overfit to stage 1 test data. He also said that his submission was not the best scoring model. If an expert like him could not figure out the best scoring model, what change do you think us, mere mortals, have to train better scoring model without public leaderboard guiding us? I'm all for clear rules and communications. (I was shocked, see 0 score when I first submitted my state 2 answer) But I think they have done a good job, making his 2 stage competition as fair as they can and I'm sure the next 2 state kaggle holds will be better.
\n\nJust my 2 cents. :)
", "votes": 0, "replies": [] }, { "id": 316234, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-18T14:00:34.890000", "content": "\n\n\nThe spirit of the rule is that you should not change your code or\n hyperparameter, unless your code tunes it automatically.
\n
The code doesn't tune anything automatically, unless you direct it to (possibly by injecting new training data, or varying certain values/hyperparameters such as the number of epochs, etc).
\n\n\n\n\nYou are also allowed to change code to handle new data.
\n
This is only allowed for non-scientific changes (e.g. file path) and not scientific ones, after you have submitted your model, since it promotes cheating.
\n\n\n\n\nSince they are not showing the score on second state leader board, I\n think it will be very tough to cheat anyway.
\n
A participant doesn't need to know his/her score on the LB to cheat. Rather, the participant can easily do a visual comparison between the predicted masks and the actual photomicrograph, to detect if there are any areas which might need to be included/removed from the masks, & subsequently include some code to do this post-processing of the predicted masks. This would already constitute a form of cheating, since the post-processing code was not in the original submitted model to begin with.
", "votes": 0, "replies": [] }, { "id": 316249, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T14:32:56.463000", "content": "@Shiraz As you said, there are many ways to break the spirit but stay in the law. The final solution may be that kaggle release an API which calls prediction on the submitted model, all submission follow that framework.
", "votes": 0, "replies": [] }, { "id": 316290, "author_name": "bsp2020", "author_url": "", "post_date": "2018-04-18T16:28:13.937000", "content": "\n\n\nA participant doesn't need to know his/her score on the LB to cheat. Rather, the participant can easily do a visual comparison ...
\n
I'm sure people who spent the time to visually inspect the output could have improved LB. Hopefully, it was the worth the time for them. If you are in the money and did something wrong, the organizer will look at their code/methodology and disqualify them. If you are not in the money and am getting away with it, shame on you.
\n\nHowever, I don't think it would affect me personally. I suspect that most people who are upset about this, are upset because they think those who did not comply with the rule may have surpassed them in LB? Due to the stochastic nature of machine learning training, I'm pretty sure more people lost their places in LB than gained if they tuned their model without the guidance of 2nd stage public LB score.
\n\nTo be honest, after I realized that they intentionally showed 0 for all 2nd stage submissions, I felt relieved and stop working on my model. I did not even retrain my model using the 1st stage test data. I just used the best weighs from the first stage training. That model weight got me 85th in 1st stage LB but my final standing is 44th.
\n\nI'm interested in machine learning/AI/computer vision. I'd rather spend my time learning the algorithm and training techniques and not worry about minute details of the competition. Though I was a bit shocked and distressed when I saw 0 score for second stage submission (my understanding is that this is the first time they did it), I think it worked out very well to keep the competition more honest and interesting. I'm sure the organizers/kaggle are learning about how to run 2 stage competition better and the next one will be even more awesome than this one.
", "votes": 3, "replies": [] }, { "id": 316378, "author_name": "YUNFEI DUAN", "author_url": "", "post_date": "2018-04-18T21:23:45.060000", "content": "We are talking about the detail and lack of the rules, not the big spirit. Big spirit is always right, like peace, honest. Nobody is against the spirit of kaggle, but there is no use to stick on the big topic while the rule is vulnerable. It is just different topic.
", "votes": 0, "replies": [] }, { "id": 316431, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-19T03:00:04.273000", "content": "\n\n\nI'm sure people who spent the time to visually inspect the output\n could have improved LB. Hopefully, it was the worth the time for them.\n If you are in the money and did something wrong, the organizer will\n look at their code/methodology and disqualify them. If you are not in\n the money and am getting away with it, shame on you.
\n
That is essentially what this whole discussion is about - on what should be permitted and what shouldn't. Are people who inspect their code visually (after testing their model on the Stage 2 test data) and realizing that they can improve their LB ranking by making some changes to their existing code/model (e.g. incorporating new filters, etc) allowed to do so, or would this be violating the competition rules?
\n\n\n\n\nHowever, I don't think it would affect me personally. I suspect that\n most people who are upset about this, are upset because they think\n those who did not comply with the rule may have surpassed them in LB?
\n
Possibly so, that is after all the spirit of any competition, i.e. to have a friendly rivalry amongst the participants, thus it would be expected of most participants to aim for medals and prizes. However, there will undeniably be some individuals who couldn't care less about the LB rankings, since they participate to gain some knowledge/experience in this respect. Personally, I'd say that I am of the second group in this context, since I did not even use any ML algorithm/packages for this competition but wanted to evaluate the accuracy and capability of my self-devised algorithm. I expected the results to be poor for this current context, since I did not even have the time to run my model on >530 images in the Stage 2 test set, so I left those as blank entries instead, which caused my solution to land in 817th place ... ;)
\n\n\n\n\nDue to the stochastic nature of machine learning training, I'm pretty\n sure more people lost their places in LB than gained if they tuned\n their model without the guidance of 2nd stage public LB score.
\n
That's not necessarily so, since it's not exactly stochastic as some individuals would realize the importance of filters for the 2nd test set, when this might not be so apparent for Stage 1 predictions, leading some to make revisions to their existing model to evaluate the Stage 2 test data (these revisions may include code for these filters in their models, making changes to the Stage 2 test images to accomodate their current model setup, etc). This is where the issue of cheating comes in - these participants who have made such revisions are rightfully gaining an unfair advantage over other contestants.
\n\n\n\n\nI'm interested in machine learning/AI/computer vision. I'd rather\n spend my time learning the algorithm and training techniques and not\n worry about minute details of the competition.
\n
That is a pretty good perspective to take on this issue, and to be frank, I share your sentiments in this context as well. However, details like these are essential and should be clearly specified for the conduct of subsequent 2-stage competitions in Kaggle to ensure fairness amongst all participants & prevent further abuses of the system and/or debates of this nature arising in this context.
", "votes": 0, "replies": [] } ] }, { "id": 314985, "author_name": "hengck23", "author_url": "", "post_date": "2018-04-16T15:48:58.913000", "content": "This is a machine learning problem on how you ranked the following words in order of importance:
\n\n\"integrity\", \"fame\", \"correct\", \"wrong\", \"rule\", \"results\", \"value\", ...
", "votes": 2, "replies": [ { "id": 314990, "author_name": "raddar", "author_url": "", "post_date": "2018-04-16T15:56:25.837000", "content": "from my personal experience, rules are not #1 in kaggle. I had the case when I asked if I had to follow the rules, and was told that for that specific case it was not necessary by a kaggle team member :\\ but I did follow them in the end...
", "votes": 0, "replies": [] }, { "id": 314993, "author_name": "raddar", "author_url": "", "post_date": "2018-04-16T15:57:51.003000", "content": "and the example of ods.ai internally sharing data in many segmentation competitions. this has been a thing nobody even cares to question these days.
", "votes": 2, "replies": [] }, { "id": 314999, "author_name": "", "author_url": "", "post_date": "2018-04-16T16:12:02.093000", "content": "@raddar do you have evidence of sharing the data on ods?
", "votes": 16, "replies": [] }, { "id": 315017, "author_name": "Cut Onion", "author_url": "", "post_date": "2018-04-16T16:32:36.167000", "content": "I first internally shared my data as a child and it traumatized me so much.
", "votes": 7, "replies": [] }, { "id": 315023, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2018-04-16T16:38:42.123000", "content": "Hi @raddar, ods.ai community is extremely careful to follow the rules of no private code or data sharing outside already formed (on kaggle) teams. For camera competition specifically please see this post by kaggle admins who double-checked that: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49268#279846
", "votes": 8, "replies": [] }, { "id": 315034, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T16:57:18.203000", "content": "@Konstantin, interesting discussion indeed. It says it is easy to join ods.ai, but I don't see how to join, neither when I go to https://opendatascience.slack.com/ nor to ods.ai site and download the slides. Can you help?
", "votes": 1, "replies": [] }, { "id": 315036, "author_name": "raddar", "author_url": "", "post_date": "2018-04-16T16:58:11.913000", "content": "Thanks Konstantin. I probably feel that I have to apologize. It's kind of frustrating at the same time, because I was referring to the same competition. It is not so obvious that flickr was that good source to extend the given data. And it seems only ods.ai teams used it (my impression after the solutions posted in forums).
\n\nI don't have anything ods.ai, in fact I wanted to join it but couldn't for some reason. That is probably why I had been suspicious about it since then. Until now I really thought that it is a closed circle russian community.
\n\nNow as to a rainfall of downvotes - im pretty sure all of them come from ods.ai. I understand why you are not too happy about my opinion, but maybe given these facts you would not look at me as a \"traumatized child\".
", "votes": 13, "replies": [] }, { "id": 315037, "author_name": "Cut Onion", "author_url": "", "post_date": "2018-04-16T16:59:36.370000", "content": "There is button \"Присоединиться\" (Join) on ods.ai, it will forward you to google form to request invite. Just fill it and you'll get there.
", "votes": 7, "replies": [] }, { "id": 315038, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2018-04-16T17:00:12.113000", "content": "@raddar, ods.ai is an Open slack communtiy with 12500 users. Feel free to join :) \nThere are Kaggle Admins there as well (@inversion).
\n\nAs the community is biased towards people interested in Data Science there are a lot of users with experience in CV/DL/ML and competitions on Kaggle, Topcoder, DrivenData.
\n\nIt is insulting to presume that no one from 12500 data scientists across the world can take top places in CV competitions...
\n\nPeople usually use opensourced solutions, for example from 1st place Carvana https://github.com/asanakoy/kaggle_carvana_segmentation \nEveryone who read Kaggle blog about Carvana got that link... Everyone who has github can use it... Using public github repositories is not a violation of the rules, right?
\n\nWinning teams usually come up with new approaches that, of course, are not shared. I'm pretty sure you know it is not enough to just take some solution from github and win.
", "votes": 3, "replies": [] }, { "id": 315039, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T17:00:45.240000", "content": "Thanks, didn't dare click on a button I did not understand!
", "votes": 2, "replies": [] }, { "id": 315041, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T17:05:36.143000", "content": "I filled the form using google translate ;)
", "votes": 2, "replies": [] }, { "id": 315043, "author_name": "Cut Onion", "author_url": "", "post_date": "2018-04-16T17:08:44.073000", "content": "@raddar, I gave you an upvote (but I still perfectly understand my friends from ods giving you downvotes, mostly it is a group dynamics, you know). \nAs for camera competition, idea of using flickr data was on a surface and after someone posted huge archive of photos from flickr it was a matter of individual laziness (or maybe skills) to get more of the same kind. \nAlso, it wasn't the only factor to win, my team head this extremely experienced Grandmaster and it was his skill and magic that allowed us to prepare (and then select!) a winning submission with score much much lower on Public LB.
", "votes": 1, "replies": [] }, { "id": 315061, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T17:49:10.687000", "content": "I got accepted on ods.ai, this is fast indeed. Now I just have to learn Russian ;)
", "votes": 5, "replies": [] }, { "id": 315064, "author_name": "", "author_url": "", "post_date": "2018-04-16T17:53:36.240000", "content": "@raddar @CPMP please write smth to #welcome channel, you will be very pleased with reaction
", "votes": 3, "replies": [] }, { "id": 315606, "author_name": "lalthan", "author_url": "", "post_date": "2018-04-17T10:41:54.423000", "content": "@CPMP, I just filled up the form using your method :)
", "votes": 1, "replies": [] } ] }, { "id": 314405, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-15T13:58:06.710000", "content": "@CPMP: Although I was not one of those who downvoted you, I have to say that I do not agree that this could be considered cheating, unless the intention of the person who submitted the code was to win a prize. Otherwise, it would amount to nothing more than a playground competition for those who did not submit their code. I, for one, did not submit my code, and I have officially withdrawn from this competition (as stated in the thread posting at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54161) as well as explicitly indicated my team (which consists solely of myself) as such (i.e. Withdrawn - Not for award consideration) on the leaderboard. Nonetheless, I'd still like to make amendments to my current model, and upload the latest results to the leaderboard to see how my revised model would fare in terms of its accuracy in ROI prediction. For me, the accuracy & capability of my model is of utmost importance, as I am an avid optical microscopist who utilizes a vast range of light microscopy & imaging modalities, and unlike most Kagglers here, I did not use any existing machine learning packages for this competition (such as deep neural networks, mask-rcnn, etc), but rather created my own algorithm from scratch, hence I'd be interested to evaluate the accuracy, capability and robustness of my model & algorithm in this context.
", "votes": 2, "replies": [ { "id": 314410, "author_name": "Liam Finnie", "author_url": "", "post_date": "2018-04-15T14:17:13.577000", "content": "@Shiraz: After stage 1 though, the LB will not give you feedback on how well you have improved your code. And, since they release the stage 1 expected results, you no longer need to submit any more, you can use the test set for local validation of whether your model has improved or not. However, @CPMP's comments are not addressed to you it seems, you seem like one of the honest ones by marking your team as ineligible for awards.
", "votes": 3, "replies": [] }, { "id": 314418, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-15T14:31:51.697000", "content": "@Liam: Yes, indeed, but as everyone here knows, the Stage 1 data set is extremely 'clean', with no images having the expected anomalies of a typical amateur-ish photomicrograph (such as background noise, poor S/N ratio, improper lighting, over-exposure, etc). Most, if not all, of these conditions were present in the Stage 2 dataset, which makes it a pretty good test of resilience for any model or algorithm, hence my interest in using this dataset to stretch the limits of my current model under extreme conditions. And yes, you're correct that the public LB does not give any indication at present on the accuracy of the submitted models, although I do expect it to do so at the end of the competition (for the 2 models we choose for final consideration in the private LB). And I also hope that the organizers will make the actual results of the test set for Stage 2 public at the end of the competition, so that we can compare our own results against the proposed solutions, which would be very helpful for everyone here ... ;)
", "votes": 2, "replies": [] }, { "id": 314419, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T14:32:26.347000", "content": "Shiraz, you raise a good point, you have a genuine need to keep submitting.
\n\nWhat I do for testing ideas is I submit after competition end, as I get the private LB feedback on it. There is no need, nor advantage, to submit before competition end something not intended for final ranking of the competition, because there is not LB feedback. I guess this was one of the reason organizers selected to not provide LB feedback in stage 2.
", "votes": 1, "replies": [] }, { "id": 314429, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-15T14:56:37.930000", "content": "@CPMP: Thanks for your feedback & suggestion. Yes, anyway, the competition draws to a close in less than a couple of days, and I have a couple of models which I am looking forward to submitting before then, as I'm not sure whether we can still keep submitting the results of our models to the competition to get the private LB feedback after the competition closes (that would depend entirely on the organizers). But if so, I'll likely keep submitting the results obtained from future revised versions of my model/(s) to gain the feedback from the private LB, even after the competition closes.
", "votes": 2, "replies": [] }, { "id": 314435, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T15:17:39.237000", "content": "Maybe you can ask in a new topic if late submissions will be allowed.
", "votes": 0, "replies": [] }, { "id": 314451, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-15T15:47:00.547000", "content": "@CPMP: Since I have withdrawn from the Competition, I'm not submitting my code at this time, just the results of my models to obtain the private LB feedback when the competition closes. But I've started a separate thread to see if future submissions will be possible after the competition closes on 16 April 2018, 23:59h.
", "votes": 0, "replies": [] }, { "id": 314457, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T15:57:32.353000", "content": "Sure, but whether you want it or not you will appear in the leaderboard with the submissions you make before competition end. There is no such thing like withdrawal and submitting at the same time. Now, if Kaggle team wants to take time to handle you case...
", "votes": 0, "replies": [] }, { "id": 314468, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-15T16:14:43.690000", "content": "Yes, and that is fine with me, since my intent is to submit the results of my model to evaluate its accuracy against the ground truth data (which may be gleaned from the LB scores), and not for competing, or pitting my model against the rest of the contestants.
", "votes": 0, "replies": [] }, { "id": 314483, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T16:47:31.587000", "content": "Sure, but if you get a medal that way then it is a medal less for others. I'm not saying you should not submit, I'm just discussing the assumptions you have with saying you withdraw yet you submit.
", "votes": 0, "replies": [] }, { "id": 314834, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T10:46:59.030000", "content": "Sure, but then again, there are participants who train their models on Stage 1 test data & the provided labels (in the Stage 1 solution set) and/or external datasets, while this would also put them at an unfair advantage to the rest of us who don't do so. And these participants continue to submit the results of their trained models to the LB for scoring and getting medals. To the rest of us who don't do this (including myself), these practices are extremely unfair as well.
", "votes": 1, "replies": [] } ] }, { "id": 315024, "author_name": "Maggie", "author_url": "", "post_date": "2018-04-16T16:39:15.833000", "content": "Hi there - Thanks for your comments. While people can compete in Stage 2 without uploading their model in Stage 1, it says that their submissions and standings will be removed after the competition is over.
", "votes": -3, "replies": [ { "id": 315029, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-16T16:48:04.520000", "content": "@Maggie, this would indeed fix the issue I had with your 2 stage FAQ. Rereading it, it still contains this:
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage?
\n \nYes. However, if you do not upload a model and finish in a prize\n position, your team will be removed from the competition standings\n entirely.
\n
Removal is only if you are in prize money according to your FAQ. This is why I wrote the post in the first place.
", "votes": 4, "replies": [] }, { "id": 315180, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-16T20:25:12.107000", "content": "@Maggie that's the inconsistency I was talking about. What you say stands only for those finishing in a prize position. And you can't suddenly change it just because some users posted their disagreement. If Kaggle thinks it should be changed then do so, but from the next 2 stage competition onwards and not in the middle of a 3500 participants competition.\nBesides, you admins should know the rules better than us, not the other way around.
\n\n\n", "votes": 4, "replies": [] }, { "id": 315966, "author_name": "Maggie", "author_url": "", "post_date": "2018-04-17T22:28:44.233000", "content": "Will I still be on the leaderboard if I don't upload a model but do\n submit in the second stage?
\n
\n Yes. However, if you do not upload a\n model and finish in a prize position, your team will be removed from\n the competition standings entirely.
The rules for this competition are that the model needs to be uploaded before the deadline for Round 1, we will remove all who did not upload a model before the deadline.
", "votes": -2, "replies": [] }, { "id": 316014, "author_name": "Giba", "author_url": "", "post_date": "2018-04-18T02:27:09.300000", "content": "Hi Maggie, I just want to make things fair to all competitors, but I read and re-read the competition rules and there is nothing there saying that teams who didn't upload the model by stage 1 deadline will be removed from LB after stage 2, where is it written?
\n\nThe only place I found something about 2-stage competitions is here: [https://www.kaggle.com/two-stage-faq][1], it says:\n\"Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely. \"
\n\nSo maybe Kaggle needs only to edit the 2-Stage Competition Rules template to avoid that issue in future competitions, but never changing the rules after the competition's end.
\n\nAlso I agree with @CPMP that allowing people to submit in stage 2 without uploading the model can be a potential way of cheating for ranking.
", "votes": 10, "replies": [] }, { "id": 316017, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-18T02:53:23.590000", "content": "@Maggie: As mentioned by @Giba, nowhere is it stated in Kaggle's FAQs or on the competition website that the rankings for teams who did not submit their models at the end of Stage 1 would be removed. If Kaggle really wanted to ensure fairness amongst participants, you would also need to remove participants who have used external datasets other than what was provided by the organizers for this competition as Stage 1 training data (and this includes the Stage 1 test set) to train their models.
", "votes": 2, "replies": [] }, { "id": 316054, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T05:20:25.757000", "content": "\n\n\nthere is nothing there saying that teams who didn't upload the model by stage 1 deadline will be removed from LB after stage 2, where is it written?
\n
In the timeline page it says models must be uploaded before stage 1 deadline:
\n\n\n\n\nApril 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.
\n
We cannot say the obligation is not written anywhere, sorry. Now if people believe they can ignore an obligation and still get points/medals then something is not working as it should.
\n\nWhat is creating confusion is the part of the FAQ that says you can still enter stage2 if you did not upload your model.
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do submit in the second stage?
\n \nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
It is because of this contradiction that I wrote the post. People who comply with the timeline page are at a disadvantage compared to people who don't. That's what is unfair. The FAQ must be rewritten to remove the contradiction before next Stage 2 competition IMHO.
", "votes": 3, "replies": [] }, { "id": 316064, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-18T05:39:17.553000", "content": "For this competition, given the FAQ, it may be hard to remove people, as some may genuinely have been encouraged by the FAQ to not upload their model/code.
", "votes": 2, "replies": [] }, { "id": 316069, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-18T05:44:02.833000", "content": "\n\n\nWe cannot say it is not written anywhere, sorry.
\n
The confusion here seems to be that it was not written specifically that models had to be uploaded for participants to continue participating during Stage 2 (while the FAQ seems to indicate otherwise, i.e. that this is not a necessity for participating in Stage 2), and in this respect, I do agree that this is an anomaly that needs to be addressed by Kaggle. However, by the uploading of models, it would also be essential to emphasize that the models should be viable and the same model is being used to obtain the results for Stage 2, without any re-training on the dataset. Otherwise, by just allowing participants to upload models to meet the Stage 2 participation requirement, a loophole is provided for participants to submit 'dummy code', with some participants continuing to change their model/re-train their model (using Stage 1 test data) even after the Stage 2 test data has been released.
\n\n\n\n\nPeople who comply with the timeline page are at a disadvantage\n compared to people who don't. That's what is unfair.
\n
It is not just people who comply with the timeline page, but also contestants who have limited themselves to using only the Stage 1 training data for training. So even if someone complies with the timeline (by submitting their model before Stage 1 closes), but then proceeds to change/re-train their model (after the Stage 2 dataset is revealed), this would be considered as being unfair to other participants.
", "votes": 2, "replies": [] }, { "id": 316073, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-18T05:52:53.347000", "content": "\n\n\nFor this competition, given the FAQ, it may be hard to remove people,\n as some may genuinely have been encouraged by the FAQ to not upload\n their model/code.
\n
This is indeed true, and in fact it gets very messy here, since the definition of what should be allowed (and what shouldn't) becomes grayed. But I partially think that the idea proposed by Kaggle of submitting the model's SHA256 hash prior to the close of Stage 1 and subsequently submitting the actual model (having the same SHA256 hash) after Stage 2 closes (when the participant lands in a prize position to win a prize) is the best alternative in this context. This is to protect the intellectual property (IP) rights of the participants (some participants may be using self-developed models which they are intending to patent after the competition ends), thus allowing the prize-winning participants the right to decide if they do indeed wish to submit their models for evaluation and to win a prize, or if they prefer to withhold their models in the event of other constraints (e.g. filing a patent, etc), in which case, they would have forsaken the award.
", "votes": 1, "replies": [] }, { "id": 316081, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-18T06:41:04.710000", "content": "@Maggie like others said, you should have changed the https://www.kaggle.com/two-stage-faqin time. Now you can only remove models that finish in a prize position. That's the way all 2-stage competitions were run up until now. It's greatly inconsistent to change it after the competition's end. Besides if that was written in time, then anyone could have just submitted a dummy model or a public kernel code model, because I'm certain you are not going to run all submitted code and verify the submissions for all participants, that's why the rule is made that way.\nAnd if it's somebody's opinion that should be seriously accounted for, with all respect to other participants, that's Giba's, top Kaggler for long and respected by everyone in the community. Even the topic's starter agrees with this.
\n\nRemoving people just for not uploading any dummy model, when the rules stated that it is ok if you didn't compete for the prizes, by changing the rules retrospectively, that's what is unfair.
", "votes": 4, "replies": [] }, { "id": 316293, "author_name": "bsp2020", "author_url": "", "post_date": "2018-04-18T16:38:43.030000", "content": "I think, when the rule of the specific competition (rules specified in the timeline) conflicts with the rule of the general Kaggle (2 stage faq) competitions, competition rule prevails. So, though I agree that there were confusions, it is within their rights to enforce the competition rule.
", "votes": 0, "replies": [] }, { "id": 316337, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-18T19:13:19.720000", "content": "@bsp2020 the code upload timeline rule is specified in every other 2-stage competition held so far, however in accordance with the faq page removal can only stand for prize winning positions. And the reason is simple. Given that Kaggle can't reproduce all uploaded models to verify the submissions, anyone could just upload a dummy code. If they wanted to enforce code restriction then they should have chosen Kernel only competition type and take care for the hardware requirements such a decision would incur. Since they chose 2-stage type and didn't change the faq clarification in time, it is unfair to remove participants.
", "votes": 3, "replies": [] }, { "id": 316388, "author_name": "bsp2020", "author_url": "", "post_date": "2018-04-18T22:17:20.917000", "content": "@gavrand Not that I want to continue this unproductive discussion, but I'll respond since you specifically replied to me.
\n\nI think your argument is that, since they did not enforce the rule in the past, they can't do it for this competition. No?
\n\nI think the rule is pretty clear. The competition specific rule prevails. Sure, they have not been enforced in the past competition. Not sure why that matters. When I got a speeding ticket, I argued that the officer did not enforce the rule properly because he did not catch all the speeders and he should not give me a ticket either. You should have seen his face. I'll never argue in such a way next time I'm stopped by a police. :)
", "votes": 0, "replies": [] }, { "id": 316439, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-19T03:19:18.603000", "content": "\n\n\nI think the rule is pretty clear. The competition specific rule\n prevails.
\n
Nowhere was it explicitly mentioned in the competition rules (up to the competition closure on 16 April 2018, 23:59H) that individuals who did not submit their models in Stage 1 would be barred from participating in Stage 2 (& I have screenshots to prove this, so no further amendments after the said date would be acceptable). Yes, you may argue that there was a timeline for model submission for Stage 1, but that is not equivalent to enforcing that individuals have to submit their models in Stage 1 to participate in Stage 2. In contrast, the Kaggle FAQ (accessible at https://www.kaggle.com/two-stage-faq) specifically addresses this matter (and which I quote for your easy reference below):
\n\n\n\n\nWill I still be on the leaderboard if I don't upload a model but do\n submit in the second stage? Yes. However, if you do not upload a model\n and finish in a prize position, your team will be removed from the\n competition standings entirely.
\n
You have to be very clear and explicit in this context, else there would be multiple interpretations, which can't be generalized in this (or any other) context for that matter.\nOn a different note, it comes across as very intriguing to me as to why some participants in this competition are earnestly arguing on the unfairness of the lack of model submission but continued participation by some contestants, even highlighting the need for the FAQ to be re-written in this respect, but gloss over the fact that changes to the submitted model (e.g. by re-training on Stage 1 test data & labels) are acceptable, stating that this was allowed by the Kaggle admin/organizers. Shouldn't fairness be enforced equally in all respects, and not be skewed to suit one's actions, else this would undeniably be considered as a form of corruption? Also, there seems to be some people who are downvoting this post, suggesting that these individuals are indeed actively promoting corruption.
", "votes": 3, "replies": [] }, { "id": 316454, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-19T04:49:17.063000", "content": "@bsp2020 let me explain my point of view because there seems to be a misunderstanding. I have nothing personal, I respect every opinion, I just express my disagreement.
\n\nMy argument is that the rule consists of two parts:\nFirst part is the code upload timeline, that stands for all participants. \nSecond part is that in accordance with the FAQ if you don't upload code then you can still be on the leaderboard, except for the case that you finish in a prize winning position and your submission will be removed.
\n\nThe reason I mention the previous 2-stage competitions is because they are testament that the rule was enforced this way so far. Now it appears as if this rule is being enforced only partially, without previous notice and retrospectively. That's unfair. If the change was published in time then all participants would be discouraged to not upload their model and would almost certainly upload their code, because getting results in such a competition requires effort, time and money, and wasting all that for such a simple thing is irrational. I can see that you are relatively new in Kaggle (3 competitions). I have done much more and I just testify my experience about previous 2-stage competitions.
\n\nAnd since you mention speeding tickets, it is as if the signs restricts you at 65mph, and the police officer fines you for traveling at 50mph because, retrospectively and with no further notice, it was decided that the speed limit should be changed to 45mph.
", "votes": 2, "replies": [] }, { "id": 316499, "author_name": "rmldj", "author_url": "", "post_date": "2018-04-19T08:01:55.553000", "content": "@Shiraz S K
\n\n\n\n\nOn a different note, it comes across as very intriguing to me as to why some participants in this competition are earnestly arguing on the unfairness of the lack of model submission but continued participation by some contestants, even highlighting the need for the FAQ to be re-written in this respect, but gloss over the fact that changes to the submitted model (e.g. by re-training on Stage 1 test data & labels) are acceptable, stating that this was allowed by the Kaggle admin/organizers.
\n
Since you mentioned this point already a couple of times, I think there is a misunderstanding: a submitted model to Kaggle does not mean the weights of a neural network or other classifier but rather the complete algorithm and processing pipeline which can be run without human intervention (up to correcting errors) to produce the stage 2 submissions.
\n\nIn particular, this algorithm may contain a training phase (which can be run on stage1 train+test data) and a subsequent prediction phase. The key point is that no manual fiddling with any parameters can take place.
", "votes": 0, "replies": [] }, { "id": 316506, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-19T08:24:59.047000", "content": "@rmldj: That is precisely the point - allowing participants to re-train their model on Stage 1 test data and submit the new model's predictions on Stage 2 test data can also be considered as condoning cheating, since it creates a completely different model from what was originally used for Stage 1 test label predictions. This would defeat the purpose of code freezing in the first instance, since the model architecture can now be varied (& quite significantly too) by subjecting it to a different training set each time. In some instances, code can also be included in the Stage 1 script to allow the model to dynamically change the hyperparameters without manual intervention, for e.g. if the user defines the number of training epochs, width or depth of the neural network (which represent hyperparameters that should not be varied) to be proportional to the size of the training dataset, a larger training dataset would in turn change these values, which would (in turn) result in a totally different model architecture. This would be highly unfair to other individuals who do not resort to such practices, but use their models constructed from Stage 1 for Stage 2 test label prediction.
", "votes": 0, "replies": [] } ] }, { "id": 314627, "author_name": "Matthew Anderson", "author_url": "", "post_date": "2018-04-16T00:56:53.533000", "content": "I fully agree. It is simply unfair: different rules are being enforced for different people (this is the exact definition of unfair). While I am not a talented data scientist, I have never participated in a two-stage competition because of these rules.
", "votes": 1, "replies": [ { "id": 314794, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T09:37:06.183000", "content": "Different rules are enforced for different competitions as well - not every competition gives you a monetary prize & not every competition gives the same number of participants monetary awards (e.g. for this competition, you get a monetary award if you make it to the top 5 on the LB, but some competitions only award the top 3 participants). Would you consider this to be unfair as well?
", "votes": -1, "replies": [] }, { "id": 315209, "author_name": "Matthew Anderson", "author_url": "", "post_date": "2018-04-16T22:02:47.833000", "content": "I would not consider that unfair. To clarify, I meant unfair as in not equal for everyone. If competition rules hold everyone to the same regulations, it seems fair.
", "votes": 0, "replies": [] }, { "id": 316455, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-19T04:52:40.623000", "content": "Yes, indeed, but this debate seems to picking on one such rule (namely the continued participation by some individuals despite not having previously submitted a model) & claiming that this is unfair, when in fact, it has been common practice for this to occur in multiple 2-stage competitions in the past, without such claims being made. This is just because of some people's grievances in this respect, and although justified, these same people overlook other occurrences (e.g. the re-training of already-submitted models), while these might also be considered to be unfair to other participants.
", "votes": 0, "replies": [] } ] }, { "id": 314552, "author_name": "DStjhb", "author_url": "", "post_date": "2018-04-15T19:46:52.640000", "content": "I completely agree with this. I have actually modified my code in a way that I think could significantly increase my score. Under the rules, it seems I could choose to submit a .csv based on this new code, as I am unlikely to be in top 5, just to increase my chance of a medal.
\n\nHowever, I see this as cheating, so I will use an upload based on my frozen model for my final submission.
\n\nI will upload a .csv with my new model just to see how well it would have done, but I will not do this until after the deadline, when it will be counted as late submission, and can simply be used to evaluate my model improvements.
\n\nI honestly think you should be disqualified from submitting in stage 2 if you have not uploaded a frozen model in stage 1. And although you couldn't manually assess if people have been honest, to make it clear that you will be removed from the leaderboard if you are discovered would discourage at least some people from cheating.
\n\nIt's annoying there isn't some sort of checksum that could be performed to ensure that each .csv has been generated by a specific uploaded model code. Maybe some genius can come up with something.
", "votes": 0, "replies": [ { "id": 314564, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-15T20:20:41.783000", "content": "Your last paragraph reminds of Zero knowledge protocol. If you are interested in this topic, it is a fascinating concept.
", "votes": 2, "replies": [] }, { "id": 314588, "author_name": "DStjhb", "author_url": "", "post_date": "2018-04-15T21:54:33.630000", "content": "Interesting stuff. Thanks for the info.
", "votes": 1, "replies": [] }, { "id": 314591, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-15T22:13:08.767000", "content": "Glad that you liked it. It was sort of a hot topic when I was in grad school. I didn't work in that area but some of my colleagues did. The wikipedia article section 5.2 is more relevant to your earlier discussion about using ZNP to enforce ethical behavior.
", "votes": 1, "replies": [] }, { "id": 314881, "author_name": "Anne Carpenter", "author_url": "", "post_date": "2018-04-16T12:31:23.110000", "content": "I wasn't involved in setting the overall design of the competition, being new to Kaggle myself, but it does indeed seem like there is no good way to guard against cheating other than manual checking.
\n\n@DStjhb: to me, it sounds sensible to not allow stage 2 submissions if you haven't uploaded code. On the other hand, it's easy for anyone to upload some nonsense code to meet that criterion. Maybe because it's such an easy workaround Kaggle has chosen not to enforce at that stage.
", "votes": 2, "replies": [] }, { "id": 314888, "author_name": "DStjhb", "author_url": "", "post_date": "2018-04-16T12:41:51.680000", "content": "@Anne: I agree it is really difficult to guard against in practise. Especially in these big competitions with thousands of entries. I guess you could have some sort of Kaggle virtual environment that you run your code inside, which could log any changes; but that would limit what languages you can use etc.
\n\nThis is the first competition I have tried, so I may simply be projecting my fears about cheating, rather than speaking with any real experience of seeing it happen. It would be good for Kaggle to do some research into the prevalence of cheating on their platform. Maybe you could do an anonymous survey (making it explicit there will be no repercussions), and ask people some questions, whether they modified code etc., so we have an idea whether our fears are grounded in reality, or if we are all just making a big fuss about a negligible percentage of users!
", "votes": 1, "replies": [] }, { "id": 314921, "author_name": "Shiraz S K", "author_url": "", "post_date": "2018-04-16T13:59:25.473000", "content": "@Anne re:
\n\n\n\n\n@DStjhb: to me, it sounds sensible to not allow stage 2 submissions if\n you haven't uploaded code. On the other hand, it's easy for anyone to\n upload some nonsense code to meet that criterion. Maybe because it's\n such an easy workaround Kaggle has chosen not to enforce at that\n stage.
\n
Yes, it might be true that nonsense code uploaded into the system will easily meet the criterion. Then there's also the possibility of contestants using external datasets and test data from Stage 1 which would also be unfair to other participants and thus be considered as \"cheating\". So I still think the use of Kernels will be an apt way to counter this issue (as highlighted in my previous post) since it ensures that the code uploaded is the one used to create the model, and that the dataset used is the one supplied by the organizers for the competition, and not some other dataset. The only issue with Kernels would be the lack of libraries supported by the current Kernel environment, unless Kaggle decides to do something about this in future.
\n\n@DStjhb re:
\n\n\n\n\n@Anne: I agree it is really difficult to guard against in practise.\n Especially in these big competitions with thousands of entries. I\n guess you could have some sort of Kaggle virtual environment that you\n run your code inside, which could log any changes; but that would\n limit what languages you can use etc.
\n
Yes, that's why I suggested that organizers on Kaggle should only utilize Kernels for future 2-stage competitions.
", "votes": 0, "replies": [] } ] }, { "id": 314421, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T14:36:09.620000", "content": "Edit: moved to main post.
", "votes": 0, "replies": [] }, { "id": 314487, "author_name": "", "author_url": "", "post_date": "2018-04-15T17:06:52.543000", "content": "", "votes": 0, "replies": [ { "id": 314489, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-15T17:12:41.737000", "content": "I think it is fair to say that some of the greatest advances in expert domains were done by experts in other domains. If you think you have an interesting idea you may want to go for it!
", "votes": 1, "replies": [] }, { "id": 314494, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T17:26:34.743000", "content": "@the1owl, I am not sure I understand what you write. Is it related to my post at all?
\n\nWhere I relate to it is that I am a beginner in image processing, and do not expect much from this competition. I came to learn, and I did learn a lot, from Heng Cher Keng, and from many others.
", "votes": 0, "replies": [] }, { "id": 314500, "author_name": "", "author_url": "", "post_date": "2018-04-15T17:34:48.573000", "content": "", "votes": 2, "replies": [] }, { "id": 314507, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-15T17:46:05.140000", "content": "@the1owl Agreed. I could only wrap my head around something simpler as UNet. I got a public score jump from 0.35 to 0.40 right on the deadline but it is dead on arrival for the second stage. It is also way short of what @CPMP and his team got with UNet too. I'd definitely would like to learn about it more.
", "votes": 0, "replies": [] }, { "id": 314518, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-15T18:12:09.893000", "content": "@the1owl I intend to share my code after competition end.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "314302": "To Kaggle staff.\n\nYour 2 stage faq (https://www.kaggle.com/two-stage-faq) reads:\n\n> The spirit of having a second stage is to prevent hand labeling and\n> leaderboard probing of the test data. In order to achieve this, we ask\n> that you upload your source code, including the correct parameters\n> that you used for generating your submission files. This is for you to\n> prove that you have written automated code to create your final\n> submission(s).\n\nYet, further down in the page it allows people to participate in Stage 2 if they have not uploaded their code. I don't get why you do this, given you can easily check that code has been uploaded in stage1.\n\nPeople that have not uploaded code are free to modify their code and get a better rank. It is only if they are in the prize money (unlikely for most of us) that they can be caught.\n\nThis is very unfair. It is very unfair to all who try to be in the spirit of the competition, i.e. reuse their stage 1 code with minimal, non scientific, change to cope with the new stage2 test data.\n\nEdit on April 16: \nSeems there is disagreement with calling this cheating. Here is why I used that word.\n\nThe timeline page of this competition reads:\n\n> April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline.\n\nAnyone not doing it is not complying with the competition description. I'm fine if some prefer to call it *not complying* rather than *cheating*. ", "314945": "here are my views:\n\n1. there is no way to prevent \"cheating\" , even for 1 stage competition. e.g. the user can hand label test data in one-stage competition. if you don't win a prize, kaggle don't check your code\n\n2. why do you want to \"cheat\"? for people who want to get good ranking for resume, it could be done like this:\n\n- submit legally for the competition, it is ok that you get poor rank\n\n- after competition, you still can do a post submission. at this point of time many top solutions would be made known. you can re-implement to get good results.\n\nat the interview, you can say although my ranking was not good, but i am fast to learn. and my post-results ranked 10, up from 100 in the period of one week after studying the top solutions.\n\n\nMaybe kaggle can made a third ranking, called post-competition ranking. Then no one would need to cheat becuase anyone can submit after the competition for good ranking\n\n\n\n\n\n\n", "314810": "@CPMP To be honest, I was very worried. I finished stage 1 in 9th place. This is my first chance to get a gold medal. In stage 2, my prediction is very disastrous and I found it may be repaired by a simple modification. I thought about cheating to get a gold medal seriously. Thank you. Your words made me stop short of cheating.", "314430": "I agree with CPMP, the rules in the two-stage-faq allow someone to circumvent the main reason for having a two stage competition in the first place. I had wanted to start a thread like this myself but then I thought that there really doesn't seem to be a good way to eliminate all \"cheating\" so I decided against it.\n\nThe main reason for having a two stage competition is to ensure that contestants cannot see the final test set until they have finalized their code. So they cannot tailor their code to this final test set. However the two-stage-faq does not enforce this. Allowing contestants to make a submission and get a ranking without having uploaded a model means that they are not bound by the model upload rule. We then have a situation where people who have uploaded a model are only allowed to make submissions based on their uploaded model. This model has not been tailored to fit the final test data. On the other hand those people who have not uploaded a model in stage 1 do not have make sure that their submission comes from any specific model. They are free to look at the final test data, change their model, and then make their submission.\n\nNow one might argue that not having uploaded a model means that one cannot win any of the prizes. But the prizes are not the only thing that matter to contestants. There are plenty of people who participate in these competitions because it affects their careers or their education. The kaggle ranking matters and has real world consequences, even if you don't win a prize.\n\nI'm not sure that there is any sure-fire way of eliminating all \"cheating\" but it can be reduced if people are only allowed to make a submission when they have uploaded a model during stage 1.\n\nAnother approach might be to allow everyone to make submissions without any model constraints. This would ensure that we have a level playing field when it comes to rankings, basically the same conditions as those for a regular single stage competition. But when it comes to the prizes, only submissions that can be verified with uploaded models would be considered, and submissions without models in prize-winning positions would be removed.", "314965": "I just love this topic. It seems that all of us forgot or some are not even aware what Paulo Pinto did in Fisheries competition https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/discussion/31607\n\nAll the points made about 2stage not being \"fair\" or is a place to cheat here are valid, but imho, there is no better solution how to make it \"fair\"", "314371": "The fact that I got down votes is a confirmation that some participants want to change their code for stage2.", "314970": "I am using this competition to learn about python and deep learning in general (I have learned a lot!). I know I have no chance of winning a prize based on my stage 1 score (top 30%) but I had to make some changes to get a valid .csv file for stage 2 submission (due to atypical data compared to stage 1). I can easily see there are many more things for me to learn. \n\n I don't think continuous learning should be considered cheating, it should actually be encouraged!\n\ntx\n\nKickback", "314384": "@CPMP No offense but I think that it's always been like that in every other 2 stage competition hosted. If they want to enforce code lock then they choose to host kernel only competition. Besides, top 10-20 teams are going to abide by the code rule enforcement, it is too big a risk to lose the prize. As far as the others are concerned, it is pretty time consuming for Kaggle to manually crosscheck codes and submissions, given the different parameters and enviroment variety, since I repeat it is not a kernel only competition where all people use the same platform. It is difficult already to do so for the top winning ones. It is this variety they want, given that the problem difficulty is such that hosting a kernel only competition would limit the solution range. So please take it cool, and note that the term \"cheating\" you chose is exaggerated and offending to other Kagglers who know it was always like that in 2 stage competitions. That's the reason you are being downvoted.", "486136": "spam :(", "314676": "True", "316023": "At the beginning of stage2, kaggle released the solution to stage1 which is an important supplement to the training data. I wonder if those got score more than 0.55 did not use those solutions to retrain their model, then how can they finalize their model before the solutions were released? \nAnd also, the test data in stage2 contained lots of unseen types, which means it is very hard to get a better score than stage1 without argument the training data to suit this situation. If not, how can the top 100 got 0.1 increase on an unfit data? Which means most of the players did not submit their prediction from the model in stage1 but did some further work on it. \nSo, the model uploaded in stage1 is just a 'sign in', no guarantee at all. If we only focus on the 'code', then we miss the effect of training data, one can just manipulate on training data without touch the code to re-train their model. \nTo fix this issue once for all, kaggle should require to submit the model file like .h5, then use the model to predict on the hidden data by themselves.", "314985": "This is a machine learning problem on how you ranked the following words in order of importance:\n\n\"integrity\", \"fame\", \"correct\", \"wrong\", \"rule\", \"results\", \"value\", ...", "314405": "@CPMP: Although I was not one of those who downvoted you, I have to say that I do not agree that this could be considered cheating, unless the intention of the person who submitted the code was to win a prize. Otherwise, it would amount to nothing more than a playground competition for those who did not submit their code. I, for one, did not submit my code, and I have officially withdrawn from this competition (as stated in the thread posting at https://www.kaggle.com/c/data-science-bowl-2018/discussion/54161) as well as explicitly indicated my team (which consists solely of myself) as such (i.e. **Withdrawn - Not for award consideration**) on the leaderboard. Nonetheless, I'd still like to make amendments to my current model, and upload the latest results to the leaderboard to see how my revised model would fare in terms of its accuracy in ROI prediction. For me, the accuracy & capability of my model is of utmost importance, as I am an avid optical microscopist who utilizes a vast range of light microscopy & imaging modalities, and unlike most Kagglers here, I did not use any existing machine learning packages for this competition (such as deep neural networks, mask-rcnn, etc), but rather created my own algorithm from scratch, hence I'd be interested to evaluate the accuracy, capability and robustness of my model & algorithm in this context.", "315024": "Hi there - Thanks for your comments. While people can compete in Stage 2 without uploading their model in Stage 1, it says that their submissions and standings will be removed after the competition is over. ", "314627": "I fully agree. It is simply unfair: different rules are being enforced for different people (this is the exact definition of unfair). While I am not a talented data scientist, I have never participated in a two-stage competition because of these rules.", "314552": "I completely agree with this. I have actually modified my code in a way that I think could significantly increase my score. Under the rules, it seems I could choose to submit a .csv based on this new code, as I am unlikely to be in top 5, just to increase my chance of a medal. \n\nHowever, I see this as cheating, so I will use an upload based on my frozen model for my final submission.\n\nI will upload a .csv with my new model just to see how well it would have done, but I will not do this until after the deadline, when it will be counted as late submission, and can simply be used to evaluate my model improvements.\n\nI honestly think you should be disqualified from submitting in stage 2 if you have not uploaded a frozen model in stage 1. And although you couldn't manually assess if people have been honest, to make it clear that you **will** be removed from the leaderboard if you are discovered would discourage at least some people from cheating.\n\nIt's annoying there isn't some sort of checksum that could be performed to ensure that each .csv has been generated by a specific uploaded model code. Maybe some genius can come up with something.", "314421": "Edit: moved to main post.", "314487": "" } }