Since they can't control people if they use the leaky data or not (except top 3), there will be always some people utilizing it. If they make it public, then the competition may not have a point anymore from the host point of view. If that is the case, I would suggest to finalize the competition with the submissions submitted before Jerome's submission (I would say including his since he deserves it) and distribute the points and medals (maybe without money prize). By this way, people's efforts are not wasted. Ofc, it may be unfair for some people but it will be fair for most of the people. Because this competition had a lot of issues and will be still having some issues, so there is no point to continue it.
", "messages": [ { "id": 402411, "postDate": "2018-10-11T17:11:28.317Z", "content": "That doesn't sound like a solid solution.
\n\nI am -1 for freezing leaderboard. It literally becomes first come first serve medals rather than awarding the actual best models.
", "rawMarkdown": "That doesn't sound like a solid solution. \n\n1. What about people who were testing their models locally and thought\n of submitting over this weekend?\n2. How can we validate current submissions (before Jerome) who were using some part of external data and got good score with a simple model?\n3. Rules and merger deadline are for a reason. So, everyone is on the same page and can submit their best models for medals to be awarded. Freezing deadline would be rude and unfair to all the participants who had faith that competition is still on for about a month.\n\nI am -1 for freezing leaderboard. It literally becomes first come first serve medals rather than awarding the actual best models. ", "votes": 14, "replies": [ { "id": 402432, "postDate": "2018-10-11T17:59:52.483Z", "content": "Agreed from my side. There will be many people who have not submitted their solution thinking we have one month left. That's really unfair. Most of them not even blended their submission and submit their single model predictions.
\n\nFrom my side: for all my submission I didn't even clip the test prediction revenues to zero.
\n\n-1 for freezing the leaderboard.
", "rawMarkdown": "Agreed from my side. There will be many people who have not submitted their solution thinking we have one month left. That's really unfair. Most of them not even blended their submission and submit their single model predictions.\n\nFrom my side: for all my submission I didn't even clip the test prediction revenues to zero.\n\n-1 for freezing the leaderboard.", "votes": 2 }, { "id": 402435, "postDate": "2018-10-11T18:05:28.500Z", "content": "As I said my solution is not 100% fair because it means that the predetermined deadline will change. I only claim that my solution is the least harmful one out of possible solutions if they cannot really recover the competition.
", "rawMarkdown": "As I said my solution is not 100% fair because it means that the predetermined deadline will change. I only claim that my solution is the least harmful one out of possible solutions if they cannot really recover the competition.", "votes": 3 }, { "id": 402450, "postDate": "2018-10-11T18:29:14.757Z", "content": "Agree with you @divrikwicky (AhmetErdem). This competition is broken irreversibly. I really hope Kaggle can come up with something hidden in their sleeve as measure plan B. (Assuming that's the reason they yet didnt respond to the leak. They usually answer queries ASAP)
\n\nI am +1 what @fatihozturk said. Awarding ranking points before the leak leaderboard is fine, but again there's no perfect solution.
", "rawMarkdown": "Agree with you @divrikwicky (AhmetErdem). This competition is broken irreversibly. I really hope Kaggle can come up with something hidden in their sleeve as measure plan B. (Assuming that's the reason they yet didnt respond to the leak. They usually answer queries ASAP)\n\n I am +1 what @fatihozturk said. Awarding ranking points before the leak leaderboard is fine, but again there's no perfect solution. ", "votes": 2 }, { "id": 402721, "postDate": "2018-10-12T07:44:24.963Z", "content": "shaz has made a good point. Now the leak (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the datamart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement.
\n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimising the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better characteristic eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement also.
\n\nSo I may suggest to change the eval metric as well to a higher the better one.
\n\nIn that way all the submitted solutions will be invalid and everyone's effort will be wasted and the competition will be fair for everybody.
", "rawMarkdown": "shaz has made a good point. Now the leak (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the datamart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement. \n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimising the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better characteristic eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement also.\n\nSo I may suggest to change the eval metric as well to a higher the better one. \n\nIn that way all the submitted solutions will be invalid and **everyone's effort will be wasted and the competition will be fair for everybody**. " } ] }, { "id": 402413, "postDate": "2018-10-11T17:12:38.807Z", "content": "I think your suggestion sounds tough but realistic. I believe in that there are many people including me who did not submit their planned blending submissions yet and this could change LB considerably. However, I would definitely go with your suggestion and get ranking points instead of a complete cancelation and having nothing at the end.
", "rawMarkdown": "I think your suggestion sounds tough but realistic. I believe in that there are many people including me who did not submit their planned blending submissions yet and this could change LB considerably. However, I would definitely go with your suggestion and get ranking points instead of a complete cancelation and having nothing at the end.", "votes": 10, "replies": [ { "id": 402414, "postDate": "2018-10-11T17:17:33.900Z", "content": "Ranking points are great. But, I expect Kaggle to restore competition by making it kernels only -- No external data. Any suggestions on this @fatihozturk ?
", "rawMarkdown": "Ranking points are great. But, I expect Kaggle to restore competition by making it kernels only -- No external data. Any suggestions on this @fatihozturk ? " }, { "id": 402421, "postDate": "2018-10-11T17:35:17.930Z", "content": "I don’t think that making kernels-only can solve this problem. Anyone can still find a way to leak. Even with a little thinking I can find some ways sneaky enough to get a gold. Didn’t you see Santander Comp? People wrote amazing scripts and spent many hours for the leak. It’s so funny our target is publicly available on some google website :) I think test set must be a new one if possible and completely unreachable.
", "rawMarkdown": "I don’t think that making kernels-only can solve this problem. Anyone can still find a way to leak. Even with a little thinking I can find some ways sneaky enough to get a gold. Didn’t you see Santander Comp? People wrote amazing scripts and spent many hours for the leak. It’s so funny our target is publicly available on some google website :) I think test set must be a new one if possible and completely unreachable.", "votes": 5 }, { "id": 402841, "postDate": "2018-10-12T12:43:14.263Z", "content": "@fatihöztürk Data overhaul would indeed be best, but I am skeptical it's going to take place - mostly because I've never seen it happen so far, and there have been quite a few instances of leaks over the last few years. Come to think about it, the proportion of leaky competitions does seem to be increasing :-/
", "rawMarkdown": "@fatihöztürk Data overhaul would indeed be best, but I am skeptical it's going to take place - mostly because I've never seen it happen so far, and there have been quite a few instances of leaks over the last few years. Come to think about it, the proportion of leaky competitions does seem to be increasing :-/", "votes": 1 }, { "id": 402857, "postDate": "2018-10-12T13:14:45.927Z", "content": "Hi Konrad. You must be right about the increasing proportion since I'm an only 1-year old Kaggler. However, data overhaul happened for current Airbus competition. You can check it here. It would be great if the same happened for last Santander competition as well, but we continued on that even it turned into be pointless. Let's hope that Kaggle will be more cautious about this motivation killer issue.
", "rawMarkdown": "Hi Konrad. You must be right about the increasing proportion since I'm an only 1-year old Kaggler. However, data overhaul happened for current Airbus competition. You can check it [here][1]. It would be great if the same happened for last Santander competition as well, but we continued on that even it turned into be pointless. Let's hope that Kaggle will be more cautious about this motivation killer issue.\n\n\n [1]: http://www.kaggle.com/c/airbus-ship-detection/discussion/68042", "votes": 1 }, { "id": 402906, "postDate": "2018-10-12T14:45:42.473Z", "content": "Hah! I missed the Airbus relaunch - thanks for pointing it out.
", "rawMarkdown": "Hah! I missed the Airbus relaunch - thanks for pointing it out." } ] }, { "id": 402378, "postDate": "2018-10-11T15:54:11.807Z", "content": "Since they can't control people if they use the leaky data or not (except top 3), there will be always some people utilizing it. If they make it public, then the competition may not have a point anymore from the host point of view. If that is the case, I would suggest to finalize the competition with the submissions submitted before Jerome's submission (I would say including his since he deserves it) and distribute the points and medals (maybe without money prize). By this way, people's efforts are not wasted. Ofc, it may be unfair for some people but it will be fair for most of the people. Because this competition had a lot of issues and will be still having some issues, so there is no point to continue it.
", "rawMarkdown": "Since they can't control people if they use the leaky data or not (except top 3), there will be always some people utilizing it. If they make it public, then the competition may not have a point anymore from the host point of view. If that is the case, I would suggest to finalize the competition with the submissions submitted before Jerome's submission (I would say including his since he deserves it) and distribute the points and medals (maybe without money prize). By this way, people's efforts are not wasted. Ofc, it may be unfair for some people but it will be fair for most of the people. Because this competition had a lot of issues and will be still having some issues, so there is no point to continue it.", "votes": 6 }, { "id": 402428, "postDate": "2018-10-11T17:45:59.337Z", "content": "I think we are all agreed that the leak means the competition cannot continue in the current format. I think many will agree with me when I say that I personally want to use kaggle to work on good techniques not on exploiting leaks. I do recognise that avoiding leaks completely is not an easy task but this one seems particularly bad.
\n\nAhmet's suggestion does have some merit in recognising work already done, but, Shaz has raised good objections as to the fairness of this approach. I think Kaggle should consider if they can replace the dataset with data from a different but comparable store for which the data is not public. This isn't ideal either because the distribution for every store would be different and some rework would be needed. But on the plus side, much existing work could likely be re-used and there should be enough time to answer all of Shaz's objections regarding fairness whilst allowing medals and ranking points to be awarded to recognise the work done to date rather than voiding the competition which would not recognise that.
", "rawMarkdown": "I think we are all agreed that the leak means the competition cannot continue in the current format. I think many will agree with me when I say that I personally want to use kaggle to work on good techniques not on exploiting leaks. I do recognise that avoiding leaks completely is not an easy task but this one seems particularly bad.\n\nAhmet's suggestion does have some merit in recognising work already done, but, Shaz has raised good objections as to the fairness of this approach. I think Kaggle should consider if they can replace the dataset with data from a different but comparable store for which the data is not public. This isn't ideal either because the distribution for every store would be different and some rework would be needed. But on the plus side, much existing work could likely be re-used and there should be enough time to answer all of Shaz's objections regarding fairness whilst allowing medals and ranking points to be awarded to recognise the work done to date rather than voiding the competition which would not recognise that.", "votes": 4 }, { "id": 426715, "postDate": "2018-11-23T17:38:55.567Z", "content": "I got a lot of downvotes on this topic but I feel like time proved that it was the best solution. I guess this competition will end up in a lottery leaderboard with no proper model and real life usability.
", "rawMarkdown": "I got a lot of downvotes on this topic but I feel like time proved that it was the best solution. I guess this competition will end up in a lottery leaderboard with no proper model and real life usability.", "votes": 1, "replies": [ { "id": 426987, "postDate": "2018-11-24T10:01:39.853Z", "content": "On the contrary, I believe this version of competition is much more useful in real life setting vs previous one. In the previous edition you could, for example, look into the future to predict a revenue for current session. What is more, you knew there was actually a session, which is already a big information.
\n\nRight now, the competition setting mimicks a real-life scenario: imagine we have 30.11.2018, we are thinking which customers to send an email with the offer, and we are judged how well we predicted who will buy.
\n\nThe big difference vs real life I can see is that normally we would use all data, eg. until 30.11.2018 to make prediction, and in this case we can only use data until 15.10.2018.
", "rawMarkdown": "On the contrary, I believe this version of competition is much more useful in real life setting vs previous one. In the previous edition you could, for example, look into the future to predict a revenue for current session. What is more, you knew there was actually a session, which is already a big information.\n\nRight now, the competition setting mimicks a real-life scenario: imagine we have 30.11.2018, we are thinking which customers to send an email with the offer, and we are judged how well we predicted who will buy.\n\nThe big difference vs real life I can see is that normally we would use all data, eg. until 30.11.2018 to make prediction, and in this case we can only use data until 15.10.2018.", "votes": 3 } ] }, { "id": 403322, "postDate": "2018-10-13T09:58:09.390Z", "content": "Ι think kaggle has only 2 simple solutions to the problem. Either they stop the competition or they leave it as it is. All the other proposed solutions based on freezing the LB don't seem to be realistic. Besides, the official statement that the host did not know the existance of the demo account reveal lack of knowledge of data boundaries which indicates that they may end up again with something worse.\nFinally congrats to Jerome. Even with the leak known for days nobody can even reach him. Great analytical work!
", "rawMarkdown": "Ι think kaggle has only 2 simple solutions to the problem. Either they stop the competition or they leave it as it is. All the other proposed solutions based on freezing the LB don't seem to be realistic. Besides, the official statement that the host did not know the existance of the demo account reveal lack of knowledge of data boundaries which indicates that they may end up again with something worse.\nFinally congrats to Jerome. Even with the leak known for days nobody can even reach him. Great analytical work!", "votes": 1 }, { "id": 403129, "postDate": "2018-10-12T23:59:01.260Z", "content": "I dont think it would be a good solution .\nWhat about the people who worked hard on it and planing to submit later day?\nWhat if few people already used the leak but to be on safe added little noise to it. how can we be sure that other than jerome no one else used the leak.?
", "rawMarkdown": "I dont think it would be a good solution .\nWhat about the people who worked hard on it and planing to submit later day?\nWhat if few people already used the leak but to be on safe added little noise to it. how can we be sure that other than jerome no one else used the leak.?", "votes": 2 }, { "id": 402840, "postDate": "2018-10-12T12:39:28.527Z", "content": "I am not completely sure a change can be realistically expected: there have been quite a few instances of leaky competitions where the final Kaggle decision was to keep calm and carry on. The ultimate winners openly admitted usage of leaks in their solution and just kept on building on top of it - and it wouldn't surprise me at all if we ended up in the same situation here.
", "rawMarkdown": "I am not completely sure a change can be realistically expected: there have been quite a few instances of leaky competitions where the final Kaggle decision was to keep calm and carry on. The ultimate winners openly admitted usage of leaks in their solution and just kept on building on top of it - and it wouldn't surprise me at all if we ended up in the same situation here. ", "votes": 2 }, { "id": 427258, "postDate": "2018-11-25T01:01:32.453Z", "content": "I think the total opposite, the new rules turned this competition into something way more realistic. Real life has not leaderboards and you cannot really see the future before it happens either.
", "rawMarkdown": "I think the total opposite, the new rules turned this competition into something way more realistic. Real life has not leaderboards and you cannot really see the future before it happens either." }, { "id": 402422, "postDate": "2018-10-11T17:40:04.863Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 402418, "postDate": "2018-10-11T17:25:35.963Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 402411, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-10-11T17:11:28.317000", "content": "That doesn't sound like a solid solution.
\n\nI am -1 for freezing leaderboard. It literally becomes first come first serve medals rather than awarding the actual best models.
", "votes": 14, "replies": [ { "id": 402432, "author_name": "redshellspy", "author_url": "", "post_date": "2018-10-11T17:59:52.483000", "content": "Agreed from my side. There will be many people who have not submitted their solution thinking we have one month left. That's really unfair. Most of them not even blended their submission and submit their single model predictions.
\n\nFrom my side: for all my submission I didn't even clip the test prediction revenues to zero.
\n\n-1 for freezing the leaderboard.
", "votes": 2, "replies": [] }, { "id": 402435, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-10-11T18:05:28.500000", "content": "As I said my solution is not 100% fair because it means that the predetermined deadline will change. I only claim that my solution is the least harmful one out of possible solutions if they cannot really recover the competition.
", "votes": 3, "replies": [] }, { "id": 402450, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-10-11T18:29:14.757000", "content": "Agree with you @divrikwicky (AhmetErdem). This competition is broken irreversibly. I really hope Kaggle can come up with something hidden in their sleeve as measure plan B. (Assuming that's the reason they yet didnt respond to the leak. They usually answer queries ASAP)
\n\nI am +1 what @fatihozturk said. Awarding ranking points before the leak leaderboard is fine, but again there's no perfect solution.
", "votes": 2, "replies": [] }, { "id": 402721, "author_name": "Arindam", "author_url": "", "post_date": "2018-10-12T07:44:24.963000", "content": "shaz has made a good point. Now the leak (to some extent) has been published and right now there is no point to continue the competition with current leakage data. I think host should re publish the datamart and replace it with the current one and extend the timeline. May change the eval metric or even the problem statement.
\n\nI have seen a leakage trend over the years in different Kaggle competitions that whenever it's minimising the error (based on MAE, RMSE, MSE, SMAPE etc) i.e. the lower the better characteristic eval statistic, the leakage for those competitions are more than wherever there is higher the better ( AUC, Accuracy etc) statistic competition. But it's completely based on problem statement also.
\n\nSo I may suggest to change the eval metric as well to a higher the better one.
\n\nIn that way all the submitted solutions will be invalid and everyone's effort will be wasted and the competition will be fair for everybody.
", "votes": 0, "replies": [] } ] }, { "id": 402413, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-11T17:12:38.807000", "content": "I think your suggestion sounds tough but realistic. I believe in that there are many people including me who did not submit their planned blending submissions yet and this could change LB considerably. However, I would definitely go with your suggestion and get ranking points instead of a complete cancelation and having nothing at the end.
", "votes": 10, "replies": [ { "id": 402414, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-10-11T17:17:33.900000", "content": "Ranking points are great. But, I expect Kaggle to restore competition by making it kernels only -- No external data. Any suggestions on this @fatihozturk ?
", "votes": 0, "replies": [] }, { "id": 402421, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-11T17:35:17.930000", "content": "I don’t think that making kernels-only can solve this problem. Anyone can still find a way to leak. Even with a little thinking I can find some ways sneaky enough to get a gold. Didn’t you see Santander Comp? People wrote amazing scripts and spent many hours for the leak. It’s so funny our target is publicly available on some google website :) I think test set must be a new one if possible and completely unreachable.
", "votes": 5, "replies": [] }, { "id": 402841, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-10-12T12:43:14.263000", "content": "@fatihöztürk Data overhaul would indeed be best, but I am skeptical it's going to take place - mostly because I've never seen it happen so far, and there have been quite a few instances of leaks over the last few years. Come to think about it, the proportion of leaky competitions does seem to be increasing :-/
", "votes": 1, "replies": [] }, { "id": 402857, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-12T13:14:45.927000", "content": "Hi Konrad. You must be right about the increasing proportion since I'm an only 1-year old Kaggler. However, data overhaul happened for current Airbus competition. You can check it here. It would be great if the same happened for last Santander competition as well, but we continued on that even it turned into be pointless. Let's hope that Kaggle will be more cautious about this motivation killer issue.
", "votes": 1, "replies": [] }, { "id": 402906, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-10-12T14:45:42.473000", "content": "Hah! I missed the Airbus relaunch - thanks for pointing it out.
", "votes": 0, "replies": [] } ] }, { "id": 402428, "author_name": "MichaelP", "author_url": "", "post_date": "2018-10-11T17:45:59.337000", "content": "I think we are all agreed that the leak means the competition cannot continue in the current format. I think many will agree with me when I say that I personally want to use kaggle to work on good techniques not on exploiting leaks. I do recognise that avoiding leaks completely is not an easy task but this one seems particularly bad.
\n\nAhmet's suggestion does have some merit in recognising work already done, but, Shaz has raised good objections as to the fairness of this approach. I think Kaggle should consider if they can replace the dataset with data from a different but comparable store for which the data is not public. This isn't ideal either because the distribution for every store would be different and some rework would be needed. But on the plus side, much existing work could likely be re-used and there should be enough time to answer all of Shaz's objections regarding fairness whilst allowing medals and ranking points to be awarded to recognise the work done to date rather than voiding the competition which would not recognise that.
", "votes": 4, "replies": [] }, { "id": 426715, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-11-23T17:38:55.567000", "content": "I got a lot of downvotes on this topic but I feel like time proved that it was the best solution. I guess this competition will end up in a lottery leaderboard with no proper model and real life usability.
", "votes": 1, "replies": [ { "id": 426987, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2018-11-24T10:01:39.853000", "content": "On the contrary, I believe this version of competition is much more useful in real life setting vs previous one. In the previous edition you could, for example, look into the future to predict a revenue for current session. What is more, you knew there was actually a session, which is already a big information.
\n\nRight now, the competition setting mimicks a real-life scenario: imagine we have 30.11.2018, we are thinking which customers to send an email with the offer, and we are judged how well we predicted who will buy.
\n\nThe big difference vs real life I can see is that normally we would use all data, eg. until 30.11.2018 to make prediction, and in this case we can only use data until 15.10.2018.
", "votes": 3, "replies": [] } ] }, { "id": 403322, "author_name": "Evdilos_Ikaria", "author_url": "", "post_date": "2018-10-13T09:58:09.390000", "content": "Ι think kaggle has only 2 simple solutions to the problem. Either they stop the competition or they leave it as it is. All the other proposed solutions based on freezing the LB don't seem to be realistic. Besides, the official statement that the host did not know the existance of the demo account reveal lack of knowledge of data boundaries which indicates that they may end up again with something worse.\nFinally congrats to Jerome. Even with the leak known for days nobody can even reach him. Great analytical work!
", "votes": 1, "replies": [] }, { "id": 403129, "author_name": "Ranjeet Jain", "author_url": "", "post_date": "2018-10-12T23:59:01.260000", "content": "I dont think it would be a good solution .\nWhat about the people who worked hard on it and planing to submit later day?\nWhat if few people already used the leak but to be on safe added little noise to it. how can we be sure that other than jerome no one else used the leak.?
", "votes": 2, "replies": [] }, { "id": 402840, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-10-12T12:39:28.527000", "content": "I am not completely sure a change can be realistically expected: there have been quite a few instances of leaky competitions where the final Kaggle decision was to keep calm and carry on. The ultimate winners openly admitted usage of leaks in their solution and just kept on building on top of it - and it wouldn't surprise me at all if we ended up in the same situation here.
", "votes": 2, "replies": [] }, { "id": 427258, "author_name": "luisduque440", "author_url": "", "post_date": "2018-11-25T01:01:32.453000", "content": "I think the total opposite, the new rules turned this competition into something way more realistic. Real life has not leaderboards and you cannot really see the future before it happens either.
", "votes": 0, "replies": [] }, { "id": 402422, "author_name": "", "author_url": "", "post_date": "2018-10-11T17:40:04.863000", "content": "", "votes": 2, "replies": [] }, { "id": 402418, "author_name": "", "author_url": "", "post_date": "2018-10-11T17:25:35.963000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "402411": "That doesn't sound like a solid solution. \n\n1. What about people who were testing their models locally and thought\n of submitting over this weekend?\n2. How can we validate current submissions (before Jerome) who were using some part of external data and got good score with a simple model?\n3. Rules and merger deadline are for a reason. So, everyone is on the same page and can submit their best models for medals to be awarded. Freezing deadline would be rude and unfair to all the participants who had faith that competition is still on for about a month.\n\nI am -1 for freezing leaderboard. It literally becomes first come first serve medals rather than awarding the actual best models. ", "402413": "I think your suggestion sounds tough but realistic. I believe in that there are many people including me who did not submit their planned blending submissions yet and this could change LB considerably. However, I would definitely go with your suggestion and get ranking points instead of a complete cancelation and having nothing at the end.", "402378": "Since they can't control people if they use the leaky data or not (except top 3), there will be always some people utilizing it. If they make it public, then the competition may not have a point anymore from the host point of view. If that is the case, I would suggest to finalize the competition with the submissions submitted before Jerome's submission (I would say including his since he deserves it) and distribute the points and medals (maybe without money prize). By this way, people's efforts are not wasted. Ofc, it may be unfair for some people but it will be fair for most of the people. Because this competition had a lot of issues and will be still having some issues, so there is no point to continue it.", "402428": "I think we are all agreed that the leak means the competition cannot continue in the current format. I think many will agree with me when I say that I personally want to use kaggle to work on good techniques not on exploiting leaks. I do recognise that avoiding leaks completely is not an easy task but this one seems particularly bad.\n\nAhmet's suggestion does have some merit in recognising work already done, but, Shaz has raised good objections as to the fairness of this approach. I think Kaggle should consider if they can replace the dataset with data from a different but comparable store for which the data is not public. This isn't ideal either because the distribution for every store would be different and some rework would be needed. But on the plus side, much existing work could likely be re-used and there should be enough time to answer all of Shaz's objections regarding fairness whilst allowing medals and ranking points to be awarded to recognise the work done to date rather than voiding the competition which would not recognise that.", "426715": "I got a lot of downvotes on this topic but I feel like time proved that it was the best solution. I guess this competition will end up in a lottery leaderboard with no proper model and real life usability.", "403322": "Ι think kaggle has only 2 simple solutions to the problem. Either they stop the competition or they leave it as it is. All the other proposed solutions based on freezing the LB don't seem to be realistic. Besides, the official statement that the host did not know the existance of the demo account reveal lack of knowledge of data boundaries which indicates that they may end up again with something worse.\nFinally congrats to Jerome. Even with the leak known for days nobody can even reach him. Great analytical work!", "403129": "I dont think it would be a good solution .\nWhat about the people who worked hard on it and planing to submit later day?\nWhat if few people already used the leak but to be on safe added little noise to it. how can we be sure that other than jerome no one else used the leak.?", "402840": "I am not completely sure a change can be realistically expected: there have been quite a few instances of leaky competitions where the final Kaggle decision was to keep calm and carry on. The ultimate winners openly admitted usage of leaks in their solution and just kept on building on top of it - and it wouldn't surprise me at all if we ended up in the same situation here. ", "427258": "I think the total opposite, the new rules turned this competition into something way more realistic. Real life has not leaderboards and you cannot really see the future before it happens either.", "402422": "", "402418": "" } }