Fellow Kagglers,
\n\nnot that the leaderboard of Stage 1 matters in any way, but can we all agree that it is better to avoid previous competitions' habit of submitting perfect scores (which are very easy to submit as they are provided) just to stay in front for a few weeks?
\n\nIt really ruins the LB, it sparks conversations about leaks (and there can't be leaks in such competition, this is what makes it special), or magic features, or other mythical creatures and generally wastes time we can spend more productively.
\n\nCan we all have this ML practitioners' agreement? :)
", "messages": [ { "id": 746781, "postDate": "2020-02-15T14:44:29.480Z", "content": "Fellow Kagglers,
\n\nnot that the leaderboard of Stage 1 matters in any way, but can we all agree that it is better to avoid previous competitions' habit of submitting perfect scores (which are very easy to submit as they are provided) just to stay in front for a few weeks?
\n\nIt really ruins the LB, it sparks conversations about leaks (and there can't be leaks in such competition, this is what makes it special), or magic features, or other mythical creatures and generally wastes time we can spend more productively.
\n\nCan we all have this ML practitioners' agreement? :)
", "rawMarkdown": "Fellow Kagglers,\n\nnot that the leaderboard of Stage 1 matters in any way, but can we all agree that it is better to avoid previous competitions' habit of submitting perfect scores (which are very easy to submit as they are provided) just to stay in front for a few weeks?\n\nIt really ruins the LB, it sparks conversations about leaks (and there can't be leaks in such competition, this is what makes it special), or magic features, or other mythical creatures and generally wastes time we can spend more productively.\n\nCan we all have this ML practitioners' agreement? :)", "votes": 50 }, { "id": 749475, "postDate": "2020-02-18T18:18:42.330Z", "content": "If you can't play nice, we'll make you play nice ๐
", "rawMarkdown": "If you can't play nice, we'll *[make you*](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131163) play nice ๐", "votes": 13 }, { "id": 746835, "postDate": "2020-02-15T15:59:33.630Z", "content": "I say: aye.
", "rawMarkdown": "I say: aye.", "votes": 2 }, { "id": 747528, "postDate": "2020-02-16T14:48:14.917Z", "content": "I guess it didn't hurt to ask :)
", "rawMarkdown": "I guess it didn't hurt to ask :)", "votes": 3 }, { "id": 754830, "postDate": "2020-02-24T05:20:05.810Z", "content": "agree with you ... i say yes
", "rawMarkdown": "agree with you ... i say yes", "votes": 1 }, { "id": 751505, "postDate": "2020-02-20T08:44:53.043Z", "content": "Agree with you. Even one rotten tomato spoils the rest.
", "rawMarkdown": "Agree with you. Even one rotten tomato spoils the rest. ", "votes": 1 }, { "id": 748025, "postDate": "2020-02-17T05:49:13.160Z", "content": "we have the actual labels of LB.\nIt will also happen those who are not willing to ruin LB would make leaked models and ruin LB.\nLB is meaningless from the beginning of the competition, so it doesn't matter if it ruins.
", "rawMarkdown": "we have the actual labels of LB.\nIt will also happen those who are not willing to ruin LB would make leaked models and ruin LB.\nLB is meaningless from the beginning of the competition, so it doesn't matter if it ruins.", "votes": 2, "replies": [ { "id": 748051, "postDate": "2020-02-17T06:33:46.190Z", "content": "I agree, I think it is important to understand which file makes leak.\nMany public kernels actually uses MNCAATourneyCompactResults.csv with 2015-2019, which means those models already uses \"leaked data\" (maybe some kernel publisher even did not know its fact).
๐ Oh you are right.I use another dateset MEvent2019 and so on.I think it isn't leaked date from beginning to end๏ผ but in fact it is.Seems that we should do data preprocessing and drop the leak row before modelling
OK that explains why they're not setting early stopping instead keep training 10000 iterations and get a low LB score
", "rawMarkdown": "OK that explains why they're not setting early stopping instead keep training 10000 iterations and get a low LB score", "votes": 1 } ] }, { "id": 746996, "postDate": "2020-02-15T20:42:23.960Z", "content": "Aye.
", "rawMarkdown": "Aye.", "votes": 2 }, { "id": 747000, "postDate": "2020-02-15T20:51:05.667Z", "content": "I COMPLETELY agree with you but unfortunately there will ALWAYS be a newbie who will be doing exactly that.
\n\nLots of LBs throughout Kaggle are ruined because of such behaviors.
", "rawMarkdown": "I COMPLETELY agree with you but unfortunately there will ALWAYS be a newbie who will be doing exactly that.\n\nLots of LBs throughout Kaggle are ruined because of such behaviors." }, { "id": 747214, "postDate": "2020-02-16T05:57:56.267Z", "content": "I don't know what they get from it,but it definetly ruins the LB!๐ค
", "rawMarkdown": "I don't know what they get from it,but it definetly ruins the LB!๐ค ", "votes": 1 }, { "id": 747050, "postDate": "2020-02-15T22:14:01.797Z", "content": "Good post; very much in favour of this agreement.
", "rawMarkdown": "Good post; very much in favour of this agreement. ", "votes": 1 }, { "id": 746954, "postDate": "2020-02-15T19:19:28.947Z", "content": "I agree with you, but I can tell you from a small amount of personal experience: being high up on a public leaderboard, no matter the reason, brings a lot of attention to your profile (e.g. linkedin views, requests, new followers, notebook views etc...) Way more than actually finishing 20-50th in a competition ;)
\n\nWhether it's the right kind of attention is not something people seem to consider!
", "rawMarkdown": "I agree with you, but I can tell you from a small amount of personal experience: being high up on a public leaderboard, no matter the reason, brings a lot of attention to your profile (e.g. linkedin views, requests, new followers, notebook views etc...) Way more than actually finishing 20-50th in a competition ;)\n\nWhether it's the right kind of attention is not something people seem to consider!\n\n", "votes": 1, "replies": [ { "id": 746998, "postDate": "2020-02-15T20:44:03.553Z", "content": "Yeah, it's kind of a shame. One would hope before hiring somebody based on such a thing they might ask some more questions or dig deeper. Kind of like they asked that woman who 'won' the Boston Marathon if she did interval training and she didn't know what that was...
", "rawMarkdown": "Yeah, it's kind of a shame. One would hope before hiring somebody based on such a thing they might ask some more questions or dig deeper. Kind of like they asked that woman who 'won' the Boston Marathon if she did interval training and she didn't know what that was...", "votes": 1 } ] }, { "id": 746916, "postDate": "2020-02-15T18:30:13.713Z", "content": "Its already started...it's hard to stop...Lets see how he got this score...
", "rawMarkdown": "Its already started...it's hard to stop...Lets see how he got this score...", "votes": 1 }, { "id": 751773, "postDate": "2020-02-20T13:54:25.853Z", "content": "Maybe this is a silly question, but in this case do you all still use the 2015-2018 tourney data to predict 2019 for stage 1? Or do you omit all of the games in the test set even though those results would have been known by 2019?
\n\nI feel like using all games prior is a better way to calibrate to previous competition leaderboards, but it does seem much easier to just omit all games 2015-2019 for training.
", "rawMarkdown": "Maybe this is a silly question, but in this case do you all still use the 2015-2018 tourney data to predict 2019 for stage 1? Or do you omit all of the games in the test set even though those results would have been known by 2019?\n\nI feel like using all games prior is a better way to calibrate to previous competition leaderboards, but it does seem much easier to just omit all games 2015-2019 for training.", "replies": [ { "id": 751836, "postDate": "2020-02-20T15:02:12.623Z", "content": "Since the test data for Stage 1 spans multiple seasons, the standard way to handle using the 2015-2019 regular season data (while using as much data as possible) would be to train multiple models. To predict the 2015 tournament results, you could train a model on any/all the data through the 2014 tournament (or technically through the 2015 regular season), whereas the model you use to predict the 2016 tournament results could be trained using all the data through the 2015 tournament, and so on. You could also use multiple rounds of training on the same model instead of separate models, feeding in the additional data as allowed, if that seems beneficial to you. You could also just leave out all data from before 2015 (as you've mentioned), although the staggered approach should lead to a better validation estimate.
\n\nThis is also a complicated issue because the last four seasons of training data make up a substantial part of the overall data set size. As you've suggested, it may be beneficial to leave out any future data to prevent a source of bias in your validation estimate caused by data leakage, but it also may be potentially harmful to your validation estimate to leave out this data because the reduction in training set size might be a source of either additional variance or additional bias (or both!) in your validation estimate. Depending on how you want to resolve this issue, a cross-validation scheme like leave-one-season-out might be more beneficial overall, and I think others may have used such an approach to finish well in the past.
\n\nThat said, I'd still say that the safest way to handle this issue would be something like the former approach. The latter approach should definitely lead to a lower validation estimate, but it's really difficult to know whether the gains from this leave-one-season-out scheme would be from the model taking advantage of legitimate or illegitimate patterns in the future data, and the former approach should still correlate well with the Stage 2 performance of any given modeling approach.
", "rawMarkdown": "Since the test data for Stage 1 spans multiple seasons, the standard way to handle using the 2015-2019 regular season data (while using as much data as possible) would be to train multiple models. To predict the 2015 tournament results, you could train a model on any/all the data through the 2014 tournament (or technically through the 2015 regular season), whereas the model you use to predict the 2016 tournament results could be trained using all the data through the 2015 tournament, and so on. You could also use multiple rounds of training on the same model instead of separate models, feeding in the additional data as allowed, if that seems beneficial to you. You could also just leave out all data from before 2015 (as you've mentioned), although the staggered approach should lead to a better validation estimate.\n\nThis is also a complicated issue because the last four seasons of training data make up a substantial part of the overall data set size. As you've suggested, it *may* be beneficial to leave out any future data to prevent a source of bias in your validation estimate caused by data leakage, but it also *may* be potentially harmful to your validation estimate to leave out this data because the reduction in training set size might be a source of either additional variance or additional bias (or both!) in your validation estimate. Depending on how you want to resolve this issue, a cross-validation scheme like leave-one-season-out *might* be more beneficial overall, and I think others may have used such an approach to finish well in the past.\n\nThat said, I'd still say that the safest way to handle this issue would be something like the former approach. The latter approach should definitely lead to a lower validation estimate, but it's really difficult to know whether the gains from this leave-one-season-out scheme would be from the model taking advantage of legitimate or illegitimate patterns in the future data, and the former approach should still correlate well with the Stage 2 performance of any given modeling approach.", "votes": 4 }, { "id": 751959, "postDate": "2020-02-20T17:09:50.407Z", "content": "\n\nThis is probably one of the most insightful posts anyone will share. There's a lot more nuance to this type of modeling problem than most realize; I keep discovering more as I keep working. \nI've settled on the approach of using only 'available' data at the time of prediction for the last six seasons (of which we have the winner's scores) and testing on six models (one for each year) - but I still haven't determined if it's the best approach.
\n\nMy next step is to create a model up through 2014 and test it on tournaments 2014-2019. I don't expect it will be better but I will find out.
", "rawMarkdown": "@joshstone \n\nThis is probably one of the most insightful posts anyone will share. There's a lot more nuance to this type of modeling problem than most realize; I keep discovering more as I keep working. \nI've settled on the approach of using only 'available' data at the time of prediction for the last six seasons (of which we have the winner's scores) and testing on six models (one for each year) - but I still haven't determined if it's the best approach. \n\nMy next step is to create a model up through 2014 and test it on tournaments 2014-2019. I don't expect it will be better but I will find out. ", "votes": 1 }, { "id": 752417, "postDate": "2020-02-21T02:20:39.990Z", "content": "Yes, thank you for the detailed response, @joshstone. Going to start with using only pre-2015 data for simplicity since I will be retraining for the 2020 predictions anyway. As a newbie to ML/data science I really such a nuanced response on some of these more complicated topics!
", "rawMarkdown": "Yes, thank you for the detailed response, @joshstone. Going to start with using only pre-2015 data for simplicity since I will be retraining for the 2020 predictions anyway. As a newbie to ML/data science I really such a nuanced response on some of these more complicated topics!" } ] }, { "id": 749175, "postDate": "2020-02-18T11:52:48.603Z", "content": "I was wondering how some people had perfect scores, it makes sense now !
", "rawMarkdown": "I was wondering how some people had perfect scores, it makes sense now !" }, { "id": 748626, "postDate": "2020-02-17T19:00:32.637Z", "content": "Yeah
", "rawMarkdown": "Yeah" }, { "id": 748208, "postDate": "2020-02-17T09:29:25.107Z", "content": "Aye
", "rawMarkdown": "Aye" }, { "id": 747978, "postDate": "2020-02-17T04:08:18.557Z", "content": "Why do people do this. It's the definition of overfitting at its finest lol...
", "rawMarkdown": "Why do people do this. It's the definition of overfitting at its finest lol..." }, { "id": 747758, "postDate": "2020-02-16T20:06:50.477Z", "content": "Top 7 have 0.0000 LB score. LOL! ๐ Let's see when does this take a drastic turn. ??? Something must be done, I agree.
", "rawMarkdown": "Top 7 have 0.0000 LB score. LOL! ๐ Let's see when does this take a drastic turn. ??? Something must be done, I agree.", "replies": [ { "id": 747789, "postDate": "2020-02-16T21:11:25.390Z", "content": "Last year half of the LB was with the perfect 0.0000, it is a problem only until Stage 2
", "rawMarkdown": "Last year half of the LB was with the perfect 0.0000, it is a problem only until Stage 2" }, { "id": 749615, "postDate": "2020-02-18T20:08:05.573Z", "content": "Kaggle is in FBI role. And all submissions with log loss < 0.2 are getting disqualified for now..
", "rawMarkdown": "Kaggle is in FBI role. And all submissions with log loss < 0.2 are getting disqualified for now.." } ] }, { "id": 747573, "postDate": "2020-02-16T15:27:58.540Z", "content": "What do you think is the highest reasonable score currently on the LB? I see an entry with something like 0.01 score , is that legit?
", "rawMarkdown": "What do you think is the highest reasonable score currently on the LB? I see an entry with something like 0.01 score , is that legit? ", "replies": [ { "id": 747586, "postDate": "2020-02-16T15:50:48.323Z", "content": "Stage 1 is based on the previous 5 seasons (2015-2019). The winning scores for those seasons were 0.43893, 0.48130, 0.43857, 0.53194 (Go UMBC! ๐), and 0.41477. SotA performance on Kaggle has advanced significantly over that period, so I don't think a simple average would make sense, but it still seems fair to estimate that anything << 0.45 is probably taking unfair advantage of the previous results.
\n\nThat said, I commend the OP for this valiant effort, but it seems ill-fated. Too many Kagglers will just want that moment of glory where they can see their name on top of the leaderboard. ๐คท
", "rawMarkdown": "Stage 1 is based on the previous 5 seasons (2015-2019). The winning scores for those seasons were 0.43893, 0.48130, 0.43857, 0.53194 (Go UMBC! ๐), and 0.41477. SotA performance on Kaggle has advanced significantly over that period, so I don't think a simple average would make sense, but it still seems fair to estimate that anything << 0.45 is probably taking unfair advantage of the previous results.\n\nThat said, I commend the OP for this valiant effort, but it seems ill-fated. Too many Kagglers will just want that moment of glory where they can see their name on top of the leaderboard. ๐คท ", "votes": 4 } ] }, { "id": 747354, "postDate": "2020-02-16T10:22:02.690Z", "content": "I didn't even know mine would give a perfect score I just continue training.
", "rawMarkdown": "I didn't even know mine would give a perfect score I just continue training." }, { "id": 747516, "postDate": "2020-02-16T14:42:02.573Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 749475, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-02-18T18:18:42.330000", "content": "If you can't play nice, we'll make you play nice ๐
", "votes": 13, "replies": [] }, { "id": 746835, "author_name": "Trigram", "author_url": "", "post_date": "2020-02-15T15:59:33.630000", "content": "I say: aye.
", "votes": 2, "replies": [] }, { "id": 747528, "author_name": "Luca Basanisi", "author_url": "", "post_date": "2020-02-16T14:48:14.917000", "content": "I guess it didn't hurt to ask :)
", "votes": 3, "replies": [] }, { "id": 754830, "author_name": "siddharth garishakurthi", "author_url": "", "post_date": "2020-02-24T05:20:05.810000", "content": "agree with you ... i say yes
", "votes": 1, "replies": [] }, { "id": 751505, "author_name": "Kunaal Naik", "author_url": "", "post_date": "2020-02-20T08:44:53.043000", "content": "Agree with you. Even one rotten tomato spoils the rest.
", "votes": 1, "replies": [] }, { "id": 748025, "author_name": "Sugawarya", "author_url": "", "post_date": "2020-02-17T05:49:13.160000", "content": "we have the actual labels of LB.\nIt will also happen those who are not willing to ruin LB would make leaked models and ruin LB.\nLB is meaningless from the beginning of the competition, so it doesn't matter if it ruins.
", "votes": 2, "replies": [ { "id": 748051, "author_name": "corochann", "author_url": "", "post_date": "2020-02-17T06:33:46.190000", "content": "I agree, I think it is important to understand which file makes leak.\nMany public kernels actually uses MNCAATourneyCompactResults.csv with 2015-2019, which means those models already uses \"leaked data\" (maybe some kernel publisher even did not know its fact).
๐ Oh you are right.I use another dateset MEvent2019 and so on.I think it isn't leaked date from beginning to end๏ผ but in fact it is.Seems that we should do data preprocessing and drop the leak row before modelling
OK that explains why they're not setting early stopping instead keep training 10000 iterations and get a low LB score
", "votes": 1, "replies": [] } ] }, { "id": 746996, "author_name": "S Charlesworth", "author_url": "", "post_date": "2020-02-15T20:42:23.960000", "content": "Aye.
", "votes": 2, "replies": [] }, { "id": 747000, "author_name": "Mourad", "author_url": "", "post_date": "2020-02-15T20:51:05.667000", "content": "I COMPLETELY agree with you but unfortunately there will ALWAYS be a newbie who will be doing exactly that.
\n\nLots of LBs throughout Kaggle are ruined because of such behaviors.
", "votes": 0, "replies": [] }, { "id": 747214, "author_name": "Shahules", "author_url": "", "post_date": "2020-02-16T05:57:56.267000", "content": "I don't know what they get from it,but it definetly ruins the LB!๐ค
", "votes": 1, "replies": [] }, { "id": 747050, "author_name": "Heads or Tails", "author_url": "", "post_date": "2020-02-15T22:14:01.797000", "content": "Good post; very much in favour of this agreement.
", "votes": 1, "replies": [] }, { "id": 746954, "author_name": "James Trotman", "author_url": "", "post_date": "2020-02-15T19:19:28.947000", "content": "I agree with you, but I can tell you from a small amount of personal experience: being high up on a public leaderboard, no matter the reason, brings a lot of attention to your profile (e.g. linkedin views, requests, new followers, notebook views etc...) Way more than actually finishing 20-50th in a competition ;)
\n\nWhether it's the right kind of attention is not something people seem to consider!
", "votes": 1, "replies": [ { "id": 746998, "author_name": "S Charlesworth", "author_url": "", "post_date": "2020-02-15T20:44:03.553000", "content": "Yeah, it's kind of a shame. One would hope before hiring somebody based on such a thing they might ask some more questions or dig deeper. Kind of like they asked that woman who 'won' the Boston Marathon if she did interval training and she didn't know what that was...
", "votes": 1, "replies": [] } ] }, { "id": 746916, "author_name": "Ashish Kumar Singh ๐ฎ๐ณ", "author_url": "", "post_date": "2020-02-15T18:30:13.713000", "content": "Its already started...it's hard to stop...Lets see how he got this score...
", "votes": 1, "replies": [] }, { "id": 751773, "author_name": "Ryan Armstrong", "author_url": "", "post_date": "2020-02-20T13:54:25.853000", "content": "Maybe this is a silly question, but in this case do you all still use the 2015-2018 tourney data to predict 2019 for stage 1? Or do you omit all of the games in the test set even though those results would have been known by 2019?
\n\nI feel like using all games prior is a better way to calibrate to previous competition leaderboards, but it does seem much easier to just omit all games 2015-2019 for training.
", "votes": 0, "replies": [ { "id": 751836, "author_name": "Josh Stone", "author_url": "", "post_date": "2020-02-20T15:02:12.623000", "content": "Since the test data for Stage 1 spans multiple seasons, the standard way to handle using the 2015-2019 regular season data (while using as much data as possible) would be to train multiple models. To predict the 2015 tournament results, you could train a model on any/all the data through the 2014 tournament (or technically through the 2015 regular season), whereas the model you use to predict the 2016 tournament results could be trained using all the data through the 2015 tournament, and so on. You could also use multiple rounds of training on the same model instead of separate models, feeding in the additional data as allowed, if that seems beneficial to you. You could also just leave out all data from before 2015 (as you've mentioned), although the staggered approach should lead to a better validation estimate.
\n\nThis is also a complicated issue because the last four seasons of training data make up a substantial part of the overall data set size. As you've suggested, it may be beneficial to leave out any future data to prevent a source of bias in your validation estimate caused by data leakage, but it also may be potentially harmful to your validation estimate to leave out this data because the reduction in training set size might be a source of either additional variance or additional bias (or both!) in your validation estimate. Depending on how you want to resolve this issue, a cross-validation scheme like leave-one-season-out might be more beneficial overall, and I think others may have used such an approach to finish well in the past.
\n\nThat said, I'd still say that the safest way to handle this issue would be something like the former approach. The latter approach should definitely lead to a lower validation estimate, but it's really difficult to know whether the gains from this leave-one-season-out scheme would be from the model taking advantage of legitimate or illegitimate patterns in the future data, and the former approach should still correlate well with the Stage 2 performance of any given modeling approach.
", "votes": 4, "replies": [] }, { "id": 751959, "author_name": "Ben Tesch", "author_url": "", "post_date": "2020-02-20T17:09:50.407000", "content": "\n\nThis is probably one of the most insightful posts anyone will share. There's a lot more nuance to this type of modeling problem than most realize; I keep discovering more as I keep working. \nI've settled on the approach of using only 'available' data at the time of prediction for the last six seasons (of which we have the winner's scores) and testing on six models (one for each year) - but I still haven't determined if it's the best approach.
\n\nMy next step is to create a model up through 2014 and test it on tournaments 2014-2019. I don't expect it will be better but I will find out.
", "votes": 1, "replies": [] }, { "id": 752417, "author_name": "Ryan Armstrong", "author_url": "", "post_date": "2020-02-21T02:20:39.990000", "content": "Yes, thank you for the detailed response, @joshstone. Going to start with using only pre-2015 data for simplicity since I will be retraining for the 2020 predictions anyway. As a newbie to ML/data science I really such a nuanced response on some of these more complicated topics!
", "votes": 0, "replies": [] } ] }, { "id": 749175, "author_name": "Mohamed Hazgui", "author_url": "", "post_date": "2020-02-18T11:52:48.603000", "content": "I was wondering how some people had perfect scores, it makes sense now !
", "votes": 0, "replies": [] }, { "id": 748626, "author_name": "Houssem Ayed", "author_url": "", "post_date": "2020-02-17T19:00:32.637000", "content": "Yeah
", "votes": 0, "replies": [] }, { "id": 748208, "author_name": "L...\"N\"", "author_url": "", "post_date": "2020-02-17T09:29:25.107000", "content": "Aye
", "votes": 0, "replies": [] }, { "id": 747978, "author_name": "MrSir", "author_url": "", "post_date": "2020-02-17T04:08:18.557000", "content": "Why do people do this. It's the definition of overfitting at its finest lol...
", "votes": 0, "replies": [] }, { "id": 747758, "author_name": "Atul Anand {Jha}", "author_url": "", "post_date": "2020-02-16T20:06:50.477000", "content": "Top 7 have 0.0000 LB score. LOL! ๐ Let's see when does this take a drastic turn. ??? Something must be done, I agree.
", "votes": 0, "replies": [ { "id": 747789, "author_name": "Luca Basanisi", "author_url": "", "post_date": "2020-02-16T21:11:25.390000", "content": "Last year half of the LB was with the perfect 0.0000, it is a problem only until Stage 2
", "votes": 0, "replies": [] }, { "id": 749615, "author_name": "Atul Anand {Jha}", "author_url": "", "post_date": "2020-02-18T20:08:05.573000", "content": "Kaggle is in FBI role. And all submissions with log loss < 0.2 are getting disqualified for now..
", "votes": 0, "replies": [] } ] }, { "id": 747573, "author_name": "Satwik", "author_url": "", "post_date": "2020-02-16T15:27:58.540000", "content": "What do you think is the highest reasonable score currently on the LB? I see an entry with something like 0.01 score , is that legit?
", "votes": 0, "replies": [ { "id": 747586, "author_name": "Josh Stone", "author_url": "", "post_date": "2020-02-16T15:50:48.323000", "content": "Stage 1 is based on the previous 5 seasons (2015-2019). The winning scores for those seasons were 0.43893, 0.48130, 0.43857, 0.53194 (Go UMBC! ๐), and 0.41477. SotA performance on Kaggle has advanced significantly over that period, so I don't think a simple average would make sense, but it still seems fair to estimate that anything << 0.45 is probably taking unfair advantage of the previous results.
\n\nThat said, I commend the OP for this valiant effort, but it seems ill-fated. Too many Kagglers will just want that moment of glory where they can see their name on top of the leaderboard. ๐คท
", "votes": 4, "replies": [] } ] }, { "id": 747354, "author_name": "", "author_url": "", "post_date": "2020-02-16T10:22:02.690000", "content": "I didn't even know mine would give a perfect score I just continue training.
", "votes": 0, "replies": [] }, { "id": 747516, "author_name": "", "author_url": "", "post_date": "2020-02-16T14:42:02.573000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "746781": "Fellow Kagglers,\n\nnot that the leaderboard of Stage 1 matters in any way, but can we all agree that it is better to avoid previous competitions' habit of submitting perfect scores (which are very easy to submit as they are provided) just to stay in front for a few weeks?\n\nIt really ruins the LB, it sparks conversations about leaks (and there can't be leaks in such competition, this is what makes it special), or magic features, or other mythical creatures and generally wastes time we can spend more productively.\n\nCan we all have this ML practitioners' agreement? :)", "749475": "If you can't play nice, we'll *[make you*](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131163) play nice ๐", "746835": "I say: aye.", "747528": "I guess it didn't hurt to ask :)", "754830": "agree with you ... i say yes", "751505": "Agree with you. Even one rotten tomato spoils the rest. ", "748025": "we have the actual labels of LB.\nIt will also happen those who are not willing to ruin LB would make leaked models and ruin LB.\nLB is meaningless from the beginning of the competition, so it doesn't matter if it ruins.", "746996": "Aye.", "747000": "I COMPLETELY agree with you but unfortunately there will ALWAYS be a newbie who will be doing exactly that.\n\nLots of LBs throughout Kaggle are ruined because of such behaviors.", "747214": "I don't know what they get from it,but it definetly ruins the LB!๐ค ", "747050": "Good post; very much in favour of this agreement. ", "746954": "I agree with you, but I can tell you from a small amount of personal experience: being high up on a public leaderboard, no matter the reason, brings a lot of attention to your profile (e.g. linkedin views, requests, new followers, notebook views etc...) Way more than actually finishing 20-50th in a competition ;)\n\nWhether it's the right kind of attention is not something people seem to consider!\n\n", "746916": "Its already started...it's hard to stop...Lets see how he got this score...", "751773": "Maybe this is a silly question, but in this case do you all still use the 2015-2018 tourney data to predict 2019 for stage 1? Or do you omit all of the games in the test set even though those results would have been known by 2019?\n\nI feel like using all games prior is a better way to calibrate to previous competition leaderboards, but it does seem much easier to just omit all games 2015-2019 for training.", "749175": "I was wondering how some people had perfect scores, it makes sense now !", "748626": "Yeah", "748208": "Aye", "747978": "Why do people do this. It's the definition of overfitting at its finest lol...", "747758": "Top 7 have 0.0000 LB score. LOL! ๐ Let's see when does this take a drastic turn. ??? Something must be done, I agree.", "747573": "What do you think is the highest reasonable score currently on the LB? I see an entry with something like 0.01 score , is that legit? ", "747354": "I didn't even know mine would give a perfect score I just continue training.", "747516": "" } }