Ready to receive downvotes but would like to publish it anyway.
\n\nThere is no such definition as a \"FAIR competition\". It's a competition and you have to use all your knowledge and instruments to have a result.
\n\nExamples? Sure.
\n\nFair competition? Never. Life is unfair. Humble yourself.
\n\nYou want to win - use all you have. Models/Data/Maths/Team/Books.
\n\nDon't devalue all efforts of hundreds and thousands of kagglers wich are investing a lot of time in data analysis and modeling. Don't like - don't participate. But don't be toxic, please. You say it's a shitty competition and with these words, you are ruining hundreds of hours of someone's work because someone stops believe in what he is doing.
\n\nYou can learn something in any competition and even without competition at all. There is no such thing as a meaningless activity but you force people to quit. I can't support you in this, sorry.
\n\nCompetition organizers are not dumb to permit win with scrapping. You don't know how the private part will be scored. Want risk and base your score on scrapping? I wouldn't do so. I would better have a stable model and a stable CV. Will I use scrapped data? Sure. It's just an instrument as any other. Is it cheat? No. As all people are in the same position and all data is shared.
\n\nWhy this post appeared?
\n\nIn the last few days, I've seen that many people were talking that it's unfair to use scrapped data and competition is meaningless. I've seen also that many people are quitting (stormed off). \nDon't invent excuses for your dealings. Decided to quit - go, decided to not participate - ok, decided to stay - also ok. But don't make a storm in a teacup. Respect other's decisions. Don't say something is shitty only because you are not in.
", "messages": [ { "id": 671986, "postDate": "2019-11-13T12:19:00.493Z", "content": "While I agree that there is always difference in skills, resources available and so on, I'l absolutely against competitions, where hosts made mistakes and the way to win competition is to exploit leaks or scrape data.\nThe hosts won't get what they want (good solutions) and this isn't what kaggle is about. Competitions are about using data and finding patterns, in real world you won't find labels in the internet.
", "rawMarkdown": "While I agree that there is always difference in skills, resources available and so on, I'l absolutely against competitions, where hosts made mistakes and the way to win competition is to exploit leaks or scrape data.\nThe hosts won't get what they want (good solutions) and this isn't what kaggle is about. Competitions are about using data and finding patterns, in real world you won't find labels in the internet.", "votes": 26, "replies": [ { "id": 671990, "postDate": "2019-11-13T12:27:33.617Z", "content": "What I have learned during kaggling is that around 50% of the tricks being used here cannot be used in real world.
", "rawMarkdown": "What I have learned during kaggling is that around 50% of the tricks being used here cannot be used in real world.", "votes": 15 }, { "id": 673055, "postDate": "2019-11-14T13:11:15.040Z", "content": "Many people are here for the other 50%
", "rawMarkdown": "Many people are here for the other 50%" }, { "id": 673266, "postDate": "2019-11-14T18:30:06.980Z", "content": "for example?
", "rawMarkdown": "for example? ", "votes": -1 }, { "id": 673422, "postDate": "2019-11-15T00:31:25.210Z", "content": "upvote !
", "rawMarkdown": "upvote !" } ] }, { "id": 671833, "postDate": "2019-11-13T07:56:05Z", "content": "If I knew that test or part of the test is available on the internet, I would never start this competition. I would rather do another competition. When I started, however, this leak did not exist. And now I have already invested time and work. Can you tell me that the other part is not ready for download anywhere? No. That's why I have a bad feeling about this competition. The leak seems to give big boost here. So should I focus on finding leaks to improve my score? I do not much like it.
", "rawMarkdown": "\nIf I knew that test or part of the test is available on the internet, I would never start this competition. I would rather do another competition. When I started, however, this leak did not exist. And now I have already invested time and work. Can you tell me that the other part is not ready for download anywhere? No. That's why I have a bad feeling about this competition. The leak seems to give big boost here. So should I focus on finding leaks to improve my score? I do not much like it.", "votes": 23, "replies": [ { "id": 671853, "postDate": "2019-11-13T08:39:40.753Z", "content": "At least, the way the leak was revealed still makes the competition quite fair currently: @gunesevitan @mpware made the leak public mostly as soon as they discovered it and prepared kernels (this and that) where all the web scrapping was curated for us. Thanks to them and to their honesty, we can all use the leak without much efforts. This would have been totally different if they had kept it secret untill the last days of the competition.
\n\nThe main issue now concerns the leaks which are not public right now. Are there other sites with the target available ?
", "rawMarkdown": "At least, the way the leak was revealed still makes the competition quite fair currently: @gunesevitan @mpware made the leak public mostly as soon as they discovered it and prepared kernels ([this](https://www.kaggle.com/gunesevitan/ashrae-ucf-spider-and-eda-full-test-labels) and [that](https://www.kaggle.com/mpware/ucf-data-limited-leakage)) where all the web scrapping was curated for us. Thanks to them and to their honesty, we can all use the leak without much efforts. This would have been totally different if they had kept it secret untill the last days of the competition.\n\nThe main issue now concerns the leaks which are not public right now. Are there other sites with the target available ?", "votes": 5 }, { "id": 671860, "postDate": "2019-11-13T08:49:09.137Z", "content": "I believe found leaks should be shared ASAP to make it fair for everyone. I would prefer to lose rather than winning with a dirty way. I came here to learn in first place after all...
\n\nand yes, there are other sites with target available.
", "rawMarkdown": "I believe found leaks should be shared ASAP to make it fair for everyone. I would prefer to lose rather than winning with a dirty way. I came here to learn in first place after all...\n\nand yes, there are other sites with target available.", "votes": 16 }, { "id": 671867, "postDate": "2019-11-13T08:54:06.133Z", "content": "\n\n\nthere are other sites with target available.
\n
which sites ?
\n\n@sohier are you already aware of all the sites than can be scraped ? Did the Kaggle team anticipate these leaks ?
", "rawMarkdown": "> there are other sites with target available.\n\nwhich sites ? \n\n@sohier are you already aware of all the sites than can be scraped ? Did the Kaggle team anticipate these leaks ?", "votes": 2 }, { "id": 671880, "postDate": "2019-11-13T09:16:26.823Z", "content": "I think so, they are prepared for all data which can be found publicly being revealed.\nAnd of course in the test set there is some data which is never published.\nThat's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word.\nThere is even one official guy upvoting this post which you can basically see the attitude of those organisers.
", "rawMarkdown": "I think so, they are prepared for all data which can be found publicly being revealed.\nAnd of course in the test set there is some data which is never published.\nThat's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word.\nThere is even one official guy upvoting this post which you can basically see the attitude of those organisers.", "votes": 3 } ] }, { "id": 671965, "postDate": "2019-11-13T11:34:26.253Z", "content": "People are right to be annoyed - it is about the uncertainty that scraping creates. Who has scraped what? How much has team #1 found? Team #2? What if there's a group out there who found it all and is just about to submit? What are Kaggle going to do? When are they going to do it? When will they acknowledge the issue? How much will the competition change? Even if they leave out site 0 as you suggest, what if more public data is discovered? What if it's discovered in the last week?
\n\nThese are not the kind of questions people came here to ponder.
\n\nExternal data rules are not the rules that apply here — external data is supposed to be something that could be gathered in the real world & used for prediction — not just scraped answers.
\n\nScraped answers are worse than useless - they actively devalue the work the sponsor put in to collect data in the first place. (Why would anyone want to mark themselves out as the kind of person who spends their time scraping answers? Gold topics & kernels?)
\n\n\n\n\nDon't like - don't participate.
\n
But competitions don't exist in isolation - compromised competitions create uncertainty about the platform. Winning by peeking at the ground truth is not winning, just cheating. The more this goes on, then the Kaggle gold standard isn't a simple gold standard any more. There are caveats. Specifics. People in general don't have time to remember those specifics - just that they are there. It pulls the whole site down.
\n\nIt can be hard to let things go, and see others pass you on the leaderboard. There will be fear of missing out - but on what? A tainted win? You don't get to decide how you spent last month, but next month is up to you, and what does a sunk cost have to do with that?
\n\nYou may want to check what happened last time someone went too far with external data.
\n\n\n\n\nany evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.
\n
(I feel for Kaggle having to sort this out. It seems like it gets harder over time to run these competitions. The discussion & kernel incentives win out and are too strong to allow any hope that a whole crowd avoids temptation to go searching for a leaderboard shortcut...)
", "rawMarkdown": "People are right to be annoyed - it is about the **uncertainty** that scraping creates. Who has scraped what? How much has team #1 found? Team #2? What if there's a group out there who found it all and is just about to submit? What are Kaggle going to do? When are they going to do it? When will they acknowledge the issue? How much will the competition change? Even if they leave out site 0 as you suggest, what if more public data is discovered? What if it's discovered in the last week?\n\nThese are not the kind of questions people came here to ponder.\n\nExternal data rules are not the rules that apply here — external data is supposed to be something that could be gathered in the real world & used for prediction — not just scraped answers.\n\nScraped answers are *worse* than useless - they actively devalue the work the sponsor put in to collect data in the first place. (Why would anyone want to mark themselves out as the kind of person who spends their time scraping answers? Gold topics & kernels?)\n\n> Don't like - don't participate.\n\nBut competitions don't exist in isolation - compromised competitions create uncertainty about the platform. Winning by peeking at the ground truth is not winning, just cheating. The more this goes on, then the Kaggle gold standard isn't a simple gold standard any more. There are caveats. Specifics. People in general don't have time to remember those specifics - just that they are there. It pulls the whole site down.\n\nIt can be hard to let things go, and see others pass you on the leaderboard. There will be fear of missing out - but on what? A tainted win? You don't get to decide how you spent *last* month, but *next* month is up to you, and what does a [sunk cost][1] have to do with that?\n\nYou may want to check [what happened last time someone went too far with external data][2].\n\n> any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.\n\n(I feel for Kaggle having to sort this out. It seems like it gets harder over time to run these competitions. The discussion & kernel incentives win out and are too strong to allow any hope that a whole crowd avoids temptation to go searching for a leaderboard shortcut...)\n\n [1]: https://en.wikipedia.org/wiki/Sunk_cost\n [2]: https://www.kaggle.com/c/champs-scalar-coupling/discussion/95496\n \n", "votes": 11 }, { "id": 671759, "postDate": "2019-11-13T05:57:22.767Z", "content": "\n\n\nThere is some \"magic\" in data but I didn't find it -> competition is badly prepared and should be canceled.
\n
Uncle CPMP would be triggered on this one :)
\n\nKostia, you have posted a tread describing your way on kaggle. And, according to it, you never was in the situation when real leak has been found. Yes, it is frustrating to invest a lot of your time doing real data science just to see yourself dropping out of the gold-silver zone because people have found a leak.
\n\nBut please don't try to excuse usage of scrapped true labels in the predictive competition.
", "rawMarkdown": "> There is some \"magic\" in data but I didn't find it -> competition is badly prepared and should be canceled.\n\nUncle CPMP would be triggered on this one :)\n\nKostia, you have posted a tread describing your way on kaggle. And, according to it, you never was in the situation when real leak has been found. Yes, it is frustrating to invest a lot of your time doing real data science just to see yourself dropping out of the gold-silver zone because people have found a leak.\n\nBut please don't try to excuse usage of scrapped true labels in the **predictive** competition.", "votes": 9 }, { "id": 671571, "postDate": "2019-11-12T22:14:06.280Z", "content": "Ready to receive downvotes but would like to publish it anyway.
\n\nThere is no such definition as a \"FAIR competition\". It's a competition and you have to use all your knowledge and instruments to have a result.
\n\nExamples? Sure.
\n\nFair competition? Never. Life is unfair. Humble yourself.
\n\nYou want to win - use all you have. Models/Data/Maths/Team/Books.
\n\nDon't devalue all efforts of hundreds and thousands of kagglers wich are investing a lot of time in data analysis and modeling. Don't like - don't participate. But don't be toxic, please. You say it's a shitty competition and with these words, you are ruining hundreds of hours of someone's work because someone stops believe in what he is doing.
\n\nYou can learn something in any competition and even without competition at all. There is no such thing as a meaningless activity but you force people to quit. I can't support you in this, sorry.
\n\nCompetition organizers are not dumb to permit win with scrapping. You don't know how the private part will be scored. Want risk and base your score on scrapping? I wouldn't do so. I would better have a stable model and a stable CV. Will I use scrapped data? Sure. It's just an instrument as any other. Is it cheat? No. As all people are in the same position and all data is shared.
\n\nWhy this post appeared?
\n\nIn the last few days, I've seen that many people were talking that it's unfair to use scrapped data and competition is meaningless. I've seen also that many people are quitting (stormed off). \nDon't invent excuses for your dealings. Decided to quit - go, decided to not participate - ok, decided to stay - also ok. But don't make a storm in a teacup. Respect other's decisions. Don't say something is shitty only because you are not in.
", "rawMarkdown": "Ready to receive downvotes but would like to publish it anyway. \n\nThere is no such definition as a \"FAIR competition\". It's a competition and you have to use all your knowledge and instruments to have a result.\n\nExamples? Sure.\n\n* It's tabular data competition but lgbm doesn't work -> it's unfair to use NN as I don't work with NN.\n* It's a CV competition but I don't have GPU on the local machine -> competition should be canceled.\n* It's NLP competition but I didn't do data augmentation because I'm lazy -> No translations should be permitted.\n* There is some \"magic\" in data but I didn't find it -> competition is badly prepared and should be canceled.\n* There is some strange metric in competition but I can't create my loss function as I'm bad it this -> Metric and labels are inappropriate and competition should be canceled.\n* I have no friends but it's hard to be on top solo - no team should be permitted.\n* ....\n* ....\n\n\nFair competition? Never. Life is unfair. Humble yourself. \n\nYou want to win - use all you have. Models/Data/Maths/Team/Books.\n\nDon't devalue all efforts of hundreds and thousands of kagglers wich are investing a lot of time in data analysis and modeling. Don't like - don't participate. But don't be toxic, please. You say it's a shitty competition and with these words, you are ruining hundreds of hours of someone's work because someone stops believe in what he is doing.\n\n\nYou can learn something in any competition and even without competition at all. There is no such thing as a meaningless activity but you force people to quit. I can't support you in this, sorry.\n\n\nCompetition organizers are not dumb to permit win with scrapping. You don't know how the private part will be scored. Want risk and base your score on scrapping? I wouldn't do so. I would better have a stable model and a stable CV. Will I use scrapped data? Sure. It's just an instrument as any other. Is it cheat? No. As all people are in the same position and all data is shared.\n\n---\n\nWhy this post appeared?\n\nIn the last few days, I've seen that many people were talking that it's unfair to use scrapped data and competition is meaningless. I've seen also that many people are quitting (stormed off). \nDon't invent excuses for your dealings. Decided to quit - go, decided to not participate - ok, decided to stay - also ok. But don't make a storm in a teacup. Respect other's decisions. Don't say something is shitty only because you are not in.", "votes": 17 }, { "id": 671825, "postDate": "2019-11-13T07:36:49.260Z", "content": "\n\n\nBut don't be toxic, please
\n
I would argue this is the most toxic post in this competition. For sure more toxic than asking what is the effect of scraped data or whether the leak will be removed from private test set.
\n\n\n\n\nWill I use scrapped data? Sure.
\n
Then make sure you write a post in the External Data thread, I don't see any post of you or your teammates there ;)\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112841#latest-671680\nAlso, make sure you make all the external data available to all other participants, as per Competition Rules:
\n\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.
A competition with test labels that can be scrapped is definitely a lose-lose case for both the organizers and the participants.
", "rawMarkdown": "A competition with test labels that can be scrapped is definitely a lose-lose case for both the organizers and the participants. ", "votes": 7 }, { "id": 671831, "postDate": "2019-11-13T07:53:13.477Z", "content": "The point is: which skill do you want to develop in your free time? I don't want to spend 60% of my time to search for hidden leak or hidden dataset online and scraping it, but i want to increase my skill in ML and DNN so...
", "rawMarkdown": "The point is: which skill do you want to develop in your free time? I don't want to spend 60% of my time to search for hidden leak or hidden dataset online and scraping it, but i want to increase my skill in ML and DNN so...", "votes": 6 }, { "id": 672158, "postDate": "2019-11-13T15:10:28.083Z", "content": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info here). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.
", "rawMarkdown": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info [here][1]). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.\n\n[1]: https://www.kaggle.com/product-feedback/116079", "votes": 4 }, { "id": 671580, "postDate": "2019-11-12T22:54:23.580Z", "content": "You make some good points above, but seriously: there ARE badly prepared competition - starting with at least two versions of Santander... Yes, people whine and moan (although I find the lgbm / nn example a bit contrived) - so what? Filtering out the noise in the forum, including people leeching for freebies, is part of the package...
", "rawMarkdown": "You make some good points above, but seriously: there ARE badly prepared competition - starting with at least two versions of Santander... Yes, people whine and moan (although I find the lgbm / nn example a bit contrived) - so what? Filtering out the noise in the forum, including people leeching for freebies, is part of the package...", "votes": 4, "replies": [ { "id": 671581, "postDate": "2019-11-12T22:56:59.733Z", "content": "And Microsoft. And ELO. And many others.
\n\nKaggle also gain experience as we all do. I think we need to support each other (not only member-member but also community-host).
\n\nI didn't see any structured proposal how Kaggle can deal with these leaks.\n* Remove Site0 - possible.\n* Private validation on site that never were published online - possible.\n* Better control data in future - possible.\n* ...
\n\nBut I saw many \"LET'S BURN IT!!\".
\n\nWhy we should? I've invested almost month in it. Should people like me drop everything because one site_id gone public (and gave 0.03 on public lb boost)?
", "rawMarkdown": "And Microsoft. And ELO. And many others.\n\nKaggle also gain experience as we all do. I think we need to support each other (not only member-member but also community-host). \n\nI didn't see any structured proposal how Kaggle can deal with these leaks.\n* Remove Site0 - possible.\n* Private validation on site that never were published online - possible.\n* Better control data in future - possible.\n* ...\n\nBut I saw many \"LET'S BURN IT!!\".\n\nWhy we should? I've invested almost month in it. Should people like me drop everything because one site_id gone public (and gave 0.03 on public lb boost)?", "votes": -6 }, { "id": 671586, "postDate": "2019-11-12T23:01:01.430Z", "content": "Admirable goal, but let's be realistic here: on the one hand you have people cutting corners (as evident from the sock puppet purges), on the other - leeches in the forum. If we can establish a baseline of civility, that's already nice.
\n\nExperience or not - leaks in Santander were just embarrasing :-)
", "rawMarkdown": "Admirable goal, but let's be realistic here: on the one hand you have people cutting corners (as evident from the sock puppet purges), on the other - leeches in the forum. If we can establish a baseline of civility, that's already nice. \n\nExperience or not - leaks in Santander were just embarrasing :-)", "votes": 5 }, { "id": 671590, "postDate": "2019-11-12T23:06:06.407Z", "content": "\n\n\nSantander were just embarrasing
\n
Shit happens)) No matter how hard you try - sometimes it just happens.
", "rawMarkdown": "> Santander were just embarrasing\n\nShit happens)) No matter how hard you try - sometimes it just happens.", "votes": -5 } ] }, { "id": 671850, "postDate": "2019-11-13T08:27:56.867Z", "content": "Problem is in expectations. There was less \"fair\" competittions then \"unfair\" for last two years. There is always some trick to use. Still you have to use ML skills to distill the trick into scores. Scrapping is not a trick - if you can find all answers to test set then it is not ML competition. Everyone feels frustrated because of that. Not so much motivating.
\n\nBut you are feel free to continue. \n\"Don't invent excuses for your dealings. Decided to scrap data - go, decided to win at all cost - ok, decided to cheat - also ok. But don't make a storm in a teacup after you will be treated as a web scrapper. Respect other's decisions. Don't say something is beautiful only because you are in.\"
", "rawMarkdown": "Problem is in expectations. There was less \"fair\" competittions then \"unfair\" for last two years. There is always some trick to use. Still you have to use ML skills to distill the trick into scores. Scrapping is not a trick - if you can find all answers to test set then it is not ML competition. Everyone feels frustrated because of that. Not so much motivating.\n\nBut you are feel free to continue. \n\"Don't invent excuses for your dealings. Decided to scrap data - go, decided to win at all cost - ok, decided to cheat - also ok. But don't make a storm in a teacup after you will be treated as a web scrapper. Respect other's decisions. Don't say something is beautiful only because you are in.\"\n", "votes": 3 }, { "id": 672854, "postDate": "2019-11-14T08:26:32.043Z", "content": "I'll go with my grain of salt, because I think this goes beyond this one competition. I haven't taken part in it either by the way, but still think this applies outside of this one competition.\nFirst of all, I kind of agree with @jtrotman that sunken cost shouldn't really matter. Just because someone worked hard, doesn't mean they should be entitled to anything, IMHO. Hard work, if done well, helps increase your odds of reaching the top, but is not, in any way, a guarantee of success. Sometimes, someone can come up with an easy solution without much work and out perform everyone else: as you said, life isn't fair.
\n\nI do agree mostly on your point, that not everyone has the same tools in their hands, and that's how most things in life are. However, I also think we should thrive to make things fairer for everyone. Not everyone can have access to GPUs, that's just how it is (and I'm a student, on the low end of the budget compared to most people here I think, so I'm not in the best place when it comes to having access to good compute). Being smart about some things can help go a long way, and outsmarting raw performance is always satisfying, to me at least.\nNow, I do think there are things where we can try to improve though. Just because some people seem to be unpleased with some aspects, wrongly apparently, doesn't mean there aren't things to fix in Kaggle competitions.
\n\nI think like a lot of things, it's not just black or white. so while I agree with the overall argument you're making, I still don't agree on everything, because I do think we should speak up when some things aren't fair and could be fixed.
\n\nGood topic to bring up though, and I'm very pleased we can have a discussion about it here, it's interesting to see what others from all levels think, and helps also rethink the way I see things.
", "rawMarkdown": "I'll go with my grain of salt, because I think this goes beyond this one competition. I haven't taken part in it either by the way, but still think this applies outside of this one competition.\nFirst of all, I kind of agree with @jtrotman that sunken cost shouldn't really matter. Just because someone worked hard, doesn't mean they should be entitled to anything, IMHO. Hard work, if done well, helps increase your odds of reaching the top, but is not, in any way, a guarantee of success. Sometimes, someone can come up with an easy solution without much work and out perform everyone else: as you said, life isn't fair.\n\nI do agree mostly on your point, that not everyone has the same tools in their hands, and that's how most things in life are. However, I also think we should thrive to make things fairer for everyone. Not everyone can have access to GPUs, that's just how it is (and I'm a student, on the low end of the budget compared to most people here I think, so I'm not in the best place when it comes to having access to good compute). Being smart about some things can help go a long way, and outsmarting raw performance is always satisfying, to me at least.\nNow, I do think there are things where we can try to improve though. Just because some people seem to be unpleased with some aspects, wrongly apparently, doesn't mean there aren't things to fix in Kaggle competitions.\n\nI think like a lot of things, it's not just black or white. so while I agree with the overall argument you're making, I still don't agree on everything, because I do think we should speak up when some things aren't fair and could be fixed. \n\nGood topic to bring up though, and I'm very pleased we can have a discussion about it here, it's interesting to see what others from all levels think, and helps also rethink the way I see things.", "votes": 2, "replies": [ { "id": 673002, "postDate": "2019-11-14T11:30:36.110Z", "content": "Thanks for the @!
\n\nI should have added that I agree with the other points too - there is a lot of unfairness in Kaggle, disparity in hardware, free time. See Jeremy Howard's responses in this twitter thread for example:
\n\n\n\n\nJeremy Howard:\n I'm similar - specifically I look for people that have achieved an unusually high level of capability despite limited opportunities or significant constraints. It's been the best hiring signal over many years and companies for me.
\n
There have been write-ups by people finishing far out of the medals that impressed me anyway. Super lightweight solutions can be very useful, and many gold medal solutions can be entirely impractical. The real skill is knowing what is useful in the real world (but hosts/sponsors do not usually comment on this).
\n\nThe bit I disagree with is saying that scraping answers is just part of the game. It feels unfair to me to try to force people into going along with it when it is so obviously against the original intent of the competition. Even downloading ground truth labels other people have scraped feels very wrong; tedious extra work, zero insight, zero learning, purely to climb the leaderboard, like being the meat puppet pulling the levers in computer-on-computer action.
\n\nAlso - it feels a bit harsh in retrospect when I said respect rational economic theory and just drop it! People don't really think that way. It is something I have to consciously tell myself if I no longer enjoy a particular competition - it can be a struggle to stop.
", "rawMarkdown": "Thanks for the @!\n\nI should have added that I agree with the other points too - there is a lot of unfairness in Kaggle, disparity in hardware, free time. See Jeremy Howard's responses in this [twitter thread][1] for example:\n\n> ***Jeremy Howard:***\n> I'm similar - specifically I look for people that have achieved an unusually high level of capability despite limited opportunities or significant constraints. It's been the best hiring signal over many years and companies for me.\n\nThere have been write-ups by people finishing far out of the medals that impressed me anyway. Super lightweight solutions can be very useful, and many gold medal solutions can be entirely impractical. The real skill is knowing what is useful in the real world (but hosts/sponsors do not usually comment on this).\n\nThe bit I disagree with is saying that scraping answers is just part of the game. It feels unfair to me to try to force people into going along with it when it is so obviously against the original intent of the competition. Even downloading ground truth labels other people have scraped feels very wrong; tedious extra work, zero insight, zero learning, purely to climb the leaderboard, like being the [meat puppet pulling the levers in computer-on-computer action][2].\n\nAlso - it feels a bit harsh in retrospect when I said *respect rational economic theory and just drop it!* People don't really think that way. It is something I have to consciously tell myself if I no longer enjoy a particular competition - it can be a struggle to stop.\n\n [1]: https://twitter.com/jeremyphoward/status/1177495814890909696\n [2]: https://www.theguardian.com/commentisfree/2007/aug/06/comment.comment\n", "votes": 1 }, { "id": 673051, "postDate": "2019-11-14T13:00:51.100Z", "content": "Thanks for the read!\nThat Scrabble story hits home a lot, and is the reason why I usually end up in the bottom in competitions, but have probably learned a lot more than a lot of people.\nIt still is frustrating to see that you've put in a lot of hard work and a lot of the people above you have copied other answers. But if we were making competition with a leader board of who has spend the most time, I wouldn't want to participate, because that also is boring.\nI think I feel pretty close to what you have been saying in this thread, so thanks for the thinking and the links, once again.
", "rawMarkdown": "Thanks for the read!\nThat Scrabble story hits home a lot, and is the reason why I usually end up in the bottom in competitions, but have probably learned a lot more than a lot of people.\nIt still is frustrating to see that you've put in a lot of hard work and a lot of the people above you have copied other answers. But if we were making competition with a leader board of who has spend the most time, I wouldn't want to participate, because that also is boring.\nI think I feel pretty close to what you have been saying in this thread, so thanks for the thinking and the links, once again. ", "votes": 1 } ] }, { "id": 674318, "postDate": "2019-11-16T09:12:57.710Z", "content": "@kyakovlev I'm deeply sorry to hear you have no friends, no GPU, you don't know NN and you're lazy.
\n\nLife is really unfair with you.
", "rawMarkdown": "@kyakovlev I'm deeply sorry to hear you have no friends, no GPU, you don't know NN and you're lazy. \n\nLife is really unfair with you.", "votes": -2, "replies": [ { "id": 678237, "postDate": "2019-11-21T06:11:02.843Z", "content": "I believe he was being overly humble. His track record shows he's none of the above!
", "rawMarkdown": "I believe he was being overly humble. His track record shows he's none of the above!", "votes": 1 } ] }, { "id": 671976, "postDate": "2019-11-13T11:53:14.673Z", "content": "\n\n\n\n\nThere is always some trick to use. \n win at all cost - ok, decided to cheat - also ok.
\n
Win is always at all costs. Cheats - 100% NO - Competition rules are very clear here what we may use and how. \nCheats in my vision:\n* Multiple accounts\n* Private sharings\n* Hide External data\n* Join team and not collaborate
\n\nNot cheats:\n- Resources - CPU/GPU/memory \n- Data cleaning\n- Data enriching\n- Multidisciplinary Teams (As for molecular competition - area experts in team is not cheat)\n- Use public Kernels/Discussions\n- Modeling (Frameworks/architecture)\n- Paper/Book reading (people that don't read well in english are very limited)
\n- External consulting with field experts (that do not participate in kaggle)
\n\n\nthis is the most toxic post in this competition
\n
Agree)) Seems that my english is not good enough to \"transmit my thoughts\". Anyway I said what I've said - it's my opinion. I'm just asking to read deeper than words and see the main idea.
\n\n\n\n\nThe point is: which skill do you want to develop in your free time?
\n
Good question and in my opinion in question you have also a part of the answer.
\n\nDefine your goal:\n* Want learn new things - you don't need to be in medal zone to read and try new things. Medals are just for EGO (beat yours own score and not lb)\n* Want to inference with ds enthusiasts - teamup and share ideas (evolve, learn, teach).\n* Want create new model architecture - Code, share, discuss but in 99% you will not be able to beat sate-of-art as BERT for NLP models. You can contribute to global ML pool. Even small change and improvement is valuable.\n* Want win - ohhh this is other story and for me not the main reason to participate kaggle competitions.\n* It's very rare case when you have all together. Focus on your goal.
\n\n\n\n\nCan you tell me that the other part is not ready for download anywhere?
\n
A month ago I've mentioned that \"When I look at data I see \"not very reach\" features dataset. Seems that we can and have to use a lot of external data.\"
\n\nYes, test labels leakage is worst type of leak but why it's so different of scrapping weather data or historical consumptions for 2014/2015? Scrapping (enriching) data is just an instrument, same as data cleaning.
\n\nI'm hoping that organisers did good analysis and will prevent such leaks in private set.
\n\n\n\n\nThen make sure you write a post in the External Data thread
\n
Be sure all our external sources will be disclosed until \"external deadline\" date. Also not to forget that any link that was posted before by other team/member not needed to be posted again - link duplication is an optional step.
\n\n\n\n\nUncle CPMP would be triggered on this one
\n
I don't remember that @cpmpml asked to cancel competition. Anyway this post was made to ask people be wise and look on each situation from different views. I'm asking not say that \"greenholders\" are miserable if you don't like green colour.
\n\n\n\n\nyou never was in the situation when real leak has been found
\n
True. But read many stories like that here and can feel the pain. Also @cdeotte told me story of Microsoft competition labeling and I'm very sorry for such cases. Can we guarantee that it will not happen again? Sure not. We just need to leave it and keep going.
\n\n\n\n\nThat's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word.
\n
That's the most alarming situation and I hope @claytonmiller @addisonhoward @chrisbalbach @sohier can break silence and clarify leak in the very short future.
\n\nFor any constructive discussions (never mind if you agree with me or not) I'm inviting continue in private messages (linkedin link in my profile).
", "rawMarkdown": "@wispzero \n> There is always some trick to use. \n> win at all cost - ok, decided to cheat - also ok.\n\nWin is always at all costs. Cheats - 100% NO - Competition rules are very clear here what we may use and how. \nCheats in my vision:\n* Multiple accounts\n* Private sharings\n* Hide External data\n* Join team and not collaborate\n\nNot cheats:\n- Resources - CPU/GPU/memory \n- Data cleaning\n- Data enriching\n- Multidisciplinary Teams (As for molecular competition - area experts in team is not cheat)\n- Use public Kernels/Discussions\n- Modeling (Frameworks/architecture)\n- Paper/Book reading (people that don't read well in english are very limited) \n- External consulting with field experts (that do not participate in kaggle)\n\n---\n\n@stecasasso \n> this is the most toxic post in this competition \n\nAgree)) Seems that my english is not good enough to \"transmit my thoughts\". Anyway I said what I've said - it's my opinion. I'm just asking to read deeper than words and see the main idea.\n\n---\n\n@stenford23 \n> The point is: which skill do you want to develop in your free time? \n\nGood question and in my opinion in question you have also a part of the answer.\n\nDefine your goal:\n* Want learn new things - you don't need to be in medal zone to read and try new things. Medals are just for EGO (beat yours own score and not lb)\n* Want to inference with ds enthusiasts - teamup and share ideas (evolve, learn, teach).\n* Want create new model architecture - Code, share, discuss but in 99% you will not be able to beat sate-of-art as BERT for NLP models. You can contribute to global ML pool. Even small change and improvement is valuable.\n* Want win - ohhh this is other story and for me not the main reason to participate kaggle competitions.\n* It's very rare case when you have all together. Focus on your goal.\n\n---\n\n@oleg90777 \n> Can you tell me that the other part is not ready for download anywhere? \n\nA month ago I've mentioned that \"When I look at data I see \"not very reach\" features dataset. Seems that we can and have to use a lot of external data.\"\n\nYes, test labels leakage is worst type of leak but why it's so different of scrapping weather data or historical consumptions for 2014/2015? Scrapping (enriching) data is just an instrument, same as data cleaning. \n\nI'm hoping that organisers did good analysis and will prevent such leaks in private set. \n\n---\n\n@stecasasso \n> Then make sure you write a post in the External Data thread \n\nBe sure all our external sources will be disclosed until \"external deadline\" date. Also not to forget that any link that was posted before by other team/member not needed to be posted again - link duplication is an optional step. \n\n---\n\n@nroman \n> Uncle CPMP would be triggered on this one \n\nI don't remember that @cpmpml asked to cancel competition. Anyway this post was made to ask people be wise and look on each situation from different views. I'm asking not say that \"greenholders\" are miserable if you don't like green colour. \n\n---\n\n@nroman \n> you never was in the situation when real leak has been found \n\nTrue. But read many stories like that here and can feel the pain. Also @cdeotte told me story of Microsoft competition labeling and I'm very sorry for such cases. Can we guarantee that it will not happen again? Sure not. We just need to leave it and keep going.\n\n---\n\n@jielu0728 \n> That's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word. \n\nThat's the most alarming situation and I hope @claytonmiller @addisonhoward @chrisbalbach @sohier can break silence and clarify leak in the very short future.\n\n---\n\nFor any constructive discussions (never mind if you agree with me or not) I'm inviting continue in private messages (linkedin link in my profile).", "votes": -3, "replies": [ { "id": 671981, "postDate": "2019-11-13T12:05:46.213Z", "content": "Yes, you did and i read it. But with \"to use a lot of external data\" i understand something other like to just copy the test data and replace the predictions.
\n\nI also hope that organisiers will prevent such leak in private.
", "rawMarkdown": "Yes, you did and i read it. But with \"to use a lot of external data\" i understand something other like to just copy the test data and replace the predictions. \n\nI also hope that organisiers will prevent such leak in private.", "votes": 5 }, { "id": 717331, "postDate": "2020-01-13T03:22:13.410Z", "content": "Hi @kyakovlev -- thanks for your patience during the competition. We understood it was a bit frustrating, but I think the Kaggle team did a good job of making sure the Private Leaderboard kept its integrity. We definitely realized that there are improvements for the next competitions we may try to host. The end goal was to get the building science and data science communities to understand each other better, and I think we at least accomplished that. I stayed quiet during the competition so as to not throw a wrench in the discussion inadvertently.
", "rawMarkdown": "Hi @kyakovlev -- thanks for your patience during the competition. We understood it was a bit frustrating, but I think the Kaggle team did a good job of making sure the Private Leaderboard kept its integrity. We definitely realized that there are improvements for the next competitions we may try to host. The end goal was to get the building science and data science communities to understand each other better, and I think we at least accomplished that. I stayed quiet during the competition so as to not throw a wrench in the discussion inadvertently.", "votes": 2 } ] }, { "id": 673348, "postDate": "2019-11-14T21:44:33.370Z", "content": "Rules clarification from Kaggle: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345
", "rawMarkdown": "Rules clarification from Kaggle: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345" }, { "id": 673038, "postDate": "2019-11-14T12:40:41.690Z", "content": "I think 78%:22% is reasonable. maybe the leaked data will not affect the private LB. But the public LB is useless for now. \nHaving a rest and waiting for clarification form host is my choice.
", "rawMarkdown": "I think 78%:22% is reasonable. maybe the leaked data will not affect the private LB. But the public LB is useless for now. \nHaving a rest and waiting for clarification form host is my choice." }, { "id": 671617, "postDate": "2019-11-13T00:34:25.010Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 671986, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-11-13T12:19:00.493000", "content": "While I agree that there is always difference in skills, resources available and so on, I'l absolutely against competitions, where hosts made mistakes and the way to win competition is to exploit leaks or scrape data.\nThe hosts won't get what they want (good solutions) and this isn't what kaggle is about. Competitions are about using data and finding patterns, in real world you won't find labels in the internet.
", "votes": 26, "replies": [ { "id": 671990, "author_name": "Roman", "author_url": "", "post_date": "2019-11-13T12:27:33.617000", "content": "What I have learned during kaggling is that around 50% of the tricks being used here cannot be used in real world.
", "votes": 15, "replies": [] }, { "id": 673055, "author_name": "bluetrain", "author_url": "", "post_date": "2019-11-14T13:11:15.040000", "content": "Many people are here for the other 50%
", "votes": 0, "replies": [] }, { "id": 673266, "author_name": "EnvyMyFieldsMedal", "author_url": "", "post_date": "2019-11-14T18:30:06.980000", "content": "for example?
", "votes": -1, "replies": [] }, { "id": 673422, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2019-11-15T00:31:25.210000", "content": "upvote !
", "votes": 0, "replies": [] } ] }, { "id": 671833, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-11-13T07:56:05", "content": "If I knew that test or part of the test is available on the internet, I would never start this competition. I would rather do another competition. When I started, however, this leak did not exist. And now I have already invested time and work. Can you tell me that the other part is not ready for download anywhere? No. That's why I have a bad feeling about this competition. The leak seems to give big boost here. So should I focus on finding leaks to improve my score? I do not much like it.
", "votes": 23, "replies": [ { "id": 671853, "author_name": "FabienDaniel", "author_url": "", "post_date": "2019-11-13T08:39:40.753000", "content": "At least, the way the leak was revealed still makes the competition quite fair currently: @gunesevitan @mpware made the leak public mostly as soon as they discovered it and prepared kernels (this and that) where all the web scrapping was curated for us. Thanks to them and to their honesty, we can all use the leak without much efforts. This would have been totally different if they had kept it secret untill the last days of the competition.
\n\nThe main issue now concerns the leaks which are not public right now. Are there other sites with the target available ?
", "votes": 5, "replies": [] }, { "id": 671860, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2019-11-13T08:49:09.137000", "content": "I believe found leaks should be shared ASAP to make it fair for everyone. I would prefer to lose rather than winning with a dirty way. I came here to learn in first place after all...
\n\nand yes, there are other sites with target available.
", "votes": 16, "replies": [] }, { "id": 671867, "author_name": "FabienDaniel", "author_url": "", "post_date": "2019-11-13T08:54:06.133000", "content": "\n\n\nthere are other sites with target available.
\n
which sites ?
\n\n@sohier are you already aware of all the sites than can be scraped ? Did the Kaggle team anticipate these leaks ?
", "votes": 2, "replies": [] }, { "id": 671880, "author_name": "Jie Lu", "author_url": "", "post_date": "2019-11-13T09:16:26.823000", "content": "I think so, they are prepared for all data which can be found publicly being revealed.\nAnd of course in the test set there is some data which is never published.\nThat's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word.\nThere is even one official guy upvoting this post which you can basically see the attitude of those organisers.
", "votes": 3, "replies": [] } ] }, { "id": 671965, "author_name": "James Trotman", "author_url": "", "post_date": "2019-11-13T11:34:26.253000", "content": "People are right to be annoyed - it is about the uncertainty that scraping creates. Who has scraped what? How much has team #1 found? Team #2? What if there's a group out there who found it all and is just about to submit? What are Kaggle going to do? When are they going to do it? When will they acknowledge the issue? How much will the competition change? Even if they leave out site 0 as you suggest, what if more public data is discovered? What if it's discovered in the last week?
\n\nThese are not the kind of questions people came here to ponder.
\n\nExternal data rules are not the rules that apply here — external data is supposed to be something that could be gathered in the real world & used for prediction — not just scraped answers.
\n\nScraped answers are worse than useless - they actively devalue the work the sponsor put in to collect data in the first place. (Why would anyone want to mark themselves out as the kind of person who spends their time scraping answers? Gold topics & kernels?)
\n\n\n\n\nDon't like - don't participate.
\n
But competitions don't exist in isolation - compromised competitions create uncertainty about the platform. Winning by peeking at the ground truth is not winning, just cheating. The more this goes on, then the Kaggle gold standard isn't a simple gold standard any more. There are caveats. Specifics. People in general don't have time to remember those specifics - just that they are there. It pulls the whole site down.
\n\nIt can be hard to let things go, and see others pass you on the leaderboard. There will be fear of missing out - but on what? A tainted win? You don't get to decide how you spent last month, but next month is up to you, and what does a sunk cost have to do with that?
\n\nYou may want to check what happened last time someone went too far with external data.
\n\n\n\n\nany evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.
\n
(I feel for Kaggle having to sort this out. It seems like it gets harder over time to run these competitions. The discussion & kernel incentives win out and are too strong to allow any hope that a whole crowd avoids temptation to go searching for a leaderboard shortcut...)
", "votes": 11, "replies": [] }, { "id": 671759, "author_name": "Roman", "author_url": "", "post_date": "2019-11-13T05:57:22.767000", "content": "\n\n\nThere is some \"magic\" in data but I didn't find it -> competition is badly prepared and should be canceled.
\n
Uncle CPMP would be triggered on this one :)
\n\nKostia, you have posted a tread describing your way on kaggle. And, according to it, you never was in the situation when real leak has been found. Yes, it is frustrating to invest a lot of your time doing real data science just to see yourself dropping out of the gold-silver zone because people have found a leak.
\n\nBut please don't try to excuse usage of scrapped true labels in the predictive competition.
", "votes": 9, "replies": [] }, { "id": 671825, "author_name": "bluetrain", "author_url": "", "post_date": "2019-11-13T07:36:49.260000", "content": "\n\n\nBut don't be toxic, please
\n
I would argue this is the most toxic post in this competition. For sure more toxic than asking what is the effect of scraped data or whether the leak will be removed from private test set.
\n\n\n\n\nWill I use scrapped data? Sure.
\n
Then make sure you write a post in the External Data thread, I don't see any post of you or your teammates there ;)\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112841#latest-671680\nAlso, make sure you make all the external data available to all other participants, as per Competition Rules:
\n\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.
A competition with test labels that can be scrapped is definitely a lose-lose case for both the organizers and the participants.
", "votes": 7, "replies": [] }, { "id": 671831, "author_name": "Davide Stenner", "author_url": "", "post_date": "2019-11-13T07:53:13.477000", "content": "The point is: which skill do you want to develop in your free time? I don't want to spend 60% of my time to search for hidden leak or hidden dataset online and scraping it, but i want to increase my skill in ML and DNN so...
", "votes": 6, "replies": [] }, { "id": 672158, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-13T15:10:28.083000", "content": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info here). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.
", "votes": 4, "replies": [] }, { "id": 671580, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-11-12T22:54:23.580000", "content": "You make some good points above, but seriously: there ARE badly prepared competition - starting with at least two versions of Santander... Yes, people whine and moan (although I find the lgbm / nn example a bit contrived) - so what? Filtering out the noise in the forum, including people leeching for freebies, is part of the package...
", "votes": 4, "replies": [ { "id": 671581, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2019-11-12T22:56:59.733000", "content": "And Microsoft. And ELO. And many others.
\n\nKaggle also gain experience as we all do. I think we need to support each other (not only member-member but also community-host).
\n\nI didn't see any structured proposal how Kaggle can deal with these leaks.\n* Remove Site0 - possible.\n* Private validation on site that never were published online - possible.\n* Better control data in future - possible.\n* ...
\n\nBut I saw many \"LET'S BURN IT!!\".
\n\nWhy we should? I've invested almost month in it. Should people like me drop everything because one site_id gone public (and gave 0.03 on public lb boost)?
", "votes": -6, "replies": [] }, { "id": 671586, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-11-12T23:01:01.430000", "content": "Admirable goal, but let's be realistic here: on the one hand you have people cutting corners (as evident from the sock puppet purges), on the other - leeches in the forum. If we can establish a baseline of civility, that's already nice.
\n\nExperience or not - leaks in Santander were just embarrasing :-)
", "votes": 5, "replies": [] }, { "id": 671590, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2019-11-12T23:06:06.407000", "content": "\n\n\nSantander were just embarrasing
\n
Shit happens)) No matter how hard you try - sometimes it just happens.
", "votes": -5, "replies": [] } ] }, { "id": 671850, "author_name": "WispZero", "author_url": "", "post_date": "2019-11-13T08:27:56.867000", "content": "Problem is in expectations. There was less \"fair\" competittions then \"unfair\" for last two years. There is always some trick to use. Still you have to use ML skills to distill the trick into scores. Scrapping is not a trick - if you can find all answers to test set then it is not ML competition. Everyone feels frustrated because of that. Not so much motivating.
\n\nBut you are feel free to continue. \n\"Don't invent excuses for your dealings. Decided to scrap data - go, decided to win at all cost - ok, decided to cheat - also ok. But don't make a storm in a teacup after you will be treated as a web scrapper. Respect other's decisions. Don't say something is beautiful only because you are in.\"
", "votes": 3, "replies": [] }, { "id": 672854, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2019-11-14T08:26:32.043000", "content": "I'll go with my grain of salt, because I think this goes beyond this one competition. I haven't taken part in it either by the way, but still think this applies outside of this one competition.\nFirst of all, I kind of agree with @jtrotman that sunken cost shouldn't really matter. Just because someone worked hard, doesn't mean they should be entitled to anything, IMHO. Hard work, if done well, helps increase your odds of reaching the top, but is not, in any way, a guarantee of success. Sometimes, someone can come up with an easy solution without much work and out perform everyone else: as you said, life isn't fair.
\n\nI do agree mostly on your point, that not everyone has the same tools in their hands, and that's how most things in life are. However, I also think we should thrive to make things fairer for everyone. Not everyone can have access to GPUs, that's just how it is (and I'm a student, on the low end of the budget compared to most people here I think, so I'm not in the best place when it comes to having access to good compute). Being smart about some things can help go a long way, and outsmarting raw performance is always satisfying, to me at least.\nNow, I do think there are things where we can try to improve though. Just because some people seem to be unpleased with some aspects, wrongly apparently, doesn't mean there aren't things to fix in Kaggle competitions.
\n\nI think like a lot of things, it's not just black or white. so while I agree with the overall argument you're making, I still don't agree on everything, because I do think we should speak up when some things aren't fair and could be fixed.
\n\nGood topic to bring up though, and I'm very pleased we can have a discussion about it here, it's interesting to see what others from all levels think, and helps also rethink the way I see things.
", "votes": 2, "replies": [ { "id": 673002, "author_name": "James Trotman", "author_url": "", "post_date": "2019-11-14T11:30:36.110000", "content": "Thanks for the @!
\n\nI should have added that I agree with the other points too - there is a lot of unfairness in Kaggle, disparity in hardware, free time. See Jeremy Howard's responses in this twitter thread for example:
\n\n\n\n\nJeremy Howard:\n I'm similar - specifically I look for people that have achieved an unusually high level of capability despite limited opportunities or significant constraints. It's been the best hiring signal over many years and companies for me.
\n
There have been write-ups by people finishing far out of the medals that impressed me anyway. Super lightweight solutions can be very useful, and many gold medal solutions can be entirely impractical. The real skill is knowing what is useful in the real world (but hosts/sponsors do not usually comment on this).
\n\nThe bit I disagree with is saying that scraping answers is just part of the game. It feels unfair to me to try to force people into going along with it when it is so obviously against the original intent of the competition. Even downloading ground truth labels other people have scraped feels very wrong; tedious extra work, zero insight, zero learning, purely to climb the leaderboard, like being the meat puppet pulling the levers in computer-on-computer action.
\n\nAlso - it feels a bit harsh in retrospect when I said respect rational economic theory and just drop it! People don't really think that way. It is something I have to consciously tell myself if I no longer enjoy a particular competition - it can be a struggle to stop.
", "votes": 1, "replies": [] }, { "id": 673051, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2019-11-14T13:00:51.100000", "content": "Thanks for the read!\nThat Scrabble story hits home a lot, and is the reason why I usually end up in the bottom in competitions, but have probably learned a lot more than a lot of people.\nIt still is frustrating to see that you've put in a lot of hard work and a lot of the people above you have copied other answers. But if we were making competition with a leader board of who has spend the most time, I wouldn't want to participate, because that also is boring.\nI think I feel pretty close to what you have been saying in this thread, so thanks for the thinking and the links, once again.
", "votes": 1, "replies": [] } ] }, { "id": 674318, "author_name": "olivier", "author_url": "", "post_date": "2019-11-16T09:12:57.710000", "content": "@kyakovlev I'm deeply sorry to hear you have no friends, no GPU, you don't know NN and you're lazy.
\n\nLife is really unfair with you.
", "votes": -2, "replies": [ { "id": 678237, "author_name": "yukiya", "author_url": "", "post_date": "2019-11-21T06:11:02.843000", "content": "I believe he was being overly humble. His track record shows he's none of the above!
", "votes": 1, "replies": [] } ] }, { "id": 671976, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2019-11-13T11:53:14.673000", "content": "\n\n\n\n\nThere is always some trick to use. \n win at all cost - ok, decided to cheat - also ok.
\n
Win is always at all costs. Cheats - 100% NO - Competition rules are very clear here what we may use and how. \nCheats in my vision:\n* Multiple accounts\n* Private sharings\n* Hide External data\n* Join team and not collaborate
\n\nNot cheats:\n- Resources - CPU/GPU/memory \n- Data cleaning\n- Data enriching\n- Multidisciplinary Teams (As for molecular competition - area experts in team is not cheat)\n- Use public Kernels/Discussions\n- Modeling (Frameworks/architecture)\n- Paper/Book reading (people that don't read well in english are very limited)
\n- External consulting with field experts (that do not participate in kaggle)
\n\n\nthis is the most toxic post in this competition
\n
Agree)) Seems that my english is not good enough to \"transmit my thoughts\". Anyway I said what I've said - it's my opinion. I'm just asking to read deeper than words and see the main idea.
\n\n\n\n\nThe point is: which skill do you want to develop in your free time?
\n
Good question and in my opinion in question you have also a part of the answer.
\n\nDefine your goal:\n* Want learn new things - you don't need to be in medal zone to read and try new things. Medals are just for EGO (beat yours own score and not lb)\n* Want to inference with ds enthusiasts - teamup and share ideas (evolve, learn, teach).\n* Want create new model architecture - Code, share, discuss but in 99% you will not be able to beat sate-of-art as BERT for NLP models. You can contribute to global ML pool. Even small change and improvement is valuable.\n* Want win - ohhh this is other story and for me not the main reason to participate kaggle competitions.\n* It's very rare case when you have all together. Focus on your goal.
\n\n\n\n\nCan you tell me that the other part is not ready for download anywhere?
\n
A month ago I've mentioned that \"When I look at data I see \"not very reach\" features dataset. Seems that we can and have to use a lot of external data.\"
\n\nYes, test labels leakage is worst type of leak but why it's so different of scrapping weather data or historical consumptions for 2014/2015? Scrapping (enriching) data is just an instrument, same as data cleaning.
\n\nI'm hoping that organisers did good analysis and will prevent such leaks in private set.
\n\n\n\n\nThen make sure you write a post in the External Data thread
\n
Be sure all our external sources will be disclosed until \"external deadline\" date. Also not to forget that any link that was posted before by other team/member not needed to be posted again - link duplication is an optional step.
\n\n\n\n\nUncle CPMP would be triggered on this one
\n
I don't remember that @cpmpml asked to cancel competition. Anyway this post was made to ask people be wise and look on each situation from different views. I'm asking not say that \"greenholders\" are miserable if you don't like green colour.
\n\n\n\n\nyou never was in the situation when real leak has been found
\n
True. But read many stories like that here and can feel the pain. Also @cdeotte told me story of Microsoft competition labeling and I'm very sorry for such cases. Can we guarantee that it will not happen again? Sure not. We just need to leave it and keep going.
\n\n\n\n\nThat's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word.
\n
That's the most alarming situation and I hope @claytonmiller @addisonhoward @chrisbalbach @sohier can break silence and clarify leak in the very short future.
\n\nFor any constructive discussions (never mind if you agree with me or not) I'm inviting continue in private messages (linkedin link in my profile).
", "votes": -3, "replies": [ { "id": 671981, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-11-13T12:05:46.213000", "content": "Yes, you did and i read it. But with \"to use a lot of external data\" i understand something other like to just copy the test data and replace the predictions.
\n\nI also hope that organisiers will prevent such leak in private.
", "votes": 5, "replies": [] }, { "id": 717331, "author_name": "Clayton Miller", "author_url": "", "post_date": "2020-01-13T03:22:13.410000", "content": "Hi @kyakovlev -- thanks for your patience during the competition. We understood it was a bit frustrating, but I think the Kaggle team did a good job of making sure the Private Leaderboard kept its integrity. We definitely realized that there are improvements for the next competitions we may try to host. The end goal was to get the building science and data science communities to understand each other better, and I think we at least accomplished that. I stayed quiet during the competition so as to not throw a wrench in the discussion inadvertently.
", "votes": 2, "replies": [] } ] }, { "id": 673348, "author_name": "Vishal Sharma", "author_url": "", "post_date": "2019-11-14T21:44:33.370000", "content": "Rules clarification from Kaggle: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345
", "votes": 0, "replies": [] }, { "id": 673038, "author_name": "Johnny Lee", "author_url": "", "post_date": "2019-11-14T12:40:41.690000", "content": "I think 78%:22% is reasonable. maybe the leaked data will not affect the private LB. But the public LB is useless for now. \nHaving a rest and waiting for clarification form host is my choice.
", "votes": 0, "replies": [] }, { "id": 671617, "author_name": "", "author_url": "", "post_date": "2019-11-13T00:34:25.010000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "671986": "While I agree that there is always difference in skills, resources available and so on, I'l absolutely against competitions, where hosts made mistakes and the way to win competition is to exploit leaks or scrape data.\nThe hosts won't get what they want (good solutions) and this isn't what kaggle is about. Competitions are about using data and finding patterns, in real world you won't find labels in the internet.", "671833": "\nIf I knew that test or part of the test is available on the internet, I would never start this competition. I would rather do another competition. When I started, however, this leak did not exist. And now I have already invested time and work. Can you tell me that the other part is not ready for download anywhere? No. That's why I have a bad feeling about this competition. The leak seems to give big boost here. So should I focus on finding leaks to improve my score? I do not much like it.", "671965": "People are right to be annoyed - it is about the **uncertainty** that scraping creates. Who has scraped what? How much has team #1 found? Team #2? What if there's a group out there who found it all and is just about to submit? What are Kaggle going to do? When are they going to do it? When will they acknowledge the issue? How much will the competition change? Even if they leave out site 0 as you suggest, what if more public data is discovered? What if it's discovered in the last week?\n\nThese are not the kind of questions people came here to ponder.\n\nExternal data rules are not the rules that apply here — external data is supposed to be something that could be gathered in the real world & used for prediction — not just scraped answers.\n\nScraped answers are *worse* than useless - they actively devalue the work the sponsor put in to collect data in the first place. (Why would anyone want to mark themselves out as the kind of person who spends their time scraping answers? Gold topics & kernels?)\n\n> Don't like - don't participate.\n\nBut competitions don't exist in isolation - compromised competitions create uncertainty about the platform. Winning by peeking at the ground truth is not winning, just cheating. The more this goes on, then the Kaggle gold standard isn't a simple gold standard any more. There are caveats. Specifics. People in general don't have time to remember those specifics - just that they are there. It pulls the whole site down.\n\nIt can be hard to let things go, and see others pass you on the leaderboard. There will be fear of missing out - but on what? A tainted win? You don't get to decide how you spent *last* month, but *next* month is up to you, and what does a [sunk cost][1] have to do with that?\n\nYou may want to check [what happened last time someone went too far with external data][2].\n\n> any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.\n\n(I feel for Kaggle having to sort this out. It seems like it gets harder over time to run these competitions. The discussion & kernel incentives win out and are too strong to allow any hope that a whole crowd avoids temptation to go searching for a leaderboard shortcut...)\n\n [1]: https://en.wikipedia.org/wiki/Sunk_cost\n [2]: https://www.kaggle.com/c/champs-scalar-coupling/discussion/95496\n \n", "671759": "> There is some \"magic\" in data but I didn't find it -> competition is badly prepared and should be canceled.\n\nUncle CPMP would be triggered on this one :)\n\nKostia, you have posted a tread describing your way on kaggle. And, according to it, you never was in the situation when real leak has been found. Yes, it is frustrating to invest a lot of your time doing real data science just to see yourself dropping out of the gold-silver zone because people have found a leak.\n\nBut please don't try to excuse usage of scrapped true labels in the **predictive** competition.", "671571": "Ready to receive downvotes but would like to publish it anyway. \n\nThere is no such definition as a \"FAIR competition\". It's a competition and you have to use all your knowledge and instruments to have a result.\n\nExamples? Sure.\n\n* It's tabular data competition but lgbm doesn't work -> it's unfair to use NN as I don't work with NN.\n* It's a CV competition but I don't have GPU on the local machine -> competition should be canceled.\n* It's NLP competition but I didn't do data augmentation because I'm lazy -> No translations should be permitted.\n* There is some \"magic\" in data but I didn't find it -> competition is badly prepared and should be canceled.\n* There is some strange metric in competition but I can't create my loss function as I'm bad it this -> Metric and labels are inappropriate and competition should be canceled.\n* I have no friends but it's hard to be on top solo - no team should be permitted.\n* ....\n* ....\n\n\nFair competition? Never. Life is unfair. Humble yourself. \n\nYou want to win - use all you have. Models/Data/Maths/Team/Books.\n\nDon't devalue all efforts of hundreds and thousands of kagglers wich are investing a lot of time in data analysis and modeling. Don't like - don't participate. But don't be toxic, please. You say it's a shitty competition and with these words, you are ruining hundreds of hours of someone's work because someone stops believe in what he is doing.\n\n\nYou can learn something in any competition and even without competition at all. There is no such thing as a meaningless activity but you force people to quit. I can't support you in this, sorry.\n\n\nCompetition organizers are not dumb to permit win with scrapping. You don't know how the private part will be scored. Want risk and base your score on scrapping? I wouldn't do so. I would better have a stable model and a stable CV. Will I use scrapped data? Sure. It's just an instrument as any other. Is it cheat? No. As all people are in the same position and all data is shared.\n\n---\n\nWhy this post appeared?\n\nIn the last few days, I've seen that many people were talking that it's unfair to use scrapped data and competition is meaningless. I've seen also that many people are quitting (stormed off). \nDon't invent excuses for your dealings. Decided to quit - go, decided to not participate - ok, decided to stay - also ok. But don't make a storm in a teacup. Respect other's decisions. Don't say something is shitty only because you are not in.", "671825": "> But don't be toxic, please \n\nI would argue this is the most toxic post in this competition. For sure more toxic than asking what is the effect of scraped data or whether the leak will be removed from private test set.\n\n > Will I use scrapped data? Sure. \n\nThen make sure you write a post in the External Data thread, I don't see any post of you or your teammates there ;)\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112841#latest-671680\nAlso, make sure you make **all** the external data available to all other participants, as per Competition Rules: \n\n`C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your models and Submissions. However, you will (i) ensure the External Data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the External Data for the participants to the official competition forum prior to the Entry Deadline.`", "674872": "A competition with test labels that can be scrapped is definitely a lose-lose case for both the organizers and the participants. ", "671831": "The point is: which skill do you want to develop in your free time? I don't want to spend 60% of my time to search for hidden leak or hidden dataset online and scraping it, but i want to increase my skill in ML and DNN so...", "672158": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info [here][1]). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.\n\n[1]: https://www.kaggle.com/product-feedback/116079", "671580": "You make some good points above, but seriously: there ARE badly prepared competition - starting with at least two versions of Santander... Yes, people whine and moan (although I find the lgbm / nn example a bit contrived) - so what? Filtering out the noise in the forum, including people leeching for freebies, is part of the package...", "671850": "Problem is in expectations. There was less \"fair\" competittions then \"unfair\" for last two years. There is always some trick to use. Still you have to use ML skills to distill the trick into scores. Scrapping is not a trick - if you can find all answers to test set then it is not ML competition. Everyone feels frustrated because of that. Not so much motivating.\n\nBut you are feel free to continue. \n\"Don't invent excuses for your dealings. Decided to scrap data - go, decided to win at all cost - ok, decided to cheat - also ok. But don't make a storm in a teacup after you will be treated as a web scrapper. Respect other's decisions. Don't say something is beautiful only because you are in.\"\n", "672854": "I'll go with my grain of salt, because I think this goes beyond this one competition. I haven't taken part in it either by the way, but still think this applies outside of this one competition.\nFirst of all, I kind of agree with @jtrotman that sunken cost shouldn't really matter. Just because someone worked hard, doesn't mean they should be entitled to anything, IMHO. Hard work, if done well, helps increase your odds of reaching the top, but is not, in any way, a guarantee of success. Sometimes, someone can come up with an easy solution without much work and out perform everyone else: as you said, life isn't fair.\n\nI do agree mostly on your point, that not everyone has the same tools in their hands, and that's how most things in life are. However, I also think we should thrive to make things fairer for everyone. Not everyone can have access to GPUs, that's just how it is (and I'm a student, on the low end of the budget compared to most people here I think, so I'm not in the best place when it comes to having access to good compute). Being smart about some things can help go a long way, and outsmarting raw performance is always satisfying, to me at least.\nNow, I do think there are things where we can try to improve though. Just because some people seem to be unpleased with some aspects, wrongly apparently, doesn't mean there aren't things to fix in Kaggle competitions.\n\nI think like a lot of things, it's not just black or white. so while I agree with the overall argument you're making, I still don't agree on everything, because I do think we should speak up when some things aren't fair and could be fixed. \n\nGood topic to bring up though, and I'm very pleased we can have a discussion about it here, it's interesting to see what others from all levels think, and helps also rethink the way I see things.", "674318": "@kyakovlev I'm deeply sorry to hear you have no friends, no GPU, you don't know NN and you're lazy. \n\nLife is really unfair with you.", "671976": "@wispzero \n> There is always some trick to use. \n> win at all cost - ok, decided to cheat - also ok.\n\nWin is always at all costs. Cheats - 100% NO - Competition rules are very clear here what we may use and how. \nCheats in my vision:\n* Multiple accounts\n* Private sharings\n* Hide External data\n* Join team and not collaborate\n\nNot cheats:\n- Resources - CPU/GPU/memory \n- Data cleaning\n- Data enriching\n- Multidisciplinary Teams (As for molecular competition - area experts in team is not cheat)\n- Use public Kernels/Discussions\n- Modeling (Frameworks/architecture)\n- Paper/Book reading (people that don't read well in english are very limited) \n- External consulting with field experts (that do not participate in kaggle)\n\n---\n\n@stecasasso \n> this is the most toxic post in this competition \n\nAgree)) Seems that my english is not good enough to \"transmit my thoughts\". Anyway I said what I've said - it's my opinion. I'm just asking to read deeper than words and see the main idea.\n\n---\n\n@stenford23 \n> The point is: which skill do you want to develop in your free time? \n\nGood question and in my opinion in question you have also a part of the answer.\n\nDefine your goal:\n* Want learn new things - you don't need to be in medal zone to read and try new things. Medals are just for EGO (beat yours own score and not lb)\n* Want to inference with ds enthusiasts - teamup and share ideas (evolve, learn, teach).\n* Want create new model architecture - Code, share, discuss but in 99% you will not be able to beat sate-of-art as BERT for NLP models. You can contribute to global ML pool. Even small change and improvement is valuable.\n* Want win - ohhh this is other story and for me not the main reason to participate kaggle competitions.\n* It's very rare case when you have all together. Focus on your goal.\n\n---\n\n@oleg90777 \n> Can you tell me that the other part is not ready for download anywhere? \n\nA month ago I've mentioned that \"When I look at data I see \"not very reach\" features dataset. Seems that we can and have to use a lot of external data.\"\n\nYes, test labels leakage is worst type of leak but why it's so different of scrapping weather data or historical consumptions for 2014/2015? Scrapping (enriching) data is just an instrument, same as data cleaning. \n\nI'm hoping that organisers did good analysis and will prevent such leaks in private set. \n\n---\n\n@stecasasso \n> Then make sure you write a post in the External Data thread \n\nBe sure all our external sources will be disclosed until \"external deadline\" date. Also not to forget that any link that was posted before by other team/member not needed to be posted again - link duplication is an optional step. \n\n---\n\n@nroman \n> Uncle CPMP would be triggered on this one \n\nI don't remember that @cpmpml asked to cancel competition. Anyway this post was made to ask people be wise and look on each situation from different views. I'm asking not say that \"greenholders\" are miserable if you don't like green colour. \n\n---\n\n@nroman \n> you never was in the situation when real leak has been found \n\nTrue. But read many stories like that here and can feel the pain. Also @cdeotte told me story of Microsoft competition labeling and I'm very sorry for such cases. Can we guarantee that it will not happen again? Sure not. We just need to leave it and keep going.\n\n---\n\n@jielu0728 \n> That's why after demanding clarification of this leak for 3 days and yet not a single organiser has stood out to say a word. \n\nThat's the most alarming situation and I hope @claytonmiller @addisonhoward @chrisbalbach @sohier can break silence and clarify leak in the very short future.\n\n---\n\nFor any constructive discussions (never mind if you agree with me or not) I'm inviting continue in private messages (linkedin link in my profile).", "673348": "Rules clarification from Kaggle: \nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345", "673038": "I think 78%:22% is reasonable. maybe the leaked data will not affect the private LB. But the public LB is useless for now. \nHaving a rest and waiting for clarification form host is my choice.", "671617": "" } }