![\"There](\"https://memegenerator.net/img/instances/49278257/this-is-not-nam-there-are-rules.jpg\")
For the last year, I have been focusing on bringing more scientific challenges to Kaggle. I personally believe it's a benefit to both our community as well as the scientific community to host these different types of problems.
\n\nScientific problems pose a different set of challenges, and we'll use this competition as an example. The obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.
\n\nBut what if a researcher wants to run a Kaggle competition for this problem? We know that some properties of the QM7 and QM9 molecules have been published, that they are expected to provide minimal-to-no benefit to the target for this competition, and using this data would be completely against the spirit of the scientific challenge. It would seem a prudent course of action to exclude this data from the competition.
\n\nThat simple request was violated before we were even able to have a discussion about it on the forums. It is already extremely difficult to find practical scientific challenges that can't be abused via external data. And, unfortunately, there are some community members who feel entitled to abuse the rules \"to make things fair\".
\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.
\n\nTo address this current competition, we are updating the rules to allow the use of QM7 and QM9 published data, if participants wish to use it. Participants still may not use quantum calculations to obtain predictions.
\n\nWith that said, I can't stress enough how wrong it is to break competition rules because you disagree with them or you want to force a particular course of action. Because this is such an important point, even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.
", "messages": [ { "id": 551457, "postDate": "2019-06-12T18:37:36.400Z", "content": "For the last year, I have been focusing on bringing more scientific challenges to Kaggle. I personally believe it's a benefit to both our community as well as the scientific community to host these different types of problems.
\n\nScientific problems pose a different set of challenges, and we'll use this competition as an example. The obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.
\n\nBut what if a researcher wants to run a Kaggle competition for this problem? We know that some properties of the QM7 and QM9 molecules have been published, that they are expected to provide minimal-to-no benefit to the target for this competition, and using this data would be completely against the spirit of the scientific challenge. It would seem a prudent course of action to exclude this data from the competition.
\n\nThat simple request was violated before we were even able to have a discussion about it on the forums. It is already extremely difficult to find practical scientific challenges that can't be abused via external data. And, unfortunately, there are some community members who feel entitled to abuse the rules \"to make things fair\".
\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.
\n\nTo address this current competition, we are updating the rules to allow the use of QM7 and QM9 published data, if participants wish to use it. Participants still may not use quantum calculations to obtain predictions.
\n\nWith that said, I can't stress enough how wrong it is to break competition rules because you disagree with them or you want to force a particular course of action. Because this is such an important point, even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.
", "rawMarkdown": "For the last year, I have been focusing on bringing more scientific challenges to Kaggle. I personally believe it's a benefit to both our community as well as the scientific community to host these different types of problems.\n\nScientific problems pose a different set of challenges, and we'll use this competition as an example. The obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.\n\nBut what if a researcher wants to run a Kaggle competition for this problem? We know that some properties of the QM7 and QM9 molecules have been published, that they are expected to provide minimal-to-no benefit to the target for this competition, and using this data would be completely against the spirit of the scientific challenge. It would seem a prudent course of action to exclude this data from the competition.\n\nThat simple request was violated before we were even able to have a discussion about it on the forums. It is already extremely difficult to find practical scientific challenges that can't be abused via external data. And, unfortunately, there are some community members who feel entitled to abuse the rules \"to make things fair\". \n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.\n\nTo address this current competition, *we are updating the rules to allow the use of QM7 and QM9 published data*, if participants wish to use it. Participants still may ***not*** use quantum calculations to obtain predictions.\n\nWith that said, I can't stress enough how wrong it is to break competition rules because you disagree with them or you want to force a particular course of action. Because this is such an important point, even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission *before* this rule change will result in that individual's (or team's) removal from the competition once it has closed.", "votes": 97 }, { "id": 558482, "postDate": "2019-06-22T12:02:59.237Z", "content": "A couple of points
\n\n1st of all - cheating is irrelevant to the scientific competitions - it happens everywhere and in all competitions, so you should not feel bad about this @inversion . This is not exclusive to scientific competitions. Private sharing is off the charge in every competition and you can definitely see patterns on the leaderboards.
\n\n2nd - No matter how good your system is in detecting violations, you will never be able to capture them all when the system is so broad, fluid and open in using external data, linking data-sets, copying the works of others. Essentially you welcome this behaviour.
\n\n3rd - something that I was afraid of since the beginning when kaggle introduced GMs from kernels and discussions is that people are gamifying the system to get the rankings at the cost of the competitions' credibility. This quite often leads to oversharing, people releasing high scoring kernel at the end of competitions (because they know they cannot get the competitions' kaggle medals, they aim for discussion/kernel medals). I need to highlight that I don't discredit the value of GMs in kernels and discussions, especially when they achieved this naturally without gamifying the system . In my opinion they are more valuable than competitions' GMs to the kaggle community. But gamification has changed things.
\n\n4th. Once its out there... then its out there! e.g once you share a \"forbidden\" dataset and people use it, all become co-conspirators. To properly assess who was using the dataset before and after is not feasible (at maximum accuracy).
\n\n5th- the rules are quite often NOT clear and you might have to read 5-10 different threads and discussions to be up to date with what is allowed and what's not. In the recent Sound classification competition I was using imagenet until before the merge deadline and only then I realised that it was not allowed (and i was actually lucky to find that). The rules (or their interpretations) are scattered everywhere!
6th- Changing the rules midway is never a good strategy. This reinforces the behaviour of cheating and I appreciate you say that people will be removed/banned, but I think this cannot happen optimally (IMO), nor I think we should fully blame the person who shared this. Kaggle is partly to blame for that.
\n\nBased on these, I have a few suggestions. I am not saying these are the perfect solutions to these problems (and some may sound a bit rough), but at least can be food for thought.
\n\n1) Less ranking points (in all competitions, discussions, kernels) for more \"dangerous\" competitions (as in prone to be hacked with lots of external information about them) . \n2) Some solo (or not) competitions with disabled kernels/discussions, disabling external data. This can make things clearer of who is doing what. Generally discussions and kernels should be blocked something like 14 days before the competition ends .\n3) EXTRA Points' penalties for cheating (not only the ones you lose if you get removed)- this will force people to be more careful. Maybe ban/block accounts for certain days if they do it consistently (like yellow card, then red card). \n4) Give clear examples of what is allowed and what is not allowed in the competitions' rules so that it is easy for people to follow. Like for example , \"you cannot use this, but you could use that \". You have done this in a few competitions as bullet points , but in my opinion, it needs to be more thorough. \n5) There needs to be some form of regulation on these private forums that keep sharing stuff (some quite large). I know it cannot be perfect, but every little helps. Also there should be people ( even volunteers) that take away what is being discussed there and bringing it into kaggle to make it fair for everybody. Maybe kaggle can give some extra prizes for these behaviours an an extra incentive?\n6) More competitions with fully annonymized data and descriptions.\n7) If competitions are very prone to cheating, maybe consider making them Masters' only?\n8) less competitions where data is public (or more can be found easily) .
\n\nAnd a few closing thoughts:
\n\nKudos to kaggle for creating a strong brand around its ranking system. Violations happen, because the ranks are valuable. I think kaggle needs to continue protecting what it has built. To be fair, even if the evaluation system has flaws (and is prone to cheating), it is much better than anything else I 've seen out there trying to do something similar. What I am trying to say is that this is not an easy task.
\n\nI think that kaggle historically has done well to adapt to new situations and address them, so I am sure they will come up with a better formula around this.
", "rawMarkdown": "A couple of points\n\n1st of all - cheating is irrelevant to the scientific competitions - it happens everywhere and in all competitions, so you should not feel bad about this @inversion . This is not exclusive to scientific competitions. Private sharing is off the charge in every competition and you can definitely see patterns on the leaderboards. \n\n2nd - No matter how good your system is in detecting violations, you will never be able to capture them all when the system is so broad, fluid and open in using external data, linking data-sets, copying the works of others. Essentially you welcome this behaviour. \n\n3rd - something that I was afraid of since the beginning when kaggle introduced GMs from kernels and discussions is that people are gamifying the system to get the rankings at the cost of the competitions' credibility. This quite often leads to oversharing, people releasing high scoring kernel at the end of competitions (because they know they cannot get the competitions' kaggle medals, they aim for discussion/kernel medals). *I need to highlight* that I don't discredit the value of GMs in kernels and discussions, especially when they achieved this naturally without gamifying the system . In my opinion they are more valuable than competitions' GMs to the kaggle community. But gamification has changed things. \n\n4th. Once its out there... then its out there! e.g once you share a \"forbidden\" dataset and people use it, all become co-conspirators. To properly assess who was using the dataset before and after is not feasible (at maximum accuracy). \n\n5th- the rules are quite often NOT clear and you might have to read 5-10 different threads and discussions to be up to date with what is allowed and what's not. In the recent *Sound classification competition* I was using `imagenet` until before the merge deadline and only then I realised that it was not allowed (and i was actually lucky to find that). The rules (or their interpretations) are scattered everywhere! \n\n6th- Changing the rules midway is never a good strategy. This reinforces the behaviour of cheating and I appreciate you say that people will be removed/banned, but I think this cannot happen optimally (IMO), nor I think we should fully blame the person who shared this. Kaggle is partly to blame for that. \n\nBased on these, I have a few suggestions. I am not saying these are the perfect solutions to these problems (and some may sound a bit rough), but at least can be food for thought. \n\n1) Less ranking points (in all competitions, discussions, kernels) for more \"dangerous\" competitions (as in prone to be hacked with lots of external information about them) . \n2) Some solo (or not) competitions with disabled kernels/discussions, disabling external data. This can make things clearer of who is doing what. Generally discussions and kernels should be blocked something like 14 days before the competition ends .\n3) EXTRA Points' penalties for cheating (not only the ones you lose if you get removed)- this will force people to be more careful. Maybe ban/block accounts for certain days if they do it consistently (like yellow card, then red card). \n4) Give clear examples of what is allowed and what is not allowed in the competitions' rules so that it is easy for people to follow. Like for example , \"you cannot use this, but you could use that \". You have done this in a few competitions as bullet points , but in my opinion, it needs to be more thorough. \n5) There needs to be some form of regulation on these private forums that keep sharing stuff (some quite large). I know it cannot be perfect, but every little helps. Also there should be people ( even volunteers) that take away what is being discussed there and bringing it into kaggle to make it fair for everybody. Maybe kaggle can give some extra prizes for these behaviours an an extra incentive?\n6) More competitions with fully annonymized data and descriptions.\n7) If competitions are very prone to cheating, maybe consider making them Masters' only?\n8) less competitions where data is public (or more can be found easily) .\n\nAnd a few closing thoughts: \n\nKudos to kaggle for creating a strong brand around its ranking system. Violations happen, because the ranks are valuable. I think kaggle needs to continue protecting what it has built. To be fair, even if the evaluation system has flaws (and is prone to cheating), it is much better than anything else I 've seen out there trying to do something similar. What I am trying to say is that this is not an easy task. \n\nI think that kaggle historically has done well to adapt to new situations and address them, so I am sure they will come up with a better formula around this. ", "votes": 36, "replies": [ { "id": 558578, "postDate": "2019-06-22T15:32:43.607Z", "content": "Good post, Μαριος Μιχαηλιδης. I like idea \"6) More competitions with fully anonymized data and descriptions.\". I wonder if it would have been possible to do that for this competition: to so disguise the data that the participants wouldn't even know it was a chemistry problem.
", "rawMarkdown": "Good post, Μαριος Μιχαηλιδης. I like idea \"6) More competitions with fully anonymized data and descriptions.\". I wonder if it would have been possible to do that for this competition: to so disguise the data that the participants wouldn't even know it was a chemistry problem.", "votes": 5 }, { "id": 558582, "postDate": "2019-06-22T15:38:34.737Z", "content": "4) Give clear examples of what is allowed [...]
\n\nAgree. On Kaggle it need not be authorative and fancy -- clear thorough explanations will suffice.
\n\n7) [...] making them Masters' only?
\n\nHow about having three rankings for all competitions? 1) Public, 2) Masters' and 3) Private?
", "rawMarkdown": "*4) Give clear examples of what is allowed [...]*\n\nAgree. On Kaggle it need not be authorative and fancy -- clear thorough explanations will suffice.\n\n*7) [...] making them Masters' only?*\n\nHow about having three rankings for all competitions? 1) Public, 2) Masters' and 3) Private?", "votes": 2 }, { "id": 558856, "postDate": "2019-06-23T03:37:32.700Z", "content": "I like the 14 days suggestion - 2) above. Or may be allow 50% duration of competition time for kernel sharing, so that both kernel enthusiasts and competetion enthusiasts have fair chance.
", "rawMarkdown": "I like the 14 days suggestion - 2) above. Or may be allow 50% duration of competition time for kernel sharing, so that both kernel enthusiasts and competetion enthusiasts have fair chance.", "votes": 1 }, { "id": 558861, "postDate": "2019-06-23T03:50:31.833Z", "content": "Agree with you @KazAnova , adding to what you have mentioned . few suggestions from my end.
1. there should be a kernels deadline let us say a month before the final submission dead line .
And after this date all new kernels will become private and people shouldnt be able to mark public.
After the competition is over kernels can be marked public.
2. Competitions where cheating can be possible with external data , split the competition in two. First one a strict one with no external data and make top x number from this LB eligible for second competition where external data can be allowed .
3. Is it possible to make a strict rule that external data has to be disclosed atleast 3 weeks before the end date ?cases where data is collected by scraping or other means they should upload the data. either them or kaggle make it public here.
@dslate You are absolutely right in this case. It's impossible to both have anonymous molecules and provide their structure.
", "rawMarkdown": "@dslate You are absolutely right in this case. It's impossible to both have anonymous molecules _and_ provide their structure.", "votes": 1 } ] }, { "id": 551501, "postDate": "2019-06-12T19:46:03.403Z", "content": "\n\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.
\n
This really pisses me off to no end. I was really hoping to see more relevant and competitive science challenges on Kaggle. In addition to all the other high-level cheating that has become rampant on Kaggle, I almost feel like giving up on competitions altogether. \nHere is just one more suggestion about how to deal with this going forward: create invitation-only Kaggle challenges. Invite only Kagglers with no history of impropriety or appearance of impropriety.
", "rawMarkdown": "> The unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.\n\nThis really pisses me off to no end. I was really hoping to see more relevant and competitive science challenges on Kaggle. In addition to all the other high-level cheating that has become rampant on Kaggle, I almost feel like giving up on competitions altogether. \nHere is just one more suggestion about how to deal with this going forward: create invitation-only Kaggle challenges. Invite only Kagglers with no history of impropriety or appearance of impropriety. ", "votes": 27, "replies": [ { "id": 551761, "postDate": "2019-06-13T04:55:54.983Z", "content": "Kaggle Social Credit Score maybe is needed. \nRespect to @inversion for giving so much oxygen to this, but for a scientific competition caving in to blatant violations of the rules is disappointing. The use of QM7, QM9 or any other dataset also has to consider if that is allowed public use. Some datasets or software may allow private research or non commercial use, but one objective of the competition - to be able to use for developing medicines, would have to consider public license/copyright.
\nBelow Two Sigma competition was mentioned, it included in the rules:
By entering a Submission, you represent and warrant that … your Submission:\n\n does not contain confidential information or trade secrets and is not\n the subject of a registered patent or pending patent application\n does not trigger any reporting or royalty obligation to any third party;\nA breach of any of these warranties will result in the corresponding Submission being invalid in addition to any other remedies available to Competition Sponsor.
\n\nPerhaps in future competitions have to include similar caveats, litigation may be the only way for rules to be understood and accepted.
", "rawMarkdown": "Kaggle Social Credit Score maybe is needed. \nRespect to @inversion for giving so much oxygen to this, but for a scientific competition caving in to blatant violations of the rules is disappointing. The use of QM7, QM9 or any other dataset also has to consider if that is allowed public use. Some datasets or software may allow private research or non commercial use, but one objective of the competition - to be able to use for developing medicines, would have to consider public license/copyright. \nBelow Two Sigma competition was mentioned, it included in the rules: \n\n By entering a Submission, you represent and warrant that … your Submission:\n\n does not contain confidential information or trade secrets and is not\n the subject of a registered patent or pending patent application\n does not trigger any reporting or royalty obligation to any third party;\n\nA breach of any of these warranties will result in the corresponding Submission being invalid in addition to any other remedies available to Competition Sponsor.\n\nPerhaps in future competitions have to include similar caveats, litigation may be the only way for rules to be understood and accepted. ", "votes": 2 }, { "id": 554642, "postDate": "2019-06-17T20:05:43.850Z", "content": "I can't agree more. iNaturalist 2019 at FGVC6 is my first try in Kaggle, but when I looked at the leaderboard of that competition at that moment... it shocked me. Competition Novice dominated the leaderboard, (at least for top 30). I cannot exclude the possibility that novice could have excellent machine learning knowledge and achieve high LB score, but it sounds weird for me.\nI checked the learderboard of that competition just now, most of the novice account disappeared... How do I know? Because I remember these novice accounts like to use famous actors' name, such as 'KunKun','ycy', as their user name, and it impressed me at that moment...\nIn my point of view, I prefer to consider it to be a kind of cheating... creating more accounts to use Kaggle cluster, but I have no evidence.
", "rawMarkdown": "I can't agree more. iNaturalist 2019 at FGVC6 is my first try in Kaggle, but when I looked at the leaderboard of that competition at that moment... it shocked me. Competition Novice dominated the leaderboard, (at least for top 30). I cannot exclude the possibility that novice could have excellent machine learning knowledge and achieve high LB score, but it sounds weird for me.\nI checked the learderboard of that competition just now, most of the novice account disappeared... How do I know? Because I remember these novice accounts like to use famous actors' name, such as 'KunKun','ycy', as their user name, and it impressed me at that moment...\nIn my point of view, I prefer to consider it to be a kind of cheating... creating more accounts to use Kaggle cluster, but I have no evidence.", "votes": 1 } ] }, { "id": 551483, "postDate": "2019-06-12T19:11:13.033Z", "content": "@inversion: Thank you for such a clear statement about the rules. Allowing use of QM7 and QM9 data may not be so bad. It seems papers on this go back a few years, so there are probably many potential chemists who would already know this data well and it would inform their modeling even if only tangentially. Allowing the rest of us to consider QM7 and QM9 might level the field some. Especially since probably some would have secretly used it anyway.
\n\nI very much hope that you will reconsider your view on future scientific competitions, these are great challenges and very interesting to many of us. This is true in spite of the issues about external data and the likely presence of it for most scientific problems. It would be a shame to see fewer scientific or engineering opportunities on Kaggle.
", "rawMarkdown": "@inversion: Thank you for such a clear statement about the rules. Allowing use of QM7 and QM9 data may not be so bad. It seems papers on this go back a few years, so there are probably many potential chemists who would already know this data well and it would inform their modeling even if only tangentially. Allowing the rest of us to consider QM7 and QM9 might level the field some. Especially since probably some would have secretly used it anyway.\n\nI very much hope that you will reconsider your view on future scientific competitions, these are great challenges and very interesting to many of us. This is true in spite of the issues about external data and the likely presence of it for most scientific problems. It would be a shame to see fewer scientific or engineering opportunities on Kaggle.", "votes": 14, "replies": [ { "id": 551518, "postDate": "2019-06-12T20:01:35.933Z", "content": "There's a good chance that using QM7 and QM9 will be a waste of time, and this drama was for nothing. (But that is for individual competitors to decide.)
\n\nThere may be ways to mitigate abuse, in the future, but everything has a cost. You can count on more scientific competitions, but we really want to have them as Featured whenever possible, and this type of thing just makes is more tricky.
", "rawMarkdown": "There's a good chance that using QM7 and QM9 will be a waste of time, and this drama was for nothing. (But that is for individual competitors to decide.)\n\nThere may be ways to mitigate abuse, in the future, but everything has a cost. You can count on more scientific competitions, but we really want to have them as Featured whenever possible, and this type of thing just makes is more tricky.", "votes": 9 } ] }, { "id": 551488, "postDate": "2019-06-12T19:27:12.540Z", "content": "Thank you for this update!
\n\nI'm really sad that this situation could case less scientific competitions in the future. I hope we (kagglers) would be able to help science in future and won't ruin competitions (though I think we didn't really ruin this competition).
", "rawMarkdown": "Thank you for this update!\n\nI'm really sad that this situation could case less scientific competitions in the future. I hope we (kagglers) would be able to help science in future and won't ruin competitions (though I think we didn't really ruin this competition).", "votes": 11 }, { "id": 551994, "postDate": "2019-06-13T11:52:06.213Z", "content": "@inversion: Thank you for your works to make the competition fair. I came from computational chemistry field and I'm really excited about what happens to chemistry with the power of data science and kagglers.
\n\nI'm worried that something like another unfairness will happen because of the ambiguity of the rule about the usage of molecular simulation schemes. When I read the rules, I could not understand which level of simulation schemes are allowed and which kind of data must be shared as external data. In this nice discussion below, the answer was given by the competition host.\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/94457#latest-546076
\n\nIn my opinion, the information given in the discussion should be officially announced (or added to rules) in easy words. I am really afraid that participants who are not familiar with chemistry or do not know the regulation may violate the terms without being aware of it.
\n\nI understand the difficulty of organizing a scientific competition. I hope the competition will be success and contribute to chemistry world! Moreover I hope there will be a lot of scientific competitions at kaggle.
", "rawMarkdown": "@inversion: Thank you for your works to make the competition fair. I came from computational chemistry field and I'm really excited about what happens to chemistry with the power of data science and kagglers.\n\nI'm worried that something like another unfairness will happen because of the ambiguity of the rule about the usage of molecular simulation schemes. When I read the rules, I could not understand which level of simulation schemes are allowed and which kind of data must be shared as external data. In this nice discussion below, the answer was given by the competition host.\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/94457#latest-546076\n\nIn my opinion, the information given in the discussion should be officially announced (or added to rules) in easy words. I am really afraid that participants who are not familiar with chemistry or do not know the regulation may violate the terms without being aware of it.\n\nI understand the difficulty of organizing a scientific competition. I hope the competition will be success and contribute to chemistry world! Moreover I hope there will be a lot of scientific competitions at kaggle.", "votes": 10 }, { "id": 551480, "postDate": "2019-06-12T19:06:23.987Z", "content": "Thanks for the clarification.
\n\nI may be wrong but I don't think these datasets will be very useful for our problem as they do not contain any magnetic information.
", "rawMarkdown": "Thanks for the clarification. \n\nI may be wrong but I don't think these datasets will be very useful for our problem as they do not contain any magnetic information.\n", "votes": 7, "replies": [ { "id": 551627, "postDate": "2019-06-13T00:24:59.557Z", "content": "Thank you for your thought, I got more motivation.\nI am going to not to use QM9. Because the model using QM9 does not solve the competition sponsor's problem.\n(And I hope that the score does not improve with the use of QM9!)
", "rawMarkdown": "Thank you for your thought, I got more motivation.\nI am going to not to use QM9. Because the model using QM9 does not solve the competition sponsor's problem.\n(And I hope that the score does not improve with the use of QM9!)", "votes": 3 } ] }, { "id": 558738, "postDate": "2019-06-22T20:42:38.103Z", "content": "@inversion Thank you for sponsoring this challenge and for writing this update.
\n\nLike the \"Higgs Boson\" and a few other challenges, this challenge will provide the science research with some new tools and techniques. I compare these challenges with old-time challenges that led Louis Pasteur and other scientists put forth their scientific concepts etc.
\n\nIn another challenge, the first winner got a few million dollars. The company could not and did not use the first winner's method. Instead of worrying about that, the company felt happy that the challenge could contribute to the Predictive Analytics field.
\n\nYour concerns are quite legitimate. To avoid such concerns in future science-related challenges:
\n\nThe sponsor may state the front-end and back-end objectives clearly.
Kaggle may evolve a multi-point scoring system, involving Leaderboard score, external dependencies, affiliations, etc.
Please note that while a few participants are using these challenges for winning prizes, most of us (including me) are using them for learning.
\n\nI encourage and plead you to sponsor more such challenges in future. I understand you may have tight budgets, but I guess that the benefits to the sponsors from a Kaggle challenge generally outweigh the cost.
\n\nThank you.
", "rawMarkdown": "@inversion Thank you for sponsoring this challenge and for writing this update.\n\nLike the \"Higgs Boson\" and a few other challenges, this challenge will provide the science research with some new tools and techniques. I compare these challenges with old-time challenges that led Louis Pasteur and other scientists put forth their scientific concepts etc. \n\nIn another challenge, the first winner got a few million dollars. The company could not and did not use the first winner's method. Instead of worrying about that, the company felt happy that the challenge could contribute to the Predictive Analytics field.\n\nYour concerns are quite legitimate. To avoid such concerns in future science-related challenges:\n\n- The sponsor may state the front-end and back-end objectives clearly.\n\n- Kaggle may evolve a multi-point scoring system, involving Leaderboard score, external dependencies, affiliations, etc.\n\nPlease note that while a few participants are using these challenges for winning prizes, most of us (including me) are using them for learning.\n\nI encourage and plead you to sponsor more such challenges in future. I understand you may have tight budgets, but I guess that the benefits to the sponsors from a Kaggle challenge generally outweigh the cost.\n\nThank you.", "votes": 7, "replies": [ { "id": 558741, "postDate": "2019-06-22T20:49:04.023Z", "content": "Thanks for that note. Sometimes the drama overshadows the real reason why most of us are here in the fist place. Great reminder!
", "rawMarkdown": "Thanks for that note. Sometimes the drama overshadows the real reason why most of us are here in the fist place. Great reminder!", "votes": 4 } ] }, { "id": 551750, "postDate": "2019-06-13T04:33:12.843Z", "content": "Excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science. Why are we allowing this sort of thing?
\n\nAlso, as others have echoed, there is so much sharing of code, data, algos etc. that is repeatedly ignored by Kaggle management. I would prefer focusing on this type of abuse vs. fretting about those who post public code with publicly available data.
", "rawMarkdown": "Excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science. Why are we allowing this sort of thing?\n\nAlso, as others have echoed, there is so much sharing of code, data, algos etc. that is repeatedly ignored by Kaggle management. I would prefer focusing on this type of abuse vs. fretting about those who post public code with publicly available data.", "votes": 7 }, { "id": 551532, "postDate": "2019-06-12T20:30:10.340Z", "content": "Wow, am I solely responsible here for \"that we will have fewer opportunities to host scientific challenges on Kaggle\", \"scientific competitions we do launch are less likely to be Featured\" and \"with smaller prize pools\"? Because you make it feel like that. No problem with barely defined rules on that topic? It is only stated that the dataset is not allowed to test and develop ones model, not that it is not allowed to study it and share. No problem with that the dataset is available to everyone anyway? No problem at all that you by yourself was interested in \"how much lift this data provides\" and you blame others for been interested in that as well? This last one is important question, because if openly available dataset with enough extra information is present in a competition it would destroy silver/bronze medals range, where the models are not checked anymore. Maybe this is something Kaggle can compromise on, and I am fine with it, I know it is very hard to design a perfect competition, but I want to know it before I decide to participate and spend significant amount of my time on the problem.
\n\nBy the way, I have no idea where you take this \"feel entitled to abuse the rules\" and \"break competition rules because you disagree with\" from. I explained myself multiple times, I understood this sentence \"not allowed to test and develop models\" in the rules as that it is not allowed that one's final submitted models are using the dataset or fitted to the dataset in some way.
\n\nMoreover, I un-shared my work the minute I understood from you that it was not allowed to share. It was up something like 24h, just a piece of code and a link to a dataset that is available online anyway.
\n\nAnyway... like many others I think this competition is a really beautiful one, with interesting data, and hope it continues with no more problems. Good luck to the participants and I hope some interesting approaches will appear.
", "rawMarkdown": "Wow, am I solely responsible here for \"that we will have fewer opportunities to host scientific challenges on Kaggle\", \"scientific competitions we do launch are less likely to be Featured\" and \"with smaller prize pools\"? Because you make it feel like that. No problem with barely defined rules on that topic? It is only stated that the dataset is not allowed to test and develop ones model, not that it is not allowed to study it and share. No problem with that the dataset is available to everyone anyway? No problem at all that you by yourself was interested in \"how much lift this data provides\" and you blame others for been interested in that as well? This last one is important question, because if openly available dataset with enough extra information is present in a competition it would destroy silver/bronze medals range, where the models are not checked anymore. Maybe this is something Kaggle can compromise on, and I am fine with it, I know it is very hard to design a perfect competition, but I want to know it before I decide to participate and spend significant amount of my time on the problem. \n\nBy the way, I have no idea where you take this \"feel entitled to abuse the rules\" and \"break competition rules because you disagree with\" from. I explained myself multiple times, I understood this sentence \"not allowed to test and develop models\" in the rules as that it is not allowed that one's final submitted models are using the dataset or fitted to the dataset in some way. \n\nMoreover, I un-shared my work the minute I understood from you that it was not allowed to share. It was up something like 24h, just a piece of code and a link to a dataset that is available online anyway. \n\nAnyway... like many others I think this competition is a really beautiful one, with interesting data, and hope it continues with no more problems. Good luck to the participants and I hope some interesting approaches will appear.", "votes": 4, "replies": [ { "id": 551545, "postDate": "2019-06-12T20:48:43.753Z", "content": "If a competitor didn't make submissions using that data (prior to the rule change), then there is nothing to worry about.
\n\nI do, though, hope you see how suboptimal it was for you to take it upon yourself to post the data that was explicitly forbidden, even if it was \"just to study\".
", "rawMarkdown": "If a competitor didn't make submissions using that data (prior to the rule change), then there is nothing to worry about.\n\nI do, though, hope you see how suboptimal it was for you to take it upon yourself to post the data that was explicitly forbidden, even if it was \"just to study\". ", "votes": 3 }, { "id": 551569, "postDate": "2019-06-12T21:41:32.647Z", "content": "@inversion You should reconsider this point:
\n\n> even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.
\n\nIn my opinion, if you used the QM7 and QM9 datasets before this rule change, then you haven't done anything wrong, because I agree with nosound's interpretation of the old rules: Your final 2 submissions should not use the QM7 and QM9.
\n\nIf the public LB gets screwed up, then that's a consequence of the data being available, not the fault of the person who submitted it. Consider the rstudio competition which had scores of 0 RMSE, or the 2sigma competition which had the public LB answers within the kernel leading to a perfect score
\n\nIn my opinion there's nothing wrong with looking at this data, training on it, and making submissions on it. The only thing that should be satisfied is your final 2 submissions cannot use the data. If your submission uses the external data, then that submission should not be counted
", "rawMarkdown": "@inversion You should reconsider this point:\n\n> even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.\n\nIn my opinion, if you used the QM7 and QM9 datasets before this rule change, then you haven't done anything wrong, because I agree with nosound's interpretation of the old rules: Your final 2 submissions should not use the QM7 and QM9.\n\nIf the public LB gets screwed up, then that's a consequence of the data being available, not the fault of the person who submitted it. Consider the rstudio competition which had scores of 0 RMSE, or the 2sigma competition which had the public LB answers within the kernel leading to a perfect score\n\nIn my opinion there's nothing wrong with looking at this data, training on it, and making submissions on it. The only thing that should be satisfied is your final 2 submissions cannot use the data. If your submission uses the external data, then that submission should not be counted\n", "votes": 7 }, { "id": 551571, "postDate": "2019-06-12T21:48:51.137Z", "content": "I am really surprised and sad that this has blown up into such a huge thing leading to Kaggle possibly having fewer opportunities to host scientific challenges.
\n\nIt seems to me like @zaharch 's kernels and posts were an honest and understandable misunderstanding of unclear rules (that apparently I too have misunderstood) and that they have not ruined this competition but possibly even prevented it from later ruin.
\n\nI suggest that in future Kaggle should spell out the rules more clearly. Please consider stating explicitly that it is also not allowed to post any kernel or discussion using/containing even a subset of the forbidden datasets to the competition, if that is the case.
", "rawMarkdown": "I am really surprised and sad that this has blown up into such a huge thing leading to Kaggle possibly having fewer opportunities to host scientific challenges. \n \nIt seems to me like @zaharch 's kernels and posts were an honest and understandable misunderstanding of unclear rules (that apparently I too have misunderstood) and that they have not ruined this competition but possibly even prevented it from later ruin.\n\nI suggest that in future Kaggle should spell out the rules more clearly. Please consider stating explicitly that it is also not allowed to post any kernel or discussion using/containing even a subset of the forbidden datasets to the competition, if that is the case.", "votes": 8 }, { "id": 551580, "postDate": "2019-06-12T22:04:19.463Z", "content": "I do not understand why you have a hard feeling against him/her. The situation is like, he/she found a jar of worms in an empty room, and said it out loud \"Hey look at this there's a jar of worms!\" While I agree this is not the best thing to do, I don't see there was anything obviously harmful being done because of this action. He/She did not bring the worms to the room, the worms were already in the room and it is just a matter of time for someone to find it. At least this is much much better than someone secretly opens the jar and scares everyone away.
", "rawMarkdown": "I do not understand why you have a hard feeling against him/her. The situation is like, he/she found a jar of worms in an empty room, and said it out loud \"Hey look at this there's a jar of worms!\" While I agree this is not the best thing to do, I don't see there was anything obviously harmful being done because of this action. He/She did not bring the worms to the room, the worms were already in the room and it is just a matter of time for someone to find it. At least this is much much better than someone secretly opens the jar and scares everyone away.", "votes": 2 }, { "id": 551586, "postDate": "2019-06-12T22:12:58.333Z", "content": "> In my opinion there's nothing wrong with looking at this data, training on it, and making submissions on it.
\n\n1) It was against the Rules, regardless of whether or not you think it was wrong.\n2) But using the data and submitting, you are gaining information that other competitors who did not break the rules do not have.\n3) At no time in the history of Kaggle has it been acceptable to break the rules as long as it wasn't for your final selected submissions.
", "rawMarkdown": "> In my opinion there's nothing wrong with looking at this data, training on it, and making submissions on it.\n\n1) It was against the Rules, regardless of whether or not you think it was wrong.\n2) But using the data and submitting, you are gaining information that other competitors _who did not break the rules_ do not have.\n3) At no time in the history of Kaggle has it been acceptable to break the rules as long as it wasn't for your final selected submissions.", "votes": 5 }, { "id": 551592, "postDate": "2019-06-12T22:18:39.713Z", "content": "@xylthx - I've stated this multiple times. This issue is not bringing up the jar of worms. There is nothing wrong with that.
\n\nThe issue is finding a jar of worms, seeing a sign that says \"no worms please\", yet still opening it up and dumping the worms all over the carpet before we even had the chance to give a response about what should be done with the worms.
", "rawMarkdown": "@xylthx - I've stated this multiple times. This issue is not bringing up the jar of worms. There is nothing wrong with that.\n\nThe issue is finding a jar of worms, seeing a sign that says \"no worms please\", yet still opening it up and dumping the worms all over the carpet _before we even had the chance to give a response about what should be done with the worms_.", "votes": 2 }, { "id": 551594, "postDate": "2019-06-12T22:23:06.800Z", "content": "@inversion - I don't see how he/she opened it up and dumped the worms all over the carpet. In my opinion you are exaggerating it. I do not see any obvious harm he/she has done because of this action.
", "rawMarkdown": "@inversion - I don't see how he/she opened it up and dumped the worms all over the carpet. In my opinion you are exaggerating it. I do not see any obvious harm he/she has done because of this action." }, { "id": 551611, "postDate": "2019-06-12T23:29:57.493Z", "rawMarkdown": "", "isDeleted": true }, { "id": 551642, "postDate": "2019-06-13T00:58:05.127Z", "content": "https://www.kaggle.com/c/two-sigma-financial-news/overview/Honor-Code
\n\n\n\n\n\"cheating\" encompasses any attempt to gain an edge in accuracy by using information that is outside of the provided dataset, or an attempt to use the provided information in a way that is not intended. Examples of cheating include (but are not limited to):
\n \nAttempting to use datasets and references beyond those made available by the competition\n Attempting to abuse the competition infrastructure to gain an edge
\n
Everyone who used the env_var07 in public submissions in Two Sigma competition should therefore be banned by your logic, since they made a public submission that \"abused the competition infrastructure to gain an edge.\" That has not been made clear yet, and I don't think it will happen because frankly it shouldn't happen. The competition had the target answers, that we weren't supposed to use, smack dab in the kernel itself. Just because you were curious and submitted to public LB, but didn't use it for private LB, shouldn't mean that you get disqualified. I think the same holds true for this competition.
@returnofsputnik - I'm not managing that competition, so really can't comment. But, in general, compliance is handled at the end of the competition.
\n\n@xylthx - I want to be clear, I'm not stating that any particular person did or did not use the data to make predictions. I'm saying that using that data (prior to the change) to make predictions was a violation of the rules. It does not matter whether or not you think the action was harmless.
", "rawMarkdown": "@returnofsputnik - I'm not managing that competition, so really can't comment. But, in general, compliance is handled at the end of the competition.\n\n@xylthx - I want to be clear, I'm not stating that any particular person did or did not use the data to make predictions. I'm saying that using that data (prior to the change) to make predictions was a violation of the rules. It does not matter whether or not you think the action was harmless." }, { "id": 551654, "postDate": "2019-06-13T01:37:55.290Z", "content": "\n\n\n\n\nPlease consider stating explicitly that it is also not allowed to post any kernel or discussion using/containing even a subset of the forbidden datasets
\n
Discussing the data? That's fine.\nPosting the data? Not against the rules. (But why, if you can't use it?)\nUsing the data to inform how a model is developed? Against the rules.\nUsing the data to make predictions? Against the rules.
\n\nAll of this could have been cleared up if I had been given more than half a business day to respond to questions on the forums.
", "rawMarkdown": "@taniaj \n\n> Please consider stating explicitly that it is also not allowed to post any kernel or discussion using/containing even a subset of the forbidden datasets\n\nDiscussing the data? That's fine.\nPosting the data? Not against the rules. (But why, if you can't use it?)\nUsing the data to inform how a model is developed? Against the rules.\nUsing the data to make predictions? Against the rules.\n\nAll of this could have been cleared up if I had been given more than half a business day to respond to questions on the forums.", "votes": -1 }, { "id": 551707, "postDate": "2019-06-13T03:16:04.800Z", "content": "@inversion - I am sure that you saw violations of rules way more than I did in Kaggle, but I've never seen an administrator so pissed off and start to criticizing individuals in the public discussion. Yes of course you could have a couple more days in advance, but no one was blaming you because of this. And of course what I think does not matter, actually what ANY kaggler thinks does not matter because you are the one who is making the rules. I just feel there is no reason to make it personal, which is exactly what you are doing now.
", "rawMarkdown": "@inversion - I am sure that you saw violations of rules way more than I did in Kaggle, but I've never seen an administrator so pissed off and start to criticizing individuals in the public discussion. Yes of course you could have a couple more days in advance, but no one was blaming you because of this. And of course what I think does not matter, actually what ANY kaggler thinks does not matter because you are the one who is making the rules. I just feel there is no reason to make it personal, which is exactly what you are doing now.", "votes": 6 }, { "id": 551714, "postDate": "2019-06-13T03:36:08.083Z", "content": "@inversion - I understand that you as an admin wants everyone to obey the rules and will be annoyed if anyone does not. But I feel that you are hostile to that particular Kaggler and also any Kagglers that seem to understand his/her idea. This actually makes normal discussion nearly impossible, because whenever someone challenges you, your answer is \"I do not care what you think, I am the person who makes the rules.\" Yes, he/she made a mistake, but I do not see why he/she deserves this treatment. You took a few days off and this happens, this is not your fault, and this is not completely his/her fault. There was not a lot chaos, actually a lot of people are happy to bring up this issue to the table at this early stage instead of three months later. And as a scientist myself, I do not see the inhibition of using some external data undermining the scientific value of the challenge, actually it is setting all these limitations undermining the usefulness of a model by making it \"fancy but useless\", because in the real world, what the users want is simply \"fast and accurate\", no one will even care about what external data it uses when making it.
", "rawMarkdown": "@inversion - I understand that you as an admin wants everyone to obey the rules and will be annoyed if anyone does not. But I feel that you are hostile to that particular Kaggler and also any Kagglers that seem to understand his/her idea. This actually makes normal discussion nearly impossible, because whenever someone challenges you, your answer is \"I do not care what you think, I am the person who makes the rules.\" Yes, he/she made a mistake, but I do not see why he/she deserves this treatment. You took a few days off and this happens, this is not your fault, and this is not completely his/her fault. There was not a lot chaos, actually a lot of people are happy to bring up this issue to the table at this early stage instead of three months later. And as a scientist myself, I do not see the inhibition of using some external data undermining the scientific value of the challenge, actually it is setting all these limitations undermining the usefulness of a model by making it \"fancy but useless\", because in the real world, what the users want is simply \"fast and accurate\", no one will even care about what external data it uses when making it.", "votes": 2 }, { "id": 551763, "postDate": "2019-06-13T04:58:10.557Z", "content": "Josef,
\n\nWe have multiple members of Kaggle monitor all of the forum posts for each competition, which typically total 2000-4000 unique posts per competition. With 15-25 competitions live at a time divided across a small staff, it's not uncommon for us to field 500+ discussion posts per day, in addition to our normal workload.
\n\nWe're thankful that the Kaggle community is so active, engaging, and often answers most of these comments and questions before we even get to them, whether it be sharing ideas/code (within the rules, of course 😉 ), troubleshooting, interpreting competition rules, or general tips and tricks on machine learning, competitions, and general data science. We're just as excited to see Kagglers reach a new tier as the community is, and even celebrate across our own internal messaging app when we see people reach Grandmaster status.
\n\n@inversion - being the OG Kaggle grandmaster, has invested countless hours in the Kaggle community, both before he was hired, and even more so (now that Kaggle pays him for his work 😄). One of the worst, and most frustrating things for someone on the Kaggle competitions team to experience is to see a lot, or all, of the hard work invested in preparing a competition unravel. Whether it be through leakage, unwanted public data, a high-scoring kernel shared last minute, or a member of the community violating the rules and tarnishing the experience of other Kagglers. As this was a very specific instance of a unique dataset in chemical engineering, an industry that @inversion is particular close, is rare on Kaggle, and is one we want to bring to the Kaggle community.
\n\nIt's one thing for a user to cheat and try to hide it (in which we often catch and remove them), or accidentally cheat and admit it. We also commonly receive inquiries on interpretations of the rules, or challenges to the structure of the competition. It's because of the Kaggle community that we try new things, new competition formats, and are willing to push the boundaries of what is considered \"standard\" to supervised machine learning competitions. We welcome the discourse and even the disagreement. We make changes to future competitions' structure and rules based on community input. We listen and respect the Kaggle community (It should be noted that we often hire them), and we want to see the community successful.
\n\nHowever, it's another thing entirely when there is a blatant disregard for the clearly stated rules, the rule is broken, announced it is broken, and those results are then shared to the masses. It's similar to if you saw a police officer on the side of the road trying to catch people speeding on a highway that had a speed limit you didn't agree with. So instead of going to city council and expressing your concerns about why that speed limit wasn't realistic and was detrimental to the public welfare, you instead accelerated right by the officer and yelled \"This is what I think about your speed limit!\" out the window, all while live streaming the event from your phone. Extreme example? Sure - but pretty funny to imagine @inversion in a Dodge Charger hiding behind a billboard trying to pull over speeding Kagglers.
\n\nThe former is proactive, people listen, and the potential for change is up for discussion. The latter assumes that the rule is arbitrary, deserves to be broken in a grand display, and results in frustration by both parties - there is usually little remorse, no collaboration, no growth, and nobody ends up better off.
\n\nWe care about the Kaggle community. We encourage open discussion. We even welcome being challenged. Discuss, disagree, and cast doubt - all fine. But by taking a rule, acknowledging it, and yet still publicly breaking it, it's not only greatly disrespectful to Kaggle but it is disrespectful to the hosts of the competition and to the rest of the Kaggle community. It not only assumes that there are no good, valid reasons for the rules to be in place, but that those rules are so outlandish they should be violated via public spectacle without the opportunity for discussion. Just because the harm/consequences of violating a rule don't seem \"obvious,\" it doesn't means that the harm isn't there, nor does it suggest that the best way to address such is to assume the worst and act recklessly.
\n\nWe want Kagglers to be successful. In fact \"user success\" is literally one of Kaggle's core values. We constantly see competition sponsors come to Kaggle with interesting and unique problems for the community to solve. They often do so because of the reputation that over three million Kagglers bring with them. But by playing vigilante justice with a competition rule, it not only impacts the hundreds, if not thousands of other Kagglers trying to learn and develop their machine learning skills and reputations, but it also jeopardizes Kaggle's ability to continue to provide these types of problems to our community in the future. We have sponsors who point to competitions from over seven years ago as references for what they'd like to emulate, as well as to discussion posts from the recent past for things they'd like to avoid. They often point to competitions of a similar nature in industry and problem type as a guide.
\n\nSo when a competition in a highly sought chemical engineering space (both close to @inversion and to many members of the scientific community) makes for a great problem, you can imagine why it stings a bit deeper when that work is insulted, rules are blatantly and fragrantly disregarded, and deference to the well-reasoned structure is defied. The Kaggle community spends more time discussing how to interpret and adhere to minutiae of the rules instead of creatively solving machine learning problems, which doesn't really benefit anyone.
", "rawMarkdown": "Josef,\n\nWe have multiple members of Kaggle monitor all of the forum posts for each competition, which typically total 2000-4000 unique posts per competition. With 15-25 competitions live at a time divided across a small staff, it's not uncommon for us to field 500+ discussion posts per day, in addition to our normal workload. \n\nWe're thankful that the Kaggle community is so active, engaging, and often answers most of these comments and questions before we even get to them, whether it be sharing ideas/code (within the rules, of course 😉 ), troubleshooting, interpreting competition rules, or general tips and tricks on machine learning, competitions, and general data science. We're just as excited to see Kagglers reach a new tier as the community is, and even celebrate across our own internal messaging app when we see people reach Grandmaster status.\n\n@inversion - being the OG Kaggle grandmaster, has invested countless hours in the Kaggle community, both before he was hired, and even more so (now that Kaggle pays him for his work 😄). One of the worst, and most frustrating things for someone on the Kaggle competitions team to experience is to see a lot, or all, of the hard work invested in preparing a competition unravel. Whether it be through leakage, unwanted public data, a high-scoring kernel shared last minute, or a member of the community violating the rules and tarnishing the experience of other Kagglers. As this was a very specific instance of a unique dataset in chemical engineering, an industry that @inversion is particular close, is rare on Kaggle, and is one we want to bring to the Kaggle community. \n\nIt's one thing for a user to cheat and try to hide it (in which we often catch and remove them), or accidentally cheat and admit it. We also commonly receive inquiries on interpretations of the rules, or challenges to the structure of the competition. It's because of the Kaggle community that we try new things, new competition formats, and are willing to push the boundaries of what is considered \"standard\" to supervised machine learning competitions. We welcome the discourse and even the disagreement. We make changes to future competitions' structure and rules based on community input. We listen and respect the Kaggle community (It should be noted that we often hire them), and we want to see the community successful.\n\nHowever, it's another thing entirely when there is a blatant disregard for the clearly stated rules, the rule is broken, announced it is broken, and those results are then shared to the masses. It's similar to if you saw a police officer on the side of the road trying to catch people speeding on a highway that had a speed limit you didn't agree with. So instead of going to city council and expressing your concerns about why that speed limit wasn't realistic and was detrimental to the public welfare, you instead accelerated right by the officer and yelled \"This is what I think about your speed limit!\" out the window, all while live streaming the event from your phone. Extreme example? Sure - but pretty funny to imagine @inversion in a Dodge Charger hiding behind a billboard trying to pull over speeding Kagglers.\n\nThe former is proactive, people listen, and the potential for change is up for discussion. The latter assumes that the rule is arbitrary, deserves to be broken in a grand display, and results in frustration by both parties - there is usually little remorse, no collaboration, no growth, and nobody ends up better off. \n\nWe care about the Kaggle community. We encourage open discussion. We even welcome being challenged. Discuss, disagree, and cast doubt - all fine. But by taking a rule, acknowledging it, and yet still publicly breaking it, it's not only greatly disrespectful to Kaggle but it is disrespectful to the hosts of the competition and to the rest of the Kaggle community. It not only assumes that there are no good, valid reasons for the rules to be in place, but that those rules are so outlandish they should be violated via public spectacle without the opportunity for discussion. Just because the harm/consequences of violating a rule don't seem \"obvious,\" it doesn't means that the harm isn't there, nor does it suggest that the best way to address such is to assume the worst and act recklessly.\n\nWe want Kagglers to be successful. In fact \"user success\" is literally one of Kaggle's core values. We constantly see competition sponsors come to Kaggle with interesting and unique problems for the community to solve. They often do so because of the reputation that over three million Kagglers bring with them. But by playing vigilante justice with a competition rule, it not only impacts the hundreds, if not thousands of other Kagglers trying to learn and develop their machine learning skills and reputations, but it also jeopardizes Kaggle's ability to continue to provide these types of problems to our community in the future. We have sponsors who point to competitions from over seven years ago as references for what they'd like to emulate, as well as to discussion posts from the recent past for things they'd like to avoid. They often point to competitions of a similar nature in industry and problem type as a guide. \n\nSo when a competition in a highly sought chemical engineering space (both close to @inversion and to many members of the scientific community) makes for a great problem, you can imagine why it stings a bit deeper when that work is insulted, rules are blatantly and fragrantly disregarded, and deference to the well-reasoned structure is defied. The Kaggle community spends more time discussing how to interpret and adhere to minutiae of the rules instead of creatively solving machine learning problems, which doesn't really benefit anyone.", "votes": 15 }, { "id": 551806, "postDate": "2019-06-13T06:12:02.230Z", "content": "@addisonhoward - I really appreciate your extensive reply. Frankly speaking, what I feel uncomfortable about this situation is that apparently the Kaggler who posted QM9 dataset is being treated like a pest to the community, while I believe his/her intention was not to ruin this competition. He/She could easily use this dataset secretly, and posted a few exaggerated threads/kernels three months later, if ruining this challenge is his/her intention.
\n\nI cannot find the old rules, but if @returnofsputnik is right, if the old rule was \"Your final 2 submissions should not use the QM7 and QM9\", I don't see any violation by simply studying QM9, or making submissions now (but not finally) using this dataset to just try it out. Even if the dataset is banned from using to train the model, there should be no limitations not to allow people to learn from it, and possibly share knowledge they gained from it. I am pretty sure my interpretation is wrong here, but misunderstanding the rules is not disrespect to Kaggle, and I am sure you will agree. What I saw from this and other threads he/she posted and replied was he/she got taunted by other users, as if everyone is trying to kick him/her out and the admin upvoted for that. People are not sure about the rules, get confused, and dare not to discuss QM9 related issue publicly for not being kicked out of the competition, while I am sure many Kagglers will still at least study this dataset. I don't think this climate is beneficial to this community.
\n\nAnd since a high-level (possibly) person like you replied, I'd like to discuss more about the science topic competitions like this one. I am a scientist (I guess I can say this) and also a data science user; I have degrees in both domains. The thing I don't feel comfortable about the current trend is that natural scientists are still very skeptical to machine learning approaches, while data scientists, instead of trying to explain how useful machine learning models can be, are aiming to beat the state-of-the-art scientific approach and claim the victory of data science over the traditional approaches (instead of trying to learn from the domain knowledge and understand what the scientists really want). Therefore, often the outcome is, data scientists built models that look pretty cool and interesting, but the natural scientists found such models useless for some reasons. Coming back to this specific competition, what I don't really understand is that, is the goal of this challenge is to find a fast and accurate model that works on any molecule, or the goal is simply \"Let's prove to those dumbass chemists what data science can do\"? Obviously the latter one does not add ANY scientific value and will most likely end up with some fancy but useless model. To be more specific, the reason not to use any quantum properties is because the computation is expensive, and if a person use a pre-calculated dataset, he/she would not be accurately predict a molecule outside of the pre-calculated dataset, making the model useless. In my opinion, providing a test set whose molecules are not in QM9 would be a good solution. Simply limiting the usage of some external dataset will actually undermine the scientific benefit, because in the real world, when using a model, you never care how it was trained, and I really doubt there is any scientific domain knowledge a chemist can learn from a model with more than tens of thousands of parameters.
", "rawMarkdown": "@addisonhoward - I really appreciate your extensive reply. Frankly speaking, what I feel uncomfortable about this situation is that apparently the Kaggler who posted QM9 dataset is being treated like a pest to the community, while I believe his/her intention was not to ruin this competition. He/She could easily use this dataset secretly, and posted a few exaggerated threads/kernels three months later, if ruining this challenge is his/her intention. \n\nI cannot find the old rules, but if @returnofsputnik is right, if the old rule was \"Your final 2 submissions should not use the QM7 and QM9\", I don't see any violation by simply studying QM9, or making submissions now (but not finally) using this dataset to just try it out. Even if the dataset is banned from using to train the model, there should be no limitations not to allow people to learn from it, and possibly share knowledge they gained from it. I am pretty sure my interpretation is wrong here, but misunderstanding the rules is not disrespect to Kaggle, and I am sure you will agree. What I saw from this and other threads he/she posted and replied was he/she got taunted by other users, as if everyone is trying to kick him/her out and the admin upvoted for that. People are not sure about the rules, get confused, and dare not to discuss QM9 related issue publicly for not being kicked out of the competition, while I am sure many Kagglers will still at least study this dataset. I don't think this climate is beneficial to this community.\n\nAnd since a high-level (possibly) person like you replied, I'd like to discuss more about the science topic competitions like this one. I am a scientist (I guess I can say this) and also a data science user; I have degrees in both domains. The thing I don't feel comfortable about the current trend is that natural scientists are still very skeptical to machine learning approaches, while data scientists, instead of trying to explain how useful machine learning models can be, are aiming to beat the state-of-the-art scientific approach and claim the victory of data science over the traditional approaches (instead of trying to learn from the domain knowledge and understand what the scientists really want). Therefore, often the outcome is, data scientists built models that look pretty cool and interesting, but the natural scientists found such models useless for some reasons. Coming back to this specific competition, what I don't really understand is that, is the goal of this challenge is to find a fast and accurate model that works on any molecule, or the goal is simply \"Let's prove to those dumbass chemists what data science can do\"? Obviously the latter one does not add ANY scientific value and will most likely end up with some fancy but useless model. To be more specific, the reason not to use any quantum properties is because the computation is expensive, and if a person use a pre-calculated dataset, he/she would not be accurately predict a molecule outside of the pre-calculated dataset, making the model useless. In my opinion, providing a test set whose molecules are not in QM9 would be a good solution. Simply limiting the usage of some external dataset will actually undermine the scientific benefit, because in the real world, when using a model, you never care how it was trained, and I really doubt there is any scientific domain knowledge a chemist can learn from a model with more than tens of thousands of parameters." }, { "id": 551986, "postDate": "2019-06-13T11:26:09.683Z", "content": "@addisonhoward , thanks you for sharing your thoughts, it completely resonates with me. However, I want to reiterate once and again, since it looks like you assume differently. I did not violate the rule knowingly or intentionally. As I wrote before, I understood the rule differently, and I didn't have any questions about my understanding of the rule, it didn't appear to me at the moment that my interpretation might be wrong, and therefore I didn't seek any clarifications on that. I understood the rule that it was not allowed to use the dataset in one's final submissions, because it was the only logical interpretation that appeared to me. Not been able to look into relevant and publicly available dataset, and not been able to study and discuss it publicly did not appear to me as a logical interpretation at the moment and I did not consider it for a second more beyond that.
\n\nIn fact, I don't even think that it is correct to say that I misunderstood the rule, because I think my interpretation is a possible one, you can see other people agreeing with me that it was a possible interpretation. I think the correct diagnosis here is that the rule was not clearly defined.
\n\nthank you for doing this hard job...
", "rawMarkdown": "@addisonhoward , thanks you for sharing your thoughts, it completely resonates with me. However, I want to reiterate once and again, since it looks like you assume differently. I did not violate the rule knowingly or intentionally. As I wrote before, I understood the rule differently, and I didn't have any questions about my understanding of the rule, it didn't appear to me at the moment that my interpretation might be wrong, and therefore I didn't seek any clarifications on that. I understood the rule that it was not allowed to use the dataset in one's final submissions, because it was the only logical interpretation that appeared to me. Not been able to look into relevant and publicly available dataset, and not been able to study and discuss it publicly did not appear to me as a logical interpretation at the moment and I did not consider it for a second more beyond that. \n\nIn fact, I don't even think that it is correct to say that I misunderstood the rule, because I think my interpretation is a possible one, you can see other people agreeing with me that it was a possible interpretation. I think the correct diagnosis here is that the rule was not clearly defined.\n\nthank you for doing this hard job... ", "votes": 5 } ] }, { "id": 551529, "postDate": "2019-06-12T20:25:41.533Z", "content": "I am bit saddened that scientific value of the competition has been lowered b/c of this issue. I have two questions:
\n\nCould Kaggle verify the top money and at least gold solutions for evidence that the forbidden data was not used? If a cheater is caught, it would be get a temp ban from Kaggle, perm bans for repeating cheaters. For money-seeking cheaters, maybe require some level (Expert/Master, etc.) to qualify to win monetary prices.
I understand molecules cannot be anonymized, but it looks like all molecules in the test set are included in the forbidden datasets. Would it make sense to have a 2-stage competition where 2nd stage test set molecules are not included in forbidden datasets?
@inversion as you value community feedback and dialogue I'd be very interested in your stance on 1) and 2) and a bit more clarity on what you call conceptual reasons: I have only high-school knowledge of chemistry, and reading the rules and the explicit restriction of quantum mechanics approaches I thought that the creation of these datasets (QM7/QM9) was very expensive as well and solutions based on them would provide less scientific value.
", "rawMarkdown": "I am bit saddened that scientific value of the competition has been lowered b/c of this issue. I have two questions:\n\n1. Could Kaggle verify the top money and at least gold solutions for evidence that the forbidden data was not used? If a cheater is caught, it would be get a temp ban from Kaggle, perm bans for repeating cheaters. For money-seeking cheaters, maybe require some level (Expert/Master, etc.) to qualify to win monetary prices.\n\n2. I understand molecules cannot be anonymized, but it looks like *all* molecules in the test set are included in the forbidden datasets. Would it make sense to have a 2-stage competition where 2nd stage test set molecules are not included in forbidden datasets?\n\n@inversion as you value community feedback and dialogue I'd be very interested in your stance on 1) and 2) and a bit more clarity on what you call _conceptual reasons_: I have only high-school knowledge of chemistry, and reading the rules and the explicit restriction of quantum mechanics approaches I thought that the creation of these datasets (QM7/QM9) was very expensive as well and solutions based on them would provide less scientific value.", "votes": 3, "replies": [ { "id": 551535, "postDate": "2019-06-12T20:34:03.867Z", "content": "I disagree that \"solutions based on them would provide less scientific value\". The solutions based on them would provide less \"data science\" value, but not \"scientific\" value. I believe that the aim of this competition, and also the host, is to find an accurate and fast model to predict the molecule properties, not to prove \"see how good using only data science knowledge can do\".
\n\nAlso, correct me if I am wrong, for \"2nd stage test set molecules are not included in forbidden datasets\", I really doubt they have any other molecules which are not in that dataset....
", "rawMarkdown": "I disagree that \"solutions based on them would provide less scientific value\". The solutions based on them would provide less \"data science\" value, but not \"scientific\" value. I believe that the aim of this competition, and also the host, is to find an accurate and fast model to predict the molecule properties, not to prove \"see how good using only data science knowledge can do\".\n\nAlso, correct me if I am wrong, for \"2nd stage test set molecules are not included in forbidden datasets\", I really doubt they have any other molecules which are not in that dataset....", "votes": -1 }, { "id": 551544, "postDate": "2019-06-12T20:47:19.190Z", "content": "Well looking at QM9 it contains ~133k molecules, and this comp is a subset (130k total), and according to the GM9 abstract \" These molecules correspond to the subset of all 133,885 species with up to nine heavy atoms (CONF) out of the GDB-17 chemical universe of 166 billion organic molecules.\" so whatever ML solutions are devised here using QM9 would not be applicable to predict scalar couplings for the vast majority of the GDB-17.
\n\nAnd obviously the organization already knows the scalar coupling values for almost all of QM9, hence my comment, which I may move from \"less scientific value\" to \"less practical value\".
", "rawMarkdown": "Well looking at QM9 it contains ~133k molecules, and this comp is a subset (130k total), and according to the GM9 abstract \" These molecules correspond to the subset of all 133,885 species with up to nine heavy atoms (CONF) out of the GDB-17 chemical universe of 166 billion organic molecules.\" so whatever ML solutions are devised here using QM9 would not be applicable to predict scalar couplings for the vast majority of the GDB-17.\n\nAnd obviously the organization already knows the scalar coupling values for almost all of QM9, hence my comment, which I may move from \"less scientific value\" to \"less practical value\".", "votes": 2 }, { "id": 551549, "postDate": "2019-06-12T20:51:53.067Z", "content": "Yea, I do hope that they can provide a new or a second test dataset without these problems...and people can use whatever resource they can find.
", "rawMarkdown": "Yea, I do hope that they can provide a new or a second test dataset without these problems...and people can use whatever resource they can find." } ] }, { "id": 551492, "postDate": "2019-06-12T19:31:08.573Z", "content": "\n\n\nIf you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.
\n
So, basically, you allowed using the data, which could harm the research, which is essentially the only valuable reason for this competition? To be honest, I'm a little shocked: you could not look after compliance with the rules and the result was not long in coming:
\n\n\n\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.
\n
Don't you think that you have to stand firm on your \"No\" in such cases (not allowing the usage of data)?
", "rawMarkdown": ">If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.\n\nSo, basically, you allowed using the data, which could harm the research, which is essentially the only valuable reason for this competition? To be honest, I'm a little shocked: you could not look after compliance with the rules and the result was not long in coming: \n\n>The unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.\n\nDon't you think that you have to stand firm on your \"No\" in such cases (not allowing the usage of data)?", "votes": 4, "replies": [ { "id": 551512, "postDate": "2019-06-12T19:52:36.563Z", "content": "The data was excluded for conceptual reasons, not because there was evidence it contained any useful information.
\n\nBest case - winning teams don't use the data.
\n\nSomewhat less optimal - Winning teams use this data, but it contributes a minuscule improvement to the score. Those teams would still win, but researches would need to recreate the models without those features.
\n\nWorst case - The external data provides significant leakage to the target values. (We don't believe this to be true for multiple reasons). If this is the case, hopefully researchers would still be able glean some learnings from the competition.
\n\nSo why not include the data in the first place? As stated above, for conceptual reasons (given the insignificance of the external data). And as is often the case in competitions, we listen and react to community feedback. This particular situation was disappointing to me because of those who effectively forced the issue, rather than having a good-faith dialog.
", "rawMarkdown": "The data was excluded for conceptual reasons, not because there was evidence it contained any useful information.\n\nBest case - winning teams don't use the data.\n\nSomewhat less optimal - Winning teams use this data, but it contributes a minuscule improvement to the score. Those teams would still win, but researches would need to recreate the models without those features.\n\nWorst case - The external data provides significant leakage to the target values. (We don't believe this to be true for multiple reasons). If this is the case, hopefully researchers would still be able glean some learnings from the competition.\n\nSo why not include the data in the first place? As stated above, for conceptual reasons (given the insignificance of the external data). And as is often the case in competitions, we listen and react to community feedback. This particular situation was disappointing to me because of those who effectively forced the issue, rather than having a good-faith dialog. ", "votes": 5 } ] }, { "id": 551572, "postDate": "2019-06-12T21:49:08.560Z", "content": "\n\n\nThe obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.
\n
If I understand correctly, the scientific benefit is to predict faster. The quantum computation properties shall not be used, only because that the quantum computation takes a very long time, not because they use physical laws to do the calculations.
\n\nBut what if there exist a big dataset with all the quantum computation properties, so that you do not need to re-calculate them? If you are a researcher, not competing for a kaggle competition but rather aiming for a paper, and you know such information makes your prediction better, would you not use it? Does intentionally not using such information do any good to science?
\n\nIn my opinion, the biggest concern is that using such external dataset might undermine the generalizability of the model, while since all the molecules in the test set are from this dataset, there is no way for you to check such overfitting. My suggestion is the same as @antorsae pointed out, having a new or a second test set where all the molecules are not in the external dataset would be the best solution.
", "rawMarkdown": "> The obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.\n\nIf I understand correctly, the scientific benefit is to predict **faster**. The quantum computation properties shall not be used, only because that the quantum computation takes a very long time, not because they use physical laws to do the calculations.\n\nBut what if there exist a big dataset with all the quantum computation properties, so that you do not need to re-calculate them? If you are a researcher, not competing for a kaggle competition but rather aiming for a paper, and you know such information makes your prediction better, would you not use it? Does intentionally not using such information do any good to science?\n\nIn my opinion, the biggest concern is that using such external dataset might undermine the generalizability of the model, while since all the molecules in the test set are from this dataset, there is no way for you to check such overfitting. My suggestion is the same as @antorsae pointed out, having a new or a second test set where all the molecules are not in the external dataset would be the best solution.", "votes": 2 }, { "id": 551514, "postDate": "2019-06-12T19:55:58.647Z", "content": "I am from the scientific world and I am applying data science in my own research to improve the current methods in my own field. There are of course a lot of improvements that can be made in natural sciences using the knowledge of our fellow data scientists. However, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy. The advantage of the machine learning ways are the speed, therefore many problems can be solved in real-time and high-throughput data can be analyzed efficiently.
\n\nI believe that the aim of this competition is also to find a machine learning model that performs similarly as the state-of-the-art quantum calculation method, while being much faster than the physical way. In my opinion, no matter what external data people use, as long as such model works on every molecules, this is a good model to me.
\n\nI think the \"the spirit of the scientific challenge\" is not \"can we beat the current physical method with pure data science\", but rather \"can we make a faster and better model using whatever we can\".
", "rawMarkdown": "I am from the scientific world and I am applying data science in my own research to improve the current methods in my own field. There are of course a lot of improvements that can be made in natural sciences using the knowledge of our fellow data scientists. However, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy. The advantage of the machine learning ways are the speed, therefore many problems can be solved in real-time and high-throughput data can be analyzed efficiently.\n\nI believe that the aim of this competition is also to find a machine learning model that performs similarly as the state-of-the-art quantum calculation method, while being much faster than the physical way. In my opinion, no matter what external data people use, as long as such model works on every molecules, this is a good model to me.\n\nI think the \"the spirit of the scientific challenge\" is not \"can we beat the current physical method with pure data science\", but rather \"can we make a faster and better model using whatever we can\".", "votes": 2, "replies": [ { "id": 551517, "postDate": "2019-06-12T19:58:29.883Z", "content": "\n\n\nHowever, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy.
\n
Challenge accepted. ;)
", "rawMarkdown": "> However, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy. \n\nChallenge accepted. ;)", "votes": 12 }, { "id": 551519, "postDate": "2019-06-12T20:03:01.797Z", "content": "Watch out . . . . Bojan's the type of guy who will find a model that is both faster and more accurate!
", "rawMarkdown": "Watch out . . . . Bojan's the type of guy who will find a model that is both faster *and* more accurate!", "votes": 10 }, { "id": 551522, "postDate": "2019-06-12T20:12:43.540Z", "content": "I honestly don't know if such a thing is really possible. My own intuition would be that the \"proper\" physical properties approach would still be well ahead of the \"pure ML\" approaches. However, a combination of the two might be the best. Something like Alpha Go for instance for Physics Equations instead of rules of Go. Which would obviously require the resources well beyond what most Kagglers have access to.
", "rawMarkdown": "I honestly don't know if such a thing is really possible. My own intuition would be that the \"proper\" physical properties approach would still be well ahead of the \"pure ML\" approaches. However, a combination of the two might be the best. Something like Alpha Go for instance for Physics Equations instead of rules of Go. Which would obviously require the resources well beyond what most Kagglers have access to. ", "votes": 5 }, { "id": 551523, "postDate": "2019-06-12T20:13:00.267Z", "content": "The problem is actually that, in many cases, the training sets are often generated using the physical and computationally expensive way. Yes, there are some fields that have a lot of experimental data, but such fields are very limited and they already have a lot of data scientists working on the problems. So often the case is, the ground truth provided is from computation, so even by some magic you find a better model, you will never be able to validate it.\nAlso, while the trend of data science is to be more and more \"generic\", scientific questions are often very specific. Precision and explainability are what scientists care most about, unless they are dealing with a huge amount of data that the traditional way takes forever.
", "rawMarkdown": "The problem is actually that, in many cases, the training sets are often generated using the physical and computationally expensive way. Yes, there are some fields that have a lot of experimental data, but such fields are very limited and they already have a lot of data scientists working on the problems. So often the case is, the ground truth provided is from computation, so even by some magic you find a better model, you will never be able to validate it.\nAlso, while the trend of data science is to be more and more \"generic\", scientific questions are often very specific. Precision and explainability are what scientists care most about, unless they are dealing with a huge amount of data that the traditional way takes forever." }, { "id": 551525, "postDate": "2019-06-12T20:16:16.770Z", "content": "I agree. And I also think that reinforcement learning has the potential to beat some physical algorithms, while it is very difficult for the traditional supervised learning to do that.
", "rawMarkdown": "I agree. And I also think that reinforcement learning has the potential to beat some physical algorithms, while it is very difficult for the traditional supervised learning to do that." } ] }, { "id": 552219, "postDate": "2019-06-13T16:40:59.843Z", "content": "
I have not been around here long, but I just want to post my support for the kaggler[s] who are willing to break rules and tarnish their own image for the sake of transparency, honesty, and fairness. I'd rather have no competitions than unfair competitions, and shame on everyone who'd rather just sweep things under the rug. I read the original thread and I was very disappointed by the initial replies to that kaggler.
", "rawMarkdown": "I have not been around here long, but I just want to post my support for the kaggler[s] who are willing to break rules and tarnish their own image for the sake of transparency, honesty, and fairness. I'd rather have no competitions than unfair competitions, and shame on everyone who'd rather just sweep things under the rug. I read the original thread and I was very disappointed by the initial replies to that kaggler.", "votes": 2, "replies": [ { "id": 552049, "postDate": "2019-06-13T12:53:56.527Z", "content": "Welcome to Kaggle Travis. Over time, you'll see that these things can be handled without breaking the rules. We're very open do discussion, feedback, and course correction.
\n\n\n\n\nI'd rather have no competitions than unfair competitions
\n
Participation in Kaggle competitions has always been and will always be voluntary. Why would you do something to take away a competition from other people, just because you don't view it as fair?
", "rawMarkdown": "Welcome to Kaggle Travis. Over time, you'll see that these things can be handled without breaking the rules. We're very open do discussion, feedback, and course correction. \n\n> I'd rather have no competitions than unfair competitions\n\nParticipation in Kaggle competitions has always been and will always be voluntary. Why would you do something to take away a competition from other people, just because you don't view it as fair?", "votes": 8 }, { "id": 552145, "postDate": "2019-06-13T14:56:58.630Z", "content": "I don't think anyone did anything to take the competition away from other people, and I certainly don't want that. I was talking about what I would personally want. I know the original post here says that because of these events there will be less opportunities for scientific competitions on kaggle, but there really wasn't any explanation as to why.
\n\nHonestly, the original post here makes it seem like it would be preferred that abuse of the rules remain hidden if there isn't a clear way to address it. Otherwise I don't see why there was such opposition to nosound telling everyone about the data in the first place.
\n\nThat said, I don't want to seem purely disagreeable or like I am attacking you, because I am not. I have the utmost respect simply for the fact that there is open uncensored discussion of this topic in the first place.
", "rawMarkdown": "I don't think anyone did anything to take the competition away from other people, and I certainly don't want that. I was talking about what I would personally want. I know the original post here says that because of these events there will be less opportunities for scientific competitions on kaggle, but there really wasn't any explanation as to why.\n\nHonestly, the original post here makes it seem like it would be preferred that abuse of the rules remain hidden if there isn't a clear way to address it. Otherwise I don't see why there was such opposition to nosound telling everyone about the data in the first place.\n\nThat said, I don't want to seem purely disagreeable or like I am attacking you, because I am not. I have the utmost respect simply for the fact that there is open uncensored discussion of this topic in the first place.", "votes": 4 }, { "id": 552168, "postDate": "2019-06-13T15:34:34.180Z", "content": "> Honestly, the original post here makes it seem like it would be preferred that abuse of the rules remain hidden if there isn't a clear way to address it.
\n\nThis is exactly how I feel about it. QM9 is simply out there in the internet, being accessible to everyone, and nosound just posted about it making everyone know this issue. Yes, you could not use it in the final submission according to the old rule, but how is Kaggle going to monitor that? He brought this question up to the table and it was indeed a tricky problem. It seems like the admin would simply prefer no one ever found this issue so that he can pretend that it does not exist, got mad and being sarcasm when someone publicly talked about this issue, accusing him/her for the possible outcome of having less scientific competitions (which does not make any sense), and encouraging other Kagglers attack him/her and whoever speaks for him/her. This is becoming personal and it should not be.
", "rawMarkdown": "> Honestly, the original post here makes it seem like it would be preferred that abuse of the rules remain hidden if there isn't a clear way to address it.\n\nThis is exactly how I feel about it. QM9 is simply out there in the internet, being accessible to everyone, and nosound just posted about it making everyone know this issue. Yes, you could not use it in the final submission according to the old rule, but how is Kaggle going to monitor that? He brought this question up to the table and it was indeed a tricky problem. It seems like the admin would simply prefer no one ever found this issue so that he can pretend that it does not exist, got mad and being sarcasm when someone publicly talked about this issue, accusing him/her for the possible outcome of having less scientific competitions (which does not make any sense), and encouraging other Kagglers attack him/her and whoever speaks for him/her. This is becoming personal and it should not be.", "votes": 5 }, { "id": 552243, "postDate": "2019-06-13T17:06:49.097Z", "content": "Travis wrote:
\n\n\n\n\nI don't see why there was such opposition to nosound telling everyone about the data in the first place.
\n
The issue has never been telling people about any data, discussing the data, etc. It was about using the data to make predictions. It's perfectly fine to say, \"I found this data that isn't supposed so be used. I'm concerned that it could be abused.\" It's against the rules say, \"I found this data that isn't supposed to be used. I submitted a model using the forbidden data and it improved my leaderboard score. Here's a kernel how you can do the same.\"
\n\nJosef wrote:
\n\n\n\n\nYes, you could not use it in the final submission according to the old rule
\n
Please stop repeating this. I've explained many times that there has never been a rule at Kaggle where you could use forbidden data as long as you did not use it in the final submission. By continuing to make this claim, you are confusing the issue.
\n\nLet me be very clear here - I will not continue to tolerate you spreading incorrect information about what has already been spelled out repeatedly and clearly, nor will I tolerate you making claims that I am encouraging anyone in the community to attack another member.
", "rawMarkdown": "Travis wrote:\n> I don't see why there was such opposition to nosound telling everyone about the data in the first place.\n\nThe issue has never been telling people about any data, discussing the data, etc. It was about using the data to make predictions. It's perfectly fine to say, \"I found this data that isn't supposed so be used. I'm concerned that it could be abused.\" It's against the rules say, \"I found this data that isn't supposed to be used. I submitted a model using the forbidden data and it improved my leaderboard score. Here's a kernel how you can do the same.\"\n\nJosef wrote:\n> Yes, you could not use it in the final submission according to the old rule\n\nPlease stop repeating this. I've explained many times that there has never been a rule at Kaggle where you could use forbidden data as long as you did not use it in the final submission. By continuing to make this claim, you are confusing the issue.\n\nLet me be very clear here - I will not continue to tolerate you spreading incorrect information about what has already been spelled out repeatedly and clearly, nor will I tolerate you making claims that I am encouraging anyone in the community to attack another member.", "votes": 1 }, { "id": 552266, "postDate": "2019-06-13T17:43:50.653Z", "content": "I don't want to play word game here. I want to be very clear as well - you are a grandmaster at Kaggle and you have made a lot of contribution to this community, and I truly respect that. However, it is not anyone's fault, if not you, that the rules are not explicit and leave room for misinterpretation. It is not anyone's responsibility to be aware of your out-of-office schedule, unless you pinned it on the top of the discussion board.
\n\nPeople come to Kaggle to learn, to compete, to meet new people, not to ruin the competition you hosted. Did nosound do things with the optimal way? Of course no. But was his/her intention to ruin your competition, to taunt the organizers? He/She removed the kernel instantly after you expressed your point, acknowledging that this was a misinterpretation of the rules. He/She tried to mend it, to minimize the effect it caused. On the other hand, you constantly, even after a few days now, accused nosound for his/her action in the public discussion board, implied that he/she should be blamed if there are fewer scientific competition, upvoted other person's harsh comment on him/her. Whatever kind of claims you think this is, this is all true and you know it.
\n\nI do not know you in person, and I don't think I will have the chance in the future. There is no point for me, or any other Kagglers here to intentionally attack you behind a computer screen, especially with the fact that you are not just an ordinary Kaggler, but an official representative of this competition. What nosound did was not optimal, but what you have been doing against him/her was totally unnecessary and was not helpful to this community in any way. Given what he/she did and the way he/she responded, nosound does not deserve the treatment that he/she has been experienced in the past few days.
\n\nAs I repeated a lot of times, there is no point to make this personal.
", "rawMarkdown": "I don't want to play word game here. I want to be very clear as well - you are a grandmaster at Kaggle and you have made a lot of contribution to this community, and I truly respect that. However, it is not anyone's fault, if not you, that the rules are not explicit and leave room for misinterpretation. It is not anyone's responsibility to be aware of your out-of-office schedule, unless you pinned it on the top of the discussion board. \n\nPeople come to Kaggle to learn, to compete, to meet new people, not to ruin the competition you hosted. Did nosound do things with the optimal way? Of course no. But was his/her intention to ruin your competition, to taunt the organizers? He/She removed the kernel instantly after you expressed your point, acknowledging that this was a misinterpretation of the rules. He/She tried to mend it, to minimize the effect it caused. On the other hand, you constantly, even after a few days now, accused nosound for his/her action in the public discussion board, implied that he/she should be blamed if there are fewer scientific competition, upvoted other person's harsh comment on him/her. Whatever kind of claims you think this is, this is all true and you know it.\n\nI do not know you in person, and I don't think I will have the chance in the future. There is no point for me, or any other Kagglers here to intentionally attack you behind a computer screen, especially with the fact that you are not just an ordinary Kaggler, but an official representative of this competition. What nosound did was not optimal, but what you have been doing against him/her was totally unnecessary and was not helpful to this community in any way. Given what he/she did and the way he/she responded, nosound does not deserve the treatment that he/she has been experienced in the past few days.\n\nAs I repeated a lot of times, there is no point to make this personal." }, { "id": 552276, "postDate": "2019-06-13T17:55:53.270Z", "content": "\n\n\nIt's perfectly fine to say, \"I found this data that isn't supposed so be used. I'm concerned that it could be abused.\" It's against the rules say, \"I found this data that isn't supposed to be used. I submitted a model using the forbidden data and it improved my leaderboard score. Here's a kernel how you can do the same.\"
\n
Same for you, it is perfectly fine to say, \"You should take your kernel down immediately since it is against the rules. I understand that there might be a misunderstanding of the rules, and we are all good if you obey the rules from now on.\" It is totally unnecessary to say, \"You should be blamed if we have fewer scientific competitions. Guys, it's nosound who makes this happen.\"
", "rawMarkdown": "> It's perfectly fine to say, \"I found this data that isn't supposed so be used. I'm concerned that it could be abused.\" It's against the rules say, \"I found this data that isn't supposed to be used. I submitted a model using the forbidden data and it improved my leaderboard score. Here's a kernel how you can do the same.\"\n\nSame for you, it is perfectly fine to say, \"You should take your kernel down immediately since it is against the rules. I understand that there might be a misunderstanding of the rules, and we are all good if you obey the rules from now on.\" It is totally unnecessary to say, \"You should be blamed if we have fewer scientific competitions. Guys, it's nosound who makes this happen.\"", "votes": 1 }, { "id": 552293, "postDate": "2019-06-13T18:13:59.520Z", "content": "Putting all the nosound issues aside, I am still not sure what Kaggle wants to achieve by these scientific challenges. I have seen many many published papers in the scientific world that use data science methods not to make any real, practical improvements, but to simply use it \"because we can and it is sexy science\". Such tools are just one-paper things, no one is going to actually use them and they just lie on your resume because they are good-looking. If Kaggle's intention is to host a fair, but domain-knowledge free (in order to attract more people and being more \"fair\") competition, while does not really care what scientists really want, I totally understand it, and that also answers many questions I have. However if Kaggle seriously wants to push science forward using the wisdom of all these data scientist users, sometimes the organizers have to think twice and discuss more with the competitors when setting limitations and designing the competition, and sometimes it is necessary to compromise the \"fairness\" because not everyone are on the same starting line when it comes to the domain knowledge.
", "rawMarkdown": "Putting all the nosound issues aside, I am still not sure what Kaggle wants to achieve by these scientific challenges. I have seen many many published papers in the scientific world that use data science methods not to make any real, practical improvements, but to simply use it \"because we can and it is sexy science\". Such tools are just one-paper things, no one is going to actually use them and they just lie on your resume because they are good-looking. If Kaggle's intention is to host a fair, but domain-knowledge free (in order to attract more people and being more \"fair\") competition, while does not really care what scientists really want, I totally understand it, and that also answers many questions I have. However if Kaggle seriously wants to push science forward using the wisdom of all these data scientist users, sometimes the organizers have to think twice and discuss more with the competitors when setting limitations and designing the competition, and sometimes it is necessary to compromise the \"fairness\" because not everyone are on the same starting line when it comes to the domain knowledge.", "votes": 2 }, { "id": 552361, "postDate": "2019-06-13T19:51:44.043Z", "content": "Josef wrote:
\n\n\n\n\nI have seen many many published papers in the scientific world that use data science methods not to make any real, practical improvements, but to simply use it \"because we can and it is sexy science\".
\n
Finally, something we can totally agree on!
\n\nI was at the APS DFD meeting last year. IMO, most of the \"using machine learning in fluid dynamics\" presentations were pretty lame . . . basically using a fancy curve fit for a small dataset. On the other hand, there was one that was absolutely amazing . . . using proper orthogonal decomposition and LSTMs to model the evolution of turbulent flow.
\n\nIt's my personal interest and preference to get more scientific competitions on Kaggle. No doubt, some will be underwhelming. But I'm hoping it's net-positive for the community and for science.
\n\n\n\n\nsometimes the organizers have to think twice
\n
I can assure you, there is a good deal of thinking that goes on prior to a competition launch. The community sees the 1 thing that slips through. They don't see 99 that didn't. You can think of a competition launch in terms of software development. We do everything we can to minimize bugs before launch. But when you have a thousand people scrutinizing your work, something is bound to be discovered.
", "rawMarkdown": "Josef wrote:\n> I have seen many many published papers in the scientific world that use data science methods not to make any real, practical improvements, but to simply use it \"because we can and it is sexy science\". \n\nFinally, something we can totally agree on!\n\nI was at the APS DFD meeting last year. IMO, most of the \"using machine learning in fluid dynamics\" presentations were pretty lame . . . basically using a fancy curve fit for a small dataset. On the other hand, there was one that was absolutely amazing . . . using proper orthogonal decomposition and LSTMs to model the evolution of turbulent flow.\n\nIt's my personal interest and preference to get more scientific competitions on Kaggle. No doubt, some will be underwhelming. But I'm hoping it's net-positive for the community and for science.\n\n> sometimes the organizers have to think twice\n\nI can assure you, there is a good deal of thinking that goes on prior to a competition launch. The community sees the 1 thing that slips through. They don't see 99 that didn't. You can think of a competition launch in terms of software development. We do everything we can to minimize bugs before launch. But when you have a thousand people scrutinizing your work, something is bound to be discovered.", "votes": 3 }, { "id": 552375, "postDate": "2019-06-13T20:46:42.393Z", "content": "Sometimes it's not about how big are the dataset, or how complex/novel the model is. Many papers lack the original scientific motivation. A good logic is, \"the traditional ways have these limitations and we proved that data science can help us overcome these problems\". But often what I see in the scientific world is, people use fancy methods to solve a non-existed problem. Many ML tools people developed have old, and sometimes even better alternatives with the traditional method. Some other ML tools look cool and try to solve problems, but they actually fail doing so and become a mediocre.
\n\nI was in an industry-oriented conference earlier this year, with all the big pharma/bioinfo and some smaller companies attending. Nearly everyone talks about AI all the time. There was a live poll during a panel discussion, the question was \"For the most successful project that used deep neural networks in your company or group, what % improvement did you find over more traditional methods?\" Nearly a half of the people chose \"10% or less\". And this was an industry conference, academia would just be worse.
\n\nPeople outside of DS field either think DS/ML/AI as a panacea or a worthless trick. The advertise they get from the media, i.e. \"how AI has changed our lives\", \"AI beats human again\", makes them think AI is good at everything and it just happens magically. When they try to use DS/ML/AI in their own work (not developing a model, just try a model), they are disappointed because usually the model cannot solve the problems they have for various reasons.
\n\nComing back to the scientific world, I am very okay if Kaggle wants to exclude as much as domain knowledge as possible when holding scientific challenges. But doing so is not always great for the improvement of science. Models developed here will most likely become a method paper in the end. A good question to ask is, if not competing in a Kaggle challenge but purely developing a model without any limitations, can I create a better model that also fulfill the needs? If so, why setting the limitations?
\n\n> I can assure you, there is a good deal of thinking that goes on prior to a competition launch.
\n\nI totally believe you. It is not trivial to hold such challenges and I appreciate your work. However, the 1 thing slipped through to the community is also important, and actually a lot of serious discussions can be made and it could even be beneficial. Most of the people here are to learn and help the community, not to undermine it. All these \"scrutinizing\" actually play a similar role as a \"peer-review\", especially when people start to discuss the motivation of a competition, \"what the sponsor really want and what can we do to make it useful, instead of just good score and looking fancy\".
", "rawMarkdown": "Sometimes it's not about how big are the dataset, or how complex/novel the model is. Many papers lack the original scientific motivation. A good logic is, \"the traditional ways have these limitations and we proved that data science can help us overcome these problems\". But often what I see in the scientific world is, people use fancy methods to solve a non-existed problem. Many ML tools people developed have old, and sometimes even better alternatives with the traditional method. Some other ML tools look cool and try to solve problems, but they actually fail doing so and become a mediocre.\n\nI was in an industry-oriented conference earlier this year, with all the big pharma/bioinfo and some smaller companies attending. Nearly everyone talks about AI all the time. There was a live poll during a panel discussion, the question was \"For the most successful project that used deep neural networks in your company or group, what % improvement did you find over more traditional methods?\" Nearly a half of the people chose \"10% or less\". And this was an industry conference, academia would just be worse.\n\nPeople outside of DS field either think DS/ML/AI as a panacea or a worthless trick. The advertise they get from the media, i.e. \"how AI has changed our lives\", \"AI beats human again\", makes them think AI is good at everything and it just happens magically. When they try to use DS/ML/AI in their own work (not developing a model, just try a model), they are disappointed because usually the model cannot solve the problems they have for various reasons.\n\nComing back to the scientific world, I am very okay if Kaggle wants to exclude as much as domain knowledge as possible when holding scientific challenges. But doing so is not always great for the improvement of science. Models developed here will most likely become a method paper in the end. A good question to ask is, if not competing in a Kaggle challenge but purely developing a model without any limitations, can I create a better model that also fulfill the needs? If so, why setting the limitations?\n\n> I can assure you, there is a good deal of thinking that goes on prior to a competition launch.\n\n\nI totally believe you. It is not trivial to hold such challenges and I appreciate your work. However, the 1 thing slipped through to the community is also important, and actually a lot of serious discussions can be made and it could even be beneficial. Most of the people here are to learn and help the community, not to undermine it. All these \"scrutinizing\" actually play a similar role as a \"peer-review\", especially when people start to discuss the motivation of a competition, \"what the sponsor really want and what can we do to make it useful, instead of just good score and looking fancy\"." }, { "id": 553147, "postDate": "2019-06-15T07:35:58.993Z", "content": "Ponder you want to test 100 G molecule configurations for certain characteristics. Your ML might give you 100 k suggestions which you then run your extended QM on?
", "rawMarkdown": "Ponder you want to test 100 G molecule configurations for certain characteristics. Your ML might give you 100 k suggestions which you then run your extended QM on?", "votes": 1 } ] }, { "id": 573645, "postDate": "2019-07-12T14:52:53.947Z", "content": "\n\n\nTo address this current competition, we are updating the rules to allow the use of QM7 and QM9 published data, if participants wish to use it. Participants still may not use quantum calculations to obtain predictions.
\n
Do you mean we can use QM7 and QM9 for training ML models ? I understand we should not use them for test data. But I'm interested in utilizing them for training and approximating their relationship.
", "rawMarkdown": "> To address this current competition, we are updating the rules to allow the use of QM7 and QM9 published data, if participants wish to use it. Participants still may not use quantum calculations to obtain predictions.\n\nDo you mean we can use QM7 and QM9 for training ML models ? I understand we should not use them for test data. But I'm interested in utilizing them for training and approximating their relationship.", "votes": 1 }, { "id": 551499, "postDate": "2019-06-12T19:36:56.780Z", "content": "\n\n\nOnce the competition finishes, CHAMPS would like to invite the top teams to present their work, discuss the details of their models, and work with them to write a joint research publication which discusses an open-source implementation of the solution.
\n
My understanding of this sentence is that you will discuss with each top team their approach. Thus, if you find any usage of quantum mechanics features (looking at the code) can't you just disqualify that team? Also, before that discussion you can ask to have access to source code to analyse it.
\n\nBesides that, like @areveillon said, if a team decided not to use quantum mechanics features what will happen? IMO and since the goal is to predict scalar coupling without the expensive quantum calculations you should focus on those approaches and do not allow QMs (from any source). In the end you can assess each top team to validate it (I now it can be cumbersome and a lot of work, but it is what it is until better way to evaluate the usage of such features)
", "rawMarkdown": "> Once the competition finishes, CHAMPS would like to invite the top teams to present their work, discuss the details of their models, and work with them to write a joint research publication which discusses an open-source implementation of the solution.\n\nMy understanding of this sentence is that you will discuss with each top team their approach. Thus, if you find any usage of quantum mechanics features (looking at the code) can't you just disqualify that team? Also, before that discussion you can ask to have access to source code to analyse it.\n\nBesides that, like @areveillon said, if a team decided not to use quantum mechanics features what will happen? IMO and since the goal is to predict scalar coupling without the expensive quantum calculations you should focus on those approaches and do not allow QMs (from any source). In the end you can assess each top team to validate it (I now it can be cumbersome and a lot of work, but it is what it is until better way to evaluate the usage of such features)", "votes": 1, "replies": [ { "id": 551516, "postDate": "2019-06-12T19:56:43.553Z", "content": "That's a great question. In this case, it would be impossible to know whether a team used the external data to, e.g., tune model hyper parameters without explicitly using them in the model. So even inspecting the final models wouldn't help.
\n\nBetter to just allow the data and end the drama. I have a feeling when it's all said and done, there are going to be many people who realize how overblown this is (since there is currently no evidence, practical or theoretical, that this data is helpful).
", "rawMarkdown": "That's a great question. In this case, it would be impossible to know whether a team used the external data to, e.g., tune model hyper parameters without explicitly using them in the model. So even inspecting the final models wouldn't help.\n\nBetter to just allow the data and end the drama. I have a feeling when it's all said and done, there are going to be many people who realize how overblown this is (since there is currently no evidence, practical or theoretical, that this data is helpful).", "votes": 2 } ] }, { "id": 553808, "postDate": "2019-06-16T11:26:42.783Z", "content": "@inversion very appreciate your effort 🥇. It's really great thoughts.
", "rawMarkdown": "@inversion very appreciate your effort 🥇. It's really great thoughts.", "votes": 2 }, { "id": 552904, "postDate": "2019-06-14T19:25:20.200Z", "content": "I am looking forward for your excellent ideas....!!!
", "rawMarkdown": "I am looking forward for your excellent ideas....!!! ", "votes": 2 }, { "id": 551481, "postDate": "2019-06-12T19:08:46.767Z", "content": "Thanks for this clarification.
\n\nYou said that there is little to no benefits of using the QM7 QM9 data set.
\n\nEven if the benefit is small, does that mean that all Kagglers who are not using it will have an handicap? I'm not planning on using it so I would like to make sure it still make sense to do the comp without it.
\n\nThanks!
", "rawMarkdown": "Thanks for this clarification.\n\nYou said that there is little to no benefits of using the QM7 QM9 data set. \n\nEven if the benefit is small, does that mean that all Kagglers who are not using it will have an handicap? I'm not planning on using it so I would like to make sure it still make sense to do the comp without it.\n\nThanks!", "votes": 2, "replies": [ { "id": 551485, "postDate": "2019-06-12T19:18:48.607Z", "content": "@areveillon \nClearly the QM7 is useless as atomization energies have nothing to do with J coupling (which is an atomic spin interaction mediated by electrons).
\n\nAccording to the website, QM9 has:\n- geometries: useless\n- harmonic frequencies: describes vibration/rotation of the molecule. Useless\n- dipole moments: this may help a bit as it is an electronic property.\n- polarizabilities: this is related to the response of the molecule under an electric field. Probably useless.\n- energies, enthalpies, and free energies of atomization: energies/enthalpies may be used but it won't help much.
", "rawMarkdown": "@areveillon \nClearly the QM7 is useless as atomization energies have nothing to do with J coupling (which is an atomic spin interaction mediated by electrons).\n\nAccording to the website, QM9 has:\n- geometries: useless\n- harmonic frequencies: describes vibration/rotation of the molecule. Useless\n- dipole moments: this may help a bit as it is an electronic property.\n- polarizabilities: this is related to the response of the molecule under an electric field. Probably useless.\n- energies, enthalpies, and free energies of atomization: energies/enthalpies may be used but it won't help much.", "votes": 11 }, { "id": 551498, "postDate": "2019-06-12T19:36:24.160Z", "content": "@areveillon - This is something you'll have do decide on your own. I can tell you that I personally used this information in a model and the feature importance was in the long tail of irrelevance. The data was excluded for conceptual reasons, not because there was evidence it contained any useful information.
", "rawMarkdown": "@areveillon - This is something you'll have do decide on your own. I can tell you that I personally used this information in a model and the feature importance was in the long tail of irrelevance. The data was excluded for conceptual reasons, not because there was evidence it contained any useful information.", "votes": 9 }, { "id": 551500, "postDate": "2019-06-12T19:42:54.960Z", "content": "Thanks for your answers!
", "rawMarkdown": "Thanks for your answers!", "votes": 1 }, { "id": 551796, "postDate": "2019-06-13T05:59:04.957Z", "content": "From the kernel:
\n\n\n\n\n\n
\n- \n
dipole_moments.csvcontains X,Y,Z values per molecule and I found thatsqrt(X^2+Y^2+Z^2)=muwhere mu is given in QM9- \n
mulliken_charges.csvmatches the mulliken charges from QM9- \n
scalar_coupling_contributions.csvI can't find this info in QM9- \n
magnetic_shielding_tensors.csvI can't find this info in QM9
1 & 2 can be calculated using existing models/packages that have already been shared on the forums/kernels. 3 & 4 cannot and aren't in QM9. So as @inversion said, it's likely that QM9 won't make much of a difference if any - which is good! 😄
", "rawMarkdown": "From the [kernel](https://www.kaggle.com/zaharch/quantum-machine-9-qm9):\n> 1. `dipole_moments.csv` contains X,Y,Z values per molecule and I found that `sqrt(X^2+Y^2+Z^2)=mu` where mu is given in QM9\n2. `mulliken_charges.csv` matches the mulliken charges from QM9\n3. `scalar_coupling_contributions.csv` I can't find this info in QM9\n4. `magnetic_shielding_tensors.csv` I can't find this info in QM9\n\n1 & 2 can be calculated using existing models/packages that have already been shared on the forums/kernels. 3 & 4 cannot and aren't in QM9. So as @inversion said, it's likely that QM9 won't make much of a difference if any - which is good! 😄 \n", "votes": 2 } ] }, { "id": 558681, "postDate": "2019-06-22T18:30:28.610Z", "content": "I might be missing something obvious, but..I really don't get why setting the \"NO QM\" rule in first place? - and tell me exactly what that means, cause almost everything here can be labeled as QM.
\n\nThe challenge is to deliver a quick solution to a problem that nowadays takes too much time and too many PhDs to be addressed.
", "rawMarkdown": "I might be missing something obvious, but..I really don't get why setting the \"NO QM\" rule in first place? - and tell me exactly what that means, cause almost everything here can be labeled as QM. \n\nThe challenge is to deliver a quick solution to a problem that nowadays takes too much time and too many PhDs to be addressed. " }, { "id": 558508, "postDate": "2019-06-22T13:28:54.127Z", "content": "\n\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.
\n
😩
", "rawMarkdown": "> The unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.\n\n😩" }, { "id": 552830, "postDate": "2019-06-14T15:59:40.737Z", "content": "Are scalar coupling constants for the private set available in the public domain or locked on organizers' hard drive? I am trying to anticipate the next leak. I am guessing it is the former.
", "rawMarkdown": "Are scalar coupling constants for the private set available in the public domain or locked on organizers' hard drive? I am trying to anticipate the next leak. I am guessing it is the former." }, { "id": 551884, "postDate": "2019-06-13T08:30:41.420Z", "content": "I don't want to be boring, but I see that kind of situation happening a lot of time in Kaggle's competitions where external data can be used and thus creates a kind of unfairness situation between competitors who play by the rules and those who don't.
\n\nThe only way is to set up a true evaluation method based on the code ! So for the private and final leaderboard, each competitor gives the code which has to take the data in input and output the prediction. In that way only we are sure that no external data can be used. Indeed, if your code is based on external data it would not be runnable without it so the code would not give any prediction.\nIt's as simple as that, even though I understand it is harder to set up this kind of evaluation method.
", "rawMarkdown": "I don't want to be boring, but I see that kind of situation happening a lot of time in Kaggle's competitions where external data can be used and thus creates a kind of unfairness situation between competitors who play by the rules and those who don't. \n\nThe only way is to set up a true evaluation method based on the code ! So for the private and final leaderboard, each competitor gives the code which has to take the data in input and output the prediction. In that way only we are sure that no external data can be used. Indeed, if your code is based on external data it would not be runnable without it so the code would not give any prediction.\nIt's as simple as that, even though I understand it is harder to set up this kind of evaluation method.", "replies": [ { "id": 552035, "postDate": "2019-06-13T12:40:47.330Z", "content": "They are not banning all external data, actually I agree with what @jpmiller said, \"excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science\". Data science is more than just finding a good model, it also requires understanding the domain knowledge, data mining, preprocessing, etc. And I believe it is not very practical for Kaggle to run the codes of even a small part of Kagglers, since one specific model may require hours of training. What you want is probably a kernel competition, yes it is more fair, but it has other problems as well.
", "rawMarkdown": "They are not banning all external data, actually I agree with what @jpmiller said, \"excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science\". Data science is more than just finding a good model, it also requires understanding the domain knowledge, data mining, preprocessing, etc. And I believe it is not very practical for Kaggle to run the codes of even a small part of Kagglers, since one specific model may require hours of training. What you want is probably a kernel competition, yes it is more fair, but it has other problems as well." }, { "id": 552355, "postDate": "2019-06-13T19:43:40.683Z", "content": "Yes I think you're right, external data should be allowed. Yet in that competition, the purpose is to give an alternative to heavy quantum computing. So giving a model which uses QM data is a nonsense.
", "rawMarkdown": "Yes I think you're right, external data should be allowed. Yet in that competition, the purpose is to give an alternative to heavy quantum computing. So giving a model which uses QM data is a nonsense. " }, { "id": 552401, "postDate": "2019-06-13T22:15:47.343Z", "content": "I think it depends on the goal of the challenge. The QM9 data is from quantum computing, but it is also a pre-calculated dataset that is available to the public. If one can learn some meaningful things from this dataset, why not?
\n\nConsider two models, 1) a pure data science model that is fast but not super accurate, and 2) a model that is equally fast, with better accuracy, but it used some quantum properties during the training. If not competing for a Kaggle challenge, you just simply want a good tool, which one would you choose?
\n\nThe potential caveat of QM9 is that although being a big dataset, there are still a lot of molecules outside QM9, and a good model should preform equally well on all the molecules. The problem now is it seems that all the molecules in the test set are in QM9 as well, making it hard to evaluate the generability of the model.
", "rawMarkdown": "I think it depends on the goal of the challenge. The QM9 data is from quantum computing, but it is also a pre-calculated dataset that is available to the public. If one can learn some meaningful things from this dataset, why not?\n\nConsider two models, 1) a pure data science model that is fast but not super accurate, and 2) a model that is equally fast, with better accuracy, but it used some quantum properties during the training. If not competing for a Kaggle challenge, you just simply want a good tool, which one would you choose?\n\nThe potential caveat of QM9 is that although being a big dataset, there are still a lot of molecules outside QM9, and a good model should preform equally well on all the molecules. The problem now is it seems that all the molecules in the test set are in QM9 as well, making it hard to evaluate the generability of the model." } ] }, { "id": 553288, "postDate": "2019-06-15T12:29:07.753Z", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": 1 }, { "id": 554216, "postDate": "2019-06-17T06:40:26.310Z", "content": "Thanks for the clarifications!
", "rawMarkdown": "Thanks for the clarifications!" } ], "comments": [ { "id": 558482, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2019-06-22T12:02:59.237000", "content": "A couple of points
\n\n1st of all - cheating is irrelevant to the scientific competitions - it happens everywhere and in all competitions, so you should not feel bad about this @inversion . This is not exclusive to scientific competitions. Private sharing is off the charge in every competition and you can definitely see patterns on the leaderboards.
\n\n2nd - No matter how good your system is in detecting violations, you will never be able to capture them all when the system is so broad, fluid and open in using external data, linking data-sets, copying the works of others. Essentially you welcome this behaviour.
\n\n3rd - something that I was afraid of since the beginning when kaggle introduced GMs from kernels and discussions is that people are gamifying the system to get the rankings at the cost of the competitions' credibility. This quite often leads to oversharing, people releasing high scoring kernel at the end of competitions (because they know they cannot get the competitions' kaggle medals, they aim for discussion/kernel medals). I need to highlight that I don't discredit the value of GMs in kernels and discussions, especially when they achieved this naturally without gamifying the system . In my opinion they are more valuable than competitions' GMs to the kaggle community. But gamification has changed things.
\n\n4th. Once its out there... then its out there! e.g once you share a \"forbidden\" dataset and people use it, all become co-conspirators. To properly assess who was using the dataset before and after is not feasible (at maximum accuracy).
\n\n5th- the rules are quite often NOT clear and you might have to read 5-10 different threads and discussions to be up to date with what is allowed and what's not. In the recent Sound classification competition I was using imagenet until before the merge deadline and only then I realised that it was not allowed (and i was actually lucky to find that). The rules (or their interpretations) are scattered everywhere!
6th- Changing the rules midway is never a good strategy. This reinforces the behaviour of cheating and I appreciate you say that people will be removed/banned, but I think this cannot happen optimally (IMO), nor I think we should fully blame the person who shared this. Kaggle is partly to blame for that.
\n\nBased on these, I have a few suggestions. I am not saying these are the perfect solutions to these problems (and some may sound a bit rough), but at least can be food for thought.
\n\n1) Less ranking points (in all competitions, discussions, kernels) for more \"dangerous\" competitions (as in prone to be hacked with lots of external information about them) . \n2) Some solo (or not) competitions with disabled kernels/discussions, disabling external data. This can make things clearer of who is doing what. Generally discussions and kernels should be blocked something like 14 days before the competition ends .\n3) EXTRA Points' penalties for cheating (not only the ones you lose if you get removed)- this will force people to be more careful. Maybe ban/block accounts for certain days if they do it consistently (like yellow card, then red card). \n4) Give clear examples of what is allowed and what is not allowed in the competitions' rules so that it is easy for people to follow. Like for example , \"you cannot use this, but you could use that \". You have done this in a few competitions as bullet points , but in my opinion, it needs to be more thorough. \n5) There needs to be some form of regulation on these private forums that keep sharing stuff (some quite large). I know it cannot be perfect, but every little helps. Also there should be people ( even volunteers) that take away what is being discussed there and bringing it into kaggle to make it fair for everybody. Maybe kaggle can give some extra prizes for these behaviours an an extra incentive?\n6) More competitions with fully annonymized data and descriptions.\n7) If competitions are very prone to cheating, maybe consider making them Masters' only?\n8) less competitions where data is public (or more can be found easily) .
\n\nAnd a few closing thoughts:
\n\nKudos to kaggle for creating a strong brand around its ranking system. Violations happen, because the ranks are valuable. I think kaggle needs to continue protecting what it has built. To be fair, even if the evaluation system has flaws (and is prone to cheating), it is much better than anything else I 've seen out there trying to do something similar. What I am trying to say is that this is not an easy task.
\n\nI think that kaggle historically has done well to adapt to new situations and address them, so I am sure they will come up with a better formula around this.
", "votes": 36, "replies": [ { "id": 558578, "author_name": "David J. Slate", "author_url": "", "post_date": "2019-06-22T15:32:43.607000", "content": "Good post, Μαριος Μιχαηλιδης. I like idea \"6) More competitions with fully anonymized data and descriptions.\". I wonder if it would have been possible to do that for this competition: to so disguise the data that the participants wouldn't even know it was a chemistry problem.
", "votes": 5, "replies": [] }, { "id": 558582, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2019-06-22T15:38:34.737000", "content": "4) Give clear examples of what is allowed [...]
\n\nAgree. On Kaggle it need not be authorative and fancy -- clear thorough explanations will suffice.
\n\n7) [...] making them Masters' only?
\n\nHow about having three rankings for all competitions? 1) Public, 2) Masters' and 3) Private?
", "votes": 2, "replies": [] }, { "id": 558856, "author_name": "Chinta", "author_url": "", "post_date": "2019-06-23T03:37:32.700000", "content": "I like the 14 days suggestion - 2) above. Or may be allow 50% duration of competition time for kernel sharing, so that both kernel enthusiasts and competetion enthusiasts have fair chance.
", "votes": 1, "replies": [] }, { "id": 558861, "author_name": "Kishore M", "author_url": "", "post_date": "2019-06-23T03:50:31.833000", "content": "Agree with you @KazAnova , adding to what you have mentioned . few suggestions from my end.
1. there should be a kernels deadline let us say a month before the final submission dead line .
And after this date all new kernels will become private and people shouldnt be able to mark public.
After the competition is over kernels can be marked public.
2. Competitions where cheating can be possible with external data , split the competition in two. First one a strict one with no external data and make top x number from this LB eligible for second competition where external data can be allowed .
3. Is it possible to make a strict rule that external data has to be disclosed atleast 3 weeks before the end date ?cases where data is collected by scraping or other means they should upload the data. either them or kaggle make it public here.
@dslate You are absolutely right in this case. It's impossible to both have anonymous molecules and provide their structure.
", "votes": 1, "replies": [] } ] }, { "id": 551501, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2019-06-12T19:46:03.403000", "content": "\n\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.
\n
This really pisses me off to no end. I was really hoping to see more relevant and competitive science challenges on Kaggle. In addition to all the other high-level cheating that has become rampant on Kaggle, I almost feel like giving up on competitions altogether. \nHere is just one more suggestion about how to deal with this going forward: create invitation-only Kaggle challenges. Invite only Kagglers with no history of impropriety or appearance of impropriety.
", "votes": 27, "replies": [ { "id": 551761, "author_name": "something4kag", "author_url": "", "post_date": "2019-06-13T04:55:54.983000", "content": "Kaggle Social Credit Score maybe is needed. \nRespect to @inversion for giving so much oxygen to this, but for a scientific competition caving in to blatant violations of the rules is disappointing. The use of QM7, QM9 or any other dataset also has to consider if that is allowed public use. Some datasets or software may allow private research or non commercial use, but one objective of the competition - to be able to use for developing medicines, would have to consider public license/copyright.
\nBelow Two Sigma competition was mentioned, it included in the rules:
By entering a Submission, you represent and warrant that … your Submission:\n\n does not contain confidential information or trade secrets and is not\n the subject of a registered patent or pending patent application\n does not trigger any reporting or royalty obligation to any third party;\nA breach of any of these warranties will result in the corresponding Submission being invalid in addition to any other remedies available to Competition Sponsor.
\n\nPerhaps in future competitions have to include similar caveats, litigation may be the only way for rules to be understood and accepted.
", "votes": 2, "replies": [] }, { "id": 554642, "author_name": "iamctR", "author_url": "", "post_date": "2019-06-17T20:05:43.850000", "content": "I can't agree more. iNaturalist 2019 at FGVC6 is my first try in Kaggle, but when I looked at the leaderboard of that competition at that moment... it shocked me. Competition Novice dominated the leaderboard, (at least for top 30). I cannot exclude the possibility that novice could have excellent machine learning knowledge and achieve high LB score, but it sounds weird for me.\nI checked the learderboard of that competition just now, most of the novice account disappeared... How do I know? Because I remember these novice accounts like to use famous actors' name, such as 'KunKun','ycy', as their user name, and it impressed me at that moment...\nIn my point of view, I prefer to consider it to be a kind of cheating... creating more accounts to use Kaggle cluster, but I have no evidence.
", "votes": 1, "replies": [] } ] }, { "id": 551483, "author_name": "Vettejeep", "author_url": "", "post_date": "2019-06-12T19:11:13.033000", "content": "@inversion: Thank you for such a clear statement about the rules. Allowing use of QM7 and QM9 data may not be so bad. It seems papers on this go back a few years, so there are probably many potential chemists who would already know this data well and it would inform their modeling even if only tangentially. Allowing the rest of us to consider QM7 and QM9 might level the field some. Especially since probably some would have secretly used it anyway.
\n\nI very much hope that you will reconsider your view on future scientific competitions, these are great challenges and very interesting to many of us. This is true in spite of the issues about external data and the likely presence of it for most scientific problems. It would be a shame to see fewer scientific or engineering opportunities on Kaggle.
", "votes": 14, "replies": [ { "id": 551518, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T20:01:35.933000", "content": "There's a good chance that using QM7 and QM9 will be a waste of time, and this drama was for nothing. (But that is for individual competitors to decide.)
\n\nThere may be ways to mitigate abuse, in the future, but everything has a cost. You can count on more scientific competitions, but we really want to have them as Featured whenever possible, and this type of thing just makes is more tricky.
", "votes": 9, "replies": [] } ] }, { "id": 551488, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-06-12T19:27:12.540000", "content": "Thank you for this update!
\n\nI'm really sad that this situation could case less scientific competitions in the future. I hope we (kagglers) would be able to help science in future and won't ruin competitions (though I think we didn't really ruin this competition).
", "votes": 11, "replies": [] }, { "id": 551994, "author_name": "girugamess", "author_url": "", "post_date": "2019-06-13T11:52:06.213000", "content": "@inversion: Thank you for your works to make the competition fair. I came from computational chemistry field and I'm really excited about what happens to chemistry with the power of data science and kagglers.
\n\nI'm worried that something like another unfairness will happen because of the ambiguity of the rule about the usage of molecular simulation schemes. When I read the rules, I could not understand which level of simulation schemes are allowed and which kind of data must be shared as external data. In this nice discussion below, the answer was given by the competition host.\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/94457#latest-546076
\n\nIn my opinion, the information given in the discussion should be officially announced (or added to rules) in easy words. I am really afraid that participants who are not familiar with chemistry or do not know the regulation may violate the terms without being aware of it.
\n\nI understand the difficulty of organizing a scientific competition. I hope the competition will be success and contribute to chemistry world! Moreover I hope there will be a lot of scientific competitions at kaggle.
", "votes": 10, "replies": [] }, { "id": 551480, "author_name": "Boris D.", "author_url": "", "post_date": "2019-06-12T19:06:23.987000", "content": "Thanks for the clarification.
\n\nI may be wrong but I don't think these datasets will be very useful for our problem as they do not contain any magnetic information.
", "votes": 7, "replies": [ { "id": 551627, "author_name": "Yuko Ishizaki", "author_url": "", "post_date": "2019-06-13T00:24:59.557000", "content": "Thank you for your thought, I got more motivation.\nI am going to not to use QM9. Because the model using QM9 does not solve the competition sponsor's problem.\n(And I hope that the score does not improve with the use of QM9!)
", "votes": 3, "replies": [] } ] }, { "id": 558738, "author_name": "Lalit", "author_url": "", "post_date": "2019-06-22T20:42:38.103000", "content": "@inversion Thank you for sponsoring this challenge and for writing this update.
\n\nLike the \"Higgs Boson\" and a few other challenges, this challenge will provide the science research with some new tools and techniques. I compare these challenges with old-time challenges that led Louis Pasteur and other scientists put forth their scientific concepts etc.
\n\nIn another challenge, the first winner got a few million dollars. The company could not and did not use the first winner's method. Instead of worrying about that, the company felt happy that the challenge could contribute to the Predictive Analytics field.
\n\nYour concerns are quite legitimate. To avoid such concerns in future science-related challenges:
\n\nThe sponsor may state the front-end and back-end objectives clearly.
Kaggle may evolve a multi-point scoring system, involving Leaderboard score, external dependencies, affiliations, etc.
Please note that while a few participants are using these challenges for winning prizes, most of us (including me) are using them for learning.
\n\nI encourage and plead you to sponsor more such challenges in future. I understand you may have tight budgets, but I guess that the benefits to the sponsors from a Kaggle challenge generally outweigh the cost.
\n\nThank you.
", "votes": 7, "replies": [ { "id": 558741, "author_name": "inversion", "author_url": "", "post_date": "2019-06-22T20:49:04.023000", "content": "Thanks for that note. Sometimes the drama overshadows the real reason why most of us are here in the fist place. Great reminder!
", "votes": 4, "replies": [] } ] }, { "id": 551750, "author_name": "JohnM", "author_url": "", "post_date": "2019-06-13T04:33:12.843000", "content": "Excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science. Why are we allowing this sort of thing?
\n\nAlso, as others have echoed, there is so much sharing of code, data, algos etc. that is repeatedly ignored by Kaggle management. I would prefer focusing on this type of abuse vs. fretting about those who post public code with publicly available data.
", "votes": 7, "replies": [] }, { "id": 551532, "author_name": "nosound", "author_url": "", "post_date": "2019-06-12T20:30:10.340000", "content": "Wow, am I solely responsible here for \"that we will have fewer opportunities to host scientific challenges on Kaggle\", \"scientific competitions we do launch are less likely to be Featured\" and \"with smaller prize pools\"? Because you make it feel like that. No problem with barely defined rules on that topic? It is only stated that the dataset is not allowed to test and develop ones model, not that it is not allowed to study it and share. No problem with that the dataset is available to everyone anyway? No problem at all that you by yourself was interested in \"how much lift this data provides\" and you blame others for been interested in that as well? This last one is important question, because if openly available dataset with enough extra information is present in a competition it would destroy silver/bronze medals range, where the models are not checked anymore. Maybe this is something Kaggle can compromise on, and I am fine with it, I know it is very hard to design a perfect competition, but I want to know it before I decide to participate and spend significant amount of my time on the problem.
\n\nBy the way, I have no idea where you take this \"feel entitled to abuse the rules\" and \"break competition rules because you disagree with\" from. I explained myself multiple times, I understood this sentence \"not allowed to test and develop models\" in the rules as that it is not allowed that one's final submitted models are using the dataset or fitted to the dataset in some way.
\n\nMoreover, I un-shared my work the minute I understood from you that it was not allowed to share. It was up something like 24h, just a piece of code and a link to a dataset that is available online anyway.
\n\nAnyway... like many others I think this competition is a really beautiful one, with interesting data, and hope it continues with no more problems. Good luck to the participants and I hope some interesting approaches will appear.
", "votes": 4, "replies": [ { "id": 551545, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T20:48:43.753000", "content": "If a competitor didn't make submissions using that data (prior to the rule change), then there is nothing to worry about.
\n\nI do, though, hope you see how suboptimal it was for you to take it upon yourself to post the data that was explicitly forbidden, even if it was \"just to study\".
", "votes": 3, "replies": [] }, { "id": 551569, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-06-12T21:41:32.647000", "content": "@inversion You should reconsider this point:
\n\n> even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission before this rule change will result in that individual's (or team's) removal from the competition once it has closed.
\n\nIn my opinion, if you used the QM7 and QM9 datasets before this rule change, then you haven't done anything wrong, because I agree with nosound's interpretation of the old rules: Your final 2 submissions should not use the QM7 and QM9.
\n\nIf the public LB gets screwed up, then that's a consequence of the data being available, not the fault of the person who submitted it. Consider the rstudio competition which had scores of 0 RMSE, or the 2sigma competition which had the public LB answers within the kernel leading to a perfect score
\n\nIn my opinion there's nothing wrong with looking at this data, training on it, and making submissions on it. The only thing that should be satisfied is your final 2 submissions cannot use the data. If your submission uses the external data, then that submission should not be counted
", "votes": 7, "replies": [] }, { "id": 551571, "author_name": "Tania J", "author_url": "", "post_date": "2019-06-12T21:48:51.137000", "content": "I am really surprised and sad that this has blown up into such a huge thing leading to Kaggle possibly having fewer opportunities to host scientific challenges.
\n\nIt seems to me like @zaharch 's kernels and posts were an honest and understandable misunderstanding of unclear rules (that apparently I too have misunderstood) and that they have not ruined this competition but possibly even prevented it from later ruin.
\n\nI suggest that in future Kaggle should spell out the rules more clearly. Please consider stating explicitly that it is also not allowed to post any kernel or discussion using/containing even a subset of the forbidden datasets to the competition, if that is the case.
", "votes": 8, "replies": [] }, { "id": 551580, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T22:04:19.463000", "content": "I do not understand why you have a hard feeling against him/her. The situation is like, he/she found a jar of worms in an empty room, and said it out loud \"Hey look at this there's a jar of worms!\" While I agree this is not the best thing to do, I don't see there was anything obviously harmful being done because of this action. He/She did not bring the worms to the room, the worms were already in the room and it is just a matter of time for someone to find it. At least this is much much better than someone secretly opens the jar and scares everyone away.
", "votes": 2, "replies": [] }, { "id": 551586, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T22:12:58.333000", "content": "> In my opinion there's nothing wrong with looking at this data, training on it, and making submissions on it.
\n\n1) It was against the Rules, regardless of whether or not you think it was wrong.\n2) But using the data and submitting, you are gaining information that other competitors who did not break the rules do not have.\n3) At no time in the history of Kaggle has it been acceptable to break the rules as long as it wasn't for your final selected submissions.
", "votes": 5, "replies": [] }, { "id": 551592, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T22:18:39.713000", "content": "@xylthx - I've stated this multiple times. This issue is not bringing up the jar of worms. There is nothing wrong with that.
\n\nThe issue is finding a jar of worms, seeing a sign that says \"no worms please\", yet still opening it up and dumping the worms all over the carpet before we even had the chance to give a response about what should be done with the worms.
", "votes": 2, "replies": [] }, { "id": 551594, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T22:23:06.800000", "content": "@inversion - I don't see how he/she opened it up and dumped the worms all over the carpet. In my opinion you are exaggerating it. I do not see any obvious harm he/she has done because of this action.
", "votes": 0, "replies": [] }, { "id": 551611, "author_name": "", "author_url": "", "post_date": "2019-06-12T23:29:57.493000", "content": "", "votes": 0, "replies": [] }, { "id": 551642, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-06-13T00:58:05.127000", "content": "https://www.kaggle.com/c/two-sigma-financial-news/overview/Honor-Code
\n\n\n\n\n\"cheating\" encompasses any attempt to gain an edge in accuracy by using information that is outside of the provided dataset, or an attempt to use the provided information in a way that is not intended. Examples of cheating include (but are not limited to):
\n \nAttempting to use datasets and references beyond those made available by the competition\n Attempting to abuse the competition infrastructure to gain an edge
\n
Everyone who used the env_var07 in public submissions in Two Sigma competition should therefore be banned by your logic, since they made a public submission that \"abused the competition infrastructure to gain an edge.\" That has not been made clear yet, and I don't think it will happen because frankly it shouldn't happen. The competition had the target answers, that we weren't supposed to use, smack dab in the kernel itself. Just because you were curious and submitted to public LB, but didn't use it for private LB, shouldn't mean that you get disqualified. I think the same holds true for this competition.
@returnofsputnik - I'm not managing that competition, so really can't comment. But, in general, compliance is handled at the end of the competition.
\n\n@xylthx - I want to be clear, I'm not stating that any particular person did or did not use the data to make predictions. I'm saying that using that data (prior to the change) to make predictions was a violation of the rules. It does not matter whether or not you think the action was harmless.
", "votes": 0, "replies": [] }, { "id": 551654, "author_name": "inversion", "author_url": "", "post_date": "2019-06-13T01:37:55.290000", "content": "\n\n\n\n\nPlease consider stating explicitly that it is also not allowed to post any kernel or discussion using/containing even a subset of the forbidden datasets
\n
Discussing the data? That's fine.\nPosting the data? Not against the rules. (But why, if you can't use it?)\nUsing the data to inform how a model is developed? Against the rules.\nUsing the data to make predictions? Against the rules.
\n\nAll of this could have been cleared up if I had been given more than half a business day to respond to questions on the forums.
", "votes": -1, "replies": [] }, { "id": 551707, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T03:16:04.800000", "content": "@inversion - I am sure that you saw violations of rules way more than I did in Kaggle, but I've never seen an administrator so pissed off and start to criticizing individuals in the public discussion. Yes of course you could have a couple more days in advance, but no one was blaming you because of this. And of course what I think does not matter, actually what ANY kaggler thinks does not matter because you are the one who is making the rules. I just feel there is no reason to make it personal, which is exactly what you are doing now.
", "votes": 6, "replies": [] }, { "id": 551714, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T03:36:08.083000", "content": "@inversion - I understand that you as an admin wants everyone to obey the rules and will be annoyed if anyone does not. But I feel that you are hostile to that particular Kaggler and also any Kagglers that seem to understand his/her idea. This actually makes normal discussion nearly impossible, because whenever someone challenges you, your answer is \"I do not care what you think, I am the person who makes the rules.\" Yes, he/she made a mistake, but I do not see why he/she deserves this treatment. You took a few days off and this happens, this is not your fault, and this is not completely his/her fault. There was not a lot chaos, actually a lot of people are happy to bring up this issue to the table at this early stage instead of three months later. And as a scientist myself, I do not see the inhibition of using some external data undermining the scientific value of the challenge, actually it is setting all these limitations undermining the usefulness of a model by making it \"fancy but useless\", because in the real world, what the users want is simply \"fast and accurate\", no one will even care about what external data it uses when making it.
", "votes": 2, "replies": [] }, { "id": 551763, "author_name": "Addison Howard", "author_url": "", "post_date": "2019-06-13T04:58:10.557000", "content": "Josef,
\n\nWe have multiple members of Kaggle monitor all of the forum posts for each competition, which typically total 2000-4000 unique posts per competition. With 15-25 competitions live at a time divided across a small staff, it's not uncommon for us to field 500+ discussion posts per day, in addition to our normal workload.
\n\nWe're thankful that the Kaggle community is so active, engaging, and often answers most of these comments and questions before we even get to them, whether it be sharing ideas/code (within the rules, of course 😉 ), troubleshooting, interpreting competition rules, or general tips and tricks on machine learning, competitions, and general data science. We're just as excited to see Kagglers reach a new tier as the community is, and even celebrate across our own internal messaging app when we see people reach Grandmaster status.
\n\n@inversion - being the OG Kaggle grandmaster, has invested countless hours in the Kaggle community, both before he was hired, and even more so (now that Kaggle pays him for his work 😄). One of the worst, and most frustrating things for someone on the Kaggle competitions team to experience is to see a lot, or all, of the hard work invested in preparing a competition unravel. Whether it be through leakage, unwanted public data, a high-scoring kernel shared last minute, or a member of the community violating the rules and tarnishing the experience of other Kagglers. As this was a very specific instance of a unique dataset in chemical engineering, an industry that @inversion is particular close, is rare on Kaggle, and is one we want to bring to the Kaggle community.
\n\nIt's one thing for a user to cheat and try to hide it (in which we often catch and remove them), or accidentally cheat and admit it. We also commonly receive inquiries on interpretations of the rules, or challenges to the structure of the competition. It's because of the Kaggle community that we try new things, new competition formats, and are willing to push the boundaries of what is considered \"standard\" to supervised machine learning competitions. We welcome the discourse and even the disagreement. We make changes to future competitions' structure and rules based on community input. We listen and respect the Kaggle community (It should be noted that we often hire them), and we want to see the community successful.
\n\nHowever, it's another thing entirely when there is a blatant disregard for the clearly stated rules, the rule is broken, announced it is broken, and those results are then shared to the masses. It's similar to if you saw a police officer on the side of the road trying to catch people speeding on a highway that had a speed limit you didn't agree with. So instead of going to city council and expressing your concerns about why that speed limit wasn't realistic and was detrimental to the public welfare, you instead accelerated right by the officer and yelled \"This is what I think about your speed limit!\" out the window, all while live streaming the event from your phone. Extreme example? Sure - but pretty funny to imagine @inversion in a Dodge Charger hiding behind a billboard trying to pull over speeding Kagglers.
\n\nThe former is proactive, people listen, and the potential for change is up for discussion. The latter assumes that the rule is arbitrary, deserves to be broken in a grand display, and results in frustration by both parties - there is usually little remorse, no collaboration, no growth, and nobody ends up better off.
\n\nWe care about the Kaggle community. We encourage open discussion. We even welcome being challenged. Discuss, disagree, and cast doubt - all fine. But by taking a rule, acknowledging it, and yet still publicly breaking it, it's not only greatly disrespectful to Kaggle but it is disrespectful to the hosts of the competition and to the rest of the Kaggle community. It not only assumes that there are no good, valid reasons for the rules to be in place, but that those rules are so outlandish they should be violated via public spectacle without the opportunity for discussion. Just because the harm/consequences of violating a rule don't seem \"obvious,\" it doesn't means that the harm isn't there, nor does it suggest that the best way to address such is to assume the worst and act recklessly.
\n\nWe want Kagglers to be successful. In fact \"user success\" is literally one of Kaggle's core values. We constantly see competition sponsors come to Kaggle with interesting and unique problems for the community to solve. They often do so because of the reputation that over three million Kagglers bring with them. But by playing vigilante justice with a competition rule, it not only impacts the hundreds, if not thousands of other Kagglers trying to learn and develop their machine learning skills and reputations, but it also jeopardizes Kaggle's ability to continue to provide these types of problems to our community in the future. We have sponsors who point to competitions from over seven years ago as references for what they'd like to emulate, as well as to discussion posts from the recent past for things they'd like to avoid. They often point to competitions of a similar nature in industry and problem type as a guide.
\n\nSo when a competition in a highly sought chemical engineering space (both close to @inversion and to many members of the scientific community) makes for a great problem, you can imagine why it stings a bit deeper when that work is insulted, rules are blatantly and fragrantly disregarded, and deference to the well-reasoned structure is defied. The Kaggle community spends more time discussing how to interpret and adhere to minutiae of the rules instead of creatively solving machine learning problems, which doesn't really benefit anyone.
", "votes": 15, "replies": [] }, { "id": 551806, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T06:12:02.230000", "content": "@addisonhoward - I really appreciate your extensive reply. Frankly speaking, what I feel uncomfortable about this situation is that apparently the Kaggler who posted QM9 dataset is being treated like a pest to the community, while I believe his/her intention was not to ruin this competition. He/She could easily use this dataset secretly, and posted a few exaggerated threads/kernels three months later, if ruining this challenge is his/her intention.
\n\nI cannot find the old rules, but if @returnofsputnik is right, if the old rule was \"Your final 2 submissions should not use the QM7 and QM9\", I don't see any violation by simply studying QM9, or making submissions now (but not finally) using this dataset to just try it out. Even if the dataset is banned from using to train the model, there should be no limitations not to allow people to learn from it, and possibly share knowledge they gained from it. I am pretty sure my interpretation is wrong here, but misunderstanding the rules is not disrespect to Kaggle, and I am sure you will agree. What I saw from this and other threads he/she posted and replied was he/she got taunted by other users, as if everyone is trying to kick him/her out and the admin upvoted for that. People are not sure about the rules, get confused, and dare not to discuss QM9 related issue publicly for not being kicked out of the competition, while I am sure many Kagglers will still at least study this dataset. I don't think this climate is beneficial to this community.
\n\nAnd since a high-level (possibly) person like you replied, I'd like to discuss more about the science topic competitions like this one. I am a scientist (I guess I can say this) and also a data science user; I have degrees in both domains. The thing I don't feel comfortable about the current trend is that natural scientists are still very skeptical to machine learning approaches, while data scientists, instead of trying to explain how useful machine learning models can be, are aiming to beat the state-of-the-art scientific approach and claim the victory of data science over the traditional approaches (instead of trying to learn from the domain knowledge and understand what the scientists really want). Therefore, often the outcome is, data scientists built models that look pretty cool and interesting, but the natural scientists found such models useless for some reasons. Coming back to this specific competition, what I don't really understand is that, is the goal of this challenge is to find a fast and accurate model that works on any molecule, or the goal is simply \"Let's prove to those dumbass chemists what data science can do\"? Obviously the latter one does not add ANY scientific value and will most likely end up with some fancy but useless model. To be more specific, the reason not to use any quantum properties is because the computation is expensive, and if a person use a pre-calculated dataset, he/she would not be accurately predict a molecule outside of the pre-calculated dataset, making the model useless. In my opinion, providing a test set whose molecules are not in QM9 would be a good solution. Simply limiting the usage of some external dataset will actually undermine the scientific benefit, because in the real world, when using a model, you never care how it was trained, and I really doubt there is any scientific domain knowledge a chemist can learn from a model with more than tens of thousands of parameters.
", "votes": 0, "replies": [] }, { "id": 551986, "author_name": "nosound", "author_url": "", "post_date": "2019-06-13T11:26:09.683000", "content": "@addisonhoward , thanks you for sharing your thoughts, it completely resonates with me. However, I want to reiterate once and again, since it looks like you assume differently. I did not violate the rule knowingly or intentionally. As I wrote before, I understood the rule differently, and I didn't have any questions about my understanding of the rule, it didn't appear to me at the moment that my interpretation might be wrong, and therefore I didn't seek any clarifications on that. I understood the rule that it was not allowed to use the dataset in one's final submissions, because it was the only logical interpretation that appeared to me. Not been able to look into relevant and publicly available dataset, and not been able to study and discuss it publicly did not appear to me as a logical interpretation at the moment and I did not consider it for a second more beyond that.
\n\nIn fact, I don't even think that it is correct to say that I misunderstood the rule, because I think my interpretation is a possible one, you can see other people agreeing with me that it was a possible interpretation. I think the correct diagnosis here is that the rule was not clearly defined.
\n\nthank you for doing this hard job...
", "votes": 5, "replies": [] } ] }, { "id": 551529, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-06-12T20:25:41.533000", "content": "I am bit saddened that scientific value of the competition has been lowered b/c of this issue. I have two questions:
\n\nCould Kaggle verify the top money and at least gold solutions for evidence that the forbidden data was not used? If a cheater is caught, it would be get a temp ban from Kaggle, perm bans for repeating cheaters. For money-seeking cheaters, maybe require some level (Expert/Master, etc.) to qualify to win monetary prices.
I understand molecules cannot be anonymized, but it looks like all molecules in the test set are included in the forbidden datasets. Would it make sense to have a 2-stage competition where 2nd stage test set molecules are not included in forbidden datasets?
@inversion as you value community feedback and dialogue I'd be very interested in your stance on 1) and 2) and a bit more clarity on what you call conceptual reasons: I have only high-school knowledge of chemistry, and reading the rules and the explicit restriction of quantum mechanics approaches I thought that the creation of these datasets (QM7/QM9) was very expensive as well and solutions based on them would provide less scientific value.
", "votes": 3, "replies": [ { "id": 551535, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T20:34:03.867000", "content": "I disagree that \"solutions based on them would provide less scientific value\". The solutions based on them would provide less \"data science\" value, but not \"scientific\" value. I believe that the aim of this competition, and also the host, is to find an accurate and fast model to predict the molecule properties, not to prove \"see how good using only data science knowledge can do\".
\n\nAlso, correct me if I am wrong, for \"2nd stage test set molecules are not included in forbidden datasets\", I really doubt they have any other molecules which are not in that dataset....
", "votes": -1, "replies": [] }, { "id": 551544, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-06-12T20:47:19.190000", "content": "Well looking at QM9 it contains ~133k molecules, and this comp is a subset (130k total), and according to the GM9 abstract \" These molecules correspond to the subset of all 133,885 species with up to nine heavy atoms (CONF) out of the GDB-17 chemical universe of 166 billion organic molecules.\" so whatever ML solutions are devised here using QM9 would not be applicable to predict scalar couplings for the vast majority of the GDB-17.
\n\nAnd obviously the organization already knows the scalar coupling values for almost all of QM9, hence my comment, which I may move from \"less scientific value\" to \"less practical value\".
", "votes": 2, "replies": [] }, { "id": 551549, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T20:51:53.067000", "content": "Yea, I do hope that they can provide a new or a second test dataset without these problems...and people can use whatever resource they can find.
", "votes": 0, "replies": [] } ] }, { "id": 551492, "author_name": "", "author_url": "", "post_date": "2019-06-12T19:31:08.573000", "content": "\n\n\nIf you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.
\n
So, basically, you allowed using the data, which could harm the research, which is essentially the only valuable reason for this competition? To be honest, I'm a little shocked: you could not look after compliance with the rules and the result was not long in coming:
\n\n\n\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.
\n
Don't you think that you have to stand firm on your \"No\" in such cases (not allowing the usage of data)?
", "votes": 4, "replies": [ { "id": 551512, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T19:52:36.563000", "content": "The data was excluded for conceptual reasons, not because there was evidence it contained any useful information.
\n\nBest case - winning teams don't use the data.
\n\nSomewhat less optimal - Winning teams use this data, but it contributes a minuscule improvement to the score. Those teams would still win, but researches would need to recreate the models without those features.
\n\nWorst case - The external data provides significant leakage to the target values. (We don't believe this to be true for multiple reasons). If this is the case, hopefully researchers would still be able glean some learnings from the competition.
\n\nSo why not include the data in the first place? As stated above, for conceptual reasons (given the insignificance of the external data). And as is often the case in competitions, we listen and react to community feedback. This particular situation was disappointing to me because of those who effectively forced the issue, rather than having a good-faith dialog.
", "votes": 5, "replies": [] } ] }, { "id": 551572, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T21:49:08.560000", "content": "\n\n\nThe obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.
\n
If I understand correctly, the scientific benefit is to predict faster. The quantum computation properties shall not be used, only because that the quantum computation takes a very long time, not because they use physical laws to do the calculations.
\n\nBut what if there exist a big dataset with all the quantum computation properties, so that you do not need to re-calculate them? If you are a researcher, not competing for a kaggle competition but rather aiming for a paper, and you know such information makes your prediction better, would you not use it? Does intentionally not using such information do any good to science?
\n\nIn my opinion, the biggest concern is that using such external dataset might undermine the generalizability of the model, while since all the molecules in the test set are from this dataset, there is no way for you to check such overfitting. My suggestion is the same as @antorsae pointed out, having a new or a second test set where all the molecules are not in the external dataset would be the best solution.
", "votes": 2, "replies": [] }, { "id": 551514, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T19:55:58.647000", "content": "I am from the scientific world and I am applying data science in my own research to improve the current methods in my own field. There are of course a lot of improvements that can be made in natural sciences using the knowledge of our fellow data scientists. However, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy. The advantage of the machine learning ways are the speed, therefore many problems can be solved in real-time and high-throughput data can be analyzed efficiently.
\n\nI believe that the aim of this competition is also to find a machine learning model that performs similarly as the state-of-the-art quantum calculation method, while being much faster than the physical way. In my opinion, no matter what external data people use, as long as such model works on every molecules, this is a good model to me.
\n\nI think the \"the spirit of the scientific challenge\" is not \"can we beat the current physical method with pure data science\", but rather \"can we make a faster and better model using whatever we can\".
", "votes": 2, "replies": [ { "id": 551517, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2019-06-12T19:58:29.883000", "content": "\n\n\nHowever, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy.
\n
Challenge accepted. ;)
", "votes": 12, "replies": [] }, { "id": 551519, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T20:03:01.797000", "content": "Watch out . . . . Bojan's the type of guy who will find a model that is both faster and more accurate!
", "votes": 10, "replies": [] }, { "id": 551522, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2019-06-12T20:12:43.540000", "content": "I honestly don't know if such a thing is really possible. My own intuition would be that the \"proper\" physical properties approach would still be well ahead of the \"pure ML\" approaches. However, a combination of the two might be the best. Something like Alpha Go for instance for Physics Equations instead of rules of Go. Which would obviously require the resources well beyond what most Kagglers have access to.
", "votes": 5, "replies": [] }, { "id": 551523, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T20:13:00.267000", "content": "The problem is actually that, in many cases, the training sets are often generated using the physical and computationally expensive way. Yes, there are some fields that have a lot of experimental data, but such fields are very limited and they already have a lot of data scientists working on the problems. So often the case is, the ground truth provided is from computation, so even by some magic you find a better model, you will never be able to validate it.\nAlso, while the trend of data science is to be more and more \"generic\", scientific questions are often very specific. Precision and explainability are what scientists care most about, unless they are dealing with a huge amount of data that the traditional way takes forever.
", "votes": 0, "replies": [] }, { "id": 551525, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-12T20:16:16.770000", "content": "I agree. And I also think that reinforcement learning has the potential to beat some physical algorithms, while it is very difficult for the traditional supervised learning to do that.
", "votes": 0, "replies": [] } ] }, { "id": 552219, "author_name": "Hassan Abedi", "author_url": "", "post_date": "2019-06-13T16:40:59.843000", "content": "
I have not been around here long, but I just want to post my support for the kaggler[s] who are willing to break rules and tarnish their own image for the sake of transparency, honesty, and fairness. I'd rather have no competitions than unfair competitions, and shame on everyone who'd rather just sweep things under the rug. I read the original thread and I was very disappointed by the initial replies to that kaggler.
", "votes": 2, "replies": [ { "id": 552049, "author_name": "inversion", "author_url": "", "post_date": "2019-06-13T12:53:56.527000", "content": "Welcome to Kaggle Travis. Over time, you'll see that these things can be handled without breaking the rules. We're very open do discussion, feedback, and course correction.
\n\n\n\n\nI'd rather have no competitions than unfair competitions
\n
Participation in Kaggle competitions has always been and will always be voluntary. Why would you do something to take away a competition from other people, just because you don't view it as fair?
", "votes": 8, "replies": [] }, { "id": 552145, "author_name": "Travis Black", "author_url": "", "post_date": "2019-06-13T14:56:58.630000", "content": "I don't think anyone did anything to take the competition away from other people, and I certainly don't want that. I was talking about what I would personally want. I know the original post here says that because of these events there will be less opportunities for scientific competitions on kaggle, but there really wasn't any explanation as to why.
\n\nHonestly, the original post here makes it seem like it would be preferred that abuse of the rules remain hidden if there isn't a clear way to address it. Otherwise I don't see why there was such opposition to nosound telling everyone about the data in the first place.
\n\nThat said, I don't want to seem purely disagreeable or like I am attacking you, because I am not. I have the utmost respect simply for the fact that there is open uncensored discussion of this topic in the first place.
", "votes": 4, "replies": [] }, { "id": 552168, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T15:34:34.180000", "content": "> Honestly, the original post here makes it seem like it would be preferred that abuse of the rules remain hidden if there isn't a clear way to address it.
\n\nThis is exactly how I feel about it. QM9 is simply out there in the internet, being accessible to everyone, and nosound just posted about it making everyone know this issue. Yes, you could not use it in the final submission according to the old rule, but how is Kaggle going to monitor that? He brought this question up to the table and it was indeed a tricky problem. It seems like the admin would simply prefer no one ever found this issue so that he can pretend that it does not exist, got mad and being sarcasm when someone publicly talked about this issue, accusing him/her for the possible outcome of having less scientific competitions (which does not make any sense), and encouraging other Kagglers attack him/her and whoever speaks for him/her. This is becoming personal and it should not be.
", "votes": 5, "replies": [] }, { "id": 552243, "author_name": "inversion", "author_url": "", "post_date": "2019-06-13T17:06:49.097000", "content": "Travis wrote:
\n\n\n\n\nI don't see why there was such opposition to nosound telling everyone about the data in the first place.
\n
The issue has never been telling people about any data, discussing the data, etc. It was about using the data to make predictions. It's perfectly fine to say, \"I found this data that isn't supposed so be used. I'm concerned that it could be abused.\" It's against the rules say, \"I found this data that isn't supposed to be used. I submitted a model using the forbidden data and it improved my leaderboard score. Here's a kernel how you can do the same.\"
\n\nJosef wrote:
\n\n\n\n\nYes, you could not use it in the final submission according to the old rule
\n
Please stop repeating this. I've explained many times that there has never been a rule at Kaggle where you could use forbidden data as long as you did not use it in the final submission. By continuing to make this claim, you are confusing the issue.
\n\nLet me be very clear here - I will not continue to tolerate you spreading incorrect information about what has already been spelled out repeatedly and clearly, nor will I tolerate you making claims that I am encouraging anyone in the community to attack another member.
", "votes": 1, "replies": [] }, { "id": 552266, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T17:43:50.653000", "content": "I don't want to play word game here. I want to be very clear as well - you are a grandmaster at Kaggle and you have made a lot of contribution to this community, and I truly respect that. However, it is not anyone's fault, if not you, that the rules are not explicit and leave room for misinterpretation. It is not anyone's responsibility to be aware of your out-of-office schedule, unless you pinned it on the top of the discussion board.
\n\nPeople come to Kaggle to learn, to compete, to meet new people, not to ruin the competition you hosted. Did nosound do things with the optimal way? Of course no. But was his/her intention to ruin your competition, to taunt the organizers? He/She removed the kernel instantly after you expressed your point, acknowledging that this was a misinterpretation of the rules. He/She tried to mend it, to minimize the effect it caused. On the other hand, you constantly, even after a few days now, accused nosound for his/her action in the public discussion board, implied that he/she should be blamed if there are fewer scientific competition, upvoted other person's harsh comment on him/her. Whatever kind of claims you think this is, this is all true and you know it.
\n\nI do not know you in person, and I don't think I will have the chance in the future. There is no point for me, or any other Kagglers here to intentionally attack you behind a computer screen, especially with the fact that you are not just an ordinary Kaggler, but an official representative of this competition. What nosound did was not optimal, but what you have been doing against him/her was totally unnecessary and was not helpful to this community in any way. Given what he/she did and the way he/she responded, nosound does not deserve the treatment that he/she has been experienced in the past few days.
\n\nAs I repeated a lot of times, there is no point to make this personal.
", "votes": 0, "replies": [] }, { "id": 552276, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T17:55:53.270000", "content": "\n\n\nIt's perfectly fine to say, \"I found this data that isn't supposed so be used. I'm concerned that it could be abused.\" It's against the rules say, \"I found this data that isn't supposed to be used. I submitted a model using the forbidden data and it improved my leaderboard score. Here's a kernel how you can do the same.\"
\n
Same for you, it is perfectly fine to say, \"You should take your kernel down immediately since it is against the rules. I understand that there might be a misunderstanding of the rules, and we are all good if you obey the rules from now on.\" It is totally unnecessary to say, \"You should be blamed if we have fewer scientific competitions. Guys, it's nosound who makes this happen.\"
", "votes": 1, "replies": [] }, { "id": 552293, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T18:13:59.520000", "content": "Putting all the nosound issues aside, I am still not sure what Kaggle wants to achieve by these scientific challenges. I have seen many many published papers in the scientific world that use data science methods not to make any real, practical improvements, but to simply use it \"because we can and it is sexy science\". Such tools are just one-paper things, no one is going to actually use them and they just lie on your resume because they are good-looking. If Kaggle's intention is to host a fair, but domain-knowledge free (in order to attract more people and being more \"fair\") competition, while does not really care what scientists really want, I totally understand it, and that also answers many questions I have. However if Kaggle seriously wants to push science forward using the wisdom of all these data scientist users, sometimes the organizers have to think twice and discuss more with the competitors when setting limitations and designing the competition, and sometimes it is necessary to compromise the \"fairness\" because not everyone are on the same starting line when it comes to the domain knowledge.
", "votes": 2, "replies": [] }, { "id": 552361, "author_name": "inversion", "author_url": "", "post_date": "2019-06-13T19:51:44.043000", "content": "Josef wrote:
\n\n\n\n\nI have seen many many published papers in the scientific world that use data science methods not to make any real, practical improvements, but to simply use it \"because we can and it is sexy science\".
\n
Finally, something we can totally agree on!
\n\nI was at the APS DFD meeting last year. IMO, most of the \"using machine learning in fluid dynamics\" presentations were pretty lame . . . basically using a fancy curve fit for a small dataset. On the other hand, there was one that was absolutely amazing . . . using proper orthogonal decomposition and LSTMs to model the evolution of turbulent flow.
\n\nIt's my personal interest and preference to get more scientific competitions on Kaggle. No doubt, some will be underwhelming. But I'm hoping it's net-positive for the community and for science.
\n\n\n\n\nsometimes the organizers have to think twice
\n
I can assure you, there is a good deal of thinking that goes on prior to a competition launch. The community sees the 1 thing that slips through. They don't see 99 that didn't. You can think of a competition launch in terms of software development. We do everything we can to minimize bugs before launch. But when you have a thousand people scrutinizing your work, something is bound to be discovered.
", "votes": 3, "replies": [] }, { "id": 552375, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T20:46:42.393000", "content": "Sometimes it's not about how big are the dataset, or how complex/novel the model is. Many papers lack the original scientific motivation. A good logic is, \"the traditional ways have these limitations and we proved that data science can help us overcome these problems\". But often what I see in the scientific world is, people use fancy methods to solve a non-existed problem. Many ML tools people developed have old, and sometimes even better alternatives with the traditional method. Some other ML tools look cool and try to solve problems, but they actually fail doing so and become a mediocre.
\n\nI was in an industry-oriented conference earlier this year, with all the big pharma/bioinfo and some smaller companies attending. Nearly everyone talks about AI all the time. There was a live poll during a panel discussion, the question was \"For the most successful project that used deep neural networks in your company or group, what % improvement did you find over more traditional methods?\" Nearly a half of the people chose \"10% or less\". And this was an industry conference, academia would just be worse.
\n\nPeople outside of DS field either think DS/ML/AI as a panacea or a worthless trick. The advertise they get from the media, i.e. \"how AI has changed our lives\", \"AI beats human again\", makes them think AI is good at everything and it just happens magically. When they try to use DS/ML/AI in their own work (not developing a model, just try a model), they are disappointed because usually the model cannot solve the problems they have for various reasons.
\n\nComing back to the scientific world, I am very okay if Kaggle wants to exclude as much as domain knowledge as possible when holding scientific challenges. But doing so is not always great for the improvement of science. Models developed here will most likely become a method paper in the end. A good question to ask is, if not competing in a Kaggle challenge but purely developing a model without any limitations, can I create a better model that also fulfill the needs? If so, why setting the limitations?
\n\n> I can assure you, there is a good deal of thinking that goes on prior to a competition launch.
\n\nI totally believe you. It is not trivial to hold such challenges and I appreciate your work. However, the 1 thing slipped through to the community is also important, and actually a lot of serious discussions can be made and it could even be beneficial. Most of the people here are to learn and help the community, not to undermine it. All these \"scrutinizing\" actually play a similar role as a \"peer-review\", especially when people start to discuss the motivation of a competition, \"what the sponsor really want and what can we do to make it useful, instead of just good score and looking fancy\".
", "votes": 0, "replies": [] }, { "id": 553147, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2019-06-15T07:35:58.993000", "content": "Ponder you want to test 100 G molecule configurations for certain characteristics. Your ML might give you 100 k suggestions which you then run your extended QM on?
", "votes": 1, "replies": [] } ] }, { "id": 573645, "author_name": "toshi_k", "author_url": "", "post_date": "2019-07-12T14:52:53.947000", "content": "\n\n\nTo address this current competition, we are updating the rules to allow the use of QM7 and QM9 published data, if participants wish to use it. Participants still may not use quantum calculations to obtain predictions.
\n
Do you mean we can use QM7 and QM9 for training ML models ? I understand we should not use them for test data. But I'm interested in utilizing them for training and approximating their relationship.
", "votes": 1, "replies": [] }, { "id": 551499, "author_name": "Luis Silva", "author_url": "", "post_date": "2019-06-12T19:36:56.780000", "content": "\n\n\nOnce the competition finishes, CHAMPS would like to invite the top teams to present their work, discuss the details of their models, and work with them to write a joint research publication which discusses an open-source implementation of the solution.
\n
My understanding of this sentence is that you will discuss with each top team their approach. Thus, if you find any usage of quantum mechanics features (looking at the code) can't you just disqualify that team? Also, before that discussion you can ask to have access to source code to analyse it.
\n\nBesides that, like @areveillon said, if a team decided not to use quantum mechanics features what will happen? IMO and since the goal is to predict scalar coupling without the expensive quantum calculations you should focus on those approaches and do not allow QMs (from any source). In the end you can assess each top team to validate it (I now it can be cumbersome and a lot of work, but it is what it is until better way to evaluate the usage of such features)
", "votes": 1, "replies": [ { "id": 551516, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T19:56:43.553000", "content": "That's a great question. In this case, it would be impossible to know whether a team used the external data to, e.g., tune model hyper parameters without explicitly using them in the model. So even inspecting the final models wouldn't help.
\n\nBetter to just allow the data and end the drama. I have a feeling when it's all said and done, there are going to be many people who realize how overblown this is (since there is currently no evidence, practical or theoretical, that this data is helpful).
", "votes": 2, "replies": [] } ] }, { "id": 553808, "author_name": "Beans", "author_url": "", "post_date": "2019-06-16T11:26:42.783000", "content": "@inversion very appreciate your effort 🥇. It's really great thoughts.
", "votes": 2, "replies": [] }, { "id": 552904, "author_name": "Himanshu Soni", "author_url": "", "post_date": "2019-06-14T19:25:20.200000", "content": "I am looking forward for your excellent ideas....!!!
", "votes": 2, "replies": [] }, { "id": 551481, "author_name": "Antoine", "author_url": "", "post_date": "2019-06-12T19:08:46.767000", "content": "Thanks for this clarification.
\n\nYou said that there is little to no benefits of using the QM7 QM9 data set.
\n\nEven if the benefit is small, does that mean that all Kagglers who are not using it will have an handicap? I'm not planning on using it so I would like to make sure it still make sense to do the comp without it.
\n\nThanks!
", "votes": 2, "replies": [ { "id": 551485, "author_name": "Boris D.", "author_url": "", "post_date": "2019-06-12T19:18:48.607000", "content": "@areveillon \nClearly the QM7 is useless as atomization energies have nothing to do with J coupling (which is an atomic spin interaction mediated by electrons).
\n\nAccording to the website, QM9 has:\n- geometries: useless\n- harmonic frequencies: describes vibration/rotation of the molecule. Useless\n- dipole moments: this may help a bit as it is an electronic property.\n- polarizabilities: this is related to the response of the molecule under an electric field. Probably useless.\n- energies, enthalpies, and free energies of atomization: energies/enthalpies may be used but it won't help much.
", "votes": 11, "replies": [] }, { "id": 551498, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T19:36:24.160000", "content": "@areveillon - This is something you'll have do decide on your own. I can tell you that I personally used this information in a model and the feature importance was in the long tail of irrelevance. The data was excluded for conceptual reasons, not because there was evidence it contained any useful information.
", "votes": 9, "replies": [] }, { "id": 551500, "author_name": "Antoine", "author_url": "", "post_date": "2019-06-12T19:42:54.960000", "content": "Thanks for your answers!
", "votes": 1, "replies": [] }, { "id": 551796, "author_name": "datasaurus", "author_url": "", "post_date": "2019-06-13T05:59:04.957000", "content": "From the kernel:
\n\n\n\n\n\n
\n- \n
dipole_moments.csvcontains X,Y,Z values per molecule and I found thatsqrt(X^2+Y^2+Z^2)=muwhere mu is given in QM9- \n
mulliken_charges.csvmatches the mulliken charges from QM9- \n
scalar_coupling_contributions.csvI can't find this info in QM9- \n
magnetic_shielding_tensors.csvI can't find this info in QM9
1 & 2 can be calculated using existing models/packages that have already been shared on the forums/kernels. 3 & 4 cannot and aren't in QM9. So as @inversion said, it's likely that QM9 won't make much of a difference if any - which is good! 😄
", "votes": 2, "replies": [] } ] }, { "id": 558681, "author_name": "andrea cazzaniga", "author_url": "", "post_date": "2019-06-22T18:30:28.610000", "content": "I might be missing something obvious, but..I really don't get why setting the \"NO QM\" rule in first place? - and tell me exactly what that means, cause almost everything here can be labeled as QM.
\n\nThe challenge is to deliver a quick solution to a problem that nowadays takes too much time and too many PhDs to be addressed.
", "votes": 0, "replies": [] }, { "id": 558508, "author_name": "Rocco Meli", "author_url": "", "post_date": "2019-06-22T13:28:54.127000", "content": "\n\n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.
\n
😩
", "votes": 0, "replies": [] }, { "id": 552830, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2019-06-14T15:59:40.737000", "content": "Are scalar coupling constants for the private set available in the public domain or locked on organizers' hard drive? I am trying to anticipate the next leak. I am guessing it is the former.
", "votes": 0, "replies": [] }, { "id": 551884, "author_name": "MaxR2D", "author_url": "", "post_date": "2019-06-13T08:30:41.420000", "content": "I don't want to be boring, but I see that kind of situation happening a lot of time in Kaggle's competitions where external data can be used and thus creates a kind of unfairness situation between competitors who play by the rules and those who don't.
\n\nThe only way is to set up a true evaluation method based on the code ! So for the private and final leaderboard, each competitor gives the code which has to take the data in input and output the prediction. In that way only we are sure that no external data can be used. Indeed, if your code is based on external data it would not be runnable without it so the code would not give any prediction.\nIt's as simple as that, even though I understand it is harder to set up this kind of evaluation method.
", "votes": 0, "replies": [ { "id": 552035, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T12:40:47.330000", "content": "They are not banning all external data, actually I agree with what @jpmiller said, \"excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science\". Data science is more than just finding a good model, it also requires understanding the domain knowledge, data mining, preprocessing, etc. And I believe it is not very practical for Kaggle to run the codes of even a small part of Kagglers, since one specific model may require hours of training. What you want is probably a kernel competition, yes it is more fair, but it has other problems as well.
", "votes": 0, "replies": [] }, { "id": 552355, "author_name": "MaxR2D", "author_url": "", "post_date": "2019-06-13T19:43:40.683000", "content": "Yes I think you're right, external data should be allowed. Yet in that competition, the purpose is to give an alternative to heavy quantum computing. So giving a model which uses QM data is a nonsense.
", "votes": 0, "replies": [] }, { "id": 552401, "author_name": "Josef K", "author_url": "", "post_date": "2019-06-13T22:15:47.343000", "content": "I think it depends on the goal of the challenge. The QM9 data is from quantum computing, but it is also a pre-calculated dataset that is available to the public. If one can learn some meaningful things from this dataset, why not?
\n\nConsider two models, 1) a pure data science model that is fast but not super accurate, and 2) a model that is equally fast, with better accuracy, but it used some quantum properties during the training. If not competing for a Kaggle challenge, you just simply want a good tool, which one would you choose?
\n\nThe potential caveat of QM9 is that although being a big dataset, there are still a lot of molecules outside QM9, and a good model should preform equally well on all the molecules. The problem now is it seems that all the molecules in the test set are in QM9 as well, making it hard to evaluate the generability of the model.
", "votes": 0, "replies": [] } ] }, { "id": 553288, "author_name": "Tony Teng", "author_url": "", "post_date": "2019-06-15T12:29:07.753000", "content": "Thanks
", "votes": 1, "replies": [] }, { "id": 554216, "author_name": "Bhavik bansal", "author_url": "", "post_date": "2019-06-17T06:40:26.310000", "content": "Thanks for the clarifications!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "551457": "For the last year, I have been focusing on bringing more scientific challenges to Kaggle. I personally believe it's a benefit to both our community as well as the scientific community to host these different types of problems.\n\nScientific problems pose a different set of challenges, and we'll use this competition as an example. The obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.\n\nBut what if a researcher wants to run a Kaggle competition for this problem? We know that some properties of the QM7 and QM9 molecules have been published, that they are expected to provide minimal-to-no benefit to the target for this competition, and using this data would be completely against the spirit of the scientific challenge. It would seem a prudent course of action to exclude this data from the competition.\n\nThat simple request was violated before we were even able to have a discussion about it on the forums. It is already extremely difficult to find practical scientific challenges that can't be abused via external data. And, unfortunately, there are some community members who feel entitled to abuse the rules \"to make things fair\". \n\nThe unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.\n\nTo address this current competition, *we are updating the rules to allow the use of QM7 and QM9 published data*, if participants wish to use it. Participants still may ***not*** use quantum calculations to obtain predictions.\n\nWith that said, I can't stress enough how wrong it is to break competition rules because you disagree with them or you want to force a particular course of action. Because this is such an important point, even though we are allowing the use of QM7 and QM9 going forward, any evidence that someone used the data to make a submission *before* this rule change will result in that individual's (or team's) removal from the competition once it has closed.", "558482": "A couple of points\n\n1st of all - cheating is irrelevant to the scientific competitions - it happens everywhere and in all competitions, so you should not feel bad about this @inversion . This is not exclusive to scientific competitions. Private sharing is off the charge in every competition and you can definitely see patterns on the leaderboards. \n\n2nd - No matter how good your system is in detecting violations, you will never be able to capture them all when the system is so broad, fluid and open in using external data, linking data-sets, copying the works of others. Essentially you welcome this behaviour. \n\n3rd - something that I was afraid of since the beginning when kaggle introduced GMs from kernels and discussions is that people are gamifying the system to get the rankings at the cost of the competitions' credibility. This quite often leads to oversharing, people releasing high scoring kernel at the end of competitions (because they know they cannot get the competitions' kaggle medals, they aim for discussion/kernel medals). *I need to highlight* that I don't discredit the value of GMs in kernels and discussions, especially when they achieved this naturally without gamifying the system . In my opinion they are more valuable than competitions' GMs to the kaggle community. But gamification has changed things. \n\n4th. Once its out there... then its out there! e.g once you share a \"forbidden\" dataset and people use it, all become co-conspirators. To properly assess who was using the dataset before and after is not feasible (at maximum accuracy). \n\n5th- the rules are quite often NOT clear and you might have to read 5-10 different threads and discussions to be up to date with what is allowed and what's not. In the recent *Sound classification competition* I was using `imagenet` until before the merge deadline and only then I realised that it was not allowed (and i was actually lucky to find that). The rules (or their interpretations) are scattered everywhere! \n\n6th- Changing the rules midway is never a good strategy. This reinforces the behaviour of cheating and I appreciate you say that people will be removed/banned, but I think this cannot happen optimally (IMO), nor I think we should fully blame the person who shared this. Kaggle is partly to blame for that. \n\nBased on these, I have a few suggestions. I am not saying these are the perfect solutions to these problems (and some may sound a bit rough), but at least can be food for thought. \n\n1) Less ranking points (in all competitions, discussions, kernels) for more \"dangerous\" competitions (as in prone to be hacked with lots of external information about them) . \n2) Some solo (or not) competitions with disabled kernels/discussions, disabling external data. This can make things clearer of who is doing what. Generally discussions and kernels should be blocked something like 14 days before the competition ends .\n3) EXTRA Points' penalties for cheating (not only the ones you lose if you get removed)- this will force people to be more careful. Maybe ban/block accounts for certain days if they do it consistently (like yellow card, then red card). \n4) Give clear examples of what is allowed and what is not allowed in the competitions' rules so that it is easy for people to follow. Like for example , \"you cannot use this, but you could use that \". You have done this in a few competitions as bullet points , but in my opinion, it needs to be more thorough. \n5) There needs to be some form of regulation on these private forums that keep sharing stuff (some quite large). I know it cannot be perfect, but every little helps. Also there should be people ( even volunteers) that take away what is being discussed there and bringing it into kaggle to make it fair for everybody. Maybe kaggle can give some extra prizes for these behaviours an an extra incentive?\n6) More competitions with fully annonymized data and descriptions.\n7) If competitions are very prone to cheating, maybe consider making them Masters' only?\n8) less competitions where data is public (or more can be found easily) .\n\nAnd a few closing thoughts: \n\nKudos to kaggle for creating a strong brand around its ranking system. Violations happen, because the ranks are valuable. I think kaggle needs to continue protecting what it has built. To be fair, even if the evaluation system has flaws (and is prone to cheating), it is much better than anything else I 've seen out there trying to do something similar. What I am trying to say is that this is not an easy task. \n\nI think that kaggle historically has done well to adapt to new situations and address them, so I am sure they will come up with a better formula around this. ", "551501": "> The unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle. In addition, it means the scientific competitions we do launch are less likely to be Featured competitions, i.e., they will be launched as Research competitions without points or medals, and with smaller prize pools.\n\nThis really pisses me off to no end. I was really hoping to see more relevant and competitive science challenges on Kaggle. In addition to all the other high-level cheating that has become rampant on Kaggle, I almost feel like giving up on competitions altogether. \nHere is just one more suggestion about how to deal with this going forward: create invitation-only Kaggle challenges. Invite only Kagglers with no history of impropriety or appearance of impropriety. ", "551483": "@inversion: Thank you for such a clear statement about the rules. Allowing use of QM7 and QM9 data may not be so bad. It seems papers on this go back a few years, so there are probably many potential chemists who would already know this data well and it would inform their modeling even if only tangentially. Allowing the rest of us to consider QM7 and QM9 might level the field some. Especially since probably some would have secretly used it anyway.\n\nI very much hope that you will reconsider your view on future scientific competitions, these are great challenges and very interesting to many of us. This is true in spite of the issues about external data and the likely presence of it for most scientific problems. It would be a shame to see fewer scientific or engineering opportunities on Kaggle.", "551488": "Thank you for this update!\n\nI'm really sad that this situation could case less scientific competitions in the future. I hope we (kagglers) would be able to help science in future and won't ruin competitions (though I think we didn't really ruin this competition).", "551994": "@inversion: Thank you for your works to make the competition fair. I came from computational chemistry field and I'm really excited about what happens to chemistry with the power of data science and kagglers.\n\nI'm worried that something like another unfairness will happen because of the ambiguity of the rule about the usage of molecular simulation schemes. When I read the rules, I could not understand which level of simulation schemes are allowed and which kind of data must be shared as external data. In this nice discussion below, the answer was given by the competition host.\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/94457#latest-546076\n\nIn my opinion, the information given in the discussion should be officially announced (or added to rules) in easy words. I am really afraid that participants who are not familiar with chemistry or do not know the regulation may violate the terms without being aware of it.\n\nI understand the difficulty of organizing a scientific competition. I hope the competition will be success and contribute to chemistry world! Moreover I hope there will be a lot of scientific competitions at kaggle.", "551480": "Thanks for the clarification. \n\nI may be wrong but I don't think these datasets will be very useful for our problem as they do not contain any magnetic information.\n", "558738": "@inversion Thank you for sponsoring this challenge and for writing this update.\n\nLike the \"Higgs Boson\" and a few other challenges, this challenge will provide the science research with some new tools and techniques. I compare these challenges with old-time challenges that led Louis Pasteur and other scientists put forth their scientific concepts etc. \n\nIn another challenge, the first winner got a few million dollars. The company could not and did not use the first winner's method. Instead of worrying about that, the company felt happy that the challenge could contribute to the Predictive Analytics field.\n\nYour concerns are quite legitimate. To avoid such concerns in future science-related challenges:\n\n- The sponsor may state the front-end and back-end objectives clearly.\n\n- Kaggle may evolve a multi-point scoring system, involving Leaderboard score, external dependencies, affiliations, etc.\n\nPlease note that while a few participants are using these challenges for winning prizes, most of us (including me) are using them for learning.\n\nI encourage and plead you to sponsor more such challenges in future. I understand you may have tight budgets, but I guess that the benefits to the sponsors from a Kaggle challenge generally outweigh the cost.\n\nThank you.", "551750": "Excluding external data makes it harder to defend our value against those who think Kaggle competitions are mere curve-fitting contests and not relevant to real-world data science. Why are we allowing this sort of thing?\n\nAlso, as others have echoed, there is so much sharing of code, data, algos etc. that is repeatedly ignored by Kaggle management. I would prefer focusing on this type of abuse vs. fretting about those who post public code with publicly available data.", "551532": "Wow, am I solely responsible here for \"that we will have fewer opportunities to host scientific challenges on Kaggle\", \"scientific competitions we do launch are less likely to be Featured\" and \"with smaller prize pools\"? Because you make it feel like that. No problem with barely defined rules on that topic? It is only stated that the dataset is not allowed to test and develop ones model, not that it is not allowed to study it and share. No problem with that the dataset is available to everyone anyway? No problem at all that you by yourself was interested in \"how much lift this data provides\" and you blame others for been interested in that as well? This last one is important question, because if openly available dataset with enough extra information is present in a competition it would destroy silver/bronze medals range, where the models are not checked anymore. Maybe this is something Kaggle can compromise on, and I am fine with it, I know it is very hard to design a perfect competition, but I want to know it before I decide to participate and spend significant amount of my time on the problem. \n\nBy the way, I have no idea where you take this \"feel entitled to abuse the rules\" and \"break competition rules because you disagree with\" from. I explained myself multiple times, I understood this sentence \"not allowed to test and develop models\" in the rules as that it is not allowed that one's final submitted models are using the dataset or fitted to the dataset in some way. \n\nMoreover, I un-shared my work the minute I understood from you that it was not allowed to share. It was up something like 24h, just a piece of code and a link to a dataset that is available online anyway. \n\nAnyway... like many others I think this competition is a really beautiful one, with interesting data, and hope it continues with no more problems. Good luck to the participants and I hope some interesting approaches will appear.", "551529": "I am bit saddened that scientific value of the competition has been lowered b/c of this issue. I have two questions:\n\n1. Could Kaggle verify the top money and at least gold solutions for evidence that the forbidden data was not used? If a cheater is caught, it would be get a temp ban from Kaggle, perm bans for repeating cheaters. For money-seeking cheaters, maybe require some level (Expert/Master, etc.) to qualify to win monetary prices.\n\n2. I understand molecules cannot be anonymized, but it looks like *all* molecules in the test set are included in the forbidden datasets. Would it make sense to have a 2-stage competition where 2nd stage test set molecules are not included in forbidden datasets?\n\n@inversion as you value community feedback and dialogue I'd be very interested in your stance on 1) and 2) and a bit more clarity on what you call _conceptual reasons_: I have only high-school knowledge of chemistry, and reading the rules and the explicit restriction of quantum mechanics approaches I thought that the creation of these datasets (QM7/QM9) was very expensive as well and solutions based on them would provide less scientific value.", "551492": ">If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.\n\nSo, basically, you allowed using the data, which could harm the research, which is essentially the only valuable reason for this competition? To be honest, I'm a little shocked: you could not look after compliance with the rules and the result was not long in coming: \n\n>The unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.\n\nDon't you think that you have to stand firm on your \"No\" in such cases (not allowing the usage of data)?", "551572": "> The obvious scientific benefit from this challenge is being able to predict molecular properties without doing the expensive quantum calculations. If you were doing this as a researcher, you wouldn't add in quantum computation properties as inputs to your model, as that defeats the purpose.\n\nIf I understand correctly, the scientific benefit is to predict **faster**. The quantum computation properties shall not be used, only because that the quantum computation takes a very long time, not because they use physical laws to do the calculations.\n\nBut what if there exist a big dataset with all the quantum computation properties, so that you do not need to re-calculate them? If you are a researcher, not competing for a kaggle competition but rather aiming for a paper, and you know such information makes your prediction better, would you not use it? Does intentionally not using such information do any good to science?\n\nIn my opinion, the biggest concern is that using such external dataset might undermine the generalizability of the model, while since all the molecules in the test set are from this dataset, there is no way for you to check such overfitting. My suggestion is the same as @antorsae pointed out, having a new or a second test set where all the molecules are not in the external dataset would be the best solution.", "551514": "I am from the scientific world and I am applying data science in my own research to improve the current methods in my own field. There are of course a lot of improvements that can be made in natural sciences using the knowledge of our fellow data scientists. However, my experience is that, if a question can be solved using a strict physical approach (such approaches are often computationally expensive), it is very hard to find a data science model to beat the accuracy. The advantage of the machine learning ways are the speed, therefore many problems can be solved in real-time and high-throughput data can be analyzed efficiently.\n\nI believe that the aim of this competition is also to find a machine learning model that performs similarly as the state-of-the-art quantum calculation method, while being much faster than the physical way. In my opinion, no matter what external data people use, as long as such model works on every molecules, this is a good model to me.\n\nI think the \"the spirit of the scientific challenge\" is not \"can we beat the current physical method with pure data science\", but rather \"can we make a faster and better model using whatever we can\".", "552219": "", "551997": "I have not been around here long, but I just want to post my support for the kaggler[s] who are willing to break rules and tarnish their own image for the sake of transparency, honesty, and fairness. I'd rather have no competitions than unfair competitions, and shame on everyone who'd rather just sweep things under the rug. I read the original thread and I was very disappointed by the initial replies to that kaggler.", "573645": "> To address this current competition, we are updating the rules to allow the use of QM7 and QM9 published data, if participants wish to use it. Participants still may not use quantum calculations to obtain predictions.\n\nDo you mean we can use QM7 and QM9 for training ML models ? I understand we should not use them for test data. But I'm interested in utilizing them for training and approximating their relationship.", "551499": "> Once the competition finishes, CHAMPS would like to invite the top teams to present their work, discuss the details of their models, and work with them to write a joint research publication which discusses an open-source implementation of the solution.\n\nMy understanding of this sentence is that you will discuss with each top team their approach. Thus, if you find any usage of quantum mechanics features (looking at the code) can't you just disqualify that team? Also, before that discussion you can ask to have access to source code to analyse it.\n\nBesides that, like @areveillon said, if a team decided not to use quantum mechanics features what will happen? IMO and since the goal is to predict scalar coupling without the expensive quantum calculations you should focus on those approaches and do not allow QMs (from any source). In the end you can assess each top team to validate it (I now it can be cumbersome and a lot of work, but it is what it is until better way to evaluate the usage of such features)", "553808": "@inversion very appreciate your effort 🥇. It's really great thoughts.", "552904": "I am looking forward for your excellent ideas....!!! ", "551481": "Thanks for this clarification.\n\nYou said that there is little to no benefits of using the QM7 QM9 data set. \n\nEven if the benefit is small, does that mean that all Kagglers who are not using it will have an handicap? I'm not planning on using it so I would like to make sure it still make sense to do the comp without it.\n\nThanks!", "558681": "I might be missing something obvious, but..I really don't get why setting the \"NO QM\" rule in first place? - and tell me exactly what that means, cause almost everything here can be labeled as QM. \n\nThe challenge is to deliver a quick solution to a problem that nowadays takes too much time and too many PhDs to be addressed. ", "558508": "> The unfortunate reality of this situation is that we will have fewer opportunities to host scientific challenges on Kaggle.\n\n😩", "552830": "Are scalar coupling constants for the private set available in the public domain or locked on organizers' hard drive? I am trying to anticipate the next leak. I am guessing it is the former.", "551884": "I don't want to be boring, but I see that kind of situation happening a lot of time in Kaggle's competitions where external data can be used and thus creates a kind of unfairness situation between competitors who play by the rules and those who don't. \n\nThe only way is to set up a true evaluation method based on the code ! So for the private and final leaderboard, each competitor gives the code which has to take the data in input and output the prediction. In that way only we are sure that no external data can be used. Indeed, if your code is based on external data it would not be runnable without it so the code would not give any prediction.\nIt's as simple as that, even though I understand it is harder to set up this kind of evaluation method.", "553288": "Thanks", "554216": "Thanks for the clarifications!" } }