As is stated clearly in the Rules, and was clarified in the External Data thread, the use of \"data from quantum mechanics computations on the molecules from the training and test set, including the QM7 and QM9 data sets\" is not allowed.
\n\nUnfortunately, over the weekend this data as well as kernels utilizing the data were posted on Kaggle.
\n\nI am currently conferring with the host on next steps.
\n\nThe specific concern was raised on Thursday. I was out of the office Thursday afternoon and Friday, and didn't even see the concern, let alone have enough time to respond to it. Regardless of next steps, I can't emphasize enough that trying to get a question or concern addressed in this manner is extremely poor form.
", "messages": [ { "id": 549548, "postDate": "2019-06-10T19:42:53.763Z", "content": "As is stated clearly in the Rules, and was clarified in the External Data thread, the use of \"data from quantum mechanics computations on the molecules from the training and test set, including the QM7 and QM9 data sets\" is not allowed.
\n\nUnfortunately, over the weekend this data as well as kernels utilizing the data were posted on Kaggle.
\n\nI am currently conferring with the host on next steps.
\n\nThe specific concern was raised on Thursday. I was out of the office Thursday afternoon and Friday, and didn't even see the concern, let alone have enough time to respond to it. Regardless of next steps, I can't emphasize enough that trying to get a question or concern addressed in this manner is extremely poor form.
", "rawMarkdown": "As is stated clearly in the Rules, and was clarified in the External Data thread, the use of \"data from quantum mechanics computations on the molecules from the training and test set, including the QM7 and QM9 data sets\" is not allowed.\n\nUnfortunately, over the weekend this data as well as kernels utilizing the data were posted on Kaggle. \n\nI am currently conferring with the host on next steps.\n\nThe specific concern was raised on Thursday. I was out of the office Thursday afternoon and Friday, and didn't even *see* the concern, let alone have enough time to respond to it. Regardless of next steps, I can't emphasize enough that trying to get a question or concern addressed in this manner is *extremely* poor form. ", "votes": 41 }, { "id": 549616, "postDate": "2019-06-10T21:31:00.813Z", "content": "This is frustrating but also a relief. As someone who didn't know anything about the QM9 data before today- I could have spent the rest of this competition believing that I was on a level playing-field with everyone else, when in fact I wouldn't have been.
\n\nThis is very similar to what happened in the LANL earthquake competition. Some of the competitors noticed there was a published paper that included images showing characteristics of the test set. The only difference between this and LANL was that using this data wasn't explicitly against the rules so long as it was disclosed. So what do you think happened?.... It was quietly posted on the external data thread, and many (if not all) of the top teams used this information to their advantage- and I don't blame them, they wanted to win. In both situations, even if you don't explicitly use the raw data to train your model- you can \"tweak\" your models based on what you know. In LANL people tweaked their models so the average prediction was the same as the average value seen in the paper. I don't see why this competition would be any different. If the QM9 data wasn't made obvious to all competitors now then some would be using it (directly or indirectly) in secret, while others would be oblivious.
\n\nI haven't been competing on Kaggle very long, but I've started to realize that in every competition there are two objectives:\n1) The one the hosts intended the competitors to solve - a real world or business issue. (except for Santander- because who knows what they want)\n2) The one the competitors are trying to solve in order to climb the leaderboard.
\n\nIn a perfect competition these two would be the same- but if there is any possible way to achieve (2) in spite of (1) - people will find it and use it.
", "rawMarkdown": "This is frustrating but also a relief. As someone who didn't know anything about the QM9 data before today- I could have spent the rest of this competition believing that I was on a level playing-field with everyone else, when in fact I wouldn't have been.\n\nThis is very similar to what happened in the LANL earthquake competition. Some of the competitors noticed there was a published paper that included images showing characteristics of the test set. The only difference between this and LANL was that using this data wasn't explicitly against the rules so long as it was disclosed. So what do you think happened?.... It was quietly posted on the external data thread, and many (if not all) of the top teams used this information to their advantage- and I don't blame them, they wanted to win. In both situations, even if you don't explicitly use the raw data to train your model- you can \"tweak\" your models based on what you know. In LANL people tweaked their models so the average prediction was the same as the average value seen in the paper. I don't see why this competition would be any different. If the QM9 data wasn't made obvious to all competitors now then some would be using it (directly or indirectly) in secret, while others would be oblivious.\n\nI haven't been competing on Kaggle very long, but I've started to realize that in every competition there are two objectives:\n1) The one the hosts intended the competitors to solve - a real world or business issue. (except for Santander- because who knows what they want)\n2) The one the competitors are trying to solve in order to climb the leaderboard.\n\nIn a perfect competition these two would be the same- but if there is any possible way to achieve (2) in spite of (1) - people will find it and use it.", "votes": 21 }, { "id": 549931, "postDate": "2019-06-11T06:37:45.690Z", "content": "I don't want to be negative, but is is really possible to restrict using the data which is open to everyone?
\n\nI'm surely not going to use this data in my models, because I don't want to violate the rules, but, as others have stated - in LANL competition \"additional\" data was used to tweak predictions on test data. This could be the same in this competition. This data could be used to create new insights on feature generation and data processing, on postprocessing and so on.
\n\nI want this competition be interesting to participants and useful to hosts. Maybe it would be better to allow using this data?
", "rawMarkdown": "I don't want to be negative, but is is really possible to restrict using the data which is open to everyone?\n\nI'm surely not going to use this data in my models, because I don't want to violate the rules, but, as others have stated - in LANL competition \"additional\" data was used to tweak predictions on test data. This could be the same in this competition. This data could be used to create new insights on feature generation and data processing, on postprocessing and so on.\n\nI want this competition be interesting to participants and useful to hosts. Maybe it would be better to allow using this data?", "votes": 13, "replies": [ { "id": 549973, "postDate": "2019-06-11T07:24:45.630Z", "content": "Isn't the concern that allowing use of QM9 means that the \"best\" models (according to the LB) will overfit to the test data, which defeats the point of the competition? I'm not sure how a bunch of overfit models would be useful to the sponsors.
", "rawMarkdown": "Isn't the concern that allowing use of QM9 means that the \"best\" models (according to the LB) will overfit to the test data, which defeats the point of the competition? I'm not sure how a bunch of overfit models would be useful to the sponsors.", "votes": 8 }, { "id": 549985, "postDate": "2019-06-11T07:39:45.940Z", "content": "My main concern is that in the current situation most of the winners of this competition will be those who would be able to use some ideas from QM9 without using that data implicitely. And other participants will be in a disadvantage. This woudn't be good for other participants and to the hosts.
", "rawMarkdown": "My main concern is that in the current situation most of the winners of this competition will be those who would be able to use some ideas from QM9 without using that data implicitely. And other participants will be in a disadvantage. This woudn't be good for other participants and to the hosts.", "votes": 6 }, { "id": 550048, "postDate": "2019-06-11T08:56:44.523Z", "content": "This is happening in most of the Competitions. NANL is the recent example.
", "rawMarkdown": "This is happening in most of the Competitions. NANL is the recent example." }, { "id": 550056, "postDate": "2019-06-11T09:03:34.360Z", "content": "Prohibit using it as a direct input in the kernels seems fair? However, if you use it to optimize the parameters or to create better meta-features, I don't see the problem. I may be wrong though
", "rawMarkdown": "Prohibit using it as a direct input in the kernels seems fair? However, if you use it to optimize the parameters or to create better meta-features, I don't see the problem. I may be wrong though" }, { "id": 550295, "postDate": "2019-06-11T13:23:09.863Z", "content": "The overfitting problem is already present in this competition. The atom indices give a huge bias, meaning the best performing model will not generalize to any dataset. I don't know why competition organizers don't take more care in preparing their datasets...
", "rawMarkdown": "The overfitting problem is already present in this competition. The atom indices give a huge bias, meaning the best performing model will not generalize to any dataset. I don't know why competition organizers don't take more care in preparing their datasets...", "votes": 1 }, { "id": 550546, "postDate": "2019-06-11T18:14:21.893Z", "content": "Comparing this data to a plot from a paper that our winning solution btw didn't even use is a bit far-fetched. That external leaky data is a general problem on Kaggle is another topic.
", "rawMarkdown": "Comparing this data to a plot from a paper that our winning solution btw didn't even use is a bit far-fetched. That external leaky data is a general problem on Kaggle is another topic.", "votes": 2 } ] }, { "id": 550570, "postDate": "2019-06-11T18:54:33.563Z", "content": "https://www.kaggle.com/c/champs-scalar-coupling/discussion/95363#latest-550565
\n\nA very important poiny!\nIf we transform xyz structure file following openly available instructions, we will receive result identical to QM9! So one could say that we already have QM9 in available data...
", "rawMarkdown": "https://www.kaggle.com/c/champs-scalar-coupling/discussion/95363#latest-550565\n\nA very important poiny!\nIf we transform xyz structure file following openly available instructions, we will receive result identical to QM9! So one could say that we already have QM9 in available data...", "votes": 11 }, { "id": 549796, "postDate": "2019-06-11T03:44:25.027Z", "content": "Although I agree that the expression came in a poor form, I am relieved that this concern came to attention early enough.
\n\nGiven what happened in the IMET competition, people will use data if it is available on the web.
\nThe competition should either be kernel only, or any kind of data should be allowed.
Wasn't this to be totally expected? A pre-defined action/response should have been anticipated, and even accounted for (e.g. having private/public test split totally uncorrelated and private split not susceptible to forbidden datasets).
\n\nI can understand when there's an unintended leak which you cannot anticipate despite best efforts, like this one https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 which our team discovered in a previous Kaggle comp, and as as result the competition relaunched (BTW, hey @inversion we're still waiting for some Kaggle swag 😂)
\n\nSorry for being blunt, but seeing a discussion/kernel pop up and then getting back to the organizer looks like improvising.
", "rawMarkdown": "Wasn't this to be totally expected? A pre-defined action/response should have been anticipated, and even accounted for (e.g. having private/public test split totally uncorrelated and private split not susceptible to forbidden datasets). \n\nI can understand when there's an unintended leak which you cannot anticipate despite best efforts, like this one https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 which our team discovered in a previous Kaggle comp, and as as result the competition relaunched (BTW, hey @inversion we're still waiting for some Kaggle swag 😂)\n\nSorry for being blunt, but seeing a discussion/kernel pop up and then getting back to the organizer looks like improvising.", "votes": 7, "replies": [ { "id": 551473, "postDate": "2019-06-12T18:53:26.487Z", "content": "Bottom line - molecules can't be anonymized when you give structure information. That means either 1) you let everyone join all external data associated with that molecule (which completely defeats the purpose of the challenge), 2) you put some reasonable rules in place, or 3) you don't host the competition.
\n\nI'm saddened that the actions of some community members are pushing the default closer to option 3 for many scientific challenges.
\n\nUpdate here:
\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496
", "rawMarkdown": "Bottom line - molecules can't be anonymized when you give structure information. That means either 1) you let everyone join all external data associated with that molecule (which completely defeats the purpose of the challenge), 2) you put some reasonable rules in place, or 3) you don't host the competition.\n\nI'm saddened that the actions of some community members are pushing the default closer to option 3 for many scientific challenges.\n\nUpdate here:\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496", "votes": 4 }, { "id": 552384, "postDate": "2019-06-13T21:22:05.360Z", "content": "I'm very happy that you didn't choose option 3 :) Thanks, really appreciate it
", "rawMarkdown": "I'm very happy that you didn't choose option 3 :) Thanks, really appreciate it", "votes": 2 } ] }, { "id": 550431, "postDate": "2019-06-11T16:08:44.910Z", "content": "Ok. I promise I will never see quantum machine dataset. O_o. \nJust kidding. But as @artgor said, it's a bit difficult to restrict public data, and talking about them is even worse (or better) since it tells other competitors (like me :-) ) directions taken by those at the top.
", "rawMarkdown": "Ok. I promise I will never see quantum machine dataset. O_o. \nJust kidding. But as @artgor said, it's a bit difficult to restrict public data, and talking about them is even worse (or better) since it tells other competitors (like me :-) ) directions taken by those at the top.", "votes": 3 }, { "id": 549552, "postDate": "2019-06-10T19:51:17.343Z", "content": "What is the remedy for people using these data ?. how are you planning to detect If the ones who don't win the prize. and win a medal be it a gold or silver.
", "rawMarkdown": "What is the remedy for people using these data ?. how are you planning to detect If the ones who don't win the prize. and win a medal be it a gold or silver.", "votes": 2, "replies": [ { "id": 549558, "postDate": "2019-06-10T19:57:20.513Z", "content": "We're working on a plan now, but it will probably be a few days before I can give definitive next steps.
\n\nRegardless, I'd love to hear whether there's evidence for how much lift this data provides.
", "rawMarkdown": "We're working on a plan now, but it will probably be a few days before I can give definitive next steps.\n\nRegardless, I'd love to hear whether there's evidence for how much lift this data provides.", "votes": 8 }, { "id": 549565, "postDate": "2019-06-10T20:03:44.387Z", "content": "Great there should be a plan \nI think this is not is not okay. you can't create a rule and put it in the grey area. There should be strict action against the one's who are using it irrespective of whether it is giving lift or not.
\n\n> Regardless, I'd love to hear whether there's evidence for how much lift this data provides.
", "rawMarkdown": "Great there should be a plan \nI think this is not is not okay. you can't create a rule and put it in the grey area. There should be strict action against the one's who are using it irrespective of whether it is giving lift or not.\n\n> Regardless, I'd love to hear whether there's evidence for how much lift this data provides.", "votes": 2 }, { "id": 549567, "postDate": "2019-06-10T20:08:17.450Z", "content": "\n\n\nThere should be strict action against the one's who are using it irrespective of whether it is giving lift or not.
\n
I vote for execution by electric chair!
", "rawMarkdown": "> There should be strict action against the one's who are using it irrespective of whether it is giving lift or not.\n\nI vote for execution by electric chair!", "votes": 8 }, { "id": 549568, "postDate": "2019-06-10T20:09:43.120Z", "content": "Lol
", "rawMarkdown": "Lol " }, { "id": 549578, "postDate": "2019-06-10T20:27:06.807Z", "content": "@mks2192 - I don't disagree. But I'm still interested. :-)
", "rawMarkdown": "@mks2192 - I don't disagree. But I'm still interested. :-)", "votes": 1 }, { "id": 549614, "postDate": "2019-06-10T21:26:55.607Z", "content": "Regarding strict action - I can't see how it'd be possible to tell if people have used it for studying, or if it guided their model selection process.
\n\nWe saw in the LANL competition how just having a picture of the target was enough to influence many top competitors' strategies.
", "rawMarkdown": "Regarding strict action - I can't see how it'd be possible to tell if people have used it for studying, or if it guided their model selection process.\n\nWe saw in the LANL competition how just having a picture of the target was enough to influence many top competitors' strategies. ", "votes": 8 }, { "id": 549903, "postDate": "2019-06-11T06:04:24.487Z", "content": "> inversion wrote\n> \n> > Regardless, I'd love to hear whether there's evidence for how much lift this data provides.
\n\nThat is funny, this is exactly the question that I asked myself 5 days ago! Important question in this competition indeed. The features from the dataset improved the score by 0.123 in my experiment, but I used a very simple model. You can read in more detail in this post... I hope you appreciate the irony.
", "rawMarkdown": "> **inversion wrote**\n> \n> > Regardless, I'd love to hear whether there's evidence for how much lift this data provides.\n\nThat is funny, this is exactly the question that I asked myself 5 days ago! Important question in this competition indeed. The features from the dataset improved the score by 0.123 in my experiment, but I used a very simple model. You can read in more detail [in this post](https://www.kaggle.com/c/champs-scalar-coupling/discussion/95047)... I hope you appreciate the irony.", "votes": 3 }, { "id": 550078, "postDate": "2019-06-11T09:27:39.477Z", "content": "\n\n\nI vote for execution by electric chair!
\n
I guess it would be a bit extreme. Probably 5 lashes on back at Times Square or Central Park whichever is more busy would do.
", "rawMarkdown": "> I vote for execution by electric chair!\n\nI guess it would be a bit extreme. Probably 5 lashes on back at Times Square or Central Park whichever is more busy would do." }, { "id": 550154, "postDate": "2019-06-11T11:00:43.863Z", "content": "\n\n\nRegardless, I'd love to hear whether there's evidence for how much lift this data provides.
\n
@zaharch, you are right. I was reading this post thinking, this is pretty much what you were trying to achieve with your discussion where you got all the down votes and criticism. ¯_(ツ)_/¯
", "rawMarkdown": "> Regardless, I'd love to hear whether there's evidence for how much lift this data provides.\n\n@zaharch, you are right. I was reading this post thinking, this is pretty much what you were trying to achieve with your discussion where you got all the down votes and criticism. ¯\\_(ツ)_/¯", "votes": 1 }, { "id": 550531, "postDate": "2019-06-11T17:48:40.737Z", "content": "@taniaj - Discussing concerns is always okay. The issue was posting data and kernels that violated the competition rules.
", "rawMarkdown": "@taniaj - Discussing concerns is always okay. The issue was posting data and kernels that violated the competition rules.", "votes": 3 }, { "id": 550556, "postDate": "2019-06-11T18:34:25.087Z", "content": "@inversion , just to clarify it a little bit more, do I understand correctly that in you message \"Regardless, I'd love to hear whether there's evidence for how much lift this data provides.\" you encourage people to study the dataset privately and report the conclusions, and the problem appears when the corresponding code is shared? I want to pinpoint the problem here more precisely, because it is still not very clear to me where exactly the boundary is. But you promised a followup, I guess you will clarify it there.
", "rawMarkdown": "@inversion , just to clarify it a little bit more, do I understand correctly that in you message \"*Regardless, I'd love to hear whether there's evidence for how much lift this data provides.*\" you encourage people to study the dataset privately and report the conclusions, and the problem appears when the corresponding code is shared? I want to pinpoint the problem here more precisely, because it is still not very clear to me where exactly the boundary is. But you promised a followup, I guess you will clarify it there.", "votes": 5 }, { "id": 550576, "postDate": "2019-06-11T19:04:37.793Z", "content": "@inversion, the way I see it is @zaharch 's data and kernels are helping to answer your question, and probably what is also on many competitor's minds, about how much lift the data provides.
", "rawMarkdown": "@inversion, the way I see it is @zaharch 's data and kernels are helping to answer your question, and probably what is also on many competitor's minds, about how much lift the data provides.", "votes": 1 }, { "id": 550578, "postDate": "2019-06-11T19:06:13.163Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 551469, "postDate": "2019-06-12T18:50:05.847Z", "content": "I don't understand about this field well. Which data is from QM9 or 7 in the concrete? structure? millikan's charge? I'm just making models from only train_data.csv at first.
", "rawMarkdown": "I don't understand about this field well. Which data is from QM9 or 7 in the concrete? structure? millikan's charge? I'm just making models from only train_data.csv at first. ", "votes": -1 }, { "id": 551474, "postDate": "2019-06-12T18:55:01.590Z", "content": "Update here:
\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496
", "rawMarkdown": "Update here:\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496" }, { "id": 550623, "postDate": "2019-06-11T20:48:39.440Z", "content": "I really struggle to understand the logic of this rule:\n* It seems that QM9 does not contains the target directly (otherwise people would have a -20 score by now)\n* If some data are very correlated with the target, worst case, it can gives away the target on the test set. However since these data are not allowed in final solution, it is like other competitions where test set is public.\n* Maybe you can say one can use this to overfit the test set with hyper parameters and seed whatever, but the test set is pretty big, I am not sure if it can be significantly overfitted.
\n\nAm I missing something?
", "rawMarkdown": "I really struggle to understand the logic of this rule:\n* It seems that QM9 does not contains the target directly (otherwise people would have a -20 score by now)\n* If some data are very correlated with the target, worst case, it can gives away the target on the test set. However since these data are not allowed in final solution, it is like other competitions where test set is public.\n* Maybe you can say one can use this to overfit the test set with hyper parameters and seed whatever, but the test set is pretty big, I am not sure if it can be significantly overfitted.\n\nAm I missing something?", "replies": [ { "id": 551000, "postDate": "2019-06-12T08:14:16.100Z", "content": "> Using state-of-the-art methods from quantum mechanics, it is possible to accurately calculate scalar coupling constants given only a 3D molecular structure as input. However, these quantum mechanics calculations are extremely expensive (days or weeks per molecule), and therefore have limited applicability in day-to-day workflows.
\n\n>A fast and reliable method to predict these interactions will allow medicinal chemists to gain structural insights faster and cheaper, enabling scientists to understand how the 3D chemical structure of a molecule affects its properties and behavior.
\n\nIf the winning method needs features from the QM data as an input, the submitted proxy model is essentially useless since they will have to do the full QM calculations to use it, so in the end, it will be no faster or cheaper.
", "rawMarkdown": "> Using state-of-the-art methods from quantum mechanics, it is possible to accurately calculate scalar coupling constants given only a 3D molecular structure as input. However, these quantum mechanics calculations are extremely expensive (days or weeks per molecule), and therefore have limited applicability in day-to-day workflows.\n\n>A fast and reliable method to predict these interactions will allow medicinal chemists to gain structural insights faster and cheaper, enabling scientists to understand how the 3D chemical structure of a molecule affects its properties and behavior.\n\nIf the winning method needs features from the QM data as an input, the submitted proxy model is essentially useless since they will have to do the full QM calculations to use it, so in the end, it will be no faster or cheaper.", "votes": 7 }, { "id": 551265, "postDate": "2019-06-12T14:22:00.183Z", "content": "So is QM data related to this specific test set? If yes, then just please change test set to new one - problem solved.
", "rawMarkdown": "So is QM data related to this specific test set? If yes, then just please change test set to new one - problem solved.", "votes": 1 }, { "id": 551450, "postDate": "2019-06-12T18:21:37.590Z", "content": "I think is not it just a matter of changing the test set. If by chance you need to retrain your model? I think the competition should be kernel only! The concern is that the developed model will be used for other purposes (commercial or non-commercial I don't know). That's why this discussion ... This goes beyond the pretension of most competitors that it's just winning the competition, getting better placed or simply learning more about DataScience.
", "rawMarkdown": "I think is not it just a matter of changing the test set. If by chance you need to retrain your model? I think the competition should be kernel only! The concern is that the developed model will be used for other purposes (commercial or non-commercial I don't know). That's why this discussion ... This goes beyond the pretension of most competitors that it's just winning the competition, getting better placed or simply learning more about DataScience." }, { "id": 551452, "postDate": "2019-06-12T18:31:53.797Z", "content": "If the competition becomes kernel only, then it will turn into programming optimization competition due to 16 GB RAM and time limitation with our big dataset.
", "rawMarkdown": "If the competition becomes kernel only, then it will turn into programming optimization competition due to 16 GB RAM and time limitation with our big dataset.", "votes": 1 }, { "id": 551466, "postDate": "2019-06-12T18:48:25.787Z", "content": "@jacekpoplawski - The problem is that there is no way to make molecules anonymous. It would be ridiculously hard to make a new database of 100k molecules that you were certain didn't have any of these properties.
\n\nAnyway, check this out:
\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496
", "rawMarkdown": "@jacekpoplawski - The problem is that there is no way to make molecules anonymous. It would be ridiculously hard to make a new database of 100k molecules that you were certain didn't have any of these properties.\n\nAnyway, check this out:\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496", "votes": 3 }, { "id": 552387, "postDate": "2019-06-13T21:38:43.210Z", "content": "But this dataset is literally the same molecules as qm9. If there were a new test set, used in a second round, the properties people used from qm9 would not be available for these molecules and they would have to recompute them. If the second round had a short time limit, this would make it impossible to cheat using the qm9 dataset, no?
", "rawMarkdown": "But this dataset is literally the same molecules as qm9. If there were a new test set, used in a second round, the properties people used from qm9 would not be available for these molecules and they would have to recompute them. If the second round had a short time limit, this would make it impossible to cheat using the qm9 dataset, no?" } ] }, { "id": 551467, "postDate": "2019-06-12T18:48:35.630Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 549616, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-10T21:31:00.813000", "content": "This is frustrating but also a relief. As someone who didn't know anything about the QM9 data before today- I could have spent the rest of this competition believing that I was on a level playing-field with everyone else, when in fact I wouldn't have been.
\n\nThis is very similar to what happened in the LANL earthquake competition. Some of the competitors noticed there was a published paper that included images showing characteristics of the test set. The only difference between this and LANL was that using this data wasn't explicitly against the rules so long as it was disclosed. So what do you think happened?.... It was quietly posted on the external data thread, and many (if not all) of the top teams used this information to their advantage- and I don't blame them, they wanted to win. In both situations, even if you don't explicitly use the raw data to train your model- you can \"tweak\" your models based on what you know. In LANL people tweaked their models so the average prediction was the same as the average value seen in the paper. I don't see why this competition would be any different. If the QM9 data wasn't made obvious to all competitors now then some would be using it (directly or indirectly) in secret, while others would be oblivious.
\n\nI haven't been competing on Kaggle very long, but I've started to realize that in every competition there are two objectives:\n1) The one the hosts intended the competitors to solve - a real world or business issue. (except for Santander- because who knows what they want)\n2) The one the competitors are trying to solve in order to climb the leaderboard.
\n\nIn a perfect competition these two would be the same- but if there is any possible way to achieve (2) in spite of (1) - people will find it and use it.
", "votes": 21, "replies": [] }, { "id": 549931, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-06-11T06:37:45.690000", "content": "I don't want to be negative, but is is really possible to restrict using the data which is open to everyone?
\n\nI'm surely not going to use this data in my models, because I don't want to violate the rules, but, as others have stated - in LANL competition \"additional\" data was used to tweak predictions on test data. This could be the same in this competition. This data could be used to create new insights on feature generation and data processing, on postprocessing and so on.
\n\nI want this competition be interesting to participants and useful to hosts. Maybe it would be better to allow using this data?
", "votes": 13, "replies": [ { "id": 549973, "author_name": "fafafa fafafafa", "author_url": "", "post_date": "2019-06-11T07:24:45.630000", "content": "Isn't the concern that allowing use of QM9 means that the \"best\" models (according to the LB) will overfit to the test data, which defeats the point of the competition? I'm not sure how a bunch of overfit models would be useful to the sponsors.
", "votes": 8, "replies": [] }, { "id": 549985, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-06-11T07:39:45.940000", "content": "My main concern is that in the current situation most of the winners of this competition will be those who would be able to use some ideas from QM9 without using that data implicitely. And other participants will be in a disadvantage. This woudn't be good for other participants and to the hosts.
", "votes": 6, "replies": [] }, { "id": 550048, "author_name": "Manoj", "author_url": "", "post_date": "2019-06-11T08:56:44.523000", "content": "This is happening in most of the Competitions. NANL is the recent example.
", "votes": 0, "replies": [] }, { "id": 550056, "author_name": "jmtest", "author_url": "", "post_date": "2019-06-11T09:03:34.360000", "content": "Prohibit using it as a direct input in the kernels seems fair? However, if you use it to optimize the parameters or to create better meta-features, I don't see the problem. I may be wrong though
", "votes": 0, "replies": [] }, { "id": 550295, "author_name": "Matt", "author_url": "", "post_date": "2019-06-11T13:23:09.863000", "content": "The overfitting problem is already present in this competition. The atom indices give a huge bias, meaning the best performing model will not generalize to any dataset. I don't know why competition organizers don't take more care in preparing their datasets...
", "votes": 1, "replies": [] }, { "id": 550546, "author_name": "Psi", "author_url": "", "post_date": "2019-06-11T18:14:21.893000", "content": "Comparing this data to a plot from a paper that our winning solution btw didn't even use is a bit far-fetched. That external leaky data is a general problem on Kaggle is another topic.
", "votes": 2, "replies": [] } ] }, { "id": 550570, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-06-11T18:54:33.563000", "content": "https://www.kaggle.com/c/champs-scalar-coupling/discussion/95363#latest-550565
\n\nA very important poiny!\nIf we transform xyz structure file following openly available instructions, we will receive result identical to QM9! So one could say that we already have QM9 in available data...
", "votes": 11, "replies": [] }, { "id": 549796, "author_name": "pocket", "author_url": "", "post_date": "2019-06-11T03:44:25.027000", "content": "Although I agree that the expression came in a poor form, I am relieved that this concern came to attention early enough.
\n\nGiven what happened in the IMET competition, people will use data if it is available on the web.
\nThe competition should either be kernel only, or any kind of data should be allowed.
Wasn't this to be totally expected? A pre-defined action/response should have been anticipated, and even accounted for (e.g. having private/public test split totally uncorrelated and private split not susceptible to forbidden datasets).
\n\nI can understand when there's an unintended leak which you cannot anticipate despite best efforts, like this one https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 which our team discovered in a previous Kaggle comp, and as as result the competition relaunched (BTW, hey @inversion we're still waiting for some Kaggle swag 😂)
\n\nSorry for being blunt, but seeing a discussion/kernel pop up and then getting back to the organizer looks like improvising.
", "votes": 7, "replies": [ { "id": 551473, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T18:53:26.487000", "content": "Bottom line - molecules can't be anonymized when you give structure information. That means either 1) you let everyone join all external data associated with that molecule (which completely defeats the purpose of the challenge), 2) you put some reasonable rules in place, or 3) you don't host the competition.
\n\nI'm saddened that the actions of some community members are pushing the default closer to option 3 for many scientific challenges.
\n\nUpdate here:
\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496
", "votes": 4, "replies": [] }, { "id": 552384, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2019-06-13T21:22:05.360000", "content": "I'm very happy that you didn't choose option 3 :) Thanks, really appreciate it
", "votes": 2, "replies": [] } ] }, { "id": 550431, "author_name": "Adriano Avelar", "author_url": "", "post_date": "2019-06-11T16:08:44.910000", "content": "Ok. I promise I will never see quantum machine dataset. O_o. \nJust kidding. But as @artgor said, it's a bit difficult to restrict public data, and talking about them is even worse (or better) since it tells other competitors (like me :-) ) directions taken by those at the top.
", "votes": 3, "replies": [] }, { "id": 549552, "author_name": "Manoj", "author_url": "", "post_date": "2019-06-10T19:51:17.343000", "content": "What is the remedy for people using these data ?. how are you planning to detect If the ones who don't win the prize. and win a medal be it a gold or silver.
", "votes": 2, "replies": [ { "id": 549558, "author_name": "inversion", "author_url": "", "post_date": "2019-06-10T19:57:20.513000", "content": "We're working on a plan now, but it will probably be a few days before I can give definitive next steps.
\n\nRegardless, I'd love to hear whether there's evidence for how much lift this data provides.
", "votes": 8, "replies": [] }, { "id": 549565, "author_name": "Manoj", "author_url": "", "post_date": "2019-06-10T20:03:44.387000", "content": "Great there should be a plan \nI think this is not is not okay. you can't create a rule and put it in the grey area. There should be strict action against the one's who are using it irrespective of whether it is giving lift or not.
\n\n> Regardless, I'd love to hear whether there's evidence for how much lift this data provides.
", "votes": 2, "replies": [] }, { "id": 549567, "author_name": "skhan", "author_url": "", "post_date": "2019-06-10T20:08:17.450000", "content": "\n\n\nThere should be strict action against the one's who are using it irrespective of whether it is giving lift or not.
\n
I vote for execution by electric chair!
", "votes": 8, "replies": [] }, { "id": 549568, "author_name": "Manoj", "author_url": "", "post_date": "2019-06-10T20:09:43.120000", "content": "Lol
", "votes": 0, "replies": [] }, { "id": 549578, "author_name": "inversion", "author_url": "", "post_date": "2019-06-10T20:27:06.807000", "content": "@mks2192 - I don't disagree. But I'm still interested. :-)
", "votes": 1, "replies": [] }, { "id": 549614, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2019-06-10T21:26:55.607000", "content": "Regarding strict action - I can't see how it'd be possible to tell if people have used it for studying, or if it guided their model selection process.
\n\nWe saw in the LANL competition how just having a picture of the target was enough to influence many top competitors' strategies.
", "votes": 8, "replies": [] }, { "id": 549903, "author_name": "nosound", "author_url": "", "post_date": "2019-06-11T06:04:24.487000", "content": "> inversion wrote\n> \n> > Regardless, I'd love to hear whether there's evidence for how much lift this data provides.
\n\nThat is funny, this is exactly the question that I asked myself 5 days ago! Important question in this competition indeed. The features from the dataset improved the score by 0.123 in my experiment, but I used a very simple model. You can read in more detail in this post... I hope you appreciate the irony.
", "votes": 3, "replies": [] }, { "id": 550078, "author_name": "cyberia", "author_url": "", "post_date": "2019-06-11T09:27:39.477000", "content": "\n\n\nI vote for execution by electric chair!
\n
I guess it would be a bit extreme. Probably 5 lashes on back at Times Square or Central Park whichever is more busy would do.
", "votes": 0, "replies": [] }, { "id": 550154, "author_name": "Tania J", "author_url": "", "post_date": "2019-06-11T11:00:43.863000", "content": "\n\n\nRegardless, I'd love to hear whether there's evidence for how much lift this data provides.
\n
@zaharch, you are right. I was reading this post thinking, this is pretty much what you were trying to achieve with your discussion where you got all the down votes and criticism. ¯_(ツ)_/¯
", "votes": 1, "replies": [] }, { "id": 550531, "author_name": "inversion", "author_url": "", "post_date": "2019-06-11T17:48:40.737000", "content": "@taniaj - Discussing concerns is always okay. The issue was posting data and kernels that violated the competition rules.
", "votes": 3, "replies": [] }, { "id": 550556, "author_name": "nosound", "author_url": "", "post_date": "2019-06-11T18:34:25.087000", "content": "@inversion , just to clarify it a little bit more, do I understand correctly that in you message \"Regardless, I'd love to hear whether there's evidence for how much lift this data provides.\" you encourage people to study the dataset privately and report the conclusions, and the problem appears when the corresponding code is shared? I want to pinpoint the problem here more precisely, because it is still not very clear to me where exactly the boundary is. But you promised a followup, I guess you will clarify it there.
", "votes": 5, "replies": [] }, { "id": 550576, "author_name": "Tania J", "author_url": "", "post_date": "2019-06-11T19:04:37.793000", "content": "@inversion, the way I see it is @zaharch 's data and kernels are helping to answer your question, and probably what is also on many competitor's minds, about how much lift the data provides.
", "votes": 1, "replies": [] }, { "id": 550578, "author_name": "", "author_url": "", "post_date": "2019-06-11T19:06:13.163000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 551469, "author_name": "ac_yasushi", "author_url": "", "post_date": "2019-06-12T18:50:05.847000", "content": "I don't understand about this field well. Which data is from QM9 or 7 in the concrete? structure? millikan's charge? I'm just making models from only train_data.csv at first.
", "votes": -1, "replies": [] }, { "id": 551474, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T18:55:01.590000", "content": "Update here:
\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496
", "votes": 0, "replies": [] }, { "id": 550623, "author_name": "Lam Dang", "author_url": "", "post_date": "2019-06-11T20:48:39.440000", "content": "I really struggle to understand the logic of this rule:\n* It seems that QM9 does not contains the target directly (otherwise people would have a -20 score by now)\n* If some data are very correlated with the target, worst case, it can gives away the target on the test set. However since these data are not allowed in final solution, it is like other competitions where test set is public.\n* Maybe you can say one can use this to overfit the test set with hyper parameters and seed whatever, but the test set is pretty big, I am not sure if it can be significantly overfitted.
\n\nAm I missing something?
", "votes": 0, "replies": [ { "id": 551000, "author_name": "datasaurus", "author_url": "", "post_date": "2019-06-12T08:14:16.100000", "content": "> Using state-of-the-art methods from quantum mechanics, it is possible to accurately calculate scalar coupling constants given only a 3D molecular structure as input. However, these quantum mechanics calculations are extremely expensive (days or weeks per molecule), and therefore have limited applicability in day-to-day workflows.
\n\n>A fast and reliable method to predict these interactions will allow medicinal chemists to gain structural insights faster and cheaper, enabling scientists to understand how the 3D chemical structure of a molecule affects its properties and behavior.
\n\nIf the winning method needs features from the QM data as an input, the submitted proxy model is essentially useless since they will have to do the full QM calculations to use it, so in the end, it will be no faster or cheaper.
", "votes": 7, "replies": [] }, { "id": 551265, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2019-06-12T14:22:00.183000", "content": "So is QM data related to this specific test set? If yes, then just please change test set to new one - problem solved.
", "votes": 1, "replies": [] }, { "id": 551450, "author_name": "Adriano Avelar", "author_url": "", "post_date": "2019-06-12T18:21:37.590000", "content": "I think is not it just a matter of changing the test set. If by chance you need to retrain your model? I think the competition should be kernel only! The concern is that the developed model will be used for other purposes (commercial or non-commercial I don't know). That's why this discussion ... This goes beyond the pretension of most competitors that it's just winning the competition, getting better placed or simply learning more about DataScience.
", "votes": 0, "replies": [] }, { "id": 551452, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-06-12T18:31:53.797000", "content": "If the competition becomes kernel only, then it will turn into programming optimization competition due to 16 GB RAM and time limitation with our big dataset.
", "votes": 1, "replies": [] }, { "id": 551466, "author_name": "inversion", "author_url": "", "post_date": "2019-06-12T18:48:25.787000", "content": "@jacekpoplawski - The problem is that there is no way to make molecules anonymous. It would be ridiculously hard to make a new database of 100k molecules that you were certain didn't have any of these properties.
\n\nAnyway, check this out:
\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496
", "votes": 3, "replies": [] }, { "id": 552387, "author_name": "Matt", "author_url": "", "post_date": "2019-06-13T21:38:43.210000", "content": "But this dataset is literally the same molecules as qm9. If there were a new test set, used in a second round, the properties people used from qm9 would not be available for these molecules and they would have to recompute them. If the second round had a short time limit, this would make it impossible to cheat using the qm9 dataset, no?
", "votes": 0, "replies": [] } ] }, { "id": 551467, "author_name": "", "author_url": "", "post_date": "2019-06-12T18:48:35.630000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "549548": "As is stated clearly in the Rules, and was clarified in the External Data thread, the use of \"data from quantum mechanics computations on the molecules from the training and test set, including the QM7 and QM9 data sets\" is not allowed.\n\nUnfortunately, over the weekend this data as well as kernels utilizing the data were posted on Kaggle. \n\nI am currently conferring with the host on next steps.\n\nThe specific concern was raised on Thursday. I was out of the office Thursday afternoon and Friday, and didn't even *see* the concern, let alone have enough time to respond to it. Regardless of next steps, I can't emphasize enough that trying to get a question or concern addressed in this manner is *extremely* poor form. ", "549616": "This is frustrating but also a relief. As someone who didn't know anything about the QM9 data before today- I could have spent the rest of this competition believing that I was on a level playing-field with everyone else, when in fact I wouldn't have been.\n\nThis is very similar to what happened in the LANL earthquake competition. Some of the competitors noticed there was a published paper that included images showing characteristics of the test set. The only difference between this and LANL was that using this data wasn't explicitly against the rules so long as it was disclosed. So what do you think happened?.... It was quietly posted on the external data thread, and many (if not all) of the top teams used this information to their advantage- and I don't blame them, they wanted to win. In both situations, even if you don't explicitly use the raw data to train your model- you can \"tweak\" your models based on what you know. In LANL people tweaked their models so the average prediction was the same as the average value seen in the paper. I don't see why this competition would be any different. If the QM9 data wasn't made obvious to all competitors now then some would be using it (directly or indirectly) in secret, while others would be oblivious.\n\nI haven't been competing on Kaggle very long, but I've started to realize that in every competition there are two objectives:\n1) The one the hosts intended the competitors to solve - a real world or business issue. (except for Santander- because who knows what they want)\n2) The one the competitors are trying to solve in order to climb the leaderboard.\n\nIn a perfect competition these two would be the same- but if there is any possible way to achieve (2) in spite of (1) - people will find it and use it.", "549931": "I don't want to be negative, but is is really possible to restrict using the data which is open to everyone?\n\nI'm surely not going to use this data in my models, because I don't want to violate the rules, but, as others have stated - in LANL competition \"additional\" data was used to tweak predictions on test data. This could be the same in this competition. This data could be used to create new insights on feature generation and data processing, on postprocessing and so on.\n\nI want this competition be interesting to participants and useful to hosts. Maybe it would be better to allow using this data?", "550570": "https://www.kaggle.com/c/champs-scalar-coupling/discussion/95363#latest-550565\n\nA very important poiny!\nIf we transform xyz structure file following openly available instructions, we will receive result identical to QM9! So one could say that we already have QM9 in available data...", "549796": "Although I agree that the expression came in a poor form, I am relieved that this concern came to attention early enough. \n\nGiven what happened in the IMET competition, people will use data if it is available on the web. \nThe competition should either be kernel only, or any kind of data should be allowed.\n", "550986": "Wasn't this to be totally expected? A pre-defined action/response should have been anticipated, and even accounted for (e.g. having private/public test split totally uncorrelated and private split not susceptible to forbidden datasets). \n\nI can understand when there's an unintended leak which you cannot anticipate despite best efforts, like this one https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 which our team discovered in a previous Kaggle comp, and as as result the competition relaunched (BTW, hey @inversion we're still waiting for some Kaggle swag 😂)\n\nSorry for being blunt, but seeing a discussion/kernel pop up and then getting back to the organizer looks like improvising.", "550431": "Ok. I promise I will never see quantum machine dataset. O_o. \nJust kidding. But as @artgor said, it's a bit difficult to restrict public data, and talking about them is even worse (or better) since it tells other competitors (like me :-) ) directions taken by those at the top.", "549552": "What is the remedy for people using these data ?. how are you planning to detect If the ones who don't win the prize. and win a medal be it a gold or silver.", "551469": "I don't understand about this field well. Which data is from QM9 or 7 in the concrete? structure? millikan's charge? I'm just making models from only train_data.csv at first. ", "551474": "Update here:\n\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/95496", "550623": "I really struggle to understand the logic of this rule:\n* It seems that QM9 does not contains the target directly (otherwise people would have a -20 score by now)\n* If some data are very correlated with the target, worst case, it can gives away the target on the test set. However since these data are not allowed in final solution, it is like other competitions where test set is public.\n* Maybe you can say one can use this to overfit the test set with hyper parameters and seed whatever, but the test set is pretty big, I am not sure if it can be significantly overfitted.\n\nAm I missing something?", "551467": "" } }