Hello I’m a Chemistry Ph.D. student from Japan.\nI have a strong aspiration to combine the chemistry and machine learning and I thought this competition is very interesting. I want to thank competition host for giving me a good opportunity .
\n\nTo make a fertile competition, I want to share my notice about the dataset.
\nI found that some molecules in dataset (including training and test) are structurally impossible (e.g. dsgdb9nsd_063191).\nI was wondering how these dataset were prepared.\n(I assume that collecting large number of chemical compound data for machine learning is painful task. )
I want to ask everyone that, how I should treat the structurally impossible data.
", "messages": [ { "id": "541315", "postDate": "06/02/2019 08:09:29", "content": "Hello I’m a Chemistry Ph.D. student from Japan.\nI have a strong aspiration to combine the chemistry and machine learning and I thought this competition is very interesting. I want to thank competition host for giving me a good opportunity .
\n\nTo make a fertile competition, I want to share my notice about the dataset.
\nI found that some molecules in dataset (including training and test) are structurally impossible (e.g. dsgdb9nsd_063191).\nI was wondering how these dataset were prepared.\n(I assume that collecting large number of chemical compound data for machine learning is painful task. )
I want to ask everyone that, how I should treat the structurally impossible data.
", "rawMarkdown": "Hello I’m a Chemistry Ph.D. student from Japan.\nI have a strong aspiration to combine the chemistry and machine learning and I thought this competition is very interesting. I want to thank competition host for giving me a good opportunity\t.\n\nTo make a fertile competition, I want to share my notice about the dataset.\t \nI found that some molecules in dataset (including training and test) are structurally impossible (e.g. dsgdb9nsd_063191).\nI was wondering how these dataset were prepared.\n(I assume that collecting large number of chemical compound data for machine learning is painful task. )\n\nI want to ask everyone that, how I should treat the structurally impossible data.", "votes": null }, { "id": "541358", "postDate": "06/02/2019 09:51:56", "content": "Interesting observation. Thanks for the insight. Hope they answer you.
", "rawMarkdown": "Interesting observation. Thanks for the insight. Hope they answer you.", "votes": null }, { "id": "541362", "postDate": "06/02/2019 10:02:01", "content": "As I see it, one of the purposes of the competition is to avoid doing theoretically correct but computationally intensive QM calculations. This would also make any resulting algorithm robust to structurally impossible data. Of course, any quick QM calculations we can do from the structure data would be useful. If for certain molecules these are impossible then the feature value becomes missing and we have to use techniques for handling missing data from statistics and ML.
", "rawMarkdown": "As I see it, one of the purposes of the competition is to avoid doing theoretically correct but computationally intensive QM calculations. This would also make any resulting algorithm robust to structurally impossible data. Of course, any quick QM calculations we can do from the structure data would be useful. If for certain molecules these are impossible then the feature value becomes missing and we have to use techniques for handling missing data from statistics and ML.", "votes": null }, { "id": "541363", "postDate": "06/02/2019 10:03:37", "content": "Btw, I used to be an Associate Professor of Theoretical Chemistry at Tohoku University and I have extensive experience in ML and am looking to team up if you are interested please email me: robert.burbidge@yahoo.co.uk.
", "rawMarkdown": "Btw, I used to be an Associate Professor of Theoretical Chemistry at Tohoku University and I have extensive experience in ML and am looking to team up if you are interested please email me: robert.burbidge@yahoo.co.uk.", "votes": null }, { "id": "541481", "postDate": "06/02/2019 14:16:01", "content": "This is my worry too. As the molecules are static, and sometimes unphysical, the calculated J-coupling will not match the measured average from NMR.
", "rawMarkdown": "This is my worry too. As the molecules are static, and sometimes unphysical, the calculated J-coupling will not match the measured average from NMR.", "votes": null }, { "id": "541548", "postDate": "06/02/2019 16:26:53", "content": "Could someone with more chemistry knowledge than me estimate how many of the records are structurally impossible?
", "rawMarkdown": "Could someone with more chemistry knowledge than me estimate how many of the records are structurally impossible?", "votes": null }, { "id": "541550", "postDate": "06/02/2019 16:29:26", "content": "From the point of view of this competition, we are trying (I assume) to estimate the calculated J-couplings even if they are unphysical, on the train and test set. How useful this is in the real world (actual NMR) is a different question. Please correct me if I have misunderstood.
", "rawMarkdown": "From the point of view of this competition, we are trying (I assume) to estimate the calculated J-couplings even if they are unphysical, on the train and test set. How useful this is in the real world (actual NMR) is a different question. Please correct me if I have misunderstood.", "votes": null }, { "id": "541621", "postDate": "06/02/2019 18:48:16", "content": "dsgdb9nsd_063191 doesn't see anything wrong to me. Sure it is highly strained, but nothing structurally impossible. I noticed some structures have some atoms placed very close each other, which tricks visualization software to display bond between them. This is due to file format being a simple XYZ format file. This is common in QM software.
", "rawMarkdown": "dsgdb9nsd_063191 doesn't see anything wrong to me. Sure it is highly strained, but nothing structurally impossible. I noticed some structures have some atoms placed very close each other, which tricks visualization software to display bond between them. This is due to file format being a simple XYZ format file. This is common in QM software.", "votes": null }, { "id": "541630", "postDate": "06/02/2019 19:19:41", "content": "Yes you are correct. Perhaps once they have a fast method for predicting J-coupling for a static molecule it will be trivial to generate a range of conformations that can be averaged to match experiment.
", "rawMarkdown": "Yes you are correct. Perhaps once they have a fast method for predicting J-coupling for a static molecule it will be trivial to generate a range of conformations that can be averaged to match experiment.", "votes": null }, { "id": "541644", "postDate": "06/02/2019 20:02:24", "content": "I don't think any are impossible.
\n\nLooking at Aaron Koch's kernel here:\nhttps://www.kaggle.com/aekoch95/bonds-from-structure-data
\n\nHe states:\n\"there is a range of valid bond counts we could get for each atom.
\n\nHydrogen\n1\nFlourine\n1\nOxygen\n1 - 2\nNitrogen\n1 - 3\nCarbon\n2 - 4\nWhen we graph the number of bonds for each element we see that these conditions are met, with the sole exception of several Nitrogen atoms forming 4 bonds.\"
\n\nExcept nitrogen does form four bonds and takes on a positive charge, which is then balanced by an anion (like Cl-, e.g. tetraethylammonium chloride). This is a common salt in active pharmaceutical ingredients, which is one of the areas this competition is interested in, so I can imagine the founders have included charged molecules in the structure sets.
\n\nSo all the molecules are possible to exist stably at 0 K. But they are not fully geometrically relaxed, which probably requires some level of DFT.
", "rawMarkdown": "I don't think any are impossible.\n\nLooking at Aaron Koch's kernel here:\nhttps://www.kaggle.com/aekoch95/bonds-from-structure-data\n\nHe states:\n\"there is a range of valid bond counts we could get for each atom.\n\nHydrogen\n1\nFlourine\n1\nOxygen\n1 - 2\nNitrogen\n1 - 3\nCarbon\n2 - 4\nWhen we graph the number of bonds for each element we see that these conditions are met, with the sole exception of several Nitrogen atoms forming 4 bonds.\"\n\nExcept nitrogen does form four bonds and takes on a positive charge, which is then balanced by an anion (like Cl-, e.g. tetraethylammonium chloride). This is a common salt in active pharmaceutical ingredients, which is one of the areas this competition is interested in, so I can imagine the founders have included charged molecules in the structure sets.\n\nSo all the molecules are possible to exist stably at 0 K. But they are not fully geometrically relaxed, which probably requires some level of DFT.", "votes": null }, { "id": "541691", "postDate": "06/02/2019 22:32:41", "content": "Thanks for the insight into the 4 bond Nitrogens. I had not considered the possibility of charged molecules in my analysis. Silly mistake in hindsight.
", "rawMarkdown": "Thanks for the insight into the 4 bond Nitrogens. I had not considered the possibility of charged molecules in my analysis. Silly mistake in hindsight.", "votes": null }, { "id": "541738", "postDate": "06/03/2019 01:23:35", "content": "I agree. While it looks highly strained, it does not seem impossible. And even though bond information is not included in the xyz file, a program like PyMol will always show the correct bonding since explicit hydrogens are included. It does not get \"tricked\" like you suggest. If you see something like a carbon with three bonds (like in dsgdb9nsd_063191), it is because one is a double bond.
", "rawMarkdown": "I agree. While it looks highly strained, it does not seem impossible. And even though bond information is not included in the xyz file, a program like PyMol will always show the correct bonding since explicit hydrogens are included. It does not get \"tricked\" like you suggest. If you see something like a carbon with three bonds (like in dsgdb9nsd_063191), it is because one is a double bond.", "votes": null }, { "id": "541933", "postDate": "06/03/2019 08:07:28", "content": "dsgdb9nsd_063191 looks sensible to me, although not necessarily very stable. There were originally a few structures with very weird structures, but they also had weird scalar coupling constants, so they were removed as outliers.\nThere might very well be nonphysical structures in the set, but that doesn't necessarily make the scalar coupling constants from it invalid. You are however free to remove outliers or unphysical structures from the training set if you feel like it would help lower the prediction error.
Thank you for the comment and offer for the team up.\nCurrently I'm working with my friend and we are sorry to turn down this proposal.\nI am looking forward to working with you another time.
", "rawMarkdown": "Thank you for the comment and offer for the team up.\nCurrently I'm working with my friend and we are sorry to turn down this proposal.\nI am looking forward to working with you another time.", "votes": null }, { "id": "542293", "postDate": "06/03/2019 16:59:16", "content": "Thank you for the comment. And thank you for holding this exciting competition.
\n\nThinking in term of bond angle and ring strain, I thought some of the structure might be impossible to exist. Although there might be some rules for the chemical compounds that yet to be defined. May be we can find some out from the machine learning. \nI understand your purposes. I will do my best.\nThank you.
", "rawMarkdown": "Thank you for the comment. And thank you for holding this exciting competition.\n\nThinking in term of bond angle and ring strain, I thought some of the structure might be impossible to exist. Although there might be some rules for the chemical compounds that yet to be defined. May be we can find some out from the machine learning. \nI understand your purposes. I will do my best.\nThank you.", "votes": null }, { "id": "542304", "postDate": "06/03/2019 17:08:33", "content": "Thank you for the comment.\nAs you mentioned, all molecules in the data set are possible in the terms of valid bond counts for each atom.\nWhile at the same time, if you want to identify the stable molecule, you need to think about the bond angle and ring strain (of course there are a lot more to concern).\nThere are rule called Bredt rule (https://doi.org/10.1002/anie.197304641) and some compounds in the data set does not qualify this rule.
", "rawMarkdown": "Thank you for the comment.\nAs you mentioned, all molecules in the data set are possible in the terms of valid bond counts for each atom.\nWhile at the same time, if you want to identify the stable molecule, you need to think about the bond angle and ring strain (of course there are a lot more to concern).\nThere are rule called Bredt rule (https://doi.org/10.1002/anie.197304641) and some compounds in the data set does not qualify this rule.", "votes": null }, { "id": "542356", "postDate": "06/03/2019 18:39:23", "content": "You are correct. Some of these structures are only stable at absolute zero, and would decompose at higher temperatures.
", "rawMarkdown": "You are correct. Some of these structures are only stable at absolute zero, and would decompose at higher temperatures.", "votes": null } ], "comments": [ { "id": 541358, "author_name": "abhinand05", "author_url": "", "post_date": "06/02/2019 09:51:56", "content": "Interesting observation. Thanks for the insight. Hope they answer you.
", "votes": null, "replies": [] }, { "id": 541362, "author_name": "robertburbidge", "author_url": "", "post_date": "06/02/2019 10:02:01", "content": "As I see it, one of the purposes of the competition is to avoid doing theoretically correct but computationally intensive QM calculations. This would also make any resulting algorithm robust to structurally impossible data. Of course, any quick QM calculations we can do from the structure data would be useful. If for certain molecules these are impossible then the feature value becomes missing and we have to use techniques for handling missing data from statistics and ML.
", "votes": null, "replies": [] }, { "id": 541363, "author_name": "robertburbidge", "author_url": "", "post_date": "06/02/2019 10:03:37", "content": "Btw, I used to be an Associate Professor of Theoretical Chemistry at Tohoku University and I have extensive experience in ML and am looking to team up if you are interested please email me: robert.burbidge@yahoo.co.uk.
", "votes": null, "replies": [ { "id": 542277, "author_name": "luckyland", "author_url": "", "post_date": "06/03/2019 16:38:06", "content": "Thank you for the comment and offer for the team up.\nCurrently I'm working with my friend and we are sorry to turn down this proposal.\nI am looking forward to working with you another time.
", "votes": null, "replies": [] } ] }, { "id": 541481, "author_name": "phucco", "author_url": "", "post_date": "06/02/2019 14:16:01", "content": "This is my worry too. As the molecules are static, and sometimes unphysical, the calculated J-coupling will not match the measured average from NMR.
", "votes": null, "replies": [ { "id": 541550, "author_name": "robertburbidge", "author_url": "", "post_date": "06/02/2019 16:29:26", "content": "From the point of view of this competition, we are trying (I assume) to estimate the calculated J-couplings even if they are unphysical, on the train and test set. How useful this is in the real world (actual NMR) is a different question. Please correct me if I have misunderstood.
", "votes": null, "replies": [] }, { "id": 541630, "author_name": "phucco", "author_url": "", "post_date": "06/02/2019 19:19:41", "content": "Yes you are correct. Perhaps once they have a fast method for predicting J-coupling for a static molecule it will be trivial to generate a range of conformations that can be averaged to match experiment.
", "votes": null, "replies": [] } ] }, { "id": 541548, "author_name": "robertburbidge", "author_url": "", "post_date": "06/02/2019 16:26:53", "content": "Could someone with more chemistry knowledge than me estimate how many of the records are structurally impossible?
", "votes": null, "replies": [ { "id": 541644, "author_name": "phucco", "author_url": "", "post_date": "06/02/2019 20:02:24", "content": "I don't think any are impossible.
\n\nLooking at Aaron Koch's kernel here:\nhttps://www.kaggle.com/aekoch95/bonds-from-structure-data
\n\nHe states:\n\"there is a range of valid bond counts we could get for each atom.
\n\nHydrogen\n1\nFlourine\n1\nOxygen\n1 - 2\nNitrogen\n1 - 3\nCarbon\n2 - 4\nWhen we graph the number of bonds for each element we see that these conditions are met, with the sole exception of several Nitrogen atoms forming 4 bonds.\"
\n\nExcept nitrogen does form four bonds and takes on a positive charge, which is then balanced by an anion (like Cl-, e.g. tetraethylammonium chloride). This is a common salt in active pharmaceutical ingredients, which is one of the areas this competition is interested in, so I can imagine the founders have included charged molecules in the structure sets.
\n\nSo all the molecules are possible to exist stably at 0 K. But they are not fully geometrically relaxed, which probably requires some level of DFT.
", "votes": null, "replies": [] }, { "id": 541691, "author_name": "aekoch95", "author_url": "", "post_date": "06/02/2019 22:32:41", "content": "Thanks for the insight into the 4 bond Nitrogens. I had not considered the possibility of charged molecules in my analysis. Silly mistake in hindsight.
", "votes": null, "replies": [] }, { "id": 542304, "author_name": "luckyland", "author_url": "", "post_date": "06/03/2019 17:08:33", "content": "Thank you for the comment.\nAs you mentioned, all molecules in the data set are possible in the terms of valid bond counts for each atom.\nWhile at the same time, if you want to identify the stable molecule, you need to think about the bond angle and ring strain (of course there are a lot more to concern).\nThere are rule called Bredt rule (https://doi.org/10.1002/anie.197304641) and some compounds in the data set does not qualify this rule.
", "votes": null, "replies": [] }, { "id": 542356, "author_name": "phucco", "author_url": "", "post_date": "06/03/2019 18:39:23", "content": "You are correct. Some of these structures are only stable at absolute zero, and would decompose at higher temperatures.
", "votes": null, "replies": [] } ] }, { "id": 541621, "author_name": "sunhwan", "author_url": "", "post_date": "06/02/2019 18:48:16", "content": "dsgdb9nsd_063191 doesn't see anything wrong to me. Sure it is highly strained, but nothing structurally impossible. I noticed some structures have some atoms placed very close each other, which tricks visualization software to display bond between them. This is due to file format being a simple XYZ format file. This is common in QM software.
", "votes": null, "replies": [ { "id": 541738, "author_name": "matthewmasters", "author_url": "", "post_date": "06/03/2019 01:23:35", "content": "I agree. While it looks highly strained, it does not seem impossible. And even though bond information is not included in the xyz file, a program like PyMol will always show the correct bonding since explicit hydrogens are included. It does not get \"tricked\" like you suggest. If you see something like a carbon with three bonds (like in dsgdb9nsd_063191), it is because one is a double bond.
", "votes": null, "replies": [] } ] }, { "id": 541933, "author_name": "larsbratholm", "author_url": "", "post_date": "06/03/2019 08:07:28", "content": "dsgdb9nsd_063191 looks sensible to me, although not necessarily very stable. There were originally a few structures with very weird structures, but they also had weird scalar coupling constants, so they were removed as outliers.\nThere might very well be nonphysical structures in the set, but that doesn't necessarily make the scalar coupling constants from it invalid. You are however free to remove outliers or unphysical structures from the training set if you feel like it would help lower the prediction error.
Thank you for the comment. And thank you for holding this exciting competition.
\n\nThinking in term of bond angle and ring strain, I thought some of the structure might be impossible to exist. Although there might be some rules for the chemical compounds that yet to be defined. May be we can find some out from the machine learning. \nI understand your purposes. I will do my best.\nThank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "541315": "Hello I’m a Chemistry Ph.D. student from Japan.\nI have a strong aspiration to combine the chemistry and machine learning and I thought this competition is very interesting. I want to thank competition host for giving me a good opportunity\t.\n\nTo make a fertile competition, I want to share my notice about the dataset.\t \nI found that some molecules in dataset (including training and test) are structurally impossible (e.g. dsgdb9nsd_063191).\nI was wondering how these dataset were prepared.\n(I assume that collecting large number of chemical compound data for machine learning is painful task. )\n\nI want to ask everyone that, how I should treat the structurally impossible data.", "541358": "Interesting observation. Thanks for the insight. Hope they answer you.", "541362": "As I see it, one of the purposes of the competition is to avoid doing theoretically correct but computationally intensive QM calculations. This would also make any resulting algorithm robust to structurally impossible data. Of course, any quick QM calculations we can do from the structure data would be useful. If for certain molecules these are impossible then the feature value becomes missing and we have to use techniques for handling missing data from statistics and ML.", "541363": "Btw, I used to be an Associate Professor of Theoretical Chemistry at Tohoku University and I have extensive experience in ML and am looking to team up if you are interested please email me: robert.burbidge@yahoo.co.uk.", "541481": "This is my worry too. As the molecules are static, and sometimes unphysical, the calculated J-coupling will not match the measured average from NMR.", "541548": "Could someone with more chemistry knowledge than me estimate how many of the records are structurally impossible?", "541550": "From the point of view of this competition, we are trying (I assume) to estimate the calculated J-couplings even if they are unphysical, on the train and test set. How useful this is in the real world (actual NMR) is a different question. Please correct me if I have misunderstood.", "541621": "dsgdb9nsd_063191 doesn't see anything wrong to me. Sure it is highly strained, but nothing structurally impossible. I noticed some structures have some atoms placed very close each other, which tricks visualization software to display bond between them. This is due to file format being a simple XYZ format file. This is common in QM software.", "541630": "Yes you are correct. Perhaps once they have a fast method for predicting J-coupling for a static molecule it will be trivial to generate a range of conformations that can be averaged to match experiment.", "541644": "I don't think any are impossible.\n\nLooking at Aaron Koch's kernel here:\nhttps://www.kaggle.com/aekoch95/bonds-from-structure-data\n\nHe states:\n\"there is a range of valid bond counts we could get for each atom.\n\nHydrogen\n1\nFlourine\n1\nOxygen\n1 - 2\nNitrogen\n1 - 3\nCarbon\n2 - 4\nWhen we graph the number of bonds for each element we see that these conditions are met, with the sole exception of several Nitrogen atoms forming 4 bonds.\"\n\nExcept nitrogen does form four bonds and takes on a positive charge, which is then balanced by an anion (like Cl-, e.g. tetraethylammonium chloride). This is a common salt in active pharmaceutical ingredients, which is one of the areas this competition is interested in, so I can imagine the founders have included charged molecules in the structure sets.\n\nSo all the molecules are possible to exist stably at 0 K. But they are not fully geometrically relaxed, which probably requires some level of DFT.", "541691": "Thanks for the insight into the 4 bond Nitrogens. I had not considered the possibility of charged molecules in my analysis. Silly mistake in hindsight.", "541738": "I agree. While it looks highly strained, it does not seem impossible. And even though bond information is not included in the xyz file, a program like PyMol will always show the correct bonding since explicit hydrogens are included. It does not get \"tricked\" like you suggest. If you see something like a carbon with three bonds (like in dsgdb9nsd_063191), it is because one is a double bond.", "541933": "`dsgdb9nsd_063191` looks sensible to me, although not necessarily very stable. There were originally a few structures with very weird structures, but they also had weird scalar coupling constants, so they were removed as outliers.\nThere might very well be nonphysical structures in the set, but that doesn't necessarily make the scalar coupling constants from it invalid. You are however free to remove outliers or unphysical structures from the training set if you feel like it would help lower the prediction error.", "542277": "Thank you for the comment and offer for the team up.\nCurrently I'm working with my friend and we are sorry to turn down this proposal.\nI am looking forward to working with you another time.", "542293": "Thank you for the comment. And thank you for holding this exciting competition.\n\nThinking in term of bond angle and ring strain, I thought some of the structure might be impossible to exist. Although there might be some rules for the chemical compounds that yet to be defined. May be we can find some out from the machine learning. \nI understand your purposes. I will do my best.\nThank you.", "542304": "Thank you for the comment.\nAs you mentioned, all molecules in the data set are possible in the terms of valid bond counts for each atom.\nWhile at the same time, if you want to identify the stable molecule, you need to think about the bond angle and ring strain (of course there are a lot more to concern).\nThere are rule called Bredt rule (https://doi.org/10.1002/anie.197304641) and some compounds in the data set does not qualify this rule.", "542356": "You are correct. Some of these structures are only stable at absolute zero, and would decompose at higher temperatures." }, "source": "meta" }