Considering the #2 spot had no chemistry background, I'm curious to see which teams among the, say, top 100 had chemistry background.
\n\nDefinitely, #2 having no domain expertise is a nice example that Machine Learning Expertise + Ability to navigate the scientific litterature can go a long way.
\n\nSince I'm new to Kaggle, I wonder if this is a common pattern or somewhat rare to see such scenarios?
", "messages": [ { "id": "612280", "postDate": "08/29/2019 17:41:12", "content": "Considering the #2 spot had no chemistry background, I'm curious to see which teams among the, say, top 100 had chemistry background.
\n\nDefinitely, #2 having no domain expertise is a nice example that Machine Learning Expertise + Ability to navigate the scientific litterature can go a long way.
\n\nSince I'm new to Kaggle, I wonder if this is a common pattern or somewhat rare to see such scenarios?
", "rawMarkdown": "[Considering the #2 spot had no chemistry background](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106468#latest-612269), I'm curious to see which teams among the, say, top 100 had chemistry background. \n\nDefinitely, #2 having no domain expertise is a nice example that Machine Learning Expertise + Ability to navigate the scientific litterature can go a long way. \n\nSince I'm new to Kaggle, I wonder if this is a common pattern or somewhat rare to see such scenarios?", "votes": null }, { "id": "612424", "postDate": "08/29/2019 19:32:39", "content": "I took AP Chemistry in High School. I think it helped me a lot with crafting useful lgbm features, but it did not help much with creating a NN model.
", "rawMarkdown": "I took AP Chemistry in High School. I think it helped me a lot with crafting useful lgbm features, but it did not help much with creating a NN model.", "votes": null }, { "id": "612427", "postDate": "08/29/2019 19:35:55", "content": "You probably care more about teams at the top of the top 100 (and not the bottom, like us!), but: we had two chemists on our team. We quickly realized, though, that chemistry knowledge doesn't help much. As for me, I took a year of chemistry back in college, but I've since forgotten it all, unfortunately.
", "rawMarkdown": "You probably care more about teams at the top of the top 100 (and not the bottom, like us!), but: we had two chemists on our team. We quickly realized, though, that chemistry knowledge doesn't help much. As for me, I took a year of chemistry back in college, but I've since forgotten it all, unfortunately.", "votes": null }, { "id": "612464", "postDate": "08/29/2019 20:01:15", "content": "Yes. I know next to nothing about machine learning though -- my teammates did the real work :)
", "rawMarkdown": "Yes. I know next to nothing about machine learning though -- my teammates did the real work :)", "votes": null }, { "id": "612479", "postDate": "08/29/2019 20:11:55", "content": "No, well sort of no.\nWe all come from CS/ML backgrounds, but have been working on applying ML to molecular data. Completely different context though, nothing to do with QM or NMR. \nSo, it mostly helped with getting the molecules into NNs and picking an arhitecture. Beyond that, all attempts to do something smart (throw in a feature or so) we a failure.
", "rawMarkdown": "No, well sort of no.\nWe all come from CS/ML backgrounds, but have been working on applying ML to molecular data. Completely different context though, nothing to do with QM or NMR. \nSo, it mostly helped with getting the molecules into NNs and picking an arhitecture. Beyond that, all attempts to do something smart (throw in a feature or so) we a failure.", "votes": null }, { "id": "612531", "postDate": "08/29/2019 21:08:26", "content": "Thanks for the share @grakocevic , and a 4th position- impressive.
", "rawMarkdown": "Thanks for the share @grakocevic , and a 4th position- impressive.", "votes": null }, { "id": "612532", "postDate": "08/29/2019 21:09:15", "content": "In what ways did you feel your chemistry background helped @mordechai1 ?
", "rawMarkdown": "In what ways did you feel your chemistry background helped @mordechai1 ?", "votes": null }, { "id": "612541", "postDate": "08/29/2019 21:16:05", "content": "Thanks @abendorf
\n\nYeah I'm already feeling a trend here - ML and DS knowledge seemed to be more important. Interesting to see that ML and data structure thinking can be applied to find the right model and features without heavy domain expertise.
\n\nI wonder if sometimes domain expertise cannot hinder you because it biases you towards features and knowledge you already possess in the field.
", "rawMarkdown": "Thanks @abendorf \n\nYeah I'm already feeling a trend here - ML and DS knowledge seemed to be more important. Interesting to see that ML and data structure thinking can be applied to find the right model and features without heavy domain expertise. \n\nI wonder if sometimes domain expertise cannot hinder you because it biases you towards features and knowledge you already possess in the field.", "votes": null }, { "id": "612554", "postDate": "08/29/2019 21:22:51", "content": "Thanks @petersk20 , it makes sense - citing this answer citing the Deep Learning book:
\n\n\nDeep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.\n
Easier for NNs to \"infer\" features - yet not automatic.
", "rawMarkdown": "Thanks @petersk20 , it makes sense - citing [this answer](https://stats.stackexchange.com/questions/349155/why-do-neural-networks-need-feature-selection-engineering) citing the Deep Learning book:\n\n```\nDeep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.\n``` \n\nEasier for NNs to \"infer\" features - [yet not automatic](https://stats.stackexchange.com/questions/349155/why-do-neural-networks-need-feature-selection-engineering).", "votes": null }, { "id": "612589", "postDate": "08/29/2019 22:11:13", "content": "For us, it helped to identify the architecture and design of the model. To give some examples,
\n\nThe FC term (the largest) is directly related to the s-like orbital amplitudes on the central atoms. This directly implies the importance of the bonding environment (e.g. sp4 hybridization) and let us design our pre-processing / labeling routines accordingly, as well as emphasize the angle terms.
The reported values rely on perturbation theory, which is sensitive to the excited states of the molecule. So here we can think about sigma and pi (anti-)bonds, and the chemistry that goes into those.
At one stage, the predictions were much worse for molecules with a hydrogen bond (O-H and N-H at >~1 Angstrom). In experiment, this chemistry will often cause NMR broadening due to its sensitivity. So we included these type of bonds too.
The electron screening depends on electron density. Fortunately, OpenBabel includes a partial charge scheme that can start off the network here.
Other physically-motivated design choices ended up not being useful, such as encoding bond distance in a Coulomb basis of 1/r^n (dipole and multipole terms). I guess the network learned this physics by itself.
\n\nThat said, many groups did excellently without much chemistry. It seems from the posted solutions that transformer-type models did a really nice job here, even without the effects of domain expertise.
", "rawMarkdown": "For us, it helped to identify the architecture and design of the model. To give some examples,\n\n- The FC term (the largest) is directly related to the s-like orbital amplitudes on the central atoms. This directly implies the importance of the bonding environment (e.g. sp4 hybridization) and let us design our pre-processing / labeling routines accordingly, as well as emphasize the angle terms.\n\n- The reported values rely on perturbation theory, which is sensitive to the excited states of the molecule. So here we can think about sigma and pi (anti-)bonds, and the chemistry that goes into those.\n\n- At one stage, the predictions were much worse for molecules with a hydrogen bond (O-H and N-H at >~1 Angstrom). In experiment, this chemistry will often cause NMR broadening due to its sensitivity. So we included these type of bonds too.\n\n- The electron screening depends on electron density. Fortunately, OpenBabel includes a partial charge scheme that can start off the network here.\n\nOther physically-motivated design choices ended up not being useful, such as encoding bond distance in a Coulomb basis of 1/r^n (dipole and multipole terms). I guess the network learned this physics by itself.\n\nThat said, many groups did excellently without much chemistry. It seems from the posted solutions that transformer-type models did a really nice job here, even without the effects of domain expertise.", "votes": null }, { "id": "612698", "postDate": "08/30/2019 00:23:19", "content": "Thanks for your time and words @mordechai1 , nice ideas! I agree with your conclusions it is impressive to see teams doing really great without domain expertise.
", "rawMarkdown": "Thanks for your time and words @mordechai1 , nice ideas! I agree with your conclusions it is impressive to see teams doing really great without domain expertise.", "votes": null } ], "comments": [ { "id": 612424, "author_name": "petersk20", "author_url": "", "post_date": "08/29/2019 19:32:39", "content": "I took AP Chemistry in High School. I think it helped me a lot with crafting useful lgbm features, but it did not help much with creating a NN model.
", "votes": null, "replies": [ { "id": 612554, "author_name": "michelml", "author_url": "", "post_date": "08/29/2019 21:22:51", "content": "Thanks @petersk20 , it makes sense - citing this answer citing the Deep Learning book:
\n\n\nDeep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.\n
Easier for NNs to \"infer\" features - yet not automatic.
", "votes": null, "replies": [] } ] }, { "id": 612427, "author_name": "abendorf", "author_url": "", "post_date": "08/29/2019 19:35:55", "content": "You probably care more about teams at the top of the top 100 (and not the bottom, like us!), but: we had two chemists on our team. We quickly realized, though, that chemistry knowledge doesn't help much. As for me, I took a year of chemistry back in college, but I've since forgotten it all, unfortunately.
", "votes": null, "replies": [ { "id": 612541, "author_name": "michelml", "author_url": "", "post_date": "08/29/2019 21:16:05", "content": "Thanks @abendorf
\n\nYeah I'm already feeling a trend here - ML and DS knowledge seemed to be more important. Interesting to see that ML and data structure thinking can be applied to find the right model and features without heavy domain expertise.
\n\nI wonder if sometimes domain expertise cannot hinder you because it biases you towards features and knowledge you already possess in the field.
", "votes": null, "replies": [] } ] }, { "id": 612464, "author_name": "mordechai1", "author_url": "", "post_date": "08/29/2019 20:01:15", "content": "Yes. I know next to nothing about machine learning though -- my teammates did the real work :)
", "votes": null, "replies": [ { "id": 612532, "author_name": "michelml", "author_url": "", "post_date": "08/29/2019 21:09:15", "content": "In what ways did you feel your chemistry background helped @mordechai1 ?
", "votes": null, "replies": [] }, { "id": 612589, "author_name": "mordechai1", "author_url": "", "post_date": "08/29/2019 22:11:13", "content": "For us, it helped to identify the architecture and design of the model. To give some examples,
\n\nThe FC term (the largest) is directly related to the s-like orbital amplitudes on the central atoms. This directly implies the importance of the bonding environment (e.g. sp4 hybridization) and let us design our pre-processing / labeling routines accordingly, as well as emphasize the angle terms.
The reported values rely on perturbation theory, which is sensitive to the excited states of the molecule. So here we can think about sigma and pi (anti-)bonds, and the chemistry that goes into those.
At one stage, the predictions were much worse for molecules with a hydrogen bond (O-H and N-H at >~1 Angstrom). In experiment, this chemistry will often cause NMR broadening due to its sensitivity. So we included these type of bonds too.
The electron screening depends on electron density. Fortunately, OpenBabel includes a partial charge scheme that can start off the network here.
Other physically-motivated design choices ended up not being useful, such as encoding bond distance in a Coulomb basis of 1/r^n (dipole and multipole terms). I guess the network learned this physics by itself.
\n\nThat said, many groups did excellently without much chemistry. It seems from the posted solutions that transformer-type models did a really nice job here, even without the effects of domain expertise.
", "votes": null, "replies": [] }, { "id": 612698, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 00:23:19", "content": "Thanks for your time and words @mordechai1 , nice ideas! I agree with your conclusions it is impressive to see teams doing really great without domain expertise.
", "votes": null, "replies": [] } ] }, { "id": 612479, "author_name": "grakocevic", "author_url": "", "post_date": "08/29/2019 20:11:55", "content": "No, well sort of no.\nWe all come from CS/ML backgrounds, but have been working on applying ML to molecular data. Completely different context though, nothing to do with QM or NMR. \nSo, it mostly helped with getting the molecules into NNs and picking an arhitecture. Beyond that, all attempts to do something smart (throw in a feature or so) we a failure.
", "votes": null, "replies": [ { "id": 612531, "author_name": "michelml", "author_url": "", "post_date": "08/29/2019 21:08:26", "content": "Thanks for the share @grakocevic , and a 4th position- impressive.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "612280": "[Considering the #2 spot had no chemistry background](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106468#latest-612269), I'm curious to see which teams among the, say, top 100 had chemistry background. \n\nDefinitely, #2 having no domain expertise is a nice example that Machine Learning Expertise + Ability to navigate the scientific litterature can go a long way. \n\nSince I'm new to Kaggle, I wonder if this is a common pattern or somewhat rare to see such scenarios?", "612424": "I took AP Chemistry in High School. I think it helped me a lot with crafting useful lgbm features, but it did not help much with creating a NN model.", "612427": "You probably care more about teams at the top of the top 100 (and not the bottom, like us!), but: we had two chemists on our team. We quickly realized, though, that chemistry knowledge doesn't help much. As for me, I took a year of chemistry back in college, but I've since forgotten it all, unfortunately.", "612464": "Yes. I know next to nothing about machine learning though -- my teammates did the real work :)", "612479": "No, well sort of no.\nWe all come from CS/ML backgrounds, but have been working on applying ML to molecular data. Completely different context though, nothing to do with QM or NMR. \nSo, it mostly helped with getting the molecules into NNs and picking an arhitecture. Beyond that, all attempts to do something smart (throw in a feature or so) we a failure.", "612531": "Thanks for the share @grakocevic , and a 4th position- impressive.", "612532": "In what ways did you feel your chemistry background helped @mordechai1 ?", "612541": "Thanks @abendorf \n\nYeah I'm already feeling a trend here - ML and DS knowledge seemed to be more important. Interesting to see that ML and data structure thinking can be applied to find the right model and features without heavy domain expertise. \n\nI wonder if sometimes domain expertise cannot hinder you because it biases you towards features and knowledge you already possess in the field.", "612554": "Thanks @petersk20 , it makes sense - citing [this answer](https://stats.stackexchange.com/questions/349155/why-do-neural-networks-need-feature-selection-engineering) citing the Deep Learning book:\n\n```\nDeep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.\n``` \n\nEasier for NNs to \"infer\" features - [yet not automatic](https://stats.stackexchange.com/questions/349155/why-do-neural-networks-need-feature-selection-engineering).", "612589": "For us, it helped to identify the architecture and design of the model. To give some examples,\n\n- The FC term (the largest) is directly related to the s-like orbital amplitudes on the central atoms. This directly implies the importance of the bonding environment (e.g. sp4 hybridization) and let us design our pre-processing / labeling routines accordingly, as well as emphasize the angle terms.\n\n- The reported values rely on perturbation theory, which is sensitive to the excited states of the molecule. So here we can think about sigma and pi (anti-)bonds, and the chemistry that goes into those.\n\n- At one stage, the predictions were much worse for molecules with a hydrogen bond (O-H and N-H at >~1 Angstrom). In experiment, this chemistry will often cause NMR broadening due to its sensitivity. So we included these type of bonds too.\n\n- The electron screening depends on electron density. Fortunately, OpenBabel includes a partial charge scheme that can start off the network here.\n\nOther physically-motivated design choices ended up not being useful, such as encoding bond distance in a Coulomb basis of 1/r^n (dipole and multipole terms). I guess the network learned this physics by itself.\n\nThat said, many groups did excellently without much chemistry. It seems from the posted solutions that transformer-type models did a really nice job here, even without the effects of domain expertise.", "612698": "Thanks for your time and words @mordechai1 , nice ideas! I agree with your conclusions it is impressive to see teams doing really great without domain expertise." }, "source": "meta" }