![\"\"](\"https://i.pinimg.com/236x/8b/41/a4/8b41a42ba45cdf6aec11485e270040d4--philosophy-memes-medieval-meme.jpg\")
(EDIT) To make some of the feature engineering ideas more accessible, I have created a kernel here:\nhttps://www.kaggle.com/scaomath/qm7-coulomb-matrix-eigenvalue-features
\n\nAfter playing with @fnands 's MPNN and @hengck23 's PyG starter kit for a while, I came to realize something. One of the most important aspects of these graph-based models are that they all treat bonds as edges in a SINGLE molecule graph, and the predictions for coupling constants will be exactly the SAME when two coupling's local topological structures agree (which depends on the edge matrix you feed to the neural net).
\n\nYet no tree-based model (XGB, LGB, you name it) has this power, no matter you do GroupKFold training based on molecule names or not. One compensation is generating more topological features. For example, one of my top performing features in LGB model for 1J and 2J is the Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms.
\n\nSo one simple way to do post-processing is just averaging the same type coupling constants within the same molecule when your LGB's predictions are close. For example, if three 3JHH's predictions are 16.6, 16.5, 16.1 in the same molecule, there is a chance that the ground truth is the same for all three. Of course you wanna try this on your CV set first.
\n\nA caveat is that if your model is not accurate enough, say MAE<0.05~0.1 for a type, this may hurt than improve. For a pure LGB prediction, post-processing 3JHH and 2JHH types gives 0.004 boost on the LB (not much but still).
\n\nEDIT 2 (A simple explanation of the caveat above): Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:
\n\nModel no.1's prediction: 1.2, 1.3, 1.4, which has MAE 0.3; after averaging, your MAE is unchanged!!\nModel no.2's prediction: 1.05+1.08+0.9, which has MAE ~0.08; after averaging, your MAE becomes 0.01!!!
\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so…
", "messages": [ { "id": "597819", "postDate": "08/12/2019 19:09:46", "content": "(EDIT) To make some of the feature engineering ideas more accessible, I have created a kernel here:\nhttps://www.kaggle.com/scaomath/qm7-coulomb-matrix-eigenvalue-features
\n\nAfter playing with @fnands 's MPNN and @hengck23 's PyG starter kit for a while, I came to realize something. One of the most important aspects of these graph-based models are that they all treat bonds as edges in a SINGLE molecule graph, and the predictions for coupling constants will be exactly the SAME when two coupling's local topological structures agree (which depends on the edge matrix you feed to the neural net).
\n\nYet no tree-based model (XGB, LGB, you name it) has this power, no matter you do GroupKFold training based on molecule names or not. One compensation is generating more topological features. For example, one of my top performing features in LGB model for 1J and 2J is the Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms.
\n\nSo one simple way to do post-processing is just averaging the same type coupling constants within the same molecule when your LGB's predictions are close. For example, if three 3JHH's predictions are 16.6, 16.5, 16.1 in the same molecule, there is a chance that the ground truth is the same for all three. Of course you wanna try this on your CV set first.
\n\nA caveat is that if your model is not accurate enough, say MAE<0.05~0.1 for a type, this may hurt than improve. For a pure LGB prediction, post-processing 3JHH and 2JHH types gives 0.004 boost on the LB (not much but still).
\n\nEDIT 2 (A simple explanation of the caveat above): Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:
\n\nModel no.1's prediction: 1.2, 1.3, 1.4, which has MAE 0.3; after averaging, your MAE is unchanged!!\nModel no.2's prediction: 1.05+1.08+0.9, which has MAE ~0.08; after averaging, your MAE becomes 0.01!!!
\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so…
", "rawMarkdown": "(EDIT) To make some of the feature engineering ideas more accessible, I have created a kernel here:\nhttps://www.kaggle.com/scaomath/qm7-coulomb-matrix-eigenvalue-features\n\n\n----\n\n\nAfter playing with @fnands 's MPNN and @hengck23 's PyG starter kit for a while, I came to realize something. One of the most important aspects of these graph-based models are that they all treat bonds as edges in a SINGLE molecule graph, and the predictions for coupling constants will be exactly the SAME when two coupling's local topological structures agree (which depends on the edge matrix you feed to the neural net). \n\nYet no tree-based model (XGB, LGB, you name it) has this power, no matter you do GroupKFold training based on molecule names or not. One compensation is generating more topological features. For example, one of my top performing features in LGB model for 1J and 2J is the Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms. \n\nSo one simple way to do post-processing is just averaging the same type coupling constants within the same molecule when your LGB's predictions are close. For example, if three 3JHH's predictions are 16.6, 16.5, 16.1 in the same molecule, there is a chance that the ground truth is the same for all three. Of course you wanna try this on your CV set first. \n\nA caveat is that if your model is not accurate enough, say MAE<0.05~0.1 for a type, this may hurt than improve. For a pure LGB prediction, post-processing 3JHH and 2JHH types gives 0.004 boost on the LB (not much but still).\n\nEDIT 2 (A simple explanation of the caveat above): Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:\n\nModel no.1's prediction: 1.2, 1.3, 1.4, which has MAE 0.3; after averaging, your MAE is unchanged!!\nModel no.2's prediction: 1.05+1.08+0.9, which has MAE ~0.08; after averaging, your MAE becomes 0.01!!!\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so…", "votes": null }, { "id": "597830", "postDate": "08/12/2019 19:33:23", "content": "\n\n\nFiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds
\n
That sentence blew my mind
", "rawMarkdown": "> Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds\n\nThat sentence blew my mind", "votes": null }, { "id": "597839", "postDate": "08/12/2019 19:40:54", "content": "I tried to understand that sentence but it is too deep for me:(
\n\n
\"Averaging couplings of same atoms in similar atomic environment (measured by similarity of atomic Coulomb matrices) may improve LB score.\" is the same as you said, right?\nDo you think, there is enough of enough similar atoms in any molecule that this would help?
", "rawMarkdown": "\"Averaging couplings of same atoms in similar atomic environment (measured by similarity of atomic Coulomb matrices) may improve LB score.\" is the same as you said, right?\nDo you think, there is enough of enough similar atoms in any molecule that this would help?", "votes": null }, { "id": "597853", "postDate": "08/12/2019 20:06:36", "content": "Well. With little domain knowledge at hand, I tried to view this problem as a mathematician. The package I used is called qml.
\n\nStep 1: atom-centered Coulomb matrices are generated for two atoms.\nStep 2: similarity matrices are built respectively (I simply use exp(-C)).\nStep 3: graph Laplacian matrices, which are semi-definitive (all eigenvalues are greater or equal to 0), are built upon the similarity matrices.\nStep 4: find the Fiedler eigenvectors (the eigenvec corresponding to the smallest non-zero eigenvalue).\nStep 5: Compute their Tanimoto coefficient.
\n\nThe Fiedler eigenvector is very useful in partitioning graphs.
", "rawMarkdown": "Well. With little domain knowledge at hand, I tried to view this problem as a mathematician. The package I used is called [qml](http://www.qmlcode.org/qml.html#module-qml.representations).\n\nStep 1: atom-centered Coulomb matrices are generated for two atoms.\nStep 2: similarity matrices are built respectively (I simply use exp(-C)).\nStep 3: [graph Laplacian matrices](https://en.wikipedia.org/wiki/Laplacian_matrix), which are semi-definitive (all eigenvalues are greater or equal to 0), are built upon the similarity matrices.\nStep 4: find the Fiedler eigenvectors (the eigenvec corresponding to the smallest non-zero eigenvalue).\nStep 5: Compute their Tanimoto coefficient.\n\nThe Fiedler eigenvector is very useful in partitioning graphs.", "votes": null }, { "id": "597854", "postDate": "08/12/2019 20:09:27", "content": "Maybe, I haven't tried that. The post-processing is kinda against my philosophy about machine learning. For example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.
", "rawMarkdown": "Maybe, I haven't tried that. The post-processing is kinda against my philosophy about machine learning. For example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.", "votes": null }, { "id": "597911", "postDate": "08/12/2019 22:28:44", "content": "Yes, consider molecule_000001: all 1JHC couplings are equal. Granted, i think LGBM can handle this on its own, since if your features are rotationally invariant then two edges should look equivalent
", "rawMarkdown": "Yes, consider molecule_000001: all 1JHC couplings are equal. Granted, i think LGBM can handle this on its own, since if your features are rotationally invariant then two edges should look equivalent", "votes": null }, { "id": "597957", "postDate": "08/13/2019 00:39:35", "content": "What a cool idea for postprocessing! Though I did not observe CV increase by taking average of similar (+-0.5) predictions within types.
\n\nTalking about topological based methods, may I ask how good your GCN could achieve?
", "rawMarkdown": "What a cool idea for postprocessing! Though I did not observe CV increase by taking average of similar (+-0.5) predictions within types. \n\nTalking about topological based methods, may I ask how good your GCN could achieve?", "votes": null }, { "id": "597971", "postDate": "08/13/2019 00:59:37", "content": "charming!!
", "rawMarkdown": "charming!!", "votes": null }, { "id": "597981", "postDate": "08/13/2019 01:27:59", "content": "I don't have a GCN model, my main graph-based model is using @fnands 's MPNN's template and has 1 fold CV about -1.4.
", "rawMarkdown": "I don't have a GCN model, my main graph-based model is using @fnands 's MPNN's template and has 1 fold CV about -1.4.", "votes": null }, { "id": "598049", "postDate": "08/13/2019 03:46:07", "content": "@scaomath Teacher, I have a question. How can you calculate 1 fold CV?
\nI suppose in 1fold train you use all training data for the estimation of model parameters.\nNo validation data. I must have a wrong understanding of CV... correct me please.
Your idea is interesting, but complex molecules might have similar coupling constants which come from different structure. I suppose making the groups for averaging is not easy and risky.
", "rawMarkdown": "Your idea is interesting, but complex molecules might have similar coupling constants which come from different structure. I suppose making the groups for averaging is not easy and risky.", "votes": null }, { "id": "598065", "postDate": "08/13/2019 04:22:41", "content": "Pardon my inaccurate expression. What I meant to express was that \"I only computed the CV score for 1 CV set\". I did a 5-fold split, yet I am focusing on 1 fold to best the model, then train other 4.
", "rawMarkdown": "Pardon my inaccurate expression. What I meant to express was that \"I only computed the CV score for 1 CV set\". I did a 5-fold split, yet I am focusing on 1 fold to best the model, then train other 4.", "votes": null }, { "id": "598068", "postDate": "08/13/2019 04:28:45", "content": "Also to answer @billbafare 's question: mathematically, this is easy to explain. Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:
\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so...
", "rawMarkdown": "Also to answer @billbafare 's question: mathematically, this is easy to explain. Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:\n\n* Model no.1's prediction: 1.2, 1.3, 1.4, which has MAE 0.3; after averaging, your MAE is *unchanged*!! \n* Model no.2's prediction: 1.05+1.08+0.9, which has MAE ~0.08; after averaging, your MAE becomes 0.01!!!\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so...", "votes": null }, { "id": "598079", "postDate": "08/13/2019 04:58:18", "content": "@scaomath I understand. Thank you !
", "rawMarkdown": "scaomath I understand. Thank you !", "votes": null }, { "id": "598147", "postDate": "08/13/2019 07:41:56", "content": "Shuhan Cao, this is is very interesting. I wonder why you disclosed your best feature given very few thought of it probably?
", "rawMarkdown": "Shuhan Cao, this is is very interesting. I wonder why you disclosed your best feature given very few thought of it probably?", "votes": null }, { "id": "598227", "postDate": "08/13/2019 09:52:24", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "598455", "postDate": "08/13/2019 15:38:36", "content": "Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms. - I was reading this line 10 times to exactly understand what it means. Meanwhile, great idea. Looking forward to more such insights. I learnt something new today. Thanks for sharing it.
", "rawMarkdown": "Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms. - I was reading this line 10 times to exactly understand what it means. Meanwhile, great idea. Looking forward to more such insights. I learnt something new today. Thanks for sharing it.", "votes": null }, { "id": "598523", "postDate": "08/13/2019 17:17:13", "content": "@scaomath : May be a very simple question. Did you import QML package in the kernel?. I was trying to import QML package and figured out that it is not part of the predefined package in the docker. Is there any workaround to add it to our session?.
", "rawMarkdown": "scaomath : May be a very simple question. Did you import QML package in the kernel?. I was trying to import QML package and figured out that it is not part of the predefined package in the docker. Is there any workaround to add it to our session?.", "votes": null }, { "id": "598552", "postDate": "08/13/2019 17:50:31", "content": "\n\n\nFor example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.
\n
I wish that was forbidden in LanL ;)
", "rawMarkdown": "> For example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.\n\nI wish that was forbidden in LanL ;)", "votes": null }, { "id": "598603", "postDate": "08/13/2019 19:11:52", "content": "Have you tried? \n!pip install qml
;) Because it is not my best feature. My best LGB feature is still distance-based (variants of Andrew's initial starter code). Yet my best group feature is based on a commonly used trick in image classification applied on Local Many-Body Tensor Representation. Now that I found that GNN is vastly superior to LGB in this case, it doesn't hurt to share a little thoughts on feature engineering.
", "rawMarkdown": ";) Because it is not my best feature. My best LGB feature is still distance-based (variants of Andrew's initial starter code). Yet my best group feature is based on a commonly used trick in image classification applied on Local Many-Body Tensor Representation. Now that I found that GNN is vastly superior to LGB in this case, it doesn't hurt to share a little thoughts on feature engineering.", "votes": null }, { "id": "598847", "postDate": "08/14/2019 06:01:36", "content": "(y)
", "rawMarkdown": "(y)", "votes": null }, { "id": "598953", "postDate": "08/14/2019 09:25:15", "content": "hi everyone
", "rawMarkdown": "hi everyone", "votes": null }, { "id": "599198", "postDate": "08/14/2019 16:40:56", "content": "Awesome!
", "rawMarkdown": "Awesome!", "votes": null }, { "id": "599322", "postDate": "08/14/2019 20:17:55", "content": "Awesome!
", "rawMarkdown": "Awesome!", "votes": null }, { "id": "599478", "postDate": "08/15/2019 02:57:08", "content": "Cool.
", "rawMarkdown": "Cool.", "votes": null }, { "id": "603737", "postDate": "08/20/2019 15:54:49", "content": "Yep. My bad, I thought it was a kernels only competition. Now, I am able to install it in my system as well as in the kernel
", "rawMarkdown": "Yep. My bad, I thought it was a kernels only competition. Now, I am able to install it in my system as well as in the kernel", "votes": null } ], "comments": [ { "id": 597830, "author_name": "antorsae", "author_url": "", "post_date": "08/12/2019 19:33:23", "content": "\n\n\nFiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds
\n
That sentence blew my mind
", "votes": null, "replies": [ { "id": 597839, "author_name": "aerdem4", "author_url": "", "post_date": "08/12/2019 19:40:54", "content": "I tried to understand that sentence but it is too deep for me:(
\n\n
Well. With little domain knowledge at hand, I tried to view this problem as a mathematician. The package I used is called qml.
\n\nStep 1: atom-centered Coulomb matrices are generated for two atoms.\nStep 2: similarity matrices are built respectively (I simply use exp(-C)).\nStep 3: graph Laplacian matrices, which are semi-definitive (all eigenvalues are greater or equal to 0), are built upon the similarity matrices.\nStep 4: find the Fiedler eigenvectors (the eigenvec corresponding to the smallest non-zero eigenvalue).\nStep 5: Compute their Tanimoto coefficient.
\n\nThe Fiedler eigenvector is very useful in partitioning graphs.
", "votes": null, "replies": [] }, { "id": 598147, "author_name": "cpmpml", "author_url": "", "post_date": "08/13/2019 07:41:56", "content": "Shuhan Cao, this is is very interesting. I wonder why you disclosed your best feature given very few thought of it probably?
", "votes": null, "replies": [] }, { "id": 598640, "author_name": "scaomath", "author_url": "", "post_date": "08/13/2019 20:23:03", "content": ";) Because it is not my best feature. My best LGB feature is still distance-based (variants of Andrew's initial starter code). Yet my best group feature is based on a commonly used trick in image classification applied on Local Many-Body Tensor Representation. Now that I found that GNN is vastly superior to LGB in this case, it doesn't hurt to share a little thoughts on feature engineering.
", "votes": null, "replies": [] } ] }, { "id": 597844, "author_name": "somnambwl", "author_url": "", "post_date": "08/12/2019 19:50:44", "content": "\"Averaging couplings of same atoms in similar atomic environment (measured by similarity of atomic Coulomb matrices) may improve LB score.\" is the same as you said, right?\nDo you think, there is enough of enough similar atoms in any molecule that this would help?
", "votes": null, "replies": [ { "id": 597854, "author_name": "scaomath", "author_url": "", "post_date": "08/12/2019 20:09:27", "content": "Maybe, I haven't tried that. The post-processing is kinda against my philosophy about machine learning. For example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.
", "votes": null, "replies": [] }, { "id": 597911, "author_name": "returnofsputnik", "author_url": "", "post_date": "08/12/2019 22:28:44", "content": "Yes, consider molecule_000001: all 1JHC couplings are equal. Granted, i think LGBM can handle this on its own, since if your features are rotationally invariant then two edges should look equivalent
", "votes": null, "replies": [] }, { "id": 598552, "author_name": "cpmpml", "author_url": "", "post_date": "08/13/2019 17:50:31", "content": "\n\n\nFor example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.
\n
I wish that was forbidden in LanL ;)
", "votes": null, "replies": [] } ] }, { "id": 597957, "author_name": "billbafare", "author_url": "", "post_date": "08/13/2019 00:39:35", "content": "What a cool idea for postprocessing! Though I did not observe CV increase by taking average of similar (+-0.5) predictions within types.
\n\nTalking about topological based methods, may I ask how good your GCN could achieve?
", "votes": null, "replies": [ { "id": 597981, "author_name": "scaomath", "author_url": "", "post_date": "08/13/2019 01:27:59", "content": "I don't have a GCN model, my main graph-based model is using @fnands 's MPNN's template and has 1 fold CV about -1.4.
", "votes": null, "replies": [] }, { "id": 598049, "author_name": "xinyan989", "author_url": "", "post_date": "08/13/2019 03:46:07", "content": "@scaomath Teacher, I have a question. How can you calculate 1 fold CV?
\nI suppose in 1fold train you use all training data for the estimation of model parameters.\nNo validation data. I must have a wrong understanding of CV... correct me please.
Pardon my inaccurate expression. What I meant to express was that \"I only computed the CV score for 1 CV set\". I did a 5-fold split, yet I am focusing on 1 fold to best the model, then train other 4.
", "votes": null, "replies": [] }, { "id": 598068, "author_name": "scaomath", "author_url": "", "post_date": "08/13/2019 04:28:45", "content": "Also to answer @billbafare 's question: mathematically, this is easy to explain. Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:
\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so...
", "votes": null, "replies": [] }, { "id": 598079, "author_name": "xinyan989", "author_url": "", "post_date": "08/13/2019 04:58:18", "content": "@scaomath I understand. Thank you !
", "votes": null, "replies": [] }, { "id": 598227, "author_name": "billbafare", "author_url": "", "post_date": "08/13/2019 09:52:24", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 597971, "author_name": "ryyghfjk89", "author_url": "", "post_date": "08/13/2019 00:59:37", "content": "charming!!
", "votes": null, "replies": [] }, { "id": 598050, "author_name": "toshik", "author_url": "", "post_date": "08/13/2019 03:47:36", "content": "Your idea is interesting, but complex molecules might have similar coupling constants which come from different structure. I suppose making the groups for averaging is not easy and risky.
", "votes": null, "replies": [] }, { "id": 598455, "author_name": "manojprabhaakr", "author_url": "", "post_date": "08/13/2019 15:38:36", "content": "Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms. - I was reading this line 10 times to exactly understand what it means. Meanwhile, great idea. Looking forward to more such insights. I learnt something new today. Thanks for sharing it.
", "votes": null, "replies": [] }, { "id": 598523, "author_name": "manojprabhaakr", "author_url": "", "post_date": "08/13/2019 17:17:13", "content": "@scaomath : May be a very simple question. Did you import QML package in the kernel?. I was trying to import QML package and figured out that it is not part of the predefined package in the docker. Is there any workaround to add it to our session?.
", "votes": null, "replies": [ { "id": 598603, "author_name": "somnambwl", "author_url": "", "post_date": "08/13/2019 19:11:52", "content": "Have you tried? \n!pip install qml
Yep. My bad, I thought it was a kernels only competition. Now, I am able to install it in my system as well as in the kernel
", "votes": null, "replies": [] } ] }, { "id": 598847, "author_name": "loyalmandal", "author_url": "", "post_date": "08/14/2019 06:01:36", "content": "(y)
", "votes": null, "replies": [] }, { "id": 598953, "author_name": "imrankhan006a", "author_url": "", "post_date": "08/14/2019 09:25:15", "content": "hi everyone
", "votes": null, "replies": [] }, { "id": 599198, "author_name": "aleksandrovich", "author_url": "", "post_date": "08/14/2019 16:40:56", "content": "Awesome!
", "votes": null, "replies": [] }, { "id": 599322, "author_name": "walterbraga", "author_url": "", "post_date": "08/14/2019 20:17:55", "content": "Awesome!
", "votes": null, "replies": [] }, { "id": 599478, "author_name": "william531", "author_url": "", "post_date": "08/15/2019 02:57:08", "content": "Cool.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "597819": "(EDIT) To make some of the feature engineering ideas more accessible, I have created a kernel here:\nhttps://www.kaggle.com/scaomath/qm7-coulomb-matrix-eigenvalue-features\n\n\n----\n\n\nAfter playing with @fnands 's MPNN and @hengck23 's PyG starter kit for a while, I came to realize something. One of the most important aspects of these graph-based models are that they all treat bonds as edges in a SINGLE molecule graph, and the predictions for coupling constants will be exactly the SAME when two coupling's local topological structures agree (which depends on the edge matrix you feed to the neural net). \n\nYet no tree-based model (XGB, LGB, you name it) has this power, no matter you do GroupKFold training based on molecule names or not. One compensation is generating more topological features. For example, one of my top performing features in LGB model for 1J and 2J is the Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms. \n\nSo one simple way to do post-processing is just averaging the same type coupling constants within the same molecule when your LGB's predictions are close. For example, if three 3JHH's predictions are 16.6, 16.5, 16.1 in the same molecule, there is a chance that the ground truth is the same for all three. Of course you wanna try this on your CV set first. \n\nA caveat is that if your model is not accurate enough, say MAE<0.05~0.1 for a type, this may hurt than improve. For a pure LGB prediction, post-processing 3JHH and 2JHH types gives 0.004 boost on the LB (not much but still).\n\nEDIT 2 (A simple explanation of the caveat above): Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:\n\nModel no.1's prediction: 1.2, 1.3, 1.4, which has MAE 0.3; after averaging, your MAE is unchanged!!\nModel no.2's prediction: 1.05+1.08+0.9, which has MAE ~0.08; after averaging, your MAE becomes 0.01!!!\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so…", "597830": "> Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds\n\nThat sentence blew my mind", "597839": "I tried to understand that sentence but it is too deep for me:(\n\n", "597844": "\"Averaging couplings of same atoms in similar atomic environment (measured by similarity of atomic Coulomb matrices) may improve LB score.\" is the same as you said, right?\nDo you think, there is enough of enough similar atoms in any molecule that this would help?", "597853": "Well. With little domain knowledge at hand, I tried to view this problem as a mathematician. The package I used is called [qml](http://www.qmlcode.org/qml.html#module-qml.representations).\n\nStep 1: atom-centered Coulomb matrices are generated for two atoms.\nStep 2: similarity matrices are built respectively (I simply use exp(-C)).\nStep 3: [graph Laplacian matrices](https://en.wikipedia.org/wiki/Laplacian_matrix), which are semi-definitive (all eigenvalues are greater or equal to 0), are built upon the similarity matrices.\nStep 4: find the Fiedler eigenvectors (the eigenvec corresponding to the smallest non-zero eigenvalue).\nStep 5: Compute their Tanimoto coefficient.\n\nThe Fiedler eigenvector is very useful in partitioning graphs.", "597854": "Maybe, I haven't tried that. The post-processing is kinda against my philosophy about machine learning. For example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.", "597911": "Yes, consider molecule_000001: all 1JHC couplings are equal. Granted, i think LGBM can handle this on its own, since if your features are rotationally invariant then two edges should look equivalent", "597957": "What a cool idea for postprocessing! Though I did not observe CV increase by taking average of similar (+-0.5) predictions within types. \n\nTalking about topological based methods, may I ask how good your GCN could achieve?", "597971": "charming!!", "597981": "I don't have a GCN model, my main graph-based model is using @fnands 's MPNN's template and has 1 fold CV about -1.4.", "598049": "scaomath Teacher, I have a question. How can you calculate 1 fold CV? \nI suppose in 1fold train you use all training data for the estimation of model parameters.\nNo validation data. I must have a wrong understanding of CV... correct me please.", "598050": "Your idea is interesting, but complex molecules might have similar coupling constants which come from different structure. I suppose making the groups for averaging is not easy and risky.", "598065": "Pardon my inaccurate expression. What I meant to express was that \"I only computed the CV score for 1 CV set\". I did a 5-fold split, yet I am focusing on 1 fold to best the model, then train other 4.", "598068": "Also to answer @billbafare 's question: mathematically, this is easy to explain. Suppose for three 3JHH in one molecule, the ground truth is 1, and you have two models as follows:\n\n* Model no.1's prediction: 1.2, 1.3, 1.4, which has MAE 0.3; after averaging, your MAE is *unchanged*!! \n* Model no.2's prediction: 1.05+1.08+0.9, which has MAE ~0.08; after averaging, your MAE becomes 0.01!!!\n\nThat's why I added a caveat in the post. Yet this little trick only applies limited molecules for some coupling types, and by the example above we can see that this trick won't change much for us whose CV > -2.5 or so. On the other hand, people with score <-2.5 hardly relies on tree-based methods, so...", "598079": "scaomath I understand. Thank you !", "598147": "Shuhan Cao, this is is very interesting. I wonder why you disclosed your best feature given very few thought of it probably?", "598227": "Thank you!", "598455": "Fiedler eigenvectors' Tanimoto similarity using the graph Laplacian built by the atom-centered Coulomb matrices of the two atoms in the bonds, which vaguely measures how similar the local topologies are for these two atoms. - I was reading this line 10 times to exactly understand what it means. Meanwhile, great idea. Looking forward to more such insights. I learnt something new today. Thanks for sharing it.", "598523": "scaomath : May be a very simple question. Did you import QML package in the kernel?. I was trying to import QML package and figured out that it is not part of the predefined package in the docker. Is there any workaround to add it to our session?.", "598552": "> For example, using a ruler to measure the length of a diagram, and then multiply the submission with a magical scaling factor.\n\nI wish that was forbidden in LanL ;)", "598603": "Have you tried? \n`!pip install qml`", "598640": ";) Because it is not my best feature. My best LGB feature is still distance-based (variants of Andrew's initial starter code). Yet my best group feature is based on a commonly used trick in image classification applied on Local Many-Body Tensor Representation. Now that I found that GNN is vastly superior to LGB in this case, it doesn't hurt to share a little thoughts on feature engineering.", "598847": "(y)", "598953": "hi everyone", "599198": "Awesome!", "599322": "Awesome!", "599478": "Cool.", "603737": "Yep. My bad, I thought it was a kernels only competition. Now, I am able to install it in my system as well as in the kernel" }, "source": "meta" }