![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2862251%2Fbfb9329eff94b50b6d393bed498f6188%2FMolTransformer.png?generation=1567079164494797&alt=media\"){kind=spacer}
First off, I would like to thank Kaggle and the hosts for a well organised competition and giving us such an interesting scientific problem to work on. I learned a great deal about training and developing neural nets (and a little bit about chemistry ;)). I hope that the final solutions are useful to the organizers as well.
\n\nThat being said, I’m excited to share my final model. In short: I used a custom GNN architecture that combines the message passing elements from the MPNN model (https://arxiv.org/pdf/1704.01212.pdf) with the heavy use of multi-head attention layers as in the Transformer Encoder (https://arxiv.org/pdf/1706.03762.pdf). On its own an 8-fold submission of this model scored -3.039 on private LB (single folds scored between -2.83 and -2.88 on their validation sets).
\n\nJourney to the final model:\nQuite quickly upon entering this competition I found several discussion topics on Graph CNNs, which led me to the MPNN paper. I started implementing this model from scratch in Pytorch and given initial promising results and later also finding Cheng’s MPNN starter kit, I was convinced I was on the right track.
\n\nAs many others have noted, the MPNN model doesn’t necessarily work well as a plug and go solution for this problem. But with some customisation, I was able to get an 8-fold MPNN model to produce a -2.873 score on private LB.
\n\nThe most important things I changed or added to the MPNN architecture to get there:\n• Add separate message passing function for scalar-coupling atom pairs. \n• Incorporate additional edge, atom and molecule level features. Most significantly, angle based features for scalar coupling edges (i.e. dihedrals for 3J, cosine angles for 2J and nearest neighbour cosine angles for all types).\n• For the Message Passing Function applied to bond connections, I added an attention mechanism over incoming messages based on cosine angles. \n• Replace set2set with a Gaussian attention layer based on Euclidean distance. \n• Introduced scalar coupling edge state that is updated alongside atom states.\n• The head of the model used dense skip connection to the final node states of the update process and raw features.\n• The write head also included a residual block that was specialised per scalar coupling type.\n• Before the final prediction I first predicted each of the four scalar coupling contributions separately. These four terms plus a learned residual are finally added to predict the scalar coupling constant. The four contribution predictions where added to the loss function.\n• Increased hidden state dimension (up to 300).
\n\nAs the long list suggests, the model grew increasingly complex and the additions started to feel increasingly hacky. Also, I wasn’t happy with how the model scaled with hidden state dimension. Anything above 300 no longer fit on the Kaggle kernel GPU memory. This mostly stems from the edge network whose final layer has a weight parameter whose size grows at a cubic rate (i.e. 300^3 = 27M params). I wasn’t able to make any compromise on the flexibility of this edge network without drastically reducing performance.
\n\nFinal Model:\nWith this in mind I started thinking of a different architecture where message passing layers and attention layers (which seemed to be the most promising elements from my MPNN) could be neatly stacked. The general architecture of the Transformer Encoder provided the basic building blocks for this: stacked encoder blocks and sublayers connected through residual connections with layer norm. In this case however the encoder blocks are build up of two message passing layers, followed by three different types of multi-head attention layers with a final point-wise feed-forward network.
\n\n
The message passing layers (one for bond connections and one for virtual scalar coupling edges) are largely as described above for the MPNN. Note that unlike the attention layers, the message passing layers' parameters are tied across encoder blocks. To allow for much larger node dimension I changed the full matrix multiplication for message passing into a convolution with a fixed kernel size of 128. In the MPNN model this significantly decreased performance but for the Transformer model the gain from increased hidden state size easily offset the loss due to less flexible message passing.
\n\nThe three multi-head attention layers are:\n1. Euclidean distance based Gaussian attention. Similar to the Gaussian attention module in my MPNN.\n2. Graph distance based attention. Computes attention through an embedding of the graph distance matrix.\n3. Scaled dot product self-attention. Exactly the same as in the Transformer Encoder.
\n\nAlthough the final layers in the block resemble the encoder blocks of the Transformer model, there are several additional layers designed specifically to capture the structure and relationships among atoms in a molecule. The final write head is the same as described for the MPNN.
\n\nTraining:\nTraining was facilitated by the fastai library. The final model was trained for 100 epochs on all folds using the one_cycle learning rate policy, a max learning rate of 5e-4 and weight decay of 1e-2. I also experimented with snapshot ensembling over an additional 40 epochs using a warm restart learning rate schedule, but the gains from this technique are marginal (~ -0.01). Up to the final submission I had done nearly all training and experimenting on Kaggle kernels. But, for the final submission I used a model with a hidden node dimension of 650, 10 encoder blocks and 10 attention heads. This model was pretty big (at least by my standards) and implied I had to do the final training on the GCP platform, using distributed training on 2 V100 GPUs. Total training time on this setup was roughly 1.5 days per fold. My top submission also included the 8-fold submission of the MPNN model. This gave me about -0.05 of a boost.
\n\nI want to specially thank @hengck23 for sharing the MPNN starter kit, which really helped me along with how to setup training and load in data for these GNN models, and the people who submitted public kernels. Nearly all my features either came from or where heavily inspired by these kernels.
\n\nI’ll try to get a kernel published in the next few days that trains a smaller version of the transformer model.
\n\nThanks for reading and I’ll happily answer any questions
\n\nUpdate: \n- github repo: https://github.com/robinniesert/kaggle-champs\n- kaggle kernel of smaller model w 15 epochs of training: https://www.kaggle.com/robin111/molecule-transformer-w-message-passing
", "messages": [ { "id": 611706, "postDate": "2019-08-29T11:50:55.577Z", "content": "First off, I would like to thank Kaggle and the hosts for a well organised competition and giving us such an interesting scientific problem to work on. I learned a great deal about training and developing neural nets (and a little bit about chemistry ;)). I hope that the final solutions are useful to the organizers as well.
\n\nThat being said, I’m excited to share my final model. In short: I used a custom GNN architecture that combines the message passing elements from the MPNN model (https://arxiv.org/pdf/1704.01212.pdf) with the heavy use of multi-head attention layers as in the Transformer Encoder (https://arxiv.org/pdf/1706.03762.pdf). On its own an 8-fold submission of this model scored -3.039 on private LB (single folds scored between -2.83 and -2.88 on their validation sets).
\n\nJourney to the final model:\nQuite quickly upon entering this competition I found several discussion topics on Graph CNNs, which led me to the MPNN paper. I started implementing this model from scratch in Pytorch and given initial promising results and later also finding Cheng’s MPNN starter kit, I was convinced I was on the right track.
\n\nAs many others have noted, the MPNN model doesn’t necessarily work well as a plug and go solution for this problem. But with some customisation, I was able to get an 8-fold MPNN model to produce a -2.873 score on private LB.
\n\nThe most important things I changed or added to the MPNN architecture to get there:\n• Add separate message passing function for scalar-coupling atom pairs. \n• Incorporate additional edge, atom and molecule level features. Most significantly, angle based features for scalar coupling edges (i.e. dihedrals for 3J, cosine angles for 2J and nearest neighbour cosine angles for all types).\n• For the Message Passing Function applied to bond connections, I added an attention mechanism over incoming messages based on cosine angles. \n• Replace set2set with a Gaussian attention layer based on Euclidean distance. \n• Introduced scalar coupling edge state that is updated alongside atom states.\n• The head of the model used dense skip connection to the final node states of the update process and raw features.\n• The write head also included a residual block that was specialised per scalar coupling type.\n• Before the final prediction I first predicted each of the four scalar coupling contributions separately. These four terms plus a learned residual are finally added to predict the scalar coupling constant. The four contribution predictions where added to the loss function.\n• Increased hidden state dimension (up to 300).
\n\nAs the long list suggests, the model grew increasingly complex and the additions started to feel increasingly hacky. Also, I wasn’t happy with how the model scaled with hidden state dimension. Anything above 300 no longer fit on the Kaggle kernel GPU memory. This mostly stems from the edge network whose final layer has a weight parameter whose size grows at a cubic rate (i.e. 300^3 = 27M params). I wasn’t able to make any compromise on the flexibility of this edge network without drastically reducing performance.
\n\nFinal Model:\nWith this in mind I started thinking of a different architecture where message passing layers and attention layers (which seemed to be the most promising elements from my MPNN) could be neatly stacked. The general architecture of the Transformer Encoder provided the basic building blocks for this: stacked encoder blocks and sublayers connected through residual connections with layer norm. In this case however the encoder blocks are build up of two message passing layers, followed by three different types of multi-head attention layers with a final point-wise feed-forward network.
\n\n
The message passing layers (one for bond connections and one for virtual scalar coupling edges) are largely as described above for the MPNN. Note that unlike the attention layers, the message passing layers' parameters are tied across encoder blocks. To allow for much larger node dimension I changed the full matrix multiplication for message passing into a convolution with a fixed kernel size of 128. In the MPNN model this significantly decreased performance but for the Transformer model the gain from increased hidden state size easily offset the loss due to less flexible message passing.
\n\nThe three multi-head attention layers are:\n1. Euclidean distance based Gaussian attention. Similar to the Gaussian attention module in my MPNN.\n2. Graph distance based attention. Computes attention through an embedding of the graph distance matrix.\n3. Scaled dot product self-attention. Exactly the same as in the Transformer Encoder.
\n\nAlthough the final layers in the block resemble the encoder blocks of the Transformer model, there are several additional layers designed specifically to capture the structure and relationships among atoms in a molecule. The final write head is the same as described for the MPNN.
\n\nTraining:\nTraining was facilitated by the fastai library. The final model was trained for 100 epochs on all folds using the one_cycle learning rate policy, a max learning rate of 5e-4 and weight decay of 1e-2. I also experimented with snapshot ensembling over an additional 40 epochs using a warm restart learning rate schedule, but the gains from this technique are marginal (~ -0.01). Up to the final submission I had done nearly all training and experimenting on Kaggle kernels. But, for the final submission I used a model with a hidden node dimension of 650, 10 encoder blocks and 10 attention heads. This model was pretty big (at least by my standards) and implied I had to do the final training on the GCP platform, using distributed training on 2 V100 GPUs. Total training time on this setup was roughly 1.5 days per fold. My top submission also included the 8-fold submission of the MPNN model. This gave me about -0.05 of a boost.
\n\nI want to specially thank @hengck23 for sharing the MPNN starter kit, which really helped me along with how to setup training and load in data for these GNN models, and the people who submitted public kernels. Nearly all my features either came from or where heavily inspired by these kernels.
\n\nI’ll try to get a kernel published in the next few days that trains a smaller version of the transformer model.
\n\nThanks for reading and I’ll happily answer any questions
\n\nUpdate: \n- github repo: https://github.com/robinniesert/kaggle-champs\n- kaggle kernel of smaller model w 15 epochs of training: https://www.kaggle.com/robin111/molecule-transformer-w-message-passing
", "rawMarkdown": "First off, I would like to thank Kaggle and the hosts for a well organised competition and giving us such an interesting scientific problem to work on. I learned a great deal about training and developing neural nets (and a little bit about chemistry ;)). I hope that the final solutions are useful to the organizers as well.\n\nThat being said, I’m excited to share my final model. In short: I used a custom GNN architecture that combines the message passing elements from the MPNN model ([https://arxiv.org/pdf/1704.01212.pdf](https://arxiv.org/pdf/1704.01212.pdf)) with the heavy use of multi-head attention layers as in the Transformer Encoder ([https://arxiv.org/pdf/1706.03762.pdf](https://arxiv.org/pdf/1706.03762.pdf)). On its own an 8-fold submission of this model scored -3.039 on private LB (single folds scored between -2.83 and -2.88 on their validation sets).\n\n*Journey to the final model:*\nQuite quickly upon entering this competition I found several discussion topics on Graph CNNs, which led me to the MPNN paper. I started implementing this model from scratch in Pytorch and given initial promising results and later also finding Cheng’s MPNN starter kit, I was convinced I was on the right track.\n\nAs many others have noted, the MPNN model doesn’t necessarily work well as a plug and go solution for this problem. But with some customisation, I was able to get an 8-fold MPNN model to produce a -2.873 score on private LB.\n\nThe most important things I changed or added to the MPNN architecture to get there:\n• Add separate message passing function for scalar-coupling atom pairs. \n• Incorporate additional edge, atom and molecule level features. Most significantly, angle based features for scalar coupling edges (i.e. dihedrals for 3J, cosine angles for 2J and nearest neighbour cosine angles for all types).\n• For the Message Passing Function applied to bond connections, I added an attention mechanism over incoming messages based on cosine angles. \n• Replace set2set with a Gaussian attention layer based on Euclidean distance. \n• Introduced scalar coupling edge state that is updated alongside atom states.\n• The head of the model used dense skip connection to the final node states of the update process and raw features.\n• The write head also included a residual block that was specialised per scalar coupling type.\n• Before the final prediction I first predicted each of the four scalar coupling contributions separately. These four terms plus a learned residual are finally added to predict the scalar coupling constant. The four contribution predictions where added to the loss function.\n• Increased hidden state dimension (up to 300).\n\nAs the long list suggests, the model grew increasingly complex and the additions started to feel increasingly hacky. Also, I wasn’t happy with how the model scaled with hidden state dimension. Anything above 300 no longer fit on the Kaggle kernel GPU memory. This mostly stems from the edge network whose final layer has a weight parameter whose size grows at a cubic rate (i.e. 300^3 = 27M params). I wasn’t able to make any compromise on the flexibility of this edge network without drastically reducing performance.\n\n*Final Model:*\nWith this in mind I started thinking of a different architecture where message passing layers and attention layers (which seemed to be the most promising elements from my MPNN) could be neatly stacked. The general architecture of the Transformer Encoder provided the basic building blocks for this: stacked encoder blocks and sublayers connected through residual connections with layer norm. In this case however the encoder blocks are build up of two message passing layers, followed by three different types of multi-head attention layers with a final point-wise feed-forward network.\n\n\n\nThe message passing layers (one for bond connections and one for virtual scalar coupling edges) are largely as described above for the MPNN. Note that unlike the attention layers, the message passing layers' parameters are tied across encoder blocks. To allow for much larger node dimension I changed the full matrix multiplication for message passing into a convolution with a fixed kernel size of 128. In the MPNN model this significantly decreased performance but for the Transformer model the gain from increased hidden state size easily offset the loss due to less flexible message passing.\n\nThe three multi-head attention layers are:\n1. Euclidean distance based Gaussian attention. Similar to the Gaussian attention module in my MPNN.\n2. Graph distance based attention. Computes attention through an embedding of the graph distance matrix.\n3. Scaled dot product self-attention. Exactly the same as in the Transformer Encoder.\n\nAlthough the final layers in the block resemble the encoder blocks of the Transformer model, there are several additional layers designed specifically to capture the structure and relationships among atoms in a molecule. The final write head is the same as described for the MPNN.\n\n*Training:*\nTraining was facilitated by the fastai library. The final model was trained for 100 epochs on all folds using the one_cycle learning rate policy, a max learning rate of 5e-4 and weight decay of 1e-2. I also experimented with snapshot ensembling over an additional 40 epochs using a warm restart learning rate schedule, but the gains from this technique are marginal (~ -0.01). Up to the final submission I had done nearly all training and experimenting on Kaggle kernels. But, for the final submission I used a model with a hidden node dimension of 650, 10 encoder blocks and 10 attention heads. This model was pretty big (at least by my standards) and implied I had to do the final training on the GCP platform, using distributed training on 2 V100 GPUs. Total training time on this setup was roughly 1.5 days per fold. My top submission also included the 8-fold submission of the MPNN model. This gave me about -0.05 of a boost.\n\nI want to specially thank @hengck23 for sharing the MPNN starter kit, which really helped me along with how to setup training and load in data for these GNN models, and the people who submitted public kernels. Nearly all my features either came from or where heavily inspired by these kernels.\n\nI’ll try to get a kernel published in the next few days that trains a smaller version of the transformer model.\n\nThanks for reading and I’ll happily answer any questions\n\n*Update:* \n- github repo: [https://github.com/robinniesert/kaggle-champs](https://github.com/robinniesert/kaggle-champs)\n- kaggle kernel of smaller model w 15 epochs of training: [https://www.kaggle.com/robin111/molecule-transformer-w-message-passing](https://www.kaggle.com/robin111/molecule-transformer-w-message-passing)", "votes": 108 }, { "id": 611806, "postDate": "2019-08-29T12:55:08.523Z", "content": "Really impressive. Good job. I hope this will not be your last challenge and I am eager to see more of you in the future.
", "rawMarkdown": "Really impressive. Good job. I hope this will not be your last challenge and I am eager to see more of you in the future.", "votes": 5 }, { "id": 612095, "postDate": "2019-08-29T15:49:18.317Z", "content": "Thanks for sharing, and congrats for the result. We were hoping to pass you but couldn't!
\n\nI'll need to reread your write up more than one to fully grasp it.
", "rawMarkdown": "Thanks for sharing, and congrats for the result. We were hoping to pass you but couldn't!\n\nI'll need to reread your write up more than one to fully grasp it.", "votes": 3 }, { "id": 611859, "postDate": "2019-08-29T13:19:51.753Z", "content": "Thanks for sharing. This is impressive. Could you elaborate more about these features that you used:\n1. scalar coupling edge state : do you mean type of bond?\n2. Graph distance: (how many step go from A-> B)? \n3. scalar coup pair feature: could you tell me precisely what they are? \nThanks.
", "rawMarkdown": "Thanks for sharing. This is impressive. Could you elaborate more about these features that you used:\n1. scalar coupling edge state : do you mean type of bond?\n2. Graph distance: (how many step go from A-> B)? \n3. scalar coup pair feature: could you tell me precisely what they are? \nThanks. ", "votes": 3, "replies": [ { "id": 611906, "postDate": "2019-08-29T13:52:34.637Z", "content": "Thanks! \n1. Similarly as for the hidden states for atoms I introduced a hidden state for the edges between atoms for which we had to predict the scalar coupling constant. In the MPNN model the hidden states for atoms are iteratively updated by means of an update function, and I used a similar mechanism to iteratively update these hidden scalar coupling edge states.\n2. I think you're understanding of graph distance is correct. It represents the number of atoms on the shortest path from atom_A -> atom_B. I computed it using the RDkit GetDistanceMatrix function (documentation here http://rdkit.org/docs/source/rdkit.Chem.rdmolops.html.\n3. These are all the scalar coupling edge features I used: 'sc_type', 'eucl_dist', 'eucl_dist_min_atom_radius', 'dist_electroneg_adj', 'normed_dist', 'diangle', 'cos_angle', 'cos_angle_nn0', 'cos_angle_nn1'. The 'sc_type' is one hot encoded. The distance features 'eucl_dist_min_atom_radius', 'dist_electroneg_adj' were taken from Giba's kernel.
", "rawMarkdown": "Thanks! \n1. Similarly as for the hidden states for atoms I introduced a hidden state for the edges between atoms for which we had to predict the scalar coupling constant. In the MPNN model the hidden states for atoms are iteratively updated by means of an update function, and I used a similar mechanism to iteratively update these hidden scalar coupling edge states.\n2. I think you're understanding of graph distance is correct. It represents the number of atoms on the shortest path from atom_A -> atom_B. I computed it using the RDkit GetDistanceMatrix function (documentation here [http://rdkit.org/docs/source/rdkit.Chem.rdmolops.html](http://rdkit.org/docs/source/rdkit.Chem.rdmolops.html).\n3. These are all the scalar coupling edge features I used: 'sc_type', 'eucl_dist', 'eucl_dist_min_atom_radius', 'dist_electroneg_adj', 'normed_dist', 'diangle', 'cos_angle', 'cos_angle_nn0', 'cos_angle_nn1'. The 'sc_type' is one hot encoded. The distance features 'eucl_dist_min_atom_radius', 'dist_electroneg_adj' were taken from Giba's kernel.", "votes": 3 } ] }, { "id": 614111, "postDate": "2019-08-31T04:54:33.110Z", "content": "@robin111 Thanks for sharing
", "rawMarkdown": "[@robin111](https://www.kaggle.com/robin111) Thanks for sharing", "votes": 1 }, { "id": 612846, "postDate": "2019-08-30T03:28:23.443Z", "content": "Congratulations! Amazing solutions! :-)
", "rawMarkdown": "Congratulations! Amazing solutions! :-)", "votes": 1 }, { "id": 612478, "postDate": "2019-08-29T20:09:38.697Z", "content": "Awesome solution and extra merit for doing this SOLO. Congratulations.
", "rawMarkdown": "Awesome solution and extra merit for doing this SOLO. Congratulations.", "votes": 1 }, { "id": 612341, "postDate": "2019-08-29T18:22:00.073Z", "content": "Hi @robin111 , superb solution. Lots of learning from here. Many congratulations.
", "rawMarkdown": "Hi @robin111 , superb solution. Lots of learning from here. Many congratulations.", "votes": 1 }, { "id": 611938, "postDate": "2019-08-29T14:09:56.110Z", "content": "Are you planning to open source your code on GitHub? That'd be great. Thanks for this post, really interesting.
", "rawMarkdown": "Are you planning to open source your code on GitHub? That'd be great. Thanks for this post, really interesting.", "votes": 1, "replies": [ { "id": 611949, "postDate": "2019-08-29T14:18:36.073Z", "content": "Thanks Michel. Yes, I plan on cleaning up my code a bit and making the repository public over the weekend.
", "rawMarkdown": "Thanks Michel. Yes, I plan on cleaning up my code a bit and making the repository public over the weekend.", "votes": 2 }, { "id": 612255, "postDate": "2019-08-29T17:20:53.600Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 611802, "postDate": "2019-08-29T12:53:29.057Z", "content": "Using attention is a cool idea! We didn't do this.
\n\nCongratulations and thanks for sharing!
", "rawMarkdown": "Using attention is a cool idea! We didn't do this.\n\nCongratulations and thanks for sharing!", "votes": 1 }, { "id": 611756, "postDate": "2019-08-29T12:17:40.677Z", "content": "Very impressive architecture, congrats! I see that you have started Kaggle recently. Was it your first time working on deep learning? If so, it is even more impressive:)
", "rawMarkdown": "Very impressive architecture, congrats! I see that you have started Kaggle recently. Was it your first time working on deep learning? If so, it is even more impressive:)", "votes": 1, "replies": [ { "id": 611771, "postDate": "2019-08-29T12:27:55.670Z", "content": "Thanks Ahmet! I had some prior experience with smaller neural nets (mostly tabular models), but nothing with models of this sort of size.
", "rawMarkdown": "Thanks Ahmet! I had some prior experience with smaller neural nets (mostly tabular models), but nothing with models of this sort of size.", "votes": 1 } ] }, { "id": 615592, "postDate": "2019-09-02T06:19:11.440Z", "content": "I’m reading your awesome kernel. \nCould you teach me where angle_in and angle_out come from ? \nI’m going to learn from you ;)
", "rawMarkdown": "I’m reading your awesome kernel. \nCould you teach me where angle_in and angle_out come from ? \nI’m going to learn from you ;)", "votes": 2, "replies": [ { "id": 615689, "postDate": "2019-09-02T08:57:23.957Z", "content": "Hi XY, glad to see the kernel is of use to you. angle_in_df and angle_out_df together contain all the angles between the chemical bonds in the molecule and the index of the bond the angle corresponds to. Take for example a molecule with the following bonds: (a0, a1), (a0, a2), (a0, a3), (a1, a4) then angle_in_df would contain the following angles and angle indices: (angle_201, 0), (angle_201, 1), (angle_301, 2), (angle_014, 3) and angle_out_df would only have (angle_014, 0). So angle out refers to the fact that it contains angles centred around the second atom in a bond pair. Check the github repo for the code used to create these inputs.
Thank you for detailed explanations. \nSorry, I must have checked your repo beforehand.\nI’m very happy with your excellent works.
", "rawMarkdown": "Thank you for detailed explanations. \nSorry, I must have checked your repo beforehand.\nI’m very happy with your excellent works. ", "votes": 1 } ] }, { "id": 615493, "postDate": "2019-09-02T02:45:02.223Z", "content": "Awesome write-up Robin. Congrats on finishing 6th in this competition.
", "rawMarkdown": "Awesome write-up Robin. Congrats on finishing 6th in this competition. ", "votes": 2 }, { "id": 612207, "postDate": "2019-08-29T16:53:07.137Z", "content": "Congratulations for your excellent work. Thanks for sharing your smart ideas in such a detailed level.
", "rawMarkdown": "Congratulations for your excellent work. Thanks for sharing your smart ideas in such a detailed level. ", "votes": 2 }, { "id": 611951, "postDate": "2019-08-29T14:19:43.227Z", "content": "2 months waiting for this kind of things... No words. Thank you for sharing your knowledge @robin111
", "rawMarkdown": "2 months waiting for this kind of things... No words. Thank you for sharing your knowledge @robin111 ", "votes": 2 }, { "id": 611927, "postDate": "2019-08-29T14:04:22.477Z", "content": "Congratulations on your finish and excellent explanation. Your approach is very unique and well planned. The introduction of features at various levels is a nice approach.
", "rawMarkdown": "Congratulations on your finish and excellent explanation. Your approach is very unique and well planned. The introduction of features at various levels is a nice approach.", "votes": 2 }, { "id": 2266478, "postDate": "2023-05-20T05:41:55.533Z", "content": "nice solution
", "rawMarkdown": "nice solution " }, { "id": 1549468, "postDate": "2021-10-19T00:57:52.717Z", "content": "Dear sir, where can I find Cheng’s MPNN starter kit? Thank you!
", "rawMarkdown": "Dear sir, where can I find Cheng’s MPNN starter kit? Thank you!" }, { "id": 612595, "postDate": "2019-08-29T22:14:03.290Z", "content": "Awesome. Thanks for sharing. Any estimate of how much money did you spent on this?
", "rawMarkdown": "Awesome. Thanks for sharing. Any estimate of how much money did you spent on this? " }, { "id": 612388, "postDate": "2019-08-29T18:56:59.550Z", "content": "Congrats for such an achievement. One question, for multi-head attention layers, did you use the edge representation from message passing, meaning each edge (coupling edge states) attend to each edge in the same molecule?
", "rawMarkdown": "Congrats for such an achievement. One question, for multi-head attention layers, did you use the edge representation from message passing, meaning each edge (coupling edge states) attend to each edge in the same molecule?", "replies": [ { "id": 612472, "postDate": "2019-08-29T20:07:40.773Z", "content": "Thank you Zhenlan. I only used a hidden coupling edge state in the MPNN model. So I have only looked at applying multi-head attention layers on atom/node states.
", "rawMarkdown": "Thank you Zhenlan. I only used a hidden coupling edge state in the MPNN model. So I have only looked at applying multi-head attention layers on atom/node states. " } ] }, { "id": 616839, "postDate": "2019-09-03T14:20:34.297Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 616458, "postDate": "2019-09-03T06:34:10.373Z", "content": "Congrats and thanks a lot for sharing.
", "rawMarkdown": "Congrats and thanks a lot for sharing.", "votes": 1 }, { "id": 1653928, "postDate": "2022-01-18T01:49:03.887Z", "content": "Thank you for sharing.
", "rawMarkdown": "Thank you for sharing." } ], "comments": [ { "id": 611806, "author_name": "Dieter", "author_url": "", "post_date": "2019-08-29T12:55:08.523000", "content": "Really impressive. Good job. I hope this will not be your last challenge and I am eager to see more of you in the future.
", "votes": 5, "replies": [] }, { "id": 612095, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-29T15:49:18.317000", "content": "Thanks for sharing, and congrats for the result. We were hoping to pass you but couldn't!
\n\nI'll need to reread your write up more than one to fully grasp it.
", "votes": 3, "replies": [] }, { "id": 611859, "author_name": "vithng", "author_url": "", "post_date": "2019-08-29T13:19:51.753000", "content": "Thanks for sharing. This is impressive. Could you elaborate more about these features that you used:\n1. scalar coupling edge state : do you mean type of bond?\n2. Graph distance: (how many step go from A-> B)? \n3. scalar coup pair feature: could you tell me precisely what they are? \nThanks.
", "votes": 3, "replies": [ { "id": 611906, "author_name": "Robin N", "author_url": "", "post_date": "2019-08-29T13:52:34.637000", "content": "Thanks! \n1. Similarly as for the hidden states for atoms I introduced a hidden state for the edges between atoms for which we had to predict the scalar coupling constant. In the MPNN model the hidden states for atoms are iteratively updated by means of an update function, and I used a similar mechanism to iteratively update these hidden scalar coupling edge states.\n2. I think you're understanding of graph distance is correct. It represents the number of atoms on the shortest path from atom_A -> atom_B. I computed it using the RDkit GetDistanceMatrix function (documentation here http://rdkit.org/docs/source/rdkit.Chem.rdmolops.html.\n3. These are all the scalar coupling edge features I used: 'sc_type', 'eucl_dist', 'eucl_dist_min_atom_radius', 'dist_electroneg_adj', 'normed_dist', 'diangle', 'cos_angle', 'cos_angle_nn0', 'cos_angle_nn1'. The 'sc_type' is one hot encoded. The distance features 'eucl_dist_min_atom_radius', 'dist_electroneg_adj' were taken from Giba's kernel.
", "votes": 3, "replies": [] } ] }, { "id": 614111, "author_name": "Pilipeyko Grigory", "author_url": "", "post_date": "2019-08-31T04:54:33.110000", "content": "@robin111 Thanks for sharing
", "votes": 1, "replies": [] }, { "id": 612846, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-30T03:28:23.443000", "content": "Congratulations! Amazing solutions! :-)
", "votes": 1, "replies": [] }, { "id": 612478, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-08-29T20:09:38.697000", "content": "Awesome solution and extra merit for doing this SOLO. Congratulations.
", "votes": 1, "replies": [] }, { "id": 612341, "author_name": "pankaj_j", "author_url": "", "post_date": "2019-08-29T18:22:00.073000", "content": "Hi @robin111 , superb solution. Lots of learning from here. Many congratulations.
", "votes": 1, "replies": [] }, { "id": 611938, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-29T14:09:56.110000", "content": "Are you planning to open source your code on GitHub? That'd be great. Thanks for this post, really interesting.
", "votes": 1, "replies": [ { "id": 611949, "author_name": "Robin N", "author_url": "", "post_date": "2019-08-29T14:18:36.073000", "content": "Thanks Michel. Yes, I plan on cleaning up my code a bit and making the repository public over the weekend.
", "votes": 2, "replies": [] }, { "id": 612255, "author_name": "", "author_url": "", "post_date": "2019-08-29T17:20:53.600000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 611802, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-08-29T12:53:29.057000", "content": "Using attention is a cool idea! We didn't do this.
\n\nCongratulations and thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 611756, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-08-29T12:17:40.677000", "content": "Very impressive architecture, congrats! I see that you have started Kaggle recently. Was it your first time working on deep learning? If so, it is even more impressive:)
", "votes": 1, "replies": [ { "id": 611771, "author_name": "Robin N", "author_url": "", "post_date": "2019-08-29T12:27:55.670000", "content": "Thanks Ahmet! I had some prior experience with smaller neural nets (mostly tabular models), but nothing with models of this sort of size.
", "votes": 1, "replies": [] } ] }, { "id": 615592, "author_name": "XY", "author_url": "", "post_date": "2019-09-02T06:19:11.440000", "content": "I’m reading your awesome kernel. \nCould you teach me where angle_in and angle_out come from ? \nI’m going to learn from you ;)
", "votes": 2, "replies": [ { "id": 615689, "author_name": "Robin N", "author_url": "", "post_date": "2019-09-02T08:57:23.957000", "content": "Hi XY, glad to see the kernel is of use to you. angle_in_df and angle_out_df together contain all the angles between the chemical bonds in the molecule and the index of the bond the angle corresponds to. Take for example a molecule with the following bonds: (a0, a1), (a0, a2), (a0, a3), (a1, a4) then angle_in_df would contain the following angles and angle indices: (angle_201, 0), (angle_201, 1), (angle_301, 2), (angle_014, 3) and angle_out_df would only have (angle_014, 0). So angle out refers to the fact that it contains angles centred around the second atom in a bond pair. Check the github repo for the code used to create these inputs.
Thank you for detailed explanations. \nSorry, I must have checked your repo beforehand.\nI’m very happy with your excellent works.
", "votes": 1, "replies": [] } ] }, { "id": 615493, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-09-02T02:45:02.223000", "content": "Awesome write-up Robin. Congrats on finishing 6th in this competition.
", "votes": 2, "replies": [] }, { "id": 612207, "author_name": "XiaokangWang", "author_url": "", "post_date": "2019-08-29T16:53:07.137000", "content": "Congratulations for your excellent work. Thanks for sharing your smart ideas in such a detailed level.
", "votes": 2, "replies": [] }, { "id": 611951, "author_name": "Nanashi", "author_url": "", "post_date": "2019-08-29T14:19:43.227000", "content": "2 months waiting for this kind of things... No words. Thank you for sharing your knowledge @robin111
", "votes": 2, "replies": [] }, { "id": 611927, "author_name": "Bill Holst", "author_url": "", "post_date": "2019-08-29T14:04:22.477000", "content": "Congratulations on your finish and excellent explanation. Your approach is very unique and well planned. The introduction of features at various levels is a nice approach.
", "votes": 2, "replies": [] }, { "id": 2266478, "author_name": "Mariam Akter", "author_url": "", "post_date": "2023-05-20T05:41:55.533000", "content": "nice solution
", "votes": 0, "replies": [] }, { "id": 1549468, "author_name": "gl huang", "author_url": "", "post_date": "2021-10-19T00:57:52.717000", "content": "Dear sir, where can I find Cheng’s MPNN starter kit? Thank you!
", "votes": 0, "replies": [] }, { "id": 612595, "author_name": "David Patton", "author_url": "", "post_date": "2019-08-29T22:14:03.290000", "content": "Awesome. Thanks for sharing. Any estimate of how much money did you spent on this?
", "votes": 0, "replies": [] }, { "id": 612388, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-29T18:56:59.550000", "content": "Congrats for such an achievement. One question, for multi-head attention layers, did you use the edge representation from message passing, meaning each edge (coupling edge states) attend to each edge in the same molecule?
", "votes": 0, "replies": [ { "id": 612472, "author_name": "Robin N", "author_url": "", "post_date": "2019-08-29T20:07:40.773000", "content": "Thank you Zhenlan. I only used a hidden coupling edge state in the MPNN model. So I have only looked at applying multi-head attention layers on atom/node states.
", "votes": 0, "replies": [] } ] }, { "id": 616839, "author_name": "", "author_url": "", "post_date": "2019-09-03T14:20:34.297000", "content": "", "votes": 1, "replies": [] }, { "id": 616458, "author_name": "Rashidul H", "author_url": "", "post_date": "2019-09-03T06:34:10.373000", "content": "Congrats and thanks a lot for sharing.
", "votes": 1, "replies": [] }, { "id": 1653928, "author_name": "Simon Moon", "author_url": "", "post_date": "2022-01-18T01:49:03.887000", "content": "Thank you for sharing.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "611706": "First off, I would like to thank Kaggle and the hosts for a well organised competition and giving us such an interesting scientific problem to work on. I learned a great deal about training and developing neural nets (and a little bit about chemistry ;)). I hope that the final solutions are useful to the organizers as well.\n\nThat being said, I’m excited to share my final model. In short: I used a custom GNN architecture that combines the message passing elements from the MPNN model ([https://arxiv.org/pdf/1704.01212.pdf](https://arxiv.org/pdf/1704.01212.pdf)) with the heavy use of multi-head attention layers as in the Transformer Encoder ([https://arxiv.org/pdf/1706.03762.pdf](https://arxiv.org/pdf/1706.03762.pdf)). On its own an 8-fold submission of this model scored -3.039 on private LB (single folds scored between -2.83 and -2.88 on their validation sets).\n\n*Journey to the final model:*\nQuite quickly upon entering this competition I found several discussion topics on Graph CNNs, which led me to the MPNN paper. I started implementing this model from scratch in Pytorch and given initial promising results and later also finding Cheng’s MPNN starter kit, I was convinced I was on the right track.\n\nAs many others have noted, the MPNN model doesn’t necessarily work well as a plug and go solution for this problem. But with some customisation, I was able to get an 8-fold MPNN model to produce a -2.873 score on private LB.\n\nThe most important things I changed or added to the MPNN architecture to get there:\n• Add separate message passing function for scalar-coupling atom pairs. \n• Incorporate additional edge, atom and molecule level features. Most significantly, angle based features for scalar coupling edges (i.e. dihedrals for 3J, cosine angles for 2J and nearest neighbour cosine angles for all types).\n• For the Message Passing Function applied to bond connections, I added an attention mechanism over incoming messages based on cosine angles. \n• Replace set2set with a Gaussian attention layer based on Euclidean distance. \n• Introduced scalar coupling edge state that is updated alongside atom states.\n• The head of the model used dense skip connection to the final node states of the update process and raw features.\n• The write head also included a residual block that was specialised per scalar coupling type.\n• Before the final prediction I first predicted each of the four scalar coupling contributions separately. These four terms plus a learned residual are finally added to predict the scalar coupling constant. The four contribution predictions where added to the loss function.\n• Increased hidden state dimension (up to 300).\n\nAs the long list suggests, the model grew increasingly complex and the additions started to feel increasingly hacky. Also, I wasn’t happy with how the model scaled with hidden state dimension. Anything above 300 no longer fit on the Kaggle kernel GPU memory. This mostly stems from the edge network whose final layer has a weight parameter whose size grows at a cubic rate (i.e. 300^3 = 27M params). I wasn’t able to make any compromise on the flexibility of this edge network without drastically reducing performance.\n\n*Final Model:*\nWith this in mind I started thinking of a different architecture where message passing layers and attention layers (which seemed to be the most promising elements from my MPNN) could be neatly stacked. The general architecture of the Transformer Encoder provided the basic building blocks for this: stacked encoder blocks and sublayers connected through residual connections with layer norm. In this case however the encoder blocks are build up of two message passing layers, followed by three different types of multi-head attention layers with a final point-wise feed-forward network.\n\n\n\nThe message passing layers (one for bond connections and one for virtual scalar coupling edges) are largely as described above for the MPNN. Note that unlike the attention layers, the message passing layers' parameters are tied across encoder blocks. To allow for much larger node dimension I changed the full matrix multiplication for message passing into a convolution with a fixed kernel size of 128. In the MPNN model this significantly decreased performance but for the Transformer model the gain from increased hidden state size easily offset the loss due to less flexible message passing.\n\nThe three multi-head attention layers are:\n1. Euclidean distance based Gaussian attention. Similar to the Gaussian attention module in my MPNN.\n2. Graph distance based attention. Computes attention through an embedding of the graph distance matrix.\n3. Scaled dot product self-attention. Exactly the same as in the Transformer Encoder.\n\nAlthough the final layers in the block resemble the encoder blocks of the Transformer model, there are several additional layers designed specifically to capture the structure and relationships among atoms in a molecule. The final write head is the same as described for the MPNN.\n\n*Training:*\nTraining was facilitated by the fastai library. The final model was trained for 100 epochs on all folds using the one_cycle learning rate policy, a max learning rate of 5e-4 and weight decay of 1e-2. I also experimented with snapshot ensembling over an additional 40 epochs using a warm restart learning rate schedule, but the gains from this technique are marginal (~ -0.01). Up to the final submission I had done nearly all training and experimenting on Kaggle kernels. But, for the final submission I used a model with a hidden node dimension of 650, 10 encoder blocks and 10 attention heads. This model was pretty big (at least by my standards) and implied I had to do the final training on the GCP platform, using distributed training on 2 V100 GPUs. Total training time on this setup was roughly 1.5 days per fold. My top submission also included the 8-fold submission of the MPNN model. This gave me about -0.05 of a boost.\n\nI want to specially thank @hengck23 for sharing the MPNN starter kit, which really helped me along with how to setup training and load in data for these GNN models, and the people who submitted public kernels. Nearly all my features either came from or where heavily inspired by these kernels.\n\nI’ll try to get a kernel published in the next few days that trains a smaller version of the transformer model.\n\nThanks for reading and I’ll happily answer any questions\n\n*Update:* \n- github repo: [https://github.com/robinniesert/kaggle-champs](https://github.com/robinniesert/kaggle-champs)\n- kaggle kernel of smaller model w 15 epochs of training: [https://www.kaggle.com/robin111/molecule-transformer-w-message-passing](https://www.kaggle.com/robin111/molecule-transformer-w-message-passing)", "611806": "Really impressive. Good job. I hope this will not be your last challenge and I am eager to see more of you in the future.", "612095": "Thanks for sharing, and congrats for the result. We were hoping to pass you but couldn't!\n\nI'll need to reread your write up more than one to fully grasp it.", "611859": "Thanks for sharing. This is impressive. Could you elaborate more about these features that you used:\n1. scalar coupling edge state : do you mean type of bond?\n2. Graph distance: (how many step go from A-> B)? \n3. scalar coup pair feature: could you tell me precisely what they are? \nThanks. ", "614111": "[@robin111](https://www.kaggle.com/robin111) Thanks for sharing", "612846": "Congratulations! Amazing solutions! :-)", "612478": "Awesome solution and extra merit for doing this SOLO. Congratulations.", "612341": "Hi @robin111 , superb solution. Lots of learning from here. Many congratulations.", "611938": "Are you planning to open source your code on GitHub? That'd be great. Thanks for this post, really interesting.", "611802": "Using attention is a cool idea! We didn't do this.\n\nCongratulations and thanks for sharing!", "611756": "Very impressive architecture, congrats! I see that you have started Kaggle recently. Was it your first time working on deep learning? If so, it is even more impressive:)", "615592": "I’m reading your awesome kernel. \nCould you teach me where angle_in and angle_out come from ? \nI’m going to learn from you ;)", "615493": "Awesome write-up Robin. Congrats on finishing 6th in this competition. ", "612207": "Congratulations for your excellent work. Thanks for sharing your smart ideas in such a detailed level. ", "611951": "2 months waiting for this kind of things... No words. Thank you for sharing your knowledge @robin111 ", "611927": "Congratulations on your finish and excellent explanation. Your approach is very unique and well planned. The introduction of features at various levels is a nice approach.", "2266478": "nice solution ", "1549468": "Dear sir, where can I find Cheng’s MPNN starter kit? Thank you!", "612595": "Awesome. Thanks for sharing. Any estimate of how much money did you spent on this? ", "612388": "Congrats for such an achievement. One question, for multi-head attention layers, did you use the edge representation from message passing, meaning each edge (coupling edge states) attend to each edge in the same molecule?", "616839": "", "616458": "Congrats and thanks a lot for sharing.", "1653928": "Thank you for sharing." }, "complete": true, "source": "existing_discussion", "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3235928, "solution_url": "/competitions/champs-scalar-coupling/writeups/robin-n-6th-place-solution" }