{
  "id": 106347,
  "title": "8th place solution - a densely connected GNN ",
  "url": "/competitions/champs-scalar-coupling/writeups/4-gm-and-the-brain-8th-place-solution-a-densely-co",
  "author_name": "",
  "post_date": "2019-08-29T08:39:49.147Z",
  "votes": 94,
  "comment_count": 32,
  "views": 0,
  "content": "<p>Thanks to the organizers for such an interesting competition. It was a lot of fun exploring the current state of the art of graph neural networks and trying to improve it. I also want to thank <a href=\"/artgor\">@artgor</a> , <a href=\"/tunguz\">@tunguz</a> , <a href=\"/borisdee\">@borisdee</a>  and <a href=\"/psilogram\">@psilogram</a>  for being amazing team mates. In short, our solution is a custom Graph Neural Network with SchNet interaction at its core, which alone scores ~ -2.95 on leaderboard. The model is a single model for all bonding types.</p>\n\n<p>Since my team already had good engineered features for their lgb (with lb score of -2.0) when I joined, I could fully concentrate on what I love: finding a strong deep learning based solution. </p>\n\n<p><strong>The journey was as following</strong></p>\n\n<p>I started with refactoring Hengs starter kit for MPNN which was at around (-1.4 LB) for speed optimization to have a better speed of iterating ideas. After 2 days I got it down from 15 min to 90 sec  per epoch on my GTX1080Ti , without losing any accuracy. That enabled to quickly explore different ideas to improve MPNN which I implemented then. However the leaderboard score was still worse than our lgb. </p>\n\n<p>So I continued to read a lot of papers related to GNN:</p>\n\n<ul>\n<li>papers referenced in forum, and </li>\n<li>newer papers referencing that papers</li>\n<li>papers and git repos found by stalking top LB positions</li>\n<li>papers shared by <a href=\"/artgor\">@artgor</a> who posted new ones in our slack before I have read the previous</li>\n</ul>\n\n<p>I implemented most of them in hacky way to see if they fit for this competition. Besides others I implemented the papers listed below. Some had usable git repositories for others I need to implement more or less from scratch because they used a different input format or were coded in tensorflow/keras whereas I had setup everything in pytorch. All of them are supposed to predict properties on molecular level, so I adjusted where necessary to work on atom pair level:</p>\n\n<ul>\n<li><a href=\"https://arxiv.org/abs/1704.01212\">MPNN</a></li>\n<li><a href=\"https://github.com/SeongokRyu/Graph-neural-networks/blob/master/tutorials/gat.ipynb\">GAT</a></li>\n<li><a href=\"https://github.com/txie-93/cgcnn\">Crystal GNN</a></li>\n<li><a href=\"https://arxiv.org/abs/1706.08566\">SchNet</a></li>\n<li><a href=\"https://github.com/materialsvirtuallab/megnet\">MegNet</a></li>\n<li><a href=\"https://github.com/blackmints/3DGCN\">3DGNN</a></li>\n</ul>\n\n<p>For me SchNet not only had the best code basis but also got decent results and was relatively fast, so I started building on that one. In general the model consists of 3 parts:</p>\n\n<ol>\n<li>encoder for nodes and edges</li>\n<li>consecutive blocks of interaction between encoded nodes and edges</li>\n<li>regression head </li>\n</ol>\n\n<p>As mentioned by other solutions SchNet lacks several things, but the core interaction between atoms and edges is good and fast.\nSo I spent some weeks improving that baseline step by step by adding new stuff I read in other papers  or I found useful during computer vision competitions. From every 10 ideas I implemented a maximum of one worked, and it felt like digging for treasures. At the end not much was left from the original SchNet apart from the core interaction function between nodes and edges.</p>\n\n<p><strong>The most architectural enhancements were the following</strong></p>\n\n<ul>\n<li>using additionally edge features like bondtype, angles</li>\n<li>additionally updating edges (as in MegNet)</li>\n<li><a href=\"https://arxiv.org/pdf/1608.06993.pdf\">densenet similar connections</a> between interaction blocks</li>\n<li><a href=\"https://arxiv.org/abs/1709.01507\">squeeze and excitation block</a> at the end of an Interaction</li>\n<li>encoding and using  also molecule level features as in Hengs starter</li>\n<li>using edge level hidden units for final regression</li>\n<li>using molecule level hidden units for final regression</li>\n<li>using a deep architecture (12 interaction blocks)</li>\n</ul>\n\n<p><strong>some details on training</strong></p>\n\n<ul>\n<li>preprocessing: save graph to disk (40 min) </li>\n<li>bs: 64</li>\n<li>optimizer: Adam</li>\n<li>loss: weighted lmae (weights by inverse frequency of coupling type counts)</li>\n<li>lr_schedule: Cyclic Cosine annealing with initial lr 0.0005 and cycles of 200 epochs which enables to  continue training easily by just adding a new cycle.</li>\n<li>validation: groupkfold (10fold) per molecule</li>\n<li>epochs: 1200 (and still underfitting)</li>\n</ul>\n\n<p><strong>our final submission:</strong></p>\n\n<p>We basically just bagged versions of the above model trained with different hyperparameters which took some resources. So we split training by <a href=\"/borisdee\">@borisdee</a> <a href=\"/tunguz\">@tunguz</a> and myself. Some version of MegNet scoring around -2.5 LB made it also in the final blend but with very minor contribution. <a href=\"/psilogram\">@psilogram</a> then used a simple mae optimizer to blend the different models on bonding type level. -&gt; private leaderboard score of -3.001 </p>\n\n<p><strong>things that did not work</strong>\n- additional task of autoencoding edge or node features\n- metric learning\n- and thousand of others.</p>\n\n<p>Happy to answer any questions and thanks for reading.</p>",
  "messages": [
    {
      "id": "611275",
      "postDate": "08/29/2019 07:41:20",
      "content": "<p>Thanks to the organizers for such an interesting competition. It was a lot of fun exploring the current state of the art of graph neural networks and trying to improve it. I also want to thank <a href=\"/artgor\">@artgor</a> , <a href=\"/tunguz\">@tunguz</a> , <a href=\"/borisdee\">@borisdee</a>  and <a href=\"/psilogram\">@psilogram</a>  for being amazing team mates. In short, our solution is a custom Graph Neural Network with SchNet interaction at its core, which alone scores ~ -2.95 on leaderboard. The model is a single model for all bonding types.</p>\n\n<p>Since my team already had good engineered features for their lgb (with lb score of -2.0) when I joined, I could fully concentrate on what I love: finding a strong deep learning based solution. </p>\n\n<p><strong>The journey was as following</strong></p>\n\n<p>I started with refactoring Hengs starter kit for MPNN which was at around (-1.4 LB) for speed optimization to have a better speed of iterating ideas. After 2 days I got it down from 15 min to 90 sec  per epoch on my GTX1080Ti , without losing any accuracy. That enabled to quickly explore different ideas to improve MPNN which I implemented then. However the leaderboard score was still worse than our lgb. </p>\n\n<p>So I continued to read a lot of papers related to GNN:</p>\n\n<ul>\n<li>papers referenced in forum, and </li>\n<li>newer papers referencing that papers</li>\n<li>papers and git repos found by stalking top LB positions</li>\n<li>papers shared by <a href=\"/artgor\">@artgor</a> who posted new ones in our slack before I have read the previous</li>\n</ul>\n\n<p>I implemented most of them in hacky way to see if they fit for this competition. Besides others I implemented the papers listed below. Some had usable git repositories for others I need to implement more or less from scratch because they used a different input format or were coded in tensorflow/keras whereas I had setup everything in pytorch. All of them are supposed to predict properties on molecular level, so I adjusted where necessary to work on atom pair level:</p>\n\n<ul>\n<li><a href=\"https://arxiv.org/abs/1704.01212\">MPNN</a></li>\n<li><a href=\"https://github.com/SeongokRyu/Graph-neural-networks/blob/master/tutorials/gat.ipynb\">GAT</a></li>\n<li><a href=\"https://github.com/txie-93/cgcnn\">Crystal GNN</a></li>\n<li><a href=\"https://arxiv.org/abs/1706.08566\">SchNet</a></li>\n<li><a href=\"https://github.com/materialsvirtuallab/megnet\">MegNet</a></li>\n<li><a href=\"https://github.com/blackmints/3DGCN\">3DGNN</a></li>\n</ul>\n\n<p>For me SchNet not only had the best code basis but also got decent results and was relatively fast, so I started building on that one. In general the model consists of 3 parts:</p>\n\n<ol>\n<li>encoder for nodes and edges</li>\n<li>consecutive blocks of interaction between encoded nodes and edges</li>\n<li>regression head </li>\n</ol>\n\n<p>As mentioned by other solutions SchNet lacks several things, but the core interaction between atoms and edges is good and fast.\nSo I spent some weeks improving that baseline step by step by adding new stuff I read in other papers  or I found useful during computer vision competitions. From every 10 ideas I implemented a maximum of one worked, and it felt like digging for treasures. At the end not much was left from the original SchNet apart from the core interaction function between nodes and edges.</p>\n\n<p><strong>The most architectural enhancements were the following</strong></p>\n\n<ul>\n<li>using additionally edge features like bondtype, angles</li>\n<li>additionally updating edges (as in MegNet)</li>\n<li><a href=\"https://arxiv.org/pdf/1608.06993.pdf\">densenet similar connections</a> between interaction blocks</li>\n<li><a href=\"https://arxiv.org/abs/1709.01507\">squeeze and excitation block</a> at the end of an Interaction</li>\n<li>encoding and using  also molecule level features as in Hengs starter</li>\n<li>using edge level hidden units for final regression</li>\n<li>using molecule level hidden units for final regression</li>\n<li>using a deep architecture (12 interaction blocks)</li>\n</ul>\n\n<p><strong>some details on training</strong></p>\n\n<ul>\n<li>preprocessing: save graph to disk (40 min) </li>\n<li>bs: 64</li>\n<li>optimizer: Adam</li>\n<li>loss: weighted lmae (weights by inverse frequency of coupling type counts)</li>\n<li>lr_schedule: Cyclic Cosine annealing with initial lr 0.0005 and cycles of 200 epochs which enables to  continue training easily by just adding a new cycle.</li>\n<li>validation: groupkfold (10fold) per molecule</li>\n<li>epochs: 1200 (and still underfitting)</li>\n</ul>\n\n<p><strong>our final submission:</strong></p>\n\n<p>We basically just bagged versions of the above model trained with different hyperparameters which took some resources. So we split training by <a href=\"/borisdee\">@borisdee</a> <a href=\"/tunguz\">@tunguz</a> and myself. Some version of MegNet scoring around -2.5 LB made it also in the final blend but with very minor contribution. <a href=\"/psilogram\">@psilogram</a> then used a simple mae optimizer to blend the different models on bonding type level. -&gt; private leaderboard score of -3.001 </p>\n\n<p><strong>things that did not work</strong>\n- additional task of autoencoding edge or node features\n- metric learning\n- and thousand of others.</p>\n\n<p>Happy to answer any questions and thanks for reading.</p>",
      "rawMarkdown": "Thanks to the organizers for such an interesting competition. It was a lot of fun exploring the current state of the art of graph neural networks and trying to improve it. I also want to thank @artgor , @tunguz , @borisdee  and @psilogram  for being amazing team mates. In short, our solution is a custom Graph Neural Network with SchNet interaction at its core, which alone scores ~ -2.95 on leaderboard. The model is a single model for all bonding types.\n\nSince my team already had good engineered features for their lgb (with lb score of -2.0) when I joined, I could fully concentrate on what I love: finding a strong deep learning based solution. \n\n**The journey was as following**\n\nI started with refactoring Hengs starter kit for MPNN which was at around (-1.4 LB) for speed optimization to have a better speed of iterating ideas. After 2 days I got it down from 15 min to 90 sec  per epoch on my GTX1080Ti , without losing any accuracy. That enabled to quickly explore different ideas to improve MPNN which I implemented then. However the leaderboard score was still worse than our lgb. \n\nSo I continued to read a lot of papers related to GNN:\n\n- papers referenced in forum, and \n- newer papers referencing that papers\n- papers and git repos found by stalking top LB positions\n- papers shared by @artgor who posted new ones in our slack before I have read the previous\n\nI implemented most of them in hacky way to see if they fit for this competition. Besides others I implemented the papers listed below. Some had usable git repositories for others I need to implement more or less from scratch because they used a different input format or were coded in tensorflow/keras whereas I had setup everything in pytorch. All of them are supposed to predict properties on molecular level, so I adjusted where necessary to work on atom pair level:\n\n- [MPNN](https://arxiv.org/abs/1704.01212)\n- [GAT](https://github.com/SeongokRyu/Graph-neural-networks/blob/master/tutorials/gat.ipynb)\n- [Crystal GNN](https://github.com/txie-93/cgcnn)\n- [SchNet](https://arxiv.org/abs/1706.08566)\n- [MegNet](https://github.com/materialsvirtuallab/megnet)\n- [3DGNN](https://github.com/blackmints/3DGCN)\n\n\nFor me SchNet not only had the best code basis but also got decent results and was relatively fast, so I started building on that one. In general the model consists of 3 parts:\n\n1. encoder for nodes and edges\n2. consecutive blocks of interaction between encoded nodes and edges\n3. regression head \n\nAs mentioned by other solutions SchNet lacks several things, but the core interaction between atoms and edges is good and fast.\nSo I spent some weeks improving that baseline step by step by adding new stuff I read in other papers  or I found useful during computer vision competitions. From every 10 ideas I implemented a maximum of one worked, and it felt like digging for treasures. At the end not much was left from the original SchNet apart from the core interaction function between nodes and edges.\n\n**The most architectural enhancements were the following**\n\n- using additionally edge features like bondtype, angles\n- additionally updating edges (as in MegNet)\n- [densenet similar connections](https://arxiv.org/pdf/1608.06993.pdf) between interaction blocks\n- [squeeze and excitation block](https://arxiv.org/abs/1709.01507) at the end of an Interaction\n- encoding and using  also molecule level features as in Hengs starter\n- using edge level hidden units for final regression\n- using molecule level hidden units for final regression\n- using a deep architecture (12 interaction blocks)\n\n\n**some details on training**\n\n- preprocessing: save graph to disk (40 min) \n- bs: 64\n- optimizer: Adam\n- loss: weighted lmae (weights by inverse frequency of coupling type counts)\n- lr_schedule: Cyclic Cosine annealing with initial lr 0.0005 and cycles of 200 epochs which enables to  continue training easily by just adding a new cycle.\n- validation: groupkfold (10fold) per molecule\n- epochs: 1200 (and still underfitting)\n\n\n**our final submission:**\n\nWe basically just bagged versions of the above model trained with different hyperparameters which took some resources. So we split training by @borisdee @tunguz and myself. Some version of MegNet scoring around -2.5 LB made it also in the final blend but with very minor contribution. @psilogram then used a simple mae optimizer to blend the different models on bonding type level. -&gt; private leaderboard score of -3.001 \n\n**things that did not work**\n- additional task of autoencoding edge or node features\n- metric learning\n- and thousand of others.\n\nHappy to answer any questions and thanks for reading.",
      "votes": null
    },
    {
      "id": "611296",
      "postDate": "08/29/2019 07:50:19",
      "content": "<p>Well done, congrats on the result!  Do you plan to share your code?</p>",
      "rawMarkdown": "Well done, congrats on the result!  Do you plan to share your code?",
      "votes": null
    },
    {
      "id": "611307",
      "postDate": "08/29/2019 07:54:55",
      "content": "<p>Good question. I don't see why not. </p>",
      "rawMarkdown": "Good question. I don't see why not.",
      "votes": null
    },
    {
      "id": "611314",
      "postDate": "08/29/2019 07:59:18",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing!",
      "votes": null
    },
    {
      "id": "611316",
      "postDate": "08/29/2019 07:59:57",
      "content": "<p>\"After 2 days I got it down from 15 min to 90 sec per epoch on my GTX1080Ti , without losing any accuracy. \"</p>\n\n<p>That is very impressive !!!</p>",
      "rawMarkdown": "\"After 2 days I got it down from 15 min to 90 sec per epoch on my GTX1080Ti , without losing any accuracy. \"\n\nThat is very impressive !!!",
      "votes": null
    },
    {
      "id": "611332",
      "postDate": "08/29/2019 08:04:19",
      "content": "<p>Congrats! I look forward to learning from your code. Not just the architectures but also how you optimized Heng's code to run so fast!</p>",
      "rawMarkdown": "Congrats! I look forward to learning from your code. Not just the architectures but also how you optimized Heng's code to run so fast!",
      "votes": null
    },
    {
      "id": "611339",
      "postDate": "08/29/2019 08:07:38",
      "content": "<p>Congrats for the result. 1200 EPOCHS; how much is your training time. I had implemented SchNEt with edges but due to lack of hardware resources and novice in deep learning I am able to train only ~150 epochs. Looking forward for your source code to learn more things.   </p>",
      "rawMarkdown": "Congrats for the result. 1200 EPOCHS; how much is your training time. I had implemented SchNEt with edges but due to lack of hardware resources and novice in deep learning I am able to train only ~150 epochs. Looking forward for your source code to learn more things.",
      "votes": null
    },
    {
      "id": "611342",
      "postDate": "08/29/2019 08:09:16",
      "content": "<p>Btw, my hardware is a DGX with 8 P100 on a single node, as well as 3 additional nodes with 4 V100 on each node.</p>",
      "rawMarkdown": "Btw, my hardware is a DGX with 8 P100 on a single node, as well as 3 additional nodes with 4 V100 on each node.",
      "votes": null
    },
    {
      "id": "611363",
      "postDate": "08/29/2019 08:20:20",
      "content": "<p>Amazing work and Congratulations to you and your team !</p>",
      "rawMarkdown": "Amazing work and Congratulations to you and your team !",
      "votes": null
    },
    {
      "id": "611366",
      "postDate": "08/29/2019 08:22:09",
      "content": "<p>Congrats! I look forward to learning from your code.... Thanks...!! <a href=\"/christofhenkel\">@christofhenkel</a> </p>",
      "rawMarkdown": "Congrats! I look forward to learning from your code.... Thanks...!! @christofhenkel",
      "votes": null
    },
    {
      "id": "611368",
      "postDate": "08/29/2019 08:23:09",
      "content": "<p>I'm pretty impressed and inspired by the amount of homework which has gone into this work. \n Thanks for sharing your methodology.  Like many others here, I'm pretty interested in what you did which reduced the epoch time by that great a factor. </p>",
      "rawMarkdown": "I'm pretty impressed and inspired by the amount of homework which has gone into this work. \n Thanks for sharing your methodology.  Like many others here, I'm pretty interested in what you did which reduced the epoch time by that great a factor.",
      "votes": null
    },
    {
      "id": "611387",
      "postDate": "08/29/2019 08:31:33",
      "content": "<p>I wanted to ask about it, thanks.</p>",
      "rawMarkdown": "I wanted to ask about it, thanks.",
      "votes": null
    },
    {
      "id": "611388",
      "postDate": "08/29/2019 08:32:10",
      "content": "<p>Thanks, I'll learn a lot given I tried to build a GNN from scratch, and I only scratched the surface!</p>",
      "rawMarkdown": "Thanks, I'll learn a lot given I tried to build a GNN from scratch, and I only scratched the surface!",
      "votes": null
    },
    {
      "id": "611391",
      "postDate": "08/29/2019 08:36:24",
      "content": "<p>I must add that a 6layer version of our model with about half the parameters still scores -2.91 LB and is significantly faster to train. We trained that in kaggle kernels.</p>",
      "rawMarkdown": "I must add that a 6layer version of our model with about half the parameters still scores -2.91 LB and is significantly faster to train. We trained that in kaggle kernels.",
      "votes": null
    },
    {
      "id": "611408",
      "postDate": "08/29/2019 08:44:14",
      "content": "<p>May I ask a very impolite question? Are you the person in your avatar photo ? </p>",
      "rawMarkdown": "May I ask a very impolite question? Are you the person in your avatar photo ?",
      "votes": null
    },
    {
      "id": "611423",
      "postDate": "08/29/2019 08:49:35",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "611537",
      "postDate": "08/29/2019 10:06:49",
      "content": "<p>Congrats!\nSo, you did not use QM9. Right? If it's true, really impressive.</p>",
      "rawMarkdown": "Congrats!\nSo, you did not use QM9. Right? If it's true, really impressive.",
      "votes": null
    },
    {
      "id": "611687",
      "postDate": "08/29/2019 11:40:27",
      "content": "<p>The speed is very impressive. What do you think is the reason that your network is so fast?</p>",
      "rawMarkdown": "The speed is very impressive. What do you think is the reason that your network is so fast?",
      "votes": null
    },
    {
      "id": "611694",
      "postDate": "08/29/2019 11:43:22",
      "content": "<p>Heng is back! Your code inspired a lot of people. Could you elaborate more what you did here? </p>",
      "rawMarkdown": "Heng is back! Your code inspired a lot of people. Could you elaborate more what you did here?",
      "votes": null
    },
    {
      "id": "611808",
      "postDate": "08/29/2019 12:57:28",
      "content": "<p>Amazing work <a href=\"/christofhenkel\">@christofhenkel</a> and congrats.</p>",
      "rawMarkdown": "Amazing work @christofhenkel and congrats.",
      "votes": null
    },
    {
      "id": "612287",
      "postDate": "08/29/2019 17:48:19",
      "content": "<p><a href=\"/xiaokangwang\">@xiaokangwang</a> \nA few reasons:\n- We preprocessed the graph generation instead of having them generated in the model\n- We used atom centered symmetry functions of lower quality\nThat lowered the training time a lot</p>",
      "rawMarkdown": "xiaokangwang \nA few reasons:\n- We preprocessed the graph generation instead of having them generated in the model\n- We used atom centered symmetry functions of lower quality\nThat lowered the training time a lot",
      "votes": null
    },
    {
      "id": "613604",
      "postDate": "08/30/2019 16:47:58",
      "content": "<p>Thank you <a href=\"/christofhenkel\">@christofhenkel</a> - very informative. Congrats to you and your team <a href=\"/psilogram\">@psilogram</a> <a href=\"/tunguz\">@tunguz</a> <a href=\"/artgor\">@artgor</a> and <a href=\"/borisdee\">@borisdee</a> .</p>",
      "rawMarkdown": "Thank you @christofhenkel - very informative. Congrats to you and your team @psilogram @tunguz @artgor and @borisdee .",
      "votes": null
    },
    {
      "id": "613652",
      "postDate": "08/30/2019 17:43:03",
      "content": "<p>I used a GAN to figure out what he really looks like:</p>\n\n<p><img src=\"https://pbs.twimg.com/media/EDPAiBeW4AAchK2?format=jpg\" alt=\"dieter\"></p>",
      "rawMarkdown": "I used a GAN to figure out what he really looks like:\n\n![dieter](https://pbs.twimg.com/media/EDPAiBeW4AAchK2?format=jpg)",
      "votes": null
    },
    {
      "id": "615591",
      "postDate": "09/02/2019 06:19:03",
      "content": "<p>working on some illustrations:</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2F08edcad3e8ea6a2446a8b8d0c69af278%2FNN%20champs.001.jpeg?generation=1567405111654062&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "working on some illustrations:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2F08edcad3e8ea6a2446a8b8d0c69af278%2FNN%20champs.001.jpeg?generation=1567405111654062&amp;alt=media)",
      "votes": null
    },
    {
      "id": "615604",
      "postDate": "09/02/2019 06:45:01",
      "content": "<p>Interaction Block:</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2Fe41816914a4d02650cfe75908d271b28%2FNN%20champs.003.jpeg?generation=1567406675179912&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Interaction Block:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2Fe41816914a4d02650cfe75908d271b28%2FNN%20champs.003.jpeg?generation=1567406675179912&amp;alt=media)",
      "votes": null
    },
    {
      "id": "615605",
      "postDate": "09/02/2019 06:46:31",
      "content": "<p>Who else is in love with the Arch?\nThanks Dieter!!! Looking forward to see the snippets :)</p>",
      "rawMarkdown": "Who else is in love with the Arch?\nThanks Dieter!!! Looking forward to see the snippets :)",
      "votes": null
    },
    {
      "id": "615635",
      "postDate": "09/02/2019 07:42:10",
      "content": "<p>Amazing Solution and Congrats to you and your team !</p>",
      "rawMarkdown": "Amazing Solution and Congrats to you and your team !",
      "votes": null
    },
    {
      "id": "615720",
      "postDate": "09/02/2019 09:39:28",
      "content": "<p>Thanks for these pictures.   My NN looks like a subset of yours, maybe it means I was on the right track? </p>\n\n<p>The most striking difference with what i tried is the molecule features.  How much do you gain from using them?  Which features are they?  QM9?</p>",
      "rawMarkdown": "Thanks for these pictures.   My NN looks like a subset of yours, maybe it means I was on the right track? \n\nThe most striking difference with what i tried is the molecule features.  How much do you gain from using them?  Which features are they?  QM9?",
      "votes": null
    },
    {
      "id": "615729",
      "postDate": "09/02/2019 09:54:35",
      "content": "<p>the molecule features add only small value to lmae (&lt; 0.05). The biggest contribution are </p>\n\n<ul>\n<li>updating bond hidden state (~ -0.15)</li>\n<li>add last bond hidden state to regression head (~ -0.15)</li>\n<li>have dense connections between interaction blocks (~ -0.1)</li>\n<li>squeeze and excitation for atom hidden (~ -0.05)</li>\n</ul>",
      "rawMarkdown": "the molecule features add only small value to lmae (&lt; 0.05). The biggest contribution are \n\n- updating bond hidden state (~ -0.15)\n- add last bond hidden state to regression head (~ -0.15)\n- have dense connections between interaction blocks (~ -0.1)\n- squeeze and excitation for atom hidden (~ -0.05)",
      "votes": null
    },
    {
      "id": "616462",
      "postDate": "09/03/2019 06:35:47",
      "content": "<p>Congrats and thanks a lot for sharing.</p>",
      "rawMarkdown": "Congrats and thanks a lot for sharing.",
      "votes": null
    },
    {
      "id": "617113",
      "postDate": "09/03/2019 18:50:59",
      "content": "<p>Impressive architecture!  I would like to catch up with the current state-of-the-art on graph neural networks so this post is a goldmine. Thank you!</p>",
      "rawMarkdown": "Impressive architecture!  I would like to catch up with the current state-of-the-art on graph neural networks so this post is a goldmine. Thank you!",
      "votes": null
    },
    {
      "id": "622448",
      "postDate": "09/09/2019 17:05:18",
      "content": "<p>Did you plan to release the code snippets for the arch?\nThnx.</p>",
      "rawMarkdown": "Did you plan to release the code snippets for the arch?\nThnx.",
      "votes": null
    },
    {
      "id": "629355",
      "postDate": "09/18/2019 17:08:58",
      "content": "<p>Hi Dieter, if you share the code, will you update the post?</p>",
      "rawMarkdown": "Hi Dieter, if you share the code, will you update the post?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 611296,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "08/29/2019 07:50:19",
      "content": "<p>Well done, congrats on the result!  Do you plan to share your code?</p>",
      "votes": null,
      "replies": [
        {
          "id": 611307,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "08/29/2019 07:54:55",
          "content": "<p>Good question. I don't see why not. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 611388,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/29/2019 08:32:10",
          "content": "<p>Thanks, I'll learn a lot given I tried to build a GNN from scratch, and I only scratched the surface!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 629355,
          "author_name": "cczaixian",
          "author_url": "",
          "post_date": "09/18/2019 17:08:58",
          "content": "<p>Hi Dieter, if you share the code, will you update the post?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 611314,
      "author_name": "stanislavblinov",
      "author_url": "",
      "post_date": "08/29/2019 07:59:18",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611316,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "08/29/2019 07:59:57",
      "content": "<p>\"After 2 days I got it down from 15 min to 90 sec per epoch on my GTX1080Ti , without losing any accuracy. \"</p>\n\n<p>That is very impressive !!!</p>",
      "votes": null,
      "replies": [
        {
          "id": 611694,
          "author_name": "xiaokangwang",
          "author_url": "",
          "post_date": "08/29/2019 11:43:22",
          "content": "<p>Heng is back! Your code inspired a lot of people. Could you elaborate more what you did here? </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 611332,
      "author_name": "taniaj",
      "author_url": "",
      "post_date": "08/29/2019 08:04:19",
      "content": "<p>Congrats! I look forward to learning from your code. Not just the architectures but also how you optimized Heng's code to run so fast!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611339,
      "author_name": "adityakumarsinha",
      "author_url": "",
      "post_date": "08/29/2019 08:07:38",
      "content": "<p>Congrats for the result. 1200 EPOCHS; how much is your training time. I had implemented SchNEt with edges but due to lack of hardware resources and novice in deep learning I am able to train only ~150 epochs. Looking forward for your source code to learn more things.   </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611342,
      "author_name": "borisdee",
      "author_url": "",
      "post_date": "08/29/2019 08:09:16",
      "content": "<p>Btw, my hardware is a DGX with 8 P100 on a single node, as well as 3 additional nodes with 4 V100 on each node.</p>",
      "votes": null,
      "replies": [
        {
          "id": 611387,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/29/2019 08:31:33",
          "content": "<p>I wanted to ask about it, thanks.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 611391,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "08/29/2019 08:36:24",
          "content": "<p>I must add that a 6layer version of our model with about half the parameters still scores -2.91 LB and is significantly faster to train. We trained that in kaggle kernels.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 611687,
          "author_name": "xiaokangwang",
          "author_url": "",
          "post_date": "08/29/2019 11:40:27",
          "content": "<p>The speed is very impressive. What do you think is the reason that your network is so fast?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 612287,
          "author_name": "borisdee",
          "author_url": "",
          "post_date": "08/29/2019 17:48:19",
          "content": "<p><a href=\"/xiaokangwang\">@xiaokangwang</a> \nA few reasons:\n- We preprocessed the graph generation instead of having them generated in the model\n- We used atom centered symmetry functions of lower quality\nThat lowered the training time a lot</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 611363,
      "author_name": "ogrellier",
      "author_url": "",
      "post_date": "08/29/2019 08:20:20",
      "content": "<p>Amazing work and Congratulations to you and your team !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611366,
      "author_name": "veeralakrishna",
      "author_url": "",
      "post_date": "08/29/2019 08:22:09",
      "content": "<p>Congrats! I look forward to learning from your code.... Thanks...!! <a href=\"/christofhenkel\">@christofhenkel</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611368,
      "author_name": "silence2",
      "author_url": "",
      "post_date": "08/29/2019 08:23:09",
      "content": "<p>I'm pretty impressed and inspired by the amount of homework which has gone into this work. \n Thanks for sharing your methodology.  Like many others here, I'm pretty interested in what you did which reduced the epoch time by that great a factor. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611408,
      "author_name": "khahuras",
      "author_url": "",
      "post_date": "08/29/2019 08:44:14",
      "content": "<p>May I ask a very impolite question? Are you the person in your avatar photo ? </p>",
      "votes": null,
      "replies": [
        {
          "id": 611423,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "08/29/2019 08:49:35",
          "content": "",
          "votes": null,
          "replies": []
        },
        {
          "id": 613652,
          "author_name": "tunguz",
          "author_url": "",
          "post_date": "08/30/2019 17:43:03",
          "content": "<p>I used a GAN to figure out what he really looks like:</p>\n\n<p><img src=\"https://pbs.twimg.com/media/EDPAiBeW4AAchK2?format=jpg\" alt=\"dieter\"></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 611537,
      "author_name": "akirasosa",
      "author_url": "",
      "post_date": "08/29/2019 10:06:49",
      "content": "<p>Congrats!\nSo, you did not use QM9. Right? If it's true, really impressive.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 611808,
      "author_name": "joelhanson",
      "author_url": "",
      "post_date": "08/29/2019 12:57:28",
      "content": "<p>Amazing work <a href=\"/christofhenkel\">@christofhenkel</a> and congrats.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 613604,
      "author_name": "michelml",
      "author_url": "",
      "post_date": "08/30/2019 16:47:58",
      "content": "<p>Thank you <a href=\"/christofhenkel\">@christofhenkel</a> - very informative. Congrats to you and your team <a href=\"/psilogram\">@psilogram</a> <a href=\"/tunguz\">@tunguz</a> <a href=\"/artgor\">@artgor</a> and <a href=\"/borisdee\">@borisdee</a> .</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 615591,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "09/02/2019 06:19:03",
      "content": "<p>working on some illustrations:</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2F08edcad3e8ea6a2446a8b8d0c69af278%2FNN%20champs.001.jpeg?generation=1567405111654062&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 615604,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "09/02/2019 06:45:01",
          "content": "<p>Interaction Block:</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2Fe41816914a4d02650cfe75908d271b28%2FNN%20champs.003.jpeg?generation=1567406675179912&amp;alt=media\" alt=\"\"></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 615720,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "09/02/2019 09:39:28",
          "content": "<p>Thanks for these pictures.   My NN looks like a subset of yours, maybe it means I was on the right track? </p>\n\n<p>The most striking difference with what i tried is the molecule features.  How much do you gain from using them?  Which features are they?  QM9?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 615729,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "09/02/2019 09:54:35",
          "content": "<p>the molecule features add only small value to lmae (&lt; 0.05). The biggest contribution are </p>\n\n<ul>\n<li>updating bond hidden state (~ -0.15)</li>\n<li>add last bond hidden state to regression head (~ -0.15)</li>\n<li>have dense connections between interaction blocks (~ -0.1)</li>\n<li>squeeze and excitation for atom hidden (~ -0.05)</li>\n</ul>",
          "votes": null,
          "replies": []
        },
        {
          "id": 617113,
          "author_name": "carlolepelaars",
          "author_url": "",
          "post_date": "09/03/2019 18:50:59",
          "content": "<p>Impressive architecture!  I would like to catch up with the current state-of-the-art on graph neural networks so this post is a goldmine. Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 615605,
      "author_name": "adityaecdrid",
      "author_url": "",
      "post_date": "09/02/2019 06:46:31",
      "content": "<p>Who else is in love with the Arch?\nThanks Dieter!!! Looking forward to see the snippets :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 615635,
      "author_name": "duykhanh99",
      "author_url": "",
      "post_date": "09/02/2019 07:42:10",
      "content": "<p>Amazing Solution and Congrats to you and your team !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 616462,
      "author_name": "rhhridoy",
      "author_url": "",
      "post_date": "09/03/2019 06:35:47",
      "content": "<p>Congrats and thanks a lot for sharing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 622448,
      "author_name": "adityaecdrid",
      "author_url": "",
      "post_date": "09/09/2019 17:05:18",
      "content": "<p>Did you plan to release the code snippets for the arch?\nThnx.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "611275": "Thanks to the organizers for such an interesting competition. It was a lot of fun exploring the current state of the art of graph neural networks and trying to improve it. I also want to thank @artgor , @tunguz , @borisdee  and @psilogram  for being amazing team mates. In short, our solution is a custom Graph Neural Network with SchNet interaction at its core, which alone scores ~ -2.95 on leaderboard. The model is a single model for all bonding types.\n\nSince my team already had good engineered features for their lgb (with lb score of -2.0) when I joined, I could fully concentrate on what I love: finding a strong deep learning based solution. \n\n**The journey was as following**\n\nI started with refactoring Hengs starter kit for MPNN which was at around (-1.4 LB) for speed optimization to have a better speed of iterating ideas. After 2 days I got it down from 15 min to 90 sec  per epoch on my GTX1080Ti , without losing any accuracy. That enabled to quickly explore different ideas to improve MPNN which I implemented then. However the leaderboard score was still worse than our lgb. \n\nSo I continued to read a lot of papers related to GNN:\n\n- papers referenced in forum, and \n- newer papers referencing that papers\n- papers and git repos found by stalking top LB positions\n- papers shared by @artgor who posted new ones in our slack before I have read the previous\n\nI implemented most of them in hacky way to see if they fit for this competition. Besides others I implemented the papers listed below. Some had usable git repositories for others I need to implement more or less from scratch because they used a different input format or were coded in tensorflow/keras whereas I had setup everything in pytorch. All of them are supposed to predict properties on molecular level, so I adjusted where necessary to work on atom pair level:\n\n- [MPNN](https://arxiv.org/abs/1704.01212)\n- [GAT](https://github.com/SeongokRyu/Graph-neural-networks/blob/master/tutorials/gat.ipynb)\n- [Crystal GNN](https://github.com/txie-93/cgcnn)\n- [SchNet](https://arxiv.org/abs/1706.08566)\n- [MegNet](https://github.com/materialsvirtuallab/megnet)\n- [3DGNN](https://github.com/blackmints/3DGCN)\n\n\nFor me SchNet not only had the best code basis but also got decent results and was relatively fast, so I started building on that one. In general the model consists of 3 parts:\n\n1. encoder for nodes and edges\n2. consecutive blocks of interaction between encoded nodes and edges\n3. regression head \n\nAs mentioned by other solutions SchNet lacks several things, but the core interaction between atoms and edges is good and fast.\nSo I spent some weeks improving that baseline step by step by adding new stuff I read in other papers  or I found useful during computer vision competitions. From every 10 ideas I implemented a maximum of one worked, and it felt like digging for treasures. At the end not much was left from the original SchNet apart from the core interaction function between nodes and edges.\n\n**The most architectural enhancements were the following**\n\n- using additionally edge features like bondtype, angles\n- additionally updating edges (as in MegNet)\n- [densenet similar connections](https://arxiv.org/pdf/1608.06993.pdf) between interaction blocks\n- [squeeze and excitation block](https://arxiv.org/abs/1709.01507) at the end of an Interaction\n- encoding and using  also molecule level features as in Hengs starter\n- using edge level hidden units for final regression\n- using molecule level hidden units for final regression\n- using a deep architecture (12 interaction blocks)\n\n\n**some details on training**\n\n- preprocessing: save graph to disk (40 min) \n- bs: 64\n- optimizer: Adam\n- loss: weighted lmae (weights by inverse frequency of coupling type counts)\n- lr_schedule: Cyclic Cosine annealing with initial lr 0.0005 and cycles of 200 epochs which enables to  continue training easily by just adding a new cycle.\n- validation: groupkfold (10fold) per molecule\n- epochs: 1200 (and still underfitting)\n\n\n**our final submission:**\n\nWe basically just bagged versions of the above model trained with different hyperparameters which took some resources. So we split training by @borisdee @tunguz and myself. Some version of MegNet scoring around -2.5 LB made it also in the final blend but with very minor contribution. @psilogram then used a simple mae optimizer to blend the different models on bonding type level. -&gt; private leaderboard score of -3.001 \n\n**things that did not work**\n- additional task of autoencoding edge or node features\n- metric learning\n- and thousand of others.\n\nHappy to answer any questions and thanks for reading.",
    "611296": "Well done, congrats on the result!  Do you plan to share your code?",
    "611307": "Good question. I don't see why not.",
    "611314": "Congratulations and thanks for sharing!",
    "611316": "\"After 2 days I got it down from 15 min to 90 sec per epoch on my GTX1080Ti , without losing any accuracy. \"\n\nThat is very impressive !!!",
    "611332": "Congrats! I look forward to learning from your code. Not just the architectures but also how you optimized Heng's code to run so fast!",
    "611339": "Congrats for the result. 1200 EPOCHS; how much is your training time. I had implemented SchNEt with edges but due to lack of hardware resources and novice in deep learning I am able to train only ~150 epochs. Looking forward for your source code to learn more things.",
    "611342": "Btw, my hardware is a DGX with 8 P100 on a single node, as well as 3 additional nodes with 4 V100 on each node.",
    "611363": "Amazing work and Congratulations to you and your team !",
    "611366": "Congrats! I look forward to learning from your code.... Thanks...!! @christofhenkel",
    "611368": "I'm pretty impressed and inspired by the amount of homework which has gone into this work. \n Thanks for sharing your methodology.  Like many others here, I'm pretty interested in what you did which reduced the epoch time by that great a factor.",
    "611387": "I wanted to ask about it, thanks.",
    "611388": "Thanks, I'll learn a lot given I tried to build a GNN from scratch, and I only scratched the surface!",
    "611391": "I must add that a 6layer version of our model with about half the parameters still scores -2.91 LB and is significantly faster to train. We trained that in kaggle kernels.",
    "611408": "May I ask a very impolite question? Are you the person in your avatar photo ?",
    "611423": "",
    "611537": "Congrats!\nSo, you did not use QM9. Right? If it's true, really impressive.",
    "611687": "The speed is very impressive. What do you think is the reason that your network is so fast?",
    "611694": "Heng is back! Your code inspired a lot of people. Could you elaborate more what you did here?",
    "611808": "Amazing work @christofhenkel and congrats.",
    "612287": "xiaokangwang \nA few reasons:\n- We preprocessed the graph generation instead of having them generated in the model\n- We used atom centered symmetry functions of lower quality\nThat lowered the training time a lot",
    "613604": "Thank you @christofhenkel - very informative. Congrats to you and your team @psilogram @tunguz @artgor and @borisdee .",
    "613652": "I used a GAN to figure out what he really looks like:\n\n![dieter](https://pbs.twimg.com/media/EDPAiBeW4AAchK2?format=jpg)",
    "615591": "working on some illustrations:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2F08edcad3e8ea6a2446a8b8d0c69af278%2FNN%20champs.001.jpeg?generation=1567405111654062&amp;alt=media)",
    "615604": "Interaction Block:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2Fe41816914a4d02650cfe75908d271b28%2FNN%20champs.003.jpeg?generation=1567406675179912&amp;alt=media)",
    "615605": "Who else is in love with the Arch?\nThanks Dieter!!! Looking forward to see the snippets :)",
    "615635": "Amazing Solution and Congrats to you and your team !",
    "615720": "Thanks for these pictures.   My NN looks like a subset of yours, maybe it means I was on the right track? \n\nThe most striking difference with what i tried is the molecule features.  How much do you gain from using them?  Which features are they?  QM9?",
    "615729": "the molecule features add only small value to lmae (&lt; 0.05). The biggest contribution are \n\n- updating bond hidden state (~ -0.15)\n- add last bond hidden state to regression head (~ -0.15)\n- have dense connections between interaction blocks (~ -0.1)\n- squeeze and excitation for atom hidden (~ -0.05)",
    "616462": "Congrats and thanks a lot for sharing.",
    "617113": "Impressive architecture!  I would like to catch up with the current state-of-the-art on graph neural networks so this post is a goldmine. Thank you!",
    "622448": "Did you plan to release the code snippets for the arch?\nThnx.",
    "629355": "Hi Dieter, if you share the code, will you update the post?"
  },
  "source": "meta"
}