{
  "id": 106864,
  "title": "5th place solution : DL guys",
  "url": "/competitions/champs-scalar-coupling/discussion/106864",
  "author_name": "Guillaume Huard",
  "post_date": "2019-08-31T11:14:14.870000",
  "votes": 39,
  "comment_count": 8,
  "views": 0,
  "content": "<h1>Thanks</h1>\n\n<p>First of all, a big Thank You to the organizers for this outstanding competition. This competition is very challenging and interesting in many points of view.</p>\n\n<p>Secondly, congratulations to all teams which completed this competition, whether you are in the medals or not. We learned a lot from your solutions and your discussions. We didn’t have any expertise in neither chemistry nor in graph neural net before this competition, this experience has been very enlightening for the 3 of us.</p>\n\n<p>Also big kudos to the top 4 teams, your usage of Transformers are quite eye opening (honestly we did think about it but was not audacious/confident enough to test it out). Maybe this competition will open a new paradigm of Deep Learning for molecular properties - Transformer is all you need :D</p>\n\n<p>A last thank you to my teemmates and coworkers Lam Dang and Thanh Tu Nguyen :) It was very fun competing with you.</p>\n\n<h1>Solution</h1>\n\n<p>Without further ado, here is a highlight of our solution:\n- On macro level our best submission is a 2 layer stacking:\n- The base level consists of different variant of the general Graph Neural Net with edge, node and global representation with some variations (cf. Architectures below)\n- It was implemented with pytorch and pytorch_geometric.\n- The 2nd level is some metamodel trained on our validation set of 5000 molecules : 1 linear stacking model and 1 LGBM (cf. Stacking section below)\n- The final submission is a blend of 2 meta model</p>\n\n<h2>Architecture:</h2>\n\n<p>The final architecture is based on the paper <a href=\"https://arxiv.org/abs/1812.05055\">https://arxiv.org/abs/1812.05055</a>.\nWe tried different variations to improve this architecture, here is a summary of what worked and what didn’t work:\n- Normalization: We found that LayerNorm worked better that BatchNorm for this data and helps improve convergence\n- Softplus vs ReLU: Softplus did provide a ~ 0.1 boost of logMAE for our models vs a ReLU baseline\n- Edge to node message gating: We found that adding some gating mechanism to the edge representation before the scatter_mean (see torch_geomrtric) for node update helps\n- Edge to edge convolution: Guillaume implemented something that seem to work very well. He noticed after a feature importance test that the most important one was by far was the angle between an edge and the edge with the closest atom to the first edge. To integrate this angle feature for more than the closest edge, we updated each edge with a convolution of the edge in question and its neighboring edges in the graph (more specifically the neighboring edges that chemically connects two atoms), and putting in this convolution the angle of the edge vectors. This architecture tweek made our architecture 5 times slower but gave us a 0.15 improvement compared to the best model without it.\n- 1 prediction tail per type: All types share a GNN “body”, but we found that having different MLP for each type helps.\n- In some variants, before feeding into output MLP layers, we pool all the edges and nodes in the chemical bond path from atom_0 to atom_1. It seems to have helped in the beginning of the competition but our best model did not use it.\n- For our architectures, we found that having a representation of the link between atom_0 and atom_1 is important. Also including the global representation as inputs of the top layers is important</p>\n\n<h2>Stacking:</h2>\n\n<ul>\n<li>Our single best model is the one with edge to edge convolution which gives us -2.9. But we have various models around -2.7 which are the variants of it. By stacking all (20 models) we got -3.13 on LB. </li>\n<li>Another thing we found out at the last day helps improve our score from -3.13 to - 3.15 is adding checkpoints of our models to stacking pipeline. So finally, we have 50 predictions to do stacking.</li>\n</ul>\n\n<p>Our final result is a blend of LGBM and HuberRegressor. \n- LGBM: 20 GroupKFold on all bond types together\n- HuberRegressor: 20 GroupKFold on every bond type separately. </p>\n\n<h2>Computation:</h2>\n\n<p>We have : \n- 1 GTX 1080ti x 2 months + 1 RTX 2080ti x 1 month\n- 1 RTX 2080ti x 3 month\n- 1 V100 x 2 month (rented) + 4 V100 x 2 weeks (rented)</p>",
  "messages": [
    {
      "id": 614330,
      "postDate": "2019-08-31T11:14:14.870Z",
      "content": "<h1>Thanks</h1>\n\n<p>First of all, a big Thank You to the organizers for this outstanding competition. This competition is very challenging and interesting in many points of view.</p>\n\n<p>Secondly, congratulations to all teams which completed this competition, whether you are in the medals or not. We learned a lot from your solutions and your discussions. We didn’t have any expertise in neither chemistry nor in graph neural net before this competition, this experience has been very enlightening for the 3 of us.</p>\n\n<p>Also big kudos to the top 4 teams, your usage of Transformers are quite eye opening (honestly we did think about it but was not audacious/confident enough to test it out). Maybe this competition will open a new paradigm of Deep Learning for molecular properties - Transformer is all you need :D</p>\n\n<p>A last thank you to my teemmates and coworkers Lam Dang and Thanh Tu Nguyen :) It was very fun competing with you.</p>\n\n<h1>Solution</h1>\n\n<p>Without further ado, here is a highlight of our solution:\n- On macro level our best submission is a 2 layer stacking:\n- The base level consists of different variant of the general Graph Neural Net with edge, node and global representation with some variations (cf. Architectures below)\n- It was implemented with pytorch and pytorch_geometric.\n- The 2nd level is some metamodel trained on our validation set of 5000 molecules : 1 linear stacking model and 1 LGBM (cf. Stacking section below)\n- The final submission is a blend of 2 meta model</p>\n\n<h2>Architecture:</h2>\n\n<p>The final architecture is based on the paper <a href=\"https://arxiv.org/abs/1812.05055\">https://arxiv.org/abs/1812.05055</a>.\nWe tried different variations to improve this architecture, here is a summary of what worked and what didn’t work:\n- Normalization: We found that LayerNorm worked better that BatchNorm for this data and helps improve convergence\n- Softplus vs ReLU: Softplus did provide a ~ 0.1 boost of logMAE for our models vs a ReLU baseline\n- Edge to node message gating: We found that adding some gating mechanism to the edge representation before the scatter_mean (see torch_geomrtric) for node update helps\n- Edge to edge convolution: Guillaume implemented something that seem to work very well. He noticed after a feature importance test that the most important one was by far was the angle between an edge and the edge with the closest atom to the first edge. To integrate this angle feature for more than the closest edge, we updated each edge with a convolution of the edge in question and its neighboring edges in the graph (more specifically the neighboring edges that chemically connects two atoms), and putting in this convolution the angle of the edge vectors. This architecture tweek made our architecture 5 times slower but gave us a 0.15 improvement compared to the best model without it.\n- 1 prediction tail per type: All types share a GNN “body”, but we found that having different MLP for each type helps.\n- In some variants, before feeding into output MLP layers, we pool all the edges and nodes in the chemical bond path from atom_0 to atom_1. It seems to have helped in the beginning of the competition but our best model did not use it.\n- For our architectures, we found that having a representation of the link between atom_0 and atom_1 is important. Also including the global representation as inputs of the top layers is important</p>\n\n<h2>Stacking:</h2>\n\n<ul>\n<li>Our single best model is the one with edge to edge convolution which gives us -2.9. But we have various models around -2.7 which are the variants of it. By stacking all (20 models) we got -3.13 on LB. </li>\n<li>Another thing we found out at the last day helps improve our score from -3.13 to - 3.15 is adding checkpoints of our models to stacking pipeline. So finally, we have 50 predictions to do stacking.</li>\n</ul>\n\n<p>Our final result is a blend of LGBM and HuberRegressor. \n- LGBM: 20 GroupKFold on all bond types together\n- HuberRegressor: 20 GroupKFold on every bond type separately. </p>\n\n<h2>Computation:</h2>\n\n<p>We have : \n- 1 GTX 1080ti x 2 months + 1 RTX 2080ti x 1 month\n- 1 RTX 2080ti x 3 month\n- 1 V100 x 2 month (rented) + 4 V100 x 2 weeks (rented)</p>",
      "rawMarkdown": "# Thanks\nFirst of all, a big Thank You to the organizers for this outstanding competition. This competition is very challenging and interesting in many points of view.\n\nSecondly, congratulations to all teams which completed this competition, whether you are in the medals or not. We learned a lot from your solutions and your discussions. We didn’t have any expertise in neither chemistry nor in graph neural net before this competition, this experience has been very enlightening for the 3 of us.\n\nAlso big kudos to the top 4 teams, your usage of Transformers are quite eye opening (honestly we did think about it but was not audacious/confident enough to test it out). Maybe this competition will open a new paradigm of Deep Learning for molecular properties - Transformer is all you need :D\n\nA last thank you to my teemmates and coworkers Lam Dang and Thanh Tu Nguyen :) It was very fun competing with you.\n\n# Solution\nWithout further ado, here is a highlight of our solution:\n- On macro level our best submission is a 2 layer stacking:\n- The base level consists of different variant of the general Graph Neural Net with edge, node and global representation with some variations (cf. Architectures below)\n- It was implemented with pytorch and pytorch_geometric.\n- The 2nd level is some metamodel trained on our validation set of 5000 molecules : 1 linear stacking model and 1 LGBM (cf. Stacking section below)\n- The final submission is a blend of 2 meta model\n\n## Architecture:\nThe final architecture is based on the paper https://arxiv.org/abs/1812.05055.\nWe tried different variations to improve this architecture, here is a summary of what worked and what didn’t work:\n- Normalization: We found that LayerNorm worked better that BatchNorm for this data and helps improve convergence\n- Softplus vs ReLU: Softplus did provide a ~ 0.1 boost of logMAE for our models vs a ReLU baseline\n- Edge to node message gating: We found that adding some gating mechanism to the edge representation before the scatter_mean (see torch_geomrtric) for node update helps\n- Edge to edge convolution: Guillaume implemented something that seem to work very well. He noticed after a feature importance test that the most important one was by far was the angle between an edge and the edge with the closest atom to the first edge. To integrate this angle feature for more than the closest edge, we updated each edge with a convolution of the edge in question and its neighboring edges in the graph (more specifically the neighboring edges that chemically connects two atoms), and putting in this convolution the angle of the edge vectors. This architecture tweek made our architecture 5 times slower but gave us a 0.15 improvement compared to the best model without it.\n- 1 prediction tail per type: All types share a GNN “body”, but we found that having different MLP for each type helps.\n- In some variants, before feeding into output MLP layers, we pool all the edges and nodes in the chemical bond path from atom_0 to atom_1. It seems to have helped in the beginning of the competition but our best model did not use it.\n- For our architectures, we found that having a representation of the link between atom_0 and atom_1 is important. Also including the global representation as inputs of the top layers is important\n\n## Stacking:\n- Our single best model is the one with edge to edge convolution which gives us -2.9. But we have various models around -2.7 which are the variants of it. By stacking all (20 models) we got -3.13 on LB. \n- Another thing we found out at the last day helps improve our score from -3.13 to - 3.15 is adding checkpoints of our models to stacking pipeline. So finally, we have 50 predictions to do stacking.\n\nOur final result is a blend of LGBM and HuberRegressor. \n- LGBM: 20 GroupKFold on all bond types together\n- HuberRegressor: 20 GroupKFold on every bond type separately. \n\n## Computation:\nWe have : \n- 1 GTX 1080ti x 2 months + 1 RTX 2080ti x 1 month\n- 1 RTX 2080ti x 3 month\n- 1 V100 x 2 month (rented) + 4 V100 x 2 weeks (rented)\n\n",
      "votes": 39
    },
    {
      "id": 615484,
      "postDate": "2019-09-02T02:29:24.423Z",
      "content": "<p>Congratulations to you and your teammates <a href=\"/grjhuard\">@grjhuard</a> <a href=\"/lamdang\">@lamdang</a> <a href=\"/thanhtu\">@thanhtu</a> .</p>",
      "rawMarkdown": "Congratulations to you and your teammates @grjhuard @lamdang @thanhtu .",
      "votes": 1
    },
    {
      "id": 995643,
      "postDate": "2020-09-02T16:33:10.330Z",
      "content": "<p>super </p>",
      "rawMarkdown": "super "
    },
    {
      "id": 987908,
      "postDate": "2020-08-27T15:57:51.843Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!"
    },
    {
      "id": 817257,
      "postDate": "2020-04-23T02:41:41.907Z",
      "content": "<p>I want to appreciate your code, can you share with me, graet thanks!  my email: believemetoo@sina.com</p>",
      "rawMarkdown": "I want to appreciate your code, can you share with me, graet thanks!  my email: believemetoo@sina.com"
    },
    {
      "id": 781195,
      "postDate": "2020-03-21T02:45:59.477Z",
      "content": "<p>I really want to see your code. This is very helpful to me. Can you give it, open source, or my email yvquan.li@gmail.com</p>",
      "rawMarkdown": "I really want to see your code. This is very helpful to me. Can you give it, open source, or my email yvquan.li@gmail.com\n\n"
    },
    {
      "id": 614579,
      "postDate": "2019-08-31T17:49:30.050Z",
      "content": "<p>Thanks for sharing your solution, and of course, congrats!</p>\n\n<p>Can I ask how about how many features did U use in LGBM and if you have time to describe some bird eye view of the features used there? Or you used the same features for LGBM and GNN?</p>",
      "rawMarkdown": "Thanks for sharing your solution, and of course, congrats!\n\nCan I ask how about how many features did U use in LGBM and if you have time to describe some bird eye view of the features used there? Or you used the same features for LGBM and GNN?"
    },
    {
      "id": 614368,
      "postDate": "2019-08-31T11:59:22.357Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 990616,
      "postDate": "2020-08-29T18:07:02.890Z",
      "content": "<p>congratulations. thanks</p>",
      "rawMarkdown": "congratulations. thanks"
    }
  ],
  "comments": [
    {
      "id": 615484,
      "author_name": "Michel ML",
      "author_url": "",
      "post_date": "2019-09-02T02:29:24.423000",
      "content": "<p>Congratulations to you and your teammates <a href=\"/grjhuard\">@grjhuard</a> <a href=\"/lamdang\">@lamdang</a> <a href=\"/thanhtu\">@thanhtu</a> .</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 995643,
      "author_name": "P Vijay Simha Reddy",
      "author_url": "",
      "post_date": "2020-09-02T16:33:10.330000",
      "content": "<p>super </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 987908,
      "author_name": "Alin Cijov",
      "author_url": "",
      "post_date": "2020-08-27T15:57:51.843000",
      "content": "<p>Congratulations!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 817257,
      "author_name": "Zhen Tian",
      "author_url": "",
      "post_date": "2020-04-23T02:41:41.907000",
      "content": "<p>I want to appreciate your code, can you share with me, graet thanks!  my email: believemetoo@sina.com</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 781195,
      "author_name": "yuquan.li",
      "author_url": "",
      "post_date": "2020-03-21T02:45:59.477000",
      "content": "<p>I really want to see your code. This is very helpful to me. Can you give it, open source, or my email yvquan.li@gmail.com</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 614579,
      "author_name": "EdgarPE",
      "author_url": "",
      "post_date": "2019-08-31T17:49:30.050000",
      "content": "<p>Thanks for sharing your solution, and of course, congrats!</p>\n\n<p>Can I ask how about how many features did U use in LGBM and if you have time to describe some bird eye view of the features used there? Or you used the same features for LGBM and GNN?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 614368,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-08-31T11:59:22.357000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 990616,
      "author_name": "Vinay Pratap Singh",
      "author_url": "",
      "post_date": "2020-08-29T18:07:02.890000",
      "content": "<p>congratulations. thanks</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "614330": "# Thanks\nFirst of all, a big Thank You to the organizers for this outstanding competition. This competition is very challenging and interesting in many points of view.\n\nSecondly, congratulations to all teams which completed this competition, whether you are in the medals or not. We learned a lot from your solutions and your discussions. We didn’t have any expertise in neither chemistry nor in graph neural net before this competition, this experience has been very enlightening for the 3 of us.\n\nAlso big kudos to the top 4 teams, your usage of Transformers are quite eye opening (honestly we did think about it but was not audacious/confident enough to test it out). Maybe this competition will open a new paradigm of Deep Learning for molecular properties - Transformer is all you need :D\n\nA last thank you to my teemmates and coworkers Lam Dang and Thanh Tu Nguyen :) It was very fun competing with you.\n\n# Solution\nWithout further ado, here is a highlight of our solution:\n- On macro level our best submission is a 2 layer stacking:\n- The base level consists of different variant of the general Graph Neural Net with edge, node and global representation with some variations (cf. Architectures below)\n- It was implemented with pytorch and pytorch_geometric.\n- The 2nd level is some metamodel trained on our validation set of 5000 molecules : 1 linear stacking model and 1 LGBM (cf. Stacking section below)\n- The final submission is a blend of 2 meta model\n\n## Architecture:\nThe final architecture is based on the paper https://arxiv.org/abs/1812.05055.\nWe tried different variations to improve this architecture, here is a summary of what worked and what didn’t work:\n- Normalization: We found that LayerNorm worked better that BatchNorm for this data and helps improve convergence\n- Softplus vs ReLU: Softplus did provide a ~ 0.1 boost of logMAE for our models vs a ReLU baseline\n- Edge to node message gating: We found that adding some gating mechanism to the edge representation before the scatter_mean (see torch_geomrtric) for node update helps\n- Edge to edge convolution: Guillaume implemented something that seem to work very well. He noticed after a feature importance test that the most important one was by far was the angle between an edge and the edge with the closest atom to the first edge. To integrate this angle feature for more than the closest edge, we updated each edge with a convolution of the edge in question and its neighboring edges in the graph (more specifically the neighboring edges that chemically connects two atoms), and putting in this convolution the angle of the edge vectors. This architecture tweek made our architecture 5 times slower but gave us a 0.15 improvement compared to the best model without it.\n- 1 prediction tail per type: All types share a GNN “body”, but we found that having different MLP for each type helps.\n- In some variants, before feeding into output MLP layers, we pool all the edges and nodes in the chemical bond path from atom_0 to atom_1. It seems to have helped in the beginning of the competition but our best model did not use it.\n- For our architectures, we found that having a representation of the link between atom_0 and atom_1 is important. Also including the global representation as inputs of the top layers is important\n\n## Stacking:\n- Our single best model is the one with edge to edge convolution which gives us -2.9. But we have various models around -2.7 which are the variants of it. By stacking all (20 models) we got -3.13 on LB. \n- Another thing we found out at the last day helps improve our score from -3.13 to - 3.15 is adding checkpoints of our models to stacking pipeline. So finally, we have 50 predictions to do stacking.\n\nOur final result is a blend of LGBM and HuberRegressor. \n- LGBM: 20 GroupKFold on all bond types together\n- HuberRegressor: 20 GroupKFold on every bond type separately. \n\n## Computation:\nWe have : \n- 1 GTX 1080ti x 2 months + 1 RTX 2080ti x 1 month\n- 1 RTX 2080ti x 3 month\n- 1 V100 x 2 month (rented) + 4 V100 x 2 weeks (rented)\n\n",
    "615484": "Congratulations to you and your teammates @grjhuard @lamdang @thanhtu .",
    "995643": "super ",
    "987908": "Congratulations!",
    "817257": "I want to appreciate your code, can you share with me, graet thanks!  my email: believemetoo@sina.com",
    "781195": "I really want to see your code. This is very helpful to me. Can you give it, open source, or my email yvquan.li@gmail.com\n\n",
    "614579": "Thanks for sharing your solution, and of course, congrats!\n\nCan I ask how about how many features did U use in LGBM and if you have time to describe some bird eye view of the features used there? Or you used the same features for LGBM and GNN?",
    "614368": "",
    "990616": "congratulations. thanks"
  }
}