{
  "id": 106293,
  "title": "10th place solution (AkiraSosa view)",
  "url": "/competitions/champs-scalar-coupling/discussion/106293",
  "author_name": "AkiraSosa",
  "post_date": "2019-08-29T02:16:01.106000",
  "votes": 42,
  "comment_count": 11,
  "views": 0,
  "content": "<p>Hi, everyone</p>\n\n<p>I would like to post my 10th place solution in my view. Though there is already another post, I have slightly different point of view, as I have spent most of my time with solo.</p>\n\n<p>At first, I pick up some essentially important points in my solution.</p>\n\n<ul>\n<li>Pipeline using model2 (model2 will be described later)\n<ul><li>It uses SchNet with Edge Update</li></ul></li>\n<li>Multitask learning using scalar coupling contributions and magnet shielding tensors.\n<ul><li>That was the only effective way for regularization for me.</li></ul></li>\n</ul>\n\n<p>Here is my brief summary.</p>\n\n<h2>Pipeline and models</h2>\n\n<p>I use 2 models, SchNet and SchNet with Edge Update. I call them as model1 and model2 respectively. Each of them were built for 8 different types, so I have 16 different models.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F0b9d522c86473d40eba4fde2e4e9f283%2F201908291758.png?generation=1567045212563825&amp;alt=media\" alt=\"pipeline\"></p>\n\n<p>Model1:</p>\n\n<ul>\n<li>Input\n<ul><li>Atom coords and type (standard input for SchNet)</li>\n<li>Mulliken charges</li>\n<li>Dihedral Graph</li></ul></li>\n<li>Output\n<ul><li>Scalar coupling constant</li>\n<li>4 Scalar coupling contributions</li>\n<li>3 atomwise values originated from magnetic shielding tensor</li></ul></li>\n<li>Architecture\n<ul><li>SchNet</li>\n<li>Concantnate 2 node features and predict atom pairwise values such as scalar coupling constant</li>\n<li>At the same time, node feature is used to predict atomwise values</li></ul></li>\n</ul>\n\n<p>Model2:</p>\n\n<ul>\n<li>Input\n<ul><li>Atom coords and type (standard input for SchNet)</li>\n<li>Mulliken charges</li>\n<li>Edge features</li>\n<li>a) Bond existence (dist=1 or not)</li>\n<li>b) Angle radian</li>\n<li>c) Angle existence (dist=2 or not)</li>\n<li>d) Dihedral cos</li>\n<li>e) Dihedral cos2</li>\n<li>f) Dihedral existence (dist=3 or not)</li>\n<li>g) Scalar coupling constant and contributions except target type derived from model1</li>\n<li>h) Scalar coupling types (one hot encoded) except target type derived from model1</li></ul></li>\n<li>Output\n<ul><li>Scalar coupling constant</li>\n<li>4 Scalar coupling contributions</li>\n<li>3 atomwise values originated from magnetic shielding tensor</li></ul></li>\n<li>Architecture\n<ul><li>SchNet with Edge Update</li>\n<li>Concantnate 2 node feats and 1 edge feats. Then, predict atom pairwise values such as scalar coupling constant</li>\n<li>At the same time, node feature is used to predict atomwise values</li></ul></li>\n</ul>\n\n<p>Finally, average the prediction of model1 and model2 </p>\n\n<h2>The motivation of model2</h2>\n\n<p>Though the model1 has relatively make good prediction by sole, it has a space for improvement. For example, when I crate a model1 for 1JHN, I don't use the information about 2JHN, 3JHN and etc. So, I create another model which uses the prediction of model1, then I can fully use all of the information. Pay attention, it's different from stacking.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F7e4e062c8574bfc9ffa1e730894bf18c%2F201908291823.png?generation=1567045265637761&amp;alt=media\" alt=\"\"></p>\n\n<h2>Discussion</h2>\n\n<p>I have struggled to regularize GCN training pretty much. How did you solve this problem? Some training using noise such as dropout show really bad result. Is it because our observation do not have noise?</p>\n\n<p>Which loss did you use? I used Log L1. It's just because it was worked. I have no idea why it was the best, because its gradient has really weird shape.</p>",
  "messages": [
    {
      "id": 610869,
      "postDate": "2019-08-29T02:16:01.107Z",
      "content": "<p>Hi, everyone</p>\n\n<p>I would like to post my 10th place solution in my view. Though there is already another post, I have slightly different point of view, as I have spent most of my time with solo.</p>\n\n<p>At first, I pick up some essentially important points in my solution.</p>\n\n<ul>\n<li>Pipeline using model2 (model2 will be described later)\n<ul><li>It uses SchNet with Edge Update</li></ul></li>\n<li>Multitask learning using scalar coupling contributions and magnet shielding tensors.\n<ul><li>That was the only effective way for regularization for me.</li></ul></li>\n</ul>\n\n<p>Here is my brief summary.</p>\n\n<h2>Pipeline and models</h2>\n\n<p>I use 2 models, SchNet and SchNet with Edge Update. I call them as model1 and model2 respectively. Each of them were built for 8 different types, so I have 16 different models.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F0b9d522c86473d40eba4fde2e4e9f283%2F201908291758.png?generation=1567045212563825&amp;alt=media\" alt=\"pipeline\"></p>\n\n<p>Model1:</p>\n\n<ul>\n<li>Input\n<ul><li>Atom coords and type (standard input for SchNet)</li>\n<li>Mulliken charges</li>\n<li>Dihedral Graph</li></ul></li>\n<li>Output\n<ul><li>Scalar coupling constant</li>\n<li>4 Scalar coupling contributions</li>\n<li>3 atomwise values originated from magnetic shielding tensor</li></ul></li>\n<li>Architecture\n<ul><li>SchNet</li>\n<li>Concantnate 2 node features and predict atom pairwise values such as scalar coupling constant</li>\n<li>At the same time, node feature is used to predict atomwise values</li></ul></li>\n</ul>\n\n<p>Model2:</p>\n\n<ul>\n<li>Input\n<ul><li>Atom coords and type (standard input for SchNet)</li>\n<li>Mulliken charges</li>\n<li>Edge features</li>\n<li>a) Bond existence (dist=1 or not)</li>\n<li>b) Angle radian</li>\n<li>c) Angle existence (dist=2 or not)</li>\n<li>d) Dihedral cos</li>\n<li>e) Dihedral cos2</li>\n<li>f) Dihedral existence (dist=3 or not)</li>\n<li>g) Scalar coupling constant and contributions except target type derived from model1</li>\n<li>h) Scalar coupling types (one hot encoded) except target type derived from model1</li></ul></li>\n<li>Output\n<ul><li>Scalar coupling constant</li>\n<li>4 Scalar coupling contributions</li>\n<li>3 atomwise values originated from magnetic shielding tensor</li></ul></li>\n<li>Architecture\n<ul><li>SchNet with Edge Update</li>\n<li>Concantnate 2 node feats and 1 edge feats. Then, predict atom pairwise values such as scalar coupling constant</li>\n<li>At the same time, node feature is used to predict atomwise values</li></ul></li>\n</ul>\n\n<p>Finally, average the prediction of model1 and model2 </p>\n\n<h2>The motivation of model2</h2>\n\n<p>Though the model1 has relatively make good prediction by sole, it has a space for improvement. For example, when I crate a model1 for 1JHN, I don't use the information about 2JHN, 3JHN and etc. So, I create another model which uses the prediction of model1, then I can fully use all of the information. Pay attention, it's different from stacking.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F7e4e062c8574bfc9ffa1e730894bf18c%2F201908291823.png?generation=1567045265637761&amp;alt=media\" alt=\"\"></p>\n\n<h2>Discussion</h2>\n\n<p>I have struggled to regularize GCN training pretty much. How did you solve this problem? Some training using noise such as dropout show really bad result. Is it because our observation do not have noise?</p>\n\n<p>Which loss did you use? I used Log L1. It's just because it was worked. I have no idea why it was the best, because its gradient has really weird shape.</p>",
      "rawMarkdown": "Hi, everyone\n\nI would like to post my 10th place solution in my view. Though there is already another post, I have slightly different point of view, as I have spent most of my time with solo.\n\nAt first, I pick up some essentially important points in my solution.\n\n* Pipeline using model2 (model2 will be described later)\n  * It uses SchNet with Edge Update\n* Multitask learning using scalar coupling contributions and magnet shielding tensors.\n  * That was the only effective way for regularization for me.\n\nHere is my brief summary.\n\n## Pipeline and models\n\nI use 2 models, SchNet and SchNet with Edge Update. I call them as model1 and model2 respectively. Each of them were built for 8 different types, so I have 16 different models.\n\n![pipeline](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F0b9d522c86473d40eba4fde2e4e9f283%2F201908291758.png?generation=1567045212563825&amp;alt=media)\n\nModel1:\n\n* Input\n  * Atom coords and type (standard input for SchNet)\n  * Mulliken charges\n  * Dihedral Graph\n* Output\n  * Scalar coupling constant\n  * 4 Scalar coupling contributions\n  * 3 atomwise values originated from magnetic shielding tensor\n* Architecture\n  * SchNet\n  * Concantnate 2 node features and predict atom pairwise values such as scalar coupling constant\n  * At the same time, node feature is used to predict atomwise values\n  \nModel2:\n\n* Input\n  * Atom coords and type (standard input for SchNet)\n  * Mulliken charges\n  * Edge features\n    * a) Bond existence (dist=1 or not)\n    * b) Angle radian\n    * c) Angle existence (dist=2 or not)\n    * d) Dihedral cos\n    * e) Dihedral cos2\n    * f) Dihedral existence (dist=3 or not)\n    * g) Scalar coupling constant and contributions except target type derived from model1\n    * h) Scalar coupling types (one hot encoded) except target type derived from model1\n* Output\n  * Scalar coupling constant\n  * 4 Scalar coupling contributions\n  * 3 atomwise values originated from magnetic shielding tensor\n* Architecture\n  * SchNet with Edge Update\n  * Concantnate 2 node feats and 1 edge feats. Then, predict atom pairwise values such as scalar coupling constant\n  * At the same time, node feature is used to predict atomwise values\n  \nFinally, average the prediction of model1 and model2 \n\n## The motivation of model2 \n\nThough the model1 has relatively make good prediction by sole, it has a space for improvement. For example, when I crate a model1 for 1JHN, I don't use the information about 2JHN, 3JHN and etc. So, I create another model which uses the prediction of model1, then I can fully use all of the information. Pay attention, it's different from stacking.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F7e4e062c8574bfc9ffa1e730894bf18c%2F201908291823.png?generation=1567045265637761&amp;alt=media)\n\n\n## Discussion\n\nI have struggled to regularize GCN training pretty much. How did you solve this problem? Some training using noise such as dropout show really bad result. Is it because our observation do not have noise?\n\nWhich loss did you use? I used Log L1. It's just because it was worked. I have no idea why it was the best, because its gradient has really weird shape.\n\n\n\n\n",
      "votes": 41
    },
    {
      "id": 610948,
      "postDate": "2019-08-29T03:22:28.623Z",
      "content": "<p>Congrats and awesome work <a href=\"/akirasosa\">@akirasosa</a> !!  This method allows you to focus on each coupling type and also incorporate information from the other types. But it does require a lot of GPUs!!  I used SchNet as well but wasn't able to incorporate additional features apart from x, y, z due to my limited PyTorch experience. </p>\n\n<p>Some questions if you don't mind:\n1.  Did you use the <a href=\"https://schnetpack.readthedocs.io/en/stable/\">https://schnetpack.readthedocs.io/en/stable/</a> library or another library or implement from scratch?</p>\n\n<ol>\n<li><p>From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models? I used SchNet with Edge Update with standard features only to predict the coupling constant only and got up to -2.34 with one 'backbone' and then 8 heads for each coupling type. For me 8 heads definitely worked better than 1 , but 8 separate models was out of the question as I only have access to Kaggle Kernels.</p></li>\n<li><p>Do the <strong>*<em>3 atomwise values originated from magnetic shielding tensor</em>*</strong> refer to the sigma, iso and kappa values in this kernel ( <a href=\"https://www.kaggle.com/tonyyy/schnet-magnetic-shielding\">https://www.kaggle.com/tonyyy/schnet-magnetic-shielding</a>)? Did you try using model 1 predictions as an input into model 2 instead of a target? Really wondering how to make good use of this data. </p></li>\n</ol>\n\n<p>My experience:\n1. I used Log L1 as well, it just works. Decaying the learning rate worked for me but occasionally the gradient would explode \n2. Adding dropout gave similar results </p>",
      "rawMarkdown": "Congrats and awesome work @akirasosa !!  This method allows you to focus on each coupling type and also incorporate information from the other types. But it does require a lot of GPUs!!  I used SchNet as well but wasn't able to incorporate additional features apart from x, y, z due to my limited PyTorch experience. \n\nSome questions if you don't mind:\n1.  Did you use the https://schnetpack.readthedocs.io/en/stable/ library or another library or implement from scratch?\n\n2. From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models? I used SchNet with Edge Update with standard features only to predict the coupling constant only and got up to -2.34 with one 'backbone' and then 8 heads for each coupling type. For me 8 heads definitely worked better than 1 , but 8 separate models was out of the question as I only have access to Kaggle Kernels.\n \n3. Do the ****3 atomwise values originated from magnetic shielding tensor**** refer to the sigma, iso and kappa values in this kernel ( https://www.kaggle.com/tonyyy/schnet-magnetic-shielding)? Did you try using model 1 predictions as an input into model 2 instead of a target? Really wondering how to make good use of this data. \n\n\nMy experience:\n1. I used Log L1 as well, it just works. Decaying the learning rate worked for me but occasionally the gradient would explode \n2. Adding dropout gave similar results ",
      "votes": 1,
      "replies": [
        {
          "id": 611154,
          "postDate": "2019-08-29T06:18:56.140Z",
          "content": "<p>Congrats your good finish <a href=\"/xang1234\">@xang1234</a> !!</p>\n\n<blockquote>\n  <p>Did you use the <a href=\"https://schnetpack.readthedocs.io/en/stable/\">https://schnetpack.readthedocs.io/en/stable/</a> library or another library or implement from scratch?</p>\n</blockquote>\n\n<p>Yes. I customized SchNet Pack for SchNet (model1). The code is well written. It's not so difficult to customize it.</p>\n\n<p>About SchNet with Edge Update, I referred <a href=\"https://github.com/NREL/nfp\">this Keras repo</a>. And I converted it into pytorch version.</p>\n\n<p>In my view, engineering is one of the important points to achieve a good result in this competition.</p>\n\n<blockquote>\n  <p>From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models?</p>\n</blockquote>\n\n<p>Actually, I tried train all types together to save my time. But, some types converges soon and some others are not. So, I gave up it.</p>\n\n<p>In my understanding, following two points improved my result.</p>\n\n<ul>\n<li>Feeding dihedral</li>\n<li>Multitask learning</li>\n</ul>\n\n<blockquote>\n  <p>Do the 3 atomwise values originated from magnetic shielding tensor refer to the sigma, iso and kappa values in this kernel ( <a href=\"https://www.kaggle.com/tonyyy/schnet-magnetic-shielding\">https://www.kaggle.com/tonyyy/schnet-magnetic-shielding</a>)? </p>\n</blockquote>\n\n<p>Yes. I scaled omega by log scale. Tony's kernel was the most important key for me.</p>",
          "rawMarkdown": "Congrats your good finish @xang1234 !!\n\n&gt; Did you use the https://schnetpack.readthedocs.io/en/stable/ library or another library or implement from scratch?\n\nYes. I customized SchNet Pack for SchNet (model1). The code is well written. It's not so difficult to customize it.\n\nAbout SchNet with Edge Update, I referred [this Keras repo](https://github.com/NREL/nfp). And I converted it into pytorch version.\n\nIn my view, engineering is one of the important points to achieve a good result in this competition.\n\n&gt; From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models?\n\nActually, I tried train all types together to save my time. But, some types converges soon and some others are not. So, I gave up it.\n\nIn my understanding, following two points improved my result.\n\n* Feeding dihedral\n* Multitask learning\n\n&gt; Do the 3 atomwise values originated from magnetic shielding tensor refer to the sigma, iso and kappa values in this kernel ( https://www.kaggle.com/tonyyy/schnet-magnetic-shielding)? \n\nYes. I scaled omega by log scale. Tony's kernel was the most important key for me.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1316939,
      "postDate": "2021-05-21T03:22:37.707Z",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/akirasosa\" target=\"_blank\">@akirasosa</a> , awesome work here !</p>\n<p>I have lost of high end GPU with not much to do, I was wondering if I can avail them to you for your work in exchange of partnering on projects here on Kaggle. I am a novice so I could really use good partners to get a good start.    </p>",
      "rawMarkdown": "Hi @akirasosa , awesome work here !\n\nI have lost of high end GPU with not much to do, I was wondering if I can avail them to you for your work in exchange of partnering on projects here on Kaggle. I am a novice so I could really use good partners to get a good start.    "
    },
    {
      "id": 611074,
      "postDate": "2019-08-29T04:51:25.187Z",
      "content": "<p>Awesome solution and explanation. The diagram is great as well! Well done. 👍 </p>",
      "rawMarkdown": "Awesome solution and explanation. The diagram is great as well! Well done. 👍 "
    },
    {
      "id": 610965,
      "postDate": "2019-08-29T03:33:12.247Z",
      "content": "<p>Congratulations and thank you for sharing!</p>",
      "rawMarkdown": "Congratulations and thank you for sharing!"
    },
    {
      "id": 611380,
      "postDate": "2019-08-29T08:27:38.720Z",
      "content": "<p>Congrats and thank you for showing your method.\nHow did you provide Mulliken charges for test data? Is it oof prediction? </p>",
      "rawMarkdown": "Congrats and thank you for showing your method.\nHow did you provide Mulliken charges for test data? Is it oof prediction? ",
      "isDeleted": true,
      "replies": [
        {
          "id": 611392,
          "postDate": "2019-08-29T08:36:53.497Z",
          "content": "<p>No. QM9 usage is allowed. We can get Mulliken charges from QM9.</p>",
          "rawMarkdown": "No. QM9 usage is allowed. We can get Mulliken charges from QM9.",
          "votes": 1
        },
        {
          "id": 611400,
          "postDate": "2019-08-29T08:40:27.263Z",
          "content": "<p>I see. Thanks for your reply.</p>",
          "rawMarkdown": "I see. Thanks for your reply.",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 610892,
      "postDate": "2019-08-29T02:40:42.247Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 611389,
      "postDate": "2019-08-29T08:33:21.333Z",
      "content": "<p>Thanks for sharing and congrats!  </p>",
      "rawMarkdown": "Thanks for sharing and congrats!  "
    },
    {
      "id": 611322,
      "postDate": "2019-08-29T08:01:27.547Z",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing!"
    }
  ],
  "comments": [
    {
      "id": 610948,
      "author_name": "David Ten",
      "author_url": "",
      "post_date": "2019-08-29T03:22:28.623000",
      "content": "<p>Congrats and awesome work <a href=\"/akirasosa\">@akirasosa</a> !!  This method allows you to focus on each coupling type and also incorporate information from the other types. But it does require a lot of GPUs!!  I used SchNet as well but wasn't able to incorporate additional features apart from x, y, z due to my limited PyTorch experience. </p>\n\n<p>Some questions if you don't mind:\n1.  Did you use the <a href=\"https://schnetpack.readthedocs.io/en/stable/\">https://schnetpack.readthedocs.io/en/stable/</a> library or another library or implement from scratch?</p>\n\n<ol>\n<li><p>From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models? I used SchNet with Edge Update with standard features only to predict the coupling constant only and got up to -2.34 with one 'backbone' and then 8 heads for each coupling type. For me 8 heads definitely worked better than 1 , but 8 separate models was out of the question as I only have access to Kaggle Kernels.</p></li>\n<li><p>Do the <strong>*<em>3 atomwise values originated from magnetic shielding tensor</em>*</strong> refer to the sigma, iso and kappa values in this kernel ( <a href=\"https://www.kaggle.com/tonyyy/schnet-magnetic-shielding\">https://www.kaggle.com/tonyyy/schnet-magnetic-shielding</a>)? Did you try using model 1 predictions as an input into model 2 instead of a target? Really wondering how to make good use of this data. </p></li>\n</ol>\n\n<p>My experience:\n1. I used Log L1 as well, it just works. Decaying the learning rate worked for me but occasionally the gradient would explode \n2. Adding dropout gave similar results </p>",
      "votes": 1,
      "replies": [
        {
          "id": 611154,
          "author_name": "AkiraSosa",
          "author_url": "",
          "post_date": "2019-08-29T06:18:56.140000",
          "content": "<p>Congrats your good finish <a href=\"/xang1234\">@xang1234</a> !!</p>\n\n<blockquote>\n  <p>Did you use the <a href=\"https://schnetpack.readthedocs.io/en/stable/\">https://schnetpack.readthedocs.io/en/stable/</a> library or another library or implement from scratch?</p>\n</blockquote>\n\n<p>Yes. I customized SchNet Pack for SchNet (model1). The code is well written. It's not so difficult to customize it.</p>\n\n<p>About SchNet with Edge Update, I referred <a href=\"https://github.com/NREL/nfp\">this Keras repo</a>. And I converted it into pytorch version.</p>\n\n<p>In my view, engineering is one of the important points to achieve a good result in this competition.</p>\n\n<blockquote>\n  <p>From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models?</p>\n</blockquote>\n\n<p>Actually, I tried train all types together to save my time. But, some types converges soon and some others are not. So, I gave up it.</p>\n\n<p>In my understanding, following two points improved my result.</p>\n\n<ul>\n<li>Feeding dihedral</li>\n<li>Multitask learning</li>\n</ul>\n\n<blockquote>\n  <p>Do the 3 atomwise values originated from magnetic shielding tensor refer to the sigma, iso and kappa values in this kernel ( <a href=\"https://www.kaggle.com/tonyyy/schnet-magnetic-shielding\">https://www.kaggle.com/tonyyy/schnet-magnetic-shielding</a>)? </p>\n</blockquote>\n\n<p>Yes. I scaled omega by log scale. Tony's kernel was the most important key for me.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1316939,
      "author_name": "KelvinMich",
      "author_url": "",
      "post_date": "2021-05-21T03:22:37.707000",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/akirasosa\" target=\"_blank\">@akirasosa</a> , awesome work here !</p>\n<p>I have lost of high end GPU with not much to do, I was wondering if I can avail them to you for your work in exchange of partnering on projects here on Kaggle. I am a novice so I could really use good partners to get a good start.    </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611074,
      "author_name": "Yassine Alouini",
      "author_url": "",
      "post_date": "2019-08-29T04:51:25.187000",
      "content": "<p>Awesome solution and explanation. The diagram is great as well! Well done. 👍 </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 610965,
      "author_name": "Andrey Lukyanenko",
      "author_url": "",
      "post_date": "2019-08-29T03:33:12.247000",
      "content": "<p>Congratulations and thank you for sharing!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611380,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-08-29T08:27:38.720000",
      "content": "<p>Congrats and thank you for showing your method.\nHow did you provide Mulliken charges for test data? Is it oof prediction? </p>",
      "votes": 0,
      "replies": [
        {
          "id": 611392,
          "author_name": "AkiraSosa",
          "author_url": "",
          "post_date": "2019-08-29T08:36:53.497000",
          "content": "<p>No. QM9 usage is allowed. We can get Mulliken charges from QM9.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 611400,
          "author_name": "",
          "author_url": "",
          "post_date": "2019-08-29T08:40:27.263000",
          "content": "<p>I see. Thanks for your reply.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 610892,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-08-29T02:40:42.247000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611389,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2019-08-29T08:33:21.333000",
      "content": "<p>Thanks for sharing and congrats!  </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611322,
      "author_name": "Stanislav Blinov",
      "author_url": "",
      "post_date": "2019-08-29T08:01:27.547000",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "610869": "Hi, everyone\n\nI would like to post my 10th place solution in my view. Though there is already another post, I have slightly different point of view, as I have spent most of my time with solo.\n\nAt first, I pick up some essentially important points in my solution.\n\n* Pipeline using model2 (model2 will be described later)\n  * It uses SchNet with Edge Update\n* Multitask learning using scalar coupling contributions and magnet shielding tensors.\n  * That was the only effective way for regularization for me.\n\nHere is my brief summary.\n\n## Pipeline and models\n\nI use 2 models, SchNet and SchNet with Edge Update. I call them as model1 and model2 respectively. Each of them were built for 8 different types, so I have 16 different models.\n\n![pipeline](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F0b9d522c86473d40eba4fde2e4e9f283%2F201908291758.png?generation=1567045212563825&amp;alt=media)\n\nModel1:\n\n* Input\n  * Atom coords and type (standard input for SchNet)\n  * Mulliken charges\n  * Dihedral Graph\n* Output\n  * Scalar coupling constant\n  * 4 Scalar coupling contributions\n  * 3 atomwise values originated from magnetic shielding tensor\n* Architecture\n  * SchNet\n  * Concantnate 2 node features and predict atom pairwise values such as scalar coupling constant\n  * At the same time, node feature is used to predict atomwise values\n  \nModel2:\n\n* Input\n  * Atom coords and type (standard input for SchNet)\n  * Mulliken charges\n  * Edge features\n    * a) Bond existence (dist=1 or not)\n    * b) Angle radian\n    * c) Angle existence (dist=2 or not)\n    * d) Dihedral cos\n    * e) Dihedral cos2\n    * f) Dihedral existence (dist=3 or not)\n    * g) Scalar coupling constant and contributions except target type derived from model1\n    * h) Scalar coupling types (one hot encoded) except target type derived from model1\n* Output\n  * Scalar coupling constant\n  * 4 Scalar coupling contributions\n  * 3 atomwise values originated from magnetic shielding tensor\n* Architecture\n  * SchNet with Edge Update\n  * Concantnate 2 node feats and 1 edge feats. Then, predict atom pairwise values such as scalar coupling constant\n  * At the same time, node feature is used to predict atomwise values\n  \nFinally, average the prediction of model1 and model2 \n\n## The motivation of model2 \n\nThough the model1 has relatively make good prediction by sole, it has a space for improvement. For example, when I crate a model1 for 1JHN, I don't use the information about 2JHN, 3JHN and etc. So, I create another model which uses the prediction of model1, then I can fully use all of the information. Pay attention, it's different from stacking.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1333985%2F7e4e062c8574bfc9ffa1e730894bf18c%2F201908291823.png?generation=1567045265637761&amp;alt=media)\n\n\n## Discussion\n\nI have struggled to regularize GCN training pretty much. How did you solve this problem? Some training using noise such as dropout show really bad result. Is it because our observation do not have noise?\n\nWhich loss did you use? I used Log L1. It's just because it was worked. I have no idea why it was the best, because its gradient has really weird shape.\n\n\n\n\n",
    "610948": "Congrats and awesome work @akirasosa !!  This method allows you to focus on each coupling type and also incorporate information from the other types. But it does require a lot of GPUs!!  I used SchNet as well but wasn't able to incorporate additional features apart from x, y, z due to my limited PyTorch experience. \n\nSome questions if you don't mind:\n1.  Did you use the https://schnetpack.readthedocs.io/en/stable/ library or another library or implement from scratch?\n\n2. From your tests, is the performance of model 1 (-2.6) due to the extra node and edge features in SchNet or due to building 8 separate models? I used SchNet with Edge Update with standard features only to predict the coupling constant only and got up to -2.34 with one 'backbone' and then 8 heads for each coupling type. For me 8 heads definitely worked better than 1 , but 8 separate models was out of the question as I only have access to Kaggle Kernels.\n \n3. Do the ****3 atomwise values originated from magnetic shielding tensor**** refer to the sigma, iso and kappa values in this kernel ( https://www.kaggle.com/tonyyy/schnet-magnetic-shielding)? Did you try using model 1 predictions as an input into model 2 instead of a target? Really wondering how to make good use of this data. \n\n\nMy experience:\n1. I used Log L1 as well, it just works. Decaying the learning rate worked for me but occasionally the gradient would explode \n2. Adding dropout gave similar results ",
    "1316939": "Hi @akirasosa , awesome work here !\n\nI have lost of high end GPU with not much to do, I was wondering if I can avail them to you for your work in exchange of partnering on projects here on Kaggle. I am a novice so I could really use good partners to get a good start.    ",
    "611074": "Awesome solution and explanation. The diagram is great as well! Well done. 👍 ",
    "610965": "Congratulations and thank you for sharing!",
    "611380": "Congrats and thank you for showing your method.\nHow did you provide Mulliken charges for test data? Is it oof prediction? ",
    "610892": "",
    "611389": "Thanks for sharing and congrats!  ",
    "611322": "Congratulations and thanks for sharing!"
  }
}