{
  "id": 106298,
  "title": "14th place solution (GCN from scratch in Kaggle kernel)",
  "url": "/competitions/champs-scalar-coupling/discussion/106298",
  "author_name": "Cédric Bény",
  "post_date": "2019-08-29T02:40:38.053000",
  "votes": 55,
  "comment_count": 18,
  "views": 0,
  "content": "<p>For this competition, I implemented a GCN using keras.</p>\n\n<p><strong>Input features</strong>: </p>\n\n<p>1/ Atom types (one-hot encoding)\n2/ Bond adjacency\n3/ Bond types (one-hot encoding, generated using Open Babel)\n4/ Bond vectors, normalized\n5/ Inverse distance between all atom pairs, together with powers up to 4\n6/ Cosine of bond angles for pairs of atoms 2 apart (for which we have to predict the coupling)\n7/ Cosine of dihedral angles for pairs of atoms 3 apart (for which we have to predict the coupling)\n8/ A flag on atoms whose dihedral contribution come from more than one path because they are part of a loop (in which case the max dihedral angle is retained, which is probably suboptimal)</p>\n\n<p>Most features were created \"offline\", and pickled in a compact form to avoid exhausting the Kaggle instance memory. The data is \"decompressed'' per-batch by a keras generator object. All the information regarding pairs of atoms are input in a 29x29 tensor, where 29 is the maximum number of atom per molecule. Entries for non-existing atoms are just zero'd, and masks are used where necessary. </p>\n\n<p><strong>Data augmentation</strong>:</p>\n\n<p>Random rotations to the bond vectors, to enforce independence from the reference frame. </p>\n\n<p><strong>Network structure</strong>:</p>\n\n<p>The network consists of 4 sections.</p>\n\n<p>1/ The vertex (atoms) and edge (atoms x atoms) information is processed by 4 layers of width 1 convolutions (i.e., the same fully connected network is apply to each vertex independently, and similarly for the edges).\n2/ 6 graph convolution layers are applied (transforming the vertex data, but leaving the edge data constant).\n3/ A special layer passes the information from the vertices to the edges.\n4/ 4 layers of width 1 convolutions are applied to the edges, to produce the magnetic coupling predictions.</p>\n\n<p>Originally I only had 2 (body), 3 (head) and 4 (tentacles) :)</p>\n\n<p><strong>Extra data</strong>:</p>\n\n<p>The extra training data not present on the test data was used as follows.</p>\n\n<p>1/ Mulliken charges, and shieleding tensor eigenvalues were predicted by two width 1 convo from the output of the the 3rd GC layer, and the prediction input into the 4th GC layer.\n2/ The magnetic contributions are predicted all together at the output of the network and summed to provide the final predictions.</p>\n\n<p>Unfortunately, only the eigenvalues of the shielding tensor can be used because the tensor are not provided in the correct reference frame.</p>\n\n<p><strong>Regularization</strong>:</p>\n\n<p>All layers are followed by ReLU nonlinearities, and most preceded by a Batch Normalization layer.</p>\n\n<p><strong>Result</strong>:</p>\n\n<p>This produces a score of -2.6 after about 27 hours runtime of the script (kernel) (3 times 9 hours continuous runs, with weights saved in-between). Averaging the predictions from about 10 runs yield the -2.9 final score.</p>\n\n<p><strong>Potential improvement</strong>:</p>\n\n<p>I was working on implementing a GCN with operations equivariant to rotations (as in <a href=\"https://arxiv.org/abs/1802.08219\">https://arxiv.org/abs/1802.08219</a> and <a href=\"https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275#\">https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275#</a>), but didn't have enough time to finish.</p>\n\n<p><strong>Script</strong>:\n<a href=\"https://www.kaggle.com/cedben/squid?scriptVersionId=19764703\">https://www.kaggle.com/cedben/squid?scriptVersionId=19764703</a></p>\n\n<p>[edited: network description]</p>",
  "messages": [
    {
      "id": 610891,
      "postDate": "2019-08-29T02:40:38.053Z",
      "content": "<p>For this competition, I implemented a GCN using keras.</p>\n\n<p><strong>Input features</strong>: </p>\n\n<p>1/ Atom types (one-hot encoding)\n2/ Bond adjacency\n3/ Bond types (one-hot encoding, generated using Open Babel)\n4/ Bond vectors, normalized\n5/ Inverse distance between all atom pairs, together with powers up to 4\n6/ Cosine of bond angles for pairs of atoms 2 apart (for which we have to predict the coupling)\n7/ Cosine of dihedral angles for pairs of atoms 3 apart (for which we have to predict the coupling)\n8/ A flag on atoms whose dihedral contribution come from more than one path because they are part of a loop (in which case the max dihedral angle is retained, which is probably suboptimal)</p>\n\n<p>Most features were created \"offline\", and pickled in a compact form to avoid exhausting the Kaggle instance memory. The data is \"decompressed'' per-batch by a keras generator object. All the information regarding pairs of atoms are input in a 29x29 tensor, where 29 is the maximum number of atom per molecule. Entries for non-existing atoms are just zero'd, and masks are used where necessary. </p>\n\n<p><strong>Data augmentation</strong>:</p>\n\n<p>Random rotations to the bond vectors, to enforce independence from the reference frame. </p>\n\n<p><strong>Network structure</strong>:</p>\n\n<p>The network consists of 4 sections.</p>\n\n<p>1/ The vertex (atoms) and edge (atoms x atoms) information is processed by 4 layers of width 1 convolutions (i.e., the same fully connected network is apply to each vertex independently, and similarly for the edges).\n2/ 6 graph convolution layers are applied (transforming the vertex data, but leaving the edge data constant).\n3/ A special layer passes the information from the vertices to the edges.\n4/ 4 layers of width 1 convolutions are applied to the edges, to produce the magnetic coupling predictions.</p>\n\n<p>Originally I only had 2 (body), 3 (head) and 4 (tentacles) :)</p>\n\n<p><strong>Extra data</strong>:</p>\n\n<p>The extra training data not present on the test data was used as follows.</p>\n\n<p>1/ Mulliken charges, and shieleding tensor eigenvalues were predicted by two width 1 convo from the output of the the 3rd GC layer, and the prediction input into the 4th GC layer.\n2/ The magnetic contributions are predicted all together at the output of the network and summed to provide the final predictions.</p>\n\n<p>Unfortunately, only the eigenvalues of the shielding tensor can be used because the tensor are not provided in the correct reference frame.</p>\n\n<p><strong>Regularization</strong>:</p>\n\n<p>All layers are followed by ReLU nonlinearities, and most preceded by a Batch Normalization layer.</p>\n\n<p><strong>Result</strong>:</p>\n\n<p>This produces a score of -2.6 after about 27 hours runtime of the script (kernel) (3 times 9 hours continuous runs, with weights saved in-between). Averaging the predictions from about 10 runs yield the -2.9 final score.</p>\n\n<p><strong>Potential improvement</strong>:</p>\n\n<p>I was working on implementing a GCN with operations equivariant to rotations (as in <a href=\"https://arxiv.org/abs/1802.08219\">https://arxiv.org/abs/1802.08219</a> and <a href=\"https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275#\">https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275#</a>), but didn't have enough time to finish.</p>\n\n<p><strong>Script</strong>:\n<a href=\"https://www.kaggle.com/cedben/squid?scriptVersionId=19764703\">https://www.kaggle.com/cedben/squid?scriptVersionId=19764703</a></p>\n\n<p>[edited: network description]</p>",
      "rawMarkdown": "For this competition, I implemented a GCN using keras.\n\n**Input features**: \n\n1/ Atom types (one-hot encoding)\n2/ Bond adjacency\n3/ Bond types (one-hot encoding, generated using Open Babel)\n4/ Bond vectors, normalized\n5/ Inverse distance between all atom pairs, together with powers up to 4\n6/ Cosine of bond angles for pairs of atoms 2 apart (for which we have to predict the coupling)\n7/ Cosine of dihedral angles for pairs of atoms 3 apart (for which we have to predict the coupling)\n8/ A flag on atoms whose dihedral contribution come from more than one path because they are part of a loop (in which case the max dihedral angle is retained, which is probably suboptimal)\n\nMost features were created \"offline\", and pickled in a compact form to avoid exhausting the Kaggle instance memory. The data is \"decompressed'' per-batch by a keras generator object. All the information regarding pairs of atoms are input in a 29x29 tensor, where 29 is the maximum number of atom per molecule. Entries for non-existing atoms are just zero'd, and masks are used where necessary. \n\n\n**Data augmentation**:\n\nRandom rotations to the bond vectors, to enforce independence from the reference frame. \n\n\n**Network structure**:\n\nThe network consists of 4 sections.\n\n1/ The vertex (atoms) and edge (atoms x atoms) information is processed by 4 layers of width 1 convolutions (i.e., the same fully connected network is apply to each vertex independently, and similarly for the edges).\n2/ 6 graph convolution layers are applied (transforming the vertex data, but leaving the edge data constant).\n3/ A special layer passes the information from the vertices to the edges.\n4/ 4 layers of width 1 convolutions are applied to the edges, to produce the magnetic coupling predictions.\n\nOriginally I only had 2 (body), 3 (head) and 4 (tentacles) :)\n\n**Extra data**:\n\nThe extra training data not present on the test data was used as follows.\n\n1/ Mulliken charges, and shieleding tensor eigenvalues were predicted by two width 1 convo from the output of the the 3rd GC layer, and the prediction input into the 4th GC layer.\n2/ The magnetic contributions are predicted all together at the output of the network and summed to provide the final predictions.\n\nUnfortunately, only the eigenvalues of the shielding tensor can be used because the tensor are not provided in the correct reference frame.\n\n\n**Regularization**:\n\nAll layers are followed by ReLU nonlinearities, and most preceded by a Batch Normalization layer.\n\n\n**Result**:\n\nThis produces a score of -2.6 after about 27 hours runtime of the script (kernel) (3 times 9 hours continuous runs, with weights saved in-between). Averaging the predictions from about 10 runs yield the -2.9 final score.\n\n\n**Potential improvement**:\n\nI was working on implementing a GCN with operations equivariant to rotations (as in https://arxiv.org/abs/1802.08219 and https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275#), but didn't have enough time to finish.\n\n**Script**:\nhttps://www.kaggle.com/cedben/squid?scriptVersionId=19764703\n\n[edited: network description]",
      "votes": 54
    },
    {
      "id": 612527,
      "postDate": "2019-08-29T21:05:21.167Z",
      "content": "<p>Hi Squid <a href=\"/cedben\">@cedben</a> </p>\n\n<p>In data description you write: \"2 atom indices of the pair, total magnetic coupling, 4 magnetic coupling contributions\" for the array <code>mag          (# couplings) x 8</code></p>\n\n<p>However this should only be 7 items not 8. I am confused by the first 3 things in these array. For example consider <code>train_data[0]</code> we have:</p>\n\n<p><code>\n array([[  0.      ,   1.      ,   0.      ,  84.8076  ,  83.0224  ,\n           0.254579,   1.25862 ,   0.27201 ],\n        [  0.      ,   1.      ,   2.      , -11.257   , -11.0347  ,\n           0.352978,   2.85839 ,  -3.4336  ],\n        [  0.      ,   1.      ,   3.      , -11.2548  , -11.0325  ,\n           0.352944,   2.85852 ,  -3.43387 ],\n        [  0.      ,   1.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352934,   2.85855 ,  -3.43393 ],\n        [  0.      ,   2.      ,   0.      ,  84.8074  ,  83.0222  ,\n           0.254585,   1.25861 ,   0.272013],\n        [  0.      ,   2.      ,   3.      , -11.2541  , -11.0317  ,\n           0.352932,   2.85856 ,  -3.43395 ],\n        [  0.      ,   2.      ,   4.      , -11.2548  , -11.0324  ,\n           0.352943,   2.85853 ,  -3.43387 ],\n        [  0.      ,   3.      ,   0.      ,  84.8093  ,  83.0241  ,\n           0.254634,   1.25856 ,   0.272012],\n        [  0.      ,   3.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352943,   2.85856 ,  -3.43393 ],\n        [  0.      ,   4.      ,   0.      ,  84.8095  ,  83.0243  ,\n           0.254628,   1.25856 ,   0.272012]], dtype=float32))\n</code></p>\n\n<p>Why is the number 0 always added to the start? It seems like you can just get rid of this. The second and third things represent the atom indices from my understanding. But why every single array preprended with 0!? Thanks</p>",
      "rawMarkdown": "Hi Squid @cedben \n\nIn data description you write: \"2 atom indices of the pair, total magnetic coupling, 4 magnetic coupling contributions\" for the array `mag          (# couplings) x 8 `\n\nHowever this should only be 7 items not 8. I am confused by the first 3 things in these array. For example consider `train_data[0]` we have:\n\n```\n array([[  0.      ,   1.      ,   0.      ,  84.8076  ,  83.0224  ,\n           0.254579,   1.25862 ,   0.27201 ],\n        [  0.      ,   1.      ,   2.      , -11.257   , -11.0347  ,\n           0.352978,   2.85839 ,  -3.4336  ],\n        [  0.      ,   1.      ,   3.      , -11.2548  , -11.0325  ,\n           0.352944,   2.85852 ,  -3.43387 ],\n        [  0.      ,   1.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352934,   2.85855 ,  -3.43393 ],\n        [  0.      ,   2.      ,   0.      ,  84.8074  ,  83.0222  ,\n           0.254585,   1.25861 ,   0.272013],\n        [  0.      ,   2.      ,   3.      , -11.2541  , -11.0317  ,\n           0.352932,   2.85856 ,  -3.43395 ],\n        [  0.      ,   2.      ,   4.      , -11.2548  , -11.0324  ,\n           0.352943,   2.85853 ,  -3.43387 ],\n        [  0.      ,   3.      ,   0.      ,  84.8093  ,  83.0241  ,\n           0.254634,   1.25856 ,   0.272012],\n        [  0.      ,   3.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352943,   2.85856 ,  -3.43393 ],\n        [  0.      ,   4.      ,   0.      ,  84.8095  ,  83.0243  ,\n           0.254628,   1.25856 ,   0.272012]], dtype=float32))\n```\n\nWhy is the number 0 always added to the start? It seems like you can just get rid of this. The second and third things represent the atom indices from my understanding. But why every single array preprended with 0!? Thanks\n\n",
      "votes": 1,
      "replies": [
        {
          "id": 612755,
          "postDate": "2019-08-30T01:27:21.323Z",
          "content": "<p>Ah, indeed I didn't realize I wasn't setting the index 0. I can't recall why... maybe because I had just switched from months of Julia :D   The code appears consistent though. Let me just update the description. Thanks for noticing!</p>",
          "rawMarkdown": "Ah, indeed I didn't realize I wasn't setting the index 0. I can't recall why... maybe because I had just switched from months of Julia :D   The code appears consistent though. Let me just update the description. Thanks for noticing!",
          "votes": 1
        },
        {
          "id": 612880,
          "postDate": "2019-08-30T04:01:03.063Z",
          "content": "<p>Why do you blame Julia? Did you write first in Julia then switch to Python? Also glad to know it doesn't affect anything 😅 </p>",
          "rawMarkdown": "Why do you blame Julia? Did you write first in Julia then switch to Python? Also glad to know it doesn't affect anything 😅 "
        },
        {
          "id": 612917,
          "postDate": "2019-08-30T04:59:12.493Z",
          "content": "<p>I didn't write it in Julia, but I got used to Julia's arrays starting with index 1 rather than 0. I actually prefer it, it makes things a bit more straightforward. </p>",
          "rawMarkdown": "I didn't write it in Julia, but I got used to Julia's arrays starting with index 1 rather than 0. I actually prefer it, it makes things a bit more straightforward. "
        }
      ]
    },
    {
      "id": 611367,
      "postDate": "2019-08-29T08:22:29.790Z",
      "content": "<p><a href=\"/cedben\">@cedben</a> this is really good! thank you so much :)\nbtw, could you make public the datasets please? I'd like to run it and do some test ;)</p>",
      "rawMarkdown": "@cedben this is really good! thank you so much :)\nbtw, could you make public the datasets please? I'd like to run it and do some test ;)",
      "votes": 1,
      "replies": [
        {
          "id": 611409,
          "postDate": "2019-08-29T08:44:44.150Z",
          "content": "<p>Ah, I made a mistake. The dataset should be public now!</p>",
          "rawMarkdown": "Ah, I made a mistake. The dataset should be public now!"
        },
        {
          "id": 611417,
          "postDate": "2019-08-29T08:46:55.230Z",
          "content": "<p>thank you so much :)</p>",
          "rawMarkdown": "thank you so much :)"
        }
      ]
    },
    {
      "id": 611244,
      "postDate": "2019-08-29T07:23:19.703Z",
      "content": "<p>Congrats on the great result! I am looking forward to learning from your code.</p>",
      "rawMarkdown": "Congrats on the great result! I am looking forward to learning from your code.",
      "votes": 1
    },
    {
      "id": 611359,
      "postDate": "2019-08-29T08:19:19.360Z",
      "content": "<p>The script is now public: <a href=\"https://www.kaggle.com/cedben/squid?scriptVersionId=19764703\">https://www.kaggle.com/cedben/squid?scriptVersionId=19764703</a></p>",
      "rawMarkdown": "The script is now public: https://www.kaggle.com/cedben/squid?scriptVersionId=19764703",
      "votes": 2
    },
    {
      "id": 612851,
      "postDate": "2019-08-30T03:37:04.020Z",
      "content": "<p>Congratulations! Amazing solutions! :-)</p>",
      "rawMarkdown": "Congratulations! Amazing solutions! :-)"
    },
    {
      "id": 612726,
      "postDate": "2019-08-30T00:53:59.217Z",
      "content": "<p>Thanks <a href=\"/cedben\">@cedben</a> , and special congrats to got this position alone!</p>",
      "rawMarkdown": "Thanks @cedben , and special congrats to got this position alone!"
    },
    {
      "id": 611418,
      "postDate": "2019-08-29T08:47:30.927Z",
      "content": "<p>Congrats on the impressive solo performance!</p>",
      "rawMarkdown": "Congrats on the impressive solo performance!"
    },
    {
      "id": 611145,
      "postDate": "2019-08-29T06:07:35.113Z",
      "content": "<p>Congratulations and thanks for sharing👍 </p>",
      "rawMarkdown": "Congratulations and thanks for sharing👍 "
    },
    {
      "id": 611043,
      "postDate": "2019-08-29T04:31:32.563Z",
      "content": "<p>Let me confirm shielding tensor eigenvalues were sorted just like large to small ?\nOne more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard.\nHow do you improve the model ? </p>\n\n<p>Congratulations. I want be a great kaggler like you. 😊 </p>",
      "rawMarkdown": "Let me confirm shielding tensor eigenvalues were sorted just like large to small ?\nOne more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard.\nHow do you improve the model ? \n\nCongratulations. I want be a great kaggler like you. 😊 ",
      "replies": [
        {
          "id": 611116,
          "postDate": "2019-08-29T05:38:19.187Z",
          "content": "<blockquote>\n  <p>Let me confirm shielding tensor eigenvalues were sorted just like large to small ?</p>\n</blockquote>\n\n<p>Indeed, the eigenvalues are sorted (maybe some sort of permutation-invariant NN would be better?)</p>\n\n<blockquote>\n  <p>One more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard. How do you improve the model ?</p>\n</blockquote>\n\n<p>I started with smaller networks, and sometimes based my judgment on the first 10 epochs, as it seemed to correlate with the final CV score. But I also used full runs, which allowed me to still do my day job in-between :)\nAlso at the end I used the full computations both for contributing to the final average and testing different parameters. </p>\n\n<p>It helped that Kaggle allowed 5 parallel jobs then...</p>\n\n<blockquote>\n  <p>Congratulations. I want be a great kaggler like you. 😊</p>\n</blockquote>\n\n<p>Thanks :)</p>",
          "rawMarkdown": "&gt;Let me confirm shielding tensor eigenvalues were sorted just like large to small ?\n\nIndeed, the eigenvalues are sorted (maybe some sort of permutation-invariant NN would be better?)\n\n&gt;One more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard. How do you improve the model ?\n\nI started with smaller networks, and sometimes based my judgment on the first 10 epochs, as it seemed to correlate with the final CV score. But I also used full runs, which allowed me to still do my day job in-between :)\nAlso at the end I used the full computations both for contributing to the final average and testing different parameters. \n\nIt helped that Kaggle allowed 5 parallel jobs then...\n\n&gt;Congratulations. I want be a great kaggler like you. 😊\n\nThanks :)"
        }
      ]
    },
    {
      "id": 610903,
      "postDate": "2019-08-29T02:49:03.147Z",
      "content": "<p>Congratulations. Interesting that averaging predictions would give a 0.3 boost ! </p>",
      "rawMarkdown": "Congratulations. Interesting that averaging predictions would give a 0.3 boost ! "
    },
    {
      "id": 611012,
      "postDate": "2019-08-29T04:04:38.423Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 610967,
      "postDate": "2019-08-29T03:36:06.647Z",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing!"
    }
  ],
  "comments": [
    {
      "id": 612527,
      "author_name": "CoreyJamesLevinson",
      "author_url": "",
      "post_date": "2019-08-29T21:05:21.167000",
      "content": "<p>Hi Squid <a href=\"/cedben\">@cedben</a> </p>\n\n<p>In data description you write: \"2 atom indices of the pair, total magnetic coupling, 4 magnetic coupling contributions\" for the array <code>mag          (# couplings) x 8</code></p>\n\n<p>However this should only be 7 items not 8. I am confused by the first 3 things in these array. For example consider <code>train_data[0]</code> we have:</p>\n\n<p><code>\n array([[  0.      ,   1.      ,   0.      ,  84.8076  ,  83.0224  ,\n           0.254579,   1.25862 ,   0.27201 ],\n        [  0.      ,   1.      ,   2.      , -11.257   , -11.0347  ,\n           0.352978,   2.85839 ,  -3.4336  ],\n        [  0.      ,   1.      ,   3.      , -11.2548  , -11.0325  ,\n           0.352944,   2.85852 ,  -3.43387 ],\n        [  0.      ,   1.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352934,   2.85855 ,  -3.43393 ],\n        [  0.      ,   2.      ,   0.      ,  84.8074  ,  83.0222  ,\n           0.254585,   1.25861 ,   0.272013],\n        [  0.      ,   2.      ,   3.      , -11.2541  , -11.0317  ,\n           0.352932,   2.85856 ,  -3.43395 ],\n        [  0.      ,   2.      ,   4.      , -11.2548  , -11.0324  ,\n           0.352943,   2.85853 ,  -3.43387 ],\n        [  0.      ,   3.      ,   0.      ,  84.8093  ,  83.0241  ,\n           0.254634,   1.25856 ,   0.272012],\n        [  0.      ,   3.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352943,   2.85856 ,  -3.43393 ],\n        [  0.      ,   4.      ,   0.      ,  84.8095  ,  83.0243  ,\n           0.254628,   1.25856 ,   0.272012]], dtype=float32))\n</code></p>\n\n<p>Why is the number 0 always added to the start? It seems like you can just get rid of this. The second and third things represent the atom indices from my understanding. But why every single array preprended with 0!? Thanks</p>",
      "votes": 1,
      "replies": [
        {
          "id": 612755,
          "author_name": "Cédric Bény",
          "author_url": "",
          "post_date": "2019-08-30T01:27:21.323000",
          "content": "<p>Ah, indeed I didn't realize I wasn't setting the index 0. I can't recall why... maybe because I had just switched from months of Julia :D   The code appears consistent though. Let me just update the description. Thanks for noticing!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 612880,
          "author_name": "CoreyJamesLevinson",
          "author_url": "",
          "post_date": "2019-08-30T04:01:03.063000",
          "content": "<p>Why do you blame Julia? Did you write first in Julia then switch to Python? Also glad to know it doesn't affect anything 😅 </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 612917,
          "author_name": "Cédric Bény",
          "author_url": "",
          "post_date": "2019-08-30T04:59:12.493000",
          "content": "<p>I didn't write it in Julia, but I got used to Julia's arrays starting with index 1 rather than 0. I actually prefer it, it makes things a bit more straightforward. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 611367,
      "author_name": "Nanashi",
      "author_url": "",
      "post_date": "2019-08-29T08:22:29.790000",
      "content": "<p><a href=\"/cedben\">@cedben</a> this is really good! thank you so much :)\nbtw, could you make public the datasets please? I'd like to run it and do some test ;)</p>",
      "votes": 1,
      "replies": [
        {
          "id": 611409,
          "author_name": "Cédric Bény",
          "author_url": "",
          "post_date": "2019-08-29T08:44:44.150000",
          "content": "<p>Ah, I made a mistake. The dataset should be public now!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 611417,
          "author_name": "Nanashi",
          "author_url": "",
          "post_date": "2019-08-29T08:46:55.230000",
          "content": "<p>thank you so much :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 611244,
      "author_name": "Tania J",
      "author_url": "",
      "post_date": "2019-08-29T07:23:19.703000",
      "content": "<p>Congrats on the great result! I am looking forward to learning from your code.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 611359,
      "author_name": "Cédric Bény",
      "author_url": "",
      "post_date": "2019-08-29T08:19:19.360000",
      "content": "<p>The script is now public: <a href=\"https://www.kaggle.com/cedben/squid?scriptVersionId=19764703\">https://www.kaggle.com/cedben/squid?scriptVersionId=19764703</a></p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 612851,
      "author_name": "FGPC",
      "author_url": "",
      "post_date": "2019-08-30T03:37:04.020000",
      "content": "<p>Congratulations! Amazing solutions! :-)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 612726,
      "author_name": "Michel ML",
      "author_url": "",
      "post_date": "2019-08-30T00:53:59.217000",
      "content": "<p>Thanks <a href=\"/cedben\">@cedben</a> , and special congrats to got this position alone!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611418,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2019-08-29T08:47:30.927000",
      "content": "<p>Congrats on the impressive solo performance!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611145,
      "author_name": "Arun Poochelvan",
      "author_url": "",
      "post_date": "2019-08-29T06:07:35.113000",
      "content": "<p>Congratulations and thanks for sharing👍 </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611043,
      "author_name": "XY",
      "author_url": "",
      "post_date": "2019-08-29T04:31:32.563000",
      "content": "<p>Let me confirm shielding tensor eigenvalues were sorted just like large to small ?\nOne more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard.\nHow do you improve the model ? </p>\n\n<p>Congratulations. I want be a great kaggler like you. 😊 </p>",
      "votes": 0,
      "replies": [
        {
          "id": 611116,
          "author_name": "Cédric Bény",
          "author_url": "",
          "post_date": "2019-08-29T05:38:19.187000",
          "content": "<blockquote>\n  <p>Let me confirm shielding tensor eigenvalues were sorted just like large to small ?</p>\n</blockquote>\n\n<p>Indeed, the eigenvalues are sorted (maybe some sort of permutation-invariant NN would be better?)</p>\n\n<blockquote>\n  <p>One more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard. How do you improve the model ?</p>\n</blockquote>\n\n<p>I started with smaller networks, and sometimes based my judgment on the first 10 epochs, as it seemed to correlate with the final CV score. But I also used full runs, which allowed me to still do my day job in-between :)\nAlso at the end I used the full computations both for contributing to the final average and testing different parameters. </p>\n\n<p>It helped that Kaggle allowed 5 parallel jobs then...</p>\n\n<blockquote>\n  <p>Congratulations. I want be a great kaggler like you. 😊</p>\n</blockquote>\n\n<p>Thanks :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 610903,
      "author_name": "David Ten",
      "author_url": "",
      "post_date": "2019-08-29T02:49:03.147000",
      "content": "<p>Congratulations. Interesting that averaging predictions would give a 0.3 boost ! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 611012,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-08-29T04:04:38.423000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 610967,
      "author_name": "Andrey Lukyanenko",
      "author_url": "",
      "post_date": "2019-08-29T03:36:06.647000",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "610891": "For this competition, I implemented a GCN using keras.\n\n**Input features**: \n\n1/ Atom types (one-hot encoding)\n2/ Bond adjacency\n3/ Bond types (one-hot encoding, generated using Open Babel)\n4/ Bond vectors, normalized\n5/ Inverse distance between all atom pairs, together with powers up to 4\n6/ Cosine of bond angles for pairs of atoms 2 apart (for which we have to predict the coupling)\n7/ Cosine of dihedral angles for pairs of atoms 3 apart (for which we have to predict the coupling)\n8/ A flag on atoms whose dihedral contribution come from more than one path because they are part of a loop (in which case the max dihedral angle is retained, which is probably suboptimal)\n\nMost features were created \"offline\", and pickled in a compact form to avoid exhausting the Kaggle instance memory. The data is \"decompressed'' per-batch by a keras generator object. All the information regarding pairs of atoms are input in a 29x29 tensor, where 29 is the maximum number of atom per molecule. Entries for non-existing atoms are just zero'd, and masks are used where necessary. \n\n\n**Data augmentation**:\n\nRandom rotations to the bond vectors, to enforce independence from the reference frame. \n\n\n**Network structure**:\n\nThe network consists of 4 sections.\n\n1/ The vertex (atoms) and edge (atoms x atoms) information is processed by 4 layers of width 1 convolutions (i.e., the same fully connected network is apply to each vertex independently, and similarly for the edges).\n2/ 6 graph convolution layers are applied (transforming the vertex data, but leaving the edge data constant).\n3/ A special layer passes the information from the vertices to the edges.\n4/ 4 layers of width 1 convolutions are applied to the edges, to produce the magnetic coupling predictions.\n\nOriginally I only had 2 (body), 3 (head) and 4 (tentacles) :)\n\n**Extra data**:\n\nThe extra training data not present on the test data was used as follows.\n\n1/ Mulliken charges, and shieleding tensor eigenvalues were predicted by two width 1 convo from the output of the the 3rd GC layer, and the prediction input into the 4th GC layer.\n2/ The magnetic contributions are predicted all together at the output of the network and summed to provide the final predictions.\n\nUnfortunately, only the eigenvalues of the shielding tensor can be used because the tensor are not provided in the correct reference frame.\n\n\n**Regularization**:\n\nAll layers are followed by ReLU nonlinearities, and most preceded by a Batch Normalization layer.\n\n\n**Result**:\n\nThis produces a score of -2.6 after about 27 hours runtime of the script (kernel) (3 times 9 hours continuous runs, with weights saved in-between). Averaging the predictions from about 10 runs yield the -2.9 final score.\n\n\n**Potential improvement**:\n\nI was working on implementing a GCN with operations equivariant to rotations (as in https://arxiv.org/abs/1802.08219 and https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275#), but didn't have enough time to finish.\n\n**Script**:\nhttps://www.kaggle.com/cedben/squid?scriptVersionId=19764703\n\n[edited: network description]",
    "612527": "Hi Squid @cedben \n\nIn data description you write: \"2 atom indices of the pair, total magnetic coupling, 4 magnetic coupling contributions\" for the array `mag          (# couplings) x 8 `\n\nHowever this should only be 7 items not 8. I am confused by the first 3 things in these array. For example consider `train_data[0]` we have:\n\n```\n array([[  0.      ,   1.      ,   0.      ,  84.8076  ,  83.0224  ,\n           0.254579,   1.25862 ,   0.27201 ],\n        [  0.      ,   1.      ,   2.      , -11.257   , -11.0347  ,\n           0.352978,   2.85839 ,  -3.4336  ],\n        [  0.      ,   1.      ,   3.      , -11.2548  , -11.0325  ,\n           0.352944,   2.85852 ,  -3.43387 ],\n        [  0.      ,   1.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352934,   2.85855 ,  -3.43393 ],\n        [  0.      ,   2.      ,   0.      ,  84.8074  ,  83.0222  ,\n           0.254585,   1.25861 ,   0.272013],\n        [  0.      ,   2.      ,   3.      , -11.2541  , -11.0317  ,\n           0.352932,   2.85856 ,  -3.43395 ],\n        [  0.      ,   2.      ,   4.      , -11.2548  , -11.0324  ,\n           0.352943,   2.85853 ,  -3.43387 ],\n        [  0.      ,   3.      ,   0.      ,  84.8093  ,  83.0241  ,\n           0.254634,   1.25856 ,   0.272012],\n        [  0.      ,   3.      ,   4.      , -11.2543  , -11.0319  ,\n           0.352943,   2.85856 ,  -3.43393 ],\n        [  0.      ,   4.      ,   0.      ,  84.8095  ,  83.0243  ,\n           0.254628,   1.25856 ,   0.272012]], dtype=float32))\n```\n\nWhy is the number 0 always added to the start? It seems like you can just get rid of this. The second and third things represent the atom indices from my understanding. But why every single array preprended with 0!? Thanks\n\n",
    "611367": "@cedben this is really good! thank you so much :)\nbtw, could you make public the datasets please? I'd like to run it and do some test ;)",
    "611244": "Congrats on the great result! I am looking forward to learning from your code.",
    "611359": "The script is now public: https://www.kaggle.com/cedben/squid?scriptVersionId=19764703",
    "612851": "Congratulations! Amazing solutions! :-)",
    "612726": "Thanks @cedben , and special congrats to got this position alone!",
    "611418": "Congrats on the impressive solo performance!",
    "611145": "Congratulations and thanks for sharing👍 ",
    "611043": "Let me confirm shielding tensor eigenvalues were sorted just like large to small ?\nOne more question, training NN take a lot of time. So, I suppose frequent modifications would be very hard.\nHow do you improve the model ? \n\nCongratulations. I want be a great kaggler like you. 😊 ",
    "610903": "Congratulations. Interesting that averaging predictions would give a 0.3 boost ! ",
    "611012": "",
    "610967": "Congratulations and thanks for sharing!"
  },
  "complete": true,
  "source": "existing_discussion",
  "competition_id": 14313,
  "competition_slug": "champs-scalar-coupling",
  "team_id": 3273456,
  "solution_url": "/competitions/champs-scalar-coupling/writeups/squid-14th-place-solution-gcn-from-scratch-in-kagg"
}