Hi everyone,
\n\nFirst I'd like to thank the organizers for the well organized competition, but I'd also like to thank other competitors for making things fun, your great scores pushed us to do much more and get much better scores than we have originally though we could achieve.
\n\nHere we'd like to share our solution which is also based on a Graph Transformer network, though with a few twists.
\n\nWe have used OpenBabel to infer bonds from atom coordinates, and have used some custom code to fix what we identified as mistakes made by OpenBabel.
\n\nWe construct a graph where each node is an atom, each bond becomes an edge, then we add artificial edges between all nodes at distance 2 and 3 in the graph (2J and 3J edges respectively).\nAll edges are directional, so we have one edge for each direction to make the graph fully bidirectional.
\n\nAttributes:\n- Nodes have atom type embedding, electronegativity, first ionization energy, electron affinity, Mulliken charge taken from the QM9 data (originally predicted using the same model architecture trained on the provided Mulliken charges)\n- All edges have distance and edge type embedding (single bond, double bond, triple bond, 2J, 3J)\n- 2J edges have bond angle on the atom they are skipping over.\n- 3J edges have dihedral angle between the atoms they are connecting
\n\nNo explicit XYZ data is used, as we wanted to make the model rotation and translation invariant.
\n\nAll input data was normalized to zero mean unit variance.
\n\nCore of the architecture is the graph attention network with multiple heads, with a few twists:\n- Attention heads do not attend to all data from previous layer, but only the output of the same head from previous layer\n- All edge embeddings are first updated from triplets (src, edge, dst), the attention then updates atom embeddings by aggregating over edge embeddings, not neighboring node embeddings\n- We use gated residual connections between attention blocks similar to: https://arxiv.org/abs/1805.10988\n- We output scalar coupling constants directly on the edges, this makes it two predictions, one for each edge direction and each is treated independently in the loss function. These two predictions are averaged to get a final prediction making this a kind of micro-ensemble.
\n\nLoss function is a mean of MAEs per coupling type. We have tried mean log MAE but it was giving us worse results. We have used both scaled targets to zero mean unit variance and zero mean targets with no variance scaling, as different types benefited from different setups.
\n\nQuickest way to outline the network would be using code:
\n\n```python\nemb = 48\nheads = 24\nbias = False
\n\ndef AttnBlock(in_emb, out_emb):\n return nn.Sequential(\n EdgeLinear(in_emb, out_emb),\n NodeLinear(in_emb, out_emb),\n GraphLambda(lambda x: x.view(x.shape[0], heads, -1)),\n TripletCat(out='triplet'),\n MagicAttn(emb, 3 * emb, heads, attn_key='triplet'),\n TripletMultiLinear(emb, emb, emb, heads, bias=bias),\n GraphLambda(torch.nn.LayerNorm(heads * emb))\n )
\n\nnet = nn.Sequential(\n Embed(emb, emb),\n AttnBlock(emb, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n EdgeLinear(emb * heads, 512, bias=True), GraphLambda(nn.PReLU(), node_key=None),\n EdgeLinear(512, 8, bias=True)\n)\n```
\n\nWe have used LAMB optimizer (https://arxiv.org/abs/1904.00962), again with a small twist, we have noticed that the weight decay is included in the step norm on which the trust ratio is calculated, this didn't make sense to us as weight decay should be independent of the update based on the batch gradient, so we moved the application of weight decay after the application of LAMB update, and this gave us better results. We call this LAMBW
\n\nWe have split the data into 90/10 train/eval, two times with two different. We have used one cycle learning rate for 30 epochs with high weight decay, then dropped weight decay and continued training until eval saturation (~70 more epochs). Then we fine tune 100 epochs for each type to get further improvement (except for *JHN types, as they didn't improve with fine tuning).
\n\nWe have used Stochastic Weight Averaging (https://arxiv.org/abs/1803.05407) of last 25 epoch to get to a better minimum.
\n\nWe then average the predictions of the two training runs for the two splits we had.
\n\nMany, many things:\n- RAdam\n- combining RAdam and LAMBW\n- many different forms of attention mechanisms\n- mean log MAE loss, mean MSE loss\n- GradNorm for multitask learning for different coupling types https://arxiv.org/abs/1711.02257\n- Multi-Task Learning as Multi-Objective Optimization https://arxiv.org/abs/1810.04650\nAnd many many other things
\n\nThe full package with transformed data, additional data (and all code, which is also included in champs_code.tgz) is 7GB, so we are providing a link https://zenodo.org/record/3406154#.XXpX_ygzabg instead of uploading directly to Kaggle forum.
", "messages": [ { "id": 612513, "postDate": "2019-08-29T20:52:10.887Z", "content": "Hi everyone,
\n\nFirst I'd like to thank the organizers for the well organized competition, but I'd also like to thank other competitors for making things fun, your great scores pushed us to do much more and get much better scores than we have originally though we could achieve.
\n\nHere we'd like to share our solution which is also based on a Graph Transformer network, though with a few twists.
\n\nWe have used OpenBabel to infer bonds from atom coordinates, and have used some custom code to fix what we identified as mistakes made by OpenBabel.
\n\nWe construct a graph where each node is an atom, each bond becomes an edge, then we add artificial edges between all nodes at distance 2 and 3 in the graph (2J and 3J edges respectively).\nAll edges are directional, so we have one edge for each direction to make the graph fully bidirectional.
\n\nAttributes:\n- Nodes have atom type embedding, electronegativity, first ionization energy, electron affinity, Mulliken charge taken from the QM9 data (originally predicted using the same model architecture trained on the provided Mulliken charges)\n- All edges have distance and edge type embedding (single bond, double bond, triple bond, 2J, 3J)\n- 2J edges have bond angle on the atom they are skipping over.\n- 3J edges have dihedral angle between the atoms they are connecting
\n\nNo explicit XYZ data is used, as we wanted to make the model rotation and translation invariant.
\n\nAll input data was normalized to zero mean unit variance.
\n\nCore of the architecture is the graph attention network with multiple heads, with a few twists:\n- Attention heads do not attend to all data from previous layer, but only the output of the same head from previous layer\n- All edge embeddings are first updated from triplets (src, edge, dst), the attention then updates atom embeddings by aggregating over edge embeddings, not neighboring node embeddings\n- We use gated residual connections between attention blocks similar to: https://arxiv.org/abs/1805.10988\n- We output scalar coupling constants directly on the edges, this makes it two predictions, one for each edge direction and each is treated independently in the loss function. These two predictions are averaged to get a final prediction making this a kind of micro-ensemble.
\n\nLoss function is a mean of MAEs per coupling type. We have tried mean log MAE but it was giving us worse results. We have used both scaled targets to zero mean unit variance and zero mean targets with no variance scaling, as different types benefited from different setups.
\n\nQuickest way to outline the network would be using code:
\n\n```python\nemb = 48\nheads = 24\nbias = False
\n\ndef AttnBlock(in_emb, out_emb):\n return nn.Sequential(\n EdgeLinear(in_emb, out_emb),\n NodeLinear(in_emb, out_emb),\n GraphLambda(lambda x: x.view(x.shape[0], heads, -1)),\n TripletCat(out='triplet'),\n MagicAttn(emb, 3 * emb, heads, attn_key='triplet'),\n TripletMultiLinear(emb, emb, emb, heads, bias=bias),\n GraphLambda(torch.nn.LayerNorm(heads * emb))\n )
\n\nnet = nn.Sequential(\n Embed(emb, emb),\n AttnBlock(emb, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n EdgeLinear(emb * heads, 512, bias=True), GraphLambda(nn.PReLU(), node_key=None),\n EdgeLinear(512, 8, bias=True)\n)\n```
\n\nWe have used LAMB optimizer (https://arxiv.org/abs/1904.00962), again with a small twist, we have noticed that the weight decay is included in the step norm on which the trust ratio is calculated, this didn't make sense to us as weight decay should be independent of the update based on the batch gradient, so we moved the application of weight decay after the application of LAMB update, and this gave us better results. We call this LAMBW
\n\nWe have split the data into 90/10 train/eval, two times with two different. We have used one cycle learning rate for 30 epochs with high weight decay, then dropped weight decay and continued training until eval saturation (~70 more epochs). Then we fine tune 100 epochs for each type to get further improvement (except for *JHN types, as they didn't improve with fine tuning).
\n\nWe have used Stochastic Weight Averaging (https://arxiv.org/abs/1803.05407) of last 25 epoch to get to a better minimum.
\n\nWe then average the predictions of the two training runs for the two splits we had.
\n\nMany, many things:\n- RAdam\n- combining RAdam and LAMBW\n- many different forms of attention mechanisms\n- mean log MAE loss, mean MSE loss\n- GradNorm for multitask learning for different coupling types https://arxiv.org/abs/1711.02257\n- Multi-Task Learning as Multi-Objective Optimization https://arxiv.org/abs/1810.04650\nAnd many many other things
\n\nThe full package with transformed data, additional data (and all code, which is also included in champs_code.tgz) is 7GB, so we are providing a link https://zenodo.org/record/3406154#.XXpX_ygzabg instead of uploading directly to Kaggle forum.
", "rawMarkdown": "Hi everyone,\n\nFirst I'd like to thank the organizers for the well organized competition, but I'd also like to thank other competitors for making things fun, your great scores pushed us to do much more and get much better scores than we have originally though we could achieve.\n\nHere we'd like to share our solution which is also based on a Graph Transformer network, though with a few twists.\n\n### Input data\nWe have used OpenBabel to infer bonds from atom coordinates, and have used some custom code to fix what we identified as mistakes made by OpenBabel.\n\nWe construct a graph where each node is an atom, each bond becomes an edge, then we add artificial edges between all nodes at distance 2 and 3 in the graph (2J and 3J edges respectively).\nAll edges are directional, so we have one edge for each direction to make the graph fully bidirectional.\n\n**Attributes**:\n- Nodes have atom type embedding, electronegativity, first ionization energy, electron affinity, Mulliken charge taken from the QM9 data (originally predicted using the same model architecture trained on the provided Mulliken charges)\n- All edges have distance and edge type embedding (single bond, double bond, triple bond, 2J, 3J)\n- 2J edges have bond angle on the atom they are skipping over.\n- 3J edges have dihedral angle between the atoms they are connecting\n\nNo explicit XYZ data is used, as we wanted to make the model rotation and translation invariant.\n\nAll input data was normalized to zero mean unit variance.\n\n### Model\nCore of the architecture is the graph attention network with multiple heads, with a few twists:\n- Attention heads do not attend to all data from previous layer, but only the output of the same head from previous layer\n- All edge embeddings are first updated from triplets (src, edge, dst), the attention then updates atom embeddings by aggregating over edge embeddings, not neighboring node embeddings\n- We use gated residual connections between attention blocks similar to: https://arxiv.org/abs/1805.10988\n- We output scalar coupling constants directly on the edges, this makes it two predictions, one for each edge direction and each is treated independently in the loss function. These two predictions are averaged to get a final prediction making this a kind of micro-ensemble.\n\nLoss function is a mean of MAEs per coupling type. We have tried mean log MAE but it was giving us worse results. We have used both scaled targets to zero mean unit variance and zero mean targets with no variance scaling, as different types benefited from different setups.\n\nQuickest way to outline the network would be using code:\n\n```python\nemb = 48\nheads = 24\nbias = False\n\ndef AttnBlock(in_emb, out_emb):\n return nn.Sequential(\n EdgeLinear(in_emb, out_emb),\n NodeLinear(in_emb, out_emb),\n GraphLambda(lambda x: x.view(x.shape[0], heads, -1)),\n TripletCat(out='triplet'),\n MagicAttn(emb, 3 * emb, heads, attn_key='triplet'),\n TripletMultiLinear(emb, emb, emb, heads, bias=bias),\n GraphLambda(torch.nn.LayerNorm(heads * emb))\n )\n\nnet = nn.Sequential(\n Embed(emb, emb),\n AttnBlock(emb, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n EdgeLinear(emb * heads, 512, bias=True), GraphLambda(nn.PReLU(), node_key=None),\n EdgeLinear(512, 8, bias=True)\n)\n```\n\n### Optimizer\nWe have used LAMB optimizer (https://arxiv.org/abs/1904.00962), again with a small twist, we have noticed that the weight decay is included in the step norm on which the trust ratio is calculated, this didn't make sense to us as weight decay should be independent of the update based on the batch gradient, so we moved the application of weight decay after the application of LAMB update, and this gave us better results. We call this LAMBW\n\n### Training regime\nWe have split the data into 90/10 train/eval, two times with two different. We have used one cycle learning rate for 30 epochs with high weight decay, then dropped weight decay and continued training until eval saturation (~70 more epochs). Then we fine tune 100 epochs for each type to get further improvement (except for *JHN types, as they didn't improve with fine tuning).\n\nWe have used Stochastic Weight Averaging (https://arxiv.org/abs/1803.05407) of last 25 epoch to get to a better minimum.\n\nWe then average the predictions of the two training runs for the two splits we had.\n\n### What didn't work\nMany, many things:\n- RAdam\n- combining RAdam and LAMBW\n- many different forms of attention mechanisms\n- mean log MAE loss, mean MSE loss\n- GradNorm for multitask learning for different coupling types https://arxiv.org/abs/1711.02257\n- Multi-Task Learning as Multi-Objective Optimization https://arxiv.org/abs/1810.04650\nAnd many many other things\n\nThe full package with transformed data, additional data (and all code, which is also included in champs_code.tgz) is 7GB, so we are providing a link https://zenodo.org/record/3406154#.XXpX_ygzabg instead of uploading directly to Kaggle forum.", "votes": 39 }, { "id": 613590, "postDate": "2019-08-30T16:36:16.757Z", "content": "Thanks so much for sharing @milosh - very interesting. Congrats to you and your teammates @grakocevic @starvingmarvin @boysha .
", "rawMarkdown": "Thanks so much for sharing @milosh - very interesting. Congrats to you and your teammates @grakocevic @starvingmarvin @boysha .", "votes": 1 }, { "id": 613303, "postDate": "2019-08-30T12:07:34.683Z", "content": "Thanks for sharing, and congrats on the result!
\n\n\n\n\nAnother question is why you use 1 kernel size of Conv1D?
\n
I relied on bonds output by openbabel. How much did you gain from this fix? How did you identify mistakes in the first place? Do you have background in chemistry that you used for that?
", "rawMarkdown": "Thanks for sharing, and congrats on the result!\n\n> Another question is why you use 1 kernel size of Conv1D?\n\nI relied on bonds output by openbabel. How much did you gain from this fix? How did you identify mistakes in the first place? Do you have background in chemistry that you used for that?", "votes": 1, "replies": [ { "id": 613656, "postDate": "2019-08-30T17:46:26.933Z", "content": "We mostly looked for stuff like N with 2 bonds and no formal charge. Some of these popped up when looking at molecule graphs.\nNo proper chemistry background, but we do work on applying DL to molecules (not in QM context, though).
", "rawMarkdown": "We mostly looked for stuff like N with 2 bonds and no formal charge. Some of these popped up when looking at molecule graphs.\nNo proper chemistry background, but we do work on applying DL to molecules (not in QM context, though)." } ] }, { "id": 612696, "postDate": "2019-08-30T00:21:08.477Z", "content": "Congrats on your great performance during this competition. I like your team name much.
", "rawMarkdown": "Congrats on your great performance during this competition. I like your team name much.", "votes": 1 }, { "id": 612740, "postDate": "2019-08-30T01:07:52.210Z", "content": "Congratulations and thanks for sharing!!
", "rawMarkdown": "Congratulations and thanks for sharing!!" }, { "id": 612904, "postDate": "2019-08-30T04:41:46.653Z", "rawMarkdown": "", "isDeleted": true }, { "id": 613848, "postDate": "2019-08-30T21:23:39.177Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing" } ], "comments": [ { "id": 613590, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T16:36:16.757000", "content": "Thanks so much for sharing @milosh - very interesting. Congrats to you and your teammates @grakocevic @starvingmarvin @boysha .
", "votes": 1, "replies": [] }, { "id": 613303, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-30T12:07:34.683000", "content": "Thanks for sharing, and congrats on the result!
\n\n\n\n\nAnother question is why you use 1 kernel size of Conv1D?
\n
I relied on bonds output by openbabel. How much did you gain from this fix? How did you identify mistakes in the first place? Do you have background in chemistry that you used for that?
", "votes": 1, "replies": [ { "id": 613656, "author_name": "Goran Rakocevic", "author_url": "", "post_date": "2019-08-30T17:46:26.933000", "content": "We mostly looked for stuff like N with 2 bonds and no formal charge. Some of these popped up when looking at molecule graphs.\nNo proper chemistry background, but we do work on applying DL to molecules (not in QM context, though).
", "votes": 0, "replies": [] } ] }, { "id": 612696, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-30T00:21:08.477000", "content": "Congrats on your great performance during this competition. I like your team name much.
", "votes": 1, "replies": [] }, { "id": 612740, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-30T01:07:52.210000", "content": "Congratulations and thanks for sharing!!
", "votes": 0, "replies": [] }, { "id": 612904, "author_name": "", "author_url": "", "post_date": "2019-08-30T04:41:46.653000", "content": "", "votes": 0, "replies": [] }, { "id": 613848, "author_name": "AMRL", "author_url": "", "post_date": "2019-08-30T21:23:39.177000", "content": "thanks for sharing
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "612513": "Hi everyone,\n\nFirst I'd like to thank the organizers for the well organized competition, but I'd also like to thank other competitors for making things fun, your great scores pushed us to do much more and get much better scores than we have originally though we could achieve.\n\nHere we'd like to share our solution which is also based on a Graph Transformer network, though with a few twists.\n\n### Input data\nWe have used OpenBabel to infer bonds from atom coordinates, and have used some custom code to fix what we identified as mistakes made by OpenBabel.\n\nWe construct a graph where each node is an atom, each bond becomes an edge, then we add artificial edges between all nodes at distance 2 and 3 in the graph (2J and 3J edges respectively).\nAll edges are directional, so we have one edge for each direction to make the graph fully bidirectional.\n\n**Attributes**:\n- Nodes have atom type embedding, electronegativity, first ionization energy, electron affinity, Mulliken charge taken from the QM9 data (originally predicted using the same model architecture trained on the provided Mulliken charges)\n- All edges have distance and edge type embedding (single bond, double bond, triple bond, 2J, 3J)\n- 2J edges have bond angle on the atom they are skipping over.\n- 3J edges have dihedral angle between the atoms they are connecting\n\nNo explicit XYZ data is used, as we wanted to make the model rotation and translation invariant.\n\nAll input data was normalized to zero mean unit variance.\n\n### Model\nCore of the architecture is the graph attention network with multiple heads, with a few twists:\n- Attention heads do not attend to all data from previous layer, but only the output of the same head from previous layer\n- All edge embeddings are first updated from triplets (src, edge, dst), the attention then updates atom embeddings by aggregating over edge embeddings, not neighboring node embeddings\n- We use gated residual connections between attention blocks similar to: https://arxiv.org/abs/1805.10988\n- We output scalar coupling constants directly on the edges, this makes it two predictions, one for each edge direction and each is treated independently in the loss function. These two predictions are averaged to get a final prediction making this a kind of micro-ensemble.\n\nLoss function is a mean of MAEs per coupling type. We have tried mean log MAE but it was giving us worse results. We have used both scaled targets to zero mean unit variance and zero mean targets with no variance scaling, as different types benefited from different setups.\n\nQuickest way to outline the network would be using code:\n\n```python\nemb = 48\nheads = 24\nbias = False\n\ndef AttnBlock(in_emb, out_emb):\n return nn.Sequential(\n EdgeLinear(in_emb, out_emb),\n NodeLinear(in_emb, out_emb),\n GraphLambda(lambda x: x.view(x.shape[0], heads, -1)),\n TripletCat(out='triplet'),\n MagicAttn(emb, 3 * emb, heads, attn_key='triplet'),\n TripletMultiLinear(emb, emb, emb, heads, bias=bias),\n GraphLambda(torch.nn.LayerNorm(heads * emb))\n )\n\nnet = nn.Sequential(\n Embed(emb, emb),\n AttnBlock(emb, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n GatedResidual(AttnBlock(emb * heads, emb * heads), emb * heads, emb * heads), GraphLambda(nn.PReLU()),\n EdgeLinear(emb * heads, 512, bias=True), GraphLambda(nn.PReLU(), node_key=None),\n EdgeLinear(512, 8, bias=True)\n)\n```\n\n### Optimizer\nWe have used LAMB optimizer (https://arxiv.org/abs/1904.00962), again with a small twist, we have noticed that the weight decay is included in the step norm on which the trust ratio is calculated, this didn't make sense to us as weight decay should be independent of the update based on the batch gradient, so we moved the application of weight decay after the application of LAMB update, and this gave us better results. We call this LAMBW\n\n### Training regime\nWe have split the data into 90/10 train/eval, two times with two different. We have used one cycle learning rate for 30 epochs with high weight decay, then dropped weight decay and continued training until eval saturation (~70 more epochs). Then we fine tune 100 epochs for each type to get further improvement (except for *JHN types, as they didn't improve with fine tuning).\n\nWe have used Stochastic Weight Averaging (https://arxiv.org/abs/1803.05407) of last 25 epoch to get to a better minimum.\n\nWe then average the predictions of the two training runs for the two splits we had.\n\n### What didn't work\nMany, many things:\n- RAdam\n- combining RAdam and LAMBW\n- many different forms of attention mechanisms\n- mean log MAE loss, mean MSE loss\n- GradNorm for multitask learning for different coupling types https://arxiv.org/abs/1711.02257\n- Multi-Task Learning as Multi-Objective Optimization https://arxiv.org/abs/1810.04650\nAnd many many other things\n\nThe full package with transformed data, additional data (and all code, which is also included in champs_code.tgz) is 7GB, so we are providing a link https://zenodo.org/record/3406154#.XXpX_ygzabg instead of uploading directly to Kaggle forum.", "613590": "Thanks so much for sharing @milosh - very interesting. Congrats to you and your teammates @grakocevic @starvingmarvin @boysha .", "613303": "Thanks for sharing, and congrats on the result!\n\n> Another question is why you use 1 kernel size of Conv1D?\n\nI relied on bonds output by openbabel. How much did you gain from this fix? How did you identify mistakes in the first place? Do you have background in chemistry that you used for that?", "612696": "Congrats on your great performance during this competition. I like your team name much.", "612740": "Congratulations and thanks for sharing!!", "612904": "", "613848": "thanks for sharing" }, "complete": true, "source": "existing_discussion", "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3248578, "solution_url": "/competitions/champs-scalar-coupling/writeups/hyperspatial-engineers-4-solution-hyperspatial-eng" }