First off, I would like to thank Kaggle, the host, and my team @senkin13 , @alexpartisan , @chenlongwang . It has been an interesting journey for me and certainly very rewarding to get my first gold. And Congrats to all who has worked hard for this competition!
\n\nI was mainly working on GNN while my teammates have worked on NN/lgb/feature engineering/stacking. So I will be focusing on GNN and leave the rest to them.
\n\nI mostly used pytorch_geometric for GNN, which is an amazing library. Network structures I have tried but failed to work include NNConv (https://arxiv.org/abs/1704.02901), SchNet (https://arxiv.org/abs/1706.08566). I guess the reason is that those networks does not update edge information and this competition aims to predict edge attributes. I tried various ways (concat/add/multiply) to combine node information of the last layer into edge to make predictions but they do not work as well.
\n\nOur major break through is based on Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals (https://arxiv.org/pdf/1812.05055.pdf), in which both edge and node information are updated together. On top of the edge prediction, I add MLP per type to make the final prediction.
\n\nAs for training, we tried both training each type separately and all types together and found that training all types together gives more stable results. We also tried fine-tune each type after training all types together first. But we found that it overfits very fast and leads to worse results.
\n\nOne question I was never able to answer is that, there are two types of graph, one based on chemical bond and one based on coupling that we seek to predict. While they have the same node, edge connection is very different. Not having any background knowledge in this domain, we were not sure which one to train GNN on and left it to trial and error. Maybe someone with domain knowledge could shed some light on this.
\n\nAgain, Congrats to all winners and look forward to their solution.
\n\nBest,
\n\nZhenlan,
", "messages": [ { "id": 610833, "postDate": "2019-08-29T01:50:03.690Z", "content": "First off, I would like to thank Kaggle, the host, and my team @senkin13 , @alexpartisan , @chenlongwang . It has been an interesting journey for me and certainly very rewarding to get my first gold. And Congrats to all who has worked hard for this competition!
\n\nI was mainly working on GNN while my teammates have worked on NN/lgb/feature engineering/stacking. So I will be focusing on GNN and leave the rest to them.
\n\nI mostly used pytorch_geometric for GNN, which is an amazing library. Network structures I have tried but failed to work include NNConv (https://arxiv.org/abs/1704.02901), SchNet (https://arxiv.org/abs/1706.08566). I guess the reason is that those networks does not update edge information and this competition aims to predict edge attributes. I tried various ways (concat/add/multiply) to combine node information of the last layer into edge to make predictions but they do not work as well.
\n\nOur major break through is based on Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals (https://arxiv.org/pdf/1812.05055.pdf), in which both edge and node information are updated together. On top of the edge prediction, I add MLP per type to make the final prediction.
\n\nAs for training, we tried both training each type separately and all types together and found that training all types together gives more stable results. We also tried fine-tune each type after training all types together first. But we found that it overfits very fast and leads to worse results.
\n\nOne question I was never able to answer is that, there are two types of graph, one based on chemical bond and one based on coupling that we seek to predict. While they have the same node, edge connection is very different. Not having any background knowledge in this domain, we were not sure which one to train GNN on and left it to trial and error. Maybe someone with domain knowledge could shed some light on this.
\n\nAgain, Congrats to all winners and look forward to their solution.
\n\nBest,
\n\nZhenlan,
", "rawMarkdown": "First off, I would like to thank Kaggle, the host, and my team @senkin13 , @alexpartisan , @chenlongwang . It has been an interesting journey for me and certainly very rewarding to get my first gold. And Congrats to all who has worked hard for this competition!\n\nI was mainly working on GNN while my teammates have worked on NN/lgb/feature engineering/stacking. So I will be focusing on GNN and leave the rest to them. \n\nI mostly used [pytorch_geometric](https://github.com/rusty1s/pytorch_geometric) for GNN, which is an amazing library. Network structures I have tried but failed to work include NNConv (https://arxiv.org/abs/1704.02901), SchNet (https://arxiv.org/abs/1706.08566). I guess the reason is that those networks does not update edge information and this competition aims to predict edge attributes. I tried various ways (concat/add/multiply) to combine node information of the last layer into edge to make predictions but they do not work as well.\n\nOur major break through is based on Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals (https://arxiv.org/pdf/1812.05055.pdf), in which both edge and node information are updated together. On top of the edge prediction, I add MLP per type to make the final prediction. \n\nAs for training, we tried both training each type separately and all types together and found that training all types together gives more stable results. We also tried fine-tune each type after training all types together first. But we found that it overfits very fast and leads to worse results.\n\nOne question I was never able to answer is that, there are two types of graph, one based on chemical bond and one based on coupling that we seek to predict. While they have the same node, edge connection is very different. Not having any background knowledge in this domain, we were not sure which one to train GNN on and left it to trial and error. Maybe someone with domain knowledge could shed some light on this.\n\nAgain, Congrats to all winners and look forward to their solution.\n\nBest,\n\nZhenlan,\n", "votes": 28 }, { "id": 612719, "postDate": "2019-08-30T00:48:03.627Z", "content": "Congrats and thank @zhenlanwang , very informative.
", "rawMarkdown": "Congrats and thank @zhenlanwang , very informative.", "votes": 1, "replies": [ { "id": 612721, "postDate": "2019-08-30T00:49:31.657Z", "content": "congrats to your teammates as well @senkin13 @chenlongwang @alexpartisan
", "rawMarkdown": "congrats to your teammates as well @senkin13 @chenlongwang @alexpartisan " } ] }, { "id": 610855, "postDate": "2019-08-29T02:02:55.570Z", "content": "Lucky to have a magnificent teammate like you, really enjoy the journey fighting with you guys.
", "rawMarkdown": "Lucky to have a magnificent teammate like you, really enjoy the journey fighting with you guys.", "votes": 1 }, { "id": 611310, "postDate": "2019-08-29T07:57:01.133Z", "content": "Congrats and thanks for sharing, so what worked best between following \"real bonds\" or prediction bonds?
", "rawMarkdown": "Congrats and thanks for sharing, so what worked best between following \"real bonds\" or prediction bonds? ", "replies": [ { "id": 611997, "postDate": "2019-08-29T14:51:07.700Z", "content": "We ended up using a combination of both. \"real bond\" followed by prediction bonds.
", "rawMarkdown": "We ended up using a combination of both. \"real bond\" followed by prediction bonds. ", "votes": 1 } ] }, { "id": 611112, "postDate": "2019-08-29T05:35:32.203Z", "content": "Congratulations! We also tried MEGNet which was super easy to implement in PyTorch geometric. Unfortunately, we couldn't get it to work as well as you. Did you make individual graphs for each id in train.csv?
Each molecule forms its own graph with node being atom and edge being chemical bond or coupling.
", "rawMarkdown": "Each molecule forms its own graph with node being atom and edge being chemical bond or coupling.", "votes": 1 }, { "id": 612008, "postDate": "2019-08-29T14:59:45.290Z", "content": "Thanks very much for sharing! How did you use one graph to predict J for multiple atom pairs? (I think this is where I went wrong)
", "rawMarkdown": "Thanks very much for sharing! How did you use one graph to predict J for multiple atom pairs? (I think this is where I went wrong)" } ] }, { "id": 611038, "postDate": "2019-08-29T04:27:14.617Z", "content": "Congrats !!
", "rawMarkdown": "Congrats !!" }, { "id": 610971, "postDate": "2019-08-29T03:37:31.647Z", "content": "Congratulations!
\n\nIt is cool that you were able to make that architecture to work!
", "rawMarkdown": "Congratulations!\n\nIt is cool that you were able to make that architecture to work!" }, { "id": 610854, "postDate": "2019-08-29T02:02:49.150Z", "content": "Congratulations! Good job at GNN, Looking forward to see your solution kernel!
", "rawMarkdown": "Congratulations! Good job at GNN, Looking forward to see your solution kernel!" }, { "id": 612241, "postDate": "2019-08-29T17:12:35.307Z", "rawMarkdown": "", "isDeleted": true }, { "id": 610899, "postDate": "2019-08-29T02:45:40.133Z", "rawMarkdown": "", "isDeleted": true }, { "id": 611313, "postDate": "2019-08-29T07:59:05.400Z", "content": "Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 } ], "comments": [ { "id": 612719, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T00:48:03.627000", "content": "Congrats and thank @zhenlanwang , very informative.
", "votes": 1, "replies": [ { "id": 612721, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T00:49:31.657000", "content": "congrats to your teammates as well @senkin13 @chenlongwang @alexpartisan
", "votes": 0, "replies": [] } ] }, { "id": 610855, "author_name": "Alexpartisan", "author_url": "", "post_date": "2019-08-29T02:02:55.570000", "content": "Lucky to have a magnificent teammate like you, really enjoy the journey fighting with you guys.
", "votes": 1, "replies": [] }, { "id": 611310, "author_name": "Optimo", "author_url": "", "post_date": "2019-08-29T07:57:01.133000", "content": "Congrats and thanks for sharing, so what worked best between following \"real bonds\" or prediction bonds?
", "votes": 0, "replies": [ { "id": 611997, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-29T14:51:07.700000", "content": "We ended up using a combination of both. \"real bond\" followed by prediction bonds.
", "votes": 1, "replies": [] } ] }, { "id": 611112, "author_name": "datasaurus", "author_url": "", "post_date": "2019-08-29T05:35:32.203000", "content": "Congratulations! We also tried MEGNet which was super easy to implement in PyTorch geometric. Unfortunately, we couldn't get it to work as well as you. Did you make individual graphs for each id in train.csv?
Each molecule forms its own graph with node being atom and edge being chemical bond or coupling.
", "votes": 1, "replies": [] }, { "id": 612008, "author_name": "datasaurus", "author_url": "", "post_date": "2019-08-29T14:59:45.290000", "content": "Thanks very much for sharing! How did you use one graph to predict J for multiple atom pairs? (I think this is where I went wrong)
", "votes": 0, "replies": [] } ] }, { "id": 611038, "author_name": "Ashesh.", "author_url": "", "post_date": "2019-08-29T04:27:14.617000", "content": "Congrats !!
", "votes": 0, "replies": [] }, { "id": 610971, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-08-29T03:37:31.647000", "content": "Congratulations!
\n\nIt is cool that you were able to make that architecture to work!
", "votes": 0, "replies": [] }, { "id": 610854, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2019-08-29T02:02:49.150000", "content": "Congratulations! Good job at GNN, Looking forward to see your solution kernel!
", "votes": 0, "replies": [] }, { "id": 612241, "author_name": "", "author_url": "", "post_date": "2019-08-29T17:12:35.307000", "content": "", "votes": 0, "replies": [] }, { "id": 610899, "author_name": "", "author_url": "", "post_date": "2019-08-29T02:45:40.133000", "content": "", "votes": 0, "replies": [] }, { "id": 611313, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-08-29T07:59:05.400000", "content": "Congratulations and thanks for sharing!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "610833": "First off, I would like to thank Kaggle, the host, and my team @senkin13 , @alexpartisan , @chenlongwang . It has been an interesting journey for me and certainly very rewarding to get my first gold. And Congrats to all who has worked hard for this competition!\n\nI was mainly working on GNN while my teammates have worked on NN/lgb/feature engineering/stacking. So I will be focusing on GNN and leave the rest to them. \n\nI mostly used [pytorch_geometric](https://github.com/rusty1s/pytorch_geometric) for GNN, which is an amazing library. Network structures I have tried but failed to work include NNConv (https://arxiv.org/abs/1704.02901), SchNet (https://arxiv.org/abs/1706.08566). I guess the reason is that those networks does not update edge information and this competition aims to predict edge attributes. I tried various ways (concat/add/multiply) to combine node information of the last layer into edge to make predictions but they do not work as well.\n\nOur major break through is based on Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals (https://arxiv.org/pdf/1812.05055.pdf), in which both edge and node information are updated together. On top of the edge prediction, I add MLP per type to make the final prediction. \n\nAs for training, we tried both training each type separately and all types together and found that training all types together gives more stable results. We also tried fine-tune each type after training all types together first. But we found that it overfits very fast and leads to worse results.\n\nOne question I was never able to answer is that, there are two types of graph, one based on chemical bond and one based on coupling that we seek to predict. While they have the same node, edge connection is very different. Not having any background knowledge in this domain, we were not sure which one to train GNN on and left it to trial and error. Maybe someone with domain knowledge could shed some light on this.\n\nAgain, Congrats to all winners and look forward to their solution.\n\nBest,\n\nZhenlan,\n", "612719": "Congrats and thank @zhenlanwang , very informative.", "610855": "Lucky to have a magnificent teammate like you, really enjoy the journey fighting with you guys.", "611310": "Congrats and thanks for sharing, so what worked best between following \"real bonds\" or prediction bonds? ", "611112": "Congratulations! We also tried MEGNet which was super easy to implement in PyTorch geometric. Unfortunately, we couldn't get it to work as well as you. Did you make individual graphs for each `id` in `train.csv`?", "611038": "Congrats !!", "610971": "Congratulations!\n\nIt is cool that you were able to make that architecture to work!", "610854": "Congratulations! Good job at GNN, Looking forward to see your solution kernel!", "612241": "", "610899": "", "611313": "Congratulations and thanks for sharing!" } }