This post is for information and results using graph CNN. For a start:
\n\nuseful Graph CNN lib:
\n\nothers:
\n\n-https://github.com/ifding/graph-neural-networks
\n\npossible approach?\n - Neural Message Passing for Quantum Chemistry (google deepmind/ICML 2017)
", "messages": [ { "id": 540459, "postDate": "2019-05-31T14:49:51.140Z", "content": "This post is for information and results using graph CNN. For a start:
\n\nuseful Graph CNN lib:
\n\nothers:
\n\n-https://github.com/ifding/graph-neural-networks
\n\npossible approach?\n - Neural Message Passing for Quantum Chemistry (google deepmind/ICML 2017)
", "rawMarkdown": "This post is for information and results using graph CNN. For a start:\n\nuseful Graph CNN lib:\n\n- https://github.com/rusty1s/pytorch_geometric\n- https://github.com/pfnet-research/chainer-chemistry\n- https://github.com/jparkhill/TensorMol\n\n\n- https://deepchem.io/\n\n\nothers:\n\n-https://github.com/ifding/graph-neural-networks\n\n\npossible approach?\n - Neural Message Passing for Quantum Chemistry (google deepmind/ICML 2017)", "votes": 218 }, { "id": 558553, "postDate": "2019-06-22T14:54:19.893Z", "content": "starter-kit is up! (train for about 160k iterations/60 epoch, you can get LB of about -0.725. Note that Google paper uses about 530 epoch in their QM9 paper and train individual separate predictors, one for each target. I am not sure how many epochs are required in this kaggle challenge, so have fun in experimenting!)
\n\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/558553/13626/starter.png\")
@hengck23 thank you for sharing this.
\n\nWhat are the train_split_by_mol.80003.npy and debug_split_by_mol.1000.npy files? They are referenced but don't seem to be part of the zip.
", "rawMarkdown": "@hengck23 thank you for sharing this.\n\nWhat are the train_split_by_mol.80003.npy and debug_split_by_mol.1000.npy files? They are referenced but don't seem to be part of the zip.", "votes": 1 }, { "id": 559524, "postDate": "2019-06-24T08:13:49.933Z", "content": "they are just random split. you can create using the code:
\n\n```\ndef run_make_split():
\n\n![\"\"](\"https://raw.githubusercontent.com/jparkhill/TensorMol/master/doc/images/tyrosine.png\")
split_dir = '/root/share/project/kaggle/2019/champs_scalar/data/split'\n\ncsv_file = '/root/share/project/kaggle/2019/champs_scalar/data/csv/train.csv'\ndf = pd.read_csv(csv_file)\nmolecule_names = df.molecule_name.unique()\nmolecule_names = np.sort(molecule_names)\n\ndebug_split = molecule_names[:1000]\nnp.save(split_dir + '/debug_split_by_mol.%d.npy'%len(debug_split), debug_split)\nprint(debug_split) #'dsgdb9nsd_001679'\n\nnp.random.shuffle(molecule_names)\nif 1: #just random\n\n num_all = len(molecule_names)\n num_valid = 5000\n num_train = num_all - num_valid\n train_split = molecule_names[num_valid:]\n valid_split = molecule_names[:num_valid]\n\n np.save(split_dir + '/train_split_by_mol.%d.npy'%num_train,train_split)\n np.save(split_dir + '/valid_split_by_mol.%d.npy'%num_valid,valid_split)\n\n pass\n```
", "rawMarkdown": "they are just random split. you can create using the code:\n\n```\ndef run_make_split():\n\n split_dir = '/root/share/project/kaggle/2019/champs_scalar/data/split'\n\n csv_file = '/root/share/project/kaggle/2019/champs_scalar/data/csv/train.csv'\n df = pd.read_csv(csv_file)\n molecule_names = df.molecule_name.unique()\n molecule_names = np.sort(molecule_names)\n\n debug_split = molecule_names[:1000]\n np.save(split_dir + '/debug_split_by_mol.%d.npy'%len(debug_split), debug_split)\n print(debug_split) #'dsgdb9nsd_001679'\n\n np.random.shuffle(molecule_names)\n if 1: #just random\n\n num_all = len(molecule_names)\n num_valid = 5000\n num_train = num_all - num_valid\n train_split = molecule_names[num_valid:]\n valid_split = molecule_names[:num_valid]\n\n np.save(split_dir + '/train_split_by_mol.%d.npy'%num_train,train_split)\n np.save(split_dir + '/valid_split_by_mol.%d.npy'%num_valid,valid_split)\n\n pass\n\n\n```", "votes": 2 }, { "id": 559838, "postDate": "2019-06-24T16:49:41.057Z", "content": "you can accelerated your training using the new loss
\n\n```
\n\ndef criterion(predict, coupling_value):\n predict = predict.view(-1)\n coupling_value = coupling_value.view(-1)\n assert(predict.shape==coupling_value.shape)
\n\nloss = torch.abs(predict-coupling_value)\nloss = loss.mean()\nloss = torch.log(loss)\nreturn loss\n```
\n\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/559838/13647/new_log.png\")
Hi Heng, maybe there is no graph.py file as shown in the picture.
", "rawMarkdown": "Hi Heng, maybe there is no graph.py file as shown in the picture." }, { "id": 575222, "postDate": "2019-07-15T07:24:46.643Z", "content": "Oh. I find it in data.py. Maybe you have changed something. \nI have read your data.py carefully and find this project so cool. \nIt is really amazing to be done by yourself. Thanks for sharing!
", "rawMarkdown": "Oh. I find it in data.py. Maybe you have changed something. \nI have read your data.py carefully and find this project so cool. \nIt is really amazing to be done by yourself. Thanks for sharing!" } ] }, { "id": 546673, "postDate": "2019-06-06T19:54:24.747Z", "content": "Hi @hengck23, I remember you from a previous competition (QuickDraw). Your posts were always insightful.
\n\nI'm working right now on my Master Thesis in geometric deep-learning but, alas, it won't be helpful for this competition. Still, I might give you my two cents:\n- As you can see in this recent survey, most GCNs are designed for graph-level or node-level outputs. The method you proposed in another comment is actually the only way to go for this kind of tasks (predicting node-pair features and then use a standard MLP to predict the edge labels, in a 1D-Conv fashion).\n- If you want a model with edge-parameters, you might want to have a look to this paper from a bunch of very smart people. They propose a very generic model that can be applied to any kind of task. They also provide an implementation in TensorFlow and Sonnet, if you're interested.\n- Since this is an edge regression task, using node-pooling might be problematic: you have to preserve all the edges, and state-of-the-art methods typically don't do this. You might pool the nodes, but the clusters need to form a partition of the graph, not a cover (i.e., you have to assign to each node one and only one cluster). Classical clustering methods (e.g. K-means) on node features (e.g. spatial position) might be an option.\n- Regarding the available frameworks, I can only speak for PyTorch-Geometric, which is the one that I'm using for my thesis and I think it's pretty solid (Matthias Fey, if you're reading this, thank you very much). It also provide an alternative implementation for the previous model under the name of MetaLayer.\n- I read in another comment that you're looking for a rotation-invariant method for the graphs. You might prefer not using the 3D coordinates at all, or embed the node distances directly in the edges, as a data preprocessing.
I hope this might help you or anyone in this competition. Good luck to everybody! :)
", "rawMarkdown": "Hi @hengck23, I remember you from a previous competition (QuickDraw). Your posts were always insightful.\n\nI'm working right now on my Master Thesis in geometric deep-learning but, alas, it won't be helpful for this competition. Still, I might give you my two cents:\n- As you can see in this [recent survey](https://arxiv.org/abs/1810.04020), most GCNs are designed for graph-level or node-level outputs. The method you proposed in another comment is actually the only way to go for this kind of tasks (predicting node-pair features and then use a standard MLP to predict the edge labels, in a 1D-Conv fashion).\n- If you want a model with edge-parameters, you might want to have a look to [this paper from a bunch of very smart people](https://arxiv.org/abs/1806.01261). They propose a very generic model that can be applied to any kind of task. They also provide an [implementation in TensorFlow and Sonnet](https://github.com/deepmind/graph_nets), if you're interested.\n- Since this is an edge regression task, using node-pooling might be problematic: you have to preserve all the edges, and state-of-the-art methods typically don't do this. You might pool the nodes, but the clusters need to form a partition of the graph, not a cover (i.e., you have to assign to each node one and only one cluster). Classical clustering methods (e.g. K-means) on node features (e.g. spatial position) might be an option.\n- Regarding the available frameworks, I can only speak for [PyTorch-Geometric](https://github.com/rusty1s/pytorch_geometric), which is the one that I'm using for my thesis and I think it's pretty solid (Matthias Fey, if you're reading this, thank you very much). It also provide an alternative implementation for the previous model under the name of [`MetaLayer`](https://rusty1s.github.io/pytorch_geometric/build/html/modules/nn.html#torch_geometric.nn.meta.MetaLayer).\n- I read in another comment that you're looking for a rotation-invariant method for the graphs. You might prefer not using the 3D coordinates at all, or embed the node distances directly in the edges, as a data preprocessing.\n\nI hope this might help you or anyone in this competition. Good luck to everybody! :)", "votes": 24 }, { "id": 561118, "postDate": "2019-06-26T06:42:20.993Z", "content": "new release in preparation .... (single graph CNN model can get LB -1.46)
\n\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/561118/13665/v3.png\")
in the next version (starter kit v3), you will learn:
\n\nhow to custom data parallel for pytorch (because of graph edge_index, we need to split and reindex the data the data ourselves before scatter to different gpu device)
how to write custom graph . Discard torch geometric nn.Messaging class use native torch.scatter_add, etc ... to write message propagation, aka. message and update function. This is because i am going to write my own graph cnn using transformer attention later and i need more customization and control)
Can anyone suggest good LSTM or RNN method or code with in-built normalization? I cannot go deep without good lstm normalization (like batch_norm) in the graph!
\n\nthanks
", "rawMarkdown": "in the next version (starter kit v3), you will learn:\n\n- how to custom data parallel for pytorch (because of graph edge_index, we need to split and reindex the data the data ourselves before scatter to different gpu device)\n\n- how to write custom graph . Discard torch geometric nn.Messaging class use native torch.scatter_add, etc ... to write message propagation, aka. message and update function. This is because i am going to write my own graph cnn using transformer attention later and i need more customization and control)\n\n### now i need kagglers' help!. \nCan anyone suggest good LSTM or RNN method or code with in-built normalization? I cannot go deep without good lstm normalization (like batch_norm) in the graph!\n\nthanks", "votes": 7 }, { "id": 562139, "postDate": "2019-06-27T02:41:30.253Z", "content": "@hengck23 \nThanks for your sharing! Have you tried IndRNN ever? In most case, it converges faster and the accuracy is better than LSTM and RNN. And also, it has batch_norm.
\n\nI usually use IndRNNv2 pytorch implementation here.\nhttps://github.com/Jyouhou/IndRNN-pytorch
", "rawMarkdown": "@hengck23 \nThanks for your sharing! Have you tried IndRNN ever? In most case, it converges faster and the accuracy is better than LSTM and RNN. And also, it has batch_norm.\n\nI usually use IndRNNv2 pytorch implementation here.\nhttps://github.com/Jyouhou/IndRNN-pytorch", "votes": 1 }, { "id": 563617, "postDate": "2019-06-28T14:01:17.190Z", "content": "\"Exploiting Edge Features in Graph Neural Networks\" a way of normalization based on graph structure
", "rawMarkdown": "\"Exploiting Edge Features in Graph Neural Networks\" a way of normalization based on graph structure" }, { "id": 564557, "postDate": "2019-06-29T15:45:26.057Z", "content": "It is released!![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F8a924c89434dd09e0ad1f46031264d7e%2Fresults-20190630.png?generation=1561823119971928&alt=media\")
Thanks for sharing. After quick scan through the code (I only looked at model.py), it seems that you are using your own implementation of Set2Set and GraphConv instead of pytorch geometric. Other than that, I do not see much changes. What do you think contribute to the improvement?
", "rawMarkdown": "Thanks for sharing. After quick scan through the code (I only looked at model.py), it seems that you are using your own implementation of Set2Set and GraphConv instead of pytorch geometric. Other than that, I do not see much changes. What do you think contribute to the improvement? ", "votes": 1 }, { "id": 564771, "postDate": "2019-06-30T00:44:11.977Z", "content": "i cross check the results. my implementation is the same as pytorch geometric version. the imprment comes from larger network and longer training time + smaller learning rate
", "rawMarkdown": "i cross check the results. my implementation is the same as pytorch geometric version. the imprment comes from larger network and longer training time + smaller learning rate", "votes": 2 }, { "id": 564828, "postDate": "2019-06-30T04:08:18.383Z", "content": "@hengck23 An easy way to parallelize the training to multiple GPUs would be using DistributedDataParallel in pytorch. It should work for your network.
", "rawMarkdown": "@hengck23 An easy way to parallelize the training to multiple GPUs would be using DistributedDataParallel in pytorch. It should work for your network.", "votes": 1 }, { "id": 565032, "postDate": "2019-06-30T10:24:59.753Z", "content": "@hengck23 , do you find that an LSTM gives you much of a performance boost over just summing the messages? Because so far my best results have been using sum. Using an LSTM gets close, but it is not better. But I agree it is probably a normalization issue, as things blow up with deeper networks.
", "rawMarkdown": "@hengck23 , do you find that an LSTM gives you much of a performance boost over just summing the messages? Because so far my best results have been using sum. Using an LSTM gets close, but it is not better. But I agree it is probably a normalization issue, as things blow up with deeper networks. ", "votes": 1 }, { "id": 580481, "postDate": "2019-07-20T08:12:30.533Z", "content": "This network get -1.4 with 13 features? or 93 features? \nI found NODE_DIM = 93 # in dataset.py and also graph.node += [acsf,] in get_item.\nI would like to know what kind of features acsf is bringing
Thank you for the nice post!
", "rawMarkdown": "Thank you for the nice post!" }, { "id": 591759, "postDate": "2019-08-04T08:04:58.090Z", "content": "@leptony I got similar results while keeping NODE_DIM = 93. I haven't yet tried changing this parameter.
![\"\"](\"https://user-images.githubusercontent.com/4609798/42087970-d47bb666-7bd2-11e8-8d75-5770ccfbdb3b.gif\")
https://github.com/pfnet-research/bayesgrad
\n\nBayesGrad: Explaining Predictions of Graph Convolutional Networks
", "rawMarkdown": "\n\nhttps://github.com/pfnet-research/bayesgrad\n\nBayesGrad: Explaining Predictions of Graph Convolutional Networks", "votes": 16, "replies": [ { "id": 544797, "postDate": "2019-06-05T23:20:38.927Z", "content": "Hi @hengck23 . I'm involved in this research, and very glad to see this thread. thank you very much for introducing it :)
\n\nI also uploaded a refactored version of saliency visualization code, please utilize.\n - blog: http://corochann.com/library-release-visualize-saliency-map-of-deep-neural-network-1478.html\n - code: https://github.com/corochann/chainer-saliency
", "rawMarkdown": "Hi @hengck23 . I'm involved in this research, and very glad to see this thread. thank you very much for introducing it :)\n\nI also uploaded a refactored version of saliency visualization code, please utilize.\n - blog: http://corochann.com/library-release-visualize-saliency-map-of-deep-neural-network-1478.html\n - code: https://github.com/corochann/chainer-saliency", "votes": 3 } ] }, { "id": 560359, "postDate": "2019-06-25T10:58:16.523Z", "content": "version.2 starter kit coming up!
\n\nNote the std of the target value and training/validation score value .... that is some trick you can exploit to get better score. I will explain later if i can confirm my hypothesis ... doing some experiments now :)
\n\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/560359/13657/starter2.png\")
hello, thanks for sharing, may I ask, what does \"from common import *\" mean, I have problems with this line
", "rawMarkdown": "hello, thanks for sharing, may I ask, what does \"from common import *\" mean, I have problems with this line", "votes": -2 } ] }, { "id": 573627, "postDate": "2019-07-12T14:20:42.793Z", "content": "@hengck23 : Come back !!
", "rawMarkdown": "@hengck23 : Come back !! \n", "votes": 7 }, { "id": 541566, "postDate": "2019-06-02T16:49:40.767Z", "content": "features![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/541566/13359/feature.png\")
example code to create above rdkit features and train using pytorch MPNN and GCN:
\n\nhttps://github.com/tencent-alchemy/Alchemy
\n\n\n\nhttps://github.com/yongqyu/MolGAN-pytorch
", "rawMarkdown": "example code to create above rdkit features and train using pytorch MPNN and GCN:\n\nhttps://github.com/tencent-alchemy/Alchemy\n\nhttps://iwatobipen.wordpress.com/2019/04/05/make-graph-convolution-model-with-geometric-deep-learning-extension-library-for-pytorch-rdkit-chemoinformatics-pytorch/\n\nhttps://github.com/yongqyu/MolGAN-pytorch", "votes": 2 } ] }, { "id": 540520, "postDate": "2019-05-31T15:44:25.867Z", "content": "my idea:
\n\n![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/540520/13343/basic.png\")
Hi @hengck23 , when you say after the Graph CNN -> *n-dim feature at index 1 *, what exactly do you mean?\nYou mean after running a Graph CNN it gives you some specific output features?Or just dimensions for features?
\n\nThanks!
", "rawMarkdown": "Hi @hengck23 , when you say after the Graph CNN -> **n-dim feature at index 1 **, what exactly do you mean?\nYou mean after running a Graph CNN it gives you some specific output features?Or just dimensions for features?\n\nThanks!" }, { "id": 541940, "postDate": "2019-06-03T08:16:08.503Z", "content": "it gives you feature at a node
", "rawMarkdown": "it gives you feature at a node " }, { "id": 542035, "postDate": "2019-06-03T11:03:17.217Z", "content": "Thanks.
\n\nAlso, I am trying to use the rdkit but since we have a csv file with coordinates and not a mol file which rdkit expects, is there a workaround for this?
", "rawMarkdown": "Thanks.\n\nAlso, I am trying to use the rdkit but since we have a csv file with coordinates and not a mol file which rdkit expects, is there a workaround for this?\n\n" }, { "id": 580078, "postDate": "2019-07-19T16:38:48.613Z", "content": "Has anyone achieved anything with this setup? My version based on chemprop doesn't achieve anything but it could be a set up problem/mistake I have made somewhere.
", "rawMarkdown": "Has anyone achieved anything with this setup? My version based on chemprop doesn't achieve anything but it could be a set up problem/mistake I have made somewhere." }, { "id": 580096, "postDate": "2019-07-19T17:07:24.167Z", "content": "Yes it works! but I haven't found a way into the -2 world with it.\nIt can also be improved even with a single gpu and a small batch.
", "rawMarkdown": "Yes it works! but I haven't found a way into the -2 world with it.\nIt can also be improved even with a single gpu and a small batch." } ] }, { "id": 570475, "postDate": "2019-07-08T10:54:21.587Z", "content": "I saw this thread after a month unfortunately, this thread has so much of cool points, visualizations and high skill level, that it should be treated as one of those videos on youtube which are titled as People are awesome and feature a guy throwing a 30 inch pizza base into the air 6 feet high at 50 revolutions per sec and catches without a glitch. Or a guy somewhere in South Asia chopping salad faster than a blink of an eye.
\n\nI mean I am amazed to see how good others are approaching this competitions I am just making line plots and scatter plots with a few bits and pieces of adding colors and thinking it looks good but then there is someone who is analyzing the structure of molecules using interactive widgets having an entire view of the dataset.
\n\nThis is great work by the author. So much to learn for me really liked the standard of work. Keep it up. 👍
", "rawMarkdown": "I saw this thread after a month unfortunately, this thread has so much of cool points, visualizations and high skill level, that it should be treated as one of those videos on youtube which are titled as **People are awesome** and feature a guy throwing a 30 inch pizza base into the air 6 feet high at 50 revolutions per sec and catches without a glitch. Or a guy somewhere in South Asia chopping salad faster than a blink of an eye.\n\nI mean I am amazed to see how good others are approaching this competitions I am just making line plots and scatter plots with a few bits and pieces of adding colors and thinking it looks good but then there is someone who is analyzing the structure of molecules using interactive widgets having an entire view of the dataset.\n\nThis is great work by the author. So much to learn for me really liked the standard of work. Keep it up. 👍 ", "votes": 5 }, { "id": 542118, "postDate": "2019-06-03T13:31:26.453Z", "content": "Excellent talk on the basics of Graph Neural Networks:
\n\nhttps://www.youtube.com/watch?v=cWIeTMklzNg
", "rawMarkdown": "Excellent talk on the basics of Graph Neural Networks:\n\nhttps://www.youtube.com/watch?v=cWIeTMklzNg", "votes": 5 }, { "id": 559256, "postDate": "2019-06-23T18:48:31.893Z", "content": "with so many variants of MPNN, NAS network architectural search for best graph cnn seems interesting:
\n\nhttps://arxiv.org/pdf/1904.09981.pdf
\n\nGraphNAS: Graph Neural Architecture Search with Reinforcement Learning
\n\n![\"\"](\"https://github.com/GraphNAS/GraphNAS-simple/raw/geometrics/images/GraphNAS.png\")
That is actually our approach :) We're not using any external features yet though
", "rawMarkdown": "That is actually our approach :) We're not using any external features yet though" } ] }, { "id": 541493, "postDate": "2019-06-02T14:42:39.823Z", "content": "some resources here: https://docs.dgl.ai/index.html
\n\nhttps://iwatobipen.wordpress.com/
\n\nhttps://iwatobipen.wordpress.com/?s=pytorch
\n\nseems useful:\nhttps://alchemy.tencent.com/
\n\n\"If you do not want to dive into RDKit, we also provide a ready-to-use pytorch dataloader which can help you easily deal with those molecules.
\n\nYou may also find a collection of baselines, including MPNN, from which you can start your journey with Alchemy!\"
", "rawMarkdown": "some resources here: https://docs.dgl.ai/index.html\n\nhttps://iwatobipen.wordpress.com/\n\nhttps://iwatobipen.wordpress.com/?s=pytorch\n\nseems useful:\nhttps://alchemy.tencent.com/\n\n\"If you do not want to dive into RDKit, we also provide a ready-to-use pytorch dataloader which can help you easily deal with those molecules.\n\nYou may also find a collection of baselines, including MPNN, from which you can start your journey with Alchemy!\"", "votes": 4 }, { "id": 545934, "postDate": "2019-06-06T04:03:27.720Z", "content": "https://github.com/Tony-Y/cgnn
\n\nThis is my PyTorch implementation of GNN for crystalline materials. It can be used for molecules, but if you want to use edge features you have to extend my code.
", "rawMarkdown": "https://github.com/Tony-Y/cgnn\n\nThis is my PyTorch implementation of GNN for crystalline materials. It can be used for molecules, but if you want to use edge features you have to extend my code.", "votes": 3 }, { "id": 565882, "postDate": "2019-07-01T13:43:17.977Z", "content": "Thanks. Didn't have enough time to learn about graph nn and really appreciate your efforts to gather all this info into one single thread.
", "rawMarkdown": "Thanks. Didn't have enough time to learn about graph nn and really appreciate your efforts to gather all this info into one single thread.", "votes": 4 }, { "id": 540550, "postDate": "2019-05-31T16:21:47.873Z", "content": "i wonder if anyone has a good method to make the results rotation-invariant (assuming that the inpute molecules can be rotated)?
", "rawMarkdown": "i wonder if anyone has a good method to make the results rotation-invariant (assuming that the inpute molecules can be rotated)? ", "votes": 4, "replies": [ { "id": 540591, "postDate": "2019-05-31T17:42:48.697Z", "content": "If you are using graphs, they are generally rotation-invariant (unless you use some kind of featurization I haven't considered).
\n\nHowever, using something like a standard CNN with voxels will NOT be rotation-invariant and is quite hard to actually achieve this. Different rotations will almost always lead to a different latent space embedding.
\n\nThere is also the PointNet approach which uses a transformer network to enforce rotation-invariance.
", "rawMarkdown": "If you are using graphs, they are generally rotation-invariant (unless you use some kind of featurization I haven't considered).\n\nHowever, using something like a standard CNN with voxels will NOT be rotation-invariant and is quite hard to actually achieve this. Different rotations will almost always lead to a different latent space embedding.\n\nThere is also the PointNet approach which uses a transformer network to enforce rotation-invariance." }, { "id": 540942, "postDate": "2019-06-01T12:07:55.500Z", "content": "You need a descriptor if you want your results to be rotationally invariant. It's a chemical representation of your molecule. The most used descriptors in computational chemistry are the Coulomb matrix (very easy to build), the SOAP and the ACSF.\nI've used the Coulomb Matrix in my kernel but it did not improve much.
", "rawMarkdown": "You need a descriptor if you want your results to be rotationally invariant. It's a chemical representation of your molecule. The most used descriptors in computational chemistry are the Coulomb matrix (very easy to build), the SOAP and the ACSF.\nI've used the Coulomb Matrix in my kernel but it did not improve much.\n", "votes": 1 }, { "id": 541153, "postDate": "2019-06-01T22:17:29.537Z", "content": "Interesting that you mentioned SOAP... These guys used a SOAP-GAP model for similar NMR predictions; https://advances.sciencemag.org/content/3/12/e1701816.full\nI haven't heard of ACSF. Do you know what the differences are?
", "rawMarkdown": "Interesting that you mentioned SOAP... These guys used a SOAP-GAP model for similar NMR predictions; https://advances.sciencemag.org/content/3/12/e1701816.full\nI haven't heard of ACSF. Do you know what the differences are?" }, { "id": 541291, "postDate": "2019-06-02T07:30:07.763Z", "content": "ACSF means atom-centered symmetry functions. Basically it describes the local environment around an atom. You define a sphere of a given radius around each atom and you \"count\" the number of atom in this sphere. The ACSFs are invariant under translation and rotations of the whole system, and independent on the ordering of the atoms. The two main type of ACSFs are two- and three-body.
\n\nSOAP is a local descriptor, that maps the local environment around a point very accurately. It eliminates rotational, and permutation redundancies by integrating the overlap of smoothed out atomic positions, by gaussian smearing, and mapping them into coefficients of orthornormal basis functions (Laplace spherical harmonics). It's more complicated than ACSF.
", "rawMarkdown": "ACSF means atom-centered symmetry functions. Basically it describes the local environment around an atom. You define a sphere of a given radius around each atom and you \"count\" the number of atom in this sphere. The ACSFs are invariant under translation and rotations of the whole system, and independent on the ordering of the atoms. The two main type of ACSFs are two- and three-body. \n\nSOAP is a local descriptor, that maps the local environment around a point very accurately. It eliminates rotational, and permutation redundancies by integrating the overlap of smoothed out atomic positions, by gaussian smearing, and mapping them into coefficients of orthornormal basis functions (Laplace spherical harmonics). It's more complicated than ACSF.", "votes": 9 }, { "id": 541294, "postDate": "2019-06-02T07:34:29.047Z", "content": "Thanks for the clear descriptions... If I can help you with any NMR queries then just ask!
", "rawMarkdown": "Thanks for the clear descriptions... If I can help you with any NMR queries then just ask!" } ] }, { "id": 564595, "postDate": "2019-06-29T17:13:15.470Z", "content": "what to train and how to ensemble? a possible solution for LB -1.6 here:
\n\n```
\n\ndef run_ensemble1():
\n\nensemble_file='/root/share/project/kaggle/2019/champs_scalar/result/separate/submit-ensemble-larger-00.csv'\ndf_test = pd.read_csv('/root/share/project/kaggle/2019/champs_scalar/data/csv/test.csv')\ndf_test = df_test.sort_values(by=['id'])\n\nid = df_test.id.values\ntype = df_test.type.values\n\nnum_test=len(id)\ncoupling_count = np.zeros(num_test)\ncoupling_value = np.zeros(num_test)\n\n['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '3JHN', '2JHH', '3JHH',]\n\n\nfor valid_type, f in [\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/ensemble/submit-00370000_model-larger--add-poola.csv'),\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all/submit/submit-00327500_model.csv'),\n\n (['3JHN'],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/3jhn-00/submit/submit-00327500_model.csv'),\n\n (['1JHC', '2JHC', '3JHC', ],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all-jhc-00/submit/submit-00262500_model.csv'),\n]:\n\n df = pd.read_csv(f)\n df = df.sort_values(by=['id'])\n for t in valid_type:\n index = np.where(type==t)[0]\n coupling_value[index] += df.scalar_coupling_constant.values[index]\n coupling_count[index] += 1\n\n\ncoupling_value = coupling_value/coupling_count\n\n\ndf = pd.DataFrame(list(zip(id, coupling_value)), columns =['id', 'scalar_coupling_constant'])\ndf.to_csv(ensemble_file,index=False)\n```
\n\nsee also the attachment train log files:
", "rawMarkdown": "what to train and how to ensemble? a possible solution for LB -1.6 here:\n\n```\n\n\ndef run_ensemble1():\n\n ensemble_file='/root/share/project/kaggle/2019/champs_scalar/result/separate/submit-ensemble-larger-00.csv'\n df_test = pd.read_csv('/root/share/project/kaggle/2019/champs_scalar/data/csv/test.csv')\n df_test = df_test.sort_values(by=['id'])\n\n id = df_test.id.values\n type = df_test.type.values\n\n num_test=len(id)\n coupling_count = np.zeros(num_test)\n coupling_value = np.zeros(num_test)\n\n ['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '3JHN', '2JHH', '3JHH',]\n\n\n for valid_type, f in [\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/ensemble/submit-00370000_model-larger--add-poola.csv'),\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all/submit/submit-00327500_model.csv'),\n\n (['3JHN'],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/3jhn-00/submit/submit-00327500_model.csv'),\n\n (['1JHC', '2JHC', '3JHC', ],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all-jhc-00/submit/submit-00262500_model.csv'),\n ]:\n\n df = pd.read_csv(f)\n df = df.sort_values(by=['id'])\n for t in valid_type:\n index = np.where(type==t)[0]\n coupling_value[index] += df.scalar_coupling_constant.values[index]\n coupling_count[index] += 1\n\n\n coupling_value = coupling_value/coupling_count\n\n\n df = pd.DataFrame(list(zip(id, coupling_value)), columns =['id', 'scalar_coupling_constant'])\n df.to_csv(ensemble_file,index=False)\n\n```\n\nsee also the attachment train log files:", "votes": 3 }, { "id": 561677, "postDate": "2019-06-26T16:33:55.313Z", "content": "self study note. rewrite graph convolution layer NNConv and set2set for easier understanding ....
\n\n```\nThe continuous kernel-based convolutional operator adapted from the “Neural Message Passing for Quantum Chemistry” paper
\n\nx′i=Θxi+∑j∈N(i)xj⋅hΘ(ei,j),
\n\nwhere hΘhΘ denotes a neural network, .i.e. a MLP.\n```
\n\n\n\n![\"\"](\"https://image.slidesharecdn.com/predictingorganicreactionoutcomeswithweisfeiler-lehmannetwork-180121081752/95/predicting-organic-reaction-outcomes-with-weisfeiler-lehman-network-18-638.jpg\")
Python-based Simulations of Chemistry Framework\nhttps://github.com/pyscf/pyscf
", "rawMarkdown": "Python-based Simulations of Chemistry Framework\nhttps://github.com/pyscf/pyscf", "votes": 2 }, { "id": 540485, "postDate": "2019-05-31T15:09:37.817Z", "content": "\"SchNet: A continuous-filter convolutional neural network for modeling quantum interactions\"
\n\n![\"\"](\"https://www.groundai.com/media/arxiv_projects/2560/figures/discrete_continuous.svg\")
SchNet is implemented in chainer-chemistry, as well as other many graph convolution models written here.
", "rawMarkdown": "SchNet is implemented in [chainer-chemistry](https://github.com/pfnet-research/chainer-chemistry/blob/master/chainer_chemistry/models/schnet.py), as well as other many graph convolution models written [here](https://github.com/pfnet-research/chainer-chemistry#supported-models).", "votes": 4 }, { "id": 601497, "postDate": "2019-08-17T17:33:39.197Z", "content": "I made an example of SchNet on Kaggle Kernel and used chainer-chemistry in it.\n- SchNet Starter Kit
https://www.kaggle.com/toshik/schnet-starter-kit
Thank you for sharing kernel. I also implemented \"WeaveNet\" example implemented in chainer-chemistry below, but your notebook performs much better! :)
\n\n", "rawMarkdown": "Thank you for sharing kernel. I also implemented \"WeaveNet\" example implemented in chainer-chemistry below, but your notebook performs much better! :)\n\n - [EDA and Graph NN baseline modeling](https://www.kaggle.com/corochann/eda-and-graph-nn-baseline-modeling)" }, { "id": 601938, "postDate": "2019-08-18T11:52:09.393Z", "content": "@toshik, thanks for sharing. Quick question. How can I save the model and continue training in a followup session using your kernel?
", "rawMarkdown": "@toshik, thanks for sharing. Quick question. How can I save the model and continue training in a followup session using your kernel?", "votes": 1 }, { "id": 602350, "postDate": "2019-08-19T01:14:08.510Z", "content": "\n\nIn chainer, you can use serializers module.
save the model\n\nfrom chainer import serializers\nserializers.save_npz('model.npz', model)\n
\nserializers.load_npz('model.npz', model)\n Reference: https://docs.chainer.org/en/stable/guides/serializers.html
", "rawMarkdown": "@sheriytm\n\nIn chainer, you can use `serializers` module.\n\n - save the model\n```\nfrom chainer import serializers\nserializers.save_npz('model.npz', model)\n```\n\n- load the model\n```\nserializers.load_npz('model.npz', model)\n``` \n\nReference: https://docs.chainer.org/en/stable/guides/serializers.html", "votes": 1 }, { "id": 602359, "postDate": "2019-08-19T01:33:40.167Z", "content": "Thanks @corochann
", "rawMarkdown": "Thanks @corochann", "votes": 1 } ] }, { "id": 561179, "postDate": "2019-06-26T07:54:50.640Z", "content": "Table I of SchNet paper shows that SchNet is better than MPNN enn-s2s on the QM9 dataset except for ZPVE, mu, and alpha. So, SchNet-like neural network models also might be better than enn-s2s models in this competition.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F224674%2F7433001e7d77492c1689763e89652c57%2F2019-06-26%2019.07.35.png?generation=1561543731573224&alt=media\")
https://github.com/atomistic-machine-learning/schnetpack
\n\nhttps://paperswithcode.com/sota/formation-energy-on-qm9
", "rawMarkdown": "Table I of [SchNet paper](https://aip.scitation.org/doi/10.1063/1.5019779) shows that SchNet is better than MPNN enn-s2s on the QM9 dataset except for ZPVE, mu, and alpha. So, SchNet-like neural network models also might be better than enn-s2s models in this competition.\n\n\n\n\nhttps://github.com/atomistic-machine-learning/schnetpack\n\nhttps://paperswithcode.com/sota/formation-energy-on-qm9", "votes": 2, "replies": [ { "id": 561694, "postDate": "2019-06-26T16:42:54.740Z", "content": "let me try to maximize the performance of MPNN first. I will try other network later (which will be better results). Till now, i still cannot overfit a set of small train data (i.e. overfit until kaggle loss is -3). Some of my training parameters/feature representation are not correct yet. Generally, my suggested approached is:
\n\nTry to over fit small train set (i.e. loss is very low, preferably close to zero. this is the lowest bound of error. Since kaggler can get to LB -2.3, i expect overfitting to give -3 to -4)
Try to generalize. See how to make sure results stays the same on validation set (and also LB test set)
Try to stabilize. Estimate shakeup and prevent difference of LB public and private set.
this seems interesting: https://arxiv.org/pdf/1806.04613.pdf
\n\nImproving Regression Performance with Distributional Losses
", "rawMarkdown": "this seems interesting: https://arxiv.org/pdf/1806.04613.pdf\n\nImproving Regression Performance with Distributional Losses", "votes": 1 }, { "id": 559723, "postDate": "2019-06-24T13:54:10.933Z", "content": "edge-based graph update:
\n\nNeural Message Passing with Edge Updates for Predicting Properties of Molecules and Materials\nhttps://arxiv.org/pdf/1806.03146.pdf
\n\ntransformer mask (for longer path dependency):
\n\nPath-Augmented Graph Transformer Network\nhttps://arxiv.org/pdf/1905.12712.pdf
\n\nothers:
\n\nDeep Neural Network Computes Electron Densities\nand Energies of a Large Set of Organic Molecules\nFaster than Density Functional Theory (DFT)\nhttps://arxiv.org/pdf/1809.02723.pdf
\n\nOperators in Machine Learning: Response Properties in Chemical Space\nhttps://arxiv.org/pdf/1807.08811.pdf
", "rawMarkdown": "edge-based graph update:\n\nNeural Message Passing with Edge Updates for Predicting Properties of Molecules and Materials\nhttps://arxiv.org/pdf/1806.03146.pdf\n\ntransformer mask (for longer path dependency):\n\nPath-Augmented Graph Transformer Network\nhttps://arxiv.org/pdf/1905.12712.pdf\n\nothers:\n\nDeep Neural Network Computes Electron Densities\nand Energies of a Large Set of Organic Molecules\nFaster than Density Functional Theory (DFT)\nhttps://arxiv.org/pdf/1809.02723.pdf\n\nOperators in Machine Learning: Response Properties in Chemical Space\nhttps://arxiv.org/pdf/1807.08811.pdf", "votes": 1 }, { "id": 572418, "postDate": "2019-07-10T23:12:55.310Z", "content": "noobie question, @hengck23 where can I find those starter kit?\nThanks for this amazing job! :)
", "rawMarkdown": "noobie question, @hengck23 where can I find those starter kit?\nThanks for this amazing job! :)", "votes": -2 }, { "id": 540779, "postDate": "2019-06-01T04:04:54.590Z", "content": "You took all the word right out of my mouth. Maybe we can team up! I can skillfully use PyG and DGL
", "rawMarkdown": "You took all the word right out of my mouth. Maybe we can team up! I can skillfully use PyG and DGL", "votes": -4 }, { "id": 591288, "postDate": "2019-08-03T12:57:33.553Z", "content": "Hi Heng,
\n\nThanks for sharing your code. I saw in your log.train.txt file, the loss kept decreasing until around -2.5. But when I trained a model using your code (the 20190630 version), I found the loss started fluctuating at around -1.2 after around 30 epochs. Did I miss anything? Below is the loss:
\n\nlr=0.00050
\n\n** start training here! **\n batch_size =12, iter_accum=1\n |--------------- VALID ----------------------------------------------------------------|-- TRAIN/BATCH ---------\n |std 18.3 4.5 3.1 10.9 3.7 1.3 4.0 3.7 | | |
\n\n0.00000 167.5* 33.5 | -0.645, -1.449, -1.187, -0.693, -1.700, -1.749, -1.649, -1.556 | -1.231 0.29 -1.33 | +0.000 | 0 hr 00 min\n0.00050 168.0 33.6 | -0.664, -1.431, -1.175, -0.735, -1.708, -1.732, -1.676, -1.542 | -1.229 0.29 -1.33 | -1.197 | 0 hr 02 min\n0.00050 168.5 33.6 | -0.671, -1.420, -1.083, -0.737, -1.632, -1.713, -1.597, -1.544 | -1.190 0.30 -1.30 | -1.225 | 0 hr 04 min\n0.00050 169.0 33.7 | -0.460, -1.331, -1.068, -0.634, -1.558, -1.620, -1.163, -1.446 | -1.068 0.34 -1.16 | -1.149 | 0 hr 06 min
", "rawMarkdown": "Hi Heng, \n\nThanks for sharing your code. I saw in your log.train.txt file, the loss kept decreasing until around -2.5. But when I trained a model using your code (the 20190630 version), I found the loss started fluctuating at around -1.2 after around 30 epochs. Did I miss anything? Below is the loss:\n\nlr=0.00050\n\n** start training here! **\n batch_size =12, iter_accum=1\n |--------------- VALID ----------------------------------------------------------------|-- TRAIN/BATCH ---------\n |std 18.3 4.5 3.1 10.9 3.7 1.3 4.0 3.7 | | |\nrate iter epoch | 1JHC, 2JHC, 3JHC, 1JHN, 2JHN, 3JHN, 2JHH, 3JHH | loss mae log_mae | loss | time\n--------------------------------------------------------------------------------------------------------------------------------------\n0.00000 167.5* 33.5 | -0.645, -1.449, -1.187, -0.693, -1.700, -1.749, -1.649, -1.556 | -1.231 0.29 -1.33 | +0.000 | 0 hr 00 min\n0.00050 168.0 33.6 | -0.664, -1.431, -1.175, -0.735, -1.708, -1.732, -1.676, -1.542 | -1.229 0.29 -1.33 | -1.197 | 0 hr 02 min\n0.00050 168.5 33.6 | -0.671, -1.420, -1.083, -0.737, -1.632, -1.713, -1.597, -1.544 | -1.190 0.30 -1.30 | -1.225 | 0 hr 04 min\n0.00050 169.0 33.7 | -0.460, -1.331, -1.068, -0.634, -1.558, -1.620, -1.163, -1.446 | -1.068 0.34 -1.16 | -1.149 | 0 hr 06 min\n" }, { "id": 589498, "postDate": "2019-08-01T01:47:02.343Z", "content": "Thank you so much for sharing. Including this model to our stack improves the score to -1.7x
", "rawMarkdown": "Thank you so much for sharing. Including this model to our stack improves the score to -1.7x" }, { "id": 589329, "postDate": "2019-07-31T18:52:39.180Z", "content": "good work!
", "rawMarkdown": "good work!" }, { "id": 579000, "postDate": "2019-07-18T11:51:42.897Z", "content": "Hi @hengck23 . \nI am recently trying to benchmark your code, and I have a quick question about the ChampsDataset \nWhen you make a batch from this function, i checked you used offset. Those offsets are necessary for your GNN model? I thought it should work without offset term.
python model.py ( run_check_net ) throws Segmentation fault error. Does anyone has the same problem?
\n\nI'm using pytorch 1.1.0 CUDA10 on AWS p2.xlarge instance.
\n\nYou are like a god and I'm like a useless student😭 .
", "rawMarkdown": "python model.py ( run_check_net ) throws Segmentation fault error. Does anyone has the same problem?\n\nI'm using pytorch 1.1.0 CUDA10 on AWS p2.xlarge instance.\n\nYou are like a god and I'm like a useless student😭 . \n\n", "replies": [ { "id": 578910, "postDate": "2019-07-18T09:29:08.323Z", "content": "I meet this problem a week ago. I think your torch_geometric library install error. Please install torch==1.1 and install torch_geometric follow pytorch_geometric
", "rawMarkdown": "I meet this problem a week ago. I think your torch_geometric library install error. Please install torch==1.1 and install torch_geometric follow [pytorch_geometric](https://github.com/rusty1s/pytorch_geometric)", "votes": 1 }, { "id": 578926, "postDate": "2019-07-18T09:46:50.137Z", "content": "Thank you! I try!
", "rawMarkdown": "Thank you! I try! " } ] }, { "id": 569357, "postDate": "2019-07-06T14:06:47.900Z", "content": "How do you know the graph structure at test time?
", "rawMarkdown": "How do you know the graph structure at test time? " }, { "id": 565433, "postDate": "2019-07-01T00:22:22.903Z", "content": "Hi Heng, thanks for sharing. I run into an error when converting molecules to graphs. The error is related to this line in the following function: \"for i,j in UA_pairs\". The UA_pairs is a list of integers. The error is 'int is not iterable. Did you run into this? BTW, the function is in data.py file.
\n\nWhen it failed on the molecule id, dsgdb9nsd_000023, the UA_pairs is a list which is [2, 1, 3, 0].
\n\ndef get_BO(AC,UA,DU,valences,UA_pairs,quick):\n BO = AC.copy()\n DU_save = []\n while DU_save != DU:\n for i,j in UA_pairs:\n BO[i,j] += 1\n BO[j,i] += 1
", "rawMarkdown": "Hi Heng, thanks for sharing. I run into an error when converting molecules to graphs. The error is related to this line in the following function: \"for i,j in UA_pairs\". The UA_pairs is a list of integers. The error is 'int is not iterable. Did you run into this? BTW, the function is in data.py file.\n\nWhen it failed on the molecule id, dsgdb9nsd_000023, the UA_pairs is a list which is [2, 1, 3, 0].\n\ndef get_BO(AC,UA,DU,valences,UA_pairs,quick):\n BO = AC.copy()\n DU_save = []\n while DU_save != DU:\n for i,j in UA_pairs:\n BO[i,j] += 1\n BO[j,i] += 1", "replies": [ { "id": 565614, "postDate": "2019-07-01T07:01:18.150Z", "content": "You need to update the networkx library. Hope this help 🤘
", "rawMarkdown": "You need to update the networkx library. Hope this help 🤘" }, { "id": 567072, "postDate": "2019-07-03T02:46:45.903Z", "content": "Thanks. It is the reason. Now I updated networkx to the verion 2.3 and it now works. Previously I had verion 1.9.1.
", "rawMarkdown": "Thanks. It is the reason. Now I updated networkx to the verion 2.3 and it now works. Previously I had verion 1.9.1." } ] }, { "id": 564661, "postDate": "2019-06-29T18:49:44.027Z", "content": "@hengck23 : thank for sharing the model. My training seems to be extremely slow compared to yours (>24h) and I haven't reach 40 epochs. May I ask what system / card you are using ?
", "rawMarkdown": "@hengck23 : thank for sharing the model. My training seems to be extremely slow compared to yours (>24h) and I haven't reach 40 epochs. May I ask what system / card you are using ?", "replies": [ { "id": 564917, "postDate": "2019-06-30T06:55:47.280Z", "content": "i am using 1080ti (4x gpu machine, i.e. train 4 models are one time)
", "rawMarkdown": "i am using 1080ti (4x gpu machine, i.e. train 4 models are one time)", "votes": 5 }, { "id": 565599, "postDate": "2019-07-01T06:45:11.597Z", "content": "@hengck23 : Thank you for a very interesting implementation. Unfortunately it hangs in my computational setup. software: Ubuntu + Cuda 10 + pytorch 1.1.0 hardware: 2xNVIDIA 1080. The situation is so severe that the server crashes and reboots every time. Does anybody else faced the same issue? The first 5000 iterations (no weight update) run normally but then it hangs after performing successfully some \n\n optimizer.step()\n optimizer.zero_grad() \n\nsteps. I also tried to reduce batch size but the issue remains. Thank you!!!
Hi, does anyone have any idea how to predict various numbers of coupling from a single molecular?\nEX: it remind me to predict bboxes in image detection, that every image have different numbers of bbox.
", "rawMarkdown": "Hi, does anyone have any idea how to predict various numbers of coupling from a single molecular?\nEX: it remind me to predict bboxes in image detection, that every image have different numbers of bbox.", "replies": [ { "id": 543963, "postDate": "2019-06-05T01:27:41.453Z", "content": "extract features of all node (atoms) in graph. Make all possible pairs. some pair will be valid bond (with given ground truth of j-coupling nJab), others will be just background or ignored
", "rawMarkdown": "extract features of all node (atoms) in graph. Make all possible pairs. some pair will be valid bond (with given ground truth of j-coupling nJab), others will be just background or ignored ", "votes": 1 }, { "id": 544173, "postDate": "2019-06-05T08:28:59.483Z", "content": "I see, Thanks Heng
", "rawMarkdown": "I see, Thanks Heng" } ] }, { "id": 540493, "postDate": "2019-05-31T15:22:33.843Z", "content": "I expect this is where some of the best models (at least initially) will come from. MPNN's have shown the best results in quantum prediction tasks. DeepChem has implementations of several GraphCNN's and already has the featurization from SMILES/SDF files. This is where I am starting. There are some necessary changes to be made to the code though.
", "rawMarkdown": "I expect this is where some of the best models (at least initially) will come from. MPNN's have shown the best results in quantum prediction tasks. DeepChem has implementations of several GraphCNN's and already has the featurization from SMILES/SDF files. This is where I am starting. There are some necessary changes to be made to the code though.", "replies": [ { "id": 541573, "postDate": "2019-06-02T16:59:38.720Z", "content": "KRR and Gaussian processes also work very well in computational chemistry/physics but we may have too many data for that, it will explode the memory
", "rawMarkdown": "KRR and Gaussian processes also work very well in computational chemistry/physics but we may have too many data for that, it will explode the memory", "votes": 1 } ] }, { "id": 581247, "postDate": "2019-07-21T16:44:04.473Z", "rawMarkdown": "", "isDeleted": true }, { "id": 540527, "postDate": "2019-05-31T15:52:19.317Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 602628, "postDate": "2019-08-19T09:33:04.307Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": 1 }, { "id": 559979, "postDate": "2019-06-24T23:00:36.353Z", "content": "thank you for sharing all of it
", "rawMarkdown": "thank you for sharing all of it", "votes": 1 }, { "id": 590571, "postDate": "2019-08-02T10:54:35.997Z", "content": "Thank you very much for sharing these.
", "rawMarkdown": "Thank you very much for sharing these.\n" }, { "id": 565003, "postDate": "2019-06-30T09:38:55.627Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing " } ], "comments": [ { "id": 558553, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-22T14:54:19.893000", "content": "starter-kit is up! (train for about 160k iterations/60 epoch, you can get LB of about -0.725. Note that Google paper uses about 530 epoch in their QM9 paper and train individual separate predictors, one for each target. I am not sure how many epochs are required in this kaggle challenge, so have fun in experimenting!)
\n\n
@hengck23 thank you for sharing this.
\n\nWhat are the train_split_by_mol.80003.npy and debug_split_by_mol.1000.npy files? They are referenced but don't seem to be part of the zip.
", "votes": 1, "replies": [] }, { "id": 559524, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-24T08:13:49.933000", "content": "they are just random split. you can create using the code:
\n\n```\ndef run_make_split():
\n\nsplit_dir = '/root/share/project/kaggle/2019/champs_scalar/data/split'\n\ncsv_file = '/root/share/project/kaggle/2019/champs_scalar/data/csv/train.csv'\ndf = pd.read_csv(csv_file)\nmolecule_names = df.molecule_name.unique()\nmolecule_names = np.sort(molecule_names)\n\ndebug_split = molecule_names[:1000]\nnp.save(split_dir + '/debug_split_by_mol.%d.npy'%len(debug_split), debug_split)\nprint(debug_split) #'dsgdb9nsd_001679'\n\nnp.random.shuffle(molecule_names)\nif 1: #just random\n\n num_all = len(molecule_names)\n num_valid = 5000\n num_train = num_all - num_valid\n train_split = molecule_names[num_valid:]\n valid_split = molecule_names[:num_valid]\n\n np.save(split_dir + '/train_split_by_mol.%d.npy'%num_train,train_split)\n np.save(split_dir + '/valid_split_by_mol.%d.npy'%num_valid,valid_split)\n\n pass\n```
", "votes": 2, "replies": [] }, { "id": 559838, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-24T16:49:41.057000", "content": "you can accelerated your training using the new loss
\n\n```
\n\ndef criterion(predict, coupling_value):\n predict = predict.view(-1)\n coupling_value = coupling_value.view(-1)\n assert(predict.shape==coupling_value.shape)
\n\nloss = torch.abs(predict-coupling_value)\nloss = loss.mean()\nloss = torch.log(loss)\nreturn loss\n```
\n\n
Hi Heng, maybe there is no graph.py file as shown in the picture.
", "votes": 0, "replies": [] }, { "id": 575222, "author_name": "PJTucker", "author_url": "", "post_date": "2019-07-15T07:24:46.643000", "content": "Oh. I find it in data.py. Maybe you have changed something. \nI have read your data.py carefully and find this project so cool. \nIt is really amazing to be done by yourself. Thanks for sharing!
", "votes": 0, "replies": [] } ] }, { "id": 546673, "author_name": "Francesco Landolfi", "author_url": "", "post_date": "2019-06-06T19:54:24.747000", "content": "Hi @hengck23, I remember you from a previous competition (QuickDraw). Your posts were always insightful.
\n\nI'm working right now on my Master Thesis in geometric deep-learning but, alas, it won't be helpful for this competition. Still, I might give you my two cents:\n- As you can see in this recent survey, most GCNs are designed for graph-level or node-level outputs. The method you proposed in another comment is actually the only way to go for this kind of tasks (predicting node-pair features and then use a standard MLP to predict the edge labels, in a 1D-Conv fashion).\n- If you want a model with edge-parameters, you might want to have a look to this paper from a bunch of very smart people. They propose a very generic model that can be applied to any kind of task. They also provide an implementation in TensorFlow and Sonnet, if you're interested.\n- Since this is an edge regression task, using node-pooling might be problematic: you have to preserve all the edges, and state-of-the-art methods typically don't do this. You might pool the nodes, but the clusters need to form a partition of the graph, not a cover (i.e., you have to assign to each node one and only one cluster). Classical clustering methods (e.g. K-means) on node features (e.g. spatial position) might be an option.\n- Regarding the available frameworks, I can only speak for PyTorch-Geometric, which is the one that I'm using for my thesis and I think it's pretty solid (Matthias Fey, if you're reading this, thank you very much). It also provide an alternative implementation for the previous model under the name of MetaLayer.\n- I read in another comment that you're looking for a rotation-invariant method for the graphs. You might prefer not using the 3D coordinates at all, or embed the node distances directly in the edges, as a data preprocessing.
I hope this might help you or anyone in this competition. Good luck to everybody! :)
", "votes": 24, "replies": [] }, { "id": 561118, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-26T06:42:20.993000", "content": "new release in preparation .... (single graph CNN model can get LB -1.46)
\n\n
in the next version (starter kit v3), you will learn:
\n\nhow to custom data parallel for pytorch (because of graph edge_index, we need to split and reindex the data the data ourselves before scatter to different gpu device)
how to write custom graph . Discard torch geometric nn.Messaging class use native torch.scatter_add, etc ... to write message propagation, aka. message and update function. This is because i am going to write my own graph cnn using transformer attention later and i need more customization and control)
Can anyone suggest good LSTM or RNN method or code with in-built normalization? I cannot go deep without good lstm normalization (like batch_norm) in the graph!
\n\nthanks
", "votes": 7, "replies": [] }, { "id": 562139, "author_name": "AkiraSosa", "author_url": "", "post_date": "2019-06-27T02:41:30.253000", "content": "@hengck23 \nThanks for your sharing! Have you tried IndRNN ever? In most case, it converges faster and the accuracy is better than LSTM and RNN. And also, it has batch_norm.
\n\nI usually use IndRNNv2 pytorch implementation here.\nhttps://github.com/Jyouhou/IndRNN-pytorch
", "votes": 1, "replies": [] }, { "id": 563617, "author_name": "Milkshakespeare", "author_url": "", "post_date": "2019-06-28T14:01:17.190000", "content": "\"Exploiting Edge Features in Graph Neural Networks\" a way of normalization based on graph structure
", "votes": 0, "replies": [] }, { "id": 564557, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-29T15:45:26.057000", "content": "It is released!
Thanks for sharing. After quick scan through the code (I only looked at model.py), it seems that you are using your own implementation of Set2Set and GraphConv instead of pytorch geometric. Other than that, I do not see much changes. What do you think contribute to the improvement?
", "votes": 1, "replies": [] }, { "id": 564771, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-30T00:44:11.977000", "content": "i cross check the results. my implementation is the same as pytorch geometric version. the imprment comes from larger network and longer training time + smaller learning rate
", "votes": 2, "replies": [] }, { "id": 564828, "author_name": "Rand Xie", "author_url": "", "post_date": "2019-06-30T04:08:18.383000", "content": "@hengck23 An easy way to parallelize the training to multiple GPUs would be using DistributedDataParallel in pytorch. It should work for your network.
", "votes": 1, "replies": [] }, { "id": 565032, "author_name": "fnands", "author_url": "", "post_date": "2019-06-30T10:24:59.753000", "content": "@hengck23 , do you find that an LSTM gives you much of a performance boost over just summing the messages? Because so far my best results have been using sum. Using an LSTM gets close, but it is not better. But I agree it is probably a normalization issue, as things blow up with deeper networks.
", "votes": 1, "replies": [] }, { "id": 580481, "author_name": "Beomki Yeo", "author_url": "", "post_date": "2019-07-20T08:12:30.533000", "content": "This network get -1.4 with 13 features? or 93 features? \nI found NODE_DIM = 93 # in dataset.py and also graph.node += [acsf,] in get_item.\nI would like to know what kind of features acsf is bringing
Thank you for the nice post!
", "votes": 0, "replies": [] }, { "id": 591759, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2019-08-04T08:04:58.090000", "content": "@leptony I got similar results while keeping NODE_DIM = 93. I haven't yet tried changing this parameter.
https://github.com/pfnet-research/bayesgrad
\n\nBayesGrad: Explaining Predictions of Graph Convolutional Networks
", "votes": 16, "replies": [ { "id": 544797, "author_name": "corochann", "author_url": "", "post_date": "2019-06-05T23:20:38.927000", "content": "Hi @hengck23 . I'm involved in this research, and very glad to see this thread. thank you very much for introducing it :)
\n\nI also uploaded a refactored version of saliency visualization code, please utilize.\n - blog: http://corochann.com/library-release-visualize-saliency-map-of-deep-neural-network-1478.html\n - code: https://github.com/corochann/chainer-saliency
", "votes": 3, "replies": [] } ] }, { "id": 560359, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-25T10:58:16.523000", "content": "version.2 starter kit coming up!
\n\nNote the std of the target value and training/validation score value .... that is some trick you can exploit to get better score. I will explain later if i can confirm my hypothesis ... doing some experiments now :)
\n\n
hello, thanks for sharing, may I ask, what does \"from common import *\" mean, I have problems with this line
", "votes": -2, "replies": [] } ] }, { "id": 573627, "author_name": "eagle4", "author_url": "", "post_date": "2019-07-12T14:20:42.793000", "content": "@hengck23 : Come back !!
", "votes": 7, "replies": [] }, { "id": 541566, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-02T16:49:40.767000", "content": "features
example code to create above rdkit features and train using pytorch MPNN and GCN:
\n\nhttps://github.com/tencent-alchemy/Alchemy
\n\n\n\nhttps://github.com/yongqyu/MolGAN-pytorch
", "votes": 2, "replies": [] } ] }, { "id": 540520, "author_name": "hengck23", "author_url": "", "post_date": "2019-05-31T15:44:25.867000", "content": "my idea:
\n\n
Hi @hengck23 , when you say after the Graph CNN -> *n-dim feature at index 1 *, what exactly do you mean?\nYou mean after running a Graph CNN it gives you some specific output features?Or just dimensions for features?
\n\nThanks!
", "votes": 0, "replies": [] }, { "id": 541940, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-03T08:16:08.503000", "content": "it gives you feature at a node
", "votes": 0, "replies": [] }, { "id": 542035, "author_name": "George", "author_url": "", "post_date": "2019-06-03T11:03:17.217000", "content": "Thanks.
\n\nAlso, I am trying to use the rdkit but since we have a csv file with coordinates and not a mol file which rdkit expects, is there a workaround for this?
", "votes": 0, "replies": [] }, { "id": 580078, "author_name": "jameschapman19", "author_url": "", "post_date": "2019-07-19T16:38:48.613000", "content": "Has anyone achieved anything with this setup? My version based on chemprop doesn't achieve anything but it could be a set up problem/mistake I have made somewhere.
", "votes": 0, "replies": [] }, { "id": 580096, "author_name": "eagle4", "author_url": "", "post_date": "2019-07-19T17:07:24.167000", "content": "Yes it works! but I haven't found a way into the -2 world with it.\nIt can also be improved even with a single gpu and a small batch.
", "votes": 0, "replies": [] } ] }, { "id": 570475, "author_name": "cyberia", "author_url": "", "post_date": "2019-07-08T10:54:21.587000", "content": "I saw this thread after a month unfortunately, this thread has so much of cool points, visualizations and high skill level, that it should be treated as one of those videos on youtube which are titled as People are awesome and feature a guy throwing a 30 inch pizza base into the air 6 feet high at 50 revolutions per sec and catches without a glitch. Or a guy somewhere in South Asia chopping salad faster than a blink of an eye.
\n\nI mean I am amazed to see how good others are approaching this competitions I am just making line plots and scatter plots with a few bits and pieces of adding colors and thinking it looks good but then there is someone who is analyzing the structure of molecules using interactive widgets having an entire view of the dataset.
\n\nThis is great work by the author. So much to learn for me really liked the standard of work. Keep it up. 👍
", "votes": 5, "replies": [] }, { "id": 542118, "author_name": "Aaron Koch", "author_url": "", "post_date": "2019-06-03T13:31:26.453000", "content": "Excellent talk on the basics of Graph Neural Networks:
\n\nhttps://www.youtube.com/watch?v=cWIeTMklzNg
", "votes": 5, "replies": [] }, { "id": 559256, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-23T18:48:31.893000", "content": "with so many variants of MPNN, NAS network architectural search for best graph cnn seems interesting:
\n\nhttps://arxiv.org/pdf/1904.09981.pdf
\n\nGraphNAS: Graph Neural Architecture Search with Reinforcement Learning
\n\n
That is actually our approach :) We're not using any external features yet though
", "votes": 0, "replies": [] } ] }, { "id": 541493, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-02T14:42:39.823000", "content": "some resources here: https://docs.dgl.ai/index.html
\n\nhttps://iwatobipen.wordpress.com/
\n\nhttps://iwatobipen.wordpress.com/?s=pytorch
\n\nseems useful:\nhttps://alchemy.tencent.com/
\n\n\"If you do not want to dive into RDKit, we also provide a ready-to-use pytorch dataloader which can help you easily deal with those molecules.
\n\nYou may also find a collection of baselines, including MPNN, from which you can start your journey with Alchemy!\"
", "votes": 4, "replies": [] }, { "id": 545934, "author_name": "Tony Y.", "author_url": "", "post_date": "2019-06-06T04:03:27.720000", "content": "https://github.com/Tony-Y/cgnn
\n\nThis is my PyTorch implementation of GNN for crystalline materials. It can be used for molecules, but if you want to use edge features you have to extend my code.
", "votes": 3, "replies": [] }, { "id": 565882, "author_name": "WispZero", "author_url": "", "post_date": "2019-07-01T13:43:17.977000", "content": "Thanks. Didn't have enough time to learn about graph nn and really appreciate your efforts to gather all this info into one single thread.
", "votes": 4, "replies": [] }, { "id": 540550, "author_name": "hengck23", "author_url": "", "post_date": "2019-05-31T16:21:47.873000", "content": "i wonder if anyone has a good method to make the results rotation-invariant (assuming that the inpute molecules can be rotated)?
", "votes": 4, "replies": [ { "id": 540591, "author_name": "Matt", "author_url": "", "post_date": "2019-05-31T17:42:48.697000", "content": "If you are using graphs, they are generally rotation-invariant (unless you use some kind of featurization I haven't considered).
\n\nHowever, using something like a standard CNN with voxels will NOT be rotation-invariant and is quite hard to actually achieve this. Different rotations will almost always lead to a different latent space embedding.
\n\nThere is also the PointNet approach which uses a transformer network to enforce rotation-invariance.
", "votes": 0, "replies": [] }, { "id": 540942, "author_name": "Boris D.", "author_url": "", "post_date": "2019-06-01T12:07:55.500000", "content": "You need a descriptor if you want your results to be rotationally invariant. It's a chemical representation of your molecule. The most used descriptors in computational chemistry are the Coulomb matrix (very easy to build), the SOAP and the ACSF.\nI've used the Coulomb Matrix in my kernel but it did not improve much.
", "votes": 1, "replies": [] }, { "id": 541153, "author_name": "phucco", "author_url": "", "post_date": "2019-06-01T22:17:29.537000", "content": "Interesting that you mentioned SOAP... These guys used a SOAP-GAP model for similar NMR predictions; https://advances.sciencemag.org/content/3/12/e1701816.full\nI haven't heard of ACSF. Do you know what the differences are?
", "votes": 0, "replies": [] }, { "id": 541291, "author_name": "Boris D.", "author_url": "", "post_date": "2019-06-02T07:30:07.763000", "content": "ACSF means atom-centered symmetry functions. Basically it describes the local environment around an atom. You define a sphere of a given radius around each atom and you \"count\" the number of atom in this sphere. The ACSFs are invariant under translation and rotations of the whole system, and independent on the ordering of the atoms. The two main type of ACSFs are two- and three-body.
\n\nSOAP is a local descriptor, that maps the local environment around a point very accurately. It eliminates rotational, and permutation redundancies by integrating the overlap of smoothed out atomic positions, by gaussian smearing, and mapping them into coefficients of orthornormal basis functions (Laplace spherical harmonics). It's more complicated than ACSF.
", "votes": 9, "replies": [] }, { "id": 541294, "author_name": "phucco", "author_url": "", "post_date": "2019-06-02T07:34:29.047000", "content": "Thanks for the clear descriptions... If I can help you with any NMR queries then just ask!
", "votes": 0, "replies": [] } ] }, { "id": 564595, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-29T17:13:15.470000", "content": "what to train and how to ensemble? a possible solution for LB -1.6 here:
\n\n```
\n\ndef run_ensemble1():
\n\nensemble_file='/root/share/project/kaggle/2019/champs_scalar/result/separate/submit-ensemble-larger-00.csv'\ndf_test = pd.read_csv('/root/share/project/kaggle/2019/champs_scalar/data/csv/test.csv')\ndf_test = df_test.sort_values(by=['id'])\n\nid = df_test.id.values\ntype = df_test.type.values\n\nnum_test=len(id)\ncoupling_count = np.zeros(num_test)\ncoupling_value = np.zeros(num_test)\n\n['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '3JHN', '2JHH', '3JHH',]\n\n\nfor valid_type, f in [\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/ensemble/submit-00370000_model-larger--add-poola.csv'),\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all/submit/submit-00327500_model.csv'),\n\n (['3JHN'],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/3jhn-00/submit/submit-00327500_model.csv'),\n\n (['1JHC', '2JHC', '3JHC', ],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all-jhc-00/submit/submit-00262500_model.csv'),\n]:\n\n df = pd.read_csv(f)\n df = df.sort_values(by=['id'])\n for t in valid_type:\n index = np.where(type==t)[0]\n coupling_value[index] += df.scalar_coupling_constant.values[index]\n coupling_count[index] += 1\n\n\ncoupling_value = coupling_value/coupling_count\n\n\ndf = pd.DataFrame(list(zip(id, coupling_value)), columns =['id', 'scalar_coupling_constant'])\ndf.to_csv(ensemble_file,index=False)\n```
\n\nsee also the attachment train log files:
", "votes": 3, "replies": [] }, { "id": 561677, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-26T16:33:55.313000", "content": "self study note. rewrite graph convolution layer NNConv and set2set for easier understanding ....
\n\n```\nThe continuous kernel-based convolutional operator adapted from the “Neural Message Passing for Quantum Chemistry” paper
\n\nx′i=Θxi+∑j∈N(i)xj⋅hΘ(ei,j),
\n\nwhere hΘhΘ denotes a neural network, .i.e. a MLP.\n```
\n\n\n\n
Python-based Simulations of Chemistry Framework\nhttps://github.com/pyscf/pyscf
", "votes": 2, "replies": [] }, { "id": 540485, "author_name": "hengck23", "author_url": "", "post_date": "2019-05-31T15:09:37.817000", "content": "\"SchNet: A continuous-filter convolutional neural network for modeling quantum interactions\"
\n\n
SchNet is implemented in chainer-chemistry, as well as other many graph convolution models written here.
", "votes": 4, "replies": [] }, { "id": 601497, "author_name": "toshi_k", "author_url": "", "post_date": "2019-08-17T17:33:39.197000", "content": "I made an example of SchNet on Kaggle Kernel and used chainer-chemistry in it.\n- SchNet Starter Kit
https://www.kaggle.com/toshik/schnet-starter-kit
Thank you for sharing kernel. I also implemented \"WeaveNet\" example implemented in chainer-chemistry below, but your notebook performs much better! :)
\n\n", "votes": 0, "replies": [] }, { "id": 601938, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-08-18T11:52:09.393000", "content": "@toshik, thanks for sharing. Quick question. How can I save the model and continue training in a followup session using your kernel?
", "votes": 1, "replies": [] }, { "id": 602350, "author_name": "corochann", "author_url": "", "post_date": "2019-08-19T01:14:08.510000", "content": "\n\nIn chainer, you can use serializers module.
save the model\n\nfrom chainer import serializers\nserializers.save_npz('model.npz', model)\n
\nserializers.load_npz('model.npz', model)\n Reference: https://docs.chainer.org/en/stable/guides/serializers.html
", "votes": 1, "replies": [] }, { "id": 602359, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-08-19T01:33:40.167000", "content": "Thanks @corochann
", "votes": 1, "replies": [] } ] }, { "id": 561179, "author_name": "Tony Y.", "author_url": "", "post_date": "2019-06-26T07:54:50.640000", "content": "Table I of SchNet paper shows that SchNet is better than MPNN enn-s2s on the QM9 dataset except for ZPVE, mu, and alpha. So, SchNet-like neural network models also might be better than enn-s2s models in this competition.
\n\n
https://github.com/atomistic-machine-learning/schnetpack
\n\nhttps://paperswithcode.com/sota/formation-energy-on-qm9
", "votes": 2, "replies": [ { "id": 561694, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-26T16:42:54.740000", "content": "let me try to maximize the performance of MPNN first. I will try other network later (which will be better results). Till now, i still cannot overfit a set of small train data (i.e. overfit until kaggle loss is -3). Some of my training parameters/feature representation are not correct yet. Generally, my suggested approached is:
\n\nTry to over fit small train set (i.e. loss is very low, preferably close to zero. this is the lowest bound of error. Since kaggler can get to LB -2.3, i expect overfitting to give -3 to -4)
Try to generalize. See how to make sure results stays the same on validation set (and also LB test set)
Try to stabilize. Estimate shakeup and prevent difference of LB public and private set.
this seems interesting: https://arxiv.org/pdf/1806.04613.pdf
\n\nImproving Regression Performance with Distributional Losses
", "votes": 1, "replies": [] }, { "id": 559723, "author_name": "hengck23", "author_url": "", "post_date": "2019-06-24T13:54:10.933000", "content": "edge-based graph update:
\n\nNeural Message Passing with Edge Updates for Predicting Properties of Molecules and Materials\nhttps://arxiv.org/pdf/1806.03146.pdf
\n\ntransformer mask (for longer path dependency):
\n\nPath-Augmented Graph Transformer Network\nhttps://arxiv.org/pdf/1905.12712.pdf
\n\nothers:
\n\nDeep Neural Network Computes Electron Densities\nand Energies of a Large Set of Organic Molecules\nFaster than Density Functional Theory (DFT)\nhttps://arxiv.org/pdf/1809.02723.pdf
\n\nOperators in Machine Learning: Response Properties in Chemical Space\nhttps://arxiv.org/pdf/1807.08811.pdf
", "votes": 1, "replies": [] }, { "id": 572418, "author_name": "Nanashi", "author_url": "", "post_date": "2019-07-10T23:12:55.310000", "content": "noobie question, @hengck23 where can I find those starter kit?\nThanks for this amazing job! :)
", "votes": -2, "replies": [] }, { "id": 540779, "author_name": "Milkshakespeare", "author_url": "", "post_date": "2019-06-01T04:04:54.590000", "content": "You took all the word right out of my mouth. Maybe we can team up! I can skillfully use PyG and DGL
", "votes": -4, "replies": [] }, { "id": 591288, "author_name": "XiaokangWang", "author_url": "", "post_date": "2019-08-03T12:57:33.553000", "content": "Hi Heng,
\n\nThanks for sharing your code. I saw in your log.train.txt file, the loss kept decreasing until around -2.5. But when I trained a model using your code (the 20190630 version), I found the loss started fluctuating at around -1.2 after around 30 epochs. Did I miss anything? Below is the loss:
\n\nlr=0.00050
\n\n** start training here! **\n batch_size =12, iter_accum=1\n |--------------- VALID ----------------------------------------------------------------|-- TRAIN/BATCH ---------\n |std 18.3 4.5 3.1 10.9 3.7 1.3 4.0 3.7 | | |
\n\n0.00000 167.5* 33.5 | -0.645, -1.449, -1.187, -0.693, -1.700, -1.749, -1.649, -1.556 | -1.231 0.29 -1.33 | +0.000 | 0 hr 00 min\n0.00050 168.0 33.6 | -0.664, -1.431, -1.175, -0.735, -1.708, -1.732, -1.676, -1.542 | -1.229 0.29 -1.33 | -1.197 | 0 hr 02 min\n0.00050 168.5 33.6 | -0.671, -1.420, -1.083, -0.737, -1.632, -1.713, -1.597, -1.544 | -1.190 0.30 -1.30 | -1.225 | 0 hr 04 min\n0.00050 169.0 33.7 | -0.460, -1.331, -1.068, -0.634, -1.558, -1.620, -1.163, -1.446 | -1.068 0.34 -1.16 | -1.149 | 0 hr 06 min
", "votes": 0, "replies": [] }, { "id": 589498, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2019-08-01T01:47:02.343000", "content": "Thank you so much for sharing. Including this model to our stack improves the score to -1.7x
", "votes": 0, "replies": [] }, { "id": 589329, "author_name": "Jason", "author_url": "", "post_date": "2019-07-31T18:52:39.180000", "content": "good work!
", "votes": 0, "replies": [] }, { "id": 579000, "author_name": "Beomki Yeo", "author_url": "", "post_date": "2019-07-18T11:51:42.897000", "content": "Hi @hengck23 . \nI am recently trying to benchmark your code, and I have a quick question about the ChampsDataset \nWhen you make a batch from this function, i checked you used offset. Those offsets are necessary for your GNN model? I thought it should work without offset term.
python model.py ( run_check_net ) throws Segmentation fault error. Does anyone has the same problem?
\n\nI'm using pytorch 1.1.0 CUDA10 on AWS p2.xlarge instance.
\n\nYou are like a god and I'm like a useless student😭 .
", "votes": 0, "replies": [ { "id": 578910, "author_name": "hadxu", "author_url": "", "post_date": "2019-07-18T09:29:08.323000", "content": "I meet this problem a week ago. I think your torch_geometric library install error. Please install torch==1.1 and install torch_geometric follow pytorch_geometric
", "votes": 1, "replies": [] }, { "id": 578926, "author_name": "XY", "author_url": "", "post_date": "2019-07-18T09:46:50.137000", "content": "Thank you! I try!
", "votes": 0, "replies": [] } ] }, { "id": 569357, "author_name": "", "author_url": "", "post_date": "2019-07-06T14:06:47.900000", "content": "", "votes": 0, "replies": [] }, { "id": 565433, "author_name": "", "author_url": "", "post_date": "2019-07-01T00:22:22.903000", "content": "", "votes": 0, "replies": [ { "id": 565614, "author_name": "", "author_url": "", "post_date": "2019-07-01T07:01:18.150000", "content": "", "votes": 0, "replies": [] }, { "id": 567072, "author_name": "", "author_url": "", "post_date": "2019-07-03T02:46:45.903000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 564661, "author_name": "", "author_url": "", "post_date": "2019-06-29T18:49:44.027000", "content": "", "votes": 0, "replies": [ { "id": 564917, "author_name": "", "author_url": "", "post_date": "2019-06-30T06:55:47.280000", "content": "", "votes": 5, "replies": [] }, { "id": 565599, "author_name": "", "author_url": "", "post_date": "2019-07-01T06:45:11.597000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 543593, "author_name": "", "author_url": "", "post_date": "2019-06-04T16:06:36.177000", "content": "", "votes": 0, "replies": [ { "id": 543963, "author_name": "", "author_url": "", "post_date": "2019-06-05T01:27:41.453000", "content": "", "votes": 1, "replies": [] }, { "id": 544173, "author_name": "", "author_url": "", "post_date": "2019-06-05T08:28:59.483000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 540493, "author_name": "", "author_url": "", "post_date": "2019-05-31T15:22:33.843000", "content": "", "votes": 0, "replies": [ { "id": 541573, "author_name": "", "author_url": "", "post_date": "2019-06-02T16:59:38.720000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 581247, "author_name": "", "author_url": "", "post_date": "2019-07-21T16:44:04.473000", "content": "", "votes": 0, "replies": [] }, { "id": 540527, "author_name": "", "author_url": "", "post_date": "2019-05-31T15:52:19.317000", "content": "", "votes": -7, "replies": [] }, { "id": 602628, "author_name": "", "author_url": "", "post_date": "2019-08-19T09:33:04.307000", "content": "", "votes": 1, "replies": [] }, { "id": 559979, "author_name": "", "author_url": "", "post_date": "2019-06-24T23:00:36.353000", "content": "", "votes": 1, "replies": [] }, { "id": 590571, "author_name": "", "author_url": "", "post_date": "2019-08-02T10:54:35.997000", "content": "", "votes": 0, "replies": [] }, { "id": 565003, "author_name": "", "author_url": "", "post_date": "2019-06-30T09:38:55.627000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "540459": "This post is for information and results using graph CNN. For a start:\n\nuseful Graph CNN lib:\n\n- https://github.com/rusty1s/pytorch_geometric\n- https://github.com/pfnet-research/chainer-chemistry\n- https://github.com/jparkhill/TensorMol\n\n\n- https://deepchem.io/\n\n\nothers:\n\n-https://github.com/ifding/graph-neural-networks\n\n\npossible approach?\n - Neural Message Passing for Quantum Chemistry (google deepmind/ICML 2017)", "558553": "starter-kit is up! (train for about 160k iterations/60 epoch, you can get LB of about -0.725. Note that Google paper uses about 530 epoch in their QM9 paper and train individual separate predictors, one for each target. I am not sure how many epochs are required in this kaggle challenge, so have fun in experimenting!) \n\n", "546673": "Hi @hengck23, I remember you from a previous competition (QuickDraw). Your posts were always insightful.\n\nI'm working right now on my Master Thesis in geometric deep-learning but, alas, it won't be helpful for this competition. Still, I might give you my two cents:\n- As you can see in this [recent survey](https://arxiv.org/abs/1810.04020), most GCNs are designed for graph-level or node-level outputs. The method you proposed in another comment is actually the only way to go for this kind of tasks (predicting node-pair features and then use a standard MLP to predict the edge labels, in a 1D-Conv fashion).\n- If you want a model with edge-parameters, you might want to have a look to [this paper from a bunch of very smart people](https://arxiv.org/abs/1806.01261). They propose a very generic model that can be applied to any kind of task. They also provide an [implementation in TensorFlow and Sonnet](https://github.com/deepmind/graph_nets), if you're interested.\n- Since this is an edge regression task, using node-pooling might be problematic: you have to preserve all the edges, and state-of-the-art methods typically don't do this. You might pool the nodes, but the clusters need to form a partition of the graph, not a cover (i.e., you have to assign to each node one and only one cluster). Classical clustering methods (e.g. K-means) on node features (e.g. spatial position) might be an option.\n- Regarding the available frameworks, I can only speak for [PyTorch-Geometric](https://github.com/rusty1s/pytorch_geometric), which is the one that I'm using for my thesis and I think it's pretty solid (Matthias Fey, if you're reading this, thank you very much). It also provide an alternative implementation for the previous model under the name of [`MetaLayer`](https://rusty1s.github.io/pytorch_geometric/build/html/modules/nn.html#torch_geometric.nn.meta.MetaLayer).\n- I read in another comment that you're looking for a rotation-invariant method for the graphs. You might prefer not using the 3D coordinates at all, or embed the node distances directly in the edges, as a data preprocessing.\n\nI hope this might help you or anyone in this competition. Good luck to everybody! :)", "561118": "new release in preparation .... (single graph CNN model can get LB -1.46)\n\n", "541488": "\n\nhttps://github.com/pfnet-research/bayesgrad\n\nBayesGrad: Explaining Predictions of Graph Convolutional Networks", "560359": "\nversion.2 starter kit coming up! \n\nNote the std of the target value and training/validation score value .... that is some trick you can exploit to get better score. I will explain later if i can confirm my hypothesis ... doing some experiments now :)\n\n", "573627": "@hengck23 : Come back !! \n", "541566": "features", "540520": "my idea:\n\n", "570475": "I saw this thread after a month unfortunately, this thread has so much of cool points, visualizations and high skill level, that it should be treated as one of those videos on youtube which are titled as **People are awesome** and feature a guy throwing a 30 inch pizza base into the air 6 feet high at 50 revolutions per sec and catches without a glitch. Or a guy somewhere in South Asia chopping salad faster than a blink of an eye.\n\nI mean I am amazed to see how good others are approaching this competitions I am just making line plots and scatter plots with a few bits and pieces of adding colors and thinking it looks good but then there is someone who is analyzing the structure of molecules using interactive widgets having an entire view of the dataset.\n\nThis is great work by the author. So much to learn for me really liked the standard of work. Keep it up. 👍 ", "542118": "Excellent talk on the basics of Graph Neural Networks:\n\nhttps://www.youtube.com/watch?v=cWIeTMklzNg", "559256": "with so many variants of MPNN, NAS network architectural search for best graph cnn seems interesting:\n\nhttps://arxiv.org/pdf/1904.09981.pdf\n\n\nGraphNAS: Graph Neural Architecture Search with Reinforcement Learning\n\n", "541493": "some resources here: https://docs.dgl.ai/index.html\n\nhttps://iwatobipen.wordpress.com/\n\nhttps://iwatobipen.wordpress.com/?s=pytorch\n\nseems useful:\nhttps://alchemy.tencent.com/\n\n\"If you do not want to dive into RDKit, we also provide a ready-to-use pytorch dataloader which can help you easily deal with those molecules.\n\nYou may also find a collection of baselines, including MPNN, from which you can start your journey with Alchemy!\"", "545934": "https://github.com/Tony-Y/cgnn\n\nThis is my PyTorch implementation of GNN for crystalline materials. It can be used for molecules, but if you want to use edge features you have to extend my code.", "565882": "Thanks. Didn't have enough time to learn about graph nn and really appreciate your efforts to gather all this info into one single thread.", "540550": "i wonder if anyone has a good method to make the results rotation-invariant (assuming that the inpute molecules can be rotated)? ", "564595": "what to train and how to ensemble? a possible solution for LB -1.6 here:\n\n```\n\n\ndef run_ensemble1():\n\n ensemble_file='/root/share/project/kaggle/2019/champs_scalar/result/separate/submit-ensemble-larger-00.csv'\n df_test = pd.read_csv('/root/share/project/kaggle/2019/champs_scalar/data/csv/test.csv')\n df_test = df_test.sort_values(by=['id'])\n\n id = df_test.id.values\n type = df_test.type.values\n\n num_test=len(id)\n coupling_count = np.zeros(num_test)\n coupling_value = np.zeros(num_test)\n\n ['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '3JHN', '2JHH', '3JHH',]\n\n\n for valid_type, f in [\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/ensemble/submit-00370000_model-larger--add-poola.csv'),\n\n (['1JHC', '2JHC', '3JHC', '1JHN', '2JHN', '2JHH', '3JHH',],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all/submit/submit-00327500_model.csv'),\n\n (['3JHN'],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/3jhn-00/submit/submit-00327500_model.csv'),\n\n (['1JHC', '2JHC', '3JHC', ],\n '/root/share/project/kaggle/2019/champs_scalar/result/separate/all-jhc-00/submit/submit-00262500_model.csv'),\n ]:\n\n df = pd.read_csv(f)\n df = df.sort_values(by=['id'])\n for t in valid_type:\n index = np.where(type==t)[0]\n coupling_value[index] += df.scalar_coupling_constant.values[index]\n coupling_count[index] += 1\n\n\n coupling_value = coupling_value/coupling_count\n\n\n df = pd.DataFrame(list(zip(id, coupling_value)), columns =['id', 'scalar_coupling_constant'])\n df.to_csv(ensemble_file,index=False)\n\n```\n\nsee also the attachment train log files:", "561677": "self study note. rewrite graph convolution layer NNConv and set2set for easier understanding ....\n\n```\nThe continuous kernel-based convolutional operator adapted from the “Neural Message Passing for Quantum Chemistry” paper\n\nx′i=Θxi+∑j∈N(i)xj⋅hΘ(ei,j),\n\nwhere hΘhΘ denotes a neural network, .i.e. a MLP.\n```\n\n(https://rusty1s.github.io/pytorch_geometric/build/html/modules/nn.html#module-torch_geometric.nn.conv.message_passing)\n\n\n", "541580": "Python-based Simulations of Chemistry Framework\nhttps://github.com/pyscf/pyscf", "540485": "\"SchNet: A continuous-filter convolutional neural network for modeling quantum interactions\"\n\n", "561179": "Table I of [SchNet paper](https://aip.scitation.org/doi/10.1063/1.5019779) shows that SchNet is better than MPNN enn-s2s on the QM9 dataset except for ZPVE, mu, and alpha. So, SchNet-like neural network models also might be better than enn-s2s models in this competition.\n\n\n\n\nhttps://github.com/atomistic-machine-learning/schnetpack\n\nhttps://paperswithcode.com/sota/formation-energy-on-qm9", "564601": "this seems interesting: https://arxiv.org/pdf/1806.04613.pdf\n\nImproving Regression Performance with Distributional Losses", "559723": "edge-based graph update:\n\nNeural Message Passing with Edge Updates for Predicting Properties of Molecules and Materials\nhttps://arxiv.org/pdf/1806.03146.pdf\n\ntransformer mask (for longer path dependency):\n\nPath-Augmented Graph Transformer Network\nhttps://arxiv.org/pdf/1905.12712.pdf\n\nothers:\n\nDeep Neural Network Computes Electron Densities\nand Energies of a Large Set of Organic Molecules\nFaster than Density Functional Theory (DFT)\nhttps://arxiv.org/pdf/1809.02723.pdf\n\nOperators in Machine Learning: Response Properties in Chemical Space\nhttps://arxiv.org/pdf/1807.08811.pdf", "572418": "noobie question, @hengck23 where can I find those starter kit?\nThanks for this amazing job! :)", "540779": "You took all the word right out of my mouth. Maybe we can team up! I can skillfully use PyG and DGL", "591288": "Hi Heng, \n\nThanks for sharing your code. I saw in your log.train.txt file, the loss kept decreasing until around -2.5. But when I trained a model using your code (the 20190630 version), I found the loss started fluctuating at around -1.2 after around 30 epochs. Did I miss anything? Below is the loss:\n\nlr=0.00050\n\n** start training here! **\n batch_size =12, iter_accum=1\n |--------------- VALID ----------------------------------------------------------------|-- TRAIN/BATCH ---------\n |std 18.3 4.5 3.1 10.9 3.7 1.3 4.0 3.7 | | |\nrate iter epoch | 1JHC, 2JHC, 3JHC, 1JHN, 2JHN, 3JHN, 2JHH, 3JHH | loss mae log_mae | loss | time\n--------------------------------------------------------------------------------------------------------------------------------------\n0.00000 167.5* 33.5 | -0.645, -1.449, -1.187, -0.693, -1.700, -1.749, -1.649, -1.556 | -1.231 0.29 -1.33 | +0.000 | 0 hr 00 min\n0.00050 168.0 33.6 | -0.664, -1.431, -1.175, -0.735, -1.708, -1.732, -1.676, -1.542 | -1.229 0.29 -1.33 | -1.197 | 0 hr 02 min\n0.00050 168.5 33.6 | -0.671, -1.420, -1.083, -0.737, -1.632, -1.713, -1.597, -1.544 | -1.190 0.30 -1.30 | -1.225 | 0 hr 04 min\n0.00050 169.0 33.7 | -0.460, -1.331, -1.068, -0.634, -1.558, -1.620, -1.163, -1.446 | -1.068 0.34 -1.16 | -1.149 | 0 hr 06 min\n", "589498": "Thank you so much for sharing. Including this model to our stack improves the score to -1.7x", "589329": "good work!", "579000": "Hi @hengck23 . \nI am recently trying to benchmark your code, and I have a quick question about the `ChampsDataset` \nWhen you make a batch from this function, i checked you used `offset`. Those offsets are necessary for your GNN model? I thought it should work without offset term.", "578905": "python model.py ( run_check_net ) throws Segmentation fault error. Does anyone has the same problem?\n\nI'm using pytorch 1.1.0 CUDA10 on AWS p2.xlarge instance.\n\nYou are like a god and I'm like a useless student😭 . \n\n", "569357": "How do you know the graph structure at test time? ", "565433": "Hi Heng, thanks for sharing. I run into an error when converting molecules to graphs. The error is related to this line in the following function: \"for i,j in UA_pairs\". The UA_pairs is a list of integers. The error is 'int is not iterable. Did you run into this? BTW, the function is in data.py file.\n\nWhen it failed on the molecule id, dsgdb9nsd_000023, the UA_pairs is a list which is [2, 1, 3, 0].\n\ndef get_BO(AC,UA,DU,valences,UA_pairs,quick):\n BO = AC.copy()\n DU_save = []\n while DU_save != DU:\n for i,j in UA_pairs:\n BO[i,j] += 1\n BO[j,i] += 1", "564661": "@hengck23 : thank for sharing the model. My training seems to be extremely slow compared to yours (>24h) and I haven't reach 40 epochs. May I ask what system / card you are using ?", "543593": "Hi, does anyone have any idea how to predict various numbers of coupling from a single molecular?\nEX: it remind me to predict bboxes in image detection, that every image have different numbers of bbox.", "540493": "I expect this is where some of the best models (at least initially) will come from. MPNN's have shown the best results in quantum prediction tasks. DeepChem has implementations of several GraphCNN's and already has the featurization from SMILES/SDF files. This is where I am starting. There are some necessary changes to be made to the code though.", "581247": "", "540527": "", "602628": "Thanks for sharing.", "559979": "thank you for sharing all of it", "590571": "Thank you very much for sharing these.\n", "565003": "Thanks for sharing " } }