Hi, all Kagglers,
\n\nI am glad to share a brief explanation of my method and why I design the model as follows. I really appreciate organizers for holding such a nice competition. I learned a lot and really enjoyed the competition. All questions and opinions are always welcome!
\n\nStrategy\nI have a few domain knowledge in quantum chemistry but not specific in this competition. The only domain expertise I used is that \"all quantum mechanical properties are determined by atom type and distances between atoms\". This was my fundamental strategy so I used only atom type and distance as input features. However, other participants showed that other features can improve the performance of their models. More surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount. In addition, I still personally believe that such conditions must be satisfied to develop robust and reliable models to predict molecular properties.
Model\nI represented a molecule with a fully connected graph. Initial node feature includes only atom type and initial edge feature includes an only distance between atoms. I started with MPNN architecture. In contrast to original MPNN, my model predict scalar coupling between two atoms from their node state obtained after the propagation step. I used 8 different fully connected layers for each type of scalar coupling. I found that not sharing interaction layers improved the performance. I used 4 interaction block with 8 towers and 256 dimensions. This model achieved about -1.7--1.8 in public LD. The followings are my major modification to my models and corresponding LB score.
\n\nFinally, I congrats all the prize winners and thank all participants. I appreciate organizers' efforts again.
", "messages": [ { "id": 613355, "postDate": "2019-08-30T12:47:21.733Z", "content": "Hi, all Kagglers,
\n\nI am glad to share a brief explanation of my method and why I design the model as follows. I really appreciate organizers for holding such a nice competition. I learned a lot and really enjoyed the competition. All questions and opinions are always welcome!
\n\nStrategy\nI have a few domain knowledge in quantum chemistry but not specific in this competition. The only domain expertise I used is that \"all quantum mechanical properties are determined by atom type and distances between atoms\". This was my fundamental strategy so I used only atom type and distance as input features. However, other participants showed that other features can improve the performance of their models. More surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount. In addition, I still personally believe that such conditions must be satisfied to develop robust and reliable models to predict molecular properties.
Model\nI represented a molecule with a fully connected graph. Initial node feature includes only atom type and initial edge feature includes an only distance between atoms. I started with MPNN architecture. In contrast to original MPNN, my model predict scalar coupling between two atoms from their node state obtained after the propagation step. I used 8 different fully connected layers for each type of scalar coupling. I found that not sharing interaction layers improved the performance. I used 4 interaction block with 8 towers and 256 dimensions. This model achieved about -1.7--1.8 in public LD. The followings are my major modification to my models and corresponding LB score.
\n\nFinally, I congrats all the prize winners and thank all participants. I appreciate organizers' efforts again.
", "rawMarkdown": "Hi, all Kagglers,\n\nI am glad to share a brief explanation of my method and why I design the model as follows. I really appreciate organizers for holding such a nice competition. I learned a lot and really enjoyed the competition. All questions and opinions are always welcome!\n\n1. Strategy\nI have a few domain knowledge in quantum chemistry but not specific in this competition. The only domain expertise I used is that \"all quantum mechanical properties are determined by atom type and distances between atoms\". This was my fundamental strategy so I used only atom type and distance as input features. However, other participants showed that other features can improve the performance of their models. More surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount. In addition, I still personally believe that such conditions must be satisfied to develop robust and reliable models to predict molecular properties. \n\n2. Model\nI represented a molecule with a fully connected graph. Initial node feature includes only atom type and initial edge feature includes an only distance between atoms. I started with MPNN architecture. In contrast to original MPNN, my model predict scalar coupling between two atoms from their node state obtained after the propagation step. I used 8 different fully connected layers for each type of scalar coupling. I found that not sharing interaction layers improved the performance. I used 4 interaction block with 8 towers and 256 dimensions. This model achieved about -1.7--1.8 in public LD. The followings are my major modification to my models and corresponding LB score.\n- Replace onehot encoding of distance to RBF proposed in Schnet paper : LB:-2.1---2.2\n- Edge update network: -2.4\n- Edge update network with GRU update function: -2.6\n- Hyperparameter tunning : -2.8\n- Ensemble with 10 models with different initial weight parameters: -2.98\n\nFinally, I congrats all the prize winners and thank all participants. I appreciate organizers' efforts again.\n", "votes": 38 }, { "id": 613409, "postDate": "2019-08-30T13:29:35.453Z", "content": "Congrats with the gold @jaechang! And thanks for the hints earlier!
", "rawMarkdown": "Congrats with the gold @jaechang! And thanks for the hints earlier!", "votes": 1 }, { "id": 613376, "postDate": "2019-08-30T13:06:26.293Z", "content": "Thanks for sharing, and congrats on the solo performance. We were afraid of you jumping ahead of us at the last minute!
\n\n\n\n\nMore surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount.
\n
I have expressed the exact same after seeing team 2 and 3 solutions. Not to downcast what they did, they did right given their model outperform ours !
", "rawMarkdown": "Thanks for sharing, and congrats on the solo performance. We were afraid of you jumping ahead of us at the last minute!\n\n> More surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount.\n\nI have expressed the exact same after seeing team 2 and 3 solutions. Not to downcast what they did, they did right given their model outperform ours !", "votes": 1, "replies": [ { "id": 613391, "postDate": "2019-08-30T13:16:02.583Z", "content": "Yes. We should appreciate them as a winner of this competition. Their results broadened my view and help to break my stereotype on developing deep learning models for molecules
", "rawMarkdown": "Yes. We should appreciate them as a winner of this competition. Their results broadened my view and help to break my stereotype on developing deep learning models for molecules", "votes": 4 }, { "id": 613579, "postDate": "2019-08-30T16:24:45.660Z", "content": "\nIt gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount.\n\nIt enlightened me to see that one can generate new insights for his research based on ml models that are generating features that are new and unknown for a particular domain of expertise. So cool @jaechang .
Great performance as a solo user !!
", "rawMarkdown": "Great performance as a solo user !!" }, { "id": 616526, "postDate": "2019-09-03T07:58:11.617Z", "content": "Re rotational invariance we needed it when doing Pointnet but did not add much with the Atom Transformer.
\n\nI may train a model from scratch with rotations again to double check, but as @jaechang said we also suspected there's a relative canonical orientation in the dataset that minimizes need for rotational alignment.
", "rawMarkdown": "Re rotational invariance we needed it when doing Pointnet but did not add much with the Atom Transformer. \n\nI may train a model from scratch with rotations again to double check, but as @jaechang said we also suspected there's a relative canonical orientation in the dataset that minimizes need for rotational alignment." }, { "id": 613574, "postDate": "2019-08-30T16:21:08.960Z", "content": "Congrats and thanks for the share @jaechang - especially on getting there in solo.
", "rawMarkdown": "Congrats and thanks for the share @jaechang - especially on getting there in solo." }, { "id": 613529, "postDate": "2019-08-30T15:32:17.097Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 613409, "author_name": "fnands", "author_url": "", "post_date": "2019-08-30T13:29:35.453000", "content": "Congrats with the gold @jaechang! And thanks for the hints earlier!
", "votes": 1, "replies": [] }, { "id": 613376, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-30T13:06:26.293000", "content": "Thanks for sharing, and congrats on the solo performance. We were afraid of you jumping ahead of us at the last minute!
\n\n\n\n\nMore surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount.
\n
I have expressed the exact same after seeing team 2 and 3 solutions. Not to downcast what they did, they did right given their model outperform ours !
", "votes": 1, "replies": [ { "id": 613391, "author_name": "Jaechang Lim", "author_url": "", "post_date": "2019-08-30T13:16:02.583000", "content": "Yes. We should appreciate them as a winner of this competition. Their results broadened my view and help to break my stereotype on developing deep learning models for molecules
", "votes": 4, "replies": [] }, { "id": 613579, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T16:24:45.660000", "content": "\nIt gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount.\n\nIt enlightened me to see that one can generate new insights for his research based on ml models that are generating features that are new and unknown for a particular domain of expertise. So cool @jaechang .
Great performance as a solo user !!
", "votes": 0, "replies": [] }, { "id": 616526, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-09-03T07:58:11.617000", "content": "Re rotational invariance we needed it when doing Pointnet but did not add much with the Atom Transformer.
\n\nI may train a model from scratch with rotations again to double check, but as @jaechang said we also suspected there's a relative canonical orientation in the dataset that minimizes need for rotational alignment.
", "votes": 0, "replies": [] }, { "id": 613574, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T16:21:08.960000", "content": "Congrats and thanks for the share @jaechang - especially on getting there in solo.
", "votes": 0, "replies": [] }, { "id": 613529, "author_name": "", "author_url": "", "post_date": "2019-08-30T15:32:17.097000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "613355": "Hi, all Kagglers,\n\nI am glad to share a brief explanation of my method and why I design the model as follows. I really appreciate organizers for holding such a nice competition. I learned a lot and really enjoyed the competition. All questions and opinions are always welcome!\n\n1. Strategy\nI have a few domain knowledge in quantum chemistry but not specific in this competition. The only domain expertise I used is that \"all quantum mechanical properties are determined by atom type and distances between atoms\". This was my fundamental strategy so I used only atom type and distance as input features. However, other participants showed that other features can improve the performance of their models. More surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount. In addition, I still personally believe that such conditions must be satisfied to develop robust and reliable models to predict molecular properties. \n\n2. Model\nI represented a molecule with a fully connected graph. Initial node feature includes only atom type and initial edge feature includes an only distance between atoms. I started with MPNN architecture. In contrast to original MPNN, my model predict scalar coupling between two atoms from their node state obtained after the propagation step. I used 8 different fully connected layers for each type of scalar coupling. I found that not sharing interaction layers improved the performance. I used 4 interaction block with 8 towers and 256 dimensions. This model achieved about -1.7--1.8 in public LD. The followings are my major modification to my models and corresponding LB score.\n- Replace onehot encoding of distance to RBF proposed in Schnet paper : LB:-2.1---2.2\n- Edge update network: -2.4\n- Edge update network with GRU update function: -2.6\n- Hyperparameter tunning : -2.8\n- Ensemble with 10 models with different initial weight parameters: -2.98\n\nFinally, I congrats all the prize winners and thank all participants. I appreciate organizers' efforts again.\n", "613409": "Congrats with the gold @jaechang! And thanks for the hints earlier!", "613376": "Thanks for sharing, and congrats on the solo performance. We were afraid of you jumping ahead of us at the last minute!\n\n> More surprisingly, some models which don't preserve permutation, rotational, and translational invariance outperformed my models. It really shocked me! I really appreciate their efforts to find such an innovative idea. It gave me inspiration in my research topic. I guess that the reason why such things don't make any problem in this competition is that there are some hidden patterns in atom ordering and 3D geometry of molecules of QM9 set. I wonder how the performance of the models change if we randomly change atom ordering, randomly rotate and translate the geometry of molecules in a large amount.\n\nI have expressed the exact same after seeing team 2 and 3 solutions. Not to downcast what they did, they did right given their model outperform ours !", "616956": "Great performance as a solo user !!", "616526": "Re rotational invariance we needed it when doing Pointnet but did not add much with the Atom Transformer. \n\nI may train a model from scratch with rotations again to double check, but as @jaechang said we also suspected there's a relative canonical orientation in the dataset that minimizes need for rotational alignment.", "613574": "Congrats and thanks for the share @jaechang - especially on getting there in solo.", "613529": "" }, "complete": true, "source": "existing_discussion", "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3229729, "solution_url": "/competitions/champs-scalar-coupling/writeups/jaechang-lim-9-solution-model-with-only-atom-type-" }