![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1155353%2F9d825e9c577db8b951a18cb9827e683b%2Ffig_1.png?generation=1567130786795298&alt=media\")
Our story
\n\nAs you know, we need to find the appropriate representation for data. That’s why we’re struggling to do feature engineering.
\n\nAs CPMP mentioned, two member of our team have domain knowledge on this competition. \nSunghwan Choi has Ph.D at quantum chemistry and works on the field of quantum chemistry and chemical application of machine learning.
\n\nI’m going on the Ph.D course in chemical engineering and have some experience for quantum calculation and dealing molecules. More important thing is that I’m kaggler :).
\n\nWe thought that our problem can be solved by conventional graph models whose edge features are distance-derived properties. Those models are quite conventional in machine learning applications on chemical system. Hence, we tried to figure out appropriate hyper parameters and edge features for models
\n\nAs many kagglers did, we also tested message passing neural networks by adopting many useful kernels. We thank to many kegglers; especially heng cher keng. :) by modification of their solution, we got a silver place 1 month ago. But, the gap between leading group and us was getting large.
\n\nWe had to find out breakthrough. We brainstormed a lot. At that time, limerobot, who is an expert of natural language processing suggested to use the raw xyz coordinates, \nI and Sunghwan didn’t agree with that, because if we use xyz coordinates instead of distance, translational and rotational invariances are not satisfied. The model which do not preserve those invariances seems to be ridiculous
\n\nBut, limerobot did on his way. And he showed that his model won my GNN. \nHis model was based on BERT model. You can see the big success in toxic competition on his profile.
\n\nMaybe, insights from toxic competition save us :).
\n\nAnyway, he don’t have any domain knowledge. He just wanted to make a model to learn the complex representation using xyz coordinates. (end-to-end)
\n\nHe just input the xyz coordinates of atom1, atom2, coupling type, distance and difference of each xyz coordinates (he thought the model can learn the distance formula based on that...amazing)
\n\nBecause we had only one month, we decided to do all things based on the transformer.\nAfter that, we did a number of experiment for hyper parameter tuning.
\n\nBecause BERT is very large model, the performances differed according to the number of hidden layers, type of embedding and learning schedule.
\n\nBefore 1 week to end of competition, we found our own scheduling and parameters and tiny modification of readout layer ( I will explain model later )
\n\nWe made multiples of models. Based on the ensemble of them, we got 3rd place :).
\n\nThanks for reading our stories. We want to share the specific magics below. Please keep going :)
\n\nHere is our overall architecture for our model. As you can see, our input sequence is transformed into output sequence by BERT-encoder.
\n\n• The number of encoder layer: 8\n• The number of heads for attention 8\n• Dropout ratio is 0.1 which is conventional choice for BERT model. \n• For each type, different readout networks but having same architecture are used
\n\n
Our float sequences composed of multiple embedding results, are the magic to get the achievement.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1155353%2Fac01c25bcce0d2e5e1638b6f6bd2c952%2Ffig_2.png?generation=1567130820868426&alt=media\")
Multi-head attention layer itself preserve permutational invariance therefore, order of couplings do not change the results but invariance when atomic_index_0 and atomic_index1 are changed, is not preserved since we use feature vector as concatenation of the embedding results of atomic charge, xyz coordinate(position), atomic number of two atoms.
\n\n• Size of embedding for atomic charge: 32\n• Size of embedding for position: 256\n• Size of embedding for atomic number: 64\n• Size of embedding for distance: 64\n• Size of embedding for type: 64
\n\nTotal feature size for single feature vector is (32+256+64)*2+64+64=832
\n\nIn order to impose pseudo-invariance on our model, we use rotational and translational noise when augmenting data.
\n\n• Translational noise: For each axis, Gaussian noise( mean: 0, std: 2) was added\n• Rotational noise: Rotational transformation whose axis is translational noise vector and angle is from Gaussian noise( mean:0, std: 3.14/2 )
\n\n• As you know that, spin-coupling (sc) value can be decomposed into four different terms (fc, sd, pso, dso)\n• After optimizing architecture and various losses, we found that auxiliary target using contributions gave a high boost.\n• During training, model minimize loss1 + loss2 with AdamW algorithm.\n• There are 8 regression layers to cover 8 different coupling types
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1155353%2F939a6ff0e4a6848477a655a2bfc6ad54%2Ffig_3.png?generation=1567130847675121&alt=media\")
• We were always using linear learning rate decay.\n• We think that there might be improvement with various learning schedule such as cycle lr. But, we didn’t have time because BERT is very large…(about 75M parameters. It took 1~2 days to get a model using 2~4 V100 machine)
\n\n• To get the more results, we needed some magics. With having an insight that there are less probability to be overfitted (Sunghwan choi’s insight) and experiment result from limerobot, we adopted pseudo-labeling.\n• After predicting for test set, we used the pseudo-labeled test dataset for training.\n• The model showed more than -3.4 CV. So, we trained model more with only train data to minigate overfitting. (finally we got ~-3.11 LB single models)\n• Overall learning process is illustrated below.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1155353%2F92bc1322f07e876a7e5f1c60684a095d%2Ffig_4.png?generation=1567130868749389&alt=media\")
• We made 14 models with various seed and hidden layers (most have 8 layers, other have 6 layers)\n• After weighted average according to cv score of types, we got -3.16.\n• After multiple procedure of pseudo labeling, we had 8 models. \n• With simple average, we got -3.19 (our final score :) 2 hours before the end of competition.
\n\n• End-to-End works!!!. Amazing BERT.\n• Learning schedule is very important for modelling of molecular property.
", "messages": [ { "id": 612778, "postDate": "2019-08-30T02:07:52.173Z", "content": "Our story
\n\nAs you know, we need to find the appropriate representation for data. That’s why we’re struggling to do feature engineering.
\n\nAs CPMP mentioned, two member of our team have domain knowledge on this competition. \nSunghwan Choi has Ph.D at quantum chemistry and works on the field of quantum chemistry and chemical application of machine learning.
\n\nI’m going on the Ph.D course in chemical engineering and have some experience for quantum calculation and dealing molecules. More important thing is that I’m kaggler :).
\n\nWe thought that our problem can be solved by conventional graph models whose edge features are distance-derived properties. Those models are quite conventional in machine learning applications on chemical system. Hence, we tried to figure out appropriate hyper parameters and edge features for models
\n\nAs many kagglers did, we also tested message passing neural networks by adopting many useful kernels. We thank to many kegglers; especially heng cher keng. :) by modification of their solution, we got a silver place 1 month ago. But, the gap between leading group and us was getting large.
\n\nWe had to find out breakthrough. We brainstormed a lot. At that time, limerobot, who is an expert of natural language processing suggested to use the raw xyz coordinates, \nI and Sunghwan didn’t agree with that, because if we use xyz coordinates instead of distance, translational and rotational invariances are not satisfied. The model which do not preserve those invariances seems to be ridiculous
\n\nBut, limerobot did on his way. And he showed that his model won my GNN. \nHis model was based on BERT model. You can see the big success in toxic competition on his profile.
\n\nMaybe, insights from toxic competition save us :).
\n\nAnyway, he don’t have any domain knowledge. He just wanted to make a model to learn the complex representation using xyz coordinates. (end-to-end)
\n\nHe just input the xyz coordinates of atom1, atom2, coupling type, distance and difference of each xyz coordinates (he thought the model can learn the distance formula based on that...amazing)
\n\nBecause we had only one month, we decided to do all things based on the transformer.\nAfter that, we did a number of experiment for hyper parameter tuning.
\n\nBecause BERT is very large model, the performances differed according to the number of hidden layers, type of embedding and learning schedule.
\n\nBefore 1 week to end of competition, we found our own scheduling and parameters and tiny modification of readout layer ( I will explain model later )
\n\nWe made multiples of models. Based on the ensemble of them, we got 3rd place :).
\n\nThanks for reading our stories. We want to share the specific magics below. Please keep going :)
\n\nHere is our overall architecture for our model. As you can see, our input sequence is transformed into output sequence by BERT-encoder.
\n\n• The number of encoder layer: 8\n• The number of heads for attention 8\n• Dropout ratio is 0.1 which is conventional choice for BERT model. \n• For each type, different readout networks but having same architecture are used
\n\n
Our float sequences composed of multiple embedding results, are the magic to get the achievement.
\n\n
Multi-head attention layer itself preserve permutational invariance therefore, order of couplings do not change the results but invariance when atomic_index_0 and atomic_index1 are changed, is not preserved since we use feature vector as concatenation of the embedding results of atomic charge, xyz coordinate(position), atomic number of two atoms.
\n\n• Size of embedding for atomic charge: 32\n• Size of embedding for position: 256\n• Size of embedding for atomic number: 64\n• Size of embedding for distance: 64\n• Size of embedding for type: 64
\n\nTotal feature size for single feature vector is (32+256+64)*2+64+64=832
\n\nIn order to impose pseudo-invariance on our model, we use rotational and translational noise when augmenting data.
\n\n• Translational noise: For each axis, Gaussian noise( mean: 0, std: 2) was added\n• Rotational noise: Rotational transformation whose axis is translational noise vector and angle is from Gaussian noise( mean:0, std: 3.14/2 )
\n\n• As you know that, spin-coupling (sc) value can be decomposed into four different terms (fc, sd, pso, dso)\n• After optimizing architecture and various losses, we found that auxiliary target using contributions gave a high boost.\n• During training, model minimize loss1 + loss2 with AdamW algorithm.\n• There are 8 regression layers to cover 8 different coupling types
\n\n
• We were always using linear learning rate decay.\n• We think that there might be improvement with various learning schedule such as cycle lr. But, we didn’t have time because BERT is very large…(about 75M parameters. It took 1~2 days to get a model using 2~4 V100 machine)
\n\n• To get the more results, we needed some magics. With having an insight that there are less probability to be overfitted (Sunghwan choi’s insight) and experiment result from limerobot, we adopted pseudo-labeling.\n• After predicting for test set, we used the pseudo-labeled test dataset for training.\n• The model showed more than -3.4 CV. So, we trained model more with only train data to minigate overfitting. (finally we got ~-3.11 LB single models)\n• Overall learning process is illustrated below.
\n\n
• We made 14 models with various seed and hidden layers (most have 8 layers, other have 6 layers)\n• After weighted average according to cv score of types, we got -3.16.\n• After multiple procedure of pseudo labeling, we had 8 models. \n• With simple average, we got -3.19 (our final score :) 2 hours before the end of competition.
\n\n• End-to-End works!!!. Amazing BERT.\n• Learning schedule is very important for modelling of molecular property.
", "rawMarkdown": "Our story\n\nAs you know, we need to find the appropriate representation for data. That’s why we’re struggling to do feature engineering.\n\nAs CPMP mentioned, two member of our team have domain knowledge on this competition. \nSunghwan Choi has Ph.D at quantum chemistry and works on the field of quantum chemistry and chemical application of machine learning. \n\nI’m going on the Ph.D course in chemical engineering and have some experience for quantum calculation and dealing molecules. More important thing is that I’m kaggler :).\n\nWe thought that our problem can be solved by conventional graph models whose edge features are distance-derived properties. Those models are quite conventional in machine learning applications on chemical system. Hence, we tried to figure out appropriate hyper parameters and edge features for models\n\nAs many kagglers did, we also tested message passing neural networks by adopting many useful kernels. We thank to many kegglers; especially heng cher keng. :) by modification of their solution, we got a silver place 1 month ago. But, the gap between leading group and us was getting large. \n\nWe had to find out breakthrough. We brainstormed a lot. At that time, limerobot, who is an expert of natural language processing suggested to use the raw xyz coordinates, \nI and Sunghwan didn’t agree with that, because if we use xyz coordinates instead of distance, translational and rotational invariances are not satisfied. The model which do not preserve those invariances seems to be ridiculous\n \nBut, limerobot did on his way. And he showed that his model won my GNN. \nHis model was based on BERT model. You can see the big success in toxic competition on his profile. \n\nMaybe, insights from toxic competition save us :).\n\nAnyway, he don’t have any domain knowledge. He just wanted to make a model to learn the complex representation using xyz coordinates. (end-to-end)\n\nHe just input the xyz coordinates of atom1, atom2, coupling type, distance and difference of each xyz coordinates (he thought the model can learn the distance formula based on that...amazing)\n\nBecause we had only one month, we decided to do all things based on the transformer.\nAfter that, we did a number of experiment for hyper parameter tuning.\n\nBecause BERT is very large model, the performances differed according to the number of hidden layers, type of embedding and learning schedule.\n\nBefore 1 week to end of competition, we found our own scheduling and parameters and tiny modification of readout layer ( I will explain model later ) \n\nWe made multiples of models. Based on the ensemble of them, we got 3rd place :).\n\nThanks for reading our stories. We want to share the specific magics below. Please keep going :)\n\n# Overall architecture\n\nHere is our overall architecture for our model. As you can see, our input sequence is transformed into output sequence by BERT-encoder.\n\n•\tThe number of encoder layer: 8\n•\tThe number of heads for attention 8\n•\tDropout ratio is 0.1 which is conventional choice for BERT model. \n•\tFor each type, different readout networks but having same architecture are used\n\n\n \n\n\n\n# Input features and embedding layer for them\n\nOur float sequences composed of multiple embedding results, are the magic to get the achievement.\n\n \n\n\nMulti-head attention layer itself preserve permutational invariance therefore, order of couplings do not change the results but invariance when atomic_index_0 and atomic_index1 are changed, is not preserved since we use feature vector as concatenation of the embedding results of atomic charge, xyz coordinate(position), atomic number of two atoms. \n\n•\tSize of embedding for atomic charge: 32\n•\tSize of embedding for position: 256\n•\tSize of embedding for atomic number: 64\n•\tSize of embedding for distance: 64\n•\tSize of embedding for type: 64\n\nTotal feature size for single feature vector is (32+256+64)*2+64+64=832\n\n\n# Augmentation\nIn order to impose pseudo-invariance on our model, we use rotational and translational noise when augmenting data.\n\n•\tTranslational noise: For each axis, Gaussian noise( mean: 0, std: 2) was added\n•\tRotational noise: Rotational transformation whose axis is translational noise vector and angle is from Gaussian noise( mean:0, std: 3.14/2 )\n\n\n# Regression layers for predicting scalar coupling constants\n•\tAs you know that, spin-coupling (sc) value can be decomposed into four different terms (fc, sd, pso, dso)\n•\tAfter optimizing architecture and various losses, we found that auxiliary target using contributions gave a high boost.\n•\tDuring training, model minimize loss1 + loss2 with AdamW algorithm.\n•\tThere are 8 regression layers to cover 8 different coupling types\n \n\n\n\n# Specific learning rate\n•\tWe were always using linear learning rate decay.\n•\tWe think that there might be improvement with various learning schedule such as cycle lr. But, we didn’t have time because BERT is very large…(about 75M parameters. It took 1~2 days to get a model using 2~4 V100 machine)\n\n\n# Pseudo labeling \n•\tTo get the more results, we needed some magics. With having an insight that there are less probability to be overfitted (Sunghwan choi’s insight) and experiment result from limerobot, we adopted pseudo-labeling.\n•\tAfter predicting for test set, we used the pseudo-labeled test dataset for training.\n•\tThe model showed more than -3.4 CV. So, we trained model more with only train data to minigate overfitting. (finally we got ~-3.11 LB single models)\n•\tOverall learning process is illustrated below.\n \n\n\n\n# Final submission\n•\tWe made 14 models with various seed and hidden layers (most have 8 layers, other have 6 layers)\n•\tAfter weighted average according to cv score of types, we got -3.16.\n•\tAfter multiple procedure of pseudo labeling, we had 8 models. \n•\tWith simple average, we got -3.19 (our final score :) 2 hours before the end of competition.\n\n# What we’ve learned\n•\tEnd-to-End works!!!. Amazing BERT.\n•\tLearning schedule is very important for modelling of molecular property.\n\n", "votes": 199 }, { "id": 612896, "postDate": "2019-08-30T04:27:39.083Z", "content": "Congratulations.....\nAmazing ideas...\nInspiring. Thanks for sharing.... @youhanlee
", "rawMarkdown": "Congratulations.....\nAmazing ideas...\nInspiring. Thanks for sharing.... @youhanlee \n", "votes": 6, "replies": [ { "id": 613020, "postDate": "2019-08-30T07:01:36.013Z", "content": "Thanks!
\n\nI've shocked for the performance of BERT :)
", "rawMarkdown": "Thanks!\n\nI've shocked for the performance of BERT :)", "votes": 2 } ] }, { "id": 613203, "postDate": "2019-08-30T09:47:50.623Z", "content": "How did you do embeddings on continous variables such as distance? Binning?\nHow did you get 32 size embedding on charge, that's an ordinalvariable with less than a dozen possible values?
", "rawMarkdown": "How did you do embeddings on continous variables such as distance? Binning?\nHow did you get 32 size embedding on charge, that's an ordinalvariable with less than a dozen possible values?", "votes": 5, "replies": [ { "id": 613948, "postDate": "2019-08-31T00:49:06.383Z", "content": "We just embedded the continous value into 32 or 64 size not using binning.
\n\nThe 32 size was arbitrary. We wanted to optimize the size, but not enough time.
\n\nWe can expect that there are optimal size of embeddings for various inputs.
", "rawMarkdown": "We just embedded the continous value into 32 or 64 size not using binning.\n\nThe 32 size was arbitrary. We wanted to optimize the size, but not enough time.\n\nWe can expect that there are optimal size of embeddings for various inputs.", "votes": 3 }, { "id": 615055, "postDate": "2019-09-01T12:09:36.753Z", "content": "So you turned 1 column into 32 columns/variables in the net?\nI'm probably missing something about your setup - I don't see how that would have improved things vs just inputting the number as a variable.
\n\nReally interesting overall!
", "rawMarkdown": "So you turned 1 column into 32 columns/variables in the net?\nI'm probably missing something about your setup - I don't see how that would have improved things vs just inputting the number as a variable. \n\nReally interesting overall!", "votes": 2 } ] }, { "id": 613144, "postDate": "2019-08-30T09:00:04.867Z", "content": "Congrats @youhanlee and your team for Amazing Solution!!
", "rawMarkdown": "Congrats @youhanlee and your team for Amazing Solution!!", "votes": 6, "replies": [ { "id": 614741, "postDate": "2019-09-01T01:35:05.160Z", "content": "Thanks for kind words!
", "rawMarkdown": "Thanks for kind words! ", "votes": 2 } ] }, { "id": 613215, "postDate": "2019-08-30T10:05:56.133Z", "content": "Thanks for sharing, and congrats on the well deserved end result. Again, using raw coordinates works surprisingly well. I wonder if there isn't some information leak in the way molecules are built here,. Would you and 2nd team get the same result if random rotations were applied to input molecules?
\n\nAnyway, your solution is very original and works amazingly well. I'll reread to grasp all details ;)
", "rawMarkdown": "Thanks for sharing, and congrats on the well deserved end result. Again, using raw coordinates works surprisingly well. I wonder if there isn't some information leak in the way molecules are built here,. Would you and 2nd team get the same result if random rotations were applied to input molecules?\n\nAnyway, your solution is very original and works amazingly well. I'll reread to grasp all details ;)", "votes": 4, "replies": [ { "id": 614733, "postDate": "2019-09-01T01:22:44.177Z", "content": "Thanks for kind words.
\n\nWe were also surprized about the performance of using raw coodinates. \nI don't think any leak in making the structure of molecules. The molecules might be optimized using QM calculations.
\n\nAnyway, we used the rotation matrics which were not too much to rotate molecules.\nSo, Rotated molecules showed very similar coupling constants compared to that of original structures.
", "rawMarkdown": "Thanks for kind words.\n\nWe were also surprized about the performance of using raw coodinates. \nI don't think any leak in making the structure of molecules. The molecules might be optimized using QM calculations.\n\nAnyway, we used the rotation matrics which were not too much to rotate molecules.\nSo, Rotated molecules showed very similar coupling constants compared to that of original structures.", "votes": 5 } ] }, { "id": 614819, "postDate": "2019-09-01T05:47:21.440Z", "content": "Great solution!\nOne question, did you use tensorflow or pytorch?
", "rawMarkdown": "Great solution!\nOne question, did you use tensorflow or pytorch?", "votes": 5, "replies": [ { "id": 616149, "postDate": "2019-09-02T18:41:27.933Z", "content": "Thanks! \nHi, we used pytorch_transformer (https://github.com/huggingface/pytorch-transformers).
", "rawMarkdown": "Thanks! \nHi, we used pytorch_transformer (https://github.com/huggingface/pytorch-transformers).", "votes": 4 }, { "id": 616605, "postDate": "2019-09-03T09:28:26.317Z", "content": "thanks!
", "rawMarkdown": "thanks! ", "votes": 2 } ] }, { "id": 930163, "postDate": "2020-07-15T08:33:27.133Z", "content": "I have a question about \"input sequence\". You show that the max coupling of the input sequence is 135. I suppose that this number is from the total number of coupling constants in molecule dsgdb9nsd_042139.
\n\nSo, is your \"input sequence\" a set of coupling constants in a molecule?
\n\nHowever, if the answer is true, your input does not contain some information, such as oxygen in molecule, because scalar coupling type is only for H, C and N.
", "rawMarkdown": "I have a question about \"input sequence\". You show that the max coupling of the input sequence is 135. I suppose that this number is from the total number of coupling constants in molecule dsgdb9nsd_042139.\n\nSo, is your \"input sequence\" a set of coupling constants in a molecule?\n\nHowever, if the answer is true, your input does not contain some information, such as oxygen in molecule, because scalar coupling type is only for H, C and N.", "votes": 1 }, { "id": 617858, "postDate": "2019-09-04T14:43:48.460Z", "content": "This is awesome!
", "rawMarkdown": "This is awesome!", "votes": 3, "replies": [ { "id": 619240, "postDate": "2019-09-06T01:31:23.353Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 617351, "postDate": "2019-09-04T03:30:49.340Z", "content": "Woww.. amazing solution
", "rawMarkdown": "Woww.. amazing solution", "votes": 3, "replies": [ { "id": 617819, "postDate": "2019-09-04T14:06:49.760Z", "content": "Thanks for kind words!
", "rawMarkdown": "Thanks for kind words!", "votes": 2 } ] }, { "id": 616930, "postDate": "2019-09-03T15:45:47.457Z", "content": "Simply Wow...!!!\nSo, you just described the journey on this competition and along with it the whole thought process.\nBig Congratulations on your win and thanks a lot for sharing this..!!!\nAwesome read...!!!
", "rawMarkdown": "Simply Wow...!!!\nSo, you just described the journey on this competition and along with it the whole thought process.\nBig Congratulations on your win and thanks a lot for sharing this..!!!\nAwesome read...!!!", "votes": 3, "replies": [ { "id": 619244, "postDate": "2019-09-06T01:32:46.717Z", "content": "Thanks for kind words! \nHope this post is informative to you!
", "rawMarkdown": "Thanks for kind words! \nHope this post is informative to you!", "votes": 1 } ] }, { "id": 616699, "postDate": "2019-09-03T11:26:46.400Z", "content": "I love this! Congratz!
", "rawMarkdown": "I love this! Congratz!", "votes": 3, "replies": [ { "id": 619241, "postDate": "2019-09-06T01:31:56.687Z", "content": "I've always liked your solutions :) Thanks for kind words!
", "rawMarkdown": "I've always liked your solutions :) Thanks for kind words!", "votes": 1 } ] }, { "id": 616516, "postDate": "2019-09-03T07:48:39.287Z", "content": "Great Solution.
", "rawMarkdown": "Great Solution.", "votes": 3, "replies": [ { "id": 619237, "postDate": "2019-09-06T01:30:56.547Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 616455, "postDate": "2019-09-03T06:32:35.337Z", "content": "Congrats and thanks a lot for sharing.
", "rawMarkdown": "**Congrats and thanks a lot for sharing.**", "votes": 3, "replies": [ { "id": 619234, "postDate": "2019-09-06T01:30:08.793Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 615927, "postDate": "2019-09-02T14:13:39.550Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": 3, "replies": [ { "id": 616153, "postDate": "2019-09-02T18:43:03.570Z", "content": "Thanks for reading!
", "rawMarkdown": "Thanks for reading!", "votes": 2 } ] }, { "id": 614995, "postDate": "2019-09-01T10:35:39.997Z", "content": "Good job!\nThanks for sharing..
", "rawMarkdown": "Good job!\nThanks for sharing..", "votes": 3, "replies": [ { "id": 616155, "postDate": "2019-09-02T18:44:39.083Z", "content": "Hi, Saad.\nHope this solution is easy to read for you :)
", "rawMarkdown": "Hi, Saad.\nHope this solution is easy to read for you :)", "votes": 2 } ] }, { "id": 614952, "postDate": "2019-09-01T09:36:33.477Z", "content": "Bravo!
", "rawMarkdown": "Bravo!", "votes": 3, "replies": [ { "id": 619232, "postDate": "2019-09-06T01:29:52.380Z", "content": "Bravo!! Thanks :)
", "rawMarkdown": "Bravo!! Thanks :)", "votes": 1 } ] }, { "id": 614738, "postDate": "2019-09-01T01:27:48.680Z", "content": "Great work. Happy to see your team do so well. It's well deserved.
", "rawMarkdown": "Great work. Happy to see your team do so well. It's well deserved.", "votes": 3, "replies": [ { "id": 616154, "postDate": "2019-09-02T18:44:10.943Z", "content": "Thanks for kind words !\nYou did also great job :) I've read your kernels which showed what's going on the competitions.
", "rawMarkdown": "Thanks for kind words !\nYou did also great job :) I've read your kernels which showed what's going on the competitions.", "votes": 3 } ] }, { "id": 614396, "postDate": "2019-08-31T12:40:56.043Z", "content": "Thank you for sharing unique method, congratulations!
", "rawMarkdown": "Thank you for sharing unique method, congratulations! ", "votes": 3, "replies": [ { "id": 616151, "postDate": "2019-09-02T18:42:24.393Z", "content": "Thanks!
\n\nHope this solution inspires you !
", "rawMarkdown": "Thanks! \n\nHope this solution inspires you !", "votes": 3 } ] }, { "id": 614313, "postDate": "2019-08-31T10:48:06.970Z", "content": "Absolutely amazing Youhan & your team!! @youhanlee !!
", "rawMarkdown": "Absolutely amazing Youhan & your team!! @youhanlee !!", "votes": 3, "replies": [ { "id": 616150, "postDate": "2019-09-02T18:42:04.517Z", "content": "Hi, Thanks for kinds words!
", "rawMarkdown": "Hi, Thanks for kinds words!", "votes": 3 } ] }, { "id": 613884, "postDate": "2019-08-30T22:24:30.130Z", "content": "Amazing job! I have little for the domain knowledge, but you have been a great collaboration. \nAt last, you got it. Again, Congraturations on yours!
", "rawMarkdown": "Amazing job! I have little for the domain knowledge, but you have been a great collaboration. \nAt last, you got it. Again, Congraturations on yours! ", "votes": 3, "replies": [ { "id": 614742, "postDate": "2019-09-01T01:36:12.337Z", "content": "I'm really lucky. Our team work was wonderful :)\nThanks for kind words!
", "rawMarkdown": "I'm really lucky. Our team work was wonderful :)\nThanks for kind words!", "votes": 3 } ] }, { "id": 613750, "postDate": "2019-08-30T19:29:51.470Z", "content": "Congrats @youhanlee and thank you for sharing. One question. how is position defined in \"embedding for position\"?
", "rawMarkdown": "Congrats @youhanlee and thank you for sharing. One question. how is position defined in \"embedding for position\"?", "votes": 3, "replies": [ { "id": 617826, "postDate": "2019-09-04T14:08:43.160Z", "content": "Thanks for kind words!
\n\nWe just embeded the xyz-coordinate (3 dimension) into 256 dimension!
", "rawMarkdown": "Thanks for kind words!\n\nWe just embeded the xyz-coordinate (3 dimension) into 256 dimension!", "votes": 1 }, { "id": 745664, "postDate": "2020-02-14T03:57:09.353Z", "content": "So xyz * A, where A is a matrix and its size is (3, 256)?
", "rawMarkdown": "So xyz * A, where A is a matrix and its size is (3, 256)?" } ] }, { "id": 613723, "postDate": "2019-08-30T19:00:18.603Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 3, "replies": [ { "id": 613931, "postDate": "2019-08-31T00:36:08.920Z", "content": "Thanks!! Very happy now :)
", "rawMarkdown": "Thanks!! Very happy now :)", "votes": 3 } ] }, { "id": 613628, "postDate": "2019-08-30T17:11:49.970Z", "content": "Congrats for the amazing solution! Thank you for sharing. @youhanlee Could you share the source code with us?
", "rawMarkdown": "Congrats for the amazing solution! Thank you for sharing. @youhanlee Could you share the source code with us?", "votes": 3, "replies": [ { "id": 613933, "postDate": "2019-08-31T00:37:47.927Z", "content": "Thanks for kind words! As illustrated in description, we will follow the policy of the CHAMPS.
", "rawMarkdown": "Thanks for kind words! As illustrated in description, we will follow the policy of the CHAMPS.", "votes": 2 } ] }, { "id": 613625, "postDate": "2019-08-30T17:10:12.117Z", "content": "Rarely have I seen a solution as complex as your team's. It's impressive. Thanks for sharing and congratulations.
", "rawMarkdown": "Rarely have I seen a solution as complex as your team's. It's impressive. Thanks for sharing and congratulations.", "votes": 3, "replies": [ { "id": 613938, "postDate": "2019-08-31T00:40:45.983Z", "content": "We did because we're team :) Thanks!
", "rawMarkdown": "We did because we're team :) Thanks!", "votes": 2 } ] }, { "id": 613599, "postDate": "2019-08-30T16:44:34.470Z", "content": "Fantastic teamwork for a very innovative approach, congrats! Hope to see you again in future competitions :-)
", "rawMarkdown": "Fantastic teamwork for a very innovative approach, congrats! Hope to see you again in future competitions :-)", "votes": 3, "replies": [ { "id": 613935, "postDate": "2019-08-31T00:38:43.847Z", "content": "Thanks for kind words!\nI will keep going on until GM.\nWe will meet always :)
", "rawMarkdown": "Thanks for kind words!\nI will keep going on until GM.\nWe will meet always :)", "votes": 4 } ] }, { "id": 613588, "postDate": "2019-08-30T16:34:24.557Z", "content": "Thanks for sharing - so interesting. Congrats to you and your teammates @youhanlee @sunghwanchoi @songwonho @limerobot @yslee04 .
", "rawMarkdown": "Thanks for sharing - so interesting. Congrats to you and your teammates @youhanlee @sunghwanchoi @songwonho @limerobot @yslee04 .", "votes": 3, "replies": [ { "id": 613941, "postDate": "2019-08-31T00:42:11.410Z", "content": "Thanks for kind words :)\nMy teamates are amazing !
", "rawMarkdown": "Thanks for kind words :)\nMy teamates are amazing !", "votes": 2 } ] }, { "id": 613416, "postDate": "2019-08-30T13:38:58.183Z", "content": "really impressive !
", "rawMarkdown": "really impressive !", "votes": 3, "replies": [ { "id": 613936, "postDate": "2019-08-31T00:39:22.610Z", "content": "Thanks for kind words.\nOur team is amazing! I'm lucky man
", "rawMarkdown": "Thanks for kind words.\nOur team is amazing! I'm lucky man", "votes": 2 } ] }, { "id": 613388, "postDate": "2019-08-30T13:14:01.483Z", "content": "Congratulations! Thanks for sharing :)
", "rawMarkdown": "Congratulations! Thanks for sharing :)", "votes": 3, "replies": [ { "id": 613937, "postDate": "2019-08-31T00:39:46.973Z", "content": "Thanks for kind words :)
", "rawMarkdown": "Thanks for kind words :)", "votes": 2 } ] }, { "id": 613297, "postDate": "2019-08-30T12:00:38.850Z", "content": "wow the idea is very inspiring! Congratulations and thank you for sharing this.
", "rawMarkdown": "wow the idea is very inspiring! Congratulations and thank you for sharing this.", "votes": 3, "replies": [ { "id": 613942, "postDate": "2019-08-31T00:43:28.553Z", "content": "Many discussions brought us amazing ideas :)\nThanks!!
", "rawMarkdown": "Many discussions brought us amazing ideas :)\nThanks!!", "votes": 3 } ] }, { "id": 613158, "postDate": "2019-08-30T09:09:06.837Z", "content": "Thanks a lot for sharing! Very insightful
", "rawMarkdown": "Thanks a lot for sharing! Very insightful", "votes": 3, "replies": [ { "id": 613944, "postDate": "2019-08-31T00:44:02.957Z", "content": "Thansk fo kind words :)\nHope this post is helpful to you!
", "rawMarkdown": "Thansk fo kind words :)\nHope this post is helpful to you!", "votes": 2 } ] }, { "id": 613117, "postDate": "2019-08-30T08:22:27.407Z", "content": "Wow. Very very similar to the Molecule Transformer we left unfinished, and I am surprised (and a bit frustrated) that using auxiliary task of individual contributions worked for you and not for us, although I must say your approach of predicting both separately and taking the mean is better than what we tried (just predicting the individual contributions and adding them later).
\n\nGreat job!
", "rawMarkdown": "Wow. Very very similar to the Molecule Transformer we left unfinished, and I am surprised (and a bit frustrated) that using auxiliary task of individual contributions worked for you and not for us, although I must say your approach of predicting both separately and taking the mean is better than what we tried (just predicting the individual contributions and adding them later). \n\nGreat job!", "votes": 3, "replies": [ { "id": 613946, "postDate": "2019-08-31T00:46:25.180Z", "content": "Hi, I really want to appreciate you!\nYoy showed the posibillities to our team.\nIt is very interesting that 1st, 2nd and 3rd team used the transformer :)\nYou did amazing job !
", "rawMarkdown": "Hi, I really want to appreciate you!\nYoy showed the posibillities to our team.\nIt is very interesting that 1st, 2nd and 3rd team used the transformer :)\nYou did amazing job !", "votes": 3 } ] }, { "id": 613106, "postDate": "2019-08-30T08:15:00.693Z", "content": "Ingenious,\nthanks for sharing
", "rawMarkdown": "Ingenious,\nthanks for sharing", "votes": 3, "replies": [ { "id": 614737, "postDate": "2019-09-01T01:27:03.757Z", "content": "Thanks for kind words!
", "rawMarkdown": "Thanks for kind words!", "votes": 2 } ] }, { "id": 613102, "postDate": "2019-08-30T08:11:15.350Z", "content": "Great! And I appreciate the very comprehensive description!
", "rawMarkdown": "Great! And I appreciate the very comprehensive description!", "votes": 3, "replies": [ { "id": 614736, "postDate": "2019-09-01T01:26:49.993Z", "content": "Thanks! Hope this post is useful to you!
", "rawMarkdown": "Thanks! Hope this post is useful to you!", "votes": 2 } ] }, { "id": 613070, "postDate": "2019-08-30T07:48:24.300Z", "content": "Amazing and very inspiring solution. Congrats @youhanlee and your teamates !
\n\nAnd kudos to @limerobot coming up with such great idea.
", "rawMarkdown": "Amazing and very inspiring solution. Congrats @youhanlee and your teamates !\n\nAnd kudos to @limerobot coming up with such great idea. ", "votes": 3, "replies": [ { "id": 614735, "postDate": "2019-09-01T01:26:24.620Z", "content": "Thanks for kind words :) \nAs you said, limerobot is amazing!
", "rawMarkdown": "Thanks for kind words :) \nAs you said, limerobot is amazing!", "votes": 2 } ] }, { "id": 613046, "postDate": "2019-08-30T07:23:29.277Z", "content": "Congrats @youhanlee and your team! Great solution and thanks for sharing!
", "rawMarkdown": "Congrats @youhanlee and your team! Great solution and thanks for sharing!", "votes": 3, "replies": [ { "id": 614734, "postDate": "2019-09-01T01:25:57.970Z", "content": "Thanks for kind words! Hope this post is easy to read!
", "rawMarkdown": "Thanks for kind words! Hope this post is easy to read!", "votes": 1 } ] }, { "id": 612996, "postDate": "2019-08-30T06:39:14.943Z", "content": "Wow, what an innovative solution! Congrats!
", "rawMarkdown": "Wow, what an innovative solution! Congrats!", "votes": 3, "replies": [ { "id": 613031, "postDate": "2019-08-30T07:04:07.377Z", "content": "Thanks for kind words :)
", "rawMarkdown": "Thanks for kind words :)", "votes": 2 } ] }, { "id": 612894, "postDate": "2019-08-30T04:20:50.627Z", "content": "Thank you for the sharing BERT is absolutely amazing idea
", "rawMarkdown": "Thank you for the sharing BERT is absolutely amazing idea", "votes": 3, "replies": [ { "id": 613022, "postDate": "2019-08-30T07:02:27.713Z", "content": "Limerobot is amazing!
\n\nMaybe BERT can be applied in other applications!
", "rawMarkdown": "Limerobot is amazing! \n\nMaybe BERT can be applied in other applications!", "votes": 2 } ] }, { "id": 612891, "postDate": "2019-08-30T04:18:37.797Z", "content": "Inspiring. Thanks for sharing.
", "rawMarkdown": "Inspiring. Thanks for sharing.", "votes": 3, "replies": [ { "id": 613023, "postDate": "2019-08-30T07:03:03.640Z", "content": "Thanks! And congratulations!
", "rawMarkdown": "Thanks! And congratulations!", "votes": 3 } ] }, { "id": 612882, "postDate": "2019-08-30T04:05:27.013Z", "content": "Big Congrats! You really deserve the long overdue gold medal!!
\n\nI have some questions.\n1. Is the BERT model fine tuned from pretrain or trained from scratch?\n2. What is the LB of a single BERT model without pseudo labels? And how long does it take on 4 V100?
\n\nThank you very much!
", "rawMarkdown": "Big Congrats! You really deserve the long overdue gold medal!!\n\nI have some questions.\n1. Is the BERT model fine tuned from pretrain or trained from scratch?\n2. What is the LB of a single BERT model without pseudo labels? And how long does it take on 4 V100?\n\nThank you very much!", "votes": 3, "replies": [ { "id": 613018, "postDate": "2019-08-30T07:01:08.800Z", "content": "Hi, thanks for kind words :)
\n\nCongratulations! Amazing solutions! :-)
", "rawMarkdown": "Congratulations! Amazing solutions! :-)", "votes": 3, "replies": [ { "id": 613024, "postDate": "2019-08-30T07:03:20.680Z", "content": "Thanks for kind words :)
", "rawMarkdown": "Thanks for kind words :)", "votes": 3 } ] }, { "id": 612798, "postDate": "2019-08-30T02:23:41.417Z", "content": "Amazing ideas
", "rawMarkdown": "Amazing ideas", "votes": 3, "replies": [ { "id": 613029, "postDate": "2019-08-30T07:04:03.597Z", "content": "Thanks for kind words :)
", "rawMarkdown": "Thanks for kind words :)", "votes": 2 } ] }, { "id": 615143, "postDate": "2019-09-01T14:17:44.533Z", "content": "good
", "rawMarkdown": "good\n", "votes": 4, "replies": [ { "id": 619233, "postDate": "2019-09-06T01:29:56.920Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 613715, "postDate": "2019-08-30T18:53:31.180Z", "content": "Congratulations! Very interesting explanation! Thanks for sharing this!
", "rawMarkdown": "Congratulations! Very interesting explanation! Thanks for sharing this!", "votes": 4, "replies": [ { "id": 613940, "postDate": "2019-08-31T00:41:30.813Z", "content": "Hope our explanation is easy to read for you! Thanks!
", "rawMarkdown": "Hope our explanation is easy to read for you! Thanks!", "votes": 2 } ] }, { "id": 612820, "postDate": "2019-08-30T03:00:26.710Z", "content": "congratulations，you guys are so amazing and totally blow me away.
", "rawMarkdown": "congratulations，you guys are so amazing and totally blow me away.", "votes": 4, "replies": [ { "id": 613025, "postDate": "2019-08-30T07:03:47.353Z", "content": "You did an amazing job as well :) congrats!
", "rawMarkdown": "You did an amazing job as well :) congrats!", "votes": 2 } ] }, { "id": 612790, "postDate": "2019-08-30T02:21:15.337Z", "content": "congratulations for winning the gold medal :)
", "rawMarkdown": "congratulations for winning the gold medal :)", "votes": 4, "replies": [ { "id": 613026, "postDate": "2019-08-30T07:03:53.187Z", "content": "Thanks for kind words :)
", "rawMarkdown": "Thanks for kind words :)", "votes": 2 } ] }, { "id": 612784, "postDate": "2019-08-30T02:18:02.370Z", "content": "amazing ideas,congratulations!
", "rawMarkdown": "amazing ideas,congratulations!", "votes": 4, "replies": [ { "id": 613027, "postDate": "2019-08-30T07:03:58.623Z", "content": "Thanks for kind words :)
", "rawMarkdown": "Thanks for kind words :)", "votes": 2 } ] }, { "id": 612780, "postDate": "2019-08-30T02:09:25.893Z", "content": "Very thanks to our teams @limerobot @sunghwanchoi @songwonho @yslee04
", "rawMarkdown": "Very thanks to our teams @limerobot @sunghwanchoi @songwonho @yslee04 ", "votes": 4 }, { "id": 616747, "postDate": "2019-09-03T12:24:12.743Z", "content": "Thanks for sharing 😄
", "rawMarkdown": "Thanks for sharing 😄 ", "votes": 2, "replies": [ { "id": 619243, "postDate": "2019-09-06T01:32:16.650Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 619306, "postDate": "2019-09-06T03:56:55.397Z", "content": "Congrats👍
", "rawMarkdown": "Congrats👍", "votes": 1 }, { "id": 619290, "postDate": "2019-09-06T03:31:45.627Z", "content": "유한님. 멋지십니다. 수상도 축하드려요. ㅎㅎㅎ 솔루션을 잘 정리하신 것 같은데, 아직 이해를 못하겠어서 안타깝네요. ㅎㅎㅎㅎ 얼른 실력이 좋아졌으면 ㅎㅎㅎㅎ
", "rawMarkdown": "유한님. 멋지십니다. 수상도 축하드려요. ㅎㅎㅎ 솔루션을 잘 정리하신 것 같은데, 아직 이해를 못하겠어서 안타깝네요. ㅎㅎㅎㅎ 얼른 실력이 좋아졌으면 ㅎㅎㅎㅎ ", "votes": 1 }, { "id": 619110, "postDate": "2019-09-05T20:47:17.307Z", "content": "Good sharing. Congratulation 👍
", "rawMarkdown": "Good sharing. Congratulation 👍 ", "votes": 2, "replies": [ { "id": 619229, "postDate": "2019-09-06T01:29:23.927Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 2 } ] }, { "id": 618973, "postDate": "2019-09-05T17:40:02.043Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 2, "replies": [ { "id": 619230, "postDate": "2019-09-06T01:29:27.763Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 618873, "postDate": "2019-09-05T15:23:22.350Z", "content": "Great Work..!!
", "rawMarkdown": "Great Work..!!", "votes": 2, "replies": [ { "id": 619231, "postDate": "2019-09-06T01:29:36.027Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 618792, "postDate": "2019-09-05T13:52:16.857Z", "content": "Good
", "rawMarkdown": "Good", "votes": 2, "replies": [ { "id": 619239, "postDate": "2019-09-06T01:31:14.297Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 617709, "postDate": "2019-09-04T12:02:31.767Z", "content": "Awesome, thanks for the write-up!
", "rawMarkdown": "Awesome, thanks for the write-up!", "votes": 2, "replies": [ { "id": 617821, "postDate": "2019-09-04T14:07:10.317Z", "content": "Thanks for kind words!\nHope this post is easy to read!
", "rawMarkdown": "Thanks for kind words!\nHope this post is easy to read!", "votes": 2 } ] }, { "id": 617624, "postDate": "2019-09-04T10:21:55.537Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": 2, "replies": [ { "id": 617820, "postDate": "2019-09-04T14:06:55.277Z", "content": "Thanks for kind words!
", "rawMarkdown": "Thanks for kind words!", "votes": 2 } ] }, { "id": 617014, "postDate": "2019-09-03T17:07:44.667Z", "content": "Congratulations and thanks a lot for sharing! One question, how do you define position embedding since (x,y,z) is already numeric?
", "rawMarkdown": "Congratulations and thanks a lot for sharing! One question, how do you define position embedding since (x,y,z) is already numeric?", "votes": 2 }, { "id": 616489, "postDate": "2019-09-03T07:15:39.907Z", "content": "Congrats @youhanlee and your teammates @sunghwanchoi @songwonho @limerobot @yslee04 . Amazingly easy to understand writeup. I hope and wish if you could share your source code in the future.
", "rawMarkdown": "Congrats @youhanlee and your teammates @sunghwanchoi @songwonho @limerobot @yslee04 . Amazingly easy to understand writeup. I hope and wish if you could share your source code in the future.", "votes": 2, "replies": [ { "id": 619236, "postDate": "2019-09-06T01:30:50.567Z", "content": "Good to hear that this post is easy to read for you. We will follow the host and kaggle's guidelines :)
", "rawMarkdown": "Good to hear that this post is easy to read for you. We will follow the host and kaggle's guidelines :)", "votes": 2 } ] }, { "id": 614443, "postDate": "2019-08-31T13:48:41.573Z", "content": "mitrondata.com .Can you able to help this.?
", "rawMarkdown": "[mitrondata.com](www.mitrondata.com) .Can you able to help this.?", "votes": 2 }, { "id": 613902, "postDate": "2019-08-30T23:19:10.383Z", "content": "Really amazing solution! Thank you for sharing😄
", "rawMarkdown": "Really amazing solution! Thank you for sharing😄 ", "votes": 2, "replies": [ { "id": 617827, "postDate": "2019-09-04T14:08:58.533Z", "content": "Thanks for kind words!
", "rawMarkdown": "Thanks for kind words!", "votes": 1 } ] }, { "id": 613769, "postDate": "2019-08-30T19:51:42.167Z", "content": "amazing solution
", "rawMarkdown": "amazing solution", "votes": 2, "replies": [ { "id": 616152, "postDate": "2019-09-02T18:42:49.107Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 2 } ] }, { "id": 3131871, "postDate": "2025-02-23T11:51:12.837Z", "content": "Great work! Tackling the bottleneck in NMR by replacing costly quantum calculations with ML for predicting scalar couplings is impressive. Excited to see how ML enhances both precision and scalability in real-world chemistry. I'm particularly interested in this intersection and actively applying these techniques myself!
", "rawMarkdown": "Great work! Tackling the bottleneck in NMR by replacing costly quantum calculations with ML for predicting scalar couplings is impressive. Excited to see how ML enhances both precision and scalability in real-world chemistry. I'm particularly interested in this intersection and actively applying these techniques myself!" }, { "id": 936179, "postDate": "2020-07-20T03:39:40.473Z", "content": "Great solution!
", "rawMarkdown": "Great solution!" }, { "id": 772083, "postDate": "2020-03-15T01:36:43.427Z", "content": "Well, great work! I wonder whether there is any publication on this scalar coupling constant prediction? I am looking forward to reading your works
", "rawMarkdown": "Well, great work! I wonder whether there is any publication on this scalar coupling constant prediction? I am looking forward to reading your works" }, { "id": 620382, "postDate": "2019-09-07T12:22:41.013Z", "content": "Congratulations!!! great job:-)
", "rawMarkdown": "Congratulations!!! great job:-)" }, { "id": 620286, "postDate": "2019-09-07T08:51:13.627Z", "content": "A very detailed explanation. And congrats. :D
", "rawMarkdown": "A very detailed explanation. And congrats. :D" }, { "id": 619873, "postDate": "2019-09-06T17:21:43.810Z", "content": "Great Work!
", "rawMarkdown": "Great Work!" }, { "id": 619952, "postDate": "2019-09-06T19:58:49.930Z", "rawMarkdown": "", "isDeleted": true }, { "id": 618549, "postDate": "2019-09-05T09:37:06.923Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 619238, "postDate": "2019-09-06T01:31:12.963Z", "content": "Thanks for kind words !
", "rawMarkdown": "Thanks for kind words !", "votes": 1 } ] }, { "id": 616731, "postDate": "2019-09-03T12:05:13.517Z", "rawMarkdown": "", "votes": 3, "isDeleted": true, "replies": [ { "id": 619242, "postDate": "2019-09-06T01:32:07.227Z", "content": "Thanks for kind words!
", "rawMarkdown": "Thanks for kind words!", "votes": 1 } ] }, { "id": 614898, "postDate": "2019-09-01T07:59:51.433Z", "rawMarkdown": "", "isDeleted": true }, { "id": 926988, "postDate": "2020-07-13T06:17:17.560Z", "content": "Thank you for sharing a great idea!
", "rawMarkdown": "Thank you for sharing a great idea!" }, { "id": 787762, "postDate": "2020-03-27T04:09:39.733Z", "content": "thanks for your discussion
", "rawMarkdown": "thanks for your discussion" }, { "id": 620082, "postDate": "2019-09-07T02:02:34.880Z", "content": "Thank you
", "rawMarkdown": "Thank you" } ], "comments": [ { "id": 612896, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-08-30T04:27:39.083000", "content": "Congratulations.....\nAmazing ideas...\nInspiring. Thanks for sharing.... @youhanlee
", "votes": 6, "replies": [ { "id": 613020, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-30T07:01:36.013000", "content": "Thanks!
\n\nI've shocked for the performance of BERT :)
", "votes": 2, "replies": [] } ] }, { "id": 613203, "author_name": "Dan Ofer", "author_url": "", "post_date": "2019-08-30T09:47:50.623000", "content": "How did you do embeddings on continous variables such as distance? Binning?\nHow did you get 32 size embedding on charge, that's an ordinalvariable with less than a dozen possible values?
", "votes": 5, "replies": [ { "id": 613948, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:49:06.383000", "content": "We just embedded the continous value into 32 or 64 size not using binning.
\n\nThe 32 size was arbitrary. We wanted to optimize the size, but not enough time.
\n\nWe can expect that there are optimal size of embeddings for various inputs.
", "votes": 3, "replies": [] }, { "id": 615055, "author_name": "Dan Ofer", "author_url": "", "post_date": "2019-09-01T12:09:36.753000", "content": "So you turned 1 column into 32 columns/variables in the net?\nI'm probably missing something about your setup - I don't see how that would have improved things vs just inputting the number as a variable.
\n\nReally interesting overall!
", "votes": 2, "replies": [] } ] }, { "id": 613144, "author_name": "KhanhVD", "author_url": "", "post_date": "2019-08-30T09:00:04.867000", "content": "Congrats @youhanlee and your team for Amazing Solution!!
", "votes": 6, "replies": [ { "id": 614741, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-01T01:35:05.160000", "content": "Thanks for kind words!
", "votes": 2, "replies": [] } ] }, { "id": 613215, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-30T10:05:56.133000", "content": "Thanks for sharing, and congrats on the well deserved end result. Again, using raw coordinates works surprisingly well. I wonder if there isn't some information leak in the way molecules are built here,. Would you and 2nd team get the same result if random rotations were applied to input molecules?
\n\nAnyway, your solution is very original and works amazingly well. I'll reread to grasp all details ;)
", "votes": 4, "replies": [ { "id": 614733, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-01T01:22:44.177000", "content": "Thanks for kind words.
\n\nWe were also surprized about the performance of using raw coodinates. \nI don't think any leak in making the structure of molecules. The molecules might be optimized using QM calculations.
\n\nAnyway, we used the rotation matrics which were not too much to rotate molecules.\nSo, Rotated molecules showed very similar coupling constants compared to that of original structures.
", "votes": 5, "replies": [] } ] }, { "id": 614819, "author_name": "Mikhail Karchevskiy", "author_url": "", "post_date": "2019-09-01T05:47:21.440000", "content": "Great solution!\nOne question, did you use tensorflow or pytorch?
", "votes": 5, "replies": [ { "id": 616149, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-02T18:41:27.933000", "content": "Thanks! \nHi, we used pytorch_transformer (https://github.com/huggingface/pytorch-transformers).
", "votes": 4, "replies": [] }, { "id": 616605, "author_name": "Mikhail Karchevskiy", "author_url": "", "post_date": "2019-09-03T09:28:26.317000", "content": "thanks!
", "votes": 2, "replies": [] } ] }, { "id": 930163, "author_name": "AkiraSosa", "author_url": "", "post_date": "2020-07-15T08:33:27.133000", "content": "I have a question about \"input sequence\". You show that the max coupling of the input sequence is 135. I suppose that this number is from the total number of coupling constants in molecule dsgdb9nsd_042139.
\n\nSo, is your \"input sequence\" a set of coupling constants in a molecule?
\n\nHowever, if the answer is true, your input does not contain some information, such as oxygen in molecule, because scalar coupling type is only for H, C and N.
", "votes": 1, "replies": [] }, { "id": 617858, "author_name": "Anubrata Bhowmick", "author_url": "", "post_date": "2019-09-04T14:43:48.460000", "content": "This is awesome!
", "votes": 3, "replies": [ { "id": 619240, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-06T01:31:23.353000", "content": "Thanks for kind words !
", "votes": 1, "replies": [] } ] }, { "id": 617351, "author_name": "ZexyZeky", "author_url": "", "post_date": "2019-09-04T03:30:49.340000", "content": "Woww.. amazing solution
", "votes": 3, "replies": [ { "id": 617819, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-04T14:06:49.760000", "content": "Thanks for kind words!
", "votes": 2, "replies": [] } ] }, { "id": 616930, "author_name": "Saikat Biswas", "author_url": "", "post_date": "2019-09-03T15:45:47.457000", "content": "Simply Wow...!!!\nSo, you just described the journey on this competition and along with it the whole thought process.\nBig Congratulations on your win and thanks a lot for sharing this..!!!\nAwesome read...!!!
", "votes": 3, "replies": [ { "id": 619244, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-06T01:32:46.717000", "content": "Thanks for kind words! \nHope this post is informative to you!
", "votes": 1, "replies": [] } ] }, { "id": 616699, "author_name": "Psi", "author_url": "", "post_date": "2019-09-03T11:26:46.400000", "content": "I love this! Congratz!
", "votes": 3, "replies": [ { "id": 619241, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-06T01:31:56.687000", "content": "I've always liked your solutions :) Thanks for kind words!
", "votes": 1, "replies": [] } ] }, { "id": 616516, "author_name": "Rahul Pareek", "author_url": "", "post_date": "2019-09-03T07:48:39.287000", "content": "Great Solution.
", "votes": 3, "replies": [ { "id": 619237, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-06T01:30:56.547000", "content": "Thanks for kind words !
", "votes": 1, "replies": [] } ] }, { "id": 616455, "author_name": "Rashidul H", "author_url": "", "post_date": "2019-09-03T06:32:35.337000", "content": "Congrats and thanks a lot for sharing.
", "votes": 3, "replies": [ { "id": 619234, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-06T01:30:08.793000", "content": "Thanks for kind words !
", "votes": 1, "replies": [] } ] }, { "id": 615927, "author_name": "Saurabh Kumar", "author_url": "", "post_date": "2019-09-02T14:13:39.550000", "content": "Thanks for sharing.
", "votes": 3, "replies": [ { "id": 616153, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-02T18:43:03.570000", "content": "Thanks for reading!
", "votes": 2, "replies": [] } ] }, { "id": 614995, "author_name": "Saad Aldin", "author_url": "", "post_date": "2019-09-01T10:35:39.997000", "content": "Good job!\nThanks for sharing..
", "votes": 3, "replies": [ { "id": 616155, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-02T18:44:39.083000", "content": "Hi, Saad.\nHope this solution is easy to read for you :)
", "votes": 2, "replies": [] } ] }, { "id": 614952, "author_name": "Sebastian Korsak", "author_url": "", "post_date": "2019-09-01T09:36:33.477000", "content": "Bravo!
", "votes": 3, "replies": [ { "id": 619232, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-06T01:29:52.380000", "content": "Bravo!! Thanks :)
", "votes": 1, "replies": [] } ] }, { "id": 614738, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-09-01T01:27:48.680000", "content": "Great work. Happy to see your team do so well. It's well deserved.
", "votes": 3, "replies": [ { "id": 616154, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-02T18:44:10.943000", "content": "Thanks for kind words !\nYou did also great job :) I've read your kernels which showed what's going on the competitions.
", "votes": 3, "replies": [] } ] }, { "id": 614396, "author_name": "Hayato", "author_url": "", "post_date": "2019-08-31T12:40:56.043000", "content": "Thank you for sharing unique method, congratulations!
", "votes": 3, "replies": [ { "id": 616151, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-02T18:42:24.393000", "content": "Thanks!
\n\nHope this solution inspires you !
", "votes": 3, "replies": [] } ] }, { "id": 614313, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-08-31T10:48:06.970000", "content": "Absolutely amazing Youhan & your team!! @youhanlee !!
", "votes": 3, "replies": [ { "id": 616150, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-02T18:42:04.517000", "content": "Hi, Thanks for kinds words!
", "votes": 3, "replies": [] } ] }, { "id": 613884, "author_name": "Wgyeong", "author_url": "", "post_date": "2019-08-30T22:24:30.130000", "content": "Amazing job! I have little for the domain knowledge, but you have been a great collaboration. \nAt last, you got it. Again, Congraturations on yours!
", "votes": 3, "replies": [ { "id": 614742, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-01T01:36:12.337000", "content": "I'm really lucky. Our team work was wonderful :)\nThanks for kind words!
", "votes": 3, "replies": [] } ] }, { "id": 613750, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-30T19:29:51.470000", "content": "Congrats @youhanlee and thank you for sharing. One question. how is position defined in \"embedding for position\"?
", "votes": 3, "replies": [ { "id": 617826, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-04T14:08:43.160000", "content": "Thanks for kind words!
\n\nWe just embeded the xyz-coordinate (3 dimension) into 256 dimension!
", "votes": 1, "replies": [] }, { "id": 745664, "author_name": "tamuhey", "author_url": "", "post_date": "2020-02-14T03:57:09.353000", "content": "So xyz * A, where A is a matrix and its size is (3, 256)?
", "votes": 0, "replies": [] } ] }, { "id": 613723, "author_name": "Murat Sarıbaş", "author_url": "", "post_date": "2019-08-30T19:00:18.603000", "content": "Congratulations!
", "votes": 3, "replies": [ { "id": 613931, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:36:08.920000", "content": "Thanks!! Very happy now :)
", "votes": 3, "replies": [] } ] }, { "id": 613628, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-30T17:11:49.970000", "content": "Congrats for the amazing solution! Thank you for sharing. @youhanlee Could you share the source code with us?
", "votes": 3, "replies": [ { "id": 613933, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:37:47.927000", "content": "Thanks for kind words! As illustrated in description, we will follow the policy of the CHAMPS.
", "votes": 2, "replies": [] } ] }, { "id": 613625, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2019-08-30T17:10:12.117000", "content": "Rarely have I seen a solution as complex as your team's. It's impressive. Thanks for sharing and congratulations.
", "votes": 3, "replies": [ { "id": 613938, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:40:45.983000", "content": "We did because we're team :) Thanks!
", "votes": 2, "replies": [] } ] }, { "id": 613599, "author_name": "A_b", "author_url": "", "post_date": "2019-08-30T16:44:34.470000", "content": "Fantastic teamwork for a very innovative approach, congrats! Hope to see you again in future competitions :-)
", "votes": 3, "replies": [ { "id": 613935, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:38:43.847000", "content": "Thanks for kind words!\nI will keep going on until GM.\nWe will meet always :)
", "votes": 4, "replies": [] } ] }, { "id": 613588, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T16:34:24.557000", "content": "Thanks for sharing - so interesting. Congrats to you and your teammates @youhanlee @sunghwanchoi @songwonho @limerobot @yslee04 .
", "votes": 3, "replies": [ { "id": 613941, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:42:11.410000", "content": "Thanks for kind words :)\nMy teamates are amazing !
", "votes": 2, "replies": [] } ] }, { "id": 613416, "author_name": "eagle4", "author_url": "", "post_date": "2019-08-30T13:38:58.183000", "content": "really impressive !
", "votes": 3, "replies": [ { "id": 613936, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:39:22.610000", "content": "Thanks for kind words.\nOur team is amazing! I'm lucky man
", "votes": 2, "replies": [] } ] }, { "id": 613388, "author_name": "Kwangnam Yu", "author_url": "", "post_date": "2019-08-30T13:14:01.483000", "content": "Congratulations! Thanks for sharing :)
", "votes": 3, "replies": [ { "id": 613937, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:39:46.973000", "content": "Thanks for kind words :)
", "votes": 2, "replies": [] } ] }, { "id": 613297, "author_name": "KelsJun", "author_url": "", "post_date": "2019-08-30T12:00:38.850000", "content": "wow the idea is very inspiring! Congratulations and thank you for sharing this.
", "votes": 3, "replies": [ { "id": 613942, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:43:28.553000", "content": "Many discussions brought us amazing ideas :)\nThanks!!
", "votes": 3, "replies": [] } ] }, { "id": 613158, "author_name": "Boris D.", "author_url": "", "post_date": "2019-08-30T09:09:06.837000", "content": "Thanks a lot for sharing! Very insightful
", "votes": 3, "replies": [ { "id": 613944, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-31T00:44:02.957000", "content": "Thansk fo kind words :)\nHope this post is helpful to you!
", "votes": 2, "replies": [] } ] }, { "id": 613117, "author_name": "", "author_url": "", "post_date": "2019-08-30T08:22:27.407000", "content": "", "votes": 3, "replies": [ { "id": 613946, "author_name": "", "author_url": "", "post_date": "2019-08-31T00:46:25.180000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 613106, "author_name": "", "author_url": "", "post_date": "2019-08-30T08:15:00.693000", "content": "", "votes": 3, "replies": [ { "id": 614737, "author_name": "", "author_url": "", "post_date": "2019-09-01T01:27:03.757000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 613102, "author_name": "", "author_url": "", "post_date": "2019-08-30T08:11:15.350000", "content": "", "votes": 3, "replies": [ { "id": 614736, "author_name": "", "author_url": "", "post_date": "2019-09-01T01:26:49.993000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 613070, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:48:24.300000", "content": "", "votes": 3, "replies": [ { "id": 614735, "author_name": "", "author_url": "", "post_date": "2019-09-01T01:26:24.620000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 613046, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:23:29.277000", "content": "", "votes": 3, "replies": [ { "id": 614734, "author_name": "", "author_url": "", "post_date": "2019-09-01T01:25:57.970000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 612996, "author_name": "", "author_url": "", "post_date": "2019-08-30T06:39:14.943000", "content": "", "votes": 3, "replies": [ { "id": 613031, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:04:07.377000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 612894, "author_name": "", "author_url": "", "post_date": "2019-08-30T04:20:50.627000", "content": "", "votes": 3, "replies": [ { "id": 613022, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:02:27.713000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 612891, "author_name": "", "author_url": "", "post_date": "2019-08-30T04:18:37.797000", "content": "", "votes": 3, "replies": [ { "id": 613023, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:03:03.640000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 612882, "author_name": "", "author_url": "", "post_date": "2019-08-30T04:05:27.013000", "content": "", "votes": 3, "replies": [ { "id": 613018, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:01:08.800000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 612836, "author_name": "", "author_url": "", "post_date": "2019-08-30T03:18:54.040000", "content": "", "votes": 3, "replies": [ { "id": 613024, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:03:20.680000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 612798, "author_name": "", "author_url": "", "post_date": "2019-08-30T02:23:41.417000", "content": "", "votes": 3, "replies": [ { "id": 613029, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:04:03.597000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 615143, "author_name": "", "author_url": "", "post_date": "2019-09-01T14:17:44.533000", "content": "", "votes": 4, "replies": [ { "id": 619233, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:29:56.920000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 613715, "author_name": "", "author_url": "", "post_date": "2019-08-30T18:53:31.180000", "content": "", "votes": 4, "replies": [ { "id": 613940, "author_name": "", "author_url": "", "post_date": "2019-08-31T00:41:30.813000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 612820, "author_name": "", "author_url": "", "post_date": "2019-08-30T03:00:26.710000", "content": "", "votes": 4, "replies": [ { "id": 613025, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:03:47.353000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 612790, "author_name": "", "author_url": "", "post_date": "2019-08-30T02:21:15.337000", "content": "", "votes": 4, "replies": [ { "id": 613026, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:03:53.187000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 612784, "author_name": "", "author_url": "", "post_date": "2019-08-30T02:18:02.370000", "content": "", "votes": 4, "replies": [ { "id": 613027, "author_name": "", "author_url": "", "post_date": "2019-08-30T07:03:58.623000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 612780, "author_name": "", "author_url": "", "post_date": "2019-08-30T02:09:25.893000", "content": "", "votes": 4, "replies": [] }, { "id": 616747, "author_name": "", "author_url": "", "post_date": "2019-09-03T12:24:12.743000", "content": "", "votes": 2, "replies": [ { "id": 619243, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:32:16.650000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 619306, "author_name": "", "author_url": "", "post_date": "2019-09-06T03:56:55.397000", "content": "", "votes": 1, "replies": [] }, { "id": 619290, "author_name": "", "author_url": "", "post_date": "2019-09-06T03:31:45.627000", "content": "", "votes": 1, "replies": [] }, { "id": 619110, "author_name": "", "author_url": "", "post_date": "2019-09-05T20:47:17.307000", "content": "", "votes": 2, "replies": [ { "id": 619229, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:29:23.927000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 618973, "author_name": "", "author_url": "", "post_date": "2019-09-05T17:40:02.043000", "content": "", "votes": 2, "replies": [ { "id": 619230, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:29:27.763000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 618873, "author_name": "", "author_url": "", "post_date": "2019-09-05T15:23:22.350000", "content": "", "votes": 2, "replies": [ { "id": 619231, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:29:36.027000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 618792, "author_name": "", "author_url": "", "post_date": "2019-09-05T13:52:16.857000", "content": "", "votes": 2, "replies": [ { "id": 619239, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:31:14.297000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 617709, "author_name": "", "author_url": "", "post_date": "2019-09-04T12:02:31.767000", "content": "", "votes": 2, "replies": [ { "id": 617821, "author_name": "", "author_url": "", "post_date": "2019-09-04T14:07:10.317000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 617624, "author_name": "", "author_url": "", "post_date": "2019-09-04T10:21:55.537000", "content": "", "votes": 2, "replies": [ { "id": 617820, "author_name": "", "author_url": "", "post_date": "2019-09-04T14:06:55.277000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 617014, "author_name": "", "author_url": "", "post_date": "2019-09-03T17:07:44.667000", "content": "", "votes": 2, "replies": [] }, { "id": 616489, "author_name": "", "author_url": "", "post_date": "2019-09-03T07:15:39.907000", "content": "", "votes": 2, "replies": [ { "id": 619236, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:30:50.567000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 614443, "author_name": "", "author_url": "", "post_date": "2019-08-31T13:48:41.573000", "content": "", "votes": 2, "replies": [] }, { "id": 613902, "author_name": "", "author_url": "", "post_date": "2019-08-30T23:19:10.383000", "content": "", "votes": 2, "replies": [ { "id": 617827, "author_name": "", "author_url": "", "post_date": "2019-09-04T14:08:58.533000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 613769, "author_name": "", "author_url": "", "post_date": "2019-08-30T19:51:42.167000", "content": "", "votes": 2, "replies": [ { "id": 616152, "author_name": "", "author_url": "", "post_date": "2019-09-02T18:42:49.107000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 3131871, "author_name": "", "author_url": "", "post_date": "2025-02-23T11:51:12.837000", "content": "", "votes": 0, "replies": [] }, { "id": 936179, "author_name": "", "author_url": "", "post_date": "2020-07-20T03:39:40.473000", "content": "", "votes": 0, "replies": [] }, { "id": 772083, "author_name": "", "author_url": "", "post_date": "2020-03-15T01:36:43.427000", "content": "", "votes": 0, "replies": [] }, { "id": 620382, "author_name": "", "author_url": "", "post_date": "2019-09-07T12:22:41.013000", "content": "", "votes": 0, "replies": [] }, { "id": 620286, "author_name": "", "author_url": "", "post_date": "2019-09-07T08:51:13.627000", "content": "", "votes": 0, "replies": [] }, { "id": 619873, "author_name": "", "author_url": "", "post_date": "2019-09-06T17:21:43.810000", "content": "", "votes": 0, "replies": [] }, { "id": 619952, "author_name": "", "author_url": "", "post_date": "2019-09-06T19:58:49.930000", "content": "", "votes": 0, "replies": [] }, { "id": 618549, "author_name": "", "author_url": "", "post_date": "2019-09-05T09:37:06.923000", "content": "", "votes": 2, "replies": [ { "id": 619238, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:31:12.963000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 616731, "author_name": "", "author_url": "", "post_date": "2019-09-03T12:05:13.517000", "content": "", "votes": 3, "replies": [ { "id": 619242, "author_name": "", "author_url": "", "post_date": "2019-09-06T01:32:07.227000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 614898, "author_name": "", "author_url": "", "post_date": "2019-09-01T07:59:51.433000", "content": "", "votes": 0, "replies": [] }, { "id": 926988, "author_name": "", "author_url": "", "post_date": "2020-07-13T06:17:17.560000", "content": "", "votes": 0, "replies": [] }, { "id": 787762, "author_name": "", "author_url": "", "post_date": "2020-03-27T04:09:39.733000", "content": "", "votes": 0, "replies": [] }, { "id": 620082, "author_name": "", "author_url": "", "post_date": "2019-09-07T02:02:34.880000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "612778": "Our story\n\nAs you know, we need to find the appropriate representation for data. That’s why we’re struggling to do feature engineering.\n\nAs CPMP mentioned, two member of our team have domain knowledge on this competition. \nSunghwan Choi has Ph.D at quantum chemistry and works on the field of quantum chemistry and chemical application of machine learning. \n\nI’m going on the Ph.D course in chemical engineering and have some experience for quantum calculation and dealing molecules. More important thing is that I’m kaggler :).\n\nWe thought that our problem can be solved by conventional graph models whose edge features are distance-derived properties. Those models are quite conventional in machine learning applications on chemical system. Hence, we tried to figure out appropriate hyper parameters and edge features for models\n\nAs many kagglers did, we also tested message passing neural networks by adopting many useful kernels. We thank to many kegglers; especially heng cher keng. :) by modification of their solution, we got a silver place 1 month ago. But, the gap between leading group and us was getting large. \n\nWe had to find out breakthrough. We brainstormed a lot. At that time, limerobot, who is an expert of natural language processing suggested to use the raw xyz coordinates, \nI and Sunghwan didn’t agree with that, because if we use xyz coordinates instead of distance, translational and rotational invariances are not satisfied. The model which do not preserve those invariances seems to be ridiculous\n \nBut, limerobot did on his way. And he showed that his model won my GNN. \nHis model was based on BERT model. You can see the big success in toxic competition on his profile. \n\nMaybe, insights from toxic competition save us :).\n\nAnyway, he don’t have any domain knowledge. He just wanted to make a model to learn the complex representation using xyz coordinates. (end-to-end)\n\nHe just input the xyz coordinates of atom1, atom2, coupling type, distance and difference of each xyz coordinates (he thought the model can learn the distance formula based on that...amazing)\n\nBecause we had only one month, we decided to do all things based on the transformer.\nAfter that, we did a number of experiment for hyper parameter tuning.\n\nBecause BERT is very large model, the performances differed according to the number of hidden layers, type of embedding and learning schedule.\n\nBefore 1 week to end of competition, we found our own scheduling and parameters and tiny modification of readout layer ( I will explain model later ) \n\nWe made multiples of models. Based on the ensemble of them, we got 3rd place :).\n\nThanks for reading our stories. We want to share the specific magics below. Please keep going :)\n\n# Overall architecture\n\nHere is our overall architecture for our model. As you can see, our input sequence is transformed into output sequence by BERT-encoder.\n\n•\tThe number of encoder layer: 8\n•\tThe number of heads for attention 8\n•\tDropout ratio is 0.1 which is conventional choice for BERT model. \n•\tFor each type, different readout networks but having same architecture are used\n\n\n \n\n\n\n# Input features and embedding layer for them\n\nOur float sequences composed of multiple embedding results, are the magic to get the achievement.\n\n \n\n\nMulti-head attention layer itself preserve permutational invariance therefore, order of couplings do not change the results but invariance when atomic_index_0 and atomic_index1 are changed, is not preserved since we use feature vector as concatenation of the embedding results of atomic charge, xyz coordinate(position), atomic number of two atoms. \n\n•\tSize of embedding for atomic charge: 32\n•\tSize of embedding for position: 256\n•\tSize of embedding for atomic number: 64\n•\tSize of embedding for distance: 64\n•\tSize of embedding for type: 64\n\nTotal feature size for single feature vector is (32+256+64)*2+64+64=832\n\n\n# Augmentation\nIn order to impose pseudo-invariance on our model, we use rotational and translational noise when augmenting data.\n\n•\tTranslational noise: For each axis, Gaussian noise( mean: 0, std: 2) was added\n•\tRotational noise: Rotational transformation whose axis is translational noise vector and angle is from Gaussian noise( mean:0, std: 3.14/2 )\n\n\n# Regression layers for predicting scalar coupling constants\n•\tAs you know that, spin-coupling (sc) value can be decomposed into four different terms (fc, sd, pso, dso)\n•\tAfter optimizing architecture and various losses, we found that auxiliary target using contributions gave a high boost.\n•\tDuring training, model minimize loss1 + loss2 with AdamW algorithm.\n•\tThere are 8 regression layers to cover 8 different coupling types\n \n\n\n\n# Specific learning rate\n•\tWe were always using linear learning rate decay.\n•\tWe think that there might be improvement with various learning schedule such as cycle lr. But, we didn’t have time because BERT is very large…(about 75M parameters. It took 1~2 days to get a model using 2~4 V100 machine)\n\n\n# Pseudo labeling \n•\tTo get the more results, we needed some magics. With having an insight that there are less probability to be overfitted (Sunghwan choi’s insight) and experiment result from limerobot, we adopted pseudo-labeling.\n•\tAfter predicting for test set, we used the pseudo-labeled test dataset for training.\n•\tThe model showed more than -3.4 CV. So, we trained model more with only train data to minigate overfitting. (finally we got ~-3.11 LB single models)\n•\tOverall learning process is illustrated below.\n \n\n\n\n# Final submission\n•\tWe made 14 models with various seed and hidden layers (most have 8 layers, other have 6 layers)\n•\tAfter weighted average according to cv score of types, we got -3.16.\n•\tAfter multiple procedure of pseudo labeling, we had 8 models. \n•\tWith simple average, we got -3.19 (our final score :) 2 hours before the end of competition.\n\n# What we’ve learned\n•\tEnd-to-End works!!!. Amazing BERT.\n•\tLearning schedule is very important for modelling of molecular property.\n\n", "612896": "Congratulations.....\nAmazing ideas...\nInspiring. Thanks for sharing.... @youhanlee \n", "613203": "How did you do embeddings on continous variables such as distance? Binning?\nHow did you get 32 size embedding on charge, that's an ordinalvariable with less than a dozen possible values?", "613144": "Congrats @youhanlee and your team for Amazing Solution!!", "613215": "Thanks for sharing, and congrats on the well deserved end result. Again, using raw coordinates works surprisingly well. I wonder if there isn't some information leak in the way molecules are built here,. Would you and 2nd team get the same result if random rotations were applied to input molecules?\n\nAnyway, your solution is very original and works amazingly well. I'll reread to grasp all details ;)", "614819": "Great solution!\nOne question, did you use tensorflow or pytorch?", "930163": "I have a question about \"input sequence\". You show that the max coupling of the input sequence is 135. I suppose that this number is from the total number of coupling constants in molecule dsgdb9nsd_042139.\n\nSo, is your \"input sequence\" a set of coupling constants in a molecule?\n\nHowever, if the answer is true, your input does not contain some information, such as oxygen in molecule, because scalar coupling type is only for H, C and N.", "617858": "This is awesome!", "617351": "Woww.. amazing solution", "616930": "Simply Wow...!!!\nSo, you just described the journey on this competition and along with it the whole thought process.\nBig Congratulations on your win and thanks a lot for sharing this..!!!\nAwesome read...!!!", "616699": "I love this! Congratz!", "616516": "Great Solution.", "616455": "**Congrats and thanks a lot for sharing.**", "615927": "Thanks for sharing.", "614995": "Good job!\nThanks for sharing..", "614952": "Bravo!", "614738": "Great work. Happy to see your team do so well. It's well deserved.", "614396": "Thank you for sharing unique method, congratulations! ", "614313": "Absolutely amazing Youhan & your team!! @youhanlee !!", "613884": "Amazing job! I have little for the domain knowledge, but you have been a great collaboration. \nAt last, you got it. Again, Congraturations on yours! ", "613750": "Congrats @youhanlee and thank you for sharing. One question. how is position defined in \"embedding for position\"?", "613723": "Congratulations!", "613628": "Congrats for the amazing solution! Thank you for sharing. @youhanlee Could you share the source code with us?", "613625": "Rarely have I seen a solution as complex as your team's. It's impressive. Thanks for sharing and congratulations.", "613599": "Fantastic teamwork for a very innovative approach, congrats! Hope to see you again in future competitions :-)", "613588": "Thanks for sharing - so interesting. Congrats to you and your teammates @youhanlee @sunghwanchoi @songwonho @limerobot @yslee04 .", "613416": "really impressive !", "613388": "Congratulations! Thanks for sharing :)", "613297": "wow the idea is very inspiring! Congratulations and thank you for sharing this.", "613158": "Thanks a lot for sharing! Very insightful", "613117": "Wow. Very very similar to the Molecule Transformer we left unfinished, and I am surprised (and a bit frustrated) that using auxiliary task of individual contributions worked for you and not for us, although I must say your approach of predicting both separately and taking the mean is better than what we tried (just predicting the individual contributions and adding them later). \n\nGreat job!", "613106": "Ingenious,\nthanks for sharing", "613102": "Great! And I appreciate the very comprehensive description!", "613070": "Amazing and very inspiring solution. Congrats @youhanlee and your teamates !\n\nAnd kudos to @limerobot coming up with such great idea. ", "613046": "Congrats @youhanlee and your team! Great solution and thanks for sharing!", "612996": "Wow, what an innovative solution! Congrats!", "612894": "Thank you for the sharing BERT is absolutely amazing idea", "612891": "Inspiring. Thanks for sharing.", "612882": "Big Congrats! You really deserve the long overdue gold medal!!\n\nI have some questions.\n1. Is the BERT model fine tuned from pretrain or trained from scratch?\n2. What is the LB of a single BERT model without pseudo labels? And how long does it take on 4 V100?\n\nThank you very much!", "612836": "Congratulations! Amazing solutions! :-)", "612798": "Amazing ideas", "615143": "good\n", "613715": "Congratulations! Very interesting explanation! Thanks for sharing this!", "612820": "congratulations，you guys are so amazing and totally blow me away.", "612790": "congratulations for winning the gold medal :)", "612784": "amazing ideas,congratulations!", "612780": "Very thanks to our teams @limerobot @sunghwanchoi @songwonho @yslee04 ", "616747": "Thanks for sharing 😄 ", "619306": "Congrats👍", "619290": "유한님. 멋지십니다. 수상도 축하드려요. ㅎㅎㅎ 솔루션을 잘 정리하신 것 같은데, 아직 이해를 못하겠어서 안타깝네요. ㅎㅎㅎㅎ 얼른 실력이 좋아졌으면 ㅎㅎㅎㅎ ", "619110": "Good sharing. Congratulation 👍 ", "618973": "Congrats!", "618873": "Great Work..!!", "618792": "Good", "617709": "Awesome, thanks for the write-up!", "617624": "thanks for sharing", "617014": "Congratulations and thanks a lot for sharing! One question, how do you define position embedding since (x,y,z) is already numeric?", "616489": "Congrats @youhanlee and your teammates @sunghwanchoi @songwonho @limerobot @yslee04 . Amazingly easy to understand writeup. I hope and wish if you could share your source code in the future.", "614443": "[mitrondata.com](www.mitrondata.com) .Can you able to help this.?", "613902": "Really amazing solution! Thank you for sharing😄 ", "613769": "amazing solution", "3131871": "Great work! Tackling the bottleneck in NMR by replacing costly quantum calculations with ML for predicting scalar couplings is impressive. Excited to see how ML enhances both precision and scalability in real-world chemistry. I'm particularly interested in this intersection and actively applying these techniques myself!", "936179": "Great solution!", "772083": "Well, great work! I wonder whether there is any publication on this scalar coupling constant prediction? I am looking forward to reading your works", "620382": "Congratulations!!! great job:-)", "620286": "A very detailed explanation. And congrats. :D", "619873": "Great Work!", "619952": "", "618549": "", "616731": "", "614898": "", "926988": "Thank you for sharing a great idea!", "787762": "thanks for your discussion", "620082": "Thank you" }, "complete": true, "source": "existing_discussion", "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3228447, "solution_url": "/competitions/champs-scalar-coupling/writeups/ka-kr-solve-chem-together-3rd-solution-bert-in-che" }