Thanks the organizers for such a great competition and great dataset. I just remembered the very recent nightmarish LANL competition, where its organizers were totally inactive and leaked dataset, then this Molecule competition is indeed one of the best ever. I love it very much.
\n\nAnd special thanks to all of my teammates, who each one actually did a great deal of work from the beginning. Thanks so much.
\n\nIn summary, main methods are:
\n\nSchNet1 (1st train) with input features: node-features (x, y, z, atom_type, mulliken_charges), edge-features (bond binary (for pairs of atoms with distance 1), angle (for pairs of atoms with distance 2), and dihedral (for pairs of atoms with distance 3). We predict 4 scc (scalar_coupling_constant) contributions and scc itself. This model gave us around -2.7 LB.
SchNet2 (2nd train): the model is the same as SchNet1, except we feed 4 out-of-fold scc contributions and scc itself as edge features where applicable, EXCEPT the edge of the learning instance itself (so in other words, we prevent leak from this, and this is not a stacking method). This model gave us -2.85 LB.
Scheduling: Cyclic LR.
Train 3 to 4 different seeds of 4-fold for each type. Some small types (xJHN) we also had an extra training of 8-fold.
Quantile regression and median averaging: since MAE cares more about predicting median of the variables, we tried to use quantile regression (with q=0.5), instead of linear regression, for 2 sub-tasks: 1) blending periodical valley model checkpoints of CyclicLR, where we removed the first 50% earlier checkpoints and only kept the last 50% checkpoints. 2) blending oof predictions of seeds of SchNet1 and SchNet2. This gave us around 0.01-0.02 boost on LB. For averaging test folds predictions, we also use median instead of mean, and found that it worked perfectly on LB.
Now, I would like to describe the process of methods from start to end, like a journey.
\n\nFirst we began with tree-based models, and without much chemistry knowledge. \n- Zidmie @zidmie in our team wrote a custom code to extract bond information from only 3D coords, which was really accurate and impressive (and later proved unnecessary since we have some chemistry toolboxes to do that). \n- We even know aromatic indicator of each atom by his method! His early work laid a good foundation to get our team to top 15, at the time we were strong enough to add in @joshxsarah and Kyle @petersk20, who had great mindsets and diverse ideas.
\n\nIt was proved that feature importance permutation is a spectacularly good technique to eliminate useless features. Josh and Kyle with their strong features, together with importance permutation once got us to 6th place. But not late after that we gradually slipped on the LB as we saw tree-based models can't compete GCN in this competition.
\n\nWe then developed a custom GCN code based on Heng's shared code in the forum, which got us -2.5 to -2.6 LB, but can never reach -2.7.
\n\nThen we merged with Akira @akirasosa , who has extremely great skills in NN!
\n\nIn the remaining of the competition onwards when having 5 people, we exploited all possible local GPUs (6) as well as 6 accounts of Google Cloud Platform (each with $300 free credits), to boost the score until the end.
\n\nMy teammates, especially Akira, will add in necessary information regarding SchNet if I missed something. You can view his separate post about SchNet architecture here: https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293#latest-610869
\n\nOur kernels are here: \n[GCN from Heng's code] https://www.kaggle.com/joshxsarah/custom-gcn-10th-place-solution\n[SchNet light Kaggle version] https://www.kaggle.com/petersk20/schnet-10th-place-solution
\n\nThanks for reading.
", "messages": [ { "id": 610727, "postDate": "2019-08-29T00:37:18.340Z", "content": "Thanks the organizers for such a great competition and great dataset. I just remembered the very recent nightmarish LANL competition, where its organizers were totally inactive and leaked dataset, then this Molecule competition is indeed one of the best ever. I love it very much.
\n\nAnd special thanks to all of my teammates, who each one actually did a great deal of work from the beginning. Thanks so much.
\n\nIn summary, main methods are:
\n\nSchNet1 (1st train) with input features: node-features (x, y, z, atom_type, mulliken_charges), edge-features (bond binary (for pairs of atoms with distance 1), angle (for pairs of atoms with distance 2), and dihedral (for pairs of atoms with distance 3). We predict 4 scc (scalar_coupling_constant) contributions and scc itself. This model gave us around -2.7 LB.
SchNet2 (2nd train): the model is the same as SchNet1, except we feed 4 out-of-fold scc contributions and scc itself as edge features where applicable, EXCEPT the edge of the learning instance itself (so in other words, we prevent leak from this, and this is not a stacking method). This model gave us -2.85 LB.
Scheduling: Cyclic LR.
Train 3 to 4 different seeds of 4-fold for each type. Some small types (xJHN) we also had an extra training of 8-fold.
Quantile regression and median averaging: since MAE cares more about predicting median of the variables, we tried to use quantile regression (with q=0.5), instead of linear regression, for 2 sub-tasks: 1) blending periodical valley model checkpoints of CyclicLR, where we removed the first 50% earlier checkpoints and only kept the last 50% checkpoints. 2) blending oof predictions of seeds of SchNet1 and SchNet2. This gave us around 0.01-0.02 boost on LB. For averaging test folds predictions, we also use median instead of mean, and found that it worked perfectly on LB.
Now, I would like to describe the process of methods from start to end, like a journey.
\n\nFirst we began with tree-based models, and without much chemistry knowledge. \n- Zidmie @zidmie in our team wrote a custom code to extract bond information from only 3D coords, which was really accurate and impressive (and later proved unnecessary since we have some chemistry toolboxes to do that). \n- We even know aromatic indicator of each atom by his method! His early work laid a good foundation to get our team to top 15, at the time we were strong enough to add in @joshxsarah and Kyle @petersk20, who had great mindsets and diverse ideas.
\n\nIt was proved that feature importance permutation is a spectacularly good technique to eliminate useless features. Josh and Kyle with their strong features, together with importance permutation once got us to 6th place. But not late after that we gradually slipped on the LB as we saw tree-based models can't compete GCN in this competition.
\n\nWe then developed a custom GCN code based on Heng's shared code in the forum, which got us -2.5 to -2.6 LB, but can never reach -2.7.
\n\nThen we merged with Akira @akirasosa , who has extremely great skills in NN!
\n\nIn the remaining of the competition onwards when having 5 people, we exploited all possible local GPUs (6) as well as 6 accounts of Google Cloud Platform (each with $300 free credits), to boost the score until the end.
\n\nMy teammates, especially Akira, will add in necessary information regarding SchNet if I missed something. You can view his separate post about SchNet architecture here: https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293#latest-610869
\n\nOur kernels are here: \n[GCN from Heng's code] https://www.kaggle.com/joshxsarah/custom-gcn-10th-place-solution\n[SchNet light Kaggle version] https://www.kaggle.com/petersk20/schnet-10th-place-solution
\n\nThanks for reading.
", "rawMarkdown": "Thanks the organizers for such a great competition and great dataset. I just remembered the very recent nightmarish LANL competition, where its organizers were totally inactive and leaked dataset, then this Molecule competition is indeed one of the best ever. I love it very much.\n\nAnd special thanks to all of my teammates, who each one actually did a great deal of work from the beginning. Thanks so much.\n\nIn summary, main methods are:\n\n- SchNet1 (1st train) with input features: node-features (x, y, z, atom_type, mulliken_charges), edge-features (bond binary (for pairs of atoms with distance 1), angle (for pairs of atoms with distance 2), and dihedral (for pairs of atoms with distance 3). We predict 4 scc (scalar_coupling_constant) contributions and scc itself. This model gave us around -2.7 LB.\n\n- SchNet2 (2nd train): the model is the same as SchNet1, except we feed 4 out-of-fold scc contributions and scc itself as edge features where applicable, EXCEPT the edge of the learning instance itself (so in other words, we prevent leak from this, and this is not a stacking method). This model gave us -2.85 LB.\n\n- Scheduling: Cyclic LR. \n\n- Train 3 to 4 different seeds of 4-fold for each type. Some small types (xJHN) we also had an extra training of 8-fold.\n\n- Quantile regression and median averaging: since MAE cares more about predicting median of the variables, we tried to use quantile regression (with q=0.5), instead of linear regression, for 2 sub-tasks: 1) blending periodical valley model checkpoints of CyclicLR, where we removed the first 50% earlier checkpoints and only kept the last 50% checkpoints. 2) blending oof predictions of seeds of SchNet1 and SchNet2. This gave us around 0.01-0.02 boost on LB. For averaging test folds predictions, we also use median instead of mean, and found that it worked perfectly on LB.\n\nNow, I would like to describe the process of methods from start to end, like a journey.\n\nFirst we began with tree-based models, and without much chemistry knowledge. \n- Zidmie @zidmie in our team wrote a custom code to extract bond information from only 3D coords, which was really accurate and impressive (and later proved unnecessary since we have some chemistry toolboxes to do that). \n- We even know aromatic indicator of each atom by his method! His early work laid a good foundation to get our team to top 15, at the time we were strong enough to add in @joshxsarah and Kyle @petersk20, who had great mindsets and diverse ideas.\n\nIt was proved that feature importance permutation is a spectacularly good technique to eliminate useless features. Josh and Kyle with their strong features, together with importance permutation once got us to 6th place. But not late after that we gradually slipped on the LB as we saw tree-based models can't compete GCN in this competition.\n\nWe then developed a custom GCN code based on Heng's shared code in the forum, which got us -2.5 to -2.6 LB, but can never reach -2.7. \n\nThen we merged with Akira @akirasosa , who has extremely great skills in NN!\n\nIn the remaining of the competition onwards when having 5 people, we exploited all possible local GPUs (6) as well as 6 accounts of Google Cloud Platform (each with $300 free credits), to boost the score until the end.\n\nMy teammates, especially Akira, will add in necessary information regarding SchNet if I missed something. You can view his separate post about SchNet architecture here: https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293#latest-610869\n\nOur kernels are here: \n[GCN from Heng's code] https://www.kaggle.com/joshxsarah/custom-gcn-10th-place-solution\n[SchNet light Kaggle version] https://www.kaggle.com/petersk20/schnet-10th-place-solution\n\nThanks for reading.", "votes": 74 }, { "id": 611682, "postDate": "2019-08-29T11:36:35.387Z", "content": "Congratulations to you and your team Kha Vo! Would you mind elaborating on your feature importance permutation work? I've gone in this direction before and am interested to see your methodology, it sounds really interesting.
", "rawMarkdown": "Congratulations to you and your team Kha Vo! Would you mind elaborating on your feature importance permutation work? I've gone in this direction before and am interested to see your methodology, it sounds really interesting.", "votes": 3, "replies": [ { "id": 612638, "postDate": "2019-08-29T23:09:51.513Z", "content": "@bigironsphere Thanks for your congratulations and great job getting silver!
\n\nWe used the approach found in this kernel - https://www.kaggle.com/speedwagon/permutation-importance
\n\nA method like RFE or sequential feature selection would take far too long so permutation importance was an efficient method for feature selection (only requires the model to be trained once). I would have liked to experiment with permutation importance on our final SchNet models but it would have been too time-consuming.
\n\nBack when 99% of people were using LGBM, it was vital to find the best features for specific types and permutation importance proved very helpful with this.
", "rawMarkdown": "@bigironsphere Thanks for your congratulations and great job getting silver! \n\nWe used the approach found in this kernel - https://www.kaggle.com/speedwagon/permutation-importance\n\nA method like RFE or sequential feature selection would take far too long so permutation importance was an efficient method for feature selection (only requires the model to be trained once). I would have liked to experiment with permutation importance on our final SchNet models but it would have been too time-consuming. \n\nBack when 99% of people were using LGBM, it was vital to find the best features for specific types and permutation importance proved very helpful with this.", "votes": 1 }, { "id": 612653, "postDate": "2019-08-29T23:27:02.110Z", "content": "Hi RNA, If you used permutation importance too, but cannot reach -2.1, I guess we might have just some better features. We crafted some binary features indicating groups of atoms, such as -CH3, -COOH, -OH... so we know that if those groups are connected to any atom of the molecule.
", "rawMarkdown": "Hi RNA, If you used permutation importance too, but cannot reach -2.1, I guess we might have just some better features. We crafted some binary features indicating groups of atoms, such as -CH3, -COOH, -OH... so we know that if those groups are connected to any atom of the molecule. ", "votes": 1 }, { "id": 617244, "postDate": "2019-09-03T23:51:43.113Z", "content": "Thank you for your replies and congratulations again on your performance!
", "rawMarkdown": "Thank you for your replies and congratulations again on your performance!", "votes": 1 } ] }, { "id": 611708, "postDate": "2019-08-29T11:52:58.863Z", "content": "Interesting share.
", "rawMarkdown": "Interesting share. ", "votes": 3, "replies": [ { "id": 611748, "postDate": "2019-08-29T12:15:29.717Z", "rawMarkdown": "" }, { "id": 612010, "postDate": "2019-08-29T15:00:05.763Z", "rawMarkdown": "\n", "votes": 3 }, { "id": 612622, "postDate": "2019-08-29T22:54:43.057Z", "content": "thanks for reading
", "rawMarkdown": "thanks for reading", "votes": 6 }, { "id": 613151, "postDate": "2019-08-30T09:05:04.297Z", "content": "No worries, it is easy to get into a heated discussion when we are both exhausted after this challenging competition. Again, congrat on the good result.
", "rawMarkdown": "No worries, it is easy to get into a heated discussion when we are both exhausted after this challenging competition. Again, congrat on the good result.", "votes": 2 } ] }, { "id": 616746, "postDate": "2019-09-03T12:23:59.847Z", "content": "good idea!
", "rawMarkdown": "good idea!", "votes": 1 }, { "id": 613134, "postDate": "2019-08-30T08:47:43.800Z", "content": "Congrats on the Gold!
", "rawMarkdown": "Congrats on the Gold!", "votes": 1 }, { "id": 612843, "postDate": "2019-08-30T03:24:20.853Z", "content": "Congratulations! Amazing solutions! :-)
", "rawMarkdown": "Congratulations! Amazing solutions! :-)", "votes": 1 }, { "id": 612270, "postDate": "2019-08-29T17:31:25.973Z", "content": "Thanks @khahuras and congrats! Would you be willing to share your computing resources for this competition?
", "rawMarkdown": "Thanks @khahuras and congrats! Would you be willing to share your computing resources for this competition? ", "votes": 1, "replies": [ { "id": 612657, "postDate": "2019-08-29T23:28:36.807Z", "content": "Akira in our team has 5 GPU 1080Ti (if I remember it right...), besides, we use 6 accounts worth of $1800 on GCP, and select P100.
", "rawMarkdown": "Akira in our team has 5 GPU 1080Ti (if I remember it right...), besides, we use 6 accounts worth of $1800 on GCP, and select P100. ", "votes": 1 }, { "id": 612683, "postDate": "2019-08-30T00:09:24.110Z", "content": "Very good to know! Thanks a lot for the valuable info @khahuras. Did you mean that each account was worth 1800$ or all six combined wwas worth that amount?
", "rawMarkdown": "Very good to know! Thanks a lot for the valuable info @khahuras. Did you mean that each account was worth 1800$ or all six combined wwas worth that amount?" }, { "id": 612688, "postDate": "2019-08-30T00:11:50.560Z", "content": "Each new credit card will have $300 first free from Google, so 6 accounts = $ 1800. However, computation on GCP is extremely expensive! We only managed to train 2 seeds (on 8 models of 8 types) with those $1800 credits.
", "rawMarkdown": "Each new credit card will have $300 first free from Google, so 6 accounts = $ 1800. However, computation on GCP is extremely expensive! We only managed to train 2 seeds (on 8 models of 8 types) with those $1800 credits." } ] }, { "id": 611302, "postDate": "2019-08-29T07:52:58.243Z", "content": "Hi @khahuras and congratulation for your achievement! :-)
\n\nAbout your tree based solution, could you detail how many features did you used and which ratio was pruned with the importance permutation technique?
\n\nWe have used it with ~150 features but the outcome was not significant and I'd like to understand why?
", "rawMarkdown": "Hi @khahuras and congratulation for your achievement! :-)\n\nAbout your tree based solution, could you detail how many features did you used and which ratio was pruned with the importance permutation technique? \n\nWe have used it with ~150 features but the outcome was not significant and I'd like to understand why? ", "votes": 1, "replies": [ { "id": 611431, "postDate": "2019-08-29T08:54:42.177Z", "content": "Thanks Sauve. Based on each type, we have 80-150 features. Best LGBM is -2.1. I will share more later.
", "rawMarkdown": "Thanks Sauve. Based on each type, we have 80-150 features. Best LGBM is -2.1. I will share more later. ", "votes": 2 }, { "id": 612543, "postDate": "2019-08-29T21:16:47.473Z", "content": "This is interesting, we used about 180 features to reach -1.9 (and 200 to 300 to go further). I'll wait for your update on this topic, thanks!
", "rawMarkdown": "This is interesting, we used about 180 features to reach -1.9 (and 200 to 300 to go further). I'll wait for your update on this topic, thanks!", "votes": 1 }, { "id": 612652, "postDate": "2019-08-29T23:26:06.227Z", "content": "Hi Sauvé. If you used permutation importance too, but cannot reach -2.1, I guess we might have just some better features. We crafted some binary features indicating groups of atoms, such as -CH3, -COOH, -OH... so we know that if those groups are connected to any atom of the molecule.
", "rawMarkdown": "Hi Sauvé. If you used permutation importance too, but cannot reach -2.1, I guess we might have just some better features. We crafted some binary features indicating groups of atoms, such as -CH3, -COOH, -OH... so we know that if those groups are connected to any atom of the molecule. ", "votes": 1 }, { "id": 616647, "postDate": "2019-09-03T10:27:45.967Z", "content": "Hi @khahuras, thanks for your answer.
\n\nJust a precision: how many hot encoded columns (approx) did you used for these groups?
\n\nWe did not used this kind of grouping, only cycles which were a significant addition. In the features crated we would have used the distance to the central atoms of these groups and probably the projection of the group direction relative to the coupling axis. However I wonder if this luxury provide a usable lever in the case of LGBM. We had some strange behaviours with some features expected to be useful and lowering significantly (and unexpectedly) the scores.
", "rawMarkdown": "Hi @khahuras, thanks for your answer.\n\nJust a precision: how many hot encoded columns (approx) did you used for these groups?\n\nWe did not used this kind of grouping, only cycles which were a significant addition. In the features crated we would have used the distance to the central atoms of these groups and probably the projection of the group direction relative to the coupling axis. However I wonder if this luxury provide a usable lever in the case of LGBM. We had some strange behaviours with some features expected to be useful and lowering significantly (and unexpectedly) the scores." } ] }, { "id": 611217, "postDate": "2019-08-29T07:04:28.660Z", "content": "Great result, congratulations!
", "rawMarkdown": "Great result, congratulations!", "votes": 1, "replies": [ { "id": 611419, "postDate": "2019-08-29T08:47:53.563Z", "content": "Thanks Agnis. I want to compete in IEEE but I’m completely worn out. I wish you get a good result, and a little bit lucky in that competition (will it shuffle? I think so...)
", "rawMarkdown": "Thanks Agnis. I want to compete in IEEE but I’m completely worn out. I wish you get a good result, and a little bit lucky in that competition (will it shuffle? I think so...)", "votes": 2 } ] }, { "id": 611040, "postDate": "2019-08-29T04:30:10.613Z", "content": "Congratulations!\nYour solution is very impressive!
", "rawMarkdown": "Congratulations!\nYour solution is very impressive!", "votes": 1 }, { "id": 610914, "postDate": "2019-08-29T02:54:24.443Z", "content": "Congratulations! Amazing work! Looking forward to your team's kernel... @khahuras
", "rawMarkdown": "Congratulations! Amazing work! Looking forward to your team's kernel... @khahuras ", "votes": 1 }, { "id": 610897, "postDate": "2019-08-29T02:44:31.990Z", "content": "You guys are legendary. Congrats.
", "rawMarkdown": "You guys are legendary. Congrats.", "votes": 1 }, { "id": 610838, "postDate": "2019-08-29T01:54:06.913Z", "content": "Congratulations! Looking forward to see your team kernel!
", "rawMarkdown": "Congratulations! Looking forward to see your team kernel!", "votes": 1 }, { "id": 610791, "postDate": "2019-08-29T01:19:27.223Z", "content": "It is great to see the simplicity of features performing so well. How long did you train the SchNet? How many epochs ?
", "rawMarkdown": "It is great to see the simplicity of features performing so well. How long did you train the SchNet? How many epochs ? ", "votes": 1, "replies": [ { "id": 611421, "postDate": "2019-08-29T08:48:47.363Z", "content": "1 day for 1/4 fold of JHC on P100. Around 500-800 epochs as cyclicLR requires a lot of epochs.
", "rawMarkdown": "1 day for 1/4 fold of JHC on P100. Around 500-800 epochs as cyclicLR requires a lot of epochs. " } ] }, { "id": 610751, "postDate": "2019-08-29T00:56:12.253Z", "content": "Amazing work! highly skilled... I couldn't do that with SchNet.\ndid you use chainer implementation?
", "rawMarkdown": "Amazing work! highly skilled... I couldn't do that with SchNet.\ndid you use **chainer** implementation?", "votes": 1, "replies": [ { "id": 610776, "postDate": "2019-08-29T01:05:30.617Z", "content": "Thanks Nanashi @jesucristo . No, we just used conventional PyTorch implementation.
", "rawMarkdown": "Thanks Nanashi @jesucristo . No, we just used conventional PyTorch implementation." } ] }, { "id": 610745, "postDate": "2019-08-29T00:50:20.397Z", "content": "Heng's starter has 70 or so ACSF features on nodes, did you guys keep it? I found adding it makes the network extremely hard to train.
", "rawMarkdown": "Heng's starter has 70 or so ACSF features on nodes, did you guys keep it? I found adding it makes the network extremely hard to train.", "votes": 1, "replies": [ { "id": 610747, "postDate": "2019-08-29T00:53:05.630Z", "content": "We had a heck of a time trying to get Heng's network to give us anything more than -1.4. We tried different ACSF settings, SOAP, adding some of the features we used in LightGBM, all sorts of different tweaks to layer sizes, learning rates, etc. I'm really looking forward to hear how people were able to achieve <-2.0 with it. Major props to Heng for releasing that code, BTW. That was awesome of him.
", "rawMarkdown": "We had a heck of a time trying to get Heng's network to give us anything more than -1.4. We tried different ACSF settings, SOAP, adding some of the features we used in LightGBM, all sorts of different tweaks to layer sizes, learning rates, etc. I'm really looking forward to hear how people were able to achieve <-2.0 with it. Major props to Heng for releasing that code, BTW. That was awesome of him.", "votes": 1 }, { "id": 610756, "postDate": "2019-08-29T00:57:45.853Z", "content": "By removing fluorine columns and hand-picking the ACSF parameters, I was able to get around -1.7 in a single fold.
", "rawMarkdown": "By removing fluorine columns and hand-picking the ACSF parameters, I was able to get around -1.7 in a single fold.", "votes": 1 }, { "id": 610764, "postDate": "2019-08-29T01:00:54.293Z", "content": "We edited Heng's code considerably much, and add a final layer which uses LGBM features. That can allow Heng's code to reach -2.5. We will share the kernel soon.
", "rawMarkdown": "We edited Heng's code considerably much, and add a final layer which uses LGBM features. That can allow Heng's code to reach -2.5. We will share the kernel soon.", "votes": 2 }, { "id": 610799, "postDate": "2019-08-29T01:26:48.617Z", "content": "@scaomath @khahuras Very interesting! Looking forward to your team's kernel, Kha.
", "rawMarkdown": "@scaomath @khahuras Very interesting! Looking forward to your team's kernel, Kha.", "votes": 1 } ] }, { "id": 618034, "postDate": "2019-09-04T18:44:49.160Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 2 }, { "id": 613221, "postDate": "2019-08-30T10:17:32.990Z", "content": "👍
", "rawMarkdown": "👍 ", "votes": 2 }, { "id": 610963, "postDate": "2019-08-29T03:31:15.813Z", "content": "Congratulations and thank you for sharing! Quantile regression and median averaging approach is quite interesting!
", "rawMarkdown": "Congratulations and thank you for sharing! Quantile regression and median averaging approach is quite interesting!", "votes": 2, "replies": [ { "id": 611426, "postDate": "2019-08-29T08:52:43.513Z", "content": "Thanks Andrew and I love all your kernels in all competitions. You are very passionate and a great asset for Kaggle community.
", "rawMarkdown": "Thanks Andrew and I love all your kernels in all competitions. You are very passionate and a great asset for Kaggle community. ", "votes": 2 }, { "id": 611812, "postDate": "2019-08-29T12:59:27.320Z", "content": "Thank you for your kind words!
", "rawMarkdown": "Thank you for your kind words!", "votes": 1 } ] }, { "id": 610928, "postDate": "2019-08-29T03:03:34.610Z", "rawMarkdown": "", "isDeleted": true }, { "id": 612173, "postDate": "2019-08-29T16:27:31.460Z", "content": "Thanks for sharing your kernel.
", "rawMarkdown": "Thanks for sharing your kernel.", "votes": 1 }, { "id": 611315, "postDate": "2019-08-29T07:59:27.563Z", "content": "Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 } ], "comments": [ { "id": 611682, "author_name": "RNA", "author_url": "", "post_date": "2019-08-29T11:36:35.387000", "content": "Congratulations to you and your team Kha Vo! Would you mind elaborating on your feature importance permutation work? I've gone in this direction before and am interested to see your methodology, it sounds really interesting.
", "votes": 3, "replies": [ { "id": 612638, "author_name": "Joshua Colmer", "author_url": "", "post_date": "2019-08-29T23:09:51.513000", "content": "@bigironsphere Thanks for your congratulations and great job getting silver!
\n\nWe used the approach found in this kernel - https://www.kaggle.com/speedwagon/permutation-importance
\n\nA method like RFE or sequential feature selection would take far too long so permutation importance was an efficient method for feature selection (only requires the model to be trained once). I would have liked to experiment with permutation importance on our final SchNet models but it would have been too time-consuming.
\n\nBack when 99% of people were using LGBM, it was vital to find the best features for specific types and permutation importance proved very helpful with this.
", "votes": 1, "replies": [] }, { "id": 612653, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T23:27:02.110000", "content": "Hi RNA, If you used permutation importance too, but cannot reach -2.1, I guess we might have just some better features. We crafted some binary features indicating groups of atoms, such as -CH3, -COOH, -OH... so we know that if those groups are connected to any atom of the molecule.
", "votes": 1, "replies": [] }, { "id": 617244, "author_name": "RNA", "author_url": "", "post_date": "2019-09-03T23:51:43.113000", "content": "Thank you for your replies and congratulations again on your performance!
", "votes": 1, "replies": [] } ] }, { "id": 611708, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-29T11:52:58.863000", "content": "Interesting share.
", "votes": 3, "replies": [ { "id": 611748, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T12:15:29.717000", "content": "", "votes": 0, "replies": [] }, { "id": 612010, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-29T15:00:05.763000", "content": "", "votes": 3, "replies": [] }, { "id": 612622, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T22:54:43.057000", "content": "thanks for reading
", "votes": 6, "replies": [] }, { "id": 613151, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-30T09:05:04.297000", "content": "No worries, it is easy to get into a heated discussion when we are both exhausted after this challenging competition. Again, congrat on the good result.
", "votes": 2, "replies": [] } ] }, { "id": 616746, "author_name": "Yongli Zhang", "author_url": "", "post_date": "2019-09-03T12:23:59.847000", "content": "good idea!
", "votes": 1, "replies": [] }, { "id": 613134, "author_name": "fnands", "author_url": "", "post_date": "2019-08-30T08:47:43.800000", "content": "Congrats on the Gold!
", "votes": 1, "replies": [] }, { "id": 612843, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-30T03:24:20.853000", "content": "Congratulations! Amazing solutions! :-)
", "votes": 1, "replies": [] }, { "id": 612270, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-29T17:31:25.973000", "content": "Thanks @khahuras and congrats! Would you be willing to share your computing resources for this competition?
", "votes": 1, "replies": [ { "id": 612657, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T23:28:36.807000", "content": "Akira in our team has 5 GPU 1080Ti (if I remember it right...), besides, we use 6 accounts worth of $1800 on GCP, and select P100.
", "votes": 1, "replies": [] }, { "id": 612683, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T00:09:24.110000", "content": "Very good to know! Thanks a lot for the valuable info @khahuras. Did you mean that each account was worth 1800$ or all six combined wwas worth that amount?
", "votes": 0, "replies": [] }, { "id": 612688, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-30T00:11:50.560000", "content": "Each new credit card will have $300 first free from Google, so 6 accounts = $ 1800. However, computation on GCP is extremely expensive! We only managed to train 2 seeds (on 8 models of 8 types) with those $1800 credits.
", "votes": 0, "replies": [] } ] }, { "id": 611302, "author_name": "Alexandre Sauvé", "author_url": "", "post_date": "2019-08-29T07:52:58.243000", "content": "Hi @khahuras and congratulation for your achievement! :-)
\n\nAbout your tree based solution, could you detail how many features did you used and which ratio was pruned with the importance permutation technique?
\n\nWe have used it with ~150 features but the outcome was not significant and I'd like to understand why?
", "votes": 1, "replies": [ { "id": 611431, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T08:54:42.177000", "content": "Thanks Sauve. Based on each type, we have 80-150 features. Best LGBM is -2.1. I will share more later.
", "votes": 2, "replies": [] }, { "id": 612543, "author_name": "Alexandre Sauvé", "author_url": "", "post_date": "2019-08-29T21:16:47.473000", "content": "This is interesting, we used about 180 features to reach -1.9 (and 200 to 300 to go further). I'll wait for your update on this topic, thanks!
", "votes": 1, "replies": [] }, { "id": 612652, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T23:26:06.227000", "content": "Hi Sauvé. If you used permutation importance too, but cannot reach -2.1, I guess we might have just some better features. We crafted some binary features indicating groups of atoms, such as -CH3, -COOH, -OH... so we know that if those groups are connected to any atom of the molecule.
", "votes": 1, "replies": [] }, { "id": 616647, "author_name": "Alexandre Sauvé", "author_url": "", "post_date": "2019-09-03T10:27:45.967000", "content": "Hi @khahuras, thanks for your answer.
\n\nJust a precision: how many hot encoded columns (approx) did you used for these groups?
\n\nWe did not used this kind of grouping, only cycles which were a significant addition. In the features crated we would have used the distance to the central atoms of these groups and probably the projection of the group direction relative to the coupling axis. However I wonder if this luxury provide a usable lever in the case of LGBM. We had some strange behaviours with some features expected to be useful and lowering significantly (and unexpectedly) the scores.
", "votes": 0, "replies": [] } ] }, { "id": 611217, "author_name": "alijs", "author_url": "", "post_date": "2019-08-29T07:04:28.660000", "content": "Great result, congratulations!
", "votes": 1, "replies": [ { "id": 611419, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T08:47:53.563000", "content": "Thanks Agnis. I want to compete in IEEE but I’m completely worn out. I wish you get a good result, and a little bit lucky in that competition (will it shuffle? I think so...)
", "votes": 2, "replies": [] } ] }, { "id": 611040, "author_name": "Yusuke Hamada", "author_url": "", "post_date": "2019-08-29T04:30:10.613000", "content": "Congratulations!\nYour solution is very impressive!
", "votes": 1, "replies": [] }, { "id": 610914, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-08-29T02:54:24.443000", "content": "Congratulations! Amazing work! Looking forward to your team's kernel... @khahuras
", "votes": 1, "replies": [] }, { "id": 610897, "author_name": "Kaggoo", "author_url": "", "post_date": "2019-08-29T02:44:31.990000", "content": "You guys are legendary. Congrats.
", "votes": 1, "replies": [] }, { "id": 610838, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2019-08-29T01:54:06.913000", "content": "Congratulations! Looking forward to see your team kernel!
", "votes": 1, "replies": [] }, { "id": 610791, "author_name": "Ashesh.", "author_url": "", "post_date": "2019-08-29T01:19:27.223000", "content": "It is great to see the simplicity of features performing so well. How long did you train the SchNet? How many epochs ?
", "votes": 1, "replies": [ { "id": 611421, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T08:48:47.363000", "content": "1 day for 1/4 fold of JHC on P100. Around 500-800 epochs as cyclicLR requires a lot of epochs.
", "votes": 0, "replies": [] } ] }, { "id": 610751, "author_name": "Nanashi", "author_url": "", "post_date": "2019-08-29T00:56:12.253000", "content": "Amazing work! highly skilled... I couldn't do that with SchNet.\ndid you use chainer implementation?
", "votes": 1, "replies": [ { "id": 610776, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T01:05:30.617000", "content": "Thanks Nanashi @jesucristo . No, we just used conventional PyTorch implementation.
", "votes": 0, "replies": [] } ] }, { "id": 610745, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2019-08-29T00:50:20.397000", "content": "Heng's starter has 70 or so ACSF features on nodes, did you guys keep it? I found adding it makes the network extremely hard to train.
", "votes": 1, "replies": [ { "id": 610747, "author_name": "fafafa fafafafa", "author_url": "", "post_date": "2019-08-29T00:53:05.630000", "content": "We had a heck of a time trying to get Heng's network to give us anything more than -1.4. We tried different ACSF settings, SOAP, adding some of the features we used in LightGBM, all sorts of different tweaks to layer sizes, learning rates, etc. I'm really looking forward to hear how people were able to achieve <-2.0 with it. Major props to Heng for releasing that code, BTW. That was awesome of him.
", "votes": 1, "replies": [] }, { "id": 610756, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2019-08-29T00:57:45.853000", "content": "By removing fluorine columns and hand-picking the ACSF parameters, I was able to get around -1.7 in a single fold.
", "votes": 1, "replies": [] }, { "id": 610764, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T01:00:54.293000", "content": "We edited Heng's code considerably much, and add a final layer which uses LGBM features. That can allow Heng's code to reach -2.5. We will share the kernel soon.
", "votes": 2, "replies": [] }, { "id": 610799, "author_name": "fafafa fafafafa", "author_url": "", "post_date": "2019-08-29T01:26:48.617000", "content": "@scaomath @khahuras Very interesting! Looking forward to your team's kernel, Kha.
", "votes": 1, "replies": [] } ] }, { "id": 618034, "author_name": "Rashidul H", "author_url": "", "post_date": "2019-09-04T18:44:49.160000", "content": "Congrats!
", "votes": 2, "replies": [] }, { "id": 613221, "author_name": "Julya Isakova", "author_url": "", "post_date": "2019-08-30T10:17:32.990000", "content": "👍
", "votes": 2, "replies": [] }, { "id": 610963, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-08-29T03:31:15.813000", "content": "Congratulations and thank you for sharing! Quantile regression and median averaging approach is quite interesting!
", "votes": 2, "replies": [ { "id": 611426, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-08-29T08:52:43.513000", "content": "Thanks Andrew and I love all your kernels in all competitions. You are very passionate and a great asset for Kaggle community.
", "votes": 2, "replies": [] }, { "id": 611812, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-08-29T12:59:27.320000", "content": "Thank you for your kind words!
", "votes": 1, "replies": [] } ] }, { "id": 610928, "author_name": "", "author_url": "", "post_date": "2019-08-29T03:03:34.610000", "content": "", "votes": 0, "replies": [] }, { "id": 612173, "author_name": "XiaokangWang", "author_url": "", "post_date": "2019-08-29T16:27:31.460000", "content": "Thanks for sharing your kernel.
", "votes": 1, "replies": [] }, { "id": 611315, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-08-29T07:59:27.563000", "content": "Congratulations and thanks for sharing!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "610727": "Thanks the organizers for such a great competition and great dataset. I just remembered the very recent nightmarish LANL competition, where its organizers were totally inactive and leaked dataset, then this Molecule competition is indeed one of the best ever. I love it very much.\n\nAnd special thanks to all of my teammates, who each one actually did a great deal of work from the beginning. Thanks so much.\n\nIn summary, main methods are:\n\n- SchNet1 (1st train) with input features: node-features (x, y, z, atom_type, mulliken_charges), edge-features (bond binary (for pairs of atoms with distance 1), angle (for pairs of atoms with distance 2), and dihedral (for pairs of atoms with distance 3). We predict 4 scc (scalar_coupling_constant) contributions and scc itself. This model gave us around -2.7 LB.\n\n- SchNet2 (2nd train): the model is the same as SchNet1, except we feed 4 out-of-fold scc contributions and scc itself as edge features where applicable, EXCEPT the edge of the learning instance itself (so in other words, we prevent leak from this, and this is not a stacking method). This model gave us -2.85 LB.\n\n- Scheduling: Cyclic LR. \n\n- Train 3 to 4 different seeds of 4-fold for each type. Some small types (xJHN) we also had an extra training of 8-fold.\n\n- Quantile regression and median averaging: since MAE cares more about predicting median of the variables, we tried to use quantile regression (with q=0.5), instead of linear regression, for 2 sub-tasks: 1) blending periodical valley model checkpoints of CyclicLR, where we removed the first 50% earlier checkpoints and only kept the last 50% checkpoints. 2) blending oof predictions of seeds of SchNet1 and SchNet2. This gave us around 0.01-0.02 boost on LB. For averaging test folds predictions, we also use median instead of mean, and found that it worked perfectly on LB.\n\nNow, I would like to describe the process of methods from start to end, like a journey.\n\nFirst we began with tree-based models, and without much chemistry knowledge. \n- Zidmie @zidmie in our team wrote a custom code to extract bond information from only 3D coords, which was really accurate and impressive (and later proved unnecessary since we have some chemistry toolboxes to do that). \n- We even know aromatic indicator of each atom by his method! His early work laid a good foundation to get our team to top 15, at the time we were strong enough to add in @joshxsarah and Kyle @petersk20, who had great mindsets and diverse ideas.\n\nIt was proved that feature importance permutation is a spectacularly good technique to eliminate useless features. Josh and Kyle with their strong features, together with importance permutation once got us to 6th place. But not late after that we gradually slipped on the LB as we saw tree-based models can't compete GCN in this competition.\n\nWe then developed a custom GCN code based on Heng's shared code in the forum, which got us -2.5 to -2.6 LB, but can never reach -2.7. \n\nThen we merged with Akira @akirasosa , who has extremely great skills in NN!\n\nIn the remaining of the competition onwards when having 5 people, we exploited all possible local GPUs (6) as well as 6 accounts of Google Cloud Platform (each with $300 free credits), to boost the score until the end.\n\nMy teammates, especially Akira, will add in necessary information regarding SchNet if I missed something. You can view his separate post about SchNet architecture here: https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293#latest-610869\n\nOur kernels are here: \n[GCN from Heng's code] https://www.kaggle.com/joshxsarah/custom-gcn-10th-place-solution\n[SchNet light Kaggle version] https://www.kaggle.com/petersk20/schnet-10th-place-solution\n\nThanks for reading.", "611682": "Congratulations to you and your team Kha Vo! Would you mind elaborating on your feature importance permutation work? I've gone in this direction before and am interested to see your methodology, it sounds really interesting.", "611708": "Interesting share. ", "616746": "good idea!", "613134": "Congrats on the Gold!", "612843": "Congratulations! Amazing solutions! :-)", "612270": "Thanks @khahuras and congrats! Would you be willing to share your computing resources for this competition? ", "611302": "Hi @khahuras and congratulation for your achievement! :-)\n\nAbout your tree based solution, could you detail how many features did you used and which ratio was pruned with the importance permutation technique? \n\nWe have used it with ~150 features but the outcome was not significant and I'd like to understand why? ", "611217": "Great result, congratulations!", "611040": "Congratulations!\nYour solution is very impressive!", "610914": "Congratulations! Amazing work! Looking forward to your team's kernel... @khahuras ", "610897": "You guys are legendary. Congrats.", "610838": "Congratulations! Looking forward to see your team kernel!", "610791": "It is great to see the simplicity of features performing so well. How long did you train the SchNet? How many epochs ? ", "610751": "Amazing work! highly skilled... I couldn't do that with SchNet.\ndid you use **chainer** implementation?", "610745": "Heng's starter has 70 or so ACSF features on nodes, did you guys keep it? I found adding it makes the network extremely hard to train.", "618034": "Congrats!", "613221": "👍 ", "610963": "Congratulations and thank you for sharing! Quantile regression and median averaging approach is quite interesting!", "610928": "", "612173": "Thanks for sharing your kernel.", "611315": "Congratulations and thanks for sharing!" } }