![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F471945%2F99bfd9d1c7c1d10ab2813ef561acf7ce%2Fds_pinkman.jpg?generation=1567088919598312&alt=media\")
First of all, I'd like to thank my team mates, without whom I would not have landed where we landed, and even would not have entered the competition. Let me also way that I'm very happy to see Ahmet become competition GM. And Outrunner is only one gold away from it. I'm sure he'll become GM in his next competition. I was very lucky to team with them. It proves 1 + 1 + 1 > 3.
\n\nI also want to thank @inversion, Kaggle staff in general, and CHAMPS for setting up this very challenging, unflawed contest. It is refreshing to not have any leak. Using test data wasn't helpful really either. And the way @inversion managed the two crisis (QM9 use, and unplanned kernel share) was very effective.
\n\nI was very reluctant to enter this competition, for two reasons. First, I was convinced that people with domain knowledge would have an unfair advantage. I was right as the top of LB shows a high proportion of teams with chemists on board. Second, it was clear from early discussions that graph NN would be key, and I had absolutely no experience with it. This second reason is a bit weak given most of us here are in same situation!
\n\nThen Ahmet asked me if I would join, because we almost teamed in a previous competition and said we would team ASAP. It motivated me to complete an initial effort using lightgbm. I developed a core set of features, then added features specific to each type. The features are based on the graph defined by bonds computed using open babel (thanks @borisd for the starter code). Angles for 2 bonds path, and dihedral angle for 3 bonds paths were key. I then added features based on the number of 1, 2, 3 bond paths starting on every atom on the path, using aggregates like number by bond type, angle averages, distance averages, etc. Depending on the type, number of features was between 500 and 2000. I trained one lgb per type. With this I got to 2.0 on LB with one fold the day I entered. Once trained on all folds, this model got 2.1 on LB.
\n\nThere are few things I did to improve the model.
\n\nFirst, I use a very large number of leaves, 2**10 -1. I am not sure why this is good, but it was. Other parameter tuning wasn't very important.
\n\nSecond, I duplicated train and test rows with types 2JHH and 3JHH by swapping the two atoms, before feature engineering, because scalar coupling is symmetric. Final prediction for one row in the original train or test was the average of the two rows predictions after expansion. This was mentioned early in the forum I think.
\n\nThird, for 3Jxx bond types, there is often more than one 3 bonds path between the two atoms. Here again I duplicated, using one row for each 3 bonds path between the two atoms, then averaging their predictions back for final prediction. I did not see this discussed anywhere.
\n\nRow duplication improves lgb score by about 0.002, not much, but still useful.
\n\nI then teamed with Ahmet, and given he had a NN with much better performance than my lgb model I gave up on my base model. I'll let Ahmet describe his NN (edit: description is available here). Let me just say that with some improvements, like using logcosh as loss, and expanding its size, we got to -2.64 on LB using single 4 fold cv run. I was very happy given this moved us way above where lgb could move us to.
\n\nLgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8.
\n\nTo make sure we get a gold medal we knew we would need another strong model. I started developing a graph NN from scratch in keras, and was getting CV o -1.9 and LB -2.08 nwith just 4 features (atom and bond type, mulliken charge, and dihedral angle for 3 bond types). I stopped working on it when we teamed with Outrunner. Indeed, Outrunner skills in NN clearly outperfom mine, by far. He had a set of 13 NN models that yield almost -2.9 LB once bagged. He further improved them to yield -2.92 or so soon after merging. I will let him describe these models if he wants to.
\n\nThe first thing we did after teaming was to run my lgb postprocessing on his validation prediction. He was using a single train/val split with only 3% or so of validation data. Yet, even with this limited data lgb could extract about 0.025 more. Blending the result with what we had added another 0.03 or so, which moved us better than -2.95.
\n\nThe last progress we made was based on the NN approach used. A lot of people have used Graph NN where one molecule is a sample, and all coupling for that molecule can be predicted together. Outrunner and Ahmet use another way, with one sample per train and test row. They have features that capture the relative position of every atom compared to the two for which the coupling is computed. For instance distances to these two atoms, and an angle, and more. But distance to the two atoms and an angle doesn't uniquely identify atom positions. There is a circular symmetry. A way to remove this symmetry is to consider other atoms. Ahmet was doing it but not Outrunner. I suggested to use middle atoms. For 2JHx, a0 and a1 are both bond to a middle atom am. For 3JHx, a0 is bond to am0 which is bond to am1 which is bond to a1. Outrunner added distance to am, or am0/am1 depending on the type. This boosted performance to where we finally ended. Unfortunately, given we teamed late, he could only retrain 3 of his models with the new feature. Bagging these leads to -2.99 LB. Running lgb on top of it and blending with Ahmet-CPMP blend yields -3.032. I guess we could have landed higher with more models retrained, but we lacked time and/or GPUs.
\n\nBottom line is I am happy this very challenging competition ended. I probably should have entered earlier, my first sub is from July 22, only 5 weeks ago. But teaming with two talented people more than made for the lack of time.
\n\nEdt, you'll find Ahmet NN description here. Outrunner told me his NN is similar to Ahmet's hence he won't provide a separate description.
", "messages": [ { "id": "611829", "postDate": "08/29/2019 13:04:14", "content": "First of all, I'd like to thank my team mates, without whom I would not have landed where we landed, and even would not have entered the competition. Let me also way that I'm very happy to see Ahmet become competition GM. And Outrunner is only one gold away from it. I'm sure he'll become GM in his next competition. I was very lucky to team with them. It proves 1 + 1 + 1 > 3.
\n\nI also want to thank @inversion, Kaggle staff in general, and CHAMPS for setting up this very challenging, unflawed contest. It is refreshing to not have any leak. Using test data wasn't helpful really either. And the way @inversion managed the two crisis (QM9 use, and unplanned kernel share) was very effective.
\n\nI was very reluctant to enter this competition, for two reasons. First, I was convinced that people with domain knowledge would have an unfair advantage. I was right as the top of LB shows a high proportion of teams with chemists on board. Second, it was clear from early discussions that graph NN would be key, and I had absolutely no experience with it. This second reason is a bit weak given most of us here are in same situation!
\n\nThen Ahmet asked me if I would join, because we almost teamed in a previous competition and said we would team ASAP. It motivated me to complete an initial effort using lightgbm. I developed a core set of features, then added features specific to each type. The features are based on the graph defined by bonds computed using open babel (thanks @borisd for the starter code). Angles for 2 bonds path, and dihedral angle for 3 bonds paths were key. I then added features based on the number of 1, 2, 3 bond paths starting on every atom on the path, using aggregates like number by bond type, angle averages, distance averages, etc. Depending on the type, number of features was between 500 and 2000. I trained one lgb per type. With this I got to 2.0 on LB with one fold the day I entered. Once trained on all folds, this model got 2.1 on LB.
\n\nThere are few things I did to improve the model.
\n\nFirst, I use a very large number of leaves, 2**10 -1. I am not sure why this is good, but it was. Other parameter tuning wasn't very important.
\n\nSecond, I duplicated train and test rows with types 2JHH and 3JHH by swapping the two atoms, before feature engineering, because scalar coupling is symmetric. Final prediction for one row in the original train or test was the average of the two rows predictions after expansion. This was mentioned early in the forum I think.
\n\nThird, for 3Jxx bond types, there is often more than one 3 bonds path between the two atoms. Here again I duplicated, using one row for each 3 bonds path between the two atoms, then averaging their predictions back for final prediction. I did not see this discussed anywhere.
\n\nRow duplication improves lgb score by about 0.002, not much, but still useful.
\n\nI then teamed with Ahmet, and given he had a NN with much better performance than my lgb model I gave up on my base model. I'll let Ahmet describe his NN (edit: description is available here). Let me just say that with some improvements, like using logcosh as loss, and expanding its size, we got to -2.64 on LB using single 4 fold cv run. I was very happy given this moved us way above where lgb could move us to.
\n\nLgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8.
\n\nTo make sure we get a gold medal we knew we would need another strong model. I started developing a graph NN from scratch in keras, and was getting CV o -1.9 and LB -2.08 nwith just 4 features (atom and bond type, mulliken charge, and dihedral angle for 3 bond types). I stopped working on it when we teamed with Outrunner. Indeed, Outrunner skills in NN clearly outperfom mine, by far. He had a set of 13 NN models that yield almost -2.9 LB once bagged. He further improved them to yield -2.92 or so soon after merging. I will let him describe these models if he wants to.
\n\nThe first thing we did after teaming was to run my lgb postprocessing on his validation prediction. He was using a single train/val split with only 3% or so of validation data. Yet, even with this limited data lgb could extract about 0.025 more. Blending the result with what we had added another 0.03 or so, which moved us better than -2.95.
\n\nThe last progress we made was based on the NN approach used. A lot of people have used Graph NN where one molecule is a sample, and all coupling for that molecule can be predicted together. Outrunner and Ahmet use another way, with one sample per train and test row. They have features that capture the relative position of every atom compared to the two for which the coupling is computed. For instance distances to these two atoms, and an angle, and more. But distance to the two atoms and an angle doesn't uniquely identify atom positions. There is a circular symmetry. A way to remove this symmetry is to consider other atoms. Ahmet was doing it but not Outrunner. I suggested to use middle atoms. For 2JHx, a0 and a1 are both bond to a middle atom am. For 3JHx, a0 is bond to am0 which is bond to am1 which is bond to a1. Outrunner added distance to am, or am0/am1 depending on the type. This boosted performance to where we finally ended. Unfortunately, given we teamed late, he could only retrain 3 of his models with the new feature. Bagging these leads to -2.99 LB. Running lgb on top of it and blending with Ahmet-CPMP blend yields -3.032. I guess we could have landed higher with more models retrained, but we lacked time and/or GPUs.
\n\nBottom line is I am happy this very challenging competition ended. I probably should have entered earlier, my first sub is from July 22, only 5 weeks ago. But teaming with two talented people more than made for the lack of time.
\n\nEdt, you'll find Ahmet NN description here. Outrunner told me his NN is similar to Ahmet's hence he won't provide a separate description.
", "rawMarkdown": "First of all, I'd like to thank my team mates, without whom I would not have landed where we landed, and even would not have entered the competition. Let me also way that I'm very happy to see Ahmet become competition GM. And Outrunner is only one gold away from it. I'm sure he'll become GM in his next competition. I was very lucky to team with them. It proves 1 + 1 + 1 > 3.\n\nI also want to thank @inversion, Kaggle staff in general, and CHAMPS for setting up this very challenging, unflawed contest. It is refreshing to not have any leak. Using test data wasn't helpful really either. And the way @inversion managed the two crisis (QM9 use, and unplanned kernel share) was very effective.\n\nI was very reluctant to enter this competition, for two reasons. First, I was convinced that people with domain knowledge would have an unfair advantage. I was right as the top of LB shows a high proportion of teams with chemists on board. Second, it was clear from early discussions that graph NN would be key, and I had absolutely no experience with it. This second reason is a bit weak given most of us here are in same situation!\n\nThen Ahmet asked me if I would join, because we almost teamed in a previous competition and said we would team ASAP. It motivated me to complete an initial effort using lightgbm. I developed a core set of features, then added features specific to each type. The features are based on the graph defined by bonds computed using open babel (thanks @borisd for the starter code). Angles for 2 bonds path, and dihedral angle for 3 bonds paths were key. I then added features based on the number of 1, 2, 3 bond paths starting on every atom on the path, using aggregates like number by bond type, angle averages, distance averages, etc. Depending on the type, number of features was between 500 and 2000. I trained one lgb per type. With this I got to 2.0 on LB with one fold the day I entered. Once trained on all folds, this model got 2.1 on LB. \n\nThere are few things I did to improve the model. \n\nFirst, I use a very large number of leaves, 2**10 -1. I am not sure why this is good, but it was. Other parameter tuning wasn't very important.\n\nSecond, I duplicated train and test rows with types 2JHH and 3JHH by swapping the two atoms, before feature engineering, because scalar coupling is symmetric. Final prediction for one row in the original train or test was the average of the two rows predictions after expansion. This was mentioned early in the forum I think.\n\nThird, for 3Jxx bond types, there is often more than one 3 bonds path between the two atoms. Here again I duplicated, using one row for each 3 bonds path between the two atoms, then averaging their predictions back for final prediction. I did not see this discussed anywhere.\n\nRow duplication improves lgb score by about 0.002, not much, but still useful.\n\nI then teamed with Ahmet, and given he had a NN with much better performance than my lgb model I gave up on my base model. I'll let Ahmet describe his NN (edit: description is available [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421#611963)). Let me just say that with some improvements, like using logcosh as loss, and expanding its size, we got to -2.64 on LB using single 4 fold cv run. I was very happy given this moved us way above where lgb could move us to. \n\nLgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8. \n\nTo make sure we get a gold medal we knew we would need another strong model. I started developing a graph NN from scratch in keras, and was getting CV o -1.9 and LB -2.08 nwith just 4 features (atom and bond type, mulliken charge, and dihedral angle for 3 bond types). I stopped working on it when we teamed with Outrunner. Indeed, Outrunner skills in NN clearly outperfom mine, by far. He had a set of 13 NN models that yield almost -2.9 LB once bagged. He further improved them to yield -2.92 or so soon after merging. I will let him describe these models if he wants to. \n\nThe first thing we did after teaming was to run my lgb postprocessing on his validation prediction. He was using a single train/val split with only 3% or so of validation data. Yet, even with this limited data lgb could extract about 0.025 more. Blending the result with what we had added another 0.03 or so, which moved us better than -2.95. \n\nThe last progress we made was based on the NN approach used. A lot of people have used Graph NN where one molecule is a sample, and all coupling for that molecule can be predicted together. Outrunner and Ahmet use another way, with one sample per train and test row. They have features that capture the relative position of every atom compared to the two for which the coupling is computed. For instance distances to these two atoms, and an angle, and more. But distance to the two atoms and an angle doesn't uniquely identify atom positions. There is a circular symmetry. A way to remove this symmetry is to consider other atoms. Ahmet was doing it but not Outrunner. I suggested to use middle atoms. For 2JHx, a0 and a1 are both bond to a middle atom am. For 3JHx, a0 is bond to am0 which is bond to am1 which is bond to a1. Outrunner added distance to am, or am0/am1 depending on the type. This boosted performance to where we finally ended. Unfortunately, given we teamed late, he could only retrain 3 of his models with the new feature. Bagging these leads to -2.99 LB. Running lgb on top of it and blending with Ahmet-CPMP blend yields -3.032. I guess we could have landed higher with more models retrained, but we lacked time and/or GPUs.\n\nBottom line is I am happy this very challenging competition ended. I probably should have entered earlier, my first sub is from July 22, only 5 weeks ago. But teaming with two talented people more than made for the lack of time.\n\nEdt, you'll find Ahmet NN description [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421#611963). Outrunner told me his NN is similar to Ahmet's hence he won't provide a separate description.", "votes": null }, { "id": "611852", "postDate": "08/29/2019 13:15:41", "content": "Very impressive work - Congrats!
", "rawMarkdown": "Very impressive work - Congrats!", "votes": null }, { "id": "611897", "postDate": "08/29/2019 13:44:26", "content": "Thanks for sharing @cpmpml, this is a very interesting solution. \nI'll keep in mind residual training for future comps. \nSo Creative of you !
", "rawMarkdown": "Thanks for sharing @cpmpml, this is a very interesting solution. \nI'll keep in mind residual training for future comps. \nSo Creative of you !", "votes": null }, { "id": "611915", "postDate": "08/29/2019 13:57:04", "content": "Excellent write up! Your approach where you used LGB to post-process the NN output is a very useful technique.
", "rawMarkdown": "Excellent write up! Your approach where you used LGB to post-process the NN output is a very useful technique.", "votes": null }, { "id": "611957", "postDate": "08/29/2019 14:24:26", "content": "Thanks for this write-up! You're an excellent writer, your #1 spot in the discussions leader-board is well-deserved.
", "rawMarkdown": "Thanks for this write-up! You're an excellent writer, your #1 spot in the discussions leader-board is well-deserved.", "votes": null }, { "id": "611963", "postDate": "08/29/2019 14:30:23", "content": "It was a great experience to team up with top Kagglers like @cpmpml and @outrunner . As CPMP explained, I had a custom NN architecture. It wouldn't be competitive enough for gold without CPMP's boosting with LGB. And we wouldn't make it to 7th place without outrunner's NN expertise.
\n\nI have started working on this competition on Kaggle Kernels and could reach -1.4 via this model: https://www.kaggle.com/divrikwicky/lightweight-version-of-2-65-custom-nn Later, I have added more interaction features, made network larger and run it longer. I have only one model for all the types. Therefore, I use sample weights to mimic the competition metric. This seems to work better than training single models for each type (there are exception types). My NN uses Conv1D extract features for each atom, and there is another Conv1D layer after which uses the feedback from the molecule.
\n\nI am also happy to be a GM. Hopefully, other people who deserve become GM too, not the ones who try to cut corners:) I have learned a lot from the Kaggle community and my (ex-)teammates. And I hope I will continue learning. Every time I learn something very novel on Kaggle, I am like:\n
Thanks Ahmet. It was great teaming with you and Outrunner.
", "rawMarkdown": "Thanks Ahmet. It was great teaming with you and Outrunner.", "votes": null }, { "id": "612194", "postDate": "08/29/2019 16:43:21", "content": "Very impressive work - Congrats!\nExcellent write up!\nThanks for sharing @cpmpml
", "rawMarkdown": "Very impressive work - Congrats!\nExcellent write up!\nThanks for sharing @cpmpml", "votes": null }, { "id": "612265", "postDate": "08/29/2019 17:27:41", "content": "Thanks @divrikwicky @cpmpml and congrats! Would you be willing to share your computing resources for this competition + time to train your models?
", "rawMarkdown": "Thanks @divrikwicky @cpmpml and congrats! Would you be willing to share your computing resources for this competition + time to train your models?", "votes": null }, { "id": "612324", "postDate": "08/29/2019 18:10:56", "content": "I have 2 1080 Ti that were mostly used to train Ahmet's model, and partly to develop mine. Ahmet only has 2 K40. And Outrunner has 2 1080 Ti as well.
", "rawMarkdown": "I have 2 1080 Ti that were mostly used to train Ahmet's model, and partly to develop mine. Ahmet only has 2 K40. And Outrunner has 2 1080 Ti as well.", "votes": null }, { "id": "612361", "postDate": "08/29/2019 18:38:20", "content": "Thanks for sharing!
\n\n@cpmpml \nCan I ask a question? \nYou used lgb to predict the residuals of the NN.\nWhat is the features for lgb?\nIs that you used the predictions of NN to be the features for lgb?
", "rawMarkdown": "Thanks for sharing!\n\n@cpmpml \nCan I ask a question? \nYou used lgb to predict the residuals of the NN.\nWhat is the features for lgb?\nIs that you used the predictions of NN to be the features for lgb?", "votes": null }, { "id": "612364", "postDate": "08/29/2019 18:40:44", "content": "I used the same features ad lgb standalone plus the NN prediction. I think this works because the features used for lgb are very different from what the NN learned. Think of it as stacking (using the NN prediction as feature), but also using original features.
", "rawMarkdown": "I used the same features ad lgb standalone plus the NN prediction. I think this works because the features used for lgb are very different from what the NN learned. Think of it as stacking (using the NN prediction as feature), but also using original features.", "votes": null }, { "id": "612375", "postDate": "08/29/2019 18:46:43", "content": "\n\n\nWith this I got to 2.0 on LB with one fold the day I entered.
\n
what the hell, so thats what a grandmaster can do... you motivate me to keep going:)
", "rawMarkdown": "> With this I got to 2.0 on LB with one fold the day I entered.\n\nwhat the hell, so thats what a grandmaster can do... you motivate me to keep going:)", "votes": null }, { "id": "612596", "postDate": "08/29/2019 22:15:22", "content": "I was watching your team through whole competition and was looking forward to see your approach. Thanks for sharing.
", "rawMarkdown": "I was watching your team through whole competition and was looking forward to see your approach. Thanks for sharing.", "votes": null }, { "id": "612701", "postDate": "08/30/2019 00:28:10", "content": "Congrats on your team performance. We have some common ideas regarding symmetry and \"middle atom\". What I learned from you in this competition is this:\n\"Lgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8.\"
", "rawMarkdown": "Congrats on your team performance. We have some common ideas regarding symmetry and \"middle atom\". What I learned from you in this competition is this:\n\"Lgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8.\"", "votes": null }, { "id": "612791", "postDate": "08/30/2019 02:21:29", "content": "Really creative solution,I want to try next time.
", "rawMarkdown": "Really creative solution,I want to try next time.", "votes": null }, { "id": "612847", "postDate": "08/30/2019 03:33:34", "content": "Congratulations! Amazing solutions! :-)
", "rawMarkdown": "Congratulations! Amazing solutions! :-)", "votes": null }, { "id": "612989", "postDate": "08/30/2019 06:29:52", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "613113", "postDate": "08/30/2019 08:19:41", "content": "Congrats on the GM-ship! And congrats to the gold for all three of you!
", "rawMarkdown": "Congrats on the GM-ship! And congrats to the gold for all three of you!", "votes": null }, { "id": "613205", "postDate": "08/30/2019 09:53:35", "content": "Thanks, and congrat for your GM title, well deserved!
", "rawMarkdown": "Thanks, and congrat for your GM title, well deserved!", "votes": null }, { "id": "613259", "postDate": "08/30/2019 11:09:57", "content": "@divrikwicky \nMade network larger is that mean add more fully connected layers?
", "rawMarkdown": "divrikwicky \nMade network larger is that mean add more fully connected layers?", "votes": null }, { "id": "613273", "postDate": "08/30/2019 11:29:30", "content": "Not new layers but more units (neurons) in each layer.
", "rawMarkdown": "Not new layers but more units (neurons) in each layer.", "votes": null }, { "id": "613285", "postDate": "08/30/2019 11:49:20", "content": "\n\nI am new for NN.\nI studied a lot of your kernels.\nThanks.
\n\nAnother question is why you use 1 kernel size of Conv1D?\nAnd why add Conv1D layer not just use fully connected layers?
", "rawMarkdown": "divrikwicky \n\nI am new for NN.\nI studied a lot of your kernels.\nThanks.\n\nAnother question is why you use 1 kernel size of Conv1D?\nAnd why add Conv1D layer not just use fully connected layers?", "votes": null }, { "id": "613336", "postDate": "08/30/2019 12:30:22", "content": "Congratulations and thanks for sharing your solution @cpmpml !
", "rawMarkdown": "Congratulations and thanks for sharing your solution @cpmpml !", "votes": null }, { "id": "613411", "postDate": "08/30/2019 13:31:12", "content": "Congratulations! I'm sure you're already aware of this, but Aaron Paul who played Jesse also does the voice for Todd in your avatar!
", "rawMarkdown": "Congratulations! I'm sure you're already aware of this, but Aaron Paul who played Jesse also does the voice for Todd in your avatar!", "votes": null }, { "id": "613667", "postDate": "08/30/2019 18:05:02", "content": "@osspringfall kernel size is 1 because there is no locality between the list of atoms.\nConv1D is preferred because I wanted to apply the same feature extraction on each atom independent of their indices.
\n\n@bigironsphere thanks! I am aware of it but I have watched Breaking Bad much later than Bojack Horseman, which is an unusual order:)
", "rawMarkdown": "osspringfall kernel size is 1 because there is no locality between the list of atoms.\nConv1D is preferred because I wanted to apply the same feature extraction on each atom independent of their indices.\n\n@bigironsphere thanks! I am aware of it but I have watched Breaking Bad much later than Bojack Horseman, which is an unusual order:)", "votes": null }, { "id": "613679", "postDate": "08/30/2019 18:16:09", "content": "Congrats to your team. Just one word exemplary work.
", "rawMarkdown": "Congrats to your team. Just one word exemplary work.", "votes": null }, { "id": "614044", "postDate": "08/31/2019 03:03:06", "content": "@divrikwicky \nAm I right?\nIf I use fully connected layer, then each atom is dependent.\nIf I use Conv1D and set kernel size 1.\nThere is one input for one neuron, so each atom are independent.
\n\nCould you explain more about why you another Conv1D layer?\nIs that same reason as the first Conv1D?
", "rawMarkdown": "divrikwicky \nAm I right?\nIf I use fully connected layer, then each atom is dependent.\nIf I use Conv1D and set kernel size 1.\nThere is one input for one neuron, so each atom are independent.\n\nCould you explain more about why you another Conv1D layer?\nIs that same reason as the first Conv1D?", "votes": null }, { "id": "614290", "postDate": "08/31/2019 10:17:13", "content": "\n\n\nIf I use fully connected layer, then each atom are dependent.
\n
The real issue in that case is that you build a model that depends on the order in which atoms are listed in the molecule.
", "rawMarkdown": "> If I use fully connected layer, then each atom are dependent.\n\nThe real issue in that case is that you build a model that depends on the order in which atoms are listed in the molecule.", "votes": null }, { "id": "614548", "postDate": "08/31/2019 17:09:54", "content": "@cpmpml \nThanks.\nI understand.\nIf I use fully connected layer, then it is origin issue.\nIf I use Conv1D will turn to the another issue that each atom is independent.
", "rawMarkdown": "cpmpml \nThanks.\nI understand.\nIf I use fully connected layer, then it is origin issue.\nIf I use Conv1D will turn to the another issue that each atom is independent.", "votes": null }, { "id": "615020", "postDate": "09/01/2019 11:18:23", "content": "Big Congrats! I loved that you share the teaming experience. I have a question:
\n\nHow much does dihedral angle for 3 bonds gain? Thanks!
", "rawMarkdown": "Big Congrats! I loved that you share the teaming experience. I have a question:\n\nHow much does dihedral angle for 3 bonds gain? Thanks!", "votes": null }, { "id": "615071", "postDate": "09/01/2019 12:32:33", "content": "\n\n\nHow much does dihedral angle for 3 bonds gain?
\n
I haven't tested in the final models, but I think that when I added it is was at least 0.05
", "rawMarkdown": "> How much does dihedral angle for 3 bonds gain?\n\nI haven't tested in the final models, but I think that when I added it is was at least 0.05", "votes": null }, { "id": "615463", "postDate": "09/02/2019 01:26:20", "content": "Congrats @divrikwicky , @cpmpml and @outrunner ! Thanks for sharing.
", "rawMarkdown": "Congrats @divrikwicky , @cpmpml and @outrunner ! Thanks for sharing.", "votes": null }, { "id": "615724", "postDate": "09/02/2019 09:42:18", "content": "Congrats @cpmpml and your team ! Great Solution and thank you for sharing!!
", "rawMarkdown": "Congrats @cpmpml and your team ! Great Solution and thank you for sharing!!", "votes": null }, { "id": "616074", "postDate": "09/02/2019 16:55:55", "content": "Congratulations and thanks for sharing, I'm curious on how and why you apply learning rate decay with LGBM? Could you please share a bit more about this @cpmpml?
", "rawMarkdown": "Congratulations and thanks for sharing, I'm curious on how and why you apply learning rate decay with LGBM? Could you please share a bit more about this @cpmpml?", "votes": null }, { "id": "616091", "postDate": "09/02/2019 17:14:11", "content": "Thanks.
\n\n\n\n\nhow and why you apply learning rate decay with LGBM?
\n
How: using learning_rates parameter of lgb.train()
Why: because of MAE. lr decay helps converge.
", "rawMarkdown": "Thanks.\n\n> how and why you apply learning rate decay with LGBM?\n\nHow: using `learning_rates` parameter of lgb.train()\n\nWhy: because of MAE. lr decay helps converge.", "votes": null }, { "id": "616457", "postDate": "09/03/2019 06:33:36", "content": "Congrats and thanks a lot for sharing.
", "rawMarkdown": "Congrats and thanks a lot for sharing.", "votes": null }, { "id": "617247", "postDate": "09/03/2019 23:53:39", "content": "Congrats and thank you for sharing!!
\n\nCould you tell me why lr decay is effective in MAE metric?
", "rawMarkdown": "Congrats and thank you for sharing!!\n\nCould you tell me why lr decay is effective in MAE metric?", "votes": null }, { "id": "617598", "postDate": "09/04/2019 09:49:32", "content": "Someone explained it early in the forum, I don't have the link handy.
", "rawMarkdown": "Someone explained it early in the forum, I don't have the link handy.", "votes": null }, { "id": "617631", "postDate": "09/04/2019 10:26:13", "content": "I tried to find it and couldn't. Here is an example.
\n\nAssume 100 samples with target 0. Assume your current lgb model prediction is 0.5. Gradient of mae is 100 and if you use lr = 0.01 then you get an additional tree that moves the prediction to -0.5. Then you oscillate between 0.5 and -0.5 each time you add a tree. You could chose a smaller learning rate, bu then convergence would be extremely slow at the beginning. That's why starting with a large lr then decreasing will help.
", "rawMarkdown": "I tried to find it and couldn't. Here is an example.\n\nAssume 100 samples with target 0. Assume your current lgb model prediction is 0.5. Gradient of mae is 100 and if you use lr = 0.01 then you get an additional tree that moves the prediction to -0.5. Then you oscillate between 0.5 and -0.5 each time you add a tree. You could chose a smaller learning rate, bu then convergence would be extremely slow at the beginning. That's why starting with a large lr then decreasing will help.", "votes": null }, { "id": "618156", "postDate": "09/05/2019 00:00:16", "content": "Thank you for clear and concrete answer!! I understand. Good information!!
\n\nThis is like a NN gradient decent!!
", "rawMarkdown": "Thank you for clear and concrete answer!! I understand. Good information!!\n\nThis is like a NN gradient decent!!", "votes": null }, { "id": "618498", "postDate": "09/05/2019 08:52:58", "content": "\n\n\nThis is like a NN gradient decent!!
\n
Indeed, gradient boosted machines like lgb, xgb or catboost perform a gradient descent.
", "rawMarkdown": "> This is like a NN gradient decent!!\n\nIndeed, gradient boosted machines like lgb, xgb or catboost perform a gradient descent.", "votes": null }, { "id": "618546", "postDate": "09/05/2019 09:34:29", "content": "Anyone tried batch training (large batch training probably) with large datasets (that don't fit in RAM) using a GBM ?
", "rawMarkdown": "Anyone tried batch training (large batch training probably) with large datasets (that don't fit in RAM) using a GBM ?", "votes": null }, { "id": "618561", "postDate": "09/05/2019 09:55:24", "content": "There is an offline option for xgboost (and maybe lightgbm too).
", "rawMarkdown": "There is an offline option for xgboost (and maybe lightgbm too).", "votes": null }, { "id": "619300", "postDate": "09/06/2019 03:41:49", "content": "Thanks sir very informative and Congrats
", "rawMarkdown": "Thanks sir very informative and Congrats", "votes": null } ], "comments": [ { "id": 611852, "author_name": "taniaj", "author_url": "", "post_date": "08/29/2019 13:15:41", "content": "Very impressive work - Congrats!
", "votes": null, "replies": [] }, { "id": 611897, "author_name": "ogrellier", "author_url": "", "post_date": "08/29/2019 13:44:26", "content": "Thanks for sharing @cpmpml, this is a very interesting solution. \nI'll keep in mind residual training for future comps. \nSo Creative of you !
", "votes": null, "replies": [] }, { "id": 611915, "author_name": "wjholst", "author_url": "", "post_date": "08/29/2019 13:57:04", "content": "Excellent write up! Your approach where you used LGB to post-process the NN output is a very useful technique.
", "votes": null, "replies": [] }, { "id": 611957, "author_name": "marcogorelli", "author_url": "", "post_date": "08/29/2019 14:24:26", "content": "Thanks for this write-up! You're an excellent writer, your #1 spot in the discussions leader-board is well-deserved.
", "votes": null, "replies": [] }, { "id": 611963, "author_name": "aerdem4", "author_url": "", "post_date": "08/29/2019 14:30:23", "content": "It was a great experience to team up with top Kagglers like @cpmpml and @outrunner . As CPMP explained, I had a custom NN architecture. It wouldn't be competitive enough for gold without CPMP's boosting with LGB. And we wouldn't make it to 7th place without outrunner's NN expertise.
\n\nI have started working on this competition on Kaggle Kernels and could reach -1.4 via this model: https://www.kaggle.com/divrikwicky/lightweight-version-of-2-65-custom-nn Later, I have added more interaction features, made network larger and run it longer. I have only one model for all the types. Therefore, I use sample weights to mimic the competition metric. This seems to work better than training single models for each type (there are exception types). My NN uses Conv1D extract features for each atom, and there is another Conv1D layer after which uses the feedback from the molecule.
\n\nI am also happy to be a GM. Hopefully, other people who deserve become GM too, not the ones who try to cut corners:) I have learned a lot from the Kaggle community and my (ex-)teammates. And I hope I will continue learning. Every time I learn something very novel on Kaggle, I am like:\n
Thanks Ahmet. It was great teaming with you and Outrunner.
", "votes": null, "replies": [] }, { "id": 613113, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 08:19:41", "content": "Congrats on the GM-ship! And congrats to the gold for all three of you!
", "votes": null, "replies": [] }, { "id": 613259, "author_name": "osspringfall", "author_url": "", "post_date": "08/30/2019 11:09:57", "content": "@divrikwicky \nMade network larger is that mean add more fully connected layers?
", "votes": null, "replies": [] }, { "id": 613273, "author_name": "aerdem4", "author_url": "", "post_date": "08/30/2019 11:29:30", "content": "Not new layers but more units (neurons) in each layer.
", "votes": null, "replies": [] }, { "id": 613285, "author_name": "osspringfall", "author_url": "", "post_date": "08/30/2019 11:49:20", "content": "\n\nI am new for NN.\nI studied a lot of your kernels.\nThanks.
\n\nAnother question is why you use 1 kernel size of Conv1D?\nAnd why add Conv1D layer not just use fully connected layers?
", "votes": null, "replies": [] }, { "id": 613411, "author_name": "bigironsphere", "author_url": "", "post_date": "08/30/2019 13:31:12", "content": "Congratulations! I'm sure you're already aware of this, but Aaron Paul who played Jesse also does the voice for Todd in your avatar!
", "votes": null, "replies": [] }, { "id": 613667, "author_name": "aerdem4", "author_url": "", "post_date": "08/30/2019 18:05:02", "content": "@osspringfall kernel size is 1 because there is no locality between the list of atoms.\nConv1D is preferred because I wanted to apply the same feature extraction on each atom independent of their indices.
\n\n@bigironsphere thanks! I am aware of it but I have watched Breaking Bad much later than Bojack Horseman, which is an unusual order:)
", "votes": null, "replies": [] }, { "id": 614044, "author_name": "osspringfall", "author_url": "", "post_date": "08/31/2019 03:03:06", "content": "@divrikwicky \nAm I right?\nIf I use fully connected layer, then each atom is dependent.\nIf I use Conv1D and set kernel size 1.\nThere is one input for one neuron, so each atom are independent.
\n\nCould you explain more about why you another Conv1D layer?\nIs that same reason as the first Conv1D?
", "votes": null, "replies": [] }, { "id": 614290, "author_name": "cpmpml", "author_url": "", "post_date": "08/31/2019 10:17:13", "content": "\n\n\nIf I use fully connected layer, then each atom are dependent.
\n
The real issue in that case is that you build a model that depends on the order in which atoms are listed in the molecule.
", "votes": null, "replies": [] }, { "id": 614548, "author_name": "osspringfall", "author_url": "", "post_date": "08/31/2019 17:09:54", "content": "@cpmpml \nThanks.\nI understand.\nIf I use fully connected layer, then it is origin issue.\nIf I use Conv1D will turn to the another issue that each atom is independent.
", "votes": null, "replies": [] } ] }, { "id": 612194, "author_name": "veeralakrishna", "author_url": "", "post_date": "08/29/2019 16:43:21", "content": "Very impressive work - Congrats!\nExcellent write up!\nThanks for sharing @cpmpml
", "votes": null, "replies": [] }, { "id": 612265, "author_name": "michelml", "author_url": "", "post_date": "08/29/2019 17:27:41", "content": "Thanks @divrikwicky @cpmpml and congrats! Would you be willing to share your computing resources for this competition + time to train your models?
", "votes": null, "replies": [ { "id": 612324, "author_name": "cpmpml", "author_url": "", "post_date": "08/29/2019 18:10:56", "content": "I have 2 1080 Ti that were mostly used to train Ahmet's model, and partly to develop mine. Ahmet only has 2 K40. And Outrunner has 2 1080 Ti as well.
", "votes": null, "replies": [] } ] }, { "id": 612361, "author_name": "osspringfall", "author_url": "", "post_date": "08/29/2019 18:38:20", "content": "Thanks for sharing!
\n\n@cpmpml \nCan I ask a question? \nYou used lgb to predict the residuals of the NN.\nWhat is the features for lgb?\nIs that you used the predictions of NN to be the features for lgb?
", "votes": null, "replies": [ { "id": 612364, "author_name": "cpmpml", "author_url": "", "post_date": "08/29/2019 18:40:44", "content": "I used the same features ad lgb standalone plus the NN prediction. I think this works because the features used for lgb are very different from what the NN learned. Think of it as stacking (using the NN prediction as feature), but also using original features.
", "votes": null, "replies": [] }, { "id": 612989, "author_name": "osspringfall", "author_url": "", "post_date": "08/30/2019 06:29:52", "content": "Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 612375, "author_name": "proletheus", "author_url": "", "post_date": "08/29/2019 18:46:43", "content": "\n\n\nWith this I got to 2.0 on LB with one fold the day I entered.
\n
what the hell, so thats what a grandmaster can do... you motivate me to keep going:)
", "votes": null, "replies": [] }, { "id": 612596, "author_name": "sukrullex", "author_url": "", "post_date": "08/29/2019 22:15:22", "content": "I was watching your team through whole competition and was looking forward to see your approach. Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 612701, "author_name": "khahuras", "author_url": "", "post_date": "08/30/2019 00:28:10", "content": "Congrats on your team performance. We have some common ideas regarding symmetry and \"middle atom\". What I learned from you in this competition is this:\n\"Lgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8.\"
", "votes": null, "replies": [] }, { "id": 612791, "author_name": "senkin13", "author_url": "", "post_date": "08/30/2019 02:21:29", "content": "Really creative solution,I want to try next time.
", "votes": null, "replies": [ { "id": 613205, "author_name": "cpmpml", "author_url": "", "post_date": "08/30/2019 09:53:35", "content": "Thanks, and congrat for your GM title, well deserved!
", "votes": null, "replies": [] } ] }, { "id": 612847, "author_name": "projdev", "author_url": "", "post_date": "08/30/2019 03:33:34", "content": "Congratulations! Amazing solutions! :-)
", "votes": null, "replies": [] }, { "id": 613336, "author_name": "rohanrao", "author_url": "", "post_date": "08/30/2019 12:30:22", "content": "Congratulations and thanks for sharing your solution @cpmpml !
", "votes": null, "replies": [] }, { "id": 613679, "author_name": "manojprabhaakr", "author_url": "", "post_date": "08/30/2019 18:16:09", "content": "Congrats to your team. Just one word exemplary work.
", "votes": null, "replies": [] }, { "id": 615020, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2019 11:18:23", "content": "Big Congrats! I loved that you share the teaming experience. I have a question:
\n\nHow much does dihedral angle for 3 bonds gain? Thanks!
", "votes": null, "replies": [ { "id": 615071, "author_name": "cpmpml", "author_url": "", "post_date": "09/01/2019 12:32:33", "content": "\n\n\nHow much does dihedral angle for 3 bonds gain?
\n
I haven't tested in the final models, but I think that when I added it is was at least 0.05
", "votes": null, "replies": [] } ] }, { "id": 615463, "author_name": "titericz", "author_url": "", "post_date": "09/02/2019 01:26:20", "content": "Congrats @divrikwicky , @cpmpml and @outrunner ! Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 615724, "author_name": "duykhanh99", "author_url": "", "post_date": "09/02/2019 09:42:18", "content": "Congrats @cpmpml and your team ! Great Solution and thank you for sharing!!
", "votes": null, "replies": [] }, { "id": 616074, "author_name": "optimo", "author_url": "", "post_date": "09/02/2019 16:55:55", "content": "Congratulations and thanks for sharing, I'm curious on how and why you apply learning rate decay with LGBM? Could you please share a bit more about this @cpmpml?
", "votes": null, "replies": [ { "id": 616091, "author_name": "cpmpml", "author_url": "", "post_date": "09/02/2019 17:14:11", "content": "Thanks.
\n\n\n\n\nhow and why you apply learning rate decay with LGBM?
\n
How: using learning_rates parameter of lgb.train()
Why: because of MAE. lr decay helps converge.
", "votes": null, "replies": [] }, { "id": 617247, "author_name": "go5kuramubon", "author_url": "", "post_date": "09/03/2019 23:53:39", "content": "Congrats and thank you for sharing!!
\n\nCould you tell me why lr decay is effective in MAE metric?
", "votes": null, "replies": [] }, { "id": 617598, "author_name": "cpmpml", "author_url": "", "post_date": "09/04/2019 09:49:32", "content": "Someone explained it early in the forum, I don't have the link handy.
", "votes": null, "replies": [] }, { "id": 617631, "author_name": "cpmpml", "author_url": "", "post_date": "09/04/2019 10:26:13", "content": "I tried to find it and couldn't. Here is an example.
\n\nAssume 100 samples with target 0. Assume your current lgb model prediction is 0.5. Gradient of mae is 100 and if you use lr = 0.01 then you get an additional tree that moves the prediction to -0.5. Then you oscillate between 0.5 and -0.5 each time you add a tree. You could chose a smaller learning rate, bu then convergence would be extremely slow at the beginning. That's why starting with a large lr then decreasing will help.
", "votes": null, "replies": [] }, { "id": 618156, "author_name": "go5kuramubon", "author_url": "", "post_date": "09/05/2019 00:00:16", "content": "Thank you for clear and concrete answer!! I understand. Good information!!
\n\nThis is like a NN gradient decent!!
", "votes": null, "replies": [] }, { "id": 618498, "author_name": "cpmpml", "author_url": "", "post_date": "09/05/2019 08:52:58", "content": "\n\n\nThis is like a NN gradient decent!!
\n
Indeed, gradient boosted machines like lgb, xgb or catboost perform a gradient descent.
", "votes": null, "replies": [] }, { "id": 618546, "author_name": "optimo", "author_url": "", "post_date": "09/05/2019 09:34:29", "content": "Anyone tried batch training (large batch training probably) with large datasets (that don't fit in RAM) using a GBM ?
", "votes": null, "replies": [] }, { "id": 618561, "author_name": "cpmpml", "author_url": "", "post_date": "09/05/2019 09:55:24", "content": "There is an offline option for xgboost (and maybe lightgbm too).
", "votes": null, "replies": [] } ] }, { "id": 616457, "author_name": "rhhridoy", "author_url": "", "post_date": "09/03/2019 06:33:36", "content": "Congrats and thanks a lot for sharing.
", "votes": null, "replies": [] }, { "id": 619300, "author_name": "vinayak123tyagi", "author_url": "", "post_date": "09/06/2019 03:41:49", "content": "Thanks sir very informative and Congrats
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "611829": "First of all, I'd like to thank my team mates, without whom I would not have landed where we landed, and even would not have entered the competition. Let me also way that I'm very happy to see Ahmet become competition GM. And Outrunner is only one gold away from it. I'm sure he'll become GM in his next competition. I was very lucky to team with them. It proves 1 + 1 + 1 > 3.\n\nI also want to thank @inversion, Kaggle staff in general, and CHAMPS for setting up this very challenging, unflawed contest. It is refreshing to not have any leak. Using test data wasn't helpful really either. And the way @inversion managed the two crisis (QM9 use, and unplanned kernel share) was very effective.\n\nI was very reluctant to enter this competition, for two reasons. First, I was convinced that people with domain knowledge would have an unfair advantage. I was right as the top of LB shows a high proportion of teams with chemists on board. Second, it was clear from early discussions that graph NN would be key, and I had absolutely no experience with it. This second reason is a bit weak given most of us here are in same situation!\n\nThen Ahmet asked me if I would join, because we almost teamed in a previous competition and said we would team ASAP. It motivated me to complete an initial effort using lightgbm. I developed a core set of features, then added features specific to each type. The features are based on the graph defined by bonds computed using open babel (thanks @borisd for the starter code). Angles for 2 bonds path, and dihedral angle for 3 bonds paths were key. I then added features based on the number of 1, 2, 3 bond paths starting on every atom on the path, using aggregates like number by bond type, angle averages, distance averages, etc. Depending on the type, number of features was between 500 and 2000. I trained one lgb per type. With this I got to 2.0 on LB with one fold the day I entered. Once trained on all folds, this model got 2.1 on LB. \n\nThere are few things I did to improve the model. \n\nFirst, I use a very large number of leaves, 2**10 -1. I am not sure why this is good, but it was. Other parameter tuning wasn't very important.\n\nSecond, I duplicated train and test rows with types 2JHH and 3JHH by swapping the two atoms, before feature engineering, because scalar coupling is symmetric. Final prediction for one row in the original train or test was the average of the two rows predictions after expansion. This was mentioned early in the forum I think.\n\nThird, for 3Jxx bond types, there is often more than one 3 bonds path between the two atoms. Here again I duplicated, using one row for each 3 bonds path between the two atoms, then averaging their predictions back for final prediction. I did not see this discussed anywhere.\n\nRow duplication improves lgb score by about 0.002, not much, but still useful.\n\nI then teamed with Ahmet, and given he had a NN with much better performance than my lgb model I gave up on my base model. I'll let Ahmet describe his NN (edit: description is available [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421#611963)). Let me just say that with some improvements, like using logcosh as loss, and expanding its size, we got to -2.64 on LB using single 4 fold cv run. I was very happy given this moved us way above where lgb could move us to. \n\nLgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8. \n\nTo make sure we get a gold medal we knew we would need another strong model. I started developing a graph NN from scratch in keras, and was getting CV o -1.9 and LB -2.08 nwith just 4 features (atom and bond type, mulliken charge, and dihedral angle for 3 bond types). I stopped working on it when we teamed with Outrunner. Indeed, Outrunner skills in NN clearly outperfom mine, by far. He had a set of 13 NN models that yield almost -2.9 LB once bagged. He further improved them to yield -2.92 or so soon after merging. I will let him describe these models if he wants to. \n\nThe first thing we did after teaming was to run my lgb postprocessing on his validation prediction. He was using a single train/val split with only 3% or so of validation data. Yet, even with this limited data lgb could extract about 0.025 more. Blending the result with what we had added another 0.03 or so, which moved us better than -2.95. \n\nThe last progress we made was based on the NN approach used. A lot of people have used Graph NN where one molecule is a sample, and all coupling for that molecule can be predicted together. Outrunner and Ahmet use another way, with one sample per train and test row. They have features that capture the relative position of every atom compared to the two for which the coupling is computed. For instance distances to these two atoms, and an angle, and more. But distance to the two atoms and an angle doesn't uniquely identify atom positions. There is a circular symmetry. A way to remove this symmetry is to consider other atoms. Ahmet was doing it but not Outrunner. I suggested to use middle atoms. For 2JHx, a0 and a1 are both bond to a middle atom am. For 3JHx, a0 is bond to am0 which is bond to am1 which is bond to a1. Outrunner added distance to am, or am0/am1 depending on the type. This boosted performance to where we finally ended. Unfortunately, given we teamed late, he could only retrain 3 of his models with the new feature. Bagging these leads to -2.99 LB. Running lgb on top of it and blending with Ahmet-CPMP blend yields -3.032. I guess we could have landed higher with more models retrained, but we lacked time and/or GPUs.\n\nBottom line is I am happy this very challenging competition ended. I probably should have entered earlier, my first sub is from July 22, only 5 weeks ago. But teaming with two talented people more than made for the lack of time.\n\nEdt, you'll find Ahmet NN description [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421#611963). Outrunner told me his NN is similar to Ahmet's hence he won't provide a separate description.", "611852": "Very impressive work - Congrats!", "611897": "Thanks for sharing @cpmpml, this is a very interesting solution. \nI'll keep in mind residual training for future comps. \nSo Creative of you !", "611915": "Excellent write up! Your approach where you used LGB to post-process the NN output is a very useful technique.", "611957": "Thanks for this write-up! You're an excellent writer, your #1 spot in the discussions leader-board is well-deserved.", "611963": "It was a great experience to team up with top Kagglers like @cpmpml and @outrunner . As CPMP explained, I had a custom NN architecture. It wouldn't be competitive enough for gold without CPMP's boosting with LGB. And we wouldn't make it to 7th place without outrunner's NN expertise.\n\nI have started working on this competition on Kaggle Kernels and could reach -1.4 via this model: https://www.kaggle.com/divrikwicky/lightweight-version-of-2-65-custom-nn Later, I have added more interaction features, made network larger and run it longer. I have only one model for all the types. Therefore, I use sample weights to mimic the competition metric. This seems to work better than training single models for each type (there are exception types). My NN uses Conv1D extract features for each atom, and there is another Conv1D layer after which uses the feedback from the molecule. \n\nI am also happy to be a GM. Hopefully, other people who deserve become GM too, not the ones who try to cut corners:) I have learned a lot from the Kaggle community and my (ex-)teammates. And I hope I will continue learning. Every time I learn something very novel on Kaggle, I am like:\n", "611975": "Thanks Ahmet. It was great teaming with you and Outrunner.", "612194": "Very impressive work - Congrats!\nExcellent write up!\nThanks for sharing @cpmpml", "612265": "Thanks @divrikwicky @cpmpml and congrats! Would you be willing to share your computing resources for this competition + time to train your models?", "612324": "I have 2 1080 Ti that were mostly used to train Ahmet's model, and partly to develop mine. Ahmet only has 2 K40. And Outrunner has 2 1080 Ti as well.", "612361": "Thanks for sharing!\n\n@cpmpml \nCan I ask a question? \nYou used lgb to predict the residuals of the NN.\nWhat is the features for lgb?\nIs that you used the predictions of NN to be the features for lgb?", "612364": "I used the same features ad lgb standalone plus the NN prediction. I think this works because the features used for lgb are very different from what the NN learned. Think of it as stacking (using the NN prediction as feature), but also using original features.", "612375": "> With this I got to 2.0 on LB with one fold the day I entered.\n\nwhat the hell, so thats what a grandmaster can do... you motivate me to keep going:)", "612596": "I was watching your team through whole competition and was looking forward to see your approach. Thanks for sharing.", "612701": "Congrats on your team performance. We have some common ideas regarding symmetry and \"middle atom\". What I learned from you in this competition is this:\n\"Lgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8.\"", "612791": "Really creative solution,I want to try next time.", "612847": "Congratulations! Amazing solutions! :-)", "612989": "Thanks a lot!", "613113": "Congrats on the GM-ship! And congrats to the gold for all three of you!", "613205": "Thanks, and congrat for your GM title, well deserved!", "613259": "divrikwicky \nMade network larger is that mean add more fully connected layers?", "613273": "Not new layers but more units (neurons) in each layer.", "613285": "divrikwicky \n\nI am new for NN.\nI studied a lot of your kernels.\nThanks.\n\nAnother question is why you use 1 kernel size of Conv1D?\nAnd why add Conv1D layer not just use fully connected layers?", "613336": "Congratulations and thanks for sharing your solution @cpmpml !", "613411": "Congratulations! I'm sure you're already aware of this, but Aaron Paul who played Jesse also does the voice for Todd in your avatar!", "613667": "osspringfall kernel size is 1 because there is no locality between the list of atoms.\nConv1D is preferred because I wanted to apply the same feature extraction on each atom independent of their indices.\n\n@bigironsphere thanks! I am aware of it but I have watched Breaking Bad much later than Bojack Horseman, which is an unusual order:)", "613679": "Congrats to your team. Just one word exemplary work.", "614044": "divrikwicky \nAm I right?\nIf I use fully connected layer, then each atom is dependent.\nIf I use Conv1D and set kernel size 1.\nThere is one input for one neuron, so each atom are independent.\n\nCould you explain more about why you another Conv1D layer?\nIs that same reason as the first Conv1D?", "614290": "> If I use fully connected layer, then each atom are dependent.\n\nThe real issue in that case is that you build a model that depends on the order in which atoms are listed in the molecule.", "614548": "cpmpml \nThanks.\nI understand.\nIf I use fully connected layer, then it is origin issue.\nIf I use Conv1D will turn to the another issue that each atom is independent.", "615020": "Big Congrats! I loved that you share the teaming experience. I have a question:\n\nHow much does dihedral angle for 3 bonds gain? Thanks!", "615071": "> How much does dihedral angle for 3 bonds gain?\n\nI haven't tested in the final models, but I think that when I added it is was at least 0.05", "615463": "Congrats @divrikwicky , @cpmpml and @outrunner ! Thanks for sharing.", "615724": "Congrats @cpmpml and your team ! Great Solution and thank you for sharing!!", "616074": "Congratulations and thanks for sharing, I'm curious on how and why you apply learning rate decay with LGBM? Could you please share a bit more about this @cpmpml?", "616091": "Thanks.\n\n> how and why you apply learning rate decay with LGBM?\n\nHow: using `learning_rates` parameter of lgb.train()\n\nWhy: because of MAE. lr decay helps converge.", "616457": "Congrats and thanks a lot for sharing.", "617247": "Congrats and thank you for sharing!!\n\nCould you tell me why lr decay is effective in MAE metric?", "617598": "Someone explained it early in the forum, I don't have the link handy.", "617631": "I tried to find it and couldn't. Here is an example.\n\nAssume 100 samples with target 0. Assume your current lgb model prediction is 0.5. Gradient of mae is 100 and if you use lr = 0.01 then you get an additional tree that moves the prediction to -0.5. Then you oscillate between 0.5 and -0.5 each time you add a tree. You could chose a smaller learning rate, bu then convergence would be extremely slow at the beginning. That's why starting with a large lr then decreasing will help.", "618156": "Thank you for clear and concrete answer!! I understand. Good information!!\n\nThis is like a NN gradient decent!!", "618498": "> This is like a NN gradient decent!!\n\nIndeed, gradient boosted machines like lgb, xgb or catboost perform a gradient descent.", "618546": "Anyone tried batch training (large batch training probably) with large datasets (that don't fit in RAM) using a GBM ?", "618561": "There is an offline option for xgboost (and maybe lightgbm too).", "619300": "Thanks sir very informative and Congrats" }, "source": "existing_discussion", "complete": true, "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3232843, "solution_url": "/competitions/champs-scalar-coupling/writeups/pinkman-chemistry-lab-solution-7th-cpmp-view" }