![\"\"](\"https://i.imgur.com/VRFiEgk.png\")
Big thanks to CHAMPS for hosting this competition. This has been by far my favorite competition that I've taken part in. Mainly because:\n- No leaks\n- No shakeup (solid public/private LB relationship)\n- No simulated data\n- Helps a good cause\n- Some very strong teams with a good mix of kagglers and subject matter experts.\n- It was fun to see the top teams surpass the host's expectations (breaking -3.0)
\n\nAs many have already discussed- tabular models could only get you so far in this competition. About three weeks ago, when this became clear, our team decided to focus only on GNN models, Prior to that I had been working exclusively on tabular models. My best personal blend of tabular models ended up getting -2.101 on the private LB (would have been ~100th place and silver medal). This is without blending with any GNN models.
\n\n
Big thanks to my teammates @titericz, @voglinio, @kingychiu, and especially @hughshaoqz - who's GNN architecture was the main reason our team ended up as high as we did.
\n\nI've made public my github repo with all of the code I used to achieve this here: https://github.com/RobMulla/kaggle-champs-2019/
\n\nEven though this competition didn't end up being much about tabular models there were a ton of things I learned from the process:\n- Ignore public kernels at first - manually create features, add them only if they help CV - I took this approach from @cpmpml . I purposefully chose to ignore features from public kernels until I completely was drained of ideas. I tried to make most of my features such that they didn't rely on the x, y, z coordinates but only each atom's relationship to each other. I wanted every feature to make sense to me why it would be useful.\n- Having a solid CV is important - This wasn't too hard in this competition, I just made sure to use GroupKFold on molecule name to make sure there was no overlap between molecules in train and validation. I found this CV had a very stable CV to LB relationship and blending of OOF predictions worked well.\n- Attack the problem like 8 mini-competitions. Early on I decided to split the feature selection, tuning and blending by bond types. This was pretty obvious but did allow me to iterate over specific types, and add features only for types where they helped, and explore parameter tuning per type.\n- Catboost + GPU is 👍 - I found that compared to LightGBM and XGBoost, catboost worked really well utilizing GPUs and converging quite quickly. I relied on it heavily for my tabular models, mainly because of speed.\n- Training was really slow. - Even with training types individually and using catboost with 2 1080tis - training was really slow for this competition. Training all types easily took 72+hours. I tried to develop new features while models were training/validating older features. Initially I was training on only 2 folds and eventually bumped it up to 3 folds.\n- Track what you've done - so you remember what's worked and what didn't - I made my code such that every feature engineering step was numbered (FE001 - FE025), and every model was tracked and numbered (M001 - M062). More than anything, this helped me mentally organize different ideas I was iterating over. It also made blending and stacking much easier to track. I also had my code append to a tracking CSV every time I trained a model: saving the CV score, parameters used, iterations taken, and a bunch more stuff. My tracking CSVs can be found in the repo here if you want to see how I slowly improved each type: https://github.com/RobMulla/kaggle-champs-2019/tree/master/tracking\n![\"\"](\"https://i.imgur.com/DvB4YYQ.png\")
\n- Will it blend? Yes it will! - using OOF predictions I used a pretty crude approach to blending that seemed to work pretty well. This is the notebook I used to create the blend that ended up scoring -2.1 private LB : https://github.com/RobMulla/kaggle-champs-2019/blob/master/notebooks/BLEND-044.ipynb
Excited to read some of the great solutions from the top teams!
\n\n......and to all the tabular folks. I'll see you over in IEEE 😄
", "messages": [ { "id": 610870, "postDate": "2019-08-29T02:18:34.750Z", "content": "Big thanks to CHAMPS for hosting this competition. This has been by far my favorite competition that I've taken part in. Mainly because:\n- No leaks\n- No shakeup (solid public/private LB relationship)\n- No simulated data\n- Helps a good cause\n- Some very strong teams with a good mix of kagglers and subject matter experts.\n- It was fun to see the top teams surpass the host's expectations (breaking -3.0)
\n\nAs many have already discussed- tabular models could only get you so far in this competition. About three weeks ago, when this became clear, our team decided to focus only on GNN models, Prior to that I had been working exclusively on tabular models. My best personal blend of tabular models ended up getting -2.101 on the private LB (would have been ~100th place and silver medal). This is without blending with any GNN models.
\n\n
Big thanks to my teammates @titericz, @voglinio, @kingychiu, and especially @hughshaoqz - who's GNN architecture was the main reason our team ended up as high as we did.
\n\nI've made public my github repo with all of the code I used to achieve this here: https://github.com/RobMulla/kaggle-champs-2019/
\n\nEven though this competition didn't end up being much about tabular models there were a ton of things I learned from the process:\n- Ignore public kernels at first - manually create features, add them only if they help CV - I took this approach from @cpmpml . I purposefully chose to ignore features from public kernels until I completely was drained of ideas. I tried to make most of my features such that they didn't rely on the x, y, z coordinates but only each atom's relationship to each other. I wanted every feature to make sense to me why it would be useful.\n- Having a solid CV is important - This wasn't too hard in this competition, I just made sure to use GroupKFold on molecule name to make sure there was no overlap between molecules in train and validation. I found this CV had a very stable CV to LB relationship and blending of OOF predictions worked well.\n- Attack the problem like 8 mini-competitions. Early on I decided to split the feature selection, tuning and blending by bond types. This was pretty obvious but did allow me to iterate over specific types, and add features only for types where they helped, and explore parameter tuning per type.\n- Catboost + GPU is 👍 - I found that compared to LightGBM and XGBoost, catboost worked really well utilizing GPUs and converging quite quickly. I relied on it heavily for my tabular models, mainly because of speed.\n- Training was really slow. - Even with training types individually and using catboost with 2 1080tis - training was really slow for this competition. Training all types easily took 72+hours. I tried to develop new features while models were training/validating older features. Initially I was training on only 2 folds and eventually bumped it up to 3 folds.\n- Track what you've done - so you remember what's worked and what didn't - I made my code such that every feature engineering step was numbered (FE001 - FE025), and every model was tracked and numbered (M001 - M062). More than anything, this helped me mentally organize different ideas I was iterating over. It also made blending and stacking much easier to track. I also had my code append to a tracking CSV every time I trained a model: saving the CV score, parameters used, iterations taken, and a bunch more stuff. My tracking CSVs can be found in the repo here if you want to see how I slowly improved each type: https://github.com/RobMulla/kaggle-champs-2019/tree/master/tracking\n\n- Will it blend? Yes it will! - using OOF predictions I used a pretty crude approach to blending that seemed to work pretty well. This is the notebook I used to create the blend that ended up scoring -2.1 private LB : https://github.com/RobMulla/kaggle-champs-2019/blob/master/notebooks/BLEND-044.ipynb
Excited to read some of the great solutions from the top teams!
\n\n......and to all the tabular folks. I'll see you over in IEEE 😄
", "rawMarkdown": "Big thanks to CHAMPS for hosting this competition. This has been by far my favorite competition that I've taken part in. Mainly because:\n- No leaks\n- No shakeup (solid public/private LB relationship)\n- No simulated data\n- Helps a good cause\n- Some very strong teams with a good mix of kagglers and subject matter experts.\n- It was fun to see the top teams surpass the host's expectations (breaking -3.0)\n\nAs many have already discussed- tabular models could only get you so far in this competition. About three weeks ago, when this became clear, our team decided to focus only on GNN models, Prior to that I had been working exclusively on tabular models. My best personal blend of tabular models ended up getting -2.101 on the private LB (would have been ~100th place and silver medal). This is without blending with any GNN models.\n\n\n\nBig thanks to my teammates @titericz, @voglinio, @kingychiu, and especially @hughshaoqz - who's GNN architecture was the main reason our team ended up as high as we did.\n\nI've made public my github repo with all of the code I used to achieve this here: https://github.com/RobMulla/kaggle-champs-2019/\n\nEven though this competition didn't end up being much about tabular models there were a ton of things I learned from the process:\n- **Ignore public kernels at first - manually create features, add them only if they help CV** - I took this approach from @cpmpml . I purposefully chose to ignore features from public kernels until I completely was drained of ideas. I tried to make most of my features such that they didn't rely on the x, y, z coordinates but only each atom's relationship to each other. I wanted every feature to make sense to me why it would be useful.\n- **Having a solid CV is important** - This wasn't too hard in this competition, I just made sure to use GroupKFold on molecule name to make sure there was no overlap between molecules in train and validation. I found this CV had a very stable CV to LB relationship and blending of OOF predictions worked well.\n- **Attack the problem like 8 mini-competitions.** Early on I decided to split the feature selection, tuning and blending by bond types. This was pretty obvious but did allow me to iterate over specific types, and add features only for types where they helped, and explore parameter tuning per type.\n- **Catboost + GPU is 👍** - I found that compared to LightGBM and XGBoost, catboost worked really well utilizing GPUs and converging quite quickly. I relied on it heavily for my tabular models, mainly because of speed.\n- **Training was really slow.** - Even with training types individually and using catboost with 2 1080tis - training was really slow for this competition. Training all types easily took 72+hours. I tried to develop new features while models were training/validating older features. Initially I was training on only 2 folds and eventually bumped it up to 3 folds.\n- **Track what you've done - so you remember what's worked and what didn't** - I made my code such that every feature engineering step was numbered (FE001 - FE025), and every model was tracked and numbered (M001 - M062). More than anything, this helped me mentally organize different ideas I was iterating over. It also made blending and stacking much easier to track. I also had my code append to a tracking CSV every time I trained a model: saving the CV score, parameters used, iterations taken, and a bunch more stuff. My tracking CSVs can be found in the repo here if you want to see how I slowly improved each type: https://github.com/RobMulla/kaggle-champs-2019/tree/master/tracking\n\n- **Will it blend? Yes it will!** - using OOF predictions I used a pretty crude approach to blending that seemed to work pretty well. This is the notebook I used to create the blend that ended up scoring -2.1 private LB : https://github.com/RobMulla/kaggle-champs-2019/blob/master/notebooks/BLEND-044.ipynb\n\nExcited to read some of the great solutions from the top teams!\n\n......and to all the tabular folks. I'll see you over in IEEE 😄 ", "votes": 37 }, { "id": 611458, "postDate": "2019-08-29T09:17:00.803Z", "content": "I was a pleasure working with you guys @titericz, @robikscube, @kingychiu, @hughshaoqz. Rob the way you organized the material and codes for this competition was inspiring! We were a team of diverse skills and realizing that was one of the best things of this competition!! I learned a lot from you. Thank you!!
", "rawMarkdown": "I was a pleasure working with you guys @titericz, @robikscube, @kingychiu, @hughshaoqz. Rob the way you organized the material and codes for this competition was inspiring! We were a team of diverse skills and realizing that was one of the best things of this competition!! I learned a lot from you. Thank you!! ", "votes": 3, "replies": [ { "id": 612101, "postDate": "2019-08-29T15:51:33.130Z", "content": "@voglinio, I'm new on Kaggle, did you guys know each other beforehand or you just partnered up based on skills/current LB/or anything else?
", "rawMarkdown": "@voglinio, I'm new on Kaggle, did you guys know each other beforehand or you just partnered up based on skills/current LB/or anything else?", "votes": 2 }, { "id": 615520, "postDate": "2019-09-02T03:33:13.510Z", "content": "@voglinio The pleasure was all mine. Glad we got to team. It was fun time learning about new types of models I've never been exposed to before this competition.
\n\n@michelml - Typically you reach out to people close to you on the leaderboard. Once you've taken part in a few competitions you also start to recognize the same people from the message boards and leaderboard and get an idea of people to reach out to about teaming up.
", "rawMarkdown": "@voglinio The pleasure was all mine. Glad we got to team. It was fun time learning about new types of models I've never been exposed to before this competition.\n\n@michelml - Typically you reach out to people close to you on the leaderboard. Once you've taken part in a few competitions you also start to recognize the same people from the message boards and leaderboard and get an idea of people to reach out to about teaming up.", "votes": 2 } ] }, { "id": 611228, "postDate": "2019-08-29T07:13:22.637Z", "content": "Nice work and thanks for sharing. Good idea with the tracking.
", "rawMarkdown": "Nice work and thanks for sharing. Good idea with the tracking.", "votes": 4 }, { "id": 616838, "postDate": "2019-09-03T14:17:11.867Z", "content": "@robikscube - I think your write-up here is a great example on how to approach any competition. Great words of wisdom!
", "rawMarkdown": "@robikscube - I think your write-up here is a great example on how to approach any competition. Great words of wisdom!", "votes": 2 }, { "id": 614543, "postDate": "2019-08-31T17:05:07.587Z", "content": "@robikscube How many rounds did you use for LGBM? I never went over 12,000 or so. Not sure if results got any better after that.
", "rawMarkdown": "@robikscube How many rounds did you use for LGBM? I never went over 12,000 or so. Not sure if results got any better after that.", "votes": 2, "replies": [ { "id": 615096, "postDate": "2019-09-01T13:01:57.597Z", "content": "You can actually see in my logs the number of estimators/boosting rounds used. I used catboost, which doesn't exactly compare to LGBM one-to-one, but many types I trained for up to 1,000,000 rounds.
", "rawMarkdown": "You can actually see in my logs the number of estimators/boosting rounds used. I used catboost, which doesn't exactly compare to LGBM one-to-one, but many types I trained for up to 1,000,000 rounds.", "votes": 2 } ] }, { "id": 613105, "postDate": "2019-08-30T08:13:50.423Z", "content": "Nicely done!
", "rawMarkdown": "Nicely done!", "votes": 2, "replies": [ { "id": 615515, "postDate": "2019-09-02T03:29:14.650Z", "content": "Thanks @fnands
", "rawMarkdown": "Thanks @fnands ", "votes": 1 } ] }, { "id": 613089, "postDate": "2019-08-30T07:59:38.613Z", "content": "Thank you @robikscube and also thanks @titericz @voglinio @kingychiu! I really enjoy working with you. Love the team!
", "rawMarkdown": "Thank you @robikscube and also thanks @titericz @voglinio @kingychiu! I really enjoy working with you. Love the team!", "votes": 2, "replies": [ { "id": 615517, "postDate": "2019-09-02T03:29:38.093Z", "content": "I enjoyed teaming! I learned a lot from you and the rest of the team.
", "rawMarkdown": "I enjoyed teaming! I learned a lot from you and the rest of the team.", "votes": 1 } ] }, { "id": 612093, "postDate": "2019-08-29T15:47:47.290Z", "content": "First github repo share - thanks @robikscube
", "rawMarkdown": "First github repo share - thanks @robikscube ", "votes": 2 }, { "id": 612044, "postDate": "2019-08-29T15:20:14.173Z", "content": "Thanks for sharing and congrats on the end result.
", "rawMarkdown": "Thanks for sharing and congrats on the end result.", "votes": 2 }, { "id": 610889, "postDate": "2019-08-29T02:37:58.493Z", "content": "Great Work...!! @robikscube
", "rawMarkdown": "Great Work...!! @robikscube ", "votes": 1 }, { "id": 612099, "postDate": "2019-08-29T15:51:01.027Z", "rawMarkdown": "", "isDeleted": true }, { "id": 611321, "postDate": "2019-08-29T08:01:20.777Z", "content": "Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 2 } ], "comments": [ { "id": 611458, "author_name": "Costas Voglis", "author_url": "", "post_date": "2019-08-29T09:17:00.803000", "content": "I was a pleasure working with you guys @titericz, @robikscube, @kingychiu, @hughshaoqz. Rob the way you organized the material and codes for this competition was inspiring! We were a team of diverse skills and realizing that was one of the best things of this competition!! I learned a lot from you. Thank you!!
", "votes": 3, "replies": [ { "id": 612101, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-29T15:51:33.130000", "content": "@voglinio, I'm new on Kaggle, did you guys know each other beforehand or you just partnered up based on skills/current LB/or anything else?
", "votes": 2, "replies": [] }, { "id": 615520, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-09-02T03:33:13.510000", "content": "@voglinio The pleasure was all mine. Glad we got to team. It was fun time learning about new types of models I've never been exposed to before this competition.
\n\n@michelml - Typically you reach out to people close to you on the leaderboard. Once you've taken part in a few competitions you also start to recognize the same people from the message boards and leaderboard and get an idea of people to reach out to about teaming up.
", "votes": 2, "replies": [] } ] }, { "id": 611228, "author_name": "Tania J", "author_url": "", "post_date": "2019-08-29T07:13:22.637000", "content": "Nice work and thanks for sharing. Good idea with the tracking.
", "votes": 4, "replies": [] }, { "id": 616838, "author_name": "Bill Holst", "author_url": "", "post_date": "2019-09-03T14:17:11.867000", "content": "@robikscube - I think your write-up here is a great example on how to approach any competition. Great words of wisdom!
", "votes": 2, "replies": [] }, { "id": 614543, "author_name": "Andy Atkinson", "author_url": "", "post_date": "2019-08-31T17:05:07.587000", "content": "@robikscube How many rounds did you use for LGBM? I never went over 12,000 or so. Not sure if results got any better after that.
", "votes": 2, "replies": [ { "id": 615096, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-09-01T13:01:57.597000", "content": "You can actually see in my logs the number of estimators/boosting rounds used. I used catboost, which doesn't exactly compare to LGBM one-to-one, but many types I trained for up to 1,000,000 rounds.
", "votes": 2, "replies": [] } ] }, { "id": 613105, "author_name": "fnands", "author_url": "", "post_date": "2019-08-30T08:13:50.423000", "content": "Nicely done!
", "votes": 2, "replies": [ { "id": 615515, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-09-02T03:29:14.650000", "content": "Thanks @fnands
", "votes": 1, "replies": [] } ] }, { "id": 613089, "author_name": "Meng Di", "author_url": "", "post_date": "2019-08-30T07:59:38.613000", "content": "Thank you @robikscube and also thanks @titericz @voglinio @kingychiu! I really enjoy working with you. Love the team!
", "votes": 2, "replies": [ { "id": 615517, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-09-02T03:29:38.093000", "content": "I enjoyed teaming! I learned a lot from you and the rest of the team.
", "votes": 1, "replies": [] } ] }, { "id": 612093, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-29T15:47:47.290000", "content": "First github repo share - thanks @robikscube
", "votes": 2, "replies": [] }, { "id": 612044, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-29T15:20:14.173000", "content": "Thanks for sharing and congrats on the end result.
", "votes": 2, "replies": [] }, { "id": 610889, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-08-29T02:37:58.493000", "content": "Great Work...!! @robikscube
", "votes": 1, "replies": [] }, { "id": 612099, "author_name": "", "author_url": "", "post_date": "2019-08-29T15:51:01.027000", "content": "", "votes": 0, "replies": [] }, { "id": 611321, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-08-29T08:01:20.777000", "content": "Congratulations and thanks for sharing!
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "610870": "Big thanks to CHAMPS for hosting this competition. This has been by far my favorite competition that I've taken part in. Mainly because:\n- No leaks\n- No shakeup (solid public/private LB relationship)\n- No simulated data\n- Helps a good cause\n- Some very strong teams with a good mix of kagglers and subject matter experts.\n- It was fun to see the top teams surpass the host's expectations (breaking -3.0)\n\nAs many have already discussed- tabular models could only get you so far in this competition. About three weeks ago, when this became clear, our team decided to focus only on GNN models, Prior to that I had been working exclusively on tabular models. My best personal blend of tabular models ended up getting -2.101 on the private LB (would have been ~100th place and silver medal). This is without blending with any GNN models.\n\n\n\nBig thanks to my teammates @titericz, @voglinio, @kingychiu, and especially @hughshaoqz - who's GNN architecture was the main reason our team ended up as high as we did.\n\nI've made public my github repo with all of the code I used to achieve this here: https://github.com/RobMulla/kaggle-champs-2019/\n\nEven though this competition didn't end up being much about tabular models there were a ton of things I learned from the process:\n- **Ignore public kernels at first - manually create features, add them only if they help CV** - I took this approach from @cpmpml . I purposefully chose to ignore features from public kernels until I completely was drained of ideas. I tried to make most of my features such that they didn't rely on the x, y, z coordinates but only each atom's relationship to each other. I wanted every feature to make sense to me why it would be useful.\n- **Having a solid CV is important** - This wasn't too hard in this competition, I just made sure to use GroupKFold on molecule name to make sure there was no overlap between molecules in train and validation. I found this CV had a very stable CV to LB relationship and blending of OOF predictions worked well.\n- **Attack the problem like 8 mini-competitions.** Early on I decided to split the feature selection, tuning and blending by bond types. This was pretty obvious but did allow me to iterate over specific types, and add features only for types where they helped, and explore parameter tuning per type.\n- **Catboost + GPU is 👍** - I found that compared to LightGBM and XGBoost, catboost worked really well utilizing GPUs and converging quite quickly. I relied on it heavily for my tabular models, mainly because of speed.\n- **Training was really slow.** - Even with training types individually and using catboost with 2 1080tis - training was really slow for this competition. Training all types easily took 72+hours. I tried to develop new features while models were training/validating older features. Initially I was training on only 2 folds and eventually bumped it up to 3 folds.\n- **Track what you've done - so you remember what's worked and what didn't** - I made my code such that every feature engineering step was numbered (FE001 - FE025), and every model was tracked and numbered (M001 - M062). More than anything, this helped me mentally organize different ideas I was iterating over. It also made blending and stacking much easier to track. I also had my code append to a tracking CSV every time I trained a model: saving the CV score, parameters used, iterations taken, and a bunch more stuff. My tracking CSVs can be found in the repo here if you want to see how I slowly improved each type: https://github.com/RobMulla/kaggle-champs-2019/tree/master/tracking\n\n- **Will it blend? Yes it will!** - using OOF predictions I used a pretty crude approach to blending that seemed to work pretty well. This is the notebook I used to create the blend that ended up scoring -2.1 private LB : https://github.com/RobMulla/kaggle-champs-2019/blob/master/notebooks/BLEND-044.ipynb\n\nExcited to read some of the great solutions from the top teams!\n\n......and to all the tabular folks. I'll see you over in IEEE 😄 ", "611458": "I was a pleasure working with you guys @titericz, @robikscube, @kingychiu, @hughshaoqz. Rob the way you organized the material and codes for this competition was inspiring! We were a team of diverse skills and realizing that was one of the best things of this competition!! I learned a lot from you. Thank you!! ", "611228": "Nice work and thanks for sharing. Good idea with the tracking.", "616838": "@robikscube - I think your write-up here is a great example on how to approach any competition. Great words of wisdom!", "614543": "@robikscube How many rounds did you use for LGBM? I never went over 12,000 or so. Not sure if results got any better after that.", "613105": "Nicely done!", "613089": "Thank you @robikscube and also thanks @titericz @voglinio @kingychiu! I really enjoy working with you. Love the team!", "612093": "First github repo share - thanks @robikscube ", "612044": "Thanks for sharing and congrats on the end result.", "610889": "Great Work...!! @robikscube ", "612099": "", "611321": "Congratulations and thanks for sharing!" } }