![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2689207%2F7af99ef7b86015faae42eca6866673ac%2Fforce_resultant.png?generation=1567102497246402&alt=media\")
Hi guys, I created two kernels explaining how we used LGB and NN models to get a silver medal in this competition. I know it is not as cool as all the graph NNs, but might be nice for someone :)
\n\n1) Features for top 5% LB with NN or LGB show how we calculated the features we used. The main one that wasn't shared publicly is calculating forces resultants on the axis of atom0 to atom1. Features based on this principle were the most important to my model. Here is one:
\n\n
2) NN and LGB tricks and pipeline for top 5% LB describes the pipeline used. There were a few tricks that greatly boosted our score, like using permutation importance to select features and setting distance for LGB as yukawa distances (exp(-dist)/dist).
\n\nWe also added some features extracted from a NN layer as features to the LGB, and they look awesome. It always impress me how beautiful neural networks are. Here is an example:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2689207%2F75927fece7206a6c97c0279d8e938dcb%2Ffeat_nn_6.png?generation=1567102931116829&alt=media\")
It took a while to upload everything and detail the code, so, if you enjoy reading it somehow, consider upvoting as a reward for my time :)
\n\nHope you guys enjoy it. Happy Kaggling.
", "messages": [ { "id": "612342", "postDate": "08/29/2019 18:22:57", "content": "Hi guys, I created two kernels explaining how we used LGB and NN models to get a silver medal in this competition. I know it is not as cool as all the graph NNs, but might be nice for someone :)
\n\n1) Features for top 5% LB with NN or LGB show how we calculated the features we used. The main one that wasn't shared publicly is calculating forces resultants on the axis of atom0 to atom1. Features based on this principle were the most important to my model. Here is one:
\n\n
2) NN and LGB tricks and pipeline for top 5% LB describes the pipeline used. There were a few tricks that greatly boosted our score, like using permutation importance to select features and setting distance for LGB as yukawa distances (exp(-dist)/dist).
\n\nWe also added some features extracted from a NN layer as features to the LGB, and they look awesome. It always impress me how beautiful neural networks are. Here is an example:
\n\n
It took a while to upload everything and detail the code, so, if you enjoy reading it somehow, consider upvoting as a reward for my time :)
\n\nHope you guys enjoy it. Happy Kaggling.
", "rawMarkdown": "Hi guys, I created two kernels explaining how we used LGB and NN models to get a silver medal in this competition. I know it is not as cool as all the graph NNs, but might be nice for someone :)\n\n1) [Features for top 5% LB with NN or LGB](https://www.kaggle.com/felipemello/features-for-top-5-lb-with-nn-or-lgb) show how we calculated the features we used. The main one that wasn't shared publicly is calculating forces resultants on the axis of atom0 to atom1. Features based on this principle were the most important to my model. Here is one:\n\n\n\n2) [NN and LGB tricks and pipeline for top 5% LB ](https://www.kaggle.com/felipemello/nn-and-lgb-tricks-and-pipeline-for-top-5-lb) describes the pipeline used. There were a few tricks that greatly boosted our score, like using [permutation importance](https://www.kaggle.com/speedwagon/permutation-importance) to select features and setting distance for LGB as yukawa distances (exp(-dist)/dist).\n\nWe also added some features extracted from a NN layer as features to the LGB, and they look awesome. It always impress me how beautiful neural networks are. Here is an example:\n\n\n\nIt took a while to upload everything and detail the code, so, if you enjoy reading it somehow, consider upvoting as a reward for my time :)\n\nHope you guys enjoy it. Happy Kaggling.", "votes": null }, { "id": "612348", "postDate": "08/29/2019 18:29:42", "content": "Green zone only!
", "rawMarkdown": "Green zone only!", "votes": null }, { "id": "612629", "postDate": "08/29/2019 23:00:52", "content": "It was a great pleasure, guys! Very nice time.\nWish you all the best and hope to collaborate, discuss, share great time and fun in the future! Good luck.
", "rawMarkdown": "It was a great pleasure, guys! Very nice time.\nWish you all the best and hope to collaborate, discuss, share great time and fun in the future! Good luck.", "votes": null }, { "id": "612691", "postDate": "08/30/2019 00:18:25", "content": "Thanks @felipemello , this is the first time I see feature extraction from an NN to transfer to another model - written in code. This is nice. I don't know if this is supposed to be basic but 'll keep that trick in mind for sure.
", "rawMarkdown": "Thanks @felipemello , this is the first time I see feature extraction from an NN to transfer to another model - written in code. This is nice. I don't know if this is supposed to be basic but 'll keep that trick in mind for sure.", "votes": null }, { "id": "612695", "postDate": "08/30/2019 00:20:28", "content": "Furthermore, I learned about permutation importance. Double thanks.
", "rawMarkdown": "Furthermore, I learned about permutation importance. Double thanks.", "votes": null }, { "id": "612704", "postDate": "08/30/2019 00:29:49", "content": "Glad you liked it. To be honest, it was also my first time with permutation importance and feature extraction. Every competition is a great opportunity to learn new great stuff :)
", "rawMarkdown": "Glad you liked it. To be honest, it was also my first time with permutation importance and feature extraction. Every competition is a great opportunity to learn new great stuff :)", "votes": null }, { "id": "612902", "postDate": "08/30/2019 04:39:24", "content": "Great Work....\nVery Informative...\nThanks for Sharing....!! @felipemello
", "rawMarkdown": "Great Work....\nVery Informative...\nThanks for Sharing....!! @felipemello", "votes": null }, { "id": "614453", "postDate": "08/31/2019 14:08:46", "content": "Thanks, @veeralakrishna
", "rawMarkdown": "Thanks, @veeralakrishna", "votes": null } ], "comments": [ { "id": 612348, "author_name": "stanislavblinov", "author_url": "", "post_date": "08/29/2019 18:29:42", "content": "Green zone only!
", "votes": null, "replies": [] }, { "id": 612629, "author_name": "muhakabartay", "author_url": "", "post_date": "08/29/2019 23:00:52", "content": "It was a great pleasure, guys! Very nice time.\nWish you all the best and hope to collaborate, discuss, share great time and fun in the future! Good luck.
", "votes": null, "replies": [] }, { "id": 612691, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 00:18:25", "content": "Thanks @felipemello , this is the first time I see feature extraction from an NN to transfer to another model - written in code. This is nice. I don't know if this is supposed to be basic but 'll keep that trick in mind for sure.
", "votes": null, "replies": [ { "id": 612695, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 00:20:28", "content": "Furthermore, I learned about permutation importance. Double thanks.
", "votes": null, "replies": [] }, { "id": 612704, "author_name": "felipemello", "author_url": "", "post_date": "08/30/2019 00:29:49", "content": "Glad you liked it. To be honest, it was also my first time with permutation importance and feature extraction. Every competition is a great opportunity to learn new great stuff :)
", "votes": null, "replies": [] } ] }, { "id": 612902, "author_name": "veeralakrishna", "author_url": "", "post_date": "08/30/2019 04:39:24", "content": "Great Work....\nVery Informative...\nThanks for Sharing....!! @felipemello
", "votes": null, "replies": [ { "id": 614453, "author_name": "felipemello", "author_url": "", "post_date": "08/31/2019 14:08:46", "content": "Thanks, @veeralakrishna
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "612342": "Hi guys, I created two kernels explaining how we used LGB and NN models to get a silver medal in this competition. I know it is not as cool as all the graph NNs, but might be nice for someone :)\n\n1) [Features for top 5% LB with NN or LGB](https://www.kaggle.com/felipemello/features-for-top-5-lb-with-nn-or-lgb) show how we calculated the features we used. The main one that wasn't shared publicly is calculating forces resultants on the axis of atom0 to atom1. Features based on this principle were the most important to my model. Here is one:\n\n\n\n2) [NN and LGB tricks and pipeline for top 5% LB ](https://www.kaggle.com/felipemello/nn-and-lgb-tricks-and-pipeline-for-top-5-lb) describes the pipeline used. There were a few tricks that greatly boosted our score, like using [permutation importance](https://www.kaggle.com/speedwagon/permutation-importance) to select features and setting distance for LGB as yukawa distances (exp(-dist)/dist).\n\nWe also added some features extracted from a NN layer as features to the LGB, and they look awesome. It always impress me how beautiful neural networks are. Here is an example:\n\n\n\nIt took a while to upload everything and detail the code, so, if you enjoy reading it somehow, consider upvoting as a reward for my time :)\n\nHope you guys enjoy it. Happy Kaggling.", "612348": "Green zone only!", "612629": "It was a great pleasure, guys! Very nice time.\nWish you all the best and hope to collaborate, discuss, share great time and fun in the future! Good luck.", "612691": "Thanks @felipemello , this is the first time I see feature extraction from an NN to transfer to another model - written in code. This is nice. I don't know if this is supposed to be basic but 'll keep that trick in mind for sure.", "612695": "Furthermore, I learned about permutation importance. Double thanks.", "612704": "Glad you liked it. To be honest, it was also my first time with permutation importance and feature extraction. Every competition is a great opportunity to learn new great stuff :)", "612902": "Great Work....\nVery Informative...\nThanks for Sharing....!! @felipemello", "614453": "Thanks, @veeralakrishna" }, "source": "meta" }