When I add features to my model it improves the CV score of some of the coupling types, but reduces the CV score of other coupling types. So far I've only noticed minor performance differences across the types, but that's with making small incremental changes to my feature set.
\n\nI want to start a discussion about how important it is to select a different set of features for each coupling type. Also, it would be neat to discuss if this translates to a better leaderboard score as well. (I would think whatever CV improvement can be gained would favorably impact the LB score in this competition)
", "messages": [ { "id": "601376", "postDate": "08/17/2019 14:18:11", "content": "When I add features to my model it improves the CV score of some of the coupling types, but reduces the CV score of other coupling types. So far I've only noticed minor performance differences across the types, but that's with making small incremental changes to my feature set.
\n\nI want to start a discussion about how important it is to select a different set of features for each coupling type. Also, it would be neat to discuss if this translates to a better leaderboard score as well. (I would think whatever CV improvement can be gained would favorably impact the LB score in this competition)
", "rawMarkdown": "When I add features to my model it improves the CV score of some of the coupling types, but reduces the CV score of other coupling types. So far I've only noticed minor performance differences across the types, but that's with making small incremental changes to my feature set.\n\nI want to start a discussion about how important it is to select a different set of features for each coupling type. Also, it would be neat to discuss if this translates to a better leaderboard score as well. (I would think whatever CV improvement can be gained would favorably impact the LB score in this competition)", "votes": null }, { "id": "601440", "postDate": "08/17/2019 16:02:35", "content": "I combined a few datasets that I've collected throughout the competition, including one large sparse matrix with atom distances and element binaries. This comes to 639 features. I haven't actually been able to successfully load this into memory for all the coupling types. This along with the fact I'm training separate models for each coupling type means that if I want to use lightgbm to select features it is practical to select different features for each type.
\n\nFeature selection hasn't yielded large CV gains for me. Going from 639 features to 100 features using lightgbm important features, building separate models for each type to gauge importance, has yielded a mediocre 0.03 CV improvement. The improvement was pretty consistent for each coupling type. Note, I was using 100 rounds for these tests. It seems to me that lightgbm does pretty well weeding out the noise of meaningless features when you have a decent sample size. I know a more meticulous approach of feature selection is warranted, but I used lightgbm feature importance because I know I had a lot of unimportant features with the feature building approach I took. I feel that my time is likely better spent feature engineering, discovering the true important features using careful consideration.
\n\nHowever, where feature selection lacked in improving my CV it definitely makes up for in runtime :)
", "rawMarkdown": "I combined a few datasets that I've collected throughout the competition, including one large sparse matrix with atom distances and element binaries. This comes to 639 features. I haven't actually been able to successfully load this into memory for all the coupling types. This along with the fact I'm training separate models for each coupling type means that if I want to use lightgbm to select features it is practical to select different features for each type.\n\nFeature selection hasn't yielded large CV gains for me. Going from 639 features to 100 features using lightgbm important features, building separate models for each type to gauge importance, has yielded a mediocre 0.03 CV improvement. The improvement was pretty consistent for each coupling type. Note, I was using 100 rounds for these tests. It seems to me that lightgbm does pretty well weeding out the noise of meaningless features when you have a decent sample size. I know a more meticulous approach of feature selection is warranted, but I used lightgbm feature importance because I know I had a lot of unimportant features with the feature building approach I took. I feel that my time is likely better spent feature engineering, discovering the true important features using careful consideration.\n\nHowever, where feature selection lacked in improving my CV it definitely makes up for in runtime :)", "votes": null }, { "id": "603849", "postDate": "08/20/2019 18:09:28", "content": "Hi, Andy,\n I am doing very similar thing as you are doing - using lighGBM to select feature from a 1000+ feature pool. For me, it yields significant improvement. There are couple of things that we may be doing differently: I am \n1. using \"gain\" as importance_type rather than the default \"split\"\n2. doing a recursive reduction 1000+ features -> 200 -> 100 -> 80 > 75 -> ...\n3. achieving best validation score at around 50 features
\n\nHope this info is of some help.
", "rawMarkdown": "Hi, Andy,\n I am doing very similar thing as you are doing - using lighGBM to select feature from a 1000+ feature pool. For me, it yields significant improvement. There are couple of things that we may be doing differently: I am \n1. using \"gain\" as importance_type rather than the default \"split\"\n2. doing a recursive reduction 1000+ features -> 200 -> 100 -> 80 > 75 -> ...\n3. achieving best validation score at around 50 features\n\nHope this info is of some help.", "votes": null }, { "id": "605040", "postDate": "08/22/2019 02:07:07", "content": "That's a great idea. I'll try the recursive reduction out. Also, I'll try boiling them down to a smaller selection too. Thanks!
", "rawMarkdown": "That's a great idea. I'll try the recursive reduction out. Also, I'll try boiling them down to a smaller selection too. Thanks!", "votes": null } ], "comments": [ { "id": 601440, "author_name": "andyatkinson", "author_url": "", "post_date": "08/17/2019 16:02:35", "content": "I combined a few datasets that I've collected throughout the competition, including one large sparse matrix with atom distances and element binaries. This comes to 639 features. I haven't actually been able to successfully load this into memory for all the coupling types. This along with the fact I'm training separate models for each coupling type means that if I want to use lightgbm to select features it is practical to select different features for each type.
\n\nFeature selection hasn't yielded large CV gains for me. Going from 639 features to 100 features using lightgbm important features, building separate models for each type to gauge importance, has yielded a mediocre 0.03 CV improvement. The improvement was pretty consistent for each coupling type. Note, I was using 100 rounds for these tests. It seems to me that lightgbm does pretty well weeding out the noise of meaningless features when you have a decent sample size. I know a more meticulous approach of feature selection is warranted, but I used lightgbm feature importance because I know I had a lot of unimportant features with the feature building approach I took. I feel that my time is likely better spent feature engineering, discovering the true important features using careful consideration.
\n\nHowever, where feature selection lacked in improving my CV it definitely makes up for in runtime :)
", "votes": null, "replies": [ { "id": 603849, "author_name": "barnwellguy", "author_url": "", "post_date": "08/20/2019 18:09:28", "content": "Hi, Andy,\n I am doing very similar thing as you are doing - using lighGBM to select feature from a 1000+ feature pool. For me, it yields significant improvement. There are couple of things that we may be doing differently: I am \n1. using \"gain\" as importance_type rather than the default \"split\"\n2. doing a recursive reduction 1000+ features -> 200 -> 100 -> 80 > 75 -> ...\n3. achieving best validation score at around 50 features
\n\nHope this info is of some help.
", "votes": null, "replies": [] }, { "id": 605040, "author_name": "andyatkinson", "author_url": "", "post_date": "08/22/2019 02:07:07", "content": "That's a great idea. I'll try the recursive reduction out. Also, I'll try boiling them down to a smaller selection too. Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "601376": "When I add features to my model it improves the CV score of some of the coupling types, but reduces the CV score of other coupling types. So far I've only noticed minor performance differences across the types, but that's with making small incremental changes to my feature set.\n\nI want to start a discussion about how important it is to select a different set of features for each coupling type. Also, it would be neat to discuss if this translates to a better leaderboard score as well. (I would think whatever CV improvement can be gained would favorably impact the LB score in this competition)", "601440": "I combined a few datasets that I've collected throughout the competition, including one large sparse matrix with atom distances and element binaries. This comes to 639 features. I haven't actually been able to successfully load this into memory for all the coupling types. This along with the fact I'm training separate models for each coupling type means that if I want to use lightgbm to select features it is practical to select different features for each type.\n\nFeature selection hasn't yielded large CV gains for me. Going from 639 features to 100 features using lightgbm important features, building separate models for each type to gauge importance, has yielded a mediocre 0.03 CV improvement. The improvement was pretty consistent for each coupling type. Note, I was using 100 rounds for these tests. It seems to me that lightgbm does pretty well weeding out the noise of meaningless features when you have a decent sample size. I know a more meticulous approach of feature selection is warranted, but I used lightgbm feature importance because I know I had a lot of unimportant features with the feature building approach I took. I feel that my time is likely better spent feature engineering, discovering the true important features using careful consideration.\n\nHowever, where feature selection lacked in improving my CV it definitely makes up for in runtime :)", "603849": "Hi, Andy,\n I am doing very similar thing as you are doing - using lighGBM to select feature from a 1000+ feature pool. For me, it yields significant improvement. There are couple of things that we may be doing differently: I am \n1. using \"gain\" as importance_type rather than the default \"split\"\n2. doing a recursive reduction 1000+ features -> 200 -> 100 -> 80 > 75 -> ...\n3. achieving best validation score at around 50 features\n\nHope this info is of some help.", "605040": "That's a great idea. I'll try the recursive reduction out. Also, I'll try boiling them down to a smaller selection too. Thanks!" }, "source": "meta" }