I have extracted about 130 columns in order to improve my score.\nUnfortunately, I think many features extracted by me are useless or redundant.\nBefore this, I select features based on the model's features importance and a little knowledge of chemistry.
\n\nHow do you select your features?
\n\nThank you for your comments.
", "messages": [ { "id": 584885, "postDate": "2019-07-26T16:10:21.713Z", "content": "Permutation importance seems to work well. \nhttps://www.kaggle.com/speedwagon/permutation-importance
", "rawMarkdown": "Permutation importance seems to work well. \n[https://www.kaggle.com/speedwagon/permutation-importance](https://www.kaggle.com/speedwagon/permutation-importance)", "votes": 6, "replies": [ { "id": 586424, "postDate": "2019-07-29T06:40:07.507Z", "content": "Thank you very much. I have made a try based on permutation importance. This method is helpful for me.
", "rawMarkdown": "Thank you very much. I have made a try based on permutation importance. This method is helpful for me.", "votes": 1 } ] }, { "id": 584814, "postDate": "2019-07-26T14:14:53Z", "content": "I have extracted about 130 columns in order to improve my score.\nUnfortunately, I think many features extracted by me are useless or redundant.\nBefore this, I select features based on the model's features importance and a little knowledge of chemistry.
\n\nHow do you select your features?
\n\nThank you for your comments.
", "rawMarkdown": "I have extracted about 130 columns in order to improve my score.\nUnfortunately, I think many features extracted by me are useless or redundant.\nBefore this, I select features based on the model's features importance and a little knowledge of chemistry. \n\nHow do you select your features?\n\nThank you for your comments.", "votes": 6 }, { "id": 585384, "postDate": "2019-07-27T10:59:35.960Z", "content": "I don't perform feature selection. More precisely, I only add features if they improve my local validation score. I don't create tons of features then struggle to find which ones are useless.
", "rawMarkdown": "I don't perform feature selection. More precisely, I only add features if they improve my local validation score. I don't create tons of features then struggle to find which ones are useless.", "votes": 3, "replies": [ { "id": 588043, "postDate": "2019-07-30T04:19:00.847Z", "content": "Thanks CPMP - I agree this is probably the most robust way of doing things. If you don't mind me asking, how do you iterate through ideas quickly? With 100's of possible feature ideas, I can see compute & time being the bottleneck if each feature is validated individually.
\n\nDo you use a small sample of the data? And/or use a small number of epochs/iteration loops? Single fold instead of full CV (if applicable to the data)?
", "rawMarkdown": "Thanks CPMP - I agree this is probably the most robust way of doing things. If you don't mind me asking, how do you iterate through ideas quickly? With 100's of possible feature ideas, I can see compute & time being the bottleneck if each feature is validated individually.\n\nDo you use a small sample of the data? And/or use a small number of epochs/iteration loops? Single fold instead of full CV (if applicable to the data)?", "votes": 2 }, { "id": 588045, "postDate": "2019-07-30T04:24:54.297Z", "content": "\n\n\nhow do you iterate through ideas quickly?
\n
I use a small number of iterations as I noticed score after these is well correlated with final score.
", "rawMarkdown": "> how do you iterate through ideas quickly? \n\nI use a small number of iterations as I noticed score after these is well correlated with final score.", "votes": 1 }, { "id": 588071, "postDate": "2019-07-30T05:40:20.477Z", "rawMarkdown": "", "isDeleted": true }, { "id": 588085, "postDate": "2019-07-30T06:07:26.433Z", "content": "\n\n\nThe reason why don't create useless score is also to avoid overfitting?
\n
yes
", "rawMarkdown": "> The reason why don't create useless score is also to avoid overfitting?\n\nyes" } ] }, { "id": 585061, "postDate": "2019-07-26T21:31:20.840Z", "content": "I use XGBoost or RandomForest to select features
", "rawMarkdown": "I use XGBoost or RandomForest to select features", "votes": 1 }, { "id": 585059, "postDate": "2019-07-26T21:25:19.517Z", "content": "LOFO is an option:\nhttps://github.com/aerdem4/lofo-importance
", "rawMarkdown": "LOFO is an option:\nhttps://github.com/aerdem4/lofo-importance", "votes": 1, "replies": [ { "id": 585268, "postDate": "2019-07-27T07:31:53.143Z", "content": "Slide 3 here has a useful overview of some of the different feature selection methods:\nhttps://github.com/aerdem4/lofo-importance/blob/master/docs/pydata_feb19_lofo.pdf
\n\nEdit: I just realized it's your slide pack! Thanks for the great work! :)
", "rawMarkdown": "Slide 3 here has a useful overview of some of the different feature selection methods:\nhttps://github.com/aerdem4/lofo-importance/blob/master/docs/pydata_feb19_lofo.pdf\n\nEdit: I just realized it's your slide pack! Thanks for the great work! :)", "votes": 2 }, { "id": 585724, "postDate": "2019-07-28T00:01:52.537Z", "content": "And while trying to figure out my results and was looking at the code at github and I realized that LOFO has you as it's proud contributor. I used it on Andrews Brute force data set that he's made public here. Got some huge numbers of features to drop, different by type and was expecting to be raising a drink in your honor. But my local and LB scores did not change - using model mostly based on Andrews. Two options 1) lightgbm is very robust to bad actors 2) I goofed and my code did not really drop the features.
\n\nOption 2 is the most likely and looking more when I get time - but what's your overall impression of lightgbm's ability to handle features with minus importance from LOFO?
\n\nA side benefit - dropping lots of features greatly improved processing time.
", "rawMarkdown": "And while trying to figure out my results and was looking at the code at github and I realized that LOFO has you as it's proud contributor. I used it on Andrews Brute force data set that he's made public here. Got some huge numbers of features to drop, different by type and was expecting to be raising a drink in your honor. But my local and LB scores did not change - using model mostly based on Andrews. Two options 1) lightgbm is very robust to bad actors 2) I goofed and my code did not really drop the features.\n\nOption 2 is the most likely and looking more when I get time - but what's your overall impression of lightgbm's ability to handle features with minus importance from LOFO?\n\nA side benefit - dropping lots of features greatly improved processing time. ", "votes": 2 }, { "id": 585756, "postDate": "2019-07-28T02:33:59.713Z", "content": "Thanks for sharing your experience with it. If you have totally random split (no tricky validation scheme like time split) and if you have non-overfitting lightgbm parameters, I don't expect you to see any negative feature importance. LOFO is more useful for feature elimination in non-random train-test splits. (see Microsoft Malware Competition Example) In random splits, I use LOFO to give me insights on which features are important so that I should engineer them more.
", "rawMarkdown": "Thanks for sharing your experience with it. If you have totally random split (no tricky validation scheme like time split) and if you have non-overfitting lightgbm parameters, I don't expect you to see any negative feature importance. LOFO is more useful for feature elimination in non-random train-test splits. (see Microsoft Malware Competition Example) In random splits, I use LOFO to give me insights on which features are important so that I should engineer them more.", "votes": 2 }, { "id": 585917, "postDate": "2019-07-28T10:24:17.100Z", "content": "Thanks for the reply - still have a code review on my to do list. I was badly burned in my very first Kaggle challenge by over-fitting - the 4th model I created would have been a gold metal - but the ones I submitted 233 models later were over-fit monsters. So my lightgbm parameters the best I can figure to avoid over fitting. Agree that LOFO should be used to understand the important features and make more like them.
", "rawMarkdown": "Thanks for the reply - still have a code review on my to do list. I was badly burned in my very first Kaggle challenge by over-fitting - the 4th model I created would have been a gold metal - but the ones I submitted 233 models later were over-fit monsters. So my lightgbm parameters the best I can figure to avoid over fitting. Agree that LOFO should be used to understand the important features and make more like them." } ] }, { "id": 585546, "postDate": "2019-07-27T16:00:46.277Z", "content": "Since I just don't have the skill for feature engineering/ selection, I try to let the model build the features itself. Or team with someone who has that skill.
\n\nSee e.g. https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44629
", "rawMarkdown": "Since I just don't have the skill for feature engineering/ selection, I try to let the model build the features itself. Or team with someone who has that skill.\n\nSee e.g. https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44629", "votes": 2, "replies": [ { "id": 585957, "postDate": "2019-07-28T11:54:28.270Z", "content": "And you have a bunch of feature engineering experts in your team ;)
", "rawMarkdown": "And you have a bunch of feature engineering experts in your team ;)\n", "votes": 1 } ] }, { "id": 588350, "postDate": "2019-07-30T12:56:09.250Z", "content": "There are lot of methods to find appropriate features for your model. Techniques such as RFE, Chi^2, Correlations, PCA etc can be used to pick features.
", "rawMarkdown": "There are lot of methods to find appropriate features for your model. Techniques such as RFE, Chi^2, Correlations, PCA etc can be used to pick features." }, { "id": 588202, "postDate": "2019-07-30T09:35:32.213Z", "content": "Check out the kernel I just made! It uses split importance because permutation importance is too much CPU for the GPU kernel.\nhttps://www.kaggle.com/nicapotato/tree-split-feature-selection-lgbm-gpu-earlystop
", "rawMarkdown": "Check out the kernel I just made! It uses split importance because permutation importance is too much CPU for the GPU kernel.\nhttps://www.kaggle.com/nicapotato/tree-split-feature-selection-lgbm-gpu-earlystop" }, { "id": 585113, "postDate": "2019-07-27T01:36:17.120Z", "content": "Like Shuhao has said, feature importance using either split or gain from LightGBM is not robust enough to reflect their true importance. You can try using model.predict(X, pred_contrib=True) to directly output the SHAP value and use that to select your feature. SHAP package also has nice built-in plotting function. \nhttps://github.com/slundberg/shap
", "rawMarkdown": "Like Shuhao has said, feature importance using either split or gain from LightGBM is not robust enough to reflect their true importance. You can try using model.predict(X, pred_contrib=True) to directly output the SHAP value and use that to select your feature. SHAP package also has nice built-in plotting function. \nhttps://github.com/slundberg/shap" }, { "id": 585056, "postDate": "2019-07-26T21:20:59.853Z", "content": "You can try and use LightGBM feature importance capabilities (you can choose few different techniques that are done in the backstage like Information gain or others, you can take a look in the documentation here - https://lightgbm.readthedocs.io/en/latest/Python-API.html#lightgbm.Booster.feature_importance).
\n\nThere is a git that I found a few months back that is doing most of the stuff for you. I tweaked it and improved it a bit (if you want the tweaked code you can contact me) - Github link - https://github.com/WillKoehrsen/feature-selector\nBest of luck!
", "rawMarkdown": "You can try and use LightGBM feature importance capabilities (you can choose few different techniques that are done in the backstage like Information gain or others, you can take a look in the documentation here - https://lightgbm.readthedocs.io/en/latest/Python-API.html#lightgbm.Booster.feature_importance).\n\nThere is a git that I found a few months back that is doing most of the stuff for you. I tweaked it and improved it a bit (if you want the tweaked code you can contact me) - Github link - https://github.com/WillKoehrsen/feature-selector\nBest of luck!", "replies": [ { "id": 585075, "postDate": "2019-07-26T22:24:51.407Z", "content": "For LGB, feature importance is measuring how important a feature is in building/splitting tree/leaves in order to minimize the objective function, in most case, a good feature is of high up in the importance chart. However, when your objective function is different from the actual metric, this may give you some false information, especially when the training error become <1, as bagging using a feature may sometimes be good in terms of minimizing the overall L^2 error (which I assume everyone is training using a smooth objective function) while not minimizing the overall L^1 error.
", "rawMarkdown": "For LGB, feature importance is measuring how important a feature is in building/splitting tree/leaves in order to minimize the objective function, in most case, a good feature is of high up in the importance chart. However, when your objective function is different from the actual metric, this may give you some false information, especially when the training error become <1, as bagging using a feature may sometimes be good in terms of minimizing the overall L^2 error (which I assume everyone is training using a smooth objective function) while not minimizing the overall L^1 error." }, { "id": 585265, "postDate": "2019-07-27T07:28:20.867Z", "content": "Of course you need to define the goal function and metric in order for you to be able to perform a good feature selection. \nBTW, the LGB default setting is to seek the split in some sort of tree as you say. But in the docs you can see that there are also other forms of metrics such as gain (I know that sometimes it might have the same effect though)
", "rawMarkdown": "Of course you need to define the goal function and metric in order for you to be able to perform a good feature selection. \nBTW, the LGB default setting is to seek the split in some sort of tree as you say. But in the docs you can see that there are also other forms of metrics such as gain (I know that sometimes it might have the same effect though)\n" } ] }, { "id": 584852, "postDate": "2019-07-26T15:04:46.097Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 585939, "postDate": "2019-07-28T11:20:50.810Z", "content": "Thanks for sharing:)
", "rawMarkdown": "Thanks for sharing:)" } ], "comments": [ { "id": 584885, "author_name": "Tania J", "author_url": "", "post_date": "2019-07-26T16:10:21.713000", "content": "Permutation importance seems to work well. \nhttps://www.kaggle.com/speedwagon/permutation-importance
", "votes": 6, "replies": [ { "id": 586424, "author_name": "little_snail", "author_url": "", "post_date": "2019-07-29T06:40:07.507000", "content": "Thank you very much. I have made a try based on permutation importance. This method is helpful for me.
", "votes": 1, "replies": [] } ] }, { "id": 585384, "author_name": "CPMP", "author_url": "", "post_date": "2019-07-27T10:59:35.960000", "content": "I don't perform feature selection. More precisely, I only add features if they improve my local validation score. I don't create tons of features then struggle to find which ones are useless.
", "votes": 3, "replies": [ { "id": 588043, "author_name": "datasaurus", "author_url": "", "post_date": "2019-07-30T04:19:00.847000", "content": "Thanks CPMP - I agree this is probably the most robust way of doing things. If you don't mind me asking, how do you iterate through ideas quickly? With 100's of possible feature ideas, I can see compute & time being the bottleneck if each feature is validated individually.
\n\nDo you use a small sample of the data? And/or use a small number of epochs/iteration loops? Single fold instead of full CV (if applicable to the data)?
", "votes": 2, "replies": [] }, { "id": 588045, "author_name": "CPMP", "author_url": "", "post_date": "2019-07-30T04:24:54.297000", "content": "\n\n\nhow do you iterate through ideas quickly?
\n
I use a small number of iterations as I noticed score after these is well correlated with final score.
", "votes": 1, "replies": [] }, { "id": 588071, "author_name": "", "author_url": "", "post_date": "2019-07-30T05:40:20.477000", "content": "", "votes": 0, "replies": [] }, { "id": 588085, "author_name": "CPMP", "author_url": "", "post_date": "2019-07-30T06:07:26.433000", "content": "\n\n\nThe reason why don't create useless score is also to avoid overfitting?
\n
yes
", "votes": 0, "replies": [] } ] }, { "id": 585061, "author_name": "Linda MacPhee-Cobb", "author_url": "", "post_date": "2019-07-26T21:31:20.840000", "content": "I use XGBoost or RandomForest to select features
", "votes": 1, "replies": [] }, { "id": 585059, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-07-26T21:25:19.517000", "content": "LOFO is an option:\nhttps://github.com/aerdem4/lofo-importance
", "votes": 1, "replies": [ { "id": 585268, "author_name": "datasaurus", "author_url": "", "post_date": "2019-07-27T07:31:53.143000", "content": "Slide 3 here has a useful overview of some of the different feature selection methods:\nhttps://github.com/aerdem4/lofo-importance/blob/master/docs/pydata_feb19_lofo.pdf
\n\nEdit: I just realized it's your slide pack! Thanks for the great work! :)
", "votes": 2, "replies": [] }, { "id": 585724, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-07-28T00:01:52.537000", "content": "And while trying to figure out my results and was looking at the code at github and I realized that LOFO has you as it's proud contributor. I used it on Andrews Brute force data set that he's made public here. Got some huge numbers of features to drop, different by type and was expecting to be raising a drink in your honor. But my local and LB scores did not change - using model mostly based on Andrews. Two options 1) lightgbm is very robust to bad actors 2) I goofed and my code did not really drop the features.
\n\nOption 2 is the most likely and looking more when I get time - but what's your overall impression of lightgbm's ability to handle features with minus importance from LOFO?
\n\nA side benefit - dropping lots of features greatly improved processing time.
", "votes": 2, "replies": [] }, { "id": 585756, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-07-28T02:33:59.713000", "content": "Thanks for sharing your experience with it. If you have totally random split (no tricky validation scheme like time split) and if you have non-overfitting lightgbm parameters, I don't expect you to see any negative feature importance. LOFO is more useful for feature elimination in non-random train-test splits. (see Microsoft Malware Competition Example) In random splits, I use LOFO to give me insights on which features are important so that I should engineer them more.
", "votes": 2, "replies": [] }, { "id": 585917, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-07-28T10:24:17.100000", "content": "Thanks for the reply - still have a code review on my to do list. I was badly burned in my very first Kaggle challenge by over-fitting - the 4th model I created would have been a gold metal - but the ones I submitted 233 models later were over-fit monsters. So my lightgbm parameters the best I can figure to avoid over fitting. Agree that LOFO should be used to understand the important features and make more like them.
", "votes": 0, "replies": [] } ] }, { "id": 585546, "author_name": "Dieter", "author_url": "", "post_date": "2019-07-27T16:00:46.277000", "content": "Since I just don't have the skill for feature engineering/ selection, I try to let the model build the features itself. Or team with someone who has that skill.
\n\nSee e.g. https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44629
", "votes": 2, "replies": [ { "id": 585957, "author_name": "Serigne ", "author_url": "", "post_date": "2019-07-28T11:54:28.270000", "content": "And you have a bunch of feature engineering experts in your team ;)
", "votes": 1, "replies": [] } ] }, { "id": 588350, "author_name": "Krishna S", "author_url": "", "post_date": "2019-07-30T12:56:09.250000", "content": "There are lot of methods to find appropriate features for your model. Techniques such as RFE, Chi^2, Correlations, PCA etc can be used to pick features.
", "votes": 0, "replies": [] }, { "id": 588202, "author_name": "nicapotato", "author_url": "", "post_date": "2019-07-30T09:35:32.213000", "content": "Check out the kernel I just made! It uses split importance because permutation importance is too much CPU for the GPU kernel.\nhttps://www.kaggle.com/nicapotato/tree-split-feature-selection-lgbm-gpu-earlystop
", "votes": 0, "replies": [] }, { "id": 585113, "author_name": "Ziteng Pang", "author_url": "", "post_date": "2019-07-27T01:36:17.120000", "content": "Like Shuhao has said, feature importance using either split or gain from LightGBM is not robust enough to reflect their true importance. You can try using model.predict(X, pred_contrib=True) to directly output the SHAP value and use that to select your feature. SHAP package also has nice built-in plotting function. \nhttps://github.com/slundberg/shap
", "votes": 0, "replies": [] }, { "id": 585056, "author_name": "Assaf Cohen", "author_url": "", "post_date": "2019-07-26T21:20:59.853000", "content": "You can try and use LightGBM feature importance capabilities (you can choose few different techniques that are done in the backstage like Information gain or others, you can take a look in the documentation here - https://lightgbm.readthedocs.io/en/latest/Python-API.html#lightgbm.Booster.feature_importance).
\n\nThere is a git that I found a few months back that is doing most of the stuff for you. I tweaked it and improved it a bit (if you want the tweaked code you can contact me) - Github link - https://github.com/WillKoehrsen/feature-selector\nBest of luck!
", "votes": 0, "replies": [ { "id": 585075, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2019-07-26T22:24:51.407000", "content": "For LGB, feature importance is measuring how important a feature is in building/splitting tree/leaves in order to minimize the objective function, in most case, a good feature is of high up in the importance chart. However, when your objective function is different from the actual metric, this may give you some false information, especially when the training error become <1, as bagging using a feature may sometimes be good in terms of minimizing the overall L^2 error (which I assume everyone is training using a smooth objective function) while not minimizing the overall L^1 error.
", "votes": 0, "replies": [] }, { "id": 585265, "author_name": "Assaf Cohen", "author_url": "", "post_date": "2019-07-27T07:28:20.867000", "content": "Of course you need to define the goal function and metric in order for you to be able to perform a good feature selection. \nBTW, the LGB default setting is to seek the split in some sort of tree as you say. But in the docs you can see that there are also other forms of metrics such as gain (I know that sometimes it might have the same effect though)
", "votes": 0, "replies": [] } ] }, { "id": 584852, "author_name": "", "author_url": "", "post_date": "2019-07-26T15:04:46.097000", "content": "", "votes": 1, "replies": [] }, { "id": 585939, "author_name": "Himanshu Padhiar", "author_url": "", "post_date": "2019-07-28T11:20:50.810000", "content": "Thanks for sharing:)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "584885": "Permutation importance seems to work well. \n[https://www.kaggle.com/speedwagon/permutation-importance](https://www.kaggle.com/speedwagon/permutation-importance)", "584814": "I have extracted about 130 columns in order to improve my score.\nUnfortunately, I think many features extracted by me are useless or redundant.\nBefore this, I select features based on the model's features importance and a little knowledge of chemistry. \n\nHow do you select your features?\n\nThank you for your comments.", "585384": "I don't perform feature selection. More precisely, I only add features if they improve my local validation score. I don't create tons of features then struggle to find which ones are useless.", "585061": "I use XGBoost or RandomForest to select features", "585059": "LOFO is an option:\nhttps://github.com/aerdem4/lofo-importance", "585546": "Since I just don't have the skill for feature engineering/ selection, I try to let the model build the features itself. Or team with someone who has that skill.\n\nSee e.g. https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44629", "588350": "There are lot of methods to find appropriate features for your model. Techniques such as RFE, Chi^2, Correlations, PCA etc can be used to pick features.", "588202": "Check out the kernel I just made! It uses split importance because permutation importance is too much CPU for the GPU kernel.\nhttps://www.kaggle.com/nicapotato/tree-split-feature-selection-lgbm-gpu-earlystop", "585113": "Like Shuhao has said, feature importance using either split or gain from LightGBM is not robust enough to reflect their true importance. You can try using model.predict(X, pred_contrib=True) to directly output the SHAP value and use that to select your feature. SHAP package also has nice built-in plotting function. \nhttps://github.com/slundberg/shap", "585056": "You can try and use LightGBM feature importance capabilities (you can choose few different techniques that are done in the backstage like Information gain or others, you can take a look in the documentation here - https://lightgbm.readthedocs.io/en/latest/Python-API.html#lightgbm.Booster.feature_importance).\n\nThere is a git that I found a few months back that is doing most of the stuff for you. I tweaked it and improved it a bit (if you want the tweaked code you can contact me) - Github link - https://github.com/WillKoehrsen/feature-selector\nBest of luck!", "584852": "", "585939": "Thanks for sharing:)" } }