I have some questions about feature selection.
\n\nThere are lots of ways to select features, like SelectKBest & SelectPercentile & SelectFromModel from sklearn.feature_selection library.
\n\nI want to know are these feature importance that ensemble trees model output reliable.
\n\nAnd what's the difference between features select from sklearn library and ensemble trees.
", "messages": [ { "id": "261803", "postDate": "12/24/2017 02:28:34", "content": "I have some questions about feature selection.
\n\nThere are lots of ways to select features, like SelectKBest & SelectPercentile & SelectFromModel from sklearn.feature_selection library.
\n\nI want to know are these feature importance that ensemble trees model output reliable.
\n\nAnd what's the difference between features select from sklearn library and ensemble trees.
", "rawMarkdown": "I have some questions about feature selection. \n\nThere are lots of ways to select features, like SelectKBest & SelectPercentile & SelectFromModel from sklearn.feature_selection library. \n\nI want to know are these feature importance that ensemble trees model output reliable. \n\nAnd what's the difference between features select from sklearn library and ensemble trees.", "votes": null }, { "id": "262024", "postDate": "12/25/2017 00:30:36", "content": "One of the best ways to perform feature selection (IMO) is to test how your algorithm performs with and without a feature. Is it better if you include the feature? By how much? Make sure when you are doing that kind of analysis that you aren't doing other things that might change the output (eg. hyperparameter tuning). Having more features is also often just better, so feature selection takes a backseat to feature engineering.
\n\nYou can also use a model architecture that is good at discovering features on its own ( eg. RNN) so that you don't have to do a lot of feature engineering/selection.
", "rawMarkdown": "One of the best ways to perform feature selection (IMO) is to test how your algorithm performs with and without a feature. Is it better if you include the feature? By how much? Make sure when you are doing that kind of analysis that you aren't doing other things that might change the output (eg. hyperparameter tuning). Having more features is also often just better, so feature selection takes a backseat to feature engineering. \n\nYou can also use a model architecture that is good at discovering features on its own ( eg. RNN) so that you don't have to do a lot of feature engineering/selection.", "votes": null }, { "id": "262044", "postDate": "12/25/2017 04:01:56", "content": "That was a good opinion...Thanks..
", "rawMarkdown": "That was a good opinion...Thanks..", "votes": null }, { "id": "263685", "postDate": "12/31/2017 11:26:43", "content": "Very interesting question. Add and delete features one-by-one may have neglible effect (less that random changes), so it's may be better to add entire group of features at one moment.
", "rawMarkdown": "Very interesting question. Add and delete features one-by-one may have neglible effect (less that random changes), so it's may be better to add entire group of features at one moment.", "votes": null }, { "id": "264060", "postDate": "01/02/2018 05:55:26", "content": "I use lightgbm.feature_importance('gain')
", "rawMarkdown": "I use lightgbm.feature_importance('gain')", "votes": null }, { "id": "264082", "postDate": "01/02/2018 07:32:18", "content": "Adding to the answer, ensemble trees feature selectors are only as good for feature selection as their performance allows - that is, a model with 60% accuracy might not be ideal for selecting features based on their importance, since it's lack of accuracy could be done because of incorrect feature importance.
\n\nWhile a single model feature importance could be used, it's ideal to have more rounds of adding and eliminating the feature to have a clearer decision of wether to keep it or not, you can do it using Boruta or RFE for example, but you pay the price by having to compute way more.
\n\nFor Boruta, you can find more details in Tili's kernel:\nhttps://www.kaggle.com/tilii7/boruta-feature-elimination
", "rawMarkdown": "Adding to the answer, ensemble trees feature selectors are only as good for feature selection as their performance allows - that is, a model with 60% accuracy might not be ideal for selecting features based on their importance, since it's lack of accuracy could be done because of incorrect feature importance.\n\nWhile a single model feature importance could be used, it's ideal to have more rounds of adding and eliminating the feature to have a clearer decision of wether to keep it or not, you can do it using Boruta or RFE for example, but you pay the price by having to compute way more.\n\nFor Boruta, you can find more details in Tili's kernel:\nhttps://www.kaggle.com/tilii7/boruta-feature-elimination", "votes": null }, { "id": "264134", "postDate": "01/02/2018 11:05:53", "content": "thank you for your reply!
", "rawMarkdown": "thank you for your reply!", "votes": null }, { "id": "264135", "postDate": "01/02/2018 11:07:32", "content": "@To Train Them Is My Cause , What is IMO?
", "rawMarkdown": "To Train Them Is My Cause , What is IMO?", "votes": null }, { "id": "264931", "postDate": "01/04/2018 06:47:27", "content": "it's just the short version of \"In my opinion\" :)
", "rawMarkdown": "it's just the short version of \"In my opinion\" :)", "votes": null }, { "id": "265040", "postDate": "01/04/2018 12:51:01", "content": "thank you for your explanation！
", "rawMarkdown": "thank you for your explanation！", "votes": null }, { "id": "265329", "postDate": "01/05/2018 07:56:29", "content": "you are welcome buddy\nHappy learning!
", "rawMarkdown": "you are welcome buddy\nHappy learning!", "votes": null } ], "comments": [ { "id": 262024, "author_name": "rhgrossm", "author_url": "", "post_date": "12/25/2017 00:30:36", "content": "One of the best ways to perform feature selection (IMO) is to test how your algorithm performs with and without a feature. Is it better if you include the feature? By how much? Make sure when you are doing that kind of analysis that you aren't doing other things that might change the output (eg. hyperparameter tuning). Having more features is also often just better, so feature selection takes a backseat to feature engineering.
\n\nYou can also use a model architecture that is good at discovering features on its own ( eg. RNN) so that you don't have to do a lot of feature engineering/selection.
", "votes": null, "replies": [ { "id": 262044, "author_name": "gsdeepakkumar", "author_url": "", "post_date": "12/25/2017 04:01:56", "content": "That was a good opinion...Thanks..
", "votes": null, "replies": [] }, { "id": 263685, "author_name": "zavodrobotov", "author_url": "", "post_date": "12/31/2017 11:26:43", "content": "Very interesting question. Add and delete features one-by-one may have neglible effect (less that random changes), so it's may be better to add entire group of features at one moment.
", "votes": null, "replies": [] }, { "id": 264082, "author_name": "andrewmvd", "author_url": "", "post_date": "01/02/2018 07:32:18", "content": "Adding to the answer, ensemble trees feature selectors are only as good for feature selection as their performance allows - that is, a model with 60% accuracy might not be ideal for selecting features based on their importance, since it's lack of accuracy could be done because of incorrect feature importance.
\n\nWhile a single model feature importance could be used, it's ideal to have more rounds of adding and eliminating the feature to have a clearer decision of wether to keep it or not, you can do it using Boruta or RFE for example, but you pay the price by having to compute way more.
\n\nFor Boruta, you can find more details in Tili's kernel:\nhttps://www.kaggle.com/tilii7/boruta-feature-elimination
", "votes": null, "replies": [] }, { "id": 264134, "author_name": "shangweichen", "author_url": "", "post_date": "01/02/2018 11:05:53", "content": "thank you for your reply!
", "votes": null, "replies": [] }, { "id": 264135, "author_name": "shangweichen", "author_url": "", "post_date": "01/02/2018 11:07:32", "content": "@To Train Them Is My Cause , What is IMO?
", "votes": null, "replies": [] }, { "id": 264931, "author_name": "andrewmvd", "author_url": "", "post_date": "01/04/2018 06:47:27", "content": "it's just the short version of \"In my opinion\" :)
", "votes": null, "replies": [] }, { "id": 265040, "author_name": "shangweichen", "author_url": "", "post_date": "01/04/2018 12:51:01", "content": "thank you for your explanation！
", "votes": null, "replies": [] }, { "id": 265329, "author_name": "andrewmvd", "author_url": "", "post_date": "01/05/2018 07:56:29", "content": "you are welcome buddy\nHappy learning!
", "votes": null, "replies": [] } ] }, { "id": 264060, "author_name": "xuanhahaha", "author_url": "", "post_date": "01/02/2018 05:55:26", "content": "I use lightgbm.feature_importance('gain')
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "261803": "I have some questions about feature selection. \n\nThere are lots of ways to select features, like SelectKBest & SelectPercentile & SelectFromModel from sklearn.feature_selection library. \n\nI want to know are these feature importance that ensemble trees model output reliable. \n\nAnd what's the difference between features select from sklearn library and ensemble trees.", "262024": "One of the best ways to perform feature selection (IMO) is to test how your algorithm performs with and without a feature. Is it better if you include the feature? By how much? Make sure when you are doing that kind of analysis that you aren't doing other things that might change the output (eg. hyperparameter tuning). Having more features is also often just better, so feature selection takes a backseat to feature engineering. \n\nYou can also use a model architecture that is good at discovering features on its own ( eg. RNN) so that you don't have to do a lot of feature engineering/selection.", "262044": "That was a good opinion...Thanks..", "263685": "Very interesting question. Add and delete features one-by-one may have neglible effect (less that random changes), so it's may be better to add entire group of features at one moment.", "264060": "I use lightgbm.feature_importance('gain')", "264082": "Adding to the answer, ensemble trees feature selectors are only as good for feature selection as their performance allows - that is, a model with 60% accuracy might not be ideal for selecting features based on their importance, since it's lack of accuracy could be done because of incorrect feature importance.\n\nWhile a single model feature importance could be used, it's ideal to have more rounds of adding and eliminating the feature to have a clearer decision of wether to keep it or not, you can do it using Boruta or RFE for example, but you pay the price by having to compute way more.\n\nFor Boruta, you can find more details in Tili's kernel:\nhttps://www.kaggle.com/tilii7/boruta-feature-elimination", "264134": "thank you for your reply!", "264135": "To Train Them Is My Cause , What is IMO?", "264931": "it's just the short version of \"In my opinion\" :)", "265040": "thank you for your explanation！", "265329": "you are welcome buddy\nHappy learning!" }, "source": "meta" }