Feature engineering is sometimes referred to as an art of intuition and I agree. However, one always need to measure the usefulness of a feature when interacting with the others i.e. measure of feature interactions. I have read people talking about 2000 features in this competition when I had only about 260 features. Have seen that in the past too during \"Home Credit Default Risk\" competition where people mentioned over 2000 and I had only 750 features.
\n\nIn the past I add features sequentially, based on CV decide what to keep or discard. This is not always practical especially when there are hundreds of features. In this competition I dug more into feature selection techniques. So here is my take:-
\n\nFE using boruta did worse than my manual technique.
Using scipy's ks_2samp mentioned by Manoj aka @mks2192 (thank you) here https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537 . This worked to a certain degree but not enough going by my CV improvement. I still had to do quite a bit of manual add and remove.
FE with null hypothesis. I missed this one during the Home credit competition posted by Olivier aka @ogrellier (thank you) here. His kernel is based on the article \"Permutation importance: a corrected feature importance measure\". I still had to add some features manually but a lot easier.
For numerical features one can also use SVD. I have used it in the past but not in this competition.
My final selection out of 260 features was only 48. With that, I jumped from public LB #311 to #239 in the last couple of days.
\n\nHow many features did you choose for your final models?
\n\nI observed that the features - feature_1 , feature_2 , and feature_3 gets removed by all the methods mentioned but adding them back improved both my CV and LB. My final selected model results are :-
\n\nLocal CV=3.642223, public LB=3.679 private LB = 3.611, and
public LB=3.678, private LB=3.609 - This one is the result of a simple averaging.
Please share your thoughts and techniques below.
", "messages": [ { "id": "479169", "postDate": "02/27/2019 02:01:56", "content": "Feature engineering is sometimes referred to as an art of intuition and I agree. However, one always need to measure the usefulness of a feature when interacting with the others i.e. measure of feature interactions. I have read people talking about 2000 features in this competition when I had only about 260 features. Have seen that in the past too during \"Home Credit Default Risk\" competition where people mentioned over 2000 and I had only 750 features.
\n\nIn the past I add features sequentially, based on CV decide what to keep or discard. This is not always practical especially when there are hundreds of features. In this competition I dug more into feature selection techniques. So here is my take:-
\n\nFE using boruta did worse than my manual technique.
Using scipy's ks_2samp mentioned by Manoj aka @mks2192 (thank you) here https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537 . This worked to a certain degree but not enough going by my CV improvement. I still had to do quite a bit of manual add and remove.
FE with null hypothesis. I missed this one during the Home credit competition posted by Olivier aka @ogrellier (thank you) here. His kernel is based on the article \"Permutation importance: a corrected feature importance measure\". I still had to add some features manually but a lot easier.
For numerical features one can also use SVD. I have used it in the past but not in this competition.
My final selection out of 260 features was only 48. With that, I jumped from public LB #311 to #239 in the last couple of days.
\n\nHow many features did you choose for your final models?
\n\nI observed that the features - feature_1 , feature_2 , and feature_3 gets removed by all the methods mentioned but adding them back improved both my CV and LB. My final selected model results are :-
\n\nLocal CV=3.642223, public LB=3.679 private LB = 3.611, and
public LB=3.678, private LB=3.609 - This one is the result of a simple averaging.
Please share your thoughts and techniques below.
", "rawMarkdown": "Feature engineering is sometimes referred to as an art of intuition and I agree. However, one always need to measure the usefulness of a feature when interacting with the others i.e. measure of feature interactions. I have read people talking about 2000 features in this competition when I had only about 260 features. Have seen that in the past too during [\"Home Credit Default Risk\"][1] competition where people mentioned over 2000 and I had only 750 features.\n\nIn the past I add features sequentially, based on CV decide what to keep or discard. This is not always practical especially when there are hundreds of features. In this competition I dug more into feature selection techniques. So here is my take:-\n\n - FE using [boruta][2] did worse than my manual technique.\n\n - Using scipy's ks_2samp mentioned by Manoj aka @mks2192 (thank you) here https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537 . This worked to a certain degree but not enough going by my CV improvement. I still had to do quite a bit of manual add and remove.\n \n - FE with null hypothesis. I missed this one during the Home credit competition posted by Olivier aka @ogrellier (thank you) [here][3]. His kernel is based on the article [\"Permutation importance: a corrected feature importance measure\"][4]. I still had to add some features manually but a lot easier.\n\n - For numerical features one can also use SVD. I have used it in the past but not in this competition.\n\nMy final selection out of 260 features was only 48. With that, I jumped from public LB #311 to #239 in the last couple of days. \n\nHow many features did you choose for your final models?\n\nI observed that the features - feature_1 , feature_2 , and feature_3 gets removed by all the methods mentioned but adding them back improved both my CV and LB. My final selected model results are :-\n\n- Local CV=3.642223, public LB=3.679 private LB = 3.611, and \n\n- public LB=3.678, private LB=3.609 - This one is the result of a simple averaging.\n\n\nPlease share your thoughts and techniques below.\n\n\n [1]: https://www.kaggle.com/c/home-credit-default-risk\n [2]: https://github.com/scikit-learn-contrib/boruta_py\n [3]: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\n [4]: https://academic.oup.com/bioinformatics/article/26/10/1340/193348", "votes": null }, { "id": "479187", "postDate": "02/27/2019 02:20:48", "content": "I generate about ~900 features, don't have good feature selection skill's, so I did not use feature selection, which gives:
\n\nI'm quite confused about feature selection.
", "rawMarkdown": "I generate about ~900 features, don't have good feature selection skill's, so I did not use feature selection, which gives:\n\n- local CV 3.642\n- public LB 3.683\n- private LB 3.604\n\nI'm quite confused about feature selection.", "votes": null }, { "id": "479194", "postDate": "02/27/2019 02:30:04", "content": "@jiazhuang, I know. It is a bit tricky hence my post hoping for a discussion. Pls. upvote the post if you like it.
", "rawMarkdown": "jiazhuang, I know. It is a bit tricky hence my post hoping for a discussion. Pls. upvote the post if you like it.", "votes": null }, { "id": "479244", "postDate": "02/27/2019 03:09:45", "content": "ok I will learn
", "rawMarkdown": "ok I will learn", "votes": null }, { "id": "479377", "postDate": "02/27/2019 05:22:23", "content": "Thanks to you for sharing this, quite a few learnings from your post here.
", "rawMarkdown": "Thanks to you for sharing this, quite a few learnings from your post here.", "votes": null }, { "id": "479381", "postDate": "02/27/2019 05:25:04", "content": "congrats YaGana Sheriff-Hussaini, I also applied the same technique for feature selection as yours. Initially my model had 125 features and was able to decrease them to 60 keeping CV entact. I guess \"Less is More\" (author-Peter Hurford) is making sense now:)\nHappy kaggling
", "rawMarkdown": "congrats YaGana Sheriff-Hussaini, I also applied the same technique for feature selection as yours. Initially my model had 125 features and was able to decrease them to 60 keeping CV entact. I guess \"Less is More\" (author-Peter Hurford) is making sense now:)\nHappy kaggling", "votes": null }, { "id": "479390", "postDate": "02/27/2019 05:29:46", "content": "Thanks @viswanathravindran and you are welcome.
", "rawMarkdown": "Thanks @viswanathravindran and you are welcome.", "votes": null }, { "id": "479392", "postDate": "02/27/2019 05:30:58", "content": "Thanks @aashish7936, less is more indeed. Congrats to you too.
", "rawMarkdown": "Thanks @aashish7936, less is more indeed. Congrats to you too.", "votes": null }, { "id": "479493", "postDate": "02/27/2019 07:38:39", "content": "Congratulations @sheriytm. Thanks for sharing your amazing work.
", "rawMarkdown": "Congratulations @sheriytm. Thanks for sharing your amazing work.", "votes": null }, { "id": "479508", "postDate": "02/27/2019 07:52:06", "content": "Thanks @karthik7395. Pls. upvote the post if you like it.
", "rawMarkdown": "Thanks @karthik7395. Pls. upvote the post if you like it.", "votes": null }, { "id": "479591", "postDate": "02/27/2019 09:09:49", "content": "Thank you for sharing ! \nHave you tried any permutation method to select your variables ? If yes base on which models ?
", "rawMarkdown": "Thank you for sharing ! \nHave you tried any permutation method to select your variables ? If yes base on which models ?", "votes": null }, { "id": "479616", "postDate": "02/27/2019 09:33:45", "content": "You are welcome @francoisdubois. Yes, I used feature permutations from which I had the most success i.e. mentioned 3rd in the list above.
", "rawMarkdown": "You are welcome @francoisdubois. Yes, I used feature permutations from which I had the most success i.e. mentioned 3rd in the list above.", "votes": null }, { "id": "479631", "postDate": "02/27/2019 09:45:12", "content": "thank you. I'm going to look at the kernel then.
", "rawMarkdown": "thank you. I'm going to look at the kernel then.", "votes": null }, { "id": "479638", "postDate": "02/27/2019 10:01:19", "content": "I must say the article \"Permutation importance: a corrected feature importance measure\" the kernel is based on is worth a read.
", "rawMarkdown": "I must say the article [\"Permutation importance: a corrected feature importance measure\"][1] the kernel is based on is worth a read.\n\n\n [1]: https://academic.oup.com/bioinformatics/article/26/10/1340/193348", "votes": null }, { "id": "479646", "postDate": "02/27/2019 10:05:09", "content": "Thank you :)
", "rawMarkdown": "Thank you :)", "votes": null }, { "id": "479796", "postDate": "02/27/2019 12:37:00", "content": "greate working. thanks
", "rawMarkdown": "greate working. thanks", "votes": null }, { "id": "479855", "postDate": "02/27/2019 13:59:08", "content": "thank you
", "rawMarkdown": "thank you", "votes": null } ], "comments": [ { "id": 479187, "author_name": "jiazhuang", "author_url": "", "post_date": "02/27/2019 02:20:48", "content": "I generate about ~900 features, don't have good feature selection skill's, so I did not use feature selection, which gives:
\n\nI'm quite confused about feature selection.
", "votes": null, "replies": [ { "id": 479194, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 02:30:04", "content": "@jiazhuang, I know. It is a bit tricky hence my post hoping for a discussion. Pls. upvote the post if you like it.
", "votes": null, "replies": [] }, { "id": 479244, "author_name": "jiazhuang", "author_url": "", "post_date": "02/27/2019 03:09:45", "content": "ok I will learn
", "votes": null, "replies": [] } ] }, { "id": 479377, "author_name": "viswanathravindran", "author_url": "", "post_date": "02/27/2019 05:22:23", "content": "Thanks to you for sharing this, quite a few learnings from your post here.
", "votes": null, "replies": [ { "id": 479390, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 05:29:46", "content": "Thanks @viswanathravindran and you are welcome.
", "votes": null, "replies": [] } ] }, { "id": 479381, "author_name": "aashish7936", "author_url": "", "post_date": "02/27/2019 05:25:04", "content": "congrats YaGana Sheriff-Hussaini, I also applied the same technique for feature selection as yours. Initially my model had 125 features and was able to decrease them to 60 keeping CV entact. I guess \"Less is More\" (author-Peter Hurford) is making sense now:)\nHappy kaggling
", "votes": null, "replies": [ { "id": 479392, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 05:30:58", "content": "Thanks @aashish7936, less is more indeed. Congrats to you too.
", "votes": null, "replies": [] } ] }, { "id": 479493, "author_name": "karthik7395", "author_url": "", "post_date": "02/27/2019 07:38:39", "content": "Congratulations @sheriytm. Thanks for sharing your amazing work.
", "votes": null, "replies": [ { "id": 479508, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 07:52:06", "content": "Thanks @karthik7395. Pls. upvote the post if you like it.
", "votes": null, "replies": [] } ] }, { "id": 479591, "author_name": "francoisdubois", "author_url": "", "post_date": "02/27/2019 09:09:49", "content": "Thank you for sharing ! \nHave you tried any permutation method to select your variables ? If yes base on which models ?
", "votes": null, "replies": [ { "id": 479616, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 09:33:45", "content": "You are welcome @francoisdubois. Yes, I used feature permutations from which I had the most success i.e. mentioned 3rd in the list above.
", "votes": null, "replies": [] }, { "id": 479631, "author_name": "francoisdubois", "author_url": "", "post_date": "02/27/2019 09:45:12", "content": "thank you. I'm going to look at the kernel then.
", "votes": null, "replies": [] }, { "id": 479638, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 10:01:19", "content": "I must say the article \"Permutation importance: a corrected feature importance measure\" the kernel is based on is worth a read.
", "votes": null, "replies": [] }, { "id": 479646, "author_name": "francoisdubois", "author_url": "", "post_date": "02/27/2019 10:05:09", "content": "Thank you :)
", "votes": null, "replies": [] } ] }, { "id": 479796, "author_name": "fuengwul5", "author_url": "", "post_date": "02/27/2019 12:37:00", "content": "greate working. thanks
", "votes": null, "replies": [] }, { "id": 479855, "author_name": "luinqiao", "author_url": "", "post_date": "02/27/2019 13:59:08", "content": "thank you
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "479169": "Feature engineering is sometimes referred to as an art of intuition and I agree. However, one always need to measure the usefulness of a feature when interacting with the others i.e. measure of feature interactions. I have read people talking about 2000 features in this competition when I had only about 260 features. Have seen that in the past too during [\"Home Credit Default Risk\"][1] competition where people mentioned over 2000 and I had only 750 features.\n\nIn the past I add features sequentially, based on CV decide what to keep or discard. This is not always practical especially when there are hundreds of features. In this competition I dug more into feature selection techniques. So here is my take:-\n\n - FE using [boruta][2] did worse than my manual technique.\n\n - Using scipy's ks_2samp mentioned by Manoj aka @mks2192 (thank you) here https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537 . This worked to a certain degree but not enough going by my CV improvement. I still had to do quite a bit of manual add and remove.\n \n - FE with null hypothesis. I missed this one during the Home credit competition posted by Olivier aka @ogrellier (thank you) [here][3]. His kernel is based on the article [\"Permutation importance: a corrected feature importance measure\"][4]. I still had to add some features manually but a lot easier.\n\n - For numerical features one can also use SVD. I have used it in the past but not in this competition.\n\nMy final selection out of 260 features was only 48. With that, I jumped from public LB #311 to #239 in the last couple of days. \n\nHow many features did you choose for your final models?\n\nI observed that the features - feature_1 , feature_2 , and feature_3 gets removed by all the methods mentioned but adding them back improved both my CV and LB. My final selected model results are :-\n\n- Local CV=3.642223, public LB=3.679 private LB = 3.611, and \n\n- public LB=3.678, private LB=3.609 - This one is the result of a simple averaging.\n\n\nPlease share your thoughts and techniques below.\n\n\n [1]: https://www.kaggle.com/c/home-credit-default-risk\n [2]: https://github.com/scikit-learn-contrib/boruta_py\n [3]: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\n [4]: https://academic.oup.com/bioinformatics/article/26/10/1340/193348", "479187": "I generate about ~900 features, don't have good feature selection skill's, so I did not use feature selection, which gives:\n\n- local CV 3.642\n- public LB 3.683\n- private LB 3.604\n\nI'm quite confused about feature selection.", "479194": "jiazhuang, I know. It is a bit tricky hence my post hoping for a discussion. Pls. upvote the post if you like it.", "479244": "ok I will learn", "479377": "Thanks to you for sharing this, quite a few learnings from your post here.", "479381": "congrats YaGana Sheriff-Hussaini, I also applied the same technique for feature selection as yours. Initially my model had 125 features and was able to decrease them to 60 keeping CV entact. I guess \"Less is More\" (author-Peter Hurford) is making sense now:)\nHappy kaggling", "479390": "Thanks @viswanathravindran and you are welcome.", "479392": "Thanks @aashish7936, less is more indeed. Congrats to you too.", "479493": "Congratulations @sheriytm. Thanks for sharing your amazing work.", "479508": "Thanks @karthik7395. Pls. upvote the post if you like it.", "479591": "Thank you for sharing ! \nHave you tried any permutation method to select your variables ? If yes base on which models ?", "479616": "You are welcome @francoisdubois. Yes, I used feature permutations from which I had the most success i.e. mentioned 3rd in the list above.", "479631": "thank you. I'm going to look at the kernel then.", "479638": "I must say the article [\"Permutation importance: a corrected feature importance measure\"][1] the kernel is based on is worth a read.\n\n\n [1]: https://academic.oup.com/bioinformatics/article/26/10/1340/193348", "479646": "Thank you :)", "479796": "greate working. thanks", "479855": "thank you" }, "source": "meta" }