It looks like for this competition, you can improve your score more by being careful about which features you don't use than which features you do use. That is, feature selection is really important.
\n\nHere are some great kernels elsewhere on Kaggle to get you started:
\n\nSome of these techniques have already lead me to shave 0.006 off my LB score for my best model.
", "messages": [ { "id": 434712, "postDate": "2018-12-06T20:44:13.033Z", "content": "It looks like for this competition, you can improve your score more by being careful about which features you don't use than which features you do use. That is, feature selection is really important.
\n\nHere are some great kernels elsewhere on Kaggle to get you started:
\n\nSome of these techniques have already lead me to shave 0.006 off my LB score for my best model.
", "rawMarkdown": "It looks like for this competition, you can improve your score more by being careful about which features *you don't use* than which features you do use. That is, feature selection is really important.\n\nHere are some great kernels elsewhere on Kaggle to get you started:\n\n* [Feature Selection with Null Importances](https://www.kaggle.com/ogrellier/feature-selection-with-null-importances)\n* [Boruta Feature Elimination](https://www.kaggle.com/tilii7/boruta-feature-elimination)\n* [Feature Ranking RFE, Random Forest, linear models](https://www.kaggle.com/arthurtok/feature-ranking-rfe-random-forest-linear-models)\n* [Recursive Feature Elimination](https://www.kaggle.com/tilii7/recursive-feature-elimination/code)\n* [Recursive Feature Elimination with LGBM](https://www.kaggle.com/kailex/recursive-feature-elimination-with-lgbm/code)\n\nSome of these techniques have already lead me to shave 0.006 off my LB score for my best model.", "votes": 129 }, { "id": 435449, "postDate": "2018-12-08T04:04:05.550Z", "content": "I like Boruta algorithm.
Here are other explanations and code examples.
\n\nThis is really helpful Maxwell. I got what i was looking for so long time.
", "rawMarkdown": "This is really helpful Maxwell. I got what i was looking for so long time." }, { "id": 631838, "postDate": "2019-09-22T19:36:01.057Z", "content": "I heard Boruta before but i didn't know it until now. It looks pretty cool and clever. The blog I read is this one :http://danielhomola.com/2015/05/08/borutapy-an-all-relevant-feature-selection-method/
", "rawMarkdown": "I heard Boruta before but i didn't know it until now. It looks pretty cool and clever. The blog I read is this one :http://danielhomola.com/2015/05/08/borutapy-an-all-relevant-feature-selection-method/" }, { "id": 631986, "postDate": "2019-09-23T04:05:00.190Z", "content": "Thanks for sharing @maxwell110 ! Just one quick question, how would you apply Boruta with different learning algorithms than Random Forest? Since different algorithms like XGBoost behave very differently on the same features. Would you run Boruta Feature Selection using Random Forest at first, and then apply the selected features to train the actual model using XGBoost for example. Thanks!
", "rawMarkdown": "Thanks for sharing @maxwell110 ! Just one quick question, how would you apply Boruta with different learning algorithms than Random Forest? Since different algorithms like XGBoost behave very differently on the same features. Would you run Boruta Feature Selection using Random Forest at first, and then apply the selected features to train the actual model using XGBoost for example. Thanks!" } ] }, { "id": 435765, "postDate": "2018-12-08T18:13:50.010Z", "content": "As far as I know permutation importance recently became popular to measure importance of variables,
", "rawMarkdown": "As far as I know permutation importance recently became popular to measure importance of variables,", "votes": 1 }, { "id": 434760, "postDate": "2018-12-06T22:39:00.263Z", "content": "Thanks!\nDo you choose to create 100 (for example) almost blindly new features and perform Feature Selection Methods to determine the most important ones or create one feature at a time and observe the changes in your CV?
", "rawMarkdown": "Thanks!\nDo you choose to create 100 (for example) almost blindly new features and perform Feature Selection Methods to determine the most important ones or create one feature at a time and observe the changes in your CV?", "votes": 1, "replies": [ { "id": 434768, "postDate": "2018-12-06T22:58:44.877Z", "content": "So far, I've been mainly starting with a large feature list and then narrowing it down from there.
", "rawMarkdown": "So far, I've been mainly starting with a large feature list and then narrowing it down from there.", "votes": 4 } ] }, { "id": 458878, "postDate": "2019-01-20T17:57:14.367Z", "content": "Guys like you are angels to us (learners) :) tons of like for the informative post, cheers!
", "rawMarkdown": "Guys like you are angels to us (learners) :) tons of like for the informative post, cheers!", "votes": 2 }, { "id": 475267, "postDate": "2019-02-20T14:43:32.867Z", "content": "Has anyone tried Boruta Feature Elimination on this dataset?\nI am trying it, but it is rejecting most of the features of my training set. This is a new algorithm to me, therefore I don't know exactly how to tune Boruta and RF parameters for the purpose of Feature Elimination.
", "rawMarkdown": "Has anyone tried Boruta Feature Elimination on this dataset?\nI am trying it, but it is rejecting most of the features of my training set. This is a new algorithm to me, therefore I don't know exactly how to tune Boruta and RF parameters for the purpose of Feature Elimination." }, { "id": 452976, "postDate": "2019-01-09T12:35:20.003Z", "content": "Well, RFE seems not to improve my LB, but it improves cv.
", "rawMarkdown": "Well, RFE seems not to improve my LB, but it improves cv.", "replies": [ { "id": 456598, "postDate": "2019-01-16T06:42:53.630Z", "content": "it also cannot work for me
", "rawMarkdown": "it also cannot work for me " } ] }, { "id": 459265, "postDate": "2019-01-21T13:27:13.383Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 459425, "postDate": "2019-01-21T17:52:01.253Z", "content": "You need to change the line 282 to use with lgb.
\n\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76977#457880
", "rawMarkdown": "You need to change the line 282 to use with lgb.\n\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76977#457880 " }, { "id": 459610, "postDate": "2019-01-22T04:41:48.073Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 440305, "postDate": "2018-12-17T11:25:41.393Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 440459, "postDate": "2018-12-17T15:13:27.627Z", "content": "You can replace missing values using pandas or sklearn. A safe value it the median of the column
", "rawMarkdown": "You can replace missing values using pandas or sklearn. A safe value it the median of the column" }, { "id": 442562, "postDate": "2018-12-20T06:00:45.090Z", "content": "If the feature is numeric, then you can replace it with mean or median. It it is categorical then you can replace it with mode.
", "rawMarkdown": "If the feature is numeric, then you can replace it with mean or median. It it is categorical then you can replace it with mode.", "votes": 1 }, { "id": 444099, "postDate": "2018-12-23T07:07:52.723Z", "content": "Don't merge new_merchanttransactions.csv to train.csv directly. Extract features from new_merchanttransactions.csv and merge the features to the train.csv
Hi Qing Liu, I don't understand the way you talked about extracting features from new_merchanttransactions.csv and merge the features , can you be more specific about it? Thanks!
", "rawMarkdown": "Hi Qing Liu, I don't understand the way you talked about extracting features from new_merchanttransactions.csv and merge the features , can you be more specific about it? Thanks!" } ] } ], "comments": [ { "id": 435449, "author_name": "Maxwell", "author_url": "", "post_date": "2018-12-08T04:04:05.550000", "content": "I like Boruta algorithm.
Here are other explanations and code examples.
\n\nThis is really helpful Maxwell. I got what i was looking for so long time.
", "votes": 0, "replies": [] }, { "id": 631838, "author_name": "MuhammedBuyukkinaci", "author_url": "", "post_date": "2019-09-22T19:36:01.057000", "content": "I heard Boruta before but i didn't know it until now. It looks pretty cool and clever. The blog I read is this one :http://danielhomola.com/2015/05/08/borutapy-an-all-relevant-feature-selection-method/
", "votes": 0, "replies": [] }, { "id": 631986, "author_name": "Terry Pham", "author_url": "", "post_date": "2019-09-23T04:05:00.190000", "content": "Thanks for sharing @maxwell110 ! Just one quick question, how would you apply Boruta with different learning algorithms than Random Forest? Since different algorithms like XGBoost behave very differently on the same features. Would you run Boruta Feature Selection using Random Forest at first, and then apply the selected features to train the actual model using XGBoost for example. Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 435765, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2018-12-08T18:13:50.010000", "content": "As far as I know permutation importance recently became popular to measure importance of variables,
", "votes": 1, "replies": [] }, { "id": 434760, "author_name": "Dimitrios Effrosynidis", "author_url": "", "post_date": "2018-12-06T22:39:00.263000", "content": "Thanks!\nDo you choose to create 100 (for example) almost blindly new features and perform Feature Selection Methods to determine the most important ones or create one feature at a time and observe the changes in your CV?
", "votes": 1, "replies": [ { "id": 434768, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-12-06T22:58:44.877000", "content": "So far, I've been mainly starting with a large feature list and then narrowing it down from there.
", "votes": 4, "replies": [] } ] }, { "id": 458878, "author_name": "Peaky", "author_url": "", "post_date": "2019-01-20T17:57:14.367000", "content": "Guys like you are angels to us (learners) :) tons of like for the informative post, cheers!
", "votes": 2, "replies": [] }, { "id": 475267, "author_name": "Feliphe Galiza", "author_url": "", "post_date": "2019-02-20T14:43:32.867000", "content": "Has anyone tried Boruta Feature Elimination on this dataset?\nI am trying it, but it is rejecting most of the features of my training set. This is a new algorithm to me, therefore I don't know exactly how to tune Boruta and RF parameters for the purpose of Feature Elimination.
", "votes": 0, "replies": [] }, { "id": 452976, "author_name": "kxx", "author_url": "", "post_date": "2019-01-09T12:35:20.003000", "content": "Well, RFE seems not to improve my LB, but it improves cv.
", "votes": 0, "replies": [ { "id": 456598, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-01-16T06:42:53.630000", "content": "it also cannot work for me
", "votes": 0, "replies": [] } ] }, { "id": 459265, "author_name": "", "author_url": "", "post_date": "2019-01-21T13:27:13.383000", "content": "", "votes": 0, "replies": [ { "id": 459425, "author_name": "silverstone", "author_url": "", "post_date": "2019-01-21T17:52:01.253000", "content": "You need to change the line 282 to use with lgb.
\n\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76977#457880
", "votes": 0, "replies": [] }, { "id": 459610, "author_name": "", "author_url": "", "post_date": "2019-01-22T04:41:48.073000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 440305, "author_name": "", "author_url": "", "post_date": "2018-12-17T11:25:41.393000", "content": "", "votes": 1, "replies": [ { "id": 440459, "author_name": "Dimitrios Effrosynidis", "author_url": "", "post_date": "2018-12-17T15:13:27.627000", "content": "You can replace missing values using pandas or sklearn. A safe value it the median of the column
", "votes": 0, "replies": [] }, { "id": 442562, "author_name": "Chandra Shekhar", "author_url": "", "post_date": "2018-12-20T06:00:45.090000", "content": "If the feature is numeric, then you can replace it with mean or median. It it is categorical then you can replace it with mode.
", "votes": 1, "replies": [] }, { "id": 444099, "author_name": "Qing Liu", "author_url": "", "post_date": "2018-12-23T07:07:52.723000", "content": "Don't merge new_merchanttransactions.csv to train.csv directly. Extract features from new_merchanttransactions.csv and merge the features to the train.csv
Hi Qing Liu, I don't understand the way you talked about extracting features from new_merchanttransactions.csv and merge the features , can you be more specific about it? Thanks!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "434712": "It looks like for this competition, you can improve your score more by being careful about which features *you don't use* than which features you do use. That is, feature selection is really important.\n\nHere are some great kernels elsewhere on Kaggle to get you started:\n\n* [Feature Selection with Null Importances](https://www.kaggle.com/ogrellier/feature-selection-with-null-importances)\n* [Boruta Feature Elimination](https://www.kaggle.com/tilii7/boruta-feature-elimination)\n* [Feature Ranking RFE, Random Forest, linear models](https://www.kaggle.com/arthurtok/feature-ranking-rfe-random-forest-linear-models)\n* [Recursive Feature Elimination](https://www.kaggle.com/tilii7/recursive-feature-elimination/code)\n* [Recursive Feature Elimination with LGBM](https://www.kaggle.com/kailex/recursive-feature-elimination-with-lgbm/code)\n\nSome of these techniques have already lead me to shave 0.006 off my LB score for my best model.", "435449": "I like `Boruta` algorithm.\n\nHere are other explanations and code examples.\n\n1. https://www.datacamp.com/community/tutorials/feature-selection-R-boruta\n2. https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/\n3. https://www.jstatsoft.org/article/view/v036i11/v36i11.pdf\n", "435765": "As far as I know permutation importance recently became popular to measure importance of variables,", "434760": "Thanks!\nDo you choose to create 100 (for example) almost blindly new features and perform Feature Selection Methods to determine the most important ones or create one feature at a time and observe the changes in your CV?", "458878": "Guys like you are angels to us (learners) :) tons of like for the informative post, cheers!", "475267": "Has anyone tried Boruta Feature Elimination on this dataset?\nI am trying it, but it is rejecting most of the features of my training set. This is a new algorithm to me, therefore I don't know exactly how to tune Boruta and RF parameters for the purpose of Feature Elimination.", "452976": "Well, RFE seems not to improve my LB, but it improves cv.", "459265": "", "440305": "" } }