{ "id": 441929, "title": "feature selection", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/441929", "author_name": "LuminooWu", "post_date": "2023-09-20T15:54:13.995000", "votes": 2, "comment_count": 7, "views": 0, "content": "

I'm trying a baseline now , If one feature makes my public score go down , such as \"month\" and \"date\" , can i just throw it ?
\nI did the feature importance analysis from a xgboost model , but i found the \"month\" feature is more important than \"weekday\" feature. I am confused about whether keep this feature or not.
\nIs it worth to throw all features with very small importance?
\nAlso, maybe i need to stick to local CV score and stop trying creating features and then look at the public score?
\nSo please give me some advice on feature selection, thank u in advance.

", "messages": [ { "id": 2448484, "postDate": "2023-09-20T16:00:37.020Z", "content": "

Dear @wjlnobody

\n

Here is a little notebook I wrote the other day which could be a place to start \"SHAP Permutation explainer + random \"probe\"\". The random probe part is creating some extra features out of random noise. Any feature that is ranked as being less important than such a random column can be deemed as unimportant and dropped from your model.

\n

All the best,
\ncarl

", "rawMarkdown": "Dear @wjlnobody \n\nHere is a little notebook I wrote the other day which could be a place to start [\"SHAP Permutation explainer + random \"probe\"\"](https://www.kaggle.com/code/carlmcbrideellis/shap-permutation-explainer-random-probe). The random probe part is creating some extra features out of random noise. Any feature that is ranked as being less important than such a random column can be deemed as unimportant and dropped from your model.\n\nAll the best,\ncarl", "votes": 7, "replies": [ { "id": 2449011, "postDate": "2023-09-21T00:00:04.590Z", "content": "

Dear carl
\nThis is a novel method for a novice like me.i will try it.
\nAll the best,
\nluminoo

", "rawMarkdown": "Dear carl\nThis is a novel method for a novice like me.i will try it.\nAll the best,\nluminoo", "votes": 2 } ] }, { "id": 2448504, "postDate": "2023-09-20T16:15:51.167Z", "content": "

For me, I try to keep as many not biased feature as possible, even if it is very small importance. I only drop the column when I am out of the memory. Also, I think that there are possibility that one feature makes the public score go down, but the private score stays same or even rise, so just trust your local CV and check if CV and LB is correlated.

", "rawMarkdown": "For me, I try to keep as many not biased feature as possible, even if it is very small importance. I only drop the column when I am out of the memory. Also, I think that there are possibility that one feature makes the public score go down, but the private score stays same or even rise, so just trust your local CV and check if CV and LB is correlated.", "votes": 3, "replies": [ { "id": 2449013, "postDate": "2023-09-21T00:05:24.970Z", "content": "

yes i also wonder that cv is more important.
\nBy the way,do you have methods to check the correlation between CV and LB?

", "rawMarkdown": "yes i also wonder that cv is more important.\nBy the way,do you have methods to check the correlation between CV and LB?" } ] }, { "id": 2448475, "postDate": "2023-09-20T15:54:13.997Z", "content": "

I'm trying a baseline now , If one feature makes my public score go down , such as \"month\" and \"date\" , can i just throw it ?
\nI did the feature importance analysis from a xgboost model , but i found the \"month\" feature is more important than \"weekday\" feature. I am confused about whether keep this feature or not.
\nIs it worth to throw all features with very small importance?
\nAlso, maybe i need to stick to local CV score and stop trying creating features and then look at the public score?
\nSo please give me some advice on feature selection, thank u in advance.

", "rawMarkdown": "I'm trying a baseline now , If one feature makes my public score go down , such as \"month\" and \"date\" , can i just throw it ? \nI did the feature importance analysis from a xgboost model , but i found the \"month\" feature is more important than \"weekday\" feature. I am confused about whether keep this feature or not.\nIs it worth to throw all features with very small importance? \nAlso, maybe i need to stick to local CV score and stop trying creating features and then look at the public score?\nSo please give me some advice on feature selection, thank u in advance.", "votes": 2 }, { "id": 2453616, "postDate": "2023-09-24T07:37:32.917Z", "content": "

Hi @wjlnobody. Please also refer to previous similar posts. Don't let good ideas get buried…

", "rawMarkdown": "Hi @wjlnobody. Please also refer to [previous similar posts](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438298). Don't let good ideas get buried..." }, { "id": 2450395, "postDate": "2023-09-21T19:58:21.910Z", "content": "

You can try permutation importances as well

\n", "rawMarkdown": "You can try permutation importances as well\n- Explanation from Kaggle's course https://www.kaggle.com/code/dansbecker/permutation-importance\n- One of the most revered notebooks on this https://www.kaggle.com/code/ogrellier/feature-selection-with-null-importances/notebook\n- Another Example https://www.kaggle.com/code/paultimothymooney/feature-selection-with-permutation-importance" }, { "id": 2450447, "postDate": "2023-09-21T21:13:55.727Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 2448484, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-09-20T16:00:37.020000", "content": "

Dear @wjlnobody

\n

Here is a little notebook I wrote the other day which could be a place to start \"SHAP Permutation explainer + random \"probe\"\". The random probe part is creating some extra features out of random noise. Any feature that is ranked as being less important than such a random column can be deemed as unimportant and dropped from your model.

\n

All the best,
\ncarl

", "votes": 7, "replies": [ { "id": 2449011, "author_name": "LuminooWu", "author_url": "", "post_date": "2023-09-21T00:00:04.590000", "content": "

Dear carl
\nThis is a novel method for a novice like me.i will try it.
\nAll the best,
\nluminoo

", "votes": 2, "replies": [] } ] }, { "id": 2448504, "author_name": "Yuchen20", "author_url": "", "post_date": "2023-09-20T16:15:51.167000", "content": "

For me, I try to keep as many not biased feature as possible, even if it is very small importance. I only drop the column when I am out of the memory. Also, I think that there are possibility that one feature makes the public score go down, but the private score stays same or even rise, so just trust your local CV and check if CV and LB is correlated.

", "votes": 3, "replies": [ { "id": 2449013, "author_name": "LuminooWu", "author_url": "", "post_date": "2023-09-21T00:05:24.970000", "content": "

yes i also wonder that cv is more important.
\nBy the way,do you have methods to check the correlation between CV and LB?

", "votes": 0, "replies": [] } ] }, { "id": 2453616, "author_name": "mh", "author_url": "", "post_date": "2023-09-24T07:37:32.917000", "content": "

Hi @wjlnobody. Please also refer to previous similar posts. Don't let good ideas get buried…

", "votes": 0, "replies": [] }, { "id": 2450395, "author_name": "x5or6", "author_url": "", "post_date": "2023-09-21T19:58:21.910000", "content": "

You can try permutation importances as well

\n", "votes": 0, "replies": [] }, { "id": 2450447, "author_name": "", "author_url": "", "post_date": "2023-09-21T21:13:55.727000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2448484": "Dear @wjlnobody \n\nHere is a little notebook I wrote the other day which could be a place to start [\"SHAP Permutation explainer + random \"probe\"\"](https://www.kaggle.com/code/carlmcbrideellis/shap-permutation-explainer-random-probe). The random probe part is creating some extra features out of random noise. Any feature that is ranked as being less important than such a random column can be deemed as unimportant and dropped from your model.\n\nAll the best,\ncarl", "2448504": "For me, I try to keep as many not biased feature as possible, even if it is very small importance. I only drop the column when I am out of the memory. Also, I think that there are possibility that one feature makes the public score go down, but the private score stays same or even rise, so just trust your local CV and check if CV and LB is correlated.", "2448475": "I'm trying a baseline now , If one feature makes my public score go down , such as \"month\" and \"date\" , can i just throw it ? \nI did the feature importance analysis from a xgboost model , but i found the \"month\" feature is more important than \"weekday\" feature. I am confused about whether keep this feature or not.\nIs it worth to throw all features with very small importance? \nAlso, maybe i need to stick to local CV score and stop trying creating features and then look at the public score?\nSo please give me some advice on feature selection, thank u in advance.", "2453616": "Hi @wjlnobody. Please also refer to [previous similar posts](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438298). Don't let good ideas get buried...", "2450395": "You can try permutation importances as well\n- Explanation from Kaggle's course https://www.kaggle.com/code/dansbecker/permutation-importance\n- One of the most revered notebooks on this https://www.kaggle.com/code/ogrellier/feature-selection-with-null-importances/notebook\n- Another Example https://www.kaggle.com/code/paultimothymooney/feature-selection-with-permutation-importance", "2450447": "" } }