{ "id": 295255, "title": "Let's talk feature selection", "url": "/competitions/g-research-crypto-forecasting/discussion/295255", "author_name": "", "post_date": "2021-12-15T06:13:11.754878200Z", "votes": 1, "comment_count": 6, "views": 0, "content": "

Hi all,

\n

It's very easy to overfit when using a lot of features in this competition, so I thought I'll create a thread to discuss feature selection.

\n

I described my approach in a series of Notebooks: Part I, Part II, and Part III.

\n

What about you? What approach do you use and how effective has it been for you?

", "messages": [ { "id": "1618594", "postDate": "12/15/2021 06:13:11", "content": "

Hi all,

\n

It's very easy to overfit when using a lot of features in this competition, so I thought I'll create a thread to discuss feature selection.

\n

I described my approach in a series of Notebooks: Part I, Part II, and Part III.

\n

What about you? What approach do you use and how effective has it been for you?

", "rawMarkdown": "Hi all,\n\nIt's very easy to overfit when using a lot of features in this competition, so I thought I'll create a thread to discuss feature selection.\n\nI described my approach in a series of Notebooks: [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), and [Part III](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-iii).\n\nWhat about you? What approach do you use and how effective has it been for you?", "votes": null }, { "id": "1618650", "postDate": "12/15/2021 07:44:07", "content": "

I'm using mutual info for feature_selection , and I also see BorutaShap in numerai, but BorutaShap cost too much time..

", "rawMarkdown": "I'm using [mutual info](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html) for feature_selection , and I also see [BorutaShap](https://forum.numer.ai/t/feature-selection-with-borutashap/4145) in numerai, but BorutaShap cost too much time..", "votes": null }, { "id": "1619146", "postDate": "12/15/2021 17:11:58", "content": "

Thanks for sharing.

\n

Do you use mutual information for univariate feature selection?

\n

If so, how do you detect/eliminate redundant features (e.g. moving averages with close enough window sizes)?

\n

And how do you detect features that are not useful when used by themselves to predict the target, but that can complement/boost other features?

", "rawMarkdown": "Thanks for sharing. \n\nDo you use mutual information for univariate feature selection? \n\nIf so, how do you detect/eliminate redundant features (e.g. moving averages with close enough window sizes)? \n\nAnd how do you detect features that are not useful when used by themselves to predict the target, but that can complement/boost other features?", "votes": null }, { "id": "1631278", "postDate": "12/28/2021 08:20:24", "content": "

no, I use correlation for the univariate feature selection, choose high and consistent(in sample and out of sample) ones, then use mutual info for the final selection. The boosting features are hard to recognize, maybe permutations could helps

", "rawMarkdown": "no, I use correlation for the univariate feature selection, choose high and consistent(in sample and out of sample) ones, then use mutual info for the final selection. The boosting features are hard to recognize, maybe permutations could helps", "votes": null }, { "id": "1653670", "postDate": "01/17/2022 19:21:13", "content": "

By only looking at the dependency between each feature and the target, you are merely scratching the surface.

\n

For instance, you will not be able to detect and eliminate features that are great but redundant.

\n

Similarly, you will not be able to find features that are great complement to other features but useless to predict the target when used by themselves.

\n

Here's a dummy example. Imagine you want to use elements of an address (i.e., street name, house number, post code, city, country) to predict GPS coordinates.

\n

Intuitively, house number is useless to predict GPS coordinates; you can find the same house number on most street, in most cities, and in most countries.

\n

Now, if I give you all elements of an address but the house number, you'll be able to locate the position within a radius of up to a few hundreds of meters on average. However, if I give you all elements of the address, including the house number, you'll be able to locate the position within the confines of a house (i.e. within a radius of up to a few tens of meters on average).

\n

Had you used correlation or pairwise mutual information, you'd have eliminated the house number on the basis that it is not informative about GPS coordinates when used by itself. However, when used in conjunction with the street name and post code, it can help shrink your error by a factor of 10.

\n

Mutual information is the way to go, so long as you use a multivariate version thereof. You need to use multivariate mutual information to take into account redundancy and complementarity between features.

\n

More on this in the following tutorial series: Part I, Part II, Part III.

", "rawMarkdown": "By only looking at the dependency between each feature and the target, you are merely scratching the surface. \n\nFor instance, you will not be able to detect and eliminate features that are great but redundant. \n\nSimilarly, you will not be able to find features that are great complement to other features but useless to predict the target when used by themselves. \n\nHere's a dummy example. Imagine you want to use elements of an address (i.e., street name, house number, post code, city, country) to predict GPS coordinates. \n\nIntuitively, house number is useless to predict GPS coordinates; you can find the same house number on most street, in most cities, and in most countries.\n\nNow, if I give you all elements of an address but the house number, you'll be able to locate the position within a radius of up to a few hundreds of meters on average. However, if I give you all elements of the address, including the house number, you'll be able to locate the position within the confines of a house (i.e. within a radius of up to a few tens of meters on average).\n\nHad you used correlation or pairwise mutual information, you'd have eliminated the house number on the basis that it is not informative about GPS coordinates when used by itself. However, when used in conjunction with the street name and post code, it can help shrink your error by a factor of 10.\n\nMutual information is the way to go, so long as you use a multivariate version thereof. You need to use multivariate mutual information to take into account redundancy and complementarity between features.\n\nMore on this in the following tutorial series: [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), [Part III](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-iii).", "votes": null }, { "id": "1654187", "postDate": "01/18/2022 08:52:57", "content": "

It seems that the market is dynamic and some features may not work in the future.

", "rawMarkdown": "It seems that the market is dynamic and some features may not work in the future.", "votes": null }, { "id": "1656911", "postDate": "01/19/2022 18:57:39", "content": "

Check out this blog post. You might find it useful.

", "rawMarkdown": "Check out [this blog post](https://medium.com/kxytechnologies/non-stationarity-and-memory-in-financial-markets-4b8d1200667c). You might find it useful.", "votes": null } ], "comments": [ { "id": 1618650, "author_name": "cymchenzelin", "author_url": "", "post_date": "12/15/2021 07:44:07", "content": "

I'm using mutual info for feature_selection , and I also see BorutaShap in numerai, but BorutaShap cost too much time..

", "votes": null, "replies": [ { "id": 1619146, "author_name": "leanboosting", "author_url": "", "post_date": "12/15/2021 17:11:58", "content": "

Thanks for sharing.

\n

Do you use mutual information for univariate feature selection?

\n

If so, how do you detect/eliminate redundant features (e.g. moving averages with close enough window sizes)?

\n

And how do you detect features that are not useful when used by themselves to predict the target, but that can complement/boost other features?

", "votes": null, "replies": [] }, { "id": 1631278, "author_name": "cymchenzelin", "author_url": "", "post_date": "12/28/2021 08:20:24", "content": "

no, I use correlation for the univariate feature selection, choose high and consistent(in sample and out of sample) ones, then use mutual info for the final selection. The boosting features are hard to recognize, maybe permutations could helps

", "votes": null, "replies": [] }, { "id": 1653670, "author_name": "leanboosting", "author_url": "", "post_date": "01/17/2022 19:21:13", "content": "

By only looking at the dependency between each feature and the target, you are merely scratching the surface.

\n

For instance, you will not be able to detect and eliminate features that are great but redundant.

\n

Similarly, you will not be able to find features that are great complement to other features but useless to predict the target when used by themselves.

\n

Here's a dummy example. Imagine you want to use elements of an address (i.e., street name, house number, post code, city, country) to predict GPS coordinates.

\n

Intuitively, house number is useless to predict GPS coordinates; you can find the same house number on most street, in most cities, and in most countries.

\n

Now, if I give you all elements of an address but the house number, you'll be able to locate the position within a radius of up to a few hundreds of meters on average. However, if I give you all elements of the address, including the house number, you'll be able to locate the position within the confines of a house (i.e. within a radius of up to a few tens of meters on average).

\n

Had you used correlation or pairwise mutual information, you'd have eliminated the house number on the basis that it is not informative about GPS coordinates when used by itself. However, when used in conjunction with the street name and post code, it can help shrink your error by a factor of 10.

\n

Mutual information is the way to go, so long as you use a multivariate version thereof. You need to use multivariate mutual information to take into account redundancy and complementarity between features.

\n

More on this in the following tutorial series: Part I, Part II, Part III.

", "votes": null, "replies": [] } ] }, { "id": 1654187, "author_name": "yuanzhezhou", "author_url": "", "post_date": "01/18/2022 08:52:57", "content": "

It seems that the market is dynamic and some features may not work in the future.

", "votes": null, "replies": [ { "id": 1656911, "author_name": "leanboosting", "author_url": "", "post_date": "01/19/2022 18:57:39", "content": "

Check out this blog post. You might find it useful.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1618594": "Hi all,\n\nIt's very easy to overfit when using a lot of features in this competition, so I thought I'll create a thread to discuss feature selection.\n\nI described my approach in a series of Notebooks: [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), and [Part III](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-iii).\n\nWhat about you? What approach do you use and how effective has it been for you?", "1618650": "I'm using [mutual info](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html) for feature_selection , and I also see [BorutaShap](https://forum.numer.ai/t/feature-selection-with-borutashap/4145) in numerai, but BorutaShap cost too much time..", "1619146": "Thanks for sharing. \n\nDo you use mutual information for univariate feature selection? \n\nIf so, how do you detect/eliminate redundant features (e.g. moving averages with close enough window sizes)? \n\nAnd how do you detect features that are not useful when used by themselves to predict the target, but that can complement/boost other features?", "1631278": "no, I use correlation for the univariate feature selection, choose high and consistent(in sample and out of sample) ones, then use mutual info for the final selection. The boosting features are hard to recognize, maybe permutations could helps", "1653670": "By only looking at the dependency between each feature and the target, you are merely scratching the surface. \n\nFor instance, you will not be able to detect and eliminate features that are great but redundant. \n\nSimilarly, you will not be able to find features that are great complement to other features but useless to predict the target when used by themselves. \n\nHere's a dummy example. Imagine you want to use elements of an address (i.e., street name, house number, post code, city, country) to predict GPS coordinates. \n\nIntuitively, house number is useless to predict GPS coordinates; you can find the same house number on most street, in most cities, and in most countries.\n\nNow, if I give you all elements of an address but the house number, you'll be able to locate the position within a radius of up to a few hundreds of meters on average. However, if I give you all elements of the address, including the house number, you'll be able to locate the position within the confines of a house (i.e. within a radius of up to a few tens of meters on average).\n\nHad you used correlation or pairwise mutual information, you'd have eliminated the house number on the basis that it is not informative about GPS coordinates when used by itself. However, when used in conjunction with the street name and post code, it can help shrink your error by a factor of 10.\n\nMutual information is the way to go, so long as you use a multivariate version thereof. You need to use multivariate mutual information to take into account redundancy and complementarity between features.\n\nMore on this in the following tutorial series: [Part I](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-i), [Part II](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii), [Part III](https://www.kaggle.com/leanboosting/automating-feature-engineering-part-iii).", "1654187": "It seems that the market is dynamic and some features may not work in the future.", "1656911": "Check out [this blog post](https://medium.com/kxytechnologies/non-stationarity-and-memory-in-financial-markets-4b8d1200667c). You might find it useful." }, "source": "meta" }