{ "id": 287804, "title": "Avoid Overfitting with Feature Neutralization", "url": "/competitions/g-research-crypto-forecasting/discussion/287804", "author_name": "", "post_date": "2021-11-15T16:38:49.813000", "votes": 54, "comment_count": 3, "views": 0, "content": "

'

\n
\n

TL;DR: You can avoid overfitting with the minimal loss of the competition metric | code

\n
\n

As an overfitting champion of the public LB,

\n

Allow me to start addressing the topic and suggest that we might want to think about methods to how to avoid overfitting.
\nSince in the end, we all aim for the private LB (the future).

\n
\n

HUGE Credit to the original author:
\nWhat you are about to read and see the code of is based on the incredible work of @code1110 during the Jane street competition earlier this year. His work not only was useful for the competition but also for my day-to-day work!
\nIf you find the following useful or Interesting go upvote the original code, please. He earned it!

\n

@code1110 Thank you for sharing!

\n
\n

There exists a powerful practice called Feature Neutralization.

\n

It was used on Jane street competition and also has been very successful on the Numerai Tournament [another financial ML competition, ongoing].

\n

Neutralization
\nThe trick is to remove a simple linear component from a column in the dataset. this column can be a feature, the target, or even the model's prediction.

\n

The intuition is that this forces the model to learn only from the information that is not correlated to that column and thus if our training set is correlated to the target during the era we train on: our models will be more resistant to this bias. [And hopefully when this bias stops being presented in real life: our model's performance won't get hurt since we removed it earlier]

\n

\"\"
\n[from https://docs.numer.ai/numerai-signals/signals-overview#neutralization]

\n

The way this is done is simply by fitting a linear function between the columns we want to neutralize and the column we want to neutralize by and then simply subtracting the linear component learned.

\n
def neutralize(df, target=\"Target\", by=None, proportion=1.0):\n    if by is None: by = [x for x in df.columns if x.startswith('feature')]\n    scores = df[target]\n    exposures = df[by].values\n    exposures = np.hstack((exposures, np.array([np.mean(scores)] * len(exposures)).reshape(-1, 1)))\n    scores -= proportion * (exposures @ (np.linalg.pinv(exposures) @ scores.values))\n    return scores / scores.std()\n
\n

Exposure
\nAfter we understand the Idea of neutralization, we can start referring to the linear component we removed as \"exposure\" since the magnitude of it represents the amount of which the column exposes us to the bias.

\n
import numpy as np\nfrom scipy.stats import spearmanr\n\nTARGET_COL = f\"Target\"\n\ndef feature_exposures(df):\n    feature_names = [f for f in df.columns if f.startswith(\"feature\")]\n    exposures = []\n    for f in feature_names:\n        fe = spearmanr(df[TARGET_COL], df[f])[0]\n        exposures.append(fe)\n    return np.array(exposures)\n\ndef max_feature_exposure(df): \n    return np.max(np.abs(feature_exposures(df)))\ndef feature_exposure(df): \n    return np.sqrt(np.mean(np.square(feature_exposures(df))))\n
\n

Summary
\nBy using feature neutralization we can transform our features to be much less correlated with targets while still keeping most of the information presented in the original features [see the top figure].

\n

I made a public notebook explaining how to do it, if you are into avoiding overfitting: check it out

\n

And If not: 1.0 LB it isssss!! 💪 👌😂

\n

\"\"
\n[from https://forum.numer.ai/t/model-diagnostics-feature-exposure/899]

\n

Cheers^^
\nHappy Kaggling!

", "messages": [ { "id": 1583233, "postDate": "2021-11-15T16:38:49.813Z", "content": "

'

\n
\n

TL;DR: You can avoid overfitting with the minimal loss of the competition metric | code

\n
\n

As an overfitting champion of the public LB,

\n

Allow me to start addressing the topic and suggest that we might want to think about methods to how to avoid overfitting.
\nSince in the end, we all aim for the private LB (the future).

\n
\n

HUGE Credit to the original author:
\nWhat you are about to read and see the code of is based on the incredible work of @code1110 during the Jane street competition earlier this year. His work not only was useful for the competition but also for my day-to-day work!
\nIf you find the following useful or Interesting go upvote the original code, please. He earned it!

\n

@code1110 Thank you for sharing!

\n
\n

There exists a powerful practice called Feature Neutralization.

\n

It was used on Jane street competition and also has been very successful on the Numerai Tournament [another financial ML competition, ongoing].

\n

Neutralization
\nThe trick is to remove a simple linear component from a column in the dataset. this column can be a feature, the target, or even the model's prediction.

\n

The intuition is that this forces the model to learn only from the information that is not correlated to that column and thus if our training set is correlated to the target during the era we train on: our models will be more resistant to this bias. [And hopefully when this bias stops being presented in real life: our model's performance won't get hurt since we removed it earlier]

\n

\"\"
\n[from https://docs.numer.ai/numerai-signals/signals-overview#neutralization]

\n

The way this is done is simply by fitting a linear function between the columns we want to neutralize and the column we want to neutralize by and then simply subtracting the linear component learned.

\n
def neutralize(df, target=\"Target\", by=None, proportion=1.0):\n    if by is None: by = [x for x in df.columns if x.startswith('feature')]\n    scores = df[target]\n    exposures = df[by].values\n    exposures = np.hstack((exposures, np.array([np.mean(scores)] * len(exposures)).reshape(-1, 1)))\n    scores -= proportion * (exposures @ (np.linalg.pinv(exposures) @ scores.values))\n    return scores / scores.std()\n
\n

Exposure
\nAfter we understand the Idea of neutralization, we can start referring to the linear component we removed as \"exposure\" since the magnitude of it represents the amount of which the column exposes us to the bias.

\n
import numpy as np\nfrom scipy.stats import spearmanr\n\nTARGET_COL = f\"Target\"\n\ndef feature_exposures(df):\n    feature_names = [f for f in df.columns if f.startswith(\"feature\")]\n    exposures = []\n    for f in feature_names:\n        fe = spearmanr(df[TARGET_COL], df[f])[0]\n        exposures.append(fe)\n    return np.array(exposures)\n\ndef max_feature_exposure(df): \n    return np.max(np.abs(feature_exposures(df)))\ndef feature_exposure(df): \n    return np.sqrt(np.mean(np.square(feature_exposures(df))))\n
\n

Summary
\nBy using feature neutralization we can transform our features to be much less correlated with targets while still keeping most of the information presented in the original features [see the top figure].

\n

I made a public notebook explaining how to do it, if you are into avoiding overfitting: check it out

\n

And If not: 1.0 LB it isssss!! 💪 👌😂

\n

\"\"
\n[from https://forum.numer.ai/t/model-diagnostics-feature-exposure/899]

\n

Cheers^^
\nHappy Kaggling!

", "rawMarkdown": "'\n> **TL;DR**: You can avoid overfitting with the minimal loss of the competition metric | [code](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/)\n\n\nAs an overfitting champion of the public LB,\n\nAllow me to start addressing the topic and suggest that we might want to think about methods to how to avoid overfitting.\nSince in the end, we all aim for the private LB (the future).\n\n____\n**HUGE Credit to the original author:** \nWhat you are about to read and see the code of is based on the incredible work of @code1110 during the Jane street competition earlier this year. His work not only was useful for the competition but also for my day-to-day work! \nIf you find the following useful or Interesting go upvote the [original code](https://www.kaggle.com/code1110/janestreet-avoid-overfit-feature-neutralization?scriptVersionId=53012639), please. He earned it!\n\n@code1110 Thank you for sharing! \n____\n\nThere exists a powerful practice called Feature Neutralization.\n\nIt was used on Jane street competition and also has been very successful on the Numerai Tournament [another financial ML competition, ongoing].\n\n**Neutralization**\nThe trick is to remove a simple linear component from a column in the dataset. this column can be a feature, the target, or even the model's prediction.\n\nThe intuition is that this forces the model to learn only from the information that is not correlated to that column and thus if our training set is correlated to the target during the era we train on: our models will be more resistant to this bias. [And hopefully when this bias stops being presented in real life: our model's performance won't get hurt since we removed it earlier]\n\n![](https://i.ibb.co/j4vmDKQ/image.png)\n[from https://docs.numer.ai/numerai-signals/signals-overview#neutralization]\n\nThe way this is done is simply by fitting a linear function between the columns we want to neutralize and the column we want to neutralize by and then simply subtracting the linear component learned. \n\n```\ndef neutralize(df, target=\"Target\", by=None, proportion=1.0):\n if by is None: by = [x for x in df.columns if x.startswith('feature')]\n scores = df[target]\n exposures = df[by].values\n exposures = np.hstack((exposures, np.array([np.mean(scores)] * len(exposures)).reshape(-1, 1)))\n scores -= proportion * (exposures @ (np.linalg.pinv(exposures) @ scores.values))\n return scores / scores.std()\n```\n\n**Exposure**\nAfter we understand the Idea of neutralization, we can start referring to the linear component we removed as \"exposure\" since the magnitude of it represents the amount of which the column exposes us to the bias. \n\n```\nimport numpy as np\nfrom scipy.stats import spearmanr\n\nTARGET_COL = f\"Target\"\n\ndef feature_exposures(df):\n feature_names = [f for f in df.columns if f.startswith(\"feature\")]\n exposures = []\n for f in feature_names:\n fe = spearmanr(df[TARGET_COL], df[f])[0]\n exposures.append(fe)\n return np.array(exposures)\n\ndef max_feature_exposure(df): \n return np.max(np.abs(feature_exposures(df)))\ndef feature_exposure(df): \n return np.sqrt(np.mean(np.square(feature_exposures(df))))\n```\n\n**Summary**\nBy using feature neutralization we can transform our features to be much less correlated with targets while still keeping most of the information presented in the original features [see the top figure].\n\n\nI made a public notebook explaining how to do it, if you are into avoiding overfitting: [check it out](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/)\n\nAnd If not: 1.0 LB it isssss!! 💪 👌😂\n\n![](https://i.ibb.co/nbFWyHv/e1d39e8f38ae51c1189ee85fd9578a2a555a3c06.jpg)\n[from https://forum.numer.ai/t/model-diagnostics-feature-exposure/899]\n\nCheers^^\nHappy Kaggling! \n", "votes": 54 }, { "id": 1593535, "postDate": "2021-11-24T03:28:19.720Z", "content": "

so, is there any intuition why feature normalization can help overfitting ? the vector used to normalize is easy to overfit ?
\nif so, what kind of features belong to this class ?

", "rawMarkdown": "so, is there any intuition why feature normalization can help overfitting ? the vector used to normalize is easy to overfit ?\nif so, what kind of features belong to this class ? " }, { "id": 1593369, "postDate": "2021-11-23T20:25:24.737Z", "content": "

How can you differentitate between good and bad exposures for the case of target neutralization? CV over public dataset?

", "rawMarkdown": "How can you differentitate between good and bad exposures for the case of target neutralization? CV over public dataset?" }, { "id": 1590736, "postDate": "2021-11-21T15:53:33.157Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1593535, "author_name": "bncdvd", "author_url": "", "post_date": "2021-11-24T03:28:19.720000", "content": "

so, is there any intuition why feature normalization can help overfitting ? the vector used to normalize is easy to overfit ?
\nif so, what kind of features belong to this class ?

", "votes": 0, "replies": [] }, { "id": 1593369, "author_name": "shay winter", "author_url": "", "post_date": "2021-11-23T20:25:24.737000", "content": "

How can you differentitate between good and bad exposures for the case of target neutralization? CV over public dataset?

", "votes": 0, "replies": [] }, { "id": 1590736, "author_name": "", "author_url": "", "post_date": "2021-11-21T15:53:33.157000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1583233": "'\n> **TL;DR**: You can avoid overfitting with the minimal loss of the competition metric | [code](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/)\n\n\nAs an overfitting champion of the public LB,\n\nAllow me to start addressing the topic and suggest that we might want to think about methods to how to avoid overfitting.\nSince in the end, we all aim for the private LB (the future).\n\n____\n**HUGE Credit to the original author:** \nWhat you are about to read and see the code of is based on the incredible work of @code1110 during the Jane street competition earlier this year. His work not only was useful for the competition but also for my day-to-day work! \nIf you find the following useful or Interesting go upvote the [original code](https://www.kaggle.com/code1110/janestreet-avoid-overfit-feature-neutralization?scriptVersionId=53012639), please. He earned it!\n\n@code1110 Thank you for sharing! \n____\n\nThere exists a powerful practice called Feature Neutralization.\n\nIt was used on Jane street competition and also has been very successful on the Numerai Tournament [another financial ML competition, ongoing].\n\n**Neutralization**\nThe trick is to remove a simple linear component from a column in the dataset. this column can be a feature, the target, or even the model's prediction.\n\nThe intuition is that this forces the model to learn only from the information that is not correlated to that column and thus if our training set is correlated to the target during the era we train on: our models will be more resistant to this bias. [And hopefully when this bias stops being presented in real life: our model's performance won't get hurt since we removed it earlier]\n\n![](https://i.ibb.co/j4vmDKQ/image.png)\n[from https://docs.numer.ai/numerai-signals/signals-overview#neutralization]\n\nThe way this is done is simply by fitting a linear function between the columns we want to neutralize and the column we want to neutralize by and then simply subtracting the linear component learned. \n\n```\ndef neutralize(df, target=\"Target\", by=None, proportion=1.0):\n if by is None: by = [x for x in df.columns if x.startswith('feature')]\n scores = df[target]\n exposures = df[by].values\n exposures = np.hstack((exposures, np.array([np.mean(scores)] * len(exposures)).reshape(-1, 1)))\n scores -= proportion * (exposures @ (np.linalg.pinv(exposures) @ scores.values))\n return scores / scores.std()\n```\n\n**Exposure**\nAfter we understand the Idea of neutralization, we can start referring to the linear component we removed as \"exposure\" since the magnitude of it represents the amount of which the column exposes us to the bias. \n\n```\nimport numpy as np\nfrom scipy.stats import spearmanr\n\nTARGET_COL = f\"Target\"\n\ndef feature_exposures(df):\n feature_names = [f for f in df.columns if f.startswith(\"feature\")]\n exposures = []\n for f in feature_names:\n fe = spearmanr(df[TARGET_COL], df[f])[0]\n exposures.append(fe)\n return np.array(exposures)\n\ndef max_feature_exposure(df): \n return np.max(np.abs(feature_exposures(df)))\ndef feature_exposure(df): \n return np.sqrt(np.mean(np.square(feature_exposures(df))))\n```\n\n**Summary**\nBy using feature neutralization we can transform our features to be much less correlated with targets while still keeping most of the information presented in the original features [see the top figure].\n\n\nI made a public notebook explaining how to do it, if you are into avoiding overfitting: [check it out](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/)\n\nAnd If not: 1.0 LB it isssss!! 💪 👌😂\n\n![](https://i.ibb.co/nbFWyHv/e1d39e8f38ae51c1189ee85fd9578a2a555a3c06.jpg)\n[from https://forum.numer.ai/t/model-diagnostics-feature-exposure/899]\n\nCheers^^\nHappy Kaggling! \n", "1593535": "so, is there any intuition why feature normalization can help overfitting ? the vector used to normalize is easy to overfit ?\nif so, what kind of features belong to this class ? ", "1593369": "How can you differentitate between good and bad exposures for the case of target neutralization? CV over public dataset?", "1590736": "" } }