{
  "id": 67795,
  "title": "How to handle categorical variable values that are present in the test set but not in the training set ?",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/67795",
  "author_name": "",
  "post_date": "2018-10-05T14:24:52.072957600Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>I'm a machine learning noob and wanted to understand how to handle categorical variables values that are only present in the test set but not in the training set.</p>\n\n<p>I haven't checked whether that is the case in this competition or not, but wanted to understand how to handle it especially given the high cardinality of some of the features in this dataset.</p>\n\n<p>Should I treat all the values in test set but not in train set as \"other\" category ? If so, I will only be reducing the number of unseen values to 1, but there will still be 1 unseen value when trying to make predictions, isn't it ?</p>\n\n<p>Also, do some models like tree based models somehow do the right thing with unseen categorical values ?</p>",
  "messages": [
    {
      "id": "399273",
      "postDate": "10/05/2018 14:24:52",
      "content": "<p>Hi,</p>\n\n<p>I'm a machine learning noob and wanted to understand how to handle categorical variables values that are only present in the test set but not in the training set.</p>\n\n<p>I haven't checked whether that is the case in this competition or not, but wanted to understand how to handle it especially given the high cardinality of some of the features in this dataset.</p>\n\n<p>Should I treat all the values in test set but not in train set as \"other\" category ? If so, I will only be reducing the number of unseen values to 1, but there will still be 1 unseen value when trying to make predictions, isn't it ?</p>\n\n<p>Also, do some models like tree based models somehow do the right thing with unseen categorical values ?</p>",
      "rawMarkdown": "Hi,\n\nI'm a machine learning noob and wanted to understand how to handle categorical variables values that are only present in the test set but not in the training set.\n\nI haven't checked whether that is the case in this competition or not, but wanted to understand how to handle it especially given the high cardinality of some of the features in this dataset.\n\nShould I treat all the values in test set but not in train set as \"other\" category ? If so, I will only be reducing the number of unseen values to 1, but there will still be 1 unseen value when trying to make predictions, isn't it ?\n\nAlso, do some models like tree based models somehow do the right thing with unseen categorical values ?",
      "votes": null
    },
    {
      "id": "399331",
      "postDate": "10/05/2018 16:05:54",
      "content": "<p>There are many angels we can deal with it. There's no single right answer to that\n1. It can well as be as an outlier that can be dropped\n2. If you are label encoding only on a sample of train data or do label/onehot encoding to test and train separately <br>\n3. Checking out the nearest neighbour to that row to understand the new label in test</p>\n\n<p>A lot would depend on the EDA. If the alien categorical variable is 1/5000 then I wouldn't care. Or probably drop it. Or, check if that's a typo by chance. Alike, <code>Hyderabad'</code> can also be written <code>'Hyderebad', 'Hyderabaad'</code> or other combinations around. In other case we might have very high value counts of 5 categories and remaining not so much. And, probably in that case I would classify it in others. </p>\n\n<p>So, there's no one right answer. It depends on the problem, analysis and what that feature is all about in first place. </p>",
      "rawMarkdown": "There are many angels we can deal with it. There's no single right answer to that\n1. It can well as be as an outlier that can be dropped\n2. If you are label encoding only on a sample of train data or do label/onehot encoding to test and train separately  \n3. Checking out the nearest neighbour to that row to understand the new label in test\n\nA lot would depend on the EDA. If the alien categorical variable is 1/5000 then I wouldn't care. Or probably drop it. Or, check if that's a typo by chance. Alike, `Hyderabad'` can also be written `'Hyderebad', 'Hyderabaad'` or other combinations around. In other case we might have very high value counts of 5 categories and remaining not so much. And, probably in that case I would classify it in others. \n\nSo, there's no one right answer. It depends on the problem, analysis and what that feature is all about in first place.",
      "votes": null
    },
    {
      "id": "930606",
      "postDate": "07/15/2020 15:34:39",
      "content": "<p>Should I treat all the values in test set but not in train set as \"other\" category ?</p>",
      "rawMarkdown": "Should I treat all the values in test set but not in train set as \"other\" category ?",
      "votes": null
    },
    {
      "id": "2014765",
      "postDate": "11/02/2022 20:13:15",
      "content": "<p>You can label them as \"other\", or use label encoding.</p>",
      "rawMarkdown": "You can label them as \"other\", or use label encoding.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2014765,
      "author_name": "jonathanlampkin",
      "author_url": "",
      "post_date": "11/02/2022 20:13:15",
      "content": "<p>You can label them as \"other\", or use label encoding.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 399331,
      "author_name": "shaz13",
      "author_url": "",
      "post_date": "10/05/2018 16:05:54",
      "content": "<p>There are many angels we can deal with it. There's no single right answer to that\n1. It can well as be as an outlier that can be dropped\n2. If you are label encoding only on a sample of train data or do label/onehot encoding to test and train separately <br>\n3. Checking out the nearest neighbour to that row to understand the new label in test</p>\n\n<p>A lot would depend on the EDA. If the alien categorical variable is 1/5000 then I wouldn't care. Or probably drop it. Or, check if that's a typo by chance. Alike, <code>Hyderabad'</code> can also be written <code>'Hyderebad', 'Hyderabaad'</code> or other combinations around. In other case we might have very high value counts of 5 categories and remaining not so much. And, probably in that case I would classify it in others. </p>\n\n<p>So, there's no one right answer. It depends on the problem, analysis and what that feature is all about in first place. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 930606,
      "author_name": "bendengr",
      "author_url": "",
      "post_date": "07/15/2020 15:34:39",
      "content": "<p>Should I treat all the values in test set but not in train set as \"other\" category ?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "399273": "Hi,\n\nI'm a machine learning noob and wanted to understand how to handle categorical variables values that are only present in the test set but not in the training set.\n\nI haven't checked whether that is the case in this competition or not, but wanted to understand how to handle it especially given the high cardinality of some of the features in this dataset.\n\nShould I treat all the values in test set but not in train set as \"other\" category ? If so, I will only be reducing the number of unseen values to 1, but there will still be 1 unseen value when trying to make predictions, isn't it ?\n\nAlso, do some models like tree based models somehow do the right thing with unseen categorical values ?",
    "399331": "There are many angels we can deal with it. There's no single right answer to that\n1. It can well as be as an outlier that can be dropped\n2. If you are label encoding only on a sample of train data or do label/onehot encoding to test and train separately  \n3. Checking out the nearest neighbour to that row to understand the new label in test\n\nA lot would depend on the EDA. If the alien categorical variable is 1/5000 then I wouldn't care. Or probably drop it. Or, check if that's a typo by chance. Alike, `Hyderabad'` can also be written `'Hyderebad', 'Hyderabaad'` or other combinations around. In other case we might have very high value counts of 5 categories and remaining not so much. And, probably in that case I would classify it in others. \n\nSo, there's no one right answer. It depends on the problem, analysis and what that feature is all about in first place.",
    "930606": "Should I treat all the values in test set but not in train set as \"other\" category ?",
    "2014765": "You can label them as \"other\", or use label encoding."
  },
  "source": "meta"
}