{
  "id": 11276,
  "title": "Levels in test is not in the train",
  "url": "/competitions/avazu-ctr-prediction/discussion/11276",
  "author_name": "",
  "post_date": "2014-12-16T21:01:39.683Z",
  "votes": 1,
  "comment_count": 3,
  "views": 2005,
  "content": "<p>Hello,</p>\n\n<p>This is my first time taking a Kaggle competition, mostly for learning purpose.&nbsp;My question could be naive.</p>\n<p>I am fitting a logistic regression, but when I do the prediction for the test dataset, for about half of the variables, there are a lot of levels/values in the test dataset that are not in the train datasets. Since all the variables are categorical, if a level/value is in the test but not in the train, how&nbsp;can I make the prediction? &nbsp;</p>\n<p>For example, for the device_ip,&nbsp;75901 out of&nbsp;107988 unique ip in the test are never present in the train.&nbsp;</p>\n<p>Any help would be appreciated!</p>",
  "messages": [
    {
      "id": "60397",
      "postDate": "12/16/2014 21:01:39",
      "content": "<p>Hello,</p>\n\n<p>This is my first time taking a Kaggle competition, mostly for learning purpose.&nbsp;My question could be naive.</p>\n<p>I am fitting a logistic regression, but when I do the prediction for the test dataset, for about half of the variables, there are a lot of levels/values in the test dataset that are not in the train datasets. Since all the variables are categorical, if a level/value is in the test but not in the train, how&nbsp;can I make the prediction? &nbsp;</p>\n<p>For example, for the device_ip,&nbsp;75901 out of&nbsp;107988 unique ip in the test are never present in the train.&nbsp;</p>\n<p>Any help would be appreciated!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60399",
      "postDate": "12/16/2014 21:07:47",
      "content": "<p>Just ignore this levels or make special 'other' level.</p>\n<p>We have nothing to do with such cases.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60400",
      "postDate": "12/16/2014 21:13:02",
      "content": "<p>You can't use unique unseen categorical features from the test set to make a prediction.</p>\n<p>There are a few options:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">Delete these features and focus on features that appear in both train and test set to keep dimensionality low.</span></li>\n<li><span style=\"line-height: 1.4\">Keep updating weights and adding features during testing.&nbsp;</span></li>\n<li><span style=\"line-height: 1.4\">Semi-supervised learning: Use predictions on the test set as extra data and add to train set. Now these unseen features will be seen during training and hopefully your model learns from them.</span></li>\n<li><span style=\"line-height: 1.4\">Encode</span><span style=\"line-height: 1.4\">&nbsp;a categorical feature for when you encounter a device_ip that hasn't been seen before. For example: has_not_seen_device_ip:1. Create a new training set from the training set, where you remove a certain % of device_ips and replace them with the unseen categorical feature, hopefully teaching your model on how to better predict cases where the device_ip is unseen. Or build two models: One for seen device_ip's in test, one for unseen device_ip's in test.</span></li>\n</ul>\n<p><span style=\"line-height: 1.4\">Good luck!</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60404",
      "postDate": "12/16/2014 21:36:58",
      "content": "<p>[quote=Triskelion;60400]</p>\n<p>You can't use unique unseen categorical features from the test set to make a prediction.</p>\n<p>There are a few options:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">Delete these features and focus on features that appear in both train and test set to keep dimensionality low.</span></li>\n<li><span style=\"line-height: 1.4\">Keep updating weights and adding features during testing.&nbsp;</span></li>\n<li><span style=\"line-height: 1.4\">Semi-supervised learning: Use predictions on the test set as extra data and add to train set. Now these unseen features will be seen during training and hopefully your model learns from them.</span></li>\n<li><span style=\"line-height: 1.4\">Encode</span><span style=\"line-height: 1.4\">&nbsp;a categorical feature for when you encounter a device_ip that hasn't been seen before. For example: has_not_seen_device_ip:1. Create a new training set from the training set, where you remove a certain % of device_ips and replace them with the unseen categorical feature, hopefully teaching your model on how to better predict cases where the device_ip is unseen. Or build two models: One for seen device_ip's in test, one for unseen device_ip's in test.</span></li>\n</ul>\n<p><span style=\"line-height: 1.4\">Good luck!</span></p>\n<p>[/quote]</p>\n\n<p>Thank you very much, this is really helpful and inspired!</p>\n<p>We are thinking that the values for some of the variables might contain some information. For example, the first three digits of site_id stand for the geographic position (I made it). So if I group the site_id by these three digits for this variable, it could turn out that there is no unseen feature in the test. The problem is that we do not have the information about what the values stand for for a variable. So, not sure if it is feasible.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 60399,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/16/2014 21:07:47",
      "content": "<p>Just ignore this levels or make special 'other' level.</p>\n<p>We have nothing to do with such cases.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60400,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "12/16/2014 21:13:02",
      "content": "<p>You can't use unique unseen categorical features from the test set to make a prediction.</p>\n<p>There are a few options:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">Delete these features and focus on features that appear in both train and test set to keep dimensionality low.</span></li>\n<li><span style=\"line-height: 1.4\">Keep updating weights and adding features during testing.&nbsp;</span></li>\n<li><span style=\"line-height: 1.4\">Semi-supervised learning: Use predictions on the test set as extra data and add to train set. Now these unseen features will be seen during training and hopefully your model learns from them.</span></li>\n<li><span style=\"line-height: 1.4\">Encode</span><span style=\"line-height: 1.4\">&nbsp;a categorical feature for when you encounter a device_ip that hasn't been seen before. For example: has_not_seen_device_ip:1. Create a new training set from the training set, where you remove a certain % of device_ips and replace them with the unseen categorical feature, hopefully teaching your model on how to better predict cases where the device_ip is unseen. Or build two models: One for seen device_ip's in test, one for unseen device_ip's in test.</span></li>\n</ul>\n<p><span style=\"line-height: 1.4\">Good luck!</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60404,
      "author_name": "louify",
      "author_url": "",
      "post_date": "12/16/2014 21:36:58",
      "content": "<p>[quote=Triskelion;60400]</p>\n<p>You can't use unique unseen categorical features from the test set to make a prediction.</p>\n<p>There are a few options:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">Delete these features and focus on features that appear in both train and test set to keep dimensionality low.</span></li>\n<li><span style=\"line-height: 1.4\">Keep updating weights and adding features during testing.&nbsp;</span></li>\n<li><span style=\"line-height: 1.4\">Semi-supervised learning: Use predictions on the test set as extra data and add to train set. Now these unseen features will be seen during training and hopefully your model learns from them.</span></li>\n<li><span style=\"line-height: 1.4\">Encode</span><span style=\"line-height: 1.4\">&nbsp;a categorical feature for when you encounter a device_ip that hasn't been seen before. For example: has_not_seen_device_ip:1. Create a new training set from the training set, where you remove a certain % of device_ips and replace them with the unseen categorical feature, hopefully teaching your model on how to better predict cases where the device_ip is unseen. Or build two models: One for seen device_ip's in test, one for unseen device_ip's in test.</span></li>\n</ul>\n<p><span style=\"line-height: 1.4\">Good luck!</span></p>\n<p>[/quote]</p>\n\n<p>Thank you very much, this is really helpful and inspired!</p>\n<p>We are thinking that the values for some of the variables might contain some information. For example, the first three digits of site_id stand for the geographic position (I made it). So if I group the site_id by these three digits for this variable, it could turn out that there is no unseen feature in the test. The problem is that we do not have the information about what the values stand for for a variable. So, not sure if it is feasible.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "60397": "",
    "60399": "",
    "60400": "",
    "60404": ""
  },
  "source": "meta"
}