{
  "id": 66334,
  "title": "How do you deal with missing values in categorical variables?",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/66334",
  "author_name": "",
  "post_date": "2018-09-20T18:46:39.942007200Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hey there,</p>\n\n<p>I hope you're all doing well with your predictions!</p>\n\n<p>My question concerns the dataset's missing values. For handling missing Values in numeric columns, there are many solutions (for example replace them with the mean), but in categorical columns I'm struggling to handle them.</p>\n\n<p>What is your solution to this problem?</p>",
  "messages": [
    {
      "id": "390737",
      "postDate": "09/20/2018 18:46:39",
      "content": "<p>Hey there,</p>\n\n<p>I hope you're all doing well with your predictions!</p>\n\n<p>My question concerns the dataset's missing values. For handling missing Values in numeric columns, there are many solutions (for example replace them with the mean), but in categorical columns I'm struggling to handle them.</p>\n\n<p>What is your solution to this problem?</p>",
      "rawMarkdown": "Hey there,\n\nI hope you're all doing well with your predictions!\n\nMy question concerns the dataset's missing values. For handling missing Values in numeric columns, there are many solutions (for example replace them with the mean), but in categorical columns I'm struggling to handle them.\n\nWhat is your solution to this problem?",
      "votes": null
    },
    {
      "id": "390757",
      "postDate": "09/20/2018 19:17:19",
      "content": "<p>The easiest solution (that often works perfectly well) is to just treat missing/null as a category in itself, meaning that you don't really have to treat it any differently than all the other categories.</p>\n\n<p>Some of your other options are more similar to the way you would handle numeric columns - replace missing values with the mode category, the mode category conditioned on some other features (e.g. groupby 2 important features and fill nulls with the mode for each group), or try a dedicated classification model that predicts the category using some other features (this is like a fancier version of the previous option).</p>",
      "rawMarkdown": "The easiest solution (that often works perfectly well) is to just treat missing/null as a category in itself, meaning that you don't really have to treat it any differently than all the other categories.\n\nSome of your other options are more similar to the way you would handle numeric columns - replace missing values with the mode category, the mode category conditioned on some other features (e.g. groupby 2 important features and fill nulls with the mode for each group), or try a dedicated classification model that predicts the category using some other features (this is like a fancier version of the previous option).",
      "votes": null
    },
    {
      "id": "391487",
      "postDate": "09/21/2018 23:05:08",
      "content": "<p>Thanks!\nFor example I could use a linear regression to predict a missing values of variables.\nBut how does that affect the quality of the revenue estimator?</p>\n\n<p>A question on top of the first one: I was wondering what do you do with missing values in the transaction_revenue column?</p>",
      "rawMarkdown": "Thanks!\nFor example I could use a linear regression to predict a missing values of variables.\nBut how does that affect the quality of the revenue estimator?\n\n\nA question on top of the first one: I was wondering what do you do with missing values in the transaction_revenue column?",
      "votes": null
    },
    {
      "id": "391618",
      "postDate": "09/22/2018 05:33:25",
      "content": "<p>Since the transaction_revenue is the response we intend to predict, we don’t impute its missing values. If your model doesn’t handle NA in the response variable, set them to 0. Because a missing revenue indicates a non-purchase. I hope that helps!</p>",
      "rawMarkdown": "Since the transaction_revenue is the response we intend to predict, we don’t impute its missing values. If your model doesn’t handle NA in the response variable, set them to 0. Because a missing revenue indicates a non-purchase. I hope that helps!",
      "votes": null
    },
    {
      "id": "391625",
      "postDate": "09/22/2018 05:53:22",
      "content": "<p>You can also make an extra feature with df[column_name].isnull(), it might give you a better perdiction.</p>",
      "rawMarkdown": "You can also make an extra feature with df[column_name].isnull(), it might give you a better perdiction.",
      "votes": null
    },
    {
      "id": "413703",
      "postDate": "11/01/2018 11:44:48",
      "content": "<p>Thanks! That was my question: How do we know missing in transactionRevenue is 0?</p>",
      "rawMarkdown": "Thanks! That was my question: How do we know missing in transactionRevenue is 0?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 390757,
      "author_name": "aquatic",
      "author_url": "",
      "post_date": "09/20/2018 19:17:19",
      "content": "<p>The easiest solution (that often works perfectly well) is to just treat missing/null as a category in itself, meaning that you don't really have to treat it any differently than all the other categories.</p>\n\n<p>Some of your other options are more similar to the way you would handle numeric columns - replace missing values with the mode category, the mode category conditioned on some other features (e.g. groupby 2 important features and fill nulls with the mode for each group), or try a dedicated classification model that predicts the category using some other features (this is like a fancier version of the previous option).</p>",
      "votes": null,
      "replies": [
        {
          "id": 391487,
          "author_name": "azacha",
          "author_url": "",
          "post_date": "09/21/2018 23:05:08",
          "content": "<p>Thanks!\nFor example I could use a linear regression to predict a missing values of variables.\nBut how does that affect the quality of the revenue estimator?</p>\n\n<p>A question on top of the first one: I was wondering what do you do with missing values in the transaction_revenue column?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 391618,
          "author_name": "mijchou",
          "author_url": "",
          "post_date": "09/22/2018 05:33:25",
          "content": "<p>Since the transaction_revenue is the response we intend to predict, we don’t impute its missing values. If your model doesn’t handle NA in the response variable, set them to 0. Because a missing revenue indicates a non-purchase. I hope that helps!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 413703,
          "author_name": "ramonafli",
          "author_url": "",
          "post_date": "11/01/2018 11:44:48",
          "content": "<p>Thanks! That was my question: How do we know missing in transactionRevenue is 0?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 391625,
      "author_name": "gsreyan",
      "author_url": "",
      "post_date": "09/22/2018 05:53:22",
      "content": "<p>You can also make an extra feature with df[column_name].isnull(), it might give you a better perdiction.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "390737": "Hey there,\n\nI hope you're all doing well with your predictions!\n\nMy question concerns the dataset's missing values. For handling missing Values in numeric columns, there are many solutions (for example replace them with the mean), but in categorical columns I'm struggling to handle them.\n\nWhat is your solution to this problem?",
    "390757": "The easiest solution (that often works perfectly well) is to just treat missing/null as a category in itself, meaning that you don't really have to treat it any differently than all the other categories.\n\nSome of your other options are more similar to the way you would handle numeric columns - replace missing values with the mode category, the mode category conditioned on some other features (e.g. groupby 2 important features and fill nulls with the mode for each group), or try a dedicated classification model that predicts the category using some other features (this is like a fancier version of the previous option).",
    "391487": "Thanks!\nFor example I could use a linear regression to predict a missing values of variables.\nBut how does that affect the quality of the revenue estimator?\n\n\nA question on top of the first one: I was wondering what do you do with missing values in the transaction_revenue column?",
    "391618": "Since the transaction_revenue is the response we intend to predict, we don’t impute its missing values. If your model doesn’t handle NA in the response variable, set them to 0. Because a missing revenue indicates a non-purchase. I hope that helps!",
    "391625": "You can also make an extra feature with df[column_name].isnull(), it might give you a better perdiction.",
    "413703": "Thanks! That was my question: How do we know missing in transactionRevenue is 0?"
  },
  "source": "meta"
}