{
  "id": 81074,
  "title": "Stop calling them outliers and you will get more ideas",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/81074",
  "author_name": "Inoddy",
  "post_date": "2019-02-19T00:47:35.329000",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I get the feeling that calling certain responses \"outliers\" is boxing some people in.</p>\n\n<p>They are NOT outliers. Don't be distracted by the label -33. Regard it as a NaN.</p>\n\n<p>The response is technically a mixed binary and continuous response variable. If you search for terms such as two part models, zero inflated models, mixed binary and continuous responses, and (the best) latent variable models you will get ideas for handling the data.</p>\n\n<p>Once you stop calling them outliers you will find that there are true outliers in the data. Some have quite large z-scores and contribute to the RMSE. You may or may not want to handle them in the usual ways.</p>\n\n<p>If I had time I would experiment with this myself and post some examples.</p>\n\n<p>That being said, gradient boosting methods and ad-hoc approaches to the mixed response are pretty good. So for many people the gains from not calling them outliers may only be small. But you may still get some ideas.</p>",
  "messages": [
    {
      "id": 474595,
      "postDate": "2019-02-19T15:51:04.920Z",
      "content": "<p>\"If I had time I would experiment with this myself and post some examples.\"</p>\n\n<p>Yeah, well, that's just it - we all have some great ideas and potential approaches that sound reasonable in theory, but until you try them out and see if they work, you don't know if they are just wishful thinking on your part. If I only had a gold medal for every brilliant idea I've had over the years that didn't pen out in the end ... </p>",
      "rawMarkdown": "\"If I had time I would experiment with this myself and post some examples.\"\n\nYeah, well, that's just it - we all have some great ideas and potential approaches that sound reasonable in theory, but until you try them out and see if they work, you don't know if they are just wishful thinking on your part. If I only had a gold medal for every brilliant idea I've had over the years that didn't pen out in the end ... ",
      "votes": 16,
      "replies": [
        {
          "id": 475585,
          "postDate": "2019-02-21T00:05:08.230Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 474114,
      "postDate": "2019-02-19T00:47:35.330Z",
      "content": "<p>I get the feeling that calling certain responses \"outliers\" is boxing some people in.</p>\n\n<p>They are NOT outliers. Don't be distracted by the label -33. Regard it as a NaN.</p>\n\n<p>The response is technically a mixed binary and continuous response variable. If you search for terms such as two part models, zero inflated models, mixed binary and continuous responses, and (the best) latent variable models you will get ideas for handling the data.</p>\n\n<p>Once you stop calling them outliers you will find that there are true outliers in the data. Some have quite large z-scores and contribute to the RMSE. You may or may not want to handle them in the usual ways.</p>\n\n<p>If I had time I would experiment with this myself and post some examples.</p>\n\n<p>That being said, gradient boosting methods and ad-hoc approaches to the mixed response are pretty good. So for many people the gains from not calling them outliers may only be small. But you may still get some ideas.</p>",
      "rawMarkdown": "I get the feeling that calling certain responses \"outliers\" is boxing some people in.\n\nThey are NOT outliers. Don't be distracted by the label -33. Regard it as a NaN.\n\nThe response is technically a mixed binary and continuous response variable. If you search for terms such as two part models, zero inflated models, mixed binary and continuous responses, and (the best) latent variable models you will get ideas for handling the data.\n\nOnce you stop calling them outliers you will find that there are true outliers in the data. Some have quite large z-scores and contribute to the RMSE. You may or may not want to handle them in the usual ways.\n\nIf I had time I would experiment with this myself and post some examples.\n\nThat being said, gradient boosting methods and ad-hoc approaches to the mixed response are pretty good. So for many people the gains from not calling them outliers may only be small. But you may still get some ideas.",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 474595,
      "author_name": "Bojan Tunguz",
      "author_url": "",
      "post_date": "2019-02-19T15:51:04.920000",
      "content": "<p>\"If I had time I would experiment with this myself and post some examples.\"</p>\n\n<p>Yeah, well, that's just it - we all have some great ideas and potential approaches that sound reasonable in theory, but until you try them out and see if they work, you don't know if they are just wishful thinking on your part. If I only had a gold medal for every brilliant idea I've had over the years that didn't pen out in the end ... </p>",
      "votes": 16,
      "replies": [
        {
          "id": 475585,
          "author_name": "",
          "author_url": "",
          "post_date": "2019-02-21T00:05:08.230000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "474595": "\"If I had time I would experiment with this myself and post some examples.\"\n\nYeah, well, that's just it - we all have some great ideas and potential approaches that sound reasonable in theory, but until you try them out and see if they work, you don't know if they are just wishful thinking on your part. If I only had a gold medal for every brilliant idea I've had over the years that didn't pen out in the end ... ",
    "474114": "I get the feeling that calling certain responses \"outliers\" is boxing some people in.\n\nThey are NOT outliers. Don't be distracted by the label -33. Regard it as a NaN.\n\nThe response is technically a mixed binary and continuous response variable. If you search for terms such as two part models, zero inflated models, mixed binary and continuous responses, and (the best) latent variable models you will get ideas for handling the data.\n\nOnce you stop calling them outliers you will find that there are true outliers in the data. Some have quite large z-scores and contribute to the RMSE. You may or may not want to handle them in the usual ways.\n\nIf I had time I would experiment with this myself and post some examples.\n\nThat being said, gradient boosting methods and ad-hoc approaches to the mixed response are pretty good. So for many people the gains from not calling them outliers may only be small. But you may still get some ideas."
  }
}