{
  "id": 24050,
  "title": "best single model?",
  "url": "/competitions/bosch-production-line-performance/discussion/24050",
  "author_name": "",
  "post_date": "2016-10-03T12:28:53.737Z",
  "votes": 10,
  "comment_count": 7,
  "views": 2158,
  "content": "<p>I have a single XG without bagging at 0.41772 LB (0.3937154 CV). And another XG with slightly higher CV but lower LB. I'm not using any features based upon the target variable in the above models. I've noticed adding categoricals seem to lower public LB while increasing local CV. Even out of fold meta or encodings of categoricals seem to hurt on public. Maybe it's just me?</p>",
  "messages": [
    {
      "id": "137620",
      "postDate": "10/03/2016 12:28:53",
      "content": "<p>I have a single XG without bagging at 0.41772 LB (0.3937154 CV). And another XG with slightly higher CV but lower LB. I'm not using any features based upon the target variable in the above models. I've noticed adding categoricals seem to lower public LB while increasing local CV. Even out of fold meta or encodings of categoricals seem to hurt on public. Maybe it's just me?</p>",
      "rawMarkdown": "I have a single XG without bagging at 0.41772 LB (0.3937154 CV). And another XG with slightly higher CV but lower LB. I'm not using any features based upon the target variable in the above models. I've noticed adding categoricals seem to lower public LB while increasing local CV. Even out of fold meta or encodings of categoricals seem to hurt on public. Maybe it's just me?",
      "votes": null
    },
    {
      "id": "137624",
      "postDate": "10/03/2016 13:21:38",
      "content": "<p>@Mike Kim : And this is using a 8GB machine as you mentioned in another thread?</p>",
      "rawMarkdown": "Mike Kim : And this is using a 8GB machine as you mentioned in another thread?",
      "votes": null
    },
    {
      "id": "137645",
      "postDate": "10/03/2016 16:22:04",
      "content": "<p>No. My single model does require more RAM (takes less than an hour to run), but it should be possible to get slightly worse results via clever chunking and bagging at the cost of significantly increased runtime.</p>",
      "rawMarkdown": "No. My single model does require more RAM (takes less than an hour to run), but it should be possible to get slightly worse results via clever chunking and bagging at the cost of significantly increased runtime.",
      "votes": null
    },
    {
      "id": "137652",
      "postDate": "10/03/2016 17:18:56",
      "content": "<p>Thank you Mike.! </p>",
      "rawMarkdown": "Thank you Mike.!",
      "votes": null
    },
    {
      "id": "137659",
      "postDate": "10/03/2016 18:26:13",
      "content": "<p>0.4632077 on CV</p>\n\n<p>0.47748 on LB</p>\n\n<p>getting consistently lower CV compared to LB as well :) I have 5 additional features engineered only;</p>",
      "rawMarkdown": "0.4632077 on CV\r\n\r\n0.47748 on LB\r\n\r\ngetting consistently lower CV compared to LB as well :) I have 5 additional features engineered only;",
      "votes": null
    },
    {
      "id": "137672",
      "postDate": "10/03/2016 22:11:44",
      "content": "<p>0.48578 LB, CV is basically identical.</p>\n\n<p>However, that's kfold. I'm not sure what the score is for a single fold but I don't think it would be much lower.</p>",
      "rawMarkdown": "0.48578 LB, CV is basically identical.\r\n\r\nHowever, that's kfold. I'm not sure what the score is for a single fold but I don't think it would be much lower.",
      "votes": null
    },
    {
      "id": "137683",
      "postDate": "10/04/2016 02:00:57",
      "content": "<p>@Mike, I have similar results like yours. I noticed that adding categorical features made my LB score lower. How about your CV AUC score? Mine is about 0.915.</p>",
      "rawMarkdown": "Mike, I have similar results like yours. I noticed that adding categorical features made my LB score lower. How about your CV AUC score? Mine is about 0.915.",
      "votes": null
    },
    {
      "id": "137775",
      "postDate": "10/05/2016 00:24:39",
      "content": "<p>AUC CV is around 0.90X right now for a single model. I might have additional features to push it into the 0.91X range, but these features do not seem to help MCC on the public LB. This is why I'm keeping them on hold in reserve. It's strange that these features that aren't that complicated (no feature looks at the target variable) do not translate to LB at all. It makes me wonder if some features have a different distribution between train and test.</p>",
      "rawMarkdown": "AUC CV is around 0.90X right now for a single model. I might have additional features to push it into the 0.91X range, but these features do not seem to help MCC on the public LB. This is why I'm keeping them on hold in reserve. It's strange that these features that aren't that complicated (no feature looks at the target variable) do not translate to LB at all. It makes me wonder if some features have a different distribution between train and test.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 137624,
      "author_name": "sudalairajkumar",
      "author_url": "",
      "post_date": "10/03/2016 13:21:38",
      "content": "<p>@Mike Kim : And this is using a 8GB machine as you mentioned in another thread?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137645,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "10/03/2016 16:22:04",
      "content": "<p>No. My single model does require more RAM (takes less than an hour to run), but it should be possible to get slightly worse results via clever chunking and bagging at the cost of significantly increased runtime.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137652,
      "author_name": "sudalairajkumar",
      "author_url": "",
      "post_date": "10/03/2016 17:18:56",
      "content": "<p>Thank you Mike.! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137659,
      "author_name": "raddar",
      "author_url": "",
      "post_date": "10/03/2016 18:26:13",
      "content": "<p>0.4632077 on CV</p>\n\n<p>0.47748 on LB</p>\n\n<p>getting consistently lower CV compared to LB as well :) I have 5 additional features engineered only;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137672,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "10/03/2016 22:11:44",
      "content": "<p>0.48578 LB, CV is basically identical.</p>\n\n<p>However, that's kfold. I'm not sure what the score is for a single fold but I don't think it would be much lower.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137683,
      "author_name": "nthanhtam",
      "author_url": "",
      "post_date": "10/04/2016 02:00:57",
      "content": "<p>@Mike, I have similar results like yours. I noticed that adding categorical features made my LB score lower. How about your CV AUC score? Mine is about 0.915.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137775,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "10/05/2016 00:24:39",
      "content": "<p>AUC CV is around 0.90X right now for a single model. I might have additional features to push it into the 0.91X range, but these features do not seem to help MCC on the public LB. This is why I'm keeping them on hold in reserve. It's strange that these features that aren't that complicated (no feature looks at the target variable) do not translate to LB at all. It makes me wonder if some features have a different distribution between train and test.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "137620": "I have a single XG without bagging at 0.41772 LB (0.3937154 CV). And another XG with slightly higher CV but lower LB. I'm not using any features based upon the target variable in the above models. I've noticed adding categoricals seem to lower public LB while increasing local CV. Even out of fold meta or encodings of categoricals seem to hurt on public. Maybe it's just me?",
    "137624": "Mike Kim : And this is using a 8GB machine as you mentioned in another thread?",
    "137645": "No. My single model does require more RAM (takes less than an hour to run), but it should be possible to get slightly worse results via clever chunking and bagging at the cost of significantly increased runtime.",
    "137652": "Thank you Mike.!",
    "137659": "0.4632077 on CV\r\n\r\n0.47748 on LB\r\n\r\ngetting consistently lower CV compared to LB as well :) I have 5 additional features engineered only;",
    "137672": "0.48578 LB, CV is basically identical.\r\n\r\nHowever, that's kfold. I'm not sure what the score is for a single fold but I don't think it would be much lower.",
    "137683": "Mike, I have similar results like yours. I noticed that adding categorical features made my LB score lower. How about your CV AUC score? Mine is about 0.915.",
    "137775": "AUC CV is around 0.90X right now for a single model. I might have additional features to push it into the 0.91X range, but these features do not seem to help MCC on the public LB. This is why I'm keeping them on hold in reserve. It's strange that these features that aren't that complicated (no feature looks at the target variable) do not translate to LB at all. It makes me wonder if some features have a different distribution between train and test."
  },
  "source": "meta"
}