{
  "id": 23867,
  "title": "Tunning For MCC",
  "url": "/competitions/bosch-production-line-performance/discussion/23867",
  "author_name": "",
  "post_date": "2016-09-23T22:20:25.280Z",
  "votes": null,
  "comment_count": 4,
  "views": 458,
  "content": "<p>Hi,</p>\n\n<p>I am not sure what is right cross validation strategy for this competition. </p>\n\n<p>I am following one of the kernels where model xbgoost early stopping is done using &quot;auc&quot; as metric and later &quot;mcc&quot; is tuned by trying out different thresholds. </p>\n\n<p>But following this helped me so far, but I am not able use this methods. I get very good &quot;mcc&quot; score on cross validation (0.255+) but on LB it regresses.  My holdout set is 1/6th of the training. </p>\n\n<p>Can anyone suggest why this behavior occurs and other validation strategies. I am thinking of trying different hold out set size and using &quot;mcc&quot; for early stopping. </p>",
  "messages": [
    {
      "id": "136672",
      "postDate": "09/23/2016 22:20:25",
      "content": "<p>Hi,</p>\n\n<p>I am not sure what is right cross validation strategy for this competition. </p>\n\n<p>I am following one of the kernels where model xbgoost early stopping is done using &quot;auc&quot; as metric and later &quot;mcc&quot; is tuned by trying out different thresholds. </p>\n\n<p>But following this helped me so far, but I am not able use this methods. I get very good &quot;mcc&quot; score on cross validation (0.255+) but on LB it regresses.  My holdout set is 1/6th of the training. </p>\n\n<p>Can anyone suggest why this behavior occurs and other validation strategies. I am thinking of trying different hold out set size and using &quot;mcc&quot; for early stopping. </p>",
      "rawMarkdown": "Hi,\r\n\r\nI am not sure what is right cross validation strategy for this competition. \r\n\r\nI am following one of the kernels where model xbgoost early stopping is done using \"auc\" as metric and later \"mcc\" is tuned by trying out different thresholds. \r\n\r\nBut following this helped me so far, but I am not able use this methods. I get very good \"mcc\" score on cross validation (0.255+) but on LB it regresses.  My holdout set is 1/6th of the training. \r\n\r\nCan anyone suggest why this behavior occurs and other validation strategies. I am thinking of trying different hold out set size and using \"mcc\" for early stopping.",
      "votes": null
    },
    {
      "id": "136674",
      "postDate": "09/23/2016 22:30:10",
      "content": "<p>I do not think the issue is your CV, your Local CV of ~0.255 is not that far apart from your LB ~0.239, I think that margin (~0.016) can be expected and acceptable. I have a ~0.01 margin in favor of my CV.</p>\n\n<p>I am also following what the script suggest.</p>",
      "rawMarkdown": "I do not think the issue is your CV, your Local CV of ~0.255 is not that far apart from your LB ~0.239, I think that margin (~0.016) can be expected and acceptable. I have a ~0.01 margin in favor of my CV.\r\n\r\nI am also following what the script suggest.\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
      "votes": null
    },
    {
      "id": "136680",
      "postDate": "09/23/2016 23:06:05",
      "content": "<p>Let me clarify my issue. My best score 0.239 on LB was achieved with CV error 0.2502, but when I submit any models with 0.255-0.259 on CV, LB score is worse than 0.239. </p>\n\n<p>So improving CV is not translating to improvements in LB, which questions my validation strategy. </p>\n\n<p>@NxGTR, do you have same cross validation strategy ? </p>",
      "rawMarkdown": "Let me clarify my issue. My best score 0.239 on LB was achieved with CV error 0.2502, but when I submit any models with 0.255-0.259 on CV, LB score is worse than 0.239. \r\n\r\nSo improving CV is not translating to improvements in LB, which questions my validation strategy. \r\n\r\n@NxGTR, do you have same cross validation strategy ?",
      "votes": null
    },
    {
      "id": "136689",
      "postDate": "09/24/2016 00:27:36",
      "content": "<p>Ohh got it, direction!, is not following, yeah.... that seems an issue, but not caused by your CV strategy, I would blame your feature engineering, maybe some leaking causing improvements.</p>\n\n<p>Yes I do.</p>",
      "rawMarkdown": "Ohh got it, direction!, is not following, yeah.... that seems an issue, but not caused by your CV strategy, I would blame your feature engineering, maybe some leaking causing improvements.\r\n\r\nYes I do.",
      "votes": null
    },
    {
      "id": "136691",
      "postDate": "09/24/2016 00:39:38",
      "content": "<p>Here is more information, two different model parameters with same data. </p>\n\n<p>0.249 CV -&gt; 0.239 LB\n0.254 CV  -&gt; 0.233 LB, attached the chart. </p>\n\n<p>Very strange. </p>\n\n<p>@NxGTR, I am using features as is, just selecting subset.  </p>",
      "rawMarkdown": "Here is more information, two different model parameters with same data. \r\n\r\n0.249 CV -> 0.239 LB\r\n0.254 CV  -> 0.233 LB, attached the chart. \r\n\r\nVery strange. \r\n\r\n@NxGTR, I am using features as is, just selecting subset.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 136674,
      "author_name": "carloshuertas",
      "author_url": "",
      "post_date": "09/23/2016 22:30:10",
      "content": "<p>I do not think the issue is your CV, your Local CV of ~0.255 is not that far apart from your LB ~0.239, I think that margin (~0.016) can be expected and acceptable. I have a ~0.01 margin in favor of my CV.</p>\n\n<p>I am also following what the script suggest.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136680,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "09/23/2016 23:06:05",
      "content": "<p>Let me clarify my issue. My best score 0.239 on LB was achieved with CV error 0.2502, but when I submit any models with 0.255-0.259 on CV, LB score is worse than 0.239. </p>\n\n<p>So improving CV is not translating to improvements in LB, which questions my validation strategy. </p>\n\n<p>@NxGTR, do you have same cross validation strategy ? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136689,
      "author_name": "carloshuertas",
      "author_url": "",
      "post_date": "09/24/2016 00:27:36",
      "content": "<p>Ohh got it, direction!, is not following, yeah.... that seems an issue, but not caused by your CV strategy, I would blame your feature engineering, maybe some leaking causing improvements.</p>\n\n<p>Yes I do.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136691,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "09/24/2016 00:39:38",
      "content": "<p>Here is more information, two different model parameters with same data. </p>\n\n<p>0.249 CV -&gt; 0.239 LB\n0.254 CV  -&gt; 0.233 LB, attached the chart. </p>\n\n<p>Very strange. </p>\n\n<p>@NxGTR, I am using features as is, just selecting subset.  </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "136672": "Hi,\r\n\r\nI am not sure what is right cross validation strategy for this competition. \r\n\r\nI am following one of the kernels where model xbgoost early stopping is done using \"auc\" as metric and later \"mcc\" is tuned by trying out different thresholds. \r\n\r\nBut following this helped me so far, but I am not able use this methods. I get very good \"mcc\" score on cross validation (0.255+) but on LB it regresses.  My holdout set is 1/6th of the training. \r\n\r\nCan anyone suggest why this behavior occurs and other validation strategies. I am thinking of trying different hold out set size and using \"mcc\" for early stopping.",
    "136674": "I do not think the issue is your CV, your Local CV of ~0.255 is not that far apart from your LB ~0.239, I think that margin (~0.016) can be expected and acceptable. I have a ~0.01 margin in favor of my CV.\r\n\r\nI am also following what the script suggest.\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
    "136680": "Let me clarify my issue. My best score 0.239 on LB was achieved with CV error 0.2502, but when I submit any models with 0.255-0.259 on CV, LB score is worse than 0.239. \r\n\r\nSo improving CV is not translating to improvements in LB, which questions my validation strategy. \r\n\r\n@NxGTR, do you have same cross validation strategy ?",
    "136689": "Ohh got it, direction!, is not following, yeah.... that seems an issue, but not caused by your CV strategy, I would blame your feature engineering, maybe some leaking causing improvements.\r\n\r\nYes I do.",
    "136691": "Here is more information, two different model parameters with same data. \r\n\r\n0.249 CV -> 0.239 LB\r\n0.254 CV  -> 0.233 LB, attached the chart. \r\n\r\nVery strange. \r\n\r\n@NxGTR, I am using features as is, just selecting subset."
  },
  "source": "meta"
}