{
  "id": 78178,
  "title": "Basic question on how to improve/debug CV",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/78178",
  "author_name": "samshipengs",
  "post_date": "2019-01-20T20:38:08.108000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Here is the result of my 5 fold cross validation (random_seed set to 42) stratified on outliers-or-not:</p>\n\n<pre><code>rmse: train: 3.6553 | val: 3.6921 | number of outliers: 442\nrmse: train: 3.6638 | val: 3.6584 | number of outliers: 442\nrmse: train: 3.6640 | val: 3.6575 | number of outliers: 441\nrmse: train: 3.6653 | val: 3.6525 | number of outliers: 441\nrmse: train: 3.6652 | val: 3.6529 | number of outliers: 441\n</code></pre>\n\n<p>as you can see, the first fold is clearly worse than the rest 4 folds, I'd like to ask what approaches/techniques do you use when you have this kind of result?</p>\n\n<p>I tried to plot the prediction distribution and difference histogram, see plts.png, but I couldn't find clues based on this.</p>\n\n<p>Or, this is just by chance, somehow there are hard instances (to generalize) in the first fold than the rest, but what makes these instances difficult while the rest folds seems like are ok?</p>",
  "messages": [
    {
      "id": 458926,
      "postDate": "2019-01-20T20:38:08.110Z",
      "content": "<p>Here is the result of my 5 fold cross validation (random_seed set to 42) stratified on outliers-or-not:</p>\n\n<pre><code>rmse: train: 3.6553 | val: 3.6921 | number of outliers: 442\nrmse: train: 3.6638 | val: 3.6584 | number of outliers: 442\nrmse: train: 3.6640 | val: 3.6575 | number of outliers: 441\nrmse: train: 3.6653 | val: 3.6525 | number of outliers: 441\nrmse: train: 3.6652 | val: 3.6529 | number of outliers: 441\n</code></pre>\n\n<p>as you can see, the first fold is clearly worse than the rest 4 folds, I'd like to ask what approaches/techniques do you use when you have this kind of result?</p>\n\n<p>I tried to plot the prediction distribution and difference histogram, see plts.png, but I couldn't find clues based on this.</p>\n\n<p>Or, this is just by chance, somehow there are hard instances (to generalize) in the first fold than the rest, but what makes these instances difficult while the rest folds seems like are ok?</p>",
      "rawMarkdown": "Here is the result of my 5 fold cross validation (random_seed set to 42) stratified on outliers-or-not:\n\n    rmse: train: 3.6553 | val: 3.6921 | number of outliers: 442\n    rmse: train: 3.6638 | val: 3.6584 | number of outliers: 442\n    rmse: train: 3.6640 | val: 3.6575 | number of outliers: 441\n    rmse: train: 3.6653 | val: 3.6525 | number of outliers: 441\n    rmse: train: 3.6652 | val: 3.6529 | number of outliers: 441\n\nas you can see, the first fold is clearly worse than the rest 4 folds, I'd like to ask what approaches/techniques do you use when you have this kind of result?\n\nI tried to plot the prediction distribution and difference histogram, see plts.png, but I couldn't find clues based on this.\n\nOr, this is just by chance, somehow there are hard instances (to generalize) in the first fold than the rest, but what makes these instances difficult while the rest folds seems like are ok?",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "458926": "Here is the result of my 5 fold cross validation (random_seed set to 42) stratified on outliers-or-not:\n\n    rmse: train: 3.6553 | val: 3.6921 | number of outliers: 442\n    rmse: train: 3.6638 | val: 3.6584 | number of outliers: 442\n    rmse: train: 3.6640 | val: 3.6575 | number of outliers: 441\n    rmse: train: 3.6653 | val: 3.6525 | number of outliers: 441\n    rmse: train: 3.6652 | val: 3.6529 | number of outliers: 441\n\nas you can see, the first fold is clearly worse than the rest 4 folds, I'd like to ask what approaches/techniques do you use when you have this kind of result?\n\nI tried to plot the prediction distribution and difference histogram, see plts.png, but I couldn't find clues based on this.\n\nOr, this is just by chance, somehow there are hard instances (to generalize) in the first fold than the rest, but what makes these instances difficult while the rest folds seems like are ok?"
  }
}