{
  "id": 113568,
  "title": "Best single model",
  "url": "/competitions/ashrae-energy-prediction/discussion/113568",
  "author_name": "",
  "post_date": "2019-10-20T12:56:36.342221700Z",
  "votes": 14,
  "comment_count": 21,
  "views": 0,
  "content": "<p>It would be interesting to know your best single models and their validation performance.  In my case:</p>\n\n<p>Validation: 0.886606\nLB: 1.24.</p>\n\n<p>As we can see, there's quite a lot of overfitting going on but I have other models with higher validation scores, i.e closer to LB, but I haven't submitted them yet.</p>",
  "messages": [
    {
      "id": "653438",
      "postDate": "10/20/2019 12:56:36",
      "content": "<p>It would be interesting to know your best single models and their validation performance.  In my case:</p>\n\n<p>Validation: 0.886606\nLB: 1.24.</p>\n\n<p>As we can see, there's quite a lot of overfitting going on but I have other models with higher validation scores, i.e closer to LB, but I haven't submitted them yet.</p>",
      "rawMarkdown": "It would be interesting to know your best single models and their validation performance.  In my case:\n\nValidation: 0.886606\nLB: 1.24.\n\nAs we can see, there's quite a lot of overfitting going on but I have other models with higher validation scores, i.e closer to LB, but I haven't submitted them yet.",
      "votes": null
    },
    {
      "id": "653448",
      "postDate": "10/20/2019 13:30:49",
      "content": "<p>Validation - 1.18\nLB - 1.29\nWhat is the validation technique are you using?</p>",
      "rawMarkdown": "Validation - 1.18\nLB - 1.29\nWhat is the validation technique are you using?",
      "votes": null
    },
    {
      "id": "653458",
      "postDate": "10/20/2019 13:45:50",
      "content": "<p>K-fold, but I'm getting similar validation scores with random subset or time split. I don't know their correlation with LB however.</p>",
      "rawMarkdown": "K-fold, but I'm getting similar validation scores with random subset or time split. I don't know their correlation with LB however.",
      "votes": null
    },
    {
      "id": "653473",
      "postDate": "10/20/2019 14:17:53",
      "content": "<p>I have a CV score of 0.92009 and an LB score of 1.09. I'm using a standard 3-fold validation. I have some ideas for different validation schemes. However my gut tells me that getting a stable CV/LB score difference is down to using features that generalize well across years. Previous experience tells me that the specifics of the validation scheme don't matter.  </p>",
      "rawMarkdown": "I have a CV score of 0.92009 and an LB score of 1.09. I'm using a standard 3-fold validation. I have some ideas for different validation schemes. However my gut tells me that getting a stable CV/LB score difference is down to using features that generalize well across years. Previous experience tells me that the specifics of the validation scheme don't matter.",
      "votes": null
    },
    {
      "id": "653488",
      "postDate": "10/20/2019 14:42:03",
      "content": "<p>Which model are you using <a href=\"/maxhalford\">@maxhalford</a>?</p>",
      "rawMarkdown": "Which model are you using @maxhalford?",
      "votes": null
    },
    {
      "id": "653489",
      "postDate": "10/20/2019 14:42:08",
      "content": "<p>Validation: 1.25\nLB: 1.25</p>",
      "rawMarkdown": "Validation: 1.25\nLB: 1.25",
      "votes": null
    },
    {
      "id": "653490",
      "postDate": "10/20/2019 14:42:43",
      "content": "<p>Wow. What's your validation technique?</p>",
      "rawMarkdown": "Wow. What's your validation technique?",
      "votes": null
    },
    {
      "id": "653509",
      "postDate": "10/20/2019 15:08:53",
      "content": "<p>LightGBM with default parameters, 300 trees, and some decent features.</p>",
      "rawMarkdown": "LightGBM with default parameters, 300 trees, and some decent features.",
      "votes": null
    },
    {
      "id": "653526",
      "postDate": "10/20/2019 15:26:05",
      "content": "<p>Thanks! That's great.</p>",
      "rawMarkdown": "Thanks! That's great.",
      "votes": null
    },
    {
      "id": "653596",
      "postDate": "10/20/2019 17:37:02",
      "content": "<p><a href=\"/themonologue\">@themonologue</a> nice try! hahaha</p>",
      "rawMarkdown": "themonologue nice try! hahaha",
      "votes": null
    },
    {
      "id": "653951",
      "postDate": "10/21/2019 07:54:40",
      "content": "<p>Validation: 1.29\nLB: 1.18</p>",
      "rawMarkdown": "Validation: 1.29\nLB: 1.18",
      "votes": null
    },
    {
      "id": "654034",
      "postDate": "10/21/2019 10:49:04",
      "content": "<p><a href=\"/maxhalford\">@maxhalford</a>  This is great</p>",
      "rawMarkdown": "maxhalford  This is great",
      "votes": null
    },
    {
      "id": "654952",
      "postDate": "10/22/2019 14:28:49",
      "content": "<p><a href=\"/maxhalford\">@maxhalford</a> did you train on whole trainset or on a sample of it?</p>",
      "rawMarkdown": "maxhalford did you train on whole trainset or on a sample of it?",
      "votes": null
    },
    {
      "id": "655179",
      "postDate": "10/22/2019 19:04:07",
      "content": "<p>As we all know K-Fold might not be the right choice for a time serie... but who knows ?</p>\n\n<p>I would bet on a LTSM.</p>",
      "rawMarkdown": "As we all know K-Fold might not be the right choice for a time serie... but who knows ?\n\nI would bet on a LTSM.",
      "votes": null
    },
    {
      "id": "661141",
      "postDate": "10/30/2019 00:35:42",
      "content": "<p>Congrats <a href=\"/maxhalford\">@maxhalford</a>  ! Have you ever try used different models for different types of measure ?</p>",
      "rawMarkdown": "Congrats @maxhalford  ! Have you ever try used different models for different types of measure ?",
      "votes": null
    },
    {
      "id": "661143",
      "postDate": "10/30/2019 00:38:17",
      "content": "<p><a href=\"/jesucristo\">@jesucristo</a> . what's your validation technique?</p>",
      "rawMarkdown": "jesucristo . what's your validation technique?",
      "votes": null
    },
    {
      "id": "661159",
      "postDate": "10/30/2019 00:54:33",
      "content": "<p>Validation: 1.0111\nLB: 1.23</p>",
      "rawMarkdown": "Validation: 1.0111\nLB: 1.23",
      "votes": null
    },
    {
      "id": "661759",
      "postDate": "10/30/2019 17:14:53",
      "content": "<p>0.99 cv 1.08 lb</p>",
      "rawMarkdown": "0.99 cv 1.08 lb",
      "votes": null
    },
    {
      "id": "661761",
      "postDate": "10/30/2019 17:19:27",
      "content": "<p>Great <a href=\"/oleg90777\">@oleg90777</a>  !  What's your validation technique?</p>",
      "rawMarkdown": "Great @oleg90777  !  What's your validation technique?",
      "votes": null
    },
    {
      "id": "661763",
      "postDate": "10/30/2019 17:19:45",
      "content": "<p>What's your validation technique <a href=\"/kimtaegwan\">@kimtaegwan</a> ?</p>",
      "rawMarkdown": "What's your validation technique @kimtaegwan ?",
      "votes": null
    },
    {
      "id": "661856",
      "postDate": "10/30/2019 19:30:53",
      "content": "<p>Validation: 1.05\nLB: 1.10</p>",
      "rawMarkdown": "Validation: 1.05\nLB: 1.10",
      "votes": null
    },
    {
      "id": "661901",
      "postDate": "10/30/2019 20:47:36",
      "content": "<p>Impressive !</p>\n\n<p>But no technique, just fact :-)</p>",
      "rawMarkdown": "Impressive !\n\nBut no technique, just fact :-)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 653448,
      "author_name": "themonologue",
      "author_url": "",
      "post_date": "10/20/2019 13:30:49",
      "content": "<p>Validation - 1.18\nLB - 1.29\nWhat is the validation technique are you using?</p>",
      "votes": null,
      "replies": [
        {
          "id": 653458,
          "author_name": "frjo0083",
          "author_url": "",
          "post_date": "10/20/2019 13:45:50",
          "content": "<p>K-fold, but I'm getting similar validation scores with random subset or time split. I don't know their correlation with LB however.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 655179,
          "author_name": "pfichou",
          "author_url": "",
          "post_date": "10/22/2019 19:04:07",
          "content": "<p>As we all know K-Fold might not be the right choice for a time serie... but who knows ?</p>\n\n<p>I would bet on a LTSM.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 653473,
      "author_name": "maxhalford",
      "author_url": "",
      "post_date": "10/20/2019 14:17:53",
      "content": "<p>I have a CV score of 0.92009 and an LB score of 1.09. I'm using a standard 3-fold validation. I have some ideas for different validation schemes. However my gut tells me that getting a stable CV/LB score difference is down to using features that generalize well across years. Previous experience tells me that the specifics of the validation scheme don't matter.  </p>",
      "votes": null,
      "replies": [
        {
          "id": 653488,
          "author_name": "themonologue",
          "author_url": "",
          "post_date": "10/20/2019 14:42:03",
          "content": "<p>Which model are you using <a href=\"/maxhalford\">@maxhalford</a>?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 653509,
          "author_name": "maxhalford",
          "author_url": "",
          "post_date": "10/20/2019 15:08:53",
          "content": "<p>LightGBM with default parameters, 300 trees, and some decent features.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 653526,
          "author_name": "themonologue",
          "author_url": "",
          "post_date": "10/20/2019 15:26:05",
          "content": "<p>Thanks! That's great.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 654034,
          "author_name": "manovirat",
          "author_url": "",
          "post_date": "10/21/2019 10:49:04",
          "content": "<p><a href=\"/maxhalford\">@maxhalford</a>  This is great</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 654952,
          "author_name": "kocman",
          "author_url": "",
          "post_date": "10/22/2019 14:28:49",
          "content": "<p><a href=\"/maxhalford\">@maxhalford</a> did you train on whole trainset or on a sample of it?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 661141,
          "author_name": "caesarlupum",
          "author_url": "",
          "post_date": "10/30/2019 00:35:42",
          "content": "<p>Congrats <a href=\"/maxhalford\">@maxhalford</a>  ! Have you ever try used different models for different types of measure ?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 653489,
      "author_name": "aerdem4",
      "author_url": "",
      "post_date": "10/20/2019 14:42:08",
      "content": "<p>Validation: 1.25\nLB: 1.25</p>",
      "votes": null,
      "replies": [
        {
          "id": 653490,
          "author_name": "themonologue",
          "author_url": "",
          "post_date": "10/20/2019 14:42:43",
          "content": "<p>Wow. What's your validation technique?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 653596,
          "author_name": "jesucristo",
          "author_url": "",
          "post_date": "10/20/2019 17:37:02",
          "content": "<p><a href=\"/themonologue\">@themonologue</a> nice try! hahaha</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 661143,
          "author_name": "caesarlupum",
          "author_url": "",
          "post_date": "10/30/2019 00:38:17",
          "content": "<p><a href=\"/jesucristo\">@jesucristo</a> . what's your validation technique?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 653951,
      "author_name": "kimtaegwan",
      "author_url": "",
      "post_date": "10/21/2019 07:54:40",
      "content": "<p>Validation: 1.29\nLB: 1.18</p>",
      "votes": null,
      "replies": [
        {
          "id": 661763,
          "author_name": "caesarlupum",
          "author_url": "",
          "post_date": "10/30/2019 17:19:45",
          "content": "<p>What's your validation technique <a href=\"/kimtaegwan\">@kimtaegwan</a> ?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 661159,
      "author_name": "caesarlupum",
      "author_url": "",
      "post_date": "10/30/2019 00:54:33",
      "content": "<p>Validation: 1.0111\nLB: 1.23</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 661759,
      "author_name": "oleg90777",
      "author_url": "",
      "post_date": "10/30/2019 17:14:53",
      "content": "<p>0.99 cv 1.08 lb</p>",
      "votes": null,
      "replies": [
        {
          "id": 661761,
          "author_name": "caesarlupum",
          "author_url": "",
          "post_date": "10/30/2019 17:19:27",
          "content": "<p>Great <a href=\"/oleg90777\">@oleg90777</a>  !  What's your validation technique?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 661856,
      "author_name": "labdmitriy",
      "author_url": "",
      "post_date": "10/30/2019 19:30:53",
      "content": "<p>Validation: 1.05\nLB: 1.10</p>",
      "votes": null,
      "replies": [
        {
          "id": 661901,
          "author_name": "pfichou",
          "author_url": "",
          "post_date": "10/30/2019 20:47:36",
          "content": "<p>Impressive !</p>\n\n<p>But no technique, just fact :-)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "653438": "It would be interesting to know your best single models and their validation performance.  In my case:\n\nValidation: 0.886606\nLB: 1.24.\n\nAs we can see, there's quite a lot of overfitting going on but I have other models with higher validation scores, i.e closer to LB, but I haven't submitted them yet.",
    "653448": "Validation - 1.18\nLB - 1.29\nWhat is the validation technique are you using?",
    "653458": "K-fold, but I'm getting similar validation scores with random subset or time split. I don't know their correlation with LB however.",
    "653473": "I have a CV score of 0.92009 and an LB score of 1.09. I'm using a standard 3-fold validation. I have some ideas for different validation schemes. However my gut tells me that getting a stable CV/LB score difference is down to using features that generalize well across years. Previous experience tells me that the specifics of the validation scheme don't matter.",
    "653488": "Which model are you using @maxhalford?",
    "653489": "Validation: 1.25\nLB: 1.25",
    "653490": "Wow. What's your validation technique?",
    "653509": "LightGBM with default parameters, 300 trees, and some decent features.",
    "653526": "Thanks! That's great.",
    "653596": "themonologue nice try! hahaha",
    "653951": "Validation: 1.29\nLB: 1.18",
    "654034": "maxhalford  This is great",
    "654952": "maxhalford did you train on whole trainset or on a sample of it?",
    "655179": "As we all know K-Fold might not be the right choice for a time serie... but who knows ?\n\nI would bet on a LTSM.",
    "661141": "Congrats @maxhalford  ! Have you ever try used different models for different types of measure ?",
    "661143": "jesucristo . what's your validation technique?",
    "661159": "Validation: 1.0111\nLB: 1.23",
    "661759": "0.99 cv 1.08 lb",
    "661761": "Great @oleg90777  !  What's your validation technique?",
    "661763": "What's your validation technique @kimtaegwan ?",
    "661856": "Validation: 1.05\nLB: 1.10",
    "661901": "Impressive !\n\nBut no technique, just fact :-)"
  },
  "source": "meta"
}