{
  "id": 304262,
  "title": "Strategies for choosing final solution",
  "url": "/competitions/g-research-crypto-forecasting/discussion/304262",
  "author_name": "",
  "post_date": "2022-01-31T12:36:34.392930100Z",
  "votes": 7,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Seems like </p>\n<ul>\n<li>in cross-validation volatility of scores between folds is quite high</li>\n<li>it's especially easy to overfit your model in this competition</li>\n<li>there are a possibility of a significant perturbation on a leaderboard (not only including test data in train set, but due to a nature of a task, and leaderboard calculated for summer 2022 may be quite different then for spring 2022)</li>\n</ul>\n<p>So what do you think is the best strategy for choosing the final solution:</p>\n<ol>\n<li>Select models that have a higher average cross-validation score. Seems reasonable if you want to optimize your \"expected place\" on the final leaderboard.</li>\n<li>Select the model with the highest maximum fold score (but not trying to optimize parameters for this particular fold). It's a risky approach, but it may improve a chance to show really good results. And if it will be no luck - no big deal (we are not betting our money).</li>\n<li>Some really complex strategy that involves comparing your approach with a bunch of other strategies. </li>\n</ol>\n<p>or some other variants?</p>",
  "messages": [
    {
      "id": "1670335",
      "postDate": "01/31/2022 12:36:34",
      "content": "<p>Seems like </p>\n<ul>\n<li>in cross-validation volatility of scores between folds is quite high</li>\n<li>it's especially easy to overfit your model in this competition</li>\n<li>there are a possibility of a significant perturbation on a leaderboard (not only including test data in train set, but due to a nature of a task, and leaderboard calculated for summer 2022 may be quite different then for spring 2022)</li>\n</ul>\n<p>So what do you think is the best strategy for choosing the final solution:</p>\n<ol>\n<li>Select models that have a higher average cross-validation score. Seems reasonable if you want to optimize your \"expected place\" on the final leaderboard.</li>\n<li>Select the model with the highest maximum fold score (but not trying to optimize parameters for this particular fold). It's a risky approach, but it may improve a chance to show really good results. And if it will be no luck - no big deal (we are not betting our money).</li>\n<li>Some really complex strategy that involves comparing your approach with a bunch of other strategies. </li>\n</ol>\n<p>or some other variants?</p>",
      "rawMarkdown": "Seems like \n* in cross-validation volatility of scores between folds is quite high\n* it's especially easy to overfit your model in this competition\n* there are a possibility of a significant perturbation on a leaderboard (not only including test data in train set, but due to a nature of a task, and leaderboard calculated for summer 2022 may be quite different then for spring 2022)\n\nSo what do you think is the best strategy for choosing the final solution:\n1. Select models that have a higher average cross-validation score. Seems reasonable if you want to optimize your \"expected place\" on the final leaderboard.\n2. Select the model with the highest maximum fold score (but not trying to optimize parameters for this particular fold). It's a risky approach, but it may improve a chance to show really good results. And if it will be no luck - no big deal (we are not betting our money).\n3. Some really complex strategy that involves comparing your approach with a bunch of other strategies. \n\nor some other variants?",
      "votes": null
    },
    {
      "id": "1670696",
      "postDate": "01/31/2022 19:04:21",
      "content": "<p>I'll let you know in May</p>",
      "rawMarkdown": "I'll let you know in May",
      "votes": null
    },
    {
      "id": "1671001",
      "postDate": "02/01/2022 04:35:42",
      "content": "<p>Hi! <a href=\"https://www.kaggle.com/yrtchn\" target=\"_blank\">@yrtchn</a>, I think I run out of time to continue implementing things. I will go with the simpler approach that I have built.</p>",
      "rawMarkdown": "Hi! @yrtchn, I think I run out of time to continue implementing things. I will go with the simpler approach that I have built.",
      "votes": null
    },
    {
      "id": "1671232",
      "postDate": "02/01/2022 09:28:23",
      "content": "<p>The extent of my ambition? Choose a model that will neither time out nor throw an error.</p>",
      "rawMarkdown": "The extent of my ambition? Choose a model that will neither time out nor throw an error.",
      "votes": null
    },
    {
      "id": "1671261",
      "postDate": "02/01/2022 10:04:24",
      "content": "<p>I use a bit more complex approach - also throw \"out of memory\" solutions.</p>",
      "rawMarkdown": "I use a bit more complex approach - also throw \"out of memory\" solutions.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1670696,
      "author_name": "julianmukaj",
      "author_url": "",
      "post_date": "01/31/2022 19:04:21",
      "content": "<p>I'll let you know in May</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1671001,
      "author_name": "cv13j0",
      "author_url": "",
      "post_date": "02/01/2022 04:35:42",
      "content": "<p>Hi! <a href=\"https://www.kaggle.com/yrtchn\" target=\"_blank\">@yrtchn</a>, I think I run out of time to continue implementing things. I will go with the simpler approach that I have built.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1671232,
      "author_name": "jbomitchell",
      "author_url": "",
      "post_date": "02/01/2022 09:28:23",
      "content": "<p>The extent of my ambition? Choose a model that will neither time out nor throw an error.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1671261,
          "author_name": "yrtchn",
          "author_url": "",
          "post_date": "02/01/2022 10:04:24",
          "content": "<p>I use a bit more complex approach - also throw \"out of memory\" solutions.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1670335": "Seems like \n* in cross-validation volatility of scores between folds is quite high\n* it's especially easy to overfit your model in this competition\n* there are a possibility of a significant perturbation on a leaderboard (not only including test data in train set, but due to a nature of a task, and leaderboard calculated for summer 2022 may be quite different then for spring 2022)\n\nSo what do you think is the best strategy for choosing the final solution:\n1. Select models that have a higher average cross-validation score. Seems reasonable if you want to optimize your \"expected place\" on the final leaderboard.\n2. Select the model with the highest maximum fold score (but not trying to optimize parameters for this particular fold). It's a risky approach, but it may improve a chance to show really good results. And if it will be no luck - no big deal (we are not betting our money).\n3. Some really complex strategy that involves comparing your approach with a bunch of other strategies. \n\nor some other variants?",
    "1670696": "I'll let you know in May",
    "1671001": "Hi! @yrtchn, I think I run out of time to continue implementing things. I will go with the simpler approach that I have built.",
    "1671232": "The extent of my ambition? Choose a model that will neither time out nor throw an error.",
    "1671261": "I use a bit more complex approach - also throw \"out of memory\" solutions."
  },
  "source": "meta"
}