{
  "id": 73466,
  "title": "3 scenarios for submission selection",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/73466",
  "author_name": "",
  "post_date": "2018-12-03T12:32:56.836946200Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>About submission selection strategy given the high uncertainty,  in my view there are <strong>three most plausible scenarios for our models</strong> :</p>\n\n<ul>\n<li><p>1) Models are worthless. In that scenario best strategy is all zeros.</p></li>\n<li><p>2) Models are on average good enough to predict individual probabilities. In that scenario submit your best model prediction.</p></li>\n<li><p>3) Model fails on average at individual level, but still is able to get right <strong>aggregate transactions</strong>. In that scenario your best bet is submitting one single value, this is, the mean of your model predictions. (It will beat all  zeros and be as good as it can be, remember here rmse is evaluated what means that best estimator is mean of the -unknown- transactions for the target period.</p></li>\n</ul>\n\n<p>In my case I considered wise to bet on both 2 and 3 for my best model. That way you bet on the same best model, but account for two levels of succes, individual-wise and aggregate-wise.</p>",
  "messages": [
    {
      "id": "432113",
      "postDate": "12/03/2018 12:32:56",
      "content": "<p>About submission selection strategy given the high uncertainty,  in my view there are <strong>three most plausible scenarios for our models</strong> :</p>\n\n<ul>\n<li><p>1) Models are worthless. In that scenario best strategy is all zeros.</p></li>\n<li><p>2) Models are on average good enough to predict individual probabilities. In that scenario submit your best model prediction.</p></li>\n<li><p>3) Model fails on average at individual level, but still is able to get right <strong>aggregate transactions</strong>. In that scenario your best bet is submitting one single value, this is, the mean of your model predictions. (It will beat all  zeros and be as good as it can be, remember here rmse is evaluated what means that best estimator is mean of the -unknown- transactions for the target period.</p></li>\n</ul>\n\n<p>In my case I considered wise to bet on both 2 and 3 for my best model. That way you bet on the same best model, but account for two levels of succes, individual-wise and aggregate-wise.</p>",
      "rawMarkdown": "About submission selection strategy given the high uncertainty,  in my view there are **three most plausible scenarios for our models** :\n\n- 1) Models are worthless. In that scenario best strategy is all zeros.\n\n- 2) Models are on average good enough to predict individual probabilities. In that scenario submit your best model prediction.\n\n- 3) Model fails on average at individual level, but still is able to get right **aggregate transactions**. In that scenario your best bet is submitting one single value, this is, the mean of your model predictions. (It will beat all  zeros and be as good as it can be, remember here rmse is evaluated what means that best estimator is mean of the -unknown- transactions for the target period.\n\nIn my case I considered wise to bet on both 2 and 3 for my best model. That way you bet on the same best model, but account for two levels of succes, individual-wise and aggregate-wise.",
      "votes": null
    },
    {
      "id": "432370",
      "postDate": "12/03/2018 19:24:45",
      "content": "<p>Or the middle ground. Bet one mean for past purchasers and one mean for everyone else. </p>",
      "rawMarkdown": "Or the middle ground. Bet one mean for past purchasers and one mean for everyone else.",
      "votes": null
    },
    {
      "id": "432401",
      "postDate": "12/03/2018 20:09:22",
      "content": "<p>I believe that the winning solution will be from 2) but within that it may have include strong safeguards to limit overfitting. Choosing a minimal value for 3) should beat 1). Even a severely underfitting 2) I believe can be found that is at least marginally better than 3).</p>",
      "rawMarkdown": "I believe that the winning solution will be from 2) but within that it may have include strong safeguards to limit overfitting. Choosing a minimal value for 3) should beat 1). Even a severely underfitting 2) I believe can be found that is at least marginally better than 3).",
      "votes": null
    },
    {
      "id": "432545",
      "postDate": "12/04/2018 03:04:00",
      "content": "<p>Due to the unpredictable nature of the data and the extreme unbalance positive and negative sample ratio, I believe the most important thing is to make the distribution of the validation set and training set(which should probably be resampled) similar to the real test set.  Mean value, which is depend on the validation scheme and resampling of the training set, should result in much better ranking than all zeros submission. And my model can only predict like 1% better RMSE than an accurate mean for my validation set. So I basically did the same as you to submit both 2 and 3 (a time series prediction of mean for the coming months). But I believe the mean in 3 for the coming value is still very unpredictable, so I believe 2 will do better.</p>\n\n<p>So 2 &gt; 3 &gt; 1</p>",
      "rawMarkdown": "Due to the unpredictable nature of the data and the extreme unbalance positive and negative sample ratio, I believe the most important thing is to make the distribution of the validation set and training set(which should probably be resampled) similar to the real test set.  Mean value, which is depend on the validation scheme and resampling of the training set, should result in much better ranking than all zeros submission. And my model can only predict like 1% better RMSE than an accurate mean for my validation set. So I basically did the same as you to submit both 2 and 3 (a time series prediction of mean for the coming months). But I believe the mean in 3 for the coming value is still very unpredictable, so I believe 2 will do better.\n\nSo 2 &gt; 3 &gt; 1",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 432370,
      "author_name": "burritodan",
      "author_url": "",
      "post_date": "12/03/2018 19:24:45",
      "content": "<p>Or the middle ground. Bet one mean for past purchasers and one mean for everyone else. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 432401,
      "author_name": "datahobbit",
      "author_url": "",
      "post_date": "12/03/2018 20:09:22",
      "content": "<p>I believe that the winning solution will be from 2) but within that it may have include strong safeguards to limit overfitting. Choosing a minimal value for 3) should beat 1). Even a severely underfitting 2) I believe can be found that is at least marginally better than 3).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 432545,
      "author_name": "zzhhrr001",
      "author_url": "",
      "post_date": "12/04/2018 03:04:00",
      "content": "<p>Due to the unpredictable nature of the data and the extreme unbalance positive and negative sample ratio, I believe the most important thing is to make the distribution of the validation set and training set(which should probably be resampled) similar to the real test set.  Mean value, which is depend on the validation scheme and resampling of the training set, should result in much better ranking than all zeros submission. And my model can only predict like 1% better RMSE than an accurate mean for my validation set. So I basically did the same as you to submit both 2 and 3 (a time series prediction of mean for the coming months). But I believe the mean in 3 for the coming value is still very unpredictable, so I believe 2 will do better.</p>\n\n<p>So 2 &gt; 3 &gt; 1</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "432113": "About submission selection strategy given the high uncertainty,  in my view there are **three most plausible scenarios for our models** :\n\n- 1) Models are worthless. In that scenario best strategy is all zeros.\n\n- 2) Models are on average good enough to predict individual probabilities. In that scenario submit your best model prediction.\n\n- 3) Model fails on average at individual level, but still is able to get right **aggregate transactions**. In that scenario your best bet is submitting one single value, this is, the mean of your model predictions. (It will beat all  zeros and be as good as it can be, remember here rmse is evaluated what means that best estimator is mean of the -unknown- transactions for the target period.\n\nIn my case I considered wise to bet on both 2 and 3 for my best model. That way you bet on the same best model, but account for two levels of succes, individual-wise and aggregate-wise.",
    "432370": "Or the middle ground. Bet one mean for past purchasers and one mean for everyone else.",
    "432401": "I believe that the winning solution will be from 2) but within that it may have include strong safeguards to limit overfitting. Choosing a minimal value for 3) should beat 1). Even a severely underfitting 2) I believe can be found that is at least marginally better than 3).",
    "432545": "Due to the unpredictable nature of the data and the extreme unbalance positive and negative sample ratio, I believe the most important thing is to make the distribution of the validation set and training set(which should probably be resampled) similar to the real test set.  Mean value, which is depend on the validation scheme and resampling of the training set, should result in much better ranking than all zeros submission. And my model can only predict like 1% better RMSE than an accurate mean for my validation set. So I basically did the same as you to submit both 2 and 3 (a time series prediction of mean for the coming months). But I believe the mean in 3 for the coming value is still very unpredictable, so I believe 2 will do better.\n\nSo 2 &gt; 3 &gt; 1"
  },
  "source": "meta"
}