{
  "id": 116878,
  "title": "A stable Down sampling approach (and other findings)",
  "url": "/competitions/ashrae-energy-prediction/discussion/116878",
  "author_name": "",
  "post_date": "2019-11-12T02:37:35.484141300Z",
  "votes": 23,
  "comment_count": 6,
  "views": 0,
  "content": "<p>As I lost interest in this competition, I decided to share my findings before quitting. Please find my findings so far:</p>\n\n<ol>\n<li>Downsampling method\nThe training set is large (around 20 million). However, you really don't need to use all the data to train the models. I down sampled the training set from 20 million to 1 million and found that the models fitted in 1 million data always performed within 0.01 gap in LB, compared to their counterparts fitted in the whole data.</li>\n</ol>\n\n<p>The downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.</p>\n\n<ol>\n<li><p>Validation method\nUnshuffle KFold with K = 2. When I improved my LB from 1.13 to 1.10 by adding engineering features, improvements in local CV always led to improvements in LB. I believe it's a stable validation approch.</p></li>\n<li><p>Some useful features:</p>\n\n<ul><li>'hour': self-explanatory</li>\n<li>'weekday': self-explanatory</li>\n<li>'day_in_month': self-explanatory</li>\n<li>'is_weekend': self-explanatory</li>\n<li>'building_id_count_full': count encoding of building_id</li>\n<li>'year_built_count_full': count encoding of year_built</li>\n<li>'site_id_cloud_coverage_48_mean': mean value of cloud_coverage in the past 48 hours for each site.</li></ul></li>\n<li><p>Data cleaning\nAs I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)...Seems the data may be not clean for the first 6 months of 2016.</p></li>\n</ol>\n\n<p>Hopes the findings can help and wish everyone a piece of good luck!!</p>",
  "messages": [
    {
      "id": "670900",
      "postDate": "11/12/2019 02:37:35",
      "content": "<p>As I lost interest in this competition, I decided to share my findings before quitting. Please find my findings so far:</p>\n\n<ol>\n<li>Downsampling method\nThe training set is large (around 20 million). However, you really don't need to use all the data to train the models. I down sampled the training set from 20 million to 1 million and found that the models fitted in 1 million data always performed within 0.01 gap in LB, compared to their counterparts fitted in the whole data.</li>\n</ol>\n\n<p>The downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.</p>\n\n<ol>\n<li><p>Validation method\nUnshuffle KFold with K = 2. When I improved my LB from 1.13 to 1.10 by adding engineering features, improvements in local CV always led to improvements in LB. I believe it's a stable validation approch.</p></li>\n<li><p>Some useful features:</p>\n\n<ul><li>'hour': self-explanatory</li>\n<li>'weekday': self-explanatory</li>\n<li>'day_in_month': self-explanatory</li>\n<li>'is_weekend': self-explanatory</li>\n<li>'building_id_count_full': count encoding of building_id</li>\n<li>'year_built_count_full': count encoding of year_built</li>\n<li>'site_id_cloud_coverage_48_mean': mean value of cloud_coverage in the past 48 hours for each site.</li></ul></li>\n<li><p>Data cleaning\nAs I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)...Seems the data may be not clean for the first 6 months of 2016.</p></li>\n</ol>\n\n<p>Hopes the findings can help and wish everyone a piece of good luck!!</p>",
      "rawMarkdown": "As I lost interest in this competition, I decided to share my findings before quitting. Please find my findings so far:\n\n1. Downsampling method\nThe training set is large (around 20 million). However, you really don't need to use all the data to train the models. I down sampled the training set from 20 million to 1 million and found that the models fitted in 1 million data always performed within 0.01 gap in LB, compared to their counterparts fitted in the whole data.\n\nThe downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.\n\n2. Validation method\nUnshuffle KFold with K = 2. When I improved my LB from 1.13 to 1.10 by adding engineering features, improvements in local CV always led to improvements in LB. I believe it's a stable validation approch.\n\n3. Some useful features:\n - 'hour': self-explanatory\n - 'weekday': self-explanatory\n -  'day_in_month': self-explanatory\n - 'is_weekend': self-explanatory\n - 'building_id_count_full': count encoding of building_id\n - 'year_built_count_full': count encoding of year_built\n - 'site_id_cloud_coverage_48_mean': mean value of cloud_coverage in the past 48 hours for each site.\n\n4. Data cleaning\nAs I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)...Seems the data may be not clean for the first 6 months of 2016.\n\nHopes the findings can help and wish everyone a piece of good luck!!",
      "votes": null
    },
    {
      "id": "670925",
      "postDate": "11/12/2019 03:58:03",
      "content": "<p>why did you lose interest? to me it seems like there's still a lot to explore</p>",
      "rawMarkdown": "why did you lose interest? to me it seems like there's still a lot to explore",
      "votes": null
    },
    {
      "id": "670944",
      "postDate": "11/12/2019 04:16:22",
      "content": "<p><code>As I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)</code></p>\n\n<p>The second fold is closer to the Public LB Data (w.r.t. time). Could that be the reason?</p>\n\n<p><code>The downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.</code></p>\n\n<p>It would be helpful if you provide some more details about the down sampling methodology (not only for this competition, but similar problem set as well). </p>",
      "rawMarkdown": "`As I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)`\n\nThe second fold is closer to the Public LB Data (w.r.t. time). Could that be the reason?\n\n`The downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.`\n\nIt would be helpful if you provide some more details about the down sampling methodology (not only for this competition, but similar problem set as well).",
      "votes": null
    },
    {
      "id": "670978",
      "postDate": "11/12/2019 05:33:55",
      "content": "<p>I think he meant the second fold validation score not the LB.  The second fold is trained then used to predict the first fold that is further away from the LB. I also have the same behavior on mine. I agree that it's probably caused by the outliers/missing data in the first 6 months. I only removed the 0s from site 1 from Jan to May 20th.  There's a lot of other buildings that had 0s in the beginning. Also the buildings that were in mid of constructions in 2016, mentioned in this thread. <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254#latest-667876\">https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254#latest-667876</a></p>",
      "rawMarkdown": "I think he meant the second fold validation score not the LB.  The second fold is trained then used to predict the first fold that is further away from the LB. I also have the same behavior on mine. I agree that it's probably caused by the outliers/missing data in the first 6 months. I only removed the 0s from site 1 from Jan to May 20th.  There's a lot of other buildings that had 0s in the beginning. Also the buildings that were in mid of constructions in 2016, mentioned in this thread. https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254#latest-667876",
      "votes": null
    },
    {
      "id": "670982",
      "postDate": "11/12/2019 05:38:04",
      "content": "<p>Thanks for sharing, especially the downsampling method, will try to test.\nProbably don't need to give up, come back in a few more weeks, see if there's anything new. From what I understand, all the external data people found need to be published here if they want to use them. </p>",
      "rawMarkdown": "Thanks for sharing, especially the downsampling method, will try to test.\nProbably don't need to give up, come back in a few more weeks, see if there's anything new. From what I understand, all the external data people found need to be published here if they want to use them.",
      "votes": null
    },
    {
      "id": "670990",
      "postDate": "11/12/2019 05:47:50",
      "content": "<p>Makes sense. Yes, the outliers and missing data would be a problem for sure.</p>\n\n<p>Another thing could be related to features engineered. If the features are more suitable to predict into the future, those will perform poorer while validating the past.</p>",
      "rawMarkdown": "Makes sense. Yes, the outliers and missing data would be a problem for sure.\n\nAnother thing could be related to features engineered. If the features are more suitable to predict into the future, those will perform poorer while validating the past.",
      "votes": null
    },
    {
      "id": "671873",
      "postDate": "11/13/2019 09:03:24",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 670925,
      "author_name": "lewington",
      "author_url": "",
      "post_date": "11/12/2019 03:58:03",
      "content": "<p>why did you lose interest? to me it seems like there's still a lot to explore</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 670944,
      "author_name": "arnabbiswas1",
      "author_url": "",
      "post_date": "11/12/2019 04:16:22",
      "content": "<p><code>As I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)</code></p>\n\n<p>The second fold is closer to the Public LB Data (w.r.t. time). Could that be the reason?</p>\n\n<p><code>The downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.</code></p>\n\n<p>It would be helpful if you provide some more details about the down sampling methodology (not only for this competition, but similar problem set as well). </p>",
      "votes": null,
      "replies": [
        {
          "id": 670978,
          "author_name": "tuqiuchiaa",
          "author_url": "",
          "post_date": "11/12/2019 05:33:55",
          "content": "<p>I think he meant the second fold validation score not the LB.  The second fold is trained then used to predict the first fold that is further away from the LB. I also have the same behavior on mine. I agree that it's probably caused by the outliers/missing data in the first 6 months. I only removed the 0s from site 1 from Jan to May 20th.  There's a lot of other buildings that had 0s in the beginning. Also the buildings that were in mid of constructions in 2016, mentioned in this thread. <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254#latest-667876\">https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254#latest-667876</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 670990,
          "author_name": "arnabbiswas1",
          "author_url": "",
          "post_date": "11/12/2019 05:47:50",
          "content": "<p>Makes sense. Yes, the outliers and missing data would be a problem for sure.</p>\n\n<p>Another thing could be related to features engineered. If the features are more suitable to predict into the future, those will perform poorer while validating the past.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 670982,
      "author_name": "tuqiuchiaa",
      "author_url": "",
      "post_date": "11/12/2019 05:38:04",
      "content": "<p>Thanks for sharing, especially the downsampling method, will try to test.\nProbably don't need to give up, come back in a few more weeks, see if there's anything new. From what I understand, all the external data people found need to be published here if they want to use them. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 671873,
      "author_name": "lewington",
      "author_url": "",
      "post_date": "11/13/2019 09:03:24",
      "content": "",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "670900": "As I lost interest in this competition, I decided to share my findings before quitting. Please find my findings so far:\n\n1. Downsampling method\nThe training set is large (around 20 million). However, you really don't need to use all the data to train the models. I down sampled the training set from 20 million to 1 million and found that the models fitted in 1 million data always performed within 0.01 gap in LB, compared to their counterparts fitted in the whole data.\n\nThe downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.\n\n2. Validation method\nUnshuffle KFold with K = 2. When I improved my LB from 1.13 to 1.10 by adding engineering features, improvements in local CV always led to improvements in LB. I believe it's a stable validation approch.\n\n3. Some useful features:\n - 'hour': self-explanatory\n - 'weekday': self-explanatory\n -  'day_in_month': self-explanatory\n - 'is_weekend': self-explanatory\n - 'building_id_count_full': count encoding of building_id\n - 'year_built_count_full': count encoding of year_built\n - 'site_id_cloud_coverage_48_mean': mean value of cloud_coverage in the past 48 hours for each site.\n\n4. Data cleaning\nAs I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)...Seems the data may be not clean for the first 6 months of 2016.\n\nHopes the findings can help and wish everyone a piece of good luck!!",
    "670925": "why did you lose interest? to me it seems like there's still a lot to explore",
    "670944": "`As I used 2 Fold CV, one strange thing I found is that the model fitted in the first fold performs significantly worse than the model fitted in the second fold (first fold 1.12 in LB, second fold 1.24 in LB, total 1.10 in LB)`\n\nThe second fold is closer to the Public LB Data (w.r.t. time). Could that be the reason?\n\n`The downsampling method is slightly modified from sampling without replacement. I upload a file containing the sample_index. If you want to use it, please use the np.load method to load the file as an array.`\n\nIt would be helpful if you provide some more details about the down sampling methodology (not only for this competition, but similar problem set as well).",
    "670978": "I think he meant the second fold validation score not the LB.  The second fold is trained then used to predict the first fold that is further away from the LB. I also have the same behavior on mine. I agree that it's probably caused by the outliers/missing data in the first 6 months. I only removed the 0s from site 1 from Jan to May 20th.  There's a lot of other buildings that had 0s in the beginning. Also the buildings that were in mid of constructions in 2016, mentioned in this thread. https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254#latest-667876",
    "670982": "Thanks for sharing, especially the downsampling method, will try to test.\nProbably don't need to give up, come back in a few more weeks, see if there's anything new. From what I understand, all the external data people found need to be published here if they want to use them.",
    "670990": "Makes sense. Yes, the outliers and missing data would be a problem for sure.\n\nAnother thing could be related to features engineered. If the features are more suitable to predict into the future, those will perform poorer while validating the past.",
    "671873": ""
  },
  "source": "meta"
}