Yes, everybody is bummed out by the massive number of leaks. Yes, everybody is spending their time finding new leaks. If that's your thing, feel free to be bummed out or excited. But I'm finding that I can quietly work on all the other aspects of the data and make forward progress, on the assumptions that the leaks will all wash out on the private LB.
\n\nSo here's what I find interesting:
\n\nThe data has distinct behaviors in a bunch of different dimensions:\n* Each type of meter works differently, even within the same building. It's unclear whether there's commonality between types which can be exploited.\n* Each site has shared weather, and potentially shared administrative policy and reporting methods within a site. It should be possible to exploit this commonality.\n* Each building is its own microcosm, with many factors not reflected in the data -- insulation, equipment use, work habits, etc. Our final model must be able to account for these per-building hidden variables.
\n\nAll of these distinct segments (along with time-series issues) must be considered when creating a representative CV split. I strongly that entire sites should be grouped together and split timewise. However, any further grouping may or may not be useful.
\n\nThey also allow for the possibility of training completely different per-segment models, but it's unclear where it will lead to a gain in predictive power through specialization or a loss due missed commonalities. My personal belief is that per-meter-type models are a good idea, but that per-building models probably throw away too much cross-over information. I'm shying away from per-site models because I fear they'll exaggerate noise without benefitting enough from commonalities -- but I could well be wrong.
\n\nThe data can, with compression, fit within a typical Kaggler's computer memory, but there's enough of it that computation time becomes a major factor. This encourages us to use subsampling to perform faster experiments but forces us to constantly re-evaluate whether such methodologies are sound.
\n\nI haven't seen much discussion around hyperparameter optimization for this competition, and the data size is probably the reason. With a single training run on the full data taking minutes, grid searches for hyperparameter optimization are impractical, and we need to get creative. Again, we can subsample but need to be able to argue that the hyperparameters are independent of sample size -- which is probably a bad bet. The net result is that we must either neglect hyperparameters in search of easier game or be prepared to spend a lot of time or cleverness in the search.
\n\nFinally, the fact that the size prompted a two-submissions-per-day limit means that there's probably a lot less leaderboard probing going on, and a lot more actual dependence on well-founded CVs. I know that's been the case for me at least, and I've found it somewhat refreshing.
\n\nTime-series problems are often popular because they allow us to exploit sequential patterns for missing-value imputation, use lag features to track trends, and exploit domain knowledge of time-of-day, seasonal, holiday, and time-zone issues.
\n\nIn this particular competition, we have the interesting (yet frustrating) fact that our training data consists of exactly one year. This makes it hard to judge common varieties of year-over-year patterns while still giving us at least one sample of each season and holiday. Is it possible, for example, to predict the effect of the 4th of July holiday based on many buildings but only one global occurrence? Does it matter that the 4th occurs on a Monday in 2016, but a Wednesday in 2018? Probably, but I don't know how much.
\n\nSince it's an international competition, we also have different holidays, different time zones, different implementations of daylight savings time, etc. This is complicated by the fact that weather is tracked in GMT, readings are tracked in local standard time, and actual building usage will be affected by local daylight savings time. How should we best account for this, or can we safely ignore it?
\n\nIt was clear from the start that data cleanup would make a huge difference in this competition, with the glaring set of zero-readings from site 0 begging to be fixed. Likewise, it very quickly became clear that building 1099 had such absurd outliers that they would warp our predictions for other buildings.
\n\nWhether we like it or not, these dirty data are a feature, not a bug. We were warned about it in the competition description, and post-facto write-ups of the previous ASHRAE competitions called out anomalous data handling as an important part of the process.
\n\nThe challenge here is to infer whether any given set of \"outliers\" will be reflected in the test data, or if they are a phenomenon that only applies in the training data. There are many buildings/meters in this competition in which the first 5 months (or so) of data are qualitatively different from the last 7. I have a personal theory that May 15th was a deadline from the organizers for \"get your reading methodology in order\", and that any anomalous reading from before this date should be distrusted. However, ignoring the entire set would be \"throwing the baby out with the bathwater\", so we need to figure out how to \"separate the wheat from the chaff\". (Hey, look! I've used my entire daily quota of stale cliches in one sentence. Oh well.)
\n\nZero-value meter readings are especially interesting. Some are clearly completely legitimate and predictable -- some utilities are clearly turned off for months at a time. Others are almost definitely illegitimate. In my opinion, this includes all zero-readings for electricity -- complete power-outages are rare and can (IMHO) be dropped as non-informative outliers. Long runs of consecutive zeros for a single meter (or within a single site) can indicate meter failures that should be disregarded (as is the case for the first 141 days of site 0 data). Finally, we have the class of \"pulse meters\", which will alternate between large readings and zero-readings, seemingly at random. If it's truly random, then we have little recourse, but I can't neglect the possibility that there is an actual pattern which can be exploited.
\n\nThe actual target leaks are annoying. Most people seem to agree on this. However, the process of hunting down and identifying the actual sites and buildings within the sites has been fascinating to watch. Given the external-data rules and Kaggler's love of sharing, this has felt like a pleasant bonus and not an annoying leak. I'm not a data-scraping guy myself, but I can enjoy watching the scrapers at work.
\n\nStrategy Evaluation: What helps and by How Much -- attempts to quantify a portion of the issues discussed here\nAshrae: easy data cleanup (LB 1.08 no leaks) -- demonstrates a particular strategy for data cleaning and a start-to-finish submission which addresses some of the issues discussed here\nThe Worst Meters -- attempts to locate the particular meters which most contribute to our error so that we can see what per-meter or systematic problems are worth addressing.
", "messages": [ { "id": "683761", "postDate": "11/28/2019 19:15:53", "content": "Yes, everybody is bummed out by the massive number of leaks. Yes, everybody is spending their time finding new leaks. If that's your thing, feel free to be bummed out or excited. But I'm finding that I can quietly work on all the other aspects of the data and make forward progress, on the assumptions that the leaks will all wash out on the private LB.
\n\nSo here's what I find interesting:
\n\nThe data has distinct behaviors in a bunch of different dimensions:\n* Each type of meter works differently, even within the same building. It's unclear whether there's commonality between types which can be exploited.\n* Each site has shared weather, and potentially shared administrative policy and reporting methods within a site. It should be possible to exploit this commonality.\n* Each building is its own microcosm, with many factors not reflected in the data -- insulation, equipment use, work habits, etc. Our final model must be able to account for these per-building hidden variables.
\n\nAll of these distinct segments (along with time-series issues) must be considered when creating a representative CV split. I strongly that entire sites should be grouped together and split timewise. However, any further grouping may or may not be useful.
\n\nThey also allow for the possibility of training completely different per-segment models, but it's unclear where it will lead to a gain in predictive power through specialization or a loss due missed commonalities. My personal belief is that per-meter-type models are a good idea, but that per-building models probably throw away too much cross-over information. I'm shying away from per-site models because I fear they'll exaggerate noise without benefitting enough from commonalities -- but I could well be wrong.
\n\nThe data can, with compression, fit within a typical Kaggler's computer memory, but there's enough of it that computation time becomes a major factor. This encourages us to use subsampling to perform faster experiments but forces us to constantly re-evaluate whether such methodologies are sound.
\n\nI haven't seen much discussion around hyperparameter optimization for this competition, and the data size is probably the reason. With a single training run on the full data taking minutes, grid searches for hyperparameter optimization are impractical, and we need to get creative. Again, we can subsample but need to be able to argue that the hyperparameters are independent of sample size -- which is probably a bad bet. The net result is that we must either neglect hyperparameters in search of easier game or be prepared to spend a lot of time or cleverness in the search.
\n\nFinally, the fact that the size prompted a two-submissions-per-day limit means that there's probably a lot less leaderboard probing going on, and a lot more actual dependence on well-founded CVs. I know that's been the case for me at least, and I've found it somewhat refreshing.
\n\nTime-series problems are often popular because they allow us to exploit sequential patterns for missing-value imputation, use lag features to track trends, and exploit domain knowledge of time-of-day, seasonal, holiday, and time-zone issues.
\n\nIn this particular competition, we have the interesting (yet frustrating) fact that our training data consists of exactly one year. This makes it hard to judge common varieties of year-over-year patterns while still giving us at least one sample of each season and holiday. Is it possible, for example, to predict the effect of the 4th of July holiday based on many buildings but only one global occurrence? Does it matter that the 4th occurs on a Monday in 2016, but a Wednesday in 2018? Probably, but I don't know how much.
\n\nSince it's an international competition, we also have different holidays, different time zones, different implementations of daylight savings time, etc. This is complicated by the fact that weather is tracked in GMT, readings are tracked in local standard time, and actual building usage will be affected by local daylight savings time. How should we best account for this, or can we safely ignore it?
\n\nIt was clear from the start that data cleanup would make a huge difference in this competition, with the glaring set of zero-readings from site 0 begging to be fixed. Likewise, it very quickly became clear that building 1099 had such absurd outliers that they would warp our predictions for other buildings.
\n\nWhether we like it or not, these dirty data are a feature, not a bug. We were warned about it in the competition description, and post-facto write-ups of the previous ASHRAE competitions called out anomalous data handling as an important part of the process.
\n\nThe challenge here is to infer whether any given set of \"outliers\" will be reflected in the test data, or if they are a phenomenon that only applies in the training data. There are many buildings/meters in this competition in which the first 5 months (or so) of data are qualitatively different from the last 7. I have a personal theory that May 15th was a deadline from the organizers for \"get your reading methodology in order\", and that any anomalous reading from before this date should be distrusted. However, ignoring the entire set would be \"throwing the baby out with the bathwater\", so we need to figure out how to \"separate the wheat from the chaff\". (Hey, look! I've used my entire daily quota of stale cliches in one sentence. Oh well.)
\n\nZero-value meter readings are especially interesting. Some are clearly completely legitimate and predictable -- some utilities are clearly turned off for months at a time. Others are almost definitely illegitimate. In my opinion, this includes all zero-readings for electricity -- complete power-outages are rare and can (IMHO) be dropped as non-informative outliers. Long runs of consecutive zeros for a single meter (or within a single site) can indicate meter failures that should be disregarded (as is the case for the first 141 days of site 0 data). Finally, we have the class of \"pulse meters\", which will alternate between large readings and zero-readings, seemingly at random. If it's truly random, then we have little recourse, but I can't neglect the possibility that there is an actual pattern which can be exploited.
\n\nThe actual target leaks are annoying. Most people seem to agree on this. However, the process of hunting down and identifying the actual sites and buildings within the sites has been fascinating to watch. Given the external-data rules and Kaggler's love of sharing, this has felt like a pleasant bonus and not an annoying leak. I'm not a data-scraping guy myself, but I can enjoy watching the scrapers at work.
\n\nStrategy Evaluation: What helps and by How Much -- attempts to quantify a portion of the issues discussed here\nAshrae: easy data cleanup (LB 1.08 no leaks) -- demonstrates a particular strategy for data cleaning and a start-to-finish submission which addresses some of the issues discussed here\nThe Worst Meters -- attempts to locate the particular meters which most contribute to our error so that we can see what per-meter or systematic problems are worth addressing.
", "rawMarkdown": "Yes, everybody is bummed out by the massive number of leaks. Yes, everybody is spending their time finding new leaks. If that's your thing, feel free to be bummed out or excited. But I'm finding that I can quietly work on all the *other* aspects of the data and make forward progress, on the assumptions that the leaks will all wash out on the private LB.\n\nSo here's what I find interesting:\n\n### Lots of different data segments:\nThe data has distinct behaviors in a bunch of different dimensions:\n* Each type of meter works differently, even within the same building. It's unclear whether there's commonality between types which can be exploited.\n* Each site has shared weather, and potentially shared administrative policy and reporting methods within a site. It should be possible to exploit this commonality.\n* Each building is its own microcosm, with many factors not reflected in the data -- insulation, equipment use, work habits, etc. Our final model must be able to account for these per-building hidden variables.\n\nAll of these distinct segments (along with time-series issues) must be considered when creating a representative CV split. I strongly that entire sites should be grouped together and split timewise. However, any further grouping may or may not be useful.\n\nThey also allow for the possibility of training completely different per-segment models, but it's unclear where it will lead to a gain in predictive power through specialization or a loss due missed commonalities. My personal belief is that per-meter-type models are a good idea, but that per-building models probably throw away too much cross-over information. I'm shying away from per-site models because I fear they'll exaggerate noise without benefitting enough from commonalities -- but I could well be wrong.\n\n### Big data:\nThe data can, with compression, fit within a typical Kaggler's computer memory, but there's enough of it that computation time becomes a major factor. This encourages us to use subsampling to perform faster experiments but forces us to constantly re-evaluate whether such methodologies are sound. \n\nI haven't seen much discussion around hyperparameter optimization for this competition, and the data size is probably the reason. With a single training run on the full data taking minutes, grid searches for hyperparameter optimization are impractical, and we need to get creative. Again, we can subsample but need to be able to argue that the hyperparameters are independent of sample size -- which is probably a bad bet. The net result is that we must either neglect hyperparameters in search of easier game or be prepared to spend a lot of time or cleverness in the search.\n\nFinally, the fact that the size prompted a two-submissions-per-day limit means that there's probably a lot less leaderboard probing going on, and a lot more actual dependence on well-founded CVs. I know that's been the case for me at least, and I've found it somewhat refreshing.\n\n### Time-series data:\nTime-series problems are often popular because they allow us to exploit sequential patterns for missing-value imputation, use lag features to track trends, and exploit domain knowledge of time-of-day, seasonal, holiday, and time-zone issues.\n\nIn this particular competition, we have the interesting (yet frustrating) fact that our training data consists of exactly one year. This makes it hard to judge common varieties of year-over-year patterns while still giving us at least one sample of each season and holiday. Is it possible, for example, to predict the effect of the 4th of July holiday based on many buildings but only one global occurrence? Does it matter that the 4th occurs on a Monday in 2016, but a Wednesday in 2018? Probably, but I don't know how much.\n\nSince it's an international competition, we also have different holidays, different time zones, different implementations of daylight savings time, etc. This is complicated by the fact that weather is tracked in GMT, readings are tracked in local standard time, and actual building usage will be affected by local daylight savings time. How should we best account for this, or can we safely ignore it?\n\n### Dirty data:\nIt was clear from the start that data cleanup would make a huge difference in this competition, with the glaring set of zero-readings from site 0 begging to be fixed. Likewise, it very quickly became clear that building 1099 had such absurd outliers that they would warp our predictions for other buildings.\n\nWhether we like it or not, these dirty data are a feature, not a bug. We were warned about it in the competition description, and post-facto write-ups of the previous ASHRAE competitions called out anomalous data handling as an important part of the process.\n\nThe challenge here is to infer whether any given set of \"outliers\" will be reflected in the test data, or if they are a phenomenon that only applies in the training data. There are many buildings/meters in this competition in which the first 5 months (or so) of data are qualitatively different from the last 7. I have a personal theory that May 15th was a deadline from the organizers for \"get your reading methodology in order\", and that any anomalous reading from before this date should be distrusted. However, ignoring the entire set would be \"throwing the baby out with the bathwater\", so we need to figure out how to \"separate the wheat from the chaff\". (Hey, look! I've used my entire daily quota of stale cliches in one sentence. Oh well.)\n\nZero-value meter readings are especially interesting. Some are clearly completely legitimate and predictable -- some utilities are clearly turned off for months at a time. Others are almost definitely illegitimate. In my opinion, this includes all zero-readings for electricity -- complete power-outages are rare and can (IMHO) be dropped as non-informative outliers. Long runs of consecutive zeros for a single meter (or within a single site) can indicate meter failures that should be disregarded (as is the case for the first 141 days of site 0 data). Finally, we have the class of \"pulse meters\", which will alternate between large readings and zero-readings, seemingly at random. If it's truly random, then we have little recourse, but I can't neglect the possibility that there is an actual pattern which can be exploited.\n\n### External data hunts:\nThe actual target leaks are annoying. Most people seem to agree on this. However, the process of hunting down and identifying the actual sites and buildings within the sites has been fascinating to watch. Given the external-data rules and Kaggler's love of sharing, this has felt like a pleasant bonus and not an annoying leak. I'm not a data-scraping guy myself, but I can enjoy watching the scrapers at work.\n\n# References:\n[Strategy Evaluation: What helps and by How Much](https://www.kaggle.com/purist1024/strategy-evaluation-what-helps-and-by-how-much/) -- attempts to quantify a portion of the issues discussed here\n[Ashrae: easy data cleanup (LB 1.08 no leaks)](https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks) -- demonstrates a particular strategy for data cleaning and a start-to-finish submission which addresses some of the issues discussed here\n[The Worst Meters](https://www.kaggle.com/purist1024/ashrae-the-worst-meters) -- attempts to locate the particular meters which most contribute to our error so that we can see what per-meter or systematic problems are worth addressing.", "votes": null }, { "id": "683779", "postDate": "11/28/2019 19:53:35", "content": "I agree. I got to around 20 on LB before leaks became widespread. I'm not convinced training on any leaked data will be a dramatic improvement to private LB. I've been making consistent progress on LB without leaks since the leaks started. Wondering if you would like to consider teaming up - my models seems to work really well with your data cleanup notebook.
", "rawMarkdown": "I agree. I got to around 20 on LB before leaks became widespread. I'm not convinced training on any leaked data will be a dramatic improvement to private LB. I've been making consistent progress on LB without leaks since the leaks started. Wondering if you would like to consider teaming up - my models seems to work really well with your data cleanup notebook.", "votes": null }, { "id": "683925", "postDate": "11/29/2019 02:33:55", "content": "May I ask your LB without leak?
", "rawMarkdown": "May I ask your LB without leak?", "votes": null }, { "id": "683927", "postDate": "11/29/2019 02:41:15", "content": "No problem. My non-leak score is 1.06, using: the various techniques I describe in my kernels; a few new features which I haven't posted yet; and 6-way CV for both early stopping and blending.
", "rawMarkdown": "No problem. My non-leak score is 1.06, using: the various techniques I describe in my kernels; a few new features which I haven't posted yet; and 6-way CV for both early stopping and blending.", "votes": null }, { "id": "683929", "postDate": "11/29/2019 02:43:19", "content": "I'm glad that my posts are providing you with new insights. I'm not quite ready to team-up yet -- I'm having more fun exploring and sharing without worrying about any over-sharing annoying my teammates.
", "rawMarkdown": "I'm glad that my posts are providing you with new insights. I'm not quite ready to team-up yet -- I'm having more fun exploring and sharing without worrying about any over-sharing annoying my teammates.", "votes": null }, { "id": "684106", "postDate": "11/29/2019 08:37:05", "content": "Interesting write-up. Do you have any insight regarding model validation? I tried vanilla KFold as well as GroupKFold (using hour of the day as group label) but the score I get on my test sets is always very different (ie. much better...) than the actual LB score.
", "rawMarkdown": "Interesting write-up. Do you have any insight regarding model validation? I tried vanilla KFold as well as GroupKFold (using hour of the day as group label) but the score I get on my test sets is always very different (ie. much better...) than the actual LB score.", "votes": null }, { "id": "684260", "postDate": "11/29/2019 14:03:40", "content": "Which CV strategy you found to work best with this data set?
", "rawMarkdown": "Which CV strategy you found to work best with this data set?", "votes": null }, { "id": "684406", "postDate": "11/29/2019 17:28:29", "content": "Yes, you have a point. There are still some fun challenges in this competition. I just hope the hosts will still be able to create a fair private leaderboard ranking when the competition ends.
", "rawMarkdown": "Yes, you have a point. There are still some fun challenges in this competition. I just hope the hosts will still be able to create a fair private leaderboard ranking when the competition ends.", "votes": null }, { "id": "684477", "postDate": "11/29/2019 21:44:20", "content": "How are you dealing with the fact that it is a timeseries problem? I`ve tried using LGBM without much sucess.
", "rawMarkdown": "How are you dealing with the fact that it is a timeseries problem? I`ve tried using LGBM without much sucess.", "votes": null }, { "id": "684489", "postDate": "11/29/2019 22:28:02", "content": "Hi Gabriel,
\n\nIt is technically a timeseries problem, but the data is very noisy and it is pretty tough to build a model that can predict 2 years ahead with one year of data. Thats why I think a lot of people are treating it mostly as a tabular prediction problem. LightGBM should work well if you remove some of time aspects (mainly month).
\n\nHowever, there are some great opportunities to focus on timeseries modeling in this competition. If you ignore the noise the energy usage pattern is quite simple, sometimes even constant.
\n\nMaybe you can try splitting by meter and train a simple LSTM model for each meter? Here is a nice kernel which implements an LSTM.
\n\nHope this helps!
", "rawMarkdown": "Hi Gabriel,\n\nIt is technically a timeseries problem, but the data is very noisy and it is pretty tough to build a model that can predict 2 years ahead with one year of data. Thats why I think a lot of people are treating it mostly as a tabular prediction problem. LightGBM should work well if you remove some of time aspects (mainly month).\n\nHowever, there are some great opportunities to focus on timeseries modeling in this competition. If you ignore the noise the energy usage pattern is quite simple, sometimes even constant.\n\nMaybe you can try splitting by meter and train a simple LSTM model for each meter? [Here is a nice kernel which implements an LSTM](https://www.kaggle.com/niyamatalmass/machine-learning-for-time-series-analysis). \n\nHope this helps!", "votes": null }, { "id": "684517", "postDate": "11/30/2019 00:23:18", "content": "I provide an implementation of a reasonable scheme in one of my kernels.
\n\n\n\n\nWe use a time-oriented split which deals in 12 equal time slices (which are almost, but not quite, months). Each fold includes a two adjacent slices for validation, 8 adjacent slices for training, and a 1 slice buffer on each side of the validation set. We use modulo arithmetic to wrap-around the year-end boundary. We thus have slices 4-11 vs 1-2; 6-12&1 vs 3-4; 8-12&1-3 vs 5-6; etc.
\n \nWhile the CV score is always better than the LB score, relative differences between strategies tend to align well between CV and LB.\n ```\n def np_sample(a, frac):\n return a if frac == 1 else np.random.choice(a, int(len(a) * frac), replace=False)
\n
def make_8121_splits(X, sample_frac):\n np.random.seed(0)\n time_sorted_idx = np.argsort(X.timestamp.values, kind='stable')\n sections = np.array_split(time_sorted_idx, 12)\n folds = []\n for start_ix in range(0, 12, 2):\n val_idxs = np.concatenate(sections[start_ix:start_ix + 2]) # no modulo necessary\n train_idxs = np.concatenate(\n [sections[ix % 12] for ix in range(start_ix + 3, start_ix + 11)])\n folds.append((np_sample(train_idxs, sample_frac), np_sample(val_idxs, sample_frac)))\n return folds\n```\nYou can do even better by having a different set of time splits for each site, but the code gets noticably more complicated at that point.
", "rawMarkdown": "I provide an implementation of a reasonable scheme in one of my kernels.\n\n> We use a time-oriented split which deals in 12 equal time slices (which are almost, but not quite, months). Each fold includes a two adjacent slices for validation, 8 adjacent slices for training, and a 1 slice buffer on each side of the validation set. We use modulo arithmetic to wrap-around the year-end boundary. We thus have slices 4-11 vs 1-2; 6-12&1 vs 3-4; 8-12&1-3 vs 5-6; etc.\n\n> While the CV score is always better than the LB score, relative differences between strategies tend to align well between CV and LB.\n```\ndef np_sample(a, frac):\n return a if frac == 1 else np.random.choice(a, int(len(a) * frac), replace=False)\n\ndef make_8121_splits(X, sample_frac):\n np.random.seed(0)\n time_sorted_idx = np.argsort(X.timestamp.values, kind='stable')\n sections = np.array_split(time_sorted_idx, 12)\n folds = []\n for start_ix in range(0, 12, 2):\n val_idxs = np.concatenate(sections[start_ix:start_ix + 2]) # no modulo necessary\n train_idxs = np.concatenate(\n [sections[ix % 12] for ix in range(start_ix + 3, start_ix + 11)])\n folds.append((np_sample(train_idxs, sample_frac), np_sample(val_idxs, sample_frac)))\n return folds\n```\nYou can do even better by having a different set of time splits for each site, but the code gets noticably more complicated at that point.", "votes": null }, { "id": "684520", "postDate": "11/30/2019 00:24:17", "content": "@jonimatix See my reply to @larryd87 above.
", "rawMarkdown": "jonimatix See my reply to @larryd87 above.", "votes": null }, { "id": "684547", "postDate": "11/30/2019 02:06:18", "content": "As @carlolepelaars notes, the easiest way is to simply add a few time-based features (i.e. hour-of-day and day-of-week) and then treat it like any other tabular problem. However, you must take the time-series properties into account when doing your validation, or you'll never be able to accurately judge your model's quality.
\n\nThe kernels I reference at the bottom of the post showcase basically this approach and include both a reasonable CV splitter and a complete end-to-end submission.
\n\nThe time-series properties can still be used in small ways while mostly using a tabular approach. Interpolation of missing weather features is one. Lag-based value-over-time features are another.
", "rawMarkdown": "As @carlolepelaars notes, the easiest way is to simply add a few time-based features (i.e. hour-of-day and day-of-week) and then treat it like any other tabular problem. However, you *must* take the time-series properties into account when doing your validation, or you'll never be able to accurately judge your model's quality.\n\nThe kernels I reference at the bottom of the post showcase basically this approach and include both a reasonable CV splitter and a complete end-to-end submission.\n\nThe time-series properties can still be used in small ways while mostly using a tabular approach. Interpolation of missing weather features is one. Lag-based value-over-time features are another.", "votes": null }, { "id": "685071", "postDate": "11/30/2019 23:16:09", "content": "Another thing I found interesting is, the data features consist of continuous features and categorical features, which make designing of my neural networks more interesting.
", "rawMarkdown": "Another thing I found interesting is, the data features consist of continuous features and categorical features, which make designing of my neural networks more interesting.", "votes": null }, { "id": "685350", "postDate": "12/01/2019 13:28:12", "content": "How do you explain the lack of top kagglers (grandmasters etc.) in this competition?
", "rawMarkdown": "How do you explain the lack of top kagglers (grandmasters etc.) in this competition?", "votes": null }, { "id": "685356", "postDate": "12/01/2019 13:44:56", "content": "Perhaps an idiom?
\n\n> One man's trash is another man's treasure
", "rawMarkdown": "Perhaps an idiom?\n\n> One man's trash is another man's treasure", "votes": null }, { "id": "685372", "postDate": "12/01/2019 14:26:00", "content": "@teeyee314 :)
\n\n@stecasasso, my feelings about this competion is mostly captured in this post by @purist1024. As for the grandmasters, they did not find web scraping as something they enjoy as stipulated by @cpmpml and a couple of others. I was as upset as anyone can be initially when the leaks was discovered but I took a step back and gathered my thoughts as I usually do in this type of situations. Since currently, there is no clean way of dropping out of a kaggle competition plus I have a little bit more time for kaggling than usual, I decided to use it as a way to try out things I have not done before. In short I am glad I stayed because I have learnt quite a bit and learning is my main goal on Kaggle.
\n\nSorry guys and girls for the long post :-)
", "rawMarkdown": "teeyee314 :)\n\n@stecasasso, my feelings about this competion is mostly captured in this post by @purist1024. As for the grandmasters, they did not find web scraping as something they enjoy as stipulated by @cpmpml and a couple of others. I was as upset as anyone can be initially when the leaks was discovered but I took a step back and gathered my thoughts as I usually do in this type of situations. Since currently, there is no clean way of dropping out of a kaggle competition plus I have a little bit more time for kaggling than usual, I decided to use it as a way to try out things I have not done before. In short I am glad I stayed because I have learnt quite a bit and learning is my main goal on Kaggle.\n\nSorry guys and girls for the long post :-)", "votes": null }, { "id": "685422", "postDate": "12/01/2019 16:27:38", "content": "> How do you explain the lack of top kagglers (grandmasters etc.) in this competition?
\n\nMany were busy with NFL, plus several competitions launched near the same time.
\n\nThat's the first explanation. The second one is that most top kagglers nowadays are computer vision experts, and this is not a computer vision competition ;)
", "rawMarkdown": "> How do you explain the lack of top kagglers (grandmasters etc.) in this competition?\n\nMany were busy with NFL, plus several competitions launched near the same time.\n\nThat's the first explanation. The second one is that most top kagglers nowadays are computer vision experts, and this is not a computer vision competition ;)", "votes": null }, { "id": "685519", "postDate": "12/01/2019 20:32:04", "content": "> Many were busy with NFL
\n\nHmmm... NFL was launched 6 days before this one: do you mean that within the first 6 days of NFL competition everyone had already joined and fully committed to that? Difficult to imagine.
\n\nI agree with the author of the post: this competition had good potential (I like the dirty data and big data parts).\nBut unfortunately it also has a poor design: 1 year of training vs. 2 years of test limits considerably the feature engineering, which is already limited because of the not-so-rich dataset.\nHowever, the most worrying thing it's the size of the leakage which I am afraid Kaggle's admins and organizers don't have under control (despite their claim). \nIf they had any clue about this, they would have not started this competition with this data => they were not aware of all this data being public => they still don't know 100% about the data which is publicly available => they cannot guarantee a leak-free private LB \nMy 2 cents
\n\nI regret not joining the NFL competition, which was also my first choice...
", "rawMarkdown": "> Many were busy with NFL\n\nHmmm... NFL was launched 6 days before this one: do you mean that within the first 6 days of NFL competition everyone had already joined and fully committed to that? Difficult to imagine. \n\nI agree with the author of the post: this competition had good potential (I like the dirty data and big data parts).\nBut unfortunately it also has a poor design: 1 year of training vs. 2 years of test limits considerably the feature engineering, which is already limited because of the not-so-rich dataset.\nHowever, the most worrying thing it's the size of the leakage which I am afraid Kaggle's admins and organizers don't have under control (despite their claim). \nIf they had any clue about this, they would have not started this competition with this data => they were not aware of all this data being public => they still don't know 100% about the data which is publicly available => they cannot guarantee a leak-free private LB \nMy 2 cents \n\nI regret not joining the NFL competition, which was also my first choice...", "votes": null }, { "id": "685803", "postDate": "12/02/2019 10:18:08", "content": "Well, I know several GMs who opted for NFL instead of ASHRAE.
", "rawMarkdown": "Well, I know several GMs who opted for NFL instead of ASHRAE.", "votes": null }, { "id": "685818", "postDate": "12/02/2019 10:59:27", "content": "Thanks! Are you keeping a fraction only of the train and validation sets for speed purpose or do you have something else in mind?
", "rawMarkdown": "Thanks! Are you keeping a fraction only of the train and validation sets for speed purpose or do you have something else in mind?", "votes": null }, { "id": "685856", "postDate": "12/02/2019 12:32:23", "content": "For evaluating new features and the like, I've used as little as 3% of the input data, though it's a bit more stable with 10-20%. This is why I have the \"sample-frac\" built into my CV splitter. For final training, of course, I use the full dataset.
", "rawMarkdown": "For evaluating new features and the like, I've used as little as 3% of the input data, though it's a bit more stable with 10-20%. This is why I have the \"sample-frac\" built into my CV splitter. For final training, of course, I use the full dataset.", "votes": null } ], "comments": [ { "id": 683779, "author_name": "teeyee314", "author_url": "", "post_date": "11/28/2019 19:53:35", "content": "I agree. I got to around 20 on LB before leaks became widespread. I'm not convinced training on any leaked data will be a dramatic improvement to private LB. I've been making consistent progress on LB without leaks since the leaks started. Wondering if you would like to consider teaming up - my models seems to work really well with your data cleanup notebook.
", "votes": null, "replies": [ { "id": 683929, "author_name": "purist1024", "author_url": "", "post_date": "11/29/2019 02:43:19", "content": "I'm glad that my posts are providing you with new insights. I'm not quite ready to team-up yet -- I'm having more fun exploring and sharing without worrying about any over-sharing annoying my teammates.
", "votes": null, "replies": [] } ] }, { "id": 683925, "author_name": "zidaoziyan", "author_url": "", "post_date": "11/29/2019 02:33:55", "content": "May I ask your LB without leak?
", "votes": null, "replies": [ { "id": 683927, "author_name": "purist1024", "author_url": "", "post_date": "11/29/2019 02:41:15", "content": "No problem. My non-leak score is 1.06, using: the various techniques I describe in my kernels; a few new features which I haven't posted yet; and 6-way CV for both early stopping and blending.
", "votes": null, "replies": [] }, { "id": 684260, "author_name": "jonimatix", "author_url": "", "post_date": "11/29/2019 14:03:40", "content": "Which CV strategy you found to work best with this data set?
", "votes": null, "replies": [] }, { "id": 684520, "author_name": "purist1024", "author_url": "", "post_date": "11/30/2019 00:24:17", "content": "@jonimatix See my reply to @larryd87 above.
", "votes": null, "replies": [] } ] }, { "id": 684106, "author_name": "larryd87", "author_url": "", "post_date": "11/29/2019 08:37:05", "content": "Interesting write-up. Do you have any insight regarding model validation? I tried vanilla KFold as well as GroupKFold (using hour of the day as group label) but the score I get on my test sets is always very different (ie. much better...) than the actual LB score.
", "votes": null, "replies": [ { "id": 684517, "author_name": "purist1024", "author_url": "", "post_date": "11/30/2019 00:23:18", "content": "I provide an implementation of a reasonable scheme in one of my kernels.
\n\n\n\n\nWe use a time-oriented split which deals in 12 equal time slices (which are almost, but not quite, months). Each fold includes a two adjacent slices for validation, 8 adjacent slices for training, and a 1 slice buffer on each side of the validation set. We use modulo arithmetic to wrap-around the year-end boundary. We thus have slices 4-11 vs 1-2; 6-12&1 vs 3-4; 8-12&1-3 vs 5-6; etc.
\n \nWhile the CV score is always better than the LB score, relative differences between strategies tend to align well between CV and LB.\n ```\n def np_sample(a, frac):\n return a if frac == 1 else np.random.choice(a, int(len(a) * frac), replace=False)
\n
def make_8121_splits(X, sample_frac):\n np.random.seed(0)\n time_sorted_idx = np.argsort(X.timestamp.values, kind='stable')\n sections = np.array_split(time_sorted_idx, 12)\n folds = []\n for start_ix in range(0, 12, 2):\n val_idxs = np.concatenate(sections[start_ix:start_ix + 2]) # no modulo necessary\n train_idxs = np.concatenate(\n [sections[ix % 12] for ix in range(start_ix + 3, start_ix + 11)])\n folds.append((np_sample(train_idxs, sample_frac), np_sample(val_idxs, sample_frac)))\n return folds\n```\nYou can do even better by having a different set of time splits for each site, but the code gets noticably more complicated at that point.
", "votes": null, "replies": [] }, { "id": 685818, "author_name": "larryd87", "author_url": "", "post_date": "12/02/2019 10:59:27", "content": "Thanks! Are you keeping a fraction only of the train and validation sets for speed purpose or do you have something else in mind?
", "votes": null, "replies": [] }, { "id": 685856, "author_name": "purist1024", "author_url": "", "post_date": "12/02/2019 12:32:23", "content": "For evaluating new features and the like, I've used as little as 3% of the input data, though it's a bit more stable with 10-20%. This is why I have the \"sample-frac\" built into my CV splitter. For final training, of course, I use the full dataset.
", "votes": null, "replies": [] } ] }, { "id": 684406, "author_name": "carlolepelaars", "author_url": "", "post_date": "11/29/2019 17:28:29", "content": "Yes, you have a point. There are still some fun challenges in this competition. I just hope the hosts will still be able to create a fair private leaderboard ranking when the competition ends.
", "votes": null, "replies": [] }, { "id": 684477, "author_name": "capiru", "author_url": "", "post_date": "11/29/2019 21:44:20", "content": "How are you dealing with the fact that it is a timeseries problem? I`ve tried using LGBM without much sucess.
", "votes": null, "replies": [ { "id": 684489, "author_name": "carlolepelaars", "author_url": "", "post_date": "11/29/2019 22:28:02", "content": "Hi Gabriel,
\n\nIt is technically a timeseries problem, but the data is very noisy and it is pretty tough to build a model that can predict 2 years ahead with one year of data. Thats why I think a lot of people are treating it mostly as a tabular prediction problem. LightGBM should work well if you remove some of time aspects (mainly month).
\n\nHowever, there are some great opportunities to focus on timeseries modeling in this competition. If you ignore the noise the energy usage pattern is quite simple, sometimes even constant.
\n\nMaybe you can try splitting by meter and train a simple LSTM model for each meter? Here is a nice kernel which implements an LSTM.
\n\nHope this helps!
", "votes": null, "replies": [] }, { "id": 684547, "author_name": "purist1024", "author_url": "", "post_date": "11/30/2019 02:06:18", "content": "As @carlolepelaars notes, the easiest way is to simply add a few time-based features (i.e. hour-of-day and day-of-week) and then treat it like any other tabular problem. However, you must take the time-series properties into account when doing your validation, or you'll never be able to accurately judge your model's quality.
\n\nThe kernels I reference at the bottom of the post showcase basically this approach and include both a reasonable CV splitter and a complete end-to-end submission.
\n\nThe time-series properties can still be used in small ways while mostly using a tabular approach. Interpolation of missing weather features is one. Lag-based value-over-time features are another.
", "votes": null, "replies": [] } ] }, { "id": 685071, "author_name": "lijunleozhang", "author_url": "", "post_date": "11/30/2019 23:16:09", "content": "Another thing I found interesting is, the data features consist of continuous features and categorical features, which make designing of my neural networks more interesting.
", "votes": null, "replies": [] }, { "id": 685350, "author_name": "stecasasso", "author_url": "", "post_date": "12/01/2019 13:28:12", "content": "How do you explain the lack of top kagglers (grandmasters etc.) in this competition?
", "votes": null, "replies": [ { "id": 685356, "author_name": "teeyee314", "author_url": "", "post_date": "12/01/2019 13:44:56", "content": "Perhaps an idiom?
\n\n> One man's trash is another man's treasure
", "votes": null, "replies": [] }, { "id": 685372, "author_name": "sheriytm", "author_url": "", "post_date": "12/01/2019 14:26:00", "content": "@teeyee314 :)
\n\n@stecasasso, my feelings about this competion is mostly captured in this post by @purist1024. As for the grandmasters, they did not find web scraping as something they enjoy as stipulated by @cpmpml and a couple of others. I was as upset as anyone can be initially when the leaks was discovered but I took a step back and gathered my thoughts as I usually do in this type of situations. Since currently, there is no clean way of dropping out of a kaggle competition plus I have a little bit more time for kaggling than usual, I decided to use it as a way to try out things I have not done before. In short I am glad I stayed because I have learnt quite a bit and learning is my main goal on Kaggle.
\n\nSorry guys and girls for the long post :-)
", "votes": null, "replies": [] }, { "id": 685422, "author_name": "cpmpml", "author_url": "", "post_date": "12/01/2019 16:27:38", "content": "> How do you explain the lack of top kagglers (grandmasters etc.) in this competition?
\n\nMany were busy with NFL, plus several competitions launched near the same time.
\n\nThat's the first explanation. The second one is that most top kagglers nowadays are computer vision experts, and this is not a computer vision competition ;)
", "votes": null, "replies": [] }, { "id": 685519, "author_name": "stecasasso", "author_url": "", "post_date": "12/01/2019 20:32:04", "content": "> Many were busy with NFL
\n\nHmmm... NFL was launched 6 days before this one: do you mean that within the first 6 days of NFL competition everyone had already joined and fully committed to that? Difficult to imagine.
\n\nI agree with the author of the post: this competition had good potential (I like the dirty data and big data parts).\nBut unfortunately it also has a poor design: 1 year of training vs. 2 years of test limits considerably the feature engineering, which is already limited because of the not-so-rich dataset.\nHowever, the most worrying thing it's the size of the leakage which I am afraid Kaggle's admins and organizers don't have under control (despite their claim). \nIf they had any clue about this, they would have not started this competition with this data => they were not aware of all this data being public => they still don't know 100% about the data which is publicly available => they cannot guarantee a leak-free private LB \nMy 2 cents
\n\nI regret not joining the NFL competition, which was also my first choice...
", "votes": null, "replies": [] }, { "id": 685803, "author_name": "cpmpml", "author_url": "", "post_date": "12/02/2019 10:18:08", "content": "Well, I know several GMs who opted for NFL instead of ASHRAE.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "683761": "Yes, everybody is bummed out by the massive number of leaks. Yes, everybody is spending their time finding new leaks. If that's your thing, feel free to be bummed out or excited. But I'm finding that I can quietly work on all the *other* aspects of the data and make forward progress, on the assumptions that the leaks will all wash out on the private LB.\n\nSo here's what I find interesting:\n\n### Lots of different data segments:\nThe data has distinct behaviors in a bunch of different dimensions:\n* Each type of meter works differently, even within the same building. It's unclear whether there's commonality between types which can be exploited.\n* Each site has shared weather, and potentially shared administrative policy and reporting methods within a site. It should be possible to exploit this commonality.\n* Each building is its own microcosm, with many factors not reflected in the data -- insulation, equipment use, work habits, etc. Our final model must be able to account for these per-building hidden variables.\n\nAll of these distinct segments (along with time-series issues) must be considered when creating a representative CV split. I strongly that entire sites should be grouped together and split timewise. However, any further grouping may or may not be useful.\n\nThey also allow for the possibility of training completely different per-segment models, but it's unclear where it will lead to a gain in predictive power through specialization or a loss due missed commonalities. My personal belief is that per-meter-type models are a good idea, but that per-building models probably throw away too much cross-over information. I'm shying away from per-site models because I fear they'll exaggerate noise without benefitting enough from commonalities -- but I could well be wrong.\n\n### Big data:\nThe data can, with compression, fit within a typical Kaggler's computer memory, but there's enough of it that computation time becomes a major factor. This encourages us to use subsampling to perform faster experiments but forces us to constantly re-evaluate whether such methodologies are sound. \n\nI haven't seen much discussion around hyperparameter optimization for this competition, and the data size is probably the reason. With a single training run on the full data taking minutes, grid searches for hyperparameter optimization are impractical, and we need to get creative. Again, we can subsample but need to be able to argue that the hyperparameters are independent of sample size -- which is probably a bad bet. The net result is that we must either neglect hyperparameters in search of easier game or be prepared to spend a lot of time or cleverness in the search.\n\nFinally, the fact that the size prompted a two-submissions-per-day limit means that there's probably a lot less leaderboard probing going on, and a lot more actual dependence on well-founded CVs. I know that's been the case for me at least, and I've found it somewhat refreshing.\n\n### Time-series data:\nTime-series problems are often popular because they allow us to exploit sequential patterns for missing-value imputation, use lag features to track trends, and exploit domain knowledge of time-of-day, seasonal, holiday, and time-zone issues.\n\nIn this particular competition, we have the interesting (yet frustrating) fact that our training data consists of exactly one year. This makes it hard to judge common varieties of year-over-year patterns while still giving us at least one sample of each season and holiday. Is it possible, for example, to predict the effect of the 4th of July holiday based on many buildings but only one global occurrence? Does it matter that the 4th occurs on a Monday in 2016, but a Wednesday in 2018? Probably, but I don't know how much.\n\nSince it's an international competition, we also have different holidays, different time zones, different implementations of daylight savings time, etc. This is complicated by the fact that weather is tracked in GMT, readings are tracked in local standard time, and actual building usage will be affected by local daylight savings time. How should we best account for this, or can we safely ignore it?\n\n### Dirty data:\nIt was clear from the start that data cleanup would make a huge difference in this competition, with the glaring set of zero-readings from site 0 begging to be fixed. Likewise, it very quickly became clear that building 1099 had such absurd outliers that they would warp our predictions for other buildings.\n\nWhether we like it or not, these dirty data are a feature, not a bug. We were warned about it in the competition description, and post-facto write-ups of the previous ASHRAE competitions called out anomalous data handling as an important part of the process.\n\nThe challenge here is to infer whether any given set of \"outliers\" will be reflected in the test data, or if they are a phenomenon that only applies in the training data. There are many buildings/meters in this competition in which the first 5 months (or so) of data are qualitatively different from the last 7. I have a personal theory that May 15th was a deadline from the organizers for \"get your reading methodology in order\", and that any anomalous reading from before this date should be distrusted. However, ignoring the entire set would be \"throwing the baby out with the bathwater\", so we need to figure out how to \"separate the wheat from the chaff\". (Hey, look! I've used my entire daily quota of stale cliches in one sentence. Oh well.)\n\nZero-value meter readings are especially interesting. Some are clearly completely legitimate and predictable -- some utilities are clearly turned off for months at a time. Others are almost definitely illegitimate. In my opinion, this includes all zero-readings for electricity -- complete power-outages are rare and can (IMHO) be dropped as non-informative outliers. Long runs of consecutive zeros for a single meter (or within a single site) can indicate meter failures that should be disregarded (as is the case for the first 141 days of site 0 data). Finally, we have the class of \"pulse meters\", which will alternate between large readings and zero-readings, seemingly at random. If it's truly random, then we have little recourse, but I can't neglect the possibility that there is an actual pattern which can be exploited.\n\n### External data hunts:\nThe actual target leaks are annoying. Most people seem to agree on this. However, the process of hunting down and identifying the actual sites and buildings within the sites has been fascinating to watch. Given the external-data rules and Kaggler's love of sharing, this has felt like a pleasant bonus and not an annoying leak. I'm not a data-scraping guy myself, but I can enjoy watching the scrapers at work.\n\n# References:\n[Strategy Evaluation: What helps and by How Much](https://www.kaggle.com/purist1024/strategy-evaluation-what-helps-and-by-how-much/) -- attempts to quantify a portion of the issues discussed here\n[Ashrae: easy data cleanup (LB 1.08 no leaks)](https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks) -- demonstrates a particular strategy for data cleaning and a start-to-finish submission which addresses some of the issues discussed here\n[The Worst Meters](https://www.kaggle.com/purist1024/ashrae-the-worst-meters) -- attempts to locate the particular meters which most contribute to our error so that we can see what per-meter or systematic problems are worth addressing.", "683779": "I agree. I got to around 20 on LB before leaks became widespread. I'm not convinced training on any leaked data will be a dramatic improvement to private LB. I've been making consistent progress on LB without leaks since the leaks started. Wondering if you would like to consider teaming up - my models seems to work really well with your data cleanup notebook.", "683925": "May I ask your LB without leak?", "683927": "No problem. My non-leak score is 1.06, using: the various techniques I describe in my kernels; a few new features which I haven't posted yet; and 6-way CV for both early stopping and blending.", "683929": "I'm glad that my posts are providing you with new insights. I'm not quite ready to team-up yet -- I'm having more fun exploring and sharing without worrying about any over-sharing annoying my teammates.", "684106": "Interesting write-up. Do you have any insight regarding model validation? I tried vanilla KFold as well as GroupKFold (using hour of the day as group label) but the score I get on my test sets is always very different (ie. much better...) than the actual LB score.", "684260": "Which CV strategy you found to work best with this data set?", "684406": "Yes, you have a point. There are still some fun challenges in this competition. I just hope the hosts will still be able to create a fair private leaderboard ranking when the competition ends.", "684477": "How are you dealing with the fact that it is a timeseries problem? I`ve tried using LGBM without much sucess.", "684489": "Hi Gabriel,\n\nIt is technically a timeseries problem, but the data is very noisy and it is pretty tough to build a model that can predict 2 years ahead with one year of data. Thats why I think a lot of people are treating it mostly as a tabular prediction problem. LightGBM should work well if you remove some of time aspects (mainly month).\n\nHowever, there are some great opportunities to focus on timeseries modeling in this competition. If you ignore the noise the energy usage pattern is quite simple, sometimes even constant.\n\nMaybe you can try splitting by meter and train a simple LSTM model for each meter? [Here is a nice kernel which implements an LSTM](https://www.kaggle.com/niyamatalmass/machine-learning-for-time-series-analysis). \n\nHope this helps!", "684517": "I provide an implementation of a reasonable scheme in one of my kernels.\n\n> We use a time-oriented split which deals in 12 equal time slices (which are almost, but not quite, months). Each fold includes a two adjacent slices for validation, 8 adjacent slices for training, and a 1 slice buffer on each side of the validation set. We use modulo arithmetic to wrap-around the year-end boundary. We thus have slices 4-11 vs 1-2; 6-12&1 vs 3-4; 8-12&1-3 vs 5-6; etc.\n\n> While the CV score is always better than the LB score, relative differences between strategies tend to align well between CV and LB.\n```\ndef np_sample(a, frac):\n return a if frac == 1 else np.random.choice(a, int(len(a) * frac), replace=False)\n\ndef make_8121_splits(X, sample_frac):\n np.random.seed(0)\n time_sorted_idx = np.argsort(X.timestamp.values, kind='stable')\n sections = np.array_split(time_sorted_idx, 12)\n folds = []\n for start_ix in range(0, 12, 2):\n val_idxs = np.concatenate(sections[start_ix:start_ix + 2]) # no modulo necessary\n train_idxs = np.concatenate(\n [sections[ix % 12] for ix in range(start_ix + 3, start_ix + 11)])\n folds.append((np_sample(train_idxs, sample_frac), np_sample(val_idxs, sample_frac)))\n return folds\n```\nYou can do even better by having a different set of time splits for each site, but the code gets noticably more complicated at that point.", "684520": "jonimatix See my reply to @larryd87 above.", "684547": "As @carlolepelaars notes, the easiest way is to simply add a few time-based features (i.e. hour-of-day and day-of-week) and then treat it like any other tabular problem. However, you *must* take the time-series properties into account when doing your validation, or you'll never be able to accurately judge your model's quality.\n\nThe kernels I reference at the bottom of the post showcase basically this approach and include both a reasonable CV splitter and a complete end-to-end submission.\n\nThe time-series properties can still be used in small ways while mostly using a tabular approach. Interpolation of missing weather features is one. Lag-based value-over-time features are another.", "685071": "Another thing I found interesting is, the data features consist of continuous features and categorical features, which make designing of my neural networks more interesting.", "685350": "How do you explain the lack of top kagglers (grandmasters etc.) in this competition?", "685356": "Perhaps an idiom?\n\n> One man's trash is another man's treasure", "685372": "teeyee314 :)\n\n@stecasasso, my feelings about this competion is mostly captured in this post by @purist1024. As for the grandmasters, they did not find web scraping as something they enjoy as stipulated by @cpmpml and a couple of others. I was as upset as anyone can be initially when the leaks was discovered but I took a step back and gathered my thoughts as I usually do in this type of situations. Since currently, there is no clean way of dropping out of a kaggle competition plus I have a little bit more time for kaggling than usual, I decided to use it as a way to try out things I have not done before. In short I am glad I stayed because I have learnt quite a bit and learning is my main goal on Kaggle.\n\nSorry guys and girls for the long post :-)", "685422": "> How do you explain the lack of top kagglers (grandmasters etc.) in this competition?\n\nMany were busy with NFL, plus several competitions launched near the same time.\n\nThat's the first explanation. The second one is that most top kagglers nowadays are computer vision experts, and this is not a computer vision competition ;)", "685519": "> Many were busy with NFL\n\nHmmm... NFL was launched 6 days before this one: do you mean that within the first 6 days of NFL competition everyone had already joined and fully committed to that? Difficult to imagine. \n\nI agree with the author of the post: this competition had good potential (I like the dirty data and big data parts).\nBut unfortunately it also has a poor design: 1 year of training vs. 2 years of test limits considerably the feature engineering, which is already limited because of the not-so-rich dataset.\nHowever, the most worrying thing it's the size of the leakage which I am afraid Kaggle's admins and organizers don't have under control (despite their claim). \nIf they had any clue about this, they would have not started this competition with this data => they were not aware of all this data being public => they still don't know 100% about the data which is publicly available => they cannot guarantee a leak-free private LB \nMy 2 cents \n\nI regret not joining the NFL competition, which was also my first choice...", "685803": "Well, I know several GMs who opted for NFL instead of ASHRAE.", "685818": "Thanks! Are you keeping a fraction only of the train and validation sets for speed purpose or do you have something else in mind?", "685856": "For evaluating new features and the like, I've used as little as 3% of the input data, though it's a bit more stable with 10-20%. This is why I have the \"sample-frac\" built into my CV splitter. For final training, of course, I use the full dataset." }, "source": "meta" }