This R kernel by kxx has really got me thinking into the time component and expected y-o-y behaviour of the data along with what feature engineering will really work.
\n\nI have ported the code as a Python kernel as well for the community to play around with.
\n\nWith the most basic feature engineering (just hour, weekday and holidays), the model scores 1.10 on the public LB beating most other benchmarks shared at the time of posting.
\n\nThe only special treatment seems to be dividing the data into two halves by time, and building two models and averaging predictions. Just this gives a very competitive score.
\n\nI am still thinking about why this simple beautiful code is able to beat so many models which do all sorts of data processing, alignments, lags and feature engineering.
\n\nOr are we just shooting in the dark?
", "messages": [ { "id": 666013, "postDate": "2019-11-05T16:32:47.820Z", "content": "This R kernel by kxx has really got me thinking into the time component and expected y-o-y behaviour of the data along with what feature engineering will really work.
\n\nI have ported the code as a Python kernel as well for the community to play around with.
\n\nWith the most basic feature engineering (just hour, weekday and holidays), the model scores 1.10 on the public LB beating most other benchmarks shared at the time of posting.
\n\nThe only special treatment seems to be dividing the data into two halves by time, and building two models and averaging predictions. Just this gives a very competitive score.
\n\nI am still thinking about why this simple beautiful code is able to beat so many models which do all sorts of data processing, alignments, lags and feature engineering.
\n\nOr are we just shooting in the dark?
", "rawMarkdown": "This [R kernel](https://www.kaggle.com/kailex/ac-dc) by [kxx](https://www.kaggle.com/kailex) has really got me thinking into the time component and expected y-o-y behaviour of the data along with what feature engineering will really work.\n\nI have ported the code as a [Python kernel](https://www.kaggle.com/rohanrao/ashrae-half-and-half) as well for the community to play around with.\n\nWith the most basic feature engineering (just hour, weekday and holidays), the model scores 1.10 on the public LB beating most other benchmarks shared at the time of posting.\n\n- No dropping of data\n- No creation of statistical aggregate features\n- No creation of lag features\n- No aligning of timestamp\n\nThe only special treatment seems to be dividing the data into two halves by time, and building two models and averaging predictions. Just this gives a very competitive score.\n\nI am still thinking about why this simple beautiful code is able to beat so many models which do all sorts of data processing, alignments, lags and feature engineering.\n\nOr are we just shooting in the dark?", "votes": 60 }, { "id": 669856, "postDate": "2019-11-10T15:19:14.033Z", "content": "The importance of the holiday list is questionable because holidays vary by nation and to some degree by regions within nations. The US does not celebrate the Lunar New Year which is a very big deal in Asia. The seasonal split theory is valid only if all buildings are in either the northern or southern hemisphere.
\n\nI got a score of 1.10 using Vopani's kernel and I show up about 20 places higher than him on the LB (which seems unfair but I did upvote the kernel!). Looks like lots of folks have scores of 1.09 and 1.10.
\n\nKickback
", "rawMarkdown": " The importance of the holiday list is questionable because holidays vary by nation and to some degree by regions within nations. The US does not celebrate the Lunar New Year which is a very big deal in Asia. The seasonal split theory is valid only if all buildings are in either the northern or southern hemisphere.\n\n I got a score of 1.10 using Vopani's kernel and I show up about 20 places higher than him on the LB (which seems unfair but I did upvote the kernel!). Looks like lots of folks have scores of 1.09 and 1.10.\n\nKickback", "votes": 3 }, { "id": 666234, "postDate": "2019-11-05T22:55:34.820Z", "content": "It's actually madness.
\n\nI saw it yesterday just after making a submission where I trained on the whole dataset before submitting. So I went back and did the exact same process except I trained two models on one half each (just like kxx), averaged them and resubmitted.
\n\nMy score (1.17) stayed exactly the same (though when I \"sort\" my scores the half-half version goes above the original one, so presumably it's a tiny bit better).
\n\nPoint being: IN GENERAL half-half doesn't seem to make a big difference. My guess is that there is something specific about these kernels that makes half-half good.
\n\nWhich is a very bad sign, because there doesn't seem to be any logical reason for that.
", "rawMarkdown": "It's actually madness.\n\nI saw it yesterday just after making a submission where I trained on the whole dataset before submitting. So I went back and did the exact same process except I trained two models on one half each (just like kxx), averaged them and resubmitted.\n\nMy score (1.17) stayed exactly the same (though when I \"sort\" my scores the half-half version goes above the original one, so presumably it's a tiny bit better). \n\nPoint being: IN GENERAL half-half doesn't seem to make a big difference. My guess is that there is something specific about these kernels that makes half-half good. \n\nWhich is a very bad sign, because there doesn't seem to be any logical reason for that.", "votes": 3, "replies": [ { "id": 666325, "postDate": "2019-11-06T02:00:56.747Z", "content": "update: I resubmitted the python kernel by @rohanrao exactly the same except with the bad site 1 readings removed and scored 1.10 but it was one of the higher 1.10's according to the leader-board, so I think its safe to say that this is an improvement.
\n\nSo that's sliiiightly good news because it means that doing something sensible increases your score by a small amount, but I still have no idea what the hell is going on in this comp.
", "rawMarkdown": "update: I resubmitted the python kernel by @rohanrao exactly the same except with the bad site 1 readings removed and scored 1.10 but it was one of the higher 1.10's according to the leader-board, so I think its safe to say that this is an improvement. \n\nSo that's sliiiightly good news because it means that doing something sensible increases your score by a small amount, but I still have no idea what the hell is going on in this comp.", "votes": 7 }, { "id": 685409, "postDate": "2019-12-01T15:47:32.600Z", "content": "I can confirm this observation
", "rawMarkdown": "I can confirm this observation" } ] }, { "id": 666232, "postDate": "2019-11-05T22:54:14.303Z", "content": "shooting in the dark is my vote - my current best model does a nice job of predicting the training data - 2016 with no strong indications of over fit.
\n\nMy leader board score still a long ways to go - I think prediction of special events in 2017 and 2018 will be key and therefore lots of shots in the dark.
", "rawMarkdown": "shooting in the dark is my vote - my current best model does a nice job of predicting the training data - 2016 with no strong indications of over fit.\n\nMy leader board score still a long ways to go - I think prediction of special events in 2017 and 2018 will be key and therefore lots of shots in the dark.", "votes": 3, "replies": [ { "id": 666241, "postDate": "2019-11-05T23:28:20.853Z", "content": "oh god.
\n\nWe had better not have to search for like, power outages in dublin or some cancer.
", "rawMarkdown": "oh god. \n\nWe had better not have to search for like, power outages in dublin or some cancer. ", "votes": 1 } ] }, { "id": 666185, "postDate": "2019-11-05T21:14:49.163Z", "content": "Yes, it's interesting and worth to investigate more. If the main reason is the 2 x 6months CV or not. Also why the OOF score is CV1: 1.33 and CV2: 1.50. I've similar imbalanced local RMSE results with 4 x 3months CV.
\n\nIt could also mean that, as you said, we're shooting in the dark, if so then big shake up is expected at the end.
", "rawMarkdown": "Yes, it's interesting and worth to investigate more. If the main reason is the 2 x 6months CV or not. Also why the OOF score is CV1: 1.33 and CV2: 1.50. I've similar imbalanced local RMSE results with 4 x 3months CV.\n\nIt could also mean that, as you said, we're shooting in the dark, if so then big shake up is expected at the end. ", "votes": 1, "replies": [ { "id": 666518, "postDate": "2019-11-06T07:05:04.440Z", "content": "4×3months CV method has a bad result in LB score like 1.21 or so
", "rawMarkdown": "4×3months CV method has a bad result in LB score like 1.21 or so", "votes": 1 }, { "id": 666538, "postDate": "2019-11-06T07:44:03.513Z", "content": "Thanks @mpware I was going to try 4×3months CV method today. You saved me a bit of time.
\n\nI also observed a similar imbalanced local RMSE results in some of my models and they seem to give a better public LB score. So I do not trust any of my CV methods so far.
", "rawMarkdown": "Thanks @mpware I was going to try 4×3months CV method today. You saved me a bit of time.\n\nI also observed a similar imbalanced local RMSE results in some of my models and they seem to give a better public LB score. So I do not trust any of my CV methods so far." }, { "id": 667480, "postDate": "2019-11-07T09:29:30.877Z", "content": "Have you tried to make separate models for each 6 months. I will try to make it. I hope that will improve the score a little bit 🙏 . If you had try it, please save me time.
", "rawMarkdown": "Have you tried to make separate models for each 6 months. I will try to make it. I hope that will improve the score a little bit 🙏 . If you had try it, please save me time.", "votes": 1 }, { "id": 667914, "postDate": "2019-11-07T20:12:51.940Z", "content": "The more I remove features the more LB score improves, it's totally negatively correlated to my local CV. I admit that some features I added could generate overfit so I'm removing them progressively.
", "rawMarkdown": "The more I remove features the more LB score improves, it's totally negatively correlated to my local CV. I admit that some features I added could generate overfit so I'm removing them progressively." }, { "id": 674185, "postDate": "2019-11-16T01:25:47.733Z", "content": "@mpware When you say \"2 x 6months CV\"/\"4 x 3months CV\", that's not enough information:
\n\nDo you mean LGBM, with a rolling timeseries split (like \"Half-and-Half\" notebook), or just a simple K-fold split (stratified or non-stratified, and if stratified, by what? building_id? And of course, turn date-shuffling off.) It looks like you meant \"simple K-fold split\", thus \"simple 2-fold\"/\"4-fold\"
Did you not use RMSLE rather than RMSE as evaluation metric? (\"I've similar imbalanced local RMSE results\") And what did you use as loss function in training? If you used the wrong functions, then sure your RMSLE results would be worse.
Third, the imbalanced OOF scores (CV1: 1.33, CV2: 1.50) are not necessarily bad. Rerun with RMSLE. And best to quote your CV-LB offset, rather than just CV numbers. A classifier with a so-so LB score, but tight CV-LB correlation, typically performs better than the opposite. Whereas overfitting is poison.
Allways look on the bright side of life !
\n\n\n\nSome things in life are bad\nThey can really make you mad\nOther things just make you swear and curse\nWhen you're chewing on life's gristle\nDon't grumble, give a whistle\nAnd this'll help things turn out for the best\nAnd\nAlways look on the bright side of life\nAlways look on the light side of life\nIf life seems jolly rotten\nThere's something you've forgotten\nAnd that's to laugh and smile and dance and sing\nWhen you're feeling in the dumps\nDon't be silly chumps\nJust purse your lips and whistle, that's the thing\nAnd\nAlways look on the bright side of life\n(Come on)\nAlways look on the right side of life\nFor life is quite absurd\nAnd death's the final word\nYou must always face the curtain with a bow\nForget about your sin\nGive the audience a grin\nEnjoy it,…
\n\nQuite this competition, da ya think?
\n\nThis is what I feel right now. ;-)
", "rawMarkdown": "Allways look on the bright side of life !\n\nhttps://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=2ahUKEwip8ePq49_lAhWQHRQKHae4B64QtwIwAHoECBMQAQ&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVOAtCOsNuVM&usg=AOvVaw2nXYSDLwmFznNo6CvwiV4x\n\nSome things in life are bad\nThey can really make you mad\nOther things just make you swear and curse\nWhen you're chewing on life's gristle\nDon't grumble, give a whistle\nAnd this'll help things turn out for the best\nAnd\nAlways look on the bright side of life\nAlways look on the light side of life\nIf life seems jolly rotten\nThere's something you've forgotten\nAnd that's to laugh and smile and dance and sing\nWhen you're feeling in the dumps\nDon't be silly chumps\nJust purse your lips and whistle, that's the thing\nAnd\nAlways look on the bright side of life\n(Come on)\nAlways look on the right side of life\nFor life is quite absurd\nAnd death's the final word\nYou must always face the curtain with a bow\nForget about your sin\nGive the audience a grin\nEnjoy it,…\n\nQuite this competition, da ya think?\n\n**This is what I feel right now. ;-)**\n" }, { "id": 681918, "postDate": "2019-11-26T17:23:07.657Z", "content": "It depends on what basis did you divide it. If there is no logic behind it, I think it must be a coincidence.🙄
", "rawMarkdown": "It depends on what basis did you divide it. If there is no logic behind it, I think it must be a coincidence.🙄 " }, { "id": 671411, "postDate": "2019-11-12T16:49:36.957Z", "content": "How did this work? Must be an accident!
", "rawMarkdown": "How did this work? Must be an accident! " }, { "id": 671176, "postDate": "2019-11-12T10:58:40.997Z", "content": "@rohanrao Could you please let us know your thought as well?
", "rawMarkdown": "@rohanrao Could you please let us know your thought as well?" }, { "id": 678107, "postDate": "2019-11-21T01:56:48.603Z", "rawMarkdown": "", "isDeleted": true }, { "id": 666115, "postDate": "2019-11-05T18:38:39.327Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 666606, "postDate": "2019-11-06T09:27:53.780Z", "content": "@raphael1123 the kernel you linked already had 1.11 using the exactly same strategy mentioned in this topic! lol\nHave a look at that kernel again, there's a print of the folds and a link to the best version\nBy using the unshuffled KFold @kimtaegwan gets half training and half validation splits by date, because the dataset is sorted by timestamp.\n```
\n\n2 -fold\nfold : 0\nval_set_range : 2016-01-01 00:00:00 2016-07-04 17:00:00\nfold : 1
\n\n```
", "rawMarkdown": "@raphael1123 the kernel you linked already had 1.11 using the exactly same strategy mentioned in this topic! lol\nHave a look at that kernel again, there's a print of the folds and a link to the [best version](https://www.kaggle.com/kimtaegwan/what-s-your-cv-method?scriptVersionId=22526687)\nBy using the unshuffled KFold @kimtaegwan gets half training and half validation splits by date, because the dataset is sorted by timestamp.\n```\nfold = KFold(n_splits=folds, shuffle=False, random_state=42)\n--------------------------------------------------\n2 -fold\nfold : 0\nval_set_range : 2016-01-01 00:00:00 2016-07-04 17:00:00\nfold : 1\nval_set_range : 2016-07-04 17:00:00 2016-12-31 23:00:00\n--------------------------------------------------\n```", "votes": 2 }, { "id": 666693, "postDate": "2019-11-06T11:37:28.300Z", "content": "That's a similar strategy👍 @hmendonca \nIn another kernel of mine, I tried Group2fold (January to June, July to December) and achieved 1.10. What's going on? haha
", "rawMarkdown": "That's a similar strategy👍 @hmendonca \nIn [another kernel ](https://www.kaggle.com/kimtaegwan/what-s-your-cv-method?scriptVersionId=22905017)of mine, I tried Group2fold (January to June, July to December) and achieved 1.10. What's going on? haha", "votes": 3 }, { "id": 671113, "postDate": "2019-11-12T09:08:13.227Z", "content": "@raphael1123 I didn't find the code for stratified by building_id. Could you please point me to that?
The importance of the holiday list is questionable because holidays vary by nation and to some degree by regions within nations. The US does not celebrate the Lunar New Year which is a very big deal in Asia. The seasonal split theory is valid only if all buildings are in either the northern or southern hemisphere.
\n\nI got a score of 1.10 using Vopani's kernel and I show up about 20 places higher than him on the LB (which seems unfair but I did upvote the kernel!). Looks like lots of folks have scores of 1.09 and 1.10.
\n\nKickback
", "votes": 3, "replies": [] }, { "id": 666234, "author_name": "Louka Ewington-Pitsos", "author_url": "", "post_date": "2019-11-05T22:55:34.820000", "content": "It's actually madness.
\n\nI saw it yesterday just after making a submission where I trained on the whole dataset before submitting. So I went back and did the exact same process except I trained two models on one half each (just like kxx), averaged them and resubmitted.
\n\nMy score (1.17) stayed exactly the same (though when I \"sort\" my scores the half-half version goes above the original one, so presumably it's a tiny bit better).
\n\nPoint being: IN GENERAL half-half doesn't seem to make a big difference. My guess is that there is something specific about these kernels that makes half-half good.
\n\nWhich is a very bad sign, because there doesn't seem to be any logical reason for that.
", "votes": 3, "replies": [ { "id": 666325, "author_name": "Louka Ewington-Pitsos", "author_url": "", "post_date": "2019-11-06T02:00:56.747000", "content": "update: I resubmitted the python kernel by @rohanrao exactly the same except with the bad site 1 readings removed and scored 1.10 but it was one of the higher 1.10's according to the leader-board, so I think its safe to say that this is an improvement.
\n\nSo that's sliiiightly good news because it means that doing something sensible increases your score by a small amount, but I still have no idea what the hell is going on in this comp.
", "votes": 7, "replies": [] }, { "id": 685409, "author_name": "pspenano", "author_url": "", "post_date": "2019-12-01T15:47:32.600000", "content": "I can confirm this observation
", "votes": 0, "replies": [] } ] }, { "id": 666232, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-11-05T22:54:14.303000", "content": "shooting in the dark is my vote - my current best model does a nice job of predicting the training data - 2016 with no strong indications of over fit.
\n\nMy leader board score still a long ways to go - I think prediction of special events in 2017 and 2018 will be key and therefore lots of shots in the dark.
", "votes": 3, "replies": [ { "id": 666241, "author_name": "Louka Ewington-Pitsos", "author_url": "", "post_date": "2019-11-05T23:28:20.853000", "content": "oh god.
\n\nWe had better not have to search for like, power outages in dublin or some cancer.
", "votes": 1, "replies": [] } ] }, { "id": 666185, "author_name": "MPWARE", "author_url": "", "post_date": "2019-11-05T21:14:49.163000", "content": "Yes, it's interesting and worth to investigate more. If the main reason is the 2 x 6months CV or not. Also why the OOF score is CV1: 1.33 and CV2: 1.50. I've similar imbalanced local RMSE results with 4 x 3months CV.
\n\nIt could also mean that, as you said, we're shooting in the dark, if so then big shake up is expected at the end.
", "votes": 1, "replies": [ { "id": 666518, "author_name": "Dxx", "author_url": "", "post_date": "2019-11-06T07:05:04.440000", "content": "4×3months CV method has a bad result in LB score like 1.21 or so
", "votes": 1, "replies": [] }, { "id": 666538, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-06T07:44:03.513000", "content": "Thanks @mpware I was going to try 4×3months CV method today. You saved me a bit of time.
\n\nI also observed a similar imbalanced local RMSE results in some of my models and they seem to give a better public LB score. So I do not trust any of my CV methods so far.
", "votes": 0, "replies": [] }, { "id": 667480, "author_name": "Aymen Khelifi", "author_url": "", "post_date": "2019-11-07T09:29:30.877000", "content": "Have you tried to make separate models for each 6 months. I will try to make it. I hope that will improve the score a little bit 🙏 . If you had try it, please save me time.
", "votes": 1, "replies": [] }, { "id": 667914, "author_name": "MPWARE", "author_url": "", "post_date": "2019-11-07T20:12:51.940000", "content": "The more I remove features the more LB score improves, it's totally negatively correlated to my local CV. I admit that some features I added could generate overfit so I'm removing them progressively.
", "votes": 0, "replies": [] }, { "id": 674185, "author_name": "Stephen McInerney", "author_url": "", "post_date": "2019-11-16T01:25:47.733000", "content": "@mpware When you say \"2 x 6months CV\"/\"4 x 3months CV\", that's not enough information:
\n\nDo you mean LGBM, with a rolling timeseries split (like \"Half-and-Half\" notebook), or just a simple K-fold split (stratified or non-stratified, and if stratified, by what? building_id? And of course, turn date-shuffling off.) It looks like you meant \"simple K-fold split\", thus \"simple 2-fold\"/\"4-fold\"
Did you not use RMSLE rather than RMSE as evaluation metric? (\"I've similar imbalanced local RMSE results\") And what did you use as loss function in training? If you used the wrong functions, then sure your RMSLE results would be worse.
Third, the imbalanced OOF scores (CV1: 1.33, CV2: 1.50) are not necessarily bad. Rerun with RMSLE. And best to quote your CV-LB offset, rather than just CV numbers. A classifier with a so-so LB score, but tight CV-LB correlation, typically performs better than the opposite. Whereas overfitting is poison.
Allways look on the bright side of life !
\n\n\n\nSome things in life are bad\nThey can really make you mad\nOther things just make you swear and curse\nWhen you're chewing on life's gristle\nDon't grumble, give a whistle\nAnd this'll help things turn out for the best\nAnd\nAlways look on the bright side of life\nAlways look on the light side of life\nIf life seems jolly rotten\nThere's something you've forgotten\nAnd that's to laugh and smile and dance and sing\nWhen you're feeling in the dumps\nDon't be silly chumps\nJust purse your lips and whistle, that's the thing\nAnd\nAlways look on the bright side of life\n(Come on)\nAlways look on the right side of life\nFor life is quite absurd\nAnd death's the final word\nYou must always face the curtain with a bow\nForget about your sin\nGive the audience a grin\nEnjoy it,…
\n\nQuite this competition, da ya think?
\n\nThis is what I feel right now. ;-)
", "votes": 0, "replies": [] }, { "id": 681918, "author_name": "Shahules", "author_url": "", "post_date": "2019-11-26T17:23:07.657000", "content": "It depends on what basis did you divide it. If there is no logic behind it, I think it must be a coincidence.🙄
", "votes": 0, "replies": [] }, { "id": 671411, "author_name": "Shahules", "author_url": "", "post_date": "2019-11-12T16:49:36.957000", "content": "How did this work? Must be an accident!
", "votes": 0, "replies": [] }, { "id": 671176, "author_name": "arnab", "author_url": "", "post_date": "2019-11-12T10:58:40.997000", "content": "@rohanrao Could you please let us know your thought as well?
", "votes": 0, "replies": [] }, { "id": 678107, "author_name": "", "author_url": "", "post_date": "2019-11-21T01:56:48.603000", "content": "", "votes": 0, "replies": [] }, { "id": 666115, "author_name": "", "author_url": "", "post_date": "2019-11-05T18:38:39.327000", "content": "", "votes": 1, "replies": [ { "id": 666606, "author_name": "Henrique Mendonça", "author_url": "", "post_date": "2019-11-06T09:27:53.780000", "content": "@raphael1123 the kernel you linked already had 1.11 using the exactly same strategy mentioned in this topic! lol\nHave a look at that kernel again, there's a print of the folds and a link to the best version\nBy using the unshuffled KFold @kimtaegwan gets half training and half validation splits by date, because the dataset is sorted by timestamp.\n```
\n\n2 -fold\nfold : 0\nval_set_range : 2016-01-01 00:00:00 2016-07-04 17:00:00\nfold : 1
\n\n```
", "votes": 2, "replies": [] }, { "id": 666693, "author_name": "Taegwan Kim", "author_url": "", "post_date": "2019-11-06T11:37:28.300000", "content": "That's a similar strategy👍 @hmendonca \nIn another kernel of mine, I tried Group2fold (January to June, July to December) and achieved 1.10. What's going on? haha
", "votes": 3, "replies": [] }, { "id": 671113, "author_name": "arnab", "author_url": "", "post_date": "2019-11-12T09:08:13.227000", "content": "@raphael1123 I didn't find the code for stratified by building_id. Could you please point me to that?