![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F72177%2F1d66f9d8f17be2cd0b13bfd666795058%2Fel1.png?generation=1572001313000036&alt=media\")
\nIt is impossible to use statistics to decide whether the usage has just reached a new level and the December mean should applied for all of the future, or whether the early year lower level will work again for the next year, right?When you plot electricity for some buildings, it seems that there may be structural changes. But how can one ever confirm that, when only 1 year is given?
\n\nFor example\n\nIt is impossible to use statistics to decide whether the usage has just reached a new level and the December mean should applied for all of the future, or whether the early year lower level will work again for the next year, right?
Or for example\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F72177%2F02706d58944505e5843675a2154ff92f%2Fel2.png?generation=1572001435882495&alt=media\")
\nHere at least January and December levels match, but it is impossible to know whether the May/June usage is an outlier and is going to repeat?
Does that makes sense? Do you see a way to compensate for these ambiguities?
\n\nMaking the wrong assumption could worsen the score considerably. Maybe only tedious LB probing for each building can resolve that? Maybe we should get another training year?
\n\nWhat do you think?
", "messages": [ { "id": "657682", "postDate": "10/25/2019 11:08:02", "content": "When you plot electricity for some buildings, it seems that there may be structural changes. But how can one ever confirm that, when only 1 year is given?
\n\nFor example\n\nIt is impossible to use statistics to decide whether the usage has just reached a new level and the December mean should applied for all of the future, or whether the early year lower level will work again for the next year, right?
Or for example\n\nHere at least January and December levels match, but it is impossible to know whether the May/June usage is an outlier and is going to repeat?
Does that makes sense? Do you see a way to compensate for these ambiguities?
\n\nMaking the wrong assumption could worsen the score considerably. Maybe only tedious LB probing for each building can resolve that? Maybe we should get another training year?
\n\nWhat do you think?
", "rawMarkdown": "When you plot electricity for some buildings, it seems that there may be structural changes. But how can one ever confirm that, when only 1 year is given?\n\nFor example\n\nIt is impossible to use statistics to decide whether the usage has just reached a new level and the December mean should applied for all of the future, or whether the early year lower level will work again for the next year, right?\n\nOr for example\n\nHere at least January and December levels match, but it is impossible to know whether the May/June usage is an outlier and is going to repeat?\n\nDoes that makes sense? Do you see a way to compensate for these ambiguities?\n\nMaking the wrong assumption could worsen the score considerably. Maybe only tedious LB probing for each building can resolve that? Maybe we should get another training year?\n\nWhat do you think?", "votes": null }, { "id": "658050", "postDate": "10/25/2019 16:16:13", "content": "very good points, I also have done some experiments and now begin to think .. about good validation.\nFirst I used last 20% as hold out set and got surprisingly a good single model lb 1.20, then I tuned something, validation got better (1.10 -> 1.06) but lb went up to 1.26. In these experiments I retrained on all data for test prediction.\nBest kernels use \"stratify data by building_id\", does this mean split train/valid by building_id in each fold? I tried this and got horrible validation error of 1.59. This means that I predict unknown building_id's, which don't make sense to me.\nRandom subset results in very low validation error of 0.5 or so.\nSo far I don't find a proper validation schema that simulates the train/test split. To capture drift over a year I need at least the whole year in train.\nIf we would have 2 years of training data I can probably safe use first year for train, second for valid (as you said).
", "rawMarkdown": "very good points, I also have done some experiments and now begin to think .. about good validation.\nFirst I used last 20% as hold out set and got surprisingly a good single model lb 1.20, then I tuned something, validation got better (1.10 -> 1.06) but lb went up to 1.26. In these experiments I retrained on all data for test prediction.\nBest kernels use \"stratify data by building_id\", does this mean split train/valid by building_id in each fold? I tried this and got horrible validation error of 1.59. This means that I predict unknown building_id's, which don't make sense to me.\nRandom subset results in very low validation error of 0.5 or so.\nSo far I don't find a proper validation schema that simulates the train/test split. To capture drift over a year I need at least the whole year in train.\nIf we would have 2 years of training data I can probably safe use first year for train, second for valid (as you said).", "votes": null }, { "id": "658091", "postDate": "10/25/2019 16:52:29", "content": "1393(building id) probably belongs to 15(site id), and you might see that this category has common missing(?) periods between feb/apr. It seems like someone artificially interpolated the gap by a line(including air temperature and dew temperature). Moreover, there are significant losses in the other features also I think. We have to deal with this.
", "rawMarkdown": "1393(building id) probably belongs to 15(site id), and you might see that this category has common missing(?) periods between feb/apr. It seems like someone artificially interpolated the gap by a line(including air temperature and dew temperature). Moreover, there are significant losses in the other features also I think. We have to deal with this.", "votes": null }, { "id": "658103", "postDate": "10/25/2019 17:02:08", "content": "Well, yes. But will there be a gap in 2017? There is absolutely nothing - apart from LB probing - that can answer that.
", "rawMarkdown": "Well, yes. But will there be a gap in 2017? There is absolutely nothing - apart from LB probing - that can answer that.", "votes": null }, { "id": "658107", "postDate": "10/25/2019 17:07:25", "content": "Yes, LB probing will be the best way for validation, but 2 subs a day really su...
", "rawMarkdown": "Yes, LB probing will be the best way for validation, but 2 subs a day really su...", "votes": null }, { "id": "658123", "postDate": "10/25/2019 17:24:56", "content": "Stratification with building id means the distribution of buildings in train and val is same.
", "rawMarkdown": "Stratification with building id means the distribution of buildings in train and val is same.", "votes": null }, { "id": "664985", "postDate": "11/04/2019 13:47:49", "content": "You can only assume that the label is highly related to weather data, otherwise, it is quite hard!
", "rawMarkdown": "You can only assume that the label is highly related to weather data, otherwise, it is quite hard!", "votes": null } ], "comments": [ { "id": 658050, "author_name": "mjahrer", "author_url": "", "post_date": "10/25/2019 16:16:13", "content": "very good points, I also have done some experiments and now begin to think .. about good validation.\nFirst I used last 20% as hold out set and got surprisingly a good single model lb 1.20, then I tuned something, validation got better (1.10 -> 1.06) but lb went up to 1.26. In these experiments I retrained on all data for test prediction.\nBest kernels use \"stratify data by building_id\", does this mean split train/valid by building_id in each fold? I tried this and got horrible validation error of 1.59. This means that I predict unknown building_id's, which don't make sense to me.\nRandom subset results in very low validation error of 0.5 or so.\nSo far I don't find a proper validation schema that simulates the train/test split. To capture drift over a year I need at least the whole year in train.\nIf we would have 2 years of training data I can probably safe use first year for train, second for valid (as you said).
", "votes": null, "replies": [ { "id": 658123, "author_name": "suchith0312", "author_url": "", "post_date": "10/25/2019 17:24:56", "content": "Stratification with building id means the distribution of buildings in train and val is same.
", "votes": null, "replies": [] } ] }, { "id": 658091, "author_name": "hanjoonchoe", "author_url": "", "post_date": "10/25/2019 16:52:29", "content": "1393(building id) probably belongs to 15(site id), and you might see that this category has common missing(?) periods between feb/apr. It seems like someone artificially interpolated the gap by a line(including air temperature and dew temperature). Moreover, there are significant losses in the other features also I think. We have to deal with this.
", "votes": null, "replies": [ { "id": 658103, "author_name": "superant", "author_url": "", "post_date": "10/25/2019 17:02:08", "content": "Well, yes. But will there be a gap in 2017? There is absolutely nothing - apart from LB probing - that can answer that.
", "votes": null, "replies": [] }, { "id": 658107, "author_name": "mjahrer", "author_url": "", "post_date": "10/25/2019 17:07:25", "content": "Yes, LB probing will be the best way for validation, but 2 subs a day really su...
", "votes": null, "replies": [] } ] }, { "id": 664985, "author_name": "longyin2", "author_url": "", "post_date": "11/04/2019 13:47:49", "content": "You can only assume that the label is highly related to weather data, otherwise, it is quite hard!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "657682": "When you plot electricity for some buildings, it seems that there may be structural changes. But how can one ever confirm that, when only 1 year is given?\n\nFor example\n\nIt is impossible to use statistics to decide whether the usage has just reached a new level and the December mean should applied for all of the future, or whether the early year lower level will work again for the next year, right?\n\nOr for example\n\nHere at least January and December levels match, but it is impossible to know whether the May/June usage is an outlier and is going to repeat?\n\nDoes that makes sense? Do you see a way to compensate for these ambiguities?\n\nMaking the wrong assumption could worsen the score considerably. Maybe only tedious LB probing for each building can resolve that? Maybe we should get another training year?\n\nWhat do you think?", "658050": "very good points, I also have done some experiments and now begin to think .. about good validation.\nFirst I used last 20% as hold out set and got surprisingly a good single model lb 1.20, then I tuned something, validation got better (1.10 -> 1.06) but lb went up to 1.26. In these experiments I retrained on all data for test prediction.\nBest kernels use \"stratify data by building_id\", does this mean split train/valid by building_id in each fold? I tried this and got horrible validation error of 1.59. This means that I predict unknown building_id's, which don't make sense to me.\nRandom subset results in very low validation error of 0.5 or so.\nSo far I don't find a proper validation schema that simulates the train/test split. To capture drift over a year I need at least the whole year in train.\nIf we would have 2 years of training data I can probably safe use first year for train, second for valid (as you said).", "658091": "1393(building id) probably belongs to 15(site id), and you might see that this category has common missing(?) periods between feb/apr. It seems like someone artificially interpolated the gap by a line(including air temperature and dew temperature). Moreover, there are significant losses in the other features also I think. We have to deal with this.", "658103": "Well, yes. But will there be a gap in 2017? There is absolutely nothing - apart from LB probing - that can answer that.", "658107": "Yes, LB probing will be the best way for validation, but 2 subs a day really su...", "658123": "Stratification with building id means the distribution of buildings in train and val is same.", "664985": "You can only assume that the label is highly related to weather data, otherwise, it is quite hard!" }, "source": "meta" }