As requested by @drexpz, I am extending a notebook to a discussion on weather files.\nAs per this Notebook, It seems that the weather files are not in local time. That basically distorts how weather features can inform the measurements (target labels) in an hourly basis. Features like \"Hours of the day\", \"day of the year\", Temperature, wind speed etc, may basically not be as informative as they should be. Especially to predict the hourly consumption of steam and chilled water, which highly depend on outdoor weather conditions.
\n\nIt seems that there is a mistake in the dataset. It would make sense that the organizers let everyone know what the time zone of each weather file is, so we can correct the issue. I guess we can work with it like it is, but it is a pity that we miss some information from weather conditions for what it seems to be a very easy-fix.
\n\nHowever, the issue could be worse if the measurements are in local time and the weather files are not. Which would mean a total mismatch between features and labels. That last point is still uncertain and should be furthermore investigated. Hopefully, there is not such a problem.
\n\nDoes somebody know about this?
", "messages": [ { "id": 658637, "postDate": "2019-10-26T09:46:38.957Z", "content": "As requested by @drexpz, I am extending a notebook to a discussion on weather files.\nAs per this Notebook, It seems that the weather files are not in local time. That basically distorts how weather features can inform the measurements (target labels) in an hourly basis. Features like \"Hours of the day\", \"day of the year\", Temperature, wind speed etc, may basically not be as informative as they should be. Especially to predict the hourly consumption of steam and chilled water, which highly depend on outdoor weather conditions.
\n\nIt seems that there is a mistake in the dataset. It would make sense that the organizers let everyone know what the time zone of each weather file is, so we can correct the issue. I guess we can work with it like it is, but it is a pity that we miss some information from weather conditions for what it seems to be a very easy-fix.
\n\nHowever, the issue could be worse if the measurements are in local time and the weather files are not. Which would mean a total mismatch between features and labels. That last point is still uncertain and should be furthermore investigated. Hopefully, there is not such a problem.
\n\nDoes somebody know about this?
", "rawMarkdown": "As requested by @drexpz, I am extending a notebook to a discussion on weather files.\nAs per this [Notebook](https://www.kaggle.com/jimeno/ashrae-weather-features-are-not-in-local-time), It seems that the weather files are not in local time. That basically distorts how weather features can inform the measurements (target labels) in an hourly basis. Features like \"Hours of the day\", \"day of the year\", Temperature, wind speed etc, may basically not be as informative as they should be. Especially to predict the hourly consumption of steam and chilled water, which highly depend on outdoor weather conditions.\n\nIt seems that there is a mistake in the dataset. It would make sense that the organizers let everyone know what the time zone of each weather file is, so we can correct the issue. I guess we can work with it like it is, but it is a pity that we miss some information from weather conditions for what it seems to be a very easy-fix.\n\nHowever, the issue could be worse if the measurements are in local time and the weather files are not. Which would mean a total mismatch between features and labels. That last point is still uncertain and should be furthermore investigated. Hopefully, there is not such a problem. \n\nDoes somebody know about this?", "votes": 27 }, { "id": 658783, "postDate": "2019-10-26T14:28:41.327Z", "content": "What's even more troubling is that you have buildings where power usage is during the expected day time hours, but the average temperature still peaks at 9pm.
\n\nFor example site id 13 has lowest temperature at noon, highest at 9pm. Office Building 1132's chilled water usage goes from 7 am to 5pm. (they only use chilled water)
\n\nSite id 3 has lows at 10am, highs at 8pm, but the electricity usage peaks from 7am to 6pm.
\n\nMy impression is that meter readings use local time while weather data uses a different time zone.
", "rawMarkdown": "What's even more troubling is that you have buildings where power usage is during the expected day time hours, but the average temperature still peaks at 9pm.\n\nFor example site id 13 has lowest temperature at noon, highest at 9pm. Office Building 1132's chilled water usage goes from 7 am to 5pm. (they only use chilled water)\n\nSite id 3 has lows at 10am, highs at 8pm, but the electricity usage peaks from 7am to 6pm.\n\nMy impression is that meter readings use local time while weather data uses a different time zone.\n", "votes": 6, "replies": [ { "id": 659775, "postDate": "2019-10-28T08:47:14.533Z", "content": "You can infer time from the meter readings, but the meter readings could conceivably be in any time offset from actual. Similarly, daylight saving time may or may not be observed. Also the meter readings themselves may or may not represent an \"hour beginning\" or \"hour ending\" time-stamp. At least in the US it all depends on the utility and the independent system operator (ISO) and they all do things differently.
", "rawMarkdown": "You can infer time from the meter readings, but the meter readings could conceivably be in any time offset from actual. Similarly, daylight saving time may or may not be observed. Also the meter readings themselves may or may not represent an \"hour beginning\" or \"hour ending\" time-stamp. At least in the US it all depends on the utility and the independent system operator (ISO) and they all do things differently. ", "isDeleted": true } ] }, { "id": 659024, "postDate": "2019-10-27T00:28:00.560Z", "content": "No doubt in my mind that the weather data is all UTC and that none of the sites are in that time zone.
\n\nI think I did a good job of determining the local time zone and than creating a new weather file with local times. Both catboost and lightgbm models seemed to have been smart enough to figure that out for themselves as local cv's got a touch worse using corrected local weather times.
\n\nOf course, always a good chance I did not correctly determine the time zones for the sites :)
", "rawMarkdown": "No doubt in my mind that the weather data is all UTC and that none of the sites are in that time zone.\n\nI think I did a good job of determining the local time zone and than creating a new weather file with local times. Both catboost and lightgbm models seemed to have been smart enough to figure that out for themselves as local cv's got a touch worse using corrected local weather times.\n\nOf course, always a good chance I did not correctly determine the time zones for the sites :)", "votes": 3, "replies": [ { "id": 659164, "postDate": "2019-10-27T07:28:38.437Z", "content": "I think the site_id can represent the time zone to a certain extent
", "rawMarkdown": "I think the site_id can represent the time zone to a certain extent" }, { "id": 659355, "postDate": "2019-10-27T14:06:20.010Z", "content": "Sites 1, 5, 12 could be in UTC time zone or close.
", "rawMarkdown": "Sites 1, 5, 12 could be in UTC time zone or close." } ] }, { "id": 661202, "postDate": "2019-10-30T02:34:39.023Z", "content": "I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")
\n\nHere's the notebook: \nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature
\n\nEnjoy!
", "rawMarkdown": "I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")\n\nHere's the notebook: \nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature\n\nEnjoy!", "votes": 1, "replies": [ { "id": 661561, "postDate": "2019-10-30T13:09:14.760Z", "content": "(I'll copy my comment here from the kernel)\nNice work! I think you must have gotten a handful right.\nMontreal could be it, as people have determined that 7 and 11 were the same, and I had hypothesized that they were both in Canada (unconverted Celsius data, July 1st holiday, American timezone)
\n\nI still don't think site 15 is in Canada (I see a small July 4 dip in office buildings but not on July 1). But if you look at other cities that have 0.90+, they're all in the northeast between Philadelphia and Toronto. Could be close.
\n\n1,5,12, I've hypothesized they were London, Loughborough, and Dublin.
\n\nYou got site 3 and site 14 in Philly, while they could use two siteIDs for the same city, would they use two different weather stations? Could be a nearby site such as Allentown, York, or Baltimore.
", "rawMarkdown": "(I'll copy my comment here from the kernel)\nNice work! I think you must have gotten a handful right.\nMontreal could be it, as people have determined that 7 and 11 were the same, and I had hypothesized that they were both in Canada (unconverted Celsius data, July 1st holiday, American timezone)\n\nI still don't think site 15 is in Canada (I see a small July 4 dip in office buildings but not on July 1). But if you look at other cities that have 0.90+, they're all in the northeast between Philadelphia and Toronto. Could be close.\n\n1,5,12, I've hypothesized they were London, Loughborough, and Dublin.\n\nYou got site 3 and site 14 in Philly, while they could use two siteIDs for the same city, would they use two different weather stations? Could be a nearby site such as Allentown, York, or Baltimore.\n", "votes": 2 }, { "id": 661594, "postDate": "2019-10-30T13:47:29.477Z", "content": "True, I totally agree with you.\nSite 15 could be an U.S. city just near Toronto, and 3 and 14 are just nearby (around Philly).\nPlus your inference about EU cities, pretty much we get all the right time zones and countries of these sites!
\n\nCheers!😁
", "rawMarkdown": "True, I totally agree with you.\nSite 15 could be an U.S. city just near Toronto, and 3 and 14 are just nearby (around Philly).\nPlus your inference about EU cities, pretty much we get all the right time zones and countries of these sites!\n\nCheers!😁 " } ] }, { "id": 658681, "postDate": "2019-10-26T10:59:19.743Z", "content": "Here is the other discussion around the same topic.
", "rawMarkdown": "[Here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112884#656125) is the other discussion around the same topic.", "votes": 1 }, { "id": 661154, "postDate": "2019-10-30T00:49:30.867Z", "content": "@claytonmiller @chrisbalbach Could the organizers double check if timezones for weather data and meter data match? Thank you.
", "rawMarkdown": "@claytonmiller @chrisbalbach Could the organizers double check if timezones for weather data and meter data match? Thank you.", "votes": 2, "replies": [ { "id": 661184, "postDate": "2019-10-30T01:49:06.247Z", "content": "Agree. Especially since official data description tab doesn’t mention weather timestamps? While meter timestamp is described.
\n\nweather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.
\n\nsite_id\nair_temperature - Degrees Celsius\ncloud_coverage - Portion of the sky covered in clouds, in oktas\ndew_temperature - Degrees Celsius\nprecip_depth_1_hr - Millimeters\nsea_level_pressure - Millibar/hectopascals\nwind_direction - Compass direction (0-360)\nwind_speed - Meters per second
", "rawMarkdown": "Agree. Especially since official data description tab doesn’t mention weather timestamps? While meter timestamp is described.\n\nweather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.\n\nsite_id\nair_temperature - Degrees Celsius\ncloud_coverage - Portion of the sky covered in clouds, in oktas\ndew_temperature - Degrees Celsius\nprecip_depth_1_hr - Millimeters\nsea_level_pressure - Millibar/hectopascals\nwind_direction - Compass direction (0-360)\nwind_speed - Meters per second" }, { "id": 717332, "postDate": "2020-01-13T03:26:28.563Z", "content": "Hi @sdoria -- The community was able to identify and find solutions for this issue and I didn't want to throw a wrench in the mix during the competition. This is actually pretty common real-world challenge and we think there were some really interesting and innovative solutions found :)
", "rawMarkdown": "Hi @sdoria -- The community was able to identify and find solutions for this issue and I didn't want to throw a wrench in the mix during the competition. This is actually pretty common real-world challenge and we think there were some really interesting and innovative solutions found :)", "votes": 1 } ] } ], "comments": [ { "id": 658783, "author_name": "S D", "author_url": "", "post_date": "2019-10-26T14:28:41.327000", "content": "What's even more troubling is that you have buildings where power usage is during the expected day time hours, but the average temperature still peaks at 9pm.
\n\nFor example site id 13 has lowest temperature at noon, highest at 9pm. Office Building 1132's chilled water usage goes from 7 am to 5pm. (they only use chilled water)
\n\nSite id 3 has lows at 10am, highs at 8pm, but the electricity usage peaks from 7am to 6pm.
\n\nMy impression is that meter readings use local time while weather data uses a different time zone.
", "votes": 6, "replies": [ { "id": 659775, "author_name": "", "author_url": "", "post_date": "2019-10-28T08:47:14.533000", "content": "You can infer time from the meter readings, but the meter readings could conceivably be in any time offset from actual. Similarly, daylight saving time may or may not be observed. Also the meter readings themselves may or may not represent an \"hour beginning\" or \"hour ending\" time-stamp. At least in the US it all depends on the utility and the independent system operator (ISO) and they all do things differently.
", "votes": 0, "replies": [] } ] }, { "id": 659024, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-10-27T00:28:00.560000", "content": "No doubt in my mind that the weather data is all UTC and that none of the sites are in that time zone.
\n\nI think I did a good job of determining the local time zone and than creating a new weather file with local times. Both catboost and lightgbm models seemed to have been smart enough to figure that out for themselves as local cv's got a touch worse using corrected local weather times.
\n\nOf course, always a good chance I did not correctly determine the time zones for the sites :)
", "votes": 3, "replies": [ { "id": 659164, "author_name": "Kevin Ling", "author_url": "", "post_date": "2019-10-27T07:28:38.437000", "content": "I think the site_id can represent the time zone to a certain extent
", "votes": 0, "replies": [] }, { "id": 659355, "author_name": "S D", "author_url": "", "post_date": "2019-10-27T14:06:20.010000", "content": "Sites 1, 5, 12 could be in UTC time zone or close.
", "votes": 0, "replies": [] } ] }, { "id": 661202, "author_name": "Chun Fu", "author_url": "", "post_date": "2019-10-30T02:34:39.023000", "content": "I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")
\n\nHere's the notebook: \nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature
\n\nEnjoy!
", "votes": 1, "replies": [ { "id": 661561, "author_name": "S D", "author_url": "", "post_date": "2019-10-30T13:09:14.760000", "content": "(I'll copy my comment here from the kernel)\nNice work! I think you must have gotten a handful right.\nMontreal could be it, as people have determined that 7 and 11 were the same, and I had hypothesized that they were both in Canada (unconverted Celsius data, July 1st holiday, American timezone)
\n\nI still don't think site 15 is in Canada (I see a small July 4 dip in office buildings but not on July 1). But if you look at other cities that have 0.90+, they're all in the northeast between Philadelphia and Toronto. Could be close.
\n\n1,5,12, I've hypothesized they were London, Loughborough, and Dublin.
\n\nYou got site 3 and site 14 in Philly, while they could use two siteIDs for the same city, would they use two different weather stations? Could be a nearby site such as Allentown, York, or Baltimore.
", "votes": 2, "replies": [] }, { "id": 661594, "author_name": "Chun Fu", "author_url": "", "post_date": "2019-10-30T13:47:29.477000", "content": "True, I totally agree with you.\nSite 15 could be an U.S. city just near Toronto, and 3 and 14 are just nearby (around Philly).\nPlus your inference about EU cities, pretty much we get all the right time zones and countries of these sites!
\n\nCheers!😁
", "votes": 0, "replies": [] } ] }, { "id": 658681, "author_name": "arnab", "author_url": "", "post_date": "2019-10-26T10:59:19.743000", "content": "Here is the other discussion around the same topic.
", "votes": 1, "replies": [] }, { "id": 661154, "author_name": "S D", "author_url": "", "post_date": "2019-10-30T00:49:30.867000", "content": "@claytonmiller @chrisbalbach Could the organizers double check if timezones for weather data and meter data match? Thank you.
", "votes": 2, "replies": [ { "id": 661184, "author_name": "Martin Elfstadius", "author_url": "", "post_date": "2019-10-30T01:49:06.247000", "content": "Agree. Especially since official data description tab doesn’t mention weather timestamps? While meter timestamp is described.
\n\nweather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.
\n\nsite_id\nair_temperature - Degrees Celsius\ncloud_coverage - Portion of the sky covered in clouds, in oktas\ndew_temperature - Degrees Celsius\nprecip_depth_1_hr - Millimeters\nsea_level_pressure - Millibar/hectopascals\nwind_direction - Compass direction (0-360)\nwind_speed - Meters per second
", "votes": 0, "replies": [] }, { "id": 717332, "author_name": "Clayton Miller", "author_url": "", "post_date": "2020-01-13T03:26:28.563000", "content": "Hi @sdoria -- The community was able to identify and find solutions for this issue and I didn't want to throw a wrench in the mix during the competition. This is actually pretty common real-world challenge and we think there were some really interesting and innovative solutions found :)
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "658637": "As requested by @drexpz, I am extending a notebook to a discussion on weather files.\nAs per this [Notebook](https://www.kaggle.com/jimeno/ashrae-weather-features-are-not-in-local-time), It seems that the weather files are not in local time. That basically distorts how weather features can inform the measurements (target labels) in an hourly basis. Features like \"Hours of the day\", \"day of the year\", Temperature, wind speed etc, may basically not be as informative as they should be. Especially to predict the hourly consumption of steam and chilled water, which highly depend on outdoor weather conditions.\n\nIt seems that there is a mistake in the dataset. It would make sense that the organizers let everyone know what the time zone of each weather file is, so we can correct the issue. I guess we can work with it like it is, but it is a pity that we miss some information from weather conditions for what it seems to be a very easy-fix.\n\nHowever, the issue could be worse if the measurements are in local time and the weather files are not. Which would mean a total mismatch between features and labels. That last point is still uncertain and should be furthermore investigated. Hopefully, there is not such a problem. \n\nDoes somebody know about this?", "658783": "What's even more troubling is that you have buildings where power usage is during the expected day time hours, but the average temperature still peaks at 9pm.\n\nFor example site id 13 has lowest temperature at noon, highest at 9pm. Office Building 1132's chilled water usage goes from 7 am to 5pm. (they only use chilled water)\n\nSite id 3 has lows at 10am, highs at 8pm, but the electricity usage peaks from 7am to 6pm.\n\nMy impression is that meter readings use local time while weather data uses a different time zone.\n", "659024": "No doubt in my mind that the weather data is all UTC and that none of the sites are in that time zone.\n\nI think I did a good job of determining the local time zone and than creating a new weather file with local times. Both catboost and lightgbm models seemed to have been smart enough to figure that out for themselves as local cv's got a touch worse using corrected local weather times.\n\nOf course, always a good chance I did not correctly determine the time zones for the sites :)", "661202": "I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")\n\nHere's the notebook: \nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature\n\nEnjoy!", "658681": "[Here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112884#656125) is the other discussion around the same topic.", "661154": "@claytonmiller @chrisbalbach Could the organizers double check if timezones for weather data and meter data match? Thank you." } }