The has been a lot of discussion about supplementing the data set with some kind of geolocation data. This might be very helpful for modeling because it could potentially give us a way of figuring out national holidays or give us access to some additional weather/climate data for different locations. So, I went to the ASHRAE web site and discovered that ASHRAE is comprised of 15 regions. The map of these regions is updated annually and can be downloaded from here.
\n\nThe number of regions is suspiciously similar to the number of the unique values of the site_id variable in the competition data set. There might be a connection here and I think it is worth investigating. Right now, I do not have time to work on this competition, but I encourage you to play with this idea and report your finding in this thread.
The has been a lot of discussion about supplementing the data set with some kind of geolocation data. This might be very helpful for modeling because it could potentially give us a way of figuring out national holidays or give us access to some additional weather/climate data for different locations. So, I went to the ASHRAE web site and discovered that ASHRAE is comprised of 15 regions. The map of these regions is updated annually and can be downloaded from here.
\n\nThe number of regions is suspiciously similar to the number of the unique values of the site_id variable in the competition data set. There might be a connection here and I think it is worth investigating. Right now, I do not have time to work on this competition, but I encourage you to play with this idea and report your finding in this thread.
Good catch! Thanks for sharing. 👍
", "rawMarkdown": "Good catch! Thanks for sharing. 👍", "votes": null }, { "id": "652326", "postDate": "10/18/2019 17:31:15", "content": "Interesting find! However:\nThe site_id variable corresponds to weather stations. Some of those ASHRAE regions are quite large geographically. It wouldn't make sense to have data from a unique weather station for each region, unless somehow the buildings are grouped geographically in each region.
", "rawMarkdown": "Interesting find! However:\nThe site_id variable corresponds to weather stations. Some of those ASHRAE regions are quite large geographically. It wouldn't make sense to have data from a unique weather station for each region, unless somehow the buildings are grouped geographically in each region.", "votes": null }, { "id": "652351", "postDate": "10/18/2019 18:05:51", "content": "I thought about that too. It could mean one of two things: (a) the buildings are grouped withing each region or (b) this idea is wrong and the ASHRAE regions have nothing to do with site_id. I think including holidays specific to each of the ASHRAE region might help to clarify this.
I plan to model on the assumption that the sites are cities from USA - your find of 15 regions is nice match to 15 sites in our data - but to minimize the weather data they have to be cities - but one or more cities from each region would make some sense.
\n\nI am going with USA assumption as I can find that multiple DOE surveys over the last decades have occurred and lots of summary data available with regards to building energy use in USA as a result. Have not seen anything similar in my google search for other parts of the world.
", "rawMarkdown": "I plan to model on the assumption that the sites are cities from USA - your find of 15 regions is nice match to 15 sites in our data - but to minimize the weather data they have to be cities - but one or more cities from each region would make some sense.\n\nI am going with USA assumption as I can find that multiple DOE surveys over the last decades have occurred and lots of summary data available with regards to building energy use in USA as a result. Have not seen anything similar in my google search for other parts of the world.", "votes": null }, { "id": "652736", "postDate": "10/19/2019 10:45:37", "content": "I think the idea of holidays is fine, but I believe seasonal time such as winter, summer,.. can bring more importance since this places might use airconditioner or heating. On top of that I will try to mark weekend days to the model, since many of the facilities \"*primary_type*\" are Education or Office and they are normally closed on weekends, a.k.a. lower energy consumption those days.
", "rawMarkdown": "I think the idea of holidays is fine, but I believe seasonal time such as winter, summer,.. can bring more importance since this places might use airconditioner or heating. On top of that I will try to mark weekend days to the model, since many of the facilities \"***primary_type***\" are Education or Office and they are normally closed on weekends, a.k.a. lower energy consumption those days.", "votes": null }, { "id": "653833", "postDate": "10/21/2019 02:58:12", "content": "Aren't there total 16 site_id in the dataset? Or am I missing something here?
", "rawMarkdown": "Aren't there total 16 site_id in the dataset? Or am I missing something here?", "votes": null }, { "id": "654072", "postDate": "10/21/2019 12:02:45", "content": "I didn't check but if it starts at 0 and ends at 15 that would be 16 indeed!
", "rawMarkdown": "I didn't check but if it starts at 0 and ends at 15 that would be 16 indeed!", "votes": null }, { "id": "654073", "postDate": "10/21/2019 12:03:02", "content": "What about Canada? Any July 1st patterns?
", "rawMarkdown": "What about Canada? Any July 1st patterns?", "votes": null }, { "id": "654139", "postDate": "10/21/2019 14:01:25", "content": "I said \"suspiciously similar\", not \"equal\".
", "rawMarkdown": "I said \"suspiciously similar\", not \"equal\".", "votes": null }, { "id": "654168", "postDate": "10/21/2019 14:45:11", "content": "\"The competition data have been anonymously donated to the competition by dozens of data\nowners, worldwide\"
\n\nhttps://www.ashrae.org/File%20Library/Technical%20Resources/Research/RT_REPORT_A19.r1.pdf
\n\nthe original plan of train=2 year, test= 1 year may be better\n(at least we can model the transition from dec to jan next year)
", "rawMarkdown": "\"The competition data have been anonymously donated to the competition by dozens of data\nowners, worldwide\"\n\n\nhttps://www.ashrae.org/File%20Library/Technical%20Resources/Research/RT_REPORT_A19.r1.pdf\n\n\nthe original plan of train=2 year, test= 1 year may be better\n(at least we can model the transition from dec to jan next year)", "votes": null }, { "id": "654253", "postDate": "10/21/2019 16:07:16", "content": "Agree. Having 2 years of data in train and 1 year of data in test would make way more sense to me.
", "rawMarkdown": "Agree. Having 2 years of data in train and 1 year of data in test would make way more sense to me.", "votes": null }, { "id": "654317", "postDate": "10/21/2019 17:44:24", "content": "Thank you, I'm curious about this topic! Have you faced any mathematical correlation about this aspect?
", "rawMarkdown": "Thank you, I'm curious about this topic! Have you faced any mathematical correlation about this aspect?", "votes": null }, { "id": "654649", "postDate": "10/22/2019 06:23:45", "content": "I try to visualize the air temperature by heatmap to check against weather data online, alot of yearly average temperature can be found in US cities itself.
\n\nMost of the temperature peak around mid of the year, it should be northern region.
\n\nCheck this out -> https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113772#latest-654627
\n\nand this kernel -> https://www.kaggle.com/ccjoshua/ashrae-airtemperature
", "rawMarkdown": "I try to visualize the air temperature by heatmap to check against weather data online, alot of yearly average temperature can be found in US cities itself. \n\nMost of the temperature peak around mid of the year, it should be northern region.\n\nCheck this out -> https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113772#latest-654627 \n\nand this kernel -> https://www.kaggle.com/ccjoshua/ashrae-airtemperature", "votes": null }, { "id": "659249", "postDate": "10/27/2019 10:45:10", "content": "Nice finding!! thank you.
", "rawMarkdown": "Nice finding!! thank you.", "votes": null }, { "id": "665803", "postDate": "11/05/2019 12:15:19", "content": "I used this info and the sites' temperature data to make educated guesses as to which site is which, see here https://www.kaggle.com/datadugong/locate-better-cities-by-weather-temp-fill-nans
", "rawMarkdown": "I used this info and the sites' temperature data to make educated guesses as to which site is which, see here https://www.kaggle.com/datadugong/locate-better-cities-by-weather-temp-fill-nans", "votes": null }, { "id": "666388", "postDate": "11/06/2019 03:59:46", "content": "Thank for that.\nIt seem like site_id: 10 is difficult to map to any of the city in ASHRAE in United Stated.
\n\nAnybody has better match for site_id: 10 ?
", "rawMarkdown": "Thank for that.\nIt seem like site_id: 10 is difficult to map to any of the city in ASHRAE in United Stated. \n\nAnybody has better match for site_id: 10 ?", "votes": null }, { "id": "667594", "postDate": "11/07/2019 12:40:40", "content": "I think it is Salt Lake City. It is definitely in the US in that area. Other good correlations were Las Vegas, San Francisco, Portland and Phoenix (if I recall correctly).
\n\nThe issue here might be that, as Salt Lake City is on a lake, the temperature might change drastically if the temp sensor is next to the lake or a couple Kms/miles to the other side of the city. I just clicked the middle of the city to get the lat longs so could be a few Kms out.
", "rawMarkdown": "I think it is Salt Lake City. It is definitely in the US in that area. Other good correlations were Las Vegas, San Francisco, Portland and Phoenix (if I recall correctly).\n\nThe issue here might be that, as Salt Lake City is on a lake, the temperature might change drastically if the temp sensor is next to the lake or a couple Kms/miles to the other side of the city. I just clicked the middle of the city to get the lat longs so could be a few Kms out.", "votes": null }, { "id": "667617", "postDate": "11/07/2019 13:06:23", "content": "Thanks! But actually i don't know how to use this external data to create new features
", "rawMarkdown": "Thanks! But actually i don't know how to use this external data to create new features", "votes": null } ], "comments": [ { "id": 652314, "author_name": "chmaxx", "author_url": "", "post_date": "10/18/2019 17:08:08", "content": "Good catch! Thanks for sharing. 👍
", "votes": null, "replies": [] }, { "id": 652326, "author_name": "sdoria", "author_url": "", "post_date": "10/18/2019 17:31:15", "content": "Interesting find! However:\nThe site_id variable corresponds to weather stations. Some of those ASHRAE regions are quite large geographically. It wouldn't make sense to have data from a unique weather station for each region, unless somehow the buildings are grouped geographically in each region.
", "votes": null, "replies": [ { "id": 652351, "author_name": "graf10a", "author_url": "", "post_date": "10/18/2019 18:05:51", "content": "I thought about that too. It could mean one of two things: (a) the buildings are grouped withing each region or (b) this idea is wrong and the ASHRAE regions have nothing to do with site_id. I think including holidays specific to each of the ASHRAE region might help to clarify this.
I think the idea of holidays is fine, but I believe seasonal time such as winter, summer,.. can bring more importance since this places might use airconditioner or heating. On top of that I will try to mark weekend days to the model, since many of the facilities \"*primary_type*\" are Education or Office and they are normally closed on weekends, a.k.a. lower energy consumption those days.
", "votes": null, "replies": [] } ] }, { "id": 652457, "author_name": "pcjimmmy", "author_url": "", "post_date": "10/18/2019 21:54:08", "content": "I plan to model on the assumption that the sites are cities from USA - your find of 15 regions is nice match to 15 sites in our data - but to minimize the weather data they have to be cities - but one or more cities from each region would make some sense.
\n\nI am going with USA assumption as I can find that multiple DOE surveys over the last decades have occurred and lots of summary data available with regards to building energy use in USA as a result. Have not seen anything similar in my google search for other parts of the world.
", "votes": null, "replies": [ { "id": 654073, "author_name": "sdoria", "author_url": "", "post_date": "10/21/2019 12:03:02", "content": "What about Canada? Any July 1st patterns?
", "votes": null, "replies": [] } ] }, { "id": 653833, "author_name": "pratikkgandhi", "author_url": "", "post_date": "10/21/2019 02:58:12", "content": "Aren't there total 16 site_id in the dataset? Or am I missing something here?
", "votes": null, "replies": [ { "id": 654072, "author_name": "sdoria", "author_url": "", "post_date": "10/21/2019 12:02:45", "content": "I didn't check but if it starts at 0 and ends at 15 that would be 16 indeed!
", "votes": null, "replies": [] }, { "id": 654139, "author_name": "graf10a", "author_url": "", "post_date": "10/21/2019 14:01:25", "content": "I said \"suspiciously similar\", not \"equal\".
", "votes": null, "replies": [] } ] }, { "id": 654168, "author_name": "hengck23", "author_url": "", "post_date": "10/21/2019 14:45:11", "content": "\"The competition data have been anonymously donated to the competition by dozens of data\nowners, worldwide\"
\n\nhttps://www.ashrae.org/File%20Library/Technical%20Resources/Research/RT_REPORT_A19.r1.pdf
\n\nthe original plan of train=2 year, test= 1 year may be better\n(at least we can model the transition from dec to jan next year)
", "votes": null, "replies": [ { "id": 654253, "author_name": "graf10a", "author_url": "", "post_date": "10/21/2019 16:07:16", "content": "Agree. Having 2 years of data in train and 1 year of data in test would make way more sense to me.
", "votes": null, "replies": [] } ] }, { "id": 654317, "author_name": "ivanvigorito", "author_url": "", "post_date": "10/21/2019 17:44:24", "content": "Thank you, I'm curious about this topic! Have you faced any mathematical correlation about this aspect?
", "votes": null, "replies": [] }, { "id": 654649, "author_name": "ccjoshua", "author_url": "", "post_date": "10/22/2019 06:23:45", "content": "I try to visualize the air temperature by heatmap to check against weather data online, alot of yearly average temperature can be found in US cities itself.
\n\nMost of the temperature peak around mid of the year, it should be northern region.
\n\nCheck this out -> https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113772#latest-654627
\n\nand this kernel -> https://www.kaggle.com/ccjoshua/ashrae-airtemperature
", "votes": null, "replies": [] }, { "id": 659249, "author_name": "corochann", "author_url": "", "post_date": "10/27/2019 10:45:10", "content": "Nice finding!! thank you.
", "votes": null, "replies": [] }, { "id": 665803, "author_name": "datadugong", "author_url": "", "post_date": "11/05/2019 12:15:19", "content": "I used this info and the sites' temperature data to make educated guesses as to which site is which, see here https://www.kaggle.com/datadugong/locate-better-cities-by-weather-temp-fill-nans
", "votes": null, "replies": [ { "id": 666388, "author_name": "ccjoshua", "author_url": "", "post_date": "11/06/2019 03:59:46", "content": "Thank for that.\nIt seem like site_id: 10 is difficult to map to any of the city in ASHRAE in United Stated.
\n\nAnybody has better match for site_id: 10 ?
", "votes": null, "replies": [] }, { "id": 667594, "author_name": "datadugong", "author_url": "", "post_date": "11/07/2019 12:40:40", "content": "I think it is Salt Lake City. It is definitely in the US in that area. Other good correlations were Las Vegas, San Francisco, Portland and Phoenix (if I recall correctly).
\n\nThe issue here might be that, as Salt Lake City is on a lake, the temperature might change drastically if the temp sensor is next to the lake or a couple Kms/miles to the other side of the city. I just clicked the middle of the city to get the lat longs so could be a few Kms out.
", "votes": null, "replies": [] } ] }, { "id": 667617, "author_name": "dxxhhh", "author_url": "", "post_date": "11/07/2019 13:06:23", "content": "Thanks! But actually i don't know how to use this external data to create new features
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "652305": "The has been a lot of discussion about supplementing the data set with some kind of geolocation data. This might be very helpful for modeling because it could potentially give us a way of figuring out national holidays or give us access to some additional weather/climate data for different locations. So, I went to the ASHRAE web site and discovered that [ASHRAE is comprised of 15 regions](https://www.ashrae.org/communities/regions). The map of these regions is updated annually and can be downloaded from [here](https://www.ashrae.org/File%20Library/Communities/Regions/ASHRAE_08072019.pdf). \n\nThe number of regions is suspiciously similar to the number of the unique values of the `site_id` variable in the competition data set. There might be a connection here and I think it is worth investigating. Right now, I do not have time to work on this competition, but I encourage you to play with this idea and report your finding in this thread.", "652314": "Good catch! Thanks for sharing. 👍", "652326": "Interesting find! However:\nThe site_id variable corresponds to weather stations. Some of those ASHRAE regions are quite large geographically. It wouldn't make sense to have data from a unique weather station for each region, unless somehow the buildings are grouped geographically in each region.", "652351": "I thought about that too. It could mean one of two things: (a) the buildings are grouped withing each region or (b) this idea is wrong and the ASHRAE regions have nothing to do with `site_id`. I think including holidays specific to each of the ASHRAE region might help to clarify this.", "652457": "I plan to model on the assumption that the sites are cities from USA - your find of 15 regions is nice match to 15 sites in our data - but to minimize the weather data they have to be cities - but one or more cities from each region would make some sense.\n\nI am going with USA assumption as I can find that multiple DOE surveys over the last decades have occurred and lots of summary data available with regards to building energy use in USA as a result. Have not seen anything similar in my google search for other parts of the world.", "652736": "I think the idea of holidays is fine, but I believe seasonal time such as winter, summer,.. can bring more importance since this places might use airconditioner or heating. On top of that I will try to mark weekend days to the model, since many of the facilities \"***primary_type***\" are Education or Office and they are normally closed on weekends, a.k.a. lower energy consumption those days.", "653833": "Aren't there total 16 site_id in the dataset? Or am I missing something here?", "654072": "I didn't check but if it starts at 0 and ends at 15 that would be 16 indeed!", "654073": "What about Canada? Any July 1st patterns?", "654139": "I said \"suspiciously similar\", not \"equal\".", "654168": "\"The competition data have been anonymously donated to the competition by dozens of data\nowners, worldwide\"\n\n\nhttps://www.ashrae.org/File%20Library/Technical%20Resources/Research/RT_REPORT_A19.r1.pdf\n\n\nthe original plan of train=2 year, test= 1 year may be better\n(at least we can model the transition from dec to jan next year)", "654253": "Agree. Having 2 years of data in train and 1 year of data in test would make way more sense to me.", "654317": "Thank you, I'm curious about this topic! Have you faced any mathematical correlation about this aspect?", "654649": "I try to visualize the air temperature by heatmap to check against weather data online, alot of yearly average temperature can be found in US cities itself. \n\nMost of the temperature peak around mid of the year, it should be northern region.\n\nCheck this out -> https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113772#latest-654627 \n\nand this kernel -> https://www.kaggle.com/ccjoshua/ashrae-airtemperature", "659249": "Nice finding!! thank you.", "665803": "I used this info and the sites' temperature data to make educated guesses as to which site is which, see here https://www.kaggle.com/datadugong/locate-better-cities-by-weather-temp-fill-nans", "666388": "Thank for that.\nIt seem like site_id: 10 is difficult to map to any of the city in ASHRAE in United Stated. \n\nAnybody has better match for site_id: 10 ?", "667594": "I think it is Salt Lake City. It is definitely in the US in that area. Other good correlations were Las Vegas, San Francisco, Portland and Phoenix (if I recall correctly).\n\nThe issue here might be that, as Salt Lake City is on a lake, the temperature might change drastically if the temp sensor is next to the lake or a couple Kms/miles to the other side of the city. I just clicked the middle of the city to get the lat longs so could be a few Kms out.", "667617": "Thanks! But actually i don't know how to use this external data to create new features" }, "source": "meta" }