![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F582454%2Fc42fe70678be86b719333fb5ac513d83%2FScreenshot%202019-10-28%20at%207.43.48%20PM.png?generation=1572272240961887&alt=media\")
(Apologies, this post won't be very visual but I don't want to spend the time to save and upload charts right now.)
\n\n*TLDR: *
\n\nMy guess is that sites 1, 5, 12 are in Europe due to the timezone and use of Celsius\nSites 7 and 11 are in Canada due to the timezone, use of C and July 1st\nOthers are potentially in the US due to the timezone and use of F
\n\nWhen I plotted a histogram of Air Temperatures (using size 1 bins), I was surprised to see that bins 9,14,19,24,29,34 had fewer records than expected.
\n\nLooking deeper into it, it seems that this is because some of the temperature values have been converted from F to C.
\n\nFor example:\n56F = 13.3C\n57F = 13.9C\n58F = 14.4C\n59F = 15C\n60F = 15.6C
\n\nFrom the examples above, I hope you can see why the 14 bin has fewer records than the 13 or 15 bins.\nSo, for a lot of the temperature data,we have 0.5 or 0.6 jumps depending on rounding.
\n\nThat's not all.
\n\nIf you look at temperature data site ID by site ID, the jumps from one value to the next can be classified into the following:
\n\n1) ~0.6 jumps (converted from F?)
\n\nsiteID's: 3, 4, 6, 10, 14
\n\n2) ~0.6 jumps (with some rare extra values, often integers):
\n\n0, 2, 6, 8, 13, 15
\n\n3) 1.0 jumps (only integer C values)
\n\n5
\n\n4) 0.1 jumps
\n\n1, 7, 11, 12
\n\nSo I guess you could separate those into two groups: Celsius vs Fahrenheit. This could give us a hint on geographical location.
\n\nNext, we can look at what time of the day peak temperature occurs.
\n\nAs others have noted, temperature sometimes peaks in the evening, which implies that the time for temperature date is not local.
\n\nI noticed 3 different types values for the hour of day with highest average temperature:
\n\n19:00-20:00-21:00: siteIDs 0, 3, 6 , 7, 8, 9, 11, 13, 14, 15
\n\n14:00: 1, 5, 12
\n\n22:00-23:00: 2, 4, 10
\n\nMy guess is that the timestamp is UTC or something close by.
\nNote that siteIDs 1, 5, 12 are both in celcius and with a 14:00 high temperature.
Difference between 19:00 and 22:00 could come from US time zones.
\n\nPotentially the other 2 siteIDS with Celsius data (7, 11) are in the same timezones as USA but different countries.
\n\nSite id 7 and 11 are in Celsius, and appears to have a 3 day weekend with lower power usage that includes Friday, July 1st => Canada day! (Only educational buildings there btw)
\n\n(Apologies, this post won't be very visual but I don't want to spend the time to save and upload charts right now.)
\n\n*TLDR: *
\n\nMy guess is that sites 1, 5, 12 are in Europe due to the timezone and use of Celsius\nSites 7 and 11 are in Canada due to the timezone, use of C and July 1st\nOthers are potentially in the US due to the timezone and use of F
\n\nWhen I plotted a histogram of Air Temperatures (using size 1 bins), I was surprised to see that bins 9,14,19,24,29,34 had fewer records than expected.
\n\nLooking deeper into it, it seems that this is because some of the temperature values have been converted from F to C.
\n\nFor example:\n56F = 13.3C\n57F = 13.9C\n58F = 14.4C\n59F = 15C\n60F = 15.6C
\n\nFrom the examples above, I hope you can see why the 14 bin has fewer records than the 13 or 15 bins.\nSo, for a lot of the temperature data,we have 0.5 or 0.6 jumps depending on rounding.
\n\nThat's not all.
\n\nIf you look at temperature data site ID by site ID, the jumps from one value to the next can be classified into the following:
\n\n1) ~0.6 jumps (converted from F?)
\n\nsiteID's: 3, 4, 6, 10, 14
\n\n2) ~0.6 jumps (with some rare extra values, often integers):
\n\n0, 2, 6, 8, 13, 15
\n\n3) 1.0 jumps (only integer C values)
\n\n5
\n\n4) 0.1 jumps
\n\n1, 7, 11, 12
\n\nSo I guess you could separate those into two groups: Celsius vs Fahrenheit. This could give us a hint on geographical location.
\n\nNext, we can look at what time of the day peak temperature occurs.
\n\nAs others have noted, temperature sometimes peaks in the evening, which implies that the time for temperature date is not local.
\n\nI noticed 3 different types values for the hour of day with highest average temperature:
\n\n19:00-20:00-21:00: siteIDs 0, 3, 6 , 7, 8, 9, 11, 13, 14, 15
\n\n14:00: 1, 5, 12
\n\n22:00-23:00: 2, 4, 10
\n\nMy guess is that the timestamp is UTC or something close by.
\nNote that siteIDs 1, 5, 12 are both in celcius and with a 14:00 high temperature.
Difference between 19:00 and 22:00 could come from US time zones.
\n\nPotentially the other 2 siteIDS with Celsius data (7, 11) are in the same timezones as USA but different countries.
\n\nSite id 7 and 11 are in Celsius, and appears to have a 3 day weekend with lower power usage that includes Friday, July 1st => Canada day! (Only educational buildings there btw)
\n\nI've found a cluster of low meter reading datapoints for May 2 and May 30 for siteid=1 (building 123). \nThose are Early May Bank Holiday and Spring Bank Holiday in the UK!
", "rawMarkdown": "I've found a cluster of low meter reading datapoints for May 2 and May 30 for siteid=1 (building 123). \nThose are Early May Bank Holiday and Spring Bank Holiday in the UK!", "votes": 7 }, { "id": 659460, "postDate": "2019-10-27T17:55:11.677Z", "content": "Recently created a kernel with timestamp auto-alignment attempt, who is interested - check this out!
As far as I see, it improves correlations with log-target, however, I didn't check it in submissions
I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")
\n\nHere's the notebook:\nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature
\n\nHope it helps!
", "rawMarkdown": "I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")\n\nHere's the notebook:\nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature\n\nHope it helps!", "votes": 4, "replies": [ { "id": 661833, "postDate": "2019-10-30T18:50:14.007Z", "content": "It seems that you discovered something interesting ...
", "rawMarkdown": "It seems that you discovered something interesting ...\n\n" } ] }, { "id": 658867, "postDate": "2019-10-26T16:44:12.707Z", "content": "Also, potentially, the timestamp for weather data is not aligned with the the timestamp for meter usage. Discussion here
\n\nCould make sense to start with buildings from sites 1,5,12, where the peak temperature occurs close to what we would expect for local time.
", "rawMarkdown": "Also, potentially, the timestamp for weather data is not aligned with the the timestamp for meter usage. Discussion [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114447#latest-658783)\n\nCould make sense to start with buildings from sites 1,5,12, where the peak temperature occurs close to what we would expect for local time.", "votes": 2, "replies": [ { "id": 659022, "postDate": "2019-10-27T00:21:59.283Z", "content": "Once you make a determination of what time zone you think the site represents you can convert the weather data to local time (rather than UTC). When you do that the peak daily temperature falls where its suppose to be at around 3 PM. Spent over a day converting both train and test weather to local. Gets messy for start and end of year where you can end up with missing values. As is normal however I was very impressed with the power of catboost and lightgbm to figure that out all by themselves - cv actually got a touch worse when I use time zone corrected weather data.
\n\nNice catch on the resolution of readings leading to Canada vs US. There are properties of solar loading were it helps to have a latitude guess.
\n\nAny guesses about the three European cities ?
", "rawMarkdown": "Once you make a determination of what time zone you think the site represents you can convert the weather data to local time (rather than UTC). When you do that the peak daily temperature falls where its suppose to be at around 3 PM. Spent over a day converting both train and test weather to local. Gets messy for start and end of year where you can end up with missing values. As is normal however I was very impressed with the power of catboost and lightgbm to figure that out all by themselves - cv actually got a touch worse when I use time zone corrected weather data.\n\nNice catch on the resolution of readings leading to Canada vs US. There are properties of solar loading were it helps to have a latitude guess. \n\nAny guesses about the three European cities ?\n\n", "votes": 1 }, { "id": 659028, "postDate": "2019-10-27T00:41:14.337Z", "content": "I'm hoping the Kaggle team will fix the time zone issue...
\n\nI think site 1 is UK based on the holidays that I've seen (May 2, May 30, Aug 29). Haven't looked at the other ones yet. Probably they are all in the UK?
", "rawMarkdown": "I'm hoping the Kaggle team will fix the time zone issue...\n\nI think site 1 is UK based on the holidays that I've seen (May 2, May 30, Aug 29). Haven't looked at the other ones yet. Probably they are all in the UK?\n" }, { "id": 673599, "postDate": "2019-11-15T07:55:29.347Z", "content": "@pcjimmmy could be because, if you aligned weather timestamp, but not the train.csv, then after the merge the meter readings could be aligned to the wrong weather?
", "rawMarkdown": "@pcjimmmy could be because, if you aligned weather timestamp, but not the train.csv, then after the merge the meter readings could be aligned to the wrong weather?", "votes": 1 } ] }, { "id": 670377, "postDate": "2019-11-11T11:40:36.703Z", "content": "Great insight ! Thank you so much !
\n\nSure from the beginning that we have to deal with temperatures in °C and °F... this is a real help.
\n\nBut now, are every temp comparable in dew and air temperatures ?
", "rawMarkdown": "Great insight ! Thank you so much !\n\nSure from the beginning that we have to deal with temperatures in °C and °F... this is a real help.\n\nBut now, are every temp comparable in dew and air temperatures ?", "replies": [ { "id": 670519, "postDate": "2019-11-11T14:43:50.140Z", "content": "To be clear, all temperatures are in °C; it's just that some of them have been converted from °F, while others were already in °C.
", "rawMarkdown": "To be clear, all temperatures are in °C; it's just that some of them have been converted from °F, while others were already in °C.", "votes": 1 } ] }, { "id": 670324, "postDate": "2019-11-11T10:21:20.537Z", "content": "it helps me a lot,thanks four your shares
", "rawMarkdown": "it helps me a lot,thanks four your shares" }, { "id": 660752, "postDate": "2019-10-29T14:48:00.830Z", "content": "I think all temperature values are in Celsius . The values for air temp are in the range of -30 & +40 which has to be celsius readings. My kernel here has a section on Weather trends sitewise which plots the daily min,max and mean temperatures sitewise .
", "rawMarkdown": "I think all temperature values are in Celsius . The values for air temp are in the range of -30 & +40 which has to be celsius readings. My kernel [here](https://www.kaggle.com/rajeshcv/exploring-buildings-weather-and-time-series-data) has a section on Weather trends sitewise which plots the daily min,max and mean temperatures sitewise .", "replies": [ { "id": 660771, "postDate": "2019-10-29T15:19:58.583Z", "content": "I agree. \nAnd as I've said in the original post, a lot of the temperatures have been converted from F to C.
", "rawMarkdown": "I agree. \nAnd as I've said in the original post, a lot of the temperatures have been converted from F to C." } ] }, { "id": 659983, "postDate": "2019-10-28T14:19:42.773Z", "content": "I still don't get it. How do you assume values to have a offset of 14 for site 1, 5, 12. Take a look at my mean highest average temperatures. They all have the range from 19 to 22 offsets. And averaging subdues any misreadings if the samples are large
\n\n
My bad. Looks like skipping nan values by condition instead of grouping led to the above chart
\n\nSite ID:0 == Offset:19, MAXAvgTemp:27.518579\nSite ID:1 == Offset:14, MAXAvgTemp:14.872055\nSite ID:2 == Offset:23, MAXAvgTemp:30.457377\nSite ID:3 == Offset:20, MAXAvgTemp:19.161096\nSite ID:4 == Offset:22, MAXAvgTemp:19.107104\nSite ID:5 == Offset:14, MAXAvgTemp:13.081967\nSite ID:6 == Offset:20, MAXAvgTemp:19.801370\nSite ID:7 == Offset:20, MAXAvgTemp:11.457423\nSite ID:8 == Offset:19, MAXAvgTemp:27.518579\nSite ID:9 == Offset:21, MAXAvgTemp:26.395902\nSite ID:10 == Offset:22, MAXAvgTemp:16.366940\nSite ID:11 == Offset:20, MAXAvgTemp:11.457423\nSite ID:12 == Offset:14, MAXAvgTemp:12.290137\nSite ID:13 == Offset:21, MAXAvgTemp:13.173770\nSite ID:14 == Offset:20, MAXAvgTemp:17.467213\nSite ID:15 == Offset:20, MAXAvgTemp:13.588202
\n\nGlad you were able to confirm my findings, thanks for producing the charts!
", "rawMarkdown": "Glad you were able to confirm my findings, thanks for producing the charts!" }, { "id": 670137, "postDate": "2019-11-11T04:12:45.193Z", "content": "Hey those charts are neat. I suggest you replot the subplots with shared x,y coords: fig, axes = plt.subplots(6, 4, sharex=True, sharey=True). Seems like the ylim for temperature will be +4..+30 F
Thank you for sharing your edged intuition.
", "rawMarkdown": "Thank you for sharing your edged intuition." }, { "id": 659002, "postDate": "2019-10-26T22:52:23.383Z", "content": "Very inspiring investigations! Good job!
", "rawMarkdown": "Very inspiring investigations! Good job!" }, { "id": 659257, "postDate": "2019-10-27T11:05:13.673Z", "rawMarkdown": "", "isDeleted": true }, { "id": 658859, "postDate": "2019-10-26T16:32:54.880Z", "content": "Great Sharing! Thanks!
", "rawMarkdown": "Great Sharing! Thanks!" } ], "comments": [ { "id": 658951, "author_name": "S D", "author_url": "", "post_date": "2019-10-26T19:57:10.167000", "content": "I've found a cluster of low meter reading datapoints for May 2 and May 30 for siteid=1 (building 123). \nThose are Early May Bank Holiday and Spring Bank Holiday in the UK!
", "votes": 7, "replies": [] }, { "id": 659460, "author_name": "Fred Navruzov", "author_url": "", "post_date": "2019-10-27T17:55:11.677000", "content": "Recently created a kernel with timestamp auto-alignment attempt, who is interested - check this out!
As far as I see, it improves correlations with log-target, however, I didn't check it in submissions
I just located most of sites according to external dataset from kaggle.\n(Already post in \"External Data Disclosure Thread\")
\n\nHere's the notebook:\nhttps://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature
\n\nHope it helps!
", "votes": 4, "replies": [ { "id": 661833, "author_name": "mezoganet", "author_url": "", "post_date": "2019-10-30T18:50:14.007000", "content": "It seems that you discovered something interesting ...
", "votes": 0, "replies": [] } ] }, { "id": 658867, "author_name": "S D", "author_url": "", "post_date": "2019-10-26T16:44:12.707000", "content": "Also, potentially, the timestamp for weather data is not aligned with the the timestamp for meter usage. Discussion here
\n\nCould make sense to start with buildings from sites 1,5,12, where the peak temperature occurs close to what we would expect for local time.
", "votes": 2, "replies": [ { "id": 659022, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-10-27T00:21:59.283000", "content": "Once you make a determination of what time zone you think the site represents you can convert the weather data to local time (rather than UTC). When you do that the peak daily temperature falls where its suppose to be at around 3 PM. Spent over a day converting both train and test weather to local. Gets messy for start and end of year where you can end up with missing values. As is normal however I was very impressed with the power of catboost and lightgbm to figure that out all by themselves - cv actually got a touch worse when I use time zone corrected weather data.
\n\nNice catch on the resolution of readings leading to Canada vs US. There are properties of solar loading were it helps to have a latitude guess.
\n\nAny guesses about the three European cities ?
", "votes": 1, "replies": [] }, { "id": 659028, "author_name": "S D", "author_url": "", "post_date": "2019-10-27T00:41:14.337000", "content": "I'm hoping the Kaggle team will fix the time zone issue...
\n\nI think site 1 is UK based on the holidays that I've seen (May 2, May 30, Aug 29). Haven't looked at the other ones yet. Probably they are all in the UK?
", "votes": 0, "replies": [] }, { "id": 673599, "author_name": "Sidhant", "author_url": "", "post_date": "2019-11-15T07:55:29.347000", "content": "@pcjimmmy could be because, if you aligned weather timestamp, but not the train.csv, then after the merge the meter readings could be aligned to the wrong weather?
", "votes": 1, "replies": [] } ] }, { "id": 670377, "author_name": "mezoganet", "author_url": "", "post_date": "2019-11-11T11:40:36.703000", "content": "Great insight ! Thank you so much !
\n\nSure from the beginning that we have to deal with temperatures in °C and °F... this is a real help.
\n\nBut now, are every temp comparable in dew and air temperatures ?
", "votes": 0, "replies": [ { "id": 670519, "author_name": "S D", "author_url": "", "post_date": "2019-11-11T14:43:50.140000", "content": "To be clear, all temperatures are in °C; it's just that some of them have been converted from °F, while others were already in °C.
", "votes": 1, "replies": [] } ] }, { "id": 670324, "author_name": "bingo", "author_url": "", "post_date": "2019-11-11T10:21:20.537000", "content": "it helps me a lot,thanks four your shares
", "votes": 0, "replies": [] }, { "id": 660752, "author_name": "Rajesh Vikraman", "author_url": "", "post_date": "2019-10-29T14:48:00.830000", "content": "I think all temperature values are in Celsius . The values for air temp are in the range of -30 & +40 which has to be celsius readings. My kernel here has a section on Weather trends sitewise which plots the daily min,max and mean temperatures sitewise .
", "votes": 0, "replies": [ { "id": 660771, "author_name": "S D", "author_url": "", "post_date": "2019-10-29T15:19:58.583000", "content": "I agree. \nAnd as I've said in the original post, a lot of the temperatures have been converted from F to C.
", "votes": 0, "replies": [] } ] }, { "id": 659983, "author_name": "VivianTitus", "author_url": "", "post_date": "2019-10-28T14:19:42.773000", "content": "I still don't get it. How do you assume values to have a offset of 14 for site 1, 5, 12. Take a look at my mean highest average temperatures. They all have the range from 19 to 22 offsets. And averaging subdues any misreadings if the samples are large
\n\n
My bad. Looks like skipping nan values by condition instead of grouping led to the above chart
\n\nSite ID:0 == Offset:19, MAXAvgTemp:27.518579\nSite ID:1 == Offset:14, MAXAvgTemp:14.872055\nSite ID:2 == Offset:23, MAXAvgTemp:30.457377\nSite ID:3 == Offset:20, MAXAvgTemp:19.161096\nSite ID:4 == Offset:22, MAXAvgTemp:19.107104\nSite ID:5 == Offset:14, MAXAvgTemp:13.081967\nSite ID:6 == Offset:20, MAXAvgTemp:19.801370\nSite ID:7 == Offset:20, MAXAvgTemp:11.457423\nSite ID:8 == Offset:19, MAXAvgTemp:27.518579\nSite ID:9 == Offset:21, MAXAvgTemp:26.395902\nSite ID:10 == Offset:22, MAXAvgTemp:16.366940\nSite ID:11 == Offset:20, MAXAvgTemp:11.457423\nSite ID:12 == Offset:14, MAXAvgTemp:12.290137\nSite ID:13 == Offset:21, MAXAvgTemp:13.173770\nSite ID:14 == Offset:20, MAXAvgTemp:17.467213\nSite ID:15 == Offset:20, MAXAvgTemp:13.588202
\n\nGlad you were able to confirm my findings, thanks for producing the charts!
", "votes": 0, "replies": [] }, { "id": 670137, "author_name": "Stephen McInerney", "author_url": "", "post_date": "2019-11-11T04:12:45.193000", "content": "Hey those charts are neat. I suggest you replot the subplots with shared x,y coords: fig, axes = plt.subplots(6, 4, sharex=True, sharey=True). Seems like the ylim for temperature will be +4..+30 F
Thank you for sharing your edged intuition.
", "votes": 0, "replies": [] }, { "id": 659002, "author_name": "Fred Navruzov", "author_url": "", "post_date": "2019-10-26T22:52:23.383000", "content": "Very inspiring investigations! Good job!
", "votes": 0, "replies": [] }, { "id": 659257, "author_name": "", "author_url": "", "post_date": "2019-10-27T11:05:13.673000", "content": "", "votes": 0, "replies": [] }, { "id": 658859, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-26T16:32:54.880000", "content": "Great Sharing! Thanks!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "658851": "(Apologies, this post won't be very visual but I don't want to spend the time to save and upload charts right now.)\n\n**TLDR: **\n\nMy guess is that sites 1, 5, 12 are in Europe due to the timezone and use of Celsius\nSites 7 and 11 are in Canada due to the timezone, use of C and July 1st\nOthers are potentially in the US due to the timezone and use of F\n\n\n-------------------------------------------------------------------------------------------------------\nWhen I plotted a histogram of Air Temperatures (using size 1 bins), I was surprised to see that bins 9,14,19,24,29,34 had fewer records than expected.\n\nLooking deeper into it, it seems that this is because some of the temperature values have been converted from F to C. \n\nFor example:\n56F = 13.3C\n57F = 13.9C\n58F = 14.4C\n59F = 15C\n60F = 15.6C\n\nFrom the examples above, I hope you can see why the 14 bin has fewer records than the 13 or 15 bins.\nSo, for a lot of the temperature data,we have 0.5 or 0.6 jumps depending on rounding.\n\n------------------------------------------------------------------------------------------------------\nThat's not all.\n\nIf you look at temperature data site ID by site ID, the jumps from one value to the next can be classified into the following:\n\n**1)** ~0.6 jumps (converted from F?)\n\nsiteID's: 3, 4, 6, 10, 14\n\n**2)** ~0.6 jumps (with some rare extra values, often integers):\n\n0, 2, 6, 8, 13, 15\n\n**3)** 1.0 jumps (only integer C values)\n\n5\n\n**4)** 0.1 jumps\n\n1, 7, 11, 12\n\n\nSo I guess you could separate those into two groups: Celsius vs Fahrenheit. This could give us a hint on geographical location.\n\n---------------------------------------------------------------------------------------------------------\n\nNext, we can look at what time of the day peak temperature occurs.\n\nAs others have noted, temperature sometimes peaks in the evening, which implies that the time for temperature date is not local.\n\nI noticed 3 different types values for the hour of day with highest average temperature:\n\n**19:00-20:00-21:00**: siteIDs 0, 3, 6 , 7, 8, 9, 11, 13, 14, 15\n\n**14:00**: 1, 5, 12\n\n**22:00-23:00**: 2, 4, 10\n\n\nMy guess is that the timestamp is UTC or something close by. \n**Note that siteIDs 1, 5, 12 are both in celcius and with a 14:00 high temperature.**\n\nDifference between 19:00 and 22:00 could come from US time zones. \n\nPotentially the other 2 siteIDS with Celsius data (7, 11) are in the same timezones as USA but different countries.\n\n-----------------------------------------------------------------------------------------------------------\n\nSite id 7 and 11 are in Celsius, and appears to have a 3 day weekend with lower power usage that includes Friday, July 1st => Canada day! (Only educational buildings there btw)\n\n-----------------------------------------------------------------------------------------------------\n\n\n\n\n", "658951": "I've found a cluster of low meter reading datapoints for May 2 and May 30 for siteid=1 (building 123). \nThose are Early May Bank Holiday and Spring Bank Holiday in the UK!", "659460": "Recently created a [kernel](https://www.kaggle.com/frednavruzov/aligning-temperature-timestamp) with **timestamp auto-alignment attempt**, who is interested - check this out!