Since several people have asked:
\n\nSince several people have asked:
\n\nThat sounds fair but it makes the current LB of no use. Thank you for clarification.
", "rawMarkdown": "That sounds fair but it makes the current LB of no use. Thank you for clarification.", "votes": null }, { "id": "673353", "postDate": "11/14/2019 21:59:13", "content": "\n\n\nWe will ensure that any meter readings that can be publicly downloaded are not included in the private set.
\n
So you will change the public/private split?
\n\nAnd I think December 12th is already too late for making the competition fair.
", "rawMarkdown": "> We will ensure that any meter readings that can be publicly downloaded are not included in the private set.\n\nSo you will change the public/private split?\n\nAnd I think December 12th is already too late for making the competition fair.", "votes": null }, { "id": "673468", "postDate": "11/15/2019 02:25:08", "content": "So I guess this is now a web scraping competition?
\nI wish we could just have all the data beforehand, so that we don't have to do all the scraping
Thanks for the clarification.
\n\nI'm satisfied with these rules.
\n\nAnyway, below 3 comments are useful.
\n\nUp, a deadline extension would be great, too much wasted time..
", "rawMarkdown": "Up, a deadline extension would be great, too much wasted time..", "votes": null }, { "id": "673620", "postDate": "11/15/2019 08:35:09", "content": "I wonder how do you find leak data exactly ? I think it's impossible if you already know all leak data from first. If so why do you open all leak data and restart this competition again.
", "rawMarkdown": "I wonder how do you find leak data exactly ? I think it's impossible if you already know all leak data from first. If so why do you open all leak data and restart this competition again.", "votes": null }, { "id": "673625", "postDate": "11/15/2019 08:36:42", "content": "I hope that meter readings that can be publicly downloaded can be official train data.
", "rawMarkdown": "I hope that meter readings that can be publicly downloaded can be official train data.", "votes": null }, { "id": "673739", "postDate": "11/15/2019 12:16:57", "content": "Thank you for the official explanation. I will continue this competition.👍
", "rawMarkdown": "Thank you for the official explanation. I will continue this competition.👍", "votes": null }, { "id": "673883", "postDate": "11/15/2019 15:40:50", "content": "\n\n\nWe will ensure that any meter readings that can be publicly downloaded are not included in the private set.
\n
@sohier Does that mean that you already know of other sites that can be scraped ?
\n\nIf so, this would be nice if somebody at Kaggle does the webscraping for us and make the data public. Otherwise, we may expect that such data will remain 'hidden' for some time before being published in the external data thread.
", "rawMarkdown": "> We will ensure that any meter readings that can be publicly downloaded are not included in the private set.\n\n@sohier Does that mean that you already know of other sites that can be scraped ? \n\nIf so, this would be nice if somebody at Kaggle does the webscraping for us and make the data public. Otherwise, we may expect that such data will remain 'hidden' for some time before being published in the external data thread.", "votes": null }, { "id": "677213", "postDate": "11/19/2019 23:27:46", "content": "why not just ban all external data?
", "rawMarkdown": "why not just ban all external data?", "votes": null }, { "id": "678178", "postDate": "11/21/2019 04:01:19", "content": "> To be clear, we have reason to believe that enough of the sites are unavailable online that there will be a meaningful final leaderboard.
\n\nMay you tell us which sites are unavailable online? Leakages are frustrating.
", "rawMarkdown": "> To be clear, we have reason to believe that enough of the sites are unavailable online that there will be a meaningful final leaderboard.\n\nMay you tell us which sites are unavailable online? Leakages are frustrating.", "votes": null }, { "id": "678304", "postDate": "11/21/2019 07:59:09", "content": "I've spent so many hours cleaning up or looking at any site_ids that are now ready to download. I also used some submissions to test this. Looks like you will be penalized in this competition if you start too early
", "rawMarkdown": "I've spent so many hours cleaning up or looking at any site_ids that are now ready to download. I also used some submissions to test this. Looks like you will be penalized in this competition if you start too early", "votes": null }, { "id": "678477", "postDate": "11/21/2019 12:55:08", "content": "I hope the range of\n\"meter readings that can be publicly downloaded\"\nis declared at 12/12 (sooner is better if possible)
\n\nAt least for me, it is not clear the range,\nfor example building_id 152 at site1 and 245 at site2.
", "rawMarkdown": "I hope the range of\n\"meter readings that can be publicly downloaded\"\nis declared at 12/12 (sooner is better if possible)\n\nAt least for me, it is not clear the range,\nfor example building_id 152 at site1 and 245 at site2.", "votes": null }, { "id": "678732", "postDate": "11/21/2019 19:14:33", "content": "Good point! This leakage will lay even more importance on building a good local validation scheme.
", "rawMarkdown": "Good point! This leakage will lay even more importance on building a good local validation scheme.", "votes": null }, { "id": "679705", "postDate": "11/23/2019 08:03:55", "content": "Now we just have to use the external data thread to download the training data 🤔
", "rawMarkdown": "Now we just have to use the external data thread to download the training data 🤔", "votes": null }, { "id": "680257", "postDate": "11/24/2019 10:39:22", "content": "@Sohier Dane I appreciate the clarification you gave. But still, I believe, there are couple of things , which is making the competitiors lose their confidence on the \"COMPETITION\".
\n\nWhat I have read in the other discussions about leakage, it certainly shows that this is not an external data that can help to find answer, but this is something that is THE ANSWER.
\n\nSo, I would really request even the kaggle team to clear their stand on this.
\n\nIsn't that an unfair advantage?
\n\nMay I have your opinion on the my following suggestions?
\n\nExternal data that involves meter readings of the buildings in the dataset should not be allowed for training. I hope my understanding is correct, that is what is happening here.
If above criteria is considered, I feel, clearing leaderboard could be helpful. Someone still may make submission with leaked data. But at least, we can hope that new submissions will meet the new criteria.
Extend the deadline, if possible.
While I share the concerns of Leo Regulus and many others on this forum, let me propose the opposite solution. Could the organizers please make the leaked data explicitly available at this forum. \nEven if removed from score calculation, this data may help to train models. I just see no sure way to effectively ban use of leaked external data from However in many cases, getting data requires web scraping and repeated heavy use of websites / APIs of data owners. I noticed that some links posted in this forum are already broken. This means that data owners noticed unusual activity and restricted access. So we have a situation where those who got early access to leaked data have advantage. True, the sources are shared in the external data thread but scraping them again is not always repeatable for the above reasons. \nIf the leaked data is available here it will put everyone into more equal position and support focus to ML part of the competition.
", "rawMarkdown": "While I share the concerns of Leo Regulus and many others on this forum, let me propose the opposite solution. Could the organizers please make the leaked data explicitly available at this forum. \nEven if removed from score calculation, this data may help to train models. I just see no sure way to effectively ban use of leaked external data from However in many cases, getting data requires web scraping and repeated heavy use of websites / APIs of data owners. I noticed that some links posted in this forum are already broken. This means that data owners noticed unusual activity and restricted access. So we have a situation where those who got early access to leaked data have advantage. True, the sources are shared in the external data thread but scraping them again is not always repeatable for the above reasons. \nIf the leaked data is available here it will put everyone into more equal position and support focus to ML part of the competition.", "votes": null }, { "id": "681890", "postDate": "11/26/2019 16:40:21", "content": "Agreed. The problem is that regulating whether someone has used leaked data or not is hard. I mean someone could quietly use it in their program and the organizers wouldn't know.
", "rawMarkdown": "Agreed. The problem is that regulating whether someone has used leaked data or not is hard. I mean someone could quietly use it in their program and the organizers wouldn't know.", "votes": null }, { "id": "681936", "postDate": "11/26/2019 18:00:33", "content": "You raise a fair point, and I was unaware that some of the links are down. With that said, are there any publicly available datasets that haven't already been rehosted in a notebook/dataset like this one? I'm happy to revisit this issue if that's not the case. https://www.kaggle.com/yamsam/ashrae-leak-data-station
", "rawMarkdown": "You raise a fair point, and I was unaware that some of the links are down. With that said, are there any publicly available datasets that haven't already been rehosted in a notebook/dataset like this one? I'm happy to revisit this issue if that's not the case. https://www.kaggle.com/yamsam/ashrae-leak-data-station", "votes": null }, { "id": "681978", "postDate": "11/26/2019 18:58:50", "content": "Dane, I have not found any extra datasets, however I have not checked personally if https://www.kaggle.com/yamsam/ashrae-leak-data-station is complete either. While Isamu Yamashita and other great Kagglers made a major effort to rehost leaked data, there is no 100% certainty that such rehosted datasets are accurately saved and processed. It may be best if such dataset comes from the organizers once the leaks are known.
", "rawMarkdown": "Dane, I have not found any extra datasets, however I have not checked personally if https://www.kaggle.com/yamsam/ashrae-leak-data-station is complete either. While Isamu Yamashita and other great Kagglers made a major effort to rehost leaked data, there is no 100% certainty that such rehosted datasets are accurately saved and processed. It may be best if such dataset comes from the organizers once the leaks are known.", "votes": null }, { "id": "682012", "postDate": "11/26/2019 19:38:37", "content": "TLDR: I'm willing to review specific cases but can't make a sweeping commitment.
\n\nI think I would need a stronger case in order to publish additional data.\n- I don't want to poach upvotes from people who have published useful data\n- Some sites may not have a clear match and I need to be careful to avoid accidentally confirming unproven theories. Similarly, I need to be careful to avoid confirming if a site is not actually a match.\n- Some of the purported public sites only have partial data available; I need to be careful to release previously inaccessible data.\n- The data was acquired through a variety of methods so we often can't confirm if a link was ever valid. It would be easy for someone to set up a reasonable looking link that returns a 404 error.\n- For some sites the purported public data only covers some buildings for a subset of the relevant time period. The forums and associated notebooks and datasets are constantly updated; it's not feasible for me to stay on top of all of them at the necessary level of detail real time. \n- The burden is on competitors to ensure the data they use complies with the rules. If someone uses external data but the data doesn't meet all of the availability requirements specified in the rules after the data disclosure deadline, I can disqualify them.
", "rawMarkdown": "TLDR: I'm willing to review specific cases but can't make a sweeping commitment.\n\nI think I would need a stronger case in order to publish additional data.\n- I don't want to poach upvotes from people who have published useful data\n- Some sites may not have a clear match and I need to be careful to avoid accidentally confirming unproven theories. Similarly, I need to be careful to avoid confirming if a site is _not_ actually a match.\n- Some of the purported public sites only have partial data available; I need to be careful to release previously inaccessible data.\n- The data was acquired through a variety of methods so we often can't confirm if a link was ever valid. It would be easy for someone to set up a reasonable looking link that returns a 404 error.\n- For some sites the purported public data only covers some buildings for a subset of the relevant time period. The forums and associated notebooks and datasets are constantly updated; it's not feasible for me to stay on top of all of them at the necessary level of detail real time. \n- The burden is on competitors to ensure the data they use complies with the rules. If someone uses external data but the data doesn't meet all of the availability requirements specified in the rules after the data disclosure deadline, I can disqualify them.", "votes": null }, { "id": "682694", "postDate": "11/27/2019 19:01:20", "content": "Dane, Your arguments are fair, indeed. Good to see that you and the other organizers are monitoring this situation.
", "rawMarkdown": "Dane, Your arguments are fair, indeed. Good to see that you and the other organizers are monitoring this situation.", "votes": null }, { "id": "682702", "postDate": "11/27/2019 19:06:53", "content": "Great idea! It is a great waste of brain power to let all Kagglers go through the nuisance of figuring out which site_ids have public test labels.
", "rawMarkdown": "Great idea! It is a great waste of brain power to let all Kagglers go through the nuisance of figuring out which site_ids have public test labels.", "votes": null }, { "id": "684536", "postDate": "11/30/2019 01:33:10", "content": "@youhanlee - I agree with your observations. I have personally held off making any submissions yet. My approach is as follows:
\n\nHopefully, this will create a solid predictor that will handle private test set without much chance of surprise.
", "rawMarkdown": "youhanlee - I agree with your observations. I have personally held off making any submissions yet. My approach is as follows:\n\n- Make a decent prediction model with a strong CV approach\n- Include no leaked data in the model design\n- Concentrate on a rich feature set, especially on missing building attributes like floor count; have more complete floor counts opens a number of critical energy factors like building volume, etc.\n- Enhance weather data with industry standard predictors like cooling degree days, cooling degree hours, prior day, prior hour temperatures, etc.\n- Use the public leaderboard to help validate the model\n\nHopefully, this will create a solid predictor that will handle private test set without much chance of surprise.", "votes": null }, { "id": "684690", "postDate": "11/30/2019 09:26:48", "content": "In Data description one can read:
\n\n\n\n\nmeter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error. UPDATE: as discussed here, the site 0 electric meter readings are in kBTU.
\n
We can reasonably expect that some measurement error is present in the test set too. Are the measurement errors removed from the test set for evaluation?\nIn other words: do we have to predict meter readings or building performance?
", "rawMarkdown": "In Data description one can read:\n> meter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which **we expect will impose a baseline level of modeling error**. UPDATE: as discussed here, the site 0 electric meter readings are in kBTU.\n\nWe can reasonably expect that some measurement error is present in the test set too. Are the measurement errors removed from the test set for evaluation?\nIn other words: do we have to predict meter readings or building performance?", "votes": null }, { "id": "685109", "postDate": "12/01/2019 01:23:07", "content": "Does this means when we upload our final data ,we can contain leak data?
", "rawMarkdown": "Does this means when we upload our final data ,we can contain leak data?", "votes": null }, { "id": "686048", "postDate": "12/02/2019 17:27:30", "content": "Per the rules, all external data used for your final submission must be disclosed and available for all other competitions to use by December 12th (the external data disclosure deadline). We will, at a minimum, disqualify any team that uses undisclosed data.
", "rawMarkdown": "Per the rules, all external data used for your final submission must be disclosed and available for all other competitions to use by December 12th (the external data disclosure deadline). We will, at a minimum, disqualify any team that uses undisclosed data.", "votes": null }, { "id": "686485", "postDate": "12/03/2019 07:56:12", "content": "That sounds very fair
", "rawMarkdown": "That sounds very fair", "votes": null }, { "id": "686540", "postDate": "12/03/2019 09:13:57", "content": "\n\n\nWe will ensure that any meter readings that can be publicly downloaded are not included in the private set. Update for clarity: shortly after the external data disclosure deadline, I'll review the disclosures and update the scoring system to ignore rows as needed. There will be a short blip on the leaderboard while existing submissions are reevaluated but the public scores will remain unchanged.
\n
Does that mean that there is no reason to tune models for publicly downloadable segments?
", "rawMarkdown": "> We will ensure that any meter readings that can be publicly downloaded are not included in the private set. Update for clarity: shortly after the external data disclosure deadline, I'll review the disclosures and update the scoring system to ignore rows as needed. There will be a short blip on the leaderboard while existing submissions are reevaluated but the public scores will remain unchanged.\n\nDoes that mean that there is no reason to tune models for publicly downloadable segments?", "votes": null }, { "id": "687815", "postDate": "12/04/2019 21:47:19", "content": "LB is likely useless as there is huge variation between years judging from the leaked sites. The only solid way to go forward is to trust your CV and even that one is kind of unpredictable. This is the worst data science setup possible.
", "rawMarkdown": "LB is likely useless as there is huge variation between years judging from the leaked sites. The only solid way to go forward is to trust your CV and even that one is kind of unpredictable. This is the worst data science setup possible.", "votes": null }, { "id": "688900", "postDate": "12/06/2019 08:24:53", "content": "Hope so, but don't think it is possible. Leaked data from external resources can be used as data augmentation for the training set, which is not submitted. Thus, to make sure this not happen, one's model and training data should be checked as well (in my opinion).
", "rawMarkdown": "Hope so, but don't think it is possible. Leaked data from external resources can be used as data augmentation for the training set, which is not submitted. Thus, to make sure this not happen, one's model and training data should be checked as well (in my opinion).", "votes": null }, { "id": "690262", "postDate": "12/08/2019 08:48:02", "content": "I have searched a lot of notebooks and threads but still a bit confused.\nMy question is about the timestamps on all of the datasets(train, test, weather train, weather test)\nare they local time or GMT time? or some of them are local time, others GMT time?
", "rawMarkdown": "I have searched a lot of notebooks and threads but still a bit confused.\nMy question is about the timestamps on all of the datasets(train, test, weather train, weather test)\nare they local time or GMT time? or some of them are local time, others GMT time?", "votes": null }, { "id": "699093", "postDate": "12/20/2019 03:34:29", "content": "i think they are all local time.
", "rawMarkdown": "i think they are all local time.", "votes": null }, { "id": "699094", "postDate": "12/20/2019 03:36:06", "content": "plot by daily temperature for each site would be a good way.
", "rawMarkdown": "plot by daily temperature for each site would be a good way.", "votes": null } ], "comments": [ { "id": 673345, "author_name": "vishal1310", "author_url": "", "post_date": "11/14/2019 21:40:56", "content": "That sounds fair but it makes the current LB of no use. Thank you for clarification.
", "votes": null, "replies": [] }, { "id": 673353, "author_name": "aerdem4", "author_url": "", "post_date": "11/14/2019 21:59:13", "content": "\n\n\nWe will ensure that any meter readings that can be publicly downloaded are not included in the private set.
\n
So you will change the public/private split?
\n\nAnd I think December 12th is already too late for making the competition fair.
", "votes": null, "replies": [ { "id": 673616, "author_name": "arthurllau", "author_url": "", "post_date": "11/15/2019 08:29:28", "content": "Up, a deadline extension would be great, too much wasted time..
", "votes": null, "replies": [] } ] }, { "id": 673468, "author_name": "pocketsuteado", "author_url": "", "post_date": "11/15/2019 02:25:08", "content": "So I guess this is now a web scraping competition?
\nI wish we could just have all the data beforehand, so that we don't have to do all the scraping
Now we just have to use the external data thread to download the training data 🤔
", "votes": null, "replies": [] } ] }, { "id": 673490, "author_name": "youhanlee", "author_url": "", "post_date": "11/15/2019 03:42:04", "content": "Thanks for the clarification.
\n\nI'm satisfied with these rules.
\n\nAnyway, below 3 comments are useful.
\n\nGood point! This leakage will lay even more importance on building a good local validation scheme.
", "votes": null, "replies": [] }, { "id": 684536, "author_name": "wjholst", "author_url": "", "post_date": "11/30/2019 01:33:10", "content": "@youhanlee - I agree with your observations. I have personally held off making any submissions yet. My approach is as follows:
\n\nHopefully, this will create a solid predictor that will handle private test set without much chance of surprise.
", "votes": null, "replies": [] }, { "id": 687815, "author_name": "snovik1975", "author_url": "", "post_date": "12/04/2019 21:47:19", "content": "LB is likely useless as there is huge variation between years judging from the leaked sites. The only solid way to go forward is to trust your CV and even that one is kind of unpredictable. This is the worst data science setup possible.
", "votes": null, "replies": [] } ] }, { "id": 673620, "author_name": "yamsam", "author_url": "", "post_date": "11/15/2019 08:35:09", "content": "I wonder how do you find leak data exactly ? I think it's impossible if you already know all leak data from first. If so why do you open all leak data and restart this competition again.
", "votes": null, "replies": [] }, { "id": 673625, "author_name": "daishu", "author_url": "", "post_date": "11/15/2019 08:36:42", "content": "I hope that meter readings that can be publicly downloaded can be official train data.
", "votes": null, "replies": [ { "id": 688900, "author_name": "alisherianvar", "author_url": "", "post_date": "12/06/2019 08:24:53", "content": "Hope so, but don't think it is possible. Leaked data from external resources can be used as data augmentation for the training set, which is not submitted. Thus, to make sure this not happen, one's model and training data should be checked as well (in my opinion).
", "votes": null, "replies": [] } ] }, { "id": 673739, "author_name": "mashlyn", "author_url": "", "post_date": "11/15/2019 12:16:57", "content": "Thank you for the official explanation. I will continue this competition.👍
", "votes": null, "replies": [] }, { "id": 673883, "author_name": "fabiendaniel", "author_url": "", "post_date": "11/15/2019 15:40:50", "content": "\n\n\nWe will ensure that any meter readings that can be publicly downloaded are not included in the private set.
\n
@sohier Does that mean that you already know of other sites that can be scraped ?
\n\nIf so, this would be nice if somebody at Kaggle does the webscraping for us and make the data public. Otherwise, we may expect that such data will remain 'hidden' for some time before being published in the external data thread.
", "votes": null, "replies": [ { "id": 682702, "author_name": "carlolepelaars", "author_url": "", "post_date": "11/27/2019 19:06:53", "content": "Great idea! It is a great waste of brain power to let all Kagglers go through the nuisance of figuring out which site_ids have public test labels.
", "votes": null, "replies": [] } ] }, { "id": 677213, "author_name": "s13658695", "author_url": "", "post_date": "11/19/2019 23:27:46", "content": "why not just ban all external data?
", "votes": null, "replies": [ { "id": 681890, "author_name": "yushg123", "author_url": "", "post_date": "11/26/2019 16:40:21", "content": "Agreed. The problem is that regulating whether someone has used leaked data or not is hard. I mean someone could quietly use it in their program and the organizers wouldn't know.
", "votes": null, "replies": [] } ] }, { "id": 678178, "author_name": "daishu", "author_url": "", "post_date": "11/21/2019 04:01:19", "content": "> To be clear, we have reason to believe that enough of the sites are unavailable online that there will be a meaningful final leaderboard.
\n\nMay you tell us which sites are unavailable online? Leakages are frustrating.
", "votes": null, "replies": [ { "id": 678304, "author_name": "oleg90777", "author_url": "", "post_date": "11/21/2019 07:59:09", "content": "I've spent so many hours cleaning up or looking at any site_ids that are now ready to download. I also used some submissions to test this. Looks like you will be penalized in this competition if you start too early
", "votes": null, "replies": [] } ] }, { "id": 678477, "author_name": "hiroyukiex", "author_url": "", "post_date": "11/21/2019 12:55:08", "content": "I hope the range of\n\"meter readings that can be publicly downloaded\"\nis declared at 12/12 (sooner is better if possible)
\n\nAt least for me, it is not clear the range,\nfor example building_id 152 at site1 and 245 at site2.
", "votes": null, "replies": [] }, { "id": 680257, "author_name": "dsnil87", "author_url": "", "post_date": "11/24/2019 10:39:22", "content": "@Sohier Dane I appreciate the clarification you gave. But still, I believe, there are couple of things , which is making the competitiors lose their confidence on the \"COMPETITION\".
\n\nWhat I have read in the other discussions about leakage, it certainly shows that this is not an external data that can help to find answer, but this is something that is THE ANSWER.
\n\nSo, I would really request even the kaggle team to clear their stand on this.
\n\nIsn't that an unfair advantage?
\n\nMay I have your opinion on the my following suggestions?
\n\nExternal data that involves meter readings of the buildings in the dataset should not be allowed for training. I hope my understanding is correct, that is what is happening here.
If above criteria is considered, I feel, clearing leaderboard could be helpful. Someone still may make submission with leaked data. But at least, we can hope that new submissions will meet the new criteria.
Extend the deadline, if possible.
While I share the concerns of Leo Regulus and many others on this forum, let me propose the opposite solution. Could the organizers please make the leaked data explicitly available at this forum. \nEven if removed from score calculation, this data may help to train models. I just see no sure way to effectively ban use of leaked external data from However in many cases, getting data requires web scraping and repeated heavy use of websites / APIs of data owners. I noticed that some links posted in this forum are already broken. This means that data owners noticed unusual activity and restricted access. So we have a situation where those who got early access to leaked data have advantage. True, the sources are shared in the external data thread but scraping them again is not always repeatable for the above reasons. \nIf the leaked data is available here it will put everyone into more equal position and support focus to ML part of the competition.
", "votes": null, "replies": [ { "id": 681936, "author_name": "sohier", "author_url": "", "post_date": "11/26/2019 18:00:33", "content": "You raise a fair point, and I was unaware that some of the links are down. With that said, are there any publicly available datasets that haven't already been rehosted in a notebook/dataset like this one? I'm happy to revisit this issue if that's not the case. https://www.kaggle.com/yamsam/ashrae-leak-data-station
", "votes": null, "replies": [] }, { "id": 681978, "author_name": "poedator", "author_url": "", "post_date": "11/26/2019 18:58:50", "content": "Dane, I have not found any extra datasets, however I have not checked personally if https://www.kaggle.com/yamsam/ashrae-leak-data-station is complete either. While Isamu Yamashita and other great Kagglers made a major effort to rehost leaked data, there is no 100% certainty that such rehosted datasets are accurately saved and processed. It may be best if such dataset comes from the organizers once the leaks are known.
", "votes": null, "replies": [] }, { "id": 682012, "author_name": "sohier", "author_url": "", "post_date": "11/26/2019 19:38:37", "content": "TLDR: I'm willing to review specific cases but can't make a sweeping commitment.
\n\nI think I would need a stronger case in order to publish additional data.\n- I don't want to poach upvotes from people who have published useful data\n- Some sites may not have a clear match and I need to be careful to avoid accidentally confirming unproven theories. Similarly, I need to be careful to avoid confirming if a site is not actually a match.\n- Some of the purported public sites only have partial data available; I need to be careful to release previously inaccessible data.\n- The data was acquired through a variety of methods so we often can't confirm if a link was ever valid. It would be easy for someone to set up a reasonable looking link that returns a 404 error.\n- For some sites the purported public data only covers some buildings for a subset of the relevant time period. The forums and associated notebooks and datasets are constantly updated; it's not feasible for me to stay on top of all of them at the necessary level of detail real time. \n- The burden is on competitors to ensure the data they use complies with the rules. If someone uses external data but the data doesn't meet all of the availability requirements specified in the rules after the data disclosure deadline, I can disqualify them.
", "votes": null, "replies": [] }, { "id": 682694, "author_name": "poedator", "author_url": "", "post_date": "11/27/2019 19:01:20", "content": "Dane, Your arguments are fair, indeed. Good to see that you and the other organizers are monitoring this situation.
", "votes": null, "replies": [] } ] }, { "id": 684690, "author_name": "lucamassidda", "author_url": "", "post_date": "11/30/2019 09:26:48", "content": "In Data description one can read:
\n\n\n\n\nmeter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error. UPDATE: as discussed here, the site 0 electric meter readings are in kBTU.
\n
We can reasonably expect that some measurement error is present in the test set too. Are the measurement errors removed from the test set for evaluation?\nIn other words: do we have to predict meter readings or building performance?
", "votes": null, "replies": [] }, { "id": 685109, "author_name": "jacobfrey", "author_url": "", "post_date": "12/01/2019 01:23:07", "content": "Does this means when we upload our final data ,we can contain leak data?
", "votes": null, "replies": [ { "id": 686048, "author_name": "sohier", "author_url": "", "post_date": "12/02/2019 17:27:30", "content": "Per the rules, all external data used for your final submission must be disclosed and available for all other competitions to use by December 12th (the external data disclosure deadline). We will, at a minimum, disqualify any team that uses undisclosed data.
", "votes": null, "replies": [] } ] }, { "id": 686485, "author_name": "yuyonod314", "author_url": "", "post_date": "12/03/2019 07:56:12", "content": "That sounds very fair
", "votes": null, "replies": [] }, { "id": 686540, "author_name": "georgegv", "author_url": "", "post_date": "12/03/2019 09:13:57", "content": "\n\n\nWe will ensure that any meter readings that can be publicly downloaded are not included in the private set. Update for clarity: shortly after the external data disclosure deadline, I'll review the disclosures and update the scoring system to ignore rows as needed. There will be a short blip on the leaderboard while existing submissions are reevaluated but the public scores will remain unchanged.
\n
Does that mean that there is no reason to tune models for publicly downloadable segments?
", "votes": null, "replies": [] }, { "id": 690262, "author_name": "navidivan", "author_url": "", "post_date": "12/08/2019 08:48:02", "content": "I have searched a lot of notebooks and threads but still a bit confused.\nMy question is about the timestamps on all of the datasets(train, test, weather train, weather test)\nare they local time or GMT time? or some of them are local time, others GMT time?
", "votes": null, "replies": [ { "id": 699093, "author_name": "knightbyj1221", "author_url": "", "post_date": "12/20/2019 03:34:29", "content": "i think they are all local time.
", "votes": null, "replies": [] }, { "id": 699094, "author_name": "knightbyj1221", "author_url": "", "post_date": "12/20/2019 03:36:06", "content": "plot by daily temperature for each site would be a good way.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "673343": "Since several people have asked:\n\n- Using publicly available meter readings is within the rules, assuming the general restrictions are also followed such as posting a link to the data in the external data thread by the entry deadline (December 12th).\n- We will ensure that any meter readings that can be publicly downloaded are not included in the private set. Update for clarity: shortly after the external data disclosure deadline, I'll review the disclosures and update the scoring system to ignore rows as needed. There will be a short blip on the leaderboard while existing submissions are reevaluated but the public scores will remain unchanged.\n- To be clear, we have reason to believe that enough of the sites are unavailable online that there will be a meaningful final leaderboard.", "673345": "That sounds fair but it makes the current LB of no use. Thank you for clarification.", "673353": "> We will ensure that any meter readings that can be publicly downloaded are not included in the private set.\n\nSo you will change the public/private split?\n\nAnd I think December 12th is already too late for making the competition fair.", "673468": "So I guess this is now a web scraping competition? \nI wish we could just have all the data beforehand, so that we don't have to do all the scraping", "673490": "Thanks for the clarification.\n\nI'm satisfied with these rules.\n\nAnyway, below 3 comments are useful.\n\n1. The ranking in public leaderboard is useless.\n2. The public leaderboard score is useful to check our cv system.\n3. The thing we need to remember is \"trust your CV\".", "673616": "Up, a deadline extension would be great, too much wasted time..", "673620": "I wonder how do you find leak data exactly ? I think it's impossible if you already know all leak data from first. If so why do you open all leak data and restart this competition again.", "673625": "I hope that meter readings that can be publicly downloaded can be official train data.", "673739": "Thank you for the official explanation. I will continue this competition.👍", "673883": "> We will ensure that any meter readings that can be publicly downloaded are not included in the private set.\n\n@sohier Does that mean that you already know of other sites that can be scraped ? \n\nIf so, this would be nice if somebody at Kaggle does the webscraping for us and make the data public. Otherwise, we may expect that such data will remain 'hidden' for some time before being published in the external data thread.", "677213": "why not just ban all external data?", "678178": "> To be clear, we have reason to believe that enough of the sites are unavailable online that there will be a meaningful final leaderboard.\n\nMay you tell us which sites are unavailable online? Leakages are frustrating.", "678304": "I've spent so many hours cleaning up or looking at any site_ids that are now ready to download. I also used some submissions to test this. Looks like you will be penalized in this competition if you start too early", "678477": "I hope the range of\n\"meter readings that can be publicly downloaded\"\nis declared at 12/12 (sooner is better if possible)\n\nAt least for me, it is not clear the range,\nfor example building_id 152 at site1 and 245 at site2.", "678732": "Good point! This leakage will lay even more importance on building a good local validation scheme.", "679705": "Now we just have to use the external data thread to download the training data 🤔", "680257": "Sohier Dane I appreciate the clarification you gave. But still, I believe, there are couple of things , which is making the competitiors lose their confidence on the \"COMPETITION\".\n\n1. Allowing usage of external data is meant really to support a better modelling or better metadata build-up, which can assist in better model creation ,i.e external data that can \"**help to find answers**\". I know, it's not perhaps, written anywhere in the rules, but that is what the intention is/should be.\n\nWhat I have read in the other discussions about leakage, it certainly shows that this is not an external data that can help to find answer, but this is something that is THE ANSWER.\n\nSo, I would really request even the kaggle team to clear their stand on this.\n\n2. Assurance of using only the private set that is unavailable publicly is good for the competitiors. But , am I really wrong in thinking, that , using such public data which caters to exact building or site, actually makes it unfair on the part of other competitors , who are practically working with only 2016 data and others who are working with leakage, are actually training their model on a bigger and better dataset?\n\nIsn't that an unfair advantage?\n\n3. Your last comment made me a little nervous. \"enough of the sites are unavailable online\" , means, 'some of the sites are available online'. \n\n\nMay I have your opinion on the my following suggestions?\n\n1. External data that involves meter readings of the buildings in the dataset should not be allowed for training. I hope my understanding is correct, that is what is happening here.\n\n2. If above criteria is considered, I feel, clearing leaderboard could be helpful. Someone still may make submission with leaked data. But at least, we can hope that new submissions will meet the new criteria.\n\n3. Extend the deadline, if possible.", "681151": "While I share the concerns of Leo Regulus and many others on this forum, let me propose the opposite solution. Could the organizers please make the leaked data explicitly available at this forum. \nEven if removed from score calculation, this data may help to train models. I just see no sure way to effectively ban use of leaked external data from However in many cases, getting data requires web scraping and repeated heavy use of websites / APIs of data owners. I noticed that some links posted in this forum are already broken. This means that data owners noticed unusual activity and restricted access. So we have a situation where those who got early access to leaked data have advantage. True, the sources are shared in the external data thread but scraping them again is not always repeatable for the above reasons. \nIf the leaked data is available here it will put everyone into more equal position and support focus to ML part of the competition.", "681890": "Agreed. The problem is that regulating whether someone has used leaked data or not is hard. I mean someone could quietly use it in their program and the organizers wouldn't know.", "681936": "You raise a fair point, and I was unaware that some of the links are down. With that said, are there any publicly available datasets that haven't already been rehosted in a notebook/dataset like this one? I'm happy to revisit this issue if that's not the case. https://www.kaggle.com/yamsam/ashrae-leak-data-station", "681978": "Dane, I have not found any extra datasets, however I have not checked personally if https://www.kaggle.com/yamsam/ashrae-leak-data-station is complete either. While Isamu Yamashita and other great Kagglers made a major effort to rehost leaked data, there is no 100% certainty that such rehosted datasets are accurately saved and processed. It may be best if such dataset comes from the organizers once the leaks are known.", "682012": "TLDR: I'm willing to review specific cases but can't make a sweeping commitment.\n\nI think I would need a stronger case in order to publish additional data.\n- I don't want to poach upvotes from people who have published useful data\n- Some sites may not have a clear match and I need to be careful to avoid accidentally confirming unproven theories. Similarly, I need to be careful to avoid confirming if a site is _not_ actually a match.\n- Some of the purported public sites only have partial data available; I need to be careful to release previously inaccessible data.\n- The data was acquired through a variety of methods so we often can't confirm if a link was ever valid. It would be easy for someone to set up a reasonable looking link that returns a 404 error.\n- For some sites the purported public data only covers some buildings for a subset of the relevant time period. The forums and associated notebooks and datasets are constantly updated; it's not feasible for me to stay on top of all of them at the necessary level of detail real time. \n- The burden is on competitors to ensure the data they use complies with the rules. If someone uses external data but the data doesn't meet all of the availability requirements specified in the rules after the data disclosure deadline, I can disqualify them.", "682694": "Dane, Your arguments are fair, indeed. Good to see that you and the other organizers are monitoring this situation.", "682702": "Great idea! It is a great waste of brain power to let all Kagglers go through the nuisance of figuring out which site_ids have public test labels.", "684536": "youhanlee - I agree with your observations. I have personally held off making any submissions yet. My approach is as follows:\n\n- Make a decent prediction model with a strong CV approach\n- Include no leaked data in the model design\n- Concentrate on a rich feature set, especially on missing building attributes like floor count; have more complete floor counts opens a number of critical energy factors like building volume, etc.\n- Enhance weather data with industry standard predictors like cooling degree days, cooling degree hours, prior day, prior hour temperatures, etc.\n- Use the public leaderboard to help validate the model\n\nHopefully, this will create a solid predictor that will handle private test set without much chance of surprise.", "684690": "In Data description one can read:\n> meter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which **we expect will impose a baseline level of modeling error**. UPDATE: as discussed here, the site 0 electric meter readings are in kBTU.\n\nWe can reasonably expect that some measurement error is present in the test set too. Are the measurement errors removed from the test set for evaluation?\nIn other words: do we have to predict meter readings or building performance?", "685109": "Does this means when we upload our final data ,we can contain leak data?", "686048": "Per the rules, all external data used for your final submission must be disclosed and available for all other competitions to use by December 12th (the external data disclosure deadline). We will, at a minimum, disqualify any team that uses undisclosed data.", "686485": "That sounds very fair", "686540": "> We will ensure that any meter readings that can be publicly downloaded are not included in the private set. Update for clarity: shortly after the external data disclosure deadline, I'll review the disclosures and update the scoring system to ignore rows as needed. There will be a short blip on the leaderboard while existing submissions are reevaluated but the public scores will remain unchanged.\n\nDoes that mean that there is no reason to tune models for publicly downloadable segments?", "687815": "LB is likely useless as there is huge variation between years judging from the leaked sites. The only solid way to go forward is to trust your CV and even that one is kind of unpredictable. This is the worst data science setup possible.", "688900": "Hope so, but don't think it is possible. Leaked data from external resources can be used as data augmentation for the training set, which is not submitted. Thus, to make sure this not happen, one's model and training data should be checked as well (in my opinion).", "690262": "I have searched a lot of notebooks and threads but still a bit confused.\nMy question is about the timestamps on all of the datasets(train, test, weather train, weather test)\nare they local time or GMT time? or some of them are local time, others GMT time?", "699093": "i think they are all local time.", "699094": "plot by daily temperature for each site would be a good way." }, "source": "meta" }