Because of last findings about site locations and public/private set available, I decided to stop participating in this competition.\nIt was the first serious competition for me, I spent a lot of hours and learn a lot from Kaggle discussion/notebooks (thanks to all authors) and internet search, but it seems that now competition is more trying to search relevant info than about creating models for accurate predictions.
\n\nBut anyway it was very interesting and useful :)
\n\nI've decided to share with my current findings and thoughts:
\n\nI think I've found two new locations for sites:
\n\nIf we plot test data by site_id in the order of row_id, you can find 3 separates increasing stepwise lines, it seems that some data was added later to main block. Number of records for 2 last blocks (grouped by site_id, building_id) are suspiciously equal in multiple cases. \nI thought that maybe there are nans/zeros, but public LB doesn't confirm it for me.
Using CatBoost and Parameter tuning in documentation (https://catboost.ai/docs/concepts/parameter-tuning.html), model training can be increased in 3-4x times with increasing (in my case) accuracy. So one model with ~15 parameters and ~22 minutes of training can give 1.09 LB score.
I was surprised that although site_id and building_id is used in almost every public model, for me not using these 2 features in model gives me better results
One of the features that I used is “feels like” feature, using meteocalc package and formula for relative humidity based on air and dew temperatures. So feature calculation includes 2 temperatures and wind speed, and gives me more feature importance than these features separately
Besides trimming data leading zeros for site_id=0 as was discussed earlier, I tried to use some empirical method to truncate leading zeros with probable outliers - for example, cut leading data while cumulative mean of meter_reading is less than specific threshold. This method is not precise of course, but it can automatically truncate almost the same data for site_id=0 and also for other sites, and it helps me to descrease rmse locally and on LB.
There are a lot of repositories in github.com/buds-lab organization that can be useful (not only \nthat were mentioned already):
\n\nYou can find there information about preprocessing, tz conversion, modeling etc for building energy data
\n\nI found the files that can be another potential data leak: \nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/DGS_322_Buildings-15m-By_Building-DST-gap-filled-3-2-18-508pm.csv\nIt contains data of 15-min meters reading with buildings which name is very similar to site_id=3 (DC) till December 2018
\nIn this folder:\nhttps://github.com/buds-lab/island-of-misfit-buildings/tree/master/data/processed \nthere is a processed version of this data (resampled by 1 hour), this data is noisy with gaps but it seems that information is correlated with train data for site_id=3
\nAlso this data contains temperature in F, so hypothesis about unit conversions to C is probably confirmed.\nDetailed information about buildings you can find here:
\nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/dgs_metadata.csv
Finally, I want to share with 2 articles what I recently read and like very much about time series / signal processing:\nhttp://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/\nhttp://ataspinar.com/2018/12/21/a-guide-for-using-the-wavelet-transform-in-machine-learning/
P.S. As final submissions I choose 2 best solutions without any leaked data, so my final result will be honest and will not hurt another participants.
\n\nThanks guys!
\n\nUPD: \nI found my notes for this competition and remember that while searching information about sites, also found probably another data leak for site_id = 13 (Twin Cities):\nhttps://facilities.umn.edu/energy-management/building-energy-dashboards
\n\nSo for example if you search for Coffman Memorial Union which was mentioned and identified by Poe Dator as building_id=1099 with very large values of steam consumption (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114830#latest-666076) and click to the link for this building, you will find some information about energy consumption.
\n\nFor example, using link \"Daily Trend Chart\" (http://meters.energy.umn.edu/enweb/HS/064_coffman/daily_trend/dte_today.html)\nyou can find current electricity/steam consumption, and for steam you will see that there are anomaly values also in the plot, so this is probably not error at all or maybe systematic error.
\n\nI didn't search anymore on this site, so maybe another useful information can be found there.
", "messages": [ { "id": "670365", "postDate": "11/11/2019 11:23:41", "content": "Because of last findings about site locations and public/private set available, I decided to stop participating in this competition.\nIt was the first serious competition for me, I spent a lot of hours and learn a lot from Kaggle discussion/notebooks (thanks to all authors) and internet search, but it seems that now competition is more trying to search relevant info than about creating models for accurate predictions.
\n\nBut anyway it was very interesting and useful :)
\n\nI've decided to share with my current findings and thoughts:
\n\nI think I've found two new locations for sites:
\n\nIf we plot test data by site_id in the order of row_id, you can find 3 separates increasing stepwise lines, it seems that some data was added later to main block. Number of records for 2 last blocks (grouped by site_id, building_id) are suspiciously equal in multiple cases. \nI thought that maybe there are nans/zeros, but public LB doesn't confirm it for me.
Using CatBoost and Parameter tuning in documentation (https://catboost.ai/docs/concepts/parameter-tuning.html), model training can be increased in 3-4x times with increasing (in my case) accuracy. So one model with ~15 parameters and ~22 minutes of training can give 1.09 LB score.
I was surprised that although site_id and building_id is used in almost every public model, for me not using these 2 features in model gives me better results
One of the features that I used is “feels like” feature, using meteocalc package and formula for relative humidity based on air and dew temperatures. So feature calculation includes 2 temperatures and wind speed, and gives me more feature importance than these features separately
Besides trimming data leading zeros for site_id=0 as was discussed earlier, I tried to use some empirical method to truncate leading zeros with probable outliers - for example, cut leading data while cumulative mean of meter_reading is less than specific threshold. This method is not precise of course, but it can automatically truncate almost the same data for site_id=0 and also for other sites, and it helps me to descrease rmse locally and on LB.
There are a lot of repositories in github.com/buds-lab organization that can be useful (not only \nthat were mentioned already):
\n\nYou can find there information about preprocessing, tz conversion, modeling etc for building energy data
\n\nI found the files that can be another potential data leak: \nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/DGS_322_Buildings-15m-By_Building-DST-gap-filled-3-2-18-508pm.csv\nIt contains data of 15-min meters reading with buildings which name is very similar to site_id=3 (DC) till December 2018
\nIn this folder:\nhttps://github.com/buds-lab/island-of-misfit-buildings/tree/master/data/processed \nthere is a processed version of this data (resampled by 1 hour), this data is noisy with gaps but it seems that information is correlated with train data for site_id=3
\nAlso this data contains temperature in F, so hypothesis about unit conversions to C is probably confirmed.\nDetailed information about buildings you can find here:
\nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/dgs_metadata.csv
Finally, I want to share with 2 articles what I recently read and like very much about time series / signal processing:\nhttp://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/\nhttp://ataspinar.com/2018/12/21/a-guide-for-using-the-wavelet-transform-in-machine-learning/
P.S. As final submissions I choose 2 best solutions without any leaked data, so my final result will be honest and will not hurt another participants.
\n\nThanks guys!
\n\nUPD: \nI found my notes for this competition and remember that while searching information about sites, also found probably another data leak for site_id = 13 (Twin Cities):\nhttps://facilities.umn.edu/energy-management/building-energy-dashboards
\n\nSo for example if you search for Coffman Memorial Union which was mentioned and identified by Poe Dator as building_id=1099 with very large values of steam consumption (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114830#latest-666076) and click to the link for this building, you will find some information about energy consumption.
\n\nFor example, using link \"Daily Trend Chart\" (http://meters.energy.umn.edu/enweb/HS/064_coffman/daily_trend/dte_today.html)\nyou can find current electricity/steam consumption, and for steam you will see that there are anomaly values also in the plot, so this is probably not error at all or maybe systematic error.
\n\nI didn't search anymore on this site, so maybe another useful information can be found there.
", "rawMarkdown": "Because of last findings about site locations and public/private set available, I decided to stop participating in this competition.\nIt was the first serious competition for me, I spent a lot of hours and learn a lot from Kaggle discussion/notebooks (thanks to all authors) and internet search, but it seems that now competition is more trying to search relevant info than about creating models for accurate predictions.\n\nBut anyway it was very interesting and useful :)\n\nI've decided to share with my current findings and thoughts:\n\n- I think I've found two new locations for sites:\n - site\\_id=4 is probably UC Berkeley - https://www.ucop.edu/facilities-management-services/_files/deep-efficiency-and-cogen.pdf\nI found matches with table 6.5\n - site\\_id=10 is probably Weber State University - https://apps.weber.edu/wsuimages/sustainability/Plans%20and%20Reports/ESIP%20II%20Manual%20wAppendices%202-1-2015.pdf\nI found multiple matches there, as I remember in tables and probably also in text\n\n- If we plot test data by site\\_id in the order of row\\_id, you can find 3 separates increasing stepwise lines, it seems that some data was added later to main block. Number of records for 2 last blocks (grouped by site\\_id, building\\_id) are suspiciously equal in multiple cases. \nI thought that maybe there are nans/zeros, but public LB doesn't confirm it for me. \n\n- Using CatBoost and Parameter tuning in documentation (https://catboost.ai/docs/concepts/parameter-tuning.html), model training can be increased in 3-4x times with increasing (in my case) accuracy. So one model with ~15 parameters and ~22 minutes of training can give 1.09 LB score.\n\n- I was surprised that although site\\_id and building\\_id is used in almost every public model, for me not using these 2 features in model gives me better results\n\n- One of the features that I used is “feels like” feature, using meteocalc package and formula for relative humidity based on air and dew temperatures. So feature calculation includes 2 temperatures and wind speed, and gives me more feature importance than these features separately\n\n- Besides trimming data leading zeros for site\\_id=0 as was discussed earlier, I tried to use some empirical method to truncate leading zeros with probable outliers - for example, cut leading data while cumulative mean of meter_reading is less than specific threshold. This method is not precise of course, but it can automatically truncate almost the same data for site\\_id=0 and also for other sites, and it helps me to descrease rmse locally and on LB.\n\n- There are a lot of repositories in github.com/buds-lab organization that can be useful (not only \nthat were mentioned already):\n - the-building-data-genome-project\n - building-prediction-benchmarking\n - island-of-misfit-buildings\n - energystar-plus-plus\n - temporal-features-for-nonres-buildings-library\n - enerNOC-100-building-open-dataset-analysis\n - forensic-analysis-of-building-energy-data \n\nYou can find there information about preprocessing, tz conversion, modeling etc for building energy data\n\n- I found the files that can be another potential data leak: \nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/DGS_322_Buildings-15m-By_Building-DST-gap-filled-3-2-18-508pm.csv\nIt contains data of 15-min meters reading with buildings which name is very similar to site\\_id=3 (DC) till December 2018 \nIn this folder:\nhttps://github.com/buds-lab/island-of-misfit-buildings/tree/master/data/processed \nthere is a processed version of this data (resampled by 1 hour), this data is noisy with gaps but it seems that information is correlated with train data for site_id=3 \nAlso this data contains temperature in F, so hypothesis about unit conversions to C is probably confirmed.\nDetailed information about buildings you can find here: \nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/dgs_metadata.csv\n\n- Finally, I want to share with 2 articles what I recently read and like very much about time series / signal processing:\nhttp://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/\nhttp://ataspinar.com/2018/12/21/a-guide-for-using-the-wavelet-transform-in-machine-learning/\n\nP.S. As final submissions I choose 2 best solutions without any leaked data, so my final result will be honest and will not hurt another participants.\n\nThanks guys!\n\n**UPD**: \nI found my notes for this competition and remember that while searching information about sites, also found probably another data leak for site\\_id = 13 (Twin Cities):\nhttps://facilities.umn.edu/energy-management/building-energy-dashboards \n\nSo for example if you search for Coffman Memorial Union which was mentioned and identified by Poe Dator as building_id=1099 with very large values of steam consumption (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114830#latest-666076) and click to the link for this building, you will find some information about energy consumption. \n\nFor example, using link \"Daily Trend Chart\" (http://meters.energy.umn.edu/enweb/HS/064_coffman/daily_trend/dte_today.html)\nyou can find current electricity/steam consumption, and for steam you will see that there are anomaly values also in the plot, so this is probably not error at all or maybe systematic error.\n\nI didn't search anymore on this site, so maybe another useful information can be found there.", "votes": null }, { "id": "670373", "postDate": "11/11/2019 11:35:59", "content": "Hey buddy thanks for your findings.\nAnd perhaps you're making the right decision to quit this one 😔
", "rawMarkdown": "Hey buddy thanks for your findings.\nAnd perhaps you're making the right decision to quit this one 😔", "votes": null }, { "id": "670376", "postDate": "11/11/2019 11:39:17", "content": "Thank you!\nI feel that for me it is correct decision :)\nGood luck!
", "rawMarkdown": "Thank you!\nI feel that for me it is correct decision :)\nGood luck!", "votes": null }, { "id": "670384", "postDate": "11/11/2019 12:10:00", "content": "Appreciate your impressive findings.\nYou really did a good job in this complex competition.\nHope to see you soon in other competitions!
", "rawMarkdown": "Appreciate your impressive findings.\nYou really did a good job in this complex competition.\nHope to see you soon in other competitions!", "votes": null }, { "id": "670386", "postDate": "11/11/2019 12:11:24", "content": "I already have the labels for site 1 too, so it's a matter of time people will discover them. I'm just curious about what will happen to this competition at this point.
", "rawMarkdown": "I already have the labels for site 1 too, so it's a matter of time people will discover them. I'm just curious about what will happen to this competition at this point.", "votes": null }, { "id": "670389", "postDate": "11/11/2019 12:13:02", "content": "Thanks Patrick! )
", "rawMarkdown": "Thanks Patrick! )", "votes": null }, { "id": "670391", "postDate": "11/11/2019 12:16:01", "content": "Thanks Gunes for sharing this finding! Hope organizers will be honest in assessing of you in this competition on really private data.
", "rawMarkdown": "Thanks Gunes for sharing this finding! Hope organizers will be honest in assessing of you in this competition on really private data.", "votes": null }, { "id": "670396", "postDate": "11/11/2019 12:20:34", "content": "Looks like this competition is going to become ASHRAE - Great Energy Leak Detector III. Guess it's a hard pass for me.
Another open data service such as southampton ... I'm also wondering if it's worth to continue.
", "rawMarkdown": "Another open data service such as southampton ... I'm also wondering if it's worth to continue.", "votes": null }, { "id": "670917", "postDate": "11/12/2019 03:36:18", "content": "I mean, in theory everyone will have to disclose any leaks they find right? So you can just sit back, monitor the data disclosure thread and then integrate everyone else's findings yourself. The competition winner will be the person who can use the leaks most effectively.
\n\nI'm half considering maybe taking a break for a few weeks and coming back once most of the leaks have been found and disclosed...
", "rawMarkdown": "I mean, in theory everyone will have to disclose any leaks they find right? So you can just sit back, monitor the data disclosure thread and then integrate everyone else's findings yourself. The competition winner will be the person who can use the leaks most effectively. \n\nI'm half considering maybe taking a break for a few weeks and coming back once most of the leaks have been found and disclosed...", "votes": null }, { "id": "670947", "postDate": "11/12/2019 04:23:55", "content": "We have to share otherwise it will be unfair for everyone.
", "rawMarkdown": "We have to share otherwise it will be unfair for everyone.", "votes": null }, { "id": "671028", "postDate": "11/12/2019 07:05:29", "content": "Me too, I'm out until the organizers say something.
", "rawMarkdown": "Me too, I'm out until the organizers say something.", "votes": null }, { "id": "671383", "postDate": "11/12/2019 15:58:45", "content": "Thank you Dmitry for putting your findings together. \nI just tried Catboost and it does seems to increase accuracy. \nThanks for sharing :)
", "rawMarkdown": "Thank you Dmitry for putting your findings together. \nI just tried Catboost and it does seems to increase accuracy. \nThanks for sharing :)", "votes": null }, { "id": "671402", "postDate": "11/12/2019 16:37:53", "content": "Great!)
", "rawMarkdown": "Great!)", "votes": null }, { "id": "673617", "postDate": "11/15/2019 08:32:21", "content": "Hello, this is my first competition as well and before starting to focus on it, I am seeing that there are quite a few problems.
\n\nThis is probably going to be a very naive/noobie question (so apologies in advance!): given the situation, that some building's test data are leaked, but that to use it, it has to be public (otherwise I assume the team would be disqualified), what use can people do of such leaked data? I think that it's only useful as an extended train set or am I missing something obvious?
\n\nOf course it's bad but in the end, in theory, everybody is going to have the same data available. Another assumption I'm doing here is that scores getting prices will have the code/models reviews or even reproduced, and therefore finding other leaks and not making them public is not possible.
\n\nDmitry thanks a lot for your sharings, they are very insightful and it seems like you invested a fair amount of work! Why not giving it a try to you current model just training when the external data disclosure finishes? (including that extra data for training).
", "rawMarkdown": "Hello, this is my first competition as well and before starting to focus on it, I am seeing that there are quite a few problems. \n\nThis is probably going to be a very naive/noobie question (so apologies in advance!): given the situation, that some building's test data are leaked, but that to use it, it has to be public (otherwise I assume the team would be disqualified), what use can people do of such leaked data? I think that it's only useful as an extended train set or am I missing something obvious? \n\nOf course it's bad but in the end, in theory, everybody is going to have the same data available. Another assumption I'm doing here is that scores getting prices will have the code/models reviews or even reproduced, and therefore finding other leaks and not making them public is not possible.\n\nDmitry thanks a lot for your sharings, they are very insightful and it seems like you invested a fair amount of work! Why not giving it a try to you current model just training when the external data disclosure finishes? (including that extra data for training).", "votes": null }, { "id": "674046", "postDate": "11/15/2019 20:50:47", "content": "It’s great that it is helpful! About competition - I don’t plan to continue)
", "rawMarkdown": "It’s great that it is helpful! About competition - I don’t plan to continue)", "votes": null }, { "id": "675367", "postDate": "11/18/2019 02:02:58", "content": "@labdmitriy Impressive findings! Thanks for sharing.\nIt seems quite a lot of effort for me to breach 1.07 without leaks on a single model, much less 1.04. Without leaked data, what is your single/blended model scores on LB? How many features would it have?
", "rawMarkdown": "labdmitriy Impressive findings! Thanks for sharing.\nIt seems quite a lot of effort for me to breach 1.07 without leaks on a single model, much less 1.04. Without leaked data, what is your single/blended model scores on LB? How many features would it have?", "votes": null }, { "id": "680770", "postDate": "11/25/2019 07:37:13", "content": "I compare ASHRAE data and DC data you've mentioned. However, I could not find any correlation between meter reading values for these data sets. I even applied timestamp transformation to DC data set.
", "rawMarkdown": "I compare ASHRAE data and DC data you've [mentioned](DGS_322_Buildings-15m-By_Building-DST-gap-filled-3-2-18-508pm.csv). However, I could not find any correlation between meter reading values for these data sets. I even applied timestamp transformation to DC data set.", "votes": null }, { "id": "680787", "postDate": "11/25/2019 08:15:22", "content": "I remember that I checked several sites by comparing plots in 1 hour scale, and as I mentioned before, there are gaps in this data from github as if it is corrupted data, but these plots were similar. I think these gaps can be the main reason why correlation can be low.
", "rawMarkdown": "I remember that I checked several sites by comparing plots in 1 hour scale, and as I mentioned before, there are gaps in this data from github as if it is corrupted data, but these plots were similar. I think these gaps can be the main reason why correlation can be low.", "votes": null }, { "id": "680791", "postDate": "11/25/2019 08:19:56", "content": "Hi @gideonteo , at the moment of this post I had 1.09 single, 1.07 blended models, single model was with ~15 features
", "rawMarkdown": "Hi @gideonteo , at the moment of this post I had 1.09 single, 1.07 blended models, single model was with ~15 features", "votes": null }, { "id": "680800", "postDate": "11/25/2019 08:42:38", "content": "I realize gap but still none of buildings have a correlation higher than 0.7 and mae /mean ratio less than 10% even null values dropped.
", "rawMarkdown": "I realize gap but still none of buildings have a correlation higher than 0.7 and mae /mean ratio less than 10% even null values dropped.", "votes": null }, { "id": "680817", "postDate": "11/25/2019 09:24:41", "content": "Maybe I'm lucky and building that I've checked has similar plots, but I found this similarity for several buildings that I checked. I have not been participating in competitions for 2 weeks, so I can't tell more details.
", "rawMarkdown": "Maybe I'm lucky and building that I've checked has similar plots, but I found this similarity for several buildings that I checked. I have not been participating in competitions for 2 weeks, so I can't tell more details.", "votes": null } ], "comments": [ { "id": 670373, "author_name": "jielu0728", "author_url": "", "post_date": "11/11/2019 11:35:59", "content": "Hey buddy thanks for your findings.\nAnd perhaps you're making the right decision to quit this one 😔
", "votes": null, "replies": [ { "id": 670376, "author_name": "labdmitriy", "author_url": "", "post_date": "11/11/2019 11:39:17", "content": "Thank you!\nI feel that for me it is correct decision :)\nGood luck!
", "votes": null, "replies": [] } ] }, { "id": 670384, "author_name": "patrick0302", "author_url": "", "post_date": "11/11/2019 12:10:00", "content": "Appreciate your impressive findings.\nYou really did a good job in this complex competition.\nHope to see you soon in other competitions!
", "votes": null, "replies": [ { "id": 670389, "author_name": "labdmitriy", "author_url": "", "post_date": "11/11/2019 12:13:02", "content": "Thanks Patrick! )
", "votes": null, "replies": [] } ] }, { "id": 670386, "author_name": "gunesevitan", "author_url": "", "post_date": "11/11/2019 12:11:24", "content": "I already have the labels for site 1 too, so it's a matter of time people will discover them. I'm just curious about what will happen to this competition at this point.
", "votes": null, "replies": [ { "id": 670391, "author_name": "labdmitriy", "author_url": "", "post_date": "11/11/2019 12:16:01", "content": "Thanks Gunes for sharing this finding! Hope organizers will be honest in assessing of you in this competition on really private data.
", "votes": null, "replies": [] }, { "id": 670814, "author_name": "mpware", "author_url": "", "post_date": "11/11/2019 22:39:10", "content": "Another open data service such as southampton ... I'm also wondering if it's worth to continue.
", "votes": null, "replies": [] } ] }, { "id": 670396, "author_name": "learnmower", "author_url": "", "post_date": "11/11/2019 12:20:34", "content": "Looks like this competition is going to become ASHRAE - Great Energy Leak Detector III. Guess it's a hard pass for me.
Me too, I'm out until the organizers say something.
", "votes": null, "replies": [] } ] }, { "id": 670917, "author_name": "lewington", "author_url": "", "post_date": "11/12/2019 03:36:18", "content": "I mean, in theory everyone will have to disclose any leaks they find right? So you can just sit back, monitor the data disclosure thread and then integrate everyone else's findings yourself. The competition winner will be the person who can use the leaks most effectively.
\n\nI'm half considering maybe taking a break for a few weeks and coming back once most of the leaks have been found and disclosed...
", "votes": null, "replies": [ { "id": 670947, "author_name": "gunesevitan", "author_url": "", "post_date": "11/12/2019 04:23:55", "content": "We have to share otherwise it will be unfair for everyone.
", "votes": null, "replies": [] } ] }, { "id": 671383, "author_name": "vishal1310", "author_url": "", "post_date": "11/12/2019 15:58:45", "content": "Thank you Dmitry for putting your findings together. \nI just tried Catboost and it does seems to increase accuracy. \nThanks for sharing :)
", "votes": null, "replies": [ { "id": 671402, "author_name": "labdmitriy", "author_url": "", "post_date": "11/12/2019 16:37:53", "content": "Great!)
", "votes": null, "replies": [] } ] }, { "id": 673617, "author_name": "rpicatoste", "author_url": "", "post_date": "11/15/2019 08:32:21", "content": "Hello, this is my first competition as well and before starting to focus on it, I am seeing that there are quite a few problems.
\n\nThis is probably going to be a very naive/noobie question (so apologies in advance!): given the situation, that some building's test data are leaked, but that to use it, it has to be public (otherwise I assume the team would be disqualified), what use can people do of such leaked data? I think that it's only useful as an extended train set or am I missing something obvious?
\n\nOf course it's bad but in the end, in theory, everybody is going to have the same data available. Another assumption I'm doing here is that scores getting prices will have the code/models reviews or even reproduced, and therefore finding other leaks and not making them public is not possible.
\n\nDmitry thanks a lot for your sharings, they are very insightful and it seems like you invested a fair amount of work! Why not giving it a try to you current model just training when the external data disclosure finishes? (including that extra data for training).
", "votes": null, "replies": [ { "id": 674046, "author_name": "labdmitriy", "author_url": "", "post_date": "11/15/2019 20:50:47", "content": "It’s great that it is helpful! About competition - I don’t plan to continue)
", "votes": null, "replies": [] } ] }, { "id": 675367, "author_name": "gideonteo", "author_url": "", "post_date": "11/18/2019 02:02:58", "content": "@labdmitriy Impressive findings! Thanks for sharing.\nIt seems quite a lot of effort for me to breach 1.07 without leaks on a single model, much less 1.04. Without leaked data, what is your single/blended model scores on LB? How many features would it have?
", "votes": null, "replies": [ { "id": 680791, "author_name": "labdmitriy", "author_url": "", "post_date": "11/25/2019 08:19:56", "content": "Hi @gideonteo , at the moment of this post I had 1.09 single, 1.07 blended models, single model was with ~15 features
", "votes": null, "replies": [] } ] }, { "id": 680770, "author_name": "serengil", "author_url": "", "post_date": "11/25/2019 07:37:13", "content": "I compare ASHRAE data and DC data you've mentioned. However, I could not find any correlation between meter reading values for these data sets. I even applied timestamp transformation to DC data set.
", "votes": null, "replies": [ { "id": 680787, "author_name": "labdmitriy", "author_url": "", "post_date": "11/25/2019 08:15:22", "content": "I remember that I checked several sites by comparing plots in 1 hour scale, and as I mentioned before, there are gaps in this data from github as if it is corrupted data, but these plots were similar. I think these gaps can be the main reason why correlation can be low.
", "votes": null, "replies": [] }, { "id": 680800, "author_name": "serengil", "author_url": "", "post_date": "11/25/2019 08:42:38", "content": "I realize gap but still none of buildings have a correlation higher than 0.7 and mae /mean ratio less than 10% even null values dropped.
", "votes": null, "replies": [] }, { "id": 680817, "author_name": "labdmitriy", "author_url": "", "post_date": "11/25/2019 09:24:41", "content": "Maybe I'm lucky and building that I've checked has similar plots, but I found this similarity for several buildings that I checked. I have not been participating in competitions for 2 weeks, so I can't tell more details.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "670365": "Because of last findings about site locations and public/private set available, I decided to stop participating in this competition.\nIt was the first serious competition for me, I spent a lot of hours and learn a lot from Kaggle discussion/notebooks (thanks to all authors) and internet search, but it seems that now competition is more trying to search relevant info than about creating models for accurate predictions.\n\nBut anyway it was very interesting and useful :)\n\nI've decided to share with my current findings and thoughts:\n\n- I think I've found two new locations for sites:\n - site\\_id=4 is probably UC Berkeley - https://www.ucop.edu/facilities-management-services/_files/deep-efficiency-and-cogen.pdf\nI found matches with table 6.5\n - site\\_id=10 is probably Weber State University - https://apps.weber.edu/wsuimages/sustainability/Plans%20and%20Reports/ESIP%20II%20Manual%20wAppendices%202-1-2015.pdf\nI found multiple matches there, as I remember in tables and probably also in text\n\n- If we plot test data by site\\_id in the order of row\\_id, you can find 3 separates increasing stepwise lines, it seems that some data was added later to main block. Number of records for 2 last blocks (grouped by site\\_id, building\\_id) are suspiciously equal in multiple cases. \nI thought that maybe there are nans/zeros, but public LB doesn't confirm it for me. \n\n- Using CatBoost and Parameter tuning in documentation (https://catboost.ai/docs/concepts/parameter-tuning.html), model training can be increased in 3-4x times with increasing (in my case) accuracy. So one model with ~15 parameters and ~22 minutes of training can give 1.09 LB score.\n\n- I was surprised that although site\\_id and building\\_id is used in almost every public model, for me not using these 2 features in model gives me better results\n\n- One of the features that I used is “feels like” feature, using meteocalc package and formula for relative humidity based on air and dew temperatures. So feature calculation includes 2 temperatures and wind speed, and gives me more feature importance than these features separately\n\n- Besides trimming data leading zeros for site\\_id=0 as was discussed earlier, I tried to use some empirical method to truncate leading zeros with probable outliers - for example, cut leading data while cumulative mean of meter_reading is less than specific threshold. This method is not precise of course, but it can automatically truncate almost the same data for site\\_id=0 and also for other sites, and it helps me to descrease rmse locally and on LB.\n\n- There are a lot of repositories in github.com/buds-lab organization that can be useful (not only \nthat were mentioned already):\n - the-building-data-genome-project\n - building-prediction-benchmarking\n - island-of-misfit-buildings\n - energystar-plus-plus\n - temporal-features-for-nonres-buildings-library\n - enerNOC-100-building-open-dataset-analysis\n - forensic-analysis-of-building-energy-data \n\nYou can find there information about preprocessing, tz conversion, modeling etc for building energy data\n\n- I found the files that can be another potential data leak: \nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/DGS_322_Buildings-15m-By_Building-DST-gap-filled-3-2-18-508pm.csv\nIt contains data of 15-min meters reading with buildings which name is very similar to site\\_id=3 (DC) till December 2018 \nIn this folder:\nhttps://github.com/buds-lab/island-of-misfit-buildings/tree/master/data/processed \nthere is a processed version of this data (resampled by 1 hour), this data is noisy with gaps but it seems that information is correlated with train data for site_id=3 \nAlso this data contains temperature in F, so hypothesis about unit conversions to C is probably confirmed.\nDetailed information about buildings you can find here: \nhttps://github.com/buds-lab/island-of-misfit-buildings/blob/master/data/raw/dgs_metadata.csv\n\n- Finally, I want to share with 2 articles what I recently read and like very much about time series / signal processing:\nhttp://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/\nhttp://ataspinar.com/2018/12/21/a-guide-for-using-the-wavelet-transform-in-machine-learning/\n\nP.S. As final submissions I choose 2 best solutions without any leaked data, so my final result will be honest and will not hurt another participants.\n\nThanks guys!\n\n**UPD**: \nI found my notes for this competition and remember that while searching information about sites, also found probably another data leak for site\\_id = 13 (Twin Cities):\nhttps://facilities.umn.edu/energy-management/building-energy-dashboards \n\nSo for example if you search for Coffman Memorial Union which was mentioned and identified by Poe Dator as building_id=1099 with very large values of steam consumption (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114830#latest-666076) and click to the link for this building, you will find some information about energy consumption. \n\nFor example, using link \"Daily Trend Chart\" (http://meters.energy.umn.edu/enweb/HS/064_coffman/daily_trend/dte_today.html)\nyou can find current electricity/steam consumption, and for steam you will see that there are anomaly values also in the plot, so this is probably not error at all or maybe systematic error.\n\nI didn't search anymore on this site, so maybe another useful information can be found there.", "670373": "Hey buddy thanks for your findings.\nAnd perhaps you're making the right decision to quit this one 😔", "670376": "Thank you!\nI feel that for me it is correct decision :)\nGood luck!", "670384": "Appreciate your impressive findings.\nYou really did a good job in this complex competition.\nHope to see you soon in other competitions!", "670386": "I already have the labels for site 1 too, so it's a matter of time people will discover them. I'm just curious about what will happen to this competition at this point.", "670389": "Thanks Patrick! )", "670391": "Thanks Gunes for sharing this finding! Hope organizers will be honest in assessing of you in this competition on really private data.", "670396": "Looks like this competition is going to become `ASHRAE - Great Energy Leak Detector III`. Guess it's a hard pass for me.", "670814": "Another open data service such as southampton ... I'm also wondering if it's worth to continue.", "670917": "I mean, in theory everyone will have to disclose any leaks they find right? So you can just sit back, monitor the data disclosure thread and then integrate everyone else's findings yourself. The competition winner will be the person who can use the leaks most effectively. \n\nI'm half considering maybe taking a break for a few weeks and coming back once most of the leaks have been found and disclosed...", "670947": "We have to share otherwise it will be unfair for everyone.", "671028": "Me too, I'm out until the organizers say something.", "671383": "Thank you Dmitry for putting your findings together. \nI just tried Catboost and it does seems to increase accuracy. \nThanks for sharing :)", "671402": "Great!)", "673617": "Hello, this is my first competition as well and before starting to focus on it, I am seeing that there are quite a few problems. \n\nThis is probably going to be a very naive/noobie question (so apologies in advance!): given the situation, that some building's test data are leaked, but that to use it, it has to be public (otherwise I assume the team would be disqualified), what use can people do of such leaked data? I think that it's only useful as an extended train set or am I missing something obvious? \n\nOf course it's bad but in the end, in theory, everybody is going to have the same data available. Another assumption I'm doing here is that scores getting prices will have the code/models reviews or even reproduced, and therefore finding other leaks and not making them public is not possible.\n\nDmitry thanks a lot for your sharings, they are very insightful and it seems like you invested a fair amount of work! Why not giving it a try to you current model just training when the external data disclosure finishes? (including that extra data for training).", "674046": "It’s great that it is helpful! About competition - I don’t plan to continue)", "675367": "labdmitriy Impressive findings! Thanks for sharing.\nIt seems quite a lot of effort for me to breach 1.07 without leaks on a single model, much less 1.04. Without leaked data, what is your single/blended model scores on LB? How many features would it have?", "680770": "I compare ASHRAE data and DC data you've [mentioned](DGS_322_Buildings-15m-By_Building-DST-gap-filled-3-2-18-508pm.csv). However, I could not find any correlation between meter reading values for these data sets. I even applied timestamp transformation to DC data set.", "680787": "I remember that I checked several sites by comparing plots in 1 hour scale, and as I mentioned before, there are gaps in this data from github as if it is corrupted data, but these plots were similar. I think these gaps can be the main reason why correlation can be low.", "680791": "Hi @gideonteo , at the moment of this post I had 1.09 single, 1.07 blended models, single model was with ~15 features", "680800": "I realize gap but still none of buildings have a correlation higher than 0.7 and mae /mean ratio less than 10% even null values dropped.", "680817": "Maybe I'm lucky and building that I've checked has similar plots, but I found this similarity for several buildings that I checked. I have not been participating in competitions for 2 weeks, so I can't tell more details." }, "source": "meta" }