{ "id": 124984, "title": "[3rd Place] Solution", "url": "/competitions/ashrae-energy-prediction/writeups/eagle4-3rd-place-solution", "author_name": "", "post_date": "2022-04-01T16:44:26.013Z", "votes": 31, "comment_count": 16, "views": 0, "content": "

Thank you to Kaggle and ASHRAE for hosting this competition

\n

My solution is likely to disappoint some of you given the lack of sophistication.

\n

I spent most of my time (too much) on trying feature engineering, trying to look a cv split correlated to public LB (didn’t happen), trying to find a neural network architecture that will give me a boost in local cv and public LB (didn’t happen) and browsing research papers related energy prediction in the subway.

\n

I am quite clueless about web scraping but thanks to @gunesevitan for showing the way and teaching me a lot in the process nevertheless.

\n

Given diverse experiments with different CV schemes I did over the period of the competition, I decided to simply combine all the results (over 30), I got into a single submission using a simple average after selection by pearson correlation (6th on private LB).
\nIn two instances over the last 3 weeks, I used a subset without leak of these experiments and use the leak to ensemble them but I gave up because the public LB was quite poor despite a better local CV and I thought it was overfitting. However, as I selected this method as my alternative solution, it appears this was the best on the final private LB.

\n

Preprocessing:

\n

By lack of time, I only used the ideas and code of some excellent public kernels.

\n

I also wrote a script that run for a few hours where I eliminated all 0s in the same period when these 0s occurs in the same site, at the same period and across all meters. @ganfear wrote an excellent visualization here. My goal was to eliminate a maximum of \"vertical\" lines especially if they were simultaneous. I believe that this is the trick that gave me an advantage. Using this preprocessed data in the top public kernels always gave me a better LB, so I was on the right track.

\n

Feature Engineering:

\n\n
latitude_dict = {0 :28.5383,\n1 :50.9097,\n2 :33.4255,\n3 :38.9072,\n4 :37.8715,\n5 :50.9097,\n6 :40.7128,\n7 :45.4215,\n8 :28.5383,\n9 :30.2672,\n10 :40.10677,\n11 :45.4215,\n12 :53.3498,\n13 :44.9375,\n14 :38.0293,\n15: 40.7128,}\n\ntrain_df['latitude'] = train_df['site_id'].map(latitude_dict)\ntrain_df['solarHour'] = (train_df['hour']-12)*15 # to be removed\ntrain_df['solarDec'] = -23.45*np.cos(np.deg2rad(360*(train_df['doy']+10)/365)) # to be removed\ntrain_df['horizsolar'] = np.cos(np.deg2rad(train_df['solarHour']))*np.cos(np.deg2rad(train_df['solarDec']))*np.cos(np.deg2rad(train_df['latitude'])) + np.sin(np.deg2rad(train_df['solarDec']))*np.sin(np.deg2rad(train_df['latitude']))\n\ntrain_df['horizsolar'] = train_df['horizsolar'].apply(lambda x: 0 if x <0 else x)\n
\n

it is supposed to calculate the solar horizontal radiation coming into the building.

\n

Models:

\n

I trained Keras CNN (@aerdem4 style), LightGBM and Catboost on diverse version of cleaned data, various feature selection (including removal of building_id) - without ever beating the best kernel without leak to my disappointment.

\n

I had better success on local CV with meter level model using decision trees rather than using all the meter in the same decision trees model. Catboost and Lightgbm are clearly using different selection of features for the splits, so they are quite complementary.

\n

Only towards the end of the competition, I realized that the NN was giving approximately the same results on both local cv and public LB than decision trees models and that most of the performance was due to the cleaning more than my feature engineering. However, the results were quite uncorrelated, so it was good for diversity.

\n

Ensembling:

\n

My best submission is an ensemble of a small samples of leak free (~10) predictions among all my experiments. Using lightgbm, I used these different predictions as feature with the addition of the original feature “meter” in order to reduce overfitting. The leaky rows are used as training set in the ensembling model.

\n

My idea was that it is likely that the prediction of first 11% of the test set (included in the public LB) are going to be much better than the last 11% of the test set (private LB), so if I were to use the leak on the test set, I may be able to correct the last 11%. I was expecting a small hit on public LB but my public LB took such a larger hit than expected (+.025) that I thought I was overfitting and didn’t pursue that path. I kept the submission as my second final submission which became my best on private LB.

\n

Didn’t work for me or didn’t give any significative improvement:

\n

Pseudo labeling (read @hengck23 posts for more information)
\nNN decoder-encoder using the leak in @mjahrer style (reniew seems to have succeed in this direction: see here (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122203#699375)
\nThermometer encoding on cloud coverage
\nTreat temperature features as categorical variables (given limited number of values) and use target encoding.
\nAdaptation of @anokas tool for time series feature
\n…

\n

Otherwise, I would like to thank all the people who taught me a lot on Kaggle among whom @Raddar, @cpmpml and @titericz during the years.

\n

Update:
\ngithub code available here

", "messages": [ { "id": "713104", "postDate": "01/07/2020 22:41:23", "content": "

Thank you to Kaggle and ASHRAE for hosting this competition

\n

My solution is likely to disappoint some of you given the lack of sophistication.

\n

I spent most of my time (too much) on trying feature engineering, trying to look a cv split correlated to public LB (didn’t happen), trying to find a neural network architecture that will give me a boost in local cv and public LB (didn’t happen) and browsing research papers related energy prediction in the subway.

\n

I am quite clueless about web scraping but thanks to @gunesevitan for showing the way and teaching me a lot in the process nevertheless.

\n

Given diverse experiments with different CV schemes I did over the period of the competition, I decided to simply combine all the results (over 30), I got into a single submission using a simple average after selection by pearson correlation (6th on private LB).
\nIn two instances over the last 3 weeks, I used a subset without leak of these experiments and use the leak to ensemble them but I gave up because the public LB was quite poor despite a better local CV and I thought it was overfitting. However, as I selected this method as my alternative solution, it appears this was the best on the final private LB.

\n

Preprocessing:

\n

By lack of time, I only used the ideas and code of some excellent public kernels.

\n

I also wrote a script that run for a few hours where I eliminated all 0s in the same period when these 0s occurs in the same site, at the same period and across all meters. @ganfear wrote an excellent visualization here. My goal was to eliminate a maximum of \"vertical\" lines especially if they were simultaneous. I believe that this is the trick that gave me an advantage. Using this preprocessed data in the top public kernels always gave me a better LB, so I was on the right track.

\n

Feature Engineering:

\n\n
latitude_dict = {0 :28.5383,\n1 :50.9097,\n2 :33.4255,\n3 :38.9072,\n4 :37.8715,\n5 :50.9097,\n6 :40.7128,\n7 :45.4215,\n8 :28.5383,\n9 :30.2672,\n10 :40.10677,\n11 :45.4215,\n12 :53.3498,\n13 :44.9375,\n14 :38.0293,\n15: 40.7128,}\n\ntrain_df['latitude'] = train_df['site_id'].map(latitude_dict)\ntrain_df['solarHour'] = (train_df['hour']-12)*15 # to be removed\ntrain_df['solarDec'] = -23.45*np.cos(np.deg2rad(360*(train_df['doy']+10)/365)) # to be removed\ntrain_df['horizsolar'] = np.cos(np.deg2rad(train_df['solarHour']))*np.cos(np.deg2rad(train_df['solarDec']))*np.cos(np.deg2rad(train_df['latitude'])) + np.sin(np.deg2rad(train_df['solarDec']))*np.sin(np.deg2rad(train_df['latitude']))\n\ntrain_df['horizsolar'] = train_df['horizsolar'].apply(lambda x: 0 if x <0 else x)\n
\n

it is supposed to calculate the solar horizontal radiation coming into the building.

\n

Models:

\n

I trained Keras CNN (@aerdem4 style), LightGBM and Catboost on diverse version of cleaned data, various feature selection (including removal of building_id) - without ever beating the best kernel without leak to my disappointment.

\n

I had better success on local CV with meter level model using decision trees rather than using all the meter in the same decision trees model. Catboost and Lightgbm are clearly using different selection of features for the splits, so they are quite complementary.

\n

Only towards the end of the competition, I realized that the NN was giving approximately the same results on both local cv and public LB than decision trees models and that most of the performance was due to the cleaning more than my feature engineering. However, the results were quite uncorrelated, so it was good for diversity.

\n

Ensembling:

\n

My best submission is an ensemble of a small samples of leak free (~10) predictions among all my experiments. Using lightgbm, I used these different predictions as feature with the addition of the original feature “meter” in order to reduce overfitting. The leaky rows are used as training set in the ensembling model.

\n

My idea was that it is likely that the prediction of first 11% of the test set (included in the public LB) are going to be much better than the last 11% of the test set (private LB), so if I were to use the leak on the test set, I may be able to correct the last 11%. I was expecting a small hit on public LB but my public LB took such a larger hit than expected (+.025) that I thought I was overfitting and didn’t pursue that path. I kept the submission as my second final submission which became my best on private LB.

\n

Didn’t work for me or didn’t give any significative improvement:

\n

Pseudo labeling (read @hengck23 posts for more information)
\nNN decoder-encoder using the leak in @mjahrer style (reniew seems to have succeed in this direction: see here (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122203#699375)
\nThermometer encoding on cloud coverage
\nTreat temperature features as categorical variables (given limited number of values) and use target encoding.
\nAdaptation of @anokas tool for time series feature
\n…

\n

Otherwise, I would like to thank all the people who taught me a lot on Kaggle among whom @Raddar, @cpmpml and @titericz during the years.

\n

Update:
\ngithub code available here

", "rawMarkdown": "Thank you to Kaggle and ASHRAE for hosting this competition\n\nMy solution is likely to disappoint some of you given the lack of sophistication. \n\nI spent most of my time (too much) on trying feature engineering, trying to look a cv split correlated to public LB (didn’t happen), trying to find a neural network architecture that will give me a boost in local cv and public LB (didn’t happen) and browsing research papers related energy prediction in the subway. \n\nI am quite clueless about web scraping but thanks to @gunesevitan for showing the way and teaching me a lot in the process nevertheless. \n\nGiven diverse experiments with different CV schemes I did over the period of the competition, I decided to simply combine all the results (over 30), I got into a single submission using a simple average after selection by pearson correlation (6th on private LB). \nIn two instances over the last 3 weeks, I used a subset without leak of these experiments and use the leak to ensemble them but I gave up because the public LB was quite poor despite a better local CV and I thought it was overfitting. However, as I selected this method as my alternative solution, it appears this was the best on the final private LB.\n\n\n# Preprocessing:\n\nBy lack of time, I only used the ideas and code of some excellent public kernels. \n\nI also wrote a script that run for a few hours where I eliminated all 0s in the same period when these 0s occurs in the same site, at the same period and across all meters. @ganfear wrote an excellent visualization [here](https://www.kaggle.com/ganfear/missing-data-and-zeros-visualized). My goal was to eliminate a maximum of \"vertical\" lines especially if they were simultaneous. I believe that this is the trick that gave me an advantage. Using this preprocessed data in the top public kernels always gave me a better LB, so I was on the right track.\n\n\n# Feature Engineering:\n\n-\tRow feature from train/test, weather metadata, and building metadata\n-\tCount features and combination of features (see excellent write up on IEEE competition)\n-\tLag on row features but only the temperature feature lags seemed to be actually useful.\n-\tFeatures mentioned in public kernel such as RH, feel_likes and presence of other meters seems to have a marginal upside.\n-\tA feature that I only included in the other submission in the last days found in a research paper was working well with meter==1 was :\n\n```\nlatitude_dict = {0 :28.5383,\n1 :50.9097,\n2 :33.4255,\n3 :38.9072,\n4 :37.8715,\n5 :50.9097,\n6 :40.7128,\n7 :45.4215,\n8 :28.5383,\n9 :30.2672,\n10 :40.10677,\n11 :45.4215,\n12 :53.3498,\n13 :44.9375,\n14 :38.0293,\n15: 40.7128,}\n\ntrain_df['latitude'] = train_df['site_id'].map(latitude_dict)\ntrain_df['solarHour'] = (train_df['hour']-12)*15 # to be removed\ntrain_df['solarDec'] = -23.45*np.cos(np.deg2rad(360*(train_df['doy']+10)/365)) # to be removed\ntrain_df['horizsolar'] = np.cos(np.deg2rad(train_df['solarHour']))*np.cos(np.deg2rad(train_df['solarDec']))*np.cos(np.deg2rad(train_df['latitude'])) + np.sin(np.deg2rad(train_df['solarDec']))*np.sin(np.deg2rad(train_df['latitude']))\n\ntrain_df['horizsolar'] = train_df['horizsolar'].apply(lambda x: 0 if x <0 else x)\n```\nit is supposed to calculate the solar horizontal radiation coming into the building.\n \n \n# Models:\n\nI trained Keras CNN (@aerdem4 style), LightGBM and Catboost on diverse version of cleaned data, various feature selection (including removal of building_id) - without ever beating the best kernel without leak to my disappointment. \n\nI had better success on local CV with meter level model using decision trees rather than using all the meter in the same decision trees model. Catboost and Lightgbm are clearly using different selection of features for the splits, so they are quite complementary.\n\nOnly towards the end of the competition, I realized that the NN was giving approximately the same results on both local cv and public LB than decision trees models and that most of the performance was due to the cleaning more than my feature engineering. However, the results were quite uncorrelated, so it was good for diversity.\n\n\n# Ensembling:\n\nMy best submission is an ensemble of a small samples of leak free (~10) predictions among all my experiments. Using lightgbm, I used these different predictions as feature with the addition of the original feature “meter” in order to reduce overfitting. The leaky rows are used as training set in the ensembling model.\n\nMy idea was that it is likely that the prediction of first 11% of the test set (included in the public LB) are going to be much better than the last 11% of the test set (private LB), so if I were to use the leak on the test set, I may be able to correct the last 11%. I was expecting a small hit on public LB but my public LB took such a larger hit than expected (+.025) that I thought I was overfitting and didn’t pursue that path. I kept the submission as my second final submission which became my best on private LB.\n\n\n\n# Didn’t work for me or didn’t give any significative improvement:\n\nPseudo labeling (read @hengck23 posts for more information)\nNN decoder-encoder using the leak in @mjahrer style (reniew seems to have succeed in this direction: see here (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122203#699375)\nThermometer encoding on cloud coverage\nTreat temperature features as categorical variables (given limited number of values) and use target encoding.\nAdaptation of @anokas tool for time series feature\n…\n\nOtherwise, I would like to thank all the people who taught me a lot on Kaggle among whom @Raddar, @cpmpml and @titericz during the years.\n\n\nUpdate:\ngithub code available [here](https://github.com/chabir/kaggle_materials/tree/master/ASHRAE%20-%20Great%20Energy%20Predictor%20III%20-%203rd%7C3614)", "votes": null }, { "id": "713157", "postDate": "01/08/2020 01:14:34", "content": "

Congrats @chabir and thanks for sharing.

", "rawMarkdown": "Congrats @chabir and thanks for sharing.", "votes": null }, { "id": "713225", "postDate": "01/08/2020 03:33:38", "content": "

Congratulations!!\nGrear Write-Up\nThanks for sharing your Approach & Insights!! @chabir

", "rawMarkdown": "Congratulations!!\nGrear Write-Up\nThanks for sharing your Approach & Insights!! @chabir", "votes": null }, { "id": "713414", "postDate": "01/08/2020 09:11:23", "content": "

Thanks for sharing. I think unsophisticated solution is more useful from learning perspective.

", "rawMarkdown": "Thanks for sharing. I think unsophisticated solution is more useful from learning perspective.", "votes": null }, { "id": "714033", "postDate": "01/09/2020 00:41:18", "content": "

Congratulations and thanks for sharing @chabir! What was the CV scheme you use?

", "rawMarkdown": "Congratulations and thanks for sharing @chabir! What was the CV scheme you use?", "votes": null }, { "id": "714530", "postDate": "01/09/2020 14:16:36", "content": "

The great things because of the leaks is that you could afford to have use several CV scheme to diversify your models and use the leaky rows for ensembling. At the end, I settled with 3 folds unshuffled.

", "rawMarkdown": "The great things because of the leaks is that you could afford to have use several CV scheme to diversify your models and use the leaky rows for ensembling. At the end, I settled with 3 folds unshuffled.", "votes": null }, { "id": "714630", "postDate": "01/09/2020 15:57:31", "content": "

Congrats on the result! And thanks for citing me ;)

", "rawMarkdown": "Congrats on the result! And thanks for citing me ;)", "votes": null }, { "id": "714684", "postDate": "01/09/2020 16:37:02", "content": "

De rien, Oncle Cpmp! Your posts are really appreciated for their technicality or for their diplomacy.

", "rawMarkdown": "De rien, Oncle Cpmp! Your posts are really appreciated for their technicality or for their diplomacy.", "votes": null }, { "id": "714792", "postDate": "01/09/2020 18:52:26", "content": "

Congrats on your solo 3rd place!\nI admit that I do a lot of Keras CNN nowadays but I have nothing special to be called as my style 😀

", "rawMarkdown": "Congrats on your solo 3rd place!\nI admit that I do a lot of Keras CNN nowadays but I have nothing special to be called as my style 😀", "votes": null }, { "id": "715069", "postDate": "01/10/2020 05:16:11", "content": "

@chabir Congratulations

", "rawMarkdown": "chabir Congratulations", "votes": null }, { "id": "715737", "postDate": "01/10/2020 19:07:34", "content": "

Congrats @chabir :)

\n\n

Thanks for the credit, was it FTIM that you tried using?

", "rawMarkdown": "Congrats @chabir :)\n\nThanks for the credit, was it FTIM that you tried using?", "votes": null }, { "id": "715750", "postDate": "01/10/2020 19:23:15", "content": "

Yes

", "rawMarkdown": "Yes", "votes": null }, { "id": "716860", "postDate": "01/12/2020 11:23:56", "content": "

Congrats @chabir and thanks for sharing.

", "rawMarkdown": "Congrats @chabir and thanks for sharing.", "votes": null }, { "id": "716903", "postDate": "01/12/2020 12:40:38", "content": "

Thanks for sharing, learned a lot.

", "rawMarkdown": "Thanks for sharing, learned a lot.", "votes": null }, { "id": "717048", "postDate": "01/12/2020 16:52:49", "content": "

Congratulations @chabir! Thanks for Sharing your Approach. Looking forward to have more awesome Kernals from you! 😄

", "rawMarkdown": "Congratulations @chabir! Thanks for Sharing your Approach. Looking forward to have more awesome Kernals from you! 😄", "votes": null }, { "id": "717352", "postDate": "01/13/2020 03:46:39", "content": "

Congratulations to our third-place winner! 🙏 👍 The third-place solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned!

\n\n

Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.

", "rawMarkdown": "Congratulations to our third-place winner! 🙏 👍 The third-place solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned! \n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "votes": null }, { "id": "717944", "postDate": "01/13/2020 20:15:34", "content": "

Thanks for sharing @chabir

", "rawMarkdown": "Thanks for sharing @chabir", "votes": null } ], "comments": [ { "id": 713157, "author_name": "sheriytm", "author_url": "", "post_date": "01/08/2020 01:14:34", "content": "

Congrats @chabir and thanks for sharing.

", "votes": null, "replies": [] }, { "id": 713225, "author_name": "veeralakrishna", "author_url": "", "post_date": "01/08/2020 03:33:38", "content": "

Congratulations!!\nGrear Write-Up\nThanks for sharing your Approach & Insights!! @chabir

", "votes": null, "replies": [] }, { "id": 713414, "author_name": "rahullalu", "author_url": "", "post_date": "01/08/2020 09:11:23", "content": "

Thanks for sharing. I think unsophisticated solution is more useful from learning perspective.

", "votes": null, "replies": [] }, { "id": 714033, "author_name": "mavillan", "author_url": "", "post_date": "01/09/2020 00:41:18", "content": "

Congratulations and thanks for sharing @chabir! What was the CV scheme you use?

", "votes": null, "replies": [ { "id": 714530, "author_name": "chabir", "author_url": "", "post_date": "01/09/2020 14:16:36", "content": "

The great things because of the leaks is that you could afford to have use several CV scheme to diversify your models and use the leaky rows for ensembling. At the end, I settled with 3 folds unshuffled.

", "votes": null, "replies": [] } ] }, { "id": 714630, "author_name": "cpmpml", "author_url": "", "post_date": "01/09/2020 15:57:31", "content": "

Congrats on the result! And thanks for citing me ;)

", "votes": null, "replies": [ { "id": 714684, "author_name": "chabir", "author_url": "", "post_date": "01/09/2020 16:37:02", "content": "

De rien, Oncle Cpmp! Your posts are really appreciated for their technicality or for their diplomacy.

", "votes": null, "replies": [] } ] }, { "id": 714792, "author_name": "aerdem4", "author_url": "", "post_date": "01/09/2020 18:52:26", "content": "

Congrats on your solo 3rd place!\nI admit that I do a lot of Keras CNN nowadays but I have nothing special to be called as my style 😀

", "votes": null, "replies": [] }, { "id": 715069, "author_name": "pradeepmuniasamy", "author_url": "", "post_date": "01/10/2020 05:16:11", "content": "

@chabir Congratulations

", "votes": null, "replies": [] }, { "id": 715737, "author_name": "anokas", "author_url": "", "post_date": "01/10/2020 19:07:34", "content": "

Congrats @chabir :)

\n\n

Thanks for the credit, was it FTIM that you tried using?

", "votes": null, "replies": [ { "id": 715750, "author_name": "chabir", "author_url": "", "post_date": "01/10/2020 19:23:15", "content": "

Yes

", "votes": null, "replies": [] } ] }, { "id": 716860, "author_name": "shahules", "author_url": "", "post_date": "01/12/2020 11:23:56", "content": "

Congrats @chabir and thanks for sharing.

", "votes": null, "replies": [] }, { "id": 716903, "author_name": "abhinand05", "author_url": "", "post_date": "01/12/2020 12:40:38", "content": "

Thanks for sharing, learned a lot.

", "votes": null, "replies": [] }, { "id": 717048, "author_name": "risabbiswas19", "author_url": "", "post_date": "01/12/2020 16:52:49", "content": "

Congratulations @chabir! Thanks for Sharing your Approach. Looking forward to have more awesome Kernals from you! 😄

", "votes": null, "replies": [] }, { "id": 717352, "author_name": "claytonmiller", "author_url": "", "post_date": "01/13/2020 03:46:39", "content": "

Congratulations to our third-place winner! 🙏 👍 The third-place solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned!

\n\n

Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.

", "votes": null, "replies": [] }, { "id": 717944, "author_name": "erkanhatipoglu", "author_url": "", "post_date": "01/13/2020 20:15:34", "content": "

Thanks for sharing @chabir

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "713104": "Thank you to Kaggle and ASHRAE for hosting this competition\n\nMy solution is likely to disappoint some of you given the lack of sophistication. \n\nI spent most of my time (too much) on trying feature engineering, trying to look a cv split correlated to public LB (didn’t happen), trying to find a neural network architecture that will give me a boost in local cv and public LB (didn’t happen) and browsing research papers related energy prediction in the subway. \n\nI am quite clueless about web scraping but thanks to @gunesevitan for showing the way and teaching me a lot in the process nevertheless. \n\nGiven diverse experiments with different CV schemes I did over the period of the competition, I decided to simply combine all the results (over 30), I got into a single submission using a simple average after selection by pearson correlation (6th on private LB). \nIn two instances over the last 3 weeks, I used a subset without leak of these experiments and use the leak to ensemble them but I gave up because the public LB was quite poor despite a better local CV and I thought it was overfitting. However, as I selected this method as my alternative solution, it appears this was the best on the final private LB.\n\n\n# Preprocessing:\n\nBy lack of time, I only used the ideas and code of some excellent public kernels. \n\nI also wrote a script that run for a few hours where I eliminated all 0s in the same period when these 0s occurs in the same site, at the same period and across all meters. @ganfear wrote an excellent visualization [here](https://www.kaggle.com/ganfear/missing-data-and-zeros-visualized). My goal was to eliminate a maximum of \"vertical\" lines especially if they were simultaneous. I believe that this is the trick that gave me an advantage. Using this preprocessed data in the top public kernels always gave me a better LB, so I was on the right track.\n\n\n# Feature Engineering:\n\n-\tRow feature from train/test, weather metadata, and building metadata\n-\tCount features and combination of features (see excellent write up on IEEE competition)\n-\tLag on row features but only the temperature feature lags seemed to be actually useful.\n-\tFeatures mentioned in public kernel such as RH, feel_likes and presence of other meters seems to have a marginal upside.\n-\tA feature that I only included in the other submission in the last days found in a research paper was working well with meter==1 was :\n\n```\nlatitude_dict = {0 :28.5383,\n1 :50.9097,\n2 :33.4255,\n3 :38.9072,\n4 :37.8715,\n5 :50.9097,\n6 :40.7128,\n7 :45.4215,\n8 :28.5383,\n9 :30.2672,\n10 :40.10677,\n11 :45.4215,\n12 :53.3498,\n13 :44.9375,\n14 :38.0293,\n15: 40.7128,}\n\ntrain_df['latitude'] = train_df['site_id'].map(latitude_dict)\ntrain_df['solarHour'] = (train_df['hour']-12)*15 # to be removed\ntrain_df['solarDec'] = -23.45*np.cos(np.deg2rad(360*(train_df['doy']+10)/365)) # to be removed\ntrain_df['horizsolar'] = np.cos(np.deg2rad(train_df['solarHour']))*np.cos(np.deg2rad(train_df['solarDec']))*np.cos(np.deg2rad(train_df['latitude'])) + np.sin(np.deg2rad(train_df['solarDec']))*np.sin(np.deg2rad(train_df['latitude']))\n\ntrain_df['horizsolar'] = train_df['horizsolar'].apply(lambda x: 0 if x <0 else x)\n```\nit is supposed to calculate the solar horizontal radiation coming into the building.\n \n \n# Models:\n\nI trained Keras CNN (@aerdem4 style), LightGBM and Catboost on diverse version of cleaned data, various feature selection (including removal of building_id) - without ever beating the best kernel without leak to my disappointment. \n\nI had better success on local CV with meter level model using decision trees rather than using all the meter in the same decision trees model. Catboost and Lightgbm are clearly using different selection of features for the splits, so they are quite complementary.\n\nOnly towards the end of the competition, I realized that the NN was giving approximately the same results on both local cv and public LB than decision trees models and that most of the performance was due to the cleaning more than my feature engineering. However, the results were quite uncorrelated, so it was good for diversity.\n\n\n# Ensembling:\n\nMy best submission is an ensemble of a small samples of leak free (~10) predictions among all my experiments. Using lightgbm, I used these different predictions as feature with the addition of the original feature “meter” in order to reduce overfitting. The leaky rows are used as training set in the ensembling model.\n\nMy idea was that it is likely that the prediction of first 11% of the test set (included in the public LB) are going to be much better than the last 11% of the test set (private LB), so if I were to use the leak on the test set, I may be able to correct the last 11%. I was expecting a small hit on public LB but my public LB took such a larger hit than expected (+.025) that I thought I was overfitting and didn’t pursue that path. I kept the submission as my second final submission which became my best on private LB.\n\n\n\n# Didn’t work for me or didn’t give any significative improvement:\n\nPseudo labeling (read @hengck23 posts for more information)\nNN decoder-encoder using the leak in @mjahrer style (reniew seems to have succeed in this direction: see here (https://www.kaggle.com/c/ashrae-energy-prediction/discussion/122203#699375)\nThermometer encoding on cloud coverage\nTreat temperature features as categorical variables (given limited number of values) and use target encoding.\nAdaptation of @anokas tool for time series feature\n…\n\nOtherwise, I would like to thank all the people who taught me a lot on Kaggle among whom @Raddar, @cpmpml and @titericz during the years.\n\n\nUpdate:\ngithub code available [here](https://github.com/chabir/kaggle_materials/tree/master/ASHRAE%20-%20Great%20Energy%20Predictor%20III%20-%203rd%7C3614)", "713157": "Congrats @chabir and thanks for sharing.", "713225": "Congratulations!!\nGrear Write-Up\nThanks for sharing your Approach & Insights!! @chabir", "713414": "Thanks for sharing. I think unsophisticated solution is more useful from learning perspective.", "714033": "Congratulations and thanks for sharing @chabir! What was the CV scheme you use?", "714530": "The great things because of the leaks is that you could afford to have use several CV scheme to diversify your models and use the leaky rows for ensembling. At the end, I settled with 3 folds unshuffled.", "714630": "Congrats on the result! And thanks for citing me ;)", "714684": "De rien, Oncle Cpmp! Your posts are really appreciated for their technicality or for their diplomacy.", "714792": "Congrats on your solo 3rd place!\nI admit that I do a lot of Keras CNN nowadays but I have nothing special to be called as my style 😀", "715069": "chabir Congratulations", "715737": "Congrats @chabir :)\n\nThanks for the credit, was it FTIM that you tried using?", "715750": "Yes", "716860": "Congrats @chabir and thanks for sharing.", "716903": "Thanks for sharing, learned a lot.", "717048": "Congratulations @chabir! Thanks for Sharing your Approach. Looking forward to have more awesome Kernals from you! 😄", "717352": "Congratulations to our third-place winner! 🙏 👍 The third-place solution will be organized and added to the other top five winners in a publication and open-source repository. Please stay tuned! \n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "717944": "Thanks for sharing @chabir" }, "source": "meta" }