Hey Dear Kagglers :)
\n\nFirst of all, congratulations to all winners and participants!\nAnd many thanks to all contributors for their amazing kernels and topics!
\n\nWanted to put my 2 cents in - so you can find a brief overview of my solution below.
\n\nStarted work on the competition quite late - just two weeks before the end and my main motivation was to experiment around the use of deep learning with tabular data (and for this kind of problems).
\n\nLike probably most of the participants I ended up with 2 layer learning architecture: base layer and ensemble.
\n\nBecause of my late start, I didn't really have the time for something fancy (FE is usually my main focus. Not this time though :smile: ).\nI ended up with 4 different datasets. On all, I applied cleansing (filtered out the bad rows) as per this great kernel (please upvote: https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks).\nUsed interpolation to fill in the missing values (on all), added \"is_missing\" features (to one of the sets).\nIn addition generated some:\n* weather-based features (aggregates/grouping, lag/rolling (moving averages/max/min etc))\n* time-based features (including holidays for one of the sets)\n* target encodings (in one of the sets).
\n\nI draw inspiration from many great kernels in the process - check the list below and please go and upvote the amazing work of the authors - they deserve it!
\n\nModels\n\nMy main focus was here and my goal was to have as diverse models as possible.\nI did NOT touche the leak data in any of the models of the base layer.\nTrained a total of 19 models. Amongst those:
\n\nDeep NNs\n\nAs I mentioned my goal was mostly to play around these - so I put significant effort here. My best NNs didn't disappoint - they were approaching the performance of the best public LGBMs (1.09 LB) - but of course, had a very different \"point of view\" on the problem. That made them extremely useful in the final ensemble, and to that, I mostly attribute my good final score. Some things that worked great here:\n* entity embeddings for the categorical variables\n* Radam optimizer (that came as a bit of surprise - Radam didn't appear to make a huge difference for me on CV competitions - but here it showed significant advantage. Maybe the size of the network, or the complexity of the problem ... will dig deeper into that for sure)\n* Adam with CyclicLR + ReduceLROnPlateau schedulers + longer training\n* weather lag features + (few) time features\nThings that didn't work:\n* more features (not surprised)\n* \"is_missing\" features (kind of surprised)
\n\nIn short - very happy with my findings. Deep NNs will definitely be part of my considerations in future tabular data competitions.
\n\nLGBMs (of course :smile: ):\n\nThe second major thing that boosted my score.\nBased on my previous experience, stacking works better than blending when testing data is not cardinally different (and that can be argued of course).\n(NOT) very small disclaimer here:\n* this is not valid in all cases\n* well... how similar is that test data really...\n* how proper your base layer is - is it diverse enough? Are you overfitting there already? Do you introduce leak in the meta model training data if use diverse validation/oof prediction schemes for the base models?\n* there are also many other (important) variables - like what algorithm will be used for the meta learner, how it will be tuned, validated etc.\n* very risky approach in general (overfitting is BIG concern...).
\n\nStill ... it worked well for me in the past.\nAnd worked this time as well.\nLet me give you some numbers:\n* my best stack: 1.245 (PL)\n* my best blend: 1.270 (PL)
\n\nWhat worked well here: simple stuff (GLM, Gaussian, 5-fold CV, some regularization, adding meter to the training set). Performed very well for me also in the past (Elo Merchant Category Recommendation competition for example).\nWhat didn't work that great: more sophisticated stuff - Random Forest and even XGB, LGBM. All tend to overfit on my base data...
\n\nThese are pretty much the points, I want to make.\nOne more time - please spend a minute to upvote the amazing work of the contributors below.\nAnd upvote this, in case you find it useful :smile:\nCheers folks and \"see\" you on the next competition.
\n\nGreat kernels that deserve your vote:\nhttps://www.kaggle.com/rohanrao/ashrae-divide-and-conquer\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type\nhttps://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15\nhttps://www.kaggle.com/roydatascience/ashrae-energy-prediction-using-stratified-kfold\nhttps://www.kaggle.com/iwatatakuya/ashrae-kfold-lightgbm-without-building-id\nhttps://www.kaggle.com/kailex/ac-dc\nhttps://www.kaggle.com/rohanrao/ashrae-half-and-half\nhttps://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\nhttps://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\nhttps://www.kaggle.com/mimoudata/ashrae-lightgbm-without-leak\nhttps://www.kaggle.com/starl1ght/ashrae-stacked-regression-lasso-ridge-lgbm\nhttps://www.kaggle.com/tunguz/ashrae-histgradientboosting\nhttps://www.kaggle.com/yamsam/ashrae-leak-data-station
", "messages": [ { "id": "714856", "postDate": "01/09/2020 21:14:16", "content": "Hey Dear Kagglers :)
\n\nFirst of all, congratulations to all winners and participants!\nAnd many thanks to all contributors for their amazing kernels and topics!
\n\nWanted to put my 2 cents in - so you can find a brief overview of my solution below.
\n\nStarted work on the competition quite late - just two weeks before the end and my main motivation was to experiment around the use of deep learning with tabular data (and for this kind of problems).
\n\nLike probably most of the participants I ended up with 2 layer learning architecture: base layer and ensemble.
\n\nBecause of my late start, I didn't really have the time for something fancy (FE is usually my main focus. Not this time though :smile: ).\nI ended up with 4 different datasets. On all, I applied cleansing (filtered out the bad rows) as per this great kernel (please upvote: https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks).\nUsed interpolation to fill in the missing values (on all), added \"is_missing\" features (to one of the sets).\nIn addition generated some:\n* weather-based features (aggregates/grouping, lag/rolling (moving averages/max/min etc))\n* time-based features (including holidays for one of the sets)\n* target encodings (in one of the sets).
\n\nI draw inspiration from many great kernels in the process - check the list below and please go and upvote the amazing work of the authors - they deserve it!
\n\nModels\n\nMy main focus was here and my goal was to have as diverse models as possible.\nI did NOT touche the leak data in any of the models of the base layer.\nTrained a total of 19 models. Amongst those:
\n\nDeep NNs\n\nAs I mentioned my goal was mostly to play around these - so I put significant effort here. My best NNs didn't disappoint - they were approaching the performance of the best public LGBMs (1.09 LB) - but of course, had a very different \"point of view\" on the problem. That made them extremely useful in the final ensemble, and to that, I mostly attribute my good final score. Some things that worked great here:\n* entity embeddings for the categorical variables\n* Radam optimizer (that came as a bit of surprise - Radam didn't appear to make a huge difference for me on CV competitions - but here it showed significant advantage. Maybe the size of the network, or the complexity of the problem ... will dig deeper into that for sure)\n* Adam with CyclicLR + ReduceLROnPlateau schedulers + longer training\n* weather lag features + (few) time features\nThings that didn't work:\n* more features (not surprised)\n* \"is_missing\" features (kind of surprised)
\n\nIn short - very happy with my findings. Deep NNs will definitely be part of my considerations in future tabular data competitions.
\n\nLGBMs (of course :smile: ):\n\nThe second major thing that boosted my score.\nBased on my previous experience, stacking works better than blending when testing data is not cardinally different (and that can be argued of course).\n(NOT) very small disclaimer here:\n* this is not valid in all cases\n* well... how similar is that test data really...\n* how proper your base layer is - is it diverse enough? Are you overfitting there already? Do you introduce leak in the meta model training data if use diverse validation/oof prediction schemes for the base models?\n* there are also many other (important) variables - like what algorithm will be used for the meta learner, how it will be tuned, validated etc.\n* very risky approach in general (overfitting is BIG concern...).
\n\nStill ... it worked well for me in the past.\nAnd worked this time as well.\nLet me give you some numbers:\n* my best stack: 1.245 (PL)\n* my best blend: 1.270 (PL)
\n\nWhat worked well here: simple stuff (GLM, Gaussian, 5-fold CV, some regularization, adding meter to the training set). Performed very well for me also in the past (Elo Merchant Category Recommendation competition for example).\nWhat didn't work that great: more sophisticated stuff - Random Forest and even XGB, LGBM. All tend to overfit on my base data...
\n\nThese are pretty much the points, I want to make.\nOne more time - please spend a minute to upvote the amazing work of the contributors below.\nAnd upvote this, in case you find it useful :smile:\nCheers folks and \"see\" you on the next competition.
\n\nGreat kernels that deserve your vote:\nhttps://www.kaggle.com/rohanrao/ashrae-divide-and-conquer\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type\nhttps://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15\nhttps://www.kaggle.com/roydatascience/ashrae-energy-prediction-using-stratified-kfold\nhttps://www.kaggle.com/iwatatakuya/ashrae-kfold-lightgbm-without-building-id\nhttps://www.kaggle.com/kailex/ac-dc\nhttps://www.kaggle.com/rohanrao/ashrae-half-and-half\nhttps://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\nhttps://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\nhttps://www.kaggle.com/mimoudata/ashrae-lightgbm-without-leak\nhttps://www.kaggle.com/starl1ght/ashrae-stacked-regression-lasso-ridge-lgbm\nhttps://www.kaggle.com/tunguz/ashrae-histgradientboosting\nhttps://www.kaggle.com/yamsam/ashrae-leak-data-station
", "rawMarkdown": "Hey Dear Kagglers :)\n\nFirst of all, congratulations to all winners and participants!\nAnd many thanks to all contributors for their amazing kernels and topics!\n\nWanted to put my 2 cents in - so you can find a brief overview of my solution below.\n\n## Motivation\nStarted work on the competition quite late - just two weeks before the end and my main motivation was to experiment around the use of deep learning with tabular data (and for this kind of problems).\n\n## Architecture\nLike probably most of the participants I ended up with 2 layer learning architecture: base layer and ensemble.\n\n### Base layer:\n#### Feature engineering\nBecause of my late start, I didn't really have the time for something fancy (FE is usually my main focus. Not this time though :smile: ).\nI ended up with 4 different datasets. On all, I applied cleansing (filtered out the bad rows) as per this great kernel (please upvote: https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks).\nUsed interpolation to fill in the missing values (on all), added \"is_missing\" features (to one of the sets).\nIn addition generated some:\n* weather-based features (aggregates/grouping, lag/rolling (moving averages/max/min etc))\n* time-based features (including holidays for one of the sets)\n* target encodings (in one of the sets).\n\nI draw inspiration from many great kernels in the process - check the list below and please go and upvote the amazing work of the authors - they deserve it!\n\n#### Models\nMy main focus was here and my goal was to have as diverse models as possible.\nI did NOT touche the leak data in any of the models of the base layer.\nTrained a total of 19 models. Amongst those:\n##### Deep NNs \nAs I mentioned my goal was mostly to play around these - so I put significant effort here. My best NNs didn't disappoint - they were approaching the performance of the best public LGBMs (1.09 LB) - but of course, had a very different \"point of view\" on the problem. That made them extremely useful in the final ensemble, and to that, I mostly attribute my good final score. Some things that worked great here:\n* entity embeddings for the categorical variables\n* Radam optimizer (that came as a bit of surprise - Radam didn't appear to make a huge difference for me on CV competitions - but here it showed significant advantage. Maybe the size of the network, or the complexity of the problem ... will dig deeper into that for sure)\n* Adam with CyclicLR + ReduceLROnPlateau schedulers + longer training\n* weather lag features + (few) time features\nThings that didn't work:\n* more features (not surprised)\n* \"is_missing\" features (kind of surprised)\n\nIn short - very happy with my findings. Deep NNs will definitely be part of my considerations in future tabular data competitions.\n \n##### LGBMs (of course :smile: ):\n* models per site, meter, half-half and on all data - trained on different datasets.\n* my best scoring model was per site (performing significantly better than the best public models (1.072 LB))\n\n##### L1, L2 regression models - just to have one more \"opinion\":\n* per site and all data\n* made sense for this more linear kind of a problem.\n \n### Ensemble\nThe second major thing that boosted my score.\nBased on my previous experience, stacking works better than blending when testing data is not cardinally different (and that can be argued of course).\n(NOT) very small disclaimer here:\n* this is not valid in all cases\n* well... how similar is that test data really...\n* how proper your base layer is - is it diverse enough? Are you overfitting there already? Do you introduce leak in the meta model training data if use diverse validation/oof prediction schemes for the base models?\n* there are also many other (important) variables - like what algorithm will be used for the meta learner, how it will be tuned, validated etc.\n* very risky approach in general (overfitting is BIG concern...).\n\nStill ... it worked well for me in the past.\nAnd worked this time as well.\nLet me give you some numbers:\n* my best stack: 1.245 (PL)\n* my best blend: 1.270 (PL)\n\n**What worked well here:** simple stuff (GLM, Gaussian, 5-fold CV, some regularization, adding meter to the training set). Performed very well for me also in the past (Elo Merchant Category Recommendation competition for example).\n**What didn't work that great:** more sophisticated stuff - Random Forest and even XGB, LGBM. All tend to overfit on my base data...\n\nThese are pretty much the points, I want to make.\nOne more time - please spend a minute to upvote the amazing work of the contributors below.\nAnd upvote this, in case you find it useful :smile:\nCheers folks and \"see\" you on the next competition.\n\nGreat kernels that deserve your vote:\nhttps://www.kaggle.com/rohanrao/ashrae-divide-and-conquer\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type\nhttps://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15\nhttps://www.kaggle.com/roydatascience/ashrae-energy-prediction-using-stratified-kfold\nhttps://www.kaggle.com/iwatatakuya/ashrae-kfold-lightgbm-without-building-id\nhttps://www.kaggle.com/kailex/ac-dc\nhttps://www.kaggle.com/rohanrao/ashrae-half-and-half\nhttps://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\nhttps://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\nhttps://www.kaggle.com/mimoudata/ashrae-lightgbm-without-leak\nhttps://www.kaggle.com/starl1ght/ashrae-stacked-regression-lasso-ridge-lgbm\nhttps://www.kaggle.com/tunguz/ashrae-histgradientboosting\nhttps://www.kaggle.com/yamsam/ashrae-leak-data-station", "votes": null }, { "id": "715028", "postDate": "01/10/2020 03:47:20", "content": "Congratulations\nGreat Write-Up!!\nThanks for sharing your Approach & Insights!! @gpamoukoff
", "rawMarkdown": "Congratulations\nGreat Write-Up!!\nThanks for sharing your Approach & Insights!! @gpamoukoff", "votes": null }, { "id": "717358", "postDate": "01/13/2020 03:51:56", "content": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "rawMarkdown": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "votes": null }, { "id": "717830", "postDate": "01/13/2020 17:08:10", "content": "Hey @claytonmiller yes I'll be happy to share the code with you - and even happier if you find some use of it (I would always support good cause like this). My entire solution is in Kaggle kernels - and it is also using output from 3-4 public kernels (like leak data station for example). Please let me know on how would you want to deliver the code (or you have access to it already?). Cheers
", "rawMarkdown": "Hey @claytonmiller yes I'll be happy to share the code with you - and even happier if you find some use of it (I would always support good cause like this). My entire solution is in Kaggle kernels - and it is also using output from 3-4 public kernels (like leak data station for example). Please let me know on how would you want to deliver the code (or you have access to it already?). Cheers", "votes": null }, { "id": "718217", "postDate": "01/14/2020 07:27:49", "content": "Hi Georgi - If your Kaggle Kernels are public, then go ahead and just insert them into your discussion writeup above and that should be good enough for us! We will let you know we need anything else.
\n\nThanks again!
", "rawMarkdown": "Hi Georgi - If your Kaggle Kernels are public, then go ahead and just insert them into your discussion writeup above and that should be good enough for us! We will let you know we need anything else. \n\nThanks again!", "votes": null }, { "id": "731616", "postDate": "01/28/2020 20:21:56", "content": "Hey @claytonmiller please find some of my key points below:
\n\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-adam-rop-sch\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-radam-lag\nhttps://www.kaggle.com/gpamoukoff/ashrae-lgbm-per-site\nhttps://www.kaggle.com/gpamoukoff/ashrae-subm-stack-fin-3
\n\nThanks and BR,
", "rawMarkdown": "Hey @claytonmiller please find some of my key points below:\n\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-adam-rop-sch\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-radam-lag\nhttps://www.kaggle.com/gpamoukoff/ashrae-lgbm-per-site\nhttps://www.kaggle.com/gpamoukoff/ashrae-subm-stack-fin-3\n\nThanks and BR,", "votes": null }, { "id": "732091", "postDate": "01/29/2020 13:40:32", "content": "Thank you sooooo much!
", "rawMarkdown": "Thank you sooooo much!", "votes": null } ], "comments": [ { "id": 715028, "author_name": "veeralakrishna", "author_url": "", "post_date": "01/10/2020 03:47:20", "content": "Congratulations\nGreat Write-Up!!\nThanks for sharing your Approach & Insights!! @gpamoukoff
", "votes": null, "replies": [] }, { "id": 717358, "author_name": "claytonmiller", "author_url": "", "post_date": "01/13/2020 03:51:56", "content": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!
\n\nAlso, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.
", "votes": null, "replies": [ { "id": 717830, "author_name": "gpamoukoff", "author_url": "", "post_date": "01/13/2020 17:08:10", "content": "Hey @claytonmiller yes I'll be happy to share the code with you - and even happier if you find some use of it (I would always support good cause like this). My entire solution is in Kaggle kernels - and it is also using output from 3-4 public kernels (like leak data station for example). Please let me know on how would you want to deliver the code (or you have access to it already?). Cheers
", "votes": null, "replies": [] }, { "id": 718217, "author_name": "claytonmiller", "author_url": "", "post_date": "01/14/2020 07:27:49", "content": "Hi Georgi - If your Kaggle Kernels are public, then go ahead and just insert them into your discussion writeup above and that should be good enough for us! We will let you know we need anything else.
\n\nThanks again!
", "votes": null, "replies": [] }, { "id": 731616, "author_name": "gpamoukoff", "author_url": "", "post_date": "01/28/2020 20:21:56", "content": "Hey @claytonmiller please find some of my key points below:
\n\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-adam-rop-sch\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-radam-lag\nhttps://www.kaggle.com/gpamoukoff/ashrae-lgbm-per-site\nhttps://www.kaggle.com/gpamoukoff/ashrae-subm-stack-fin-3
\n\nThanks and BR,
", "votes": null, "replies": [] }, { "id": 732091, "author_name": "claytonmiller", "author_url": "", "post_date": "01/29/2020 13:40:32", "content": "Thank you sooooo much!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "714856": "Hey Dear Kagglers :)\n\nFirst of all, congratulations to all winners and participants!\nAnd many thanks to all contributors for their amazing kernels and topics!\n\nWanted to put my 2 cents in - so you can find a brief overview of my solution below.\n\n## Motivation\nStarted work on the competition quite late - just two weeks before the end and my main motivation was to experiment around the use of deep learning with tabular data (and for this kind of problems).\n\n## Architecture\nLike probably most of the participants I ended up with 2 layer learning architecture: base layer and ensemble.\n\n### Base layer:\n#### Feature engineering\nBecause of my late start, I didn't really have the time for something fancy (FE is usually my main focus. Not this time though :smile: ).\nI ended up with 4 different datasets. On all, I applied cleansing (filtered out the bad rows) as per this great kernel (please upvote: https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks).\nUsed interpolation to fill in the missing values (on all), added \"is_missing\" features (to one of the sets).\nIn addition generated some:\n* weather-based features (aggregates/grouping, lag/rolling (moving averages/max/min etc))\n* time-based features (including holidays for one of the sets)\n* target encodings (in one of the sets).\n\nI draw inspiration from many great kernels in the process - check the list below and please go and upvote the amazing work of the authors - they deserve it!\n\n#### Models\nMy main focus was here and my goal was to have as diverse models as possible.\nI did NOT touche the leak data in any of the models of the base layer.\nTrained a total of 19 models. Amongst those:\n##### Deep NNs \nAs I mentioned my goal was mostly to play around these - so I put significant effort here. My best NNs didn't disappoint - they were approaching the performance of the best public LGBMs (1.09 LB) - but of course, had a very different \"point of view\" on the problem. That made them extremely useful in the final ensemble, and to that, I mostly attribute my good final score. Some things that worked great here:\n* entity embeddings for the categorical variables\n* Radam optimizer (that came as a bit of surprise - Radam didn't appear to make a huge difference for me on CV competitions - but here it showed significant advantage. Maybe the size of the network, or the complexity of the problem ... will dig deeper into that for sure)\n* Adam with CyclicLR + ReduceLROnPlateau schedulers + longer training\n* weather lag features + (few) time features\nThings that didn't work:\n* more features (not surprised)\n* \"is_missing\" features (kind of surprised)\n\nIn short - very happy with my findings. Deep NNs will definitely be part of my considerations in future tabular data competitions.\n \n##### LGBMs (of course :smile: ):\n* models per site, meter, half-half and on all data - trained on different datasets.\n* my best scoring model was per site (performing significantly better than the best public models (1.072 LB))\n\n##### L1, L2 regression models - just to have one more \"opinion\":\n* per site and all data\n* made sense for this more linear kind of a problem.\n \n### Ensemble\nThe second major thing that boosted my score.\nBased on my previous experience, stacking works better than blending when testing data is not cardinally different (and that can be argued of course).\n(NOT) very small disclaimer here:\n* this is not valid in all cases\n* well... how similar is that test data really...\n* how proper your base layer is - is it diverse enough? Are you overfitting there already? Do you introduce leak in the meta model training data if use diverse validation/oof prediction schemes for the base models?\n* there are also many other (important) variables - like what algorithm will be used for the meta learner, how it will be tuned, validated etc.\n* very risky approach in general (overfitting is BIG concern...).\n\nStill ... it worked well for me in the past.\nAnd worked this time as well.\nLet me give you some numbers:\n* my best stack: 1.245 (PL)\n* my best blend: 1.270 (PL)\n\n**What worked well here:** simple stuff (GLM, Gaussian, 5-fold CV, some regularization, adding meter to the training set). Performed very well for me also in the past (Elo Merchant Category Recommendation competition for example).\n**What didn't work that great:** more sophisticated stuff - Random Forest and even XGB, LGBM. All tend to overfit on my base data...\n\nThese are pretty much the points, I want to make.\nOne more time - please spend a minute to upvote the amazing work of the contributors below.\nAnd upvote this, in case you find it useful :smile:\nCheers folks and \"see\" you on the next competition.\n\nGreat kernels that deserve your vote:\nhttps://www.kaggle.com/rohanrao/ashrae-divide-and-conquer\nhttps://www.kaggle.com/corochann/ashrae-training-lgbm-by-meter-type\nhttps://www.kaggle.com/isaienkov/keras-nn-with-embeddings-for-cat-features-1-15\nhttps://www.kaggle.com/roydatascience/ashrae-energy-prediction-using-stratified-kfold\nhttps://www.kaggle.com/iwatatakuya/ashrae-kfold-lightgbm-without-building-id\nhttps://www.kaggle.com/kailex/ac-dc\nhttps://www.kaggle.com/rohanrao/ashrae-half-and-half\nhttps://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\nhttps://www.kaggle.com/aitude/ashrae-kfold-lightgbm-without-leak-1-08\nhttps://www.kaggle.com/mimoudata/ashrae-lightgbm-without-leak\nhttps://www.kaggle.com/starl1ght/ashrae-stacked-regression-lasso-ridge-lgbm\nhttps://www.kaggle.com/tunguz/ashrae-histgradientboosting\nhttps://www.kaggle.com/yamsam/ashrae-leak-data-station", "715028": "Congratulations\nGreat Write-Up!!\nThanks for sharing your Approach & Insights!! @gpamoukoff", "717358": "Thank you very much for sharing! We would like to add your solution to our analysis of the top processes and models. Please let us know how you would like to be acknowledged. The default will be what you have on your profile. Thanks again!\n\n**Also, for everyone else, we are in the process of curating a directory of Notebooks for building performance engineers and analysts from the competition that will serve as a recipe book for beginners as well as experts who are curious to learn about certain topics. We highly encourage you all to share your solutions (or any part of your process) as a notebook for this purpose. We will acknowledge the authors of all notebooks in this process, so please annotate your notebook or profile with how you would like to be acknowledged. We will also include general solutions and process information that is not in notebooks as well. It would also be helpful if you could indicate whether you are an ASHRAE member.**", "717830": "Hey @claytonmiller yes I'll be happy to share the code with you - and even happier if you find some use of it (I would always support good cause like this). My entire solution is in Kaggle kernels - and it is also using output from 3-4 public kernels (like leak data station for example). Please let me know on how would you want to deliver the code (or you have access to it already?). Cheers", "718217": "Hi Georgi - If your Kaggle Kernels are public, then go ahead and just insert them into your discussion writeup above and that should be good enough for us! We will let you know we need anything else. \n\nThanks again!", "731616": "Hey @claytonmiller please find some of my key points below:\n\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-adam-rop-sch\nhttps://www.kaggle.com/gpamoukoff/ashrae-keras-nn-entity-embeddings-radam-lag\nhttps://www.kaggle.com/gpamoukoff/ashrae-lgbm-per-site\nhttps://www.kaggle.com/gpamoukoff/ashrae-subm-stack-fin-3\n\nThanks and BR,", "732091": "Thank you sooooo much!" }, "source": "existing_discussion", "complete": true, "competition_id": 9994, "competition_slug": "ashrae-energy-prediction", "team_id": 3803285, "solution_url": "/competitions/ashrae-energy-prediction/writeups/georgi-pamukov-25th-place-solution" }