Hey everyone. New to kaggle and ML, enjoying my first competition here (despite the leaks). I've had some success using leak validation for blending (Thanks to @yamsam for this useful kernel : https://www.kaggle.com/yamsam/ashrae-leak-validation-and-more ) but when I tried using it for single model validation my model (LGBM) score suffered. I realize that the leak data is not fully representative because it is only a few sites, but my naive hope was that it might do better than time based CV splits where some of the information about seasonal patterns is hidden from each fold.
\n\nWhat I'm wondering is:
\n\n1) Is the mismatch between the leak validation score and lb score because it somehow stops training at a point where it has only learnt well for some sites/buildings but not others and will this differ based on what kind of model (Tree based, Neural Net, etc) is used?
\n\n2) Or is it that the time split CV approach did better only because that was multiple models and it was already harnessing the advantages of ensembling somehow?
\n\nAnyone here tried/had any success using the leaks as a validation set? Curious to hear what your thoughts and experience regarding this are :)
", "messages": [ { "id": 684658, "postDate": "2019-11-30T07:26:01.177Z", "content": "Hey everyone. New to kaggle and ML, enjoying my first competition here (despite the leaks). I've had some success using leak validation for blending (Thanks to @yamsam for this useful kernel : https://www.kaggle.com/yamsam/ashrae-leak-validation-and-more ) but when I tried using it for single model validation my model (LGBM) score suffered. I realize that the leak data is not fully representative because it is only a few sites, but my naive hope was that it might do better than time based CV splits where some of the information about seasonal patterns is hidden from each fold.
\n\nWhat I'm wondering is:
\n\n1) Is the mismatch between the leak validation score and lb score because it somehow stops training at a point where it has only learnt well for some sites/buildings but not others and will this differ based on what kind of model (Tree based, Neural Net, etc) is used?
\n\n2) Or is it that the time split CV approach did better only because that was multiple models and it was already harnessing the advantages of ensembling somehow?
\n\nAnyone here tried/had any success using the leaks as a validation set? Curious to hear what your thoughts and experience regarding this are :)
", "rawMarkdown": "Hey everyone. New to kaggle and ML, enjoying my first competition here (despite the leaks). I've had some success using leak validation for blending (Thanks to @yamsam for this useful kernel : https://www.kaggle.com/yamsam/ashrae-leak-validation-and-more ) but when I tried using it for single model validation my model (LGBM) score suffered. I realize that the leak data is not fully representative because it is only a few sites, but my naive hope was that it might do better than time based CV splits where some of the information about seasonal patterns is hidden from each fold.\n\nWhat I'm wondering is:\n\n1) Is the mismatch between the leak validation score and lb score because it somehow stops training at a point where it has only learnt well for some sites/buildings but not others and will this differ based on what kind of model (Tree based, Neural Net, etc) is used?\n\n2) Or is it that the time split CV approach did better only because that was multiple models and it was already harnessing the advantages of ensembling somehow?\n\nAnyone here tried/had any success using the leaks as a validation set? Curious to hear what your thoughts and experience regarding this are :) ", "votes": 1 } ], "comments": [], "raw_markdown_by_id": { "684658": "Hey everyone. New to kaggle and ML, enjoying my first competition here (despite the leaks). I've had some success using leak validation for blending (Thanks to @yamsam for this useful kernel : https://www.kaggle.com/yamsam/ashrae-leak-validation-and-more ) but when I tried using it for single model validation my model (LGBM) score suffered. I realize that the leak data is not fully representative because it is only a few sites, but my naive hope was that it might do better than time based CV splits where some of the information about seasonal patterns is hidden from each fold.\n\nWhat I'm wondering is:\n\n1) Is the mismatch between the leak validation score and lb score because it somehow stops training at a point where it has only learnt well for some sites/buildings but not others and will this differ based on what kind of model (Tree based, Neural Net, etc) is used?\n\n2) Or is it that the time split CV approach did better only because that was multiple models and it was already harnessing the advantages of ensembling somehow?\n\nAnyone here tried/had any success using the leaks as a validation set? Curious to hear what your thoughts and experience regarding this are :) " } }