This may be a naive question but here it is anyway. I never used level 2 (or higher) ensembling. It is clearly next step for me to close some of the gap with top Kagglers.
\nMy understanding of L2 learning is that you train models on data (L1 learning), then construct new data set with predictions of L1 models and some (if not all) the original data. My question is this:
isn't there a risk of overfitting to the L1 predictions? Shouldn't one keep some of the original data out of L1 and keep it for L2 learning? Alternatively, shouldn't one use out of fold L1 predictions?
", "messages": [ { "id": "144604", "postDate": "11/14/2016 19:43:36", "content": "This may be a naive question but here it is anyway. I never used level 2 (or higher) ensembling. It is clearly next step for me to close some of the gap with top Kagglers.
\nMy understanding of L2 learning is that you train models on data (L1 learning), then construct new data set with predictions of L1 models and some (if not all) the original data. My question is this:
isn't there a risk of overfitting to the L1 predictions? Shouldn't one keep some of the original data out of L1 and keep it for L2 learning? Alternatively, shouldn't one use out of fold L1 predictions?
", "rawMarkdown": "This may be a naive question but here it is anyway. I never used level 2 (or higher) ensembling. It is clearly next step for me to close some of the gap with top Kagglers. \r\nMy understanding of L2 learning is that you train models on data (L1 learning), then construct new data set with predictions of L1 models and some (if not all) the original data. My question is this:\r\n\r\nisn't there a risk of overfitting to the L1 predictions? Shouldn't one keep some of the original data out of L1 and keep it for L2 learning? Alternatively, shouldn't one use out of fold L1 predictions?", "votes": null }, { "id": "144784", "postDate": "11/15/2016 11:29:19", "content": "Well, I guess I will have to learn it the hard way given the amount of responses. Fair enough.
", "rawMarkdown": "Well, I guess I will have to learn it the hard way given the amount of responses. Fair enough.", "votes": null }, { "id": "144786", "postDate": "11/15/2016 11:54:14", "content": "Using fix CV folds and OOF predictions overfitting usually is not an issue.
\n\nI found these posts useful:
\n\nhttp://kaggler.com/winning-data-science-competitions-datascience-la/
\n\nhttp://mlwave.com/kaggle-ensembling-guide/
", "rawMarkdown": "Using fix CV folds and OOF predictions overfitting usually is not an issue.\r\n\r\nI found these posts useful:\r\n\r\nhttp://kaggler.com/winning-data-science-competitions-datascience-la/\r\n\r\nhttp://mlwave.com/kaggle-ensembling-guide/", "votes": null }, { "id": "144789", "postDate": "11/15/2016 12:07:34", "content": "Thanks again Beluga, will study these.
", "rawMarkdown": "Thanks again Beluga, will study these.", "votes": null }, { "id": "144833", "postDate": "11/15/2016 16:35:05", "content": "Darragh pointed me to his post and the enclosing thread: https://www.kaggle.com/c/predicting-red-hat-business-value/forums/t/23617/how-to-build-an-ensemble/135350#post135350
\n\nWorth reading too.
\n\nBeware, I'm going to stack like hell in my next competition ;)
", "rawMarkdown": "Darragh pointed me to his post and the enclosing thread: https://www.kaggle.com/c/predicting-red-hat-business-value/forums/t/23617/how-to-build-an-ensemble/135350#post135350\r\n\r\nWorth reading too.\r\n\r\nBeware, I'm going to stack like hell in my next competition ;)", "votes": null }, { "id": "228888", "postDate": "10/08/2017 07:10:07", "content": "may be this helps https://youtu.be/9IyJ4HvubGo
", "rawMarkdown": "may be this helps https://youtu.be/9IyJ4HvubGo", "votes": null } ], "comments": [ { "id": 144784, "author_name": "cpmpml", "author_url": "", "post_date": "11/15/2016 11:29:19", "content": "Well, I guess I will have to learn it the hard way given the amount of responses. Fair enough.
", "votes": null, "replies": [] }, { "id": 144786, "author_name": "gaborfodor", "author_url": "", "post_date": "11/15/2016 11:54:14", "content": "Using fix CV folds and OOF predictions overfitting usually is not an issue.
\n\nI found these posts useful:
\n\nhttp://kaggler.com/winning-data-science-competitions-datascience-la/
\n\nhttp://mlwave.com/kaggle-ensembling-guide/
", "votes": null, "replies": [] }, { "id": 144789, "author_name": "cpmpml", "author_url": "", "post_date": "11/15/2016 12:07:34", "content": "Thanks again Beluga, will study these.
", "votes": null, "replies": [] }, { "id": 144833, "author_name": "cpmpml", "author_url": "", "post_date": "11/15/2016 16:35:05", "content": "Darragh pointed me to his post and the enclosing thread: https://www.kaggle.com/c/predicting-red-hat-business-value/forums/t/23617/how-to-build-an-ensemble/135350#post135350
\n\nWorth reading too.
\n\nBeware, I'm going to stack like hell in my next competition ;)
", "votes": null, "replies": [] }, { "id": 228888, "author_name": "burgje", "author_url": "", "post_date": "10/08/2017 07:10:07", "content": "may be this helps https://youtu.be/9IyJ4HvubGo
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "144604": "This may be a naive question but here it is anyway. I never used level 2 (or higher) ensembling. It is clearly next step for me to close some of the gap with top Kagglers. \r\nMy understanding of L2 learning is that you train models on data (L1 learning), then construct new data set with predictions of L1 models and some (if not all) the original data. My question is this:\r\n\r\nisn't there a risk of overfitting to the L1 predictions? Shouldn't one keep some of the original data out of L1 and keep it for L2 learning? Alternatively, shouldn't one use out of fold L1 predictions?", "144784": "Well, I guess I will have to learn it the hard way given the amount of responses. Fair enough.", "144786": "Using fix CV folds and OOF predictions overfitting usually is not an issue.\r\n\r\nI found these posts useful:\r\n\r\nhttp://kaggler.com/winning-data-science-competitions-datascience-la/\r\n\r\nhttp://mlwave.com/kaggle-ensembling-guide/", "144789": "Thanks again Beluga, will study these.", "144833": "Darragh pointed me to his post and the enclosing thread: https://www.kaggle.com/c/predicting-red-hat-business-value/forums/t/23617/how-to-build-an-ensemble/135350#post135350\r\n\r\nWorth reading too.\r\n\r\nBeware, I'm going to stack like hell in my next competition ;)", "228888": "may be this helps https://youtu.be/9IyJ4HvubGo" }, "source": "meta" }