Ensembling seems to be everywhere, and I feel like I have heard people use Gradient Boosting to ensemble models together. How does that work? (If that is a thing?) Right now I have just been building n models and then averaging their results together. Is there some reading that describes this method clearly and why it works? (if it is a method!)
Ensembling seems to be everywhere, and I feel like I have heard people use Gradient Boosting to ensemble models together. How does that work? (If that is a thing?) Right now I have just been building n models and then averaging their results together. Is there some reading that describes this method clearly and why it works? (if it is a method!)
How's the Bias-Variance trade-off?\nWell, Weak Learners (United we Stand) => Stronger (Ensemble)\nSee if this helps:\nhttps://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205
\n\nLet me know if you find anything else. Thanks.
", "rawMarkdown": "How's the Bias-Variance trade-off?\nWell, Weak Learners (United we Stand) => Stronger (Ensemble)\nSee if this helps:\nhttps://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205\n\nLet me know if you find anything else. Thanks.", "votes": null }, { "id": "752272", "postDate": "02/20/2020 20:37:45", "content": "Glad you've asked, but first lets talk about two very popular ensemble methods, and YES there are more.
\n\nThat's basically what you're doing, another popular example would be Random forests where you train a bunch of decision tree models and perform a majority vote to make the final prediction.
\n\nAnd that's what you're asking about buddy. Now let me explain two popular methods:
\n\nAdaBoost or Adaptive Boosting
\n\nLets keep decision trees as the basic model in mind, AdaBoost will learn by running decision trees on the full dataset sequentially where the weights of the ith model miss classified data points will be increased for the (i + 1)th model. This means it will be less likely for it to pick that wrong path to this miss classified data point again.
\n\nGradient Boosting
\n\nAnd here we're with the one you're looking for. In contrast to AdaBoost, instead of updating the weights, the model learns from its predecessor's mistakes through looking directly at the residuals(Actual - Prediction). Not saying that's exactly how it is done but to give you an intuition lets say we've the following predictions [1, 2, 3, 4] where model A predicted [1, 2, 4, 3] with the residuals being [0, 0, -1, 1]. Model B comes to predict [1, 2, 4 ,3] but with the residuals added making it [1, 2, 3, 4].
\n\nI hope I gave you the basic idea to motivate you to research forward!
\n\nThere are tons of great articles and videos you can reach via Google ;)
\n\nIf you've found this useful, please don't forget to up vote. Thank you :))
", "rawMarkdown": "Glad you've asked, but first lets talk about two very popular ensemble methods, and YES there are more.\n\n- **Bagging**: Training a bunch of models in parallel on different subsets of data.\n\nThat's basically what you're doing, another popular example would be Random forests where you train a bunch of decision tree models and perform a majority vote to make the final prediction.\n\n- **Boosting**: Training a bunch of models sequentially so that they learn from each other.\n\nAnd that's what you're asking about buddy. Now let me explain two popular methods:\n\n**AdaBoost or Adaptive Boosting**\n\nLets keep decision trees as the basic model in mind, AdaBoost will learn by running decision trees on the full dataset sequentially where the weights of the ith model miss classified data points will be increased for the (i + 1)th model. This means it will be less likely for it to pick that wrong path to this miss classified data point again.\n\n**Gradient Boosting**\n\nAnd here we're with the one you're looking for. In contrast to AdaBoost, instead of updating the weights, the model learns from its predecessor's mistakes through looking directly at the residuals(Actual - Prediction). Not saying that's exactly how it is done but to give you an intuition lets say we've the following predictions [1, 2, 3, 4] where model A predicted [1, 2, 4, 3] with the residuals being [0, 0, -1, 1]. Model B comes to predict [1, 2, 4 ,3] but with the residuals added making it [1, 2, 3, 4].\n\nI hope I gave you the basic idea to motivate you to research forward! \n\nThere are tons of great articles and videos you can reach via Google ;) \n\nIf you've found this useful, please don't forget to up vote. Thank you :))", "votes": null }, { "id": "752283", "postDate": "02/20/2020 20:45:10", "content": "Thank you for such a clear explanation! That helps immensely! So if I were to try to adapt AdaBoost to a fastai NN. Would I then do the following setup (for a two model setup):
\n\nIs this a realistic understanding of that process?
", "rawMarkdown": "Thank you for such a clear explanation! That helps immensely! So if I were to try to adapt AdaBoost to a fastai NN. Would I then do the following setup (for a two model setup):\n\n1. Train my base model\n2. For training and validation data calculate and include my residual\n3. Using the validation residual average, run said trained model on the test set (which now includes this residual)\n\nIs this a realistic understanding of that process?", "votes": null }, { "id": "752285", "postDate": "02/20/2020 20:45:33", "content": "Thank you very much! This helped in making things clearer to me.
", "rawMarkdown": "Thank you very much! This helped in making things clearer to me.", "votes": null }, { "id": "752915", "postDate": "02/21/2020 14:41:47", "content": "Didn't quite get what you mean in step 3 but if your intention is to adapt AdaBoost to NNs then I suggest you give this paper a read. It's a great reference on the topic!
", "rawMarkdown": "Didn't quite get what you mean in step 3 but if your intention is to adapt AdaBoost to NNs then I suggest you give this [paper](http://www.iro.umontreal.ca/~lisa/pointeurs/ada-nc.pdf) a read. It's a great reference on the topic!", "votes": null } ], "comments": [ { "id": 752243, "author_name": "binaicrai", "author_url": "", "post_date": "02/20/2020 19:59:46", "content": "How's the Bias-Variance trade-off?\nWell, Weak Learners (United we Stand) => Stronger (Ensemble)\nSee if this helps:\nhttps://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205
\n\nLet me know if you find anything else. Thanks.
", "votes": null, "replies": [ { "id": 752285, "author_name": "muellerzr", "author_url": "", "post_date": "02/20/2020 20:45:33", "content": "Thank you very much! This helped in making things clearer to me.
", "votes": null, "replies": [] } ] }, { "id": 752272, "author_name": "msheriey", "author_url": "", "post_date": "02/20/2020 20:37:45", "content": "Glad you've asked, but first lets talk about two very popular ensemble methods, and YES there are more.
\n\nThat's basically what you're doing, another popular example would be Random forests where you train a bunch of decision tree models and perform a majority vote to make the final prediction.
\n\nAnd that's what you're asking about buddy. Now let me explain two popular methods:
\n\nAdaBoost or Adaptive Boosting
\n\nLets keep decision trees as the basic model in mind, AdaBoost will learn by running decision trees on the full dataset sequentially where the weights of the ith model miss classified data points will be increased for the (i + 1)th model. This means it will be less likely for it to pick that wrong path to this miss classified data point again.
\n\nGradient Boosting
\n\nAnd here we're with the one you're looking for. In contrast to AdaBoost, instead of updating the weights, the model learns from its predecessor's mistakes through looking directly at the residuals(Actual - Prediction). Not saying that's exactly how it is done but to give you an intuition lets say we've the following predictions [1, 2, 3, 4] where model A predicted [1, 2, 4, 3] with the residuals being [0, 0, -1, 1]. Model B comes to predict [1, 2, 4 ,3] but with the residuals added making it [1, 2, 3, 4].
\n\nI hope I gave you the basic idea to motivate you to research forward!
\n\nThere are tons of great articles and videos you can reach via Google ;)
\n\nIf you've found this useful, please don't forget to up vote. Thank you :))
", "votes": null, "replies": [ { "id": 752283, "author_name": "muellerzr", "author_url": "", "post_date": "02/20/2020 20:45:10", "content": "Thank you for such a clear explanation! That helps immensely! So if I were to try to adapt AdaBoost to a fastai NN. Would I then do the following setup (for a two model setup):
\n\nIs this a realistic understanding of that process?
", "votes": null, "replies": [] }, { "id": 752915, "author_name": "msheriey", "author_url": "", "post_date": "02/21/2020 14:41:47", "content": "Didn't quite get what you mean in step 3 but if your intention is to adapt AdaBoost to NNs then I suggest you give this paper a read. It's a great reference on the topic!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "752151": "Ensembling seems to be everywhere, and I feel like I have heard people use Gradient Boosting to ensemble models together. How does that work? (If that is a thing?) Right now I have just been building `n` models and then averaging their results together. Is there some reading that describes this method clearly and why it works? (if it is a method!)", "752243": "How's the Bias-Variance trade-off?\nWell, Weak Learners (United we Stand) => Stronger (Ensemble)\nSee if this helps:\nhttps://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205\n\nLet me know if you find anything else. Thanks.", "752272": "Glad you've asked, but first lets talk about two very popular ensemble methods, and YES there are more.\n\n- **Bagging**: Training a bunch of models in parallel on different subsets of data.\n\nThat's basically what you're doing, another popular example would be Random forests where you train a bunch of decision tree models and perform a majority vote to make the final prediction.\n\n- **Boosting**: Training a bunch of models sequentially so that they learn from each other.\n\nAnd that's what you're asking about buddy. Now let me explain two popular methods:\n\n**AdaBoost or Adaptive Boosting**\n\nLets keep decision trees as the basic model in mind, AdaBoost will learn by running decision trees on the full dataset sequentially where the weights of the ith model miss classified data points will be increased for the (i + 1)th model. This means it will be less likely for it to pick that wrong path to this miss classified data point again.\n\n**Gradient Boosting**\n\nAnd here we're with the one you're looking for. In contrast to AdaBoost, instead of updating the weights, the model learns from its predecessor's mistakes through looking directly at the residuals(Actual - Prediction). Not saying that's exactly how it is done but to give you an intuition lets say we've the following predictions [1, 2, 3, 4] where model A predicted [1, 2, 4, 3] with the residuals being [0, 0, -1, 1]. Model B comes to predict [1, 2, 4 ,3] but with the residuals added making it [1, 2, 3, 4].\n\nI hope I gave you the basic idea to motivate you to research forward! \n\nThere are tons of great articles and videos you can reach via Google ;) \n\nIf you've found this useful, please don't forget to up vote. Thank you :))", "752283": "Thank you for such a clear explanation! That helps immensely! So if I were to try to adapt AdaBoost to a fastai NN. Would I then do the following setup (for a two model setup):\n\n1. Train my base model\n2. For training and validation data calculate and include my residual\n3. Using the validation residual average, run said trained model on the test set (which now includes this residual)\n\nIs this a realistic understanding of that process?", "752285": "Thank you very much! This helped in making things clearer to me.", "752915": "Didn't quite get what you mean in step 3 but if your intention is to adapt AdaBoost to NNs then I suggest you give this [paper](http://www.iro.umontreal.ca/~lisa/pointeurs/ada-nc.pdf) a read. It's a great reference on the topic!" }, "source": "meta" }