Please share your story if you blended just other's model results. How this affected on you standing on the LB.
\n\nThank you very much for attention.
", "messages": [ { "id": "610735", "postDate": "08/29/2019 00:43:30", "content": "Please share your story if you blended just other's model results. How this affected on you standing on the LB.
\n\nThank you very much for attention.
", "rawMarkdown": "Please share your story if you blended just other's model results. How this affected on you standing on the LB. \n\nThank you very much for attention.", "votes": null }, { "id": "610750", "postDate": "08/29/2019 00:54:58", "content": "Blending is useful when you have a consistent CV strategy and OOFs at hand.
", "rawMarkdown": "Blending is useful when you have a consistent CV strategy and OOFs at hand.", "votes": null }, { "id": "610758", "postDate": "08/29/2019 00:58:01", "content": "sure, there are many techniques, and for example the theorem central limit about estimating real mu (mean) is a good argument if you are doing average blending :)
", "rawMarkdown": "sure, there are many techniques, and for example the *theorem central limit* about estimating real mu (mean) is a good argument if you are doing average blending :)", "votes": null }, { "id": "610771", "postDate": "08/29/2019 01:04:10", "content": "Using 4 kernels with LB scores -1.83,-2.02,-2.05,-1.87, I was able to reach -2.20. -1.83 was from the public kernel. Others were private kernels. Weighted average was used. It works !
", "rawMarkdown": "Using 4 kernels with LB scores -1.83,-2.02,-2.05,-1.87, I was able to reach -2.20. -1.83 was from the public kernel. Others were private kernels. Weighted average was used. It works !", "votes": null }, { "id": "610964", "postDate": "08/29/2019 03:32:33", "content": "There's a mathematical formula for blending regression models. If one model has mae=x and one model has mae=y where x < y then if 0.5*(1-x/y) percentage of prediction errors change sign (i.e. model one has positive error on a certain perdiction while model two has negative error) then an ensemble with weights y/(x+y) and x/(x+y) will increase mae.
I blended my several of my own weaker models and got a .2 bump
", "rawMarkdown": "I blended my several of my own weaker models and got a .2 bump", "votes": null }, { "id": "611603", "postDate": "08/29/2019 10:45:15", "content": "Do you have a reference for this? I'm curious to learn more about the theoretical justification for blending. I keep hearing people reference the CLT, but standard forms of the CLT (like you'd find in a first probability course) don't seem applicable (but maybe I'm missing something obvious). This sounds like a more promising justification.
", "rawMarkdown": "Do you have a reference for this? I'm curious to learn more about the theoretical justification for blending. I keep hearing people reference the CLT, but standard forms of the CLT (like you'd find in a first probability course) don't seem applicable (but maybe I'm missing something obvious). This sounds like a more promising justification.", "votes": null }, { "id": "612811", "postDate": "08/30/2019 02:44:23", "content": "When I gain access to a computer next week, I'll post the equations. The general idea is as follows.
\n\nWhen a model makes a prediction, it is either above true value (positive error) or below true value (negative error). If two models with mae=x and mae=y have postive error on every prediction then their ensemble has mae=(x+y)/2 and ensemble does not help.
But if one model predicts a coupling with +1 error and another predicts with -1 error, then when you average the predictions together, the ensemble has zero error on that prediction.
", "rawMarkdown": "When I gain access to a computer next week, I'll post the equations. The general idea is as follows.\n\nWhen a model makes a prediction, it is either above true value (positive error) or below true value (negative error). If two models with `mae=x` and `mae=y` have postive error on every prediction then their ensemble has `mae=(x+y)/2` and ensemble does not help.\n\nBut if one model predicts a coupling with +1 error and another predicts with -1 error, then when you average the predictions together, the ensemble has zero error on that prediction.", "votes": null }, { "id": "613310", "postDate": "08/30/2019 12:12:38", "content": "\n\n\nweights 1/x and 1/y will increase mae.
\n
It must be a bit different, as the weights should not be modified if I scale everything by a constant factor.
", "rawMarkdown": "> weights 1/x and 1/y will increase mae.\n\nIt must be a bit different, as the weights should not be modified if I scale everything by a constant factor.", "votes": null }, { "id": "613326", "postDate": "08/30/2019 12:20:56", "content": "@cdeotte Interesting. I'd love to look at the equations when you get some time. Thanks!
", "rawMarkdown": "cdeotte Interesting. I'd love to look at the equations when you get some time. Thanks!", "votes": null }, { "id": "613335", "postDate": "08/30/2019 12:29:22", "content": "The equation for the sum of two random variates is
\n\n$$Var(X1+X2)=Var(X1)+Var(X2)+2Cov(X1,X2)$$
\n\nFrom this you can see that there's the biggest improvement to the error of the mean when two methods are completely uncorrelated. Specifically for the case variates following normal distributions, you can have a look at this wikipedia page.
", "rawMarkdown": "The equation for the sum of two random variates is\n\n$$Var(X1+X2)=Var(X1)+Var(X2)+2Cov(X1,X2)$$\n\nFrom this you can see that there's the biggest improvement to the error of the mean when two methods are completely uncorrelated. Specifically for the case variates following normal distributions, you can have a look at [this wikipedia page.](https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables)", "votes": null }, { "id": "613346", "postDate": "08/30/2019 12:41:47", "content": "Lars, your analysis applies to mean squared error, not mae ;)
", "rawMarkdown": "Lars, your analysis applies to mean squared error, not mae ;)", "votes": null }, { "id": "613373", "postDate": "08/30/2019 13:04:56", "content": "Only if you assume normal distributed variates (which the latter link refers to). For Laplacian distributions, which usually resembles test errors better than normal distributions, you would get something more closely related to the MAEs. I highlighted Normal distributions since they tend to be a bit easier to understand and work with. @cdeotte might be thinking about some other sets of equations though.
", "rawMarkdown": "Only if you assume normal distributed variates (which the latter link refers to). For Laplacian distributions, which usually resembles test errors better than normal distributions, you would get something more closely related to the MAEs. I highlighted Normal distributions since they tend to be a bit easier to understand and work with. @cdeotte might be thinking about some other sets of equations though.", "votes": null }, { "id": "613382", "postDate": "08/30/2019 13:08:41", "content": "Laers, given you are active here I'm asking it here: why did you never answer Cédric Bény question about the coordinate system of shielding data?
", "rawMarkdown": "Laers, given you are active here I'm asking it here: why did you never answer Cédric Bény question about the coordinate system of shielding data?", "votes": null }, { "id": "613561", "postDate": "08/30/2019 16:12:11", "content": "@cpmpml Thanks for the correction. I meant multiply model one by 1/x and multiply model two by 1/y then divide by 1/x + 1/y. So the weights are y/(x+y) and x/(x+y).
@cpmpml Thanks for the reminder. I have made a response over here.
", "rawMarkdown": "cpmpml Thanks for the reminder. I have made a response [over here.](https://www.kaggle.com/c/champs-scalar-coupling/discussion/99055613592)", "votes": null } ], "comments": [ { "id": 610750, "author_name": "scaomath", "author_url": "", "post_date": "08/29/2019 00:54:58", "content": "Blending is useful when you have a consistent CV strategy and OOFs at hand.
", "votes": null, "replies": [] }, { "id": 610758, "author_name": "jesucristo", "author_url": "", "post_date": "08/29/2019 00:58:01", "content": "sure, there are many techniques, and for example the theorem central limit about estimating real mu (mean) is a good argument if you are doing average blending :)
", "votes": null, "replies": [] }, { "id": 610771, "author_name": "silence2", "author_url": "", "post_date": "08/29/2019 01:04:10", "content": "Using 4 kernels with LB scores -1.83,-2.02,-2.05,-1.87, I was able to reach -2.20. -1.83 was from the public kernel. Others were private kernels. Weighted average was used. It works !
", "votes": null, "replies": [] }, { "id": 610964, "author_name": "cdeotte", "author_url": "", "post_date": "08/29/2019 03:32:33", "content": "There's a mathematical formula for blending regression models. If one model has mae=x and one model has mae=y where x < y then if 0.5*(1-x/y) percentage of prediction errors change sign (i.e. model one has positive error on a certain perdiction while model two has negative error) then an ensemble with weights y/(x+y) and x/(x+y) will increase mae.
Do you have a reference for this? I'm curious to learn more about the theoretical justification for blending. I keep hearing people reference the CLT, but standard forms of the CLT (like you'd find in a first probability course) don't seem applicable (but maybe I'm missing something obvious). This sounds like a more promising justification.
", "votes": null, "replies": [] }, { "id": 612811, "author_name": "cdeotte", "author_url": "", "post_date": "08/30/2019 02:44:23", "content": "When I gain access to a computer next week, I'll post the equations. The general idea is as follows.
\n\nWhen a model makes a prediction, it is either above true value (positive error) or below true value (negative error). If two models with mae=x and mae=y have postive error on every prediction then their ensemble has mae=(x+y)/2 and ensemble does not help.
But if one model predicts a coupling with +1 error and another predicts with -1 error, then when you average the predictions together, the ensemble has zero error on that prediction.
", "votes": null, "replies": [] }, { "id": 613310, "author_name": "cpmpml", "author_url": "", "post_date": "08/30/2019 12:12:38", "content": "\n\n\nweights 1/x and 1/y will increase mae.
\n
It must be a bit different, as the weights should not be modified if I scale everything by a constant factor.
", "votes": null, "replies": [] }, { "id": 613326, "author_name": "abendorf", "author_url": "", "post_date": "08/30/2019 12:20:56", "content": "@cdeotte Interesting. I'd love to look at the equations when you get some time. Thanks!
", "votes": null, "replies": [] }, { "id": 613335, "author_name": "larsbratholm", "author_url": "", "post_date": "08/30/2019 12:29:22", "content": "The equation for the sum of two random variates is
\n\n$$Var(X1+X2)=Var(X1)+Var(X2)+2Cov(X1,X2)$$
\n\nFrom this you can see that there's the biggest improvement to the error of the mean when two methods are completely uncorrelated. Specifically for the case variates following normal distributions, you can have a look at this wikipedia page.
", "votes": null, "replies": [] }, { "id": 613346, "author_name": "cpmpml", "author_url": "", "post_date": "08/30/2019 12:41:47", "content": "Lars, your analysis applies to mean squared error, not mae ;)
", "votes": null, "replies": [] }, { "id": 613373, "author_name": "larsbratholm", "author_url": "", "post_date": "08/30/2019 13:04:56", "content": "Only if you assume normal distributed variates (which the latter link refers to). For Laplacian distributions, which usually resembles test errors better than normal distributions, you would get something more closely related to the MAEs. I highlighted Normal distributions since they tend to be a bit easier to understand and work with. @cdeotte might be thinking about some other sets of equations though.
", "votes": null, "replies": [] }, { "id": 613382, "author_name": "cpmpml", "author_url": "", "post_date": "08/30/2019 13:08:41", "content": "Laers, given you are active here I'm asking it here: why did you never answer Cédric Bény question about the coordinate system of shielding data?
", "votes": null, "replies": [] }, { "id": 613561, "author_name": "cdeotte", "author_url": "", "post_date": "08/30/2019 16:12:11", "content": "@cpmpml Thanks for the correction. I meant multiply model one by 1/x and multiply model two by 1/y then divide by 1/x + 1/y. So the weights are y/(x+y) and x/(x+y).
@cpmpml Thanks for the reminder. I have made a response over here.
", "votes": null, "replies": [] } ] }, { "id": 611233, "author_name": "julianmukaj", "author_url": "", "post_date": "08/29/2019 07:16:05", "content": "I blended my several of my own weaker models and got a .2 bump
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "610735": "Please share your story if you blended just other's model results. How this affected on you standing on the LB. \n\nThank you very much for attention.", "610750": "Blending is useful when you have a consistent CV strategy and OOFs at hand.", "610758": "sure, there are many techniques, and for example the *theorem central limit* about estimating real mu (mean) is a good argument if you are doing average blending :)", "610771": "Using 4 kernels with LB scores -1.83,-2.02,-2.05,-1.87, I was able to reach -2.20. -1.83 was from the public kernel. Others were private kernels. Weighted average was used. It works !", "610964": "There's a mathematical formula for blending regression models. If one model has `mae=x` and one model has `mae=y` where `x < y` then if `0.5*(1-x/y)` percentage of prediction errors change sign (i.e. model one has positive error on a certain perdiction while model two has negative error) then an ensemble with weights `y/(x+y)` and `x/(x+y)` will increase mae.", "611233": "I blended my several of my own weaker models and got a .2 bump", "611603": "Do you have a reference for this? I'm curious to learn more about the theoretical justification for blending. I keep hearing people reference the CLT, but standard forms of the CLT (like you'd find in a first probability course) don't seem applicable (but maybe I'm missing something obvious). This sounds like a more promising justification.", "612811": "When I gain access to a computer next week, I'll post the equations. The general idea is as follows.\n\nWhen a model makes a prediction, it is either above true value (positive error) or below true value (negative error). If two models with `mae=x` and `mae=y` have postive error on every prediction then their ensemble has `mae=(x+y)/2` and ensemble does not help.\n\nBut if one model predicts a coupling with +1 error and another predicts with -1 error, then when you average the predictions together, the ensemble has zero error on that prediction.", "613310": "> weights 1/x and 1/y will increase mae.\n\nIt must be a bit different, as the weights should not be modified if I scale everything by a constant factor.", "613326": "cdeotte Interesting. I'd love to look at the equations when you get some time. Thanks!", "613335": "The equation for the sum of two random variates is\n\n$$Var(X1+X2)=Var(X1)+Var(X2)+2Cov(X1,X2)$$\n\nFrom this you can see that there's the biggest improvement to the error of the mean when two methods are completely uncorrelated. Specifically for the case variates following normal distributions, you can have a look at [this wikipedia page.](https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables)", "613346": "Lars, your analysis applies to mean squared error, not mae ;)", "613373": "Only if you assume normal distributed variates (which the latter link refers to). For Laplacian distributions, which usually resembles test errors better than normal distributions, you would get something more closely related to the MAEs. I highlighted Normal distributions since they tend to be a bit easier to understand and work with. @cdeotte might be thinking about some other sets of equations though.", "613382": "Laers, given you are active here I'm asking it here: why did you never answer Cédric Bény question about the coordinate system of shielding data?", "613561": "cpmpml Thanks for the correction. I meant multiply model one by `1/x` and multiply model two by `1/y` then divide by `1/x + 1/y`. So the weights are `y/(x+y)` and `x/(x+y)`.", "613596": "cpmpml Thanks for the reminder. I have made a response [over here.](https://www.kaggle.com/c/champs-scalar-coupling/discussion/99055613592)" }, "source": "meta" }