Hi guys and girls,
\n\nFirst off all, I’d like to congrats the winners and thank all the people who contributed in the forum, Tilii, Vladimir Iglovikov, MT, Faron … to name just a few.
\n\nI’m new to data science and this is my first competition, as much this is solution sharing it’s also about validating it. So feel free to challenge it, point out any unnecessary/wrong steps and how I could have improved it.
\n\nI used 4 different algorithms: XGB, LGBM, Keras, and sklearn GBR. All models are 10 fold cv, with two main transformations: power(loss+0, 0.26) and log(loss+200), but the first transformation was better.
\n\nXGB: I used fair objective with constant=1.5, Best model: CV=1124.79, LB=1105.25. (10 models)
\n\nKeras: same architecture that the one shared in the forum with default MAE. Best model: CV= 1130.59, LB=??. (7 models)
\n\nLGBM: best model: CV= 1128.08, LB= ??. (6 models)
\n\nSKlearn GBR: best model: CV=1136.37, LB=??. (2 models).
\n\nI had 25 first level models that I stacked using the same 4 algorithms. I used the two transformations above, but also with no transformation of loss for sklearn GBR, which significantly helped my score (I didn’t get the time to experiment with the other ones). I also added three extra features: the mean of all 25 models, cont14 and cont7 from training set.
\n\nBest CV scores are:\nXGB= 1117.37, bagged by changing the model’s seed.\nKeras= 1118.44, with 2 layers: 200(0.35), 80(0.1) and a constraint on the weights (maxnorm=0.5).\nSklearn GBR= 1116.65, bagged dozens of times by changing the model’s seed.
\n\nMy final score is an optimized weighted average of the first 25 models and 14 second level models. I used the scipy minimize package, with the BFGS method. The score on the oob predictions is 1115.66, LB=1099.48 and 1111.38 on private LB, which gave me my current rank.
\n\nThese are some of my questions:
\n\nIs adding the mean, and features from the training set common practice in stacking?
\n\nThe BFGS method gives negative weights, is this normal, how can we interpret this? In all cases, it outperformed the other method that just give positive weights (SLSQP), in both oob predictions, public and private LB.
\n\nMy weighted average includes first and second level models, is this common practice?
\n\nMy 2nd level stacked models performed worse than the optimized weighted average, is it because all models are similar in performance and that I should have incorporates some “worse” ones?
\n\nThanks to kaggle and to kagglers, and I to the pleasure of racing with you again in the LB :)
", "comments": [ { "id": 150929, "author_name": "Hakim Idjis", "author_url": "", "post_date": "2016-12-17T16:32:03.253000", "content": "@qmaruf to the best of my knowledge, oob (out of bag) and oof (out of fold) are exactly the same. To generate them, you can refer to this script, look at lines:136, 154, 162, 163. For the optimized weights, I am not sure that i understood the question. In any case, you can refer to this thread to see an example.
", "votes": 0, "replies": [] }, { "id": 150896, "author_name": "Quazi Marufur Rahman", "author_url": "", "post_date": "2016-12-17T12:01:42.407000", "content": "Hi, I have some questions.
\n\nHow are you generating the oob predictions? And what is the difference between oob and oof? What is the use case of optimized weights? Any example available?
", "votes": 0, "replies": [] }, { "id": 150015, "author_name": "Hakim Idjis", "author_url": "", "post_date": "2016-12-13T15:16:42.387000", "content": "Thanks Micheal and congrats to you also!\nYes they are correlated. I spent so much time trying to fine tune them, that they ended up looking very similar. I will bear that in mind for my next competition :)
", "votes": 0, "replies": [] }, { "id": 150013, "author_name": "Michael Maguire", "author_url": "", "post_date": "2016-12-13T14:58:28.007000", "content": "Congratulations Hakim! And thanks for posting your solution. Here's some answers to your questions (based on what I have noticed while competing here)..
\n\nYes, I think using engineered features (like the mean) built from the original feature set is common. I think it is also common to combine engineered features and stacked results into feature sets for additional models. The same with inclusion of level 1 and level 2 models in your weighted average. The \"overlap\" of models and features can give algorithms like XGBoost extra data to work with and can speed up and improve its predictive capability. Tree-based algorithms can generally account for linear combinations of features, but not necessarily non-linear combinations (like stacked results) so including them with the original features can provide algorithms more to work with.
\n\nYour 2nd level stacked models performing worse might be because of high correlation and similarity in results. Hard to tell without knowing exactly what you used. But, yes inclusion of \"worse\" ones can provide big improvements. I used \"worse\" models in my stacking and saw 2-3 point gains in CV by adding in models that had CV's in the 1200 range. So did some other solutions I've read so far.
", "votes": 0, "replies": [] }, { "id": 149980, "author_name": "Hakim Idjis", "author_url": "", "post_date": "2016-12-13T11:45:36.707000", "content": "You're welcome, mainly by changing the parameters (I had two set off different parameters) ,the transformation (log, power(0.26)), and the shift value (0,200, 500).
\n\nThe pearson correlation is between 0.989 and 0.997. Zach's post suggested that having negative weight takes care of that
\n\n[quote=Zach;149863]
\n\nAlso, I let my weights go negative. I think since all my models were so highly correlated, allowing negative weights allowed the optimizer to find a better solution.
\n\n[/quote]
", "votes": 0, "replies": [] }, { "id": 149977, "author_name": "Øystein Schønning-Johansen", "author_url": "", "post_date": "2016-12-13T11:23:12.043000", "content": "First: Thanks for sharing your solution!
\n\nHere is my question: How do you make sure your individual L1 models are not too much correlated to each other? You say 10 XGB models, how does these 10 models differ from each other? Can you state the L1 model correlation?
\n\n(I think I'm too afraid to mix/stack highly correlated models.)
\n\nBTW: I also shared your experience that (L)BFGS and SLSQP (and everything else I tried) gave negative weights in a linear model. I ended up manually tuning. I tuned against the public LB, and that seems to have been a big mistake...
", "votes": 0, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 149961, "postDate": "2016-12-13T10:19:44.927Z", "content": "Hi guys and girls,
\n\nFirst off all, I’d like to congrats the winners and thank all the people who contributed in the forum, Tilii, Vladimir Iglovikov, MT, Faron … to name just a few.
\n\nI’m new to data science and this is my first competition, as much this is solution sharing it’s also about validating it. So feel free to challenge it, point out any unnecessary/wrong steps and how I could have improved it.
\n\nI used 4 different algorithms: XGB, LGBM, Keras, and sklearn GBR. All models are 10 fold cv, with two main transformations: power(loss+0, 0.26) and log(loss+200), but the first transformation was better.
\n\nXGB: I used fair objective with constant=1.5, Best model: CV=1124.79, LB=1105.25. (10 models)
\n\nKeras: same architecture that the one shared in the forum with default MAE. Best model: CV= 1130.59, LB=??. (7 models)
\n\nLGBM: best model: CV= 1128.08, LB= ??. (6 models)
\n\nSKlearn GBR: best model: CV=1136.37, LB=??. (2 models).
\n\nI had 25 first level models that I stacked using the same 4 algorithms. I used the two transformations above, but also with no transformation of loss for sklearn GBR, which significantly helped my score (I didn’t get the time to experiment with the other ones). I also added three extra features: the mean of all 25 models, cont14 and cont7 from training set.
\n\nBest CV scores are:\nXGB= 1117.37, bagged by changing the model’s seed.\nKeras= 1118.44, with 2 layers: 200(0.35), 80(0.1) and a constraint on the weights (maxnorm=0.5).\nSklearn GBR= 1116.65, bagged dozens of times by changing the model’s seed.
\n\nMy final score is an optimized weighted average of the first 25 models and 14 second level models. I used the scipy minimize package, with the BFGS method. The score on the oob predictions is 1115.66, LB=1099.48 and 1111.38 on private LB, which gave me my current rank.
\n\nThese are some of my questions:
\n\nIs adding the mean, and features from the training set common practice in stacking?
\n\nThe BFGS method gives negative weights, is this normal, how can we interpret this? In all cases, it outperformed the other method that just give positive weights (SLSQP), in both oob predictions, public and private LB.
\n\nMy weighted average includes first and second level models, is this common practice?
\n\nMy 2nd level stacked models performed worse than the optimized weighted average, is it because all models are similar in performance and that I should have incorporates some “worse” ones?
\n\nThanks to kaggle and to kagglers, and I to the pleasure of racing with you again in the LB :)
", "rawMarkdown": "Hi guys and girls,\r\n\r\nFirst off all, I’d like to congrats the winners and thank all the people who contributed in the forum, Tilii, Vladimir Iglovikov, MT, Faron … to name just a few.\r\n\r\nI’m new to data science and this is my first competition, as much this is solution sharing it’s also about validating it. So feel free to challenge it, point out any unnecessary/wrong steps and how I could have improved it.\r\n\r\nI used 4 different algorithms: XGB, LGBM, Keras, and sklearn GBR. All models are 10 fold cv, with two main transformations: power(loss+0, 0.26) and log(loss+200), but the first transformation was better.\r\n\r\n**XGB**: I used fair objective with constant=1.5, Best model: CV=1124.79, LB=1105.25. (10 models)\r\n\r\n**Keras**: same architecture that the one shared in the forum with default MAE. Best model: CV= 1130.59, LB=??. (7 models)\r\n\r\n**LGBM**: best model: CV= 1128.08, LB= ??. (6 models)\r\n\r\n**SKlearn GBR**: best model: CV=1136.37, LB=??. (2 models).\r\n\r\nI had 25 first level models that I stacked using the same 4 algorithms. I used the two transformations above, but also with no transformation of loss for sklearn GBR, which significantly helped my score (I didn’t get the time to experiment with the other ones). I also added three extra features: the mean of all 25 models, cont14 and cont7 from training set.\r\n\r\n**Best CV scores are:**\r\nXGB= 1117.37, bagged by changing the model’s seed.\r\nKeras= 1118.44, with 2 layers: 200(0.35), 80(0.1) and a constraint on the weights (maxnorm=0.5).\r\nSklearn GBR= 1116.65, bagged dozens of times by changing the model’s seed.\r\n\r\nMy final score is an optimized weighted average of the first 25 models and 14 second level models. I used the scipy minimize package, with the BFGS method. The score on the oob predictions is **1115.66, LB=1099.48 and 1111.38** on private LB, which gave me my current rank.\r\n\r\n**These are some of my questions**:\r\n\r\nIs adding the mean, and features from the training set common practice in stacking?\r\n\r\nThe BFGS method gives negative weights, is this normal, how can we interpret this? In all cases, it outperformed the other method that just give positive weights (SLSQP), in both oob predictions, public and private LB.\r\n\r\nMy weighted average includes first and second level models, is this common practice?\r\n\r\nMy 2nd level stacked models performed worse than the optimized weighted average, is it because all models are similar in performance and that I should have incorporates some “worse” ones?\r\n\r\nThanks to kaggle and to kagglers, and I to the pleasure of racing with you again in the LB :)\r\n", "votes": 16 }, { "id": 150929, "postDate": "2016-12-17T16:32:03.253Z", "content": "@qmaruf to the best of my knowledge, oob (out of bag) and oof (out of fold) are exactly the same. To generate them, you can refer to this script, look at lines:136, 154, 162, 163. For the optimized weights, I am not sure that i understood the question. In any case, you can refer to this thread to see an example.
", "rawMarkdown": "@qmaruf to the best of my knowledge, oob (out of bag) and oof (out of fold) are exactly the same. To generate them, you can refer to this [scrip][1]t, look at lines:136, 154, 162, 163. For the optimized weights, I am not sure that i understood the question. In any case, you can refer to this [thread][2] to see an example.\r\n\r\n\r\n [1]: https://www.kaggle.com/mtinti/allstate-claims-severity/keras-starter-with-bagging-1111-84364\r\n [2]: https://www.kaggle.com/c/allstate-claims-severity/forums/t/25240/weights-in-an-ensemble" }, { "id": 150896, "postDate": "2016-12-17T12:01:42.407Z", "content": "Hi, I have some questions.
\n\nHow are you generating the oob predictions? And what is the difference between oob and oof? What is the use case of optimized weights? Any example available?
", "rawMarkdown": "Hi, I have some questions. \r\n\r\nHow are you generating the oob predictions? And what is the difference between oob and oof? What is the use case of optimized weights? Any example available?" }, { "id": 150015, "postDate": "2016-12-13T15:16:42.387Z", "content": "Thanks Micheal and congrats to you also!\nYes they are correlated. I spent so much time trying to fine tune them, that they ended up looking very similar. I will bear that in mind for my next competition :)
", "rawMarkdown": "Thanks Micheal and congrats to you also!\r\nYes they are correlated. I spent so much time trying to fine tune them, that they ended up looking very similar. I will bear that in mind for my next competition :)" }, { "id": 150013, "postDate": "2016-12-13T14:58:28.007Z", "content": "Congratulations Hakim! And thanks for posting your solution. Here's some answers to your questions (based on what I have noticed while competing here)..
\n\nYes, I think using engineered features (like the mean) built from the original feature set is common. I think it is also common to combine engineered features and stacked results into feature sets for additional models. The same with inclusion of level 1 and level 2 models in your weighted average. The \"overlap\" of models and features can give algorithms like XGBoost extra data to work with and can speed up and improve its predictive capability. Tree-based algorithms can generally account for linear combinations of features, but not necessarily non-linear combinations (like stacked results) so including them with the original features can provide algorithms more to work with.
\n\nYour 2nd level stacked models performing worse might be because of high correlation and similarity in results. Hard to tell without knowing exactly what you used. But, yes inclusion of \"worse\" ones can provide big improvements. I used \"worse\" models in my stacking and saw 2-3 point gains in CV by adding in models that had CV's in the 1200 range. So did some other solutions I've read so far.
", "rawMarkdown": "Congratulations Hakim! And thanks for posting your solution. Here's some answers to your questions (based on what I have noticed while competing here)..\r\n\r\nYes, I think using engineered features (like the mean) built from the original feature set is common. I think it is also common to combine engineered features and stacked results into feature sets for additional models. The same with inclusion of level 1 and level 2 models in your weighted average. The \"overlap\" of models and features can give algorithms like XGBoost extra data to work with and can speed up and improve its predictive capability. Tree-based algorithms can generally account for linear combinations of features, but not necessarily non-linear combinations (like stacked results) so including them with the original features can provide algorithms more to work with.\r\n\r\nYour 2nd level stacked models performing worse might be because of high correlation and similarity in results. Hard to tell without knowing exactly what you used. But, yes inclusion of \"worse\" ones can provide big improvements. I used \"worse\" models in my stacking and saw 2-3 point gains in CV by adding in models that had CV's in the 1200 range. So did some other solutions I've read so far. \r\n\r\n" }, { "id": 149980, "postDate": "2016-12-13T11:45:36.707Z", "content": "You're welcome, mainly by changing the parameters (I had two set off different parameters) ,the transformation (log, power(0.26)), and the shift value (0,200, 500).
\n\nThe pearson correlation is between 0.989 and 0.997. Zach's post suggested that having negative weight takes care of that
\n\n[quote=Zach;149863]
\n\nAlso, I let my weights go negative. I think since all my models were so highly correlated, allowing negative weights allowed the optimizer to find a better solution.
\n\n[/quote]
", "rawMarkdown": "You're welcome, mainly by changing the parameters (I had two set off different parameters) ,the transformation (log, power(0.26)), and the shift value (0,200, 500).\r\n\r\nThe pearson correlation is between 0.989 and 0.997. [Zach's post][1] suggested that having negative weight takes care of that\r\n\r\n[quote=Zach;149863]\r\n\r\nAlso, I let my weights go negative. I think since all my models were so highly correlated, allowing negative weights allowed the optimizer to find a better solution.\r\n\r\n[/quote]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/allstate-claims-severity/forums/t/26411/my-approach?forumMessageId=149863#post149863" }, { "id": 149977, "postDate": "2016-12-13T11:23:12.043Z", "content": "First: Thanks for sharing your solution!
\n\nHere is my question: How do you make sure your individual L1 models are not too much correlated to each other? You say 10 XGB models, how does these 10 models differ from each other? Can you state the L1 model correlation?
\n\n(I think I'm too afraid to mix/stack highly correlated models.)
\n\nBTW: I also shared your experience that (L)BFGS and SLSQP (and everything else I tried) gave negative weights in a linear model. I ended up manually tuning. I tuned against the public LB, and that seems to have been a big mistake...
", "rawMarkdown": "First: Thanks for sharing your solution!\r\n\r\nHere is my question: How do you make sure your individual L1 models are not too much correlated to each other? You say 10 XGB models, how does these 10 models differ from each other? Can you state the L1 model correlation?\r\n\r\n(I think I'm too afraid to mix/stack highly correlated models.)\r\n\r\nBTW: I also shared your experience that (L)BFGS **and** SLSQP (and everything else I tried) gave negative weights in a linear model. I ended up manually tuning. I tuned against the public LB, and that seems to have been a big mistake..." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:16:29Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 383427, "solution_url": "/competitions/allstate-claims-severity/writeups/hakim-idjis-16-place-solution-and-some-questions-a" }