Hey guys, that was a fun competition. Thanks to all organizers. Congrats to the winners and other top 10 competitors!!!
\n\nMy solution is very simple and I didn't spent so much time in this comp.\nI have basically only 4 base models trained on different target transformations and same 10 folds: XGB, Keras, SVR and libFM.
\n\n1)- XGB: I trained some XGBs adding some 2-way features interactios and on different target transformations like: t^1, t^(1/2), t^(1/4), t^(1/8), log(t), log(t+100), log(t+200), log(t+400), 10/t. CV varies from 1123.2 to 1135. I used maxdepth no more than 10. Even a maxdepth of 7 can build good models. I changed a little the other parameters according the transformation used.
\n\n2)- Keras NN: Same as XGB I trained on different target transformations and bagged 15 times each fold: t^1, t^(1/3), t^(1/5), t^(1/9), log(t+50), log(t+150), log(t+300), log(t+500), 100/t. CV varies from 1128 to 1145. Number of neurons varies from 32 to 512 per layer.
\n\n3)- Only 1 SVR model adapted from a Kernel to run on my folds. It took 3 days to run on all folds in parallel. Scored about 1159 CV. C=1
\n\n4)- Trained two libFM models. First one using raw data. Second using all features as categorical. Scored CV around 1190 and 1185.
\n\nTo blend all models I used a simple scipy minimize function with Nelder-Mead solver. I used meta features from 1st level prediction as input and searched to the best weights that directly minimize the MAE function. It worked very well with CV and public LB, but overfited a little Private LB. My CV using that aproach was around 1116.2.\nAlso I used another trick to improve a little more. I optimized the models weights for different predictions ranges, for example. I run minimize for predictions range [0..2000] then other for [2000..]. \nBlending preditions of global optimizer with by range optimizer improved CV to 1115.80 and that is my final model.
\n\nI quit the competition basically 3 days before the end to have some vacations and I still on it. On my last competition day I found that using R gbm with laplace loss builds a second level model that scores around CV 1117.3. I'm sure blending that GBM model and my optimizer solution could improve a little more the results. Maybe a second level NNet can improve results also.
\n\nWhat I found is that feature interactions, bagging results and correct hyperparameter tunning is the best way to improve models for that noise dataset and MAE metric.
\n\nSee you at Kaggle,
\n\nGiba
", "comments": [ { "id": 150669, "author_name": "", "author_url": "", "post_date": "2016-12-16T09:16:16.373000", "content": "Gilberto Thanks for sharing the solution.
", "votes": 1, "replies": [] }, { "id": 150650, "author_name": "Gippy", "author_url": "", "post_date": "2016-12-16T06:44:22.110000", "content": "@Gilberto: Thanks for sharing the solution. Very elegant solution.
\n\nCan you please once explain what is \"laplace loss builds\"? Thanks
", "votes": 1, "replies": [] }, { "id": 150645, "author_name": "Quazi Marufur Rahman", "author_url": "", "post_date": "2016-12-16T06:00:47.363000", "content": "Any example of how to use Nelder-Mead solver?
", "votes": 1, "replies": [] }, { "id": 150569, "author_name": "Andrey Gavrilenko", "author_url": "", "post_date": "2016-12-15T20:11:35.610000", "content": "It looks really simple and fast. Superb job has done! Thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 150551, "author_name": "Aleksandr Meshkov", "author_url": "", "post_date": "2016-12-15T18:22:36.310000", "content": "Brilliant!
", "votes": 2, "replies": [] }, { "id": 150590, "author_name": "Achal", "author_url": "", "post_date": "2016-12-15T22:05:49.957000", "content": "Thanks for sharing the solution.
\n\nI have a question regarding 2-way interactions and feature engineering part. How do you decide which features to use for generating 2-way interactions.
\n\nAnd if I am not wrong, you built (9 x xgb models + 9 x NN + 1 SVR + 2 LibFM = 21 models) in level 1 and for level 2, you only used Nelder-Mead solver which is quite interesting.
", "votes": 0, "replies": [] }, { "id": 152290, "author_name": "AK", "author_url": "", "post_date": "2016-12-24T20:05:34.600000", "content": "Can someone please explain me what is two way interaction?
", "votes": 0, "replies": [ { "id": 152293, "author_name": "Tilii", "author_url": "", "post_date": "2016-12-24T20:39:16.853000", "content": "[quote=AmanKapoor;152290]
\n\nCan someone please explain me what is two way interaction?
\n\n[/quote]
\n\nThis may help.
", "votes": 2, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 150547, "postDate": "2016-12-15T17:49:36.043Z", "content": "Hey guys, that was a fun competition. Thanks to all organizers. Congrats to the winners and other top 10 competitors!!!
\n\nMy solution is very simple and I didn't spent so much time in this comp.\nI have basically only 4 base models trained on different target transformations and same 10 folds: XGB, Keras, SVR and libFM.
\n\n1)- XGB: I trained some XGBs adding some 2-way features interactios and on different target transformations like: t^1, t^(1/2), t^(1/4), t^(1/8), log(t), log(t+100), log(t+200), log(t+400), 10/t. CV varies from 1123.2 to 1135. I used maxdepth no more than 10. Even a maxdepth of 7 can build good models. I changed a little the other parameters according the transformation used.
\n\n2)- Keras NN: Same as XGB I trained on different target transformations and bagged 15 times each fold: t^1, t^(1/3), t^(1/5), t^(1/9), log(t+50), log(t+150), log(t+300), log(t+500), 100/t. CV varies from 1128 to 1145. Number of neurons varies from 32 to 512 per layer.
\n\n3)- Only 1 SVR model adapted from a Kernel to run on my folds. It took 3 days to run on all folds in parallel. Scored about 1159 CV. C=1
\n\n4)- Trained two libFM models. First one using raw data. Second using all features as categorical. Scored CV around 1190 and 1185.
\n\nTo blend all models I used a simple scipy minimize function with Nelder-Mead solver. I used meta features from 1st level prediction as input and searched to the best weights that directly minimize the MAE function. It worked very well with CV and public LB, but overfited a little Private LB. My CV using that aproach was around 1116.2.\nAlso I used another trick to improve a little more. I optimized the models weights for different predictions ranges, for example. I run minimize for predictions range [0..2000] then other for [2000..]. \nBlending preditions of global optimizer with by range optimizer improved CV to 1115.80 and that is my final model.
\n\nI quit the competition basically 3 days before the end to have some vacations and I still on it. On my last competition day I found that using R gbm with laplace loss builds a second level model that scores around CV 1117.3. I'm sure blending that GBM model and my optimizer solution could improve a little more the results. Maybe a second level NNet can improve results also.
\n\nWhat I found is that feature interactions, bagging results and correct hyperparameter tunning is the best way to improve models for that noise dataset and MAE metric.
\n\nSee you at Kaggle,
\n\nGiba
", "rawMarkdown": "#7 Place Solution\r\n\r\nHey guys, that was a fun competition. Thanks to all organizers. Congrats to the winners and other top 10 competitors!!!\r\n\r\nMy solution is very simple and I didn't spent so much time in this comp.\r\nI have basically only 4 base models trained on different target transformations and same 10 folds: XGB, Keras, SVR and libFM.\r\n\r\n1)- XGB: I trained some XGBs adding some 2-way features interactios and on different target transformations like: t^1, t^(1/2), t^(1/4), t^(1/8), log(t), log(t+100), log(t+200), log(t+400), 10/t. CV varies from 1123.2 to 1135. I used maxdepth no more than 10. Even a maxdepth of 7 can build good models. I changed a little the other parameters according the transformation used.\r\n\r\n2)- Keras NN: Same as XGB I trained on different target transformations and bagged 15 times each fold: t^1, t^(1/3), t^(1/5), t^(1/9), log(t+50), log(t+150), log(t+300), log(t+500), 100/t. CV varies from 1128 to 1145. Number of neurons varies from 32 to 512 per layer.\r\n\r\n3)- Only 1 SVR model adapted from a Kernel to run on my folds. It took 3 days to run on all folds in parallel. Scored about 1159 CV. C=1\r\n\r\n4)- Trained two libFM models. First one using raw data. Second using all features as categorical. Scored CV around 1190 and 1185. \r\n\r\n\r\nTo blend all models I used a simple scipy minimize function with Nelder-Mead solver. I used meta features from 1st level prediction as input and searched to the best weights that directly minimize the MAE function. It worked very well with CV and public LB, but overfited a little Private LB. My CV using that aproach was around 1116.2.\r\nAlso I used another trick to improve a little more. I optimized the models weights for different predictions ranges, for example. I run minimize for predictions range [0..2000] then other for [2000..]. \r\nBlending preditions of global optimizer with by range optimizer improved CV to 1115.80 and that is my final model.\r\n\r\nI quit the competition basically 3 days before the end to have some vacations and I still on it. On my last competition day I found that using R gbm with laplace loss builds a second level model that scores around CV 1117.3. I'm sure blending that GBM model and my optimizer solution could improve a little more the results. Maybe a second level NNet can improve results also.\r\n\r\nWhat I found is that feature interactions, bagging results and correct hyperparameter tunning is the best way to improve models for that noise dataset and MAE metric.\r\n\r\nSee you at Kaggle,\r\n\r\nGiba", "votes": 45 }, { "id": 150669, "postDate": "2016-12-16T09:16:16.373Z", "content": "Gilberto Thanks for sharing the solution.
", "rawMarkdown": "Gilberto Thanks for sharing the solution. ", "votes": 1 }, { "id": 150650, "postDate": "2016-12-16T06:44:22.110Z", "content": "@Gilberto: Thanks for sharing the solution. Very elegant solution.
\n\nCan you please once explain what is \"laplace loss builds\"? Thanks
", "rawMarkdown": "@Gilberto: Thanks for sharing the solution. Very elegant solution.\r\n\r\nCan you please once explain what is \"laplace loss builds\"? Thanks", "votes": 1 }, { "id": 150645, "postDate": "2016-12-16T06:00:47.363Z", "content": "Any example of how to use Nelder-Mead solver?
", "rawMarkdown": "Any example of how to use Nelder-Mead solver?", "votes": 1 }, { "id": 150569, "postDate": "2016-12-15T20:11:35.610Z", "content": "It looks really simple and fast. Superb job has done! Thanks for sharing!
", "rawMarkdown": "It looks really simple and fast. Superb job has done! Thanks for sharing!", "votes": 1 }, { "id": 150551, "postDate": "2016-12-15T18:22:36.310Z", "content": "Brilliant!
", "rawMarkdown": "Brilliant!", "votes": 2 }, { "id": 150590, "postDate": "2016-12-15T22:05:49.957Z", "content": "Thanks for sharing the solution.
\n\nI have a question regarding 2-way interactions and feature engineering part. How do you decide which features to use for generating 2-way interactions.
\n\nAnd if I am not wrong, you built (9 x xgb models + 9 x NN + 1 SVR + 2 LibFM = 21 models) in level 1 and for level 2, you only used Nelder-Mead solver which is quite interesting.
", "rawMarkdown": "Thanks for sharing the solution.\r\n\r\nI have a question regarding 2-way interactions and feature engineering part. How do you decide which features to use for generating 2-way interactions.\r\n\r\nAnd if I am not wrong, you built (9 x xgb models + 9 x NN + 1 SVR + 2 LibFM = 21 models) in level 1 and for level 2, you only used Nelder-Mead solver which is quite interesting. " }, { "id": 152290, "postDate": "2016-12-24T20:05:34.600Z", "content": "Can someone please explain me what is two way interaction?
", "rawMarkdown": "Can someone please explain me what is two way interaction?", "replies": [ { "id": 152293, "postDate": "2016-12-24T20:39:16.853Z", "content": "[quote=AmanKapoor;152290]
\n\nCan someone please explain me what is two way interaction?
\n\n[/quote]
\n\nThis may help.
", "rawMarkdown": "[quote=AmanKapoor;152290]\r\n\r\nCan someone please explain me what is two way interaction?\r\n\r\n[/quote]\r\n\r\n[__This__](https://www.kaggle.com/c/allstate-claims-severity/forums/t/26527/could-anyone-kindly-explain-how-to-find-those-important-two-way-interaction) may help.", "votes": 2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:15:13Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 386170, "solution_url": "/competitions/allstate-claims-severity/writeups/on-vacation-giba-7-place-solution" }