Hi!
\n\nThanks for all participants and especially forum/script authors, it was a very fun competition and I learned A LOT!
\n\nI ended in 9th place in public LB, but moved to 2nd in private which was a pleasant surprise =)
\n\nHere is my solution description. As most of solutions, I suppose, it's based on building a lot of different models and stacking.
\n\nFirst-level models
\n\nI've trained a lot of diverse models on top of different data transformations, target transformations and different parameters.
\n\nThe main ones were XGB and Keras NN (all of them with 4-6 bags) which provide me with following best single model results:
\n\nBest XGB - 1122.64977 CV, 1105.01969 Public LB, 1116.74215 Private LB
\n\nBest NN - 1130.29286 CV, 1111.90884 Public LB, 1122.06610 Private LB
\n\nAlso, there was LightGBM (which is pretty fast, but seem to provide less tuning options then XGB, I've used pyLightGBM for Python integration), sklearn models (Random Forest, Extra Trees, Gradient Boosting, Linear regression, Support Vector Regression and KNN) and LibFM.
\n\nAs SVR and KNN took a lot of time to train, I've trained them on small data samples and then averaged results.
\n\nIn total there were about 70 L1 models which I used in my solution.
\n\nData transformations
\n\nI've used different categorical encodings in different models - lexical, dummy, bayes, the best ones being lexical for tree-based models and dummy for other ones.
\n\nI've tried applying SVD to numeric + dummy categoricals and it seem to help for non-tree models (for example, CV of libfm model with SVD improved from 1196 to 1177).
\n\nAlso, what helped for non-tree models is clustering data and then transforming it to cluster-distance space and applying RBF function. It was especially useful for linear regression (improved from 1237 to 1202), but also good for NN models.
\n\nIn my late XGB models I also used categorical combinations, the best way to select combinations was using excellent Xgbfi tool - just take dump of good XGB model without combinations, run Xgbfi and take some number of 2-way interactions.
\n\nParameter tuning
\n\nTo select parameter sets I've used BayesOptimization package for smaller models (like RF and ET), but XGB and NN were tuned manually - there were several good links in the forums which provided me with some intuition on how to tune XGB params (especially this one).
\n\nSecond level
\n\nIn second level I've mainly trained XGB and Keras NN models, with different params, but also included linear regression with different target transformations, random forests and gradient boosting from sklearn, because it can optimize MAE directly.
\n\nIn total I've trained around 50 L2 models, which were futher groupped and averaged (see below).
\n\nThird level
\n\nFor my third level I used quantile regression from statsmodels package. It doesn't have any regularization and seems to produce noisy results, so in last couple of days I tried to reduce model noise:
\n\nAlso, in the end I've selected around of ten my best submissions from different time and averaged them, what gave me additional 0.1 in public LB.
\n\nThats all. I'll post a link to repository with all the code in a day or two (though it may be a bit messy) (UPD: https://github.com/alno/kaggle-allstate-claims-severity).
\n\nThanks again for competing!
", "comments": [ { "id": 150113, "author_name": "Alexey Noskov", "author_url": "", "post_date": "2016-12-13T22:27:54.687000", "content": "Here is my code: https://github.com/alno/kaggle-allstate-claims-severity
\n\nScripts starting with prepare transform dataset, all training code is in train.py, all others are util modules and small helper scripts.
Yes, I've selected top interactions and added them as new categorical variables, encoding with function from this script
", "votes": 1, "replies": [] }, { "id": 149984, "author_name": "Ilya Baryshnikov", "author_url": "", "post_date": "2016-12-13T12:06:35.947000", "content": "Cheers! Your win seems unexpected and beautiful :)
\n\nIt's too late, but Microsoft released native python wrapper for LightGBM recently. It's API very similar to XGBoost, lib works pretty fast and have more features, e.g. custom objective...
\n\nhttps://github.com/Microsoft/LightGBM/tree/master/python-package
", "votes": 1, "replies": [] }, { "id": 152573, "author_name": "Alexey Noskov", "author_url": "", "post_date": "2016-12-27T08:25:37.137000", "content": "You are right, with n_splits I may try different seeds in KFold, and n_bags specify number of different model seeds inside each fold (but train set sampling may be enabled for some models, it's done in third level, for example).
\n\nIt works for XGB, NN and LightGB, as their models may vary with the seed, but there is no need to use it with RF/ET as they consist from number of independent trees by itself, so we may just increase number of trees if needed. For LR it also not very useful as it doesn't depend on seed very much.
\n\nIn the third level I use n_bags with training set sampling, to get more robust model.
\n\nAs for early stopping - I wasn't sure how to combine different number rounds from different folds for full-set training.
", "votes": 2, "replies": [] }, { "id": 150035, "author_name": "dmi3kno", "author_url": "", "post_date": "2016-12-13T16:26:13.793000", "content": "xgbfi gracefully brought to us by @Far0n compiles significance of feature interactions using xgb model dump. You can read about it online. Just search xgbfi and you will find excellent examples of use (including on this forum) and walk-through guides like this one.
\n\nAnd here's Python implementation
", "votes": 2, "replies": [] }, { "id": 152694, "author_name": "Chaofan Han", "author_url": "", "post_date": "2016-12-27T23:07:53.333000", "content": "Your comments are really useful!! Thanks.
", "votes": 0, "replies": [] }, { "id": 152526, "author_name": "Chaofan Han", "author_url": "", "post_date": "2016-12-26T23:53:46.863000", "content": "Hi Alexey Noskov,
\n\nI am wondering what the meanings of n_splits, n_bags in your code. I understand it like this. For the param n_splits, you wanna try different fold splitting in KFold function. For n_bags, you wanna try different seeds for a particular model which is not what the true bagging means as the model is still trained on the full training data set. I guess this kind of new bagging works for Xgb, nn, LightGB and may not work well for lr, et, rf which I observed in your code.
\n\nAnother quick question is that why you didn't use early_stopping_rounds in training your xgb models. Are there any tricks behind that?
", "votes": 0, "replies": [] }, { "id": 152181, "author_name": "mezoganet", "author_url": "", "post_date": "2016-12-23T21:21:26.350000", "content": "@Alexey Noskov
\n\n\"I applied boxcox after log\"...
\n\nOk, that's what I did to but it may miss something later on...
\n\nFor me, it was'nt enough. Even with a \"log\", \"BoxCox\", \"Center\", \"Scale\"...
\n\nDid I miss something ?
\n\nI join everybody here and congrat you for your performance, really, honnestly !
\n\nBe proud Alexey
\n\nPS : \"Naive Bayes\" ? hum...
", "votes": 0, "replies": [] }, { "id": 152109, "author_name": "Chaofan Han", "author_url": "", "post_date": "2016-12-23T12:00:50.850000", "content": "Thanks for your help. I really appreciate your efforts. You deserve it!!!
", "votes": 0, "replies": [] }, { "id": 152068, "author_name": "Alexey Noskov", "author_url": "", "post_date": "2016-12-23T07:59:43.127000", "content": "Hi!
\n\nI applied boxcox after log, and as I remember I've added 1 before log to avoid negative values, but for now I don't see much sense in it.
Where was hypothesis in forums what some columns represent some time values, so I tried to add their difference as it may provide value to the model. It didn't help.
When looking on feature value distributions it may be seen what some features have significant amount of values equal to exactly 0 or 1, so I created special features for such cases. It also didn't help.
There is a lot of models and features in code which I tried and they didn't help, really.
", "votes": 0, "replies": [] }, { "id": 152060, "author_name": "Chaofan Han", "author_url": "", "post_date": "2016-12-23T07:10:11.063000", "content": "Thanks a lot for sharing! I really learn a lot from you. I have several questions to ask.
\n\n[quote=Alexey Noskov;149950]
\n\nIn my late XGB models I also used categorical combinations, the best way to select combinations was using excellent [Xgbfi][3] tool - just take dump of good XGB model without combinations, run Xgbfi and take some number of 2-way interactions.
\n\n[/quote]
\n\nThanks a lot for sharing!\nWhat is your approach to use interactions after xgbfi? Just take top-interactions from \"Interaction Depth X\" sheets and consider it as a new categorical vars?
", "votes": 0, "replies": [] }, { "id": 150161, "author_name": "", "author_url": "", "post_date": "2016-12-14T04:04:15.957000", "content": "Congrats Alexey, and thanks for sharing the code. I have a question, is there any hiararchy relationship between the level one and level two?
", "votes": 0, "replies": [ { "id": 150210, "author_name": "Alexey Noskov", "author_url": "", "post_date": "2016-12-14T08:00:29.007000", "content": "[quote=Weiwei Liu;150161]
\n\nCongrats Alexey, and thanks for sharing the code. I have a question, is there any hiararchy relationship between the level one and level two?
\n\n[/quote]
\n\nLevel 2 models use predictions of Level 1 models as a features (and Level 3 use predictions of Level 2), that's all hierarchy.
", "votes": 0, "replies": [] } ] }, { "id": 150032, "author_name": "Nirupam Kar", "author_url": "", "post_date": "2016-12-13T16:18:34.657000", "content": "could you please explain how you did use xgbfi?
", "votes": 0, "replies": [] }, { "id": 149981, "author_name": "", "author_url": "", "post_date": "2016-12-13T11:45:46.843000", "content": "Congrats Alexey . Thanks for sharing solution
", "votes": 0, "replies": [] }, { "id": 149973, "author_name": "Bishwarup B", "author_url": "", "post_date": "2016-12-13T10:47:42.463000", "content": "Congrats Alexey. Nice third level model :)
", "votes": 0, "replies": [] }, { "id": 149956, "author_name": "bit_hammer", "author_url": "", "post_date": "2016-12-13T10:10:17.837000", "content": "Wow, very cool ~ I have learned a lot from your experience and look forward to your code.
", "votes": 0, "replies": [] }, { "id": 149955, "author_name": "", "author_url": "", "post_date": "2016-12-13T10:10:16.180000", "content": "", "votes": 0, "replies": [] }, { "id": 149996, "author_name": "MukeshKumar", "author_url": "", "post_date": "2016-12-13T13:42:03.980000", "content": "Thanks for the information !!
", "votes": 0, "replies": [] } ], "comment_count": 20, "messages": [ { "id": 149950, "postDate": "2016-12-13T09:52:04.163Z", "content": "Hi!
\n\nThanks for all participants and especially forum/script authors, it was a very fun competition and I learned A LOT!
\n\nI ended in 9th place in public LB, but moved to 2nd in private which was a pleasant surprise =)
\n\nHere is my solution description. As most of solutions, I suppose, it's based on building a lot of different models and stacking.
\n\nFirst-level models
\n\nI've trained a lot of diverse models on top of different data transformations, target transformations and different parameters.
\n\nThe main ones were XGB and Keras NN (all of them with 4-6 bags) which provide me with following best single model results:
\n\nBest XGB - 1122.64977 CV, 1105.01969 Public LB, 1116.74215 Private LB
\n\nBest NN - 1130.29286 CV, 1111.90884 Public LB, 1122.06610 Private LB
\n\nAlso, there was LightGBM (which is pretty fast, but seem to provide less tuning options then XGB, I've used pyLightGBM for Python integration), sklearn models (Random Forest, Extra Trees, Gradient Boosting, Linear regression, Support Vector Regression and KNN) and LibFM.
\n\nAs SVR and KNN took a lot of time to train, I've trained them on small data samples and then averaged results.
\n\nIn total there were about 70 L1 models which I used in my solution.
\n\nData transformations
\n\nI've used different categorical encodings in different models - lexical, dummy, bayes, the best ones being lexical for tree-based models and dummy for other ones.
\n\nI've tried applying SVD to numeric + dummy categoricals and it seem to help for non-tree models (for example, CV of libfm model with SVD improved from 1196 to 1177).
\n\nAlso, what helped for non-tree models is clustering data and then transforming it to cluster-distance space and applying RBF function. It was especially useful for linear regression (improved from 1237 to 1202), but also good for NN models.
\n\nIn my late XGB models I also used categorical combinations, the best way to select combinations was using excellent Xgbfi tool - just take dump of good XGB model without combinations, run Xgbfi and take some number of 2-way interactions.
\n\nParameter tuning
\n\nTo select parameter sets I've used BayesOptimization package for smaller models (like RF and ET), but XGB and NN were tuned manually - there were several good links in the forums which provided me with some intuition on how to tune XGB params (especially this one).
\n\nSecond level
\n\nIn second level I've mainly trained XGB and Keras NN models, with different params, but also included linear regression with different target transformations, random forests and gradient boosting from sklearn, because it can optimize MAE directly.
\n\nIn total I've trained around 50 L2 models, which were futher groupped and averaged (see below).
\n\nThird level
\n\nFor my third level I used quantile regression from statsmodels package. It doesn't have any regularization and seems to produce noisy results, so in last couple of days I tried to reduce model noise:
\n\nAlso, in the end I've selected around of ten my best submissions from different time and averaged them, what gave me additional 0.1 in public LB.
\n\nThats all. I'll post a link to repository with all the code in a day or two (though it may be a bit messy) (UPD: https://github.com/alno/kaggle-allstate-claims-severity).
\n\nThanks again for competing!
", "rawMarkdown": "Hi! \r\n\r\nThanks for all participants and especially forum/script authors, it was a very fun competition and I learned A LOT!\r\n\r\nI ended in 9th place in public LB, but moved to 2nd in private which was a pleasant surprise =)\r\n\r\nHere is my solution description. As most of solutions, I suppose, it's based on building a lot of different models and stacking.\r\n\r\n**First-level models**\r\n\r\nI've trained a lot of diverse models on top of different data transformations, target transformations and different parameters.\r\n\r\nThe main ones were XGB and Keras NN (all of them with 4-6 bags) which provide me with following best single model results:\r\n\r\nBest XGB - 1122.64977 CV, 1105.01969 Public LB, 1116.74215 Private LB\r\n\r\nBest NN - 1130.29286 CV, 1111.90884 Public LB, 1122.06610 Private LB\r\n\r\nAlso, there was [LightGBM][1] (which is pretty fast, but seem to provide less tuning options then XGB, I've used [pyLightGBM][2] for Python integration), sklearn models (Random Forest, Extra Trees, Gradient Boosting, Linear regression, Support Vector Regression and KNN) and LibFM.\r\n\r\nAs SVR and KNN took a lot of time to train, I've trained them on small data samples and then averaged results.\r\n\r\nIn total there were about 70 L1 models which I used in my solution.\r\n\r\n**Data transformations**\r\n\r\nI've used different categorical encodings in different models - lexical, dummy, bayes, the best ones being lexical for tree-based models and dummy for other ones.\r\n\r\nI've tried applying SVD to numeric + dummy categoricals and it seem to help for non-tree models (for example, CV of libfm model with SVD improved from 1196 to 1177).\r\n\r\nAlso, what helped for non-tree models is clustering data and then transforming it to cluster-distance space and applying RBF function. It was especially useful for linear regression (improved from 1237 to 1202), but also good for NN models.\r\n\r\nIn my late XGB models I also used categorical combinations, the best way to select combinations was using excellent [Xgbfi][3] tool - just take dump of good XGB model without combinations, run Xgbfi and take some number of 2-way interactions.\r\n\r\n**Parameter tuning**\r\n\r\nTo select parameter sets I've used [BayesOptimization][4] package for smaller models (like RF and ET), but XGB and NN were tuned manually - there were several good links in the forums which provided me with some intuition on how to tune XGB params (especially [this][5] one). \r\n\r\n**Second level**\r\n\r\nIn second level I've mainly trained XGB and Keras NN models, with different params, but also included linear regression with different target transformations, random forests and [gradient boosting from sklearn][6], because it can optimize MAE directly.\r\n\r\nIn total I've trained around 50 L2 models, which were futher groupped and averaged (see below).\r\n\r\n**Third level**\r\n\r\nFor my third level I used [quantile regression from statsmodels package][7]. It doesn't have any regularization and seems to produce noisy results, so in last couple of days I tried to reduce model noise:\r\n\r\n* I've trained L3 model with 8 bags in each fold and used fold-average predictions for submission.\r\n* I've groupped L2 models by their similariity and averaged predictions in each group, so producing 10 features for L3 level. Also I've applied power correction to some of groups, like X^1.03.\r\n\r\n\r\nAlso, in the end I've selected around of ten my best submissions from different time and averaged them, what gave me additional 0.1 in public LB.\r\n\r\n\r\nThats all. I'll post a link to repository with all the code in a day or two (though it may be a bit messy) (UPD: https://github.com/alno/kaggle-allstate-claims-severity).\r\n\r\nThanks again for competing!\r\n\r\n\r\n [1]: https://github.com/Microsoft/LightGBM/\r\n [2]: https://github.com/ArdalanM/pyLightGBM\r\n [3]: https://github.com/Far0n/xgbfi\r\n [4]: https://github.com/fmfn/BayesianOptimization/\r\n [5]: https://www.kaggle.com/c/santander-customer-satisfaction/forums/t/20662/overtuning-hyper-parameters-especially-re-xgboost?forumMessageId=118487\r\n [6]: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html\r\n [7]: http://statsmodels.sourceforge.net/devel/examples/notebooks/generated/quantile_regression.html", "votes": 65 }, { "id": 150113, "postDate": "2016-12-13T22:27:54.687Z", "content": "Here is my code: https://github.com/alno/kaggle-allstate-claims-severity
\n\nScripts starting with prepare transform dataset, all training code is in train.py, all others are util modules and small helper scripts.
Yes, I've selected top interactions and added them as new categorical variables, encoding with function from this script
", "rawMarkdown": "Yes, I've selected top interactions and added them as new categorical variables, encoding with function from [this script][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/modkzs/allstate-claims-severity/lexical-encoding-feature-comb", "votes": 1 }, { "id": 149984, "postDate": "2016-12-13T12:06:35.947Z", "content": "Cheers! Your win seems unexpected and beautiful :)
\n\nIt's too late, but Microsoft released native python wrapper for LightGBM recently. It's API very similar to XGBoost, lib works pretty fast and have more features, e.g. custom objective...
\n\nhttps://github.com/Microsoft/LightGBM/tree/master/python-package
", "rawMarkdown": "Cheers! Your win seems unexpected and beautiful :)\r\n\r\nIt's too late, but Microsoft released native python wrapper for LightGBM recently. It's API very similar to XGBoost, lib works pretty fast and have more features, e.g. custom objective...\r\n\r\nhttps://github.com/Microsoft/LightGBM/tree/master/python-package\r\n", "votes": 1 }, { "id": 152573, "postDate": "2016-12-27T08:25:37.137Z", "content": "You are right, with n_splits I may try different seeds in KFold, and n_bags specify number of different model seeds inside each fold (but train set sampling may be enabled for some models, it's done in third level, for example).
\n\nIt works for XGB, NN and LightGB, as their models may vary with the seed, but there is no need to use it with RF/ET as they consist from number of independent trees by itself, so we may just increase number of trees if needed. For LR it also not very useful as it doesn't depend on seed very much.
\n\nIn the third level I use n_bags with training set sampling, to get more robust model.
\n\nAs for early stopping - I wasn't sure how to combine different number rounds from different folds for full-set training.
", "rawMarkdown": "You are right, with n_splits I may try different seeds in KFold, and n_bags specify number of different model seeds inside each fold (but train set sampling may be enabled for some models, it's done in third level, for example). \r\n\r\nIt works for XGB, NN and LightGB, as their models may vary with the seed, but there is no need to use it with RF/ET as they consist from number of independent trees by itself, so we may just increase number of trees if needed. For LR it also not very useful as it doesn't depend on seed very much.\r\n\r\nIn the third level I use n_bags with training set sampling, to get more robust model.\r\n\r\nAs for early stopping - I wasn't sure how to combine different number rounds from different folds for full-set training.", "votes": 2 }, { "id": 150035, "postDate": "2016-12-13T16:26:13.793Z", "content": "xgbfi gracefully brought to us by @Far0n compiles significance of feature interactions using xgb model dump. You can read about it online. Just search xgbfi and you will find excellent examples of use (including on this forum) and walk-through guides like this one.
\n\nAnd here's Python implementation
", "rawMarkdown": "xgbfi gracefully brought to us by @Far0n compiles significance of feature interactions using xgb model dump. You can read about it online. Just search xgbfi and you will find excellent examples of use (including on this forum) and walk-through guides like [this one][1]. \r\n\r\nAnd [here's][2] Python implementation\r\n\r\n\r\n [1]: http://projects.rajivshah.com/blog/2016/08/01/xgbfi/\r\n [2]: https://github.com/limexp/xgbfir", "votes": 2 }, { "id": 152694, "postDate": "2016-12-27T23:07:53.333Z", "content": "Your comments are really useful!! Thanks.
", "rawMarkdown": "Your comments are really useful!! Thanks." }, { "id": 152526, "postDate": "2016-12-26T23:53:46.863Z", "content": "Hi Alexey Noskov,
\n\nI am wondering what the meanings of n_splits, n_bags in your code. I understand it like this. For the param n_splits, you wanna try different fold splitting in KFold function. For n_bags, you wanna try different seeds for a particular model which is not what the true bagging means as the model is still trained on the full training data set. I guess this kind of new bagging works for Xgb, nn, LightGB and may not work well for lr, et, rf which I observed in your code.
\n\nAnother quick question is that why you didn't use early_stopping_rounds in training your xgb models. Are there any tricks behind that?
", "rawMarkdown": "Hi Alexey Noskov,\r\n\r\nI am wondering what the meanings of n_splits, n_bags in your code. I understand it like this. For the param n_splits, you wanna try different fold splitting in KFold function. For n_bags, you wanna try different seeds for a particular model which is not what the true bagging means as the model is still trained on the full training data set. I guess this kind of new bagging works for Xgb, nn, LightGB and may not work well for lr, et, rf which I observed in your code. \r\n\r\nAnother quick question is that why you didn't use early_stopping_rounds in training your xgb models. Are there any tricks behind that?\r\n\r\n" }, { "id": 152181, "postDate": "2016-12-23T21:21:26.350Z", "content": "@Alexey Noskov
\n\n\"I applied boxcox after log\"...
\n\nOk, that's what I did to but it may miss something later on...
\n\nFor me, it was'nt enough. Even with a \"log\", \"BoxCox\", \"Center\", \"Scale\"...
\n\nDid I miss something ?
\n\nI join everybody here and congrat you for your performance, really, honnestly !
\n\nBe proud Alexey
\n\nPS : \"Naive Bayes\" ? hum...
", "rawMarkdown": "@Alexey Noskov\r\n\r\n\"I applied boxcox after log\"...\r\n\r\nOk, that's what I did to but it may miss something later on...\r\n\r\nFor me, it was'nt enough. Even with a \"log\", \"BoxCox\", \"Center\", \"Scale\"...\r\n\r\nDid I miss something ? \r\n\r\nI join everybody here and congrat you for your performance, really, honnestly !\r\n\r\nBe proud Alexey \r\n\r\nPS : \"Naive Bayes\" ? hum..." }, { "id": 152109, "postDate": "2016-12-23T12:00:50.850Z", "content": "Thanks for your help. I really appreciate your efforts. You deserve it!!!
", "rawMarkdown": "Thanks for your help. I really appreciate your efforts. You deserve it!!!" }, { "id": 152068, "postDate": "2016-12-23T07:59:43.127Z", "content": "Hi!
\n\nI applied boxcox after log, and as I remember I've added 1 before log to avoid negative values, but for now I don't see much sense in it.
Where was hypothesis in forums what some columns represent some time values, so I tried to add their difference as it may provide value to the model. It didn't help.
When looking on feature value distributions it may be seen what some features have significant amount of values equal to exactly 0 or 1, so I created special features for such cases. It also didn't help.
There is a lot of models and features in code which I tried and they didn't help, really.
", "rawMarkdown": "Hi!\r\n\r\n1. I applied boxcox after log, and as I remember I've added 1 before log to avoid negative values, but for now I don't see much sense in it.\r\n\r\n2. Where was hypothesis in forums what some columns represent some time values, so I tried to add their difference as it may provide value to the model. It didn't help.\r\n\r\n3. When looking on feature value distributions it may be seen what some features have significant amount of values equal to exactly 0 or 1, so I created special features for such cases. It also didn't help. \r\n\r\nThere is a lot of models and features in code which I tried and they didn't help, really." }, { "id": 152060, "postDate": "2016-12-23T07:10:11.063Z", "content": "Thanks a lot for sharing! I really learn a lot from you. I have several questions to ask.
\n\n[quote=Alexey Noskov;149950]
\n\nIn my late XGB models I also used categorical combinations, the best way to select combinations was using excellent [Xgbfi][3] tool - just take dump of good XGB model without combinations, run Xgbfi and take some number of 2-way interactions.
\n\n[/quote]
\n\nThanks a lot for sharing!\nWhat is your approach to use interactions after xgbfi? Just take top-interactions from \"Interaction Depth X\" sheets and consider it as a new categorical vars?
", "rawMarkdown": "[quote=Alexey Noskov;149950]\r\n\r\nIn my late XGB models I also used categorical combinations, the best way to select combinations was using excellent [Xgbfi][3] tool - just take dump of good XGB model without combinations, run Xgbfi and take some number of 2-way interactions.\r\n\r\n[/quote]\r\n\r\n\r\nThanks a lot for sharing!\r\nWhat is your approach to use interactions after xgbfi? Just take top-interactions from \"Interaction Depth X\" sheets and consider it as a new categorical vars?" }, { "id": 150161, "postDate": "2016-12-14T04:04:15.957Z", "content": "Congrats Alexey, and thanks for sharing the code. I have a question, is there any hiararchy relationship between the level one and level two?
", "rawMarkdown": "Congrats Alexey, and thanks for sharing the code. I have a question, is there any hiararchy relationship between the level one and level two?", "replies": [ { "id": 150210, "postDate": "2016-12-14T08:00:29.007Z", "content": "[quote=Weiwei Liu;150161]
\n\nCongrats Alexey, and thanks for sharing the code. I have a question, is there any hiararchy relationship between the level one and level two?
\n\n[/quote]
\n\nLevel 2 models use predictions of Level 1 models as a features (and Level 3 use predictions of Level 2), that's all hierarchy.
", "rawMarkdown": "[quote=Weiwei Liu;150161]\r\n\r\nCongrats Alexey, and thanks for sharing the code. I have a question, is there any hiararchy relationship between the level one and level two?\r\n\r\n[/quote]\r\n\r\nLevel 2 models use predictions of Level 1 models as a features (and Level 3 use predictions of Level 2), that's all hierarchy.\r\n" } ] }, { "id": 150032, "postDate": "2016-12-13T16:18:34.657Z", "content": "could you please explain how you did use xgbfi?
", "rawMarkdown": "could you please explain how you did use xgbfi?" }, { "id": 149981, "postDate": "2016-12-13T11:45:46.843Z", "content": "Congrats Alexey . Thanks for sharing solution
", "rawMarkdown": "Congrats Alexey . Thanks for sharing solution" }, { "id": 149973, "postDate": "2016-12-13T10:47:42.463Z", "content": "Congrats Alexey. Nice third level model :)
", "rawMarkdown": "Congrats Alexey. Nice third level model :)" }, { "id": 149956, "postDate": "2016-12-13T10:10:17.837Z", "content": "Wow, very cool ~ I have learned a lot from your experience and look forward to your code.
", "rawMarkdown": "Wow, very cool ~ I have learned a lot from your experience and look forward to your code." }, { "id": 149955, "postDate": "2016-12-13T10:10:16.180Z", "rawMarkdown": "", "isDeleted": true }, { "id": 149996, "postDate": "2016-12-13T13:42:03.980Z", "content": "Thanks for the information !!
", "rawMarkdown": "Thanks for the information !!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:16:07Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 386284, "solution_url": "/competitions/allstate-claims-severity/writeups/alexey-noskov-2nd-place-solution" }