{ "id": 26416, "title": "#1st Place Solution", "content": "

Hi Guys,

\n\n

Here is a short summary of my final solution: Nothing extra-oridinary but a lot of diverse models and data preprocessing. So after my 1st submission which scored 1118 on public LB and I landed the 3rd position at the very beginning of the competiton, I thought I would give a try for my solo gold medal here. I had just relocated to a new city and joined a new company at that moment, but I tried my best to keep in touch with the leaderboard progression and hence update my models going ahead.

\n\n

Best Single Model:

\n\n

My best single model is an XGBoost with the two-way interactions reported on the forum along with some chosen three, four, five and seven-way interactions. It score 1105.12 in the public LB with 10-fold CV around 1124. I chose the higher order interactions primarily based on mutual information. I used fair_obj with a constant of 0.7 for training that model and (1 + loss)**0.25 transformation reported by MrOijer.

\n\n

My NN architecture

\n\n

My best NN scores 1110 in the public leaderboard with a 10-fold cv of 1134.xx. It is very similar to the public scirpt scoring 1111.8x , just some more neurons in each layer. I used EarlyStopping, ModelCheckpoint and derived metric to monitor (I used log(loss + SHIFT) transformation) and 10-fold 10 times bagged model. Almost all my NN models follow the same architecture.

\n\n

I used mostly the above two models to train a stacked model (details later in the post) in the second layer, but there are other models with significant contribution. Among them:

\n\n

Regularized Greedy Forest\nI had three RGF mixed in to my ensemble trained with slighly different data and loss transformation. My best RGF scores 1113.xx with a cv around 1136.xx. It is a fantastic library that I explored back in BNP Paribas competiton and was determined to use it in one of my competitions. Just one setback is that it's a bit on the slower side being single-threaded.

\n\n

LightGBM\nAnother very fast and efficient gradient boosting library beside XGboost. I loved it. Best single model 1111.xx on public LB.

\n\n

Vowpal Wabbit\nAlways found it worth giving a shot and yes it beat random forest and extra trees which I never could get executed with MAE criterion in sklearn. Best single model 1133.xx with a cv of 1154.xx. I used quadtraic interactions in the categorical namespace.

\n\n

LibFM\nNot much from single model, but definitely helped my ensemble. Best model 1166.xx with 10-fold cv of 1188.xx.

\n\n

LibFFM\nAlmost at par with my VW. Best model 10-fold cv 1158.xx with public LB 1139.xx.

\n\n

I also used RF(R-h20), ExtraTrees(sklearn), glmnet (R) and some other models to diversify my final ensemble but none of them yielded significant individual performance.

\n\n

Stacking pipeline

\n\n

I used 10-fold stacking in this competiton and tried a number of preprocessing on the data:

\n\n\n\n

I had a total of 81 models in the 1st level and trained an XGB and two NN on them on the second layer.

\n\n

Best Second Level Model\nMy best L2 model has a 10-fold cv of 1114.645 and public LB 1097.87.

\n\n

Final Submission

\n\n

My final submission is an wegthed average of the following:

\n\n
    \n
  1. w1*NN1^w2 + w3*NN2^w4 + w5*XGB1^w6 + w7 - weights optimized by using optim (Nelder-Mead) in a 1-fold manner => apply weights to test predictions => average 10 test predictions for 10x optimized weights.
    2. \n
  2. If NN1 < w1 , then w2*NN1^w3 + w4 Else if NN1 > w5, then w6*NN1^w7\n + w8 Else NN1
    3. \n
\n\n

I extend my congratulations to all other winners and my sincere thanks to Kaggle and Allstate group for arranging this exciting competition.

\n\n

Special thanks to Scirpus, Vladimir, Tilli, Lauea (as always), Danijel, d3miekno and all the other people who kept the forum so active all the time with their brilliant ideas and scripts :)

", "comments": [ { "id": 149922, "author_name": "Bishwarup B", "author_url": "", "post_date": "2016-12-13T07:35:54.940000", "content": "

@DDgg: I am not aware of any such open/commercial software who will automate the whole stacking pipleline. There are libraries in both R and python to do the ensembling part for you but I haven't looked into those very deeply yet.

\n\n

@qianqian: My NN models took 24 hours on an average to complete. XGboost was much faster. None of my XGBoost models has an execution duration of more than 4 hours (I did 10-fold CV).

\n\n

@Balaji SR: I actually did not train on whole data. None of my models are trained on whole data as it was pretty tricky to find the optimum number of rounds. So I chose to do 10-fold CV average approach which also seemed quite stable in terms of CV-LB agreement. However, if I calculate the optimum round from my best model by the formula average best round/ proportion sample in training set it comes around 2700. 15000 seems too much, I think you were using a very small learning rate which resulted in large number of iterations. I tried to cut down the learning rate as well but that didn't improve my cv or lb. I almost always used learning rate = 0.03, the best number of rounds was pretty dependent on the gamma though.

\n\n

@Gippy: I calculated the mutual information between the interaction features and the transformed target. If you are looking for how to calculate MI, you can look into the packages like mRMRe or infotheo. In python, you can look into the function sklearn.metrics.mutual_info_score. I took the interactions which had high MI score.

\n\n

@olivier: Thanks for the link. I actually came across that but didn't give it a try yet. Does it have cv support find the optimum model number given other parameters? I actually used my own wrapper in python for RGF but that is not multithreaded.

", "votes": 15, "replies": [] }, { "id": 149903, "author_name": "Siddharth Chandrakant", "author_url": "", "post_date": "2016-12-13T06:05:29.923000", "content": "

Congrats Bishwarup, not only for the win, but also for becoming the highest ranked Indian Kaggler when the rankings get updated! :)

", "votes": 11, "replies": [] }, { "id": 150253, "author_name": "SRK", "author_url": "", "post_date": "2016-12-14T12:06:16.590000", "content": "

Excellent Bishwarup. The solution shows the amount of hard work that you have put in. Well deserved victory. Congratulations.!

\n\n

Congrats on the overall top 10 as well. Time to party :)

", "votes": 9, "replies": [] }, { "id": 149948, "author_name": "Qingchen", "author_url": "", "post_date": "2016-12-13T09:41:33.230000", "content": "

Congrats! I'm happy to see that you got so much success from ensembling many models. I pushed my best xgboost to ~1103 MAE three weeks ago and didn't know how to make improvements anymore. Really nice to see the effect of ensembling of different variations. Learned a lot from you here, definitely saving your post :)

", "votes": 7, "replies": [ { "id": 150017, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T15:24:17.087000", "content": "

[quote=Qingchen;149948]

\n\n

Congrats! I'm happy to see that you got so much success from ensembling many models. I pushed my best xgboost to ~1103 MAE three weeks ago and didn't know how to make improvements anymore.

\n\n

[/quote]\n1103 is very impressive. Could you please share code, hyperparameters, feature engineering?

", "votes": 3, "replies": [] } ] }, { "id": 149902, "author_name": "DDgg", "author_url": "", "post_date": "2016-12-13T06:01:16.833000", "content": "

Thank you,\nmuch appreciated!

\n\n

Questions of course:
\n1) Is there an open/commercial pipeline/workflow that could match those results consistently
\n2) Is there an open or commercial pipeline/workflow that could deploy the whole solution consistently

\n\n

Basically, would it be feasible to implement a similar model
\nthat would consistently get good results on your proposed workflow.

\n\n

Surely interesting for commercial/open ML providers for regression problems.

", "votes": 3, "replies": [] }, { "id": 150706, "author_name": "Bishwarup B", "author_url": "", "post_date": "2016-12-16T12:01:13.023000", "content": "

@Gippy: I didn't use bootstrapping, I simply used oof-data for EarlyStopping.

\n\n

@ Ronin: I have written my own wrapper for VW in python for performing cross-validation and other stuff.

", "votes": 1, "replies": [] }, { "id": 150356, "author_name": "deltap", "author_url": "", "post_date": "2016-12-14T21:50:11.247000", "content": "

Thank you so much for sharing such a great solution! I have a question about generating out-of-fold predictions from bagged models (e.g. your NN bag) for stacking. I assume you used the same 10-fold split (call it 'a') for all of your base models. When you generate the meta/oof features of your NN models do you do another 10-fold cv (call it 'b') on the training portion of (a), early stop each of the models based on the valid portion of (b), and bag them to make predictions/oof features on the valid fold of (a)? In another word are the meta features you computed via the first 10-fold cv(a) generated by a similar 10-fold 10 times bagged model on the training portions of (a)

\n\n

I wonder if this type of 'nested' cv is necessary for all kinds of bagged models when the individual ones are terminated by the validation set

\n\n

I am not sure of I have made myself clear. Let me know if it is still confusing.

", "votes": 1, "replies": [] }, { "id": 149941, "author_name": "Bishwarup B", "author_url": "", "post_date": "2016-12-13T09:05:50.817000", "content": "

@Engine: I tuned the parameter manually for RGF (RGF, reg_L2 = 0.035, min_pop = 80, max_leaf_forest = 4500). Without interaction it took around 3.5 hrs and with interaction around 7 hours.

\n\n

@Evgeny_Semyonov: I used Ubuntu15.10 for the VW model. I used python to convert the data into vw format and execute the command line from within the script only. For your second query, the idea is when the NN (second layer) predicts low and higher value, I optimized the adjustment parameters differently.

", "votes": 1, "replies": [] }, { "id": 149937, "author_name": "RR", "author_url": "", "post_date": "2016-12-13T08:57:48.990000", "content": "

Congrats Bishwarup. great achievement.

", "votes": 1, "replies": [] }, { "id": 149920, "author_name": "olivier", "author_url": "", "post_date": "2016-12-13T07:24:41.397000", "content": "

Bishwarup Congratulations !
\nThank you for detailing your solution in this way. \nThese algos will certainly end up in my arsenal. \nYou're probably aware but just in case, there's a multi core RGF available here: https://github.com/baidu/fast_rgf\nI still need to test it though.

", "votes": 1, "replies": [] }, { "id": 154664, "author_name": "ShashankHegdeK", "author_url": "", "post_date": "2017-01-07T14:18:10.450000", "content": "

@Bishwarup B Thanks for sharing the solution. Can you please share the code as well?

", "votes": 2, "replies": [] }, { "id": 150635, "author_name": "Ronin", "author_url": "", "post_date": "2016-12-16T05:05:14.267000", "content": "

How did you perform your CV in VW? Did you do it manually? I've always had trouble optimizing VW as the \"scoring\" functions used in competitions tend not to be implemented directly in vw (although the quintile loss function seemed to be working well this competition) as well as having to manually split the data set.

\n\n

I've never had good luck with the VW wrappers, although the program itself is slick as snot once I figured out what I was doing.

", "votes": 2, "replies": [] }, { "id": 150416, "author_name": "FredTony", "author_url": "", "post_date": "2016-12-15T02:58:56.450000", "content": "

And for python: https://docs.scipy.org/doc/scipy-0.18.1/reference/optimize.minimize-neldermead.html

", "votes": 2, "replies": [ { "id": 2186320, "author_name": "Tony6512", "author_url": "", "post_date": "2023-03-17T17:53:46.860000", "content": "

This is helpful. Thanks.

", "votes": 0, "replies": [] } ] }, { "id": 149927, "author_name": "Evgeny_Semyonov", "author_url": "", "post_date": "2016-12-13T07:44:44.397000", "content": "

Congratulations! Very cool solution.

\n\n

Questions:

\n\n

1) i tried to use Vowpal Wabbit, but it's terrible on Windows. What's your platform?(Windows/Unix)\nAn did you use it from Python or converted data to Vowpal Wabbit format and start from console?

\n\n

2) \"If NN1 < w1 , then w2*NN1^w3 + w4 Else if NN1 > w5, then w6*NN1^w7 + w8 Else NN1\".

\n\n

NN1 - it's predictions from Neural second level model and w1 - it's weitht (scalar)? How can we compare them?

", "votes": 2, "replies": [] }, { "id": 149923, "author_name": "Engine", "author_url": "", "post_date": "2016-12-13T07:37:03.353000", "content": "

congratulations for the 1st place!\nTell, please, about parameters of RGF (algorithm modification (RGF|RGF Opt|RGF Sib), num of leaves, kind of regularization parameters). How much of time takes for rgf to make prediction? \nThanks for sharing your ideas

", "votes": 2, "replies": [] }, { "id": 149917, "author_name": "Abhay Shukla", "author_url": "", "post_date": "2016-12-13T07:16:37.597000", "content": "

So this is what it takes. Phew! Congratulations :)

", "votes": 2, "replies": [] }, { "id": 970385, "author_name": "Murilo Palomo Sebilla", "author_url": "", "post_date": "2020-08-14T12:03:20.523000", "content": "

Thank you, it is really helpful to new guys like me ^^.

", "votes": 0, "replies": [] }, { "id": 261183, "author_name": "tedfyw", "author_url": "", "post_date": "2017-12-22T00:21:34.733000", "content": "

Thanks for sharing. That is really helpful!

", "votes": 0, "replies": [] }, { "id": 169239, "author_name": "deadpitman", "author_url": "", "post_date": "2017-03-20T08:16:19.507000", "content": "

Thx for sharing your strategy.

", "votes": 0, "replies": [] }, { "id": 162633, "author_name": "Amit Gupta", "author_url": "", "post_date": "2017-02-20T11:45:17.850000", "content": "

Many congrats Bishwarup B!!!!. Could you please share your code also.

", "votes": 0, "replies": [] }, { "id": 150903, "author_name": "Bishwarup B", "author_url": "", "post_date": "2016-12-17T12:51:55.397000", "content": "

@Gippy: I didn't change any parameter as such, just the random_seed to make the model more robust as NN is very sentsitive to noise in the data.

", "votes": 0, "replies": [] }, { "id": 150718, "author_name": "Gippy", "author_url": "", "post_date": "2016-12-16T13:30:04.677000", "content": "

@Bishwarup: Then can you please let me know what do you mean by 10 bags for each fold? What parameter do you change for each bag? Thanks

", "votes": 0, "replies": [] }, { "id": 150506, "author_name": "Gippy", "author_url": "", "post_date": "2016-12-15T13:38:28.027000", "content": "

@Bishwarup: 10 fold 10 bags: Does it mean that for every fold, you take bootstrap sample (10 times) and use that for training (if yes, then do you use the oob data for early stopping or oof fold data for early stopping). Or have i misunderstood the approach? Thanks

", "votes": 0, "replies": [] }, { "id": 150490, "author_name": "Bishwarup B", "author_url": "", "post_date": "2016-12-15T12:22:42.593000", "content": "

@Achal: I used percentile based binning.

\n\n

@ SRK: Thanks man :)

\n\n

@ deltap: 10 fold 10 bags -> so 100 folds in total -> EarlyStopping in each of the folds -> predict oof and test -> average test for all the 100 runs.

\n\n

@Finaly Liu: I think ClusterFork and Tony already cleared your doubts.

", "votes": 0, "replies": [] }, { "id": 150406, "author_name": "Finlay", "author_url": "", "post_date": "2016-12-15T02:17:41.223000", "content": "

optim (Nelder-Mead) , what does this mean?

", "votes": 0, "replies": [ { "id": 150414, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2016-12-15T02:55:29.700000", "content": "

[quote=Finlay Liu;150406]

\n\n

optim (Nelder-Mead) , what does this mean?

\n\n

[/quote]\n?optim in R console:

\n\n

optim {stats} R Documentation\nGeneral-purpose Optimization

\n\n

Description

\n\n

General-purpose optimization based on Nelder–Mead, quasi-Newton and conjugate-gradient algorithms. It includes an option for box-constrained optimization and simulated annealing.

", "votes": 0, "replies": [] } ] }, { "id": 150076, "author_name": "Achal", "author_url": "", "post_date": "2016-12-13T19:14:10.190000", "content": "
\n

\"I chose the higher order interactions primarily based on mutual\n information\"

\n \n

\" I calculated the mutual information between the interaction features\n and the transformed target. If you are looking for how to calculate\n MI, you can look into the packages like mRMRe or infotheo. In python,\n you can look into the function sklearn.metrics.mutual_info_score. I\n took the interactions which had high MI score.\"

\n
\n\n

Did you calculate mutual information among all categorical features, right? Our loss function is continuous, so how did you calculate MI to calculate interactions? Any readings or link will be helpful.

", "votes": 0, "replies": [] }, { "id": 150021, "author_name": "Travis Glines", "author_url": "", "post_date": "2016-12-13T15:36:33.727000", "content": "

Congrats Bishwarup! Thanks for the play-by-play.

", "votes": 0, "replies": [] }, { "id": 150020, "author_name": "InnerVoice", "author_url": "", "post_date": "2016-12-13T15:29:45.517000", "content": "

I have tried to use vowpal and max CV I got was around 1165. Need some tuning tips for vowpal. I also tried to use RGF during last couple of days but not able to complete the training. I found your solution easy to understand as compared to others Multilayer stacking.. and will use some of the tips in near future.

", "votes": 0, "replies": [] }, { "id": 150001, "author_name": "Vipin", "author_url": "", "post_date": "2016-12-13T14:08:13.037000", "content": "

Congratulations Bishwarup B.This was my first competition I never even thought about ensembling many models.Thanks for explaining your solution.

", "votes": 0, "replies": [] }, { "id": 149918, "author_name": "", "author_url": "", "post_date": "2016-12-13T07:21:06.597000", "content": "", "votes": 0, "replies": [] }, { "id": 149910, "author_name": "", "author_url": "", "post_date": "2016-12-13T06:40:55.803000", "content": "", "votes": 0, "replies": [] }, { "id": 149908, "author_name": "", "author_url": "", "post_date": "2016-12-13T06:29:33.187000", "content": "", "votes": 0, "replies": [] }, { "id": 149907, "author_name": "", "author_url": "", "post_date": "2016-12-13T06:28:07.327000", "content": "", "votes": 0, "replies": [] }, { "id": 149905, "author_name": "", "author_url": "", "post_date": "2016-12-13T06:23:39.387000", "content": "", "votes": 0, "replies": [] }, { "id": 149904, "author_name": "", "author_url": "", "post_date": "2016-12-13T06:11:42.260000", "content": "", "votes": 0, "replies": [] }, { "id": 150011, "author_name": "", "author_url": "", "post_date": "2016-12-13T14:53:35.440000", "content": "", "votes": 0, "replies": [] }, { "id": 1913465, "author_name": "", "author_url": "", "post_date": "2022-08-25T10:33:50.040000", "content": "", "votes": 0, "replies": [] }, { "id": 150984, "author_name": "", "author_url": "", "post_date": "2016-12-18T03:15:11.007000", "content": "", "votes": 0, "replies": [] }, { "id": 150522, "author_name": "", "author_url": "", "post_date": "2016-12-15T15:05:05.073000", "content": "", "votes": 0, "replies": [] }, { "id": 149991, "author_name": "", "author_url": "", "post_date": "2016-12-13T13:20:00.253000", "content": "", "votes": 0, "replies": [] }, { "id": 149900, "author_name": "", "author_url": "", "post_date": "2016-12-13T05:50:17.413000", "content": "", "votes": 0, "replies": [] }, { "id": 149897, "author_name": "", "author_url": "", "post_date": "2016-12-13T05:30:24.663000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 45, "messages": [ { "id": 149896, "postDate": "2016-12-13T05:23:14.197Z", "content": "

Hi Guys,

\n\n

Here is a short summary of my final solution: Nothing extra-oridinary but a lot of diverse models and data preprocessing. So after my 1st submission which scored 1118 on public LB and I landed the 3rd position at the very beginning of the competiton, I thought I would give a try for my solo gold medal here. I had just relocated to a new city and joined a new company at that moment, but I tried my best to keep in touch with the leaderboard progression and hence update my models going ahead.

\n\n

Best Single Model:

\n\n

My best single model is an XGBoost with the two-way interactions reported on the forum along with some chosen three, four, five and seven-way interactions. It score 1105.12 in the public LB with 10-fold CV around 1124. I chose the higher order interactions primarily based on mutual information. I used fair_obj with a constant of 0.7 for training that model and (1 + loss)**0.25 transformation reported by MrOijer.

\n\n

My NN architecture

\n\n

My best NN scores 1110 in the public leaderboard with a 10-fold cv of 1134.xx. It is very similar to the public scirpt scoring 1111.8x , just some more neurons in each layer. I used EarlyStopping, ModelCheckpoint and derived metric to monitor (I used log(loss + SHIFT) transformation) and 10-fold 10 times bagged model. Almost all my NN models follow the same architecture.

\n\n

I used mostly the above two models to train a stacked model (details later in the post) in the second layer, but there are other models with significant contribution. Among them:

\n\n

Regularized Greedy Forest\nI had three RGF mixed in to my ensemble trained with slighly different data and loss transformation. My best RGF scores 1113.xx with a cv around 1136.xx. It is a fantastic library that I explored back in BNP Paribas competiton and was determined to use it in one of my competitions. Just one setback is that it's a bit on the slower side being single-threaded.

\n\n

LightGBM\nAnother very fast and efficient gradient boosting library beside XGboost. I loved it. Best single model 1111.xx on public LB.

\n\n

Vowpal Wabbit\nAlways found it worth giving a shot and yes it beat random forest and extra trees which I never could get executed with MAE criterion in sklearn. Best single model 1133.xx with a cv of 1154.xx. I used quadtraic interactions in the categorical namespace.

\n\n

LibFM\nNot much from single model, but definitely helped my ensemble. Best model 1166.xx with 10-fold cv of 1188.xx.

\n\n

LibFFM\nAlmost at par with my VW. Best model 10-fold cv 1158.xx with public LB 1139.xx.

\n\n

I also used RF(R-h20), ExtraTrees(sklearn), glmnet (R) and some other models to diversify my final ensemble but none of them yielded significant individual performance.

\n\n

Stacking pipeline

\n\n

I used 10-fold stacking in this competiton and tried a number of preprocessing on the data:

\n\n\n\n

I had a total of 81 models in the 1st level and trained an XGB and two NN on them on the second layer.

\n\n

Best Second Level Model\nMy best L2 model has a 10-fold cv of 1114.645 and public LB 1097.87.

\n\n

Final Submission

\n\n

My final submission is an wegthed average of the following:

\n\n
    \n
  1. w1*NN1^w2 + w3*NN2^w4 + w5*XGB1^w6 + w7 - weights optimized by using optim (Nelder-Mead) in a 1-fold manner => apply weights to test predictions => average 10 test predictions for 10x optimized weights.
    2. \n
  2. If NN1 < w1 , then w2*NN1^w3 + w4 Else if NN1 > w5, then w6*NN1^w7\n + w8 Else NN1
    3. \n
\n\n

I extend my congratulations to all other winners and my sincere thanks to Kaggle and Allstate group for arranging this exciting competition.

\n\n

Special thanks to Scirpus, Vladimir, Tilli, Lauea (as always), Danijel, d3miekno and all the other people who kept the forum so active all the time with their brilliant ideas and scripts :)

", "rawMarkdown": "Hi Guys,\r\n\r\nHere is a short summary of my final solution: Nothing extra-oridinary but a lot of diverse models and data preprocessing. So after my 1st submission which scored 1118 on public LB and I landed the 3rd position at the very beginning of the competiton, I thought I would give a try for my solo gold medal here. I had just relocated to a new city and joined a new company at that moment, but I tried my best to keep in touch with the leaderboard progression and hence update my models going ahead.\r\n\r\n**Best Single Model:**\r\n\r\nMy best single model is an XGBoost with the two-way interactions reported on the forum along with some chosen three, four, five and seven-way interactions. It score 1105.12 in the public LB with 10-fold CV around 1124. I chose the higher order interactions primarily based on mutual information. I used [fair_obj][1] with a constant of 0.7 for training that model and `(1 + loss)**0.25` transformation reported by [MrOijer][2].\r\n\r\n**My NN architecture**\r\n\r\nMy best NN scores 1110 in the public leaderboard with a 10-fold cv of 1134.xx. It is very similar to the [public scirpt scoring 1111.8x][3] , just some more neurons in each layer. I used EarlyStopping, ModelCheckpoint and derived metric to monitor (I used `log(loss + SHIFT)` transformation) and 10-fold 10 times bagged model. Almost all my NN models follow the same architecture.\r\n\r\nI used mostly the above two models to train a stacked model (details later in the post) in the second layer, but there are other models with significant contribution. Among them:\r\n\r\n**Regularized Greedy Forest**\r\nI had three RGF mixed in to my ensemble trained with slighly different data and loss transformation. My best RGF scores 1113.xx with a cv around 1136.xx. It is a fantastic library that I explored back in BNP Paribas competiton and was determined to use it in one of my competitions. Just one setback is that it's a bit on the slower side being single-threaded.\r\n\r\n**LightGBM**\r\nAnother very fast and efficient gradient boosting library beside XGboost. I loved it. Best single model 1111.xx on public LB.\r\n\r\n**Vowpal Wabbit**\r\nAlways found it worth giving a shot and yes it beat random forest and extra trees which I never could get executed with `MAE` criterion in sklearn. Best single model 1133.xx with a cv of 1154.xx. I used quadtraic interactions in the categorical namespace.\r\n\r\n**LibFM**\r\nNot much from single model, but definitely helped my ensemble. Best model 1166.xx with 10-fold cv of 1188.xx.\r\n\r\n**LibFFM**\r\nAlmost at par with my VW. Best model 10-fold cv 1158.xx with public LB 1139.xx.\r\n\r\nI also used RF(R-h20), ExtraTrees(sklearn), glmnet (R) and some other models to diversify my final ensemble but none of them yielded significant individual performance.\r\n\r\n**Stacking pipeline**\r\n\r\nI used 10-fold stacking in this competiton and tried a number of preprocessing on the data:\r\n\r\n - Different order interactions between categorical features.\r\n - TF-IDF on categorical features.\r\n - [Category Embedding][4] with NN and different embedding layer sizes.\r\n - Standard Scalar/ Minmax scalar\r\n - Different loss function in xgb incuding count:poisson on the rounded and log transformed targets. Surprisingly it gave me a public LB of 1168.xx.\r\n - Trained XGB on the bottom 5 and 10 percentile and top 70, 80 and 90 percentile of data. The motivation was to get a better estimate of the exorbitant loss values.\r\n\r\nI had a total of 81 models in the 1st level and trained an XGB and two NN on them on the second layer.\r\n\r\n**Best Second Level Model**\r\nMy best L2 model has a 10-fold cv of 1114.645 and public LB 1097.87.\r\n\r\n**Final Submission**\r\n\r\nMy final submission is an wegthed average of the following: \r\n \r\n\r\n 1. `w1*NN1^w2 + w3*NN2^w4 + w5*XGB1^w6 + w7` - weights optimized by using `optim` (Nelder-Mead) in a 1-fold manner => apply weights to test predictions => average 10 test predictions for 10x optimized weights.\r\n 2. If NN1 < w1 , then w2*NN1^w3 + w4 Else if NN1 > w5, then w6*NN1^w7\r\n + w8 Else NN1\r\n\r\nI extend my congratulations to all other winners and my sincere thanks to Kaggle and Allstate group for arranging this exciting competition. \r\n\r\nSpecial thanks to Scirpus, Vladimir, Tilli, Lauea (as always), Danijel, d3miekno and all the other people who kept the forum so active all the time with their brilliant ideas and scripts :)\r\n\r\n [1]: https://www.kaggle.com/c/allstate-claims-severity/forums/t/24520/effect-of-mae?forumMessageId=140255#post140255\r\n [2]: https://www.kaggle.com/mrooijer\r\n [3]: https://www.kaggle.com/mtinti/allstate-claims-severity/keras-starter-with-bagging-1111-84364/code\r\n [4]: https://github.com/entron/entity-embedding-rossmann", "votes": 180 }, { "id": 149922, "postDate": "2016-12-13T07:35:54.940Z", "content": "

@DDgg: I am not aware of any such open/commercial software who will automate the whole stacking pipleline. There are libraries in both R and python to do the ensembling part for you but I haven't looked into those very deeply yet.

\n\n

@qianqian: My NN models took 24 hours on an average to complete. XGboost was much faster. None of my XGBoost models has an execution duration of more than 4 hours (I did 10-fold CV).

\n\n

@Balaji SR: I actually did not train on whole data. None of my models are trained on whole data as it was pretty tricky to find the optimum number of rounds. So I chose to do 10-fold CV average approach which also seemed quite stable in terms of CV-LB agreement. However, if I calculate the optimum round from my best model by the formula average best round/ proportion sample in training set it comes around 2700. 15000 seems too much, I think you were using a very small learning rate which resulted in large number of iterations. I tried to cut down the learning rate as well but that didn't improve my cv or lb. I almost always used learning rate = 0.03, the best number of rounds was pretty dependent on the gamma though.

\n\n

@Gippy: I calculated the mutual information between the interaction features and the transformed target. If you are looking for how to calculate MI, you can look into the packages like mRMRe or infotheo. In python, you can look into the function sklearn.metrics.mutual_info_score. I took the interactions which had high MI score.

\n\n

@olivier: Thanks for the link. I actually came across that but didn't give it a try yet. Does it have cv support find the optimum model number given other parameters? I actually used my own wrapper in python for RGF but that is not multithreaded.

", "rawMarkdown": "@DDgg: I am not aware of any such open/commercial software who will automate the whole stacking pipleline. There are libraries in both R and python to do the ensembling part for you but I haven't looked into those very deeply yet.\r\n\r\n@qianqian: My NN models took 24 hours on an average to complete. XGboost was much faster. None of my XGBoost models has an execution duration of more than 4 hours (I did 10-fold CV).\r\n\r\n@Balaji SR: I actually did not train on whole data. None of my models are trained on whole data as it was pretty tricky to find the optimum number of rounds. So I chose to do 10-fold CV average approach which also seemed quite stable in terms of CV-LB agreement. However, if I calculate the optimum round from my best model by the formula `average best round/ proportion sample in training set` it comes around 2700. 15000 seems too much, I think you were using a very small learning rate which resulted in large number of iterations. I tried to cut down the learning rate as well but that didn't improve my cv or lb. I almost always used learning rate = 0.03, the best number of rounds was pretty dependent on the `gamma` though.\r\n\r\n@Gippy: I calculated the mutual information between the interaction features and the transformed target. If you are looking for how to calculate MI, you can look into the packages like [mRMRe][1] or [infotheo][2]. In python, you can look into the function [sklearn.metrics.mutual_info_score][3]. I took the interactions which had high MI score.\r\n\r\n@olivier: Thanks for the link. I actually came across that but didn't give it a try yet. Does it have cv support find the optimum model number given other parameters? I actually used my own wrapper in python for RGF but that is not multithreaded.\r\n\r\n\r\n [1]: https://cran.r-project.org/web/packages/mRMRe/mRMRe.pdf\r\n [2]: https://cran.r-project.org/web/packages/infotheo/infotheo.pdf\r\n [3]: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mutual_info_score.html", "votes": 15 }, { "id": 149903, "postDate": "2016-12-13T06:05:29.923Z", "content": "

Congrats Bishwarup, not only for the win, but also for becoming the highest ranked Indian Kaggler when the rankings get updated! :)

", "rawMarkdown": "Congrats Bishwarup, not only for the win, but also for becoming the highest ranked Indian Kaggler when the rankings get updated! :)", "votes": 11 }, { "id": 150253, "postDate": "2016-12-14T12:06:16.590Z", "content": "

Excellent Bishwarup. The solution shows the amount of hard work that you have put in. Well deserved victory. Congratulations.!

\n\n

Congrats on the overall top 10 as well. Time to party :)

", "rawMarkdown": "Excellent Bishwarup. The solution shows the amount of hard work that you have put in. Well deserved victory. Congratulations.!\r\n\r\nCongrats on the overall top 10 as well. Time to party :) ", "votes": 9 }, { "id": 149948, "postDate": "2016-12-13T09:41:33.230Z", "content": "

Congrats! I'm happy to see that you got so much success from ensembling many models. I pushed my best xgboost to ~1103 MAE three weeks ago and didn't know how to make improvements anymore. Really nice to see the effect of ensembling of different variations. Learned a lot from you here, definitely saving your post :)

", "rawMarkdown": "Congrats! I'm happy to see that you got so much success from ensembling many models. I pushed my best xgboost to ~1103 MAE three weeks ago and didn't know how to make improvements anymore. Really nice to see the effect of ensembling of different variations. Learned a lot from you here, definitely saving your post :) ", "votes": 7, "replies": [ { "id": 150017, "postDate": "2016-12-13T15:24:17.087Z", "content": "

[quote=Qingchen;149948]

\n\n

Congrats! I'm happy to see that you got so much success from ensembling many models. I pushed my best xgboost to ~1103 MAE three weeks ago and didn't know how to make improvements anymore.

\n\n

[/quote]\n1103 is very impressive. Could you please share code, hyperparameters, feature engineering?

", "rawMarkdown": "[quote=Qingchen;149948]\r\n\r\nCongrats! I'm happy to see that you got so much success from ensembling many models. I pushed my best xgboost to ~1103 MAE three weeks ago and didn't know how to make improvements anymore.\r\n\r\n[/quote]\r\n1103 is very impressive. Could you please share code, hyperparameters, feature engineering?", "votes": 3 } ] }, { "id": 149902, "postDate": "2016-12-13T06:01:16.833Z", "content": "

Thank you,\nmuch appreciated!

\n\n

Questions of course:
\n1) Is there an open/commercial pipeline/workflow that could match those results consistently
\n2) Is there an open or commercial pipeline/workflow that could deploy the whole solution consistently

\n\n

Basically, would it be feasible to implement a similar model
\nthat would consistently get good results on your proposed workflow.

\n\n

Surely interesting for commercial/open ML providers for regression problems.

", "rawMarkdown": "Thank you,\r\nmuch appreciated!\r\n\r\nQuestions of course: \r\n1) Is there an open/commercial pipeline/workflow that could **match** those results **consistently** \r\n2) Is there an open or commercial pipeline/workflow that could **deploy** the whole solution consistently \r\n\r\nBasically, would it be feasible to implement a similar model \r\nthat would consistently get good results on your proposed workflow. \r\n\r\nSurely interesting for commercial/open ML providers for regression problems. \r\n\r\n", "votes": 3 }, { "id": 150706, "postDate": "2016-12-16T12:01:13.023Z", "content": "

@Gippy: I didn't use bootstrapping, I simply used oof-data for EarlyStopping.

\n\n

@ Ronin: I have written my own wrapper for VW in python for performing cross-validation and other stuff.

", "rawMarkdown": "@Gippy: I didn't use bootstrapping, I simply used oof-data for EarlyStopping.\r\n\r\n@ Ronin: I have written my own wrapper for VW in python for performing cross-validation and other stuff.", "votes": 1 }, { "id": 150356, "postDate": "2016-12-14T21:50:11.247Z", "content": "

Thank you so much for sharing such a great solution! I have a question about generating out-of-fold predictions from bagged models (e.g. your NN bag) for stacking. I assume you used the same 10-fold split (call it 'a') for all of your base models. When you generate the meta/oof features of your NN models do you do another 10-fold cv (call it 'b') on the training portion of (a), early stop each of the models based on the valid portion of (b), and bag them to make predictions/oof features on the valid fold of (a)? In another word are the meta features you computed via the first 10-fold cv(a) generated by a similar 10-fold 10 times bagged model on the training portions of (a)

\n\n

I wonder if this type of 'nested' cv is necessary for all kinds of bagged models when the individual ones are terminated by the validation set

\n\n

I am not sure of I have made myself clear. Let me know if it is still confusing.

", "rawMarkdown": "Thank you so much for sharing such a great solution! I have a question about generating out-of-fold predictions from bagged models (e.g. your NN bag) for stacking. I assume you used the same 10-fold split (call it 'a') for all of your base models. When you generate the meta/oof features of your NN models do you do another 10-fold cv (call it 'b') on the training portion of (a), early stop each of the models based on the valid portion of (b), and bag them to make predictions/oof features on the valid fold of (a)? In another word are the meta features you computed via the first 10-fold cv(a) generated by a similar 10-fold 10 times bagged model on the training portions of (a)\r\n\r\nI wonder if this type of 'nested' cv is necessary for all kinds of bagged models when the individual ones are terminated by the validation set\r\n\r\nI am not sure of I have made myself clear. Let me know if it is still confusing.", "votes": 1 }, { "id": 149941, "postDate": "2016-12-13T09:05:50.817Z", "content": "

@Engine: I tuned the parameter manually for RGF (RGF, reg_L2 = 0.035, min_pop = 80, max_leaf_forest = 4500). Without interaction it took around 3.5 hrs and with interaction around 7 hours.

\n\n

@Evgeny_Semyonov: I used Ubuntu15.10 for the VW model. I used python to convert the data into vw format and execute the command line from within the script only. For your second query, the idea is when the NN (second layer) predicts low and higher value, I optimized the adjustment parameters differently.

", "rawMarkdown": "@Engine: I tuned the parameter manually for RGF (RGF, reg_L2 = 0.035, min_pop = 80, max_leaf_forest = 4500). Without interaction it took around 3.5 hrs and with interaction around 7 hours.\r\n\r\n@Evgeny_Semyonov: I used Ubuntu15.10 for the VW model. I used python to convert the data into vw format and execute the command line from within the script only. For your second query, the idea is when the NN (second layer) predicts low and higher value, I optimized the adjustment parameters differently.", "votes": 1 }, { "id": 149937, "postDate": "2016-12-13T08:57:48.990Z", "content": "

Congrats Bishwarup. great achievement.

", "rawMarkdown": "Congrats Bishwarup. great achievement.\r\n", "votes": 1 }, { "id": 149920, "postDate": "2016-12-13T07:24:41.397Z", "content": "

Bishwarup Congratulations !
\nThank you for detailing your solution in this way. \nThese algos will certainly end up in my arsenal. \nYou're probably aware but just in case, there's a multi core RGF available here: https://github.com/baidu/fast_rgf\nI still need to test it though.

", "rawMarkdown": "Bishwarup Congratulations ! \r\nThank you for detailing your solution in this way. \r\nThese algos will certainly end up in my arsenal. \r\nYou're probably aware but just in case, there's a multi core RGF available here: https://github.com/baidu/fast_rgf\r\nI still need to test it though.\r\n", "votes": 1 }, { "id": 154664, "postDate": "2017-01-07T14:18:10.450Z", "content": "

@Bishwarup B Thanks for sharing the solution. Can you please share the code as well?

", "rawMarkdown": "@Bishwarup B Thanks for sharing the solution. Can you please share the code as well?\r\n", "votes": 2 }, { "id": 150635, "postDate": "2016-12-16T05:05:14.267Z", "content": "

How did you perform your CV in VW? Did you do it manually? I've always had trouble optimizing VW as the \"scoring\" functions used in competitions tend not to be implemented directly in vw (although the quintile loss function seemed to be working well this competition) as well as having to manually split the data set.

\n\n

I've never had good luck with the VW wrappers, although the program itself is slick as snot once I figured out what I was doing.

", "rawMarkdown": "How did you perform your CV in VW? Did you do it manually? I've always had trouble optimizing VW as the \"scoring\" functions used in competitions tend not to be implemented directly in vw (although the quintile loss function seemed to be working well this competition) as well as having to manually split the data set.\r\n\r\nI've never had good luck with the VW wrappers, although the program itself is slick as snot once I figured out what I was doing.", "votes": 2 }, { "id": 150416, "postDate": "2016-12-15T02:58:56.450Z", "content": "

And for python: https://docs.scipy.org/doc/scipy-0.18.1/reference/optimize.minimize-neldermead.html

", "rawMarkdown": "And for python: [https://docs.scipy.org/doc/scipy-0.18.1/reference/optimize.minimize-neldermead.html][1]\r\n\r\n\r\n [1]: https://docs.scipy.org/doc/scipy-0.18.1/reference/optimize.minimize-neldermead.html", "votes": 2, "replies": [ { "id": 2186320, "postDate": "2023-03-17T17:53:46.860Z", "content": "

This is helpful. Thanks.

", "rawMarkdown": "This is helpful. Thanks. " } ] }, { "id": 149927, "postDate": "2016-12-13T07:44:44.397Z", "content": "

Congratulations! Very cool solution.

\n\n

Questions:

\n\n

1) i tried to use Vowpal Wabbit, but it's terrible on Windows. What's your platform?(Windows/Unix)\nAn did you use it from Python or converted data to Vowpal Wabbit format and start from console?

\n\n

2) \"If NN1 < w1 , then w2*NN1^w3 + w4 Else if NN1 > w5, then w6*NN1^w7 + w8 Else NN1\".

\n\n

NN1 - it's predictions from Neural second level model and w1 - it's weitht (scalar)? How can we compare them?

", "rawMarkdown": "Congratulations! Very cool solution.\r\n\r\nQuestions: \r\n\r\n1) i tried to use Vowpal Wabbit, but it's terrible on Windows. What's your platform?(Windows/Unix)\r\nAn did you use it from Python or converted data to Vowpal Wabbit format and start from console?\r\n\r\n2) \"If NN1 < w1 , then w2*NN1^w3 + w4 Else if NN1 > w5, then w6*NN1^w7 + w8 Else NN1\". \r\n\r\nNN1 - it's predictions from Neural second level model and w1 - it's weitht (scalar)? How can we compare them?", "votes": 2 }, { "id": 149923, "postDate": "2016-12-13T07:37:03.353Z", "content": "

congratulations for the 1st place!\nTell, please, about parameters of RGF (algorithm modification (RGF|RGF Opt|RGF Sib), num of leaves, kind of regularization parameters). How much of time takes for rgf to make prediction? \nThanks for sharing your ideas

", "rawMarkdown": "congratulations for the 1st place!\r\nTell, please, about parameters of RGF (algorithm modification (RGF|RGF Opt|RGF Sib), num of leaves, kind of regularization parameters). How much of time takes for rgf to make prediction? \r\nThanks for sharing your ideas", "votes": 2 }, { "id": 149917, "postDate": "2016-12-13T07:16:37.597Z", "content": "

So this is what it takes. Phew! Congratulations :)

", "rawMarkdown": "So this is what it takes. Phew! Congratulations :)", "votes": 2 }, { "id": 970385, "postDate": "2020-08-14T12:03:20.523Z", "content": "

Thank you, it is really helpful to new guys like me ^^.

", "rawMarkdown": "Thank you, it is really helpful to new guys like me ^^." }, { "id": 261183, "postDate": "2017-12-22T00:21:34.733Z", "content": "

Thanks for sharing. That is really helpful!

", "rawMarkdown": "Thanks for sharing. That is really helpful!" }, { "id": 169239, "postDate": "2017-03-20T08:16:19.507Z", "content": "

Thx for sharing your strategy.

", "rawMarkdown": "Thx for sharing your strategy." }, { "id": 162633, "postDate": "2017-02-20T11:45:17.850Z", "content": "

Many congrats Bishwarup B!!!!. Could you please share your code also.

", "rawMarkdown": "Many congrats Bishwarup B!!!!. Could you please share your code also." }, { "id": 150903, "postDate": "2016-12-17T12:51:55.397Z", "content": "

@Gippy: I didn't change any parameter as such, just the random_seed to make the model more robust as NN is very sentsitive to noise in the data.

", "rawMarkdown": "@Gippy: I didn't change any parameter as such, just the random_seed to make the model more robust as NN is very sentsitive to noise in the data." }, { "id": 150718, "postDate": "2016-12-16T13:30:04.677Z", "content": "

@Bishwarup: Then can you please let me know what do you mean by 10 bags for each fold? What parameter do you change for each bag? Thanks

", "rawMarkdown": "@Bishwarup: Then can you please let me know what do you mean by 10 bags for each fold? What parameter do you change for each bag? Thanks" }, { "id": 150506, "postDate": "2016-12-15T13:38:28.027Z", "content": "

@Bishwarup: 10 fold 10 bags: Does it mean that for every fold, you take bootstrap sample (10 times) and use that for training (if yes, then do you use the oob data for early stopping or oof fold data for early stopping). Or have i misunderstood the approach? Thanks

", "rawMarkdown": "@Bishwarup: 10 fold 10 bags: Does it mean that for every fold, you take bootstrap sample (10 times) and use that for training (if yes, then do you use the oob data for early stopping or oof fold data for early stopping). Or have i misunderstood the approach? Thanks" }, { "id": 150490, "postDate": "2016-12-15T12:22:42.593Z", "content": "

@Achal: I used percentile based binning.

\n\n

@ SRK: Thanks man :)

\n\n

@ deltap: 10 fold 10 bags -> so 100 folds in total -> EarlyStopping in each of the folds -> predict oof and test -> average test for all the 100 runs.

\n\n

@Finaly Liu: I think ClusterFork and Tony already cleared your doubts.

", "rawMarkdown": "@Achal: I used percentile based binning.\r\n\r\n@ SRK: Thanks man :)\r\n\r\n@ deltap: 10 fold 10 bags -> so 100 folds in total -> EarlyStopping in each of the folds -> predict oof and test -> average test for all the 100 runs.\r\n\r\n@Finaly Liu: I think ClusterFork and Tony already cleared your doubts.\r\n" }, { "id": 150406, "postDate": "2016-12-15T02:17:41.223Z", "content": "

optim (Nelder-Mead) , what does this mean?

", "rawMarkdown": "optim (Nelder-Mead) , what does this mean?", "replies": [ { "id": 150414, "postDate": "2016-12-15T02:55:29.700Z", "content": "

[quote=Finlay Liu;150406]

\n\n

optim (Nelder-Mead) , what does this mean?

\n\n

[/quote]\n?optim in R console:

\n\n

optim {stats} R Documentation\nGeneral-purpose Optimization

\n\n

Description

\n\n

General-purpose optimization based on Nelder–Mead, quasi-Newton and conjugate-gradient algorithms. It includes an option for box-constrained optimization and simulated annealing.

", "rawMarkdown": "[quote=Finlay Liu;150406]\r\n\r\noptim (Nelder-Mead) , what does this mean?\r\n\r\n[/quote]\r\n?optim in R console:\r\n\r\noptim {stats}\tR Documentation\r\nGeneral-purpose Optimization\r\n\r\nDescription\r\n\r\nGeneral-purpose optimization based on Nelder–Mead, quasi-Newton and conjugate-gradient algorithms. It includes an option for box-constrained optimization and simulated annealing.\r\n" } ] }, { "id": 150076, "postDate": "2016-12-13T19:14:10.190Z", "content": "
\n

\"I chose the higher order interactions primarily based on mutual\n information\"

\n \n

\" I calculated the mutual information between the interaction features\n and the transformed target. If you are looking for how to calculate\n MI, you can look into the packages like mRMRe or infotheo. In python,\n you can look into the function sklearn.metrics.mutual_info_score. I\n took the interactions which had high MI score.\"

\n
\n\n

Did you calculate mutual information among all categorical features, right? Our loss function is continuous, so how did you calculate MI to calculate interactions? Any readings or link will be helpful.

", "rawMarkdown": "> \"I chose the higher order interactions primarily based on mutual\r\n> information\"\r\n> \r\n> \" I calculated the mutual information between the interaction features\r\n> and the transformed target. If you are looking for how to calculate\r\n> MI, you can look into the packages like mRMRe or infotheo. In python,\r\n> you can look into the function sklearn.metrics.mutual_info_score. I\r\n> took the interactions which had high MI score.\"\r\n\r\n\r\nDid you calculate mutual information among all categorical features, right? Our loss function is continuous, so how did you calculate MI to calculate interactions? Any readings or link will be helpful." }, { "id": 150021, "postDate": "2016-12-13T15:36:33.727Z", "content": "

Congrats Bishwarup! Thanks for the play-by-play.

", "rawMarkdown": "Congrats Bishwarup! Thanks for the play-by-play." }, { "id": 150020, "postDate": "2016-12-13T15:29:45.517Z", "content": "

I have tried to use vowpal and max CV I got was around 1165. Need some tuning tips for vowpal. I also tried to use RGF during last couple of days but not able to complete the training. I found your solution easy to understand as compared to others Multilayer stacking.. and will use some of the tips in near future.

", "rawMarkdown": "I have tried to use vowpal and max CV I got was around 1165. Need some tuning tips for vowpal. I also tried to use RGF during last couple of days but not able to complete the training. I found your solution easy to understand as compared to others Multilayer stacking.. and will use some of the tips in near future." }, { "id": 150001, "postDate": "2016-12-13T14:08:13.037Z", "content": "

Congratulations Bishwarup B.This was my first competition I never even thought about ensembling many models.Thanks for explaining your solution.

", "rawMarkdown": "Congratulations Bishwarup B.This was my first competition I never even thought about ensembling many models.Thanks for explaining your solution." }, { "id": 149918, "postDate": "2016-12-13T07:21:06.597Z", "content": "

Congrats for the 1st Rank and thanks for sharing your approach. I have one question: Can you please explain \"higher order interactions primarily based on mutual information\"? Thanks

", "rawMarkdown": "Congrats for the 1st Rank and thanks for sharing your approach. I have one question: Can you please explain \"higher order interactions primarily based on mutual information\"? Thanks\r\n\r\n" }, { "id": 149910, "postDate": "2016-12-13T06:40:55.803Z", "content": "

Gongrats with #1st place and thank you very much for sharing solution

", "rawMarkdown": "Gongrats with #1st place and thank you very much for sharing solution" }, { "id": 149908, "postDate": "2016-12-13T06:29:33.187Z", "content": "

Bishwarup Congratulations!

\n\n

If you do not mind,May I know number of rounds in your best xgboost model. Mine went beyond 15000 and did not try more than that.

\n\n

Thanks

", "rawMarkdown": "Bishwarup Congratulations!\r\n\r\nIf you do not mind,May I know number of rounds in your best xgboost model. Mine went beyond 15000 and did not try more than that.\r\n\r\nThanks" }, { "id": 149907, "postDate": "2016-12-13T06:28:07.327Z", "content": "

Genius :)

", "rawMarkdown": "Genius :)" }, { "id": 149905, "postDate": "2016-12-13T06:23:39.387Z", "content": "

Congratulations! Awesome work! Btw I am curious about how long it took to run all these models.

", "rawMarkdown": "Congratulations! Awesome work! Btw I am curious about how long it took to run all these models." }, { "id": 149904, "postDate": "2016-12-13T06:11:42.260Z", "content": "

congratulation and Thanks for sharing your solution.

", "rawMarkdown": "congratulation and Thanks for sharing your solution." }, { "id": 150011, "postDate": "2016-12-13T14:53:35.440Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1913465, "postDate": "2022-08-25T10:33:50.040Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing" }, { "id": 150984, "postDate": "2016-12-18T03:15:11.007Z", "content": "

Congrats and Thanks for sharing.

", "rawMarkdown": "Congrats and Thanks for sharing." }, { "id": 150522, "postDate": "2016-12-15T15:05:05.073Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!\r\n\r\n " }, { "id": 149991, "postDate": "2016-12-13T13:20:00.253Z", "content": "

Thank you.

", "rawMarkdown": "Thank you." }, { "id": 149900, "postDate": "2016-12-13T05:50:17.413Z", "content": "

Learn a lot from your solution! Thanks~

", "rawMarkdown": "Learn a lot from your solution! Thanks~" }, { "id": 149897, "postDate": "2016-12-13T05:30:24.663Z", "content": "

Great solution! Thanks!

", "rawMarkdown": "Great solution! Thanks!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:15:48Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 385195, "solution_url": "/competitions/allstate-claims-severity/writeups/bishwarup-b-1st-place-solution" }