It seems the other top 10 had way more base models than I did. So I guess that was probably my mistake!
\n\nI had less than 20 base models (17 to be precise). 10 xgboost + 5 nn + 1 gbm + 1 lightgbm. The best single model is xgboost with 1103 - 1104 public LB.
\n\nI used 5 fold instead of 10 fold to mainly save time training (probably another mistake!). My best single xgboost used eta = 0.0025, with about 13K trees. I would love to hear what others can get with higher eta.
\n\nI found that 0.7 * FAIR + 0.15 * RMSE + 0.15 * MAE is a better loss compared to FAIR alone for xgboost (about 2 MAE better for same hyper parameters).
\n\nI mainly used the raw features + cat interactions. I also had two xgboost and one nn model trained on features where categorical are oof encoded based on loss.
\n\nAnd then one model each (xgboost, nn, gbm, lightgbm, random forest) for second layer.
\n\nFinally, quantile regression on first layer + second layer.
\n\nI didn't really have much \"human time\" for this one, but I did have a new computer with some \"computing time\" so I let my computer run 24/7.
\n\nThanks everybody. The forum discussion here is purely amazing!
", "comments": [ { "id": 150056, "author_name": "Little Boat", "author_url": "", "post_date": "2016-12-13T17:46:02.033000", "content": "def logexpexp(preds, dtrain):
\n\nlabels = dtrain.get_label()\nc = 0.7\nx = preds - labels\ngrad = c * x /(np.abs(x) + c)\nhess = c ** 2 / (np.abs(x) + c) ** 2\n\n# rmse grad and hess\ngrad_rmse = x\nhess_rmse = 1.0\n\n# mae grad and hess\ngrad_mae = np.array(x)\ngrad_mae[grad_mae > 0] = 1.\ngrad_mae[grad_mae <= 0] = -1.\nhess_mae = 1.0\n\n#coef = [0.7, 0.2, 0.1]\ncoef = [0.7, 0.15, 0.15]\nreturn coef[0] * grad + coef[1] * grad_rmse + coef[2] * grad_mae, coef[0] * hess + \\\n coef[1] * hess_rmse + coef[2] * hess_mae\nHey guys, here is the code for the loss function for xgboost.
\n\nIf you used fair as loss function, replace that should give a fair boost, for my side, it was at least 1 MAE (and 2 if fine tuned) better than fair.
\n\n@Vladimir\nI changed features along the way and never looked back :( I was committing things in git in the beginning but then I got lazy and didn't do that for feature engineering. So my feature list part cannot be recovered now but I did nothing fancy but included more categorical features for the two way interactions same as the public script did. I remembered I had about 1300 features or so.\nThis is my last list (which was not the best sadly)
\n\n #COMB_FEATURE = ['cat%i' % i for i in xrange(97, 117)]\n COMB_FEATURE = ['cat%i' % i for i in [1,5,9,17,25,31,39,45,55,63,77, 105, 101,102,103,104]]\n COMB_FEATURE += 'cat80,cat87,cat57,cat12,cat79,cat10,cat7,cat89,cat2,cat72,cat81,cat11,cat1,cat13,cat9,cat3,cat16,cat90,cat23,cat36,cat73,cat103,cat40,cat28,cat111,cat6,cat76,cat50,cat5,cat4,cat14,cat38,cat24,cat82,cat25'.split(',')\n COMB_FEATURE = list(set(COMB_FEATURE))\nBut here is the hyperparameter set.
\n\neta = 0.0025\nalpha = 3.2364\ncolsample_bylevel = 1.\ncolsample_bytree = 0.18\ngamma = 0.\nlamb = 0.\nmax_depth = 10\nmin_child_weight = 180\nsubsample = 1\nnum_boost_round = 13666
\n\nHope it helps.
", "votes": 5, "replies": [] }, { "id": 150058, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T17:49:30.987000", "content": "Thank you.
\n\nHow did you choose your hyperparameters?
\n\nLooks like intelligent manual tuning, except alpha = 3.2364
", "votes": 1, "replies": [ { "id": 150063, "author_name": "Little Boat", "author_url": "", "post_date": "2016-12-13T17:56:09.693000", "content": "[quote=Vladimir Iglovikov;150058]
\n\nThank you.
\n\nHow did you choose your hyperparameters?
\n\nLooks like intelligent manual tuning, except alpha = 3.2364
\n\n[/quote]
\n\nI used bayesian optimization, several times with eta = 0.05. \nThat is, I ran it several rounds. Each round had (inital = 10, iteration = 100). And I picked the best one out of them, and then reduced eta to eta/2 while grid searching other \"important parameters\" (max_depth, subsample, and colsample_bytree) within their neighbours.
\n\nAnd when eta is reduced to 0.0025, I stopped mainly because of the running time. And then I tweaked a little with other parameters (alpha, lambda, gamma), and I found that alpha was the best to stay as is (that is why it is this weird number).
", "votes": 4, "replies": [] } ] }, { "id": 150047, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T17:20:13.443000", "content": "@Little Boat, Could you please share your hyperparameters, feature engineering for 1103 XGB?
", "votes": 2, "replies": [] }, { "id": 150062, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T17:54:56.127000", "content": "Somehow it worked for me similar to Randomized Grid Search
\n\n[quote=Vladimir Iglovikov;150062]
\n\nSomehow it worked for me similar to Randomized Grid Search
\n\n[/quote]
\n\nI started with fairly big range for each parameter (say, max_depth in (2, 20), and gamma in (0, 10)).
\n\nI passed all parameters in it (which now I think about it, might not be the best idea, since it will take forever to converge). I haven't tried but I think maybe pass in the parameters that are more important (max_depth, subsample and colsample_bytree) and then reduce the range for them, include those not so important ones and then optimize again, might save some time.
\n\nI used 5 fold, and I found that the 2nd fold had about the same MAE score as the cv, so I used that fold only to save time.
\n\nYes, I started with 0.05 and then reduced by a factor of 2 each time while doing grid search like I described above.
\n\nI also didn't find bayesian optimization optimal for xgboost, mainly probably because of the dependency between parameters, so some grid search might be needed somewhere in the loop.
", "votes": 1, "replies": [] } ] }, { "id": 150054, "author_name": "Marcus Luck", "author_url": "", "post_date": "2016-12-13T17:42:55.017000", "content": "Thanks for sharing! I also feel like I don't have enough human time but plenty of computer time, do you have any specific packages you recommend for finding hyperparameters ?
", "votes": 0, "replies": [ { "id": 150059, "author_name": "Little Boat", "author_url": "", "post_date": "2016-12-13T17:49:45.927000", "content": "[quote=Marcus Luck;150054]
\n\nThanks for sharing! I also feel like I don't have enough human time but plenty of computer time, do you have any specific packages you recommend for finding hyperparameters ?
\n\n[/quote]
\n\nI just used bayesian optimization python package.
", "votes": 2, "replies": [] } ] }, { "id": 150051, "author_name": "Nirupam Kar", "author_url": "", "post_date": "2016-12-13T17:31:30.597000", "content": "How did you get the weights for loss? I did kind of similar thing, however went back to Fair. The issue for me was to find proper weight . As MSE will always be higher than MAP (if abs(loss)>1, which is true in most of the cases), so the final model will be more aligned to mse. Another issue I faced, not selecting proper weight leading to divergence .. \nThanks for reply in advance!!
", "votes": 0, "replies": [] }, { "id": 150046, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2016-12-13T17:17:57.607000", "content": "congrats! only 20 base models has to be a record for top finishers.\ncould you elaborate on your objective function? code? thanks
\n\np.s. I am a slow typist :)
", "votes": 0, "replies": [] }, { "id": 150042, "author_name": "zhen", "author_url": "", "post_date": "2016-12-13T17:14:13.220000", "content": "@Little Boat, thanks for sharing your solution! Could you be more specific on \"0.7 * FAIR + 0.15 * RMSE + 0.15 * MAE\"? This might be a silly question. Did you create a loss function combining the three? If so, did you use such linear combination for grad and hess as the output? Thank you!
", "votes": 0, "replies": [] }, { "id": 150057, "author_name": "Joe", "author_url": "", "post_date": "2016-12-13T17:49:03.820000", "content": "Thanks! Very interesting.
", "votes": 0, "replies": [] } ], "comment_count": 12, "messages": [ { "id": 150036, "postDate": "2016-12-13T16:47:30.617Z", "content": "It seems the other top 10 had way more base models than I did. So I guess that was probably my mistake!
\n\nI had less than 20 base models (17 to be precise). 10 xgboost + 5 nn + 1 gbm + 1 lightgbm. The best single model is xgboost with 1103 - 1104 public LB.
\n\nI used 5 fold instead of 10 fold to mainly save time training (probably another mistake!). My best single xgboost used eta = 0.0025, with about 13K trees. I would love to hear what others can get with higher eta.
\n\nI found that 0.7 * FAIR + 0.15 * RMSE + 0.15 * MAE is a better loss compared to FAIR alone for xgboost (about 2 MAE better for same hyper parameters).
\n\nI mainly used the raw features + cat interactions. I also had two xgboost and one nn model trained on features where categorical are oof encoded based on loss.
\n\nAnd then one model each (xgboost, nn, gbm, lightgbm, random forest) for second layer.
\n\nFinally, quantile regression on first layer + second layer.
\n\nI didn't really have much \"human time\" for this one, but I did have a new computer with some \"computing time\" so I let my computer run 24/7.
\n\nThanks everybody. The forum discussion here is purely amazing!
", "rawMarkdown": "It seems the other top 10 had way more base models than I did. So I guess that was probably my mistake!\r\n\r\nI had less than 20 base models (17 to be precise). 10 xgboost + 5 nn + 1 gbm + 1 lightgbm. The best single model is xgboost with 1103 - 1104 public LB.\r\n\r\nI used 5 fold instead of 10 fold to mainly save time training (probably another mistake!). My best single xgboost used eta = 0.0025, with about 13K trees. I would love to hear what others can get with higher eta. \r\n\r\nI found that 0.7 * FAIR + 0.15 * RMSE + 0.15 * MAE is a better loss compared to FAIR alone for xgboost (about 2 MAE better for same hyper parameters).\r\n\r\nI mainly used the raw features + cat interactions. I also had two xgboost and one nn model trained on features where categorical are oof encoded based on loss.\r\n\r\nAnd then one model each (xgboost, nn, gbm, lightgbm, random forest) for second layer.\r\n\r\nFinally, quantile regression on first layer + second layer.\r\n\r\n\r\nI didn't really have much \"human time\" for this one, but I did have a new computer with some \"computing time\" so I let my computer run 24/7.\r\n\r\nThanks everybody. The forum discussion here is purely amazing!", "votes": 21 }, { "id": 150056, "postDate": "2016-12-13T17:46:02.033Z", "content": "def logexpexp(preds, dtrain):
\n\nlabels = dtrain.get_label()\nc = 0.7\nx = preds - labels\ngrad = c * x /(np.abs(x) + c)\nhess = c ** 2 / (np.abs(x) + c) ** 2\n\n# rmse grad and hess\ngrad_rmse = x\nhess_rmse = 1.0\n\n# mae grad and hess\ngrad_mae = np.array(x)\ngrad_mae[grad_mae > 0] = 1.\ngrad_mae[grad_mae <= 0] = -1.\nhess_mae = 1.0\n\n#coef = [0.7, 0.2, 0.1]\ncoef = [0.7, 0.15, 0.15]\nreturn coef[0] * grad + coef[1] * grad_rmse + coef[2] * grad_mae, coef[0] * hess + \\\n coef[1] * hess_rmse + coef[2] * hess_mae\nHey guys, here is the code for the loss function for xgboost.
\n\nIf you used fair as loss function, replace that should give a fair boost, for my side, it was at least 1 MAE (and 2 if fine tuned) better than fair.
\n\n@Vladimir\nI changed features along the way and never looked back :( I was committing things in git in the beginning but then I got lazy and didn't do that for feature engineering. So my feature list part cannot be recovered now but I did nothing fancy but included more categorical features for the two way interactions same as the public script did. I remembered I had about 1300 features or so.\nThis is my last list (which was not the best sadly)
\n\n #COMB_FEATURE = ['cat%i' % i for i in xrange(97, 117)]\n COMB_FEATURE = ['cat%i' % i for i in [1,5,9,17,25,31,39,45,55,63,77, 105, 101,102,103,104]]\n COMB_FEATURE += 'cat80,cat87,cat57,cat12,cat79,cat10,cat7,cat89,cat2,cat72,cat81,cat11,cat1,cat13,cat9,cat3,cat16,cat90,cat23,cat36,cat73,cat103,cat40,cat28,cat111,cat6,cat76,cat50,cat5,cat4,cat14,cat38,cat24,cat82,cat25'.split(',')\n COMB_FEATURE = list(set(COMB_FEATURE))\nBut here is the hyperparameter set.
\n\neta = 0.0025\nalpha = 3.2364\ncolsample_bylevel = 1.\ncolsample_bytree = 0.18\ngamma = 0.\nlamb = 0.\nmax_depth = 10\nmin_child_weight = 180\nsubsample = 1\nnum_boost_round = 13666
\n\nHope it helps.
", "rawMarkdown": "def logexpexp(preds, dtrain):\r\n\r\n labels = dtrain.get_label()\r\n c = 0.7\r\n x = preds - labels\r\n grad = c * x /(np.abs(x) + c)\r\n hess = c ** 2 / (np.abs(x) + c) ** 2\r\n\r\n # rmse grad and hess\r\n grad_rmse = x\r\n hess_rmse = 1.0\r\n\r\n # mae grad and hess\r\n grad_mae = np.array(x)\r\n grad_mae[grad_mae > 0] = 1.\r\n grad_mae[grad_mae <= 0] = -1.\r\n hess_mae = 1.0\r\n\r\n #coef = [0.7, 0.2, 0.1]\r\n coef = [0.7, 0.15, 0.15]\r\n return coef[0] * grad + coef[1] * grad_rmse + coef[2] * grad_mae, coef[0] * hess + \\\r\n coef[1] * hess_rmse + coef[2] * hess_mae\r\nHey guys, here is the code for the loss function for xgboost.\r\n\r\nIf you used fair as loss function, replace that should give a fair boost, for my side, it was at least 1 MAE (and 2 if fine tuned) better than fair. \r\n\r\n@Vladimir\r\nI changed features along the way and never looked back :( I was committing things in git in the beginning but then I got lazy and didn't do that for feature engineering. So my feature list part cannot be recovered now but I did nothing fancy but included more categorical features for the two way interactions same as the public script did. I remembered I had about 1300 features or so.\r\nThis is my last list (which was not the best sadly)\r\n\r\n #COMB_FEATURE = ['cat%i' % i for i in xrange(97, 117)]\r\n COMB_FEATURE = ['cat%i' % i for i in [1,5,9,17,25,31,39,45,55,63,77, 105, 101,102,103,104]]\r\n COMB_FEATURE += 'cat80,cat87,cat57,cat12,cat79,cat10,cat7,cat89,cat2,cat72,cat81,cat11,cat1,cat13,cat9,cat3,cat16,cat90,cat23,cat36,cat73,cat103,cat40,cat28,cat111,cat6,cat76,cat50,cat5,cat4,cat14,cat38,cat24,cat82,cat25'.split(',')\r\n COMB_FEATURE = list(set(COMB_FEATURE))\r\n\r\n\r\n But here is the hyperparameter set.\r\n\r\neta = 0.0025\r\nalpha = 3.2364\r\ncolsample_bylevel = 1.\r\ncolsample_bytree = 0.18\r\ngamma = 0.\r\nlamb = 0.\r\nmax_depth = 10\r\nmin_child_weight = 180\r\nsubsample = 1\r\nnum_boost_round = 13666\r\n\r\n\r\nHope it helps.", "votes": 5 }, { "id": 150058, "postDate": "2016-12-13T17:49:30.987Z", "content": "Thank you.
\n\nHow did you choose your hyperparameters?
\n\nLooks like intelligent manual tuning, except alpha = 3.2364
", "rawMarkdown": "Thank you. \r\n\r\nHow did you choose your hyperparameters? \r\n\r\nLooks like intelligent manual tuning, except *alpha = 3.2364*", "votes": 1, "replies": [ { "id": 150063, "postDate": "2016-12-13T17:56:09.693Z", "content": "[quote=Vladimir Iglovikov;150058]
\n\nThank you.
\n\nHow did you choose your hyperparameters?
\n\nLooks like intelligent manual tuning, except alpha = 3.2364
\n\n[/quote]
\n\nI used bayesian optimization, several times with eta = 0.05. \nThat is, I ran it several rounds. Each round had (inital = 10, iteration = 100). And I picked the best one out of them, and then reduced eta to eta/2 while grid searching other \"important parameters\" (max_depth, subsample, and colsample_bytree) within their neighbours.
\n\nAnd when eta is reduced to 0.0025, I stopped mainly because of the running time. And then I tweaked a little with other parameters (alpha, lambda, gamma), and I found that alpha was the best to stay as is (that is why it is this weird number).
", "rawMarkdown": "[quote=Vladimir Iglovikov;150058]\r\n\r\nThank you. \r\n\r\nHow did you choose your hyperparameters? \r\n\r\nLooks like intelligent manual tuning, except *alpha = 3.2364*\r\n\r\n[/quote]\r\n\r\nI used bayesian optimization, several times with eta = 0.05. \r\nThat is, I ran it several rounds. Each round had (inital = 10, iteration = 100). And I picked the best one out of them, and then reduced eta to eta/2 while grid searching other \"important parameters\" (max_depth, subsample, and colsample_bytree) within their neighbours.\r\n\r\nAnd when eta is reduced to 0.0025, I stopped mainly because of the running time. And then I tweaked a little with other parameters (alpha, lambda, gamma), and I found that alpha was the best to stay as is (that is why it is this weird number).", "votes": 4 } ] }, { "id": 150047, "postDate": "2016-12-13T17:20:13.443Z", "content": "@Little Boat, Could you please share your hyperparameters, feature engineering for 1103 XGB?
", "rawMarkdown": "@Little Boat, Could you please share your hyperparameters, feature engineering for 1103 XGB?", "votes": 2 }, { "id": 150062, "postDate": "2016-12-13T17:54:56.127Z", "content": "Somehow it worked for me similar to Randomized Grid Search
\n\n[quote=Vladimir Iglovikov;150062]
\n\nSomehow it worked for me similar to Randomized Grid Search
\n\n[/quote]
\n\nI started with fairly big range for each parameter (say, max_depth in (2, 20), and gamma in (0, 10)).
\n\nI passed all parameters in it (which now I think about it, might not be the best idea, since it will take forever to converge). I haven't tried but I think maybe pass in the parameters that are more important (max_depth, subsample and colsample_bytree) and then reduce the range for them, include those not so important ones and then optimize again, might save some time.
\n\nI used 5 fold, and I found that the 2nd fold had about the same MAE score as the cv, so I used that fold only to save time.
\n\nYes, I started with 0.05 and then reduced by a factor of 2 each time while doing grid search like I described above.
\n\nI also didn't find bayesian optimization optimal for xgboost, mainly probably because of the dependency between parameters, so some grid search might be needed somewhere in the loop.
", "rawMarkdown": "[quote=Vladimir Iglovikov;150062]\r\n\r\nSomehow it worked for me similar to Randomized Grid Search\r\n\r\n - How did you define range of the hyper parameters to explore? \r\n - What parameters did you pass into the maximize function?\r\n - Did you use holdout set or N fold CV?\r\n - Did you explore with high eta and made it smaller later? If yes, what was your initial value for exploration?\r\n\r\n\r\n[/quote]\r\n\r\nI started with fairly big range for each parameter (say, max_depth in (2, 20), and gamma in (0, 10)). \r\n\r\nI passed all parameters in it (which now I think about it, might not be the best idea, since it will take forever to converge). I haven't tried but I think maybe pass in the parameters that are more important (max_depth, subsample and colsample_bytree) and then reduce the range for them, include those not so important ones and then optimize again, might save some time.\r\n\r\nI used 5 fold, and I found that the 2nd fold had about the same MAE score as the cv, so I used that fold only to save time.\r\n\r\nYes, I started with 0.05 and then reduced by a factor of 2 each time while doing grid search like I described above.\r\n\r\nI also didn't find bayesian optimization optimal for xgboost, mainly probably because of the dependency between parameters, so some grid search might be needed somewhere in the loop.", "votes": 1 } ] }, { "id": 150054, "postDate": "2016-12-13T17:42:55.017Z", "content": "Thanks for sharing! I also feel like I don't have enough human time but plenty of computer time, do you have any specific packages you recommend for finding hyperparameters ?
", "rawMarkdown": "Thanks for sharing! I also feel like I don't have enough human time but plenty of computer time, do you have any specific packages you recommend for finding hyperparameters ?", "replies": [ { "id": 150059, "postDate": "2016-12-13T17:49:45.927Z", "content": "[quote=Marcus Luck;150054]
\n\nThanks for sharing! I also feel like I don't have enough human time but plenty of computer time, do you have any specific packages you recommend for finding hyperparameters ?
\n\n[/quote]
\n\nI just used bayesian optimization python package.
", "rawMarkdown": "[quote=Marcus Luck;150054]\r\n\r\nThanks for sharing! I also feel like I don't have enough human time but plenty of computer time, do you have any specific packages you recommend for finding hyperparameters ?\r\n\r\n[/quote]\r\n\r\nI just used bayesian optimization python package.", "votes": 2 } ] }, { "id": 150051, "postDate": "2016-12-13T17:31:30.597Z", "content": "How did you get the weights for loss? I did kind of similar thing, however went back to Fair. The issue for me was to find proper weight . As MSE will always be higher than MAP (if abs(loss)>1, which is true in most of the cases), so the final model will be more aligned to mse. Another issue I faced, not selecting proper weight leading to divergence .. \nThanks for reply in advance!!
", "rawMarkdown": "How did you get the weights for loss? I did kind of similar thing, however went back to Fair. The issue for me was to find proper weight . As MSE will always be higher than MAP (if abs(loss)>1, which is true in most of the cases), so the final model will be more aligned to mse. Another issue I faced, not selecting proper weight leading to divergence .. \r\nThanks for reply in advance!!" }, { "id": 150046, "postDate": "2016-12-13T17:17:57.607Z", "content": "congrats! only 20 base models has to be a record for top finishers.\ncould you elaborate on your objective function? code? thanks
\n\np.s. I am a slow typist :)
", "rawMarkdown": "congrats! only 20 base models has to be a record for top finishers.\r\ncould you elaborate on your objective function? code? thanks\r\n\r\np.s. I am a slow typist :)" }, { "id": 150042, "postDate": "2016-12-13T17:14:13.220Z", "content": "@Little Boat, thanks for sharing your solution! Could you be more specific on \"0.7 * FAIR + 0.15 * RMSE + 0.15 * MAE\"? This might be a silly question. Did you create a loss function combining the three? If so, did you use such linear combination for grad and hess as the output? Thank you!
", "rawMarkdown": "@Little Boat, thanks for sharing your solution! Could you be more specific on \"0.7 * FAIR + 0.15 * RMSE + 0.15 * MAE\"? This might be a silly question. Did you create a loss function combining the three? If so, did you use such linear combination for grad and hess as the output? Thank you!" }, { "id": 150057, "postDate": "2016-12-13T17:49:03.820Z", "content": "Thanks! Very interesting.
", "rawMarkdown": "Thanks! Very interesting." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:15:58Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 387073, "solution_url": "/competitions/allstate-claims-severity/writeups/goosify-8-solution" }