I had to confess that I was shocked. In fact, my best XGB is a bit better than the solution of Alexey, who's 2nd in this competition, and my NN is a bit worse. Below are the comparisons:
\n\nMy solution:
\n\nBest XGB - 1124.82374149 CV, 1103.76376 Public LB, 1115.89595 Private LB
\n\nBest NN -1127.59973367 CV, 1112.55274 Public LB, 1122.06610 Private LB
\n\nAlexey's:
\n\nBest XGB - 1122.64977 CV, 1105.01969 Public LB, 1116.74215 Private LB
\n\nBest NN - 1130.29286 CV, 1111.90884 Public LB, 1122.06610 Private LB
\n\nI did have 10+ XGBoost, 10+ LightGBM and approx. 20+ NN blended with out of fold predictions (10 folds).
\n\nNo idea what happened. Can anyone shed some light into this? Much appreciated.
\n\nBTW, for L2, I was using Ridge. Could this be the issue?
", "comments": [ { "id": 150092, "author_name": "Alexey Noskov", "author_url": "", "post_date": "2016-12-13T20:34:04.127000", "content": "I think what there may be several possible reasons.
\n\nFirst of all, my L2+L3 is a lot more sophisticated when just linear model. As an example - one of my L2 models (which is Keras NN) has CV 1116.84297, public LB 1098.97434 and private LB 1111.21766. Last L3 model has CV 1115.79637, public LB 1098.17233 and private LB 1110.14853. And my best submission (which is average of others have public LB 1098.07061).
\n\nSo, it's not enough to just build good L1 models, you need to combine their predictions effectively.
\n\nSecond, Ridge for L2 is not best idea, I think, as it's optimizes MSE which is far from our target metric.
\n\nThird, I suppose, you've used it with some target transformation, like log(y + 200) or (y + 1) ^ 0.25, isn't it?
As follows from my tests, such models are good when predicting low loss values, but fail predicting large ones. For example, I've computed MAE by loss quantiles for two of my L2 models, both having similar CV MAE (around 1118.4).
\n\nFirst is XGB with target transformation:
\n\n0 627.941580\n1 547.817976\n2 681.354639\n3 1110.023789\n4 2625.146912\nSecond is XGB with raw target:
\n\n0 646.347681\n1 567.701967\n2 699.029329\n3 1116.448895\n4 2562.774978\nSee the pattern? Model with raw target is better at predicting large loss values, and it's just one bucket (20% of the data), and it's worse in other 80%, but it's enough to make average MAE equal!
\n\nSo these large values (which were called \"outliers\" in some forum posts) bring significant risk - if private test set have just slightly more of them then public, then it will be enough for second model to outperform the first.
\n\nIf we suppose what test target distribution is similar to public and take into account what public LB scores are better when CV (which is effect most of us observed, I think) then it's pretty possible what private LB may really have slightly more large loss values and so we need to take care of this risk. It was discussed in \"shakeup topic\".
\n\nAnd I think, it was partly realized, that's why your Ridge model failed so badly.
", "votes": 3, "replies": [] }, { "id": 150082, "author_name": "Qingchen", "author_url": "", "post_date": "2016-12-13T19:47:47.120000", "content": "I think there's a significant amount of public LB overfitting going on. I just made a submission of a xgboost prediction that scored 1103.5 on the public LB and is 1117.01201 on the private LB. My guess is that ensembling of many different variations of xgboost helped Alexey hedge against overfitting. When we see the individual submission results we'll get a clearer answer unless someone wants to resubmit everything :)
", "votes": 1, "replies": [ { "id": 150084, "author_name": "ChrisCC", "author_url": "", "post_date": "2016-12-13T19:54:34.120000", "content": "[quote=Qingchen;150082]
\n\nI think there's a significant amount of public LB overfitting going on. I just made a submission of a xgboost prediction that scored 1103.5 on the public LB and is 1117.01201 on the private LB. My guess is that ensembling of many different variations of xgboost helped Alexey hedge against overfitting. When we see the individual submission results we'll get a clearer answer unless someone wants to resubmit everything :)
\n\n[/quote]
\n\nThank you QIngchen for your comments. I understood I could have been overfitting the public board. However, my best single XGBoost performed consistently on both public and private boards, as well as the NN model. What drove me crazy was after ensembling they didn't synergize at all which has been working fine for my previous competitions.
", "votes": 0, "replies": [] } ] }, { "id": 150133, "author_name": "Tilii", "author_url": "", "post_date": "2016-12-14T00:09:15.053000", "content": "[quote=ChrisCC;150080]
\n\nNo idea what happened. Can anyone shed some light into this? Much appreciated.
\n\nBTW, for L2, I was using Ridge. Could this be the issue?
\n\n[/quote]
\n\nI don't know if this will help you, but I will share several observations based on my scores and on published scores of others.
\n\nA gap between CV and public LB scores for XGBoost models was 18-22 points (yours are in that category). A gap between CV and public LB scores for Keras models was 20-28 points (yours are decidedly not in that category). A gap between CV and public LB scores for 2nd-level models was 17-18 points (or ~5 points between the best individual model and the final public LB score). Although I don't have your 2nd-level model scores, you don't seem to be in this category as your single best model alone (~1104) should have yielded ~1099 on public LB. So something wrong happened while you were making 2nd level models.
\n\nIt is tough to do forensics with limited data, but your undoing could have been in using 20 NN models. If your best-model gaps are any indication (as they do not align with general trends), you had 20 NN models with ~15 points between CV and LB, and 10 XGBoost models with ~21 points between CV and LB.
\n\nTo use your best models as an example: your XGBoost model is ~9 points better than NN on LB, but only ~3 points better based on CV. Since 2nd-level predictor uses local CV scores reflected in OOF files, it was basically mixing the data from two groups of models that didn't seem very different, even though they were. Given 20:10 ratio between quite a bit worse model (NN) and quite a bit better model (XGBoost) that seemed similar in terms of local CV, my guess is that NN models weighed more heavily in your solution than they should have.
\n\nI suggest you trim down the number of NN models to 5-10, and pick not necessarily the ones with best LB scores but the ones with (CV-LB) gaps more similar to those of XGBoost models. I'd be surprised if that doesn't improve your ensemble score.
", "votes": 2, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 150092, "postDate": "2016-12-13T20:34:04.127Z", "content": "I think what there may be several possible reasons.
\n\nFirst of all, my L2+L3 is a lot more sophisticated when just linear model. As an example - one of my L2 models (which is Keras NN) has CV 1116.84297, public LB 1098.97434 and private LB 1111.21766. Last L3 model has CV 1115.79637, public LB 1098.17233 and private LB 1110.14853. And my best submission (which is average of others have public LB 1098.07061).
\n\nSo, it's not enough to just build good L1 models, you need to combine their predictions effectively.
\n\nSecond, Ridge for L2 is not best idea, I think, as it's optimizes MSE which is far from our target metric.
\n\nThird, I suppose, you've used it with some target transformation, like log(y + 200) or (y + 1) ^ 0.25, isn't it?
As follows from my tests, such models are good when predicting low loss values, but fail predicting large ones. For example, I've computed MAE by loss quantiles for two of my L2 models, both having similar CV MAE (around 1118.4).
\n\nFirst is XGB with target transformation:
\n\n0 627.941580\n1 547.817976\n2 681.354639\n3 1110.023789\n4 2625.146912\nSecond is XGB with raw target:
\n\n0 646.347681\n1 567.701967\n2 699.029329\n3 1116.448895\n4 2562.774978\nSee the pattern? Model with raw target is better at predicting large loss values, and it's just one bucket (20% of the data), and it's worse in other 80%, but it's enough to make average MAE equal!
\n\nSo these large values (which were called \"outliers\" in some forum posts) bring significant risk - if private test set have just slightly more of them then public, then it will be enough for second model to outperform the first.
\n\nIf we suppose what test target distribution is similar to public and take into account what public LB scores are better when CV (which is effect most of us observed, I think) then it's pretty possible what private LB may really have slightly more large loss values and so we need to take care of this risk. It was discussed in \"shakeup topic\".
\n\nAnd I think, it was partly realized, that's why your Ridge model failed so badly.
", "rawMarkdown": "I think what there may be several possible reasons.\r\n\r\n First of all, my L2+L3 is a lot more sophisticated when just linear model. As an example - one of my L2 models (which is Keras NN) has CV 1116.84297, public LB 1098.97434 and private LB 1111.21766. Last L3 model has CV 1115.79637, public LB 1098.17233 and private LB 1110.14853. And my best submission (which is average of others have public LB 1098.07061). \r\n\r\nSo, it's not enough to just build good L1 models, you need to combine their predictions effectively.\r\n\r\nSecond, Ridge for L2 is not best idea, I think, as it's optimizes MSE which is far from our target metric.\r\n\r\nThird, I suppose, you've used it with some target transformation, like `log(y + 200)` or `(y + 1) ^ 0.25`, isn't it?\r\n\r\nAs follows from my tests, such models are good when predicting low loss values, but fail predicting large ones. For example, I've computed MAE by loss quantiles for two of my L2 models, both having similar CV MAE (around 1118.4).\r\n\r\nFirst is XGB with target transformation:\r\n\r\n 0 627.941580\r\n 1 547.817976\r\n 2 681.354639\r\n 3 1110.023789\r\n 4 2625.146912\r\n\r\nSecond is XGB with raw target:\r\n\r\n 0 646.347681\r\n 1 567.701967\r\n 2 699.029329\r\n 3 1116.448895\r\n 4 2562.774978\r\n\r\nSee the pattern? Model with raw target is better at predicting large loss values, and it's just one bucket (20% of the data), and it's worse in other 80%, but it's enough to make average MAE equal! \r\n\r\nSo these large values (which were called \"outliers\" in some forum posts) bring significant risk - if private test set have just slightly more of them then public, then it will be enough for second model to outperform the first.\r\n\r\nIf we suppose what test target distribution is similar to public and take into account what public LB scores are better when CV (which is effect most of us observed, I think) then it's pretty possible what private LB may really have slightly more large loss values and so we need to take care of this risk. It was discussed in \"[shakeup topic][1]\".\r\n\r\nAnd I think, it was partly realized, that's why your Ridge model failed so badly.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/allstate-claims-severity/forums/t/26271/time-to-discuss-shake-up", "votes": 3 }, { "id": 150082, "postDate": "2016-12-13T19:47:47.120Z", "content": "I think there's a significant amount of public LB overfitting going on. I just made a submission of a xgboost prediction that scored 1103.5 on the public LB and is 1117.01201 on the private LB. My guess is that ensembling of many different variations of xgboost helped Alexey hedge against overfitting. When we see the individual submission results we'll get a clearer answer unless someone wants to resubmit everything :)
", "rawMarkdown": "I think there's a significant amount of public LB overfitting going on. I just made a submission of a xgboost prediction that scored 1103.5 on the public LB and is 1117.01201 on the private LB. My guess is that ensembling of many different variations of xgboost helped Alexey hedge against overfitting. When we see the individual submission results we'll get a clearer answer unless someone wants to resubmit everything :) ", "votes": 1, "replies": [ { "id": 150084, "postDate": "2016-12-13T19:54:34.120Z", "content": "[quote=Qingchen;150082]
\n\nI think there's a significant amount of public LB overfitting going on. I just made a submission of a xgboost prediction that scored 1103.5 on the public LB and is 1117.01201 on the private LB. My guess is that ensembling of many different variations of xgboost helped Alexey hedge against overfitting. When we see the individual submission results we'll get a clearer answer unless someone wants to resubmit everything :)
\n\n[/quote]
\n\nThank you QIngchen for your comments. I understood I could have been overfitting the public board. However, my best single XGBoost performed consistently on both public and private boards, as well as the NN model. What drove me crazy was after ensembling they didn't synergize at all which has been working fine for my previous competitions.
", "rawMarkdown": "[quote=Qingchen;150082]\r\n\r\nI think there's a significant amount of public LB overfitting going on. I just made a submission of a xgboost prediction that scored 1103.5 on the public LB and is 1117.01201 on the private LB. My guess is that ensembling of many different variations of xgboost helped Alexey hedge against overfitting. When we see the individual submission results we'll get a clearer answer unless someone wants to resubmit everything :) \r\n\r\n[/quote]\r\n\r\nThank you QIngchen for your comments. I understood I could have been overfitting the public board. However, my best single XGBoost performed consistently on both public and private boards, as well as the NN model. What drove me crazy was after ensembling they didn't synergize at all which has been working fine for my previous competitions.\r\n" } ] }, { "id": 150133, "postDate": "2016-12-14T00:09:15.053Z", "content": "[quote=ChrisCC;150080]
\n\nNo idea what happened. Can anyone shed some light into this? Much appreciated.
\n\nBTW, for L2, I was using Ridge. Could this be the issue?
\n\n[/quote]
\n\nI don't know if this will help you, but I will share several observations based on my scores and on published scores of others.
\n\nA gap between CV and public LB scores for XGBoost models was 18-22 points (yours are in that category). A gap between CV and public LB scores for Keras models was 20-28 points (yours are decidedly not in that category). A gap between CV and public LB scores for 2nd-level models was 17-18 points (or ~5 points between the best individual model and the final public LB score). Although I don't have your 2nd-level model scores, you don't seem to be in this category as your single best model alone (~1104) should have yielded ~1099 on public LB. So something wrong happened while you were making 2nd level models.
\n\nIt is tough to do forensics with limited data, but your undoing could have been in using 20 NN models. If your best-model gaps are any indication (as they do not align with general trends), you had 20 NN models with ~15 points between CV and LB, and 10 XGBoost models with ~21 points between CV and LB.
\n\nTo use your best models as an example: your XGBoost model is ~9 points better than NN on LB, but only ~3 points better based on CV. Since 2nd-level predictor uses local CV scores reflected in OOF files, it was basically mixing the data from two groups of models that didn't seem very different, even though they were. Given 20:10 ratio between quite a bit worse model (NN) and quite a bit better model (XGBoost) that seemed similar in terms of local CV, my guess is that NN models weighed more heavily in your solution than they should have.
\n\nI suggest you trim down the number of NN models to 5-10, and pick not necessarily the ones with best LB scores but the ones with (CV-LB) gaps more similar to those of XGBoost models. I'd be surprised if that doesn't improve your ensemble score.
", "rawMarkdown": "[quote=ChrisCC;150080]\r\n\r\nNo idea what happened. Can anyone shed some light into this? Much appreciated.\r\n\r\nBTW, for L2, I was using Ridge. Could this be the issue?\r\n\r\n[/quote]\r\n\r\nI don't know if this will help you, but I will share several observations based on my scores and on published scores of others.\r\n\r\nA gap between CV and public LB scores for XGBoost models was 18-22 points (yours are in that category). A gap between CV and public LB scores for Keras models was 20-28 points (yours are __decidedly not__ in that category). A gap between CV and public LB scores for 2nd-level models was 17-18 points (or ~5 points between the best individual model and the final public LB score). Although I don't have your 2nd-level model scores, you don't seem to be in this category as your single best model alone (~1104) should have yielded ~1099 on public LB. So something wrong happened while you were making 2nd level models.\r\n\r\nIt is tough to do forensics with limited data, but your undoing could have been in using 20 NN models. If your best-model gaps are any indication (as they do not align with general trends), you had 20 NN models with ~15 points between CV and LB, and 10 XGBoost models with ~21 points between CV and LB.\r\n\r\nTo use your best models as an example: your XGBoost model is ~9 points better than NN on LB, but only ~3 points better based on CV. Since 2nd-level predictor uses local CV scores reflected in OOF files, it was basically mixing the data from two groups of models that didn't seem very different, even though they were. Given 20:10 ratio between quite a bit worse model (NN) and quite a bit better model (XGBoost) that seemed similar in terms of local CV, my guess is that NN models weighed more heavily in your solution than they should have.\r\n\r\nI suggest you trim down the number of NN models to 5-10, and pick not necessarily the ones with best LB scores but the ones with (CV-LB) gaps more similar to those of XGBoost models. I'd be surprised if that doesn't improve your ensemble score.\r\n", "votes": 2 }, { "id": 150080, "postDate": "2016-12-13T19:22:38.007Z", "content": "I had to confess that I was shocked. In fact, my best XGB is a bit better than the solution of Alexey, who's 2nd in this competition, and my NN is a bit worse. Below are the comparisons:
\n\nMy solution:
\n\nBest XGB - 1124.82374149 CV, 1103.76376 Public LB, 1115.89595 Private LB
\n\nBest NN -1127.59973367 CV, 1112.55274 Public LB, 1122.06610 Private LB
\n\nAlexey's:
\n\nBest XGB - 1122.64977 CV, 1105.01969 Public LB, 1116.74215 Private LB
\n\nBest NN - 1130.29286 CV, 1111.90884 Public LB, 1122.06610 Private LB
\n\nI did have 10+ XGBoost, 10+ LightGBM and approx. 20+ NN blended with out of fold predictions (10 folds).
\n\nNo idea what happened. Can anyone shed some light into this? Much appreciated.
\n\nBTW, for L2, I was using Ridge. Could this be the issue?
", "rawMarkdown": "I had to confess that I was shocked. In fact, my best XGB is a bit better than the solution of Alexey, who's 2nd in this competition, and my NN is a bit worse. Below are the comparisons:\r\n\r\nMy solution:\r\n\r\nBest XGB - 1124.82374149 CV, 1103.76376 Public LB, **1115.89595** Private LB\r\n\r\nBest NN -1127.59973367 CV, 1112.55274 Public LB, **1122.06610** Private LB\r\n\r\nAlexey's:\r\n\r\nBest XGB - 1122.64977 CV, 1105.01969 Public LB, **1116.74215** Private LB\r\n\r\nBest NN - 1130.29286 CV, 1111.90884 Public LB, **1122.06610** Private LB\r\n\r\n\r\nI did have 10+ XGBoost, 10+ LightGBM and approx. 20+ NN blended with out of fold predictions (10 folds).\r\n\r\nNo idea what happened. Can anyone shed some light into this? Much appreciated.\r\n\r\nBTW, for L2, I was using Ridge. Could this be the issue?\r\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:17:10Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 388494, "solution_url": "/competitions/allstate-claims-severity/writeups/chriscc-public-board-1101-47642-to-private-board-1" }