{
  "id": 26447,
  "title": "Faron's 3rd Place Solution",
  "content": "<p>Hi fellow Kagglers,</p>\n\n<p>first of all, I want to congratulate <a href=\"https://www.kaggle.com/bishwarup\">Bishwarup B </a> for winning this competition. Kudos! I also want to thank all the diligent forum &amp; kernel contributors: There are alot of interesting discussions and valuable kernels, which made this competition even more interesting and a good learning trip. Besides, my special thanks go to Marios (<a href=\"https://www.kaggle.com/kazanova\">KazAnova)</a> and Henk (<a href=\"https://www.kaggle.com/triskelion\">Triskelion</a>), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).</p>\n\n<hr>\n\n<p><br>\n<strong>Short summary of my final solution</strong>\n<br>\nI ended up with using XGB and Keras exclusively for my final solution, which is an \nensemble of around 100 base models (70% XGB &amp; 30% Keras models). The test set predictions have been generated by a 20-times bagged Keras model with one hidden layer as stacker at the 2nd level. </p>\n\n<p><br>\n<strong>Cross validation</strong>\n<br>\nI used 5-Fold unstratified CV (same folds has been used at base and 2nd level).</p>\n\n<p><br>\n<strong>Best single model</strong>\n<br>\nMy best single models are adapted from the public XGB &amp; Keras kernels (early stopping removed, creation of oof-predictions). The best CV-score I got is  1124.080 (1107.44746 @ public LB) for an xgb based on the <a href=\"https://www.kaggle.com/misfyre/allstate-claims-severity/encoding-feature-comb-modkzs-1108-72665/\">Encoding + Feature Comb (modkzs) (1108.72665)</a> kernel by <a href=\"https://www.kaggle.com/misfyre\">Misfyre</a> after post processing the predictions via\nyhat_opt = x1 * yhat ** x2 + x3 (yhat CV: 1125.876, public LB: 1108.95367)</p>\n\n<p>In general I did not spend much time in single model tuning, because I found it easier to improve the 2nd-level CV-score by training models on different input data and/or with transformed target.</p>\n\n<p><br>\n<strong>Input data</strong>\n<br>\nFeature engineering  like summary statistics did not help me much and hence I concentrated on different representations of the given features and their interactions.<br>\n<br>\n<strong>processing on categoricals:</strong></p>\n\n<ul>\n<li>one hot encoding</li>\n<li>label encoding (in lexicographical order)</li>\n<li>creation of interactions (up to 4-way)</li>\n<li>target loss encoding</li>\n<li>tf-idf encoding</li>\n<li>XGB-embedding (leaf indices of single tree with depth &gt;75 and min_child_weight &gt; 100 used as feature for Neural Networks)</li>\n<li>Keras-embedding via embedding layers</li>\n<li>removal of rare values</li>\n<li>count of values</li>\n</ul>\n\n<p><br>\n<strong>processing on numericals:</strong></p>\n\n<ul>\n<li>quantile-based binning</li>\n<li>for NN: standard scaling, min-max scaling, Box-Cox transformation, percentile transformation</li>\n<li>target loss encoding (except <em>cont14</em>)</li>\n<li>count of values</li>\n</ul>\n\n<p><br>\n<strong>Used target transformations</strong></p>\n\n<ul>\n<li>y</li>\n<li>log(y + n)</li>\n<li>sqrt(y)</li>\n<li>boxcox(y)</li>\n<li>y =&gt; [0,1] (for logistic regression)</li>\n</ul>\n\n<p><br>\n<strong>Base models</strong>\n<br>\n<strong>XGB:</strong>\nAround 70 models trained on different combinations of transformed target loss, input data and objective functions (including fairobj, reglinear, gblinear, gammareg, logreg, poission on counts)</p>\n\n<p><strong>NN:</strong>\nAround 30 models trained on different combinations of transformed target loss and input data. The objective has either been \"mae\" or  pin-ball loss (quantile regression).</p>\n\n<p><br>\n<strong>Ensembling</strong>\n<br>\nFor me, a bagged Keras model provided the best CV-score at the 2nd level. I also noticed that Keras was the only model, which was able to correct the systematical errors all my base models suffered from. That is, there was no more gain in post processing the 2nd level predictions and CV-wise it made no difference whether the base models predictions had been post processed / optimized or not.  </p>\n\n<p>XGB performed around 3 MAE worse here for me and did not blend with the Keras stacker (neither did ET, RF nor friends). Another interesting observation has been, that in contrast to the base models, I was only able to get a good CV with Keras at the 2nd level when training on the  orginal scaled target loss.</p>\n\n<p><br>\n<strong>Final model selection</strong>\n<br>\nThis was easy this time, because my best local CV model also provided the best LB score.</p>\n\n<p><br>\ncheers,<br>\nFaron</p>",
  "comments": [
    {
      "id": 150200,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-12-14T07:32:41.070000",
      "content": "<p>In case someone is interested: <a href=\"https://www.kaggle.com/mmueller/allstate-claims-severity/categorical-embedding-with-xgb\">Categorical Embedding with XGB</a></p>",
      "votes": 10,
      "replies": []
    },
    {
      "id": 150125,
      "author_name": "Oussama Errabia",
      "author_url": "",
      "post_date": "2016-12-13T23:38:36.937000",
      "content": "<p>congrats Faron.</p>\n\n<p>may i ask for \"the XGB-embeddeding (leaf indices of single tree with depth &gt;75 and min_child_weight &gt; 100 used as feature for Neural Networks)\"if you can link me to some references or code examples i would appreciated  </p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 150178,
      "author_name": "Pradeep",
      "author_url": "",
      "post_date": "2016-12-14T06:10:54.017000",
      "content": "<p>Gippy, a good example of using embedding layers with Keras can be found at <a href=\"https://github.com/entron/entity-embedding-rossmann\">https://github.com/entron/entity-embedding-rossmann</a></p>\n\n<p>Faron, congratulations and thank you for sharing the solution. It would be great if you have can explain how this is done:\nXGB-embeddeding (leaf indices of single tree with depth &gt;75 and min_child_weight &gt; 100 used as feature for Neural Networks)</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 150175,
      "author_name": "Gippy",
      "author_url": "",
      "post_date": "2016-12-14T05:57:26.353000",
      "content": "<p>Congrats Faron and Thanks for sharing the approach. Can you please share some intuition/explanation on Keras-embedding via embedding layers and XGB embedding?</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 150328,
      "author_name": "Μαριος Μιχαηλιδης KazAnova",
      "author_url": "",
      "post_date": "2016-12-14T19:40:42.033000",
      "content": "<p>[quote=Faron;150075]</p>\n\n<p>my special thanks go to Marios ([KazAnova)][2] and Henk ([Triskelion][3]), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).</p>\n\n<p>[/quote]</p>\n\n<p>Me and Tris dug our own graves! Haha</p>\n\n<p>Well done! </p>\n\n<p>The Feeling is mutual, would not have achieved as much as I did without your help . </p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 150321,
      "author_name": "Vladimir Iglovikov",
      "author_url": "",
      "post_date": "2016-12-14T19:17:09.980000",
      "content": "<p>Thank you.</p>\n\n<p>Before this competition I did not think that <strong>max_depth &gt; 10</strong> is worth considering.  And at the end my best hyperparameters for xgb involved <strong>max_depth = 13</strong></p>\n\n<p>But <strong>25</strong>? I would never even guess to try it. </p>\n\n<ol>\n<li><p>How common it is to have max_depth that is so big?</p></li>\n<li><p>What was your space for RandomizedGridSearch that you started with\nfor this competition?</p></li>\n</ol>",
      "votes": 4,
      "replies": [
        {
          "id": 150325,
          "author_name": "Μαριος Μιχαηλιδης KazAnova",
          "author_url": "",
          "post_date": "2016-12-14T19:36:27.473000",
          "content": "<p>[quote=Vladimir Iglovikov;150321]</p>\n\n<p>But <strong>25</strong>? I would never even guess to try it. </p>\n\n<ol>\n<li>How common it is to have max_depth that is so big?</li>\n</ol>\n\n<p>[/quote]</p>\n\n<p>Something you need to consider is that Faron's Xgboost  (and keras) is not the same as everybody else's. I know this from experience. We may run the same thing and same hyper params, but his models are always 10% better ! If it was max_depth 40 ...it would still work in Faron's laptop...hah</p>\n\n<p>Kidding aside  you can always increase max_depth if u also increase min_child_weight. Same goes with other tree-based models. You can increase depth and min_leaf too. The reasoning is that you can become bolder with how greedy (and large) your model becomes as long as you account for over fitting somehow. </p>",
          "votes": 12,
          "replies": [
            {
              "id": 150492,
              "author_name": "Bishwarup B",
              "author_url": "",
              "post_date": "2016-12-15T12:29:13.817000",
              "content": "<p>@Faron: I used something like this:</p>\n\n<pre><code>IF (oof_pred &lt; cutoff[1]) {\n     oof_pred = w[1]*oof_pred^w[2] + w[3]\n} \nELSE IF (oof_pred &gt; cutoff[2]) {\n     oof_pred = w[4]*oof_pred^w[5] + w[6]\n} ELSE oof_pred\n</code></pre>\n\n<p>Then I optimized the weights and the cut-offs.</p>\n\n<p>[/quote]</p>\n\n<p>Something you need to consider is that Faron's Xgboost  (and keras) is not the same as everybody else's. I know this from experience. We may run the same thing and same hyper params, but his models are always 10% better ! If it was max_depth 40 ...it would still work in Faron's laptop...hah</p>\n\n<p>[/quote]</p>\n\n<p>Any plan to release Faron's Xgboost soon in future? :P</p>",
              "votes": 4,
              "replies": []
            }
          ]
        },
        {
          "id": 150460,
          "author_name": "Faron",
          "author_url": "",
          "post_date": "2016-12-15T09:23:54.760000",
          "content": "<p>[quote=Vladimir Iglovikov;150321]</p>\n\n<ol>\n<li><p>How common it is to have max_depth that is so big?</p></li>\n<li><p>What was your space for RandomizedGridSearch that you started with\nfor this competition?</p></li>\n</ol>\n\n<p>[/quote]</p>\n\n<p>Not very common to my best knowledge. In general neither max_depth of 25 nor min_child_weight of 50 are part of my initial search space, but I like try more or less extreme values later on to see what happens. I started with something like</p>\n\n<pre><code>params = \n{\n    'eta': random.choice([0.075]),\n    'colsample_bytree': random.choice([0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),\n    'subsample': random.choice([0.5, 0.7, 0.9, 0.95]),\n    'max_depth': random.choice([7]),\n    'min_child_weight': random.choice([1,5,10]),\n    'max_delta_step': random.choice([0, 0.25, 0.5]),\n    'alpha': random.choice([0, 0.25, 0.5, 0.75, 1.]),\n    'lambda': random.choice([1, 1.5, 2, 2.5]),\n    'gamma': random.choice([0, 0.5, 1., 1.5, 2.]),\n    'base_score': np.mean(y_train),\n    'seed': 0\n}\n</code></pre>\n\n<p>and my last one has been:</p>\n\n<pre><code>params = \n{\n    'eta': random.choice([0.015]),\n    'colsample_bytree': random.choice([0.3, 0.4, 0.5, 0.7]),\n    'subsample': random.choice([0.8, 0.9, 0.925, 0.975, 0.995]),\n    'max_depth': random.choice([17, 19, 21, 23, 25, 27, 29]),\n    'min_child_weight': random.choice([15, 40, 50, 60, 70, 100, 150]),\n    'max_delta_step': random.choice([0, 0.5]),\n    'alpha': random.choice([1, 1.5, 2., 2.5, 3.0]),\n    'lambda': random.choice([1, 1.5, 2., 2.5, 3.0]),\n    'gamma': random.choice([1., 1.5, 2., 2.5, 3.0]),\n    'base_score': np.mean(y_train),\n    'seed': 0,\n}\n</code></pre>",
          "votes": 4,
          "replies": []
        }
      ]
    },
    {
      "id": 150319,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-12-14T19:12:02.293000",
      "content": "<p>@Vladimir: I only reported <a href=\"https://www.kaggle.com/c/allstate-claims-severity/forums/t/24611/single-model-performance?forumMessageId=140571#post140571\">1109</a> on raw features and this I got with the following params:</p>\n\n<pre><code>xgb_params = {\n    'seed': 0,\n    'colsample_bytree': 0.4,\n    'silent': 1,\n    'subsample': 0.975,\n    'learning_rate': 0.015,\n    'gamma': 1.5,\n    'lambda': 2,\n    'alpha': 2,\n    'max_depth': 25,\n    'num_parallel_tree': 1,\n    'min_child_weight': 50,\n    'eval_metric': 'mae',\n    'max_delta_step': 0,\n}\n</code></pre>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 150273,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-12-14T14:03:16.157000",
      "content": "<p>@ATS: I used oof-target-encoding on all categoricals. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 150199,
      "author_name": "DDgg",
      "author_url": "",
      "post_date": "2016-12-14T07:32:04.893000",
      "content": "<p>thanks for the insight!  </p>\n\n<p>publishing high ranking starter kernels is really pushing the envelope. <br>\nit also positively multiplies across the community</p>",
      "votes": 1,
      "replies": [
        {
          "id": 150207,
          "author_name": "ATS",
          "author_url": "",
          "post_date": "2016-12-14T07:53:23.373000",
          "content": "<p>[quote=Faron;150075]</p>\n\n<p><strong>processing on categoricals:</strong></p>\n\n<ul>\n<li>creation of interactions (up to 4-way)</li>\n<li>target loss encoding</li>\n</ul>\n\n<p>[/quote]</p>\n\n<p>Thanks for detailed description. </p>\n\n<p>How did you encode categorical interactions?</p>\n\n<p>Did you use oof-target-encoding or straight full train? Did you use all cat_s for this encoding or only high cardinality?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 150197,
      "author_name": "Bishwarup B",
      "author_url": "",
      "post_date": "2016-12-14T07:26:33.847000",
      "content": "<p>@ Congrats to you too Far0n. Your approach on preprocessing the categorical seems very similar to mine. Great idea to use the leaf indices as a meta feature in second level model. I don't know exactly where it was different, but I got pretty good gain by post-processing my second layer keras model (almost 0.2 on an average, which got carried to private leaderboard as well).</p>",
      "votes": 1,
      "replies": [
        {
          "id": 150204,
          "author_name": "Faron",
          "author_url": "",
          "post_date": "2016-12-14T07:45:10.067000",
          "content": "<p>[quote=Bishwarup B;150197]</p>\n\n<p>@ Congrats to you too Far0n. Your approach on preprocessing the categorical seems very similar to mine. Great idea to use the leaf indices as a meta feature in second level model. I don't know exactly where it was different, but I got pretty good gain by post-processing my second layer keras model (almost 0.2 on an average, which got carried to private leaderboard as well).</p>\n\n<p>[/quote]</p>\n\n<p>Thank you @Bishwarup B. It's interesting, that you was able to improve your 2nd level predictions via post processing. How exactly did you do it? My optimization worked nicely for all my base models (at least 1 MAE gain on the oof-CV) but made my 2nd-level CV always worse. Besides, the 2nd level CV of my keras model did not change significantly after applying the post processing on my base models. I concluded, that the stacker was able to correct those errors. Maybe I missed something here.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 150079,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-12-13T19:18:03.887000",
      "content": "<p>@itod: I used this one: <a href=\"http://pandas.pydata.org/pandas-docs/version/0.18.1/generated/pandas.qcut.html\">pandas qcut</a></p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 150077,
      "author_name": "itod",
      "author_url": "",
      "post_date": "2016-12-13T19:15:13.963000",
      "content": "<p>Congrats, Faron.</p>\n\n<p>Can you please give some more detail how you did quantile-based binning?</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 162081,
      "author_name": "Kyungsu Kim",
      "author_url": "",
      "post_date": "2017-02-17T01:25:41.013000",
      "content": "<p>Congrats and thanks for your sharing!</p>\n\n<p>The idea of using Neural Network is interesting.</p>\n\n<p>Can you tell the architecture of Neural Networks?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 151798,
      "author_name": "Marius Bock",
      "author_url": "",
      "post_date": "2016-12-22T05:06:48.413000",
      "content": "<p>Thanks for sharing, Faron! And of course: Congratulations!</p>\n\n<p>Following up on the question above: What does oof-encoding in general? I tried to googling it, but did not find an answer...</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 151480,
      "author_name": "lll",
      "author_url": "",
      "post_date": "2016-12-20T23:13:46.597000",
      "content": "<p>Thanks for sharing!</p>\n\n<p>I have a question.</p>\n\n<p>What is target loss encoding? To encode categories with the mean of the target loss?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 150313,
      "author_name": "Vladimir Iglovikov",
      "author_url": "",
      "post_date": "2016-12-14T18:37:07.723000",
      "content": "<p>@Faron, question that I have for more than a month now. You mentioned in another thread that you got 1107 on a raw features on the log transformed target.</p>\n\n<p>What were your hyperparameters?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 150177,
      "author_name": "Andy Shkabko",
      "author_url": "",
      "post_date": "2016-12-14T06:07:34.327000",
      "content": "<p>can someone confirm that Feature Combination (modkzs) (1108.72665)  gave better CVs with Keras model?</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 22,
  "messages": [
    {
      "id": 150075,
      "postDate": "2016-12-13T19:02:36.897Z",
      "content": "<p>Hi fellow Kagglers,</p>\n\n<p>first of all, I want to congratulate <a href=\"https://www.kaggle.com/bishwarup\">Bishwarup B </a> for winning this competition. Kudos! I also want to thank all the diligent forum &amp; kernel contributors: There are alot of interesting discussions and valuable kernels, which made this competition even more interesting and a good learning trip. Besides, my special thanks go to Marios (<a href=\"https://www.kaggle.com/kazanova\">KazAnova)</a> and Henk (<a href=\"https://www.kaggle.com/triskelion\">Triskelion</a>), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).</p>\n\n<hr>\n\n<p><br>\n<strong>Short summary of my final solution</strong>\n<br>\nI ended up with using XGB and Keras exclusively for my final solution, which is an \nensemble of around 100 base models (70% XGB &amp; 30% Keras models). The test set predictions have been generated by a 20-times bagged Keras model with one hidden layer as stacker at the 2nd level. </p>\n\n<p><br>\n<strong>Cross validation</strong>\n<br>\nI used 5-Fold unstratified CV (same folds has been used at base and 2nd level).</p>\n\n<p><br>\n<strong>Best single model</strong>\n<br>\nMy best single models are adapted from the public XGB &amp; Keras kernels (early stopping removed, creation of oof-predictions). The best CV-score I got is  1124.080 (1107.44746 @ public LB) for an xgb based on the <a href=\"https://www.kaggle.com/misfyre/allstate-claims-severity/encoding-feature-comb-modkzs-1108-72665/\">Encoding + Feature Comb (modkzs) (1108.72665)</a> kernel by <a href=\"https://www.kaggle.com/misfyre\">Misfyre</a> after post processing the predictions via\nyhat_opt = x1 * yhat ** x2 + x3 (yhat CV: 1125.876, public LB: 1108.95367)</p>\n\n<p>In general I did not spend much time in single model tuning, because I found it easier to improve the 2nd-level CV-score by training models on different input data and/or with transformed target.</p>\n\n<p><br>\n<strong>Input data</strong>\n<br>\nFeature engineering  like summary statistics did not help me much and hence I concentrated on different representations of the given features and their interactions.<br>\n<br>\n<strong>processing on categoricals:</strong></p>\n\n<ul>\n<li>one hot encoding</li>\n<li>label encoding (in lexicographical order)</li>\n<li>creation of interactions (up to 4-way)</li>\n<li>target loss encoding</li>\n<li>tf-idf encoding</li>\n<li>XGB-embedding (leaf indices of single tree with depth &gt;75 and min_child_weight &gt; 100 used as feature for Neural Networks)</li>\n<li>Keras-embedding via embedding layers</li>\n<li>removal of rare values</li>\n<li>count of values</li>\n</ul>\n\n<p><br>\n<strong>processing on numericals:</strong></p>\n\n<ul>\n<li>quantile-based binning</li>\n<li>for NN: standard scaling, min-max scaling, Box-Cox transformation, percentile transformation</li>\n<li>target loss encoding (except <em>cont14</em>)</li>\n<li>count of values</li>\n</ul>\n\n<p><br>\n<strong>Used target transformations</strong></p>\n\n<ul>\n<li>y</li>\n<li>log(y + n)</li>\n<li>sqrt(y)</li>\n<li>boxcox(y)</li>\n<li>y =&gt; [0,1] (for logistic regression)</li>\n</ul>\n\n<p><br>\n<strong>Base models</strong>\n<br>\n<strong>XGB:</strong>\nAround 70 models trained on different combinations of transformed target loss, input data and objective functions (including fairobj, reglinear, gblinear, gammareg, logreg, poission on counts)</p>\n\n<p><strong>NN:</strong>\nAround 30 models trained on different combinations of transformed target loss and input data. The objective has either been \"mae\" or  pin-ball loss (quantile regression).</p>\n\n<p><br>\n<strong>Ensembling</strong>\n<br>\nFor me, a bagged Keras model provided the best CV-score at the 2nd level. I also noticed that Keras was the only model, which was able to correct the systematical errors all my base models suffered from. That is, there was no more gain in post processing the 2nd level predictions and CV-wise it made no difference whether the base models predictions had been post processed / optimized or not.  </p>\n\n<p>XGB performed around 3 MAE worse here for me and did not blend with the Keras stacker (neither did ET, RF nor friends). Another interesting observation has been, that in contrast to the base models, I was only able to get a good CV with Keras at the 2nd level when training on the  orginal scaled target loss.</p>\n\n<p><br>\n<strong>Final model selection</strong>\n<br>\nThis was easy this time, because my best local CV model also provided the best LB score.</p>\n\n<p><br>\ncheers,<br>\nFaron</p>",
      "rawMarkdown": "Hi fellow Kagglers,\r\n\r\nfirst of all, I want to congratulate [Bishwarup B ][1] for winning this competition. Kudos! I also want to thank all the diligent forum & kernel contributors: There are alot of interesting discussions and valuable kernels, which made this competition even more interesting and a good learning trip. Besides, my special thanks go to Marios ([KazAnova)][2] and Henk ([Triskelion][3]), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).\r\n\r\n\r\n----------\r\n<br>\r\n**Short summary of my final solution**\r\n<br>\r\nI ended up with using XGB and Keras exclusively for my final solution, which is an \r\nensemble of around 100 base models (70% XGB & 30% Keras models). The test set predictions have been generated by a 20-times bagged Keras model with one hidden layer as stacker at the 2nd level. \r\n\r\n<br>\r\n**Cross validation**\r\n<br>\r\nI used 5-Fold unstratified CV (same folds has been used at base and 2nd level).\r\n\r\n<br>\r\n**Best single model**\r\n<br>\r\nMy best single models are adapted from the public XGB & Keras kernels (early stopping removed, creation of oof-predictions). The best CV-score I got is  1124.080 (1107.44746 @ public LB) for an xgb based on the [Encoding + Feature Comb (modkzs) (1108.72665)][4] kernel by [Misfyre][5] after post processing the predictions via\r\nyhat_opt = x1 * yhat ** x2 + x3 (yhat CV: 1125.876, public LB: 1108.95367)\r\n\r\nIn general I did not spend much time in single model tuning, because I found it easier to improve the 2nd-level CV-score by training models on different input data and/or with transformed target.\r\n\r\n<br>\r\n**Input data**\r\n<br>\r\nFeature engineering  like summary statistics did not help me much and hence I concentrated on different representations of the given features and their interactions.<br>\r\n<br>\r\n**processing on categoricals:**\r\n\r\n - one hot encoding\r\n - label encoding (in lexicographical order)\r\n - creation of interactions (up to 4-way)\r\n - target loss encoding\r\n - tf-idf encoding\r\n - XGB-embedding (leaf indices of single tree with depth >75 and min_child_weight > 100 used as feature for Neural Networks)\r\n - Keras-embedding via embedding layers\r\n - removal of rare values\r\n - count of values\r\n\r\n<br>\r\n**processing on numericals:**\r\n\r\n - quantile-based binning\r\n - for NN: standard scaling, min-max scaling, Box-Cox transformation, percentile transformation\r\n - target loss encoding (except *cont14*)\r\n - count of values\r\n\r\n<br>\r\n**Used target transformations**\r\n\r\n - y\r\n - log(y + n)\r\n - sqrt(y)\r\n - boxcox(y)\r\n - y => [0,1] (for logistic regression)\r\n\r\n<br>\r\n**Base models**\r\n<br>\r\n**XGB:**\r\nAround 70 models trained on different combinations of transformed target loss, input data and objective functions (including fairobj, reglinear, gblinear, gammareg, logreg, poission on counts)\r\n\r\n**NN:**\r\nAround 30 models trained on different combinations of transformed target loss and input data. The objective has either been \"mae\" or  pin-ball loss (quantile regression).\r\n \r\n<br>\r\n**Ensembling**\r\n<br>\r\nFor me, a bagged Keras model provided the best CV-score at the 2nd level. I also noticed that Keras was the only model, which was able to correct the systematical errors all my base models suffered from. That is, there was no more gain in post processing the 2nd level predictions and CV-wise it made no difference whether the base models predictions had been post processed / optimized or not.  \r\n\r\nXGB performed around 3 MAE worse here for me and did not blend with the Keras stacker (neither did ET, RF nor friends). Another interesting observation has been, that in contrast to the base models, I was only able to get a good CV with Keras at the 2nd level when training on the  orginal scaled target loss.\r\n\r\n<br>\r\n**Final model selection**\r\n<br>\r\nThis was easy this time, because my best local CV model also provided the best LB score.\r\n\r\n<br>\r\ncheers,<br>\r\nFaron\r\n\r\n  [1]: https://www.kaggle.com/bishwarup\r\n  [2]: https://www.kaggle.com/kazanova\r\n  [3]: https://www.kaggle.com/triskelion\r\n  [4]: https://www.kaggle.com/misfyre/allstate-claims-severity/encoding-feature-comb-modkzs-1108-72665/\r\n  [5]: https://www.kaggle.com/misfyre",
      "votes": 61
    },
    {
      "id": 150200,
      "postDate": "2016-12-14T07:32:41.070Z",
      "content": "<p>In case someone is interested: <a href=\"https://www.kaggle.com/mmueller/allstate-claims-severity/categorical-embedding-with-xgb\">Categorical Embedding with XGB</a></p>",
      "rawMarkdown": "In case someone is interested: [Categorical Embedding with XGB][1]\r\n\r\n  [1]: https://www.kaggle.com/mmueller/allstate-claims-severity/categorical-embedding-with-xgb",
      "votes": 10
    },
    {
      "id": 150125,
      "postDate": "2016-12-13T23:38:36.937Z",
      "content": "<p>congrats Faron.</p>\n\n<p>may i ask for \"the XGB-embeddeding (leaf indices of single tree with depth &gt;75 and min_child_weight &gt; 100 used as feature for Neural Networks)\"if you can link me to some references or code examples i would appreciated  </p>",
      "rawMarkdown": "congrats Faron.\r\n\r\n may i ask for \"the XGB-embeddeding (leaf indices of single tree with depth >75 and min_child_weight > 100 used as feature for Neural Networks)\"if you can link me to some references or code examples i would appreciated  ",
      "votes": 5
    },
    {
      "id": 150178,
      "postDate": "2016-12-14T06:10:54.017Z",
      "content": "<p>Gippy, a good example of using embedding layers with Keras can be found at <a href=\"https://github.com/entron/entity-embedding-rossmann\">https://github.com/entron/entity-embedding-rossmann</a></p>\n\n<p>Faron, congratulations and thank you for sharing the solution. It would be great if you have can explain how this is done:\nXGB-embeddeding (leaf indices of single tree with depth &gt;75 and min_child_weight &gt; 100 used as feature for Neural Networks)</p>",
      "rawMarkdown": "Gippy, a good example of using embedding layers with Keras can be found at https://github.com/entron/entity-embedding-rossmann\r\n\r\nFaron, congratulations and thank you for sharing the solution. It would be great if you have can explain how this is done:\r\nXGB-embeddeding (leaf indices of single tree with depth >75 and min_child_weight > 100 used as feature for Neural Networks)",
      "votes": 3
    },
    {
      "id": 150175,
      "postDate": "2016-12-14T05:57:26.353Z",
      "content": "<p>Congrats Faron and Thanks for sharing the approach. Can you please share some intuition/explanation on Keras-embedding via embedding layers and XGB embedding?</p>",
      "rawMarkdown": "Congrats Faron and Thanks for sharing the approach. Can you please share some intuition/explanation on Keras-embedding via embedding layers and XGB embedding?",
      "votes": 3
    },
    {
      "id": 150328,
      "postDate": "2016-12-14T19:40:42.033Z",
      "content": "<p>[quote=Faron;150075]</p>\n\n<p>my special thanks go to Marios ([KazAnova)][2] and Henk ([Triskelion][3]), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).</p>\n\n<p>[/quote]</p>\n\n<p>Me and Tris dug our own graves! Haha</p>\n\n<p>Well done! </p>\n\n<p>The Feeling is mutual, would not have achieved as much as I did without your help . </p>",
      "rawMarkdown": "[quote=Faron;150075]\r\n\r\n my special thanks go to Marios ([KazAnova)][2] and Henk ([Triskelion][3]), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).\r\n\r\n[/quote]\r\n\r\nMe and Tris dug our own graves! Haha\r\n\r\nWell done! \r\n\r\nThe Feeling is mutual, would not have achieved as much as I did without your help . \r\n ",
      "votes": 4
    },
    {
      "id": 150321,
      "postDate": "2016-12-14T19:17:09.980Z",
      "content": "<p>Thank you.</p>\n\n<p>Before this competition I did not think that <strong>max_depth &gt; 10</strong> is worth considering.  And at the end my best hyperparameters for xgb involved <strong>max_depth = 13</strong></p>\n\n<p>But <strong>25</strong>? I would never even guess to try it. </p>\n\n<ol>\n<li><p>How common it is to have max_depth that is so big?</p></li>\n<li><p>What was your space for RandomizedGridSearch that you started with\nfor this competition?</p></li>\n</ol>",
      "rawMarkdown": "Thank you.\r\n\r\nBefore this competition I did not think that **max_depth > 10** is worth considering.  And at the end my best hyperparameters for xgb involved **max_depth = 13**\r\n\r\nBut **25**? I would never even guess to try it. \r\n\r\n 1. How common it is to have max_depth that is so big?\r\n   \r\n 2. What was your space for RandomizedGridSearch that you started with\r\n   for this competition?",
      "votes": 4,
      "replies": [
        {
          "id": 150325,
          "postDate": "2016-12-14T19:36:27.473Z",
          "content": "<p>[quote=Vladimir Iglovikov;150321]</p>\n\n<p>But <strong>25</strong>? I would never even guess to try it. </p>\n\n<ol>\n<li>How common it is to have max_depth that is so big?</li>\n</ol>\n\n<p>[/quote]</p>\n\n<p>Something you need to consider is that Faron's Xgboost  (and keras) is not the same as everybody else's. I know this from experience. We may run the same thing and same hyper params, but his models are always 10% better ! If it was max_depth 40 ...it would still work in Faron's laptop...hah</p>\n\n<p>Kidding aside  you can always increase max_depth if u also increase min_child_weight. Same goes with other tree-based models. You can increase depth and min_leaf too. The reasoning is that you can become bolder with how greedy (and large) your model becomes as long as you account for over fitting somehow. </p>",
          "rawMarkdown": "[quote=Vladimir Iglovikov;150321]\r\n\r\n\r\nBut **25**? I would never even guess to try it. \r\n\r\n 1. How common it is to have max_depth that is so big?\r\n\r\n[/quote]\r\n\r\nSomething you need to consider is that Faron's Xgboost  (and keras) is not the same as everybody else's. I know this from experience. We may run the same thing and same hyper params, but his models are always 10% better ! If it was max_depth 40 ...it would still work in Faron's laptop...hah\r\n\r\nKidding aside  you can always increase max_depth if u also increase min_child_weight. Same goes with other tree-based models. You can increase depth and min_leaf too. The reasoning is that you can become bolder with how greedy (and large) your model becomes as long as you account for over fitting somehow. ",
          "votes": 12,
          "replies": [
            {
              "id": 150492,
              "postDate": "2016-12-15T12:29:13.817Z",
              "content": "<p>@Faron: I used something like this:</p>\n\n<pre><code>IF (oof_pred &lt; cutoff[1]) {\n     oof_pred = w[1]*oof_pred^w[2] + w[3]\n} \nELSE IF (oof_pred &gt; cutoff[2]) {\n     oof_pred = w[4]*oof_pred^w[5] + w[6]\n} ELSE oof_pred\n</code></pre>\n\n<p>Then I optimized the weights and the cut-offs.</p>\n\n<p>[/quote]</p>\n\n<p>Something you need to consider is that Faron's Xgboost  (and keras) is not the same as everybody else's. I know this from experience. We may run the same thing and same hyper params, but his models are always 10% better ! If it was max_depth 40 ...it would still work in Faron's laptop...hah</p>\n\n<p>[/quote]</p>\n\n<p>Any plan to release Faron's Xgboost soon in future? :P</p>",
              "rawMarkdown": "@Faron: I used something like this:\r\n\r\n    IF (oof_pred < cutoff[1]) {\r\n         oof_pred = w[1]*oof_pred^w[2] + w[3]\r\n    } \r\n    ELSE IF (oof_pred > cutoff[2]) {\r\n         oof_pred = w[4]*oof_pred^w[5] + w[6]\r\n    } ELSE oof_pred\r\n\r\nThen I optimized the weights and the cut-offs.\r\n\r\n[/quote]\r\n\r\nSomething you need to consider is that Faron's Xgboost  (and keras) is not the same as everybody else's. I know this from experience. We may run the same thing and same hyper params, but his models are always 10% better ! If it was max_depth 40 ...it would still work in Faron's laptop...hah\r\n\r\n[/quote]\r\n\r\nAny plan to release Faron's Xgboost soon in future? :P",
              "votes": 4
            }
          ]
        },
        {
          "id": 150460,
          "postDate": "2016-12-15T09:23:54.760Z",
          "content": "<p>[quote=Vladimir Iglovikov;150321]</p>\n\n<ol>\n<li><p>How common it is to have max_depth that is so big?</p></li>\n<li><p>What was your space for RandomizedGridSearch that you started with\nfor this competition?</p></li>\n</ol>\n\n<p>[/quote]</p>\n\n<p>Not very common to my best knowledge. In general neither max_depth of 25 nor min_child_weight of 50 are part of my initial search space, but I like try more or less extreme values later on to see what happens. I started with something like</p>\n\n<pre><code>params = \n{\n    'eta': random.choice([0.075]),\n    'colsample_bytree': random.choice([0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),\n    'subsample': random.choice([0.5, 0.7, 0.9, 0.95]),\n    'max_depth': random.choice([7]),\n    'min_child_weight': random.choice([1,5,10]),\n    'max_delta_step': random.choice([0, 0.25, 0.5]),\n    'alpha': random.choice([0, 0.25, 0.5, 0.75, 1.]),\n    'lambda': random.choice([1, 1.5, 2, 2.5]),\n    'gamma': random.choice([0, 0.5, 1., 1.5, 2.]),\n    'base_score': np.mean(y_train),\n    'seed': 0\n}\n</code></pre>\n\n<p>and my last one has been:</p>\n\n<pre><code>params = \n{\n    'eta': random.choice([0.015]),\n    'colsample_bytree': random.choice([0.3, 0.4, 0.5, 0.7]),\n    'subsample': random.choice([0.8, 0.9, 0.925, 0.975, 0.995]),\n    'max_depth': random.choice([17, 19, 21, 23, 25, 27, 29]),\n    'min_child_weight': random.choice([15, 40, 50, 60, 70, 100, 150]),\n    'max_delta_step': random.choice([0, 0.5]),\n    'alpha': random.choice([1, 1.5, 2., 2.5, 3.0]),\n    'lambda': random.choice([1, 1.5, 2., 2.5, 3.0]),\n    'gamma': random.choice([1., 1.5, 2., 2.5, 3.0]),\n    'base_score': np.mean(y_train),\n    'seed': 0,\n}\n</code></pre>",
          "rawMarkdown": "[quote=Vladimir Iglovikov;150321]\r\n\r\n 1. How common it is to have max_depth that is so big?\r\n   \r\n 2. What was your space for RandomizedGridSearch that you started with\r\n   for this competition?\r\n\r\n[/quote]\r\n\r\nNot very common to my best knowledge. In general neither max_depth of 25 nor min_child_weight of 50 are part of my initial search space, but I like try more or less extreme values later on to see what happens. I started with something like\r\n\r\n    params = \r\n    {\r\n        'eta': random.choice([0.075]),\r\n        'colsample_bytree': random.choice([0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),\r\n        'subsample': random.choice([0.5, 0.7, 0.9, 0.95]),\r\n        'max_depth': random.choice([7]),\r\n        'min_child_weight': random.choice([1,5,10]),\r\n        'max_delta_step': random.choice([0, 0.25, 0.5]),\r\n        'alpha': random.choice([0, 0.25, 0.5, 0.75, 1.]),\r\n        'lambda': random.choice([1, 1.5, 2, 2.5]),\r\n        'gamma': random.choice([0, 0.5, 1., 1.5, 2.]),\r\n        'base_score': np.mean(y_train),\r\n        'seed': 0\r\n    }\r\n\r\nand my last one has been:\r\n\r\n    params = \r\n    {\r\n        'eta': random.choice([0.015]),\r\n        'colsample_bytree': random.choice([0.3, 0.4, 0.5, 0.7]),\r\n        'subsample': random.choice([0.8, 0.9, 0.925, 0.975, 0.995]),\r\n        'max_depth': random.choice([17, 19, 21, 23, 25, 27, 29]),\r\n        'min_child_weight': random.choice([15, 40, 50, 60, 70, 100, 150]),\r\n        'max_delta_step': random.choice([0, 0.5]),\r\n        'alpha': random.choice([1, 1.5, 2., 2.5, 3.0]),\r\n        'lambda': random.choice([1, 1.5, 2., 2.5, 3.0]),\r\n        'gamma': random.choice([1., 1.5, 2., 2.5, 3.0]),\r\n        'base_score': np.mean(y_train),\r\n        'seed': 0,\r\n    }\r\n\r\n\r\n",
          "votes": 4
        }
      ]
    },
    {
      "id": 150319,
      "postDate": "2016-12-14T19:12:02.293Z",
      "content": "<p>@Vladimir: I only reported <a href=\"https://www.kaggle.com/c/allstate-claims-severity/forums/t/24611/single-model-performance?forumMessageId=140571#post140571\">1109</a> on raw features and this I got with the following params:</p>\n\n<pre><code>xgb_params = {\n    'seed': 0,\n    'colsample_bytree': 0.4,\n    'silent': 1,\n    'subsample': 0.975,\n    'learning_rate': 0.015,\n    'gamma': 1.5,\n    'lambda': 2,\n    'alpha': 2,\n    'max_depth': 25,\n    'num_parallel_tree': 1,\n    'min_child_weight': 50,\n    'eval_metric': 'mae',\n    'max_delta_step': 0,\n}\n</code></pre>",
      "rawMarkdown": "@Vladimir: I only reported [1109][1] on raw features and this I got with the following params:\r\n\r\n    xgb_params = {\r\n        'seed': 0,\r\n        'colsample_bytree': 0.4,\r\n        'silent': 1,\r\n        'subsample': 0.975,\r\n        'learning_rate': 0.015,\r\n        'gamma': 1.5,\r\n        'lambda': 2,\r\n        'alpha': 2,\r\n        'max_depth': 25,\r\n        'num_parallel_tree': 1,\r\n        'min_child_weight': 50,\r\n        'eval_metric': 'mae',\r\n        'max_delta_step': 0,\r\n    }\r\n\r\n\r\n  [1]: https://www.kaggle.com/c/allstate-claims-severity/forums/t/24611/single-model-performance?forumMessageId=140571#post140571",
      "votes": 4
    },
    {
      "id": 150273,
      "postDate": "2016-12-14T14:03:16.157Z",
      "content": "<p>@ATS: I used oof-target-encoding on all categoricals. </p>",
      "rawMarkdown": "@ATS: I used oof-target-encoding on all categoricals. ",
      "votes": 1
    },
    {
      "id": 150199,
      "postDate": "2016-12-14T07:32:04.893Z",
      "content": "<p>thanks for the insight!  </p>\n\n<p>publishing high ranking starter kernels is really pushing the envelope. <br>\nit also positively multiplies across the community</p>",
      "rawMarkdown": "thanks for the insight!  \r\n\r\npublishing high ranking starter kernels is really pushing the envelope.  \r\nit also positively multiplies across the community\r\n\r\n",
      "votes": 1,
      "replies": [
        {
          "id": 150207,
          "postDate": "2016-12-14T07:53:23.373Z",
          "content": "<p>[quote=Faron;150075]</p>\n\n<p><strong>processing on categoricals:</strong></p>\n\n<ul>\n<li>creation of interactions (up to 4-way)</li>\n<li>target loss encoding</li>\n</ul>\n\n<p>[/quote]</p>\n\n<p>Thanks for detailed description. </p>\n\n<p>How did you encode categorical interactions?</p>\n\n<p>Did you use oof-target-encoding or straight full train? Did you use all cat_s for this encoding or only high cardinality?</p>",
          "rawMarkdown": "[quote=Faron;150075]\r\n\r\n**processing on categoricals:**\r\n\r\n - creation of interactions (up to 4-way)\r\n - target loss encoding\r\n\r\n[/quote]\r\n\r\nThanks for detailed description. \r\n\r\nHow did you encode categorical interactions?\r\n\r\nDid you use oof-target-encoding or straight full train? Did you use all cat_s for this encoding or only high cardinality?"
        }
      ]
    },
    {
      "id": 150197,
      "postDate": "2016-12-14T07:26:33.847Z",
      "content": "<p>@ Congrats to you too Far0n. Your approach on preprocessing the categorical seems very similar to mine. Great idea to use the leaf indices as a meta feature in second level model. I don't know exactly where it was different, but I got pretty good gain by post-processing my second layer keras model (almost 0.2 on an average, which got carried to private leaderboard as well).</p>",
      "rawMarkdown": "@ Congrats to you too Far0n. Your approach on preprocessing the categorical seems very similar to mine. Great idea to use the leaf indices as a meta feature in second level model. I don't know exactly where it was different, but I got pretty good gain by post-processing my second layer keras model (almost 0.2 on an average, which got carried to private leaderboard as well).\r\n",
      "votes": 1,
      "replies": [
        {
          "id": 150204,
          "postDate": "2016-12-14T07:45:10.067Z",
          "content": "<p>[quote=Bishwarup B;150197]</p>\n\n<p>@ Congrats to you too Far0n. Your approach on preprocessing the categorical seems very similar to mine. Great idea to use the leaf indices as a meta feature in second level model. I don't know exactly where it was different, but I got pretty good gain by post-processing my second layer keras model (almost 0.2 on an average, which got carried to private leaderboard as well).</p>\n\n<p>[/quote]</p>\n\n<p>Thank you @Bishwarup B. It's interesting, that you was able to improve your 2nd level predictions via post processing. How exactly did you do it? My optimization worked nicely for all my base models (at least 1 MAE gain on the oof-CV) but made my 2nd-level CV always worse. Besides, the 2nd level CV of my keras model did not change significantly after applying the post processing on my base models. I concluded, that the stacker was able to correct those errors. Maybe I missed something here.</p>",
          "rawMarkdown": "[quote=Bishwarup B;150197]\r\n\r\n@ Congrats to you too Far0n. Your approach on preprocessing the categorical seems very similar to mine. Great idea to use the leaf indices as a meta feature in second level model. I don't know exactly where it was different, but I got pretty good gain by post-processing my second layer keras model (almost 0.2 on an average, which got carried to private leaderboard as well).\r\n\r\n\r\n[/quote]\r\n\r\nThank you @Bishwarup B. It's interesting, that you was able to improve your 2nd level predictions via post processing. How exactly did you do it? My optimization worked nicely for all my base models (at least 1 MAE gain on the oof-CV) but made my 2nd-level CV always worse. Besides, the 2nd level CV of my keras model did not change significantly after applying the post processing on my base models. I concluded, that the stacker was able to correct those errors. Maybe I missed something here."
        }
      ]
    },
    {
      "id": 150079,
      "postDate": "2016-12-13T19:18:03.887Z",
      "content": "<p>@itod: I used this one: <a href=\"http://pandas.pydata.org/pandas-docs/version/0.18.1/generated/pandas.qcut.html\">pandas qcut</a></p>",
      "rawMarkdown": "@itod: I used this one: [pandas qcut][1]\r\n\r\n\r\n  [1]: http://pandas.pydata.org/pandas-docs/version/0.18.1/generated/pandas.qcut.html",
      "votes": 2
    },
    {
      "id": 150077,
      "postDate": "2016-12-13T19:15:13.963Z",
      "content": "<p>Congrats, Faron.</p>\n\n<p>Can you please give some more detail how you did quantile-based binning?</p>",
      "rawMarkdown": "Congrats, Faron.\r\n\r\nCan you please give some more detail how you did quantile-based binning?",
      "votes": 2
    },
    {
      "id": 162081,
      "postDate": "2017-02-17T01:25:41.013Z",
      "content": "<p>Congrats and thanks for your sharing!</p>\n\n<p>The idea of using Neural Network is interesting.</p>\n\n<p>Can you tell the architecture of Neural Networks?</p>",
      "rawMarkdown": "Congrats and thanks for your sharing!\n\nThe idea of using Neural Network is interesting.\n\nCan you tell the architecture of Neural Networks?"
    },
    {
      "id": 151798,
      "postDate": "2016-12-22T05:06:48.413Z",
      "content": "<p>Thanks for sharing, Faron! And of course: Congratulations!</p>\n\n<p>Following up on the question above: What does oof-encoding in general? I tried to googling it, but did not find an answer...</p>",
      "rawMarkdown": "Thanks for sharing, Faron! And of course: Congratulations!\r\n\r\nFollowing up on the question above: What does oof-encoding in general? I tried to googling it, but did not find an answer..."
    },
    {
      "id": 151480,
      "postDate": "2016-12-20T23:13:46.597Z",
      "content": "<p>Thanks for sharing!</p>\n\n<p>I have a question.</p>\n\n<p>What is target loss encoding? To encode categories with the mean of the target loss?</p>",
      "rawMarkdown": "Thanks for sharing!\r\n\r\nI have a question.\r\n\r\n What is target loss encoding? To encode categories with the mean of the target loss?\r\n"
    },
    {
      "id": 150313,
      "postDate": "2016-12-14T18:37:07.723Z",
      "content": "<p>@Faron, question that I have for more than a month now. You mentioned in another thread that you got 1107 on a raw features on the log transformed target.</p>\n\n<p>What were your hyperparameters?</p>",
      "rawMarkdown": "@Faron, question that I have for more than a month now. You mentioned in another thread that you got 1107 on a raw features on the log transformed target.\r\n\r\nWhat were your hyperparameters?"
    },
    {
      "id": 150177,
      "postDate": "2016-12-14T06:07:34.327Z",
      "content": "<p>can someone confirm that Feature Combination (modkzs) (1108.72665)  gave better CVs with Keras model?</p>",
      "rawMarkdown": "can someone confirm that Feature Combination (modkzs) (1108.72665)  gave better CVs with Keras model?"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:15:38Z",
  "competition_id": 5325,
  "competition_slug": "allstate-claims-severity",
  "team_id": 383370,
  "solution_url": "/competitions/allstate-claims-severity/writeups/faron-faron-s-3rd-place-solution"
}