{
  "id": 25434,
  "title": "1st place solution",
  "url": "/competitions/bosch-production-line-performance/writeups/ash-beluga-1st-place-solution",
  "author_name": "",
  "post_date": "2016-11-14T16:27:24.080Z",
  "votes": 80,
  "comment_count": 24,
  "views": 3112,
  "content": "<p>Sorry for the delay. It was good to have a kaggle free weekend after two months. (Ok not completely kaggle free but at least in read only mode:).</p>\n\n<p>As we had 6-7 hours local time difference (USA, EU) we tried to split the work. \nI was mostly doing feature engineering and tried to improve our best single xgb score. Ash tried all kinds of different modeling techniques and built the L2/L3 ensemble.</p>\n\n<h2>Data Exploration</h2>\n\n<p>I spent the first two weeks with data discovery. Numeric feature plots, statistics, station to station transition probabilities, correlation matrices for each stations was necessary to have ideas how to handle the thousands of anonymous features.</p>\n\n<p>I start each competition with desktop machine (16GB RAM). It takes some extra effort in the beginning to deal with lower level data manipulations due to the memory constraints. Getting closer to the raw data helps sometimes later. </p>\n\n<p>As the numeric/date features had .3f/.2f precision we could rescale (*=1000, *=100) everything to integers. We kept only the min time for each station.\nSparse matrices came handy as we had a lot of missing values. </p>\n\n<h2>Leak/Magic/Data Property Features</h2>\n\n<p>Ash found quite early that consecutive rows had feature duplication and correlated Response. We called consecutive Ids ordered by StartStation, StartTime *chunk*s.</p>\n\n<p>For each chunk we used features</p>\n\n<ul>\n<li>number of records</li>\n<li>rank ascending</li>\n<li>rank descending</li>\n</ul>\n\n<p>We used the original order to add features based on previous and next records.</p>\n\n<ul>\n<li>Response</li>\n<li>feature hash equality for each station</li>\n<li>StartTime, EndTime</li>\n<li>Numeric raw features</li>\n</ul>\n\n<h2>Time Features</h2>\n\n<p>I figured out in the beginning that <a href=\"https://www.kaggle.com/gaborfodor/bosch-production-line-performance/notebookd19d11e4f2\">0.01 time granularity means probably 6 mins</a>.</p>\n\n<p>So we had to deal with 2 years of manufacturing data. This observation did not gave direct performance boost but gave us enough intuition to construct a lot of time based features.</p>\n\n<ul>\n<li>StartStationTimes</li>\n<li>StartTime, EndTime, Duration</li>\n<li>StationTimeDiff</li>\n<li>Start/End part of week (mod 1680)</li>\n<li>Number of records in next/last 2.5h, 24h, 168h for each station</li>\n<li>Number of records in the same time (6 mins)</li>\n<li>MeanTimeDiff since last 1/5/10 failure(s)</li>\n<li>MeanTimeDiff till next 1/5/10 failure(s)</li>\n</ul>\n\n<h2>Numeric Features</h2>\n\n<ul>\n<li>Raw numeric features (most of the time we used the raw numeric features or simple subsets based on xgb feature importance)</li>\n<li>Z-scaled features for each week</li>\n<li>Count encoding for each value </li>\n<li>Feature combinations (f1 + -  * f2)</li>\n</ul>\n\n<p>We saw interesting similarity in the station transition probability matrix between station S0-S11 and S12 - S23. You have probably seen the same in the Shopfloor visualizations. We also noticed that the number of features are the same across these station groups.</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/144544/5351/StationFeaturSimilarity.png?sv=2012-02-12&se=2016-11-17T16:25:47Z&sr=b&sp=r&sig=pEgr%2B4FRBrS2Owl2ctg5sOEMBzMbWQoL78R%2FEed8h%2BA%3D\" alt=\"enter image description here\" title></p>\n\n<p>The correlation matrices were similar for these stations so we tried combining the numeric features for the same stages. (e.g. L0_S0_F0 + L0_S12_F330 etc.)</p>\n\n<h2>Categorical Features</h2>\n\n<p>We could not squeeze much out of the categorical features. Most of the time we just dropped them. We had a few models with aggregated categorical features for each station in [24, 25, 26, 27, 28, 29, 32, 44, 47]. 'T1' was replaced by 1 all the other values were replaced 100.</p>\n\n<h2>Validation</h2>\n\n<p>Given the imbalanced labels and binary evaluation criteria we noticed quite high 5-FOLD CV stds (~0.01) in the beginning. After a few experiments we decided to use 4 FOLD &quot;leak-stratified&quot; CV (forcing to have the same number of duplication for each fold). \nEach &quot;single&quot; L1 model was trained with 3 different seeds on the same folds. Our latest ensemble scores were quite stable (std &lt; 0.004) and very often our CV/LB score improved hand in hand.</p>\n\n<h2>Rebalancing</h2>\n\n<p>We kept every failure and records with duplication. For the remaining 0s 50-90% down sampling was used. It made the training a bit faster and worked as bagging for the model averaging part helping the later ensemble stages.</p>\n\n<h2>Results</h2>\n\n<p>In the last month of the competition we continuously improved and tried to keep us in the top 3. Fortunately we could improve significantly both CV and LB  with our last submissions on Friday. This made the submission selection easier.\n<img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/144544/5348/results.PNG?sv=2012-02-12&se=2016-11-17T16:01:24Z&sr=b&sp=r&sig=%2FBvQoJg5kEuzvBWFBoHFs7GU%2Fgr7AotglS54yYgZk3M%3D\" alt=\"enter image description here\" title></p>",
  "messages": [
    {
      "id": "144544",
      "postDate": "11/14/2016 15:51:36",
      "content": "<p>Sorry for the delay. It was good to have a kaggle free weekend after two months. (Ok not completely kaggle free but at least in read only mode:).</p>\n\n<p>As we had 6-7 hours local time difference (USA, EU) we tried to split the work. \nI was mostly doing feature engineering and tried to improve our best single xgb score. Ash tried all kinds of different modeling techniques and built the L2/L3 ensemble.</p>\n\n<h2>Data Exploration</h2>\n\n<p>I spent the first two weeks with data discovery. Numeric feature plots, statistics, station to station transition probabilities, correlation matrices for each stations was necessary to have ideas how to handle the thousands of anonymous features.</p>\n\n<p>I start each competition with desktop machine (16GB RAM). It takes some extra effort in the beginning to deal with lower level data manipulations due to the memory constraints. Getting closer to the raw data helps sometimes later. </p>\n\n<p>As the numeric/date features had .3f/.2f precision we could rescale (*=1000, *=100) everything to integers. We kept only the min time for each station.\nSparse matrices came handy as we had a lot of missing values. </p>\n\n<h2>Leak/Magic/Data Property Features</h2>\n\n<p>Ash found quite early that consecutive rows had feature duplication and correlated Response. We called consecutive Ids ordered by StartStation, StartTime *chunk*s.</p>\n\n<p>For each chunk we used features</p>\n\n<ul>\n<li>number of records</li>\n<li>rank ascending</li>\n<li>rank descending</li>\n</ul>\n\n<p>We used the original order to add features based on previous and next records.</p>\n\n<ul>\n<li>Response</li>\n<li>feature hash equality for each station</li>\n<li>StartTime, EndTime</li>\n<li>Numeric raw features</li>\n</ul>\n\n<h2>Time Features</h2>\n\n<p>I figured out in the beginning that <a href=\"https://www.kaggle.com/gaborfodor/bosch-production-line-performance/notebookd19d11e4f2\">0.01 time granularity means probably 6 mins</a>.</p>\n\n<p>So we had to deal with 2 years of manufacturing data. This observation did not gave direct performance boost but gave us enough intuition to construct a lot of time based features.</p>\n\n<ul>\n<li>StartStationTimes</li>\n<li>StartTime, EndTime, Duration</li>\n<li>StationTimeDiff</li>\n<li>Start/End part of week (mod 1680)</li>\n<li>Number of records in next/last 2.5h, 24h, 168h for each station</li>\n<li>Number of records in the same time (6 mins)</li>\n<li>MeanTimeDiff since last 1/5/10 failure(s)</li>\n<li>MeanTimeDiff till next 1/5/10 failure(s)</li>\n</ul>\n\n<h2>Numeric Features</h2>\n\n<ul>\n<li>Raw numeric features (most of the time we used the raw numeric features or simple subsets based on xgb feature importance)</li>\n<li>Z-scaled features for each week</li>\n<li>Count encoding for each value </li>\n<li>Feature combinations (f1 + -  * f2)</li>\n</ul>\n\n<p>We saw interesting similarity in the station transition probability matrix between station S0-S11 and S12 - S23. You have probably seen the same in the Shopfloor visualizations. We also noticed that the number of features are the same across these station groups.</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/144544/5351/StationFeaturSimilarity.png?sv=2012-02-12&se=2016-11-17T16:25:47Z&sr=b&sp=r&sig=pEgr%2B4FRBrS2Owl2ctg5sOEMBzMbWQoL78R%2FEed8h%2BA%3D\" alt=\"enter image description here\" title></p>\n\n<p>The correlation matrices were similar for these stations so we tried combining the numeric features for the same stages. (e.g. L0_S0_F0 + L0_S12_F330 etc.)</p>\n\n<h2>Categorical Features</h2>\n\n<p>We could not squeeze much out of the categorical features. Most of the time we just dropped them. We had a few models with aggregated categorical features for each station in [24, 25, 26, 27, 28, 29, 32, 44, 47]. 'T1' was replaced by 1 all the other values were replaced 100.</p>\n\n<h2>Validation</h2>\n\n<p>Given the imbalanced labels and binary evaluation criteria we noticed quite high 5-FOLD CV stds (~0.01) in the beginning. After a few experiments we decided to use 4 FOLD &quot;leak-stratified&quot; CV (forcing to have the same number of duplication for each fold). \nEach &quot;single&quot; L1 model was trained with 3 different seeds on the same folds. Our latest ensemble scores were quite stable (std &lt; 0.004) and very often our CV/LB score improved hand in hand.</p>\n\n<h2>Rebalancing</h2>\n\n<p>We kept every failure and records with duplication. For the remaining 0s 50-90% down sampling was used. It made the training a bit faster and worked as bagging for the model averaging part helping the later ensemble stages.</p>\n\n<h2>Results</h2>\n\n<p>In the last month of the competition we continuously improved and tried to keep us in the top 3. Fortunately we could improve significantly both CV and LB  with our last submissions on Friday. This made the submission selection easier.\n<img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/144544/5348/results.PNG?sv=2012-02-12&se=2016-11-17T16:01:24Z&sr=b&sp=r&sig=%2FBvQoJg5kEuzvBWFBoHFs7GU%2Fgr7AotglS54yYgZk3M%3D\" alt=\"enter image description here\" title></p>",
      "rawMarkdown": "Sorry for the delay. It was good to have a kaggle free weekend after two months. (Ok not completely kaggle free but at least in read only mode:).\r\n\r\nAs we had 6-7 hours local time difference (USA, EU) we tried to split the work. \r\nI was mostly doing feature engineering and tried to improve our best single xgb score. Ash tried all kinds of different modeling techniques and built the L2/L3 ensemble.\r\n\r\n## Data Exploration\r\nI spent the first two weeks with data discovery. Numeric feature plots, statistics, station to station transition probabilities, correlation matrices for each stations was necessary to have ideas how to handle the thousands of anonymous features.\r\n\r\nI start each competition with desktop machine (16GB RAM). It takes some extra effort in the beginning to deal with lower level data manipulations due to the memory constraints. Getting closer to the raw data helps sometimes later. \r\n\r\nAs the numeric/date features had .3f/.2f precision we could rescale (*=1000, *=100) everything to integers. We kept only the min time for each station.\r\nSparse matrices came handy as we had a lot of missing values. \r\n\r\n\r\n## Leak/Magic/Data Property Features\r\nAsh found quite early that consecutive rows had feature duplication and correlated Response. We called consecutive Ids ordered by StartStation, StartTime *chunk*s.\r\n\r\nFor each chunk we used features\r\n\r\n* number of records\r\n* rank ascending\r\n* rank descending\r\n\r\nWe used the original order to add features based on previous and next records.\r\n\r\n* Response\r\n* feature hash equality for each station\r\n* StartTime, EndTime\r\n* Numeric raw features\r\n\r\n## Time Features\r\nI figured out in the beginning that [0.01 time granularity means probably 6 mins][1].\r\n\r\nSo we had to deal with 2 years of manufacturing data. This observation did not gave direct performance boost but gave us enough intuition to construct a lot of time based features.\r\n\r\n* StartStationTimes\r\n* StartTime, EndTime, Duration\r\n* StationTimeDiff\r\n* Start/End part of week (mod 1680)\r\n* Number of records in next/last 2.5h, 24h, 168h for each station\r\n* Number of records in the same time (6 mins)\r\n* MeanTimeDiff since last 1/5/10 failure(s)\r\n* MeanTimeDiff till next 1/5/10 failure(s)\r\n\r\n\r\n## Numeric Features\r\n* Raw numeric features (most of the time we used the raw numeric features or simple subsets based on xgb feature importance)\r\n* Z-scaled features for each week\r\n* Count encoding for each value \r\n* Feature combinations (f1 + -  * f2)\r\n\r\nWe saw interesting similarity in the station transition probability matrix between station S0-S11 and S12 - S23. You have probably seen the same in the Shopfloor visualizations. We also noticed that the number of features are the same across these station groups.\r\n\r\n![enter image description here][2]\r\n\r\nThe correlation matrices were similar for these stations so we tried combining the numeric features for the same stages. (e.g. L0_S0_F0 + L0_S12_F330 etc.)\r\n\r\n\r\n\r\n## Categorical Features\r\nWe could not squeeze much out of the categorical features. Most of the time we just dropped them. We had a few models with aggregated categorical features for each station in [24, 25, 26, 27, 28, 29, 32, 44, 47]. 'T1' was replaced by 1 all the other values were replaced 100.\r\n\r\n## Validation\r\nGiven the imbalanced labels and binary evaluation criteria we noticed quite high 5-FOLD CV stds (~0.01) in the beginning. After a few experiments we decided to use 4 FOLD \"leak-stratified\" CV (forcing to have the same number of duplication for each fold). \r\nEach \"single\" L1 model was trained with 3 different seeds on the same folds. Our latest ensemble scores were quite stable (std < 0.004) and very often our CV/LB score improved hand in hand.\r\n\r\n## Rebalancing\r\nWe kept every failure and records with duplication. For the remaining 0s 50-90% down sampling was used. It made the training a bit faster and worked as bagging for the model averaging part helping the later ensemble stages.\r\n\r\n## Results\r\nIn the last month of the competition we continuously improved and tried to keep us in the top 3. Fortunately we could improve significantly both CV and LB  with our last submissions on Friday. This made the submission selection easier.\r\n![enter image description here][3]\r\n\r\n\r\n  [1]: https://www.kaggle.com/gaborfodor/bosch-production-line-performance/notebookd19d11e4f2\r\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/144544/5351/StationFeaturSimilarity.png?sv=2012-02-12&se=2016-11-17T16%3A25%3A47Z&sr=b&sp=r&sig=pEgr%2B4FRBrS2Owl2ctg5sOEMBzMbWQoL78R%2FEed8h%2BA%3D\r\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/144544/5348/results.PNG?sv=2012-02-12&se=2016-11-17T16%3A01%3A24Z&sr=b&sp=r&sig=%2FBvQoJg5kEuzvBWFBoHFs7GU%2Fgr7AotglS54yYgZk3M%3D",
      "votes": null
    },
    {
      "id": "144552",
      "postDate": "11/14/2016 16:19:23",
      "content": "<p>Thanks a lot for sharing (plus the kernels you shared too).</p>\n\n<p>Can you explain a bit more this:</p>\n\n<ul>\n<li>MeanTimeDiff since last 1/5/10 failure(s)</li>\n<li>MeanTimeDiff till next 1/5/10 failure(s)</li>\n</ul>\n\n<p>Mean over what?  </p>",
      "rawMarkdown": "Thanks a lot for sharing (plus the kernels you shared too).\r\n\r\nCan you explain a bit more this:\r\n\r\n- MeanTimeDiff since last 1/5/10 failure(s)\r\n- MeanTimeDiff till next 1/5/10 failure(s)\r\n\r\nMean over what?",
      "votes": null
    },
    {
      "id": "144553",
      "postDate": "11/14/2016 16:23:53",
      "content": "<p>Impressive !</p>",
      "rawMarkdown": "Impressive !",
      "votes": null
    },
    {
      "id": "144558",
      "postDate": "11/14/2016 16:39:27",
      "content": "<p>[quote=CPMP;144552]\nCan you explain a bit more this:</p>\n\n<ul>\n<li>MeanTimeDiff since last 1/5/10 failure(s)</li>\n<li>MeanTimeDiff till next 1/5/10 failure(s)</li>\n</ul>\n\n<p>Mean over what? <br>\n[/quote]</p>\n\n<p>Mean(EndTime for latest K failures where EndTime &lt; CurrentEndTime) - CurrentEndTime</p>\n\n<p>Mean(EndTime for the first K failures where EndTime &gt; CurrentEndTime) - CurrentEndTime</p>",
      "rawMarkdown": "[quote=CPMP;144552]\r\nCan you explain a bit more this:\r\n\r\n- MeanTimeDiff since last 1/5/10 failure(s)\r\n- MeanTimeDiff till next 1/5/10 failure(s)\r\n\r\nMean over what?  \r\n[/quote]\r\n\r\nMean(EndTime for latest K failures where EndTime < CurrentEndTime) - CurrentEndTime\r\n\r\nMean(EndTime for the first K failures where EndTime > CurrentEndTime) - CurrentEndTime",
      "votes": null
    },
    {
      "id": "144568",
      "postDate": "11/14/2016 17:07:56",
      "content": "<p>Thanks.</p>",
      "rawMarkdown": "Thanks.",
      "votes": null
    },
    {
      "id": "144584",
      "postDate": "11/14/2016 18:11:33",
      "content": "<p>Awesome job Beluga and Ash; you crushed it!  A few follow up questions:</p>\n\n<ol>\n<li><p>For &quot;leak stratified&quot;, does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?</p></li>\n<li><p>What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?</p></li>\n</ol>",
      "rawMarkdown": "Awesome job Beluga and Ash; you crushed it!  A few follow up questions:\r\n\r\n1. For \"leak stratified\", does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?\r\n\r\n2. What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?",
      "votes": null
    },
    {
      "id": "144628",
      "postDate": "11/14/2016 21:53:55",
      "content": "<p>[quote=Russ W;144584]</p>\n\n<ol>\n<li><p>For &quot;leak stratified&quot;, does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?</p></li>\n<li><p>What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?</p></li>\n</ol>\n\n<p>[/quote]</p>\n\n<ol>\n<li><p>We allowed the same chunk to appear in multiple folds. We assumed that Train/Test and Public/Private were simple random splits.</p></li>\n<li><p>I tried ExtraTreesRegressor a few times. Deeper trees worked better for me but we had all sorts of different xgbs in the ensemble.\nOne of the best xgb parameters:</p>\n\n<ul><li>early stopping <strong>auc</strong></li>\n<li>alpha <strong>0</strong></li>\n<li>booster <strong>gbtree</strong></li>\n<li>colsample_bytree <strong>0.6</strong></li>\n<li>min_child_weight <strong>5</strong></li>\n<li>subsample    <strong>0.9</strong></li>\n<li>eta  <strong>0.03</strong></li>\n<li>objective    <strong>binary:logistic</strong></li>\n<li>max_depth    <strong>14</strong></li>\n<li>lambda    <strong>4</strong></li></ul></li>\n</ol>\n\n<p>I found it strange that decreasing eta did not help to get better models.</p>",
      "rawMarkdown": "[quote=Russ W;144584]\r\n\r\n1. For \"leak stratified\", does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?\r\n\r\n2. What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?\r\n\r\n[/quote]\r\n\r\n1. We allowed the same chunk to appear in multiple folds. We assumed that Train/Test and Public/Private were simple random splits.\r\n\r\n2. I tried ExtraTreesRegressor a few times. Deeper trees worked better for me but we had all sorts of different xgbs in the ensemble.\r\nOne of the best xgb parameters:\r\n * early stopping **auc**\r\n * alpha **0**\r\n * booster **gbtree**\r\n * colsample_bytree\t**0.6**\r\n * min_child_weight\t**5**\r\n * subsample\t**0.9**\r\n * eta\t**0.03**\r\n * objective\t**binary:logistic**\r\n * max_depth\t**14**\r\n* lambda\t**4**\r\n \r\nI found it strange that decreasing eta did not help to get better models.",
      "votes": null
    },
    {
      "id": "144788",
      "postDate": "11/15/2016 12:03:34",
      "content": "<p>very impressive overall work , congratulations !</p>",
      "rawMarkdown": "very impressive overall work , congratulations !",
      "votes": null
    },
    {
      "id": "144798",
      "postDate": "11/15/2016 12:51:16",
      "content": "<p>I also found that small eta did not really help, hence I stick with a quite large value for it as it yield way faster training times.</p>",
      "rawMarkdown": "I also found that small eta did not really help, hence I stick with a quite large value for it as it yield way faster training times.",
      "votes": null
    },
    {
      "id": "145000",
      "postDate": "11/16/2016 09:02:36",
      "content": "<p>great work! congratulations!</p>",
      "rawMarkdown": "great work! congratulations!",
      "votes": null
    },
    {
      "id": "145046",
      "postDate": "11/16/2016 14:48:55",
      "content": "<p>Congratulations!\nThere is a question about how to 'count'...... Which way is proper?\n1) Combine traindata and testdata to groupby().count(), and split them, or \n2) traindata.groupby().count() and testdata.groupby().count(), or\n3) traindata.groupby().count() and testdata.merge(, on='').\nThanks ;)</p>",
      "rawMarkdown": "Congratulations!\r\nThere is a question about how to 'count'...... Which way is proper?\r\n1) Combine traindata and testdata to groupby().count(), and split them, or \r\n2) traindata.groupby().count() and testdata.groupby().count(), or\r\n3) traindata.groupby().count() and testdata.merge(, on='').\r\nThanks ;)",
      "votes": null
    },
    {
      "id": "145050",
      "postDate": "11/16/2016 15:01:40",
      "content": "<p>Congratulations guys. Just to get a feel for, how many man hours in total do you reckon you have put into this competition?</p>",
      "rawMarkdown": "Congratulations guys. Just to get a feel for, how many man hours in total do you reckon you have put into this competition?",
      "votes": null
    },
    {
      "id": "145052",
      "postDate": "11/16/2016 15:08:12",
      "content": "<p>Thank you for the solution! \nYour effort is really appreciate, especially that two weeks for data discovery.</p>",
      "rawMarkdown": "Thank you for the solution! \r\nYour effort is really appreciate, especially that two weeks for data discovery.",
      "votes": null
    },
    {
      "id": "145156",
      "postDate": "11/17/2016 00:49:13",
      "content": "<p>Z-scaled features for each week</p>\n\n<p>What is Z-Scaled ? </p>",
      "rawMarkdown": "Z-scaled features for each week\r\n\r\nWhat is Z-Scaled ?",
      "votes": null
    },
    {
      "id": "145210",
      "postDate": "11/17/2016 09:09:03",
      "content": "<p>I guess it is standardization (substract mean then divide by variance), see for instance </p>\n\n<p><a href=\"http://sebastianraschka.com/Articles/2014_about_feature_scaling.html\">http://sebastianraschka.com/Articles/2014_about_feature_scaling.html</a></p>\n\n<p>or</p>\n\n<p><a href=\"https://en.wikipedia.org/wiki/Standard_score\">https://en.wikipedia.org/wiki/Standard_score</a></p>\n\n<p>Goal is to have a mean of 0 and variance of 1.</p>\n\n<p>Note that original numeric values were already having a mean of 0.  Variance was not one, but we have <code>max - min   &lt;= 1</code>, which is another way to standardize.</p>",
      "rawMarkdown": "I guess it is standardization (substract mean then divide by variance), see for instance \r\n\r\nhttp://sebastianraschka.com/Articles/2014_about_feature_scaling.html\r\n\r\nor\r\n\r\nhttps://en.wikipedia.org/wiki/Standard_score\r\n\r\nGoal is to have a mean of 0 and variance of 1.\r\n\r\nNote that original numeric values were already having a mean of 0.  Variance was not one, but we have `max - min   <= 1`, which is another way to standardize.",
      "votes": null
    },
    {
      "id": "145373",
      "postDate": "11/18/2016 01:16:15",
      "content": "<p>Wow, this is awesome! :)</p>",
      "rawMarkdown": "Wow, this is awesome! :)",
      "votes": null
    },
    {
      "id": "145506",
      "postDate": "11/18/2016 19:44:17",
      "content": "<p>Can you explain more about your L2/L3 ensemble?</p>",
      "rawMarkdown": "Can you explain more about your L2/L3 ensemble?",
      "votes": null
    },
    {
      "id": "145530",
      "postDate": "11/18/2016 23:53:13",
      "content": "<p>[quote=HueyKwik;145506]</p>\n\n<p>Can you explain more about your L2/L3 ensemble?</p>\n\n<p>[/quote]</p>\n\n<p>As beluga already mentioned, we ran a bagged, 4-fold cv, on a shared leak-stratified split. </p>\n\n<p>All the L1 models were trained multiple times per fold, each with a different seed. Each of these runs predicted on both the hold-out set, as well as the test set. The averaged hold-out and test predictions for each fold were stored for use in L2.</p>\n\n<p>In L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. Again the same 4-fold splits were used, predictions made and bagged on both hold-out and test set, and saved. These predictions were also rank normalized. </p>\n\n<p>To generate a submission based on an L2 model, the 4 test set predictions were averaged together, while the 4 holdout set predictions were combined into one set, and used to determine the thresholding percentage of 1s to use. That same percentage of 1s was then used to find the corresponding threshold level value to use in the test set. </p>\n\n<p>L3 was similar - it built on top of the saved L1 predictions and some of the saved L2 predictions.</p>",
      "rawMarkdown": "[quote=HueyKwik;145506]\r\n\r\nCan you explain more about your L2/L3 ensemble?\r\n\r\n[/quote]\r\n\r\nAs beluga already mentioned, we ran a bagged, 4-fold cv, on a shared leak-stratified split. \r\n\r\nAll the L1 models were trained multiple times per fold, each with a different seed. Each of these runs predicted on both the hold-out set, as well as the test set. The averaged hold-out and test predictions for each fold were stored for use in L2.\r\n\r\nIn L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. Again the same 4-fold splits were used, predictions made and bagged on both hold-out and test set, and saved. These predictions were also rank normalized. \r\n\r\nTo generate a submission based on an L2 model, the 4 test set predictions were averaged together, while the 4 holdout set predictions were combined into one set, and used to determine the thresholding percentage of 1s to use. That same percentage of 1s was then used to find the corresponding threshold level value to use in the test set. \r\n\r\nL3 was similar - it built on top of the saved L1 predictions and some of the saved L2 predictions.",
      "votes": null
    },
    {
      "id": "145840",
      "postDate": "11/21/2016 11:26:40",
      "content": "<p>[quote=ash hafez;145530]</p>\n\n<p>[quote=HueyKwik;145506]</p>\n\n<p>[/quote]</p>\n\n<p>In L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. </p>\n\n<p>[/quote]</p>\n\n<p>Thanks for the solution share,  another question about the max rank normalized processing.\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?</p>",
      "rawMarkdown": "[quote=ash hafez;145530]\r\n\r\n[quote=HueyKwik;145506]\r\n\r\n[/quote]\r\n\r\nIn L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. \r\n\r\n[/quote]\r\n\r\nThanks for the solution share,  another question about the max rank normalized processing.\r\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?",
      "votes": null
    },
    {
      "id": "145954",
      "postDate": "11/21/2016 23:54:13",
      "content": "<p>[quote=Zongyi Liu;145840]</p>\n\n<p>[quote=ash hafez;145530]</p>\n\n<p>[quote=HueyKwik;145506]</p>\n\n<p>[/quote]</p>\n\n<p>In L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. </p>\n\n<p>[/quote]</p>\n\n<p>Thanks for the solution share,  another question about the max rank normalized processing.\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?</p>\n\n<p>[/quote]</p>\n\n<p>When you rank the fold predictions, you can divide by the maximum rank to turn it into a range between 0 and 1. Do the same for the test predictions as well. Then assuming you are stratified in your folds, and the target response rate in test matches train in general, you can translate the top x% of your ranked normalized predictions from train to an equivalent percentage in the test set.</p>",
      "rawMarkdown": "[quote=Zongyi Liu;145840]\r\n\r\n[quote=ash hafez;145530]\r\n\r\n[quote=HueyKwik;145506]\r\n\r\n[/quote]\r\n\r\nIn L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. \r\n\r\n[/quote]\r\n\r\nThanks for the solution share,  another question about the max rank normalized processing.\r\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?\r\n\r\n[/quote]\r\n\r\nWhen you rank the fold predictions, you can divide by the maximum rank to turn it into a range between 0 and 1. Do the same for the test predictions as well. Then assuming you are stratified in your folds, and the target response rate in test matches train in general, you can translate the top x% of your ranked normalized predictions from train to an equivalent percentage in the test set.",
      "votes": null
    },
    {
      "id": "173499",
      "postDate": "04/07/2017 06:38:21",
      "content": "<p>Hi Beluga\n  Belated  Congrats .  I know its almost 5 months back. But i am new to kaggle  by 2 weeks.\nI am new to data science as well. </p>\n\n<p>I am working on this as POC in our company.</p>\n\n<p>Can you share what and all steps you did from the beginning to the end.\nI know i am bit <em>ambitious</em> here asking you to provide each piece of details.</p>\n\n<p>But if possible what steps you have taken can you share , like why certain features were selected, some discared. </p>\n\n<p>Thanks </p>",
      "rawMarkdown": "Hi Beluga\n  Belated  Congrats .  I know its almost 5 months back. But i am new to kaggle  by 2 weeks.\nI am new to data science as well. \n\n I am working on this as POC in our company.\n\nCan you share what and all steps you did from the beginning to the end.\nI know i am bit *ambitious* here asking you to provide each piece of details.\n\n But if possible what steps you have taken can you share , like why certain features were selected, some discared. \n\nThanks",
      "votes": null
    },
    {
      "id": "837071",
      "postDate": "05/07/2020 13:54:12",
      "content": "<p>Could you please share your code, thanks:)</p>",
      "rawMarkdown": "Could you please share your code, thanks:)",
      "votes": null
    },
    {
      "id": "849560",
      "postDate": "05/15/2020 22:32:28",
      "content": "<p>Any chance of sharing your code? It would be highly appreciated, thank you!</p>",
      "rawMarkdown": "Any chance of sharing your code? It would be highly appreciated, thank you!",
      "votes": null
    },
    {
      "id": "1673300",
      "postDate": "02/02/2022 16:28:06",
      "content": "<p>i am using this data set for benchmarking but clearly there is the test data's labels that are missing , could you please share the labels that were predicted for the test data if its possible?</p>",
      "rawMarkdown": "i am using this data set for benchmarking but clearly there is the test data's labels that are missing , could you please share the labels that were predicted for the test data if its possible?",
      "votes": null
    },
    {
      "id": "3274668",
      "postDate": "08/25/2025 08:43:45",
      "content": "<p>thanks for the details</p>",
      "rawMarkdown": "thanks for the details",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1673300,
      "author_name": "mohammednadeem7673",
      "author_url": "",
      "post_date": "02/02/2022 16:28:06",
      "content": "<p>i am using this data set for benchmarking but clearly there is the test data's labels that are missing , could you please share the labels that were predicted for the test data if its possible?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3274668,
      "author_name": "xiaoman521",
      "author_url": "",
      "post_date": "08/25/2025 08:43:45",
      "content": "<p>thanks for the details</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144552,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "11/14/2016 16:19:23",
      "content": "<p>Thanks a lot for sharing (plus the kernels you shared too).</p>\n\n<p>Can you explain a bit more this:</p>\n\n<ul>\n<li>MeanTimeDiff since last 1/5/10 failure(s)</li>\n<li>MeanTimeDiff till next 1/5/10 failure(s)</li>\n</ul>\n\n<p>Mean over what?  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144553,
      "author_name": "alanwuxm",
      "author_url": "",
      "post_date": "11/14/2016 16:23:53",
      "content": "<p>Impressive !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144558,
      "author_name": "gaborfodor",
      "author_url": "",
      "post_date": "11/14/2016 16:39:27",
      "content": "<p>[quote=CPMP;144552]\nCan you explain a bit more this:</p>\n\n<ul>\n<li>MeanTimeDiff since last 1/5/10 failure(s)</li>\n<li>MeanTimeDiff till next 1/5/10 failure(s)</li>\n</ul>\n\n<p>Mean over what? <br>\n[/quote]</p>\n\n<p>Mean(EndTime for latest K failures where EndTime &lt; CurrentEndTime) - CurrentEndTime</p>\n\n<p>Mean(EndTime for the first K failures where EndTime &gt; CurrentEndTime) - CurrentEndTime</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144568,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "11/14/2016 17:07:56",
      "content": "<p>Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144584,
      "author_name": "sasrdw",
      "author_url": "",
      "post_date": "11/14/2016 18:11:33",
      "content": "<p>Awesome job Beluga and Ash; you crushed it!  A few follow up questions:</p>\n\n<ol>\n<li><p>For &quot;leak stratified&quot;, does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?</p></li>\n<li><p>What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?</p></li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144628,
      "author_name": "gaborfodor",
      "author_url": "",
      "post_date": "11/14/2016 21:53:55",
      "content": "<p>[quote=Russ W;144584]</p>\n\n<ol>\n<li><p>For &quot;leak stratified&quot;, does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?</p></li>\n<li><p>What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?</p></li>\n</ol>\n\n<p>[/quote]</p>\n\n<ol>\n<li><p>We allowed the same chunk to appear in multiple folds. We assumed that Train/Test and Public/Private were simple random splits.</p></li>\n<li><p>I tried ExtraTreesRegressor a few times. Deeper trees worked better for me but we had all sorts of different xgbs in the ensemble.\nOne of the best xgb parameters:</p>\n\n<ul><li>early stopping <strong>auc</strong></li>\n<li>alpha <strong>0</strong></li>\n<li>booster <strong>gbtree</strong></li>\n<li>colsample_bytree <strong>0.6</strong></li>\n<li>min_child_weight <strong>5</strong></li>\n<li>subsample    <strong>0.9</strong></li>\n<li>eta  <strong>0.03</strong></li>\n<li>objective    <strong>binary:logistic</strong></li>\n<li>max_depth    <strong>14</strong></li>\n<li>lambda    <strong>4</strong></li></ul></li>\n</ol>\n\n<p>I found it strange that decreasing eta did not help to get better models.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144788,
      "author_name": "neongen",
      "author_url": "",
      "post_date": "11/15/2016 12:03:34",
      "content": "<p>very impressive overall work , congratulations !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144798,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "11/15/2016 12:51:16",
      "content": "<p>I also found that small eta did not really help, hence I stick with a quite large value for it as it yield way faster training times.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145000,
      "author_name": "schtirlitz",
      "author_url": "",
      "post_date": "11/16/2016 09:02:36",
      "content": "<p>great work! congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145046,
      "author_name": "hczxwyx",
      "author_url": "",
      "post_date": "11/16/2016 14:48:55",
      "content": "<p>Congratulations!\nThere is a question about how to 'count'...... Which way is proper?\n1) Combine traindata and testdata to groupby().count(), and split them, or \n2) traindata.groupby().count() and testdata.groupby().count(), or\n3) traindata.groupby().count() and testdata.merge(, on='').\nThanks ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145050,
      "author_name": "",
      "author_url": "",
      "post_date": "11/16/2016 15:01:40",
      "content": "<p>Congratulations guys. Just to get a feel for, how many man hours in total do you reckon you have put into this competition?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145052,
      "author_name": "a11rand0m",
      "author_url": "",
      "post_date": "11/16/2016 15:08:12",
      "content": "<p>Thank you for the solution! \nYour effort is really appreciate, especially that two weeks for data discovery.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145156,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "11/17/2016 00:49:13",
      "content": "<p>Z-scaled features for each week</p>\n\n<p>What is Z-Scaled ? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145210,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "11/17/2016 09:09:03",
      "content": "<p>I guess it is standardization (substract mean then divide by variance), see for instance </p>\n\n<p><a href=\"http://sebastianraschka.com/Articles/2014_about_feature_scaling.html\">http://sebastianraschka.com/Articles/2014_about_feature_scaling.html</a></p>\n\n<p>or</p>\n\n<p><a href=\"https://en.wikipedia.org/wiki/Standard_score\">https://en.wikipedia.org/wiki/Standard_score</a></p>\n\n<p>Goal is to have a mean of 0 and variance of 1.</p>\n\n<p>Note that original numeric values were already having a mean of 0.  Variance was not one, but we have <code>max - min   &lt;= 1</code>, which is another way to standardize.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145373,
      "author_name": "vuchetichbalint",
      "author_url": "",
      "post_date": "11/18/2016 01:16:15",
      "content": "<p>Wow, this is awesome! :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145506,
      "author_name": "",
      "author_url": "",
      "post_date": "11/18/2016 19:44:17",
      "content": "<p>Can you explain more about your L2/L3 ensemble?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145530,
      "author_name": "ashhafez",
      "author_url": "",
      "post_date": "11/18/2016 23:53:13",
      "content": "<p>[quote=HueyKwik;145506]</p>\n\n<p>Can you explain more about your L2/L3 ensemble?</p>\n\n<p>[/quote]</p>\n\n<p>As beluga already mentioned, we ran a bagged, 4-fold cv, on a shared leak-stratified split. </p>\n\n<p>All the L1 models were trained multiple times per fold, each with a different seed. Each of these runs predicted on both the hold-out set, as well as the test set. The averaged hold-out and test predictions for each fold were stored for use in L2.</p>\n\n<p>In L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. Again the same 4-fold splits were used, predictions made and bagged on both hold-out and test set, and saved. These predictions were also rank normalized. </p>\n\n<p>To generate a submission based on an L2 model, the 4 test set predictions were averaged together, while the 4 holdout set predictions were combined into one set, and used to determine the thresholding percentage of 1s to use. That same percentage of 1s was then used to find the corresponding threshold level value to use in the test set. </p>\n\n<p>L3 was similar - it built on top of the saved L1 predictions and some of the saved L2 predictions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145840,
      "author_name": "zonemercy",
      "author_url": "",
      "post_date": "11/21/2016 11:26:40",
      "content": "<p>[quote=ash hafez;145530]</p>\n\n<p>[quote=HueyKwik;145506]</p>\n\n<p>[/quote]</p>\n\n<p>In L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. </p>\n\n<p>[/quote]</p>\n\n<p>Thanks for the solution share,  another question about the max rank normalized processing.\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145954,
      "author_name": "ashhafez",
      "author_url": "",
      "post_date": "11/21/2016 23:54:13",
      "content": "<p>[quote=Zongyi Liu;145840]</p>\n\n<p>[quote=ash hafez;145530]</p>\n\n<p>[quote=HueyKwik;145506]</p>\n\n<p>[/quote]</p>\n\n<p>In L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. </p>\n\n<p>[/quote]</p>\n\n<p>Thanks for the solution share,  another question about the max rank normalized processing.\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?</p>\n\n<p>[/quote]</p>\n\n<p>When you rank the fold predictions, you can divide by the maximum rank to turn it into a range between 0 and 1. Do the same for the test predictions as well. Then assuming you are stratified in your folds, and the target response rate in test matches train in general, you can translate the top x% of your ranked normalized predictions from train to an equivalent percentage in the test set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 173499,
      "author_name": "bmvsprasad",
      "author_url": "",
      "post_date": "04/07/2017 06:38:21",
      "content": "<p>Hi Beluga\n  Belated  Congrats .  I know its almost 5 months back. But i am new to kaggle  by 2 weeks.\nI am new to data science as well. </p>\n\n<p>I am working on this as POC in our company.</p>\n\n<p>Can you share what and all steps you did from the beginning to the end.\nI know i am bit <em>ambitious</em> here asking you to provide each piece of details.</p>\n\n<p>But if possible what steps you have taken can you share , like why certain features were selected, some discared. </p>\n\n<p>Thanks </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 837071,
      "author_name": "fengyuan1993",
      "author_url": "",
      "post_date": "05/07/2020 13:54:12",
      "content": "<p>Could you please share your code, thanks:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 849560,
      "author_name": "abouhassanmd",
      "author_url": "",
      "post_date": "05/15/2020 22:32:28",
      "content": "<p>Any chance of sharing your code? It would be highly appreciated, thank you!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "144544": "Sorry for the delay. It was good to have a kaggle free weekend after two months. (Ok not completely kaggle free but at least in read only mode:).\r\n\r\nAs we had 6-7 hours local time difference (USA, EU) we tried to split the work. \r\nI was mostly doing feature engineering and tried to improve our best single xgb score. Ash tried all kinds of different modeling techniques and built the L2/L3 ensemble.\r\n\r\n## Data Exploration\r\nI spent the first two weeks with data discovery. Numeric feature plots, statistics, station to station transition probabilities, correlation matrices for each stations was necessary to have ideas how to handle the thousands of anonymous features.\r\n\r\nI start each competition with desktop machine (16GB RAM). It takes some extra effort in the beginning to deal with lower level data manipulations due to the memory constraints. Getting closer to the raw data helps sometimes later. \r\n\r\nAs the numeric/date features had .3f/.2f precision we could rescale (*=1000, *=100) everything to integers. We kept only the min time for each station.\r\nSparse matrices came handy as we had a lot of missing values. \r\n\r\n\r\n## Leak/Magic/Data Property Features\r\nAsh found quite early that consecutive rows had feature duplication and correlated Response. We called consecutive Ids ordered by StartStation, StartTime *chunk*s.\r\n\r\nFor each chunk we used features\r\n\r\n* number of records\r\n* rank ascending\r\n* rank descending\r\n\r\nWe used the original order to add features based on previous and next records.\r\n\r\n* Response\r\n* feature hash equality for each station\r\n* StartTime, EndTime\r\n* Numeric raw features\r\n\r\n## Time Features\r\nI figured out in the beginning that [0.01 time granularity means probably 6 mins][1].\r\n\r\nSo we had to deal with 2 years of manufacturing data. This observation did not gave direct performance boost but gave us enough intuition to construct a lot of time based features.\r\n\r\n* StartStationTimes\r\n* StartTime, EndTime, Duration\r\n* StationTimeDiff\r\n* Start/End part of week (mod 1680)\r\n* Number of records in next/last 2.5h, 24h, 168h for each station\r\n* Number of records in the same time (6 mins)\r\n* MeanTimeDiff since last 1/5/10 failure(s)\r\n* MeanTimeDiff till next 1/5/10 failure(s)\r\n\r\n\r\n## Numeric Features\r\n* Raw numeric features (most of the time we used the raw numeric features or simple subsets based on xgb feature importance)\r\n* Z-scaled features for each week\r\n* Count encoding for each value \r\n* Feature combinations (f1 + -  * f2)\r\n\r\nWe saw interesting similarity in the station transition probability matrix between station S0-S11 and S12 - S23. You have probably seen the same in the Shopfloor visualizations. We also noticed that the number of features are the same across these station groups.\r\n\r\n![enter image description here][2]\r\n\r\nThe correlation matrices were similar for these stations so we tried combining the numeric features for the same stages. (e.g. L0_S0_F0 + L0_S12_F330 etc.)\r\n\r\n\r\n\r\n## Categorical Features\r\nWe could not squeeze much out of the categorical features. Most of the time we just dropped them. We had a few models with aggregated categorical features for each station in [24, 25, 26, 27, 28, 29, 32, 44, 47]. 'T1' was replaced by 1 all the other values were replaced 100.\r\n\r\n## Validation\r\nGiven the imbalanced labels and binary evaluation criteria we noticed quite high 5-FOLD CV stds (~0.01) in the beginning. After a few experiments we decided to use 4 FOLD \"leak-stratified\" CV (forcing to have the same number of duplication for each fold). \r\nEach \"single\" L1 model was trained with 3 different seeds on the same folds. Our latest ensemble scores were quite stable (std < 0.004) and very often our CV/LB score improved hand in hand.\r\n\r\n## Rebalancing\r\nWe kept every failure and records with duplication. For the remaining 0s 50-90% down sampling was used. It made the training a bit faster and worked as bagging for the model averaging part helping the later ensemble stages.\r\n\r\n## Results\r\nIn the last month of the competition we continuously improved and tried to keep us in the top 3. Fortunately we could improve significantly both CV and LB  with our last submissions on Friday. This made the submission selection easier.\r\n![enter image description here][3]\r\n\r\n\r\n  [1]: https://www.kaggle.com/gaborfodor/bosch-production-line-performance/notebookd19d11e4f2\r\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/144544/5351/StationFeaturSimilarity.png?sv=2012-02-12&se=2016-11-17T16%3A25%3A47Z&sr=b&sp=r&sig=pEgr%2B4FRBrS2Owl2ctg5sOEMBzMbWQoL78R%2FEed8h%2BA%3D\r\n  [3]: https://kaggle2.blob.core.windows.net/forum-message-attachments/144544/5348/results.PNG?sv=2012-02-12&se=2016-11-17T16%3A01%3A24Z&sr=b&sp=r&sig=%2FBvQoJg5kEuzvBWFBoHFs7GU%2Fgr7AotglS54yYgZk3M%3D",
    "144552": "Thanks a lot for sharing (plus the kernels you shared too).\r\n\r\nCan you explain a bit more this:\r\n\r\n- MeanTimeDiff since last 1/5/10 failure(s)\r\n- MeanTimeDiff till next 1/5/10 failure(s)\r\n\r\nMean over what?",
    "144553": "Impressive !",
    "144558": "[quote=CPMP;144552]\r\nCan you explain a bit more this:\r\n\r\n- MeanTimeDiff since last 1/5/10 failure(s)\r\n- MeanTimeDiff till next 1/5/10 failure(s)\r\n\r\nMean over what?  \r\n[/quote]\r\n\r\nMean(EndTime for latest K failures where EndTime < CurrentEndTime) - CurrentEndTime\r\n\r\nMean(EndTime for the first K failures where EndTime > CurrentEndTime) - CurrentEndTime",
    "144568": "Thanks.",
    "144584": "Awesome job Beluga and Ash; you crushed it!  A few follow up questions:\r\n\r\n1. For \"leak stratified\", does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?\r\n\r\n2. What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?",
    "144628": "[quote=Russ W;144584]\r\n\r\n1. For \"leak stratified\", does that mean that you kept chunks together in separate folds, or you split up the chunks equally across folds?\r\n\r\n2. What xgb parms did you typically use for L1, L2, and L3?   Did you use any other modeling methods to ensemble?\r\n\r\n[/quote]\r\n\r\n1. We allowed the same chunk to appear in multiple folds. We assumed that Train/Test and Public/Private were simple random splits.\r\n\r\n2. I tried ExtraTreesRegressor a few times. Deeper trees worked better for me but we had all sorts of different xgbs in the ensemble.\r\nOne of the best xgb parameters:\r\n * early stopping **auc**\r\n * alpha **0**\r\n * booster **gbtree**\r\n * colsample_bytree\t**0.6**\r\n * min_child_weight\t**5**\r\n * subsample\t**0.9**\r\n * eta\t**0.03**\r\n * objective\t**binary:logistic**\r\n * max_depth\t**14**\r\n* lambda\t**4**\r\n \r\nI found it strange that decreasing eta did not help to get better models.",
    "144788": "very impressive overall work , congratulations !",
    "144798": "I also found that small eta did not really help, hence I stick with a quite large value for it as it yield way faster training times.",
    "145000": "great work! congratulations!",
    "145046": "Congratulations!\r\nThere is a question about how to 'count'...... Which way is proper?\r\n1) Combine traindata and testdata to groupby().count(), and split them, or \r\n2) traindata.groupby().count() and testdata.groupby().count(), or\r\n3) traindata.groupby().count() and testdata.merge(, on='').\r\nThanks ;)",
    "145050": "Congratulations guys. Just to get a feel for, how many man hours in total do you reckon you have put into this competition?",
    "145052": "Thank you for the solution! \r\nYour effort is really appreciate, especially that two weeks for data discovery.",
    "145156": "Z-scaled features for each week\r\n\r\nWhat is Z-Scaled ?",
    "145210": "I guess it is standardization (substract mean then divide by variance), see for instance \r\n\r\nhttp://sebastianraschka.com/Articles/2014_about_feature_scaling.html\r\n\r\nor\r\n\r\nhttps://en.wikipedia.org/wiki/Standard_score\r\n\r\nGoal is to have a mean of 0 and variance of 1.\r\n\r\nNote that original numeric values were already having a mean of 0.  Variance was not one, but we have `max - min   <= 1`, which is another way to standardize.",
    "145373": "Wow, this is awesome! :)",
    "145506": "Can you explain more about your L2/L3 ensemble?",
    "145530": "[quote=HueyKwik;145506]\r\n\r\nCan you explain more about your L2/L3 ensemble?\r\n\r\n[/quote]\r\n\r\nAs beluga already mentioned, we ran a bagged, 4-fold cv, on a shared leak-stratified split. \r\n\r\nAll the L1 models were trained multiple times per fold, each with a different seed. Each of these runs predicted on both the hold-out set, as well as the test set. The averaged hold-out and test predictions for each fold were stored for use in L2.\r\n\r\nIn L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. Again the same 4-fold splits were used, predictions made and bagged on both hold-out and test set, and saved. These predictions were also rank normalized. \r\n\r\nTo generate a submission based on an L2 model, the 4 test set predictions were averaged together, while the 4 holdout set predictions were combined into one set, and used to determine the thresholding percentage of 1s to use. That same percentage of 1s was then used to find the corresponding threshold level value to use in the test set. \r\n\r\nL3 was similar - it built on top of the saved L1 predictions and some of the saved L2 predictions.",
    "145840": "[quote=ash hafez;145530]\r\n\r\n[quote=HueyKwik;145506]\r\n\r\n[/quote]\r\n\r\nIn L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. \r\n\r\n[/quote]\r\n\r\nThanks for the solution share,  another question about the max rank normalized processing.\r\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?",
    "145954": "[quote=Zongyi Liu;145840]\r\n\r\n[quote=ash hafez;145530]\r\n\r\n[quote=HueyKwik;145506]\r\n\r\n[/quote]\r\n\r\nIn L2, the predictions were (max) rank normalized (scipy.stats.rankdata) per fold before being used in the L2 models. \r\n\r\n[/quote]\r\n\r\nThanks for the solution share,  another question about the max rank normalized processing.\r\nIf I understand correctly, max rank normalized aims to find the best N predictions per fold. But the rank lists for each fold are different, how to deal with the test set predictions for L2 model ?\r\n\r\n[/quote]\r\n\r\nWhen you rank the fold predictions, you can divide by the maximum rank to turn it into a range between 0 and 1. Do the same for the test predictions as well. Then assuming you are stratified in your folds, and the target response rate in test matches train in general, you can translate the top x% of your ranked normalized predictions from train to an equivalent percentage in the test set.",
    "173499": "Hi Beluga\n  Belated  Congrats .  I know its almost 5 months back. But i am new to kaggle  by 2 weeks.\nI am new to data science as well. \n\n I am working on this as POC in our company.\n\nCan you share what and all steps you did from the beginning to the end.\nI know i am bit *ambitious* here asking you to provide each piece of details.\n\n But if possible what steps you have taken can you share , like why certain features were selected, some discared. \n\nThanks",
    "837071": "Could you please share your code, thanks:)",
    "849560": "Any chance of sharing your code? It would be highly appreciated, thank you!",
    "1673300": "i am using this data set for benchmarking but clearly there is the test data's labels that are missing , could you please share the labels that were predicted for the test data if its possible?",
    "3274668": "thanks for the details"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 5357,
  "competition_slug": "bosch-production-line-performance",
  "team_id": 375915,
  "solution_url": "/competitions/bosch-production-line-performance/writeups/ash-beluga-1st-place-solution"
}