{ "id": 25359, "title": "#3 place solution", "url": "/competitions/bosch-production-line-performance/writeups/data-property-avengers-3-place-solution", "author_name": "", "post_date": "2016-11-13T10:40:24.903Z", "votes": 37, "comment_count": 20, "views": 3233, "content": "

Before starting... a big thanks to Bosch team and all of the folks who contributed a lot to the community's learning in this competition – among others, JohnM's (superb!) and gingermans' flowpath visualisations; CPMP and Daniel FG's super fast MCC measurement functions for R and python; Mathias (Faron) and Marios for releasing the leak early and making it a competition; all the wrapper's for lightgbm (I used Raddar's R functions, not to take away from great ones by Laurae(great top10 finish!), Aradalan and others – and Microsoft team for releasing it).
\nCongrats to Gabor, Ash, Daniel FG, scnndl (looking forward to all your solutions!) and all other players.... and of course my team mates Marios, Mathias and Stas! I told Marios at the start I wanted to get GM status from this competition, and to the end he was determined to keep to that :)

\n\n

For feature engineering, here is what worked roughly in order of effectiveness,

\n\n\n\n

We had about 50 models in a stack, made up at second level of xgb, neural net.. and I think linear models. First level models were xgb, lightgbm, extratrees, rf, glmnet, neural net and others. Best single model was lightgbm with LB 0.502. As I guess everyone experienced, the stack was very instable, which lead to a lot of frustration and long periods of stagnation.

\n\n

We had an unplanned boost in the last hour of the competition :) With the instability, as BreakfastPriate pointed out, there seemed to be a few 100 rows being the main drivers for the last bit of score. We downloaded around 25 subs (containing the 1's and 0's) from LB which scored over 0.51, then column binded these subs and took the median value, row wise, of the 25 Response values. This gave a nice boost - around 0.003 on public and private – moved us up to fourth place on public LB .... with 20 mins to go. Had 2 more subs, and tried different thresholds, as opposed to median, but this didn't improve the straight median of all the 25 best subs.
\nA histogram (cut off on the y-axis) of the row wise sum of the 25 subs can be seen below...

\n\n

\"enter

", "messages": [ { "id": "144106", "postDate": "11/12/2016 11:13:27", "content": "

Before starting... a big thanks to Bosch team and all of the folks who contributed a lot to the community's learning in this competition – among others, JohnM's (superb!) and gingermans' flowpath visualisations; CPMP and Daniel FG's super fast MCC measurement functions for R and python; Mathias (Faron) and Marios for releasing the leak early and making it a competition; all the wrapper's for lightgbm (I used Raddar's R functions, not to take away from great ones by Laurae(great top10 finish!), Aradalan and others – and Microsoft team for releasing it).
\nCongrats to Gabor, Ash, Daniel FG, scnndl (looking forward to all your solutions!) and all other players.... and of course my team mates Marios, Mathias and Stas! I told Marios at the start I wanted to get GM status from this competition, and to the end he was determined to keep to that :)

\n\n

For feature engineering, here is what worked roughly in order of effectiveness,

\n\n\n\n

We had about 50 models in a stack, made up at second level of xgb, neural net.. and I think linear models. First level models were xgb, lightgbm, extratrees, rf, glmnet, neural net and others. Best single model was lightgbm with LB 0.502. As I guess everyone experienced, the stack was very instable, which lead to a lot of frustration and long periods of stagnation.

\n\n

We had an unplanned boost in the last hour of the competition :) With the instability, as BreakfastPriate pointed out, there seemed to be a few 100 rows being the main drivers for the last bit of score. We downloaded around 25 subs (containing the 1's and 0's) from LB which scored over 0.51, then column binded these subs and took the median value, row wise, of the 25 Response values. This gave a nice boost - around 0.003 on public and private – moved us up to fourth place on public LB .... with 20 mins to go. Had 2 more subs, and tried different thresholds, as opposed to median, but this didn't improve the straight median of all the 25 best subs.
\nA histogram (cut off on the y-axis) of the row wise sum of the 25 subs can be seen below...

\n\n

\"enter

", "rawMarkdown": "Before starting... a big thanks to Bosch team and all of the folks who contributed a lot to the community's learning in this competition – among others, JohnM's (superb!) and gingermans' flowpath visualisations; CPMP and Daniel FG's super fast MCC measurement functions for R and python; Mathias (Faron) and Marios for releasing the leak early and making it a competition; all the wrapper's for lightgbm (I used Raddar's R functions, not to take away from great ones by Laurae(great top10 finish!), Aradalan and others – and Microsoft team for releasing it). \r\nCongrats to Gabor, Ash, Daniel FG, scnndl (looking forward to all your solutions!) and all other players.... and of course my team mates Marios, Mathias and Stas! I told Marios at the start I wanted to get GM status from this competition, and to the end he was determined to keep to that :)\r\n\r\nFor feature engineering, here is what worked roughly in order of effectiveness,\r\n\r\n- Mathias's leak shown in the [public script][1] \r\n- Second leak, used per station, which worked especially well on L3_S29 and L3_S30. If you check JohnM's [viz][2], you can see these lines have very few NA's so the order by date can be captured very well. On other lines, with more NA's, the date order is more broken. We basically sorted on the dates in each station, and put an indicator for sequential duplicate numeric rows within the station. Created a leak feature like this for each of the stations, but L3_S29 and L3_S30 worked particularly well. \r\n- Related to second leak above, within each station, column bind the numeric values to a string, and get the count of the unique strings. \r\n- Row-wise summed the count and the leak (found in the last two bullet points), over all the stations. \r\n- Out of fold rolling mean of target (using 2 fold to replicate train/test split); sorted based on start and end dates of all stations. Got different rolling mean windows – 5, 10, 20, 100, 1000, 5000 – interestingly it was the really big windows which helped most; probably because it captured periods with low failure rate. \r\n- Lag and lead of target out of fold. \r\n- Encoded a few categorical columns with out of fold Bayesian Mean -there was one cat var with a lot of levels which captured a lot of 1's particularly well. This encoding was limited to the non duplicated categoricals which had over circa 7 unique values. \r\n- Counts of non-duplicated categoricals columns, counts of non-duplicated date and numeric columns. \r\n- Encode the paths of whether each row passed through stations. \r\n- Row wise NA counts for numeric, as well as max/min per stations. \r\n\r\nWe had about 50 models in a stack, made up at second level of xgb, neural net.. and I think linear models. First level models were xgb, lightgbm, extratrees, rf, glmnet, neural net and others. Best single model was lightgbm with LB 0.502. As I guess everyone experienced, the stack was very instable, which lead to a lot of frustration and long periods of stagnation. \r\n\r\nWe had an unplanned boost in the last hour of the competition :) With the instability, as BreakfastPriate pointed out, there seemed to be a few 100 rows being the main drivers for the last bit of score. We downloaded around 25 subs (containing the 1's and 0's) from LB which scored over 0.51, then column binded these subs and took the median value, row wise, of the 25 Response values. This gave a nice boost - around 0.003 on public and private – moved us up to fourth place on public LB .... with 20 mins to go. Had 2 more subs, and tried different thresholds, as opposed to median, but this didn't improve the straight median of all the 25 best subs. \r\nA histogram (cut off on the y-axis) of the row wise sum of the 25 subs can be seen below... \r\n\r\n![enter image description here][3]\r\n\r\n\r\n [1]: https://www.kaggle.com/mmueller/bosch-production-line-performance/road-2-0-4\r\n [2]: https://www.kaggle.com/jpmiller/bosch-production-line-performance/flowpath-viz/code\r\n [3]: https://www.dropbox.com/s/z9b91qrh2cg0mhy/25subs.png?dl=1", "votes": null }, { "id": "144118", "postDate": "11/12/2016 12:30:05", "content": "

thank you for share,can you share your code.learning from your code should be very nice

", "rawMarkdown": "thank you for share,can you share your code.learning from your code should be very nice", "votes": null }, { "id": "144120", "postDate": "11/12/2016 12:39:44", "content": "

Sure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though.

", "rawMarkdown": "Sure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though.", "votes": null }, { "id": "144138", "postDate": "11/12/2016 14:37:28", "content": "

How many folds did you use for stacking....

", "rawMarkdown": "How many folds did you use for stacking....", "votes": null }, { "id": "144139", "postDate": "11/12/2016 14:38:48", "content": "

Five

", "rawMarkdown": "Five", "votes": null }, { "id": "144148", "postDate": "11/12/2016 16:21:42", "content": "

[quote=Darragh;144120]

\n\n

Sure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though.

\n\n

[/quote]

\n\n

Thanks a lot for sharing the solution. Looking forward to learn more from your code.

", "rawMarkdown": "[quote=Darragh;144120]\r\n\r\nSure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though. \r\n\r\n[/quote]\r\n\r\nThanks a lot for sharing the solution. Looking forward to learn more from your code.", "votes": null }, { "id": "144149", "postDate": "11/12/2016 16:28:27", "content": "

Congratulations on becoming GM, well deserved! And thanks for sharing all this info.

", "rawMarkdown": "Congratulations on becoming GM, well deserved! And thanks for sharing all this info.", "votes": null }, { "id": "144470", "postDate": "11/14/2016 09:58:29", "content": "

Out of fold rolling mean of target (using 2 fold to replicate train/test split); sorted based on start and end dates of all stations. Got different rolling mean windows – 5, 10, 20, 100, 1000, 5000 – interestingly it was the really big windows which helped most; probably because it captured periods with low failure rate

\n\n

hi would you please explain this paragraph more?\n thank you

", "rawMarkdown": "**Out of fold rolling mean of target (using 2 fold to replicate train/test split); sorted based on start and end dates of all stations. Got different rolling mean windows – 5, 10, 20, 100, 1000, 5000 – interestingly it was the really big windows which helped most; probably because it captured periods with low failure rate**\r\n\r\n\r\n\r\nhi would you please explain this paragraph more?\r\n thank you", "votes": null }, { "id": "144485", "postDate": "11/14/2016 11:15:59", "content": "

Hope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also.

\n\n
> library(zoo);library(caret);set.seed(2)\n> vec = rep(0, 40)\n> vec[sample(1:40, 10)] = 1\n> # Create a vector will use as our target\n> vec\n [1] 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0\n> # The next line is the infold rollmean, with window of 5. this will overfit on your training data\n> rollmean(vec, 5, fill=NA)\n [1]  NA  NA 0.2 0.2 0.4 0.6 0.6 0.4 0.4 0.2 0.0 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.4\n[35] 0.4 0.4 0.4 0.4  NA  NA\n> # Now, lets get the out-of-fold rollmean with window of 5\n> # We need an index based on the vector with two folds\n> idx = createFolds(vec, 2, list=F)\n> idx\n [1] 1 1 2 1 2 1 1 1 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 1 2 1 1 1 2 1 1 2 2 1 1 1 1 1 2 2\n> # Create a vector of results for each fold.\n> fold1 = fold2 = vec\n> # get OOF rollmean of fold1\n> fold1[idx==2] = 0\n> fold1 = rollmean(fold1, 5, fill=NA)\n> # get OOF rollmean of fold2\n> fold2[idx==1] = 0\n> fold2 = rollmean(fold2, 5, fill=NA)\n> # Get the result using only OOF elements\n> result = fold1\n> result[idx==1] = fold2[idx==1]\n> result\n [1]  NA  NA 0.0 0.2 0.2 0.2 0.2 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.2 0.0\n[35] 0.0 0.0 0.2 0.2  NA  NA\n> plot(vec, xlab = \"Sequence of Components\", ylab = \"Passed-0    Failure-1\", type=\"l\", col = \"blue\", ylim = c(0,1), lwd=1)\n> lines(rollmean(vec, 5, fill=NA), type=\"l\", col=\"red\", lwd=3, lty=2)\n> lines(result, type=\"l\", col=\"green\", lwd=3, lty=2)\n> legend(\"topright\", c(\"Actual failures\", \"Rolling mean of failures\", \"Out of fold rolling mean\"), \n+        col=c(\"blue\", \"red\", \"green\"), lty=c(1,3,3), lwd=c(1,2,2))\n
\n\n

\"enter

", "rawMarkdown": "Hope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also. \r\n\r\n > library(zoo);library(caret);set.seed(2)\r\n > vec = rep(0, 40)\r\n > vec[sample(1:40, 10)] = 1\r\n > # Create a vector will use as our target\r\n > vec\r\n [1] 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0\r\n > # The next line is the infold rollmean, with window of 5. this will overfit on your training data\r\n > rollmean(vec, 5, fill=NA)\r\n [1] NA NA 0.2 0.2 0.4 0.6 0.6 0.4 0.4 0.2 0.0 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.4\r\n [35] 0.4 0.4 0.4 0.4 NA NA\r\n > # Now, lets get the out-of-fold rollmean with window of 5\r\n > # We need an index based on the vector with two folds\r\n > idx = createFolds(vec, 2, list=F)\r\n > idx\r\n [1] 1 1 2 1 2 1 1 1 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 1 2 1 1 1 2 1 1 2 2 1 1 1 1 1 2 2\r\n > # Create a vector of results for each fold.\r\n > fold1 = fold2 = vec\r\n > # get OOF rollmean of fold1\r\n > fold1[idx==2] = 0\r\n > fold1 = rollmean(fold1, 5, fill=NA)\r\n > # get OOF rollmean of fold2\r\n > fold2[idx==1] = 0\r\n > fold2 = rollmean(fold2, 5, fill=NA)\r\n > # Get the result using only OOF elements\r\n > result = fold1\r\n > result[idx==1] = fold2[idx==1]\r\n > result\r\n [1] NA NA 0.0 0.2 0.2 0.2 0.2 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.2 0.0\r\n [35] 0.0 0.0 0.2 0.2 NA NA\r\n > plot(vec, xlab = \"Sequence of Components\", ylab = \"Passed-0 Failure-1\", type=\"l\", col = \"blue\", ylim = c(0,1), lwd=1)\r\n > lines(rollmean(vec, 5, fill=NA), type=\"l\", col=\"red\", lwd=3, lty=2)\r\n > lines(result, type=\"l\", col=\"green\", lwd=3, lty=2)\r\n > legend(\"topright\", c(\"Actual failures\", \"Rolling mean of failures\", \"Out of fold rolling mean\"), \r\n + col=c(\"blue\", \"red\", \"green\"), lty=c(1,3,3), lwd=c(1,2,2))\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://www.dropbox.com/s/t75djdlf5l10r9x/OOFRollingMean.png?dl=1", "votes": null }, { "id": "144492", "postDate": "11/14/2016 11:56:26", "content": "

Hi \nwhat does "Lag and lead of target out of fold" mean?

", "rawMarkdown": "Hi \r\nwhat does \"**Lag and lead of target out of fold**\" mean?", "votes": null }, { "id": "144496", "postDate": "11/14/2016 12:13:11", "content": "

Sort of the same as the previous rollmean code; but instead of calulating the rollmean, put int fold1 and fold2 what was the n-1 value (lag), or the n+1 (lead) value for each element. Mathias implemented this so I may not have it exactly correct.

", "rawMarkdown": "Sort of the same as the previous rollmean code; but instead of calulating the rollmean, put int fold1 and fold2 what was the n-1 value (lag), or the n+1 (lead) value for each element. Mathias implemented this so I may not have it exactly correct.", "votes": null }, { "id": "144503", "postDate": "11/14/2016 12:36:16", "content": "

Hello\nDoes Row-wise mean Best row that extract from unique flowpath ?\nis the flow path created from L3_S29 and L3_S30?

", "rawMarkdown": "Hello\r\nDoes Row-wise mean Best row that extract from unique flowpath ?\r\nis the flow path created from L3_S29 and L3_S30?", "votes": null }, { "id": "144528", "postDate": "11/14/2016 14:27:16", "content": "

[quote=Darragh;144485]

\n\n

Hope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also.

\n\n
> library(zoo);library(caret);set.seed(1)\n> vec = rep(0, 40)\n> vec[sample(1:40, 10)] = 1\n> # Create a vector will use as our target\n> vec\n [1] 0 1 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0\n> # The next line is the infold rollmean, with window of 5. this will overfit on your training data\n> rollmean(vec, 5, fill=NA)\n [1]  NA  NA 0.2 0.2 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.4 0.4 0.4 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.6 0.6 0.6 0.4 0.4 0.2\n[38] 0.2  NA  NA\n> # Now, lets get the out-of-fold rollmean with window of 5\n> # We need an index based on the vector with two folds\n> idx = createFolds(vec, 2, list=F)\n> idx\n [1] 1 1 1 1 2 2 2 2 1 1 2 2 2 1 2 2 1 1 2 1 1 1 2 2 2 1 2 2 2 1 1 2 2 1 2 2 1 1 1 1\n> # Create a vector of results for each fold.\n> fold1 = fold2 = vec\n> # get OOF rollmean of fold1\n> fold1[idx==2] = 0\n> fold1 = rollmean(fold1, 5, fill=NA)\n> # get OOF rollmean of fold2\n> fold2[idx==1] = 0\n> fold2 = rollmean(fold2, 5, fill=NA)\n> # Get the result using only OOF elements\n> result = fold1\n> result[idx==1] = fold2[idx==1]\n> result\n [1]  NA  NA 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.4 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.0 0.2 0.0 0.0 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.2 0.4 0.2 0.4 0.0\n[38] 0.0  NA  NA\n> \n
\n\n

To visualise it : \n\"enter

\n\n

[/quote]

\n\n

thanks for your explanation :)

", "rawMarkdown": "[quote=Darragh;144485]\r\n\r\nHope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also. \r\n\r\n\r\n > library(zoo);library(caret);set.seed(1)\r\n > vec = rep(0, 40)\r\n > vec[sample(1:40, 10)] = 1\r\n > # Create a vector will use as our target\r\n > vec\r\n [1] 0 1 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0\r\n > # The next line is the infold rollmean, with window of 5. this will overfit on your training data\r\n > rollmean(vec, 5, fill=NA)\r\n [1] NA NA 0.2 0.2 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.4 0.4 0.4 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.6 0.6 0.6 0.4 0.4 0.2\r\n [38] 0.2 NA NA\r\n > # Now, lets get the out-of-fold rollmean with window of 5\r\n > # We need an index based on the vector with two folds\r\n > idx = createFolds(vec, 2, list=F)\r\n > idx\r\n [1] 1 1 1 1 2 2 2 2 1 1 2 2 2 1 2 2 1 1 2 1 1 1 2 2 2 1 2 2 2 1 1 2 2 1 2 2 1 1 1 1\r\n > # Create a vector of results for each fold.\r\n > fold1 = fold2 = vec\r\n > # get OOF rollmean of fold1\r\n > fold1[idx==2] = 0\r\n > fold1 = rollmean(fold1, 5, fill=NA)\r\n > # get OOF rollmean of fold2\r\n > fold2[idx==1] = 0\r\n > fold2 = rollmean(fold2, 5, fill=NA)\r\n > # Get the result using only OOF elements\r\n > result = fold1\r\n > result[idx==1] = fold2[idx==1]\r\n > result\r\n [1] NA NA 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.4 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.0 0.2 0.0 0.0 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.2 0.4 0.2 0.4 0.0\r\n [38] 0.0 NA NA\r\n > \r\n\r\nTo visualise it : \r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://www.dropbox.com/s/qa638tyl3k578re/oof.png?dl=1\r\n\r\n[/quote]\r\n\r\nthanks for your explanation :)", "votes": null }, { "id": "144585", "postDate": "11/14/2016 18:15:05", "content": "

@mr.z .. Flowpath is from all stations

", "rawMarkdown": "mr.z .. Flowpath is from all stations", "votes": null }, { "id": "144626", "postDate": "11/14/2016 21:50:02", "content": "

Just FYI, R's RcppRoll package does blazing fast rolling window operations. Quite a lot faster than zoo or TTR.

\n\n

And R's data.table package has a shift function, which helps in calculating lagged or leading values.

", "rawMarkdown": "Just FYI, R's RcppRoll package does blazing fast rolling window operations. Quite a lot faster than zoo or TTR.\r\n\r\nAnd R's data.table package has a shift function, which helps in calculating lagged or leading values.", "votes": null }, { "id": "144629", "postDate": "11/14/2016 21:55:30", "content": "

@willie, thanks! will try that

", "rawMarkdown": "willie, thanks! will try that", "votes": null }, { "id": "145896", "postDate": "11/21/2016 16:23:19", "content": "

Could author or anybody help to explain this bullet? Thanks.

\n\n", "rawMarkdown": "Could author or anybody help to explain this bullet? Thanks.\r\n\r\n \r\n\r\n - **Related to second leak above, within each station, column bind the numeric values to a string, and get the count of the unique strings**. (*Why did column bind of numeric values? In my understanding every string is unique in this way...*)", "votes": null }, { "id": "146012", "postDate": "11/22/2016 09:13:04", "content": "

It is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc.

", "rawMarkdown": "It is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc.", "votes": null }, { "id": "146075", "postDate": "11/22/2016 16:37:25", "content": "

[quote=Darragh;146012]

\n\n

It is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc.

\n\n

[/quote]\nSo after this processing you will have 53 numbers corresponding to all 53 stations?

", "rawMarkdown": "[quote=Darragh;146012]\r\n\r\nIt is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc. \r\n\r\n[/quote]\r\nSo after this processing you will have 53 numbers corresponding to all 53 stations?", "votes": null }, { "id": "154408", "postDate": "01/06/2017 03:12:16", "content": "

Where is ur code in github?

", "rawMarkdown": "Where is ur code in github?", "votes": null }, { "id": "837057", "postDate": "05/07/2020 13:38:17", "content": "

I would appreciate it if you can share you code. thanks :)

", "rawMarkdown": "I would appreciate it if you can share you code. thanks :)", "votes": null } ], "comments": [ { "id": 144118, "author_name": "jiandong", "author_url": "", "post_date": "11/12/2016 12:30:05", "content": "

thank you for share,can you share your code.learning from your code should be very nice

", "votes": null, "replies": [] }, { "id": 144120, "author_name": "darraghdog", "author_url": "", "post_date": "11/12/2016 12:39:44", "content": "

Sure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though.

", "votes": null, "replies": [] }, { "id": 144138, "author_name": "outliar", "author_url": "", "post_date": "11/12/2016 14:37:28", "content": "

How many folds did you use for stacking....

", "votes": null, "replies": [] }, { "id": 144139, "author_name": "darraghdog", "author_url": "", "post_date": "11/12/2016 14:38:48", "content": "

Five

", "votes": null, "replies": [] }, { "id": 144148, "author_name": "msaber", "author_url": "", "post_date": "11/12/2016 16:21:42", "content": "

[quote=Darragh;144120]

\n\n

Sure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though.

\n\n

[/quote]

\n\n

Thanks a lot for sharing the solution. Looking forward to learn more from your code.

", "votes": null, "replies": [] }, { "id": 144149, "author_name": "cpmpml", "author_url": "", "post_date": "11/12/2016 16:28:27", "content": "

Congratulations on becoming GM, well deserved! And thanks for sharing all this info.

", "votes": null, "replies": [] }, { "id": 144470, "author_name": "svasva", "author_url": "", "post_date": "11/14/2016 09:58:29", "content": "

Out of fold rolling mean of target (using 2 fold to replicate train/test split); sorted based on start and end dates of all stations. Got different rolling mean windows – 5, 10, 20, 100, 1000, 5000 – interestingly it was the really big windows which helped most; probably because it captured periods with low failure rate

\n\n

hi would you please explain this paragraph more?\n thank you

", "votes": null, "replies": [] }, { "id": 144485, "author_name": "darraghdog", "author_url": "", "post_date": "11/14/2016 11:15:59", "content": "

Hope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also.

\n\n
> library(zoo);library(caret);set.seed(2)\n> vec = rep(0, 40)\n> vec[sample(1:40, 10)] = 1\n> # Create a vector will use as our target\n> vec\n [1] 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0\n> # The next line is the infold rollmean, with window of 5. this will overfit on your training data\n> rollmean(vec, 5, fill=NA)\n [1]  NA  NA 0.2 0.2 0.4 0.6 0.6 0.4 0.4 0.2 0.0 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.4\n[35] 0.4 0.4 0.4 0.4  NA  NA\n> # Now, lets get the out-of-fold rollmean with window of 5\n> # We need an index based on the vector with two folds\n> idx = createFolds(vec, 2, list=F)\n> idx\n [1] 1 1 2 1 2 1 1 1 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 1 2 1 1 1 2 1 1 2 2 1 1 1 1 1 2 2\n> # Create a vector of results for each fold.\n> fold1 = fold2 = vec\n> # get OOF rollmean of fold1\n> fold1[idx==2] = 0\n> fold1 = rollmean(fold1, 5, fill=NA)\n> # get OOF rollmean of fold2\n> fold2[idx==1] = 0\n> fold2 = rollmean(fold2, 5, fill=NA)\n> # Get the result using only OOF elements\n> result = fold1\n> result[idx==1] = fold2[idx==1]\n> result\n [1]  NA  NA 0.0 0.2 0.2 0.2 0.2 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.2 0.0\n[35] 0.0 0.0 0.2 0.2  NA  NA\n> plot(vec, xlab = \"Sequence of Components\", ylab = \"Passed-0    Failure-1\", type=\"l\", col = \"blue\", ylim = c(0,1), lwd=1)\n> lines(rollmean(vec, 5, fill=NA), type=\"l\", col=\"red\", lwd=3, lty=2)\n> lines(result, type=\"l\", col=\"green\", lwd=3, lty=2)\n> legend(\"topright\", c(\"Actual failures\", \"Rolling mean of failures\", \"Out of fold rolling mean\"), \n+        col=c(\"blue\", \"red\", \"green\"), lty=c(1,3,3), lwd=c(1,2,2))\n
\n\n

\"enter

", "votes": null, "replies": [] }, { "id": 144492, "author_name": "zahraf", "author_url": "", "post_date": "11/14/2016 11:56:26", "content": "

Hi \nwhat does "Lag and lead of target out of fold" mean?

", "votes": null, "replies": [] }, { "id": 144496, "author_name": "darraghdog", "author_url": "", "post_date": "11/14/2016 12:13:11", "content": "

Sort of the same as the previous rollmean code; but instead of calulating the rollmean, put int fold1 and fold2 what was the n-1 value (lag), or the n+1 (lead) value for each element. Mathias implemented this so I may not have it exactly correct.

", "votes": null, "replies": [] }, { "id": 144503, "author_name": "mahsarz", "author_url": "", "post_date": "11/14/2016 12:36:16", "content": "

Hello\nDoes Row-wise mean Best row that extract from unique flowpath ?\nis the flow path created from L3_S29 and L3_S30?

", "votes": null, "replies": [] }, { "id": 144528, "author_name": "svasva", "author_url": "", "post_date": "11/14/2016 14:27:16", "content": "

[quote=Darragh;144485]

\n\n

Hope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also.

\n\n
> library(zoo);library(caret);set.seed(1)\n> vec = rep(0, 40)\n> vec[sample(1:40, 10)] = 1\n> # Create a vector will use as our target\n> vec\n [1] 0 1 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0\n> # The next line is the infold rollmean, with window of 5. this will overfit on your training data\n> rollmean(vec, 5, fill=NA)\n [1]  NA  NA 0.2 0.2 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.4 0.4 0.4 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.6 0.6 0.6 0.4 0.4 0.2\n[38] 0.2  NA  NA\n> # Now, lets get the out-of-fold rollmean with window of 5\n> # We need an index based on the vector with two folds\n> idx = createFolds(vec, 2, list=F)\n> idx\n [1] 1 1 1 1 2 2 2 2 1 1 2 2 2 1 2 2 1 1 2 1 1 1 2 2 2 1 2 2 2 1 1 2 2 1 2 2 1 1 1 1\n> # Create a vector of results for each fold.\n> fold1 = fold2 = vec\n> # get OOF rollmean of fold1\n> fold1[idx==2] = 0\n> fold1 = rollmean(fold1, 5, fill=NA)\n> # get OOF rollmean of fold2\n> fold2[idx==1] = 0\n> fold2 = rollmean(fold2, 5, fill=NA)\n> # Get the result using only OOF elements\n> result = fold1\n> result[idx==1] = fold2[idx==1]\n> result\n [1]  NA  NA 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.4 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.0 0.2 0.0 0.0 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.2 0.4 0.2 0.4 0.0\n[38] 0.0  NA  NA\n> \n
\n\n

To visualise it : \n\"enter

\n\n

[/quote]

\n\n

thanks for your explanation :)

", "votes": null, "replies": [] }, { "id": 144585, "author_name": "darraghdog", "author_url": "", "post_date": "11/14/2016 18:15:05", "content": "

@mr.z .. Flowpath is from all stations

", "votes": null, "replies": [] }, { "id": 144626, "author_name": "willieliao", "author_url": "", "post_date": "11/14/2016 21:50:02", "content": "

Just FYI, R's RcppRoll package does blazing fast rolling window operations. Quite a lot faster than zoo or TTR.

\n\n

And R's data.table package has a shift function, which helps in calculating lagged or leading values.

", "votes": null, "replies": [] }, { "id": 144629, "author_name": "darraghdog", "author_url": "", "post_date": "11/14/2016 21:55:30", "content": "

@willie, thanks! will try that

", "votes": null, "replies": [] }, { "id": 145896, "author_name": "jianshefeng", "author_url": "", "post_date": "11/21/2016 16:23:19", "content": "

Could author or anybody help to explain this bullet? Thanks.

\n\n", "votes": null, "replies": [] }, { "id": 146012, "author_name": "darraghdog", "author_url": "", "post_date": "11/22/2016 09:13:04", "content": "

It is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc.

", "votes": null, "replies": [] }, { "id": 146075, "author_name": "jianshefeng", "author_url": "", "post_date": "11/22/2016 16:37:25", "content": "

[quote=Darragh;146012]

\n\n

It is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc.

\n\n

[/quote]\nSo after this processing you will have 53 numbers corresponding to all 53 stations?

", "votes": null, "replies": [] }, { "id": 154408, "author_name": "karengreen", "author_url": "", "post_date": "01/06/2017 03:12:16", "content": "

Where is ur code in github?

", "votes": null, "replies": [] }, { "id": 837057, "author_name": "fengyuan1993", "author_url": "", "post_date": "05/07/2020 13:38:17", "content": "

I would appreciate it if you can share you code. thanks :)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "144106": "Before starting... a big thanks to Bosch team and all of the folks who contributed a lot to the community's learning in this competition – among others, JohnM's (superb!) and gingermans' flowpath visualisations; CPMP and Daniel FG's super fast MCC measurement functions for R and python; Mathias (Faron) and Marios for releasing the leak early and making it a competition; all the wrapper's for lightgbm (I used Raddar's R functions, not to take away from great ones by Laurae(great top10 finish!), Aradalan and others – and Microsoft team for releasing it). \r\nCongrats to Gabor, Ash, Daniel FG, scnndl (looking forward to all your solutions!) and all other players.... and of course my team mates Marios, Mathias and Stas! I told Marios at the start I wanted to get GM status from this competition, and to the end he was determined to keep to that :)\r\n\r\nFor feature engineering, here is what worked roughly in order of effectiveness,\r\n\r\n- Mathias's leak shown in the [public script][1] \r\n- Second leak, used per station, which worked especially well on L3_S29 and L3_S30. If you check JohnM's [viz][2], you can see these lines have very few NA's so the order by date can be captured very well. On other lines, with more NA's, the date order is more broken. We basically sorted on the dates in each station, and put an indicator for sequential duplicate numeric rows within the station. Created a leak feature like this for each of the stations, but L3_S29 and L3_S30 worked particularly well. \r\n- Related to second leak above, within each station, column bind the numeric values to a string, and get the count of the unique strings. \r\n- Row-wise summed the count and the leak (found in the last two bullet points), over all the stations. \r\n- Out of fold rolling mean of target (using 2 fold to replicate train/test split); sorted based on start and end dates of all stations. Got different rolling mean windows – 5, 10, 20, 100, 1000, 5000 – interestingly it was the really big windows which helped most; probably because it captured periods with low failure rate. \r\n- Lag and lead of target out of fold. \r\n- Encoded a few categorical columns with out of fold Bayesian Mean -there was one cat var with a lot of levels which captured a lot of 1's particularly well. This encoding was limited to the non duplicated categoricals which had over circa 7 unique values. \r\n- Counts of non-duplicated categoricals columns, counts of non-duplicated date and numeric columns. \r\n- Encode the paths of whether each row passed through stations. \r\n- Row wise NA counts for numeric, as well as max/min per stations. \r\n\r\nWe had about 50 models in a stack, made up at second level of xgb, neural net.. and I think linear models. First level models were xgb, lightgbm, extratrees, rf, glmnet, neural net and others. Best single model was lightgbm with LB 0.502. As I guess everyone experienced, the stack was very instable, which lead to a lot of frustration and long periods of stagnation. \r\n\r\nWe had an unplanned boost in the last hour of the competition :) With the instability, as BreakfastPriate pointed out, there seemed to be a few 100 rows being the main drivers for the last bit of score. We downloaded around 25 subs (containing the 1's and 0's) from LB which scored over 0.51, then column binded these subs and took the median value, row wise, of the 25 Response values. This gave a nice boost - around 0.003 on public and private – moved us up to fourth place on public LB .... with 20 mins to go. Had 2 more subs, and tried different thresholds, as opposed to median, but this didn't improve the straight median of all the 25 best subs. \r\nA histogram (cut off on the y-axis) of the row wise sum of the 25 subs can be seen below... \r\n\r\n![enter image description here][3]\r\n\r\n\r\n [1]: https://www.kaggle.com/mmueller/bosch-production-line-performance/road-2-0-4\r\n [2]: https://www.kaggle.com/jpmiller/bosch-production-line-performance/flowpath-viz/code\r\n [3]: https://www.dropbox.com/s/z9b91qrh2cg0mhy/25subs.png?dl=1", "144118": "thank you for share,can you share your code.learning from your code should be very nice", "144120": "Sure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though.", "144138": "How many folds did you use for stacking....", "144139": "Five", "144148": "[quote=Darragh;144120]\r\n\r\nSure jiandong, will put up a lot of the feature part on github anyway... may be a couple of weeks though. \r\n\r\n[/quote]\r\n\r\nThanks a lot for sharing the solution. Looking forward to learn more from your code.", "144149": "Congratulations on becoming GM, well deserved! And thanks for sharing all this info.", "144470": "**Out of fold rolling mean of target (using 2 fold to replicate train/test split); sorted based on start and end dates of all stations. Got different rolling mean windows – 5, 10, 20, 100, 1000, 5000 – interestingly it was the really big windows which helped most; probably because it captured periods with low failure rate**\r\n\r\n\r\n\r\nhi would you please explain this paragraph more?\r\n thank you", "144485": "Hope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also. \r\n\r\n > library(zoo);library(caret);set.seed(2)\r\n > vec = rep(0, 40)\r\n > vec[sample(1:40, 10)] = 1\r\n > # Create a vector will use as our target\r\n > vec\r\n [1] 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0\r\n > # The next line is the infold rollmean, with window of 5. this will overfit on your training data\r\n > rollmean(vec, 5, fill=NA)\r\n [1] NA NA 0.2 0.2 0.4 0.6 0.6 0.4 0.4 0.2 0.0 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.2 0.2 0.2 0.0 0.2 0.2 0.4\r\n [35] 0.4 0.4 0.4 0.4 NA NA\r\n > # Now, lets get the out-of-fold rollmean with window of 5\r\n > # We need an index based on the vector with two folds\r\n > idx = createFolds(vec, 2, list=F)\r\n > idx\r\n [1] 1 1 2 1 2 1 1 1 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 1 2 1 1 1 2 1 1 2 2 1 1 1 1 1 2 2\r\n > # Create a vector of results for each fold.\r\n > fold1 = fold2 = vec\r\n > # get OOF rollmean of fold1\r\n > fold1[idx==2] = 0\r\n > fold1 = rollmean(fold1, 5, fill=NA)\r\n > # get OOF rollmean of fold2\r\n > fold2[idx==1] = 0\r\n > fold2 = rollmean(fold2, 5, fill=NA)\r\n > # Get the result using only OOF elements\r\n > result = fold1\r\n > result[idx==1] = fold2[idx==1]\r\n > result\r\n [1] NA NA 0.0 0.2 0.2 0.2 0.2 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.2 0.0\r\n [35] 0.0 0.0 0.2 0.2 NA NA\r\n > plot(vec, xlab = \"Sequence of Components\", ylab = \"Passed-0 Failure-1\", type=\"l\", col = \"blue\", ylim = c(0,1), lwd=1)\r\n > lines(rollmean(vec, 5, fill=NA), type=\"l\", col=\"red\", lwd=3, lty=2)\r\n > lines(result, type=\"l\", col=\"green\", lwd=3, lty=2)\r\n > legend(\"topright\", c(\"Actual failures\", \"Rolling mean of failures\", \"Out of fold rolling mean\"), \r\n + col=c(\"blue\", \"red\", \"green\"), lty=c(1,3,3), lwd=c(1,2,2))\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://www.dropbox.com/s/t75djdlf5l10r9x/OOFRollingMean.png?dl=1", "144492": "Hi \r\nwhat does \"**Lag and lead of target out of fold**\" mean?", "144496": "Sort of the same as the previous rollmean code; but instead of calulating the rollmean, put int fold1 and fold2 what was the n-1 value (lag), or the n+1 (lead) value for each element. Mathias implemented this so I may not have it exactly correct.", "144503": "Hello\r\nDoes Row-wise mean Best row that extract from unique flowpath ?\r\nis the flow path created from L3_S29 and L3_S30?", "144528": "[quote=Darragh;144485]\r\n\r\nHope this explains the oof rollmean, bit easier to code than explain. If R is not to your liking, pandas has a rolling_mean function also. \r\n\r\n\r\n > library(zoo);library(caret);set.seed(1)\r\n > vec = rep(0, 40)\r\n > vec[sample(1:40, 10)] = 1\r\n > # Create a vector will use as our target\r\n > vec\r\n [1] 0 1 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0\r\n > # The next line is the infold rollmean, with window of 5. this will overfit on your training data\r\n > rollmean(vec, 5, fill=NA)\r\n [1] NA NA 0.2 0.2 0.0 0.2 0.2 0.2 0.4 0.4 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.0 0.2 0.4 0.4 0.4 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.6 0.6 0.6 0.4 0.4 0.2\r\n [38] 0.2 NA NA\r\n > # Now, lets get the out-of-fold rollmean with window of 5\r\n > # We need an index based on the vector with two folds\r\n > idx = createFolds(vec, 2, list=F)\r\n > idx\r\n [1] 1 1 1 1 2 2 2 2 1 1 2 2 2 1 2 2 1 1 2 1 1 1 2 2 2 1 2 2 2 1 1 2 2 1 2 2 1 1 1 1\r\n > # Create a vector of results for each fold.\r\n > fold1 = fold2 = vec\r\n > # get OOF rollmean of fold1\r\n > fold1[idx==2] = 0\r\n > fold1 = rollmean(fold1, 5, fill=NA)\r\n > # get OOF rollmean of fold2\r\n > fold2[idx==1] = 0\r\n > fold2 = rollmean(fold2, 5, fill=NA)\r\n > # Get the result using only OOF elements\r\n > result = fold1\r\n > result[idx==1] = fold2[idx==1]\r\n > result\r\n [1] NA NA 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.4 0.0 0.0 0.0 0.2 0.0 0.0 0.2 0.0 0.2 0.0 0.0 0.0 0.4 0.2 0.0 0.0 0.0 0.0 0.0 0.2 0.4 0.2 0.2 0.4 0.2 0.4 0.0\r\n [38] 0.0 NA NA\r\n > \r\n\r\nTo visualise it : \r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://www.dropbox.com/s/qa638tyl3k578re/oof.png?dl=1\r\n\r\n[/quote]\r\n\r\nthanks for your explanation :)", "144585": "mr.z .. Flowpath is from all stations", "144626": "Just FYI, R's RcppRoll package does blazing fast rolling window operations. Quite a lot faster than zoo or TTR.\r\n\r\nAnd R's data.table package has a shift function, which helps in calculating lagged or leading values.", "144629": "willie, thanks! will try that", "145896": "Could author or anybody help to explain this bullet? Thanks.\r\n\r\n \r\n\r\n - **Related to second leak above, within each station, column bind the numeric values to a string, and get the count of the unique strings**. (*Why did column bind of numeric values? In my understanding every string is unique in this way...*)", "146012": "It is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc.", "146075": "[quote=Darragh;146012]\r\n\r\nIt is another way to say get count of leak values within a station. Two components may have the exact same numeric values in a station - if no other components have those exact values, those two components get a count of two. Likewise for three components which have the same numeric values, get a count of 3 assigned etc. \r\n\r\n[/quote]\r\nSo after this processing you will have 53 numbers corresponding to all 53 stations?", "154408": "Where is ur code in github?", "837057": "I would appreciate it if you can share you code. thanks :)" }, "source": "meta" }