![\"enter](\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/5337/bosch_sac_mn.JPG\")
The competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:
\n\n• We only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.
\n\n• We also only predicted for rows where N was very similar to N+1. We set all others to zero.
\n\n• Some features were:
\n\n• How many rows were in a particular time-group. If there were many rows, there tended to be higher failure, and if there were few rows, there tended to be higher failure. Lowest failure was in the middle ranges.
\n\n• Some time groups simply had higher failure rates than others. So proximity by time was an important feature.
\n\n• Total duration of a row from first timestamp to last timestamp.
\n\n• Some values of L3_S32_F3854 were particularly valuable so we one-hot-encoded that.
\n\n• The rows immediately before and after row N were important. Specifically the Response on those rows; were they exact duplicates of row N or simply partial duplicates; the values of L3_S38_D3957 and L1_S24_F1844 on row N+1 were helpful.
\n\n• Not all of the duplicates came in pairs. Some were in triples, quadruples, or more. So we measured how many rows were in the duplicate “set” and what position this row was (e.g. 3rd of 4).
\n\n• After feature creation, we used XGBoost and Random Forests and created a weighted average of their predictions.
\n\n• I usually prefer simpler solutions to more complex ones, but for this competition, we found that to squeeze the last 0.01 out, we had to create some complex ensembles of runs that had different subsets of the features, different algorithms, etc. In fact, one of our selected submissions simply took all of our submissions from the last month and predicted 1 if that row was 1 in 11 or more of the submissions.
\n\nWe will see how it goes.
", "messages": [ { "id": "144019", "postDate": "11/11/2016 23:57:52", "content": "The competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:
\n\n• We only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.
\n\n• We also only predicted for rows where N was very similar to N+1. We set all others to zero.
\n\n• Some features were:
\n\n• How many rows were in a particular time-group. If there were many rows, there tended to be higher failure, and if there were few rows, there tended to be higher failure. Lowest failure was in the middle ranges.
\n\n• Some time groups simply had higher failure rates than others. So proximity by time was an important feature.
\n\n• Total duration of a row from first timestamp to last timestamp.
\n\n• Some values of L3_S32_F3854 were particularly valuable so we one-hot-encoded that.
\n\n• The rows immediately before and after row N were important. Specifically the Response on those rows; were they exact duplicates of row N or simply partial duplicates; the values of L3_S38_D3957 and L1_S24_F1844 on row N+1 were helpful.
\n\n• Not all of the duplicates came in pairs. Some were in triples, quadruples, or more. So we measured how many rows were in the duplicate “set” and what position this row was (e.g. 3rd of 4).
\n\n• After feature creation, we used XGBoost and Random Forests and created a weighted average of their predictions.
\n\n• I usually prefer simpler solutions to more complex ones, but for this competition, we found that to squeeze the last 0.01 out, we had to create some complex ensembles of runs that had different subsets of the features, different algorithms, etc. In fact, one of our selected submissions simply took all of our submissions from the last month and predicted 1 if that row was 1 in 11 or more of the submissions.
\n\nWe will see how it goes.
", "rawMarkdown": "The competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:\r\n\r\n•\tWe only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.\r\n\r\n•\tWe also only predicted for rows where N was very similar to N+1. We set all others to zero.\r\n\r\n•\tSome features were:\r\n\r\n•\tHow many rows were in a particular time-group. If there were many rows, there tended to be higher failure, and if there were few rows, there tended to be higher failure. Lowest failure was in the middle ranges.\r\n\r\n•\tSome time groups simply had higher failure rates than others. So proximity by time was an important feature.\r\n\r\n•\tTotal duration of a row from first timestamp to last timestamp.\r\n\r\n•\tSome values of L3_S32_F3854 were particularly valuable so we one-hot-encoded that.\r\n\r\n•\tThe rows immediately before and after row N were important. Specifically the Response on those rows; were they exact duplicates of row N or simply partial duplicates; the values of L3_S38_D3957 and L1_S24_F1844 on row N+1 were helpful.\r\n\r\n•\tNot all of the duplicates came in pairs. Some were in triples, quadruples, or more. So we measured how many rows were in the duplicate “set” and what position this row was (e.g. 3rd of 4).\r\n\r\n•\tAfter feature creation, we used XGBoost and Random Forests and created a weighted average of their predictions.\r\n\r\n•\tI usually prefer simpler solutions to more complex ones, but for this competition, we found that to squeeze the last 0.01 out, we had to create some complex ensembles of runs that had different subsets of the features, different algorithms, etc. In fact, one of our selected submissions simply took all of our submissions from the last month and predicted 1 if that row was 1 in 11 or more of the submissions.\r\n\r\nWe will see how it goes.", "votes": null }, { "id": "144033", "postDate": "11/12/2016 00:20:15", "content": "I also wanted to make models on only cluster part of the data. But my experiments showed that there was some extra value from running full dataset models:)
\n\nanyway, interesting observation about L1_S24; I dont know how you used it, so will have to check why its so special.
", "rawMarkdown": "I also wanted to make models on only cluster part of the data. But my experiments showed that there was some extra value from running full dataset models:)\r\n\r\nanyway, interesting observation about L1_S24; I dont know how you used it, so will have to check why its so special.", "votes": null }, { "id": "144043", "postDate": "11/12/2016 00:42:13", "content": "Main aspects of our approach (9th on private):
\n\nEngineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.
Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.
Forming time-adjusted numeric features by binning each numeric variable along its corresponding time variable and computing means and residuals.
Differencing, lag, moving average, and basic statistical features both across rows sorted by ID and/or time and across columns.
Employing a nested 5-fold cross-validation scheme, first splitting the data randomly in half (stratified on response) then doing 5-fold in each then cross-checking. This seemed like a good way to go and produced a lot of useful info along the way, but scores typically did not align well with leaderboard performance. Switched to straight 10-fold for the final two weeks.
Using xgboost and JMP. For xgb, reg:linear on the original 0s and 1s tended to produce better MCC scores than binary:logistic; however, out-of-fold predictions from the resulting models seemed difficult to ensemble. JMP was very useful for data exploration, feature engineering, and quick screening with the Partition platform.
Big congrats to the top finishers--this was a tough problem and really looking forward to hearing how you approached it!
", "rawMarkdown": "Main aspects of our approach (9th on private):\r\n\r\n- Engineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.\r\n\r\n- Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.\r\n\r\n- Forming time-adjusted numeric features by binning each numeric variable along its corresponding time variable and computing means and residuals.\r\n\r\n- Differencing, lag, moving average, and basic statistical features both across rows sorted by ID and/or time and across columns.\r\n\r\n- Employing a nested 5-fold cross-validation scheme, first splitting the data randomly in half (stratified on response) then doing 5-fold in each then cross-checking. This seemed like a good way to go and produced a lot of useful info along the way, but scores typically did not align well with leaderboard performance. Switched to straight 10-fold for the final two weeks.\r\n\r\n- Using xgboost and JMP. For xgb, reg:linear on the original 0s and 1s tended to produce better MCC scores than binary:logistic; however, out-of-fold predictions from the resulting models seemed difficult to ensemble. JMP was very useful for data exploration, feature engineering, and quick screening with the Partition platform.\r\n\r\nBig congrats to the top finishers--this was a tough problem and really looking forward to hearing how you approached it!", "votes": null }, { "id": "144052", "postDate": "11/12/2016 01:24:59", "content": "[quote=Russ W;144043]
\n\nEngineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.
Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.
[/quote]
\n\nHi Russ, \nYour seemed to get metadata from the station pattern, which i did not manage to figure out . I thought the magic feature was powerful enough, never knew there could be another :)
\n\nFor your strongest magical feature, you mean you made 'station history' into a variable, and found those correlated with higher failure rates? \nFor the second feature, i do not really understand :(
\n\nCongrats to your 9th placing,
\n\nleexa
", "rawMarkdown": "[quote=Russ W;144043]\r\n\r\n- Engineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.\r\n\r\n- Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.\r\n\r\n\r\n[/quote]\r\n\r\nHi Russ, \r\nYour seemed to get metadata from the station pattern, which i did not manage to figure out . I thought the magic feature was powerful enough, never knew there could be another :)\r\n\r\nFor your strongest magical feature, you mean you made 'station history' into a variable, and found those correlated with higher failure rates? \r\nFor the second feature, i do not really understand :(\r\n\r\nCongrats to your 9th placing,\r\n\r\nleexa", "votes": null }, { "id": "144055", "postDate": "11/12/2016 01:46:26", "content": "Hi leexa, For the first one, here's a plot, with each point corresponding to a different station activity pattern. Creating a 0-1 feature that flags the points in red is very predictive.
\n\n
The second one is much more laborious, but you can cut the number of numeric features approximately in half by condensing them by missing value pattern. When parts branch out to cloned stations, it makes sense to combine the resulting features to make them more powerful, but care must be taken to combine the correct ones. Note they are not directly detectable with correlations because the missing value blocks have little to no overlapping rows, but you can compare relative order within a station, distributions, and within-station correlation matrices.
", "rawMarkdown": "Hi leexa, For the first one, here's a plot, with each point corresponding to a different station activity pattern. Creating a 0-1 feature that flags the points in red is very predictive.\r\n\r\n![enter image description here][1]\r\n\r\nThe second one is much more laborious, but you can cut the number of numeric features approximately in half by condensing them by missing value pattern. When parts branch out to cloned stations, it makes sense to combine the resulting features to make them more powerful, but care must be taken to combine the correct ones. Note they are not directly detectable with correlations because the missing value blocks have little to no overlapping rows, but you can compare relative order within a station, distributions, and within-station correlation matrices.\r\n\r\n\r\n [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/5337/bosch_sac_mn.JPG", "votes": null }, { "id": "144057", "postDate": "11/12/2016 02:00:15", "content": "Thanks for sharing guys! Amazing insights. Russ, did you think you were going to overfit with the binary red/black feature before your cross validated? If I were looking at this graph I wouldn't bother to assign a binary variable like that because i'd assume it would only lead to my alg. overfitting the curve. Did you think about Nrows more as a categorical variable rather than a numeric one? Congratz on your placement.
", "rawMarkdown": "Thanks for sharing guys! Amazing insights. Russ, did you think you were going to overfit with the binary red/black feature before your cross validated? If I were looking at this graph I wouldn't bother to assign a binary variable like that because i'd assume it would only lead to my alg. overfitting the curve. Did you think about Nrows more as a categorical variable rather than a numeric one? Congratz on your placement.", "votes": null }, { "id": "144058", "postDate": "11/12/2016 02:00:49", "content": "@BreakfastPirate why didn't you tell me this a week ago..damn! I was so close, I knew something was up with those duplicates ...living to fight another day!.
", "rawMarkdown": "BreakfastPirate why didn't you tell me this a week ago..damn! I was so close, I knew something was up with those duplicates ...living to fight another day!.", "votes": null }, { "id": "144059", "postDate": "11/12/2016 02:12:22", "content": "Vape, yes, whenever you use the response to build a feature there is always a concern of overfitting / leakage. Probably the safest way to check is to build the feature separately for each fold, or of course the leaderboard will usually tell you. Here the signal-to-noise ratio is high enough to where it holds up, and in fact a good complementary plot is the t-statistic versus the mean. I also played some with best linear unbiased predictors / shrunken means but they tended to overfit.
", "rawMarkdown": "Vape, yes, whenever you use the response to build a feature there is always a concern of overfitting / leakage. Probably the safest way to check is to build the feature separately for each fold, or of course the leaderboard will usually tell you. Here the signal-to-noise ratio is high enough to where it holds up, and in fact a good complementary plot is the t-statistic versus the mean. I also played some with best linear unbiased predictors / shrunken means but they tended to overfit.", "votes": null }, { "id": "144069", "postDate": "11/12/2016 03:34:48", "content": "[quote=BreakfastPirate;144019]
\n\nThe competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:
\n\n• We only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.
\n\n• We also only predicted for rows where N was very similar to N+1. We set all others to zero.
\n\n[/quote]
\n\nThanks a lto for sharing so much.
\n\nI looked at this but never submitted unfortunately. Shame on me. I didn't dare set everything else to zero.
\n\n[edit] you just made me will to enter a competition again. I felt exhausted, but you show how simple and bold ideas can make a difference.
", "rawMarkdown": "[quote=BreakfastPirate;144019]\r\n\r\nThe competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:\r\n\r\n•\tWe only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.\r\n\r\n•\tWe also only predicted for rows where N was very similar to N+1. We set all others to zero.\r\n\r\n[/quote]\r\n\r\nThanks a lto for sharing so much.\r\n\r\nI looked at this but never submitted unfortunately. Shame on me. I didn't dare set everything else to zero.\r\n\r\n[edit] you just made me will to enter a competition again. I felt exhausted, but you show how simple and bold ideas can make a difference.", "votes": null }, { "id": "144085", "postDate": "11/12/2016 07:12:33", "content": "I group samples by hash values of data features(exclude site 38), then copy numeric features of site 38 to the samples in the same group, it improved compare to the original magic leak.
", "rawMarkdown": "I group samples by hash values of data features(exclude site 38), then copy numeric features of site 38 to the samples in the same group, it improved compare to the original magic leak.", "votes": null }, { "id": "144096", "postDate": "11/12/2016 10:22:31", "content": "I sort of lost interest after the leak
\n\nA simple improvement to get ~.43 was to use a variation of my script and do a logical "or" on the output from the original script plus the output when swapping visible and blind.
\n\nCongrats to the winners
", "rawMarkdown": "I sort of lost interest after the leak\r\n\r\nA simple improvement to get ~.43 was to use a variation of my script and do a logical \"or\" on the output from the original script plus the output when swapping visible and blind.\r\n\r\nCongrats to the winners", "votes": null }, { "id": "144152", "postDate": "11/12/2016 16:39:01", "content": "@BreakfastPirate,\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?
", "rawMarkdown": "BreakfastPirate,\r\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?", "votes": null }, { "id": "144154", "postDate": "11/12/2016 16:47:42", "content": "[quote=CPMP;144152]
\n\n@BreakfastPirate,\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?
\n\n[/quote]
\n\nActually since MCC was the evaluation metric, and our score was about 0.50, we calculated that a row had to have >= ~30% chance of being Response = 1 in order to improve our score. Any less than that, and the added False Positives would have brought the MCC down. I'm not sure of the exact numbers, but I think about only 0.1% of non-duplicate rows had Response = 1. So it was really difficult to find a sub-population of non-dups with probability >= 30%. I found sub-populations with probability >= 10%, but that was not good enough to improve MCC.
", "rawMarkdown": "[quote=CPMP;144152]\r\n\r\n@BreakfastPirate,\r\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?\r\n\r\n[/quote]\r\n\r\nActually since MCC was the evaluation metric, and our score was about 0.50, we calculated that a row had to have >= ~30% chance of being Response = 1 in order to improve our score. Any less than that, and the added False Positives would have brought the MCC down. I'm not sure of the exact numbers, but I think about only 0.1% of non-duplicate rows had Response = 1. So it was really difficult to find a sub-population of non-dups with probability >= 30%. I found sub-populations with probability >= 10%, but that was not good enough to improve MCC.", "votes": null }, { "id": "144155", "postDate": "11/12/2016 16:50:32", "content": "Helo Russ,\nCan you please provide more details about the figure you provided ? \nThank you.
", "rawMarkdown": "Helo Russ,\r\nCan you please provide more details about the figure you provided ? \r\nThank you.", "votes": null }, { "id": "144158", "postDate": "11/12/2016 17:54:47", "content": "Hello Jack, The first step is to create a station activity string of length 52, e.g. "11101....", indicating how that row passed through stations. Then by looking at the very nice floor visualizations from JohnM or gingerman, you can reduce the number of distinct strings, e.g. move a 1 for station 12 to a 1 for station 0. Now use this modified string variable as a grouping variable and compute means and N Rows for the response, and then use these points to make the plot. The plot shown above has 649 points.
\n\nI am wondering if perhaps this feature is a bit damp. If so, it should help to grab fewer points closer to the upper right corner. Will be curious to see if any of the top teams did anything similar, or ideally this kind of feature could play well with other great features I'm sure they engineered.
", "rawMarkdown": "Hello Jack, The first step is to create a station activity string of length 52, e.g. \"11101....\", indicating how that row passed through stations. Then by looking at the very nice floor visualizations from JohnM or gingerman, you can reduce the number of distinct strings, e.g. move a 1 for station 12 to a 1 for station 0. Now use this modified string variable as a grouping variable and compute means and N Rows for the response, and then use these points to make the plot. The plot shown above has 649 points. \r\n\r\nI am wondering if perhaps this feature is a bit damp. If so, it should help to grab fewer points closer to the upper right corner. Will be curious to see if any of the top teams did anything similar, or ideally this kind of feature could play well with other great features I'm sure they engineered.", "votes": null }, { "id": "144162", "postDate": "11/12/2016 19:14:11", "content": "Thanks to everyone who has shared their tricks so far and thanks to those who have shared their ideas and knowledge of programming in the kernels.
\n\nIn addition to a lot of dead ends, the feature that gave me the biggest edge over the starter scripts came from domain experience. It amounts to a overall material balance around the facility.
\n\nI used both the rate of accumulation and the cumulative sum (cheap integration) of accumulation. It's a bit tricky for me to describe how it was constructed in practice so here's the snippet:
\n\nNewFeats=pd.merge(NewFeats,NewFeats.groupby('Min').size().to_frame(name='groupsizemin'),\n left_on='Min',right_index=True,how='outer')\n\nNewFeats=pd.merge(NewFeats,NewFeats.groupby('Max').size().to_frame(name='groupsizemax'),\n left_on='Max',right_index=True,how="outer")\n\nratefeats=pd.merge(NewFeats.groupby('Min').size().to_frame(name='groupsizemin'),\n NewFeats.groupby('Max').size().to_frame(name='groupsizemax'),\n left_index=True,\n right_index=True,\n how='outer').fillna(0)\n\nratefeats['Accum']=ratefeats['groupsizemax'].fillna(0)-ratefeats['groupsizemin'].fillna(0)\n\nratefeats['CumSum']=ratefeats['Accum'].cumsum()\n\nratefeatsmin=NewFeats.groupby('Min').size().index\n\nratefeatsmin=pd.merge(ratefeats,pd.DataFrame(ratefeatsmin),left_index=True,right_on="Min")\n\nratefeatsmax=NewFeats.groupby('Max').size().index\n\nratefeatsmax=pd.merge(ratefeats,pd.DataFrame(ratefeatsmax),left_index=True,right_on="Max")\n\nNewFeats=pd.merge(NewFeats,\n ratefeatsmin[['Min','Accum','CumSum']],\n left_on='Min',\n right_on='Min',\n how='outer')\n\nNewFeats=pd.merge(NewFeats,\n ratefeatsmax[['Max','Accum','CumSum']],\n left_on='Max',\n right_on='Max',\n how='outer')\nThats a nice approach ClockWorkWorld - "Rate Accumulation"
", "rawMarkdown": "Thats a nice approach ClockWorkWorld - \"Rate Accumulation\"", "votes": null }, { "id": "144407", "postDate": "11/14/2016 05:07:16", "content": "[quote=Russ W;144158]
\n\n, e.g. move a 1 for station 12 to a 1 for station 0\n[/quote]
\n\n@Russ, Can you explain this step a bit clearly.
", "rawMarkdown": "[quote=Russ W;144158]\r\n\r\n, e.g. move a 1 for station 12 to a 1 for station 0\r\n[/quote]\r\n\r\n\r\n@Russ, Can you explain this step a bit clearly.", "votes": null }, { "id": "144490", "postDate": "11/14/2016 11:49:35", "content": "@neuron, For this specific example, suppose you have a station activity string with a 1 in the 13th position, indicating that that part passed through station 12. You would replace this value with a 0 and change the first value in the string to a 1, as if the part had passed through station 0 instead. A full implementation involves logical checks for all stations deemed to be clones of each other. You could alternatively reduce the overall length of the strings when making replacements, but I found it easier just to do the switching trick above.
", "rawMarkdown": "neuron, For this specific example, suppose you have a station activity string with a 1 in the 13th position, indicating that that part passed through station 12. You would replace this value with a 0 and change the first value in the string to a 1, as if the part had passed through station 0 instead. A full implementation involves logical checks for all stations deemed to be clones of each other. You could alternatively reduce the overall length of the strings when making replacements, but I found it easier just to do the switching trick above.", "votes": null }, { "id": "144497", "postDate": "11/14/2016 12:18:03", "content": "Thanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.
\n\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485
\n\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.
", "rawMarkdown": "Thanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \r\nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\r\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.\r\n\r\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485\r\n\r\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.", "votes": null }, { "id": "144508", "postDate": "11/14/2016 12:56:37", "content": "[quote=Daniel FG;144497]
\n\nAlso I agree with @raddar about training the whole dataset.
\n\n[/quote]
\n\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?
", "rawMarkdown": "[quote=Daniel FG;144497]\r\n\r\nAlso I agree with @raddar about training the whole dataset.\r\n\r\n[/quote]\r\n\r\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?", "votes": null }, { "id": "144511", "postDate": "11/14/2016 13:16:15", "content": "I predict in the whole dataset. I found that some of the L1 models were predicting about 100-150 1s of non-duplicates, with a rate TP/1s of 0.45. Averaging some of this models I predicted about 300 1s. I split the data according to the first date (LK_DT$P1 == TRUE in the code I attached), so maybe this 300 1s were also duplicated in any other way, I didn't check it.
", "rawMarkdown": "I predict in the whole dataset. I found that some of the L1 models were predicting about 100-150 1s of non-duplicates, with a rate TP/1s of 0.45. Averaging some of this models I predicted about 300 1s. I split the data according to the first date (LK_DT$P1 == TRUE in the code I attached), so maybe this 300 1s were also duplicated in any other way, I didn't check it.", "votes": null }, { "id": "144514", "postDate": "11/14/2016 13:35:14", "content": "[quote=BreakfastPirate;144508]
\n\n[quote=Daniel FG;144497]
\n\nAlso I agree with @raddar about training the whole dataset.
\n\n[/quote]
\n\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?
\n\n[/quote]
\n\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.
\n\nI guess the upside is that you could try ideas way faster than those training on the full data set.
", "rawMarkdown": "[quote=BreakfastPirate;144508]\r\n\r\n[quote=Daniel FG;144497]\r\n\r\nAlso I agree with @raddar about training the whole dataset.\r\n\r\n[/quote]\r\n\r\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?\r\n\r\n[/quote]\r\n\r\n\r\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.\r\n\r\nI guess the upside is that you could try ideas way faster than those training on the full data set.", "votes": null }, { "id": "144517", "postDate": "11/14/2016 13:56:01", "content": "[quote=CPMP;144514]
\n\n[quote=BreakfastPirate;144508]
\n\n[quote=Daniel FG;144497]
\n\nAlso I agree with @raddar about training the whole dataset.
\n\n[/quote]
\n\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?
\n\n[/quote]
\n\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.
\n\nI guess the upside is that you could try ideas way faster than those training on the full data set.
\n\n[/quote]
\n\nInteresting. Thank you. I had an early run where using only duplicates gave a slightly better CV, and I went with that from then on, but maybe it was random noise.
\n\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.
", "rawMarkdown": "[quote=CPMP;144514]\r\n\r\n[quote=BreakfastPirate;144508]\r\n\r\n[quote=Daniel FG;144497]\r\n\r\nAlso I agree with @raddar about training the whole dataset.\r\n\r\n[/quote]\r\n\r\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?\r\n\r\n[/quote]\r\n\r\n\r\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.\r\n\r\nI guess the upside is that you could try ideas way faster than those training on the full data set.\r\n\r\n\r\n\r\n[/quote]\r\n\r\nInteresting. Thank you. I had an early run where using only duplicates gave a slightly better CV, and I went with that from then on, but maybe it was random noise. \r\n\r\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.", "votes": null }, { "id": "144520", "postDate": "11/14/2016 14:00:22", "content": "[quote=BreakfastPirate;144517]
\n\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.
\n\n[/quote]
\n\nI discovered the virtues of this averaging in the last few days. Given I had a small machine, I only used a rather small subset of numeric features (280) + 40 engineered features, hence could run multiple iterations of xgboost rather quickly.
", "rawMarkdown": "[quote=BreakfastPirate;144517]\r\n\r\n\r\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.\r\n\r\n[/quote]\r\n\r\n\r\nI discovered the virtues of this averaging in the last few days. Given I had a small machine, I only used a rather small subset of numeric features (280) + 40 engineered features, hence could run multiple iterations of xgboost rather quickly.", "votes": null }, { "id": "144522", "postDate": "11/14/2016 14:08:50", "content": "Thanks BreakfastPirate, Daniel, CPMP for nice discussion and results. We also used the full data set and as a result could only manage a couple of xgboost reps on each model due to long run times.
\n\nDid you guys use binary:logistic the whole time in xgboost? We actually found reg:linear to work a little better in terms of local CV MCC performance. It produces a small number of predictions outside [0,1] but these don't matter since we're discretizing to 0s and 1s anyway. Solutions with 2850-3150 1s performed best on the LB for us.
", "rawMarkdown": "Thanks BreakfastPirate, Daniel, CPMP for nice discussion and results. We also used the full data set and as a result could only manage a couple of xgboost reps on each model due to long run times. \r\n\r\nDid you guys use binary:logistic the whole time in xgboost? We actually found reg:linear to work a little better in terms of local CV MCC performance. It produces a small number of predictions outside [0,1] but these don't matter since we're discretizing to 0s and 1s anyway. Solutions with 2850-3150 1s performed best on the LB for us.", "votes": null }, { "id": "144535", "postDate": "11/14/2016 15:23:21", "content": "I used binary:logistic only.
\n\nI also used a parameter that proved quite effective: max_delta_step=1.5
It is recommended on xgboost doc for unbalanced data set. 1.5 is the value that worked best on my local cv.
", "rawMarkdown": "I used binary:logistic only.\r\n\r\nI also used a parameter that proved quite effective: `max_delta_step=1.5`\r\n\r\nIt is recommended on xgboost doc for unbalanced data set. 1.5 is the value that worked best on my local cv.", "votes": null }, { "id": "144543", "postDate": "11/14/2016 15:47:14", "content": "Shucks, should have tried tuning max_delta_step. Our default parms were
\n\np = list(max_depth=8, eta=0.02, colsample_bytree=0.8, subsample=0.8, min_child_weight=1, \n alpha=1, gamma=0.1)
\n\nfor level 1 models then reduce the depth to 5 for level 2. This contest seemed to be way more about feature engineering so we set these kind of early and did not spend much more time on them.
", "rawMarkdown": "Shucks, should have tried tuning max_delta_step. Our default parms were\r\n\r\np = list(max_depth=8, eta=0.02, colsample_bytree=0.8, subsample=0.8, min_child_weight=1, \r\n alpha=1, gamma=0.1)\r\n\r\nfor level 1 models then reduce the depth to 5 for level 2. This contest seemed to be way more about feature engineering so we set these kind of early and did not spend much more time on them.", "votes": null }, { "id": "145428", "postDate": "11/18/2016 09:15:35", "content": "Hi Everyone,
\n\nEnding final validation, I share my solution.
\nSolutions of Prize winners remind me that ensemble is also important.
\nCode: https://github.com/toshi-k/kaggle-bosch-production-line-performance
![\"conceptual](\"https://raw.githubusercontent.com/toshi-k/kaggle-bosch-production-line-performance/master/img/solution.png\" "\\"\\"")
Lovely Picture!
", "rawMarkdown": "Lovely Picture!", "votes": null }, { "id": "145899", "postDate": "11/21/2016 17:04:30", "content": "[quote=Daniel FG;144497]
\n\nThanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.
\n\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485
\n\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.
\n\n[/quote]
\n\nThank Daniel FG for sharing. Could you give a brief description about Feature Engineering (1) and Feature Engineering (2) parts in your code? I am new to R, so it's a little bit difficult to understand your work. Thanks!
", "rawMarkdown": "[quote=Daniel FG;144497]\r\n\r\nThanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \r\nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\r\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.\r\n\r\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485\r\n\r\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.\r\n\r\n[/quote]\r\n\r\nThank Daniel FG for sharing. Could you give a brief description about Feature Engineering (1) and Feature Engineering (2) parts in your code? I am new to R, so it's a little bit difficult to understand your work. Thanks!", "votes": null }, { "id": "146028", "postDate": "11/22/2016 12:18:46", "content": "Hi JiansheFeng,
\n\nI hope this helps:
\n\n[quote=Daniel FG;146028]
\n\nHi JiansheFeng,
\n\nI hope this helps:
\n\n[/quote]\nHi Daniel FG, Thanks so much! It really helps.
", "rawMarkdown": "[quote=Daniel FG;146028]\r\n\r\nHi JiansheFeng,\r\n\r\nI hope this helps:\r\n\r\n- P1: Boolean, whether the first date is equal to the first date of the previous/next row (ID sorted). This split the data in 'leak' or 'out-of-leak'.\r\n- ord: When P1 is true, relative order of subgroup of correlatives examples with same first date.\r\n- group_len: When P1 is true, size of subgroup.\r\n- fst: First date (ordered by stations, as given).\r\n- L3.fst: First date in Line 3.\r\n- L3.lst: Last date in Line 3 (last date in the series, not always the maximum date value).\r\n- time_dtL3: difference between time_L3 [total time in Line 3, max(date) - min(date)] and time_L3 for the previous row.\r\n- time_idtL3: difference between time_L3 and time_L3 for the next row\r\n- NAs_dtL3: difference between L3.NAs [NA count] and L3.NAs for the previous row.\r\n- NAs_idtL3: difference between L3.NAs and L3.NAs for the next row.\r\n\r\n\r\n[/quote]\r\nHi Daniel FG, Thanks so much! It really helps.", "votes": null }, { "id": "183861", "postDate": "05/19/2017 14:13:46", "content": "i have run in a problem in your code. made an issue in github. please check. thanks :)
", "rawMarkdown": "i have run in a problem in your code. made an issue in github. please check. thanks :)", "votes": null } ], "comments": [ { "id": 144033, "author_name": "raddar", "author_url": "", "post_date": "11/12/2016 00:20:15", "content": "I also wanted to make models on only cluster part of the data. But my experiments showed that there was some extra value from running full dataset models:)
\n\nanyway, interesting observation about L1_S24; I dont know how you used it, so will have to check why its so special.
", "votes": null, "replies": [] }, { "id": 144043, "author_name": "sasrdw", "author_url": "", "post_date": "11/12/2016 00:42:13", "content": "Main aspects of our approach (9th on private):
\n\nEngineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.
Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.
Forming time-adjusted numeric features by binning each numeric variable along its corresponding time variable and computing means and residuals.
Differencing, lag, moving average, and basic statistical features both across rows sorted by ID and/or time and across columns.
Employing a nested 5-fold cross-validation scheme, first splitting the data randomly in half (stratified on response) then doing 5-fold in each then cross-checking. This seemed like a good way to go and produced a lot of useful info along the way, but scores typically did not align well with leaderboard performance. Switched to straight 10-fold for the final two weeks.
Using xgboost and JMP. For xgb, reg:linear on the original 0s and 1s tended to produce better MCC scores than binary:logistic; however, out-of-fold predictions from the resulting models seemed difficult to ensemble. JMP was very useful for data exploration, feature engineering, and quick screening with the Partition platform.
Big congrats to the top finishers--this was a tough problem and really looking forward to hearing how you approached it!
", "votes": null, "replies": [] }, { "id": 144052, "author_name": "leexiongan", "author_url": "", "post_date": "11/12/2016 01:24:59", "content": "[quote=Russ W;144043]
\n\nEngineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.
Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.
[/quote]
\n\nHi Russ, \nYour seemed to get metadata from the station pattern, which i did not manage to figure out . I thought the magic feature was powerful enough, never knew there could be another :)
\n\nFor your strongest magical feature, you mean you made 'station history' into a variable, and found those correlated with higher failure rates? \nFor the second feature, i do not really understand :(
\n\nCongrats to your 9th placing,
\n\nleexa
", "votes": null, "replies": [] }, { "id": 144055, "author_name": "sasrdw", "author_url": "", "post_date": "11/12/2016 01:46:26", "content": "Hi leexa, For the first one, here's a plot, with each point corresponding to a different station activity pattern. Creating a 0-1 feature that flags the points in red is very predictive.
\n\n
The second one is much more laborious, but you can cut the number of numeric features approximately in half by condensing them by missing value pattern. When parts branch out to cloned stations, it makes sense to combine the resulting features to make them more powerful, but care must be taken to combine the correct ones. Note they are not directly detectable with correlations because the missing value blocks have little to no overlapping rows, but you can compare relative order within a station, distributions, and within-station correlation matrices.
", "votes": null, "replies": [] }, { "id": 144057, "author_name": "vapenaysh", "author_url": "", "post_date": "11/12/2016 02:00:15", "content": "Thanks for sharing guys! Amazing insights. Russ, did you think you were going to overfit with the binary red/black feature before your cross validated? If I were looking at this graph I wouldn't bother to assign a binary variable like that because i'd assume it would only lead to my alg. overfitting the curve. Did you think about Nrows more as a categorical variable rather than a numeric one? Congratz on your placement.
", "votes": null, "replies": [] }, { "id": 144058, "author_name": "godaibo", "author_url": "", "post_date": "11/12/2016 02:00:49", "content": "@BreakfastPirate why didn't you tell me this a week ago..damn! I was so close, I knew something was up with those duplicates ...living to fight another day!.
", "votes": null, "replies": [] }, { "id": 144059, "author_name": "sasrdw", "author_url": "", "post_date": "11/12/2016 02:12:22", "content": "Vape, yes, whenever you use the response to build a feature there is always a concern of overfitting / leakage. Probably the safest way to check is to build the feature separately for each fold, or of course the leaderboard will usually tell you. Here the signal-to-noise ratio is high enough to where it holds up, and in fact a good complementary plot is the t-statistic versus the mean. I also played some with best linear unbiased predictors / shrunken means but they tended to overfit.
", "votes": null, "replies": [] }, { "id": 144069, "author_name": "cpmpml", "author_url": "", "post_date": "11/12/2016 03:34:48", "content": "[quote=BreakfastPirate;144019]
\n\nThe competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:
\n\n• We only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.
\n\n• We also only predicted for rows where N was very similar to N+1. We set all others to zero.
\n\n[/quote]
\n\nThanks a lto for sharing so much.
\n\nI looked at this but never submitted unfortunately. Shame on me. I didn't dare set everything else to zero.
\n\n[edit] you just made me will to enter a competition again. I felt exhausted, but you show how simple and bold ideas can make a difference.
", "votes": null, "replies": [] }, { "id": 144085, "author_name": "breakhearts", "author_url": "", "post_date": "11/12/2016 07:12:33", "content": "I group samples by hash values of data features(exclude site 38), then copy numeric features of site 38 to the samples in the same group, it improved compare to the original magic leak.
", "votes": null, "replies": [] }, { "id": 144096, "author_name": "scirpus", "author_url": "", "post_date": "11/12/2016 10:22:31", "content": "I sort of lost interest after the leak
\n\nA simple improvement to get ~.43 was to use a variation of my script and do a logical "or" on the output from the original script plus the output when swapping visible and blind.
\n\nCongrats to the winners
", "votes": null, "replies": [] }, { "id": 144152, "author_name": "cpmpml", "author_url": "", "post_date": "11/12/2016 16:39:01", "content": "@BreakfastPirate,\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?
", "votes": null, "replies": [] }, { "id": 144154, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/12/2016 16:47:42", "content": "[quote=CPMP;144152]
\n\n@BreakfastPirate,\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?
\n\n[/quote]
\n\nActually since MCC was the evaluation metric, and our score was about 0.50, we calculated that a row had to have >= ~30% chance of being Response = 1 in order to improve our score. Any less than that, and the added False Positives would have brought the MCC down. I'm not sure of the exact numbers, but I think about only 0.1% of non-duplicate rows had Response = 1. So it was really difficult to find a sub-population of non-dups with probability >= 30%. I found sub-populations with probability >= 10%, but that was not good enough to improve MCC.
", "votes": null, "replies": [] }, { "id": 144155, "author_name": "jackabdo", "author_url": "", "post_date": "11/12/2016 16:50:32", "content": "Helo Russ,\nCan you please provide more details about the figure you provided ? \nThank you.
", "votes": null, "replies": [] }, { "id": 144158, "author_name": "sasrdw", "author_url": "", "post_date": "11/12/2016 17:54:47", "content": "Hello Jack, The first step is to create a station activity string of length 52, e.g. "11101....", indicating how that row passed through stations. Then by looking at the very nice floor visualizations from JohnM or gingerman, you can reduce the number of distinct strings, e.g. move a 1 for station 12 to a 1 for station 0. Now use this modified string variable as a grouping variable and compute means and N Rows for the response, and then use these points to make the plot. The plot shown above has 649 points.
\n\nI am wondering if perhaps this feature is a bit damp. If so, it should help to grab fewer points closer to the upper right corner. Will be curious to see if any of the top teams did anything similar, or ideally this kind of feature could play well with other great features I'm sure they engineered.
", "votes": null, "replies": [] }, { "id": 144162, "author_name": "clockworkworld", "author_url": "", "post_date": "11/12/2016 19:14:11", "content": "Thanks to everyone who has shared their tricks so far and thanks to those who have shared their ideas and knowledge of programming in the kernels.
\n\nIn addition to a lot of dead ends, the feature that gave me the biggest edge over the starter scripts came from domain experience. It amounts to a overall material balance around the facility.
\n\nI used both the rate of accumulation and the cumulative sum (cheap integration) of accumulation. It's a bit tricky for me to describe how it was constructed in practice so here's the snippet:
\n\nNewFeats=pd.merge(NewFeats,NewFeats.groupby('Min').size().to_frame(name='groupsizemin'),\n left_on='Min',right_index=True,how='outer')\n\nNewFeats=pd.merge(NewFeats,NewFeats.groupby('Max').size().to_frame(name='groupsizemax'),\n left_on='Max',right_index=True,how="outer")\n\nratefeats=pd.merge(NewFeats.groupby('Min').size().to_frame(name='groupsizemin'),\n NewFeats.groupby('Max').size().to_frame(name='groupsizemax'),\n left_index=True,\n right_index=True,\n how='outer').fillna(0)\n\nratefeats['Accum']=ratefeats['groupsizemax'].fillna(0)-ratefeats['groupsizemin'].fillna(0)\n\nratefeats['CumSum']=ratefeats['Accum'].cumsum()\n\nratefeatsmin=NewFeats.groupby('Min').size().index\n\nratefeatsmin=pd.merge(ratefeats,pd.DataFrame(ratefeatsmin),left_index=True,right_on="Min")\n\nratefeatsmax=NewFeats.groupby('Max').size().index\n\nratefeatsmax=pd.merge(ratefeats,pd.DataFrame(ratefeatsmax),left_index=True,right_on="Max")\n\nNewFeats=pd.merge(NewFeats,\n ratefeatsmin[['Min','Accum','CumSum']],\n left_on='Min',\n right_on='Min',\n how='outer')\n\nNewFeats=pd.merge(NewFeats,\n ratefeatsmax[['Max','Accum','CumSum']],\n left_on='Max',\n right_on='Max',\n how='outer')\nThats a nice approach ClockWorkWorld - "Rate Accumulation"
", "votes": null, "replies": [] }, { "id": 144407, "author_name": "kumaran5555", "author_url": "", "post_date": "11/14/2016 05:07:16", "content": "[quote=Russ W;144158]
\n\n, e.g. move a 1 for station 12 to a 1 for station 0\n[/quote]
\n\n@Russ, Can you explain this step a bit clearly.
", "votes": null, "replies": [] }, { "id": 144490, "author_name": "sasrdw", "author_url": "", "post_date": "11/14/2016 11:49:35", "content": "@neuron, For this specific example, suppose you have a station activity string with a 1 in the 13th position, indicating that that part passed through station 12. You would replace this value with a 0 and change the first value in the string to a 1, as if the part had passed through station 0 instead. A full implementation involves logical checks for all stations deemed to be clones of each other. You could alternatively reduce the overall length of the strings when making replacements, but I found it easier just to do the switching trick above.
", "votes": null, "replies": [] }, { "id": 144497, "author_name": "danielfg", "author_url": "", "post_date": "11/14/2016 12:18:03", "content": "Thanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.
\n\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485
\n\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.
", "votes": null, "replies": [] }, { "id": 144508, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/14/2016 12:56:37", "content": "[quote=Daniel FG;144497]
\n\nAlso I agree with @raddar about training the whole dataset.
\n\n[/quote]
\n\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?
", "votes": null, "replies": [] }, { "id": 144511, "author_name": "danielfg", "author_url": "", "post_date": "11/14/2016 13:16:15", "content": "I predict in the whole dataset. I found that some of the L1 models were predicting about 100-150 1s of non-duplicates, with a rate TP/1s of 0.45. Averaging some of this models I predicted about 300 1s. I split the data according to the first date (LK_DT$P1 == TRUE in the code I attached), so maybe this 300 1s were also duplicated in any other way, I didn't check it.
", "votes": null, "replies": [] }, { "id": 144514, "author_name": "cpmpml", "author_url": "", "post_date": "11/14/2016 13:35:14", "content": "[quote=BreakfastPirate;144508]
\n\n[quote=Daniel FG;144497]
\n\nAlso I agree with @raddar about training the whole dataset.
\n\n[/quote]
\n\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?
\n\n[/quote]
\n\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.
\n\nI guess the upside is that you could try ideas way faster than those training on the full data set.
", "votes": null, "replies": [] }, { "id": 144517, "author_name": "breakfastpirate", "author_url": "", "post_date": "11/14/2016 13:56:01", "content": "[quote=CPMP;144514]
\n\n[quote=BreakfastPirate;144508]
\n\n[quote=Daniel FG;144497]
\n\nAlso I agree with @raddar about training the whole dataset.
\n\n[/quote]
\n\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?
\n\n[/quote]
\n\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.
\n\nI guess the upside is that you could try ideas way faster than those training on the full data set.
\n\n[/quote]
\n\nInteresting. Thank you. I had an early run where using only duplicates gave a slightly better CV, and I went with that from then on, but maybe it was random noise.
\n\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.
", "votes": null, "replies": [] }, { "id": 144520, "author_name": "cpmpml", "author_url": "", "post_date": "11/14/2016 14:00:22", "content": "[quote=BreakfastPirate;144517]
\n\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.
\n\n[/quote]
\n\nI discovered the virtues of this averaging in the last few days. Given I had a small machine, I only used a rather small subset of numeric features (280) + 40 engineered features, hence could run multiple iterations of xgboost rather quickly.
", "votes": null, "replies": [] }, { "id": 144522, "author_name": "sasrdw", "author_url": "", "post_date": "11/14/2016 14:08:50", "content": "Thanks BreakfastPirate, Daniel, CPMP for nice discussion and results. We also used the full data set and as a result could only manage a couple of xgboost reps on each model due to long run times.
\n\nDid you guys use binary:logistic the whole time in xgboost? We actually found reg:linear to work a little better in terms of local CV MCC performance. It produces a small number of predictions outside [0,1] but these don't matter since we're discretizing to 0s and 1s anyway. Solutions with 2850-3150 1s performed best on the LB for us.
", "votes": null, "replies": [] }, { "id": 144535, "author_name": "cpmpml", "author_url": "", "post_date": "11/14/2016 15:23:21", "content": "I used binary:logistic only.
\n\nI also used a parameter that proved quite effective: max_delta_step=1.5
It is recommended on xgboost doc for unbalanced data set. 1.5 is the value that worked best on my local cv.
", "votes": null, "replies": [] }, { "id": 144543, "author_name": "sasrdw", "author_url": "", "post_date": "11/14/2016 15:47:14", "content": "Shucks, should have tried tuning max_delta_step. Our default parms were
\n\np = list(max_depth=8, eta=0.02, colsample_bytree=0.8, subsample=0.8, min_child_weight=1, \n alpha=1, gamma=0.1)
\n\nfor level 1 models then reduce the depth to 5 for level 2. This contest seemed to be way more about feature engineering so we set these kind of early and did not spend much more time on them.
", "votes": null, "replies": [] }, { "id": 145428, "author_name": "toshik", "author_url": "", "post_date": "11/18/2016 09:15:35", "content": "Hi Everyone,
\n\nEnding final validation, I share my solution.
\nSolutions of Prize winners remind me that ensemble is also important.
\nCode: https://github.com/toshi-k/kaggle-bosch-production-line-performance
i have run in a problem in your code. made an issue in github. please check. thanks :)
", "votes": null, "replies": [] } ] }, { "id": 145459, "author_name": "scirpus", "author_url": "", "post_date": "11/18/2016 13:41:22", "content": "Lovely Picture!
", "votes": null, "replies": [] }, { "id": 145899, "author_name": "jianshefeng", "author_url": "", "post_date": "11/21/2016 17:04:30", "content": "[quote=Daniel FG;144497]
\n\nThanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.
\n\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485
\n\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.
\n\n[/quote]
\n\nThank Daniel FG for sharing. Could you give a brief description about Feature Engineering (1) and Feature Engineering (2) parts in your code? I am new to R, so it's a little bit difficult to understand your work. Thanks!
", "votes": null, "replies": [] }, { "id": 146028, "author_name": "danielfg", "author_url": "", "post_date": "11/22/2016 12:18:46", "content": "Hi JiansheFeng,
\n\nI hope this helps:
\n\n[quote=Daniel FG;146028]
\n\nHi JiansheFeng,
\n\nI hope this helps:
\n\n[/quote]\nHi Daniel FG, Thanks so much! It really helps.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "144019": "The competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:\r\n\r\n•\tWe only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.\r\n\r\n•\tWe also only predicted for rows where N was very similar to N+1. We set all others to zero.\r\n\r\n•\tSome features were:\r\n\r\n•\tHow many rows were in a particular time-group. If there were many rows, there tended to be higher failure, and if there were few rows, there tended to be higher failure. Lowest failure was in the middle ranges.\r\n\r\n•\tSome time groups simply had higher failure rates than others. So proximity by time was an important feature.\r\n\r\n•\tTotal duration of a row from first timestamp to last timestamp.\r\n\r\n•\tSome values of L3_S32_F3854 were particularly valuable so we one-hot-encoded that.\r\n\r\n•\tThe rows immediately before and after row N were important. Specifically the Response on those rows; were they exact duplicates of row N or simply partial duplicates; the values of L3_S38_D3957 and L1_S24_F1844 on row N+1 were helpful.\r\n\r\n•\tNot all of the duplicates came in pairs. Some were in triples, quadruples, or more. So we measured how many rows were in the duplicate “set” and what position this row was (e.g. 3rd of 4).\r\n\r\n•\tAfter feature creation, we used XGBoost and Random Forests and created a weighted average of their predictions.\r\n\r\n•\tI usually prefer simpler solutions to more complex ones, but for this competition, we found that to squeeze the last 0.01 out, we had to create some complex ensembles of runs that had different subsets of the features, different algorithms, etc. In fact, one of our selected submissions simply took all of our submissions from the last month and predicted 1 if that row was 1 in 11 or more of the submissions.\r\n\r\nWe will see how it goes.", "144033": "I also wanted to make models on only cluster part of the data. But my experiments showed that there was some extra value from running full dataset models:)\r\n\r\nanyway, interesting observation about L1_S24; I dont know how you used it, so will have to check why its so special.", "144043": "Main aspects of our approach (9th on private):\r\n\r\n- Engineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.\r\n\r\n- Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.\r\n\r\n- Forming time-adjusted numeric features by binning each numeric variable along its corresponding time variable and computing means and residuals.\r\n\r\n- Differencing, lag, moving average, and basic statistical features both across rows sorted by ID and/or time and across columns.\r\n\r\n- Employing a nested 5-fold cross-validation scheme, first splitting the data randomly in half (stratified on response) then doing 5-fold in each then cross-checking. This seemed like a good way to go and produced a lot of useful info along the way, but scores typically did not align well with leaderboard performance. Switched to straight 10-fold for the final two weeks.\r\n\r\n- Using xgboost and JMP. For xgb, reg:linear on the original 0s and 1s tended to produce better MCC scores than binary:logistic; however, out-of-fold predictions from the resulting models seemed difficult to ensemble. JMP was very useful for data exploration, feature engineering, and quick screening with the Partition platform.\r\n\r\nBig congrats to the top finishers--this was a tough problem and really looking forward to hearing how you approached it!", "144052": "[quote=Russ W;144043]\r\n\r\n- Engineering one really strong feature (higher in xgb importance than magic ones) obtained by creating a collapsed station activity character string of 0s and 1s based on the nice floor plot by gingerman, then forming means and N of the response grouped by this character var. Using the resulting summary data, plot the means vs N on a log-log scale and around a dozen of the values stick out towards the upper right. Hand select them and form an indicator variable for them in the original data.\r\n\r\n- Creating condensed numeric features based on the checkerboard missing value pattern; kind of like the game of tetris, but need to verify variables with similar distributions are merged correctly. Especially difficult for Line 1.\r\n\r\n\r\n[/quote]\r\n\r\nHi Russ, \r\nYour seemed to get metadata from the station pattern, which i did not manage to figure out . I thought the magic feature was powerful enough, never knew there could be another :)\r\n\r\nFor your strongest magical feature, you mean you made 'station history' into a variable, and found those correlated with higher failure rates? \r\nFor the second feature, i do not really understand :(\r\n\r\nCongrats to your 9th placing,\r\n\r\nleexa", "144055": "Hi leexa, For the first one, here's a plot, with each point corresponding to a different station activity pattern. Creating a 0-1 feature that flags the points in red is very predictive.\r\n\r\n![enter image description here][1]\r\n\r\nThe second one is much more laborious, but you can cut the number of numeric features approximately in half by condensing them by missing value pattern. When parts branch out to cloned stations, it makes sense to combine the resulting features to make them more powerful, but care must be taken to combine the correct ones. Note they are not directly detectable with correlations because the missing value blocks have little to no overlapping rows, but you can compare relative order within a station, distributions, and within-station correlation matrices.\r\n\r\n\r\n [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/5337/bosch_sac_mn.JPG", "144057": "Thanks for sharing guys! Amazing insights. Russ, did you think you were going to overfit with the binary red/black feature before your cross validated? If I were looking at this graph I wouldn't bother to assign a binary variable like that because i'd assume it would only lead to my alg. overfitting the curve. Did you think about Nrows more as a categorical variable rather than a numeric one? Congratz on your placement.", "144058": "BreakfastPirate why didn't you tell me this a week ago..damn! I was so close, I knew something was up with those duplicates ...living to fight another day!.", "144059": "Vape, yes, whenever you use the response to build a feature there is always a concern of overfitting / leakage. Probably the safest way to check is to build the feature separately for each fold, or of course the leaderboard will usually tell you. Here the signal-to-noise ratio is high enough to where it holds up, and in fact a good complementary plot is the t-statistic versus the mean. I also played some with best linear unbiased predictors / shrunken means but they tended to overfit.", "144069": "[quote=BreakfastPirate;144019]\r\n\r\nThe competition only has a couple minutes left so I think it is safe to start sharing. Don’t know whether our solution will drop like a rock or not, but here it is:\r\n\r\n•\tWe only trained on rows that were part of duplicates – specifically row N was very similar to row N+1. It turns out that this set was almost exactly the same as Faron’s feature that measured whether the IDs were consecutive in a time-group.\r\n\r\n•\tWe also only predicted for rows where N was very similar to N+1. We set all others to zero.\r\n\r\n[/quote]\r\n\r\nThanks a lto for sharing so much.\r\n\r\nI looked at this but never submitted unfortunately. Shame on me. I didn't dare set everything else to zero.\r\n\r\n[edit] you just made me will to enter a competition again. I felt exhausted, but you show how simple and bold ideas can make a difference.", "144085": "I group samples by hash values of data features(exclude site 38), then copy numeric features of site 38 to the samples in the same group, it improved compare to the original magic leak.", "144096": "I sort of lost interest after the leak\r\n\r\nA simple improvement to get ~.43 was to use a variation of my script and do a logical \"or\" on the output from the original script plus the output when swapping visible and blind.\r\n\r\nCongrats to the winners", "144152": "BreakfastPirate,\r\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?", "144154": "[quote=CPMP;144152]\r\n\r\n@BreakfastPirate,\r\nI may miss something, but I reckon that anything better than random on the rows you set to 0 would improve your score. Did you try it?\r\n\r\n[/quote]\r\n\r\nActually since MCC was the evaluation metric, and our score was about 0.50, we calculated that a row had to have >= ~30% chance of being Response = 1 in order to improve our score. Any less than that, and the added False Positives would have brought the MCC down. I'm not sure of the exact numbers, but I think about only 0.1% of non-duplicate rows had Response = 1. So it was really difficult to find a sub-population of non-dups with probability >= 30%. I found sub-populations with probability >= 10%, but that was not good enough to improve MCC.", "144155": "Helo Russ,\r\nCan you please provide more details about the figure you provided ? \r\nThank you.", "144158": "Hello Jack, The first step is to create a station activity string of length 52, e.g. \"11101....\", indicating how that row passed through stations. Then by looking at the very nice floor visualizations from JohnM or gingerman, you can reduce the number of distinct strings, e.g. move a 1 for station 12 to a 1 for station 0. Now use this modified string variable as a grouping variable and compute means and N Rows for the response, and then use these points to make the plot. The plot shown above has 649 points. \r\n\r\nI am wondering if perhaps this feature is a bit damp. If so, it should help to grab fewer points closer to the upper right corner. Will be curious to see if any of the top teams did anything similar, or ideally this kind of feature could play well with other great features I'm sure they engineered.", "144162": "Thanks to everyone who has shared their tricks so far and thanks to those who have shared their ideas and knowledge of programming in the kernels.\r\n\r\nIn addition to a lot of dead ends, the feature that gave me the biggest edge over the starter scripts came from domain experience. It amounts to a overall material balance around the facility.\r\n\r\n - [Rate In] - [Rate Out] = [Rate Accumulation]\r\n\r\nI used both the rate of accumulation and the cumulative sum (cheap integration) of accumulation. It's a bit tricky for me to describe how it was constructed in practice so here's the snippet:\r\n\r\n NewFeats=pd.merge(NewFeats,NewFeats.groupby('Min').size().to_frame(name='groupsizemin'),\r\n left_on='Min',right_index=True,how='outer')\r\n\r\n NewFeats=pd.merge(NewFeats,NewFeats.groupby('Max').size().to_frame(name='groupsizemax'),\r\n left_on='Max',right_index=True,how=\"outer\")\r\n\r\n ratefeats=pd.merge(NewFeats.groupby('Min').size().to_frame(name='groupsizemin'),\r\n NewFeats.groupby('Max').size().to_frame(name='groupsizemax'),\r\n left_index=True,\r\n right_index=True,\r\n how='outer').fillna(0)\r\n\r\n ratefeats['Accum']=ratefeats['groupsizemax'].fillna(0)-ratefeats['groupsizemin'].fillna(0)\r\n\r\n ratefeats['CumSum']=ratefeats['Accum'].cumsum()\r\n\r\n ratefeatsmin=NewFeats.groupby('Min').size().index\r\n\r\n ratefeatsmin=pd.merge(ratefeats,pd.DataFrame(ratefeatsmin),left_index=True,right_on=\"Min\")\r\n\r\n ratefeatsmax=NewFeats.groupby('Max').size().index\r\n\r\n ratefeatsmax=pd.merge(ratefeats,pd.DataFrame(ratefeatsmax),left_index=True,right_on=\"Max\")\r\n\r\n NewFeats=pd.merge(NewFeats,\r\n ratefeatsmin[['Min','Accum','CumSum']],\r\n left_on='Min',\r\n right_on='Min',\r\n how='outer')\r\n\r\n NewFeats=pd.merge(NewFeats,\r\n ratefeatsmax[['Max','Accum','CumSum']],\r\n left_on='Max',\r\n right_on='Max',\r\n how='outer')", "144172": "Thats a nice approach ClockWorkWorld - \"Rate Accumulation\"", "144407": "[quote=Russ W;144158]\r\n\r\n, e.g. move a 1 for station 12 to a 1 for station 0\r\n[/quote]\r\n\r\n\r\n@Russ, Can you explain this step a bit clearly.", "144490": "neuron, For this specific example, suppose you have a station activity string with a 1 in the 13th position, indicating that that part passed through station 12. You would replace this value with a 0 and change the first value in the string to a 1, as if the part had passed through station 0 instead. A full implementation involves logical checks for all stations deemed to be clones of each other. You could alternatively reduce the overall length of the strings when making replacements, but I found it easier just to do the switching trick above.", "144497": "Thanks to Bosh and Kaggle for this competition, and congratulation for the top teams, especially the winners ash&beluga. \r\nFor feature engineering I did pretty much the same than other teams already explained (@BreakfastPirate). Also I agree with @raddar about training the whole dataset.\r\nAs an example of feature engineering I have uploaded this single model (LB: 0.485). I hope this can help those who couldn't get that score. It can be improved a lot, but I had to tune parameters to fit the 20 minutes limitation.\r\n\r\nhttps://www.kaggle.com/danielfg/bosch-production-line-performance/xgboost-reg-linear-lb-0-485\r\n\r\nI trained about 100 L1 models with subset of the data in multiple combinations. My final solution is an average of about 10 L2 models. I only used XGB for this competition.", "144508": "[quote=Daniel FG;144497]\r\n\r\nAlso I agree with @raddar about training the whole dataset.\r\n\r\n[/quote]\r\n\r\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?", "144511": "I predict in the whole dataset. I found that some of the L1 models were predicting about 100-150 1s of non-duplicates, with a rate TP/1s of 0.45. Averaging some of this models I predicted about 300 1s. I split the data according to the first date (LK_DT$P1 == TRUE in the code I attached), so maybe this 300 1s were also duplicated in any other way, I didn't check it.", "144514": "[quote=BreakfastPirate;144508]\r\n\r\n[quote=Daniel FG;144497]\r\n\r\nAlso I agree with @raddar about training the whole dataset.\r\n\r\n[/quote]\r\n\r\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?\r\n\r\n[/quote]\r\n\r\n\r\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.\r\n\r\nI guess the upside is that you could try ideas way faster than those training on the full data set.", "144517": "[quote=CPMP;144514]\r\n\r\n[quote=BreakfastPirate;144508]\r\n\r\n[quote=Daniel FG;144497]\r\n\r\nAlso I agree with @raddar about training the whole dataset.\r\n\r\n[/quote]\r\n\r\nInteresting. You both beat us so I guess I can't argue with success :-) Did you also predict on the whole test dataset? If so, did you have any 1s that were for non-duplicates?\r\n\r\n[/quote]\r\n\r\n\r\nFor what its worth given my final rank, I implemented your idea after reading about it, i.e. running my model only on duplicated rows, then complementing with 0s. I found that local cv and LB degraded a bit.\r\n\r\nI guess the upside is that you could try ideas way faster than those training on the full data set.\r\n\r\n\r\n\r\n[/quote]\r\n\r\nInteresting. Thank you. I had an early run where using only duplicates gave a slightly better CV, and I went with that from then on, but maybe it was random noise. \r\n\r\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.", "144520": "[quote=BreakfastPirate;144517]\r\n\r\n\r\nSomething related to this. When we created an XGBoost model, we ran XGboost 10 times with 10 different random seeds. We then averaged the predicted pre-binary values and derived our binary predictions from that average. So running on only the duplicates speeded things up enough to allow us to average multiple iterations of XGBoost.\r\n\r\n[/quote]\r\n\r\n\r\nI discovered the virtues of this averaging in the last few days. Given I had a small machine, I only used a rather small subset of numeric features (280) + 40 engineered features, hence could run multiple iterations of xgboost rather quickly.", "144522": "Thanks BreakfastPirate, Daniel, CPMP for nice discussion and results. We also used the full data set and as a result could only manage a couple of xgboost reps on each model due to long run times. \r\n\r\nDid you guys use binary:logistic the whole time in xgboost? We actually found reg:linear to work a little better in terms of local CV MCC performance. It produces a small number of predictions outside [0,1] but these don't matter since we're discretizing to 0s and 1s anyway. Solutions with 2850-3150 1s performed best on the LB for us.", "144535": "I used binary:logistic only.\r\n\r\nI also used a parameter that proved quite effective: `max_delta_step=1.5`\r\n\r\nIt is recommended on xgboost doc for unbalanced data set. 1.5 is the value that worked best on my local cv.", "144543": "Shucks, should have tried tuning max_delta_step. Our default parms were\r\n\r\np = list(max_depth=8, eta=0.02, colsample_bytree=0.8, subsample=0.8, min_child_weight=1, \r\n alpha=1, gamma=0.1)\r\n\r\nfor level 1 models then reduce the depth to 5 for level 2. This contest seemed to be way more about feature engineering so we set these kind of early and did not spend much more time on them.", "145428": "Hi Everyone,\n\nEnding final validation, I share my solution.