There is a difference of 0.1 between 1st and 2nd team. They are also leading with the maximum number of submissions.
\n\nThere are two possibilities:
\n\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models.
\n\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.
", "messages": [ { "id": "136789", "postDate": "09/25/2016 08:53:08", "content": "There is a difference of 0.1 between 1st and 2nd team. They are also leading with the maximum number of submissions.
\n\nThere are two possibilities:
\n\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models.
\n\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.
", "rawMarkdown": "There is a difference of 0.1 between 1st and 2nd team. They are also leading with the maximum number of submissions. \r\n\r\nThere are two possibilities:\r\n\r\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models. \r\n\r\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.", "votes": null }, { "id": "136790", "postDate": "09/25/2016 09:26:43", "content": "I think they found something .
", "rawMarkdown": "I think they found something .", "votes": null }, { "id": "136791", "postDate": "09/25/2016 09:52:54", "content": "Indeed I think they found a leak but it's hard to say "over-fitting" even their model might not work well with other test dataset. In the end as long as they don't violate the rule they deserve the top ranking.
\n\nSo curious about their solution as I have never been able to get that score locally in my CV even I tried to overfit the train dataset :-)
", "rawMarkdown": "Indeed I think they found a leak but it's hard to say \"over-fitting\" even their model might not work well with other test dataset. In the end as long as they don't violate the rule they deserve the top ranking.\r\n\r\nSo curious about their solution as I have never been able to get that score locally in my CV even I tried to overfit the train dataset :-)", "votes": null }, { "id": "136792", "postDate": "09/25/2016 10:05:15", "content": "1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve.
\n\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.
\n\nGood luck! ;)
\n\n- Mikel.
", "rawMarkdown": "1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve. \r\n\r\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.\r\n\r\nGood luck! ;)\r\n\r\n\\- Mikel.", "votes": null }, { "id": "136793", "postDate": "09/25/2016 10:10:48", "content": "good luck :)
", "rawMarkdown": "good luck :)", "votes": null }, { "id": "136795", "postDate": "09/25/2016 10:12:04", "content": "[quote=Mohd Shadab Alam;136789]
\n\nThere are two possibilities:
\n\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models.
\n\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.
\n\n[/quote]
\n\nWhy does it have to be over-fitting or a leak which causes the significant increase? There is probably a little bit of overfitting because you can try different cutoff-levels for each new "model", but your argument lends little evidence towards how overfitting the public LB would lead to a 25% improvement on top of the 2nd place team.
\n\nIn general a leak is caused by information that is provided in the dataset that you wouldn't have in real life. Can you explain why there could/would be a leak in this dataset? Just because there may be a feature that has significant predictive power (which others haven't discovered yet) doesn't mean that there is a leak. Let's say that they manually found that a specific station at a specific time is causing 50% of the faulty observations in the data, that would seem pretty reasonable to me, and your machine learning model probably cannot pick up specifically on that given the size of the dataset.
", "rawMarkdown": "[quote=Mohd Shadab Alam;136789] \r\n\r\nThere are two possibilities:\r\n\r\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models. \r\n\r\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.\r\n\r\n[/quote]\r\n\r\nWhy does it have to be over-fitting or a leak which causes the significant increase? There is probably a little bit of overfitting because you can try different cutoff-levels for each new \"model\", but your argument lends little evidence towards how overfitting the public LB would lead to a 25% improvement on top of the 2nd place team. \r\n\r\nIn general a leak is caused by information that is provided in the dataset that you wouldn't have in real life. Can you explain why there could/would be a leak in this dataset? Just because there may be a feature that has significant predictive power (which others haven't discovered yet) doesn't mean that there is a leak. Let's say that they manually found that a specific station at a specific time is causing 50% of the faulty observations in the data, that would seem pretty reasonable to me, and your machine learning model probably cannot pick up specifically on that given the size of the dataset.", "votes": null }, { "id": "136798", "postDate": "09/25/2016 10:49:53", "content": "Qingchen:\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature.
\n\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays.
\n\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.
\n\nHope this data is created in such a manner,otherwise well just model postevent.
", "rawMarkdown": "Qingchen:\r\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature. \r\n\r\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays. \r\n\r\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.\r\n\r\nHope this data is created in such a manner,otherwise well just model postevent.", "votes": null }, { "id": "136800", "postDate": "09/25/2016 11:02:28", "content": "I tried using Id as a feature in xgbtree and it gives a slight decrease in local CV (for AUC). If there really is an egregious leak 47 days out, it will likely come out in the forums in a few weeks anyways.
", "rawMarkdown": "I tried using Id as a feature in xgbtree and it gives a slight decrease in local CV (for AUC). If there really is an egregious leak 47 days out, it will likely come out in the forums in a few weeks anyways.", "votes": null }, { "id": "136801", "postDate": "09/25/2016 11:03:52", "content": "[quote=Mohd Shadab Alam;136798]
\n\nQingchen:\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature.
\n\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays.
\n\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.
\n\nHope this data is created in such a manner,otherwise well just model postevent.
\n\n[/quote]
\n\nI would like to point out that I don't think that using the future to predict the past is leaking in this scenario (I agree with what @JeffH says here).
\n\nIn a real life model, you don't need to know something is failing the very moment it is produced; you can find out an hour later or a day later and since the product is still in the warehouse you can still remove it/check it. This is why I think the data is intentionally prepared so that the training and testing sets span the same time period.
\n\nIn a real-world scenario, it would have to be done something like this: For every production batch, you randomly check say 10% of products for failures (training set), and use that to predict the failures of the 90% (testing set). For this dataset, you can think of it like checking 50% of the products, and then trying to detect faults in the other 50% automatically (which would save a lot of time and money for Bosch.) Although I may be wrong, since I don't work at Bosch ;)
\n\nAlso, @Mike, I have searched far and wide for a leak without success. This dataset seems well-prepared.
", "rawMarkdown": "[quote=Mohd Shadab Alam;136798]\r\n\r\nQingchen:\r\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature. \r\n\r\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays. \r\n\r\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.\r\n\r\nHope this data is created in such a manner,otherwise well just model postevent.\r\n\r\n[/quote]\r\n\r\nI would like to point out that I don't think that using the future to predict the past is leaking in this scenario (I agree with what [@JeffH says here](https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089)). \r\n\r\nIn a real life model, you don't need to know something is failing the very moment it is produced; you can find out an hour later or a day later and since the product is still in the warehouse you can still remove it/check it. This is why I think the data is intentionally prepared so that the training and testing sets span the same time period.\r\n\r\nIn a real-world scenario, it would have to be done something like this: For every production batch, you randomly check say 10% of products for failures (training set), and use that to predict the failures of the 90% (testing set). For this dataset, you can think of it like checking 50% of the products, and then trying to detect faults in the other 50% automatically (which would save a lot of time and money for Bosch.) Although I may be wrong, since I don't work at Bosch ;)\r\n\r\nAlso, @Mike, I have searched far and wide for a leak without success. This dataset seems well-prepared.", "votes": null }, { "id": "136806", "postDate": "09/25/2016 12:33:52", "content": "@anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?
", "rawMarkdown": "anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?", "votes": null }, { "id": "136809", "postDate": "09/25/2016 12:45:48", "content": "[quote=salka;136806]
\n\n@anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?
\n\n[/quote]
\n\nWith 8GB of RAM you could definitely get 0.25+, but you wouldn't be able to run our model. Ours peaks at about 40GB RAM (loading data), and we are running it on a dual-xeon machine with 128GB RAM, although we can also run it fine on a 32GB machine with i7 (using swap), it just takes longer.
", "rawMarkdown": "[quote=salka;136806]\r\n\r\n@anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?\r\n\r\n[/quote]\r\n\r\nWith 8GB of RAM you could definitely get 0.25+, but you wouldn't be able to run our model. Ours peaks at about 40GB RAM (loading data), and we are running it on a dual-xeon machine with 128GB RAM, although we can also run it fine on a 32GB machine with i7 (using swap), it just takes longer.", "votes": null }, { "id": "136810", "postDate": "09/25/2016 12:45:49", "content": "@salka to compete 8 GB is not enough (at first, you will find trouble loading data, not to mention doing some modeling). I barely fit in 128 GB (though can be little bit optimized).
", "rawMarkdown": "salka to compete 8 GB is not enough (at first, you will find trouble loading data, not to mention doing some modeling). I barely fit in 128 GB (though can be little bit optimized).", "votes": null }, { "id": "136811", "postDate": "09/25/2016 12:48:24", "content": "Congratulations to the team Team ZFTurbo & anokas !!! Both CV and LB scores are 0.467. The solution must be so elegant.
", "rawMarkdown": "Congratulations to the team Team ZFTurbo & anokas !!! Both CV and LB scores are 0.467. The solution must be so elegant.", "votes": null }, { "id": "136814", "postDate": "09/25/2016 13:57:15", "content": "I sincerely admire them.
", "rawMarkdown": "I sincerely admire them.", "votes": null }, { "id": "136835", "postDate": "09/25/2016 18:10:05", "content": "Yup, impressive work! Cheers! (edit: And doubly impressive that you CV it perfectly!)
\n\nI hope I can figure it out myself... at least there's time ;)
\n\nI think if you're clever 16GB is going to be enough to get a silver medal. 8GB is probably doable if you're very clever and patient - but if you're already at that level you probably have a bigger machine already ;)
\n\nThere are a lot of features that can be dropped which helps. And if you remove the T's from the categorical file (something like sed s/T//g ) importing it becomes easier.
", "rawMarkdown": "Yup, impressive work! Cheers! (edit: And doubly impressive that you CV it perfectly!)\r\n\r\nI hope I can figure it out myself... at least there's time ;)\r\n\r\n---\r\n\r\nI think *if you're clever* 16GB is going to be enough to get a silver medal. 8GB is probably doable if you're *very* clever and patient - but if you're already at that level you probably have a bigger machine already ;)\r\n\r\nThere are a lot of features that can be dropped which helps. And if you remove the T's from the categorical file (something like sed s/T//g ) importing it becomes easier.", "votes": null }, { "id": "136841", "postDate": "09/25/2016 18:58:45", "content": "[quote=anokas;136792]
\n\n1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve.
\n\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.
\n\nGood luck! ;)
\n\n- Mikel.
\n\n[/quote]
\n\nGreat work! It gives us the goal to compete, making the competition more fun.
", "rawMarkdown": "[quote=anokas;136792]\r\n\r\n1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve. \r\n\r\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.\r\n\r\nGood luck! ;)\r\n\r\n\\- Mikel.\r\n\r\n[/quote]\r\n\r\n\r\nGreat work! It gives us the goal to compete, making the competition more fun.", "votes": null }, { "id": "136843", "postDate": "09/25/2016 19:02:50", "content": "I hope to catch them as well :)
", "rawMarkdown": "I hope to catch them as well :)", "votes": null }, { "id": "136848", "postDate": "09/25/2016 20:06:27", "content": "Thanks everyone for your kind words :D
\n\nraddar, you just won the last competition. Have some pity on the people that haven't won a prize in 3 months :P
", "rawMarkdown": "Thanks everyone for your kind words :D\r\n\r\nraddar, you just won the last competition. Have some pity on the people that haven't won a prize in 3 months :P", "votes": null }, { "id": "136874", "postDate": "09/26/2016 03:08:00", "content": "Gilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.
", "rawMarkdown": "Gilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.", "votes": null }, { "id": "136879", "postDate": "09/26/2016 03:56:11", "content": "[quote=Tam T. Nguyen;136874]
\n\nGilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.
\n\n[/quote]
\n\nUh oh... is it an ID based feature?
", "rawMarkdown": "[quote=Tam T. Nguyen;136874]\r\n\r\nGilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.\r\n\r\n[/quote]\r\n\r\nUh oh... is it an ID based feature?", "votes": null }, { "id": "136889", "postDate": "09/26/2016 07:09:35", "content": "[quote=anokas;136792]\nI am extremely confident we have not modelled a leak accidently\n[/quote]
\n\n[quote=Tam T. Nguyen;136874]\nI've just added a nonsense feature and my LB score improves around 0.1\n[/quote]
\n\nOk... This is going to be very interesting, maybe we are just about to see near ~0.6 in the LB very very soon :/
", "rawMarkdown": "[quote=anokas;136792]\r\nI am extremely confident we have not modelled a leak accidently\r\n[/quote]\r\n\r\n[quote=Tam T. Nguyen;136874]\r\nI've just added a nonsense feature and my LB score improves around 0.1\r\n[/quote]\r\n\r\nOk... This is going to be very interesting, maybe we are just about to see near ~0.6 in the LB very very soon :/", "votes": null }, { "id": "136892", "postDate": "09/26/2016 07:21:17", "content": "It could be that they just have different definition for leak ;)
\n\nIMHO using the same time period for train and test is generally a mistake.
\n\n"Prediction is very difficult, especially if it's about the future." Niels Bohr
\n\nAnyway I was 100% percent sure that there are a few leaks/goldenfeatures in such large dataset. At least we have enough time to explore them. I just hope that kind of last week madness that happened in TalkingData won't happen again.
", "rawMarkdown": "It could be that they just have different definition for leak ;)\r\n\r\nIMHO using the same time period for train and test is generally a mistake. \r\n\r\n\r\n*\"Prediction is very difficult, especially if it's about the future.\"* **Niels Bohr**\r\n\r\nAnyway I was 100% percent sure that there are a few leaks/goldenfeatures in such large dataset. At least we have enough time to explore them. I just hope that kind of last week madness that happened in TalkingData won't happen again.", "votes": null }, { "id": "136893", "postDate": "09/26/2016 07:33:07", "content": "The way I see it, what people have found may not be a leak but rather a data property.
\n\n@Tam It might be useless, but it might not; maybe that nonsense itself would be useful to Bosch. Keep on digging;)
", "rawMarkdown": "The way I see it, what people have found may not be a leak but rather a data property.\r\n\r\n@Tam It might be useless, but it might not; maybe that nonsense itself would be useful to Bosch. Keep on digging;)", "votes": null }, { "id": "136894", "postDate": "09/26/2016 07:48:39", "content": "[quote=beluga;136892]
\n\nIMHO using the same time period for train and test is generally a mistake.
\n\n[/quote]
\n\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?
", "rawMarkdown": "[quote=beluga;136892]\r\n\r\nIMHO using the same time period for train and test is generally a mistake. \r\n\r\n[/quote]\r\n\r\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?", "votes": null }, { "id": "136895", "postDate": "09/26/2016 08:04:38", "content": "[quote=Tony Schroeder;136894]\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?\n[/quote]
\n\nIt depends. I have seen competitions where the train/test was split by time. (Expedia2, WestNile, ClickPredictFix, etc.)
\n\nOf course I am not saying that splitting by time would solve all the problems. Each listed competitions had their special leakage (distance matching, LB feedback to identify outbreaks, LB feedback to scale predictions).
\n\nThough in real-world I always split train/test based on time.
", "rawMarkdown": "[quote=Tony Schroeder;136894]\r\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?\r\n[/quote]\r\n\r\nIt depends. I have seen competitions where the train/test was split by time. ([Expedia2][1], [WestNile][2], [ClickPredictFix][3], etc.) \r\n\r\nOf course I am not saying that splitting by time would solve all the problems. Each listed competitions had their special leakage (distance matching, LB feedback to identify outbreaks, LB feedback to scale predictions).\r\n\r\nThough in real-world I always split train/test based on time.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/expedia-hotel-recommendations\r\n [2]: https://www.kaggle.com/c/predict-west-nile-virus\r\n [3]: https://www.kaggle.com/c/see-click-predict-fix", "votes": null }, { "id": "136897", "postDate": "09/26/2016 08:25:24", "content": "So would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.
", "rawMarkdown": "So would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.", "votes": null }, { "id": "136899", "postDate": "09/26/2016 08:53:37", "content": "[quote=LEE;136790]
\n\nI think they found something .
\n\n[/quote]
\n\nSomething like the dependence of failure rate on date?
", "rawMarkdown": "[quote=LEE;136790]\r\n\r\nI think they found something . \r\n\r\n[/quote]\r\n\r\nSomething like the dependence of [failure rate][1] on date?\r\n\r\n\r\n [1]: https://www.kaggle.com/sionek/bosch-production-line-performance/failure-rate", "votes": null }, { "id": "136901", "postDate": "09/26/2016 09:35:12", "content": "[quote=Tony Schroeder;136897]
\n\nSo would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.
\n\n[/quote]
\n\nIn competitions where the organiser wants to predict the future, this is what usually happens. However, I don't think this is the case with Bosch, I think instead they want us to predict for other products in the same time period (since they have prepared the date file, they probably want us to use it)
\n\nIn this dataset, we have a time that each measurement was taken, and the training and testing sets are randomly split in the same time period.
", "rawMarkdown": "[quote=Tony Schroeder;136897]\r\n\r\nSo would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.\r\n\r\n[/quote]\r\n\r\nIn competitions where the organiser wants to predict the future, this is what usually happens. However, I don't think this is the case with Bosch, I think instead they want us to predict for other products in the same time period (since they have prepared the date file, they probably want us to use it)\r\n\r\nIn this dataset, we have a time that each measurement was taken, and the training and testing sets are randomly split in the same time period.", "votes": null }, { "id": "136922", "postDate": "09/26/2016 14:36:48", "content": "@anokas Alright, I figured so. Thanks for clarifying.
", "rawMarkdown": "anokas Alright, I figured so. Thanks for clarifying.", "votes": null }, { "id": "136925", "postDate": "09/26/2016 14:58:13", "content": "Hi folks, so I tested using an ID feature, and it looks like ID number / np.arange(total_number_of_IDs) is the top feature in terms of the number of splits (get_fscore from XGBoost). But perhaps the model I am using is not that great, such that ID is relevant.
", "rawMarkdown": "Hi folks, so I tested using an ID feature, and it looks like ID number / np.arange(total_number_of_IDs) is the top feature in terms of the number of splits (get_fscore from XGBoost). But perhaps the model I am using is not that great, such that ID is relevant.", "votes": null }, { "id": "136957", "postDate": "09/26/2016 22:00:23", "content": "Whatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the "learning" stage.
", "rawMarkdown": "Whatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the \"learning\" stage.", "votes": null }, { "id": "136958", "postDate": "09/26/2016 22:11:19", "content": "[quote=Bob, Lord of Cats;136957]
\n\nWhatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the "learning" stage.
\n\n[/quote]
\n\nWe will definitely do this if we end up in a good position. I also plan to release code (but I can't promise that)
", "rawMarkdown": "[quote=Bob, Lord of Cats;136957]\r\n\r\nWhatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the \"learning\" stage.\r\n\r\n[/quote]\r\n\r\nWe will definitely do this if we end up in a good position. I also _plan_ to release code (but I can't promise that)", "votes": null }, { "id": "137031", "postDate": "09/27/2016 17:03:28", "content": "First, I'm thrilled to finally participate in a manufacturing themed contest on Kaggle. For me, feature anonymization makes this competition both exciting and frustrating. Compared to other datasets, what Bosch has provided represent processes that I think are more physical with an artificial layer of abstraction. Even without removing the information the names contained (even the SI units!), it's a challenge to separate the superimposed process and raw material variability. I'm curious to know if some variables have analogues across stations. Despite the mystification, I'm certain predictable mechanisms are lurking in there. Someone could conceivably extract an effective first principle or dimensionless ratio without catastrophic overfitting.
\n\nAs a ridiculously nonlinear example, dependent on the chunk of steel, the rotational speed of the drillbit, the axial velocity, and the flow rate of coolant at the bit, more energy will be imparted either as heat or mechanically. If this particular coolant is chilled by a unit that has run nonstop for a week maybe it lost efficiency and returned an out of spec temperature, imparting a defect.
\n\nRegardless, I'm very interested to learn from the overfitting/predicting story as it plays out.
", "rawMarkdown": "First, I'm thrilled to finally participate in a manufacturing themed contest on Kaggle. For me, feature anonymization makes this competition both exciting and frustrating. Compared to other datasets, what Bosch has provided represent processes that I think are more physical with an artificial layer of abstraction. Even without removing the information the names contained (even the SI units!), it's a challenge to separate the superimposed process and raw material variability. I'm curious to know if some variables have analogues across stations. Despite the mystification, I'm certain predictable mechanisms are lurking in there. Someone could conceivably extract an effective first principle or dimensionless ratio without catastrophic overfitting.\r\n\r\nAs a ridiculously nonlinear example, dependent on the chunk of steel, the rotational speed of the drillbit, the axial velocity, and the flow rate of coolant at the bit, more energy will be imparted either as heat or mechanically. If this particular coolant is chilled by a unit that has run nonstop for a week maybe it lost efficiency and returned an out of spec temperature, imparting a defect.\r\n\r\nRegardless, I'm very interested to learn from the overfitting/predicting story as it plays out.", "votes": null }, { "id": "137094", "postDate": "09/28/2016 06:29:32", "content": "[quote=Grzegorz Sionkowski;136899]
\n\n[quote=LEE;136790]
\n\nI think they found something .
\n\n[/quote]
\n\nSomething like the dependence of failure rate on date?
\n\n[/quote]
\n\nOf course, I do not know. But Date-file contains a lot of information than we thought.
", "rawMarkdown": "[quote=Grzegorz Sionkowski;136899]\r\n\r\n[quote=LEE;136790]\r\n\r\nI think they found something . \r\n\r\n[/quote]\r\n\r\nSomething like the dependence of [failure rate][1] on date?\r\n\r\n\r\n [1]: https://www.kaggle.com/sionek/bosch-production-line-performance/failure-rate\r\n\r\n[/quote]\r\n\r\nOf course, I do not know. But Date-file contains a lot of information than we thought.", "votes": null }, { "id": "137142", "postDate": "09/28/2016 16:26:20", "content": "[quote=<<>>;137094]
\n\n[quote=Grzegorz Sionkowski;136899]
\n\n[quote=LEE;136790]
\n\nI think they found something .
\n\n[/quote]
\n\nSomething like the dependence of failure rate on date?
\n\n[/quote]
\n\nOf course, I do not know. But Date-file contains a lot of information than we thought.
\n\n[/quote]
\n\nWe don't know if the dataset here is complete. However, my intuition tells me that looking beyond individual parts and considering the time series of equipment conditions they represent can explain equipment malfunctions or changes. For example, you can have a jam or break a tool due to repeated out of control conditions. Of course, the cost to sort by time can be high.
", "rawMarkdown": "[quote=<<I think they found something .
", "votes": null, "replies": [] }, { "id": 136791, "author_name": "tarobxl", "author_url": "", "post_date": "09/25/2016 09:52:54", "content": "Indeed I think they found a leak but it's hard to say "over-fitting" even their model might not work well with other test dataset. In the end as long as they don't violate the rule they deserve the top ranking.
\n\nSo curious about their solution as I have never been able to get that score locally in my CV even I tried to overfit the train dataset :-)
", "votes": null, "replies": [] }, { "id": 136792, "author_name": "anokas", "author_url": "", "post_date": "09/25/2016 10:05:15", "content": "1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve.
\n\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.
\n\nGood luck! ;)
\n\n- Mikel.
", "votes": null, "replies": [] }, { "id": 136793, "author_name": "outliar", "author_url": "", "post_date": "09/25/2016 10:10:48", "content": "good luck :)
", "votes": null, "replies": [] }, { "id": 136795, "author_name": "qwang88", "author_url": "", "post_date": "09/25/2016 10:12:04", "content": "[quote=Mohd Shadab Alam;136789]
\n\nThere are two possibilities:
\n\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models.
\n\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.
\n\n[/quote]
\n\nWhy does it have to be over-fitting or a leak which causes the significant increase? There is probably a little bit of overfitting because you can try different cutoff-levels for each new "model", but your argument lends little evidence towards how overfitting the public LB would lead to a 25% improvement on top of the 2nd place team.
\n\nIn general a leak is caused by information that is provided in the dataset that you wouldn't have in real life. Can you explain why there could/would be a leak in this dataset? Just because there may be a feature that has significant predictive power (which others haven't discovered yet) doesn't mean that there is a leak. Let's say that they manually found that a specific station at a specific time is causing 50% of the faulty observations in the data, that would seem pretty reasonable to me, and your machine learning model probably cannot pick up specifically on that given the size of the dataset.
", "votes": null, "replies": [] }, { "id": 136798, "author_name": "outliar", "author_url": "", "post_date": "09/25/2016 10:49:53", "content": "Qingchen:\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature.
\n\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays.
\n\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.
\n\nHope this data is created in such a manner,otherwise well just model postevent.
", "votes": null, "replies": [] }, { "id": 136800, "author_name": "mikeskim", "author_url": "", "post_date": "09/25/2016 11:02:28", "content": "I tried using Id as a feature in xgbtree and it gives a slight decrease in local CV (for AUC). If there really is an egregious leak 47 days out, it will likely come out in the forums in a few weeks anyways.
", "votes": null, "replies": [] }, { "id": 136801, "author_name": "anokas", "author_url": "", "post_date": "09/25/2016 11:03:52", "content": "[quote=Mohd Shadab Alam;136798]
\n\nQingchen:\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature.
\n\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays.
\n\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.
\n\nHope this data is created in such a manner,otherwise well just model postevent.
\n\n[/quote]
\n\nI would like to point out that I don't think that using the future to predict the past is leaking in this scenario (I agree with what @JeffH says here).
\n\nIn a real life model, you don't need to know something is failing the very moment it is produced; you can find out an hour later or a day later and since the product is still in the warehouse you can still remove it/check it. This is why I think the data is intentionally prepared so that the training and testing sets span the same time period.
\n\nIn a real-world scenario, it would have to be done something like this: For every production batch, you randomly check say 10% of products for failures (training set), and use that to predict the failures of the 90% (testing set). For this dataset, you can think of it like checking 50% of the products, and then trying to detect faults in the other 50% automatically (which would save a lot of time and money for Bosch.) Although I may be wrong, since I don't work at Bosch ;)
\n\nAlso, @Mike, I have searched far and wide for a leak without success. This dataset seems well-prepared.
", "votes": null, "replies": [] }, { "id": 136806, "author_name": "salkaa", "author_url": "", "post_date": "09/25/2016 12:33:52", "content": "@anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?
", "votes": null, "replies": [] }, { "id": 136809, "author_name": "anokas", "author_url": "", "post_date": "09/25/2016 12:45:48", "content": "[quote=salka;136806]
\n\n@anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?
\n\n[/quote]
\n\nWith 8GB of RAM you could definitely get 0.25+, but you wouldn't be able to run our model. Ours peaks at about 40GB RAM (loading data), and we are running it on a dual-xeon machine with 128GB RAM, although we can also run it fine on a 32GB machine with i7 (using swap), it just takes longer.
", "votes": null, "replies": [] }, { "id": 136810, "author_name": "raddar", "author_url": "", "post_date": "09/25/2016 12:45:49", "content": "@salka to compete 8 GB is not enough (at first, you will find trouble loading data, not to mention doing some modeling). I barely fit in 128 GB (though can be little bit optimized).
", "votes": null, "replies": [] }, { "id": 136811, "author_name": "tarobxl", "author_url": "", "post_date": "09/25/2016 12:48:24", "content": "Congratulations to the team Team ZFTurbo & anokas !!! Both CV and LB scores are 0.467. The solution must be so elegant.
", "votes": null, "replies": [] }, { "id": 136814, "author_name": "niks2000", "author_url": "", "post_date": "09/25/2016 13:57:15", "content": "I sincerely admire them.
", "votes": null, "replies": [] }, { "id": 136835, "author_name": "happycube", "author_url": "", "post_date": "09/25/2016 18:10:05", "content": "Yup, impressive work! Cheers! (edit: And doubly impressive that you CV it perfectly!)
\n\nI hope I can figure it out myself... at least there's time ;)
\n\nI think if you're clever 16GB is going to be enough to get a silver medal. 8GB is probably doable if you're very clever and patient - but if you're already at that level you probably have a bigger machine already ;)
\n\nThere are a lot of features that can be dropped which helps. And if you remove the T's from the categorical file (something like sed s/T//g ) importing it becomes easier.
", "votes": null, "replies": [] }, { "id": 136841, "author_name": "amazingbob", "author_url": "", "post_date": "09/25/2016 18:58:45", "content": "[quote=anokas;136792]
\n\n1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve.
\n\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.
\n\nGood luck! ;)
\n\n- Mikel.
\n\n[/quote]
\n\nGreat work! It gives us the goal to compete, making the competition more fun.
", "votes": null, "replies": [] }, { "id": 136843, "author_name": "raddar", "author_url": "", "post_date": "09/25/2016 19:02:50", "content": "I hope to catch them as well :)
", "votes": null, "replies": [] }, { "id": 136848, "author_name": "anokas", "author_url": "", "post_date": "09/25/2016 20:06:27", "content": "Thanks everyone for your kind words :D
\n\nraddar, you just won the last competition. Have some pity on the people that haven't won a prize in 3 months :P
", "votes": null, "replies": [] }, { "id": 136874, "author_name": "nthanhtam", "author_url": "", "post_date": "09/26/2016 03:08:00", "content": "Gilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.
", "votes": null, "replies": [] }, { "id": 136879, "author_name": "garcia274", "author_url": "", "post_date": "09/26/2016 03:56:11", "content": "[quote=Tam T. Nguyen;136874]
\n\nGilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.
\n\n[/quote]
\n\nUh oh... is it an ID based feature?
", "votes": null, "replies": [] }, { "id": 136889, "author_name": "carloshuertas", "author_url": "", "post_date": "09/26/2016 07:09:35", "content": "[quote=anokas;136792]\nI am extremely confident we have not modelled a leak accidently\n[/quote]
\n\n[quote=Tam T. Nguyen;136874]\nI've just added a nonsense feature and my LB score improves around 0.1\n[/quote]
\n\nOk... This is going to be very interesting, maybe we are just about to see near ~0.6 in the LB very very soon :/
", "votes": null, "replies": [] }, { "id": 136892, "author_name": "gaborfodor", "author_url": "", "post_date": "09/26/2016 07:21:17", "content": "It could be that they just have different definition for leak ;)
\n\nIMHO using the same time period for train and test is generally a mistake.
\n\n"Prediction is very difficult, especially if it's about the future." Niels Bohr
\n\nAnyway I was 100% percent sure that there are a few leaks/goldenfeatures in such large dataset. At least we have enough time to explore them. I just hope that kind of last week madness that happened in TalkingData won't happen again.
", "votes": null, "replies": [] }, { "id": 136893, "author_name": "raddar", "author_url": "", "post_date": "09/26/2016 07:33:07", "content": "The way I see it, what people have found may not be a leak but rather a data property.
\n\n@Tam It might be useless, but it might not; maybe that nonsense itself would be useful to Bosch. Keep on digging;)
", "votes": null, "replies": [] }, { "id": 136894, "author_name": "fredtony", "author_url": "", "post_date": "09/26/2016 07:48:39", "content": "[quote=beluga;136892]
\n\nIMHO using the same time period for train and test is generally a mistake.
\n\n[/quote]
\n\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?
", "votes": null, "replies": [] }, { "id": 136895, "author_name": "gaborfodor", "author_url": "", "post_date": "09/26/2016 08:04:38", "content": "[quote=Tony Schroeder;136894]\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?\n[/quote]
\n\nIt depends. I have seen competitions where the train/test was split by time. (Expedia2, WestNile, ClickPredictFix, etc.)
\n\nOf course I am not saying that splitting by time would solve all the problems. Each listed competitions had their special leakage (distance matching, LB feedback to identify outbreaks, LB feedback to scale predictions).
\n\nThough in real-world I always split train/test based on time.
", "votes": null, "replies": [] }, { "id": 136897, "author_name": "fredtony", "author_url": "", "post_date": "09/26/2016 08:25:24", "content": "So would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.
", "votes": null, "replies": [] }, { "id": 136899, "author_name": "sionek", "author_url": "", "post_date": "09/26/2016 08:53:37", "content": "[quote=LEE;136790]
\n\nI think they found something .
\n\n[/quote]
\n\nSomething like the dependence of failure rate on date?
", "votes": null, "replies": [] }, { "id": 136901, "author_name": "anokas", "author_url": "", "post_date": "09/26/2016 09:35:12", "content": "[quote=Tony Schroeder;136897]
\n\nSo would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.
\n\n[/quote]
\n\nIn competitions where the organiser wants to predict the future, this is what usually happens. However, I don't think this is the case with Bosch, I think instead they want us to predict for other products in the same time period (since they have prepared the date file, they probably want us to use it)
\n\nIn this dataset, we have a time that each measurement was taken, and the training and testing sets are randomly split in the same time period.
", "votes": null, "replies": [] }, { "id": 136922, "author_name": "fredtony", "author_url": "", "post_date": "09/26/2016 14:36:48", "content": "@anokas Alright, I figured so. Thanks for clarifying.
", "votes": null, "replies": [] }, { "id": 136925, "author_name": "garcia274", "author_url": "", "post_date": "09/26/2016 14:58:13", "content": "Hi folks, so I tested using an ID feature, and it looks like ID number / np.arange(total_number_of_IDs) is the top feature in terms of the number of splits (get_fscore from XGBoost). But perhaps the model I am using is not that great, such that ID is relevant.
", "votes": null, "replies": [] }, { "id": 136957, "author_name": "rdslater", "author_url": "", "post_date": "09/26/2016 22:00:23", "content": "Whatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the "learning" stage.
", "votes": null, "replies": [] }, { "id": 136958, "author_name": "anokas", "author_url": "", "post_date": "09/26/2016 22:11:19", "content": "[quote=Bob, Lord of Cats;136957]
\n\nWhatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the "learning" stage.
\n\n[/quote]
\n\nWe will definitely do this if we end up in a good position. I also plan to release code (but I can't promise that)
", "votes": null, "replies": [] }, { "id": 137031, "author_name": "clockworkworld", "author_url": "", "post_date": "09/27/2016 17:03:28", "content": "First, I'm thrilled to finally participate in a manufacturing themed contest on Kaggle. For me, feature anonymization makes this competition both exciting and frustrating. Compared to other datasets, what Bosch has provided represent processes that I think are more physical with an artificial layer of abstraction. Even without removing the information the names contained (even the SI units!), it's a challenge to separate the superimposed process and raw material variability. I'm curious to know if some variables have analogues across stations. Despite the mystification, I'm certain predictable mechanisms are lurking in there. Someone could conceivably extract an effective first principle or dimensionless ratio without catastrophic overfitting.
\n\nAs a ridiculously nonlinear example, dependent on the chunk of steel, the rotational speed of the drillbit, the axial velocity, and the flow rate of coolant at the bit, more energy will be imparted either as heat or mechanically. If this particular coolant is chilled by a unit that has run nonstop for a week maybe it lost efficiency and returned an out of spec temperature, imparting a defect.
\n\nRegardless, I'm very interested to learn from the overfitting/predicting story as it plays out.
", "votes": null, "replies": [] }, { "id": 137094, "author_name": "niks2000", "author_url": "", "post_date": "09/28/2016 06:29:32", "content": "[quote=Grzegorz Sionkowski;136899]
\n\n[quote=LEE;136790]
\n\nI think they found something .
\n\n[/quote]
\n\nSomething like the dependence of failure rate on date?
\n\n[/quote]
\n\nOf course, I do not know. But Date-file contains a lot of information than we thought.
", "votes": null, "replies": [] }, { "id": 137142, "author_name": "clockworkworld", "author_url": "", "post_date": "09/28/2016 16:26:20", "content": "[quote=<<>>;137094]
\n\n[quote=Grzegorz Sionkowski;136899]
\n\n[quote=LEE;136790]
\n\nI think they found something .
\n\n[/quote]
\n\nSomething like the dependence of failure rate on date?
\n\n[/quote]
\n\nOf course, I do not know. But Date-file contains a lot of information than we thought.
\n\n[/quote]
\n\nWe don't know if the dataset here is complete. However, my intuition tells me that looking beyond individual parts and considering the time series of equipment conditions they represent can explain equipment malfunctions or changes. For example, you can have a jam or break a tool due to repeated out of control conditions. Of course, the cost to sort by time can be high.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "136789": "There is a difference of 0.1 between 1st and 2nd team. They are also leading with the maximum number of submissions. \r\n\r\nThere are two possibilities:\r\n\r\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models. \r\n\r\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.", "136790": "I think they found something .", "136791": "Indeed I think they found a leak but it's hard to say \"over-fitting\" even their model might not work well with other test dataset. In the end as long as they don't violate the rule they deserve the top ranking.\r\n\r\nSo curious about their solution as I have never been able to get that score locally in my CV even I tried to overfit the train dataset :-)", "136792": "1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve. \r\n\r\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.\r\n\r\nGood luck! ;)\r\n\r\n\\- Mikel.", "136793": "good luck :)", "136795": "[quote=Mohd Shadab Alam;136789] \r\n\r\nThere are two possibilities:\r\n\r\n1.Chances of over-fitting is highest in their case, as they are the ones with the maximum number of submissions /models. \r\n\r\n2.It is also probable that they have discovered some rule because of leak. Now, this rule can be specific to public the test set (as in Santander : Guys with age <24 are always happy), which will again confirm 1.\r\n\r\n[/quote]\r\n\r\nWhy does it have to be over-fitting or a leak which causes the significant increase? There is probably a little bit of overfitting because you can try different cutoff-levels for each new \"model\", but your argument lends little evidence towards how overfitting the public LB would lead to a 25% improvement on top of the 2nd place team. \r\n\r\nIn general a leak is caused by information that is provided in the dataset that you wouldn't have in real life. Can you explain why there could/would be a leak in this dataset? Just because there may be a feature that has significant predictive power (which others haven't discovered yet) doesn't mean that there is a leak. Let's say that they manually found that a specific station at a specific time is causing 50% of the faulty observations in the data, that would seem pretty reasonable to me, and your machine learning model probably cannot pick up specifically on that given the size of the dataset.", "136798": "Qingchen:\r\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature. \r\n\r\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays. \r\n\r\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.\r\n\r\nHope this data is created in such a manner,otherwise well just model postevent.", "136800": "I tried using Id as a feature in xgbtree and it gives a slight decrease in local CV (for AUC). If there really is an egregious leak 47 days out, it will likely come out in the forums in a few weeks anyways.", "136801": "[quote=Mohd Shadab Alam;136798]\r\n\r\nQingchen:\r\nMikel has made it very clear that its not a leak, i agree with you that there must be some hidden feature. \r\n\r\nWell , any model based on time wont be useful, until timestamp is measured from the instance you run a machine because it wont make sense to say that we are getting faulty readings on sundays. \r\n\r\nIn ideal scenario, train and test readings must be taken at different times although timestamp may look similar, to make the model useful. As it makes sense to say that machine at station 32 is producing faulty reading after running fot 100hours.\r\n\r\nHope this data is created in such a manner,otherwise well just model postevent.\r\n\r\n[/quote]\r\n\r\nI would like to point out that I don't think that using the future to predict the past is leaking in this scenario (I agree with what [@JeffH says here](https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089)). \r\n\r\nIn a real life model, you don't need to know something is failing the very moment it is produced; you can find out an hour later or a day later and since the product is still in the warehouse you can still remove it/check it. This is why I think the data is intentionally prepared so that the training and testing sets span the same time period.\r\n\r\nIn a real-world scenario, it would have to be done something like this: For every production batch, you randomly check say 10% of products for failures (training set), and use that to predict the failures of the 90% (testing set). For this dataset, you can think of it like checking 50% of the products, and then trying to detect faults in the other 50% automatically (which would save a lot of time and money for Bosch.) Although I may be wrong, since I don't work at Bosch ;)\r\n\r\nAlso, @Mike, I have searched far and wide for a leak without success. This dataset seems well-prepared.", "136806": "anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?", "136809": "[quote=salka;136806]\r\n\r\n@anokas, i haven't started this competition yet, can you tell me if 8GB of ram are enough to make some good results ? and if its not secret what machine you are using ?\r\n\r\n[/quote]\r\n\r\nWith 8GB of RAM you could definitely get 0.25+, but you wouldn't be able to run our model. Ours peaks at about 40GB RAM (loading data), and we are running it on a dual-xeon machine with 128GB RAM, although we can also run it fine on a 32GB machine with i7 (using swap), it just takes longer.", "136810": "salka to compete 8 GB is not enough (at first, you will find trouble loading data, not to mention doing some modeling). I barely fit in 128 GB (though can be little bit optimized).", "136811": "Congratulations to the team Team ZFTurbo & anokas !!! Both CV and LB scores are 0.467. The solution must be so elegant.", "136814": "I sincerely admire them.", "136835": "Yup, impressive work! Cheers! (edit: And doubly impressive that you CV it perfectly!)\r\n\r\nI hope I can figure it out myself... at least there's time ;)\r\n\r\n---\r\n\r\nI think *if you're clever* 16GB is going to be enough to get a silver medal. 8GB is probably doable if you're *very* clever and patient - but if you're already at that level you probably have a bigger machine already ;)\r\n\r\nThere are a lot of features that can be dropped which helps. And if you remove the T's from the categorical file (something like sed s/T//g ) importing it becomes easier.", "136841": "[quote=anokas;136792]\r\n\r\n1) I don't think we've overfit (although it's possible :P) considering we improved our score 0.32 -> 0.47 without submitting once during that process, only using CV to improve. \r\n\r\n2) No santander-like rules, no intentional leak (and I am extremely confident we have not modelled a leak accidently). Just a lot of hard work :) After these last few competitions, I think Kaggle is too quick to say good score = leak. Anyway, I don't consider what happened in Santander a leak, since it was modelling a real aspect of the data, not something that was only there due to bad preparation.\r\n\r\nGood luck! ;)\r\n\r\n\\- Mikel.\r\n\r\n[/quote]\r\n\r\n\r\nGreat work! It gives us the goal to compete, making the competition more fun.", "136843": "I hope to catch them as well :)", "136848": "Thanks everyone for your kind words :D\r\n\r\nraddar, you just won the last competition. Have some pity on the people that haven't won a prize in 3 months :P", "136874": "Gilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.", "136879": "[quote=Tam T. Nguyen;136874]\r\n\r\nGilberto and I have teamed up twice and we did not win any recent competitions due to leak. I've just added a nonsense feature and my LB score improves around 0.1. I don't think this kind of feature would be useful in practice. I think it is another leak hunt competition.\r\n\r\n[/quote]\r\n\r\nUh oh... is it an ID based feature?", "136889": "[quote=anokas;136792]\r\nI am extremely confident we have not modelled a leak accidently\r\n[/quote]\r\n\r\n[quote=Tam T. Nguyen;136874]\r\nI've just added a nonsense feature and my LB score improves around 0.1\r\n[/quote]\r\n\r\nOk... This is going to be very interesting, maybe we are just about to see near ~0.6 in the LB very very soon :/", "136892": "It could be that they just have different definition for leak ;)\r\n\r\nIMHO using the same time period for train and test is generally a mistake. \r\n\r\n\r\n*\"Prediction is very difficult, especially if it's about the future.\"* **Niels Bohr**\r\n\r\nAnyway I was 100% percent sure that there are a few leaks/goldenfeatures in such large dataset. At least we have enough time to explore them. I just hope that kind of last week madness that happened in TalkingData won't happen again.", "136893": "The way I see it, what people have found may not be a leak but rather a data property.\r\n\r\n@Tam It might be useless, but it might not; maybe that nonsense itself would be useful to Bosch. Keep on digging;)", "136894": "[quote=beluga;136892]\r\n\r\nIMHO using the same time period for train and test is generally a mistake. \r\n\r\n[/quote]\r\n\r\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?", "136895": "[quote=Tony Schroeder;136894]\r\nIsn't it a basic premise of the train/test sets in these competitions and data sets in general that they are IID and not from different data sets?\r\n[/quote]\r\n\r\nIt depends. I have seen competitions where the train/test was split by time. ([Expedia2][1], [WestNile][2], [ClickPredictFix][3], etc.) \r\n\r\nOf course I am not saying that splitting by time would solve all the problems. Each listed competitions had their special leakage (distance matching, LB feedback to identify outbreaks, LB feedback to scale predictions).\r\n\r\nThough in real-world I always split train/test based on time.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/expedia-hotel-recommendations\r\n [2]: https://www.kaggle.com/c/predict-west-nile-virus\r\n [3]: https://www.kaggle.com/c/see-click-predict-fix", "136897": "So would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.", "136899": "[quote=LEE;136790]\r\n\r\nI think they found something . \r\n\r\n[/quote]\r\n\r\nSomething like the dependence of [failure rate][1] on date?\r\n\r\n\r\n [1]: https://www.kaggle.com/sionek/bosch-production-line-performance/failure-rate", "136901": "[quote=Tony Schroeder;136897]\r\n\r\nSo would that imply that the train set takes place before some specific time and test set after that time? Are they usually shuffled within those blocks or do you think they are ordered by time in each set? I'm just asking because I'm trying to wrap my head around this and haven't really analyzed the data from this perspective if so.\r\n\r\n[/quote]\r\n\r\nIn competitions where the organiser wants to predict the future, this is what usually happens. However, I don't think this is the case with Bosch, I think instead they want us to predict for other products in the same time period (since they have prepared the date file, they probably want us to use it)\r\n\r\nIn this dataset, we have a time that each measurement was taken, and the training and testing sets are randomly split in the same time period.", "136922": "anokas Alright, I figured so. Thanks for clarifying.", "136925": "Hi folks, so I tested using an ID feature, and it looks like ID number / np.arange(total_number_of_IDs) is the top feature in terms of the number of splits (get_fscore from XGBoost). But perhaps the model I am using is not that great, such that ID is relevant.", "136957": "Whatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the \"learning\" stage.", "136958": "[quote=Bob, Lord of Cats;136957]\r\n\r\nWhatever it is, please make sure you do a nice post at the end. Some of the more recent competitions have been less than forthcoming on what they did. While competition helps--looking at successful teams helps much too--especially for those of us in the \"learning\" stage.\r\n\r\n[/quote]\r\n\r\nWe will definitely do this if we end up in a good position. I also _plan_ to release code (but I can't promise that)", "137031": "First, I'm thrilled to finally participate in a manufacturing themed contest on Kaggle. For me, feature anonymization makes this competition both exciting and frustrating. Compared to other datasets, what Bosch has provided represent processes that I think are more physical with an artificial layer of abstraction. Even without removing the information the names contained (even the SI units!), it's a challenge to separate the superimposed process and raw material variability. I'm curious to know if some variables have analogues across stations. Despite the mystification, I'm certain predictable mechanisms are lurking in there. Someone could conceivably extract an effective first principle or dimensionless ratio without catastrophic overfitting.\r\n\r\nAs a ridiculously nonlinear example, dependent on the chunk of steel, the rotational speed of the drillbit, the axial velocity, and the flow rate of coolant at the bit, more energy will be imparted either as heat or mechanically. If this particular coolant is chilled by a unit that has run nonstop for a week maybe it lost efficiency and returned an out of spec temperature, imparting a defect.\r\n\r\nRegardless, I'm very interested to learn from the overfitting/predicting story as it plays out.", "137094": "[quote=Grzegorz Sionkowski;136899]\r\n\r\n[quote=LEE;136790]\r\n\r\nI think they found something . \r\n\r\n[/quote]\r\n\r\nSomething like the dependence of [failure rate][1] on date?\r\n\r\n\r\n [1]: https://www.kaggle.com/sionek/bosch-production-line-performance/failure-rate\r\n\r\n[/quote]\r\n\r\nOf course, I do not know. But Date-file contains a lot of information than we thought.", "137142": "[quote=<<