{
  "id": 23354,
  "title": "So, is there a leak?",
  "url": "/competitions/bosch-production-line-performance/discussion/23354",
  "author_name": "",
  "post_date": "2016-09-04T15:13:09.050Z",
  "votes": 2,
  "comment_count": 15,
  "views": 2702,
  "content": "<p>Before I start spending hours munging for this competition, has anyone found any kind of leak? I really don't want to waste my time doing data science for competitions that don't require data science.  </p>",
  "messages": [
    {
      "id": "133952",
      "postDate": "09/04/2016 15:13:09",
      "content": "<p>Before I start spending hours munging for this competition, has anyone found any kind of leak? I really don't want to waste my time doing data science for competitions that don't require data science.  </p>",
      "rawMarkdown": "Before I start spending hours munging for this competition, has anyone found any kind of leak? I really don't want to waste my time doing data science for competitions that don't require data science.",
      "votes": null
    },
    {
      "id": "133979",
      "postDate": "09/04/2016 16:23:35",
      "content": "<p>Do you have any hint or are you just asking?</p>",
      "rawMarkdown": "Do you have any hint or are you just asking?",
      "votes": null
    },
    {
      "id": "133996",
      "postDate": "09/04/2016 17:08:05",
      "content": "<p>No hint. I've looked at data and thought of some feature engineering but then I got Leak Dread</p>",
      "rawMarkdown": "No hint. I've looked at data and thought of some feature engineering but then I got Leak Dread",
      "votes": null
    },
    {
      "id": "134024",
      "postDate": "09/04/2016 18:39:53",
      "content": "<p>Unfortunately for you, my ML-process does not include looking for leaks, so my only way to find them is by error and therefore I am unaware of them.</p>",
      "rawMarkdown": "Unfortunately for you, my ML-process does not include looking for leaks, so my only way to find them is by error and therefore I am unaware of them.",
      "votes": null
    },
    {
      "id": "134030",
      "postDate": "09/04/2016 19:27:20",
      "content": "<p>I must be really lucky, I can create them on my own :-) Maybe one day I'm gonna show this super power to the world...</p>",
      "rawMarkdown": "I must be really lucky, I can create them on my own :-) Maybe one day I'm gonna show this super power to the world...",
      "votes": null
    },
    {
      "id": "134033",
      "postDate": "09/04/2016 19:32:51",
      "content": "<p>Does anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. </p>\n\n<p>For instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.</p>\n\n<p>But I can't see how that benefits Bosch. They're trying to predict failure of parts and this data only predicts failure based on parts in the same batch that are already known to fail. </p>\n\n<p>Is there something I'm missing? Maybe they are testing half the parts as they move down the line and want to predict the other half? Or maybe there is something about how the measurements are taken?</p>",
      "rawMarkdown": "Does anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. \r\n\r\nFor instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.\r\n\r\nBut I can't see how that benefits Bosch. They're trying to predict failure of parts and this data only predicts failure based on parts in the same batch that are already known to fail. \r\n\r\nIs there something I'm missing? Maybe they are testing half the parts as they move down the line and want to predict the other half? Or maybe there is something about how the measurements are taken?",
      "votes": null
    },
    {
      "id": "134089",
      "postDate": "09/05/2016 01:42:35",
      "content": "<p>[quote=JohnM;134033]</p>\n\n<p>Does anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. </p>\n\n<p>For instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.\n[/quote]</p>\n\n<p>That is not a leak, that is one of the things they are actually trying to find! I work in semiconductor manufacturing and we frequently see this behavior. Essentially one of the stations has a bad day and damages many parts processed on it during that day.</p>",
      "rawMarkdown": "[quote=JohnM;134033]\r\n\r\nDoes anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. \r\n\r\nFor instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.\r\n[/quote]\r\n\r\nThat is not a leak, that is one of the things they are actually trying to find! I work in semiconductor manufacturing and we frequently see this behavior. Essentially one of the stations has a bad day and damages many parts processed on it during that day.",
      "votes": null
    },
    {
      "id": "134093",
      "postDate": "09/05/2016 02:27:28",
      "content": "<p>Thanks for the explanation, Jeff. </p>",
      "rawMarkdown": "Thanks for the explanation, Jeff.",
      "votes": null
    },
    {
      "id": "134143",
      "postDate": "09/05/2016 12:14:25",
      "content": "<p>JeffH - great insight - it might be worth applying a time-based gaussian filter on the predictions which will spot these clusters </p>",
      "rawMarkdown": "JeffH - great insight - it might be worth applying a time-based gaussian filter on the predictions which will spot these clusters",
      "votes": null
    },
    {
      "id": "134158",
      "postDate": "09/05/2016 13:16:02",
      "content": "<p>Here are a couple interesting examples. Part of the challenge with this kind of behavior is identifying if an increase in failures is related to the tool performance, or the amount of work running through the tool.</p>\n\n<p>Station L3_S32 is heavily used. Around time 750 there was an increase in the number of bad parts off that tool.  The station was stopped around time 800, possibly when manufacturing found a problem. When it came back online, the failure rate was back to normal.</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4781/L3_S32_D3852_trend.png\" alt=\"enter image description here\" title></p>\n\n<p>Station L2_S26 might be easier to interpret. It seems to run in batches of many parts. From time 100 through 400 it ran many parts with no failures. Around time 400 there were quite a few failures. The tool was stopped around time 450. When it came back online, the failure rate appears proportional to the run rate. </p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4782/L2_S26_trend.png\" alt=\"enter image description here\" title></p>",
      "rawMarkdown": "Here are a couple interesting examples. Part of the challenge with this kind of behavior is identifying if an increase in failures is related to the tool performance, or the amount of work running through the tool.\r\n\r\nStation L3_S32 is heavily used. Around time 750 there was an increase in the number of bad parts off that tool.  The station was stopped around time 800, possibly when manufacturing found a problem. When it came back online, the failure rate was back to normal.\r\n\r\n![enter image description here][1]\r\n\r\nStation L2_S26 might be easier to interpret. It seems to run in batches of many parts. From time 100 through 400 it ran many parts with no failures. Around time 400 there were quite a few failures. The tool was stopped around time 450. When it came back online, the failure rate appears proportional to the run rate. \r\n\r\n![enter image description here][2]\r\n\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4781/L3_S32_D3852_trend.png\r\n  [2]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4782/L2_S26_trend.png",
      "votes": null
    },
    {
      "id": "134179",
      "postDate": "09/05/2016 16:34:15",
      "content": "<p>That's a nice graph/format for exploration. One thing I've noticed is the challenge of identifying where the defect occurs, since parts travel through the stations in batches (more or less). I could be wrong, but my sampling seemed to paint that picture.</p>\n\n<p>Except for Station 32, which has a &quot;normal&quot; failure rate of 2x the average and an average failure rate of 10x. No idea what they do there, but it's probably a complex operation for specialty parts.</p>\n\n<p>Does any of this matter for the competition? Maybe not, but since the good folks at Bosch don't want to tell us anything about their factory, it makes for a very interesting puzzle:) </p>",
      "rawMarkdown": "That's a nice graph/format for exploration. One thing I've noticed is the challenge of identifying where the defect occurs, since parts travel through the stations in batches (more or less). I could be wrong, but my sampling seemed to paint that picture.\r\n\r\nExcept for Station 32, which has a \"normal\" failure rate of 2x the average and an average failure rate of 10x. No idea what they do there, but it's probably a complex operation for specialty parts.\r\n\r\nDoes any of this matter for the competition? Maybe not, but since the good folks at Bosch don't want to tell us anything about their factory, it makes for a very interesting puzzle:)",
      "votes": null
    },
    {
      "id": "134195",
      "postDate": "09/05/2016 17:47:09",
      "content": "<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>",
      "rawMarkdown": "Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!",
      "votes": null
    },
    {
      "id": "134198",
      "postDate": "09/05/2016 18:04:56",
      "content": "<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>OK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? </p>\n\n<p>Other possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... </p>",
      "rawMarkdown": "[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nOK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? \r\n\r\nOther possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too...",
      "votes": null
    },
    {
      "id": "134212",
      "postDate": "09/05/2016 20:18:07",
      "content": "<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>Thanks. Because of the size of data and munging concerned, this will be a heart-breaker if a leak is found in the last week!</p>",
      "rawMarkdown": "[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nThanks. Because of the size of data and munging concerned, this will be a heart-breaker if a leak is found in the last week!",
      "votes": null
    },
    {
      "id": "134226",
      "postDate": "09/05/2016 21:57:59",
      "content": "<p>[quote=VictorGarcia;134198]</p>\n\n<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>OK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? </p>\n\n<p>Other possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... </p>\n\n<p>[/quote]</p>\n\n<p>Hi Victor,</p>\n\n<p>I don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089\">here</a>.</p>\n\n<p>I'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.</p>\n\n<p>I think after the whole situation with Expedia, RedHat &amp; TalkingData people are too quick to attribute every bit of useful information as a &quot;leak&quot;. Back in my day, we just called those features :D</p>\n\n<p>- Mikel</p>",
      "rawMarkdown": "[quote=VictorGarcia;134198]\r\n\r\n[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nOK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? \r\n\r\nOther possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... \r\n\r\n[/quote]\r\n\r\nHi Victor,\r\n\r\nI don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well [here](https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089).\r\n\r\nI'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.\r\n\r\nI think after the whole situation with Expedia, RedHat & TalkingData people are too quick to attribute every bit of useful information as a \"leak\". Back in my day, we just called those features :D\r\n\r\n\\- Mikel",
      "votes": null
    },
    {
      "id": "134234",
      "postDate": "09/05/2016 22:37:05",
      "content": "<p>[quote=anokas;134226]</p>\n\n<p>[quote=VictorGarcia;134198]</p>\n\n<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>OK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? </p>\n\n<p>Other possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... </p>\n\n<p>[/quote]</p>\n\n<p>Hi Victor,</p>\n\n<p>I don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089\">here</a>.</p>\n\n<p>I'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.</p>\n\n<p>I think after the whole situation with Expedia, RedHat &amp; TalkingData people are too quick to attribute every bit of useful information as a &quot;leak&quot;. Back in my day, we just called those features :D</p>\n\n<p>- Mikel</p>\n\n<p>[/quote]</p>\n\n<p>OK well even if those aren't necessarily leaks I think we should take the possibility of a data leak seriously.  The scientist in me wants to know exactly what kind of actual tests on the data have been performed to confirm that there are no super obvious data leaks I.e using features that are not based in the real world, but on the choice of how the training/test data sets are made and distributed? </p>",
      "rawMarkdown": "[quote=anokas;134226]\r\n\r\n[quote=VictorGarcia;134198]\r\n\r\n[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nOK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? \r\n\r\nOther possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... \r\n\r\n[/quote]\r\n\r\nHi Victor,\r\n\r\nI don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well [here](https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089).\r\n\r\nI'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.\r\n\r\nI think after the whole situation with Expedia, RedHat & TalkingData people are too quick to attribute every bit of useful information as a \"leak\". Back in my day, we just called those features :D\r\n\r\n\\- Mikel\r\n\r\n[/quote]\r\n\r\nOK well even if those aren't necessarily leaks I think we should take the possibility of a data leak seriously.  The scientist in me wants to know exactly what kind of actual tests on the data have been performed to confirm that there are no super obvious data leaks I.e using features that are not based in the real world, but on the choice of how the training/test data sets are made and distributed?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 133979,
      "author_name": "kailicht",
      "author_url": "",
      "post_date": "09/04/2016 16:23:35",
      "content": "<p>Do you have any hint or are you just asking?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133996,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "09/04/2016 17:08:05",
      "content": "<p>No hint. I've looked at data and thought of some feature engineering but then I got Leak Dread</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134024,
      "author_name": "carloshuertas",
      "author_url": "",
      "post_date": "09/04/2016 18:39:53",
      "content": "<p>Unfortunately for you, my ML-process does not include looking for leaks, so my only way to find them is by error and therefore I am unaware of them.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134030,
      "author_name": "kailicht",
      "author_url": "",
      "post_date": "09/04/2016 19:27:20",
      "content": "<p>I must be really lucky, I can create them on my own :-) Maybe one day I'm gonna show this super power to the world...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134033,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "09/04/2016 19:32:51",
      "content": "<p>Does anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. </p>\n\n<p>For instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.</p>\n\n<p>But I can't see how that benefits Bosch. They're trying to predict failure of parts and this data only predicts failure based on parts in the same batch that are already known to fail. </p>\n\n<p>Is there something I'm missing? Maybe they are testing half the parts as they move down the line and want to predict the other half? Or maybe there is something about how the measurements are taken?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134089,
      "author_name": "jeffhebert",
      "author_url": "",
      "post_date": "09/05/2016 01:42:35",
      "content": "<p>[quote=JohnM;134033]</p>\n\n<p>Does anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. </p>\n\n<p>For instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.\n[/quote]</p>\n\n<p>That is not a leak, that is one of the things they are actually trying to find! I work in semiconductor manufacturing and we frequently see this behavior. Essentially one of the stations has a bad day and damages many parts processed on it during that day.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134093,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "09/05/2016 02:27:28",
      "content": "<p>Thanks for the explanation, Jeff. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134143,
      "author_name": "scirpus",
      "author_url": "",
      "post_date": "09/05/2016 12:14:25",
      "content": "<p>JeffH - great insight - it might be worth applying a time-based gaussian filter on the predictions which will spot these clusters </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134158,
      "author_name": "jeffhebert",
      "author_url": "",
      "post_date": "09/05/2016 13:16:02",
      "content": "<p>Here are a couple interesting examples. Part of the challenge with this kind of behavior is identifying if an increase in failures is related to the tool performance, or the amount of work running through the tool.</p>\n\n<p>Station L3_S32 is heavily used. Around time 750 there was an increase in the number of bad parts off that tool.  The station was stopped around time 800, possibly when manufacturing found a problem. When it came back online, the failure rate was back to normal.</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4781/L3_S32_D3852_trend.png\" alt=\"enter image description here\" title></p>\n\n<p>Station L2_S26 might be easier to interpret. It seems to run in batches of many parts. From time 100 through 400 it ran many parts with no failures. Around time 400 there were quite a few failures. The tool was stopped around time 450. When it came back online, the failure rate appears proportional to the run rate. </p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4782/L2_S26_trend.png\" alt=\"enter image description here\" title></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134179,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "09/05/2016 16:34:15",
      "content": "<p>That's a nice graph/format for exploration. One thing I've noticed is the challenge of identifying where the defect occurs, since parts travel through the stations in batches (more or less). I could be wrong, but my sampling seemed to paint that picture.</p>\n\n<p>Except for Station 32, which has a &quot;normal&quot; failure rate of 2x the average and an average failure rate of 10x. No idea what they do there, but it's probably a complex operation for specialty parts.</p>\n\n<p>Does any of this matter for the competition? Maybe not, but since the good folks at Bosch don't want to tell us anything about their factory, it makes for a very interesting puzzle:) </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134195,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "09/05/2016 17:47:09",
      "content": "<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134198,
      "author_name": "garcia274",
      "author_url": "",
      "post_date": "09/05/2016 18:04:56",
      "content": "<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>OK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? </p>\n\n<p>Other possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134212,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "09/05/2016 20:18:07",
      "content": "<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>Thanks. Because of the size of data and munging concerned, this will be a heart-breaker if a leak is found in the last week!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134226,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "09/05/2016 21:57:59",
      "content": "<p>[quote=VictorGarcia;134198]</p>\n\n<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>OK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? </p>\n\n<p>Other possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... </p>\n\n<p>[/quote]</p>\n\n<p>Hi Victor,</p>\n\n<p>I don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089\">here</a>.</p>\n\n<p>I'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.</p>\n\n<p>I think after the whole situation with Expedia, RedHat &amp; TalkingData people are too quick to attribute every bit of useful information as a &quot;leak&quot;. Back in my day, we just called those features :D</p>\n\n<p>- Mikel</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134234,
      "author_name": "garcia274",
      "author_url": "",
      "post_date": "09/05/2016 22:37:05",
      "content": "<p>[quote=anokas;134226]</p>\n\n<p>[quote=VictorGarcia;134198]</p>\n\n<p>[quote=anokas;134195]</p>\n\n<p>Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)</p>\n\n<p>Good luck!</p>\n\n<p>[/quote]</p>\n\n<p>OK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? </p>\n\n<p>Other possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... </p>\n\n<p>[/quote]</p>\n\n<p>Hi Victor,</p>\n\n<p>I don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089\">here</a>.</p>\n\n<p>I'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.</p>\n\n<p>I think after the whole situation with Expedia, RedHat &amp; TalkingData people are too quick to attribute every bit of useful information as a &quot;leak&quot;. Back in my day, we just called those features :D</p>\n\n<p>- Mikel</p>\n\n<p>[/quote]</p>\n\n<p>OK well even if those aren't necessarily leaks I think we should take the possibility of a data leak seriously.  The scientist in me wants to know exactly what kind of actual tests on the data have been performed to confirm that there are no super obvious data leaks I.e using features that are not based in the real world, but on the choice of how the training/test data sets are made and distributed? </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "133952": "Before I start spending hours munging for this competition, has anyone found any kind of leak? I really don't want to waste my time doing data science for competitions that don't require data science.",
    "133979": "Do you have any hint or are you just asking?",
    "133996": "No hint. I've looked at data and thought of some feature engineering but then I got Leak Dread",
    "134024": "Unfortunately for you, my ML-process does not include looking for leaks, so my only way to find them is by error and therefore I am unaware of them.",
    "134030": "I must be really lucky, I can create them on my own :-) Maybe one day I'm gonna show this super power to the world...",
    "134033": "Does anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. \r\n\r\nFor instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.\r\n\r\nBut I can't see how that benefits Bosch. They're trying to predict failure of parts and this data only predicts failure based on parts in the same batch that are already known to fail. \r\n\r\nIs there something I'm missing? Maybe they are testing half the parts as they move down the line and want to predict the other half? Or maybe there is something about how the measurements are taken?",
    "134089": "[quote=JohnM;134033]\r\n\r\nDoes anyone consider the time stamps in the two date files a type of leak? I get a MCC of around .16 just by looking at where parts were at different times. \r\n\r\nFor instance, around t = 1376, there was a high percentage of failures in the train set between S33 and S37. So, an algorithm gives a higher probability of failure to parts in the test set which were also at that time and place.\r\n[/quote]\r\n\r\nThat is not a leak, that is one of the things they are actually trying to find! I work in semiconductor manufacturing and we frequently see this behavior. Essentially one of the stations has a bad day and damages many parts processed on it during that day.",
    "134093": "Thanks for the explanation, Jeff.",
    "134143": "JeffH - great insight - it might be worth applying a time-based gaussian filter on the predictions which will spot these clusters",
    "134158": "Here are a couple interesting examples. Part of the challenge with this kind of behavior is identifying if an increase in failures is related to the tool performance, or the amount of work running through the tool.\r\n\r\nStation L3_S32 is heavily used. Around time 750 there was an increase in the number of bad parts off that tool.  The station was stopped around time 800, possibly when manufacturing found a problem. When it came back online, the failure rate was back to normal.\r\n\r\n![enter image description here][1]\r\n\r\nStation L2_S26 might be easier to interpret. It seems to run in batches of many parts. From time 100 through 400 it ran many parts with no failures. Around time 400 there were quite a few failures. The tool was stopped around time 450. When it came back online, the failure rate appears proportional to the run rate. \r\n\r\n![enter image description here][2]\r\n\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4781/L3_S32_D3852_trend.png\r\n  [2]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4782/L2_S26_trend.png",
    "134179": "That's a nice graph/format for exploration. One thing I've noticed is the challenge of identifying where the defect occurs, since parts travel through the stations in batches (more or less). I could be wrong, but my sampling seemed to paint that picture.\r\n\r\nExcept for Station 32, which has a \"normal\" failure rate of 2x the average and an average failure rate of 10x. No idea what they do there, but it's probably a complex operation for specialty parts.\r\n\r\nDoes any of this matter for the competition? Maybe not, but since the good folks at Bosch don't want to tell us anything about their factory, it makes for a very interesting puzzle:)",
    "134195": "Message from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!",
    "134198": "[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nOK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? \r\n\r\nOther possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too...",
    "134212": "[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nThanks. Because of the size of data and munging concerned, this will be a heart-breaker if a leak is found in the last week!",
    "134226": "[quote=VictorGarcia;134198]\r\n\r\n[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nOK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? \r\n\r\nOther possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... \r\n\r\n[/quote]\r\n\r\nHi Victor,\r\n\r\nI don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well [here](https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089).\r\n\r\nI'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.\r\n\r\nI think after the whole situation with Expedia, RedHat & TalkingData people are too quick to attribute every bit of useful information as a \"leak\". Back in my day, we just called those features :D\r\n\r\n\\- Mikel",
    "134234": "[quote=anokas;134226]\r\n\r\n[quote=VictorGarcia;134198]\r\n\r\n[quote=anokas;134195]\r\n\r\nMessage from the top: We haven't found a leak and I think it's unlikely that there is one - the data looks well prepared :)\r\n\r\nGood luck!\r\n\r\n[/quote]\r\n\r\nOK so just to be 100% sure: Someone has performed tests using the ID# as a feature, and haven't found any use above random in using the ID number to predict the response? \r\n\r\nOther possible leaks could be using the number of digits in train_numeric.csv and test_numeric to predict response, I am sure there are other possibilities too... \r\n\r\n[/quote]\r\n\r\nHi Victor,\r\n\r\nI don't consider these two scenarios 'leaks' in this particular competition. In the data, we know that the train and tests were random splits of the original unshuffled dataset, and we are explicitly given the original order prior to splitting (as the IDs are not separate for train and test) - I believe that we are supposed to incorporate this data that we were given into our models, as @JeffH put well [here](https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23354/so-is-there-a-leak/134089#post134089).\r\n\r\nI'm assuming that by the number of digits you mean something like the total length of each line. This isn't a leak, but rather a feature, as we are applying a transformation to the training data to create some new data which allows the model to discover new information.\r\n\r\nI think after the whole situation with Expedia, RedHat & TalkingData people are too quick to attribute every bit of useful information as a \"leak\". Back in my day, we just called those features :D\r\n\r\n\\- Mikel\r\n\r\n[/quote]\r\n\r\nOK well even if those aren't necessarily leaks I think we should take the possibility of a data leak seriously.  The scientist in me wants to know exactly what kind of actual tests on the data have been performed to confirm that there are no super obvious data leaks I.e using features that are not based in the real world, but on the choice of how the training/test data sets are made and distributed?"
  },
  "source": "meta"
}