![\"enter](\"https://s3.amazonaws.com/nonwebstorage/ProductFamilies.png\" "\\"\\"")
Heres a \"production map\" that gives a sense of how parts make their way through the factory. Which machines each part touches, which machines it skips, etc. It gives a rough grouping of parts with similar flow paths. Should be useful to understanding quality!
\n\nLater this week I'll post the code so you can check for issues, improve on it, use the data, etc.
\n\n
Heres a \"production map\" that gives a sense of how parts make their way through the factory. Which machines each part touches, which machines it skips, etc. It gives a rough grouping of parts with similar flow paths. Should be useful to understanding quality!
\n\nLater this week I'll post the code so you can check for issues, improve on it, use the data, etc.
\n\n
cool
", "rawMarkdown": "cool", "votes": null }, { "id": "132164", "postDate": "08/23/2016 07:15:25", "content": "Super-cool!
", "rawMarkdown": "Super-cool!", "votes": null }, { "id": "132176", "postDate": "08/23/2016 09:01:22", "content": "I just started looking through the data (mainly the categoricals). It's not clear to me what you are using for "Product Families". If it is something obvious, feel free to tell me to just look harder. :-)
", "rawMarkdown": "I just started looking through the data (mainly the categoricals). It's not clear to me what you are using for \"Product Families\". If it is something obvious, feel free to tell me to just look harder. :-)", "votes": null }, { "id": "132222", "postDate": "08/23/2016 17:58:38", "content": "Here's more of an explanation on the logic and code for looking at parts flow:
\n\nThe VIM package for R does all the heavy lifting for the drawing. By counting and sorting NAs for each part, it gives us a nice count and sort of the non-NAs as well. Some notes on the drawing:
\n\nWe can see a few things from the picture:
\n\nOf course the possible explanations I've added are hypothetical only. We should be able to use the dates file and add some more insight into the flow. My next step is probably to add the Response variable into this and see if anything stands out before going much further.
\n\nPS - code in R is at https://github.com/JohnM-TX/Kaggle-Bosch . The biggest challenge was handling the reshaping within memory limits. I started in Python-Pandas, but then switched to R and reshaped/aggregated in chunks before feeding it to the VIM function.
", "rawMarkdown": "Here's more of an explanation on the logic and code for looking at parts flow:\r\n\r\n - The goal is to trace each part across the 52 stations and get a feel for the manufacturing flow. Many times, quality is a function of the part type and the specific machine used. \r\n - Measurements are only used here in a binary sense. If the part has any measurement at a station, categorical or numeric, then the part was operated on by that station. Otherwise the part did not go through the station.\r\n - Grouping parts with similar manufacturing patterns will help us identify families of parts and possibly shed light on the nature of the stations.\r\n\r\n\r\nThe VIM package for R does all the heavy lifting for the drawing. By counting and sorting NAs for each part, it gives us a nice count and sort of the non-NAs as well. Some notes on the drawing:\r\n\r\n - The x-axis represents each station, obviously. The y-axis represents a stack of parts, sorted by similar patterns of NAs. \r\n - The height of the blocks represents the relative frequency of the groups. I called the groups \"product families\" which is bit confusing here. \"Parts groups\" may make more sense. \r\n - The bars across the top show the % of NAs for each station.\r\n\r\nWe can see a few things from the picture:\r\n\r\n - There are 3 basic part groups. Those that go through L0(S0-S11), those that go through L0(S12-23), and those that go through L1 & L2. All parts go through L3.\r\n - L0(S0-S11) and L0(S12-S23) have similar patterns so these may be similar part types with a differing characteristic. For instance, regular size parts might go through one group of machines, and a smaller number of jumbo parts go through the other. \r\n - Within each major group there are complementary stations. S09-S11, for instance. These could be 3 stations with the same equipment, or they could have different equipment to deal with part options. Given the equal volume, I think it's more likely in this case they are 3 instances of the same thing.\r\n - Everything runs through L3. Most parts go through the same 5 stations and then there are many stations with low volume. This might be the finishing line, where almost everything gets deburred, sand blasted, painted, etc. and a small number of parts get some sort of custom finish. Or maybe this is an assembly step?\r\n\r\nOf course the possible explanations I've added are hypothetical only. We should be able to use the dates file and add some more insight into the flow. My next step is probably to add the Response variable into this and see if anything stands out before going much further.\r\n\r\nPS - code in R is at https://github.com/JohnM-TX/Kaggle-Bosch . The biggest challenge was handling the reshaping within memory limits. I started in Python-Pandas, but then switched to R and reshaped/aggregated in chunks before feeding it to the VIM function.", "votes": null }, { "id": "132263", "postDate": "08/24/2016 02:09:57", "content": "picture is worth a 1000 words !
", "rawMarkdown": "picture is worth a 1000 words !", "votes": null }, { "id": "132523", "postDate": "08/26/2016 13:09:24", "content": "Hi John,
\n\nAs promised, here's what I got from the train_date.csv file:
\n\nBased on the first 500 (data, non-header) rows of train_date.csv and train_numeric.csv, I was able to deduce four common workflow patterns unique to each row, as represented in the tab 'flow' in the attached excel file. Broadly speaking, there are rows that:
\n\n(1) Initialize at S0, then end at either S37 or S51;\n(2) Initialize at S12, then end at either S37 or S38;\n(3) Initialize at S24, then end at S37;\n(4) Initialize at S25, then end at S37 or S51.
\n\nAs you defined above, if there were null values under a station (and all the features under that station) then a part was not considered to have gone through that station.
\n\nNotice that along the way, parts are sometimes processed in one of 2 or 3 stations then everything returns back to a common station. The basis for this conclusion was a bit arbitrary although at best grounded by the timestamps. You'll notice, for example, for rows that initialize at S0, after passing through S1, the next timestamps that continue from S1 are either in S2 or S3. That is, for those rows that started at S0, when S2 are null (or in my case replaced with zero), S3 is activated; and vice-versa, when S3 is null, S2 is activated.
\n\nLooking at the 'time stamps normalized' tab, you can see the from a normalized starting time value of 1, the times at S2/S3 are 1.01 at most. Similarly, S4/S5 show the same either/or/but/not/both pattern, and the normalized time stamps there are 1.02/1.03. Also, when looking at similar rows in train_numeric.csv (which is not included in the excel file for brevity), you'll notice the same either/or pattern with the same number of features for either alternative station.
\n\nBtw, I 'normalized' the time values by doing an element-wise subtraction of ALL non-zero values by the non-zero minimum and added one back. This means that the lowest actual available value in any row is 1. And all the higher time stamps will just show the incremental time units since the 'earliest' timestamp (+1). This helps understand the processing speed more uniformly across all parts. (All remaining zeros in train_numeric were either really zero or blanks, since my extraction replaced blanks/NAs with zero.)
\n\nIn any case, this matches the symmetric flip in your blue bars for S2 and S3 in your picture above; some parts simply go to S2 instead of S3 and vice versa. Same for S4/S5, and S6/S7 alternating. Then S8 receives everything that started in S0. And this is why the flip is across three stations in S9/S10/S11 -- those guys take turns doing something to parts depending on the part. And so on and so forth... the downstream stations were a bit harder to discern, and they seem to take on pretty much everything, but I didn't focus on that too much.
\n\nIn any case, the implication is one of two things:
\n\n(1) either pairs like S4/S5 do something similar and thus, these columns have to be merged. (This would in of itself be a challenge since we do not know if the features in S4 are ordered in the same columns as correspondent features in S5.) Treating features under them as separate features will yield horrible probability density calculations as both positive and null samples will have multiple common zeros across features in both stations if treated separately, which will dull any useful differences between anomalies and non-anomalies; OR\n(2) pairs like S4/S5 do something different for each part, and thus, these columns will need to stay separated BUT different probability density calculations for each unique row path will need to be used depending on what has values.
\n\nI was surprised by the relative symmetry the assumptions I made took on when rendered on a flowchart. I wouldn't trust the end station representations that much because again, the data is so sparse and I only used 500 rows, but for the 'earliest' features activated towards the middle, I have a bit more confidence about. Corroborating these results with your graphic was very reassuring though!
\n\nOh lastly, not sure yet if each row(/part) then feeds into another row(part); since the parts all merge somehow eventually to form a machine, I'm sure some rows relate to others somehow, but I can't get a sense of that yet. The fact that a part can have its first 'timestamp' (or feature for that matter) largely either at S00, S12, S24, or S25 may matter, or even the fact that the timestamps do not all start from the same value. In this case the 'raw' non-normalized timestamp may shed some light.
\n\n(PS - the 'chunks' I labeled timestamps as were for my own purposes when I was guessing which features were measured at the same time... it became apparent that it was station-specific, except for S24 and S25; within those stations certain features represent entirely different processes and imply 'sub-stations'.)
\n\nCheers.
", "rawMarkdown": "Hi John,\r\n\r\nAs promised, here's what I got from the train_date.csv file:\r\n\r\nBased on the first 500 (data, non-header) rows of train_date.csv and train_numeric.csv, I was able to deduce four common workflow patterns unique to each row, as represented in the tab 'flow' in the attached excel file. Broadly speaking, there are rows that:\r\n\r\n(1) Initialize at S0, then end at either S37 or S51;\r\n(2) Initialize at S12, then end at either S37 or S38;\r\n(3) Initialize at S24, then end at S37;\r\n(4) Initialize at S25, then end at S37 or S51.\r\n\r\nAs you defined above, if there were null values under a station (and all the features under that station) then a part was not considered to have gone through that station.\r\n\r\nNotice that along the way, parts are sometimes processed in one of 2 or 3 stations then everything returns back to a common station. The basis for this conclusion was a bit arbitrary although at best grounded by the timestamps. You'll notice, for example, for rows that initialize at S0, after passing through S1, the next timestamps that continue from S1 are *either* in S2 or S3. That is, for those rows that started at S0, when S2 are null (or in my case replaced with zero), S3 is activated; and vice-versa, when S3 is null, S2 is activated.\r\n\r\nLooking at the 'time stamps normalized' tab, you can see the from a normalized starting time value of 1, the times at S2/S3 are 1.01 at most. Similarly, S4/S5 show the same either/or/but/not/both pattern, and the normalized time stamps there are 1.02/1.03. Also, when looking at similar rows in train_numeric.csv (which is not included in the excel file for brevity), you'll notice the same either/or pattern with the same number of features for either alternative station.\r\n\r\nBtw, I 'normalized' the time values by doing an element-wise subtraction of ALL non-zero values by the non-zero minimum and added one back. This means that the lowest actual available value in any row is 1. And all the higher time stamps will just show the incremental time units since the 'earliest' timestamp (+1). This helps understand the processing speed more uniformly across all parts. (All remaining zeros in train_numeric were either really zero or blanks, since my extraction replaced blanks/NAs with zero.)\r\n\r\nIn any case, this matches the symmetric flip in your blue bars for S2 and S3 in your picture above; some parts simply go to S2 instead of S3 and vice versa. Same for S4/S5, and S6/S7 alternating. Then S8 receives everything that started in S0. And this is why the flip is across three stations in S9/S10/S11 -- those guys take turns doing something to parts depending on the part. And so on and so forth... the downstream stations were a bit harder to discern, and they seem to take on pretty much everything, but I didn't focus on that too much.\r\n\r\nIn any case, the implication is one of two things:\r\n\r\n(1) either pairs like S4/S5 do something similar and thus, these columns have to be merged. (This would in of itself be a challenge since we do not know if the features in S4 are ordered in the same columns as correspondent features in S5.) Treating features under them as separate features will yield horrible probability density calculations as both positive and null samples will have multiple common zeros across features in both stations if treated separately, which will dull any useful differences between anomalies and non-anomalies; OR\r\n(2) pairs like S4/S5 do something different for each part, and thus, these columns will need to stay separated BUT different probability density calculations for each unique row path will need to be used depending on what has values.\r\n\r\nI was surprised by the relative symmetry the assumptions I made took on when rendered on a flowchart. I wouldn't trust the end station representations that much because again, the data is so sparse and I only used 500 rows, but for the 'earliest' features activated towards the middle, I have a bit more confidence about. Corroborating these results with your graphic was very reassuring though!\r\n\r\nOh lastly, not sure yet if each row(/part) then feeds into another row(part); since the parts all merge somehow eventually to form a machine, I'm sure some rows relate to others somehow, but I can't get a sense of that yet. The fact that a part can have its first 'timestamp' (or feature for that matter) largely either at S00, S12, S24, or S25 may matter, or even the fact that the timestamps do not all start from the same value. In this case the 'raw' non-normalized timestamp may shed some light.\r\n\r\n(PS - the 'chunks' I labeled timestamps as were for my own purposes when I was guessing which features were measured at the same time... it became apparent that it was station-specific, except for S24 and S25; within those stations certain features represent entirely different processes and imply 'sub-stations'.)\r\n\r\nCheers.", "votes": null }, { "id": "132746", "postDate": "08/29/2016 00:27:13", "content": "Hello John, I read your code. Why did you allot the first 60 rows into part1? And with regard to the figure you posted, is that base on the part1 one or the whole data? Thank you.
", "rawMarkdown": "Hello John, I read your code. Why did you allot the first 60 rows into part1? And with regard to the figure you posted, is that base on the part1 one or the whole data? Thank you.", "votes": null }, { "id": "132756", "postDate": "08/29/2016 03:59:42", "content": "Hi Haom - the figure I posted is based on all training data. As for the 60 rows in part 1, that was just to demonstrate the output with a small piece. Depending on your RAM, you will want many more rows per part to process the whole file.
\n\nI was able to digest 600,000 rows at a time on an Amazon cluster with ~60GB (only $0.12/hr!). The limit was not actually with fread, but with the data.table.melt function in the next step.
\n\nFor "normal" RAM, you can set the part size at whatever works and run a for loop. It may take a while, but it will work. And you can process categorical and numeric files separately, and combine them at the very end - the result should be the same.
", "rawMarkdown": "Hi Haom - the figure I posted is based on all training data. As for the 60 rows in part 1, that was just to demonstrate the output with a small piece. Depending on your RAM, you will want many more rows per part to process the whole file.\r\n\r\nI was able to digest 600,000 rows at a time on an Amazon cluster with ~60GB (only $0.12/hr!). The limit was not actually with fread, but with the data.table.melt function in the next step. \r\n\r\nFor \"normal\" RAM, you can set the part size at whatever works and run a for loop. It may take a while, but it will work. And you can process categorical and numeric files separately, and combine them at the very end - the result should be the same.", "votes": null }, { "id": "132760", "postDate": "08/29/2016 04:32:43", "content": "Hi John, thank you for your reply so fast. I see, I tried different parts with different size, they all got the same results! Thank you for what you have done on this project, it really helpful for me to have a comprehensive understanding about the data. The visualization is cool, I never used VIM before, and I learned a lot from your work. Thanks again, looking forward to your fellow up on this. \nAnother thing, I think the gc() command that you used is a really good method, it can save a lot of time and space.
", "rawMarkdown": "Hi John, thank you for your reply so fast. I see, I tried different parts with different size, they all got the same results! Thank you for what you have done on this project, it really helpful for me to have a comprehensive understanding about the data. The visualization is cool, I never used VIM before, and I learned a lot from your work. Thanks again, looking forward to your fellow up on this. \r\nAnother thing, I think the gc() command that you used is a really good method, it can save a lot of time and space.", "votes": null }, { "id": "133366", "postDate": "09/01/2016 17:37:29", "content": "Hi John, Thanks a lot for the great analysis. Based on your direction, I just looked at them by initial station number (specifically, first column in train_numeric.csv with a non-null value), and then looked to see where Response was 0 or 1.
\n\nBased on this, it gives the appearance to me of S0 as the highest runner, S12 and S24 as significantly lower, and then so on to lower and lower frequency parts. It looks to my untrained eye like a Pareto distribution, where each step down the frequency chart has a big drop, but there is a long tail of parts with only a few rows each.
\n\nIt's not much of an improvement on your method but it has the advantage of being very easy to implement (for each line in train_numeric, just look for the first station with a non-null result), and it gives an idea of what different parts there are. If it looks wrong to you, of course feel free to point that out. :)
\n\nThanks again for the great analysis.
", "rawMarkdown": "Hi John, Thanks a lot for the great analysis. Based on your direction, I just looked at them by initial station number (specifically, first column in train_numeric.csv with a non-null value), and then looked to see where Response was 0 or 1.\r\n\r\nBased on this, it gives the appearance to me of S0 as the highest runner, S12 and S24 as significantly lower, and then so on to lower and lower frequency parts. It looks to my untrained eye like a Pareto distribution, where each step down the frequency chart has a big drop, but there is a long tail of parts with only a few rows each.\r\n\r\nIt's not much of an improvement on your method but it has the advantage of being very easy to implement (for each line in train_numeric, just look for the first station with a non-null result), and it gives an idea of what different parts there are. If it looks wrong to you, of course feel free to point that out. :)\r\n\r\nThanks again for the great analysis.", "votes": null }, { "id": "133395", "postDate": "09/01/2016 20:50:59", "content": "David - These numbers look consistent with the picture I made. More importantly, it seems you have identified a meaningful difference between parts starting at S24 vs S25!
", "rawMarkdown": "David - These numbers look consistent with the picture I made. More importantly, it seems you have identified a meaningful difference between parts starting at S24 vs S25!", "votes": null }, { "id": "133411", "postDate": "09/02/2016 00:00:58", "content": "Good point! I think most of these "part types" (if that's what they are) don't have enough data to identify meaningful differences, but those are two of the ones that we have enough data on.
", "rawMarkdown": "Good point! I think most of these \"part types\" (if that's what they are) don't have enough data to identify meaningful differences, but those are two of the ones that we have enough data on.", "votes": null }, { "id": "133423", "postDate": "09/02/2016 04:16:37", "content": "Great Visualization of the production line..
", "rawMarkdown": "Great Visualization of the production line..", "votes": null }, { "id": "133441", "postDate": "09/02/2016 08:05:09", "content": "[quote=JohnM;133441]
\n\nThe x-axis represents each station, obviously. The y-axis represents a stack of parts, sorted by similar patterns of NAs.
\n\n[/quote]
\n\nI was wondering how did you separate them so nicely by parts, what kind of algorithm was used ?
\n\nI suppose "parts" here mean a cluster of data points with similar patterns of NAs?
\n\nSorry for clarifying, but some terminology is confusing for someone without any engineering background.
", "rawMarkdown": "[quote=JohnM;133441]\r\n\r\n\r\nThe x-axis represents each station, obviously. The y-axis represents a stack of parts, sorted by similar patterns of NAs. \r\n\r\n\r\n[/quote]\r\n\r\nI was wondering how did you separate them so nicely by parts, what kind of algorithm was used ? \r\n\r\nI suppose \"parts\" here mean a *cluster* of data points with similar patterns of NAs?\r\n\r\nSorry for clarifying, but some terminology is confusing for someone without any engineering background.", "votes": null }, { "id": "133443", "postDate": "09/02/2016 08:25:19", "content": "leexa - he just sorted the "hashes" lexicographically, that's done all the job.
", "rawMarkdown": "**leexa** - he just sorted the \"hashes\" lexicographically, that's done all the job.", "votes": null }, { "id": "134055", "postDate": "09/04/2016 21:41:23", "content": "Hey John,
\n\nThe map is really nice..What I can see is that stations like 9-10-11 have same product , because where one ends other starts. I think it's like same product is going through 3 different stations (in case of 9-10-11) for various operations which cannot be done on single station. Same is the case with few other stations as well. If we combine these stations and then plot this same map, we should be able to eliminate most of the NAs from the data.\nWhat do you think?
", "rawMarkdown": "Hey John,\r\n\r\nThe map is really nice..What I can see is that stations like 9-10-11 have same product , because where one ends other starts. I think it's like same product is going through 3 different stations (in case of 9-10-11) for various operations which cannot be done on single station. Same is the case with few other stations as well. If we combine these stations and then plot this same map, we should be able to eliminate most of the NAs from the data.\r\nWhat do you think?", "votes": null }, { "id": "134077", "postDate": "09/05/2016 00:16:43", "content": "Hi Mrugank - as long as you preserve the features and encode the station number into the data, it might work. One way would be to have a column populated with either '9', '10', or '11' depending on which station the part went through, followed by columns with the features from that station. I haven't tried different ways yet to see what works best with various algos.
", "rawMarkdown": "Hi Mrugank - as long as you preserve the features and encode the station number into the data, it might work. One way would be to have a column populated with either '9', '10', or '11' depending on which station the part went through, followed by columns with the features from that station. I haven't tried different ways yet to see what works best with various algos.", "votes": null }, { "id": "134567", "postDate": "09/07/2016 09:45:34", "content": "Can anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?
", "rawMarkdown": "Can anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?", "votes": null }, { "id": "139516", "postDate": "10/14/2016 16:18:45", "content": "[quote=shubhijain;134567]
\n\nCan anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?
\n\n[/quote]
\n\nWell, the Response column indicates if a part failed or not. It's the observed value which is compared with the model's predicted value. Both of these are input to the MCC, which you'll find plenty of information about on the forum.
\n\nBy decimal values, I assume you mean the numeric data other than the Response column? These are the feature values under each feature column. The values range between -1 and 1, which shows they've been normalised. We don't know anything else about the data but my guess is that numerical data represent measured part tolerance for a given feature.
", "rawMarkdown": "[quote=shubhijain;134567]\r\n\r\nCan anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?\r\n\r\n\r\n[/quote]\r\n\r\nWell, the Response column indicates if a part failed or not. It's the observed value which is compared with the model's predicted value. Both of these are input to the MCC, which you'll find plenty of information about on the forum.\r\n\r\nBy decimal values, I assume you mean the numeric data other than the Response column? These are the feature values under each feature column. The values range between -1 and 1, which shows they've been normalised. We don't know anything else about the data but my guess is that numerical data represent measured part tolerance for a given feature.", "votes": null }, { "id": "139520", "postDate": "10/14/2016 16:40:32", "content": "Of course, you've probably got an answer already after 5 weeks... :)
", "rawMarkdown": "Of course, you've probably got an answer already after 5 weeks... :)", "votes": null } ], "comments": [ { "id": 132161, "author_name": "harriken", "author_url": "", "post_date": "08/23/2016 05:35:33", "content": "cool
", "votes": null, "replies": [] }, { "id": 132164, "author_name": "sakvaua", "author_url": "", "post_date": "08/23/2016 07:15:25", "content": "Super-cool!
", "votes": null, "replies": [] }, { "id": 132176, "author_name": "inversion", "author_url": "", "post_date": "08/23/2016 09:01:22", "content": "I just started looking through the data (mainly the categoricals). It's not clear to me what you are using for "Product Families". If it is something obvious, feel free to tell me to just look harder. :-)
", "votes": null, "replies": [] }, { "id": 132222, "author_name": "jpmiller", "author_url": "", "post_date": "08/23/2016 17:58:38", "content": "Here's more of an explanation on the logic and code for looking at parts flow:
\n\nThe VIM package for R does all the heavy lifting for the drawing. By counting and sorting NAs for each part, it gives us a nice count and sort of the non-NAs as well. Some notes on the drawing:
\n\nWe can see a few things from the picture:
\n\nOf course the possible explanations I've added are hypothetical only. We should be able to use the dates file and add some more insight into the flow. My next step is probably to add the Response variable into this and see if anything stands out before going much further.
\n\nPS - code in R is at https://github.com/JohnM-TX/Kaggle-Bosch . The biggest challenge was handling the reshaping within memory limits. I started in Python-Pandas, but then switched to R and reshaped/aggregated in chunks before feeding it to the VIM function.
", "votes": null, "replies": [] }, { "id": 132263, "author_name": "akumthek", "author_url": "", "post_date": "08/24/2016 02:09:57", "content": "picture is worth a 1000 words !
", "votes": null, "replies": [] }, { "id": 132523, "author_name": "jedsd3285", "author_url": "", "post_date": "08/26/2016 13:09:24", "content": "Hi John,
\n\nAs promised, here's what I got from the train_date.csv file:
\n\nBased on the first 500 (data, non-header) rows of train_date.csv and train_numeric.csv, I was able to deduce four common workflow patterns unique to each row, as represented in the tab 'flow' in the attached excel file. Broadly speaking, there are rows that:
\n\n(1) Initialize at S0, then end at either S37 or S51;\n(2) Initialize at S12, then end at either S37 or S38;\n(3) Initialize at S24, then end at S37;\n(4) Initialize at S25, then end at S37 or S51.
\n\nAs you defined above, if there were null values under a station (and all the features under that station) then a part was not considered to have gone through that station.
\n\nNotice that along the way, parts are sometimes processed in one of 2 or 3 stations then everything returns back to a common station. The basis for this conclusion was a bit arbitrary although at best grounded by the timestamps. You'll notice, for example, for rows that initialize at S0, after passing through S1, the next timestamps that continue from S1 are either in S2 or S3. That is, for those rows that started at S0, when S2 are null (or in my case replaced with zero), S3 is activated; and vice-versa, when S3 is null, S2 is activated.
\n\nLooking at the 'time stamps normalized' tab, you can see the from a normalized starting time value of 1, the times at S2/S3 are 1.01 at most. Similarly, S4/S5 show the same either/or/but/not/both pattern, and the normalized time stamps there are 1.02/1.03. Also, when looking at similar rows in train_numeric.csv (which is not included in the excel file for brevity), you'll notice the same either/or pattern with the same number of features for either alternative station.
\n\nBtw, I 'normalized' the time values by doing an element-wise subtraction of ALL non-zero values by the non-zero minimum and added one back. This means that the lowest actual available value in any row is 1. And all the higher time stamps will just show the incremental time units since the 'earliest' timestamp (+1). This helps understand the processing speed more uniformly across all parts. (All remaining zeros in train_numeric were either really zero or blanks, since my extraction replaced blanks/NAs with zero.)
\n\nIn any case, this matches the symmetric flip in your blue bars for S2 and S3 in your picture above; some parts simply go to S2 instead of S3 and vice versa. Same for S4/S5, and S6/S7 alternating. Then S8 receives everything that started in S0. And this is why the flip is across three stations in S9/S10/S11 -- those guys take turns doing something to parts depending on the part. And so on and so forth... the downstream stations were a bit harder to discern, and they seem to take on pretty much everything, but I didn't focus on that too much.
\n\nIn any case, the implication is one of two things:
\n\n(1) either pairs like S4/S5 do something similar and thus, these columns have to be merged. (This would in of itself be a challenge since we do not know if the features in S4 are ordered in the same columns as correspondent features in S5.) Treating features under them as separate features will yield horrible probability density calculations as both positive and null samples will have multiple common zeros across features in both stations if treated separately, which will dull any useful differences between anomalies and non-anomalies; OR\n(2) pairs like S4/S5 do something different for each part, and thus, these columns will need to stay separated BUT different probability density calculations for each unique row path will need to be used depending on what has values.
\n\nI was surprised by the relative symmetry the assumptions I made took on when rendered on a flowchart. I wouldn't trust the end station representations that much because again, the data is so sparse and I only used 500 rows, but for the 'earliest' features activated towards the middle, I have a bit more confidence about. Corroborating these results with your graphic was very reassuring though!
\n\nOh lastly, not sure yet if each row(/part) then feeds into another row(part); since the parts all merge somehow eventually to form a machine, I'm sure some rows relate to others somehow, but I can't get a sense of that yet. The fact that a part can have its first 'timestamp' (or feature for that matter) largely either at S00, S12, S24, or S25 may matter, or even the fact that the timestamps do not all start from the same value. In this case the 'raw' non-normalized timestamp may shed some light.
\n\n(PS - the 'chunks' I labeled timestamps as were for my own purposes when I was guessing which features were measured at the same time... it became apparent that it was station-specific, except for S24 and S25; within those stations certain features represent entirely different processes and imply 'sub-stations'.)
\n\nCheers.
", "votes": null, "replies": [] }, { "id": 132746, "author_name": "freshfishing", "author_url": "", "post_date": "08/29/2016 00:27:13", "content": "Hello John, I read your code. Why did you allot the first 60 rows into part1? And with regard to the figure you posted, is that base on the part1 one or the whole data? Thank you.
", "votes": null, "replies": [] }, { "id": 132756, "author_name": "jpmiller", "author_url": "", "post_date": "08/29/2016 03:59:42", "content": "Hi Haom - the figure I posted is based on all training data. As for the 60 rows in part 1, that was just to demonstrate the output with a small piece. Depending on your RAM, you will want many more rows per part to process the whole file.
\n\nI was able to digest 600,000 rows at a time on an Amazon cluster with ~60GB (only $0.12/hr!). The limit was not actually with fread, but with the data.table.melt function in the next step.
\n\nFor "normal" RAM, you can set the part size at whatever works and run a for loop. It may take a while, but it will work. And you can process categorical and numeric files separately, and combine them at the very end - the result should be the same.
", "votes": null, "replies": [] }, { "id": 132760, "author_name": "freshfishing", "author_url": "", "post_date": "08/29/2016 04:32:43", "content": "Hi John, thank you for your reply so fast. I see, I tried different parts with different size, they all got the same results! Thank you for what you have done on this project, it really helpful for me to have a comprehensive understanding about the data. The visualization is cool, I never used VIM before, and I learned a lot from your work. Thanks again, looking forward to your fellow up on this. \nAnother thing, I think the gc() command that you used is a really good method, it can save a lot of time and space.
", "votes": null, "replies": [] }, { "id": 133366, "author_name": "rossdavidh", "author_url": "", "post_date": "09/01/2016 17:37:29", "content": "Hi John, Thanks a lot for the great analysis. Based on your direction, I just looked at them by initial station number (specifically, first column in train_numeric.csv with a non-null value), and then looked to see where Response was 0 or 1.
\n\nBased on this, it gives the appearance to me of S0 as the highest runner, S12 and S24 as significantly lower, and then so on to lower and lower frequency parts. It looks to my untrained eye like a Pareto distribution, where each step down the frequency chart has a big drop, but there is a long tail of parts with only a few rows each.
\n\nIt's not much of an improvement on your method but it has the advantage of being very easy to implement (for each line in train_numeric, just look for the first station with a non-null result), and it gives an idea of what different parts there are. If it looks wrong to you, of course feel free to point that out. :)
\n\nThanks again for the great analysis.
", "votes": null, "replies": [] }, { "id": 133395, "author_name": "jpmiller", "author_url": "", "post_date": "09/01/2016 20:50:59", "content": "David - These numbers look consistent with the picture I made. More importantly, it seems you have identified a meaningful difference between parts starting at S24 vs S25!
", "votes": null, "replies": [] }, { "id": 133411, "author_name": "rossdavidh", "author_url": "", "post_date": "09/02/2016 00:00:58", "content": "Good point! I think most of these "part types" (if that's what they are) don't have enough data to identify meaningful differences, but those are two of the ones that we have enough data on.
", "votes": null, "replies": [] }, { "id": 133423, "author_name": "azacharia", "author_url": "", "post_date": "09/02/2016 04:16:37", "content": "Great Visualization of the production line..
", "votes": null, "replies": [] }, { "id": 133441, "author_name": "leexiongan", "author_url": "", "post_date": "09/02/2016 08:05:09", "content": "[quote=JohnM;133441]
\n\nThe x-axis represents each station, obviously. The y-axis represents a stack of parts, sorted by similar patterns of NAs.
\n\n[/quote]
\n\nI was wondering how did you separate them so nicely by parts, what kind of algorithm was used ?
\n\nI suppose "parts" here mean a cluster of data points with similar patterns of NAs?
\n\nSorry for clarifying, but some terminology is confusing for someone without any engineering background.
", "votes": null, "replies": [] }, { "id": 133443, "author_name": "zfturbo", "author_url": "", "post_date": "09/02/2016 08:25:19", "content": "leexa - he just sorted the "hashes" lexicographically, that's done all the job.
", "votes": null, "replies": [] }, { "id": 134055, "author_name": "mrugankakarte", "author_url": "", "post_date": "09/04/2016 21:41:23", "content": "Hey John,
\n\nThe map is really nice..What I can see is that stations like 9-10-11 have same product , because where one ends other starts. I think it's like same product is going through 3 different stations (in case of 9-10-11) for various operations which cannot be done on single station. Same is the case with few other stations as well. If we combine these stations and then plot this same map, we should be able to eliminate most of the NAs from the data.\nWhat do you think?
", "votes": null, "replies": [] }, { "id": 134077, "author_name": "jpmiller", "author_url": "", "post_date": "09/05/2016 00:16:43", "content": "Hi Mrugank - as long as you preserve the features and encode the station number into the data, it might work. One way would be to have a column populated with either '9', '10', or '11' depending on which station the part went through, followed by columns with the features from that station. I haven't tried different ways yet to see what works best with various algos.
", "votes": null, "replies": [] }, { "id": 134567, "author_name": "shubhij", "author_url": "", "post_date": "09/07/2016 09:45:34", "content": "Can anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?
", "votes": null, "replies": [] }, { "id": 139516, "author_name": "rshpeley", "author_url": "", "post_date": "10/14/2016 16:18:45", "content": "[quote=shubhijain;134567]
\n\nCan anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?
\n\n[/quote]
\n\nWell, the Response column indicates if a part failed or not. It's the observed value which is compared with the model's predicted value. Both of these are input to the MCC, which you'll find plenty of information about on the forum.
\n\nBy decimal values, I assume you mean the numeric data other than the Response column? These are the feature values under each feature column. The values range between -1 and 1, which shows they've been normalised. We don't know anything else about the data but my guess is that numerical data represent measured part tolerance for a given feature.
", "votes": null, "replies": [] }, { "id": 139520, "author_name": "rshpeley", "author_url": "", "post_date": "10/14/2016 16:40:32", "content": "Of course, you've probably got an answer already after 5 weeks... :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "132141": "Heres a \"production map\" that gives a sense of how parts make their way through the factory. Which machines each part touches, which machines it skips, etc. It gives a rough grouping of parts with similar flow paths. Should be useful to understanding quality!\n\nLater this week I'll post the code so you can check for issues, improve on it, use the data, etc.\n\n![enter image description here][1]\n\n\n [1]: https://s3.amazonaws.com/nonwebstorage/ProductFamilies.png", "132161": "cool", "132164": "Super-cool!", "132176": "I just started looking through the data (mainly the categoricals). It's not clear to me what you are using for \"Product Families\". If it is something obvious, feel free to tell me to just look harder. :-)", "132222": "Here's more of an explanation on the logic and code for looking at parts flow:\r\n\r\n - The goal is to trace each part across the 52 stations and get a feel for the manufacturing flow. Many times, quality is a function of the part type and the specific machine used. \r\n - Measurements are only used here in a binary sense. If the part has any measurement at a station, categorical or numeric, then the part was operated on by that station. Otherwise the part did not go through the station.\r\n - Grouping parts with similar manufacturing patterns will help us identify families of parts and possibly shed light on the nature of the stations.\r\n\r\n\r\nThe VIM package for R does all the heavy lifting for the drawing. By counting and sorting NAs for each part, it gives us a nice count and sort of the non-NAs as well. Some notes on the drawing:\r\n\r\n - The x-axis represents each station, obviously. The y-axis represents a stack of parts, sorted by similar patterns of NAs. \r\n - The height of the blocks represents the relative frequency of the groups. I called the groups \"product families\" which is bit confusing here. \"Parts groups\" may make more sense. \r\n - The bars across the top show the % of NAs for each station.\r\n\r\nWe can see a few things from the picture:\r\n\r\n - There are 3 basic part groups. Those that go through L0(S0-S11), those that go through L0(S12-23), and those that go through L1 & L2. All parts go through L3.\r\n - L0(S0-S11) and L0(S12-S23) have similar patterns so these may be similar part types with a differing characteristic. For instance, regular size parts might go through one group of machines, and a smaller number of jumbo parts go through the other. \r\n - Within each major group there are complementary stations. S09-S11, for instance. These could be 3 stations with the same equipment, or they could have different equipment to deal with part options. Given the equal volume, I think it's more likely in this case they are 3 instances of the same thing.\r\n - Everything runs through L3. Most parts go through the same 5 stations and then there are many stations with low volume. This might be the finishing line, where almost everything gets deburred, sand blasted, painted, etc. and a small number of parts get some sort of custom finish. Or maybe this is an assembly step?\r\n\r\nOf course the possible explanations I've added are hypothetical only. We should be able to use the dates file and add some more insight into the flow. My next step is probably to add the Response variable into this and see if anything stands out before going much further.\r\n\r\nPS - code in R is at https://github.com/JohnM-TX/Kaggle-Bosch . The biggest challenge was handling the reshaping within memory limits. I started in Python-Pandas, but then switched to R and reshaped/aggregated in chunks before feeding it to the VIM function.", "132263": "picture is worth a 1000 words !", "132523": "Hi John,\r\n\r\nAs promised, here's what I got from the train_date.csv file:\r\n\r\nBased on the first 500 (data, non-header) rows of train_date.csv and train_numeric.csv, I was able to deduce four common workflow patterns unique to each row, as represented in the tab 'flow' in the attached excel file. Broadly speaking, there are rows that:\r\n\r\n(1) Initialize at S0, then end at either S37 or S51;\r\n(2) Initialize at S12, then end at either S37 or S38;\r\n(3) Initialize at S24, then end at S37;\r\n(4) Initialize at S25, then end at S37 or S51.\r\n\r\nAs you defined above, if there were null values under a station (and all the features under that station) then a part was not considered to have gone through that station.\r\n\r\nNotice that along the way, parts are sometimes processed in one of 2 or 3 stations then everything returns back to a common station. The basis for this conclusion was a bit arbitrary although at best grounded by the timestamps. You'll notice, for example, for rows that initialize at S0, after passing through S1, the next timestamps that continue from S1 are *either* in S2 or S3. That is, for those rows that started at S0, when S2 are null (or in my case replaced with zero), S3 is activated; and vice-versa, when S3 is null, S2 is activated.\r\n\r\nLooking at the 'time stamps normalized' tab, you can see the from a normalized starting time value of 1, the times at S2/S3 are 1.01 at most. Similarly, S4/S5 show the same either/or/but/not/both pattern, and the normalized time stamps there are 1.02/1.03. Also, when looking at similar rows in train_numeric.csv (which is not included in the excel file for brevity), you'll notice the same either/or pattern with the same number of features for either alternative station.\r\n\r\nBtw, I 'normalized' the time values by doing an element-wise subtraction of ALL non-zero values by the non-zero minimum and added one back. This means that the lowest actual available value in any row is 1. And all the higher time stamps will just show the incremental time units since the 'earliest' timestamp (+1). This helps understand the processing speed more uniformly across all parts. (All remaining zeros in train_numeric were either really zero or blanks, since my extraction replaced blanks/NAs with zero.)\r\n\r\nIn any case, this matches the symmetric flip in your blue bars for S2 and S3 in your picture above; some parts simply go to S2 instead of S3 and vice versa. Same for S4/S5, and S6/S7 alternating. Then S8 receives everything that started in S0. And this is why the flip is across three stations in S9/S10/S11 -- those guys take turns doing something to parts depending on the part. And so on and so forth... the downstream stations were a bit harder to discern, and they seem to take on pretty much everything, but I didn't focus on that too much.\r\n\r\nIn any case, the implication is one of two things:\r\n\r\n(1) either pairs like S4/S5 do something similar and thus, these columns have to be merged. (This would in of itself be a challenge since we do not know if the features in S4 are ordered in the same columns as correspondent features in S5.) Treating features under them as separate features will yield horrible probability density calculations as both positive and null samples will have multiple common zeros across features in both stations if treated separately, which will dull any useful differences between anomalies and non-anomalies; OR\r\n(2) pairs like S4/S5 do something different for each part, and thus, these columns will need to stay separated BUT different probability density calculations for each unique row path will need to be used depending on what has values.\r\n\r\nI was surprised by the relative symmetry the assumptions I made took on when rendered on a flowchart. I wouldn't trust the end station representations that much because again, the data is so sparse and I only used 500 rows, but for the 'earliest' features activated towards the middle, I have a bit more confidence about. Corroborating these results with your graphic was very reassuring though!\r\n\r\nOh lastly, not sure yet if each row(/part) then feeds into another row(part); since the parts all merge somehow eventually to form a machine, I'm sure some rows relate to others somehow, but I can't get a sense of that yet. The fact that a part can have its first 'timestamp' (or feature for that matter) largely either at S00, S12, S24, or S25 may matter, or even the fact that the timestamps do not all start from the same value. In this case the 'raw' non-normalized timestamp may shed some light.\r\n\r\n(PS - the 'chunks' I labeled timestamps as were for my own purposes when I was guessing which features were measured at the same time... it became apparent that it was station-specific, except for S24 and S25; within those stations certain features represent entirely different processes and imply 'sub-stations'.)\r\n\r\nCheers.", "132746": "Hello John, I read your code. Why did you allot the first 60 rows into part1? And with regard to the figure you posted, is that base on the part1 one or the whole data? Thank you.", "132756": "Hi Haom - the figure I posted is based on all training data. As for the 60 rows in part 1, that was just to demonstrate the output with a small piece. Depending on your RAM, you will want many more rows per part to process the whole file.\r\n\r\nI was able to digest 600,000 rows at a time on an Amazon cluster with ~60GB (only $0.12/hr!). The limit was not actually with fread, but with the data.table.melt function in the next step. \r\n\r\nFor \"normal\" RAM, you can set the part size at whatever works and run a for loop. It may take a while, but it will work. And you can process categorical and numeric files separately, and combine them at the very end - the result should be the same.", "132760": "Hi John, thank you for your reply so fast. I see, I tried different parts with different size, they all got the same results! Thank you for what you have done on this project, it really helpful for me to have a comprehensive understanding about the data. The visualization is cool, I never used VIM before, and I learned a lot from your work. Thanks again, looking forward to your fellow up on this. \r\nAnother thing, I think the gc() command that you used is a really good method, it can save a lot of time and space.", "133366": "Hi John, Thanks a lot for the great analysis. Based on your direction, I just looked at them by initial station number (specifically, first column in train_numeric.csv with a non-null value), and then looked to see where Response was 0 or 1.\r\n\r\nBased on this, it gives the appearance to me of S0 as the highest runner, S12 and S24 as significantly lower, and then so on to lower and lower frequency parts. It looks to my untrained eye like a Pareto distribution, where each step down the frequency chart has a big drop, but there is a long tail of parts with only a few rows each.\r\n\r\nIt's not much of an improvement on your method but it has the advantage of being very easy to implement (for each line in train_numeric, just look for the first station with a non-null result), and it gives an idea of what different parts there are. If it looks wrong to you, of course feel free to point that out. :)\r\n\r\nThanks again for the great analysis.", "133395": "David - These numbers look consistent with the picture I made. More importantly, it seems you have identified a meaningful difference between parts starting at S24 vs S25!", "133411": "Good point! I think most of these \"part types\" (if that's what they are) don't have enough data to identify meaningful differences, but those are two of the ones that we have enough data on.", "133423": "Great Visualization of the production line..", "133441": "[quote=JohnM;133441]\r\n\r\n\r\nThe x-axis represents each station, obviously. The y-axis represents a stack of parts, sorted by similar patterns of NAs. \r\n\r\n\r\n[/quote]\r\n\r\nI was wondering how did you separate them so nicely by parts, what kind of algorithm was used ? \r\n\r\nI suppose \"parts\" here mean a *cluster* of data points with similar patterns of NAs?\r\n\r\nSorry for clarifying, but some terminology is confusing for someone without any engineering background.", "133443": "**leexa** - he just sorted the \"hashes\" lexicographically, that's done all the job.", "134055": "Hey John,\r\n\r\nThe map is really nice..What I can see is that stations like 9-10-11 have same product , because where one ends other starts. I think it's like same product is going through 3 different stations (in case of 9-10-11) for various operations which cannot be done on single station. Same is the case with few other stations as well. If we combine these stations and then plot this same map, we should be able to eliminate most of the NAs from the data.\r\nWhat do you think?", "134077": "Hi Mrugank - as long as you preserve the features and encode the station number into the data, it might work. One way would be to have a column populated with either '9', '10', or '11' depending on which station the part went through, followed by columns with the features from that station. I haven't tried different ways yet to see what works best with various algos.", "134567": "Can anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?", "139516": "[quote=shubhijain;134567]\r\n\r\nCan anyone please explain the meaning of response variable in test-numeric data? What do the decimal values indicate? Is it kind of weight being given to a particular feature?\r\n\r\n\r\n[/quote]\r\n\r\nWell, the Response column indicates if a part failed or not. It's the observed value which is compared with the model's predicted value. Both of these are input to the MCC, which you'll find plenty of information about on the forum.\r\n\r\nBy decimal values, I assume you mean the numeric data other than the Response column? These are the feature values under each feature column. The values range between -1 and 1, which shows they've been normalised. We don't know anything else about the data but my guess is that numerical data represent measured part tolerance for a given feature.", "139520": "Of course, you've probably got an answer already after 5 weeks... :)" }, "source": "meta" }