{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"5f4df752-cefe-a0bc-3799-db51588d571b"},"source":"I had two questions about the `timestamp` info contained in `train_date.csv`:\n\n1. Are there specific times when the number of bad parts suddenly goes up ? (this as the result of a machine being down for example)\n2. Do bad parts have a different timing in their manufacturing process ?  (it could be a diversion from the standard path, a rework, ...)\n\n\nThe answer seems to be NO to both questions !\n---------------------------------------------------\n\n\nThe following code is an attempt at addressing these 2 questions:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"93b4d171-9253-dab9-405b-cd5a2d8f04d1"},"outputs":[],"source":"library(data.table)\nlibrary(ggplot2)\nlibrary(dplyr)\nlibrary(tidyr)"},{"cell_type":"markdown","metadata":{"_cell_guid":"62b02fbf-2b1f-57ae-738b-d48981e86eb7"},"source":"The size of `train_date.csv` requires reading it by pieces \n\nThe following defines the parameters to cut the file into batches"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"82fe27d3-254f-7146-ba4f-1b61d22c3803"},"outputs":[],"source":"n_rows = 50000  # batch size for reading file\nL <- nrow(fread(\"../input/train_date.csv\", select=1, showProgress = FALSE)) # reading 1 column only to get nrow\nrest <- L %% n_rows\nK <- round(L/n_rows - .5 )"},{"cell_type":"markdown","metadata":{"_cell_guid":"b0c1367a-b38c-0396-4f35-7d3e55e73f8a"},"source":"Reading the file, together with two colums of the `train_numeric` file where the `Response` info is stored"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8e160931-ee63-1ec7-b38e-ecebb55398b6"},"outputs":[],"source":"train_dat <- fread('../input/train_date.csv', nrows= rest, showProgress = FALSE)\nnoms_td <- names(train_dat)\nIdResp <- fread('../input/train_numeric.csv', select = c(1,970), nrows= rest, showProgress = FALSE)\nnoms_IR <- names(IdResp)"},{"cell_type":"markdown","metadata":{"_cell_guid":"e074d1ab-5e21-0ad0-5e0d-c4ae56669232"},"source":"The `Response` info is now added to the `timestamp` data"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e76cc74d-24d3-6be4-3ec9-8b607b724bf3"},"outputs":[],"source":"train_dat <- left_join(train_dat, IdResp)\nrm(IdResp)"},{"cell_type":"markdown","metadata":{"_cell_guid":"b7f079fe-e83f-8d1a-deb4-c216dd5accf2"},"source":"I use `tidyr::gather` to restructure the `train_dshort` file where the `timestamp` info is now stored!\n\n> Note that with 8G RAM, I could not use the na.rm option in the `gather` function unless I was taking much smaller batches..."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"16905c4e-9c21-a131-f3ad-76162deb5056"},"outputs":[],"source":"train_dshort <- gather(train_dat, feature, timestamp, -Id, - Response)\nrm(train_dat)\ntrain_dshort <- as.data.table(train_dshort)"},{"cell_type":"markdown","metadata":{"_cell_guid":"b0f1067b-9242-e7b1-3ed0-fda484c3b23d"},"source":"The NAs are taken out now !  We can see the first rows of the resulting file `train_dshort`"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"01c35e21-27eb-e63b-4d4c-946ab85c76da"},"outputs":[],"source":"train_dshort <- train_dshort[!is.na(timestamp),]\nhead(train_dshort)"},{"cell_type":"markdown","metadata":{"_cell_guid":"1a3aadbc-d272-274f-9c27-a10656d611eb"},"source":"The same commands can be repeated to read the whole `train_date` file\n\n> Note: to read the whole file could take about 30mn.  In order to test the code and see a quick result, replace (k in 1:K) by (1 in 1:1) in the second line of the following code... You will get about 170k rows to play with !"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4a8f4fb2-3fba-70f0-3340-6648efff6066"},"outputs":[],"source":"for (k in 1:1){\n  toskip <- rest  + n_rows * (k-1)\n  train_dat <- fread('../input/train_date.csv', skip= toskip, nrows= n_rows,\n                     header= FALSE, showProgress = FALSE)\n  colnames(train_dat) <- noms_td\n  IdResp <- fread('../input/train_numeric.csv', select = c(1,970), \n                  skip= toskip, nrows= n_rows, header = FALSE,\n                  showProgress = FALSE)\n  colnames(IdResp) <- noms_IR\n  \n  # add Response info to train_dat\n  train_dat <- left_join(train_dat, IdResp)\n  rm(IdResp)\n    \n  # restructuring of the file struture to allow different representations\n  # train_dshort <- gather(train_dat, feature, measure, -Id, na.rm= TRUE) # works only for small batches a few 10000's\n  train_dtemp<- gather(train_dat, feature, timestamp, -Id, - Response)\n  rm(train_dat)\n  train_dtemp <- as.data.table(train_dtemp)\n  \n  # take out the measure = NA lines\n  train_dtemp <- train_dtemp[!is.na(timestamp),]\n  \n  train_dshort <- rbind (train_dshort, train_dtemp)\n  rm(train_dtemp)\n}"},{"cell_type":"markdown","metadata":{"_cell_guid":"ffb678b2-dfc9-6e2a-cbe6-ee6efbf6299c"},"source":"We can now add a number of additional variables to help the graphical representation of the data\n\n 1. a  `pc` variable to identify the point of control number which are buried in the Station number... i.e.  I take the '103' out of the 'L0_SO_D103' ! \n 2. The `start_time` variable records the starting `timestamp` of each part manufacturing process\n 3. The `time0` variable will allow the comparison of parts manufacturing processes, as if they were starting at the same instant !"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"2bc2d366-bc1e-2262-acc3-853266f8e66c"},"outputs":[],"source":"train_dshort[,cp := as.integer(gsub(\"[A-Z]\\\\d+_[A-Z]\\\\d+_[A-Z]\", \"\", feature))]\ntrain_dshort[,start_time := min(timestamp), by = Id]\ntrain_dshort[,time0 := (timestamp - start_time), by=Id]"},{"cell_type":"markdown","metadata":{"_cell_guid":"3137ee34-c8a5-0e64-d0bf-bec6ddd20237"},"source":"The graphs can now be easily produced:\n\nThe following shows the timing of the manufacturing process of about 10 parts"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"53d5477e-68b2-a75b-9a0d-a8f27e4f41e6"},"outputs":[],"source":"ggplot(train_dshort[Id %in% 1:20], aes(y=cp, x = timestamp, col = factor(Id)))+ geom_line() + \n  scale_color_discrete(name=\"Part Id\") + ylab(\"Control points\") +\n  ggtitle(\"Manufacturing process timing\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"b49dad9d-7d55-592a-1e1b-1a669657c5a2"},"source":"The sames graph for about 300 part, 3 of which anomalies, i.e. Response ==1 (only 15% of the whole file though in this particular graph ! )\n\n\n  [1]: http://C:%5CUsers%5Ccdamp_000%5CDocuments%5CKaggle%20BIG%20datasets%5CBosh,_fail_rate.jpeg"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"31805089-3dd5-3f65-a1c4-d737ac7b1fa6"},"outputs":[],"source":"ggplot(train_dshort[Id %in% 1050:1351], aes(y=cp, x = timestamp, col = factor(Response), group= Id))+ \n  geom_line(alpha=.4)+ \n  scale_color_discrete(name=\"Response\")+ ylab(\"Control points\") +\n  ggtitle(\"Manufacturing process timing\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"2d2cd4b4-2cb7-f981-5d4d-a34a4f074151"},"source":"Same plot again, but with only  the anomalies (Response == 1) - only 15% of the whole file though in this particular graph ! -"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8e9a9b79-c448-a03a-dcee-ce4398ff7341"},"outputs":[],"source":"ggplot(train_dshort[Response==1], aes(y=cp, x = timestamp, group= Id))+ \n  geom_line(alpha=.6, col=\"red\") + scale_color_discrete(labels=\"\")+ ylab(\"Control points\") +\n  ggtitle(\"Anomalies Manufacturing process timing\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"fc81a33f-bdd9-6fbe-71d1-309de00a4a85"},"source":"Difficult to see anything special about the way the manufacturing process of anomalies unfolds !  \n\n> **The answer to the first question would be a NO !**\n------------------------------------------------------\n\nTo address question 2, it could be interesting to look at the density distribution of the starting time for good parts, and for bad parts (anomalies)."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"28f9b514-0eab-4a93-2326-4822c7bcec27"},"outputs":[],"source":"ggplot(train_dshort[,.(start_time, Response), by = Id], aes(x= start_time, fill= factor(Response)))+ \n  geom_histogram(aes(y=..density..), position= 'dodge', bins= 50 )+ scale_fill_discrete(name = \"Response\")+ \n  ggtitle(\"Start of Manufacturing process density\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"9e901eae-671f-71ae-43b7-257e0bc4432b"},"source":"The same information base gives a glimpse at how the percentage of bad parts manufactured evolved with time `start_time`"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"22d88fe6-e6ea-9b8f-0974-21a5fb6151c5"},"outputs":[],"source":"  ggplot(train_dshort[,.(start_time, Response), by = Id], aes(x= start_time, fill= factor(!Response)))+ \n  geom_histogram(position= 'fill', bins= 50 )+ \n  scale_fill_discrete(name = \"Response\", labels=c(\"1\", \"0\")) +  \n  coord_cartesian(xlim= c(0,1750), ylim=c(0,.02)) + ylab('failure rate')\n  ggtitle(\"Evolution of failure rate\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"5887a5df-29ed-1f75-8203-5871335ed7e2"},"source":"Until now, we can appreciate when the manufacturing process of a given part is initated, but not how the detailed time of the manufacturing process unfolds.  The `time0` variable helps us here..."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c9f40bec-ed4e-cd40-47e9-ac95004e832f"},"outputs":[],"source":"ggplot(train_dshort[Id %in% 1:100], aes(y=cp, x = time0, col = factor(Id)))+ geom_line() +  \n  scale_color_discrete(name=\"Part Id\") +\n  ylab(\"Control points\") +\n  ggtitle(\"Timing of manufacturing process comparison - good parts\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"4bab3d42-a4d1-7610-cca0-62579dfa69cd"},"source":"More specifically, for the bad parts ..."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"6c649d70-d324-c771-bf2b-c911571c122b"},"outputs":[],"source":"ggplot(train_dshort[Response==1], aes(y=cp, x = time0, group = Id, col= factor(Response)))+ geom_line()+  \n  scale_color_discrete(name=\"Response\") + ylab(\"Control points\") +\n  ggtitle(\"Timing of manufacturing process comparison - all bad parts\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"072a2c4d-8bbe-562d-5e2d-935f0bb5cca4"},"source":"Again no big difference in the way timing goes when a bad part is manufactured !\n\n> **The answer to the second question looks like a NO !**\n-------------------------------------------------------------------"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"codemirror_mode":"r","file_extension":".r","mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.3.1"}},"nbformat":4,"nbformat_minor":0}