{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"e629bf62-5e23-fa27-ffcc-0b8492d2704e"},"source":"Failures after Failures\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"854d2cbe-f044-eb71-88b9-4bf5c610d8e5"},"outputs":[],"source":"#options(warn=-1) #surpress warnings\n#options(repr.plot.width=6, repr.plot.height=3)\n\nlibrary(data.table)\nlibrary(dtplyr)\nlibrary(dplyr)\n\n\ndtNum <- fread(\"../input/train_numeric.csv\",select = c(\"Id\", \"Response\"))\n\ndesiredLag = 1 #can try out other relationships, e.g. pre-predecesssor\ndtNum <- select(dtNum, Id, Response)\ndtNum$nextId = dt$Id + desiredLag\n\n\n\nselect(dtNum, Id,Response) %>%\n  mutate(nextId = Id+desiredLag) %>%\n  filter(nextId %in% Id) -> Ids\n\n\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"4ec0713a-613a-8d48-14b7-253909e31bda"},"source":"Ok this histogram looks even more drastic then amaia`s. Can we translate it into probabilities?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"412643cc-a871-a8d4-9bfd-2ba0eee78c19"},"outputs":[],"source":"\n\ndtNum[Id %in% Ids$Id] -> pre\ndtNum[Id %in% Ids$nextId] -> succ\n\n\nresult = table(pre$Response,succ$Response)\nprint(result)\n\nprob1 = result[2,2]/(result[2,1]+result[2,2])\nprob2 = result[2,1]/(result[1,1]+result[2,1])\n\nprint(paste(\"Probability of defect after defect\",round(prob1*100,2),\"%\"))\nprint(paste(\"Probability of defect after no defect\",round(prob2*100,2),\"%\"))"},{"cell_type":"markdown","metadata":{"_cell_guid":"a636ce04-4387-7905-e90e-73e0cda95555"},"source":"So here we have it: After observing a non-defect the probability of seeing a defect is our baseline of 0.5%, if we saw a defect it increases twenty-fold to 10%! This may not be the silver bullet but it helped me improve quite a bit."}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"codemirror_mode":"r","file_extension":".r","mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.3.2"}},"nbformat":4,"nbformat_minor":0}