{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e629bf62-5e23-fa27-ffcc-0b8492d2704e"
      },
      "source": [
        "As also noted by amaia [here][1] subsequent IDs are more likely to fail if there predecessor failed as well. My version of the script corrects for the fact that many subsequent IDs are nor present in the train data set as they were split into the test data set.\n",
        "\n",
        "\n",
        "  [1]: https://www.kaggle.com/aamaia/bosch-production-line-performance/failures-in-pairs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "854d2cbe-f044-eb71-88b9-4bf5c610d8e5"
      },
      "outputs": [],
      "source": [
        "options(warn=-1) #surpress warnings\n",
        "options(repr.plot.width=6, repr.plot.height=3)\n",
        "\n",
        "library(data.table)\n",
        "library(dtplyr)\n",
        "library(dplyr)\n",
        "\n",
        "\n",
        "dtNum <- fread(\"../input/train_numeric.csv\",select = c(\"Id\", \"Response\"))\n",
        "\n",
        "desiredLag = 1 #can try out other relationships, e.g. pre-predecesssor\n",
        "\n",
        "dtNum %>%\n",
        "  select(Id,Response) %>%\n",
        "  mutate(nextId = Id+desiredLag) %>%\n",
        "  filter(nextId %in% Id) -> Ids\n",
        "\n",
        "hist(diff(Ids[Ids$Response==1,]$Id),\n",
        "     breaks = 4000,\n",
        "     main=\"Errors in Sequence\", \n",
        "     xlab=\"Distance to last error\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4ec0713a-613a-8d48-14b7-253909e31bda"
      },
      "source": [
        "Ok this histogram looks even more drastic then amaia`s. Can we translate it into probabilities?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "412643cc-a871-a8d4-9bfd-2ba0eee78c19"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "dtNum[Id %in% Ids$Id] -> pre\n",
        "dtNum[Id %in% Ids$nextId] -> succ\n",
        "\n",
        "result = table(pre$Response,succ$Response)\n",
        "print(result)\n",
        "\n",
        "prob1 = result[2,2]/(result[2,1]+result[2,2])\n",
        "prob2 = result[2,1]/(result[1,1]+result[2,1])\n",
        "\n",
        "print(paste(\"Probability of defect after defect\",round(prob1*100,2),\"%\"))\n",
        "print(paste(\"Probability of defect after no defect\",round(prob2*100,2),\"%\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a636ce04-4387-7905-e90e-73e0cda95555"
      },
      "source": [
        "So here we have it: After observing a non-defect the probability of seeing a defect is our baseline of 0.5%, if we saw a defect it increases twenty-fold to 10%! This may not be the silver bullet but it helped me improve quite a bit."
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "R",
      "language": "R",
      "name": "ir"
    },
    "language_info": {
      "codemirror_mode": "r",
      "file_extension": ".r",
      "mimetype": "text/x-r-source",
      "name": "R",
      "pygments_lexer": "r",
      "version": "3.3.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}