{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5f4df752-cefe-a0bc-3799-db51588d571b"
      },
      "source": [
        "I had two questions about the `timestamp` info contained in `train_date.csv`:\n",
        "\n",
        "1. Are there specific times when the number of bad parts suddenly goes up ? (this as the result of a machine being down for example)\n",
        "2. Do bad parts have a different timing in their manufacturing process ?  (it could be a diversion from the standard path, a rework, ...)\n",
        "\n",
        "\n",
        "The answer seems to be NO to both questions !\n",
        "---------------------------------------------------\n",
        "\n",
        "\n",
        "The following code is an attempt at addressing these 2 questions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "93b4d171-9253-dab9-405b-cd5a2d8f04d1"
      },
      "outputs": [],
      "source": [
        "library(data.table)\n",
        "library(ggplot2)\n",
        "library(dplyr)\n",
        "library(tidyr)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "62b02fbf-2b1f-57ae-738b-d48981e86eb7"
      },
      "source": [
        "The size of `train_date.csv` requires reading it by pieces \n",
        "\n",
        "The following defines the parameters to cut the file into batches"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "82fe27d3-254f-7146-ba4f-1b61d22c3803"
      },
      "outputs": [],
      "source": [
        "n_rows = 50000  # batch size for reading file\n",
        "L <- nrow(fread(\"../input/train_date.csv\", select=1, showProgress = FALSE)) # reading 1 column only to get nrow\n",
        "rest <- L %% n_rows\n",
        "K <- round(L/n_rows - .5 )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b0c1367a-b38c-0396-4f35-7d3e55e73f8a"
      },
      "source": [
        "Reading the file, together with two colums of the `train_numeric` file where the `Response` info is stored"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8e160931-ee63-1ec7-b38e-ecebb55398b6"
      },
      "outputs": [],
      "source": [
        "train_dat <- fread('../input/train_date.csv', nrows= rest, showProgress = FALSE)\n",
        "noms_td <- names(train_dat)\n",
        "IdResp <- fread('../input/train_numeric.csv', select = c(1,970), nrows= rest, showProgress = FALSE)\n",
        "noms_IR <- names(IdResp)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e074d1ab-5e21-0ad0-5e0d-c4ae56669232"
      },
      "source": [
        "The `Response` info is now added to the `timestamp` data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e76cc74d-24d3-6be4-3ec9-8b607b724bf3"
      },
      "outputs": [],
      "source": [
        "train_dat <- left_join(train_dat, IdResp)\n",
        "rm(IdResp)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b7f079fe-e83f-8d1a-deb4-c216dd5accf2"
      },
      "source": [
        "I use `tidyr::gather` to restructure the `train_dshort` file where the `timestamp` info is now stored!\n",
        "\n",
        "> Note that with 8G RAM, I could not use the na.rm option in the `gather` function unless I was taking much smaller batches..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "16905c4e-9c21-a131-f3ad-76162deb5056"
      },
      "outputs": [],
      "source": [
        "train_dshort <- gather(train_dat, feature, timestamp, -Id, - Response)\n",
        "rm(train_dat)\n",
        "train_dshort <- as.data.table(train_dshort)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b0f1067b-9242-e7b1-3ed0-fda484c3b23d"
      },
      "source": [
        "The NAs are taken out now !  We can see the first rows of the resulting file `train_dshort`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "01c35e21-27eb-e63b-4d4c-946ab85c76da"
      },
      "outputs": [],
      "source": [
        "train_dshort <- train_dshort[!is.na(timestamp),]\n",
        "head(train_dshort)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1a3aadbc-d272-274f-9c27-a10656d611eb"
      },
      "source": [
        "The same commands can be repeated to read the whole `train_date` file\n",
        "\n",
        "> Note: to read the whole file could take about 30mn.  In order to test the code and see a quick result, replace (k in 1:K) by (1 in 1:1) in the second line of the following code... You will get about 170k rows to play with !"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4a8f4fb2-3fba-70f0-3340-6648efff6066"
      },
      "outputs": [],
      "source": [
        "for (k in 1:1){\n",
        "  toskip <- rest  + n_rows * (k-1)\n",
        "  train_dat <- fread('../input/train_date.csv', skip= toskip, nrows= n_rows,\n",
        "                     header= FALSE, showProgress = FALSE)\n",
        "  colnames(train_dat) <- noms_td\n",
        "  IdResp <- fread('../input/train_numeric.csv', select = c(1,970), \n",
        "                  skip= toskip, nrows= n_rows, header = FALSE,\n",
        "                  showProgress = FALSE)\n",
        "  colnames(IdResp) <- noms_IR\n",
        "  \n",
        "  # add Response info to train_dat\n",
        "  train_dat <- left_join(train_dat, IdResp)\n",
        "  rm(IdResp)\n",
        "    \n",
        "  # restructuring of the file struture to allow different representations\n",
        "  # train_dshort <- gather(train_dat, feature, measure, -Id, na.rm= TRUE) # works only for small batches a few 10000's\n",
        "  train_dtemp<- gather(train_dat, feature, timestamp, -Id, - Response)\n",
        "  rm(train_dat)\n",
        "  train_dtemp <- as.data.table(train_dtemp)\n",
        "  \n",
        "  # take out the measure = NA lines\n",
        "  train_dtemp <- train_dtemp[!is.na(timestamp),]\n",
        "  \n",
        "  train_dshort <- rbind (train_dshort, train_dtemp)\n",
        "  rm(train_dtemp)\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ffb678b2-dfc9-6e2a-cbe6-ee6efbf6299c"
      },
      "source": [
        "We can now add a number of additional variables to help the graphical representation of the data\n",
        "\n",
        " 1. a  `pc` variable to identify the point of control number which are buried in the Station number... i.e.  I take the '103' out of the 'L0_SO_D103' ! \n",
        " 2. The `start_time` variable records the starting `timestamp` of each part manufacturing process\n",
        " 3. The `time0` variable will allow the comparison of parts manufacturing processes, as if they were starting at the same instant !"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2bc2d366-bc1e-2262-acc3-853266f8e66c"
      },
      "outputs": [],
      "source": [
        "train_dshort[,cp := as.integer(gsub(\"[A-Z]\\\\d+_[A-Z]\\\\d+_[A-Z]\", \"\", feature))]\n",
        "train_dshort[,start_time := min(timestamp), by = Id]\n",
        "train_dshort[,time0 := (timestamp - start_time), by=Id]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3137ee34-c8a5-0e64-d0bf-bec6ddd20237"
      },
      "source": [
        "The graphs can now be easily produced:\n",
        "\n",
        "The following shows the timing of the manufacturing process of about 10 parts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "53d5477e-68b2-a75b-9a0d-a8f27e4f41e6"
      },
      "outputs": [],
      "source": [
        "ggplot(train_dshort[Id %in% 1:20], aes(y=cp, x = timestamp, col = factor(Id)))+ geom_line() + \n",
        "  scale_color_discrete(name=\"Part Id\") + ylab(\"Control points\") +\n",
        "  ggtitle(\"Manufacturing process timing\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b49dad9d-7d55-592a-1e1b-1a669657c5a2"
      },
      "source": [
        "The sames graph for about 300 part, 3 of which anomalies, i.e. Response ==1 (only 15% of the whole file though in this particular graph ! )\n",
        "\n",
        "\n",
        "  [1]: http://C:%5CUsers%5Ccdamp_000%5CDocuments%5CKaggle%20BIG%20datasets%5CBosh,_fail_rate.jpeg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "31805089-3dd5-3f65-a1c4-d737ac7b1fa6"
      },
      "outputs": [],
      "source": [
        "ggplot(train_dshort[Id %in% 1050:1351], aes(y=cp, x = timestamp, col = factor(Response), group= Id))+ \n",
        "  geom_line(alpha=.4)+ \n",
        "  scale_color_discrete(name=\"Response\")+ ylab(\"Control points\") +\n",
        "  ggtitle(\"Manufacturing process timing\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2d2cd4b4-2cb7-f981-5d4d-a34a4f074151"
      },
      "source": [
        "Same plot again, but with only  the anomalies (Response == 1) - only 15% of the whole file though in this particular graph ! -"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8e9a9b79-c448-a03a-dcee-ce4398ff7341"
      },
      "outputs": [],
      "source": [
        "ggplot(train_dshort[Response==1], aes(y=cp, x = timestamp, group= Id))+ \n",
        "  geom_line(alpha=.6, col=\"red\") + scale_color_discrete(labels=\"\")+ ylab(\"Control points\") +\n",
        "  ggtitle(\"Anomalies Manufacturing process timing\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fc81a33f-bdd9-6fbe-71d1-309de00a4a85"
      },
      "source": [
        "Difficult to see anything special about the way the manufacturing process of anomalies unfolds !  \n",
        "\n",
        "> **The answer to the first question would be a NO !**\n",
        "------------------------------------------------------\n",
        "\n",
        "To address question 2, it could be interesting to look at the density distribution of the starting time for good parts, and for bad parts (anomalies)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "28f9b514-0eab-4a93-2326-4822c7bcec27"
      },
      "outputs": [],
      "source": [
        "ggplot(train_dshort[,.(start_time, Response), by = Id], aes(x= start_time, fill= factor(Response)))+ \n",
        "  geom_histogram(aes(y=..density..), position= 'dodge', bins= 50 )+ scale_fill_discrete(name = \"Response\")+ \n",
        "  ggtitle(\"Start of Manufacturing process density\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9e901eae-671f-71ae-43b7-257e0bc4432b"
      },
      "source": [
        "The same information base gives a glimpse at how the percentage of bad parts manufactured evolved with time `start_time`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "22d88fe6-e6ea-9b8f-0974-21a5fb6151c5"
      },
      "outputs": [],
      "source": [
        "  ggplot(train_dshort[,.(start_time, Response), by = Id], aes(x= start_time, fill= factor(!Response)))+ \n",
        "  geom_histogram(position= 'fill', bins= 50 )+ \n",
        "  scale_fill_discrete(name = \"Response\", labels=c(\"1\", \"0\")) +  \n",
        "  coord_cartesian(xlim= c(0,1750), ylim=c(0,.02)) + ylab('failure rate')\n",
        "  ggtitle(\"Evolution of failure rate\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5887a5df-29ed-1f75-8203-5871335ed7e2"
      },
      "source": [
        "Until now, we can appreciate when the manufacturing process of a given part is initated, but not how the detailed time of the manufacturing process unfolds.  The `time0` variable helps us here..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c9f40bec-ed4e-cd40-47e9-ac95004e832f"
      },
      "outputs": [],
      "source": [
        "ggplot(train_dshort[Id %in% 1:100], aes(y=cp, x = time0, col = factor(Id)))+ geom_line() +  \n",
        "  scale_color_discrete(name=\"Part Id\") +\n",
        "  ylab(\"Control points\") +\n",
        "  ggtitle(\"Timing of manufacturing process comparison - good parts\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4bab3d42-a4d1-7610-cca0-62579dfa69cd"
      },
      "source": [
        "More specifically, for the bad parts ..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6c649d70-d324-c771-bf2b-c911571c122b"
      },
      "outputs": [],
      "source": [
        "ggplot(train_dshort[Response==1], aes(y=cp, x = time0, group = Id, col= factor(Response)))+ geom_line()+  \n",
        "  scale_color_discrete(name=\"Response\") + ylab(\"Control points\") +\n",
        "  ggtitle(\"Timing of manufacturing process comparison - all bad parts\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "072a2c4d-8bbe-562d-5e2d-935f0bb5cca4"
      },
      "source": [
        "Again no big difference in the way timing goes when a bad part is manufactured !\n",
        "\n",
        "> **The answer to the second question looks like a NO !**\n",
        "-------------------------------------------------------------------"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "R",
      "language": "R",
      "name": "ir"
    },
    "language_info": {
      "codemirror_mode": "r",
      "file_extension": ".r",
      "mimetype": "text/x-r-source",
      "name": "R",
      "pygments_lexer": "r",
      "version": "3.3.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}