{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c3f40457-d99c-e724-125d-8cc4e57ee172"
      },
      "source": [
        "## Error/Failure Rate EDA and Feature Engineering Ideas\n",
        "\n",
        "I wanted to know if certain lines or stations were correlated to higher error rates. While exploring, I found this code is also somewhat useful for feature engineering, such as taking the min/max values at each station.\n",
        "\n",
        "### Station 32\n",
        "\n",
        "A total of 24,543 samples run through station 32, with a 4.7% error rate, compared the mean error rate 0.6%. It only has one feature, L3_S32_F3850, which has come up on other Kernels ranking feature importance.\n",
        "\n",
        "Coincidentally (or maybe not), Station 31 is associated with the lowest failure rate at 0.27%"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "af527e99-8ad8-76d4-f6ea-67e9030845b4"
      },
      "outputs": [],
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import seaborn as sns\n",
        "sns.set_color_codes(\"muted\")\n",
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline\n",
        "\n",
        "numeric = \"../input/train_numeric.csv\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0c0fce6c-8cfe-3656-0e74-794819b2952c"
      },
      "source": [
        "## Sorting out Lines, Stations, and Features\n",
        "The function below creates a two dicts that isolate all the features that belong to specific Lines and Stations. This makes it easy to slice the data on a Line-by-Line or Station-by-Station basis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "04d7fd23-272c-65ef-87e6-9b1cca612fad"
      },
      "outputs": [],
      "source": [
        "features = pd.read_csv(numeric, nrows=1).drop(['Response', 'Id'], axis=1).columns.values\n",
        "\n",
        "def orgainize(features):\n",
        "    line_features = {}\n",
        "    station_features = {}\n",
        "    lines = set([f.split('_')[0] for f in features])\n",
        "    stations = set([f.split('_')[1] for f in features])\n",
        "    \n",
        "    for l in lines:\n",
        "        line_features[l] = [f for f in features if l+'_' in f]\n",
        "        \n",
        "    for s in stations:\n",
        "        station_features[s] = [f for f in features if s+'_' in f]\n",
        "        \n",
        "            \n",
        "    return line_features, station_features\n",
        "\n",
        "line_features, station_features = orgainize(features)\n",
        "\n",
        "print(\"Features in Station 32: {}\".format( station_features['S32'] ))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "651a96e0-2013-87e4-859e-aa79734b6948"
      },
      "source": [
        "## Exploring Stations\n",
        "\n",
        "- Features - Total features in the Station\n",
        "- Samples - Total samples with measured values (non-NaN) >= 1 in the Station\n",
        "- Error rate - (Response==1) rate for samples in the Station.\n",
        "\n",
        "Note* Samples run through multiple stations/lines, which was not taken into account, but may be important.\n",
        "\n",
        "Note** A small percentage of samples have no data and were dropped."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fc133c67-36e7-7291-8878-de0b226975f4"
      },
      "outputs": [],
      "source": [
        "station_error = []\n",
        "for s in station_features:\n",
        "    cols = ['Id', 'Response']\n",
        "    cols.extend(station_features[s])\n",
        "    df = pd.read_csv(numeric, usecols=cols).dropna(subset=station_features[s], how='all')\n",
        "    error_rate = df[df.Response == 1].size / float(df[df.Response == 0].size)\n",
        "    station_error.append([df.shape[1]-2, df.shape[0], error_rate]) \n",
        "    \n",
        "station_data = pd.DataFrame(station_error, \n",
        "                         columns=['Features', 'Samples', 'Error_Rate'], \n",
        "                         index=station_features).sort_index()\n",
        "station_data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "db0a665b-a57b-6830-1b1a-f9394d1413f2"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(8, 20))\n",
        "sns.barplot(x='Error_Rate', y=station_data.index.values, data=station_data, color=\"red\")\n",
        "plt.title('Error Rate between Production Stations')\n",
        "\n",
        "plt.xlabel('Station Error Rate')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "552797b0-94f9-b659-408e-bc92796bca8f"
      },
      "source": [
        "## Quick Feature Engineering Example\n",
        "Here's an example of how to use the station dict can be used to create new features or just reduce the size of the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "36c392b7-5484-d266-cc4d-bedc4dd766d9"
      },
      "outputs": [],
      "source": [
        "data = pd.read_csv(numeric, nrows=100)\n",
        "\n",
        "def make_features(df):\n",
        "    new_features = pd.DataFrame({})\n",
        "    for s in station_features.keys():\n",
        "        station_data = df[station_features[s]]\n",
        "        col = s+'_max'\n",
        "        new_features[col] = station_data.max(axis=1).fillna(-1.)\n",
        "        col = s+'_min'\n",
        "        new_features[col] = station_data.min(axis=1).fillna(-1.)\n",
        "    return new_features\n",
        "\n",
        "data = make_features(data)\n",
        "data.head()"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}