{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"43678144-79a2-0ee0-8430-7a6a9324da9d"},"source":"## 1- How big is Basch data? ##\nin this first section we'll do a little inspection of the data file and see what we can learn."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0aefeb14-51b4-0ea4-640d-e503e5322108"},"outputs":[],"source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3ecd7b79-fdd8-de02-8971-49d34884dde8"},"outputs":[],"source":"INPUT_PATH = \"../input/%s.csv\"\n\ndef get_failure_rate(file_name):\n    # will calculate the error rate from data on the file based on response col\n    # Respons: 1 = Failed QC , 0 = Passed QC\n    rows = pd.read_csv(INPUT_PATH % file_name, usecols=[\"Response\"])\n    failure_rate = rows[rows.Response == 1].size / float(rows[rows.Response == 0].size)\n    return failure_rate\n\n\ndef explore_data_size():\n    # explore the size (rows, cols) of each file\n    data_files = ['train_numeric', 'train_date', 'train_categorical', 'test_numeric',\n                  'test_date', 'test_categorical']\n    stats = []\n    for file_name in data_files:\n        cols = pd.read_csv(INPUT_PATH % file_name, nrows=1)\n        rows = pd.read_csv(INPUT_PATH % file_name, usecols=[\"Id\"])\n        stats.append({'File': file_name, 'Rows': rows.shape[0], 'Columns': cols.shape[1]})\n    # convert the result into a DataFrame so we can do plotting.\n    df = pd.DataFrame(stats, columns=[\"File\", \"Rows\", \"Columns\"])\n    failure_rate = get_failure_rate('train_numeric')\n    df[\"Error\"] = 0\n    df.loc[df.File == 'train_numeric', 'Error'] = failure_rate\n    return df\n    \n\nres = explore_data_size()\nres\n\nres.plot(kind='bar', x='File', y='Columns', title=\"Columns per file\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"78a60393-af27-2d25-342b-fa664cb4e089"},"source":"From the results above we can say:\n\n 1. We are dealing with more 1 million record per file, so we just need to be careful not read the whole file at once, especially if you have limited memory.\n 2. The number of columns per file is also huge 1000 to 2000 per file. Each column represents a feature the corresponds to some measurement on a station or a machine.\n 3. Each file contains the \"Id\" column which represent the unique identifier of the product unit that we are tracking\n 4. Finally we notice that the error rate is about 0.5%, so it will be challenging to try to predict such small number from this huge data set. But we'll see...\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"59e7481b-f0cc-c504-5c6f-6b3b42918c0d"},"source":"## 2- Based on the features, how many stations, lines, failure rate per station? ##\nin this section we'll take a look at the feature or the columns and try to see how many station and line of production we have? how many features does each station have? how many products were processed in each station? and what's the failure rate per station?\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5f51c11c-51d0-c882-a887-accd10f5bd73"},"outputs":[],"source":"def get_features(feature_list):\n    line_features = {}\n    station_features = {}\n    lines = set([item.split('_')[0] for item in feature_list])\n    stations = set([item.split('_')[1] for item in feature_list])\n\n    for l in lines:\n        line_features[l] = [item for item in feature_list if '%s_' % l in item]\n\n    for s in stations:\n        station_features[s] = [item for item in feature_list if '%s_' % s in item]\n\n    return (line_features, station_features)\n\ndef explore_features(data_set_name):\n    # how many units processed in each station\n    # also percentage of failed/passed QS in each station\n    features = pd.read_csv(\"../input/%s\" % data_set_name, nrows=1).drop([\"Response\", \"Id\"], axis=1).columns.values\n    line_features, station_features = get_features(features)\n    # create a dataframe cols: station, features_count\n    sdf = pd.DataFrame(list({int(key[1:]): len(station_features[key]) for\n                             key in station_features.keys()}.items()),\n                       columns=[\"Station\", \"FeatureCount\"])\n    ldf = pd.DataFrame(list({int(key[1:]): len(line_features[key]) for\n                             key in line_features.keys()}.items()),\n                       columns=[\"Line\", \"FeatureCount\"])\n    \n    stations_plot = sdf.plot(x=\"Station\", y=\"FeatureCount\", kind=\"bar\", title=\"Features by Station\")\n\nexplore_features(\"train_numeric.csv\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"0ae0f848-1e78-1977-83c1-075ffca604dd"},"source":"Wow! most stations has less than 20 features, except station 24, 25 which has more than 200 features each! those must be the biggest machines in this chocolate mousse factory.\n\nSo let head to the next section where we will try to calculate the number of units processed by each station (machine) and also what's the failure rate for each station."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"584b1257-a891-01e3-0c9f-b895444120d2"},"outputs":[],"source":"\ndef explore_product_by_station(data_set_name):\n    # need to see how many product per station\n    features = pd.read_csv(\"../input/%s\" % data_set_name, nrows=1).drop([\"Response\", \"Id\"], axis=1).columns.values\n    line_features, station_features = get_features(features)\n    result = []\n    for station in station_features:\n        station_data = pd.read_csv(\n            \"../input/%s\" % data_set_name,\n            usecols=station_features[station] + ['Id', 'Response']).dropna(how=\"any\")\n        result.append([int(station[1:]), station_data.shape[0],\n                       station_data[station_data[\"Response\"] == 1].shape[0],\n                       station_data[station_data[\"Response\"] == 0].shape[0]])\n    df = pd.DataFrame(\n        result, columns=[\"Station\", \"Count\", \"Failed\", \"Passed\"]).sort_values(by=[\"Station\"])\n    # calculate the error rate for each station\n    df[\"Error\"] = df.apply(lambda row: float(row[\"Failed\"]) / row[\"Passed\"], axis=1)\n    \n    # draw the stacked bar plot tot product/ station\n    my_plot = df[[\"Station\", \"Failed\", \"Passed\"]].plot(kind=\"bar\", stacked=True, x=\"Station\",\n        title=\"Products by station\")\n    my_plot.set_xlabel(\"Stations\")\n    my_plot.set_ylabel(\"Record Count\")\n    \n    # draw error rate per station\n    my_plot = df[[\"Station\", \"Error\"]].plot(\n        kind=\"bar\", x=\"Station\",\n        title=\"Error by station\")\n    my_plot.set_xlabel(\"Stations\")\n    my_plot.set_ylabel(\"Error %\")\n        \nexplore_product_by_station(\"train_numeric.csv\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"a506837c-bcd6-5287-4e5a-f2d83e893d35"},"source":"## So what can we say by just looking at the 3 plots above: ##\n\n 1. Station 24, 25 has by far the most number of features, more than 200 feature per stations! at the same time most of the stations has less than 20 features. And worth noticing that station 32 has only one feature.\n\n 2. Not all stations have the same number of records or product! Does it mean that each product doesn't go through each station? could it be that we have different types of products and each one get processed through different station? Based on my understanding on manufacturing if let's say we are producing Orange Juice, then each juice box will go through the same process and the same machines until the juice is ready. Can someone tell me why the numbers are not equal for all the stations?\n\n 3. The suspicious station 32 has really a minimal number of records. So it doesn't process a lot of products.\n\n 4. Also worth noting that station 32 has by far most of QC failures almost 0.5%. At the same time all of the other stations has less than 0.1% as a failure rate! But wait S32 doesn't process a lot of records, so 0.5% of low number of records is minimal. However if each other station has a failure rate of 0.1% of the total production quantity then this number will add up eventually.\n\n\n"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}