{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"43678144-79a2-0ee0-8430-7a6a9324da9d"},"source":"## 1- How big is Basch data? ##\nin this first section we'll do a little inspection of the data file and see what we can learn."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0aefeb14-51b4-0ea4-640d-e503e5322108"},"outputs":[],"source":"import matplotlib\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\nINPUT_PATH = \"../input/%s.csv\"\nmatplotlib.style.use('ggplot')"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3ecd7b79-fdd8-de02-8971-49d34884dde8"},"outputs":[],"source":"def get_failure_rate(file_name):\n    # will calculate the error rate from data on the file based on response col\n    # Respons: 1 = Failed QC , 0 = Passed QC\n    rows = pd.read_csv(INPUT_PATH % file_name, usecols=[\"Response\"])\n    failure_rate = rows[rows.Response == 1].size / float(rows[rows.Response == 0].size)\n    return failure_rate\n\n\ndef explore_data_size():\n    # explore the size (rows, cols) of each file\n    data_files = ['train_numeric', 'train_date', 'train_categorical', 'test_numeric',\n                  'test_date', 'test_categorical']\n    stats = []\n    for file_name in data_files:\n        cols = pd.read_csv(INPUT_PATH % file_name, nrows=1)\n        rows = pd.read_csv(INPUT_PATH % file_name, usecols=[\"Id\"])\n        stats.append({'File': file_name, 'Rows': rows.shape[0], 'Columns': cols.shape[1]})\n    # convert the result into a DataFrame so we can do plotting.\n    df = pd.DataFrame(stats, columns=[\"File\", \"Rows\", \"Columns\"])\n    failure_rate = get_failure_rate('train_numeric')\n    df[\"Error\"] = 0\n    df.loc[df.File == 'train_numeric', 'Error'] = failure_rate\n    return df\n    \n\n# due to time/memory limitation I decided to comment this part, but you can run it localy if you need to\n# res = explore_data_size()\n# res.plot(kind='bar', x='File', y='Columns', title=\"Fig:1 - Columns per file\",\n#         figsize=(13,5), fontsize=14)\n# print(res)"},{"cell_type":"markdown","metadata":{"_cell_guid":"0fcef473-2056-0f27-4a02-f1d4a50900e0"},"source":"The file size exploration results\n\n \n\n                   File     Rows  Columns     Error\n    0      train_numeric  1183747      970  0.005845\n    1         train_date  1183747     1157  0.000000\n    2  train_categorical  1183747     2141  0.000000\n    3       test_numeric  1183748      969  0.000000\n    4          test_date  1183748     1157  0.000000\n    5   test_categorical  1183748     2141  0.000000"},{"cell_type":"markdown","metadata":{"_cell_guid":"78a60393-af27-2d25-342b-fa664cb4e089"},"source":"From the results above we can say:\n\n 1. We are dealing with more 1 million record per file, so we just need to be careful not read the whole file at once, especially if you have limited memory.\n 2. The number of columns per file is also huge 1000 to 2000 per file. Each column represents a feature the corresponds to some measurement on a station or a machine.\n 3. Each file contains the \"Id\" column which represent the unique identifier of the product unit that we are tracking\n 4. Finally we notice that the error rate is about 0.5%, so it will be challenging to try to predict such small number from this huge data set. But we'll see..."},{"cell_type":"markdown","metadata":{"_cell_guid":"59e7481b-f0cc-c504-5c6f-6b3b42918c0d"},"source":"## 2- Based on the features, how many stations, lines, failure rate per station? ##\nin this section we'll take a look at the features or the columns and try to see how many station and line of production we have? how many features does each station have? how many products were processed in each station? and what's the failure rate per station?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5de5558e-9ea9-1324-7fdd-7b8c9c582250"},"outputs":[],"source":"def get_features(feature_list):\n    # function to group features by station or line of production sinc the convention is:\n    # L1_S15_F232 means Line 1, Station 15, Feature 232\n    line_features = {}\n    station_features = {}\n    lines = set([item.split('_')[0] for item in feature_list])\n    stations = set([item.split('_')[1] for item in feature_list])\n\n    for l in lines:\n        line_features[l] = [item for item in feature_list if '%s_' % l in item]\n\n    for s in stations:\n        station_features[s] = [item for item in feature_list if '%s_' % s in item]\n\n    return (line_features, station_features)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5f51c11c-51d0-c882-a887-accd10f5bd73"},"outputs":[],"source":"def explore_features(data_set_name):\n    # how many units processed in each station\n    # also percentage of failed/passed QS in each station\n    features = pd.read_csv(INPUT_PATH % data_set_name, nrows=1).drop([\"Response\", \"Id\"], axis=1).columns.values\n    line_features, station_features = get_features(features)\n    # create a dataframe cols: station, features_count\n    sdf = pd.DataFrame(list({int(key[1:]): len(station_features[key]) for\n                             key in station_features.keys()}.items()),\n                       columns=[\"Station\", \"FeatureCount\"])\n    ldf = pd.DataFrame(list({int(key[1:]): len(line_features[key]) for\n                             key in line_features.keys()}.items()),\n                       columns=[\"Line\", \"FeatureCount\"])\n    \n    stations_plot = sdf.plot(x=\"Station\", y=\"FeatureCount\", kind=\"bar\",\n                             title=\"Fig:2 - Features by Station\",\n                             figsize=(13,6), fontsize=12)\n    \n    return line_features, station_features\n\nlf, sf = explore_features(\"train_numeric\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"0ae0f848-1e78-1977-83c1-075ffca604dd"},"source":"Wow! most stations has less than 20 features, except station 24, 25 which has more than 200 features each! those must be the biggest machines in this chocolate mousse factory.\n\nSo let's dive into the next section where we will try to calculate the number of units processed by each station (machine) and also what's the failure rate for each station."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"584b1257-a891-01e3-0c9f-b895444120d2"},"outputs":[],"source":"def explore_product_by_station(data_set_name):\n    # need to see how many product per station\n    # features = pd.read_csv(INPUT_PATH % data_set_name, nrows=1).drop([\"Response\", \"Id\"], axis=1).columns.values\n    # line_features, station_features = get_features(features)\n    station_features = sf\n    dwb_result = []\n    ppbs_result = []\n    rows = 0\n    # inside this loop we will try to calculate different data sets, this way we only read the\n    # files once since it takes a while and produce multiple results\n    for station in station_features:\n        station_data = pd.read_csv(\n            INPUT_PATH % data_set_name,\n            usecols=station_features[station] + ['Id', 'Response'])\n        \n        # need to get how many rows in this data set\n        if not rows:\n            rows = station_data.shape[0]\n        # need to store how many processed units in each station\n        # only if all features in that station has value we conside unit processed in this station\n        ppbs_result.append(\n            {'Station': int(station[1:]),\n             'Processed': station_data[station_features[station]].isnull().all(axis=1).sum()})\n        \n        # data without blanks (dwb): take all rows and drop any row that has any blank\n        # in any station column\n        dwb = station_data.dropna(how=\"any\")\n        dwb_result.append([int(station[1:]), dwb.shape[0],\n                       dwb[dwb[\"Response\"] == 1].shape[0],\n                       dwb[dwb[\"Response\"] == 0].shape[0]])\n        \n    # convert list to a dataframe and prepare for the plot\n    dwb_df = pd.DataFrame(\n        dwb_result, columns=[\"Station\", \"Count\", \"Failed\", \"Passed\"]).sort_values(by=[\"Station\"])\n    # calculate the error rate for each station\n    dwb_df[\"Error\"] = dwb_df.apply(lambda row: float(row[\"Failed\"]) / row[\"Passed\"], axis=1)\n    \n    # plot the stacked bar tot product/ station\n    my_plot = dwb_df[[\"Station\", \"Failed\", \"Passed\"]].plot(kind=\"bar\", stacked=True, x=\"Station\",\n        title=\"Fig:3 - Products by station\", figsize=(13,6), fontsize=12)\n    my_plot.set_xlabel(\"Stations\")\n    my_plot.set_ylabel(\"Record Count\")\n    \n    # plot error rate per station\n    my_plot = dwb_df[[\"Station\", \"Error\"]].plot(\n        kind=\"bar\", x=\"Station\",\n        title=\"Fig:4 - Error by station\", figsize=(13,6), fontsize=12)\n    my_plot.set_xlabel(\"Stations\")\n    my_plot.set_ylabel(\"Error %\")\n    \n    # process the ppbs result to see how many products has been process by each station.\n    ppbs_df = pd.DataFrame(\n        ppbs_result, columns=['Station', 'Processed']).sort(columns=['Station'])\n    # calculate the missed product for each station: all count - processed count\n    ppbs_df[\"Missed\"] = ppbs_df[\"Processed\"].apply(lambda x: rows - x)\n    # the plot\n    ppbs_df.plot(x=\"Station\", kind=\"bar\", stacked=True,\n            title=\"Fig: 5 - Products processed by each station\",\n            figsize=(13,6), fontsize=12)\n    \n\n# explore_product_by_station(\"train_numeric\")"},{"cell_type":"markdown","metadata":{"_cell_guid":"a506837c-bcd6-5287-4e5a-f2d83e893d35"},"source":"## So what can we say by just looking at the 3 plots above: ##\n\n 1. Station 24, 25 has by far the most number of features, more than 200 feature per stations! at the same time most of the stations has less than 20 features. And worth noticing that station 32 has only one feature.\n\n 2. Not all stations have the same number of records or product! Does it mean that each product doesn't go through each station? could it be that we have different types of products and each one get processed through different station? Based on my understanding on manufacturing if let's say we are producing Orange Juice, then each juice box will go through the same process and the same machines until the juice is ready. Can someone tell me why the numbers are not equal for all the stations?\n\n 3. The suspicious station 32 has really a minimal number of records. So it doesn't process a lot of products.\n\n 4. Also worth noting that station 32 has by far most of QC failures almost 0.5%. At the same time all of the other stations has less than 0.1% as a failure rate! But wait S32 doesn't process a lot of records, so 0.5% of low number of records is minimal. However if each other station has a failure rate of 0.1% of the total production quantity then this number will add up eventually."}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}