{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"c3f40457-d99c-e724-125d-8cc4e57ee172"},"source":"## Error/Failure Rate EDA and Feature Engineering Ideas\n\nI wanted to know if certain lines or stations were correlated to higher error rates. While exploring, I found this code is also somewhat useful for feature engineering, such as taking the min/max values at each station.\n\n### Station 32\n\nA total of 24,543 samples run through station 32, with a 4.7% error rate, compared the mean error rate 0.6%. It only has one feature, L3_S32_F3850, which has come up on other Kernels ranking feature importance.\n\nCoincidentally (or maybe not), Station 31 is associated with the lowest failure rate at 0.27%"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"af527e99-8ad8-76d4-f6ea-67e9030845b4"},"outputs":[],"source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nsns.set_color_codes(\"muted\")\nimport matplotlib.pyplot as plt\n% matplotlib inline\n\nnumeric = \"../input/train_numeric.csv\""},{"cell_type":"markdown","metadata":{"_cell_guid":"0c0fce6c-8cfe-3656-0e74-794819b2952c"},"source":"## Sorting out Lines, Stations, and Features\nThe function below creates a two dicts that isolate all the features that belong to specific Lines and Stations. This makes it easy to slice the data on a Line-by-Line or Station-by-Station basis."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"04d7fd23-272c-65ef-87e6-9b1cca612fad"},"outputs":[],"source":"features = pd.read_csv(numeric, nrows=1).drop(['Response', 'Id'], axis=1).columns.values\n\ndef orgainize(features):\n    line_features = {}\n    station_features = {}\n    lines = set([f.split('_')[0] for f in features])\n    stations = set([f.split('_')[1] for f in features])\n    \n    for l in lines:\n        line_features[l] = [f for f in features if l+'_' in f]\n        \n    for s in stations:\n        station_features[s] = [f for f in features if s+'_' in f]\n        \n            \n    return line_features, station_features\n\nline_features, station_features = orgainize(features)\n\nprint(\"Features in Station 32: {}\".format( station_features['S32'] ))"},{"cell_type":"markdown","metadata":{"_cell_guid":"651a96e0-2013-87e4-859e-aa79734b6948"},"source":"## Exploring Stations\n\n- Features - Total features in the Station\n- Samples - Total samples with measured values (non-NaN) >= 1 in the Station\n- Error rate - (Response==1) rate for samples in the Station.\n\nNote* Samples run through multiple stations/lines, which was not taken into account, but may be important.\n\nNote** A small percentage of samples have no data and were dropped."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"fc133c67-36e7-7291-8878-de0b226975f4"},"outputs":[],"source":"station_error = []\nfor s in station_features:\n    cols = ['Id', 'Response']\n    cols.extend(station_features[s])\n    df = pd.read_csv(numeric, usecols=cols).dropna(subset=station_features[s], how='all')\n    error_rate = df[df.Response == 1].size / float(df[df.Response == 0].size)\n    station_error.append([df.shape[1]-2, df.shape[0], error_rate]) \n    \nstation_data = pd.DataFrame(station_error, \n                         columns=['Features', 'Samples', 'Error_Rate'], \n                         index=station_features).sort_index()\nstation_data"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"db0a665b-a57b-6830-1b1a-f9394d1413f2"},"outputs":[],"source":"plt.figure(figsize=(8, 20))\nsns.barplot(x='Error_Rate', y=station_data.index.values, data=station_data, color=\"red\")\nplt.title('Error Rate between Production Stations')\n\nplt.xlabel('Station Error Rate')\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"552797b0-94f9-b659-408e-bc92796bca8f"},"source":"## Quick Feature Engineering Example\nHere's an example of how to use the station dict can be used to create new features or just reduce the size of the data."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"36c392b7-5484-d266-cc4d-bedc4dd766d9"},"outputs":[],"source":"data = pd.read_csv(numeric, nrows=100)\n\ndef make_features(df):\n    new_features = pd.DataFrame({})\n    for s in station_features.keys():\n        station_data = df[station_features[s]]\n        col = s+'_max'\n        new_features[col] = station_data.max(axis=1).fillna(-1.)\n        col = s+'_min'\n        new_features[col] = station_data.min(axis=1).fillna(-1.)\n    return new_features\n\ndata = make_features(data)\ndata.head()"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}