{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"#Importing Python Libraries to use different functions \n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport tqdm\nimport gc\nimport sys\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":1.045176,"end_time":"2022-04-25T20:02:40.828686","exception":false,"start_time":"2022-04-25T20:02:39.78351","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preparation","metadata":{"papermill":{"duration":0.029886,"end_time":"2022-04-25T20:02:40.888822","exception":false,"start_time":"2022-04-25T20:02:40.858936","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"As dataset is quite large , we will be solving the problem by breaking down dataset into Chunks and Merging the results afterwards.","metadata":{"papermill":{"duration":0.029236,"end_time":"2022-04-25T20:02:40.948336","exception":false,"start_time":"2022-04-25T20:02:40.9191","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#reading csv files of dataset \ndate = pd.read_csv('../input/bosch-production-line-performance/train_date.csv.zip', nrows=10000)\nnumeric = pd.read_csv('../input/bosch-production-line-performance/train_numeric.csv.zip', nrows=10000)\ncategory = pd.read_csv('../input/bosch-production-line-performance/train_categorical.csv.zip', nrows=10000)","metadata":{"papermill":{"duration":7.160523,"end_time":"2022-04-25T20:02:48.139613","exception":false,"start_time":"2022-04-25T20:02:40.97909","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"date","metadata":{"papermill":{"duration":0.082035,"end_time":"2022-04-25T20:02:48.255283","exception":false,"start_time":"2022-04-25T20:02:48.173248","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The data represents measurements of parts as they move through Bosch's production lines. Each part has a unique Id. The goal is to predict which parts will fail quality control (represented by a 'Response' = 1).\n\nThe dataset contains an extremely large number of anonymized features. Features are named according to a convention that tells you the production line, the station on the line, and a feature number. E.g. L3_S36_F3939 is a feature measured on line 3, station 36, and is feature number 3939.\n\nOn account of the large size of the dataset, we have separated the files by the type of feature they contain: numerical, categorical, and finally, a file with date features. The date features provide a timestamp for when each measurement was taken. Each date column ends in a number that corresponds to the previous feature number. E.g. the value of L0_S0_D1 is the time at which L0_S0_F0 was taken.","metadata":{"papermill":{"duration":0.031005,"end_time":"2022-04-25T20:02:48.317778","exception":false,"start_time":"2022-04-25T20:02:48.286773","status":"completed"},"tags":[]}},{"cell_type":"code","source":"numeric","metadata":{"papermill":{"duration":0.070967,"end_time":"2022-04-25T20:02:48.419257","exception":false,"start_time":"2022-04-25T20:02:48.34829","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"category","metadata":{"papermill":{"duration":0.071411,"end_time":"2022-04-25T20:02:48.522447","exception":false,"start_time":"2022-04-25T20:02:48.451036","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# FEATURE ENGINEERING","metadata":{"papermill":{"duration":0.031694,"end_time":"2022-04-25T20:02:48.588025","exception":false,"start_time":"2022-04-25T20:02:48.556331","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### The list of numeric features is selected based on the other XGBOOST classifier check the [numericclassifier notebook](https://www.kaggle.com/nitinnitj/feature-selection-using-xgboost)","metadata":{"papermill":{"duration":0.031813,"end_time":"2022-04-25T20:02:48.652018","exception":false,"start_time":"2022-04-25T20:02:48.620205","status":"completed"},"tags":[]}},{"cell_type":"code","source":"num_feats = ['Id',\n       'L3_S30_F3514', 'L0_S9_F200', 'L3_S29_F3430', 'L0_S11_F314',\n       'L0_S0_F18', 'L3_S35_F3896', 'L0_S12_F350', 'L3_S36_F3918',\n       'L0_S0_F20', 'L3_S30_F3684', 'L1_S24_F1632', 'L0_S2_F48',\n       'L3_S29_F3345', 'L0_S18_F449', 'L0_S21_F497', 'L3_S29_F3433',\n       'L3_S30_F3764', 'L0_S1_F24', 'L3_S30_F3554', 'L0_S11_F322',\n       'L3_S30_F3564', 'L3_S29_F3327', 'L0_S2_F36', 'L0_S9_F180',\n       'L3_S33_F3855', 'L0_S0_F4', 'L0_S21_F477', 'L0_S5_F114',\n       'L0_S6_F122', 'L1_S24_F1122', 'L0_S9_F165', 'L0_S18_F439',\n       'L1_S24_F1490', 'L0_S6_F132', 'L3_S29_F3379', 'L3_S29_F3336',\n       'L0_S3_F80', 'L3_S30_F3749', 'L1_S24_F1763', 'L0_S10_F219',\n 'Response']","metadata":{"papermill":{"duration":0.043547,"end_time":"2022-04-25T20:02:48.727933","exception":false,"start_time":"2022-04-25T20:02:48.684386","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"modified the dataset , by dropping non-required colummns from dataset And sorting them as necessary ","metadata":{}},{"cell_type":"code","source":"length = date.drop('Id', axis=1).count()\ndate_cols = length.reset_index().sort_values(by=0, ascending=False)\nstations = sorted(date_cols['index'].str.split('_',expand=True)[1].unique().tolist())\ndate_cols['station'] = date_cols['index'].str.split('_',expand=True)[1]\ndate_cols = date_cols.drop_duplicates('station', keep='first')['index'].tolist()","metadata":{"papermill":{"duration":0.120787,"end_time":"2022-04-25T20:02:48.8808","exception":false,"start_time":"2022-04-25T20:02:48.760013","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"diving the data into chunks for train files . ","metadata":{}},{"cell_type":"code","source":"data = None\nfor chunk in pd.read_csv('../input/bosch-production-line-performance/train_date.csv.zip',usecols=['Id'] + date_cols,chunksize=50000,low_memory=False):\n\n    chunk.columns = ['Id'] + stations\n    chunk['start_station'] = -1\n    chunk['end_station'] = -1\n    \n    for s in stations:\n        chunk[s] = 1 * (chunk[s] >= 0)\n        id_not_null = chunk[chunk[s] == 1].Id\n        chunk.loc[(chunk['start_station']== -1) & (chunk.Id.isin(id_not_null)),'start_station'] = int(s[1:])\n        chunk.loc[chunk.Id.isin(id_not_null),'end_station'] = int(s[1:])   \n    data = pd.concat([data, chunk])","metadata":{"papermill":{"duration":84.04085,"end_time":"2022-04-25T20:04:12.954809","exception":false,"start_time":"2022-04-25T20:02:48.913959","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"dividing the data into chunks for test files ","metadata":{}},{"cell_type":"code","source":"for chunk in pd.read_csv('../input/bosch-production-line-performance/test_date.csv.zip',usecols=['Id'] + date_cols,chunksize=50000,low_memory=False):\n    \n    chunk.columns = ['Id'] + stations\n    chunk['start_station'] = -1\n    chunk['end_station'] = -1\n    for s in stations:\n        chunk[s] = 1 * (chunk[s] >= 0)\n        id_not_null = chunk[chunk[s] == 1].Id\n        chunk.loc[(chunk['start_station']== -1) & (chunk.Id.isin(id_not_null)),'start_station'] = int(s[1:])\n        chunk.loc[chunk.Id.isin(id_not_null),'end_station'] = int(s[1:])   \n    data = pd.concat([data, chunk])\ndel chunk\ngc.collect()   ","metadata":{"papermill":{"duration":87.765033,"end_time":"2022-04-25T20:05:40.753402","exception":false,"start_time":"2022-04-25T20:04:12.988369","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = data[['Id','start_station','end_station']]\nusefuldatefeatures = ['Id']+date_cols","metadata":{"papermill":{"duration":0.065362,"end_time":"2022-04-25T20:05:40.851214","exception":false,"start_time":"2022-04-25T20:05:40.785852","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"defining min and max date of each process for train dataset ","metadata":{}},{"cell_type":"code","source":"minmaxfeatures = None\nfor chunk in pd.read_csv('../input/bosch-production-line-performance/train_date.csv.zip',usecols=usefuldatefeatures,chunksize=50000,low_memory=False):\n    features = chunk.columns.values.tolist()\n    features.remove('Id')\n    df_mindate_chunk = chunk[['Id']].copy()\n    df_mindate_chunk['mindate'] = chunk[features].min(axis=1).values\n    df_mindate_chunk['maxdate'] = chunk[features].max(axis=1).values\n    df_mindate_chunk['min_time_station'] =  chunk[features].idxmin(axis = 1).apply(lambda s: int(s.split('_')[1][1:]) if s is not np.nan else -1)\n    df_mindate_chunk['max_time_station'] =  chunk[features].idxmax(axis = 1).apply(lambda s: int(s.split('_')[1][1:]) if s is not np.nan else -1)\n    minmaxfeatures = pd.concat([minmaxfeatures, df_mindate_chunk])\n\ndel chunk\ngc.collect()","metadata":{"papermill":{"duration":65.194788,"end_time":"2022-04-25T20:06:46.078582","exception":false,"start_time":"2022-04-25T20:05:40.883794","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"defining min and max date of each process for test dataset ","metadata":{}},{"cell_type":"code","source":"for chunk in pd.read_csv('../input/bosch-production-line-performance/test_date.csv.zip',usecols=usefuldatefeatures,chunksize=50000,low_memory=False):\n    features = chunk.columns.values.tolist()\n    features.remove('Id')\n    df_mindate_chunk = chunk[['Id']].copy()\n    df_mindate_chunk['mindate'] = chunk[features].min(axis=1).values\n    df_mindate_chunk['maxdate'] = chunk[features].max(axis=1).values\n    df_mindate_chunk['min_time_station'] =  chunk[features].idxmin(axis = 1).apply(lambda s: int(s.split('_')[1][1:]) if s is not np.nan else -1)\n    df_mindate_chunk['max_time_station'] =  chunk[features].idxmax(axis = 1).apply(lambda s: int(s.split('_')[1][1:]) if s is not np.nan else -1)\n    minmaxfeatures = pd.concat([minmaxfeatures, df_mindate_chunk])\n\ndel chunk\ngc.collect()","metadata":{"papermill":{"duration":66.369857,"end_time":"2022-04-25T20:07:52.481903","exception":false,"start_time":"2022-04-25T20:06:46.112046","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sorted the columns by mindate and then id. After that added two columns of features , one by shift of one row and one as it is . ","metadata":{}},{"cell_type":"code","source":"minmaxfeatures.sort_values(by=['mindate', 'Id'], inplace=True)\nminmaxfeatures['min_Id_rev'] = -minmaxfeatures.Id.diff().shift(-1)\nminmaxfeatures['min_Id'] = minmaxfeatures.Id.diff()","metadata":{"papermill":{"duration":2.263792,"end_time":"2022-04-25T20:07:54.779261","exception":false,"start_time":"2022-04-25T20:07:52.515469","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols = [['Id']+date_cols,num_feats]","metadata":{"papermill":{"duration":0.041799,"end_time":"2022-04-25T20:07:54.854141","exception":false,"start_time":"2022-04-25T20:07:54.812342","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"traindata = None\ntestdata = None","metadata":{"papermill":{"duration":0.041379,"end_time":"2022-04-25T20:07:54.928874","exception":false,"start_time":"2022-04-25T20:07:54.887495","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"allocated two arrays conatining all testing and training zip folders respectively ","metadata":{}},{"cell_type":"code","source":"trainfiles = ['train_date.csv.zip','train_numeric.csv.zip']\ntestfiles = ['test_date.csv.zip','test_numeric.csv.zip']","metadata":{"papermill":{"duration":0.04166,"end_time":"2022-04-25T20:07:55.003848","exception":false,"start_time":"2022-04-25T20:07:54.962188","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i,f in enumerate(trainfiles):\n    \n    subset = None\n    \n    for chunk in pd.read_csv('../input/bosch-production-line-performance/' + f,usecols=cols[i],chunksize=100000,low_memory=False):\n        subset = pd.concat([subset, chunk])\n    \n    if traindata is None:\n        traindata = subset.copy()\n    else:\n        traindata = pd.merge(traindata, subset.copy(), on=\"Id\")\n        \ndel subset,chunk\ngc.collect()\ndel cols[1][-1]","metadata":{"papermill":{"duration":113.943525,"end_time":"2022-04-25T20:09:48.980969","exception":false,"start_time":"2022-04-25T20:07:55.037444","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"for i, f in enumerate(testfiles):\n    subset = None\n    \n    for chunk in pd.read_csv('../input/bosch-production-line-performance/' + f,usecols=cols[i],chunksize=100000,low_memory=False):\n        subset = pd.concat([subset, chunk])\n        \n    if testdata is None:\n        testdata = subset.copy()\n    else:\n        testdata = pd.merge(testdata, subset.copy(), on=\"Id\")\n    \ndel subset,chunk\ngc.collect()","metadata":{"papermill":{"duration":113.650432,"end_time":"2022-04-25T20:11:42.665618","exception":false,"start_time":"2022-04-25T20:09:49.015186","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"merged our train and test dataframes with two other dataframes features and data by comparing the ids","metadata":{}},{"cell_type":"code","source":"traindata = traindata.merge(minmaxfeatures, on='Id')\ntraindata = traindata.merge(data, on='Id')\ntestdata = testdata.merge(minmaxfeatures, on='Id')\ntestdata = testdata.merge(data, on='Id')","metadata":{"papermill":{"duration":4.987297,"end_time":"2022-04-25T20:11:47.686501","exception":false,"start_time":"2022-04-25T20:11:42.699204","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del minmaxfeatures,data\ngc.collect()","metadata":{"papermill":{"duration":0.116718,"end_time":"2022-04-25T20:11:47.838282","exception":false,"start_time":"2022-04-25T20:11:47.721564","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"traindata","metadata":{"papermill":{"duration":0.381456,"end_time":"2022-04-25T20:11:48.254176","exception":false,"start_time":"2022-04-25T20:11:47.87272","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"testdata","metadata":{"papermill":{"duration":0.381836,"end_time":"2022-04-25T20:11:48.671242","exception":false,"start_time":"2022-04-25T20:11:48.289406","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"filled all rows with value null or undefined or NAN with '0'","metadata":{}},{"cell_type":"code","source":"traindata.fillna(value=0,inplace=True)\ntestdata.fillna(value=0,inplace=True)","metadata":{"papermill":{"duration":2.161807,"end_time":"2022-04-25T20:11:50.869275","exception":false,"start_time":"2022-04-25T20:11:48.707468","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.set_printoptions(suppress=True)","metadata":{"papermill":{"duration":0.044223,"end_time":"2022-04-25T20:11:51.199813","exception":false,"start_time":"2022-04-25T20:11:51.15559","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc","metadata":{"papermill":{"duration":0.044026,"end_time":"2022-04-25T20:11:51.280602","exception":false,"start_time":"2022-04-25T20:11:51.236576","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"rearranged the failure and non - failure components and stored all in single dataframe total","metadata":{}},{"cell_type":"code","source":"total = traindata[traindata['Response']==0].sample(frac=1).head(400000)\ntotal = pd.concat([total,traindata[traindata['Response']==1]]).sample(frac=1)","metadata":{"papermill":{"duration":2.713324,"end_time":"2022-04-25T20:11:54.030734","exception":false,"start_time":"2022-04-25T20:11:51.31741","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split","metadata":{"papermill":{"duration":0.164985,"end_time":"2022-04-25T20:11:54.234396","exception":false,"start_time":"2022-04-25T20:11:54.069411","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"initialising the input and output dataframe","metadata":{}},{"cell_type":"code","source":"X,y = total.drop(['Response','Id'],axis=1),total['Response']","metadata":{"papermill":{"duration":0.151661,"end_time":"2022-04-25T20:11:54.425946","exception":false,"start_time":"2022-04-25T20:11:54.274285","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Spilt the data in ratio ( 7/3) i.e. 30% test and 70 % train ","metadata":{}},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42,stratify=y)","metadata":{"papermill":{"duration":0.656558,"end_time":"2022-04-25T20:11:55.118272","exception":false,"start_time":"2022-04-25T20:11:54.461714","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MODELLING","metadata":{"papermill":{"duration":0.037303,"end_time":"2022-04-25T20:11:55.192093","exception":false,"start_time":"2022-04-25T20:11:55.15479","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Importing all the models decided to use for training and testing ","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.ensemble import BaggingClassifier","metadata":{"papermill":{"duration":0.246726,"end_time":"2022-04-25T20:11:55.477947","exception":false,"start_time":"2022-04-25T20:11:55.231221","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"appended all models in a single array  , n_estimators represent the no. of rows in subsets that will be created , the hyperparameters defined within models are set by hit and trial method .  All the predicted values are stored in ypredicted array . and MCC error function is used to see the accuracy achieveable . ","metadata":{}},{"cell_type":"code","source":" # models selected for testing \ndef get_models():\n\tmodels = list()\n\tmodels.append(AdaBoostClassifier(n_estimators=500,random_state=11))\n\tmodels.append(BaggingClassifier(n_estimators=500,n_jobs=-1,verbose=1,random_state=11))\n\tmodels.append(RandomForestClassifier(n_estimators=500,n_jobs=-1,verbose=1,random_state=11))\n\treturn models \n\n# loading the models \nmodels = get_models()\n\n#fit the models according to training dataset  \n#create loaded_models folder in google collab \ndef fit_base_models(X_train , y_train , models):\n\tfor model in models:\n\t\tmodel.fit(X_train, y_train) \n\n\nfit_base_models(X_train , y_train , models) \nfrom sklearn.metrics import matthews_corrcoef\n\n# storing the predicted values by each model in given array and printing the rmse ( error)\nypredicted = []\ndef evaluate_models(X_test, y_test, models):\n    for model in models:\n        yhat = model.predict(X_test) \n        print(model)\n        ypredicted.append(yhat)\n        mcc = matthews_corrcoef(y_test, yhat)\n        print('%s: MCC %.3f' % (model.__class__.__name__, mcc))\n\n#predicted values on our testdataset testX \nevaluate_models(X_test , y_test , models)","metadata":{"papermill":{"duration":6778.303396,"end_time":"2022-04-25T22:04:53.820089","exception":false,"start_time":"2022-04-25T20:11:55.516693","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]}]}