{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Notebook Submission","metadata":{}},{"cell_type":"markdown","source":"In this notebook we will discuss the leverage of path signature method with ensemble machine learning on anomaly detection. Signature method has shown to be effective in extracting patterns in data and thus improve the predictive power of the model.\n\nData Processing can be found in [this notebook](https://www.kaggle.com/yutianxiao/data-processing). In the following notebook, we will only discuss the training and prediction stages.\n\n\n#### Section 1: **Ensemble model**\n#### Section 2: **Prediction**","metadata":{}},{"cell_type":"code","source":"import sklearn\nimport numpy as np\nimport pandas as pd\nfrom xgboost import XGBClassifier\nimport lightgbm as lgb\n# from tqdm import tqdm, trange\nfrom sklearn import metrics\nfrom sklearn.preprocessing import StandardScaler\n\n# set seed for all random processes in this notebook\nr_s = 5","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:02.072975Z","iopub.execute_input":"2022-11-06T13:57:02.073414Z","iopub.status.idle":"2022-11-06T13:57:02.082444Z","shell.execute_reply.started":"2022-11-06T13:57:02.073379Z","shell.execute_reply":"2022-11-06T13:57:02.080381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Import the pre-computed training set with signature features of order 5 from the directory.","metadata":{}},{"cell_type":"code","source":"sig_order = 5","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:02.084297Z","iopub.execute_input":"2022-11-06T13:57:02.084694Z","iopub.status.idle":"2022-11-06T13:57:02.095692Z","shell.execute_reply.started":"2022-11-06T13:57:02.084663Z","shell.execute_reply":"2022-11-06T13:57:02.094016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"whole_set = pd.read_pickle('../input/largescale-energy-anomaly/whole_set_with_sig_{}.pkl'.format(sig_order))","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:02.097598Z","iopub.execute_input":"2022-11-06T13:57:02.098418Z","iopub.status.idle":"2022-11-06T13:57:39.811593Z","shell.execute_reply.started":"2022-11-06T13:57:02.098379Z","shell.execute_reply":"2022-11-06T13:57:39.809755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble model","metadata":{}},{"cell_type":"markdown","source":"Split the original training set to trainig and validation sets according to building ids. ","metadata":{}},{"cell_type":"code","source":"train_en = whole_set[whole_set['building_id']%5<4]\ntest_en = whole_set[whole_set['building_id']%5==4]","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:39.813883Z","iopub.execute_input":"2022-11-06T13:57:39.814271Z","iopub.status.idle":"2022-11-06T13:57:43.329106Z","shell.execute_reply.started":"2022-11-06T13:57:39.814242Z","shell.execute_reply":"2022-11-06T13:57:43.328080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_en[train_en.anomaly==1].shape)\nprint(train_en[train_en.anomaly==0].shape)","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:43.330377Z","iopub.execute_input":"2022-11-06T13:57:43.331859Z","iopub.status.idle":"2022-11-06T13:57:50.802182Z","shell.execute_reply.started":"2022-11-06T13:57:43.331790Z","shell.execute_reply":"2022-11-06T13:57:50.800494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use ensemble learning method here. The size of anomalies ($n_a=29529$) are far larger than the size of normal points. To balance each class in the training set, we randomly sample the same amount of normal points from the original training set ${X}$ and concatenate it to the set of all anomalies $X_a$ to form a new training set $X_{new}$. Classifiers specifically to this set are then trained. The ensemble training process can be simplified as:\n\n**Step 0**: Initialisation. Create an empty list $L_{classifier}$ for storing all trained classifiers.\n\n**Step 1**: Randomly sample $n_a$ normal points and remove them from $X$. Cancatenate the sampled points to anomaly set $X_a$ to form a new training set $X_{new}$ of size $2n_a$.\n\n**Step 2**: Train XGBoost and LGBoost classifiers (i.e. $C_{xgb}$ and $C_{lgb}$ respectively) on $X_{new}$. Add $C_{xgb}$ and $C_{lgb}$ to the list $L_{classifier}$.\n\n**Step 3**: Repeat Step 1 and Step 2 until the size of $X$ is smaller than $n_a$.\n\n**Step 4**: Output the list of trained classifiers $L_{classifier}$.","metadata":{}},{"cell_type":"code","source":"def train_model(train, test=None):\n    # if test is None, we are training on the original training set (i.e. before train-test split)\n    if test is not None:\n#         x_test = test.iloc[:, :-1]\n#         y_test = test.iloc[:, -1]\n        x_test = test.drop(columns='anomaly')\n        y_test = test.anomaly\n    \n    # separate normal points and anomalies\n    neg = train[train['anomaly'] == 0]\n    pos = train[train['anomaly'] == 1]\n    len_pos = pos.shape[0]\n    em_models = []\n    scalers = []\n    count = 0\n    num_of_models = neg.shape[0]//len_pos\n    while neg.shape[0] >= len_pos:\n        print('{}/{}'.format(count, num_of_models-1))\n        print('neg size:{}, pos size:{}'.format(neg.shape[0], len_pos))\n        count += 1\n        \n        # randomly sample and remove n_a normal points from the training set\n        negs = neg.sample(n=len_pos, random_state=r_s)\n        neg.drop(index=negs.index, inplace=True)\n        \n        # concatenate sampled points with anomalies to form a new training set\n        df_eq = pd.concat([negs, pos], axis=0)\n        del negs\n        \n        # train XGB and LGB classifiers\n        models = [XGBClassifier(), lgb.LGBMClassifier()]\n        scaler = StandardScaler()\n#         x_train_local = df_eq.iloc[:, :-1]\n#         y_train = df_eq.iloc[:, -1]\n        x_train_local = df_eq.drop(columns='anomaly')\n        y_train = df_eq.anomaly\n        x_train_local = scaler.fit_transform(x_train_local)\n        if test is not None:\n            x_test_local = scaler.transform(x_test)\n        for i, model in enumerate(models):\n            model.fit(x_train_local, y_train)\n#             print(model)\n#             print('train set accuracy: ',metrics.roc_auc_score(y_train, model.predict_proba(x_train_local)[:,1]))\n#             if test is not None:\n#                 print('test set accuracy: ',metrics.roc_auc_score(y_test, model.predict_proba(x_test_local)[:,1]))\n#         return None, None\n        em_models.append(models)\n        scalers.append(scaler)\n    return scalers, em_models","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:50.807373Z","iopub.execute_input":"2022-11-06T13:57:50.807738Z","iopub.status.idle":"2022-11-06T13:57:50.821401Z","shell.execute_reply.started":"2022-11-06T13:57:50.807711Z","shell.execute_reply":"2022-11-06T13:57:50.819834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scalers, em_models = train_model(train_en, test_en)","metadata":{"execution":{"iopub.status.busy":"2022-11-06T13:57:50.822717Z","iopub.execute_input":"2022-11-06T13:57:50.823107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Predictions generated by each classifier is averaged and output.","metadata":{}},{"cell_type":"code","source":"def ensembled_predictions(x, scalers, em_models):\n    predictions = []\n    for i, models in enumerate(em_models):\n        x_local = scalers[i].transform(x)\n        for model in models:\n            predictions.append(model.predict_proba(x_local)[:,1])\n    return np.array(predictions).mean(axis=0)","metadata":{"execution":{"iopub.status.busy":"2022-11-04T18:08:15.400208Z","iopub.execute_input":"2022-11-04T18:08:15.401022Z","iopub.status.idle":"2022-11-04T18:08:15.414074Z","shell.execute_reply.started":"2022-11-04T18:08:15.400977Z","shell.execute_reply":"2022-11-04T18:08:15.413110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('overall ensembled model test set accuracy: ',metrics.roc_auc_score(test_en.iloc[:, -1], ensembled_predictions(test_en.iloc[:, :-1], scalers, em_models)))","metadata":{"execution":{"iopub.status.busy":"2022-11-04T18:08:15.415408Z","iopub.execute_input":"2022-11-04T18:08:15.416190Z","iopub.status.idle":"2022-11-04T18:11:20.715654Z","shell.execute_reply.started":"2022-11-04T18:08:15.416152Z","shell.execute_reply":"2022-11-04T18:11:20.714116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction","metadata":{"execution":{"iopub.status.busy":"2022-10-16T21:02:26.538274Z","iopub.execute_input":"2022-10-16T21:02:26.538618Z","iopub.status.idle":"2022-10-16T21:02:26.548660Z","shell.execute_reply.started":"2022-10-16T21:02:26.538592Z","shell.execute_reply":"2022-10-16T21:02:26.548020Z"}}},{"cell_type":"markdown","source":"To completely exploit the information in the original dataset, we will train the ensemble model on the whole training set (i.e. before train-test split). ","metadata":{}},{"cell_type":"code","source":"del train_en, test_en, scalers, em_models\nscalers, em_models = train_model(whole_set)\ndel whole_set","metadata":{"execution":{"iopub.status.busy":"2022-11-04T18:51:32.105999Z","iopub.execute_input":"2022-11-04T18:51:32.106387Z","iopub.status.idle":"2022-11-04T22:21:08.459033Z","shell.execute_reply.started":"2022-11-04T18:51:32.106357Z","shell.execute_reply":"2022-11-04T22:21:08.455666Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We then import the pre-processed prediction set to predict results for the submission.","metadata":{}},{"cell_type":"code","source":"x_predict = pd.read_pickle('../input/largescale-energy-anomaly/x_predict_with_sigs_5.pkl')","metadata":{"execution":{"iopub.status.busy":"2022-11-04T22:53:10.445085Z","iopub.execute_input":"2022-11-04T22:53:10.445568Z","iopub.status.idle":"2022-11-04T22:54:14.038049Z","shell.execute_reply.started":"2022-11-04T22:53:10.445534Z","shell.execute_reply":"2022-11-04T22:54:14.036722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Predict on the submission set with classifiers trained on the whole training set. ","metadata":{}},{"cell_type":"code","source":"predictions = ensembled_predictions(x_predict, scalers, em_models)","metadata":{"execution":{"iopub.status.busy":"2022-11-04T22:54:14.040389Z","iopub.execute_input":"2022-11-04T22:54:14.040893Z","iopub.status.idle":"2022-11-04T23:08:15.031281Z","shell.execute_reply.started":"2022-11-04T22:54:14.040849Z","shell.execute_reply":"2022-11-04T23:08:15.029473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Assign $0$s to the 'anomaly' column of the points which has missing meter_readings.","metadata":{}},{"cell_type":"code","source":"test = pd.read_pickle('../input/largescale-energy-anomaly/test_win_with_miss.pkl')","metadata":{"execution":{"iopub.status.busy":"2022-11-04T23:09:14.026250Z","iopub.execute_input":"2022-11-04T23:09:14.026693Z","iopub.status.idle":"2022-11-04T23:09:18.397929Z","shell.execute_reply.started":"2022-11-04T23:09:14.026657Z","shell.execute_reply":"2022-11-04T23:09:18.396809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = test[~np.isnan(test.row_id)]\ntest['row_id'] = test['row_id'].apply(lambda x: int(x))","metadata":{"execution":{"iopub.status.busy":"2022-11-04T23:09:18.399997Z","iopub.execute_input":"2022-11-04T23:09:18.400388Z","iopub.status.idle":"2022-11-04T23:09:19.926765Z","shell.execute_reply.started":"2022-11-04T23:09:18.400350Z","shell.execute_reply":"2022-11-04T23:09:19.925641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pds = pd.read_csv('../input/largescale-energy-anomaly/sample_submission.csv/sample_submission.csv')\noutput_df = test[['row_id', 'missing']]\noutput_df.sort_values(by='row_id', inplace=True)\noutput_df.reset_index(drop=True, inplace=True)\npds['anomaly'] = predictions\npds.loc[output_df[output_df['missing']==1].index, 'anomaly'] = 0\npds.to_csv('Submission_Final_1.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-04T23:09:19.928174Z","iopub.execute_input":"2022-11-04T23:09:19.929269Z","iopub.status.idle":"2022-11-04T23:09:24.581063Z","shell.execute_reply.started":"2022-11-04T23:09:19.929222Z","shell.execute_reply":"2022-11-04T23:09:24.580129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Acknowledgement\n1. Chun Fu https://www.kaggle.com/code/patrick0302/2-anomaly-detection-features-csv-revised-ensem\n2. Datasig https://github.com/datasig-ac-uk/Kaggle22/blob/main/GradientBoostingNotebook.ipynb","metadata":{}}]}