{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

📈 Time Series Outlier and Anomaly Detection 📈

","metadata":{"papermill":{"duration":0.092719,"end_time":"2022-02-23T02:27:22.75519","exception":false,"start_time":"2022-02-23T02:27:22.662471","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Time-Series-Anomaly-Detection\n\n# Introduction:\n\nAnomaly detection is the process of discover the event or the points which are unexpected at \nthis position of the dataset or deviates from the normal pattern of the dataset. \nSo, the detection of those points very important; because it give us an early step to make the \nemergency movements to control that un usual change. \nWe used many techniques to reach best one to apply it on our way of the project. \n\n \n# Anomaly Detection Methods: \nThe outlairs and anomaly detection different methods:\n\n
  • 1- Tukey’s box plot method\n
  • 2- Isolation forest.\n
  • 3- Anomly based on Machine learning Predication.","metadata":{"papermill":{"duration":0.124123,"end_time":"2022-02-23T02:27:22.971808","exception":false,"start_time":"2022-02-23T02:27:22.847685","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport plotly.express as px\nimport seaborn as sns\nimport warnings\nimport matplotlib.pyplot as plt\nfrom datetime import datetime, timezone\nimport datetime\nfrom scipy.stats import norm\nfrom sklearn.preprocessing import StandardScaler\nfrom scipy import stats\nimport plotly.express as px\nimport plotly.graph_objects as go\nwarnings.filterwarnings('ignore')","metadata":{"papermill":{"duration":3.35866,"end_time":"2022-02-23T02:27:26.422256","exception":false,"start_time":"2022-02-23T02:27:23.063596","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T15:25:44.487461Z","iopub.execute_input":"2022-05-15T15:25:44.487844Z","iopub.status.idle":"2022-05-15T15:25:46.120426Z","shell.execute_reply.started":"2022-05-15T15:25:44.487793Z","shell.execute_reply":"2022-05-15T15:25:46.119409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/energy-anomaly-detection/train.csv',parse_dates= [\"timestamp\"])","metadata":{"papermill":{"duration":7.144449,"end_time":"2022-02-23T02:27:33.658885","exception":false,"start_time":"2022-02-23T02:27:26.514436","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T15:25:11.520736Z","iopub.status.idle":"2022-05-15T15:25:11.521474Z","shell.execute_reply.started":"2022-05-15T15:25:11.521185Z","shell.execute_reply":"2022-05-15T15:25:11.521213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"papermill":{"duration":0.111216,"end_time":"2022-02-23T02:27:33.860822","exception":false,"start_time":"2022-02-23T02:27:33.749606","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:03.796851Z","iopub.execute_input":"2022-05-15T12:18:03.797257Z","iopub.status.idle":"2022-05-15T12:18:03.82029Z","shell.execute_reply.started":"2022-05-15T12:18:03.797208Z","shell.execute_reply":"2022-05-15T12:18:03.819351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"papermill":{"duration":0.101567,"end_time":"2022-02-23T02:27:34.054268","exception":false,"start_time":"2022-02-23T02:27:33.952701","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:03.822814Z","iopub.execute_input":"2022-05-15T12:18:03.82314Z","iopub.status.idle":"2022-05-15T12:18:03.831121Z","shell.execute_reply.started":"2022-05-15T12:18:03.823098Z","shell.execute_reply":"2022-05-15T12:18:03.830021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"papermill":{"duration":0.115561,"end_time":"2022-02-23T02:27:34.263332","exception":false,"start_time":"2022-02-23T02:27:34.147771","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:03.833201Z","iopub.execute_input":"2022-05-15T12:18:03.833647Z","iopub.status.idle":"2022-05-15T12:18:03.854762Z","shell.execute_reply.started":"2022-05-15T12:18:03.833528Z","shell.execute_reply":"2022-05-15T12:18:03.853895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe()","metadata":{"papermill":{"duration":0.116705,"end_time":"2022-02-23T02:27:34.47266","exception":false,"start_time":"2022-02-23T02:27:34.355955","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:03.855877Z","iopub.execute_input":"2022-05-15T12:18:03.856562Z","iopub.status.idle":"2022-05-15T12:18:04.08101Z","shell.execute_reply.started":"2022-05-15T12:18:03.856524Z","shell.execute_reply":"2022-05-15T12:18:04.080221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['anomaly'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:04.082134Z","iopub.execute_input":"2022-05-15T12:18:04.082408Z","iopub.status.idle":"2022-05-15T12:18:04.101825Z","shell.execute_reply.started":"2022-05-15T12:18:04.082377Z","shell.execute_reply":"2022-05-15T12:18:04.100589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['meter_reading'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:04.10308Z","iopub.execute_input":"2022-05-15T12:18:04.103332Z","iopub.status.idle":"2022-05-15T12:18:04.191437Z","shell.execute_reply.started":"2022-05-15T12:18:04.103305Z","shell.execute_reply":"2022-05-15T12:18:04.190243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(x=df['meter_reading'])","metadata":{"papermill":{"duration":0.31816,"end_time":"2022-02-23T02:27:34.886782","exception":false,"start_time":"2022-02-23T02:27:34.568622","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:04.193181Z","iopub.execute_input":"2022-05-15T12:18:04.193528Z","iopub.status.idle":"2022-05-15T12:18:04.689337Z","shell.execute_reply.started":"2022-05-15T12:18:04.193486Z","shell.execute_reply":"2022-05-15T12:18:04.688374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure()\nax = fig.gca()\nsns.distplot(df['meter_reading'])\nplt.show()\n\n#Probability plot\nfig = plt.figure()\nres = stats.probplot(df['meter_reading'], plot=plt)","metadata":{"papermill":{"duration":1.395582,"end_time":"2022-02-23T02:27:36.389224","exception":false,"start_time":"2022-02-23T02:27:34.993642","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:04.692489Z","iopub.execute_input":"2022-05-15T12:18:04.692758Z","iopub.status.idle":"2022-05-15T12:18:14.524738Z","shell.execute_reply.started":"2022-05-15T12:18:04.692725Z","shell.execute_reply":"2022-05-15T12:18:14.522127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_val_count_by_column = (df.isnull().sum())\nprint(missing_val_count_by_column[missing_val_count_by_column > 0])","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:11.522738Z","iopub.status.idle":"2022-05-15T15:25:11.523413Z","shell.execute_reply.started":"2022-05-15T15:25:11.523145Z","shell.execute_reply":"2022-05-15T15:25:11.523171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_name = df.iloc[:,[0,2]].columns","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:11.524728Z","iopub.status.idle":"2022-05-15T15:25:11.525373Z","shell.execute_reply.started":"2022-05-15T15:25:11.525101Z","shell.execute_reply":"2022-05-15T15:25:11.525138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# there is a problem with missing values so i will try to handel it.\n\nfrom sklearn.impute import SimpleImputer\n\nmy_imputer = SimpleImputer()\ndf_clean = my_imputer.fit_transform(df.iloc[:,[0,2]])\n\ndf_clean = pd.DataFrame(df_clean,columns = df_name)\n\nmissing_val_count_by_column = (df_clean.isnull().sum())\nprint(missing_val_count_by_column)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:11.534330Z","iopub.execute_input":"2022-05-15T15:25:11.534971Z","iopub.status.idle":"2022-05-15T15:25:12.962101Z","shell.execute_reply.started":"2022-05-15T15:25:11.534917Z","shell.execute_reply":"2022-05-15T15:25:12.959861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.iloc[:,[1,3]].shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.963437Z","iopub.status.idle":"2022-05-15T15:25:12.963838Z","shell.execute_reply.started":"2022-05-15T15:25:12.963613Z","shell.execute_reply":"2022-05-15T15:25:12.963632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_clean.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.965679Z","iopub.status.idle":"2022-05-15T15:25:12.966250Z","shell.execute_reply.started":"2022-05-15T15:25:12.966008Z","shell.execute_reply":"2022-05-15T15:25:12.966039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.concat([df.iloc[:,[1,3]], df_clean], axis=1)\n","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.967473Z","iopub.status.idle":"2022-05-15T15:25:12.967851Z","shell.execute_reply.started":"2022-05-15T15:25:12.967644Z","shell.execute_reply":"2022-05-15T15:25:12.967668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.968980Z","iopub.status.idle":"2022-05-15T15:25:12.969318Z","shell.execute_reply.started":"2022-05-15T15:25:12.969142Z","shell.execute_reply":"2022-05-15T15:25:12.969164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.columns","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.970625Z","iopub.status.idle":"2022-05-15T15:25:12.970964Z","shell.execute_reply.started":"2022-05-15T15:25:12.970787Z","shell.execute_reply":"2022-05-15T15:25:12.970811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df[['timestamp', 'building_id', 'meter_reading','anomaly']]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.972632Z","iopub.status.idle":"2022-05-15T15:25:12.973494Z","shell.execute_reply.started":"2022-05-15T15:25:12.973220Z","shell.execute_reply":"2022-05-15T15:25:12.973251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.974956Z","iopub.status.idle":"2022-05-15T15:25:12.975309Z","shell.execute_reply.started":"2022-05-15T15:25:12.975113Z","shell.execute_reply":"2022-05-15T15:25:12.975142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['building_id'] = df['building_id'].apply(np.int64)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:25:12.976475Z","iopub.status.idle":"2022-05-15T15:25:12.976823Z","shell.execute_reply.started":"2022-05-15T15:25:12.976624Z","shell.execute_reply":"2022-05-15T15:25:12.976646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:17.558226Z","iopub.execute_input":"2022-05-15T12:18:17.558522Z","iopub.status.idle":"2022-05-15T12:18:17.570554Z","shell.execute_reply.started":"2022-05-15T12:18:17.558481Z","shell.execute_reply":"2022-05-15T12:18:17.569252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

    1- Tukey’s box plot method

    \n\n\n\n\nin this method we depend on the pox plot to determine if the point is outlier or not and not \nonly that it gives us the ability to decide if this outlier is possible or probable outlier point; by \ncalculate the following parameters: \n- 25th percentile (Q1) \n- 75th percentile (Q3) \n- interquartile range (IQR = Q3 – Q1) \n- Lower inner fence: Q1 – (1.5 * IQR) \n- Upper inner fence: Q3 + (1.5 * IQR) \n- Lower outer fence: Q1 – (3 * IQR) \n- Upper outer fence: Q3 + (3 * IQR) \n\nThen decide if the point between the inner fence and outer fence it considered as a possible \noutlier point. And if the point lies outside the outer fence, it will be considered as probable \noutlier. ","metadata":{"papermill":{"duration":0.09625,"end_time":"2022-02-23T02:27:36.624817","exception":false,"start_time":"2022-02-23T02:27:36.528567","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### First apply on a random grid","metadata":{"papermill":{"duration":0.095291,"end_time":"2022-02-23T02:27:36.8149","exception":false,"start_time":"2022-02-23T02:27:36.719609","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df.columns","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:17.571921Z","iopub.execute_input":"2022-05-15T12:18:17.572146Z","iopub.status.idle":"2022-05-15T12:18:17.586752Z","shell.execute_reply.started":"2022-05-15T12:18:17.572117Z","shell.execute_reply":"2022-05-15T12:18:17.585837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['building_id'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:17.588314Z","iopub.execute_input":"2022-05-15T12:18:17.588648Z","iopub.status.idle":"2022-05-15T12:18:17.615475Z","shell.execute_reply.started":"2022-05-15T12:18:17.588605Z","shell.execute_reply":"2022-05-15T12:18:17.614314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.set_index('timestamp',inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:17.616882Z","iopub.execute_input":"2022-05-15T12:18:17.617618Z","iopub.status.idle":"2022-05-15T12:18:17.629764Z","shell.execute_reply.started":"2022-05-15T12:18:17.617579Z","shell.execute_reply":"2022-05-15T12:18:17.628813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.groupby(\"building_id\").get_group(108)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:17.631416Z","iopub.execute_input":"2022-05-15T12:18:17.632169Z","iopub.status.idle":"2022-05-15T12:18:17.782932Z","shell.execute_reply.started":"2022-05-15T12:18:17.632124Z","shell.execute_reply":"2022-05-15T12:18:17.782007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(30,10))\nax.plot(df.groupby(\"building_id\").get_group(108)['meter_reading'].index, \n df.groupby(\"building_id\").get_group(108)['meter_reading'], color='blue', label = 'Normal')\nax.set_title('Random building_id(108) ', fontsize=20)\nplt.legend()\nplt.show();","metadata":{"papermill":{"duration":0.406016,"end_time":"2022-02-23T02:27:37.315982","exception":false,"start_time":"2022-02-23T02:27:36.909966","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:17.784569Z","iopub.execute_input":"2022-05-15T12:18:17.785029Z","iopub.status.idle":"2022-05-15T12:18:18.334681Z","shell.execute_reply.started":"2022-05-15T12:18:17.784988Z","shell.execute_reply":"2022-05-15T12:18:18.333974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tukeys_method(df, variable):\n #Takes two parameters: dataframe & variable of interest as string\n q1 = df[variable].quantile(0.25)\n q3 = df[variable].quantile(0.75)\n iqr = q3-q1\n inner_fence = 1.5*iqr\n outer_fence = 3*iqr\n \n #inner fence lower and upper end\n inner_fence_le = q1-inner_fence\n inner_fence_ue = q3+inner_fence\n \n #outer fence lower and upper end\n outer_fence_le = q1-outer_fence\n outer_fence_ue = q3+outer_fence\n \n outliers_prob = []\n outliers_poss = []\n for index, x in enumerate(df[variable]):\n if x <= outer_fence_le or x >= outer_fence_ue:\n outliers_prob.append(index)\n for index, x in enumerate(df[variable]):\n if x <= inner_fence_le or x >= inner_fence_ue:\n outliers_poss.append(index)\n return outliers_prob, outliers_poss\n","metadata":{"papermill":{"duration":0.108455,"end_time":"2022-02-23T02:27:37.524004","exception":false,"start_time":"2022-02-23T02:27:37.415549","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.34011Z","iopub.execute_input":"2022-05-15T12:18:18.34147Z","iopub.status.idle":"2022-05-15T12:18:18.351008Z","shell.execute_reply.started":"2022-05-15T12:18:18.341428Z","shell.execute_reply":"2022-05-15T12:18:18.349793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id = df.groupby(\"building_id\").get_group(108)\n\nprobable_outliers_tm, possible_outliers_tm = tukeys_method(random_building_id, \"meter_reading\")\n\nprint(probable_outliers_tm)\nprint(\"*****************************************************************************************\")\nprint(possible_outliers_tm)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:18.353759Z","iopub.execute_input":"2022-05-15T12:18:18.354139Z","iopub.status.idle":"2022-05-15T12:18:18.507396Z","shell.execute_reply.started":"2022-05-15T12:18:18.354092Z","shell.execute_reply":"2022-05-15T12:18:18.50607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(probable_outliers_tm)","metadata":{"papermill":{"duration":0.106331,"end_time":"2022-02-23T02:27:37.952214","exception":false,"start_time":"2022-02-23T02:27:37.845883","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.50881Z","iopub.execute_input":"2022-05-15T12:18:18.509052Z","iopub.status.idle":"2022-05-15T12:18:18.516326Z","shell.execute_reply.started":"2022-05-15T12:18:18.509024Z","shell.execute_reply":"2022-05-15T12:18:18.515432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(possible_outliers_tm)","metadata":{"papermill":{"duration":0.107396,"end_time":"2022-02-23T02:27:38.158705","exception":false,"start_time":"2022-02-23T02:27:38.051309","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.517795Z","iopub.execute_input":"2022-05-15T12:18:18.518891Z","iopub.status.idle":"2022-05-15T12:18:18.531143Z","shell.execute_reply.started":"2022-05-15T12:18:18.518839Z","shell.execute_reply":"2022-05-15T12:18:18.530371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomaly = pd.DataFrame(possible_outliers_tm)\nanomaly['Anomaly'] = 1\nanomaly.set_index(0, inplace=True)","metadata":{"papermill":{"duration":0.107279,"end_time":"2022-02-23T02:27:38.365338","exception":false,"start_time":"2022-02-23T02:27:38.258059","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.532396Z","iopub.execute_input":"2022-05-15T12:18:18.532951Z","iopub.status.idle":"2022-05-15T12:18:18.547256Z","shell.execute_reply.started":"2022-05-15T12:18:18.532902Z","shell.execute_reply":"2022-05-15T12:18:18.546404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id = pd.concat([random_building_id.reset_index(), anomaly], axis=1)","metadata":{"papermill":{"duration":0.111106,"end_time":"2022-02-23T02:27:38.57626","exception":false,"start_time":"2022-02-23T02:27:38.465154","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.548756Z","iopub.execute_input":"2022-05-15T12:18:18.549231Z","iopub.status.idle":"2022-05-15T12:18:18.569904Z","shell.execute_reply.started":"2022-05-15T12:18:18.549139Z","shell.execute_reply":"2022-05-15T12:18:18.568576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:18.571563Z","iopub.execute_input":"2022-05-15T12:18:18.57179Z","iopub.status.idle":"2022-05-15T12:18:18.590773Z","shell.execute_reply.started":"2022-05-15T12:18:18.571763Z","shell.execute_reply":"2022-05-15T12:18:18.589776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id['Anomaly'] = random_building_id['Anomaly'].replace(np.nan, False)\nrandom_building_id['Anomaly'] = random_building_id['Anomaly'].replace(1.0, True)","metadata":{"papermill":{"duration":0.115508,"end_time":"2022-02-23T02:27:38.790926","exception":false,"start_time":"2022-02-23T02:27:38.675418","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.592146Z","iopub.execute_input":"2022-05-15T12:18:18.592874Z","iopub.status.idle":"2022-05-15T12:18:18.603175Z","shell.execute_reply.started":"2022-05-15T12:18:18.592824Z","shell.execute_reply":"2022-05-15T12:18:18.601732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id['Anomaly'].value_counts()","metadata":{"papermill":{"duration":0.110899,"end_time":"2022-02-23T02:27:39.21823","exception":false,"start_time":"2022-02-23T02:27:39.107331","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.605412Z","iopub.execute_input":"2022-05-15T12:18:18.606029Z","iopub.status.idle":"2022-05-15T12:18:18.621901Z","shell.execute_reply.started":"2022-05-15T12:18:18.605977Z","shell.execute_reply":"2022-05-15T12:18:18.620994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(30,10))\n\nanomaly = random_building_id.loc[random_building_id['Anomaly'] == True, ['meter_reading']] \n\nax.plot(random_building_id.index, random_building_id['meter_reading'], color='blue', label = 'Normal')\nax.scatter(anomaly.index,anomaly['meter_reading'], color='red', label = 'Anomaly')\nax.set_title('Random Building Id(108) anomalies points for box blot method', fontsize=20)\nplt.legend()\nplt.show();","metadata":{"papermill":{"duration":0.431309,"end_time":"2022-02-23T02:27:39.750756","exception":false,"start_time":"2022-02-23T02:27:39.319447","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.623222Z","iopub.execute_input":"2022-05-15T12:18:18.623921Z","iopub.status.idle":"2022-05-15T12:18:18.975818Z","shell.execute_reply.started":"2022-05-15T12:18:18.623878Z","shell.execute_reply":"2022-05-15T12:18:18.974659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig1 = px.line(random_building_id, y=\"meter_reading\")\n# fig1.update_traces(line=dict(color = 'magenta'))\n\n# anomaly = random_building_id.loc[random_building_id['Anomaly'] == True, ['meter_reading']] \n# fig2 = px.scatter(anomaly,y=\"meter_reading\")\n\n# fig3 = go.Figure(data=fig1.data + fig2.data)\n# fig3.update_layout(title=\"Random Building Id(108) anomalies points for box blot method\")\n# fig3.show()","metadata":{"papermill":{"duration":0.112908,"end_time":"2022-02-23T02:27:39.989619","exception":false,"start_time":"2022-02-23T02:27:39.876711","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:18.977607Z","iopub.execute_input":"2022-05-15T12:18:18.978332Z","iopub.status.idle":"2022-05-15T12:18:18.983192Z","shell.execute_reply.started":"2022-05-15T12:18:18.978269Z","shell.execute_reply":"2022-05-15T12:18:18.982477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"my first model predication for only one building has accurcy \",round((random_building_id[random_building_id['anomaly'] == random_building_id['Anomaly']].shape[0]/random_building_id.shape[0])*100,2),\"%\")","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:18.9846Z","iopub.execute_input":"2022-05-15T12:18:18.985074Z","iopub.status.idle":"2022-05-15T12:18:19.000745Z","shell.execute_reply.started":"2022-05-15T12:18:18.985043Z","shell.execute_reply":"2022-05-15T12:18:18.99964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Second apply on all grids","metadata":{"papermill":{"duration":0.103988,"end_time":"2022-02-23T02:27:40.198544","exception":false,"start_time":"2022-02-23T02:27:40.094556","status":"completed"},"tags":[]}},{"cell_type":"code","source":"All_buildings = df.groupby(\"building_id\")\nbuildings = list(All_buildings.groups.keys())","metadata":{"papermill":{"duration":0.184959,"end_time":"2022-02-23T02:27:47.378698","exception":false,"start_time":"2022-02-23T02:27:47.193739","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:19.002185Z","iopub.execute_input":"2022-05-15T12:18:19.002467Z","iopub.status.idle":"2022-05-15T12:18:19.086305Z","shell.execute_reply.started":"2022-05-15T12:18:19.002435Z","shell.execute_reply":"2022-05-15T12:18:19.085331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data=[]\nfor building in buildings:\n All_buildings = df.groupby(\"building_id\").get_group(building)\n data.append(All_buildings)","metadata":{"papermill":{"duration":0.164109,"end_time":"2022-02-23T02:27:47.908944","exception":false,"start_time":"2022-02-23T02:27:47.744835","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:19.087856Z","iopub.execute_input":"2022-05-15T12:18:19.088104Z","iopub.status.idle":"2022-05-15T12:18:38.400746Z","shell.execute_reply.started":"2022-05-15T12:18:19.088074Z","shell.execute_reply":"2022-05-15T12:18:38.399983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x=len(buildings)\nx","metadata":{"papermill":{"duration":0.113008,"end_time":"2022-02-23T02:27:48.128826","exception":false,"start_time":"2022-02-23T02:27:48.015818","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:38.40235Z","iopub.execute_input":"2022-05-15T12:18:38.402807Z","iopub.status.idle":"2022-05-15T12:18:38.408788Z","shell.execute_reply.started":"2022-05-15T12:18:38.402772Z","shell.execute_reply":"2022-05-15T12:18:38.407698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"buildings","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:38.410153Z","iopub.execute_input":"2022-05-15T12:18:38.41092Z","iopub.status.idle":"2022-05-15T12:18:38.427229Z","shell.execute_reply.started":"2022-05-15T12:18:38.410881Z","shell.execute_reply":"2022-05-15T12:18:38.426451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test Tukey’s box plot method\n\ndata_2=pd.DataFrame()\nanomalies= pd.DataFrame()\nfor i in range(x):\n probable_outliers_tm, possible_outliers_tm = tukeys_method(data[i], \"meter_reading\")\n \n if len(possible_outliers_tm)==0:\n print(\"================ NO Anomaly in building number {} ================\".format(i+1))\n print(\"=================================================================\")\n anomaly = pd.DataFrame(possible_outliers_tm)\n data_2 = pd.concat([data[i].reset_index(drop=True), anomaly], axis=1)\n anomalies=anomalies.append(data_2)\n continue\n\n \n anomaly = pd.DataFrame(possible_outliers_tm)\n anomaly['Anomaly'] = 1\n anomaly.set_index(0, inplace=True)\n\n data_2 = pd.concat([data[i].reset_index(drop=True), anomaly], axis=1)\n \n print(\"==================== building number {} done ====================\".format(i+1))\n\n print(\"-\" *14,\"have number of anomalies points \",data_2['Anomaly'].value_counts().index[0],\"-\" *14)\n print(\"=================================================================\")\n \n anomalies=anomalies.append(data_2)\n #print(data_2)\n #print(anomalies)","metadata":{"papermill":{"duration":0.282653,"end_time":"2022-02-23T02:27:48.515727","exception":false,"start_time":"2022-02-23T02:27:48.233074","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:38.428804Z","iopub.execute_input":"2022-05-15T12:18:38.429267Z","iopub.status.idle":"2022-05-15T12:18:44.31863Z","shell.execute_reply.started":"2022-05-15T12:18:38.429221Z","shell.execute_reply":"2022-05-15T12:18:44.317422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"check if the results are true","metadata":{}},{"cell_type":"code","source":"anomalies.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:44.320232Z","iopub.execute_input":"2022-05-15T12:18:44.32054Z","iopub.status.idle":"2022-05-15T12:18:44.327162Z","shell.execute_reply.started":"2022-05-15T12:18:44.320507Z","shell.execute_reply":"2022-05-15T12:18:44.32634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:44.328826Z","iopub.execute_input":"2022-05-15T12:18:44.329077Z","iopub.status.idle":"2022-05-15T12:18:44.340924Z","shell.execute_reply.started":"2022-05-15T12:18:44.329047Z","shell.execute_reply":"2022-05-15T12:18:44.340192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies.groupby(\"building_id\").get_group(1323)[\"Anomaly\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:44.342258Z","iopub.execute_input":"2022-05-15T12:18:44.342521Z","iopub.status.idle":"2022-05-15T12:18:44.489437Z","shell.execute_reply.started":"2022-05-15T12:18:44.342489Z","shell.execute_reply":"2022-05-15T12:18:44.488554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies.groupby(\"building_id\").get_group(1007)[\"Anomaly\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:44.490704Z","iopub.execute_input":"2022-05-15T12:18:44.490942Z","iopub.status.idle":"2022-05-15T12:18:44.565668Z","shell.execute_reply.started":"2022-05-15T12:18:44.490912Z","shell.execute_reply":"2022-05-15T12:18:44.564599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.groupby(\"building_id\").get_group(1007).shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:44.567321Z","iopub.execute_input":"2022-05-15T12:18:44.567663Z","iopub.status.idle":"2022-05-15T12:18:44.691691Z","shell.execute_reply.started":"2022-05-15T12:18:44.56762Z","shell.execute_reply":"2022-05-15T12:18:44.690696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies.groupby(\"building_id\").get_group(1007).shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:44.693015Z","iopub.execute_input":"2022-05-15T12:18:44.69349Z","iopub.status.idle":"2022-05-15T12:18:44.764898Z","shell.execute_reply.started":"2022-05-15T12:18:44.693454Z","shell.execute_reply":"2022-05-15T12:18:44.764047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies.reset_index(drop=True,inplace=True)","metadata":{"papermill":{"duration":0.112716,"end_time":"2022-02-23T02:27:48.952435","exception":false,"start_time":"2022-02-23T02:27:48.839719","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:44.766187Z","iopub.execute_input":"2022-05-15T12:18:44.766579Z","iopub.status.idle":"2022-05-15T12:18:44.770939Z","shell.execute_reply.started":"2022-05-15T12:18:44.766541Z","shell.execute_reply":"2022-05-15T12:18:44.769935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies['Anomaly'] = anomalies['Anomaly'].replace(np.nan, False)\nanomalies['Anomaly'] = anomalies['Anomaly'].replace(1.0, True)","metadata":{"papermill":{"duration":0.132628,"end_time":"2022-02-23T02:27:49.191636","exception":false,"start_time":"2022-02-23T02:27:49.059008","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:44.772251Z","iopub.execute_input":"2022-05-15T12:18:44.772521Z","iopub.status.idle":"2022-05-15T12:18:45.318883Z","shell.execute_reply.started":"2022-05-15T12:18:44.772491Z","shell.execute_reply":"2022-05-15T12:18:45.317928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies['Anomaly'].value_counts()","metadata":{"papermill":{"duration":0.115545,"end_time":"2022-02-23T02:27:49.414171","exception":false,"start_time":"2022-02-23T02:27:49.298626","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:45.320159Z","iopub.execute_input":"2022-05-15T12:18:45.320507Z","iopub.status.idle":"2022-05-15T12:18:45.33826Z","shell.execute_reply.started":"2022-05-15T12:18:45.320469Z","shell.execute_reply":"2022-05-15T12:18:45.337534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies['anomaly'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:45.339732Z","iopub.execute_input":"2022-05-15T12:18:45.34075Z","iopub.status.idle":"2022-05-15T12:18:45.362527Z","shell.execute_reply.started":"2022-05-15T12:18:45.340704Z","shell.execute_reply":"2022-05-15T12:18:45.361607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies['anomaly'] = anomalies['anomaly'].replace(0, False)\nanomalies['anomaly'] = anomalies['anomaly'].replace(1, True)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:45.364054Z","iopub.execute_input":"2022-05-15T12:18:45.365006Z","iopub.status.idle":"2022-05-15T12:18:45.756684Z","shell.execute_reply.started":"2022-05-15T12:18:45.364956Z","shell.execute_reply":"2022-05-15T12:18:45.755634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anomalies['anomaly'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:45.758524Z","iopub.execute_input":"2022-05-15T12:18:45.758894Z","iopub.status.idle":"2022-05-15T12:18:45.776593Z","shell.execute_reply.started":"2022-05-15T12:18:45.758847Z","shell.execute_reply":"2022-05-15T12:18:45.775754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig, ax = plt.subplots(figsize=(30,10))\n# ax.plot(df.index, df['meter_reading'], color='blue', label = 'Normal')\n# ax.set_title('Total grids', fontsize=20)\n# plt.show();","metadata":{"papermill":{"duration":0.449527,"end_time":"2022-02-23T02:27:49.970784","exception":false,"start_time":"2022-02-23T02:27:49.521257","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:45.777965Z","iopub.execute_input":"2022-05-15T12:18:45.77821Z","iopub.status.idle":"2022-05-15T12:18:45.789047Z","shell.execute_reply.started":"2022-05-15T12:18:45.77818Z","shell.execute_reply":"2022-05-15T12:18:45.787948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(30,10))\n\nanomaly = anomalies.loc[anomalies['Anomaly'] == True, ['meter_reading']] \n\nax.plot(anomalies.index, anomalies['meter_reading'], color='blue', label = 'Normal')\nax.scatter(anomaly.index,anomaly['meter_reading'], color='red', label = 'Anomaly')\nax.set_title('Total buldings anomalies points for box blot method', fontsize=20)\nplt.legend()\nplt.show();","metadata":{"papermill":{"duration":0.530877,"end_time":"2022-02-23T02:27:50.614669","exception":false,"start_time":"2022-02-23T02:27:50.083792","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:45.790594Z","iopub.execute_input":"2022-05-15T12:18:45.791212Z","iopub.status.idle":"2022-05-15T12:18:47.721966Z","shell.execute_reply.started":"2022-05-15T12:18:45.791167Z","shell.execute_reply":"2022-05-15T12:18:47.720811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig1 = px.line(df, y=\"meter_reading\")\n# fig1.update_traces(line=dict(color = 'rgba(250,100,10,0.2)'))\n# anomaly = anomalies.loc[anomalies['Anomaly'] == -1, ['meter_reading']] \n\n# fig2 = px.scatter(anomaly,y='meter_reading')\n\n# fig3 = go.Figure(data=fig1.data + fig2.data)\n# fig3.update_layout(title=\"Total buldings anomalies points for isolation forest\")\n# fig3.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:47.723567Z","iopub.execute_input":"2022-05-15T12:18:47.723862Z","iopub.status.idle":"2022-05-15T12:18:47.729647Z","shell.execute_reply.started":"2022-05-15T12:18:47.723825Z","shell.execute_reply":"2022-05-15T12:18:47.728601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"my first model predication has accurcy \",round((anomalies[anomalies['anomaly'] == anomalies['Anomaly']].shape[0]/anomalies.shape[0])*100,2),\"%\")","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:47.731083Z","iopub.execute_input":"2022-05-15T12:18:47.73137Z","iopub.status.idle":"2022-05-15T12:18:47.792158Z","shell.execute_reply.started":"2022-05-15T12:18:47.731331Z","shell.execute_reply":"2022-05-15T12:18:47.791369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

    2- Isolation forest.

    \n\n\nIn this method we will depend on the detection using some Machine Learning algorithms. In the \nwe will depend on Isolation Forest. \n\nOne of those algorithms is the Isolation Forest method. Isolation Forest build using the decision \ntrees which depend on the points that go deeper into the tree are not anomalies and points \nwhich go short distance have big probability to be anomalies, and it is unsupervised learning \nmodel which used without labeled data. \n\nThe algorithm goes by selecting a sample of the dataset then branch it on the binary tress by \nsetting a threshold if the sample we selected is less than this threshold it will be in the left \nbranch and if it not it will be in the right branch. This process repeated until we every point in \nthe dataset is isolated. ","metadata":{"papermill":{"duration":0.11352,"end_time":"2022-02-23T02:27:50.844607","exception":false,"start_time":"2022-02-23T02:27:50.731087","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nfrom sklearn.covariance import EllipticEnvelope\nfrom sklearn.ensemble import IsolationForest","metadata":{"papermill":{"duration":0.128337,"end_time":"2022-02-23T02:27:51.089934","exception":false,"start_time":"2022-02-23T02:27:50.961597","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:47.793238Z","iopub.execute_input":"2022-05-15T12:18:47.793852Z","iopub.status.idle":"2022-05-15T12:18:47.906346Z","shell.execute_reply.started":"2022-05-15T12:18:47.793818Z","shell.execute_reply":"2022-05-15T12:18:47.905153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### First apply on a random grid","metadata":{"papermill":{"duration":0.23394,"end_time":"2022-02-23T02:27:58.286462","exception":false,"start_time":"2022-02-23T02:27:58.052522","status":"completed"},"tags":[]}},{"cell_type":"code","source":"random_building_id = df.groupby(\"building_id\").get_group(108)\nrandom_building_id.info()","metadata":{"papermill":{"duration":0.147084,"end_time":"2022-02-23T02:27:58.672312","exception":false,"start_time":"2022-02-23T02:27:58.525228","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:47.907973Z","iopub.execute_input":"2022-05-15T12:18:47.908218Z","iopub.status.idle":"2022-05-15T12:18:48.022672Z","shell.execute_reply.started":"2022-05-15T12:18:47.90819Z","shell.execute_reply":"2022-05-15T12:18:48.021417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train isolation forest\noutliers_fraction = float(.01)\nscaler = StandardScaler()\nnp_scaled = scaler.fit_transform(random_building_id[\"meter_reading\"].values.reshape(-1, 1))\ndf_data = pd.DataFrame(np_scaled)\n\nmodel = IsolationForest(contamination=outliers_fraction)\nmodel.fit(df_data) ","metadata":{"papermill":{"duration":0.575338,"end_time":"2022-02-23T02:27:59.655571","exception":false,"start_time":"2022-02-23T02:27:59.080233","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:48.024362Z","iopub.execute_input":"2022-05-15T12:18:48.0246Z","iopub.status.idle":"2022-05-15T12:18:48.484951Z","shell.execute_reply.started":"2022-05-15T12:18:48.024572Z","shell.execute_reply":"2022-05-15T12:18:48.483883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predict isolation forest\nAnomaly = model.predict(df_data)","metadata":{"papermill":{"duration":0.341732,"end_time":"2022-02-23T02:28:00.130368","exception":false,"start_time":"2022-02-23T02:27:59.788636","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:48.486652Z","iopub.execute_input":"2022-05-15T12:18:48.487164Z","iopub.status.idle":"2022-05-15T12:18:48.719643Z","shell.execute_reply.started":"2022-05-15T12:18:48.487131Z","shell.execute_reply":"2022-05-15T12:18:48.718531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Anomaly = pd.DataFrame(Anomaly,columns=['Anomaly'])\nAnomaly","metadata":{"papermill":{"duration":0.130892,"end_time":"2022-02-23T02:28:00.379857","exception":false,"start_time":"2022-02-23T02:28:00.248965","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:48.721176Z","iopub.execute_input":"2022-05-15T12:18:48.72145Z","iopub.status.idle":"2022-05-15T12:18:48.73405Z","shell.execute_reply.started":"2022-05-15T12:18:48.72142Z","shell.execute_reply":"2022-05-15T12:18:48.732637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id = pd.concat([random_building_id.reset_index(), Anomaly], axis=1)\nrandom_building_id","metadata":{"papermill":{"duration":0.133397,"end_time":"2022-02-23T02:28:00.630517","exception":false,"start_time":"2022-02-23T02:28:00.49712","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:48.735816Z","iopub.execute_input":"2022-05-15T12:18:48.73616Z","iopub.status.idle":"2022-05-15T12:18:48.762582Z","shell.execute_reply.started":"2022-05-15T12:18:48.736114Z","shell.execute_reply":"2022-05-15T12:18:48.761375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id['Anomaly'].value_counts()","metadata":{"papermill":{"duration":0.131508,"end_time":"2022-02-23T02:28:00.880646","exception":false,"start_time":"2022-02-23T02:28:00.749138","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:48.764269Z","iopub.execute_input":"2022-05-15T12:18:48.7651Z","iopub.status.idle":"2022-05-15T12:18:48.774082Z","shell.execute_reply.started":"2022-05-15T12:18:48.765034Z","shell.execute_reply":"2022-05-15T12:18:48.773157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig, ax = plt.subplots(figsize=(30,10))\n\nanomaly = random_building_id.loc[random_building_id['Anomaly'] == -1, ['meter_reading']] \n\nax.plot(random_building_id.index, random_building_id['meter_reading'], color='blue', label = 'Normal')\nax.scatter(anomaly.index,anomaly['meter_reading'], color='red', label = 'Anomaly')\nax.set_title('Random Building Id(108) anomalies points', fontsize=20)\nplt.legend()\nplt.show();","metadata":{"papermill":{"duration":0.841441,"end_time":"2022-02-23T02:28:01.839699","exception":false,"start_time":"2022-02-23T02:28:00.998258","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:48.775608Z","iopub.execute_input":"2022-05-15T12:18:48.776745Z","iopub.status.idle":"2022-05-15T12:18:49.130896Z","shell.execute_reply.started":"2022-05-15T12:18:48.776687Z","shell.execute_reply":"2022-05-15T12:18:49.130095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id['Anomaly'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:49.132151Z","iopub.execute_input":"2022-05-15T12:18:49.132417Z","iopub.status.idle":"2022-05-15T12:18:49.142873Z","shell.execute_reply.started":"2022-05-15T12:18:49.132386Z","shell.execute_reply":"2022-05-15T12:18:49.141368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_building_id['Anomaly'] = random_building_id['Anomaly'].replace(1, 0)\nrandom_building_id['Anomaly'] = random_building_id['Anomaly'].replace(-1, 1)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:49.145342Z","iopub.execute_input":"2022-05-15T12:18:49.146001Z","iopub.status.idle":"2022-05-15T12:18:49.159117Z","shell.execute_reply.started":"2022-05-15T12:18:49.145938Z","shell.execute_reply":"2022-05-15T12:18:49.157664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"my first model predication for only one building has accurcy \",round((random_building_id[random_building_id['anomaly'] == random_building_id['Anomaly']].shape[0]/random_building_id.shape[0])*100,2),\"%\")","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:18:49.160987Z","iopub.execute_input":"2022-05-15T12:18:49.162054Z","iopub.status.idle":"2022-05-15T12:18:49.175702Z","shell.execute_reply.started":"2022-05-15T12:18:49.161999Z","shell.execute_reply":"2022-05-15T12:18:49.174008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Second apply on all grids ","metadata":{"papermill":{"duration":0.122045,"end_time":"2022-02-23T02:28:02.088964","exception":false,"start_time":"2022-02-23T02:28:01.966919","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data=[]\nfor building in buildings:\n All_buildings = df.groupby(\"building_id\").get_group(building)\n data.append(All_buildings)\n#data","metadata":{"papermill":{"duration":0.197691,"end_time":"2022-02-23T02:28:02.40871","exception":false,"start_time":"2022-02-23T02:28:02.211019","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:18:49.177649Z","iopub.execute_input":"2022-05-15T12:18:49.178811Z","iopub.status.idle":"2022-05-15T12:19:10.185396Z","shell.execute_reply.started":"2022-05-15T12:18:49.178762Z","shell.execute_reply":"2022-05-15T12:19:10.184335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[0]","metadata":{"papermill":{"duration":0.1377,"end_time":"2022-02-23T02:28:02.671555","exception":false,"start_time":"2022-02-23T02:28:02.533855","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:19:10.186748Z","iopub.execute_input":"2022-05-15T12:19:10.187005Z","iopub.status.idle":"2022-05-15T12:19:10.203134Z","shell.execute_reply.started":"2022-05-15T12:19:10.186972Z","shell.execute_reply":"2022-05-15T12:19:10.202093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train isolation forest\nfull_anomaly = pd.DataFrame()\nfor i in range (x):\n \n All_buildings = data[i]\n \n outliers_fraction = float(.01)\n\n scaler = StandardScaler()\n np_scaled = scaler.fit_transform(All_buildings[\"meter_reading\"].values.reshape(-1, 1))\n df_data = pd.DataFrame(np_scaled)\n\n model = IsolationForest(contamination=outliers_fraction)\n model.fit(df_data) \n Anomaly = model.predict(df_data)\n Anomaly = pd.DataFrame(Anomaly,columns=['Anomaly'])\n print(\"========== building number {} predicted ==========\".format(i+1))\n full_anomaly = full_anomaly.append(Anomaly)","metadata":{"papermill":{"duration":6.00266,"end_time":"2022-02-23T02:28:08.797603","exception":false,"start_time":"2022-02-23T02:28:02.794943","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:19:10.204401Z","iopub.execute_input":"2022-05-15T12:19:10.204729Z","iopub.status.idle":"2022-05-15T12:21:27.472977Z","shell.execute_reply.started":"2022-05-15T12:19:10.204694Z","shell.execute_reply":"2022-05-15T12:21:27.471769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_anomaly.reset_index(inplace= True)\nfull_anomaly.drop('index', axis=1, inplace=True)\nfull_anomaly","metadata":{"papermill":{"duration":0.254967,"end_time":"2022-02-23T02:28:09.203225","exception":false,"start_time":"2022-02-23T02:28:08.948258","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:21:27.474753Z","iopub.execute_input":"2022-05-15T12:21:27.475114Z","iopub.status.idle":"2022-05-15T12:21:27.521849Z","shell.execute_reply.started":"2022-05-15T12:21:27.475079Z","shell.execute_reply":"2022-05-15T12:21:27.520902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full_anomaly = pd.concat([df.reset_index(), full_anomaly], axis=1)\ndf_full_anomaly","metadata":{"papermill":{"duration":0.142836,"end_time":"2022-02-23T02:28:09.865036","exception":false,"start_time":"2022-02-23T02:28:09.7222","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:21:27.523458Z","iopub.execute_input":"2022-05-15T12:21:27.523805Z","iopub.status.idle":"2022-05-15T12:21:27.60884Z","shell.execute_reply.started":"2022-05-15T12:21:27.523772Z","shell.execute_reply":"2022-05-15T12:21:27.608157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig1 = px.line(df_full_anomaly, y=\"meter_reading\")\n# fig1.update_traces(line=dict(color = 'rgba(250,100,10,0.2)'))\n# fig1.update_layout(title=\"Total buildings\")\n# fig1.show()","metadata":{"papermill":{"duration":0.697733,"end_time":"2022-02-23T02:28:10.955437","exception":false,"start_time":"2022-02-23T02:28:10.257704","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:21:27.610264Z","iopub.execute_input":"2022-05-15T12:21:27.611665Z","iopub.status.idle":"2022-05-15T12:21:27.616688Z","shell.execute_reply.started":"2022-05-15T12:21:27.611598Z","shell.execute_reply":"2022-05-15T12:21:27.615534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig1 = px.line(df_full_anomaly, y=\"meter_reading\")\n# fig1.update_traces(line=dict(color = 'rgba(250,100,10,0.2)'))\n# anomaly = df_full_anomaly.loc[df_full_anomaly['Anomaly'] == -1, ['meter_reading']] \n\n# fig2 = px.scatter(anomaly,y='meter_reading')\n\n# fig3 = go.Figure(data=fig1.data + fig2.data)\n# fig3.update_layout(title=\"Total buildings anomalies points for isolation forest\")\n# fig3.show()","metadata":{"papermill":{"duration":0.753744,"end_time":"2022-02-23T02:28:11.871803","exception":false,"start_time":"2022-02-23T02:28:11.118059","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:21:27.618701Z","iopub.execute_input":"2022-05-15T12:21:27.619221Z","iopub.status.idle":"2022-05-15T12:21:27.629699Z","shell.execute_reply.started":"2022-05-15T12:21:27.619177Z","shell.execute_reply":"2022-05-15T12:21:27.628807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig, ax = plt.subplots(figsize=(30,10))\n\nanomaly = df_full_anomaly.loc[df_full_anomaly['Anomaly'] == -1, ['meter_reading']] \n\nax.plot(df_full_anomaly.index, df_full_anomaly['meter_reading'], color='blue', label = 'Normal')\nax.scatter(anomaly.index,anomaly['meter_reading'], color='red', label = 'Anomaly')\nax.set_title('All buildings anomalies points', fontsize=20)\nplt.legend()\nplt.show();","metadata":{"papermill":{"duration":0.198521,"end_time":"2022-02-23T02:28:12.265066","exception":false,"start_time":"2022-02-23T02:28:12.066545","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-15T12:21:27.630807Z","iopub.execute_input":"2022-05-15T12:21:27.631493Z","iopub.status.idle":"2022-05-15T12:21:32.104731Z","shell.execute_reply.started":"2022-05-15T12:21:27.631456Z","shell.execute_reply":"2022-05-15T12:21:32.103609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full_anomaly['Anomaly'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:21:32.106297Z","iopub.execute_input":"2022-05-15T12:21:32.106539Z","iopub.status.idle":"2022-05-15T12:21:32.123984Z","shell.execute_reply.started":"2022-05-15T12:21:32.106511Z","shell.execute_reply":"2022-05-15T12:21:32.123359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full_anomaly['Anomaly'] = df_full_anomaly['Anomaly'].replace(1, 0)\ndf_full_anomaly['Anomaly'] = df_full_anomaly['Anomaly'].replace(-1, 1)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:21:32.12505Z","iopub.execute_input":"2022-05-15T12:21:32.126535Z","iopub.status.idle":"2022-05-15T12:21:32.186636Z","shell.execute_reply.started":"2022-05-15T12:21:32.126458Z","shell.execute_reply":"2022-05-15T12:21:32.185728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"my first model predication for only one building has accurcy \",round((df_full_anomaly[df_full_anomaly['anomaly'] == df_full_anomaly['Anomaly']].shape[0]/df.shape[0])*100,2),\"%\")","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:21:32.197076Z","iopub.execute_input":"2022-05-15T12:21:32.198235Z","iopub.status.idle":"2022-05-15T12:21:32.276031Z","shell.execute_reply.started":"2022-05-15T12:21:32.198177Z","shell.execute_reply":"2022-05-15T12:21:32.274705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

    3- Anomly based on Machine learning Predication.

    \n","metadata":{}},{"cell_type":"code","source":"# Load libraries\n\nfrom pandas import set_option\nfrom pandas.plotting import scatter_matrix\n\nfrom sklearn.preprocessing import StandardScaler\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\n\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.svm import SVC\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.ensemble import ExtraTreesClassifier\n\nSEED = 7\nnp.random.seed(SEED)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:02.245530Z","iopub.execute_input":"2022-05-15T15:26:02.246393Z","iopub.status.idle":"2022-05-15T15:26:02.257155Z","shell.execute_reply.started":"2022-05-15T15:26:02.246337Z","shell.execute_reply":"2022-05-15T15:26:02.256366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/energy-anomaly-detection/train_features.csv')\ntest = pd.read_csv('../input/energy-anomaly-detection/test_features.csv')\n\n#take sample to test the code quickly\n# train = train.iloc[:train.shape[0]//1500,:]\n# test = test.iloc[:test.shape[0]//1500,:]\n","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:04.677371Z","iopub.execute_input":"2022-05-15T15:26:04.678324Z","iopub.status.idle":"2022-05-15T15:26:56.213440Z","shell.execute_reply.started":"2022-05-15T15:26:04.678271Z","shell.execute_reply":"2022-05-15T15:26:56.211937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:22:27.449293Z","iopub.execute_input":"2022-05-15T12:22:27.449873Z","iopub.status.idle":"2022-05-15T12:22:27.489977Z","shell.execute_reply.started":"2022-05-15T12:22:27.449819Z","shell.execute_reply":"2022-05-15T12:22:27.488711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:22:27.491693Z","iopub.execute_input":"2022-05-15T12:22:27.492387Z","iopub.status.idle":"2022-05-15T12:22:27.499236Z","shell.execute_reply.started":"2022-05-15T12:22:27.492339Z","shell.execute_reply":"2022-05-15T12:22:27.498414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"take only important features","metadata":{}},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:22:27.501013Z","iopub.execute_input":"2022-05-15T12:22:27.501384Z","iopub.status.idle":"2022-05-15T12:22:27.515172Z","shell.execute_reply.started":"2022-05-15T12:22:27.501337Z","shell.execute_reply":"2022-05-15T12:22:27.514298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train[['building_id', 'meter_reading', 'square_feet',\n 'year_built', 'floor_count','air_temperature', 'dew_temperature',\n 'precip_depth_1_hr', 'sea_level_pressure','is_holiday','anomaly']]\n\ntest = test[['building_id', 'meter_reading','square_feet',\n 'year_built', 'floor_count','air_temperature', 'dew_temperature',\n 'precip_depth_1_hr', 'sea_level_pressure','is_holiday']]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.216904Z","iopub.execute_input":"2022-05-15T15:26:56.217587Z","iopub.status.idle":"2022-05-15T15:26:56.545346Z","shell.execute_reply.started":"2022-05-15T15:26:56.217523Z","shell.execute_reply":"2022-05-15T15:26:56.544184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def HeatMap(df,x=True):\n correlations = df.corr()\n ## Create color map ranging between two colors\n cmap = sns.diverging_palette(220, 10, as_cmap=True)\n fig, ax = plt.subplots(figsize=(10, 10))\n fig = sns.heatmap(correlations, cmap=cmap, vmax=1.0, center=0, fmt='.2f',square=True, linewidths=.5, annot=x, cbar_kws={\"shrink\": .75})\n fig.set_xticklabels(fig.get_xticklabels(), rotation = 90, fontsize = 10)\n fig.set_yticklabels(fig.get_yticklabels(), rotation = 0, fontsize = 10)\n plt.tight_layout()\n plt.show()\n\nHeatMap(train,x=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:22:27.797134Z","iopub.execute_input":"2022-05-15T12:22:27.797457Z","iopub.status.idle":"2022-05-15T12:22:29.536389Z","shell.execute_reply.started":"2022-05-15T12:22:27.797421Z","shell.execute_reply":"2022-05-15T12:22:29.535333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" There is no highly corrolated feature in the data set to remove it.","metadata":{}},{"cell_type":"markdown","source":"there is a problem with missing values so i will try to handel it.","metadata":{}},{"cell_type":"code","source":"missing_val_count_by_column = (train.isnull().sum())\nprint(missing_val_count_by_column[missing_val_count_by_column > 0])","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:22:29.537902Z","iopub.execute_input":"2022-05-15T12:22:29.538194Z","iopub.status.idle":"2022-05-15T12:22:29.581325Z","shell.execute_reply.started":"2022-05-15T12:22:29.538155Z","shell.execute_reply":"2022-05-15T12:22:29.580035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfrom sklearn.impute import SimpleImputer\n\nmy_imputer = SimpleImputer()\ndf_clean = my_imputer.fit_transform(train.iloc[:,1:2])\ndf_name =train.iloc[:,1:2].columns\ndf_clean = pd.DataFrame(df_clean,columns = df_name)\n\nmissing_val_count_by_column = (df_clean.isnull().sum())\nprint(missing_val_count_by_column)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.546665Z","iopub.execute_input":"2022-05-15T15:26:56.546967Z","iopub.status.idle":"2022-05-15T15:26:56.618819Z","shell.execute_reply.started":"2022-05-15T15:26:56.546930Z","shell.execute_reply":"2022-05-15T15:26:56.617750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"meter_reading\"] = df_clean[\"meter_reading\"]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.621119Z","iopub.execute_input":"2022-05-15T15:26:56.621395Z","iopub.status.idle":"2022-05-15T15:26:56.635323Z","shell.execute_reply.started":"2022-05-15T15:26:56.621364Z","shell.execute_reply":"2022-05-15T15:26:56.633868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_val_count_by_column = (test.isnull().sum())\nprint(missing_val_count_by_column[missing_val_count_by_column > 0])","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.637455Z","iopub.execute_input":"2022-05-15T15:26:56.638116Z","iopub.status.idle":"2022-05-15T15:26:56.677181Z","shell.execute_reply.started":"2022-05-15T15:26:56.638071Z","shell.execute_reply":"2022-05-15T15:26:56.676507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.impute import SimpleImputer\n\nmy_imputer = SimpleImputer()\ndf_clean = my_imputer.fit_transform(test.iloc[:,1:2])\ndf_name =test.iloc[:,1:2].columns\ndf_clean = pd.DataFrame(df_clean,columns = df_name)\n\nmissing_val_count_by_column = (df_clean.isnull().sum())\nprint(missing_val_count_by_column)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.678839Z","iopub.execute_input":"2022-05-15T15:26:56.679228Z","iopub.status.idle":"2022-05-15T15:26:56.743726Z","shell.execute_reply.started":"2022-05-15T15:26:56.679189Z","shell.execute_reply":"2022-05-15T15:26:56.742640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test[\"meter_reading\"] = df_clean[\"meter_reading\"]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.745424Z","iopub.execute_input":"2022-05-15T15:26:56.745755Z","iopub.status.idle":"2022-05-15T15:26:56.759139Z","shell.execute_reply.started":"2022-05-15T15:26:56.745710Z","shell.execute_reply":"2022-05-15T15:26:56.758058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Modeling","metadata":{}},{"cell_type":"code","source":"y = train[['anomaly']]\nX = train.drop(['anomaly'], axis = 1)\n\nX_train, X_test, y_train, y_test =train_test_split(X,y,\n test_size=0.25,\n random_state=0,\n stratify=y)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:26:56.760593Z","iopub.execute_input":"2022-05-15T15:26:56.761162Z","iopub.status.idle":"2022-05-15T15:27:07.607193Z","shell.execute_reply.started":"2022-05-15T15:26:56.761119Z","shell.execute_reply":"2022-05-15T15:27:07.606088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\nfrom scipy.stats import uniform\n\nclass GridSearch(object):\n \n def __init__(self,X_train,y_train,model,hyperparameters):\n \n self.X_train = X_train\n self.y_train = y_train\n self.model = model\n self.hyperparameters = hyperparameters\n \n def GridSearch(self):\n cv = 10\n clf = GridSearchCV(self.model,\n self.hyperparameters,\n cv=cv,\n verbose=0,\n n_jobs=-1,\n )\n # Fit randomized search\n best_model = clf.fit(self.X_train, self.y_train.values.ravel())\n message = (best_model.best_score_, best_model.best_params_)\n print(\"Best: %f using %s\" % (message))\n\n return best_model,best_model.best_params_\n \n def BestModelPridict(self,X_test):\n \n best_model,_ = self.GridSearch()\n pred = best_model.predict(X_test)\n return pred\n \n\n \nclass RandomSearch(object):\n \n def __init__(self,X_train,y_train,model,hyperparameters):\n \n self.X_train = X_train\n self.y_train = y_train\n self.model = model\n self.hyperparameters = hyperparameters\n \n def RandomSearch(self):\n # Create randomized search 10-fold cross validation and 100 iterations\n cv = 10\n clf = RandomizedSearchCV(self.model,\n self.hyperparameters,\n random_state=1,\n n_iter=100,\n cv=cv,\n verbose=0,\n n_jobs=-1,\n )\n # Fit randomized search\n best_model = clf.fit(self.X_train, self.y_train.values.ravel())\n message = (best_model.best_score_, best_model.best_params_)\n print(\"Best: %f using %s\" % (message))\n\n return best_model,best_model.best_params_\n \n def BestModelPridict(self,X_test):\n \n best_model,_ = self.RandomSearch()\n pred = best_model.predict(X_test)\n return pred","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:27:07.608584Z","iopub.execute_input":"2022-05-15T15:27:07.608842Z","iopub.status.idle":"2022-05-15T15:27:07.625844Z","shell.execute_reply.started":"2022-05-15T15:27:07.608811Z","shell.execute_reply":"2022-05-15T15:27:07.624644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_KNN = KNeighborsClassifier()\n\nneighbors = [1,2,3,4,5,6,7,8,9]\nparam_grid = dict(n_neighbors=neighbors)\n\nKNN_GridSearch = GridSearch(X_train,y_train,model_KNN,param_grid)\nPrediction_KNN = KNN_GridSearch.BestModelPridict(test)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:28:32.963985Z","iopub.execute_input":"2022-05-15T15:28:32.964473Z","iopub.status.idle":"2022-05-15T15:28:32.969884Z","shell.execute_reply.started":"2022-05-15T15:28:32.964432Z","shell.execute_reply":"2022-05-15T15:28:32.968983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Prediction_KNN = pd.DataFrame(data=Prediction_KNN, columns=[\"anomaly\"])\nPrediction_KNN.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T12:46:57.113474Z","iopub.execute_input":"2022-05-15T12:46:57.11375Z","iopub.status.idle":"2022-05-15T12:46:57.133685Z","shell.execute_reply.started":"2022-05-15T12:46:57.113717Z","shell.execute_reply":"2022-05-15T12:46:57.132426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from scipy.stats import randint\nmax_depth_value = [3, None]\nmax_features_value = [1,2,3]\nmin_samples_leaf_value = [1,2,3]\ncriterion_value = [\"gini\", \"entropy\"]\n\n\nparam_grid = dict(max_depth = max_depth_value,\n max_features = max_features_value,\n min_samples_leaf = min_samples_leaf_value,\n criterion = criterion_value)\n\n\nmodel_DT = DecisionTreeClassifier()\nDT_RandSearch = GridSearch(X_train,y_train,model_DT,param_grid)\nPrediction_DT = DT_RandSearch.BestModelPridict(test)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T13:42:01.928719Z","iopub.execute_input":"2022-05-15T13:42:01.929112Z","iopub.status.idle":"2022-05-15T13:47:25.31165Z","shell.execute_reply.started":"2022-05-15T13:42:01.929063Z","shell.execute_reply":"2022-05-15T13:47:25.310559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Prediction_DT = pd.DataFrame(data=Prediction_DT, columns=[\"anomaly\"])\nPrediction_DT.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T13:47:25.314334Z","iopub.execute_input":"2022-05-15T13:47:25.314918Z","iopub.status.idle":"2022-05-15T13:47:25.335476Z","shell.execute_reply.started":"2022-05-15T13:47:25.314862Z","shell.execute_reply":"2022-05-15T13:47:25.334387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ensemble method","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import VotingClassifier\n\nparam = {'n_neighbors': 2}\nmodel1 = KNeighborsClassifier(**param)\n\nparam = {'criterion': 'entropy', 'max_depth': None, 'max_features': 3, 'min_samples_leaf': 2}\nmodel2 = DecisionTreeClassifier(**param)\n\nmodel3 = ExtraTreesClassifier()\n\nestimators = [ ('KNN',model1),('DT',model2), ('ET',model3)]\n ","metadata":{"execution":{"iopub.status.busy":"2022-05-15T13:00:18.442667Z","iopub.execute_input":"2022-05-15T13:00:18.443052Z","iopub.status.idle":"2022-05-15T13:00:18.450702Z","shell.execute_reply.started":"2022-05-15T13:00:18.443015Z","shell.execute_reply":"2022-05-15T13:00:18.44958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kfold = StratifiedKFold(n_splits=10,shuffle=True, random_state=SEED)\nensemble = VotingClassifier(estimators)\n\nresults = cross_val_score(ensemble, X_train,y_train, cv=kfold)\nprint('Accuracy on train: ',round(results.mean(),4))\n\nensemble_model = ensemble.fit(X,y)\n\nPrediction_EN = ensemble_model.predict(test)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T13:47:58.940877Z","iopub.execute_input":"2022-05-15T13:47:58.941251Z","iopub.status.idle":"2022-05-15T14:30:51.304488Z","shell.execute_reply.started":"2022-05-15T13:47:58.941212Z","shell.execute_reply":"2022-05-15T14:30:51.302855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Prediction_EN = pd.DataFrame(data=Prediction_EN, columns=[\"anomaly\"])\nPrediction_EN.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T14:30:51.31001Z","iopub.execute_input":"2022-05-15T14:30:51.310984Z","iopub.status.idle":"2022-05-15T14:30:51.375296Z","shell.execute_reply.started":"2022-05-15T14:30:51.310899Z","shell.execute_reply":"2022-05-15T14:30:51.373934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stacked model","metadata":{}},{"cell_type":"code","source":"from numpy import hstack\nfrom numpy import vstack\nfrom numpy import asarray\nfrom sklearn.datasets import make_blobs\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.ensemble import BaggingClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.ensemble import ExtraTreesClassifier","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# base-models\ndef get_models():\n models = list()\n# models.append(LogisticRegression(solver='liblinear'))\n models.append(DecisionTreeClassifier())\n# models.append(SVC(gamma='scale', probability=True))\n# models.append(GaussianNB())\n models.append(KNeighborsClassifier())\n# models.append(AdaBoostClassifier())\n models.append(BaggingClassifier(n_estimators=10))\n models.append(RandomForestClassifier(n_estimators=10))\n models.append(ExtraTreesClassifier(n_estimators=10))\n return models\n\n# collect out of fold predictions form k-fold cross validation\ndef get_out_of_fold_predictions(X, y, models):\n meta_X, meta_y = list(), list()\n \n kfold = KFold(n_splits=10, shuffle=True)\n # enumerate splits\n for train_ix, test_ix in kfold.split(X):\n fold_yhats = list()\n # get data\n train_X, test_X = X[train_ix], X[test_ix]\n train_y, test_y = y[train_ix], y[test_ix]\n meta_y.extend(test_y)\n # fit and make predictions with each sub-model\n for model in models:\n model.fit(train_X, train_y)\n yhat = model.predict_proba(test_X)\n # store columns\n fold_yhats.append(yhat)\n # store fold yhats as columns\n meta_X.append(hstack(fold_yhats))\n return vstack(meta_X), asarray(meta_y)\n\n# fit all base models on the training dataset\ndef fit_base_models(X, y, models):\n for model in models:\n model.fit(X, y)\n\n# fit a meta model\ndef fit_meta_model(X, y):\n model = RandomForestClassifier(n_estimators=10)\n model.fit(X, y)\n return model\n\n# evaluate a list of models on a dataset\ndef evaluate_models(X, y, models):\n for model in models:\n yhat = model.predict(X)\n acc = accuracy_score(y, yhat)\n print('%s: %.3f' % (model.__class__.__name__, acc*100))\n\n# make predictions with stacked model\ndef super_learner_predictions(X, models, meta_model):\n meta_X = list()\n for model in models:\n yhat = model.predict_proba(X)\n meta_X.append(yhat)\n meta_X = hstack(meta_X)\n\n return meta_model.predict(meta_X)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T16:32:30.393820Z","iopub.execute_input":"2022-05-15T16:32:30.394185Z","iopub.status.idle":"2022-05-15T16:32:30.409740Z","shell.execute_reply.started":"2022-05-15T16:32:30.394149Z","shell.execute_reply":"2022-05-15T16:32:30.408738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models = get_models()\n\nmeta_X, meta_y = get_out_of_fold_predictions(X_train.to_numpy(), y_train.to_numpy(), models)\n\n# fit base models\nfit_base_models(X_train, y_train, models)\n# fit the meta model\nmeta_model = fit_meta_model(meta_X, meta_y)\n# evaluate base models\nevaluate_models(X_test, y_test, models)\n# evaluate meta model\nPrediction_ST = super_learner_predictions(test, models, meta_model)\n\nprint(\"=\"*100)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T15:33:08.352264Z","iopub.execute_input":"2022-05-15T15:33:08.353325Z","iopub.status.idle":"2022-05-15T16:23:31.228352Z","shell.execute_reply.started":"2022-05-15T15:33:08.353275Z","shell.execute_reply":"2022-05-15T16:23:31.226651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Prediction_ST = pd.DataFrame(data=Prediction_ST, columns=[\"anomaly\"])\nPrediction_ST.value_counts()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submission","metadata":{}},{"cell_type":"code","source":"submission_df = pd.read_csv('../input/energy-anomaly-detection/sample_submission.csv')\nprint(submission_df.shape)\nprint(Prediction_EN.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T14:30:51.376807Z","iopub.execute_input":"2022-05-15T14:30:51.377099Z","iopub.status.idle":"2022-05-15T14:30:51.791667Z","shell.execute_reply.started":"2022-05-15T14:30:51.377029Z","shell.execute_reply":"2022-05-15T14:30:51.790253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"KNN\",Prediction_KNN.value_counts())\nprint(\"DT\",Prediction_DT.value_counts())\nprint(\"EN\",Prediction_EN.value_counts())\nprint(\"LR\",Prediction_ST.value_counts())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df[\"anomaly\"] = Prediction_KNN['anomaly']\nprint(submission_df['anomaly'].value_counts())\nsubmission_df.to_csv('my_submission_final_1.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df[\"anomaly\"] = Prediction_DT['anomaly']\nprint(submission_df['anomaly'].value_counts())\nsubmission_df.to_csv('my_submission_final_2.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df[\"anomaly\"] = Prediction_EN['anomaly']\nprint(submission_df['anomaly'].value_counts())\nsubmission_df.to_csv('my_submission_final_3.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df[\"anomaly\"] = Prediction_ST['anomaly']\nprint(submission_df['anomaly'].value_counts())\nsubmission_df.to_csv('my_submission_final_4.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T14:30:51.794566Z","iopub.execute_input":"2022-05-15T14:30:51.794947Z","iopub.status.idle":"2022-05-15T14:30:54.815274Z","shell.execute_reply.started":"2022-05-15T14:30:51.7949Z","shell.execute_reply":"2022-05-15T14:30:54.814253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## What is next?\nAfter determining the anomalies points what we should do about them? In the most common \napplication like Microsoft anomaly detection, they have some web application to send some \nemails to the concerned persons about the sudden and didn’t expected change, then they take \nthe right decision to solve this issue. \n\nBut in our scope here we concerned with if this point is outlier or point of interest. \n\n\n\nSo, if the point we detected as outlier is a data that we did not need we will do on it some \ncleaning data processing. But in the other hand some time did not mean if there is some \nunusual event that we didn’t need it. Some event most be important to us because might this \nevent will happen in the future so by studying it will make us have the ability to avoid this \nsudden change in the future by handle it by some control flow process. \nHere we after studying those points we will make all anomalies point as a nan value so will \nhandle it in the coming part, Missing value imputation. ","metadata":{"papermill":{"duration":0.428982,"end_time":"2022-02-23T02:51:41.32133","exception":false,"start_time":"2022-02-23T02:51:40.892348","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"###

    Thanks a lot for having a look at this notebook. If you found this notebook useful, Do Upvote.🎯

    ","metadata":{"execution":{"iopub.status.busy":"2022-05-14T00:05:47.726974Z","iopub.execute_input":"2022-05-14T00:05:47.727259Z","iopub.status.idle":"2022-05-14T00:05:47.732015Z","shell.execute_reply.started":"2022-05-14T00:05:47.727231Z","shell.execute_reply":"2022-05-14T00:05:47.730956Z"}}}]}