{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from matplotlib import pyplot as plt\nimport matplotlib.dates as mdates\n\nimport pandas as pd\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:22.771491Z","iopub.execute_input":"2022-05-15T20:26:22.772110Z","iopub.status.idle":"2022-05-15T20:26:22.777190Z","shell.execute_reply.started":"2022-05-15T20:26:22.772073Z","shell.execute_reply":"2022-05-15T20:26:22.776545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_data = pd.read_csv('../input/energy-anomaly-detection/train.csv')\ntraining_data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:22.823614Z","iopub.execute_input":"2022-05-15T20:26:22.824460Z","iopub.status.idle":"2022-05-15T20:26:23.907749Z","shell.execute_reply.started":"2022-05-15T20:26:22.824404Z","shell.execute_reply":"2022-05-15T20:26:23.906806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_ids = training_data.groupby('building_id')\ntraining_data['building_id'].unique().size","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-05-15T20:26:23.909242Z","iopub.execute_input":"2022-05-15T20:26:23.909477Z","iopub.status.idle":"2022-05-15T20:26:23.933181Z","shell.execute_reply.started":"2022-05-15T20:26:23.909449Z","shell.execute_reply":"2022-05-15T20:26:23.932171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Missing values\n\n### Number of observations\n\nAmong many aspects of data that can affect our work, missing values hold a special place.\nDepending on the situation they can mean various things, from broken meters, to blackouts. At any rate they can be considered anomalous in their own right, even if they are not denominated as such in the dataset.\n\nOne way or another we have to account for them. But first of all we must know where they occur and how frequently.","metadata":{}},{"cell_type":"code","source":"grouped_ids['timestamp'].count().max()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:23.934378Z","iopub.execute_input":"2022-05-15T20:26:23.934596Z","iopub.status.idle":"2022-05-15T20:26:24.153175Z","shell.execute_reply.started":"2022-05-15T20:26:23.934570Z","shell.execute_reply":"2022-05-15T20:26:24.152368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_ids['timestamp'].count().hist()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:24.155014Z","iopub.execute_input":"2022-05-15T20:26:24.155564Z","iopub.status.idle":"2022-05-15T20:26:24.610826Z","shell.execute_reply.started":"2022-05-15T20:26:24.155518Z","shell.execute_reply":"2022-05-15T20:26:24.609938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the buildings monitored have a full set of 8784 readings. Only a handful have slightly less and even those have at least over 7k observations.","metadata":{}},{"cell_type":"markdown","source":"### Total number of anomalies","metadata":{}},{"cell_type":"code","source":"grouped_ids.sum('anomaly').hist('anomaly', bins=10)\nplt.show()","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-05-15T20:26:24.612245Z","iopub.execute_input":"2022-05-15T20:26:24.612831Z","iopub.status.idle":"2022-05-15T20:26:24.875671Z","shell.execute_reply.started":"2022-05-15T20:26:24.612787Z","shell.execute_reply":"2022-05-15T20:26:24.874658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observed anomalies number in low hundreds for most buildings monitored. However, there are few exceptions with almost a thousand recorded anomalies. These should be given more attention and their characteristics will be highlighted further.","metadata":{}},{"cell_type":"markdown","source":"### Relative prevalence of anomalous readings","metadata":{}},{"cell_type":"code","source":"grouped_ids.mean('anomaly').hist('anomaly', bins=10)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:24.876986Z","iopub.execute_input":"2022-05-15T20:26:24.878150Z","iopub.status.idle":"2022-05-15T20:26:25.104133Z","shell.execute_reply.started":"2022-05-15T20:26:24.878108Z","shell.execute_reply":"2022-05-15T20:26:25.103379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As with absolute number of observed anomalies, the relative prevalance of anomalous readins is rather low, and most buildings in the data set have less than 3% of their readings classified as anomalous.\n\nNext we compare the two measures and extract building ids with most anomalouse readings by both. As there is only a minute difference between number of observations for individual buildings we should have the same ids at the top of both lists.","metadata":{}},{"cell_type":"code","source":"grouped_ids.sum('anomaly').sort_values('anomaly', ascending=False)[:10]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:25.105779Z","iopub.execute_input":"2022-05-15T20:26:25.106101Z","iopub.status.idle":"2022-05-15T20:26:25.148242Z","shell.execute_reply.started":"2022-05-15T20:26:25.106066Z","shell.execute_reply":"2022-05-15T20:26:25.147471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_ids.mean('anomaly').sort_values('anomaly', ascending=False)[:10]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:25.149450Z","iopub.execute_input":"2022-05-15T20:26:25.149677Z","iopub.status.idle":"2022-05-15T20:26:25.196679Z","shell.execute_reply.started":"2022-05-15T20:26:25.149650Z","shell.execute_reply":"2022-05-15T20:26:25.192961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, top positions are dominated by the same ids. We've expected no surprise and got none. Now we will plot the values recorded for top three entries to see if we can find any interesting characteristics. ","metadata":{}},{"cell_type":"markdown","source":"## Plotting the most anomalous buildings","metadata":{}},{"cell_type":"code","source":"top_ids = grouped_ids.mean('anomaly').sort_values('anomaly', ascending=False)[:3].index.values\nfig, axes = plt.subplots(top_ids.size)\nfig.set_size_inches((40,20))\nfor top_id, ax in zip(top_ids, axes):\n    ax.set_title(\"Power consumption - \" + str(top_id))\n    anomaly_ax = ax.twinx()\n    selected_data = training_data[training_data['building_id'] == top_id]\n    ax.plot(selected_data['timestamp'].values, selected_data['meter_reading'])\n    anomaly_ax.plot(\n        selected_data['timestamp'].values, selected_data['anomaly'],\n        color='red',\n        linestyle='dashed')\n    ax.xaxis.set_major_locator(mdates.YearLocator())\n    ax.xaxis.set_minor_locator(mdates.MonthLocator())\n    ax.grid(True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:25.198447Z","iopub.execute_input":"2022-05-15T20:26:25.198986Z","iopub.status.idle":"2022-05-15T20:26:39.659547Z","shell.execute_reply.started":"2022-05-15T20:26:25.198942Z","shell.execute_reply":"2022-05-15T20:26:39.658860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Couple of interesting things pop up from these charts. First of all that these three, most anomalous, buildings all behave in quite different fashion from one another.\n\nBuilding 1319 has couple of large gaps in observations, unlike 439 and 1258.\nAt the same time power consumption of 1258 appears to drop for several months, to rapidly recover afterwards.\nFinally the power consumption of building 439 abruptly increases only to slowly return to previous levels with somewhat decreased variance.\n\nFurthermore, average power consumption of building 1258 is almost order of magnitude higher than that of other two.\n","metadata":{}},{"cell_type":"markdown","source":"### Rolling average and standard deviation\n\nTo make large scale behavior a bit more obvious we can calculate rolling average and standard deviation with a window of 20 observations.\n\nIn this case we'll keep the anomaly out of consideration to keep the charts tidy. ","metadata":{}},{"cell_type":"code","source":"window = 50\n\nfig, axes = plt.subplots(top_ids.size)\nfig.set_size_inches((40,20))\nfor top_id, ax in zip(top_ids, axes):\n    ax.set_title(\"Power consumption - \" + str(top_id))\n    avg_ax = ax.twinx()\n    selected_data = training_data[training_data['building_id'] == top_id]\n    \n    selected_data.loc[\n        pd.isna(selected_data['meter_reading']),'meter_reading'] = selected_data['meter_reading'].fillna(0)\n    ax.plot(\n        selected_data['timestamp'].values, selected_data['meter_reading'].rolling(window).std(),\n        linestyle='dashed',\n        label='Std',\n        color='blue')\n    avg_ax.plot(\n       selected_data['timestamp'].values, selected_data['meter_reading'].rolling(window).mean(),\n        color='red',\n        linestyle='dotted')\n    ax.set_ylabel('Std', color='blue')\n    avg_ax.set_ylabel('Avg', color='red')\n    ax.xaxis.set_major_locator(mdates.YearLocator())\n    ax.xaxis.set_minor_locator(mdates.MonthLocator())\n    ax.grid(True)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:26:39.661630Z","iopub.execute_input":"2022-05-15T20:26:39.662075Z","iopub.status.idle":"2022-05-15T20:26:54.182688Z","shell.execute_reply.started":"2022-05-15T20:26:39.662039Z","shell.execute_reply":"2022-05-15T20:26:54.181816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plots of rollling average and standard deviation values reveal further differences between the selected buildings.\n\nThe 439 displays obvious periodicity in both averge power consumption and std, this periodicity is especially apparent during the first third of the time period. Afterwards we see several dips in std followed by raising peak average power consumption with longer periods.\n\nPeriodic behavior, although not as pronounced, can be observed in the average values of power consumption for building 1258 recover previous pattern of behavior after experiencing dips.\n\nPerhaps least regular behavior belongs to building 1319. At least with the selected window size, not periodicity in std or average values appears present.","metadata":{}}]}