{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview","metadata":{}},{"cell_type":"markdown","source":"1448棟のビルの4種類のメーター（電力、冷水、スチーム、温水）の値を予測するコンペである。\n\n背景としては、エネルギー使用量（4種類のメーターの値）が正確に予測できるようになることで<br>\n省エネ投資を活発化させて、環境問題に貢献したいという狙いがある。\n\nビルの所有者は、ビルのエネルギー効率を改善するような投資を行うことで<br>\nエネルギー効率が改善された分、コストを抑えることができる。","metadata":{}},{"cell_type":"markdown","source":"# Module","metadata":{}},{"cell_type":"code","source":"import gc\nimport sys\nimport optuna\nimport datetime\nimport warnings\nimport matplotlib\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport lightgbm as lgb\nimport statsmodels.api as sm\nimport matplotlib.pyplot as plt\n\nfrom matplotlib.pylab import rcParams\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import mean_squared_log_error\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:05.088796Z","iopub.execute_input":"2021-06-15T01:58:05.089226Z","iopub.status.idle":"2021-06-15T01:58:08.827761Z","shell.execute_reply.started":"2021-06-15T01:58:05.089146Z","shell.execute_reply":"2021-06-15T01:58:08.826736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Datasets","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\ntrain, test = train_test_split(data, test_size=0.3, random_state=123)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:08.829572Z","iopub.execute_input":"2021-06-15T01:58:08.830031Z","iopub.status.idle":"2021-06-15T01:58:29.773347Z","shell.execute_reply.started":"2021-06-15T01:58:08.829983Z","shell.execute_reply":"2021-06-15T01:58:29.772240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del data\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:29.775573Z","iopub.execute_input":"2021-06-15T01:58:29.775883Z","iopub.status.idle":"2021-06-15T01:58:29.971089Z","shell.execute_reply.started":"2021-06-15T01:58:29.775852Z","shell.execute_reply":"2021-06-15T01:58:29.969708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.reset_index(drop=True)\n\nprint(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:29.973269Z","iopub.execute_input":"2021-06-15T01:58:29.973653Z","iopub.status.idle":"2021-06-15T01:58:30.342611Z","shell.execute_reply.started":"2021-06-15T01:58:29.973615Z","shell.execute_reply":"2021-06-15T01:58:30.341621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1時間ごとのメータ値を保持するテーブル（学習データ）<br>\n・building_id：ビルのID<br>\n・meter：0は電力、1は冷水、2はスチーム、3は温水<br>\n・timestamp：日付と時刻<br>\n・meter_reading：メーターの使用量（目的変数）<br>","metadata":{"execution":{"iopub.status.busy":"2021-06-04T05:10:36.573045Z","iopub.execute_input":"2021-06-04T05:10:36.573413Z","iopub.status.idle":"2021-06-04T05:10:36.579555Z","shell.execute_reply.started":"2021-06-04T05:10:36.573384Z","shell.execute_reply":"2021-06-04T05:10:36.578197Z"}}},{"cell_type":"code","source":"test = test.reset_index(drop=True)\n\nprint(test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.344322Z","iopub.execute_input":"2021-06-15T01:58:30.344731Z","iopub.status.idle":"2021-06-15T01:58:30.509271Z","shell.execute_reply.started":"2021-06-15T01:58:30.344688Z","shell.execute_reply":"2021-06-15T01:58:30.508424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1時間ごとのメータ値を保持するテーブル（テストデータ）<br>\n・building_id：ビルのID<br>\n・meter：0は電力、1は冷水、2はスチーム、3は温水<br>\n・timestamp：日付と時刻<br>\n・meter_reading：メーターの使用量（目的変数）<br>","metadata":{}},{"cell_type":"code","source":"usecols = {'site_id', 'timestamp', 'air_temperature', 'dew_temperature'}","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.510471Z","iopub.execute_input":"2021-06-15T01:58:30.510738Z","iopub.status.idle":"2021-06-15T01:58:30.514102Z","shell.execute_reply.started":"2021-06-15T01:58:30.510713Z","shell.execute_reply":"2021-06-15T01:58:30.513361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weather_data = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv', usecols=usecols)\n\nprint(weather_data.shape)\nweather_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.515217Z","iopub.execute_input":"2021-06-15T01:58:30.515662Z","iopub.status.idle":"2021-06-15T01:58:30.743885Z","shell.execute_reply.started":"2021-06-15T01:58:30.515634Z","shell.execute_reply":"2021-06-15T01:58:30.742907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"気象情報を保持するテーブル<br>\n・site_id：ビルの物理的な位置<br>\n・timestamp：日付と時刻<br>\n・air_temperature：気温<br>\n・dew_temperature：湿度<br>\n・precip_depth：降水量<br>\n・sea_level_pressure：海圧<br>\n・wind_direction：風向<br>\n・wind_speed：風速<br>","metadata":{}},{"cell_type":"code","source":"usecols = {'site_id', 'building_id', 'primary_use', 'square_feet', 'year_built'}","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.747140Z","iopub.execute_input":"2021-06-15T01:58:30.747761Z","iopub.status.idle":"2021-06-15T01:58:30.751777Z","shell.execute_reply.started":"2021-06-15T01:58:30.747730Z","shell.execute_reply":"2021-06-15T01:58:30.750963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"building = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv', usecols=usecols)\n\nprint(building.shape)\nbuilding.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.754015Z","iopub.execute_input":"2021-06-15T01:58:30.754334Z","iopub.status.idle":"2021-06-15T01:58:30.784464Z","shell.execute_reply.started":"2021-06-15T01:58:30.754285Z","shell.execute_reply":"2021-06-15T01:58:30.783513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ビルの情報を保持するテーブル<br>\n・site_id：ビルの物理的な位置<br>\n・building_id：ビルのID<br>\n・primary_use：使用目的<br>\n・square_feet：面積<br>\n・year_built：施工した年<br>\n・floor_count：階数<br>","metadata":{}},{"cell_type":"markdown","source":"# Lags","metadata":{}},{"cell_type":"code","source":"lag_cols = ['air_temperature', 'dew_temperature']\n\nfor i in lag_cols:\n    weather_data['DIFF_' + i] = weather_data.groupby(by = ['site_id'])[i].shift(-1)\n    weather_data['PREVIOUS_DIFF_' + i] = np.abs(weather_data[i] - weather_data['DIFF_' + i])\n    weather_data = weather_data.drop(['DIFF_' + i], axis=1)\n    \nweather_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.785973Z","iopub.execute_input":"2021-06-15T01:58:30.786426Z","iopub.status.idle":"2021-06-15T01:58:30.836576Z","shell.execute_reply.started":"2021-06-15T01:58:30.786382Z","shell.execute_reply":"2021-06-15T01:58:30.835478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Join","metadata":{}},{"cell_type":"code","source":"def reduce_mem_usage(df, use_float16=False):\n    \n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.838219Z","iopub.execute_input":"2021-06-15T01:58:30.838640Z","iopub.status.idle":"2021-06-15T01:58:30.853482Z","shell.execute_reply.started":"2021-06-15T01:58:30.838600Z","shell.execute_reply":"2021-06-15T01:58:30.852583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = reduce_mem_usage(train)\n# test = reduce_mem_usage(test)\n\nweather_data= reduce_mem_usage(weather_data)\nbuilding = reduce_mem_usage(building)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.854998Z","iopub.execute_input":"2021-06-15T01:58:30.855307Z","iopub.status.idle":"2021-06-15T01:58:30.917079Z","shell.execute_reply.started":"2021-06-15T01:58:30.855263Z","shell.execute_reply":"2021-06-15T01:58:30.916043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.merge(building, on='building_id', how='left')\ntest = test.merge(building, on='building_id', how='left')\n\nprint(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:30.918541Z","iopub.execute_input":"2021-06-15T01:58:30.918944Z","iopub.status.idle":"2021-06-15T01:58:34.623485Z","shell.execute_reply.started":"2021-06-15T01:58:30.918900Z","shell.execute_reply":"2021-06-15T01:58:34.622411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del building\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:34.624632Z","iopub.execute_input":"2021-06-15T01:58:34.624957Z","iopub.status.idle":"2021-06-15T01:58:34.761645Z","shell.execute_reply.started":"2021-06-15T01:58:34.624931Z","shell.execute_reply":"2021-06-15T01:58:34.760853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.merge(weather_data, on=['site_id', 'timestamp'], how='left')\ntest = test.merge(weather_data, on=['site_id', 'timestamp'], how='left')\n\nprint(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:34.762587Z","iopub.execute_input":"2021-06-15T01:58:34.762862Z","iopub.status.idle":"2021-06-15T01:58:43.034469Z","shell.execute_reply.started":"2021-06-15T01:58:34.762835Z","shell.execute_reply":"2021-06-15T01:58:43.033380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del weather_data\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:43.036066Z","iopub.execute_input":"2021-06-15T01:58:43.036512Z","iopub.status.idle":"2021-06-15T01:58:43.177686Z","shell.execute_reply.started":"2021-06-15T01:58:43.036467Z","shell.execute_reply":"2021-06-15T01:58:43.176573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = reduce_mem_usage(train)\n# test = reduce_mem_usage(test)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:43.179038Z","iopub.execute_input":"2021-06-15T01:58:43.179545Z","iopub.status.idle":"2021-06-15T01:58:43.186960Z","shell.execute_reply.started":"2021-06-15T01:58:43.179501Z","shell.execute_reply":"2021-06-15T01:58:43.185988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.reset_index(drop=True)\ntest = test.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:43.188031Z","iopub.execute_input":"2021-06-15T01:58:43.188405Z","iopub.status.idle":"2021-06-15T01:58:43.874507Z","shell.execute_reply.started":"2021-06-15T01:58:43.188332Z","shell.execute_reply":"2021-06-15T01:58:43.873622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sys.getsizeof(train))\nprint(sys.getsizeof(test))","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:43.875775Z","iopub.execute_input":"2021-06-15T01:58:43.876059Z","iopub.status.idle":"2021-06-15T01:58:47.514235Z","shell.execute_reply.started":"2021-06-15T01:58:43.876033Z","shell.execute_reply":"2021-06-15T01:58:47.513128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.memory_usage()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:47.515952Z","iopub.execute_input":"2021-06-15T01:58:47.516547Z","iopub.status.idle":"2021-06-15T01:58:47.529205Z","shell.execute_reply.started":"2021-06-15T01:58:47.516500Z","shell.execute_reply":"2021-06-15T01:58:47.528238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.memory_usage()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:47.530847Z","iopub.execute_input":"2021-06-15T01:58:47.531358Z","iopub.status.idle":"2021-06-15T01:58:47.541606Z","shell.execute_reply.started":"2021-06-15T01:58:47.531279Z","shell.execute_reply":"2021-06-15T01:58:47.540481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:47.542769Z","iopub.execute_input":"2021-06-15T01:58:47.543153Z","iopub.status.idle":"2021-06-15T01:58:47.568426Z","shell.execute_reply.started":"2021-06-15T01:58:47.543120Z","shell.execute_reply":"2021-06-15T01:58:47.567385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analytics","metadata":{}},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:47.572841Z","iopub.execute_input":"2021-06-15T01:58:47.573129Z","iopub.status.idle":"2021-06-15T01:58:47.590827Z","shell.execute_reply.started":"2021-06-15T01:58:47.573100Z","shell.execute_reply":"2021-06-15T01:58:47.590026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.nunique()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:47.593914Z","iopub.execute_input":"2021-06-15T01:58:47.594203Z","iopub.status.idle":"2021-06-15T01:58:53.575329Z","shell.execute_reply.started":"2021-06-15T01:58:47.594177Z","shell.execute_reply":"2021-06-15T01:58:53.574424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:53.576485Z","iopub.execute_input":"2021-06-15T01:58:53.576764Z","iopub.status.idle":"2021-06-15T01:58:55.064341Z","shell.execute_reply.started":"2021-06-15T01:58:53.576738Z","shell.execute_reply":"2021-06-15T01:58:55.063412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.corr()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:58:55.065732Z","iopub.execute_input":"2021-06-15T01:58:55.066102Z","iopub.status.idle":"2021-06-15T01:59:00.435210Z","shell.execute_reply.started":"2021-06-15T01:58:55.066071Z","shell.execute_reply":"2021-06-15T01:59:00.434320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualization","metadata":{"execution":{"iopub.status.busy":"2021-06-09T04:35:08.004275Z","iopub.execute_input":"2021-06-09T04:35:08.004653Z","iopub.status.idle":"2021-06-09T04:35:08.008716Z","shell.execute_reply.started":"2021-06-09T04:35:08.004621Z","shell.execute_reply":"2021-06-09T04:35:08.007602Z"}}},{"cell_type":"code","source":"sns.set()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:00.436577Z","iopub.execute_input":"2021-06-15T01:59:00.436879Z","iopub.status.idle":"2021-06-15T01:59:00.442337Z","shell.execute_reply.started":"2021-06-15T01:59:00.436850Z","shell.execute_reply":"2021-06-15T01:59:00.441395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['timestamp'] = pd.to_datetime(train['timestamp'])\ntest['timestamp'] = pd.to_datetime(test['timestamp'])","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:00.443884Z","iopub.execute_input":"2021-06-15T01:59:00.444283Z","iopub.status.idle":"2021-06-15T01:59:06.586227Z","shell.execute_reply.started":"2021-06-15T01:59:00.444241Z","shell.execute_reply":"2021-06-15T01:59:06.585454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(8, 4))\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14)\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='day', alpha=0.8).set_ylabel('Meter reading', fontsize=14)\naxes.legend()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:06.587397Z","iopub.execute_input":"2021-06-15T01:59:06.587859Z","iopub.status.idle":"2021-06-15T01:59:22.646624Z","shell.execute_reply.started":"2021-06-15T01:59:06.587811Z","shell.execute_reply":"2021-06-15T01:59:22.645646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(8, 4))\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('W').mean()['meter_reading'].plot(ax=axes, label='week', alpha=1).set_ylabel('Meter reading', fontsize=14)\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='month', alpha=1).set_ylabel('Meter reading', fontsize=14)\naxes.legend()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:22.648125Z","iopub.execute_input":"2021-06-15T01:59:22.648744Z","iopub.status.idle":"2021-06-15T01:59:38.516943Z","shell.execute_reply.started":"2021-06-15T01:59:22.648688Z","shell.execute_reply":"2021-06-15T01:59:38.516269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Outliers","metadata":{}},{"cell_type":"code","source":"def to_season(month_num):\n    season = 'winter'\n    if 3 <= month_num <= 5:\n        season = 'spring'\n    elif 6 <= month_num <= 8:\n        season = 'summer'\n    elif 9 <= month_num <= 11:\n        season = 'autumn'\n    return season","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:38.518019Z","iopub.execute_input":"2021-06-15T01:59:38.518475Z","iopub.status.idle":"2021-06-15T01:59:38.522862Z","shell.execute_reply.started":"2021-06-15T01:59:38.518432Z","shell.execute_reply":"2021-06-15T01:59:38.522182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['month'] = train['timestamp'].dt.month\ntrain['season'] = train['month'].apply(lambda x: to_season(x))","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:38.523962Z","iopub.execute_input":"2021-06-15T01:59:38.524389Z","iopub.status.idle":"2021-06-15T01:59:48.138774Z","shell.execute_reply.started":"2021-06-15T01:59:38.524349Z","shell.execute_reply":"2021-06-15T01:59:48.137987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.drop(train[(train['meter'] == 0) & (train['meter_reading'] == 0)].index, axis=0)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:48.140104Z","iopub.execute_input":"2021-06-15T01:59:48.140927Z","iopub.status.idle":"2021-06-15T01:59:49.914510Z","shell.execute_reply.started":"2021-06-15T01:59:48.140875Z","shell.execute_reply":"2021-06-15T01:59:49.913675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.drop(train[(train['meter'] == 1) & (train['meter_reading'] == 0) & (train['season'] == 'spring')].index, axis=0)\ntrain = train.drop(train[(train['meter'] == 1) & (train['meter_reading'] == 0) & (train['season'] == 'summer')].index, axis=0)\ntrain = train.drop(train[(train['meter'] == 1) & (train['meter_reading'] == 0) & (train['season'] == 'autumn')].index, axis=0)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T01:59:49.915859Z","iopub.execute_input":"2021-06-15T01:59:49.916438Z","iopub.status.idle":"2021-06-15T02:00:07.972057Z","shell.execute_reply.started":"2021-06-15T01:59:49.916395Z","shell.execute_reply":"2021-06-15T02:00:07.970528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.drop(train[(train['meter'] == 2) & (train['meter_reading'] == 0) & (train['season'] == 'spring')].index, axis=0)\ntrain = train.drop(train[(train['meter'] == 2) & (train['meter_reading'] == 0) & (train['season'] == 'autumn')].index, axis=0)\ntrain = train.drop(train[(train['meter'] == 2) & (train['meter_reading'] == 0) & (train['season'] == 'winter')].index, axis=0)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:07.974255Z","iopub.execute_input":"2021-06-15T02:00:07.974821Z","iopub.status.idle":"2021-06-15T02:00:23.573896Z","shell.execute_reply.started":"2021-06-15T02:00:07.974777Z","shell.execute_reply":"2021-06-15T02:00:23.572806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.drop(train[(train['meter'] == 3) & (train['meter_reading'] == 0) & (train['season'] == 'spring')].index, axis=0)\ntrain = train.drop(train[(train['meter'] == 3) & (train['meter_reading'] == 0) & (train['season'] == 'autumn')].index, axis=0)\ntrain = train.drop(train[(train['meter'] == 3) & (train['meter_reading'] == 0) & (train['season'] == 'winter')].index, axis=0)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:23.575145Z","iopub.execute_input":"2021-06-15T02:00:23.575488Z","iopub.status.idle":"2021-06-15T02:00:37.896499Z","shell.execute_reply.started":"2021-06-15T02:00:23.575457Z","shell.execute_reply":"2021-06-15T02:00:37.895562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.drop('season', axis=1).reset_index(drop=True)\ngc.collect()\n\ntrain = reduce_mem_usage(train)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:37.897707Z","iopub.execute_input":"2021-06-15T02:00:37.897985Z","iopub.status.idle":"2021-06-15T02:00:40.463411Z","shell.execute_reply.started":"2021-06-15T02:00:37.897959Z","shell.execute_reply":"2021-06-15T02:00:40.462486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:40.464667Z","iopub.execute_input":"2021-06-15T02:00:40.464957Z","iopub.status.idle":"2021-06-15T02:00:40.486970Z","shell.execute_reply.started":"2021-06-15T02:00:40.464930Z","shell.execute_reply":"2021-06-15T02:00:40.486319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"code","source":"train['hour'] = train['timestamp'].dt.hour\n#train['day'] = train['timestamp'].dt.day\n#train['weekend'] = train['timestamp'].dt.weekday","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:40.487967Z","iopub.execute_input":"2021-06-15T02:00:40.488403Z","iopub.status.idle":"2021-06-15T02:00:41.967982Z","shell.execute_reply.started":"2021-06-15T02:00:40.488373Z","shell.execute_reply":"2021-06-15T02:00:41.967127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.drop(['timestamp', 'site_id'], axis=1)\ngc.collect()\n\ntrain = reduce_mem_usage(train)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:41.969028Z","iopub.execute_input":"2021-06-15T02:00:41.969492Z","iopub.status.idle":"2021-06-15T02:00:43.990274Z","shell.execute_reply.started":"2021-06-15T02:00:41.969458Z","shell.execute_reply":"2021-06-15T02:00:43.989340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_group = train.groupby('building_id')['meter_reading']\n\nbuilding_mean = df_group.mean()\nbuilding_median = df_group.median()\nbuilding_std = df_group.std()\n\n# building_min = df_group.min()\n# building_max = df_group.max()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:43.991614Z","iopub.execute_input":"2021-06-15T02:00:43.991901Z","iopub.status.idle":"2021-06-15T02:00:45.122080Z","shell.execute_reply.started":"2021-06-15T02:00:43.991873Z","shell.execute_reply":"2021-06-15T02:00:45.121038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['building_mean'] = train['building_id'].map(building_mean)\ntrain['building_median'] = train['building_id'].map(building_median)\ntrain['building_std'] = train['building_id'].map(building_std)\n\n# train['building_min'] = train['building_id'].map(building_min)\n# train['building_max'] = train['building_id'].map(building_max)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:45.123929Z","iopub.execute_input":"2021-06-15T02:00:45.124438Z","iopub.status.idle":"2021-06-15T02:00:45.656749Z","shell.execute_reply.started":"2021-06-15T02:00:45.124390Z","shell.execute_reply":"2021-06-15T02:00:45.655630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_group\ngc.collect()\n\ntrain = reduce_mem_usage(train)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:45.658021Z","iopub.execute_input":"2021-06-15T02:00:45.658354Z","iopub.status.idle":"2021-06-15T02:00:47.445284Z","shell.execute_reply.started":"2021-06-15T02:00:45.658321Z","shell.execute_reply":"2021-06-15T02:00:47.444297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:47.446405Z","iopub.execute_input":"2021-06-15T02:00:47.446674Z","iopub.status.idle":"2021-06-15T02:00:47.468898Z","shell.execute_reply.started":"2021-06-15T02:00:47.446650Z","shell.execute_reply":"2021-06-15T02:00:47.467790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocess","metadata":{}},{"cell_type":"code","source":"# drop_col = ['precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'wind_speed', 'floor_count']      ","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:47.470147Z","iopub.execute_input":"2021-06-15T02:00:47.470499Z","iopub.status.idle":"2021-06-15T02:00:47.477758Z","shell.execute_reply.started":"2021-06-15T02:00:47.470469Z","shell.execute_reply":"2021-06-15T02:00:47.476821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = train.drop(drop_col, axis = 1)\n# gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:47.478962Z","iopub.execute_input":"2021-06-15T02:00:47.479270Z","iopub.status.idle":"2021-06-15T02:00:47.488146Z","shell.execute_reply.started":"2021-06-15T02:00:47.479244Z","shell.execute_reply":"2021-06-15T02:00:47.487128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['square_feet'] = np.log(train['square_feet'])\ntest['square_feet'] = np.log(test['square_feet'])","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:47.489662Z","iopub.execute_input":"2021-06-15T02:00:47.490008Z","iopub.status.idle":"2021-06-15T02:00:48.103683Z","shell.execute_reply.started":"2021-06-15T02:00:47.489941Z","shell.execute_reply":"2021-06-15T02:00:48.101387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"le = LabelEncoder()\n\ntrain['primary_use'] = le.fit_transform(train['primary_use'])\ntest['primary_use'] = le.fit_transform(test['primary_use'])","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:48.104957Z","iopub.execute_input":"2021-06-15T02:00:48.105386Z","iopub.status.idle":"2021-06-15T02:00:54.612230Z","shell.execute_reply.started":"2021-06-15T02:00:48.105339Z","shell.execute_reply":"2021-06-15T02:00:54.611375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = np.log1p(train['meter_reading'])\ntrain = train.drop(['meter_reading'], axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:54.613736Z","iopub.execute_input":"2021-06-15T02:00:54.614152Z","iopub.status.idle":"2021-06-15T02:00:55.605923Z","shell.execute_reply.started":"2021-06-15T02:00:54.614099Z","shell.execute_reply":"2021-06-15T02:00:55.604935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del le\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:55.607608Z","iopub.execute_input":"2021-06-15T02:00:55.608047Z","iopub.status.idle":"2021-06-15T02:00:55.758843Z","shell.execute_reply.started":"2021-06-15T02:00:55.608000Z","shell.execute_reply":"2021-06-15T02:00:55.758046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = reduce_mem_usage(train)\ntest = reduce_mem_usage(test)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:55.760132Z","iopub.execute_input":"2021-06-15T02:00:55.760447Z","iopub.status.idle":"2021-06-15T02:00:57.903359Z","shell.execute_reply.started":"2021-06-15T02:00:55.760420Z","shell.execute_reply":"2021-06-15T02:00:57.902233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.reset_index(drop=True)\ntest = test.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:57.904761Z","iopub.execute_input":"2021-06-15T02:00:57.905061Z","iopub.status.idle":"2021-06-15T02:00:58.211799Z","shell.execute_reply.started":"2021-06-15T02:00:57.905031Z","shell.execute_reply":"2021-06-15T02:00:58.210668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sys.getsizeof(train))\nprint(sys.getsizeof(test))","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:58.213108Z","iopub.execute_input":"2021-06-15T02:00:58.213439Z","iopub.status.idle":"2021-06-15T02:00:58.223203Z","shell.execute_reply.started":"2021-06-15T02:00:58.213407Z","shell.execute_reply":"2021-06-15T02:00:58.222169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.to_pickle('train.pickle')\n\nprint(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:58.224803Z","iopub.execute_input":"2021-06-15T02:00:58.225212Z","iopub.status.idle":"2021-06-15T02:00:58.250674Z","shell.execute_reply.started":"2021-06-15T02:00:58.225170Z","shell.execute_reply":"2021-06-15T02:00:58.249491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling","metadata":{}},{"cell_type":"code","source":"'''\nX_tr, X_val, y_tr, y_val = train_test_split(train, target, test_size=0.2, random_state=666)\n\ndef create_model(trial):\n    num_leaves = trial.suggest_int('num_leaves', 2, 30)\n    n_estimators = trial.suggest_int('n_estimators', 50, 300)\n    learning_rate = trial.suggest_uniform('learning_rate', 0.0001, 0.99)\n    max_depth = trial.suggest_int('max_depth', 2, 10)\n    min_child_samples = trial.suggest_int('min_child_samples', 100, 1200)\n    min_data_in_leaf = trial.suggest_int('min_data_in_leaf', 5, 90)\n    bagging_freq = trial.suggest_int('bagging_freq', 1, 7)\n    bagging_fraction = trial.suggest_uniform('bagging_fraction', 0.0001, 1.0)\n    feature_fraction = trial.suggest_uniform('feature_fraction', 0.0001, 1.0)\n    subsample = trial.suggest_uniform('subsample', 0.1, 1.0)\n    colsample_bytree = trial.suggest_uniform('colsample_bytree', 0.1, 1.0)\n    \n    model = lgb.LGBMRegressor(\n        num_leaves=num_leaves,\n        n_estimators=n_estimators,\n        learning_rate=learning_rate,\n        max_depth=max_depth, \n        min_child_samples=min_child_samples, \n        min_data_in_leaf=min_data_in_leaf,\n        bagging_freq=bagging_freq,\n        bagging_fraction=bagging_fraction,\n        feature_fraction=feature_fraction,\n        subsample=subsample,\n        colsample_bytree=colsample_bytree,\n        metric='rsme',\n        random_state=666)\n    return model\n\ndef objective(trial):\n    model = create_model(trial)\n    model.fit(X_tr, y_tr)\n    y_pred = model.predict(X_val)\n    rsme = np.sqrt(mean_squared_log_error(y_val, y_pred))\n    return rsme\n\nstudy = optuna.create_study(direction='minimize')\nstudy.optimize(objective, n_trials=40)\nparams = study.best_params\nprint(params)\n'''","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:58.252097Z","iopub.execute_input":"2021-06-15T02:00:58.252415Z","iopub.status.idle":"2021-06-15T02:00:58.260540Z","shell.execute_reply.started":"2021-06-15T02:00:58.252385Z","shell.execute_reply":"2021-06-15T02:00:58.259585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params = {'num_leaves': 30,\n          'n_estimators': 270,\n          'learning_rate': 0.7879053219502106,\n          'max_depth': 10,\n          'min_child_samples': 229,\n          'min_data_in_leaf': 29,\n          'bagging_freq': 2,\n          'bagging_fraction': 0.5812963822819995,\n          'feature_fraction': 0.935225000564576,\n          'subsample': 0.5646113055393944,\n          'colsample_bytree': 0.9821303332132698,\n          'random_state': 666}","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:58.261740Z","iopub.execute_input":"2021-06-15T02:00:58.262093Z","iopub.status.idle":"2021-06-15T02:00:58.275652Z","shell.execute_reply.started":"2021-06-15T02:00:58.262067Z","shell.execute_reply":"2021-06-15T02:00:58.274682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cls = lgb.LGBMRegressor(**params)\ncls.fit(train, target)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:00:58.277028Z","iopub.execute_input":"2021-06-15T02:00:58.277437Z","iopub.status.idle":"2021-06-15T02:03:12.803589Z","shell.execute_reply.started":"2021-06-15T02:00:58.277400Z","shell.execute_reply":"2021-06-15T02:03:12.802515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del params\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:12.805007Z","iopub.execute_input":"2021-06-15T02:03:12.805571Z","iopub.status.idle":"2021-06-15T02:03:12.969736Z","shell.execute_reply.started":"2021-06-15T02:03:12.805534Z","shell.execute_reply":"2021-06-15T02:03:12.968688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb.plot_importance(cls)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:12.971401Z","iopub.execute_input":"2021-06-15T02:03:12.971834Z","iopub.status.idle":"2021-06-15T02:03:13.267088Z","shell.execute_reply.started":"2021-06-15T02:03:12.971782Z","shell.execute_reply":"2021-06-15T02:03:13.265813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction","metadata":{}},{"cell_type":"code","source":"y_true = test['meter_reading']\ntimestamp = test['timestamp']\nmeter = test['meter']\n\ntest = test[[col for col in test.columns if col != 'meter_reading']]","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:13.268701Z","iopub.execute_input":"2021-06-15T02:03:13.269037Z","iopub.status.idle":"2021-06-15T02:03:13.602498Z","shell.execute_reply.started":"2021-06-15T02:03:13.269002Z","shell.execute_reply":"2021-06-15T02:03:13.601380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['hour'] = test['timestamp'].dt.hour\n# test['day'] = test['timestamp'].dt.day\n# test['weekend'] = test['timestamp'].dt.weekday\ntest['month'] = test['timestamp'].dt.month","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:13.604473Z","iopub.execute_input":"2021-06-15T02:03:13.605004Z","iopub.status.idle":"2021-06-15T02:03:14.973765Z","shell.execute_reply.started":"2021-06-15T02:03:13.604958Z","shell.execute_reply":"2021-06-15T02:03:14.972457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = test.drop(['timestamp', 'site_id'], axis=1)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:14.975535Z","iopub.execute_input":"2021-06-15T02:03:14.976048Z","iopub.status.idle":"2021-06-15T02:03:15.469282Z","shell.execute_reply.started":"2021-06-15T02:03:14.975994Z","shell.execute_reply":"2021-06-15T02:03:15.468252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['building_mean'] = test['building_id'].map(building_mean)\ntest['building_median'] = test['building_id'].map(building_median)\ntest['building_std'] = test['building_id'].map(building_std)\n\n# test['building_min'] = test['building_id'].map(building_min)\n# test['building_max'] = test['building_id'].map(building_max)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:15.470845Z","iopub.execute_input":"2021-06-15T02:03:15.471152Z","iopub.status.idle":"2021-06-15T02:03:15.667503Z","shell.execute_reply.started":"2021-06-15T02:03:15.471122Z","shell.execute_reply":"2021-06-15T02:03:15.666324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del building_mean, building_median, building_std\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:15.669236Z","iopub.execute_input":"2021-06-15T02:03:15.669726Z","iopub.status.idle":"2021-06-15T02:03:15.892790Z","shell.execute_reply.started":"2021-06-15T02:03:15.669678Z","shell.execute_reply":"2021-06-15T02:03:15.891697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = test.reset_index(drop=True)\ntest = reduce_mem_usage(test)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:15.894193Z","iopub.execute_input":"2021-06-15T02:03:15.894580Z","iopub.status.idle":"2021-06-15T02:03:16.963484Z","shell.execute_reply.started":"2021-06-15T02:03:15.894545Z","shell.execute_reply":"2021-06-15T02:03:16.962211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(pd.DataFrame([[i for i in dir()], [sys.getsizeof(eval(i)) for i in dir()]],\n                   index=['name','size']).T.sort_values('size', ascending=False).reset_index(drop=True).head(10))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-15T02:03:16.965226Z","iopub.execute_input":"2021-06-15T02:03:16.965640Z","iopub.status.idle":"2021-06-15T02:03:17.017049Z","shell.execute_reply.started":"2021-06-15T02:03:16.965604Z","shell.execute_reply":"2021-06-15T02:03:17.015883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.to_pickle('test.pickle')\n\nprint(test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:17.018496Z","iopub.execute_input":"2021-06-15T02:03:17.019037Z","iopub.status.idle":"2021-06-15T02:03:17.049880Z","shell.execute_reply.started":"2021-06-15T02:03:17.018983Z","shell.execute_reply":"2021-06-15T02:03:17.048631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = np.expm1(cls.predict(test))","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:17.057329Z","iopub.execute_input":"2021-06-15T02:03:17.057695Z","iopub.status.idle":"2021-06-15T02:03:41.885330Z","shell.execute_reply.started":"2021-06-15T02:03:17.057663Z","shell.execute_reply":"2021-06-15T02:03:41.884156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del test, cls\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:41.887481Z","iopub.execute_input":"2021-06-15T02:03:41.888113Z","iopub.status.idle":"2021-06-15T02:03:42.119153Z","shell.execute_reply.started":"2021-06-15T02:03:41.888053Z","shell.execute_reply":"2021-06-15T02:03:42.118024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Alert","metadata":{}},{"cell_type":"code","source":"data = pd.DataFrame(timestamp, columns=['timestamp'])\n\ndata['meter'] = meter\ndata['prediction'] = y_pred\ndata['meter_reading'] = y_true\n\ndata = data.sort_values('timestamp')\ndata = data.reset_index(drop=True)\n\ndata.to_pickle('data.pickle')\ndata.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:42.120985Z","iopub.execute_input":"2021-06-15T02:03:42.121362Z","iopub.status.idle":"2021-06-15T02:03:44.668057Z","shell.execute_reply.started":"2021-06-15T02:03:42.121325Z","shell.execute_reply":"2021-06-15T02:03:44.667155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del y_pred, y_true, timestamp, meter\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:44.669214Z","iopub.execute_input":"2021-06-15T02:03:44.669508Z","iopub.status.idle":"2021-06-15T02:03:44.845499Z","shell.execute_reply.started":"2021-06-15T02:03:44.669481Z","shell.execute_reply":"2021-06-15T02:03:44.844503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data['timestamp'] = pd.to_datetime(data['timestamp'])","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:44.846769Z","iopub.execute_input":"2021-06-15T02:03:44.847087Z","iopub.status.idle":"2021-06-15T02:03:45.000724Z","shell.execute_reply.started":"2021-06-15T02:03:44.847045Z","shell.execute_reply":"2021-06-15T02:03:44.999969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(8, 4))\ndata[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='Meter_reading', alpha=0.8).set_ylabel('Meter_reading', fontsize=14)\ndata[['timestamp', 'prediction']].set_index('timestamp').resample('H').mean()['prediction'].plot(ax=axes, label='Prediction', alpha=0.8).set_ylabel('Prediction', fontsize=14)\naxes.legend()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:45.001901Z","iopub.execute_input":"2021-06-15T02:03:45.002481Z","iopub.status.idle":"2021-06-15T02:03:46.178607Z","shell.execute_reply.started":"2021-06-15T02:03:45.002441Z","shell.execute_reply":"2021-06-15T02:03:46.177382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(8, 4))\ndata[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='Meter_reading', alpha=0.8).set_ylabel('Meter_reading', fontsize=14)\ndata[['timestamp', 'prediction']].set_index('timestamp').resample('M').mean()['prediction'].plot(ax=axes, label='Prediction', alpha=0.8).set_ylabel('Prediction', fontsize=14)\naxes.legend()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"異常度 = 実測値 / 予測値 と設定する。<br>\n値が0.5以下であればsmall、2以上であればbigのアラートを出す。","metadata":{}},{"cell_type":"code","source":"data['abnormality'] = data['meter_reading'] / data['prediction']\ndata['alert'] = 'nomal'","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:46.181480Z","iopub.execute_input":"2021-06-15T02:03:46.181814Z","iopub.status.idle":"2021-06-15T02:03:46.239359Z","shell.execute_reply.started":"2021-06-15T02:03:46.181785Z","shell.execute_reply":"2021-06-15T02:03:46.238393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(data.shape[0]):\n    if i < 50:\n        if data.iloc[i, 4] <= 1/2:\n            data.iloc[i, 5] = 'small'\n            print(data.iloc[i, 0], '検出値が小さすぎます')\n            \n        elif data.iloc[i, 4] >= 2:\n            data.iloc[i, 5] = 'big'\n            print(data.iloc[i, 0], '検出値が大きすぎます')\n            \n        else:\n            pass\n    \n    else:\n        break","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:46.240890Z","iopub.execute_input":"2021-06-15T02:03:46.241318Z","iopub.status.idle":"2021-06-15T02:03:49.786525Z","shell.execute_reply.started":"2021-06-15T02:03:46.241261Z","shell.execute_reply":"2021-06-15T02:03:49.785404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:49.788009Z","iopub.execute_input":"2021-06-15T02:03:49.788576Z","iopub.status.idle":"2021-06-15T02:03:49.802877Z","shell.execute_reply.started":"2021-06-15T02:03:49.788529Z","shell.execute_reply":"2021-06-15T02:03:49.801804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ネットワークやサーバー機器のログであれば、smallやbigなどのエラーメッセージがあらかじめ表示されるので<br>\nそれをもとに、条件分岐で次のアクションを設定してあげるのが良いと考えられる。\n\nまた、以下2つを組み合わせることで、より精度の高い検知を行うことができそうである。\n\n①エラーメッセージによる検知<br>\n②AIの異常度（実測値/予測値）を活用した検知","metadata":{}},{"cell_type":"markdown","source":"# Stats","metadata":{}},{"cell_type":"code","source":"# data = data.set_index('timestamp')\n# data.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-15T02:03:49.804522Z","iopub.execute_input":"2021-06-15T02:03:49.804951Z","iopub.status.idle":"2021-06-15T02:03:49.949842Z","shell.execute_reply.started":"2021-06-15T02:03:49.804897Z","shell.execute_reply":"2021-06-15T02:03:49.948767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References","metadata":{"execution":{"iopub.status.busy":"2021-06-14T08:31:26.276232Z","iopub.execute_input":"2021-06-14T08:31:26.276717Z","iopub.status.idle":"2021-06-14T08:31:26.281041Z","shell.execute_reply.started":"2021-06-14T08:31:26.276676Z","shell.execute_reply":"2021-06-14T08:31:26.280145Z"}}},{"cell_type":"markdown","source":"・[異常検知の手法（外れ値検出・異常部位検出・変化点検出）](https://www.albert2005.co.jp/knowledge/machine_learning/anomaly_detection_basics/anomaly_detection_time#hazure)<br>\n・[Pythonのstatsmodelsを用いた時系列分析](https://qiita.com/hcpmiyuki/items/b1783956dee20c6d4700)","metadata":{"execution":{"iopub.status.busy":"2021-06-14T08:32:22.659585Z","iopub.execute_input":"2021-06-14T08:32:22.659978Z","iopub.status.idle":"2021-06-14T08:32:22.666411Z","shell.execute_reply.started":"2021-06-14T08:32:22.659947Z","shell.execute_reply":"2021-06-14T08:32:22.664924Z"}}}]}