{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# KELOMPOK E\nAnggota:\n1. Nurkhalisha Humaira (2106703802)\n2. Rima Fitrianti Azahra (2106701974)\n3. Farah Ramadhani Putri (2106654832)\n4. Daniella Putri Shalomita (2106631072)\n5. Musarofah Kurnia (2106652543)","metadata":{}},{"cell_type":"markdown","source":"# PENDAHULUAN\n## 1. Deskripsi Data\nData ini diambil dari site ASHRAE (American Society of Heating and Air-Conditioning Engineers). Data ini berisi tentang penggunaan 4 energi yang berbeda pada gedung-gedung di beberapa wilayah dalam satu tahun. Variabel yang ada dari data ini adalah: <br>\n\n**train.csv:** \n  1. building_id - ID Gedung\n  2. meter - Meter id code. Dengan Keterangan {0: electricity, 1: chilledwater, 2: steam, 3: hotwater}.\n  3. timestamp - Waktu Pengukuran\n  4. meter_reading - Variabel target yaitu energi yang digunakan dalam kWh. \n  5. site_id - Foreign key untuk data cuaca.\n  6. building_id - Foreign key untuk data train\n  7. primary_use - Penggunaan gedung sehari-hari\n  8. square_feet - luas lantai gedung\n  9. year_built - Tahun gedung dibuka\n  10. floor_count - jumlah lantai\n\n\n**weather_[train/test].csv**: Data Cuaca dari stasiun metereologi terdekat site gedung\n\n  1. site_id - ID site gedung\n  2. air_temperature - Suhu Udara (Derajat Celsius)\n  3. cloud_coverage - Jumlah awan yang menutupi langit (oktas)\n  4. dew_temperature - Suhu Embun (Derajat Celsius)\n  5. precip_depth_1_hr - Presipitasi Air Hujan (Millimeter)\n  6. sea_level_pressure - Millibar/hectopascals\n  7. wind_direction - Arah Angin berdasarkan Compass direction (0-360)\n  8. wind_speed - Kecepatan angin (m/s)\n\n**test.csv**: \n\n  1. row_id - Row id file submission\n  2. building_id - Building id code\n  3. meter - meter id code\n  4. timestamp - Jangka waktu data test\n  \n## 2. Permasalahan\nPermasalahan yang akan dianalisa adalah bagaimana melihat penggunaan 4 energi tersebut, yaitu:<br>\n- Energi Listrik/Electricity (0)\n- Air Dingin/Chilled Water (1)\n- Uap/Steam (2)\n- Air Panas/Hot Water (3)\n\ndengan informasi gedung, cuaca, dan meter reading. Nantinya informasi ini akan digunakan untuk menentukan efisiensi energi.\n","metadata":{}},{"cell_type":"markdown","source":"# LOADING DATA\n## Inisasi Data","metadata":{}},{"cell_type":"code","source":"#importing packages\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-04-05T06:55:26.902453Z","iopub.execute_input":"2022-04-05T06:55:26.902994Z","iopub.status.idle":"2022-04-05T06:55:27.949660Z","shell.execute_reply.started":"2022-04-05T06:55:26.902865Z","shell.execute_reply":"2022-04-05T06:55:27.948572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Data Gedung\ngd = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\n#Data Meter Reading\nen_test = pd.read_csv('../input/ashrae-energy-prediction/test.csv')\nen_train = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\n#Data Cuaca\ncu_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv')\ncu_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:55:27.951669Z","iopub.execute_input":"2022-04-05T06:55:27.952025Z","iopub.status.idle":"2022-04-05T06:56:14.445179Z","shell.execute_reply.started":"2022-04-05T06:55:27.951981Z","shell.execute_reply":"2022-04-05T06:56:14.444306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Menampilkan Data","metadata":{}},{"cell_type":"code","source":"gd","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.450681Z","iopub.execute_input":"2022-04-05T06:56:14.450925Z","iopub.status.idle":"2022-04-05T06:56:14.486699Z","shell.execute_reply.started":"2022-04-05T06:56:14.450893Z","shell.execute_reply":"2022-04-05T06:56:14.485581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en_test","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.506931Z","iopub.execute_input":"2022-04-05T06:56:14.507262Z","iopub.status.idle":"2022-04-05T06:56:14.523732Z","shell.execute_reply.started":"2022-04-05T06:56:14.507217Z","shell.execute_reply":"2022-04-05T06:56:14.522687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en_train","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.525450Z","iopub.execute_input":"2022-04-05T06:56:14.525784Z","iopub.status.idle":"2022-04-05T06:56:14.544395Z","shell.execute_reply.started":"2022-04-05T06:56:14.525742Z","shell.execute_reply":"2022-04-05T06:56:14.543518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cu_test","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.545789Z","iopub.execute_input":"2022-04-05T06:56:14.546092Z","iopub.status.idle":"2022-04-05T06:56:14.576574Z","shell.execute_reply.started":"2022-04-05T06:56:14.546018Z","shell.execute_reply":"2022-04-05T06:56:14.575585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cu_train","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.578163Z","iopub.execute_input":"2022-04-05T06:56:14.578531Z","iopub.status.idle":"2022-04-05T06:56:14.618207Z","shell.execute_reply.started":"2022-04-05T06:56:14.578468Z","shell.execute_reply":"2022-04-05T06:56:14.617156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#melihat info masing-masing data untuk data understanding\ngd.info()\n\nen_test.info()\nen_train.info()\n\ncu_test.info()\ncu_train.info()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.619775Z","iopub.execute_input":"2022-04-05T06:56:14.620883Z","iopub.status.idle":"2022-04-05T06:56:14.744074Z","shell.execute_reply.started":"2022-04-05T06:56:14.620834Z","shell.execute_reply":"2022-04-05T06:56:14.743192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reducing Memory\nKarena data yang digunakan cukup besar, sering terjadi *crash* saat run program, maka akan direduce data untuk mengurangi besar memori data","metadata":{}},{"cell_type":"code","source":"#fungsi rekursif untuk reducing memory\ndef reduce_mem_usage(df, df_name):\n\n    start_mem = df.memory_usage().sum() / 1024**2\n    \n    for col in df.columns:\n\n        col_type = df[col].dtype\n\n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n\n    end_mem = df.memory_usage().sum() / 1024**2\n\n    print('Memory usage of {} is reduced by {:.2f} %. Usage dropped from {:.2f} MB to {:.2f} MB.'.format(df_name, (100 * (start_mem - end_mem) / start_mem), start_mem, end_mem))\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.745505Z","iopub.execute_input":"2022-04-05T06:56:14.746317Z","iopub.status.idle":"2022-04-05T06:56:14.762867Z","shell.execute_reply.started":"2022-04-05T06:56:14.746265Z","shell.execute_reply":"2022-04-05T06:56:14.761416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#reducing memory dataframe yang telah diinisiasi\ngd = reduce_mem_usage(gd, 'Data Gedung')\n\nen_train = reduce_mem_usage(en_train, 'Data Train')\nen_test = reduce_mem_usage(en_test, 'Data Test')\n\ncu_test = reduce_mem_usage(cu_test, \"Data Cuaca Test\")\ncu_train = reduce_mem_usage(cu_train, 'Data Cuaca Train')","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:14.764287Z","iopub.execute_input":"2022-04-05T06:56:14.764699Z","iopub.status.idle":"2022-04-05T06:56:23.787716Z","shell.execute_reply.started":"2022-04-05T06:56:14.764653Z","shell.execute_reply":"2022-04-05T06:56:23.786638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MERGING DATA\nData akan dikelompokkan menjadi 2 Dataframe, namun EDA akan difokuskan terhadap dataframe train.","metadata":{}},{"cell_type":"code","source":"train = en_train.merge(gd, on='building_id', how='left')\ntrain = train.merge(cu_train, on=['site_id', 'timestamp'], how='left')\n\ntest = en_test.merge(gd, on='building_id', how='left')\ntest = test.merge(cu_test, on=['site_id', 'timestamp'], how='left')\n\ndisplay(train.head(), test.head())","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:23.789053Z","iopub.execute_input":"2022-04-05T06:56:23.789784Z","iopub.status.idle":"2022-04-05T06:56:48.514973Z","shell.execute_reply.started":"2022-04-05T06:56:23.789736Z","shell.execute_reply":"2022-04-05T06:56:48.514133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:48.516034Z","iopub.execute_input":"2022-04-05T06:56:48.516249Z","iopub.status.idle":"2022-04-05T06:56:48.531647Z","shell.execute_reply.started":"2022-04-05T06:56:48.516219Z","shell.execute_reply":"2022-04-05T06:56:48.530901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Melihat Statistika Deskriptif Dari Dataframe","metadata":{}},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:56:48.532689Z","iopub.execute_input":"2022-04-05T06:56:48.533396Z","iopub.status.idle":"2022-04-05T06:57:25.157753Z","shell.execute_reply.started":"2022-04-05T06:56:48.533355Z","shell.execute_reply":"2022-04-05T06:57:25.156843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.describe()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:57:25.159075Z","iopub.execute_input":"2022-04-05T06:57:25.159380Z","iopub.status.idle":"2022-04-05T06:58:41.313968Z","shell.execute_reply.started":"2022-04-05T06:57:25.159336Z","shell.execute_reply":"2022-04-05T06:58:41.313181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PREPROCESSING","metadata":{}},{"cell_type":"markdown","source":"## Mengubah dtype dan adding feature baru\nVariabel timestamps masih terbaca sebagai objek, maka akan diubah menjadi datetime dan di breakdown menjadi \"day\", \"hour\", \"weekday\", \"month\", dan \"year\" untuk kemudahan visualisasi nantinya.","metadata":{}},{"cell_type":"code","source":"import datetime as dt\ntrain[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\ntrain[\"day\"]       = train[\"timestamp\"].dt.day\ntrain[\"hour\"]      = train[\"timestamp\"].dt.hour\ntrain[\"week\"]      = train[\"timestamp\"].dt.weekday\ntrain[\"month\"]     = train[\"timestamp\"].dt.month\ntrain[\"year\"]      = train[\"timestamp\"].dt.year\ntrain              = train.drop(\"timestamp\", axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:58:41.314928Z","iopub.execute_input":"2022-04-05T06:58:41.315125Z","iopub.status.idle":"2022-04-05T06:58:54.681120Z","shell.execute_reply.started":"2022-04-05T06:58:41.315101Z","shell.execute_reply":"2022-04-05T06:58:54.679840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test[\"timestamp\"] = pd.to_datetime(test[\"timestamp\"])\ntest[\"day\"]       = test[\"timestamp\"].dt.day\ntest[\"hour\"]      = test[\"timestamp\"].dt.hour\ntest[\"week\"]      = test[\"timestamp\"].dt.weekday\ntest[\"month\"]     = test[\"timestamp\"].dt.month\ntest[\"year\"]      = test[\"timestamp\"].dt.year\ntest              = test.drop(\"timestamp\", axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:58:54.684948Z","iopub.execute_input":"2022-04-05T06:58:54.685572Z","iopub.status.idle":"2022-04-05T06:59:22.934711Z","shell.execute_reply.started":"2022-04-05T06:58:54.685513Z","shell.execute_reply":"2022-04-05T06:59:22.933023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Mencari missing value\nAkan dicari persentase missing value dari tiap-tiap variabel","metadata":{}},{"cell_type":"markdown","source":"### Data Train","metadata":{}},{"cell_type":"code","source":"train_missing_values = pd.DataFrame(train.isnull().sum() * 100 / len(train))\ntrain_missing_values.columns = [\"Missing Values\"]\n\ntrain_missing_values = train_missing_values[train_missing_values[\"Missing Values\"] != 0]\ntrain_missing_values.sort_values(by = \"Missing Values\", axis = 0, ascending = False, inplace = True)\ntrain_missing_values","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:59:22.936892Z","iopub.execute_input":"2022-04-05T06:59:22.937270Z","iopub.status.idle":"2022-04-05T06:59:24.434037Z","shell.execute_reply.started":"2022-04-05T06:59:22.937235Z","shell.execute_reply":"2022-04-05T06:59:24.433056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Visualisasi\nplt.figure(figsize=(8,8))\nsns.barplot(x = train_missing_values['Missing Values'], y = train_missing_values.index).set_title(\"Persentase Missing Value Variabel Pada Data Train\", fontsize=16)\nplt.xlabel('Persentase Missing', fontsize=12)\nplt.ylabel('Feature', fontsize=12)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:59:24.435290Z","iopub.execute_input":"2022-04-05T06:59:24.435919Z","iopub.status.idle":"2022-04-05T06:59:24.755168Z","shell.execute_reply.started":"2022-04-05T06:59:24.435882Z","shell.execute_reply":"2022-04-05T06:59:24.754374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Test","metadata":{}},{"cell_type":"code","source":"test_missing_values = pd.DataFrame(test.isnull().sum() * 100 / len(test))\ntest_missing_values.columns = [\"Missing Values\"]\n\ntest_missing_values = test_missing_values[test_missing_values[\"Missing Values\"] != 0]\ntest_missing_values.sort_values(by = \"Missing Values\", axis = 0, ascending = False, inplace = True)\ntest_missing_values","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:59:24.756475Z","iopub.execute_input":"2022-04-05T06:59:24.756713Z","iopub.status.idle":"2022-04-05T06:59:27.288839Z","shell.execute_reply.started":"2022-04-05T06:59:24.756684Z","shell.execute_reply":"2022-04-05T06:59:27.287958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Visualisasi\nplt.figure(figsize=(8,8))\nsns.barplot(x = test_missing_values['Missing Values'], y = test_missing_values.index).set_title(\"Persentase Missing Value Pada Variabel Data Test\", fontsize=16)\nplt.xlabel('Persentase Missing', fontsize=12)\nplt.ylabel('Feature', fontsize=12)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:59:27.289994Z","iopub.execute_input":"2022-04-05T06:59:27.290284Z","iopub.status.idle":"2022-04-05T06:59:27.541993Z","shell.execute_reply.started":"2022-04-05T06:59:27.290254Z","shell.execute_reply":"2022-04-05T06:59:27.541369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Melihat Distribusi\nAkan dilihat distribusi dari masing-masing variabel untuk filling missing value yang ada","metadata":{}},{"cell_type":"code","source":"#fungsi rekursif untuk melihat distribusi data \nsns.set_theme()\n\ndef draw_distribution_graph(dataframe, column_name):\n    \n    fig, ax = plt.subplots(figsize=(8, 8))\n    sns.distplot(dataframe[column_name], color='purple', ax = ax).set_title(\"Distribution Graph for {}\".format(column_name), fontsize=16)\n    plt.xlabel(column_name, fontsize=12)\n    plt.ylabel('Density', fontsize=12)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:59:27.543212Z","iopub.execute_input":"2022-04-05T06:59:27.544290Z","iopub.status.idle":"2022-04-05T06:59:27.552383Z","shell.execute_reply.started":"2022-04-05T06:59:27.544140Z","shell.execute_reply":"2022-04-05T06:59:27.551385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribusi Dari Variabel yang memiliki Missing Value >10% Pada Data Train","metadata":{}},{"cell_type":"code","source":"draw_distribution_graph(train, 'precip_depth_1_hr')","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:59:27.557727Z","iopub.execute_input":"2022-04-05T06:59:27.557964Z","iopub.status.idle":"2022-04-05T07:00:26.395618Z","shell.execute_reply.started":"2022-04-05T06:59:27.557937Z","shell.execute_reply":"2022-04-05T07:00:26.394567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observasi**:<br>\nTerlihat distribusinya berada di sekitar 0, sedangkan pada statistika deskriptifnya terdapat nilai 343, sehingga nilai tersebut dapat diasumsikan sebagai outlier. Maka, untuk mengisi missing value pada variabel ini akan digunakan mediannya.","metadata":{}},{"cell_type":"markdown","source":"### Distribusi Dari Variabel yang memiliki Missing Value >10% Pada Data Test","metadata":{}},{"cell_type":"code","source":"draw_distribution_graph(test, 'precip_depth_1_hr')","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:00:26.396994Z","iopub.execute_input":"2022-04-05T07:00:26.397257Z","iopub.status.idle":"2022-04-05T07:02:22.612699Z","shell.execute_reply.started":"2022-04-05T07:00:26.397228Z","shell.execute_reply":"2022-04-05T07:02:22.611661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observasi**:<br> Terlihat distribusinya berada di sekitar 0, sedangkan pada statistika deskriptifnya terdapat nilai maksimumnya adalah 597, sehingga nilai tersebut dapat diasumsikan sebagai outlier. Maka, untuk mengisi missing value pada variabel ini akan digunakan mediannya.","metadata":{}},{"cell_type":"markdown","source":"### Melihat letak missing value pada variabel yang missing valuesnya <5% pada data train","metadata":{}},{"cell_type":"code","source":"cek_na = train[train.air_temperature.isnull()]\ndisplay(cek_na.head(), cek_na.tail(), cek_na.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:22.614282Z","iopub.execute_input":"2022-04-05T07:02:22.614632Z","iopub.status.idle":"2022-04-05T07:02:22.766639Z","shell.execute_reply.started":"2022-04-05T07:02:22.614588Z","shell.execute_reply":"2022-04-05T07:02:22.765669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cek_na1 = train[train.wind_speed.isnull()]\ndisplay(cek_na1.head(), cek_na1.tail(), cek_na1.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:22.768021Z","iopub.execute_input":"2022-04-05T07:02:22.768278Z","iopub.status.idle":"2022-04-05T07:02:22.914707Z","shell.execute_reply.started":"2022-04-05T07:02:22.768245Z","shell.execute_reply":"2022-04-05T07:02:22.913732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cek_na2 = train[train.dew_temperature.isnull()]\ndisplay(cek_na2.head(), cek_na2.tail(), cek_na2.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:22.916311Z","iopub.execute_input":"2022-04-05T07:02:22.916577Z","iopub.status.idle":"2022-04-05T07:02:23.058636Z","shell.execute_reply.started":"2022-04-05T07:02:22.916545Z","shell.execute_reply":"2022-04-05T07:02:23.057550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Melihat letak missing value pada variabel yang missing valuesnya < 5% pada data test","metadata":{}},{"cell_type":"code","source":"cek_na = test[test.air_temperature.isnull()]\ndisplay(cek_na.head(), cek_na.tail(), cek_na.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:23.060240Z","iopub.execute_input":"2022-04-05T07:02:23.060591Z","iopub.status.idle":"2022-04-05T07:02:23.304484Z","shell.execute_reply.started":"2022-04-05T07:02:23.060545Z","shell.execute_reply":"2022-04-05T07:02:23.303550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cek_na1 = test[test.wind_speed.isnull()]\ndisplay(cek_na1.head(), cek_na1.tail(), cek_na1.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:23.306034Z","iopub.execute_input":"2022-04-05T07:02:23.306377Z","iopub.status.idle":"2022-04-05T07:02:23.570179Z","shell.execute_reply.started":"2022-04-05T07:02:23.306332Z","shell.execute_reply":"2022-04-05T07:02:23.569544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cek_na2 = test[test.dew_temperature.isnull()]\ndisplay(cek_na2.head(), cek_na2.tail(), cek_na2.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:23.571809Z","iopub.execute_input":"2022-04-05T07:02:23.572033Z","iopub.status.idle":"2022-04-05T07:02:23.817621Z","shell.execute_reply.started":"2022-04-05T07:02:23.572005Z","shell.execute_reply":"2022-04-05T07:02:23.816585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Mengatasi missing value pada data train\nkarena variabel \"floor_count\" dan \"year_built\" mengandung sekitar 80% dan 60% missing value, maka kedua kolom ini akan di drop, karena jika diisi akan menyebabkan data tidak akurat karena besarnya >50%, selain itu, kedua variabel tersebut juga tidak berpengaruh terhadap variabel target meter_reading.","metadata":{}},{"cell_type":"code","source":"threshold = len(train) * 0.5\ntrain.dropna(axis=1, thresh = threshold, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:23.818966Z","iopub.execute_input":"2022-04-05T07:02:23.819224Z","iopub.status.idle":"2022-04-05T07:02:26.284940Z","shell.execute_reply.started":"2022-04-05T07:02:23.819191Z","shell.execute_reply":"2022-04-05T07:02:26.283694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Untuk variabel lainnya, missing value akan diimputasi dengan nilai sesuai dengan pengamatan distribusi yang telah dilakukan sebelumnya.\nUntuk cloud_coverage, sea_level_pressure, dan wind_direction telah terlihat distribusinya pada statistika deskriptif, dan distribusinya tidak terlalu jauh antara nilai maksimum dan minimumnya, maka missing valuenya akan diisi dengan mediannya.\nUntuk precip_depth_1_hr juga akan diisi dengan mediannya sesuai dengan observasi yang telah dilakukan di atas.","metadata":{}},{"cell_type":"code","source":"train['cloud_coverage'].fillna(train['cloud_coverage'].median(), inplace=True)\ntrain['sea_level_pressure'].fillna(train['sea_level_pressure'].median(), inplace=True)\ntrain['precip_depth_1_hr'].fillna(train['precip_depth_1_hr'].median(), inplace=True)\ntrain['wind_direction'].fillna(train['wind_direction'].median(), inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:26.286243Z","iopub.execute_input":"2022-04-05T07:02:26.286700Z","iopub.status.idle":"2022-04-05T07:02:27.970035Z","shell.execute_reply.started":"2022-04-05T07:02:26.286659Z","shell.execute_reply":"2022-04-05T07:02:27.968793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Untuk variabel yang missing valuenya < 5%, berdasarkan observasi di atas, imputasi dapat dilakukan dengan dropna karena missing value terjadi karena ketiadaan data di waktu itu (terlihat dari banyaknya kolom yang juga kosong)","metadata":{}},{"cell_type":"code","source":"train = train.dropna(subset=['air_temperature'])\ntrain = train.dropna(subset=['wind_speed'])\ntrain = train.dropna(subset=['dew_temperature'])","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:27.971646Z","iopub.execute_input":"2022-04-05T07:02:27.971959Z","iopub.status.idle":"2022-04-05T07:02:37.585302Z","shell.execute_reply.started":"2022-04-05T07:02:27.971926Z","shell.execute_reply":"2022-04-05T07:02:37.584586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#akan dilihat missing value yang tersisa\ntrain.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:37.586397Z","iopub.execute_input":"2022-04-05T07:02:37.586680Z","iopub.status.idle":"2022-04-05T07:02:38.590036Z","shell.execute_reply.started":"2022-04-05T07:02:37.586646Z","shell.execute_reply":"2022-04-05T07:02:38.589414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Mengatasi missing value pada data test\nkarena variabel \"floor_count\" dan \"year_built\" mengandung sekitar 80% dan 60% missing value, maka kedua kolom ini akan di drop, karena jika diisi akan menyebabkan data tidak akurat karena besarnya >50%","metadata":{}},{"cell_type":"code","source":"threshold = len(test) * 0.5\ntest.dropna(axis=1, thresh = threshold, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:38.591295Z","iopub.execute_input":"2022-04-05T07:02:38.591715Z","iopub.status.idle":"2022-04-05T07:02:43.479566Z","shell.execute_reply.started":"2022-04-05T07:02:38.591672Z","shell.execute_reply":"2022-04-05T07:02:43.478832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Untuk variabel lainnya, missing value akan diimputasi dengan nilai sesuai dengan pengamatan distribusi yang telah dilakukan sebelumnya.\nUntuk cloud_coverage, sea_level_pressure, dan wind_direction telah terlihat distribusinya pada statistika deskriptif, dan distribusinya tidak terlalu jauh antara nilai maksimum dan minimumnya, maka missing valuenya akan diisi dengan mediannya.\nUntuk precip_depth_1_hr juga akan diisi dengan mediannya sesuai dengan observasi yang telah dilakukan di atas.","metadata":{}},{"cell_type":"code","source":"test['cloud_coverage'].fillna(test['cloud_coverage'].median(), inplace=True)\ntest['precip_depth_1_hr'].fillna(test['precip_depth_1_hr'].median(), inplace=True)\ntest['sea_level_pressure'].fillna(test['sea_level_pressure'].median(), inplace=True)\ntest['wind_direction'].fillna(test['wind_direction'].median(), inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:43.480839Z","iopub.execute_input":"2022-04-05T07:02:43.481195Z","iopub.status.idle":"2022-04-05T07:02:47.779076Z","shell.execute_reply.started":"2022-04-05T07:02:43.481165Z","shell.execute_reply":"2022-04-05T07:02:47.778069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Untuk variabel yang missing valuenya < 5%, berdasarkan observasi di atas, imputasi dapat dilakukan dengan dropna karena missing value terjadi karena ketiadaan data di waktu itu (terlihat dari banyaknya kolom yang juga kosong)","metadata":{}},{"cell_type":"code","source":"test = test.dropna(subset=['air_temperature'])\ntest = test.dropna(subset=['wind_speed'])\ntest = test.dropna(subset=['dew_temperature'])","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:02:47.780728Z","iopub.execute_input":"2022-04-05T07:02:47.780986Z","iopub.status.idle":"2022-04-05T07:03:03.893308Z","shell.execute_reply.started":"2022-04-05T07:02:47.780954Z","shell.execute_reply":"2022-04-05T07:03:03.892419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#akan dilihat missing value yangtersisa\ntest.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:03:03.894598Z","iopub.execute_input":"2022-04-05T07:03:03.894840Z","iopub.status.idle":"2022-04-05T07:03:05.941207Z","shell.execute_reply.started":"2022-04-05T07:03:03.894812Z","shell.execute_reply":"2022-04-05T07:03:05.940040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# OUTLIER\nKarena dataframe yang akan dipakai untuk analisis adalah dataframe train, outlier pada dataframe test dapat diabaikan.\n\nPertama-tama akan dilihat boxplot dari masing-masing variabel untuk dilihat persebarannya dan mendeteksi outliernya.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\n\nfor i, col in enumerate(train.describe().columns):\n    plt.subplot(4,5, i+1)\n    sns.boxplot(x=train[col])\n    plt.tight_layout()\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:03:05.942468Z","iopub.execute_input":"2022-04-05T07:03:05.942735Z","iopub.status.idle":"2022-04-05T07:04:29.153775Z","shell.execute_reply.started":"2022-04-05T07:03:05.942703Z","shell.execute_reply":"2022-04-05T07:04:29.153059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dari boxplot, terlihat bahwa masih terdapat banyak outlier. Yang akan diatasi adalah outlier pada meter_reading karena variabel tersebut menjadi fokus (target value) pada analisa ini. Sebelum itu akan dilihat distribusi dari variabel meter_reading.","metadata":{}},{"cell_type":"code","source":"draw_distribution_graph(train, 'meter_reading')","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:04:29.154951Z","iopub.execute_input":"2022-04-05T07:04:29.155636Z","iopub.status.idle":"2022-04-05T07:05:41.700952Z","shell.execute_reply.started":"2022-04-05T07:04:29.155595Z","shell.execute_reply":"2022-04-05T07:05:41.700205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Terlihat bahwa distribusinya terlalu condong ke kiri (tidak normal). Hal ini disebabkan karena outlier tersebut, maka dari itu untuk menormalkan distribusinya, akan digunakan metode log transformation. Hal ini dilakukan untuk mengurangi \"influence\" outlier tersebut terhadap analisa data nantinya.","metadata":{}},{"cell_type":"code","source":"train['meter_reading']=np.log1p(train['meter_reading'])\ndraw_distribution_graph(train, 'meter_reading')","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:05:41.702155Z","iopub.execute_input":"2022-04-05T07:05:41.702922Z","iopub.status.idle":"2022-04-05T07:06:59.183842Z","shell.execute_reply.started":"2022-04-05T07:05:41.702884Z","shell.execute_reply":"2022-04-05T07:06:59.182921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"terlihat bahwa meter_reading sudah lebih \"normal\" dan efek dari outlier tersebut telah tereduksi. Untuk lebih meyakinkan, akan dicek visualisasi boxplotnya.","metadata":{}},{"cell_type":"code","source":"sns.boxplot(x=train['meter_reading'])","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:06:59.185360Z","iopub.execute_input":"2022-04-05T07:06:59.185624Z","iopub.status.idle":"2022-04-05T07:07:01.256604Z","shell.execute_reply.started":"2022-04-05T07:06:59.185591Z","shell.execute_reply":"2022-04-05T07:07:01.255564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ANALISA\nKarena fokusnya pada penggunaan energi, maka akan dianalisa:\n## Jenis Meteran Yang Digunakan Di Gedung-Gedung","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8,8))\nmeter_types_labels= ['Listrik', 'Air Dingin', 'Uap', 'Air Panas']\nsns.countplot(x = 'meter', data = train).set_title(\"Frek Pengukuran Yang Dilakukan Tiap-Tiap Meteran\", fontsize=16)\nax.set_xticklabels(meter_types_labels)\nplt.xlabel('Meter Type', fontsize=12)\nplt.ylabel('Readings Count', fontsize=12)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:01.258240Z","iopub.execute_input":"2022-04-05T07:07:01.258523Z","iopub.status.idle":"2022-04-05T07:07:03.168612Z","shell.execute_reply.started":"2022-04-05T07:07:01.258465Z","shell.execute_reply":"2022-04-05T07:07:03.167758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**observasi**: Meteran listrik adalah jenis meteran yang paling banyak di gedung-gedung","metadata":{}},{"cell_type":"markdown","source":"## Rata-rata energi yang dikonsumsi dari masing-masing listrik, air hangat, uap, dan air panas pada seluruh gedung","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8,8))\nmeter_types_labels= ['Listrik', 'Air Dingin', 'Uap', 'Air Panas']\nmeter_types = train.groupby('meter').meter_reading.mean().reset_index()\nsns.barplot(x ='meter', y ='meter_reading', data = meter_types).set_title(\"Rata-Rata Penggunaan Energi Yang Dibaca Tiap-Tiap Meter Type\", fontsize=16)\nax.set_xticklabels(meter_types_labels)\nplt.xlabel('Meter Type', fontsize=12)\nplt.ylabel('Rata-Rata', fontsize=12)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:03.170070Z","iopub.execute_input":"2022-04-05T07:07:03.170954Z","iopub.status.idle":"2022-04-05T07:07:03.888556Z","shell.execute_reply.started":"2022-04-05T07:07:03.170914Z","shell.execute_reply":"2022-04-05T07:07:03.887503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observasi**: Uap menggunakan energi paling banyak\n\n## Rata-rata penggunaan energi tiap gedung","metadata":{}},{"cell_type":"code","source":"building_groups = train.groupby(['building_id']).meter_reading.mean().reset_index()\n\nplt.figure(figsize=(8,8))\nsns.lineplot(x = building_groups['building_id'], y = building_groups.meter_reading).set_title(\"Rata-Rata Pengukuran Energi Yang Dilakukan Tiap Gedung\", fontsize=16)\nplt.xlabel('Building Id', fontsize=12)\nplt.ylabel('Average Log Meter Reading', fontsize=12)\nplt.xticks(rotation=45)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:03.890104Z","iopub.execute_input":"2022-04-05T07:07:03.890451Z","iopub.status.idle":"2022-04-05T07:07:04.772110Z","shell.execute_reply.started":"2022-04-05T07:07:03.890418Z","shell.execute_reply":"2022-04-05T07:07:04.771183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Rata-Rata Penggunaan Energi Perjam dan Perbulan","metadata":{}},{"cell_type":"code","source":"hour_groups = train.groupby(['hour']).meter_reading.mean().reset_index()\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize = (20,8))\nfig.suptitle('Rata-Rata Penggunaan Energi')\nsns.lineplot(x = hour_groups['hour'], y = hour_groups['meter_reading'], color='green', ax = ax1)\nax1.set_title('Rata-Rata Penggunaan Energi Per Jam')\nmonth_groups = train.groupby(['month']).meter_reading.mean().reset_index()\nsns.lineplot(x = month_groups['month'], y = month_groups['meter_reading'], color='purple', ax = ax2)\nax2.set_title('Rata-Rata Penggunaan Energi Per Bulan')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:04.773812Z","iopub.execute_input":"2022-04-05T07:07:04.774448Z","iopub.status.idle":"2022-04-05T07:07:06.215594Z","shell.execute_reply.started":"2022-04-05T07:07:04.774407Z","shell.execute_reply":"2022-04-05T07:07:06.214511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**observasi**: Energi paling banyak digunakan di pukul 10-15 yang mana waktu tersebut merupakan waktu produktif (waktu kerja)\n## Rata-Rata Penggunaan Energi Berdasarkan Primary Use","metadata":{}},{"cell_type":"code","source":"prim_use = train.groupby(['primary_use'])\nprim_use = prim_use['meter_reading'].mean().reset_index()\n\nprim_use","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:06.216918Z","iopub.execute_input":"2022-04-05T07:07:06.217186Z","iopub.status.idle":"2022-04-05T07:07:06.442045Z","shell.execute_reply.started":"2022-04-05T07:07:06.217145Z","shell.execute_reply":"2022-04-05T07:07:06.441173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**observasi**: Gedung Serbaguna menggunakan energi paling tinggi","metadata":{}},{"cell_type":"markdown","source":"# HASIL PREPROCESSING","metadata":{}},{"cell_type":"code","source":"test","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:06.443658Z","iopub.execute_input":"2022-04-05T07:07:06.443910Z","iopub.status.idle":"2022-04-05T07:07:06.475616Z","shell.execute_reply.started":"2022-04-05T07:07:06.443879Z","shell.execute_reply":"2022-04-05T07:07:06.474575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:06.477214Z","iopub.execute_input":"2022-04-05T07:07:06.477472Z","iopub.status.idle":"2022-04-05T07:07:06.514209Z","shell.execute_reply.started":"2022-04-05T07:07:06.477428Z","shell.execute_reply":"2022-04-05T07:07:06.513156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.to_csv('EDA_KELOMPOKE.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:07:06.516062Z","iopub.execute_input":"2022-04-05T07:07:06.516615Z","iopub.status.idle":"2022-04-05T07:14:05.294323Z","shell.execute_reply.started":"2022-04-05T07:07:06.516565Z","shell.execute_reply":"2022-04-05T07:14:05.293384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.to_csv('EDA_KELOMPOKE_TRAIN.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T07:14:05.296235Z","iopub.execute_input":"2022-04-05T07:14:05.296606Z","iopub.status.idle":"2022-04-05T07:17:41.509503Z","shell.execute_reply.started":"2022-04-05T07:14:05.296557Z","shell.execute_reply":"2022-04-05T07:17:41.508559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}