{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### 학습 목표\n\n- 새로운 데이터가 주어지면, 우리는 어떻게 데이터와 친해져야 할까요? (EDA process)\n\n\n- 데이터가 대용량일 때, 특정 기술을 포함한 다양한 방법으로 어떻게 해결해야 할까요?\n\n\n- 여러 개의 데이터를 합쳐서 평가할 때는 어떤 기준을 사용해야 할까요?\n\n\n- 공부했던 예시들을 가지고, 여러분 만의 EDA Notebook을 만들어보세요!","metadata":{}},{"cell_type":"markdown","source":"## ASHRAE ENERGY PREDICTION III","metadata":{"id":"PRDnm0lhzEyp","papermill":{"duration":0.070817,"end_time":"2022-01-13T16:00:07.942753","exception":false,"start_time":"2022-01-13T16:00:07.871936","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"- 건물 에너지 사용량 예측\n\n- 분석 문제 정의에 따라 시계열 분석 방법으로도 분석이 가능하다.\n(우리는 시계열 분석으로 풀지는 않습니다)\n\n- 주어진 건물별 에너지 사용량 정보와, 날씨 정보를 통해서 건물별 에너지 사용량 예측 문제를 풀어봅시다.\n\n[Data] https://www.kaggle.com/c/ashrae-energy-prediction/data","metadata":{"id":"0s21TZMROe2Y","papermill":{"duration":0.067229,"end_time":"2022-01-13T16:00:08.506704","exception":false,"start_time":"2022-01-13T16:00:08.439475","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 데이터 및 라이브러리 불러오기","metadata":{"id":"V1xIUKqK_mQn","papermill":{"duration":0.067506,"end_time":"2022-01-13T16:00:10.312724","exception":false,"start_time":"2022-01-13T16:00:10.245218","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\n\ndata_path = \"../input/ashrae-energy-prediction/\"\n\ntrain_path = data_path + \"train.csv\"\n#test_path = data_path + \"test.csv\"\n\nbuilding_path = data_path + \"building_metadata.csv\"\n\nweather_train_path = data_path + \"weather_train.csv\"\n#weather_test_path = data_path + \"weather_test.csv\"","metadata":{"papermill":{"duration":0.053846,"end_time":"2022-01-13T16:00:11.902890","exception":false,"start_time":"2022-01-13T16:00:11.849044","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-20T05:10:42.020773Z","iopub.execute_input":"2022-05-20T05:10:42.021094Z","iopub.status.idle":"2022-05-20T05:10:42.496493Z","shell.execute_reply.started":"2022-05-20T05:10:42.021036Z","shell.execute_reply":"2022-05-20T05:10:42.495724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ***질문 환영!!!***","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv(train_path)\n#test_data = pd.read_csv(test_path)\n\nbuilding_data = pd.read_csv(building_path)\n\nweather_train_data = pd.read_csv(weather_train_path)\n#weather_test_data = pd.read_csv(weather_test_path)\nprint(train_data.shape, building_data.shape, weather_train_data.shape)","metadata":{"papermill":{"duration":43.698341,"end_time":"2022-01-13T16:00:55.646301","exception":false,"start_time":"2022-01-13T16:00:11.947960","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-20T05:10:42.501364Z","iopub.execute_input":"2022-05-20T05:10:42.501617Z","iopub.status.idle":"2022-05-20T05:10:50.234223Z","shell.execute_reply.started":"2022-05-20T05:10:42.501580Z","shell.execute_reply":"2022-05-20T05:10:50.233368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### (OPTIONAL) Reducing the memory usage of the data\n\n- ASHRAE 데이터는 데이터셋이 매우 큽니다. 메모리를 줄여서 처리하기 편하게 만듭니다.\n\n\n- 이전 실습교안에서 만들었던 함수를 불러와서 사용합시다.","metadata":{"id":"la1s0NjRJvGK","papermill":{"duration":0.050451,"end_time":"2022-01-13T16:01:07.799163","exception":false,"start_time":"2022-01-13T16:01:07.748712","status":"completed"},"tags":[]}},{"cell_type":"code","source":"## Reducing DataFrame memory size by ~65%\n#####\ndef reduce_mem_usage(props):\n    start_mem_usg = props.memory_usage().sum() / 1024**2 \n    print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n    NAlist = [] # Keeps track of columns that have missing values filled in. \n    for col in props.columns:\n        if props[col].dtype != object:  # Exclude strings\n            \n            # Print current column type\n            print(\"******************************\")\n            print(\"Column: \",col)\n            print(\"dtype before: \",props[col].dtype)\n            \n            # make variables for Int, max and min\n            IsInt = False\n            mx = props[col].max()\n            mn = props[col].min()\n            \n            # Integer does not support NA, therefore, NA needs to be filled\n            if not np.isfinite(props[col]).all(): \n                NAlist.append(col)\n                props[col].fillna(mn-1,inplace=True)  \n            # test if column can be converted to an integer\n            asint = props[col].fillna(0).astype(np.int64)\n            result = (props[col] - asint)\n            result = result.sum()\n            if result > -0.01 and result < 0.01:\n                IsInt = True\n\n            \n            # Make Integer/unsigned Integer datatypes\n            if IsInt:\n                if mn >= 0:\n                    if mx < 255:\n                        props[col] = props[col].astype(np.uint8)\n                    elif mx < 65535:\n                        props[col] = props[col].astype(np.uint16)\n                    elif mx < 4294967295:\n                        props[col] = props[col].astype(np.uint32)\n                    else:\n                        props[col] = props[col].astype(np.uint64)\n                else:\n                    if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n                        props[col] = props[col].astype(np.int8)\n                    elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n                        props[col] = props[col].astype(np.int16)\n                    elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n                        props[col] = props[col].astype(np.int32)\n                    elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n                        props[col] = props[col].astype(np.int64)    \n            \n            # Make float datatypes 32 bit\n            else:\n                props[col] = props[col].astype(np.float32)\n            \n            # Print new column type\n            print(\"dtype after: \",props[col].dtype)\n            print(\"******************************\")\n    \n    # Print final result\n    print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n    mem_usg = props.memory_usage().sum() / 1024**2 \n    print(\"Memory usage is: \",mem_usg,\" MB\")\n    print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\\n\\n\")\n    return props, NAlist","metadata":{"id":"v3yIjOevKKKY","papermill":{"duration":0.06777,"end_time":"2022-01-13T16:01:07.915936","exception":false,"start_time":"2022-01-13T16:01:07.848166","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-20T05:10:50.235452Z","iopub.execute_input":"2022-05-20T05:10:50.235705Z","iopub.status.idle":"2022-05-20T05:10:50.250501Z","shell.execute_reply.started":"2022-05-20T05:10:50.235648Z","shell.execute_reply":"2022-05-20T05:10:50.249552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## apply\ntrain_data, _ = reduce_mem_usage(train_data)\nweather_train_data, _ = reduce_mem_usage(weather_train_data)\nbuilding_data, _ = reduce_mem_usage(building_data)","metadata":{"id":"XYZL4jkJeVyB","outputId":"01e04941-ed2c-47a7-d13f-80fe439f5148","papermill":{"duration":7.493875,"end_time":"2022-01-13T16:01:15.458868","exception":false,"start_time":"2022-01-13T16:01:07.964993","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-20T05:34:04.847826Z","iopub.execute_input":"2022-05-20T05:34:04.848155Z","iopub.status.idle":"2022-05-20T05:34:06.705410Z","shell.execute_reply.started":"2022-05-20T05:34:04.848124Z","shell.execute_reply":"2022-05-20T05:34:06.704695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Do your Own EDA\n\n- 각자의 방법대로 EDA를 해봅시다.\n\n\n- EDA를 할 때 중요한 사항은, \"내가 이 데이터에서 보고 싶은 포인트가 무엇인가?\" 입니다.\n\n\n- EDA에서 정답은 없습니다. 다양한 케이스를 살펴보면서 감을 익힐뿐입니다.\n\n\n- 전혀 감을 잡지 못하겠다고 하면, kaggle에 있는 좋은 EDA notebook을 필사해보는 것도 좋습니다.\n\n> 추천 1. https://www.kaggle.com/code/caesarlupum/ashrae-start-here-a-gentle-introduction\n\n> 추천 2. https://www.kaggle.com/code/jaseziv83/a-deep-dive-eda-into-all-variables\n\n> 추천 3. https://www.kaggle.com/code/hmendonca/starter-eda-and-feature-selection-ashrae3","metadata":{}},{"cell_type":"code","source":"for col in train_data.columns: # 각 column마다\n    print(f\"{col}:\\t{train_data[col].isnull().mean() * 100:.2f}%\") # column별 결측치 비율","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:19:48.900424Z","iopub.execute_input":"2022-05-20T05:19:48.901011Z","iopub.status.idle":"2022-05-20T05:19:50.341439Z","shell.execute_reply.started":"2022-05-20T05:19:48.900959Z","shell.execute_reply":"2022-05-20T05:19:50.340475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in building_data.columns: # 각 column마다\n    print(f\"{col}:\\t{building_data[col].isnull().mean() * 100:.2f}%\") # column별 결측치 비율","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:19:23.217495Z","iopub.execute_input":"2022-05-20T05:19:23.218428Z","iopub.status.idle":"2022-05-20T05:19:23.227863Z","shell.execute_reply.started":"2022-05-20T05:19:23.218385Z","shell.execute_reply":"2022-05-20T05:19:23.226930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in weather_train_data.columns: # 각 column마다\n    print(f\"{col}:\\t{weather_train_data[col].isnull().mean() * 100:.2f}%\") # column별 결측치 비율","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:19:37.248754Z","iopub.execute_input":"2022-05-20T05:19:37.249042Z","iopub.status.idle":"2022-05-20T05:19:37.282033Z","shell.execute_reply.started":"2022-05-20T05:19:37.249013Z","shell.execute_reply":"2022-05-20T05:19:37.281130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 데이터를 모두 합쳐보세요!\nfull_data = pd.merge(train_data, building_data, on=\"building_id\")\nfull_data = pd.merge(full_data, weather_train_data, on=[\"site_id\", \"timestamp\"], how=\"left\")\nfull_data","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:34:38.774753Z","iopub.execute_input":"2022-05-20T05:34:38.775523Z","iopub.status.idle":"2022-05-20T05:34:51.899348Z","shell.execute_reply.started":"2022-05-20T05:34:38.775486Z","shell.execute_reply":"2022-05-20T05:34:51.898342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_data.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:34:51.901103Z","iopub.execute_input":"2022-05-20T05:34:51.901600Z","iopub.status.idle":"2022-05-20T05:34:51.914253Z","shell.execute_reply.started":"2022-05-20T05:34:51.901554Z","shell.execute_reply":"2022-05-20T05:34:51.913697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_data[full_data.isnull().any(axis=1)]","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:15:46.473385Z","iopub.execute_input":"2022-05-20T05:15:46.474432Z","iopub.status.idle":"2022-05-20T05:15:53.973877Z","shell.execute_reply.started":"2022-05-20T05:15:46.474382Z","shell.execute_reply":"2022-05-20T05:15:53.973012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in full_data.columns: # 각 column마다\n    print(f\"{col}:\\t{full_data[col].isnull().mean() * 100:.2f}%\") # column별 결측치 비율","metadata":{"execution":{"iopub.status.busy":"2022-05-20T05:17:13.154293Z","iopub.execute_input":"2022-05-20T05:17:13.154909Z","iopub.status.idle":"2022-05-20T05:17:15.320315Z","shell.execute_reply.started":"2022-05-20T05:17:13.154867Z","shell.execute_reply":"2022-05-20T05:17:15.319434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion\n\n- 여러분들의 내린 이 데이터에서 중요한 지표는 무엇인가요?\n\n\n- 가장 중요하다고 생각하는 그래프 3개를 제출해주세요. (핵심 포인트가 들어간 그래프)\n(plt.savefig 함수를 사용합니다)\n\n\n- Energy prediction을 하기 위해서 어떤 feature를 사용하는 것이 좋은 방법인지 생각을 정리해보세요.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}