{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### 학습 목표\n\n- 초대용량 데이터는 어떻게 처리할까요?\n\n\n- 데이터를 분할하여 불러오고 처리하는 방법에 대해서 연습해봅니다.\n\n\n- 불량 검출 문제에 대해 살펴봅니다.\n\n\n- Class imbalance 문제를 해결하는 방법에 대해 알아봅니다.","metadata":{}},{"cell_type":"markdown","source":"## Bosch Production Line Performance\n\n- 공정과정 데이터를 통해 제품 내부 불량 검출하기\n\n- 주어진 공정변수를 이용하여 정상/불량 분류하기","metadata":{"id":"U0IhpuicpE1o"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ntrain_cat_part = pd.read_csv(\"../input/bosch-production-line-performance/train_categorical.csv.zip\",\n                            nrows=1000)\ntrain_numeric_part = pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                                nrows=1000)\ntrain_date_part = pd.read_csv(\"../input/bosch-production-line-performance/train_date.csv.zip\",\n                             nrows=1000)\nprint(train_cat_part.shape, train_numeric_part.shape, train_date_part.shape)","metadata":{"id":"improving-limitation","outputId":"874047c9-4bfc-47a0-ee85-24ceb5cf49e1","execution":{"iopub.status.busy":"2022-05-20T03:37:47.048238Z","iopub.execute_input":"2022-05-20T03:37:47.048609Z","iopub.status.idle":"2022-05-20T03:37:49.328486Z","shell.execute_reply.started":"2022-05-20T03:37:47.048488Z","shell.execute_reply":"2022-05-20T03:37:49.326963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 대용량 데이터가 메인 메모리에 한번에 적재가 안될때!\n\n1. 사용할 column을 몇개 불러온다. (usecols)\n\n2. 데이터를 쪼개서 임시로 불러와서 계산하고, 다시 불러온다. (chunksize)","metadata":{}},{"cell_type":"code","source":"train_date_part # start_station, end_station","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:37:49.331211Z","iopub.execute_input":"2022-05-20T03:37:49.331579Z","iopub.status.idle":"2022-05-20T03:37:49.385259Z","shell.execute_reply.started":"2022-05-20T03:37:49.331545Z","shell.execute_reply":"2022-05-20T03:37:49.384160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### train_date 데이터를 요약해봅시다.\n\n- 각 row마다 측정이 처음된 station 번호와, 마지막으로 측정이 된 station 번호를 찾습니다.","metadata":{}},{"cell_type":"code","source":"# 1. station별 column 찾기 : 1157개 -> 52개\ncounts = train_date_part.count() # column이 index로 들어가 있는 Series를 만들려고.\ndate_cols = counts.reset_index()[\"index\"].str.split(\"_\", expand=True)\ncol_idx = date_cols.drop_duplicates(1).index  # column 1을 기준으로 중복 제거\ndate_cols = train_date_part.columns[col_idx]\ndate_cols","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:37:49.386651Z","iopub.execute_input":"2022-05-20T03:37:49.386908Z","iopub.status.idle":"2022-05-20T03:37:49.426294Z","shell.execute_reply.started":"2022-05-20T03:37:49.386877Z","shell.execute_reply":"2022-05-20T03:37:49.425455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2. start_station, end_station 찾기\n\ntrain_date = pd.read_csv(\"../input/bosch-production-line-performance/train_date.csv.zip\",\n                        usecols=date_cols)\n\n## 전체 데이터에서 column 순서대로 scan하면서 처음으로 NaN이 아닌 column의 station이 start_station,\n## 마지막까지 scan하면서 마지막으로 NaN이 아니었던 column의 station이 end_station.\ntrain_date[\"start_station\"] = -1\ntrain_date[\"end_station\"] = -1\n\nfor col in train_date.drop(columns=[\"Id\", \"start_station\", \"end_station\"]).columns:\n    notnulls = ~train_date[col].isnull() # null이 아닌 row들의 위치\n    station_name = int(col.split(\"_\")[1][1:])\n    \n    train_date.loc[(notnulls) & (train_date.start_station == -1), \"start_station\"] = station_name\n    train_date.loc[(notnulls), \"end_station\"] = station_name\n    \ntrain_date","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:37:49.428749Z","iopub.execute_input":"2022-05-20T03:37:49.429024Z","iopub.status.idle":"2022-05-20T03:38:39.558034Z","shell.execute_reply.started":"2022-05-20T03:37:49.428993Z","shell.execute_reply":"2022-05-20T03:38:39.557389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_date = train_date[[\"Id\", \"start_station\", \"end_station\"]]\ntrain_date","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:38:39.559583Z","iopub.execute_input":"2022-05-20T03:38:39.560435Z","iopub.status.idle":"2022-05-20T03:38:39.580395Z","shell.execute_reply.started":"2022-05-20T03:38:39.560385Z","shell.execute_reply":"2022-05-20T03:38:39.579474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 차원의 저주(Curse of Dimensionality)\n\n- 머신러닝 모델이 데이터가 존재하는 공간이 너무 고차원일 때, <=> feature(column)가 너무 많을 때\n\n- 머신러닝 모델이 예측력을 잃어버리는 문제. (성능이 저하)\n\n\n- 해결책\n\n> 차원을 줄입니다! (= column수를 줄인다)\n\n> 1. feature selection : 분석에 사용할 column을 직접 고른다.\n>> 결측치가 많은 column을 버리거나, 도메인 지식으로 필요한 변수를 고르거나, ....\n\n> 2. feature extraction : dimensionality reduction method(e.g. PCA)를 사용한다.","metadata":{}},{"cell_type":"code","source":"missing_ratio = pd.Series(index=train_numeric_part.columns,\n                         data=np.zeros(len(train_numeric_part.columns))) # 970\ncolumn_means = pd.Series(index=train_numeric_part.columns,\n                         data=np.zeros(len(train_numeric_part.columns)))\nlength = 0\n\nfor chunk in pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                         chunksize=100000):\n    #display(chunk.isnull().sum())\n    temp = chunk.isnull().sum()  # chunk별 결측치 합\n    temp2 = chunk.sum()\n    \n    length = length + len(chunk)\n    missing_ratio = missing_ratio + temp  # 결측치 누적합\n    column_means = column_means + temp2  # 실제 값의 누적합\n    #display(missing_ratio + chunk.isnull().sum())\n    #break\n\ndisplay(missing_ratio) # 결측치 개수\ndisplay(column_means / length)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:38:39.582242Z","iopub.execute_input":"2022-05-20T03:38:39.582773Z","iopub.status.idle":"2022-05-20T03:40:52.873002Z","shell.execute_reply.started":"2022-05-20T03:38:39.582726Z","shell.execute_reply":"2022-05-20T03:40:52.871369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_ratio = missing_ratio / length  # 비율로 변환","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:40:52.874789Z","iopub.execute_input":"2022-05-20T03:40:52.875640Z","iopub.status.idle":"2022-05-20T03:40:52.881731Z","shell.execute_reply.started":"2022-05-20T03:40:52.875588Z","shell.execute_reply":"2022-05-20T03:40:52.880292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 결측 비율이 50%를 넘지 않는 column들만 뽑아오겠다.\nusecols = train_numeric_part.columns[missing_ratio <= 0.5]\nusecols","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:40:52.883127Z","iopub.execute_input":"2022-05-20T03:40:52.883369Z","iopub.status.idle":"2022-05-20T03:40:52.899747Z","shell.execute_reply.started":"2022-05-20T03:40:52.883339Z","shell.execute_reply":"2022-05-20T03:40:52.898137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. 대용량 데이터 다루기\n\nTechnique1. 잘라서 불러오기\n\n\nTechnique2. Reducing Memory Technique (dtype 변경하기)","metadata":{}},{"cell_type":"code","source":"n_components = 15\n\nfrom sklearn.decomposition import PCA\ndata = pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                        usecols=usecols)\ndata = data.fillna(0) # 결측치를 0으로 채움\nids = data.Id\nresponses = data.Response\ndata = data.drop(columns=[\"Id\", \"Response\"])\npca = PCA(n_components=n_components)\nX = pca.fit_transform(data)\npca_df = pd.DataFrame(columns=[f\"PC{i}\" for i in range(1, n_components+1)],\n                     data=X)\ndisplay(pca_df)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:40:52.902250Z","iopub.execute_input":"2022-05-20T03:40:52.902640Z","iopub.status.idle":"2022-05-20T03:42:19.476531Z","shell.execute_reply.started":"2022-05-20T03:40:52.902589Z","shell.execute_reply":"2022-05-20T03:42:19.475310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 데이터 병합\n\n- 학습에 사용할 데이터로 합쳐줍니다.\n\n- Numeric Data는 PCA로, Date Data는 start_station, end_station 정보를 사용했습니다.","metadata":{}},{"cell_type":"code","source":"## 데이터를 합쳐줍니다.\n# pca_df (numeric 데이터로 만든 데이터) + train_date(date 데이터로 만든 데이터)\npca_df[\"Id\"] = ids\npca_df[\"Response\"] = responses\nX = pd.merge(train_date, pca_df, on=\"Id\")\nX","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:42:19.479225Z","iopub.execute_input":"2022-05-20T03:42:19.479557Z","iopub.status.idle":"2022-05-20T03:42:20.326529Z","shell.execute_reply.started":"2022-05-20T03:42:19.479504Z","shell.execute_reply":"2022-05-20T03:42:20.325624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### if, ML","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n#from sklearn.ensemble import RandomForestClassifier\nfrom lightgbm.sklearn import LGBMClassifier\nfrom sklearn.metrics import matthews_corrcoef\n\n\ny = X.Response\nX = X.drop(columns=[\"Id\", \"Response\"])\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=0xC0FFEE,\n                                                 stratify=y)\n\nclf = LGBMClassifier(max_depth=8, n_estimators=200, random_state=0xC0FFEE, verbose=1)\nclf.fit(X_train, y_train)\n\npred = clf.predict(X_train)\npred2 = clf.predict(X_val)\n\nprint(\"Train MCC : %.4f\" % matthews_corrcoef(y_train, pred))\nprint(\"Validation MCC : %.4f\" % matthews_corrcoef(y_val, pred2))","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:46:42.479092Z","iopub.execute_input":"2022-05-20T03:46:42.479459Z","iopub.status.idle":"2022-05-20T03:46:58.734596Z","shell.execute_reply.started":"2022-05-20T03:46:42.479418Z","shell.execute_reply":"2022-05-20T03:46:58.733814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\n# 사용한 feature vector 저장\nwith open(\"train_data.pk\", \"wb\") as f:\n    pickle.dump(X, f)\n    \n# 학습한 LightGBM 모델 저장\nwith open(\"LGBM_model.pk\", \"wb\") as f:\n    pickle.dump(clf, f)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:57:24.297990Z","iopub.execute_input":"2022-05-20T03:57:24.298300Z","iopub.status.idle":"2022-05-20T03:57:24.763192Z","shell.execute_reply.started":"2022-05-20T03:57:24.298268Z","shell.execute_reply":"2022-05-20T03:57:24.762149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Prediction\n\n- Test data를 불러와서, \"동일한\" 전처리를 수행합니다.\n\n- 동일한 column(=feature)로 구성된 test data를 학습시킨 모델로 inference한 결과를 제출합니다.","metadata":{}},{"cell_type":"code","source":"# test를 하기전에 이미 만들었던 데이터들은 메모리 최적화를 위해 지워줍니다.\nimport gc\n\ndel X\ndel train_date\ndel train_cat_part\ndel train_date_part\ndel train_numeric_part\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:49:45.928437Z","iopub.execute_input":"2022-05-20T03:49:45.928768Z","iopub.status.idle":"2022-05-20T03:49:46.240873Z","shell.execute_reply.started":"2022-05-20T03:49:45.928735Z","shell.execute_reply":"2022-05-20T03:49:46.239492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 1. test data Load\ntest_cat_part = pd.read_csv(\"../input/bosch-production-line-performance/test_categorical.csv.zip\",\n                            nrows=1000)\ntest_numeric_part = pd.read_csv(\"../input/bosch-production-line-performance/test_numeric.csv.zip\",\n                                nrows=1000)\ntest_date_part = pd.read_csv(\"../input/bosch-production-line-performance/test_date.csv.zip\",\n                             nrows=1000)\nprint(test_cat_part.shape, test_numeric_part.shape, test_date_part.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:48:45.086008Z","iopub.execute_input":"2022-05-20T03:48:45.086731Z","iopub.status.idle":"2022-05-20T03:48:45.963760Z","shell.execute_reply.started":"2022-05-20T03:48:45.086684Z","shell.execute_reply":"2022-05-20T03:48:45.962727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2. preprocessing\n# 2-1. test_date_part\ncounts = test_date_part.count() # column이 index로 들어가 있는 Series를 만들려고.\ndate_cols = counts.reset_index()[\"index\"].str.split(\"_\", expand=True)\ncol_idx = date_cols.drop_duplicates(1).index  # column 1을 기준으로 중복 제거\ndate_cols = test_date_part.columns[col_idx]\n\n\n# 2. start_station, end_station 찾기\n\ntest_date = pd.read_csv(\"../input/bosch-production-line-performance/test_date.csv.zip\",\n                        usecols=date_cols)\n\n## 전체 데이터에서 column 순서대로 scan하면서 처음으로 NaN이 아닌 column의 station이 start_station,\n## 마지막까지 scan하면서 마지막으로 NaN이 아니었던 column의 station이 end_station.\ntest_date[\"start_station\"] = -1\ntest_date[\"end_station\"] = -1\n\nfor col in test_date.drop(columns=[\"Id\", \"start_station\", \"end_station\"]).columns:\n    notnulls = ~test_date[col].isnull() # null이 아닌 row들의 위치\n    station_name = int(col.split(\"_\")[1][1:])\n    \n    test_date.loc[(notnulls) & (test_date.start_station == -1), \"start_station\"] = station_name\n    test_date.loc[(notnulls), \"end_station\"] = station_name\n    \ndisplay(test_date)\ntest_date = test_date[[\"Id\", \"start_station\", \"end_station\"]]","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:53:06.185284Z","iopub.execute_input":"2022-05-20T03:53:06.185649Z","iopub.status.idle":"2022-05-20T03:53:54.633416Z","shell.execute_reply.started":"2022-05-20T03:53:06.185608Z","shell.execute_reply":"2022-05-20T03:53:54.632431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2-2. Numeric data에서 missing_value 지우고 PCA\n# missing_columns는 이미 train에서 구했던걸 사용합니다.\n# PCA도 이미 사용했던 걸 사용합니다.\nusecols = usecols.drop(\"Response\") # Response column이 없음.\n\ndata = pd.read_csv(\"../input/bosch-production-line-performance/test_numeric.csv.zip\",\n                        usecols=usecols)\ndata = data.fillna(0) # 결측치를 0으로 채움\nids = data.Id\ndata = data.drop(columns=[\"Id\"])\nX = pca.transform(data) # fit_transform이 아니라 transform 함수.\npca_df = pd.DataFrame(columns=[f\"PC{i}\" for i in range(1, n_components+1)],\n                     data=X)\ndisplay(pca_df)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:55:32.389358Z","iopub.execute_input":"2022-05-20T03:55:32.389749Z","iopub.status.idle":"2022-05-20T03:56:37.112285Z","shell.execute_reply.started":"2022-05-20T03:55:32.389719Z","shell.execute_reply":"2022-05-20T03:56:37.111273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2-3. 데이터 병합\n# pca_df (numeric 데이터로 만든 데이터) + test_date(date 데이터로 만든 데이터)\npca_df[\"Id\"] = ids\nX = pd.merge(test_date, pca_df, on=\"Id\")\nX","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:58:23.640697Z","iopub.execute_input":"2022-05-20T03:58:23.641055Z","iopub.status.idle":"2022-05-20T03:58:24.403417Z","shell.execute_reply.started":"2022-05-20T03:58:23.641023Z","shell.execute_reply":"2022-05-20T03:58:24.402542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test data Inference","metadata":{}},{"cell_type":"code","source":"X_test = X.drop(columns=[\"Id\"])\nresult = clf.predict(X_test)\nresult","metadata":{"execution":{"iopub.status.busy":"2022-05-20T03:59:55.397386Z","iopub.execute_input":"2022-05-20T03:59:55.397747Z","iopub.status.idle":"2022-05-20T03:59:59.971289Z","shell.execute_reply.started":"2022-05-20T03:59:55.397712Z","shell.execute_reply":"2022-05-20T03:59:59.969558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make submission\nsubmission = pd.read_csv(\"../input/bosch-production-line-performance/sample_submission.csv.zip\")\nsubmission","metadata":{"execution":{"iopub.status.busy":"2022-05-20T04:00:14.245508Z","iopub.execute_input":"2022-05-20T04:00:14.246099Z","iopub.status.idle":"2022-05-20T04:00:14.600551Z","shell.execute_reply.started":"2022-05-20T04:00:14.246032Z","shell.execute_reply":"2022-05-20T04:00:14.599337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission[\"Response\"] = result\nsubmission.to_csv(\"submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T04:00:33.631297Z","iopub.execute_input":"2022-05-20T04:00:33.631958Z","iopub.status.idle":"2022-05-20T04:00:35.786177Z","shell.execute_reply.started":"2022-05-20T04:00:33.631917Z","shell.execute_reply":"2022-05-20T04:00:35.785007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}