{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### 학습 목표\n\n- 초대용량 데이터는 어떻게 처리할까요?\n\n\n- 데이터를 분할하여 불러오고 처리하는 방법에 대해서 연습해봅니다.\n\n\n- 불량 검출 문제에 대해 살펴봅니다.\n\n\n- Class imbalance 문제를 해결하는 방법에 대해 알아봅니다.","metadata":{}},{"cell_type":"markdown","source":"## Bosch Production Line Performance\n\n- 공정과정 데이터를 통해 제품 내부 불량 검출하기\n\n- 주어진 공정변수를 이용하여 정상/불량 분류하기","metadata":{"id":"U0IhpuicpE1o"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ntrain_cat_part = pd.read_csv(\"../input/bosch-production-line-performance/train_categorical.csv.zip\",\n                            nrows=1000)\ntrain_numeric_part = pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                                nrows=1000)\ntrain_date_part = pd.read_csv(\"../input/bosch-production-line-performance/train_date.csv.zip\",\n                             nrows=1000)\nprint(train_cat_part.shape, train_numeric_part.shape, train_date_part.shape)","metadata":{"id":"improving-limitation","outputId":"874047c9-4bfc-47a0-ee85-24ceb5cf49e1","execution":{"iopub.status.busy":"2022-05-20T00:04:13.412309Z","iopub.execute_input":"2022-05-20T00:04:13.413195Z","iopub.status.idle":"2022-05-20T00:04:15.312019Z","shell.execute_reply.started":"2022-05-20T00:04:13.413062Z","shell.execute_reply":"2022-05-20T00:04:15.311194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 대용량 데이터가 메인 메모리에 한번에 적재가 안될때!\n\n1. 사용할 column을 몇개 불러온다. (usecols)\n\n2. 데이터를 쪼개서 임시로 불러와서 계산하고, 다시 불러온다. (chunksize)","metadata":{}},{"cell_type":"code","source":"train_date_part # start_station, end_station","metadata":{"execution":{"iopub.status.busy":"2022-05-20T00:06:08.016115Z","iopub.execute_input":"2022-05-20T00:06:08.016668Z","iopub.status.idle":"2022-05-20T00:06:08.061559Z","shell.execute_reply.started":"2022-05-20T00:06:08.016628Z","shell.execute_reply":"2022-05-20T00:06:08.060951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### train_date 데이터를 요약해봅시다.\n\n- 각 row마다 측정이 처음된 station 번호와, 마지막으로 측정이 된 station 번호를 찾습니다.","metadata":{}},{"cell_type":"code","source":"row = train_date_part.loc[0]\nrow[row.index.str.contains(\"S2_\")] # 0.01 == 6분\n\n# 1. station별 column 찾기 : 1157개 -> 52개\ncounts = train_date_part.count() # column이 index로 들어가 있는 Series를 만들려고.\ndate_cols = counts.reset_index()[\"index\"].str.split(\"_\", expand=True)\ncol_idx = date_cols.drop_duplicates(1).index  # column 1을 기준으로 중복 제거\ndate_cols = train_date_part.columns[col_idx]\ndate_cols","metadata":{"execution":{"iopub.status.busy":"2022-05-20T01:17:20.928355Z","iopub.execute_input":"2022-05-20T01:17:20.928797Z","iopub.status.idle":"2022-05-20T01:17:20.956564Z","shell.execute_reply.started":"2022-05-20T01:17:20.928754Z","shell.execute_reply":"2022-05-20T01:17:20.955758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"row = train_date_part.loc[999]\nrow[row.index.str.contains(\"S51_\")] # 0.01 == 6분","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:00:53.776808Z","iopub.execute_input":"2022-05-20T02:00:53.777088Z","iopub.status.idle":"2022-05-20T02:00:53.787915Z","shell.execute_reply.started":"2022-05-20T02:00:53.777059Z","shell.execute_reply":"2022-05-20T02:00:53.786798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2. start_station, end_station 찾기\n\ntrain_date = pd.read_csv(\"../input/bosch-production-line-performance/train_date.csv.zip\",\n                        usecols=date_cols)\n\n## 전체 데이터에서 column 순서대로 scan하면서 처음으로 NaN이 아닌 column의 station이 start_station,\n## 마지막까지 scan하면서 마지막으로 NaN이 아니었던 column의 station이 end_station.\ntrain_date[\"start_station\"] = -1\ntrain_date[\"end_station\"] = -1\n\nfor col in train_date.drop(columns=[\"Id\", \"start_station\", \"end_station\"]).columns:\n    notnulls = ~train_date[col].isnull() # null이 아닌 row들의 위치\n    station_name = int(col.split(\"_\")[1][1:])\n    \n    train_date.loc[(notnulls) & (train_date.start_station == -1), \"start_station\"] = station_name\n    train_date.loc[(notnulls), \"end_station\"] = station_name\n    \ntrain_date","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:10:04.697124Z","iopub.execute_input":"2022-05-20T02:10:04.697540Z","iopub.status.idle":"2022-05-20T02:10:47.647688Z","shell.execute_reply.started":"2022-05-20T02:10:04.697498Z","shell.execute_reply":"2022-05-20T02:10:47.646896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_date = train_date[[\"Id\", \"start_station\", \"end_station\"]]\ntrain_date","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:26:47.705713Z","iopub.execute_input":"2022-05-20T02:26:47.706032Z","iopub.status.idle":"2022-05-20T02:26:47.725382Z","shell.execute_reply.started":"2022-05-20T02:26:47.706000Z","shell.execute_reply":"2022-05-20T02:26:47.724734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 차원의 저주(Curse of Dimensionality)\n\n- 머신러닝 모델이 데이터가 존재하는 공간이 너무 고차원일 때, <=> feature(column)가 너무 많을 때\n\n- 머신러닝 모델이 예측력을 잃어버리는 문제. (성능이 저하)\n\n\n- 해결책\n\n> 차원을 줄입니다! (= column수를 줄인다)\n\n> 1. feature selection : 분석에 사용할 column을 직접 고른다.\n>> 결측치가 많은 column을 버리거나, 도메인 지식으로 필요한 변수를 고르거나, ....\n\n> 2. feature extraction : dimensionality reduction method(e.g. PCA)를 사용한다.","metadata":{}},{"cell_type":"code","source":"missing_ratio = pd.Series(index=train_numeric_part.columns,\n                         data=np.zeros(len(train_numeric_part.columns))) # 970\ncolumn_means = pd.Series(index=train_numeric_part.columns,\n                         data=np.zeros(len(train_numeric_part.columns)))\nlength = 0\n\nfor chunk in pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                         chunksize=100000):\n    #display(chunk.isnull().sum())\n    temp = chunk.isnull().sum()  # chunk별 결측치 합\n    temp2 = chunk.sum()\n    \n    length = length + len(chunk)\n    missing_ratio = missing_ratio + temp  # 결측치 누적합\n    column_means = column_means + temp2  # 실제 값의 누적합\n    #display(missing_ratio + chunk.isnull().sum())\n    #break\n\ndisplay(missing_ratio) # 결측치 개수\ndisplay(column_means / length)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:33:04.500824Z","iopub.execute_input":"2022-05-20T02:33:04.501117Z","iopub.status.idle":"2022-05-20T02:34:51.143282Z","shell.execute_reply.started":"2022-05-20T02:33:04.501089Z","shell.execute_reply":"2022-05-20T02:34:51.142426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"length","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.144687Z","iopub.execute_input":"2022-05-20T02:34:51.144950Z","iopub.status.idle":"2022-05-20T02:34:51.150818Z","shell.execute_reply.started":"2022-05-20T02:34:51.144921Z","shell.execute_reply":"2022-05-20T02:34:51.149984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_ratio = missing_ratio / length  # 비율로 변환","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.151950Z","iopub.execute_input":"2022-05-20T02:34:51.152156Z","iopub.status.idle":"2022-05-20T02:34:51.162054Z","shell.execute_reply.started":"2022-05-20T02:34:51.152129Z","shell.execute_reply":"2022-05-20T02:34:51.161271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 결측치가 50%를 넘는 column을 지우겠다.\ntrain_numeric_part.drop(columns=train_numeric_part.columns[missing_ratio > 0.5])","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.163830Z","iopub.execute_input":"2022-05-20T02:34:51.164593Z","iopub.status.idle":"2022-05-20T02:34:51.213153Z","shell.execute_reply.started":"2022-05-20T02:34:51.164550Z","shell.execute_reply":"2022-05-20T02:34:51.212280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 결측 비율이 50%를 넘지 않는 column들만 뽑아오겠다.\nusecols = train_numeric_part.columns[missing_ratio <= 0.5]\nusecols","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.214556Z","iopub.execute_input":"2022-05-20T02:34:51.214870Z","iopub.status.idle":"2022-05-20T02:34:51.222035Z","shell.execute_reply.started":"2022-05-20T02:34:51.214828Z","shell.execute_reply":"2022-05-20T02:34:51.221258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_numeric_part2 = pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n#                                 usecols=[\"Id\", \"Response\"]) # 불러온 column을 지정.\n# train_numeric_part2","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.223492Z","iopub.execute_input":"2022-05-20T02:34:51.223972Z","iopub.status.idle":"2022-05-20T02:34:51.231977Z","shell.execute_reply.started":"2022-05-20T02:34:51.223929Z","shell.execute_reply":"2022-05-20T02:34:51.231232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_numeric_part2.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.233259Z","iopub.execute_input":"2022-05-20T02:34:51.233557Z","iopub.status.idle":"2022-05-20T02:34:51.241555Z","shell.execute_reply.started":"2022-05-20T02:34:51.233515Z","shell.execute_reply":"2022-05-20T02:34:51.240937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # 불량품 비율\n# #train_numeric_part.Response.value_counts() # 99.5 : 0.05\n# sns.countplot(data=train_numeric_part, x=\"Response\")  # target value distribution\n# ## Class imbalance problem, 클래스 불균형 문제!","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.242738Z","iopub.execute_input":"2022-05-20T02:34:51.243599Z","iopub.status.idle":"2022-05-20T02:34:51.251833Z","shell.execute_reply.started":"2022-05-20T02:34:51.243556Z","shell.execute_reply":"2022-05-20T02:34:51.251160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. 대용량 데이터 다루기\n\nTechnique1. 잘라서 불러오기\n\n\nTechnique2. Reducing Memory Technique (dtype 변경하기)","metadata":{}},{"cell_type":"code","source":"# %%timeit\n# X = pca.fit_transform(data)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:50:37.756899Z","iopub.execute_input":"2022-05-19T08:50:37.759651Z","iopub.status.idle":"2022-05-19T08:50:37.777887Z","shell.execute_reply.started":"2022-05-19T08:50:37.759268Z","shell.execute_reply":"2022-05-19T08:50:37.774570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pca = PCA(n_components=0.90) # 전체 데이터의 90% 정보를 보존하는 저차원 공간으로 변환해주세요.\n# X = pca.fit_transform(data)\n# pca_df = pd.DataFrame(columns=[f\"PC{i}\" for i in range(1, X.shape[1]+1)],\n#                      data=X)\n# display(pca_df)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:50:37.783930Z","iopub.execute_input":"2022-05-19T08:50:37.786190Z","iopub.status.idle":"2022-05-19T08:50:37.803117Z","shell.execute_reply.started":"2022-05-19T08:50:37.785321Z","shell.execute_reply":"2022-05-19T08:50:37.800384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.decomposition import PCA\ndata = pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                        usecols=usecols)\ndata = data.fillna(0) # 결측치를 0으로 채움\nids = data.Id\nresponses = data.Response\ndata = data.drop(columns=[\"Id\", \"Response\"])\npca = PCA(n_components=6)\nX = pca.fit_transform(data)\npca_df = pd.DataFrame(columns=[f\"PC{i}\" for i in range(1, 7)],\n                     data=X)\ndisplay(pca_df)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:34:51.253515Z","iopub.execute_input":"2022-05-20T02:34:51.254147Z","iopub.status.idle":"2022-05-20T02:35:57.399444Z","shell.execute_reply.started":"2022-05-20T02:34:51.254102Z","shell.execute_reply":"2022-05-20T02:35:57.398493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## 데이터를 합쳐줍니다.\n# pca_df (numeric 데이터로 만든 데이터) + train_date(date 데이터로 만든 데이터)\npca_df[\"Id\"] = ids\npca_df[\"Response\"] = responses\nX = pd.merge(train_date, pca_df, on=\"Id\")\nX","metadata":{"execution":{"iopub.status.busy":"2022-05-20T02:43:34.122464Z","iopub.execute_input":"2022-05-20T02:43:34.123086Z","iopub.status.idle":"2022-05-20T02:43:34.556397Z","shell.execute_reply.started":"2022-05-20T02:43:34.123034Z","shell.execute_reply":"2022-05-20T02:43:34.555568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Technique 1. PCA (ML) / AutoEncoder (DL)\n# from sklearn.decomposition import PCA\n\n# for chunk in pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n#                         usecols=usecols,\n#                         chunksize=10000):\n#     #display(chunk) # dropna()\n#     chunk = chunk.fillna(0) # 결측치를 0으로 채움\n#     temp = chunk.drop(columns=[\"Id\", \"Response\"])\n#     pca = PCA(n_components=6)\n#     X = pca.fit_transform(temp)\n#     pca_df = pd.DataFrame(columns=[f\"PC{i}\" for i in range(1, 7)],\n#                          data=X)\n#     display(pca_df)\n#     break","metadata":{"id":"isolated-barbados","execution":{"iopub.status.busy":"2022-05-19T08:51:50.460230Z","iopub.execute_input":"2022-05-19T08:51:50.460483Z","iopub.status.idle":"2022-05-19T08:51:50.464233Z","shell.execute_reply.started":"2022-05-19T08:51:50.460450Z","shell.execute_reply":"2022-05-19T08:51:50.463585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### dtype\n\n- np.int8, np.int16, np.int32, np.int64 : 정수 하나를 몇 bit로 표현할 것인가?\n\n> int8 : -2^7 ~ 2^7 - 1  --> -128 ~ 127  --> 00000000 ~ 11111111\n\n> int16 : -2^15 ~ 2^15 - 1 --> -65536 ~ 65535 --> 0000000000000000 ~ 11111111111111111111\n\n...\n\n- np.float32 / np.float64 : 실수 하나를 몇 bit로 표현할 것인가?\n (single precision / double precision)\n\n> 실수 표현은 항상 근사치를 쓰기 때문에, 얼마나 자세하게 표현할지의 차이!\n\n> 필연적으로 발생하는 수치적 차이를 \"numerical error\"라고 부릅니다.","metadata":{}},{"cell_type":"code","source":"# Technique 2. dtype 변경\npca_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:51:50.465242Z","iopub.execute_input":"2022-05-19T08:51:50.466022Z","iopub.status.idle":"2022-05-19T08:51:50.556972Z","shell.execute_reply.started":"2022-05-19T08:51:50.465987Z","shell.execute_reply":"2022-05-19T08:51:50.555997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_df.astype(np.float32).info() # float64 -> float32로 메모리 반토막.","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:51:50.558409Z","iopub.execute_input":"2022-05-19T08:51:50.558693Z","iopub.status.idle":"2022-05-19T08:51:50.670721Z","shell.execute_reply.started":"2022-05-19T08:51:50.558663Z","shell.execute_reply":"2022-05-19T08:51:50.669515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ## Reducing DataFrame memory size by ~65%\n# #####\n# def reduce_mem_usage(props):\n#     start_mem_usg = props.memory_usage().sum() / 1024**2 \n#     print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n#     NAlist = [] # Keeps track of columns that have missing values filled in. \n#     for col in props.columns:\n#         if props[col].dtype != object:  # Exclude strings\n            \n#             # Print current column type\n#             print(\"******************************\")\n#             print(\"Column: \",col)\n#             print(\"dtype before: \",props[col].dtype)\n            \n#             # make variables for Int, max and min\n#             IsInt = False\n#             mx = props[col].max()\n#             mn = props[col].min()\n            \n#             # Integer does not support NA, therefore, NA needs to be filled\n#             if not np.isfinite(props[col]).all(): \n#                 NAlist.append(col)\n#                 props[col].fillna(mn-1,inplace=True)  \n#             # test if column can be converted to an integer\n#             asint = props[col].fillna(0).astype(np.int64)\n#             result = (props[col] - asint)\n#             result = result.sum()\n#             if result > -0.01 and result < 0.01:\n#                 IsInt = True\n\n            \n#             # Make Integer/unsigned Integer datatypes\n#             if IsInt:\n#                 if mn >= 0:\n#                     if mx < 255:\n#                         props[col] = props[col].astype(np.uint8)\n#                     elif mx < 65535:\n#                         props[col] = props[col].astype(np.uint16)\n#                     elif mx < 4294967295:\n#                         props[col] = props[col].astype(np.uint32)\n#                     else:\n#                         props[col] = props[col].astype(np.uint64)\n#             else:\n#                     if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n#                         props[col] = props[col].astype(np.int8)\n#                     elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n#                         props[col] = props[col].astype(np.int16)\n#                     elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n#                         props[col] = props[col].astype(np.int32)\n#                     elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n#                         props[col] = props[col].astype(np.int64)    \n            \n#             # Make float datatypes 32 bit\n#             else:\n#                 props[col] = props[col].astype(np.float32)\n            \n#             # Print new column type\n#             print(\"dtype after: \",props[col].dtype)\n#             print(\"******************************\")\n    \n#     # Print final result\n#     print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n#     mem_usg = props.memory_usage().sum() / 1024**2 \n#     print(\"Memory usage is: \",mem_usg,\" MB\")\n#     print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\n#     return props, NAlist","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:51:50.672805Z","iopub.execute_input":"2022-05-19T08:51:50.673373Z","iopub.status.idle":"2022-05-19T08:51:50.695050Z","shell.execute_reply.started":"2022-05-19T08:51:50.673319Z","shell.execute_reply":"2022-05-19T08:51:50.693523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. 불량 검출 문제 다루기 (Class Imbalance problem)\n\nTechnique 1. Sampling Method (Under / Over)\n> binary classification (이진 분류) : 정상이냐 불량이냐로 예측하는 문제!\n\n> 일반적으로 불량에 해당하는 케이스가 매우매우 적습니다. (minority class)\n\n> 일반적으로 정상에 해당하는 케이스가 아주아주 많습니다. (majority class)\n\n> 머신러닝 분류 문제에서는 클래스의 비율이 비슷할수록 학습이 잘됩니다.\n\n>> minority -> majority (oversampling)\n\n>> majority -> minority (undersampling)\n\n\n\n\nTechnique 2. Change to Outlier Detection problem\n\n- 주류(또는 대다수) : majority\n\n- 주류가 아닌 데이터를 outlier(이상치)라고 판단하는 방법.\n\n- IQR, IsolationForest (ML), DBSCAN (ML)\n\n\n\n\n\n(Advanced) Technique 3. Semi-supervised Learning (DL), MakinaRocks","metadata":{}},{"cell_type":"code","source":"# Technique 1.\n# X : feature vector (독립변수), 학습 데이터(문제)\n# y : target value   (종속변수), 목표 값 (정답)\nX = pca_df\ny = responses\nprint(X.shape, y.shape)\n\n## 1-1. Undersampling - y에서 1의 개수로 0인 데이터를 줄이는 것. (99.5 --> 0.05)\nnormal = X[y == 0] # response가 0인 데이터\nfailure = X[y == 1] # response가 1인 데이터\nprint(normal.shape, failure.shape) ## 1176868 --> 6879\n\nnormal_under = normal.sample(n=failure.shape[0], random_state=0xC0FFEE) # 불량 데이터 개수만큼 샘플링!\nprint(normal_under.shape, failure.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:14.737844Z","iopub.execute_input":"2022-05-19T08:52:14.738368Z","iopub.status.idle":"2022-05-19T08:52:14.896820Z","shell.execute_reply.started":"2022-05-19T08:52:14.738311Z","shell.execute_reply":"2022-05-19T08:52:14.895727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## 1-2. Oversampilng - y가 1인 데이터를 y가 0인 데이터의 개수로 늘리는 것. (0.05 --> 99.5)\nfrom imblearn.over_sampling import SMOTE\n\nsm = SMOTE()\n# X, y 데이터를 가지고 y에서 자동으로 minority class를 찾은 뒤에\n# majority class의 크기와 같게 oversampling 해주는 함수.\nX_oversampled, y_oversampled = sm.fit_resample(X, y)\nprint(X_oversampled.shape, y_oversampled.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:15.424040Z","iopub.execute_input":"2022-05-19T08:52:15.424721Z","iopub.status.idle":"2022-05-19T08:52:16.410795Z","shell.execute_reply.started":"2022-05-19T08:52:15.424662Z","shell.execute_reply":"2022-05-19T08:52:16.409808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"failure.shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:16.412331Z","iopub.execute_input":"2022-05-19T08:52:16.412587Z","iopub.status.idle":"2022-05-19T08:52:16.421450Z","shell.execute_reply.started":"2022-05-19T08:52:16.412555Z","shell.execute_reply":"2022-05-19T08:52:16.420697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#X_oversampled[y_oversampled == 0] # == normal\nX_oversampled[y_oversampled == 1][6879:] # 새로 생긴 데이터.","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:16.504554Z","iopub.execute_input":"2022-05-19T08:52:16.504891Z","iopub.status.idle":"2022-05-19T08:52:16.643077Z","shell.execute_reply.started":"2022-05-19T08:52:16.504856Z","shell.execute_reply":"2022-05-19T08:52:16.642049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## 1-3. Hybrid Approach : Undersampling + Oversampling\n\n### 정상 데이터를 500,000개로 undersampling한 뒤, 불량 데이터를 oversampling을 수행하여\n### 데이터를 1,000,000개가 되도록 만들어주세요!!\n\nX_under = normal.sample(n=500000).reset_index(drop=True)\ny_under = np.zeros(len(X_under))\nX_total = pd.concat([X_under, failure]).reset_index(drop=True)\ny_total = np.hstack([y_under, np.ones(len(failure))]).astype(int)\n\n\n# SMOTE에 정상과 불량 데이터가 합쳐진 전체 데이터를 넣으면 알아서, (기준은 항상 y)\n# majority class 크기에 맞게 oversampling이 됩니다.\nX_hybrid, y_hybrid = sm.fit_resample(X_total, y_total)\nprint(X_hybrid.shape, y_hybrid.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:17.150052Z","iopub.execute_input":"2022-05-19T08:52:17.150349Z","iopub.status.idle":"2022-05-19T08:52:17.606511Z","shell.execute_reply.started":"2022-05-19T08:52:17.150318Z","shell.execute_reply":"2022-05-19T08:52:17.605534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# write csv file\npca_df.to_csv(\"PCA_data.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:53:45.089629Z","iopub.execute_input":"2022-05-19T08:53:45.089968Z","iopub.status.idle":"2022-05-19T08:54:00.496966Z","shell.execute_reply.started":"2022-05-19T08:53:45.089936Z","shell.execute_reply":"2022-05-19T08:54:00.495840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\n# Data dump\nwith open(\"pca_data.pk\", \"wb\") as f:\n    pickle.dump(pca_df, f)\n    \n# Data load\nwith open(\"pca_data.pk\", \"rb\") as f:\n    pca_df = pickle.load(f)","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:34.263720Z","iopub.execute_input":"2022-05-19T08:52:34.264000Z","iopub.status.idle":"2022-05-19T08:52:34.723578Z","shell.execute_reply.started":"2022-05-19T08:52:34.263969Z","shell.execute_reply":"2022-05-19T08:52:34.722594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_df","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:52:34.725378Z","iopub.execute_input":"2022-05-19T08:52:34.725757Z","iopub.status.idle":"2022-05-19T08:52:34.742605Z","shell.execute_reply.started":"2022-05-19T08:52:34.725691Z","shell.execute_reply":"2022-05-19T08:52:34.741752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Technique 2. IQR\n#sns.boxplot(train_numeric_part.L0_S0_F0)\n#sns.boxplot(data=train_numeric_part, x=\"L0_S0_F0\")\n\ndef IQR_outlier(data, column):\n    temp = data[column].dropna() ## 결측치가 없어야함!\n    q25 = np.percentile(temp, 25) # 25%\n    q75 = np.percentile(temp, 75) # 75%\n    IQR = q75 - q25 # Inter-Quartile Range\n    lower_bound = q25 - IQR*1.5\n    upper_bound = q75 + IQR*1.5\n    outlier = temp.loc[(temp < lower_bound) | (temp > upper_bound)]\n    return outlier","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:38:17.565651Z","iopub.execute_input":"2022-05-19T08:38:17.566522Z","iopub.status.idle":"2022-05-19T08:38:17.573125Z","shell.execute_reply.started":"2022-05-19T08:38:17.566478Z","shell.execute_reply":"2022-05-19T08:38:17.572132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(data=train_numeric_part, x=\"L0_S0_F0\")","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:38:17.662922Z","iopub.execute_input":"2022-05-19T08:38:17.663603Z","iopub.status.idle":"2022-05-19T08:38:18.459903Z","shell.execute_reply.started":"2022-05-19T08:38:17.663565Z","shell.execute_reply":"2022-05-19T08:38:18.458645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IQR_outlier(train_numeric_part, \"L0_S0_F0\")","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:38:18.462012Z","iopub.execute_input":"2022-05-19T08:38:18.462876Z","iopub.status.idle":"2022-05-19T08:38:18.471312Z","shell.execute_reply.started":"2022-05-19T08:38:18.462836Z","shell.execute_reply":"2022-05-19T08:38:18.470569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from sklearn.cluster import DBSCAN\n\n# db = DBSCAN(n_jobs=-1)\n# pred = db.fit_predict(X)\n# pred","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:44:16.775051Z","iopub.execute_input":"2022-05-19T08:44:16.775386Z","iopub.status.idle":"2022-05-19T08:45:30.997515Z","shell.execute_reply.started":"2022-05-19T08:44:16.775349Z","shell.execute_reply":"2022-05-19T08:45:30.995122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X[pred == -1] # outlier","metadata":{"execution":{"iopub.status.busy":"2022-05-19T08:45:46.281337Z","iopub.execute_input":"2022-05-19T08:45:46.281665Z","iopub.status.idle":"2022-05-19T08:45:46.300347Z","shell.execute_reply.started":"2022-05-19T08:45:46.281630Z","shell.execute_reply":"2022-05-19T08:45:46.299233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### if, ML","metadata":{}},{"cell_type":"code","source":"# from sklearn.ensemble import RandomForestClassifier\n\n","metadata":{},"execution_count":null,"outputs":[]}]}