{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Machine Learning 프로젝트 수행을 위한 코드 구조화\n\n`(분류, 회귀 Task)`","metadata":{"id":"8f3119aa"}},{"cell_type":"markdown","source":"- ML project를 위해서 사용하는 템플릿 코드를 만듭니다.\n\n1. **필요한 라이브러리와 데이터를 불러옵니다.**\n\n\n2. **EDA를 수행합니다.** 이 때 EDA의 목적은 풀어야하는 문제를 위해서 수행됩니다.\n\n\n3. **전처리를 수행합니다.** 이 때 중요한건 **feature engineering**을 어떻게 하느냐 입니다.\n\n\n4. **데이터 분할을 합니다.** 이 때 train data와 test data 간의 분포 차이가 없는지 확인합니다.\n\n\n5. **학습을 진행합니다.** 어떤 모델을 사용하여 학습할지 정합니다. 성능이 잘 나오는 GBM을 추천합니다.\n\n\n6. **hyper-parameter tuning을 수행합니다.** 원하는 목표 성능이 나올 때 까지 진행합니다. 검증 단계를 통해 지속적으로 **overfitting이 되지 않게 주의**하세요.\n\n\n7. **최종 테스트를 진행합니다.** 데이터 분석 대회 포맷에 맞는 submission 파일을 만들어서 성능을 확인해보세요.","metadata":{"id":"9f7610ea"}},{"cell_type":"markdown","source":"## 1. 라이브러리, 데이터 불러오기","metadata":{"id":"bd2f7530"}},{"cell_type":"code","source":"# 데이터분석 4종 세트\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# 모델들, 성능 평가\n# (저는 일반적으로 정형데이터로 머신러닝 분석할 때는 이 2개 모델은 그냥 돌려봅니다. 특히 RF가 테스트하기 좋습니다.)\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.ensemble import RandomForestRegressor\nfrom lightgbm.sklearn import LGBMClassifier\nfrom lightgbm.sklearn import LGBMRegressor\n\n# 상관관계 분석, VIF : 다중공선성 제거\nfrom statsmodels.stats.outliers_influence import variance_inflation_factor\n\n# KFold(CV), partial : optuna를 사용하기 위함\nfrom sklearn.model_selection import KFold\nfrom functools import partial\n\n# hyper-parameter tuning을 위한 라이브러리, optuna\nimport optuna","metadata":{"id":"125fc348","execution":{"iopub.status.busy":"2022-11-10T13:09:12.229203Z","iopub.execute_input":"2022-11-10T13:09:12.229634Z","iopub.status.idle":"2022-11-10T13:09:12.237944Z","shell.execute_reply.started":"2022-11-10T13:09:12.229602Z","shell.execute_reply":"2022-11-10T13:09:12.236517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# flag setting\ndata_reducing = False ## memory reducing technique\nfeature_reducing = 'fi' ## 'pca' or 'fi'\nonline = False ## use online training\nsampling = 'hybrid' ## 'under' or 'over' or 'hybrid'","metadata":{"id":"0e4d49ce","execution":{"iopub.status.busy":"2022-11-10T13:09:12.240334Z","iopub.execute_input":"2022-11-10T13:09:12.240686Z","iopub.status.idle":"2022-11-10T13:09:12.254355Z","shell.execute_reply.started":"2022-11-10T13:09:12.240656Z","shell.execute_reply":"2022-11-10T13:09:12.252700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 데이터를 불러옵니다.\n# train_cat_part = pd.read_csv('../input/bosch-production-line-performance/train_categorical.csv.zip',\n#                             nrows=1000)\ntrain_num_part = pd.read_csv('../input/bosch-production-line-performance/train_numeric.csv.zip',\n                            nrows=1000)\ntrain_date_part = pd.read_csv('../input/bosch-production-line-performance/train_date.csv.zip',\n                             nrows=1000)\n\n#print(train_cat_part.shape, train_num_part.shape, train_date_part.shape)\nprint(train_num_part.shape, train_date_part.shape)","metadata":{"id":"3615c24a","execution":{"iopub.status.busy":"2022-11-10T13:09:12.256049Z","iopub.execute_input":"2022-11-10T13:09:12.256549Z","iopub.status.idle":"2022-11-10T13:09:12.515736Z","shell.execute_reply.started":"2022-11-10T13:09:12.256503Z","shell.execute_reply":"2022-11-10T13:09:12.514899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"categorical data는 EDA 결과 분석에 큰 도움이 없어, 사용하지 않겠습니다.","metadata":{}},{"cell_type":"code","source":"# total = train_cat_part.count() * 0\n# N = 0\n\n# for chunk in pd.read_csv('../input/bosch-production-line-performance/train_categorical.csv.zip',\n#                         chunksize=100000):  # chunksize가 클수록, 속도는 빨라지고 메모리 사용량이 늘어납니다.\n    \n#     total = total + chunk.isnull().sum()  # 전체 데이터의 column별 결측치 개수\n#     N = N + len(chunk)  # 전체 데이터의 row 수\n    \n# display(total)\n# print(N)\n# usecols = total.index[(total / N) < 0.5] # train_categorical.csv에 있는 결측치가 50% 미만인 column들.\n# usecols","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:12.517249Z","iopub.execute_input":"2022-11-10T13:09:12.517575Z","iopub.status.idle":"2022-11-10T13:09:12.522781Z","shell.execute_reply.started":"2022-11-10T13:09:12.517546Z","shell.execute_reply":"2022-11-10T13:09:12.521530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_cat = pd.read_csv('../input/bosch-production-line-performance/train_categorical.csv.zip',\n#                        usecols=usecols)\n# train_cat","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:12.525583Z","iopub.execute_input":"2022-11-10T13:09:12.525916Z","iopub.status.idle":"2022-11-10T13:09:12.537952Z","shell.execute_reply.started":"2022-11-10T13:09:12.525887Z","shell.execute_reply":"2022-11-10T13:09:12.536726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"train_date.csv에서는 파생 변수를 만들어서 사용합니다.\n\n> 1. 각 station별로 측정 시간이 모두 있거나, 없거나의 케이스만 존재한다.\n>> 데이터 중에서 station별로 하나의 column만 뽑아서 체크해도 괜찮다.\n\n> 2. 각 station별로 시간이 모두 같지는 않다. 중간에 측정시간이 바뀌는 케이스도 있다.\n>> 데이터 별로 min, max값을 계산하면 된다.\n\n- start_station : 해당 product가 처음으로 공정을 거친 station_name\n\n- end_station : 해당 product가 마지막으로 공정을 거친 station_name\n\n- time_diff : 해당 product가 처음 시작한 timestamp와 마지막으로 끝낸 timestamp의 차이 (공정에 걸린 시간)","metadata":{}},{"cell_type":"code","source":"# 1157 cols -> station 개수(S0 ~ S51)\n# 1. column들이 나와있는 series를 하나 만듭니다.\ntrain_date_part.count()\n# 2. 그 series의 index를 뽑아서 가운데 _S0_ 를 뽑습니다.\ntrain_date_part.count().reset_index()['index'].str.split('_', expand=True)[1]\n# 3. 각 station별로 하나씩만 남깁니다. + Id column 제거\ntrain_date_part.count().reset_index()['index'].str.split('_', expand=True)[1].drop_duplicates()[1:]\n# 4. 각 station별 index에 해당하는 column들을 뽑습니다.\ntemp = train_date_part.count().reset_index()['index'].str.split('_', expand=True)[1].drop_duplicates()[1:].index\ndatecols = train_date_part.columns[temp]\ndatecols","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:12.539839Z","iopub.execute_input":"2022-11-10T13:09:12.540344Z","iopub.status.idle":"2022-11-10T13:09:12.602630Z","shell.execute_reply.started":"2022-11-10T13:09:12.540289Z","shell.execute_reply":"2022-11-10T13:09:12.601385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"train_date_part 데이터를 가지고, 원하는 3개의 파생변수가 잘 생성되는지 확인합니다.","metadata":{}},{"cell_type":"code","source":"train_date_part[\"start_station\"] = -1  # 초기화(initialization)\ntrain_date_part[\"end_station\"] = -1\n\nfor col in train_date_part[datecols]:\n    ## TODO : start_station과 end_station을 찾아주세요\n    station_name = col.split(\"_\")[1] # S0\n    station_num = int(col.split(\"_\")[1][1:]) # 0\n    notnulls = (~train_date_part[col].isnull()) # 해당 column에서 NaN 아닌 값들의 위치 (boolean mask)\n    \n    # NaN이라면 해당 station을 지나간 것. 아니라면 해당 station에서 측정이 된 것.\n    # start_station은 해당 column에서 \"처음으로\" 수치값이 등장해야한다.(NaN이 아님) 즉, start_station이 -1이며 notnull이어야 함.\n    # end_station은 해당 column에 수치값이 등장하면 매번 update한다. 즉, notnull이기만 하면 됨.\n    train_date_part.loc[(notnulls & (train_date_part.start_station == -1)), \"start_station\"] = station_num\n    train_date_part.loc[notnulls, \"end_station\"] = station_num\n\nstart_time = train_date_part.drop(columns=[\"Id\", \"start_station\", \"end_station\"]).min(axis=1) # start_time\nend_time = train_date_part.drop(columns=[\"Id\", \"start_station\", \"end_station\"]).max(axis=1) # end_time\ntime_diff = end_time - start_time  # 공정에 걸린 시간.\ntrain_date_part['time_diff'] = time_diff\ntrain_date = train_date_part[[\"Id\", \"start_station\", \"end_station\", \"time_diff\"]]\ntrain_date","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:12.604285Z","iopub.execute_input":"2022-11-10T13:09:12.604774Z","iopub.status.idle":"2022-11-10T13:09:12.728228Z","shell.execute_reply.started":"2022-11-10T13:09:12.604720Z","shell.execute_reply":"2022-11-10T13:09:12.727065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## TO-DO: train_date.csv를 불러와서, start_station, end_station, time_diff 파생 변수를 생성한 뒤에,\n##        1183747 x 4 에 해당하는 train_date DataFrame을 만들어주세요.\n\n## [HINT]\n# 1. chunk (datecols)\n# 2. start, end station 만들었던거 그대로 사용\n# 3. min, max 계산해서 time_diff 구하기\nextract_cols = [\"Id\", \"start_station\", \"end_station\", \"time_diff\"]\ntrain_date = pd.DataFrame(columns=extract_cols)\n\n# file I/O\nfor chunk in pd.read_csv('../input/bosch-production-line-performance/train_date.csv.zip',\n                        usecols=[\"Id\"] + datecols.tolist(), chunksize=300000):\n\n    chunk[\"start_station\"] = -1  # 초기화(initialization)\n    chunk[\"end_station\"] = -1\n\n    for col in chunk[datecols]: # Id, start_station, end_station 제외한것과 같음.\n        ## TODO : start_station과 end_station을 찾아주세요\n        station_num = int(col.split(\"_\")[1][1:])\n        notnulls = (~chunk[col].isnull()) # 해당 column에서 NaN 아닌 값들의 위치 (boolean mask)\n\n        chunk.loc[(notnulls & (chunk.start_station == -1)), \"start_station\"] = station_num\n        chunk.loc[notnulls, \"end_station\"] = station_num\n        \n        \n    start_time = chunk[datecols].min(axis=1)\n    end_time = chunk[datecols].max(axis=1)\n    time_diff = end_time - start_time\n    chunk['time_diff'] = time_diff\n    chunk = chunk[extract_cols]\n    train_date = pd.concat([train_date, chunk])\n    \ntrain_date","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:12.730073Z","iopub.execute_input":"2022-11-10T13:09:12.731311Z","iopub.status.idle":"2022-11-10T13:09:54.982854Z","shell.execute_reply.started":"2022-11-10T13:09:12.731258Z","shell.execute_reply":"2022-11-10T13:09:54.981798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"train_numeric.csv 데이터에서 중요한 feature를 뽑기 위해서, PCA(truncatedSVD) 또는 feature importance를 통해 뽑는 방법을 사용합니다.","metadata":{}},{"cell_type":"markdown","source":"#### [WARNING] 이 데이터는 1000개의 데이터에 대해서만 계산한 값으로, 제대로 하려면 전체 데이터를 불러와서 계산해야 합니다!","metadata":{}},{"cell_type":"code","source":"# train_num.csv에서 결측치가 70% 이하인 데이터.\nnullcols = train_num_part.columns[train_num_part.isnull().mean() < 0.7]\nnullcols","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:54.984350Z","iopub.execute_input":"2022-11-10T13:09:54.984752Z","iopub.status.idle":"2022-11-10T13:09:54.999352Z","shell.execute_reply.started":"2022-11-10T13:09:54.984660Z","shell.execute_reply":"2022-11-10T13:09:54.998120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if feature_reducing == 'pca':\n    from sklearn.decomposition import TruncatedSVD ## for sparse input\n\n    train_num = pd.read_csv('../input/bosch-production-line-performance/train_numeric.csv.zip',\n                           usecols=nullcols)\n\n    X = train_num.drop([\"Id\", \"Response\"], axis=1).fillna(0)  # 1183747 x 186 features\n    svd = TruncatedSVD(n_components=15, n_iter=30, random_state=42) ## more than 300 iterations!\n    X_svd = svd.fit_transform(X)\n    print(X_svd.shape) # 186 features -> 15 features.","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:55.000680Z","iopub.execute_input":"2022-11-10T13:09:55.001247Z","iopub.status.idle":"2022-11-10T13:09:55.008129Z","shell.execute_reply.started":"2022-11-10T13:09:55.001212Z","shell.execute_reply":"2022-11-10T13:09:55.007086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"SVD로 만든 numpy array를 보기 좋게 DataFrame으로 변환합니다.","metadata":{}},{"cell_type":"code","source":"if feature_reducing == 'pca':\n    sv_cols = [f\"SV{i}\" for i in range(1, X_svd.shape[1]+1)]\n    df_svd = pd.DataFrame(data=X_svd, columns=sv_cols)\n    df_svd[\"Id\"] = train_num[\"Id\"] # Id column 추가\n    df_svd[\"Response\"] = train_num[\"Response\"] # target value 추가\n    display(df_svd)\n    \n    # release memory\n    del train_num\n    del X\n    del X_svd\n\n    gc.collect()\n    \n    # train_date에서 추출한 정보와, train_numeric에서 추출한 정보를 합쳐줍니다.\n    X_train = pd.merge(df_svd, train_date, on=\"Id\")\n    display(X_train)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:55.009944Z","iopub.execute_input":"2022-11-10T13:09:55.010359Z","iopub.status.idle":"2022-11-10T13:09:55.025245Z","shell.execute_reply.started":"2022-11-10T13:09:55.010323Z","shell.execute_reply":"2022-11-10T13:09:55.024137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"RandomForestClassifier를 이용해서 feature importance가 높은 변수 추출하기","metadata":{}},{"cell_type":"code","source":"if feature_reducing == 'fi' and False:  ## 현재 버젼에서는 training을 안시키려고 강제로 False로 만듬.\n    # Online learning (= batch training)\n    if online:\n        from tqdm.notebook import tqdm\n\n        clf = RandomForestClassifier(n_estimators=50, max_depth=31, max_features=0.7, n_jobs=-1,\n                                    random_state=42)#, warm_start=True)\n\n        cnt = 0\n        for chunk in pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                                chunksize=300000):  # each iteration, training with 50000 rows\n            cnt += 1\n            X = chunk.drop([\"Id\", \"Response\"], axis=1).fillna(0)  # 10000 x 978 features\n            y = chunk.Response\n            clf.fit(X, y)\n            print(f\"in {cnt} iterations.\")\n     \n    # offline learning(=full-batch training)\n    else:\n        train_num = pd.read_csv('../input/bosch-production-line-performance/train_numeric.csv.zip',\n                               nrows=300000)\n\n        X = train_num.drop([\"Id\", \"Response\"], axis=1).fillna(0)  # 10000 x 978 features\n        y = train_num.Response\n\n        # max_features : sqrt(# of features) ~ 31\n        clf = RandomForestClassifier(n_estimators=50, max_depth=31, max_features=0.7, n_jobs=-1,\n                                    random_state=42)\n        clf.fit(X, y)\n        \n        feature_importances = clf.feature_importances_\n        fi_index = feature_importances.argsort()[::-1][:15] # feature_importance가 높은 순서대로의 index\n        usecols = train_num.columns[fi_index]\n        print(\"Features from RF: \", usecols)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:55.026906Z","iopub.execute_input":"2022-11-10T13:09:55.027739Z","iopub.status.idle":"2022-11-10T13:09:55.038458Z","shell.execute_reply.started":"2022-11-10T13:09:55.027655Z","shell.execute_reply":"2022-11-10T13:09:55.037383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Using known Numeric feature from [this notebook](https://github.com/aakashveera/bosch-production-line-performance/blob/master/Numeric%20Classifier.ipynb)","metadata":{}},{"cell_type":"code","source":"if feature_reducing == 'fi':\n    usecols = ['Id',\n               'L3_S30_F3514', 'L0_S9_F200', 'L3_S29_F3430', 'L0_S11_F314',\n               'L0_S0_F18', 'L3_S35_F3896', 'L0_S12_F350', 'L3_S36_F3918',\n               'L0_S0_F20', 'L3_S30_F3684', 'L1_S24_F1632', 'L0_S2_F48',\n               'L3_S29_F3345', 'L0_S18_F449', 'L0_S21_F497', 'L3_S29_F3433',\n               'L3_S30_F3764', 'L0_S1_F24', 'L3_S30_F3554', 'L0_S11_F322',\n               'L3_S30_F3564', 'L3_S29_F3327', 'L0_S2_F36', 'L0_S9_F180',\n               'L3_S33_F3855', 'L0_S0_F4', 'L0_S21_F477', 'L0_S5_F114',\n               'L0_S6_F122', 'L1_S24_F1122', 'L0_S9_F165', 'L0_S18_F439',\n               'L1_S24_F1490', 'L0_S6_F132', 'L3_S29_F3379', 'L3_S29_F3336',\n               'L0_S3_F80', 'L3_S30_F3749', 'L1_S24_F1763', 'L0_S10_F219',\n               'Response'][:15] + [\"Response\"]\n\n    train_num = pd.read_csv(\"../input/bosch-production-line-performance/train_numeric.csv.zip\",\n                           usecols=usecols)\n    display(train_num) # 1183747 x 42","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:09:55.039863Z","iopub.execute_input":"2022-11-10T13:09:55.040321Z","iopub.status.idle":"2022-11-10T13:10:27.836372Z","shell.execute_reply.started":"2022-11-10T13:09:55.040276Z","shell.execute_reply":"2022-11-10T13:10:27.834867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if feature_reducing == 'fi':\n    # train_date에서 추출한 정보와, train_numeric에서 추출한 정보를 합쳐줍니다.\n    X_train = pd.merge(train_num.fillna(0), train_date, on=\"Id\")\n    display(X_train)\n    \n    del train_num\n    del train_date\n    import gc\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:10:27.842606Z","iopub.execute_input":"2022-11-10T13:10:27.843641Z","iopub.status.idle":"2022-11-10T13:10:29.102846Z","shell.execute_reply.started":"2022-11-10T13:10:27.843592Z","shell.execute_reply":"2022-11-10T13:10:29.101624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. EDA","metadata":{"id":"c9c9acb8"}},{"cell_type":"markdown","source":"- 데이터에서 찾아야 하는 기초적인 내용들을 확인합니다.\n\n\n- class imbalance, target distribution, outlier, correlation을 확인합니다.","metadata":{"id":"6fdf620b"}},{"cell_type":"code","source":"## On your Own\ndisplay(X_train.Response.value_counts()) # 0.995 : 0.005 = 995 : 5 = 199 : 1\nsns.countplot(data=X_train, x=\"Response\")","metadata":{"id":"adb06474","execution":{"iopub.status.busy":"2022-11-10T13:10:29.104389Z","iopub.execute_input":"2022-11-10T13:10:29.104875Z","iopub.status.idle":"2022-11-10T13:10:29.331649Z","shell.execute_reply.started":"2022-11-10T13:10:29.104830Z","shell.execute_reply":"2022-11-10T13:10:29.330453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"이런 식으로 여러가지 그래프를 그려가며, 데이터에 대한 인사이트를 얻습니다!","metadata":{"id":"f2c602bd"}},{"cell_type":"markdown","source":"### 3. 전처리\n\nClass Imbalance가 존재하는 경우에는 data sampling 기법을 이용하여 문제를 해결합니다.\n- majority class : 다수가 속해있는 class\n\n- minority class : 소수가 속해있는 class\n\n\n\n**1) Under sampling : majority class의 수를 minority class로 맞춰주기 위해서 random sampling을 수행하는 방법.**\n\ne.g. 1176868 : 6879 -> 6879 : 6879\n\n\n**2) Over sampling : minority class에서 데이터를 생성하여 majority class의 수를 맞춰주는 방법.**\n\ne.g. 1176868 : 6879 -> 1176868 : 1176868\n\n\n**3) Hybrid approach : under sampling & over sampling을 써서 적당한 수로 맞춰주는 방법.**\n\ne.g. 1176868 : 6879 -> 20000 : 20000","metadata":{"id":"9dbb8802"}},{"cell_type":"code","source":"# 모든 데이터가 NaN이라서, time_diff가 NaN이고 train_date에 아무 정보가 없는 데이터들을 제거합니다.\nX_train = X_train.dropna() # 1183767 - 582\nX_train","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:10:29.333184Z","iopub.execute_input":"2022-11-10T13:10:29.334134Z","iopub.status.idle":"2022-11-10T13:10:29.845329Z","shell.execute_reply.started":"2022-11-10T13:10:29.334086Z","shell.execute_reply":"2022-11-10T13:10:29.844205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 1) Undersampling\nif sampling == 'under':\n    N = 20000\n    normal = X_train[X_train.Response == 0]\n    abnormal = X_train[X_train.Response == 1]\n    # do undersampling\n    normal = normal.sample(n=N, random_state=42)\n    print(normal.shape, abnormal.shape)\n    \n    temp = pd.concat([normal, abnormal])\n    X_s = temp.drop(columns=[\"Id\", \"Response\"])\n    y_s = temp.Response","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:10:29.847110Z","iopub.execute_input":"2022-11-10T13:10:29.848384Z","iopub.status.idle":"2022-11-10T13:10:29.856022Z","shell.execute_reply.started":"2022-11-10T13:10:29.848336Z","shell.execute_reply":"2022-11-10T13:10:29.854764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if sampling == 'over':\n    # 2) Oversampling - SMOTE (only numeric features)\n    from imblearn.over_sampling import SMOTE\n\n    X = X_train.drop(columns=[\"Id\", \"Response\"])\n    y = X_train.Response\n\n    X_s, y_s = SMOTE(random_state=42).fit_resample(X, y)\n    print(X_s.shape, y_s.shape)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:10:29.858345Z","iopub.execute_input":"2022-11-10T13:10:29.858918Z","iopub.status.idle":"2022-11-10T13:10:29.869554Z","shell.execute_reply.started":"2022-11-10T13:10:29.858869Z","shell.execute_reply":"2022-11-10T13:10:29.868317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if sampling == 'hybrid':\n    # 3) Hybrid approach\n    N_sample = 20000\n    normal = X_train[X_train.Response == 0].sample(n=N_sample, random_state=42) # 10000개 샘플\n    abnormal = X_train[X_train.Response == 1]\n    temp = pd.concat([normal, abnormal]) # 10000 + 6877\n\n    from imblearn.over_sampling import SMOTE\n\n    X = temp.drop(columns=[\"Id\", \"Response\"])\n    y = temp.Response\n\n    X_s, y_s = SMOTE(random_state=42).fit_resample(X, y)\n    print(X_s.shape, y_s.shape) # (20000, 43), (20000, )","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:10:29.871907Z","iopub.execute_input":"2022-11-10T13:10:29.874507Z","iopub.status.idle":"2022-11-10T13:10:31.541544Z","shell.execute_reply.started":"2022-11-10T13:10:29.874464Z","shell.execute_reply":"2022-11-10T13:10:31.540164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# after sampling, convert float to int.\nX_s['start_station'] = X_s.start_station.astype(np.int8) # 64 -> 8 (reduce x8)\nX_s['end_station'] = X_s.end_station.astype(np.int8)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:10:31.543464Z","iopub.execute_input":"2022-11-10T13:10:31.544202Z","iopub.status.idle":"2022-11-10T13:10:31.557636Z","shell.execute_reply.started":"2022-11-10T13:10:31.544115Z","shell.execute_reply":"2022-11-10T13:10:31.556379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4. 학습 데이터 분할","metadata":{"id":"f497a2d8"}},{"cell_type":"code","source":"# 첫번째 테스트용으로 사용하고, 실제 학습시에는 K-Fold CV를 사용합니다.\n# train : test = 8 : 2\nfrom sklearn.model_selection import train_test_split\n\nX = X_s\ny = y_s\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)\nprint(X_train.shape, X_val.shape, y_train.shape, y_val.shape)","metadata":{"id":"47306aaf","execution":{"iopub.status.busy":"2022-11-10T13:10:31.559139Z","iopub.execute_input":"2022-11-10T13:10:31.559539Z","iopub.status.idle":"2022-11-10T13:10:31.594332Z","shell.execute_reply.started":"2022-11-10T13:10:31.559504Z","shell.execute_reply":"2022-11-10T13:10:31.593021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5. 학습 및 평가","metadata":{"id":"58056e51"}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nmodel = RandomForestClassifier(n_estimators=20, max_depth=7, random_state=42)\nmodel.fit(X_train, y_train)","metadata":{"id":"39fd2515","execution":{"iopub.status.busy":"2022-11-10T13:10:31.595742Z","iopub.execute_input":"2022-11-10T13:10:31.596615Z","iopub.status.idle":"2022-11-10T13:10:32.347608Z","shell.execute_reply.started":"2022-11-10T13:10:31.596576Z","shell.execute_reply":"2022-11-10T13:10:32.346745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# metric은 그때마다 맞게 바꿔줘야 합니다.\nfrom sklearn.metrics import matthews_corrcoef # MCC\nevaluation_metric = matthews_corrcoef","metadata":{"id":"6c8b0259","execution":{"iopub.status.busy":"2022-11-10T13:10:32.348811Z","iopub.execute_input":"2022-11-10T13:10:32.349863Z","iopub.status.idle":"2022-11-10T13:10:32.354564Z","shell.execute_reply.started":"2022-11-10T13:10:32.349825Z","shell.execute_reply":"2022-11-10T13:10:32.353231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Prediction\")\npred_train = model.predict(X_train)\npred_test = model.predict(X_val)\n\ntrain_score = evaluation_metric(y_train, pred_train)\ntest_score = evaluation_metric(y_val, pred_test)\n\nprint(\"Train Score : %.4f\" % train_score)\nprint(\"Test Score : %.4f\" % test_score)","metadata":{"id":"a6b39be5","execution":{"iopub.status.busy":"2022-11-10T13:10:32.355895Z","iopub.execute_input":"2022-11-10T13:10:32.356239Z","iopub.status.idle":"2022-11-10T13:10:32.473565Z","shell.execute_reply.started":"2022-11-10T13:10:32.356208Z","shell.execute_reply":"2022-11-10T13:10:32.472137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6. Hyper-parameter Tuning","metadata":{"id":"bc755b17"}},{"cell_type":"markdown","source":"> GridSearchCV","metadata":{"id":"60070d0e"}},{"cell_type":"markdown","source":"** LightGBM의 hyperparameter **\n\n[Official Documentation] https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html \n\n[Blog 1] https://smecsm.tistory.com/133\n\n[Blog 2] https://towardsdatascience.com/kagglers-guide-to-lightgbm-hyperparameter-tuning-with-optuna-in-2021-ed048d9838b5\n\n[Blog 3] https://nurilee.com/2020/04/03/lightgbm-definition-parameter-tuning/","metadata":{"id":"2bf886a9"}},{"cell_type":"markdown","source":"> optuna를 사용해봅시다 !","metadata":{"id":"97e5302d"}},{"cell_type":"code","source":"def optimizer(trial, X, y, K):\n    # 조절할 hyper-parameter 조합을 적어줍니다.\n    n_estimators = trial.suggest_int('n_estimators', 200, 500)\n    max_depth = trial.suggest_int('max_depth', 5, 8)\n    max_features = trial.suggest_float('max_features', 0.7, 0.9)\n    \n    \n    # 원하는 모델을 지정합니다, optuna는 시간이 오래걸리기 때문에 저는 보통 RF로 일단 테스트를 해본 뒤에 LGBM을 사용합니다.\n    model = RandomForestClassifier(n_estimators=n_estimators,\n                                 max_depth=max_depth,\n                                 max_features=max_features)\n    \n    \n    # K-Fold Cross validation을 구현합니다.\n    from sklearn.model_selection import StratifiedKFold\n    folds = StratifiedKFold(n_splits=K, shuffle=True, random_state=42)\n    scores = []\n    \n    for train_idx, val_idx in folds.split(X, y):\n        X_train = X.iloc[train_idx, :]\n        y_train = y.iloc[train_idx]\n        \n        X_val = X.iloc[val_idx, :]\n        y_val = y.iloc[val_idx]\n        \n        model.fit(X_train, y_train)\n        preds = model.predict(X_val)\n        score = evaluation_metric(y_val, preds)\n        scores.append(score)\n    \n    del model\n    gc.collect()\n    # K-Fold의 평균 loss값을 돌려줍니다.\n    return np.mean(scores)","metadata":{"id":"34ce4986","execution":{"iopub.status.busy":"2022-11-10T13:10:32.475133Z","iopub.execute_input":"2022-11-10T13:10:32.475479Z","iopub.status.idle":"2022-11-10T13:10:32.485839Z","shell.execute_reply.started":"2022-11-10T13:10:32.475450Z","shell.execute_reply":"2022-11-10T13:10:32.484989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"K = 5 # Kfold 수\nopt_func = partial(optimizer, X=X, y=y, K=K)\n\nstudy = optuna.create_study(direction=\"maximize\") # 최소/최대 어느 방향의 최적값을 구할 건지.\nstudy.optimize(opt_func, n_trials=30)","metadata":{"id":"7150b210","execution":{"iopub.status.busy":"2022-11-10T13:10:32.487268Z","iopub.execute_input":"2022-11-10T13:10:32.487613Z","iopub.status.idle":"2022-11-10T14:42:38.118282Z","shell.execute_reply.started":"2022-11-10T13:10:32.487582Z","shell.execute_reply":"2022-11-10T14:42:38.117135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# optuna가 시도했던 모든 실험 관련 데이터\nstudy.trials_dataframe()","metadata":{"id":"72d0a118","execution":{"iopub.status.busy":"2022-11-10T14:42:38.120350Z","iopub.execute_input":"2022-11-10T14:42:38.120797Z","iopub.status.idle":"2022-11-10T14:42:38.150761Z","shell.execute_reply.started":"2022-11-10T14:42:38.120753Z","shell.execute_reply":"2022-11-10T14:42:38.149301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Best Score: %.4f\" % study.best_value) # best score 출력\nprint(\"Best params: \", study.best_trial.params) # best score일 때의 하이퍼파라미터들","metadata":{"id":"a805da05","execution":{"iopub.status.busy":"2022-11-10T14:42:38.152408Z","iopub.execute_input":"2022-11-10T14:42:38.153484Z","iopub.status.idle":"2022-11-10T14:42:38.164492Z","shell.execute_reply.started":"2022-11-10T14:42:38.153436Z","shell.execute_reply":"2022-11-10T14:42:38.163330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 실험 기록 시각화\noptuna.visualization.plot_optimization_history(study)","metadata":{"id":"051ae1eb","execution":{"iopub.status.busy":"2022-11-10T14:42:38.166244Z","iopub.execute_input":"2022-11-10T14:42:38.166842Z","iopub.status.idle":"2022-11-10T14:42:38.325367Z","shell.execute_reply.started":"2022-11-10T14:42:38.166795Z","shell.execute_reply":"2022-11-10T14:42:38.324115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# hyper-parameter들의 중요도\noptuna.visualization.plot_param_importances(study)","metadata":{"id":"efbf8f65","execution":{"iopub.status.busy":"2022-11-10T14:42:38.327201Z","iopub.execute_input":"2022-11-10T14:42:38.327684Z","iopub.status.idle":"2022-11-10T14:42:38.899917Z","shell.execute_reply.started":"2022-11-10T14:42:38.327639Z","shell.execute_reply":"2022-11-10T14:42:38.898958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7. 테스트 및 제출 파일 생성","metadata":{"id":"24b360ec"}},{"cell_type":"code","source":"# release memory\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T14:42:38.901986Z","iopub.execute_input":"2022-11-10T14:42:38.902361Z","iopub.status.idle":"2022-11-10T14:42:39.061634Z","shell.execute_reply.started":"2022-11-10T14:42:38.902328Z","shell.execute_reply":"2022-11-10T14:42:39.060266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_num_part = pd.read_csv('../input/bosch-production-line-performance/test_numeric.csv.zip',\n                           nrows=1000)\ntest_date_part = pd.read_csv('../input/bosch-production-line-performance/test_date.csv.zip',\n                            nrows=1000)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T14:42:39.063540Z","iopub.execute_input":"2022-11-10T14:42:39.064081Z","iopub.status.idle":"2022-11-10T14:42:39.412014Z","shell.execute_reply.started":"2022-11-10T14:42:39.063981Z","shell.execute_reply":"2022-11-10T14:42:39.410961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 전처리 과정은 train data와 동일하게 진행합니다.\n\n1. test_numeric : 위에서 사용한 usecols에 해당하는 column만 read_csv를 통해서 가져옵니다.\n\n2. test_date : start_station, end_station, time_diff의 파생변수를 생성한 방법과 동일한 처리를 해줍니다.\n\n3. X_test : 위의 2개의 DataFrame을 합칩니다.","metadata":{}},{"cell_type":"code","source":"## TO-DO: train_date.csv를 불러와서, start_station, end_station, time_diff 파생 변수를 생성한 뒤에,\n##        1183747 x 4 에 해당하는 train_date DataFrame을 만들어주세요.\n\n## [HINT]\n# 1. chunk (usecols)\n# 2. start, end station 만들었던거 그대로 사용\n# 3. min, max 계산해서 time_diff 구하기\nextract_cols = [\"Id\", \"start_station\", \"end_station\", \"time_diff\"]\ntest_date = pd.DataFrame(columns=extract_cols)\n\nfor chunk in pd.read_csv('../input/bosch-production-line-performance/test_date.csv.zip',\n                        usecols=[\"Id\"] + datecols.tolist(), chunksize=100000):\n\n    chunk[\"start_station\"] = -1  # 초기화(initialization)\n    chunk[\"end_station\"] = -1\n\n    for col in chunk[datecols]: # Id, start_station, end_station 제외한것과 같음.\n        ## TODO : start_station과 end_station을 찾아주세요\n        station_num = int(col.split(\"_\")[1][1:])\n        notnulls = (~chunk[col].isnull()) # 해당 column에서 NaN 아닌 값들의 위치 (boolean mask)\n\n        chunk.loc[(notnulls & (chunk.start_station == -1)), \"start_station\"] = station_num\n        chunk.loc[notnulls, \"end_station\"] = station_num\n        \n        \n    start_time = chunk[datecols].min(axis=1)\n    end_time = chunk[datecols].max(axis=1)\n    time_diff = end_time - start_time\n    chunk['time_diff'] = time_diff\n    chunk = chunk[extract_cols]\n    test_date = pd.concat([test_date, chunk])\n    \ntest_date","metadata":{"execution":{"iopub.status.busy":"2022-11-10T14:42:39.413911Z","iopub.execute_input":"2022-11-10T14:42:39.414427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if feature_reducing == 'fi':\n    usecols.remove(\"Response\") # Response를 제거.\n    test_num = pd.read_csv('../input/bosch-production-line-performance/test_numeric.csv.zip',\n                          usecols=usecols)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if feature_reducing == 'fi':\n    X_test = pd.merge(test_num, test_date, on=\"Id\").drop(\"Id\", axis=1).fillna(0)\n    display(X_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if feature_reducing == 'pca':\n    nullcols.tolist().remove(\"Response\")\n    # train_num.csv에서 결측치가 70% 이하인 column을 그대로 가져옵니다.\n    test_num = pd.read_csv('../input/bosch-production-line-performance/test_numeric.csv.zip',\n                      usecols=nullcols)\n    \n    X = test_num.drop([\"Id\"], axis=1)\n    X_svd = svd.transform(X)\n    print(X_svd.shape) # 185 features -> 15 features.\n    \n    sv_cols = [f\"SV{i}\" for i in range(1, X_svd.shape[1]+1)]\n    df_svd = pd.DataFrame(data=X_svd, columns=sv_cols)\n    df_svd[\"Id\"] = train_num[\"Id\"] # Id column 추가\n    display(df_svd)\n    \n    # release memory\n    del test_num\n    del X\n    del X_svd\n\n    gc.collect()\n    \n    # train_date에서 추출한 정보와, train_numeric에서 추출한 정보를 합쳐줍니다.\n    X_test = pd.merge(df_svd, train_date, on=\"Id\")\n    display(X_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_params = study.best_trial.params\n\nmodel = RandomForestClassifier(**best_params) # 찾은 best hyper-parameter.\n\nmodel.fit(X, y) # 전체 데이터에 대해서 학습!\npreds = model.predict(X_test)\npreds","metadata":{"id":"a0daf54e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"evaluation_metric(y, model.predict(X))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n\nConfusionMatrixDisplay.from_estimator(model, X, y, values_format='d')\n\n#confusion_matrix(y, model.predict(X))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv(\"../input/bosch-production-line-performance/sample_submission.csv.zip\")\nsubmission['Response'] = preds","metadata":{"id":"8ff2070c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index=False)","metadata":{"id":"55a2c13f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}