{"cells":[{"metadata":{"_uuid":"da257aad9b04cbebc1e67e6465414e7e379fc1e9"},"cell_type":"markdown","source":"# Introduction","execution_count":null},{"metadata":{"_uuid":"a04a68f5edf3bcd4f18524049ac02d185fdf0e76"},"cell_type":"markdown","source":"이 커널은 [base model - v2 user level solution](https://www.kaggle.com/augustmarvel/base-model-v2-user-level-solution)을 한국어로 번역한 커널입니다.\n\n### 이 커널은 세 부분으로 구성됩니다.\n* [**1. Data loading**](#Data loading)\n* [**2. Data preprocessing**](#Data preprocessing)\n* [**3. Model building**](#Model building)\n\n###  이 커널의 핵심:\n*  데이터 전체 chunk: \n* 210일의 학습 기간, 45일의 격차 기간, 2개월의 타겟 기간\n* 훈련 기간에서 데이터를 집계하고, 간격 기간은 무시하고, 목표기간에 대해서 목표를 얻습니다.\n* 유효성 검증셋은 12월-1월로 설정되어있으며, 테스트셋의 대상 기간과 동일한 기간입니다.\n\n### Summary:\n이 컴피티션은 솔루션이 \"all-zeros\"를 이길 수 있는지 말하기 힘들정도로 데이터셋의 불균형이 심합니다. 몇가지 기본적인 EDA를 수행한 후 확실한 결론이 났습니다:\n\n1. 고객이 비용을 지불하면 첫 달에 거래가 발생할 가능성이 높으며, 고객이 처음으로 나타난 후 2개월을 넘지 않아야 합니다.\n2. 최소 거래 수익은 1e+07 이상입ㄴ디ㅏ.\n---\n* 첫 번째 테스트 세트의 경우 테스트 기간 사이에 1.5 개월의 간격이 있습니다. 즉, 태그가 두 그룹으로 나뉘어 있습니다. 첫 번째는 이미 지불 여부를 생각하는 데 45일 이상 소요된 것입니다. 두 번째는 부분 서비스 비용을 지불하고 추가 서비스 비용을 지불하는 사람입니다. 첫 번째 그룹에게는 고객이 지불 할 가능성이 적습니다. 두 번째 고객에게는 고객이 이전에 지불 한 금액과 동일한 금액을 지불 할 가능성이 높습니다. 그러한 조건 하에서, 지불 할 사람들의 수에 대한 나의 예측은 200 정도입니다.\n* 두 번째로, 모델의 예측은 1E + 07보다 작은 숫자들로 가득합니다. 그러나 해당 숫자를 0으로 설정하면 점수가 떨어집니다. 우리의 모델은 RMSE를 최소화하기 위해 현명하게 베팅을 유지하지만 결과는 실제 숫자에서 멀어집니다.\n\n### random thoughts:\n* 높은 페널티를 주는 사용자 정의 목표 기능을 설정하려면 작은 값을 피하는게 좋습니다.\n* 시간 특징은 최우선 순위에 있어야 합니다.\n* 두번째 그룹 사람들에게 클러스터링으로 지정할 수 있다면?* 1년의 서비스 기간이 만료된 후 고객이 돌아올 경우?\n* 데이터셋에는 사용자 웹 사이트의 페이지 뷰와 같은 일부 중요한 기능이 없습니다. 저용량 사용자에게 무료 계정이 이미 모든 요구사항을 충족하는 경우, 사전 서비스 요금을 청구하는 이유는 무엇?","execution_count":null},{"metadata":{"trusted":true,"_uuid":"dddabb983cddc6cc669dfede6c3cb434a6d269cf"},"cell_type":"markdown","source":"* Data are generated from this script : https://www.kaggle.com/qnkhuat/make-data-ready \n* Stacking part is from this script: https://www.kaggle.com/ashishpatel26/updated-bayesian-lgbm-xgb-cat-fe-kfold-cv","execution_count":null},{"metadata":{"_uuid":"c980fc990df8ec129a8143b9358cfc256c04eca9"},"cell_type":"markdown","source":"## Data loading","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom datetime import datetime\n\nimport os\nfrom os.path import join as pjoin\n\ndata_root = '../input/make-data-ready'\nprint(os.listdir(data_root))\n\npd.set_option('display.max_rows',200)\n\nfrom sklearn.preprocessing import LabelEncoder\n\nfrom pprint import pprint\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3cc9e31f9c51f1fd66593d2fd0cb3963e880f12"},"cell_type":"code","source":"def load_data(data='train',n=2):\n    df = pd.DataFrame()\n    for i in range(n) :\n        if data=='train':\n            if i > 8 :\n                break\n            dfpart = pd.read_pickle(pjoin(data_root,f'train_{i}.pkl'))\n        elif data=='test':\n            if i > 2 :\n                break\n            dfpart = pd.read_pickle(pjoin(data_root,f'test_{i}.pkl'))\n        df = pd.concat([df,dfpart])\n        del dfpart\n    return df\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"711b2ff1d23636474cbf8cbd63ed852d4853499d","scrolled":false},"cell_type":"code","source":"df_train = load_data(n=9)\ndf_test = load_data('test',n=4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3609ea5ecb4ea0a721136c3fc804b48877df40fd"},"cell_type":"markdown","source":"## Data preprocessing","execution_count":null},{"metadata":{"trusted":true,"_uuid":"3d05fcea733cf0d106981b1012bc9bec45535d75"},"cell_type":"code","source":"df = pd.concat([df_train, df_test])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f385d9764874cc85395f13e16ceecabb510a828"},"cell_type":"markdown","source":"### Drop some features and items","execution_count":null},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"68df28767f05caa0636a05ed247205f989d86e91"},"cell_type":"code","source":"col_drop = ['Date_Year', 'Date_Month', 'Date_Week','Date_Hour','device_isMobile','device_deviceCategory',\n       'Date_Day', 'Date_Dayofweek', 'Date_Dayofyear', 'Date_Is_month_end',\n       'Date_Is_month_start', 'Date_Is_quarter_end', 'Date_Is_quarter_start',\n       'Date_Is_year_end', 'Date_Is_year_start','totals_visits',\n           'date','visitId','totals_totalTransactionRevenue','geoNetwork_city','geoNetwork_continent',\n            'geoNetwork_metro','geoNetwork_networkDomain',\n'geoNetwork_region','geoNetwork_subContinent','trafficSource_adContent',\n            'trafficSource_adwordsClickInfo.adNetworkType','trafficSource_adwordsClickInfo.gclId',\n'trafficSource_adwordsClickInfo.slot','trafficSource_campaign',\n            'trafficSource_keyword','trafficSource_referralPath','trafficSource_medium',\n            'customDimensions_value','customDimensions_index','trafficSource_source',\n           'totals_bounces','visitNumber','totals_newVisits']\ndf.drop(col_drop, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"3d908877ca829b6379b2812a2ca8fb4f6f95724f"},"cell_type":"code","source":"country_drop=df.groupby('geoNetwork_country')['totals_transactions'].sum()[df.groupby('geoNetwork_country')['totals_transactions'].sum().sort_values()<4].index.tolist()\ndf.loc[df[df.geoNetwork_country.isin(country_drop)].index,'geoNetwork_country'] = 'NaN'\n\ndf.loc[df[~df.device_browser.isin(['Edge', 'Internet Explorer', 'Firefox', 'Safari', 'Chrome'])].index,'device_browser'] = 'NaN'\ndf.loc[df[~df.device_operatingSystem.isin(['Android', 'iOS', 'Linux', 'Chrome OS', 'Windows', 'Macintosh'])].index,'device_operatingSystem'] = 'NaN'\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c893481ff845810da8afa0b9c74f421d5040737"},"cell_type":"markdown","source":"### Label encoding","execution_count":null},{"metadata":{"trusted":true,"_uuid":"d57562be04b714d000cc04b9f228af5415c2fb3a"},"cell_type":"code","source":"col_lb = ['channelGrouping','device_browser','device_operatingSystem', 'geoNetwork_country',\n          'trafficSource_adwordsClickInfo.isVideoAd','trafficSource_isTrueDirect']\nfor col in col_lb:\n    lb = LabelEncoder()\n    df[col]=lb.fit_transform(df[col])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"daa3c2dd554b4892cc8f797b4151f9c673238a5f"},"cell_type":"markdown","source":"### Features to user level\nThere is also a feature called time_diff, which is directly coded in generating part. And this time- relative feature really works well","execution_count":null},{"metadata":{"trusted":true,"_uuid":"08422247faa12dfb05b56eb5c968fc17b61dcfdf"},"cell_type":"code","source":"to_median = ['channelGrouping','device_browser','device_operatingSystem','geoNetwork_country','trafficSource_adwordsClickInfo.isVideoAd','trafficSource_isTrueDirect','trafficSource_adwordsClickInfo.page']\nto_sum =['totals_hits','totals_pageviews','totals_timeOnSite','totals_transactionRevenue', 'totals_transactions']\nto_mean =['totals_hits','totals_pageviews','totals_sessionQualityDim']\nto_std = ['totals_hits','totals_pageviews']\nto_time = 'visitStartTime'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73bfce7889aee0d47dfa4329f5467b886d45e501"},"cell_type":"markdown","source":"### Time period\n* 학습셋은 시험셋과 동일한 목표셋과 45일 간격이 있습니다.\n* 학습셋은 시험셋과 지속 시간이 거의 같습니다.\n* 검증 셋은 12월-1월로 설정되며, 시험셋의 목표 기간과 동일한 기간입니다.","execution_count":null},{"metadata":{"trusted":true,"_uuid":"4f462ee9657626056b3d37ec053f96e3759dda46"},"cell_type":"code","source":"target_period = pd.date_range(start='2016-08-01',end='2018-12-01', freq='2MS')\ntrain_period = target_period.to_series().shift(periods=-210, freq='d',axis= 0)\ntime_to = train_period[train_period.index>np.datetime64('2016-08-01')].astype('int')//10**9\ntime_end = target_period.to_series().shift(periods=-45, freq='d',axis= 0)[4:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a0d781e54279f9ed2035b6f83ea0597e01d4139"},"cell_type":"markdown","source":"### Test data","execution_count":null},{"metadata":{"trusted":true,"_uuid":"b559aa5ad7ba5edd36f787b7d6877e5595960507","scrolled":true},"cell_type":"code","source":"    user_x = df.iloc[df_train.shape[0]:,:]\n    test_x = pd.concat([user_x.groupby('fullVisitorId')[to_median].median().add_suffix('_median'),\n    user_x.groupby('fullVisitorId')['visitStartTime'].agg(['first','last']).add_suffix('_time').sub(time_to.values[-1]).abs(),\n    user_x.groupby('fullVisitorId')['visitStartTime'].apply(lambda x: x.max() -x.min()).rename('time_diff'),\n    user_x.groupby('fullVisitorId')[to_sum].sum().add_suffix('_sum'),\n    user_x.groupby('fullVisitorId')[to_mean].mean().add_suffix('_mean'),\n    user_x.groupby('fullVisitorId')[to_std].std(ddof=0).add_suffix('_std')],axis=1).reset_index()\n    \n    test_ID= test_x.fullVisitorId\n    test_x = test_x.drop(['fullVisitorId'], axis=1,errors='ignore')\n    test_x = test_x.astype('int')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa3d17404b1de64779d970d0649feb95fbc5df82"},"cell_type":"markdown","source":"### Valiation data","execution_count":null},{"metadata":{"trusted":true,"_uuid":"6af113d251ba3348c1c50274c684aed1ba95f022","scrolled":true},"cell_type":"code","source":"    i=4\n    user_x = df[(df.visitStartTime>=(time_to.index.astype('int')//10**9)[i]) & (df.visitStartTime<(time_end.index.astype('int')//10**9)[i])]\n    user_y = df[(df.visitStartTime>=time_to.values[i]) & (df.visitStartTime<time_to.values[i+1])]\n    train_x = pd.concat([user_x.groupby('fullVisitorId')[to_median].median().add_suffix('_median'),\n    user_x.groupby('fullVisitorId')['visitStartTime'].agg(['first','last']).add_suffix('_time').sub(time_to.values[i]).abs(),\n    user_x.groupby('fullVisitorId')['visitStartTime'].apply(lambda x: x.max() -x.min()).rename('time_diff'),\n    user_x.groupby('fullVisitorId')[to_sum].sum().add_suffix('_sum'),\n    user_x.groupby('fullVisitorId')[to_mean].mean().add_suffix('_mean'),\n    user_x.groupby('fullVisitorId')[to_std].std(ddof=0).add_suffix('_std')],axis=1).reset_index()\n    \n    merged=train_x.merge(user_y.groupby('fullVisitorId')['totals_transactionRevenue'].sum().reset_index(),\\\n                              how='left', on='fullVisitorId')\n    val_y = merged.totals_transactionRevenue\n    val_y.fillna(0, inplace=True)\n    val_x = merged.drop(['fullVisitorId','totals_transactionRevenue'], axis=1,errors='ignore')\n    val_x = val_x.astype('int')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65f0aee10ca05964d8db36cdc354f5325b9eba31"},"cell_type":"markdown","source":"## Model building","execution_count":null},{"metadata":{"trusted":true,"_uuid":"6c41bf33202785c56ad6401fdb764b31e50c623b"},"cell_type":"code","source":"import lightgbm as lgb\nfrom xgboost import XGBRegressor\nfrom catboost import CatBoostRegressor\n\nfrom sklearn.metrics import mean_squared_error","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"89019c852b56064c69ffc8d34f50373929fc5254"},"cell_type":"code","source":"params={'learning_rate': 0.01,\n        'objective':'regression',\n        'metric':'rmse',\n        'num_leaves': 31,\n        'verbose': 1,\n        'bagging_fraction': 0.9,\n        'feature_fraction': 0.9,\n        \"random_state\":42,\n        'max_depth': 5,\n        \"bagging_seed\" : 42,\n        \"verbosity\" : -1,\n        \"bagging_frequency\" : 5,\n        'lambda_l2': 0.5,\n        'lambda_l1': 0.5,\n        'min_child_samples': 36\n       }\nxgb_params = {\n        'objective': 'reg:linear',\n        'booster': 'gbtree',\n        'learning_rate': 0.02,\n        'max_depth': 22,\n        'min_child_weight': 57,\n        'gamma' : 1.45,\n        'alpha': 0.0,\n        'lambda': 0.0,\n        'subsample': 0.67,\n        'colsample_bytree': 0.054,\n        'colsample_bylevel': 0.50,\n        'n_jobs': -1,\n        'random_state': 456,\n        'importance_type': 'total_gain'\n    }\n\ncat_param = {\n    'learning_rate' :0.03,\n    'depth' :10,\n    'eval_metric' :'RMSE',\n    'od_type' :'Iter',\n    'metric_period ' : 50,\n    'od_wait' : 20,\n    'seed' : 42\n    \n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"324eb17bc0c5b2a6db2584d99490e7efb10d874b"},"cell_type":"code","source":"oof_reg_preds = np.zeros(val_x.shape[0])\noof_reg_preds1 = np.zeros(val_x.shape[0])\noof_reg_preds2 = np.zeros(val_x.shape[0])\nmerge_pred = np.zeros(val_x.shape[0])\nmerge_preds = np.zeros(val_x.shape[0])\nsub_preds = np.zeros(test_x.shape[0])\nalist = list(range(time_to.shape[0]-1))\nalist.remove(4)\nfolds = alist\nfolds=range(len(alist)-1)\n\nfor i in alist:\n    print(i)\n    user_x = df[(df.visitStartTime>=(time_to.index.astype('int')//10**9)[i]) & (df.visitStartTime<(time_end.index.astype('int')//10**9)[i])]\n    user_y = df[(df.visitStartTime>=time_to.values[i]) & (df.visitStartTime<time_to.values[i+1])]\n    train_x = pd.concat([user_x.groupby('fullVisitorId')[to_median].median().add_suffix('_median'),\n    user_x.groupby('fullVisitorId')['visitStartTime'].agg(['first','last']).add_suffix('_time').sub(time_to.values[i]).abs(),\n    user_x.groupby('fullVisitorId')['visitStartTime'].apply(lambda x: x.max() -x.min()).rename('time_diff'),\n    user_x.groupby('fullVisitorId')[to_sum].sum().add_suffix('_sum'),\n    user_x.groupby('fullVisitorId')[to_mean].mean().add_suffix('_mean'),\n    user_x.groupby('fullVisitorId')[to_std].std(ddof=0).add_suffix('_std')],axis=1).reset_index()\n    \n    merged=train_x.merge(user_y.groupby('fullVisitorId')['totals_transactionRevenue'].sum().reset_index(),\\\n                              how='left', on='fullVisitorId')\n    train_y = merged.totals_transactionRevenue\n    train_y.fillna(0, inplace=True)\n    train_x = merged.drop(['fullVisitorId','totals_transactionRevenue'], axis=1,errors='ignore')\n    train_x = train_x.astype('int')    \n    \n    reg = lgb.LGBMRegressor(**params,n_estimators=1100)\n    xgb = XGBRegressor(**xgb_params, n_estimators=1000)\n    cat = CatBoostRegressor(iterations=1000,learning_rate=0.03,\n                            depth=10,\n                            eval_metric='RMSE',\n                            random_seed = 42,\n                            bagging_temperature = 0.2,\n                            od_type='Iter',\n                            metric_period = 50,\n                            od_wait=20)\n    print(\"-\"* 20 + \"LightGBM Training\" + \"-\"* 20)\n    reg.fit(train_x, np.log1p(train_y),eval_set=[(val_x, np.log1p(val_y))],early_stopping_rounds=50,verbose=100,eval_metric='rmse')\n    print(\"-\"* 20 + \"XGboost Training\" + \"-\"* 20)\n    xgb.fit(train_x, np.log1p(train_y),eval_set=[(val_x, np.log1p(val_y))],early_stopping_rounds=50,eval_metric='rmse',verbose=100)\n    print(\"-\"* 20 + \"Catboost Training\" + \"-\"* 20)\n    cat.fit(train_x, np.log1p(train_y), eval_set=[(val_x, np.log1p(val_y))],early_stopping_rounds=50,use_best_model=True,verbose=100)\n    \n    imp_df = pd.DataFrame()\n    imp_df['feature'] = train_x.columns\n    imp_df['gain_reg'] = np.zeros(train_x.shape[1])\n    imp_df['gain_xgb'] = np.zeros(train_x.shape[1])\n    imp_df['gain_cat'] = np.zeros(train_x.shape[1])\n    imp_df['gain_reg'] += reg.booster_.feature_importance(importance_type='gain')/ len(folds)\n    imp_df['gain_xgb'] += xgb.feature_importances_/ len(folds)\n    imp_df['gain_cat'] += np.array(cat.get_feature_importance())/ len(folds)\n    \n    # LightGBM\n    oof_reg_preds = reg.predict(val_x, num_iteration=reg.best_iteration_)\n    oof_reg_preds[oof_reg_preds < 0] = 0\n    lgb_preds = reg.predict(test_x, num_iteration=reg.best_iteration_)\n    lgb_preds[lgb_preds < 0] = 0\n    \n    \n    # Xgboost\n    oof_reg_preds1 = xgb.predict(val_x)\n    oof_reg_preds1[oof_reg_preds1 < 0] = 0\n    xgb_preds = xgb.predict(test_x)\n    xgb_preds[xgb_preds < 0] = 0\n    \n    # catboost\n    oof_reg_preds2 = cat.predict(val_x)\n    oof_reg_preds2[oof_reg_preds2 < 0] = 0\n    cat_preds = cat.predict(test_x)\n    cat_preds[cat_preds < 0] = 0\n        \n    #merge all prediction\n    merge_pred = oof_reg_preds * 0.4 + oof_reg_preds1 * 0.3 +oof_reg_preds2 * 0.3\n    merge_preds += (oof_reg_preds / len(folds)) * 0.4 + (oof_reg_preds1 / len(folds)) * 0.3 + (oof_reg_preds2 / len(folds)) * 0.3\n    sub_preds += (lgb_preds / len(folds)) * 0.4 + (xgb_preds / len(folds)) * 0.3 + (cat_preds / len(folds)) * 0.3\n    \n    \nprint(\"LGBM  \", mean_squared_error(np.log1p(val_y), oof_reg_preds) ** .5)\nprint(\"XGBoost  \", mean_squared_error(np.log1p(val_y), oof_reg_preds1) ** .5)\nprint(\"CatBoost  \", mean_squared_error(np.log1p(val_y), oof_reg_preds2) ** .5)\nprint(\"merged  \", mean_squared_error(np.log1p(val_y), merge_pred) ** .5)\nprint(\"stack_merged  \", mean_squared_error(np.log1p(val_y), merge_preds) ** .5)\nprint(\"Zeros  \", mean_squared_error(np.log1p(val_y), np.zeros(val_x.shape[0])) ** .5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0569007f5fc9fafa9130e055266a0b4df818fdf"},"cell_type":"markdown","source":"## Display feature importances","execution_count":null},{"metadata":{"trusted":true,"_uuid":"b14a69ce8864555f41e52d796fee3d4616449aa6"},"cell_type":"code","source":"plt.figure(figsize=(8, 12))\nsns.barplot(x='gain_reg', y='feature', data=imp_df.sort_values('gain_reg', ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"483933a11116f62ab7e1a48dbacd175e92008041"},"cell_type":"code","source":"plt.figure(figsize=(8, 12))\nsns.barplot(x='gain_xgb', y='feature', data=imp_df.sort_values('gain_xgb', ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea46d8c9ed90321b1554bada0c455e08b865f4e8"},"cell_type":"code","source":"plt.figure(figsize=(8, 12))\nsns.barplot(x='gain_cat', y='feature', data=imp_df.sort_values('gain_cat', ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff432574f6bffcf1b02919da5658cf805e9d2ef1"},"cell_type":"markdown","source":"## Save result","execution_count":null},{"metadata":{"trusted":true,"_uuid":"67273ccb4272e4f9682c20eff386492fe5ca4a08"},"cell_type":"code","source":"sub_df = pd.DataFrame(test_ID)\nsub_df[\"PredictedLogRevenue\"] = sub_preds\nsub_df.to_csv(\"stacked_result.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}