{"cells":[{"metadata":{"_uuid":"c97f1497b42d357c1929916f8b2f965d766021a3"},"cell_type":"markdown","source":"# Background\n- As you know, there are features which have high cardinality in this competition.\n- I've studied and read some discussions, blogs and articles about high cardinality.\n- In this kernel, I'll experiment to see which encoding works better.\n- Label encoding, Frequency encoding and Mean encoding will be tested.\n- Because I'm student, I welcome your feedback on anything of this contents.!\n- Ok, Let's see!.\n- To compare, I forked great kernel https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue.\n- I set the same rrandom number and same parameters for each cases.\n- I want to recommend you, this kernel. https://www.kaggle.com/vprokopev/mean-likelihood-encodings-a-comprehensive-study. \n- This kernel experiments and explains these encodings. Very useful!"},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:28:54.366049Z","start_time":"2018-09-25T09:28:53.982745Z"},"trusted":true,"_uuid":"2a14b2ab334a1422479f66606fd38bac624be368"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport lightgbm as lgb\nfrom tqdm import tqdm\nimport warnings\nwarnings.filterwarnings('ignore')\nfrom sklearn import model_selection, preprocessing, metrics\nimport datetime\\\n\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:08.122157Z","start_time":"2018-09-25T09:28:54.367550Z"},"trusted":true,"_uuid":"b9edcb9cfe5d7c4970dd8e61b90446840f03a803"},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': str}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}_{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6083b8076522ec4adec2fcca4bde8af5f6bd8b81"},"cell_type":"code","source":"%%time\ndf_train = load_df()\ndf_test = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:08.129814Z","start_time":"2018-09-25T09:29:08.123806Z"},"trusted":true,"_uuid":"6511c695b027e48bbd8590751e903b57a7d9a497"},"cell_type":"code","source":"df_train[\"totals_transactionRevenue\"] = df_train[\"totals_transactionRevenue\"].astype('float')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:10.074241Z","start_time":"2018-09-25T09:29:08.131328Z"},"trusted":true,"_uuid":"38aca860f4c0eaa79ee4229937edab70590b2825"},"cell_type":"code","source":"const_cols = [c for c in df_train.columns if df_train[c].nunique(dropna=False)==1 ]\nconst_cols","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:10.431481Z","start_time":"2018-09-25T09:29:10.075821Z"},"trusted":true,"_uuid":"a6dae4ab1a5ba3615d7fbadd9879446a1044aa45"},"cell_type":"code","source":"cols_to_drop = const_cols + ['sessionId']\n\ndf_train = df_train.drop(cols_to_drop + [\"trafficSource_campaignCode\"], axis=1)\ndf_test = df_test.drop(cols_to_drop, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:10.436270Z","start_time":"2018-09-25T09:29:10.433221Z"},"trusted":true,"_uuid":"f0ad3c32bc4c7f6581ac75b672742e71e66eec13"},"cell_type":"code","source":"print(df_train.shape, df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:10.547422Z","start_time":"2018-09-25T09:29:10.437810Z"},"trusted":true,"_uuid":"d99414c34311ddd8accbde02eb4d9550f505651e"},"cell_type":"code","source":"# Impute 0 for missing target values\ndf_train[\"totals_transactionRevenue\"].fillna(0, inplace=True)\ntrain_y = df_train[\"totals_transactionRevenue\"].values\ntrain_id = df_train[\"fullVisitorId\"].values\ntest_id = df_test[\"fullVisitorId\"].values\n\n\n# label encode the categorical variables and convert the numerical variables to float\ncat_cols = [\"channelGrouping\", \"device_browser\", \n            \"device_deviceCategory\", \"device_operatingSystem\", \n            \"geoNetwork_city\", \"geoNetwork_continent\", \n            \"geoNetwork_country\", \"geoNetwork_metro\",\n            \"geoNetwork_networkDomain\", \"geoNetwork_region\", \n            \"geoNetwork_subContinent\", \"trafficSource_adContent\", \n            \"trafficSource_adwordsClickInfo.adNetworkType\", \n            \"trafficSource_adwordsClickInfo.gclId\", \n            \"trafficSource_adwordsClickInfo.page\", \n            \"trafficSource_adwordsClickInfo.slot\", \"trafficSource_campaign\",\n            \"trafficSource_keyword\", \"trafficSource_medium\", \n            \"trafficSource_referralPath\", \"trafficSource_source\",\n            'trafficSource_adwordsClickInfo.isVideoAd',\n            'trafficSource_isTrueDirect', 'device_isMobile']","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:11.846757Z","start_time":"2018-09-25T09:29:10.552253Z"},"trusted":true,"_uuid":"be71c2f7629ae170e57577e8f389e3f2d4a36600"},"cell_type":"code","source":"df_train['date'] = df_train['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ndf_train['date'] = pd.to_datetime(df_train['date'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"051cc70e531d6a7b59d1d66cc29a65522cefb0c4"},"cell_type":"code","source":"num_cols = [\"totals_hits\", \"totals_pageviews\", \n            \"visitNumber\", \"visitStartTime\", \n            'totals_bounces',  'totals_newVisits']    \n\nfor col in num_cols:\n    df_train[col] = df_train[col].astype(float)\n    df_test[col] = df_test[col].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:12.058823Z","start_time":"2018-09-25T09:29:11.848305Z"},"trusted":true,"_uuid":"edd8f9e2a8a899d62ec1cb416ed8929b892f769e"},"cell_type":"code","source":"df_train_copy = df_train.copy()\ndf_test_copy = df_test.copy()\n\ndf_train = df_train_copy.copy()\ndf_test = df_test_copy.copy()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:29:12.275588Z","start_time":"2018-09-25T09:29:12.271869Z"},"trusted":true,"_uuid":"4d7762bed9e3caf6a2d6a2a81bdd9391013288d9"},"cell_type":"code","source":"train_dates = df_train['date'].copy()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72a8873eb15cf003697cd3c5b97b26778b018dfe"},"cell_type":"markdown","source":"# Label-encoding"},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:12.394295Z","start_time":"2018-09-25T09:29:12.277215Z"},"trusted":true,"_uuid":"d098d19521ed1b4ade2c4e86e49f008655b69472"},"cell_type":"code","source":"for col in cat_cols:\n    print(col)\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(df_train[col].values.astype('str')) + list(df_test[col].values.astype('str')))\n    df_train[col] = lbl.transform(list(df_train[col].values.astype('str')))\n    df_test[col] = lbl.transform(list(df_test[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:12.623907Z","start_time":"2018-09-25T09:31:12.556413Z"},"trusted":true,"_uuid":"0602a6f20204b6269c565a5137aa7207ee79a35b"},"cell_type":"code","source":"df_train.shape","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:12.740011Z","start_time":"2018-09-25T09:31:12.626703Z"},"trusted":true,"_uuid":"ed74b5bc18de6bbcbb4db4de1683ca98f98b3ad4"},"cell_type":"code","source":"for col in df_train.columns:\n    if col not in num_cols and col not in cat_cols:\n        print(col)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:12.846879Z","start_time":"2018-09-25T09:31:12.744677Z"},"trusted":true,"_uuid":"a216914ff7961e5fbc8c455a9e74e3e6deb85134"},"cell_type":"code","source":"not_use_cols = ['date', 'fullVisitorId', 'visitId', 'totals_transactionRevenue']","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:12.995131Z","start_time":"2018-09-25T09:31:12.850542Z"},"trusted":true,"_uuid":"fe03664e2d968e5c08479729b735dc9f3029c437"},"cell_type":"code","source":"len(cat_cols) + len(num_cols)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:13.100848Z","start_time":"2018-09-25T09:31:12.999462Z"},"trusted":true,"_uuid":"88e626777431366daae0d5c42d1ae483f4303b8d"},"cell_type":"code","source":"len(not_use_cols)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:30.290397Z","start_time":"2018-09-25T09:31:13.105649Z"},"trusted":true,"_uuid":"14f7206db3e649874350e78d43039544d4eb7965"},"cell_type":"code","source":"# Split the train dataset into development and valid based on time \ndev_df = df_train[df_train['date']<=datetime.date(2017,5,31)]\nval_df = df_train[df_train['date']>datetime.date(2017,5,31)]\ndev_y = np.log1p(dev_df[\"totals_transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals_transactionRevenue\"].values)\n\nuse_cols = [col for col in df_train.columns if col not in not_use_cols]\n\ndev_X = dev_df[use_cols] \nval_X = val_df[use_cols] \ntest_X = df_test[use_cols] \n\n# custom function to run light gbm model\ndef run_lgb(train_X, train_y, val_X, val_y, test_X):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \"num_leaves\" : 30,\n        \"min_child_samples\" : 100,\n        \"learning_rate\" : 0.1,\n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.5,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 1989,\n        \"verbosity\" : -1,\n        'seed': 1989\n    }\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    model = lgb.train(params, lgtrain, 10000, valid_sets=[lgtrain, lgval], early_stopping_rounds=500, verbose_eval=100)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    return pred_test_y, model\n\n# Training the model #\npred_test, model = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04dbf31865451f5f6b072e5740e7352e8630a960"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"Label_encoding.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ba2fa335a5a997b38395d15f70702f77629d94c"},"cell_type":"markdown","source":"# Frequency encoding"},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:30.294024Z","start_time":"2018-09-25T09:31:30.291803Z"},"trusted":true,"_uuid":"5b16ed07c085827ff4cf92d6ea6448f202e458a0"},"cell_type":"code","source":"def frequency_encoding(frame, col):\n    freq_encoding = frame.groupby([col]).size()/frame.shape[0] \n    freq_encoding = freq_encoding.reset_index().rename(columns={0:'{}_Frequency'.format(col)})\n    return frame.merge(freq_encoding, on=col, how='left')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:31:30.747896Z","start_time":"2018-09-25T09:31:30.295180Z"},"trusted":true,"_uuid":"8475de6ff85035ad890ab0623ef0d47d805efa01"},"cell_type":"code","source":"df_train = df_train_copy.copy()\ndf_test = df_test_copy.copy()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:32:16.041986Z","start_time":"2018-09-25T09:31:30.749287Z"},"trusted":true,"_uuid":"a2d34b6d53011ed1d68388866cff1e39d34ddf49"},"cell_type":"code","source":"len_train = df_train.shape[0]\ndf_all = pd.concat([df_train, df_test])\n\nfor col in tqdm(cat_cols):\n    df_all = frequency_encoding(df_all, col)\n\ndf_train = df_all[:len_train]\ndf_test = df_all[len_train:]\n\nprint(df_train.shape, df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:32:34.157774Z","start_time":"2018-09-25T09:32:16.043763Z"},"trusted":true,"scrolled":true,"_uuid":"3073df7a975ae894cd94c950ee408ab9a97363ed"},"cell_type":"code","source":"df_train.drop(cat_cols, axis=1, inplace=True)\ndf_test.drop(cat_cols, axis=1, inplace=True)\n\nfreq_cat_cols = ['{}_Frequency'.format(col) for col in cat_cols]\n\n# Split the train dataset into development and valid based on time \ndev_df = df_train[train_dates<=datetime.date(2017,5,31)]\nval_df = df_train[train_dates>datetime.date(2017,5,31)]\n\ndev_y = np.log1p(dev_df[\"totals_transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals_transactionRevenue\"].values)\n\nuse_cols = [col for col in df_train.columns if col not in not_use_cols]\n\ndev_X = dev_df[use_cols] \nval_X = val_df[use_cols] \ntest_X = df_test[use_cols]  \n\n# Training the model #\npred_test, model = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b6789e0a7032a8f1eeedd1501d7ce5ad971b325"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"Freq_encoding.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e696a170209add199a33a00393755149d105c07"},"cell_type":"markdown","source":"# Mean encoding"},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:32:34.161554Z","start_time":"2018-09-25T09:32:34.159270Z"},"trusted":true,"_uuid":"82252e8a8944a413b229276056bb48db49184fb7"},"cell_type":"code","source":"from sklearn.model_selection import KFold","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:32:34.246307Z","start_time":"2018-09-25T09:32:34.162684Z"},"trusted":true,"_uuid":"c6af63cde8168c4c83cbca90ff550c0873a8e01e"},"cell_type":"code","source":"def mean_k_fold_encoding(col, alpha):\n    target_name = 'totals_transactionRevenue'\n    target_mean_global = df_train[target_name].mean()\n    \n    nrows_cat = df_train.groupby(col)[target_name].count()\n    target_means_cats = df_train.groupby(col)[target_name].mean()\n    target_means_cats_adj = (target_means_cats*nrows_cat + \n                             target_mean_global*alpha)/(nrows_cat+alpha)\n    # Mapping means to test data\n    encoded_col_test = df_test[col].map(target_means_cats_adj)\n    \n    kfold = KFold(n_splits=5, shuffle=True, random_state=1989)\n    parts = []\n    for trn_inx, val_idx in kfold.split(df_train):\n        df_for_estimation, df_estimated = df_train.iloc[trn_inx], df_train.iloc[val_idx]\n        nrows_cat = df_for_estimation.groupby(col)[target_name].count()\n        target_means_cats = df_for_estimation.groupby(col)[target_name].mean()\n\n        target_means_cats_adj = (target_means_cats * nrows_cat + \n                                target_mean_global * alpha) / (nrows_cat + alpha)\n\n        encoded_col_train_part = df_estimated[col].map(target_means_cats_adj)\n        parts.append(encoded_col_train_part)\n        \n    encoded_col_train = pd.concat(parts, axis=0)\n    encoded_col_train.fillna(target_mean_global, inplace=True)\n    encoded_col_train.sort_index(inplace=True)\n    \n    return encoded_col_train, encoded_col_test","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:32:34.698010Z","start_time":"2018-09-25T09:32:34.248095Z"},"trusted":true,"_uuid":"52b070dfcf0931cd92407455ed6e678cc6a415e7"},"cell_type":"code","source":"df_train = df_train_copy.copy()\ndf_test = df_test_copy.copy()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:33:14.399588Z","start_time":"2018-09-25T09:32:34.699312Z"},"trusted":true,"_uuid":"fbfbce90c253f033d6a389fb68b302929799a0de"},"cell_type":"code","source":"for col in tqdm(cat_cols):\n    temp_encoded_tr, temp_encoded_te = mean_k_fold_encoding(col, 5)\n    new_feat_name = 'mean_k_fold_{}'.format(col)\n    df_train[new_feat_name] = temp_encoded_tr.values\n    df_test[new_feat_name] = temp_encoded_te.values","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:33:29.906968Z","start_time":"2018-09-25T09:33:14.400826Z"},"trusted":true,"_uuid":"0aad2ca6813c67501afd1d63569e2b934ec1a79c"},"cell_type":"code","source":"df_train.drop(cat_cols, axis=1, inplace=True)\ndf_test.drop(cat_cols, axis=1, inplace=True)\n\nmean_cat_cols = ['mean_k_fold_{}'.format(col) for col in cat_cols]\n\n# Split the train dataset into development and valid based on time \ndev_df = df_train[train_dates<=datetime.date(2017,5,31)]\nval_df = df_train[train_dates>datetime.date(2017,5,31)]\n\ndev_y = np.log1p(dev_df[\"totals_transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals_transactionRevenue\"].values)\n\nuse_cols = [col for col in df_train.columns if col not in not_use_cols]\n\ndev_X = dev_df[use_cols] \nval_X = val_df[use_cols] \ntest_X = df_test[use_cols] \n\n# Training the model #\npred_test, model = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d92c53cd8033e991b0f342dac79e522a6a50376"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"mean_encoding.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1fc370c75d3e2a3411e2fb7c0415dedfe73bbf01"},"cell_type":"markdown","source":"# Label + Frequency + Mean encoding"},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:35:14.049651Z","start_time":"2018-09-25T09:33:29.908345Z"},"trusted":true,"_uuid":"657ba4ce47c96595c05b0ccf621f36d67bc0af96"},"cell_type":"code","source":"df_train = df_train_copy.copy()\ndf_test = df_test_copy.copy()\n\nlen_train = df_train.shape[0]\ndf_all = pd.concat([df_train, df_test])\n\nfor col in tqdm(cat_cols):\n    df_all = frequency_encoding(df_all, col)\n\ndf_train = df_all[:len_train]\ndf_test = df_all[len_train:]\n\nprint(df_train.shape, df_test.shape)\n\nfor col in tqdm(cat_cols):\n    temp_encoded_tr, temp_encoded_te = mean_k_fold_encoding(col, 5)\n    new_feat_name = 'mean_k_fold_{}'.format(col)\n    df_train[new_feat_name] = temp_encoded_tr.values\n    df_test[new_feat_name] = temp_encoded_te.values","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:37:18.429690Z","start_time":"2018-09-25T09:35:14.050824Z"},"trusted":true,"_uuid":"5bab79056d14d622a68035a2363414d82a737d80"},"cell_type":"code","source":"for col in cat_cols:\n    print(col)\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(df_train[col].values.astype('str')) + list(df_test[col].values.astype('str')))\n    df_train[col] = lbl.transform(list(df_train[col].values.astype('str')))\n    df_test[col] = lbl.transform(list(df_test[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:37:18.433008Z","start_time":"2018-09-25T09:37:18.431015Z"},"trusted":true,"_uuid":"71d67008c137c71b87ed878bee2a0a03fdf266a3"},"cell_type":"code","source":"freq_cat_cols = ['{}_Frequency'.format(col) for col in cat_cols]\nmean_cat_cols = ['mean_k_fold_{}'.format(col) for col in cat_cols]","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-09-25T09:37:47.844444Z","start_time":"2018-09-25T09:37:18.434225Z"},"trusted":true,"_uuid":"e9947146f2dfc7ab7333c1c9baeb216f4a4ff9af"},"cell_type":"code","source":"print(df_train.shape, df_test.shape)\n\n# Split the train dataset into development and valid based on time \ndev_df = df_train[train_dates<=datetime.date(2017,5,31)]\nval_df = df_train[train_dates>datetime.date(2017,5,31)]\ndev_y = np.log1p(dev_df[\"totals_transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals_transactionRevenue\"].values)\n\nuse_cols = [col for col in df_train.columns if col not in not_use_cols]\n\ndev_X = dev_df[use_cols] \nval_X = val_df[use_cols] \ntest_X = df_test[use_cols]  \n\n# Training the model #\npred_test, model = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0fe4e1ecf35d001ab9e231f9e151326e68b4d5f"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"all.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"71e60a383949756963850b929d674129813728ce"},"cell_type":"markdown","source":"# Result\n| Encoding  | Training RMSE  | VALID RMSE  |  RMSE(Tr) / RMSE(vld)  | LB \n|---|---|---|---|---|\n| Label encoding |  1.52503  | 1.69546  |  1.111755 |  1.4470\n|  Frequency encoding | 1.52039  | 1.69291 | 1.113471 | 1.4545\n|  Mean encoding |  1.52247 | 1.6955  | 1.113651  |  1.4448\n|Label + Fre + Mean (All) | 1.51965 | 1.69179 | 1.1132761 |  1.4417"},{"metadata":{"trusted":true,"_uuid":"b3c46174eda46ba2176f597ddcc1d323ce9d8636"},"cell_type":"markdown","source":"- Traning RMSE: All  < Freq < Mean < Label\n- Valid RMSE: All < Freq < Label < Mean\n- Tr/vld ratio: Label < All < Freq < Mean"},{"metadata":{"_uuid":"d0dda30390eb96bd6484ac400bdb391bb0b43a6c"},"cell_type":"markdown","source":"- Freq, Mean encoding tend to overfit to training set.\n- But LB of Mean encoding is lower than LB of Freq."},{"metadata":{"_uuid":"7b8b9154b87812f061cb58a653e14c4c20223a47"},"cell_type":"markdown","source":"# Conclusion"},{"metadata":{"_uuid":"dd66f8de4ca14ef6669e2fc0111cab14a3f0e0fe"},"cell_type":"markdown","source":"- Of course, it's not easy to say the all(Label + Fre + Mean) is the best choice.\n- But, adding other encoding performs better than only using label encoding."},{"metadata":{"_uuid":"2251b9a4db64a05fc57e798ea03b45653c40eb74"},"cell_type":"markdown","source":"# More\n- I know, this experiment is so simple. The result could be changed depending on hyper-parameters and algoritm.\n- But, I think adding various encoding give us better performance than using only one encoding strategy because applying various approachs (like ensemble)  commoly shows good result."},{"metadata":{"trusted":true,"_uuid":"c6938751facc768f5aefdecb088d5c5c274407cb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4e76f9ff601260a440b8809fc1dc5a65a1dfc60"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"384px"},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":1}