{"cells":[{"metadata":{"_uuid":"503b27f4751cef3db7bed4a5cf78d660df1ae146"},"cell_type":"markdown","source":"## LGBM (RF) starter\n*aknowledgment: a quick hello at [Olivier](https://www.kaggle.com/ogrellier) to whom I borrowed many lines of code*"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dd1714f80d3aba317196e64400d62f6d85ff873d"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport time\nfrom datetime import datetime\nimport lightgbm as lgb\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e250f1cde4b580075fd1e1152ff13e211f4635a"},"cell_type":"markdown","source":"First, we load the dataset and thanks to [Juliàn Peller](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields), this is made quite simple:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\n\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae71114ba1bfc9a440c9113af30103bdead2341b"},"cell_type":"code","source":"%%time\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e568f400d41ef81e7672d96419e26d5924780f56"},"cell_type":"markdown","source":"The target we want to predict, `transactionRevenue`, is contained in one of the JSON columns, ie. the `totals` column. While loading the dataset, it was renamed as `totals.transactionRevenue`. The target only contains a few non-null values and before taking its log, we fill the NAs:"},{"metadata":{"trusted":true,"_uuid":"932021b5905411aa27bc4a55e906c7a35141e849"},"cell_type":"code","source":"target = train_df['totals.transactionRevenue'].fillna(0).astype(float)\ntarget = target.apply(lambda x: np.log1p(x))\ndel train_df['totals.transactionRevenue']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7824bde10f1d17b2fac153b9e75366c049e82cdd"},"cell_type":"markdown","source":"## Variable selection\nSome variables have a unique value:"},{"metadata":{"trusted":true,"_uuid":"5032502bf6628e875609fc421721dd585e9faef7"},"cell_type":"code","source":"columns_to_remove = [col for col in train_df.columns if train_df[col].nunique() == 1]\nprint(\"Nb. of variables with unique value: {}\".format(len(columns_to_remove)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"273ab1c905cf091ba591de8c4382408cc288779f"},"cell_type":"markdown","source":"However, among these variables, the `nan` values could make sense, [according to the organizers](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65691):"},{"metadata":{"trusted":true,"_uuid":"82e543661a420e598b43cf2ea25517a7bf77d675"},"cell_type":"code","source":"for col in columns_to_remove:\n    if set(['not available in demo dataset']) ==  set(train_df[col].unique()): continue\n    print(col, train_df[col].dtypes, train_df[col].unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99f730a2c7a472f6bee5da20e01b36583e8b52cc"},"cell_type":"code","source":"train_df['totals.bounces'] = train_df['totals.bounces'].fillna('0')\ntest_df['totals.bounces'] = test_df['totals.bounces'].fillna('0')\n\ntrain_df['totals.newVisits'] = train_df['totals.newVisits'].fillna('0')\ntest_df['totals.newVisits'] = test_df['totals.newVisits'].fillna('0')\n\ntrain_df['trafficSource.adwordsClickInfo.isVideoAd'] = train_df['trafficSource.adwordsClickInfo.isVideoAd'].fillna(True)\ntest_df['trafficSource.adwordsClickInfo.isVideoAd'] = test_df['trafficSource.adwordsClickInfo.isVideoAd'].fillna(True)\n\ntrain_df['trafficSource.isTrueDirect'] = train_df['trafficSource.isTrueDirect'].fillna(False)\ntest_df['trafficSource.isTrueDirect'] = test_df['trafficSource.isTrueDirect'].fillna(False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7624906186020596c829aa10ca5ebf57d2f77a0a"},"cell_type":"markdown","source":"Many variables only contain a single class and we remove them:"},{"metadata":{"trusted":true,"_uuid":"c2832954eb6e410c61759b4ea32f30a54e45af48"},"cell_type":"code","source":"columns = [col for col in train_df.columns if train_df[col].nunique() > 1]\n#____________________________\ntrain_df = train_df[columns]\ntest_df = test_df[columns]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6004e1e2bb400bbbd735825b1a6aa86bb613b056"},"cell_type":"markdown","source":"Before performing label encoding, we merge the test and train sets to insure we have consistent labels in the two sets:"},{"metadata":{"trusted":true,"_uuid":"312d61e88204fa0c1011fb9ef1057ca42832ec07"},"cell_type":"code","source":"trn_len = train_df.shape[0]\nmerged_df = pd.concat([train_df, test_df])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"665b948e382d12a7aed0b70255ae10801aa2d4cd"},"cell_type":"markdown","source":"## Feature Engineering"},{"metadata":{"trusted":true,"_uuid":"052591c15d6250b14499a32034cbbae3a65939f6"},"cell_type":"code","source":"merged_df['diff_visitId_time'] = merged_df['visitId'] - merged_df['visitStartTime']\nmerged_df['diff_visitId_time'] = (merged_df['diff_visitId_time'] != 0).astype(int)\ndel merged_df['visitId']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b9ed5dff4f54b419bf6c75e2d509e6e67a253ec3"},"cell_type":"code","source":"del merged_df['sessionId']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd4eaa8ccc7d1bbc5d88510310a0f11e9825975e"},"cell_type":"markdown","source":"We perform some feature engineering on dates:"},{"metadata":{"trusted":true,"_uuid":"514b21266931eaf0930b798e434ce7e48cd63565"},"cell_type":"code","source":"format_str = '%Y%m%d' \nmerged_df['formated_date'] = merged_df['date'].apply(lambda x: datetime.strptime(str(x), format_str))\nmerged_df['month'] = merged_df['formated_date'].apply(lambda x:x.month)\nmerged_df['quarter_month'] = merged_df['formated_date'].apply(lambda x:x.day//8)\nmerged_df['day'] = merged_df['formated_date'].apply(lambda x:x.day)\nmerged_df['weekday'] = merged_df['formated_date'].apply(lambda x:x.weekday())\n\ndel merged_df['date']\ndel merged_df['formated_date']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c7afb5611fee6e8a20df964cd5a17b7b97e17569"},"cell_type":"code","source":"merged_df['totals.hits'] = merged_df['totals.hits'].astype(int)\nmerged_df['mean_hits_per_day'] = merged_df.groupby(['day'])['totals.hits'].transform('mean')\ndel  merged_df['day']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a9859efee1a93b712593866b24c15115796c757"},"cell_type":"code","source":"merged_df['formated_visitStartTime'] = merged_df['visitStartTime'].apply(\n    lambda x: time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(x)))\nmerged_df['formated_visitStartTime'] = pd.to_datetime(merged_df['formated_visitStartTime'])\nmerged_df['visit_hour'] = merged_df['formated_visitStartTime'].apply(lambda x: x.hour)\n\ndel merged_df['visitStartTime']\ndel merged_df['formated_visitStartTime']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dec4aa907a9b136b220e3ad183a02a2307495e22"},"cell_type":"code","source":"# for col in ['totals.newVisits', 'totals.pageviews', 'totals.bounces']:\n#     merged_df[col] = merged_df[col].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f94a7b24d956fa28b007b5648eae8594032d492c"},"cell_type":"code","source":"# aggs = {\n#         #'date': ['min', 'max'],\n#         'totals.hits': ['sum', 'min', 'max', 'mean', 'median'],\n#         'totals.pageviews': ['sum', 'min', 'max', 'mean'],\n#         'totals.bounces': ['sum'],\n#         'totals.newVisits': ['sum']\n#     }\n# users = merged_df.groupby('fullVisitorId').agg(aggs)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dbb443ac5c2aabbd0dfb05a22a9c1f66cccda9bc"},"cell_type":"markdown","source":"## label encoding"},{"metadata":{"trusted":true,"_uuid":"d12f90ca3cb751c8ab5b620a8f387ed0f39265c8"},"cell_type":"code","source":"for col in merged_df.columns:\n    if col in ['fullVisitorId', 'month', 'quarter_month', 'weekday', 'visit_hour', 'WoY']: continue\n    if merged_df[col].dtypes == object or merged_df[col].dtypes == bool:\n        merged_df[col], indexer = pd.factorize(merged_df[col])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5cac9481407cad9e7e60f4740694c4787c921ab"},"cell_type":"code","source":"numerics = [col for col in merged_df.columns if 'totals.' in col]\nnumerics += ['visitNumber', 'mean_hits_per_day', 'fullVisitorId']\ncategorical_feats =  [col for col in merged_df.columns if col not in numerics]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f1eef165602ffc5f698aea92675dd57ba2cfc4d"},"cell_type":"code","source":"for col in categorical_feats:\n    merged_df[col] = merged_df[col].astype(int)\n#merged_df['fullVisitorId'] = merged_df['fullVisitorId'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08c80630e7040e393da014c720138687c9d4b995"},"cell_type":"code","source":"train_df = merged_df[:trn_len]\ntest_df = merged_df[trn_len:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c52a9d042f9a074d204a479e0c2b49ed39511c6"},"cell_type":"markdown","source":"## LGBM\nWe adopt some ad-hoc hyperparameters, set the objective function to regression and use a **random forest** as learning method:"},{"metadata":{"trusted":true,"_uuid":"d9f7f01cbf9d56641a9c709cfd83c4d02fbc8d0a"},"cell_type":"code","source":"param = {'num_leaves': 300,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.005,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.8 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 1,\n         \"verbosity\": -1}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b9ee707d98cf0b78164c685ac6384c2d792fb8c"},"cell_type":"markdown","source":"The train set is split with a Kfold method and the prediction on the test set are averaged:"},{"metadata":{"trusted":true,"_uuid":"82cc1f70a7f7f1fd628292353faf81460f20e98f"},"cell_type":"code","source":"trn_cols = [col for col in train_df.columns if col not in ['fullVisitorId']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35ee1800e472a1ba87b6d3d2b7bb35cdfbd50812"},"cell_type":"code","source":"folds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train_df))\npredictions = np.zeros(len(test_df))\nstart = time.time()\nfeatures = list(train_df[trn_cols].columns)\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_df.values, target.values)):\n    trn_data = lgb.Dataset(train_df.iloc[trn_idx][trn_cols], label=target.iloc[trn_idx], categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(train_df.iloc[val_idx][trn_cols], label=target.iloc[val_idx], categorical_feature=categorical_feats)\n    \n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 100)\n    oof[val_idx] = clf.predict(train_df.iloc[val_idx][trn_cols], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(test_df[trn_cols], num_iteration=clf.best_iteration) / folds.n_splits","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3aa9b2e3205994067f2828976ec33769da2d4f10"},"cell_type":"code","source":"print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"03379b774a37cf3b03b90a98f4b85584c0bb5ebc"},"cell_type":"markdown","source":"We have a look at the most import features:"},{"metadata":{"trusted":true,"_uuid":"6e7d1a2011bdd548d9cba8bb2b6a657dab7fc39e"},"cell_type":"code","source":"cols = feature_importance_df[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n    by=\"importance\", ascending=False)[:1000].index\n\nbest_features = feature_importance_df.loc[feature_importance_df.feature.isin(cols)]\n\nplt.figure(figsize=(14,10))\nsns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5b7d489b80a384d35f5f72beb55b3d861bad351"},"cell_type":"markdown","source":"We create the submission file:"},{"metadata":{"trusted":true,"_uuid":"0e0adb075c576ec68ddd2ef8f461b68b8ae08487"},"cell_type":"code","source":"submission = test_df[['fullVisitorId']].copy()\nsubmission.loc[:, 'PredictedLogRevenue'] = np.expm1(predictions)\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test[\"PredictedLogRevenue\"] = np.log1p(grouped_test[\"PredictedLogRevenue\"])\ngrouped_test.to_csv('submit.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9833298e153d08d1a86255342ba82b23ea027d4e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7885bae28ffa2f6c3bfb1e974e537512749ce3b1"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}