{"cells":[{"metadata":{"_uuid":"44d8f4d13bbbe6e827d22f1230fba10e60389060"},"cell_type":"markdown","source":"## Introduction\n\nI believe the main issue we have in this challenge is not to predict revenues but more to get these zeros right since less than 1.3 % of the sessions have a non-zero revenue.\n\nThe idea in this kernel is to classify non-zero transactions first and use that to help our regressor get better results.\n\nThe kernel only presents one way of doing it. No special feature engineering or set of hyperparameters, just a code shell/structure ;-) "},{"metadata":{"_uuid":"1db520441d58d29f05ea6afca18dc56a541e3d39"},"cell_type":"markdown","source":"### Check file structure"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec8e0981e8c26bdd8a097ff838ecf09cc9af8c21"},"cell_type":"markdown","source":"### Import packages"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport matplotlib.cbook as cbook\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error, roc_auc_score, log_loss\nimport gc\nimport time\nfrom pandas.core.common import SettingWithCopyWarning\nimport warnings\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold, GroupKFold\n\nwarnings.simplefilter('error', SettingWithCopyWarning)\ngc.enable()\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f866bfe2f783c5b535de3e90d11534405f59384"},"cell_type":"markdown","source":"### Get data"},{"metadata":{"trusted":true,"_uuid":"6f0c0aed82437dd6efcd3731016e236035b35215"},"cell_type":"code","source":"train = pd.read_csv('../input/create-extracted-json-fields-dataset/extracted_fields_train.gz', \n                    dtype={'date': str, 'fullVisitorId': str, 'sessionId':str}, nrows=None)\ntest = pd.read_csv('../input/create-extracted-json-fields-dataset/extracted_fields_test.gz', \n                   dtype={'date': str, 'fullVisitorId': str, 'sessionId':str}, nrows=None)\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f17e9157ce05123e680d573691d121850ac1f4f5"},"cell_type":"markdown","source":"### Get targets"},{"metadata":{"trusted":true,"_uuid":"abfebee8d07a27278692e63e25405e1454eeb771"},"cell_type":"code","source":"y_clf = (train['totals.transactionRevenue'].fillna(0) > 0).astype(np.uint8)\ny_reg = train['totals.transactionRevenue'].fillna(0)\ndel train['totals.transactionRevenue']\ny_clf.mean(), y_reg.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54fafb2a4c679a4bf7b2a03d8d738c800e9fac84"},"cell_type":"markdown","source":"### Add date features"},{"metadata":{"trusted":true,"_uuid":"35e3931adb5dbeac55a57951d5fdc8efa8fa3934"},"cell_type":"code","source":"for df in [train, test]:\n    df['date'] = pd.to_datetime(df['date'])\n    df['vis_date'] = pd.to_datetime(df['visitStartTime'])\n    df['sess_date_dow'] = df['vis_date'].dt.dayofweek\n    df['sess_date_hours'] = df['vis_date'].dt.hour\n    df['sess_date_dom'] = df['vis_date'].dt.day","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f815321ded2e3a29a8c27b4ee9944eb59042329"},"cell_type":"markdown","source":"### Create list of features"},{"metadata":{"trusted":true,"_uuid":"20e3c879a70c91cd750c6c047afcd1c16fd7ca37"},"cell_type":"code","source":"excluded_features = [\n    'date', 'fullVisitorId', 'sessionId', 'totals.transactionRevenue', \n    'visitId', 'visitStartTime', 'non_zero_proba', 'vis_date'\n]\n\ncategorical_features = [\n    _f for _f in train.columns\n    if (_f not in excluded_features) & (train[_f].dtype == 'object')\n]\n\nif 'totals.transactionRevenue' in train.columns:\n    del train['totals.transactionRevenue']\n\nif 'totals.transactionRevenue' in test.columns:\n    del test['totals.transactionRevenue']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9557c48c2f32b0af57bf25fe23d98e8aef3a67b"},"cell_type":"markdown","source":"### Factorize categoricals"},{"metadata":{"trusted":true,"_uuid":"b56ea7dd7ad87f7ddac82f88b892d0ee62376c2a"},"cell_type":"code","source":"for f in categorical_features:\n    train[f], indexer = pd.factorize(train[f])\n    test[f] = indexer.get_indexer(test[f])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2ca8af2e7219a6c9b552912878f27f7c7c48c02"},"cell_type":"markdown","source":"### Classify non-zero revenues"},{"metadata":{"trusted":true,"_uuid":"3a1575b33dcfa55cff56949ae58ddd2606a9d691"},"cell_type":"code","source":"folds = GroupKFold(n_splits=5)\n\ntrain_features = [_f for _f in train.columns if _f not in excluded_features]\nprint(train_features)\noof_clf_preds = np.zeros(train.shape[0])\nsub_clf_preds = np.zeros(test.shape[0])\nfor fold_, (trn_, val_) in enumerate(folds.split(y_clf, y_clf, groups=train['fullVisitorId'])):\n    trn_x, trn_y = train[train_features].iloc[trn_], y_clf.iloc[trn_]\n    val_x, val_y = train[train_features].iloc[val_], y_clf.iloc[val_]\n    \n    clf = lgb.LGBMClassifier(\n        num_leaves=31,\n        learning_rate=0.03,\n        n_estimators=1000,\n        subsample=.9,\n        colsample_bytree=.9,\n        random_state=1\n    )\n    clf.fit(\n        trn_x, trn_y,\n        eval_set=[(val_x, val_y)],\n        early_stopping_rounds=50,\n        verbose=50\n    )\n    \n    oof_clf_preds[val_] = clf.predict_proba(val_x, num_iteration=clf.best_iteration_)[:, 1]\n    print(roc_auc_score(val_y, oof_clf_preds[val_]))\n    sub_clf_preds += clf.predict_proba(test[train_features], num_iteration=clf.best_iteration_)[:, 1] / folds.n_splits\n    \nroc_auc_score(y_clf, oof_clf_preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2ad2c5f72d7885831bd4823caf786c68e2ea5cb"},"cell_type":"markdown","source":"### Add classification to dataset"},{"metadata":{"trusted":true,"_uuid":"a96c096e4779041b4da97599a25cd1667006abed"},"cell_type":"code","source":"train['non_zero_proba'] = oof_clf_preds\ntest['non_zero_proba'] = sub_clf_preds","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e9c88f125e8b4f1087b07510505228bb2463f0d"},"cell_type":"markdown","source":"### Predict revenues at session level"},{"metadata":{"trusted":true,"_uuid":"9093926371607ab06e8c20bb946e74d1f0f13082"},"cell_type":"code","source":"train_features = [_f for _f in train.columns if _f not in excluded_features] + ['non_zero_proba']\nprint(train_features)\n\noof_reg_preds = np.zeros(train.shape[0])\nsub_reg_preds = np.zeros(test.shape[0])\nimportances = pd.DataFrame()\n\nfor fold_, (trn_, val_) in enumerate(folds.split(y_reg, y_reg, groups=train['fullVisitorId'])):\n    trn_x, trn_y = train[train_features].iloc[trn_], y_reg.iloc[trn_].fillna(0)\n    val_x, val_y = train[train_features].iloc[val_], y_reg.iloc[val_].fillna(0)\n    \n    reg = lgb.LGBMRegressor(\n        num_leaves=31,\n        learning_rate=0.03,\n        n_estimators=1000,\n        subsample=.9,\n        colsample_bytree=.9,\n        random_state=1\n    )\n    reg.fit(\n        trn_x, np.log1p(trn_y),\n        eval_set=[(val_x, np.log1p(val_y))],\n        early_stopping_rounds=50,\n        verbose=50\n    )\n    imp_df = pd.DataFrame()\n    imp_df['feature'] = train_features\n    imp_df['gain'] = reg.booster_.feature_importance(importance_type='gain')\n    \n    imp_df['fold'] = fold_ + 1\n    importances = pd.concat([importances, imp_df], axis=0, sort=False)\n    \n    oof_reg_preds[val_] = reg.predict(val_x, num_iteration=reg.best_iteration_)\n    oof_reg_preds[oof_reg_preds < 0] = 0\n    _preds = reg.predict(test[train_features], num_iteration=reg.best_iteration_)\n    _preds[_preds < 0] = 0\n    sub_reg_preds += np.expm1(_preds) / folds.n_splits\n    \nmean_squared_error(np.log1p(y_reg.fillna(0)), oof_reg_preds) ** .5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bbf589f7bd04e7aa4386af04b23edfec23bba3bb"},"cell_type":"code","source":"import warnings\nwarnings.simplefilter('ignore', FutureWarning)\n\nimportances['gain_log'] = np.log1p(importances['gain'])\nmean_gain = importances[['gain', 'feature']].groupby('feature').mean()\nimportances['mean_gain'] = importances['feature'].map(mean_gain['gain'])\n\nplt.figure(figsize=(8, 12))\nsns.barplot(x='gain_log', y='feature', data=importances.sort_values('mean_gain', ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b26745de1a6dab24ffeb29829cfa80183446e9e"},"cell_type":"markdown","source":"### Save predictions\n\nMaybe one day Kaggle will support file compression for submissions from kernels...\n\nI'm aware I sum the logs instead of summing the actual revenues..."},{"metadata":{"trusted":true,"_uuid":"00c39d147aabcf384f328b01fdaade7f1fb5a3ab"},"cell_type":"code","source":"test['PredictedLogRevenue'] = sub_reg_preds\ntest[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum()['PredictedLogRevenue'].apply(np.log1p).reset_index()\\\n    .to_csv('test_clf_reg_log_of_sum.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2460f0519f6f18bef60f6ef267fc3af9c067340a"},"cell_type":"markdown","source":"### Plot Actual Dollar estimates per dates"},{"metadata":{"trusted":true,"_uuid":"60c10960c9317c732f9716b11aed5021036267c1"},"cell_type":"code","source":"# Go to actual revenues\ntrain['PredictedRevenue'] = np.expm1(oof_reg_preds)\ntest['PredictedRevenue'] = sub_reg_preds\ntrain['totals.transactionRevenue'] = y_reg\n\n# Sum by date on train and test\ntrn_group = train[['date', 'PredictedRevenue', 'totals.transactionRevenue']].groupby('date').sum().reset_index()\nsub_group = test[['date', 'PredictedRevenue']].groupby('date').sum().reset_index()\n\n# Now plot all this\nyears = mdates.YearLocator()   # every year\nmonths = mdates.MonthLocator()  # every month\nyearsFmt = mdates.DateFormatter('%Y-%m')\n\nfig, ax = plt.subplots(figsize=(15, 6))\nax.set_title('Actual Dollar Revenues - we are way off...', fontsize=15, fontweight='bold')\nax.plot(pd.to_datetime(trn_group['date']).values, trn_group['totals.transactionRevenue'].values)\nax.plot(pd.to_datetime(trn_group['date']).values, trn_group['PredictedRevenue'].values)\nax.plot(pd.to_datetime(sub_group['date']).values, sub_group['PredictedRevenue'].values)\n\n# # format the ticks\nax.xaxis.set_major_locator(months)\nax.xaxis.set_major_formatter(yearsFmt)\nax.xaxis.set_minor_locator(months)\n\nax.format_xdata = mdates.DateFormatter('%Y-%m-%d')\n# # ax.format_ydata = price\nax.grid(True)\n\n# rotates and right aligns the x labels, and moves the bottom of the\n# axes up to make room for them\nfig.autofmt_xdate()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89237540966cad34cbd385617bb09e2fd4a3dd39"},"cell_type":"markdown","source":"### Display using np.log1p"},{"metadata":{"trusted":true,"_uuid":"73e098514fc6b2f2a4b5da0aa728475e1ea68e5c","_kg_hide-input":true},"cell_type":"code","source":"# Go to actual revenues\ntrain['PredictedRevenue'] = np.expm1(oof_reg_preds)\ntest['PredictedRevenue'] = sub_reg_preds\ntrain['totals.transactionRevenue'] = y_reg\n\n# Sum by date on train and test\ntrn_group = train[['date', 'PredictedRevenue', 'totals.transactionRevenue']].groupby('date').sum().reset_index()\nsub_group = test[['date', 'PredictedRevenue']].groupby('date').sum().reset_index()\n\nyears = mdates.YearLocator()   # every year\nmonths = mdates.MonthLocator()  # every month\nyearsFmt = mdates.DateFormatter('%Y-%m')\n\nfig, ax = plt.subplots(figsize=(15, 6))\nax.set_title('We are also off in logs... or am I just stupid ?', fontsize=15, fontweight='bold')\nax.plot(pd.to_datetime(trn_group['date']).values, np.log1p(trn_group['totals.transactionRevenue'].values))\nax.plot(pd.to_datetime(trn_group['date']).values, np.log1p(trn_group['PredictedRevenue'].values))\nax.plot(pd.to_datetime(sub_group['date']).values, np.log1p(sub_group['PredictedRevenue'].values))\n\n# # format the ticks\nax.xaxis.set_major_locator(months)\nax.xaxis.set_major_formatter(yearsFmt)\nax.xaxis.set_minor_locator(months)\n\nax.format_xdata = mdates.DateFormatter('%Y-%m-%d')\n# # ax.format_ydata = price\nax.grid(True)\n\n# rotates and right aligns the x labels, and moves the bottom of the\n# axes up to make room for them\nfig.autofmt_xdate()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b0ff54d8ecad195c91b475d1a4b72d05dcc5708"},"cell_type":"markdown","source":"### Using sum of logs - no really ?"},{"metadata":{"trusted":true,"_uuid":"7968c3596908f77a3e58014b935cf54b0a98e38e","_kg_hide-input":true},"cell_type":"code","source":"# Keep amounts in logs\ntrain['PredictedRevenue'] = oof_reg_preds\ntest['PredictedRevenue'] = np.log1p(sub_reg_preds)\ntrain['totals.transactionRevenue'] = np.log1p(y_reg)\n\n# You really mean summing up the logs ???\ntrn_group = train[['date', 'PredictedRevenue', 'totals.transactionRevenue']].groupby('date').sum().reset_index()\nsub_group = test[['date', 'PredictedRevenue']].groupby('date').sum().reset_index()\n\nyears = mdates.YearLocator()   # every year\nmonths = mdates.MonthLocator()  # every month\nyearsFmt = mdates.DateFormatter('%Y-%m')\n\nfig, ax = plt.subplots(figsize=(15, 6))\nax.set_title('Summing up logs looks a lot better !?! Is the challenge to find the correct metric ???', fontsize=15, fontweight='bold')\nax.plot(pd.to_datetime(trn_group['date']).values, trn_group['totals.transactionRevenue'].values)\nax.plot(pd.to_datetime(trn_group['date']).values, trn_group['PredictedRevenue'].values)\nax.plot(pd.to_datetime(sub_group['date']).values, sub_group['PredictedRevenue'].values)\n\n# # format the ticks\nax.xaxis.set_major_locator(months)\nax.xaxis.set_major_formatter(yearsFmt)\nax.xaxis.set_minor_locator(months)\n\nax.format_xdata = mdates.DateFormatter('%Y-%m-%d')\n# # ax.format_ydata = price\nax.grid(True)\n\n# rotates and right aligns the x labels, and moves the bottom of the\n# axes up to make room for them\nfig.autofmt_xdate()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c3422139efe35e811f0528790112cc3b6b4142c"},"cell_type":"markdown","source":"The issue is that the model highly underestimates the log revenues. As a consequence, going back to actual dollar revenues underestimates even more transactions. Now if you sum up a high number of transactions (like we do to display things at date level) will get estimates really off. This is what the 1st plot demonstrates.\n\nThe 2nd plot shows the same thing but in log. Again this is due to the number of transactions we need to aggregate at date level.\n\nDue to the underestimation, summing up log revenues (i.e. in fact multiplying them) will get revenues on the same range as actual revenues. This is just a proof of our underestimation at session level.\n\nAs it's been said on the forum we don't have a lot of Visitors with lots of sessions and this is why Visitor RMSE and Session RMSE are on the same scale.\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}