{"cells":[{"metadata":{"_uuid":"726990b51a779af2ee0566d87038d721b56c6ca6"},"cell_type":"markdown","source":"**About the competition**\n\nIn this competition, we’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. Hopefully, the outcome will be more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data.\n\n**Objectives of the Notebook**\n\nIn this notebook we will go through the features of the dataset. We will try to clean the data by preprocessing with Pandas and try to make it model ready for Baseline LightGBM model. Then Finally we will apply LightGBM model to predict the outcome.\n\n> This is my first notebook where I have tried to present what I have done in a documented manner. Please point out any noticeable mistake that I have done. Thank you. :)\n\n**Inspirations of the Notebook**\n\nSome parts of the notebook is inspired by the following notebooks.\n* [Simple Exploration+Baseline - GA Customer Revenue](https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue) by [SRK](https://www.kaggle.com/sudalairajkumar)\n*  [2 - Quick study: LGBM, XGB and Catboost [LB: 1.66]](https://www.kaggle.com/julian3833/2-quick-study-lgbm-xgb-and-catboost-lb-1-66) by [Julian](https://www.kaggle.com/julian3833)\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport time\nimport gc\nimport json\nimport os\nfrom datetime import datetime\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport seaborn as sns\nimport xgboost as xgb\nimport lightgbm as lgb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import Imputer, LabelEncoder\nfrom sklearn.metrics import accuracy_score, f1_score, mean_absolute_error\nsns.set_style(\"dark\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     parse_dates=['date'],\n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e38aaf971cee9b55c0e33f7eaed86a980c45082"},"cell_type":"markdown","source":"**About the Features**\n\n* fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n* channelGrouping - The channel via which the user came to the Store.\n* date - The date on which the user visited the Store.\n* device - The specifications for the device used to access the Store.\n* geoNetwork - This section contains information about the geography of the user.\n* sessionId - A unique identifier for this visit to the store.\n* socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n* totals - This section contains aggregate values across the session.\n* trafficSource - This section contains information about the Traffic Source from which the session originated.\n* visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n* visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n* visitStartTime - The timestamp (expressed as POSIX time)."},{"metadata":{"trusted":true,"_uuid":"1e3e5e5adbc277a71fe669de4083bbf2a659bd70"},"cell_type":"code","source":"train_df.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5551556c9e73b3b7a2811f79b2c18a2f5dd3d9c"},"cell_type":"markdown","source":"**Preprocessing**\n\nFirst we will merge the train and test dataset to make the basic operations on the whole dataset easier."},{"metadata":{"trusted":true,"_uuid":"0598680ebdcea9c2622c42ac0ad9d8db315d2001"},"cell_type":"code","source":"train_df['train_or_test'] = 'train'\ntest_df['train_or_test'] = 'test'\ntest_df['totals.transactionRevenue'] = np.nan\ndf = pd.concat([train_df, test_df], sort=False, ignore_index=True)\ndel train_df\ndel test_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12f98e1e2acfb6153b697472dd912bf7763baeae"},"cell_type":"code","source":"df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"080e9d2678d45a2d1eb9f6d366c0b6aa8471ff6e"},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4c9793c2861de3362a562d502a3f21b49053c78"},"cell_type":"markdown","source":"Based on the time-series nature of the data, I have created some new features which might be helpful to get new insights in the future."},{"metadata":{"trusted":true,"_uuid":"6d2808904377429b69ea6db94c4bbd74554f079c"},"cell_type":"code","source":"df['year'] = df.date.dt.year\ndf['month'] = df.date.dt.month\ndf['dayofmonth'] = df.date.dt.day\ndf['dayofweek'] = df.date.dt.dayofweek\ndf['dayofyear'] = df.date.dt.dayofyear\ndf['weekofyear'] = df.date.dt.weekofyear\ndf['is_month_start'] = (df.date.dt.is_month_start).astype(int)\ndf['is_month_end'] = (df.date.dt.is_month_end).astype(int)\ndf['quarter'] = df.date.dt.quarter\ndf['week_block_num'] = [int(x) for x in np.floor((df.date - pd.to_datetime('2012-12-31')).dt.days/7) + 1]\ndf['quarter_block_num'] = (df['year'] - 2013) * 4 + df['quarter']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af544ba500c30ac38f6affe6b0c8a9ea19694352"},"cell_type":"code","source":"df.describe(include=\"all\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab46bb9669f0b3756507a2743dc40284efccaf0c"},"cell_type":"markdown","source":"Many of the columns contain constant values for all the training and test examples. Removing them is wise to make predictions faster."},{"metadata":{"trusted":true,"_uuid":"433658608ccf2245b4c221f85edc0023136dcf17"},"cell_type":"code","source":"dropcols = [c for c in df.columns if df[c].nunique(dropna=True)==1]\ndropcols.remove('totals.bounces')\ndropcols.remove('totals.newVisits')\nprint(dropcols)\ndf.drop(dropcols,axis=1,inplace=True,errors='ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6150b05627a61e53a098c25c5b6f4913331a103"},"cell_type":"markdown","source":"These following 4 features seem to have integer format and there missing values cell should most probably be 0."},{"metadata":{"trusted":true,"_uuid":"158a4387f64fc5de3daa1acd163d35b1b5eb0581"},"cell_type":"code","source":"df['totals.bounces'].fillna(0,inplace=True)\ndf['totals.hits'].fillna(0,inplace=True)\ndf['totals.pageviews'].fillna(0,inplace=True)\ndf['totals.newVisits'].fillna(0,inplace=True)\ndf[['totals.bounces','totals.hits','totals.pageviews','totals.newVisits']] = df[['totals.bounces','totals.hits','totals.pageviews','totals.newVisits']].astype(np.int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a7187f2ed16cf1d6cac270c550354e62f0a1c047"},"cell_type":"markdown","source":"Trying to find out the reason behind all the other missing values in the dataset."},{"metadata":{"trusted":true,"_uuid":"0147e468a4f8eb376d114ce31da7a7721520ca3e"},"cell_type":"code","source":"null_df = df.isnull().sum().reset_index()\nnull_df[0] = null_df[0] / df.shape[0]\nnull_df[null_df[0] > 0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b0c5d147f2918153da41e384005994a7e41d6c6"},"cell_type":"markdown","source":"These features has similar reason for missing values. I am trying to put appropriate fill for the missing cells for these columns."},{"metadata":{"trusted":true,"_uuid":"832f95f21d1411b4297c878e04dcaa0cc0cb8db1"},"cell_type":"code","source":"cols = ['trafficSource.adwordsClickInfo.adNetworkType','trafficSource.adwordsClickInfo.gclId','trafficSource.adwordsClickInfo.slot','trafficSource.adContent']\ndf[cols] = df[cols].fillna(\"No_Ad\")\ndf['trafficSource.adwordsClickInfo.page'].fillna(0,inplace=True)\ndf['trafficSource.referralPath'].fillna(\"No_Path\",inplace=True)\ndf['trafficSource.adContent'].fillna(\"No_Ad\",inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69a04bb079a952bf98d2935f97d085b4029828a0"},"cell_type":"code","source":"df.describe(include=[\"O\"]).T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a81c8865ac3d158495a232be9d45db316895d76"},"cell_type":"markdown","source":"I am categorizing the object features in two different groups. The group having many categorical values will be used for Label Encoding and the other having only a few categorical values will be used for One Hot Encoding."},{"metadata":{"trusted":true,"_uuid":"e1628c1fd16f03e95d1d48f94ca9b10c756fbd5e"},"cell_type":"code","source":"cat_many_label_cols = [\"channelGrouping\", \"device.browser\", \"device.operatingSystem\", \n            \"geoNetwork.city\", \"geoNetwork.continent\", \n            \"geoNetwork.country\", \"geoNetwork.metro\",\n            \"geoNetwork.networkDomain\", \"geoNetwork.region\", \n            \"geoNetwork.subContinent\", \"trafficSource.adContent\", \n            \"trafficSource.adwordsClickInfo.gclId\", \n            \"trafficSource.adwordsClickInfo.page\", \n            \"trafficSource.campaign\",\n            \"trafficSource.keyword\", \"trafficSource.medium\", \n            \"trafficSource.referralPath\", \"trafficSource.source\"]\n\ncat_few_label_cols = [\"device.deviceCategory\",\"trafficSource.adwordsClickInfo.adNetworkType\",\n                     \"trafficSource.adwordsClickInfo.slot\"]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87e24df5bbde665707ef44454b3bc82cb878f5ef"},"cell_type":"markdown","source":"Performing Label Encoding and One Hot Encoding of the vabove grouped features."},{"metadata":{"trusted":true,"_uuid":"ba7fc089f2d8a6018ac944a8df166816ed54d2cb"},"cell_type":"code","source":"for col in cat_many_label_cols:\n    print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(df[col].values.astype('str')))\n    df[col] = lbl.transform(list(df[col].values.astype('str')))\n    \ndf = pd.get_dummies(df,columns=cat_few_label_cols)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f4d5a19830a3a572c54c1a2547214b9d30d6c60"},"cell_type":"code","source":"df.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f6c5f37dcf7ce9b775acffd51027bfa8a0ea907"},"cell_type":"markdown","source":"Now the target variable also has many missing value in the training dataset. The reason is those instances did not generate any revenues. Thus setting all the missing values to zero is the only option."},{"metadata":{"trusted":true,"_uuid":"d982e4fd29b32f1a9ef87d60aac7d1d5d7421c5f"},"cell_type":"code","source":"df[\"totals.transactionRevenue\"].fillna(0,inplace=True)\ndf[\"totals.transactionRevenue\"] = df[\"totals.transactionRevenue\"].astype(np.float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8da46ad3a67f1bcb69e5d78ca6e3b47b7ed59f6"},"cell_type":"markdown","source":"Now I think most of the data cleaning and feature engineering is over. I will get back the original train and test DataFrame from the merged DataFrame and then extract a validation dataset from the last part of the time sequence."},{"metadata":{"trusted":true,"_uuid":"5cdfcb1f4ceac940f6b54ba94e3b0b619e00c61b"},"cell_type":"code","source":"train_df = df[df.train_or_test=='train']\ntest_df = df[df.train_or_test=='test'].drop('totals.transactionRevenue',axis=1)\nval_df = train_df[train_df['date']>datetime(2017,5,31)]\nprint(train_df.shape)\nprint(val_df.shape)\nprint(test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2f7ecb5f5124b7d269d079da0e07e0b25006aed","trusted":true},"cell_type":"code","source":"dropcols = ['fullVisitorId','sessionId','visitId']\ntrain_x = train_df.drop(dropcols,axis=1)\ntest_x = test_df.drop(dropcols,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d0c651d4b2764f0ea5cbc41c6c56fd6891535c8a"},"cell_type":"code","source":"dev_x = train_x[train_x['date']<=datetime(2017,5,31)]\nval_x = train_x[train_x['date']>datetime(2017,5,31)]\ndev_y = np.log1p(dev_x[\"totals.transactionRevenue\"].values)\nval_y = np.log1p(val_x[\"totals.transactionRevenue\"].values)\ndev_x.drop([\"totals.transactionRevenue\",\"date\",\"train_or_test\"],axis=1,inplace=True)\nval_x.drop([\"totals.transactionRevenue\",\"date\",\"train_or_test\"],axis=1,inplace=True)\ntest_x.drop([\"date\",\"train_or_test\"],axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"763f7abe30e3ac2b8a7e8eba5ccb6a0bc79017b6"},"cell_type":"markdown","source":"**Modeling**\n\nNow we are ready to apply machine learning models on the dataset. I will use gradient boosting framework LightGBM as it is fast and very accurate in modeling big datasets."},{"metadata":{"trusted":true,"_uuid":"badd04360ae7c35924fba392adefdc5a08b81351"},"cell_type":"code","source":"lgb_params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \"num_leaves\" : 1024,\n        'max_depth': 16,  \n        'max_bin': 255,\n        \"min_child_samples\" : 100,\n        \"learning_rate\" : 0.005,\n        'verbose': 0,\n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.7,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018\n    }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"972290211b128df63dec333ff47d3c0b61d03c95"},"cell_type":"code","source":"dtrain = lgb.Dataset(dev_x, label=dev_y)\ndvalid = lgb.Dataset(val_x, label=val_y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff64a6ec36036efb0e0f61c9dfc849d8f124b1c4"},"cell_type":"code","source":"evals_results = {}\nprint(\"Training the model...\")\n\nstart = datetime.now()\nlgb_model = lgb.train(lgb_params, \n                 dtrain, \n                 valid_sets=[dtrain, dvalid], \n                 valid_names=['train','valid'], \n                 evals_result=evals_results, \n                 num_boost_round=1000,\n                 early_stopping_rounds=70,\n                 verbose_eval=50, \n                 feval=None)\nprint(\"Total time taken : \", datetime.now()-start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea3250b92b35732ea2bc3f6855375817bef8ce33"},"cell_type":"code","source":"pred_test_lgb = lgb_model.predict(test_x, num_iteration=lgb_model.best_iteration)\npred_val_lgb = lgb_model.predict(val_x, num_iteration=lgb_model.best_iteration)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef8f232e5c8051c8e3eff70fb25033591bf4b95e"},"cell_type":"markdown","source":"Will now test the accuracy on the validation dataset. "},{"metadata":{"trusted":true,"_uuid":"914fec53b01774fbaaeda772fdc2307d3889021d"},"cell_type":"code","source":"from sklearn import metrics\npred_val_lgb[pred_val_lgb<0] = 0\nval_pred_df = pd.DataFrame({\"fullVisitorId\":val_df[\"fullVisitorId\"].values})\nval_pred_df[\"transactionRevenue\"] = val_df[\"totals.transactionRevenue\"].values\nval_pred_df[\"PredictedRevenue\"] = np.expm1(pred_val_lgb)\nval_pred_df = val_pred_df.groupby(\"fullVisitorId\")[\"transactionRevenue\", \"PredictedRevenue\"].sum().reset_index()\nprint(np.sqrt(metrics.mean_squared_error(np.log1p(val_pred_df[\"transactionRevenue\"].values), np.log1p(val_pred_df[\"PredictedRevenue\"].values))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d315dfd74710e5b7ab2d33991216fc27961960f"},"cell_type":"markdown","source":"Now plotting the feature importance graph as calculated by the above LightGBM model."},{"metadata":{"trusted":true,"_uuid":"a7b36529563a1eb80ae44fce6663a23ced4f04a2"},"cell_type":"code","source":"fold_importance_df = pd.DataFrame()\nfold_importance_df[\"feature\"] = val_x.columns\nfold_importance_df[\"importance\"] = lgb_model.feature_importance()\nplt.figure(figsize=(18,20))\nsns.barplot(x='importance',y='feature',data=fold_importance_df.sort_values(by=\"importance\", ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e01a3c8a4ad895a2f58797c9de583953f9d8ad2"},"cell_type":"markdown","source":"Now applying the model on the test dataset and trying to generate submission files."},{"metadata":{"trusted":true,"_uuid":"0e463a1d23e2af90b2962a9d14ca646aca654a80"},"cell_type":"code","source":"train_id = train_df[\"fullVisitorId\"].values\ntest_id = test_df[\"fullVisitorId\"].values\nsub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test_lgb[pred_test_lgb<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test_lgb)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}