{"cells":[{"metadata":{"_uuid":"bb32cc5d2722e96bb02725aaece54c5509bdacb9"},"cell_type":"markdown","source":"# <strong><span style=\"color: #ff0000;\">Google</span> <span style=\"color: #339966;\">Analytics</span> <span style=\"color: #00ccff;\">Customer</span> <span style=\"color: #ff6600;\">Revenue</span> <span style=\"color: #3366ff;\">Prediction</span></strong>\n\n## <strong><span style=\"color: #ff0000;\">O</span> <span style=\"color: #339966;\">u</span> <span style=\"color: #00ccff;\">t</span> <span style=\"color: #ff6600;\">L i</span> <span style=\"color: #3366ff;\">n e</span></strong>\n----\n1. [**Feature Engineering Functions**](#Feature-Engineering-Function)\n2. [**Load Data**](#2.Load-Data)\n3. [**Drop columns**](#3.Drop-columns)\n4. [**Only one not null value**](#4.Only-one-not-null-value)\n5. [**Converting columns format**](#5.Converting-columns-format)\n6. [**Features engineering**](#6.Features-engineering)\n7. [**Categorical columns(Label Encoding)**](#7.Categorical-columns-Label-Encoding)\n8. [**Data Prepare For Modeling**](#8.Data-Prepare-For-Modeling)\n9. [**Light GBM Model Training**](#9.Light-GBM-Model-Training)  \n9.1.[**Light GBM**](#9.1.Light-GBM)    \n9.2.[**Feature Importance**](#9.2.Feature-Importance)    \n9.3.[**LightGBM Submission**](#9.3.LightGBM-Submission)    \n10. [**10. XGB Model Training**](#10.-XGB-Model-Training)  \n10.1.[**XGB Model**](#10.1-XGB-Model)    \n10.2.[**XGB Feature Importance**](#10.2.XGB-Feature-Importance)    \n10.3.[**XGB Submission**](#10.3.XGB-Submission)   \n11. [**Catboost Model Training**](#11.-Catboost-Model-Training)  \n11.1.[**Catboost Model**](#11.1-Catboost-Model)    \n11.2.[**Catboost Feature Importance**](#11.2.Catboost-Feature-Importance)    \n11.3.[**Catboost Submission**](#11.3.Catboost-Submission)\n\n12.[**Ensemble**](#12.-Ensemble)\n\n**Objective of the notebook:**\n\nIn this notebook, let us explore the given dataset and make some inferences along the way. Also finally we will build a baseline light gbm model to get started. \n\n**Objective of the competition:**\n\nIn this competition, we a’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncolor = sns.color_palette()\n\n%matplotlib inline\n\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nfrom sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb\nimport xgboost as xgb\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6868c4286025cdb431e9f6c51e4ee714d594fca"},"cell_type":"markdown","source":"**About the dataset:**\n\nSimilar to most other kaggle competitions, we are given two datasets\n* train.csv\n* test.csv\n\nEach row in the dataset is one visit to the store. We are predicting the natural log of the sum of all transactions per user. \n    \nThe data fields in the given files are \n* fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n* channelGrouping - The channel via which the user came to the Store.\n* date - The date on which the user visited the Store.\n* device - The specifications for the device used to access the Store.\n* geoNetwork - This section contains information about the geography of the user.\n* sessionId - A unique identifier for this visit to the store.\n* socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n* totals - This section contains aggregate values across the session.\n* trafficSource - This section contains information about the Traffic Source from which the session originated.\n* visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n* visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n* visitStartTime - The timestamp (expressed as POSIX time).\n\nAlso it is important to note that some of the fields are in json format. \n\nThanks to this [wonderful kernel](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook) by [Julian](https://www.kaggle.com/julian3833), we can convert all the json fields in the file to a flattened csv format which generally use in other competitions. also thanks to [Julian](https://www.kaggle.com/julian3833/2-quick-study-lgbm-xgb-and-catboost-lb-1-66/notebook)"},{"metadata":{"_uuid":"6be7ebd7776a156b7db885fa7697ca0dd4e5c68f"},"cell_type":"markdown","source":"# 1.Feature Engineering Function"},{"metadata":{"trusted":true,"_uuid":"d37237352fc2eb969b83d5a9989b040017500a25"},"cell_type":"code","source":"IS_LOCAL=False\nif(IS_LOCAL):\n    PATH=\"../input/google-analytics-customer-revenue-prediction/\"    \nelse:\n    PATH=\"../input/\"\n\n#the columns that will be parsed to extract the fields from the jsons\ncols_to_parse = ['device', 'geoNetwork', 'totals', 'trafficSource']\ndef read_parse_dataframe(file_name):\n    #full path for the data file\n    path = PATH + file_name\n    #read the data file, convert the columns in the list of columns to parse using json loader,\n    #convert the `fullVisitorId` field as a string\n    data_df = pd.read_csv(path, \n        converters={column: json.loads for column in cols_to_parse}, \n        dtype={'fullVisitorId': 'str'})\n    #parse the json-type columns\n    for col in cols_to_parse:\n        #each column became a dataset, with the columns the fields of the Json type object\n        json_col_df = json_normalize(data_df[col])\n        json_col_df.columns = [f\"{col}_{sub_col}\" for sub_col in json_col_df.columns]\n        #we drop the object column processed and we add the columns created from the json fields\n        data_df = data_df.drop(col, axis=1).merge(json_col_df, right_index=True, left_index=True)\n    return data_df\n\ndef process_date_time(data_df):\n    print(\"process date time ...\")\n    data_df['date'] = data_df['date'].astype(str)\n    data_df[\"date\"] = data_df[\"date\"].apply(lambda x : x[:4] + \"-\" + x[4:6] + \"-\" + x[6:])\n    data_df[\"date\"] = pd.to_datetime(data_df[\"date\"])   \n    data_df[\"year\"] = data_df['date'].dt.year\n    data_df[\"month\"] = data_df['date'].dt.month\n    data_df[\"day\"] = data_df['date'].dt.day\n    data_df[\"weekday\"] = data_df['date'].dt.weekday\n    data_df['weekofyear'] = data_df['date'].dt.weekofyear\n    data_df['month_unique_user_count'] = data_df.groupby('month')['fullVisitorId'].transform('nunique')\n    data_df['day_unique_user_count'] = data_df.groupby('day')['fullVisitorId'].transform('nunique')\n    data_df['weekday_unique_user_count'] = data_df.groupby('weekday')['fullVisitorId'].transform('nunique')\n    return data_df\n\ndef process_format(data_df):\n    print(\"process format ...\")\n    for col in ['visitNumber', 'totals_hits', 'totals_pageviews']:\n        data_df[col] = data_df[col].astype(float)\n    data_df['trafficSource_adwordsClickInfo.isVideoAd'].fillna(True, inplace=True)\n    data_df['trafficSource_isTrueDirect'].fillna(False, inplace=True)\n    return data_df\n    \ndef process_device(data_df):\n    print(\"process device ...\")\n    data_df['browser_category'] = data_df['device_browser'] + '_' + data_df['device_deviceCategory']\n    data_df['browser_operatingSystem'] = data_df['device_browser'] + '_' + data_df['device_operatingSystem']\n    data_df['source_country'] = data_df['trafficSource_source'] + '_' + data_df['geoNetwork_country']\n    return data_df\n\ndef process_totals(data_df):\n    print(\"process totals ...\")\n    data_df['visitNumber'] = np.log1p(data_df['visitNumber'])\n    data_df['totals_hits'] = np.log1p(data_df['totals_hits'])\n    data_df['totals_pageviews'] = np.log1p(data_df['totals_pageviews'].fillna(0))\n    data_df['mean_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('mean')\n    data_df['sum_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('sum')\n    data_df['max_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('max')\n    data_df['min_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('min')\n    data_df['var_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('var')\n    return data_df\n\ndef process_geo_network(data_df):\n    print(\"process geo network ...\")\n    data_df['sum_pageviews_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_pageviews'].transform('sum')\n    data_df['count_pageviews_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_pageviews'].transform('count')\n    data_df['mean_pageviews_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_pageviews'].transform('mean')\n    data_df['sum_hits_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_hits'].transform('sum')\n    data_df['count_hits_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_hits'].transform('count')\n    data_df['mean_hits_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_hits'].transform('mean')\n    return data_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"983c7fb950f7d84f905caa92c7e185ac606a4cce"},"cell_type":"markdown","source":"## 2.Load data"},{"metadata":{"trusted":true,"_uuid":"9c731e4afbd801d7d02dec3bc2c89e51fcfcacb3"},"cell_type":"code","source":"train_df = read_parse_dataframe('../input/train.csv')\ntrain_df = process_date_time(train_df)\ntest_df = read_parse_dataframe('../input/test.csv')\ntest_df = process_date_time(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52c17f8883b9b7110bad945e35a97f151c2a240b"},"cell_type":"markdown","source":"## 3.Drop columns"},{"metadata":{"trusted":true,"_uuid":"a6f3ed2574e5a59ec8a1c79b92b29c5f11643018"},"cell_type":"code","source":"cols_to_drop = [col for col in train_df.columns if train_df[col].nunique(dropna=False) == 1]\ntrain_df.drop(cols_to_drop, axis=1, inplace=True)\ntest_df.drop([col for col in cols_to_drop if col in test_df.columns], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da68cd6bcd0c4db2f17b06fd53b4811fc86c7d6d"},"cell_type":"markdown","source":"## 4.Only one not null value"},{"metadata":{"trusted":true,"_uuid":"9bef5f78b661bc143142c64149281c78335ebb72"},"cell_type":"code","source":"train_df.drop(['trafficSource_campaignCode'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3fb5242898b37029e704883127cb00155ffa8a19"},"cell_type":"markdown","source":"## 5.Converting columns format"},{"metadata":{"trusted":true,"_uuid":"ba4a170cf043f2462e6a3c1e2764a3556f47f695"},"cell_type":"code","source":"train_df['totals_transactionRevenue'] = train_df['totals_transactionRevenue'].astype(float)\ntrain_df['totals_transactionRevenue'] = train_df['totals_transactionRevenue'].fillna(0)\ntrain_df['totals_transactionRevenue'] = np.log1p(train_df['totals_transactionRevenue'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f54178b203c6fbbce7f069b9ae23e228d4795f86"},"cell_type":"markdown","source":"## 6.Features engineering"},{"metadata":{"trusted":true,"_uuid":"5b2f85181dc69e75d15305ec920a4632a9f4ea45"},"cell_type":"code","source":"train_df = process_format(train_df)\ntrain_df = process_device(train_df)\ntrain_df = process_totals(train_df)\ntrain_df = process_geo_network(train_df)\n\ntest_df = process_format(test_df)\ntest_df = process_device(test_df)\ntest_df = process_totals(test_df)\ntest_df = process_geo_network(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18a192adde48976a7fb0ff1ea4dfc2dcbaf303a0"},"cell_type":"markdown","source":"## 7.Categorical columns Label Encoding"},{"metadata":{"trusted":true,"_uuid":"f7c2c505f19d8a3c6ba929bf839914ab022d5061"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nprint(\"process categorical columns ...\")\nnum_cols = ['month_unique_user_count', 'day_unique_user_count', 'weekday_unique_user_count',\n            'visitNumber', 'totals_hits', 'totals_pageviews', \n            'mean_hits_per_day', 'sum_hits_per_day', 'min_hits_per_day', 'max_hits_per_day', 'var_hits_per_day',\n            'sum_pageviews_per_network_domain', 'count_pageviews_per_network_domain', 'mean_pageviews_per_network_domain',\n            'sum_hits_per_network_domain', 'count_hits_per_network_domain', 'mean_hits_per_network_domain']\n            \nnot_used_cols = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \n        \"visitId\", \"visitStartTime\", 'totals_transactionRevenue', 'trafficSource_referralPath']\ncat_cols = [col for col in train_df.columns if col not in num_cols and col not in not_used_cols]\nfor col in cat_cols:\n    print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e07ce3724c9c6bb66f4839204f54f59c5826bf53"},"cell_type":"markdown","source":"# 8.Data Prepare For Modeling"},{"metadata":{"trusted":true,"_uuid":"00b3c090da16bd8c60d33186345486bcef9244c3"},"cell_type":"code","source":"# Model\nprint(\"prepare model ...\")\ntrain_df = train_df.sort_values('date')\nX = train_df.drop(not_used_cols, axis=1)\ny = train_df['totals_transactionRevenue']\nX_test = test_df.drop([col for col in not_used_cols if col in test_df.columns], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dcc528a15dda0f853516375bd7e2e73aec09ac75"},"cell_type":"code","source":"print(\"Number of unique visitors in train set : \",train_df.fullVisitorId.nunique(), \" out of rows : \",train_df.shape[0])\nprint(\"Number of unique visitors in train set : \",test_df.fullVisitorId.nunique(), \" out of rows : \",test_df.shape[0])\nprint(\"Number of common visitors in train and test set : \",len(set(train_df.fullVisitorId.unique()).intersection(set(test_df.fullVisitorId.unique())) ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a769623b979e4d931f915385f26348d3d1db68b7"},"cell_type":"code","source":"plt.figure(figsize=(20,20))\nsns.heatmap(train_df.corr(),annot=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ce519ee7bca13eba79f51d585035ebce144b355"},"cell_type":"markdown","source":"### Correlation coefficient of the variables"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"a73a146792aa9fc26ef853ae38beae454f6ec59f"},"cell_type":"code","source":"train_df_new = train_df\n\n#print(train_df_new)\n\n#Now let us look at the correlation coefficient of each of these variables #\nx_cols = [col for col in train_df_new.columns if col not in ['totals_transactionRevenue'] if train_df_new[col].dtype=='float64']\n\nlabels = []\nvalues = []\nfor col in x_cols:\n    labels.append(col)\n    values.append(np.corrcoef(train_df_new[col].values, train_df_new.totals_transactionRevenue.values)[0,1])\ncorr_df = pd.DataFrame({'col_labels':labels, 'corr_values':values})\ncorr_df = corr_df.sort_values(by='corr_values')\n    \nind = np.arange(len(labels))\nwidth = 0.9\nfig, ax = plt.subplots(figsize=(12,40))\nrects = ax.barh(ind, np.array(corr_df.corr_values.values), color='y')\nax.set_yticks(ind)\nax.set_yticklabels(corr_df.col_labels.values, rotation='horizontal')\nax.set_xlabel(\"Correlation coefficient\")\nax.set_title(\"Correlation coefficient of the variables\")\n#autolabel(rects)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b64d48cf54da544ff835becfc2d2e84397c50327"},"cell_type":"code","source":"# corr_df_sel = corr_df.ix[(corr_df['corr_values']>0.05)]\n# corr_df_sel['col_labels'].values\ncol_x = ['sum_pageviews_per_network_domain', 'sum_hits_per_network_domain',\n       'mean_hits_per_network_domain',\n       'mean_pageviews_per_network_domain', 'totals_hits',\n       'totals_pageviews']\n\nfor i in col_x:\n    X[i+'_square'] =  X[i] ** 2\n    X_test[i+'_square'] = X_test[i] ** 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f8f8d5dfda8432f4a2ca6f487a50929dfce66c8"},"cell_type":"code","source":"X.shape,X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"961119f995a317c8b937571890cf30ac66bd1969"},"cell_type":"markdown","source":"**Columns with constant values: **\n\nLooks like there are quite a few features with constant value in the train set. Let us get the list of these features."},{"metadata":{"trusted":true,"_uuid":"5eed694270d93fd7c2bc0a16a620fd329d59c2c1"},"cell_type":"code","source":"[c for c in train_df.columns if train_df[c].nunique()==1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d008bbab982f36a922928f1dbe079202c380df33"},"cell_type":"code","source":"print(\"Variables not in test but in train : \", set(train_df.columns).difference(set(test_df.columns)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08497a93e57b8d6c27145a99d27ed0a85183cfa9"},"cell_type":"code","source":"from sklearn.model_selection import KFold\n# import lightgbm as lgb\nimport xgboost as xgb\nfrom catboost import CatBoostRegressor\n\n# params = {\"objective\" : \"regression\", \"metric\" : \"rmse\", \"max_depth\": 8, \"min_child_samples\": 20, \"reg_alpha\": 1, \"reg_lambda\": 1,'min_data_in_leaf': 20,\n#         \"num_leaves\" : 350, \"learning_rate\" : 0.01, \"subsample\" : 0.8, \"colsample_bytree\" : 0.85, \"subsample_freq \": 5, \"feature_fraction\":0.95,\n#          \"bagging_freq\":1,\"bagging_fraction\":0.85 ,\"bagging_seed\": 32,\"lambda_l1\": 0.89}\nparams = {\"objective\" : \"regression\", \"metric\" : \"rmse\",\"num_leaves\" : 30, \"learning_rate\" : 0.01, \"bagging_fraction\" : 0.9,\"feature_fraction\" : 0.3, \"bagging_seed\" : 0}\nfolds = KFold(n_splits=10, shuffle=True, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a20a865679a0c1d9ba6226ab25bd63562185520c"},"cell_type":"markdown","source":"## 9.Light GBM Model Training\n\n9. [**Light GBM Model Training**](#9.Light-GBM-Model-Training)  \n9.1.[**Light GBM**](#9.1.Light-GBM)    \n9.2.[**Feature Importance**](#9.2.Feature-Importance)    \n9.3.[**LightGBM Submission**](#9.3.LightGBM-Submission)  "},{"metadata":{"_uuid":"1cd082e4aae294f8ab0ffac931e6163032577552"},"cell_type":"markdown","source":"### 9.1.Light GBM"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"11c4ec0cdf3168bd3e0c0bba235624a097f8f77a"},"cell_type":"code","source":"NUM_ROUNDS = 20000\nVERBOSE_EVAL = 1000\nSTOP_ROUNDS = 100\nprint(\"lgb_model ...\")\nlgb_model = lgb.LGBMRegressor(**params, n_estimators = NUM_ROUNDS, nthread = 4, n_jobs = -1)\n\nprediction = np.zeros(test_df.shape[0])\n\nfor fold_n, (train_index, test_index) in enumerate(folds.split(X)):\n    print('Fold:', fold_n)\n    X_train, X_valid = X.iloc[train_index], X.iloc[test_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[test_index]\n    \n    lgb_model.fit(X_train, y_train, \n            eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric='rmse',\n            verbose=VERBOSE_EVAL, early_stopping_rounds=STOP_ROUNDS)\n    \n    y_pred = lgb_model.predict(X_test, num_iteration=lgb_model.best_iteration_)\n    prediction += y_pred\nprediction /= 10","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f6e46fbabab15c6ddcd5fe41937a0f276594279"},"cell_type":"markdown","source":"### 9.2.Feature Importance"},{"metadata":{"trusted":true,"_uuid":"7a0da4195ac3e9683c29579b7700a21eb58fdcb8"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(lgb_model, max_num_features=50, height=0.8, ax=ax)\nlgb_features = lgb_model.feature_importances_\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8855da82b1933a98d9e7d80c527b0219d00a433c"},"cell_type":"markdown","source":"### 9.3.LightGBM Submission"},{"metadata":{"trusted":true,"_uuid":"4f3d891613226accac6b7147f686c6b5391ae6d2"},"cell_type":"code","source":"# Submission\nprint(\"prepare submission ...\")\nsubmission = test_df[['fullVisitorId']].copy()\nsubmission.loc[:, 'PredictedLogRevenue'] = prediction\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].apply(lambda x : 0.0 if x < 0 else x)\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].fillna(0.0)\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test.to_csv('lgb.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23e8205d2a55d4137206c84d76b90ef1fab1e752"},"cell_type":"markdown","source":"# 10. XGB Model Training"},{"metadata":{"_uuid":"51965c4f9b9564572fc9ef6b385973ccec65158c"},"cell_type":"markdown","source":"### 10.1 XGB Model\n\n10. [**XGB Model Training**](#10.-XGB-Model-Training)  \n10.1.[**XGB Model**](#10.1-XGB-Model)    \n10.2.[**XGB Feature Importance**](#10.2.XGB-Feature-Importance)    \n10.3.[**XGB Submission**](#10.3.XGB-Submission)   \n"},{"metadata":{"trusted":true,"_uuid":"de9153d4b2c464a6c020578bff10eb5c8fad7f6b"},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import train_test_split\nfolds1 = KFold(n_splits=10, random_state=42)\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.15, random_state=1)\n\nprint(f\"Train shape: {X_train.shape}\")\nprint(f\"Validation shape: {X_val.shape}\")\nprint(f\"Test (submit) shape: {X_test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b1ad86a6f1cfd366f5713129a31f70bba19c4df"},"cell_type":"code","source":"print(\"model ...\")\nparams1 = {\n            'objective': 'reg:linear',\n            'eval_metric': 'rmse',\n            'eta': 0.001,\n            'max_depth': 7,\n            'gamma': 1.3250360141843498, \n            'min_child_weight': 13.0958516960316, \n            'max_delta_step': 8.88492863796954, \n            'subsample': 0.9864199446951019, \n            'colsample_bytree': 0.8376539278239742,\n            'subsample': 0.6,\n            'colsample_bytree': 0.8,\n            'alpha':0.001,\n            \"num_leaves\" : 40,\n            'random_state': 42,\n            'silent': True,\n            }\n\n\n#  params = {'max_depth': 7, \n#            'gamma': 1.3250360141843498, \n#            'min_child_weight': 13.0958516960316, \n#            'max_delta_step': 8.88492863796954, \n#            'subsample': 0.9864199446951019, \n#            'colsample_bytree': 0.8376539278239742}\n\nprediction = np.zeros(test_df.shape[0])\n\nfor fold_n1, (train_index, test_index) in enumerate(folds1.split(X)):\n    print('Fold:', fold_n1)\n    xgb_train_data = xgb.DMatrix(X_train, y_train)\n    xgb_val_data = xgb.DMatrix(X_val, y_val)\n    xgb_submit_data = xgb.DMatrix(X_test)\n\n    xgb_model = xgb.train(params1, xgb_train_data, \n                      # Note: I disabled XGB to make the notebook run faster\n                      # Set to 2000 to obtain the results reported in Conclusion\n                      num_boost_round=1000, \n                      evals= [(xgb_train_data, 'train'), (xgb_val_data, 'valid')],\n                      early_stopping_rounds=100, \n                      verbose_eval=500\n                     )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7fa4ac5bca213b12ba0c0f7f6071f12255c38c0"},"cell_type":"code","source":"y_pred = xgb_model.predict(xgb_submit_data, ntree_limit=model.best_ntree_limit)\nprediction += y_pred\nprediction /= 10","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0ae81aab904dabe8b9c0ba1eb1100c7c8d3c7e2"},"cell_type":"markdown","source":"### 10.2.XGB Feature Importance"},{"metadata":{"trusted":true,"_uuid":"02a1b60fe785ffcd19686ea4f4ecfea4341769ce"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nxgb_model.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nxgb_model.feature_importance_\nax.grid(False)\nplt.title(\"XGB - Feature Importance\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dbf3b34e4d4822056290966165392112fadb662d"},"cell_type":"code","source":"xgb.plot_tree(model,num_trees=0, rankdir='LR')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"7cbb6ec9a84c87eaa88ba434fa5d6919c207ee68"},"cell_type":"code","source":"xgb.to_graphviz(model, num_trees=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd0f0e31ff1f1093fbf6e61f28fbadc843255a9e"},"cell_type":"markdown","source":"### 10.3.XGB Submission"},{"metadata":{"trusted":true,"_uuid":"284bc31d88079cbb74613b44cfeb6879799e51ce"},"cell_type":"code","source":"# Submission\nprint(\"prepare submission ...\")\nsubmission = test_df[['fullVisitorId']].copy()\nsubmission.loc[:, 'PredictedLogRevenue'] = prediction\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].apply(lambda x : 0.0 if x < 0 else x)\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].fillna(0.0)\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test.to_csv('xgb.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed53e783d44af871c5007b9504249d7759076b63"},"cell_type":"markdown","source":"# 11. Catboost Model Training\n\n11. [**Catboost Model Training**](#11.-Catboost-Model-Training)  \n11.1.[**Catboost Model**](#11.1-Catboost-Model)    \n11.2.[**Catboost Feature Importance**](#11.2.Catboost-Feature-Importance)    \n11.3.[**Catboost Submission**](#11.3.Catboost-Submission)  \n"},{"metadata":{"trusted":true,"_uuid":"51f6c78f7e8dd3176919982c2ada2d42ee9f6cee"},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import train_test_split\nfolds2 = KFold(n_splits=10, random_state=42)\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.15, random_state=1)\n\nprint(f\"Train shape: {X_train.shape}\")\nprint(f\"Validation shape: {X_val.shape}\")\nprint(f\"Test (submit) shape: {X_test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4c5144ec02020c7635e445c5cc30919b20cbc5b6"},"cell_type":"markdown","source":"### 11.1 Catboost Model"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"5590ef305ec4a41ae73540a3cccf18b292256938"},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\ndef rmse(y_true, y_pred):\n    return round(np.sqrt(mean_squared_error(y_true, y_pred)), 5)\nprint(\"model ...\")\n\nprediction = np.zeros(test_df.shape[0])\nmodel = CatBoostRegressor(iterations=1000,\n                             learning_rate=0.05,\n                             depth=10,\n                             eval_metric='RMSE',\n                             random_seed = 42,\n                             bagging_temperature = 0.2,\n                             od_type='Iter',\n                             metric_period = 50,\n                             od_wait=20)\n\nfor fold_n2, (train_index, test_index) in enumerate(folds2.split(X)):\n    print('Fold:', fold_n2)\n    model.fit(X_train, y_train,eval_set=(X_val, y_val),use_best_model=True,verbose=True)\n    y_pred_train = model.predict(X_train)\n    y_pred_val = model.predict(X_val)\n    y_pred_submit = model.predict(X_test)\n    print(f\"CatB: RMSE val: {rmse(y_val, y_pred_val)}  - RMSE train: {rmse(y_train, y_pred_train)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7627a56aec879f7bd1465647b3e8cb7907489691"},"cell_type":"code","source":"y_pred_submit = model.predict(X_test)\nprediction += y_pred_submit\nprediction /= 10","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7a7310f4fe7d61a13065d3e861676157634c79d"},"cell_type":"markdown","source":"### 11.2.Catboost Feature Importance"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"2c21ffdbd30306aa2aeac97ce5ac43e5e2991cc2"},"cell_type":"code","source":"feat_imp = pd.DataFrame({'importance':model.feature_importances_})    \nfeat_imp['feature'] = X_train.columns\nfeat_imp.sort_values(by='importance', ascending=False, inplace=True)\nfeat_imp.sort_values(by='importance', inplace=True)\nfeat_imp = feat_imp.set_index('feature', drop=True)\nfeat_imp.plot.barh(title='Cat Boost Feature Importance', figsize = (20,40), fontsize=20)\nplt.xlabel('Feature Importance Score')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee14fc676f52fd81a3836b24ce337ac37b70a74c"},"cell_type":"markdown","source":"### 11.3.Catboost Submission  "},{"metadata":{"trusted":true,"_uuid":"52538e1892a636777d15970c5d4fe679a77ca84f"},"cell_type":"code","source":"# Submission\nprint(\"prepare submission ...\")\nsubmission = test_df[['fullVisitorId']].copy()\nsubmission.loc[:, 'PredictedLogRevenue'] = prediction\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].apply(lambda x : 0.0 if x < 0 else x)\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].fillna(0.0)\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test.to_csv('catb.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b63b89d0817657871a6bbff02213a66ec243bbc0"},"cell_type":"markdown","source":"# 12. Ensemble"},{"metadata":{"trusted":true,"_uuid":"c28c6e9e15a7948ff7e87b98e4a3319984b3b4a9"},"cell_type":"code","source":"# Note: I disabled XGB to make the notebook run faster, it was 70-25-5 before\nlgb = pd.read_csv(\"lgb.csv\")\nxgb = pd.read_csv(\"xgb.csv\")\ncatb = pd.read_csv(\"catb.csv\")\nsubmission = pd.read_csv(\"../input/sample_submission.csv\")\n\nsubmission[\"PredictedLogRevenue\"] = lgb[\"PredictedLogRevenue\"] * 0.6 + catb[\"PredictedLogRevenue\"] * 0.3 + xgb[\"PredictedLogRevenue\"] * 0.1\n\nsubmission.to_csv(\"Mix_model.csv\", index = False)\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}