{"cells":[{"metadata":{"_uuid":"5eedb3c713ceebdb4135e231a32b25c831870d6c"},"cell_type":"markdown","source":"[https://github.com/Shravankp/Emplay_kaggle_challenge/blob/master/customer_revenue_prediction.ipynb](http://)\n\nGoogle Analytics Customer Revenue Prediction\n\nTo predict - How much GStore's Revenue from each customers, using EDA and ML models.\n\nGiven in problem description, the 80/20 rule - 80% of revenue comes from only 20% of the customers.So our task is to predict the Gstore's revenue from those customers (using natural log of sum of all transactions per user).\n\nFeatures of the given dataset:\n\nfullVisitorId- A unique identifier for each user of the Google Merchandise Store.\nchannelGrouping - The channel via which the user came to the Store.\ndate - The date on which the user visited the Store.\ndevice - The specifications for the device used to access the Store.\ngeoNetwork - This section contains information about the geography of the user.\nsessionId - A unique identifier for this visit to the store.\nsocialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\ntotals - This section contains aggregate values across the session.\ntrafficSource - This section contains information about the Traffic Source from which the session originated.\nvisitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\nvisitNumber - The session number for this user. If this is the first session, then this is set to 1.\nvisitStartTime - The timestamp (expressed as POSIX time)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32f9b39d400e3c4bf6659a0b67041764beb221a8"},"cell_type":"markdown","source":"Note: Some columns contain serialised JSON as strings which should be deserialised and converted to seperate columns.\n\n"},{"metadata":{"trusted":true,"_uuid":"9c8a8b2bd3ae9e2f6e5da2cbaf7c1f71d59232f0"},"cell_type":"code","source":"def load_df(csv_path='./train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, #json.loads takes in a string and converts to dict or list object.\n                     dtype={'fullVisitorId': 'str'}, #convert id to string\n                     nrows=nrows)\n    \n\n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])  #converts semi-structured json to flat table.\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"179efbadb78155a94a6a5fc96609458592c30fed"},"cell_type":"code","source":"dftrain = load_df(\"../input/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"068b1001b42a9989ef4c2b99144220651dd81aa1"},"cell_type":"code","source":"dftrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6abdfbc9fac28d6a81ccb043c036e04579a91e86"},"cell_type":"code","source":"test_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a86e942e9e56e578fa5dc008916897fe9c3bff1"},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"042eeecc6699e874a1078b24a3655f083e2de46d"},"cell_type":"markdown","source":"Lets perform EDA:\n\nEach row in the dataset is one visit to the store.\n\nLet us look at what all are the numerical variables and categorical variables we have in train dataset."},{"metadata":{"trusted":true,"_uuid":"3b21eceae8a5f41953408b12d7016187a57897d2"},"cell_type":"code","source":"numeric_features = dftrain.select_dtypes(include=[np.number])\nprint(numeric_features.columns)\n\ncategorical_features = dftrain.select_dtypes(include=[np.object])\nprint(categorical_features.columns)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b414f366d111d7e480de86236f9d1b42a5396d69"},"cell_type":"markdown","source":"**Note**: There are some variables in categorical_features which can be converted to fload and can be used as numeric variables, such as ['totals.bounces', 'totals.hits', 'totals.newVisits', 'totals.pageviews', 'totals.transactionRevenue', 'totals.visits'].\n\nNow lets check whether the given dataset conforms 80/20 rule :\n\n* First convert values of totals.transactionRevenue to float\n* Group values according to fullVisitorId ( i.e we are calculating revenue from each customer )\n* Then consider only those customers who have revenue more than zero and find out the ratio."},{"metadata":{"trusted":true,"_uuid":"596886a4d84f72f72125d548c6aa479220564ec2"},"cell_type":"code","source":"dftrain[\"totals.transactionRevenue\"] = dftrain[\"totals.transactionRevenue\"].astype('float')\ngrouped_revenue = dftrain.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\n\nnon_zero_customers = (grouped_revenue[\"totals.transactionRevenue\"]>0).sum()\nprint(\"Number of unique customers with non-zero revenue : \", non_zero_customers, \"and the ratio is : \", non_zero_customers / grouped_revenue.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"adc230ebc7d08259b33c002f285ba31d8c4fb0ea"},"cell_type":"markdown","source":"\nNumber of unique customers with non-zero revenue :  9996 and the ratio is :  0.013996726255903731\nSo the ratio of revenue generating customers to total number of customers is 1.3%. From the above analysis it is confirmed that only 1.3% of the customers bring in revenue to Gstore.\n\n**Data pre-processing:**\n\nSome columns have constant values and missing values. So, Lets examine that and drop those columns from feature set which would make our dataset more useful while training models."},{"metadata":{"trusted":true,"_uuid":"b2502e07cebbd11f346b6f2f150d49db5e5324e2"},"cell_type":"code","source":"# convert the 'date' column values to datetime object\nimport datetime\ndftrain['date'] = dftrain['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ntest_df['date'] = test_df['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d3d99019ebdc4eefd0c46f449a2e5c9eaaec0d5c"},"cell_type":"code","source":"consts = [c for c in dftrain.columns if dftrain[c].nunique(dropna=False)==1 ] #lets include nan values in the count(nunique())\nconsts","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4d847144eca63078f1b54650a04dcf7e1e0e70b"},"cell_type":"markdown","source":"As we know that train set has 55 columns whereas test set has 53 ,lets examine what are those two extra variables in train set and which of those two can be target variable."},{"metadata":{"trusted":true,"_uuid":"8cf2cb6bf87890a430990c29fc4557ac0ac02f13"},"cell_type":"code","source":"set(dftrain.columns).difference(set(test_df.columns))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8fda2a86bb1f2708506d9432f86eeb68255be9fd"},"cell_type":"markdown","source":"As 'trafficSource.campaignCode' is an extra feature not in test set (other than target variable 'totals.transactionRevenue' ).Lets drop this also. Even sessionId can be removed as this is just a unique number for each visit.\n\nSo lets drop consts (columns with constant values declared above), 'sessionId', 'trafficSource.campaignCode'."},{"metadata":{"trusted":true,"_uuid":"ac7bdb2a5e23edc5d2cac9505d563240c983249f"},"cell_type":"code","source":"cols_to_drop = consts + [\"sessionId\"] + [\"trafficSource.campaignCode\"]\ndftrain = dftrain.drop(cols_to_drop, axis=1)\ntest_df = test_df.drop(cols_to_drop[:-1], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc3014692959fe799f299fe4b90f8f1dcfe0dcf0"},"cell_type":"markdown","source":"* Fill in missing values to 0.\n* Now, Identify categorical variables and convert to numbers i.e label encode them.\n* Identify numeric variables and convert them to floats.\n\n**Note:** Do not include IDs and dates to any of the above operations."},{"metadata":{"trusted":true,"_uuid":"ac040c3f0b53fbe7982482f57dee51f10dab5ccb"},"cell_type":"code","source":"#dftrain.head()\n#dftrain.info()\n#dftrain.describe()\n\ndftrain[\"totals.transactionRevenue\"].fillna(0, inplace=True)\ntrain_y = dftrain[\"totals.transactionRevenue\"].values\n\n#identify categorical variables and label encode them.\ncategorical_cols = [\"channelGrouping\", \"device.browser\", \n            \"device.deviceCategory\", \"device.operatingSystem\", \n            \"geoNetwork.city\", \"geoNetwork.continent\", \n            \"geoNetwork.country\", \"geoNetwork.metro\",\n            \"geoNetwork.networkDomain\", \"geoNetwork.region\", \n            \"geoNetwork.subContinent\", \"trafficSource.adContent\", \n            \"trafficSource.adwordsClickInfo.adNetworkType\", \n            \"trafficSource.adwordsClickInfo.gclId\", \n            \"trafficSource.adwordsClickInfo.page\", \n            \"trafficSource.adwordsClickInfo.slot\", \"trafficSource.campaign\",\n            \"trafficSource.keyword\", \"trafficSource.medium\", \n            \"trafficSource.referralPath\", \"trafficSource.source\",\n            'trafficSource.adwordsClickInfo.isVideoAd', 'trafficSource.isTrueDirect']\n\nfor col in categorical_cols:\n    print(col)\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(dftrain[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    dftrain[col] = lbl.transform(list(dftrain[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))\n\n\nnumeric_cols = [\"totals.hits\", \"totals.pageviews\", \"visitNumber\", \"visitStartTime\", 'totals.bounces',  'totals.newVisits']    \nfor col in numeric_cols:\n    dftrain[col] = dftrain[col].astype(float)\n    test_df[col] = test_df[col].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d88a026c12fdf8b8a8fe936195f8d9905d7004dd"},"cell_type":"markdown","source":"As the training set contains data from August 1st 2016 to August 1st 2017. Then take cross-validation set as last three month's data which makes the ratio of train set to cross-val set roughly 7.5 : 2.5 .(This is considering months but not no of examples in trainset , so might not be exactly 7.5 : 2.5)."},{"metadata":{"trusted":true,"_uuid":"57dddb0d63b639a0ebc3bd5933d9a1da7e3d1861"},"cell_type":"code","source":"# Split the train dataset into development and valid based on time \ndev_df = dftrain[dftrain['date']<=datetime.date(2017,5,31)]\nval_df = dftrain[dftrain['date']>datetime.date(2017,5,31)]\n\ndev_y = np.log1p(dev_df[\"totals.transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals.transactionRevenue\"].values)\n\ndev_X = dev_df[categorical_cols + numeric_cols] \nval_X = val_df[categorical_cols + numeric_cols] \ntest_X = test_df[categorical_cols + numeric_cols]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eaacd3313a32b25e08ee6abaccf90506c7c77150"},"cell_type":"markdown","source":"\nChoosing a model to train:\n\nAs we have large dataset (with more columns >30 and rows > 10000)its better to use ensemble learning (gradient boosting models).we'll use LightGBM to train our model and predict revenue from test set."},{"metadata":{"trusted":true,"_uuid":"4e869476d9112ab9e3ce2930b340b4049997676f"},"cell_type":"code","source":"def run_lgb(train_X, train_y, val_X, val_y, test_X):\n    params = {\"objective\" : \"regression\",\"metric\" : \"rmse\", \n            \"subsample\" : 0.9,\"colsample_bytree\" : 0.9,\n            \"num_leaves\" : 31,\"min_child_samples\" : 100,\n            \"learning_rate\" : 0.03,\"bagging_fraction\" : 0.7,\n            \"feature_fraction\" : 0.5,\"bagging_frequency\" : 5,\n            \"bagging_seed\" : 2018,\"verbosity\" : -1}\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    model = lgb.train(params, lgtrain, 1000,early_stopping_rounds=100, valid_sets=[lgval],  verbose_eval=100)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    pred_val_y = model.predict(val_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, pred_val_y\n\n# Training the model #\npred_test, model, pred_val = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cae3f8895bce8533e8b4ea6986b42922a6660833"},"cell_type":"markdown","source":"\nLet us compute the evaluation metric on the validation data.\n\n* Assign zero if predicted value is less than 0.\n* Prepare a validation dataframe. As predicted values are logarithmic values apply np.exp ( we'll convert this to log values again after grouping values acc to 'fullVisitorId' and calculating sum of revenue per user )\n* Apply log on sum for all the transactions per user (or apply sum first on grouped data and then apply log).\n* Calculate rms error."},{"metadata":{"trusted":true,"_uuid":"8af3b782b0c9b8d5f634b33e8d8732562aa394c3"},"cell_type":"code","source":"from sklearn import metrics\n\npred_val[pred_val<0] = 0\nval_pred_df = pd.DataFrame({\"fullVisitorId\":val_df[\"fullVisitorId\"].values})\nval_pred_df[\"transactionRevenue\"] = val_df[\"totals.transactionRevenue\"].values\nval_pred_df[\"PredictedRevenue\"] = np.expm1(pred_val) #exp(x) -1 can also be used but expm1 gives greater precision when converting log\n\nval_pred_df = val_pred_df.groupby(\"fullVisitorId\")[\"transactionRevenue\", \"PredictedRevenue\"].sum().reset_index()\nval_pred_df[\"transactionRevenue\"] = np.log1p(val_pred_df[\"transactionRevenue\"].values)\nval_pred_df[\"PredictedRevenue\"] =  np.log1p(val_pred_df[\"PredictedRevenue\"].values)\n\n#Now apply rms to find out error\nprint(np.sqrt(metrics.mean_squared_error(val_pred_df[\"transactionRevenue\"].values, val_pred_df[\"PredictedRevenue\"].values)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8010be3cb371c124491dc76893c82f25c4ace36"},"cell_type":"code","source":"train_id = dftrain[\"fullVisitorId\"].values\ntest_id = test_df[\"fullVisitorId\"].values   \nsubmit_df = pd.DataFrame({\"fullVisitorId\":test_id})\n\n#Repeat same steps as we did for cross-validation\npred_test[pred_test<0] = 0\nsubmit_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsubmit_df = submit_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsubmit_df[\"PredictedLogRevenue\"] = np.log1p(submit_df[\"PredictedLogRevenue\"])\n\nsubmit_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nprint(submit_df.head())\nsubmit_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f04abac870f55a4c62aba6f21daa6d74883f2076"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}