{"cells":[{"metadata":{"_uuid":"cb70a6fe4f7893aa335160d238dbaf82f7364db2"},"cell_type":"markdown","source":"

Introduction

\n\nThis is a simple EDA for the updated data (v2 files) with focus on the new columns. I'm using data from [this script](https://www.kaggle.com/jsaguiar/parse-json-v2-without-hits-column) and therefore the hits column is not included."},{"metadata":{"trusted":true,"_uuid":"82aebaa4df737bcdb114e5817713d5c96f126de8","_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n# Seaborn and matplotlib\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n# Plotly\nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\ninit_notebook_mode(connected=True)\n\n# Read data from kernel (see above)\npath = \"../input/parse-json-v2-without-hits-column/\"\ntrain = pd.read_pickle(path + 'train_v2_clean.pkl')\ntest = pd.read_pickle(path + 'test_v2_clean.pkl')\n\n# Unique visitors\nunique_vis_train = train.fullVisitorId.nunique()\nunique_vis_test = test.fullVisitorId.nunique()\nprint(\"Unique fullVisitorId - train: {}, test: {}\".format(unique_vis_train, unique_vis_test))\n\n# Print shapes and first 5 rows\nprint(\"Train shape: {}, test shape: {}\".format(train.shape, test.shape))\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0cc62655993a3fcb819f4596059fcdc92f2ef96c"},"cell_type":"markdown","source":"

1. Revenue and date

"},{"metadata":{"_uuid":"d974277dcc0b9f4f2c63f1ea440d06faa8e9b0d8"},"cell_type":"markdown","source":"

Transaction Revenue

\n\nIn this new version we have two transaction revenue columns in totals:\n\n> totalTransactionRevenue: Total transaction revenue, expressed as the value passed to Analytics multiplied by 10^6 (e.g., 2.40 would be given as 2400000).\n\n> transactionRevenue: This field is deprecated. Use \"totals.totalTransactionRevenue\" instead (see above)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"df0ef37b5a6a6dae68ce7ad82d6ece818de3d8a5"},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Train logn scale - transactionsRevenue vs totalTransactionRevenue\")\nfor col in ['totals_totalTransactionRevenue', 'totals_transactionRevenue']:\n train[col] = train[col].astype('float64')\n revenue = train[train[col] > 0][col].dropna()\n ax1 = sns.kdeplot(np.log(revenue))\n \nplt.figure(figsize=(10,4))\nplt.title(\"Test logn scale - transactionsRevenue vs totalTransactionRevenue\")\nfor col in ['totals_totalTransactionRevenue', 'totals_transactionRevenue']:\n test[col] = test[col].astype('float64')\n revenue = test[test[col] > 0][col].dropna()\n ax1 = sns.kdeplot(np.log(revenue))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cae5fd4c08a6ea78600debb8c8100de2250cef28"},"cell_type":"markdown","source":"

Timeseries

\n\nTrain data: Aug 01, 2016 to April 30, 2018 (21 months)\n\nTest data: May 01, 2018 to Oct 15, 2018 (5.5 months)"},{"metadata":{"trusted":true,"_uuid":"00c07a8375db08b453006c5d7fb60eec6198ef12","_kg_hide-input":true},"cell_type":"code","source":"# Revenue by time\ntrain_date_sum = train.groupby('date')['totals_transactionRevenue'].sum().to_frame().reset_index()\ntest_date_sum = test.groupby('date')['totals_transactionRevenue'].sum().to_frame().reset_index()\n# Plot\ntrace_train = go.Scatter(x = pd.to_datetime(train_date_sum.date.astype(str)),\n y=train_date_sum['totals_transactionRevenue'].apply(lambda x: np.log(x)),\n opacity=0.8, name='Train')\n\ntrace_test = go.Scatter(x = pd.to_datetime(test_date_sum.date.astype(str)),\n y=test_date_sum['totals_transactionRevenue'].apply(lambda x: np.log(x)),\n opacity=0.8, name='Test')\nlayout = dict(\n title= \"Log transactionRevenue by date\",\n xaxis=dict(rangeslider=dict(visible=True), type='date')\n)\nfig = dict(data= [trace_train, trace_test], layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c1f431de0490b4e0b223812d6f34d6a7886c8d8d"},"cell_type":"code","source":"def train_test_distribution(col, dtype='float64'):\n \"\"\"Plot a single numerical column distribution in linear and log scale.\"\"\"\n fig, axis = plt.subplots(1, 2, figsize=(12,4))\n axis[0].set_title(\"Linear scale\")\n axis[1].set_title(\"Log scale\")\n \n train[col], test[col] = train[col].astype(dtype), test[col].astype(dtype)\n ax1 = sns.kdeplot(train[col].dropna(), label='train', ax=axis[0])\n ax2 = sns.kdeplot(test[col].dropna(), label='test', ax=axis[0])\n ax3 = sns.kdeplot(np.log(train[col].dropna()), label='train', ax=axis[1])\n ax4 = sns.kdeplot(np.log(test[col].dropna()), label='test', ax=axis[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"880cc8c1770247d1c2ad9c15baebb58db0e600af"},"cell_type":"markdown","source":"

2. A few more new columns..

\n\nI'm listing only the new columns since there are many kernels for the others.\n\nNote: I'm not including Hits JSON data"},{"metadata":{"_uuid":"e7f3be9a8b0d55103f24fb6a2bf697165c240b9c"},"cell_type":"markdown","source":"

Transaction

\n\n> Total number of ecommerce transactions within the session.\n\nMost sessions have zero or one transaction, but there are sessions with 25 transactions."},{"metadata":{"trusted":true,"_uuid":"b2b2b4ca38fe8b6318a3ab8b5aa8374c304e67d7"},"cell_type":"code","source":"print(train.totals_transactions.value_counts(dropna=False).head())\ntrain_test_distribution('totals_transactions')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec30dfbcc3d084a68dc8d5aed5e5a7b1bf7d7d72"},"cell_type":"markdown","source":"

Session Quality Dim

\n\n> An estimate of how close a particular session was to transacting, ranging from 1 to 100, calculated for each session. A value closer to 1 indicates a low session quality, or far from transacting, while a value closer to 100 indicates a high session quality, or very close to transacting. A value of 0 indicates that Session Quality is not calculated for the selected time range."},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"a5ee13002b9e006350a635eccba90d090f9af8e2"},"cell_type":"code","source":"train_test_distribution('totals_sessionQualityDim')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cacc84dd3a72c992221f39596ca5e2b3f3ea517c"},"cell_type":"markdown","source":"

Time On Site

\n\n> Total time of the session expressed in seconds."},{"metadata":{"trusted":true,"_uuid":"0cbed3b78cc6fbfcee78ad374994b97de5d41431"},"cell_type":"code","source":"train_test_distribution('totals_timeOnSite')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"967c56ec2436f9416259e7da8f4b9147421bcab9"},"cell_type":"markdown","source":"

Custom Dimensions value

"},{"metadata":{"trusted":true,"_uuid":"ef6f4fd51f3ea19e39b46e68006fde8b7917a2b6","_kg_hide-input":true},"cell_type":"code","source":"counts = train.customDimensions_value.value_counts(dropna=False).to_frame().reset_index()\nplt.figure(figsize=(8,4))\ng = sns.barplot(x='index', y='customDimensions_value', data=counts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63571c01d6bd91c98a5ef058568c21ceb36dbe98"},"cell_type":"markdown","source":"

3. What are we predicting?

\n\nLet's have a look at what we are predicting after this update:\n\n> you will be predicting the target for ALL users in the posted test set: test_v2.csv, for their transactions in the future time period of December 1st 2018 through January 31st 2019.\n\nSo we have to predict the log revenue for each fullVisitorId in test_v2 for the next two months... quite challenging."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"917e527fa0865e1d7fdd79569a9c855f3ea5a947"},"cell_type":"code","source":"unique_test = test.fullVisitorId.unique()\nprint(\"There are\", len(unique_test), \"unique users (fullVisitorId) in test\")\nrev = test.groupby('fullVisitorId')['totals_transactionRevenue'].sum()\nprint(\"There are\", len(rev[rev > 0]), \"unique users with revenue (greater than 0) in test\")\nprint(\"So only {:.1f}% of users have revenue\".format(100 * len(rev[rev > 0]) / len(unique_test)))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"34233ad34d23c03671c30f5a810c3ef3f402c8c7"},"cell_type":"code","source":"plt.figure(figsize=(8,4))\nplt.title(\"Distribution of revenue per user (only users with revenue > 0)\")\nax = sns.kdeplot(np.log(rev[rev > 0]), label='Log transactionRevenue')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d297fb8316b984829cf5c9971601861482de5a8f"},"cell_type":"markdown","source":"

4. How difficult is it?

\n\nIn this section I'll try to reproduce our problem with past data. So we'll be looking at users at 5.5 months periods and checking if they returned in the next two months (with a 45 days interval)."},{"metadata":{"trusted":true,"_uuid":"dc187ee1849e38f37abe8b5898799cceecefc0b1"},"cell_type":"code","source":"users_period = [\n (20160801, 20170115),\n (20170115, 20170630),\n (20170701, 20171215),\n (20171216, 20180601),\n # Using the same months!\n (20170501, 20171015),\n]\n\npredict_period = [\n (20170301, 20170430),\n (20170715, 20170915),\n (20180201, 20180331),\n (20180715, 20180915),\n # Using the same months!\n (20171201, 20180131),\n]\n\n# Join train and test\ndf = pd.concat([train, test])\n# Save revenue for plot\nrevenues_list = []\n\nfor i in range(5):\n print(\"\\nPeriod\", i+1)\n a, b = users_period[i]\n batch = df[(df.date >= a) & (df.date <= b)]\n batch_visitors = batch.fullVisitorId.unique()\n print(\"There are\", len(batch_visitors), \"visitors in 5.5 months\")\n \n c, d = predict_period[i]\n pred = df[(df.date >= c) & (df.date <= d)]\n pred_visitors = pred.fullVisitorId.unique()\n print(\"There are\", len(pred_visitors), \"visitors in 2 months\")\n # Returning visitors\n same_visitors = np.intersect1d(batch_visitors, pred_visitors)\n print(\"{} visitors returned or {:.2f}%\".format(len(same_visitors), 100*len(same_visitors)/len(batch_visitors)))\n # Returning visitors revenue\n with_rev = pred[(pred.fullVisitorId.isin(same_visitors)) & (pred.totals_transactionRevenue > 0)]\n print(\"And only {} returning visitors have revenue or {:.2f}% from total\".format(len(with_rev), 100*len(with_rev)/len(batch_visitors)))\n print(\"The total revenue for this users is U$$ {:.2f}\".format(with_rev.totals_transactionRevenue.sum()/1000000))\n revenues_list.append(with_rev.copy(deep=True))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"089d72d94a5507c02267e186d9b9fd037e458ed3"},"cell_type":"markdown","source":"This looks like finding a needle in a haystack... Finally, let's have a look at revenue distribution for the returning customers with transactions in the last period:"},{"metadata":{"trusted":true,"_uuid":"41587d6849d23330a6ce31237b44db42eed68c9a"},"cell_type":"code","source":"plt.figure(figsize=(8,4))\nplt.title(\"Distribution of revenue for returning customers\")\nax = sns.kdeplot(np.log(revenues_list[-1].totals_transactionRevenue), label='Log transactionRevenue')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84cbe99608a7a183cf2074e962024dfef9790f55"},"cell_type":"markdown","source":"What do you think is the best approach after the update? Predicting returning customers maybe? Leave a comment with your ideias and upvote if you find usefull."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}