{"cells":[{"metadata":{"_uuid":"e3029a60c0db45c1e0329a276620f7e4985a409c"},"cell_type":"markdown","source":"<h2>1. Introduction</h2>\n\nThere are 34 usefull features in this dataset, of these 12 are about traffic source. Most missing values are also from traffic source, but there is a pattern on the data as we shall see. In this notebook I will try to explain how traffic source features are related and plot their values. If you find any wrong information please leave a comment.\n\nFor a complete exploration on this dataset: [Complete exploratory analysis](https://www.kaggle.com/jsaguiar/complete-exploratory-analysis-all-columns)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\n# Seaborn and matplotlib\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n# Plotly\nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\ninit_notebook_mode(connected=True)\n\npath = \"../input/python-quick-start-clean-and-pickle-data\"\nsource_cols = ['trafficSource_adContent', 'trafficSource_adwordsClickInfo.adNetworkType',\n               'trafficSource_adwordsClickInfo.gclId', 'trafficSource_adwordsClickInfo.isVideoAd', \n               'trafficSource_adwordsClickInfo.page', 'trafficSource_adwordsClickInfo.slot', \n               'trafficSource_campaign', 'trafficSource_isTrueDirect',\n               'trafficSource_keyword', 'trafficSource_medium', 'trafficSource_referralPath', 'trafficSource_source']","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_pickle(os.path.join(path, 'train_clean.pkl'))\ntest = pd.read_pickle(os.path.join(path, 'test_clean.pkl'))\ntrain[source_cols].head()    # Head of traffic source columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee51d1dac7ea4eec7bf969c741588b7eadf8626e"},"cell_type":"markdown","source":"<h2>2. Features</h2>"},{"metadata":{"_uuid":"22f64fa9bad131098bc670a2bef950abe2cbada8"},"cell_type":"markdown","source":"<h3>Ads</h3>\n\nThere are six features related to Ads. The first one is adContent and the other five are informations about AdWords:\n* adContent\n\n<b>AdWords</b>\n* adNetworkType\n* gclId\n> Gclid is a globally unique tracking parameter (Google Click Identifier) used by Google to pass information back and forth between Google AdWords and Google Analytics. If you enable URL auto tagging in Google AdWords, Google will append a unique ?gclid parameter on your destination URLs at run-time. Because it is a redirect, you won't see any gclid parameters on your ad words text ad destination url's, but it will show up in your Web server log files. Auto tagging was introduced in 2004 and is on by default in any Google AdWords accounts. \nThe gclid parameter enables data sharing between Google Analytics and Google AdWords. For example, the \"Traffic Sources\" feature in Google Analytics to segment your campaigns based on ad text and other PPC-related campaign metrics and dimensions, relies on gclid.\n\n* isVideoAd: If the Ad is a video (boolean)\n\n* page\t\n* slot"},{"metadata":{"_uuid":"3fb06c6c2b668d3375beca18664f1f1d6c214c5d"},"cell_type":"markdown","source":"<h3>Other features</h3>\n\nThere are six more features:\n* campaign\n* isTrueDirect\n* keyword\n* medium\n* referralPath\n* source"},{"metadata":{"_uuid":"dcb8978bdfab3a1bad31f6f9ac43a929b8bdf225"},"cell_type":"markdown","source":"<h2>3. medium</h2>\n\nThe most important feature to understand traffic source is medium, which tells how did the user arrived at the store."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3d00e2f701b8ac2df9765326252f9746ec9d3399"},"cell_type":"code","source":"def plotbar(df, col, title, top=None):\n    frame = pd.DataFrame()\n    frame['totals_transactionRevenue'] = df['totals_transactionRevenue'].copy()\n    frame[col] = df[col].fillna('missing')\n    # Percentage of revenue\n    tmp_rev = frame.groupby(col)['totals_transactionRevenue'].sum().to_frame().reset_index()\n    tmp_rev = tmp_rev.sort_values('totals_transactionRevenue', ascending=False)\n    tmp_rev = tmp_rev.rename({'totals_transactionRevenue': 'Revenue percentage'},axis=1)\n    tmp_rev['Revenue percentage'] = 100*tmp_rev['Revenue percentage']/df['totals_transactionRevenue'].sum()\n    # Percentage of visits\n    tmp = frame[col].value_counts().to_frame().reset_index()\n    tmp.sort_values(col, ascending=False)\n    tmp = tmp.rename({'index': col, col: 'Percentage of Visits'},axis=1)\n    tmp['Percentage of Visits'] = 100*tmp['Percentage of Visits']/len(df)\n    tmp = pd.merge(tmp, tmp_rev, on=col, how='left')\n    if top:\n        tmp = tmp.head(top)\n    # Barplot\n    trace1 = go.Bar(x=tmp[col], y=tmp['Percentage of Visits'],\n                    name='Visits', marker=dict(color='rgb(55, 83, 109)'))\n    trace2 = go.Bar(x=tmp[col], y=tmp['Revenue percentage'],\n                    name='Revenue', marker=dict(color='rgb(26, 118, 255)'))\n\n    layout = go.Layout(\n        barmode='group',\n        title=title,\n    )\n    \n    layout = go.Layout(\n        title=title,\n        xaxis=dict(tickfont=dict(size=14, color='rgb(107, 107, 107)')),\n        yaxis=dict(\n            title='Percentage',\n            titlefont=dict(size=16, color='rgb(107, 107, 107)'),\n            tickfont=dict(size=14, color='rgb(107, 107, 107)')\n        ),\n        legend=dict(x=0.95, y=1.0, bgcolor='rgba(255, 255, 255, 0)',\n                    bordercolor='rgba(255, 255, 255, 0)'),\n        barmode='group',\n        bargap=0.15,\n        bargroupgap=0.1\n    )\n    \n    fig = go.Figure(data=[trace1, trace2], layout=layout)\n    iplot(fig)\n    \nplotbar(train, 'trafficSource_medium', 'Train set - visits and revenue by medium')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c81f866be520b755fa867de69b893a370e4bbd9"},"cell_type":"markdown","source":"There are seven labels (and no missing values) for this feature. We will go through all values and relate to the other 11 columns:"},{"metadata":{"_uuid":"0987ccc8f9625f15b74f5614d566de3b64eeab1d"},"cell_type":"markdown","source":"<h3> Organic</h3>\n\nOrganic refers to searchs performed in search mechanisms like Google and Baidu. Almost all searchs are from google. The plot below shows the percentage of visits and revenue w.r.t. organic searches for each source. \n\nNote: the percentage is relative to the total revenue/visits for organic search, not for the total."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"af4a45370023b32b7a820cc045f617220a7737d8"},"cell_type":"code","source":"plotbar(train[train['trafficSource_medium'] == 'organic'], 'trafficSource_source', 'Visits and revenue by source for Organic search')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40a91a5d3bd81d95fe5006495b461604cd139972"},"cell_type":"markdown","source":"This category:\n* Has no <b>referralPath</b> (missing)\n* Has no <b>Ad variables</b> (missing)\n* All <b>Campaigns</b> are '(not set)'\n* channelGrouping is 'Organic Search'\n\nAs you can see here:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dc034a5c75c5e3857c5bbb6ccf00146ea017c7e6"},"cell_type":"code","source":"def print_value_counts(category):\n    cols = ['trafficSource_adContent', 'trafficSource_adwordsClickInfo.adNetworkType',\n           'trafficSource_adwordsClickInfo.gclId', 'trafficSource_adwordsClickInfo.isVideoAd',\n           'trafficSource_adwordsClickInfo.page', 'trafficSource_adwordsClickInfo.slot',\n           'trafficSource_referralPath', 'trafficSource_campaign']\n    for c in cols:\n        nunique = train[train['trafficSource_medium'] == category][c].nunique()\n        if nunique < 5:\n            print(train[train['trafficSource_medium'] == category][c].value_counts(dropna=False))\n        else:\n            print(train[train['trafficSource_medium'] == category][c].describe())\n            \nprint_value_counts('organic')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbfcac78e80b9523da888cf407e4a4eaf5a14bd0"},"cell_type":"markdown","source":"<h3>Referral</h3>\n\nUsers that clicked in a link or were redirected from another webpage. The feature trafficSource_referralPath is exclusive for this category. It's interesting that youtube is responsible for more than 60% of the visits coming from referral, but 93% of the revenue is from mail.googleplex.com.\n\nThe plot below shows the top 8 sources for referral. As before, percentages are based on the totals for this category."},{"metadata":{"trusted":true,"_uuid":"ce607ec99aabf2d9f7dcec0da68c201f0f68948c"},"cell_type":"code","source":"plotbar(train[train['trafficSource_medium'] == 'referral'], 'trafficSource_source', 'Visits and revenue by source for Referral', top=8)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9b34626815871c66003735eb393ab1019782dd4"},"cell_type":"markdown","source":"This category:\n* Has all <b>referral paths</b> that are not missing\n* Has no <b>ad variables</b> (all missing)\n* All <b>campaigns</b> are '(not set)'\n* channelGrouping is 'Referral' or 'Social'"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ac1729a9397072f989a44798cb48c9555954c382"},"cell_type":"code","source":"print_value_counts('referral')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6430f68c8824399b35cf0e3347f9ed73ea81f962"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"a75a3a3c7188b92196a57e03fb530e297a2f123d"},"cell_type":"markdown","source":"The feature channelGrouping, which is not included in traffic source, can have two values for referral medium. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"31a61399082ea7ce12abed4365a402399f8298ad"},"cell_type":"code","source":"plotbar(train[train['trafficSource_medium'] == 'referral'], 'channelGrouping', 'Visits and revenue by channel for REFERRAL MEDIUM')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ce677dc797d19c2ab613840625d446f873d5213"},"cell_type":"markdown","source":"<h3>cpc</h3>\n\nWhen channel is 'Paid Search', the medium is cpc and it's basically google paid search results:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8c0a26b5dd973d2a1254325fb45301f1a62158a4"},"cell_type":"code","source":"plotbar(train[train['trafficSource_medium'] == 'cpc'], 'trafficSource_source', 'Visits and revenue by source for CPC', top=None)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"860b4320de1d9457dae56176a4a895f118a9bb64"},"cell_type":"markdown","source":"Details:\n* <b>Ad Network</b> is mostly 'Google Search'\n* Ad features are usually not missing\n* referral path is always missing\n* Campaign has very few missing values"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7c9a7e2b9055b30bb2ddb63abcac488df511c7cc"},"cell_type":"code","source":"print_value_counts('cpc')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8372eabd7b9efd543cade91ef5b14f1f9d89e3e"},"cell_type":"markdown","source":"There are 8 possible campaigns for paid searches:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1c8fee53627f128723330d83472eafb9247c0ec3"},"cell_type":"code","source":"plotbar(train[train['trafficSource_medium'] == 'cpc'], 'trafficSource_campaign', 'Visits and revenue by campaign for CPC')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ca20f4f893945f9f8c197406aac847477cb08c9"},"cell_type":"markdown","source":"<h3>cpm</h3>\n\nPossible sources for CPM:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d7e582312ee210bf09d2bbed002f0162c6d55b7c"},"cell_type":"code","source":"plotbar(train[train['trafficSource_medium'] == 'cpm'], 'trafficSource_source', 'Visits and revenue by source for CPM')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7737d9af14db956749f38c036ad9c824f3aa0ca5"},"cell_type":"markdown","source":"I don't know what this category means, but we can conclude that in the train set:\n* Most sources are dfa\n* Channel grouping is always 'Display'\n* Ad variables and referral path are all missing\n* Campaign is always (not set)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"aeabf1a805042d841174fbde62c612b898c3edb5"},"cell_type":"code","source":"print_value_counts('cpm')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"774edcf582cf7ae2dc94dcee0819b9942c587cf2"},"cell_type":"markdown","source":"<h3>Affiliate</h3>\n\n* All traffic sources are 'Partners'\n* Channel grouping is always 'Affiliates'\n* All Campaigns are 'Data Share Promo'\n* Ad variables are missing"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0fceca868a00b2c3fccb1ec68a4b668be3e1584a"},"cell_type":"code","source":"#print(train[train['trafficSource_medium'] == 'affiliate']['trafficSource_campaign'].value_counts())\nprint_value_counts('affiliate')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60e5664dd9feba576fbb727d65b1e0545e77dc75"},"cell_type":"markdown","source":"<h3>(none)</h3>\n\nMedium (none) are direct acesses to the store\n* traffic source is always '(direct)'\n* channel grouping is always 'Direct'\n* Ad features are missing\n* Campaign is missing\n* referral path is '(not set)'"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b6cd6081a94916e27d656a92fe53672350f565c1"},"cell_type":"code","source":"print_value_counts('(none)')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"814ab693c3bbce28dabc53df6c95f67a3b0f569e"},"cell_type":"markdown","source":"<h3>4. Work in progress...</h3>"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}