{"cells":[{"metadata":{"_uuid":"d21ae5d68a0b39c29977b8fb1db129163fb11aa9"},"cell_type":"markdown","source":"<h2>1. Introduction</h2>\n\nIn this competition we are predicting the natural log of the sum of all transactions per user. The dataset contains information about visits to [GStore](https://www.googlemerchandisestore.com/) (Google swag online store), each row is a unique visit and each user has a unique 'fullVisitorId'. There are two data tables for the competition: the train set with 903k rows and the test set with 804k rows. This EDA will go through all columns in the dataset.\n\n<b>Columns and features:</b>\n\nThere are 12 original columns, but four of them have a json object that can be converted:\n\n1. device: 16 new columns\n2.  geoNetwork: 11 new columns\n3. totals: 6 new columns\n4. trafficSource: 14 new columns\n\nAnd therefore we have <b>55 total columns</b>. After droping useless columns (section 1.4) we end up with <b>34 features</b> for this competition, most of them with categorical values."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nfrom scipy.stats import norm\nimport json\nimport datetime\n# Seaborn and matplotlib\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n# Plotly\nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\ninit_notebook_mode(connected=True)\n\n# 'train.csv', 'sample_submission.csv', 'test.csv'\ndef load_df(file_name = 'train.csv', nrows = None):\n    \"\"\" Read csv and convert json columns. Author: Julián Peller. \"\"\"\n    \n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    df = pd.read_csv('../input/{}'.format(file_name),\n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}_{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"07d837847ee5e7d319b5c6f9b350ed6493854dce"},"cell_type":"code","source":"train, test = load_df('train.csv'), load_df('test.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"307b2d4922d070259e66b25a6c5ad14c9865e620"},"cell_type":"markdown","source":"* Train set has 55 columns and the test set has 53 columns\n* Extra columns are: totals_transactionRevenue (our target variable) and trafficSource_campaignCode\n\nNow let's plot the number of rows/visits and unique users for each dataset. \n\n<b>Note</b>: There are 7679 users that are common to both datasets."},{"metadata":{"trusted":true,"_uuid":"53743ec9b48d0d102bffd97698ee46c3199db07e","_kg_hide-input":true},"cell_type":"code","source":"values_name = ['Train rows (visits)', 'Test rows (visits)', 'Train unique users', 'Test unique users']\nvalues = [len(train), len(test), train['fullVisitorId'].nunique(), test['fullVisitorId'].nunique()]\nplt.figure(figsize=(8,4))\nplt.title(\"Basic statistics\")\nax = sns.barplot(x=values_name, y=values, palette='Blues_d')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fa239d5ec219a3e46efd91fb4595e7f6211211b"},"cell_type":"markdown","source":"<h3>1.1 Target variable</h3>\n\nThe 'totals_transactionRevenue' column has the transaction value for each visit. However, our <b>train set has 98.72% of missing values for revenue which we can consider as zero revenue (no purchase) </b>. As stated [in this discussion](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65775#latest-388425) the revenue unit is million USD (10^6).\n\nThe black lines are the closest normal distribution that we can fit to each distribution."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c5d731a79df18504318d5359352607c5adcee7a2"},"cell_type":"code","source":"non_missing = len(train[~train['totals_transactionRevenue'].isnull()])\nnum_visitors = train[~train['totals_transactionRevenue'].isnull()]['fullVisitorId'].nunique()\nprint(\"totals_transactionRevenue has {} non-missing values or {:.3f}% (train set)\"\n      .format(non_missing, 100*non_missing/len(train)))\nprint(\"Only {} unique users have transactions or {:.3f}% (train set)\"\n      .format(num_visitors, num_visitors/train['fullVisitorId'].nunique()))\n# Logn Distplot\nrevenue = train['totals_transactionRevenue'].dropna().astype('float64')\nplt.figure(figsize=(10,4))\nplt.title(\"Natural log Distribution - Transactions revenue\")\nax1 = sns.distplot(np.log(revenue), color=\"#006633\", fit=norm)\n# Log10 Distplot\nplt.figure(figsize=(10,4))\nplt.title(\"Log10 Distribution - Transactions revenue\")\nax1 = sns.distplot(np.log10(revenue), color=\"#006633\", fit=norm)\n\n# Fill missing with 0 and convert to numerical\ntrain['totals_transactionRevenue'] = train['totals_transactionRevenue'].fillna(0).astype('int64')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5cbbecccb23ea2acc5245735951d7da66c8986fc"},"cell_type":"markdown","source":"We have only 11,515 purchases in the training set with the lowest value being 10k and the highest value around 23 billions. Converting to USD we would have the lowest purchase value being 1 cent and the highest 23 million dollars.\n\nLet's look at the sum of transactions for each user:"},{"metadata":{"trusted":true,"_uuid":"0b8d636bd1825d768631c9a024cb6545d8ffd19e","_kg_hide-input":true},"cell_type":"code","source":"g = train[train['totals_transactionRevenue'] > 0][['fullVisitorId', 'totals_transactionRevenue']]\nsum_transactions = g.groupby('fullVisitorId')['totals_transactionRevenue'].sum()\nplt.figure(figsize=(10,4))\nplt.title(\"Log10 Distribution - Sum of transactions per user\")\nax1 = sns.distplot(np.log10(sum_transactions), color=\"#005c99\", fit=norm)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4af70bfb58e7bd08aaaa1fa1b3ce4081997cb281"},"cell_type":"markdown","source":"<b>1.2 Timeseries</b>"},{"metadata":{"trusted":true,"_uuid":"23d4225abddba5052e093eb7037ebb0cf744a9f7","_kg_hide-input":true},"cell_type":"code","source":"def convert_to_datetime(frame):\n    frame['date'] = frame['date'].astype(str)\n    frame['date'] = frame['date'].apply(lambda x : x[:4] + \"-\" + x[4:6] + \"-\" + x[6:])\n    frame['date'] = pd.to_datetime(frame['date'])\n    return frame\n\ntrain = convert_to_datetime(train)\ntest = convert_to_datetime(test)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9e25eb56eeccd8012f4e355b46bde864086fae5c"},"cell_type":"code","source":"# Visits by time train\ntmp_train = train['date'].value_counts().to_frame().reset_index().sort_values('index')\ntmp_train = tmp_train.rename(columns = {\"date\" : \"visits\"}).rename(columns = {\"index\" : \"date\"})\n# Visits by time test\ntmp_test = test['date'].value_counts().to_frame().reset_index().sort_values('index')\ntmp_test = tmp_test.rename(columns = {\"date\" : \"visits\"}).rename(columns = {\"index\" : \"date\"})\n# Plot visits\ntrace1 = go.Scatter(x=tmp_train.date.astype(str), y=tmp_train.visits,\n                    opacity = 0.8, line = dict(color = '#ff751a'), name= 'Train')\ntrace2 = go.Scatter(x=tmp_test.date.astype(str), y=tmp_test.visits,\n                    opacity = 0.8, line = dict(color = '#75a3a3'), name= 'Test')\ntraces = [trace1, trace2]\n\nlayout = dict(\n    title= \"Visits by date\",\n    xaxis=dict(\n        rangeselector=dict(\n            buttons=list([\n                dict(count=1, label='1m', step='month', stepmode='backward'),\n                dict(count=3, label='3m', step='month', stepmode='backward'),\n                dict(count=6, label='6m', step='month', stepmode='backward'),\n                dict(step='all')\n            ])\n        ),\n        rangeslider=dict(visible = True),\n        type='date'\n    )\n)\n\nfig = dict(data= traces, layout=layout)\niplot(fig)\n\n\n# Revenue by time\ntrain_date_sum = train.groupby('date')['totals_transactionRevenue'].sum().to_frame().reset_index()\n# Plot\ntrace_date = go.Scatter(x=train_date_sum.date.astype(str), \n                        y=train_date_sum['totals_transactionRevenue'].apply(lambda x: np.log(x)), opacity = 0.8)\nlayout = dict(\n    title= \"Log Revenue by date\",\n    xaxis=dict(\n        rangeselector=dict(\n            buttons=list([\n                dict(count=1, label='1m', step='month', stepmode='backward'),\n                dict(count=3, label='3m', step='month', stepmode='backward'),\n                dict(count=6, label='6m', step='month', stepmode='backward'),\n                dict(step='all')\n            ])\n        ),\n        rangeslider=dict(visible = True),\n        type='date'\n    )\n)\n\nfig = dict(data= [trace_date], layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64c916f96623c383fe44e6cdadc59318c0e8ed24"},"cell_type":"markdown","source":"<h3> 1.3 Missing values</h3> \n\nMost missing values are in trafficSource and a few in totals."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3b84b689544d810289080247d9545270024c1dd3"},"cell_type":"code","source":"def missing_plot(frame, set_name, palette):\n    nan_ratio = frame.isna().sum()/len(frame)\n    nan_ratio = nan_ratio.to_frame().reset_index().rename({'index': 'column', 0: 'missing_percentage'},axis=1)\n    nan_ratio.sort_values(by=['missing_percentage'], ascending=False, inplace=True)\n    plt.figure(figsize=(8,6))\n    plt.title(\"Columns with missing values - {}\".format(set_name))\n    ax = sns.barplot(x='missing_percentage', y='column', orient='h',\n                     data=nan_ratio[nan_ratio['missing_percentage'] > 0],\n                     palette= palette)\n\nmissing_plot(train, 'train', 'Blues_d')\nmissing_plot(train, 'test', 'Greens_d')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"79a1c4dd880d520ebbda3c7ae8e5eae64d038e25"},"cell_type":"markdown","source":"<h3>1.4 Constant features</h3>\n\nThere are many columns with a unique value (constant) that can be dropped."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"69305480e49df4bcce199346a6880cbee1a9b440"},"cell_type":"code","source":"train_const_cols = [c for c in train.columns if len(train[c].unique()) == 1]\ntest_const_cols = [c for c in test.columns if len(test[c].unique()) == 1]\nprint(\"{} columns with a unique value on train set\".format(len(train_const_cols)))\nprint(\"{} columns with a unique value on test set\".format(len(test_const_cols)))\nprint(\"Same columns for train and test set: \", train_const_cols == test_const_cols)\ntrain.drop(train_const_cols, axis=1, inplace=True)\ntest.drop(test_const_cols, axis=1, inplace=True)\nprint(\"Shape after dropping: train {}, test {}\".format(train.shape, test.shape))\n# Flag visits with revenue\ntrain['has_revenue'] = train['totals_transactionRevenue'].apply(lambda x: 1 if x > 0 else 0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca746337b6b453157a76ad2a96a9d32280b9af43"},"cell_type":"markdown","source":"<h2> 2. Channels and visit number</h2>"},{"metadata":{"_uuid":"fd78926e8263abe3b786edbda3492bc2727cac7a"},"cell_type":"markdown","source":"<b>Channel grouping</b>\n\n* The first plot shows the number of visits for each channel on train and test set\n* The second plot is the percentage of visits with revenue for each channel (train set)\n* The last one is the mean revenue for each group"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8c9b6293fc0a5abce0fa1998b6cbc3b12d8e616a"},"cell_type":"code","source":"def barplot_percentage(count_feat, color1= 'rgb(55, 83, 109)', \n                       color2= 'rgb(26, 118, 255)',num_bars= None):\n\n    train_channel = 100*train[count_feat].value_counts()/len(train)\n    train_channel = train_channel.to_frame().reset_index()\n    test_channel = 100*test[count_feat].value_counts()/len(test)\n    test_channel = test_channel.to_frame().reset_index()\n    if num_bars:\n        train_channel = train_channel.head(num_bars)\n        test_channel = test_channel.head(num_bars)\n\n    trace0 = go.Bar(\n        x=train_channel['index'],\n        y=train_channel[count_feat],\n        name='Train set',\n        marker=dict(color=color1)\n    )\n    trace1 = go.Bar(\n        x=test_channel['index'],\n        y=test_channel[count_feat],\n        name='Test set',\n        marker=dict(color=color2,)\n    )\n\n    layout = go.Layout(\n        title='{} grouping'.format(count_feat),\n        xaxis=dict(\n            tickfont=dict(size=14, color='rgb(107, 107, 107)')\n        ),\n        yaxis=dict(\n            title='Percentage of visits',\n            titlefont=dict(size=16, color='rgb(107, 107, 107)'),\n            tickfont=dict(size=14, color='rgb(107, 107, 107)')\n        ),\n        legend=dict(\n            x=1.0,\n            y=1.0,\n            bgcolor='rgba(255, 255, 255, 0)',\n            bordercolor='rgba(255, 255, 255, 0)'\n        ),\n        barmode='group',\n        bargap=0.15,\n        bargroupgap=0.1\n    )\n\n    fig = go.Figure(data=[trace0, trace1], layout=layout)\n    iplot(fig)\nbarplot_percentage('channelGrouping')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dbba02c3c3cf79b77d59485b3500808d8cab5b21"},"cell_type":"code","source":"channel_order = ['Referral', 'Display', 'Paid Search', 'Direct', 'Organic Search', '(Other)', 'Social', 'Affiliates']\nplt.figure(figsize=(10,4))\nplt.title(\"Percentage of visits with revenue per channel\")\nsns.barplot(x='channelGrouping', y='has_revenue', data=train, order=channel_order, palette='Greens_d')\nrevenue_channel = train.groupby('channelGrouping')['totals_transactionRevenue'].sum()\nrevenue_channel = revenue_channel.to_frame().reset_index()\nplt.figure(figsize=(10,4))\nplt.title(\"Mean revenue for each channel\")\nax = sns.barplot(x='channelGrouping', y='totals_transactionRevenue', data=revenue_channel, order=channel_order, palette='Greens_d')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"059c2e8bd8fbe7db1bf05457178c175d08034128"},"cell_type":"markdown","source":"<b>Visit Number</b>\n \nThe number of visits per user is almost the same for the train and test set"},{"metadata":{"trusted":true,"_uuid":"7f50d0784f7834c752d02878db5e61e009d32893","_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Visit number distribution\")\nax1 = sns.kdeplot(train['visitNumber'], label='Train set', color=\"#005c99\")\nax2 = sns.kdeplot(test['visitNumber'], label='Test set', color=\"#e68a00\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"959d9c21f2e50c62fb721994c3c89dab3536da53"},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Number of visits and revenue\")\nax = sns.scatterplot(x='visitNumber', y='totals_transactionRevenue',\n                     data=train,color='orange', hue='has_revenue') #[train['has_revenue'] > 0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1a758270f225f1edb7b86d080c090abb5ce22fe"},"cell_type":"markdown","source":"<h2>3. Geographic Data</h2>\n\nNow let's look at each specific section: geoNetwork, totals, device and trafficSource"},{"metadata":{"_uuid":"c2d19b3281276e69d9988c3bf93af569a4bfaac6"},"cell_type":"markdown","source":"<h3>3.1 Location</h3>"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"6e415eac7781f2b96b7b4f0472b06b2012161239"},"cell_type":"code","source":"def plotmap(frame, z_var, countries_col, title, colorscale, rcolor=True):\n\n    data = [ dict(\n            type = 'choropleth',\n            autocolorscale = False,\n            colorscale = colorscale,\n            showscale = True,\n            reversescale = rcolor,\n            locations = frame[countries_col],\n            z = frame[z_var],\n            locationmode = 'country names',\n            text = frame[countries_col],\n            marker = dict(line = dict(color = '#fff', width = 2))\n        )           \n    ]\n\n    layout = dict(\n        height=680,\n        #width=1200,\n        title = title,\n        geo = dict(\n            showframe = False,\n            showcoastlines = False,\n            projection = dict(type = 'mercator'),\n        ),\n    )\n    fig = dict(data=data, layout=layout)\n    iplot(fig)\n\ncolorscale = [[0, 'rgb(102,194,165)'], [0.005, 'rgb(102,194,165)'], \n              [0.01, 'rgb(171,221,164)'], [0.02, 'rgb(230,245,152)'], \n              [0.04, 'rgb(255,255,191)'], [0.05, 'rgb(254,224,139)'], \n              [0.10, 'rgb(253,174,97)'], [0.25, 'rgb(213,62,79)'], [1.0, 'rgb(158,1,66)']]\n    \n# Plot world map - total visits\ntmp = train[\"geoNetwork_country\"].value_counts().to_frame().reset_index()\nplotmap(tmp, 'geoNetwork_country', 'index', 'Total visits by Country', colorscale, False)\n\ncolorscale = [[0,\"rgb(5, 10, 172)\"],[0.35,\"rgb(40, 60, 190)\"],[0.5,\"rgb(70, 100, 245)\"],\\\n        [0.6,\"rgb(90, 120, 245)\"],[0.7,\"rgb(106, 137, 247)\"],[1,\"rgb(220, 220, 220)\"]]\n# Plot world map - mean revenue\ntmp = train.groupby(\"geoNetwork_country\").agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\nplotmap(tmp, 'totals_transactionRevenue','geoNetwork_country', 'Mean revenue by country', colorscale)\n# Plot world map - total revenue\ntmp = train.groupby(\"geoNetwork_country\").agg({\"totals_transactionRevenue\" : \"sum\"}).reset_index()\nplotmap(tmp, 'totals_transactionRevenue','geoNetwork_country', 'Total revenue by country', colorscale)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e73a97876fe372f5883b37f715c771842f09edb","_kg_hide-input":true},"cell_type":"code","source":"tmp1 = train[\"geoNetwork_continent\"].value_counts().to_frame().reset_index()\ntmp2 = train.groupby(\"geoNetwork_continent\")[\"totals_transactionRevenue\"].sum().to_frame().reset_index()\nfig = {\n  \"data\": [\n    {\n        \"values\": tmp1['geoNetwork_continent'],\n        \"labels\": tmp1['index'],\n        \"name\": \"Visits\",\n        \"domain\": {\"x\": [0, 0.46]},\n        \"hoverinfo\":\"label+percent+name\",\n        \"hole\": .5,\n        \"type\": \"pie\",\n        #\"textinfo\": \"none\"\n    },\n    {\n        \"values\": tmp2['totals_transactionRevenue'],\n        \"labels\": tmp2['geoNetwork_continent'],\n        \"name\": \"Revenue\",\n        #\"textposition\":\"inside\",\n        \"domain\": {\"x\": [.54, 1]},\n        \"hoverinfo\":\"label+percent+name\",\n        \"hole\": .5,\n        \"type\": \"pie\",\n        #\"textinfo\": \"none\"\n    }],\n  \"layout\": {\n        \"title\":\"Visits and Revenue by Continent\",\n        \"annotations\": [\n            {\n                \"font\": {\n                    \"size\": 20\n                },\n                \"showarrow\": False,\n                \"text\": \"Visits\",\n                \"x\": 0.18,\n                \"y\": 0.5\n            },\n            {\n                \"font\": {\n                    \"size\": 20\n                },\n                \"showarrow\": False,\n                \"text\": \"Revenue\",\n                \"x\": 0.85,\n                \"y\": 0.5\n            }\n        ]\n    }\n}\niplot(fig, filename='donut')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2ed8d365c49a46c9325442873b13375c88a01651"},"cell_type":"markdown","source":"Only half the number of visits is from the US, but more than 94% of the revenue."},{"metadata":{"_uuid":"7c14ee2917984366ba4f41a69c7a5ec78515c858"},"cell_type":"markdown","source":"<h3>3.2 Network domain</h3>\n\nTop 10 domains in both train and test set:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b13d9d5545d4cc20ce9c489c0f7da0413d5c5c8b"},"cell_type":"code","source":"barplot_percentage('geoNetwork_networkDomain', num_bars= 10, \n                   color1='rgb(38, 115, 77)', color2='rgb(102, 204, 153)')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5dcae49ab86ef3d2e471e4e8ed35626e14f2079"},"cell_type":"markdown","source":"<h2>4. totals</h2>\n\nThere are four features in totals: hits, pageviews, bounces and newVisits."},{"metadata":{"_uuid":"bf585891aa844793f3d20a2f54d270f10764bfe0"},"cell_type":"markdown","source":"<h3>4.1 Bounces</h3>\n\nHere we have an interesting information: bounces has only 1's and missing values, but only the former has revenue:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3331df007d0738ac7d0a2060be5ae86403f947b6"},"cell_type":"code","source":"bounces_nan = train[train['totals_bounces'].isnull()]['totals_transactionRevenue'].sum()\nbounces_1 = train[~train['totals_bounces'].isnull()]['totals_transactionRevenue'].sum()\nprint(\"Revenue for bounce missing: {}, revenue for bounce '1': {}\".format(bounces_nan, bounces_1))\nplt.figure(figsize=(8,4))\nplt.title(\"Bounces count\")\nax = sns.countplot(x='totals_bounces', data=train.fillna('nan'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5fb4b21f68a767c491f2b726a0a11fd409ab09c2"},"cell_type":"markdown","source":"<h3>4.2 New Visits<h3>\n\nAnother binary column:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"997cadcd9304d733994a14905b5430fc9b5dc2ce"},"cell_type":"code","source":"plt.figure(figsize=(8,4))\nplt.title(\"New visits count\")\nax = sns.countplot(x='totals_newVisits', data=train.fillna('nan'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff40f4ccad62840eed6d490ef27254dfee748341"},"cell_type":"markdown","source":"<h3>4.3 Hits</h3>\n\nHere we have a numeric feature with min 1 and max 500. More than 75% of values are less or equal to 4."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5437cd3444eea35086bbea3d9ef6ddf1f267badb"},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Hits distribution\")\nax1 = sns.kdeplot(train['totals_hits'].astype('float64'), color=\"#006633\", shade=True)\nplt.figure(figsize=(10,4))\nplt.title(\"Page views distribution\")\nax2 = sns.kdeplot(train[train['has_revenue'] == 0]['totals_hits'].astype('float64'),\n                  label='No revenue', color=\"#0000ff\")\nax2 = sns.kdeplot(train[train['has_revenue'] == 1]['totals_hits'].astype('float64'),\n                  label='Has revenue', color=\"#ff6600\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4375f505dd45911b7498da41ab7b3b31c724c57c"},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Hits vs Log1p Revenue\")\nax = sns.scatterplot(x=train['totals_hits'].astype('float64'), y=np.log1p(train['totals_transactionRevenue']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8caef5bf4e392d94d415579efb72dc4990b96b7b"},"cell_type":"markdown","source":"Transactions with revenue are more concentrated in the 0 to 100 range"},{"metadata":{"_uuid":"1c9e1dd216a0dc36ee727104bfa1f7976bec2aac"},"cell_type":"markdown","source":"<h3>4.4 Page views</h3>\n\nNumber of pages visited in a single session. There are 100 missing values"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"0901d83d5329f60fb63c79ed081880c70eff2545"},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Page views distribution\")\nax3 = sns.kdeplot(train['totals_pageviews'].dropna().astype('float64'), color=\"#006633\", shade=True)\nplt.figure(figsize=(10,4))\nplt.title(\"Page views distribution\")\nax4 = sns.kdeplot(train[train['has_revenue'] == 0]['totals_pageviews'].dropna().astype('float64'),\n                  label='No revenue', color=\"#0000ff\")\nax4 = sns.kdeplot(train[train['has_revenue'] == 1]['totals_pageviews'].dropna().astype('float64'),\n                  label='Has revenue', color=\"#ff6600\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d608b445eab76cf433fcb6eb8895d52d9c23c7b6"},"cell_type":"code","source":"plt.figure(figsize=(10,4))\nplt.title(\"Page views vs Log1p Revenue\")\nax = sns.scatterplot(x=train['totals_pageviews'].dropna().astype('float64'), y=np.log1p(train['totals_transactionRevenue']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3edd2b96d47e1fda888388e6894436afd667122d"},"cell_type":"markdown","source":"And the two numerical features combined:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"98458fb4556c1169999b5a5d6229981dbb65ad94"},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nplt.title(\"Hits vs Page views\")\ntmp = train[['totals_hits', 'totals_pageviews', 'has_revenue']].copy()\ntmp['totals_hits'] = tmp['totals_hits'].astype('float64')\ntmp['totals_pageviews'] = tmp['totals_pageviews'].dropna().astype('float64')\nax = sns.scatterplot(x='totals_hits', y='totals_pageviews', hue='has_revenue', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb2f33635677f67c9766888ca855ab63c21a105f"},"cell_type":"markdown","source":"<h2>5. Device</h2>\n\nHere we also have four columns, all of them categorical and without missing values: \n* device_browser: 54 categories\n* device_operatingSystem: 20 categories\n* device_deviceCategory: desktop, mobile or table\n* device_isMobile: boolean column"},{"metadata":{"_uuid":"dffc35986f0259f4c5c3f0ef4487cfa442255a93"},"cell_type":"markdown","source":"<h3>5.1 Browser</h3>\n\nThere are only 9 browsers with revenue and Chrome has most visits and revenue, but the highest mean revenue is from Firefox."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f594bbfddcec1236a82e9e4cf8b3e6a30e772590"},"cell_type":"code","source":"# Group and plot revenue\ndef group_revenue(group_col, title, sum_values=True, palette='Blues_d', size=(8,5)):\n    if sum_values:\n        tmp = train.groupby(group_col)['totals_transactionRevenue'].sum()\n    else:\n        tmp = train.groupby(group_col)['totals_transactionRevenue'].mean()\n    tmp = tmp.to_frame().reset_index().sort_values('totals_transactionRevenue', ascending=False)\n    tmp = tmp[tmp['totals_transactionRevenue'] > 0]\n    plt.figure(figsize=size)\n    plt.title(title)\n    ax = sns.barplot(y=tmp[group_col], x= tmp.totals_transactionRevenue, orient='h', palette=palette)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d5f47dd1c6e90c1b57b82f46688db6540083a838"},"cell_type":"code","source":"# Visits\nbarplot_percentage('device_browser', num_bars= 7)\n# Revenue\ngroup_revenue('device_browser', 'Total revenue by browser', True, size=(9,5))\ngroup_revenue('device_browser', 'Mean revenue by browser', False, 'Greens_d',size=(9,5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff450380a750f8cc774f5a6383de7546a6ac22c7"},"cell_type":"markdown","source":"<h3>5.2 Device category</h3>\n\nHere we have three possible categories."},{"metadata":{"trusted":true,"_uuid":"0f3fc167c0bc4a3590907fa5dd4d4a9aab3dee98","_kg_hide-input":true},"cell_type":"code","source":"# Pie chart\ncolors = ['#5c8a8a', '#94b8b8', '#b3cccc']\ndev_train = train['device_deviceCategory'].value_counts().to_frame().reset_index()\ndev_test = test['device_deviceCategory'].value_counts().to_frame().reset_index()\ntrace1 = go.Pie(labels=dev_train['index'], values=dev_train.device_deviceCategory,\n                domain= {'x': [0, .48]}, marker=dict(colors=colors))\ntrace2 = go.Pie(labels=dev_test['index'], values=dev_test.device_deviceCategory,\n                domain= {'x': [0.52, 1]}, marker=dict(colors=colors))\nlayout = dict(title= \"Device category - train and test\", height=400)\nfig = dict(data=[trace1, trace2], layout=layout)\niplot(fig)\n# Revenue\ngroup_revenue('device_deviceCategory', 'Mean revenue by device category', False, 'Greens_d', size=(8, 3.6))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4a3c4b98c8c414dda059afeeed6567dc983b153"},"cell_type":"markdown","source":"<h3>5.3 Operating system</h3>\n\nThere are 20 operating systems, but many OS have very few visits and only seven are responsible for all revenue."},{"metadata":{"trusted":true,"_uuid":"65215f901aa75037ba90bcc74fc6a05fc0be8191","_kg_hide-input":true},"cell_type":"code","source":"barplot_percentage('device_operatingSystem', num_bars = 7,\n                   color1='rgb(204, 82, 0)', color2='rgb(255, 163, 102)')\n# Revenue\ngroup_revenue('device_operatingSystem', 'Total revenue by OS', True, 'Blues_d', size=(9, 4))\ngroup_revenue('device_operatingSystem', 'Mean revenue by OS', False, 'Greens_d', size=(9, 4))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22fe610fe94c2d92c092769c2c781c54993aeb63"},"cell_type":"markdown","source":"<h2>6. Traffic Source</h2>\n\nWe have 13 columns in this category and many missing values. The campaignCode feature has all values missing except one and can be safely removed, so we are left with twelve features.\n\n<b>This section was moved to:</b> [Understanding Traffic Source: in-depth analysis](https://www.kaggle.com/jsaguiar/understanding-traffic-source-in-depth-analysis?scriptVersionId=5801526)"},{"metadata":{"_uuid":"f4ed41de0a869369738fd05d15563cbea8450281"},"cell_type":"markdown","source":"<h2>7. User analysis</h2>\n\nIn this section we will be looking at some statistics for each user (fullVisitorId)."},{"metadata":{"_uuid":"71fc5da2ecf18595fc627288cc930dcd6d667bc1"},"cell_type":"markdown","source":"<h3>7.1 Visits per user</h3>"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ac8e7b032b9c28cd5bcc9c9f0412f077bbe46506"},"cell_type":"code","source":"tmp = train.groupby('fullVisitorId').size().value_counts().to_frame().reset_index()\nsum_ = tmp[tmp['index'].astype('int16') > 5][0].sum()\ntmp = tmp.head(5).append(pd.DataFrame({'index': ['more than 5'], 0: [sum_]})).reset_index()\nplt.figure(figsize=(10,4))\nplt.title(\"Visits per user\")\nax = sns.barplot(x=tmp['index'], y=tmp[0], palette='Blues_d')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0c85c1b6c88e782716165db36528aabf3b6a4f9"},"cell_type":"markdown","source":"<h3>7.2 Totals per user</h3>"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e15a2d7a7ec305ac0f9ff06197d5a5dbcdae184a"},"cell_type":"code","source":"# Plot from https://www.kaggle.com/shivamb/exploratory-analysis-ga-customer-revenue\ndef getbin_hits(x):\n    if x < 5:\n        return \"1-5\"\n    elif x < 10:\n        return \"5-10\"\n    elif x < 30:\n        return \"10-30\"\n    elif x < 50:\n        return \"30-50\"\n    elif x < 100:\n        return \"50-100\"\n    else:\n        return \"100+\"\n\nagg_dict = {}\nfor col in [\"totals_bounces\", \"totals_hits\", \"totals_newVisits\", \"totals_pageviews\", \"totals_transactionRevenue\"]:\n    train[col] = train[col].astype('float')\n    agg_dict[col] = \"sum\"\ntmp = train.groupby(\"fullVisitorId\").agg(agg_dict).reset_index()\ntmp[\"total_hits_bin\"] = tmp[\"totals_hits\"].apply(getbin_hits)\ntmp[\"totals_bounces_bin\"] = tmp[\"totals_bounces\"].apply(lambda x : str(x) if x <= 5 else \"5+\")\ntmp[\"totals_pageviews_bin\"] = tmp[\"totals_pageviews\"].apply(lambda x : str(x) if x <= 50 else \"50+\")\n\nt1 = tmp[\"total_hits_bin\"].value_counts()\nt2 = tmp[\"totals_bounces_bin\"].value_counts()\nt3 = tmp[\"totals_newVisits\"].value_counts()\nt4 = tmp[\"totals_pageviews_bin\"].value_counts()\n\nfig = tools.make_subplots(rows=2, cols=2, subplot_titles=[\"Total Hits per User\", \"Total Bounces per User\", \n                                                         \"Total NewVisits per User\", \"Total PageViews per User\"], print_grid=False)\n\ntr1 = go.Bar(x = t1.index[:20], y = t1.values[:20])\ntr2 = go.Bar(x = t2.index[:20], y = t2.values[:20])\ntr3 = go.Bar(x = t3.index[:20], y = t3.values[:20])\ntr4 = go.Bar(x = t4.index, y = t4.values)\n\nfig.append_trace(tr1, 1, 1)\nfig.append_trace(tr2, 1, 2)\nfig.append_trace(tr3, 2, 1)\nfig.append_trace(tr4, 2, 2)\n\nfig['layout'].update(height=700, showlegend=False)\niplot(fig)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}