{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"@author: André Daniël VOLSCHENK  \n\nKaggle project {Google Analytics Customer Revenue Prediction}  \nkaggle.com/andredanielvolschenk  \n\n# Preface\nThe intention of this notebook is to continue from Part 1 by visualizing and performing Exploratory Data Analysis on the data. Features will be engineered and explored.\n\n# Contents\n* [Setup](#Setup)\n* [Visualization functions](#Visualization-functions)  \nFeature collections:\n* [1. channelGrouping](#1.-channelGrouping)\n* [2. customDimensions](#2.-customDimensions)\n* [3. date](#3.-date)\n* [4. device](#4.-device)\n* [5. fullVisitorId](#5.-fullVisitorId)\n* [6. geoNetwork](#6.-geoNetwork)\n* [7. hits](#7.-hits)\n* [8. totals](#8.-totals)\n* [9. trafficSource](#9.-trafficSource)\n* [10. visitId](#10.-visitId)\n* [11. visitNumber](#11.-visitNumber)\n* [12. visitStartTime](#12.-visitStartTime)\n\n\n# Setup\n\n## Problem Statement\nPlease see Part 1 : https://www.kaggle.com/andredanielvolschenk/gstore-part-1-data-cleansing  \n\n## Import libraries\nLets import libraries and see what datafiles we have in our environment."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport statistics as ss\n\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nmpl.style.use('ggplot') # if error, use plt.style.use('ggplot') instead\n\n#ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport datetime as dt\n\nimport plotly as ply\nimport plotly.offline as plyo\nplyo.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nimport seaborn as sns\n\n\nimport json\nimport pandas.io.json as pdjson\nimport ast\n\nimport gc\ngc.enable()\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41c0fc82d53313d8239ef49a922efca3f107872c"},"cell_type":"markdown","source":"## Load data as per Part 1\nIn Part 1 (https://www.kaggle.com/andredanielvolschenk/gstore-part-1-data-cleansing-with-v2-data) we listed the columns we want to keep.  \nHere we shall only import the training set for now and save only selected columns to `data1`.  \nFirst we save a variable with the columns declared at the end of Part 1, and a variable with JSON columns too:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"73de8b413443eda92b137d2d86dfe362043234d7"},"cell_type":"code","source":"json_vars = ['device', 'geoNetwork', 'totals', 'trafficSource', 'hits', 'customDimensions']\n\nfinal_vars = ['channelGrouping','customDimensions_index','customDimensions_value','date',\n'device_browser','device_deviceCategory','device_isMobile','device_operatingSystem',\n'fullVisitorId','geoNetwork_city','geoNetwork_continent','geoNetwork_country',\n'geoNetwork_metro','geoNetwork_networkDomain','geoNetwork_region','geoNetwork_subContinent',\n'hits_appInfo.exitScreenName','hits_appInfo.landingScreenName','hits_appInfo.screenDepth',\n'hits_appInfo.screenName','hits_contentGroup.contentGroup1','hits_contentGroup.contentGroup2',\n'hits_contentGroup.contentGroup3','hits_contentGroup.contentGroup4','hits_contentGroup.contentGroup5',\n'hits_contentGroup.contentGroupUniqueViews1','hits_contentGroup.contentGroupUniqueViews2',\n'hits_contentGroup.contentGroupUniqueViews3','hits_contentGroup.previousContentGroup1',\n'hits_contentGroup.previousContentGroup2','hits_contentGroup.previousContentGroup3',\n'hits_contentGroup.previousContentGroup4','hits_contentGroup.previousContentGroup5',\n'hits_customDimensions','hits_customMetrics','hits_customVariables','hits_dataSource',\n'hits_eCommerceAction.action_type','hits_eCommerceAction.option','hits_eCommerceAction.step',\n'hits_eventInfo.eventAction','hits_eventInfo.eventCategory','hits_eventInfo.eventLabel',\n'hits_exceptionInfo.isFatal','hits_experiment','hits_hitNumber','hits_hour','hits_isEntrance',\n'hits_isExit','hits_isInteraction','hits_item.currencyCode','hits_item.transactionId',\n'hits_latencyTracking.domContentLoadedTime','hits_latencyTracking.domInteractiveTime',\n'hits_latencyTracking.domLatencyMetricsSample','hits_latencyTracking.domainLookupTime',\n'hits_latencyTracking.pageDownloadTime','hits_latencyTracking.pageLoadSample',\n'hits_latencyTracking.pageLoadTime','hits_latencyTracking.redirectionTime',\n'hits_latencyTracking.serverConnectionTime','hits_latencyTracking.serverResponseTime',\n'hits_latencyTracking.speedMetricsSample','hits_minute','hits_page.hostname','hits_page.pagePath',\n'hits_page.pagePathLevel1','hits_page.pagePathLevel2','hits_page.pagePathLevel3',\n'hits_page.pagePathLevel4','hits_page.pageTitle','hits_page.searchCategory','hits_page.searchKeyword',\n'hits_promotionActionInfo.promoIsClick','hits_promotionActionInfo.promoIsView','hits_publisher_infos',\n'hits_referer','hits_social.hasSocialSourceReferral','hits_social.socialInteractionNetworkAction',\n'hits_social.socialNetwork','hits_time','hits_transaction.affiliation','hits_transaction.currencyCode',\n'hits_transaction.localTransactionRevenue','hits_transaction.localTransactionShipping',\n'hits_transaction.localTransactionTax','hits_transaction.transactionId',\n'hits_transaction.transactionRevenue','hits_transaction.transactionShipping',\n'hits_transaction.transactionTax','hits_type','totals_bounces','totals_hits','totals_newVisits',\n'totals_pageviews','totals_sessionQualityDim','totals_timeOnSite','totals_totalTransactionRevenue',\n'totals_transactionRevenue','totals_transactions','trafficSource_adContent',\n'trafficSource_adwordsClickInfo.adNetworkType','trafficSource_adwordsClickInfo.gclId',\n'trafficSource_adwordsClickInfo.isVideoAd','trafficSource_adwordsClickInfo.page',\n'trafficSource_adwordsClickInfo.slot','trafficSource_campaign','trafficSource_isTrueDirect',\n'trafficSource_keyword','trafficSource_medium','trafficSource_referralPath','trafficSource_source',\n'visitId','visitNumber','visitStartTime']\n\nprint('created json_var and final_var')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3eb413461696301002b509bc24e7db98fe7b1d83"},"cell_type":"markdown","source":"Because our memory is limited, we import 500'000 rows from `train_v2` to do this notebook. This should be representative of the entire data set which has 1'708'337 rows."},{"metadata":{"trusted":true,"_uuid":"47d59554b95cb2edbf71b58a2eaf28c39053e0f1","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"# lets append json_vars with final_vars, because we still need to import the json vars before expanding them\nall_vars  = json_vars + final_vars # the master list of columns to import\n\ndef load_df(csv_path, usecols=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    ans = pd.DataFrame()\n    \n    dfs = pd.read_csv(csv_path, sep=',',\n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                    chunksize = 100000,   # 100000\n                     nrows=500000  # TODO: take out\n                     )\n    \n    for df in dfs:\n        df.reset_index(drop = True,inplace = True)\n        \n        device_list=df['device'].tolist()\n        #deleting unwanted columns before normalizing\n        for device in device_list:\n            del device['browserVersion'],device['browserSize'],device['flashVersion'],device['mobileInputSelector'],device['operatingSystemVersion'],device['screenResolution'],device['screenColors']\n        df['device']=pd.Series(device_list)\n        \n        geoNetwork_list=df['geoNetwork'].tolist()\n        for network in geoNetwork_list:\n            del network['latitude'],network['longitude'],network['networkLocation'],network['cityId']\n        df['geoNetwork']=pd.Series(geoNetwork_list)\n        \n        df['hits']=df['hits'].apply(ast.literal_eval)\n        df['hits']=df['hits'].str[0]\n        df['hits']=df['hits'].apply(lambda x: {'index':np.NaN,'value':np.NaN} if pd.isnull(x) else x)\n    \n        df['customDimensions']=df['customDimensions'].apply(ast.literal_eval)\n        df['customDimensions']=df['customDimensions'].str[0]\n        df['customDimensions']=df['customDimensions'].apply(lambda x: {'index':np.NaN,'value':np.NaN} if pd.isnull(x) else x)\n    \n        JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource','hits','customDimensions']\n\n        for column in JSON_COLUMNS:\n            column_as_df = pdjson.json_normalize(df[column])\n            column_as_df.columns = [f\"{column}_{subcolumn}\" for subcolumn in column_as_df.columns]\n            df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n        \n        \n        print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n        \n        # we wont see each and every column in each chunk that we load, so we need to find where our master list intersects with the actual data\n        usecols = set(usecols).intersection(df.columns)\n        usecols = list(usecols)\n        use_df = df[usecols]\n        del df\n        gc.collect()\n        ans = pd.concat([ans, use_df], axis = 0).reset_index(drop = True)\n        print('Stored shape:', ans.shape)\n        \n    return ans\n\n\n\ndata1 = load_df('../input/train_v2.csv', usecols=final_vars)\n#data2 = load_df(\"../input/test_v2.csv\", usecols=final_vars)\n\nprint('data1 shape: ', data1.shape)\n#print('data2 shape: ', data2.shape)\n\nprint(\"data1 loaded\")\n#print(\"data2 loaded\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d045942ca680acd3d2992c421d5f67431ce828e"},"cell_type":"markdown","source":"## Target variable preparation\nTwo variables are given: `totals_transactionRevenue` and `totals_totalTransactionRevenue`.  \n\nNote that the description for `totals_transactionRevenue` is given as:  \n\"This field is deprecated. Use \"totals.totalTransactionRevenue\" instead (see above).\"  \n(see https://support.google.com/analytics/answer/3437719?hl=en)  \nHowever ! The problem statement for the v2 competition requires us to predict based on  `totals_transactionRevenue`. So we will drop `totals_totalTransactionRevenue`.\n\nBefore we start, it is worth noting that our target variabe (`totals_transactionRevenue`) is the actual purchase amount in $ multiplied by 1 million.  \nLets divide this by 1x10^6 in order to get the actual purchase amount again.  \nBefore any of this, we must replace NaNs with 0, because a 'no purchase' is encoded as NaN."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"fd014e7ce161cc9b3eae41b77f3dfc91db529cf2"},"cell_type":"code","source":"data1['totals_transactionRevenue'].fillna(0, inplace=True)\ndata1['totals_transactionRevenue'] = data1['totals_transactionRevenue'].astype('float')/1000000\n\ndata1[['totals_transactionRevenue']].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25675be3dd2e86f487d98e5497331e42f9002ec3"},"cell_type":"markdown","source":"# Visualization functions\nLets look at features to see whether they have information about the target  \nWe will go through each one, and even create some new ones.\n\nSpecifically there are 4 plots per feature that may give us some insight:\n* 1. a count of how many times an instance is encounterred in the dataset\n* 2. a count of how many purchases were made for that instance\n* 3. the mean revenue of the purchases for that instance\n* 4. the total revenue for that instance\n\nSince we are doing comparisons to `totals_transactionRevenue`, we want to look at the training data only.  \n\nDeclare a `bar_plots` function to handle qualitative variables:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a32bd39ce18a7e8b7a03b6ba63148a53f55518f8"},"cell_type":"code","source":"def bar_plots(cstr, data, n_bars=None):\n    \n    \n    if n_bars:\n        counts=data[cstr].value_counts()\n        counts = counts.iloc[:n_bars]\n        data = data[data[cstr].isin(counts.index)]\n    #\n    \n    \n    df = data.copy()\n    df[cstr] = df[cstr].astype('str')\n    \n    fig = plt.figure(figsize=(14,10))   # define (new) plot area\n    ax = fig.gca()   # define axis\n    plt.suptitle(str('Plots for '+cstr))\n    ax.get_xaxis().set_visible(False)\n    ax.get_yaxis().set_visible(False)\n    \n    # FIGURE 1\n    ax = fig.add_subplot(221)\n    ax.set_xlabel('Count of instances')   # Set text for the x axis\n    counts = df[cstr].value_counts() # This is important to get the full nr of instances\n    counts.plot.barh(ax = ax)   # Use the plot.bar method on the counts data frame\n    \n    for i, v in enumerate( counts ):  # give each 3 decimal points\n        d=''\n        if v>1000000000:\n            v=v/1000000000\n            d='B'\n        elif v>1000000:\n            v=v/1000000\n            d='M'\n        elif v>1000:\n            v=v/1000\n            d='k'\n        ax.text(v + 0.5, i + .25, str(round(v,3))+d, color='black', fontweight='bold')\n    \n    \n    # FIGURE 2\n    ax = fig.add_subplot(222)\n    ax.set_xlabel('Count of purchases')   # Set text for the x axis\n    counts[:] = 0  # set all to 0\n    counts2=df[cstr].where(df.totals_transactionRevenue>0).value_counts()\n    counts.update(counts2)\n    counts.plot.barh(ax = ax)   # Use the plot.bar method on the counts data frame\n    \n    for i, v in enumerate( counts ):  # give each 3 decimal points\n        d=''\n        if v>1000000000:\n            v=v/1000000000\n            d='B'\n        elif v>1000000:\n            v=v/1000000\n            d='M'\n        elif v>1000:\n            v=v/1000\n            d='k'\n        ax.text(v + 0.5, i + .25, str(round(v,3))+d, color='black', fontweight='bold')\n    \n    \n    # FIGURE 3\n    ax = fig.add_subplot(223)\n    ax.set_xlabel('Mean of purchases')   # Set text for the x axis\n    counts[:] = 0  # set all to 0\n    counts2 = df[df.totals_transactionRevenue > 0].groupby(cstr)['totals_transactionRevenue'].agg(['mean'])\n    idx=counts2.index\n    counts2=pd.Series(counts2.values.reshape(-1,))\n    counts2.index = idx\n    counts.update(counts2)\n    counts = counts.astype('int64')\n    counts.plot.barh(ax = ax)   # Use the plot.bar method on the counts data frame\n    \n    for i, v in enumerate( counts ):  # give each 3 decimal points\n        d=''\n        if v>1000000000:\n            v=v/1000000000\n            d='B'\n        elif v>1000000:\n            v=v/1000000\n            d='M'\n        elif v>1000:\n            v=v/1000\n            d='k'\n        ax.text(v + 0.5, i + .25, str(round(v,3))+d, color='black', fontweight='bold')\n    \n    \n    # FIGURE 4\n    ax = fig.add_subplot(224)\n    ax.set_xlabel('Sum of purchases')   # Set text for the x axis\n    counts[:] = 0  # set all to 0\n    counts2 = df.groupby(cstr)['totals_transactionRevenue'].agg(['sum'])\n    idx=counts2.index\n    counts2=pd.Series(counts2.values.reshape(-1,))\n    counts2.index = idx\n    counts.update(counts2)\n    counts = counts.astype('int64')\n    counts.plot.barh(ax = ax)   # Use the plot.bar method on the counts data frame\n    \n    for i, v in enumerate( counts ):  # give each 3 decimal points\n        d=''\n        if v>1000000000:\n            v=v/1000000000\n            d='B'\n        elif v>1000000:\n            v=v/1000000\n            d='M'\n        elif v>1000:\n            v=v/1000\n            d='k'\n        ax.text(v + 0.5, i + .25, str(round(v,3))+d, color='black', fontweight='bold')\n#\n\nprint(\"'bar_plots' function declared\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c40fed4b8790c124e039741d79c2949ad1462802"},"cell_type":"markdown","source":"Declare  a `scatter_plots` function to handle datetime variables:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f1e3807a89da43813f3ec6de2376f7c888fd5ff0"},"cell_type":"code","source":"def scatter_plots(train_df, test_df):\n    \n    \n    def scatter_plot(cnt_srs, color):\n        trace = go.Scatter(\n            x=cnt_srs.index[::-1],\n            y=cnt_srs.values[::-1],\n            showlegend=False,\n            marker=dict(\n                color=color,\n            ),\n        )\n        return trace\n    \n    train_df['date'] = train_df['date'].apply(lambda x: dt.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\n    cnt_srs = train_df.groupby('date')['totals_transactionRevenue'].agg(['size', 'count', 'mean', 'sum'])\n    cnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\", \"sum\"]\n    cnt_srs = cnt_srs.sort_index()\n    #cnt_srs.index = cnt_srs.index.astype('str')\n    trace1 = scatter_plot(cnt_srs[\"count\"], 'blue')\n    trace2 = scatter_plot(cnt_srs[\"count of non-zero revenue\"], 'blue')\n    trace3 = scatter_plot(cnt_srs[\"mean\"], 'blue')\n    trace4 = scatter_plot(cnt_srs[\"sum\"], 'blue')\n    \n    fig = ply.tools.make_subplots(rows=4, cols=1, vertical_spacing=0.08,\n                              subplot_titles=[\"Date - Count\", \"Date - Non-zero Revenue count\",\n                                              \"Date - Mean of purchases\", \"Date - Sum of purchases\"])\n    fig.append_trace(trace1, 1, 1)\n    fig.append_trace(trace2, 2, 1)\n    fig.append_trace(trace3, 3, 1)\n    fig.append_trace(trace4, 4, 1)\n    fig['layout'].update(height=800, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Date Plots\")\n    plyo.iplot(fig, filename='date-plots')\n    \n    \n    # test set\n    test_df['date'] = test_df['date'].apply(lambda x: dt.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\n    cnt_srs = test_df.groupby('date')['fullVisitorId'].size()\n    \n    \n    trace = scatter_plot(cnt_srs, 'red')\n    \n    layout = go.Layout(\n        height=400,\n        width=800,\n        paper_bgcolor='rgb(233,233,233)',\n        title='Dates in Test set'\n    )\n    \n    data = [trace]\n    fig = go.Figure(data=data, layout=layout)\n    plyo.iplot(fig, filename=\"ActivationDate\")\n    \n    \n#\nprint(\"'scatter_plots' function declared\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"143394e89062c65fc59f1d553444231b17b10355"},"cell_type":"markdown","source":"Declare a `kde_scatter_plots` function to handle quantitative variables:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"42e011a3640a387ef491498d5192391875bc0c0e","scrolled":false},"cell_type":"code","source":"def kde_scatter_plots(cstr, train):\n    \n    # Figure 1\n    plt.figure(figsize=(15,5))\n    plt.title(str(cstr)+\" distribution\")\n    ax1 = sns.kdeplot(train[cstr].astype('float64'), color=\"#006633\", shade=True)\n    \n    # Figure 2\n    plt.figure(figsize=(15,5))\n    plt.title(str(cstr)+\" distribution\")\n    ax2 = sns.kdeplot(train[np.isnan(train['totals_transactionRevenue'])][cstr].astype('float64'),\n                      label='No revenue', color=\"#0000ff\")\n    ax2 = sns.kdeplot(train[train['totals_transactionRevenue'] >0][cstr].astype('float64'),\n                      label='Has revenue', color=\"#ff6600\")\n    \n    # Figure 3\n    temp = train.groupby(cstr, as_index=False)['totals_transactionRevenue'].mean()\n    \n    plt.figure(figsize=(15,5))\n    plt.title(str(cstr)+\" distribution\")\n    ax3 = sns.scatterplot(x=temp[cstr], y=temp.totals_transactionRevenue, label='Mean of purchases', color=\"#ff6600\")\n    \n    # Figure 3\n    temp = train.groupby(cstr, as_index=False)['totals_transactionRevenue'].sum()\n    \n    plt.figure(figsize=(15,5))\n    plt.title(str(cstr)+\" distribution\")\n    ax4 = sns.scatterplot(x=temp[cstr], y=temp.totals_transactionRevenue, label='Sum of purchases', color=\"#ff6600\")\n    \n#\nprint(\"'kde_scatter_plots' function declared\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1831d614cedfd0ff385da1145e3a8f6ab9a1c589"},"cell_type":"markdown","source":"Declare a `globe_plot` function for nationality-based geographical variables:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d23c1e5c5ed57565f6d310b322aa64cb8b954e6c"},"cell_type":"code","source":"def globe_plot(train):\n    \n    def globe(tmp, title, var2=None):\n        \n        if var2 != None:\n            locations = tmp.geoNetwork_country\n            z = tmp.totals_transactionRevenue\n            text = tmp.totals_transactionRevenue\n        else:\n            locations = tmp.index\n            z = tmp.values\n            text = tmp.values\n        \n        # plotly globe credits - https://www.kaggle.com/arthurtok/generation-unemployed-interactive-plotly-visuals\n        colorscale = [[0, 'rgb(102,194,165)'], [0.005, 'rgb(102,194,165)'], \n                      [0.01, 'rgb(171,221,164)'], [0.02, 'rgb(230,245,152)'], \n                      [0.04, 'rgb(255,255,191)'], [0.05, 'rgb(254,224,139)'], \n                      [0.10, 'rgb(253,174,97)'], [0.25, 'rgb(213,62,79)'], [1.0, 'rgb(158,1,66)']]\n        \n        data = [ dict(\n                type = 'choropleth',\n                autocolorscale = False,\n                colorscale = colorscale,\n                showscale = True,\n                locations = locations,\n                z = z,\n                locationmode = 'country names',\n                text = text,\n                marker = dict(\n                    line = dict(color = '#fff', width = 2)) )           ]\n        \n        layout = dict(\n            height=500,\n            title = title,\n            geo = dict(\n                showframe = True,\n                showocean = True,\n                oceancolor = '#222',\n                projection = dict(\n                type = 'orthographic',\n                    rotation = dict(\n                            lon = 60,\n                            lat = 10),\n                ),\n                lonaxis =  dict(\n                        showgrid = False,\n                        gridcolor = 'rgb(102, 102, 102)'\n                    ),\n                lataxis = dict(\n                        showgrid = False,\n                        gridcolor = 'rgb(102, 102, 102)'\n                        )\n                    ),\n                )\n        fig = dict(data=data, layout=layout)\n        plyo.iplot(fig)\n    #\n    \n    \n    \n    tmp = train[\"geoNetwork_country\"].value_counts()\n    title = 'Visits by Country'\n    globe(tmp, title, var2=None)\n    \n    tmp = train[train.totals_transactionRevenue > 0][\"geoNetwork_country\"].value_counts()\n    title = 'Number of purchases by Country'\n    globe(tmp, title, var2=None)\n    \n    tmp = train.groupby(\"geoNetwork_country\").agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\n    var2='totals_transactionRevenue'\n    title = 'Mean Revenue by Countries'\n    globe(tmp, title, var2)\n    \n    tmp = train.groupby(\"geoNetwork_country\").agg({\"totals_transactionRevenue\" : \"sum\"}).reset_index()\n    var2='totals_transactionRevenue'\n    title = 'Sum Revenue by Countries'\n    globe(tmp, title, var2)\n    \n#\nprint(\"'globe_plot' function defined\")\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5117862a50eedb55955e81415324c7eb82453583"},"cell_type":"markdown","source":"## Review column names\nLets look at the list of column names one last time"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"059837de85deaf0cf3097fd3d0b676e6cd852290"},"cell_type":"code","source":"data1 = data1.reindex_axis(sorted(data1.columns), axis=1)\nprint('Number of columns:', len(data1.columns))\nfor col in data1.columns:\n    print(\"'\"+col+\"',\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b98c80e9a3c4195f4128146667c6e85382ce9e35"},"cell_type":"markdown","source":"We will discuss the variables in alphabetical order as shown above.\n\n# 1. channelGrouping\nSTRING \tThe Default Channel Group associated with an end user's session for this View.  \nLets start with the first variable: `channelGrouping`.  \n`channelGrouping` is the channel through which the user came to the store.  \nWe will print out the unique values first:"},{"metadata":{"trusted":true,"_uuid":"4eed3803416fdbaa5564158f17572809554e2d36","_kg_hide-input":true},"cell_type":"code","source":"print(data1.channelGrouping.unique())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ebd4f4c049499093cfa075ce9a587f3fceb50aed"},"cell_type":"markdown","source":"First we describe the unique channels:  \n* 'Organic search' is a method for entering one or several search terms as a single string of text into a search engine.  \n* 'Referral' traffic is Google's method of reporting visits that came to your site from sources outside of its search engine. When someone clicks on a hyperlink to go to a new page on a different website, Analytics tracks the click as a referral visit to the second site.  \n* 'Paid search' marketing means you advertise within the sponsored listings of a search engine or a partner site.  \n* 'Affiliate' marketing is a marketing arrangement by which an online retailer pays commission to an external website for traffic or sales generated from its referrals.  \n* 'Direct' traffic is most often the result of a user entering a URL into their browser or using a bookmark to directly access the site. Essentially, Direct sessions occur any time Google Analytics cannot determine another referring source or channel.  \n* Display marketing channel can be made up of any number of traffic sources as long as the medium of the traffic sources is 'display', 'cpm' or 'banner' and 'Ad Distribution Network' matches 'content'.  \n* 'Social' marketing channel can be made up of any number of traffic sources as long as the medium of the traffic sources is 'social', 'social media', 'social-media', 'social network', or 'social-network'.  \n* Other: anything else.  \n \n\nNow lets Visualize:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e87dbcb4150a1eb09aabdd2f805abe6c4c01b337","scrolled":false},"cell_type":"code","source":"bar_plots('channelGrouping', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc027b36735d67ab746a855b5a25de775f07fbf1"},"cell_type":"markdown","source":"After reviewing the Figures above, I would advize the marketing team in this way:\n\n`Organic Search`\n* Has the highest proportion of user visits, but the 2nd highest proportion of buyers.  \n* The mean of the purchases made is not particularly high, but Organic search still makes 3rd most of our revenue due to sheer number.  \n* I hypothesize that when a user arrives by Organic Search, they may leave to view other options on the search engine.  \n* Users arriving through Organic Search are clearly important. Perhaps we can encourage more buying if we promote the product as *the* choice option. Clearly state why it is the 'recommeded' product. This might result in higher buying ratio like that for 'Referral'.  \n\n`Social`\n* Has a high portion of the traffic, but a tiny fraction of the purchases.  \n* Mean purchase amount is low, so is the portion of our revenue.  \n* It would be easy to write 'Social' off as worthless. However, this channel presents a unique opportunity! Why is it that our views are not translated into purchases?  \n* I reccomend the marketing team takes advantage of this huge traffic stream and focus on converting these views to purchases! This will require some in-depth research.\n\n`Direct`\n* 3rd highest traffic share, 3rd highest purchase share.  \n* The mean purchase amount is high, and this generates the 2nd most of our revenue.  \n* This is an important channel! I think some users save the URL and come back later to buy. How they see the URL before saving is most likely after an Organic Search, Social, or Referral, respectively. But we canot know for sure. The only thing we can do is as I recommended for 'Organic Search'. Promote the product as *the* choice.\n\n`Referral`\n* Has a fairly sizeable portion of the traffic, but a disproportionally large portion of purchases!  \n* Mean purchase amount is not particularly high, but this is where most of our revenue comes from currently.  \n* When a user is on some starting website and that site refers them to some product, then the user probably feels like the authors of the starting website has done the research and narrowing down of options on their behalf, and that they can trust their reccomendation. This explains the high purchase ratio.  \n* Referrals are arguably the most important for the company right now. Focus on promoting the products to authors of websites!!!\n\n`Display`\n* Has a small traffic share, and a small number of purchases.  \n* The mean purchase amount is, however, very high!!! Only due to the small number of visitors is the share of our revenue low.  \n* Here is another excellent opportunity for the marketing team! The Display channel certainly encourages users to spend highly. Perhaps the alure of the visual banner is enticing enough to convince users to spend highly?  \n* I reccomend increasing the number of these advertisements! Specifically target demohraphics that are likely to spend big! We will discover who these demograhics are later...\n\n`Paid Search`, `Affiliates`, and `Other`\n* These have low traffic share, and low number of our purchases.  \n* Their mean purchase and share of our revenue are likewise not very high.  \n* Im not sure that these are worhtwhile to expand on. Perhaps the 'Paid Search' can be said to have decent mean purchase amount. Maybe this is the only channel in this bunch worth considering seriously.  \n\n\n\n# 2. customDimensions\nRECORD \tThis section contains any user-level or session-level custom dimensions that are set for a session. This is a repeated field and has an entry for each dimension that is set.\n\n## 2.1. customDimensions_index\n INTEGER \tThe index of the custom dimension."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8e798b05f044b752ecd515df335aac2e567b2e78"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1.customDimensions_index.nunique(dropna=False))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"05f5592328d650c7e31b253191d5aaa2fa2cc4cb"},"cell_type":"markdown","source":"There are only 2 unique values, so this variable can be encoded as a boolean.  \nFor now lets explore this variable graphically:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2a4c8af3c68aea851afafc51f3927bdd238e50e8"},"cell_type":"code","source":"bar_plots('customDimensions_index', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce21f6cdab3bb78efaac1901ad0e10b0693b6e1f"},"cell_type":"markdown","source":"This feature seems to  have powerful predictive value!  \nConsider the disparity between the 'nan' category's count and purchase count.\n\n## 2.2. customDimensions_value\nSTRING \tThe value of the custom dimension.  \nLets see how many unique values there are, and visualize:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"608208204c295a53347147c683df8b6894bfa840"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1.customDimensions_value.nunique(dropna=False))\n\nbar_plots('customDimensions_value', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0be3ed7775083f53141b3e03ead6f0efea6ebf12"},"cell_type":"markdown","source":"This feature also seems to have predictive power!  \nIt seems that when the custom dimensions are set to 'North America', then sales are high relative to other settings.  \nWe could group 'North America' as a category against the category of all others.\n\n# 3. date\nSTRING \tThe date of the session in YYYYMMDD format.\nWe also neeed to look at the `date` feature.  \nIn order to compare the dates from the train and test sets, we need to load the test set's `date` column:  "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"cf173ed3b0fa28bd17341ed1cf505f16871bd924"},"cell_type":"code","source":"data2 = load_df(\"../input/test_v2.csv\", usecols=['date', 'fullVisitorId'])\n\nscatter_plots(data1, data2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a1fa2753d52a4eb1f3816e611b37051a8ac09bd"},"cell_type":"markdown","source":"What do we learn from this?\n* the testing data `data2` does not overlap in time with `data1`.  \n* This means we need to ensure that our testing and training sets throughout this notebook follows the rule:  \n* train set is followed by non-overlapping test set.\n* Lets keep this in mind for when we split our data later !!!!  \n\nThis feature is not useful in this format. Let us extract some useful features like:  \n* month: The month (1 to 12)\n* week: The week (1 to 52)\n* weekday: Return the day of the week as an integer, where Monday is 0 and Sunday is 6  \n\nWe can now drop `date` from further analysis."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a4114fa820f9b107e46ccdaff974738ab20ee1e5"},"cell_type":"code","source":"data1['date'] = data1['date'].apply(lambda x: dt.date(int(str(x)[:4]), int(str(x)[5:7]), int(str(x)[8:])))\n\n#% feature representation\ndata1.date = pd.to_datetime(data1.date, errors='coerce')\n\n#% feature extraction - time and date features\n\n# Get the month value from date\ndata1['month'] = data1['date'].dt.month\n# Get the week value from date\ndata1['week'] = data1['date'].dt.week\n# Get the weekday value from date\ndata1['weekday'] = data1['date'].dt.weekday\n\ndata1 = data1.drop(labels=['date'], axis=1)\ndata1.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8d971b774ca965aff80a4b94a4f8fd84dbc0dc2"},"cell_type":"markdown","source":"Now lets look at the extracted features graphically:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7579eb5a9d47827c633cbf9a4c1e6ed79a5b5897","scrolled":false},"cell_type":"code","source":"kde_scatter_plots('month', data1)\nkde_scatter_plots('week', data1)\nkde_scatter_plots('weekday', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"586212892e5b7f524f35a6afdbab10a6808a2a82"},"cell_type":"markdown","source":"# 4. device\nRECORD \tThis section contains information about the user devices.\n## 4.1. device_browser\n STRING \tThe browser used (e.g., \"Chrome\" or \"Firefox\").  \n Lets look at the data. First lets see how many unique browsers there were:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d4c3314da5bb28b34f2b9eabe26dcc83813b1826"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1.device_browser.nunique(dropna=False), 'out of', data1.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2e22150845027f42984690098735eba78fa867"},"cell_type":"markdown","source":"That is a lot of different browsers!  \nFar more than the common Chrome, Firefox, Opera, Safari, Internet Explorer, and Microsoft Edge that we are used to!  \nLets visualize the top 10.  \nI hypothesize that almost all of the traffic will be through these:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f95da1af27a5102e676a9ec54561f2cff7d06254"},"cell_type":"code","source":"bar_plots('device_browser', data1, n_bars=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c54e6973e1df68b6fc3bf36ae15aa4f65c083ff3"},"cell_type":"markdown","source":"The hypothesis was correct. The first 3 or 4 have nearly all the traffic.  \n* by far most of our traffic is from Chrome, and correspondingly, so is our number of purchasers.\n* Strangely, there is a *significantly* higher mean purchase for Firefox users over any other users. Other browsers are quite even in this category.\n* Unsurprisingly, most of our revenue is from Chrome users. This is mainly due to their sheer number.\n* If Firefox users visited the GStore more often, I hypothesize that they would become a large part of our revenue.\n* I reccomend GStore seriously focus on supporting Firefox users to encourage purchases by those users.  \nWe can categorize the lesser used browsers as one group.  \n\n## 4.2. device_deviceCategory\nSTRING \tThe type of device (Mobile, Tablet, Desktop)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"35f054d9a56a09784f92696889d5425fbdc4b9d9"},"cell_type":"code","source":"bar_plots('device_deviceCategory', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c78a33bcd759dff7a6b02223b9fbecab508aa6b7"},"cell_type":"markdown","source":"This visual is straightforward:\n* most users access the site with their desktop, follwed by mobile, and then tablet.\n* the count of purchases for these categories correspond accordingly, however, desktop has an even higher share on purchases that of pure counts.\n* the mean purchase amount is highest for desktop, at almost double that of tablet and mobile.\n* these factors account for the fact that the revenue generated is simply dominated by the desktop users.\n* marketing can keep focussing on desktop users.\n\n## 4.3. device_isMobile\nThis field is deprecated. Use device.deviceCategory instead. \t\nBOOLEAN \tIf the user is on a mobile device, this value is true, otherwise false.  \nThis is simply a boolean logic of the previous feature. We do not expect much new information:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2480f06983ccb16adac8eed495ff33b8b41ccc41"},"cell_type":"code","source":"bar_plots('device_isMobile', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32e8d0acb9e83f75d18215c5ead55f610e9c41bd"},"cell_type":"markdown","source":"Similar to the previous feature, we can say that Mobile users are the minority, and that they spend less on average than 'others'.  \nWe can remove this feature from our data.  \n\n## 4.4. device_operatingSystem\n STRING \tThe operating system of the device (e.g., \"Macintosh\" or \"Windows\")."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0d37b5c241fa0802fe2cfe7e70b1cdb43bfb9915"},"cell_type":"code","source":"del(data1['device_isMobile']) # remove\n\nprint('Number of unique values (incl. nans) is:', data1.device_operatingSystem.nunique(dropna=False))\nbar_plots('device_operatingSystem', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec35a24bd4be5cd1597103277bd614326d688097"},"cell_type":"markdown","source":"It is not a surprise that the visits are dominated by Windows, Macintosh, Android, iOS, and then Linux. These are the larger operating systems.  \n`Windows` users are the most frequent, and therefore it makes sense that they have the 2nd most purchases. They also have the 2nd highest mean purchase amount, and make 2nd most of our revenue. These users have potential - if marketing can convince a higher proportion of these users to purchase, then their contribution will almost certainly rival that of the Mac users.  \n`Macintosh` users are 2nd most frequent visitors, but dominate in purchases. They have the 3rd highest mean purchase amounts, and therefore are the primary source of our income. This is an important demographic! If marketing can manage to increase their mean purchase even slightly, this will result in massive earnings!  \n`Android` users are the 3rd highest count of visitors, but are unlikely to purchase. Their mean purchase is 4th highest, but due to low number of purchases they make very little of our revenue. Marketing needs to find a way to encourage a higher fraction of purchases to go through. Andoid users need to be supported, as they could otherwise have made a significant share of our income.  \n`iOS`: Similar to Android, they have a fairly large share of visitors, but the number of purchases are relatively small. Their mean purchase amount is only 6th highest, which all leads to a tiny share on our revenue. This is a difficult situation, because we need to encourage more purchases, but also higher mean spending amount. Perhaps special products could be designed for these users, but I doubt that the investment to do that will translate to significantly higher revenue generated.  \n`Linux` users are a small number, but they make 4th most of our sales. They also have 4th highest mean purchase amounts. Due to the small number of visitors, their revenue share is very small. Perhaps more Linux users need to be encouraged to visit the store.  \n`Chrome OS` is an interesting one! The number of users are small, but they are 3rd highest in number of sales! Their mean purchase amount is also the highest overall! They are, inceredibly, 3rd highest revenue generators. It is only due to their small number that they do not have a significantly higher revenue share. Chrome is part of Google, so it should not be too surprising that they are relevent in Google Store (GStore). Here, marketing should focus on promoting Chrome OS as a product. The technical team should maintiain excellent support for these users. If the number of users grow due to great technical support, then it would translate to a significant boost in income for GStore.  \n**All others**: Other OSs have an almost insignificant share of visitors, and therefore of sales. They do not seem to be particularly promising for investing in.  \nWe can categorize these lesser used OSs into one category.  \n\n# 5. fullVisitorId\nSTRING \tThe unique visitor ID (also known as client ID).  \nThere is not much to say about `fullVisitorId` at this point.  \nWe know that this is unique for each user. We can look at how many unique users we have:"},{"metadata":{"_uuid":"e0fb4a4b79716082ba9c8c0660c8bd883d23d9ef","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Number of unique visitors in train set : \",data1.fullVisitorId.nunique(), \" out of rows : \",data1.shape[0])\nprint(\"Number of unique visitors in test set : \",data2.fullVisitorId.nunique(), \" out of rows : \",data2.shape[0])\nprint(\"Number of common visitors in train and test set : \",len(set(data1.fullVisitorId.unique()).intersection(set(data2.fullVisitorId.unique())) ))\n\ndel(data2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"360d00decc69ef3b403f62a6ac5b27888ab7948b"},"cell_type":"markdown","source":"Evidently, most of the visits to the GStore are different users. There are some who return a few times.  \n\n# 6. geoNetwork\nRECORD \tThis section contains information about the geography of the user.\n\n# 6.1. geoNetwork_city\nSTRING \tUsers' city, derived from their IP addresses or Geographical IDs.  \nThere are a lot of cities on this planet. Lets find out how many there represented at GStore:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2987305c2123863b07f86f1b3e3de1335bc98c39"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1.geoNetwork_city.nunique(dropna=False), 'out of', data1.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"909fb2a43fc5665a8310bf8a53dda92c85213a2e"},"cell_type":"markdown","source":"That is too many cities to visualize on a bar chart. So let only look at the top... say... 25:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"978eecede571c791a5837bb4995afab6e04df55b"},"cell_type":"code","source":"bar_plots('geoNetwork_city', data1, n_bars=25)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5d7338d1eae98086d70176d0adeb078418273cd"},"cell_type":"markdown","source":"These figures are not particularly useful, and it feels like a bit of an information overload.  \nWhat we can infer is that the vast majority visits do not have the city location available. Altough mean purchases vary significantly, I doubt that this is indicative of some underlying truth, since the number of purchases for most cities are tiny, and as such the mean purchase amount os probably not a good indicator.  \nIt even difficult to comment with confidence on the fact that `not available in dataset` and `(not set)` seem to have different mean purchases. The `(not set)` category has vew sales to take the mean from. It is unclear whether these two categories should be combined, or whether they represent a significantly different type of user.   \n\nLets aggregate cities in terms of how many times they appear. We will also agrgate cities based on `totals_hits` and `totals_pageviews`.  \nWe print these new columns.  \n\nWe shall create a `make_countsum` function to aggregate features with high cardinality in terms of its frequency (count) as well as the sums of `totals_hits` and `totals_pageviews`.  Features with high cardinality are those with a large number of unique categories. High cardinality is a serious problem for Data Science problems."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"b8c7e31de64538ba3c3bd2dcc489b6b51addcbf8"},"cell_type":"code","source":"def make_countsum(df, dfstr):\n    df['totals_hits']=df['totals_hits'].fillna(0).astype('int')\n    df['totals_pageviews']=df['totals_pageviews'].fillna(0).astype('int')\n    \n    df[str(dfstr+'_count')] = df[dfstr]\n    df[str(dfstr+'_count')]=df.groupby(dfstr).transform('count')\n    \n    df[str(dfstr+'_hitssum')] = df.groupby(dfstr)['totals_hits'].transform('sum')\n    df[str(dfstr+'_viewssum')] = df.groupby(dfstr)['totals_pageviews'].transform('sum')\n    #del(df[dfstr])\n    return df\n\nprint('make_countsum function created')\n\ndata1 = make_countsum(data1, 'geoNetwork_city')\n\nprint( 'created geoNetwork_city_count,geoNetwork_city_hitssum,and geoNetwork_city_viewssum features')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4792140a1e0607136b2b5cef1b2793a7fd1a09da"},"cell_type":"markdown","source":"Now lets view these new features grahically:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"c519ea28bead8e0002ca2e4244aa18337790b78f"},"cell_type":"code","source":"bar_plots('geoNetwork_city_count', data1, n_bars=10)\nbar_plots('geoNetwork_city_hitssum', data1, n_bars=10)\nbar_plots('geoNetwork_city_viewssum', data1, n_bars=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d2034da616ebf478e905aad84000dff253157b0"},"cell_type":"markdown","source":"\n\n# 6.2. geoNetwork_metro\nSTRING \tThe Designated Market Area (DMA) from which sessions originate.  \nPerhaps if we zoom out slightly we can make better inferences? We will look at the top 20 metros:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5f7d5a8b03a7350833c21b77b231b7a9867417f2"},"cell_type":"code","source":"bar_plots('geoNetwork_metro', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d81e3d478c2123d58482a064ab6e775b9ec879b"},"cell_type":"markdown","source":"Unfortunately most of the metros are unavailable. We do however see a significant portion of San Francisco-Oakland-San Jose users. They, together with New York, seem to have a disproportionately large number of sales, and therefore revenue share.  \nI wuold recommend that GStore support these users in particular and perhaps encourage these users to promote GStore to their peers.   \nWe will be using the `make_countsum` function for this feature too.\n\n# 6.3. geoNetwork_region\nSTRING \tThe region from which sessions originate, derived from IP addresses. In the U.S., a region is a state, such as New York.  \nLets zoom out even more, to look at regions of the world."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2b06697ede8d29b61e98cfdfd82a4997af44ffdf"},"cell_type":"code","source":"bar_plots('geoNetwork_region', data1, 20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"618413a31bd55d2d2a4c2b0d585f1b1cded66226"},"cell_type":"markdown","source":"Similar to cities and metros, we have the issue that most regions are 'not available in dataset'.  \nPerhaps one inference we *can* make is that California, New York, Texas, Washington, and Illnois have a disproportionately high share of purchases. These regions clearly make most of our revenue (aside from the 'not available in dataset category').\nTake note that, again, there seems to be a difference between the 'not available in dataset', and the '(not set)' categories, however we cant be sure yet.  \nWe will be using the `make_countsum` function for this feature too.\n\n# 6.4. geoNetwork_continent\nSTRING \tThe continent from which sessions originated, based on IP address.  \nThere are just too many cities, metros, and regions. The low number of sales in each of these make our comparisons weak with uncertainty.  \nLets zoom out... way out... to look at the continents!"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f7e7a913d4c60c71e297450173d5f389a4dacbd3"},"cell_type":"code","source":"bar_plots('geoNetwork_continent', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3136de943b5448cd74be7ce608172a0c05214643"},"cell_type":"markdown","source":"Hmmm... not what i expected.  \nLets break down what our inferences are:\n* The majority of our visitors are from the Americas, Europe, and Asia.\n* The Americas totally dominate in number of purchases. In fact the number of purchases from other continents are surprisingly small!\n* Aside from the Americas, the estimate for mean purchase amount is probably not very reliable.\n* Africa has an enormous mean purchase amount. Keep in mind however, that there were only 8 sales in Africa. A single high spender may have caused the mean purchase amout to inflate to this extent.\n* Since the Americas are the vast majority of our sales numbers, they also contribute to nearly the entirety of our revenue.  \nMarketting should focus on converting European and Asian visits into purchases. This is currently a lost opportunity for revenue generation. Finally, the americas should be supported technically, and targetted by consistent marketing to keep the numbers up.   \nLooks like 'Americas' can be one category and all others can be grouped into another.\n\n# 6.5. geoNetwork_subContinent\n STRING \tThe sub-continent from which sessions originated, based on IP address of the visitor.  \n Perhaps if we zoom in slightly, we can get a better idea of what is going on in some of the more relevant continents:"},{"metadata":{"trusted":true,"_uuid":"b927b4c82546b7934c174bf830875efa56290206","_kg_hide-input":true},"cell_type":"code","source":"bar_plots('geoNetwork_subContinent', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1580eb478a290d55abdc072d1305038a313d5817"},"cell_type":"markdown","source":"Somehow it is not very surprising that it is Northern America that is driving the Americas figures so highly. In fact, the Carribbean, Southern- and Central America seem underrepresented here.  \nWe have a number of subcontinents making a respectable share of visitors, but again, their sales numbers are completely overshadowed by Northern America.  \nSome exceptionally high mean purchases in Africa are still likely due to the aforementioned outliers. This can occur due to the tiny number of sales in these regions.  \nThe reccomendations are similar to before: subcontinents other than northern america need to be encouraged to purchase. Their respectable visitor numbers are not being utilized!  \nIt looks like we can again make a binary variable, where all categories except 'Northern America' can be grouped together.\n\n# 6.6. geoNetwork_country\nSTRING \tThe country from which sessions originated, based on IP address.  \nPerhaps subContinent is still too vague. Countries within the same continent can be *vastly* different! Lets look at countries.  \nFirst see how many countries are represented:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6b6100bd959cda4b6b77eefc5b00050de893c56a"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1.geoNetwork_country.nunique(dropna=False), 'out of', data1.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1d113f1bbe5d7feef17c0bf28f5142c05428bcc"},"cell_type":"markdown","source":"This is far too many to represent on a bar chart.  \nLets look at only the 20 most frequent countries..."},{"metadata":{"trusted":true,"_uuid":"97d2a7fdac0475d2845d8a0a9907ce5f036543a6","_kg_hide-input":true},"cell_type":"code","source":"bar_plots('geoNetwork_country', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e8353716754a315f1028a3f0b0b170b5d85b3c5"},"cell_type":"markdown","source":"Well, this really matches what we saw previously. The United States Seem to be dominating visiting representation, as well as number of purchases. However, it is still noticeable that other nations seem to visit the store, but not make purchases.  \n\nReally what all this has boiled down to is that marketing need to target countries with significant visits, who are not currently, for some reason, making purchases. India, the UK, Canada, *etc* have potential to become a significant source of revenue.  \n\nLets take a look at all countries now to see which to target for converting visits into purchases. We will also look at countries with large populations that have low visits for some reason. The best way to quickly look at this type of information is by looking at some globes:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"da93703243a46a5f3ea7d4f7390a11341608de45"},"cell_type":"code","source":"globe_plot(data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9d3341acdbae9f52236ec0d84b236d841c7909b"},"cell_type":"markdown","source":"Now we can quickly make inferences about any country in the world!  \n\nLets look in particular at the 10 nations in the world with the largest populations: China, India, USA, Indonesia, Brazil, Pakistan, Nigeria, Bangladesh, Russia, Mexico.  \nOf these nations, the USA clearly dominates in number of visitors. India features too, but at a number far lower than it should, given its population. The other large nations barely feature at all. This remains an untapped source of visitors that marketing needs to target!  \nIn order to diminish cardinality, we will be using the `make_countsum` function for this feature.\n\n# 6.7. geoNetwork_networkDomain\nSTRING \tThe domain name of user's ISP, derived from the domain name registered to the ISP's IP address.  \nThe domain name of user's ISP, derived from the domain name registered to the ISP's IP address.  \nThere may be very many of these.. lets see how many:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"497ee381f39ddb3fb3c2e0167655a9ad1f4d0e53"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1.geoNetwork_networkDomain.nunique(dropna=False), 'out of', data1.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2892941c4cc318d9527c78720348066b8c5c7942"},"cell_type":"markdown","source":"That is a lot of network domains...  \nLets only look at the top 20 for now:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e199c6aa2073c4b5d348f70a4cb99068fef56879"},"cell_type":"code","source":"bar_plots('geoNetwork_networkDomain', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57bdf989c62137ae9074f61991ed1f33f168aa6b"},"cell_type":"markdown","source":"Our data capturing records the majority of visits as having unset '(not set)' network domains, or 'unknown.unknown'.  \nThese may initially sound like they should be the same thing, however a look at the number of purchases plot shows that users with 'unknown.unknown' are far less likely to purchase than users with '(not set)'.  \nThis is a good example of why Visualization and EDA is important. We may have otherwise grouped these as one category, however they seem to be different, and may thus lead to predictive insight in our model.  \n\nOne interesting observation is the mean purchase amount of the 'comcastbusiness.net' domain. Users enterring the GStore through this domain are purchasing expensive products. The number of purchases for this domain are 294, so it is unlikely that the mean are unreasonably boosted by a few outliers.  \nLets take a quick look to be sure:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"53e75bb093c7b0e0d250f180a8b6ea1e1a610a77"},"cell_type":"code","source":"fig = plt.figure(figsize=(8,6))   # define (new) plot area\nax = fig.gca()   # define axis\nplt.suptitle('Boxplot for comcastbusiness.net')\ntemp = data1[data1.geoNetwork_networkDomain=='comcastbusiness.net'][['totals_transactionRevenue']]\nprint(temp.describe())\ntemp.boxplot(ax=ax)\ndel(temp)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50703862cfd6b7ccad055e7a86c28e574bdaa7e9"},"cell_type":"markdown","source":"Surprisingly, it seems that the mean may well be inflated due to some large outliers.  \nComparing the mean and median indicates that this is the case.  \nRegardless, the 'comcastbusiness.net' domain is significant, as it is the second highest source of revenue, despite having a tiny visitor share!  \nWe will be using the `make_countsum` function for this feature too.\n\n# 7. hits\nRECORD \tThis row and nested fields are populated for any and all types of hits.\n## 7.1. hits_appInfo\nRECORD \tThis section will be populated for each hit with type = \"APPVIEW\" or \"EXCEPTION\".\n\n### 7.1.1. hits_appInfo.exitScreenName\nSTRING \tThe exit screen of the session."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e62720844a73f766b3c7e4d947513e067df2295a"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1['hits_appInfo.exitScreenName'].nunique(dropna=False), 'out of', data1.shape[0])\nbar_plots('hits_appInfo.exitScreenName', data1, n_bars=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a929858429d089f000151fc94c5517950718331"},"cell_type":"markdown","source":"We will be using the `make_countsum` function for this feature too.\n\n### 7.1.2. hits_appInfo.landingScreenName\nSTRING \tThe landing screen of the session."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"cccb5b3ea9876ca834b6e29340e3a9dcfb2f6612"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1['hits_appInfo.landingScreenName'].nunique(dropna=False), 'out of', data1.shape[0])\nbar_plots('hits_appInfo.landingScreenName', data1, n_bars=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca812e9bcd5c87760964a0bf4ae09ade3f549bec"},"cell_type":"markdown","source":"We will be using the `make_countsum` function for this feature too.\n\n### 7.1.3. hits_appInfo.screenDepth\nSTRING \tThe number of screenviews per session reported as a string. Can be useful for historgrams."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"d8854b7d56ddff5b0cc464e15891d3f12a1ab284"},"cell_type":"code","source":"print('Number of unique values (incl. nans) is:', data1['hits_appInfo.screenDepth'].nunique(dropna=False), 'out of', data1.shape[0])\ndata1['hits_appInfo.screenDepth']=data1['hits_appInfo.screenDepth'].astype('str')\nbar_plots('hits_appInfo.screenDepth', data1, n_bars=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e8311b21d5feb270e6b8f95c6bb9154f66c8492"},"cell_type":"markdown","source":"This feature does not seem useful at all. Lets drop it !  \n\n### 7.1.4. hits_appInfo.screenName\nSTRING  The name of the string."},{"metadata":{"trusted":true,"_uuid":"920f658eb306f2edb000fa41999bbee35954efea","_kg_hide-input":true},"cell_type":"code","source":"del(data1['hits_appInfo.screenDepth'])\n\nprint('Number of unique values (incl. nans) is:', data1['hits_appInfo.screenName'].nunique(dropna=False), 'out of', data1.shape[0])\nbar_plots('hits_appInfo.screenName', data1, n_bars=10)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"59d2501c07e3c66fbf12999453486ccb5fb3da2b"},"cell_type":"markdown","source":"We will be using the `make_countsum` function for this feature too.\n\n## 7.2. hits_contentGroup\nRECORD \tThis section contains information about content grouping.\n\n### 7.2.1. hits_contentGroup.contentGroup1\n\nhits.contentGroup.contentGroupX   :    STRING \tThe content group on a property. A content group is a collection of content that provides a logical structure that can be determined by tracking-code or page-title/URL regex match, or predefined rules. (Index X can range from 1 to 5.)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"cdd8f76ffbf55a76512662461d7352279a7d7869"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroup1'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroup1', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1bd95e4d1dfc16ce7003f16046f27e61efc91437"},"cell_type":"markdown","source":"This feature seems weak, and will be dropped!  \n\n### 7.2.2. hits_contentGroup.contentGroup2"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6ffe4b420a3463618420d6474b80d76caad984ee"},"cell_type":"code","source":"del(data1['hits_contentGroup.contentGroup1'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroup2'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroup2', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5c2b0c58ba0610e371c2f1ad71a8f42b66ebc4d"},"cell_type":"markdown","source":"### 7.2.3. hits_contentGroup.contentGroup3"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"3c4e7b60d7363c801e4c50a5a3f9977f65f14df5"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroup3'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroup3', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c046051d5dbe97311fc1195a981e48691eae22d4"},"cell_type":"markdown","source":"This faeture does not seem to have much redictive power, it will be dropped!\n\n### 7.2.4. hits_contentGroup.contentGroup4"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4c439b7ce02b4073da349e1d96e34d60db426955"},"cell_type":"code","source":"del(data1['hits_contentGroup.contentGroup3'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroup4'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroup4', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"988e983c1c8999ab151be1320701173a6241a052"},"cell_type":"markdown","source":"This feature seems weak in predictive power and will be dropped!\n\n### 7.2.5. hits_contentGroup.contentGroup5"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"aafa3585f65bfd232c66b7ad23a3fd1dfa90da16"},"cell_type":"code","source":"del(data1['hits_contentGroup.contentGroup4'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroup5'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroup5', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9eb22671014e6c62fb526943fca67d9ad1c6241"},"cell_type":"markdown","source":"This feature seems the same as `hits_contentGroup.contentGroup4` and will be dropped!\n\n### 7.2.6. hits_contentGroup.contentGroupUniqueViews1\n\nhits.contentGroup.contentGroupUniqueViewsX \t:   STRING \tThe number of unique content group views. Content group views in different sessions are counted as unique content group views. Both the pagePath and pageTitle are used to determine content group view uniqueness. (Index X can range from 1 to 5.)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"102e5539f199ca57dd6355d3609e1e756ee8dc91"},"cell_type":"code","source":"del(data1['hits_contentGroup.contentGroup5'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroupUniqueViews1'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroupUniqueViews1', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a92b721531dcae9720eab318eb55a9319806c076"},"cell_type":"markdown","source":"This feature does not seem to have predictive power. It will be dropped.\n\n### 7.2.7. hits_contentGroup.contentGroupUniqueViews2"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8122aea7f0ffe995f36954df22c3fad997ebe4f6"},"cell_type":"code","source":"del(data1['hits_contentGroup.contentGroupUniqueViews1'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroupUniqueViews2'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroupUniqueViews2', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"79185239814a4feec01ca724446003ab103cf582"},"cell_type":"markdown","source":"### 7.2.8. hits_contentGroup.contentGroupUniqueViews3"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f5d8dcb33b7a94298e7306e6ccb4ddb9f5b1773f"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_contentGroup.contentGroupUniqueViews3'].nunique(dropna=False))\nbar_plots('hits_contentGroup.contentGroupUniqueViews3', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"224ad12973dda37b09067c0ee0defbb659dfd90d"},"cell_type":"markdown","source":"This feature is not powerful. Lets drop it.\n\n### 7.2.9. hits_contentGroup.previousContentGroup1\n\nhits.contentGroup.previousContentGroupX   :\tSTRING \tContent group that was visited before another content group. (Index X can range from 1 to 5.)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5ab0610c26f5de034c9dc95f4a0bbd0a6da225ea"},"cell_type":"code","source":"del(data1['hits_contentGroup.contentGroupUniqueViews3'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.previousContentGroup1'].nunique(dropna=False))\nbar_plots('hits_contentGroup.previousContentGroup1', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92a0faa262a3e431301b1a4aecd1c66e698e5389"},"cell_type":"markdown","source":"This feature is not discriminative, so it will be dropped.\n\n### 7.2.10. hits_contentGroup.previousContentGroup2"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"75ab0b57c75650c97b8f79ab90854585b7a7dcc7"},"cell_type":"code","source":"del(data1['hits_contentGroup.previousContentGroup1'])\n\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.previousContentGroup2'].nunique(dropna=False))\nbar_plots('hits_contentGroup.previousContentGroup2', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b8abe6cb6f7e42135c7c1fe96220ea996e118d3"},"cell_type":"markdown","source":"Similar to group1, group2 is not powerful, and so it will be dropped. \n\n### 7.2.11. hits_contentGroup.previousContentGroup3"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9a5a29883f21208fd025ad8b6942ee716478bd0f"},"cell_type":"code","source":"del(data1['hits_contentGroup.previousContentGroup2'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.previousContentGroup3'].nunique(dropna=False))\nbar_plots('hits_contentGroup.previousContentGroup3', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4be90c3239fad4be20594cb02bcde7b89a7cd07e"},"cell_type":"markdown","source":"Group3 is also not helpful, and so it will be dropped.\n\n### 7.2.12. hits_contentGroup.previousContentGroup4"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8229102a88b7a87af7f1586ef07a04213b4ee6d9"},"cell_type":"code","source":"del(data1['hits_contentGroup.previousContentGroup3'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.previousContentGroup4'].nunique(dropna=False))\nbar_plots('hits_contentGroup.previousContentGroup4', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca74af8f08ab14b07027d8f19246cb6019461846"},"cell_type":"markdown","source":"Group4 is also useless and will be dropped.\n\n### 7.2.13. hits_contentGroup.previousContentGroup5"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"289168b3ae3b6b6e912064d6acd2ac0e119e74bb"},"cell_type":"code","source":"del(data1['hits_contentGroup.previousContentGroup4'])\n\nprint('Number of unique values (incl. nan):', data1['hits_contentGroup.previousContentGroup5'].nunique(dropna=False))\nbar_plots('hits_contentGroup.previousContentGroup5', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d60ad9487bc6ac4c824f66a5a27e659bdfce4934"},"cell_type":"markdown","source":"Group5 also does not offer anything new. These were all similar and pointless. This feature will be dropped.\n\n## 7.3. hits_customDimensions\nRECORD \tThis section contains any hit-level custom dimensions. This is a repeated field and has an entry for each dimension that is set."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4111330008ffd441ee6fddfbea1e1b5809ff02b0"},"cell_type":"code","source":"del(data1['hits_contentGroup.previousContentGroup5'])\n\nprint('Number of unique values (incl. nan):', data1['hits_customDimensions'].astype('str').nunique(dropna=False)) # convert to str otherwise error\nbar_plots('hits_customDimensions', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9654b96cbece5166457bd7e2f555f9260bc17c91"},"cell_type":"markdown","source":"Just like the features before, this is not usefull and will be dropped.\n\n## 7.4. hits_customMetrics\nRECORD \tThis section contains any hit-level custom metrics. This is a repeated field and has an entry for each metric that is set."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9b24ffb610fcbc4ec252324bfb1e1ab5cc2f4ea5"},"cell_type":"code","source":"del(data1['hits_customDimensions'])\n\nprint('Number of unique values (incl. nan):', data1['hits_customMetrics'].astype('str').nunique(dropna=False)) # convert to str otherwise error\nbar_plots('hits_customMetrics', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d17f7a9fbd863fd9a33eda137002cc4e7c285f1f"},"cell_type":"markdown","source":"Similar to all those before... this is not useful and will be dopped.\n\n## 7.5. hits_customVariables\nRECORD \tThis section contains any hit-level custom variables. This is a repeated field and has an entry for each variable that is set."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"44dc3d4fb6d0afc7d996e67d29963f24d5385953","scrolled":false},"cell_type":"code","source":"del(data1['hits_customMetrics'])\n\nprint('Number of unique values (incl. nan):', data1['hits_customVariables'].astype('str').nunique(dropna=False)) # convert to str otherwise error\nbar_plots('hits_customVariables', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3190f7b0d1f35a80a3cc6565bb2a0fde1c7f0efb"},"cell_type":"markdown","source":"It seems all the 'custom' fields were not usefull. This faeture shall be dropped.\n\n## 7.6. hits_dataSource\nSTRING \tThe data source of a hit. By default, hits sent from analytics.js are reported as \"web\" and hits sent from the mobile SDKs are reported as \"app\"."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b553d875200298d3416b7dab13ff6e7c3a5f4386"},"cell_type":"code","source":"del(data1['hits_customVariables'])\n\nprint('Number of unique values (incl. nan):', data1['hits_dataSource'].nunique(dropna=False))\nbar_plots('hits_dataSource', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd1eca951fb4cac68d7e4ebd8b2d211c53c8181b"},"cell_type":"markdown","source":"## 7.7. hits_eCommerceAction\nRECORD \tThis section contains all of the ecommerce hits that occurred during the session. This is a repeated field and has an entry for each hit that was collected.\n\n### 7.7.1. hits_eCommerceAction.action_type\nSTRING \t\n\nThe action type. Click through of product lists = 1, Product detail views = 2, Add product(s) to cart = 3, Remove product(s) from cart = 4, Check out = 5, Completed purchase = 6, Refund of purchase = 7, Checkout options = 8, Unknown = 0.  \nUsually this action type applies to all the products in a hit, with the following exception: when hits.product.isImpression = TRUE, the corresponding product is a product impression that is seen while the product action is taking place (i.e., a \"product in list view\")."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9e6eb92f61160a84e293b7faec1d4dde4de197ef"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_eCommerceAction.action_type'].nunique(dropna=False))\nbar_plots('hits_eCommerceAction.action_type', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b35a2bf4c29cbdc3eec6d3df007c0675a2ce7ed8"},"cell_type":"markdown","source":"This feature does not have sufficient observations in its categories to be usefull. Lets drop it.\n\n### 7.7.2. hits_eCommerceAction.step\nINTEGER \tThis field is populated when a checkout step is specified with the hit."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"927de249ac358032f013d72af1f30a46f5c84a26"},"cell_type":"code","source":"del(data1['hits_eCommerceAction.action_type'])\n\nprint('Number of unique values (incl. nan):', data1['hits_eCommerceAction.step'].nunique(dropna=False))\nbar_plots('hits_eCommerceAction.step', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b7b2b5e87e2fc9f2cffa3f6b746d2d7672e2105"},"cell_type":"markdown","source":"Again, not enough observations in its categories to be useful. Lets drop it.\n\n## 7.8.  hits_eventInfo\nRECORD \tThis section is populated for each hit with type = \"EVENT\".\n\n### 7.8.1. hits_eventInfo.eventAction\nSTRING \tThe event action."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d7d856a2791807aa2d3a0df01bd3543dd965ec67"},"cell_type":"code","source":"del(data1['hits_eCommerceAction.step'])\n\nprint('Number of unique values (incl. nan):', data1['hits_eventInfo.eventAction'].nunique(dropna=False))\nbar_plots('hits_eventInfo.eventAction', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b94d6f7f489c0e912b76d8f1a55fa5672e69811a"},"cell_type":"markdown","source":"This faeture also does not have enough observations in its categories. Lets drop it.\n\n### 7.8.2. hits_eventInfo.eventCategory\nSTRING \tThe event category."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e0f1641912ad15fb6a074c83409ac1ead77f420a"},"cell_type":"code","source":"del(data1['hits_eventInfo.eventAction'])\n\nprint('Number of unique values (incl. nan):', data1['hits_eventInfo.eventCategory'].nunique(dropna=False))\nbar_plots('hits_eventInfo.eventCategory', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97e20992ed75bf983b36c6bb60baa73be1f17fee"},"cell_type":"markdown","source":"### 7.8.3. hits_eventInfo.eventLabel\nSTRING \tThe event label."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ef57f52c4bb96d13eec8610a5bc4551f90adea21"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_eventInfo.eventLabel'].nunique(dropna=False))\nbar_plots('hits_eventInfo.eventLabel', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b58ebc8a30de38b450f7b8f90a9364d6fa56b5fd"},"cell_type":"markdown","source":"This is another high cardinality feature for which we shall use `make_countsum`.\n\n## 7.9. hits_exceptionInfo.isFatal\nBOOLEAN     If the exception was fatal, this is set to true."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0df8a8613b924a9d38704dd8ac97ed0bbcc396a1"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_exceptionInfo.isFatal'].nunique(dropna=False))\nbar_plots('hits_exceptionInfo.isFatal', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"248b8f248a3340af9054a2edb2c7020820879f4d"},"cell_type":"markdown","source":"Another feature with the same information as many before it, and it shall be removed.\n\n## 7.10. hits_experiment\n\nRECORD   This row and the nested fields are populated for each hit that contains data for an experiment."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f88859238b999d119d80efc767918daea9bba6ee"},"cell_type":"code","source":"del(data1['hits_exceptionInfo.isFatal'])\n\nprint('Number of unique values (incl. nan):', data1['hits_experiment'].astype('str').nunique(dropna=False))   # convert to 'str' otherwise error\nbar_plots('hits_experiment', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91e867bcbf2db0bc30b3a17efbf5d24eb70f39b3"},"cell_type":"markdown","source":"Similar to the previous feature, this feature shows the same useless info.\n\n## 7.11. hits_hitNumber\nINTEGER \tThe sequenced hit number. For the first hit of each session, this is set to 1."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"71a1fcbe54df378a45402ffd6c965cf9fb7fb7c0","scrolled":false},"cell_type":"code","source":"del(data1['hits_experiment'])\n\n#print('Number of unique values (incl. nan):', data1['hits_hitNumber'].nunique(dropna=False))\nkde_scatter_plots('hits_hitNumber', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f494f4fcad34b8dce41de1ef9bba90e7ec523ef"},"cell_type":"markdown","source":"## 7.12. hits_hour\nINTEGER \tThe hour in which the hit occurred (0 to 23)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1821acaeb38e5289c54180a344e60c6e4e310bef"},"cell_type":"code","source":"print('Unique values (incl. nan):', data1['hits_hour'].unique())\nprint('Number of nans:', data1.hits_hour.isnull().sum() )\nprint('The mode is:', ss.mode(data1.hits_hour) )\nprint(\"Replace nans with mode\")\ndata1.hits_hour.fillna(ss.mode(data1.hits_hour))\nkde_scatter_plots('hits_hour', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cfac690316c7352fdcf7fdd960af28be848a9d1"},"cell_type":"markdown","source":"## 7.13. hits_isEntrance\nBOOLEAN \tIf this hit was the first pageview or screenview hit of a session, this is set to true."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8196098a9c339b64dfd9e66edb5d9376e13770b3"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_isEntrance'].nunique(dropna=False))\nbar_plots('hits_isEntrance', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c779d9bedf54e4328d8466b11c733c98036d97cc"},"cell_type":"markdown","source":"## 7.14. hits_isExit\nBOOLEAN \tIf this hit was the last pageview or screenview hit of a session, this is set to true."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"54fa73fcd16c1714f2350c0665ae82461db89cac"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_isExit'].nunique(dropna=False))\nbar_plots('hits_isExit', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d50de0db0ef8d76079ee9857981c5a22cefa8d5"},"cell_type":"markdown","source":"## 7.15. hits_isInteraction\nBOOLEAN \tIf this hit was an interaction, this is set to true. If this was a non-interaction hit (i.e., an event with interaction set to false), this is false."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7f070415fbd8d49fc3d75c3133d9b714c7506edf"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_isInteraction'].nunique(dropna=False))\nbar_plots('hits_isInteraction', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0da80cbc9979c8b5a7268acc0d86bdf1a7a8236"},"cell_type":"markdown","source":"This is yet another fearture like those in the past. It offers no power and will be dropped.\n\n## 7.16. hits_item.currencyCode\nhits_item: RECORD \tThis section will be populated for each hit with type = \"ITEM\".\ncurrencyCode:   STRING \tThe local currency code for the transaction."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dfc784a5767d3c6634d928fabf9920a8649dbef0"},"cell_type":"code","source":"del(data1['hits_isInteraction'])\n\nprint('Number of unique values (incl. nan):', data1['hits_item.currencyCode'].nunique(dropna=False))\nbar_plots('hits_item.currencyCode', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5fd023f07596ef6dac80ae9d488e1fc4c6a2f973"},"cell_type":"markdown","source":"## 7.17. hits_minute\nINTEGER \tThe minute in which the hit occurred (0 to 59).  \nLogically, the minute over all hours should not have any statistically significant influence on whether a purchase is made.  \nLets investigate this nonetheless:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"01a091cc1e9cc1f009fd2d308778fb6c4a4543a5"},"cell_type":"code","source":"kde_scatter_plots('hits_minute', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1977267ba79cacfa30c594eff9ab177eb3ecc305"},"cell_type":"markdown","source":"As expected. there seems to be no discernable pattern or logical indicator. Lets drop this feature immediately.\n\n## 7.18. hits_page\nThis section is populated for each hit with type = \"PAGE\".\n\n### 7.18.1. hits_page.hostname\nSTRING \tThe hostname of the URL."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"8eca49aca6bd9cda88b69aba546e080064c2cdd0","_kg_hide-input":true},"cell_type":"code","source":"del(data1['hits_minute'])  # drop immediately\n\nprint('Number of unique values (incl. nan):', data1['hits_page.hostname'].nunique(dropna=False))\nbar_plots('hits_page.hostname', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74c19ca88ba23fc38d232601b7f8f2a6f5d0e222"},"cell_type":"markdown","source":"### 7.18.2. hits_page.pagePath\nSTRING \tThe URL path of the page."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"d317bbb1d08e58bbd24158e97a587c8a73862485"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.pagePath'].nunique(dropna=False))\n#bar_plots('hits_page.pagePath', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3454f2f960b67ac944e209d79104bc506732b5bc"},"cell_type":"markdown","source":"We will have to use the `make_countsum` function to change this high cardinality feature to numeric features.\n\n### 7.18.3. hits_page.pagePathLevel1\nSTRING \tThis dimension rolls up all the page paths in the 1st hierarchical level in pagePath."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"941584d499c5fb5010662bf8ffaec2cf6133038e"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.pagePathLevel1'].nunique(dropna=False))\n#bar_plots('hits_page.pagePathLevel1', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9ff3ac48df06ff95ef48ed300f439c6e41b47ac"},"cell_type":"markdown","source":"We can use `make_countsum` to numerize this feature.\n\n### 7.18.4. hits_page.pagePathLevel2\nSTRING \tThis dimension rolls up all the page paths in the 2nd hierarchical level in pagePath."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"97938968a0808ff0bd4d5a6bf67c8ec83cbfb4cc"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.pagePathLevel2'].nunique(dropna=False))\n#bar_plots('hits_page.pagePathLevel2', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3eb3c877133824cde264f255ce3c04bf40f385db"},"cell_type":"markdown","source":"We shall use `make_countsum` here too.\n\n### 7.18.5. hits_page.pagePathLevel3\nSTRING \tThis dimension rolls up all the page paths in the 3rd hierarchical level in pagePath."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"63812a2018b0845ac7c934d8aa95522ba71e01f7"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.pagePathLevel3'].nunique(dropna=False))\n#bar_plots('hits_page.pagePathLevel3', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"841abe736f20c4a61f6428dec49c40833c38ace3"},"cell_type":"markdown","source":"We shall use `make_countsum` here too.\n\n### 7.18.5. hits_page.pagePathLevel4\nSTRING \tThis dimension rolls up all the page paths in the 4th hierarchical level in pagePath."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c0ce28cc69b8b8a826323a21882bc32c5700ca91"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.pagePathLevel4'].nunique(dropna=False))\n#bar_plots('hits_page.pagePathLevel4', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58f8d6c65d7f703fb463a726d1223a1a7172ab0d"},"cell_type":"markdown","source":"We shall use `make_countsum` here too.\n\n### 7.18.6. hits_page.pageTitle\nSTRING \tThe page title."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"5fee943a1a8a8583cab5cfa288ed36757be56bcc"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.pageTitle'].nunique(dropna=False))\n#bar_plots('hits_page.pageTitle', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf1a3f5ce261778b7d026efdeb050a46ec61a7d0"},"cell_type":"markdown","source":"Again, we will have to use `make_countsum`\n\n### 7.18.7. hits_page.searchCategory\nSTRING \tIf this was a search-results page, this is the category selected."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"d41c2ac64f4a05cdea8dddbeb099855039d6eba1"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_page.searchCategory'].nunique(dropna=False))\nbar_plots('hits_page.searchCategory', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75bbd5c86e59cc13e9b8d991df291e54f0639d67"},"cell_type":"markdown","source":"This feature seems powerfless, and shall be deleted.\n\n### 7.18.8. hits_page.searchKeyword\nSTRING \tIf this was a search results page, this is the keyword entered."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8daf1bb86cec2d4743c4ec91135b097e8550f8e8","scrolled":false},"cell_type":"code","source":"del(data1['hits_page.searchCategory'])\n\nprint('Number of unique values (incl. nan):', data1['hits_page.searchKeyword'].nunique(dropna=False))\nbar_plots('hits_page.searchKeyword', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23d94005d5e51c6b2c3381b00dd2c92d435a5a53"},"cell_type":"markdown","source":"Ths feature seems to have some low representation in all but 1 categories. Lets delete it.\n\n## 7.19. hits_promotionActionInfo\nRECORD \tThis row and nested fields are populated for each hit that contains Enhanced Ecommerce PROMOTION action information.\n### 7.19.1. hits_promotionActionInfo.promoIsClick\nBOOLEAN \tTrue if the Enhanced Ecommerce action is a promo click."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ba66d9c1f07b9bf6e8a18c90dd3de0e883dbe415","scrolled":false},"cell_type":"code","source":"del(data1['hits_page.searchKeyword'])\n\n#print('Number of unique values (incl. nan):', data1['hits_promotionActionInfo.promoIsClick'].nunique(dropna=False))\nbar_plots('hits_promotionActionInfo.promoIsClick', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32450a492912e3fad0c6d64e05d33653ef81e79a"},"cell_type":"markdown","source":"This feature seems low power. Lets delete it,\n\n### 7.19.3. hits_promotionActionInfo.promoIsView\nBOOLEAN \tTrue if the Enhanced Ecommerce action is a promo view."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"55991c8f9671c3b338da62ddb3af89be1bb6ee78"},"cell_type":"code","source":"del(data1['hits_promotionActionInfo.promoIsClick'])\n\n#print('Number of unique values (incl. nan):', data1['hits_promotionActionInfo.promoIsView'].nunique(dropna=False))\nbar_plots('hits_promotionActionInfo.promoIsView', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cb3b9d98c163d9747fcb3cb986275ad808324db"},"cell_type":"markdown","source":"## 7.20. hits_publisher_infos\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0d1e2780c82a15183e40753f3673e33da8a98fd2"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_publisher_infos'].astype('str').nunique(dropna=False))\nbar_plots('hits_publisher_infos', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d63ce2500e3eb21796410462e28f09c20eb4e83"},"cell_type":"markdown","source":"This feature is similar to many before, it offers no predictive power. Lets delete it.\n\n## 7.21. hits_referer\nSTRING \tThe referring page, if the session has a goal completion or transaction. If this page is from the same domain, this is blank."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"145b67d5065bb92113c8608f468c5455ea96afa9"},"cell_type":"code","source":"del(data1['hits_publisher_infos'])\n\nprint('Number of unique values (incl. nan):', data1['hits_referer'].nunique(dropna=False))\n#bar_plots('hits_referer', data1, n_bars=15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5f4036cb6d021e5ac29c045c6a135fb39977e03"},"cell_type":"markdown","source":"We shall use `make_countsum` for this feature.\n\n## 7.22. hits_social\nRECORD \tThis section is populated for each hit with type = \"SOCIAL\".  \n\n### 7.22.1. hits_social.hasSocialSourceReferral\nSTRING \tA string, either Yes or No, that indicates whether sessions to the property are from a social source."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"586b02d1afb7200abc1f6d7b2c3dcd34a17e2344"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_social.hasSocialSourceReferral'].nunique(dropna=False))\ndata1['hits_social.hasSocialSourceReferral']=data1['hits_social.hasSocialSourceReferral'].astype('str')\nbar_plots('hits_social.hasSocialSourceReferral', data1, n_bars=15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f01614473fb042ce0027ee818d06252cb5be51c1"},"cell_type":"markdown","source":"### 7.22.2. hits_social.socialInteractionNetworkAction\nSTRING \tFor social interactions, this represents the social network being tracked."},{"metadata":{"trusted":true,"_uuid":"553d2360e39f23b2b85bd0dbb9c8dc236bc65ee1","_kg_hide-input":true},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_social.socialInteractionNetworkAction'].nunique(dropna=False))\nbar_plots('hits_social.socialInteractionNetworkAction', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"457e4ecc7e35c6f2220107c121aa97e3f31f3fbb"},"cell_type":"markdown","source":"Another repeat feature as before. We drop this feature immediately.\n\n### 7.22.3. hits_social.socialNetwork\nSTRING \tThe social network name. This is related to the referring social network for traffic sources; e.g., Google+, Blogger."},{"metadata":{"trusted":true,"_uuid":"473bc80eeab258674b27a192cb323e6f86c38ed0","_kg_hide-input":true},"cell_type":"code","source":"del(data1['hits_social.socialInteractionNetworkAction'])\n\nprint('Number of unique values (incl. nan):', data1['hits_social.socialNetwork'].nunique(dropna=False))\nbar_plots('hits_social.socialNetwork', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a3d5502d70b75b7aba1adb66920d9cca575dcef"},"cell_type":"markdown","source":"For this feature I reccommend grouping all categories (except 'not set' and 'Youtube) into 1 category. We will then have 3 categories.\n\n## 7.23. hits_time\nINTEGER \tThe number of milliseconds after the visitStartTime when this hit was registered. The first hit has a hits.time of 0 "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2f999a15e219575268942f927f87f17a8c3fb601"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_time'].nunique(dropna=False))\nbar_plots('hits_time', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"558f4d5a86422f4ddf9bc45f9daabb999568903d"},"cell_type":"markdown","source":"This is again a repeat of what we had before so many times. We will delete this feature immediately.\n\n## 7.24. hits_transaction.currencyCode\nSTRING \tThe local currency code for the transaction."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"58e16d117316138a676b5655971bf57bce94d8ee"},"cell_type":"code","source":"del(data1['hits_time'])\n\nprint('Number of unique values (incl. nan):', data1['hits_transaction.currencyCode'].nunique(dropna=False))\nbar_plots('hits_transaction.currencyCode', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"45cf2d7b05ec68e3a6ea2048118784168c5b78b2"},"cell_type":"markdown","source":"## 7.25. hits_type\nSTRING \tThe type of hit. One of: \"PAGE\", \"TRANSACTION\", \"ITEM\", \"EVENT\", \"SOCIAL\", \"APPVIEW\", \"EXCEPTION\"."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a1eab2862e53cb22cc0b509d45ba2edd3e780004"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['hits_type'].nunique(dropna=False))\nbar_plots('hits_type', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8119706551511f58b00f0351d5a082f2dbc05543"},"cell_type":"markdown","source":"# 8. totals\nRECORD \tThis section contains aggregate values across the session.\n## 8.1. totals_bounces\nINTEGER \tTotal bounces (for convenience). For a bounced session, the value is 1, otherwise it is null.  \nBounces are used to determine 'bounce rate':  \nBounce rate is the percentage of single page visits (or web sessions). It is the percentage of visits in which a person leaves your website from the landing page without browsing any further. Google analytics calculates and report the bounce rate of a web page and bounce rate of a website.  So if a visit has a bounce of 1, then the user did not browse any further after they landed on the page. If the user *did* browse further, then `totals_bounces` is nan.  \nLets find out how that affects sales:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4a17062a11e1aba3766e374415012631a1424664"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['totals_bounces'].nunique(dropna=False))\nbar_plots('totals_bounces', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0bee7caaf7d1be7945f9842132f93abd8da4b9d"},"cell_type":"markdown","source":"This seems incredibly powerful!  \nFrom our data, we can infer that\n* bounced sessions never have sales!\n* approximately half of store visits are bounced.\n* i hypothesize that many users who land in the store and do not see imediately what they want will simply leave.\n* Users should ideally be directed in a helpful way from the start and convinced that they are in the right place with the right product where they are now!\n* This recommendation matches the recommendation given under `channelGrouping` \n\n## 8.2. totals_hits\nINTEGER \tTotal number of hits within the session.  \n`totals_hits` is the total number of hits within the session.  \nGoogle Analytics counts hits as “interactions”, which sends data to Analytics and is recorded as a user activity.\n\nSome hit types can include:\n* Page (e.g. when a page is loaded on a website, or even inside a mobile application)\n* Event (e.g. when a user clicks play on a video)\n* Ecommerce (e.g. making a purchase online)\n* Social Interaction (e.g. clicking an embedded “Like” or “Retweet” button)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f1ff2eb060885d4f23bf9374c8805ee9d45c2f9e"},"cell_type":"code","source":"kde_scatter_plots('totals_hits', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18530d24d71884cab7d2ae02cea20788951745c1"},"cell_type":"markdown","source":"The inferences and observations to be made here are similar to that of `visitNumber`.\n* Users who interact more with the website seem to also be more inclined to purchase higher.\n* Perhaps the site should have embedded (do not take user to another site/page/tab) videos or infographics to encourage purchases.\n\n## 8.3. totals_newVisits\ntotals_newVisits: Total number of new users in session (for convenience). If this is the first visit, this value is 1, otherwise it is null. This is essentially a binned version of `visitNumber`"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d07b924667c7287f18dbd6ecb0bfe1c7b06f42bf"},"cell_type":"code","source":"bar_plots('totals_newVisits', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"922edada06a87e57637c85a9ec0aa13c646aa18a"},"cell_type":"markdown","source":"Our inferences here are:\n* Most visitors are first time visitors\n* Most purchases are made by returning visitors\n* Returning visitors spend a higher amount on average\n* Returning visitors make us most of our money\n\n## 8.4. totals_pageviews\nPageviews is the total number of pages viewed. Repeated views of a single page are counted. According to Google’s Google Analytics support site, a pageview is: \nA pageview (or pageview hit, page tracking hit) is an instance of a page being loaded (or reloaded) in a browser. Pageviews is a metric defined as the total number of pages viewed. The metric itself says nothing about how many visitors saw that page or how many times the page was viewed per session. It’s just the total number of pageviews per page."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"347449332a4b994ddbff13ed2bcf133966453261"},"cell_type":"code","source":"kde_scatter_plots('totals_pageviews', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a3a065139feb1248cdc5ecb3ef321b6726186b3"},"cell_type":"markdown","source":"The inferences are much of the same:\n* users generally see a given page only a few times per session\n* Purchases are therefore usually made for low pageviews\n* There seems to be some correlation between number of pageviews and average \n\nspending amount.\n* Most of our money comes, unsurprisingly, from low pageviews\n\n## 8.5. totals_sessionQualityDim\nINTEGER \tAn estimate of how close a particular session was to transacting, ranging from 1 to 100, calculated for each session. A value closer to 1 indicates a low session quality, or far from transacting, while a value closer to 100 indicates a high session quality, or very close to transacting. A value of 0 indicates that Session Quality is not calculated for the selected time range."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e236dd4e83b72e83a1382b60bd960b72ca7b8f11"},"cell_type":"code","source":"kde_scatter_plots('totals_sessionQualityDim', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"435eff481964db87e6fdf0fd1d07530efa9ba050"},"cell_type":"markdown","source":"This feature seems to be powerful in predicting whether a user makes a purchase !!! There also seems to be a correlation between this variable and mean- and sum of purchase amount.\n\n## 8.6. totals_timeOnSite\nINTEGER \tTotal time of the session expressed in seconds."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"40bb06293c256334a94b99c67cc753834353fd8f"},"cell_type":"code","source":"kde_scatter_plots('totals_timeOnSite', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2aac70d493b6ddbc737308ae3b3a95d639e2a2e"},"cell_type":"markdown","source":"## 8.7. totals_transactionRevenue"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"27b904de7595cd7c18782ae771c8ce1c0665f444"},"cell_type":"code","source":"data1['totals_transactionRevenue'].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55056e81200637249e0ad6208ed09fd7948d7670"},"cell_type":"markdown","source":"Lets see the distribution of revenue graphically:"},{"metadata":{"trusted":true,"_uuid":"aa011c0028b15870125fc598e5ba10520b77838b"},"cell_type":"code","source":"data1[\"totals_transactionRevenue\"] = data1[\"totals_transactionRevenue\"].astype('float')\n\n\ngdf = data1.groupby(\"fullVisitorId\")[\"totals_transactionRevenue\"].sum().reset_index()    # sum each user's revenue\nplt.figure(figsize=(8,6))\nplt.scatter(range(gdf.shape[0]), np.sort(np.log1p(gdf[\"totals_transactionRevenue\"].values)))    # take log of revenue\nplt.xlabel('index', fontsize=12)\nplt.ylabel('totals_transactionRevenue', fontsize=12)\nplt.show()\n\n# NOTE\n# Why are using natural log plus 1 (np.log1p) instead of natural log only (np.log) ?  \n# Answer: np.log1p is used to deal with log(0). np.log1p(0) = np.log(0+1) = 0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6a235d1ad943ce38b2cca350ac6c47d5e3fa65b"},"cell_type":"markdown","source":"The Figure above shows that only a small number of customers make all GStore's revenue.  \n\n## 8.8. totals_totalTransactionRevenue"},{"metadata":{"trusted":true,"_uuid":"c7e96b4fe5475d60b69f95c5213cc0ad474e0917","_kg_hide-input":true},"cell_type":"code","source":"del(data1['totals_totalTransactionRevenue'])\nprint('totals_totalTransactionRevenue deleted')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60c2dcbef211a29ca981602262ad927ffbab40a8"},"cell_type":"markdown","source":"## 8.9. totals_transactions\nINTEGER \tTotal number of ecommerce transactions within the session."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7a4f90b5cf453dcf040633fdf01db6c598b222a8"},"cell_type":"code","source":"#kde_scatter_plots('totals_transactions', data1)\nbar_plots('totals_transactions', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78ed4c631b69dfcdcd11386d4cb2d1ed578a8da6"},"cell_type":"markdown","source":"Low representation in categories suggest reducing to 2 categories: nan vs all else.\n\n# 9. trafficSource\nRECORD \tThis section contains information about the Traffic Source from which the session originated.\n## 9.1. trafficSource_adContent\nSTRING \tThe ad content of the traffic source. Can be set by the utm_content URL parameter. Lets look at the top 15 most frequent categories."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2de63969b781a3e6eba73d42a980d0b7ec2a420b"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_adContent'].nunique(dropna=False))\nbar_plots('trafficSource_adContent', data1, n_bars=15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f971c3ac7ff957807f4b97af18946d3068a0d4b8"},"cell_type":"markdown","source":"For this feature we could use `make_countsum` to deal with high cardinality.\n\n## 9.2. trafficSource_adwordsClickInfo\nRECORD \tThis section contains information about the Google Ads click info if there is any associated with this session. Analytics uses the last-click model.\n### 9.2.1. trafficSource_adwordsClickInfo.adNetworkType\nSTRING \tNetwork Type. Takes one of the following values: {“Google Search\", \"Content\", \"Search partners\", \"Ad Exchange\", \"Yahoo Japan Search\", \"Yahoo Japan AFS\", “unknown”}"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"56531bd3b9edf5ce899d8064cc12b8c116e3d6d3"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_adwordsClickInfo.adNetworkType'].nunique(dropna=False))\nbar_plots('trafficSource_adwordsClickInfo.adNetworkType', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39bc2f505092298687b809a5f75604876137d7a6"},"cell_type":"markdown","source":"### 9.2.2. trafficSource_adwordsClickInfo.gclId\nSTRING \tThe Google Click ID."},{"metadata":{"trusted":true,"_uuid":"a5c09ae524341893763a75722848db3bc62e81b2","_kg_hide-input":true},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_adwordsClickInfo.gclId'].nunique(dropna=False))\n#bar_plots('trafficSource_adwordsClickInfo.adNetworkType', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f77288095e793c350cc29f80cbdb34f64c8e4a73"},"cell_type":"markdown","source":"We will use `make_countsum` on this feature.\n\n### 9.2.3. trafficSource_adwordsClickInfo.page\nINTEGER \tPage number in search results where the ad was shown."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"51d19c269164510e54923896924f6bd1414233f4"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_adwordsClickInfo.page'].nunique(dropna=False))\nbar_plots('trafficSource_adwordsClickInfo.page', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"064fd0ac46aae04e3532c5240bfea5411eff2e39"},"cell_type":"markdown","source":"This feature does not seem very useful. Low representation in most categories. Lets delete it!\n\n### 9.2.4. trafficSource_adwordsClickInfo.slot\nSTRING \tPosition of the Ad. Takes one of the following values:{“RHS\", \"Top\"}"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ba8e792b62d2848c2f15f178eb0e4c6ce2248109"},"cell_type":"code","source":"del(data1['trafficSource_adwordsClickInfo.page'])\n\nprint('Number of unique values (incl. nan):', data1['trafficSource_adwordsClickInfo.slot'].nunique(dropna=False))\nbar_plots('trafficSource_adwordsClickInfo.slot', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a14f88f1f6d4112f4104d3667360b211f4bda36"},"cell_type":"markdown","source":"## 9.3. trafficSource_campaign\nSTRING \tThe campaign value. Usually set by the utm_campaign URL parameter."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3d50169700fe7d3f8428d7a707ffef5cbfd710d8"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_campaign'].nunique(dropna=False))\nbar_plots('trafficSource_campaign', data1, n_bars=15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a58f334e4307075a63a6ddcc6506d099e636ff4"},"cell_type":"markdown","source":"Low representation in all but 1 category makes this feature low power. Lets delete it.\n\n## 9.4. trafficSource_isTrueDirect\nBOOLEAN \tTrue if the source of the session was Direct (meaning the user typed the name of your website URL into the browser or came to your site via a bookmark), This field will also be true if 2 successive but distinct sessions have exactly the same campaign details. Otherwise NULL."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"78587cbe381e9fd7e8594a871a2a3697918ee2e7"},"cell_type":"code","source":"del(data1['trafficSource_campaign'])\n\nprint('Number of unique values (incl. nan):', data1['trafficSource_isTrueDirect'].nunique(dropna=False))\nbar_plots('trafficSource_isTrueDirect', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b859690760cbbe82e3ad77cb066748b7e0da67d"},"cell_type":"markdown","source":"## 9.5. trafficSource_keyword\nSTRING \tThe keyword of the traffic source, usually set when the trafficSource.medium is \"organic\" or \"cpc\". Can be set by the utm_term URL parameter."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"755419d27777de6ff3a79a422e10f34d5988058f"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_keyword'].nunique(dropna=False))\nbar_plots('trafficSource_keyword', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c84857e55be1f0d8057f2b84059fcb3005ec655b"},"cell_type":"markdown","source":"Low frrequenct in all but 1 category. Lets delete this feature.\n\n## 9.6. trafficSource_medium\nSTRING \tThe medium of the traffic source. Could be \"organic\", \"cpc\", \"referral\", or the value of the utm_medium URL parameter."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"38e6d7ef0c686310af3a81d9c3c24732d697cecf"},"cell_type":"code","source":"del(data1['trafficSource_keyword'])\n\nprint('Number of unique values (incl. nan):', data1['trafficSource_medium'].nunique(dropna=False))\nbar_plots('trafficSource_medium', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53dc44157ac484067c9fd0748ada2b2d580dbffa"},"cell_type":"markdown","source":"## 9.7. trafficSource_referralPath\nSTRING \tIf trafficSource.medium is \"referral\", then this is set to the path of the referrer. (The host name of the referrer is in trafficSource.source.)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3ee0156348331551bd93a0bfd6cb7107743908e0"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_referralPath'].nunique(dropna=False))\nbar_plots('trafficSource_referralPath', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b101f994d112841827055e7f7529fc1f79657ad9"},"cell_type":"markdown","source":"This feature could perhaps be reduced to 2 categories. With '/' being a category on its own.\n\n## 9.8. trafficSource_source\nSTRING \tThe source of the traffic source. Could be the name of the search engine, the referring hostname, or a value of the utm_source URL parameter."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"be62e176611a065162a227e235f6f4286b08786c"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['trafficSource_source'].nunique(dropna=False))\nbar_plots('trafficSource_source', data1, n_bars=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27eec7d9976c39fb12f1e5a66189adf083d70994"},"cell_type":"markdown","source":"This feature seems to hold some predictive power. We can for example see that 'youtube.com' does not contribute to purchases relative to its frequency of occurence.  We could perhaps group categories that are less frequent than 'Youtube' as 1 category.\n\n# 10. visitId\nINTEGER \tAn identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"031b2c85b9a4e024f3c6a1ab29cd1d72ef55b5ea"},"cell_type":"code","source":"print('For data1:')\nprint(\"Number of unique 'fullVisitorId' entries\", data1.fullVisitorId.nunique(dropna=False))\nprint(\"Number of unique 'visitId' entries\", data1.visitId.nunique(dropna=False))\nfull_Vis = data1.fullVisitorId + data1.visitId.astype('str')\nprint('Number of unique combinations (out of ',data1.shape[0],' possible):',full_Vis.nunique(dropna=False))\ndel(full_Vis)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6eefc0c58b575913696a6c719ffa089cbc45a8a7"},"cell_type":"markdown","source":"Oddly enough, combining `fullVisitorId` and `visitId` we do NOT obtain a totally unique identifier.\n\n# 11. visitNumber\nINTEGER \tThe session number for this user. If this is the first session, then this is set to 1."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3784736663c11d8c55d52d23bb597183d4e50b90"},"cell_type":"code","source":"print('Number of unique values (incl. nan):', data1['visitNumber'].nunique(dropna=False))\nkde_scatter_plots('visitNumber', data1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b00179c6e049540fb02653de6161da7b66614026"},"cell_type":"markdown","source":"# 12. visitStartTime\nINTEGER \tThe timestamp (expressed as POSIX time).  \n\nThis feature is useful when converted. We can extract the hour of the day from the timestamp. We shall call this feature `hour`. We can delete `visitStartTime` directly after extracting `hour`."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d7199227f20356fd4fcabf0b762bebc8207df326"},"cell_type":"code","source":"data1['visitStartTime'] = pd.to_datetime(data1['visitStartTime'], unit='s')\ndata1['hour'] = data1['visitStartTime'].dt.hour\nkde_scatter_plots('hour', data1)\n\ndata1 = data1.drop(labels=['visitStartTime'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a9e96a106383fd2f89061c74e5d31f1ba9685fc"},"cell_type":"markdown","source":"It is interesting to see that this is not the same as the `hits_hour` feature.  \n\nLets see the final feature list to be used in Part 3:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"aadb9e421c6acf6d35092764f42a4f793e6fbc72"},"cell_type":"code","source":"print('Number of columns:', len(data1.columns))\nfor col in data1.columns:\n    print(\"'\"+col+\"',\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56a2e7b5206df7a27c20f2ac302a49e9c286fbe8"},"cell_type":"markdown","source":"# Concluding remarks\nWe have visualized and explored the features in our dataset.  \n\nAcknowledgements  \nSpecial thank you to the following authors for their insightful kernels:  \nhttps://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize  \nhttps://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue  \nhttps://www.kaggle.com/jsaguiar/complete-exploratory-analysis-all-columns  \nhttps://www.kaggle.com/shivamb/exploratory-analysis-ga-customer-revenue  \nhttps://www.kaggle.com/yoshoku/gacrp-v2-starter-kit/code  \n\nI welcome comments and suggestions for improvement!  "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}