{"cells":[{"metadata":{"_uuid":"c9c6ca83599e511354494d7c8f2a43255449cced"},"cell_type":"markdown","source":"**More To Come. Stay Tuned. !!**\nIf there are any suggestions/changes you would like to see in the Kernel please let me know :). \n\nThis notebook will always be a work in progress. Please leave any comments about further improvements to the notebook! If you like it or it helps you , you can upvote and/or leave a comment :)."},{"metadata":{"_uuid":"6b8f2d7bda0ba3105fac6fa1bebedd4bb3dabfb4"},"cell_type":"markdown","source":"### Introduction\n\nGoogle Merchandise Store sells Google-branded T-shirts & hoodies as well as gifts like coffee mugs, tote bags & pens. The objective of this competetion is to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. The outcome can help in doing more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data.\n\n\n"},{"metadata":{"_uuid":"96c86ffb2b6a5f697646ff4d423e18fbc8512459"},"cell_type":"markdown","source":"**Loading Libraries**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport json\nimport bq_helper\nfrom pandas.io.json import json_normalize\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\nimport plotly.figure_factory as ff\nimport numpy as np\ninit_notebook_mode(connected = True)\nimport datetime\nimport missingno as msno\n\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a8712fd6d8e954f5bb133e9cf146a4f63c48e17"},"cell_type":"markdown","source":"### Read the Data\n\nUtility function whcih is used to flatten json. This function is taken from https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"json_cols = ['device', 'geoNetwork', 'totals', 'trafficSource']\n\nnan_list = [\"not available in demo dataset\",\n            \"unknown.unknown\",\n            \"(not provided)\",\n            \"(not set)\", \n            \"Not Socially Engaged\"] # this last one is borderline\nnan_dict = {nl:np.nan for nl in nan_list}\n\ndef load_data(file):\n    path = '../input/' + file\n    df = pd.read_csv(path, converters = {column: json.loads for column in json_cols},\n                     dtype={'fullVisitorId': str, 'date': str}, \n            parse_dates=['date'])\n    \n    for column in json_cols:\n        dataframe = json_normalize(df[column])\n        dataframe.columns = [f\"{column}_{subcolumn}\" for subcolumn in dataframe.columns]\n        df = df.drop(column, axis = 1).merge(dataframe, right_index = True, left_index = True)\n        df.replace(nan_dict, inplace=True)\n    return df\n\ntrain = load_data(\"train.csv\")\n#test = load_data('test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c49de5071bc403263069d8d1909dbcaf065a7be8"},"cell_type":"markdown","source":"### Glimpse of data"},{"metadata":{"trusted":true,"_uuid":"dea580b8e02b0024d421f385004acd64f9c65791","_kg_hide-input":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f84fe574e1250793c252ba5115718acf305c58bc"},"cell_type":"markdown","source":"### Utility functions for plots"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d438f60e2df1853edecf7ac25c9c4622ed9de2d4"},"cell_type":"code","source":"def plotlybar(labels1 = None, values1 = None, name1 = None,labels2 = None, values2 = None, name2 =None, markercolor1 = '#9ECAE1', markercolor2 = '#ff99ff', title= None, mode = 'group', orientation = 'v'):\n    trace1 = go.Bar(\n    x = labels1,\n    y = values1,\n    orientation = orientation, \n    text = values1,\n    name = name1,\n    textposition = 'auto',\n    marker=dict(\n        color=markercolor1#'rgb(58,200,225)',\n        )\n    )    \n    \n    if labels2 is not None:\n        trace2 = go.Bar(\n            x = labels2,\n            y = values2,\n            name = name2,\n            text = values2,\n            textposition = 'auto',\n            marker=dict(\n                color=markercolor2#'rgb(58,200,225)',\n            )\n        )\n        data = [trace1, trace2]\n    \n    else:\n        data = [trace1]\n    layout = go.Layout(title = title,xaxis=dict(tickangle=-25),\n    barmode=mode)\n    fig = go.Figure(data = data, layout = layout)\n    iplot(fig)\n    \ndef scatter_plot(x,y, title):\n    data = [go.Scatter(\n    x = x,\n    y = y,\n    mode = 'lines+markers',\n    name = 'lines+markers'\n    )]\n    layout = go.Layout(title = title,xaxis=dict(tickangle=-25))\n    fig = go.Figure(data = data, layout = layout)\n    iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c24d2c39f07ea1bf8b18190016a881018103eb0"},"cell_type":"markdown","source":"### Data Preparation\n\n#### Checking for missing data"},{"metadata":{"trusted":true,"_uuid":"e37835798e8472394a360d2b4cd00ded58918e7e"},"cell_type":"code","source":"total = train.isnull().sum().sort_values(ascending = True)\npercent = (train.isnull().sum() / train.isnull().count() * 100).sort_values(ascending = True)\nmissing_data = pd.concat([total, percent], axis = 1, keys = ['Total','Percent'])\nmissing_data","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9232aa98ed31d0de58d0849356eb95bf2ac4a57b"},"cell_type":"markdown","source":"### Statistical overview of data"},{"metadata":{"trusted":true,"_uuid":"21b31b3777c5f1ec7c1157161346855742cd66b4"},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84a1374c60f6f37ce3b4071389a5f67a7f1553a3"},"cell_type":"markdown","source":"### Drop columns\n\nThere are few columns which has constant values. Removing those columns"},{"metadata":{"trusted":true,"_uuid":"0b6bef18e8591b517ee69b3f444c8b6e302563e2"},"cell_type":"code","source":"cols_to_drop = [column for column in train.columns if train[column].nunique()==1]\ntrain = train.drop(cols_to_drop , axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9367a8cd238957f7857c843ca63ab2abeb2722ab"},"cell_type":"markdown","source":"### Data Exploration\n\n#### Target Variable"},{"metadata":{"trusted":true,"_uuid":"e08e583b811644683eec6b09edbef3792b7c9fbd"},"cell_type":"code","source":"train['month'] = train['date'].dt.month\ntrain['year'] = train['date'].dt.year\ntrain['day'] = train['date'].dt.weekday\ntrain['totals_transactionRevenue'] = pd.to_numeric(train['totals_transactionRevenue'])\ntrain['totals_transactionRevenue'] = train['totals_transactionRevenue'].fillna(0)\ntrain['totals_hits'] = pd.to_numeric(train['totals_hits']).fillna(0)\ntrain['totals_pageviews'] = pd.to_numeric(train['totals_pageviews']).fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ed513231001426be754117cc5af63df0160c68d"},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nsns.distplot(train[train['totals_transactionRevenue'] > 0]['totals_transactionRevenue'],color=\"darkgreen\",bins=50)\nplt.xlabel(\"total transaction revenue\");\nplt.title(\"Distribution of total transaction revenue (non-zeros)\");\nplt.figure(figsize=(12,6))\nsns.distplot(np.log1p(train[train['totals_transactionRevenue'] > 0]['totals_transactionRevenue']),color=\"darkgreen\",bins=50)\nplt.xlabel(\"total transaction revenue\");\nplt.title(\"Lograthemic Distribution of total transaction revenue (non-zeros)\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fc12d14d1e05a49a9f5fca993e623feffd6ede9"},"cell_type":"code","source":"train_ = train.loc[train['totals_transactionRevenue'] > 0]\nfig, ax1 = plt.subplots(figsize=(16, 8))\nplt.title(\"Revenue and Non Revenue transactions\");\ntrain.groupby(['date'])['totals_transactionRevenue'].count().plot(color='#E7638B')\nax1.set_ylabel('Transaction count', color='#E7638B')\nplt.legend(['Non-Revenue users'])\nax2 = ax1.twinx()\ntrain_.groupby(['date'])['totals_transactionRevenue'].count().plot(color='#7debca')\nax2.set_ylabel('Transaction count', color='#7debca')\nplt.legend(['Revenue users'], loc=(0.875, 0.9))\nplt.grid(False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a39dffedf6696ef5fe73458337e69554cd6a4dae"},"cell_type":"markdown","source":"### Visitors"},{"metadata":{"trusted":true,"_uuid":"6e5b785873141c02a81c3a4c90b625de58e1c42e"},"cell_type":"code","source":"group_by_date = train.groupby('date')['fullVisitorId'].size()\nscatter_plot(group_by_date.index, group_by_date.values, 'Visitors by date')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15f7e14c21f2c5b3929f84348d91082298f14167"},"cell_type":"code","source":"\nvisit_by_month = train['month'].value_counts()\nvisits_by_day = train['day'].value_counts()\nplotlybar(visit_by_month.index, visit_by_month.values, 'Month', markercolor1 = '#E7638B' , title = 'Visits By Month')\nplotlybar(visits_by_day.index[:10], visits_by_day.values[:10], 'Day', markercolor1 = '#7dd5eb' , title = 'Visits By Day')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16f5673336f78e149142c1ffa40135c65ae9bbcd"},"cell_type":"code","source":"train['time_of_visit'] = train['visitStartTime'].apply(lambda x : x % (24 * 3600) // 3600)\ntime_of_visit = train['time_of_visit'].value_counts()\n#time_of_visit\nplotlybar(time_of_visit.index, time_of_visit.values, 'Time', markercolor1 = '#7debca' , title = 'Time of Visit')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"10e2abb209f668b12bd06f6a3b6ba9748472c3d2"},"cell_type":"code","source":"Channels = train['channelGrouping'].value_counts()\nplotlybar(Channels.index, Channels.values, 'Channel Grouping', markercolor1 = '#E7638B' , title = 'Count of Channel Grouping')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ba71471bb270ed5afa8901827860e305475d022"},"cell_type":"code","source":"monthy_chnl_rev = train.groupby(['channelGrouping','month'])['totals_transactionRevenue'].mean().reset_index()\n\nfig = {\n    'data': [\n        {\n            'x' : monthy_chnl_rev[monthy_chnl_rev['channelGrouping']==channel]['month'],\n            'y': monthy_chnl_rev[monthy_chnl_rev['channelGrouping']==channel]['totals_transactionRevenue'],\n            'name': channel\n        } for channel in ['Organic Search', 'Social', 'Direct', 'Referral', 'Paid Search', 'Affiliates', 'Display', '(Other)']\n    ],\n    'layout': {\n        'title' : 'Month wise mean revenue from each Channel Grouping',\n        'xaxis': {'title': 'Month'},\n        'yaxis': {'title': \"Mean Revenue\"}\n    }\n}\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2d83ccf4a372a8a02fcc61f183e16613227817a"},"cell_type":"markdown","source":" Organic search had the most sessions, followed by Social , Direct and Referrel Search."},{"metadata":{"trusted":true,"_uuid":"da54c865ef9f8932a14c7a67ee98616fb8b42a0f"},"cell_type":"code","source":"Browsers = train['device_browser'].value_counts()\nplotlybar(Browsers.index[:10], Browsers.values[:10], 'Browsers', markercolor1 = '#7debca' , title = 'Types of browsers used')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"03327fbc72e4f1ce90f6ec02514e9d32435faa09"},"cell_type":"markdown","source":"* Most of there customers use  web browser Chrome to land on the site."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7adcb8db9287487383621dffe736663803c18033"},"cell_type":"code","source":"#device_deviceCategory\nDevice_category = train['device_deviceCategory'].value_counts()\nplotlybar(Device_category.index, Device_category.values, 'Device category', markercolor1 = '#7dd5eb' , title = 'Category of Devices')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bdb66e2631fece90f0203b8c1c86a9fac6fa29c3"},"cell_type":"code","source":"monthy_dev_rev = train.groupby(['device_deviceCategory','month'])['totals_transactionRevenue'].mean().reset_index()\n\nfig = {\n    'data': [\n        {\n            'x' : monthy_dev_rev[monthy_dev_rev['device_deviceCategory']==device]['month'],\n            'y': monthy_dev_rev[monthy_dev_rev['device_deviceCategory']==device]['totals_transactionRevenue'],\n            'name': device\n        } for device in ['desktop', 'mobile', 'tablet']\n    ],\n    'layout': {\n        'title' : 'Month wise mean revenue from each device category',\n        'xaxis': {'title': 'Month'},\n        'yaxis': {'title': \"Mean Revenue\"}\n    }\n}\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9e7064bba6c690857eb544008a1ac96f76cffb7"},"cell_type":"markdown","source":"The use of a desktop computer brings the most users which also gives higher revenues, followed by mobile and tablet devices"},{"metadata":{"trusted":true,"_uuid":"f6406657efc5660152146d1cd18c54ce233c3184"},"cell_type":"code","source":"os = train['device_operatingSystem'].value_counts()\nplotlybar(os.index, os.values, 'os', markercolor1 = '#7debca' , title = 'Different OS')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42f34fafe0738e85a428db44267a6d767d2e32b0"},"cell_type":"code","source":"#device_deviceCategory\ngeo_continent = train['geoNetwork_continent'].value_counts()\nplotlybar(geo_continent.index, geo_continent.values, 'continent', markercolor1 = '#7debca' , title = 'Country wise count')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4639c7fc12f0a00914d68874b274b65e155645b9"},"cell_type":"code","source":"#device_deviceCategory\ngeo_country = train['geoNetwork_country'].value_counts()\nplotlybar(geo_country.index[:10], geo_country.values[:10], 'country', markercolor1 = '#E7638B' , title = 'Top 10 countries')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78ef7ccd1a54a17653680b08948b181e12b24661"},"cell_type":"code","source":"network_country = train[\"geoNetwork_country\"].value_counts()\ncolorscale = [[0, '#14f962'], [0.005, '#15ed11'], \n              [0.01, '#5fe10f'], [0.02, '#a2d50d'], \n              [0.04, '#c9b50b'], [0.05, '#bd6909'], \n              [0.10, '#b12407'], [0.25, '#a50624'], [1.0, '#990557']]\n\ndata = [\n    dict(\n    type = 'choropleth',\n        autocolorscale = False,\n        colorscale = colorscale,\n        showscale = True,\n        locations = network_country.index,\n        z = network_country.values,\n        locationmode = \"country names\",\n        text = network_country.values,\n        marker = dict(\n            line = dict(color = '#fff' , width = 2)\n        )\n        \n    )\n]\n\nlayout = dict(\n    height = 700,\n    title = \"Geo Network countries\",\n    geo =dict(\n        showframe = True,\n        showocean = True,\n        oceancolor = '#0077be',\n        projection = dict(\n            type = 'orthographic',\n            rotation = dict(\n                lon = 60,\n                lat = 10\n            )\n        ),\n        lonaxis = dict(\n            showgrid = False,\n            gridcolor = 'rgb(102, 102, 102)'\n        ),\n        \n        lataxis = dict(\n            showgrid = False,\n            gridcolor = 'rgb(102,102,102)'\n        )\n    )\n)\n\nfig = dict(data= data, layout = layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0191f7a42dd8497ab4ae60f780f475476e689986"},"cell_type":"markdown","source":"Most of there customers are from english speaking countries. "},{"metadata":{"trusted":true,"_uuid":"26e1b6925fd470cefbf4a5379479c63783c6a3ed","_kg_hide-input":true},"cell_type":"code","source":"\nmonthy_geo_rev = train.groupby(['geoNetwork_continent','month'])['totals_transactionRevenue'].mean().reset_index()\n\nfig = {\n    'data': [\n        {\n            'x' : monthy_geo_rev[monthy_geo_rev['geoNetwork_continent']==geo]['month'],\n            'y': monthy_geo_rev[monthy_geo_rev['geoNetwork_continent']==geo]['totals_transactionRevenue'],\n            'name': geo\n        } for geo in ['Americas', 'Asia', 'Europe','Oceania','Africa']\n    ],\n    'layout': {\n        'title' : 'Month wise mean revenue from each geography region',\n        'xaxis': {'title': 'Month'},\n        'yaxis': {'title': \"Mean Revenue\"}\n    }\n}\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2a9f84e6981dccc9b8126bddce827f5a2f75339c"},"cell_type":"code","source":"plt.figure(figsize = (18,8))\nplt.figure(1)\nsns.set(style=\"darkgrid\")\nplt.subplot(221)\nsns.countplot(x=\"trafficSource_adwordsClickInfo.adNetworkType\", data=train ,color=\"#7dd5eb\")\nplt.subplot(222)\nsns.countplot(x=\"trafficSource_adwordsClickInfo.slot\", data=train ,color=\"#7dd5eb\" )\nplt.subplot(223)\nsns.countplot(x=\"trafficSource_campaign\", data=train ,color=\"#7dd5eb\")\nplt.xticks(rotation = 90)\nplt.subplot(224)\nsns.countplot(x=\"trafficSource_medium\", data=train ,color=\"#7dd5eb\" )\nplt.xticks(rotation = 90)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}