{"cells":[{"metadata":{"_uuid":"bb32cc5d2722e96bb02725aaece54c5509bdacb9"},"cell_type":"markdown","source":"Objective of the notebook:\n\nIn this notebook,I tried to explore the given dataset and make some inferences along the way. The description of the dataset is given in the below cells.\n\nI have taken this dataset from a kaggle competition for revenue prediction. Here I just analysed the data, without building a model."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncolor = sns.color_palette()\n\n%matplotlib inline\n\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6868c4286025cdb431e9f6c51e4ee714d594fca"},"cell_type":"markdown","source":"**About the dataset:**\n\nSimilar to most other kaggle competitions, we are given two datasets\n* train.csv\n* test.csv\n\nEach row in the dataset is one visit to the store. We are predicting the natural log of the sum of all transactions per user. \n    \nThe data fields in the given files are \n* fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n* channelGrouping - The channel via which the user came to the Store.\n* date - The date on which the user visited the Store.\n* device - The specifications for the device used to access the Store.\n* geoNetwork - This section contains information about the geography of the user.\n* sessionId - A unique identifier for this visit to the store.\n* socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n* totals - This section contains aggregate values across the session.\n* trafficSource - This section contains information about the Traffic Source from which the session originated.\n* visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n* visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n* visitStartTime - The timestamp (expressed as POSIX time)."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e08fe7f8ec7419ba476cd52664140ed6ae736a8c"},"cell_type":"code","source":"%%time\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"302c2b9449b6173ed44ad1d27d3547573368afd4"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcaa5e0839118e5e2d9192f450569309e8c3798d"},"cell_type":"markdown","source":"**Target Variable Exploration:**\n\nSince we are predicting the natural log of sum of all transactions of the user, let us sum up the transaction revenue at user level and take a log and then do a scatter plot."},{"metadata":{"trusted":true,"_uuid":"09744db3d19167bcb29c298ee5b248004bdb01c1"},"cell_type":"code","source":"train_df[\"totals.transactionRevenue\"] = train_df[\"totals.transactionRevenue\"].astype('float')\ngdf = train_df.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\n\nplt.figure(figsize=(8,6))\nplt.scatter(range(gdf.shape[0]), np.sort(np.log1p(gdf[\"totals.transactionRevenue\"].values)))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('TransactionRevenue', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a86c2c2d3a8191845bd799e1124c6d94fcc04ff"},"cell_type":"code","source":"nzi = pd.notnull(train_df[\"totals.transactionRevenue\"]).sum()\nnzr = (gdf[\"totals.transactionRevenue\"]>0).sum()\nprint(\"Number of instances in train set with non-zero revenue : \", nzi, \" and ratio is : \", nzi / train_df.shape[0])\nprint(\"Number of unique customers with non-zero revenue : \", nzr, \"and the ratio is : \", nzr / gdf.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"569d56cf56f6134872a47e89c8654aff2ca7562c"},"cell_type":"markdown","source":"So the ratio of revenue generating customers to customers with no revenue is in the ratio os 1.3%\n\nSince most of the rows have non-zero revenues, in the following plots let us have a look at the count of each category of the variable along with the number of instances where the revenue is not zero.\n\n** Number of visitors and common visitors:**\n\nNow let us look at the number of unique visitors in the train and test set and also the number of common visitors."},{"metadata":{"trusted":true,"_uuid":"dcc528a15dda0f853516375bd7e2e73aec09ac75"},"cell_type":"code","source":"print(\"Number of unique visitors in train set : \",train_df.fullVisitorId.nunique(), \" out of rows : \",train_df.shape[0])\nprint(\"Number of unique visitors in test set : \",test_df.fullVisitorId.nunique(), \" out of rows : \",test_df.shape[0])\nprint(\"Number of common visitors in train and test set : \",len(set(train_df.fullVisitorId.unique()).intersection(set(test_df.fullVisitorId.unique())) ))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"961119f995a317c8b937571890cf30ac66bd1969"},"cell_type":"markdown","source":"**Columns with constant values: **\n\nLooks like there are quite a few features with constant value in the train set."},{"metadata":{"trusted":true,"_uuid":"5eed694270d93fd7c2bc0a16a620fd329d59c2c1"},"cell_type":"code","source":"const_cols = [c for c in train_df.columns if train_df[c].nunique(dropna=False)==1 ]\nconst_cols","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66245f8c2294398511b88d68538578a962cbdf07"},"cell_type":"markdown","source":"They are quite a few. Since the values are constant, we can just drop them from our feature list and save some memory and time in our analysis. \n\n**Device Information:**"},{"metadata":{"trusted":true,"_uuid":"c57e7e05554e2e19a694a934ffcef530baef47f4","_kg_hide-input":false},"cell_type":"code","source":"def horizontal_bar_chart(cnt_srs, color):\n    trace = go.Bar(\n        y=cnt_srs.index[::-1],\n        x=cnt_srs.values[::-1],\n        showlegend=False,\n        orientation = 'h',\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\n# Device Browser\ncnt_srs = train_df.groupby('device.browser')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace3 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\n# Device Category\ncnt_srs = train_df.groupby('device.deviceCategory')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace4 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(71, 58, 131, 0.8)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(71, 58, 131, 0.8)')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(71, 58, 131, 0.8)')\n\n# Operating system\ncnt_srs = train_df.groupby('device.operatingSystem')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace7 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(246, 78, 139, 0.6)')\ntrace8 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10),'rgba(246, 78, 139, 0.6)')\ntrace9 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10),'rgba(246, 78, 139, 0.6)')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=3, vertical_spacing=0.04, \n                          subplot_titles=[\"Device Browser - Count\", \"Device Browser - Non-zero Revenue Count\", \"Device Browser - Mean Revenue\",\n                                          \"Device Category - Count\",  \"Device Category - Non-zero Revenue Count\", \"Device Category - Mean Revenue\", \n                                          \"Device OS - Count\", \"Device OS - Non-zero Revenue Count\", \"Device OS - Mean Revenue\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 1, 3)\nfig.append_trace(trace4, 2, 1)\nfig.append_trace(trace5, 2, 2)\nfig.append_trace(trace6, 2, 3)\nfig.append_trace(trace7, 3, 1)\nfig.append_trace(trace8, 3, 2)\nfig.append_trace(trace9, 3, 3)\n\nfig['layout'].update(height=1200, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"832da86bcdc9fd153d4142f4bd5430c37915e271"},"cell_type":"markdown","source":"Inferences:\n* Device browser distribution looks similar on both the count and count of non-zero revenue plots\n* On the device category front, desktop seem to have higher percentage of non-zero revenue counts compared to mobile devices.\n* In device operating system, though the number of counts is more from windows, the number of counts where revenue is not zero is more for Macintosh.\n* Chrome OS also has higher percentage of non-zero revenue counts\n* On the mobile OS side, iOS has more percentage of non-zero revenue counts compared to Android \n\n**Date Exploration:**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"59eb3e903949d8345fd8747aac85eab66c0f42cd"},"cell_type":"code","source":"import datetime\n\ndef scatter_plot(cnt_srs, color):\n    trace = go.Scatter(\n        x=cnt_srs.index[::-1],\n        y=cnt_srs.values[::-1],\n        showlegend=False,\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\ntrain_df['date'] = train_df['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = train_df.groupby('date')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_index()\n#cnt_srs.index = cnt_srs.index.astype('str')\ntrace1 = scatter_plot(cnt_srs[\"count\"], 'red')\ntrace2 = scatter_plot(cnt_srs[\"count of non-zero revenue\"], 'blue')\n\nfig = tools.make_subplots(rows=2, cols=1, vertical_spacing=0.08,\n                          subplot_titles=[\"Date - Count\", \"Date - Non-zero Revenue count\"])\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig['layout'].update(height=800, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Date Plots\")\npy.iplot(fig, filename='date-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f35771ef1b28f298d1a266d24e8d2a77626ccf0"},"cell_type":"markdown","source":"Inferences:\n* We have data from 1 Aug, 2016 to 31 July, 2017 in our training dataset\n* In Nov 2016, though there is an increase in the count of visitors, there is no increase in non-zero revenue counts during that time period (relative to the mean)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"08f4a090a5928029e4b47b660afd71d952002146"},"cell_type":"code","source":"test_df['date'] = test_df['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = test_df.groupby('date')['fullVisitorId'].size()\n\n\ntrace = scatter_plot(cnt_srs, 'red')\n\nlayout = go.Layout(\n    height=400,\n    width=800,\n    paper_bgcolor='rgb(233,233,233)',\n    title='Dates in Test set'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f98518587981622a86b31c3e3ce7ab70a2a349e"},"cell_type":"markdown","source":"In the test set, we have dates from 2 Aug, 2017 to 30 Apr, 2018. So there are no common dates between train and test set. So it might be a good idea to do time based validation for this dataset.\n\n**Geographic Information:**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"fcb64598e3f9d0931e140f56c70d4dd5dc6f7c75"},"cell_type":"code","source":"# Continent\ncnt_srs = train_df.groupby('geoNetwork.continent')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(58, 71, 80, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(58, 71, 80, 0.6)')\ntrace3 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(58, 71, 80, 0.6)')\n\n# Sub-continent\ncnt_srs = train_df.groupby('geoNetwork.subContinent')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace4 = horizontal_bar_chart(cnt_srs[\"count\"], 'orange')\ntrace5 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"], 'orange')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"], 'orange')\n\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=2, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Continent - Count\", \"Continent - Non-zero Revenue Count\", \"Continent - Mean Revenue\",\n                                          \"Sub Continent - Count\",  \"Sub Continent - Non-zero Revenue Count\", \"Sub Continent - Mean Revenue\",])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace5, 3, 1)\nfig.append_trace(trace6, 3, 2)\n\nfig['layout'].update(height=1500, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Geography Plots\")\npy.iplot(fig, filename='geo-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acba2fcee5fe10d5985a7daf9cc88aaa7a55f4b0"},"cell_type":"markdown","source":"Inferences:\n* On the continent plot, we can see that America has both higher number of counts as well as highest number of counts where the revenue is non-zero\n* Though Asia and Europe has high number of counts, the number of non-zero revenue counts from these continents are comparatively low. \n* We can infer the first two points from the sub-continents plot too.\n* If the network domain is \"unknown.unknown\" rather than \"(not set)\", then the number of counts with non-zero revenue tend to be lower. \n\nVisitor Profile:\n\nNow let us look at the visitor profile variables like number of pageviews by the visitor, number of hits by the visitor and see how they look."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6b4042af5aa1c3b5aa79ba6d60d4af3554e510e7"},"cell_type":"code","source":"\n# Page views\ncnt_srs = train_df.groupby('totals.pageviews')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(60), 'cyan')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(60), 'cyan')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(60), 'cyan')\n\n# Hits\ncnt_srs = train_df.groupby('totals.hits')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", 'mean']\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(60), 'black')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(60), 'black')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"].head(60), 'black')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=2, cols=3, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Total Pageviews - Count\", \"Total Pageviews - Non-zero Revenue Count\", \"Total Pageviews - Mean Revenue\",\n                                          \"Total Hits - Count\",  \"Total Hits - Non-zero Revenue Count\", \"Total Hits - Mean Revenue\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace6, 2, 3)\n\nfig['layout'].update(height=1200, width=900, paper_bgcolor='rgb(233,233,233)', title=\"Visitor Profile Plots\")\npy.iplot(fig, filename='visitor-profile-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa7d921a36f47ed34e55fc566f9781e625432db0"},"cell_type":"markdown","source":"Inferences:\n* Both these variables look very predictive\n* Count plot shows decreasing nature i.e. we have  a very high total count for less number of hits and page views per visitor transaction and the overall count decreases when the number of hits per visitor transaction increases.\n* On the other hand, we can clearly see that when the number of hits / pageviews per visitor transaction increases, we see that there is a high number of non-zero revenue counts. "},{"metadata":{},"cell_type":"markdown","source":"I have tried to analyse this data in my perspective and tried to explore web data in this notebook."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}