{"cells":[{"metadata":{"_uuid":"f7061ba1c7e07908087693f6087785f98e84a606"},"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction EDA\n"},{"metadata":{"_uuid":"0cd098c1aab3715f348d78f89c282aa2d291d0cb"},"cell_type":"markdown","source":"This kernel is a simple EDA of the Google Analytics Customer Revenue Prediction train set.\n\nI used Plotly as data plotting tool: Interactive graphs are really useful to explore data!\n\nIt's my first attempt at giving out a Kaggle kernel, feel free to give any advice to get better :)\n\n# Summary\n1. [Some preliminary table formatting](#chap1)\n2. [Basic first questions answered](#chap2)\n3. [Visits metrics over time](#chap3)\n4. [transactionRevenue distribution](#chap4)\n5. [A closer look at each categorical field](#chap5)\n6. [Impact of previous visits](#chap6)\n7. [Impact of hits and pageviews](#chap7)\n8. [A quick look at the test set](#chap8)"},{"metadata":{"_uuid":"d608d554263de18289827d7f0063c3c2f68fbc3c"},"cell_type":"markdown","source":"# Some preliminary table formatting <a name=\"chap1\"></a>"},{"metadata":{"_uuid":"94287da5f2b213bf5c9cc69573d21b208f933ff1"},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set(style=\"darkgrid\")\nimport datetime as dt\nfrom pandas.io.json import json_normalize\n\n\n\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport cufflinks as cf\ncf.go_offline()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4a752024378f1eb589e5d9eea38447ae8ca1863"},"cell_type":"markdown","source":"## Load and format"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c41326f1cf859a7c91306f9f8aae9dc2ec5304d","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"\"\"\" Format tables \"\"\"\n\ndef format_table(df):\n    df_formatted = df\n    \n    # Format date\n    df_formatted[\"date\"] = df_formatted[\"date\"]\\\n        .apply(lambda x: dt.datetime.strptime(str(x),\"%Y%m%d\"))\n\n    for dict_col_name in [\"device\", \"totals\", \"trafficSource\", \"geoNetwork\"]:\n        df_formatted[dict_col_name] = df_formatted[dict_col_name]\\\n            .apply(lambda x: x.replace('true', 'True').replace('false', 'False'))\\\n            .apply(lambda x: dict(eval(x)))\n        \n    # Explode json columns\n    for col_name in [\"browser\", \"operatingSystem\", \"isMobile\", \"deviceCategory\"]:\n        df_formatted[col_name] = df_formatted[\"device\"]\\\n            .apply(lambda x: x.get(col_name))\n\n    for col_name in [\"visits\", \"hits\", \"pageviews\", \"bounces\", \"newVisits\", \"transactionRevenue\"]:\n        df_formatted[col_name] = df_formatted[\"totals\"]\\\n            .apply(lambda x: float(x.get(col_name) if x.get(col_name) else 0))\n\n    for col_name in [\"campaign\", \"source\", \"medium\", \"keyword\"]:\n        df_formatted[col_name] = df_formatted[\"trafficSource\"]\\\n            .apply(lambda x: x.get(col_name))\n\n    for col_name in [\"continent\", \"subContinent\", \"country\", \"region\"]:\n        df_formatted[col_name] = df_formatted[\"geoNetwork\"]\\\n            .apply(lambda x: x.get(col_name))\n\n    df_formatted = df_formatted.drop(columns=[\"device\", \"totals\", \"trafficSource\", \"geoNetwork\"])    \n    \n    # Add has_bought column\n    df_formatted[\"has_bought\"]=(df_formatted[\"transactionRevenue\"]>0).apply(int)\n    \n    return df_formatted\n\ntrain = format_table(train)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5ba81fb8bc2d01838efe6978683fc0b91674159","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"# A simple look at the train set\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46aae5fab7ada10127d5f428c206c77c34c3d9d2","_kg_hide-input":true},"cell_type":"code","source":"print(\"Training set contains %s lines\" % train.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30fa0032f954f4185ac6dc069f997fcb00ebd555"},"cell_type":"markdown","source":"# Basic first questions <a name=\"chap2\"></a>"},{"metadata":{"_uuid":"46f7f0d662f72c8fd48e4e7f69e3c3ba35a6b906"},"cell_type":"markdown","source":"What's the probability that a visit leads to a payment ? If yes, what's the mean basket price ?"},{"metadata":{"trusted":true,"_uuid":"1920e7f229d5a83d173d8ee18edad1a98a3d6a08","_kg_hide-input":true},"cell_type":"code","source":"print(\"Probability: {}%\".format(round(100 * train[train['transactionRevenue'] > 0].shape[0] / train.shape[0], 2)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"32510aad82d860d40598b7a98a296ea25327f282","_kg_hide-input":true},"cell_type":"code","source":"mean = train[train['transactionRevenue'] > 0]\\\n          .agg({'transactionRevenue': 'mean'})['transactionRevenue']\n\nprint(\"Mean basket price if payment: %s\" % round(mean, 2))\n\nmedian = train[train['transactionRevenue'] > 0]\\\n          .agg({'transactionRevenue': 'median'})['transactionRevenue']\n\nprint(\"Mean basket price if payment: %s\" % round(median, 2))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32329a0376a970f11da2d81c6a100c0179448115"},"cell_type":"markdown","source":"The probability makes sense. The mean basket price however doesn't without any currency information. Not a big deal but it's always nice to have an intuition over what we're handling\n(Update: saw on other kernels that it's in micro$, it makes more sense then. The median being lower than the average, we can expect that there are outliers payments with great amounts)"},{"metadata":{"_uuid":"d5229f8e3e9c8af08b365a93290c178f919fdaed"},"cell_type":"markdown","source":"# Evolution of visits and sales in time <a name=\"chap3\"></a>"},{"metadata":{"trusted":true,"_uuid":"22490dc8cf58be180ada4f7ef908dbb6b1d0863c","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Count nb_sales\ntrain_count = train\\\n    .groupby(\"date\", as_index=False)\\\n    .agg({'visitId' : ['count'], 'transactionRevenue' : ['sum']})\\\n    .rename(columns={'visitId': 'nb_visits', 'transactionRevenue': 'total_revenue'})\n\ntrain_count.columns=train_count.columns.droplevel(1)\n\ntrain_count['revenue_per_visit'] = train_count['total_revenue'] / train_count['nb_visits']\n\n# Add moving average\nwindow_size = 7 # Weekly seasonality\ntrain_count[\"visits_moving_average\"] = train_count[\"nb_visits\"].rolling(window_size, center=True).sum() / window_size\ntrain_count[\"revenue_moving_average\"] = train_count[\"total_revenue\"].rolling(window_size, center=True).sum() / window_size\ntrain_count[\"ratio_moving_average\"] = train_count[\"revenue_per_visit\"].rolling(window_size, center=True).sum() / window_size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"135983648e631d29209d56a1156a7c667f257394","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Prepare layout for plotting\nlayout = go.Layout(\n    legend = dict(orientation=\"h\")\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7de8f80bd9ba9c6471129d6f189af74249a4ebb5","_kg_hide-input":true},"cell_type":"code","source":"#Create traces\nlayout.update({\"title\": \"Evolution of number of daily visits over time\"})\n\ntrace_daily = go.Scatter(\n    x = train_count.date,\n    y = train_count.nb_visits,\n    name = 'nb_visits'\n)\ntrace_average = go.Scatter(\n    x = train_count.date,\n    y = train_count.visits_moving_average,\n    name = 'Moving average'\n)\n\ndata = [trace_daily, trace_average]\nfig = go.Figure(data=data, layout=layout)\n\npy.iplot(fig, filename='line-mode')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df19d05ee69b11698537cfceabd622dd1fed083d"},"cell_type":"markdown","source":"Notes :\n- Peak during Late Fall / Early Winter 2017 (Christmas ?)\n- Weekly seasonality (less visits on weekends, easier to see with zoom in functionality)"},{"metadata":{"trusted":true,"_uuid":"64ee6aada2a66f7511234aef5b013c7dcccfbaeb","_kg_hide-input":true},"cell_type":"code","source":"layout.update({\"title\": \"Evolution of daily revenues over time\"})\n\ntrace_transactions = go.Scatter(\n    x = train_count.date,\n    y = train_count.total_revenue,\n    name = 'Revenue'\n)\n\ntrace_average = go.Scatter(\n    x = train_count.date,\n    y = train_count.revenue_moving_average,\n    name = 'Moving average'\n)\n\ndata = [trace_transactions, trace_average]\nfig = go.Figure(data=data, layout=layout)\n\npy.iplot(fig, filename='line-mode')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"faadcd6c2e456e58b9f4813507c8e60a89b5c12f"},"cell_type":"markdown","source":"Notes:\n - Spikes during Valentines day and easter\n - rise of average sales during christmas"},{"metadata":{"_uuid":"57d6e214d756ca504fa9d6e5b94a57b26471bcc1"},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_uuid":"25dae2051cfd9b4907a91b837c0f4d47ff97b552","_kg_hide-input":true},"cell_type":"code","source":"layout.update({\"title\": \"Evolution of daily SumRevenue/NbVisits over time\"})\n\n#Create traces\ntrace_daily = go.Scatter(\n    x = train_count.date,\n    y = train_count.revenue_per_visit,\n    name = 'Revenue per visit'\n)\ntrace_average = go.Scatter(\n    x = train_count.date,\n    y = train_count.ratio_moving_average,\n    name = 'Moving average'\n)\n\ndata = [trace_daily, trace_average]\nfig = go.Figure(data=data, layout=layout)\n\npy.iplot(fig, filename='line-mode')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e377507d6746d717d682e8db235f8b6037efdc1"},"cell_type":"markdown","source":"Date can have a strong impact on revenue per visit. We can also see that we didn't lose the seasonality, meaning that people are likely to buy more on certain days of the week"},{"metadata":{"_uuid":"8d2779b2d91597e8c4160b07edaa2793eea1123e"},"cell_type":"markdown","source":"# Transaction revenue distribution <a name=\"chap4\"></a>"},{"metadata":{"trusted":true,"_uuid":"4831abf32dd2a51b57aee6680ec0393b139ac655","_kg_hide-input":true},"cell_type":"code","source":"train[(train['transactionRevenue'] > 0) & (train['transactionRevenue'] < 1000000000)]['transactionRevenue'].iplot(\n    kind='histogram',\n    barmode='stack',\n    bins=1000,\n    histnorm='probability',\n    filename='cufflinks/customized-histogram',\n    title='transactionRevenue distribution')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"957587a96f942cdd0a9a03cf86e1e3b03f316480"},"cell_type":"markdown","source":"As expected the disrtibution is skewed, there are far more small payments than big ones. if we account for the fact that we are trying to predict log of revenues, it shows that small payments are of bigger importance here."},{"metadata":{"_uuid":"03af7327fb5e20b930d2ea8c0b30e1ab295f7ffd"},"cell_type":"markdown","source":"# Metrics by categories (Device/continent/...) <a name=\"chap5\"></a>"},{"metadata":{"trusted":true,"_uuid":"47793cf47bc807d0b353d251822d97d384bb9a9b","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"def plot_metrics_on_type(train, t):\n    \"\"\" Top Part \"\"\"\n    # Daily visits evolutions\n    df = train\\\n        .groupby([\"date\", t], as_index=False)\\\n        .visitId.agg(['count']).reset_index()\\\n        .pivot(index=\"date\", columns=t, values=\"count\")\n    \n    figure = df.iplot(kind='area', fill=True, asFigure=True)\n    figure['layout'].update({'height': 400, 'paper_bgcolor':'#ffffff', 'title': 'Evolution of visits over time per %s' % t})\n    py.iplot(figure)\n    \n    \"\"\" Bottom part \"\"\"\n    # Compare revenues and probability to buy \n    df_revenue_per_type = train[train[\"transactionRevenue\"] > 0]\\\n        .groupby(t)\\\n        .agg({'transactionRevenue': 'mean'})\n    \n    df = train\\\n        .groupby([t], as_index=False)\\\n        .agg({'has_bought': 'mean'}).reset_index()\\\n        .merge(df_revenue_per_type, on=t)\n    \n    trace1 = go.Bar(\n        x=df[t],\n        y=df.has_bought,\n        name='Has_bought'\n    )\n    trace2 = go.Bar(\n        x=df[t],\n        y=df.transactionRevenue,\n        xaxis='x2',\n        yaxis='y2',\n        name='Mean revenue when a payment is done'\n    )\n    \n    # Global repartition of visits\n    df2 = train\\\n        .groupby([t], as_index=False)\\\n        .visitId.agg(['count']).reset_index()\\\n    \n    trace3 = go.Bar(\n        x=df2[t],\n        y=df2[\"count\"],\n        name='Number of visits',\n        xaxis='x3',\n        yaxis='y3',\n    )\n    \n    data = [trace3, trace1, trace2]\n    layout = go.Layout(\n        title=\"%s metrics\" % t,\n        showlegend=False,\n        barmode='group',\n        height=400,\n        xaxis=dict(\n            title='Probability of payment by %s' % t,\n            domain=[0, 0.28],\n            titlefont=dict(\n                size=9,\n            )\n        ),\n        yaxis=dict(\n            domain=[0, 1]\n        ),\n        xaxis2=dict(\n            title='Mean revenue by %s' % t,\n            domain=[0.38, 0.62],\n            titlefont=dict(\n                size=9,\n            )\n        ),\n        yaxis2=dict(\n            domain=[0, 1],\n            anchor='x2'\n        ),\n        xaxis3=dict(\n            title='Number of visits by %s' % t,\n            domain=[0.72, 1],\n            titlefont=dict(\n                size=9,\n            )\n        ),\n        yaxis3=dict(\n            domain=[0, 1],\n            anchor='x3'\n        )\n    )\n    \n    fig = go.Figure(data=data, layout=layout)\n    py.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e677cc5033d27855678298e5f3b36f54ede60126"},"cell_type":"markdown","source":"## channelGrouping field"},{"metadata":{"trusted":true,"_uuid":"90710dfa98f0bcea787060e7f35b172fc814ec2b","_kg_hide-input":true},"cell_type":"code","source":"plot_metrics_on_type(train, \"channelGrouping\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"985754304096690f6484775c7c8ddd37b6f96ab1"},"cell_type":"markdown","source":"Display channel has far fewer visits but customers are more generous.\n\nReferral has logically a better probability of payment.\n\nSocial channel seems to get boosted in the end of the year while other channels seem stable."},{"metadata":{"_uuid":"7518d41978d87a4849405aa20c58c5c8f0d95113"},"cell_type":"markdown","source":"## deviceCategory field"},{"metadata":{"trusted":true,"_uuid":"1a9785356ad0e53539ae304ab79e99fe2d4e0b94","_kg_hide-input":true},"cell_type":"code","source":"plot_metrics_on_type(train, \"deviceCategory\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dee2689405852764764234d316e0f9a95a824026"},"cell_type":"markdown","source":"Customers are 3 times more likely to pay on desktops, and are likely to make bigger payments as well."},{"metadata":{"_uuid":"384b9aed2f41dc90ff09ff4cd87a5b56a397fadd"},"cell_type":"markdown","source":"## operatingSystem field"},{"metadata":{"trusted":true,"_uuid":"9b1df147ae876de2d829a4ff846786f6f1d53fa5","_kg_hide-input":true},"cell_type":"code","source":"plot_metrics_on_type(train, \"operatingSystem\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"94d73fbd9f01694a578de5bf4f71446bf9adf38e"},"cell_type":"markdown","source":"The number of visits shows that only Windows, Mac, iOS and Android come out. Between these 4 OS Mac has a far greater probability of payment."},{"metadata":{"_uuid":"a9fb39f7c4014bbad8f282333f88093e3255f885"},"cell_type":"markdown","source":"## continent field"},{"metadata":{"trusted":true,"_uuid":"602198fa2fbb1b8080ce7f3452ca4ea2f1e1e35c"},"cell_type":"code","source":"plot_metrics_on_type(train, \"continent\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61646ddd7051d1b72d1f4a1261016ac1d48aae29"},"cell_type":"markdown","source":"There are very few payments from Africa but they are usually bigger than average.\nAmericans are far more likely to buy than the average"},{"metadata":{"_uuid":"b7a72b40d389c9bd8c8c89023b46fef9464afc4b"},"cell_type":"markdown","source":"## country field"},{"metadata":{"trusted":true,"_uuid":"c74d833a55aacc3cff0c3116c70dad6ff558f1b6"},"cell_type":"code","source":"#plot_metrics_on_type(train, \"country\") #filter biggest countries first","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80f0f3ca64089c61f7365c13e5033c9542ee9c7d"},"cell_type":"markdown","source":"# Impact of previous visits <a name=\"chap6\"></a>"},{"metadata":{"_uuid":"ac3a1dbaa08e33e8138025f585f97c3b2c19ab07","trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"df_payment_prob = train\\\n    .groupby(\"newVisits\")\\\n    .agg({\"visitId\": \"count\", \"has_bought\": \"mean\"})\\\n    .reset_index()\n\ndf_transaction_mean_when_payment = \\\n    train[train[\"transactionRevenue\"] > 0]\\\n        .groupby(\"newVisits\")\\\n        .agg({\"transactionRevenue\": \"mean\"})\\\n        .reset_index()\n\ntrace1 = go.Bar(\n    x=df_payment_prob[\"newVisits\"],\n    y=df_payment_prob[\"has_bought\"],\n    name='Payment probability'\n)\n\ntrace2 = go.Bar(\n    x=df_transaction_mean_when_payment[\"newVisits\"],\n    y=df_transaction_mean_when_payment[\"transactionRevenue\"],\n    xaxis='x2',\n    yaxis='y2',\n    name='Mean revenue when payment'\n)\n\ntrace3 = go.Bar(\n    x=df_payment_prob[\"newVisits\"],\n    y=df_payment_prob[\"visitId\"],\n    xaxis='x3',\n    yaxis='y3',\n    name='Number of visits'\n)\n\n\nlayout = go.Layout(\n    title=\"Analysis of newVisits field\",\n    showlegend=False,\n    barmode='group',\n    height=400,\n    xaxis=dict(\n        title='Probability of payment',\n        domain=[0, 0.28]\n    ),\n    yaxis=dict(\n        domain=[0, 1]\n    ),\n    xaxis2=dict(\n        title='Mean revenue when a payment is done',\n        domain=[0.38, 0.62]\n    ),\n    yaxis2=dict(\n        domain=[0, 1],\n        anchor='x2'\n    ),\n    xaxis3=dict(\n        title='Number of visits',\n        domain=[0.72, 1]\n    ),\n    yaxis3=dict(\n        domain=[0, 1],\n        anchor='x3'\n    )\n)\n\ndata = [trace1, trace2, trace3]\n\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e3d5e349c45e6851dceeefec5b709fae2543b8a6"},"cell_type":"markdown","source":"newVisits = 1 <-> It is not the first time customers join the service.\n\nIt seems than previous visitors are less likely to make a payment and are less generous.\n\nLet's now go further than just the \"newVisits\" info."},{"metadata":{"trusted":true,"_uuid":"f74a8b59e9ddd870ebcb701376f9cf99c5422e49","_kg_hide-input":true},"cell_type":"code","source":"train[(train['transactionRevenue'] < 1000000000)]\\\n    .groupby(\"visitNumber\")\\\n    .agg({\"has_bought\": \"mean\"})\\\n    .iplot(kind='bar', title=\"Probability of payment given visitNumber field\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a28b318222333b23b821a4ef24096ac0c8dba47b","_kg_hide-input":true},"cell_type":"code","source":"train[(train['transactionRevenue'] > 0) & (train['transactionRevenue'] < 1000000000)]\\\n    .groupby(\"visitNumber\")\\\n    .agg({\"transactionRevenue\": \"mean\"})\\\n    .iplot(kind='bar', title=\"Mean transaction Revenue for visits that led to a payment by visitNumber field\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42de636b8956f6baf77668fe5678f5744698e62e"},"cell_type":"markdown","source":"This clearly shows that the probability of buying is impacted by previous visits if any, and that the price of the basket if a payment is made is impacted as well. \n"},{"metadata":{"_uuid":"4308b09fadd372cf1449ab198a7d918154a7b0c1"},"cell_type":"markdown","source":"# Analysis of hits &  pageviews <a name=\"chap7\"></a>"},{"metadata":{"_uuid":"843e72a4c098bb44a3f4980df292566f85a87959"},"cell_type":"markdown","source":"Other categories we looked  (Countries, devices used...) at described the source of the visit. These features qualify the visit itself."},{"metadata":{"trusted":true,"_uuid":"92e21962b10420c264e04467beb0d982a2e1cf8b","_kg_hide-input":true},"cell_type":"code","source":"train\\\n    .groupby(\"hits\")\\\n    .agg({'has_bought':'mean'})\\\n    .iplot(kind=\"bar\", title=\"Probability of payment given the number of hits\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e59630f3b1fcc8446d542acface0122c426d47ea"},"cell_type":"markdown","source":"This looks great ! As we could have guessed the more hits the more likely you are to make a payment. After a certain threshold (hits ~ > 200) we even make out that over 2/3 of visits led to payment. What about the transactionRevenue of these visits ?"},{"metadata":{"trusted":true,"_uuid":"2b55f77ec84682e88a860fa9c2ae1d601fa100f1","_kg_hide-input":true},"cell_type":"code","source":"train[train[\"transactionRevenue\"] > 0]\\\n    .groupby(\"hits\")\\\n    .agg({'transactionRevenue':'mean'})\\\n    .iplot(kind=\"bar\", title=\"Mean transactionRevenue for visits that led to payments, given the number of hits\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ceda20be0dcc0cb48fe12bcb46e5337e80a60a18"},"cell_type":"markdown","source":"It seems that more hits slightly lead to bigger transactions, and a greater variance in these sums. (You can zoom in the plot)"},{"metadata":{"trusted":true,"_uuid":"e2f429882df5e1a884f65416d6a4a75575cfa77c","_kg_hide-input":true},"cell_type":"code","source":"train\\\n    .groupby(\"pageviews\")\\\n    .agg({'has_bought':'mean'})\\\n    .iplot(kind=\"bar\", title=\"Probability of payment given the number of pageviews\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9998b5541ef4e43b158ceb08c1d61c326110a2d3","_kg_hide-input":true},"cell_type":"code","source":"train[train[\"transactionRevenue\"] > 0]\\\n    .groupby(\"pageviews\")\\\n    .agg({'transactionRevenue':'mean'})\\\n    .iplot(kind=\"bar\", title=\"Mean transactionRevenue for visits that led to payments, given the number of pageviews\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57caa24da018d9ca562edd08479b28c38cbe818d"},"cell_type":"markdown","source":"Same thing for pageviews, which seems normal since pageviews and hits are highly correlated as you can see below\n"},{"metadata":{"trusted":true,"_uuid":"5403f3cc8844729e5ca03e40cf9b02bce503cafe"},"cell_type":"code","source":"data = train[['pageviews','hits']]\ndata.corr(method='pearson')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"befe016a2922e918664414d87fb8fe0a196d6a59"},"cell_type":"markdown","source":"# A quick look at the Test set <a name=\"chap8****\"></a>"},{"metadata":{"trusted":true,"_uuid":"f4ef663818395c3bbbef59acf5caedac3d8da1d0","_kg_hide-input":true},"cell_type":"code","source":"print( \"The test set contains %s lines\" % test.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"645afe49a296546b9bafd7133846eac93dbf9759","_kg_hide-input":true},"cell_type":"code","source":"common_ids = set(test[\"fullVisitorId\"]).intersection(train[\"fullVisitorId\"])\nprint( \"%s visitors IDs are present in bot train and test set. (out of %s)\" % (len(common_ids), len(set(test[\"fullVisitorId\"]))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c698f8d3ea48b2bcf21cd386cdb1c053508f457"},"cell_type":"markdown","source":"to be continued..."},{"metadata":{"trusted":true,"_uuid":"8685bc34812838922a3a307cb265a2c3c81cf6d0"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}