{"cells":[{"metadata":{},"cell_type":"markdown","source":"The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.\n\nGStore\n\nIn this notebook we will analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. Hopefully, the outcome will be more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data."},{"metadata":{},"cell_type":"markdown","source":"### **Codes cells have been minimized to improve readiblity, Please click on the code button on the right of each cell to expand**"},{"metadata":{},"cell_type":"markdown","source":"### Please give a thumbs up if you like the notebook!!!"},{"metadata":{},"cell_type":"markdown","source":"#### Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\nimport os\nimport random \n\nimport pandas as pd \nimport numpy as np \nfrom scipy.stats import kurtosis, skew \nfrom scipy import stats\n\nimport matplotlib.pyplot as plt \nimport seaborn as sns \nimport squarify # to better understand proportion of categorys - it's a treemap layout algorithm\n\n# Importing librarys to use on interactive graphs\nimport plotly.offline as plty\nfrom plotly import tools\nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot, plot \nimport plotly.graph_objs as go \n\nimport json # to convert json in df\nfrom pandas.io.json import json_normalize # to normalize the json file\n\n# to set a style to all graphs\nplt.style.use('fivethirtyeight')\ninit_notebook_mode(connected=True)\nsns.set_style(\"whitegrid\")\nsns.set_context(\"paper\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1. Reading datafile (handling columns with Json Values)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"columns = ['device', 'geoNetwork', 'totals', 'trafficSource'] # Columns that have json format\n\ndir_path = \"../input/ga-customer-revenue-prediction/\"\n\n# p is a fractional number to skiprows and read just a random sample of the our dataset. \np = 0.07 # *** In this case we will use 50% of data set *** #\n\n#Code to transform the json format columns in table\ndef json_read(df):\n    #joining the [ path + df received]\n    data_frame = dir_path + df\n    \n    #Importing the dataset\n    df = pd.read_csv(data_frame, \n                     converters={column: json.loads for column in columns}, # loading the json columns properly\n                     dtype={'fullVisitorId': 'str'}, # transforming this column to string\n                     skiprows=lambda i: i>0 and random.random() > p)# Number of rows that will be imported randomly\n    \n    for column in columns: #loop to finally transform the columns in data frame\n        #It will normalize and set the json to a table\n        column_as_df = json_normalize(df[column]) \n        # here will be set the name using the category and subcategory of json columns\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns] \n        # after extracting the values, let drop the original columns\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n        \n    # Printing the shape of dataframes that was imported     \n    print(f\"Loaded {os.path.basename(data_frame)}. Shape: {df.shape}\")\n    return df ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time \n# %%time is used to calculate the timing of code chunk execution #\n\n\ndf_train = json_read(\"train.csv\") ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.options.display.max_columns = 100\ndf_train.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Function to Describe each column"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def DataDesc(df):\n    print(f\"Dataset Shape: {df.shape}\")\n    summary = pd.DataFrame(df.dtypes,columns=['dtypes'])\n    summary = summary.reset_index()\n    summary['Name'] = summary['index']\n    summary = summary[['Name','dtypes']]\n    summary['Missing'] = df.isnull().sum().values    \n    summary['Uniques'] = df.nunique().values\n    summary['First Value'] = df.loc[0].values\n    summary['Second Value'] = df.loc[1].values\n    summary['Third Value'] = df.loc[2].values\n\n    for name in summary['Name'].value_counts().index:\n        summary.loc[summary['Name'] == name, 'Entropy'] = round(stats.entropy(df[name].value_counts(normalize=True), base=2),2) \n    \n    return summary","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DataDesc(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. Notice there are a lot of constant columns having only one unique value\n2. Target column = totals.transactionRevenue (Null value = No Revenue/ Total 62847 Null Values)\n3. Columns with prefix trafficSource contain Null values majorly, hence we will drop these columns"},{"metadata":{},"cell_type":"markdown","source":"### 3. Describing Columns with Null Values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def Null_Count(df):\n    df_null = df.isnull().sum().sort_values(ascending = False).rename('Null').reset_index()\n\n    null_count = df_null['Null']\n    null_percent = (null_count * 100) / (df.shape[0])\n\n    df_null = pd.concat([df_null['index'],null_count,null_percent], axis=1, keys=['Column','Null_Count','Null_Percent'])\n\n    return df_null[df_null['Null_Count'] != 0]\n\n\n\nNull_Count(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4. Function to Fill Null Values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def fill_na(df):   \n    df['totals.pageviews'].fillna(1, inplace=True)\n    df['totals.newVisits'].fillna(0, inplace=True)\n    df['totals.bounces'].fillna(0, inplace=True) \n    df[\"totals.transactionRevenue\"].fillna(0.0, inplace=True)\n    \n    # Changing datatypes from object to desired ones\n    df['totals.pageviews'] = df['totals.pageviews'].astype(int)\n    df['totals.newVisits'] = df['totals.newVisits'].astype(int)\n    df['totals.bounces'] = df['totals.bounces'].astype(int)\n    df[\"totals.transactionRevenue\"] = df[\"totals.transactionRevenue\"].astype(float)\n    \n    \n    df['trafficSource.isTrueDirect'].fillna(False, inplace=True) \n    df['trafficSource.adwordsClickInfo.isVideoAd'].fillna(True, inplace=True) # filling boolean with True\n    df[df_train['geoNetwork.city'] == \"(not set)\"]['geoNetwork.city'] = np.nan\n    df['geoNetwork.city'].fillna(\"NaN\", inplace=True)\n    \n    return df\n\ndf_train = fill_na(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_train['fullVisitorId'] = df_train['fullVisitorId'].astype(float)\ndf_train['sessionId'] = df_train['sessionId'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5. Processing Date/Time Columns"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from datetime import datetime\n\n# This function is to extract date features\ndef date_process(df):\n    df[\"date\"] = pd.to_datetime(df[\"date\"], format=\"%Y%m%d\") # seting the column as pandas datetime\n    df[\"weekday\"] = df['date'].dt.weekday #extracting week day\n    df[\"day\"] = df['date'].dt.day # extracting day\n    df[\"month\"] = df['date'].dt.month # extracting day\n    df[\"year\"] = df['date'].dt.year # extracting day\n    df['visitHour'] = (df['visitStartTime'].apply(lambda x: str(datetime.fromtimestamp(x).hour))).astype(int)\n    \n    return df\n\ndf_train = date_process(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6. Removing Constant Columns"},{"metadata":{"trusted":true},"cell_type":"code","source":"constant_columns = [col for col in df_train.columns if df_train[col].nunique() == 1]\nprint(f'Columns : {constant_columns}, \\n Num of Columns : {len(constant_columns)}')\n\n\ndf_train.drop(constant_columns, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 7. Dropping Null Columns"},{"metadata":{"trusted":true},"cell_type":"code","source":"Null_Count(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.drop(list(Null_Count(df_train)['Column']), axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 8. Exploring Dataset after cleaning and processing"},{"metadata":{"trusted":true},"cell_type":"code","source":"DataDesc(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"We are good to go now.........!"},{"metadata":{},"cell_type":"markdown","source":"# Exploring Revenue"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\nfig = plt.figure(figsize=(20,5))\nplt.suptitle('Distribuition of Revenue', fontsize=30)\n\nax1 = fig.add_subplot(121)\n_ = sns.distplot(np.log(df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"]), bins=40,color='#e56b6f', ax=ax1)\n_ = ax1.set_ylabel('Distribution', fontsize=20)\n_ = ax1.set_xlabel('Transaction Revenue Log', fontsize=20)\n\n\nax2 = fig.add_subplot(122)\n_ = plt.scatter(range(df_train.shape[0]), np.sort(df_train['totals.transactionRevenue'].values), color='#2a9d8f')\n_ = ax2.set_ylabel('Distribution', fontsize=20)\n_ = ax2.set_xlabel('Revenue', fontsize=20)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"According to 80/20 it seems to be true that only a small percentage of customers produce most of the revenue"},{"metadata":{},"cell_type":"markdown","source":"### Calculating Outliers"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class color:\n   PURPLE = '\\033[95m'\n   CYAN = '\\033[96m'\n   DARKCYAN = '\\033[36m'\n   BLUE = '\\033[94m'\n   GREEN = '\\033[92m'\n   YELLOW = '\\033[93m'\n   RED = '\\033[91m'\n   BOLD = '\\033[1m'\n   UNDERLINE = '\\033[4m'\n   END = '\\033[0m'\n\ndef CalOutliers(df_num): \n\n    # calculating mean and std of the array\n    data_mean, data_std = np.mean(df_num), np.std(df_num)\n\n    # seting the cut line to both higher and lower values\n    # You can change this value\n    cut = data_std * 3\n\n    #Calculating the higher and lower cut values\n    lower, upper = data_mean - cut, data_mean + cut\n\n    # creating an array of lower, higher and total outlier values \n    outliers_lower = [x for x in df_num if x < lower]\n    outliers_higher = [x for x in df_num if x > upper]\n    outliers_total = [x for x in df_num if x < lower or x > upper]\n\n    # array without outlier values\n    outliers_removed = [x for x in df_num if x > lower and x < upper]\n    \n    print(color.BOLD+f'Lower outliers: {len(outliers_lower)}'+ color.END) # printing total number of values in lower cut of outliers\n    print(color.BOLD+f'Upper outliers: {len(outliers_higher)}'+ color.END) # printing total number of values in higher cut of outliers\n    print(color.BOLD+f'Total outliers: {len(outliers_total)}'+ color.END) # printing total number of values outliers of both sides\n    print(color.BOLD+f'Non - outliers: {len(outliers_removed)}'+ color.END) # printing total number of non outlier values\n    print(color.BOLD+f'% of Outliers : {round((len(outliers_total) / len(outliers_removed) )*100, 4)}'+ color.END ) # Percentage of outliers in points","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"CalOutliers(df_train['totals.transactionRevenue'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Revenue based on Browsers/Device/Operating System"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_train['device.browser'].value_counts()[:10].reset_index()\n\n\ndef horizontal_bar_chart(cnt_srs, color):\n    trace = go.Bar(\n        y=cnt_srs.index[::-1],\n        x=cnt_srs.values[::-1],\n        showlegend=False,\n        orientation = 'h',\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\n# Device Browser\ncnt_srs = df_train.groupby('device.browser')['totals.transactionRevenue'].agg(['size', 'sum', 'mean'])\ncnt_srs.columns = [\"count\", \"total revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), '#073b4c')\ntrace2 = horizontal_bar_chart(cnt_srs[\"total revenue\"].head(10), '#073b4c')\ntrace3 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), '#073b4c')\n\n# Device Category\ncnt_srs = df_train.groupby('device.deviceCategory')['totals.transactionRevenue'].agg(['size', 'sum', 'mean'])\ncnt_srs.columns = [\"count\", \"total revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace4 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), '#118ab2')\ntrace5 = horizontal_bar_chart(cnt_srs[\"total revenue\"].head(10), '#118ab2')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), '#118ab2')\n\n# Operating system\ncnt_srs = df_train.groupby('device.operatingSystem')['totals.transactionRevenue'].agg(['size', 'sum', 'mean'])\ncnt_srs.columns = [\"count\", \"total revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace7 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), '#ef476f')\ntrace8 = horizontal_bar_chart(cnt_srs[\"total revenue\"].head(10),'#ef476f')\ntrace9 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10),'#ef476f')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=3, vertical_spacing=0.04, \n                          subplot_titles=[\"Device Browser - Count\", \"Device Browser - Total Revenue\", \"Device Browser - Mean Revenue\",\n                                          \"Device Category - Count\",  \"Device Category - Total Revenue \", \"Device Category - Mean Revenue\", \n                                          \"Device OS - Count\", \"Device OS - Total Revenue\", \"Device OS - Mean Revenue\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 1, 3)\nfig.append_trace(trace4, 2, 1)\nfig.append_trace(trace5, 2, 2)\nfig.append_trace(trace6, 2, 3)\nfig.append_trace(trace7, 3, 1)\nfig.append_trace(trace8, 3, 2)\nfig.append_trace(trace9, 3, 3)\n\nfig['layout'].update(height=1200, width=1500, template='plotly_white',paper_bgcolor='#ffffff', title=\"Device Plots\")\nplty.iplot(fig, filename='device-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Channel Groups and their Revenue"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"group = df_train.groupby('channelGrouping')['totals.transactionRevenue'].agg(['count','sum','mean']).reset_index()\n\ncolor = [\"#ffa69e\",\"#faf3dd\",\"#b8f2e6\",\"#aed9e0\",\"#5e6472\",'#f6bd60','#84a59d','#f8edeb']\ncustomPalette = sns.set_palette(sns.color_palette(color))\n\nfig = plt.figure(figsize=(10,10))\n\nax1 = fig.add_subplot(311)\n_ = sns.barplot(data=group, x='channelGrouping', y='count', palette= customPalette, ax=ax1)\nxlabels = group['channelGrouping'].to_list()\nylabels = group['count']\n_ = ax1.set_title('Number of Users', fontsize=20)\n_ = ax1.set_ylabel('Number of Users', fontsize=14)\n_ = ax1.set_xlabel('')\n_ = ax1.set_xticklabels(xlabels, rotation=30, fontsize=10)\n\nax2 = fig.add_subplot(312)\n_ = sns.barplot(data=group, x='channelGrouping', y='sum', palette= customPalette, ax=ax2)\nxlabels = group['channelGrouping'].to_list()\nylabels = group['sum']\n_ = ax2.set_title('Total Revenue', fontsize=20)\n_ = ax2.set_ylabel('Total Revenue', fontsize=14)\n_ = ax2.set_xlabel('')\n_ = ax2.set_xticklabels(xlabels, rotation=30, fontsize=10)\n\nax3 = fig.add_subplot(313)\n_ = sns.barplot(data=group, x='channelGrouping', y='mean', palette= customPalette, ax=ax3)\nxlabels = group['channelGrouping'].to_list()\nylabels = group['mean']\n_ = ax3.set_title('Average Revenue', fontsize=20)\n_ = ax3.set_ylabel('Average Revenue', fontsize=14)\n_ = ax3.set_xlabel('')\n_ = ax3.set_xticklabels(xlabels, rotation=30, fontsize=10)\n\nfig.tight_layout(pad=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"group = df_train.groupby(['channelGrouping','device.browser']).size().rename('Count').reset_index()\n\ngroup_sorted = group.groupby(['channelGrouping']).apply(lambda x: x.sort_values(['Count'], ascending=False)).reset_index(drop=True)\n\ngroup_top_four = group_sorted.groupby(['channelGrouping']).head(4)\n\ncolors = [\"#264653\",\"#2a9d8f\",\"#e9c46a\",\"#f4a261\",\"#e76f51\",'#457b9d']\npx.bar(data_frame=group_top_four,x='channelGrouping',y='Count', color='device.browser', template='plotly_white', color_discrete_sequence= colors)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Which browsers are popularly used on Different Devices?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"group = df_train.groupby(['device.operatingSystem','device.browser']).size().rename('Count').reset_index()\n\ngroup_sorted = group.groupby(['device.operatingSystem']).apply(lambda x: x.sort_values(['Count'], ascending=False)).reset_index(drop=True)\n\ngroup_top_four = group_sorted.groupby(['device.operatingSystem']).head(4)\n\ncolors = [\"#ff9f1c\",\"#ffbf69\",\"#cbf3f0\",\"#2ec4b6\",\"#264653\",\"#2a9d8f\",\"#e9c46a\",\"#f4a261\",\"#e76f51\",\"#e63946\",\"#f1faee\",\"#a8dadc\",\"#457b9d\",\"#1d3557\"]\npx.bar(data_frame=group_top_four,x='device.operatingSystem',y='Count', color='device.browser', template='plotly_white', color_discrete_sequence= colors)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. Safari is mostly used on Mac OS and IOS devices\n2. Chrome is quite popular in most of the devices"},{"metadata":{},"cell_type":"markdown","source":"## Normalizing Transaction Revenue "},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['totals.transactionRevenue'] = df_train['totals.transactionRevenue'].apply(lambda x: np.log1p(x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of transaction Revenue of OS's"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"group = df_train[(df_train['device.operatingSystem'].isin\\\n         (df_train[df_train['totals.transactionRevenue'] > 0]['device.operatingSystem'].value_counts().reset_index()[:6]['index']))\n        & (df_train['totals.transactionRevenue'] > 0)]\n\n_ = sns.FacetGrid(group,\n               hue='device.operatingSystem', height=5, aspect=2.5)\\\n  .map(sns.kdeplot, 'totals.transactionRevenue', shade=True)\\\n .add_legend()\n\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# seting the graph size\nplt.figure(figsize=(14,5))\n\nplt.subplot(1,2,1)\nsns.countplot(df_train[\"device.deviceCategory\"], palette= ['#14213d','#d90429','#fca311']) \nplt.title(\"Device Category Count\", fontsize=20) \nplt.xlabel(\"Device Category\", fontsize=18) \nplt.ylabel(\"Count\", fontsize=16) \nplt.xticks(fontsize=18) \n\nplt.subplot(1,2,2)\nsns.boxenplot(x=\"device.deviceCategory\", y = 'totals.transactionRevenue', \n              data=df_train[df_train['totals.transactionRevenue'] > 0], palette=['#14213d','#fca311','#d90429']) \nplt.title(\"Device Category Revenue Distribuition\", fontsize=20) \nplt.xlabel(\"Device Category\", fontsize=18) \nplt.ylabel(\"Revenue(Log)\", fontsize=16) \nplt.xticks(fontsize=18)\n\nplt.subplots_adjust(hspace = 0.9, wspace = 0.5)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"group = df_train.groupby('device.deviceCategory').size().rename('Count').reset_index()\nfig = px.pie(group, \n             values='Count', names='device.deviceCategory', \n             color_discrete_sequence=['#14213d','#fca311','#d90429'],\n            title='Device Category',\n            width=800,\n            height=500)\n\nfig.update_layout(\n    margin=dict(l=25, r=20, t=30, b=50),\n    width = 600,\n    height = 400,\n    paper_bgcolor=\"#ffffff\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"color = ['#14213d','#fca311','#d90429']\n\ncustomPalette = sns.set_palette(sns.color_palette(color))\n\n_ = sns.FacetGrid(df_train[df_train['totals.transactionRevenue'] > 0],\n               hue='device.deviceCategory', height=5, aspect=2.5, palette=customPalette)\\\n  .map(sns.kdeplot, 'totals.transactionRevenue', shade=True)\\\n .add_legend()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Different continents"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['geoNetwork.continent'].value_counts().reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Vistis/Revenue/Sellings by Day"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#seting some static color options\ncolor_op = ['#5527A0', '#BB93D7', '#834CF7', '#6C941E', '#93EAEA', '#7425FF', '#F2098A', '#7E87AC', \n            '#EBE36F', '#7FD394', '#49C35D', '#3058EE', '#44FDCF', '#A38F85', '#C4CEE0', '#B63A05', \n            '#4856BF', '#F0DB1B', '#9FDBD9', '#B123AC']\n\n# Visits by time train\n\n# couting all entries by date to get number of visits by each date\ndates_temp = df_train['date'].value_counts().reset_index().sort_values('index') \n# renaming the columns to apropriate names\ndates_temp.columns = ['date','visits'] \n\n# creating the first trace with the necessary parameters\ntrace = go.Scatter(x=dates_temp.date.astype(str), y=dates_temp.visits,\n                    opacity = 0.8, line = dict(color = '#38C788'), name= 'Visits by day')\n\n# Below we will get the total values by Transaction Revenue Log by date\ndates_temp_sum = df_train.groupby('date')['totals.transactionRevenue'].sum().reset_index()\n\n# using the new dates_temp_sum we will create the second trace\ntrace1 = go.Scatter(x=dates_temp_sum.date.astype(str), line = dict(color = '#C73877'), name=\"RevenueLog by day\",\n                        y=dates_temp_sum['totals.transactionRevenue'], opacity = 0.8)\n\n# Getting the total values by Transactions by each date\ndates_temp_count = df_train[df_train['totals.transactionRevenue'] > 0].groupby('date')['totals.transactionRevenue'].count().reset_index()\n\n# using the new dates_temp_count we will create the third trace\ntrace2 = go.Scatter(x=dates_temp_count.date.astype(str), line = dict(color = color_op[5]), name=\"Sellings by day\",\n                        y=dates_temp_count['totals.transactionRevenue'], opacity = 0.8)\n\n#creating the layout the will allow us to give an title and \n# give us some interesting options to handle with the outputs of graphs\nlayout = dict(\n    title= \"Informations by Date\",\n    paper_bgcolor='#ffffff',\n    template='plotly_white',\n    xaxis=dict(\n        rangeselector=dict(\n            buttons=list([\n                dict(count=1, label='1m', step='month', stepmode='backward'),\n                dict(count=3, label='3m', step='month', stepmode='backward'),\n                dict(count=6, label='6m', step='month', stepmode='backward'),\n                dict(step='all')\n            ])\n        ),\n        rangeslider=dict(visible = True),\n        type='date'\n    )\n)\n\n# creating figure with the both traces and layout\nfig = dict(data= [trace, trace1, trace2], layout=layout)\n\n#rendering the graphs\niplot(fig) #it's an equivalent to plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Hour of day when the Revenue is high!!"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pd.crosstab(df_train['visitHour'],\n            df_train['weekday'], \n            values=df_train['totals.transactionRevenue'], \n            aggfunc='sum').style.background_gradient(cmap='viridis')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* After 2 pm till mid night the Revenue is high"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def PieChart(df_colum, title, limit=15):\n    \"\"\"\n    This function helps to investigate the proportion of visits and total of transction revenue \n    by each category\n    \"\"\"\n\n    count_trace = df_train[df_colum].value_counts()[:limit].to_frame().reset_index()\n    rev_trace = df_train.groupby(df_colum)[\"totals.transactionRevenue\"].sum().nlargest(10).to_frame().reset_index()\n\n    trace1 = go.Pie(labels=count_trace['index'], values=count_trace[df_colum], name= \"% Acesses\", hole= .5, \n                    hoverinfo=\"label+percent+name\", showlegend=True,domain= {'x': [0, .48]}, \n                    marker=dict(colors=color))\n\n    trace2 = go.Pie(labels=rev_trace[df_colum], \n                    values=rev_trace['totals.transactionRevenue'], name=\"% Revenue\", hole= .5, \n                    hoverinfo=\"label+percent+name\", showlegend=False, domain= {'x': [.52, 1]})\n\n    layout = dict(title= title, height=450, font=dict(size=15),\n                  annotations = [\n                      dict(\n                          x=.25, y=.5,\n                          text='Visits', \n                          showarrow=False,\n                          font=dict(size=20)\n                      ),\n                      dict(\n                          x=.80, y=.5,\n                          text='Revenue', \n                          showarrow=False,\n                          font=dict(size=20)\n                      )\n        ])\n\n    fig = dict(data=[trace1, trace2], layout=layout)\n    iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PieChart(\"device.deviceCategory\", \"Device Category\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PieChart(\"geoNetwork.city\", \"Top Cities by Accesses and Revenue\", limit=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PieChart(\"channelGrouping\", \"Channel Grouping Visits and Revenues\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"countMaps = pd.DataFrame(df_train['geoNetwork.country'].value_counts()).reset_index()\ncountMaps.columns=['country', 'counts'] #renaming columns\ncountMaps = countMaps.reset_index().drop('index', axis=1) #reseting index and droping the column\n\ndata = [ dict(\n        type = 'choropleth',\n        locations = countMaps['country'],\n        locationmode = 'country names',\n        z = countMaps['counts'],\n        text = countMaps['country'],\n        colorscale = 'YlGnBu',\n        autocolorscale = False,\n        marker = dict(\n            line = dict (\n                color = 'rgb(180,180,180)',\n                width = 0.5\n            ) ),\n        colorbar = dict(\n            autotick = False,\n            tickprefix = '',\n            title = 'Number of Visits'),\n      ) ]\n\nlayout = dict(\n    title = 'Couting Visits Per Country',\n    geo = dict(\n        showframe = False,\n        showcoastlines = True,\n        projection = dict(\n            type = 'natural earth'\n        )\n    )\n)\n\nfigure = dict( data=data, layout=layout )\niplot(figure, validate=False, filename='map-countrys-count')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"countMaps = df_train.groupby('geoNetwork.country')['totals.transactionRevenue'].sum().reset_index()\ncountMaps.columns=['country', 'revenue'] #renaming columns\ncountMaps = countMaps.reset_index().drop('index', axis=1) #reseting index and droping the column\n\ndata = [ dict(\n        type = 'choropleth',\n        locations = countMaps['country'],\n        locationmode = 'country names',\n        z = countMaps['revenue'],\n        text = countMaps['country'],\n        colorscale = 'Jet',\n        autocolorscale = False,\n        marker = dict(\n            line = dict (\n                color = 'rgb(180,180,180)',\n                width = 0.5\n            ) ),\n        colorbar = dict(\n            autotick = False,\n            tickprefix = '',\n            title = 'Revenue'),\n      ) ]\n\nlayout = dict(\n    title = 'Total Revenue Per Country',\n    geo = dict(\n        showframe = False,\n        showcoastlines = True,\n        projection = dict(\n            type = 'natural earth'\n        )\n    )\n)\n\nfigure = dict( data=data, layout=layout )\niplot(figure, validate=False, filename='map-countrys-count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling"},{"metadata":{},"cell_type":"markdown","source":"## Encoding Categorical Columns"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\ncat_columns = list(df_train.dtypes[df_train.dtypes == 'object'].reset_index()['index'])\nnum_columns = num_cols = [\"totals.hits\", \"totals.pageviews\", \"visitNumber\", \"visitStartTime\", 'totals.bounces',  'totals.newVisits'] \n\nfor col in cat_columns:\n    df_train[col] = LabelEncoder().fit_transform(df_train[col])\n\nX = df_train[cat_columns + num_columns]\ny = df_train['totals.transactionRevenue']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nimport lightgbm as lgb\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=123)\n\nparams = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \"num_leaves\" : 30,\n        \"min_child_samples\" : 100,\n        \"learning_rate\" : 0.01,\n        \"bagging_fraction\" : 0.8,\n        \"feature_fraction\" : 0.5,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n\nlgb_train = lgb.Dataset(X_train, label=y_train)\nlgb_val = lgb.Dataset(X_test, label=y_test)\n\n\nmodel = lgb.train(params, lgb_train, 1000, valid_sets=[lgb_val], early_stopping_rounds=200, verbose_eval=100)\n    \ny_pred = model.predict(X_test, num_iteration=model.best_iteration)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred[y_pred < 0] = 0\ny_df = pd.DataFrame(data=y_pred, columns=['y_pred'])\ny_df['y_test'] = y_test.values\n\n\nfrom sklearn.metrics import mean_squared_error\n\nnp.sqrt(mean_squared_error(y_df['y_test'],y_df['y_pred']))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}