{"cells":[{"metadata":{"_uuid":"bb32cc5d2722e96bb02725aaece54c5509bdacb9"},"cell_type":"markdown","source":"**Objective:**\nWe are challenged to analyze a Google Merchandise Store  (where Google swag is sold) customer dataset to predict revenue per customer. "},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn import linear_model ,neighbors,preprocessing,svm,tree\nfrom sklearn.neighbors import NearestNeighbors\nfrom sklearn.preprocessing import StandardScaler,LabelEncoder\nimport pandas as pd\nfrom IPython.display import display\nfrom sklearn.utils import shuffle\nfrom sklearn.model_selection import train_test_split\nimport warnings\nfrom sklearn.naive_bayes import GaussianNB\nimport matplotlib.pyplot as plt\nfrom sklearn import linear_model , neighbors,preprocessing,svm,tree\nfrom sklearn.linear_model import LogisticRegression,LinearRegression\nfrom sklearn.neighbors import NearestNeighbors,KNeighborsClassifier\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.metrics import r2_score,accuracy_score,make_scorer\nimport seaborn as sns\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.ensemble import RandomForestClassifier,VotingClassifier,ExtraTreesClassifier,BaggingClassifier\nfrom sklearn.linear_model import Lasso, ElasticNet, LinearRegression\nimport sys\nimport os \nfrom xgboost import XGBRegressor\npd.set_option('display.max_rows', 100)\npd.set_option('display.max_columns', 100)\nplt.style.use('dark_background')\nimport vecstack\nfrom vecstack import stacking\nfrom xgboost import XGBClassifier\nwarnings.filterwarnings('ignore')\nfrom scipy.stats import probplot\nfrom itertools import combinations\nfrom sklearn.neighbors import NearestNeighbors\nfrom sklearn.metrics import r2_score,accuracy_score,make_scorer,log_loss,precision_score\nimport seaborn as sns\nfrom sklearn.model_selection import GridSearchCV,KFold,cross_val_score\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.svm import NuSVC,SVC\nfrom scipy import std ,mean\nfrom scipy.stats import norm\nfrom scipy import stats\nwarnings.filterwarnings('ignore')\n%matplotlib inline\nimport lightgbm as lgb\n\n\n\nimport json \nfrom pandas.io.json import json_normalize\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6868c4286025cdb431e9f6c51e4ee714d594fca"},"cell_type":"markdown","source":"**About the data:**\nEach row in the dataset is one visit to the store. We are predicting the natural log of the sum of all transactions per user. \nThe data fields in the given files are \n* fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n* channelGrouping - The channel via which the user came to the Store.\n* date - The date on which the user visited the Store.\n* device - The specifications for the device used to access the Store.\n* geoNetwork - This section contains information about the geography of the user.\n* sessionId - A unique identifier for this visit to the store.\n* socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n* totals - This section contains aggregate values across the session.\n* trafficSource - This section contains information about the Traffic Source from which the session originated.\n* visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n* visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n* visitStartTime - The timestamp (expressed as POSIX time)."},{"metadata":{},"cell_type":"markdown","source":"# WEBSITE TERMINOLOGY\n1. Visit - This is the one piece of information that you really want to know. A visit is one individual visitor who arrives at your web site and proceeds to browse. A visit counts all visitors, no matter how many times the same visitor may have been to your site.\n \n\n2. Page View - This is also called Impression.  Once a visitor arrives at your website, they will search around on a few more pages. On average, a visitor will look at about 2.5 pages. Each individual page a visitor views is tracked as a page view.\n \n3. Hits - The real Black Sheep in the family. The average website owner thinks that a hit means a visit but it is very different (see item 1).  A Hit actually refers to the number of files downloaded on your site, this could include photos, graphics, etc. Picture the average web page, it has photos (each photo is a file and hence a hit) and lots of buttons (each button is a file and hence a hit). On average, each page will include 15 hits.\n \nTo give you an example -  Using the average statistics , 1 Visit to an average web site will generate 3 Page Views and 45 Hits.\n \n4. Traffic Sources - How do visitors find your site\n\nDirect Navigation (type URL in traffic, bookmarks, email links w/o tracking codes, etc.) \n\nReferral Traffic (from links across the web, social media, in trackable email, promotion & branding campaign links)\n\nOrganic Search (queries that sent traffic from any major or minor web search engines)\n\nPPC (click through from Pay Per click sponsored ads, triggered by targeted keyphrases)"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def load_df(csv_path=\"C:/Users/DELL/Downloads/train.csv\", nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, \n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading data with the previus function to handle json columns.. Because it takes time we save it to a csv file!"},{"metadata":{"_uuid":"e08fe7f8ec7419ba476cd52664140ed6ae736a8c","trusted":true},"cell_type":"code","source":"# data = load_df()\n# test_data = load_df(\"C:/Users/DELL/Downloads/googleStoreTest.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# data.to_csv(r'C:/Users/DELL/Downloads/googleStore.csv')\n# test_data.to_csv(r'C:/Users/DELL/Downloads/googleStoreTest.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data=pd.read_csv(\"../input/googlestore-loaded-json/googleStore.csv\", dtype={'fullVisitorId': 'str'},low_memory=False)\ntest_data=pd.read_csv('../input/googlestore-loaded-json/googleStoreTest.csv', dtype={'fullVisitorId': 'str'},low_memory=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# allData=pd.concat([data,test_data],axis=0,keys=['x','y'])\n# test and train together","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# FILLING NA "},{"metadata":{"trusted":true},"cell_type":"code","source":"data.isna().sum()[data.isna().sum()>0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# channel Grouping\n\n * Traffic Sources - How do visitors find your site\n\nDirect Navigation (type URL in traffic, bookmarks, email links w/o tracking codes, etc.) \n\nReferral Traffic (from links across the web, social media, in trackable email, promotion & branding campaign links)\n\nOrganic Search ( traffic from any major or minor web search engines)\n\nPaid search(click through from Pay Per click sponsored ads, triggered by targeted keyphrases)\n\nAffiliate (from  ads on websites)"},{"metadata":{"trusted":false},"cell_type":"code","source":"\ndata['positiveRevenue']=data['totals.transactionRevenue'].map(lambda x: 1 if (x>0)  else 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.style.use('dark_background')\n\n\nsns.set(rc={'figure.figsize':(16.7,7)})\n\nsns.countplot(data['channelGrouping'],hue=data['positiveRevenue'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Referral has a good ratio of positive revenues,Organic brings the most visitors."},{"metadata":{},"cell_type":"markdown","source":"# Visits"},{"metadata":{"trusted":false,"_kg_hide-input":true},"cell_type":"code","source":"print('ratio of visits with non zero revenue/total Visits:',1-data['totals.transactionRevenue'].isna().sum()/len(data))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nuniqueCostumers=data['fullVisitorId'].value_counts()\nprint('most of the visits are from unique costumers',len(uniqueCostumers)/len(data))\nuniqueCostumers\nsns.barplot(x=['all visits','unique visits'],y=[len(data),data['fullVisitorId'].nunique()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_kg_hide-input":true},"cell_type":"code","source":"print('only', 100*(1-data.groupby('fullVisitorId').mean()['totals.transactionRevenue'].isnull().sum()/len(uniqueCostumers)),'% of unique costumers produce revenue' ) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"data['totals.transactionRevenue'].fillna(0,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":false},"cell_type":"code","source":"\ndata['totals.transactionRevenue']=np.log(data['totals.transactionRevenue']+1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_kg_hide-input":true},"cell_type":"code","source":"ax=sns.distplot(data[data['totals.transactionRevenue']>0]['totals.transactionRevenue'])\nprint('Overall mean of visits:',data['totals.transactionRevenue'].mean(),\n      \n      '\\n mean of positive revenue visits:',data[data['totals.transactionRevenue']>0]['totals.transactionRevenue'].mean())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" * gini coefficient of total transaction Revenue (0,1) bigger means bigger inequality in revenue distribution"},{"metadata":{"trusted":false},"cell_type":"code","source":"def gini(list_of_values):\n    sorted_list = sorted(list_of_values)\n    height, area = 0, 0\n    for value in sorted_list:\n        height += value\n        area += height - value / 2.\n    fair_area = height * len(list_of_values) / 2.\n    return (fair_area - area) / fair_area\nprint('gini of positive Revenue',gini(data['totals.transactionRevenue'][data['positiveRevenue']>0]),'\\n \\n gini of total Revenue',gini(data['totals.transactionRevenue']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def dataNumberOfVisitsGreaterThan(n):\n    \n    return data[data['fullVisitorId'].isin(data['fullVisitorId'].value_counts()[data['fullVisitorId'].value_counts()>n].index)]\n\ndef dataNumberOfVisitsLessThan(n):\n    \n    return data[data['fullVisitorId'].isin(data['fullVisitorId'].value_counts()[data['fullVisitorId'].value_counts()<n].index)]\n\n\ndef dataNumberOfVisits(n):\n    \n    return data[data['fullVisitorId'].isin(data['fullVisitorId'].value_counts()[data['fullVisitorId'].value_counts()==n].index)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TRANSACTION--VISITS"},{"metadata":{},"cell_type":"markdown","source":" Mean Transacion increases when visits increase both in positive-revenue and general-revenue visits\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"from datetime import datetime\ndef date_process(df):\n    df[\"date\"] = pd.to_datetime(df[\"date\"], format=\"%Y%m%d\")\n    df[\"_weekday\"] = df['date'].dt.weekday \n    df[\"_day\"] = df['date'].dt.day \n    df[\"_month\"] = df['date'].dt.month \n    df[\"_year\"] = df['date'].dt.year \n    df['_visitHour'] = (df['visitStartTime'].apply(lambda x: str(datetime.fromtimestamp(x).hour))).astype(int)\n    \n    return df\n\ndate_process(data)\ndate_process(test_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DEVICE DATA VISUALAZATION"},{"metadata":{"trusted":false},"cell_type":"code","source":"browsers=data.groupby(data['device.browser']).filter(lambda x:x['device.browser'].size*100/len(data)>1)['device.browser'].value_counts()\n\npbrowsers=data[data['positiveRevenue']>0].groupby(data['device.browser']).filter(lambda x:x['device.browser'].size*100/len(data)>1)['device.browser'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"browsers with more than 1% usage"},{"metadata":{"trusted":false,"_kg_hide-input":true},"cell_type":"code","source":"sns.set(palette='dark')\nplt.style.use('dark_background')\nfig,axes=plt.subplots(2,3,figsize=(19,12))\naxes[0,1].set_title('DEVICE GRAPH',fontsize=26)\nsns.barplot(browsers.index,browsers.values,ax=axes[0,0])\nsns.barplot(data['device.deviceCategory'].value_counts().index,data['device.deviceCategory'].value_counts().values,ax=axes[0,1])\nsns.barplot(data['device.operatingSystem'].value_counts().index[0:6],data['device.operatingSystem'].value_counts().values[0:6],ax=axes[0,2])\n\naxes[1,1].set_title('DEVICE GRAPH FOR POSITIVE REVENUE',fontsize=26)\nsns.barplot(pbrowsers.index,pbrowsers.values,ax=axes[1,0])\nsns.countplot(data[data['positiveRevenue']>0]['device.deviceCategory'],ax=axes[1,1])\nsns.countplot(data[data['positiveRevenue']>0]['device.operatingSystem'],ax=axes[1,2],order=data['device.operatingSystem'].value_counts().index[0:6])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* only chrome users have positive revenue.Others have  low amount of visits\n* mobile and tablets have worse ratio from desktops\n* Macintosh  way bigger  ratio, chrome OS and linux too,while everything less is smaller with emphasis to windows"},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(1,3,figsize=(21,10))\naxes[1].set_title('DEVICE GRAPH',fontsize=26)\nsns.boxplot('device.operatingSystem','totals.transactionRevenue',data=data[(data['device.browser'].isin(browsers.index))&data['positiveRevenue']==1],ax=axes[0])\nsns.boxplot('device.deviceCategory','totals.transactionRevenue',data=data[(data['device.browser'].isin(browsers.index))&data['positiveRevenue']==1],ax=axes[1])\nsns.boxplot('device.browser','totals.transactionRevenue',data=data[(data['device.browser'].isin(browsers.index))&data['positiveRevenue']==1],ax=axes[2])\nylim = (11, 22)\nplt.setp(axes, ylim=ylim);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"chrome OS ,desktop,and chrome are the highest of each category"},{"metadata":{"trusted":false},"cell_type":"code","source":"sns.set(palette='bright')\ncrosstab_eda = pd.crosstab(columns=data['device.browser'][data['device.browser'].isin(browsers.index)], \n                           index=data['device.deviceCategory'])\ncrosstab_eda.plot(kind=\"bar\",    \n                 figsize=(14,7), \n                 stacked=True)  \nplt.title(\"Most frequent Browser's by Device Category\", fontsize=22) \nplt.xlabel(\"Device Name\", fontsize=19)        ;  \nplt.ylabel(\"Browser Count\", fontsize=19)         \nplt.xticks(rotation=0) ;","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"\ndf_train=data\ncrosstab_eda = pd.crosstab(index=df_train['device.deviceCategory'], \n                           columns=df_train[df_train['device.operatingSystem']\\\n                                            .isin(df_train['device.operatingSystem']\\\n                                                  .value_counts()[:6].index.values)]['device.operatingSystem'])\ncrosstab_eda.plot(kind=\"bar\",    \n                 figsize=(14,7), \n                 stacked=True)  \nplt.title(\"Most frequent OS's by Device Category\", fontsize=22) \nplt.xlabel(\"Device Name\", fontsize=19)        \nplt.ylabel(\"Count Device x OS\", fontsize=19)         \nplt.xticks(rotation=0) ;","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATES"},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(1,2,figsize=(21,10))\naxes[1].set_title('Positive---0 revenue',fontsize=26)\nsns.countplot(data['_year'],hue=data['positiveRevenue'])\nsns.countplot(data['_year'],ax=axes[0])\nmin(data['date']),max(data['date'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2016 data has only 4 months while 2017 8. so 2017 is a worse year \n"},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(2,1,figsize=(20,11))\nsns.countplot(data['_weekday'],ax=axes[0])\nweekdayDist=data[data['positiveRevenue']==1].groupby('_weekday')['date'].count()\nsns.barplot(weekdayDist.index,weekdayDist.values,ax=axes[1])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lowest counts on weekends!Also,weekends go even worse for positive revenue ratio compared to others."},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(2,1,figsize=(20,11))\nsns.countplot(data['_month'],ax=axes[0])\naxes[1].set_title('How many people buy',fontsize=19)\naxes[0].set_title('How many people visit',fontsize=19)\nmonthDist=data[data['positiveRevenue']==1].groupby('_month')['date'].count()\nsns.barplot(monthDist.index,monthDist.values,ax=axes[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Novmber & October have the highest counts.But they dont go so well revenue wise. December's ratio is the largest"},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(2,1,figsize=(20,11))\nsns.countplot(data['_day'],ax=axes[0])\naxes[1].set_title('How many people buy',fontsize=19)\naxes[0].set_title('How many people visit',fontsize=19)\nmonthDist=data[data['positiveRevenue']==1].groupby('_day')['date'].count()\nsns.barplot(monthDist.index,monthDist.values,ax=axes[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(2,1,figsize=(20,11))\nsns.countplot(data['_visitHour'],ax=axes[0])\naxes[1].set_title('How many people buy',fontsize=19)\naxes[0].set_title('How many people visit',fontsize=19)\nmonthDist=data[data['positiveRevenue']==1].groupby('_visitHour')['date'].count()\nsns.barplot(monthDist.index,monthDist.values,ax=axes[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"18 to 21 highest traffic.\n\nHuge decrease revenue-wise from 5 to 19"},{"metadata":{"trusted":false},"cell_type":"code","source":"date_sales = ['_visitHour', '_weekday']\n\ncm = sns.light_palette(\"yellow\", as_cmap=True)\npd.crosstab(df_train[date_sales[0]], df_train[date_sales[1]], \n            values=df_train[\"totals.transactionRevenue\"], aggfunc=[np.sum]).style.background_gradient(cmap = cm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LOCATION\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"country_tree = df_train[\"geoNetwork.country\"].value_counts() \nimport squarify\ncountry_tree = round((df_train[\"geoNetwork.country\"].value_counts()[:30]/len(df_train['geoNetwork.country']) * 100),2)\nfig,axes=plt.subplots(2,1,figsize=(16,14))\ng = squarify.plot(sizes=country_tree.values, label=country_tree.index, value=country_tree.values, alpha=.7,ax=axes[0])\ng.set_title(\"Top countries\",fontsize=20)\ng.set_axis_off()\n\ncountry_tree = round((df_train[df_train['positiveRevenue']>0][\"geoNetwork.country\"].value_counts()[:10]/len(df_train[df_train['positiveRevenue']>0]['geoNetwork.country']) * 100),2)\ng = squarify.plot(sizes=country_tree.values, label=country_tree.index, value=country_tree.values, alpha=.7,ax=axes[1])\ng.set_title(\"Top positive Revenue countries \",fontsize=20)\ng.set_axis_off()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unites States takes 95% of positive revenues & only 40% of total visits"},{"metadata":{"trusted":false},"cell_type":"code","source":"\nsns.set(palette='bright')\nplt.style.use('dark_background')\nplt.figure(figsize=(16,8))\nsns.countplot(df_train[df_train['geoNetwork.subContinent'].isin(df_train['geoNetwork.subContinent'].value_counts()[:13].index.values)]['geoNetwork.subContinent'], palette=\"hls\") \nplt.title(\"Most frequent SubContinents\", fontsize=22) \nplt.xlabel(\"subContinent\", fontsize=20) \nplt.ylabel(\"SubContinent Count\", fontsize=20) \nplt.xticks(rotation=52);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TOTAL "},{"metadata":{},"cell_type":"markdown","source":"A bounce occurs when a web site visitor only views a single page on a website, that is, the visitor leaves a site without visiting any other pages before a specified session-timeout occurs. "},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(1,2)\n\nsns.regplot(data['totals.pageviews'],data['totals.transactionRevenue'],ax=axes[0])\n\nsns.regplot(data['totals.hits'],data['totals.transactionRevenue'],ax=axes[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" Almost Similar plots"},{"metadata":{"trusted":false},"cell_type":"code","source":"fig,axes=plt.subplots(1,3)\n\na=sns.barplot(data['positiveRevenue'],data['totals.pageviews'],ax=axes[0])\na.set_ylabel('PAGEVIEWS',fontsize=20)\n\na=sns.barplot(data['positiveRevenue'],data['totals.hits'],ax=axes[1])\na.set_ylabel('HITS',fontsize=20)\n\n\na=sns.barplot(data['positiveRevenue'],data['totals.bounces'],ax=axes[2])\na.set_ylabel('BOUNCES',fontsize=20);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As expected when we have positive revenue we dont have bounce"},{"metadata":{"_uuid":"0f5ae722a5beceacaf32f52a6691a57f05884be0"},"cell_type":"markdown","source":"# Cleaning the data:"},{"metadata":{},"cell_type":"markdown","source":" * Label encoder to categorical columns"},{"metadata":{"_uuid":"1b4cc2fa9b1580c9d8a7ad43b5f75657e150fba3","trusted":false},"cell_type":"code","source":"cols = [\"channelGrouping\", \"device.browser\", \n            \"device.deviceCategory\", \"device.operatingSystem\", \n            \"device.isMobile\",\"geoNetwork.city\", \"geoNetwork.continent\",\"geoNetwork.country\",\n            \"geoNetwork.metro\",\"geoNetwork.networkDomain\", \"geoNetwork.region\",\n            \"geoNetwork.subContinent\", \"trafficSource.adContent\", \"trafficSource.adwordsClickInfo.adNetworkType\", \n            \"trafficSource.adwordsClickInfo.gclId\", \n            \"trafficSource.adwordsClickInfo.page\", \n            \"trafficSource.adwordsClickInfo.slot\", \"trafficSource.campaign\",\n            \"trafficSource.keyword\", \"trafficSource.medium\", \n            \"trafficSource.referralPath\", \"trafficSource.source\",\n            'trafficSource.adwordsClickInfo.isVideoAd', 'trafficSource.isTrueDirect']\nfor col in cols:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(data[col].values.astype('str')))\n    data[col] = lbl.transform(list(data[col].values.astype('str')))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\nfor col in cols:\n    lbl.fit(list(test_data[col].values.astype('str')))\n    test_data[col] = lbl.transform(list(test_data[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"code_folding":[],"trusted":false},"cell_type":"code","source":"%%time\n\ndef transform(data):\n            \n    data['_visitStartHour'] = data['visitStartTime'].apply(lambda x: str(datetime.fromtimestamp(x).hour))\n    data['_visitStartHour'] = data['_visitStartHour'].astype(int)\n        \n\n        \n    data['totals.pageviews'].fillna(1, inplace=True)\n    data['totals.newVisits'].fillna(0, inplace=True)\n    data['totals.bounces'].fillna(0, inplace=True)\n    data['totals.pageviews'] = data['totals.pageviews'].astype(int)\n    data['totals.newVisits'] = data['totals.newVisits'].astype(int)\n    data['totals.bounces'] = data['totals.bounces'].astype(int)\n\n    \n    \n    data['meanHitsHour'] = data.groupby(['_visitHour'])['totals.hits'].transform('mean')\n    data['meanHitsDay'] = data.groupby(['_day'])['totals.hits'].transform('mean')\n    data['meanHitsWeekday'] = data.groupby(['_weekday'])['totals.hits'].transform('mean')\n    data['meanHitsMonth'] = data.groupby(['_month'])['totals.hits'].transform('mean')\n    data['hitsHour'] = data.groupby(['_visitHour'])['totals.hits'].transform('sum') \n    data['hitsDay'] = data.groupby(['_day'])['totals.hits'].transform('sum')\n    data['hitsWeekday'] = data.groupby(['_weekday'])['totals.hits'].transform('sum')\n    data['hitsMonth'] = data.groupby(['_month'])['totals.hits'].transform('sum')\n    \n    \n    for i in ['device.isMobile']: #using mobile or not in particular visit vs mean\n        x = data.groupby('fullVisitorId')[i].mean()\n        data['m'] = data.fullVisitorId.map(x)\n        data['isMobileDifference']=data['device.isMobile']-data['m']\n        data.drop('m',1,inplace=True)\n        \n    \n    for i in ['totals.bounces','totals.newVisits']: ##mean \n        x = data.groupby('fullVisitorId')[i].mean()\n        data['visitorMean_' +i] = data.fullVisitorId.map(x)\n\n\n    for i in ['totals.hits', 'totals.pageviews']: ##mean ,max,min,sum\n        x = data.groupby('fullVisitorId')[i].mean()\n        maxx = data.groupby('fullVisitorId')[i].max()\n        minn = data.groupby('fullVisitorId')[i].min()\n        summ = data.groupby('fullVisitorId')[i].sum()\n        data['visitorMean_' +i] = data.fullVisitorId.map(x)\n        data['visitorMax_' +i] = data.fullVisitorId.map(maxx)\n        data['visitorMin_' +i] = data.fullVisitorId.map(minn)\n        data['visitorMean_' +i] = data.fullVisitorId.map(summ)\n\n    for i in ['visitNumber']:\n        maxx = data.groupby('fullVisitorId')[i].max()\n        minn=data.groupby('fullVisitorId')[i].min()\n        data['visitorMax_' + i] = data.fullVisitorId.map(maxx) \n#         data['visitorDif_'+i]=data.fullVisitorId.map(maxx-minn)\n        \n        \n    for i in ['date']: #date\n        maxx = data.groupby('fullVisitorId')[i].max()\n        minn=data.groupby('fullVisitorId')[i].min()\n        data['visitorDiff_'+i]=data.fullVisitorId.map((maxx-minn))\n\n\ntransform(data)\ntransform(test_data)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test_data.isnull().sum().any(),data.isnull().sum().any()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"\nfor j in data.columns:\n    if j not in test_data.columns:\n        print(j)\n        \n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def datadrop(data):\n    data.drop(['sessionId','visitId','visitStartTime','date'],1,inplace=True)\n    \n    \n    \ndatadrop(data)\ndatadrop(test_data)\n\n\nconst_cols = [c for c in data.columns if data[c].nunique(dropna=False)==1 ]\ndata.drop(const_cols,1,inplace=True)\ntest_data.drop(const_cols,1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"data['visitorDiff_date']=data['visitorDiff_date'].apply(lambda x : x.days)\ntest_data['visitorDiff_date']=test_data['visitorDiff_date'].apply(lambda x : x.days)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"x=data.drop(['totals.transactionRevenue','positiveRevenue','fullVisitorId','Unnamed: 0','trafficSource.campaignCode'],1)\ny=data['totals.transactionRevenue']\n\n\nxtr,xtest,ytr,ytest=train_test_split(x,y,test_size=0.25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"xpred=test_data.drop(['fullVisitorId','Unnamed: 0'],1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MODEL & PARAMETER TUNING"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nfrom math import sqrt\nfrom sklearn.linear_model import Lasso, ElasticNet, LinearRegression\ndef model(model):\n    reg=model\n    reg.fit(xtr,ytr)\n\n    y_pred=reg.predict(xtest)\n    y_pred[y_pred<1]=0\n    score=mean_squared_error(ytest, y_pred)\n    print(sqrt(score))\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"\nfrom sklearn.linear_model import BayesianRidge\n\nmodel(BayesianRidge())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"   params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.5,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n\ngridModel=lgb.LGBMRegressor(**params)    \n\ngridParams = { \n        \"min_child_samples\" : [100,50,200],\n    'learning_rate': [0.1,0.01,0.001],\n    'num_leaves': [12,20,30,40,50]\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#make mean_squared_error scorrer\n\n# grid = GridSearchCV(gridModel, gridParams,\n#                     verbose=0,\n#                     cv=5, n_jobs=-1)\n# grid.fit(xtr, ytr)\n\n# Print the best parameters found\n# print(grid.best_params_)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3de965f248a0bf1a536d9230ef0f3cb9789b0763","scrolled":true,"trusted":false},"cell_type":"code","source":"def lgbCustom(train_X, train_y, val_X, val_y, test_X):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \n        \n        \"num_leaves\" : 50,\n        \"min_child_samples\" : 200, #best params from grid Search!!\n        \"learning_rate\" : 0.1,\n        \n        \n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.5,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=300, verbose_eval=100)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    pred_test_y[pred_test_y<1]=0\n    pred_val_y = model.predict(val_X, num_iteration=model.best_iteration)\n    \n    pred_val_y[pred_val_y<1]=0\n\n    return pred_test_y, model, pred_val_y\n\n\npred, model, val = lgbCustom(xtr, ytr, xtest, ytest, xpred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(sqrt(mean_squared_error(val,ytest)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"360ffb4d501ac2e35b274ac15efd031495c85ec3","trusted":false},"cell_type":"code","source":"submission = pd.DataFrame({'fullVisitorId':test_data['fullVisitorId'], 'PredictedLogRevenue':pred})\n\nsubmission[\"PredictedLogRevenue\"] = np.expm1(submission[\"PredictedLogRevenue\"])\n\nsubmission_sum = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\nsubmission_sum[\"PredictedLogRevenue\"] = np.log1p(submission_sum[\"PredictedLogRevenue\"])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission_sum.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sns.kdeplot(data=submission_sum['PredictedLogRevenue'][submission_sum['PredictedLogRevenue']>0],shade=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# submission_sum.to_csv(\"submission.csv\")\n# submission_sum.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cbf691d8f8ce6bebe84b17f729cf7c7a0ad8c9b"},"cell_type":"markdown","source":"# Feature Importance:\n\n"},{"metadata":{"_uuid":"7a0da4195ac3e9683c29579b7700a21eb58fdcb8","trusted":false},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,12))\nlgb.plot_importance(model, max_num_features=45, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM Feature Importance\", fontsize=15)\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2ed97686a372b89e874864f271765cdcbe9129b"},"cell_type":"markdown","source":"if you have questions feel free to answer \n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false}},"nbformat":4,"nbformat_minor":1}