{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"**Objective of the competition**: In this competition, we’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. \n\n**Data Description** \nWe are given two datasets; train.csv and test.csv.\n\nEach row in the dataset is one visit to the store. We are predicting the natural log of the sum of all transactions per user.\n\nThe data fields are:\n\n*fullVisitorId*- A unique identifier for each user of the Google Merchandise Store.\n\n*channelGrouping* - The channel via which the user came to the Store.\n\n*date* - The date on which the user visited the Store.\n\n*device* - The specifications for the device used to access the Store. \n\n*geoNetwork* - This section contains information about the geography of the user.\n\n*sessionId* - A unique identifier for this visit to the store.\n\n*socialEngagementType* - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n\n*totals* - This section contains aggregate values across the session.\n\n*trafficSource* - This section contains information about the Traffic Source from which the session originated.\n\n*visitId* - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the \nuser. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n\n*visitNumber* - The session number for this user. If this is the first session, then this is set to 1.\n\n*visitStartTime* - The timestamp (expressed as POSIX time).\n\nP.s: It is important to note that some of the fields are in json format ('device', 'geoNetwork', 'totals', 'trafficSource')\nP.s: There is more information on the columns in this link: https://support.google.com/analytics/answer/3437719?hl=en\n\n\n"},{"metadata":{"_uuid":"372b3da7ceb23ebb7017cee6bf291191a1f54e72"},"cell_type":"markdown","source":"In order to convert all the json fields in the file to a flattened csv format we refer to the code from [this kernel](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook) from[ Julian](https://www.kaggle.com/julian3833)."},{"metadata":{"trusted":true,"_uuid":"62711295f09955667b01dc830b8e76e6612888e8"},"cell_type":"code","source":"# Importing libraries\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom datetime import datetime\n\n\n# Function in order to flatten the JSON filds\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"03020e444800d4527b80951ccbc2aa559b064736"},"cell_type":"code","source":"%%time\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"211811e73c7a773ee15e4d62f759fbb700f70e76"},"cell_type":"code","source":"train_df_copy = train_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c58048d06dc6570dfb867af4f0614d2acb45336"},"cell_type":"code","source":"train_df_copy.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"031afede3d7c88d1155271f322e7ee0a9284c0f3"},"cell_type":"markdown","source":"Let's **decide** for all the preprocessing (missing value imputation, etc) based on **only the train data** and then apply the preprocessing to the test data. if we concatenate the train and test data and decide, then our decisions will be very optimistic because they are also based on the test data that we are not allowed to see. "},{"metadata":{"trusted":true,"_uuid":"f77372ea424df0cdfd8c7b6ec70930f005669125"},"cell_type":"code","source":"train_df_copy.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"352e0b228bb19337f4833e75e74f1d75d550ff34"},"cell_type":"code","source":"#sns.countplot(train_df_copy['geoNetwork.city'])\n#plt.rcParams[\"figure.figsize\"] = (25, 10)\n\ntrain_df_copy['trafficSource.adwordsClickInfo.criteriaParameters'].value_counts()\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"318a2a6974c8d1459a426dee1c8569f4ded3a62a"},"cell_type":"code","source":"# Having a look at unique values of geoNetwork.city, geoNetwork.metro, geoNetwork.region, \n# and trafficSource.campaign and trafficSource.adwordsClickInfo.criteriaParameters\n# these fields have more than half missing values\n# Dropping these columns\n#train_df_copy['geoNetwork.city'].describe()\ntrain_df_copy.drop(['trafficSource.adwordsClickInfo.criteriaParameters'],axis=1,inplace=True)\ntest_df.drop(['trafficSource.adwordsClickInfo.criteriaParameters'],axis=1,inplace=True)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ab4768420c99c244233f5f9f449586c08ddd1e2b"},"cell_type":"code","source":"train_df_copy.drop(['fullVisitorId','sessionId',\n                    'visitId'],axis=1,inplace=True)\ntest_df.drop(['sessionId', 'visitId'],axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12b1d1d9638112a69073a39a5f263cdf7a66e2a6"},"cell_type":"code","source":"train_df_copy['trafficSource.adwordsClickInfo.gclId'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0caef61bfba565d17e516b8402b70613b4f7d3b"},"cell_type":"code","source":"train_df_copy.drop(['trafficSource.adwordsClickInfo.gclId'],axis=1,inplace=True)\ntest_df.drop(['trafficSource.adwordsClickInfo.gclId'],axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"97a8daed724334239968b782355d96e1e7ca425d"},"cell_type":"markdown","source":"Based on the extensive column descriptions on https://support.google.com/analytics/answer/3437719?hl=en, we can see that for some columns, a NULL value actually means something. Here I will write the list of these features together with their meaning:\n\n**totals.bounces** (Integer):Total bounces (for convenience). For a bounced session, the value is 1, otherwise it is null.\n\n**totals.newVisits** (Integer):Total number of new users in session (for convenience). If this is the first visit, this value is 1, otherwise it is null.\n\n**totals.visits** (Integer): The number of sessions (for convenience). This value is 1 for sessions with interaction events. The value is null if there are no interaction events in the session.\n\n**TrafficSource.isTrueDirect** (Boolean):True if the source of the session was Direct (meaning the user typed the name of your website URL into the browser or came to your site via a bookmark), This field will also be true if 2 successive but distinct sessions have exactly the same campaign details. Otherwise NULL.\n\nSo I am going to keep the NULL values for these fields and recode them.\n\n\n    "},{"metadata":{"trusted":true,"_uuid":"88be7237b200d5c8918ea3dd9fbfbb7928598d88"},"cell_type":"code","source":"for col in ['totals.bounces','totals.newVisits','totals.visits','totals.transactionRevenue']:\n    train_df_copy[col].fillna(0,inplace = True)\ntrain_df_copy['trafficSource.isTrueDirect'].fillna(False,inplace=True)    \n\n\nfor col in ['totals.bounces','totals.newVisits','totals.visits']:\n    test_df[col].fillna(0,inplace = True)\ntest_df['trafficSource.isTrueDirect'].fillna(False,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af834ff7243053e26ae3ab3299eaa1dc8452fc39"},"cell_type":"markdown","source":"Investigating other missing values:"},{"metadata":{"trusted":true,"_uuid":"ac218c03327ec1d8d30a6b87bcdb776bc2ca6ee8"},"cell_type":"code","source":"# Check missing values\ntrain_df_copy.isnull().sum().sort_values()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59ad3191f765fd73a02f0ab4bef8177c4ac9320e"},"cell_type":"markdown","source":"**'pageviews'** and **'trafficSource.adwordsClickInfo.page'**"},{"metadata":{"trusted":true,"_uuid":"d12e3c660d42932b9fe9bea1da29d4d39d47f2d4"},"cell_type":"code","source":"train_df_copy['totals.pageviews'].describe()\ntrain_df_copy['trafficSource.adwordsClickInfo.page'].describe()\n\n# We fill in the missing values with the most frequent occurence of these columns --> 1\ntrain_df_copy['totals.pageviews'].fillna(1, inplace=True)\ntrain_df_copy['trafficSource.adwordsClickInfo.page'].fillna(1, inplace=True)\n\ntest_df['totals.pageviews'].fillna(1, inplace=True)\ntest_df['trafficSource.adwordsClickInfo.page'].fillna(1, inplace=True)\n\n# Converting totals.pageviews and trafficSource.adwordsClickInfo.page from object to integer\ntrain_df_copy['totals.pageviews'] = train_df_copy['totals.pageviews'].astype(int)\ntrain_df_copy['trafficSource.adwordsClickInfo.page'] = train_df_copy['trafficSource.adwordsClickInfo.page'].astype(int)\n\ntest_df['totals.pageviews'] = test_df['totals.pageviews'].astype(int)\ntest_df['trafficSource.adwordsClickInfo.page'] = test_df['trafficSource.adwordsClickInfo.page'].astype(int)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5200de4eeca043daaab6d3abab1e0049da33655c"},"cell_type":"code","source":"#trafficSource.campaignCode    \n# Only one non-missing. Drop the column\ntrain_df_copy.drop('trafficSource.campaignCode', axis = 1, inplace = True)\n\n#test_df.drop('trafficSource.campaignCode', axis = 1, inplace = True)   #Doesnt exist in test data\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92d2150419859ae87f726c9dcfcdb9b9c6adc056"},"cell_type":"code","source":"train_df_copy['geoNetwork.city'].replace('not available in demo dataset',np.NaN,inplace=True)\ntrain_df_copy['geoNetwork.metro'].replace('not available in demo dataset',np.NaN, inplace=True)\ntrain_df_copy['geoNetwork.region'].replace('not available in demo dataset',np.NaN, inplace=True)\ntrain_df_copy['trafficSource.campaign'].replace('(not set)',np.NaN,inplace=True)\n\ntest_df['geoNetwork.city'].replace('not available in demo dataset',np.NaN,inplace=True)\ntest_df['geoNetwork.metro'].replace('not available in demo dataset',np.NaN, inplace=True)\ntest_df['geoNetwork.region'].replace('not available in demo dataset',np.NaN, inplace=True)\ntest_df['trafficSource.campaign'].replace('(not set)',np.NaN,inplace=True)\n\n#geoNetwork.city, geoNetwork.metro, geoNetwork.region, \n# and trafficSource.campaign","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d3febe48f394e81f4e81a5e6e52abb58ab04c6a"},"cell_type":"code","source":"## fillna for the other objects\nfor col in ['trafficSource.keyword',\n            'trafficSource.referralPath',\n            'trafficSource.adwordsClickInfo.adNetworkType',\n            'trafficSource.adwordsClickInfo.isVideoAd',\n            'trafficSource.adwordsClickInfo.slot',\n            'trafficSource.adContent',\n            'geoNetwork.city',\n            'geoNetwork.metro',\n            'geoNetwork.region',\n            'trafficSource.campaign'\n            ]:\n    \n    train_df_copy[col].fillna('unknown', inplace=True)\n    test_df[col].fillna('unknown', inplace=True)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e473dfb418a2f12312c851decaa6c382d40f3f6a"},"cell_type":"code","source":"# drop constant columns\nconstant_column = [col for col in train_df_copy.columns if train_df_copy[col].nunique() == 1]\n\nprint('drop columns:', constant_column)\ntrain_df_copy.drop(constant_column, axis=1, inplace=True)\n\ntest_df.drop(constant_column, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b81abb520609dd94f293d95f675ab429fd516030"},"cell_type":"code","source":"train_df_copy.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f842425b8ca27b1b2fa42166516d5b0c6154c400"},"cell_type":"markdown","source":"Converting features that are objects but should be integers"},{"metadata":{"trusted":true,"_uuid":"49070b7a43ea827b9d54b68469b750954958f14e"},"cell_type":"code","source":"# totals.bounces, totals.hits, totals.newVisits\ntrain_df_copy['totals.bounces'] = train_df_copy['totals.bounces'].astype(int)\ntrain_df_copy['totals.hits'] = train_df_copy['totals.hits'].astype(int)\ntrain_df_copy['totals.newVisits'] = train_df_copy['totals.newVisits'].astype(int)\n\ntrain_df_copy['trafficSource.adwordsClickInfo.isVideoAd'] = train_df_copy['trafficSource.adwordsClickInfo.isVideoAd'].astype(bool)\n\ntest_df['totals.bounces'] = test_df['totals.bounces'].astype(int)\ntest_df['totals.hits'] = test_df['totals.hits'].astype(int)\ntest_df['totals.newVisits'] = test_df['totals.newVisits'].astype(int)\n\ntest_df['trafficSource.adwordsClickInfo.isVideoAd'] = test_df['trafficSource.adwordsClickInfo.isVideoAd'].astype(bool)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"156dd56306f40c097b263b444c9b244d4b2826d8"},"cell_type":"markdown","source":"Now that we have a clean and not-missing dataframe, we start thinking about the date and visitStartTime columns.\n\n**Date**: I only keep the day and the weekday features from the date column because in the test set, we would have different months and different year, so including them in the train set will only make the model overfit\n\n**VisitStartTime**: For now I remove this column because it is all in UTC time (time of the main google server) and it is only useful if we can combine it with the geographical information in order to see this VisitStartTime in local time. "},{"metadata":{"trusted":true,"_uuid":"2fae9a8e1a89e6ea84fcd33f9a4d97a37bd6f351"},"cell_type":"code","source":"# Parsing the date. \nformat_str = '%Y%m%d'\ntrain_df_copy['formated_date'] = train_df_copy['date'].apply(lambda x: datetime.strptime(str(x), format_str))\n\ntrain_df_copy['day'] = train_df_copy['formated_date'].apply(lambda x:x.day)\ntrain_df_copy['weekday'] = train_df_copy['formated_date'].apply(lambda x:x.weekday())\n\ntrain_df_copy.drop(['formated_date','date'], axis=1, inplace=True)\n\n##TEST\ntest_df['formated_date'] = test_df['date'].apply(lambda x: datetime.strptime(str(x), format_str))\n\ntest_df['day'] = test_df['formated_date'].apply(lambda x:x.day)\ntest_df['weekday'] = test_df['formated_date'].apply(lambda x:x.weekday())\n\ntest_df.drop(['formated_date','date'], axis=1, inplace=True)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"576dc214645799ec2b3656611e6eb0038178dfb9"},"cell_type":"code","source":"train_df_copy['day'] = train_df_copy['day'].astype('category')\ntrain_df_copy['weekday'] = train_df_copy['weekday'].astype('category')\n\n##TEST\ntest_df['day'] = test_df['day'].astype('category')\ntest_df['weekday'] = test_df['weekday'].astype('category')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a2c68f8a27f54b6cc1ff0a4d1f94aadb904ea07"},"cell_type":"code","source":"train_df_copy.drop(['visitStartTime'], axis=1, inplace=True)\ntest_df.drop(['visitStartTime'], axis=1, inplace=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f00f845192dc6ca1da41356b9d7035328a93ac1"},"cell_type":"code","source":"train_df_copy.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7298a8dc32ff7ef3d9db1e702e1b40a6df2338c0"},"cell_type":"markdown","source":"Investigating which features might have correlations with the transactionRevenue"},{"metadata":{"trusted":true,"_uuid":"1fd2756e36282f008bf1620281eb57b69389cbb6"},"cell_type":"code","source":"train_df_copy['totals.transactionRevenue'] = train_df_copy['totals.transactionRevenue'].astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6932b831fc8158d481f5cbe7b1e5bd80834e8df2"},"cell_type":"code","source":"def mean_transactionRevenue_plot(feature):\n    ax = train_df_copy.groupby(feature)['totals.transactionRevenue'].mean().plot.bar()\n    ax.set_ylabel('mean transaction revenue')\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0147655cabbad44b67fff38b64ca44aef3e1ba7c"},"cell_type":"code","source":"mean_transactionRevenue_plot('channelGrouping')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2dde63f6e78bbb7f917e1e9b71379b92478d4a7d"},"cell_type":"code","source":"mean_transactionRevenue_plot('device.browser')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48c5080e251bc88732bf7517ce9fb71b91d48129"},"cell_type":"code","source":"mean_transactionRevenue_plot('device.deviceCategory')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c191327fa63703e16acc4a0ea50c949da81425f"},"cell_type":"code","source":"mean_transactionRevenue_plot('device.operatingSystem')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"681768c7f22aa80082bde018e59f6cfa42ac6d89"},"cell_type":"code","source":"mean_transactionRevenue_plot('geoNetwork.continent')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a6ba1a01a905e1c96d46688eea4fdc411f54436"},"cell_type":"code","source":"mean_transactionRevenue_plot('geoNetwork.country')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3389c54b8c23cf23dcdbfb30d62643fb404e94f"},"cell_type":"code","source":"mean_transactionRevenue_plot('geoNetwork.subContinent')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d983fac545ecafed4a06e92013ea108c55a6288d"},"cell_type":"code","source":"train_df_copy.groupby('totals.bounces')['totals.transactionRevenue'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"269e2563c74c2577211ac795d4a18fdab2f5b5a6"},"cell_type":"code","source":"mean_transactionRevenue_plot('totals.bounces')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1304082eab8a6d2a9bdba889daed8739ba92d067"},"cell_type":"code","source":"mean_transactionRevenue_plot('totals.hits')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"805f8d759e91db0ab379c9e342b9d45cb5998deb"},"cell_type":"code","source":"mean_transactionRevenue_plot('totals.newVisits')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"660b52463d1564da2ad5beb410a8d1fad7e74c4a"},"cell_type":"code","source":"mean_transactionRevenue_plot('totals.pageviews')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6cfed112597c34e1cfd2d8aa876b7afc9b631e95"},"cell_type":"code","source":"mean_transactionRevenue_plot('trafficSource.adContent')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c48638f7f5e6084b6e1db21ac51407289fa3ec75"},"cell_type":"code","source":"train_df_copy['trafficSource.adwordsClickInfo.isVideoAd'].describe()\nmean_transactionRevenue_plot('trafficSource.adwordsClickInfo.isVideoAd')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad2a5f1d714a76cd78329234c83082be5b6635ac"},"cell_type":"code","source":"train_df_copy['trafficSource.adwordsClickInfo.page'].describe()\nmean_transactionRevenue_plot('trafficSource.adwordsClickInfo.page')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"728f9427bc381ad31379e8e14772714e36bbbd42"},"cell_type":"code","source":"train_df_copy['trafficSource.adwordsClickInfo.slot'].describe()\nmean_transactionRevenue_plot('trafficSource.adwordsClickInfo.slot')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6953daec1c5d30ccd757f18cddb166c9aa9a7650"},"cell_type":"code","source":"train_df_copy['trafficSource.isTrueDirect'].describe()\nmean_transactionRevenue_plot('trafficSource.isTrueDirect')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e6b961ab72a9a8e6f41a3ea538539170850687fc"},"cell_type":"code","source":"train_df_copy['trafficSource.keyword'].describe()\ntrain_df_copy.drop('trafficSource.keyword',axis=1,inplace=True)\ntest_df.drop('trafficSource.keyword',axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c4c1eece5c628aa9583c0942b43041951f620c5"},"cell_type":"code","source":"train_df_copy['trafficSource.medium'].describe()\nmean_transactionRevenue_plot('trafficSource.medium')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35d1d5d34fae7b6ca3c35ea8fe5801e6959be542"},"cell_type":"code","source":"train_df_copy['trafficSource.referralPath'].value_counts()\n#train_df_copy.drop('trafficSource.referralPath',axis=1,inplace=True)\n#test_df.drop('trafficSource.referralPath',axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0aab8468ad6eea5863d6a0aa8fbae951f9846f2"},"cell_type":"code","source":"train_df_copy['trafficSource.source'].describe()\nmean_transactionRevenue_plot('trafficSource.source')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc67e3480b63f103d771950769124838b5b6e1b1"},"cell_type":"code","source":"mean_transactionRevenue_plot('weekday')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"880fdffea3cefb3615afbb67981edacd26b21935"},"cell_type":"code","source":"mean_transactionRevenue_plot('geoNetwork.continent')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e6a4e028a8e15d14cef4aa702696984f325e0a25"},"cell_type":"code","source":"mean_transactionRevenue_plot('geoNetwork.country')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0cb98665679ee25fd874f019a599e415b8c1f3c3"},"cell_type":"code","source":"train_df_copy.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4eec2fc45bac336c02ef902ac3914da1ac2ac47"},"cell_type":"markdown","source":"Let's now get all Dummy variables"},{"metadata":{"trusted":true,"_uuid":"5c0ef33f7d96e41c253e21b9411d1ecb25f9741d"},"cell_type":"code","source":"# %% Categorical columns\n# List of categorical columns to recode\ncatCols = ['channelGrouping', 'device.browser','device.deviceCategory',\n           'device.operatingSystem', 'geoNetwork.continent','geoNetwork.country',\n           'geoNetwork.networkDomain',\n           'geoNetwork.subContinent', 'trafficSource.adContent', \n           'trafficSource.adwordsClickInfo.adNetworkType',\n           'trafficSource.adwordsClickInfo.slot',\n           'trafficSource.source','trafficSource.medium',\n           'day','weekday', 'trafficSource.referralPath',\n           'geoNetwork.city', \n           'geoNetwork.metro', \n           'geoNetwork.region', \n           'trafficSource.campaign'\n          ]\n\n# Recode\nfor c in catCols:\n    # Convert column to pd.Categotical\n    train_df_copy[c] = pd.Categorical(train_df_copy[c])\n    test_df[c] = pd.Categorical(test_df[c])\n\n    # Extract the cat.codes and replace the column with these\n    train_df_copy[c] = train_df_copy[c].cat.codes\n    test_df[c] = test_df[c].cat.codes\n\n    # Convert the cat codes to categotical...\n    train_df_copy[c] = pd.Categorical(train_df_copy[c])\n    test_df[c] = pd.Categorical(test_df[c])\n\n\n\n# Generate a logical index of categorical columns to maybe use with LightGBM later\ncatCols = [i for i,v in enumerate(train_df_copy.dtypes) if str(v)=='category']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5aed4946ad28ca06c31f33b83835b41479c7cec5"},"cell_type":"code","source":"#train_df_copy.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de491fcaa86fd06cd1cf9cbd18137526f90345da"},"cell_type":"markdown","source":"Remove the highly correlated features:\n\ntotals.newVisits and trafficSource.isTrueDirect --> drop totals.newVisits\n\ntrafficSource.adwordsClickInfo.isVideoAd and trafficSource.adwordsClickInfo.slot  --> drop both\n\ntotals.pageviews and totals.hits --> drop totals.hits\n"},{"metadata":{"trusted":true,"_uuid":"7c301025ad2e0fe0cac5a4af2bda0787cbdc3f59"},"cell_type":"code","source":"train_df_copy.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6943d0e83f6c3580f9332e866a4b8d60f88f84d2"},"cell_type":"code","source":"#%% Prepare data\ndef prepLGB(data,\n            classCol='',\n            IDCol='',\n            fDrop=[]):\n\n        # Drop class column\n        if classCol != '':\n            labels = data[classCol]\n            fDrop = fDrop + [classCol]\n        else:\n            labels = []\n\n        if IDCol != '':\n            IDs = data[IDCol]\n        else:\n            IDs = []\n\n        if fDrop != []:\n           data = data.drop(fDrop,\n                            axis=1)\n\n        # Create LGB mats\n        lData = lgb.Dataset(data, label=labels,\n                            free_raw_data=False,\n                            feature_name=list(data.columns),\n                            categorical_feature='auto')\n\n        return lData, labels, IDs, data\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ec2c44ee68325c4e6950dfb380c70b046c32892"},"cell_type":"code","source":"train_df_copy['totals.transactionRevenue'] = np.log1p(train_df_copy['totals.transactionRevenue'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cd9037725abacb21e245a3952222ae71b8a8e15c"},"cell_type":"code","source":"import lightgbm as lgb\n#reg = lgb.train(params, d_train, 100)\nfrom sklearn.model_selection import train_test_split\nimport pandas as pd\nfrom sklearn.externals import joblib\nfrom sklearn.model_selection import RandomizedSearchCV,GridSearchCV\n\nfrom sklearn.metrics import make_scorer, mean_squared_error,roc_auc_score, roc_curve\n\nfrom sklearn.pipeline import Pipeline\nimport numpy as np","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9d806f1fc7aa82b317decb53e3b3634e5e6f616"},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1dc657bde058a1c78b1444a97c6d6ceb072f2fff"},"cell_type":"code","source":"# Split training data in to training and validation sets.\n# Validation set is used for early stopping.\ntrainData, validData = train_test_split(train_df_copy,\n                                        test_size=0.3)\n\n# Prepare the data sets\ntrainDataL, trainLabels, trainIDs, trainData = prepLGB(trainData,\n                                                 classCol='totals.transactionRevenue',\n                                                 IDCol='',\n                                                 fDrop=[])\n\nvalidDataL, validLabels, validIDs, validData = prepLGB(validData,\n                                                 classCol='totals.transactionRevenue',\n                                                 IDCol='',\n                                                 fDrop=[])\n\ntestDataL, _, _ , testData = prepLGB(test_df,\n                                  classCol='',\n                                     IDCol='fullVisitorId',\n                                     fDrop=[])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e4e91fb9cf6c2431844843f94bf65789258bde1"},"cell_type":"code","source":"# Prepare data set using all the training data\nallTrainDataL, allTrainLabels, _ , allTrainData = prepLGB(train_df_copy,\n                                                 classCol='totals.transactionRevenue',\n                                                 IDCol='',\n                                                 fDrop=[])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51949bd12a4605ccdfdb636c001853706cf4e465"},"cell_type":"markdown","source":"We can start training the model"},{"metadata":{"trusted":true,"_uuid":"ea524c0ca8a407a40063200152a7492423b76870"},"cell_type":"code","source":"#inspecting the target variable\nsns.kdeplot(np.log(train_df_copy['totals.transactionRevenue']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"400841368464f20b06f44083e5d3cfce844f7872"},"cell_type":"code","source":"#y = np.log1p(train_df_copy['totals.transactionRevenue'])\n#X = train_df_copy.drop('totals.transactionRevenue',axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f02a3defb00e3fe36c73955569f2476b8e95f4a2"},"cell_type":"code","source":"#X.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c3962f17c61843e6102f799bda1df5723c06ff4c"},"cell_type":"markdown","source":"Good EXplanation on LGBM: https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc"},{"metadata":{"trusted":true,"_uuid":"59a7879353c06321f6011a798950defbb7f0ddf4"},"cell_type":"code","source":"#https://www.kaggle.com/garethjns/microsoft-lightgbm-with-parameter-tuning-0-823\n#https://lightgbm.readthedocs.io/en/latest/Python-API.html?highlight=fit","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f22e9323eaabdb395a6129f7e277d0484ec0325f"},"cell_type":"markdown","source":"LightGBM offers good accuracy with integer-encoded categorical features. LightGBM applies Fisher (1958) to find the optimal split over categories as described here. This often performs better than one-hot encoding.\nUse categorical_feature to specify the categorical features. Refer to the parameter categorical_feature in Parameters.\nCategorical features must be encoded as non-negative integers (int) less than Int32.MaxValue (2147483647). It is best to use a contiguous range of integers started from zero.\nIt is common to represent categorical features with one-hot encoding, but this approach is suboptimal for tree learners. Particularly for high-cardinality categorical features, a tree built on one-hot features tends to be unbalanced and needs to grow very deep to achieve good accuracy.\n\nInstead of one-hot encoding, the optimal solution is to split on a categorical feature by partitioning its categories into 2 subsets. "},{"metadata":{"_uuid":"76f14499dd146f59445fda99738880569a27e4a9"},"cell_type":"markdown","source":"Trying the simplest model"},{"metadata":{"trusted":true,"_uuid":"93ca3e1e5fb729a9fc730cad3e519239f9376a94"},"cell_type":"code","source":"#grouped_test = test_df[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\n#grouped_test.to_csv('submit.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2ba68e3255910cf50e7dc67de6f522e6403f47bc"},"cell_type":"markdown","source":" \"categorical_feature\"=name: ['channelGrouping','device.browser','device.deviceCategory',\n                                     'device.operatingSystem','geoNetwork.continent',\n                                     'geoNetwork.country','geoNetwork.subContinent',\n                                      'trafficSource.adContent',\n                                      'trafficSource.adwordsClickInfo.adNetworkType',\n                                      'trafficSource.medium', 'trafficSource.source',\n                                      'day', 'weekday'\n                                     ]"},{"metadata":{"_uuid":"a6588b97af586a6ac86f0b77afec2a8b45b12009"},"cell_type":"markdown","source":"My sources for LBGM:\nhttps://www.kaggle.com/garethjns/microsoft-lightgbm-with-parameter-tuning-0-823\nhttps://www.kaggle.com/ievgenvp/lgbm-custom-randomizedsearchcv-lb-283\nhttps://www.kaggle.com/vinnsvinay/introduction-to-boosting-using-lgbm-lb-0-68357\nhttps://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue\nhttps://www.kaggle.com/karkun/sergey-ivanov-msu-mmp\nhttps://www.kaggle.com/ogrellier/i-have-seen-the-future\nhttps://www.kaggle.com/plasticgrammer/customer-revenue-prediction-playground"},{"metadata":{"trusted":true,"_uuid":"36cc240ca73aead5f5150864e8288a26780074b9"},"cell_type":"code","source":"lgbm_model.get_params().keys()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2d07bb76b58497b290df1902f160608a7dfe924"},"cell_type":"code","source":"import lightgbm as lgb\n#reg = lgb.train(params, d_train, 100)\nfrom sklearn.model_selection import train_test_split\nimport pandas as pd\nfrom sklearn.externals import joblib\nfrom sklearn.model_selection import RandomizedSearchCV,GridSearchCV\n\nfrom sklearn.metrics import make_scorer, mean_squared_error,roc_auc_score, roc_curve\n\nfrom sklearn.pipeline import Pipeline\nimport numpy as np\n\n    \n    #Hiper parameters to explore\nparams = {\"learning_rate\":np.arange(0.1,0.2,0.05),\n              \"metric\": ['rmse'],\n              \"num_leaves\": [70,80],\n              \"min_data_in_leaf\": [100,1000],\n              \"max_depth\": [7],\n              'objective': ['regression'],\n              'boosting': ['gbdt'],\n          'bagging_fraction': [0.7,0.8],\n        'bagging_freq': [1,3,5],\n        'feature_fraction': [0.5,0.7],\n        'max_bin': [128, 255],\n          \"min_child_samples\" : [100]\n             \n                  }\n                \n#lgbm_model = lgb.train()\nlgbm_model = lgb.LGBMRegressor(boosting_type= 'gbdt',\n          objective = 'regression',\n          verbose = 5,\n          max_depth = params['max_depth'],\n          learning_rate = params['learning_rate'],\n          metric = ['rmse'],\n                               num_leaves = params['num_leaves'],\n                               min_data_in_leaf = params['min_data_in_leaf'],\n                               bagging_fraction= params['bagging_fraction'],\n        bagging_freq= params['bagging_freq'],\n                               feature_fraction= params['feature_fraction'],\n                               max_bin= params['max_bin'],\n                               min_child_samples = params['min_child_samples']\n                                         \n                              ) #Regressor\n\n# To view the default model params:\n\nlgbm_model.get_params().keys()\n\ngrid = RandomizedSearchCV(lgbm_model, params, cv=5, n_jobs=4, verbose = 8)\ngrid.fit(allTrainData, allTrainLabels)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"57c897c2a25e4afe83f31ccb6e23e998a0fc774c"},"cell_type":"code","source":"print(grid.best_params_)\nprint(grid.best_score_)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ecf4d63e2eb43ac70db0e114390e4f953113416"},"cell_type":"code","source":"# Using parameters already set above, replace in the best from the grid search\nparams['max_depth'] = grid.best_params_['max_depth']\nparams['learning_rate'] = grid.best_params_['learning_rate']\nparams['num_leaves'] = grid.best_params_['num_leaves']\nparams['min_data_in_leaf'] = grid.best_params_['min_data_in_leaf']\n\nparams['min_child_samples'] = grid.best_params_['min_child_samples']\n\nparams['bagging_fraction'] = grid.best_params_['bagging_fraction']\nparams['bagging_freq'] = grid.best_params_['bagging_freq']\nparams['feature_fraction'] = grid.best_params_['feature_fraction']\nparams['max_bin'] = grid.best_params_['max_bin']\n\n\nprint('Fitting with params: ')\nprint(params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f89a682147809109ad67185b4c60ee37892dc87"},"cell_type":"code","source":"k = 4\n\nfor i in range(0, k):\n    print('Fitting model', k)\n\n    # Prepare the data set for fold\n    trainData, validData = train_test_split(train_df_copy,\n                                            test_size=0.4)\n    trainDataL, trainLabels, trainIDs, trainData = prepLGB(trainData,\n                                                     classCol='totals.transactionRevenue',\n                                                     IDCol='',\n                                                     fDrop=[])\n    validDataL, validLabels, validIDs, validData = prepLGB(validData,\n                                                     classCol='totals.transactionRevenue',\n                                                     IDCol='',\n                                                     fDrop=[])\n    # Train\n    gbm = lgb.train(params,\n                    trainDataL,\n                    100000,\n                    valid_sets=[trainDataL, validDataL],\n                    early_stopping_rounds=50,\n                    verbose_eval=4)\n\n                    #valid_sets=[trainDataL, validDataL],\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8aef5bcca9c62da5ac104a528ec1a1736679daf8"},"cell_type":"code","source":"lgb.plot_importance(gbm,height=0.5,figsize=(10,10))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9fd585d647aeeb53b9de934ada9b48b4945d2e4"},"cell_type":"code","source":"testData.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3a516d021c02e0526a98936488e7b494c7aef94e"},"cell_type":"code","source":"predsValid = 0\npredsTrain = 0\npredsTest = 0\n\npredsValid += gbm.predict(validData,num_iteration=gbm.best_iteration)/k\npredsTrain += gbm.predict(trainData,num_iteration=gbm.best_iteration)/k\npredsTest += gbm.predict(testData.drop('fullVisitorId',axis=1),num_iteration=gbm.best_iteration)/k\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b32b2efa65e089bd2190ded066e2192f085edb35"},"cell_type":"code","source":"#preds_Test = pd.concat(testData['fullVisitorId'],predsTest,axis=1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f724f4d80e96cc0c80436c5a459966b38cbb34dc"},"cell_type":"code","source":"predsTest[predsTest<0] = 0\nsub = pd.DataFrame()\nsub[\"PredictedLogRevenue\"] = np.expm1(predsTest)\nsub['fullVisitorId'] = testData['fullVisitorId']\n\nsub = sub.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub[\"PredictedLogRevenue\"] = np.log1p(sub[\"PredictedLogRevenue\"])\nsub.to_csv(\"baseline_lgb1.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2165159c23e8e88d4e59991cf3be1ad0e56b93f"},"cell_type":"code","source":"sub.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b99ca6c2b20f17f2d1986d768fabcdb0f983aea3"},"cell_type":"code","source":"sub.head(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"783848dab367f678660b6fd9ab98a35622465b52"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}