{"cells":[{"metadata":{"trusted":true,"_uuid":"85e86a3b091c3ad573a201120fedff75a89876e4"},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4750fdb6f13eec95bcad83a016c57a63fe11952"},"cell_type":"markdown","source":"Function to load json files.. taken from https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aea8d808ddedd900aff77abfd66bdd7f466c3d61"},"cell_type":"markdown","source":"Not sure how we got the input files here... will revert later"},{"metadata":{"_uuid":"679d84d693888c93c63762754e36a58749be5b4d","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport json\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport datetime as datetime","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14307ee628d04b24be8bd2199903c777d8081fff"},"cell_type":"markdown","source":"Import a lot of stuff. I don't know how to use most of these."},{"metadata":{"trusted":true,"_uuid":"a6430eaf18e19cc83ff3bd1a322668fde9f43f2a"},"cell_type":"code","source":"%%time \ntrain_data = load_df()\ntest_data = load_df(\"../input/test.csv\")\ntrain_data.head()\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82264c263a0654d5a2cf5a76e0daebedb37c9e61"},"cell_type":"markdown","source":"Shows us some data- some of these in the curly brackets look useless. And by that I mean, I don't know how to use them."},{"metadata":{"trusted":true,"_uuid":"86ef2c2d21767682cf351db547c3119a2339bd5e"},"cell_type":"code","source":"train_data.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b8ad0e4d6bd6900b0f25d6f33a7312e6b82f48f"},"cell_type":"markdown","source":"Any way to show these in appropriate units?"},{"metadata":{"trusted":true,"_uuid":"2a74d89743bd1ff994e390ffb6a028aeaca5f4b7"},"cell_type":"code","source":"list(train_data.columns.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2e880ed5017f078fda5067e0b17c53c980d4fea1"},"cell_type":"markdown","source":"**Features descriptions**:\n\nReturning back to Data description for understanding features.\n\n*     channelGrouping - The channel via which the user came to the Store.\n*     date - The date on which the user visited the Store.\n*     geoNetwork - This section contains information about the geography of the user.\n*     sessionId - A unique identifier for this visit to the store.\n*     socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n*     totals - This section contains aggregate values across the session.\n*     trafficSource - This section contains information about the Traffic Source from which the session originated.\n*     visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n*     visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n*     visitStartTime - The timestamp (expressed as POSIX time).\n"},{"metadata":{"trusted":true,"_uuid":"dcb51011a920d5741ce007c15fe6f9cc122ab234"},"cell_type":"markdown","source":"**1. channelGrouping** - the channel via which the user came to the store. By far the largest was organic search. I don't know what social or direct means anyways, so who knows how useful this will be."},{"metadata":{"trusted":true,"_uuid":"24d6d732b0e816e2a890c817dd5543ab3e5e561c","scrolled":false},"cell_type":"code","source":"(train_data['channelGrouping'].value_counts()).plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98d005134d4677798e3c1431a6d95e8ac55ebe3a"},"cell_type":"markdown","source":"**2. date and visitStartTime**\n\n"},{"metadata":{"trusted":true,"_uuid":"4a21ac9e2c038995911eb25f643e157878ae9c12"},"cell_type":"code","source":"train_data['date'] = pd.to_datetime(train_data['date'], format=\"%Y%m%d\")\n(train_data['date'].value_counts()).plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9426c06830d5872f44195d2f1c4acbafea4f6649"},"cell_type":"markdown","source":"Spikes during holiday seasons to no surprise... also random spikes during summer? Hopefully will add in more granular labels."},{"metadata":{"_uuid":"337926fa6261abcbd7fa6fd0804f300c81316a46"},"cell_type":"markdown","source":"**3. socialEngagementType**\nThis is useless.. no one is socially engaged."},{"metadata":{"trusted":true,"_uuid":"7b3911b4fee714c5f1d2c5dc2b033f7c7c7d1e01","scrolled":false},"cell_type":"code","source":"(train_data['socialEngagementType'].value_counts()).plot(kind ='bar')\nplt.xticks(rotation='horizontal')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a1a7ed010b0ae312a18612e20fc3a4a0e78f3a8"},"cell_type":"markdown","source":"**4. visitNumber**\n\nAlmost no one visits more than once!"},{"metadata":{"trusted":true,"_uuid":"280d50807c7221484f954f4baef407d583d55795"},"cell_type":"code","source":"(train_data['visitNumber'].value_counts()).plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fef0db81633664554f871a768cee80fe0ee949b"},"cell_type":"markdown","source":"**5. Devices**"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"7fb3f63ebce8c18b47c5ee7b36ca83cda5ed8562"},"cell_type":"code","source":"train_data.groupby('device.browser')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\ntrain_data.groupby('device.deviceCategory')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\ntrain_data.groupby('device.operatingSystem')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f83a9cf96287c7628d6cfea8212b20788f8a598a"},"cell_type":"markdown","source":"**6. Traffic Source**"},{"metadata":{"trusted":true,"_uuid":"374b77d48f0a866ce4e7b0b3ad1f9d5768fef110"},"cell_type":"code","source":"train_data.groupby('trafficSource.source')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\ntrain_data.groupby('trafficSource.medium')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1fe71d696dd9ffa7acbdaa3f4e59973c25f95b2c"},"cell_type":"markdown","source":"**7. Continent**"},{"metadata":{"trusted":true,"_uuid":"75e8832d43958518b04e609fb8c23816d99b357a"},"cell_type":"code","source":"train_data.groupby('geoNetwork.continent')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\ntrain_data.groupby('geoNetwork.subContinent')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\ntrain_data.groupby('geoNetwork.networkDomain')['totals.transactionRevenue'].agg(['mean']).sort_values(by=\"mean\", ascending=True).head(10).plot(kind = 'barh')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"202ae6ffee582e7550d0846c5655c965b68a4ef2"},"cell_type":"markdown","source":"**8. Unique customers**"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"8479e8e55655fa93d7c912926eda7ca563a4713a"},"cell_type":"code","source":"train_data[\"totals.transactionRevenue\"] = train_data[\"totals.transactionRevenue\"].astype('float')\ngdf = train_data.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\nNonzero_instance = pd.notnull(train_data[\"totals.transactionRevenue\"]).sum()\nUnique_customer = (gdf[\"totals.transactionRevenue\"] > 0).sum()\n\nprint(\"There were\", Nonzero_instance, \"instances of activities that involved non-zero revenues.\")\nprint(\"The ratio of activities that involved non-zero revenues to total activities was\", '{:.2%}'.format(Nonzero_instance/train_data.shape[0]))\nprint()\nprint(\"There were\", Unique_customer, \"instances of activities that involved non-zero revenues from unique customers.\")\nprint(\"The ratio of unique activities that involved non-zero revenues to unique total activities was\", '{:.2%}'.format(Unique_customer/gdf.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b11146f0b7a185b25616d97de55d1b1eeac330a1"},"cell_type":"markdown","source":"This means that the vast majority of predicted values should be 0. The 1% will feature heavily in the analysis of this dataset. Let's take a look at the training set and the test set, just from the perspective of unique visitors."},{"metadata":{"trusted":true,"_uuid":"41bffff73273e0007301b72dced0d726032d0a51"},"cell_type":"code","source":"print(\"In the train set, there were\", train_data.fullVisitorId.nunique(), \"unique visitors. There were\", train_data.shape[0], \"total non-unique visitors.\")\nprint(\"In the test set, there were\", test_data.fullVisitorId.nunique(), \"unique visitors. There were\", test_data.shape[0], \"total non-unique visitors.\")\nprint(\"There were\", len(set(train_data.fullVisitorId.unique()).intersection(set(test_data.fullVisitorId.unique()))), \"common visitors in the two data sets.\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5e66ff585db4f17ffd2b6ef7b332598b61586da"},"cell_type":"markdown","source":"Why don't we just set these common visitors to what they were in the training set? It may actually be a good idea to see if non-unique visitors change their spending habits over time. Two hypotheses:\n1) If they visit more than once, they are much more likely to spend money on the second time around.\n2) If they spent the first time, they will also spend the second time. The likelihood of spending the first time and not spending the second time is near 0."},{"metadata":{"_uuid":"cf5acfed966501b2dd89beb73ec431f4a8adc288"},"cell_type":"markdown","source":"**II. Regression**\n\nBelow regression borrows heavily from: https://www.kaggle.com/vik1511/a-simple-data-visualization-lgbm"},{"metadata":{"trusted":true,"_uuid":"ca9df9647155122b0c7d2882aeec771a4c709e90"},"cell_type":"code","source":"\ntrain_x = train_data.drop(['date', 'fullVisitorId', 'sessionId', 'visitId', 'totals.transactionRevenue', 'socialEngagementType', 'device.browserSize', 'device.browserVersion', 'device.deviceCategory',\n        'device.flashVersion', 'device.isMobile','device.screenResolution',  'device.language',\n        'device.mobileDeviceBranding', 'device.mobileDeviceInfo',\n        'device.mobileDeviceMarketingName', 'device.mobileDeviceModel',\n        'device.mobileInputSelector', 'device.operatingSystem',\n        'device.operatingSystemVersion', 'device.screenColors','geoNetwork.cityId',\n        'geoNetwork.continent', 'geoNetwork.latitude',\n        'geoNetwork.longitude', 'geoNetwork.networkDomain',\n        'geoNetwork.networkLocation', 'geoNetwork.region',\n        'geoNetwork.subContinent', 'visitStartTime', 'totals.bounces', 'totals.hits', 'totals.newVisits', 'totals.visits',\n        'trafficSource.adContent',\n        'trafficSource.adwordsClickInfo.adNetworkType',\n        'trafficSource.adwordsClickInfo.criteriaParameters',\n        'trafficSource.adwordsClickInfo.gclId',\n        'trafficSource.adwordsClickInfo.isVideoAd',\n        'trafficSource.adwordsClickInfo.page',\n        'trafficSource.adwordsClickInfo.slot', 'trafficSource.campaign',\n        'trafficSource.campaignCode', 'trafficSource.isTrueDirect',\n        'trafficSource.keyword', 'trafficSource.referralPath'], axis=1)\ntrain_y = train_data['totals.transactionRevenue']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ef57019ab6dda2593b2bb83eec9e49abe21b9f6d"},"cell_type":"code","source":"train_x['totals.pageviews'] = train_x['totals.pageviews'].astype(float)\ncategorical_col = train_x.select_dtypes(include = [np.object]).columns\nnumerical_col = train_x.select_dtypes(include = [np.number]).columns\ncategorical_col, numerical_col","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1afeb80918838908b13af85da71880a3c3bb8621"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nle = LabelEncoder()\nfor col in categorical_col:\n    train_col = list(train_x[col].values.astype(str))\n    le.fit(train_col)\n    train_x[col] = le.transform(train_col)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4ac777e1c50d5909a58537a9413317fdfaddc1ec"},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nscaler = StandardScaler()\nfor col in numerical_col :\n    train_x[col] = (train_x[col] - np.mean(train_x[col])/(np.max(train_x[col]) - np.min(train_x[col])))\ntrain_x.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86e921de4a2aa4770f2b04b813f832bfee65476b"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ntrainX, crossX, trainY, crossY = train_test_split(train_x.values, train_y, test_size = 0.25, random_state = 20)\ntrainX","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffd0ffd248d07bb02b8357bd277f0e433c025d26"},"cell_type":"code","source":"import lightgbm as lgb\n\nlgb_params = {\"objective\":\"regression\", \"metric\": \"rmse\", \"num_leaves\": 500, \"learning_rate\" : 0.02, \"max_bin\":500, \"bagging_fraction\" : 0.75, \"feature_fraction\" : 0.8, \"bagging_frequency\" : 9, \"num_interation\": 1200}\nlgb_train = lgb.Dataset(trainX, label=trainY)\nlgb_val = lgb.Dataset(crossX, label=crossY)\nmodel = lgb.train(lgb_params, lgb_train, valid_sets=[lgb_val], early_stopping_rounds=100, verbose_eval=20)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}