{"cells":[{"metadata":{"toc":true,"_uuid":"85953a28a810794472fe876bbfbc00d5e3b0d30f"},"cell_type":"markdown","source":"

Table of Contents

\n
"},{"metadata":{"_uuid":"46bad00407fe240a59ccfd4e10f8e8eda8378f9f"},"cell_type":"markdown","source":"# Baseline Model 快速尝试\n\n*By Jared Gong*\n\n本文档专注于尝试快速建立 baseline model."},{"metadata":{"_uuid":"35d89641894acc2f745314268e4ba32c22ccfc4a"},"cell_type":"markdown","source":"## 比赛介绍\n\n>> Google Analytics Customer Revenue Prediction \n\n**Objective of the competition:**\n\n在本次竞赛中，我们面临的挑战是分析Google Merchandise Store（也称为GStore，销售谷歌周边）客户数据集，predicting the log of the total revenue per user, be aware that not all rows in `test.csv` will correspond to a row in the submission, but all unique `fullVisitorIds` will correspond to a row in the submission.\n\n\n有多个列包含不同深度的JSON blob。在其中一个JSON列（`total`）中，子列 `transactionRevenue` 包含我们尝试预测的收入信息。该子列仅存在于训练数据中。具体的来说，我们预测每个用户的所有交易总和的自然对数。对于测试集中的每个用户，目标是：\n\n$$\n\\text{target}^\\text{user} = \\ln \\sum_{i=1}^n \\text{transaction}_i^\\text{user}\n$$"},{"metadata":{"trusted":true,"_uuid":"ce9db2fc501be9db6f7ac302202539536f1182ca"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e00ee28bb0c6288f7f39439d8c433d042335b16"},"cell_type":"markdown","source":"## 相关有趣的知识点"},{"metadata":{"_uuid":"510d5477f418b38fc7ad73aa22c7481176f26966"},"cell_type":"markdown","source":"## 读入数据\n\n本数据有多个列包含不同深度的JSON blob。"},{"metadata":{"_uuid":"46591007b1ef380080e2886e174eda84faeaca95"},"cell_type":"markdown","source":"\n关于数据集：\n\n与大多数其他的kaggle比赛类似，我们给出了两个数据集\n\n train.csv\n test.csv\n\n数据集中的每一行都是对商店的一次访问。我们预测每个用户的所有交易总和的自然对数。\n\n需要注意的是，有多个列包含不同深度的JSON blob。在其中一个JSON列（`total`）中，子列 `transactionRevenue` 包含我们尝试预测的收入信息。"},{"metadata":{"code_folding":[4],"trusted":true,"_uuid":"ab47b4a20fbf2662e675e2c8a9db4ba9677c51d2"},"cell_type":"code","source":"import os\nimport json\nfrom pandas.io.json import json_normalize\n\ndef load_df(csv_path='../input/train.csv', nrows=None):\n JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n \n df = pd.read_csv(csv_path, \n converters={column: json.loads for column in JSON_COLUMNS}, \n dtype={'fullVisitorId': 'str'}, \n nrows=nrows) # Important!!\n \n for column in JSON_COLUMNS:\n column_as_df = json_normalize(df[column])\n column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n return df\n\nprint(os.listdir(\"../input\"))\n\ntrain = load_df(nrows=100000)\ntest =load_df('../input/test.csv', nrows=100000)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"045c2bf498d0ab6f7d8acbf7b61422c1b0724604"},"cell_type":"markdown","source":"\n总共12个变量：\n\n1. `channelGrouping` - 用户通过其进入商店的渠道。\n1. `date` - 用户访问Store的日期\n1. `device` - 用户访问 Store 的设备类型\n1. `fullVisitorId` - Google Merchandise Store的每个用户的唯一标识符。\n1. `geoNetwork` - 包含有关用户地理位置的信息。\n1. `sessionId` - 此次访问商店的唯一标识符。\n1. `socialEngagementType` - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n1. `totals` - This section contains aggregate values across the session.\n1. `trafficSource` - This section contains information about the Traffic Source from which the session originated.\n1. `visitId` - 此会话的标识符。这是通常存储为_utmb cookie的值的一部分。这仅对用户而言是唯一的。对于完全唯一的ID，您应该使用fullVisitorId和visitId的组合。\n1. `visitNumber` - 此用户的会话号。如果这是第一个会话，则将其设置为1。\n1. `visitStartTime` - 时间戳（表示为POSIX时间）。\n\n其中的四个变量 `device, geoNetwork, totals, trafficSource` 进一步处理成多个变量，最后分解成54个特征。"},{"metadata":{"_uuid":"8038b701f38976d5188e490e9886064cc55e59b3"},"cell_type":"markdown","source":"### Target variable"},{"metadata":{"_uuid":"c9f76502de219684b4dd65e7af35cf23f5b4b77b"},"cell_type":"markdown","source":"\n我们预测每个用户的所有交易总和的自然对数。对于测试集中的每个用户，目标是：\n\n$$\n\\text{target}^\\text{user} = \\ln \\sum_{i=1}^n \\text{transaction}_i^\\text{user}\n$$\n\n看看进行散点图。"},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"a75cad597b35e214688b81de969f3ee5838d4d3e"},"cell_type":"code","source":"train[\"totals.transactionRevenue\"] = train[\"totals.transactionRevenue\"].astype('float')\ngdf = train.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\n\nplt.figure(figsize=(6,4))\nplt.scatter(range(gdf.shape[0]), np.sort(np.log1p(gdf[\"totals.transactionRevenue\"].values)))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('TransactionRevenue', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b15437b24323b92fa7b747bba3c51a3085672365"},"cell_type":"code","source":"gdf.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"881293cb71bff519f430a1d6141f8cd554ede0fb"},"cell_type":"markdown","source":"从中可以看出一个重要的点：\n\n*The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.*"},{"metadata":{"_uuid":"84fc31d4c55e0c2c832ae28722e5d3da25ac1345"},"cell_type":"markdown","source":"### full Visitor ld\n\n与客户 Id 相关的变量包括 `fullVisitorId, sessionId, visitId, visitNumber`，一个 `fullVisitorId` 包含多个 `sessionId`, 而一个 `sessionId` 唯一对应一个 `visitId`，个数和 `visitNumber` 相关。\n\n> 我们需要创建一个以 `fullVisitorId` 为 index 的数据框。"},{"metadata":{"_uuid":"3917ba69094fb1d5ed0c3c028f14e276ab6396ed"},"cell_type":"markdown","source":"## 基本数据分析"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"265cba22ee33ad5e8e67bc4f915c432f5d2262fb"},"cell_type":"code","source":"zeroColumns = [c for c in train.columns if train[c].nunique() == 1]\nzeroColumns\n# 数据中方差不为0的特征","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5837134c3d46e6bcceb167cf057ad6516c549362"},"cell_type":"markdown","source":"### 目标变量"},{"metadata":{"trusted":true,"_uuid":"0778f7bbb4ab3b075972baa68e51101375919078"},"cell_type":"code","source":"nzi = pd.notnull(train[\"totals.transactionRevenue\"]).sum()\nnzr = (gdf[\"totals.transactionRevenue\"] > 0).sum()\nprint(\"Nusmber of instances in train set with non-zero revenue : \", nzi, \" and ratio is : \", nzi / train.shape[0])\nprint(\"Number of unique customers with non-zero revenue : \", nzr, \"and the ratio is : \", nzr / gdf.shape[0])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"989529c3dd8e1e8c37f5bb78e1fff52e9cb89da1"},"cell_type":"markdown","source":"访客人数和共同访客人数：\n\n现在让我们看一下训练和测试集中的不同访客数量，以及共同的访客的数量。"},{"metadata":{"trusted":true,"_uuid":"e76135c02dd09e4a28dbffacb3afac5e2b666388"},"cell_type":"code","source":"print(\"Number of unique visitors in train set : \",train.fullVisitorId.nunique(), \" out of rows : \",train.shape[0])\nprint(\"Number of unique visitors in test set : \",test.fullVisitorId.nunique(), \" out of rows : \",test.shape[0])\nprint(\"Number of common visitors in train and test set : \",len(set(train.fullVisitorId.unique()).intersection(set(test.fullVisitorId.unique())) ))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e169930f1b011cfb3816a083f32c1e305c331515"},"cell_type":"markdown","source":"### 设备与 target 的关系\n\n与设备相关的三个重要属性是：\n\n- 浏览器(chrome,firefox, IE)\n- 操作系统（mac, linux, windows）\n- 设备类型（移动端，桌面端和平板）\n\n我们考察这三个变量和目标变量的关系。"},{"metadata":{"trusted":true,"_uuid":"3802135f14ff558adc55b94dc242430e2bd5cfd2"},"cell_type":"code","source":"import seaborn as sns\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\ndef horizontal_bar_chart(cnt_srs, color):\n trace = go.Bar(\n y=cnt_srs.index[::-1],\n x=cnt_srs.values[::-1],\n showlegend=False,\n orientation = 'h',\n marker=dict(\n color=color,\n ),\n )\n return trace","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b96ce1dcf14a9f8cf02477b6e7f849b6391bac97"},"cell_type":"code","source":"# Device Browser\ncnt_srs = train.groupby('device.browser')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\n# Device Category\ncnt_srs = train.groupby('device.deviceCategory')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(71, 58, 131, 0.8)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(71, 58, 131, 0.8)')\n\n# Operating system\ncnt_srs = train.groupby('device.operatingSystem')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace5 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(246, 78, 139, 0.6)')\ntrace6 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10),'rgba(246, 78, 139, 0.6)')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=2, vertical_spacing=0.04, \n subplot_titles=[\"Device Browser - Count\", \"Device Browser - Non-zero Revenue Count\",\n \"Device Category - Count\", \"Device Category - Non-zero Revenue Count\",\n \"Device Operating System - Count\", \"Device Operating System - Non-zero Revenue Count\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace5, 3, 1)\nfig.append_trace(trace6, 3, 2)\n\nfig['layout'].update(height=1200, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"416d6f8c1e406b5b3ead15e8045b1fb07d70a7f5"},"cell_type":"markdown","source":"\n推论：\n\n- 设备浏览器分布在非零收入图的计数和计数方面看起来相似\n- 在设备类别方面，与移动设备相比，桌面似乎具有更高的非零收入计数百分比。\n- 在设备操作系统中，虽然计数的次数多来自Windows，但收入不为零的计数数量对于Macintosh来说更多。\n- Chrome操作系统的非零收入占比也较高\n- 在移动操作系统方面，与Android相比，iOS拥有更多的非零收入计数"},{"metadata":{"_uuid":"be2762c546fe72b2fe59048456939d3f80b1f479"},"cell_type":"markdown","source":"### target 当成时间序列\n\n"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"c16e6ffd1d70208e5055f709006eb6a2bb04b2c9"},"cell_type":"code","source":"import datetime\n\ndef scatter_plot(cnt_srs, color):\n trace = go.Scatter(\n x=cnt_srs.index[::-1],\n y=cnt_srs.values[::-1],\n showlegend=False,\n marker=dict(\n color=color,\n ),\n )\n return trace\n\ntrain['date'] = train['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = train.groupby('date')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_index()\n#cnt_srs.index = cnt_srs.index.astype('str')\ntrace1 = scatter_plot(cnt_srs[\"count\"], 'red')\ntrace2 = scatter_plot(cnt_srs[\"count of non-zero revenue\"], 'blue')\n\nfig = tools.make_subplots(rows=2, cols=1, vertical_spacing=0.08,\n subplot_titles=[\"Date - Count\", \"Date - Non-zero Revenue count\"])\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig['layout'].update(height=800, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Date Plots\")\npy.iplot(fig, filename='date-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad0f843c26ca2ad1d2f69cffa48e59d9cca81aaf"},"cell_type":"markdown","source":"\n推论：\n\n- 我们的训练数据集中包含2016年8月1日至2017年7月31日的数据\n- 2016年11月，虽然访客人数有所增加，但在此期间非零收入计数没有增加。"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"2b665dac32166528ba3f3b832f7613032fdb62fd"},"cell_type":"code","source":"test['date'] = test['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = test.groupby('date')['fullVisitorId'].size()\n\n\ntrace = scatter_plot(cnt_srs, 'red')\n\nlayout = go.Layout(\n height=400,\n width=800,\n paper_bgcolor='rgb(233,233,233)',\n title='Dates in Test set'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f8d957902e62aa2f541421a4672b0c277ca28d2"},"cell_type":"markdown","source":"\n在测试集中，我们的日期为2017年8月2日至2018年4月30日。因此，训练和测试集之间没有共同日期。因此，对此数据集进行基于时间的验证可能是个好主意。"},{"metadata":{"_uuid":"90df7c0369476ed7c0a55fc540bd1f48f9e63189"},"cell_type":"markdown","source":"### geographic 与 target 关系"},{"metadata":{"trusted":true,"_uuid":"d50ee41c7a85f0d21365c58d12908dc039cbf7c9"},"cell_type":"code","source":"# Geographic Information:\n\n# Continent\ncnt_srs = train.groupby('geoNetwork.continent')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(58, 71, 80, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(58, 71, 80, 0.6)')\n\n# Sub-continent\ncnt_srs = train.groupby('geoNetwork.subContinent')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"], 'orange')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"], 'orange')\n\n# Network domain\ncnt_srs = train.groupby('geoNetwork.networkDomain')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace5 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'blue')\ntrace6 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'blue')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=2, vertical_spacing=0.08, horizontal_spacing=0.15, \n subplot_titles=[\"Continent - Count\", \"Continent - Non-zero Revenue Count\",\n \"Sub Continent - Count\", \"Sub Continent - Non-zero Revenue Count\",\n \"Network Domain - Count\", \"Network Domain - Non-zero Revenue Count\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace5, 3, 1)\nfig.append_trace(trace6, 3, 2)\n\nfig['layout'].update(height=1200, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Geography Plots\")\npy.iplot(fig, filename='geo-plots')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96e1329684139bb75d30ae6cb916d9907e0f9b6a"},"cell_type":"markdown","source":"\n推论：\n\n- 在非洲大陆的情节中，我们可以看到，美国的计数数量更多，而且收入非零的计数最多\n- 虽然亚洲和欧洲的计数很多，但来自这些大陆的非零收入数量相对较低。\n- 我们也可以从次大陆的图推断前两点。\n- 如果网络域是“unknown.unknown”而不是“（未设置）”，则具有非零收入的计数数量往往较低。"},{"metadata":{"_uuid":"90d6dba1056bfeb68e1932e4db9fe19de8a85101"},"cell_type":"markdown","source":"### traffic source 和 target 的关系"},{"metadata":{"trusted":true,"_uuid":"d019d1580e578dbd4ae447934949f780a3136b10"},"cell_type":"code","source":"cnt_srs = train.groupby('trafficSource.source')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'green')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'green')\n\n# Sub-continent\ncnt_srs = train.groupby('trafficSource.medium')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"], 'purple')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"], 'purple')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=2, cols=2, vertical_spacing=0.08, horizontal_spacing=0.15, \n subplot_titles=[\"Traffic Source - Count\", \"Traffic Source - Non-zero Revenue Count\",\n \"Traffic Source Medium - Count\", \"Traffic Source Medium - Non-zero Revenue Count\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\n\nfig['layout'].update(height=1000, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Traffic Source Plots\")\npy.iplot(fig, filename='traffic-source-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd07021c4d6a1d40d47b4abefe3263073dc6362e"},"cell_type":"markdown","source":"\n推论：\n\n- 在流量来源图中，尽管Youtube在数据集中具有大量计数，但非零收入计数的数量非常少。\n- Google plex的非零收入计数与流量来源图中的总计数之比很高。\n- 在流量来源媒体上，与“organic”媒体相比，“referral”具有更多的非零收入计数。"},{"metadata":{"_uuid":"0652abd79bf560e8442452b91c1152ccadde12fc"},"cell_type":"markdown","source":"## Baseline model 建立\n\n我们希望快速建立一个 Baseline Model.\n\n**比XGBOOST更快--LightGBM介绍:**\n\nxgboost的出现，让数据民工们告别了传统的机器学习算法们：RF、GBM、SVM、LASSO........。现在，微软推出了一个新的boosting框架，想要挑战xgboost的江湖地位。 https://zhuanlan.zhihu.com/p/25308051"},{"metadata":{"_uuid":"87e3c5cb1dc1fae225c7be24de6035443c331367"},"cell_type":"markdown","source":"### 数据处理"},{"metadata":{"trusted":true,"_uuid":"a74565abfd0ea18e32b0937317f14688d73832ad"},"cell_type":"code","source":"zeroColumns = [c for c in train.columns if train[c].nunique() == 1]\nzeroColumns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3837356d40fd5864e0b782dc6129b90d2b7fc995"},"cell_type":"code","source":"cols_to_drop = ['socialEngagementType',\n 'device.browserSize',\n 'device.browserVersion',\n 'device.flashVersion',\n 'device.language',\n 'device.mobileDeviceBranding',\n 'device.mobileDeviceInfo',\n 'device.mobileDeviceMarketingName',\n 'device.mobileDeviceModel',\n 'device.mobileInputSelector',\n 'device.operatingSystemVersion',\n 'device.screenColors',\n 'device.screenResolution',\n 'geoNetwork.cityId',\n 'geoNetwork.latitude',\n 'geoNetwork.longitude',\n 'geoNetwork.networkLocation',\n 'totals.bounces',\n 'totals.newVisits',\n 'totals.visits',\n 'trafficSource.adwordsClickInfo.criteriaParameters',\n 'trafficSource.adwordsClickInfo.isVideoAd',\n 'trafficSource.isTrueDirect',\n 'sessionId']\n\ntrain = train.drop(cols_to_drop + [\"trafficSource.campaignCode\"], axis=1)\ntest = test.drop(cols_to_drop, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4494fd7c03162542a417f90271b5f5aac36bc131"},"cell_type":"code","source":"train.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4afc169963c077e44f1f4835004da5fee3d2b810"},"cell_type":"markdown","source":"### 划分数据集合\n\n现在让我们根据时间创建开发和验证拆分来构建模型。我们可以将最近两个月作为验证样本。"},{"metadata":{"trusted":false,"_uuid":"165df043a8e4f3dc25aafb077bbe69de6e554ca8"},"cell_type":"code","source":"# Impute 0 for missing target values\ntrain[\"totals.transactionRevenue\"].fillna(0, inplace=True)\ntrain_y = train[\"totals.transactionRevenue\"].values\ntrain_id = train[\"fullVisitorId\"].values\ntest_id = test[\"fullVisitorId\"].values\n\n# label encode the categorical variables and convert the numerical variables to float\ncat_cols = [\"channelGrouping\", \"device.browser\", \"device.deviceCategory\", \"device.operatingSystem\", \n \"geoNetwork.city\", \"geoNetwork.continent\", \"geoNetwork.country\", \"geoNetwork.metro\",\n \"geoNetwork.networkDomain\", \"geoNetwork.region\", \"geoNetwork.subContinent\", \"trafficSource.adContent\", \n \"trafficSource.adwordsClickInfo.adNetworkType\", \"trafficSource.adwordsClickInfo.gclId\", \n \"trafficSource.adwordsClickInfo.page\", \"trafficSource.adwordsClickInfo.slot\", \"trafficSource.campaign\",\n \"trafficSource.keyword\", \"trafficSource.medium\", \"trafficSource.referralPath\", \"trafficSource.source\"]\nnum_cols = [\"totals.hits\", \"totals.pageviews\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c8c8e9fb1a10b14a803b6d1beeb6bae720c13278"},"cell_type":"code","source":"from sklearn import preprocessing\n\nfor col in cat_cols:\n print(col)\n lbl = preprocessing.LabelEncoder()\n lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n train[col] = lbl.transform(list(train[col].values.astype('str')))\n test[col] = lbl.transform(list(test[col].values.astype('str')))\nfor col in num_cols:\n train[col] = train[col].astype(float)\n test[col] = test[col].astype(float)\n\n# Split the train dataset into development and valid based on time \ndev = train[train['date']<=datetime.date(2017,5,31)]\nval = train[train['date']>datetime.date(2017,5,31)]\ndev_y = np.log1p(dev[\"totals.transactionRevenue\"].values)\nval_y = np.log1p(val[\"totals.transactionRevenue\"].values)\n\ndev_X = dev[cat_cols + num_cols]\nval_X = val[cat_cols + num_cols]\ntest_X = test[cat_cols + num_cols]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e7182064675839e02791483f006b71622fcb7ffb"},"cell_type":"code","source":"[d.shape for d in [dev_X, val_X, test_X]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e1812e5a332b56f940ef56425764d10c7e26e97"},"cell_type":"markdown","source":"### 建立和训练模型"},{"metadata":{"code_folding":[1],"trusted":false,"_uuid":"a1026dbca786c3eface6d381b31aef64a07a653d"},"cell_type":"code","source":"import lightgbm as lgb\n\n# custom function to run light gbm model\ndef run_lgb(train_X, train_y, val_X, val_y, test_X):\n params = {\n \"objective\" : \"regression\",\n \"metric\" : \"rmse\",\n \"num_leaves\" : 30,\n \"learning_rate\" : 0.1,\n \"bagging_fraction\" : 0.7,\n \"feature_fraction\" : 0.5,\n \"bagging_frequency\" : 5,\n \"bagging_seed\" : 2018,\n \"verbosity\" : -1\n }\n \n lgtrain = lgb.Dataset(train_X, label=train_y)\n lgval = lgb.Dataset(val_X, label=val_y)\n model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100)\n \n pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n return pred_test_y, model\n\n# Training the model #\npred_test, model = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b24d7ea4b324c4311be52f4607097ad755509c2c"},"cell_type":"markdown","source":"### 模型预测和评估"},{"metadata":{"trusted":false,"_uuid":"66c14bc268ba0daf66de7d2802d19d21588f455f"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"10996d1718749533a960bf4a7a948812eac6e432"},"cell_type":"code","source":"sub_df.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9cca4b635c6b19891879a5726edfcb2776cdca52"},"cell_type":"markdown","source":"**Feature Importance:**\n\nNow let us have a look at the important features of the light gbm model."},{"metadata":{"trusted":false,"_uuid":"ea78814d90d8bed604f35fbb41f5fe6033ca4578"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"986593d8364c9e08bb0d9ffa8e5845c93d7d6c67"},"cell_type":"markdown","source":"### 封装\n\n把整个建模过程都封装成一段代码。"},{"metadata":{"trusted":false,"_uuid":"ab04a020320bf6d5b004a005205c907ab12f58ef"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nimport os\nimport json\nfrom pandas.io.json import json_normalize\n%matplotlib inline\n\n# 1. 读入数据\ndef load_df(csv_path='../input/train.csv', nrows=None):\n JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n \n df = pd.read_csv(csv_path, \n converters={column: json.loads for column in JSON_COLUMNS}, \n dtype={'fullVisitorId': 'str'}, \n nrows=nrows) # Important!!\n \n for column in JSON_COLUMNS:\n column_as_df = json_normalize(df[column])\n column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n return df\n\nprint(os.listdir(\"../input\"))\n\ntrain = load_df(nrows=100000)\ntest =load_df('../input/test.csv', nrows=100000)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"d5aa8e472b9de0a7f16fea3756940627e431c2e0"},"cell_type":"code","source":"# 2. 处理数据\ntrain[\"totals.transactionRevenue\"] = train[\"totals.transactionRevenue\"].astype('float')\n# test[\"totals.transactionRevenue\"] = test[\"totals.transactionRevenue\"].astype('float') # test 数据集合就是要预测该项\n\ntrain['date'] = train['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ntest['date'] = test['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\n\ncols_to_drop = ['socialEngagementType',\n 'device.browserSize',\n 'device.browserVersion',\n 'device.flashVersion',\n 'device.language',\n 'device.mobileDeviceBranding',\n 'device.mobileDeviceInfo',\n 'device.mobileDeviceMarketingName',\n 'device.mobileDeviceModel',\n 'device.mobileInputSelector',\n 'device.operatingSystemVersion',\n 'device.screenColors',\n 'device.screenResolution',\n 'geoNetwork.cityId',\n 'geoNetwork.latitude',\n 'geoNetwork.longitude',\n 'geoNetwork.networkLocation',\n 'totals.bounces',\n 'totals.newVisits',\n 'totals.visits',\n 'trafficSource.adwordsClickInfo.criteriaParameters',\n 'trafficSource.adwordsClickInfo.isVideoAd',\n 'trafficSource.isTrueDirect',\n 'sessionId']\n\ntrain = train.drop(cols_to_drop + [\"trafficSource.campaignCode\"], axis=1)\ntest = test.drop(cols_to_drop, axis=1)\n\n# Impute 0 for missing target values\ntrain[\"totals.transactionRevenue\"].fillna(0, inplace=True)\ntrain_y = train[\"totals.transactionRevenue\"].values\ntrain_id = train[\"fullVisitorId\"].values\ntest_id = test[\"fullVisitorId\"].values\n\n# label encode the categorical variables and convert the numerical variables to float\ncat_cols = [\"channelGrouping\", \"device.browser\", \"device.deviceCategory\", \"device.operatingSystem\", \n \"geoNetwork.city\", \"geoNetwork.continent\", \"geoNetwork.country\", \"geoNetwork.metro\",\n \"geoNetwork.networkDomain\", \"geoNetwork.region\", \"geoNetwork.subContinent\", \"trafficSource.adContent\", \n \"trafficSource.adwordsClickInfo.adNetworkType\", \"trafficSource.adwordsClickInfo.gclId\", \n \"trafficSource.adwordsClickInfo.page\", \"trafficSource.adwordsClickInfo.slot\", \"trafficSource.campaign\",\n \"trafficSource.keyword\", \"trafficSource.medium\", \"trafficSource.referralPath\", \"trafficSource.source\"]\nnum_cols = [\"totals.hits\", \"totals.pageviews\"]\n\nfrom sklearn import preprocessing\n\nfor col in cat_cols:\n print(col)\n lbl = preprocessing.LabelEncoder()\n lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n train[col] = lbl.transform(list(train[col].values.astype('str')))\n test[col] = lbl.transform(list(test[col].values.astype('str')))\nfor col in num_cols:\n train[col] = train[col].astype(float)\n test[col] = test[col].astype(float)\n\n# Split the train dataset into development and valid based on time \ndev = train[train['date']<=datetime.date(2017,5,31)]\nval = train[train['date']>datetime.date(2017,5,31)]\ndev_y = np.log1p(dev[\"totals.transactionRevenue\"].values)\nval_y = np.log1p(val[\"totals.transactionRevenue\"].values)\n\ndev_X = dev[cat_cols + num_cols]\nval_X = val[cat_cols + num_cols]\ntest_X = test[cat_cols + num_cols]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8171e08324aedcbabf60df7d9041dcac15d1b398"},"cell_type":"code","source":"# 3. 构建和训练模型\nimport lightgbm as lgb\n\n# custom function to run light gbm model\ndef run_lgb(train_X, train_y, val_X, val_y, test_X):\n params = {\n \"objective\" : \"regression\",\n \"metric\" : \"rmse\",\n \"num_leaves\" : 30,\n \"learning_rate\" : 0.1,\n \"bagging_fraction\" : 0.7,\n \"feature_fraction\" : 0.5,\n \"bagging_frequency\" : 5,\n \"bagging_seed\" : 2018,\n \"verbosity\" : -1\n }\n \n lgtrain = lgb.Dataset(train_X, label=train_y)\n lgval = lgb.Dataset(val_X, label=val_y)\n model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100)\n \n pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n return pred_test_y, model\n\n# Training the model #\npred_test, model = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n\n# 4. 模型预测和评估\nsub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c77110256ead12b477084aeaa9070a9527f0c8d9"},"cell_type":"markdown","source":"## XGBoost"},{"metadata":{"_uuid":"21104dc53585b8565577fe1708b672edf4d37021"},"cell_type":"markdown","source":"### 构建和训练模型\n"},{"metadata":{"trusted":false,"_uuid":"b59e4141593cfea9e898e64a817e8901b25fa573"},"cell_type":"code","source":"from xgboost import XGBRegressor\nmy_model = XGBRegressor(n_estimators=1000, learning_rate=0.05)\nmy_model.fit(dev_X, dev_y, early_stopping_rounds=5, \n eval_set=[(val_X, val_y)], verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"49f067029573441f4b7a82f9795bb70fc6747bab"},"cell_type":"code","source":"pred_test_y = my_model.predict(test_X)\nsub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"efec4f02a2283e2b08ed53332ce473cf1036919c"},"cell_type":"code","source":"pred_test","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9aa3f4c00170822a2dbf087f9428e58eedf83593"},"cell_type":"markdown","source":"# 参考资料"},{"metadata":{"_uuid":"7bd2b3c44ef3c9f5f1a1508a3d0759b73018cc17"},"cell_type":"markdown","source":"- Simple Exploration+Baseline - GA Customer Revenue\n https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue\n \n- 02-data analysis 相关变量分析\n https://www.kaggle.com/gongheayng/02-data-analysis"},{"metadata":{"trusted":false,"_uuid":"0af534764483be4203a395436d933c5d93025d4a"},"cell_type":"code","source":"import xgboost\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ae0e1f20dcb338df548bc27cf189c5848c304d2e"},"cell_type":"code","source":"[d.shape for d in [dev_X, val_X, test_X]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"fc7bf6a024dfb6041da5415c9dfc9eb190191e95"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":true,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"305px"},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":1}