{"cells":[{"metadata":{"_uuid":"da257aad9b04cbebc1e67e6465414e7e379fc1e9"},"cell_type":"markdown","source":"# Introduction"},{"metadata":{"_uuid":"a04a68f5edf3bcd4f18524049ac02d185fdf0e76"},"cell_type":"markdown","source":"### This kernel is mainly made up of three parts:\n* [**1. Data loading**](#Data loading)\n* [**2. Data preprocessing**](#Data preprocessing)\n* [**3. Model building**](#Model building)\n\n### Main of the kernel:\n* Chunk whole of data set by the period as such a structure: \n* [210 days of training period, 45 days of gap period, 2 months of traget perod]. \n* Aggregating data from the training period, ignoring the gap period, getting the target from the traget period. \n* The valiation set is set to Dec-Jan which is the same monthly period as the target peroid of the test set.\n\n### Summary:\n In this competition, the data set is so unbalanced that it's hard to say whether our solution can beat all-zeros. After doing some basic EDA, there are some conclusions are for sure: \n\n1. if a customer will pay, the transaction will be highly likely happened at the first month, and no longer than two months after the customer shows up in first time. \n2. the minimum of transaction revenue is no less than 1E+07.\n---\n* For the first one, our test set has a 1.5 months' gap between the traget period which means our taget is divided into two groups: the first is the one who has already spent no less than 45 days on thinking whether to pay. The second is the one who has payed for partial services and is going to pay for additional services. To the first group, the customers are terrific unlikely to pay. To the second one, the customers are likely to pay much the same as they payed before. Under those conditions, my prediction of the number of people to pay is 200 or so.\n* For the second one, as we see, the prediction of our model is full of numbers less than 1E+07. But you'll get a worse score if you set those numbers to zero. Our model keeps betting wisely on minimize RMSE but the result keeps leaving away from the real numbers. \n\n### random thoughts:\n* To set a user-defined objective function, which gives a high penalty once the floor level is breached, will be good for avoiding small values.\n* Time features should be under the first priority.\n* To the second group people, if it's possible to specify them by clustering.\n* if the customers wil return after a full year of services are expired?\n* the data set is lack of some important features such as page views of user's website. To the low volume users, why do they pay the bill for advance services if the free account already meets all the needs?"},{"metadata":{"trusted":true,"_uuid":"dddabb983cddc6cc669dfede6c3cb434a6d269cf"},"cell_type":"markdown","source":"* Data are generated from this script : https://www.kaggle.com/qnkhuat/make-data-ready \n* Stacking part is from this script: https://www.kaggle.com/ashishpatel26/updated-bayesian-lgbm-xgb-cat-fe-kfold-cv"},{"metadata":{"_uuid":"c980fc990df8ec129a8143b9358cfc256c04eca9"},"cell_type":"markdown","source":"## Data loading"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom datetime import datetime\n\nimport os\nfrom os.path import join as pjoin\n\ndata_root = '../input/make-data-ready'\nprint(os.listdir(data_root))\n\npd.set_option('display.max_rows',200)\n\nfrom sklearn.preprocessing import LabelEncoder\n\nfrom pprint import pprint\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3cc9e31f9c51f1fd66593d2fd0cb3963e880f12"},"cell_type":"code","source":"def load_data(data='train',n=2):\n df = pd.DataFrame()\n for i in range(n) :\n if data=='train':\n if i > 8 :\n break\n dfpart = pd.read_pickle(pjoin(data_root,f'train_{i}.pkl'))\n elif data=='test':\n if i > 2 :\n break\n dfpart = pd.read_pickle(pjoin(data_root,f'test_{i}.pkl'))\n df = pd.concat([df,dfpart])\n del dfpart\n return df\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"711b2ff1d23636474cbf8cbd63ed852d4853499d","scrolled":false},"cell_type":"code","source":"df_train = load_data(n=9)\ndf_test = load_data('test',n=4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3609ea5ecb4ea0a721136c3fc804b48877df40fd"},"cell_type":"markdown","source":"## Data preprocessing"},{"metadata":{"trusted":true,"_uuid":"3d05fcea733cf0d106981b1012bc9bec45535d75"},"cell_type":"code","source":"df = pd.concat([df_train, df_test])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f385d9764874cc85395f13e16ceecabb510a828"},"cell_type":"markdown","source":"### Drop some features and items"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"68df28767f05caa0636a05ed247205f989d86e91"},"cell_type":"code","source":"col_drop = ['Date_Year', 'Date_Month', 'Date_Week','Date_Hour','device_isMobile','device_deviceCategory',\n 'Date_Day', 'Date_Dayofweek', 'Date_Dayofyear', 'Date_Is_month_end',\n 'Date_Is_month_start', 'Date_Is_quarter_end', 'Date_Is_quarter_start',\n 'Date_Is_year_end', 'Date_Is_year_start','totals_visits',\n 'date','visitId','totals_totalTransactionRevenue','geoNetwork_city','geoNetwork_continent',\n 'geoNetwork_metro','geoNetwork_networkDomain',\n'geoNetwork_region','geoNetwork_subContinent','trafficSource_adContent',\n 'trafficSource_adwordsClickInfo.adNetworkType','trafficSource_adwordsClickInfo.gclId',\n'trafficSource_adwordsClickInfo.slot','trafficSource_campaign',\n 'trafficSource_keyword','trafficSource_referralPath','trafficSource_medium',\n 'customDimensions_value','customDimensions_index','trafficSource_source',\n 'totals_bounces','visitNumber','totals_newVisits']\ndf.drop(col_drop, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"3d908877ca829b6379b2812a2ca8fb4f6f95724f"},"cell_type":"code","source":"country_drop=df.groupby('geoNetwork_country')['totals_transactions'].sum()[df.groupby('geoNetwork_country')['totals_transactions'].sum().sort_values()<4].index.tolist()\ndf.loc[df[df.geoNetwork_country.isin(country_drop)].index,'geoNetwork_country'] = 'NaN'\n\ndf.loc[df[~df.device_browser.isin(['Edge', 'Internet Explorer', 'Firefox', 'Safari', 'Chrome'])].index,'device_browser'] = 'NaN'\ndf.loc[df[~df.device_operatingSystem.isin(['Android', 'iOS', 'Linux', 'Chrome OS', 'Windows', 'Macintosh'])].index,'device_operatingSystem'] = 'NaN'\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c893481ff845810da8afa0b9c74f421d5040737"},"cell_type":"markdown","source":"### Label encoding"},{"metadata":{"trusted":true,"_uuid":"d57562be04b714d000cc04b9f228af5415c2fb3a"},"cell_type":"code","source":"col_lb = ['channelGrouping','device_browser','device_operatingSystem', 'geoNetwork_country',\n 'trafficSource_adwordsClickInfo.isVideoAd','trafficSource_isTrueDirect']\nfor col in col_lb:\n lb = LabelEncoder()\n df[col]=lb.fit_transform(df[col])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"daa3c2dd554b4892cc8f797b4151f9c673238a5f"},"cell_type":"markdown","source":"### Features to user level\nThere is also a feature called time_diff, which is directly coded in generating part. And this time- relative feature really works well"},{"metadata":{"trusted":true,"_uuid":"08422247faa12dfb05b56eb5c968fc17b61dcfdf"},"cell_type":"code","source":"to_median = ['channelGrouping','device_browser','device_operatingSystem','geoNetwork_country','trafficSource_adwordsClickInfo.isVideoAd','trafficSource_isTrueDirect','trafficSource_adwordsClickInfo.page']\nto_sum =['totals_hits','totals_pageviews','totals_timeOnSite','totals_transactionRevenue', 'totals_transactions']\nto_mean =['totals_hits','totals_pageviews','totals_sessionQualityDim']\nto_std = ['totals_hits','totals_pageviews']\nto_time = 'visitStartTime'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73bfce7889aee0d47dfa4329f5467b886d45e501"},"cell_type":"markdown","source":"### Time period\n* the training set has a 45 days gap to its target set that is same as the test set \n* the training set has almost the same duration as the test set\n* the valiation set is set to Dec-Jan which is the same monthly period as the target peroid of the test set"},{"metadata":{"trusted":true,"_uuid":"4f462ee9657626056b3d37ec053f96e3759dda46"},"cell_type":"code","source":"target_period = pd.date_range(start='2016-08-01',end='2018-12-01', freq='2MS')\ntrain_period = target_period.to_series().shift(periods=-210, freq='d',axis= 0)\ntime_to = train_period[train_period.index>np.datetime64('2016-08-01')].astype('int')//10**9\ntime_end = target_period.to_series().shift(periods=-45, freq='d',axis= 0)[4:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a0d781e54279f9ed2035b6f83ea0597e01d4139"},"cell_type":"markdown","source":"### Test data"},{"metadata":{"trusted":true,"_uuid":"b559aa5ad7ba5edd36f787b7d6877e5595960507","scrolled":true},"cell_type":"code","source":" user_x = df.iloc[df_train.shape[0]:,:]\n test_x = pd.concat([user_x.groupby('fullVisitorId')[to_median].median().add_suffix('_median'),\n user_x.groupby('fullVisitorId')['visitStartTime'].agg(['first','last']).add_suffix('_time').sub(time_to.values[-1]).abs(),\n user_x.groupby('fullVisitorId')['visitStartTime'].apply(lambda x: x.max() -x.min()).rename('time_diff'),\n user_x.groupby('fullVisitorId')[to_sum].sum().add_suffix('_sum'),\n user_x.groupby('fullVisitorId')[to_mean].mean().add_suffix('_mean'),\n user_x.groupby('fullVisitorId')[to_std].std(ddof=0).add_suffix('_std')],axis=1).reset_index()\n \n test_ID= test_x.fullVisitorId\n test_x = test_x.drop(['fullVisitorId'], axis=1,errors='ignore')\n test_x = test_x.astype('int')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa3d17404b1de64779d970d0649feb95fbc5df82"},"cell_type":"markdown","source":"### Valiation data"},{"metadata":{"trusted":true,"_uuid":"6af113d251ba3348c1c50274c684aed1ba95f022","scrolled":true},"cell_type":"code","source":" i=4\n user_x = df[(df.visitStartTime>=(time_to.index.astype('int')//10**9)[i]) & (df.visitStartTime<(time_end.index.astype('int')//10**9)[i])]\n user_y = df[(df.visitStartTime>=time_to.values[i]) & (df.visitStartTime=(time_to.index.astype('int')//10**9)[i]) & (df.visitStartTime<(time_end.index.astype('int')//10**9)[i])]\n user_y = df[(df.visitStartTime>=time_to.values[i]) & (df.visitStartTime