{"cells":[{"metadata":{},"cell_type":"markdown","source":"**About the Model**\n\nIn this kernel, using [ML-Ensemble](http://ml-ensemble.com/), I developed a Super Learner model with Python to predict the customer revenue through the Google Analytics dataset.\nThis Super Learner model, consists on LinearRegression as a meta-model and LightGBM, XGBRegressor, and CatBoostRegressor as base-models.\nAlso, thanks to the pandas new updates (Version 1.0.0) specially in resolving issues with “named aggregation”, I was able to make a transparent and well organized codes for this project.\n\n**Solution Structure**\n\nBased on the train and test files and dates, we are going to predict returning customers behavior after a given time interval. Let's get started with calculating time interval.\nBased on the project specs:\n1. train_v2.csv file contains user transactions from August 1st, 2016 to April 30th, 2018 (637 days).\n2. test_v2.csv file contains user transactions from May 1st, 2018 to October 15th, 2018 (167 days).\n3. Final goal is to predict revenue for each of the visitors for the timeframe of December 1st, 2018 to January 31st, 2019 (61 days).\n4. The gap between #2 and #3 is 47 days.\n\nBased on the #2, #3, and #4, in this project we are going to predict revenue of returning visitores of 61 days interval (A) which visited the site in previous 167 days interval (B) and gap between the A and B is 47 days.\nWe can split our train data out based on this pattern and train our model to get ready for final predictions.\n\n\n**References**\n* ML_Ensemble documents, Link: http://ml-ensemble.com/\n* [How to Develop Super Learner Ensembles in Python](https://machinelearningmastery.com/super-learner-ensemble-in-python/), by [Jason Brownlee](https://machinelearningmastery.com/author/jasonb/) on December 11, 2019 in Python Machine Learning.\n* Data generation script: [https://www.kaggle.com/qnkhuat/make-data-ready](https://www.kaggle.com/qnkhuat/make-data-ready)\n* Stacking part is from this script: https://www.kaggle.com/ashishpatel26/updated-bayesian-lgbm-xgb-cat-fe-kfold-cv\n* Reference scripts for selecting important features, dataset splitting patterns, and others: https://www.kaggle.com/kostoglot/winning-solution, https://www.kaggle.com/augustmarvel/base-model-v2-user-level-solution\n\n\n\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Install the latest version of the pandas\n!pip install pandas --upgrade\nimport importlib\nimportlib.invalidate_caches()\nimport pandas as pd\nprint(pd.__version__)\n\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nfrom numpy import savetxt,inf\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport json\nfrom pandas.io.json import json_normalize\nimport time\nfrom ast import literal_eval\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.preprocessing import MinMaxScaler\nimport datetime\nfrom math import sqrt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.linear_model import LogisticRegression\nfrom mlens.ensemble import SuperLearner\nimport lightgbm as lgb\nfrom xgboost import XGBRegressor\nfrom xgboost import XGBClassifier\nfrom catboost import CatBoostRegressor\nfrom catboost import CatBoostClassifier\nimport os as os\nfrom os.path import join as pjoin\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\n%matplotlib inline\nprint(pd.__version__)\np = sns.color_palette()\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n\n### LOAD & PREPARE FILES\n  # Loading train_v2.csv and test_v2.csv files to the Pandas dataframe\n  # Flattening the JSON fields in the input files\n  # Casting the data types for fields\n  # Dealing with the missing values based on the columns data types\n  # Preprocessing the categorical fields. We will use the regression models and need to convert catecorical fields to numerical fields.\n###   \n\n# Dealing with the missing values based on the columns data types\n# We're using make-data-ready script at this link: https://www.kaggle.com/qnkhuat/make-data-ready\ndef to_na(df):\n    # Each type of columns that need to replace with the right na values\n    to_NA_cols = ['trafficSource_adContent','trafficSource_adwordsClickInfo.adNetworkType',\n                'trafficSource_adwordsClickInfo.slot','trafficSource_adwordsClickInfo.gclId',\n                'trafficSource_keyword','trafficSource_referralPath','customDimensions_value']\n\n    to_0_cols = ['totals_transactionRevenue','trafficSource_adwordsClickInfo.page','totals_sessionQualityDim','totals_bounces',\n                 'totals_timeOnSite','totals_newVisits','totals_pageviews','customDimensions_index','totals_transactions','totals_totalTransactionRevenue']\n\n    to_true_cols = ['trafficSource_adwordsClickInfo.isVideoAd']\n    to_false_cols = ['trafficSource_isTrueDirect']\n    \n    df[to_NA_cols] = df[to_NA_cols].fillna('NaN')\n    df[to_0_cols] = df[to_0_cols].fillna(0)\n    df[to_true_cols] = df[to_true_cols].fillna(True)\n    df[to_false_cols] = df[to_false_cols].fillna(False)\n    \n    return df\n    \ndef encode_date(df):\n    fld = pd.to_datetime(df['date'], infer_datetime_format=True)\n    \n    attrs = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',\n        'Is_month_end', 'Is_month_start', 'Is_quarter_end', \n        'Is_quarter_start', 'Is_year_end', 'Is_year_start','Hour']\n        \n    for attr in attrs:\n        df['Date_'+attr] = getattr(fld.dt,attr.lower())\n        \n    return df\n\ndef weird_na(df):\n    cols_to_replace = {\n        'socialEngagementType' : 'Not Socially Engaged',\n        'device_browserSize' : 'not available in demo dataset', \n        'device_flashVersion' : 'not available in demo dataset', \n        'device_browserVersion' : 'not available in demo dataset', \n        'device_language' : 'not available in demo dataset',\n        'device_mobileDeviceBranding' : 'not available in demo dataset',\n        'device_mobileDeviceInfo' : 'not available in demo dataset',\n        'device_mobileDeviceMarketingName' : 'not available in demo dataset',\n        'device_mobileDeviceModel' : 'not available in demo dataset',\n        'device_mobileInputSelector' : 'not available in demo dataset',\n        'device_operatingSystemVersion' : 'not available in demo dataset',\n        'device_screenColors' : 'not available in demo dataset',\n        'device_screenResolution' : 'not available in demo dataset',\n        'geoNetwork_city' : 'not available in demo dataset',\n        'geoNetwork_cityId' : 'not available in demo dataset',\n        'geoNetwork_latitude' : 'not available in demo dataset',\n        'geoNetwork_longitude' : 'not available in demo dataset',\n        'geoNetwork_metro' : ['not available in demo dataset', '(not set)'], \n        'geoNetwork_networkDomain' : ['unknown.unknown', '(not set)'], \n        'geoNetwork_networkLocation' : 'not available in demo dataset',\n        'geoNetwork_region' : 'not available in demo dataset',\n        'trafficSource_adwordsClickInfo.criteriaParameters' : 'not available in demo dataset',\n        'trafficSource_campaign' : '(not set)', \n        'trafficSource_keyword' : '(not provided)',\n        'trafficSource_medium' : '(none)',\n        'networkDomain': '(not set)', \n        'city': '(not set)', \n    }\n    df = df.replace(cols_to_replace,'NaN')\n    return df\n\ndef del_const(df):\n    const_col = []\n    for col in df.columns:\n        if df[col].nunique() == 1 and df[col].isnull().sum()==0 :\n            const_col.append(col)\n            \n    df.drop(const_col,axis=1,inplace=True)\n    return df, const_col\n    \ndef json_it(df):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column]) \n        column_as_df.columns = [f\"{column}_{subcolumn}\" for subcolumn in column_as_df.columns] \n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n            \n     # Normalize customDimensions\n    df['customDimensions']=df['customDimensions'].apply(literal_eval)\n    df['customDimensions']=df['customDimensions'].str[0]\n    df['customDimensions']=df['customDimensions'].apply(lambda x: {'index':np.NaN,'value':np.NaN} if pd.isnull(x) else x)\n\n    column_as_df = json_normalize(df['customDimensions'])\n    column_as_df.columns = [f\"customDimensions_{subcolumn}\" for subcolumn in column_as_df.columns]\n    df = df.drop('customDimensions', axis=1).merge(column_as_df, right_index=True, left_index=True)\n    \n    return df\n\ndef convert_it(df):\n    # Convert weird string to na\n    df = weird_na(df)\n    \n    # Convert columns to Na on it own type\n    df = to_na(df)\n    \n    # Create new columsn with data\n    df_train = encode_date(df)\n    \n    return df\n    \ndef fix_type(df):\n    try:\n        df.drop('trafficSource_campaignCode',axis=1,inplace=True)\n    except:\n        pass\n    # Fill na and rename the Revenue column\n    df['totals_transactionRevenue'] = df['totals_transactionRevenue'].fillna(0).astype(float)\n\n    to_int = ['totals_bounces','totals_newVisits','totals_pageviews',\n            'customDimensions_index','totals_hits','totals_sessionQualityDim',\n            'totals_visits','totals_timeOnSite','trafficSource_adwordsClickInfo.page',\n            'totals_transactions','totals_totalTransactionRevenue']\n    for col in to_int :\n        df[col] = df[col].astype(int)\n\n    return df\n\n# Start Loading & Preprocessing    \ndef load_it(csv_path,name):\n    CONST_COLLUMNS = ['socialEngagementType','device_browserSize',\n         'device_browserVersion','device_flashVersion',\n         'device_language','device_mobileDeviceBranding',\n         'device_mobileDeviceInfo','device_mobileDeviceMarketingName',\n         'device_mobileDeviceModel','device_mobileInputSelector',\n         'device_operatingSystemVersion','device_screenColors',\n         'device_screenResolution','geoNetwork_cityId',\n         'geoNetwork_latitude','geoNetwork_longitude',\n         'geoNetwork_networkLocation',\n         'trafficSource_adwordsClickInfo.criteriaParameters',]\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    dfs = pd.read_csv(csv_path, sep=',',\n                    parse_dates=['date'],\n                    converters={column: json.loads for column in JSON_COLUMNS}, \n                    dtype={'fullVisitorId': 'str'}, # Important!!\n                    chunksize = 200000)\n    \n    for idx,df in enumerate(dfs):\n        print(idx)\n        df.reset_index(drop = True,inplace = True)\n        df = json_it(df)\n        df = convert_it(df)\n        df.drop(CONST_COLLUMNS,axis=1,inplace=True)\n        # Heavy as hell this column\n        df.drop('hits',axis=1,inplace=True)\n        df = fix_type(df)\n        df.to_pickle(f'{name}_{idx}.pkl')\n        \n        del df\n        gc.collect()\n\n# Read the input files\ntrain_file = \"../input/ga-customer-revenue-prediction/train_v2.csv\"\ntest_file = \"../input/ga-customer-revenue-prediction/test_v2.csv\"\ndf_train_raw = load_it(train_file,'train')\ndf_test_raw = load_it(test_file,'test')\n\n# Main function to make a final train and test data frames\ndef load_data(data='train',n=2):\n    df = pd.DataFrame()\n    for i in range(n) :\n        if data=='train':\n            if i > 8 :\n                break\n            dfpart = pd.read_pickle(pjoin('../working',f'train_{i}.pkl'))\n        elif data=='test':\n            if i > 2 :\n                break\n            dfpart = pd.read_pickle(pjoin('../working',f'test_{i}.pkl'))\n        df = pd.concat([df,dfpart])\n        del dfpart    \n    return df\n\ndf_train = load_data(n=9)\ndf_test = load_data('test',n=4) \ndf = pd.concat([df_train,df_test])\n\n# Picking up a target dataset\ntest_date = datetime.datetime(2018, 5, 1)\ndf_test_set = df[df['date'] >=  test_date]\n\n### DATA VISUALIZATION\n  # Function to calculate correlation coefficient between two features in 2018\n###\ndef corr(x, y, **kwargs):\n    \n    # Calculate the value\n    coef = np.corrcoef(x, y)[0][1]\n    # Make the label\n    label = r'$\\rho$ = ' + str(round(coef, 2))\n    \n    # Add the label to the plot\n    ax = plt.gca()\n    ax.annotate(label, xy = (0.2, 0.95), size = 20, xycoords = ax.transAxes)\n    \n# Create a pair grid instance\nendyear_date = datetime.datetime(2018, 1, 1)\ngrid = sns.PairGrid(data= df[(df['date'] >= endyear_date) & (df['totals_transactionRevenue'] > 0)],\n                    vars = ['totals_hits', 'totals_timeOnSite', 'totals_transactionRevenue'], height = 4)\n\n# Map the plots to the locations\ngrid = grid.map_upper(plt.scatter, color = 'darkred')\ngrid = grid.map_upper(corr)\ngrid = grid.map_lower(sns.kdeplot, cmap = 'Reds')\ngrid = grid.map_diag(plt.hist, bins = 10, edgecolor =  'k', color = 'darkred')\nplt.show()\n\n### SPLIT The DATASET UP\n# The training set duration is 167 days \n# 45 days gab between training and test set\n# The test set statrts from 167 + 47 and last for 61 days\ntrain_range = pd.date_range(start='2016-08-01',end='2018-10-15', freq='275D')\ntrain_end = train_range.to_series().shift(periods=167, freq='d',axis= 0)\ntest_strart = train_range.to_series().shift(periods=214, freq='d',axis= 0)\ntrain_range = train_range.to_series()\n\n# Generate dataset\nimportant_cols = ['channelGrouping','date','fullVisitorId','visitId','visitNumber','visitStartTime','device_browser','device_deviceCategory',\n                  'device_operatingSystem', 'device_isMobile','geoNetwork_continent', 'geoNetwork_subContinent','geoNetwork_country','geoNetwork_region','geoNetwork_metro','geoNetwork_city','geoNetwork_networkDomain','totals_bounces',\n                  'totals_hits','totals_newVisits','totals_pageviews','totals_sessionQualityDim','totals_timeOnSite','totals_totalTransactionRevenue','totals_transactionRevenue',\n                  'totals_transactions','trafficSource_adContent','trafficSource_adwordsClickInfo.adNetworkType', 'trafficSource_adwordsClickInfo.gclId','trafficSource_adwordsClickInfo.isVideoAd',\n                  'trafficSource_adwordsClickInfo.page', 'trafficSource_adwordsClickInfo.slot','trafficSource_campaign','trafficSource_isTrueDirect','trafficSource_keyword','trafficSource_medium','trafficSource_referralPath','trafficSource_source']\n\ndf = df[important_cols]\ndef proc_data(df, i = 0):    \n    train_set = df[(df.date>=train_range.index[i]) & (df.date <=train_end.index[i])]\n    if i == 2:\n        test_set_ids = df[(df.date >= test_strart.index[i])].fullVisitorId.unique()\n    else:\n        test_set_ids = df[(df.date >= test_strart.index[i]) & (df.date < train_range.index[i + 1])].fullVisitorId.unique()\n    test_set_ids = pd.Series(test_set_ids)\n    train_returned =  pd.merge(train_set, test_set_ids.rename('fullVisitorId'), how='inner', on=['fullVisitorId'])\n    returned_set_ids = train_returned.fullVisitorId.unique()\n    returned_set_ids = pd.Series(returned_set_ids)\n    if i==2:\n        test_set = df[(df.date >= test_strart.index[i])]\n    else:\n        test_set = df[(df.date >= test_strart.index[i]) & (df.date < train_range.index[i + 1])]\n    test_returned = pd.merge(test_set, returned_set_ids.rename('fullVisitorId'), how='inner', on=['fullVisitorId'])\n    test_returned = test_returned.groupby(\"fullVisitorId\")['totals_transactionRevenue'].apply(lambda x: np.log(sum(x))).reset_index()\n    test_returned['target'] = test_returned['totals_transactionRevenue'].apply(lambda x: 0 if x == -inf else x)\n    test_returned = test_returned.drop(['totals_transactionRevenue'],axis = 1)\n    test_returned['retuningVisitor'] = 1\n    test_tmp_nonret = pd.merge(train_set, returned_set_ids.rename('fullVisitorId'), how='left', on=['fullVisitorId'], indicator=True).query('_merge==\"left_only\"')\n    test_tmp_ids = test_tmp_nonret.fullVisitorId.unique()\n    test_nonret = pd.DataFrame(data =test_tmp_ids, columns= ['fullVisitorId'])\n    test_nonret['target'] = 0\n    test_nonret['retuningVisitor'] = 0\n    test_target = pd.concat((test_returned,test_nonret), axis = 0)\n    test_target.fillna(0, inplace=True)\n\n  # Encoding numerical columns\n    numerical_ix = df[df.columns[df.columns != 'totals_transactionRevenue']].select_dtypes(include=['int64', 'float64']).columns\n\n    # Iterate and encode mumerical columns\n    for idx,column in enumerate(numerical_ix):    \n            le = MinMaxScaler()\n            train_set[column] = le.fit_transform(train_set[column].values.reshape(-1, 1))            \n\n    # Determine categorical features\n    categorical_ix = df[['channelGrouping','device_browser','device_deviceCategory','device_isMobile','device_operatingSystem','geoNetwork_continent','geoNetwork_subContinent', 'geoNetwork_country','geoNetwork_region',\n                        'geoNetwork_metro','geoNetwork_city','geoNetwork_networkDomain','trafficSource_adContent','trafficSource_adwordsClickInfo.adNetworkType','trafficSource_adwordsClickInfo.gclId','trafficSource_adwordsClickInfo.slot',\n                        'trafficSource_campaign','trafficSource_adwordsClickInfo.isVideoAd','trafficSource_isTrueDirect','trafficSource_keyword','trafficSource_medium','trafficSource_referralPath','trafficSource_source', 'date']]\n\n    # Label encoding categorical features\n    for idx,column in enumerate(categorical_ix):    \n            le = LabelEncoder()\n            train_set[column] = le.fit_transform(train_set[column].values.reshape(-1, 1))\n      \n    # Make a final data frame\n    train_set = train_set.groupby('fullVisitorId').agg(\n        channelGrouping = pd.NamedAgg(column = 'channelGrouping', aggfunc = max),\n        first_visit = pd.NamedAgg(column = 'date', aggfunc = min),\n        last_visit = pd.NamedAgg(column = 'date', aggfunc = max),\n        visit_interval = pd.NamedAgg(column = \"date\", aggfunc = lambda x: (max(x) - min(x))),\n        unique_dates = pd.NamedAgg(column = \"date\", aggfunc = lambda x: len(x.unique())),\n        max_visits = pd.NamedAgg(column = 'visitNumber', aggfunc = max),\n        visits_average = pd.NamedAgg(column =\"visitStartTime\", aggfunc= lambda x: (max(x) - min(x))),\n        browser = pd.NamedAgg(column = 'device_browser', aggfunc = max),\n        device_category = pd.NamedAgg(column = 'device_deviceCategory', aggfunc = max),\n        device_operatingSystem = pd.NamedAgg(column = 'device_operatingSystem', aggfunc = max),\n        device_isMobile = pd.NamedAgg(column = 'device_isMobile', aggfunc = max),\n        geoNetwork_continent = pd.NamedAgg(column = 'geoNetwork_continent', aggfunc = max),\n        geoNetwork_subContinent = pd.NamedAgg(column = 'geoNetwork_subContinent', aggfunc = max),    \n        geoNetwork_country = pd.NamedAgg(column = 'geoNetwork_country', aggfunc = max),\n        geoNetwork_region = pd.NamedAgg(column = 'geoNetwork_region', aggfunc = max),\n        geoNetwork_metro = pd.NamedAgg(column = 'geoNetwork_metro', aggfunc = max),\n        geoNetwork_city = pd.NamedAgg(column = 'geoNetwork_city', aggfunc = max),\n        geoNetwork_networkDomain = pd.NamedAgg(column = 'geoNetwork_networkDomain', aggfunc = max),\n        totals_bounces = pd.NamedAgg(column = 'totals_bounces', aggfunc = sum),\n        totals_hits_sum = pd.NamedAgg(column = 'totals_hits', aggfunc = sum),\n        totals_hits_mean = pd.NamedAgg(column = 'totals_hits', aggfunc = np.mean),\n        totals_hits_min = pd.NamedAgg(column = 'totals_hits', aggfunc = min),\n        totals_hits_max = pd.NamedAgg(column = 'totals_hits', aggfunc = max),\n        totals_hits_meidan = pd.NamedAgg(column = 'totals_hits', aggfunc = np.median),\n        totals_hits_std = pd.NamedAgg(column = 'totals_hits', aggfunc = np.std),\n        totals_pageviews_sum = pd.NamedAgg(column = 'totals_pageviews', aggfunc = sum),\n        totals_pageviews_mean = pd.NamedAgg(column = 'totals_pageviews', aggfunc = np.mean),\n        totals_pageviews_min = pd.NamedAgg(column = 'totals_pageviews', aggfunc = min),\n        totals_pageviews_max = pd.NamedAgg(column = 'totals_pageviews', aggfunc = max),\n        totals_pageviews_median = pd.NamedAgg(column = 'totals_pageviews', aggfunc = np.median),\n        totals_pageviews_std = pd.NamedAgg(column = 'totals_pageviews', aggfunc = np.std),\n        totals_sessionQualityDim_mean = pd.NamedAgg(column = 'totals_sessionQualityDim', aggfunc = np.mean),\n        totals_timeOnSite_sum = pd.NamedAgg(column = 'totals_timeOnSite', aggfunc = sum),\n        totals_totalTransactionRevenue_sum = pd.NamedAgg(column = 'totals_totalTransactionRevenue', aggfunc = sum),\n        totals_transactions_sum = pd.NamedAgg(column = 'totals_transactions', aggfunc = sum),\n        totals_transactionRevenue = pd.NamedAgg(column = 'totals_transactionRevenue', aggfunc = sum)\n    ).reset_index()\n    df_FT = (train_set.join(test_target.set_index('fullVisitorId'), on='fullVisitorId', how= 'inner'))\n    return df_FT\n\n# Preprocessing test dataset\ndf_test_set = df_test_set[important_cols]\n\n# Encoding numerical columns\nnumerical_ix = df_test_set[df_test_set.columns[df_test_set.columns != 'totals_transactionRevenue']].select_dtypes(include=['int64', 'float64']).columns\n\n# Iterate and encode mumerical columns\nfor idx,column in enumerate(numerical_ix):    \n        le = MinMaxScaler()\n        df_test_set[column] = le.fit_transform(df_test_set[column].values.reshape(-1, 1))\n\n# Determine categorical features\ncategorical_ix = df_test_set[['channelGrouping','device_browser','device_deviceCategory','device_isMobile','device_operatingSystem','geoNetwork_continent','geoNetwork_subContinent', 'geoNetwork_country','geoNetwork_region',\n                    'geoNetwork_metro','geoNetwork_city','geoNetwork_networkDomain','trafficSource_adContent','trafficSource_adwordsClickInfo.adNetworkType','trafficSource_adwordsClickInfo.gclId','trafficSource_adwordsClickInfo.slot',\n                    'trafficSource_campaign','trafficSource_adwordsClickInfo.isVideoAd','trafficSource_isTrueDirect','trafficSource_keyword','trafficSource_medium','trafficSource_referralPath','trafficSource_source', 'date']]\n\n# Label encoding categorical features\nfor idx,column in enumerate(categorical_ix):    \n        le = LabelEncoder()\n        df_test_set[column] = le.fit_transform(df_test_set[column].values.reshape(-1, 1))\n\n# Make a final dataframe\ndf_test_FN = df_test_set.groupby('fullVisitorId').agg(\n        channelGrouping = pd.NamedAgg(column = 'channelGrouping', aggfunc = max),\n        first_visit = pd.NamedAgg(column = 'date', aggfunc = min),\n        last_visit = pd.NamedAgg(column = 'date', aggfunc = max),\n        visit_interval = pd.NamedAgg(column = \"date\", aggfunc = lambda x: (max(x) - min(x))),\n        unique_dates = pd.NamedAgg(column = \"date\", aggfunc = lambda x: len(x.unique())),\n        max_visits = pd.NamedAgg(column = 'visitNumber', aggfunc = max),\n        visits_average = pd.NamedAgg(column =\"visitStartTime\", aggfunc= lambda x: (max(x) - min(x))),\n        browser = pd.NamedAgg(column = 'device_browser', aggfunc = max),\n        device_category = pd.NamedAgg(column = 'device_deviceCategory', aggfunc = max),\n        device_operatingSystem = pd.NamedAgg(column = 'device_operatingSystem', aggfunc = max),\n        device_isMobile = pd.NamedAgg(column = 'device_isMobile', aggfunc = max),\n        geoNetwork_continent = pd.NamedAgg(column = 'geoNetwork_continent', aggfunc = max),\n        geoNetwork_subContinent = pd.NamedAgg(column = 'geoNetwork_subContinent', aggfunc = max),    \n        geoNetwork_country = pd.NamedAgg(column = 'geoNetwork_country', aggfunc = max),\n        geoNetwork_region = pd.NamedAgg(column = 'geoNetwork_region', aggfunc = max),\n        geoNetwork_metro = pd.NamedAgg(column = 'geoNetwork_metro', aggfunc = max),\n        geoNetwork_city = pd.NamedAgg(column = 'geoNetwork_city', aggfunc = max),\n        geoNetwork_networkDomain = pd.NamedAgg(column = 'geoNetwork_networkDomain', aggfunc = max),\n        totals_bounces = pd.NamedAgg(column = 'totals_bounces', aggfunc = sum),\n        totals_hits_sum = pd.NamedAgg(column = 'totals_hits', aggfunc = sum),\n        totals_hits_mean = pd.NamedAgg(column = 'totals_hits', aggfunc = np.mean),\n        totals_hits_min = pd.NamedAgg(column = 'totals_hits', aggfunc = min),\n        totals_hits_max = pd.NamedAgg(column = 'totals_hits', aggfunc = max),\n        totals_hits_meidan = pd.NamedAgg(column = 'totals_hits', aggfunc = np.median),\n        totals_hits_std = pd.NamedAgg(column = 'totals_hits', aggfunc = np.std),\n        totals_pageviews_sum = pd.NamedAgg(column = 'totals_pageviews', aggfunc = sum),\n        totals_pageviews_mean = pd.NamedAgg(column = 'totals_pageviews', aggfunc = np.mean),\n        totals_pageviews_min = pd.NamedAgg(column = 'totals_pageviews', aggfunc = min),\n        totals_pageviews_max = pd.NamedAgg(column = 'totals_pageviews', aggfunc = max),\n        totals_pageviews_median = pd.NamedAgg(column = 'totals_pageviews', aggfunc = np.median),\n        totals_pageviews_std = pd.NamedAgg(column = 'totals_pageviews', aggfunc = np.std),\n        totals_sessionQualityDim_mean = pd.NamedAgg(column = 'totals_sessionQualityDim', aggfunc = np.mean),\n        totals_timeOnSite_sum = pd.NamedAgg(column = 'totals_timeOnSite', aggfunc = sum),\n        totals_totalTransactionRevenue_sum = pd.NamedAgg(column = 'totals_totalTransactionRevenue', aggfunc = sum),\n        totals_transactions_sum = pd.NamedAgg(column = 'totals_transactions', aggfunc = sum),\n        totals_transactionRevenue = pd.NamedAgg(column = 'totals_transactionRevenue', aggfunc = sum)\n    ).reset_index() \n\ndf_train_0 = proc_data(df, 0)\ndf_train_1 = proc_data(df, 1)\ndf_train_2 = proc_data(df, 2)\ntrain_set = pd.concat((df_train_0,df_train_1,df_train_2,df_test_FN), axis=0)\ntrain = train_set.loc[~pd.isnull(train_set.retuningVisitor)]\ntest = train_set.loc[pd.isnull(train_set.retuningVisitor)]\ntrain.fillna(0, inplace=True)\ntest.fillna(0, inplace=True)\ntrain_ret = train[train['retuningVisitor'] == 1]\n\n# Free memory\ndel df_train_0,df_train_1,df_train_2,train_set\ngc.collect()\n\n# Super learner for regression using the mlens library\n# This Super Learner algorithm, consists on LinearRegression as a meta-model and LightGBM, XGBRegressor, and CatBoostRegressor as base-models \nregression_params={'learning_rate': 0.01,\n        'objective':'regression',\n        'metric':'rmse',\n        'num_leaves': 32,\n        'verbose': 1,\n        'bagging_fraction': 0.9,\n        'feature_fraction': 0.9,\n        \"random_state\":42,\n        'max_depth': 5,        \n        \"verbosity\" : -1,\n        \"bagging_frequency\" : 5,\n        'lambda_l2': 0.5,\n        'lambda_l1': 0.5,\n        'min_child_samples': 36        \n       }\nxgb_regression_params = {\n        'objective': 'reg:squarederror',\n        'booster': 'gbtree',\n        'learning_rate': 0.02,      \n        'gamma' : 1.45,\n        'alpha': 0.0,\n        'lambda': 0.0,\n        'subsample': 0.67,\n        'colsample_bytree': 0.054,\n        'colsample_bylevel': 0.50,\n        'n_jobs': -1,        \n        'importance_type': 'total_gain'\n    }\n\ncat_regression_param = {\n    'learning_rate' :0.03,\n    'eval_metric' :'RMSE',\n    'od_type' :'Iter',\n    'metric_period' : 50,\n    'od_wait' : 20,        \n    'bagging_temperature' : 0.2\n    \n}\n\n# Super learner for binary classification using the mlens library\n# This Super Learner algorithm, consists on LogisticRegression as a meta-model and LightGBM, XGBRegressor, and CatBoostclassifier as base-models.\n \nclassification_params={'learning_rate': 0.01,\n        'objective':'binary',\n        'metric':'binary_logloss',\n        'num_leaves': 32,\n        'verbose': 1,\n        'bagging_fraction': 0.9,\n        'feature_fraction': 0.8,\n        \"random_state\":42,\n        'max_depth': 5,        \n        \"verbosity\" : -1,\n        \"bagging_frequency\" : 5,\n        'lambda_l2': 0.5,\n        'lambda_l1': 0.5,\n        'min_child_samples': 36        \n       }\nxgb_classification_params = {\n        'objective': 'binary:hinge',\n        'booster': 'gbtree',\n        'learning_rate': 0.02,      \n        'gamma' : 1.45,\n        'alpha': 0.0,\n        'lambda': 0.0,\n        'subsample': 0.67,\n        'colsample_bytree': 0.054,\n        'colsample_bylevel': 0.50,\n        'n_jobs': -1,        \n        'importance_type': 'total_gain'\n    }\n\n# Create a list of base-models\ndef get_models(i,j):\n    if j == 0:\n            reg = lgb.LGBMClassifier(**classification_params, n_estimators = 368 + i, bagging_seed = 42 + i,feature_fraction_seed = 13 + i)\n            xgb = XGBClassifier(**xgb_classification_params, n_estimators = 350 + i, random_state = 446 + i, max_depth = 12 + i, min_child_weight = 47 + i )\n            meta_model = LogisticRegression(solver='liblinear')\n    else:\n        reg = lgb.LGBMRegressor(**regression_params, n_estimators = 1100 + i, bagging_seed = 13 + i,feature_fraction_seed = 42 + i)\n        xgb = XGBRegressor(**xgb_regression_params, n_estimators = 1000 + i, random_state = 446 + i, max_depth = 12 + i, min_child_weight = 47 + i )\n        cat = CatBoostRegressor(**cat_regression_param, iterations = 1000 + i, random_seed = 32 + i, depth = 2 + i)\n        meta_model = LinearRegression()\n          \n    models = list()\n    models.append(meta_model)\n    models.append(reg)\n    models.append(xgb)\n    if j == 1:\n        models.append(cat)       \n    return models\n \n# Cost function for base regressor models\ndef rmse(yreal, yhat):\n    return sqrt(mean_squared_error(yreal, yhat))\n\n \n# Create the super learner\ndef get_super_learner(X,i,j):    \n    if j == 0:\n        ensemble = SuperLearner(folds=2, shuffle=True, sample_size=len(X))\n    else:\n        ensemble = SuperLearner(scorer=rmse, folds=10, shuffle=True, sample_size=len(X))\n        \n    # Add base models\n    models = get_models(i,j)    \n    ensemble.add(models)\n    # Add the meta model\n    if j == 0:\n        ensemble.add_meta(LogisticRegression(solver='liblinear'))\n    else:\n        ensemble.add_meta(LinearRegression())    \n        \n    return ensemble\n\n\n\nX_all, y_all = train[train.columns[train.columns != 'retuningVisitor']],train['retuningVisitor']\nX_ret, y_ret = train_ret[train_ret.columns[train_ret.columns != 'target']],train_ret['target']\nX_test_all = test[test.columns[test.columns != 'retuningVisitor']]\nX_test = test[test.columns[test.columns != 'target']]\nX_all.fillna(0, inplace=True)\nX_ret.fillna(0, inplace=True)\nX_test_all.fillna(0, inplace=True)\nX_test.fillna(0, inplace=True)\ny_all.fillna(0, inplace=True)\ny_ret.fillna(0, inplace=True)\n\ndef predict(X = 0,y = 0, Xtest = 0, i = 0, j = 0):\n    \n    # Split data to train, test samples \n    X, X_val, y, y_val = train_test_split(X, y, test_size=0.40)\n    print('Train', X.shape, y.shape, 'Test', X_val.shape, y_val.shape)\n    X=X.to_numpy()\n    X_val = X_val.to_numpy()\n    y=y.to_numpy()\n    y_val = y_val.to_numpy()\n    Xtest = Xtest.to_numpy()\n\n    # Create the super learner\n    ensemble = get_super_learner(X,i,j)\n\n    # Fit the super learner\n    print('__' *40)\n    ensemble.fit(X, y)\n\n    # Summarize base learners\n    print(ensemble.data)\n    # Evaluate meta model    \n    if j == 0:\n        yhat_0 = ensemble.predict_proba(X_val)\n        yhat = ensemble.predict_proba(Xtest)\n    else:\n        yhat_0 = ensemble.predict(X_val)\n        print('Super Learner: RMSE %.3f' % (rmse(y_val, yhat_0)))\n        yhat = ensemble.predict(Xtest)\n\n    return yhat\n\nyhat_1 = np.zeros(y_all.shape[0])\nyhat_2 = np.zeros(y_ret.shape[0])\nyhat_all = np.zeros(X_test.shape[0])\nyhat_FNL = np.zeros(X_test.shape[0])\n\nfor i in range(1, 11):\n    print('***Set ' + str(i))\n    # Will the customer return back to the store?(Classification)\n    # Overall method is same as this notebook: https://www.kaggle.com/kostoglot/winning-solution\n    yhat_1 = predict(X_all,y_all,X_test_all,i, j = 0)\n    # How much returned customer will spend? (Reggression)\n    yhat_2 = predict(X_ret,y_ret,X_test,i, j = 1)\n    yhat_all = yhat_all + (yhat_1 * yhat_2)\nyhat_FNL = yhat_all/10\nyhat_FNL[yhat_FNL < 0] = 0\n\n# Preparing submissin list\ntest_ID= test.fullVisitorId\n# Final predicts\n#Final predictions to csv\nsub_df = pd.DataFrame(test_ID) \nsub_df[\"PredictedLogRevenue\"] = yhat_FNL\nsub_df.to_csv(\"stacked_result.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}