{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Intro / motivation\n\nThis kernel is basically an refactor of [my-first-kernel](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) by [chauhuynh](https://www.kaggle.com/chauhuynh).  \nI tried to made the code more understandable, removed some redundancy and add some (hopefully somehow useful) comments along the code. The main difference imo is that I could not use all of the features, specifically these two:\n    1. new_hist_purchase_date_max\n    2. new_hist_purchase_date_min\n\nReason is because at some point some `NaT` showed up and I wasn't able to figure it out why. Therefore I just did not used them.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport datetime\nimport gc\nfrom pathlib import Path\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# for dirname, _, filenames in os.walk('../'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n# copy-paste\nimport numpy as np\nimport pandas as pd\nimport datetime\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import mean_squared_error\nimport warnings\nwarnings.filterwarnings('ignore')\nnp.random.seed(4590)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nPATH = Path('../input/elo-merchant-category-recommendation')\ndf_train = pd.read_csv(PATH/'train.csv', parse_dates=['first_active_month']);\ndf_test = pd.read_csv(PATH/'test.csv', parse_dates=['first_active_month']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_hist_trans = pd.read_csv(PATH/'historical_transactions.csv', parse_dates=['purchase_date']);\ndf_new_merch_trans = pd.read_csv(PATH/'new_merchant_transactions.csv', parse_dates=['purchase_date']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/fabiendaniel/elo-world\ndef reduce_mem_usage(df, verbose=True):\n    prefixes = ['int', 'float']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = str(df[col].dtype)\n        if not col_type.startswith('int') and not col_type.startswith('float'):\n            continue\n        c_min = df[col].min()\n        c_max = df[col].max()\n        if col_type.startswith('int'):\n            if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                df[col] = df[col].astype(np.int8)\n            elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                df[col] = df[col].astype(np.int16)\n            elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                df[col] = df[col].astype(np.int32)\n            elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                df[col] = df[col].astype(np.int64)  \n        elif col_type.startswith('float'):\n            if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                df[col] = df[col].astype(np.float16)\n            elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                df[col] = df[col].astype(np.float32)\n            else:\n                df[col] = df[col].astype(np.float64)    \n    if verbose:\n        end_mem = df.memory_usage().sum() / 1024**2\n        print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\n# Same logic, but clearer (I hope so)\ndef fillna_mode(df, cols, pipeline=False):\n    for c in cols:\n        df[c].fillna(df[c].mode()[0], inplace=True)\n    return df if pipeline else None\n\n\ndef get_nan_col_names(df: pd.DataFrame, df_name='<not named>'):\n    total = df.shape[0]\n    missing_cols = []\n    for c in df.columns:\n        quo = (total - pd.notna(df[c]).sum())/total\n        if quo != 0:\n            missing_cols.append(c)\n    print(df_name, 'MISSING COLS:', missing_cols)\n    return missing_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduce_mem_usage(df_train);\nreduce_mem_usage(df_test);\nreduce_mem_usage(df_hist_trans);\nreduce_mem_usage(df_new_merch_trans);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('authorized_flag: ', df_hist_trans.authorized_flag.unique())\n# print('category_1: ', df_hist_trans.category_1.unique())\n# print('category_2: ', df_hist_trans.category_2.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_month_diff(df, pipeline=False):\n    # still not sure why is useful. I mean, it is a constant changing feature.\n    #https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73244\n    df['month_diff'] = ((datetime.datetime.today() - df['purchase_date']).dt.days) // 30\n    df['month_diff'] += df['month_lag']\n    return df if pipeline else None\n\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\ndef pre_process_trans(df, date_col, date_formated=False, add_month=True):\n    fillna_mode(df, cols=get_nan_col_names(df))\n    df['authorized_flag'] = df['authorized_flag'].map({'Y':1, 'N':0}).astype(np.int8)\n    df['category_1'] = df['category_1'].map({'Y': 1, 'N': 0}).astype(np.int8)    \n    df['category_2'] = df['category_2'].astype(np.int8)\n\n    if not date_formated:\n        df[date_col] = pd.to_datetime(df[date_col])\n    # Separate each date component so that the model\n    # can use each of them independently. This way\n    # model will be feed with more potentially\n    # useful information.\n    df['year'] = df[date_col].dt.year\n    df['weekofyear'] = df[date_col].dt.weekofyear\n    df['month'] = df[date_col].dt.month\n    df['dayofweek'] = df[date_col].dt.dayofweek\n    df['weekend'] = (df[date_col].dt.weekday >= 5).astype(np.int8)\n    df['hour'] = df[date_col].dt.hour\n    if add_month:\n        add_month_diff(df)\n\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\ndef get_new_columns(name,aggs):\n    return [name + '_' + k + '_' + agg for k in aggs.keys() for agg in aggs[k]]\n\n\n# Refactor of https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\ndef custom_group_by(df: pd.DataFrame, df_name: str, agg_by: dict):\n    for col in ['category_2','category_3']:\n        df[col+'_mean'] = df.groupby([col])['purchase_amount'].transform('mean')\n        aggs[col+'_mean'] = ['mean']\n    new_columns = get_new_columns(df_name, agg_by)\n    \n    df_group = df.groupby('card_id').agg(agg_by)\n    df_group.columns = new_columns\n    df_group.reset_index(drop=False, inplace=True)\n    df_group[df_name + '_purchase_date_diff'] = (\n        df_group[df_name + '_purchase_date_max']\n            - df_group[df_name + '_purchase_date_min']\n        ).dt.days\n    df_group[df_name + '_purchase_date_average'] = (\n        df_group[df_name + '_purchase_date_diff']\n            / df_group[df_name + '_card_id_size']\n        )\n#     df_group[df_name + '_purchase_date_uptonow'] = (\n#         datetime.datetime.today()\n#         - df_group[df_name + '_purchase_date_max']\n#         ).dt.days\n\n    return df_group","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature engeneering\nFeatures are taken from:\n1. [my-first-kernel][mfk]\n\n[mfk]: https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"aggs = {}\n\n# Count number of unique values at each column.\n# Maybe this can be useful to calculate the\n# probability an value is sampled\nfor col in ['month','hour','weekofyear','dayofweek','year','subsector_id','merchant_id','merchant_category_id']:\n    aggs[col] = ('nunique',)\n\n\naggs['purchase_amount'] = ('sum','max','min','mean','var')\n\n\n# Maybe there are patterns regarding to loyal consumers installments\naggs['installments'] = ('sum','max','min','mean','var')\n\n# purchase range\naggs['purchase_date'] = ('max','min')\n\naggs['month_lag'] = ('max','min','mean','var')\n# aggs['month_diff'] = ('mean',)\n\n# How many transactions:\n# - were on the weekend, and the percentage\n# - has category_1 as 1 (binary feature), and the percentage\n# - were authorized, and its percentage\n\naggs['weekend'] = ('sum', 'mean')\naggs['category_1'] = ('sum', 'mean')\n\n# How many times and how ofter a card had a\n# transaction accepted?\naggs['authorized_flag'] = ('sum', 'mean')\n\n# How many transactions each card did?\naggs['card_id'] = ('size', )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# _old_train = df_train.copy()\n# _odl_test = df_test.copy()\n_old_train = pd.read_csv(PATH/'train.csv', parse_dates=['first_active_month']);\n_odl_test = pd.read_csv(PATH/'test.csv', parse_dates=['first_active_month']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ngps = []\nfor name, df in [('hist', df_hist_trans), ('new_hist', df_new_merch_trans)]:    \n    pre_process_trans(df, date_col='purchase_date', add_month=False)\n    df_group = custom_group_by(df, name, aggs.copy())\n    gps.append(df_group)\n    df_train = df_train.merge(df_group, on='card_id', how='left')\n    df_test = df_test.merge(df_group, on='card_id', how='left')\n#     del df_group; gc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# del df_hist_trans; print(gc.collect())\n# del df_new_merch_trans;print(gc.collect())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prob = [\n    'hist_purchase_date_max',\n    'hist_purchase_date_min',\n    'new_hist_purchase_date_max',\n    'new_hist_purchase_date_min',\n]\n\ndef post_process_df(df):\n    global prob\n    \n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    # \"Feature-spliting\" just like was performed for the date columnn\n    # in pre_process_trans\n    df['fam_dayofweek'] = df['first_active_month'].dt.dayofweek\n    df['fam_weekofyear'] = df['first_active_month'].dt.weekofyear\n    df['fam_month'] = df['first_active_month'].dt.month\n    df['fam_elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n    \n    df['hist_first_buy'] = (df['hist_purchase_date_min'] - df['first_active_month']).dt.days\n    df['new_hist_first_buy'] = (df['new_hist_purchase_date_min'] - df['first_active_month']).dt.days\n\n    df[prob[:2]] = df[prob[:2]].astype(np.int64) * 1e-9\n\n    \n    df['transactions_total'] = df['new_hist_card_id_size']+df['hist_card_id_size']\n    df['purchase_amount_total'] = df['new_hist_purchase_amount_sum']+df['hist_purchase_amount_sum']\n\n    for f in ['feature_1','feature_2','feature_3']:\n        order_label = df_train.groupby([f])['outliers'].mean()\n        df_train[f] = df_train[f].map(order_label)\n        df_test[f] = df_test[f].map(order_label)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_train.hist_purchase_date_max.notna().sum())\nprint(df_train.hist_purchase_date_min.notna().sum())\nprint(df_train.new_hist_purchase_date_max.notna().sum())\nprint(df_train.new_hist_purchase_date_min.notna().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['outliers'] = (df_train.target < -30).astype(np.int8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"post_process_df(df_train)\npost_process_df(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.drop(['new_hist_purchase_date_max', 'new_hist_purchase_date_min'], axis=1, inplace=True)\ndf_train.drop(['new_hist_purchase_date_max', 'new_hist_purchase_date_min'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_columns = [c for c in df_train.columns if c not in ['card_id', 'first_active_month', 'outliers', 'target']]\ntarget = df_train['target']\ndel df_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# copy-paste\n%time\nparam = {'num_leaves': 31,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.01,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1,\n         \"nthread\": 4,\n         \"random_state\": 4590}\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=4590)\noof = np.zeros(len(df_train))\npredictions = np.zeros(len(df_test))\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(df_train,df_train['outliers'].values)):\n    print(\"fold {}\".format(fold_))\n    trn_data = lgb.Dataset(df_train.iloc[trn_idx][df_train_columns], label=target.iloc[trn_idx])#, categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(df_train.iloc[val_idx][df_train_columns], label=target.iloc[val_idx])#, categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 100)\n    oof[val_idx] = clf.predict(df_train.iloc[val_idx][df_train_columns], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"Feature\"] = df_train_columns\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(df_test[df_train_columns], num_iteration=clf.best_iteration) / folds.n_splits\n\nnp.sqrt(mean_squared_error(oof, target))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions\nsub_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Some comments\n\nI still don't understanf what is the point of the widespread used feature `new_hist_purchase_date_uptonow`. Interestingly, when compared to the original kernel, this feature seems to have **more importance now than 2 years ago**. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Resources\n\n0. [base kernel][base]\n1. [Why LGBM][lgbm]\n2. [LGBM x XGB][lgbm-xgb]\n3. [Parameter Tuning][param-tuning]\n\n\n[base]: https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/comments\n[lgbm]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73762\n[lgbm-xgb]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/80484\n[param-tuning]: https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}