{"cells":[{"metadata":{"trusted":true,"_uuid":"332fa118684da9a9278d2caf565620686fd73a68"},"cell_type":"markdown","source":"## Elo Merchant Category Recommendation\n---\n> ***Help understand customer loyalty***\n\n![](https://www.cartaoelo.com.br/images/home-promo-new.jpg)\n\n---\n\n> ## Objective\n> <p style=\"text-align:justify\">Elo has built machine learning models to understand the most important aspects and preferences in their customers’ lifecycle, from food to shopping. But so far none of them is specifically tailored for an individual or profile. This is where you come in.</p>\n> ## Solution thought by me\n> In this kernel, I build a LGBM model that aggregates the `new_merchant_transactions.csv` and `historical_transactions.csv` tables to the main train table. New features are built by successive grouping on`card_id` and `month_lag`, in order to recover some information from the time serie.\n</div></div></div>\n\n\n> ## Notebook  Content\n> 1. [***Loading the data***](#1)\n> 1. [***Feature engineering***](#2)\n> 1. [***Training the model***](#3)\n> 1. [***Feature importance***](#4)\n> 1. [***Submission***](#5)\n> 1. [***Stacking***](#6)\n---"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error\nfrom tqdm import tqdm_notebook as tqdm, tqdm_pandas\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.linear_model import Ridge, BayesianRidge\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nimport gc\nimport time\nimport datetime","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a059a8dcf9d93a650f1ccaa8e2bfa3e087219f3"},"cell_type":"markdown","source":"## 1. Loading the data <a id=\"1\"></a> <br>\n\nFirst, we load the `new_merchant_transactions.csv` and `historical_transactions.csv`. In practice, these two files contain the same variables and the difference between the two tables only concern the position with respect to a reference date.  Also, booleans features are made numeric:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7a7877dff5c337c09ca111cdcbf527362c9217c7"},"cell_type":"code","source":"%%time\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b6b5a505bfecabd70c6dac0ee30386d074b03953"},"cell_type":"code","source":"%%time\nnew_transactions = pd.read_csv('../input/new_merchant_transactions.csv', parse_dates=['purchase_date'])\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv', parse_dates=['purchase_date'])\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"156acd1e83dfba2c2896561b75a6a5a7782cab1d"},"cell_type":"markdown","source":"We then load the main files, formatting the dates and extracting the target:"},{"metadata":{"trusted":true,"_uuid":"a24cf99ad6b0785b5af2b101e06b400e26360d1e"},"cell_type":"code","source":"%%time\ndef read_data(input_file):\n    df = pd.read_csv(input_file)\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['elapsed_time'] = (datetime.date(2018, 2, 1) - df['first_active_month'].dt.date).dt.days\n    return df\n\n# Read data train and test file\ntrain = read_data('../input/train.csv')\ntest = read_data('../input/test.csv')\n\ntarget = train['target']\ndel train['target']\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e50c9764e02edac89eaafd080b64ffca21987b0"},"cell_type":"markdown","source":"## 2.Feature engineering <a id=\"2\"></a> <br>\n* First, following [Robin Denz](https://www.kaggle.com/denzo123/a-closer-look-at-date-variables) and [konradb](https://www.kaggle.com/konradb/lgb-fe-lb-3-707) analysis, I define a few dates features.\n* Binarize the categorical variables where it makes sense"},{"metadata":{"_uuid":"281683448f9e67925acee26067dc09d25ae26dc9"},"cell_type":"markdown","source":"### Historical Transactions"},{"metadata":{"trusted":true,"_uuid":"6d6e3aebeeaa2a46cd4a80ca3d6de0db3b2a1838"},"cell_type":"code","source":"%%time\n\nhistorical_transactions['authorized_flag'] = historical_transactions['authorized_flag'].map({'Y':1, 'N':0})\nhistorical_transactions['category_1'] = historical_transactions['category_1'].map({'Y':1, 'N':0})\n\nhistorical_transactions['category_2x1'] = (historical_transactions['category_2'] == 1) + 0\nhistorical_transactions['category_2x2'] = (historical_transactions['category_2'] == 2) + 0\nhistorical_transactions['category_2x3'] = (historical_transactions['category_2'] == 3) + 0\nhistorical_transactions['category_2x4'] = (historical_transactions['category_2'] == 4) + 0\nhistorical_transactions['category_2x5'] = (historical_transactions['category_2'] == 5) + 0\n\nhistorical_transactions['category_3A'] = (historical_transactions['category_3'].astype(str) == 'A') + 0\nhistorical_transactions['category_3B'] = (historical_transactions['category_3'].astype(str) == 'B') + 0\nhistorical_transactions['category_3C'] = (historical_transactions['category_3'].astype(str) == 'C') + 0\n\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d0ccef73b0393a6d2122539620e40638ef56fd80"},"cell_type":"code","source":"%%time\ndef aggregate_historical_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'category_1': ['sum', 'mean'],\n        'category_2': ['nunique'],\n        'category_3A': ['sum'],\n        'category_3B': ['sum'],\n        'category_3C': ['sum'],\n        'category_2x1': ['sum','mean'],\n        'category_2x2': ['sum','mean'],\n        'category_2x3': ['sum','mean'],\n        'category_2x4': ['sum','mean'],\n        'category_2x5': ['sum','mean'],        \n        'city_id': ['nunique'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'merchant_category_id': ['nunique'],\n        'merchant_id': ['nunique'],\n        'month_lag': ['min', 'max'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp, 'max', 'min'],\n        'state_id': ['nunique'],\n        'subsector_id': ['nunique'],\n\n        }\n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['hist_' + '_'.join(col).strip() \n                           for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='hist_transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n\nhistory = aggregate_historical_transactions(historical_transactions)\nhistory.columns = ['hist_' + c if c != 'card_id' else c for c in history.columns]\ndisplay(history[:5])\n\ndel historical_transactions\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a1262cc77dccebd1b770e51f1c434774154d2de"},"cell_type":"markdown","source":"### New Transaction"},{"metadata":{"_uuid":"7493b32cb783d6fb6afdad60964eb41c9e42c2e3"},"cell_type":"markdown","source":"Then I define two functions that aggregate the info contained in these two tables. The first function aggregates the function by grouping on `card_id`:"},{"metadata":{"trusted":true,"_uuid":"c83bab7b5e07d2541a45b53f2ace0e4bb001f362"},"cell_type":"code","source":"%%time\nnew_transactions['authorized_flag'] = new_transactions['authorized_flag'].map({'Y':1, 'N':0})\n\nnew_transactions['category_1'] = new_transactions['category_1'].map({'Y':1, 'N':0})\nnew_transactions['category_3A'] = (new_transactions['category_3'].astype(str) == 'A') + 0\nnew_transactions['category_3B'] = (new_transactions['category_3'].astype(str) == 'B') + 0\nnew_transactions['category_3C'] = (new_transactions['category_3'].astype(str) == 'C') + 0\n\nnew_transactions['category_2x1'] = (new_transactions['category_2'] == 1) + 0\nnew_transactions['category_2x2'] = (new_transactions['category_2'] == 2) + 0\nnew_transactions['category_2x3'] = (new_transactions['category_2'] == 3) + 0\nnew_transactions['category_2x4'] = (new_transactions['category_2'] == 4) + 0\nnew_transactions['category_2x5'] = (new_transactions['category_2'] == 5) + 0\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82c25c7cd0d075195fb7bd63211c66f0dac9304b"},"cell_type":"code","source":"def aggregate_new_transactions(new_trans):    \n    \n    new_transactions['purchase_date'] = pd.DatetimeIndex(new_transactions['purchase_date']).astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'category_1': ['sum', 'mean'],\n        'category_2': ['nunique'],\n        'category_3A': ['sum'],\n        'category_3B': ['sum'],\n        'category_3C': ['sum'],     \n        'category_2x1': ['sum','mean'],\n        'category_2x2': ['sum','mean'],\n        'category_2x3': ['sum','mean'],\n        'category_2x4': ['sum','mean'],\n        'category_2x5': ['sum','mean'],        \n\n        'city_id': ['nunique'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'merchant_category_id': ['nunique'],\n        'merchant_id': ['nunique'],\n        'month_lag': ['min', 'max'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp, 'max', 'min'],\n        'state_id': ['nunique'],\n        'subsector_id': ['nunique']        \n        }\n    agg_new_trans = new_trans.groupby(['card_id']).agg(agg_func)\n    agg_new_trans.columns = ['new_' + '_'.join(col).strip() \n                           for col in agg_new_trans.columns.values]\n    agg_new_trans.reset_index(inplace=True)\n    \n    df = (new_trans.groupby('card_id')\n          .size()\n          .reset_index(name='new_transactions_count'))\n    \n    agg_new_trans = pd.merge(df, agg_new_trans, on='card_id', how='left')\n    \n    return agg_new_trans\n\nnew = aggregate_new_transactions(new_transactions)\nnew.columns = ['new_' + c if c != 'card_id' else c for c in new.columns]\ndisplay(new[:5])\n\ndel new_transactions\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"843a929ddea5e317f882c58c3b69f5e5a4476a38"},"cell_type":"markdown","source":"The second function first aggregates on the two variables `card_id` and `month_lag`. Then a second grouping is performed to aggregate over time:"},{"metadata":{"_uuid":"dcf0403c10b8ee817257a51e5edf8f1f81fcd593"},"cell_type":"markdown","source":"## 3. Training the model<a id=\"3\"></a> \nWe now train the model with the features we previously defined. A first step consists in merging all the dataframes:"},{"metadata":{"trusted":true,"_uuid":"d27264c6c7f0af6af7ba141177bfd38f7a68dec3"},"cell_type":"code","source":"%%time\nprint(train.shape)\nprint(test.shape)\n\ntrain = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')\n\nprint(train.shape)\nprint(test.shape)\n\ntrain = pd.merge(train, new, on='card_id', how='left')\ntest = pd.merge(test, new, on='card_id', how='left')\n\nprint(train.shape)\nprint(test.shape)\n\n# train = pd.merge(train, final_group, on='card_id')\n# test = pd.merge(test, final_group, on='card_id')\n# print(train.shape)\n# print(test.shape)\ndel history\ndel new\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e06ddc8d941360b1b3e32bc59221e5f6fe729763"},"cell_type":"markdown","source":"and to define the features we want to keep to train the model:"},{"metadata":{"trusted":true,"_uuid":"6b0eba3e733ecdbab96d631eb46d42453d82aa20"},"cell_type":"code","source":"features = [c for c in train.columns if c not in ['card_id', 'first_active_month']]\ncategorical_feats = [c for c in features if 'feature_' in c]\n\n# categorical_feats = ['feature_1', 'feature_2', 'feature_3']\n\nfor col in categorical_feats:\n    print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n    train[col] = lbl.transform(list(train[col].values.astype('str')))\n    test[col] = lbl.transform(list(test[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9008798ce0be96807c746912f8834653e1d58e7c"},"cell_type":"markdown","source":"## Alpha Value in Bayesian Ridge Regression\n\n* So, if the **alpha** value is **0,** it means that it is just an **Ordinary Least Squares Regression model.** So, the **larger is the alpha**, the **higher** is the **smoothness constraint.** So, the **smaller** the value of **alpha,** the **higher would be the magnitude of the coefficients.**"},{"metadata":{"trusted":true,"_uuid":"9ab40561e87810df82c5e8a2d66348f4fc0a0a8d"},"cell_type":"code","source":"from sklearn.model_selection import RepeatedKFold\nfolds = RepeatedKFold(n_splits=5, n_repeats=2, random_state=4520)\noof_ridge = np.zeros(train.shape[0])\npredictions_ridge = np.zeros(test.shape[0])\n\ntst_data = test.copy()\ntst_data.fillna((tst_data.mean()), inplace=True)\n\ntst_data = tst_data[features].values\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train, target)):\n    print(\"fold n°{}\".format(fold_+1))\n    trn_data, trn_y = train.iloc[trn_idx][features], target.iloc[trn_idx].values\n    val_data, val_y = train.iloc[val_idx][features], target.iloc[val_idx].values\n    \n    trn_data.fillna((trn_data.mean()), inplace=True)\n    val_data.fillna((val_data.mean()), inplace=True)\n    \n    trn_data = trn_data.values\n    val_data = val_data.values\n\n    clf = BayesianRidge()\n    clf.fit(trn_data, trn_y)\n    \n    oof_ridge[val_idx] = clf.predict(val_data)\n    predictions_ridge += clf.predict(tst_data) / 10\n\nnp.save('oof_ridge', oof_ridge)\nnp.save('predictions_ridge', predictions_ridge)\nnp.sqrt(mean_squared_error(target.values, oof_ridge))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c610f51450145101732f4e9ed3247f2a9fa0b091"},"cell_type":"code","source":"param = {'num_leaves': 31,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.01,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1,\n         \"nthread\": 4,\n         \"random_state\": 4590}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7a7377cd7401f2cbd13ea707fbc2a2bebe229a6"},"cell_type":"markdown","source":"# Light GBM\nWe now train the model. Here, we use a standard KFold split of the dataset in order to validate the results and to stop the training. Interstingly, during the writing of this kernel, the model was enriched adding new features, which improved the CV score. The variations observed on the CV were found to be quite similar to the variations on the LB: it seems that the current competition won't give us headaches to define the correct validation scheme:"},{"metadata":{"trusted":true,"_uuid":"b23550968ef3fb49ae0fcc5533551d702297c990","scrolled":false},"cell_type":"code","source":"from sklearn.model_selection import RepeatedKFold\nfolds = RepeatedKFold(n_splits=5, n_repeats=2, random_state=4520)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features], label=target.iloc[trn_idx], categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(train.iloc[val_idx][features], label=target.iloc[val_idx], categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=500, early_stopping_rounds = 150)\n    oof[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(test[features], num_iteration=clf.best_iteration) / 10\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"257095c29b97e52f8eb721fbefdea3cd2c9ff581"},"cell_type":"markdown","source":"# CATBOOST"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"8504aad214e4549666147260140e6ead43ebf420","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# %%time\n# from catboost import CatBoostRegressor\n# folds = KFold(n_splits=5, shuffle=True, random_state=15)\n# oof_cat = np.zeros(len(train))\n# predictions_cat = np.zeros(len(test))\n\n# for fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n#     print(\"fold n°{}\".format(fold_ + 1))\n#     trn_data, trn_y = train.iloc[trn_idx][features], target.iloc[trn_idx].values\n#     val_data, val_y = train.iloc[val_idx][features], target.iloc[val_idx].values\n#     print(\"-\" * 10 + \"Catboost \" + str(fold_) + \"-\" * 10)\n#     cb_model = CatBoostRegressor(iterations=3000, learning_rate=0.1, depth=8, l2_leaf_reg=20, bootstrap_type='Bernoulli',  eval_metric='RMSE', metric_period=50, od_type='Iter', od_wait=45, random_seed=17, allow_writing_files=False)\n#     cb_model.fit(trn_data, trn_y, eval_set=(val_data, val_y), cat_features=[], use_best_model=True, verbose=True)\n    \n#     oof_cat[val_idx] = cb_model.predict(val_data)\n#     predictions_cat += cb_model.predict(test[features]) / folds.n_splits\n    \n# np.save('oof_cat', oof_cat)\n# np.save('predictions_cat', predictions_cat)\n# np.sqrt(mean_squared_error(target.values, oof_cat))\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69fdc18dd97234d5a13e69ab5b970eeb7d116f7b"},"cell_type":"markdown","source":"# XGBOOST"},{"metadata":{"trusted":true,"_uuid":"6aefad9f04bf48dd776ca16b83400e7ef7c31d0d","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"%%time\n\nimport xgboost as xgb\n\nxgb_params = {'eta': 0.005, 'max_depth': 3, 'subsample': 0.8, 'colsample_bytree': 0.8, 'alpha':0.1,\n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True, 'random_state':folds}\n\n\nfolds = RepeatedKFold(n_splits=5, n_repeats=2, random_state=4520)\noof_xgb = np.zeros(len(train))\npredictions_xgb = np.zeros(len(test))\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print(\"fold n°{}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][features], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 11000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / 10\n    \nnp.save('oof_xgb', oof_xgb)\nnp.save('predictions_xgb', predictions_xgb)\nprint(\"RMSE : \",np.sqrt(mean_squared_error(target.values, oof_xgb)))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a1f0a866e05f8a450960e2d787a641fc35991a1"},"cell_type":"markdown","source":"## 4. Feature importance <a id=\"4\"></a> <br>\nFinally, we can have a look at the features that were used by the model:"},{"metadata":{"trusted":true,"_uuid":"d479e83032448481b40c216264a039cacdb2f9a1"},"cell_type":"code","source":"cols = (feature_importance_df[[\"feature\", \"importance\"]]\n        .groupby(\"feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.feature.isin(cols)]\n\nplt.figure(figsize=(14,16))\nsns.barplot(x=\"importance\",\n            y=\"feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"003ae1b1bd522b1b0d992ff220ed98d2a6d7477a"},"cell_type":"markdown","source":"## 5. Submission<a id=\"5\"></a> <br>\nNow, we just need to prepare the submission file:"},{"metadata":{"_uuid":"c53815a6af8dedbae98139add8f01fd2cd3289e1"},"cell_type":"markdown","source":"## Lightgbm"},{"metadata":{"trusted":true,"_uuid":"82d5ac08a13603b2a66c59d98584c4b709daee2d"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions\nsub_df.to_csv(\"submit_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3a537323519283dd65da6a16bbb5d6ee4f82d72"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_xgb\nsub_df.to_csv(\"submit_xgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f197baedf41262462000a9352bcd816a3e09cded"},"cell_type":"markdown","source":"## 6. Stacking Using LightGBM<a id=\"6\"></a> <br>"},{"metadata":{"trusted":true,"_uuid":"c1c6c1d40dbb2137ddd6404670b5f3dd3d381207"},"cell_type":"code","source":"train_stack = np.vstack([oof_ridge, oof, oof_xgb]).transpose()\ntest_stack = np.vstack([predictions_ridge, predictions,predictions_xgb]).transpose()\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_stack = np.zeros(train_stack.shape[0])\npredictions_stack = np.zeros(test_stack.shape[0])\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_stack, target)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data, trn_y = train_stack[trn_idx], target.iloc[trn_idx].values\n    val_data, val_y = train_stack[val_idx], target.iloc[val_idx].values\n\n    print(\"-\" * 10 + \"Ridge Regression\" + str(fold_) + \"-\" * 10)\n#     cb_model = CatBoostRegressor(iterations=3000, learning_rate=0.1, depth=8, l2_leaf_reg=20, bootstrap_type='Bernoulli',  eval_metric='RMSE', metric_period=50, od_type='Iter', od_wait=45, random_seed=17, allow_writing_files=False)\n#     cb_model.fit(trn_data, trn_y, eval_set=(val_data, val_y), cat_features=[], use_best_model=True, verbose=True)\n    clf = Ridge(alpha=100)\n    clf.fit(trn_data, trn_y)\n    \n    oof_stack[val_idx] = clf.predict(val_data)\n    predictions_stack += clf.predict(test_stack) / 5\n\n\nnp.sqrt(mean_squared_error(target.values, oof))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3a29986bdf91fcb42c53479cd244aae5d7895a14"},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/sample_submission.csv')\nsample_submission['target'] = predictions_stack\nsample_submission.to_csv('RLS.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}