{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Elo world \n\nIn this kernel, I build a LGBM model that aggregates the `new_merchant_transactions.csv` and `historical_transactions.csv` tables to the main train table. New features are built by successive grouping on`card_id` and `month_lag`, in order to recover some information from the time serie.\n\nDuring the competition, I took into account the enlightments provided by others kernels, and included a few features that appeared to be important. In particular, I closely looked at the following kernels (ordered by release time):\n1.  [You're Going to Want More Categories [LB 3.737] by Peter Hurford](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737)\n2. [EloDA with Feature Engineering and Stacking by Bojan Tunguz](https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking)\n3. [A Closer Look at Date Variables by Robin Denz](https://www.kaggle.com/denzo123/a-closer-look-at-date-variables)\n4. [LGB + FE (LB 3.707) by Konrad Banachewicz](https://www.kaggle.com/konradb/lgb-fe-lb-3-707)\n5. [My first kernel (3.699) by Chau Ngoc Huynh](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/)\n\n## Notebook  Content\n1. [Loading the data](#1)\n1. [Feature engineering](#2)\n1. [Training the model](#3)\n1. [Feature importance](#4)\n1. [Submission](#5)","metadata":{"_uuid":"332fa118684da9a9278d2caf565620686fd73a68","trusted":true}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nimport xgboost as xgb\nfrom xgboost import XGBRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom sklearn.model_selection import KFold\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error\nimport gc\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.max_columns', 500)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-21T03:12:48.517576Z","iopub.execute_input":"2022-11-21T03:12:48.517975Z","iopub.status.idle":"2022-11-21T03:12:48.525437Z","shell.execute_reply.started":"2022-11-21T03:12:48.517947Z","shell.execute_reply":"2022-11-21T03:12:48.524443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3. Training the model\nWe now train the model with the features we previously defined. A first step consists in merging all the dataframes:","metadata":{"_uuid":"dcf0403c10b8ee817257a51e5edf8f1f81fcd593"}},{"cell_type":"code","source":"train = pd.read_csv('../input/elo-processed-data/train.csv')\ntest = pd.read_csv('../input/elo-processed-data/test.csv')\ntarget = train['target']\ndel train['target']","metadata":{"_uuid":"d27264c6c7f0af6af7ba141177bfd38f7a68dec3","execution":{"iopub.status.busy":"2022-11-21T03:12:48.527253Z","iopub.execute_input":"2022-11-21T03:12:48.527795Z","iopub.status.idle":"2022-11-21T03:12:53.121947Z","shell.execute_reply.started":"2022-11-21T03:12:48.527745Z","shell.execute_reply":"2022-11-21T03:12:53.120719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [c for c in train.columns if c not in ['card_id', 'first_active_month', 'Unnamed: 0']]\n#features = [f for f in features if f not in unimportant_features]\ncategorical_feats = ['feature_2', 'feature_3']","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:12:53.123050Z","iopub.execute_input":"2022-11-21T03:12:53.123301Z","iopub.status.idle":"2022-11-21T03:12:53.128715Z","shell.execute_reply.started":"2022-11-21T03:12:53.123278Z","shell.execute_reply":"2022-11-21T03:12:53.127813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We then set the hyperparameters of the LGBM model, these parameters are obtained by an [bayesian optimization done in another kernel](https://www.kaggle.com/fabiendaniel/hyperparameter-tuning/edit):","metadata":{"_uuid":"e67399b25b3886e306b274069ab240af7d060397"}},{"cell_type":"code","source":"param = {\n    'objective': 'reg:linear', \n    'eval_metric': 'rmse', \n    'silent': True, \n    'booster': 'gbtree', \n    'n_jobs': 4, \n    'n_estimators': 2500, \n    #'tree_method': 'gpu_hist', \n    'grow_policy': 'lossguide', \n    'max_depth': 12, \n    'seed': 538, \n    'colsample_bylevel': 0.9, \n    'colsample_bytree': 0.8, \n    'gamma': 0.0001, \n    'learning_rate': 0.006, \n    'max_bin': 128, \n    'max_leaves': 47, \n    'min_child_weight': 40, \n    'reg_alpha': 10.0, \n    'reg_lambda': 10.0, \n    'subsample': 0.9}","metadata":{"_uuid":"c610f51450145101732f4e9ed3247f2a9fa0b091","execution":{"iopub.status.busy":"2022-11-21T03:12:53.130693Z","iopub.execute_input":"2022-11-21T03:12:53.130956Z","iopub.status.idle":"2022-11-21T03:12:53.140049Z","shell.execute_reply.started":"2022-11-21T03:12:53.130933Z","shell.execute_reply":"2022-11-21T03:12:53.139108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We now train the model. Here, we use a standard KFold split of the dataset in order to validate the results and to stop the training. Interstingly, during the writing of this kernel, the model was enriched adding new features, which improved the CV score. **The variations observed on the CV were found to be quite similar to the variations on the LB**: it seems that the current competition won't give us headaches to define the correct validation scheme:","metadata":{"_uuid":"b7a7377cd7401f2cbd13ea707fbc2a2bebe229a6"}},{"cell_type":"code","source":"train = train\ntest = test\ntarget = target","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:30:41.233088Z","iopub.execute_input":"2022-11-21T03:30:41.233446Z","iopub.status.idle":"2022-11-21T03:30:41.239129Z","shell.execute_reply.started":"2022-11-21T03:30:41.233419Z","shell.execute_reply":"2022-11-21T03:30:41.238071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[categorical_feats] = train[categorical_feats].astype('category')","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:30:42.708349Z","iopub.execute_input":"2022-11-21T03:30:42.708717Z","iopub.status.idle":"2022-11-21T03:30:42.716192Z","shell.execute_reply.started":"2022-11-21T03:30:42.708678Z","shell.execute_reply":"2022-11-21T03:30:42.715234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_hot_encoder(df, categorical_columns = categorical_feats, nan_as_category = True):\n    original_columns = df.columns.tolist()\n\n    #categorical_columns = list(filter(lambda c: c in ['object'], df.dtypes))\n    df = pd.get_dummies(df, columns=categorical_columns, dummy_na=nan_as_category)\n\n    new_columns = list(filter(lambda c: c not in original_columns, df.columns))\n    return df, new_columns\n\ntrain_onehot, train_cols = one_hot_encoder(train[features], categorical_columns = categorical_feats, nan_as_category=False)\ntest_onehot, test_cols = one_hot_encoder(test[features], categorical_columns = categorical_feats, nan_as_category=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:30:43.837682Z","iopub.execute_input":"2022-11-21T03:30:43.838040Z","iopub.status.idle":"2022-11-21T03:30:43.856119Z","shell.execute_reply.started":"2022-11-21T03:30:43.838013Z","shell.execute_reply":"2022-11-21T03:30:43.854834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#xgb\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n    \n    model = xgb.XGBRegressor(**param)\n    model.fit(\n        X_train, y_train,\n        # eval_set=[(X_train, y_train), (X_valid, y_valid)],\n        eval_set=[(X_valid, y_valid)],\n        verbose=1000, #eval_metric='rmse',\n        early_stopping_rounds=500,\n    )\n    \n    model.save_model(f'xsg_reg_fold{fold_}')\n    clfs.append(model)\n    oof_preds[val_idx] = model.predict(X_valid, ntree_limit=model.best_ntree_limit)\n    \n    del X_train, y_train, X_valid, y_valid\n    gc.collect()\n    \n    predictions += model.predict(test_onehot, ntree_limit=model.best_ntree_limit) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:58:03.430059Z","iopub.execute_input":"2022-11-21T03:58:03.430422Z","iopub.status.idle":"2022-11-21T03:58:28.023113Z","shell.execute_reply.started":"2022-11-21T03:58:03.430393Z","shell.execute_reply":"2022-11-21T03:58:28.022409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 4. Feature importance\nFinally, we can have a look at the features that were used by the model:","metadata":{"_uuid":"9a1f0a866e05f8a450960e2d787a641fc35991a1"}},{"cell_type":"code","source":"X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\nX_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\nmodel.fit(\n    X_train, y_train,\n    eval_set=[(X_valid, y_valid)],\n    verbose=1000, #eval_metric='rmse',\n    early_stopping_rounds=500,\n)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:53:22.992385Z","iopub.execute_input":"2022-11-21T03:53:22.992755Z","iopub.status.idle":"2022-11-21T03:53:26.481445Z","shell.execute_reply.started":"2022-11-21T03:53:22.992725Z","shell.execute_reply":"2022-11-21T03:53:26.480374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#在整个数据集上训练\nX_train, y_train = train_onehot, target\nmodel = xgb.XGBRegressor(**param)\nmodel.fit(\n    X_train, y_train,\n    # eval_set=[(X_train, y_train), (X_valid, y_valid)],\n    verbose=0, \n)\nmodel.save_model('xgb_reg_all')","metadata":{"execution":{"iopub.status.busy":"2022-11-21T03:47:18.851287Z","iopub.execute_input":"2022-11-21T03:47:18.852425Z","iopub.status.idle":"2022-11-21T03:47:30.105392Z","shell.execute_reply.started":"2022-11-21T03:47:18.852371Z","shell.execute_reply":"2022-11-21T03:47:30.104756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"reg_importance = pd.Series(model.feature_importances_,index = X_train.columns, name = 'Var')\nplt.figure(figsize=(8,8))\nreg_importance.sort_values(ascending=False)[:30].sort_values().plot(kind='barh')\nplt.title(\"Variances Importances\")\nplt.savefig('xgb_importances.png')","metadata":{"_uuid":"d479e83032448481b40c216264a039cacdb2f9a1","execution":{"iopub.status.busy":"2022-11-21T03:47:30.106475Z","iopub.execute_input":"2022-11-21T03:47:30.106917Z","iopub.status.idle":"2022-11-21T03:47:30.495143Z","shell.execute_reply.started":"2022-11-21T03:47:30.106893Z","shell.execute_reply":"2022-11-21T03:47:30.494119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5. Submission\nNow, we just need to prepare the submission file:","metadata":{"_uuid":"003ae1b1bd522b1b0d992ff220ed98d2a6d7477a"}},{"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions\nsub_df.to_csv(\"submit.csv\", index=False)","metadata":{"_uuid":"82d5ac08a13603b2a66c59d98584c4b709daee2d","execution":{"iopub.status.busy":"2022-11-21T03:47:30.497360Z","iopub.execute_input":"2022-11-21T03:47:30.497607Z","iopub.status.idle":"2022-11-21T03:47:30.506139Z","shell.execute_reply.started":"2022-11-21T03:47:30.497585Z","shell.execute_reply":"2022-11-21T03:47:30.505077Z"},"trusted":true},"execution_count":null,"outputs":[]}]}