{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Elo world \n\nIn this kernel, I build a LGBM model that aggregates the `new_merchant_transactions.csv` and `historical_transactions.csv` tables to the main train table. New features are built by successive grouping on`card_id` and `month_lag`, in order to recover some information from the time serie.\n\nDuring the competition, I took into account the enlightments provided by others kernels, and included a few features that appeared to be important. In particular, I closely looked at the following kernels (ordered by release time):\n1.  [You're Going to Want More Categories [LB 3.737] by Peter Hurford](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737)\n2. [EloDA with Feature Engineering and Stacking by Bojan Tunguz](https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking)\n3. [A Closer Look at Date Variables by Robin Denz](https://www.kaggle.com/denzo123/a-closer-look-at-date-variables)\n4. [LGB + FE (LB 3.707) by Konrad Banachewicz](https://www.kaggle.com/konradb/lgb-fe-lb-3-707)\n5. [My first kernel (3.699) by Chau Ngoc Huynh](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/)\n\n## Notebook  Content\n1. [Loading the data](#1)\n1. [Feature engineering](#2)\n1. [Training the model](#3)\n1. [Feature importance](#4)\n1. [Submission](#5)","metadata":{"_uuid":"332fa118684da9a9278d2caf565620686fd73a68","trusted":true}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nimport xgboost as xgb\nfrom xgboost import XGBRegressor\nfrom sklearn import tree\nfrom sklearn.svm import SVR\nfrom sklearn import linear_model\nfrom sklearn.neighbors import KNeighborsRegressor\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom sklearn.model_selection import KFold\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error\nimport gc\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.max_columns', 500)\n\ndebug = False","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-09T09:14:05.488515Z","iopub.execute_input":"2022-12-09T09:14:05.488972Z","iopub.status.idle":"2022-12-09T09:14:05.500046Z","shell.execute_reply.started":"2022-12-09T09:14:05.488928Z","shell.execute_reply":"2022-12-09T09:14:05.49841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3. Training the model\nWe now train the model with the features we previously defined. A first step consists in merging all the dataframes:","metadata":{"_uuid":"dcf0403c10b8ee817257a51e5edf8f1f81fcd593"}},{"cell_type":"code","source":"train = pd.read_csv('../input/elo-processed-data/train.csv')\ntest = pd.read_csv('../input/elo-processed-data/test.csv')\ntarget = train['target']\ntarget.head(3)\ndel train['target']\n\n\nfeatures = [c for c in train.columns if c not in ['card_id', 'first_active_month', 'Unnamed: 0']]\n#features = [f for f in features if f not in unimportant_features]\ncategorical_feats = ['feature_2', 'feature_3']\n\nif debug:\n    train = train.iloc[:200,]\n    test = test.iloc[:20,]\n    target = target[:200]\n    \ntrain[categorical_feats] = train[categorical_feats].astype('category')","metadata":{"_uuid":"d27264c6c7f0af6af7ba141177bfd38f7a68dec3","execution":{"iopub.status.busy":"2022-12-09T09:14:05.5027Z","iopub.execute_input":"2022-12-09T09:14:05.503236Z","iopub.status.idle":"2022-12-09T09:14:13.605427Z","shell.execute_reply.started":"2022-12-09T09:14:05.503189Z","shell.execute_reply":"2022-12-09T09:14:13.604287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_hot_encoder(df, categorical_columns = categorical_feats, nan_as_category = True):\n    original_columns = df.columns.tolist()\n\n    #categorical_columns = list(filter(lambda c: c in ['object'], df.dtypes))\n    df = pd.get_dummies(df, columns=categorical_columns, dummy_na=nan_as_category)\n\n    new_columns = list(filter(lambda c: c not in original_columns, df.columns))\n    return df, new_columns\n\ntrain_onehot, train_cols = one_hot_encoder(train[features], categorical_columns = categorical_feats, nan_as_category=False)\ntest_onehot, test_cols = one_hot_encoder(test[features], categorical_columns = categorical_feats, nan_as_category=False)\n\n# 缺失值采用中位数填补\nimp = SimpleImputer(add_indicator=False, strategy='median')\ncolnames = train_onehot.columns\ntrain_onehot = pd.DataFrame(imp.fit_transform(train_onehot))\ntrain_onehot.columns = colnames\n\ncolnames = test_onehot.columns\ntest_onehot = pd.DataFrame(imp.fit_transform(test_onehot))\ntest_onehot.columns = colnames","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:13.849452Z","iopub.execute_input":"2022-12-09T09:14:13.850275Z","iopub.status.idle":"2022-12-09T09:14:24.316958Z","shell.execute_reply.started":"2022-12-09T09:14:13.850222Z","shell.execute_reply":"2022-12-09T09:14:24.315597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# todo： \n1. 超参数搜索\n2. 每个fold的训练，测试误差\n3. 综合比较结果图 -->","metadata":{}},{"cell_type":"markdown","source":"## 1.决策树","metadata":{}},{"cell_type":"code","source":"#grid search\n\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#tree\nfolds = KFold(n_splits=5, shuffle=True, random_state=42)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\ntrain_losses = []\nval_losses = []\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n    \n    clf = tree.DecisionTreeRegressor(criterion='squared_error', \n                                     random_state=42,\n                                     splitter='random', \n                                     max_depth=4, \n                                     min_samples_leaf=10,\n                                     min_samples_split=20)\n\n    clf.fit(X_train,y_train, )\n    #clf.save_model(f'tree_reg_fold{fold_}')\n    clfs.append(clf)\n    oof_preds[val_idx] = clf.predict(X_valid)\n    train_losses.append(mean_squared_error(clf.predict(X_train), y_train)**0.5)\n    val_losses.append(mean_squared_error(oof_preds[val_idx], target[val_idx])**0.5)\n    print(\"fold {} train loss: {:<8.5f}\".format(fold_, train_losses[-1]))\n    print(\"fold {} val loss: {:<8.5f}\".format(fold_, val_losses[-1]))\n    \n    #del X_train, y_train, X_valid, y_valid\n    gc.collect()\n    \n    predictions += clf.predict(test_onehot) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_preds, target)**0.5))\n","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:56.223382Z","iopub.execute_input":"2022-12-09T09:14:56.223992Z","iopub.status.idle":"2022-12-09T09:15:03.775166Z","shell.execute_reply.started":"2022-12-09T09:14:56.223939Z","shell.execute_reply":"2022-12-09T09:15:03.77363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plt.figure(figure = (10.8/2, 9.6/2))\ntr_l, = plt.plot(train_losses, label = 'train loss')\nva_l, = plt.plot(val_losses, label = 'val loss')\nplt.legend((tr_l, va_l), ['train loss', 'val loss'],loc = 'upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:31.715222Z","iopub.execute_input":"2022-12-09T09:14:31.715611Z","iopub.status.idle":"2022-12-09T09:14:31.939513Z","shell.execute_reply.started":"2022-12-09T09:14:31.715567Z","shell.execute_reply":"2022-12-09T09:14:31.938294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 画出决策树\nimport graphviz \ndot_data = tree.export_graphviz(clf, out_file=None, \n                      feature_names=X_train.columns, \n                      filled=True, rounded=True,  \n                      special_characters=True)  \ngraph = graphviz.Source(dot_data)  \ngraph ","metadata":{"execution":{"iopub.status.busy":"2022-12-09T08:50:29.720308Z","iopub.execute_input":"2022-12-09T08:50:29.720934Z","iopub.status.idle":"2022-12-09T08:50:31.109479Z","shell.execute_reply.started":"2022-12-09T08:50:29.720883Z","shell.execute_reply":"2022-12-09T08:50:31.108008Z"}}},{"cell_type":"code","source":"#变量重要性\nreg_importance = pd.Series(clf.feature_importances_,index = X_train.columns, name = 'Var')\nplt.figure(figsize=(8,8))\nreg_importance.sort_values(ascending=False)[:20].sort_values().plot(kind='barh')\nplt.title(\"Variances Importances\")\nplt.savefig('xgb_importances.png')","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:31.941169Z","iopub.execute_input":"2022-12-09T09:14:31.941553Z","iopub.status.idle":"2022-12-09T09:14:32.532372Z","shell.execute_reply.started":"2022-12-09T09:14:31.941519Z","shell.execute_reply":"2022-12-09T09:14:32.530932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#random forest\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\ntrain_losses = []\nval_losses = []\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n    \n    clf = RandomForestRegressor(n_estimators=100, max_features=3, max_depth=4, \n                            bootstrap=True, oob_score=True, random_state=1)\n    clf.fit(X_train,y_train, )\n    #clf.save_model(f'tree_reg_fold{fold_}')\n    clfs.append(clf)\n    oof_preds[val_idx] = clf.predict(X_valid)\n    train_losses.append(mean_squared_error(clf.predict(X_train), y_train)**0.5)\n    val_losses.append(mean_squared_error(oof_preds[val_idx], target[val_idx])**0.5)\n    print(\"fold {} train loss: {:<8.5f}\".format(fold_, train_losses[-1]))\n    print(\"fold {} val loss: {:<8.5f}\".format(fold_, val_losses[-1]))\n    \n    #del X_train, y_train, X_valid, y_valid\n    gc.collect()\n    \n    predictions += clf.predict(test_onehot) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_preds, target)**0.5))\n","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:15:09.838908Z","iopub.execute_input":"2022-12-09T09:15:09.839332Z","iopub.status.idle":"2022-12-09T09:16:30.107037Z","shell.execute_reply.started":"2022-12-09T09:15:09.839301Z","shell.execute_reply":"2022-12-09T09:16:30.10583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plt.figure(figure = (10.8/2, 9.6/2))\ntr_l, = plt.plot(train_losses, label = 'train loss')\nva_l, = plt.plot(val_losses, label = 'val loss')\nplt.legend((tr_l, va_l), ['train loss', 'val loss'],loc = 'upper left')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#变量重要性\nreg_importance = pd.Series(clf.feature_importances_,index = X_train.columns, name = 'Var')\nplt.figure(figsize=(8,8))\nreg_importance.sort_values(ascending=False)[:20].sort_values().plot(kind='barh')\nplt.title(\"Variances Importances\")\nplt.savefig('xgb_importances.png')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SVR\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\ntrain_losses = []\nval_losses = []\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n    \n    clf = SVR(C=2,gamma=2)\n    clf.fit(X_train,y_train, )\n    #clf.save_model(f'tree_reg_fold{fold_}')\n    clfs.append(clf)\n    oof_preds[val_idx] = clf.predict(X_valid)\n    train_losses.append(mean_squared_error(clf.predict(X_train), y_train)**0.5)\n    val_losses.append(mean_squared_error(oof_preds[val_idx], target[val_idx])**0.5)\n    print(\"fold {} train loss: {:<8.5f}\".format(fold_, train_losses[-1]))\n    print(\"fold {} val loss: {:<8.5f}\".format(fold_, val_losses[-1]))\n    \n    del X_train, y_train, X_valid, y_valid\n    gc.collect()\n    \n    predictions += clf.predict(test_onehot) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_preds, target)**0.5))\n","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:51.60986Z","iopub.status.idle":"2022-12-09T09:14:51.610879Z","shell.execute_reply.started":"2022-12-09T09:14:51.610531Z","shell.execute_reply":"2022-12-09T09:14:51.610566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plt.figure(figure = (10.8/2, 9.6/2))\ntr_l, = plt.plot(train_losses, label = 'train loss')\nva_l, = plt.plot(val_losses, label = 'val loss')\nplt.legend((tr_l, va_l), ['train loss', 'val loss'],loc = 'upper left')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# KNN\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\ntrain_losses = []\nval_losses = []\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n    \n    #等权重情况\n    clf = KNeighborsRegressor(n_neighbors=3, \n                               p=1, \n                               metric='minkowski', \n                               weights='uniform')\n\n    #为便于探索，先不将X标准化处理\n    clf = clf.fit(X_train,y_train)\n    \n    #clf.save_model(f'tree_reg_fold{fold_}')\n    clfs.append(clf)\n    oof_preds[val_idx] = clf.predict(X_valid)\n    train_losses.append(mean_squared_error(clf.predict(X_train), y_train)**0.5)\n    val_losses.append(mean_squared_error(oof_preds[val_idx], target[val_idx])**0.5)\n    print(\"fold {} train loss: {:<8.5f}\".format(fold_, train_losses[-1]))\n    print(\"fold {} val loss: {:<8.5f}\".format(fold_, val_losses[-1]))\n    \n    del X_train, y_train, X_valid, y_valid\n    gc.collect()\n    \n    predictions += clf.predict(test_onehot) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_preds, target)**0.5))","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:51.61282Z","iopub.status.idle":"2022-12-09T09:14:51.613803Z","shell.execute_reply.started":"2022-12-09T09:14:51.613456Z","shell.execute_reply":"2022-12-09T09:14:51.61349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plt.figure(figure = (10.8/2, 9.6/2))\ntr_l, = plt.plot(train_losses, label = 'train loss')\nva_l, = plt.plot(val_losses, label = 'val loss')\nplt.legend((tr_l, va_l), ['train loss', 'val loss'],loc = 'upper left')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 岭回归\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n    \n    #等权重情况\n    clf = linear_model.Ridge(alpha=.5)\n\n    #为便于探索，先不将X标准化处理\n    clf = clf.fit(X_train,y_train)\n    \n    #clf.save_model(f'tree_reg_fold{fold_}')\n    clfs.append(clf)\n        oof_preds[val_idx] = clf.predict(X_valid)\n    train_losses.append(mean_squared_error(clf.predict(X_train), y_train)**0.5)\n    val_losses.append(mean_squared_error(oof_preds[val_idx], target[val_idx])**0.5)\n    print(\"fold {} train loss: {:<8.5f}\".format(fold_, train_losses[-1]))\n    print(\"fold {} val loss: {:<8.5f}\".format(fold_, val_losses[-1]))\n    \n    del X_train, y_train, X_valid, y_valid\n    gc.collect()\n    \n    predictions += clf.predict(test_onehot) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_preds, target)**0.5))\n","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:51.615674Z","iopub.status.idle":"2022-12-09T09:14:51.616651Z","shell.execute_reply.started":"2022-12-09T09:14:51.616322Z","shell.execute_reply":"2022-12-09T09:14:51.616357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 4. Feature importance\n这里都是sklearn的包，通用的方法","metadata":{"_uuid":"9a1f0a866e05f8a450960e2d787a641fc35991a1"}},{"cell_type":"code","source":"#在整个数据集上训练\nX_train, y_train = train_onehot, target\nclf = tree.DecisionTreeRegressor(max_depth=4)\nclf.fit(X_train,y_train, )\n#clf.save_model('xgb_reg_all')","metadata":{"execution":{"iopub.status.busy":"2022-12-09T09:14:51.618571Z","iopub.status.idle":"2022-12-09T09:14:51.619568Z","shell.execute_reply.started":"2022-12-09T09:14:51.619252Z","shell.execute_reply":"2022-12-09T09:14:51.619284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"reg_importance = pd.Series(clf.feature_importances_,index = X_train.columns, name = 'Var')\nplt.figure(figsize=(8,8))\nreg_importance.sort_values(ascending=False)[:30].sort_values().plot(kind='barh')\nplt.title(\"Variances Importances\")\nplt.savefig('xgb_importances.png')","metadata":{"_uuid":"d479e83032448481b40c216264a039cacdb2f9a1","execution":{"iopub.status.busy":"2022-12-09T09:14:51.621363Z","iopub.status.idle":"2022-12-09T09:14:51.622365Z","shell.execute_reply.started":"2022-12-09T09:14:51.622033Z","shell.execute_reply":"2022-12-09T09:14:51.622072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5. Submission\nNow, we just need to prepare the submission file:","metadata":{"_uuid":"003ae1b1bd522b1b0d992ff220ed98d2a6d7477a"}},{"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions\nsub_df.to_csv(\"submit.csv\", index=False)","metadata":{"_uuid":"82d5ac08a13603b2a66c59d98584c4b709daee2d","execution":{"iopub.status.busy":"2022-12-09T09:14:51.624217Z","iopub.status.idle":"2022-12-09T09:14:51.625223Z","shell.execute_reply.started":"2022-12-09T09:14:51.624845Z","shell.execute_reply":"2022-12-09T09:14:51.624877Z"},"trusted":true},"execution_count":null,"outputs":[]}]}